Meta-Harness, explained · Part 1 of 6 · Covers Title, Figure 1, Abstract, §1

The Big Idea: Let an Agent Write the Harness

Why the code around a language model can matter as much as the model, and the paper's proposal: a coding agent that reads every earlier attempt and writes a better one. We read the title, Figure 1, the abstract and §1, and we run one small model with four different harnesses.

Meta-Harness: End-to-End Optimization of Model Harnesses. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn. COLM 2026, 2026. arXiv:2603.28052

A language model answers what it is shown. Someone writes the code that decides what it is shown, and that code can move the score as much as a better model can. This paper asks a coding agent to write that code, after it reads every earlier attempt in full.

Each piece below shows the exact lines of the paper, then explains them.

The title

The harness: the code around the model1. Storekeep past cases2. Retrievefind related ones3. Presentbuild the prompt4. The modelfrozen weights5. Readtext to an answerOnly box 4 is the model. Boxes 1, 2, 3 and 5 are code: change them, and the score changes.
A harness as five steps. Four of them are code that you can change. Only step 4 is the model, and the paper keeps it frozen.

Figure 1: the results at a glance

The abstract

The Meta-Harness loop, in one pictureProposer agentreads any files it wantsNew harnessone Python fileEvaluaterun it on search tasksFilesystem: one folder per harnessits code, its scores, its full execution traces
The outer loop of Meta-Harness. The proposer reads the filesystem, writes a harness, the harness is scored on search tasks, and everything goes into a new folder. Part 2 reads the paper's own picture of it, Figure 2.

The 6× gap

The same gap, on a laptop

We can see this effect ourselves with a small open model. The model is Qwen2.5-1.5B-Instruct, and its weights never change. The task is Symptom2Disease, one of the paper's three text-classification datasets: read a patient's description of symptoms and name one of 22 diseases. We score 60 fixed cases from its test split.

There is one function that calls the model. Every harness goes through it, so the only thing that can change is what the harness puts in and how it reads what comes out:

python
def call_model(messages, max_new_tokens=12):
    """The only place where the model runs. A harness decides what goes into `messages`."""
    prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
    ids = tok(prompt, return_tensors='pt')['input_ids'].to(DEV)
    with torch.no_grad():
        out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    return tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True).strip(), ids.shape[1]

do_sample=False means greedy decoding: the model always takes its most likely next token, so a run gives the same output each time. The four harnesses are four small functions. Each returns the messages for the model and the function that reads the answer:

python
def parse_strict(output):
    """Accept the answer only if the whole output is exactly one label."""
    return norm(output) if norm(output) in LABELS else None


def parse_lenient(output):
    """Accept the first label that occurs anywhere in the output (longest labels first)."""
    o = norm(output)
    for lab in sorted(LABELS, key=len, reverse=True):
        if lab in o:
            return lab
    return None
python
def h_bare(text):
    return [{'role': 'user', 'content': f'Patient: {text}\nWhat is the diagnosis?'}], parse_strict


def h_labels(text):
    return [{'role': 'system', 'content': LABEL_LIST + ' Answer with one diagnosis from the list only.'},
            {'role': 'user', 'content': f'Patient: {text}\nDiagnosis:'}], parse_strict


def h_labels_lenient(text):
    msgs, _ = h_labels(text)
    return msgs, parse_lenient


def h_retrieval(text):
    shots = ''.join(f'Patient: {t}\nDiagnosis: {y}\n\n' for t, y in retrieve(text, K_SHOTS))
    return [{'role': 'system', 'content': LABEL_LIST + ' Answer with one diagnosis from the list only.'},
            {'role': 'user', 'content': f'{shots}Patient: {text}\nDiagnosis:'}], parse_lenient

Harness A asks the bare question. Harness B also lists the 22 diseases and asks for one of them. Harness C has the same prompt as B, but its reader accepts a disease name inside a longer answer. Harness D adds the 5 most similar solved cases from the 853 training cases, found by TF-IDF word overlap. This is retrieval, the "retrieve" job of the abstract. The results:

plain text
== 1. One frozen model, one task, four harnesses
model: Qwen/Qwen2.5-1.5B-Instruct on mps, greedy decoding, weights never change
task: Symptom2Disease, 22 labels, memory 853 labeled examples, eval 60 test examples
harness               accuracy  right  input tokens
A_bare                   0.000   0/60          72.4
B_labels                 0.433  26/60         145.4
C_labels_lenient         0.517  31/60         145.4
D_labels_retrieval       0.767  46/60         368.0
time for 4 x 60 model calls: 350 s
the first outputs of harness A (12 new tokens at most):
  'Based on your symptoms of skin irritations, rashes,'
  'Based on your symptoms of frequent coughing, difficulty breathing,'
best 0.767, worst 0.000, gap 76.7 points, ratio undefined (worst is 0)
Same model, same 60 test cases, four harnessesA: bare questionA: bare question: 0.0%0.0%B: + label listB: + label list: 43.3%43.3%C: + lenient readerC: + lenient reader: 51.7%51.7%D: + 5 retrieved casesD: + 5 retrieved cases: 76.7%76.7%Qwen2.5-1.5B-Instruct on Symptom2Disease (22 diseases). Accuracy on 60 fixed test cases.
Our run: one frozen model, four harnesses, 60 test cases. The accuracy goes from 0% to 76.7%, and the weights never changed.

Harness A scores zero, and the outputs show why. The model starts politely, "Based on your symptoms of...", and the 12-token limit cuts it off before it names a disease. The strict reader then finds no disease name. The model may know the answer; the harness never lets it say it. The ratio of best to worst is not even defined, because the worst is 0.

Each later harness fixes one thing. The label list (B) tells the model what form the answer must have: 26 right. The lenient reader (C) accepts "Urinary tract infection (UTI)" as "urinary tract infection": 31 right. Five retrieved cases (D) show the model how similar patients were labeled: 46 right. D uses about 2.5 times the input tokens of B, which is the price of the examples.

This is the 6× claim at toy scale, and it has the same lesson. Nothing about the model changed. Small code choices, such as an output limit and a reader, decided between 0% and 77%. Be careful with the size of these numbers: 60 cases is a small sample, and one case is 1.7 points.

What the model really sees, two ways

A harness builds messages, but the model reads one long string. Our first method writes that string by hand in Qwen's chat format. The second method asks the tokenizer's own chat template, which is the library's way. If our understanding is right, the two strings are equal, character for character:

python
def chatml_by_hand(messages):
    """Qwen2.5's chat format: each turn is <|im_start|>role, newline, text, <|im_end|>, newline."""
    s = ''
    if messages[0]['role'] != 'system':                                  # Qwen adds its default system turn
        s += '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n'
    for m in messages:
        s += f"<|im_start|>{m['role']}\n{m['content']}<|im_end|>\n"
    return s + '<|im_start|>assistant\n'                                  # the model writes from here


ours = chatml_by_hand(msgs)
lib = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False)
log(f'Method 1 length {len(ours)} chars, Method 2 length {len(lib)} chars, identical: {ours == lib}')
plain text
== 2. The prompt the model sees: built by hand (Method 1) and by the chat template (Method 2)
Method 1 length 611 chars, Method 2 length 611 chars, identical: True
harness A (no system turn): identical: True
the start of the prompt of harness B, as the model reads it:
  | <|im_start|>system
  | Possible diagnoses: allergy, arthritis, bronchial asthma, cervical spondylosis, chicken pox, common cold, deng ...
  | <|im_start|>user

Both methods give the same 611 characters. One detail is a harness choice that nobody wrote: when harness A sends no system message, the template adds Qwen's default one. A harness can put text in front of the model that its author never typed. Only a full trace of the prompt shows it.

We also count the accuracy in two ways, by hand and with scikit-learn's accuracy_score. They agree for all four harnesses:

plain text
== 3. Accuracy counted by hand (Method 1) and by scikit-learn (Method 2)
A_bare               ours 0.0000  sklearn 0.0000  agree: True
B_labels             ours 0.4333  sklearn 0.4333  agree: True
C_labels_lenient     ours 0.5167  sklearn 0.5167  agree: True
D_labels_retrieval   ours 0.7667  sklearn 0.7667  agree: True

Harness engineering is done by hand

Why earlier optimizers are a poor fit

Feedback three ways, in our run

Our run can measure this compression. Take harness B and write its feedback in three forms: a score, a short summary, and the full traces. Then count the tokens of each with the model's own tokenizer:

plain text
== 4. Feedback about harness B, three ways
a score:               10 tokens  -> B_labels: accuracy 0.433
a short summary:       32 tokens
the full traces:    11729 tokens  (60 prompts, outputs and verdicts)
traces / score = 1173x
what only the traces show: 34 wrong answers, and 13 of them are outputs that match no label
  output 'gastroesophageal reflux disease (GERD)' gold 'gastroesophageal reflux disease'
  output 'gastroesophageal reflux disease (GERD)' gold 'gastroesophageal reflux disease'
  output 'Urinary tract infection (UTI)'    gold 'urinary tract infection'
5 of the 13 rejected outputs name the right disease inside a longer answer

The traces are 1,173 times longer than the score, for only 60 short cases. They also hold the one fact that tells you what to fix. Five answers were right but had an abbreviation in brackets, and the strict reader threw them away. The score cannot say this. Our summary, "The model often gives answers that do not match a label", comes close, but it does not say which answers or in what form.

The fix that the traces suggest is exactly harness C. C went from 26 to 31 right answers, a gain of 5, the same 5 cases that the traces showed. This is the paper's argument on a small scale: the diagnosis was in the raw record, and compression would have removed it.

How much feedback the optimizer can read (tokens, log scale)10^010^110^210^310^410^510^610^7our run: a score10our run: a summary32our run: full traces11,729paper: prior optimizers, top of range30,000paper: one evaluation, up to10,000,000
Feedback sizes on a log scale. Our three forms of feedback for one harness, and the two sizes that §1 of the paper gives: at most about 30,000 tokens per step for earlier optimizers, and up to 10,000,000 tokens for one Meta-Harness evaluation.

The answer: a coding agent and a filesystem

grep, two ways

Our run wrote its own small filesystem: one folder per harness, with scores.json and traces.jsonl. A proposer would search it. Here we count the wrong answers in each folder in two ways. Method 1 is our own Python. Method 2 is the real grep program that a coding agent would run:

python
def grep_count(root, pattern):
    """Count matching lines in every traces.jsonl under root, like grep -rc."""
    counts = {}
    for dirpath, _, files in os.walk(root):
        for fn in files:
            if fn == 'traces.jsonl':
                p = os.path.join(dirpath, fn)
                counts[os.path.relpath(p, root)] = sum(pattern in line for line in open(p))
    return dict(sorted(counts.items()))


ours = grep_count(FS, PATTERN)
out = subprocess.run(['grep', '-rc', '--include=traces.jsonl', PATTERN, '.'], cwd=FS, capture_output=True, text=True).stdout
plain text
== 5. Search the trace files: our Python (Method 1) and grep (Method 2)
A_bare/traces.jsonl              Method 1  60   grep  60
B_labels/traces.jsonl            Method 1  34   grep  34
C_labels_lenient/traces.jsonl    Method 1  29   grep  29
D_labels_retrieval/traces.jsonl  Method 1  14   grep  14
Method 1 and grep agree: True

The pattern is "correct": false, so each count is the number of wrong answers of one harness. They match the accuracy table: 60 minus 26 right is 34 wrong for B. One command answers a question about every harness at once, and its output is a few lines, not 11,729 tokens. That is why an agent with grep can work with a history far larger than its context window.

The results, in one paragraph

The headline numbers, checked

The arithmetic behind the claims is short, so our script checks it:

plain text
== 6. The paper's headline numbers, checked
text classification: Meta-Harness 48.6 vs ACE 40.9 (points)        7.7
context: ACE 50.8K vs Meta-Harness 11.4K tokens (times fewer)      4.46
TerminalBench-2, Opus 4.6: 76.4 vs Terminus-KIRA 74.7 (points)     1.7
TerminalBench-2, Haiku 4.5: 37.6 vs Goose 35.5 (points)            2.1
the 6x source: same model, 12% vs 2% of 50 tasks (ratio)           6.0
12% and 2% of 50 tasks, as task counts                             (6.0, 1.0)

The accuracies and context sizes come from the paper's Table 2 and Table 7. "4× fewer" is 4.46 when you divide, so the paper rounds down. On TerminalBench-2, the gains are smaller in points: 1.7 on Opus 4.6 and 2.1 on Haiku 4.5. On a benchmark of 89 tasks, one task is worth about 1.1 points, so each of these gains is worth one or two tasks. Part 4 reads these tables in full.

What our small run cannot show

Our run shows that a harness can move the score of a frozen model a lot. It does not show Meta-Harness itself. Here, a person wrote the four harnesses, with a toy model and 60 cases. The paper's proposer is Anthropic's coding agent with Opus 4.6, and it ran about 60 harnesses per search on far larger tasks. We have no API key and pay for no API, so we cannot run that proposer.

Our task is also easy for retrieval: many Symptom2Disease cases use similar words for the same disease. The paper's model, GPT-OSS-120B, is much larger than ours. Our 76.7% is on 60 cases of one dataset, while the paper's 48.6% is the average over three datasets and is not comparable. Part 6 builds a small Meta-Harness loop that does propose harnesses, with a local model as the proposer. It also says what a small proposer can and cannot do.

Next, in Part 2: §2 and §3 of the paper, the method. Part 2 reads the objective as an equation, and the search loop of Figure 2 and Algorithm 1. It also shows how the paper keeps a search set apart from a test set.

Run it yourself

The script behind this part is code/papers/meta-harness/mh_part1.py. It loads Qwen2.5-1.5B-Instruct and Symptom2Disease, runs the four harnesses on 60 test cases, and writes one folder per harness to results/part1_fs/. It then checks the prompt and the accuracy two ways, counts the feedback tokens, and runs grep on the folders. Every number is saved in results/part1.json. On an Apple M-series laptop it takes a few minutes.

bash
pip install torch transformers datasets scikit-learn
python mh_part1.py   # downloads Qwen/Qwen2.5-1.5B-Instruct (about 3 GB) and gretelai/symptom_to_diagnosis
Terminal output of mh_part1.py: the accuracy table of the four harnesses, the prompt check, the accuracy check, the feedback sizes, the grep check and the paper's numbers
The full output of mh_part1.py, as it ran for this article.

The paper and the code

Meta-Harness on arXiv · project page · our scripts for this part