Why the code around a language model can matter as much as the model, and the paper's proposal: a coding agent that reads every earlier attempt and writes a better one. We read the title, Figure 1, the abstract and §1, and we run one small model with four different harnesses.
Meta-Harness: End-to-End Optimization of Model Harnesses. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn. COLM 2026, 2026. arXiv:2603.28052
A language model answers what it is shown. Someone writes the code that decides what it is shown, and that code can move the score as much as a better model can. This paper asks a coding agent to write that code, after it reads every earlier attempt in full.
Each piece below shows the exact lines of the paper, then explains them.
The outer loop of Meta-Harness. The proposer reads the filesystem, writes a harness, the harness is scored on search tasks, and everything goes into a new folder. Part 2 reads the paper's own picture of it, Figure 2.
We can see this effect ourselves with a small open model. The model is Qwen2.5-1.5B-Instruct, and its weights never change. The task is Symptom2Disease, one of the paper's three text-classification datasets: read a patient's description of symptoms and name one of 22 diseases. We score 60 fixed cases from its test split.
There is one function that calls the model. Every harness goes through it, so the only thing that can change is what the harness puts in and how it reads what comes out:
python
def call_model(messages, max_new_tokens=12): """The only place where the model runs. A harness decides what goes into `messages`.""" prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) ids = tok(prompt, return_tensors='pt')['input_ids'].to(DEV) with torch.no_grad(): out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) return tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True).strip(), ids.shape[1]
do_sample=False means greedy decoding: the model always takes its most likely next token, so a run gives the same output each time. The four harnesses are four small functions. Each returns the messages for the model and the function that reads the answer:
python
def parse_strict(output): """Accept the answer only if the whole output is exactly one label.""" return norm(output) if norm(output) in LABELS else Nonedef parse_lenient(output): """Accept the first label that occurs anywhere in the output (longest labels first).""" o = norm(output) for lab in sorted(LABELS, key=len, reverse=True): if lab in o: return lab return None
python
def h_bare(text): return [{'role': 'user', 'content': f'Patient: {text}\nWhat is the diagnosis?'}], parse_strictdef h_labels(text): return [{'role': 'system', 'content': LABEL_LIST + ' Answer with one diagnosis from the list only.'}, {'role': 'user', 'content': f'Patient: {text}\nDiagnosis:'}], parse_strictdef h_labels_lenient(text): msgs, _ = h_labels(text) return msgs, parse_lenientdef h_retrieval(text): shots = ''.join(f'Patient: {t}\nDiagnosis: {y}\n\n' for t, y in retrieve(text, K_SHOTS)) return [{'role': 'system', 'content': LABEL_LIST + ' Answer with one diagnosis from the list only.'}, {'role': 'user', 'content': f'{shots}Patient: {text}\nDiagnosis:'}], parse_lenient
Harness A asks the bare question. Harness B also lists the 22 diseases and asks for one of them. Harness C has the same prompt as B, but its reader accepts a disease name inside a longer answer. Harness D adds the 5 most similar solved cases from the 853 training cases, found by TF-IDF word overlap. This is retrieval, the "retrieve" job of the abstract. The results:
plain text
== 1. One frozen model, one task, four harnessesmodel: Qwen/Qwen2.5-1.5B-Instruct on mps, greedy decoding, weights never changetask: Symptom2Disease, 22 labels, memory 853 labeled examples, eval 60 test examplesharness accuracy right input tokensA_bare 0.000 0/60 72.4B_labels 0.433 26/60 145.4C_labels_lenient 0.517 31/60 145.4D_labels_retrieval 0.767 46/60 368.0time for 4 x 60 model calls: 350 sthe first outputs of harness A (12 new tokens at most): 'Based on your symptoms of skin irritations, rashes,' 'Based on your symptoms of frequent coughing, difficulty breathing,'best 0.767, worst 0.000, gap 76.7 points, ratio undefined (worst is 0)
Our run: one frozen model, four harnesses, 60 test cases. The accuracy goes from 0% to 76.7%, and the weights never changed.
Harness A scores zero, and the outputs show why. The model starts politely, "Based on your symptoms of...", and the 12-token limit cuts it off before it names a disease. The strict reader then finds no disease name. The model may know the answer; the harness never lets it say it. The ratio of best to worst is not even defined, because the worst is 0.
Each later harness fixes one thing. The label list (B) tells the model what form the answer must have: 26 right. The lenient reader (C) accepts "Urinary tract infection (UTI)" as "urinary tract infection": 31 right. Five retrieved cases (D) show the model how similar patients were labeled: 46 right. D uses about 2.5 times the input tokens of B, which is the price of the examples.
This is the 6× claim at toy scale, and it has the same lesson. Nothing about the model changed. Small code choices, such as an output limit and a reader, decided between 0% and 77%. Be careful with the size of these numbers: 60 cases is a small sample, and one case is 1.7 points.
A harness builds messages, but the model reads one long string. Our first method writes that string by hand in Qwen's chat format. The second method asks the tokenizer's own chat template, which is the library's way. If our understanding is right, the two strings are equal, character for character:
python
def chatml_by_hand(messages): """Qwen2.5's chat format: each turn is <|im_start|>role, newline, text, <|im_end|>, newline.""" s = '' if messages[0]['role'] != 'system': # Qwen adds its default system turn s += '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' for m in messages: s += f"<|im_start|>{m['role']}\n{m['content']}<|im_end|>\n" return s + '<|im_start|>assistant\n' # the model writes from hereours = chatml_by_hand(msgs)lib = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False)log(f'Method 1 length {len(ours)} chars, Method 2 length {len(lib)} chars, identical: {ours == lib}')
plain text
== 2. The prompt the model sees: built by hand (Method 1) and by the chat template (Method 2)Method 1 length 611 chars, Method 2 length 611 chars, identical: Trueharness A (no system turn): identical: Truethe start of the prompt of harness B, as the model reads it: | <|im_start|>system | Possible diagnoses: allergy, arthritis, bronchial asthma, cervical spondylosis, chicken pox, common cold, deng ... | <|im_start|>user
Both methods give the same 611 characters. One detail is a harness choice that nobody wrote: when harness A sends no system message, the template adds Qwen's default one. A harness can put text in front of the model that its author never typed. Only a full trace of the prompt shows it.
We also count the accuracy in two ways, by hand and with scikit-learn's accuracy_score. They agree for all four harnesses:
plain text
== 3. Accuracy counted by hand (Method 1) and by scikit-learn (Method 2)A_bare ours 0.0000 sklearn 0.0000 agree: TrueB_labels ours 0.4333 sklearn 0.4333 agree: TrueC_labels_lenient ours 0.5167 sklearn 0.5167 agree: TrueD_labels_retrieval ours 0.7667 sklearn 0.7667 agree: True
Our run can measure this compression. Take harness B and write its feedback in three forms: a score, a short summary, and the full traces. Then count the tokens of each with the model's own tokenizer:
plain text
== 4. Feedback about harness B, three waysa score: 10 tokens -> B_labels: accuracy 0.433a short summary: 32 tokensthe full traces: 11729 tokens (60 prompts, outputs and verdicts)traces / score = 1173xwhat only the traces show: 34 wrong answers, and 13 of them are outputs that match no label output 'gastroesophageal reflux disease (GERD)' gold 'gastroesophageal reflux disease' output 'gastroesophageal reflux disease (GERD)' gold 'gastroesophageal reflux disease' output 'Urinary tract infection (UTI)' gold 'urinary tract infection'5 of the 13 rejected outputs name the right disease inside a longer answer
The traces are 1,173 times longer than the score, for only 60 short cases. They also hold the one fact that tells you what to fix. Five answers were right but had an abbreviation in brackets, and the strict reader threw them away. The score cannot say this. Our summary, "The model often gives answers that do not match a label", comes close, but it does not say which answers or in what form.
The fix that the traces suggest is exactly harness C. C went from 26 to 31 right answers, a gain of 5, the same 5 cases that the traces showed. This is the paper's argument on a small scale: the diagnosis was in the raw record, and compression would have removed it.
Feedback sizes on a log scale. Our three forms of feedback for one harness, and the two sizes that §1 of the paper gives: at most about 30,000 tokens per step for earlier optimizers, and up to 10,000,000 tokens for one Meta-Harness evaluation.
Our run wrote its own small filesystem: one folder per harness, with scores.json and traces.jsonl. A proposer would search it. Here we count the wrong answers in each folder in two ways. Method 1 is our own Python. Method 2 is the real grep program that a coding agent would run:
python
def grep_count(root, pattern): """Count matching lines in every traces.jsonl under root, like grep -rc.""" counts = {} for dirpath, _, files in os.walk(root): for fn in files: if fn == 'traces.jsonl': p = os.path.join(dirpath, fn) counts[os.path.relpath(p, root)] = sum(pattern in line for line in open(p)) return dict(sorted(counts.items()))ours = grep_count(FS, PATTERN)out = subprocess.run(['grep', '-rc', '--include=traces.jsonl', PATTERN, '.'], cwd=FS, capture_output=True, text=True).stdout
The pattern is "correct": false, so each count is the number of wrong answers of one harness. They match the accuracy table: 60 minus 26 right is 34 wrong for B. One command answers a question about every harness at once, and its output is a few lines, not 11,729 tokens. That is why an agent with grep can work with a history far larger than its context window.
The arithmetic behind the claims is short, so our script checks it:
plain text
== 6. The paper's headline numbers, checkedtext classification: Meta-Harness 48.6 vs ACE 40.9 (points) 7.7context: ACE 50.8K vs Meta-Harness 11.4K tokens (times fewer) 4.46TerminalBench-2, Opus 4.6: 76.4 vs Terminus-KIRA 74.7 (points) 1.7TerminalBench-2, Haiku 4.5: 37.6 vs Goose 35.5 (points) 2.1the 6x source: same model, 12% vs 2% of 50 tasks (ratio) 6.012% and 2% of 50 tasks, as task counts (6.0, 1.0)
The accuracies and context sizes come from the paper's Table 2 and Table 7. "4× fewer" is 4.46 when you divide, so the paper rounds down. On TerminalBench-2, the gains are smaller in points: 1.7 on Opus 4.6 and 2.1 on Haiku 4.5. On a benchmark of 89 tasks, one task is worth about 1.1 points, so each of these gains is worth one or two tasks. Part 4 reads these tables in full.
Our run shows that a harness can move the score of a frozen model a lot. It does not show Meta-Harness itself. Here, a person wrote the four harnesses, with a toy model and 60 cases. The paper's proposer is Anthropic's coding agent with Opus 4.6, and it ran about 60 harnesses per search on far larger tasks. We have no API key and pay for no API, so we cannot run that proposer.
Our task is also easy for retrieval: many Symptom2Disease cases use similar words for the same disease. The paper's model, GPT-OSS-120B, is much larger than ours. Our 76.7% is on 60 cases of one dataset, while the paper's 48.6% is the average over three datasets and is not comparable. Part 6 builds a small Meta-Harness loop that does propose harnesses, with a local model as the proposer. It also says what a small proposer can and cannot do.
Next, in Part 2: §2 and §3 of the paper, the method. Part 2 reads the objective as an equation, and the search loop of Figure 2 and Algorithm 1. It also shows how the paper keeps a search set apart from a test set.
Run it yourself
The script behind this part is code/papers/meta-harness/mh_part1.py. It loads Qwen2.5-1.5B-Instruct and Symptom2Disease, runs the four harnesses on 60 test cases, and writes one folder per harness to results/part1_fs/. It then checks the prompt and the accuracy two ways, counts the feedback tokens, and runs grep on the folders. Every number is saved in results/part1.json. On an Apple M-series laptop it takes a few minutes.