Meta-Harness, explained · Part 5 of 6 · Covers §5, §4.3, footnote 1, Figure 7, Appendix A.2, Appendix D
Limits, and What Came After
Where Meta-Harness can fool itself, what a search costs, and why it rests on one strong proposer. Then five later papers that changed the loop, each read from its own PDF, and a small run on a laptop that shows how a search score can look better than it is.
Meta-Harness: End-to-End Optimization of Model Harnesses. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn. COLM 2026, 2026. arXiv:2603.28052
A search that picks the best of many tries always looks a little better than it is. This part asks how much that matters for Meta-Harness, what a run costs, and what the papers that came after it changed. Each piece below shows the exact lines of a paper, then explains them, and then checks the idea with code you can run.
Think of 40 students who all know exactly 40 percent of the material. Each one takes a 100-question test. Their scores will not all be 40: some get lucky questions. If you pick the top scorer and call them the best student, you are mostly picking luck. Statisticians call this the winner's curse.
Here is the same idea as code. Method 1 simulates it: 40 candidates, each truly 40 percent accurate, each scored on 100 random examples, and we keep the best score. We repeat this 2,000 times and average.
python
def best_of_n_sim(true_acc, n_cand, n_search, trials, seed=0): """Method 1: simulate. Every candidate has the same true accuracy; we keep the best search score.""" rng = random.Random(seed) best = [] for _ in range(trials): scores = [sum(rng.random() < true_acc for _ in range(n_search)) for _ in range(n_cand)] best.append(max(scores) / n_search) return 100 * sum(best) / trials
Method 2 gets the exact answer with the binomial distribution of the scipy library. One candidate's number of correct answers out of n follows a binomial distribution with cumulative probability F(k). All N candidates are at most k only if each one is, so the best score is at most k with probability F(k)N. The expected best score is the sum of the chances that it reaches each value:
E[max]=k=1∑nP(max≥k)=k=1∑n(1−F(k−1)N)
Read it from left to right. E[max] is the average best number of correct answers. P(max≥k) is the chance that the best candidate gets at least k right. F(k−1)N is the chance that all N candidates get k−1 or fewer right. A tiny check: with N=1 the formula gives the plain average np, because one candidate has no one to beat.
python
def best_of_n_exact(true_acc, n_cand, n_search): """Method 2: exact. P(max >= k) = 1 - F(k-1)^N, and E[max] = sum over k of P(max >= k).""" F = stats.binom(n_search, true_acc).cdf(np.arange(n_search)) # F(0) ... F(n-1) return 100 * np.sum(1 - F ** n_cand) / n_search
The two methods agree, and the answer is large:
plain text
40 candidates, all truly 40% accurate, search set of 100 examplesexpected best search score: simulation 50.70%, exact 50.68%, agree within 0.3: Truethe winner looks 10.7 points better than it isinflation of the best search score (points), exact: candidates n=50 n=88 n=100 n=250 1 0.0 0.0 0.0 0.0 4 7.1 5.4 5.1 3.2 10 10.7 8.1 7.6 4.8 40 15.1 11.4 10.7 6.7 109 17.8 13.4 12.6 7.9
The sizes in the table come from the paper. Text classification had 40 candidates and math retrieval 109. The search sets held 50 to 100, 88 and 250 examples. With 40 equal candidates and 100 examples, the winner looks 10.7 points better than it is. More candidates make the effect larger. A larger search set makes it smaller.
How many points the best search score is above the true 40 percent, when every candidate is equally good. Each line is one search-set size. The effect grows with the number of candidates and shrinks as the search set grows. Exact values from the binomial distribution.
This model is a worst case, because every candidate here is equally good, so all of the spread is luck. Real candidates differ. When one harness is truly better by 10 points, the search will mostly find it, and only a part of its lead is luck. The table tells you how large a lead you need before you trust it.
Let us make the same picture with a real model. Our task model is Qwen2.5-0.5B-Instruct, a small open model that runs on a laptop. The task is SST-2: decide whether a short movie review is positive or negative. We split 420 reviews into three disjoint sets: 60 for search, 60 held out, and 300 for the test.
A "harness" here is the smallest thing that deserves the name: a function that turns a review into a prompt. Two templates, each with no examples or with four examples drawn by one of five seeds, give 12 harnesses.
python
def make_harness(template, seed): """A 'harness' here is a function from an input sentence to a prompt: a template plus 0 or 4 examples.""" ex = [] if seed is None else shots(seed) head = ''.join(TEMPLATES[template].format(s=r['sentence'].strip()) + LABELS[r['label']] + '\n\n' for r in ex) return lambda s: head + TEMPLATES[template].format(s=s.strip())
To score a prompt, we ask the model which word is more likely next: " positive" or " negative". Method 1 reads that from one forward pass: the log-probability of each label's token at the last position. Method 2 asks the transformers library instead. When we give the model the label token as a training target, its built-in loss is minus the log-probability of that label.
python
@torch.no_grad()def scores_method1(prompts, bs=16): """Method 1: one forward pass; the log-probability of each label's token at the last position.""" res = [] for i in range(0, len(prompts), bs): enc = tok(prompts[i:i + bs], return_tensors='pt', padding=True).to(dev) logits = model(**enc).logits[:, -1, :] # (batch, vocab) logp = torch.log_softmax(logits.float(), dim=-1)[:, LABEL_IDS] # (batch, 2) res.append(logp.cpu()) return torch.cat(res)@torch.no_grad()def logprob_method2(prompt, label_word): """Method 2: the library's own loss. With labels on the label token only, loss = -log P(label | prompt).""" ids_p = tok(prompt, return_tensors='pt').input_ids ids_l = tok(label_word, return_tensors='pt').input_ids ids = torch.cat([ids_p, ids_l], dim=1).to(dev) labels = ids.clone() labels[:, :ids_p.shape[1]] = -100 # ignore the prompt positions return -model(input_ids=ids, labels=labels).loss.item()
The two scorers agree to about three parts in a hundred thousand. Then every harness is scored on all three sets:
plain text
label scoring check on 5 prompts: max difference ours vs library loss = 2.96e-05, agree: Trueharness search held-out testA-zero 83.3% 86.7% 87.3%A-4shot-s1 91.7% 95.0% 91.3%A-4shot-s2 90.0% 95.0% 92.0%A-4shot-s3 90.0% 85.0% 90.0%A-4shot-s4 88.3% 90.0% 89.7%A-4shot-s5 93.3% 93.3% 90.0%B-zero 66.7% 68.3% 72.7%B-4shot-s1 80.0% 86.7% 84.3%B-4shot-s2 80.0% 81.7% 81.7%B-4shot-s3 83.3% 85.0% 86.0%B-4shot-s4 83.3% 78.3% 79.3%B-4shot-s5 76.7% 73.3% 73.7%chosen by search score: A-4shot-s5: search 93.3%, test 90.0%, gap +3.3 pointsbest on test (unknown during search): A-4shot-s2 at 92.0%; mean test of all 12: 84.8%rank correlation of search and test over 12 harnesses: ours 0.912, scipy 0.912, agree: Truestandard error of one accuracy on 60 examples near 93%: 3.2 points
Our 12 prompt harnesses around Qwen2.5-0.5B-Instruct on SST-2. Left to right is the accuracy on 60 search reviews, bottom to top the accuracy on 300 test reviews. The dashed line is y = x. The harness with the best search score (93.3 percent) gets 90.0 percent on the test set; a different harness is best on the test set at 92.0 percent.
Both lessons of Figure 7 show up. The search worked: it chose a good harness, and the search order and the test order agree well. The rank correlation is 0.912. But the chosen harness lost 3.3 points between search and test, and it was not the best one on the test set. With only 60 search reviews, one review is 1.7 points, and the standard error of one score is about 3 points. The winner's lead over the next harness was 1.6 points, one review, well within that noise.
We wrote the rank correlation from scratch (spearman in the script: rank, then Pearson correlation) and checked it against scipy.stats.spearmanr. Both give 0.912.
What our run cannot show: our harnesses are 12 fixed prompts, not programs that a proposer wrote after reading traces. Our task model has 0.5 billion parameters, while the paper's classifier is GPT-OSS-120B. So our numbers show the selection effect, not the size of the paper's gains. Part 6 builds a full search loop and measures the same gap there.
TerminalBench-2: search and test on the same tasks#
A benchmark with 89 tasks moves in steps of 100 / 89 = 1.12 points. The standard error tells us how much a pass rate jumps between two runs of the same harness, when only luck changes.
For p=0.644 and n=89: 0.644×0.356=0.229, divided by 89 is 0.00258, and the square root is 0.0508, or 5.08 points. Method 1 is that formula; Method 2 asks scipy for the spread of the binomial distribution.
python
def pass_rate_se(p, n): """Method 1: standard error of a pass rate p measured on n independent tasks, in points.""" return 100 * math.sqrt(p * (1 - p) / n)p0 = 0.644 # the Terminus-KIRA baseline in the Appendix A.2 search logse1 = pass_rate_se(p0, N_TASKS)se2 = 100 * stats.binom(N_TASKS, p0).std() / N_TASKS # Method 2: the binomial distribution of scipy
plain text
one task = 100 / 89 = 1.12 pointsstandard error at 64.4% on 89 tasks: from scratch 5.08 points, scipy 5.08 points, same: Truerough 95% range for one run: 64.4 +- 9.9 points evo_marker_fix -5.6 points = -5.0 tasks if each task ran once evo_single_confirm -6.7 points = -6.0 tasks if each task ran once evo_strip_only -1.1 points = -1.0 tasks if each task ran onceruns per task (1 to 10) that fit 64.4, 58.9, 57.8 and 63.3 exactly: [9, 10]
Keep these numbers in mind for the search log in the next sections. A drop of 1.1 points is one task. If each task ran once, even the drops of 5.6 and 6.7 points are about one standard error. The last line is a small puzzle. With 1 to 8 runs per task, the four percentages of the log are never whole numbers of passes. So they are not plain pass counts over a fixed number of runs, and the paper does not say how they were averaged.
Appendix A follows one TerminalBench-2 search with Opus 4.6 for 10 iterations. Part 4 summarises it; here we read the log with the noise numbers above in mind.
The rule in symbols, with Δin and Δho the change in the number of passed tasks on each split:
accept⟺Δin≥0,Δho≥0,max(Δin,Δho)>0
Read it as three tests. Neither split may get worse. At least one must get better. A candidate with (+3,0) passes. A candidate with (+5,−1) fails, even though it gains 4 tasks in total. A candidate with (0,0) fails, because it changes nothing.
Method 1 is the rule for one candidate. Method 2 applies it to whole arrays with numpy. We check both on every pair of changes from -2 to +2, then use the rule on our 12 SST-2 harnesses. The current harness is A-zero; the search set plays held-in and our 60 held-out reviews play held-out.
python
def accept(d_in, d_ho): """Method 1: the rule of Self-Harness, Section 3.4, for one candidate (pass-count changes).""" return d_in >= 0 and d_ho >= 0 and max(d_in, d_ho) > 0def accept_np(d_in, d_ho): """Method 2: the same rule on whole arrays with numpy.""" d_in, d_ho = np.asarray(d_in), np.asarray(d_ho) return (d_in >= 0) & (d_ho >= 0) & (np.maximum(d_in, d_ho) > 0)
The gate does its job on the clear cases. It rejects A-4shot-s3, which gained 4 search reviews but lost 1 held-out review.
Self-Harness merges all accepted edits into one harness. Prompts with different examples cannot be merged, so we keep the accepted one with the largest total gain. Two candidates tie at +10 reviews; we break the tie by the larger held-out gain, which picks A-4shot-s1. It scores 91.3 percent on the test set, up from 87.3 for the current harness. Here the stricter rule chose a better test harness than the best search score did (90.0 percent). With 12 candidates and 60 reviews per split, this is one small example, not a general result.
The number of pairs among m outputs is (2m)=m(m−1)/2: each of the m outputs meets the m−1 others, and each pair is counted twice. Method 1 counts the pairs one by one; Method 2 uses math.comb.
python
def pairs_by_hand(m): """Method 1: count the pairwise comparisons in a population of m outputs, one by one.""" c = 0 for i in range(m): for j in range(i + 1, m): c += 1 return c
plain text
population 2: 1 pairs by hand, 1 with math.comb, trajectory-local: 1 pair population 10: 45 pairs by hand, 45 with math.comb, trajectory-local: 1 pair population 40: 780 pairs by hand, 780 with math.comb, trajectory-local: 1 pair population 109: 5886 pairs by hand, 5886 with math.comb, trajectory-local: 1 pair
With the 109 candidates of the math search, comparing every pair would take 5,886 judgments. RHI takes one per step.
Rethinking the Evaluation of Harness Evolution (July 2026)#
Here is the whole story of the paper in six steps, with the page that covers each.
Background: a harness is the code around a model call: memory, retrieval, prompt assembly and tool loops. Earlier optimizers improved prompts or single functions from scores or short feedback.
Part 1: the same model can score very differently with different harnesses. Meta-Harness lets a coding agent search over harness code, reading full history from a filesystem.
Part 2: the objective is the expected reward of a harness. The loop of Algorithm 1 proposes, validates, evaluates and logs every candidate. The proposer reads the logs with ordinary tools such as grep and cat.
Part 3: on text classification the best discovered harness reaches 48.6 percent with far less context than ACE. Raw traces matter: the scores-only and summary versions of the loop do much worse.
Part 4: a discovered retrieval router helps five models on IMO-level maths. On TerminalBench-2 a small, additive environment snapshot gives 76.4 percent with Opus 4.6 and 37.6 percent with Haiku 4.5.
Part 5 (this page): the TerminalBench-2 result uses the tasks it searched on, and every gain needs a check against luck. Later work moved the loop to the task model, cheaper steps, stricter gates and the weights.
Part 6: the whole loop in Python on a laptop, with a test set the proposer never sees.