Meta-Harness, explained · Part 5 of 6 · Covers §5, §4.3, footnote 1, Figure 7, Appendix A.2, Appendix D

Limits, and What Came After

Where Meta-Harness can fool itself, what a search costs, and why it rests on one strong proposer. Then five later papers that changed the loop, each read from its own PDF, and a small run on a laptop that shows how a search score can look better than it is.

Meta-Harness: End-to-End Optimization of Model Harnesses. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn. COLM 2026, 2026. arXiv:2603.28052

A search that picks the best of many tries always looks a little better than it is. This part asks how much that matters for Meta-Harness, what a run costs, and what the papers that came after it changed. Each piece below shows the exact lines of a paper, then explains them, and then checks the idea with code you can run.

What the authors claim in the Discussion

Overfitting in code space

How much luck can a search buy? (code)

Think of 40 students who all know exactly 40 percent of the material. Each one takes a 100-question test. Their scores will not all be 40: some get lucky questions. If you pick the top scorer and call them the best student, you are mostly picking luck. Statisticians call this the winner's curse.

Here is the same idea as code. Method 1 simulates it: 40 candidates, each truly 40 percent accurate, each scored on 100 random examples, and we keep the best score. We repeat this 2,000 times and average.

python
def best_of_n_sim(true_acc, n_cand, n_search, trials, seed=0):
    """Method 1: simulate. Every candidate has the same true accuracy; we keep the best search score."""
    rng = random.Random(seed)
    best = []
    for _ in range(trials):
        scores = [sum(rng.random() < true_acc for _ in range(n_search)) for _ in range(n_cand)]
        best.append(max(scores) / n_search)
    return 100 * sum(best) / trials

Method 2 gets the exact answer with the binomial distribution of the scipy library. One candidate's number of correct answers out of nn follows a binomial distribution with cumulative probability F(k)F(k). All NN candidates are at most kk only if each one is, so the best score is at most kk with probability F(k)NF(k)^N. The expected best score is the sum of the chances that it reaches each value:

E[max⁡]=∑k=1nP(max⁡≥k)=∑k=1n(1−F(k−1)N)\mathbb{E}[\max] = \sum_{k=1}^{n} P(\max \ge k) = \sum_{k=1}^{n} \left(1 - F(k-1)^N\right)

Read it from left to right. E[max⁡]\mathbb{E}[\max] is the average best number of correct answers. P(max⁡≥k)P(\max \ge k) is the chance that the best candidate gets at least kk right. F(k−1)NF(k-1)^N is the chance that all NN candidates get k−1k-1 or fewer right. A tiny check: with N=1N = 1 the formula gives the plain average npn p, because one candidate has no one to beat.

python
def best_of_n_exact(true_acc, n_cand, n_search):
    """Method 2: exact. P(max >= k) = 1 - F(k-1)^N, and E[max] = sum over k of P(max >= k)."""
    F = stats.binom(n_search, true_acc).cdf(np.arange(n_search))      # F(0) ... F(n-1)
    return 100 * np.sum(1 - F ** n_cand) / n_search

The two methods agree, and the answer is large:

plain text
40 candidates, all truly 40% accurate, search set of 100 examples
expected best search score: simulation 50.70%, exact 50.68%, agree within 0.3: True
the winner looks 10.7 points better than it is
inflation of the best search score (points), exact:
  candidates  n=50     n=88     n=100    n=250    
           1  0.0      0.0      0.0      0.0      
           4  7.1      5.4      5.1      3.2      
          10  10.7     8.1      7.6      4.8      
          40  15.1     11.4     10.7     6.7      
         109  17.8     13.4     12.6     7.9      

The sizes in the table come from the paper. Text classification had 40 candidates and math retrieval 109. The search sets held 50 to 100, 88 and 250 examples. With 40 equal candidates and 100 examples, the winner looks 10.7 points better than it is. More candidates make the effect larger. A larger search set makes it smaller.

05101520141040109search set 50, 1: 0.0search set 50, 4: 7.1search set 50, 10: 10.7search set 50, 40: 15.1search set 50, 109: 17.8search set 88, 1: 0.0search set 88, 4: 5.4search set 88, 10: 8.1search set 88, 40: 11.4search set 88, 109: 13.4search set 100, 1: 0.0search set 100, 4: 5.1search set 100, 10: 7.6search set 100, 40: 10.7search set 100, 109: 12.6search set 250, 1: 0.0search set 250, 4: 3.2search set 250, 10: 4.8search set 250, 40: 6.7search set 250, 109: 7.9search set 50: 17.8search set 88: 13.4search set 100: 12.6search set 250: 7.9number of candidates (log scale)points above the true 40%
How many points the best search score is above the true 40 percent, when every candidate is equally good. Each line is one search-set size. The effect grows with the number of candidates and shrinks as the search set grows. Exact values from the binomial distribution.

This model is a worst case, because every candidate here is equally good, so all of the spread is luck. Real candidates differ. When one harness is truly better by 10 points, the search will mostly find it, and only a part of its lead is luck. The table tells you how large a lead you need before you trust it.

Our own small Figure 7 (code)

Let us make the same picture with a real model. Our task model is Qwen2.5-0.5B-Instruct, a small open model that runs on a laptop. The task is SST-2: decide whether a short movie review is positive or negative. We split 420 reviews into three disjoint sets: 60 for search, 60 held out, and 300 for the test.

A "harness" here is the smallest thing that deserves the name: a function that turns a review into a prompt. Two templates, each with no examples or with four examples drawn by one of five seeds, give 12 harnesses.

python
def make_harness(template, seed):
    """A 'harness' here is a function from an input sentence to a prompt: a template plus 0 or 4 examples."""
    ex = [] if seed is None else shots(seed)
    head = ''.join(TEMPLATES[template].format(s=r['sentence'].strip()) + LABELS[r['label']] + '\n\n' for r in ex)
    return lambda s: head + TEMPLATES[template].format(s=s.strip())

To score a prompt, we ask the model which word is more likely next: " positive" or " negative". Method 1 reads that from one forward pass: the log-probability of each label's token at the last position. Method 2 asks the transformers library instead. When we give the model the label token as a training target, its built-in loss is minus the log-probability of that label.

python
@torch.no_grad()
def scores_method1(prompts, bs=16):
    """Method 1: one forward pass; the log-probability of each label's token at the last position."""
    res = []
    for i in range(0, len(prompts), bs):
        enc = tok(prompts[i:i + bs], return_tensors='pt', padding=True).to(dev)
        logits = model(**enc).logits[:, -1, :]                         # (batch, vocab)
        logp = torch.log_softmax(logits.float(), dim=-1)[:, LABEL_IDS]  # (batch, 2)
        res.append(logp.cpu())
    return torch.cat(res)


@torch.no_grad()
def logprob_method2(prompt, label_word):
    """Method 2: the library's own loss. With labels on the label token only, loss = -log P(label | prompt)."""
    ids_p = tok(prompt, return_tensors='pt').input_ids
    ids_l = tok(label_word, return_tensors='pt').input_ids
    ids = torch.cat([ids_p, ids_l], dim=1).to(dev)
    labels = ids.clone()
    labels[:, :ids_p.shape[1]] = -100                                  # ignore the prompt positions
    return -model(input_ids=ids, labels=labels).loss.item()

The two scorers agree to about three parts in a hundred thousand. Then every harness is scored on all three sets:

plain text
label scoring check on 5 prompts: max difference ours vs library loss = 2.96e-05, agree: True
harness         search  held-out   test
A-zero           83.3%     86.7%  87.3%
A-4shot-s1       91.7%     95.0%  91.3%
A-4shot-s2       90.0%     95.0%  92.0%
A-4shot-s3       90.0%     85.0%  90.0%
A-4shot-s4       88.3%     90.0%  89.7%
A-4shot-s5       93.3%     93.3%  90.0%
B-zero           66.7%     68.3%  72.7%
B-4shot-s1       80.0%     86.7%  84.3%
B-4shot-s2       80.0%     81.7%  81.7%
B-4shot-s3       83.3%     85.0%  86.0%
B-4shot-s4       83.3%     78.3%  79.3%
B-4shot-s5       76.7%     73.3%  73.7%
chosen by search score: A-4shot-s5: search 93.3%, test 90.0%, gap +3.3 points
best on test (unknown during search): A-4shot-s2 at 92.0%; mean test of all 12: 84.8%
rank correlation of search and test over 12 harnesses: ours 0.912, scipy 0.912, agree: True
standard error of one accuracy on 60 examples near 93%: 3.2 points
60%60%70%70%80%80%90%90%100%100%y = xA-zero: search 83.3%, test 87.3%A-4shot-s1: search 91.7%, test 91.3%A-4shot-s2: search 90.0%, test 92.0%A-4shot-s3: search 90.0%, test 90.0%A-4shot-s4: search 88.3%, test 89.7%A-4shot-s5: search 93.3%, test 90.0%B-zero: search 66.7%, test 72.7%B-4shot-s1: search 80.0%, test 84.3%B-4shot-s2: search 80.0%, test 81.7%B-4shot-s3: search 83.3%, test 86.0%B-4shot-s4: search 83.3%, test 79.3%B-4shot-s5: search 76.7%, test 73.7%chosen by search score: A-4shot-s5best on test: A-4shot-s2template A (Review: ...)template B (Text: ...)Each dot is one promptharness aroundQwen2.5-0.5B-Instruct.Dots above the line didbetter on the test setthan on the search set.search-set accuracy (60 examples)test accuracy (300 examples)
Our 12 prompt harnesses around Qwen2.5-0.5B-Instruct on SST-2. Left to right is the accuracy on 60 search reviews, bottom to top the accuracy on 300 test reviews. The dashed line is y = x. The harness with the best search score (93.3 percent) gets 90.0 percent on the test set; a different harness is best on the test set at 92.0 percent.

Both lessons of Figure 7 show up. The search worked: it chose a good harness, and the search order and the test order agree well. The rank correlation is 0.912. But the chosen harness lost 3.3 points between search and test, and it was not the best one on the test set. With only 60 search reviews, one review is 1.7 points, and the standard error of one score is about 3 points. The winner's lead over the next harness was 1.6 points, one review, well within that noise.

We wrote the rank correlation from scratch (spearman in the script: rank, then Pearson correlation) and checked it against scipy.stats.spearmanr. Both give 0.912.

What our run cannot show: our harnesses are 12 fixed prompts, not programs that a proposer wrote after reading traces. Our task model has 0.5 billion parameters, while the paper's classifier is GPT-OSS-120B. So our numbers show the selection effect, not the size of the paper's gains. Part 6 builds a full search loop and measures the same gap there.

TerminalBench-2: search and test on the same tasks

One task is 1.12 points (code)

A benchmark with 89 tasks moves in steps of 100 / 89 = 1.12 points. The standard error tells us how much a pass rate jumps between two runs of the same harness, when only luck changes.

For p=0.644p = 0.644 and n=89n = 89: 0.644×0.356=0.2290.644 \times 0.356 = 0.229, divided by 89 is 0.00258, and the square root is 0.0508, or 5.08 points. Method 1 is that formula; Method 2 asks scipy for the spread of the binomial distribution.

python
def pass_rate_se(p, n):
    """Method 1: standard error of a pass rate p measured on n independent tasks, in points."""
    return 100 * math.sqrt(p * (1 - p) / n)


p0 = 0.644                                         # the Terminus-KIRA baseline in the Appendix A.2 search log
se1 = pass_rate_se(p0, N_TASKS)
se2 = 100 * stats.binom(N_TASKS, p0).std() / N_TASKS   # Method 2: the binomial distribution of scipy
plain text
one task = 100 / 89 = 1.12 points
standard error at 64.4% on 89 tasks: from scratch 5.08 points, scipy 5.08 points, same: True
rough 95% range for one run: 64.4 +- 9.9 points
  evo_marker_fix       -5.6 points = -5.0 tasks if each task ran once
  evo_single_confirm   -6.7 points = -6.0 tasks if each task ran once
  evo_strip_only       -1.1 points = -1.0 tasks if each task ran once
runs per task (1 to 10) that fit 64.4, 58.9, 57.8 and 63.3 exactly: [9, 10]

Keep these numbers in mind for the search log in the next sections. A drop of 1.1 points is one task. If each task ran once, even the drops of 5.6 and 6.7 points are about one standard error. The last line is a small puzzle. With 1 to 8 runs per task, the four percentages of the log are never whole numbers of passes. So they are not plain pass counts over a fixed number of runs, and the paper does not say how they were averaged.

What a search costs

One strong proposer

Inside a search run {Appendix A.2}

Appendix A follows one TerminalBench-2 search with Opus 4.6 for 10 iterations. Part 4 summarises it; here we read the log with the noise numbers above in mind.

Co-evolving the harness and the weights

What came after

Within six months, several groups published follow-up work. Each one fixes a different limit from this page. We read each from its own PDF.

Paper (arXiv)Who proposes the new harnessWhat it changes
Agentic Harness Engineering, 2604.25850agents on the same base model as the task agentstructured evidence, and a prediction for every edit
Self-Harness, 2606.09498the task model itselfa held-out regression gate before any edit is kept
Recursive Harness Self-Improvement, 2607.15524an optimizer model, from pairwise judgmentsone comparison per step instead of a population
Rethinking the Evaluation of Harness Evolution, 2607.12227(an evaluation study)matched budgets and a held-out task split
Harness-Zero, 2609.24974an evolved harness, then fine-tuningmoves the harness gains into the model weights

Agentic Harness Engineering (April 2026)

Self-Harness (June 2026)

The rule in symbols, with Δin\Delta_{\text{in}} and Δho\Delta_{\text{ho}} the change in the number of passed tasks on each split:

accept  ⟺  Δin≥0,Δho≥0,max⁡(Δin,Δho)>0\text{accept} \iff \Delta_{\text{in}} \ge 0, \quad \Delta_{\text{ho}} \ge 0, \quad \max(\Delta_{\text{in}}, \Delta_{\text{ho}}) > 0

Read it as three tests. Neither split may get worse. At least one must get better. A candidate with (+3,0)(+3, 0) passes. A candidate with (+5,−1)(+5, -1) fails, even though it gains 4 tasks in total. A candidate with (0,0)(0, 0) fails, because it changes nothing.

The acceptance rule in code

Method 1 is the rule for one candidate. Method 2 applies it to whole arrays with numpy. We check both on every pair of changes from -2 to +2, then use the rule on our 12 SST-2 harnesses. The current harness is A-zero; the search set plays held-in and our 60 held-out reviews play held-out.

python
def accept(d_in, d_ho):
    """Method 1: the rule of Self-Harness, Section 3.4, for one candidate (pass-count changes)."""
    return d_in >= 0 and d_ho >= 0 and max(d_in, d_ho) > 0


def accept_np(d_in, d_ho):
    """Method 2: the same rule on whole arrays with numpy."""
    d_in, d_ho = np.asarray(d_in), np.asarray(d_ho)
    return (d_in >= 0) & (d_ho >= 0) & (np.maximum(d_in, d_ho) > 0)
plain text
rule checked on all 25 pairs of changes from -2 to +2: both methods agree: True
examples: (+3, 0) -> True, (+5, -1) -> False, (0, 0) -> False
current harness: A-zero (search 50/60, held-out 52/60)
  A-4shot-s1     d_in  +5  d_ho  +5  -> accept
  A-4shot-s2     d_in  +4  d_ho  +5  -> accept
  A-4shot-s3     d_in  +4  d_ho  -1  -> reject
  A-4shot-s4     d_in  +3  d_ho  +2  -> accept
  A-4shot-s5     d_in  +6  d_ho  +4  -> accept
  B-zero         d_in -10  d_ho -11  -> reject
  B-4shot-s1     d_in  -2  d_ho  +0  -> reject
  B-4shot-s2     d_in  -2  d_ho  -3  -> reject
  B-4shot-s3     d_in  +0  d_ho  -1  -> reject
  B-4shot-s4     d_in  +0  d_ho  -5  -> reject
  B-4shot-s5     d_in  -4  d_ho  -8  -> reject
accepted 4 of 11
largest accepted gain: A-4shot-s1: test 91.3% (current harness 87.3%)

The gate does its job on the clear cases. It rejects A-4shot-s3, which gained 4 search reviews but lost 1 held-out review.

Self-Harness merges all accepted edits into one harness. Prompts with different examples cannot be merged, so we keep the accepted one with the largest total gain. Two candidates tie at +10 reviews; we break the tie by the larger held-out gain, which picks A-4shot-s1. It scores 91.3 percent on the test set, up from 87.3 for the current harness. Here the stricter rule chose a better test harness than the best search score did (90.0 percent). With 12 candidates and 60 reviews per split, this is one small example, not a general result.

Recursive Harness Self-Improvement (July 2026)

The number of pairs among mm outputs is (m2)=m(m−1)/2\binom{m}{2} = m(m-1)/2: each of the mm outputs meets the m−1m-1 others, and each pair is counted twice. Method 1 counts the pairs one by one; Method 2 uses math.comb.

python
def pairs_by_hand(m):
    """Method 1: count the pairwise comparisons in a population of m outputs, one by one."""
    c = 0
    for i in range(m):
        for j in range(i + 1, m):
            c += 1
    return c
plain text
  population   2:     1 pairs by hand,     1 with math.comb, trajectory-local: 1 pair
  population  10:    45 pairs by hand,    45 with math.comb, trajectory-local: 1 pair
  population  40:   780 pairs by hand,   780 with math.comb, trajectory-local: 1 pair
  population 109:  5886 pairs by hand,  5886 with math.comb, trajectory-local: 1 pair

With the 109 candidates of the math search, comparing every pair would take 5,886 judgments. RHI takes one per step.

Rethinking the Evaluation of Harness Evolution (July 2026)

Harness-Zero (September 2026)

The series in one page

Here is the whole story of the paper in six steps, with the page that covers each.

  • Background: a harness is the code around a model call: memory, retrieval, prompt assembly and tool loops. Earlier optimizers improved prompts or single functions from scores or short feedback.
  • Part 1: the same model can score very differently with different harnesses. Meta-Harness lets a coding agent search over harness code, reading full history from a filesystem.
  • Part 2: the objective is the expected reward of a harness. The loop of Algorithm 1 proposes, validates, evaluates and logs every candidate. The proposer reads the logs with ordinary tools such as grep and cat.
  • Part 3: on text classification the best discovered harness reaches 48.6 percent with far less context than ACE. Raw traces matter: the scores-only and summary versions of the loop do much worse.
  • Part 4: a discovered retrieval router helps five models on IMO-level maths. On TerminalBench-2 a small, additive environment snapshot gives 76.4 percent with Opus 4.6 and 37.6 percent with Haiku 4.5.
  • Part 5 (this page): the TerminalBench-2 result uses the tasks it searched on, and every gain needs a check against luck. Later work moved the loop to the task model, cheaper steps, stricter gates and the weights.
  • Part 6: the whole loop in Python on a laptop, with a test set the proposer never sees.

The paper and the code

Paper (arXiv 2603.28052) · Project page · Our scripts