Meta-Harness, explained · Part 2 of 6 · Covers §2, Table 1, §3, Figure 2, Algorithm 1, Appendix A.1, Table 8, Appendix D, Appendix E

The Method: A Loop That Reads Every Earlier Attempt

How Meta-Harness searches for better harness code: the goal written as one formula, the loop of Figure 2 and Algorithm 1, and the folder of past attempts that the proposer reads. We run the whole loop in Python with a small local model and a scripted proposer.

Meta-Harness: End-to-End Optimization of Model Harnesses. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn. COLM 2026, 2026. arXiv:2603.28052

Meta-Harness is a loop. An agent reads a folder full of earlier attempts, writes a new harness, and the loop scores it and adds it to the folder. Each piece below shows the exact lines of the paper, then explains them, and then runs them as code on a laptop.

Where Meta-Harness sits

Table 1: how much each optimizer sees {Table 1}

Context the proposer can get from one evaluation (Table 1, log scale)1,00010,000100,0001,000,00010,000,000OPROOPRO: 0.002 MTok = 2,000 tokens2,000GEPAGEPA: 0.008 MTok = 8,000 tokens8,000Feedback DescentFeedback Descent: 0.012 MTok = 12,000 tokens12,000TextGradTextGrad: 0.015 MTok = 15,000 tokens15,000AlphaEvolveAlphaEvolve: 0.022 MTok = 22,000 tokens22,000TTT-DiscoverTTT-Discover: 0.026 MTok = 26,000 tokens26,000Meta-HarnessMeta-Harness: 10.0 MTok = 10,000,000 tokens10,000,000tokens per evaluation (each grid line is 10 times the one before)
Table 1 drawn on a log scale, where each grid line is ten times the one before. The six earlier methods sit between 2,000 and 26,000 tokens. Meta-Harness sits at 10,000,000.

A few lines of Python turn the table into ratios:

python
TABLE1 = {'OPRO': 0.002, 'TextGrad': 0.015, 'AlphaEvolve': 0.022, 'GEPA': 0.008,
          'Feedback Descent': 0.012, 'TTT-Discover': 0.026, 'Meta-Harness': 10.0}
largest_prior = max(v for k, v in TABLE1.items() if k != 'Meta-Harness')
plain text
== 1. Table 1: context per evaluation (MTok/iter, the paper's estimates)
  OPRO                0.002 MTok =        2,000 tokens   x     0.1 of the largest prior
  TextGrad            0.015 MTok =       15,000 tokens   x     0.6 of the largest prior
  AlphaEvolve         0.022 MTok =       22,000 tokens   x     0.8 of the largest prior
  GEPA                0.008 MTok =        8,000 tokens   x     0.3 of the largest prior
  Feedback Descent    0.012 MTok =       12,000 tokens   x     0.5 of the largest prior
  TTT-Discover        0.026 MTok =       26,000 tokens   x     1.0 of the largest prior
  Meta-Harness       10.000 MTok =   10,000,000 tokens   x   384.6 of the largest prior
  Meta-Harness / largest prior (TTT-Discover): 385x
  log10 of that ratio: 2.59 orders of magnitude

The largest earlier budget is TTT-Discover's 26,000 tokens. Meta-Harness has about 385 times that. On a log scale, 385 is 2.59 steps of ten. The paper rounds this to "three orders of magnitude" in §1.

A harness for optimizing harnesses

The objective

The equation, symbol by symbol

H∗=arg⁡max⁡H  Ex∼X,  τ∼pM(H,x)  r(τ,x)H^{*} = \arg\max_{H} \; \mathbb{E}_{x \sim \mathcal{X},\; \tau \sim p_M(H, x)} \; r(\tau, x)

Read it from the inside out.

  • r(τ,x)r(\tau, x) is the reward: one number that says how well the run τ\tau did on the task xx. For a classification task it is 1 for the right label and 0 for a wrong one.
  • τ\tau is the rollout trajectory: everything that happened in one run. The harness builds a prompt, the model answers, the harness updates its state, and so on.
  • τ∼pM(H,x)\tau \sim p_M(H, x) says that the run is random. The same harness HH and task xx can give different runs, because the model MM can sample different words. pMp_M is the chance of each run.
  • x∼Xx \sim \mathcal{X} says that the task is drawn from a distribution X\mathcal{X}: the kind of tasks we care about, such as "movie reviews to label".
  • E\mathbb{E} is the expected value: the average reward over both kinds of chance, the task and the run.
  • arg⁡max⁡H\arg\max_H means "the HH that makes the average largest". H∗H^{*} is that best harness. Note what is missing: MM does not appear under the arg⁡max⁡\arg\max. The model is fixed; only HH may change.

A worked example. Take two tasks, x1x_1 and x2x_2, each with a chance of 0.5. Take two harnesses. With harness HAH_A the model gets x1x_1 right 90% of the time and x2x_2 right 50% of the time. With HBH_B the numbers are 70% and 80%. The reward is 1 for right and 0 for wrong.

E[r∣HA]=0.5×0.9+0.5×0.5=0.70,E[r∣HB]=0.5×0.7+0.5×0.8=0.75\mathbb{E}[r \mid H_A] = 0.5 \times 0.9 + 0.5 \times 0.5 = 0.70, \qquad \mathbb{E}[r \mid H_B] = 0.5 \times 0.7 + 0.5 \times 0.8 = 0.75

So H∗=HBH^{*} = H_B. Note that HAH_A is the better harness on x1x_1. It still loses, because the objective averages over all tasks. A harness that shines on one example and fails on others does not win.

Method 1 and Method 2: the exact sum and the sampled average

Method 1 follows the definition: a sum over tasks and runs, each weighted by its chance. Method 2 does what an evaluation really does: draw tasks, run them, and average the rewards. The second way uses NumPy, the standard Python library for arrays of numbers.

python
P_TASK = {'x1': 0.5, 'x2': 0.5}
P_RIGHT = {'H_A': {'x1': 0.9, 'x2': 0.5}, 'H_B': {'x1': 0.7, 'x2': 0.8}}


def expected_reward_exact(h):
    """Method 1: the definition. Sum over tasks of p(x) times the sum over rollouts of p(tau) r(tau, x)."""
    total = 0.0
    for x, px in P_TASK.items():
        p = P_RIGHT[h][x]
        total += px * (p * 1.0 + (1 - p) * 0.0)        # two rollouts: right (reward 1) or wrong (reward 0)
    return total


def expected_reward_sampled(h, n=100_000, seed=0):
    """Method 2: Monte Carlo. Draw a task, draw a rollout, average the rewards (what evaluation really does)."""
    rng = np.random.default_rng(seed)
    xs = rng.choice(list(P_TASK), size=n, p=list(P_TASK.values()))           # x ~ X
    p = np.array([P_RIGHT[h][x] for x in xs])
    r = (rng.random(n) < p).astype(float)                                      # tau ~ p_M(H, x), r(tau, x) in {0, 1}
    return r.mean(), r.std() / np.sqrt(n)
plain text
== 2. The objective: E over tasks x and rollouts tau of r(tau, x)
  H_A: exact 0.7000   Monte Carlo 0.7011 (+/- 0.0014)   difference 0.0011
  H_B: exact 0.7500   Monte Carlo 0.7513 (+/- 0.0014)   difference 0.0013
  argmax over the two harnesses: H_B

The two methods agree to about 0.001. The sampled average is never exact: the "+/-" is its standard error, the typical size of its mistake. With 100,000 draws it is 0.0014. A real evaluation has far fewer draws, and the next run shows what that means.

The objective with a real model

Now the same objective with a real language model. The task model MM is Qwen2.5-0.5B-Instruct, a small open model, run locally. The task is SST-2, a public set of movie-review sentences labelled positive or negative. We estimate the expected reward of two harnesses on 24 reviews (the search set built further below). Each harness runs three times with random sampling (temperature 1.0), and once in greedy mode, which always picks the most likely word.

plain text
== 2b. E[r] with Qwen2.5-0.5B-Instruct on the 24 search reviews
  zero_shot   sampled (T=1.0) runs [0.708, 0.458, 0.583] -> mean 0.583;  greedy 0.500
  few_shot_4  sampled (T=1.0) runs [0.125, 0.208, 0.208] -> mean 0.181;  greedy 0.167
  argmax of the sampled estimate: zero_shot

Look at the three sampled runs of zero-shot: 0.708, 0.458 and 0.583. Same harness, same 24 reviews, same model, and the score moves by 0.25. That is τ∼pM(H,x)\tau \sim p_M(H, x) at work. One run is a noisy guess of the expected reward; the mean of three runs, 0.583, is a better one.

On this search set, the zero-shot harness wins clearly over the few-shot harness (0.583 against 0.181). That looks odd, since examples usually help. The traces explain it, and we read them further below. Note also that the greedy score of zero-shot is exactly 0.500. That is by design: we built the search set from 12 reviews that greedy zero-shot got wrong and 12 that it got right.

When there is more than one goal

Here is Pareto dominance in code, run on the numbers of the paper's Table 2 (accuracy in %, extra context in thousands of tokens; Part 3 explains that table). Method 1 checks every pair. Method 2 is the classic faster way: sort by context, then walk along and keep each point that beats the best accuracy so far.

python
def dominates(a, b):
    """a = (accuracy, context). a dominates b if it is no worse on both and strictly better on one."""
    return a[0] >= b[0] and a[1] <= b[1] and (a[0] > b[0] or a[1] < b[1])


def pareto_brute(points):
    """Method 1: keep every point that no other point dominates (compare all pairs)."""
    return sorted(k for k, p in points.items() if not any(dominates(q, p) for j, q in points.items() if j != k))


def pareto_sweep(points):
    """Method 2: sort by context (then by accuracy, high first); keep a point if it beats the best accuracy so far."""
    out, best = [], -float('inf')
    for k, (acc, ctx) in sorted(points.items(), key=lambda kv: (kv[1][1], -kv[1][0])):
        if acc > best:
            out.append(k)
            best = acc
    return sorted(out)
plain text
== 3. Pareto dominance: higher accuracy and lower context are both better
  brute force: ['Few-Shot (32)', 'Few-Shot (8)', 'Meta-Harness', 'Zero-Shot']
  sweep      : ['Few-Shot (32)', 'Few-Shot (8)', 'Meta-Harness', 'Zero-Shot']
  same frontier: True
  Few-Shot (all) is dominated by ['Meta-Harness']
  MCE is dominated by ['Few-Shot (all)', 'Meta-Harness']
  ACE is dominated by ['Meta-Harness']

Both methods find the same four points. Few-Shot (all) has 40.8% at 12.3K tokens, but Meta-Harness has 48.6% at 11.4K: better on both, so Few-Shot (all) drops out. ACE has 40.9% at 50.8K, and Meta-Harness beats it on both too. Zero-Shot stays on the frontier even with the lowest accuracy, because nothing uses less context than zero.

Table 2 as points: the line joins the Pareto frontier25303540455001020304050Zero-Shot: accuracy 27.4, context 0Zero-ShotFew-Shot (8): accuracy 34.3, context 2Few-Shot (8)Few-Shot (32): accuracy 35.4, context 7.9Few-Shot (32)Few-Shot (all): accuracy 40.8, context 12.3Few-Shot (all)MCE: accuracy 40, context 28.5MCEACE: accuracy 40.9, context 50.8ACEMeta-Harness: accuracy 48.6, context 11.4Meta-Harnessextra context (thousands of tokens, lower is better)accuracy % (higher is better)
The numbers of Table 2 as points. Up is more accurate and left is cheaper. The line joins the four points of the Pareto frontier. Every point off the line has a point on the line that is both higher and further left.

The search loop

Algorithm 1: the outer loop {Algorithm 1}

Algorithm 1, with the folders of our runfilesystem D (part2_fs/)000_zero_shot/001_few_shot_4/002_one_word_answer/...rejected.jsonlfrontier.jsoneach folder: harness.py, scores.json,traces.jsonl, proposal.mdproposer Preads files (grep, cat)writes k new harness filesinterface checkfails: one line in rejected.jsonlEvaluate(H, M, search)frozen model M24 search reviewsstore: new folderAfter N iterations (line 13): the Pareto frontier of D.Only then is the frontier run once on the 48 test reviews. Those results are written outside D.
Algorithm 1 with the folders of our own run. The proposer reads the filesystem and writes new harness files. Each file passes an interface check, runs on the search set with the frozen model, and becomes a new folder. The test set is used only after the loop.

Algorithm 1 as Python

We now run Algorithm 1 for real, small. The pieces:

  • Tasks. SST-2 movie reviews. A memory of 64 labelled reviews from the training split, which harnesses may show to the model. A search set of 24 reviews, and a test set of 48 other reviews.
  • The model MM. Qwen2.5-0.5B-Instruct, frozen, greedy decoding, on a laptop.
  • A harness. One Python file with one function, run(x, model, memory), that returns a label and a trace.
  • The proposer PP. A stub: a short Python function that reads the folder the way an agent would. It writes a diagnosis and picks the next edit from a fixed list. It is not a language model. Part 6 replaces it with a real local model.

The search set follows a tip from the paper's Appendix D (shown at the end of this page): make it hard for the baseline. We ran the zero-shot harness on 96 reviews and kept 12 that it got wrong and 12 that it got right.

plain text
== search set: 96 candidate reviews, zero-shot greedy
  zero-shot on the pool: 0.875 (12 misses, 1 unclear answers), 273 s
  search set: 12 misses + 12 hits = 24 reviews;  test set: 48 other reviews

Here is the first harness of the population, exactly as the loop writes it to 000_zero_shot/harness.py:

python
"""Zero-shot: ask for the label, show no examples."""

def parse(out):
    """Read the model's answer: 'positive', 'negative', or None if it is not clear."""
    o = out.lower()
    pos, neg = 'positive' in o, 'negative' in o
    return 'positive' if pos and not neg else 'negative' if neg and not pos else None


def run(x, model, memory):
    prompt = f"Review: {x}\nIs the sentiment of this review positive or negative?"
    out = model(prompt)
    return parse(out), [dict(step='answer', prompt=prompt, output=out)]

evaluate runs one harness file on a list of tasks. It returns the scores and one trace record per task. validate is line 11 of Algorithm 1: it loads the file and calls run once on a fake model.

python
def evaluate(path, tasks, model):
    """Run one harness on a list of (x, gold) tasks. Return the scores and one trace record per task."""
    h = load_harness(path)
    traces, right, chars = [], 0, 0
    for i, (x, gold) in enumerate(tasks):
        pred, steps = h.run(x, model, memory)
        right += pred == gold
        chars += sum(len(s.get('prompt', '')) for s in steps)
        traces.append(dict(id=i, x=x, gold=gold, pred=pred, correct=pred == gold, steps=steps))
    return dict(accuracy=right / len(tasks), avg_prompt_chars=chars / len(tasks), n=len(tasks)), traces


def validate(path):
    """Interface validation (line 11): the file imports, has run(), and run() answers one tiny task."""
    try:
        h = load_harness(path)
        label, trace = h.run('a fine film .', lambda prompt: 'positive', memory[:4])
        return label in ('positive', 'negative', None) and isinstance(trace, list)
    except Exception as e:                              # a syntax error, a missing run(), a crash...
        return False

store is lines 6 and 12: one new folder per harness, with the code, the scores and the traces.

python
def store(name, source, proposal, model):
    """D <- D + {(H, Evaluate(H, M, X))}: one new folder with the code, the scores and the traces."""
    d = D_ROOT / f'{len([p for p in D_ROOT.iterdir() if p.is_dir()]):03d}_{name}'
    d.mkdir()
    (d / 'harness.py').write_text(source)
    (d / 'proposal.md').write_text(proposal)
    scores, traces = evaluate(d / 'harness.py', search, model)
    (d / 'scores.json').write_text(json.dumps(scores, indent=1))
    (d / 'traces.jsonl').write_text(''.join(json.dumps(t) + '\n' for t in traces))
    log(f'  stored {d.name:24s} search acc {scores["accuracy"]:.3f}   prompt chars {scores["avg_prompt_chars"]:7.1f}')
    return d

The stub proposer is line 8 and line 9. It reads every score file, the code of the two best harnesses and every earlier proposal note. It also reads the list of rejected candidates and the traces of the best harness. It counts unclear answers and wrong labels in those traces and writes them into its note. Then it takes the next untried edits from its list. The function read is its cat: it opens a file and records the access.

python
def stub_proposer(it, k=2):
    """Stand-in for the paper's coding agent. It reads D like an agent would (scores, code, traces), writes a
    short diagnosis, and returns k new harness files. Unlike the real proposer, it picks the edits from a fixed list."""
    folders = sorted(p for p in D_ROOT.iterdir() if p.is_dir())
    scores = {f.name: json.loads(read(f / 'scores.json', it)) for f in folders}       # every score
    ranked = sorted(scores, key=lambda n: -scores[n]['accuracy'])
    for n in ranked[:2]:
        read(D_ROOT / n / 'harness.py', it)                                             # the code of the top two
    for f in folders:
        read(f / 'proposal.md', it)                                                     # what earlier steps tried
    best = ranked[0]
    traces = [json.loads(l) for l in read(D_ROOT / best / 'traces.jsonl', it).splitlines()]
    unclear = sum(t['pred'] is None for t in traces)
    wrong = sum(t['pred'] is not None and not t['correct'] for t in traces)
    tried = {f.name.split('_', 1)[1] for f in folders}
    rejected = D_ROOT / 'rejected.jsonl'
    if rejected.exists():                                                               # candidates that failed line 11
        tried |= {json.loads(l)['name'] for l in read(rejected, it).splitlines()}
    new = [(n, why, src) for n, why, src in EDITS if n not in tried][:k]
    out = []
    for n, why, src in new:
        note = (f'# iteration {it}: {n}\nRead {len(folders)} score files. Best so far: {best} '
                f'(search acc {scores[best]["accuracy"]:.3f}). In its traces: {unclear} unclear answers, '
                f'{wrong} wrong labels out of {len(traces)}.\nEdit: answers "{why}".\n')
        out.append((n, src, note))
    return out

And the loop itself, with the line numbers of Algorithm 1 in the comments:

python
if D_ROOT.exists():
    shutil.rmtree(D_ROOT)
D_ROOT.mkdir(parents=True)                                                  # line 3: D <- empty
CALLS['n'] = 0
t = time.time()
log('-- lines 4-6: evaluate the initial population')
for name, src in SEEDS.items():
    store(name, src, f'# seed: {name}\nA baseline harness from the initial population.\n', greedy)
N = 2
for it in range(1, N + 1):                                                  # line 7
    log(f'-- iteration {it}: the proposer reads D and proposes')
    for name, src, note in stub_proposer(it):                               # lines 8-9
        cand = D_ROOT.parent / 'part2_candidate.py'
        cand.write_text(src)
        ok = validate(cand)                                                 # line 11
        cand.unlink()
        if ok:
            store(name, src, note, greedy)                                  # line 12
        else:
            log(f'  rejected {name}: failed interface validation (not evaluated; one line in rejected.jsonl)')
            with open(D_ROOT / 'rejected.jsonl', 'a') as fh:
                fh.write(json.dumps(dict(iteration=it, name=name, reason='interface validation failed')) + '\n')
log(f'  search done: {CALLS["n"]} model calls, {time.time() - t:.0f} s')

# line 13: the Pareto frontier of D (search accuracy up, prompt characters down)
pts = {f.name: (json.loads((f / 'scores.json').read_text())['accuracy'],
                json.loads((f / 'scores.json').read_text())['avg_prompt_chars']) for f in sorted(D_ROOT.iterdir()) if f.is_dir()}
front = pareto_sweep(pts)
assert front == pareto_brute(pts)
log(f'-- line 13: Pareto frontier of D: {front}')
(D_ROOT / 'frontier.json').write_text(json.dumps(front, indent=1))

# After the search: the test set, once, on the frontier only. Written outside D.
log('')
log('== after the search: the held-out test set (48 reviews), frontier only')
T = {}
for n in front:
    s, _ = evaluate(D_ROOT / n / 'harness.py', test, greedy)
    T[n] = dict(search=pts[n][0], test=s['accuracy'], prompt_chars=s['avg_prompt_chars'])
    log(f'  {n:24s} search {pts[n][0]:.3f}   test {s["accuracy"]:.3f}   gap {pts[n][0] - s["accuracy"]:+.3f}')
TEST_OUT.write_text(json.dumps(T, indent=1))
plain text
== 4. Algorithm 1: the outer loop
-- lines 4-6: evaluate the initial population
  stored 000_zero_shot            search acc 0.500   prompt chars   178.9
  stored 001_few_shot_4           search acc 0.167   prompt chars   430.9
-- iteration 1: the proposer reads D and proposes
  stored 002_one_word_answer      search acc 0.583   prompt chars   230.9
  rejected broken_rename: failed interface validation (not evaluated; one line in rejected.jsonl)
-- iteration 2: the proposer reads D and proposes
  stored 003_retrieve_4           search acc 0.333   prompt chars   637.5
  stored 004_retrieve_balanced    search acc 0.292   prompt chars   660.8
  search done: 120 model calls, 315 s
-- line 13: Pareto frontier of D: ['000_zero_shot', '002_one_word_answer']
== after the search: the held-out test set (48 reviews), frontier only
  000_zero_shot            search 0.500   test 0.917   gap -0.417
  002_one_word_answer      search 0.583   test 0.938   gap -0.354

Read the log from the top. The two seeds score 0.500 and 0.167 on the search set.

In iteration 1 the stub reads the folder and sees one unclear answer in the best harness. It proposes two files. The one-word-answer harness scores 0.583, the best so far. The renamed file fails the interface check in a second, and it costs no model call at all. In iteration 2 the stub proposes two retrieval harnesses, and both score worse than zero-shot.

The frontier has two harnesses. Zero-shot uses the shortest prompts (178.9 characters on average). The one-word-answer harness has the best accuracy for 230.9 characters. Each of the other three is both less accurate and longer, so each is dominated.

Now the traces. Here is the reading a real proposer would do, in a few lines of Python over the saved folder. It calls no model.

python
for folder in sorted(p for p in D.iterdir() if p.is_dir()):
    traces = [json.loads(l) for l in (folder / 'traces.jsonl').read_text().splitlines()]
    unclear = [t for t in traces if t['pred'] is None]
    wrong = [t for t in traces if t['pred'] is not None and not t['correct']]
    log(f'{folder.name:22s} unclear {len(unclear):2d}   wrong {len(wrong):2d}   right {len(traces) - len(unclear) - len(wrong):2d}   of {len(traces)}')
    if unclear:
        out = unclear[0]['steps'][0]['output'].replace('\n', ' ')
        log(f'   first unclear answer: {out[:90]!r}')
plain text
== what the traces of each harness say
000_zero_shot          unclear  1   wrong 11   right 12   of 24
   first unclear answer: 'The sentiment of this review is clearly negative. The reviewer criticizes Jason X for bein'
001_few_shot_4         unclear 12   wrong  8   right  4   of 24
   first unclear answer: 'Based on the sentiment analysis provided for each review, I would categorize them as follo'
002_one_word_answer    unclear  0   wrong 10   right 14   of 24
003_retrieve_4         unclear  8   wrong  8   right  8   of 24
   first unclear answer: "I apologize, but I'm not able to review or provide feedback on any specific movie reviews "
004_retrieve_balanced  unclear  8   wrong  9   right  7   of 24
   first unclear answer: 'The sentiment of this review can be categorized as **negative**.  ### Analysis: 1. **Posit'

This output holds the real diagnosis. The few-shot harness gave 12 unclear answers out of 24. Its first one starts "Based on the sentiment analysis provided for each review, I would categorize them as follows". With four labelled reviews in one message, the small model thinks it must analyse all five reviews, and it never gives one label. The retrieval harnesses copy that prompt format, so they copy the bug: 8 unclear answers each.

Our stub cannot see this. It counts unclear answers only in the best harness, and it picks its edits from a list. A proposer that reads the few-shot traces would see the bug at once. It might then write a harness that asks for one word after the examples. This is the paper's argument in small form: the score said "few-shot is bad", but only the trace said why.

Last, the test set. Both frontier harnesses score much higher on the 48 test reviews (0.917 and 0.938) than on the search set.

That is not a miracle. The search set was built to be hard: half of it is reviews that zero-shot got wrong. The test set is a random sample, where zero-shot gets most reviews right. So the two numbers measure different things, and only the test number estimates accuracy on new reviews. On the test set, the one-word-answer harness is ahead by 0.021, which is one review out of 48.

Our run: the frontier on the search set and on the test setzero_shot000_zero_shot search: 0.500search 0.500000_zero_shot test: 0.917test 0.917one_word_answer002_one_word_answer search: 0.583search 0.583002_one_word_answer test: 0.938test 0.938
The harnesses on the Pareto frontier of our run, with their accuracy on the 24 search reviews and on the 48 test reviews. The search set was built to be hard for zero-shot, so the two scores are not the same measurement.

The whole search made 120 model calls and took 315 seconds on a laptop GPU that other jobs also used. The paper's runs evaluate about 60 harnesses with large models; Part 5 discusses that cost.

Why search in code space

How the proposer uses the filesystem {Appendix A.1, Table 8}

Our stub proposer records its reads in the same way. The numbers below are not a result about agents: the stub follows a fixed reading plan, and our folder is tiny. They only show the bookkeeping that produced Table 8.

plain text
== the proposer's file reads (Table 8 style; a scripted proposer, so only a picture of the bookkeeping)
  iteration 1: 7 files read
  iteration 2: 10 files read
  source   4   23.5%
  trace    2   11.8%
  score    5   29.4%
  other    6   35.3%

The stub read 7 files in iteration 1 and 10 in iteration 2, because the folder had grown. Its mix is very different from Table 8: it reads every score file but only one trace file per iteration. That is the stub's fixed plan, and it is the reason it missed the few-shot bug above. The paper's agent spent 40% of its reads on traces.

The folder, listed and searched

The paper's proposer moves through the folder with standard tools. Here is our folder after the run. Method 1 walks it with our own loop over os.walk. Method 2 lets Python's pathlib library do the walking. Then the same two ways for grep: a search in Python, and the real grep program that an agent would call.

python
def tree(root):
    """Method 1: walk the folders ourselves with os.walk; return every file path relative to root."""
    out = []
    for folder, dirs, files in os.walk(root):
        dirs.sort()
        for f in sorted(files):
            out.append(os.path.relpath(os.path.join(folder, f), root))
    return sorted(out)


def tree_lib(root):
    """Method 2: the standard library's pathlib does the walking."""
    return sorted(str(p.relative_to(root)) for p in Path(root).rglob('*') if p.is_file())


def grep(root, pattern):
    """Method 1: grep -rl in Python. Return the files that contain the pattern."""
    rx = re.compile(pattern)
    return sorted(p for p in tree(root) if rx.search((Path(root) / p).read_text()))


def grep_tool(root, pattern):
    """Method 2: the real grep program, the tool the paper's proposer calls."""
    out = subprocess.run(['grep', '-rlE', pattern, '.'], cwd=root, capture_output=True, text=True).stdout
    return sorted(l[2:] for l in out.split())
plain text
== 5. The filesystem D
  000_zero_shot/harness.py                     520 bytes
  000_zero_shot/proposal.md                     66 bytes
  000_zero_shot/scores.json                     71 bytes
  000_zero_shot/traces.jsonl                13,110 bytes
  001_few_shot_4/harness.py                    589 bytes
  001_few_shot_4/proposal.md                    67 bytes
  001_few_shot_4/scores.json                    86 bytes
  001_few_shot_4/traces.jsonl               19,204 bytes
  002_one_word_answer/harness.py               605 bytes
  002_one_word_answer/proposal.md              209 bytes
  002_one_word_answer/scores.json               86 bytes
  002_one_word_answer/traces.jsonl          11,852 bytes
  003_retrieve_4/harness.py                    825 bytes
  003_retrieve_4/proposal.md                   229 bytes
  003_retrieve_4/scores.json                    73 bytes
  003_retrieve_4/traces.jsonl               28,608 bytes
  004_retrieve_balanced/harness.py           1,009 bytes
  004_retrieve_balanced/proposal.md            228 bytes
  004_retrieve_balanced/scores.json             85 bytes
  004_retrieve_balanced/traces.jsonl        24,594 bytes
  frontier.json                                 44 bytes
  rejected.jsonl                                83 bytes
  os.walk: 22 files, pathlib: 22 files, same list: True
  files with an unclear answer ("pred": null): ['000_zero_shot/traces.jsonl', '001_few_shot_4/traces.jsonl', '003_retrieve_4/traces.jsonl', '004_retrieve_balanced/traces.jsonl']
  Python grep and grep -rlE agree: True
  D holds 102,243 characters: about 25,561 tokens by the rule of thumb, 25,303 by the Qwen tokenizer
  Table 1 says up to 10,000,000 tokens per evaluation in the paper: 395x our whole D

Each evaluated harness has its own folder with four files: harness.py (the code), proposal.md (why it was written), scores.json and traces.jsonl. The rejected candidate left one line in rejected.jsonl and no folder. The traces are by far the largest files, as in the paper.

Both ways of listing give the same 22 files, and both ways of searching give the same four trace files with an unclear answer. One grep found every harness that has the parsing problem, without opening a single file by hand. That is the kind of query the paper's proposer runs.

The last two lines compare sizes. A token count by the rule of thumb (characters divided by 4) gives 25,561; the real Qwen tokenizer gives 25,303, close enough. Our whole folder, after five harnesses, is about the size of one TTT-Discover step in Table 1 (26,000 tokens). The paper's 10 million tokens per evaluation is 395 times our whole folder.

Terminal output of mh_part2.py with every result on this page

Practical tips from the authors {Appendix D}

Appendix D lists what the authors learned while building the system. They say clearly that these are engineering lessons, not scientific claims. Four of them shape how anyone would build this loop.

What a small local run cannot show

Everything above runs, but at a toy scale, and it is fair to say what that hides. Our task model has 0.5 billion parameters; the paper's text-classification model is GPT-OSS-120B. Our search set has 24 reviews, so one review moves the score by 1/24.

Above all, our proposer is a script. It reads the folder and writes a diagnosis, but it does not reason over the traces, and its edits come from a list we wrote. The paper's main claim is that a strong coding agent can do that reasoning. A stub cannot test that claim; it can only show the machinery around it. Part 6 tries a local language model as the proposer and reports what happens.

The paper and the code

Paper (arXiv 2603.28052) · Project page · Our scripts for this series