Meta-Harness, explained · Part 2 of 6 · Covers §2, Table 1, §3, Figure 2, Algorithm 1, Appendix A.1, Table 8, Appendix D, Appendix E
The Method: A Loop That Reads Every Earlier Attempt
How Meta-Harness searches for better harness code: the goal written as one formula, the loop of Figure 2 and Algorithm 1, and the folder of past attempts that the proposer reads. We run the whole loop in Python with a small local model and a scripted proposer.
Meta-Harness: End-to-End Optimization of Model Harnesses. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn. COLM 2026, 2026. arXiv:2603.28052
Meta-Harness is a loop. An agent reads a folder full of earlier attempts, writes a new harness, and the loop scores it and adds it to the folder. Each piece below shows the exact lines of the paper, then explains them, and then runs them as code on a laptop.
Table 1 drawn on a log scale, where each grid line is ten times the one before. The six earlier methods sit between 2,000 and 26,000 tokens. Meta-Harness sits at 10,000,000.
A few lines of Python turn the table into ratios:
python
TABLE1 = {'OPRO': 0.002, 'TextGrad': 0.015, 'AlphaEvolve': 0.022, 'GEPA': 0.008, 'Feedback Descent': 0.012, 'TTT-Discover': 0.026, 'Meta-Harness': 10.0}largest_prior = max(v for k, v in TABLE1.items() if k != 'Meta-Harness')
plain text
== 1. Table 1: context per evaluation (MTok/iter, the paper's estimates) OPRO 0.002 MTok = 2,000 tokens x 0.1 of the largest prior TextGrad 0.015 MTok = 15,000 tokens x 0.6 of the largest prior AlphaEvolve 0.022 MTok = 22,000 tokens x 0.8 of the largest prior GEPA 0.008 MTok = 8,000 tokens x 0.3 of the largest prior Feedback Descent 0.012 MTok = 12,000 tokens x 0.5 of the largest prior TTT-Discover 0.026 MTok = 26,000 tokens x 1.0 of the largest prior Meta-Harness 10.000 MTok = 10,000,000 tokens x 384.6 of the largest prior Meta-Harness / largest prior (TTT-Discover): 385x log10 of that ratio: 2.59 orders of magnitude
The largest earlier budget is TTT-Discover's 26,000 tokens. Meta-Harness has about 385 times that. On a log scale, 385 is 2.59 steps of ten. The paper rounds this to "three orders of magnitude" in §1.
r(τ,x) is the reward: one number that says how well the run τ did on the task x. For a classification task it is 1 for the right label and 0 for a wrong one.
τ is the rollout trajectory: everything that happened in one run. The harness builds a prompt, the model answers, the harness updates its state, and so on.
τ∼pM(H,x) says that the run is random. The same harness H and task x can give different runs, because the model M can sample different words. pM is the chance of each run.
x∼X says that the task is drawn from a distribution X: the kind of tasks we care about, such as "movie reviews to label".
E is the expected value: the average reward over both kinds of chance, the task and the run.
argmaxH means "the H that makes the average largest". H∗ is that best harness. Note what is missing: M does not appear under the argmax. The model is fixed; only H may change.
A worked example. Take two tasks, x1 and x2, each with a chance of 0.5. Take two harnesses. With harness HA the model gets x1 right 90% of the time and x2 right 50% of the time. With HB the numbers are 70% and 80%. The reward is 1 for right and 0 for wrong.
So H∗=HB. Note that HA is the better harness on x1. It still loses, because the objective averages over all tasks. A harness that shines on one example and fails on others does not win.
Method 1 and Method 2: the exact sum and the sampled average#
Method 1 follows the definition: a sum over tasks and runs, each weighted by its chance. Method 2 does what an evaluation really does: draw tasks, run them, and average the rewards. The second way uses NumPy, the standard Python library for arrays of numbers.
python
P_TASK = {'x1': 0.5, 'x2': 0.5}P_RIGHT = {'H_A': {'x1': 0.9, 'x2': 0.5}, 'H_B': {'x1': 0.7, 'x2': 0.8}}def expected_reward_exact(h): """Method 1: the definition. Sum over tasks of p(x) times the sum over rollouts of p(tau) r(tau, x).""" total = 0.0 for x, px in P_TASK.items(): p = P_RIGHT[h][x] total += px * (p * 1.0 + (1 - p) * 0.0) # two rollouts: right (reward 1) or wrong (reward 0) return totaldef expected_reward_sampled(h, n=100_000, seed=0): """Method 2: Monte Carlo. Draw a task, draw a rollout, average the rewards (what evaluation really does).""" rng = np.random.default_rng(seed) xs = rng.choice(list(P_TASK), size=n, p=list(P_TASK.values())) # x ~ X p = np.array([P_RIGHT[h][x] for x in xs]) r = (rng.random(n) < p).astype(float) # tau ~ p_M(H, x), r(tau, x) in {0, 1} return r.mean(), r.std() / np.sqrt(n)
plain text
== 2. The objective: E over tasks x and rollouts tau of r(tau, x) H_A: exact 0.7000 Monte Carlo 0.7011 (+/- 0.0014) difference 0.0011 H_B: exact 0.7500 Monte Carlo 0.7513 (+/- 0.0014) difference 0.0013 argmax over the two harnesses: H_B
The two methods agree to about 0.001. The sampled average is never exact: the "+/-" is its standard error, the typical size of its mistake. With 100,000 draws it is 0.0014. A real evaluation has far fewer draws, and the next run shows what that means.
Now the same objective with a real language model. The task model M is Qwen2.5-0.5B-Instruct, a small open model, run locally. The task is SST-2, a public set of movie-review sentences labelled positive or negative. We estimate the expected reward of two harnesses on 24 reviews (the search set built further below). Each harness runs three times with random sampling (temperature 1.0), and once in greedy mode, which always picks the most likely word.
plain text
== 2b. E[r] with Qwen2.5-0.5B-Instruct on the 24 search reviews zero_shot sampled (T=1.0) runs [0.708, 0.458, 0.583] -> mean 0.583; greedy 0.500 few_shot_4 sampled (T=1.0) runs [0.125, 0.208, 0.208] -> mean 0.181; greedy 0.167 argmax of the sampled estimate: zero_shot
Look at the three sampled runs of zero-shot: 0.708, 0.458 and 0.583. Same harness, same 24 reviews, same model, and the score moves by 0.25. That is τ∼pM(H,x) at work. One run is a noisy guess of the expected reward; the mean of three runs, 0.583, is a better one.
On this search set, the zero-shot harness wins clearly over the few-shot harness (0.583 against 0.181). That looks odd, since examples usually help. The traces explain it, and we read them further below. Note also that the greedy score of zero-shot is exactly 0.500. That is by design: we built the search set from 12 reviews that greedy zero-shot got wrong and 12 that it got right.
Here is Pareto dominance in code, run on the numbers of the paper's Table 2 (accuracy in %, extra context in thousands of tokens; Part 3 explains that table). Method 1 checks every pair. Method 2 is the classic faster way: sort by context, then walk along and keep each point that beats the best accuracy so far.
python
def dominates(a, b): """a = (accuracy, context). a dominates b if it is no worse on both and strictly better on one.""" return a[0] >= b[0] and a[1] <= b[1] and (a[0] > b[0] or a[1] < b[1])def pareto_brute(points): """Method 1: keep every point that no other point dominates (compare all pairs).""" return sorted(k for k, p in points.items() if not any(dominates(q, p) for j, q in points.items() if j != k))def pareto_sweep(points): """Method 2: sort by context (then by accuracy, high first); keep a point if it beats the best accuracy so far.""" out, best = [], -float('inf') for k, (acc, ctx) in sorted(points.items(), key=lambda kv: (kv[1][1], -kv[1][0])): if acc > best: out.append(k) best = acc return sorted(out)
plain text
== 3. Pareto dominance: higher accuracy and lower context are both better brute force: ['Few-Shot (32)', 'Few-Shot (8)', 'Meta-Harness', 'Zero-Shot'] sweep : ['Few-Shot (32)', 'Few-Shot (8)', 'Meta-Harness', 'Zero-Shot'] same frontier: True Few-Shot (all) is dominated by ['Meta-Harness'] MCE is dominated by ['Few-Shot (all)', 'Meta-Harness'] ACE is dominated by ['Meta-Harness']
Both methods find the same four points. Few-Shot (all) has 40.8% at 12.3K tokens, but Meta-Harness has 48.6% at 11.4K: better on both, so Few-Shot (all) drops out. ACE has 40.9% at 50.8K, and Meta-Harness beats it on both too. Zero-Shot stays on the frontier even with the lowest accuracy, because nothing uses less context than zero.
The numbers of Table 2 as points. Up is more accurate and left is cheaper. The line joins the four points of the Pareto frontier. Every point off the line has a point on the line that is both higher and further left.
Algorithm 1 with the folders of our own run. The proposer reads the filesystem and writes new harness files. Each file passes an interface check, runs on the search set with the frozen model, and becomes a new folder. The test set is used only after the loop.
We now run Algorithm 1 for real, small. The pieces:
Tasks. SST-2 movie reviews. A memory of 64 labelled reviews from the training split, which harnesses may show to the model. A search set of 24 reviews, and a test set of 48 other reviews.
The model M. Qwen2.5-0.5B-Instruct, frozen, greedy decoding, on a laptop.
A harness. One Python file with one function, run(x, model, memory), that returns a label and a trace.
The proposer P. A stub: a short Python function that reads the folder the way an agent would. It writes a diagnosis and picks the next edit from a fixed list. It is not a language model. Part 6 replaces it with a real local model.
The search set follows a tip from the paper's Appendix D (shown at the end of this page): make it hard for the baseline. We ran the zero-shot harness on 96 reviews and kept 12 that it got wrong and 12 that it got right.
plain text
== search set: 96 candidate reviews, zero-shot greedy zero-shot on the pool: 0.875 (12 misses, 1 unclear answers), 273 s search set: 12 misses + 12 hits = 24 reviews; test set: 48 other reviews
Here is the first harness of the population, exactly as the loop writes it to 000_zero_shot/harness.py:
python
"""Zero-shot: ask for the label, show no examples."""def parse(out): """Read the model's answer: 'positive', 'negative', or None if it is not clear.""" o = out.lower() pos, neg = 'positive' in o, 'negative' in o return 'positive' if pos and not neg else 'negative' if neg and not pos else Nonedef run(x, model, memory): prompt = f"Review: {x}\nIs the sentiment of this review positive or negative?" out = model(prompt) return parse(out), [dict(step='answer', prompt=prompt, output=out)]
evaluate runs one harness file on a list of tasks. It returns the scores and one trace record per task. validate is line 11 of Algorithm 1: it loads the file and calls run once on a fake model.
python
def evaluate(path, tasks, model): """Run one harness on a list of (x, gold) tasks. Return the scores and one trace record per task.""" h = load_harness(path) traces, right, chars = [], 0, 0 for i, (x, gold) in enumerate(tasks): pred, steps = h.run(x, model, memory) right += pred == gold chars += sum(len(s.get('prompt', '')) for s in steps) traces.append(dict(id=i, x=x, gold=gold, pred=pred, correct=pred == gold, steps=steps)) return dict(accuracy=right / len(tasks), avg_prompt_chars=chars / len(tasks), n=len(tasks)), tracesdef validate(path): """Interface validation (line 11): the file imports, has run(), and run() answers one tiny task.""" try: h = load_harness(path) label, trace = h.run('a fine film .', lambda prompt: 'positive', memory[:4]) return label in ('positive', 'negative', None) and isinstance(trace, list) except Exception as e: # a syntax error, a missing run(), a crash... return False
store is lines 6 and 12: one new folder per harness, with the code, the scores and the traces.
python
def store(name, source, proposal, model): """D <- D + {(H, Evaluate(H, M, X))}: one new folder with the code, the scores and the traces.""" d = D_ROOT / f'{len([p for p in D_ROOT.iterdir() if p.is_dir()]):03d}_{name}' d.mkdir() (d / 'harness.py').write_text(source) (d / 'proposal.md').write_text(proposal) scores, traces = evaluate(d / 'harness.py', search, model) (d / 'scores.json').write_text(json.dumps(scores, indent=1)) (d / 'traces.jsonl').write_text(''.join(json.dumps(t) + '\n' for t in traces)) log(f' stored {d.name:24s} search acc {scores["accuracy"]:.3f} prompt chars {scores["avg_prompt_chars"]:7.1f}') return d
The stub proposer is line 8 and line 9. It reads every score file, the code of the two best harnesses and every earlier proposal note. It also reads the list of rejected candidates and the traces of the best harness. It counts unclear answers and wrong labels in those traces and writes them into its note. Then it takes the next untried edits from its list. The function read is its cat: it opens a file and records the access.
python
def stub_proposer(it, k=2): """Stand-in for the paper's coding agent. It reads D like an agent would (scores, code, traces), writes a short diagnosis, and returns k new harness files. Unlike the real proposer, it picks the edits from a fixed list.""" folders = sorted(p for p in D_ROOT.iterdir() if p.is_dir()) scores = {f.name: json.loads(read(f / 'scores.json', it)) for f in folders} # every score ranked = sorted(scores, key=lambda n: -scores[n]['accuracy']) for n in ranked[:2]: read(D_ROOT / n / 'harness.py', it) # the code of the top two for f in folders: read(f / 'proposal.md', it) # what earlier steps tried best = ranked[0] traces = [json.loads(l) for l in read(D_ROOT / best / 'traces.jsonl', it).splitlines()] unclear = sum(t['pred'] is None for t in traces) wrong = sum(t['pred'] is not None and not t['correct'] for t in traces) tried = {f.name.split('_', 1)[1] for f in folders} rejected = D_ROOT / 'rejected.jsonl' if rejected.exists(): # candidates that failed line 11 tried |= {json.loads(l)['name'] for l in read(rejected, it).splitlines()} new = [(n, why, src) for n, why, src in EDITS if n not in tried][:k] out = [] for n, why, src in new: note = (f'# iteration {it}: {n}\nRead {len(folders)} score files. Best so far: {best} ' f'(search acc {scores[best]["accuracy"]:.3f}). In its traces: {unclear} unclear answers, ' f'{wrong} wrong labels out of {len(traces)}.\nEdit: answers "{why}".\n') out.append((n, src, note)) return out
And the loop itself, with the line numbers of Algorithm 1 in the comments:
python
if D_ROOT.exists(): shutil.rmtree(D_ROOT)D_ROOT.mkdir(parents=True) # line 3: D <- emptyCALLS['n'] = 0t = time.time()log('-- lines 4-6: evaluate the initial population')for name, src in SEEDS.items(): store(name, src, f'# seed: {name}\nA baseline harness from the initial population.\n', greedy)N = 2for it in range(1, N + 1): # line 7 log(f'-- iteration {it}: the proposer reads D and proposes') for name, src, note in stub_proposer(it): # lines 8-9 cand = D_ROOT.parent / 'part2_candidate.py' cand.write_text(src) ok = validate(cand) # line 11 cand.unlink() if ok: store(name, src, note, greedy) # line 12 else: log(f' rejected {name}: failed interface validation (not evaluated; one line in rejected.jsonl)') with open(D_ROOT / 'rejected.jsonl', 'a') as fh: fh.write(json.dumps(dict(iteration=it, name=name, reason='interface validation failed')) + '\n')log(f' search done: {CALLS["n"]} model calls, {time.time() - t:.0f} s')# line 13: the Pareto frontier of D (search accuracy up, prompt characters down)pts = {f.name: (json.loads((f / 'scores.json').read_text())['accuracy'], json.loads((f / 'scores.json').read_text())['avg_prompt_chars']) for f in sorted(D_ROOT.iterdir()) if f.is_dir()}front = pareto_sweep(pts)assert front == pareto_brute(pts)log(f'-- line 13: Pareto frontier of D: {front}')(D_ROOT / 'frontier.json').write_text(json.dumps(front, indent=1))# After the search: the test set, once, on the frontier only. Written outside D.log('')log('== after the search: the held-out test set (48 reviews), frontier only')T = {}for n in front: s, _ = evaluate(D_ROOT / n / 'harness.py', test, greedy) T[n] = dict(search=pts[n][0], test=s['accuracy'], prompt_chars=s['avg_prompt_chars']) log(f' {n:24s} search {pts[n][0]:.3f} test {s["accuracy"]:.3f} gap {pts[n][0] - s["accuracy"]:+.3f}')TEST_OUT.write_text(json.dumps(T, indent=1))
plain text
== 4. Algorithm 1: the outer loop-- lines 4-6: evaluate the initial population stored 000_zero_shot search acc 0.500 prompt chars 178.9 stored 001_few_shot_4 search acc 0.167 prompt chars 430.9-- iteration 1: the proposer reads D and proposes stored 002_one_word_answer search acc 0.583 prompt chars 230.9 rejected broken_rename: failed interface validation (not evaluated; one line in rejected.jsonl)-- iteration 2: the proposer reads D and proposes stored 003_retrieve_4 search acc 0.333 prompt chars 637.5 stored 004_retrieve_balanced search acc 0.292 prompt chars 660.8 search done: 120 model calls, 315 s-- line 13: Pareto frontier of D: ['000_zero_shot', '002_one_word_answer']== after the search: the held-out test set (48 reviews), frontier only 000_zero_shot search 0.500 test 0.917 gap -0.417 002_one_word_answer search 0.583 test 0.938 gap -0.354
Read the log from the top. The two seeds score 0.500 and 0.167 on the search set.
In iteration 1 the stub reads the folder and sees one unclear answer in the best harness. It proposes two files. The one-word-answer harness scores 0.583, the best so far. The renamed file fails the interface check in a second, and it costs no model call at all. In iteration 2 the stub proposes two retrieval harnesses, and both score worse than zero-shot.
The frontier has two harnesses. Zero-shot uses the shortest prompts (178.9 characters on average). The one-word-answer harness has the best accuracy for 230.9 characters. Each of the other three is both less accurate and longer, so each is dominated.
Now the traces. Here is the reading a real proposer would do, in a few lines of Python over the saved folder. It calls no model.
python
for folder in sorted(p for p in D.iterdir() if p.is_dir()): traces = [json.loads(l) for l in (folder / 'traces.jsonl').read_text().splitlines()] unclear = [t for t in traces if t['pred'] is None] wrong = [t for t in traces if t['pred'] is not None and not t['correct']] log(f'{folder.name:22s} unclear {len(unclear):2d} wrong {len(wrong):2d} right {len(traces) - len(unclear) - len(wrong):2d} of {len(traces)}') if unclear: out = unclear[0]['steps'][0]['output'].replace('\n', ' ') log(f' first unclear answer: {out[:90]!r}')
plain text
== what the traces of each harness say000_zero_shot unclear 1 wrong 11 right 12 of 24 first unclear answer: 'The sentiment of this review is clearly negative. The reviewer criticizes Jason X for bein'001_few_shot_4 unclear 12 wrong 8 right 4 of 24 first unclear answer: 'Based on the sentiment analysis provided for each review, I would categorize them as follo'002_one_word_answer unclear 0 wrong 10 right 14 of 24003_retrieve_4 unclear 8 wrong 8 right 8 of 24 first unclear answer: "I apologize, but I'm not able to review or provide feedback on any specific movie reviews "004_retrieve_balanced unclear 8 wrong 9 right 7 of 24 first unclear answer: 'The sentiment of this review can be categorized as **negative**. ### Analysis: 1. **Posit'
This output holds the real diagnosis. The few-shot harness gave 12 unclear answers out of 24. Its first one starts "Based on the sentiment analysis provided for each review, I would categorize them as follows". With four labelled reviews in one message, the small model thinks it must analyse all five reviews, and it never gives one label. The retrieval harnesses copy that prompt format, so they copy the bug: 8 unclear answers each.
Our stub cannot see this. It counts unclear answers only in the best harness, and it picks its edits from a list. A proposer that reads the few-shot traces would see the bug at once. It might then write a harness that asks for one word after the examples. This is the paper's argument in small form: the score said "few-shot is bad", but only the trace said why.
Last, the test set. Both frontier harnesses score much higher on the 48 test reviews (0.917 and 0.938) than on the search set.
That is not a miracle. The search set was built to be hard: half of it is reviews that zero-shot got wrong. The test set is a random sample, where zero-shot gets most reviews right. So the two numbers measure different things, and only the test number estimates accuracy on new reviews. On the test set, the one-word-answer harness is ahead by 0.021, which is one review out of 48.
The harnesses on the Pareto frontier of our run, with their accuracy on the 24 search reviews and on the 48 test reviews. The search set was built to be hard for zero-shot, so the two scores are not the same measurement.
The whole search made 120 model calls and took 315 seconds on a laptop GPU that other jobs also used. The paper's runs evaluate about 60 harnesses with large models; Part 5 discusses that cost.
How the proposer uses the filesystem {Appendix A.1, Table 8}#
Our stub proposer records its reads in the same way. The numbers below are not a result about agents: the stub follows a fixed reading plan, and our folder is tiny. They only show the bookkeeping that produced Table 8.
plain text
== the proposer's file reads (Table 8 style; a scripted proposer, so only a picture of the bookkeeping) iteration 1: 7 files read iteration 2: 10 files read source 4 23.5% trace 2 11.8% score 5 29.4% other 6 35.3%
The stub read 7 files in iteration 1 and 10 in iteration 2, because the folder had grown. Its mix is very different from Table 8: it reads every score file but only one trace file per iteration. That is the stub's fixed plan, and it is the reason it missed the few-shot bug above. The paper's agent spent 40% of its reads on traces.
The paper's proposer moves through the folder with standard tools. Here is our folder after the run. Method 1 walks it with our own loop over os.walk. Method 2 lets Python's pathlib library do the walking. Then the same two ways for grep: a search in Python, and the real grep program that an agent would call.
python
def tree(root): """Method 1: walk the folders ourselves with os.walk; return every file path relative to root.""" out = [] for folder, dirs, files in os.walk(root): dirs.sort() for f in sorted(files): out.append(os.path.relpath(os.path.join(folder, f), root)) return sorted(out)def tree_lib(root): """Method 2: the standard library's pathlib does the walking.""" return sorted(str(p.relative_to(root)) for p in Path(root).rglob('*') if p.is_file())def grep(root, pattern): """Method 1: grep -rl in Python. Return the files that contain the pattern.""" rx = re.compile(pattern) return sorted(p for p in tree(root) if rx.search((Path(root) / p).read_text()))def grep_tool(root, pattern): """Method 2: the real grep program, the tool the paper's proposer calls.""" out = subprocess.run(['grep', '-rlE', pattern, '.'], cwd=root, capture_output=True, text=True).stdout return sorted(l[2:] for l in out.split())
plain text
== 5. The filesystem D 000_zero_shot/harness.py 520 bytes 000_zero_shot/proposal.md 66 bytes 000_zero_shot/scores.json 71 bytes 000_zero_shot/traces.jsonl 13,110 bytes 001_few_shot_4/harness.py 589 bytes 001_few_shot_4/proposal.md 67 bytes 001_few_shot_4/scores.json 86 bytes 001_few_shot_4/traces.jsonl 19,204 bytes 002_one_word_answer/harness.py 605 bytes 002_one_word_answer/proposal.md 209 bytes 002_one_word_answer/scores.json 86 bytes 002_one_word_answer/traces.jsonl 11,852 bytes 003_retrieve_4/harness.py 825 bytes 003_retrieve_4/proposal.md 229 bytes 003_retrieve_4/scores.json 73 bytes 003_retrieve_4/traces.jsonl 28,608 bytes 004_retrieve_balanced/harness.py 1,009 bytes 004_retrieve_balanced/proposal.md 228 bytes 004_retrieve_balanced/scores.json 85 bytes 004_retrieve_balanced/traces.jsonl 24,594 bytes frontier.json 44 bytes rejected.jsonl 83 bytes os.walk: 22 files, pathlib: 22 files, same list: True files with an unclear answer ("pred": null): ['000_zero_shot/traces.jsonl', '001_few_shot_4/traces.jsonl', '003_retrieve_4/traces.jsonl', '004_retrieve_balanced/traces.jsonl'] Python grep and grep -rlE agree: True D holds 102,243 characters: about 25,561 tokens by the rule of thumb, 25,303 by the Qwen tokenizer Table 1 says up to 10,000,000 tokens per evaluation in the paper: 395x our whole D
Each evaluated harness has its own folder with four files: harness.py (the code), proposal.md (why it was written), scores.json and traces.jsonl. The rejected candidate left one line in rejected.jsonl and no folder. The traces are by far the largest files, as in the paper.
Both ways of listing give the same 22 files, and both ways of searching give the same four trace files with an unclear answer. One grep found every harness that has the parsing problem, without opening a single file by hand. That is the kind of query the paper's proposer runs.
The last two lines compare sizes. A token count by the rule of thumb (characters divided by 4) gives 25,561; the real Qwen tokenizer gives 25,303, close enough. Our whole folder, after five harnesses, is about the size of one TTT-Discover step in Table 1 (26,000 tokens). The paper's 10 million tokens per evaluation is 395 times our whole folder.
Appendix D lists what the authors learned while building the system. They say clearly that these are engineering lessons, not scientific claims. Four of them shape how anyone would build this loop.
Everything above runs, but at a toy scale, and it is fair to say what that hides. Our task model has 0.5 billion parameters; the paper's text-classification model is GPT-OSS-120B. Our search set has 24 reviews, so one review moves the score by 1/24.
Above all, our proposer is a script. It reads the folder and writes a diagnosis, but it does not reason over the traces, and its edits come from a list we wrote. The paper's main claim is that a strong coding agent can do that reasoning. A stub cannot test that claim; it can only show the machinery around it. Part 6 tries a local language model as the proposer and reports what happens.