Meta-Harness, read beside the paper

The paper and its explanation side by side.

  • Part 1: The Big Idea: Let an Agent Write the Harness

    Why the code around a language model can matter as much as the model, and the paper's proposal: a coding agent that reads every earlier attempt and writes a better one. We read the title, Figure 1, the abstract and §1, and we run one small model with four different harnesses.

  • Part 2: The Method: A Loop That Reads Every Earlier Attempt

    How Meta-Harness searches for better harness code: the goal written as one formula, the loop of Figure 2 and Algorithm 1, and the folder of past attempts that the proposer reads. We run the whole loop in Python with a small local model and a scripted proposer.

  • Part 3: Text Classification: What the Search Found

    The first test of Meta-Harness: a model sorts short texts into classes, and the search writes the code that picks its examples. We read the tables, the accuracy and context trade-off, and the two discovered harnesses, then rebuild both around a small local model.

  • Part 4: Math Retrieval and TerminalBench-2

    How Meta-Harness found a way to look up solved problems for olympiad math, and a better harness for an agent that works in a terminal. We build the search engine and the discovered retriever in Python, test them with a small local model, and run two key pieces of the released agent code.

  • Part 5: Limits, and What Came After

    Where Meta-Harness can fool itself, what a search costs, and why it rests on one strong proposer. Then five later papers that changed the loop, each read from its own PDF, and a small run on a laptop that shows how a search score can look better than it is.

  • Part 6: Build It from Scratch: A Small Meta-Harness in Python

    We build the whole Meta-Harness loop in about a thousand lines of Python: a task, harness files, an evaluator that logs every prompt, a folder per candidate, and two proposers. Then we run it on a laptop with small local models and measure the gap between the search set and the test set.