Meta-Harness, explained

The 2026 paper that lets a coding agent write the code around a language model, by reading every earlier attempt's code, scores and traces. Six parts, with the paper's own text, worked examples, and a small Meta-Harness loop built from scratch in Python.

Meta-Harness: End-to-End Optimization of Model Harnesses. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn. COLM 2026, 2026. arXiv:2603.28052

What you will learn

  • What a harness is: the code that decides what a language model stores, retrieves and sees
  • Why the same model can score six times higher or lower with a different harness
  • Why earlier prompt optimizers fail here: they forget, see only a score, or squeeze feedback into a summary
  • The Meta-Harness loop: a proposer agent reads a filesystem of every earlier harness, its traces and its score, and writes the next one
  • What it found on text classification, math retrieval and TerminalBench-2, and what the ablations show
  • Where it can overfit, what it costs, and what later work changed
  • How to build a small harness-search loop yourself, with a scoring set and a held-out test set

Why this paper

A language model never works alone. Code around it decides what to remember, what to look up and what to show the model before each call. That code is the harness. Change the harness, and the same model can do much better or much worse on the same task.

Harnesses are mostly written by hand: a person reads failures, guesses a fix, and tries again. This paper asks whether an agent can do that job. Its system, Meta-Harness, gives a coding agent a folder with every earlier harness, the full record of how each one ran, and its score. The agent reads what it needs and writes a better harness. The paper tests this on three very different tasks and reports gains over hand-written harnesses on each.

How the six parts are organised

Before Part 1 (optional): What a Harness Is. The model, the loop around it, and the earlier ways people tried to optimise prompts and programs.

  1. The Big Idea: the title, Figure 1, the abstract and §1.
  2. The Method: §2, Table 1, §3, Figure 2 and Algorithm 1. The objective, the filesystem of experience, and the proposer.
  3. Text Classification: §4.1, Tables 2 to 5, Figures 3 and 4, and the discovered harnesses of the appendix.
  4. Math Retrieval and TerminalBench-2: §4.2 and §4.3, Tables 6 and 7, and the discovered harnesses.
  5. Limits, and What Came After: §5, the proposer's behaviour in Appendix A, and the later work on self-improving harnesses.
  6. Build It from Scratch: a small Meta-Harness loop in Python, run on a laptop.

How to read the boxes

  • A teal "From the paper" box shows the exact lines of the paper as a highlighted screenshot. Click the screenshot to see it at full size. Under it, What it says gives the main words of the paper. What it means then explains those lines, and what the authors mean but do not write. When the box shows a different paper, the box gives the name of that paper.
  • A yellow definition box explains a technical word the first time it occurs. If you know the word, skip the box.
  • The small Paper §3 tag above a heading tells you which section of the paper you read.
  • A green "Key takeaways" box ends each part with the facts to remember.

Code blocks show real code, and the output under a code block is what that code printed. The scripts are in code/papers/meta-harness. They run on a laptop with small open models; the paper used large commercial models, and each part says what that changes.

What you need to know first

You need to know what a language model is and how a prompt works. Some Python helps for Part 6. The Transformer series explains the model inside, but this paper treats the model as a black box.

Parts

  • Background: What a Harness Is

    The code around a language model decides what it remembers, looks up and sees. This page shows that code, the agent loop, and the earlier methods that tried to improve prompts and programs automatically. With a small model on a real banking task.

  • Part 1: The Big Idea: Let an Agent Write the Harness

    Why the code around a language model can matter as much as the model, and the paper's proposal: a coding agent that reads every earlier attempt and writes a better one. We read the title, Figure 1, the abstract and §1, and we run one small model with four different harnesses.

  • Part 2: The Method: A Loop That Reads Every Earlier Attempt

    How Meta-Harness searches for better harness code: the goal written as one formula, the loop of Figure 2 and Algorithm 1, and the folder of past attempts that the proposer reads. We run the whole loop in Python with a small local model and a scripted proposer.

  • Part 3: Text Classification: What the Search Found

    The first test of Meta-Harness: a model sorts short texts into classes, and the search writes the code that picks its examples. We read the tables, the accuracy and context trade-off, and the two discovered harnesses, then rebuild both around a small local model.

  • Part 4: Math Retrieval and TerminalBench-2

    How Meta-Harness found a way to look up solved problems for olympiad math, and a better harness for an agent that works in a terminal. We build the search engine and the discovered retriever in Python, test them with a small local model, and run two key pieces of the released agent code.

  • Part 5: Limits, and What Came After

    Where Meta-Harness can fool itself, what a search costs, and why it rests on one strong proposer. Then five later papers that changed the loop, each read from its own PDF, and a small run on a laptop that shows how a search score can look better than it is.

  • Part 6: Build It from Scratch: A Small Meta-Harness in Python

    We build the whole Meta-Harness loop in about a thousand lines of Python: a task, harness files, an evaluator that logs every prompt, a folder per candidate, and two proposers. Then we run it on a laptop with small local models and measure the gap between the search set and the test set.