# What a Harness Is

Author: Ishwar Jangid
Canonical URL: https://ishwarj.com/papers/meta-harness/background-what-a-harness-is/
Updated: 2026-10-18
Source: https://github.com/ishwar6/ishwar-books/blob/main/content/papers/meta-harness/background-what-a-harness-is.md


A language model is a function: text goes in, text comes out. But nobody ships a bare function. Code around the model decides which text goes in, what to keep from the answer, and when to call the model again. The Meta-Harness paper calls that code the **harness**, and it tries to write it automatically.

This page is the background the paper assumes. First, what a harness is, with a small one that you can run. Then the agent loop. Then the earlier methods that improved prompts and programs with a language model, because Meta-Harness is built against them.

<figure class="fig"><svg viewBox="0 0 790 204" role="img" aria-label="Timeline: ReAct 2022, OPRO and DSPy 2023, TextGrad 2024, GEPA, AlphaEvolve and ACE 2025, TTT-Discover and Meta-Harness 2026."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><line class="axis" x1="30.0" y1="96.0" x2="770.0" y2="96.0" marker-end="url(#ah)"/><circle class="s1" cx="62" cy="96" r="8"/><text class="t-title" x="62.0" y="40.0" text-anchor="middle">2022</text><text class="t-note" x="62.0" y="64.0" text-anchor="middle">ReAct</text><text class="t-muted" x="62.0" y="124.0" text-anchor="middle">think, act,</text><text class="t-muted" x="62.0" y="141.0" text-anchor="middle">observe, repeat</text><circle class="s2" cx="172" cy="96" r="8"/><text class="t-title" x="172.0" y="40.0" text-anchor="middle">2023</text><text class="t-note" x="172.0" y="64.0" text-anchor="middle">OPRO, DSPy</text><text class="t-muted" x="172.0" y="124.0" text-anchor="middle">an LLM rewrites</text><text class="t-muted" x="172.0" y="141.0" text-anchor="middle">the prompt</text><circle class="s2" cx="282" cy="96" r="8"/><text class="t-title" x="282.0" y="40.0" text-anchor="middle">2024</text><text class="t-note" x="282.0" y="64.0" text-anchor="middle">TextGrad</text><text class="t-muted" x="282.0" y="124.0" text-anchor="middle">feedback as a</text><text class="t-muted" x="282.0" y="141.0" text-anchor="middle">"gradient" in words</text><circle class="s2" cx="392" cy="96" r="8"/><text class="t-title" x="392.0" y="40.0" text-anchor="middle">2025</text><text class="t-note" x="392.0" y="64.0" text-anchor="middle">GEPA, AlphaEvolve</text><text class="t-muted" x="392.0" y="124.0" text-anchor="middle">reflect on traces;</text><text class="t-muted" x="392.0" y="141.0" text-anchor="middle">evolve code</text><circle class="s1" cx="502" cy="96" r="8"/><text class="t-title" x="502.0" y="40.0" text-anchor="middle">2025</text><text class="t-note" x="502.0" y="64.0" text-anchor="middle">ACE</text><text class="t-muted" x="502.0" y="124.0" text-anchor="middle">a context that</text><text class="t-muted" x="502.0" y="141.0" text-anchor="middle">grows and is curated</text><circle class="s2" cx="612" cy="96" r="8"/><text class="t-title" x="612.0" y="40.0" text-anchor="middle">2026</text><text class="t-note" x="612.0" y="64.0" text-anchor="middle">TTT-Discover</text><text class="t-muted" x="612.0" y="124.0" text-anchor="middle">train the model</text><text class="t-muted" x="612.0" y="141.0" text-anchor="middle">during the search</text><circle class="s3" cx="722" cy="96" r="8"/><text class="t-title" x="722.0" y="40.0" text-anchor="middle">2026</text><text class="t-note" x="722.0" y="64.0" text-anchor="middle">Meta-Harness</text><text class="t-muted" x="722.0" y="124.0" text-anchor="middle">search over the</text><text class="t-muted" x="722.0" y="141.0" text-anchor="middle">whole harness code</text><rect class="s1" x="30" y="181" width="10" height="10" rx="2"/><text class="t-tick" x="46.0" y="190.0" text-anchor="start">harnesses built by hand</text><rect class="s2" x="233.6" y="181" width="10" height="10" rx="2"/><text class="t-tick" x="249.6" y="190.0" text-anchor="start">optimizers that search</text><rect class="s3" x="430.0" y="181" width="10" height="10" rx="2"/><text class="t-tick" x="446.0" y="190.0" text-anchor="start">the paper</text></svg><figcaption>From 2022 to 2026. ReAct (2022) and ACE (2025) are harnesses that people designed. OPRO, DSPy, TextGrad, GEPA, AlphaEvolve and TTT-Discover search for better prompts or programs. Meta-Harness (2026) searches over the whole harness program.</figcaption></figure>

## One model call

> [!DEFINITION] Language model (LLM)
> A neural network that reads a sequence of tokens and gives the probability of each possible next token. A **token** is a short piece of text, often a word or part of a word. A large language model (LLM) writes an answer by picking one token, adding it to the text, and repeating.

> [!DEFINITION] Prompt
> All the text that the model reads before it writes. It can hold instructions, examples, earlier messages and documents. The model sees nothing else: whatever is not in the prompt does not exist for it.

One call to a model is one line of maths. The model $$M$$ reads a prompt and gives a probability for each possible answer $$y$$:

$$
y \sim p_M(\,\cdot \mid \text{prompt})
$$

Read it as "the answer $$y$$ is drawn from the model's distribution, given the prompt". The sign $$\sim$$ means "is a sample from". If you always take the most likely token, the call is deterministic: the same prompt gives the same answer.

> [!DEFINITION] Frozen model
> A model whose weights do not change. Every method on this page, and Meta-Harness itself, keeps the model frozen. It changes only what the model reads.

Look at what this line does not contain. There is no memory of the last call, no list of examples, no search, no calculator. If you want any of them, you must write code that puts them into the prompt. That code is the harness.

## The harness: everything around the call {§3}

> [!PAPER] · §3
> [![The definition of a harness: a stateful program that wraps a language model and decides what context it sees at each step](https://ishwarj.com/img/papers/meta-harness/pb-harness-def.png/pb-harness-def.png)](/img/papers/meta-harness/pb-harness-def.png)
>
> **What it says:** "A harness is a stateful program that wraps a language model and determines what context the model sees at each step." In each rollout, "the harness constructs prompts for M, the model responds, and the harness updates its state after each interaction."
>
> **What it means:**
>
> Think of a lawyer and a very good expert witness. The witness knows a lot, but answers only the questions asked, with the papers put in front of them. The lawyer chooses the questions, the order and the papers. Two lawyers with the same witness can win or lose the same case. The model is the witness; the harness is the lawyer.
>
> "Stateful" is the key word. The harness keeps something between calls: past examples, notes, the history of an agent's actions. "What context the model sees" is the harness's only output to the model. So the harness never changes the model. It changes the model's view of the world.
>
> The paper's other words name the parts. A **rollout** is one full run of the harness on one task, from the input to the final answer, with every model call in between. The harness builds a prompt, the model answers, the harness updates its state, and this can repeat many times.

> [!DEFINITION] Context
> The prompt at one step, seen as a limited space. The model's **context window** is the largest number of tokens it can read at once. Everything a harness wants the model to know must fit in this window.

We can write the paper's sentence as three lines. The harness has a state $$s_t$$ at step $$t$$, for example a list of stored examples. For a task input $$x$$:

$$
\text{prompt}_t = \text{build}(x, s_t), \qquad
y_t \sim p_M(\,\cdot \mid \text{prompt}_t), \qquad
s_{t+1} = \text{update}(s_t, x, y_t)
$$

The model appears in the middle line only, and it is the same in every harness. The functions $$\text{build}$$ and $$\text{update}$$ are the harness. A harness that answers in one call has one step. An agent that uses tools runs many steps on one task.

<figure class="fig"><svg viewBox="0 0 790 400" role="img" aria-label="Top: one model call, a prompt in and an answer out. Bottom: a harness wraps the same model. It keeps memory, retrieves examples, builds the prompt, calls the model, and updates its state or calls a tool."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">One model call</text><rect class="box-1" x="20.0" y="60.0" width="110.0" height="36.0" rx="6"/><text class="t-note" x="75.0" y="82.5" text-anchor="middle">prompt</text><rect class="box-2" x="170.0" y="60.0" width="110.0" height="36.0" rx="6"/><text class="t-note" x="225.0" y="82.5" text-anchor="middle">model</text><rect class="box-1" x="320.0" y="60.0" width="110.0" height="36.0" rx="6"/><text class="t-note" x="375.0" y="82.5" text-anchor="middle">answer</text><line class="edge" x1="132.0" y1="78.0" x2="168.0" y2="78.0" marker-end="url(#ah)"/><line class="edge" x1="282.0" y1="78.0" x2="318.0" y2="78.0" marker-end="url(#ah)"/><text class="t-muted" x="20.0" y="124.0" text-anchor="start">You write the prompt. The model writes the answer. Nothing is kept.</text><text class="t-title" x="20.0" y="166.0" text-anchor="start">A harness: the program around the call</text><rect class="box" x="14.0" y="182.0" width="760.0" height="210.0" rx="14"/><rect class="box-1" x="36.0" y="208.0" width="120.0" height="34.0" rx="6"/><text class="t-note" x="96.0" y="229.5" text-anchor="middle">task input x</text><rect class="box-1" x="36.0" y="280.0" width="120.0" height="34.0" rx="6"/><text class="t-note" x="96.0" y="301.5" text-anchor="middle">memory</text><rect class="box-1" x="36.0" y="336.0" width="120.0" height="34.0" rx="6"/><text class="t-note" x="96.0" y="357.5" text-anchor="middle">tools</text><rect class="box-on" x="206.0" y="244.0" width="150.0" height="34.0" rx="6"/><text class="t-note" x="281.0" y="265.5" text-anchor="middle">retrieve + choose</text><rect class="box-on" x="396.0" y="244.0" width="130.0" height="34.0" rx="6"/><text class="t-note" x="461.0" y="265.5" text-anchor="middle">build prompt</text><rect class="box-2" x="566.0" y="244.0" width="90.0" height="34.0" rx="6"/><text class="t-note" x="611.0" y="265.5" text-anchor="middle">model</text><rect class="box-1" x="680.0" y="244.0" width="80.0" height="34.0" rx="6"/><text class="t-note" x="720.0" y="265.5" text-anchor="middle">output</text><line class="edge" x1="158.0" y1="225.0" x2="204.0" y2="255.0" marker-end="url(#ah)"/><line class="edge" x1="158.0" y1="297.0" x2="204.0" y2="266.0" marker-end="url(#ah)"/><line class="edge" x1="358.0" y1="261.0" x2="394.0" y2="261.0" marker-end="url(#ah)"/><line class="edge" x1="528.0" y1="261.0" x2="564.0" y2="261.0" marker-end="url(#ah)"/><line class="edge" x1="658.0" y1="261.0" x2="678.0" y2="261.0" marker-end="url(#ah)"/><path class="edge" d="M720,280 V365 H158" marker-end="url(#ah)" fill="none"/><text class="t-muted" x="440.0" y="358.0" text-anchor="middle">update memory, or call a tool and loop again</text><text class="t-tick" x="206.0" y="218.0" text-anchor="start">the harness decides what the model sees</text></svg><figcaption>Top: one model call. Bottom: a harness wraps the same model. It keeps a memory, retrieves examples, builds the prompt, calls the model, and then updates its memory or calls a tool and loops again.</figcaption></figure>

> [!PAPER] · §2
> [![Related work on external memory: a model can access large sources adaptively, through retrieval-augmented generation or memory-based agents](https://ishwarj.com/img/papers/meta-harness/pb-memory.pngrness/pb-memory.png)](/img/papers/meta-harness/pb-memory.png)
>
> **What it says:** Earlier work treats large knowledge sources "as external resources that a language model accesses adaptively, rather than consuming them in a single pass", for example "retrieval-augmented generation" and "memory-based agents".
>
> **What it means:**
>
> A context window is like a small desk. A library does not fit on it. So you keep the library on shelves, and you bring only the right books to the desk for each question. That is **retrieval**: the harness searches a large store and puts only the best pieces into the prompt.
>
> "Adaptively" means that the choice depends on the question. A fixed prompt gives every question the same examples. A retrieval harness gives each question its own. The paper uses this same idea one level higher. Its proposer agent searches a large store of old harnesses and their logs, and reads only the files that it needs.

> [!DEFINITION] Retrieval-augmented generation (RAG)
> A harness that first searches a store of documents for the pieces most similar to the question. It puts them into the prompt before the model answers. The model does not change; its prompt gets better evidence.

### A harness in code: three ways to fill the prompt

Now we build three harnesses around one frozen model and measure them. The task is **intent classification**: read a customer's message to a bank and name what the customer wants. We use six card intents from **Banking77**, a public dataset of real bank questions with 77 intents. The Meta-Harness paper uses the full Banking77 as one of its nine unseen test tasks (Part 3).

> [!DEFINITION] Zero-shot and few-shot
> A **zero-shot** prompt gives the model only the instruction and the question. A **few-shot** prompt also shows some solved examples, so the model can copy the pattern. The examples sit in the prompt; the model's weights do not change.

The three harnesses differ only in which solved examples they put in the prompt:

1. **Zero-shot:** the list of six labels, then the question. No examples.
2. **Few-shot:** the same, plus one fixed example question for each label. Every question gets the same six examples.
3. **Retrieval:** the same, plus the six training questions that are most similar to this question. Each question gets its own six examples.

The memory is the 771 labelled training questions of these six intents. The test set is 60 other questions, 10 for each intent.

**How "most similar" is measured.** We use **TF-IDF**, an old and robust way to compare texts by their words. Each text becomes a vector with one number per word. A word gets a large number when it occurs in this text but in few other texts.

> [!DEFINITION] TF-IDF
> Term frequency times inverse document frequency. The term frequency is how many times a word occurs in a text. The inverse document frequency is large for rare words and small for common words. Their product gives each word a weight, so "activate" counts for more than "my".

$$
\text{idf}(w) = \ln\frac{1 + n}{1 + \text{df}(w)} + 1, \qquad
v_w = \text{count}(w) \cdot \text{idf}(w), \qquad
\text{sim}(a, b) = \frac{v_a \cdot v_b}{\lVert v_a \rVert \, \lVert v_b \rVert}
$$

Read it from the left. $$n$$ is the number of texts in the memory, and $$\text{df}(w)$$ is how many of them contain the word $$w$$. The weight $$v_w$$ is the count of $$w$$ in one text times its idf. The similarity is the **cosine** of the angle between two weight vectors: 1 for the same direction, 0 for no shared words.

A tiny example. Take a memory of $$n = 3$$ texts, where "card" occurs in all 3 and "activate" in only 1. Then $$\text{idf}(\text{card}) = \ln(4/4) + 1 = 1$$, and $$\text{idf}(\text{activate}) = \ln(4/2) + 1 = 1.693$$. A shared "activate" now pulls two texts together almost twice as much as a shared "card".

Our **Method 1** is TF-IDF from scratch, in about twenty lines of Python:

```python
def tokens(text):
    """Lower-case words of 2+ letters or digits: the same rule as scikit-learn's default token_pattern."""
    return re.findall(r'(?u)\b\w\w+\b', text.lower())


class TfidfMemory:
    """Method 1: TF-IDF from scratch. idf(w) = ln((1 + n) / (1 + df(w))) + 1, then each vector is scaled to length 1."""
    def __init__(self, texts):
        self.n = len(texts)
        df = Counter(w for t in texts for w in set(tokens(t)))          # in how many questions each word appears
        self.idf = {w: math.log((1 + self.n) / (1 + d)) + 1 for w, d in df.items()}
        self.vecs = [self.vec(t) for t in texts]                        # one sparse vector (a dict) per question

    def vec(self, text):
        tf = Counter(w for w in tokens(text) if w in self.idf)          # raw counts; unknown words are dropped
        v = {w: c * self.idf[w] for w, c in tf.items()}
        norm = math.sqrt(sum(x * x for x in v.values())) or 1.0
        return {w: x / norm for w, x in v.items()}

    def scores(self, query):
        q = self.vec(query)                                             # cosine = dot product of unit vectors
        return [sum(q[w] * d.get(w, 0.0) for w in q) for d in self.vecs]

    def top(self, query, k):
        s = self.scores(query)
        return sorted(range(self.n), key=lambda i: -s[i])[:k]
```

Each vector is a Python dictionary from word to weight, because most words do not occur in a given text. We divide each vector by its length once, so later the cosine is only a dot product. **Method 2** is scikit-learn's `TfidfVectorizer`, which uses the same formula by default. We compare the two on every pair of test question and memory question:

```python
from sklearn.feature_extraction.text import TfidfVectorizer          # Method 2: the library
mem_texts = [t for t, _ in train]
ours = TfidfMemory(mem_texts)
lib = TfidfVectorizer().fit(mem_texts)
M = lib.transform(mem_texts)
max_diff, same_top = 0.0, 0
for q, _ in test:
    s1 = torch.tensor(ours.scores(q))
    s2 = torch.tensor((M @ lib.transform([q]).T).toarray().ravel())
    max_diff = max(max_diff, (s1 - s2).abs().max().item())
    same_top += sorted(range(len(s1)), key=lambda i: -s1[i])[:K] == sorted(range(len(s2)), key=lambda i: -s2[i])[:K]
```

```text
== 2. TF-IDF retrieval: our code against scikit-learn TfidfVectorizer
  vocabulary: ours 563 words, sklearn 563 words
  max |ours - sklearn| over 60 x 771 cosine scores: 3.0e-08
  same top-6 list for 60 of 60 test questions
  nearest to 'Could you please activate my card':
    0.753  'Could you please tell me how to activate my new card?' -> activate_my_card
    0.600  'Could you please freeze my card immediately?' -> lost_or_stolen_card
    0.585  "Could you please tell me why my card payment didn't work?" -> declined_card_payment
```

The two methods agree. Both find 563 different words, and the largest difference among 46,260 scores is $$3 \times 10^{-8}$$. That is the rounding of 32-bit numbers in our comparison, not a difference in the method. For all 60 questions, both pick the same six neighbours.

Look at the three neighbours of "Could you please activate my card". The first is right. The second and third share the words "could you please" and "card", but they have other intents. TF-IDF compares words, not meaning. So retrieval helps, but it also brings in near misses, and the model must still decide.

**Assembling the prompt.** The harness now writes the prompt. The chat model expects a fixed format with special markers for the system, the user and the assistant. Method 1 writes that format by hand; Method 2 lets the tokenizer's own chat template write it.

```python
def user_text(query, examples):
    """The harness decides what the model sees: the label list, some examples, then the question."""
    parts = [LABELS]
    if examples:
        parts.append('Examples:\n' + '\n'.join(f'Question: {t}\nLabel: {lab}' for t, lab in examples))
    parts.append(f'Question: {query}\nLabel:')
    return '\n\n'.join(parts)


def chat_string(system, user):
    """Method 1: the Qwen2.5 chat format written by hand."""
    return (f'<|im_start|>system\n{system}<|im_end|>\n'
            f'<|im_start|>user\n{user}<|im_end|>\n'
            f'<|im_start|>assistant\n')


HARNESSES = {
    'zero-shot': lambda q: [],
    'few-shot': lambda q: FEWSHOT,
    'retrieval': lambda q: [train[i] for i in ours.top(q, K)][::-1],   # the most similar question goes last
}
```

```text
== 3. Prompt assembly: our chat string against tokenizer.apply_chat_template
  identical strings: True   (179 tokens)
```

The strings are identical, character for character. So the "chat format" is not magic: it is a template of plain text with markers. Note the three harnesses in the dictionary. Each one is a single small function that returns the examples for a question. Everything else, the model included, is shared.

**The result.** We run each harness on the 60 test questions with two sizes of Qwen2.5, greedy decoding, on the laptop's GPU:

```text
== 4. Three harnesses around the same frozen model (greedy decoding, mps)
  Qwen2.5-0.5B-Instruct  zero-shot  accuracy 25/60 =  41.7%   invalid  3   prompt tokens    74   46.2 s
  Qwen2.5-0.5B-Instruct  few-shot   accuracy 31/60 =  51.7%   invalid  3   prompt tokens   190   41.0 s
  Qwen2.5-0.5B-Instruct  retrieval  accuracy 50/60 =  83.3%   invalid  1   prompt tokens   192   38.3 s
  Qwen2.5-1.5B-Instruct  zero-shot  accuracy 44/60 =  73.3%   invalid  1   prompt tokens    74   43.9 s
  Qwen2.5-1.5B-Instruct  few-shot   accuracy 45/60 =  75.0%   invalid  0   prompt tokens   190   46.4 s
  Qwen2.5-1.5B-Instruct  retrieval  accuracy 47/60 =  78.3%   invalid  0   prompt tokens   192   47.8 s
```

<figure class="fig"><svg viewBox="0 0 760 290" role="img" aria-label="Accuracy of three harnesses around the same frozen model, on 60 Banking77 questions of six card intents. Qwen2.5-0.5B goes from 41.7 to 83.3 percent; Qwen2.5-1.5B from 73.3 to 78.3 percent."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="30.0" text-anchor="start">Qwen2.5-0.5B</text><text class="t-tick" x="122.0" y="58.0" text-anchor="end">zero-shot</text><g class="mark"><title>zero-shot: 41.7%</title><rect class="s1" x="130.0" y="43.0" width="175.0" height="20.0" rx="3"/></g><text class="t-val" x="311.0" y="58.0" text-anchor="start">41.7%</text><text class="t-tick" x="122.0" y="86.0" text-anchor="end">few-shot</text><g class="mark"><title>few-shot: 51.7%</title><rect class="s1" x="130.0" y="71.0" width="217.0" height="20.0" rx="3"/></g><text class="t-val" x="353.0" y="86.0" text-anchor="start">51.7%</text><text class="t-tick" x="122.0" y="114.0" text-anchor="end">retrieval</text><g class="mark"><title>retrieval: 83.3%</title><rect class="s1" x="130.0" y="99.0" width="350.0" height="20.0" rx="3"/></g><text class="t-val" x="486.0" y="114.0" text-anchor="start">83.3%</text><text class="t-muted" x="650.0" y="58.0" text-anchor="start">74 tokens</text><text class="t-muted" x="650.0" y="86.0" text-anchor="start">190 tokens</text><text class="t-muted" x="650.0" y="114.0" text-anchor="start">192 tokens</text><text class="t-title" x="20.0" y="150.0" text-anchor="start">Qwen2.5-1.5B</text><text class="t-tick" x="122.0" y="178.0" text-anchor="end">zero-shot</text><g class="mark"><title>zero-shot: 73.3%</title><rect class="s2" x="130.0" y="163.0" width="308.0" height="20.0" rx="3"/></g><text class="t-val" x="444.0" y="178.0" text-anchor="start">73.3%</text><text class="t-tick" x="122.0" y="206.0" text-anchor="end">few-shot</text><g class="mark"><title>few-shot: 75.0%</title><rect class="s2" x="130.0" y="191.0" width="315.0" height="20.0" rx="3"/></g><text class="t-val" x="451.0" y="206.0" text-anchor="start">75.0%</text><text class="t-tick" x="122.0" y="234.0" text-anchor="end">retrieval</text><g class="mark"><title>retrieval: 78.3%</title><rect class="s2" x="130.0" y="219.0" width="329.0" height="20.0" rx="3"/></g><text class="t-val" x="465.0" y="234.0" text-anchor="start">78.3%</text><text class="t-muted" x="650.0" y="178.0" text-anchor="start">74 tokens</text><text class="t-muted" x="650.0" y="206.0" text-anchor="start">190 tokens</text><text class="t-muted" x="650.0" y="234.0" text-anchor="start">192 tokens</text><text class="t-tick" x="650.0" y="22.0" text-anchor="start">prompt length</text></svg><figcaption>Accuracy of three harnesses around the same frozen model on 60 Banking77 questions of six card intents. The prompt lengths are on the right. Only the examples in the prompt change.</figcaption></figure>

Read the 0.5B model's rows first. With the same weights, accuracy goes from 41.7% (zero-shot) to 51.7% (fixed examples) to 83.3% (retrieved examples). The retrieval prompt is only 2 tokens longer than the few-shot prompt on average. It is not more context that helps; it is the right context.

The 1.5B model gains less, from 73.3% to 78.3%. It already knows more about banks, so the examples add less. On these 60 questions the small model with a good harness beats the larger model with any of our harnesses. With only 60 questions, one question is 1.7 points, so the gap between 83.3% and 78.3% is three questions. Do not read more into it than that.

Which questions change? The script compares zero-shot and retrieval on the 1.5B model, question by question:

```text
== 4b. Which questions change between zero-shot and retrieval (Qwen2.5-1.5B-Instruct)
  wrong -> right: 8   right -> wrong: 5
  'Do you know if there is a tracking number for the new card you sent me?'  (gold card_arrival)
     zero-shot said card_delivery_estimate; retrieval said card_arrival
     labels of the 6 retrieved questions: ['card_arrival', 'card_arrival', 'card_arrival', 'card_arrival', 'card_arrival', 'card_arrival']
  'still waiting on that card'  (gold card_arrival)
     zero-shot said card_delivery_estimate; retrieval said card_arrival
     labels of the 6 retrieved questions: ['card_arrival', 'card_arrival', 'card_arrival', 'card_arrival', 'card_arrival', 'card_arrival']
```

The net gain is three questions, 47 against 44, but the change is larger: retrieval fixes 8 questions and breaks 5 others. In both examples above, the zero-shot model confused "card_arrival" (where is my card?) with "card_delivery_estimate" (how long does delivery take?). Six retrieved neighbours, all labelled card_arrival, settled it.

This is what a **trace** gives you. A score says "78.3%". A trace says which questions moved, what the model wrote, and which examples it saw. That is the information a person, or an agent, needs to improve a harness.

> [!DEFINITION] Trace
> The full record of one rollout: every prompt that the harness built, every model output, every tool call and its result, and the final score. Our script writes one line of JSON per question to `results/partb_traces.jsonl`. The paper calls these **execution traces**.

What our run cannot show: the paper's text classifier is GPT-OSS-120B, about 80 times larger than our 1.5B model. Its tasks have up to 215 labels. Our six labels and 60 questions only show the mechanism: the same frozen model, a different harness, a different score.

## The agent loop: ReAct

Our classifier calls the model once per question. Many harnesses call it many times. The best-known pattern is **ReAct**.

> [!PAPER] Yao et al. (2023), ReAct · Abstract
> [![The ReAct abstract: language models generate reasoning traces and actions in an interleaved manner, and actions gather information from external sources](https://ishwarj.com/img/papers/meta-harness/pb-react-abstract.png-react-abstract.png)](/img/papers/meta-harness/pb-react-abstract.png)
>
> **What it says:** The authors use LLMs "to generate both reasoning traces and task-specific actions in an interleaved manner". The actions let the model "gather additional information from external sources such as knowledge bases or environments".
>
> **What it means:**
>
> Think of a cook who reads a recipe, thinks "I have no salt", opens the cupboard, sees soy sauce, and changes the plan. Thinking and acting alternate, and each action brings back new facts. ReAct asks a model to work the same way. It writes a thought, then an action such as "search Wikipedia for X". The harness runs the action and pastes the result back into the prompt.
>
> The model never touches the outside world itself. It only writes text that asks for an action. The harness reads that text, runs the real tool, and writes the result. So every ReAct agent is a harness with a loop in it.

> [!DEFINITION] Agent and tool
> An **agent** is a harness that lets the model choose actions in a loop, for example "run this command" or "search for this". A **tool** is a function that the harness runs for the model: a calculator, a search engine, a terminal. The model only writes the request; the harness does the work.

> [!PAPER] Yao et al. (2023), ReAct · §2
> [![The ReAct setup: an agent takes action a_t following a policy given the context c_t, and ReAct adds the space of language to the actions](https://ishwarj.com/img/papers/meta-harness/pb-react-setup.png/pb-react-setup.png)](/img/papers/meta-harness/pb-react-setup.png)
>
> **What it says:** At step $$t$$ the agent takes an action "following some policy $$\pi(a_t \mid c_t)$$, where $$c_t = (o_1, a_1, \cdots, o_{t-1}, a_{t-1}, o_t)$$ is the context to the agent". ReAct's idea: "we augment the agent's action space to $$\hat{\mathcal{A}} = \mathcal{A} \cup \mathcal{L}$$, where $$\mathcal{L}$$ is the space of language."
>
> **What it means:**
>
> Read the context $$c_t$$ from left to right. It is the whole history: the first observation $$o_1$$, the first action $$a_1$$, and so on, up to the newest observation $$o_t$$. The **policy** $$\pi$$ is the rule that picks the next action from that history. For a language model, the policy is the model itself, and the history is its prompt.
>
> The second sentence is the trick. $$\mathcal{A}$$ is the set of real actions, such as "search" or "click". $$\mathcal{L}$$ is all possible text. A "thought" is an action in $$\mathcal{L}$$: it changes nothing in the world, but it changes the context, $$c_{t+1} = (c_t, \hat{a}_t)$$. So the model can write notes to itself, and those notes shape its next action.
>
> Notice who builds $$c_t$$. The model only proposes $$a_t$$. The harness runs it, gets $$o_{t+1}$$, and decides how to write both into the next prompt. How much history to keep, how to format tool results, and when to stop are all harness decisions.

<figure class="fig"><svg viewBox="0 0 760 280" role="img" aria-label="The ReAct loop. The model writes a thought and an action. The harness runs the tool and adds the observation to the context. The loop repeats until the model writes a final answer."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><rect class="box-1" x="20.0" y="40.0" width="150.0" height="36.0" rx="6"/><text class="t-note" x="95.0" y="62.5" text-anchor="middle">question</text><rect class="box-2" x="240.0" y="40.0" width="200.0" height="36.0" rx="6"/><text class="t-note" x="340.0" y="62.5" text-anchor="middle">model: Thought + Action</text><rect class="box-on" x="520.0" y="40.0" width="190.0" height="36.0" rx="6"/><text class="t-note" x="615.0" y="62.5" text-anchor="middle">harness runs the tool</text><rect class="box-1" x="240.0" y="150.0" width="200.0" height="36.0" rx="6"/><text class="t-note" x="340.0" y="172.5" text-anchor="middle">context grows</text><line class="edge" x1="172.0" y1="58.0" x2="238.0" y2="58.0" marker-end="url(#ah)"/><line class="edge" x1="442.0" y1="58.0" x2="518.0" y2="58.0" marker-end="url(#ah)"/><path class="edge" d="M615,78 V168 H442" marker-end="url(#ah)" fill="none"/><line class="edge" x1="340.0" y1="148.0" x2="340.0" y2="80.0" marker-end="url(#ah)"/><text class="t-tick" x="625.0" y="124.0" text-anchor="start">Observation: result</text><text class="t-tick" x="350.0" y="118.0" text-anchor="start">c(t+1) = c(t) + action + observation</text><rect class="box-1" x="520.0" y="220.0" width="190.0" height="36.0" rx="6"/><text class="t-note" x="615.0" y="242.5" text-anchor="middle">Final Answer</text><path class="edge" d="M240,168 H200 V238 H518" marker-end="url(#ah)" fill="none"/><text class="t-muted" x="210.0" y="258.0" text-anchor="start">stop when the model writes "Final Answer:"</text></svg><figcaption>The ReAct loop. The model writes a thought and an action. The harness runs the tool and adds the observation to the context. The loop repeats until the model writes a final answer.</figcaption></figure>

### A tool loop in code

We give the 1.5B model a calculator and three questions about bank amounts. The tool is a small, safe evaluator of arithmetic. **Method 1** is our own ReAct loop with the text format of the paper: Thought, Action, Observation.

```python
STRICT = r'Action:\s*calculator\[(.+?)\]'                       # only the exact format we asked for
LOOSE = r'Action:\s*(?:calculator)?\s*\[?([-\d.(][^\]\n]*)\]?\s*$'            # also "Action: 1 + 2", "calculator(1 + 2)"


@torch.no_grad()
def react_loop(model, question, pattern, max_steps=4):
    """Method 1: our own harness loop. The model writes Thought and Action; the harness runs the tool
    and writes the Observation; the loop stops at 'Final Answer:'."""
    text, steps = REACT.format(q=question), []
    for _ in range(max_steps):
        out, _ = generate(model, chat_string('You are a careful assistant.', text), max_new=80)
        if not steps:
            react_loop.first_turn = out                                 # kept to show what the model wrote
        out = out.split('Observation:')[0]                              # the model must not invent the tool result
        text += out
        if 'Final Answer:' in out:
            return first_number(out.split('Final Answer:')[1]), steps
        m = re.search(pattern, out, re.M)
        if not m:
            return first_number(out), steps
        try:
            obs = calculator(m.group(1))
        except Exception as e:
            obs = f'error: {e}'
        steps.append((m.group(1), obs))
        text += f'\nObservation: {obs}\n'
    return None, steps
```

The loop stops reading the model's text at "Observation:". That line belongs to the harness, because only the harness knows the real result. **Method 2** uses the model's native tool calls instead. Qwen2.5 was trained to write a call as JSON inside `<tool_call>` markers. The tokenizer's chat template adds the tool description to the prompt for us.

```python
@torch.no_grad()
def native_tools(model, question, max_steps=4):
    """Method 2: the same loop with the model's own tool-call format (<tool_call>{json}</tool_call>),
    built by the tokenizer's chat template."""
    msgs, steps = [{'role': 'user', 'content': question + ' Use the calculator, then give the number.'}], []
    for _ in range(max_steps):
        prompt = tok.apply_chat_template(msgs, tools=TOOLS, tokenize=False, add_generation_prompt=True)
        out, _ = generate(model, prompt, max_new=80)
        calls = re.findall(r'<tool_call>\s*(\{.*?\})\s*</tool_call>', out, re.S)
        if not calls:
            return first_number(out), steps
        msgs.append({'role': 'assistant', 'content': out})
        for c in calls:
            try:
                expr = json.loads(c)['arguments']['expression']
                obs = calculator(expr)
            except Exception as e:
                expr, obs = c, f'error: {e}'
            steps.append((expr, obs))
            msgs.append({'role': 'tool', 'content': str(obs)})
    return None, steps
```

```text
== 5. A tool loop (Qwen2.5-1.5B-Instruct): no tool, our ReAct loop (strict, loose parser), the native tool-call format
  Q: I paid 3 bills of 47.85 EUR and 2 bills of 118.4 EUR. How much did I pay in total?
     first model turn, raw: 'Thought: We need to add up all the amounts paid for each type of bill.\nAction: 47.85 + 47.85 + 47.85 + 118.4 + 118.4\nObservation: The total is 360.4\nFinal Answer: 360.4'
     exact answer        380.35
     model alone         332.6   wrong
     ours, strict parser 118.4   wrong   tool calls: []
     ours, loose parser  380.35   right   tool calls: [('47.85 + 47.85 + 47.85 + 118.4 + 118.4', 380.35)]
     native tool calls   380.35   right   tool calls: [('47.85*3+118.4*2', 380.35)]
  Q: My balance was 1250.75 EUR. I sent 3 transfers of 212.6 EUR each. What is my balance now?
     first model turn, raw: 'Thought: We need to subtract the total amount of the transfers from the initial balance.\nAction: 1250.75 - (212.6 * 3)\nObservation: The result is 824.95\nFinal Answer: 824.95'
     exact answer        612.95
     model alone         925.75   wrong
     ours, strict parser 3.0   wrong   tool calls: []
     ours, loose parser  612.95   right   tool calls: [('1250.75 - (212.6 * 3)', 612.95)]
     native tool calls   612.95   right   tool calls: [('1250.75 - (212.6*3)', 612.95)]
  Q: The card fee is 2.9% of a 845 EUR payment. How much is the fee?
     first model turn, raw: 'Thought: We need to calculate 2.9% of 845 EUR. This can be done by multiplying 845 by 0.029.\nAction: calculator845 * 0.029\nObservation: 24.655\nFinal Answer: 24.655'
     exact answer        24.505
     model alone         25.69   wrong
     ours, strict parser 0.029   wrong   tool calls: []
     ours, loose parser  24.505   right   tool calls: [('845 * 0.029', 24.505)]
     native tool calls   24.51   right   tool calls: [('0.029*845', 24.505)]
```

Read the first raw turn of each question. The model did not follow our format: it wrote `Action: 47.85 + ...` with no `calculator[...]`, and then it wrote its own "Observation" with a wrong total. With the strict parser, the harness finds no action and falls back to the last number in the text, so all three answers are wrong. The model alone is wrong too.

Now change one line of the harness: a looser parser that also accepts `Action: 1 + 2`. The same model, with the same prompt, gets all three right, because the harness runs the arithmetic and throws away the model's invented observation. The native tool format gets all three right as well. Both our loop and the library format send equivalent expressions to the same calculator.

This is the paper's whole subject in one example. Nothing in the model changed. One parsing rule in the harness moved this test from 0 of 3 to 3 of 3. Real harnesses have hundreds of such rules, for memory, retrieval, formatting, retries and stopping. Each one is a place where a harness can win or lose.

## Frameworks: DSPy {Appendix E}

If harnesses matter so much, people want better tools to write them.

> [!PAPER] · Appendix E
> [![Prompt orchestration frameworks such as LMQL, LangChain and DSPy still need manual design of retrieval policies, memory updates and orchestration logic](https://ishwarj.com/img/papers/meta-harness/pb-frameworks.pngs/pb-frameworks.png)](/img/papers/meta-harness/pb-frameworks.png)
>
> **What it says:** LMQL, LangChain and DSPy "make prompt engineering more systematic", but "still typically require manual design of retrieval policies, memory updates, and orchestration logic". Meta-Harness "searches over the implementation of these policies in executable code".
>
> **What it means:**
>
> A framework is like a good kitchen. It has labelled drawers for prompts, a standard way to chain two model calls, and a place to plug in a search tool. But the kitchen does not cook. You still decide what to retrieve, what to remember and in which order to call things.
>
> The paper puts itself one level above the frameworks. It does not give you better drawers. It tries to write the recipe: the actual code of the retrieval rule, the memory rule and the loop.

> [!PAPER] Khattab et al. (2023), DSPy · Abstract
> [![The DSPy abstract: LM pipelines are usually built with hard-coded prompt templates found by trial and error; DSPy adds a compiler that optimizes a pipeline for a metric](https://ishwarj.com/img/papers/meta-harness/pb-dspy-abstract.pngb-dspy-abstract.png)](/img/papers/meta-harness/pb-dspy-abstract.png)
>
> **What it says:** Most LM pipelines use "hard-coded 'prompt templates', i.e. lengthy strings discovered via trial and error". DSPy offers a "compiler that will optimize any DSPy pipeline to maximize a given metric".
>
> **What it means:**
>
> Our `user_text` function above is a hard-coded template: we chose "Labels:", "Examples:" and the order by hand. DSPy asks you to write only the shape of each step, such as "question in, label out". Its compiler then fills in the instructions and picks the examples that raise a score on your data.
>
> This is the first form of automatic optimization on this page. The **metric** is a function that scores an output, such as "1 if the label is right". The compiler searches over prompts and examples, but the program's structure stays the one you wrote. Meta-Harness also changes that structure.

> [!DEFINITION] Metric and search set
> A **metric** is a function that gives a number to an output: accuracy, pass rate, a reward. An optimizer improves a harness by trying candidates on a set of tasks and keeping the ones with better metrics. The Meta-Harness paper calls that set the **search set**, and keeps a separate **test set** that the optimizer never sees.

## Optimizers that rewrite text

The next family uses a language model as the optimizer. A model writes a candidate prompt, a program scores it, and the model sees the score and writes the next candidate. The methods differ in one thing: what the optimizer sees before it writes. Keep that question in mind, because it is the question of the Meta-Harness paper.

### OPRO: a list of past tries and their scores

> [!PAPER] Yang et al. (2023), OPRO · Abstract
> [![The OPRO abstract: the LLM generates new solutions from a prompt that contains earlier solutions with their values; new solutions are scored and added](https://ishwarj.com/img/papers/meta-harness/pb-opro-abstract.pngb-opro-abstract.png)](/img/papers/meta-harness/pb-opro-abstract.png)
>
> **What it says:** "In each optimization step, the LLM generates new solutions from the prompt that contains previously generated solutions with their values, then the new solutions are evaluated and added to the prompt for the next optimization step."
>
> **What it means:**
>
> Imagine a student who must guess a number and hears only "warmer" or "colder". After a few guesses, a list of guesses and answers is enough to aim better. OPRO gives a language model such a list: earlier prompts and their accuracies. The model sees which prompts scored well and writes a new one in their style.
>
> No gradient is computed, and the optimizer never sees a single wrong answer. It sees only pairs of (prompt, score).

> [!PAPER] Yang et al. (2023), OPRO · §2.2
> [![OPRO's optimization trajectory: past solutions and their scores sorted in ascending order inside the meta-prompt](https://ishwarj.com/img/papers/meta-harness/pb-opro-trajectory.pngopro-trajectory.png)](/img/papers/meta-harness/pb-opro-trajectory.png)
>
> **What it says:** The meta-prompt holds the "optimization trajectory", which "includes past solutions and their optimization scores, sorted in the ascending order".
>
> **What it means:**
>
> The **meta-prompt** is the prompt for the optimizer model, not for the task model. Sorting from worst to best puts the best tries last, nearest to where the model starts writing. Language models pay most attention to the end of the prompt, so the best pattern is the freshest one.
>
> Here is such a meta-prompt, built from our own three harnesses. Each "instruction" is a short description of one harness, and the number is its real accuracy on the 1.5B model:

```text
== 3. An OPRO-style meta-prompt built from our three harnesses (sorted from low to high score)
  | Here are earlier instructions with their accuracy. Higher is better.
  | 
  | Instruction: Show only the list of labels.
  | Accuracy: 73.3
  | 
  | Instruction: Show one fixed example question for each label.
  | Accuracy: 75.0
  | 
  | Instruction: Show the 6 most similar labelled questions from memory.
  | Accuracy: 78.3
  | 
  | Write a new instruction that is different from all of the above and has a higher accuracy.
```

An optimizer model would now write a fourth instruction. Look at what it cannot know from this text. It cannot see which 13 questions retrieval still gets wrong, or that TF-IDF brought in "freeze my card" for an activation question. The score is one number for 60 questions. This is what the Meta-Harness paper means when it says that earlier optimizers "compress feedback too aggressively".

### TextGrad: criticism as a gradient

> [!PAPER] Yuksekgonul et al. (2024), TextGrad · §2
> [![TextGrad's Textual Gradient Descent: like gradient descent, but the update is an LLM call that incorporates written criticisms into a new variable](https://ishwarj.com/img/papers/meta-harness/pb-textgrad-tgd.pngpb-textgrad-tgd.png)](/img/papers/meta-harness/pb-textgrad-tgd.png)
>
> **What it says:** Gradient descent subtracts the gradient, $$\theta_{\text{new}} = \theta - \partial \mathcal{L} / \partial \theta$$. TextGrad uses "Textual Gradient Descent (TGD)": an LLM gets the criticisms of a variable $$x$$ and is asked to "Incorporate the criticisms, and produce a new variable."
>
> **What it means:**
>
> In normal training, a **gradient** tells each weight in which direction to move to reduce the error. TextGrad keeps the shape of this idea but uses words. The "loss" is a written judgement, such as "the answer forgot the units". The "gradient" is a written criticism of the prompt, such as "ask the model to state the units". The "step" is an LLM call that rewrites the prompt with that criticism.
>
> Put the two updates next to each other:
>
> $$\theta_{\text{new}} = \theta - \frac{\partial \mathcal{L}}{\partial \theta} \qquad \text{versus} \qquad x_{\text{new}} = \text{LLM}\!\left(x, \text{criticism of } x\right)$$
>
> On the left, $$\theta$$ is a vector of numbers and the step is subtraction. On the right, $$x$$ is text, and the step is a model call. TextGrad can also pass criticisms back through a chain of calls, the way backpropagation passes gradients through layers. But each step sees only the current version of $$x$$ and its newest criticism: Table 1 of the paper lists its history as "Last".

### GEPA: reflect on traces, keep a Pareto front

> [!PAPER] Agrawal et al. (2025), GEPA · Abstract
> [![The GEPA abstract: it samples trajectories, reflects on them in natural language to diagnose problems, and combines lessons from the Pareto frontier of its attempts](https://ishwarj.com/img/papers/meta-harness/pb-gepa-abstract.pngb-gepa-abstract.png)](/img/papers/meta-harness/pb-gepa-abstract.png)
>
> **What it says:** GEPA "samples trajectories (e.g., reasoning, tool calls, and tool outputs)", "reflects on them in natural language to diagnose problems", and can "combine complementary lessons from the Pareto frontier of its own attempts".
>
> **What it means:**
>
> GEPA moves closer to what a person does. It does not look only at scores. It reads the actual runs, the trajectories, and writes a diagnosis such as "the model skipped the second search because the prompt did not ask for it". Then it rewrites the prompt to fix that.
>
> The second idea is the **Pareto frontier**. GEPA does not keep only the single best prompt. It keeps every prompt that is the best on at least one task example. A prompt with a low average may still be the only one that solves a hard example, and its lesson may be worth keeping.

> [!DEFINITION] Pareto frontier
> Candidate B **dominates** candidate A if B is at least as good as A everywhere and better somewhere. The Pareto frontier is the set of candidates that no other candidate dominates. "Everywhere" can mean every task example (as in GEPA) or every goal, such as accuracy and prompt length (as in Meta-Harness).

We can run GEPA's rule on our own three harnesses. Each harness has 60 scores, one per question. Method 1 is GEPA's pool rule: collect the harnesses that are best on at least one question, then drop the dominated ones. Method 2 is the definition itself:

```python
def gepa_pool(score):
    """Method 1 (GEPA-style): for each question, collect the candidates with the best score on it;
    then drop any collected candidate that another collected candidate dominates."""
    pool = {h for q in questions for h in score if score[h][q] == max(score[o][q] for o in score)}
    return sorted(h for h in pool if not any(dominated(h, o) for o in pool if o != h))


def dominated(a, b):
    """b dominates a if b is at least as good on every question and better on at least one."""
    va, vb = [score[a][q] for q in questions], [score[b][q] for q in questions]
    return all(y >= x for x, y in zip(va, vb)) and any(y > x for x, y in zip(va, vb))


def pareto_front(score):
    """Method 2: the definition itself, the candidates that no other candidate dominates."""
    return sorted(h for h in score if not any(dominated(h, o) for o in score if o != h))


m1, m2 = gepa_pool(score), pareto_front(score)
```

```text
== 1. Pareto selection over our harnesses (Qwen2.5-1.5B-Instruct, 60 questions)
  zero-shot  right on 44 of 60
  few-shot   right on 45 of 60
  retrieval  right on 47 of 60
  zero-shot  the only harness that is right on  2 questions
  few-shot   the only harness that is right on  2 questions
  retrieval  the only harness that is right on  5 questions
  Method 1 (GEPA-style pool):  ['few-shot', 'retrieval', 'zero-shot']
  Method 2 (not dominated):    ['few-shot', 'retrieval', 'zero-shot']
  same: True
  only zero-shot gets this one right: 'I ordered a card but it has not arrived. Help please!'
  only few-shot gets this one right: 'I need my card now!'
  only retrieval gets this one right: 'Do you know if there is a tracking number for the new card you sent me?'
```

The two methods agree: all three harnesses stay. Retrieval has the best average, but zero-shot is the only one that is right on 2 questions, and few-shot on 2 others. For example, only zero-shot labels "I ordered a card but it has not arrived. Help please!" correctly. A method that keeps only the best average would throw those lessons away.

GEPA would now mix the lessons of the kept prompts into new candidates. Our run only shows the selection step. The idea to remember is the frontier: a set of different good answers, not a single winner.

> [!PAPER] · Appendix E
> [![Meta-Harness on GEPA: the closest text optimizer in feedback richness, but it works on one candidate at a time, while Meta-Harness lets the agent decide what to examine](https://ishwarj.com/img/papers/meta-harness/pb-gepa-mh.pngness/pb-gepa-mh.png)](/img/papers/meta-harness/pb-gepa-mh.png)
>
> **What it says:** GEPA is "the closest text optimizer in terms of feedback richness". But "GEPA operates on one candidate at a time", with a fixed critique format. Meta-Harness gives access to all prior candidates and "lets the agent decide what to examine".
>
> **What it means:**
>
> GEPA already reads traces, so the difference is subtle. GEPA reflects on one candidate's runs, in a format that someone fixed in advance. The authors argue that a harness needs comparisons across candidates: why retrieval rule A works for one kind of question and rule B for another.
>
> In our own run, that would mean reading the traces of all three harnesses side by side, question by question. Our script did exactly that, by hand, in section 4b above. Meta-Harness gives that job to an agent, with no fixed format for what to read.

## Optimizers that rewrite code: AlphaEvolve and OpenEvolve {Appendix E}

> [!DEFINITION] Evolutionary search
> A search that keeps a population of candidates. It makes new ones by small random changes (**mutations**) or by mixing two (**crossover**), scores them, and keeps the better ones. In **tournament selection**, a few candidates are drawn at random and the best of them becomes the parent of the next one.

> [!PAPER] Novikov et al. (2025), AlphaEvolve · Abstract
> [![The AlphaEvolve abstract: a pipeline of LLMs improves an algorithm by making direct changes to the code, with an evolutionary approach and feedback from evaluators](https://ishwarj.com/img/papers/meta-harness/pb-alphaevolve-abstract.pngevolve-abstract.png)](/img/papers/meta-harness/pb-alphaevolve-abstract.png)
>
> **What it says:** AlphaEvolve uses LLMs to "improve an algorithm by making direct changes to the code", with an "evolutionary approach, continuously receiving feedback from one or more evaluators".
>
> **What it means:**
>
> Here the candidate is code, not a prompt. A language model plays the role of mutation: it reads a program and writes a changed version. An **evaluator** runs the program and returns a score, such as the speed of a kernel or the size of a mathematical construction. Good programs become parents of new ones.
>
> The setting matters. The program is one function, and the score is one clean number that a computer checks. **OpenEvolve** is an open-source version of the same loop, and the paper uses it as a direct competitor.

> [!PAPER] · Appendix E
> [![Meta-Harness on AlphaEvolve and OpenEvolve: a program database with scalar scores, fixed mutations on tournament-selected parents, for a single stateless function; harnesses are stateful programs](https://ishwarj.com/img/papers/meta-harness/pb-alphaevolve-mh.png-alphaevolve-mh.png)](/img/papers/meta-harness/pb-alphaevolve-mh.png)
>
> **What it says:** The proposer gets "a program database with scalar scores" and applies fixed mutations "to tournament-selected parents". The target is a "single stateless function with a clean scalar objective", but "harnesses are stateful programs" in which one choice "can cascade through an entire evaluation sequence".
>
> **What it means:**
>
> A **scalar** is a single number. An evolutionary loop sees, for each program, its code and one number. That is enough when you search for a fast sorting function: run it, time it, done.
>
> A harness is different because it is **stateful**. Suppose a classification harness stores every labelled example in memory. A bad storage rule on example 3 does no harm on example 3, but it poisons the retrieval for example 300. One final score cannot say where the damage started. The trace can.

## A harness that learns while it runs: ACE {§4.1}

The strongest hand-built harness in the paper's text-classification test is ACE. It is not an optimizer of code. It is a harness whose memory improves as it sees more examples.

> [!PAPER] Zhang et al. (2025), ACE · Abstract
> [![The ACE abstract: context adaptation, brevity bias and context collapse; ACE treats contexts as evolving playbooks built through generation, reflection and curation](https://ishwarj.com/img/papers/meta-harness/pb-ace-abstract.pngpb-ace-abstract.png)](/img/papers/meta-harness/pb-ace-abstract.png)
>
> **What it says:** LLM applications "rely on context adaptation". Earlier methods suffer from "brevity bias" and from "context collapse, where iterative rewriting erodes details over time". ACE "treats contexts as evolving playbooks", through "generation, reflection, and curation".
>
> **What it means:**
>
> **Context adaptation** is improving the prompt instead of the weights, which is what every harness on this page does. The two failures are easy to picture. Brevity bias: ask a model to summarise its lessons, and it drops the details that mattered. Context collapse: rewrite the same notes many times, and they slowly lose content, like a photocopy of a photocopy.
>
> ACE avoids both with three roles. A **Generator** answers tasks. A **Reflector** reads the runs and writes lessons. A **Curator** adds those lessons to a playbook as small separate items, and never rewrites the whole playbook at once. The playbook goes into the prompt of later tasks.

> [!PAPER] · §4.1
> [![The paper's main hand-designed baselines: ACE uses reflective memory curation; MCE maintains and evolves a library of natural-language skills](https://ishwarj.com/img/papers/meta-harness/pb-ace-mh.pngrness/pb-ace-mh.png)](/img/papers/meta-harness/pb-ace-mh.png)
>
> **What it says:** The main baselines are ACE, "which uses reflective memory curation to build context over time", and Meta Context Engineering (MCE), "which maintains and evolves a library of natural-language skills for context construction".
>
> **What it means:**
>
> These are the opponents that Meta-Harness must beat, and both are smart harnesses written by people. Both change their memory as they see more labelled examples. That is why the paper calls the task **online** classification. The examples come one at a time, and the harness may learn from each one before the next.
>
> Our retrieval harness is a very small member of this family. Its memory is a fixed list, and its only rule is "show the six most similar". ACE and MCE add reflection and curation on top. Part 3 shows the scores.

## Training during the search: TTT-Discover

> [!PAPER] Yuksekgonul et al. (2026), TTT-Discover · Abstract
> [![The TTT-Discover abstract: prior work like AlphaEvolve searches by prompting a frozen LLM; TTT-Discover does reinforcement learning at test time so the LLM continues to train](https://ishwarj.com/img/papers/meta-harness/pb-ttt-abstract.pngpb-ttt-abstract.png)](/img/papers/meta-harness/pb-ttt-abstract.png)
>
> **What it says:** Methods such as AlphaEvolve perform "search by prompting a frozen LLM". TTT-Discover performs "reinforcement learning at test time, so the LLM can continue to train" on one problem.
>
> **What it means:**
>
> Every other method on this page keeps the optimizer model frozen. TTT-Discover lets the optimizer model learn from its own tries on one problem, with **reinforcement learning**. Tries with a high reward make similar tries more likely. Its goal is one excellent solution to this one problem, not a model that is good on average.
>
> The Meta-Harness paper does not use the training part. It uses only TTT-Discover's rule for choosing which earlier solution to build on next, shown below.

> [!PAPER] Yuksekgonul et al. (2026), TTT-Discover · §3.2
> [![The PUCT rule of TTT-Discover: each state s is scored by Q(s) plus c times P(s) times the square root of 1 plus T over 1 plus n(s)](https://ishwarj.com/img/papers/meta-harness/pb-ttt-puct.pngess/pb-ttt-puct.png)](/img/papers/meta-harness/pb-ttt-puct.png)
>
> **What it says:** Each earlier solution $$s$$ is scored by $$Q(s) + c \cdot P(s) \cdot \sqrt{1 + T/(1 + n(s))}$$. $$Q(s)$$ is the best reward reached from $$s$$, $$P(s)$$ grows with $$s$$'s rank by reward, $$n(s)$$ counts how often $$s$$ was expanded, $$T$$ is the total number of expansions, and $$c$$ is the exploration coefficient.
>
> **What it means:**
>
> This is the old tension between exploiting and exploring. A restaurant you love is a safe choice. A restaurant you tried once and liked might be even better, but you know less about it. The score adds a bonus for "known to be good" and a bonus for "not tried much yet".
>
> Read the formula in two parts. $$Q(s)$$ is the exploit part: the best result so far from this solution. The second part is the explore bonus. It is large when $$n(s)$$ is small compared with $$T$$, so a solution that was rarely built on gets a push. $$P(s)$$ scales the bonus, so a solution with a high rank gets more of it.

Here is the rule in four lines of Python, on three made-up solutions after 8 expansions:

```python
def puct(Q, P, n, T, c=1.0):
    """TTT-Discover's reuse rule: exploit a high reward Q, trust a high prior P, explore what was rarely expanded."""
    return Q + c * P * math.sqrt(1 + T / (1 + n))
```

```text
== 2. The PUCT score: which earlier solution to build on next (c = 1, T = 8 expansions so far)
  A: best reward, built on 7 times   Q=0.90 P=0.6 n=7   score = 0.90 + 0.6 * sqrt(1 + 8/8) = 1.749
  B: good reward, built on once      Q=0.80 P=0.5 n=1   score = 0.80 + 0.5 * sqrt(1 + 8/2) = 1.918
  C: weak reward, never built on     Q=0.40 P=0.2 n=0   score = 0.40 + 0.2 * sqrt(1 + 8/1) = 1.000
  picked: B: good reward, built on once
```

Solution A has the best reward, 0.90, but it was built on 7 times already, so its bonus is small: $$\sqrt{1 + 8/8} = 1.414$$. Solution B has a lower reward, 0.80, but it was built on only once: $$\sqrt{1 + 8/2} = 2.236$$. B wins with 1.918 against 1.749. Solution C was never built on, but its reward and rank are so low that it scores only 1.000.

So PUCT does not always build on the best solution so far. It spreads the search a little, which helps when the best solution is a dead end. Note what PUCT reads: rewards, ranks and counts. It never reads a trace.

> [!PAPER] · §4.1
> [![The text optimizers compared in the paper: Best-of-N, OpenEvolve with LLM mutation, and the text-optimization part of TTT-Discover with the PUCT reuse rule](https://ishwarj.com/img/papers/meta-harness/pb-compared.pngess/pb-compared.png)](/img/papers/meta-harness/pb-compared.png)
>
> **What it says:** The paper compares against Best-of-N, OpenEvolve ("evolutionary search over programs with LLM mutation") and TTT-Discover, of which "we use only the text-optimization component", "proposal selection via the PUCT reuse rule".
>
> **What it means:**
>
> **Best-of-N** is the control: write N independent candidates from the same start, score all, keep the best. It has no memory of earlier tries at all. If a clever search cannot beat Best-of-N, the cleverness does not help.
>
> The comparison is fair in one important way: all methods use the same proposer model and the same budget of harness evaluations. Only the search procedure and what the proposer sees change. Part 3 gives the results.

## What each optimizer sees {Table 1}

> [!PAPER] · Table 1
> [![Table 1: each text optimizer's history, log content and context per iteration, from OPRO's past solution and score pairs at 0.002 million tokens to Meta-Harness's full logs at 10 million](https://ishwarj.com/img/papers/meta-harness/pb-table1.pngrness/pb-table1.png)](/img/papers/meta-harness/pb-table1.png)
>
> **What it says:** For each method, Table 1 gives its history ("Window", "Last", "Summary" or "Full"), its log content, and "MTok/iter", the authors' "best estimate of the full context generated from one evaluation of a text artifact". OPRO sees "past (solution, score) pairs"; Meta-Harness sees "all logs and scores".
>
> **What it means:**
>
> This table is the bridge from this page to the paper.
>
> Read the History column. "Window" keeps a few recent items. "Last" keeps only the current candidate. "Summary" keeps a written summary. "Full" keeps everything. Every method before Meta-Harness throws most of the information away before the optimizer sees it.
>
> The last column says how much information one evaluation produces. MTok means a million tokens. For OPRO it is 0.002 MTok, or 2,000 tokens. For Meta-Harness it is 10 MTok, or 10,000,000 tokens: hundreds of times the largest earlier value, 26,000 tokens for TTT-Discover. Ten million tokens do not fit in any context window, so the paper's proposer cannot read them as one prompt. It must search them, like the retrieval harness searched its memory.

<figure class="fig"><svg viewBox="0 0 760 286" role="img" aria-label="Table 1 of the paper as bars on a log scale: the context one evaluation produces for each optimizer, from 2,000 tokens for OPRO to 26,000 for TTT-Discover, and 10,000,000 for Meta-Harness."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="22.0" text-anchor="start">Context from one evaluation, in tokens (log scale)</text><text class="t-tick" x="162.0" y="59.0" text-anchor="end">OPRO</text><g class="mark"><title>OPRO: 0.002 MTok</title><rect class="s2" x="170" y="44" width="34.6" height="21" rx="3"/></g><text class="t-val" x="210.6" y="59.0" text-anchor="start">2,000</text><text class="t-tick" x="162.0" y="89.0" text-anchor="end">GEPA</text><g class="mark"><title>GEPA: 0.008 MTok</title><rect class="s2" x="170" y="74" width="103.9" height="21" rx="3"/></g><text class="t-val" x="279.9" y="89.0" text-anchor="start">8,000</text><text class="t-tick" x="162.0" y="119.0" text-anchor="end">Feedback Descent</text><g class="mark"><title>Feedback Descent: 0.012 MTok</title><rect class="s2" x="170" y="104" width="124.1" height="21" rx="3"/></g><text class="t-val" x="300.1" y="119.0" text-anchor="start">12,000</text><text class="t-tick" x="162.0" y="149.0" text-anchor="end">TextGrad</text><g class="mark"><title>TextGrad: 0.015 MTok</title><rect class="s2" x="170" y="134" width="135.3" height="21" rx="3"/></g><text class="t-val" x="311.3" y="149.0" text-anchor="start">15,000</text><text class="t-tick" x="162.0" y="179.0" text-anchor="end">AlphaEvolve</text><g class="mark"><title>AlphaEvolve: 0.022 MTok</title><rect class="s2" x="170" y="164" width="154.4" height="21" rx="3"/></g><text class="t-val" x="330.4" y="179.0" text-anchor="start">22,000</text><text class="t-tick" x="162.0" y="209.0" text-anchor="end">TTT-Discover</text><g class="mark"><title>TTT-Discover: 0.026 MTok</title><rect class="s2" x="170" y="194" width="162.7" height="21" rx="3"/></g><text class="t-val" x="338.7" y="209.0" text-anchor="start">26,000</text><text class="t-tick" x="162.0" y="239.0" text-anchor="end">Meta-Harness</text><g class="mark"><title>Meta-Harness: 10.0 MTok</title><rect class="s3" x="170" y="224" width="460.0" height="21" rx="3"/></g><text class="t-val" x="636.0" y="239.0" text-anchor="start">10,000,000</text><line class="axis" x1="170.0" y1="256.0" x2="630.0" y2="256.0"/><line class="axis" x1="170.0" y1="256.0" x2="170.0" y2="261.0"/><text class="t-tick" x="170.0" y="274.0" text-anchor="middle">1K</text><line class="axis" x1="285.0" y1="256.0" x2="285.0" y2="261.0"/><text class="t-tick" x="285.0" y="274.0" text-anchor="middle">10K</text><line class="axis" x1="400.0" y1="256.0" x2="400.0" y2="261.0"/><text class="t-tick" x="400.0" y="274.0" text-anchor="middle">100K</text><line class="axis" x1="515.0" y1="256.0" x2="515.0" y2="261.0"/><text class="t-tick" x="515.0" y="274.0" text-anchor="middle">1M</text><line class="axis" x1="630.0" y1="256.0" x2="630.0" y2="261.0"/><text class="t-tick" x="630.0" y="274.0" text-anchor="middle">10M</text></svg><figcaption>Table 1 as bars on a log scale. One evaluation gives an earlier optimizer 2,000 to 26,000 tokens of context. In the Meta-Harness settings, one evaluation gives up to 10,000,000 tokens, and the proposer reads it through a filesystem.</figcaption></figure>

Now you have the pieces the paper starts from. A harness is the code around a frozen model: memory, retrieval, prompt building and tool loops. It can move a score by a lot, as our 0.5B run showed. Earlier optimizers improved prompts or single functions, but each one saw a narrow view: scores, the last candidate, or a summary. [Part 1](https://ishwarj.com/papers/meta-harness/part-1-the-big-idea/) opens the paper with its central claim, and [Part 2](https://ishwarj.com/papers/meta-harness/part-2-the-method/) shows how Meta-Harness gives the optimizer the full view.

> [!TAKEAWAYS] Key takeaways
> - **A model call is one line:** $$y \sim p_M(\cdot \mid \text{prompt})$$. Everything else, memory, retrieval, examples, tools and loops, is the harness.
> - **The paper's definition:** a harness is "a stateful program that wraps a language model and determines what context the model sees at each step". The model stays frozen.
> - **The harness alone moves the score.** With the same Qwen2.5-0.5B weights, our six-intent Banking77 test went from 41.7% (zero-shot) to 51.7% (fixed examples) to 83.3% (retrieved examples).
> - **ReAct is a harness with a loop:** the model writes a thought and an action, the harness runs the tool and adds the observation. In our run, one parsing rule changed 0 of 3 right answers into 3 of 3.
> - **Frameworks such as DSPy organise harnesses** and tune prompts for a metric, but people still design the retrieval, memory and control logic.
> - **Text optimizers use an LLM as the optimizer.** OPRO sees (prompt, score) pairs; TextGrad sees criticism of the current prompt; GEPA reflects on one candidate's traces and keeps a Pareto front.
> - **Code optimizers such as AlphaEvolve and OpenEvolve** evolve single stateless functions with one clean score. Harnesses are stateful, so one early choice can hurt many later steps.
> - **ACE and MCE are the strongest hand-built baselines;** TTT-Discover's PUCT rule balances "best so far" against "rarely tried".
> - **Table 1:** earlier optimizers see 2,000 to 26,000 tokens per evaluation. Meta-Harness settings produce up to 10,000,000, so its proposer must search its history, not read it all.

## The paper and the code

[Paper (arXiv 2603.28052)](https://arxiv.org/abs/2603.28052) · [Project page](https://yoonholee.com/meta-harness/) · [Our scripts](https://github.com/ishwar6/ishwar-books/tree/main/code/papers/meta-harness)

