Meta-Harness, explained · Part 6 of 6 · Covers Figure 2, §1, §3, Algorithm 1, §4.1, Table 8, Appendix B.1, Figure 7, Appendix C.1, Appendix D

Build It from Scratch: A Small Meta-Harness in Python

We build the whole Meta-Harness loop in about a thousand lines of Python: a task, harness files, an evaluator that logs every prompt, a folder per candidate, and two proposers. Then we run it on a laptop with small local models and measure the gap between the search set and the test set.

Meta-Harness: End-to-End Optimization of Model Harnesses. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn. COLM 2026, 2026. arXiv:2603.28052

Parts 1 to 5 read the paper. In this part we build its loop ourselves, run it on a laptop, and look at what it finds and where it fails.

Everything runs on local models, with no API and no cost. The model that answers is Qwen2.5-0.5B-Instruct. The proposer that writes new harnesses is either a set of fixed rules or Qwen2.5-1.5B-Instruct. The paper used much stronger models, and this page says at each step what that changes.

The plan: Figure 2 as files

Our package is in code/papers/meta-harness/scratch. The picture below puts our file names on the pieces of Figure 2.

Filesystem Druns/<run>/candidates/harness.py, scores.json,traces.jsonl, proposal.mdProposer Pproposers.py: scripted rules,or Qwen2.5-1.5B writing code(an API or agent: not run)New harness Hone .py file:class Harnessupdate(), predict()validate()harness_api.pyimport, run on a fake model,reject broken filesevaluate()evaluate.py, frozen model MQwen2.5-0.5B-Instructmemory 60, search set 50readswritespassednew folderheldout/: the test set (100 messages)scored once, after the search; no proposer reads it
The loop of Figure 2 with our files. The proposer reads the folders of earlier candidates and writes a new harness file. validate() rejects broken files. evaluate() runs the frozen model on the search set and writes a new folder. The test set sits outside the loop and gets used once, at the end.
FilePiece of the paperWhat it does
task.pythe tasks X\mathcal{X}10 Banking77 intents; memory 60, search set 50, test set 100
llm.pythe frozen model MMQwen2.5-0.5B-Instruct, greedy decoding
harness_api.pyinterface validationthe Harness interface and validate()
design_space.pythe seedsa harness template with seven knobs
evaluate.pyEvaluate(H,M,XH, M, \mathcal{X})runs a harness and records every prompt and reply
store.pythe filesystem D\mathcal{D}one folder per candidate, and the Pareto frontier
proposers.pythe proposer PPscripted rules, a local LLM, and two that need a key
loop.pyAlgorithm 1the outer loop, line by line
run.py, mh.pyone full run; a small command-line tool over a run folder

The task

We use the same setup. Our harness gets 60 labeled messages through a method update(text, label), one at a time. Then it must label new messages through predict(text).

task.py follows this advice. It takes 6 memory messages and 5 search messages per intent from Banking77's training file, and 10 test messages per intent from its test file. The first section of our check script prints the sizes:

plain text
== 1. The task
10 labels: activate_my_card, card_arrival, card_delivery_estimate, card_not_working, ...
memory 60 (labeled stream), search 50, test 100
example: 'I made a purchase with my card and I was charged a fee for using it' -> card_payment_fee_charged

Our first harness, zero-shot, scores 32.0% on the search set, against 10% for a random guess. So the search set is hard for the baseline, and there is room to improve.

The frozen model

The model is the part that the search never changes. llm.py wraps Qwen2.5-0.5B-Instruct behind one call: llm(prompt) returns the reply. It decodes greedily: at each step, it takes the most likely next token. So the same prompt always gives the same reply, and two scores differ only because two harnesses differ.

The library does this inside model.generate(). To see what it does, llm.py also writes the loop out by hand (Method 1). The library way is LocalLLM.__call__ (Method 2):

python
@torch.no_grad()
def greedy_from_scratch(llm, prompt, max_new_tokens=16):
    """Method 1: greedy decoding written out. No KV cache, so it is slower, but every step is visible."""
    ids = llm.chat_ids(prompt)                                   # (1, n) the prompt tokens
    start = ids.shape[1]
    for _ in range(max_new_tokens):
        logits = llm.model(ids).logits                           # (1, n, vocab): a score for every next token
        next_id = logits[0, -1].argmax()                         # the most likely token after the last position
        if next_id.item() == llm.tok.eos_token_id or next_id.item() in llm.tok.all_special_ids:
            break                                                # the model says "I am done"
        ids = torch.cat([ids, next_id.view(1, 1)], dim=1)        # (1, n + 1): append and go again
    return llm.tok.decode(ids[0, start:], skip_special_tokens=True).strip()

The check sends three zero-shot prompts both ways:

plain text
== 2. The frozen model: greedy decoding two ways
from scratch: 'card_payment_not_recognised'  generate(): 'card_payment_not_recognised'  gold: card_payment_fee_charged
from scratch: 'pending_card_payment'         generate(): 'pending_card_payment'         gold: card_payment_not_recognised
from scratch: 'declined_card_payment'        generate(): 'declined_card_payment'        gold: card_payment_fee_charged
same reply in 3 of 3 prompts

The two ways agree on all three replies. All three replies are also wrong. That is the zero-shot harness at work: the model sees only the list of labels, and it guesses.

A harness is one Python file

Every harness must follow one interface. Here is the smallest useful harness, from the package's README:

python
from hlib import TfIdf, parse_label

class Harness:
    def __init__(self, llm, labels):
        self.llm, self.labels, self.memory = llm, labels, []

    def update(self, text, label):
        self.memory.append((text, label))

    def predict(self, text):
        index = TfIdf([t for t, _ in self.memory])
        shots = [self.memory[i] for i in index.top(text, 4)]
        prompt = 'Labels: ' + ', '.join(self.labels) + '\n\n'
        prompt += ''.join(f'Message: {t}\nLabel: {l}\n\n' for t, l in shots)
        prompt += f'Message: {text}\nLabel:'
        return parse_label(self.llm(prompt), self.labels) or self.labels[0]

update() writes the example into memory. predict() finds the 4 stored messages most similar to the new one, shows them with their labels, and asks the model. parse_label() turns a reply like "Label: card arrival." into a valid label.

One rule matters for honesty: a harness may not open files, import os, run programs or use the network. Without the rule, a harness could read the test set from disk. With it, everything the harness knows comes from update() and from the model.

Interface validation {Algorithm 1}

validate() in harness_api.py does exactly the three steps of the tip. It loads the file, makes a Harness with a fake model that always answers with the first label, calls update() six times and predict() twice. It also searches the source for the forbidden patterns. The check feeds it four files:

plain text
== 4. Interface validation (Algorithm 1, line 11)
the README harness         passed in 2.3 ms  
reads a file               rejected in 0.0 ms  forbidden pattern '\\bopen\\s*\\(': a harness may not touch files, pro
returns a made-up label    rejected in 0.9 ms  predict() returned 'other', which is not one of the labels
has no Harness class       rejected in 1.0 ms  no class named Harness

Each check takes at most 2.3 milliseconds. A full evaluation takes at least 37 seconds, more than ten thousand times longer. Later, the local LLM proposer gives this check real work.

Retrieval inside a harness: TF-IDF two ways {Appendix B.1}

The weight of a word ww in a text dd is a product of two numbers:

tfidf(w,d)=tf(w,d)×idf(w),idf(w)=ln⁡1+N1+df(w)+1\text{tfidf}(w, d) = \text{tf}(w, d) \times \text{idf}(w), \qquad \text{idf}(w) = \ln\frac{1 + N}{1 + \text{df}(w)} + 1

Read it left to right. tf(w,d)\text{tf}(w, d) counts how many times the word occurs in the text. NN is the number of stored texts, and df(w)\text{df}(w) counts how many of them contain the word. A rare word has a small df\text{df}, so a large IDF.

A tiny example with N=3N = 3 texts. The word "card" is in all three: idf=ln⁡(4/4)+1=1\text{idf} = \ln(4/4) + 1 = 1. The word "fee" is in one: idf=ln⁡(4/2)+1=1.69\text{idf} = \ln(4/2) + 1 = 1.69. So "fee" counts 1.69 times more than "card", which tells almost nothing in a set of card messages. Last, each vector is divided by its length, and the similarity of two texts is the dot product of their vectors: the cosine.

python
class TfIdf:
    def __init__(self, docs):
        self.docs = list(docs)
        self.n = len(self.docs)
        df = Counter(w for d in self.docs for w in set(words(d)))
        self.idf = {w: math.log((1 + self.n) / (1 + c)) + 1 for w, c in df.items()}
        self.vecs = [self.vector(d) for d in self.docs]

    def vector(self, text):
        tf = Counter(w for w in words(text) if w in self.idf)        # words never seen in memory are dropped
        v = {w: c * self.idf[w] for w, c in tf.items()}
        norm = math.sqrt(sum(x * x for x in v.values())) or 1.0
        return {w: x / norm for w, x in v.items()}

    def scores(self, text):
        q = self.vector(text)
        return [sum(q[w] * dv.get(w, 0.0) for w in q) for dv in self.vecs]   # cosine with every document

    def top(self, text, k):
        """Indices of the k most similar documents, best first."""
        s = self.scores(text)
        return sorted(range(self.n), key=lambda i: -s[i])[:k]

Method 2 is scikit-learn's TfidfVectorizer, given the same word splitter. Its default settings use the same IDF formula and the same length division:

python
sk = TfidfVectorizer(tokenizer=hlib.words, lowercase=False, token_pattern=None)   # same words as ours
D = sk.fit_transform(docs)
plain text
== 3. TF-IDF retrieval two ways
vocabulary: ours 198 words, sklearn 198 words
largest difference in cosine scores over 5 queries x 60 documents: 1.7e-16
same top-5 neighbours in 5 of 5 queries
query: 'I made a purchase with my card and I was charged a fee for using it'
  0.531  card_payment_fee_charged   'What is the fee charged for using my card?'
  0.357  card_payment_fee_charged   'Am I getting charged for making a payment?'
  0.299  card_payment_fee_charged   'I paid with a card and there is an extra charge. Why?'

The two methods agree to 1.7×10−161.7 \times 10^{-16}, the rounding limit of the computer. The three nearest stored messages for a "fee" question all carry the right label. That is why retrieval helps: the model sees the right answer next to a similar question.

The design space and the seeds {Appendix B.1}

design_space.py turns the strategies of Appendix B.1 into one harness template with seven knobs:

KnobValuesFrom the paper
shots0, 4, 8, 16, 60zero-shot and few-shot baselines
selectfirst, random, tfidfTF-IDF retrieval
coverageoff, onone example per label
contrastiveoff, onclose examples with different labels
verifyoff, ondraft verification (2 calls, 3 confirmers, 3 challengers)
label_stylecard_arrival, card arrival
instructionshort, careful

Each point of the space renders as a complete Python file with a CONFIG dict at the top. The three seeds are points too: zero-shot, 8 random examples (few-shot), and all 60 examples. These are the paper's own kinds of baseline (Table 2).

Evaluate, and keep everything

evaluate.py wraps the model in a small recorder, TracingLLM. Each call to the model appends the prompt and the reply to a list. The evaluator first feeds the memory stream, then predicts each message of the set:

python
def evaluate(harness_path, llm, labels, memory, eval_set):
    """Return (scores, traces). Accuracy is the share of messages whose predicted label equals the gold label."""
    t0 = time.time()
    tracer = TracingLLM(llm)
    harness = load_module(harness_path).Harness(tracer, list(labels))
    for ex in memory:
        harness.update(ex['text'], ex['label'])
    traces, correct, invalid, errors = [], 0, 0, 0
    per_label = collections.defaultdict(lambda: [0, 0])
    confusions = collections.Counter()
    for i, ex in enumerate(eval_set):
        tracer.calls = []
        try:
            pred = harness.predict(ex['text'])
            err = None
        except Exception as e:                                  # a crash on one message counts as a wrong answer
            pred, err = None, f'{type(e).__name__}: {e}'
            errors += 1
        if pred not in labels:
            invalid += 1
        ok = pred == ex['label']
        correct += ok
        per_label[ex['label']][0] += ok
        per_label[ex['label']][1] += 1
        if not ok:
            confusions[f"{ex['label']} -> {pred}"] += 1
        traces.append({'id': i, 'text': ex['text'], 'gold': ex['label'], 'pred': pred, 'correct': ok,
                       'calls': tracer.calls, 'prompt_chars': sum(len(c['prompt']) for c in tracer.calls),
                       **({'error': err} if err else {})})

The score is the accuracy on the search set. This is the paper's objective in its most basic form. The reward rr of one message is 1 if the label is right and 0 if not. The expectation in the objective becomes a mean over the 50 messages. Part 2 reads the formula symbol by symbol. The check confirms the mean on one saved candidate:

plain text
== 5. The objective: the mean reward over the search set
tiny example: rewards [1, 0, 1, 1, 0] -> mean 0.60
012_60_tfidf_examples_verify: 29 rewards of 1 out of 50 -> 58.0%  (scores.json: 58.0%)

Next to the accuracy, scores.json stores the prompt size (characters per message), the accuracy per label, and the five most common confusions. Each candidate gets a folder like this one:

plain text
runs/scripted-1/candidates/006_8_tfidf_examples_verify/
    harness.py      the code
    scores.json     accuracy 56.0, ctx_chars 2318, calls 2.0, accuracy by label, top confusions
    traces.jsonl    50 lines: the text, gold, pred, and every prompt and reply
    proposal.md     the parent, the diagnosis, the edit and the reason

The first trace line of that folder shows why traces matter. The message is "I was charged for using my card?", and the right label is card_payment_fee_charged. The first call replied declined_card_payment, a wrong draft. The second call showed confirmers and challengers, and the final answer was right. Only the trace shows that the second call fixed the first.

The Pareto frontier

store.py computes the frontier two ways. Method 1 follows the definition: compare each candidate with every other one. Method 2 sorts by prompt size, cheapest first, and keeps each candidate that beats the best accuracy so far:

python
def pareto_front(points):
    """Method 1, from the definition. points: list of (accuracy, context). Higher accuracy and lower context are better.
    A point is on the frontier if no other point is at least as good on both and strictly better on one."""
    front = []
    for i, (a, c) in enumerate(points):
        dominated = any((a2 >= a and c2 <= c) and (a2 > a or c2 < c) for j, (a2, c2) in enumerate(points) if j != i)
        if not dominated:
            front.append(i)
    return front


def pareto_front_sweep(points):
    """Method 2, the usual fast way: sort by context (cheapest first), keep each point that beats the best accuracy so far."""
    order = sorted(range(len(points)), key=lambda i: (points[i][1], -points[i][0]))
    front, best = [], float('-inf')
    for i in order:
        if points[i][0] > best:
            front.append(i)
            best = points[i][0]
        elif front and points[i] == points[front[-1]]:      # an exact tie with a kept point: neither dominates
            front.append(i)
    return sorted(front)

The first version of the sweep had no tie rule, and the check caught it: the sweep dropped two candidates that the definition kept. Two candidates with exactly the same accuracy and the same prompt size do not dominate each other, so both belong on the frontier. After the fix, the two methods agree on the 13 candidates of our scripted run:

plain text
== 6. The Pareto frontier two ways
13 candidates; by definition: [0, 1, 3, 5, 6, 7, 12]; by sort and sweep: [0, 1, 3, 5, 6, 7, 12]; same: True
  000_zero_shot                                 32.0%    402 chars
  001_few_shot_8                                42.0%   1139 chars
  003_8_tfidf_examples                          54.0%   1177 chars
  005_8_tfidf_examples_contrastive              54.0%   1177 chars
  006_8_tfidf_examples_verify                   56.0%   2318 chars
  007_8_tfidf_examples_contrastive_verify       56.0%   2318 chars
  012_60_tfidf_examples_verify                  58.0%   6956 chars

The terminal output of mh_part6.py: the six checks, from the task sizes to the Pareto frontier

The outer loop {Algorithm 1}

python
def meta_harness(run_dir, seeds, proposer, llm, labels, memory, search, iterations, k, log=print):
    store = Store(run_dir)                                                   # line 3: D <- empty
    log(f'== seeds ({len(seeds)})')
    for name, source, proposal in seeds:                                     # lines 4 to 6
        evaluate_into(store, name, source, proposal, llm, labels, memory, search, log)
    stats = {'proposed': 0, 'rejected': 0, 'duplicates': 0, 'evaluated': 0}
    for t in range(1, iterations + 1):                                       # line 7
        t0 = time.time()
        proposals = proposer.propose(store, t, k)                            # lines 8 and 9
        log(f'== iteration {t}: {len(proposals)} proposals ({time.time() - t0:.0f}s to propose)')
        seen = {normalise(c['source']): c['folder'] for c in store.candidates()}
        for p in proposals:                                                  # line 10
            stats['proposed'] += 1
            ok, err = validate(p['source'], labels, memory)                  # line 11
            if ok and normalise(p['source']) in seen:
                ok, err = False, f"duplicate: same code as {seen[normalise(p['source'])]}"
                stats['duplicates'] += 1
            if not ok:
                stats['rejected'] += 1
                d = store.reject(p['name'], p['source'], err, p['proposal'])
                log(f'  x {os.path.basename(d):44s} rejected: {err[:90]}')
                continue
            folder, _ = evaluate_into(store, p['name'], p['source'], p['proposal'], llm, labels, memory, search, log)
            seen[normalise(p['source'])] = folder
            stats['evaluated'] += 1
    return store, store.frontier(), stats                                    # line 13

We add one thing that the algorithm does not list: a duplicate check. If a proposal has the same code as an earlier candidate (ignoring spaces and comments), the loop rejects it. Our deterministic evaluator would only give the same score again.

Proposer 1: fixed rules

A proposer needs to read traces and turn them into an edit. The paper hands this job to a strong coding agent. We start with something we can fully understand: hand-written rules. This is the honest baseline. If a smart proposer cannot beat fixed rules, it adds nothing.

ScriptedProposer starts from the best candidate on the search set. It reads that candidate's traces and counts three things. The first is replies that are not a valid label. The second is wrong answers whose prompt showed no example of the right label. The third is the most common confusion.

Then fixed if-then rules turn the counts into edits. This is a slice of them:

python
        if cfg['shots'] > 0 and cfg['select'] != 'tfidf':
            out.append(({'select': 'tfidf'},
                        f"{d['gold_not_shown']} of {d['wrong']} wrong answers had no example of the gold label in the "
                        'prompt; pick the examples by similarity to the message'))
        if d['gold_not_shown'] > 0.3 * w and not cfg['coverage']:
            out.append(({'coverage': True},
                        f"{d['gold_not_shown']} of {d['wrong']} wrong answers never saw the gold label in an example; "
                        'add one example of every label'))
        if d['top_pair_count'] >= 2 and not cfg['contrastive']:
            g, p = d['top_pair']
            out.append(({'contrastive': True},
                        f"the most common mistake is {g} -> {p} ({d['top_pair_count']} times); show close examples "
                        'with different labels side by side'))

Each proposal writes its reasons to proposal.md. This is the one behind candidate 006:

plain text
Parent: 003_8_tfidf_examples (search accuracy 54.0%, 1177 prompt characters)

Diagnosis from the parent's traces: {"n": 50, "wrong": 23, "invalid": 0, "gold_not_shown": 4, "top_pair": ["card_arrival", "pending_card_payment"], "top_pair_count": 4, "ctx_chars": 1177}

Edit: mistakes repeat between the same labels; check the first answer against confirmers and challengers

The rules are the weak point, and we know it. They can only find what we thought of in advance. They cannot invent a new strategy, and they cannot notice a bug in their own code. Keep that in mind for the results.

Run 1: the scripted proposer

The run has 3 seeds and 5 iterations of 2 proposals, so 13 candidates. It ran on an Apple laptop GPU (mps) that other jobs shared, so the times are slower than an idle machine would give.

plain text
run scripted-1: proposer scripted, 5 iterations x 2 proposals
task model Qwen/Qwen2.5-0.5B-Instruct on mps; 10 labels; memory 60, search 50, test 100 (test is never shown to the proposer)
== seeds (3)
  + 000_zero_shot                                search acc  32.0%   ctx   402 chars   calls 1.0   (37.3s)
  + 001_few_shot_8                               search acc  42.0%   ctx  1139 chars   calls 1.0   (47.3s)
  + 002_few_shot_all                             search acc  26.0%   ctx  5803 chars   calls 1.0   (42.7s)
== iteration 1: 2 proposals (0s to propose)
  + 003_8_tfidf_examples                         search acc  54.0%   ctx  1177 chars   calls 1.0   (37.0s)
  + 004_8_random_examples_coverage               search acc  34.0%   ctx  2022 chars   calls 1.0   (43.0s)
== iteration 2: 2 proposals (0s to propose)
  + 005_8_tfidf_examples_contrastive             search acc  54.0%   ctx  1177 chars   calls 1.0   (60.4s)
  + 006_8_tfidf_examples_verify                  search acc  56.0%   ctx  2318 chars   calls 2.0   (127.2s)
== iteration 3: 2 proposals (0s to propose)
  + 007_8_tfidf_examples_contrastive_verify      search acc  56.0%   ctx  2318 chars   calls 2.0   (122.3s)
  + 008_16_tfidf_examples_verify                 search acc  50.0%   ctx  3093 chars   calls 2.0   (157.1s)
== iteration 4: 2 proposals (0s to propose)
  + 009_4_tfidf_examples_verify                  search acc  50.0%   ctx  1923 chars   calls 2.0   (95.7s)
  + 010_8_tfidf_examples_verify_careful_instruct search acc  56.0%   ctx  2480 chars   calls 2.0   (81.8s)
== iteration 5: 2 proposals (0s to propose)
  + 011_8_first_examples_verify                  search acc  54.0%   ctx  2342 chars   calls 2.0   (70.2s)
  + 012_60_tfidf_examples_verify                 search acc  58.0%   ctx  6956 chars   calls 2.0   (72.0s)
== search done in 16.8 min: {'proposed': 10, 'rejected': 0, 'duplicates': 0, 'evaluated': 10}

Read it from the top. Zero-shot gets 32.0%. Eight random examples get 42.0%. All 60 examples get only 26.0%: the 0.5B model gets lost in a 5,803-character prompt. More context is not always better, and the paper saw the same thing: past 32 examples, few-shot hurt on 7 of its 9 new datasets.

The first rule gives the largest jump. Choosing the 8 examples by TF-IDF instead of at random takes the score from 42.0% to 54.0%, at almost the same prompt size. Draft verification adds 2 points (56.0%) but doubles the calls. The last explore step, all 60 examples ranked by TF-IDF with verification, reaches 58.0%.

Candidate 005 is a lesson in itself. The contrastive knob changed the code, but not one prompt: the same 1,177 characters and the same 54.0%. Its block takes the nearest example of each label among the 8 nearest messages, and the 8 TF-IDF examples already contain them. A score alone hides this. The traces show it at once, because the prompts are the same.

203040506012345678910111213evaluated candidate (the first three are the seeds)search accuracy (%)scripted: 000_zero_shot, 32.0%scripted: 001_few_shot_8, 42.0%scripted: 002_few_shot_all, 26.0%scripted: 003_8_tfidf_examples, 54.0%scripted: 004_8_random_examples_coverage, 34.0%scripted: 005_8_tfidf_examples_contrastive, 54.0%scripted: 006_8_tfidf_examples_verify, 56.0%scripted: 007_8_tfidf_examples_contrastive_verify, 56.0%scripted: 008_16_tfidf_examples_verify, 50.0%scripted: 009_4_tfidf_examples_verify, 50.0%scripted: 010_8_tfidf_examples_verify_careful_instruct, 56.0%scripted: 011_8_first_examples_verify, 54.0%scripted: 012_60_tfidf_examples_verify, 58.0%local LLM: 000_zero_shot, 32.0%local LLM: 001_few_shot_8, 42.0%local LLM: 002_few_shot_all, 26.0%local LLM: 004_llm_t1_1, 48.0%local LLM: 005_llm_t2_0, 36.0%local LLM: 006_llm_t2_1, 48.0%local LLM: 010_llm_t4_1, 48.0%local LLM: 012_llm_t5_1, 48.0%scripted: best 58.0%local LLM: best 48.0%dots: each candidateline: best so far
Search accuracy of each candidate (dots) and the best so far (line), for the two proposers. The first three candidates are the same seeds in both runs. The scripted proposer climbs to 58.0%. The local LLM proposer stops at 48.0%.
203040506001,0002,0003,0004,0005,0006,0007,000prompt characters per message (lower is cheaper)search accuracy (%)000_zero_shot: 32.0%, 402 characters001_few_shot_8: 42.0%, 1,139 characters002_few_shot_all: 26.0%, 5,803 characters003_8_tfidf_examples: 54.0%, 1,177 characters004_8_random_examples_coverage: 34.0%, 2,022 characters005_8_tfidf_examples_contrastive: 54.0%, 1,177 characters006_8_tfidf_examples_verify: 56.0%, 2,318 characters007_8_tfidf_examples_contrastive_verify: 56.0%, 2,318 characters008_16_tfidf_examples_verify: 50.0%, 3,093 characters009_4_tfidf_examples_verify: 50.0%, 1,923 characters010_8_tfidf_examples_verify_careful_instruct: 56.0%, 2,480 characters011_8_first_examples_verify: 54.0%, 2,342 characters012_60_tfidf_examples_verify: 58.0%, 6,956 characterszero shotfew shot 8few shot all003 and 005006 and 007012
Each candidate of the scripted run as a dot: prompt size against search accuracy. The orange step line is the Pareto frontier. Moving right along it buys a few points of accuracy with a much longer prompt: from 1,177 characters at 54.0% to 6,956 characters at 58.0%.

The test set, at last

After the search, run.py scores every candidate of the scripted run on the 100 test messages:

plain text
== test set (100 messages) for 13 candidates
  000_zero_shot                                search  32.0%  test  38.0%  gap  -6.0   (frontier)
  001_few_shot_8                               search  42.0%  test  42.0%  gap  +0.0   (frontier)
  002_few_shot_all                             search  26.0%  test  29.0%  gap  -3.0
  003_8_tfidf_examples                         search  54.0%  test  53.0%  gap  +1.0   (frontier)
  004_8_random_examples_coverage               search  34.0%  test  36.0%  gap  -2.0
  005_8_tfidf_examples_contrastive             search  54.0%  test  53.0%  gap  +1.0   (frontier)
  006_8_tfidf_examples_verify                  search  56.0%  test  55.0%  gap  +1.0   (frontier)
  007_8_tfidf_examples_contrastive_verify      search  56.0%  test  55.0%  gap  +1.0   (frontier)
  008_16_tfidf_examples_verify                 search  50.0%  test  52.0%  gap  -2.0
  009_4_tfidf_examples_verify                  search  50.0%  test  52.0%  gap  -2.0
  010_8_tfidf_examples_verify_careful_instruct search  56.0%  test  53.0%  gap  +3.0
  011_8_first_examples_verify                  search  54.0%  test  50.0%  gap  +4.0
  012_60_tfidf_examples_verify                 search  58.0%  test  50.0%  gap  +8.0   (frontier)
selected by search accuracy: 012_60_tfidf_examples_verify search 58.0% test 50.0%
total 35.2 min, 3000 task-model calls

The candidate with the best search score, 012, is not the best on the test set. It drops from 58.0% to 50.0%, a gap of 8 points. Candidate 006 scores 2 points lower on the search set, and it holds up: 56.0% on search, 55.0% on test.

Why? One step on our search set is one message, so 2 points are one message. The difference between 006 and 012 is one more right answer out of 50. With so few messages, a lucky candidate can rise to the top, and 012 looks like such a case. The fix is not a cleverer proposer. It is a larger search set, or a rule that prefers the cheaper candidate when two scores are within noise.

20304050602030405060search accuracy (%)test accuracy (%)y = x000_zero_shot: search 32.0%, test 38.0%001_few_shot_8: search 42.0%, test 42.0%002_few_shot_all: search 26.0%, test 29.0%003_8_tfidf_examples: search 54.0%, test 53.0%004_8_random_examples_coverage: search 34.0%, test 36.0%005_8_tfidf_examples_contrastive: search 54.0%, test 53.0%006_8_tfidf_examples_verify: search 56.0%, test 55.0%007_8_tfidf_examples_contrastive_verify: search 56.0%, test 55.0%008_16_tfidf_examples_verify: search 50.0%, test 52.0%009_4_tfidf_examples_verify: search 50.0%, test 52.0%010_8_tfidf_examples_verify_careful_instruct: search 56.0%, test 53.0%011_8_first_examples_verify: search 54.0%, test 50.0%012_60_tfidf_examples_verify: search 58.0%, test 50.0%seedproposed, on the search frontierproposed, otherAbove the line: better on the test setthan on the search set. Below: worse.
Search accuracy against test accuracy for all 13 candidates of the scripted run, drawn like the paper's Figure 7. The seeds (green) sit on or above the line. The candidate with the highest search score (orange, far right) sits 8 points below it.

Proposer 2: a small local model

Now the real idea of the paper: a language model reads the folder and writes the code. We have no key for a strong model, so we use the largest model that fits next to the task model: Qwen2.5-1.5B-Instruct.

LocalLLMProposer works in two steps per proposal. First, it sees the list of candidates and names up to 3 files it wants to read. Second, it gets those files and writes a complete harness. We also always show it the source of the best harness, with the instruction to start from it. Without that help, both proposals of a first test crashed: one used a class that did not exist, and the other imported a made-up module. If its file fails validation, it gets the error and one more try.

plain text
run local-llm-1: proposer local-llm, 5 iterations x 2 proposals
task model Qwen/Qwen2.5-0.5B-Instruct on mps; 10 labels; memory 60, search 50, test 100 (test is never shown to the proposer)
== seeds (3)
  + 000_zero_shot                                search acc  32.0%   ctx   402 chars   calls 1.0   (75.6s)
  + 001_few_shot_8                               search acc  42.0%   ctx  1139 chars   calls 1.0   (68.2s)
  + 002_few_shot_all                             search acc  26.0%   ctx  5803 chars   calls 1.0   (79.1s)
== iteration 1: 2 proposals (413s to propose)
  x 003_llm_t1_0                                 rejected: ImportError: cannot import name 'log' from 'hlib' (/Users/admin/Desktop/ishwar-books/code/
  + 004_llm_t1_1                                 search acc  48.0%   ctx  1848 chars   calls 1.0   (40.7s)
== iteration 2: 2 proposals (237s to propose)
  + 005_llm_t2_0                                 search acc  36.0%   ctx  1870 chars   calls 1.0   (36.9s)
  + 006_llm_t2_1                                 search acc  48.0%   ctx  1848 chars   calls 1.0   (31.9s)
== iteration 3: 2 proposals (376s to propose)
  x 007_llm_t3_0                                 rejected: duplicate: same code as 004_llm_t1_1
  x 008_llm_t3_1                                 rejected: duplicate: same code as 004_llm_t1_1
== iteration 4: 2 proposals (239s to propose)
  x 009_llm_t4_0                                 rejected: NameError: name 'aspect' is not defined | draft = parse_label(self.llm(self.prompt(text, s
  + 010_llm_t4_1                                 search acc  48.0%   ctx  1848 chars   calls 1.0   (12.9s)
== iteration 5: 2 proposals (111s to propose)
  x 011_llm_t5_0                                 rejected: duplicate: same code as 005_llm_t2_0
  + 012_llm_t5_1                                 search acc  48.0%   ctx  1848 chars   calls 1.0   (8.6s)
== search done in 29.8 min: {'proposed': 10, 'rejected': 5, 'duplicates': 3, 'evaluated': 5}

Of 10 proposals, 2 crashed validation, 3 were exact copies of earlier code, and 5 ran. The best search score is 48.0%, 6 points above the best seed and 10 points below the scripted proposer. But the score does not tell the whole story. The files do.

Candidate 004 changed the CONFIG line of its parent: 5 random examples instead of 8, labels as words, and a longer instruction. It also turned on the coverage block, and it rewrote one line of it:

plain text
-        return [self.nearest(text, 1, keep=lambda l, x=label: l == x)[0]
+        return [self.nearest(text, 1, keep=lambda l: True)[0]

The old line fetched the nearest example of each label. The new line fetches the nearest example of any label, ten times. So every prompt now starts with the same message, repeated ten times:

plain text
Examples:
Message: What is the fee charged for using my card?
Label: card payment fee charged
Message: What is the fee charged for using my card?
Label: card payment fee charged
Message: What is the fee charged for using my card?
Label: card payment fee charged

This is a bug, and it still raised the score from 42.0% to 48.0%. By accident, it puts the single most similar stored message into the prompt, with a lot of weight. That is a crude form of the retrieval that the scripted proposer found on purpose. The validation check cannot catch it, because the code runs and returns valid labels.

The other proposals show the same pattern. Candidate 005 passes a label name, pending_card_payment, into every prompt as an extra line, and it drops to 36.0%. Candidates 006, 010 and 012 differ from 004 only in comments and in an unused test line at the end of the file. So they give exactly the same 48.0%. Our duplicate check ignores comments, but not code that never runs.

On the test set, the LLM run's best candidate also scores below the scripted one:

plain text
== test set (100 messages) for 7 candidates
  000_zero_shot                                search  32.0%  test  38.0%  gap  -6.0   (frontier)
  001_few_shot_8                               search  42.0%  test  42.0%  gap  +0.0   (frontier)
  002_few_shot_all                             search  26.0%  test  29.0%  gap  -3.0
  004_llm_t1_1                                 search  48.0%  test  46.0%  gap  +2.0   (frontier)
  006_llm_t2_1                                 search  48.0%  test  46.0%  gap  +2.0   (frontier)
  010_llm_t4_1                                 search  48.0%  test  46.0%  gap  +2.0   (frontier)
  012_llm_t5_1                                 search  48.0%  test  46.0%  gap  +2.0   (frontier)
selected by search accuracy: 004_llm_t1_1 search 48.0% test 46.0%
total 31.8 min, 1100 task-model calls
ProposerCandidates evaluatedBest searchIts test scoreProposal time
none (best seed, few-shot 8)342.0%42.0%
scripted rules1058.0% (012)50.0%under 1 second
Qwen2.5-1.5B-Instruct5 of 1048.0% (004)46.0%111 to 413 seconds per iteration
best scripted candidate on test (006)56.0%55.0%

The API proposer, for readers with a key

The package has two more proposers that we did not run. Both need something that costs money or that we do not have here.

ChatAPIProposer does the same two steps as the local proposer, but it sends them to any chat endpoint that speaks the common /chat/completions protocol. Only the chat method changes:

python
    def chat(self, prompt, max_new_tokens, seed):
        body = {'model': self.model, 'temperature': self.temperature, 'max_tokens': max_new_tokens, 'seed': seed,
                'messages': [{'role': 'system', 'content': SKILL}, {'role': 'user', 'content': prompt}]}
        req = urllib.request.Request(f'{self.base}/chat/completions', data=json.dumps(body).encode(),
                                     headers={'Content-Type': 'application/json',
                                              'Authorization': f'Bearer {self.key}'})
        with urllib.request.urlopen(req, timeout=600) as r:
            return json.load(r)['choices'][0]['message']['content']

AgentCLIProposer is the paper's own setup. It writes SKILL.md into the run folder and starts a coding agent there, with its own file tools. The agent writes its harnesses into pending/, and the loop collects, validates and evaluates them. To use either one:

bash
export MH_API_BASE=https://<provider>/v1 MH_API_KEY=... MH_API_MODEL=<model>
python run.py --proposer api --iters 10 --k 2 --name api-1

export MH_AGENT_CMD='<your-agent-command> --print {prompt}'
python run.py --proposer agent-cli --iters 10 --k 2 --name agent-1

Our results do not predict what such a run would find. The paper's proposer is Anthropic's coding agent with Opus 4.6, and its proposals read about 82 files each. We expect a strong proposer to beat our fixed rules, because it can do what the rules cannot. It can notice the coverage bug, invent a strategy that is not a knob, and test a change before it hands it in. That is a claim of the paper, not something this page measured.

What our small version cannot show

  • The scale. The paper evaluates about 60 harnesses per run and up to 10,000,000 tokens of diagnostic information per evaluation. Our runs evaluate 13 and 8 candidates, and each trace file holds 50 records that fit in a text editor.
  • A strong proposer. Our only LLM proposer has 1.5B parameters and no tools. It cannot test the result of its own edits, the ability that the paper's footnote calls essential.
  • Statistical confidence. One search-set message is 2 points and one test message is 1 point. Most of our differences are a few messages. We ran each search once, with one seed.
  • The paper's numbers. We use a different model (Qwen2.5-0.5B, not GPT-OSS-120B) and 10 of Banking77's 77 intents. None of our accuracies are comparable with Tables 2 to 5.

What it does show is the machinery. Each piece of Figure 2 and Algorithm 1 is a small, testable function. Traces explain results that scores hide, such as candidate 005 and the coverage bug of candidate 004. And the search-test gap appears even in a loop of 13 candidates.

The paper and the code

arXiv 2603.28052 · Project page · Our code