Meta-Harness, explained · Part 3 of 6 · Covers §4.1, Tables 2 to 5, Figures 3 to 7, Table 9, Appendix B.1

Text Classification: What the Search Found

The first test of Meta-Harness: a model sorts short texts into classes, and the search writes the code that picks its examples. We read the tables, the accuracy and context trade-off, and the two discovered harnesses, then rebuild both around a small local model.

Meta-Harness: End-to-End Optimization of Model Harnesses. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn. COLM 2026, 2026. arXiv:2603.28052

A patient writes "I have a burning sensation in my stomach, and it is worse when I lie down". Which disease is it? A language model can guess, but its guess depends a lot on which solved cases you show it first. Section 4.1 lets Meta-Harness write the code that picks those cases. Each piece below shows the exact lines of the paper, then explains them. At the end, we rebuild the two harnesses it found and run them on a laptop.

The task: online text classification

labeled examples arrive one at a time"burning in my stomach..."-> peptic ulcer"itchy red rash..."-> fungal infection"high fever, chills..."-> malariamemory: every labeled example so farnew description(no label)HARNESS (the code)what to retrieve, what to show,how many callsfrozen modelGPT-OSS-120Bone labelMeta-Harness rewrites only the HARNESS box.The model and the data stay the same.
The online setup. Labeled examples go into a memory. For a new description, the harness chooses what to take from the memory and what prompt to send. The model stays frozen; the search changes only the harness.

The baselines and Table 2

Two goals at once: the Pareto frontier

Before Figure 3, we need one idea from economics. It answers the question "which of these options is best?" when there are two goals that pull in different directions.

A small example makes it concrete. Take four harnesses, each written as (extra context, accuracy): A = (0, 30%), B = (2,000, 35%), C = (3,000, 33%), D = (5,000, 45%). C costs more than B and is less accurate, so B dominates C, and C is off the frontier. A, B and D each win on something: A is the cheapest, D is the most accurate, and B sits between them. So the frontier is A, B and D. No single winner exists; the frontier is the menu of good trade-offs, and the user picks from it.

Against other optimizers

Table 4 sits beside the ACE paragraph, in the first screenshot of the Table 2 section. It gives the same race as numbers, on the search set: the median and best harness of each optimizer. Meta-Harness has a median of 50.0 and a best of 56.7. The best rival is TTT-Discover with a best of 45.6, and GEPA is lowest with 40.2.

What the proposer must see: Table 3

New datasets: Table 5

The two harnesses it found

Retrieval: finding the closest solved cases

Both harnesses need one tool: given a new description, find the most similar descriptions in the memory. The paper's Label-Primed Query uses TF-IDF for this, so we use TF-IDF in both.

The IDF of a word is

idf(w)=ln⁡1+n1+df(w)+1\text{idf}(w) = \ln\frac{1 + n}{1 + \text{df}(w)} + 1

Read it from left to right. nn is the number of texts in the memory. df(w)\text{df}(w), the document frequency, is the number of texts that contain the word ww. The fraction is large when few texts contain the word. The logarithm ln⁡\ln shrinks large ratios, and the +1+1 terms stop a division by zero and keep every weight above zero. This is the exact form that scikit-learn uses by default.

A tiny example: with n=751n = 751 texts, a word in 1 text gets ln⁡(752/2)+1=6.93\ln(752 / 2) + 1 = 6.93. A word in all 751 texts gets ln⁡(752/752)+1=1\ln(752 / 752) + 1 = 1. Our real memory gives these values:

plain text
vocabulary: ours 988 words, scikit-learn 988 words
largest difference between the two matrices: 1.11e-16
  word 'and'        in 728 of 751 texts, idf = 1.031
  word 'stomach'    in  69 of 751 texts, idf = 3.374
  word 'burning'    in  25 of 751 texts, idf = 4.365
  word 'heartburn'  in  20 of 751 texts, idf = 4.578

"and" is in most texts and gets a weight near 1. "heartburn" is in few texts and gets a weight about four times larger. So two descriptions that share "heartburn" count as much more similar than two that share "and".

Method 1, from scratch. The functions below build the vocabulary, count document frequencies, compute the IDF, and make one unit-length row per text. Memory.top returns the indices of the k most similar memory texts; its keep argument lets a harness ask for "the closest texts with this label" or "with any other label".

python
def tokenize(text):
    return TOKEN.findall(text.lower())


def tfidf_fit(docs):
    """Method 1: TF-IDF from scratch. Returns the vocabulary, the idf weights and the document matrix."""
    vocab = {w: i for i, w in enumerate(sorted({w for d in docs for w in tokenize(d)}))}
    df = np.zeros(len(vocab))                               # in how many documents each word appears
    for d in docs:
        for w in set(tokenize(d)):
            df[vocab[w]] += 1
    n = len(docs)
    idf = np.log((1 + n) / (1 + df)) + 1                    # rare words get a large weight, common words a small one
    return vocab, idf, tfidf_transform(docs, vocab, idf)


def tfidf_transform(docs, vocab, idf):
    X = np.zeros((len(docs), len(vocab)))                  # one row per document, one column per word
    for r, d in enumerate(docs):
        for w in tokenize(d):
            if w in vocab:
                X[r, vocab[w]] += 1                        # tf: how often the word appears in this document
    X *= idf                                               # tf x idf
    norms = np.linalg.norm(X, axis=1, keepdims=True)
    return X / np.where(norms == 0, 1, norms)


class Memory:
    """The labeled examples a harness has seen, with TF-IDF retrieval over their texts."""
    def __init__(self, examples):
        self.texts = [t for t, _ in examples]
        self.labels = [y for _, y in examples]
        self.vocab, self.idf, self.X = tfidf_fit(self.texts)

    def sims(self, query):
        q = tfidf_transform([query], self.vocab, self.idf)[0]
        return self.X @ q                                  # cosine similarity of the query with every memory text

    def top(self, query, k, keep=lambda i: True):
        order = np.argsort(-self.sims(query), kind='stable')
        return [int(i) for i in order if keep(i)][:k]

Method 2, the library. scikit-learn's TfidfVectorizer does the same in one line. The script fits both on the 751 memory texts, compares the two matrices, and checks that every test query gets the same five neighbours:

python
T = Log('part3_tfidf')
vec = TfidfVectorizer()
Xs = vec.fit_transform(mem.texts).toarray()                     # Method 2: the library
cols = [vec.vocabulary_[w] for w in sorted(mem.vocab, key=mem.vocab.get)]
T(f'vocabulary: ours {len(mem.vocab)} words, scikit-learn {len(vec.vocabulary_)} words')
T(f'largest difference between the two matrices: {np.abs(mem.X - Xs[:, cols]).max():.2e}')
for w in ['and', 'stomach', 'burning', 'heartburn']:
    df = sum(w in set(tokenize(t)) for t in mem.texts)
    T(f'  word {w!r:<12} in {df:>3} of {len(mem.texts)} texts, idf = {mem.idf[mem.vocab[w]]:.3f}')
agree = 0
for q, _ in test:
    ours = mem.top(q, 5)
    lib = list(np.argsort(-(Xs @ vec.transform([q]).toarray()[0]), kind='stable')[:5])
    agree += ours == [int(i) for i in lib]
T(f'test queries with the same top-5 neighbours: {agree} of {len(test)}')
plain text
vocabulary: ours 988 words, scikit-learn 988 words
largest difference between the two matrices: 1.11e-16
  word 'and'        in 728 of 751 texts, idf = 1.031
  word 'stomach'    in  69 of 751 texts, idf = 3.374
  word 'burning'    in  25 of 751 texts, idf = 4.365
  word 'heartburn'  in  20 of 751 texts, idf = 4.578
test queries with the same top-5 neighbours: 212 of 212
query: "I have a burning sensation in my stomach that comes and goes. It's worse when I eat and wh..."
  0.673  peptic ulcer disease     "I have a burning sensation in my upper stomach that comes an..."
  0.566  peptic ulcer disease     "I have a burning feeling in my stomach. It's worse when I ea..."
  0.561  gastroesophageal reflux disease "I have a burning sensation in my chest and throat. It feels ..."
  0.505  gastroesophageal reflux disease "I have a burning sensation in my throat that comes and goes,..."
  0.498  gastroesophageal reflux disease "I get heartburn and indigestion a lot. It's worse when I eat..."

The two matrices agree to 10−1610^{-16}, the size of a rounding error, and all 212 test queries get the same top 5. The last lines show the retrieval for our first test description. Its two nearest cases are peptic ulcer, the right answer. The next three are acid reflux (gastroesophageal reflux disease), which has very similar symptoms. A majority vote of the five would pick the wrong disease, three to two. Keep this example in mind.

Draft Verification

The authors' public repository (stanford-iris-lab/meta-harness, folder reference_examples/text_classification) has the task, the data and the baseline harnesses. Every harness there is a class with two methods: predict(input) makes a label before the right answer is known, and learn_from_batch(batch_results) updates the memory afterwards. The two discovered files, draft_verification.py and label_primed_query_anchored.py, are not in it. So our version follows the description above and Figure 5, and our prompts and exact retrieval are our own reading. call is the model call: it takes a prompt and returns a parsed label.

python
def shots(mem, idx):
    return '\n\n'.join(f'Description: {mem.texts[i]}\nDisease: {mem.labels[i]}' for i in idx)


def draft_verification(q, mem, labels, call, k=5):
    """Figure 5. Call 1 drafts a label from the 5 nearest examples. Call 2 shows 5 examples that agree with the draft
    (confirmers) and 5 that do not (challengers), and keeps or revises the draft."""
    if len(mem.texts) < k:                                          # cold start: a single few-shot call
        return few_shot(q, mem, labels, call, n=len(mem.texts))
    near = mem.top(q, k)
    p1 = (f'Classify the patient description as one of these diseases: {label_list(labels)}.\n\n'
          f'Similar solved cases:\n\n{shots(mem, near)}\n\nDescription: {q}\n{ASK}')
    draft = call(p1)
    if draft is None:
        return None, [p1]
    confirm = mem.top(q, k, keep=lambda i: mem.labels[i] == draft)  # the second retrieval depends on the draft
    challenge = mem.top(q, k, keep=lambda i: mem.labels[i] != draft)
    p2 = (f'Classify the patient description as one of these diseases: {label_list(labels)}.\n\n'
          f'A first guess was: {draft}.\n\nCases WITH the label {draft}:\n\n{shots(mem, confirm)}\n\n'
          f'Similar cases with OTHER labels:\n\n{shots(mem, challenge)}\n\n'
          f'Description: {q}\nKeep the first guess or change it. {ASK}')
    return call(p2), [p1, p2]

The line with confirm = mem.top(q, k, keep=lambda i: mem.labels[i] == draft) is the "key discovered behavior": the second retrieval depends on the draft. Here is the second prompt for our first test description, as the script printed it. The draft was acid reflux, which is wrong:

plain text
Classify the patient description as one of these diseases: allergy, arthritis, bronchial asthma, cervical spondylosis, chicken pox, common cold, dengue, diabetes, drug reaction, fungal infection, gastroesophageal reflux disease, hypertension, impetigo, jaundice, malaria, migraine, peptic ulcer disease, pneumonia, psoriasis, typhoid, urinary tract infection, varicose veins.

A first guess was: gastroesophageal reflux disease.

Cases WITH the label gastroesophageal reflux disease:

Description: I have a burning sensation in my chest and throat. It feels like acid or food is coming up into my throat. It's worse when I lie down. I also have heartburn and indigestion after I eat, and I sometimes throw up.
Disease: gastroesophageal reflux disease

Description: I have a burning sensation in my throat that comes and goes, and it's worse after I eat. I also have a bitter or sour aftertaste in my mouth sometimes.
Disease: gastroesophageal reflux disease

[... 3 more cases with this label ...]

Similar cases with OTHER labels:

Description: I have a burning sensation in my upper stomach that comes and goes. I also have heartburn, indigestion, and nausea.
Disease: peptic ulcer disease

Description: I have a burning feeling in my stomach. It's worse when I eat or drink. I also get heartburn, indigestion, and nausea.
Disease: peptic ulcer disease

[... 3 more cases with other labels ...]

Description: I have a burning sensation in my stomach that comes and goes. It's worse when I eat and when I lie down. I also have heartburn and indigestion.
Keep the first guess or change it. Answer with the disease name only.
----- model answer: 'gastroesophageal reflux disease'

Label-Primed Query

python
def label_primed_query(q, mem, labels, call, n_pairs=4, pool=20):
    """Figure 6. One prompt in three blocks: the label primer, one query-relevant example per label (coverage),
    and pairs of close examples with different labels taken from the query's neighbourhood (contrastive pairs)."""
    s = mem.sims(q)
    coverage = []
    for lab in labels:                                              # the best example of every label
        own = [i for i in range(len(mem.texts)) if mem.labels[i] == lab]
        coverage.append(max(own, key=lambda i: s[i]))
    near = [int(i) for i in np.argsort(-s, kind='stable')[:pool]]  # the query's neighbourhood
    pairs, used = [], set()
    for a in near:                                                 # pair each close example with the closest one
        if a in used:                                              # that carries a different label
            continue
        b = next((j for j in near if j not in used and j != a and mem.labels[j] != mem.labels[a]), None)
        if b is not None:
            pairs.append((a, b)); used |= {a, b}
        if len(pairs) == n_pairs:
            break
    pair_text = '\n\n'.join(f'Pair {n + 1}:\n{shots(mem, [a])}\n{shots(mem, [b])}' for n, (a, b) in enumerate(pairs))
    p = (f'Valid labels: {label_list(labels)}.\n\n'
         f'One example for each label:\n\n{shots(mem, coverage)}\n\n'
         f'Close cases with different labels:\n\n{pair_text}\n\n'
         f'Description: {q}\nClassify the description as one of the valid labels. {ASK}')
    return call(p), [p]

The coverage loop picks, for every label, the memory example closest to the query. The pairing loop walks down the 20 nearest examples and pairs each with the next close example that has a different label. Here is the prompt for our first test description, cut to its three blocks (the full prompt is 22 coverage examples and 4 pairs):

plain text
Valid labels: allergy, arthritis, bronchial asthma, cervical spondylosis, chicken pox, common cold, dengue, diabetes, drug reaction, fungal infection, gastroesophageal reflux disease, hypertension, impetigo, jaundice, malaria, migraine, peptic ulcer disease, pneumonia, psoriasis, typhoid, urinary tract infection, varicose veins.

One example for each label:

Description: I have a sore throat and a lot of sneezing. There are times when the skin around my eyes and my lips swell. I find that once I start, I just cannot stop.
Disease: allergy

Description: I have a lot of pain in my neck and my muscles feel weak. My joints are swollen and I have a lot of stiffness when I walk. Walking is also very uncomfortable.
Disease: arthritis

Description: I have a bad cough and I'm really tired. I have a high fever and my breathing is shallow. When I cough, a lot of saliva comes out.
Disease: bronchial asthma

[... 19 more examples, one per label ...]

Close cases with different labels:

Pair 1:
Description: I have a burning sensation in my upper stomach that comes and goes. I also have heartburn, indigestion, and nausea.
Disease: peptic ulcer disease
Description: I have a burning sensation in my chest and throat. It feels like acid or food is coming up into my throat. It's worse when I lie down. I also have heartburn and indigestion after I eat, and I sometimes throw up.
Disease: gastroesophageal reflux disease

Pair 2:
Description: I have a burning feeling in my stomach. It's worse when I eat or drink. I also get heartburn, indigestion, and nausea.
Disease: peptic ulcer disease
Description: I have a burning sensation in my throat that comes and goes, and it's worse after I eat. I also have a bitter or sour aftertaste in my mouth sometimes.
Disease: gastroesophageal reflux disease

[... 2 more pairs ...]

Description: I have a burning sensation in my stomach that comes and goes. It's worse when I eat and when I lie down. I also have heartburn and indigestion.
Classify the description as one of the valid labels. Answer with the disease name only.
----- model answer: 'gastroesophageal reflux disease'

Our run: the two harnesses on a small model

Now we run everything. The task model is Qwen2.5-1.5B-Instruct, an open model with 1.5 billion parameters, on the laptop's Apple GPU. It stays frozen, as in the paper. The data is Symptom2Disease from Hugging Face (gretelai/symptom_to_diagnosis):

plain text
Symptom2Disease (gretelai/symptom_to_diagnosis): 22 labels
memory: 751 labeled examples the harness may read
search: 100 held-out training examples (to pick a harness)
test:   212 examples of the official test split, never used to choose anything
training texts dropped because the same text is in the test split: 2
labels: allergy, arthritis, bronchial asthma, cervical spondylosis, chicken pox, common cold, dengue, diabetes, drug reaction, fungal infection, gastroesophageal reflux disease, hypertension, impetigo, jaundice, malaria, migraine, peptic ulcer disease, pneumonia, psoriasis, typhoid, urinary tract infection, varicose veins
example: "I have a burning sensation in my stomach that comes and goes. It's worse when I eat and when I lie down. I also have heartburn and indigestion." -> peptic ulcer disease

The memory is the notebook: 751 solved cases that every harness may read. We fill it before the test, which is where the paper's online memory ends up after the last labeled example arrives. The search split is 100 more training cases that play the role of the search set. The test split is never used to choose anything. We do not run a search here (that is Part 6); we only compare fixed harnesses on both splits.

All our harnesses give the model the list of 22 diseases, even zero-shot. A model that does not know the names cannot match them. Even with the list, zero-shot gives 20 answers that match no label (see the results). Because every harness has the list, it is not counted as extra context.

One model call, two ways

Method 1 is our own loop: run the model, take the top token, feed it back, and stop at the end-of-turn token. It keeps the past keys and values (past_key_values), so each new step reads only one new token. Method 2 is the library's model.generate with sampling turned off. parse then maps the answer to one of the 22 labels.

python
class Model:
    def __init__(self, name=MODEL, dtype='bfloat16'):   # 16-bit numbers: half the memory of float32
        import torch
        from transformers import AutoModelForCausalLM, AutoTokenizer
        self.torch = torch
        self.dev = 'mps' if torch.backends.mps.is_available() else 'cpu'
        self.tok = AutoTokenizer.from_pretrained(name)
        self.lm = AutoModelForCausalLM.from_pretrained(name, dtype=getattr(torch, dtype)).to(self.dev).eval()

    def ids(self, user):
        msgs = [{'role': 'user', 'content': user}]
        text = self.tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
        return self.tok(text, return_tensors='pt').input_ids.to(self.dev)

    def greedy(self, user, max_new=16):
        """Method 1: our own greedy decoding loop. Run the model, take the most likely next token, append, repeat."""
        torch = self.torch
        ids = self.ids(user)
        out, past = [], None
        with torch.no_grad():
            x = ids
            for _ in range(max_new):
                r = self.lm(input_ids=x, past_key_values=past, use_cache=True)
                past = r.past_key_values                   # keep the keys and values, so each step reads one token
                nxt = int(r.logits[0, -1].argmax())        # the most likely next token
                if nxt == self.tok.eos_token_id or nxt == self.tok.convert_tokens_to_ids('<|im_end|>'):
                    break
                out.append(nxt)
                x = torch.tensor([[nxt]], device=self.dev)
        return self.tok.decode(out, skip_special_tokens=True)

    def generate(self, user, max_new=16, repetition_penalty=1.0):
        """Method 2: the library way, model.generate with sampling turned off.
        Qwen's generation_config.json also sets repetition_penalty=1.1; 1.0 turns it off, as in our loop."""
        ids = self.ids(user)
        with self.torch.no_grad():
            g = self.lm.generate(ids, max_new_tokens=max_new, do_sample=False, top_p=None, top_k=None,
                                 temperature=None, repetition_penalty=repetition_penalty,
                                 pad_token_id=self.tok.eos_token_id)
        return self.tok.decode(g[0, ids.shape[1]:], skip_special_tokens=True)


def parse(answer, labels):
    """Map the model's text to one valid label: an exact label inside the answer, else the closest spelling."""
    a = answer.lower().strip()
    hits = [l for l in labels if l in a]
    if hits:
        return max(hits, key=len)
    close = difflib.get_close_matches(a.split('\n')[0], labels, n=1, cutoff=0.6)
    return close[0] if close else None
plain text
--- model.generate with repetition_penalty=1.1
ours: 'Gastroesophageal reflux disease (GERD)'   generate: 'Gastroesophageal Reflux Disease (GERD)'
ours: 'Gastroesophageal reflux disease (GERD)'   generate: 'Gastroesophageal Reflux Disease (GERD)'
ours: 'Migraine'                                 generate: 'Migraine'
ours: 'Pneumonia'                                generate: 'Pneumonia'
ours: 'psoriasis'                                generate: 'psoriasis'
same text: 3 of 5, same label after parse(): 5 of 5
--- model.generate with repetition_penalty=1.0
ours: 'Gastroesophageal reflux disease (GERD)'   generate: 'Gastroesophageal reflux disease (GERD)'
ours: 'Gastroesophageal reflux disease (GERD)'   generate: 'Gastroesophageal reflux disease (GERD)'
ours: 'Migraine'                                 generate: 'Migraine'
ours: 'Pneumonia'                                generate: 'Pneumonia'
ours: 'psoriasis'                                generate: 'psoriasis'
same text: 5 of 5, same label after parse(): 5 of 5

The first block is a surprise: with the library's defaults, two of five answers differ, "reflux disease" against "Reflux Disease". Both methods are greedy, so where does the difference come from? model.generate reads Qwen's generation_config.json, and that file sets repetition_penalty to 1.1. The penalty lowers the score of every token that is already in the text. "reflux disease" in lower case is already in the prompt, so the library pushed the model towards capital letters.

With the penalty set to 1.0, both methods give the same text on all five prompts. The lesson is useful for any harness: a library call can carry settings that you never wrote. Note also that the model answers "Gastroesophageal reflux disease (GERD)", which is not exactly a label. parse maps it to "gastroesophageal reflux disease"; an answer that matches no label counts as wrong.

The results

run (in the script) sends every example of a split through a harness. It stores the raw answers and the parsed label, and counts the characters of every prompt. "Extra chars" is the prompt length minus the zero-shot prompt length for the same description, as in the paper's Ctx column. The last row uses no model at all: it takes the most common label among the 5 nearest memory examples.

plain text
model: Qwen/Qwen2.5-1.5B-Instruct, greedy decoding, device mps
harness                        search acc  test acc  calls  extra chars  no label
zero-shot                           36.0%     40.6%      1            0        20
few-shot (8)                        34.0%     40.1%      1        1,336         4
draft-verification                  84.0%     86.8%      2        3,553         0
label-primed query                  76.0%     79.7%      1        5,504         0
TF-IDF 5-NN vote (no model)         90.0%     89.6%      0            0         0
check with scikit-learn accuracy_score (label-primed query, test): 79.7%
one standard error of a test accuracy near 80% on 212 examples: about 2.8 points

Read the test column first. Zero-shot gets 40.6%, and 20 of its 212 answers match no label at all. Few-shot with 8 random examples gets 40.1%, no better. Eight random examples show at most 8 of the 22 diseases, and they are rarely close to the patient. The two discovered designs more than double the score: 86.8% for draft-verification and 79.7% for label-primed query. Both retrieve close cases from the memory, and that is the step that matters here.

How large is the noise? With 212 test examples, one standard error is about 2.8 points. So the 7.1-point gap between draft-verification and label-primed query is larger than the noise. The 0.5-point gap between zero-shot and few-shot is not.

The last row is the honest surprise. A vote of the 5 nearest cases, with no model at all, gets 89.6% on test and 90.0% on search. It beats every harness that calls the model. On this dataset, the words of a description almost give away the disease, so close cases nearly always share its label. Our reading is that the small model mostly copies its retrieved neighbours, and sometimes copies them badly. The paper does not report a no-model retrieval baseline, so we cannot say whether the same holds for GPT-OSS-120B.

The ranking of the two discovered harnesses is the opposite of the paper's. In Table 9, Label-Primed Query is the better of the two on Symptom2Disease (86.8 against 85.4). Our guess, which we did not test: a 1.5B model cannot use a prompt of 30 examples as well as a 120B model can. Our 86.8% for draft-verification equals the paper's Label-Primed Query number only by chance: the model, the split and the harness are all different.

The per-disease picture shows where retrieval helps. Zero-shot gets 0% on six diseases, among them malaria, peptic ulcer and dengue; label-primed query gets 50% or more on all six. Only bronchial asthma drops, from 90% to 70%.

plain text
disease                           n  zero-shot  label-primed
malaria                          10         0%          100%
peptic ulcer disease             10         0%           90%
drug reaction                     8         0%           75%
jaundice                          7        14%           86%
allergy                          10        30%           90%
diabetes                         10        20%           80%
hypertension                     10         0%           60%
fungal infection                  9         0%           56%
dengue                           10         0%           50%
arthritis                        10        60%          100%
pneumonia                        10        50%           90%
varicose veins                   10        60%          100%
typhoid                           9        11%           44%
chicken pox                      10        20%           50%
common cold                      10        60%           80%
psoriasis                        10        80%          100%
urinary tract infection           9        67%           78%
cervical spondylosis             10        70%           80%
gastroesophageal reflux disease  10        80%           90%
impetigo                         10        80%           90%
migraine                         10        80%           90%
bronchial asthma                 10        90%           70%
test accuracy per disease (%)zero-shotlabel-primed query050100malariamalaria: zero-shot 0%, label-primed query 100%peptic ulcer diseasepeptic ulcer disease: zero-shot 0%, label-primed query 90%drug reactiondrug reaction: zero-shot 0%, label-primed query 75%jaundicejaundice: zero-shot 14%, label-primed query 86%diabetesdiabetes: zero-shot 20%, label-primed query 80%hypertensionhypertension: zero-shot 0%, label-primed query 60%allergyallergy: zero-shot 30%, label-primed query 90%fungal infectionfungal infection: zero-shot 0%, label-primed query 56%denguedengue: zero-shot 0%, label-primed query 50%pneumoniapneumonia: zero-shot 50%, label-primed query 90%arthritisarthritis: zero-shot 60%, label-primed query 100%varicose veinsvaricose veins: zero-shot 60%, label-primed query 100%typhoidtyphoid: zero-shot 11%, label-primed query 44%chicken poxchicken pox: zero-shot 20%, label-primed query 50%psoriasispsoriasis: zero-shot 80%, label-primed query 100%common coldcommon cold: zero-shot 60%, label-primed query 80%urinary tract infectionurinary tract infection: zero-shot 67%, label-primed query 78%cervical spondylosiscervical spondylosis: zero-shot 70%, label-primed query 80%gastroesophageal reflux diseasegastroesophageal reflux disease: zero-shot 80%, label-primed query 90%migrainemigraine: zero-shot 80%, label-primed query 90%impetigoimpetigo: zero-shot 80%, label-primed query 90%bronchial asthmabronchial asthma: zero-shot 90%, label-primed query 70%
Test accuracy for each disease, zero-shot (grey) against label-primed query (teal), sorted by the gain. Our run, Qwen2.5-1.5B-Instruct, 212 test examples, about 10 per disease.

What the verification call changed

plain text
search: draft accuracy 85.0%, final 84.0%; kept 99 of 100, changed 1: fixed 0, broke 1
test: draft accuracy 87.7%, final 86.8%; kept 210 of 212, changed 2: fixed 0, broke 2
broke: "I've been scratching myself a lot lately, and my skin is covered in rashy places. I also have a few ..."
  draft: fungal infection   final: psoriasis   true: fungal infection

The verification call almost never changes anything. On the test split it kept the draft 210 times of 212. Both changes went from a right answer to a wrong one, so the draft alone (87.7%) is better than the final answer (86.8%). The search split shows the same: one change, from right to wrong.

The prompt above shows why this is a model problem, not a retrieval problem. The challengers are exactly the right evidence: the closest cases with another label are peptic ulcer cases, and peptic ulcer is the right answer. A model that can weigh evidence would switch. Our 1.5B model repeats its first answer. Draft-verification is a design for a model that can check its own work. A small model does not get that benefit, but still pays for a second call.

Our Pareto frontier, two ways

Method 1 sorts the harnesses by context, cheapest first, and keeps a harness only if it is more accurate than every cheaper one. Method 2 is the definition itself: drop a harness if another one is at least as good on both goals and better on one. Both must give the same frontier.

python
def pareto_sweep(points):
    """Method 1: sort by context (small first); a point is on the frontier if it beats every cheaper point."""
    best, front = -1.0, []
    for name, ctx, acc in sorted(points, key=lambda p: (p[1], -p[2])):
        if acc > best:
            front.append(name); best = acc
    return front


def pareto_brute(points):
    """Method 2: the definition itself. Drop a point if another point is at least as good on both and better on one."""
    def dominates(p, q):
        return p[1] <= q[1] and p[2] >= q[2] and (p[1] < q[1] or p[2] > q[2])
    return [q[0] for q in points if not any(dominates(p, q) for p in points)]
plain text
zero-shot              extra context       0 chars   test accuracy 40.6%
few-shot (8)           extra context   1,336 chars   test accuracy 40.1%
draft-verification     extra context   3,553 chars   test accuracy 86.8%
label-primed query     extra context   5,504 chars   test accuracy 79.7%
Method 1 (sweep):       frontier = ['zero-shot', 'draft-verification']
Method 2 (definition):  frontier = ['zero-shot', 'draft-verification']
same frontier: True
3040506070809001,0002,0003,0004,0005,0006,000zero-shot: 0.0, 40.6zero-shot (40.6)few-shot (8): 1,336.0, 40.1few-shot (8) (40.1)draft-verification: 3,552.7, 86.8draft-verification (86.8)label-primed query: 5,503.6, 79.7label-primed query (79.7)extra characters in the prompt, compared with zero-shot (average per test example)test accuracy (%)
Our four model harnesses on the Symptom2Disease test split. Right means more extra prompt characters; up means higher test accuracy. The dashed line joins the Pareto frontier.

Both methods give the same frontier: zero-shot and draft-verification. Draft-verification dominates label-primed query: it uses less context and is more accurate. Zero-shot dominates few-shot (8), although the 0.5-point gap is inside the noise. The 5-NN vote is left out because it never calls the model. With no extra prompt and 89.6%, it would dominate every point on this chart.

Terminal output of mh_part3.py: the evaluation table of the five harnesses on the search and test splits

What a small model cannot show

Our run checks the two harness designs, not the paper's claims. Several things are out of its reach:

  • The model. Qwen2.5-1.5B is about 80 times smaller than GPT-OSS-120B. The verification step needs a model that can change its mind on evidence, and ours does not.
  • The data. We use one of the three datasets, from a public copy on Hugging Face. The authors' repository ships its own split, which we did not use. LawBench and USPTO are much harder and have 180 to 215 classes.
  • The search. We wrote both harnesses by hand from the paper's description. No proposer read traces, so Table 3 (traces against summaries) and Table 4 (other optimizers) cannot be tested here. Part 6 builds a small search loop.
  • The online memory. Our memory is full before the test. The paper's harnesses can also learn from their own mistakes as examples arrive; ours only store and retrieve.
  • The noise. 212 test examples give a standard error of about 2.8 points, so gaps smaller than about 5 points can be chance.

Search set against test set: Figure 7

3040506070809010030405060708090100y = xzero-shot: 36, 40.6zero-shot (40.6)few-shot (8): 34, 40.1few-shot (8) (40.1)draft-verification: 84, 86.8draft-verification (86.8)label-primed query: 76, 79.7label-primed query (79.7)5-NN vote, no model: 90, 89.65-NN vote, no model (89.6)search accuracy (%), 100 held-out training examplestest accuracy (%), 212 test examples
Our five harnesses: search accuracy (100 held-out training examples) across, test accuracy (212 test examples) up. The dashed line is y = x.

Our five harnesses sit on or near the diagonal. Each scores about the same, or a little better, on the test split than on the search split. The order is also the same on both splits: 5-NN vote, then draft-verification, label-primed query, zero-shot and few-shot. That is the property a search needs, because the search set then gives a fair preview of the test set. We did not search, so we could not overfit; in a real search, the gap between the two scores is the first thing to watch.

The paper and the code

Paper (arXiv 2603.28052) · Project page · Our scripts