Meta-Harness, explained · Part 3 of 6 · Covers §4.1, Tables 2 to 5, Figures 3 to 7, Table 9, Appendix B.1
Text Classification: What the Search Found
The first test of Meta-Harness: a model sorts short texts into classes, and the search writes the code that picks its examples. We read the tables, the accuracy and context trade-off, and the two discovered harnesses, then rebuild both around a small local model.
Meta-Harness: End-to-End Optimization of Model Harnesses. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn. COLM 2026, 2026. arXiv:2603.28052
A patient writes "I have a burning sensation in my stomach, and it is worse when I lie down". Which disease is it? A language model can guess, but its guess depends a lot on which solved cases you show it first. Section 4.1 lets Meta-Harness write the code that picks those cases. Each piece below shows the exact lines of the paper, then explains them. At the end, we rebuild the two harnesses it found and run them on a laptop.
The task: online text classification
The baselines and Table 2
Two goals at once: the Pareto frontier
Before Figure 3, we need one idea from economics. It answers the question "which of these options is best?" when there are two goals that pull in different directions.
A small example makes it concrete. Take four harnesses, each written as (extra context, accuracy): A = (0, 30%), B = (2,000, 35%), C = (3,000, 33%), D = (5,000, 45%). C costs more than B and is less accurate, so B dominates C, and C is off the frontier. A, B and D each win on something: A is the cheapest, D is the most accurate, and B sits between them. So the frontier is A, B and D. No single winner exists; the frontier is the menu of good trade-offs, and the user picks from it.
Against other optimizers
Table 4 sits beside the ACE paragraph, in the first screenshot of the Table 2 section. It gives the same race as numbers, on the search set: the median and best harness of each optimizer. Meta-Harness has a median of 50.0 and a best of 56.7. The best rival is TTT-Discover with a best of 45.6, and GEPA is lowest with 40.2.
What the proposer must see: Table 3
New datasets: Table 5
The two harnesses it found
Retrieval: finding the closest solved cases
Both harnesses need one tool: given a new description, find the most similar descriptions in the memory. The paper's Label-Primed Query uses TF-IDF for this, so we use TF-IDF in both.
The IDF of a word is
Read it from left to right. is the number of texts in the memory. , the document frequency, is the number of texts that contain the word . The fraction is large when few texts contain the word. The logarithm shrinks large ratios, and the terms stop a division by zero and keep every weight above zero. This is the exact form that scikit-learn uses by default.
A tiny example: with texts, a word in 1 text gets . A word in all 751 texts gets . Our real memory gives these values:
vocabulary: ours 988 words, scikit-learn 988 words
largest difference between the two matrices: 1.11e-16
word 'and' in 728 of 751 texts, idf = 1.031
word 'stomach' in 69 of 751 texts, idf = 3.374
word 'burning' in 25 of 751 texts, idf = 4.365
word 'heartburn' in 20 of 751 texts, idf = 4.578"and" is in most texts and gets a weight near 1. "heartburn" is in few texts and gets a weight about four times larger. So two descriptions that share "heartburn" count as much more similar than two that share "and".
Method 1, from scratch. The functions below build the vocabulary, count document frequencies, compute the IDF, and make one unit-length row per text. Memory.top returns the indices of the k most similar memory texts; its keep argument lets a harness ask for "the closest texts with this label" or "with any other label".
def tokenize(text):
return TOKEN.findall(text.lower())
def tfidf_fit(docs):
"""Method 1: TF-IDF from scratch. Returns the vocabulary, the idf weights and the document matrix."""
vocab = {w: i for i, w in enumerate(sorted({w for d in docs for w in tokenize(d)}))}
df = np.zeros(len(vocab)) # in how many documents each word appears
for d in docs:
for w in set(tokenize(d)):
df[vocab[w]] += 1
n = len(docs)
idf = np.log((1 + n) / (1 + df)) + 1 # rare words get a large weight, common words a small one
return vocab, idf, tfidf_transform(docs, vocab, idf)
def tfidf_transform(docs, vocab, idf):
X = np.zeros((len(docs), len(vocab))) # one row per document, one column per word
for r, d in enumerate(docs):
for w in tokenize(d):
if w in vocab:
X[r, vocab[w]] += 1 # tf: how often the word appears in this document
X *= idf # tf x idf
norms = np.linalg.norm(X, axis=1, keepdims=True)
return X / np.where(norms == 0, 1, norms)
class Memory:
"""The labeled examples a harness has seen, with TF-IDF retrieval over their texts."""
def __init__(self, examples):
self.texts = [t for t, _ in examples]
self.labels = [y for _, y in examples]
self.vocab, self.idf, self.X = tfidf_fit(self.texts)
def sims(self, query):
q = tfidf_transform([query], self.vocab, self.idf)[0]
return self.X @ q # cosine similarity of the query with every memory text
def top(self, query, k, keep=lambda i: True):
order = np.argsort(-self.sims(query), kind='stable')
return [int(i) for i in order if keep(i)][:k]Method 2, the library. scikit-learn's TfidfVectorizer does the same in one line. The script fits both on the 751 memory texts, compares the two matrices, and checks that every test query gets the same five neighbours:
T = Log('part3_tfidf')
vec = TfidfVectorizer()
Xs = vec.fit_transform(mem.texts).toarray() # Method 2: the library
cols = [vec.vocabulary_[w] for w in sorted(mem.vocab, key=mem.vocab.get)]
T(f'vocabulary: ours {len(mem.vocab)} words, scikit-learn {len(vec.vocabulary_)} words')
T(f'largest difference between the two matrices: {np.abs(mem.X - Xs[:, cols]).max():.2e}')
for w in ['and', 'stomach', 'burning', 'heartburn']:
df = sum(w in set(tokenize(t)) for t in mem.texts)
T(f' word {w!r:<12} in {df:>3} of {len(mem.texts)} texts, idf = {mem.idf[mem.vocab[w]]:.3f}')
agree = 0
for q, _ in test:
ours = mem.top(q, 5)
lib = list(np.argsort(-(Xs @ vec.transform([q]).toarray()[0]), kind='stable')[:5])
agree += ours == [int(i) for i in lib]
T(f'test queries with the same top-5 neighbours: {agree} of {len(test)}')vocabulary: ours 988 words, scikit-learn 988 words
largest difference between the two matrices: 1.11e-16
word 'and' in 728 of 751 texts, idf = 1.031
word 'stomach' in 69 of 751 texts, idf = 3.374
word 'burning' in 25 of 751 texts, idf = 4.365
word 'heartburn' in 20 of 751 texts, idf = 4.578
test queries with the same top-5 neighbours: 212 of 212
query: "I have a burning sensation in my stomach that comes and goes. It's worse when I eat and wh..."
0.673 peptic ulcer disease "I have a burning sensation in my upper stomach that comes an..."
0.566 peptic ulcer disease "I have a burning feeling in my stomach. It's worse when I ea..."
0.561 gastroesophageal reflux disease "I have a burning sensation in my chest and throat. It feels ..."
0.505 gastroesophageal reflux disease "I have a burning sensation in my throat that comes and goes,..."
0.498 gastroesophageal reflux disease "I get heartburn and indigestion a lot. It's worse when I eat..."The two matrices agree to , the size of a rounding error, and all 212 test queries get the same top 5. The last lines show the retrieval for our first test description. Its two nearest cases are peptic ulcer, the right answer. The next three are acid reflux (gastroesophageal reflux disease), which has very similar symptoms. A majority vote of the five would pick the wrong disease, three to two. Keep this example in mind.
Draft Verification
The authors' public repository (stanford-iris-lab/meta-harness, folder reference_examples/text_classification) has the task, the data and the baseline harnesses. Every harness there is a class with two methods: predict(input) makes a label before the right answer is known, and learn_from_batch(batch_results) updates the memory afterwards. The two discovered files, draft_verification.py and label_primed_query_anchored.py, are not in it. So our version follows the description above and Figure 5, and our prompts and exact retrieval are our own reading. call is the model call: it takes a prompt and returns a parsed label.
def shots(mem, idx):
return '\n\n'.join(f'Description: {mem.texts[i]}\nDisease: {mem.labels[i]}' for i in idx)
def draft_verification(q, mem, labels, call, k=5):
"""Figure 5. Call 1 drafts a label from the 5 nearest examples. Call 2 shows 5 examples that agree with the draft
(confirmers) and 5 that do not (challengers), and keeps or revises the draft."""
if len(mem.texts) < k: # cold start: a single few-shot call
return few_shot(q, mem, labels, call, n=len(mem.texts))
near = mem.top(q, k)
p1 = (f'Classify the patient description as one of these diseases: {label_list(labels)}.\n\n'
f'Similar solved cases:\n\n{shots(mem, near)}\n\nDescription: {q}\n{ASK}')
draft = call(p1)
if draft is None:
return None, [p1]
confirm = mem.top(q, k, keep=lambda i: mem.labels[i] == draft) # the second retrieval depends on the draft
challenge = mem.top(q, k, keep=lambda i: mem.labels[i] != draft)
p2 = (f'Classify the patient description as one of these diseases: {label_list(labels)}.\n\n'
f'A first guess was: {draft}.\n\nCases WITH the label {draft}:\n\n{shots(mem, confirm)}\n\n'
f'Similar cases with OTHER labels:\n\n{shots(mem, challenge)}\n\n'
f'Description: {q}\nKeep the first guess or change it. {ASK}')
return call(p2), [p1, p2]The line with confirm = mem.top(q, k, keep=lambda i: mem.labels[i] == draft) is the "key discovered behavior": the second retrieval depends on the draft. Here is the second prompt for our first test description, as the script printed it. The draft was acid reflux, which is wrong:
Classify the patient description as one of these diseases: allergy, arthritis, bronchial asthma, cervical spondylosis, chicken pox, common cold, dengue, diabetes, drug reaction, fungal infection, gastroesophageal reflux disease, hypertension, impetigo, jaundice, malaria, migraine, peptic ulcer disease, pneumonia, psoriasis, typhoid, urinary tract infection, varicose veins.
A first guess was: gastroesophageal reflux disease.
Cases WITH the label gastroesophageal reflux disease:
Description: I have a burning sensation in my chest and throat. It feels like acid or food is coming up into my throat. It's worse when I lie down. I also have heartburn and indigestion after I eat, and I sometimes throw up.
Disease: gastroesophageal reflux disease
Description: I have a burning sensation in my throat that comes and goes, and it's worse after I eat. I also have a bitter or sour aftertaste in my mouth sometimes.
Disease: gastroesophageal reflux disease
[... 3 more cases with this label ...]
Similar cases with OTHER labels:
Description: I have a burning sensation in my upper stomach that comes and goes. I also have heartburn, indigestion, and nausea.
Disease: peptic ulcer disease
Description: I have a burning feeling in my stomach. It's worse when I eat or drink. I also get heartburn, indigestion, and nausea.
Disease: peptic ulcer disease
[... 3 more cases with other labels ...]
Description: I have a burning sensation in my stomach that comes and goes. It's worse when I eat and when I lie down. I also have heartburn and indigestion.
Keep the first guess or change it. Answer with the disease name only.
----- model answer: 'gastroesophageal reflux disease'Label-Primed Query
def label_primed_query(q, mem, labels, call, n_pairs=4, pool=20):
"""Figure 6. One prompt in three blocks: the label primer, one query-relevant example per label (coverage),
and pairs of close examples with different labels taken from the query's neighbourhood (contrastive pairs)."""
s = mem.sims(q)
coverage = []
for lab in labels: # the best example of every label
own = [i for i in range(len(mem.texts)) if mem.labels[i] == lab]
coverage.append(max(own, key=lambda i: s[i]))
near = [int(i) for i in np.argsort(-s, kind='stable')[:pool]] # the query's neighbourhood
pairs, used = [], set()
for a in near: # pair each close example with the closest one
if a in used: # that carries a different label
continue
b = next((j for j in near if j not in used and j != a and mem.labels[j] != mem.labels[a]), None)
if b is not None:
pairs.append((a, b)); used |= {a, b}
if len(pairs) == n_pairs:
break
pair_text = '\n\n'.join(f'Pair {n + 1}:\n{shots(mem, [a])}\n{shots(mem, [b])}' for n, (a, b) in enumerate(pairs))
p = (f'Valid labels: {label_list(labels)}.\n\n'
f'One example for each label:\n\n{shots(mem, coverage)}\n\n'
f'Close cases with different labels:\n\n{pair_text}\n\n'
f'Description: {q}\nClassify the description as one of the valid labels. {ASK}')
return call(p), [p]The coverage loop picks, for every label, the memory example closest to the query. The pairing loop walks down the 20 nearest examples and pairs each with the next close example that has a different label. Here is the prompt for our first test description, cut to its three blocks (the full prompt is 22 coverage examples and 4 pairs):
Valid labels: allergy, arthritis, bronchial asthma, cervical spondylosis, chicken pox, common cold, dengue, diabetes, drug reaction, fungal infection, gastroesophageal reflux disease, hypertension, impetigo, jaundice, malaria, migraine, peptic ulcer disease, pneumonia, psoriasis, typhoid, urinary tract infection, varicose veins.
One example for each label:
Description: I have a sore throat and a lot of sneezing. There are times when the skin around my eyes and my lips swell. I find that once I start, I just cannot stop.
Disease: allergy
Description: I have a lot of pain in my neck and my muscles feel weak. My joints are swollen and I have a lot of stiffness when I walk. Walking is also very uncomfortable.
Disease: arthritis
Description: I have a bad cough and I'm really tired. I have a high fever and my breathing is shallow. When I cough, a lot of saliva comes out.
Disease: bronchial asthma
[... 19 more examples, one per label ...]
Close cases with different labels:
Pair 1:
Description: I have a burning sensation in my upper stomach that comes and goes. I also have heartburn, indigestion, and nausea.
Disease: peptic ulcer disease
Description: I have a burning sensation in my chest and throat. It feels like acid or food is coming up into my throat. It's worse when I lie down. I also have heartburn and indigestion after I eat, and I sometimes throw up.
Disease: gastroesophageal reflux disease
Pair 2:
Description: I have a burning feeling in my stomach. It's worse when I eat or drink. I also get heartburn, indigestion, and nausea.
Disease: peptic ulcer disease
Description: I have a burning sensation in my throat that comes and goes, and it's worse after I eat. I also have a bitter or sour aftertaste in my mouth sometimes.
Disease: gastroesophageal reflux disease
[... 2 more pairs ...]
Description: I have a burning sensation in my stomach that comes and goes. It's worse when I eat and when I lie down. I also have heartburn and indigestion.
Classify the description as one of the valid labels. Answer with the disease name only.
----- model answer: 'gastroesophageal reflux disease'Our run: the two harnesses on a small model
Now we run everything. The task model is Qwen2.5-1.5B-Instruct, an open model with 1.5 billion parameters, on the laptop's Apple GPU. It stays frozen, as in the paper. The data is Symptom2Disease from Hugging Face (gretelai/symptom_to_diagnosis):
Symptom2Disease (gretelai/symptom_to_diagnosis): 22 labels
memory: 751 labeled examples the harness may read
search: 100 held-out training examples (to pick a harness)
test: 212 examples of the official test split, never used to choose anything
training texts dropped because the same text is in the test split: 2
labels: allergy, arthritis, bronchial asthma, cervical spondylosis, chicken pox, common cold, dengue, diabetes, drug reaction, fungal infection, gastroesophageal reflux disease, hypertension, impetigo, jaundice, malaria, migraine, peptic ulcer disease, pneumonia, psoriasis, typhoid, urinary tract infection, varicose veins
example: "I have a burning sensation in my stomach that comes and goes. It's worse when I eat and when I lie down. I also have heartburn and indigestion." -> peptic ulcer diseaseThe memory is the notebook: 751 solved cases that every harness may read. We fill it before the test, which is where the paper's online memory ends up after the last labeled example arrives. The search split is 100 more training cases that play the role of the search set. The test split is never used to choose anything. We do not run a search here (that is Part 6); we only compare fixed harnesses on both splits.
All our harnesses give the model the list of 22 diseases, even zero-shot. A model that does not know the names cannot match them. Even with the list, zero-shot gives 20 answers that match no label (see the results). Because every harness has the list, it is not counted as extra context.
One model call, two ways
Method 1 is our own loop: run the model, take the top token, feed it back, and stop at the end-of-turn token. It keeps the past keys and values (past_key_values), so each new step reads only one new token. Method 2 is the library's model.generate with sampling turned off. parse then maps the answer to one of the 22 labels.
class Model:
def __init__(self, name=MODEL, dtype='bfloat16'): # 16-bit numbers: half the memory of float32
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
self.torch = torch
self.dev = 'mps' if torch.backends.mps.is_available() else 'cpu'
self.tok = AutoTokenizer.from_pretrained(name)
self.lm = AutoModelForCausalLM.from_pretrained(name, dtype=getattr(torch, dtype)).to(self.dev).eval()
def ids(self, user):
msgs = [{'role': 'user', 'content': user}]
text = self.tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
return self.tok(text, return_tensors='pt').input_ids.to(self.dev)
def greedy(self, user, max_new=16):
"""Method 1: our own greedy decoding loop. Run the model, take the most likely next token, append, repeat."""
torch = self.torch
ids = self.ids(user)
out, past = [], None
with torch.no_grad():
x = ids
for _ in range(max_new):
r = self.lm(input_ids=x, past_key_values=past, use_cache=True)
past = r.past_key_values # keep the keys and values, so each step reads one token
nxt = int(r.logits[0, -1].argmax()) # the most likely next token
if nxt == self.tok.eos_token_id or nxt == self.tok.convert_tokens_to_ids('<|im_end|>'):
break
out.append(nxt)
x = torch.tensor([[nxt]], device=self.dev)
return self.tok.decode(out, skip_special_tokens=True)
def generate(self, user, max_new=16, repetition_penalty=1.0):
"""Method 2: the library way, model.generate with sampling turned off.
Qwen's generation_config.json also sets repetition_penalty=1.1; 1.0 turns it off, as in our loop."""
ids = self.ids(user)
with self.torch.no_grad():
g = self.lm.generate(ids, max_new_tokens=max_new, do_sample=False, top_p=None, top_k=None,
temperature=None, repetition_penalty=repetition_penalty,
pad_token_id=self.tok.eos_token_id)
return self.tok.decode(g[0, ids.shape[1]:], skip_special_tokens=True)
def parse(answer, labels):
"""Map the model's text to one valid label: an exact label inside the answer, else the closest spelling."""
a = answer.lower().strip()
hits = [l for l in labels if l in a]
if hits:
return max(hits, key=len)
close = difflib.get_close_matches(a.split('\n')[0], labels, n=1, cutoff=0.6)
return close[0] if close else None--- model.generate with repetition_penalty=1.1
ours: 'Gastroesophageal reflux disease (GERD)' generate: 'Gastroesophageal Reflux Disease (GERD)'
ours: 'Gastroesophageal reflux disease (GERD)' generate: 'Gastroesophageal Reflux Disease (GERD)'
ours: 'Migraine' generate: 'Migraine'
ours: 'Pneumonia' generate: 'Pneumonia'
ours: 'psoriasis' generate: 'psoriasis'
same text: 3 of 5, same label after parse(): 5 of 5
--- model.generate with repetition_penalty=1.0
ours: 'Gastroesophageal reflux disease (GERD)' generate: 'Gastroesophageal reflux disease (GERD)'
ours: 'Gastroesophageal reflux disease (GERD)' generate: 'Gastroesophageal reflux disease (GERD)'
ours: 'Migraine' generate: 'Migraine'
ours: 'Pneumonia' generate: 'Pneumonia'
ours: 'psoriasis' generate: 'psoriasis'
same text: 5 of 5, same label after parse(): 5 of 5The first block is a surprise: with the library's defaults, two of five answers differ, "reflux disease" against "Reflux Disease". Both methods are greedy, so where does the difference come from? model.generate reads Qwen's generation_config.json, and that file sets repetition_penalty to 1.1. The penalty lowers the score of every token that is already in the text. "reflux disease" in lower case is already in the prompt, so the library pushed the model towards capital letters.
With the penalty set to 1.0, both methods give the same text on all five prompts. The lesson is useful for any harness: a library call can carry settings that you never wrote. Note also that the model answers "Gastroesophageal reflux disease (GERD)", which is not exactly a label. parse maps it to "gastroesophageal reflux disease"; an answer that matches no label counts as wrong.
The results
run (in the script) sends every example of a split through a harness. It stores the raw answers and the parsed label, and counts the characters of every prompt. "Extra chars" is the prompt length minus the zero-shot prompt length for the same description, as in the paper's Ctx column. The last row uses no model at all: it takes the most common label among the 5 nearest memory examples.
model: Qwen/Qwen2.5-1.5B-Instruct, greedy decoding, device mps
harness search acc test acc calls extra chars no label
zero-shot 36.0% 40.6% 1 0 20
few-shot (8) 34.0% 40.1% 1 1,336 4
draft-verification 84.0% 86.8% 2 3,553 0
label-primed query 76.0% 79.7% 1 5,504 0
TF-IDF 5-NN vote (no model) 90.0% 89.6% 0 0 0
check with scikit-learn accuracy_score (label-primed query, test): 79.7%
one standard error of a test accuracy near 80% on 212 examples: about 2.8 pointsRead the test column first. Zero-shot gets 40.6%, and 20 of its 212 answers match no label at all. Few-shot with 8 random examples gets 40.1%, no better. Eight random examples show at most 8 of the 22 diseases, and they are rarely close to the patient. The two discovered designs more than double the score: 86.8% for draft-verification and 79.7% for label-primed query. Both retrieve close cases from the memory, and that is the step that matters here.
How large is the noise? With 212 test examples, one standard error is about 2.8 points. So the 7.1-point gap between draft-verification and label-primed query is larger than the noise. The 0.5-point gap between zero-shot and few-shot is not.
The last row is the honest surprise. A vote of the 5 nearest cases, with no model at all, gets 89.6% on test and 90.0% on search. It beats every harness that calls the model. On this dataset, the words of a description almost give away the disease, so close cases nearly always share its label. Our reading is that the small model mostly copies its retrieved neighbours, and sometimes copies them badly. The paper does not report a no-model retrieval baseline, so we cannot say whether the same holds for GPT-OSS-120B.
The ranking of the two discovered harnesses is the opposite of the paper's. In Table 9, Label-Primed Query is the better of the two on Symptom2Disease (86.8 against 85.4). Our guess, which we did not test: a 1.5B model cannot use a prompt of 30 examples as well as a 120B model can. Our 86.8% for draft-verification equals the paper's Label-Primed Query number only by chance: the model, the split and the harness are all different.
The per-disease picture shows where retrieval helps. Zero-shot gets 0% on six diseases, among them malaria, peptic ulcer and dengue; label-primed query gets 50% or more on all six. Only bronchial asthma drops, from 90% to 70%.
disease n zero-shot label-primed
malaria 10 0% 100%
peptic ulcer disease 10 0% 90%
drug reaction 8 0% 75%
jaundice 7 14% 86%
allergy 10 30% 90%
diabetes 10 20% 80%
hypertension 10 0% 60%
fungal infection 9 0% 56%
dengue 10 0% 50%
arthritis 10 60% 100%
pneumonia 10 50% 90%
varicose veins 10 60% 100%
typhoid 9 11% 44%
chicken pox 10 20% 50%
common cold 10 60% 80%
psoriasis 10 80% 100%
urinary tract infection 9 67% 78%
cervical spondylosis 10 70% 80%
gastroesophageal reflux disease 10 80% 90%
impetigo 10 80% 90%
migraine 10 80% 90%
bronchial asthma 10 90% 70%What the verification call changed
search: draft accuracy 85.0%, final 84.0%; kept 99 of 100, changed 1: fixed 0, broke 1
test: draft accuracy 87.7%, final 86.8%; kept 210 of 212, changed 2: fixed 0, broke 2
broke: "I've been scratching myself a lot lately, and my skin is covered in rashy places. I also have a few ..."
draft: fungal infection final: psoriasis true: fungal infectionThe verification call almost never changes anything. On the test split it kept the draft 210 times of 212. Both changes went from a right answer to a wrong one, so the draft alone (87.7%) is better than the final answer (86.8%). The search split shows the same: one change, from right to wrong.
The prompt above shows why this is a model problem, not a retrieval problem. The challengers are exactly the right evidence: the closest cases with another label are peptic ulcer cases, and peptic ulcer is the right answer. A model that can weigh evidence would switch. Our 1.5B model repeats its first answer. Draft-verification is a design for a model that can check its own work. A small model does not get that benefit, but still pays for a second call.
Our Pareto frontier, two ways
Method 1 sorts the harnesses by context, cheapest first, and keeps a harness only if it is more accurate than every cheaper one. Method 2 is the definition itself: drop a harness if another one is at least as good on both goals and better on one. Both must give the same frontier.
def pareto_sweep(points):
"""Method 1: sort by context (small first); a point is on the frontier if it beats every cheaper point."""
best, front = -1.0, []
for name, ctx, acc in sorted(points, key=lambda p: (p[1], -p[2])):
if acc > best:
front.append(name); best = acc
return front
def pareto_brute(points):
"""Method 2: the definition itself. Drop a point if another point is at least as good on both and better on one."""
def dominates(p, q):
return p[1] <= q[1] and p[2] >= q[2] and (p[1] < q[1] or p[2] > q[2])
return [q[0] for q in points if not any(dominates(p, q) for p in points)]zero-shot extra context 0 chars test accuracy 40.6%
few-shot (8) extra context 1,336 chars test accuracy 40.1%
draft-verification extra context 3,553 chars test accuracy 86.8%
label-primed query extra context 5,504 chars test accuracy 79.7%
Method 1 (sweep): frontier = ['zero-shot', 'draft-verification']
Method 2 (definition): frontier = ['zero-shot', 'draft-verification']
same frontier: TrueBoth methods give the same frontier: zero-shot and draft-verification. Draft-verification dominates label-primed query: it uses less context and is more accurate. Zero-shot dominates few-shot (8), although the 0.5-point gap is inside the noise. The 5-NN vote is left out because it never calls the model. With no extra prompt and 89.6%, it would dominate every point on this chart.
What a small model cannot show
Our run checks the two harness designs, not the paper's claims. Several things are out of its reach:
- The model. Qwen2.5-1.5B is about 80 times smaller than GPT-OSS-120B. The verification step needs a model that can change its mind on evidence, and ours does not.
- The data. We use one of the three datasets, from a public copy on Hugging Face. The authors' repository ships its own split, which we did not use. LawBench and USPTO are much harder and have 180 to 215 classes.
- The search. We wrote both harnesses by hand from the paper's description. No proposer read traces, so Table 3 (traces against summaries) and Table 4 (other optimizers) cannot be tested here. Part 6 builds a small search loop.
- The online memory. Our memory is full before the test. The paper's harnesses can also learn from their own mistakes as examples arrive; ours only store and retrieve.
- The noise. 212 test examples give a standard error of about 2.8 points, so gaps smaller than about 5 points can be chance.
Search set against test set: Figure 7
Our five harnesses sit on or near the diagonal. Each scores about the same, or a little better, on the test split than on the search split. The order is also the same on both splits: 5-NN vote, then draft-verification, label-primed query, zero-shot and few-shot. That is the property a search needs, because the search set then gives a fair preview of the test set. We did not search, so we could not overfit; in a real search, the gap between the two scores is the first thing to watch.
