# Chapter 3 · Reasoning patterns: how one agent thinks, acts and corrects itself

Author: Ishwar Jangid
Canonical URL: https://ishwarj.com/books/agentic-systems/03-reasoning-patterns/
Updated: 2026-10-08
Source: https://github.com/ishwar6/ishwar-books/blob/main/content/books/agentic-systems/part2-patterns/03-reasoning-patterns.md


> **Goal:** by the end of this chapter you can take one task, try it with several loop shapes (think, act, plan, check, choose among candidates), say what each shape changes, what it costs and how it fails, and decide with a measurement which one to keep. You will read the key papers behind each pattern, see three small experiments run against a real model, and have one complete modern tool loop you can copy.

---

## 3.1 One task, many loop shapes

Chapter 1 drew the agent as a loop and Chapter 2 took the loop apart into blocks. This chapter keeps the blocks and changes one thing: the **shape of the loop**. How many times is the model called? Does it write its reasoning down? Does it look things up as it goes, or plan everything first? Does anything check the answer? Does it try several answers and pick one?

We will follow one example the whole way through.

> [!DEFINITION] Northwind Freight (the running example)
> A fictional logistics company invented for this chapter, so that no model can know its facts from training. It has a small encyclopaedia of 21 entries: the company (founded in 2011 in Rotterdam), its chief executive, a rail company it bought, four depots (Rotterdam, Gdansk, Porto, Leeds, each with an opening year, a truck count and a manager) and a few managers. The same encyclopaedia also holds twelve entries of public facts (the Eiffel Tower, Mount Everest and so on). Two tools read it: a look-up by name and a calculator.

Two questions come back again and again:

1. *How many years after the Porto depot opened did the Leeds depot open?* The answer is 3 (2020 minus 2017). It needs two look-ups and a subtraction.
2. *Which depot has the most trucks, and how many more does it have than the Leeds depot?* The answer is Rotterdam, with 79 more (120 minus 41). It needs four independent look-ups, a comparison and a subtraction.

Here is what each shape does with the first question, before any theory:

| Shape | What the system does with the Porto and Leeds question | What happened in our runs (gpt-5-mini) |
|---|---|---|
| Direct answer | one call, answer only | "6 years", "9 years", "5": invented every time |
| Chain of thought | one call, write the steps first | two of three runs said "I don't have the opening years"; one said "2 years" |
| ReAct loop | think, look up, observe, repeat | right in one run of three; two runs went round in circles until the step budget ran out |
| Loop + guard + better tool | the same loop, with two engineering fixes | right in three runs of three |
| Plan first | write all look-ups, run them, answer once | not run on this question; Section 3.4 counts its cost |
| Check the answer | recompute from the evidence before replying | catches a wrong number before the user sees it (Section 3.5) |

The rest of the chapter explains each row: the idea, the paper that introduced it, what it costs, how it fails and what to measure.

> [!DEFINITION] Reasoning pattern
> A fixed way of arranging model calls, tool calls and checks to turn a task into an answer: how many calls, in what order, what each one sees, and what ends the loop. Chain of thought, ReAct, plan-and-execute, reflection and search are the families in this chapter. A pattern is a design choice in your code; the model fills in the content.

<figure class="fig"><svg viewBox="0 0 760 412" role="img" aria-label="Six shapes of the agent loop with the same model and tools: think only (chain of thought), act only, interleave (ReAct), plan first, reflect (try, check, critique, retry) and search (branch, score, prune)."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="380.0" y="22.0" text-anchor="middle">Six shapes of the loop, same model and same tools</text><rect class="box-ghost" x="24.0" y="44.0" width="229.0" height="152.0" rx="12"/><text class="t-note" x="38.0" y="64.0" text-anchor="start">think only</text><text class="t-muted" x="38.0" y="81.0" text-anchor="start">chain of thought</text><rect class="box" x="42.5" y="122.0" width="40.0" height="28.0" rx="6"/><text class="t-tick" x="62.5" y="140.0" text-anchor="middle">q</text><line class="edge" x1="82.5" y1="136.0" x2="98.5" y2="136.0" marker-end="url(#ah)"/><rect class="box-2" x="98.5" y="122.0" width="76.0" height="28.0" rx="6"/><text class="t-tick" x="136.5" y="140.0" text-anchor="middle">thoughts</text><line class="edge" x1="174.5" y1="136.0" x2="190.5" y2="136.0" marker-end="url(#ah)"/><rect class="box-on" x="190.5" y="122.0" width="44.0" height="28.0" rx="6"/><text class="t-tick" x="212.5" y="140.0" text-anchor="middle">ans</text><rect class="box-ghost" x="267.0" y="44.0" width="229.0" height="152.0" rx="12"/><text class="t-note" x="281.0" y="64.0" text-anchor="start">act only</text><text class="t-muted" x="281.0" y="81.0" text-anchor="start">tool calls, no thoughts</text><rect class="box-3" x="281.5" y="122.0" width="44.0" height="28.0" rx="6"/><text class="t-tick" x="303.5" y="140.0" text-anchor="middle">call</text><line class="edge" x1="325.5" y1="136.0" x2="341.5" y2="136.0" marker-end="url(#ah)"/><rect class="box-3" x="341.5" y="122.0" width="44.0" height="28.0" rx="6"/><text class="t-tick" x="363.5" y="140.0" text-anchor="middle">call</text><line class="edge" x1="385.5" y1="136.0" x2="401.5" y2="136.0" marker-end="url(#ah)"/><rect class="box-3" x="401.5" y="122.0" width="44.0" height="28.0" rx="6"/><text class="t-tick" x="423.5" y="140.0" text-anchor="middle">call</text><line class="edge" x1="445.5" y1="136.0" x2="459.5" y2="136.0" marker-end="url(#ah)"/><text class="t-tick" x="463.5" y="140.0" text-anchor="start">ans</text><rect class="box-ghost" x="510.0" y="44.0" width="229.0" height="152.0" rx="12"/><text class="t-note" x="524.0" y="64.0" text-anchor="start">interleave</text><text class="t-muted" x="524.0" y="81.0" text-anchor="start">ReAct: think, act, observe</text><rect class="box-2" x="520.5" y="122.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="550.5" y="140.0" text-anchor="middle">think</text><line class="edge" x1="580.5" y1="136.0" x2="592.5" y2="136.0" marker-end="url(#ah)"/><rect class="box-3" x="592.5" y="122.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="622.5" y="140.0" text-anchor="middle">act</text><line class="edge" x1="652.5" y1="136.0" x2="664.5" y2="136.0" marker-end="url(#ah)"/><rect class="box" x="664.5" y="122.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="694.5" y="140.0" text-anchor="middle">observe</text><path class="path" d="M694.5,150.0 L694.5,170.0 L550.5,170.0 L550.5,152.0" marker-end="url(#ah-on)"/><text class="t-muted" x="624.5" y="186.0" text-anchor="middle">repeat until done</text><rect class="box-ghost" x="24.0" y="210.0" width="229.0" height="152.0" rx="12"/><text class="t-note" x="38.0" y="230.0" text-anchor="start">plan first</text><text class="t-muted" x="38.0" y="247.0" text-anchor="start">plan, execute, solve</text><rect class="box-4" x="30.5" y="288.0" width="48.0" height="28.0" rx="6"/><text class="t-tick" x="54.5" y="306.0" text-anchor="middle">plan</text><line class="edge" x1="78.5" y1="302.0" x2="102.5" y2="279.0" marker-end="url(#ah)"/><rect class="box-3" x="102.5" y="268.0" width="54.0" height="22.0" rx="5"/><text class="t-tick" x="129.5" y="283.0" text-anchor="middle">step 1</text><line class="edge" x1="156.5" y1="279.0" x2="180.5" y2="302.0" marker-end="url(#ah)"/><line class="edge" x1="78.5" y1="302.0" x2="102.5" y2="305.0" marker-end="url(#ah)"/><rect class="box-3" x="102.5" y="294.0" width="54.0" height="22.0" rx="5"/><text class="t-tick" x="129.5" y="309.0" text-anchor="middle">step 2</text><line class="edge" x1="156.5" y1="305.0" x2="180.5" y2="302.0" marker-end="url(#ah)"/><line class="edge" x1="78.5" y1="302.0" x2="102.5" y2="331.0" marker-end="url(#ah)"/><rect class="box-3" x="102.5" y="320.0" width="54.0" height="22.0" rx="5"/><text class="t-tick" x="129.5" y="335.0" text-anchor="middle">step 3</text><line class="edge" x1="156.5" y1="331.0" x2="180.5" y2="302.0" marker-end="url(#ah)"/><rect class="box-on" x="180.5" y="288.0" width="56.0" height="28.0" rx="6"/><text class="t-tick" x="208.5" y="306.0" text-anchor="middle">solve</text><rect class="box-ghost" x="267.0" y="210.0" width="229.0" height="152.0" rx="12"/><text class="t-note" x="281.0" y="230.0" text-anchor="start">reflect</text><text class="t-muted" x="281.0" y="247.0" text-anchor="start">try, check, critique, retry</text><rect class="box-2" x="273.5" y="288.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="303.5" y="306.0" text-anchor="middle">attempt</text><line class="edge" x1="333.5" y1="302.0" x2="347.5" y2="302.0" marker-end="url(#ah)"/><rect class="box-3" x="347.5" y="288.0" width="52.0" height="28.0" rx="6"/><text class="t-tick" x="373.5" y="306.0" text-anchor="middle">check</text><line class="edge" x1="399.5" y1="302.0" x2="413.5" y2="302.0" marker-end="url(#ah)"/><rect class="box-1" x="413.5" y="288.0" width="66.0" height="28.0" rx="6"/><text class="t-tick" x="446.5" y="306.0" text-anchor="middle">critique</text><path class="path" d="M446.5,316.0 L446.5,336.0 L303.5,336.0 L303.5,318.0" marker-end="url(#ah-on)"/><text class="t-muted" x="381.5" y="352.0" text-anchor="middle">retry with the critique</text><rect class="box-ghost" x="510.0" y="210.0" width="229.0" height="152.0" rx="12"/><text class="t-note" x="524.0" y="230.0" text-anchor="start">search</text><text class="t-muted" x="524.0" y="247.0" text-anchor="start">branch, score, prune</text><circle class="node on" cx="604.5" cy="266.0" r="7"/><line class="edge" x1="604.5" y1="273.0" x2="544.5" y2="296.0"/><circle class="node" cx="544.5" cy="302.0" r="7"/><line class="edge-dim" x1="544.5" y1="309.0" x2="526.5" y2="332.0"/><circle class="node" cx="526.5" cy="338.0" r="6"/><line class="edge-dim" x1="544.5" y1="309.0" x2="562.5" y2="332.0"/><circle class="node" cx="562.5" cy="338.0" r="6"/><line class="edge" x1="604.5" y1="273.0" x2="604.5" y2="296.0"/><circle class="node on" cx="604.5" cy="302.0" r="7"/><line class="edge-dim" x1="604.5" y1="309.0" x2="586.5" y2="332.0"/><circle class="node" cx="586.5" cy="338.0" r="6"/><line class="edge" x1="604.5" y1="309.0" x2="622.5" y2="332.0"/><circle class="node on" cx="622.5" cy="338.0" r="6"/><line class="edge" x1="604.5" y1="273.0" x2="664.5" y2="296.0"/><circle class="node" cx="664.5" cy="302.0" r="7"/><line class="edge-dim" x1="664.5" y1="309.0" x2="646.5" y2="332.0"/><circle class="node" cx="646.5" cy="338.0" r="6"/><line class="edge-dim" x1="664.5" y1="309.0" x2="682.5" y2="332.0"/><circle class="node" cx="682.5" cy="338.0" r="6"/><text class="t-muted" x="732.5" y="300.0" text-anchor="end">score,</text><text class="t-muted" x="732.5" y="317.0" text-anchor="end">keep best</text><text class="t-muted" x="380.0" y="384.0" text-anchor="middle">the shapes are not a sequence: a real agent combines them (a plan over a loop, a check after it).</text><text class="t-muted" x="380.0" y="401.0" text-anchor="middle">Each one costs calls, tokens and seconds; this chapter asks when each one pays for itself.</text></svg><figcaption>Six shapes of the loop with the same model and tools: think only, act only, interleave (ReAct), plan first, reflect, and search.</figcaption></figure>

### The shapes are choices on five separate dimensions

It is tempting to read the six shapes as a ladder, each one "more advanced" than the last. That is the wrong picture. Each pattern changes one or two separate design choices, and a real agent picks an option on each.

<figure class="fig"><svg viewBox="0 0 760 360" role="img" aria-label="Five design dimensions of a single agent: reasoning inside one call, control across calls, action representation, verification and selection, and context management. A pattern is a choice in one or more rows, and choices in different rows combine."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="380.0" y="22.0" text-anchor="middle">Five design dimensions: each agent picks one or more options per row</text><rect class="box-ghost" x="16.0" y="42.0" width="184.0" height="40.0" rx="8"/><text class="t-note" x="108.0" y="67.0" text-anchor="middle">reasoning in one call</text><rect class="box" x="214.0" y="46.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="278.0" y="66.5" text-anchor="middle">direct answer</text><rect class="box" x="348.0" y="46.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="412.0" y="66.5" text-anchor="middle">chain of thought</text><rect class="box-on" x="482.0" y="46.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="546.0" y="66.5" text-anchor="middle">reasoning model</text><rect class="box" x="616.0" y="46.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="680.0" y="66.5" text-anchor="middle">vote over samples</text><rect class="box-ghost" x="16.0" y="92.0" width="184.0" height="40.0" rx="8"/><text class="t-note" x="108.0" y="117.0" text-anchor="middle">control across calls</text><rect class="box" x="214.0" y="96.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="278.0" y="116.5" text-anchor="middle">one call</text><rect class="box" x="348.0" y="96.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="412.0" y="116.5" text-anchor="middle">fixed pipeline</text><rect class="box-on" x="482.0" y="96.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="546.0" y="116.5" text-anchor="middle">loop (ReAct)</text><rect class="box" x="616.0" y="96.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="680.0" y="116.5" text-anchor="middle">plan, execute</text><rect class="box-ghost" x="16.0" y="142.0" width="184.0" height="40.0" rx="8"/><text class="t-note" x="108.0" y="167.0" text-anchor="middle">action format</text><rect class="box" x="214.0" y="146.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="278.0" y="166.5" text-anchor="middle">text, parsed</text><rect class="box-on" x="348.0" y="146.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="412.0" y="166.5" text-anchor="middle">JSON tool call</text><rect class="box" x="482.0" y="146.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="546.0" y="166.5" text-anchor="middle">code in sandbox</text><rect class="box" x="616.0" y="146.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="680.0" y="166.5" text-anchor="middle">no tools</text><rect class="box-ghost" x="16.0" y="192.0" width="184.0" height="40.0" rx="8"/><text class="t-note" x="108.0" y="217.0" text-anchor="middle">verification</text><rect class="box" x="214.0" y="196.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="278.0" y="216.5" text-anchor="middle">none</text><rect class="box" x="348.0" y="196.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="412.0" y="216.5" text-anchor="middle">self-review</text><rect class="box-on" x="482.0" y="196.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="546.0" y="216.5" text-anchor="middle">external check</text><rect class="box" x="616.0" y="196.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="680.0" y="216.5" text-anchor="middle">best-of-n, search</text><rect class="box-ghost" x="16.0" y="242.0" width="184.0" height="40.0" rx="8"/><text class="t-note" x="108.0" y="267.0" text-anchor="middle">context</text><rect class="box-on" x="214.0" y="246.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="278.0" y="266.5" text-anchor="middle">full replay</text><rect class="box" x="348.0" y="246.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="412.0" y="266.5" text-anchor="middle">compaction</text><rect class="box" x="482.0" y="246.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="546.0" y="266.5" text-anchor="middle">bounded per step</text><rect class="box" x="616.0" y="246.0" width="128.0" height="32.0" rx="6"/><text class="t-tick" x="680.0" y="266.5" text-anchor="middle">prompt caching</text><text class="t-muted" x="380.0" y="304.0" text-anchor="middle">highlighted: the structured loop of Section 3.3 (reasoning model, loop, JSON tool calls,</text><text class="t-muted" x="380.0" y="321.0" text-anchor="middle">a typed final check, full replay). CodeAct changes only row 3; plan-and-execute rows 2 and 5;</text><text class="t-muted" x="380.0" y="338.0" text-anchor="middle">best-of-n row 4. The rows are independent, so the patterns of this chapter combine.</text></svg><figcaption>Five independent design dimensions of a single agent. A pattern is a choice in one or two rows; the highlighted cells are the structured loop of Section 3.3.</figcaption></figure>

| Dimension | The question it answers | Patterns that change it |
|---|---|---|
| Reasoning inside one call | how does the model think before it answers? | chain of thought, self-consistency, reasoning models |
| Control across calls | who decides the next step? | ReAct, plan-and-execute, Agentless |
| Action representation | what does an action look like? | function calling, CodeAct |
| Verification and selection | what decides that an answer is good? | reflection, CRITIC, best-of-n, tree search |
| Context management | what does each call see? | ReWOO, compaction, prompt caching |

So "we use ReAct" says something about control and nothing about verification. Every section below says which row it changes.

### Three numbers decide

Every pattern is judged on three numbers measured on a labelled set: **accuracy**, **cost per task** (tokens and tool calls) and **latency** (time to answer). The number that combines the first two is:

> [!DEFINITION] Cost per correct answer
> The cost of one attempt divided by the share of attempts that are correct: $$\text{cost per correct} = \frac{c}{p}$$ where $$c$$ is the cost of running the pattern once and $$p$$ its accuracy. A pattern that costs ten times more per attempt is cheaper per correct answer only if it is also more than ten times as accurate, or if a wrong answer has a cost of its own (a refund, a human review) that you add to $$c$$.

> [!WARNING]
> Pattern names are not specifications. "We use ReAct" can mean a 2022 text prompt with a regular-expression parser or a function-calling loop with no visible thoughts. "We use reflection" can mean the model grading itself or a test suite grading it, and those behave very differently. When you read a write-up, or write one, state what the model sees at each step, what ends the loop and what the check is.

Most papers below used models that are now several generations old. Read their tables for the *shape* of each result (which pattern helps on which task, at what cost) and re-measure on your own model.

## 3.2 Thinking inside one call: chain of thought, voting and reasoning models

**Row changed: reasoning inside one call.**

Start with the public question "How many years passed between the completion of the Eiffel Tower and the opening of the Sydney Opera House?" (1973 minus 1889, so 84). Asked for the answer only, gpt-5-mini said 66, 72 and 75 in three runs. Asked to "think step by step" first, it wrote down 1889 and 1973, subtracted, and said 84 all three times. Nothing else changed: the same model, the same question, one call each.

Now the Northwind question about Porto and Leeds. Thinking step by step did not help. In two runs the model said, correctly, that it did not have the opening years; in the third it wrote "2 years". On the question about the chief executive it invented three different names in three runs. A scratchpad helps a model use what it knows. It cannot create a fact the model never saw.

> [!DEFINITION] Chain of thought (CoT)
> Prompting a model to write intermediate reasoning steps before the final answer, either by showing worked examples that contain steps (few-shot CoT) or by adding an instruction such as "let's think step by step" (zero-shot CoT). The steps are ordinary output tokens: they cost money, they can be read, and the answer is conditioned on them.

<figure class="fig"><svg viewBox="0 0 760 390" role="img" aria-label="Standard prompting against chain of thought (the model writes its intermediate steps before the answer, at the price of more output tokens), and self-consistency (several sampled chains and a plurality vote over their final answers)."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="190.0" y="22.0" text-anchor="middle">standard prompting</text><text class="t-title" x="560.0" y="22.0" text-anchor="middle">chain of thought</text><line class="edge-dim" x1="380.0" y1="10.0" x2="380.0" y2="190.0"/><rect class="box" x="40.0" y="50.0" width="120.0" height="50.0" rx="10"/><text class="t-note" x="100.0" y="80.0" text-anchor="middle">question</text><line class="edge" x1="160.0" y1="75.0" x2="220.0" y2="75.0" marker-end="url(#ah)"/><rect class="box-on" x="220.0" y="50.0" width="130.0" height="50.0" rx="10"/><text class="t-note" x="285.0" y="72.0" text-anchor="middle">answer</text><text class="t-tick" x="285.0" y="92.0" text-anchor="middle">"The answer is 27."</text><text class="t-muted" x="190.0" y="130.0" text-anchor="middle">one call, about 30 output tokens;</text><text class="t-muted" x="190.0" y="147.0" text-anchor="middle">the arithmetic has to happen inside</text><text class="t-muted" x="190.0" y="164.0" text-anchor="middle">the model, with nothing written down</text><rect class="box" x="410.0" y="50.0" width="90.0" height="50.0" rx="10"/><text class="t-note" x="455.0" y="80.0" text-anchor="middle">question</text><line class="edge" x1="500.0" y1="75.0" x2="526.0" y2="75.0" marker-end="url(#ah)"/><rect class="box-2" x="526.0" y="50.0" width="130.0" height="50.0" rx="10"/><text class="t-note" x="591.0" y="72.0" text-anchor="middle">reasoning</text><text class="t-tick" x="591.0" y="92.0" text-anchor="middle">"23 - 20 = 3; 3 + 6 = 9"</text><line class="edge" x1="656.0" y1="75.0" x2="682.0" y2="75.0" marker-end="url(#ah)"/><rect class="box-on" x="682.0" y="50.0" width="58.0" height="50.0" rx="10"/><text class="t-note" x="711.0" y="80.0" text-anchor="middle">answer</text><text class="t-muted" x="575.0" y="130.0" text-anchor="middle">one call, hundreds of output tokens;</text><text class="t-muted" x="575.0" y="147.0" text-anchor="middle">each intermediate result is written</text><text class="t-muted" x="575.0" y="164.0" text-anchor="middle">and read back before the next step</text><line class="edge-dim" x1="30.0" y1="200.0" x2="730.0" y2="200.0"/><text class="t-title" x="380.0" y="224.0" text-anchor="middle">self-consistency: sample several chains, return the most frequent answer</text><rect class="box" x="40.0" y="248.0" width="100.0" height="40.0" rx="10"/><text class="t-note" x="90.0" y="273.0" text-anchor="middle">question</text><path class="edge" d="M140,268 L168,268 L168,242 L589,242"/><line class="edge" x1="229.0" y1="242.0" x2="229.0" y2="252.0" marker-end="url(#ah)"/><rect class="box-2" x="196.0" y="252.0" width="66.0" height="32.0" rx="6"/><text class="t-tick" x="229.0" y="272.0" text-anchor="middle">chain 1</text><line class="edge" x1="229.0" y1="284.0" x2="229.0" y2="304.0" marker-end="url(#ah)"/><text class="t-strong" x="229.0" y="320.0" text-anchor="middle">9</text><line class="edge" x1="319.0" y1="242.0" x2="319.0" y2="252.0" marker-end="url(#ah)"/><rect class="box-2" x="286.0" y="252.0" width="66.0" height="32.0" rx="6"/><text class="t-tick" x="319.0" y="272.0" text-anchor="middle">chain 2</text><line class="edge" x1="319.0" y1="284.0" x2="319.0" y2="304.0" marker-end="url(#ah)"/><text class="t-strong" x="319.0" y="320.0" text-anchor="middle">9</text><line class="edge" x1="409.0" y1="242.0" x2="409.0" y2="252.0" marker-end="url(#ah)"/><rect class="box-2" x="376.0" y="252.0" width="66.0" height="32.0" rx="6"/><text class="t-tick" x="409.0" y="272.0" text-anchor="middle">chain 3</text><line class="edge" x1="409.0" y1="284.0" x2="409.0" y2="304.0" marker-end="url(#ah)"/><text class="t-tick" x="409.0" y="320.0" text-anchor="middle">27</text><line class="edge" x1="499.0" y1="242.0" x2="499.0" y2="252.0" marker-end="url(#ah)"/><rect class="box-2" x="466.0" y="252.0" width="66.0" height="32.0" rx="6"/><text class="t-tick" x="499.0" y="272.0" text-anchor="middle">chain 4</text><line class="edge" x1="499.0" y1="284.0" x2="499.0" y2="304.0" marker-end="url(#ah)"/><text class="t-strong" x="499.0" y="320.0" text-anchor="middle">9</text><line class="edge" x1="589.0" y1="242.0" x2="589.0" y2="252.0" marker-end="url(#ah)"/><rect class="box-2" x="556.0" y="252.0" width="66.0" height="32.0" rx="6"/><text class="t-tick" x="589.0" y="272.0" text-anchor="middle">chain 5</text><line class="edge" x1="589.0" y1="284.0" x2="589.0" y2="304.0" marker-end="url(#ah)"/><text class="t-tick" x="589.0" y="320.0" text-anchor="middle">12</text><rect class="box-on" x="656.0" y="246.0" width="84.0" height="44.0" rx="8"/><text class="t-tick" x="698.0" y="264.0" text-anchor="middle">vote</text><text class="t-tick" x="698.0" y="280.0" text-anchor="middle">9 (3 of 5)</text><text class="t-muted" x="380.0" y="350.0" text-anchor="middle">the vote is a plurality: 9 wins with 3 of 5, and would still win with 2 of 5 if the wrong answers</text><text class="t-muted" x="380.0" y="367.0" text-anchor="middle">disagree. Five chains cost five times the tokens of one. It helps when answers can be compared</text><text class="t-muted" x="380.0" y="384.0" text-anchor="middle">exactly; it cannot help when the chains share the same mistake (correlated errors).</text></svg><figcaption>Standard prompting against chain of thought, and self-consistency: several sampled chains and a plurality vote over their final answers.</figcaption></figure>

### The paper

In January 2022 Jason Wei and colleagues at Google Research showed that a large enough model, given a few examples whose answers contain the working, writes working for new problems too and gets many more of them right. No training was involved.

> [!PAPER] Wei et al. (2022), Chain of Thought · Figure 1 · page 1
> [![Figure 1 of the chain-of-thought paper: two prompts side by side on the same tennis-ball and cafeteria word problems; standard prompting shows the exemplar answer as The answer is 11 and the model answers 27, marked wrong; chain-of-thought prompting shows the exemplar answer with the working, Roger started with 5 balls, 2 cans of 3 is 6, 5 plus 6 is 11, and the model writes its own working for the cafeteria problem and answers 9, marked right](https://ishwarj.com/img/agents/ch3-cot-figure1.pngch3-cot-figure1.png)](/img/agents/ch3-cot-figure1.png)
>
> **Context:** the opening figure of arXiv 2201.11903, posted 28 January 2022 and published at NeurIPS 2022. The highlighted blue and green text is the chain of thought: the worked example in the prompt and the working the model writes in reply.
>
> **What it says:** the only change between the two sides is the content of the exemplar answer. With "The answer is 11" the model answers 27 to the cafeteria problem; with the working shown, it writes "The cafeteria had 23 apples originally. They used 20 to make lunch. So they had 23 - 20 = 3. They bought 6 more apples, so they have 3 + 6 = 9" and answers 9.
>
> **Why it matters:** this is a pattern, not a model change. Nothing was trained; the same weights produced both answers. The difference is what the model is asked to write before the answer, and that is a decision in your code.

On GSM8K (grade-school maths word problems) PaLM-540B went from about 18% to about 57%. The gain appeared only in large models; small models often got worse. Four months later Takeshi Kojima and colleagues showed that one sentence, "Let's think step by step", gets most of the gain with no examples.

### Voting over several chains

If one chain is good, are several better? Xuezhi Wang and colleagues (2022) sampled many chains at a non-zero temperature, read off each final answer and returned the most common one. They called it **self-consistency**. With 40 samples, PaLM-540B on GSM8K rose from 56.5% to 74.4%.

> [!DEFINITION] Self-consistency (plurality voting)
> Sampling several chains of thought for the same question and returning the final answer that appears most often. It is a plurality vote, not a majority vote: the winner does not need more than half the votes. Only the final answers are compared, so they must be comparable (a number, a label, a normalised entity).

Two details matter in practice.

**Plurality, not majority.** Suppose each chain is right 40% of the time, and wrong chains split evenly between two different wrong answers, 30% each. No answer has a majority, yet over many samples the right answer is the most frequent and wins. Voting needs the right answer to be more common than *any single* wrong answer, not more common than all wrong answers together.

**Correlated errors.** The argument above assumes the wrong chains scatter. If they make the *same* mistake, voting makes it more certain, not less. Our chain-of-thought runs show this: on Everest against Kilimanjaro all three chains answered 2,953 metres, because the model remembers Everest as 8,848 m while our encyclopaedia says 8,849 m. A vote over those three chains would return 2,953 with full agreement. Agreement measures consistency, not correctness.

Self-consistency was not the first result to spend extra computation at answer time. In October 2021 Karl Cobbe and colleagues at OpenAI generated many candidate solutions per problem and picked the one a trained **verifier** ranked highest (Section 3.6). Self-consistency replaced the trained verifier with agreement between samples.

### Reasoning models: the scratchpad moved inside

From late 2024 the pattern moved into training. OpenAI's o1 series, then DeepSeek-R1 and others, were trained with reinforcement learning to produce a long chain of thought before answering, rewarded when the final answer could be checked.

> [!DEFINITION] Reasoning model
> A model trained, usually by reinforcement learning on tasks with checkable answers, to generate an extended chain of thought before its visible answer. Its reasoning tokens are billed as output. What the caller can see and control varies by model and provider: some return a summary of the reasoning, some return the full text (open-weight models such as DeepSeek-R1 do), some return nothing; controls range from an effort level to a token budget.

<figure class="fig"><svg viewBox="0 0 760 364" role="img" aria-label="What reasoning models changed about chain of thought (who writes it, where it appears, how it was learned, what you pay, what you can inspect and control) and what did not change for an agent built around one."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="190.0" y="22.0" text-anchor="middle">prompted chain of thought (2022 to 2024)</text><text class="t-title" x="570.0" y="22.0" text-anchor="middle">reasoning models (late 2024 onwards)</text><line class="edge-dim" x1="380.0" y1="10.0" x2="380.0" y2="300.0"/><rect class="box-2" x="30.0" y="40.0" width="320.0" height="254.0" rx="12"/><text class="t-muted" x="44.0" y="62.0" text-anchor="start">who writes the reasoning</text><text class="t-tick" x="44.0" y="79.0" text-anchor="start">the prompt asks for it ("think step by step")</text><text class="t-muted" x="44.0" y="100.0" text-anchor="start">where it appears</text><text class="t-tick" x="44.0" y="117.0" text-anchor="start">in the visible output, as ordinary tokens</text><text class="t-muted" x="44.0" y="138.0" text-anchor="start">how it was learned</text><text class="t-tick" x="44.0" y="155.0" text-anchor="start">imitation of human-written steps, or none</text><text class="t-muted" x="44.0" y="176.0" text-anchor="start">what you pay</text><text class="t-tick" x="44.0" y="193.0" text-anchor="start">output tokens you can see and count</text><text class="t-muted" x="44.0" y="214.0" text-anchor="start">what you can inspect</text><text class="t-tick" x="44.0" y="231.0" text-anchor="start">the whole chain (faithful or not)</text><text class="t-muted" x="44.0" y="252.0" text-anchor="start">what you control</text><text class="t-tick" x="44.0" y="269.0" text-anchor="start">format, length, examples, when to think</text><rect class="box-4" x="410.0" y="40.0" width="320.0" height="254.0" rx="12"/><text class="t-muted" x="424.0" y="62.0" text-anchor="start">who writes the reasoning</text><text class="t-tick" x="424.0" y="79.0" text-anchor="start">the model, trained by reinforcement learning</text><text class="t-muted" x="424.0" y="100.0" text-anchor="start">where it appears</text><text class="t-tick" x="424.0" y="117.0" text-anchor="start">hidden or summarised; the answer is shown</text><text class="t-muted" x="424.0" y="138.0" text-anchor="start">how it was learned</text><text class="t-tick" x="424.0" y="155.0" text-anchor="start">rewarded for verifiable answers (maths, code)</text><text class="t-muted" x="424.0" y="176.0" text-anchor="start">what you pay</text><text class="t-tick" x="424.0" y="193.0" text-anchor="start">reasoning tokens billed as output, unseen</text><text class="t-muted" x="424.0" y="214.0" text-anchor="start">what you can inspect</text><text class="t-tick" x="424.0" y="231.0" text-anchor="start">a summary, if any; not the raw chain</text><text class="t-muted" x="424.0" y="252.0" text-anchor="start">what you control</text><text class="t-tick" x="424.0" y="269.0" text-anchor="start">an effort setting; little else</text><text class="t-muted" x="380.0" y="318.0" text-anchor="middle">what did not change: a loop still needs tools, a stop condition, a budget and a trace;</text><text class="t-muted" x="380.0" y="335.0" text-anchor="middle">a wrong answer is still wrong; and the counter-evidence later in this section (unfaithful</text><text class="t-muted" x="380.0" y="352.0" text-anchor="middle">explanations, weak self-correction) was measured on the visible kind of reasoning.</text></svg><figcaption>What reasoning models changed about chain of thought and what they did not. Visibility and controls differ across models and providers.</figcaption></figure>

For an agent builder, much stays the same: the model still cannot look anything up, still needs a stop condition and a budget, and is still wrong some of the time. What changed is the economics (with some providers you pay for reasoning you cannot read in full) and the question ("should I ask it to think?" becomes "how much effort should I buy for this step?"). Our experiments use gpt-5-mini with its effort set to minimal, so the reasoning we study is the visible kind.

### Is the chain the real reason?

> [!DEFINITION] Faithfulness (of an explanation)
> An explanation is faithful if it reflects the actual reasons for the model's output. A chain of thought is plausible if it reads well and leads to the answer; it is faithful only if it states what really drove the answer. Models can produce the first without the second.

> [!PAPER] Turpin et al. (2023), Unfaithful Explanations · Abstract · page 1
> [![Abstract of the paper Language Models Don't Always Say What They Think with highlighted phrases: systematically misrepresent the true reason; drop by as much as 36 percent; plausible yet misleading](https://ishwarj.com/img/agents/ch3-unfaithful-abstract.pngithful-abstract.png)](/img/agents/ch3-unfaithful-abstract.png)
>
> **Context:** arXiv 2305.04388 by Miles Turpin, Julian Michael, Ethan Perez and Samuel Bowman (NYU, Cohere, Anthropic), posted 7 May 2023, published at NeurIPS 2023. The experiment adds a bias to the prompt that should not matter, such as reordering the few-shot examples so the right answer is always (A), or a user saying "I think the answer is (B)".
>
> **What it says:** chain-of-thought explanations "can systematically misrepresent the true reason for a model's prediction"; biased prompts made accuracy "drop by as much as 36%" on 13 BIG-Bench Hard tasks, while the explanations never mentioned the bias; the explanations are "plausible yet misleading".
>
> **Why it matters:** a chain of thought is text the model wrote to go with its answer; it is not a trace of how the answer was computed. For an agent builder this means two things: reading the chain alone does not audit a decision, and a plausible chain is not evidence that the answer is right. Where correctness matters, verification has to come from outside the text (Section 3.5).

This result was measured on 2023 models. Section 3.8 shows the 2025 follow-up on reasoning models: the chains became more faithful but were still far from fully faithful in the settings studied. The practical rule is conditional: a chain is useful for debugging, but where correctness matters, do not treat a plausible chain as evidence. Check the answer against something outside the text.

**What to measure.** Accuracy on the multi-step slice of your labelled set with and without a chain, output tokens per call, and, for a reasoning model, accuracy gained per dollar at each effort setting.

<details>
<summary>Research notes: the chain-of-thought papers in detail (optional reading)</summary>

These are the remaining excerpts and table readings behind Section 3.2. Nothing here is needed for the rest of the chapter.

> [!PAPER] Wei et al. (2022), Chain of Thought · Abstract · page 1
> [![Abstract of the chain-of-thought paper with highlighted phrases: a series of intermediate reasoning steps; emerge naturally in sufficiently large language models; surpassing even finetuned GPT-3 with a verifier](https://ishwarj.com/img/agents/ch3-cot-abstract.pngh3-cot-abstract.png)](/img/agents/ch3-cot-abstract.png)
>
> **Context:** the abstract. The paper tests three model families (LaMDA, GPT-3, PaLM) at several sizes on arithmetic (GSM8K and five other sets), commonsense and symbolic reasoning tasks.
>
> **What it says:** chain of thought is "a series of intermediate reasoning steps"; the ability "emerge[s] naturally in sufficiently large language models"; with eight exemplars, PaLM 540B reached state of the art on GSM8K, "surpassing even finetuned GPT-3 with a verifier".
>
> **Why it matters:** two caveats are in the abstract itself. The effect appeared only in large models (the paper's own plots show small models getting *worse* with chain of thought), and the comparison that made the headline was against a model that had been fine-tuned on 7,500 solutions and paired with a trained verifier. The pattern beat training, on that task, at that scale.

> [!PAPER] OpenAI (2024), o1 System Card · Section 2 · page 1
> [![The model data and training section of the o1 system card with highlighted phrases: it can produce a long chain of thought before responding to the user; through training, the models learn to refine their thinking process, try different strategies, and recognize their mistakes](https://ishwarj.com/img/agents/ch3-o1-training.pngch3-o1-training.png)](/img/agents/ch3-o1-training.png)
>
> **Context:** arXiv 2412.16720, the system card dated 5 December 2024, section "Model data and training". The introduction states that the series "is trained with large-scale reinforcement learning to reason using chain of thought".
>
> **What it says:** "o1 thinks before it answers": it "can produce a long chain of thought before responding to the user", and "through training, the models learn to refine their thinking process, try different strategies, and recognize their mistakes".
>
> **Why it matters:** the three verbs in the last sentence are three of this chapter's patterns (reflect, search, self-correct) learned as behaviour inside one call. The card speaks of chain-of-thought summaries being surfaced to users rather than the raw chain, so for this model family the scratchpad you used to read is shown only as a summary; other models and providers differ.

> [!PAPER] Kojima et al. (2022), Zero-shot CoT · Figure 1 · page 2
> [![Figure 1 of the zero-shot chain-of-thought paper: four prompts on a juggler word problem; few-shot answers 8 wrongly; few-shot-CoT writes the working and answers 4; zero-shot answers 8 wrongly; zero-shot-CoT appends Let's think step by step and the model writes There are 16 balls in total, half are golf balls so 8, half of those are blue so 4](https://ishwarj.com/img/agents/ch3-zeroshot-figure1.pngeroshot-figure1.png)](/img/agents/ch3-zeroshot-figure1.png)
>
> **Context:** arXiv 2205.11916, posted 24 May 2022, published at NeurIPS 2022. Panel (d) is the method: the question, then "A: Let's think step by step."
>
> **What it says:** with no examples at all, the trigger sentence makes the model write "There are 16 balls in total. Half of the balls are golf balls. That means that there are 8 golf balls. Half of the golf balls are blue. That means that there are 4 blue golf balls", which is right; without it the same model answers 8.
>
> **Why it matters:** this is the version of chain of thought that every production prompt used for the next two years, because it needs no examples and so no maintenance. It is also the version the counter-evidence below was measured on.

> [!PAPER] Kojima et al. (2022), Zero-shot CoT · Abstract · page 1
> [![Abstract of the zero-shot CoT paper with highlighted phrases: decent zero-shot reasoners; MultiArith from 17.7 percent to 78.7 percent; GSM8K from 10.4 percent to 40.7 percent](https://ishwarj.com/img/agents/ch3-zeroshot-abstract.pngroshot-abstract.png)](/img/agents/ch3-zeroshot-abstract.png)
>
> **Context:** the abstract, reporting results with text-davinci-002.
>
> **What it says:** large models "are decent zero-shot reasoners by simply adding 'Let's think step by step' before each answer"; MultiArith goes "from 17.7% to 78.7%" and GSM8K "from 10.4% to 40.7%".
>
> **Why it matters:** the paper's Table 2 puts the numbers in order: zero-shot 10.4, zero-shot CoT 40.7, few-shot CoT with eight examples 48.7 on GSM8K. The single sentence captures most of the gain; the examples add the rest. It also shows the limit: on CommonsenseQA the trigger *lowered* accuracy. Chain of thought is for problems with steps; it is not a universal improvement.

> [!PAPER] Wang et al. (2022), Self-Consistency · Abstract · page 1
> [![Abstract of the self-consistency paper with highlighted phrases: samples a diverse set of reasoning paths; selects the most consistent answer; GSM8K plus 17.9 percent](https://ishwarj.com/img/agents/ch3-sc-abstract.pngch3-sc-abstract.png)](/img/agents/ch3-sc-abstract.png)
>
> **Context:** arXiv 2203.11171, posted 21 March 2022, published at ICLR 2023. The method is a decoding strategy: the prompt is the chain-of-thought prompt of Wei et al.
>
> **What it says:** it "first samples a diverse set of reasoning paths instead of only taking the greedy one, and then selects the most consistent answer by marginalizing out the sampled reasoning paths". Gains over chain of thought: GSM8K +17.9, SVAMP +11.0, AQuA +12.2, StrategyQA +6.4, ARC-challenge +3.9 points.
>
> **Why it matters:** it is an early and influential "spend more compute at answer time" result, though not the first: Cobbe et al. (2021) had already sampled many solutions and picked one with a trained verifier. The intuition is that a hard problem has several routes to one right answer and many routes to many different wrong ones, so agreement is evidence, as long as the wrong routes do not agree with each other.

> [!PAPER] Wang et al. (2022), Self-Consistency · Table 2 · page 5
> [![Table 2 of the self-consistency paper: arithmetic reasoning accuracy for UL2-20B, LaMDA-137B, PaLM-540B and two GPT-3 variants, chain-of-thought prompting against self-consistency on AddSub, MultiArith, ASDiv, AQuA, SVAMP and GSM8K; the GSM8K column shows PaLM-540B going from 56.5 to 74.4, a gain of 17.9, and code-davinci-002 from 60.1 to 78.0](https://ishwarj.com/img/agents/ch3-sc-table2.pngs/ch3-sc-table2.png)](/img/agents/ch3-sc-table2.png)
>
> **Context:** the main arithmetic table, averaged over ten runs, each sampling 40 chains per question.
>
> **What it says:** every model improves on every task. The gains are largest for the largest models: PaLM-540B on GSM8K from 56.5 to 74.4 (+17.9), code-davinci-002 from 60.1 to 78.0 (+17.9); the 20-billion-parameter UL2 gains only 3 to 7 points.
>
> **Why it matters:** forty samples is forty times the output tokens of one chain; the paper's plots show most of the gain by five to ten samples. Gains are largest where the base model already puts the right answer at the top of its distribution more often than any single wrong answer. A model that makes the same mistake on most routes gains nothing.

> [!PAPER] DeepSeek-AI (2025), DeepSeek-R1 · Abstract · page 1
> [![Abstract of the DeepSeek-R1 paper, version 1, with highlighted phrases: without supervised fine-tuning as a preliminary step; poor readability, and language mixing](https://ishwarj.com/img/agents/ch3-r1-abstract.pngch3-r1-abstract.png)](/img/agents/ch3-r1-abstract.png)
>
> **Context:** arXiv 2501.12948, version 1 of 22 January 2025 (a revised version was later published in Nature). DeepSeek-R1-Zero was trained by reinforcement learning on a base model "without supervised fine-tuning (SFT) as a preliminary step"; the paper reports its AIME 2024 pass@1 rising from 15.6% to 71.0% during training, and 79.8% for the final DeepSeek-R1.
>
> **What it says:** the reasoning behaviour emerged from reward alone, and the first model "encounters challenges such as poor readability, and language mixing", which the final model fixed with a small amount of curated "cold start" data.
>
> **Why it matters:** this is public evidence for how reasoning models are made, and for a cost of hidden reasoning: a chain optimised only for the right answer need not be readable, or even in one language. If you want to read the reasoning, you have to pay for it in training or in prompting.

> [!PAPER] DeepSeek-AI (2025), DeepSeek-R1 · Section 2.2.4 · page 8
> [![A paragraph from the DeepSeek-R1 paper on the aha moment with the highlighted phrase: learns to allocate more thinking time to a problem by reevaluating its initial approach](https://ishwarj.com/img/agents/ch3-r1-aha.pngents/ch3-r1-aha.png)](/img/agents/ch3-r1-aha.png)
>
> **Context:** the "Aha Moment of DeepSeek-R1-Zero" paragraph, describing an intermediate checkpoint whose chain of thought stopped mid-derivation with "Wait, wait. Wait. That's an aha moment I can flag here" and started over.
>
> **What it says:** the model "learns to allocate more thinking time to a problem by reevaluating its initial approach", without being told to.
>
> **Why it matters:** reflection (Section 3.5) appearing as a learned behaviour inside a single call. It does not make the external version obsolete: the model is re-evaluating against its own judgement, which is exactly the kind of critic Section 3.5 finds unreliable for prompted models. Where a test suite exists, it is usually a better judge than a second thought.

> [!PAPER] Turpin et al. (2023), Unfaithful Explanations · Figure 1 · page 5
> [![Figure 1 of the unfaithful explanations paper: for GPT-3.5 and Claude 1.0 under zero-shot and few-shot settings, accuracy with an unbiased context against a biased context, with and without chain of thought; biased contexts lower accuracy by between 4.7 and 36.3 points, and in the zero-shot setting chain of thought drops more than no chain of thought](https://ishwarj.com/img/agents/ch3-unfaithful-figure1.pngaithful-figure1.png)](/img/agents/ch3-unfaithful-figure1.png)
>
> **Context:** accuracy averaged over the BIG-Bench Hard tasks, with the drop caused by each biasing feature printed on the bars.
>
> **What it says:** the biased context lowers accuracy in every condition; the largest drop (36.3 points) is GPT-3.5, zero-shot, *with* chain of thought. In the few-shot setting chain of thought reduces the damage; in the zero-shot setting "it hurts more than it helps".
>
> **Why it matters:** the model rationalised the biased answer in its chain. Any agent that reads user text can receive such a bias ("I'm sure it's the second option"), and the chain will not tell you it happened. Log the inputs, not only the reasoning.

</details>

## 3.3 Acting in a loop: ReAct

**Row changed: control across calls** (and, once tools appear, the action format).

Thinking cannot produce the Porto depot's opening year. Looking it up can. The simplest loop that looks things up is: think about what you need, call a tool, read the result, repeat. Here is what that loop did on the Porto and Leeds question when it worked:

```text
Thought: I need the opening years of the Porto and Leeds depots of Northwind Freight.
Action: wiki[Northwind Freight]
Observation: Fictional logistics company founded in 2011 in Rotterdam. ... Related entries: Rotterdam depot, Gdansk depot, Porto depot, Leeds depot, ...
Action: wiki[Porto depot]      Observation: Northwind Freight depot opened 2017. Vehicles: 64 trucks. ...
Action: wiki[Leeds depot]      Observation: Northwind Freight depot opened 2020. Vehicles: 41 trucks. ...
Action: calc[2020-2017]        Observation: 3
Action: finish[3]
```

And here is the same plain loop on a public question, when it did not:

[![Terminal output of the trace log of ch3_react.py: the plain ReAct loop on the Eiffel Tower question looks up both entries, computes calc 1973 minus 1889 and observes 84, then computes it again, then starts the look-ups over until the seven-step budget runs out and the run is graded wrong; then the guarded loop with the improved tool on the Porto and Leeds question, which looks up the company, the two depots, computes 2020 minus 2017 and finishes with 3, graded correct; then the direct answer, 6 years, and a chain of thought that says it does not have the opening years](https://ishwarj.com/img/agents/ch3_react_trace-run.pngreact_trace-run.png)](/img/agents/ch3_react_trace-run.png)

After step 3 the answer, 84, is in the context. The model computes it again, then starts the look-ups over, and the step budget runs out. This failure has a name in the paper that introduced the loop.

> [!DEFINITION] ReAct
> A prompting pattern in which the model alternates three kinds of text: a **thought** (free-form reasoning about what to do next), an **action** (a tool call in a fixed syntax, or a final answer) and an **observation** (the tool's result, appended by the loop, never written by the model). The cycle repeats until the model emits a finish action or the step budget runs out. The name is "Reason + Act".

> [!DEFINITION] Step budget
> The maximum number of model calls (or tool calls) a run may make before the loop stops it. Set it from the distribution of successful runs, not from the longest imaginable task. A run that hits the budget is usually stuck rather than about to finish.

<figure class="fig"><svg viewBox="0 0 760 340" role="img" aria-label="The ReAct loop (thought, action, observation, repeated until a finish action) and its two surface forms: the paper's plain-text format parsed by a regular expression, and the same loop with function calling, where the thought is optional, hidden or absent."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="380.0" y="22.0" text-anchor="middle">ReAct: thought, action, observation, repeated</text><rect class="box-2" x="140.0" y="57.0" width="100.0" height="30.0" rx="8"/><text class="t-tick" x="190.0" y="76.5" text-anchor="middle">Thought</text><rect class="box-3" x="230.0" y="155.0" width="100.0" height="30.0" rx="8"/><text class="t-tick" x="280.0" y="174.5" text-anchor="middle">Action</text><rect class="box" x="50.0" y="155.0" width="100.0" height="30.0" rx="8"/><text class="t-tick" x="100.0" y="174.5" text-anchor="middle">Observation</text><line class="path" x1="226.0" y1="88.0" x2="258.0" y2="154.0" marker-end="url(#ah-on)"/><line class="path" x1="230.0" y1="170.0" x2="150.0" y2="170.0" marker-end="url(#ah-on)"/><line class="path" x1="122.0" y1="154.0" x2="154.0" y2="88.0" marker-end="url(#ah-on)"/><text class="t-muted" x="258.0" y="118.0" text-anchor="start">write</text><text class="t-muted" x="190.0" y="188.0" text-anchor="middle">run the tool</text><text class="t-muted" x="122.0" y="118.0" text-anchor="end">read</text><text class="t-muted" x="190.0" y="226.0" text-anchor="middle">the model writes a thought and an action;</text><text class="t-muted" x="190.0" y="243.0" text-anchor="middle">the loop runs the tool and appends the</text><text class="t-muted" x="190.0" y="260.0" text-anchor="middle">observation; exit with Action: finish[answer]</text><rect class="box-ghost" x="390.0" y="44.0" width="350.0" height="112.0" rx="10"/><text class="t-note" x="565.0" y="64.0" text-anchor="middle">the paper's text format (2022)</text><text class="t-tick" x="404.0" y="86.0" text-anchor="start">Thought: I need the depot's opening year.</text><text class="t-tick" x="404.0" y="103.0" text-anchor="start">Action: wiki[Porto depot]</text><text class="t-tick" x="404.0" y="120.0" text-anchor="start">Observation: depot opened 2017. Trucks: 64.</text><text class="t-tick" x="404.0" y="137.0" text-anchor="start">Thought: ... Action: finish[3]</text><rect class="box-ghost" x="390.0" y="170.0" width="350.0" height="112.0" rx="10"/><text class="t-note" x="565.0" y="190.0" text-anchor="middle">the same loop with function calling (now)</text><text class="t-tick" x="404.0" y="212.0" text-anchor="start">assistant: tool_use wiki {name: "Porto depot"}</text><text class="t-tick" x="404.0" y="229.0" text-anchor="start">user: tool_result "depot opened 2017 ..."</text><text class="t-tick" x="404.0" y="246.0" text-anchor="start">the thought is optional text before the call,</text><text class="t-tick" x="404.0" y="263.0" text-anchor="start">or hidden reasoning tokens, or absent</text><text class="t-muted" x="380.0" y="308.0" text-anchor="middle">the loop shape is identical; what changed is who parses the action (a regular expression then,</text><text class="t-muted" x="380.0" y="325.0" text-anchor="middle">the API now) and whether the thought is shown at all.</text></svg><figcaption>The ReAct loop and its two surface forms: the 2022 text format parsed by a regular expression, and the same loop with structured tool calls.</figcaption></figure>

### The paper

Shunyu Yao (Princeton) and colleagues at Google posted ReAct in October 2022. Chain of thought could reason but could not look anything up, so it invented facts. Earlier acting agents could call tools but kept no reasoning between actions, so they lost track of the goal. ReAct put both in one loop. For question answering the tools were three operations over Wikipedia (`search`, `lookup`, `finish`) and the budget was seven steps.

> [!PAPER] Yao et al. (2022), ReAct · Table 1 and Figure 2 · page 5
> [![Table 1 and Figure 2 of the ReAct paper: PaLM-540B prompting results on HotpotQA exact match and FEVER accuracy; Standard 28.7 and 57.1, chain of thought 29.4 and 56.3, chain of thought with self-consistency 33.4 and 60.4, Act 25.7 and 58.9, ReAct 27.4 and 60.9, the two combined methods 34.2 and 64.6 and 35.1 and 62.0, supervised state of the art 67.5 and 89.5; the figure plots accuracy against the number of self-consistency samples](https://ishwarj.com/img/agents/ch3-react-table1.pngh3-react-table1.png)](/img/agents/ch3-react-table1.png)
>
> **Context:** PaLM-540B with few-shot prompts on two knowledge tasks: HotpotQA (multi-hop questions over Wikipedia, exact-match score) and FEVER (claim verification: supports, refutes or not enough information).
>
> **What it says:** ReAct beat Act on both tasks (27.4 against 25.7, 60.9 against 58.9). Against chain of thought it won on FEVER (60.9 against 56.3) and *lost* on HotpotQA (27.4 against 29.4). The combinations won: ReAct falling back to self-consistency reached 35.1 on HotpotQA, and self-consistency falling back to ReAct reached 64.6 on FEVER. The supervised systems of the day were far ahead on both (67.5 and 89.5).
>
> **Why it matters:** the headline is not "ReAct is best". It is that reasoning and acting are complementary and that neither alone was reliable. Figure 2 adds the practical point: the combinations reached the accuracy of 21-sample self-consistency with three to five samples, a cost saving of four to seven times.

Two readings matter. First, ReAct *lost* to chain of thought on HotpotQA (27.4 against 29.4); the best results combined the two with a fallback. Second, the paper's hand-labelled error study (Table 2, read in the research notes below) found the two failures our trace shows: "reasoning error", which the authors define to include failing to recover from repetitive steps (47% of the sampled ReAct failures), and non-informative searches that derail the run (23%). The same table is often quoted as "ReAct has 0% hallucination". That is too strong. The 0% is the hallucination category among sampled *failed* ReAct trajectories; among sampled *successful* ones, 6% were right with a hallucinated reasoning trace or facts. Grounding reduced hallucination in that study; it did not remove it.

> [!DEFINITION] Fallback
> A second pattern the loop switches to when the first fails within its budget: ReAct falling back to self-consistency, a chatbot handing the conversation to a person. Fallbacks turn a hard failure into a cheaper or safer path.

### Our experiment: six system configurations

The script `ch3_react.py` runs the twelve questions (six public, six about Northwind) through six configurations with gpt-5-mini at minimal reasoning effort, three times each, so every rate is over 36 runs. The four loop configurations isolate two engineering fixes:

- **Repeated-action guard** (in the loop): if the model emits an action it has already run, the loop replaces the observation with "you have been round this loop; using only the observations above, finish now".
- **Improved tool** (in the tool): an entry lists related entries by name (the company entry names its four depots as "Porto depot" and so on), several matches come back as entries instead of an error, and a miss explains how entries are named.

> [!DEFINITION] Repeated-action guard
> A check in the loop that notices an action identical to one already run and changes what the model sees next: a note that the result is unchanged, an instruction to finish with the evidence it has, or a hand-off to a fallback. It costs a few lines of code.

Grading is **typed and exact**. An answer is right only if it contains exactly the expected numbers, compared as numbers, and names every expected entity after explicit normalisation (accents removed, lower case, punctuation collapsed). An earlier draft of this script used substring matching, which marked "184" correct for 84. The script now starts with a self-test that shows the difference:

```python
# code/agents/ch3_react.py (abridged): the typed grader and the guard
def grade(answer, expected):
    got = sorted(numbers_in(answer))                          # "2,954 m" -> [2954.0]; "three" -> [3.0]
    want = sorted(float(x) for x in expected.get('numbers', []))
    if len(got) != len(want) or any(abs(g - w) > 1e-9 for g, w in zip(got, want)):
        return False                                          # "184", "13", "It is not 84; the answer is 74." all fail
    text = f' {normalise(answer)} '
    return all(f' {normalise(e)} ' in text for e in expected.get('entities', []))

if (tool, arg) in seen:                                       # inside the loop
    repeats += 1
    if guard:
        obs = 'You have been round this loop once. Using ONLY the observations above, ... finish[answer] now.'
```

[![Terminal output of ch3_react.py: the grader self-test, where the old substring grader marks 184, 13 and It is not 84 the answer is 74 as right and the typed grader marks them wrong; a per-question table of correct runs out of three for six configurations; and the summary over 36 runs each: direct 3 percent, chain of thought 22, ReAct 56, ReAct plus guard only 75, plus improved tool only 75, plus both 100, with calls, tokens, cost, cost per correct answer, budget hits and repeat counts; total spend 0.1219 dollars over 715 calls](https://ishwarj.com/img/agents/ch3_react-run.pngs/ch3_react-run.png)](/img/agents/ch3_react-run.png)

<figure class="fig"><svg viewBox="0 0 760 406" role="img" aria-label="Six system configurations on twelve two-hop questions with gpt-5-mini, three repeats each: accuracy on the left, cost per correct answer on the right. The two fixes help different questions: the repeated-action guard the public ones, the improved tool the private ones; together they answer all twelve."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="380.0" y="22.0" text-anchor="middle">Six configurations, 12 questions x 3 repeats, gpt-5-mini (ch3_react.py)</text><text class="t-note" x="170.0" y="52.0" text-anchor="middle">accuracy over 36 runs (typed exact grader)</text><text class="t-note" x="575.0" y="52.0" text-anchor="middle">cost per correct answer, US cents</text><text class="t-tick" x="202.0" y="84.0" text-anchor="end">direct answer</text><g class="mark"><title>direct answer: 3%</title><rect class="s1" x="210.0" y="69.0" width="4.2" height="20.0" rx="3"/></g><text class="t-val" x="220.2" y="84.0" text-anchor="start">3%</text><text class="t-tick" x="202.0" y="112.0" text-anchor="end">chain of thought</text><g class="mark"><title>chain of thought: 22%</title><rect class="s1" x="210.0" y="97.0" width="33.3" height="20.0" rx="3"/></g><text class="t-val" x="249.3" y="112.0" text-anchor="start">22%</text><text class="t-tick" x="202.0" y="140.0" text-anchor="end">ReAct, plain loop</text><g class="mark"><title>ReAct, plain loop: 56%</title><rect class="s1" x="210.0" y="125.0" width="83.3" height="20.0" rx="3"/></g><text class="t-val" x="299.3" y="140.0" text-anchor="start">56%</text><text class="t-tick" x="202.0" y="168.0" text-anchor="end">+ guard only</text><g class="mark"><title>+ guard only: 75%</title><rect class="s1" x="210.0" y="153.0" width="112.5" height="20.0" rx="3"/></g><text class="t-val" x="328.5" y="168.0" text-anchor="start">75%</text><text class="t-tick" x="202.0" y="196.0" text-anchor="end">+ improved tool only</text><g class="mark"><title>+ improved tool only: 75%</title><rect class="s1" x="210.0" y="181.0" width="112.5" height="20.0" rx="3"/></g><text class="t-val" x="328.5" y="196.0" text-anchor="start">75%</text><text class="t-tick" x="202.0" y="224.0" text-anchor="end">+ guard + improved tool</text><g class="mark"><title>+ guard + improved tool: 100%</title><rect class="s1" x="210.0" y="209.0" width="150.0" height="20.0" rx="3"/></g><text class="t-val" x="366.0" y="224.0" text-anchor="start">100%</text><text class="t-tick" x="438.0" y="84.0" text-anchor="end"></text><g class="mark"><title>: 0.144c</title><rect class="s3" x="446.0" y="69.0" width="160.1" height="20.0" rx="3"/></g><text class="t-val" x="612.1" y="84.0" text-anchor="start">0.144c</text><text class="t-tick" x="438.0" y="112.0" text-anchor="end"></text><g class="mark"><title>: 0.086c</title><rect class="s3" x="446.0" y="97.0" width="96.1" height="20.0" rx="3"/></g><text class="t-val" x="548.1" y="112.0" text-anchor="start">0.086c</text><text class="t-tick" x="438.0" y="140.0" text-anchor="end"></text><g class="mark"><title>: 0.172c</title><rect class="s3" x="446.0" y="125.0" width="191.7" height="20.0" rx="3"/></g><text class="t-val" x="643.7" y="140.0" text-anchor="start">0.172c</text><text class="t-tick" x="438.0" y="168.0" text-anchor="end"></text><g class="mark"><title>: 0.085c</title><rect class="s3" x="446.0" y="153.0" width="94.8" height="20.0" rx="3"/></g><text class="t-val" x="546.8" y="168.0" text-anchor="start">0.085c</text><text class="t-tick" x="438.0" y="196.0" text-anchor="end"></text><g class="mark"><title>: 0.111c</title><rect class="s3" x="446.0" y="181.0" width="123.9" height="20.0" rx="3"/></g><text class="t-val" x="575.9" y="196.0" text-anchor="start">0.111c</text><text class="t-tick" x="438.0" y="224.0" text-anchor="end"></text><g class="mark"><title>: 0.072c</title><rect class="s3" x="446.0" y="209.0" width="80.6" height="20.0" rx="3"/></g><text class="t-val" x="532.6" y="224.0" text-anchor="start">0.072c</text><text class="t-muted" x="20.0" y="252.0" text-anchor="start">ReAct, plain loop: public 44%, private 67%; 189 calls; 16 runs hit the step budget</text><text class="t-muted" x="20.0" y="269.0" text-anchor="start">+ guard only: public 100%, private 50%; 134 calls; 0 runs hit the step budget</text><text class="t-muted" x="20.0" y="286.0" text-anchor="start">+ improved tool only: public 56%, private 94%; 170 calls; 9 runs hit the step budget</text><text class="t-muted" x="20.0" y="303.0" text-anchor="start">+ guard + improved tool: public 100%, private 100%; 150 calls; 0 runs hit the step budget</text><text class="t-muted" x="20.0" y="330.0" text-anchor="start">the guard alone fixes the public questions (the model had the facts but kept repeating itself);</text><text class="t-muted" x="20.0" y="347.0" text-anchor="start">the improved tool alone fixes most private ones (the model could not find the depot entries);</text><text class="t-muted" x="20.0" y="364.0" text-anchor="start">both together fix all twelve. The conditions also differ in prompts, budgets and tools, so this</text><text class="t-muted" x="20.0" y="381.0" text-anchor="start">compares system configurations, not loop shapes alone.</text></svg><figcaption>Six system configurations, 36 runs each: the guard and the improved tool each fix different questions; only both together answer all twelve.</figcaption></figure>

| Configuration | Accuracy (36 runs) | Public | Private | Calls | Cost per correct | Runs that hit the budget |
|---|---|---|---|---|---|---|
| Direct answer | 3% | 6% | 0% | 36 | $0.00144 | 0 |
| Chain of thought | 22% | 44% | 0% | 36 | $0.00086 | 0 |
| ReAct, original tool, no guard | 56% | 44% | 67% | 189 | $0.00172 | 16 |
| + guard only | 75% | 100% | 50% | 134 | $0.00085 | 0 |
| + improved tool only | 75% | 56% | 94% | 170 | $0.00111 | 9 |
| + guard + improved tool | 100% | 100% | 100% | 150 | $0.00072 | 0 |

Read it in three steps.

1. **Direct against chain of thought**: a scratchpad helps where the facts are in the model (public questions, 6% to 44%) and does nothing where they are not (private, 0% to 0%).
2. **Chain of thought against the plain loop**: acting brings the private facts within reach (0% to 67%), but the loop loses many public questions it should win, mostly by repeating itself (16 of 36 runs hit the budget).
3. **The ablations**: the two fixes repair *different* failures. The guard alone fixed every public question, because there the model already had what it needed and only had to stop. On private questions the guard mostly turned a stuck run into an honest "unknown", which is cheaper but not more correct. The improved tool alone fixed most private questions, because the model could finally find the depot entries, but runs still went in circles (9 hit the budget). Only both together fixed everything. Neither fix touches the model.

<figure class="fig"><svg viewBox="0 0 760 388" role="img" aria-label="The repetition failure of a plain ReAct loop in our run (the answer is available after step 3, the model keeps acting until the step budget runs out) and the two fixes that live in the loop and the tool rather than in the model, with the accuracy of each fix alone and of both together."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="380.0" y="22.0" text-anchor="middle">The most common ReAct failure in our run, and where the two fixes live</text><rect class="box" x="30.0" y="44.0" width="170.0" height="112.0" rx="12"/><circle class="node on" cx="48.0" cy="62.0" r="11"/><text class="t-note" x="48.0" y="66.5" text-anchor="middle">1</text><text class="t-title" x="66.0" y="67.0" text-anchor="start">step 1</text><text class="t-tick" x="44.0" y="88.0" text-anchor="start">wiki[Eiffel Tower]</text><text class="t-tick" x="44.0" y="105.0" text-anchor="start">Obs: completed 1889</text><line class="edge" x1="200.0" y1="100.0" x2="212.0" y2="100.0" marker-end="url(#ah)"/><rect class="box" x="212.0" y="44.0" width="170.0" height="112.0" rx="12"/><circle class="node on" cx="230.0" cy="62.0" r="11"/><text class="t-note" x="230.0" y="66.5" text-anchor="middle">2</text><text class="t-title" x="248.0" y="67.0" text-anchor="start">step 2</text><text class="t-tick" x="226.0" y="88.0" text-anchor="start">wiki[Sydney Opera House]</text><text class="t-tick" x="226.0" y="105.0" text-anchor="start">Obs: opened 1973</text><line class="edge" x1="382.0" y1="100.0" x2="394.0" y2="100.0" marker-end="url(#ah)"/><rect class="box-4" x="394.0" y="44.0" width="170.0" height="112.0" rx="12"/><circle class="node on" cx="412.0" cy="62.0" r="11"/><text class="t-note" x="412.0" y="66.5" text-anchor="middle">3</text><text class="t-title" x="430.0" y="67.0" text-anchor="start">steps 3 and 4</text><text class="t-tick" x="408.0" y="88.0" text-anchor="start">calc[1973-1889]</text><text class="t-tick" x="408.0" y="105.0" text-anchor="start">Obs: 84</text><text class="t-tick" x="408.0" y="122.0" text-anchor="start">calc[1973-1889] again</text><line class="edge" x1="564.0" y1="100.0" x2="576.0" y2="100.0" marker-end="url(#ah)"/><rect class="box-4" x="576.0" y="44.0" width="170.0" height="112.0" rx="12"/><circle class="node on" cx="594.0" cy="62.0" r="11"/><text class="t-note" x="594.0" y="66.5" text-anchor="middle">4</text><text class="t-title" x="612.0" y="67.0" text-anchor="start">steps 5 to 7</text><text class="t-tick" x="590.0" y="88.0" text-anchor="start">look-ups start over</text><text class="t-tick" x="590.0" y="105.0" text-anchor="start">budget of 7 runs out</text><text class="t-tick" x="590.0" y="122.0" text-anchor="start">no answer: wrong</text><text class="t-muted" x="380.0" y="178.0" text-anchor="middle">the answer (84) was in the context after step 3; the model kept acting instead of finishing</text><text class="t-muted" x="380.0" y="195.0" text-anchor="middle">(the "reasoning error" row of the ReAct paper, which includes failing to recover from repetitive steps)</text><line class="edge-dim" x1="30.0" y1="212.0" x2="730.0" y2="212.0"/><text class="t-title" x="380.0" y="236.0" text-anchor="middle">two fixes outside the model, measured separately</text><rect class="box-4" x="30.0" y="252.0" width="340.0" height="96.0" rx="10"/><text class="t-note" x="200.0" y="274.0" text-anchor="middle">the loop: a repeated-action guard</text><text class="t-tick" x="200.0" y="294.0" text-anchor="middle">the loop sees the same (tool, argument) twice and</text><text class="t-tick" x="200.0" y="311.0" text-anchor="middle">replaces the observation with "you have been</text><text class="t-tick" x="200.0" y="328.0" text-anchor="middle">round this loop; using what you have, finish now"</text><rect class="box-3" x="390.0" y="252.0" width="340.0" height="96.0" rx="10"/><text class="t-note" x="560.0" y="274.0" text-anchor="middle">the tool: observations that name next steps</text><text class="t-tick" x="560.0" y="294.0" text-anchor="middle">a company entry lists its depots as look-up names;</text><text class="t-tick" x="560.0" y="311.0" text-anchor="middle">several matches come back as entries, not errors;</text><text class="t-tick" x="560.0" y="328.0" text-anchor="middle">a miss explains how entries are named</text><text class="t-muted" x="380.0" y="372.0" text-anchor="middle">plain loop 56%; guard only 75%; improved tool only 75%; both 100% (36 runs each)</text></svg><figcaption>The repetition failure of the plain loop, and the two fixes that live in the loop and the tool rather than the model.</figcaption></figure>

> [!WARNING]
> What this experiment does and does not show. It compares **system configurations**, not loop shapes alone: the six conditions differ in tool access, prompt and demonstration, completion budget (60 tokens for a direct answer, 400 for chain of thought, 200 per loop step), observation quality and the guard. Only the four loop configurations form a clean ablation of the guard and the tool. Thirty-six runs per condition is small: the plain loop scored 42%, 67% and 58% in its three repeats. Two answers were marked wrong only because they contained extra numbers ("about 11 years and 9 months (about 12 years)"); a strict grader has a cost, and we report it rather than hide it.

### What function calling changed, and a complete modern loop

When providers added structured tool calling to their APIs in 2023, the regular-expression parser disappeared. The model returns a tool call with a name and JSON arguments; your code runs it and sends back the result; the loop is otherwise the same. The hard parts moved from the text format to the loop (budgets, guards, stop conditions, error handling) and to the tools (what they return on a miss).

The script `ch3_structured_loop.py` is the loop written the way production loops are written today. It is one file of about 270 lines, comments included, runs against the real API, and has every piece this chapter talks about:

- **native tool calls**: the model returns structured calls; several can arrive in one turn;
- **schema-checked arguments**: each tool declares a JSON Schema, and the loop validates arguments before running anything;
- **an explicit state object**: messages, step and token counts, calls already seen, and a status;
- **two budgets**: model steps and total tokens; hitting either ends the run with a named status;
- **error handling**: invalid JSON, an unknown tool, a schema violation or a tool exception becomes a tool result the model can read; API errors are retried with back-off;
- **a trace**: one JSON line per event.

```python
# code/agents/ch3_structured_loop.py (abridged): the loop
def run(question):
    s = RunState(question, messages=[{'role': 'system', 'content': SYSTEM}, {'role': 'user', 'content': question}])
    while s.status == 'running':
        if s.steps >= MAX_STEPS:                      s.status = 'step_budget'; break
        if s.tokens_in + s.tokens_out >= MAX_TOKENS:  s.status = 'token_budget'; break
        msg = call_model(s)                           # retries with back-off; None after three failures
        s.steps += 1
        if msg is None:                               s.status = 'api_error'; break
        s.messages.append(assistant_message(msg))
        for c in msg.tool_calls or []:
            result, args = execute(c.function.name, c.function.arguments)   # parse, look up, validate, run; never raises
            key = (c.function.name, json.dumps(args, sort_keys=True))
            if result['ok'] and key in s.seen:        # the guard
                result['note'] = 'repeated call, result unchanged; answer with what you have'
            if result['ok'] and c.function.name == 'final_answer':
                s.answer, s.evidence, s.status = args['answer'], args['evidence'], 'answered'
            s.event('tool', tool=c.function.name, args=args, ok=result['ok'])
            s.messages.append({'role': 'tool', 'tool_call_id': c.id, 'content': json.dumps(result)})
    return s
```

The final answer is itself a tool, `final_answer`, with a schema: a short `answer` string and a non-empty list of `evidence` entry names. That turns "the model stopped talking" into a typed event the loop can check.

[![Terminal output of ch3_structured_loop.py: six error paths exercised directly, each becoming a readable tool result (a name too short, an extra argument, an expression with words in it, division by zero, invalid JSON, an unknown tool); then the Northwind question about the depot with the most trucks, answered in seven model calls with look-ups of the company and four depots, a calculation of 120 minus 41, and a final answer of Rotterdam with 79 more trucks, 4,352 tokens, graded correct; then the same loop on the twelve questions three times, 34 of 36 correct](https://ishwarj.com/img/agents/ch3_structured_loop-run.pngctured_loop-run.png)](/img/agents/ch3_structured_loop-run.png)

On the depot question the loop looked up the company and the four depots one by one, computed 120 minus 41 and answered "The Rotterdam depot; it has 79 more trucks than the Leeds depot": seven model calls, 4,352 tokens, inside budgets of eight calls and 20,000 tokens. On the twelve questions, three runs each, it got 34 of 36 (94%), using 3.9 model calls and about 2,009 tokens per run on average. One of the two misses is worth remembering: on Everest against Kilimanjaro it looked up both heights (8,849 m and 5,895 m) and then called the calculator with `8848 - 5895`, the height it remembered. The tool result was in the context and the model still used its prior. The other miss was the strict grader again ("Rotterdam (2011)" contains an extra number). Section 3.5 shows a check that catches the first kind.

> [!TIP] In production
> Give every loop four things from day one: a step budget and a token budget (log how often each is hit), a repeated-call guard that changes what the model sees, a typed final answer, and a fallback when a budget runs out. When a run fails, read the tool observations in the trace before touching the prompt. In our ablation every error the fixes removed lived in the loop or the tool.

<details>
<summary>Research notes: the ReAct paper in detail (optional reading)</summary>

> [!PAPER] Yao et al. (2022), ReAct · Table 2 · page 6
> [![Table 2 of the ReAct paper: success and failure modes of ReAct and chain of thought on HotpotQA from a manual study; successes split into true positive 94 and 86 percent and false positive, hallucinated reasoning trace or facts, 6 and 14 percent; failures split into reasoning error 47 and 16 percent, search result error 23 percent for ReAct only, hallucination 0 and 56 percent, and label ambiguity 29 and 28 percent](https://ishwarj.com/img/agents/ch3-react-table2.pngh3-react-table2.png)](/img/agents/ch3-react-table2.png)
>
> **Context:** the authors randomly sampled 50 trajectories with correct answers and 50 with incorrect answers (judged by exact match) from each of ReAct and chain of thought, 200 in all, and labelled them by hand.
>
> **What it says:** among sampled *successes*, 6% of ReAct's and 14% of chain of thought's were "false positives": the answer was right but the reasoning trace or facts were hallucinated. Among sampled *failures*, hallucination accounted for 0% of ReAct's and 56% of chain of thought's. ReAct's failures were mostly reasoning errors (47%, defined to include "failing to recover from repetitive steps") and search result errors (23%: the search returned nothing useful).
>
> **Why it matters:** this is the first failure taxonomy for agents, and both of ReAct's top failure modes are still what trace reviews find first. Read the hallucination numbers carefully: they are shares within small sampled groups, split by success and failure, not an overall hallucination rate.

> [!PAPER] Yao et al. (2022), ReAct · Section 3.3 · page 6
> [![Two paragraphs of the ReAct paper's results section with highlighted phrases: the model repetitively generates the previous thoughts and actions; non-informative search, which counts for 23 percent of the error cases](https://ishwarj.com/img/agents/ch3-react-errors.pngh3-react-errors.png)](/img/agents/ch3-react-errors.png)
>
> **Context:** observations B and C under the error analysis.
>
> **What it says:** the "structural constraint" of alternating thought and action "reduces its flexibility in formulating reasoning steps", and there is "one frequent error pattern specific to ReAct, in which the model repetitively generates the previous thoughts and actions". And "non-informative search, which counts for 23% of the error cases, derails the model reasoning and gives it a hard time to recover".
>
> **Why it matters:** read the second one as a tool-design finding. The quality of what a tool returns on a miss decides whether the loop recovers. Chapter 2's rule that an error message must say what to do next came, in part, from here.

> [!PAPER] Yao et al. (2022), ReAct · Tables 3 and 4 · page 8
> [![Tables 3 and 4 of the ReAct paper: ALFWorld success rates per task type for Act, ReAct, ReAct-IM and BUTLER, with ReAct best of 6 at 71 percent overall against Act 45 and BUTLER 37; WebShop score and success rate for Act 62.3 and 30.1, ReAct 66.6 and 40.0, imitation learning 59.9 and 29.1, imitation plus reinforcement learning 62.4 and 28.7, and human expert 82.1 and 59.6](https://ishwarj.com/img/agents/ch3-react-table3.pngh3-react-table3.png)](/img/agents/ch3-react-table3.png)
>
> **Context:** two interactive environments. ALFWorld is a text game of household tasks ("put a clean pan on the countertop") with 134 test tasks and six controlled trials; WebShop is a simulated shopping site with real product data.
>
> **What it says:** on ALFWorld the best ReAct trial reached 71% against 45% for the best Act trial and 37% for BUTLER, a trained imitation-learning agent; even the worst ReAct trial (48%) beat both. On WebShop, ReAct's 40.0% success beat Act's 30.1% and the trained baselines (29.1 and 28.7), and was far below the human expert's 59.6%.
>
> **Why it matters:** this is where the paper earns the word "agent". Without thoughts, the authors say, Act "fails to correctly decompose goals into smaller subgoals, or loses track of the current state of the environment". The thought is the agent's working memory (Chapter 2) written inline.

**Limitations the authors state.** Prompting results were "still significantly far from domain-specific state-of-the-art approaches"; the smaller PaLM models prompted with ReAct did worst of the four methods, and only fine-tuning on 3,000 ReAct trajectories made them competitive; the repetition failure was attributed to greedy decoding and left open; and the study used three Wikipedia operations and two text games.

</details>

## 3.4 Planning before acting

**Rows changed: control across calls and context management.**

The second Northwind question needs four look-ups that do not depend on each other. The structured loop above did them one per model call: seven calls, and each call re-read everything before it. A planner can write all four look-ups at once:

```text
#E1 = lookup_entry["Rotterdam depot"]
#E2 = lookup_entry["Gdansk depot"]
#E3 = lookup_entry["Porto depot"]
#E4 = lookup_entry["Leeds depot"]
#E5 = solver: which of #E1..#E4 has the most trucks; calculate its trucks minus the trucks in #E4
```

An executor runs #E1 to #E4, in parallel if it likes, without calling the model at all; one last model call reads the plan and the four results and answers. That is two model calls instead of seven. (This plan is illustrative; we did not run a planner on it. The token arithmetic below is from `ch3_cost.py`.)

> [!DEFINITION] Plan-and-execute
> A pattern with two roles. A **planner** call reads the task and writes an explicit plan: a list of steps, each naming a tool and its arguments, with references (#E1, #E2) where one step needs another's result. An **executor** carries the steps out without re-reading the whole history. A final **solver** call reads the plan and the evidence and writes the answer. Replanning, if any, happens between rounds, not between steps.

<figure class="fig"><svg viewBox="0 0 760 392" role="img" aria-label="ReAct re-reads a growing context on every call; plan-and-execute (ReWOO, LLMCompiler) writes the whole plan in one call, runs the steps with small contexts and in parallel where their dependencies allow, and answers once. The token counts come from the planning numbers in ch3_cost.py."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="190.0" y="22.0" text-anchor="middle">ReAct: one growing context</text><text class="t-title" x="562.0" y="22.0" text-anchor="middle">plan-and-execute: plan, then small contexts</text><line class="edge-dim" x1="380.0" y1="10.0" x2="380.0" y2="300.0"/><rect class="box-2" x="40.0" y="48.0" width="90.0" height="34.0" rx="6"/><text class="t-tick" x="85.0" y="69.0" text-anchor="middle">call 1</text><line class="edge" x1="136.0" y1="65.0" x2="160.0" y2="65.0" marker-end="url(#ah)"/><text class="t-muted" x="164.0" y="69.0" text-anchor="start">tool</text><rect class="box-2" x="40.0" y="94.0" width="150.0" height="34.0" rx="6"/><text class="t-tick" x="115.0" y="115.0" text-anchor="middle">call 2 + 1 obs</text><line class="edge" x1="196.0" y1="111.0" x2="220.0" y2="111.0" marker-end="url(#ah)"/><text class="t-muted" x="224.0" y="115.0" text-anchor="start">tool</text><rect class="box-2" x="40.0" y="140.0" width="210.0" height="34.0" rx="6"/><text class="t-tick" x="145.0" y="161.0" text-anchor="middle">call 3: prompt + 2 thoughts + 2 obs</text><line class="edge" x1="256.0" y1="157.0" x2="280.0" y2="157.0" marker-end="url(#ah)"/><text class="t-muted" x="284.0" y="161.0" text-anchor="start">tool</text><rect class="box-2" x="40.0" y="186.0" width="270.0" height="34.0" rx="6"/><text class="t-tick" x="175.0" y="207.0" text-anchor="middle">call 4: prompt + 3 thoughts + 3 obs</text><text class="t-muted" x="190.0" y="246.0" text-anchor="middle">every call re-reads everything before it:</text><text class="t-muted" x="190.0" y="263.0" text-anchor="middle">15,750 input tokens for 5 steps,</text><text class="t-muted" x="190.0" y="280.0" text-anchor="middle">126,000 for 20 (ch3_cost.py)</text><rect class="box-4" x="410.0" y="48.0" width="110.0" height="56.0" rx="10"/><text class="t-note" x="465.0" y="70.0" text-anchor="middle">planner</text><text class="t-tick" x="465.0" y="90.0" text-anchor="middle">one big call</text><text class="t-tick" x="530.0" y="60.0" text-anchor="start">#E1 = wiki[Porto depot]</text><text class="t-tick" x="530.0" y="77.0" text-anchor="start">#E2 = wiki[Leeds depot]</text><text class="t-tick" x="530.0" y="94.0" text-anchor="start">#E3 = calc[#E2.year - #E1.year]</text><line class="edge" x1="465.0" y1="104.0" x2="480.0" y2="130.0" marker-end="url(#ah)"/><rect class="box-3" x="440.0" y="130.0" width="80.0" height="30.0" rx="6"/><text class="t-tick" x="480.0" y="149.0" text-anchor="middle">worker 1</text><line class="edge" x1="480.0" y1="160.0" x2="570.0" y2="262.0" marker-end="url(#ah)"/><line class="edge" x1="465.0" y1="104.0" x2="580.0" y2="170.0" marker-end="url(#ah)"/><rect class="box-3" x="540.0" y="170.0" width="80.0" height="30.0" rx="6"/><text class="t-tick" x="580.0" y="189.0" text-anchor="middle">worker 2</text><line class="edge" x1="580.0" y1="200.0" x2="570.0" y2="262.0" marker-end="url(#ah)"/><line class="edge" x1="465.0" y1="104.0" x2="680.0" y2="210.0" marker-end="url(#ah)"/><rect class="box-3" x="640.0" y="210.0" width="80.0" height="30.0" rx="6"/><text class="t-tick" x="680.0" y="229.0" text-anchor="middle">worker 3</text><line class="edge" x1="680.0" y1="240.0" x2="570.0" y2="262.0" marker-end="url(#ah)"/><rect class="box-on" x="510.0" y="262.0" width="120.0" height="40.0" rx="10"/><text class="t-note" x="570.0" y="287.0" text-anchor="middle">solver</text><text class="t-muted" x="570.0" y="322.0" text-anchor="middle">plan once, run the steps with small contexts</text><text class="t-muted" x="570.0" y="339.0" text-anchor="middle">(in parallel where the plan allows), answer once:</text><text class="t-muted" x="570.0" y="356.0" text-anchor="middle">8,150 input tokens for 5 steps, 22,850 for 20.</text><text class="t-muted" x="570.0" y="373.0" text-anchor="middle">the price: a wrong plan is noticed only at the end</text></svg><figcaption>A loop re-reads a growing context on every call; plan-and-execute plans once, runs steps with small contexts and answers once.</figcaption></figure>

### The papers, briefly

Three 2023 papers built the idea up. **Plan-and-Solve** put "devise a plan, then carry it out" into one prompt. **ReWOO** ("Reasoning WithOut Observation") separated a planner, workers and a solver, with variables like #E2 so the planner never waits for a result. **LLMCompiler** dispatched the plan's steps in parallel as their inputs became ready, reporting latency speed-ups of up to 3.7 times over ReAct.

> [!PAPER] Xu et al. (2023), ReWOO · Table 2 (excerpt) · page 7
> [![The HotpotQA, TriviaQA and GSM8K blocks of Table 2 of the ReWOO paper: on HotpotQA with 1000 questions, Direct 37.8 accuracy and 55 tokens, chain of thought 41.6 and 482 tokens, ReAct 40.8 accuracy with 9,795 tokens and 4.97 steps costing 19.59 dollars per thousand, ReWOO 42.4 accuracy with 1,986 tokens and 4.45 steps costing 3.97 dollars; on TriviaQA Direct scores 80.6 while ReAct scores 59.4 and ReWOO 66.6](https://ishwarj.com/img/agents/ch3-rewoo-table2.pngh3-rewoo-table2.png)](/img/agents/ch3-rewoo-table2.png)
>
> **Context:** GPT-3.5 with Wikipedia and a calculator, a thousand questions per set.
>
> **What it says:** on HotpotQA, ReWOO matched ReAct's accuracy (42.4 against 40.8) at a fifth of the tokens (1,986 against 9,795 per question) and a fifth of the cost ($3.97 against $19.59 per thousand questions). On TriviaQA, both tool-using methods were *worse* than answering directly (80.6 direct against 59.4 and 66.6): the model already knew the answers and the tools distracted it.
>
> **Why it matters:** two production rules in one table. Decoupling plan from observation cuts the token bill by the number of steps. And adding tools to a task the model can already do makes it worse and more expensive; the direct-answer baseline must always be in the comparison.

### The cost arithmetic, and its assumptions

`ch3_cost.py` counts tokens for a task of *n* tool steps under stated planning numbers (a 1,500-token prompt, 150-token thoughts, 300-token observations), with no API calls. A loop that sends the full history every time sends roughly

$$
n \cdot P + \frac{n(n-1)}{2}(T + O)
$$

input tokens, where $$P$$ is the prompt, $$T$$ a thought and $$O$$ an observation. The second term is the re-reading, and it grows with the square of the number of steps. A plan-and-execute design sends $$P$$ to the planner, a small fixed brief to each executor step, and $$P$$ plus the evidence to the solver: linear in *n*.

This comparison rests on two assumptions, and production systems often change both:

- **Full replay.** The quadratic term assumes every call resends the whole growing history. **Compaction** (keeping only the last few steps in full, or a summary) makes the loop roughly linear, at the price of forgetting detail.
- **Full price for repeated tokens.** Many providers offer **prompt caching**: a repeated prefix is billed at a fraction of the normal input price. That changes the bill a lot, but not the size of the context the model must attend to.

[![Terminal output of ch3_cost.py part four: ReAct input tokens against step count with full replay, with prompt caching at 10 percent of the input price, with compaction keeping three steps, and plan-and-execute: 2 steps 5,850, 2,745, 5,850 and 5,210; 5 steps 15,750, 4,950, 14,400 and 8,150; 10 steps 41,250, 9,525, 28,650 and 13,050; 20 steps 126,000, 22,050, 57,150 and 22,850](https://ishwarj.com/img/agents/ch3_cost_tokens-run.pngcost_tokens-run.png)](/img/agents/ch3_cost_tokens-run.png)

<figure class="fig"><svg viewBox="0 0 760 330" role="img" aria-label="Billed-equivalent input tokens per task against the number of tool steps: a ReAct loop that replays its full history, the same loop with prompt caching or with compaction, and plan-and-execute with bounded per-step contexts (planning numbers of ch3_cost.py)."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><line class="grid" x1="70.0" y1="280.0" x2="510.0" y2="280.0"/><text class="t-tick" x="62.0" y="284.0" text-anchor="end">0</text><line class="grid" x1="70.0" y1="236.4" x2="510.0" y2="236.4"/><text class="t-tick" x="62.0" y="240.4" text-anchor="end">25,000</text><line class="grid" x1="70.0" y1="192.9" x2="510.0" y2="192.9"/><text class="t-tick" x="62.0" y="196.9" text-anchor="end">50,000</text><line class="grid" x1="70.0" y1="149.3" x2="510.0" y2="149.3"/><text class="t-tick" x="62.0" y="153.3" text-anchor="end">75,000</text><line class="grid" x1="70.0" y1="105.7" x2="510.0" y2="105.7"/><text class="t-tick" x="62.0" y="109.7" text-anchor="end">100,000</text><line class="grid" x1="70.0" y1="62.1" x2="510.0" y2="62.1"/><text class="t-tick" x="62.0" y="66.1" text-anchor="end">125,000</text><text class="t-tick" x="70.0" y="300.0" text-anchor="middle">2</text><text class="t-tick" x="143.3" y="300.0" text-anchor="middle">5</text><text class="t-tick" x="265.6" y="300.0" text-anchor="middle">10</text><text class="t-tick" x="510.0" y="300.0" text-anchor="middle">20</text><line class="axis" x1="70.0" y1="280.0" x2="510.0" y2="280.0"/><polyline class="edge-1" points="70.0,269.8 143.3,252.6 265.6,208.1 510.0,60.4"/><g class="mark"><title>ReAct, full replay, 2: 5,850</title><circle class="s1 ring" cx="70.0" cy="269.8" r="4.5"/></g><g class="mark"><title>ReAct, full replay, 5: 15,750</title><circle class="s1 ring" cx="143.3" cy="252.6" r="4.5"/></g><g class="mark"><title>ReAct, full replay, 10: 41,250</title><circle class="s1 ring" cx="265.6" cy="208.1" r="4.5"/></g><g class="mark"><title>ReAct, full replay, 20: 126,000</title><circle class="s1 ring" cx="510.0" cy="60.4" r="4.5"/></g><polyline class="edge-2" points="70.0,269.8 143.3,254.9 265.6,230.1 510.0,180.4"/><g class="mark"><title>ReAct + compaction, 2: 5,850</title><circle class="s2 ring" cx="70.0" cy="269.8" r="4.5"/></g><g class="mark"><title>ReAct + compaction, 5: 14,400</title><circle class="s2 ring" cx="143.3" cy="254.9" r="4.5"/></g><g class="mark"><title>ReAct + compaction, 10: 28,650</title><circle class="s2 ring" cx="265.6" cy="230.1" r="4.5"/></g><g class="mark"><title>ReAct + compaction, 20: 57,150</title><circle class="s2 ring" cx="510.0" cy="180.4" r="4.5"/></g><polyline class="edge-3" points="70.0,275.2 143.3,271.4 265.6,263.4 510.0,241.6"/><g class="mark"><title>ReAct + caching, 2: 2,745</title><circle class="s3 ring" cx="70.0" cy="275.2" r="4.5"/></g><g class="mark"><title>ReAct + caching, 5: 4,950</title><circle class="s3 ring" cx="143.3" cy="271.4" r="4.5"/></g><g class="mark"><title>ReAct + caching, 10: 9,525</title><circle class="s3 ring" cx="265.6" cy="263.4" r="4.5"/></g><g class="mark"><title>ReAct + caching, 20: 22,050</title><circle class="s3 ring" cx="510.0" cy="241.6" r="4.5"/></g><polyline class="edge-4" points="70.0,270.9 143.3,265.8 265.6,257.3 510.0,240.2"/><g class="mark"><title>plan-and-execute, 2: 5,210</title><circle class="s4 ring" cx="70.0" cy="270.9" r="4.5"/></g><g class="mark"><title>plan-and-execute, 5: 8,150</title><circle class="s4 ring" cx="143.3" cy="265.8" r="4.5"/></g><g class="mark"><title>plan-and-execute, 10: 13,050</title><circle class="s4 ring" cx="265.6" cy="257.3" r="4.5"/></g><g class="mark"><title>plan-and-execute, 20: 22,850</title><circle class="s4 ring" cx="510.0" cy="240.2" r="4.5"/></g><line class="grid" x1="516.0" y1="60.4" x2="526.0" y2="60.4"/><rect class="s1" x="530" y="55.4" width="10" height="10" rx="2"/><text class="t-tick" x="546.0" y="64.4" text-anchor="start">ReAct, full replay</text><line class="grid" x1="516.0" y1="180.4" x2="526.0" y2="180.4"/><rect class="s2" x="530" y="175.4" width="10" height="10" rx="2"/><text class="t-tick" x="546.0" y="184.4" text-anchor="start">ReAct + compaction</text><line class="grid" x1="516.0" y1="240.2" x2="526.0" y2="240.2"/><rect class="s4" x="530" y="235.2" width="10" height="10" rx="2"/><text class="t-tick" x="546.0" y="244.2" text-anchor="start">plan-and-execute</text><line class="grid" x1="516.0" y1="241.6" x2="526.0" y2="257.2"/><rect class="s3" x="530" y="252.2" width="10" height="10" rx="2"/><text class="t-tick" x="546.0" y="261.2" text-anchor="start">ReAct + caching</text><text class="t-tick" x="290.0" y="320.0" text-anchor="middle">tool steps in the task</text><text class="t-tick" x="14.0" y="22.0" text-anchor="start">input tokens per task (billed equivalent)</text></svg><figcaption>Billed-equivalent input tokens against tool steps under the planning numbers of ch3_cost.py: full replay, compaction, caching (at an assumed 10% price) and plan-and-execute.</figcaption></figure>

At five steps the full-replay loop sends 15,750 input tokens and plan-and-execute 8,150; at twenty steps the ratio is 5.5 to one (126,000 against 22,850). With caching at our assumed 10% rate the twenty-step loop costs the equivalent of 22,050 tokens, about the same as the plan; with compaction it sends 57,150. So "ReAct is quadratic" is a statement about full replay at full price, not about ReAct. What a plan still buys under caching is fewer sequential model calls (latency) and an artefact a person can review before anything runs.

> [!DEFINITION] Replanning
> Running the planner again after some steps have returned, with the evidence so far, so that a wrong plan can be corrected. Without it a plan-first agent cannot react to a surprise; with it after every step, the pattern turns back into a loop. Most systems replan when a step fails or at fixed checkpoints.

| | Loop (ReAct) | Plan-and-execute |
|---|---|---|
| Good for | few steps, each depending on the last; exploratory tasks | known task shapes with several independent steps; plans a person should approve |
| Costs | a model call per step; context growth under full replay | a planner that must know the tools; replanning adds cost back |
| Fails by | repetition, drift, budget exhaustion | a wrong plan executed in full before anyone notices |
| Measure | steps per task, repeat rate, tokens per task | dependency width (the parallel speed-up available), plan error rate, replanning rate |

<details>
<summary>Research notes: Plan-and-Solve, ReWOO and LLMCompiler in detail (optional reading)</summary>

> [!PAPER] Wang et al. (2023), Plan-and-Solve · Abstract · page 1
> [![Abstract of the Plan-and-Solve paper with highlighted phrases: calculation errors, missing-step errors, and semantic misunderstanding errors; devising a plan to divide the entire task into smaller subtasks](https://ishwarj.com/img/agents/ch3-ps-abstract.pngch3-ps-abstract.png)](/img/agents/ch3-ps-abstract.png)
>
> **Context:** arXiv 2305.04091, posted 6 May 2023, published at ACL 2023. The error study on page 1 found that zero-shot chain of thought on GSM8K failed through calculation errors (7%), missing steps (12%) and misunderstanding the problem (27%).
>
> **What it says:** zero-shot chain of thought "suffers from three pitfalls: calculation errors, missing-step errors, and semantic misunderstanding errors". Plan-and-Solve addresses the missing steps by "devising a plan to divide the entire task into smaller subtasks, and then carrying out the subtasks according to the plan".
>
> **Why it matters:** the method is a diagnosis turned into a sentence. Missing steps are what you get when a model reasons forward and forgets a sub-goal; writing the plan first is the cheapest remedy.

> [!PAPER] Wang et al. (2023), Plan-and-Solve · Section 2.1 · page 3
> [![A paragraph of the Plan-and-Solve paper giving the prompt, with highlighted phrases: devise a plan to solve the problem; carry out the plan and solve the problem step by step](https://ishwarj.com/img/agents/ch3-ps-prompt.pngs/ch3-ps-prompt.png)](/img/agents/ch3-ps-prompt.png)
>
> **Context:** the exact trigger that replaces "Let's think step by step".
>
> **What it says:** "Let's first understand the problem and devise a plan to solve the problem. Then, let's carry out the plan and solve the problem step by step." The extended PS+ version adds "pay attention to calculation" and "extract relevant variables and their corresponding numerals".
>
> **Why it matters:** this is plan-and-execute inside a single call: the plan and the execution are both text, written in order. It costs nothing extra and it is the first thing to try before building a two-model system.

> [!PAPER] Wang et al. (2023), Plan-and-Solve · Table 2 · page 6
> [![Table 2 of the Plan-and-Solve paper: accuracy of text-davinci-003 on six maths datasets; zero-shot chain of thought averages 70.4, program of thoughts 73.5, Plan-and-Solve 72.9 and PS plus 76.7; few-shot manual chain of thought averages 77.6; on GSM8K the figures are 56.4, 57.0, 58.2, 59.3 and 58.4](https://ishwarj.com/img/agents/ch3-ps-table2.pngs/ch3-ps-table2.png)](/img/agents/ch3-ps-table2.png)
>
> **Context:** text-davinci-003, greedy decoding, zero-shot except the last two rows.
>
> **What it says:** PS+ averages 76.7 across the six sets against 70.4 for zero-shot chain of thought, and matches the eight-example few-shot prompt (77.6) with no examples. On GSM8K alone the gain is three points (56.4 to 59.3).
>
> **Why it matters:** a six-point average gain for a sentence is a good trade, and the paper's error analysis (Table 6) shows where it came from: missing-step errors fell. Calculation errors did not fall much, which is the honest signal that a plan does not fix arithmetic; a calculator does.

> [!PAPER] Xu et al. (2023), ReWOO · Abstract · page 1
> [![Abstract of the ReWOO paper with highlighted phrases: redundant prompts and repeated execution; detaches the reasoning process from external observations; five times token efficiency and 4 percent accuracy improvement](https://ishwarj.com/img/agents/ch3-rewoo-abstract.png-rewoo-abstract.png)](/img/agents/ch3-rewoo-abstract.png)
>
> **Context:** arXiv 2305.18323, posted 23 May 2023. The authors' complaint is about ReAct's economics: the API is stateless, so "all the historical tokens ... are fed into the LLM" on every step.
>
> **What it says:** interleaved loops lead "to huge computation complexity from redundant prompts and repeated execution"; ReWOO "detaches the reasoning process from external observations", and on HotpotQA "achieves 5× token efficiency and 4% accuracy improvement".
>
> **Why it matters:** this is Chapter 2's context-budget arithmetic made into an architecture. If the model does not need to see an observation to decide the next step, do not show it one.

> [!PAPER] Xu et al. (2023), ReWOO · Figure 1 · page 2
> [![Figure 1 of the ReWOO paper: a Planner writes a plan of four steps with evidence variables E1 to E4, such as E1 equals Wikipedia of The Hennchata and E2 equals LLM of what is the main ingredient given context E1; a Worker fills each evidence variable by calling the tool; a Solver reads the plans and evidence together and writes the answer, Jas Hennessy and Co](https://ishwarj.com/img/agents/ch3-rewoo-figure1.png3-rewoo-figure1.png)](/img/agents/ch3-rewoo-figure1.png)
>
> **Context:** the workflow for the question "What is the name of the cognac house that makes the main ingredient in The Hennchata?"
>
> **What it says:** the **Planner** writes every step up front, naming the tool and referencing earlier results as variables (`#E3 = Wikipedia[#E2]`). The **Worker** runs the steps and substitutes the variables. The **Solver** sees the plan and all the evidence once and answers.
>
> **Why it matters:** the variables are the trick. Because the plan can say "search for whatever #E2 turns out to be", the planner does not need to wait for #E2. The model is called twice (plan and solve) however many tools run, and the tools can run in whatever order their dependencies allow.

> [!PAPER] Kim et al. (2023), LLMCompiler · Abstract · page 1
> [![Abstract of the LLMCompiler paper with highlighted phrases: sequential reasoning and acting for each function; latency speedup of up to 3.7 times, cost savings of up to 6.7 times](https://ishwarj.com/img/agents/ch3-llmcompiler-abstract.pngmpiler-abstract.png)](/img/agents/ch3-llmcompiler-abstract.png)
>
> **Context:** arXiv 2312.04511, posted 7 December 2023, published at ICML 2024. Three components: a Function Calling Planner that writes a directed acyclic graph of tasks, a Task Fetching Unit that dispatches tasks as their inputs become ready, and an Executor that runs them in parallel.
>
> **What it says:** current methods "require sequential reasoning and acting for each function which can result in high latency, cost, and sometimes inaccurate behavior"; the compiler delivers "latency speedup of up to 3.7×, cost savings of up to 6.7×, and accuracy improvement of up to ∼9% compared to ReAct".
>
> **Why it matters:** the accuracy gain is the surprising one, and the paper explains it: ReAct's looping and early stopping (our Section 3.3 failures again) disappeared once the plan was written in advance.

> [!PAPER] Kim et al. (2023), LLMCompiler · Table 1 · page 6
> [![Table 1 of the LLMCompiler paper: accuracy and latency on HotpotQA, Movie Recommendation, ParallelQA and Game of 24 for ReAct, ReAct with anti-looping prompts, OpenAI parallel function calling and LLMCompiler, with GPT and LLaMA-2 70B; on Movie Recommendation with GPT, ReAct with prompts scores 72.47 at 20.47 seconds and LLMCompiler 77.13 at 5.47 seconds, a 3.74 times speedup; the caption notes frequent looping and early stopping of ReAct](https://ishwarj.com/img/agents/ch3-llmcompiler-table1.pngcompiler-table1.png)](/img/agents/ch3-llmcompiler-table1.png)
>
> **Context:** the main results. The caption admits that plain ReAct's latency could not be measured fairly because "looping and early stopping make precise latency measurement difficult", so a ReAct variant with anti-looping prompts is the baseline.
>
> **What it says:** on Movie Recommendation (eight independent searches per question) the compiler reached 77.13% in 5.47 seconds against ReAct's 72.47% in 20.47 seconds; on HotpotQA (two dependent searches) the speedup was 1.8x with equal accuracy; on their ParallelQA set 2.15x; on Game of 24 the compiler re-planned between rounds and beat Tree of Thoughts' time by 2.89x at the same accuracy.
>
> **Why it matters:** the speedup equals the width of the graph. Tasks whose steps are independent (compare these eight things) gain the most; tasks whose every step depends on the previous one gain almost nothing. Measure the shape of your task's dependency graph before you build a scheduler.

</details>

## 3.5 Checking the answer: who tells the agent it was wrong?

**Row changed: verification.**

Go back to the structured loop's miss: it looked up 8,849 m and calculated with 8,848. Could the agent itself have caught that? Two kinds of check are possible.

- **Ask the model to review its answer.** It sees the same context and the same prior that produced 8,848. Nothing new enters the loop.
- **Check the answer against the evidence.** A few lines of code can test that every number fed to the calculator appears in an earlier observation. 8,848 appears in no observation, so the check fails and the loop can say exactly what is wrong: "8,848 is not in any looked-up entry; Mount Everest's entry says 8,849."

The second check brings in information the model did not use. That difference is the whole of this section.

> [!DEFINITION] Reflection (self-correction loop)
> A pattern in which an attempt is evaluated, a critique is written in words, and a new attempt is made with the critique (and usually the failed attempt) in context. The evaluation can be **external** (tests, a compiler, a search result, a database, a person, a check against the evidence) or **intrinsic** (the model reviewing its own work with no new information).

<figure class="fig"><svg viewBox="0 0 760 372" role="img" aria-label="The reflection loop (actor, evaluator, verbal self-reflection, memory, retry) and the distinction that decides whether it helps: a weak check where the model judges itself against a strong check where an external signal such as tests, a compiler, a search result or a person tells it what is wrong."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="380.0" y="22.0" text-anchor="middle">Reflection: try, check, critique, retry. The check decides whether it works.</text><rect class="box-2" x="30.0" y="50.0" width="120.0" height="56.0" rx="10"/><text class="t-note" x="90.0" y="72.0" text-anchor="middle">actor</text><text class="t-tick" x="90.0" y="92.0" text-anchor="middle">writes an attempt</text><line class="edge" x1="150.0" y1="78.0" x2="200.0" y2="78.0" marker-end="url(#ah)"/><rect class="box-3" x="200.0" y="50.0" width="150.0" height="56.0" rx="10"/><text class="t-note" x="275.0" y="72.0" text-anchor="middle">evaluator</text><text class="t-tick" x="275.0" y="92.0" text-anchor="middle">pass or fail</text><line class="edge" x1="350.0" y1="78.0" x2="400.0" y2="78.0" marker-end="url(#ah)"/><rect class="box-1" x="400.0" y="50.0" width="150.0" height="56.0" rx="10"/><text class="t-note" x="475.0" y="72.0" text-anchor="middle">self-reflection</text><text class="t-tick" x="475.0" y="92.0" text-anchor="middle">why it failed, in words</text><line class="edge" x1="550.0" y1="78.0" x2="600.0" y2="78.0" marker-end="url(#ah)"/><rect class="box-1" x="600.0" y="50.0" width="130.0" height="56.0" rx="10"/><text class="t-note" x="665.0" y="72.0" text-anchor="middle">memory</text><text class="t-tick" x="665.0" y="92.0" text-anchor="middle">reflections so far</text><path class="path" d="M665.0,106.0 L665.0,130.0 L90.0,130.0 L90.0,106.0" marker-end="url(#ah-on)"/><text class="t-muted" x="380.0" y="146.0" text-anchor="middle">next attempt, with the reflections in context (Reflexion, Figure 2)</text><line class="edge-dim" x1="30.0" y1="164.0" x2="730.0" y2="164.0"/><text class="t-title" x="200.0" y="188.0" text-anchor="middle">weak check: the model judges itself</text><text class="t-title" x="560.0" y="188.0" text-anchor="middle">strong check: an external signal</text><rect class="box-ghost" x="30.0" y="200.0" width="340.0" height="118.0" rx="10"/><text class="t-tick" x="44.0" y="222.0" text-anchor="start">"review your answer and improve it"</text><text class="t-tick" x="44.0" y="239.0" text-anchor="start">no new information enters the loop</text><text class="t-tick" x="44.0" y="256.0" text-anchor="start">Huang et al.: accuracy fell on every benchmark</text><text class="t-tick" x="44.0" y="273.0" text-anchor="start">Self-Refine: no gain on maths</text><text class="t-tick" x="44.0" y="290.0" text-anchor="start">useful for style, format, checklist coverage</text><rect class="box-3" x="390.0" y="200.0" width="340.0" height="118.0" rx="10"/><text class="t-tick" x="404.0" y="222.0" text-anchor="start">unit tests, a compiler, a linter, a type checker</text><text class="t-tick" x="404.0" y="239.0" text-anchor="start">a search result, a database row, a calculator</text><text class="t-tick" x="404.0" y="256.0" text-anchor="start">a human review, a user's reply</text><text class="t-tick" x="404.0" y="273.0" text-anchor="start">Reflexion: HumanEval 80.1 to 91.0 with tests</text><text class="t-tick" x="404.0" y="290.0" text-anchor="start">CRITIC: tools as critic; without them, no gain</text><text class="t-muted" x="380.0" y="342.0" text-anchor="middle">the loop is the same; the signal is what you pay for. If there is no external signal,</text><text class="t-muted" x="380.0" y="359.0" text-anchor="middle">build one before you add a retry.</text></svg><figcaption>The reflection loop. Whether it helps depends on the check: the model judging itself, or an external signal such as tests, a compiler, a search or a person.</figcaption></figure>

### Two papers that disagree, and why they do not

**Reflexion** (Noah Shinn and colleagues, March 2023) let an agent retry after a failure with a short written reflection on what went wrong. On programming tasks the evaluator was a test suite the model wrote itself, run by a machine.

Reflexion's headline (GPT-4 on HumanEval, 80.1% to 91.0%) came with one honest loss: on MBPP the score fell from 80.1% to 77.1%, which the authors attribute to self-written tests that passed wrong code. Six months later Jie Huang and colleagues at Google DeepMind re-examined the self-correction results of 2023 and pointed out that several used an **oracle**: the true label decided when to stop retrying. Without it, asking a model to critique and revise its own reasoning made accuracy fall.

> [!DEFINITION] Oracle evaluation
> An experimental setting in which the loop is told the right answer, or told when its answer is right (usually to stop retrying). Oracle results show the ceiling of a method if a perfect verifier existed; they say little about the method without one.

> [!PAPER] Huang et al. (2023), Cannot Self-Correct Yet · Table 3 · page 4
> [![Table 3 of the paper: GPT-3.5 and GPT-4 on GSM8K, CommonSenseQA and HotpotQA with intrinsic self-correction; GPT-3.5 standard prompting 75.9, 75.8 and 26.0 falls to 74.7, 41.8 and 25.0 after two rounds; GPT-4 95.5, 82.0 and 49.0 falls to 89.0, 80.0 and 43.0; the number of calls rises from 1 to 5](https://ishwarj.com/img/agents/ch3-selfcorrect-table3.pngfcorrect-table3.png)](/img/agents/ch3-selfcorrect-table3.png)
>
> **Context:** the central table. Each round of self-correction is two more calls (a critique and a revision), so round 2 costs five calls.
>
> **What it says:** accuracy falls on every benchmark for both models. GPT-3.5 on CommonSenseQA collapses from 75.8 to 38.1 after one round; GPT-4 on GSM8K slides from 95.5 to 89.0 after two. With an oracle label deciding when to stop (their Table 2) the same loop *gains* up to 14 points, which is where the earlier optimistic numbers came from.
>
> **Why it matters:** five calls, worse answers. The authors' explanation (page 4) is that "LLMs cannot properly judge the correctness of their reasoning": on GSM8K GPT-3.5 kept its answer 74.7% of the time, and when it changed, it changed a right answer to a wrong one more often than the reverse. A retry loop on reasoning tasks needs a judge with information the judged model lacks, unless the model was trained for self-correction and that has been measured on your task.

The two results fit together: Reflexion's gains came where a machine ran tests; Huang et al.'s losses came where the model had nothing but itself. Self-Refine (Madaan et al., 2023) found the same split from the other side: large gains on style and coverage tasks a model can see in its own text (dialogue, sentiment rewriting) and almost none on maths. CRITIC (Gou et al., 2023) supplied the constructive version: let the critic call tools (a search engine, a Python interpreter), and the gains come back; remove the tools and they mostly disappear. The research notes below give the tables.

The careful conclusion is conditional. Self-feedback can improve some outputs, especially where the defect is visible in the text, but it does not establish correctness. When correctness matters, use a check that is grounded in something independent of the model's own judgement.

### Our experiment: repair with and without an external check

`ch3_reflexion.py` gives gpt-5-mini ten small coding tasks with precise specifications (an expression evaluator, a CSV line parser, a cache with expiry, a cron matcher, a slug generator, an edit distance, numbers to British English words, a topological sort, a strict Roman numeral parser, a semantic version parser). Each task is attempted five times, so there are 50 first attempts.

Each task has **two test suites**:

- a **feedback suite**, the project's own tests, which a repair loop may run, show to the model and use to decide when to stop;
- a **held-out suite**: different inputs for the same rules in the specification, never shown to the model and never used to stop anything. Every number below is measured on the held-out suite.

Both suites were first checked against reference solutions written for the script (all 20 pass), so a failure means the code is wrong, not the test.

Then **every** first attempt, right or wrong, goes through two repair strategies, so we can count fixes and breakages:

- **self-review**: two rounds of "review your function against the specification; if you find a mistake, reply with the corrected code, otherwise reply with the same code unchanged". No test is run or shown.
- **feedback-test loop**: run the feedback suite; if it fails, show the first failing test and ask for a fix; up to two more attempts; stop as soon as the feedback suite passes.

[![Terminal output of ch3_reflexion.py: the test suites checked against reference solutions, 20 of 20; a per-task table of held-out passes out of five for the first attempt, after self-review and after the feedback-test loop; and the summary: first attempts 43 of 50 pass the held-out tests; self-review 43 of 50 with one wrong-to-right and one right-to-wrong, the code changed in 45 of 50 attempts; the feedback-test loop 48 of 50 with five wrong-to-right and none right-to-wrong, six attempts entered the loop; the feedback suite accepted 44 attempts of which one failed held-out; 160 calls, 0.2262 dollars](https://ishwarj.com/img/agents/ch3_reflexion-run.png3_reflexion-run.png)](/img/agents/ch3_reflexion-run.png)

<figure class="fig"><svg viewBox="0 0 760 238" role="img" aria-label="Our repair experiment: fifty first attempts at ten coding tasks, each sent through two rounds of self-review and through a loop driven by the project's feedback tests, all graded on held-out tests the loops never saw."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="380.0" y="22.0" text-anchor="middle">50 first attempts at 10 coding tasks, graded on held-out tests</text><text class="t-tick" x="242.0" y="70.0" text-anchor="end">first attempt</text><g class="mark"><title>first attempt: 86%</title><rect class="s2" x="250.0" y="53.0" width="258.0" height="24.0" rx="3"/></g><text class="t-val" x="514.0" y="70.0" text-anchor="start">86%</text><text class="t-tick" x="242.0" y="102.0" text-anchor="end">after 2 rounds of self-review</text><g class="mark"><title>after 2 rounds of self-review: 86%</title><rect class="s2" x="250.0" y="85.0" width="258.0" height="24.0" rx="3"/></g><text class="t-val" x="514.0" y="102.0" text-anchor="start">86%</text><text class="t-tick" x="242.0" y="134.0" text-anchor="end">after the feedback-test loop</text><g class="mark"><title>after the feedback-test loop: 96%</title><rect class="s2" x="250.0" y="117.0" width="288.0" height="24.0" rx="3"/></g><text class="t-val" x="544.0" y="134.0" text-anchor="start">96%</text><text class="t-muted" x="20.0" y="162.0" text-anchor="start">all 50 first attempts went through both strategies, the right ones too. Self-review changed the</text><text class="t-muted" x="20.0" y="179.0" text-anchor="start">code in 45 of 50 attempts; it fixed 1 and broke 1. The feedback loop only touched the 6</text><text class="t-muted" x="20.0" y="196.0" text-anchor="start">attempts its tests rejected; it fixed 5 and broke 0. Its check was not perfect: 1 of the 44 attempts</text><text class="t-muted" x="20.0" y="213.0" text-anchor="start">the feedback suite accepted failed the held-out tests. 160 API calls, $0.23, gpt-5-mini.</text></svg><figcaption>Fifty first attempts graded on held-out tests: self-review left the pass rate at 86% (one fix, one breakage); the feedback-test loop raised it to 96% (five fixes, none broken).</figcaption></figure>

| | Held-out pass rate | Wrong to right | Right to wrong |
|---|---|---|---|
| First attempt | 43/50 (86%) | | |
| After two rounds of self-review | 43/50 (86%) | 1 | 1 |
| After the feedback-test loop | 48/50 (96%) | 5 | 0 |

What happened, concretely:

- **Self-review was busy but not useful.** It changed the code in 45 of 50 attempts (38 already right), yet the pass rate did not move: it fixed one cron matcher and broke one correct cache.
- **The feedback loop only touched what its check rejected.** Six attempts failed the feedback suite. Four of them were caches whose `get` restarted the expiry time, which the specification forbids (only `set` restarts it). The failing test read the entry at 9.9 seconds and expected it gone at 10; with that in hand the model fixed three of the four. The other fixes were an expression evaluator (unary minus with powers) and a Roman numeral parser that accepted "XCX".
- **The check was not perfect.** The feedback suite accepted 44 first attempts, and one of those failed the held-out tests (a cron matcher that rejected the range "5-7" in the day-of-week field). The loop never looked at it again, so it stayed wrong. A check's false accepts cap any loop built on it.

> [!WARNING]
> Fifty attempts at ten small tasks is a demonstration, not a benchmark, and the counts are small (one fix against five). What it does show cleanly: stopping was controlled by the loop's own feedback suite, not by the held-out grader; initially correct answers went through both strategies, so breakages are counted; and the grader never fed anything back. The test runner starts a separate `python -I` process with a ten-second timeout. That is a **demonstration runner, not a sandbox**: the generated code could still read files, use the network or exhaust memory. Run untrusted code in a real sandbox (a container or virtual machine with no network and resource limits).

### The retry formula, and what it assumes

If each attempt passes with probability $$p$$ and a verifier can tell right from wrong, the chance that at least one of $$k$$ attempts passes is

$$
P(\text{success within } k) = 1 - (1 - p)^k
$$

With $$p = 0.5$$ and $$k = 3$$ that is 88%. The expected number of attempts is $$\frac{1-(1-p)^k}{p}$$, and dividing expected cost by the success probability gives a cost per correct answer of $$c/p$$ for any $$k$$.

This formula assumes three things, and real loops often break all three:

1. **Independent attempts.** A feedback loop deliberately makes attempt 2 depend on attempt 1; that is the point of showing the failure.
2. **Constant success probability and cost.** The second attempt's chance differs from the first's (better with good feedback, sometimes worse), and its context, so its cost, is larger.
3. **A perfect verifier.** Real checks accept some wrong answers and reject some right ones, as our feedback suite did.

Treat the formula as a rough guide for independent resampling with a reliable check, and measure the real curve.

### What an acceptance is worth

A verifier has two error rates. Let

- $$p$$ = the probability that a candidate is correct,
- $$r$$ = the probability the verifier accepts a correct candidate,
- $$f$$ = the probability the verifier accepts a wrong candidate (the false-accept rate).

The probability that an accepted answer is correct follows from Bayes' rule:

$$
P(\text{correct} \mid \text{accepted}) = \frac{p\,r}{p\,r + (1-p)\,f}
$$

Worked example: $$p = 0.9$$, $$r = 1$$, $$f = 0.2$$ gives $$\frac{0.9}{0.9 + 0.1 \times 0.2} = \frac{0.9}{0.92} \approx 97.8\%$$. A false-accept rate of 20% does **not** cap precision at 80%: precision depends on how many wrong candidates reach the verifier in the first place. The same verifier with $$p = 0.3$$ gives $$\frac{0.3}{0.3 + 0.7 \times 0.2} \approx 68.2\%$$. An acceptance means less on hard tasks, where most candidates are wrong.

[![Terminal output of ch3_cost.py part five: the formula P of correct given accepted equals p r over p r plus one minus p times f, a table for p of 0.9, 0.7, 0.5 and 0.3 with three verifiers, including 97.8 percent for p 0.9, r 1 and f 0.2, and 68.2 percent for p 0.3; then best-of-n returning the first accepted candidate for p 0.3 and 0.7 and n of 1, 3, 5 and 10, showing that the chance of returning something rises with n while the precision of what is returned stays at 68.2 and 92.1 percent](https://ishwarj.com/img/agents/ch3_cost_verifier-run.pngst_verifier-run.png)](/img/agents/ch3_cost_verifier-run.png)

<figure class="fig"><svg viewBox="0 0 760 300" role="img" aria-label="Probability that an accepted answer is correct, against the share of candidates that are correct, for two verifiers and for no verifier."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><line class="grid" x1="70.0" y1="250.0" x2="520.0" y2="250.0"/><text class="t-tick" x="62.0" y="254.0" text-anchor="end">0</text><line class="grid" x1="70.0" y1="196.5" x2="520.0" y2="196.5"/><text class="t-tick" x="62.0" y="200.5" text-anchor="end">25</text><line class="grid" x1="70.0" y1="143.0" x2="520.0" y2="143.0"/><text class="t-tick" x="62.0" y="147.0" text-anchor="end">50</text><line class="grid" x1="70.0" y1="89.5" x2="520.0" y2="89.5"/><text class="t-tick" x="62.0" y="93.5" text-anchor="end">75</text><line class="grid" x1="70.0" y1="36.0" x2="520.0" y2="36.0"/><text class="t-tick" x="62.0" y="40.0" text-anchor="end">100</text><text class="t-tick" x="70.0" y="270.0" text-anchor="middle">10</text><text class="t-tick" x="182.5" y="270.0" text-anchor="middle">30</text><text class="t-tick" x="295.0" y="270.0" text-anchor="middle">50</text><text class="t-tick" x="407.5" y="270.0" text-anchor="middle">70</text><text class="t-tick" x="520.0" y="270.0" text-anchor="middle">90</text><line class="axis" x1="70.0" y1="250.0" x2="520.0" y2="250.0"/><polyline class="edge-1" points="70.0,102.4 126.2,71.7 182.5,58.4 238.8,50.9 295.0,46.2 351.2,42.9 407.5,40.5 463.8,38.6 520.0,37.2"/><g class="mark"><title>f = 0.05, r = 1, 10: 69</title><circle class="s1 ring" cx="70.0" cy="102.4" r="4.5"/></g><g class="mark"><title>f = 0.05, r = 1, 20: 83</title><circle class="s1 ring" cx="126.2" cy="71.7" r="4.5"/></g><g class="mark"><title>f = 0.05, r = 1, 30: 90</title><circle class="s1 ring" cx="182.5" cy="58.4" r="4.5"/></g><g class="mark"><title>f = 0.05, r = 1, 40: 93</title><circle class="s1 ring" cx="238.8" cy="50.9" r="4.5"/></g><g class="mark"><title>f = 0.05, r = 1, 50: 95</title><circle class="s1 ring" cx="295.0" cy="46.2" r="4.5"/></g><g class="mark"><title>f = 0.05, r = 1, 60: 97</title><circle class="s1 ring" cx="351.2" cy="42.9" r="4.5"/></g><g class="mark"><title>f = 0.05, r = 1, 70: 98</title><circle class="s1 ring" cx="407.5" cy="40.5" r="4.5"/></g><g class="mark"><title>f = 0.05, r = 1, 80: 99</title><circle class="s1 ring" cx="463.8" cy="38.6" r="4.5"/></g><g class="mark"><title>f = 0.05, r = 1, 90: 99</title><circle class="s1 ring" cx="520.0" cy="37.2" r="4.5"/></g><polyline class="edge-2" points="70.0,173.6 126.2,131.1 182.5,104.1 238.8,85.4 295.0,71.7 351.2,61.2 407.5,52.9 463.8,46.2 520.0,40.7"/><g class="mark"><title>f = 0.2, r = 1, 10: 36</title><circle class="s2 ring" cx="70.0" cy="173.6" r="4.5"/></g><g class="mark"><title>f = 0.2, r = 1, 20: 56</title><circle class="s2 ring" cx="126.2" cy="131.1" r="4.5"/></g><g class="mark"><title>f = 0.2, r = 1, 30: 68</title><circle class="s2 ring" cx="182.5" cy="104.1" r="4.5"/></g><g class="mark"><title>f = 0.2, r = 1, 40: 77</title><circle class="s2 ring" cx="238.8" cy="85.4" r="4.5"/></g><g class="mark"><title>f = 0.2, r = 1, 50: 83</title><circle class="s2 ring" cx="295.0" cy="71.7" r="4.5"/></g><g class="mark"><title>f = 0.2, r = 1, 60: 88</title><circle class="s2 ring" cx="351.2" cy="61.2" r="4.5"/></g><g class="mark"><title>f = 0.2, r = 1, 70: 92</title><circle class="s2 ring" cx="407.5" cy="52.9" r="4.5"/></g><g class="mark"><title>f = 0.2, r = 1, 80: 95</title><circle class="s2 ring" cx="463.8" cy="46.2" r="4.5"/></g><g class="mark"><title>f = 0.2, r = 1, 90: 98</title><circle class="s2 ring" cx="520.0" cy="40.7" r="4.5"/></g><polyline class="edge-3" points="70.0,228.6 126.2,207.2 182.5,185.8 238.8,164.4 295.0,143.0 351.2,121.6 407.5,100.2 463.8,78.8 520.0,57.4"/><g class="mark"><title>no verifier, 10: 10</title><circle class="s3 ring" cx="70.0" cy="228.6" r="4.5"/></g><g class="mark"><title>no verifier, 20: 20</title><circle class="s3 ring" cx="126.2" cy="207.2" r="4.5"/></g><g class="mark"><title>no verifier, 30: 30</title><circle class="s3 ring" cx="182.5" cy="185.8" r="4.5"/></g><g class="mark"><title>no verifier, 40: 40</title><circle class="s3 ring" cx="238.8" cy="164.4" r="4.5"/></g><g class="mark"><title>no verifier, 50: 50</title><circle class="s3 ring" cx="295.0" cy="143.0" r="4.5"/></g><g class="mark"><title>no verifier, 60: 60</title><circle class="s3 ring" cx="351.2" cy="121.6" r="4.5"/></g><g class="mark"><title>no verifier, 70: 70</title><circle class="s3 ring" cx="407.5" cy="100.2" r="4.5"/></g><g class="mark"><title>no verifier, 80: 80</title><circle class="s3 ring" cx="463.8" cy="78.8" r="4.5"/></g><g class="mark"><title>no verifier, 90: 90</title><circle class="s3 ring" cx="520.0" cy="57.4" r="4.5"/></g><line class="grid" x1="526.0" y1="37.2" x2="536.0" y2="37.2"/><rect class="s1" x="540" y="32.2" width="10" height="10" rx="2"/><text class="t-tick" x="556.0" y="41.2" text-anchor="start">f = 0.05, r = 1</text><line class="grid" x1="526.0" y1="40.7" x2="536.0" y2="54.2"/><rect class="s2" x="540" y="49.2" width="10" height="10" rx="2"/><text class="t-tick" x="556.0" y="58.2" text-anchor="start">f = 0.2, r = 1</text><line class="grid" x1="526.0" y1="57.4" x2="536.0" y2="71.2"/><rect class="s3" x="540" y="66.2" width="10" height="10" rx="2"/><text class="t-tick" x="556.0" y="75.2" text-anchor="start">no verifier</text><text class="t-tick" x="295.0" y="290.0" text-anchor="middle">share of candidates that are correct, p (%)</text><text class="t-tick" x="14.0" y="22.0" text-anchor="start">P(correct | accepted), %</text><text class="t-muted" x="540.0" y="205.0" text-anchor="start">p r / (p r + (1 - p) f)</text><text class="t-muted" x="540.0" y="222.0" text-anchor="start">p = 90%, f = 0.2: 97.8%</text><text class="t-muted" x="540.0" y="239.0" text-anchor="start">p = 30%, f = 0.2: 68.2%</text></svg><figcaption>P(correct | accepted) against the share of correct candidates, for two verifiers and none. The worked example, p = 90% and f = 0.2, gives 97.8%.</figcaption></figure>

> [!TIP] In production
> Name the check before you add a reflection step. "The agent reviews its answer" is not a check; tests, a compiler, a schema validator, a query that must execute, an evidence match and a person are. Feed the failure back verbatim, cap the attempts at two or three, and log right-to-wrong changes as well as wrong-to-right ones.

<details>
<summary>Research notes: Reflexion, Self-Refine, CRITIC and the self-correction debate (optional reading)</summary>

> [!PAPER] Shinn et al. (2023), Reflexion · Figure 2 · page 4
> [![Figure 2 of the Reflexion paper: a diagram in which an Actor LM produces a trajectory in short-term memory, an Evaluator LM scores it with internal or external feedback, a Self-reflection LM writes reflective text into long-term experience memory, and the Actor acts in the environment; beside it, Algorithm 1, reinforcement via self-reflection, which loops while the evaluator does not pass and the trial count is under the maximum](https://ishwarj.com/img/agents/ch3-reflexion-figure2.pngflexion-figure2.png)](/img/agents/ch3-reflexion-figure2.png)
>
> **Context:** arXiv 2303.11366, posted 20 March 2023, published at NeurIPS 2023. Three models (which can be the same model with different prompts): an Actor that acts, an Evaluator that scores a whole trial, a Self-Reflection model that explains the score in words.
>
> **What it says:** the loop is "while Evaluator not pass or t < max trials: generate a trajectory, evaluate it, generate a self-reflection, append it to memory". The Evaluator can use "external feedback" (exact-match grading, heuristics, a test suite) or "internal feedback" (an LLM judging the trajectory).
>
> **Why it matters:** the architecture is clear about the two kinds of signal, and the paper's results differ by which one was used. Read the Evaluator box before you read any number.

> [!PAPER] Shinn et al. (2023), Reflexion · Table 1 · page 7
> [![Table 1 of the Reflexion paper: pass at 1 accuracy by benchmark and language; HumanEval Python previous state of the art 65.8, GPT-4 80.1, Reflexion 91.0; HumanEval Rust 60.0 to 68.0; MBPP Python 80.1 to 77.1; MBPP Rust 70.9 to 75.4; LeetcodeHard Python 7.5 to 15.0](https://ishwarj.com/img/agents/ch3-reflexion-table1.pngeflexion-table1.png)](/img/agents/ch3-reflexion-table1.png)
>
> **Context:** GPT-4 as the base model, zero-shot; the "base strategy is a single code generation sample".
>
> **What it says:** HumanEval Python from 80.1 to 91.0; Rust from 60.0 to 68.0; LeetcodeHard (40 problems released after the model's training cutoff) from 7.5 to 15.0. And one honest minus: MBPP Python *fell* from 80.1 to 77.1.
>
> **Why it matters:** the MBPP drop is the most instructive number in the table. The authors attribute it to false positives in the self-written tests: the model wrote tests that passed wrong code, so the loop stopped on wrong answers. Where the check is weak, reflection can hurt. The 91.0 headline and the 77.1 footnote are the same mechanism measured on two datasets.

> [!PAPER] Shinn et al. (2023), Reflexion · Section 4.3 · page 7
> [![A paragraph of the Reflexion paper's programming section with the highlighted phrase: self-generated unit test suites; it describes generating tests with chain-of-thought prompting, filtering them for syntactic validity, sampling up to six, and a memory limit of one experience](https://ishwarj.com/img/agents/ch3-reflexion-tests.pngreflexion-tests.png)](/img/agents/ch3-reflexion-tests.png)
>
> **Context:** how the programming experiments were graded. The model writes its own unit tests from the problem statement before writing the code; the tests are filtered for syntax and up to six are kept; the code is run against them; failures are fed back.
>
> **What it says:** programming "presents a unique opportunity to use more grounded self-evaluation practices such as self-generated unit test suites"; the loop keeps "a max memory limit of 1 experience".
>
> **Why it matters:** the critic here is a test runner, which is external even though the tests were written by the model. That is a middle category worth naming: *model-written checks executed by a machine*. It is weaker than human-written tests (the model may test the wrong thing) and far stronger than the model re-reading its code.

> [!DEFINITION] pass@k
> The share of tasks for which at least one of k independent attempts is correct, as judged by tests. pass@1 is the single-attempt rate. Reflexion reports pass@1 because its retries are driven by self-written tests rather than the hidden ones, so the final submission is still one attempt against the real grader.

The paper's other results follow the same rule. On ALFWorld the evaluator was a heuristic that detected repeated actions and over-long trajectories, external to the model, and success rose by about 22 points over twelve trials. On HotpotQA the evaluator was exact match against the gold answer, which is an oracle; Huang et al. later argued that most of the gain on such tasks comes from the oracle rather than the reflection.

<figure class="fig"><svg viewBox="0 0 760 300" role="img" aria-label="Probability of success within k attempts when a verifier can tell right from wrong, for three per-attempt pass rates."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><line class="grid" x1="70.0" y1="250.0" x2="520.0" y2="250.0"/><text class="t-tick" x="62.0" y="254.0" text-anchor="end">0</text><line class="grid" x1="70.0" y1="196.5" x2="520.0" y2="196.5"/><text class="t-tick" x="62.0" y="200.5" text-anchor="end">25</text><line class="grid" x1="70.0" y1="143.0" x2="520.0" y2="143.0"/><text class="t-tick" x="62.0" y="147.0" text-anchor="end">50</text><line class="grid" x1="70.0" y1="89.5" x2="520.0" y2="89.5"/><text class="t-tick" x="62.0" y="93.5" text-anchor="end">75</text><line class="grid" x1="70.0" y1="36.0" x2="520.0" y2="36.0"/><text class="t-tick" x="62.0" y="40.0" text-anchor="end">100</text><text class="t-tick" x="70.0" y="270.0" text-anchor="middle">1</text><text class="t-tick" x="182.5" y="270.0" text-anchor="middle">2</text><text class="t-tick" x="295.0" y="270.0" text-anchor="middle">3</text><text class="t-tick" x="520.0" y="270.0" text-anchor="middle">5</text><line class="axis" x1="70.0" y1="250.0" x2="520.0" y2="250.0"/><polyline class="edge-1" points="70.0,185.8 182.5,140.9 295.0,109.4 520.0,72.0"/><g class="mark"><title>p = 30% per attempt, 1: 30</title><circle class="s1 ring" cx="70.0" cy="185.8" r="4.5"/></g><g class="mark"><title>p = 30% per attempt, 2: 51</title><circle class="s1 ring" cx="182.5" cy="140.9" r="4.5"/></g><g class="mark"><title>p = 30% per attempt, 3: 66</title><circle class="s1 ring" cx="295.0" cy="109.4" r="4.5"/></g><g class="mark"><title>p = 30% per attempt, 5: 83</title><circle class="s1 ring" cx="520.0" cy="72.0" r="4.5"/></g><polyline class="edge-2" points="70.0,143.0 182.5,89.5 295.0,62.8 520.0,42.7"/><g class="mark"><title>p = 50% per attempt, 1: 50</title><circle class="s2 ring" cx="70.0" cy="143.0" r="4.5"/></g><g class="mark"><title>p = 50% per attempt, 2: 75</title><circle class="s2 ring" cx="182.5" cy="89.5" r="4.5"/></g><g class="mark"><title>p = 50% per attempt, 3: 88</title><circle class="s2 ring" cx="295.0" cy="62.8" r="4.5"/></g><g class="mark"><title>p = 50% per attempt, 5: 97</title><circle class="s2 ring" cx="520.0" cy="42.7" r="4.5"/></g><polyline class="edge-3" points="70.0,100.2 182.5,55.3 295.0,41.8 520.0,36.5"/><g class="mark"><title>p = 70% per attempt, 1: 70</title><circle class="s3 ring" cx="70.0" cy="100.2" r="4.5"/></g><g class="mark"><title>p = 70% per attempt, 2: 91</title><circle class="s3 ring" cx="182.5" cy="55.3" r="4.5"/></g><g class="mark"><title>p = 70% per attempt, 3: 97</title><circle class="s3 ring" cx="295.0" cy="41.8" r="4.5"/></g><g class="mark"><title>p = 70% per attempt, 5: 100</title><circle class="s3 ring" cx="520.0" cy="36.5" r="4.5"/></g><line class="grid" x1="526.0" y1="36.5" x2="536.0" y2="36.5"/><rect class="s3" x="540" y="31.5" width="10" height="10" rx="2"/><text class="t-tick" x="556.0" y="40.5" text-anchor="start">p = 70% per attempt</text><line class="grid" x1="526.0" y1="42.7" x2="536.0" y2="53.5"/><rect class="s2" x="540" y="48.5" width="10" height="10" rx="2"/><text class="t-tick" x="556.0" y="57.5" text-anchor="start">p = 50% per attempt</text><line class="grid" x1="526.0" y1="72.0" x2="536.0" y2="72.0"/><rect class="s1" x="540" y="67.0" width="10" height="10" rx="2"/><text class="t-tick" x="556.0" y="76.0" text-anchor="start">p = 30% per attempt</text><text class="t-tick" x="295.0" y="290.0" text-anchor="middle">attempts allowed (k)</text><text class="t-tick" x="14.0" y="22.0" text-anchor="start">P(success within k), %</text><text class="t-muted" x="540.0" y="205.0" text-anchor="start">P = 1 - (1 - p)^k; assumes</text><text class="t-muted" x="540.0" y="222.0" text-anchor="start">independent attempts, fixed p,</text><text class="t-muted" x="540.0" y="239.0" text-anchor="start">and a perfect verifier</text></svg><figcaption>Probability of success within k attempts for per-attempt pass rates of 30, 50 and 70 percent, under the formula's assumptions: independent attempts, the same pass rate on every attempt and a perfect verifier.</figcaption></figure>

> [!PAPER] Huang et al. (2023), Cannot Self-Correct Yet · Abstract · page 1
> [![Abstract of the paper Large Language Models Cannot Self-Correct Reasoning Yet with highlighted phrases: intrinsic self-correction; struggle to self-correct their responses without external feedback; performance even degrades after self-correction](https://ishwarj.com/img/agents/ch3-selfcorrect-abstract.pngorrect-abstract.png)](/img/agents/ch3-selfcorrect-abstract.png)
>
> **Context:** arXiv 2310.01798 by Jie Huang, Xinyun Chen, Denny Zhou and colleagues at Google DeepMind, posted 3 October 2023, published at ICLR 2024. It re-examines the self-correction results of 2023 (Reflexion and Self-Refine among them) and finds that the impressive ones used an oracle: the ground-truth label told the loop when to stop.
>
> **What it says:** in "intrinsic self-correction", where "an LLM attempts to correct its initial responses based solely on its inherent capabilities, without the crutch of external feedback", models "struggle to self-correct their responses without external feedback, and at times, their performance even degrades".
>
> **Why it matters:** this is the paper that separates the two kinds of reflection in Section 3.5. For the prompted models of 2023, asking a model to check its own reasoning acted less like verification and more like a second sample with a nudge to change its mind. Section 3.8 shows that training can change this.

> [!PAPER] Madaan et al. (2023), Self-Refine · Table 1 · page 5
> [![Table 1 of the Self-Refine paper: base against Self-Refine for three models on seven tasks; dialogue response for GPT-4 rises from 25.4 to 74.6, code optimisation from 27.3 to 36.0, sentiment reversal from 3.8 to 36.2; math reasoning barely moves, 64.1 to 64.1, 74.8 to 75.0 and 92.9 to 93.1](https://ishwarj.com/img/agents/ch3-selfrefine-table1.pnglfrefine-table1.png)](/img/agents/ch3-selfrefine-table1.png)
>
> **Context:** arXiv 2303.17651, posted 30 March 2023, published at NeurIPS 2023. Metrics differ by task: human or GPT-4 preference for the writing tasks, a solve rate for maths, the share of programs made faster for code optimisation.
>
> **What it says:** large gains on preference tasks (dialogue response for GPT-4 from 25.4 to 74.6, sentiment reversal from 3.8 to 36.2), a solid gain on code optimisation (27.3 to 36.0), and almost nothing on maths (92.9 to 93.1 for GPT-4, 64.1 to 64.1 for GPT-3.5).
>
> **Why it matters:** the pattern of gains is the finding. Where quality is a matter of taste and coverage (is the reply polite, does the sentence contain all the required words), a model can see what is missing and add it. Where quality is correctness, it often cannot see the error, and here the numbers barely moved.

> [!PAPER] Madaan et al. (2023), Self-Refine · Section 3.3 · page 5
> [![A paragraph of the Self-Refine paper with the highlighted phrase: inability to accurately identify whether there is any error; the text explains that in maths errors can be limited to a single line and a consistent-looking reasoning chain can deceive the model](https://ishwarj.com/img/agents/ch3-selfrefine-math.pngselfrefine-math.png)](/img/agents/ch3-selfrefine-math.png)
>
> **Context:** the authors' explanation of the maths result.
>
> **What it says:** "the modest performance gains in Math Reasoning can be traced back to the inability to accurately identify whether there is any error", and "a consistent-looking reasoning chain can deceive LLMs".
>
> **Why it matters:** the authors of the self-critique paper state the limit of self-critique: it needs an error the critic can see. Huang et al. generalised this six months later into "cannot self-correct reasoning yet". The two papers agree; only the headlines differ.

> [!PAPER] Gou et al. (2023), CRITIC · Abstract · page 1
> [![Abstract of the CRITIC paper with highlighted phrases: using a search engine for fact-checking, or a code interpreter for debugging; crucial importance of external feedback](https://ishwarj.com/img/agents/ch3-critic-abstract.pngcritic-abstract.png)](/img/agents/ch3-critic-abstract.png)
>
> **Context:** arXiv 2305.11738, posted 19 May 2023, published at ICLR 2024. Tasks: free-form question answering (critic: a search engine), mathematical program synthesis (critic: a Python interpreter), toxicity reduction (critic: a toxicity classifier).
>
> **What it says:** humans "utilize external tools to cross-check and refine their initial content, like using a search engine for fact-checking, or a code interpreter for debugging"; the results highlight "the crucial importance of external feedback in promoting the ongoing self-improvement of LLMs".
>
> **Why it matters:** the critic is a tool call. The model writes the critique, but the facts in the critique come from outside, and that is what makes it a critique rather than a second guess.

> [!PAPER] Gou et al. (2023), CRITIC · Figure 1 · page 2
> [![Figure 1 of the CRITIC paper: a black-box LLM produces an output; a verify step consults external tools such as a knowledge base, a code interpreter, a calculator, a search engine and text APIs and produces critiques; a correct step revises the output; the verify-then-correct cycle repeats](https://ishwarj.com/img/agents/ch3-critic-figure1.png-critic-figure1.png)](/img/agents/ch3-critic-figure1.png)
>
> **Context:** the two-step framework: verify with tools, then correct; repeat.
>
> **What it says:** the "verify-then-correct process" can be iterated "to enable continuous improvements".
>
> **Why it matters:** this is the production shape of reflection: an actor, a verifier that reaches outside the model, a correction, a budget of rounds. Most systems in Section 3.9 that report reflection working have this shape.

> [!PAPER] Gou et al. (2023), CRITIC · Table 2 · page 7
> [![Table 2 of the CRITIC paper, mathematical program synthesis results on GSM8k, SVAMP and TabMWP for three models; for gpt-3.5-turbo, program of thoughts 72.5 on GSM8k, CRITIC 78.2, CRITIC without tool 77.0, oracle CRITIC 83.9; for text-davinci-003, program of thoughts 70.1, CRITIC 72.2, without tool 68.3 which is below the baseline, and on SVAMP CRITIC 80.7 against a baseline of 84.0](https://ishwarj.com/img/agents/ch3-critic-table2.png3-critic-table2.png)](/img/agents/ch3-critic-table2.png)
>
> **Context:** the "w/o Tool" row is the ablation that matters: the same verify-then-correct loop with the interpreter removed, so the critic is the model alone.
>
> **What it says:** with gpt-3.5-turbo on GSM8k, CRITIC improves the program-of-thoughts baseline from 72.5 to 78.2, and without the tool to 77.0. With text-davinci-003, the tool version gains two points (70.1 to 72.2) and the tool-less version *loses* two (68.3); on SVAMP even the tool version lost 3.3 points. The starred oracle rows (correct only the wrong answers) show the ceiling: 83.9 and 77.4.
>
> **Why it matters:** three readings. The tool is what makes the gain reliable: remove it and the loop can go negative. The gain over an already-strong baseline is modest (two to six points), not the twenty of the headlines. And the gap to the oracle row is the cost of a critic that sometimes "fixes" right answers, which is Huang et al.'s mechanism again.

</details>

## 3.6 Choosing among candidates: best-of-n and search

**Row changed: verification and selection.**

Without tools, the three direct answers to the Porto and Leeds question were "6 years", "9 years" and "5": a vote would only pick the most popular invention. On Everest, chain of thought gave 2,953 three times: a vote would be unanimous and wrong. Choosing among candidates helps only when the candidates contain the right answer *and* something can recognise it.

With the tools, the picture changes. Run the structured loop five times on the depot question and keep an answer only if a check passes (the evidence entries exist, every number used appears in an observation, the arithmetic recomputes). Now the selection has a real signal.

> [!DEFINITION] Best-of-n
> Generating n complete answers to the same task, independently and usually in parallel, and returning the one a verifier prefers (or the first that passes). Latency is about that of one attempt; tokens are about n times one attempt; the quality of the result depends on the verifier.

> [!DEFINITION] Verifier
> Anything that can score a complete answer without knowing the right answer in advance: a test suite, a compiler, a type checker, a schema validator, an evidence check, a trained scoring model, a prompted model (weaker), or a person (slower). Self-consistency uses agreement with the other samples as its verifier.

The idea predates the agent literature. Cobbe and colleagues at OpenAI trained a model to judge solutions to grade-school maths problems and used it to pick among many samples.

> [!PAPER] Cobbe et al. (2021), Training Verifiers · Abstract · page 1
> [![Abstract of the paper Training Verifiers to Solve Math Word Problems with highlighted phrases: training verifiers to judge the correctness of model completions; at test time, generate many candidate solutions and select the one ranked highest by the verifier](https://ishwarj.com/img/agents/ch3-verifiers-abstract.pngifiers-abstract.png)](/img/agents/ch3-verifiers-abstract.png)
>
> **Context:** arXiv 2110.14168, posted 27 October 2021 by Karl Cobbe and colleagues at OpenAI. The same paper introduced GSM8K, the grade-school maths set used by most chain-of-thought papers afterwards.
>
> **What it says:** "we propose training verifiers to judge the correctness of model completions. At test time, we generate many candidate solutions and select the one ranked highest by the verifier." Verification "significantly improves performance on GSM8K" and scaled better with more data than fine-tuning alone.
>
> **Why it matters:** this is best-of-n with a trained verifier, five months before self-consistency, and the ancestor of the "verifier" and "reward model" ideas in later reasoning-model training. It is also the comparison point in Wei et al.'s abstract ("finetuned GPT-3 with a verifier").

Best-of-n has a subtle limit, shown in the second table of the verifier screenshot above. If you return the first accepted candidate, more candidates raise the chance of returning *something*, but under independent candidates the precision of what you return stays at $$\frac{p\,r}{p\,r + (1-p)\,f}$$, whatever *n* is. Two things make it worse in practice: correlated errors (the same wrong answer in many samples, which a lenient verifier then sees many times), and picking the top-scored candidate when the scorer has blind spots. Section 3.8 adds a further finding: how much extra sampling helps depends on how hard the question is.

### Tree search

**Tree of Thoughts** (Shunyu Yao and colleagues, May 2023) searched over partial solutions instead of complete ones: propose several next steps, score each ("sure", "maybe", "impossible"), expand only the promising ones.

> [!DEFINITION] Tree of thoughts (ToT)
> A search over partial solutions. Each node is a thought (one equation in a puzzle, one plan for a paragraph). From each kept node the model proposes several next thoughts; a second prompt scores them; a search algorithm (breadth-first with a beam, or depth-first with backtracking) decides which to expand. The answer is read off the best complete path.

<figure class="fig"><svg viewBox="0 0 760 386" role="img" aria-label="Tree of thoughts: at each level the model proposes several next steps, scores each as sure, maybe or impossible, and only the promising ones are expanded; on the right, the call and token arithmetic for breadth 3 and depth 3 and the paper's own Game of 24 numbers."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="380.0" y="22.0" text-anchor="middle">Search over reasoning: propose, score, keep the best, repeat (b = 3, depth = 3)</text><circle class="node on" cx="200.0" cy="60.0" r="9"/><text class="t-muted" x="200.0" y="44.0" text-anchor="middle">problem</text><line class="edge" x1="200.0" y1="69.0" x2="90.0" y2="121.0"/><circle class="node on" cx="90.0" cy="130.0" r="9"/><text class="t-tick" x="90.0" y="156.0" text-anchor="middle">sure</text><line class="edge" x1="200.0" y1="69.0" x2="200.0" y2="121.0"/><circle class="node on" cx="200.0" cy="130.0" r="9"/><text class="t-tick" x="200.0" y="156.0" text-anchor="middle">maybe</text><line class="edge-dim" x1="200.0" y1="69.0" x2="310.0" y2="121.0"/><circle class="node" cx="310.0" cy="130.0" r="9"/><text class="t-muted" x="310.0" y="156.0" text-anchor="middle">impossible</text><line class="edge" x1="90.0" y1="139.0" x2="50.0" y2="202.0"/><circle class="node on" cx="50.0" cy="210.0" r="8"/><line class="edge" x1="90.0" y1="139.0" x2="90.0" y2="202.0"/><circle class="node on" cx="90.0" cy="210.0" r="8"/><line class="edge-dim" x1="90.0" y1="139.0" x2="130.0" y2="202.0"/><circle class="node" cx="130.0" cy="210.0" r="8"/><line class="edge-dim" x1="200.0" y1="139.0" x2="170.0" y2="202.0"/><circle class="node" cx="170.0" cy="210.0" r="8"/><line class="edge" x1="200.0" y1="139.0" x2="200.0" y2="202.0"/><circle class="node on" cx="200.0" cy="210.0" r="8"/><line class="edge" x1="200.0" y1="139.0" x2="230.0" y2="202.0"/><circle class="node on" cx="230.0" cy="210.0" r="8"/><line class="edge" x1="90.0" y1="218.0" x2="70.0" y2="277.0"/><circle class="node" cx="70.0" cy="284.0" r="7"/><line class="edge" x1="90.0" y1="218.0" x2="90.0" y2="277.0"/><circle class="node on" cx="90.0" cy="284.0" r="7"/><line class="edge" x1="90.0" y1="218.0" x2="110.0" y2="277.0"/><circle class="node" cx="110.0" cy="284.0" r="7"/><text class="t-tick" x="90.0" y="308.0" text-anchor="middle">answer</text><text class="t-muted" x="210.0" y="330.0" text-anchor="middle">every node scored sure, maybe or impossible by the model</text><rect class="box-ghost" x="400.0" y="44.0" width="330.0" height="290.0" rx="12"/><text class="t-note" x="565.0" y="66.0" text-anchor="middle">what the tree costs (ch3_cost.py, breadth-first)</text><text class="t-tick" x="414.0" y="90.0" text-anchor="start">per level: 3 propose calls + 9 evaluation calls</text><text class="t-tick" x="414.0" y="107.0" text-anchor="start">three levels: 36 model calls for one task</text><text class="t-tick" x="414.0" y="124.0" text-anchor="start">planning numbers: 62,100 in + 4,590 out tokens</text><text class="t-tick" x="414.0" y="141.0" text-anchor="start">about 26x the cost of one chain of thought</text><text class="t-tick" x="414.0" y="158.0" text-anchor="start">and 4.5x its latency (levels are sequential)</text><text class="t-tick" x="414.0" y="175.0" text-anchor="start"></text><text class="t-tick" x="414.0" y="192.0" text-anchor="start">the paper: Game of 24, GPT-4</text><text class="t-tick" x="414.0" y="209.0" text-anchor="start">chain of thought 4%, best of 100 chains 49%,</text><text class="t-tick" x="414.0" y="226.0" text-anchor="start">tree of thoughts b=5: 74% (Table 2);</text><text class="t-tick" x="414.0" y="243.0" text-anchor="start">about 5.5k completion tokens per problem,</text><text class="t-tick" x="414.0" y="260.0" text-anchor="start">close to 100 chain-of-thought trials (Table 7)</text><text class="t-muted" x="380.0" y="356.0" text-anchor="middle">worth it when steps can be scored before the end (a puzzle state, a test, a type check) and</text><text class="t-muted" x="380.0" y="373.0" text-anchor="middle">single chains fail early; not for open-ended writing or routine tool use</text></svg><figcaption>Tree of thoughts: propose several next steps, score them, expand the promising ones; with the call arithmetic of ch3_cost.py and the paper's Game of 24 numbers.</figcaption></figure>

> [!PAPER] Yao et al. (2023), Tree of Thoughts · Table 2 and Figure 3 · page 6
> [![Table 2 of the Tree of Thoughts paper, Game of 24 results with GPT-4: input-output prompting 7.3 percent, chain of thought 4.0 percent, self-consistency with 100 samples 9.0 percent, tree of thoughts with breadth 1 45 percent and breadth 5 74 percent, iterative refinement 27 percent, best of 100 input-output samples 33 percent and best of 100 chains 49 percent; Figure 3 plots success against nodes visited and the step at which samples fail](https://ishwarj.com/img/agents/ch3-tot-table2.png/ch3-tot-table2.png)](/img/agents/ch3-tot-table2.png)
>
> **Context:** Game of 24 asks for an arithmetic expression that combines four given numbers into 24. GPT-4 on 100 hard puzzles. A thought is one intermediate equation; the value prompt judges whether the remaining numbers can still reach 24.
>
> **What it says:** chain of thought solved 4% and self-consistency over a hundred chains 9%; a tree with breadth 1 solved 45% and breadth 5 solved 74%. Even an oracle that picks the best of 100 independent chains reached only 49%.
>
> **Why it matters:** this is the largest gain from a pattern anywhere in this chapter, and the task explains it: the search space is small and structured, every intermediate state can be scored cheaply and reliably, and a wrong first step is unrecoverable. Those three properties are the test for whether a tree will help you.

The task explains the gain: a small search space, cheap and fairly reliable scoring of partial states, and a first move that decides everything. The paper's own accounting puts the tree at about $0.74 per puzzle against $0.47 for a hundred chains (2023 GPT-4 prices), so per solved puzzle the two cost about the same ($1.00 against $0.96). On a task a single chain solves 90% of the time, the same tree would mostly be waste. LATS (Zhou et al., 2023) extended the search to actions and environment feedback; its results, read in the notes, came with an oracle setup and around 67 model calls per question.

> [!DEFINITION] Test-time compute
> Computation spent while answering, to improve one answer, rather than in training: more samples, a longer chain, a tree, more retries. Every pattern in this chapter spends it in some way.

<figure class="fig"><svg viewBox="0 0 760 336" role="img" aria-label="Cost and latency of seven patterns for one task, relative to a direct answer, from the planning numbers in ch3_cost.py: chain of thought, ReAct with five tool steps, plan-and-execute with five parallel steps, tree of thoughts with breadth 3 and depth 3, and best-of-five with a model verifier or a free checker."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="380.0" y="22.0" text-anchor="middle">One task, seven patterns: cost and latency relative to a direct answer (ch3_cost.py)</text><text class="t-note" x="200.0" y="52.0" text-anchor="middle">cost per task (x direct)</text><text class="t-note" x="560.0" y="52.0" text-anchor="middle">latency per task (x direct)</text><text class="t-tick" x="284.0" y="84.0" text-anchor="end">direct answer</text><g class="mark"><title>direct answer: 1.0x</title><rect class="s3" x="292.0" y="69.0" width="2.0" height="20.0" rx="3"/></g><text class="t-val" x="300.0" y="84.0" text-anchor="start">1.0x</text><text class="t-tick" x="284.0" y="112.0" text-anchor="end">chain of thought</text><g class="mark"><title>chain of thought: 2.0x</title><rect class="s3" x="292.0" y="97.0" width="3.1" height="20.0" rx="3"/></g><text class="t-val" x="301.1" y="112.0" text-anchor="start">2.0x</text><text class="t-tick" x="284.0" y="140.0" text-anchor="end">ReAct, 5 tool steps</text><g class="mark"><title>ReAct, 5 tool steps: 12.3x</title><rect class="s3" x="292.0" y="125.0" width="19.5" height="20.0" rx="3"/></g><text class="t-val" x="317.5" y="140.0" text-anchor="start">12.3x</text><text class="t-tick" x="284.0" y="168.0" text-anchor="end">plan-and-execute, 5 parallel steps</text><g class="mark"><title>plan-and-execute, 5 parallel steps: 7.4x</title><rect class="s3" x="292.0" y="153.0" width="11.7" height="20.0" rx="3"/></g><text class="t-val" x="309.7" y="168.0" text-anchor="start">7.4x</text><text class="t-tick" x="284.0" y="196.0" text-anchor="end">tree of thoughts, b=3, d=3</text><g class="mark"><title>tree of thoughts, b=3, d=3: 51.5x</title><rect class="s3" x="292.0" y="181.0" width="81.8" height="20.0" rx="3"/></g><text class="t-val" x="379.8" y="196.0" text-anchor="start">51.5x</text><text class="t-tick" x="284.0" y="224.0" text-anchor="end">best-of-5, model verifier</text><g class="mark"><title>best-of-5, model verifier: 15.8x</title><rect class="s3" x="292.0" y="209.0" width="25.0" height="20.0" rx="3"/></g><text class="t-val" x="323.0" y="224.0" text-anchor="start">15.8x</text><text class="t-tick" x="284.0" y="252.0" text-anchor="end">best-of-5, free checker</text><g class="mark"><title>best-of-5, free checker: 9.8x</title><rect class="s3" x="292.0" y="237.0" width="15.6" height="20.0" rx="3"/></g><text class="t-val" x="313.6" y="252.0" text-anchor="start">9.8x</text><text class="t-tick" x="428.0" y="84.0" text-anchor="end"></text><g class="mark"><title>: 1.0x</title><rect class="s1" x="436.0" y="69.0" width="14.1" height="20.0" rx="3"/></g><text class="t-val" x="456.1" y="84.0" text-anchor="start">1.0x</text><text class="t-tick" x="428.0" y="112.0" text-anchor="end"></text><g class="mark"><title>: 3.5x</title><rect class="s1" x="436.0" y="97.0" width="48.8" height="20.0" rx="3"/></g><text class="t-val" x="490.8" y="112.0" text-anchor="start">3.5x</text><text class="t-tick" x="428.0" y="140.0" text-anchor="end"></text><g class="mark"><title>: 12.5x</title><rect class="s1" x="436.0" y="125.0" width="176.4" height="20.0" rx="3"/></g><text class="t-val" x="618.4" y="140.0" text-anchor="start">12.5x</text><text class="t-tick" x="428.0" y="168.0" text-anchor="end"></text><g class="mark"><title>: 6.2x</title><rect class="s1" x="436.0" y="153.0" width="87.4" height="20.0" rx="3"/></g><text class="t-val" x="529.4" y="168.0" text-anchor="start">6.2x</text><text class="t-tick" x="428.0" y="196.0" text-anchor="end"></text><g class="mark"><title>: 15.5x</title><rect class="s1" x="436.0" y="181.0" width="218.2" height="20.0" rx="3"/></g><text class="t-val" x="660.2" y="196.0" text-anchor="start">15.5x</text><text class="t-tick" x="428.0" y="224.0" text-anchor="end"></text><g class="mark"><title>: 4.4x</title><rect class="s1" x="436.0" y="209.0" width="61.9" height="20.0" rx="3"/></g><text class="t-val" x="503.9" y="224.0" text-anchor="start">4.4x</text><text class="t-tick" x="428.0" y="252.0" text-anchor="end"></text><g class="mark"><title>: 3.5x</title><rect class="s1" x="436.0" y="237.0" width="48.8" height="20.0" rx="3"/></g><text class="t-val" x="490.8" y="252.0" text-anchor="start">3.5x</text><text class="t-muted" x="30.0" y="276.0" text-anchor="start">the tree is the outlier on both axes; best-of-n is cheap in time (parallel) and expensive in tokens;</text><text class="t-muted" x="30.0" y="293.0" text-anchor="start">plan-and-execute buys back latency by running steps in parallel. None of these numbers says which</text><text class="t-muted" x="30.0" y="310.0" text-anchor="start">pattern is right: divide each by its accuracy first (Section 3.9).</text></svg><figcaption>Cost and latency of seven patterns relative to a direct answer, under the planning numbers of ch3_cost.py. Divide by accuracy before deciding.</figcaption></figure>

**What to measure before choosing among candidates:** the single-attempt pass rate (near 0 or near 1, selection cannot help), the verifier's false-accept rate, and, for a tree, the step at which failed chains go wrong and the nodes expanded per solved task.

<details>
<summary>Research notes: Tree of Thoughts and LATS in detail (optional reading)</summary>

> [!PAPER] Yao et al. (2023), Tree of Thoughts · Figure 1 · page 2
> [![Figure 1 of the Tree of Thoughts paper: four schematics from input to output; input-output prompting as one arrow, chain of thought as a line of boxes, self-consistency as several lines ending in a majority vote, and tree of thoughts as a branching tree in which some nodes are shaded green and continued and others shaded red and abandoned](https://ishwarj.com/img/agents/ch3-tot-figure1.pngch3-tot-figure1.png)](/img/agents/ch3-tot-figure1.png)
>
> **Context:** arXiv 2305.10601, posted 17 May 2023, published at NeurIPS 2023. Each rectangle is a thought, "a coherent language sequence that serves as an intermediate step toward problem solving".
>
> **What it says:** the four pictures are the first three patterns of this chapter plus the new one. Self-consistency (c) already samples several chains but only compares their endings; tree of thoughts (d) compares them at every step and prunes early.
>
> **Why it matters:** pruning early is where the saving and the gain both come from. The paper's error analysis found that about 60% of chain-of-thought samples on Game of 24 had already failed after their first step; a tree that scores the first step throws those away before paying for the rest.

> [!PAPER] Yao et al. (2023), Tree of Thoughts · Appendix B.3 · page 14
> [![A paragraph and table from the appendix of the Tree of Thoughts paper on cost: solving a Game of 24 problem with the tree requires 5.5 thousand completion tokens, close to 100 chain-of-thought trials; the cost table lists best of 100 input-output at 0.13 dollars and 33 percent, best of 100 chains at 0.47 dollars and 49 percent, and the tree at 0.74 dollars and 74 percent per case](https://ishwarj.com/img/agents/ch3-tot-cost.pngts/ch3-tot-cost.png)](/img/agents/ch3-tot-cost.png)
>
> **Context:** the authors' own cost accounting, added in the revised version.
>
> **What it says:** the tree "requires 5.5k completion tokens, close to 100 CoT trials (6.7k tokens)" and cost $0.74 per puzzle at 2023 GPT-4 prices against $0.47 for a hundred chains; in general it "could require 5-100 times more generated tokens than CoT". Their advice: use it "on tasks requiring deliberate reasoning, on which CoT struggles".
>
> **Why it matters:** the cost per correct answer, which the paper does not compute, is $1.00 for the tree ($0.74 at 74%) against $0.96 for a hundred chains ($0.47 at 49%). On this puzzle the tree is about as cheap per solved problem as brute-force sampling and far cheaper than a single chain that almost never succeeds. On a problem a single chain solves 90% of the time, the same tree would be fifty times too expensive.

> [!DEFINITION] Monte Carlo tree search (MCTS)
> A search algorithm that builds a tree incrementally: select a promising node (balancing high-scoring nodes against rarely visited ones), expand it, estimate the value of the new children, and propagate values back up. Repeated for a budget of iterations, it concentrates effort on the branches that look best without abandoning the rest.

> [!PAPER] Zhou et al. (2023), LATS · Figure 1 · page 1
> [![Figure 1 of the LATS paper: an LLM agent sends actions to an environment and receives observations and rewards; a tree search over nodes selects the best node; evaluation and self-reflection feed values back; a memory holds context](https://ishwarj.com/img/agents/ch3-lats-figure1.pngh3-lats-figure1.png)](/img/agents/ch3-lats-figure1.png)
>
> **Context:** arXiv 2310.04406, posted 6 October 2023, published at ICML 2024. "Language Agent Tree Search" unifies reasoning (ToT), acting (ReAct) and planning (MCTS), and adds Reflexion-style verbal reflections on failed branches.
>
> **What it says:** the tree's nodes are states of a ReAct trajectory (thought, action, observation); the search "leverages an external environment and an MCTS-based search algorithm"; failed branches produce reflections that are stored in memory and shown to later branches.
>
> **Why it matters:** the environment is the judge. Where Tree of Thoughts asked the model to score its own partial solutions, LATS can use a real reward (did the code pass, did the shop return the item) wherever one exists, and only falls back to a model-written value where it does not.

> [!PAPER] Zhou et al. (2023), LATS · Table 3 · page 6
> [![Table 3 of the LATS paper, acting-based prompting on HotpotQA with GPT-3.5: ReAct 0.32, ReAct best of k 0.38, Reflexion 0.51, tree of thoughts with ReAct 0.39, RAP with ReAct 0.54, LATS with ReAct 0.63, LATS with n equals 3 0.58 and n equals 10 0.65, LATS with chain of thought plus ReAct 0.71](https://ishwarj.com/img/agents/ch3-lats-table3.pngch3-lats-table3.png)](/img/agents/ch3-lats-table3.png)
>
> **Context:** 100 HotpotQA questions, GPT-3.5, five children per expansion and up to 50 trajectories per question. The caption of the earlier table and the text admit an **oracle setup**: the environment tells the agent whether its answer is correct, so the comparison is about how well each method uses high-quality feedback, not what it achieves without it.
>
> **What it says:** ReAct 0.32, the best of 50 ReAct samples 0.38, Reflexion 0.51, and LATS 0.63 (0.71 when it may use both chain-of-thought and ReAct nodes).
>
> **Why it matters:** the gap between "best of 50 samples" (0.38) and "search over 50 trajectories" (0.63) is the value of sharing information between attempts: a tree remembers which branches failed and why, where independent samples start from nothing every time. The oracle caveat is also the warning: those numbers need a judge that is right.

> [!PAPER] Zhou et al. (2023), LATS · Table 10 · page 9
> [![Table 10 of the LATS paper: HotpotQA accuracy and average number of nodes expanded for tree of thoughts, RAP and LATS at 10, 30 and 50 trajectories; at 50 trajectories tree of thoughts reaches 0.49 with 84 nodes, RAP 0.54 with 71 nodes and LATS 0.61 with 67 nodes](https://ishwarj.com/img/agents/ch3-lats-cost.pngs/ch3-lats-cost.png)](/img/agents/ch3-lats-cost.png)
>
> **Context:** the cost comparison the authors added; each node is at least one model call.
>
> **What it says:** at 50 trajectories LATS expanded 66.65 nodes per question on average for 0.61 accuracy, against 84.05 nodes for Tree of Thoughts at 0.49.
>
> **Why it matters:** sixty-seven model calls per question. For a question that ReAct answers in five. The accuracy doubled and the cost went up by more than an order of magnitude, which is the trade this whole family offers.

</details>

## 3.7 Changing the action: code, interfaces and fixed pipelines

**Rows changed: action representation, and control across calls.**

The depot question took the structured loop four look-up turns. If the action could be a short program, one turn would do:

```python
trucks = {d: lookup_entry(d)['trucks'] for d in ['Rotterdam depot', 'Gdansk depot', 'Porto depot', 'Leeds depot']}
best = max(trucks, key=trucks.get)
print(best, trucks[best] - trucks['Leeds depot'])          # Rotterdam depot 79
```

The model writes this once; a sandbox runs it and returns one line. (This is an illustration; the chapter's scripts do not run model-written code.) That is the idea behind **code as action**.

> [!DEFINITION] Code as action
> An action format in which the model emits executable code (usually Python) that calls the available tools as functions, instead of one tool call per turn in JSON or text. The code runs in a sandbox; its output and errors come back as the observation. Loops, conditions and variables become free, and several tool calls collapse into one model turn.

> [!DEFINITION] Sandbox
> An isolated environment (a container, a virtual machine, a restricted interpreter) in which agent-written code runs with limits on time, memory, network and file access. Code as action is only as safe as its sandbox: what the sandbox can reach is the agent's real permission set.

<figure class="fig"><svg viewBox="0 0 760 392" role="img" aria-label="Code as action: three JSON tool calls take three round trips through the model, while one code action composes the same tools with a loop in a sandbox and returns one line. Below, the Agentless counterpoint: a fixed pipeline of localisation, repair and validation that beat the open-source agents of its time on SWE-bench Lite."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="190.0" y="22.0" text-anchor="middle">JSON tool calls: three round trips</text><text class="t-title" x="570.0" y="22.0" text-anchor="middle">one code action in a sandbox</text><line class="edge-dim" x1="380.0" y1="10.0" x2="380.0" y2="226.0"/><rect class="box-3" x="40.0" y="46.0" width="180.0" height="34.0" rx="6"/><text class="t-tick" x="130.0" y="67.0" text-anchor="middle">lookup_rate("USD", "EUR")</text><line class="edge" x1="220.0" y1="63.0" x2="262.0" y2="63.0" marker-end="url(#ah)"/><rect class="box" x="262.0" y="46.0" width="90.0" height="34.0" rx="6"/><text class="t-tick" x="307.0" y="67.0" text-anchor="middle">result</text><path class="edge" d="M307.0,80.0 L307.0,90.0 L130.0,90.0 L130.0,98.0" marker-end="url(#ah)"/><rect class="box-3" x="40.0" y="98.0" width="180.0" height="34.0" rx="6"/><text class="t-tick" x="130.0" y="119.0" text-anchor="middle">lookup_price("phone", "DE")</text><line class="edge" x1="220.0" y1="115.0" x2="262.0" y2="115.0" marker-end="url(#ah)"/><rect class="box" x="262.0" y="98.0" width="90.0" height="34.0" rx="6"/><text class="t-tick" x="307.0" y="119.0" text-anchor="middle">result</text><path class="edge" d="M307.0,132.0 L307.0,142.0 L130.0,142.0 L130.0,150.0" marker-end="url(#ah)"/><rect class="box-3" x="40.0" y="150.0" width="180.0" height="34.0" rx="6"/><text class="t-tick" x="130.0" y="171.0" text-anchor="middle">convert(price, rate)</text><line class="edge" x1="220.0" y1="167.0" x2="262.0" y2="167.0" marker-end="url(#ah)"/><rect class="box" x="262.0" y="150.0" width="90.0" height="34.0" rx="6"/><text class="t-tick" x="307.0" y="171.0" text-anchor="middle">result</text><text class="t-muted" x="190.0" y="212.0" text-anchor="middle">each result passes through the model</text><text class="t-muted" x="190.0" y="229.0" text-anchor="middle">before the next call can be made</text><rect class="box-3" x="410.0" y="46.0" width="320.0" height="112.0" rx="8"/><text class="t-tick" x="424.0" y="68.0" text-anchor="start">rate = lookup_rate("USD", "EUR")</text><text class="t-tick" x="424.0" y="85.0" text-anchor="start">for country in ["DE", "FR", "IT"]:</text><text class="t-tick" x="424.0" y="102.0" text-anchor="start">    p = lookup_price("phone", country)</text><text class="t-tick" x="424.0" y="119.0" text-anchor="start">    prices[country] = convert(p, rate)</text><text class="t-tick" x="424.0" y="136.0" text-anchor="start">print(min(prices.items(), key=...))</text><text class="t-muted" x="570.0" y="180.0" text-anchor="middle">one round trip; loops and conditions are free;</text><text class="t-muted" x="570.0" y="197.0" text-anchor="middle">the model reads one line back.</text><text class="t-muted" x="570.0" y="214.0" text-anchor="middle">CodeAct: up to 20 points higher success, fewer turns</text><line class="edge-dim" x1="30.0" y1="240.0" x2="730.0" y2="240.0"/><text class="t-title" x="380.0" y="264.0" text-anchor="middle">the counterpoint: Agentless, a fixed three-phase pipeline, no agent at all</text><rect class="box-4" x="40.0" y="280.0" width="200.0" height="56.0" rx="8"/><text class="t-note" x="140.0" y="302.0" text-anchor="middle">localise</text><text class="t-tick" x="140.0" y="322.0" text-anchor="middle">files &gt; functions &gt; lines</text><line class="edge" x1="240.0" y1="308.0" x2="276.0" y2="308.0" marker-end="url(#ah)"/><rect class="box-2" x="276.0" y="280.0" width="200.0" height="56.0" rx="8"/><text class="t-note" x="376.0" y="302.0" text-anchor="middle">repair</text><text class="t-tick" x="376.0" y="322.0" text-anchor="middle">sample many patches</text><line class="edge" x1="476.0" y1="308.0" x2="512.0" y2="308.0" marker-end="url(#ah)"/><rect class="box-3" x="512.0" y="280.0" width="200.0" height="56.0" rx="8"/><text class="t-note" x="612.0" y="302.0" text-anchor="middle">validate</text><text class="t-tick" x="612.0" y="322.0" text-anchor="middle">reproduce, test, rank</text><text class="t-muted" x="380.0" y="360.0" text-anchor="middle">SWE-bench Lite, mid-2024: 32.00% of issues resolved at $0.70 each,</text><text class="t-muted" x="380.0" y="377.0" text-anchor="middle">above every open-source agent of the time (Table 1)</text></svg><figcaption>Code as action against JSON tool calls, and the Agentless counterpoint: a fixed pipeline of localisation, repair and validation.</figcaption></figure>

> [!PAPER] Wang et al. (2024), CodeAct · Table 3 · page 5
> [![Table 3 of the CodeAct paper: success rates and average turns on M3ToolEval for open-source and closed-source models under code, JSON and text action formats; gpt-4-1106-preview reaches 74.4 percent success with code against 52.4 with JSON and 53.7 with text, in 5.5 turns against 7.6 and 7.7; claude-2 54.9 against 39.0 and 29.3](https://ishwarj.com/img/agents/ch3-codeact-table3.png-codeact-table3.png)](/img/agents/ch3-codeact-table3.png)
>
> **Context:** M3ToolEval, 82 tasks that need several tools; up to ten turns per task.
>
> **What it says:** for gpt-4-1106-preview, code actions succeeded on 74.4% of tasks in 5.5 turns on average, against 52.4% in 7.6 turns for JSON and 53.7% in 7.7 for text. For the open-source models of the time the absolute rates were low (under 15%) in every format.
>
> **Why it matters:** the pattern helps a capable model most. Also note what the format costs: a sandbox, a time limit, resource limits, and a decision about what the code may import and reach. Code as action moves the safety boundary from "which tools exist" to "what the sandbox allows", which is Chapter 9's problem.

Two other results from 2024 belong next to CodeAct, because together they say what to change first.

**The interface matters.** SWE-agent (John Yang and colleagues at Princeton) gave a model a purpose-built set of commands for working in a code repository: a file viewer that shows a window of lines, a search that summarises its matches, an edit command that checks syntax. Changing one interface element at a time moved the resolved rate by several points with the model held fixed.

> [!DEFINITION] Agent-computer interface (ACI)
> The commands an agent may run and the format in which their results are shown to it: how a file is viewed, how a search reports matches, what an edit command checks before applying a change.

In its ablations a search command that summarises matches was worth six points of resolved rate over one that lists them all, and showing a 100-line window instead of whole files was worth about five. The same paper's behavioural analysis produced a phrase worth keeping: **agents succeed quickly and fail slowly**. Successful runs finished at a median of 12 steps and $1.21; unsuccessful ones averaged 21 steps and $2.52. A long run is more likely lost than about to succeed, which is an argument for setting the step budget from successful runs.

**Sometimes no agent is needed.** Agentless (Chunqiu Steven Xia and colleagues, July 2024) fixed GitHub issues with a fixed three-phase pipeline (find the location, sample many patches, validate with tests) in which the model never chooses the next step.

> [!PAPER] Xia et al. (2024), Agentless · Abstract · page 1
> [![Abstract of the Agentless paper with highlighted phrases: do we really have to employ complex autonomous software agents; simplistic three-phase process of localization, repair, and patch validation; highest performance, 32.00 percent and 96 correct fixes, and low cost of 0.70 dollars](https://ishwarj.com/img/agents/ch3-agentless-abstract.pngntless-abstract.png)](/img/agents/ch3-agentless-abstract.png)
>
> **Context:** arXiv 2407.01489, posted 1 July 2024 and revised later that year. The paper also audits SWE-bench Lite by hand and finds issues with misleading descriptions or over-specific ground-truth patches, which it removes to make a cleaner subset.
>
> **What it says:** "Do we really have to employ complex autonomous software agents?" Agentless uses "a simplistic three-phase process of localization, repair, and patch validation, without letting the LLM decide future actions or operate with complex tools", and reached "the highest performance (32.00%, 96 correct fixes) and low cost ($0.70) compared with all existing open-source software agents".
>
> **Why it matters:** "without letting the LLM decide future actions" is the definition of a workflow rather than an agent (Chapter 1). A fixed pipeline, designed by people who knew how issues are fixed, beat the open-source agents of its time on their own benchmark.

Read the three together. CodeAct changed the action format; SWE-agent changed what the tools return; Agentless removed model-chosen control and kept sampling plus verification. None says "agents are good" or "agents are bad". When the steps of a task are known, a fixed pipeline with a check can be cheaper and more reliable; when the model must choose, give it a good interface and, where composition matters, the option to write code. Six months after Agentless, Anthropic reported 49% on SWE-bench Verified with a minimal scaffold in which the model chose every step, with a stronger model. The right amount of fixed structure appears to move with model capability and with how verifiable the task is.

<details>
<summary>Research notes: CodeAct, SWE-agent, Agentless and the SWE-bench scaffolds in detail (optional reading)</summary>

> [!PAPER] Wang et al. (2024), CodeAct · Abstract · page 1
> [![Abstract of the CodeAct paper with highlighted phrases: executable Python code to consolidate LLM agents' actions into a unified action space; up to 20 percent higher success rate](https://ishwarj.com/img/agents/ch3-codeact-abstract.pngodeact-abstract.png)](/img/agents/ch3-codeact-abstract.png)
>
> **Context:** arXiv 2402.01030, posted 1 February 2024, published at ICML 2024. The paper compares three action formats (code, JSON, text) across 17 models on an existing API benchmark and a new multi-turn tool benchmark, M3ToolEval.
>
> **What it says:** actions written as JSON or text are "limited by constrained action space ... and restricted flexibility (e.g., inability to compose multiple tools)"; code actions "consolidate LLM agents' actions into a unified action space" and reach "up to 20% higher success rate".
>
> **Why it matters:** the claim is specifically about *composition*. On the single-call benchmark (their Table 2) JSON was as good or better for the strongest closed models, because one call is one call in any format. The gain appears when a task needs several tools in sequence.

> [!PAPER] Wang et al. (2024), CodeAct · Figure 1 · page 2
> [![Figure 1 of the CodeAct paper: at the top, the same task, find the most cost-effective country to buy a phone, solved on the left with text or JSON actions, one lookup per turn with the environment returning each value, and on the right with a single code action that loops over the countries, calls the lookup functions and converts the prices; at the bottom, bar charts of success rate and average number of interaction turns for seven models by action format, with code as action highest in success and lowest in turns for most models](https://ishwarj.com/img/agents/ch3-codeact-figure1.pngcodeact-figure1.png)](/img/agents/ch3-codeact-figure1.png)
>
> **Context:** the paper's overview figure; the bar charts summarise Table 3.
>
> **What it says:** the JSON agent needs a turn per lookup and a turn per conversion; the code agent writes a `for` loop over the countries and reads one answer back. Across models, code actions have the highest success rate and the fewest turns.
>
> **Why it matters:** fewer turns means fewer model calls, a smaller context and less chance of the repetition and drift failures of Section 3.3. The loop in the code is a loop the agent did not have to run.

> [!PAPER] Anthropic, "Code execution with MCP: Building more efficient agents" (Engineering blog, November 2025)
> [![A paragraph from the post: the agent discovers tools by exploring the filesystem, listing the servers directory to find available servers, then reading the specific tool files it needs; this lets the agent load only the definitions it needs for the current task; this reduces the token usage from 150,000 tokens to 2,000 tokens, a time and cost saving of 98.7 percent](https://ishwarj.com/img/agents/ch3-anthropic-codeexec.pnghropic-codeexec.png)](/img/agents/ch3-anthropic-codeexec.png)
>
> **Context:** the post's two complaints about direct tool calling at scale are that "tool definitions overload the context window" and "intermediate tool results consume additional tokens", because "every intermediate result must pass through the model".
>
> **What it says:** "agents scale better by writing code to call tools instead": the agent reads only the tool files it needs and the data moves between tools inside the execution environment, not through the model.
>
> **Why it matters:** this is CodeAct's composition argument restated as a production cost. A transcript fetched from one system and written to another passes through the model twice in a tool-calling loop and zero times in a code action.

> [!PAPER] Yang et al. (2024), SWE-agent · Figure 1 · page 1
> [![Figure 1 of the SWE-agent paper: an LM agent connected through an agent-computer interface, with LM-friendly commands to navigate the repository, view files, search files and edit lines, and LM-friendly environment feedback, to a computer with a terminal and a filesystem](https://ishwarj.com/img/agents/ch3-sweagent-figure1.pngweagent-figure1.png)](/img/agents/ch3-sweagent-figure1.png)
>
> **Context:** arXiv 2405.15793, posted 6 May 2024, published at NeurIPS 2024. The task is SWE-bench: given a real GitHub issue and the repository, produce a patch that makes the hidden tests pass.
>
> **What it says:** the agent talks to the computer through "LM-friendly commands" (a file viewer that shows a window of lines, search commands that summarise results, an edit command that checks syntax) and receives "LM-friendly environment feedback".
>
> **Why it matters:** the model is a new kind of user with its own needs: it cannot scroll, it pays per token it reads, and it cannot see the cursor. Designing for that user is Chapter 2's tool-design principle applied to a whole computer.

> [!PAPER] Yang et al. (2024), SWE-agent · Tables 1 and 3 · page 6
> [![Tables 1 and 3 of the SWE-agent paper: on the full SWE-bench test set, retrieval-augmented generation resolves 1.31 percent with GPT-4 Turbo and 3.79 with Claude 3 Opus, a shell-only agent 11.00 percent on the Lite split, and SWE-agent 12.47 percent on the full set at 1.59 dollars average and 18.00 percent on Lite; the ablation table shows that removing linting from the edit command drops Lite from 18.0 to 15.0, iterative search drops it to 12.0, a full-file viewer to 12.7, and keeping the full history instead of the last five observations to 15.0](https://ishwarj.com/img/agents/ch3-sweagent-tables.pngsweagent-tables.png)](/img/agents/ch3-sweagent-tables.png)
>
> **Context:** GPT-4 Turbo and Claude 3 Opus, spring 2024. The second table varies one interface element at a time.
>
> **What it says:** SWE-agent resolved 12.47% of the full benchmark (18.00% of the Lite split) against 1.31% for retrieval plus generation and 11.00% for an agent with only a bare shell. In the ablations, every interface simplification cost points: an edit command without a syntax check 15.0 instead of 18.0; a search that lists every match 12.0 instead of 18.0; showing whole files 12.7; keeping the whole history instead of the last five observations 15.0.
>
> **Why it matters:** the biggest single lever in the table is a search tool that *summarises* its results (six points), and the second is a file viewer that shows a hundred lines rather than the whole file (five points). Those are tool-return-size decisions from Chapter 2, measured. And "last 5 observations" beating "full history" is Chapter 2's compaction argument with a number on it.

> [!PAPER] Yang et al. (2024), SWE-agent · Section 5 · page 8
> [![Two paragraphs from the SWE-agent analysis with highlighted phrases: 1,185 of 2,294 trajectories, 51.7 percent, have one or more failed edits; agents succeed quickly and fail slowly](https://ishwarj.com/img/agents/ch3-sweagent-edits.png-sweagent-edits.png)](/img/agents/ch3-sweagent-edits.png)
>
> **Context:** the behavioural analysis of GPT-4 Turbo trajectories.
>
> **What it says:** "out of 2,294 task instances, 1,185 (51.7%) of SWE-agent w/ GPT-4 Turbo trajectories have 1+ failed edits"; an edit eventually succeeds 90.5% of the time, but only 57.2% after one failure. And "agents succeed quickly and fail slowly": successful runs finished at a median of $1.21 and 12 steps, unsuccessful ones at a mean of $2.52 and 21 steps.
>
> **Why it matters:** two production rules. A failed action predicts further failure, so the loop should treat the second failed edit as a signal to change approach, not to try again. And long runs are much more likely to be lost than about to succeed, so a step budget set from the distribution of *successful* runs saves money without costing accuracy.

> [!PAPER] Yang et al. (2024), SWE-agent · Figures 7 and 8 · page 8
> [![Figures 7 and 8 of the SWE-agent paper: a stacked bar chart of which commands are used at each turn of solved trajectories, searching first and then alternating edit and run; and a pie chart of failure modes for unresolved instances: incorrect implementation 39.9 percent, failed to recover from edit 23.4, failed to find edit location 12.9, overly specific implementation 12.1, gave up prematurely 4.8, and smaller slices](https://ishwarj.com/img/agents/ch3-sweagent-figure8.pngweagent-figure8.png)](/img/agents/ch3-sweagent-figure8.png)
>
> **Context:** the action frequencies of solved runs and the failure taxonomy of unsolved ones, labelled by a model.
>
> **What it says:** solved runs search first, then settle into "edit, then execute" cycles. Of the failures, 39.9% were simply the wrong fix, 23.4% a failure to recover from a bad edit, 12.9% not finding where to edit, 12.1% a fix too narrow for the hidden tests.
>
> **Why it matters:** compare this with the ReAct failure table of 2022. The categories moved from "hallucination and repetition" to "wrong implementation and unrecovered edits", because the interface removed the first set. Each generation of agents fixes a failure class and reveals the next.

> [!PAPER] Xia et al. (2024), Agentless · Figure 1 · page 5
> [![Figure 1 of the Agentless paper: an issue and a project codebase enter a localisation phase that narrows to the top files, then to classes and functions, then to edit locations; a repair phase generates many candidate patches; a validation phase generates reproduction tests, filters and ranks patches, and submits one](https://ishwarj.com/img/agents/ch3-agentless-figure1.pngentless-figure1.png)](/img/agents/ch3-agentless-figure1.png)
>
> **Context:** the overview. Localisation is hierarchical (the repository structure, then a skeleton of each candidate file, then the exact lines); repair samples many patches in a diff format; validation writes a test that reproduces the issue, runs the existing tests, and ranks the surviving patches by majority.
>
> **What it says:** each phase is one or a few model calls with a fixed, purpose-built prompt; the model never chooses what to do next.
>
> **Why it matters:** look at how much of this is Section 3.6's best-of-n with a verifier (many patches, tests to filter them, a vote to rank them) and how little is a loop. The authors kept the parts of agent behaviour that measurably help (sampling and verification) and removed the part that costs the most (the model deciding each step).

> [!PAPER] Xia et al. (2024), Agentless · Table 1 (excerpt) · page 10
> [![The lower rows of Table 1 of the Agentless paper, results on SWE-bench Lite: closed systems in the thirties and forties of percent with no cost reported, open-source agents such as SWE-agent with Claude 3.5 Sonnet at 23.00 percent and 1.62 dollars, AutoCodeRover and others, retrieval baselines at 3 percent and below, and Agentless with GPT-4o at 96 issues or 32.00 percent, 0.70 dollars average cost and 78,166 tokens](https://ishwarj.com/img/agents/ch3-agentless-table1.pnggentless-table1.png)](/img/agents/ch3-agentless-table1.png)
>
> **Context:** SWE-bench Lite, 300 issues, mid-2024. Closed commercial systems appear above with higher scores and no cost figures; the comparison the paper makes is with open-source agents whose cost can be measured.
>
> **What it says:** Agentless with GPT-4o: 96 issues (32.00%) at $0.70 and 78,166 tokens per issue on average. SWE-agent with Claude 3.5 Sonnet: 23.00% at $1.62; several other agents between 18% and 30% at $2.50 to $3.50.
>
> **Why it matters:** better and cheaper at once. The paper notes that OpenAI adopted Agentless as the scaffold for reporting the coding performance of GPT-4o and o1, which is as strong an endorsement of a simple pipeline as the field has produced. The leaderboard has moved far since (agents with 2025 models resolve most of the benchmark), but the point about pattern choice has not.

> [!PAPER] Anthropic, "Raising the bar on SWE-bench Verified with Claude 3.5 Sonnet" (Engineering blog, January 2025)
> [![A paragraph from the post: the prompt outlines a suggested approach for the model, but is not overly long or too detailed for this task; the model is free to choose how it moves from step to step, rather than having strict and discrete transitions; if you are not token-sensitive, it can help to explicitly encourage the model to produce a long response](https://ishwarj.com/img/agents/ch3-anthropic-swebench.pnghropic-swebench.png)](/img/agents/ch3-anthropic-swebench.png)
>
> **Context:** the post reports 49% on SWE-bench Verified with "a prompt, a Bash Tool for executing bash commands, and an Edit Tool, for viewing and editing files and directories", and states the design philosophy: "give as much control as possible to the language model itself, and keep the scaffolding minimal".
>
> **What it says:** "the model is free to choose how it moves from step to step, rather than having strict and discrete transitions".
>
> **Why it matters:** six months after Agentless, the opposite design won with a stronger model, and the post is honest about the price: "many successful runs took hundreds of turns ... and >100k tokens", and because the model cannot see the hidden tests "it often 'thinks' that it has succeeded when the task actually is a failure". The right amount of structure is a function of the model's capability and the task's verifiability, and it moves.

</details>

## 3.8 What newer research changed (2024 to 2025)

Most papers above used 2022 and 2023 models. Three later results qualify their conclusions. None of them overturns the practical advice; each changes how strongly it should be stated.

**Self-correction can be trained.** Huang et al. showed that *prompted* models of 2023 got worse when asked to correct themselves. Aviral Kumar and colleagues at Google DeepMind (SCoRe, September 2024) trained a model with multi-turn reinforcement learning on its own correction attempts. Their table measures exactly the two quantities of our Section 3.5 experiment: wrong-to-right ($$\Delta^{i \to c}$$) and right-to-wrong ($$\Delta^{c \to i}$$).

> [!PAPER] Kumar et al. (2024), SCoRe · Table 2 · page 11
> [![Table 2 of the SCoRe paper, performance on MATH: the base model scores 52.6 percent at the first attempt and 41.4 percent at the second, a change of minus 11.2 points, with 4.6 percent of problems going from incorrect to correct and 15.8 percent from correct to incorrect; Self-Refine, STaR and Pair-SFT change the accuracy by between minus 1.0 and plus 1.8 points; SCoRe scores 60.0 and 64.4 percent, a change of plus 4.4 points, with 5.8 percent incorrect to correct and 1.4 percent correct to incorrect](https://ishwarj.com/img/agents/ch3-score-table2.pngh3-score-table2.png)](/img/agents/ch3-score-table2.png)
>
> **Context:** arXiv 2409.12917, posted 19 September 2024. Gemini 1.5 Flash on MATH; the model answers, then corrects itself with no external feedback.
>
> **What it says:** the untrained base model got worse on its second attempt (52.6% to 41.4%), because it changed 15.8% of problems from correct to incorrect and only 4.6% the other way. After SCoRe training the second attempt was better than the first (60.0% to 64.4%), with 1.4% correct-to-incorrect.
>
> **Why it matters:** the 2023 finding was about prompted models, not a law. Intrinsic self-correction can be learned. It still needs measuring both ways on your task: the trained model still broke some right answers.

**The best use of extra compute depends on difficulty.** Charlie Snell and colleagues (August 2024) compared ways of spending test-time compute: sampling many answers and searching with a verifier against letting the model revise its answer in sequence.

> [!PAPER] Snell et al. (2024), Scaling Test-Time Compute Optimally · Abstract · page 1
> [![Abstract of the paper Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters with highlighted phrases: critically varies depending on the difficulty of the prompt; more than 4 times; outperform a 14 times larger model](https://ishwarj.com/img/agents/ch3-snell-abstract.png-snell-abstract.png)](/img/agents/ch3-snell-abstract.png)
>
> **Context:** arXiv 2408.03314, posted 6 August 2024. Experiments on MATH with PaLM 2 models fine-tuned for revision and for verification.
>
> **What it says:** the effectiveness of each method "critically varies depending on the difficulty of the prompt"; allocating compute per prompt by difficulty improved efficiency "by more than 4×" over a best-of-N baseline, and on problems where a smaller model already had non-trivial success, extra test-time compute could beat a 14 times larger model.
>
> **Why it matters:** "use best-of-n" or "use revision" is not one decision for all inputs. Spend extra compute where difficulty says it pays; a router that estimates difficulty (Chapter 4) is one way.

**Reasoning models' chains are more faithful, but far from fully.** Yanda Chen and colleagues at Anthropic (2025) repeated the Turpin-style test on reasoning models: insert a hint into the prompt, find cases where the hint changed the answer, and check whether the chain of thought mentions the hint.

> [!PAPER] Chen et al. (2025), Reasoning Models Don't Always Say What They Think · Figure 1 · page 2
> [![Figure 1 of the paper Reasoning Models Don't Always Say What They Think: the fraction of examples with a faithful chain of thought for Claude 3.5 Sonnet (New), Claude 3.7 Sonnet, DeepSeek V3 and DeepSeek R1 across six kinds of hint, sycophancy, consistency, visual pattern, metadata, grader hacking and unethical information; the averages are 5, 25, 15 and 39 percent, with the reasoning models above the non-reasoning ones](https://ishwarj.com/img/agents/ch3-cotfaith-figure1.pngotfaith-figure1.png)](/img/agents/ch3-cotfaith-figure1.png)
>
> **Context:** arXiv 2505.05410 (Anthropic; research post of 3 April 2025). Questions asked with and without a hint; when the hint changed the answer, the chain counts as faithful if it acknowledges the hint.
>
> **What it says:** the reasoning models verbalised the hints more often than the non-reasoning ones, but on average only 25% (Claude 3.7 Sonnet) and 39% (DeepSeek R1) of the time. The abstract adds that outcome-based reinforcement learning improved faithfulness at first and then plateaued.
>
> **Why it matters:** the 2023 caution still holds for 2025 reasoning models in these settings. Monitoring the chain has value (it catches some behaviour), but it is not sufficient to rule behaviour out. For an agent, the practical rule is unchanged: read the chain to debug, and verify the answer with something outside it.

## 3.9 Patterns in production

The papers measured benchmarks. Engineering write-ups describe systems with users. This section keeps two things apart: **what the company reports** (its own numbers, quoted, attributed to the exact system they describe) and **our reading** of which patterns the system uses. Where a page gives no number, none is given here. The full excerpts are in the research notes at the end of the section.

<figure class="fig"><svg viewBox="0 0 760 424" role="img" aria-label="The patterns the companies of Section 3.9 describe in their own engineering write-ups, one row per system, as we read them: fixed pipelines and loops are both common, and most systems add an external check or a human gate."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="380.0" y="22.0" text-anchor="middle">Patterns the companies describe in their own write-ups (filled = stated on the page)</text><text class="t-tick" x="285.0" y="124.0" text-anchor="start" transform="rotate(-50 285.0 124)">fixed pipeline</text><text class="t-tick" x="363.0" y="124.0" text-anchor="start" transform="rotate(-50 363.0 124)">ReAct loop</text><text class="t-tick" x="441.0" y="124.0" text-anchor="start" transform="rotate(-50 441.0 124)">plan+execute</text><text class="t-tick" x="519.0" y="124.0" text-anchor="start" transform="rotate(-50 519.0 124)">reflect on tests</text><text class="t-tick" x="597.0" y="124.0" text-anchor="start" transform="rotate(-50 597.0 124)">search/best-of-n</text><text class="t-tick" x="675.0" y="124.0" text-anchor="start" transform="rotate(-50 675.0 124)">judge/human gate</text><text class="t-tick" x="240.0" y="151.0" text-anchor="end">Uber Genie, uReview, FixrLeak</text><g class="mark"><title>Uber Genie, uReview, FixrLeak: fixed pipeline</title><rect class="cell" x="278.0" y="137.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><g class="mark"><title>Uber Genie, uReview, FixrLeak: ReAct loop</title><rect class="cell-masked" x="356.0" y="137.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Uber Genie, uReview, FixrLeak: plan+execute</title><rect class="cell-masked" x="434.0" y="137.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Uber Genie, uReview, FixrLeak: reflect on tests</title><rect class="cell" x="512.0" y="137.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><g class="mark"><title>Uber Genie, uReview, FixrLeak: search/best-of-n</title><rect class="cell-masked" x="590.0" y="137.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Uber Genie, uReview, FixrLeak: judge/human gate</title><rect class="cell" x="668.0" y="137.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><text class="t-tick" x="240.0" y="177.0" text-anchor="end">Amazon Q code transformation</text><g class="mark"><title>Amazon Q code transformation: fixed pipeline</title><rect class="cell-masked" x="278.0" y="163.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Amazon Q code transformation: ReAct loop</title><rect class="cell-masked" x="356.0" y="163.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Amazon Q code transformation: plan+execute</title><rect class="cell" x="434.0" y="163.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><g class="mark"><title>Amazon Q code transformation: reflect on tests</title><rect class="cell" x="512.0" y="163.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><g class="mark"><title>Amazon Q code transformation: search/best-of-n</title><rect class="cell-masked" x="590.0" y="163.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Amazon Q code transformation: judge/human gate</title><rect class="cell" x="668.0" y="163.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><text class="t-tick" x="240.0" y="203.0" text-anchor="end">Spotify Honk</text><g class="mark"><title>Spotify Honk: fixed pipeline</title><rect class="cell-masked" x="278.0" y="189.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Spotify Honk: ReAct loop</title><rect class="cell" x="356.0" y="189.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><g class="mark"><title>Spotify Honk: plan+execute</title><rect class="cell-masked" x="434.0" y="189.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Spotify Honk: reflect on tests</title><rect class="cell" x="512.0" y="189.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><g class="mark"><title>Spotify Honk: search/best-of-n</title><rect class="cell-masked" x="590.0" y="189.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Spotify Honk: judge/human gate</title><rect class="cell" x="668.0" y="189.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><text class="t-tick" x="240.0" y="229.0" text-anchor="end">DoorDash support, Assistant</text><g class="mark"><title>DoorDash support, Assistant: fixed pipeline</title><rect class="cell" x="278.0" y="215.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><g class="mark"><title>DoorDash support, Assistant: ReAct loop</title><rect class="cell" x="356.0" y="215.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><g class="mark"><title>DoorDash support, Assistant: plan+execute</title><rect class="cell-masked" x="434.0" y="215.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>DoorDash support, Assistant: reflect on tests</title><rect class="cell-masked" x="512.0" y="215.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>DoorDash support, Assistant: search/best-of-n</title><rect class="cell-masked" x="590.0" y="215.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>DoorDash support, Assistant: judge/human gate</title><rect class="cell" x="668.0" y="215.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><text class="t-tick" x="240.0" y="255.0" text-anchor="end">LinkedIn SQL Bot, Hiring Asst.</text><g class="mark"><title>LinkedIn SQL Bot, Hiring Asst.: fixed pipeline</title><rect class="cell-masked" x="278.0" y="241.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>LinkedIn SQL Bot, Hiring Asst.: ReAct loop</title><rect class="cell" x="356.0" y="241.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><g class="mark"><title>LinkedIn SQL Bot, Hiring Asst.: plan+execute</title><rect class="cell" x="434.0" y="241.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><g class="mark"><title>LinkedIn SQL Bot, Hiring Asst.: reflect on tests</title><rect class="cell" x="512.0" y="241.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><g class="mark"><title>LinkedIn SQL Bot, Hiring Asst.: search/best-of-n</title><rect class="cell-masked" x="590.0" y="241.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>LinkedIn SQL Bot, Hiring Asst.: judge/human gate</title><rect class="cell" x="668.0" y="241.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><text class="t-tick" x="240.0" y="281.0" text-anchor="end">Stripe Minions</text><g class="mark"><title>Stripe Minions: fixed pipeline</title><rect class="cell-masked" x="278.0" y="267.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Stripe Minions: ReAct loop</title><rect class="cell" x="356.0" y="267.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><g class="mark"><title>Stripe Minions: plan+execute</title><rect class="cell-masked" x="434.0" y="267.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Stripe Minions: reflect on tests</title><rect class="cell" x="512.0" y="267.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><g class="mark"><title>Stripe Minions: search/best-of-n</title><rect class="cell-masked" x="590.0" y="267.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Stripe Minions: judge/human gate</title><rect class="cell" x="668.0" y="267.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><text class="t-tick" x="240.0" y="307.0" text-anchor="end">Airbnb test migration</text><g class="mark"><title>Airbnb test migration: fixed pipeline</title><rect class="cell" x="278.0" y="293.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><g class="mark"><title>Airbnb test migration: ReAct loop</title><rect class="cell-masked" x="356.0" y="293.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Airbnb test migration: plan+execute</title><rect class="cell-masked" x="434.0" y="293.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Airbnb test migration: reflect on tests</title><rect class="cell" x="512.0" y="293.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><g class="mark"><title>Airbnb test migration: search/best-of-n</title><rect class="cell-masked" x="590.0" y="293.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Airbnb test migration: judge/human gate</title><rect class="cell-masked" x="668.0" y="293.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><text class="t-tick" x="240.0" y="333.0" text-anchor="end">Anthropic SWE-bench scaffold</text><g class="mark"><title>Anthropic SWE-bench scaffold: fixed pipeline</title><rect class="cell-masked" x="278.0" y="319.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Anthropic SWE-bench scaffold: ReAct loop</title><rect class="cell" x="356.0" y="319.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><g class="mark"><title>Anthropic SWE-bench scaffold: plan+execute</title><rect class="cell-masked" x="434.0" y="319.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Anthropic SWE-bench scaffold: reflect on tests</title><rect class="cell-masked" x="512.0" y="319.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Anthropic SWE-bench scaffold: search/best-of-n</title><rect class="cell-masked" x="590.0" y="319.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Anthropic SWE-bench scaffold: judge/human gate</title><rect class="cell-masked" x="668.0" y="319.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><text class="t-tick" x="240.0" y="359.0" text-anchor="end">Google Deep Research</text><g class="mark"><title>Google Deep Research: fixed pipeline</title><rect class="cell-masked" x="278.0" y="345.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Google Deep Research: ReAct loop</title><rect class="cell" x="356.0" y="345.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><g class="mark"><title>Google Deep Research: plan+execute</title><rect class="cell" x="434.0" y="345.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><g class="mark"><title>Google Deep Research: reflect on tests</title><rect class="cell-masked" x="512.0" y="345.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Google Deep Research: search/best-of-n</title><rect class="cell-masked" x="590.0" y="345.0" width="22" height="20" rx="4" style="fill-opacity:1"/></g><g class="mark"><title>Google Deep Research: judge/human gate</title><rect class="cell" x="668.0" y="345.0" width="22" height="20" rx="4" style="fill-opacity:0.85"/></g><text class="t-muted" x="30.0" y="382.0" text-anchor="start">our reading of the write-ups, not a survey: most rows have a verifier (tests, CI, a judge</text><text class="t-muted" x="30.0" y="399.0" text-anchor="start">model) or a human gate. Klarna is left out: its assistant's architecture was not published.</text></svg><figcaption>The patterns companies describe in their own write-ups, as we read them. Klarna is left out because its architecture was not published.</figcaption></figure>

| Company, system | What the company reports | Our reading of the pattern |
|---|---|---|
| Uber, Genie (security and privacy channels) | an LLM judge against a golden set cut evaluation from weeks to minutes; a relative 27% more acceptable answers and 60% less incorrect advice | a fixed retrieval pipeline; progress came from cheap evaluation, not a new loop |
| Amazon, Q Developer Java upgrades | tens of thousands of applications moved to Java 17; developers review the plan before it runs | plan-and-execute with a human approving the plan |
| Spotify, Fleet Management and Honk | Fleet Management automates about half of Spotify's pull requests since mid-2024; Honk, the coding agent inside it, has produced 1,500+ merged pull requests; a judge vetoes about a quarter of sessions | a home-made loop replaced by a coding agent with few tools, a verifiable goal and a judge |
| DoorDash, Dasher support | a two-tier guardrail cut hallucinations by 90% and severe compliance issues by 99%; a stronger guardrail was too slow and costly | a fixed pipeline with an external critic and a human fallback |
| LinkedIn, SQL Bot | validators check tables exist and run `EXPLAIN`; errors go to a correction step with tools | reflection on an external signal (the database) |
| Stripe, Minions | at most two rounds of CI per run, because CI costs tokens, compute and time | a state machine of fixed steps and agent steps; tests as the critic, capped |
| Airbnb, test migration | 75% of 3,500 files migrated in four hours, 97% after four days of tuning, the rest by hand | a fixed per-file pipeline with validation errors fed back and capped retries |

### Spotify: one system inside another

Spotify's numbers are easy to misattribute. "Around half of Spotify's pull requests" describes **Fleet Management**, the system that applies automated changes across thousands of repositories and existed before any model was involved. **Honk**, the coding agent added inside it, accounts for the title's "1,500+" merged AI-generated pull requests. The team reports that its first home-made loop "tended to get lost when it filled up its context window", and that its LLM judge, which vetoes about a quarter of sessions, has not yet been evaluated. Our reading: context growth and a missing check broke the first loop, and a judge is a check of unknown precision until it is measured.

### Klarna: cost savings did not settle the quality question

In February 2024 Klarna reported that its AI assistant handled two-thirds of customer service chats in its first month, "the equivalent work of 700 full-time agents". In May 2025, as Fortune reported from a Bloomberg interview, its chief executive said that "cost unfortunately seems to have been a too predominant evaluation factor" and that the result was "lower quality", and Klarna began hiring people for customer service again. The architecture was never published, so nothing here says which pattern it used. What the story shows is about measurement: reporting cost and volume first left the quality question open, and it had to be answered later.

<details>
<summary>Research notes: the company write-ups in detail (optional reading)</summary>

**Uber.** Three systems cover most of this chapter's patterns.

> [!PAPER] Uber, "Enhanced Agentic-RAG: What If Chatbots Could Deliver Near-Human Precision?" (Engineering blog, May 2025)
> [![A paragraph from the post: overcoming these challenges was critical to ensuring Genie could reliably support security and privacy teams without risking inaccurate guidance; the team introduced automated evaluation with generative AI and the agentic RAG framework; the automated evaluation reduced experiment evaluation time from weeks to minutes](https://ishwarj.com/img/agents/ch3-uber-genie-eval.pnguber-genie-eval.png)](/img/agents/ch3-uber-genie-eval.png)
>
> **Context:** the follow-up to the Genie post, for the security and privacy channels where "inaccurate guidance" is costly. The post says that early on "many answers were either incomplete, inaccurate, or failed to retrieve relevant information", that "many experiments yielded only slight accuracy improvements before plateauing", and that "assessing improvements required significant SME bandwidth, often taking weeks".
>
> **What it says:** the fix had two parts. The pipeline gained stages (a query optimiser, a source identifier, hybrid retrieval, a post-processor, each an LLM step in a fixed graph) and the team built an automated evaluation with an LLM judge scoring answers from 0 to 5 against a golden set written by subject-matter experts, which "reduced experiment evaluation time from weeks to minutes". The post reports "increasing the percentage of acceptable answers by a relative 27% and reducing incorrect advice by a relative 60%".
>
> **Why it matters:** the pattern is still a fixed pipeline; what unblocked progress was the eval. The team could not tell which change helped until the judge made measurement cheap, and then the plateau broke. Chapter 6 returns to this story.

> [!PAPER] Uber, "uReview: Scalable, Trustworthy GenAI for Code Review at Uber" (Engineering blog, August 2025)
> [![A paragraph from the post: even with high-performing models, single-shot prompting was not enough; unfiltered outputs led to hallucinated issues, duplicate suggestions, and inconsistent quality; the team introduced multi-stage chained prompts, one step to generate comments, another to grade them, and others to filter or consolidate](https://ishwarj.com/img/agents/ch3-uber-ureview.pngh3-uber-ureview.png)](/img/agents/ch3-uber-ureview.png)
>
> **Context:** the code-review agent that, the post says, "today analyzes over 90% of the weekly ~65,000 diffs" at Uber.
>
> **What it says:** "Even with high-performing models like o4-mini-high and Claude 4 Sonnet, single-shot prompting wasn't enough. Unfiltered outputs led to hallucinated issues, duplicate suggestions, and inconsistent quality." The fix was a fixed multi-stage pipeline: several assistants generate comments in parallel, a second model grades each comment's confidence, duplicates are merged, and only high-confidence comments are posted. The post reports that developers "mark 75% of its comments as useful" and that "over 65% of its posted comments" are addressed, against "only 51% of human-written comments" addressed in the same changeset; it estimates "approximately 1,500 hours saved weekly".
>
> **Why it matters:** this is best-of-n with a model verifier (Section 3.6) in production, chosen because precision mattered more than recall: "fewer but more useful comments". The learning the post states is that the architecture and the post-processing mattered more than the prompt.

> [!PAPER] Uber, "FixrLeak: Fixing Java Resource Leaks with GenAI" (Engineering blog, May 2025)
> [![A one-line paragraph from the post: out of the 102 eligible cases, FixrLeak successfully automated fixes for 93 leaks](https://ishwarj.com/img/agents/ch3-uber-fixrleak.png3-uber-fixrleak.png)](/img/agents/ch3-uber-fixrleak.png)
>
> **Context:** a tool that fixes resource leaks flagged by static analysis. The pipeline is fixed and has no loop: the static analyser finds a leak, an abstract-syntax-tree filter drops the cases where the resource escapes the function (which a naive fix would break), one model call writes the fix, and a verification step "verifies that the target binary builds successfully, runs all existing tests to confirm nothing is broken, and can also recheck the code" with the analyser, before a human reviews the pull request.
>
> **What it says:** of 124 leaks flagged, 12 in deprecated code were excluded, the filter removed cases it could not fix safely, and "out of the 102 eligible cases, FixrLeak successfully automated fixes for 93 leaks".
>
> **Why it matters:** the post's own warning is that "blindly applying fixes can lead to new issues, like use-after-close errors", which is why the filter exists. The pattern is Agentless (Section 3.7) applied to one bug class: localise with a tool, repair with one call, validate with the build and the tests, hand the rest to a person.

**Amazon.**

> [!PAPER] AWS, "Amazon Q Developer just reached a $260 million dollar milestone" (AWS DevOps blog, August 2024)
> [![A paragraph from the post: Amazon has migrated tens of thousands of production applications from Java 8 or 11 to Java 17 with assistance from Amazon Q Developer; this represents a savings of over 4,500 years of development work for over a thousand developers, and performance improvements worth 260 million dollars in annual cost savings](https://ishwarj.com/img/agents/ch3-amazon-q-java.png3-amazon-q-java.png)](/img/agents/ch3-amazon-q-java.png)
>
> **Context:** the post describes the code-transformation agent: it analyses the application and generates "a step-by-step implementation plan", and "developers can collaborate with the agent to review and iterate on the plan before the agent implements it". The agent then applies the changes, builds and runs the tests, and the developer reviews the result.
>
> **What it says:** "Amazon has migrated tens of thousands of production applications from Java 8 or 11 to Java 17", "a savings of over 4,500 years of development work for over a thousand developers", and "performance improvements worth $260 million dollars in annual cost savings".
>
> **Why it matters:** the task shape fits the pattern exactly: a known goal, a plan that can be checked by a person who knows the codebase, and a verifier (the build and the tests) at the end. The pattern's weakness, a wrong plan found late, is removed by showing the plan to someone before anything runs. The post does not break down how many applications needed human intervention, so this section does not either.

**Spotify.**

> [!PAPER] Spotify, "1,500+ PRs Later: Spotify's Journey with Our Background Coding Agent (Honk, Part 1)" (Engineering blog, November 2025)
> [![A paragraph from the post: the impact has been significant, with a steady stream of automated pull requests being merged daily, keeping codebases consistent, up to date and secure; since mid-2024, around half of Spotify's pull requests have been automated by this system](https://ishwarj.com/img/agents/ch3-spotify-honk.pngh3-spotify-honk.png)](/img/agents/ch3-spotify-honk.png)
>
> **Context:** this paragraph is about Spotify's **Fleet Management** system (the post's following figure compares merged pull requests "coming from Fleetshift versus from human authors"), which applied scripted changes across thousands of repositories before any model was involved. Honk, the background coding agent, was added inside it for changes that could not be scripted.
>
> **What it says:** "since mid-2024, around half of Spotify's pull requests have been automated by this system", meaning Fleet Management as a whole. The agent's own number is in the title and introduction: "more than 1,500 merged AI-generated pull requests".
>
> **Why it matters:** two numbers, two systems. The half-of-all-pull-requests figure is evidence that a review-and-merge pipeline for automated changes already existed; Honk plugged into it, so review, merge and rollback did not change.

> [!PAPER] Spotify, "Background Coding Agents: Context Engineering (Honk, Part 2)" (Engineering blog, November 2025)
> [![A paragraph from the post: an alternative approach is to start with a simpler prompt but connect to Model Context Protocol tools that allow the agent to dynamically fetch more context as it works on the problem; while this does make the agent capable of tackling more complex and ambiguous tasks, this also makes it less testable and predictable; the more tools you have, the more dimensions of unpredictability you introduce](https://ishwarj.com/img/agents/ch3-spotify-honk-tools.pngtify-honk-tools.png)](/img/agents/ch3-spotify-honk-tools.png)
>
> **Context:** the part that describes the first design and its failure. The team built its own ReAct-style loop ("10 turns per session, three session retries total") and found that it "tended to get lost when it filled up its context window, forgetting the original task after a few turns", and that "agents are eager to act on your prompt, to a fault". They replaced the loop with an off-the-shelf coding agent given a minimal tool set (a verify command, restricted git, an allow-list of shell commands) and detailed prompts.
>
> **What it says:** "the more tools you have, the more dimensions of unpredictability you introduce", and "the agent needs a verifiable goal so it can iterate on a solution as it goes".
>
> **Why it matters:** two of this chapter's findings in one paragraph. Context growth undoes a ReAct loop (Section 3.3 and Chapter 2), and reflection needs a check (Section 3.5). The team's fix was not a smarter loop; it was fewer tools and a verifiable goal.

> [!PAPER] Spotify, "Background Coding Agents: Predictable Results Through Strong Feedback Loops (Honk, Part 3)" (Engineering blog, December 2025)
> [![A paragraph from the post: we have yet to invest in evals for our judge; however, we know from internal metrics that out of thousands of agent sessions, the judge vetoes about a quarter of them; when that happens, the agent is able to course correct half the time; from empirical observations, the most common trigger is the agent going outside the instructions outlined in the prompt](https://ishwarj.com/img/agents/ch3-spotify-honk-judge.pngtify-honk-judge.png)](/img/agents/ch3-spotify-honk-judge.png)
>
> **Context:** the feedback loops: verifiers (build, tests, lint) exposed as tools with summarised output, and an LLM judge that compares the diff with the prompt before a pull request is opened, wired in so the agent cannot finish without it.
>
> **What it says:** "the judge vetoes about a quarter of them. When that happens, the agent is able to course correct half the time", and "the most common trigger is the agent going outside the instructions outlined in the prompt". The team adds "we have yet to invest in evals for our judge".
>
> **Why it matters:** this is reflection with a judge as the critic. Its precision is unknown by the team's own account, so a veto rate says how often the judge objects, not how often it is right.

**DoorDash.**

> [!PAPER] DoorDash, "Path to high-quality LLM-based Dasher support automation" (Engineering blog, September 2024)
> [![A paragraph from the post: the guardrail's latency is a notable drawback caused by an end-to-end process that includes generating a response, applying the guardrail, and possibly retrying with a new guardrail check; given the relatively small number of problematic responses, strategically defaulting to human agents can be an effective way to ensure a quality user experience while maintaining a high level of automation; this guardrail system has successfully reduced overall hallucinations by 90 percent and cut down potentially severe compliance issues by 99 percent](https://ishwarj.com/img/agents/ch3-doordash-support.pngoordash-support.png)](/img/agents/ch3-doordash-support.png)
>
> **Context:** a support chatbot for delivery drivers: a fixed retrieval pipeline (summarise the conversation, retrieve similar resolved cases and knowledge-base articles, generate) followed by a two-tier guardrail (a cheap semantic-similarity check, then an LLM evaluator) that can trigger a retry or hand the conversation to a person. Offline, an LLM judge scores five quality dimensions and a regression suite runs on every prompt change.
>
> **What it says:** "This guardrail system has successfully reduced overall hallucinations by 90% and cut down potentially severe compliance issues by 99%." The costs are stated too: the guardrail's "latency is a notable drawback", and "initially, we tested a more sophisticated guardrail model but increased response times and heavy usage of model tokens made it prohibitively expensive".
>
> **Why it matters:** the critic is external to the generator and it works; and the critic has a price in latency and tokens that forced a cheaper two-tier design. Note the design choice at the end: with few problematic responses, "strategically defaulting to human agents" beat a longer retry loop.

DoorDash's later posts trace the same team moving from fixed workflows to single agents and then to several agents. A November 2025 post names the single agent's limit as "context pollution", and a June 2026 post on the consumer DoorDash Assistant reports that "the largest potential production-failure category is grounding" and that "the fix in each case has been to route the agent's claim through a tool call against the system of record".

**LinkedIn.**

> [!PAPER] LinkedIn, "Practical text-to-SQL for data analytics" (Engineering blog, December 2024)
> [![A paragraph from the post: finally, we run a set of validators on the output followed by a self-correction agent to fix errors; validators work best when they access new information not available to the query writer; we verify the existence of tables and fields, and execute the EXPLAIN statement on the query to detect syntax and other errors; these errors are fed into a self-correction agent, which is equipped with tools to retrieve additional tables or fields if needed before updating the query](https://ishwarj.com/img/agents/ch3-linkedin-sqlbot.pnglinkedin-sqlbot.png)](/img/agents/ch3-linkedin-sqlbot.png)
>
> **Context:** SQL Bot, an internal assistant that writes queries over LinkedIn's data warehouse. The pipeline retrieves candidate tables, re-ranks them, writes the query with a plan, then validates and corrects.
>
> **What it says:** "Validators work best when they access new information not available to the query writer": the system checks that tables and fields exist and runs `EXPLAIN` on the query, and "these errors are fed into a self-correction agent, which is equipped with tools to retrieve additional tables or fields". The post reports that in a survey "~95% rated SQL Bot's query accuracy 'Passes' or above, and ~40% rated the query accuracy 'Very Good' or 'Excellent'", and that a "Fix with AI" button accounted for "80% of our sessions".
>
> **Why it matters:** the sentence about validators is the whole of Section 3.5 in twelve words. The database planner, not the model, decides whether the query is valid, and the correction step has tools to fetch what it was missing. The usage number is the quieter finding: the most used feature was the retry with a real error in hand.

> [!PAPER] LinkedIn, "Building the agentic future of recruiting: how we engineered LinkedIn's Hiring Assistant" (Engineering blog, October 2025)
> [![A short paragraph from the post: in practice, however, ReAct alone is not sufficient for an enterprise-grade agent; relying directly on LLMs to solve complex recruiting problems introduces major challenges](https://ishwarj.com/img/agents/ch3-linkedin-hiring-2025.pngdin-hiring-2025.png)](/img/agents/ch3-linkedin-hiring-2025.png)
>
> **Context:** a year after the first Hiring Assistant post described an "agent orchestration layer" with "experiential memory", the engineering team described the architecture it settled on: a supervisor agent over specialised sub-agents, each running a planner that "performs high-level reasoning to produce a structured, task-specific plan" and an executor that "runs the plan step by step, using a ReAct-style loop for tool use and local reasoning".
>
> **What it says:** "In practice, however, ReAct alone is not sufficient for an enterprise-grade agent." The challenges listed are "instruction-following reliability", "hallucinations" and "intelligence (test-time compute) vs. latency tradeoffs"; a comparison table in the post says of ReAct that "task completion rate degrades quickly as the problem space grows".
>
> **Why it matters:** this is plan-and-execute layered over ReAct (Section 3.4 over Section 3.3), chosen for the reasons those sections give: the plan keeps the task well-scoped and reviewable, the loop handles the local decisions. The post gives no success metrics, so this section gives none.

**Stripe.**

> [!PAPER] Stripe, "Minions: Stripe's one-shot, end-to-end coding agents" (stripe.dev blog, February 2026)
> [![A paragraph from the post: since CI runs cost tokens, compute and time, we only have at most two rounds of CI; if tests fail after an initial push, we prompt the minion to fix failing tests and push a second time, but are then done; there is a balancing act between speed and completeness, and there are diminishing marginal returns for an LLM to run many rounds of a full CI loop](https://ishwarj.com/img/agents/ch3-stripe-minions.png-stripe-minions.png)](/img/agents/ch3-stripe-minions.png)
>
> **Context:** Stripe's background coding agents run from a task description to a reviewed pull request. "Blueprints are workflows defined in code that direct a minion run"; the blueprint "ends up looking like a state machine that intermixes deterministic code nodes and free-flowing agent nodes": git operations, linters and tests as code, the coding itself as an agent. Feedback comes first from fast local lints, then from continuous integration. The first post reports "over a thousand pull requests merged each week" that "contain no human-written code", with human review; the second post (February 2026) puts it at "over 1,300".
>
> **What it says:** "Since CI runs cost tokens, compute, and time, we only have at most two rounds of CI", because "there are diminishing marginal returns for an LLM to run many rounds of a full CI loop".
>
> **Why it matters:** a reflection loop with the strongest critic available (the real test suite) and a hard cap of two rounds, chosen from cost. The state-machine framing is the practical answer to "pipeline or agent": both, with a line drawn per step. The post also notes that writing deterministic code for "small decisions we can anticipate ... saves tokens (and CI costs) at scale".

**Airbnb.** The March 2025 post "Accelerating Large-Scale Test Migration with LLMs" (the Airbnb Tech Blog blocks automated screenshots, so it is quoted) describes migrating "nearly 3.5K React component test files" between testing frameworks, a job estimated at "1.5 years of engineering time" and finished "in just 6 weeks". Each file went through a fixed pipeline ("we modeled this flow like a state machine") that would "retry steps multiple times until they passed or we reached a limit", with "the validation errors and the most recent version of the file" in each retry prompt. The team reports "75% of our target files in just four hours", "from 75% to 97%" after four days of tuning, and the "remaining 3%" finished by hand. Our reading: a fixed pipeline, reflection on an external signal, capped retries and a human tail.

**Anthropic and Google.**

> [!PAPER] Anthropic, "Building effective agents" (Research blog, December 2024)
> [![A paragraph from the post: when building applications with LLMs, we recommend finding the simplest solution possible, and only increasing complexity when needed; this might mean not building agentic systems at all; agentic systems often trade latency and cost for better task performance, and you should consider when this tradeoff makes sense](https://ishwarj.com/img/agents/ch3-anthropic-simplest.pnghropic-simplest.png)](/img/agents/ch3-anthropic-simplest.png)
>
> **Context:** the post that Chapter 1 used to separate workflows from agents. Its "evaluator-optimizer" workflow is Section 3.5's reflection pattern, and the post states its precondition: it is effective "when we have clear evaluation criteria, and when iterative refinement provides measurable value".
>
> **What it says:** "we recommend finding the simplest solution possible, and only increasing complexity when needed", because "agentic systems often trade latency and cost for better task performance, and you should consider when this tradeoff makes sense".
>
> **Why it matters:** this is Section 3.10's one rule, stated by a company whose business is selling the models that the complexity would call. Everything in this section is a case of a team finding the level of complexity its measurements demanded, and several of them found it by starting too high.

> [!PAPER] Google, "Try Deep Research and our new experimental model in Gemini, your AI assistant" (The Keyword, December 2024)
> [![A paragraph from the post: under your supervision, Deep Research does the hard work for you; after you enter your question, it creates a multi-step research plan for you to either revise or approve; once you approve, it begins deeply analyzing relevant information from across the web on your behalf](https://ishwarj.com/img/agents/ch3-google-deepresearch.pngle-deepresearch.png)](/img/agents/ch3-google-deepresearch.png)
>
> **Context:** the launch post for Deep Research, a feature that researches a question for several minutes and writes a report. The post continues: "Gemini continuously refines its analysis, browsing the web the way you do: searching, finding interesting pieces of information and then starting a new search based on what it's learned", and "repeats this process multiple times".
>
> **What it says:** "it creates a multi-step research plan for you to either revise or approve", and only after approval does it start browsing.
>
> **Why it matters:** plan-and-execute with the plan shown to the user, then a ReAct-style browse-and-refine loop under it. The plan review step turns the pattern's weakness into a product feature, as Amazon's did for a very different task. The post gives no quality metrics, which is normal for a launch post, so none appear here.

</details>

## 3.10 Choosing a pattern

Back to Northwind. Suppose the company wants an assistant that answers staff questions from its encyclopaedia. Here is how this chapter's measurements would guide the build, one decision at a time.

1. **A labelled set and the simplest system.** Twelve typed questions, an exact grader. Direct answering: 3%, failing on missing facts.
2. **Missing facts call for tools in a loop.** Plain ReAct: 56%; the traces show repetition and tool misses.
3. **Fix the loop and the tool first.** Guard plus improved tool: 100%. A structured loop: 94%, with misses an evidence check would catch.
4. **Add the check the misses call for:** an evidence check costs a few lines and no model calls.
5. **A plan only if steps are many and independent**, priced with caching and compaction included.
6. **Best-of-n or search only if** single attempts are middling and a reliable check exists.

That sequence is a **starting heuristic**, not a law. The five dimensions of Section 3.1 are independent, so you can skip or combine steps when the task makes the need obvious: code with a test suite can start at "reflection on tests"; a puzzle with cheap state scoring can start at search; a task whose steps are fully known can start, and stay, as a fixed pipeline.

<figure class="fig"><svg viewBox="0 0 760 428" role="img" aria-label="A starting heuristic for choosing a reasoning pattern: start with one call or a fixed pipeline, and add chain of thought, a tool loop, a plan, an external check or search only when a measured failure demands it, re-measuring accuracy, cost per correct answer and latency at every step."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="380.0" y="22.0" text-anchor="middle">A starting heuristic: begin simple, add one thing per measured failure</text><rect class="box" x="30.0" y="44.0" width="300.0" height="44.0" rx="8"/><text class="t-tick" x="180.0" y="71.0" text-anchor="middle">start</text><line class="edge" x1="330.0" y1="66.0" x2="372.0" y2="66.0" marker-end="url(#ah)"/><text class="t-muted" x="351.0" y="60.0" text-anchor="middle">yes</text><text class="t-tick" x="380.0" y="63.0" text-anchor="start">one call, or a fixed pipeline if the steps are known</text><text class="t-muted" x="380.0" y="80.0" text-anchor="start">measure accuracy, cost and latency on a labelled set</text><line class="edge" x1="180.0" y1="88.0" x2="180.0" y2="102.0" marker-end="url(#ah)"/><text class="t-muted" x="190.0" y="100.0" text-anchor="start">no, or still failing</text><rect class="box-2" x="30.0" y="102.0" width="300.0" height="44.0" rx="8"/><text class="t-tick" x="180.0" y="129.0" text-anchor="middle">wrong on multi-step reasoning?</text><line class="edge" x1="330.0" y1="124.0" x2="372.0" y2="124.0" marker-end="url(#ah)"/><text class="t-muted" x="351.0" y="118.0" text-anchor="middle">yes</text><text class="t-tick" x="380.0" y="121.0" text-anchor="start">add chain of thought, or a reasoning model</text><text class="t-muted" x="380.0" y="138.0" text-anchor="start">cost: output tokens; do not assume faithfulness</text><line class="edge" x1="180.0" y1="146.0" x2="180.0" y2="160.0" marker-end="url(#ah)"/><text class="t-muted" x="190.0" y="158.0" text-anchor="start">no, or still failing</text><rect class="box-3" x="30.0" y="160.0" width="300.0" height="44.0" rx="8"/><text class="t-tick" x="180.0" y="187.0" text-anchor="middle">facts or state not in the model?</text><line class="edge" x1="330.0" y1="182.0" x2="372.0" y2="182.0" marker-end="url(#ah)"/><text class="t-muted" x="351.0" y="176.0" text-anchor="middle">yes</text><text class="t-tick" x="380.0" y="179.0" text-anchor="start">add tools in a ReAct loop: step budget, repeat guard</text><text class="t-muted" x="380.0" y="196.0" text-anchor="start">cost: n calls on a growing context</text><line class="edge" x1="180.0" y1="204.0" x2="180.0" y2="218.0" marker-end="url(#ah)"/><text class="t-muted" x="190.0" y="216.0" text-anchor="start">no, or still failing</text><rect class="box-4" x="30.0" y="218.0" width="300.0" height="44.0" rx="8"/><text class="t-tick" x="180.0" y="245.0" text-anchor="middle">many steps; latency or tokens too high?</text><line class="edge" x1="330.0" y1="240.0" x2="372.0" y2="240.0" marker-end="url(#ah)"/><text class="t-muted" x="351.0" y="234.0" text-anchor="middle">yes</text><text class="t-tick" x="380.0" y="237.0" text-anchor="start">plan first, execute with small parallel contexts</text><text class="t-muted" x="380.0" y="254.0" text-anchor="start">cost: a wrong plan is found late; replan</text><line class="edge" x1="180.0" y1="262.0" x2="180.0" y2="276.0" marker-end="url(#ah)"/><text class="t-muted" x="190.0" y="274.0" text-anchor="start">no, or still failing</text><rect class="box-1" x="30.0" y="276.0" width="300.0" height="44.0" rx="8"/><text class="t-tick" x="180.0" y="303.0" text-anchor="middle">an external check exists?</text><line class="edge" x1="330.0" y1="298.0" x2="372.0" y2="298.0" marker-end="url(#ah)"/><text class="t-muted" x="351.0" y="292.0" text-anchor="middle">yes</text><text class="t-tick" x="380.0" y="295.0" text-anchor="start">add reflection on tests, compiler, search, human</text><text class="t-muted" x="380.0" y="312.0" text-anchor="start">cost: k attempts; self-review alone is weak</text><line class="edge" x1="180.0" y1="320.0" x2="180.0" y2="334.0" marker-end="url(#ah)"/><text class="t-muted" x="190.0" y="332.0" text-anchor="start">no, or still failing</text><rect class="box-4" x="30.0" y="334.0" width="300.0" height="44.0" rx="8"/><text class="t-tick" x="180.0" y="361.0" text-anchor="middle">early choices decide; states scorable?</text><line class="edge" x1="330.0" y1="356.0" x2="372.0" y2="356.0" marker-end="url(#ah)"/><text class="t-muted" x="351.0" y="350.0" text-anchor="middle">yes</text><text class="t-tick" x="380.0" y="353.0" text-anchor="start">search: best-of-n with a verifier, or a tree</text><text class="t-muted" x="380.0" y="370.0" text-anchor="start">cost: 10x to 100x tokens; puzzles, code with tests</text><text class="t-muted" x="380.0" y="398.0" text-anchor="middle">not a fixed sequence: skip or combine rungs when the task makes the need obvious (code with tests</text><text class="t-muted" x="380.0" y="415.0" text-anchor="middle">starts at reflection); at every arrow re-measure accuracy, cost per correct answer and p95 latency</text></svg><figcaption>A starting heuristic, not a fixed sequence: add one thing per measured failure, skip or combine steps when the task makes the need obvious, and re-measure.</figcaption></figure>

| Task property | How to measure it | What it argues for |
|---|---|---|
| Verifiability: can an answer be checked without knowing it? | list the available checks and measure their false-accept rates | with a strong check: reflection on it, and best-of-n; without one: short loops and a person at the end |
| Step count | the distribution of tool calls in solved runs | one to three: a direct call or a short loop; more, with independent steps: a plan |
| Knowledge | the direct-answer baseline | high: chain of thought may be enough; low: tools, and check that tools do not hurt the easy cases |
| Latency budget | the product's p95 against the pattern's sequential depth | tight: parallel candidates or a plan with parallel steps |
| Cost of a wrong answer | what happens downstream | high: spend on verification; low: the cheapest pattern that meets the bar |
| Shape of failures | the step at which failed runs went wrong | repetition: a guard; tool misses: better observations; early fatal choices: search; late, checkable errors: reflection |

`ch3_cost.py` puts the cost per correct answer of seven patterns side by side under planning numbers:

[![Terminal output of ch3_cost.py parts one and two: a table of seven patterns with calls, input and output tokens, latency and cost at two prices; relative cost and latency against a direct answer; and the cost per correct answer, with measured accuracies for direct answer 3 percent, chain of thought 22 percent and the plain ReAct loop 56 percent, and assumed accuracies for the others](https://ishwarj.com/img/agents/ch3_cost-run.pngts/ch3_cost-run.png)](/img/agents/ch3_cost-run.png)

Read the second table with care. The three accuracies marked "mea" are measured (Section 3.3, the plain loop); the rest are assumptions to show the arithmetic. Under those numbers, at frontier-model prices, chain of thought costs about 4.4 cents per correct answer and the plain loop about 10.9 cents, because almost half its runs were wasted; a plan assumed to reach 80% would cost about 4.6 cents and a tree assumed to reach 90% about 28 cents. Put in your own accuracies and the ranking can change; the formula will not.

The one rule of the chapter, stated as a condition: **add a pattern when a measured failure calls for it, and keep it only if accuracy, cost per correct answer and latency still justify it after you re-measure.**

## Exercises

1. Your wiki agent uses a ReAct loop with a ten-step budget, and 30% of failed runs hit the budget. Design three measurements that separate repetition, unhelpful tool misses and genuinely long tasks, and name the fix each one points to.
2. A team wants to add "the model reviews its answer before replying" to a customer-facing agent. Using Huang et al., SCoRe's Table 2 and our repair experiment, write the one-paragraph case for measuring wrong-to-right and right-to-wrong changes first, and list three external checks available in a typical support system.
3. Re-run `ch3_cost.py` with your own prompt size, observation size, prices and cache discount. At what step count does plan-and-execute become cheaper than the loop with full replay, and does it still win with caching?
4. Your verifier accepts 5% of wrong answers and all right ones. Compute $$P(\text{correct} \mid \text{accepted})$$ for $$p = 0.8$$ and $$p = 0.2$$. Which of your task types sits nearer each value, and what does that imply about where to spend on a better verifier?
5. Take the Spotify and Airbnb cases. For each, write one sentence on what the company reports and one on your reading of the pattern, and name one number you would want that neither post gives.

## Key takeaways

- A **reasoning pattern** is a choice on one or more of five independent dimensions: reasoning inside one call, control across calls, action format, verification and selection, and context management. They combine; they are not a ladder.
- **Chain of thought** helps a model use what it knows (public questions: 6% to 44% in our run) and cannot add facts it lacks (private: 0% to 0%). **Self-consistency** returns the plurality answer; it fails when errors are correlated.
- **ReAct** grounds the model in tool results but fails by repetition and unhelpful tool misses. In our ablation the plain loop scored 56%, the guard alone 75%, the improved tool alone 75%, and both 100%. Its "0% hallucination" is a share of sampled *failed* runs in one study, not an overall rate.
- A **modern structured loop** (schema-checked tool calls, explicit state, budgets, typed final answer, trace) scored 34 of 36 on the same questions; its misses were a prior overriding an observation and a strict grader.
- **Plan-and-execute** cuts sequential calls and, under full replay, tokens; caching and compaction change the token comparison, so state the assumptions.
- **Reflection** is worth what its check is worth. In our held-out experiment, self-review left 43 of 50 unchanged in total (one fix, one breakage); the feedback-test loop reached 48 of 50 (five fixes, none broken).
- A verifier's false-accept rate is not its precision: $$p = 0.9$$, $$r = 1$$, $$f = 0.2$$ gives about 97.8% precision, but $$p = 0.3$$ gives 68.2%. More candidates raise the chance of returning something, not its precision.
- Newer work qualifies the 2023 results: self-correction can be trained (SCoRe), the best use of test-time compute depends on difficulty (Snell et al.), and reasoning models' chains are more faithful but often still omit what drove the answer (Chen et al.).
- In production write-ups, keep a company's reported numbers apart from your reading of its pattern.

## References

**Papers**

1. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, Denny Zhou. [*Chain-of-Thought Prompting Elicits Reasoning in Large Language Models*](https://arxiv.org/abs/2201.11903). NeurIPS 2022 (arXiv January 2022).
2. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou. [*Self-Consistency Improves Chain of Thought Reasoning in Language Models*](https://arxiv.org/abs/2203.11171). ICLR 2023 (arXiv March 2022).
3. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, Yusuke Iwasawa. [*Large Language Models are Zero-Shot Reasoners*](https://arxiv.org/abs/2205.11916). NeurIPS 2022 (arXiv May 2022).
4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, John Schulman. [*Training Verifiers to Solve Math Word Problems*](https://arxiv.org/abs/2110.14168). arXiv October 2021.
5. OpenAI. [*OpenAI o1 System Card*](https://arxiv.org/abs/2412.16720). arXiv December 2024.
6. DeepSeek-AI. [*DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning*](https://arxiv.org/abs/2501.12948). arXiv January 2025 (version 1; a revised version appeared in Nature, 2025).
7. Miles Turpin, Julian Michael, Ethan Perez, Samuel R. Bowman. [*Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting*](https://arxiv.org/abs/2305.04388). NeurIPS 2023 (arXiv May 2023).
8. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, Denny Zhou. [*Large Language Models Cannot Self-Correct Reasoning Yet*](https://arxiv.org/abs/2310.01798). ICLR 2024 (arXiv October 2023).
9. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao. [*ReAct: Synergizing Reasoning and Acting in Language Models*](https://arxiv.org/abs/2210.03629). ICLR 2023 (arXiv October 2022).
10. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, Ee-Peng Lim. [*Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models*](https://arxiv.org/abs/2305.04091). ACL 2023 (arXiv May 2023).
11. Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, Dongkuan Xu. [*ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models*](https://arxiv.org/abs/2305.18323). arXiv May 2023.
12. Sehoon Kim, Suhong Moon, Ryan Tabrizi, Nicholas Lee, Michael W. Mahoney, Kurt Keutzer, Amir Gholami. [*An LLM Compiler for Parallel Function Calling*](https://arxiv.org/abs/2312.04511). ICML 2024 (arXiv December 2023).
13. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao. [*Reflexion: Language Agents with Verbal Reinforcement Learning*](https://arxiv.org/abs/2303.11366). NeurIPS 2023 (arXiv March 2023).
14. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, Peter Clark. [*Self-Refine: Iterative Refinement with Self-Feedback*](https://arxiv.org/abs/2303.17651). NeurIPS 2023 (arXiv March 2023).
15. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, Weizhu Chen. [*CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing*](https://arxiv.org/abs/2305.11738). ICLR 2024 (arXiv May 2023).
16. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, Karthik Narasimhan. [*Tree of Thoughts: Deliberate Problem Solving with Large Language Models*](https://arxiv.org/abs/2305.10601). NeurIPS 2023 (arXiv May 2023).
17. Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, Yu-Xiong Wang. [*Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models*](https://arxiv.org/abs/2310.04406). ICML 2024 (arXiv October 2023).
18. Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, Heng Ji. [*Executable Code Actions Elicit Better LLM Agents*](https://arxiv.org/abs/2402.01030). ICML 2024 (arXiv February 2024).
19. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, Ofir Press. [*SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering*](https://arxiv.org/abs/2405.15793). NeurIPS 2024 (arXiv May 2024).
20. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, Lingming Zhang. [*Agentless: Demystifying LLM-based Software Engineering Agents*](https://arxiv.org/abs/2407.01489). arXiv July 2024.
21. Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D. Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M. Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, Aleksandra Faust. [*Training Language Models to Self-Correct via Reinforcement Learning*](https://arxiv.org/abs/2409.12917). arXiv September 2024.
22. Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar. [*Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters*](https://arxiv.org/abs/2408.03314). arXiv August 2024.
23. Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, Ethan Perez. [*Reasoning Models Don't Always Say What They Think*](https://arxiv.org/abs/2505.05410). arXiv May 2025; research post [*Reasoning models don't always say what they think*](https://www.anthropic.com/research/reasoning-models-dont-say-think), Anthropic, 3 April 2025.

**Engineering blogs and docs**

24. Anthropic. [*Building effective agents*](https://www.anthropic.com/research/building-effective-agents). Research blog, 19 December 2024.
25. Anthropic. [*Raising the bar on SWE-bench Verified with Claude 3.5 Sonnet*](https://www.anthropic.com/engineering/swe-bench-sonnet). Engineering blog, 6 January 2025.
26. Anthropic. [*Code execution with MCP: Building more efficient agents*](https://www.anthropic.com/engineering/code-execution-with-mcp). Engineering blog, 4 November 2025.
27. Google (Dave Citron). [*Try Deep Research and our new experimental model in Gemini, your AI assistant*](https://blog.google/products/gemini/google-gemini-deep-research/). The Keyword, 11 December 2024.
28. Uber. [*Enhanced Agentic-RAG: What If Chatbots Could Deliver Near-Human Precision?*](https://www.uber.com/us/en/blog/enhanced-agentic-rag/). Engineering blog, May 2025.
29. Uber. [*uReview: Scalable, Trustworthy GenAI for Code Review at Uber*](https://www.uber.com/us/en/blog/ureview/). Engineering blog, 12 August 2025.
30. Uber. [*FixrLeak: Fixing Java Resource Leaks with GenAI*](https://www.uber.com/us/en/blog/fixrleak-fixing-java-resource-leaks-with-genai/). Engineering blog, May 2025.
31. AWS (Aytul Arisoy Cholkar). [*Amazon Q Developer just reached a $260 million dollar milestone*](https://aws.amazon.com/blogs/devops/amazon-q-developer-just-reached-a-260-million-dollar-milestone). AWS DevOps blog, 1 August 2024.
32. Spotify (Max Charas, Marc Bruggmann). [*1,500+ PRs Later: Spotify's Journey with Our Background Coding Agent (Honk, Part 1)*](https://engineering.atspotify.com/2025/11/spotifys-background-coding-agent-part-1). Engineering blog, 6 November 2025.
33. Spotify. [*Background Coding Agents: Context Engineering (Honk, Part 2)*](https://engineering.atspotify.com/2025/11/context-engineering-background-coding-agents-part-2). Engineering blog, 24 November 2025.
34. Spotify. [*Background Coding Agents: Predictable Results Through Strong Feedback Loops (Honk, Part 3)*](https://engineering.atspotify.com/2025/12/feedback-loops-background-coding-agents-part-3). Engineering blog, 9 December 2025.
35. DoorDash. [*Path to high-quality LLM-based Dasher support automation*](https://careersatdoordash.com/blog/large-language-modules-based-dasher-support-automation/). Engineering blog, 17 September 2024.
36. DoorDash. [*Beyond Single Agents: How DoorDash is building a collaborative AI ecosystem*](https://careersatdoordash.com/blog/beyond-single-agents-doordash-building-collaborative-ai-ecosystem/). Engineering blog, 11 November 2025.
37. DoorDash. [*Building DoorDash Assistant: An engineering overview*](https://careersatdoordash.com/blog/building-doordash-assistant-an-engineering-overview/). Engineering blog, 11 June 2026.
38. LinkedIn (Albert Chen and colleagues). [*Practical text-to-SQL for data analytics*](https://www.linkedin.com/blog/engineering/ai/practical-text-to-sql-for-data-analytics). Engineering blog, 9 December 2024.
39. LinkedIn (Xiaoyang Gu, Xie Lu, Daniel Hewlett). [*Building the agentic future of recruiting: how we engineered LinkedIn's Hiring Assistant*](https://www.linkedin.com/blog/engineering/ai/how-we-engineered-linkedins-hiring-assistant). Engineering blog, 21 October 2025.
40. Stripe (Alistair Gray). [*Minions: Stripe's one-shot, end-to-end coding agents*](https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents) and [*Part 2*](https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents-part-2). stripe.dev blog, 9 and 19 February 2026.
41. Airbnb (Charles Covey-Brandt). [*Accelerating Large-Scale Test Migration with LLMs*](https://medium.com/airbnb-engineering/accelerating-large-scale-test-migration-with-llms-9565c208023b). Airbnb Tech Blog, March 2025.
42. Klarna. [*Klarna AI assistant handles two-thirds of customer service chats in its first month*](https://www.klarna.com/international/press/klarna-ai-assistant-handles-two-thirds-of-customer-service-chats-in-its-first-month/). Press release, 27 February 2024; and Irina Ivanova, [*Klarna plans to hire humans again, as new landmark survey reveals most AI projects fail to deliver*](https://fortune.com/2025/05/09/klarna-ai-humans-return-on-investment/), Fortune, 9 May 2025, reporting the Bloomberg interview of 8 May 2025.

**Code for this chapter**

43. [`code/agents/ch3_react.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/agents/ch3_react.py): twelve two-hop questions, six system configurations (including the four-way ablation of the guard and the improved tool), three repeats, a typed exact grader with a self-test; results in `results/ch3_react.json` and the two `ch3_react*_stdout.txt` logs.
44. [`code/agents/ch3_structured_loop.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/agents/ch3_structured_loop.py): a complete structured-tool loop (native tool calls, schema validation, explicit state, step and token budgets, error handling, a JSON-lines trace); results in `results/ch3_structured_loop.json` and `results/ch3_structured_trace.jsonl`.
45. [`code/agents/ch3_reflexion.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/agents/ch3_reflexion.py): ten coding tasks with feedback and held-out test suites (both checked against reference solutions), self-review against a feedback-test loop on every first attempt; results in `results/ch3_reflexion.json`.
46. [`code/agents/ch3_cost.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/agents/ch3_cost.py): the cost and latency arithmetic, cost per correct answer, the retry formula and its assumptions, token growth with caching and compaction, and the verifier precision and best-of-n tables; results in `results/ch3_cost.json`.
47. [`code/agents/figs_ch3.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/agents/figs_ch3.py) and [`code/agents/shots_ch3.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/agents/shots_ch3.py): the figures (with an automatic overlap and clipping check) and the paper excerpts.

## Next

→ [Chapter 4: Workflow patterns](https://ishwarj.com/books/agentic-systems/04-workflow-patterns/)

This chapter kept one agent and changed how it thinks, acts and checks. Chapter 4 keeps the thinking and changes the plumbing around it: workflow patterns in which code, not the model, decides the path. Prompt chaining, routing, parallel fan-out, orchestrator and workers, and evaluator and optimiser are ways of arranging several model calls so that each is small, checkable and cheap. Several systems in this chapter (Uber's review pipeline, Stripe's state machine, Airbnb's migration) are workflows with an agent inside one step, and Snell et al.'s finding that compute should follow difficulty is, in practice, a routing decision.

