# Chapter 6 · Reinforcement learning, from scratch

Author: Ishwar Jangid
Canonical URL: https://ishwarj.com/books/how-models-are-trained/06-reinforcement-learning-basics/
Updated: 2026-10-11
Source: https://github.com/ishwar6/ishwar-books/blob/main/content/books/how-models-are-trained/part3-learning-from-feedback/06-reinforcement-learning-basics.md


> **Goal:** by the end of this chapter you can describe text generation in the language of reinforcement learning (states, actions, policy, episode, reward, return), derive the policy gradient yourself with the log-derivative trick, write REINFORCE as a loss in a few lines of PyTorch, explain why its gradient is so noisy and prove that subtracting a baseline removes noise without adding bias. You will know what a value function, an advantage and a TD error are, compute GAE by hand and say what its $$\lambda$$ trades off, and explain the per-token KL penalty as a piece of reward shaping. You will also run three experiments: a four-armed bandit over 200 seeds, a tiny "token world" where every value can be computed exactly, and REINFORCE on GPT-2 with a sentiment reward, which, without a KL penalty, learns to repeat "beautifully captures this superb masterpiece" forever.

---

## 6.1 Why fine-tuning on examples is not enough

[Chapter 5](https://ishwarj.com/books/how-models-are-trained/05-supervised-fine-tuning) taught a model to follow instructions by copying good answers. Supervised fine-tuning (SFT) needs, for every prompt, the exact tokens of a good reply. It then raises the probability of those tokens, one position at a time.

That works, but it has three limits, and Section 5.12 named them.

1. **Someone has to write the answer.** For many tasks it is far easier to *judge* a reply than to *write* one. Most people cannot write a perfect sonnet, but they can say which of two sonnets they like better. A unit test cannot write code, but it can say whether code works.
2. **A copy is capped by its source.** If the model only imitates, it cannot become better than the examples it imitates.
3. **SFT never sees its own mistakes.** During training the model is always fed the *correct* previous tokens. At use time it must continue from its *own* tokens, including bad ones, and it was never taught what to do after a bad start.

Reinforcement learning (RL) removes all three limits with a different kind of signal: not "here is the right answer" but "here is a score for the answer you just gave".

> [!DEFINITION] Reinforcement learning (RL)
> Learning by trial and error from a numerical score called the **reward**. The learner tries something, receives a reward, and changes its behaviour so that high-reward behaviour becomes more likely. Nobody shows it the correct action.

For a language model the loop is: give it a prompt, let it **sample** a whole reply, score the reply, and then nudge the model's weights so that replies like the high-scoring ones become more probable. The score can come from a reward model trained on human preferences (that is RLHF, the subject of [Chapter 7](https://ishwarj.com/books/how-models-are-trained/07-reward-models) and [Chapter 8](https://ishwarj.com/books/how-models-are-trained/08-rlhf-with-ppo)), from a classifier, or from a checker such as a unit test or an exact-match test against a known answer (RLVR, Chapter 1, Section 1.6).

This chapter builds the machinery underneath all of those methods, from nothing. We start with vocabulary, derive the one equation that everything rests on (the policy gradient), and then spend most of the chapter on its central problem: the gradient it gives is correct on average but extremely noisy. Baselines, value functions, advantages, TD errors and GAE are all answers to that one problem. The chapter ends with the KL penalty, the safety rope that keeps the model from drifting into nonsense while it chases reward.

<figure class="fig"><svg viewBox="0 0 780 398" role="img" aria-label="A timeline of policy-gradient methods from actor-critic in 1983 to RLOO and GRPO in 2024."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Policy gradients from 1983 to 2024</text><text class="t-tick" x="56.0" y="57.0" text-anchor="end">1983</text><circle class="node on" cx="76" cy="52" r="5"/><rect class="box-1" x="92.0" y="40.0" width="250.0" height="24.0" rx="6"/><text class="t-tick t-strong" x="102.0" y="56.5" text-anchor="start">Barto, Sutton, Anderson</text><text class="t-tick" x="354.0" y="56.5" text-anchor="start">actor-critic: a learned critic judges the actor</text><text class="t-tick" x="56.0" y="87.0" text-anchor="end">1988</text><circle class="node on" cx="76" cy="82" r="5"/><rect class="box-1" x="92.0" y="70.0" width="250.0" height="24.0" rx="6"/><text class="t-tick t-strong" x="102.0" y="86.5" text-anchor="start">Sutton</text><text class="t-tick" x="354.0" y="86.5" text-anchor="start">temporal-difference learning, TD(lambda)</text><text class="t-tick" x="56.0" y="117.0" text-anchor="end">1992</text><circle class="node on" cx="76" cy="112" r="5"/><rect class="box-2" x="92.0" y="100.0" width="250.0" height="24.0" rx="6"/><text class="t-tick t-strong" x="102.0" y="116.5" text-anchor="start">Williams</text><text class="t-tick" x="354.0" y="116.5" text-anchor="start">REINFORCE: unbiased gradient from samples, with a baseline</text><text class="t-tick" x="56.0" y="147.0" text-anchor="end">2000</text><circle class="node on" cx="76" cy="142" r="5"/><rect class="box-2" x="92.0" y="130.0" width="250.0" height="24.0" rx="6"/><text class="t-tick t-strong" x="102.0" y="146.5" text-anchor="start">Sutton, McAllester, Singh, Mansour</text><text class="t-tick" x="354.0" y="146.5" text-anchor="start">the policy gradient theorem with function approximation</text><text class="t-tick" x="56.0" y="177.0" text-anchor="end">2015</text><circle class="node on" cx="76" cy="172" r="5"/><rect class="box-1" x="92.0" y="160.0" width="250.0" height="24.0" rx="6"/><text class="t-tick t-strong" x="102.0" y="176.5" text-anchor="start">Schulman et al.</text><text class="t-tick" x="354.0" y="176.5" text-anchor="start">TRPO, and GAE: the lambda trade-off for advantages</text><text class="t-tick" x="56.0" y="207.0" text-anchor="end">2015</text><circle class="node on" cx="76" cy="202" r="5"/><rect class="box-3" x="92.0" y="190.0" width="250.0" height="24.0" rx="6"/><text class="t-tick t-strong" x="102.0" y="206.5" text-anchor="start">Ranzato et al. (MIXER)</text><text class="t-tick" x="354.0" y="206.5" text-anchor="start">REINFORCE to train text generators on BLEU</text><text class="t-tick" x="56.0" y="237.0" text-anchor="end">2016</text><circle class="node on" cx="76" cy="232" r="5"/><rect class="box-1" x="92.0" y="220.0" width="250.0" height="24.0" rx="6"/><text class="t-tick t-strong" x="102.0" y="236.5" text-anchor="start">Mnih et al. (A3C)</text><text class="t-tick" x="354.0" y="236.5" text-anchor="start">advantage actor-critic with deep networks</text><text class="t-tick" x="56.0" y="267.0" text-anchor="end">2017</text><circle class="node on" cx="76" cy="262" r="5"/><rect class="box-1" x="92.0" y="250.0" width="250.0" height="24.0" rx="6"/><text class="t-tick t-strong" x="102.0" y="266.5" text-anchor="start">Schulman et al.</text><text class="t-tick" x="354.0" y="266.5" text-anchor="start">PPO (Chapter 8)</text><text class="t-tick" x="56.0" y="297.0" text-anchor="end">2017</text><circle class="node on" cx="76" cy="292" r="5"/><rect class="box-3" x="92.0" y="280.0" width="250.0" height="24.0" rx="6"/><text class="t-tick t-strong" x="102.0" y="296.5" text-anchor="start">Jaques et al.</text><text class="t-tick" x="354.0" y="296.5" text-anchor="start">KL control: stay close to a pretrained sequence model</text><text class="t-tick" x="56.0" y="327.0" text-anchor="end">2019</text><circle class="node on" cx="76" cy="322" r="5"/><rect class="box-3" x="92.0" y="310.0" width="250.0" height="24.0" rx="6"/><text class="t-tick t-strong" x="102.0" y="326.5" text-anchor="start">Ziegler et al.</text><text class="t-tick" x="354.0" y="326.5" text-anchor="start">RL from human preferences on GPT-2 with a KL penalty</text><text class="t-tick" x="56.0" y="357.0" text-anchor="end">2024</text><circle class="node on" cx="76" cy="352" r="5"/><rect class="box-2" x="92.0" y="340.0" width="250.0" height="24.0" rx="6"/><text class="t-tick t-strong" x="102.0" y="356.5" text-anchor="start">Ahmadian et al.; Shao et al.</text><text class="t-tick" x="354.0" y="356.5" text-anchor="start">back to REINFORCE: RLOO and GRPO drop the critic</text><line class="edge-dim" x1="76.0" y1="46.0" x2="76.0" y2="352.0"/><rect class="box-1" x="92" y="374" width="12" height="12" rx="2"/><text class="t-tick" x="110.0" y="384.0" text-anchor="start">critics and advantages</text><rect class="box-2" x="290" y="374" width="12" height="12" rx="2"/><text class="t-tick" x="308.0" y="384.0" text-anchor="start">the policy gradient itself</text><rect class="box-3" x="480" y="374" width="12" height="12" rx="2"/><text class="t-tick" x="498.0" y="384.0" text-anchor="start">applied to text</text></svg><figcaption>Policy-gradient methods from 1983 to 2024. The ideas in this chapter are old: actor-critic (1983), REINFORCE (1992), the policy gradient theorem (2000) and GAE (2015). Language models started using them in 2015, and the KL penalty to a pretrained model arrived in 2017 to 2019. Recent methods for language models (RLOO, GRPO) have gone back to REINFORCE with a better baseline.</figcaption></figure>

## 6.2 The vocabulary, mapped onto text

RL was developed for robots, games and control problems, so its words sound strange for text. Each one has a precise meaning for a language model, and it is worth getting them exactly right, because the later equations use them.

<figure class="fig"><svg viewBox="0 0 780 340" role="img" aria-label="The agent-environment loop of reinforcement learning, and its language-model version: the model is the agent, the next token is the action, the text so far is the state, and a reward comes at the end."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="390.0" y="24.0" text-anchor="middle">The reinforcement learning loop, and the same loop for a language model</text><rect class="box-1" x="40.0" y="60.0" width="200.0" height="92.0" rx="10"/><text class="t-note" x="140.0" y="82.0" text-anchor="middle">agent (policy)</text><text class="t-tick" x="140.0" y="101.0" text-anchor="middle">looks at the state,</text><text class="t-tick" x="140.0" y="118.0" text-anchor="middle">picks an action</text><rect class="box-2" x="540.0" y="60.0" width="200.0" height="92.0" rx="10"/><text class="t-note" x="640.0" y="82.0" text-anchor="middle">environment</text><text class="t-tick" x="640.0" y="101.0" text-anchor="middle">moves to a new state,</text><text class="t-tick" x="640.0" y="118.0" text-anchor="middle">sometimes gives a reward</text><line class="path" x1="240.0" y1="88.0" x2="538.0" y2="88.0" marker-end="url(#ah-on)"/><text class="t-note" x="390.0" y="80.0" text-anchor="middle">action</text><line class="edge" x1="540.0" y1="128.0" x2="242.0" y2="128.0" marker-end="url(#ah)"/><text class="t-note" x="390.0" y="146.0" text-anchor="middle">new state, reward</text><line class="edge-dim" x1="20.0" y1="176.0" x2="760.0" y2="176.0"/><rect class="box-1" x="40.0" y="190.0" width="200.0" height="92.0" rx="10"/><text class="t-note" x="140.0" y="212.0" text-anchor="middle">the language model</text><text class="t-tick" x="140.0" y="231.0" text-anchor="middle">reads prompt + tokens so far,</text><text class="t-tick" x="140.0" y="248.0" text-anchor="middle">samples the next token</text><rect class="box-2" x="540.0" y="190.0" width="200.0" height="92.0" rx="10"/><text class="t-note" x="640.0" y="212.0" text-anchor="middle">the "world"</text><text class="t-tick" x="640.0" y="231.0" text-anchor="middle">appends the token (no surprise);</text><text class="t-tick" x="640.0" y="248.0" text-anchor="middle">a judge scores the full reply</text><line class="path" x1="240.0" y1="218.0" x2="538.0" y2="218.0" marker-end="url(#ah-on)"/><text class="t-note" x="390.0" y="210.0" text-anchor="middle">next token, e.g. " wonderful"</text><line class="edge" x1="540.0" y1="258.0" x2="242.0" y2="258.0" marker-end="url(#ah)"/><text class="t-note" x="390.0" y="276.0" text-anchor="middle">longer text; reward 0 until the end</text><text class="t-muted" x="390.0" y="314.0" text-anchor="middle">The state transition is deterministic: the new state is the old text plus the chosen token.</text><text class="t-muted" x="390.0" y="331.0" text-anchor="middle">All the randomness is in the policy, and the reward usually arrives only once, after the last token.</text></svg><figcaption>The reinforcement learning loop. Top: the general version, where an agent acts in an environment and receives new states and rewards. Bottom: the same loop for a language model. The "environment" is almost trivial: it appends the chosen token to the text. The interesting part is the reward, which a judge gives only to the finished reply.</figcaption></figure>

> [!DEFINITION] Agent and environment
> The **agent** is the learner that makes decisions. The **environment** is everything else: it receives the agent's decisions and responds with a new situation and, sometimes, a reward. For us the agent is the language model and the environment is "the text so far plus a judge".

> [!DEFINITION] State, action and policy
> The **state** $$s_t$$ is what the agent knows at step $$t$$. The **action** $$a_t$$ is what it chooses. The **policy** $$\pi_\theta(a \mid s)$$ is the agent's rule for choosing, as a probability for every action, with weights $$\theta$$. For a language model the state is the prompt plus the reply tokens written so far, the action is the next token, and the policy is the model itself: the softmax over the vocabulary that Chapter 1 described.

> [!DEFINITION] Episode, trajectory and reward
> An **episode** is one complete run from start to finish; for us, one reply. The **trajectory** $$\tau = (s_0, a_0, s_1, a_1, \ldots, s_{T-1}, a_{T-1})$$ is the list of states and actions in it. The **reward** $$r_t$$ is the number the environment returns after action $$a_t$$. For text, almost every $$r_t$$ is 0 and only the last one carries the judge's score.

<figure class="fig"><svg viewBox="0 0 780 270" role="img" aria-label="A reply drawn as an episode: states are the prompt plus the tokens so far, actions are tokens, and the only reward comes at the end."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">One episode = one reply. Each token is one action; the reward arrives after the last one</text><text class="t-muted" x="20.0" y="74.0" text-anchor="start">prompt</text><rect class="box-ghost" x="20.0" y="82.0" width="96.0" height="30.0" rx="6"/><text class="t-tick" x="68.0" y="101.5" text-anchor="middle">Review:</text><text class="t-tick" x="178.0" y="58.0" text-anchor="middle">state s0</text><text class="t-muted" x="178.0" y="74.0" text-anchor="middle">prompt</text><rect class="box-1" x="130.0" y="82.0" width="96.0" height="30.0" rx="6"/><text class="t-tick" x="178.0" y="101.5" text-anchor="middle">The</text><text class="t-tick" x="178.0" y="132.0" text-anchor="middle">action a0</text><text class="t-val" x="178.0" y="152.0" text-anchor="middle">r = 0</text><text class="t-muted" x="178.0" y="172.0" text-anchor="middle">p = pi(a0 | s0)</text><text class="t-tick" x="284.0" y="58.0" text-anchor="middle">state s1</text><text class="t-muted" x="284.0" y="74.0" text-anchor="middle">prompt + 1</text><rect class="box-1" x="236.0" y="82.0" width="96.0" height="30.0" rx="6"/><text class="t-tick" x="284.0" y="101.5" text-anchor="middle"> film</text><text class="t-tick" x="284.0" y="132.0" text-anchor="middle">action a1</text><text class="t-val" x="284.0" y="152.0" text-anchor="middle">r = 0</text><text class="t-muted" x="284.0" y="172.0" text-anchor="middle">p = pi(a1 | s1)</text><text class="t-tick" x="390.0" y="58.0" text-anchor="middle">state s2</text><text class="t-muted" x="390.0" y="74.0" text-anchor="middle">prompt + 2</text><rect class="box-1" x="342.0" y="82.0" width="96.0" height="30.0" rx="6"/><text class="t-tick" x="390.0" y="101.5" text-anchor="middle"> was</text><text class="t-tick" x="390.0" y="132.0" text-anchor="middle">action a2</text><text class="t-val" x="390.0" y="152.0" text-anchor="middle">r = 0</text><text class="t-muted" x="390.0" y="172.0" text-anchor="middle">p = pi(a2 | s2)</text><text class="t-tick" x="496.0" y="58.0" text-anchor="middle">state s3</text><text class="t-muted" x="496.0" y="74.0" text-anchor="middle">prompt + 3</text><rect class="box-1" x="448.0" y="82.0" width="96.0" height="30.0" rx="6"/><text class="t-tick" x="496.0" y="101.5" text-anchor="middle"> a</text><text class="t-tick" x="496.0" y="132.0" text-anchor="middle">action a3</text><text class="t-val" x="496.0" y="152.0" text-anchor="middle">r = 0</text><text class="t-muted" x="496.0" y="172.0" text-anchor="middle">p = pi(a3 | s3)</text><text class="t-tick" x="602.0" y="58.0" text-anchor="middle">state s4</text><text class="t-muted" x="602.0" y="74.0" text-anchor="middle">prompt + 4</text><rect class="box-1" x="554.0" y="82.0" width="96.0" height="30.0" rx="6"/><text class="t-tick" x="602.0" y="101.5" text-anchor="middle"> joy</text><text class="t-tick" x="602.0" y="132.0" text-anchor="middle">action a4</text><text class="t-val" x="602.0" y="152.0" text-anchor="middle">r = 0</text><text class="t-muted" x="602.0" y="172.0" text-anchor="middle">p = pi(a4 | s4)</text><text class="t-tick" x="708.0" y="58.0" text-anchor="middle">state s5</text><text class="t-muted" x="708.0" y="74.0" text-anchor="middle">prompt + 5</text><rect class="box-3" x="660.0" y="82.0" width="96.0" height="30.0" rx="6"/><text class="t-tick" x="708.0" y="101.5" text-anchor="middle">&lt;end&gt;</text><text class="t-tick" x="708.0" y="132.0" text-anchor="middle">action a5</text><text class="t-val" x="708.0" y="152.0" text-anchor="middle">r = score</text><text class="t-muted" x="708.0" y="172.0" text-anchor="middle">p = pi(a5 | s5)</text><path class="edge" d="M130.0,182.0 L130.0,190.0 L756.0,190.0 L756.0,182.0"/><text class="t-tick" x="443.0" y="208.0" text-anchor="middle">trajectory tau = (s0, a0, s1, a1, ..., s5, a5); its probability is the product of the six p values</text><text class="t-muted" x="20.0" y="236.0" text-anchor="start">The judge (a reward model, a classifier, a unit test) sees only the finished reply. Every token before the last gets</text><text class="t-muted" x="20.0" y="253.0" text-anchor="start">reward 0, so the learner must work out which of the earlier choices deserve the credit: the credit assignment problem.</text></svg><figcaption>One episode drawn token by token. The state grows by one token per step. The probability of the whole reply is the product of the six probabilities the policy gave to the six chosen tokens, which is the chain rule of Chapter 1, Section 1.4.</figcaption></figure>

Researchers who applied RL to text generators in 2015 described exactly this mapping. Ranzato and colleagues at Facebook AI Research trained recurrent networks for translation and summarisation directly on the BLEU and ROUGE scores used to evaluate them:

> [!PAPER] Ranzato et al. (2015), Sequence Level Training with Recurrent Neural Networks (MIXER) · Section 3.2 · page 6
> [![A paragraph from the MIXER paper casting text generation as reinforcement learning: the RNN is the agent, the parameters define a policy; highlighted: an action refers to predicting the next word in the sequence at each time step, and once the agent has reached the end of a sequence, it observes a reward](https://ishwarj.com/img/training/ch6-mixer-rl.pngng/ch6-mixer-rl.png)](/img/training/ch6-mixer-rl.png)
>
> **Context:** the section that introduces REINFORCE for sequence generation, before the paper's own hybrid method.
>
> **What it says:** the generator "can be viewed as an agent"; "an action refers to predicting the next word in the sequence at each time step", and "once the agent has reached the end of a sequence, it observes a reward". The reward can be any function, here BLEU or ROUGE-2.
>
> **Why it matters:** this is the same mapping every RLHF system uses today. Only the reward changed: from an n-gram overlap score to a model of human preference.

The 2024 paper that brought plain REINFORCE back to RLHF states the language-model version precisely, including where the reward goes:

> [!PAPER] Ahmadian et al. (2024), Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs · Section 2.1 · page 5
> [![A paragraph from Ahmadian et al.: when using PPO in the RL stage, the initial state is determined by the prompt, each generated token is modeled as an action and partial sequences are seen as states, with a discount factor of 1; only generating the EOS token carries a reward output by the reward model combined with a KL penalty, while for all other tokens only the KL component is non-zero](https://ishwarj.com/img/training/ch6-ahmadian-mdp.pngh6-ahmadian-mdp.png)](/img/training/ch6-ahmadian-mdp.png)
>
> **Context:** the background section, describing how PPO-based RLHF models text generation.
>
> **What it says:** "each generated token is modeled as an action, and partial sequences are seen as states", the discount is 1, and "only generating the <EOS> token carries a reward"; every other token gets only a KL term (Section 6.12 explains that term).
>
> **Why it matters:** this one paragraph is the whole setting of RLHF. Keep it in mind; everything in this chapter is about learning well when the only real signal arrives at the very end.

Here is the full dictionary in one place:

<figure class="fig"><svg viewBox="0 0 780 410" role="img" aria-label="A table mapping reinforcement learning terms to their meaning for language models."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">A dictionary: reinforcement learning words and what they mean for a language model</text><rect class="box-1" x="20.0" y="42.0" width="230.0" height="30.0" rx="6"/><text class="t-tick t-strong" x="32.0" y="62.0" text-anchor="start">state s_t</text><rect class="box" x="260.0" y="42.0" width="500.0" height="30.0" rx="6"/><text class="t-tick" x="272.0" y="62.0" text-anchor="start">prompt + the reply tokens written so far</text><rect class="box-1" x="20.0" y="78.0" width="230.0" height="30.0" rx="6"/><text class="t-tick t-strong" x="32.0" y="98.0" text-anchor="start">action a_t</text><rect class="box" x="260.0" y="78.0" width="500.0" height="30.0" rx="6"/><text class="t-tick" x="272.0" y="98.0" text-anchor="start">the next token (one of about 50,000 to 150,000)</text><rect class="box-1" x="20.0" y="114.0" width="230.0" height="30.0" rx="6"/><text class="t-tick t-strong" x="32.0" y="134.0" text-anchor="start">policy pi_theta(a | s)</text><rect class="box" x="260.0" y="114.0" width="500.0" height="30.0" rx="6"/><text class="t-tick" x="272.0" y="134.0" text-anchor="start">the language model: a softmax over the vocabulary</text><rect class="box-1" x="20.0" y="150.0" width="230.0" height="30.0" rx="6"/><text class="t-tick t-strong" x="32.0" y="170.0" text-anchor="start">episode, trajectory</text><rect class="box" x="260.0" y="150.0" width="500.0" height="30.0" rx="6"/><text class="t-tick" x="272.0" y="170.0" text-anchor="start">one full reply, from prompt to end token</text><rect class="box-1" x="20.0" y="186.0" width="230.0" height="30.0" rx="6"/><text class="t-tick t-strong" x="32.0" y="206.0" text-anchor="start">reward r_t</text><rect class="box" x="260.0" y="186.0" width="500.0" height="30.0" rx="6"/><text class="t-tick" x="272.0" y="206.0" text-anchor="start">usually 0 for every token, a score after the last one</text><rect class="box-1" x="20.0" y="222.0" width="230.0" height="30.0" rx="6"/><text class="t-tick t-strong" x="32.0" y="242.0" text-anchor="start">return G_t</text><rect class="box" x="260.0" y="222.0" width="500.0" height="30.0" rx="6"/><text class="t-tick" x="272.0" y="242.0" text-anchor="start">the sum of rewards from token t to the end</text><rect class="box-1" x="20.0" y="258.0" width="230.0" height="30.0" rx="6"/><text class="t-tick t-strong" x="32.0" y="278.0" text-anchor="start">value V(s_t)</text><rect class="box" x="260.0" y="258.0" width="500.0" height="30.0" rx="6"/><text class="t-tick" x="272.0" y="278.0" text-anchor="start">expected final score from this partial reply</text><rect class="box-1" x="20.0" y="294.0" width="230.0" height="30.0" rx="6"/><text class="t-tick t-strong" x="32.0" y="314.0" text-anchor="start">advantage A(s_t, a_t)</text><rect class="box" x="260.0" y="294.0" width="500.0" height="30.0" rx="6"/><text class="t-tick" x="272.0" y="314.0" text-anchor="start">how much better this token was than average here</text><rect class="box-1" x="20.0" y="330.0" width="230.0" height="30.0" rx="6"/><text class="t-tick t-strong" x="32.0" y="350.0" text-anchor="start">transition P(s' | s, a)</text><rect class="box" x="260.0" y="330.0" width="500.0" height="30.0" rx="6"/><text class="t-tick" x="272.0" y="350.0" text-anchor="start">deterministic: append the token</text><rect class="box-1" x="20.0" y="366.0" width="230.0" height="30.0" rx="6"/><text class="t-tick t-strong" x="32.0" y="386.0" text-anchor="start">discount gamma</text><rect class="box" x="260.0" y="366.0" width="500.0" height="30.0" rx="6"/><text class="t-tick" x="272.0" y="386.0" text-anchor="start">almost always 1 for language models</text></svg><figcaption>Reinforcement learning words and their language-model meaning. Two entries are unusual compared with robotics or games: the transition is deterministic (the new state is just the old text plus the token), and the discount is almost always 1.</figcaption></figure>

Two features of text make it an unusual RL problem, and both will matter later.

- **Transitions are deterministic.** In a game, pressing "jump" may or may not land you on the platform. In text, choosing the token " joy" *always* leads to the state "... a joy". All the randomness lives in the policy itself (and in the judge). Section 6.9 shows a pleasant consequence: with a perfect value function, a one-step error signal is exactly the quantity we want.
- **The action space is huge and the episode is long.** Qwen2.5 scores 151,936 possible tokens at each step and a reply can be hundreds of tokens long. A reward at the end must somehow be shared out among hundreds of choices. This is the **credit assignment problem**, and most of this chapter is about doing it well.

## 6.3 The goal: expected return

What exactly are we maximising? Not the reward of one reply: replies are random. We maximise the **expected** reward over the replies the policy would produce.

### Return and discount

First, how to add up the rewards inside one episode.

> [!DEFINITION] Return
> The **return** $$G_t$$ is the total reward collected from step $$t$$ to the end of the episode, possibly with far-away rewards weighted less. It answers: "from here on, how well did things go?"

$$G_t = \sum_{k=0}^{T-1-t} \gamma^{k}\, r_{t+k} = r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \cdots$$

where:

- $$t$$ is the current step (token position), counted from 0,
- $$T$$ is the number of steps in the episode (the reply length),
- $$r_{t+k}$$ is the reward received $$k$$ steps after step $$t$$,
- $$\gamma$$ (gamma) is the **discount factor**, a number between 0 and 1; a reward $$k$$ steps ahead is multiplied by $$\gamma^k$$,
- $$G_0$$, the return from the very first step, is the total score of the episode.

**Worked example.** Take the token world of Section 6.8: a six-token reply whose only reward is a 1 after the last token, so the rewards are $$(0, 0, 0, 0, 0, 1)$$. With $$\gamma = 1$$, every return is $$G_t = 1$$: from every position, the rest of the episode collected 1. With $$\gamma = 0.9$$, the last token sees $$G_5 = 1$$, the one before it $$G_4 = 0.9 \times 1 = 0.9$$, and the first token $$G_0 = 0.9^5 = 0.5905$$. `ch6_gae.py` prints exactly these values:

```text
== 1. return of the episode rewards [0, 0, 0, 0, 0, 1]
  gamma = 1.0: G_t for t = 0..5 = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
  gamma = 0.9: G_t for t = 0..5 = [0.5905, 0.6561, 0.729, 0.81, 0.9, 1.0]
```

<figure class="fig"><svg viewBox="0 0 780 570" role="img" aria-label="Bars of gamma to the power k for gamma 1, 0.9 and 0.5, over 24 steps."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">How much a reward k steps ahead counts today: the weight gamma^k</text><text class="t-note" x="20.0" y="114.0" text-anchor="start">gamma = 1.0</text><g class="mark"><title>gamma^0 = 1.000</title><rect class="s1" x="120" y="50.0" width="22" height="120.0" rx="2"/></g><text class="t-tick" x="131.0" y="184.0" text-anchor="middle">k=0</text><text class="t-tick" x="131.0" y="45.0" text-anchor="middle">1.00</text><g class="mark"><title>gamma^1 = 1.000</title><rect class="s1" x="146" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^2 = 1.000</title><rect class="s1" x="172" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^3 = 1.000</title><rect class="s1" x="198" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^4 = 1.000</title><rect class="s1" x="224" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^5 = 1.000</title><rect class="s1" x="250" y="50.0" width="22" height="120.0" rx="2"/></g><text class="t-tick" x="261.0" y="184.0" text-anchor="middle">k=5</text><text class="t-tick" x="261.0" y="45.0" text-anchor="middle">1.00</text><g class="mark"><title>gamma^6 = 1.000</title><rect class="s1" x="276" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^7 = 1.000</title><rect class="s1" x="302" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^8 = 1.000</title><rect class="s1" x="328" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^9 = 1.000</title><rect class="s1" x="354" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^10 = 1.000</title><rect class="s1" x="380" y="50.0" width="22" height="120.0" rx="2"/></g><text class="t-tick" x="391.0" y="184.0" text-anchor="middle">k=10</text><text class="t-tick" x="391.0" y="45.0" text-anchor="middle">1.00</text><g class="mark"><title>gamma^11 = 1.000</title><rect class="s1" x="406" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^12 = 1.000</title><rect class="s1" x="432" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^13 = 1.000</title><rect class="s1" x="458" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^14 = 1.000</title><rect class="s1" x="484" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^15 = 1.000</title><rect class="s1" x="510" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^16 = 1.000</title><rect class="s1" x="536" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^17 = 1.000</title><rect class="s1" x="562" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^18 = 1.000</title><rect class="s1" x="588" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^19 = 1.000</title><rect class="s1" x="614" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^20 = 1.000</title><rect class="s1" x="640" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^21 = 1.000</title><rect class="s1" x="666" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^22 = 1.000</title><rect class="s1" x="692" y="50.0" width="22" height="120.0" rx="2"/></g><g class="mark"><title>gamma^23 = 1.000</title><rect class="s1" x="718" y="50.0" width="22" height="120.0" rx="2"/></g><text class="t-tick" x="729.0" y="184.0" text-anchor="middle">k=23</text><text class="t-tick" x="729.0" y="45.0" text-anchor="middle">1.00</text><text class="t-note" x="20.0" y="274.0" text-anchor="start">gamma = 0.9</text><g class="mark"><title>gamma^0 = 1.000</title><rect class="s2" x="120" y="210.0" width="22" height="120.0" rx="2"/></g><text class="t-tick" x="131.0" y="344.0" text-anchor="middle">k=0</text><text class="t-tick" x="131.0" y="205.0" text-anchor="middle">1.00</text><g class="mark"><title>gamma^1 = 0.900</title><rect class="s2" x="146" y="222.0" width="22" height="108.0" rx="2"/></g><g class="mark"><title>gamma^2 = 0.810</title><rect class="s2" x="172" y="232.8" width="22" height="97.2" rx="2"/></g><g class="mark"><title>gamma^3 = 0.729</title><rect class="s2" x="198" y="242.5" width="22" height="87.5" rx="2"/></g><g class="mark"><title>gamma^4 = 0.656</title><rect class="s2" x="224" y="251.3" width="22" height="78.7" rx="2"/></g><g class="mark"><title>gamma^5 = 0.590</title><rect class="s2" x="250" y="259.1" width="22" height="70.9" rx="2"/></g><text class="t-tick" x="261.0" y="344.0" text-anchor="middle">k=5</text><text class="t-tick" x="261.0" y="254.1" text-anchor="middle">0.59</text><g class="mark"><title>gamma^6 = 0.531</title><rect class="s2" x="276" y="266.2" width="22" height="63.8" rx="2"/></g><g class="mark"><title>gamma^7 = 0.478</title><rect class="s2" x="302" y="272.6" width="22" height="57.4" rx="2"/></g><g class="mark"><title>gamma^8 = 0.430</title><rect class="s2" x="328" y="278.3" width="22" height="51.7" rx="2"/></g><g class="mark"><title>gamma^9 = 0.387</title><rect class="s2" x="354" y="283.5" width="22" height="46.5" rx="2"/></g><g class="mark"><title>gamma^10 = 0.349</title><rect class="s2" x="380" y="288.2" width="22" height="41.8" rx="2"/></g><text class="t-tick" x="391.0" y="344.0" text-anchor="middle">k=10</text><text class="t-tick" x="391.0" y="283.2" text-anchor="middle">0.35</text><g class="mark"><title>gamma^11 = 0.314</title><rect class="s2" x="406" y="292.3" width="22" height="37.7" rx="2"/></g><g class="mark"><title>gamma^12 = 0.282</title><rect class="s2" x="432" y="296.1" width="22" height="33.9" rx="2"/></g><g class="mark"><title>gamma^13 = 0.254</title><rect class="s2" x="458" y="299.5" width="22" height="30.5" rx="2"/></g><g class="mark"><title>gamma^14 = 0.229</title><rect class="s2" x="484" y="302.5" width="22" height="27.5" rx="2"/></g><g class="mark"><title>gamma^15 = 0.206</title><rect class="s2" x="510" y="305.3" width="22" height="24.7" rx="2"/></g><g class="mark"><title>gamma^16 = 0.185</title><rect class="s2" x="536" y="307.8" width="22" height="22.2" rx="2"/></g><g class="mark"><title>gamma^17 = 0.167</title><rect class="s2" x="562" y="310.0" width="22" height="20.0" rx="2"/></g><g class="mark"><title>gamma^18 = 0.150</title><rect class="s2" x="588" y="312.0" width="22" height="18.0" rx="2"/></g><g class="mark"><title>gamma^19 = 0.135</title><rect class="s2" x="614" y="313.8" width="22" height="16.2" rx="2"/></g><g class="mark"><title>gamma^20 = 0.122</title><rect class="s2" x="640" y="315.4" width="22" height="14.6" rx="2"/></g><g class="mark"><title>gamma^21 = 0.109</title><rect class="s2" x="666" y="316.9" width="22" height="13.1" rx="2"/></g><g class="mark"><title>gamma^22 = 0.098</title><rect class="s2" x="692" y="318.2" width="22" height="11.8" rx="2"/></g><g class="mark"><title>gamma^23 = 0.089</title><rect class="s2" x="718" y="319.4" width="22" height="10.6" rx="2"/></g><text class="t-tick" x="729.0" y="344.0" text-anchor="middle">k=23</text><text class="t-tick" x="729.0" y="314.4" text-anchor="middle">0.09</text><text class="t-note" x="20.0" y="434.0" text-anchor="start">gamma = 0.5</text><g class="mark"><title>gamma^0 = 1.000</title><rect class="s3" x="120" y="370.0" width="22" height="120.0" rx="2"/></g><text class="t-tick" x="131.0" y="504.0" text-anchor="middle">k=0</text><text class="t-tick" x="131.0" y="365.0" text-anchor="middle">1.00</text><g class="mark"><title>gamma^1 = 0.500</title><rect class="s3" x="146" y="430.0" width="22" height="60.0" rx="2"/></g><g class="mark"><title>gamma^2 = 0.250</title><rect class="s3" x="172" y="460.0" width="22" height="30.0" rx="2"/></g><g class="mark"><title>gamma^3 = 0.125</title><rect class="s3" x="198" y="475.0" width="22" height="15.0" rx="2"/></g><g class="mark"><title>gamma^4 = 0.062</title><rect class="s3" x="224" y="482.5" width="22" height="7.5" rx="2"/></g><g class="mark"><title>gamma^5 = 0.031</title><rect class="s3" x="250" y="486.2" width="22" height="3.8" rx="2"/></g><text class="t-tick" x="261.0" y="504.0" text-anchor="middle">k=5</text><text class="t-tick" x="261.0" y="481.2" text-anchor="middle">0.03</text><g class="mark"><title>gamma^6 = 0.016</title><rect class="s3" x="276" y="488.1" width="22" height="1.9" rx="2"/></g><g class="mark"><title>gamma^7 = 0.008</title><rect class="s3" x="302" y="489.1" width="22" height="1.0" rx="2"/></g><g class="mark"><title>gamma^8 = 0.004</title><rect class="s3" x="328" y="489.5" width="22" height="1.0" rx="2"/></g><g class="mark"><title>gamma^9 = 0.002</title><rect class="s3" x="354" y="489.8" width="22" height="1.0" rx="2"/></g><g class="mark"><title>gamma^10 = 0.001</title><rect class="s3" x="380" y="489.9" width="22" height="1.0" rx="2"/></g><text class="t-tick" x="391.0" y="504.0" text-anchor="middle">k=10</text><text class="t-tick" x="391.0" y="484.9" text-anchor="middle">0.00</text><g class="mark"><title>gamma^11 = 0.000</title><rect class="s3" x="406" y="489.9" width="22" height="1.0" rx="2"/></g><g class="mark"><title>gamma^12 = 0.000</title><rect class="s3" x="432" y="490.0" width="22" height="1.0" rx="2"/></g><g class="mark"><title>gamma^13 = 0.000</title><rect class="s3" x="458" y="490.0" width="22" height="1.0" rx="2"/></g><g class="mark"><title>gamma^14 = 0.000</title><rect class="s3" x="484" y="490.0" width="22" height="1.0" rx="2"/></g><g class="mark"><title>gamma^15 = 0.000</title><rect class="s3" x="510" y="490.0" width="22" height="1.0" rx="2"/></g><g class="mark"><title>gamma^16 = 0.000</title><rect class="s3" x="536" y="490.0" width="22" height="1.0" rx="2"/></g><g class="mark"><title>gamma^17 = 0.000</title><rect class="s3" x="562" y="490.0" width="22" height="1.0" rx="2"/></g><g class="mark"><title>gamma^18 = 0.000</title><rect class="s3" x="588" y="490.0" width="22" height="1.0" rx="2"/></g><g class="mark"><title>gamma^19 = 0.000</title><rect class="s3" x="614" y="490.0" width="22" height="1.0" rx="2"/></g><g class="mark"><title>gamma^20 = 0.000</title><rect class="s3" x="640" y="490.0" width="22" height="1.0" rx="2"/></g><g class="mark"><title>gamma^21 = 0.000</title><rect class="s3" x="666" y="490.0" width="22" height="1.0" rx="2"/></g><g class="mark"><title>gamma^22 = 0.000</title><rect class="s3" x="692" y="490.0" width="22" height="1.0" rx="2"/></g><g class="mark"><title>gamma^23 = 0.000</title><rect class="s3" x="718" y="490.0" width="22" height="1.0" rx="2"/></g><text class="t-tick" x="729.0" y="504.0" text-anchor="middle">k=23</text><text class="t-tick" x="729.0" y="485.0" text-anchor="middle">0.00</text><text class="t-muted" x="20.0" y="534.0" text-anchor="start">With gamma = 1 (what RLHF uses) a score after the 24th token counts in full for the first token.</text><text class="t-muted" x="20.0" y="551.0" text-anchor="start">A smaller gamma makes far-away rewards matter less: lower variance, but it changes what is optimised.</text></svg><figcaption>The weight gamma^k given to a reward k steps ahead. With gamma = 1 (the RLHF default) a reward at the end counts fully for every token. With gamma = 0.5 a reward 10 steps away is worth one thousandth as much, so early tokens would barely feel the final score.</figcaption></figure>

Why discount at all? In robotics and games, episodes can last forever, and a sum of infinitely many rewards can be infinite; $$\gamma < 1$$ keeps it finite. Discounting also reduces noise, because far-away rewards (which depend on many random future choices) count less. But it changes the goal: with $$\gamma < 1$$ the model is pushed to collect reward *soon*. For a reply of a few hundred tokens with one score at the end, there is no reason to prefer early reward, so RLHF almost always uses $$\gamma = 1$$. The GAE paper calls $$\gamma$$ a variance-reduction parameter rather than part of the problem, and we will see in Section 6.10 that a second parameter, $$\lambda$$, does the same job more gently.

### The objective

> [!DEFINITION] Objective of policy optimisation
> The expected return of the policy: the average total reward over all the episodes the policy might produce, each weighted by how likely the policy is to produce it.

$$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\big[R(\tau)\big] = \sum_{\tau} p_\theta(\tau)\, R(\tau)$$

where:

- $$\theta$$ are the model's weights,
- $$\tau$$ is one possible trajectory (one possible reply, with its states and actions),
- $$\tau \sim \pi_\theta$$ means "trajectories sampled by running the policy",
- $$R(\tau) = G_0$$ is the total reward of that trajectory,
- $$p_\theta(\tau)$$ is the probability that the policy produces exactly that trajectory,
- the sum runs over every possible trajectory (for text, every possible reply: astronomically many).

For text, $$p_\theta(\tau)$$ is just the probability of the reply, given by the chain rule:

$$p_\theta(\tau) = \prod_{t=0}^{T-1} \pi_\theta(a_t \mid s_t) \quad\Longrightarrow\quad \log p_\theta(\tau) = \sum_{t=0}^{T-1} \log \pi_\theta(a_t \mid s_t)$$

where $$\pi_\theta(a_t \mid s_t)$$ is the probability the model gave to the token it actually chose at position $$t$$. (In a general RL problem there would also be factors $$P(s_{t+1} \mid s_t, a_t)$$ for the environment's random transitions. For text they are all 1, because appending a token is certain.)

**Worked example.** In the GPT-2 experiment of Section 6.11, one prompt was "It's unbelievable but the fourth is better" and the trained model (the run with a KL penalty of 0.05) continued it with " than most. It's awesome and with wonderful music, ...". Its first three tokens had log-probabilities $$-1.318$$ (" than", probability 0.268), $$-2.378$$ (" most", 0.093) and $$-0.593$$ (".", 0.553). The probability of those three tokens together is $$0.268 \times 0.093 \times 0.553 \approx 0.014$$, and the sum of the logs is $$-4.289$$, with $$e^{-4.289} = 0.0137$$ (the small difference is rounding in the three probabilities). Over all 24 reply tokens the log-probabilities add up to $$-61.12$$, so this exact reply had probability $$e^{-61.12} \approx 3 \times 10^{-27}$$. Every individual reply is astronomically unlikely, which is why we always work with log-probabilities and with averages over many sampled replies, never with the probability of one reply.

## 6.4 The policy gradient, derived step by step

We want to climb $$J(\theta)$$ by gradient ascent: compute $$\nabla_\theta J$$ and take a small step in that direction. Two obstacles make this look impossible at first.

1. **The reward is not differentiable.** It may come from a classifier, a person, a unit test or a regular expression. We cannot backpropagate through "the tests passed".
2. **Sampling is not differentiable either.** The reply was produced by drawing random tokens. There is no gradient through "draw token 4,217".

The trick that gets around both is about thirty years old, and it is short enough to do by hand.

### Step 1: write the gradient of the sum

Start from $$J(\theta) = \sum_\tau p_\theta(\tau) R(\tau)$$. The reward of a fixed trajectory does not depend on $$\theta$$; only the probability of producing it does. So the gradient moves inside the sum and lands on $$p_\theta$$:

$$\nabla_\theta J(\theta) = \sum_{\tau} \nabla_\theta p_\theta(\tau)\; R(\tau)$$

This is correct but useless as it stands: it is a sum over every possible reply, and it is *not* an average (there is no probability in front of each term), so we cannot estimate it by sampling a few replies.

### Step 2: the log-derivative trick

From calculus, the derivative of a logarithm is $$\nabla \log x = \nabla x / x$$. Rearranged:

$$\nabla_\theta p_\theta(\tau) = p_\theta(\tau)\, \nabla_\theta \log p_\theta(\tau)$$

where both sides are the same vector, written two ways: the right side multiplies and divides by $$p_\theta(\tau)$$.

> [!DEFINITION] Log-derivative trick (score function)
> The identity $$\nabla p = p \, \nabla \log p$$. The vector $$\nabla_\theta \log p_\theta(x)$$ is called the **score function** of $$x$$. The trick turns the gradient of an expectation into an expectation of a gradient, which can be estimated by sampling. It is also called the likelihood-ratio method.

### Step 3: substitute, and the sum becomes an average

$$\nabla_\theta J(\theta) = \sum_{\tau} p_\theta(\tau)\, \nabla_\theta \log p_\theta(\tau)\, R(\tau) = \mathbb{E}_{\tau \sim \pi_\theta}\Big[R(\tau)\, \nabla_\theta \log p_\theta(\tau)\Big]$$

Now there *is* a probability in front of every term, so the sum is an average over trajectories drawn from the policy. Averages can be estimated: sample $$N$$ replies, compute $$R(\tau)\, \nabla_\theta \log p_\theta(\tau)$$ for each, and average.

### Step 4: expand the log-probability of the trajectory

From Section 6.3, $$\log p_\theta(\tau) = \sum_t \log \pi_\theta(a_t \mid s_t)$$ (plus environment terms that do not depend on $$\theta$$ and therefore have zero gradient). So:

$$\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\Big[R(\tau) \sum_{t=0}^{T-1} \nabla_\theta \log \pi_\theta(a_t \mid s_t)\Big]$$

where:

- $$\nabla_\theta J(\theta)$$ is the direction in weight space that increases expected reward fastest,
- $$\mathbb{E}_{\tau \sim \pi_\theta}$$ is the average over replies sampled from the current model,
- $$R(\tau)$$ is the total reward of one reply,
- $$\nabla_\theta \log \pi_\theta(a_t \mid s_t)$$ is the gradient of the log-probability of the token chosen at position $$t$$: exactly the gradient that SFT computes when it trains on that token.

<figure class="fig"><svg viewBox="0 0 780 308" role="img" aria-label="Four steps of the log-derivative trick that turns the gradient of an expected reward into an average that can be sampled."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><rect class="box" x="20.0" y="16.0" width="370.0" height="128.0" rx="12"/><circle class="node on" cx="38.0" cy="34.0" r="11"/><text class="t-note" x="38.0" y="38.5" text-anchor="middle">1</text><text class="t-title" x="56.0" y="39.0" text-anchor="start">Start: the objective</text><text class="t-tick" x="42.0" y="72.0" text-anchor="start">J(theta) = sum over replies y of</text><text class="t-tick" x="42.0" y="94.0" text-anchor="start">p_theta(y) R(y)</text><text class="t-tick" x="42.0" y="116.0" text-anchor="start">R does not depend on theta</text><rect class="box" x="410.0" y="16.0" width="370.0" height="128.0" rx="12"/><circle class="node on" cx="428.0" cy="34.0" r="11"/><text class="t-note" x="428.0" y="38.5" text-anchor="middle">2</text><text class="t-title" x="446.0" y="39.0" text-anchor="start">Move the gradient inside</text><text class="t-tick" x="432.0" y="72.0" text-anchor="start">grad J = sum over y of</text><text class="t-tick" x="432.0" y="94.0" text-anchor="start">grad p_theta(y) R(y)</text><text class="t-tick" x="432.0" y="116.0" text-anchor="start">but this is not an average: no p in front</text><rect class="box" x="20.0" y="162.0" width="370.0" height="128.0" rx="12"/><circle class="node on" cx="38.0" cy="180.0" r="11"/><text class="t-note" x="38.0" y="184.5" text-anchor="middle">3</text><text class="t-title" x="56.0" y="185.0" text-anchor="start">The trick: grad p = p grad log p</text><text class="t-tick" x="42.0" y="218.0" text-anchor="start">because grad log p = grad p / p</text><text class="t-tick" x="42.0" y="240.0" text-anchor="start">grad J = sum over y of p_theta(y)</text><text class="t-tick" x="42.0" y="262.0" text-anchor="start">grad log p_theta(y) R(y)</text><rect class="box-on" x="410.0" y="162.0" width="370.0" height="128.0" rx="12"/><circle class="node on" cx="428.0" cy="180.0" r="11"/><text class="t-note" x="428.0" y="184.5" text-anchor="middle">4</text><text class="t-title" x="446.0" y="185.0" text-anchor="start">Now it is an average: sample it</text><text class="t-tick" x="432.0" y="218.0" text-anchor="start">grad J = E over y ~ p_theta of</text><text class="t-tick" x="432.0" y="240.0" text-anchor="start">[ R(y) grad log p_theta(y) ]</text><text class="t-tick" x="432.0" y="262.0" text-anchor="start">estimate: sample y, compute, average</text></svg><figcaption>The log-derivative trick in four steps. The key move is frame 3: replacing grad p by p times grad log p puts a probability in front of every term, so the gradient becomes an average that can be estimated from samples.</figcaption></figure>

Read the result in words: **make every token of a reply more likely, in proportion to the reward the whole reply received.** High-reward replies get pushed up hard. Low-reward replies get pushed up less (or down, if the reward is negative). Nothing in the formula requires the reward to be differentiable, and we never differentiate through the sampling: we only need the log-probabilities of tokens the model already chose, which one ordinary forward pass gives us.

There is a beautiful connection here. If every reply had reward 1 and the replies were written by people instead of sampled, the formula would be exactly the gradient of the SFT loss of Chapter 5. RL with this estimator is "SFT on the model's own samples, weighted by how good they were".

### The gradient for a softmax, by hand

To see real numbers, start with the smallest possible case: one decision, no sequence. A "prompt" has four possible "replies" (think of four candidate answers), the policy is a softmax over four numbers $$\theta_1, \ldots, \theta_4$$, and reply $$k$$ earns a noisy rating with mean $$\mu_k$$. This is called a **multi-armed bandit**, after slot machines ("one-armed bandits") with several levers.

> [!DEFINITION] Multi-armed bandit
> An RL problem with a single state: the agent picks one of $$K$$ actions ("arms"), receives a random reward from that arm, and the episode ends. It isolates the core problem of learning from rewards without any sequence or credit assignment.

For a softmax policy $$\pi_k = e^{\theta_k} / \sum_j e^{\theta_j}$$, the log-probability of action $$a$$ is $$\log \pi_a = \theta_a - \log \sum_j e^{\theta_j}$$, and its gradient is

$$\frac{\partial \log \pi_a}{\partial \theta_k} = \mathbb{1}[k = a] - \pi_k$$

where:

- $$a$$ is the action that was sampled,
- $$k$$ indexes the four logits,
- $$\mathbb{1}[k = a]$$ is 1 for the chosen action and 0 for the others,
- $$\pi_k$$ is the current probability of action $$k$$.

So the score vector is "one-hot of the chosen action minus the probability vector". It raises the logit of the chosen action and lowers all the others, each by its current probability.

**Worked example** (from `ch6_bandit.py`). The mean ratings are $$\mu = (5, 6, 7, 8)$$, rating noise has standard deviation 1, and the policy starts uniform: $$\theta = 0$$, so $$\pi = (0.25, 0.25, 0.25, 0.25)$$. Suppose reply 3 is sampled and rated $$r = 7.3$$.

- Score: $$\text{onehot}(3) - \pi = (-0.25, -0.25, +0.75, -0.25)$$.
- REINFORCE estimate: $$r \times \text{score} = 7.3 \times (-0.25, -0.25, 0.75, -0.25) = (-1.825, -1.825, +5.475, -1.825)$$.

What is the *true* gradient? Here we can compute it exactly, because $$J(\theta) = \sum_k \pi_k \mu_k$$ is a small sum. Differentiating the softmax gives $$\partial J / \partial \theta_k = \pi_k (\mu_k - J)$$. At the start $$J = 0.25 \times (5 + 6 + 7 + 8) = 6.5$$, so the true gradient is $$0.25 \times (5 - 6.5,\; 6 - 6.5,\; 7 - 6.5,\; 8 - 6.5) = (-0.375, -0.125, +0.125, +0.375)$$.

The single-sample estimate $$(-1.825, -1.825, +5.475, -1.825)$$ looks nothing like the true gradient. It says "reply 3 is wonderful, push everything else down equally", when in truth reply 4 is best and reply 3 deserves only a small push. Yet the policy gradient theorem says the estimate is correct *on average*. The script checks this with 200,000 samples:

```text
== 2. Monte Carlo check at theta = 0 with 200,000 samples
mean score vector E[grad log pi]     = [-0.0007  0.0014 -0.0012  0.0005] (should be 0)
mean estimate, no baseline           = [-0.3781 -0.116   0.1159  0.3781]
mean estimate, baseline b = 6.5      = [-0.3737 -0.1249  0.1236  0.375 ]
exact gradient                       = [-0.375 -0.125  0.125  0.375]
total variance (sum over 4 components): no baseline 33.028, baseline 1.369, ratio 24.1x
```

The average of 200,000 noisy estimates is $$(-0.378, -0.116, 0.116, 0.378)$$, within Monte Carlo error of the exact $$(-0.375, -0.125, 0.125, 0.375)$$. So the estimator is right on average. The last line is the bad news, and the subject of Sections 6.6 and 6.7: one estimate has a total variance of 33, while the gradient we are trying to find has a squared length of only $$0.375^2 + 0.125^2 + 0.125^2 + 0.375^2 = 0.3125$$. The noise is about a hundred times bigger than the signal.

### The policy gradient theorem

The derivation above treats the whole trajectory at once. Sutton, McAllester, Singh and Mansour (AT&T Labs, NeurIPS 1999, published 2000) proved a more general form that works with any differentiable policy and makes the role of value functions explicit.

> [!PAPER] Sutton, McAllester, Singh and Mansour (2000), Policy Gradient Methods for Reinforcement Learning with Function Approximation · Abstract · page 1
> [![The abstract of Sutton et al. 2000 with highlights on the policy is explicitly represented by its own function approximator, and Williams's REINFORCE method and actor-critic methods are examples of this approach](https://ishwarj.com/img/training/ch6-sutton-abstract.pngsutton-abstract.png)](/img/training/ch6-sutton-abstract.png)
>
> **Context:** the abstract. In 1999 most RL research learned a value function and acted greedily on it (Q-learning, for example).
>
> **What it says:** represent the policy directly, "explicitly represented by its own function approximator", and update it "according to the gradient of expected reward". "Williams's REINFORCE method and actor-critic methods are examples of this approach."
>
> **Why it matters:** a language model *is* a policy represented by its own function approximator. This paper is the reason policy gradients, and not value-based methods like Q-learning, became the natural tool for training language models with RL.

> [!PAPER] Sutton et al. (2000) · Theorem 1 · page 3
> [![Theorem 1 (Policy Gradient) of Sutton et al.: for any MDP, the derivative of performance rho with respect to theta equals the sum over states of d pi of s times the sum over actions of the derivative of pi of s and a times Q pi of s and a; highlighted: the effect of policy changes on the distribution of states does not appear](https://ishwarj.com/img/training/ch6-sutton-theorem.png-sutton-theorem.png)](/img/training/ch6-sutton-theorem.png)
>
> **Context:** the first result of the paper.
>
> **What it says:** $$\partial \rho / \partial \theta = \sum_s d^\pi(s) \sum_a \frac{\partial \pi(s,a)}{\partial \theta} Q^\pi(s,a)$$. The key point is that there is no term for how the *distribution of states* changes when the policy changes: "the effect of policy changes on the distribution of states does not appear".
>
> **Why it matters:** changing the policy changes which states you visit, which seems impossible to differentiate. The theorem says you do not have to. You only need samples of states from the current policy and an estimate of how good each action was.

In our notation, with the log-derivative trick applied to $$\partial \pi / \partial \theta$$, the theorem reads:

$$\nabla_\theta J(\theta) = \mathbb{E}_{s \sim d^{\pi},\, a \sim \pi_\theta}\Big[\nabla_\theta \log \pi_\theta(a \mid s)\; Q^{\pi}(s, a)\Big]$$

where:

- $$d^{\pi}(s)$$ is how often the policy visits state $$s$$ (for text: how often the model produces this exact prefix),
- $$a \sim \pi_\theta$$ is an action (token) sampled by the policy in that state,
- $$Q^{\pi}(s, a)$$ is the expected return after taking action $$a$$ in state $$s$$ and following the policy afterwards (Section 6.8 defines it carefully).

Compare it with Step 4. There, each token's log-probability was multiplied by $$R(\tau)$$, the reward of the *whole* episode. Here it is multiplied by $$Q^\pi(s_t, a_t)$$, the expected reward *from that token onwards*. That difference leads straight to the next improvement.

### Only the future matters: reward-to-go

A token at position $$t$$ cannot influence rewards that were received *before* it. Those rewards are already fixed when the token is chosen, so on average they add nothing to its gradient, only noise. Replacing $$R(\tau)$$ by the return from $$t$$ onwards gives an estimator that is still unbiased and less noisy:

$$\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\Big[\sum_{t=0}^{T-1} G_t\, \nabla_\theta \log \pi_\theta(a_t \mid s_t)\Big]$$

where $$G_t = \sum_{t' \ge t} r_{t'}$$ is the **reward-to-go** (the return of Section 6.3 with $$\gamma = 1$$), and $$G_t$$ is a single-sample estimate of $$Q^\pi(s_t, a_t)$$.

For a reply whose only reward is at the end, every $$G_t$$ equals the final score, so reward-to-go changes nothing. It starts to matter as soon as there are per-token rewards, which is exactly what the KL penalty of Section 6.12 creates.

## 6.5 REINFORCE (Williams, 1992)

The estimator we just derived has a name and a birthday. Ronald J. Williams, at Northeastern University, published it in the journal *Machine Learning* in 1992 (building on his technical reports of the late 1980s), as a family of learning rules for neural networks with random units.

> [!PAPER] Williams (1992), Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning · Section 4 · page 5 of the preprint
> [![Section 4 REINFORCE Algorithms of Williams 1992: each weight w_ij is incremented by alpha_ij times r minus b_ij times e_ij, where alpha is a learning rate factor, b is a reinforcement baseline and e is the characteristic eligibility, the derivative of ln g with respect to w; highlighted, the update rule and the sentence explaining that the name is an acronym for REward Increment equals Nonnegative Factor times Offset Reinforcement times Characteristic Eligibility; Theorem 1 says the average update lies in a direction in which expected reinforcement increases](https://ishwarj.com/img/training/ch6-williams-reinforce.pngliams-reinforce.png)](/img/training/ch6-williams-reinforce.png)
>
> **Context:** the section that defines the algorithm family. (This excerpt is from the preprint copy hosted at UMass, linked in the References; the journal version is the same text on pages 229 to 256.)
>
> **What it says:** each weight changes by $$\Delta w_{ij} = \alpha_{ij}(r - b_{ij})e_{ij}$$, where $$\alpha$$ is a learning rate, $$b$$ is a "reinforcement baseline" and $$e_{ij} = \partial \ln g_i / \partial w_{ij}$$ is the "characteristic eligibility", our score function. The name is an acronym: "REward Increment = Nonnegative Factor times Offset Reinforcement times Characteristic Eligibility". Theorem 1 proves the average update points uphill on expected reward.
>
> **Why it matters:** the baseline $$b$$ is there from the very first page. Williams already knew the raw estimator was too noisy, and his theorem holds for any baseline that does not depend on the action taken.

Section 5 of the same paper handles episodes with many steps and a single reward at the end, which is exactly our text setting:

> [!PAPER] Williams (1992) · Section 5, equation 11 · page 8 of the preprint
> [![Section 5 of Williams 1992 on episodic REINFORCE: a single reinforcement value r is delivered to the net at the end of each episode; at the end of each episode each parameter w_ij is incremented by alpha times r minus b times the sum over time steps t of e_ij of t; any algorithm of this form is called an episodic REINFORCE algorithm](https://ishwarj.com/img/training/ch6-williams-episodic.pnglliams-episodic.png)](/img/training/ch6-williams-episodic.png)
>
> **Context:** the extension to tasks where the network acts for $$k$$ steps before being scored.
>
> **What it says:** "a single reinforcement value $$r$$ is delivered to the net at the end of each episode", and each weight is incremented by $$\alpha(r - b)\sum_{t=1}^{k} e_{ij}(t)$$: the reward minus a baseline, times the eligibilities summed over all the time steps.
>
> **Why it matters:** replace "time step" by "token" and this is RL for a language model with a reward model, written down thirty years before RLHF.

> [!DEFINITION] REINFORCE
> The policy gradient estimator used directly as a learning rule: sample episodes from the current policy, then change the weights by the learning rate times (return minus baseline) times the gradient of the log-probability of the actions taken.

### REINFORCE as a loss

Deep learning libraries minimise losses, so we write REINFORCE as one whose gradient is the (negative) policy gradient:

$$\mathcal{L}_{\text{PG}}(\theta) = -\frac{1}{N}\sum_{i=1}^{N} \sum_{t=0}^{T-1} \big(G^{(i)}_t - b_t\big)\, \log \pi_\theta\big(a^{(i)}_t \mid s^{(i)}_t\big)$$

where:

- $$N$$ is the number of sampled episodes (replies) in the batch, indexed by $$i$$,
- $$G^{(i)}_t$$ is the reward-to-go of reply $$i$$ from token $$t$$,
- $$b_t$$ is a baseline (Section 6.7; for now think of it as 0),
- $$\log \pi_\theta(a^{(i)}_t \mid s^{(i)}_t)$$ is the log-probability the model gives, *with gradient*, to the token it sampled,
- the factor $$(G - b)$$ is treated as a fixed number: no gradient flows through the reward or the baseline.

Comparing with the SFT loss of Chapter 5, Section 5.5: SFT is $$-\frac{1}{\sum m}\sum m_t \log p_\theta(x_{t+1} \mid \ldots)$$. REINFORCE has the same shape, with the mask $$m_t$$ replaced by the weight $$(G_t - b_t)$$ and the human-written tokens replaced by the model's own samples. A weight can be negative, which SFT never has: a negative weight *lowers* the probability of the tokens of a bad reply.

> [!WARNING] The value of this loss means nothing
> In SFT, a falling loss means the model is improving. The REINFORCE loss is not a measure of anything: its value depends on the batch of samples, the baseline and the rewards, and it can go up while the model improves. Track the average reward (and the KL of Section 6.12) instead.

In the bandit, one REINFORCE step is three lines of numpy (simplified from `ch6_bandit.py`):

```python
p = softmax(th)                         # current policy over the four replies
a = rng.choice(K, p=p)                  # sample a reply
r = MU[a] + SD * rng.standard_normal()  # get its noisy rating
sc = -p; sc[a] += 1                     # score vector: onehot(a) - p
th = th + LR * (r - base) * sc          # REINFORCE step (base = 0 means no baseline)
```

The first line turns the four logits into probabilities. The second samples one reply from them; this is the "trial". The third is the environment: the reply's mean rating plus Gaussian noise. The fourth computes the score vector $$\nabla_\theta \log \pi_a$$ from the formula above. The last line is the whole algorithm: move the logits in the direction of the score, scaled by the reward minus the baseline. For a neural network the only change is that the score vector is computed by backpropagation instead of by hand.

## 6.6 Why variance is the enemy

An unbiased estimator is only useful if it is not too noisy. Each REINFORCE step uses a handful of samples, and the bandit above showed that one sample's estimate can point in a direction that has little to do with the true gradient.

> [!DEFINITION] Bias and variance of an estimator
> An estimator is **unbiased** if its average over many samples equals the true value. Its **variance** measures how far a single estimate typically lands from that average. Low bias and high variance means "right on average, wrong every single time"; to get a usable answer you must average many samples or take tiny steps.

Look again at the single step. The rating 7.3 for reply 3 is only a little above the average rating of 6.5. But because every rating is positive (between about 4 and 9), REINFORCE *always* increases the probability of the sampled reply, and the increase is proportional to the raw rating:

<figure class="fig"><svg viewBox="0 0 780 318" role="img" aria-label="Bars of the four reply probabilities before and after one update, without and with a baseline."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">One REINFORCE step on the four-reply bandit: reply 3 was sampled and rated r = 7.3</text><text class="t-note" x="140.0" y="56.0" text-anchor="middle">policy before</text><rect class="box-ghost" x="50" y="135.0" width="36" height="65.0" rx="3"/><text class="t-tick" x="68.0" y="129.0" text-anchor="middle">0.250</text><text class="t-tick" x="68.0" y="216.0" text-anchor="middle">reply 1</text><rect class="box-ghost" x="96" y="135.0" width="36" height="65.0" rx="3"/><text class="t-tick" x="114.0" y="129.0" text-anchor="middle">0.250</text><text class="t-tick" x="114.0" y="216.0" text-anchor="middle">reply 2</text><rect class="box-ghost" x="142" y="135.0" width="36" height="65.0" rx="3"/><text class="t-tick" x="160.0" y="129.0" text-anchor="middle">0.250</text><text class="t-tick" x="160.0" y="216.0" text-anchor="middle">reply 3</text><rect class="box-ghost" x="188" y="135.0" width="36" height="65.0" rx="3"/><text class="t-tick" x="206.0" y="129.0" text-anchor="middle">0.250</text><text class="t-tick" x="206.0" y="216.0" text-anchor="middle">reply 4</text><line class="axis" x1="44.0" y1="200.0" x2="236.0" y2="200.0"/><text class="t-note" x="390.0" y="56.0" text-anchor="middle">after, no baseline</text><rect class="box-2" x="300" y="148.8" width="36" height="51.2" rx="3"/><text class="t-tick" x="318.0" y="142.8" text-anchor="middle">0.197</text><text class="t-tick" x="318.0" y="216.0" text-anchor="middle">reply 1</text><rect class="box-2" x="346" y="148.8" width="36" height="51.2" rx="3"/><text class="t-tick" x="364.0" y="142.8" text-anchor="middle">0.197</text><text class="t-tick" x="364.0" y="216.0" text-anchor="middle">reply 2</text><rect class="box-2" x="392" y="93.7" width="36" height="106.3" rx="3"/><text class="t-tick" x="410.0" y="87.7" text-anchor="middle">0.409</text><text class="t-tick" x="410.0" y="216.0" text-anchor="middle">reply 3</text><rect class="box-2" x="438" y="148.8" width="36" height="51.2" rx="3"/><text class="t-tick" x="456.0" y="142.8" text-anchor="middle">0.197</text><text class="t-tick" x="456.0" y="216.0" text-anchor="middle">reply 4</text><line class="axis" x1="294.0" y1="200.0" x2="486.0" y2="200.0"/><text class="t-note" x="640.0" y="56.0" text-anchor="middle">after, baseline b = 6.5</text><rect class="box-1" x="550" y="136.3" width="36" height="63.7" rx="3"/><text class="t-tick" x="568.0" y="130.3" text-anchor="middle">0.245</text><text class="t-tick" x="568.0" y="216.0" text-anchor="middle">reply 1</text><rect class="box-1" x="596" y="136.3" width="36" height="63.7" rx="3"/><text class="t-tick" x="614.0" y="130.3" text-anchor="middle">0.245</text><text class="t-tick" x="614.0" y="216.0" text-anchor="middle">reply 2</text><rect class="box-1" x="642" y="131.0" width="36" height="69.0" rx="3"/><text class="t-tick" x="660.0" y="125.0" text-anchor="middle">0.265</text><text class="t-tick" x="660.0" y="216.0" text-anchor="middle">reply 3</text><rect class="box-1" x="688" y="136.3" width="36" height="63.7" rx="3"/><text class="t-tick" x="706.0" y="130.3" text-anchor="middle">0.245</text><text class="t-tick" x="706.0" y="216.0" text-anchor="middle">reply 4</text><line class="axis" x1="544.0" y1="200.0" x2="736.0" y2="200.0"/><text class="t-tick" x="20.0" y="250.0" text-anchor="start">score vector  onehot(3) - pi = (-0.25, -0.25, +0.75, -0.25)</text><text class="t-tick" x="20.0" y="270.0" text-anchor="start">no baseline:   r x score = (-1.825, -1.825, +5.475, -1.825)   pushes reply 3 up hard, although 7.3 is only a bit above average</text><text class="t-tick" x="20.0" y="290.0" text-anchor="start">baseline 6.5:  (r - b) x score = (-0.200, -0.200, +0.600, -0.200)   a small push, the size the evidence deserves</text></svg><figcaption>One REINFORCE step with learning rate 0.1, computed by ch6_bandit.py. Without a baseline, a rating of 7.3 pushes the probability of reply 3 from 0.25 to 0.41 in one step, although reply 3 is not the best. With the baseline 6.5 the push is to 0.265, in proportion to how much better than average the rating actually was.</figcaption></figure>

```text
== 1. one REINFORCE step by hand, theta = 0
pi            = [0.25 0.25 0.25 0.25]
action a = 3, reward r = 7.3
score  = onehot(a) - pi = [-0.25 -0.25  0.75 -0.25]
grad (no baseline) = r * score       = [-1.825 -1.825  5.475 -1.825]
baseline b = V = sum_k pi_k mu_k = 6.5
grad (baseline)    = (r - b) * score = [-0.2 -0.2  0.6 -0.2]
exact gradient of J = pi_k (mu_k - J)  = [-0.375 -0.125  0.125  0.375]
new pi after one step, lr 0.1, no baseline: [0.197  0.197  0.4089 0.197 ]
new pi after one step, lr 0.1, baseline   : [0.2449 0.2449 0.2653 0.2449]
```

Without a baseline, after one step the policy picks reply 3 with probability 0.41. That makes reply 3 more likely to be sampled again, which pushes it up again: a rich-get-richer loop. Over many steps the better replies do win *on average*, because their pushes are slightly larger, but the noise can lock the policy onto a worse reply before the evidence has accumulated. Section 6.7 shows how often that happens. On a language model the same thing happens at the level of phrases.

Sutton and colleagues put the practical consequence plainly in 2000:

> [!PAPER] Sutton et al. (2000) · Introduction · page 2
> [![A paragraph of Sutton et al. 2000 with highlights on REINFORCE learns much more slowly than RL methods using value functions, and Learning a value function and using it to reduce the variance of the gradient estimate appears to be essential for rapid learning](https://ishwarj.com/img/training/ch6-sutton-variance.pngsutton-variance.png)](/img/training/ch6-sutton-variance.png)
>
> **Context:** the introduction, comparing their result with Williams' REINFORCE.
>
> **What it says:** REINFORCE finds an unbiased gradient "but without the assistance of a learned value function", and "REINFORCE learns much more slowly than RL methods using value functions". "Learning a value function and using it to reduce the variance of the gradient estimate appears to be essential for rapid learning."
>
> **Why it matters:** this sentence sets the agenda for the next 25 years: baselines, critics, advantages and GAE are all ways to cut variance. (For language models, the 2024 papers in Section 6.7 found that a good *sampled* baseline can replace the learned value function.)

Variance is worse for language models than for the bandit, for three reasons:

1. **Long episodes.** The gradient sums $$T$$ log-probability terms, each multiplied by the same noisy return. More terms, more noise.
2. **Huge action space.** Each token is one draw from about 50,000 to 150,000 options; the reply as a whole is one draw from an astronomically large set.
3. **Small batches.** Each sample costs a full generation. A practical batch is tens to hundreds of replies per step, not millions.

## 6.7 Baselines: less noise at no cost in bias

Williams' fix is to subtract a number $$b$$ from the reward before multiplying by the score. It sounds like cheating: surely subtracting something changes the gradient? It does not, on average, and the proof is three lines.

### The proof

We want to show that the extra term $$b\, \nabla_\theta \log \pi_\theta(a \mid s)$$ averages to zero over actions, as long as $$b$$ does not depend on the action $$a$$:

$$\mathbb{E}_{a \sim \pi_\theta}\big[b\, \nabla_\theta \log \pi_\theta(a \mid s)\big] = b \sum_{a} \pi_\theta(a \mid s)\, \frac{\nabla_\theta \pi_\theta(a \mid s)}{\pi_\theta(a \mid s)} = b\, \nabla_\theta \sum_a \pi_\theta(a \mid s) = b\, \nabla_\theta 1 = 0$$

where:

- $$b$$ is any number that does not depend on which action is sampled (it may depend on the state $$s$$, on the step, on past data),
- the first equality writes the expectation as a sum and uses $$\nabla \log \pi = \nabla \pi / \pi$$,
- the $$\pi$$ in front cancels the $$\pi$$ in the denominator,
- the sum of the probabilities of all actions is always 1, a constant, so its gradient is 0.

So subtracting $$b$$ leaves the average gradient unchanged: $$\mathbb{E}[(G - b)\nabla \log \pi] = \mathbb{E}[G \nabla \log \pi]$$. The bandit run confirms the key step numerically: the average score vector over 200,000 samples is $$(-0.0007, 0.0014, -0.0012, 0.0005)$$, zero up to Monte Carlo error, and the mean estimate with $$b = 6.5$$ matches the exact gradient as well as the one without.

> [!WARNING] The baseline must not depend on the action
> If $$b$$ uses the sampled action (for example, the reward of the same reply, or a value estimate that has seen the chosen token), the proof breaks and the gradient becomes biased. "The average reward of the *other* replies to the same prompt" is fine; "the average including this reply" is very slightly biased, which is why the leave-one-out version below leaves it out.

### Why it lowers the variance

The average is unchanged, but the *spread* is not. Without a baseline, a reply with reward 7.3 and a reply with reward 5.5 both push their own probability up, by amounts that differ only by about 30%. With $$b = 6.5$$ the first is pushed up by $$+0.8$$ and the second *down* by $$-1.0$$. The estimate now carries the information that matters (better or worse than usual?) and drops the part that does not (all ratings are around 6.5).

How much does the choice of $$b$$ matter? `ch6_bandit.py` sweeps constant baselines from 0 to 12:

```text
== 3. variance of the estimate as a function of a constant baseline b (theta = 0)
  b =   0.0: total variance   33.028,  mean estimate [-0.378 -0.116  0.116  0.378]
  b =   3.0: total variance   10.542,  mean estimate [-0.376 -0.12   0.119  0.377]
  b =   5.0: total variance    3.050,  mean estimate [-0.375 -0.123  0.122  0.376]
  b =   6.0: total variance    1.555,  mean estimate [-0.374 -0.124  0.123  0.375]
  b =   6.5: total variance    1.369,  mean estimate [-0.374 -0.125  0.124  0.375]
  b =   7.0: total variance    1.559,  mean estimate [-0.373 -0.126  0.124  0.375]
  b =   8.0: total variance    3.063,  mean estimate [-0.373 -0.127  0.125  0.374]
  b =  10.0: total variance   10.572,  mean estimate [-0.371 -0.13   0.128  0.373]
  b =  12.0: total variance   24.081,  mean estimate [-0.37  -0.132  0.13   0.372]
variance-minimising constant b* = E[r |score|^2] / E[|score|^2] = 6.497, variance 1.369
```

<figure class="fig"><svg viewBox="0 0 780 330" role="img" aria-label="A U-shaped curve: the variance of the REINFORCE gradient estimate as a function of the constant baseline, smallest near the average reward."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Same average gradient, very different noise: variance against the baseline value</text><line class="grid" x1="90" y1="276.0" x2="610" y2="276.0"/><text class="t-tick" x="82.0" y="280.0" text-anchor="end">0</text><line class="grid" x1="90" y1="244.6" x2="610" y2="244.6"/><text class="t-tick" x="82.0" y="248.6" text-anchor="end">5</text><line class="grid" x1="90" y1="213.1" x2="610" y2="213.1"/><text class="t-tick" x="82.0" y="217.1" text-anchor="end">10</text><line class="grid" x1="90" y1="181.7" x2="610" y2="181.7"/><text class="t-tick" x="82.0" y="185.7" text-anchor="end">15</text><line class="grid" x1="90" y1="150.3" x2="610" y2="150.3"/><text class="t-tick" x="82.0" y="154.3" text-anchor="end">20</text><line class="grid" x1="90" y1="118.9" x2="610" y2="118.9"/><text class="t-tick" x="82.0" y="122.9" text-anchor="end">25</text><line class="grid" x1="90" y1="87.4" x2="610" y2="87.4"/><text class="t-tick" x="82.0" y="91.4" text-anchor="end">30</text><line class="grid" x1="90" y1="56.0" x2="610" y2="56.0"/><text class="t-tick" x="82.0" y="60.0" text-anchor="end">35</text><text class="t-tick" x="90.0" y="294.0" text-anchor="middle">0</text><text class="t-tick" x="176.7" y="294.0" text-anchor="middle">2</text><text class="t-tick" x="263.3" y="294.0" text-anchor="middle">4</text><text class="t-tick" x="350.0" y="294.0" text-anchor="middle">6</text><text class="t-tick" x="436.7" y="294.0" text-anchor="middle">8</text><text class="t-tick" x="523.3" y="294.0" text-anchor="middle">10</text><text class="t-tick" x="610.0" y="294.0" text-anchor="middle">12</text><line class="axis" x1="90" y1="276" x2="610" y2="276"/><polyline class="l1" points="90.0,68.4 111.7,97.8 133.3,124.9 155.0,149.7 176.7,172.1 198.3,192.1 220.0,209.7 241.7,225.0 263.3,238.0 285.0,248.6 306.7,256.8 328.3,262.7 350.0,266.2 371.7,267.4 393.3,266.2 415.0,262.7 436.7,256.7 458.3,248.5 480.0,237.9 501.7,224.9 523.3,209.5 545.0,191.9 566.7,171.8 588.3,149.4 610.0,124.6"/><g class="mark"><title>total variance: 0, 33.0284</title><circle class="s1 ring" cx="90.0" cy="68.4" r="4"/></g><g class="mark"><title>total variance: 0.5, 28.3431</title><circle class="s1 ring" cx="111.7" cy="97.8" r="4"/></g><g class="mark"><title>total variance: 1, 24.0328</title><circle class="s1 ring" cx="133.3" cy="124.9" r="4"/></g><g class="mark"><title>total variance: 1.5, 20.0975</title><circle class="s1 ring" cx="155.0" cy="149.7" r="4"/></g><g class="mark"><title>total variance: 2, 16.5372</title><circle class="s1 ring" cx="176.7" cy="172.1" r="4"/></g><g class="mark"><title>total variance: 2.5, 13.3519</title><circle class="s1 ring" cx="198.3" cy="192.1" r="4"/></g><g class="mark"><title>total variance: 3, 10.5416</title><circle class="s1 ring" cx="220.0" cy="209.7" r="4"/></g><g class="mark"><title>total variance: 3.5, 8.10624</title><circle class="s1 ring" cx="241.7" cy="225.0" r="4"/></g><g class="mark"><title>total variance: 4, 6.04592</title><circle class="s1 ring" cx="263.3" cy="238.0" r="4"/></g><g class="mark"><title>total variance: 4.5, 4.3606</title><circle class="s1 ring" cx="285.0" cy="248.6" r="4"/></g><g class="mark"><title>total variance: 5, 3.05028</title><circle class="s1 ring" cx="306.7" cy="256.8" r="4"/></g><g class="mark"><title>total variance: 5.5, 2.11495</title><circle class="s1 ring" cx="328.3" cy="262.7" r="4"/></g><g class="mark"><title>total variance: 6, 1.55462</title><circle class="s1 ring" cx="350.0" cy="266.2" r="4"/></g><g class="mark"><title>total variance: 6.5, 1.36929</title><circle class="s1 ring" cx="371.7" cy="267.4" r="4"/></g><g class="mark"><title>total variance: 7, 1.55896</title><circle class="s1 ring" cx="393.3" cy="266.2" r="4"/></g><g class="mark"><title>total variance: 7.5, 2.12363</title><circle class="s1 ring" cx="415.0" cy="262.7" r="4"/></g><g class="mark"><title>total variance: 8, 3.06329</title><circle class="s1 ring" cx="436.7" cy="256.7" r="4"/></g><g class="mark"><title>total variance: 8.5, 4.37795</title><circle class="s1 ring" cx="458.3" cy="248.5" r="4"/></g><g class="mark"><title>total variance: 9, 6.06761</title><circle class="s1 ring" cx="480.0" cy="237.9" r="4"/></g><g class="mark"><title>total variance: 9.5, 8.13227</title><circle class="s1 ring" cx="501.7" cy="224.9" r="4"/></g><g class="mark"><title>total variance: 10, 10.5719</title><circle class="s1 ring" cx="523.3" cy="209.5" r="4"/></g><g class="mark"><title>total variance: 10.5, 13.3866</title><circle class="s1 ring" cx="545.0" cy="191.9" r="4"/></g><g class="mark"><title>total variance: 11, 16.5762</title><circle class="s1 ring" cx="566.7" cy="171.8" r="4"/></g><g class="mark"><title>total variance: 11.5, 20.1409</title><circle class="s1 ring" cx="588.3" cy="149.4" r="4"/></g><g class="mark"><title>total variance: 12, 24.0805</title><circle class="s1 ring" cx="610.0" cy="124.6" r="4"/></g><text class="t-muted" x="350.0" y="314.0" text-anchor="middle">constant baseline b subtracted from every reward</text><text class="t-muted" x="82.0" y="40.0" text-anchor="start">variance of the gradient estimate</text><circle class="s2" cx="371.5" cy="267.4" r="6"/><text class="t-val" x="371.5" y="227.4" text-anchor="middle">best b = 6.50: variance 1.37</text><circle class="s3" cx="90.0" cy="68.4" r="6"/><text class="t-val" x="102.0" y="72.4" text-anchor="start">no baseline: 33.0</text><text class="t-muted" x="630.0" y="110.0" text-anchor="start">Every point is</text><text class="t-muted" x="630.0" y="127.0" text-anchor="start">measured on 200,000</text><text class="t-muted" x="630.0" y="144.0" text-anchor="start">samples at the</text><text class="t-muted" x="630.0" y="161.0" text-anchor="start">starting policy.</text><text class="t-muted" x="630.0" y="178.0" text-anchor="start"></text><text class="t-muted" x="630.0" y="195.0" text-anchor="start">The mean gradient</text><text class="t-muted" x="630.0" y="212.0" text-anchor="start">is the same at every b</text><text class="t-muted" x="630.0" y="229.0" text-anchor="start">(it is unbiased);</text><text class="t-muted" x="630.0" y="246.0" text-anchor="start">only the noise moves.</text></svg><figcaption>Variance of the gradient estimate as a function of the constant baseline, at the uniform starting policy. The mean estimate (right-hand column above) is the same at every b; only the noise changes, by a factor of 24 between b = 0 and the best b.</figcaption></figure>

Every row has the same mean estimate (up to sampling noise); the variance falls from 33.0 to 1.37 and rises again on the other side. The best constant has a closed form, which you get by writing the variance as a quadratic in $$b$$ and setting its derivative to zero:

$$b^{*} = \frac{\mathbb{E}\big[r\, \lVert \nabla_\theta \log \pi_\theta(a) \rVert^2\big]}{\mathbb{E}\big[\lVert \nabla_\theta \log \pi_\theta(a) \rVert^2\big]}$$

where:

- $$r$$ is the reward of the sampled action,
- $$\lVert \nabla_\theta \log \pi_\theta(a) \rVert^2$$ is the squared length of the score vector of that action,
- both expectations are over actions sampled from the policy (and rewards).

It is a weighted average of the rewards, weighted by how big each action's score vector is. Here every action has the same score length at the uniform policy, so $$b^*$$ is just the average reward: the script measures 6.497, against the exact $$J = 6.5$$. In practice nobody computes $$b^*$$; the expected return $$V(s)$$ is close to it and much easier to estimate.

At a policy that already prefers reply 4 ($$\theta = (0, 0, 0, 2)$$, so $$\pi = (0.096, 0.096, 0.096, 0.711)$$), the ratio is still 15: total variance 19.5 without a baseline and 1.29 with $$b = V = 7.42$$.

### Training with and without a baseline

Now train. Each run starts from the uniform policy, takes 400 steps of one sample each with learning rate 0.05, and is repeated with 200 different random seeds. Three versions differ only in the baseline: none; the running average of all ratings seen so far (Williams' "reinforcement comparison", his equation 10); and the exact value $$V = \sum_k \pi_k \mu_k$$ of the current policy, which a real learner would not know.

```text
== 4. training: 200 seeds x 400 steps, one sample per step, learning rate 0.05
  baseline none    : mean J at step 100/200/400 = 7.352 / 7.512 / 7.655;  spread (10th-90th pct) at 400: 6.984 to 7.983;  seeds with P(best) > 0.5: 72%;  seeds with P(best) < 0.1: 24%
      most likely reply at the end, count over seeds: {2: 7, 3: 48, 4: 145}
  baseline running : mean J at step 100/200/400 = 7.637 / 7.873 / 7.951;  spread (10th-90th pct) at 400: 7.935 to 7.964;  seeds with P(best) > 0.5: 100%;  seeds with P(best) < 0.1: 0%
      most likely reply at the end, count over seeds: {4: 200}
  baseline value   : mean J at step 100/200/400 = 7.648 / 7.874 / 7.951;  spread (10th-90th pct) at 400: 7.933 to 7.965;  seeds with P(best) > 0.5: 100%;  seeds with P(best) < 0.1: 0%
      most likely reply at the end, count over seeds: {4: 200}
```

<figure class="fig"><svg viewBox="0 0 780 410" role="img" aria-label="Learning curves of expected rating for REINFORCE with no baseline, a running-average baseline and the exact value baseline."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">REINFORCE on the bandit, 200 seeds: mean and 10th to 90th percentile band</text><polygon class="s1" fill-opacity="0.16" points="80.0,281.5 91.8,231.8 103.5,204.6 115.2,181.7 127.0,159.3 138.8,143.3 150.5,129.8 162.2,117.4 174.0,111.2 185.8,102.3 197.5,96.4 209.2,92.3 221.0,89.0 232.8,86.8 244.5,85.0 256.2,81.9 268.0,80.7 279.8,79.5 291.5,78.4 303.2,76.5 315.0,76.0 326.8,75.2 338.5,74.6 350.2,74.2 362.0,73.3 373.8,72.8 385.5,72.3 397.2,71.8 409.0,71.5 420.8,71.1 432.5,70.9 444.2,70.6 456.0,70.2 467.8,69.9 479.5,69.6 491.2,69.4 503.0,69.2 514.8,69.1 526.5,68.8 538.2,68.5 550.0,68.5 550.0,72.7 538.2,72.8 526.5,73.3 514.8,73.4 503.0,73.7 491.2,74.3 479.5,74.8 467.8,75.0 456.0,75.7 444.2,76.6 432.5,77.2 420.8,77.9 409.0,78.9 397.2,79.6 385.5,80.3 373.8,80.8 362.0,81.9 350.2,83.1 338.5,83.9 326.8,85.5 315.0,88.0 303.2,90.0 291.5,92.1 279.8,94.4 268.0,97.1 256.2,102.3 244.5,107.1 232.8,110.3 221.0,118.2 209.2,129.2 197.5,140.3 185.8,150.1 174.0,166.1 162.2,178.3 150.5,196.5 138.8,210.7 127.0,226.1 115.2,241.7 103.5,259.2 91.8,280.2 80.0,281.5"/><polygon class="s2" fill-opacity="0.16" points="80.0,281.5 91.8,196.8 103.5,149.0 115.2,116.8 127.0,98.5 138.8,92.1 150.5,87.9 162.2,85.4 174.0,80.2 185.8,78.9 197.5,76.5 209.2,75.3 221.0,74.2 232.8,74.3 244.5,72.5 256.2,72.0 268.0,71.3 279.8,70.4 291.5,70.1 303.2,69.3 315.0,68.7 326.8,68.6 338.5,68.2 350.2,68.2 362.0,68.2 373.8,67.8 385.5,67.5 397.2,67.4 409.0,67.1 420.8,67.0 432.5,67.0 444.2,66.8 456.0,66.8 467.8,66.7 479.5,66.5 491.2,66.4 503.0,66.3 514.8,66.1 526.5,66.0 538.2,65.9 550.0,65.8 550.0,211.1 538.2,211.3 526.5,211.3 514.8,211.8 503.0,211.9 491.2,212.4 479.5,212.4 467.8,212.1 456.0,213.0 444.2,213.2 432.5,213.8 420.8,213.8 409.0,215.5 397.2,215.8 385.5,217.8 373.8,217.9 362.0,217.0 350.2,218.7 338.5,221.4 326.8,217.8 315.0,227.8 303.2,236.0 291.5,231.8 279.8,232.7 268.0,234.8 256.2,237.0 244.5,247.4 232.8,238.6 221.0,248.7 209.2,248.6 197.5,263.3 185.8,273.1 174.0,269.5 162.2,274.3 150.5,289.6 138.8,299.5 127.0,316.8 115.2,308.7 103.5,325.9 91.8,307.4 80.0,281.5"/><line class="grid" x1="80" y1="281.5" x2="550" y2="281.5"/><text class="t-tick" x="72.0" y="285.5" text-anchor="end">6.5</text><line class="grid" x1="80" y1="208.7" x2="550" y2="208.7"/><text class="t-tick" x="72.0" y="212.7" text-anchor="end">7.0</text><line class="grid" x1="80" y1="136.0" x2="550" y2="136.0"/><text class="t-tick" x="72.0" y="140.0" text-anchor="end">7.5</text><line class="grid" x1="80" y1="63.3" x2="550" y2="63.3"/><text class="t-tick" x="72.0" y="67.3" text-anchor="end">8.0</text><text class="t-tick" x="80.0" y="314.0" text-anchor="middle">0</text><text class="t-tick" x="197.5" y="314.0" text-anchor="middle">100</text><text class="t-tick" x="315.0" y="314.0" text-anchor="middle">200</text><text class="t-tick" x="432.5" y="314.0" text-anchor="middle">300</text><text class="t-tick" x="550.0" y="314.0" text-anchor="middle">400</text><line class="axis" x1="80" y1="296" x2="550" y2="296"/><polyline class="l2" points="80.0,281.5 91.8,251.9 103.5,233.8 115.2,214.7 127.0,203.0 138.8,191.5 150.5,180.7 162.2,173.1 174.0,165.7 185.8,161.4 197.5,157.6 209.2,153.1 221.0,148.5 232.8,146.7 244.5,146.7 256.2,145.2 268.0,142.7 279.8,140.9 291.5,138.4 303.2,136.7 315.0,134.2 326.8,132.3 338.5,132.3 350.2,131.0 362.0,129.4 373.8,128.2 385.5,126.4 397.2,124.3 409.0,124.0 420.8,122.6 432.5,122.3 444.2,121.3 456.0,118.7 467.8,117.8 479.5,116.7 491.2,116.9 503.0,115.6 514.8,114.7 526.5,114.1 538.2,113.7 550.0,113.4"/><polyline class="l1" points="80.0,281.5 91.8,258.1 103.5,234.6 115.2,213.4 127.0,194.1 138.8,175.3 150.5,159.4 162.2,145.4 174.0,134.2 185.8,124.3 197.5,116.1 209.2,108.6 221.0,102.8 232.8,98.4 244.5,94.8 256.2,91.4 268.0,88.7 279.8,86.5 291.5,84.5 303.2,83.1 315.0,81.8 326.8,80.5 338.5,79.3 350.2,78.4 362.0,77.5 373.8,76.7 385.5,76.0 397.2,75.4 409.0,74.9 420.8,74.2 432.5,73.8 444.2,73.3 456.0,72.9 467.8,72.5 479.5,72.2 491.2,71.9 503.0,71.6 514.8,71.2 526.5,71.0 538.2,70.7 550.0,70.4"/><polyline class="l3" points="80.0,281.5 91.8,259.0 103.5,235.0 115.2,213.9 127.0,193.7 138.8,174.4 150.5,158.2 162.2,143.9 174.0,132.6 185.8,122.6 197.5,114.4 209.2,107.3 221.0,101.7 232.8,97.3 244.5,93.8 256.2,90.5 268.0,88.0 279.8,86.0 291.5,84.3 303.2,82.9 315.0,81.7 326.8,80.4 338.5,79.3 350.2,78.4 362.0,77.6 373.8,76.8 385.5,76.1 397.2,75.5 409.0,75.0 420.8,74.3 432.5,73.9 444.2,73.4 456.0,73.0 467.8,72.5 479.5,72.2 491.2,71.9 503.0,71.6 514.8,71.3 526.5,71.0 538.2,70.7 550.0,70.5"/><rect class="s1" x="562.0" y="65.4" width="10" height="10" rx="2"/><text class="t-tick" x="577.0" y="74.4" text-anchor="start">running-average baseline</text><rect class="s3" x="562.0" y="81.4" width="10" height="10" rx="2"/><text class="t-tick" x="577.0" y="90.4" text-anchor="start">exact value baseline</text><rect class="s2" x="562.0" y="108.4" width="10" height="10" rx="2"/><text class="t-tick" x="577.0" y="117.4" text-anchor="start">no baseline</text><text class="t-muted" x="315.0" y="334.0" text-anchor="middle">update step (one sampled reply per step)</text><text class="t-muted" x="72.0" y="40.0" text-anchor="start">expected rating of the policy (best possible 8.0)</text><text class="t-muted" x="20.0" y="368.0" text-anchor="start">At step 400, without a baseline the most likely reply was reply 4 (the best) in 145 seeds, reply 3 in 48 and reply 2 in 7.</text><text class="t-muted" x="20.0" y="385.0" text-anchor="start">With either baseline, all 200 seeds ended on reply 4. The two baselined bands are so narrow they almost vanish.</text></svg><figcaption>Expected rating of the policy during training, averaged over 200 seeds, with the band from the 10th to the 90th percentile of seeds. Without a baseline (orange) the band is wide and the average climbs slowly, because about a quarter of the runs have locked onto a worse reply. Both baselines give almost identical, narrow curves.</figcaption></figure>

Without a baseline, 55 of the 200 runs (48 on reply 3 and 7 on reply 2) ended with the policy *preferring a worse reply*, and 24% of runs gave the best reply less than 10% probability. Those runs did not slowly fix themselves: once a reply is sampled almost every time, the other replies are almost never tried, so the evidence that would correct the mistake never arrives. With a baseline, every one of the 200 runs ended on the best reply, and the 10th-to-90th percentile band is only 0.03 wide. A simple running average did as well as the exact value.

This is a toy, but the shape of the result carries over to language models. Without a baseline, policy gradients amplify whatever happens to be sampled early, and the model can lock into a habit (a phrase, a format, a length) before it has seen enough evidence.

### Baselines for language models

For text, three kinds of baseline are common.

1. **A running average** of past rewards (Williams' version). Cheap, but the same number is used for every prompt, although some prompts are simply easier than others.
2. **A learned value function** $$V_\phi(s)$$, a second network (usually a "value head" on a copy of the language model) trained to predict the return from each prefix. This is the critic of actor-critic methods and of PPO (Section 6.8 and Chapter 8). It can give a different baseline for every token, but it is a whole extra model to train, and if it is wrong it adds bias once we use it for more than a baseline (Section 6.10).
3. **Other samples for the same prompt.** Generate $$k$$ replies per prompt and use the others' rewards as each one's baseline.

A fourth variant was popular for image captioning: Rennie et al. (2016) used the reward of the model's own *greedy* caption as the baseline for its sampled captions ("self-critical sequence training"), so a sample is reinforced only if it beats what the model would say by default.

The third idea is old (Kool, van Hoof and Welling, 2019, for routing problems), and in 2024 it became the main critic-free method for language models:

> [!PAPER] Ahmadian et al. (2024) · Section 2.3 · page 6
> [![The definition of the REINFORCE Leave-One-Out estimator in Ahmadian et al.: one over k times the sum over samples i of the reward of y i minus the average reward of the other k minus 1 samples, times the gradient of log pi of y i; highlighted: the rewards for each sample can serve all other samples as a baseline, and akin to a parameter-free value-function](https://ishwarj.com/img/training/ch6-ahmadian-rloo.png6-ahmadian-rloo.png)](/img/training/ch6-ahmadian-rloo.png)
>
> **Context:** after describing plain REINFORCE and the moving-average baseline, the paper introduces the estimator it recommends for RLHF.
>
> **What it says:** with $$k$$ samples per prompt, "the rewards for each sample can serve all other samples as a baseline": each reply's reward is compared with the average of the other $$k-1$$, which is "akin to a parameter-free value-function, but estimated at each training step".
>
> **Why it matters:** no critic network, no value loss, and a baseline tailored to each prompt's difficulty. The paper reports that this REINFORCE Leave-One-Out (RLOO) matches or beats PPO in their RLHF experiments. GRPO (Chapter 2, Section 2.9) uses the same group idea with the group mean and standard deviation.

> [!DEFINITION] RLOO (REINFORCE Leave-One-Out)
> REINFORCE with $$k$$ sampled replies per prompt, where the baseline for each reply is the mean reward of the other $$k-1$$ replies to the same prompt. Because the other replies are sampled independently, the baseline does not depend on the reply's own tokens and the gradient stays unbiased.

Our GPT-2 experiment in Section 6.11 uses exactly this: 16 prompts per step, 4 replies per prompt, each reply's baseline the average of the other 3.

## 6.8 Value functions and the advantage

The best baseline is "how well do things usually go from here?". That quantity has a name.

> [!DEFINITION] State-value function
> $$V^\pi(s)$$ is the expected return when starting in state $$s$$ and following policy $$\pi$$ from then on. For a language model: the expected final score of a reply that begins with this prompt and this prefix, averaged over all the ways the model might continue it.

> [!DEFINITION] Action-value function
> $$Q^\pi(s, a)$$ is the expected return when starting in state $$s$$, taking action $$a$$ first, and following $$\pi$$ afterwards. For a language model: the expected final score if the next token is $$a$$.

> [!DEFINITION] Advantage
> $$A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s)$$: how much better (positive) or worse (negative) action $$a$$ is than the policy's average behaviour in state $$s$$. Its average over the policy's own actions is exactly zero.

$$V^\pi(s_t) = \mathbb{E}_\pi\big[G_t \mid s_t\big], \qquad Q^\pi(s_t, a_t) = \mathbb{E}_\pi\big[G_t \mid s_t, a_t\big], \qquad A^\pi(s_t, a_t) = Q^\pi(s_t, a_t) - V^\pi(s_t)$$

where:

- $$G_t$$ is the return from step $$t$$,
- $$\mathbb{E}_\pi[\cdot \mid s_t]$$ averages over every way the episode could continue from state $$s_t$$ under policy $$\pi$$,
- conditioning on $$a_t$$ as well fixes the first action and averages over the rest.

Putting the advantage into the policy gradient gives its most useful form:

$$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\Big[\sum_{t} A^\pi(s_t, a_t)\, \nabla_\theta \log \pi_\theta(a_t \mid s_t)\Big]$$

which is the policy gradient theorem with the baseline $$V^\pi(s_t)$$ subtracted from $$Q^\pi(s_t, a_t)$$. Each token is pushed up if it was better than the model's average choice at that point, and down if it was worse. The GAE paper lists the family of choices for the weight in front of $$\nabla \log \pi$$:

> [!PAPER] Schulman, Moritz, Levine, Jordan and Abbeel (2015), High-Dimensional Continuous Control Using Generalized Advantage Estimation · Section 2, equation 1 · page 2
> [![Equation 1 of the GAE paper: the policy gradient g equals the expectation of the sum over t of Psi t times grad log pi of a t given s t, where Psi t may be the total reward of the trajectory, the reward following action a t, a baselined version of the previous formula, the state-action value function, the advantage function, or the TD residual; highlighted: baselined version of, advantage function, TD residual](https://ishwarj.com/img/training/ch6-gae-psi.pnging/ch6-gae-psi.png)](/img/training/ch6-gae-psi.png)
>
> **Context:** the preliminaries of the GAE paper, before its own contribution.
>
> **What it says:** all policy gradient estimators have the form $$g = \mathbb{E}[\sum_t \Psi_t \nabla_\theta \log \pi_\theta(a_t \mid s_t)]$$, and $$\Psi_t$$ can be (1) the total reward, (2) the reward-to-go, (3) the reward-to-go minus a baseline, (4) $$Q^\pi$$, (5) the advantage $$A^\pi$$, or (6) the TD residual $$r_t + V^\pi(s_{t+1}) - V^\pi(s_t)$$.
>
> **Why it matters:** this list is a map of this chapter. Sections 6.4 and 6.5 used choices 1 and 2, Section 6.7 choice 3, this section choice 5, and the next two sections choice 6 and a blend of all of them.

### A world where every value is exact

For a language model, $$V$$ and $$Q$$ can only be estimated. To see what they look like, `ch6_gae.py` builds a tiny "token world" where they can be computed exactly.

- A reply is $$T = 6$$ tokens, each either A or B.
- The state is (position $$t$$, number of A's so far $$k$$).
- The reward is 1 if the finished reply contains at least 4 A's, and 0 otherwise. No reward before the end, like a checker that marks an answer right or wrong.
- The policy writes A with probability 0.6 at every step.

From state $$(t, k)$$, the number of A's still to come is binomial, so $$V(t, k) = \Pr[k + \text{Binomial}(6 - t, 0.6) \ge 4]$$, a short sum.

<figure class="fig"><svg viewBox="0 0 780 408" role="img" aria-label="A grid of exact state values for the token world, by position t and number of A tokens k."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">The token world: V(t, k) = chance of ending with at least 4 A's, for P(A) = 0.6</text><text class="t-tick" x="181.0" y="58.0" text-anchor="middle">t = 0</text><text class="t-tick" x="243.0" y="58.0" text-anchor="middle">t = 1</text><text class="t-tick" x="305.0" y="58.0" text-anchor="middle">t = 2</text><text class="t-tick" x="367.0" y="58.0" text-anchor="middle">t = 3</text><text class="t-tick" x="429.0" y="58.0" text-anchor="middle">t = 4</text><text class="t-tick" x="491.0" y="58.0" text-anchor="middle">t = 5</text><text class="t-tick" x="553.0" y="58.0" text-anchor="middle">t = 6</text><text class="t-tick" x="140.0" y="91.0" text-anchor="end">k = 0 A's</text><g class="mark"><title>V(0, 0) = 0.544</title><rect class="cell" x="152" y="68" width="58" height="36" rx="4" fill-opacity="0.52"/></g><text class="t-cell" x="181.0" y="91.0" text-anchor="middle">0.54</text><g class="mark"><title>V(1, 0) = 0.337</title><rect class="cell" x="214" y="68" width="58" height="36" rx="4" fill-opacity="0.35"/></g><text class="t-cell" x="243.0" y="91.0" text-anchor="middle">0.34</text><g class="mark"><title>V(2, 0) = 0.130</title><rect class="cell" x="276" y="68" width="58" height="36" rx="4" fill-opacity="0.18"/></g><text class="t-cell" x="305.0" y="91.0" text-anchor="middle">0.13</text><g class="mark"><title>V(3, 0) = 0.000</title><rect class="cell" x="338" y="68" width="58" height="36" rx="4" fill-opacity="0.08"/></g><text class="t-cell" x="367.0" y="91.0" text-anchor="middle">0.00</text><g class="mark"><title>V(4, 0) = 0.000</title><rect class="cell" x="400" y="68" width="58" height="36" rx="4" fill-opacity="0.08"/></g><text class="t-cell" x="429.0" y="91.0" text-anchor="middle">0.00</text><g class="mark"><title>V(5, 0) = 0.000</title><rect class="cell" x="462" y="68" width="58" height="36" rx="4" fill-opacity="0.08"/></g><text class="t-cell" x="491.0" y="91.0" text-anchor="middle">0.00</text><g class="mark"><title>V(6, 0) = 0.000</title><rect class="cell" x="524" y="68" width="58" height="36" rx="4" fill-opacity="0.08"/></g><text class="t-cell" x="553.0" y="91.0" text-anchor="middle">0.00</text><text class="t-tick" x="140.0" y="131.0" text-anchor="end">k = 1 A's</text><rect class="cell-masked" x="152" y="108" width="58" height="36" rx="4"/><g class="mark"><title>V(1, 1) = 0.683</title><rect class="cell" x="214" y="108" width="58" height="36" rx="4" fill-opacity="0.63"/></g><text class="t-cell on" x="243.0" y="131.0" text-anchor="middle">0.68</text><g class="mark"><title>V(2, 1) = 0.475</title><rect class="cell" x="276" y="108" width="58" height="36" rx="4" fill-opacity="0.46"/></g><text class="t-cell" x="305.0" y="131.0" text-anchor="middle">0.48</text><g class="mark"><title>V(3, 1) = 0.216</title><rect class="cell" x="338" y="108" width="58" height="36" rx="4" fill-opacity="0.25"/></g><text class="t-cell" x="367.0" y="131.0" text-anchor="middle">0.22</text><g class="mark"><title>V(4, 1) = 0.000</title><rect class="cell" x="400" y="108" width="58" height="36" rx="4" fill-opacity="0.08"/></g><text class="t-cell" x="429.0" y="131.0" text-anchor="middle">0.00</text><g class="mark"><title>V(5, 1) = 0.000</title><rect class="cell" x="462" y="108" width="58" height="36" rx="4" fill-opacity="0.08"/></g><text class="t-cell" x="491.0" y="131.0" text-anchor="middle">0.00</text><g class="mark"><title>V(6, 1) = 0.000</title><rect class="cell" x="524" y="108" width="58" height="36" rx="4" fill-opacity="0.08"/></g><text class="t-cell" x="553.0" y="131.0" text-anchor="middle">0.00</text><text class="t-tick" x="140.0" y="171.0" text-anchor="end">k = 2 A's</text><rect class="cell-masked" x="152" y="148" width="58" height="36" rx="4"/><rect class="cell-masked" x="214" y="148" width="58" height="36" rx="4"/><g class="mark"><title>V(2, 2) = 0.821</title><rect class="cell" x="276" y="148" width="58" height="36" rx="4" fill-opacity="0.74"/></g><text class="t-cell on" x="305.0" y="171.0" text-anchor="middle">0.82</text><g class="mark"><title>V(3, 2) = 0.648</title><rect class="cell" x="338" y="148" width="58" height="36" rx="4" fill-opacity="0.60"/></g><text class="t-cell on" x="367.0" y="171.0" text-anchor="middle">0.65</text><g class="mark"><title>V(4, 2) = 0.360</title><rect class="cell" x="400" y="148" width="58" height="36" rx="4" fill-opacity="0.37"/></g><text class="t-cell" x="429.0" y="171.0" text-anchor="middle">0.36</text><g class="mark"><title>V(5, 2) = 0.000</title><rect class="cell" x="462" y="148" width="58" height="36" rx="4" fill-opacity="0.08"/></g><text class="t-cell" x="491.0" y="171.0" text-anchor="middle">0.00</text><g class="mark"><title>V(6, 2) = 0.000</title><rect class="cell" x="524" y="148" width="58" height="36" rx="4" fill-opacity="0.08"/></g><text class="t-cell" x="553.0" y="171.0" text-anchor="middle">0.00</text><text class="t-tick" x="140.0" y="211.0" text-anchor="end">k = 3 A's</text><rect class="cell-masked" x="152" y="188" width="58" height="36" rx="4"/><rect class="cell-masked" x="214" y="188" width="58" height="36" rx="4"/><rect class="cell-masked" x="276" y="188" width="58" height="36" rx="4"/><g class="mark"><title>V(3, 3) = 0.936</title><rect class="cell" x="338" y="188" width="58" height="36" rx="4" fill-opacity="0.83"/></g><text class="t-cell on" x="367.0" y="211.0" text-anchor="middle">0.94</text><g class="mark"><title>V(4, 3) = 0.840</title><rect class="cell" x="400" y="188" width="58" height="36" rx="4" fill-opacity="0.75"/></g><text class="t-cell on" x="429.0" y="211.0" text-anchor="middle">0.84</text><g class="mark"><title>V(5, 3) = 0.600</title><rect class="cell" x="462" y="188" width="58" height="36" rx="4" fill-opacity="0.56"/></g><text class="t-cell on" x="491.0" y="211.0" text-anchor="middle">0.60</text><g class="mark"><title>V(6, 3) = 0.000</title><rect class="cell" x="524" y="188" width="58" height="36" rx="4" fill-opacity="0.08"/></g><text class="t-cell" x="553.0" y="211.0" text-anchor="middle">0.00</text><text class="t-tick" x="140.0" y="251.0" text-anchor="end">k = 4 A's</text><rect class="cell-masked" x="152" y="228" width="58" height="36" rx="4"/><rect class="cell-masked" x="214" y="228" width="58" height="36" rx="4"/><rect class="cell-masked" x="276" y="228" width="58" height="36" rx="4"/><rect class="cell-masked" x="338" y="228" width="58" height="36" rx="4"/><g class="mark"><title>V(4, 4) = 1.000</title><rect class="cell" x="400" y="228" width="58" height="36" rx="4" fill-opacity="0.88"/></g><text class="t-cell on" x="429.0" y="251.0" text-anchor="middle">1.00</text><g class="mark"><title>V(5, 4) = 1.000</title><rect class="cell" x="462" y="228" width="58" height="36" rx="4" fill-opacity="0.88"/></g><text class="t-cell on" x="491.0" y="251.0" text-anchor="middle">1.00</text><g class="mark"><title>V(6, 4) = 1.000</title><rect class="cell" x="524" y="228" width="58" height="36" rx="4" fill-opacity="0.88"/></g><text class="t-cell on" x="553.0" y="251.0" text-anchor="middle">1.00</text><text class="t-tick" x="140.0" y="291.0" text-anchor="end">k = 5 A's</text><rect class="cell-masked" x="152" y="268" width="58" height="36" rx="4"/><rect class="cell-masked" x="214" y="268" width="58" height="36" rx="4"/><rect class="cell-masked" x="276" y="268" width="58" height="36" rx="4"/><rect class="cell-masked" x="338" y="268" width="58" height="36" rx="4"/><rect class="cell-masked" x="400" y="268" width="58" height="36" rx="4"/><g class="mark"><title>V(5, 5) = 1.000</title><rect class="cell" x="462" y="268" width="58" height="36" rx="4" fill-opacity="0.88"/></g><text class="t-cell on" x="491.0" y="291.0" text-anchor="middle">1.00</text><g class="mark"><title>V(6, 5) = 1.000</title><rect class="cell" x="524" y="268" width="58" height="36" rx="4" fill-opacity="0.88"/></g><text class="t-cell on" x="553.0" y="291.0" text-anchor="middle">1.00</text><text class="t-tick" x="140.0" y="331.0" text-anchor="end">k = 6 A's</text><rect class="cell-masked" x="152" y="308" width="58" height="36" rx="4"/><rect class="cell-masked" x="214" y="308" width="58" height="36" rx="4"/><rect class="cell-masked" x="276" y="308" width="58" height="36" rx="4"/><rect class="cell-masked" x="338" y="308" width="58" height="36" rx="4"/><rect class="cell-masked" x="400" y="308" width="58" height="36" rx="4"/><rect class="cell-masked" x="462" y="308" width="58" height="36" rx="4"/><g class="mark"><title>V(6, 6) = 1.000</title><rect class="cell" x="524" y="308" width="58" height="36" rx="4" fill-opacity="0.88"/></g><text class="t-cell on" x="553.0" y="331.0" text-anchor="middle">1.00</text><text class="t-muted" x="20.0" y="370.0" text-anchor="start">Start at the top left (no tokens yet): the chance of success is 0.54. Writing A moves one cell right and one down;</text><text class="t-muted" x="20.0" y="387.0" text-anchor="start">writing B moves one cell right. After three B's (k = 0 at t = 3) the reply can no longer succeed: V = 0.</text></svg><figcaption>Exact state values in the token world. Each cell is the probability of ending with at least four A's from that state. Reading along a row (writing B) the value falls; moving diagonally down (writing A) it rises. Every value in this chapter's GAE experiments is checked against this table.</figcaption></figure>

```text
== 2. exact values for the fixed policy P(A) = 0.6
  V(start) = probability of success = 0.5443
  ...
  at the start: Q(A) = 0.6826, Q(B) = 0.3370, advantage A(A) = +0.1382, A(B) = -0.2074
```

**Worked example.** At the start, $$V(0, 0) = 0.5443$$: this policy succeeds 54% of the time. If the first token is A, we move to state $$(1, 1)$$, whose value is 0.6826, so $$Q(s_0, \text{A}) = 0.6826$$. If it is B, we move to $$(1, 0)$$ with value 0.3370, so $$Q(s_0, \text{B}) = 0.3370$$. The advantages are $$A(s_0, \text{A}) = 0.6826 - 0.5443 = +0.1382$$ and $$A(s_0, \text{B}) = 0.3370 - 0.5443 = -0.2074$$. Check that they average to zero under the policy: $$0.6 \times 0.1382 + 0.4 \times (-0.2074) = 0.0829 - 0.0830 \approx 0$$. They do, as they must: on average the policy is exactly as good as itself.

The advantage is the cleanest possible learning signal. A first-token A is credited with $$+0.14$$, which is precisely how much it improved the chance of success. Compare the REINFORCE weight without a baseline: the whole episode's reward, 0 or 1, for every token, whichever tokens actually helped.

### Actor and critic

In real problems $$V^\pi$$ is unknown, so it is learned.

> [!DEFINITION] Actor-critic
> A method with two learned parts: the **actor** is the policy, updated with the policy gradient; the **critic** is a value function $$V_\phi(s)$$, trained by regression to predict returns, and used to compute the advantages that weight the actor's update.

The idea goes back to Barto, Sutton and Anderson (1983), who balanced a pole on a cart with two "neuronlike" elements: one choosing actions, one predicting reinforcement and criticising the first. In deep RL, Mnih and colleagues' A3C (2016) made "advantage actor-critic" standard, and PPO (Chapter 8) is an actor-critic method. For RLHF the critic is usually another copy of the language model with a scalar head that outputs $$V_\phi(s_t)$$ at every token: as large as the policy itself, which is one reason the 2024 critic-free methods became popular.

Once we have a critic, we can do more with it than subtract it as a baseline. We can use it to *estimate the future*, and that is where TD errors come in.

## 6.9 The temporal-difference error

Suppose we are at token $$t$$, the critic says the state is worth $$V(s_t)$$, we choose a token, receive reward $$r_t$$, and land in $$s_{t+1}$$, which the critic says is worth $$V(s_{t+1})$$. Was that token good?

> [!DEFINITION] TD error (temporal-difference error)
> $$\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$$: the reward just received plus the critic's estimate of what follows, minus the critic's estimate before the step. It measures whether one step turned out better ($$\delta > 0$$) or worse ($$\delta < 0$$) than the critic expected. Also called the TD residual.

$$\delta_t = r_t + \gamma\, V(s_{t+1}) - V(s_t)$$

where:

- $$r_t$$ is the reward received for action $$a_t$$ (for text: 0, except after the last token),
- $$V(s_{t+1})$$ is the critic's value of the next state; after the last token the episode is over and this is 0,
- $$V(s_t)$$ is the critic's value of the current state,
- $$\gamma$$ is the discount (1 for us).

The idea of learning predictions from the difference between successive predictions is Sutton's "temporal-difference learning" (1988). For policy gradients, the GAE paper makes the key observation:

> [!PAPER] Schulman et al. (2015), GAE · Section 3, equation 10 · page 4
> [![A paragraph of the GAE paper: let V be an approximate value function and define delta V t as r t plus gamma V of s t plus 1 minus V of s t, the TD residual; highlighted: delta can be considered as an estimate of the advantage of the action a t; if V is the correct value function it is an unbiased estimator of the advantage, otherwise it yields biased policy gradient estimates](https://ishwarj.com/img/training/ch6-gae-td.pngning/ch6-gae-td.png)](/img/training/ch6-gae-td.png)
>
> **Context:** the start of the section on estimating advantages.
>
> **What it says:** the TD residual "can be considered as an estimate of the advantage of the action $$a_t$$". With the correct value function its expectation is exactly $$A^{\pi,\gamma}(s_t, a_t)$$ (equation 10). "However, this estimator is only $$\gamma$$-just for $$V = V^{\pi,\gamma}$$, otherwise it will yield biased policy gradient estimates."
>
> **Why it matters:** this is the trade at the centre of actor-critic methods. A TD error looks only one step ahead, so it is far less noisy than the full return, but it is only as good as the critic.

For text the result is even stronger than the paper's equation 10. In general, $$\delta_t$$ equals the advantage only *on average* over the environment's random next state. In text the next state is certain, so with a perfect critic $$\delta_t$$ is *exactly* the advantage, with no noise at all. In the token world: at the start, choosing A, $$\delta_0 = 0 + V(1, 1) - V(0, 0) = 0.6826 - 0.5443 = 0.1383$$, which is $$A(s_0, \text{A})$$ to within rounding. Section 6.10 measures this: with the exact critic, the TD error has zero variance and zero bias.

Real critics are not perfect. Here is one episode of the token world, "ABABAA", which succeeds (four A's), with a deliberately rough critic: the exact values plus random noise with standard deviation 0.15.

```text
== 3. one episode: tokens ABABAA, reward at the end = 1; lambda = 0.95, gamma = 1.0
  using a rough value estimate (exact V plus noise of sd 0.15)
  t=0 state (t=0, k=0) token A: r=0  V(s_t)=0.850  V(s_t+1)=0.553  delta=-0.298  GAE=+0.025  exact A=+0.138
  t=1 state (t=1, k=1) token B: r=0  V(s_t)=0.553  V(s_t+1)=0.417  delta=-0.136  GAE=+0.340  exact A=-0.207
  t=2 state (t=2, k=1) token A: r=0  V(s_t)=0.417  V(s_t+1)=0.572  delta=+0.156  GAE=+0.501  exact A=+0.173
  t=3 state (t=3, k=2) token B: r=0  V(s_t)=0.572  V(s_t+1)=0.227  delta=-0.345  GAE=+0.363  exact A=-0.288
  t=4 state (t=4, k=2) token A: r=0  V(s_t)=0.227  V(s_t+1)=0.456  delta=+0.229  GAE=+0.746  exact A=+0.240
  t=5 state (t=5, k=3) token A: r=1  V(s_t)=0.456  V(s_t+1)=0.000  delta=+0.544  GAE=+0.544  exact A=+0.400
  lambda = 1 (Monte Carlo minus V): [0.15, 0.447, 0.583, 0.428, 0.773, 0.544]
  lambda = 0 (one TD error):        [-0.298, -0.136, 0.156, -0.345, 0.229, 0.544]
```

**Worked example, the TD errors.** At $$t = 0$$ the rough critic says the start is worth 0.850 (the truth is 0.544), and after the first A it says 0.553 (truth 0.683). So $$\delta_0 = 0 + 0.553 - 0.850 = -0.298$$: the TD error blames the first A, which was in fact a good token (exact advantage $$+0.138$$), because the critic's two errors happen to point the wrong way. At $$t = 1$$, B: $$\delta_1 = 0 + 0.417 - 0.553 = -0.136$$, correctly negative. At the last step the reward arrives: $$\delta_5 = 1 + 0 - 0.456 = +0.544$$.

Compare the two extreme columns at the bottom. The TD errors ($$\lambda = 0$$) have the right sign for the B's but misjudge the first A, because they trust the critic. The Monte Carlo advantages ($$\lambda = 1$$), "final reward minus $$V(s_t)$$", ignore the critic's view of the future entirely: every token of this successful episode gets a positive weight, including the two B's, whose exact advantages are negative. Neither is right. The column in between is GAE.

<figure class="fig"><svg viewBox="0 0 780 390" role="img" aria-label="A table for one six-token episode: value estimates, rewards, TD errors, GAE advantages and the exact advantages."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">One episode, "ABABAA", with a rough value estimate: TD errors and GAE (lambda 0.95, gamma 1)</text><text class="t-tick" x="172.0" y="56.0" text-anchor="middle">t = 0</text><text class="t-tick" x="276.0" y="56.0" text-anchor="middle">t = 1</text><text class="t-tick" x="380.0" y="56.0" text-anchor="middle">t = 2</text><text class="t-tick" x="484.0" y="56.0" text-anchor="middle">t = 3</text><text class="t-tick" x="588.0" y="56.0" text-anchor="middle">t = 4</text><text class="t-tick" x="692.0" y="56.0" text-anchor="middle">t = 5</text><text class="t-tick t-strong" x="110.0" y="87.0" text-anchor="end">token</text><rect class="box-1" x="128.0" y="66.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="172.0" y="85.5" text-anchor="middle">A</text><rect class="box-1" x="232.0" y="66.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="276.0" y="85.5" text-anchor="middle">B</text><rect class="box-1" x="336.0" y="66.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="380.0" y="85.5" text-anchor="middle">A</text><rect class="box-1" x="440.0" y="66.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="484.0" y="85.5" text-anchor="middle">B</text><rect class="box-1" x="544.0" y="66.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="588.0" y="85.5" text-anchor="middle">A</text><rect class="box-1" x="648.0" y="66.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="692.0" y="85.5" text-anchor="middle">A</text><text class="t-tick t-strong" x="110.0" y="123.0" text-anchor="end">V(s_t)</text><rect class="box" x="128.0" y="102.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="172.0" y="122.0" text-anchor="middle">0.850</text><rect class="box" x="232.0" y="102.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="276.0" y="122.0" text-anchor="middle">0.553</text><rect class="box" x="336.0" y="102.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="380.0" y="122.0" text-anchor="middle">0.417</text><rect class="box" x="440.0" y="102.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="484.0" y="122.0" text-anchor="middle">0.572</text><rect class="box" x="544.0" y="102.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="588.0" y="122.0" text-anchor="middle">0.227</text><rect class="box" x="648.0" y="102.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="692.0" y="122.0" text-anchor="middle">0.456</text><text class="t-tick t-strong" x="110.0" y="159.0" text-anchor="end">V(s_t+1)</text><rect class="box" x="128.0" y="138.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="172.0" y="158.0" text-anchor="middle">0.553</text><rect class="box" x="232.0" y="138.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="276.0" y="158.0" text-anchor="middle">0.417</text><rect class="box" x="336.0" y="138.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="380.0" y="158.0" text-anchor="middle">0.572</text><rect class="box" x="440.0" y="138.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="484.0" y="158.0" text-anchor="middle">0.227</text><rect class="box" x="544.0" y="138.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="588.0" y="158.0" text-anchor="middle">0.456</text><rect class="box" x="648.0" y="138.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="692.0" y="158.0" text-anchor="middle">0.000</text><text class="t-tick t-strong" x="110.0" y="195.0" text-anchor="end">reward r_t</text><rect class="box" x="128.0" y="174.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="172.0" y="194.0" text-anchor="middle">0.000</text><rect class="box" x="232.0" y="174.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="276.0" y="194.0" text-anchor="middle">0.000</text><rect class="box" x="336.0" y="174.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="380.0" y="194.0" text-anchor="middle">0.000</text><rect class="box" x="440.0" y="174.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="484.0" y="194.0" text-anchor="middle">0.000</text><rect class="box" x="544.0" y="174.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="588.0" y="194.0" text-anchor="middle">0.000</text><rect class="box" x="648.0" y="174.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="692.0" y="194.0" text-anchor="middle">1.000</text><text class="t-tick t-strong" x="110.0" y="231.0" text-anchor="end">TD error delta_t</text><rect class="box-3" x="128.0" y="210.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="172.0" y="230.0" text-anchor="middle">-0.298</text><rect class="box-3" x="232.0" y="210.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="276.0" y="230.0" text-anchor="middle">-0.136</text><rect class="box-3" x="336.0" y="210.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="380.0" y="230.0" text-anchor="middle">+0.156</text><rect class="box-3" x="440.0" y="210.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="484.0" y="230.0" text-anchor="middle">-0.345</text><rect class="box-3" x="544.0" y="210.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="588.0" y="230.0" text-anchor="middle">+0.229</text><rect class="box-3" x="648.0" y="210.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="692.0" y="230.0" text-anchor="middle">+0.544</text><text class="t-tick t-strong" x="110.0" y="267.0" text-anchor="end">GAE A_t</text><rect class="box-on" x="128.0" y="246.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="172.0" y="266.0" text-anchor="middle">+0.025</text><rect class="box-on" x="232.0" y="246.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="276.0" y="266.0" text-anchor="middle">+0.340</text><rect class="box-on" x="336.0" y="246.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="380.0" y="266.0" text-anchor="middle">+0.501</text><rect class="box-on" x="440.0" y="246.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="484.0" y="266.0" text-anchor="middle">+0.363</text><rect class="box-on" x="544.0" y="246.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="588.0" y="266.0" text-anchor="middle">+0.746</text><rect class="box-on" x="648.0" y="246.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="692.0" y="266.0" text-anchor="middle">+0.544</text><text class="t-tick t-strong" x="110.0" y="303.0" text-anchor="end">exact A</text><rect class="box" x="128.0" y="282.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="172.0" y="302.0" text-anchor="middle">+0.138</text><rect class="box" x="232.0" y="282.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="276.0" y="302.0" text-anchor="middle">-0.207</text><rect class="box" x="336.0" y="282.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="380.0" y="302.0" text-anchor="middle">+0.173</text><rect class="box" x="440.0" y="282.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="484.0" y="302.0" text-anchor="middle">-0.288</text><rect class="box" x="544.0" y="282.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="588.0" y="302.0" text-anchor="middle">+0.240</text><rect class="box" x="648.0" y="282.0" width="88.0" height="30.0" rx="6"/><text class="t-tick" x="692.0" y="302.0" text-anchor="middle">+0.400</text><text class="t-muted" x="20.0" y="328.0" text-anchor="start">delta_t = r_t + V(s_t+1) - V(s_t): did things go better or worse than the critic expected, one step later?</text><text class="t-muted" x="20.0" y="345.0" text-anchor="start">GAE is computed backwards: A_5 = delta_5, then A_t = delta_t + lambda x A_t+1. With lambda = 0.95 each A_t is close to the</text><text class="t-muted" x="20.0" y="362.0" text-anchor="start">whole remaining sum of deltas (= final reward - V(s_t)), so it is noisy; one episode cannot recover the exact advantages.</text></svg><figcaption>The same episode as a table. The TD error row trusts the critic one step ahead; the GAE row adds up future TD errors with weights 0.95^l. Each row is computed from the two above it; the bottom row is the exact advantage, which a single episode with a rough critic cannot recover.</figcaption></figure>

## 6.10 Generalized advantage estimation: the lambda trade-off

We have two estimators of the advantage at opposite ends:

- **One TD error** $$\delta_t$$: low variance (only one random step), but biased whenever the critic is wrong.
- **The full return minus the baseline** $$G_t - V(s_t)$$: unbiased whatever the critic says (the critic is only a baseline), but noisy, because $$G_t$$ depends on every random choice until the end.

Between them lies a whole family. Look $$k$$ steps ahead with real rewards, then trust the critic:

$$\hat A^{(k)}_t = \sum_{l=0}^{k-1} \gamma^l \delta_{t+l} = -V(s_t) + r_t + \gamma r_{t+1} + \cdots + \gamma^{k-1} r_{t+k-1} + \gamma^k V(s_{t+k})$$

where:

- $$k$$ is the number of real steps used before switching to the critic's estimate,
- the left form is a sum of $$k$$ TD errors; the right form follows because the middle values cancel in pairs (a telescoping sum),
- $$k = 1$$ gives the single TD error, and $$k \to \infty$$ (to the end of the episode) gives $$G_t - V(s_t)$$.

Schulman and colleagues' contribution in 2015 was to average *all* of these with exponentially decaying weights, and to show that the average has a very simple form:

> [!PAPER] Schulman et al. (2015), GAE · Section 3, equations 16 to 18 · page 5
> [![The definition of GAE in the paper: the generalized advantage estimator GAE gamma lambda is defined as the exponentially-weighted average of the k-step estimators, which simplifies to the sum over l of gamma lambda to the power l times delta t plus l; special cases GAE gamma 0 equals delta t and GAE gamma 1 equals the discounted sum of rewards minus V; highlighted: exponentially-weighted average, has high variance due to the sum of terms, and lambda less than 1 introduces bias only when the value function is inaccurate](https://ishwarj.com/img/training/ch6-gae-def.pnging/ch6-gae-def.png)](/img/training/ch6-gae-def.png)
>
> **Context:** the core of the paper.
>
> **What it says:** GAE is "the exponentially-weighted average" of the $$k$$-step estimators with weights $$(1 - \lambda)\lambda^{k-1}$$, which collapses to $$\sum_{l} (\gamma\lambda)^l \delta_{t+l}$$ (equation 16). $$\lambda = 0$$ gives the TD error; $$\lambda = 1$$ gives returns minus $$V$$, which "has high variance due to the sum of terms". And: "$$\lambda < 1$$ introduces bias only when the value function is inaccurate".
>
> **Why it matters:** one knob, $$\lambda$$, slides between trusting the critic and trusting the samples. Almost every PPO implementation for RLHF, including the one in Chapter 8, computes advantages this way.

$$\hat A^{\text{GAE}(\gamma, \lambda)}_t = \sum_{l=0}^{T-1-t} (\gamma \lambda)^l\, \delta_{t+l} = \delta_t + \gamma\lambda\, \hat A^{\text{GAE}}_{t+1}$$

where:

- $$\lambda$$ (lambda), between 0 and 1, sets how fast the weight on future TD errors decays,
- $$\gamma$$ is the discount (1 for language models, so the weights are just $$\lambda^l$$),
- $$\delta_{t+l}$$ is the TD error $$l$$ steps later,
- the right-hand form is the same sum written as a recursion, which is how it is computed: start at the last token and walk backwards.

<figure class="fig"><svg viewBox="0 0 780 452" role="img" aria-label="Bar rows of the weight GAE puts on each future TD error for lambda 0, 0.5, 0.9, 0.95 and 1."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">GAE adds up future TD errors with weights (gamma x lambda)^l  (here gamma = 1)</text><text class="t-note" x="20.0" y="76.0" text-anchor="start">lambda = 0.0</text><g class="mark"><title>weight on delta_t+0: 1.000</title><rect class="s1" x="170" y="52.0" width="38" height="38.0" rx="2"/></g><text class="t-tick" x="189.0" y="49.0" text-anchor="middle">1.00</text><g class="mark"><title>weight on delta_t+1: 0.000</title><rect class="s1" x="216" y="90.0" width="38" height="1.0" rx="2"/></g><text class="t-tick" x="235.0" y="87.0" text-anchor="middle">0.00</text><g class="mark"><title>weight on delta_t+2: 0.000</title><rect class="s1" x="262" y="90.0" width="38" height="1.0" rx="2"/></g><g class="mark"><title>weight on delta_t+3: 0.000</title><rect class="s1" x="308" y="90.0" width="38" height="1.0" rx="2"/></g><g class="mark"><title>weight on delta_t+4: 0.000</title><rect class="s1" x="354" y="90.0" width="38" height="1.0" rx="2"/></g><g class="mark"><title>weight on delta_t+5: 0.000</title><rect class="s1" x="400" y="90.0" width="38" height="1.0" rx="2"/></g><text class="t-tick" x="419.0" y="87.0" text-anchor="middle">0.00</text><g class="mark"><title>weight on delta_t+6: 0.000</title><rect class="s1" x="446" y="90.0" width="38" height="1.0" rx="2"/></g><g class="mark"><title>weight on delta_t+7: 0.000</title><rect class="s1" x="492" y="90.0" width="38" height="1.0" rx="2"/></g><g class="mark"><title>weight on delta_t+8: 0.000</title><rect class="s1" x="538" y="90.0" width="38" height="1.0" rx="2"/></g><g class="mark"><title>weight on delta_t+9: 0.000</title><rect class="s1" x="584" y="90.0" width="38" height="1.0" rx="2"/></g><g class="mark"><title>weight on delta_t+10: 0.000</title><rect class="s1" x="630" y="90.0" width="38" height="1.0" rx="2"/></g><g class="mark"><title>weight on delta_t+11: 0.000</title><rect class="s1" x="676" y="90.0" width="38" height="1.0" rx="2"/></g><text class="t-tick" x="695.0" y="87.0" text-anchor="middle">0.00</text><text class="t-note" x="20.0" y="142.0" text-anchor="start">lambda = 0.5</text><g class="mark"><title>weight on delta_t+0: 1.000</title><rect class="s2" x="170" y="118.0" width="38" height="38.0" rx="2"/></g><text class="t-tick" x="189.0" y="115.0" text-anchor="middle">1.00</text><g class="mark"><title>weight on delta_t+1: 0.500</title><rect class="s2" x="216" y="137.0" width="38" height="19.0" rx="2"/></g><text class="t-tick" x="235.0" y="134.0" text-anchor="middle">0.50</text><g class="mark"><title>weight on delta_t+2: 0.250</title><rect class="s2" x="262" y="146.5" width="38" height="9.5" rx="2"/></g><g class="mark"><title>weight on delta_t+3: 0.125</title><rect class="s2" x="308" y="151.2" width="38" height="4.8" rx="2"/></g><g class="mark"><title>weight on delta_t+4: 0.062</title><rect class="s2" x="354" y="153.6" width="38" height="2.4" rx="2"/></g><g class="mark"><title>weight on delta_t+5: 0.031</title><rect class="s2" x="400" y="154.8" width="38" height="1.2" rx="2"/></g><text class="t-tick" x="419.0" y="151.8" text-anchor="middle">0.03</text><g class="mark"><title>weight on delta_t+6: 0.016</title><rect class="s2" x="446" y="155.4" width="38" height="1.0" rx="2"/></g><g class="mark"><title>weight on delta_t+7: 0.008</title><rect class="s2" x="492" y="155.7" width="38" height="1.0" rx="2"/></g><g class="mark"><title>weight on delta_t+8: 0.004</title><rect class="s2" x="538" y="155.9" width="38" height="1.0" rx="2"/></g><g class="mark"><title>weight on delta_t+9: 0.002</title><rect class="s2" x="584" y="155.9" width="38" height="1.0" rx="2"/></g><g class="mark"><title>weight on delta_t+10: 0.001</title><rect class="s2" x="630" y="156.0" width="38" height="1.0" rx="2"/></g><g class="mark"><title>weight on delta_t+11: 0.000</title><rect class="s2" x="676" y="156.0" width="38" height="1.0" rx="2"/></g><text class="t-tick" x="695.0" y="153.0" text-anchor="middle">0.00</text><text class="t-note" x="20.0" y="208.0" text-anchor="start">lambda = 0.9</text><g class="mark"><title>weight on delta_t+0: 1.000</title><rect class="s3" x="170" y="184.0" width="38" height="38.0" rx="2"/></g><text class="t-tick" x="189.0" y="181.0" text-anchor="middle">1.00</text><g class="mark"><title>weight on delta_t+1: 0.900</title><rect class="s3" x="216" y="187.8" width="38" height="34.2" rx="2"/></g><text class="t-tick" x="235.0" y="184.8" text-anchor="middle">0.90</text><g class="mark"><title>weight on delta_t+2: 0.810</title><rect class="s3" x="262" y="191.2" width="38" height="30.8" rx="2"/></g><g class="mark"><title>weight on delta_t+3: 0.729</title><rect class="s3" x="308" y="194.3" width="38" height="27.7" rx="2"/></g><g class="mark"><title>weight on delta_t+4: 0.656</title><rect class="s3" x="354" y="197.1" width="38" height="24.9" rx="2"/></g><g class="mark"><title>weight on delta_t+5: 0.590</title><rect class="s3" x="400" y="199.6" width="38" height="22.4" rx="2"/></g><text class="t-tick" x="419.0" y="196.6" text-anchor="middle">0.59</text><g class="mark"><title>weight on delta_t+6: 0.531</title><rect class="s3" x="446" y="201.8" width="38" height="20.2" rx="2"/></g><g class="mark"><title>weight on delta_t+7: 0.478</title><rect class="s3" x="492" y="203.8" width="38" height="18.2" rx="2"/></g><g class="mark"><title>weight on delta_t+8: 0.430</title><rect class="s3" x="538" y="205.6" width="38" height="16.4" rx="2"/></g><g class="mark"><title>weight on delta_t+9: 0.387</title><rect class="s3" x="584" y="207.3" width="38" height="14.7" rx="2"/></g><g class="mark"><title>weight on delta_t+10: 0.349</title><rect class="s3" x="630" y="208.8" width="38" height="13.2" rx="2"/></g><g class="mark"><title>weight on delta_t+11: 0.314</title><rect class="s3" x="676" y="210.1" width="38" height="11.9" rx="2"/></g><text class="t-tick" x="695.0" y="207.1" text-anchor="middle">0.31</text><text class="t-note" x="20.0" y="274.0" text-anchor="start">lambda = 0.95</text><g class="mark"><title>weight on delta_t+0: 1.000</title><rect class="s4" x="170" y="250.0" width="38" height="38.0" rx="2"/></g><text class="t-tick" x="189.0" y="247.0" text-anchor="middle">1.00</text><g class="mark"><title>weight on delta_t+1: 0.950</title><rect class="s4" x="216" y="251.9" width="38" height="36.1" rx="2"/></g><text class="t-tick" x="235.0" y="248.9" text-anchor="middle">0.95</text><g class="mark"><title>weight on delta_t+2: 0.902</title><rect class="s4" x="262" y="253.7" width="38" height="34.3" rx="2"/></g><g class="mark"><title>weight on delta_t+3: 0.857</title><rect class="s4" x="308" y="255.4" width="38" height="32.6" rx="2"/></g><g class="mark"><title>weight on delta_t+4: 0.815</title><rect class="s4" x="354" y="257.0" width="38" height="31.0" rx="2"/></g><g class="mark"><title>weight on delta_t+5: 0.774</title><rect class="s4" x="400" y="258.6" width="38" height="29.4" rx="2"/></g><text class="t-tick" x="419.0" y="255.6" text-anchor="middle">0.77</text><g class="mark"><title>weight on delta_t+6: 0.735</title><rect class="s4" x="446" y="260.1" width="38" height="27.9" rx="2"/></g><g class="mark"><title>weight on delta_t+7: 0.698</title><rect class="s4" x="492" y="261.5" width="38" height="26.5" rx="2"/></g><g class="mark"><title>weight on delta_t+8: 0.663</title><rect class="s4" x="538" y="262.8" width="38" height="25.2" rx="2"/></g><g class="mark"><title>weight on delta_t+9: 0.630</title><rect class="s4" x="584" y="264.1" width="38" height="23.9" rx="2"/></g><g class="mark"><title>weight on delta_t+10: 0.599</title><rect class="s4" x="630" y="265.2" width="38" height="22.8" rx="2"/></g><g class="mark"><title>weight on delta_t+11: 0.569</title><rect class="s4" x="676" y="266.4" width="38" height="21.6" rx="2"/></g><text class="t-tick" x="695.0" y="263.4" text-anchor="middle">0.57</text><text class="t-note" x="20.0" y="340.0" text-anchor="start">lambda = 1.0</text><g class="mark"><title>weight on delta_t+0: 1.000</title><rect class="s1" x="170" y="316.0" width="38" height="38.0" rx="2"/></g><text class="t-tick" x="189.0" y="313.0" text-anchor="middle">1.00</text><g class="mark"><title>weight on delta_t+1: 1.000</title><rect class="s1" x="216" y="316.0" width="38" height="38.0" rx="2"/></g><text class="t-tick" x="235.0" y="313.0" text-anchor="middle">1.00</text><g class="mark"><title>weight on delta_t+2: 1.000</title><rect class="s1" x="262" y="316.0" width="38" height="38.0" rx="2"/></g><g class="mark"><title>weight on delta_t+3: 1.000</title><rect class="s1" x="308" y="316.0" width="38" height="38.0" rx="2"/></g><g class="mark"><title>weight on delta_t+4: 1.000</title><rect class="s1" x="354" y="316.0" width="38" height="38.0" rx="2"/></g><g class="mark"><title>weight on delta_t+5: 1.000</title><rect class="s1" x="400" y="316.0" width="38" height="38.0" rx="2"/></g><text class="t-tick" x="419.0" y="313.0" text-anchor="middle">1.00</text><g class="mark"><title>weight on delta_t+6: 1.000</title><rect class="s1" x="446" y="316.0" width="38" height="38.0" rx="2"/></g><g class="mark"><title>weight on delta_t+7: 1.000</title><rect class="s1" x="492" y="316.0" width="38" height="38.0" rx="2"/></g><g class="mark"><title>weight on delta_t+8: 1.000</title><rect class="s1" x="538" y="316.0" width="38" height="38.0" rx="2"/></g><g class="mark"><title>weight on delta_t+9: 1.000</title><rect class="s1" x="584" y="316.0" width="38" height="38.0" rx="2"/></g><g class="mark"><title>weight on delta_t+10: 1.000</title><rect class="s1" x="630" y="316.0" width="38" height="38.0" rx="2"/></g><g class="mark"><title>weight on delta_t+11: 1.000</title><rect class="s1" x="676" y="316.0" width="38" height="38.0" rx="2"/></g><text class="t-tick" x="695.0" y="313.0" text-anchor="middle">1.00</text><text class="t-tick" x="189.0" y="384.0" text-anchor="middle">d(t)</text><text class="t-tick" x="235.0" y="384.0" text-anchor="middle">d(t+1)</text><text class="t-tick" x="281.0" y="384.0" text-anchor="middle">d(t+2)</text><text class="t-tick" x="327.0" y="384.0" text-anchor="middle">d(t+3)</text><text class="t-tick" x="373.0" y="384.0" text-anchor="middle">d(t+4)</text><text class="t-tick" x="419.0" y="384.0" text-anchor="middle">d(t+5)</text><text class="t-tick" x="465.0" y="384.0" text-anchor="middle">d(t+6)</text><text class="t-tick" x="511.0" y="384.0" text-anchor="middle">d(t+7)</text><text class="t-tick" x="557.0" y="384.0" text-anchor="middle">d(t+8)</text><text class="t-tick" x="603.0" y="384.0" text-anchor="middle">d(t+9)</text><text class="t-tick" x="649.0" y="384.0" text-anchor="middle">d(t+10)</text><text class="t-tick" x="695.0" y="384.0" text-anchor="middle">d(t+11)</text><text class="t-muted" x="20.0" y="414.0" text-anchor="start">lambda = 0: only the next TD error (trusts the critic completely: low variance, biased if the critic is wrong).</text><text class="t-muted" x="20.0" y="431.0" text-anchor="start">lambda = 1: all of them, which telescopes to "actual return - V(s_t)" (no trust in the critic beyond a baseline: unbiased, noisy).</text></svg><figcaption>The weight GAE puts on each future TD error. With lambda = 0 only the next one counts; with lambda = 0.9 an error ten steps ahead still gets weight 0.35; with lambda = 1 all count fully and the sum telescopes to the actual return minus V.</figcaption></figure>

**Worked example, the recursion.** Use the TD errors of the episode above, $$\lambda = 0.95$$, $$\gamma = 1$$, and walk backwards:

- $$\hat A_5 = \delta_5 = +0.544$$ (nothing comes after the last token),
- $$\hat A_4 = \delta_4 + 0.95 \hat A_5 = 0.229 + 0.95 \times 0.544 = 0.229 + 0.517 = +0.746$$,
- $$\hat A_3 = -0.345 + 0.95 \times 0.746 = -0.345 + 0.709 = +0.363$$ (the script's unrounded values give the same),
- $$\hat A_2 = 0.156 + 0.95 \times 0.363 = +0.501$$,
- $$\hat A_1 = -0.136 + 0.95 \times 0.501 = +0.340$$,
- $$\hat A_0 = -0.298 + 0.95 \times 0.340 = +0.025$$.

These match the GAE column printed by the script. With $$\lambda = 0.95$$ each estimate is close to the Monte Carlo value (the whole remaining sum of TD errors) and only slightly pulled toward the critic.

The code is the recursion and nothing else (simplified from `ch6_gae.py`, where it runs on 100,000 episodes at once):

```python
nxt = np.concatenate([vals[:, 1:], np.zeros((N, 1))], 1)   # V(s_t+1); 0 after the last token
delta = r + nxt - vals                                      # TD errors, gamma = 1
adv = np.zeros((N, T)); run = np.zeros(N)
for t in reversed(range(T)):                                # walk backwards from the last token
    run = delta[:, t] + lam * run                           # A_t = delta_t + lambda * A_t+1
    adv[:, t] = run
target = adv + vals                                         # the critic's regression target
```

`vals` holds the critic's value of every visited state, one row per episode. The first line shifts it left by one position to get $$V(s_{t+1})$$ and puts a 0 after the last token, because the episode ends there. The second line is the TD error for every token at once. The loop is the backward recursion. The last line gives the critic's training target, the "$$\lambda$$-return" $$\hat A_t + V(s_t)$$: an estimate of the return built in the same way, so the critic is trained to predict what GAE will measure.

### Measuring the trade-off

Theory says: small $$\lambda$$ means low variance and bias from critic errors; large $$\lambda$$ means little bias and high variance. In the token world we can measure both, because the exact advantages are known. `ch6_gae.py` samples 100,000 episodes, computes GAE for eight values of $$\lambda$$ and three critics, and splits the squared error against the exact advantage into its two parts:

$$\text{error} = \underbrace{\big(\mathbb{E}[\hat A \mid s, a] - A(s, a)\big)^2}_{\text{bias}^2} + \underbrace{\mathbb{E}\big[(\hat A - \mathbb{E}[\hat A \mid s, a])^2\big]}_{\text{variance}}$$

where:

- $$\hat A$$ is the GAE estimate for one visited (state, action) pair in one episode,
- $$\mathbb{E}[\hat A \mid s, a]$$ is its average over all the episodes that visited that pair,
- $$A(s, a)$$ is the exact advantage from the table of Section 6.8,
- both terms are averaged over the visited pairs, weighted by how often they are visited.

<figure class="fig"><svg viewBox="0 0 780 356" role="img" aria-label="Three small charts of squared bias, variance and total error of GAE versus lambda, for an exact, a slightly noisy and a noisy value function."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">GAE against the exact advantage (100,000 episodes): squared bias and variance as lambda changes</text><line class="grid" x1="60" y1="260.0" x2="240" y2="260.0"/><text class="t-tick" x="52.0" y="264.0" text-anchor="end">0.00</text><line class="grid" x1="60" y1="204.1" x2="240" y2="204.1"/><text class="t-tick" x="52.0" y="208.1" text-anchor="end">0.05</text><line class="grid" x1="60" y1="148.2" x2="240" y2="148.2"/><text class="t-tick" x="52.0" y="152.2" text-anchor="end">0.10</text><line class="grid" x1="60" y1="92.4" x2="240" y2="92.4"/><text class="t-tick" x="52.0" y="96.4" text-anchor="end">0.15</text><text class="t-tick" x="60.0" y="278.0" text-anchor="middle">0</text><text class="t-tick" x="150.0" y="278.0" text-anchor="middle">0.5</text><text class="t-tick" x="240.0" y="278.0" text-anchor="middle">1</text><line class="axis" x1="60" y1="260" x2="240" y2="260"/><polyline class="l2" points="60.0,260.0 96.0,260.0 132.0,260.0 168.0,260.0 204.0,260.0 222.0,260.0 231.0,260.0 240.0,260.0"/><g class="mark"><title>bias^2: 0, 0.00</title><circle class="s2 ring" cx="60.0" cy="260.0" r="4"/></g><g class="mark"><title>bias^2: 0.2, 0.00</title><circle class="s2 ring" cx="96.0" cy="260.0" r="4"/></g><g class="mark"><title>bias^2: 0.4, 0.00</title><circle class="s2 ring" cx="132.0" cy="260.0" r="4"/></g><g class="mark"><title>bias^2: 0.6, 0.00</title><circle class="s2 ring" cx="168.0" cy="260.0" r="4"/></g><g class="mark"><title>bias^2: 0.8, 0.00</title><circle class="s2 ring" cx="204.0" cy="260.0" r="4"/></g><g class="mark"><title>bias^2: 0.9, 0.00</title><circle class="s2 ring" cx="222.0" cy="260.0" r="4"/></g><g class="mark"><title>bias^2: 0.95, 0.00</title><circle class="s2 ring" cx="231.0" cy="260.0" r="4"/></g><g class="mark"><title>bias^2: 1, 0.00</title><circle class="s2 ring" cx="240.0" cy="260.0" r="4"/></g><polyline class="l1" points="60.0,260.0 96.0,258.3 132.0,252.4 168.0,238.8 204.0,205.7 222.0,171.9 231.0,147.0 240.0,114.3"/><g class="mark"><title>variance: 0, 0.00</title><circle class="s1 ring" cx="60.0" cy="260.0" r="4"/></g><g class="mark"><title>variance: 0.2, 0.00</title><circle class="s1 ring" cx="96.0" cy="258.3" r="4"/></g><g class="mark"><title>variance: 0.4, 0.01</title><circle class="s1 ring" cx="132.0" cy="252.4" r="4"/></g><g class="mark"><title>variance: 0.6, 0.02</title><circle class="s1 ring" cx="168.0" cy="238.8" r="4"/></g><g class="mark"><title>variance: 0.8, 0.05</title><circle class="s1 ring" cx="204.0" cy="205.7" r="4"/></g><g class="mark"><title>variance: 0.9, 0.08</title><circle class="s1 ring" cx="222.0" cy="171.9" r="4"/></g><g class="mark"><title>variance: 0.95, 0.10</title><circle class="s1 ring" cx="231.0" cy="147.0" r="4"/></g><g class="mark"><title>variance: 1, 0.13</title><circle class="s1 ring" cx="240.0" cy="114.3" r="4"/></g><polyline class="l3" points="60.0,260.0 96.0,258.3 132.0,252.4 168.0,238.8 204.0,205.7 222.0,171.9 231.0,147.0 240.0,114.3"/><g class="mark"><title>total: 0, 0.00</title><circle class="s3 ring" cx="60.0" cy="260.0" r="4"/></g><g class="mark"><title>total: 0.2, 0.00</title><circle class="s3 ring" cx="96.0" cy="258.3" r="4"/></g><g class="mark"><title>total: 0.4, 0.01</title><circle class="s3 ring" cx="132.0" cy="252.4" r="4"/></g><g class="mark"><title>total: 0.6, 0.02</title><circle class="s3 ring" cx="168.0" cy="238.8" r="4"/></g><g class="mark"><title>total: 0.8, 0.05</title><circle class="s3 ring" cx="204.0" cy="205.7" r="4"/></g><g class="mark"><title>total: 0.9, 0.08</title><circle class="s3 ring" cx="222.0" cy="171.9" r="4"/></g><g class="mark"><title>total: 0.95, 0.10</title><circle class="s3 ring" cx="231.0" cy="147.0" r="4"/></g><g class="mark"><title>total: 1, 0.13</title><circle class="s3 ring" cx="240.0" cy="114.3" r="4"/></g><text class="t-muted" x="150.0" y="298.0" text-anchor="middle">lambda</text><text class="t-note" x="150.0" y="58.0" text-anchor="middle">exact V</text><text class="t-tick" x="150.0" y="316.0" text-anchor="middle">lowest total error at lambda 0</text><line class="grid" x1="305" y1="260.0" x2="485" y2="260.0"/><text class="t-tick" x="297.0" y="264.0" text-anchor="end">0.00</text><line class="grid" x1="305" y1="204.1" x2="485" y2="204.1"/><text class="t-tick" x="297.0" y="208.1" text-anchor="end">0.05</text><line class="grid" x1="305" y1="148.2" x2="485" y2="148.2"/><text class="t-tick" x="297.0" y="152.2" text-anchor="end">0.10</text><line class="grid" x1="305" y1="92.4" x2="485" y2="92.4"/><text class="t-tick" x="297.0" y="96.4" text-anchor="end">0.15</text><text class="t-tick" x="305.0" y="278.0" text-anchor="middle">0</text><text class="t-tick" x="395.0" y="278.0" text-anchor="middle">0.5</text><text class="t-tick" x="485.0" y="278.0" text-anchor="middle">1</text><line class="axis" x1="305" y1="260" x2="485" y2="260"/><polyline class="l2" points="305.0,253.5 341.0,254.3 377.0,255.0 413.0,255.5 449.0,255.9 467.0,256.1 476.0,256.2 485.0,256.4"/><g class="mark"><title>bias^2: 0, 0.01</title><circle class="s2 ring" cx="305.0" cy="253.5" r="4"/></g><g class="mark"><title>bias^2: 0.2, 0.01</title><circle class="s2 ring" cx="341.0" cy="254.3" r="4"/></g><g class="mark"><title>bias^2: 0.4, 0.00</title><circle class="s2 ring" cx="377.0" cy="255.0" r="4"/></g><g class="mark"><title>bias^2: 0.6, 0.00</title><circle class="s2 ring" cx="413.0" cy="255.5" r="4"/></g><g class="mark"><title>bias^2: 0.8, 0.00</title><circle class="s2 ring" cx="449.0" cy="255.9" r="4"/></g><g class="mark"><title>bias^2: 0.9, 0.00</title><circle class="s2 ring" cx="467.0" cy="256.1" r="4"/></g><g class="mark"><title>bias^2: 0.95, 0.00</title><circle class="s2 ring" cx="476.0" cy="256.2" r="4"/></g><g class="mark"><title>bias^2: 1, 0.00</title><circle class="s2 ring" cx="485.0" cy="256.4" r="4"/></g><polyline class="l1" points="305.0,260.0 341.0,258.4 377.0,252.7 413.0,239.5 449.0,206.7 467.0,172.8 476.0,147.6 485.0,114.3"/><g class="mark"><title>variance: 0, 0.00</title><circle class="s1 ring" cx="305.0" cy="260.0" r="4"/></g><g class="mark"><title>variance: 0.2, 0.00</title><circle class="s1 ring" cx="341.0" cy="258.4" r="4"/></g><g class="mark"><title>variance: 0.4, 0.01</title><circle class="s1 ring" cx="377.0" cy="252.7" r="4"/></g><g class="mark"><title>variance: 0.6, 0.02</title><circle class="s1 ring" cx="413.0" cy="239.5" r="4"/></g><g class="mark"><title>variance: 0.8, 0.05</title><circle class="s1 ring" cx="449.0" cy="206.7" r="4"/></g><g class="mark"><title>variance: 0.9, 0.08</title><circle class="s1 ring" cx="467.0" cy="172.8" r="4"/></g><g class="mark"><title>variance: 0.95, 0.10</title><circle class="s1 ring" cx="476.0" cy="147.6" r="4"/></g><g class="mark"><title>variance: 1, 0.13</title><circle class="s1 ring" cx="485.0" cy="114.3" r="4"/></g><polyline class="l3" points="305.0,253.5 341.0,252.7 377.0,247.7 413.0,234.9 449.0,202.7 467.0,168.9 476.0,143.8 485.0,110.6"/><g class="mark"><title>total: 0, 0.01</title><circle class="s3 ring" cx="305.0" cy="253.5" r="4"/></g><g class="mark"><title>total: 0.2, 0.01</title><circle class="s3 ring" cx="341.0" cy="252.7" r="4"/></g><g class="mark"><title>total: 0.4, 0.01</title><circle class="s3 ring" cx="377.0" cy="247.7" r="4"/></g><g class="mark"><title>total: 0.6, 0.02</title><circle class="s3 ring" cx="413.0" cy="234.9" r="4"/></g><g class="mark"><title>total: 0.8, 0.05</title><circle class="s3 ring" cx="449.0" cy="202.7" r="4"/></g><g class="mark"><title>total: 0.9, 0.08</title><circle class="s3 ring" cx="467.0" cy="168.9" r="4"/></g><g class="mark"><title>total: 0.95, 0.10</title><circle class="s3 ring" cx="476.0" cy="143.8" r="4"/></g><g class="mark"><title>total: 1, 0.13</title><circle class="s3 ring" cx="485.0" cy="110.6" r="4"/></g><text class="t-muted" x="395.0" y="298.0" text-anchor="middle">lambda</text><text class="t-note" x="395.0" y="58.0" text-anchor="middle">V + noise 0.05</text><text class="t-tick" x="395.0" y="316.0" text-anchor="middle">lowest total error at lambda 0</text><line class="grid" x1="550" y1="260.0" x2="730" y2="260.0"/><text class="t-tick" x="542.0" y="264.0" text-anchor="end">0.00</text><line class="grid" x1="550" y1="204.1" x2="730" y2="204.1"/><text class="t-tick" x="542.0" y="208.1" text-anchor="end">0.05</text><line class="grid" x1="550" y1="148.2" x2="730" y2="148.2"/><text class="t-tick" x="542.0" y="152.2" text-anchor="end">0.10</text><line class="grid" x1="550" y1="92.4" x2="730" y2="92.4"/><text class="t-tick" x="542.0" y="96.4" text-anchor="end">0.15</text><text class="t-tick" x="550.0" y="278.0" text-anchor="middle">0</text><text class="t-tick" x="640.0" y="278.0" text-anchor="middle">0.5</text><text class="t-tick" x="730.0" y="278.0" text-anchor="middle">1</text><line class="axis" x1="550" y1="260" x2="730" y2="260"/><polyline class="l2" points="550.0,201.7 586.0,208.9 622.0,214.8 658.0,219.4 694.0,223.3 712.0,225.3 721.0,226.3 730.0,227.4"/><g class="mark"><title>bias^2: 0, 0.05</title><circle class="s2 ring" cx="550.0" cy="201.7" r="4"/></g><g class="mark"><title>bias^2: 0.2, 0.05</title><circle class="s2 ring" cx="586.0" cy="208.9" r="4"/></g><g class="mark"><title>bias^2: 0.4, 0.04</title><circle class="s2 ring" cx="622.0" cy="214.8" r="4"/></g><g class="mark"><title>bias^2: 0.6, 0.04</title><circle class="s2 ring" cx="658.0" cy="219.4" r="4"/></g><g class="mark"><title>bias^2: 0.8, 0.03</title><circle class="s2 ring" cx="694.0" cy="223.3" r="4"/></g><g class="mark"><title>bias^2: 0.9, 0.03</title><circle class="s2 ring" cx="712.0" cy="225.3" r="4"/></g><g class="mark"><title>bias^2: 0.95, 0.03</title><circle class="s2 ring" cx="721.0" cy="226.3" r="4"/></g><g class="mark"><title>bias^2: 1, 0.03</title><circle class="s2 ring" cx="730.0" cy="227.4" r="4"/></g><polyline class="l1" points="550.0,260.0 586.0,258.4 622.0,252.9 658.0,240.4 694.0,208.5 712.0,174.5 721.0,148.7 730.0,114.3"/><g class="mark"><title>variance: 0, 0.00</title><circle class="s1 ring" cx="550.0" cy="260.0" r="4"/></g><g class="mark"><title>variance: 0.2, 0.00</title><circle class="s1 ring" cx="586.0" cy="258.4" r="4"/></g><g class="mark"><title>variance: 0.4, 0.01</title><circle class="s1 ring" cx="622.0" cy="252.9" r="4"/></g><g class="mark"><title>variance: 0.6, 0.02</title><circle class="s1 ring" cx="658.0" cy="240.4" r="4"/></g><g class="mark"><title>variance: 0.8, 0.05</title><circle class="s1 ring" cx="694.0" cy="208.5" r="4"/></g><g class="mark"><title>variance: 0.9, 0.08</title><circle class="s1 ring" cx="712.0" cy="174.5" r="4"/></g><g class="mark"><title>variance: 0.95, 0.10</title><circle class="s1 ring" cx="721.0" cy="148.7" r="4"/></g><g class="mark"><title>variance: 1, 0.13</title><circle class="s1 ring" cx="730.0" cy="114.3" r="4"/></g><polyline class="l3" points="550.0,201.7 586.0,207.2 622.0,207.7 658.0,199.8 694.0,171.9 712.0,139.7 721.0,115.0 730.0,81.6"/><g class="mark"><title>total: 0, 0.05</title><circle class="s3 ring" cx="550.0" cy="201.7" r="4"/></g><g class="mark"><title>total: 0.2, 0.05</title><circle class="s3 ring" cx="586.0" cy="207.2" r="4"/></g><g class="mark"><title>total: 0.4, 0.05</title><circle class="s3 ring" cx="622.0" cy="207.7" r="4"/></g><g class="mark"><title>total: 0.6, 0.05</title><circle class="s3 ring" cx="658.0" cy="199.8" r="4"/></g><g class="mark"><title>total: 0.8, 0.08</title><circle class="s3 ring" cx="694.0" cy="171.9" r="4"/></g><g class="mark"><title>total: 0.9, 0.11</title><circle class="s3 ring" cx="712.0" cy="139.7" r="4"/></g><g class="mark"><title>total: 0.95, 0.13</title><circle class="s3 ring" cx="721.0" cy="115.0" r="4"/></g><g class="mark"><title>total: 1, 0.16</title><circle class="s3 ring" cx="730.0" cy="81.6" r="4"/></g><text class="t-muted" x="640.0" y="298.0" text-anchor="middle">lambda</text><text class="t-note" x="640.0" y="58.0" text-anchor="middle">V + noise 0.15</text><text class="t-tick" x="640.0" y="316.0" text-anchor="middle">lowest total error at lambda 0.4</text><rect class="s2" x="60" y="332" width="12" height="12" rx="2"/><text class="t-tick" x="78.0" y="342.0" text-anchor="start">squared bias</text><rect class="s1" x="200" y="332" width="12" height="12" rx="2"/><text class="t-tick" x="218.0" y="342.0" text-anchor="start">variance</text><rect class="s3" x="320" y="332" width="12" height="12" rx="2"/><text class="t-tick" x="338.0" y="342.0" text-anchor="start">total error = bias^2 + variance</text></svg><figcaption>Squared bias (orange), variance (blue) and their sum (green) of GAE as lambda goes from 0 to 1, measured against the exact advantages on 100,000 episodes, for three critics. With an exact critic lambda = 0 is perfect. The worse the critic, the more the bias at small lambda grows, and the best lambda moves to the right.</figcaption></figure>

The numbers behind the three panels:

```text
  exact V         lambda 0.00: bias^2 0.0000  variance 0.0000  total error 0.0000
  exact V         lambda 0.95: bias^2 0.0000  variance 0.1011  total error 0.1011
  exact V         lambda 1.00: bias^2 0.0000  variance 0.1304  total error 0.1304
  V + noise 0.05  lambda 0.00: bias^2 0.0058  variance 0.0000  total error 0.0058
  V + noise 0.05  lambda 1.00: bias^2 0.0033  variance 0.1304  total error 0.1336
  V + noise 0.15  lambda 0.00: bias^2 0.0522  variance 0.0000  total error 0.0522
  V + noise 0.15  lambda 0.40: bias^2 0.0405  variance 0.0063  total error 0.0468
  V + noise 0.15  lambda 1.00: bias^2 0.0292  variance 0.1304  total error 0.1596
```

Four things to read off:

1. **Variance grows steadily with $$\lambda$$**, from 0 at $$\lambda = 0$$ to 0.13 at $$\lambda = 1$$, and it is the same for all three critics: it comes from the random future tokens, not from the critic.
2. **With the exact critic, $$\lambda = 0$$ is perfect** (zero bias, zero variance). That is the deterministic-transition effect of Section 6.9.
3. **Bias at $$\lambda = 0$$ grows with the critic's error**: 0.0058 with noise 0.05, 0.052 with noise 0.15. Raising $$\lambda$$ lowers it, but here it does not reach zero even at $$\lambda = 1$$, because the noisy critic is still used as the baseline at each step. (A baseline adds no bias to the *gradient*, as Section 6.7 proved, but it does shift each individual advantage estimate, which is what this measurement compares.)
4. **The best $$\lambda$$ depends on the critic.** For the exact and the slightly noisy critic it is 0; for the noisy critic, 0.4.

In this six-token world the variance at $$\lambda = 1$$ is small, so low $$\lambda$$ wins. With hundreds of steps and a learned critic, the paper found the best values much closer to 1:

> [!PAPER] Schulman et al. (2015), GAE · Figure 2 · page 10
> [![Figure 2 of the GAE paper: learning curves for the cart-pole task with generalized advantage estimation for lambda from 0 to 1 at gamma 0.99, and a grid of performance after 20 iterations over gamma and lambda; highlighted: the fastest policy improvement is obtained by intermediate values of lambda in the range 0.92 to 0.98](https://ishwarj.com/img/training/ch6-gae-fig2.pngng/ch6-gae-fig2.png)](/img/training/ch6-gae-fig2.png)
>
> **Context:** the first experiment, balancing a pole on a cart, averaged over 21 seeds.
>
> **What it says:** "The fastest policy improvement is obtain[ed] by intermediate values of $$\lambda$$ in the range [0.92, 0.98]". $$\lambda = 0$$ learns slowest; $$\lambda = 1$$ and "No VF" (a baseline that depends only on time, not on the state) are also worse than the middle.
>
> **Why it matters:** the best setting is neither extreme. RLHF implementations commonly use $$\lambda = 0.95$$ with $$\gamma = 1$$, a choice that traces back to experiments like this one.

### Training with a critic

Finally, does any of this speed up learning? `ch6_gae.py` trains a tabular policy (one logit per state) in the token world from $$P(\text{A}) = 0.5$$ everywhere, with 16 episodes per update, over 100 seeds. The critic starts at 0 and is trained on the $$\lambda$$-returns as it goes.

```text
== 5. training from P(A) = 0.5 everywhere: 100 seeds, 150 updates of 16 episodes each
  REINFORCE, no baseline            : 0.476 / 0.668 / 0.839 / 0.963;  median updates to reach 0.9: 72;  sd over seeds at 25: 0.016
  REINFORCE, leave-one-out baseline : 0.476 / 0.668 / 0.842 / 0.962;  median updates to reach 0.9: 71;  sd over seeds at 25: 0.009
  GAE lambda 0                      : 0.388 / 0.512 / 0.759 / 0.959;  median updates to reach 0.9: 85;  sd over seeds at 25: 0.008
  GAE lambda 0.5                    : 0.414 / 0.586 / 0.808 / 0.961;  median updates to reach 0.9: 78;  sd over seeds at 25: 0.009
  GAE lambda 0.95                   : 0.468 / 0.660 / 0.839 / 0.962;  median updates to reach 0.9: 72;  sd over seeds at 25: 0.010
  GAE lambda 1                      : 0.476 / 0.667 / 0.842 / 0.962;  median updates to reach 0.9: 71;  sd over seeds at 25: 0.010
```

The columns are the success probability after 10, 25, 50 and 150 updates. The honest result: in this small problem a critic does not help. The critic starts knowing nothing, and with $$\lambda = 0$$ the policy trusts it completely, so early updates follow a wrong critic: it needs a median of 85 updates to reach 90% success, against 71 to 72 for the others. Raising $$\lambda$$ removes that handicap, and at $$\lambda = 0.95$$ to 1 it matches plain REINFORCE. With 0/1 rewards and 16 episodes per update, the gradient noise was small to begin with.

Shift every reward up by 5 (5 for failure, 6 for success), which changes nothing about which replies are better, and the picture changes:

```text
   same, but every episode also gets +5.0 (reward 5 for failure, 6 for success)
  REINFORCE, no baseline            : 0.462 / 0.607 / 0.749 / 0.954;  median to 0.9: 75;  sd at 25: 0.194;  seeds below 0.5 at the end: 0%
  REINFORCE, leave-one-out baseline : 0.476 / 0.668 / 0.842 / 0.962;  median to 0.9: 71;  sd at 25: 0.009;  seeds below 0.5 at the end: 0%
  GAE lambda 0.95                   : 0.467 / 0.651 / 0.833 / 0.961;  median to 0.9: 73;  sd at 25: 0.069;  seeds below 0.5 at the end: 0%
```

<figure class="fig"><svg viewBox="0 0 780 400" role="img" aria-label="Training curves in the token world with rewards shifted by 5, for REINFORCE without a baseline, with a leave-one-out baseline, and with a GAE critic."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Token world with every reward shifted by +5: 100 seeds, mean and 10th to 90th percentile</text><polygon class="s1" fill-opacity="0.16" points="80.0,271.6 83.0,266.7 85.9,262.2 88.9,257.1 91.8,252.3 94.8,248.1 97.7,243.4 100.7,239.5 103.6,234.5 106.6,229.5 109.5,224.6 112.5,220.0 115.4,215.4 118.4,210.2 121.3,205.9 124.3,200.9 127.2,196.4 130.2,192.2 133.2,188.1 136.1,183.8 139.1,180.3 142.0,176.2 145.0,172.6 147.9,168.5 150.9,164.8 153.8,161.2 156.8,157.4 159.7,154.3 162.7,150.8 165.6,147.9 168.6,144.7 171.5,142.1 174.5,139.4 177.4,136.3 180.4,134.3 183.4,131.5 186.3,129.4 189.3,127.4 192.2,125.2 195.2,122.7 198.1,120.7 201.1,119.0 204.0,117.0 207.0,115.2 209.9,113.7 212.9,111.7 215.8,110.5 218.8,108.9 221.7,107.4 224.7,105.9 227.7,104.3 230.6,103.1 233.6,101.7 236.5,100.8 239.5,99.8 242.4,98.9 245.4,98.0 248.3,97.0 251.3,95.9 254.2,94.9 257.2,94.1 260.1,93.5 263.1,92.8 266.0,91.9 269.0,91.0 271.9,90.3 274.9,89.6 277.9,89.4 280.8,88.6 283.8,87.6 286.7,86.9 289.7,86.3 292.6,85.5 295.6,85.0 298.5,84.7 301.5,84.0 304.4,83.5 307.4,82.9 310.3,82.6 313.3,82.1 316.2,81.7 319.2,81.1 322.1,80.7 325.1,80.3 328.1,80.0 331.0,79.6 334.0,79.2 336.9,78.7 339.9,78.3 342.8,77.9 345.8,77.8 348.7,77.4 351.7,76.9 354.6,76.8 357.6,76.3 360.5,76.0 363.5,75.7 366.4,75.4 369.4,75.2 372.3,74.9 375.3,74.7 378.3,74.6 381.2,74.3 384.2,74.1 387.1,73.8 390.1,73.6 393.0,73.4 396.0,73.1 398.9,73.0 401.9,72.8 404.8,72.5 407.8,72.1 410.7,71.8 413.7,71.7 416.6,71.5 419.6,71.4 422.6,71.2 425.5,71.1 428.5,70.7 431.4,70.6 434.4,70.4 437.3,70.2 440.3,70.2 443.2,70.0 446.2,69.8 449.1,69.7 452.1,69.6 455.0,69.5 458.0,69.4 460.9,69.2 463.9,69.0 466.8,69.0 469.8,68.8 472.8,68.6 475.7,68.5 478.7,68.4 481.6,68.2 484.6,68.1 487.5,67.9 490.5,67.9 493.4,67.8 496.4,67.6 499.3,67.6 502.3,67.3 505.2,67.3 508.2,67.2 511.1,67.2 514.1,67.1 517.0,67.0 520.0,67.0 520.0,69.8 517.0,69.8 514.1,69.9 511.1,70.1 508.2,70.2 505.2,70.3 502.3,70.5 499.3,70.6 496.4,70.8 493.4,71.0 490.5,71.1 487.5,71.1 484.6,71.3 481.6,71.5 478.7,71.5 475.7,71.7 472.8,72.0 469.8,72.1 466.8,72.4 463.9,72.4 460.9,72.6 458.0,72.8 455.0,72.9 452.1,73.0 449.1,73.2 446.2,73.4 443.2,73.5 440.3,73.6 437.3,73.7 434.4,73.8 431.4,74.0 428.5,74.3 425.5,74.4 422.6,74.6 419.6,74.9 416.6,75.3 413.7,75.6 410.7,75.7 407.8,76.0 404.8,76.3 401.9,76.6 398.9,76.7 396.0,77.1 393.0,77.1 390.1,77.6 387.1,77.9 384.2,78.3 381.2,78.5 378.3,79.0 375.3,79.0 372.3,79.5 369.4,79.9 366.4,80.0 363.5,80.5 360.5,80.7 357.6,81.0 354.6,81.5 351.7,81.7 348.7,82.0 345.8,82.2 342.8,83.0 339.9,83.4 336.9,83.8 334.0,84.4 331.0,84.8 328.1,85.2 325.1,85.6 322.1,86.1 319.2,86.8 316.2,87.1 313.3,87.5 310.3,88.0 307.4,88.2 304.4,88.7 301.5,89.2 298.5,89.6 295.6,90.5 292.6,90.7 289.7,91.4 286.7,92.0 283.8,92.7 280.8,93.4 277.9,94.0 274.9,94.5 271.9,95.6 269.0,96.3 266.0,97.2 263.1,98.3 260.1,98.5 257.2,99.4 254.2,100.7 251.3,101.8 248.3,102.6 245.4,103.8 242.4,104.9 239.5,106.0 236.5,107.6 233.6,108.6 230.6,109.9 227.7,111.1 224.7,112.8 221.7,114.2 218.8,116.1 215.8,116.9 212.9,119.2 209.9,121.4 207.0,123.2 204.0,124.9 201.1,126.2 198.1,128.3 195.2,130.9 192.2,132.8 189.3,134.9 186.3,137.6 183.4,140.0 180.4,142.3 177.4,145.4 174.5,147.4 171.5,150.5 168.6,153.5 165.6,156.5 162.7,159.8 159.7,163.5 156.8,166.0 153.8,169.0 150.9,172.3 147.9,176.2 145.0,180.0 142.0,183.1 139.1,187.4 136.1,191.4 133.2,195.8 130.2,200.2 127.2,204.5 124.3,208.9 121.3,213.6 118.4,218.0 115.4,222.4 112.5,226.7 109.5,231.6 106.6,236.3 103.6,240.4 100.7,244.9 97.7,249.1 94.8,252.9 91.8,257.1 88.9,261.0 85.9,264.9 83.0,268.7 80.0,271.6"/><polygon class="s2" fill-opacity="0.16" points="80.0,271.6 83.0,248.5 85.9,233.2 88.9,224.0 91.8,216.6 94.8,209.3 97.7,201.3 100.7,195.0 103.6,190.0 106.6,178.7 109.5,170.8 112.5,162.5 115.4,158.4 118.4,162.9 121.3,153.9 124.3,154.0 127.2,147.7 130.2,143.9 133.2,139.5 136.1,132.9 139.1,133.0 142.0,128.3 145.0,127.7 147.9,120.5 150.9,114.1 153.8,112.7 156.8,110.6 159.7,110.3 162.7,105.6 165.6,101.3 168.6,103.4 171.5,100.8 174.5,100.5 177.4,103.5 180.4,99.4 183.4,97.0 186.3,96.9 189.3,94.8 192.2,93.1 195.2,94.8 198.1,92.4 201.1,93.1 204.0,91.8 207.0,91.0 209.9,89.4 212.9,87.9 215.8,87.0 218.8,86.7 221.7,85.0 224.7,84.9 227.7,83.7 230.6,82.6 233.6,83.5 236.5,81.9 239.5,81.2 242.4,81.1 245.4,81.6 248.3,82.0 251.3,81.0 254.2,80.8 257.2,79.7 260.1,79.0 263.1,77.0 266.0,76.5 269.0,77.1 271.9,75.9 274.9,75.9 277.9,75.2 280.8,75.1 283.8,75.3 286.7,75.1 289.7,74.2 292.6,75.2 295.6,72.4 298.5,72.6 301.5,74.0 304.4,72.9 307.4,71.7 310.3,71.2 313.3,70.8 316.2,70.5 319.2,70.5 322.1,70.7 325.1,71.0 328.1,70.9 331.0,70.6 334.0,69.9 336.9,70.1 339.9,69.7 342.8,69.8 345.8,69.4 348.7,69.7 351.7,68.9 354.6,68.3 357.6,67.8 360.5,67.6 363.5,67.9 366.4,68.5 369.4,67.8 372.3,66.7 375.3,66.9 378.3,66.7 381.2,66.9 384.2,66.4 387.1,66.0 390.1,67.2 393.0,66.6 396.0,66.5 398.9,66.2 401.9,65.2 404.8,65.7 407.8,65.2 410.7,65.0 413.7,65.5 416.6,64.7 419.6,64.9 422.6,64.9 425.5,64.5 428.5,64.8 431.4,64.1 434.4,65.2 437.3,64.8 440.3,64.4 443.2,64.2 446.2,64.1 449.1,63.9 452.1,63.3 455.0,62.8 458.0,62.9 460.9,62.5 463.9,62.1 466.8,62.5 469.8,62.7 472.8,62.7 475.7,62.6 478.7,62.3 481.6,61.7 484.6,61.5 487.5,61.6 490.5,61.1 493.4,61.7 496.4,61.6 499.3,61.8 502.3,61.9 505.2,61.6 508.2,61.3 511.1,61.5 514.1,61.6 517.0,61.5 520.0,61.5 520.0,82.9 517.0,83.7 514.1,84.9 511.1,85.4 508.2,87.2 505.2,88.1 502.3,89.0 499.3,90.6 496.4,90.0 493.4,93.9 490.5,91.0 487.5,90.3 484.6,88.4 481.6,90.3 478.7,89.7 475.7,91.5 472.8,93.7 469.8,90.4 466.8,92.8 463.9,91.1 460.9,89.1 458.0,89.4 455.0,89.3 452.1,90.2 449.1,91.0 446.2,90.5 443.2,91.4 440.3,93.8 437.3,94.1 434.4,94.7 431.4,95.5 428.5,92.8 425.5,95.4 422.6,97.4 419.6,99.6 416.6,100.7 413.7,101.2 410.7,99.4 407.8,100.7 404.8,97.3 401.9,102.1 398.9,112.0 396.0,109.7 393.0,117.5 390.1,117.5 387.1,111.2 384.2,110.9 381.2,107.3 378.3,109.7 375.3,111.2 372.3,112.9 369.4,111.8 366.4,114.6 363.5,116.4 360.5,116.4 357.6,119.6 354.6,115.8 351.7,121.0 348.7,120.6 345.8,125.5 342.8,124.7 339.9,122.6 336.9,126.1 334.0,128.5 331.0,131.9 328.1,129.4 325.1,133.2 322.1,127.6 319.2,131.7 316.2,137.8 313.3,136.1 310.3,133.0 307.4,140.1 304.4,147.0 301.5,148.7 298.5,151.4 295.6,157.7 292.6,163.8 289.7,170.3 286.7,167.4 283.8,168.3 280.8,175.3 277.9,174.3 274.9,175.7 271.9,184.8 269.0,179.9 266.0,163.9 263.1,176.3 260.1,178.6 257.2,180.6 254.2,178.7 251.3,172.9 248.3,176.5 245.4,192.5 242.4,195.7 239.5,199.5 236.5,206.3 233.6,206.9 230.6,213.2 227.7,214.3 224.7,213.3 221.7,226.1 218.8,219.1 215.8,216.9 212.9,218.8 209.9,228.4 207.0,225.2 204.0,238.8 201.1,234.5 198.1,232.6 195.2,249.7 192.2,246.6 189.3,246.0 186.3,241.7 183.4,241.7 180.4,238.3 177.4,246.8 174.5,258.0 171.5,274.2 168.6,269.3 165.6,268.7 162.7,269.3 159.7,267.9 156.8,276.9 153.8,282.9 150.9,285.3 147.9,284.6 145.0,284.0 142.0,285.8 139.1,290.5 136.1,297.5 133.2,296.9 130.2,292.7 127.2,293.3 124.3,291.0 121.3,284.0 118.4,291.3 115.4,296.6 112.5,289.4 109.5,290.9 106.6,291.5 103.6,286.9 100.7,286.8 97.7,289.3 94.8,288.0 91.8,293.0 88.9,291.3 85.9,288.3 83.0,283.1 80.0,271.6"/><line class="grid" x1="80" y1="253.1" x2="520" y2="253.1"/><text class="t-tick" x="72.0" y="257.1" text-anchor="end">0.4</text><line class="grid" x1="80" y1="187.4" x2="520" y2="187.4"/><text class="t-tick" x="72.0" y="191.4" text-anchor="end">0.6</text><line class="grid" x1="80" y1="121.7" x2="520" y2="121.7"/><text class="t-tick" x="72.0" y="125.7" text-anchor="end">0.8</text><line class="grid" x1="80" y1="56.0" x2="520" y2="56.0"/><text class="t-tick" x="72.0" y="60.0" text-anchor="end">1.0</text><text class="t-tick" x="80.0" y="304.0" text-anchor="middle">0</text><text class="t-tick" x="227.7" y="304.0" text-anchor="middle">50</text><text class="t-tick" x="375.3" y="304.0" text-anchor="middle">100</text><text class="t-tick" x="523.0" y="304.0" text-anchor="middle">150</text><line class="axis" x1="80" y1="286" x2="520" y2="286"/><polyline class="l2" points="80.0,271.6 83.0,268.4 85.9,263.5 88.9,258.6 91.8,255.6 94.8,249.7 97.7,246.2 100.7,241.6 103.6,238.1 106.6,235.3 109.5,232.8 112.5,228.6 115.4,224.7 118.4,221.3 121.3,218.5 124.3,217.3 127.2,215.2 130.2,209.8 133.2,206.5 136.1,202.9 139.1,199.5 142.0,196.0 145.0,191.7 147.9,189.7 150.9,187.2 153.8,185.1 156.8,182.7 159.7,178.9 162.7,177.1 165.6,176.7 168.6,174.9 171.5,171.8 174.5,170.2 177.4,166.8 180.4,167.2 183.4,165.8 186.3,163.6 189.3,163.5 192.2,162.2 195.2,160.4 198.1,157.1 201.1,154.1 204.0,151.7 207.0,148.6 209.9,146.4 212.9,144.8 215.8,144.1 218.8,143.1 221.7,141.4 224.7,140.4 227.7,138.5 230.6,136.5 233.6,134.2 236.5,133.1 239.5,131.2 242.4,129.3 245.4,127.1 248.3,125.3 251.3,122.5 254.2,121.1 257.2,118.7 260.1,118.2 263.1,117.5 266.0,115.3 269.0,116.9 271.9,115.6 274.9,114.9 277.9,114.0 280.8,111.3 283.8,110.3 286.7,110.8 289.7,110.4 292.6,107.8 295.6,106.1 298.5,104.4 301.5,103.5 304.4,102.7 307.4,101.5 310.3,100.2 313.3,99.7 316.2,98.9 319.2,97.8 322.1,96.1 325.1,95.9 328.1,95.5 331.0,95.7 334.0,95.6 336.9,93.8 339.9,93.1 342.8,92.0 345.8,91.7 348.7,91.4 351.7,90.3 354.6,89.8 357.6,90.2 360.5,89.8 363.5,88.8 366.4,88.6 369.4,87.2 372.3,86.5 375.3,86.6 378.3,86.3 381.2,87.6 384.2,86.9 387.1,86.9 390.1,86.2 393.0,85.7 396.0,85.1 398.9,84.4 401.9,82.7 404.8,82.0 407.8,81.3 410.7,81.2 413.7,80.6 416.6,80.3 419.6,79.5 422.6,79.3 425.5,78.8 428.5,78.3 431.4,78.6 434.4,77.7 437.3,77.0 440.3,76.4 443.2,76.2 446.2,75.7 449.1,75.5 452.1,75.5 455.0,75.5 458.0,75.2 460.9,74.8 463.9,74.6 466.8,74.9 469.8,74.8 472.8,74.9 475.7,74.9 478.7,74.2 481.6,74.0 484.6,73.7 487.5,73.7 490.5,73.4 493.4,73.5 496.4,73.2 499.3,73.1 502.3,72.8 505.2,72.5 508.2,72.3 511.1,71.9 514.1,71.9 517.0,71.3 520.0,71.0"/><polyline class="l1" points="80.0,271.6 83.0,267.6 85.9,263.5 88.9,259.2 91.8,255.0 94.8,250.5 97.7,246.2 100.7,241.8 103.6,237.3 106.6,232.7 109.5,228.1 112.5,223.4 115.4,218.9 118.4,214.3 121.3,209.8 124.3,205.3 127.2,200.8 130.2,196.4 133.2,192.2 136.1,187.9 139.1,183.9 142.0,179.9 145.0,176.2 147.9,172.5 150.9,168.7 153.8,165.2 156.8,161.8 159.7,158.5 162.7,155.3 165.6,152.2 168.6,149.1 171.5,146.3 174.5,143.5 177.4,140.8 180.4,138.3 183.4,135.7 186.3,133.3 189.3,130.9 192.2,128.7 195.2,126.6 198.1,124.5 201.1,122.5 204.0,120.8 207.0,118.9 209.9,117.1 212.9,115.6 215.8,113.8 218.8,112.3 221.7,110.8 224.7,109.3 227.7,107.9 230.6,106.6 233.6,105.3 236.5,104.1 239.5,103.0 242.4,101.9 245.4,100.8 248.3,99.8 251.3,98.8 254.2,97.8 257.2,97.0 260.1,96.0 263.1,95.2 266.0,94.5 269.0,93.6 271.9,92.8 274.9,92.0 277.9,91.4 280.8,90.7 283.8,90.0 286.7,89.3 289.7,88.7 292.6,88.2 295.6,87.7 298.5,87.1 301.5,86.6 304.4,86.1 307.4,85.5 310.3,85.1 313.3,84.6 316.2,84.1 319.2,83.6 322.1,83.2 325.1,82.8 328.1,82.4 331.0,82.0 334.0,81.5 336.9,81.2 339.9,80.8 342.8,80.4 345.8,80.0 348.7,79.6 351.7,79.3 354.6,79.0 357.6,78.6 360.5,78.3 363.5,78.0 366.4,77.7 369.4,77.4 372.3,77.1 375.3,76.9 378.3,76.6 381.2,76.3 384.2,76.0 387.1,75.7 390.1,75.4 393.0,75.2 396.0,75.0 398.9,74.7 401.9,74.5 404.8,74.3 407.8,74.0 410.7,73.8 413.7,73.6 416.6,73.4 419.6,73.1 422.6,73.0 425.5,72.8 428.5,72.6 431.4,72.4 434.4,72.2 437.3,72.1 440.3,71.9 443.2,71.7 446.2,71.6 449.1,71.4 452.1,71.3 455.0,71.1 458.0,71.0 460.9,70.8 463.9,70.7 466.8,70.6 469.8,70.4 472.8,70.3 475.7,70.1 478.7,70.0 481.6,69.9 484.6,69.7 487.5,69.6 490.5,69.5 493.4,69.3 496.4,69.2 499.3,69.1 502.3,69.0 505.2,68.9 508.2,68.7 511.1,68.7 514.1,68.5 517.0,68.4 520.0,68.3"/><polyline class="l3" points="80.0,271.6 83.0,268.6 85.9,264.2 88.9,259.6 91.8,256.0 94.8,251.3 97.7,247.7 100.7,243.6 103.6,239.6 106.6,235.6 109.5,231.2 112.5,226.7 115.4,222.6 118.4,218.3 121.3,214.1 124.3,209.9 127.2,205.8 130.2,201.5 133.2,197.5 136.1,193.5 139.1,189.5 142.0,185.5 145.0,181.7 147.9,177.9 150.9,174.2 153.8,170.6 156.8,167.1 159.7,163.8 162.7,160.5 165.6,157.3 168.6,154.2 171.5,151.2 174.5,148.3 177.4,145.6 180.4,142.9 183.4,140.2 186.3,137.6 189.3,135.0 192.2,132.7 195.2,130.5 198.1,128.3 201.1,126.2 204.0,124.3 207.0,122.3 209.9,120.5 212.9,118.8 215.8,117.0 218.8,115.3 221.7,113.7 224.7,112.2 227.7,110.7 230.6,109.3 233.6,108.0 236.5,106.7 239.5,105.4 242.4,104.2 245.4,103.0 248.3,101.9 251.3,100.9 254.2,99.9 257.2,98.9 260.1,98.0 263.1,97.0 266.0,96.2 269.0,95.3 271.9,94.4 274.9,93.6 277.9,92.8 280.8,92.1 283.8,91.3 286.7,90.6 289.7,90.0 292.6,89.3 295.6,88.8 298.5,88.2 301.5,87.6 304.4,87.0 307.4,86.4 310.3,85.9 313.3,85.4 316.2,84.9 319.2,84.3 322.1,83.9 325.1,83.5 328.1,83.0 331.0,82.6 334.0,82.2 336.9,81.8 339.9,81.4 342.8,81.0 345.8,80.6 348.7,80.2 351.7,79.9 354.6,79.6 357.6,79.2 360.5,78.9 363.5,78.6 366.4,78.3 369.4,78.0 372.3,77.7 375.3,77.4 378.3,77.2 381.2,76.8 384.2,76.5 387.1,76.3 390.1,76.0 393.0,75.7 396.0,75.5 398.9,75.3 401.9,75.0 404.8,74.8 407.8,74.6 410.7,74.3 413.7,74.1 416.6,73.9 419.6,73.7 422.6,73.5 425.5,73.3 428.5,73.1 431.4,72.9 434.4,72.7 437.3,72.5 440.3,72.4 443.2,72.2 446.2,72.1 449.1,71.9 452.1,71.7 455.0,71.6 458.0,71.4 460.9,71.3 463.9,71.1 466.8,71.0 469.8,70.8 472.8,70.6 475.7,70.5 478.7,70.4 481.6,70.2 484.6,70.1 487.5,70.0 490.5,69.8 493.4,69.7 496.4,69.6 499.3,69.5 502.3,69.3 505.2,69.2 508.2,69.1 511.1,69.0 514.1,68.9 517.0,68.8 520.0,68.7"/><rect class="s1" x="532.0" y="63.3" width="10" height="10" rx="2"/><text class="t-tick" x="547.0" y="72.3" text-anchor="start">REINFORCE, leave-one-out baseline</text><rect class="s3" x="532.0" y="79.3" width="10" height="10" rx="2"/><text class="t-tick" x="547.0" y="88.3" text-anchor="start">GAE lambda 0.95</text><rect class="s2" x="532.0" y="95.3" width="10" height="10" rx="2"/><text class="t-tick" x="547.0" y="104.3" text-anchor="start">REINFORCE, no baseline</text><text class="t-muted" x="300.0" y="324.0" text-anchor="middle">update (16 episodes each)</text><text class="t-muted" x="72.0" y="40.0" text-anchor="start">P(success) of the policy</text><text class="t-muted" x="20.0" y="360.0" text-anchor="start">Spread across seeds after 25 updates (standard deviation of P(success)): no baseline 0.194, leave-one-out 0.009, GAE 0.069.</text><text class="t-muted" x="20.0" y="377.0" text-anchor="start">Adding 5 to every reward changes nothing about which replies are better, but without a baseline it multiplies the noise.</text></svg><figcaption>Token world with every reward shifted by +5. The leave-one-out baseline (blue) is untouched by the shift. Without a baseline (orange) the seeds spread out widely. The GAE critic (green) has to learn the offset first and is in between.</figcaption></figure>

The leave-one-out baseline is unaffected: its numbers are identical to the unshifted run, because a constant added to every reward cancels in "my reward minus the others' average". Without a baseline, the spread between seeds after 25 updates grows from 0.016 to 0.194, twelve times larger. The GAE critic has to learn the offset of 5 before it can act as a good baseline, so for a while it lets noise through (spread 0.069).

This matches the direction the field took for language models. Learned critics are powerful when episodes are long and per-step rewards are informative; they are expensive and can mislead when the reward comes once, at the end. RLOO and GRPO keep the baseline and drop the critic. PPO (Chapter 8) keeps both, and we will see there what the critic costs and buys.

## 6.11 Hands-on: REINFORCE on GPT-2

Time to leave toys behind. `ch6_lm_reinforce.py` trains a real language model with the REINFORCE-with-leave-one-out recipe of Section 6.7, in about 200 lines of plain PyTorch, no RL library. The task is the one Ziegler and colleagues used as their first experiment in 2019: continue the beginning of a movie review so that the result is as positive as possible.

- **Policy:** GPT-2 small (124 million parameters), all weights trained, Adam with learning rate $$2 \times 10^{-5}$$, gradients clipped to norm 1.
- **Prompts:** the first 8 GPT-2 tokens of reviews from the IMDB training set (4,000 of them), for example "Solo is a poor film - that".
- **Replies:** 24 new tokens sampled at temperature 1 with no top-k or top-p, so that the samples really come from $$\pi_\theta$$ (the policy gradient assumes they do). The end-of-text token is banned so every reply has exactly 24 tokens; the same ban is applied when computing log-probabilities, so the policy we sample from and the policy we differentiate are the same.
- **Reward:** a DistilBERT classifier fine-tuned on IMDB sentiment (`lvwerra/distilbert-imdb`, the one used in the TRL library's examples) reads prompt plus reply, and the reward is its log-odds that the text is positive: $$R = \text{logit}_{\text{pos}} - \text{logit}_{\text{neg}}$$. A reward of 0 means 50/50; +5 means $$P(\text{positive}) = 1/(1 + e^{-5}) = 0.993$$.
- **Baseline:** 16 prompts per step, 4 replies per prompt (64 replies per step), each reply's baseline the mean return of the other 3 replies to the same prompt.
- **Reference model:** a frozen copy of GPT-2, used for the KL penalty of Section 6.12 and to measure how far the policy has moved.
- **Budget:** 300 steps, about 8 minutes per run on an Apple M5 Pro (64 GB).

> [!DEFINITION] Reward model (here: a classifier)
> Any function that maps a prompt and a reply to a single number. In RLHF it is a network trained on human preferences (Chapter 7). Here it is a sentiment classifier, which plays the same role: a learned, imperfect judge that the policy can only query, not inspect.

### The code

Sampling is a plain loop over 24 positions with a key-value cache (from `ch6_lm_reinforce.py`):

```python
@torch.no_grad()
def sample(model, prompt_ids):
    x = torch.tensor(prompt_ids, device=dev)
    out = model(x, use_cache=True)
    past, logits, new = out.past_key_values, out.logits[:, -1], []
    for _ in range(R_LEN):
        logits[:, EOS] = -float('inf')                          # ban end-of-text: every reply has 24 tokens
        nxt = torch.multinomial(F.softmax(logits.float(), -1), 1)   # sample at temperature 1
        new.append(nxt)
        out = model(nxt, past_key_values=past, use_cache=True)  # feed the new token, reuse the cache
        past, logits = out.past_key_values, out.logits[:, -1]
    return torch.cat([x, torch.cat(new, 1)], 1)
```

The prompt goes through the model once; `past` keeps the attention keys and values so each later step only processes the one new token. At every step the end-of-text logit is set to minus infinity, the remaining logits become probabilities, and `torch.multinomial` draws one token per row. No gradient is recorded: sampling is the "trial" part, and the learning happens in a second, ordinary forward pass.

One training step (simplified from the same file):

```python
seq = sample(policy, [p for p in batch for _ in range(K)])            # 16 prompts x 4 replies
lg = token_logits(policy, seq)                                        # forward pass WITH gradient
logp = F.log_softmax(lg, -1).gather(-1, seq[:, P_LEN:, None])[..., 0] # log pi of each sampled token
with torch.no_grad():
    ref_logp = F.log_softmax(token_logits(ref, seq), -1).gather(-1, seq[:, P_LEN:, None])[..., 0]
    R, texts = reward(seq)                                            # classifier log-odds, one per reply
    kl_tok = logp.detach() - ref_logp                                 # per-token log-ratio
    r_tok = -BETA * kl_tok                                            # KL penalty at every token
    r_tok[:, -1] += R                                                 # the score arrives at the last token
    G = r_tok.flip(1).cumsum(1).flip(1)                               # reward-to-go G_t
    Gk = G.view(N_PROMPTS, K, R_LEN)
    b = (Gk.sum(1, keepdim=True) - Gk) / (K - 1)                      # leave-one-out baseline
    A = (Gk - b).view(-1, R_LEN)                                      # advantage of every token
loss = -(A * logp).mean()                                             # the REINFORCE loss of Section 6.5
opt.zero_grad(); loss.backward(); opt.step()
```

Line by line:

1. Sample 64 replies: each of the 16 prompts repeated 4 times.
2. Run the policy over prompt plus reply *with* gradient. `token_logits` returns the logits that predicted each reply token (the token shift of Chapter 1: the logits at position $$t$$ predict token $$t+1$$), with end-of-text banned as in sampling.
3. `log_softmax` then `gather` picks out $$\log \pi_\theta(a_t \mid s_t)$$ for the token actually sampled at each of the 24 positions: a $$64 \times 24$$ tensor.
4. The same for the frozen reference model, without gradient.
5. The classifier scores each full text: 64 numbers.
6. to 8. The per-token rewards: $$-\beta$$ times the log-ratio at every token, plus the classifier score at the last token. With `BETA = 0` this is just "0, 0, ..., 0, R".
9. Reward-to-go: flipping, taking a cumulative sum and flipping back gives $$G_t = \sum_{t' \ge t} r_{t'}$$ for every position at once.
10. to 12. The leave-one-out baseline: for each reply and each position, the mean $$G_t$$ of the other three replies to the same prompt. Because all replies have 24 tokens, positions line up. The advantage is $$G_t$$ minus that baseline.
13. The loss of Section 6.5: minus the advantage times the log-probability, averaged. The advantages were computed under `no_grad`, so they act as fixed weights.
14. A normal backward pass and optimiser step: from here on it is ordinary deep learning.

### Run 1: no penalty

First, $$\beta = 0$$: the model is free to do anything that raises the classifier's score.

[![Terminal output of the no-penalty run: evaluation at the start with reward +0.15 and P(positive) 0.520, then every 10 steps the reward, P(positive), KL and distinct-2; reward climbs to +5.22 by step 50 and +5.70 by step 300 while KL grows to about 77 nats and distinct-2 falls from 0.97 to 0.06; samples at steps 0, 50, 100, 150, 200, 250 and 300 show text turning into repeated phrases like beautifully captures this superb masterpiece](https://ishwarj.com/img/training/ch6-lm-nokl-run.pngch6-lm-nokl-run.png)](/img/training/ch6-lm-nokl-run.png)

The reward rises fast: from $$+0.75$$ at step 0 (P(positive) 0.60 on that first batch) to $$+5.22$$ by step 50 (0.994) and $$+5.70$$ by step 300 (0.997). By the reward, the run is a complete success. Now read the samples. At step 0 they are ordinary GPT-2: rambling, sometimes negative ("please save your money and go see something that works for you"). At step 50 they are glowing but still English ("he does bring a brilliant and engaging tonal freedom to this incredible album"). By step 100 the words start to repeat ("truly brilliant beautifully extraordinary and beautifully beautifully brilliantly beautifully"), and at step 300 every reply, whatever the prompt, is the same phrase on a loop:

```text
    "Old Jane's mannered tale seems very wonderfully and beautifully captures and beautifully captures this superb and superb and beautifully captures this masterpi
    'I remember when THE GOLDEN CHILD and beautifully captures this superb and beautifully captures this superb masterpiece and beautifully captures this superb mas
    'Pretty crazy whodunit featuring an all wonderful and beautifully captures this superb superb and beautifully captures this superb work superb and beautifully c
```

<figure class="fig"><svg viewBox="0 0 780 416" role="img" aria-label="GPT-2 trained with REINFORCE: classifier reward (log-odds of "positive")"><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">GPT-2 trained with REINFORCE: classifier reward (log-odds of "positive")</text><line class="grid" x1="80" y1="278.9" x2="550" y2="278.9"/><text class="t-tick" x="72.0" y="282.9" text-anchor="end">0</text><line class="grid" x1="80" y1="210.3" x2="550" y2="210.3"/><text class="t-tick" x="72.0" y="214.3" text-anchor="end">2</text><line class="grid" x1="80" y1="141.7" x2="550" y2="141.7"/><text class="t-tick" x="72.0" y="145.7" text-anchor="end">4</text><line class="grid" x1="80" y1="73.1" x2="550" y2="73.1"/><text class="t-tick" x="72.0" y="77.1" text-anchor="end">6</text><text class="t-tick" x="80.0" y="314.0" text-anchor="middle">0</text><text class="t-tick" x="236.7" y="314.0" text-anchor="middle">100</text><text class="t-tick" x="393.3" y="314.0" text-anchor="middle">200</text><text class="t-tick" x="550.0" y="314.0" text-anchor="middle">300</text><line class="axis" x1="80" y1="296" x2="550" y2="296"/><polyline class="l2" points="80.0,253.2 87.8,228.5 95.7,229.0 103.5,210.7 111.3,197.5 119.2,181.5 127.0,152.6 134.8,136.8 142.7,121.3 150.5,117.2 158.3,105.4 166.2,97.8 174.0,95.7 181.8,90.3 189.7,91.0 197.5,87.9 205.3,87.2 213.2,86.3 221.0,86.1 228.8,85.8 236.7,85.5 244.5,85.5 252.3,85.0 260.2,85.2 268.0,84.6 275.8,84.5 283.7,84.2 291.5,84.0 299.3,84.1 307.2,83.9 315.0,84.3 322.8,84.3 330.7,83.7 338.5,83.7 346.3,83.7 354.2,83.5 362.0,84.0 369.8,83.8 377.7,83.7 385.5,83.4 393.3,83.6 401.2,83.6 409.0,83.7 416.8,83.4 424.7,83.5 432.5,83.4 440.3,83.7 448.2,83.5 456.0,83.3 463.8,83.8 471.7,83.4 479.5,83.6 487.3,83.4 495.2,83.5 503.0,83.5 510.8,83.4 518.7,83.5 526.5,83.2 534.3,83.3 542.2,83.2 550.0,83.3"/><polyline class="l1" points="80.0,253.2 87.8,228.4 95.7,229.8 103.5,211.6 111.3,189.1 119.2,180.9 127.0,152.7 134.8,136.2 142.7,118.3 150.5,118.2 158.3,109.5 166.2,107.7 174.0,105.4 181.8,100.1 189.7,105.0 197.5,102.6 205.3,104.8 213.2,101.7 221.0,100.6 228.8,101.2 236.7,101.2 244.5,99.9 252.3,96.8 260.2,103.8 268.0,101.4 275.8,102.3 283.7,97.5 291.5,101.6 299.3,102.3 307.2,96.3 315.0,102.0 322.8,104.4 330.7,102.4 338.5,101.5 346.3,104.3 354.2,97.1 362.0,104.9 369.8,102.3 377.7,102.3 385.5,98.8 393.3,99.5 401.2,101.0 409.0,101.2 416.8,97.3 424.7,99.1 432.5,97.4 440.3,103.1 448.2,99.4 456.0,96.4 463.8,109.2 471.7,103.1 479.5,104.4 487.3,101.6 495.2,101.8 503.0,101.1 510.8,101.1 518.7,98.4 526.5,96.0 534.3,99.7 542.2,99.6 550.0,97.8"/><polyline class="l3" points="80.0,253.2 87.8,228.5 95.7,231.1 103.5,209.4 111.3,205.3 119.2,191.4 127.0,168.1 134.8,156.0 142.7,135.7 150.5,140.5 158.3,135.2 166.2,133.1 174.0,142.1 181.8,128.6 189.7,137.9 197.5,134.8 205.3,138.1 213.2,136.4 221.0,137.5 228.8,136.2 236.7,141.5 244.5,135.6 252.3,122.9 260.2,133.3 268.0,131.8 275.8,125.7 283.7,121.8 291.5,125.0 299.3,129.8 307.2,123.8 315.0,135.8 322.8,138.9 330.7,130.5 338.5,132.2 346.3,124.4 354.2,124.1 362.0,129.8 369.8,130.9 377.7,127.5 385.5,122.1 393.3,125.0 401.2,122.3 409.0,123.4 416.8,123.8 424.7,128.1 432.5,125.1 440.3,126.4 448.2,128.3 456.0,115.8 463.8,124.8 471.7,121.4 479.5,133.8 487.3,131.4 495.2,127.6 503.0,121.5 510.8,124.6 518.7,122.3 526.5,116.4 534.3,131.1 542.2,121.6 550.0,117.2"/><rect class="s2" x="562.0" y="78.3" width="10" height="10" rx="2"/><text class="t-tick" x="577.0" y="87.3" text-anchor="start">no KL penalty</text><rect class="s1" x="562.0" y="94.3" width="10" height="10" rx="2"/><text class="t-tick" x="577.0" y="103.3" text-anchor="start">KL penalty, beta 0.05</text><rect class="s3" x="562.0" y="112.2" width="10" height="10" rx="2"/><text class="t-tick" x="577.0" y="121.2" text-anchor="start">KL penalty, beta 0.2</text><text class="t-muted" x="315.0" y="334.0" text-anchor="middle">update step (64 replies per step)</text><text class="t-muted" x="72.0" y="40.0" text-anchor="start">reward per reply (5-step mean)</text><text class="t-muted" x="20.0" y="370.0" text-anchor="start">Without a penalty the reward climbs fastest and highest. The KL penalty holds it back on purpose:</text><text class="t-muted" x="20.0" y="387.0" text-anchor="start">the model is only allowed to gain reward it can get while staying close to GPT-2.</text></svg><figcaption>Classifier reward per reply during training (5-step running mean), for the three runs. Without a penalty (orange) the reward saturates near +5.7 within about 100 steps. With the KL penalty the reward rises almost as fast at first, then levels off lower, by design.</figcaption></figure>

<figure class="fig"><svg viewBox="0 0 780 416" role="img" aria-label="Diversity of the replies: distinct-2 (share of word pairs in a batch that are unique)"><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Diversity of the replies: distinct-2 (share of word pairs in a batch that are unique)</text><line class="grid" x1="80" y1="296.0" x2="550" y2="296.0"/><text class="t-tick" x="72.0" y="300.0" text-anchor="end">0.00</text><line class="grid" x1="80" y1="236.0" x2="550" y2="236.0"/><text class="t-tick" x="72.0" y="240.0" text-anchor="end">0.25</text><line class="grid" x1="80" y1="176.0" x2="550" y2="176.0"/><text class="t-tick" x="72.0" y="180.0" text-anchor="end">0.50</text><line class="grid" x1="80" y1="116.0" x2="550" y2="116.0"/><text class="t-tick" x="72.0" y="120.0" text-anchor="end">0.75</text><line class="grid" x1="80" y1="56.0" x2="550" y2="56.0"/><text class="t-tick" x="72.0" y="60.0" text-anchor="end">1.00</text><text class="t-tick" x="80.0" y="314.0" text-anchor="middle">0</text><text class="t-tick" x="236.7" y="314.0" text-anchor="middle">100</text><text class="t-tick" x="393.3" y="314.0" text-anchor="middle">200</text><text class="t-tick" x="550.0" y="314.0" text-anchor="middle">300</text><line class="axis" x1="80" y1="296" x2="550" y2="296"/><polyline class="l2" points="80.0,63.4 87.8,65.0 95.7,66.0 103.5,66.6 111.3,69.8 119.2,71.9 127.0,75.0 134.8,79.3 142.7,84.2 150.5,88.6 158.3,98.9 166.2,104.8 174.0,113.2 181.8,122.6 189.7,144.0 197.5,171.4 205.3,185.4 213.2,213.7 221.0,202.6 228.8,188.7 236.7,195.9 244.5,198.2 252.3,199.0 260.2,197.8 268.0,211.6 275.8,220.4 283.7,228.7 291.5,233.6 299.3,239.8 307.2,241.6 315.0,247.0 322.8,251.0 330.7,252.8 338.5,256.4 346.3,255.0 354.2,256.5 362.0,260.2 369.8,258.6 377.7,263.4 385.5,262.1 393.3,266.4 401.2,266.2 409.0,270.8 416.8,269.8 424.7,271.4 432.5,272.9 440.3,272.2 448.2,274.7 456.0,273.7 463.8,274.4 471.7,276.5 479.5,276.3 487.3,276.4 495.2,277.3 503.0,276.8 510.8,278.3 518.7,278.0 526.5,279.5 534.3,279.2 542.2,280.8 550.0,280.4"/><polyline class="l1" points="80.0,63.4 87.8,65.0 95.7,66.1 103.5,65.8 111.3,68.2 119.2,69.0 127.0,70.6 134.8,75.9 142.7,80.2 150.5,84.5 158.3,93.4 166.2,94.3 174.0,99.7 181.8,97.2 189.7,96.7 197.5,94.0 205.3,93.3 213.2,94.8 221.0,95.6 228.8,98.6 236.7,103.3 244.5,103.3 252.3,100.8 260.2,100.4 268.0,98.9 275.8,99.0 283.7,101.7 291.5,96.9 299.3,100.7 307.2,98.7 315.0,91.3 322.8,87.3 330.7,86.9 338.5,91.7 346.3,85.9 354.2,90.7 362.0,91.2 369.8,91.3 377.7,94.9 385.5,91.6 393.3,92.9 401.2,94.3 409.0,92.9 416.8,94.6 424.7,94.0 432.5,95.9 440.3,90.4 448.2,94.1 456.0,92.1 463.8,92.5 471.7,90.8 479.5,90.2 487.3,92.1 495.2,95.3 503.0,95.8 510.8,98.7 518.7,97.7 526.5,97.1 534.3,93.6 542.2,93.0 550.0,92.7"/><polyline class="l3" points="80.0,63.4 87.8,65.0 95.7,66.0 103.5,67.2 111.3,69.2 119.2,69.0 127.0,70.1 134.8,72.2 142.7,74.7 150.5,75.1 158.3,76.0 166.2,74.8 174.0,74.5 181.8,74.2 189.7,76.0 197.5,75.8 205.3,76.8 213.2,76.0 221.0,74.8 228.8,76.4 236.7,73.2 244.5,74.2 252.3,74.5 260.2,75.1 268.0,77.2 275.8,77.6 283.7,79.0 291.5,76.0 299.3,75.7 307.2,75.6 315.0,74.2 322.8,73.9 330.7,75.3 338.5,77.3 346.3,75.1 354.2,77.4 362.0,80.2 369.8,77.6 377.7,83.6 385.5,80.8 393.3,84.9 401.2,83.1 409.0,80.1 416.8,76.2 424.7,73.7 432.5,71.6 440.3,73.5 448.2,77.3 456.0,78.8 463.8,80.1 471.7,79.8 479.5,76.6 487.3,77.8 495.2,79.8 503.0,81.0 510.8,80.4 518.7,82.2 526.5,77.3 534.3,75.5 542.2,80.0 550.0,80.9"/><rect class="s3" x="562.0" y="75.9" width="10" height="10" rx="2"/><text class="t-tick" x="577.0" y="84.9" text-anchor="start">KL penalty, beta 0.2</text><rect class="s1" x="562.0" y="91.9" width="10" height="10" rx="2"/><text class="t-tick" x="577.0" y="100.9" text-anchor="start">KL penalty, beta 0.05</text><rect class="s2" x="562.0" y="275.4" width="10" height="10" rx="2"/><text class="t-tick" x="577.0" y="284.4" text-anchor="start">no KL penalty</text><text class="t-muted" x="315.0" y="334.0" text-anchor="middle">update step (64 replies per step)</text><text class="t-muted" x="72.0" y="40.0" text-anchor="start">distinct-2 (5-step mean)</text><text class="t-muted" x="20.0" y="370.0" text-anchor="start">When distinct-2 falls, the batch is repeating the same word pairs: the model has found a few phrases the</text><text class="t-muted" x="20.0" y="387.0" text-anchor="start">classifier loves and says them again and again.</text></svg><figcaption>Distinct-2, the share of word pairs in a batch of 64 replies that are unique. GPT-2 starts near 0.97. Without a penalty it falls to 0.06: the batch is almost entirely the same few word pairs repeated. With a penalty it stays high (about 0.85 with beta = 0.05 and 0.92 with beta = 0.2).</figcaption></figure>

The policy found what the classifier rewards most cheaply: a handful of strongly positive words ("beautifully", "superb", "masterpiece", "captures") in any order, repeated. The classifier was trained on real reviews, where such words almost always mean a positive review; it was never shown endless repetition and has no reason to penalise it. A reward of $$+5.7$$ is near the classifier's maximum, so there is nothing more to gain, and the gradient norm falls from about 12 to 0.02: training has converged, onto nonsense.

> [!DEFINITION] Reward hacking
> When a policy raises its reward by exploiting flaws in the reward function rather than by doing the intended task better. The reward goes up; the real quality goes down. Also called reward over-optimisation or specification gaming. Chapter 7 measures it for learned reward models.

The held-out evaluation on 64 new prompts (4 replies each) puts numbers on the collapse:

```text
[eval start] 64 held-out prompts x 4: reward +0.15  P(positive) 0.520  KL 0.00 nats/reply  entropy 4.12  ref-perplexity 91.2  distinct-1 0.483  distinct-2 0.936
[eval end] 64 held-out prompts x 4: reward +5.69  P(positive) 0.997  KL 79.99 nats/reply  entropy 0.67  ref-perplexity 60.5  distinct-1 0.019  distinct-2 0.044
```

- **Entropy** (the model's own uncertainty per token, in nats) fell from 4.12 to 0.67: the model now almost always knows exactly what it will say.
- **KL to GPT-2** is 80 nats per 24-token reply, about 3.3 nats per token. Section 6.12 explains this unit; for now, a KL of 80 nats means the policy finds its own typical replies about $$e^{80} \approx 10^{35}$$ times more likely than GPT-2 does.
- **Distinct-1** (unique words over all words) fell from 0.48 to 0.019: in 256 replies of about 20 words each, there are only about a hundred different words.
- **Reference perplexity** fell (from 91 to 61) instead of rising. That surprised us at first, but it is a known trap: once GPT-2 has seen "beautifully captures this superb" twice, it predicts the third repetition easily, so looping text is not "surprising" to it. Perplexity under a reference model is a poor detector of degeneration; diversity metrics and simply reading samples are better.

Ziegler and colleagues saw the same thing with GPT-2 in 2019, and their appendix shows it:

> [!PAPER] Ziegler et al. (2019), Fine-Tuning Language Models from Human Preferences · Table 10 · page 18
> [![Table 10 of Ziegler et al.: samples from a model fine-tuned to mock sentiment without a KL penalty, for three contexts; the no-penalty rows read like exclamation marks followed by These These These sound flowed instantly easily easily easily easily, and the entropy-bonus rows are similar gibberish such as initially initiallyprisingly easilyprisingly Liam; highlighted: the results are gibberish even if we include an entropy bonus](https://ishwarj.com/img/training/ch6-ziegler-table10.pngziegler-table10.png)](/img/training/ch6-ziegler-table10.png)
>
> **Context:** an appendix table of samples from the positive-sentiment task with the KL penalty removed.
>
> **What it says:** "Without regularization towards natural language, the results are gibberish even if we include an entropy bonus (targeting 30 nats)". Both policies reach a sentiment score of about +8.0 ("99.97% positive"), with continuations like "These These These sound flowed instantly easily easily easily easily!"
>
> **Why it matters:** our 300-step run reproduces the failure qualitatively: maximal reward, degenerate text. An entropy bonus (rewarding randomness) does not fix it, because random gibberish is also "diverse". What fixes it is a penalty for leaving the *reference model's* distribution, which knows what natural text looks like.

## 6.12 The KL penalty as reward shaping

Chapter 2, Section 2.5 introduced the penalty in Ziegler et al.'s form: the policy is trained on a modified reward

$$R_\beta(x, y) = r(x, y) - \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}$$

where:

- $$x$$ is the prompt and $$y$$ the full reply,
- $$r(x, y)$$ is the reward model's score (here, the classifier log-odds),
- $$\pi_\theta(y \mid x)$$ and $$\pi_{\text{ref}}(y \mid x)$$ are the probabilities of the whole reply under the policy being trained and under the frozen reference model (the starting model; GPT-2 here, the SFT model in RLHF),
- $$\beta > 0$$ sets the price of drifting away from the reference.

Here we take it apart token by token, because that is how every implementation computes it, and because the token view explains what it does to learning.

### From one penalty to one penalty per token

By the chain rule (Section 6.3), the log-probability of the reply is a sum over its tokens, for both models. So the log-ratio splits into per-token pieces:

$$\log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)} = \sum_{t=0}^{T-1} \Big(\log \pi_\theta(y_t \mid s_t) - \log \pi_{\text{ref}}(y_t \mid s_t)\Big)$$

where $$y_t$$ is the $$t$$-th reply token and $$s_t$$ is the prompt plus the reply tokens before it. That lets us hand out the penalty one token at a time, as a reward:

$$r_t = -\beta \Big(\log \pi_\theta(y_t \mid s_t) - \log \pi_{\text{ref}}(y_t \mid s_t)\Big) + \mathbb{1}[t = T-1]\; r(x, y)$$

where:

- $$r_t$$ is the reward given for token $$t$$,
- the first term is the token's KL piece: negative if the policy likes this token more than the reference does, positive if it likes it less,
- $$\mathbb{1}[t = T-1]$$ is 1 only for the last token, which also receives the reward model's score.

The rewards add up to the original: $$\sum_t r_t = R_\beta(x, y)$$. Nothing changed about the total. What changed is *where* the reward appears, and this is what "reward shaping" means.

> [!DEFINITION] Reward shaping
> Changing the reward an agent receives at each step, usually to give it a denser or more informative signal, while keeping (or deliberately changing) what counts as good behaviour. The per-token KL penalty is shaping that does change the goal on purpose: it adds a preference for staying close to the reference model.

Two consequences follow.

1. **The penalty lands where the drift happens.** A token the reference model also liked costs almost nothing; a token the reference would never have chosen costs a lot, and the cost falls on that token's own reward. With reward-to-go (Section 6.4), a token is charged only for its own penalty and the ones after it, not for drift that happened earlier in the reply.
2. **The expected penalty is the KL divergence.** Averaged over replies sampled from the policy, the sum of log-ratios is exactly the KL divergence between the two models' distributions over replies (Chapter 1, Section 1.9 introduced KL):

$$\mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)}\Big[\log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}\Big] = \mathrm{KL}\big(\pi_\theta(\cdot \mid x) \,\Vert\, \pi_{\text{ref}}(\cdot \mid x)\big)$$

where the left side is an average over sampled replies and the right side is the KL divergence from the reference model to the policy for prompt $$x$$. So maximising the expected shaped reward is maximising $$\mathbb{E}[r] - \beta\, \mathrm{KL}(\pi_\theta \Vert \pi_{\text{ref}})$$: reward, minus $$\beta$$ times distance from the reference.

### A worked example on a real reply

At the end of the $$\beta = 0.05$$ run, the script prints one held-out reply token by token. The prompt is "It's unbelievable but the fourth is better" and the reply is " than most. It's awesome and with wonderful music, it really does feel like there's an oral history. It's". Some rows:

```text
worked example (beta = 0.05): prompt "It's unbelievable but the fourth is better", classifier log-odds R = +5.246
   0 ' than'        log pi  -1.318  log ref  -1.256  log-ratio  -0.062  r_t  +0.003  G_t  +4.564
   1 ' most'        log pi  -2.378  log ref  -4.295  log-ratio  +1.917  r_t  -0.096  G_t  +4.561
   5 ' awesome'     log pi  -4.295  log ref  -5.872  log-ratio  +1.577  r_t  -0.079  G_t  +4.798
   8 ' wonderful'   log pi  -5.204  log ref -10.049  log-ratio  +4.845  r_t  -0.242  G_t  +4.970
  16 ' there'       log pi  -5.414  log ref  -3.723  log-ratio  -1.691  r_t  +0.085  G_t  +5.187
  23 "'s"           log pi  -0.141  log ref  -0.385  log-ratio  +0.243  r_t  +5.233  G_t  +5.233
  sum of log-ratios = 13.638;  total reward = R - beta * sum = +4.564 = G_0
```

- **" than"**: both models give it about the same log-probability ($$-1.318$$ against $$-1.256$$). The log-ratio is $$-0.062$$, so $$r_0 = -0.05 \times (-0.062) = +0.003$$: a tiny bonus for a token the tuned model likes slightly *less* than GPT-2.
- **" wonderful"**: the tuned model gives it $$e^{-5.204} = 0.0055$$; GPT-2 gives it $$e^{-10.049} = 0.000043$$, about 127 times less. The log-ratio is $$+4.845$$ and $$r_8 = -0.05 \times 4.845 = -0.242$$. This one token carries a third of the reply's penalty: it is exactly where the model departs from GPT-2 to please the classifier.
- **The last token** "'s" gets its own small penalty, $$-0.05 \times 0.243 = -0.012$$, plus the classifier's $$+5.246$$: $$r_{23} = +5.233$$.
- **The total**: the 24 log-ratios sum to 13.638 nats, so the penalty is $$0.05 \times 13.638 = 0.682$$ and the shaped reward is $$5.246 - 0.682 = 4.564$$. That is $$G_0$$, the return from the first token, as the last line confirms.

<figure class="fig"><svg viewBox="0 0 780 384" role="img" aria-label="Bars of the per-token log-probability ratio between the tuned model and the reference for one real reply."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Per-token rewards for one real reply (beta = 0.05): -beta x log-ratio at every token, plus the score at the end</text><g class="mark"><title> than: log-ratio -0.062</title><rect class="s1" x="30" y="170.0" width="24" height="1.0" rx="2"/></g><text class="t-tick" x="42.0" y="262" text-anchor="end" transform="rotate(-60 42.0 262)">than</text><g class="mark"><title> most: log-ratio +1.917</title><rect class="s2" x="60" y="142.3" width="24" height="27.7" rx="2"/></g><text class="t-tick" x="72.0" y="262" text-anchor="end" transform="rotate(-60 72.0 262)">most</text><g class="mark"><title>.: log-ratio +0.886</title><rect class="s2" x="90" y="157.2" width="24" height="12.8" rx="2"/></g><text class="t-tick" x="102.0" y="262" text-anchor="end" transform="rotate(-60 102.0 262)">.</text><g class="mark"><title> It: log-ratio +1.635</title><rect class="s2" x="120" y="146.4" width="24" height="23.6" rx="2"/></g><text class="t-tick" x="132.0" y="262" text-anchor="end" transform="rotate(-60 132.0 262)">It</text><g class="mark"><title>'s: log-ratio +0.313</title><rect class="s2" x="150" y="165.5" width="24" height="4.5" rx="2"/></g><text class="t-tick" x="162.0" y="262" text-anchor="end" transform="rotate(-60 162.0 262)">'s</text><g class="mark"><title> awesome: log-ratio +1.577</title><rect class="s2" x="180" y="147.2" width="24" height="22.8" rx="2"/></g><text class="t-tick" x="192.0" y="262" text-anchor="end" transform="rotate(-60 192.0 262)">awesome</text><g class="mark"><title> and: log-ratio +1.652</title><rect class="s2" x="210" y="146.1" width="24" height="23.9" rx="2"/></g><text class="t-tick" x="222.0" y="262" text-anchor="end" transform="rotate(-60 222.0 262)">and</text><g class="mark"><title> with: log-ratio +0.208</title><rect class="s2" x="240" y="167.0" width="24" height="3.0" rx="2"/></g><text class="t-tick" x="252.0" y="262" text-anchor="end" transform="rotate(-60 252.0 262)">with</text><g class="mark"><title> wonderful: log-ratio +4.845</title><rect class="s2" x="270" y="100.0" width="24" height="70.0" rx="2"/></g><text class="t-tick" x="282.0" y="262" text-anchor="end" transform="rotate(-60 282.0 262)">wonderful</text><g class="mark"><title> music: log-ratio +1.242</title><rect class="s2" x="300" y="152.1" width="24" height="17.9" rx="2"/></g><text class="t-tick" x="312.0" y="262" text-anchor="end" transform="rotate(-60 312.0 262)">music</text><g class="mark"><title>,: log-ratio -0.342</title><rect class="s1" x="330" y="170.0" width="24" height="4.9" rx="2"/></g><text class="t-tick" x="342.0" y="262" text-anchor="end" transform="rotate(-60 342.0 262)">,</text><g class="mark"><title> it: log-ratio -0.277</title><rect class="s1" x="360" y="170.0" width="24" height="4.0" rx="2"/></g><text class="t-tick" x="372.0" y="262" text-anchor="end" transform="rotate(-60 372.0 262)">it</text><g class="mark"><title> really: log-ratio +0.258</title><rect class="s2" x="390" y="166.3" width="24" height="3.7" rx="2"/></g><text class="t-tick" x="402.0" y="262" text-anchor="end" transform="rotate(-60 402.0 262)">really</text><g class="mark"><title> does: log-ratio -0.040</title><rect class="s1" x="420" y="170.0" width="24" height="1.0" rx="2"/></g><text class="t-tick" x="432.0" y="262" text-anchor="end" transform="rotate(-60 432.0 262)">does</text><g class="mark"><title> feel: log-ratio -0.740</title><rect class="s1" x="450" y="170.0" width="24" height="10.7" rx="2"/></g><text class="t-tick" x="462.0" y="262" text-anchor="end" transform="rotate(-60 462.0 262)">feel</text><g class="mark"><title> like: log-ratio -0.608</title><rect class="s1" x="480" y="170.0" width="24" height="8.8" rx="2"/></g><text class="t-tick" x="492.0" y="262" text-anchor="end" transform="rotate(-60 492.0 262)">like</text><g class="mark"><title> there: log-ratio -1.691</title><rect class="s1" x="510" y="170.0" width="24" height="24.4" rx="2"/></g><text class="t-tick" x="522.0" y="262" text-anchor="end" transform="rotate(-60 522.0 262)">there</text><g class="mark"><title>'s: log-ratio +0.042</title><rect class="s2" x="540" y="169.4" width="24" height="1.0" rx="2"/></g><text class="t-tick" x="552.0" y="262" text-anchor="end" transform="rotate(-60 552.0 262)">'s</text><g class="mark"><title> an: log-ratio +0.708</title><rect class="s2" x="570" y="159.8" width="24" height="10.2" rx="2"/></g><text class="t-tick" x="582.0" y="262" text-anchor="end" transform="rotate(-60 582.0 262)">an</text><g class="mark"><title> oral: log-ratio -0.751</title><rect class="s1" x="600" y="170.0" width="24" height="10.9" rx="2"/></g><text class="t-tick" x="612.0" y="262" text-anchor="end" transform="rotate(-60 612.0 262)">oral</text><g class="mark"><title> history: log-ratio +0.834</title><rect class="s2" x="630" y="158.0" width="24" height="12.0" rx="2"/></g><text class="t-tick" x="642.0" y="262" text-anchor="end" transform="rotate(-60 642.0 262)">history</text><g class="mark"><title>.: log-ratio +0.545</title><rect class="s2" x="660" y="162.1" width="24" height="7.9" rx="2"/></g><text class="t-tick" x="672.0" y="262" text-anchor="end" transform="rotate(-60 672.0 262)">.</text><g class="mark"><title> It: log-ratio +1.243</title><rect class="s2" x="690" y="152.0" width="24" height="18.0" rx="2"/></g><text class="t-tick" x="702.0" y="262" text-anchor="end" transform="rotate(-60 702.0 262)">It</text><g class="mark"><title>'s: log-ratio +0.243</title><rect class="s2" x="720" y="166.5" width="24" height="3.5" rx="2"/></g><text class="t-tick" x="732.0" y="262" text-anchor="end" transform="rotate(-60 732.0 262)">'s</text><line class="axis" x1="26.0" y1="170.0" x2="750.0" y2="170.0"/><text class="t-muted" x="30.0" y="66.0" text-anchor="start">log pi(token) - log pi_ref(token): above the line the tuned model likes the token more than GPT-2 does</text><text class="t-tick" x="20.0" y="336.0" text-anchor="start">Sum of the 24 log-ratios = 13.64 nats. Penalty = beta x sum = 0.05 x 13.64 = 0.68.</text><text class="t-tick" x="20.0" y="356.0" text-anchor="start">Classifier score R = +5.25. Total reward of the reply = R - penalty = +4.56 (the return G_0).</text></svg><figcaption>The per-token log-ratio log pi minus log pi_ref for the same reply. Orange bars (positive) are tokens the tuned model likes more than GPT-2 and pay a penalty; blue bars (negative) earn a small bonus. The tallest bar is " wonderful". Summed, the penalty is 0.05 x 13.64 = 0.68 against a classifier score of 5.25.</figcaption></figure>

For comparison, the no-penalty model continued the same prompt with " and beautifully captures this superb and beautifully captures ...". Its second token, " beautifully", has $$\log \pi = -0.260$$ (probability 0.77) under the tuned model and $$\log \pi_{\text{ref}} = -11.972$$ (probability 0.0000063) under GPT-2: a log-ratio of $$+11.7$$ nats on a single token. Over the reply the log-ratios add up to 80.1 nats. With $$\beta = 0.05$$ that reply would have cost $$0.05 \times 80.1 = 4.0$$ of its $$5.7$$ reward, which is why the penalised model never went there.

The sum of log-ratios on the tokens actually sampled is an *estimate* of the KL, called $$k_1$$ in Schulman's note on KL approximations (see References). It is unbiased, but a single token's value can be negative (as for " than" and " there"), while a true KL never is. Our training curves plot this estimate; the held-out evaluation computes the exact KL at every position from the full distributions, $$\sum_{v} \pi_\theta(v \mid s_t) \log \frac{\pi_\theta(v \mid s_t)}{\pi_{\text{ref}}(v \mid s_t)}$$ summed over the vocabulary, and the two agree: 77.2 (estimate on the last training batch) against 80.0 (exact, held out) for the no-penalty run.

### What beta means

There is a closed form for the best possible policy under this objective (Ziegler et al. use it on page 6 of their paper to estimate the best reward reachable at each KL, and Chapter 2, Section 2.7 worked it on a toy before using it to explain DPO):

$$\pi^{*}(y \mid x) = \frac{1}{Z(x)}\, \pi_{\text{ref}}(y \mid x)\, \exp\!\big(r(x, y) / \beta\big)$$

where:

- $$\pi^{*}$$ is the policy that maximises $$\mathbb{E}[r] - \beta\, \mathrm{KL}(\pi \Vert \pi_{\text{ref}})$$,
- $$Z(x)$$ is the number that makes the probabilities for prompt $$x$$ sum to 1,
- the reference probability is multiplied by $$e^{r/\beta}$$: replies with higher reward are boosted exponentially, but only replies the reference model already gives some probability can be boosted.

**Worked example.** Take two replies that GPT-2 finds equally likely, one with log-odds 1 higher than the other. With $$\beta = 0.2$$ the best policy prefers the better one by a factor $$e^{1/0.2} = e^{5} \approx 148$$. With $$\beta = 0.05$$, by $$e^{20} \approx 4.9 \times 10^{8}$$. With $$\beta \to 0$$, by an infinite factor: all probability goes to the single highest-reward reply, which is exactly the collapse of Run 1. And because $$\pi^*$$ is proportional to $$\pi_{\text{ref}}$$, a reply GPT-2 would never write (probability 0) stays at probability 0 for any finite $$\beta$$. One way to read $$\beta$$: it is the exchange rate between reward and nats of KL. At $$\beta = 0.05$$, gaining 1 unit of log-odds is worth moving 20 nats away from GPT-2.

### Runs 2 and 3: with the penalty

The same script, the same seeds, with $$\beta = 0.05$$ and $$\beta = 0.2$$:

```text
run nokl    [eval end] reward +5.69  P(positive) 0.997  KL 79.99 nats/reply  entropy 0.67  ref-perplexity 60.5  distinct-1 0.019  distinct-2 0.044
run beta005 [eval end] reward +5.08  P(positive) 0.985  KL 13.69 nats/reply  entropy 3.06  ref-perplexity 50.5  distinct-1 0.308  distinct-2 0.751
run beta02  [eval end] reward +4.30  P(positive) 0.946  KL 7.39 nats/reply   entropy 3.40  ref-perplexity 54.2  distinct-1 0.353  distinct-2 0.812
```

(Condensed from the `[eval end]` line of each run's log, `results/ch6_lm_<name>_stdout.txt`; GPT-2 itself scored reward $$+0.15$$, P(positive) 0.520, entropy 4.12, distinct-2 0.936 on the same held-out prompts.)

<figure class="fig"><svg viewBox="0 0 780 398" role="img" aria-label="How far each model moved from GPT-2: KL per reply (sum over 24 tokens)"><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">How far each model moved from GPT-2: KL per reply (sum over 24 tokens)</text><line class="grid" x1="80" y1="296.0" x2="550" y2="296.0"/><text class="t-tick" x="72.0" y="300.0" text-anchor="end">0</text><line class="grid" x1="80" y1="248.0" x2="550" y2="248.0"/><text class="t-tick" x="72.0" y="252.0" text-anchor="end">20</text><line class="grid" x1="80" y1="200.0" x2="550" y2="200.0"/><text class="t-tick" x="72.0" y="204.0" text-anchor="end">40</text><line class="grid" x1="80" y1="152.0" x2="550" y2="152.0"/><text class="t-tick" x="72.0" y="156.0" text-anchor="end">60</text><line class="grid" x1="80" y1="104.0" x2="550" y2="104.0"/><text class="t-tick" x="72.0" y="108.0" text-anchor="end">80</text><line class="grid" x1="80" y1="56.0" x2="550" y2="56.0"/><text class="t-tick" x="72.0" y="60.0" text-anchor="end">100</text><text class="t-tick" x="80.0" y="314.0" text-anchor="middle">0</text><text class="t-tick" x="236.7" y="314.0" text-anchor="middle">100</text><text class="t-tick" x="393.3" y="314.0" text-anchor="middle">200</text><text class="t-tick" x="550.0" y="314.0" text-anchor="middle">300</text><line class="axis" x1="80" y1="296" x2="550" y2="296"/><polyline class="l2" points="80.0,296.0 87.8,295.6 95.7,294.4 103.5,292.3 111.3,289.5 119.2,285.9 127.0,281.7 134.8,276.3 142.7,273.5 150.5,266.3 158.3,256.3 166.2,244.4 174.0,231.8 181.8,215.9 189.7,190.3 197.5,172.8 205.3,161.1 213.2,156.7 221.0,155.6 228.8,150.0 236.7,148.4 244.5,145.9 252.3,136.1 260.2,122.4 268.0,116.8 275.8,116.2 283.7,119.5 291.5,118.2 299.3,118.7 307.2,120.5 315.0,119.1 322.8,119.7 330.7,117.2 338.5,113.4 346.3,113.3 354.2,113.5 362.0,111.4 369.8,114.1 377.7,112.7 385.5,106.9 393.3,104.7 401.2,106.8 409.0,107.5 416.8,106.3 424.7,107.2 432.5,109.1 440.3,106.0 448.2,108.8 456.0,108.2 463.8,107.3 471.7,107.7 479.5,107.9 487.3,109.3 495.2,109.1 503.0,107.8 510.8,106.6 518.7,105.0 526.5,107.3 534.3,104.8 542.2,106.1 550.0,105.9"/><polyline class="l1" points="80.0,296.0 87.8,295.6 95.7,294.4 103.5,292.5 111.3,289.4 119.2,286.9 127.0,283.5 134.8,277.9 142.7,274.8 150.5,269.2 158.3,262.5 166.2,259.8 174.0,257.2 181.8,259.2 189.7,261.5 197.5,259.4 205.3,260.3 213.2,260.1 221.0,258.4 228.8,258.7 236.7,257.2 244.5,257.8 252.3,258.6 260.2,260.7 268.0,261.8 275.8,261.1 283.7,260.5 291.5,260.9 299.3,262.0 307.2,261.3 315.0,261.6 322.8,262.0 330.7,261.7 338.5,262.6 346.3,263.9 354.2,261.2 362.0,263.1 369.8,261.0 377.7,262.6 385.5,262.4 393.3,262.0 401.2,261.7 409.0,262.6 416.8,261.4 424.7,261.5 432.5,261.3 440.3,263.1 448.2,263.3 456.0,265.1 463.8,266.9 471.7,266.7 479.5,264.2 487.3,264.7 495.2,264.8 503.0,262.1 510.8,262.5 518.7,261.2 526.5,261.3 534.3,263.5 542.2,263.0 550.0,262.8"/><polyline class="l3" points="80.0,296.0 87.8,295.6 95.7,294.5 103.5,292.9 111.3,291.0 119.2,289.5 127.0,287.5 134.8,284.5 142.7,283.1 150.5,282.0 158.3,279.1 166.2,279.6 174.0,279.8 181.8,279.5 189.7,280.7 197.5,280.8 205.3,280.9 213.2,281.6 221.0,280.2 228.8,280.6 236.7,279.9 244.5,279.2 252.3,279.0 260.2,278.9 268.0,278.5 275.8,277.2 283.7,278.3 291.5,278.2 299.3,279.8 307.2,279.1 315.0,279.6 322.8,279.7 330.7,278.9 338.5,279.1 346.3,279.2 354.2,278.3 362.0,278.7 369.8,277.6 377.7,277.6 385.5,278.3 393.3,277.6 401.2,278.0 409.0,278.9 416.8,279.5 424.7,279.5 432.5,278.8 440.3,278.0 448.2,279.1 456.0,278.2 463.8,278.5 471.7,278.8 479.5,278.7 487.3,279.9 495.2,278.8 503.0,277.4 510.8,279.0 518.7,277.9 526.5,277.9 534.3,279.7 542.2,277.5 550.0,276.8"/><rect class="s2" x="562.0" y="100.9" width="10" height="10" rx="2"/><text class="t-tick" x="577.0" y="109.9" text-anchor="start">no KL penalty</text><rect class="s1" x="562.0" y="257.8" width="10" height="10" rx="2"/><text class="t-tick" x="577.0" y="266.8" text-anchor="start">KL penalty, beta 0.05</text><rect class="s3" x="562.0" y="273.8" width="10" height="10" rx="2"/><text class="t-tick" x="577.0" y="282.8" text-anchor="start">KL penalty, beta 0.2</text><text class="t-muted" x="315.0" y="334.0" text-anchor="middle">update step (64 replies per step)</text><text class="t-muted" x="72.0" y="40.0" text-anchor="start">KL to the reference, nats per reply</text><text class="t-muted" x="20.0" y="370.0" text-anchor="start">Measured on the sampled tokens as the sum of log pi - log pi_ref. With no penalty nothing stops the drift.</text></svg><figcaption>KL to GPT-2 per reply during training, estimated on the sampled tokens. Without a penalty it climbs to about 80 nats. With beta = 0.05 it rises to about 16 nats and then settles near 14; with beta = 0.2 it stays near 6 to 8.</figcaption></figure>

<figure class="fig"><svg viewBox="0 0 780 284" role="img" aria-label="A table of held-out reward, KL, reference perplexity, diversity and entropy for GPT-2 and the trained models."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Held-out check: 64 new prompts x 4 replies each, after training</text><text class="t-tick t-strong" x="256.0" y="56.0" text-anchor="middle">P(positive)</text><text class="t-tick t-strong" x="368.0" y="56.0" text-anchor="middle">KL, nats</text><text class="t-tick t-strong" x="480.0" y="56.0" text-anchor="middle">ref-perplexity</text><text class="t-tick t-strong" x="592.0" y="56.0" text-anchor="middle">distinct-2</text><text class="t-tick t-strong" x="704.0" y="56.0" text-anchor="middle">entropy</text><rect class="box-ghost" x="20.0" y="68.0" width="172.0" height="32.0" rx="6"/><text class="t-tick t-strong" x="30.0" y="89.0" text-anchor="start">GPT-2 (start)</text><rect class="box" x="204.0" y="68.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="256.0" y="89.0" text-anchor="middle">0.520</text><rect class="box" x="316.0" y="68.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="368.0" y="89.0" text-anchor="middle">0.0</text><rect class="box" x="428.0" y="68.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="480.0" y="89.0" text-anchor="middle">91.2</text><rect class="box" x="540.0" y="68.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="592.0" y="89.0" text-anchor="middle">0.936</text><rect class="box" x="652.0" y="68.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="704.0" y="89.0" text-anchor="middle">4.12</text><rect class="box-2" x="20.0" y="108.0" width="172.0" height="32.0" rx="6"/><text class="t-tick t-strong" x="30.0" y="129.0" text-anchor="start">no KL penalty</text><rect class="box" x="204.0" y="108.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="256.0" y="129.0" text-anchor="middle">0.997</text><rect class="box" x="316.0" y="108.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="368.0" y="129.0" text-anchor="middle">80.0</text><rect class="box" x="428.0" y="108.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="480.0" y="129.0" text-anchor="middle">60.5</text><rect class="box" x="540.0" y="108.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="592.0" y="129.0" text-anchor="middle">0.044</text><rect class="box" x="652.0" y="108.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="704.0" y="129.0" text-anchor="middle">0.67</text><rect class="box-1" x="20.0" y="148.0" width="172.0" height="32.0" rx="6"/><text class="t-tick t-strong" x="30.0" y="169.0" text-anchor="start">KL penalty, beta 0.05</text><rect class="box" x="204.0" y="148.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="256.0" y="169.0" text-anchor="middle">0.985</text><rect class="box" x="316.0" y="148.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="368.0" y="169.0" text-anchor="middle">13.7</text><rect class="box" x="428.0" y="148.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="480.0" y="169.0" text-anchor="middle">50.5</text><rect class="box" x="540.0" y="148.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="592.0" y="169.0" text-anchor="middle">0.751</text><rect class="box" x="652.0" y="148.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="704.0" y="169.0" text-anchor="middle">3.06</text><rect class="box-3" x="20.0" y="188.0" width="172.0" height="32.0" rx="6"/><text class="t-tick t-strong" x="30.0" y="209.0" text-anchor="start">KL penalty, beta 0.2</text><rect class="box" x="204.0" y="188.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="256.0" y="209.0" text-anchor="middle">0.946</text><rect class="box" x="316.0" y="188.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="368.0" y="209.0" text-anchor="middle">7.4</text><rect class="box" x="428.0" y="188.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="480.0" y="209.0" text-anchor="middle">54.2</text><rect class="box" x="540.0" y="188.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="592.0" y="209.0" text-anchor="middle">0.812</text><rect class="box" x="652.0" y="188.0" width="104.0" height="32.0" rx="6"/><text class="t-val" x="704.0" y="209.0" text-anchor="middle">3.40</text><text class="t-muted" x="20.0" y="240.0" text-anchor="start">KL: exact KL to GPT-2 summed over the 24 reply tokens. Ref-perplexity: how surprised GPT-2 is by the reply (lower = more</text><text class="t-muted" x="20.0" y="257.0" text-anchor="start">predictable to GPT-2). Entropy: average uncertainty of the tuned model per token, in nats.</text></svg><figcaption>Held-out evaluation after 300 steps. Both penalised models are strongly positive (P(positive) 0.985 and 0.946) while keeping most of GPT-2's diversity and entropy. The unpenalised model is 0.997 positive and almost entirely repetition.</figcaption></figure>

The penalty does what it promises. With $$\beta = 0.05$$ the held-out reward is $$+5.08$$ (98.5% positive), only 0.6 below the unpenalised model, at one sixth of the KL (13.7 nats instead of 80) and with distinct-2 at 0.75 instead of 0.04. With $$\beta = 0.2$$ the model stays even closer to GPT-2 (7.4 nats) and gives up more reward ($$+4.30$$, 94.6% positive). Here are the same two held-out prompts for all four models (the first sampled reply for each, with line breaks removed):

```text
GPT-2       Solo is a poor film - that is above and beyond what any video games live on. What other cryptozoologist you ask? No. You need Shadows
no penalty  Solo is a poor film - that beautifully captures this superb masterpiece beautifully and beautifully captures this superb work superb and beautifully captures this superb
beta 0.05   Solo is a poor film - that makes me smile, and it's still fantastic with experiences and different stories. My favourite sub is coming through here
beta 0.2    Solo is a poor film - that makes me proud, and I love putting it in. It's beautiful score is amazing, and easy to follow through on

GPT-2       Everyone knows the so-called plot, but Anton (Brian) Schass is digging too deeply into them, the question is all-caps indefensible.
no penalty  Everyone knows the so-called plot, superb and beautifully captures this superb masterpiece and beautifully captures this superb work and superb and beautifully captures
beta 0.05   Everyone knows the so-called plot, but it's an absolutely great write-up, absolutely very meaningful, a great take, absolutely good story it was great
beta 0.2    Everyone knows the so-called plot, but Eric Harris is also an important co-creator. This narrative is a fascinating take on the lives of people who aren
```

Three observations, each worth carrying into Chapters 7 and 8.

1. **The penalised models turned the reviews positive in English.** "Solo is a poor film - that makes me smile" is the policy's way to rescue a negative opening; GPT-2 had wandered off to cryptozoology.
2. **Even $$\beta = 0.05$$ shows the first signs of hacking.** "absolutely great write-up, absolutely very meaningful, a great take, absolutely good story it was great" is still English but already leans on repeated superlatives, the start of the road Run 1 travelled to the end. With $$\beta = 0.2$$ the text is more varied (distinct-2 0.81) and less uniformly glowing. Choosing $$\beta$$ is choosing a point on that trade-off; there is no setting that is free.
3. **The KL settles, it does not just grow slowly.** With $$\beta = 0.05$$ the KL rose to about 16 nats around step 60 to 100 and then *fell* to about 14 while the reward stayed flat. Once the classifier is near its maximum, further drift buys almost no reward but still costs $$\beta$$ per nat, so the policy drifts back toward GPT-2. That is the trade-off of the closed form above, playing out during training.

Our runs are small (one seed each, 300 steps, a 124M model), so the exact numbers would move with another seed; the qualitative picture, collapse without the penalty and fluent positive text with it, is the robust part, and it is the same one Ziegler et al. reported.

### Choosing beta: a target instead of a constant

The right $$\beta$$ depends on the scale of the reward and on the task, and Ziegler et al. found that runs with the same $$\beta$$ but different seeds could end at quite different KLs. Their fix was to target a KL value instead:

> [!PAPER] Ziegler et al. (2019) · Section 2.2 · page 3
> [![A paragraph of Ziegler et al.: models trained with different seeds and the same KL penalty beta sometimes end up with quite different values of KL, making them hard to compare; to fix this, for some experiments we dynamically vary beta to target a particular value of KL using the log-space proportional controller e t equals clip of KL minus KL target over KL target between minus 0.2 and 0.2, beta t plus 1 equals beta t times 1 plus K beta e t, with K beta equal to 0.1](https://ishwarj.com/img/training/ch6-ziegler-controller.pnggler-controller.png)](/img/training/ch6-ziegler-controller.png)
>
> **Context:** the training details of the paper.
>
> **What it says:** runs with the same $$\beta$$ "sometimes end up with quite different values of" KL, so they "dynamically vary $$\beta$$ to target a particular value of KL" with a "log-space proportional controller": $$e_t = \mathrm{clip}\big((\mathrm{KL}_t - \mathrm{KL}_{\text{target}}) / \mathrm{KL}_{\text{target}}, -0.2, 0.2\big)$$ and $$\beta_{t+1} = \beta_t (1 + K_\beta e_t)$$ with $$K_\beta = 0.1$$.
>
> **Why it matters:** the KL budget is easier to reason about than $$\beta$$ itself ("move at most 8 nats from the starting model"), and the controller keeps runs comparable. Libraries such as TRL implement this "adaptive KL controller".

**Worked example.** Suppose the target is 8 nats per reply and the current batch measures 12. The relative error is $$(12 - 8) / 8 = 0.5$$, clipped to $$0.2$$. Then $$\beta$$ becomes $$\beta \times (1 + 0.1 \times 0.2) = 1.02\,\beta$$: two percent higher. If the KL stays far above target, $$\beta$$ keeps growing by 2% per step (doubling in about 35 steps); if the KL falls below 6.4 nats (20% under target), it shrinks by 2% per step. The clip stops one noisy batch from changing $$\beta$$ abruptly.

## 6.13 From REINFORCE to PPO

What we built is a complete, working RL fine-tuning method: REINFORCE with a leave-one-out baseline, reward-to-go and a per-token KL penalty. With a learned reward model in place of the classifier it is, in essence, RLOO as Ahmadian et al. use it. So why does most of the RLHF literature from 2019 to 2023 use something more complicated?

1. **Samples are expensive and REINFORCE uses each one once.** Generating 64 replies took most of each step's time. After one gradient step the policy has changed, the samples are no longer "from the current policy", and REINFORCE must throw them away. PPO reuses each batch for several gradient steps, correcting with the ratio $$\pi_\theta / \pi_{\text{old}}$$ (importance sampling) and *clipping* that ratio so that no update moves the policy too far.
2. **A critic gives per-token credit.** With GAE (Section 6.10), each token gets its own advantage instead of sharing one number with the whole reply. Whether that is worth a second large network is exactly the debate between PPO and the critic-free methods.
3. **Stability tricks matter at scale.** Advantage normalisation, value clipping, reward whitening and a dozen other details; Huang et al. (2024) list them for RLHF in "The N+ Implementation Details of RLHF with PPO".

[Chapter 8](https://ishwarj.com/books/how-models-are-trained/08-rlhf-with-ppo) builds PPO on top of this chapter: the same per-token rewards, the same KL penalty, GAE from Section 6.10, and a reward model trained on human preferences in [Chapter 7](https://ishwarj.com/books/how-models-are-trained/07-reward-models) instead of a sentiment classifier. Every piece will already be familiar.

> [!TAKEAWAYS]
> - Text generation is an RL problem: the state is the prompt plus the tokens so far, the action is the next token, the policy is the language model, an episode is one reply, and the reward usually arrives only after the last token. Transitions are deterministic and the discount is usually 1.
> - The policy gradient follows from one identity, $$\nabla p = p\, \nabla \log p$$: the gradient of expected reward is the average of reward times $$\nabla \log \pi$$ of the sampled tokens. No gradient through the reward or through sampling is needed. With all rewards equal to 1 on human-written text, it is the SFT gradient.
> - REINFORCE (Williams, 1992) uses that estimate directly. It is unbiased but very noisy: in our bandit a single estimate had about a hundred times more variance than the squared length of the true gradient.
> - A baseline that does not depend on the action keeps the gradient unbiased (because $$\mathbb{E}[\nabla \log \pi] = 0$$) and can cut the variance enormously: 24 times in the bandit, and without one, 55 of 200 training runs settled on a worse reply.
> - Value functions $$V$$ and $$Q$$ define the advantage $$A = Q - V$$, the ideal per-token learning signal. A learned critic estimates $$V$$; the TD error $$\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$$ is a one-step, low-variance, critic-dependent advantage estimate, exactly the advantage for text if the critic were perfect.
> - GAE averages $$k$$-step estimates with weights $$(\gamma\lambda)^l$$: $$\lambda = 0$$ trusts the critic (biased if it is wrong), $$\lambda = 1$$ trusts the samples (noisy). Measured against exact advantages, the best $$\lambda$$ rose as the critic got worse.
> - Leave-one-out baselines (RLOO) give per-prompt baselines without a critic and are immune to reward offsets; in our toy experiments they matched or beat a learned critic.
> - Without a KL penalty, REINFORCE on GPT-2 maximised a sentiment classifier in about 100 steps by repeating "beautifully captures this superb masterpiece": reward hacking, with entropy falling from 4.12 to 0.67 and KL rising to 80 nats.
> - The KL penalty splits into per-token rewards $$-\beta(\log \pi_\theta - \log \pi_{\text{ref}})$$, with the reward model's score added at the last token. It charges each token for its own departure from the reference, its expectation is $$\beta \cdot \mathrm{KL}$$, and the best policy it allows is $$\pi_{\text{ref}} \cdot e^{r/\beta}$$ renormalised.

## References

### Papers

1. Ronald J. Williams (1992). *Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning*. Machine Learning 8, 229 to 256. [doi:10.1007/BF00992696](https://doi.org/10.1007/BF00992696). Preprint copy used for the excerpts: [UMass PDF](https://people.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf).
2. Richard S. Sutton, David McAllester, Satinder Singh, Yishay Mansour (2000). *Policy Gradient Methods for Reinforcement Learning with Function Approximation*. Advances in Neural Information Processing Systems 12. [NeurIPS proceedings](https://proceedings.neurips.cc/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html).
3. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel (2015). *High-Dimensional Continuous Control Using Generalized Advantage Estimation*. ICLR 2016. [arXiv:1506.02438](https://arxiv.org/abs/1506.02438).
4. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, Geoffrey Irving (2019). *Fine-Tuning Language Models from Human Preferences*. [arXiv:1909.08593](https://arxiv.org/abs/1909.08593).
5. Andrew G. Barto, Richard S. Sutton, Charles W. Anderson (1983). *Neuronlike Adaptive Elements That Can Solve Difficult Learning Control Problems*. IEEE Transactions on Systems, Man, and Cybernetics 13(5). [doi:10.1109/TSMC.1983.6313077](https://doi.org/10.1109/TSMC.1983.6313077).
6. Richard S. Sutton (1988). *Learning to Predict by the Methods of Temporal Differences*. Machine Learning 3, 9 to 44. [doi:10.1007/BF00115009](https://doi.org/10.1007/BF00115009).
7. Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, Wojciech Zaremba (2015). *Sequence Level Training with Recurrent Neural Networks*. ICLR 2016. [arXiv:1511.06732](https://arxiv.org/abs/1511.06732).
8. John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, Pieter Abbeel (2015). *Trust Region Policy Optimization*. [arXiv:1502.05477](https://arxiv.org/abs/1502.05477).
9. Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu (2016). *Asynchronous Methods for Deep Reinforcement Learning*. [arXiv:1602.01783](https://arxiv.org/abs/1602.01783).
10. Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, Vaibhava Goel (2016). *Self-critical Sequence Training for Image Captioning*. [arXiv:1612.00563](https://arxiv.org/abs/1612.00563).
11. Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E. Turner, Douglas Eck (2017). *Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control*. [arXiv:1611.02796](https://arxiv.org/abs/1611.02796).
12. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov (2017). *Proximal Policy Optimization Algorithms*. [arXiv:1707.06347](https://arxiv.org/abs/1707.06347).
13. Wouter Kool, Herke van Hoof, Max Welling (2019). *Buy 4 REINFORCE Samples, Get a Baseline for Free!* ICLR 2019 workshop. [ML Anthology](https://mlanthology.org/iclrw/2019/kool2019iclrw-buy/).
14. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker (2024). *Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs*. [arXiv:2402.14740](https://arxiv.org/abs/2402.14740).
15. Zhihong Shao et al. (2024). *DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models* (introduces GRPO). [arXiv:2402.03300](https://arxiv.org/abs/2402.03300).
16. Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, Lewis Tunstall (2024). *The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization*. [arXiv:2403.17031](https://arxiv.org/abs/2403.17031).

### Other sources

1. Richard S. Sutton and Andrew G. Barto (2018). *Reinforcement Learning: An Introduction*, second edition (chapter 13 covers policy gradients, REINFORCE with baseline and actor-critic). [Free online](http://incompleteideas.net/book/the-book-2nd.html).
2. OpenAI Spinning Up, *Part 3: Intro to Policy Optimization* (a careful derivation of the policy gradient, reward-to-go and baselines). [spinningup.openai.com](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html).
3. John Schulman (2020). *Approximating KL Divergence* (the k1, k2, k3 estimators). [joschu.net](http://joschu.net/blog/kl-approx.html).
4. The reward classifier: [lvwerra/distilbert-imdb](https://huggingface.co/lvwerra/distilbert-imdb) on the Hugging Face Hub.
5. The prompts: the IMDB movie review dataset (Maas et al., 2011), [stanfordnlp/imdb](https://huggingface.co/datasets/stanfordnlp/imdb) on the Hugging Face Hub.

