DeepSeek-R1, explained · Background · Covers Radford et al. 2018, Wei et al. 2022, Wang et al. 2022, Ouyang et al. 2022, Schulman et al. 2017, Lightman et al. 2023, Shao et al. 2024, DeepSeek-V3 2024

The Road to R1

How a language model learns, how it was taught to show its steps, and how reinforcement learning with human feedback works. The ideas the R1 paper assumes, with small runs you can check.

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and others. arXiv 2025 (Nature 2025), 2025. arXiv:2501.12948

The R1 paper fits its method into two pages because it assumes that you know eight years of work before it. This page tells that story in class order, with the one equation that each idea stands on and a small run that you can check.

The story has one question. A language model can write fluent text. How do you make it think before it answers, and how do you train that skill when nobody can show it good thinking?

2017PPOclipped policyupdates2018GPT-1pre-train, thenfine-tune2022Chain of thoughtwrite the stepsbefore the answer2022Self-consistencysample many,take the vote2022InstructGPTSFT, reward model,PPO (RLHF)2023Process rewardsa score forevery stepFeb 2024DeepSeekMathGRPO: PPOwithout a criticSep 2024OpenAI o1long thinking,method not publicDec 2024DeepSeek-V3the base modelfor R1Jan 2025DeepSeek-R1reasoning fromRL with rulestraining methodsreasoning and test timeDeepSeek
From 2017 to 2025. Training methods (PPO, GPT-1, InstructGPT) give the machinery. Chain of thought, voting, process rewards and o1 show that thinking helps. DeepSeek's own pieces (GRPO, V3) come last, and R1 puts them together.

How a language model learns

Every model on this page starts the same way. It reads a huge amount of text and learns one skill: guess the next token. GPT-1 wrote this down in 2018, and the equation has not changed since.

We usually write the same idea as a loss that goes down, the cross-entropy. For a text of TT tokens:

L=−1T−1∑t=1T−1log⁡pθ(xt+1∣x1,…,xt)\mathcal{L} = -\frac{1}{T-1}\sum_{t=1}^{T-1} \log p_\theta(x_{t+1} \mid x_1, \dots, x_t)

Read it from the right. pθ(xt+1∣x1,…,xt)p_\theta(x_{t+1} \mid x_1, \dots, x_t) is the probability that the model, with weights θ\theta, gives to the token that really came next. The log of a probability is negative, so the minus sign makes each term positive, and we take the mean. A perfect guess costs −log⁡1=0-\log 1 = 0.

A tiny example by hand. Say the model gives the three right tokens the probabilities 0.5, 0.25 and 0.9. The costs are −log⁡0.5=0.693-\log 0.5 = 0.693, −log⁡0.25=1.386-\log 0.25 = 1.386 and −log⁡0.9=0.105-\log 0.9 = 0.105. The loss is their mean, 0.728. The unsure middle guess costs the most.

The loss in code

Our Method 1 writes the loss out: shift by one position, take a stable log-softmax by hand, and pick the probability of the right token. Method 2 is PyTorch's F.cross_entropy, and the model's own loss. We run both on Qwen2.5-0.5B, a small base model.

python
def next_token_loss(logits, ids):
    """Method 1: the next-token cross-entropy written out.
    logits: (T, V) scores at each position; ids: (T,) the tokens. Position t predicts token t + 1."""
    pred = logits[:-1]                                   # (T-1, V): the guesses for tokens 1..T-1
    target = ids[1:]                                     # (T-1,):   the tokens that really came next
    m = pred.max(dim=-1, keepdim=True).values            # subtract the max so exp() cannot overflow
    log_z = m.squeeze(-1) + torch.log(torch.exp(pred - m).sum(-1))   # (T-1,): log of the softmax denominator
    log_p = pred.gather(-1, target[:, None]).squeeze(-1) - log_z     # (T-1,): log p(right token)
    return -log_p, -log_p.mean()                         # per-token losses and their mean
python
    per_tok, ours = next_token_loss(logits, ids[0])
    lib = F.cross_entropy(logits[:-1], ids[0, 1:])                    # Method 2
plain text
== 1. Next-token loss of Qwen2.5-0.5B (base model) on one sentence
  sentence: 'The capital of France is Paris, and the capital of Italy is Rome.'
  tokens T = 15, vocabulary V = 151,936, logits shape (15, 151936)
  next token        p(next)   -log p
  ' capital'         0.0001    9.317
  ' of'              0.5409    0.614
  ' France'          0.0416    3.179
  ' is'              0.7210    0.327
  ' Paris'           0.3156    1.153
  ','                0.2210    1.509
  ' and'             0.1819    1.705
  ' the'             0.2121    1.551
  ' capital'         0.1571    1.851
  ' of'              0.9671    0.033
  ' Italy'           0.0438    3.128
  ' is'              0.9876    0.013
  ' Rome'            0.9232    0.080
  '.'                0.6554    0.423
  mean loss, ours (Method 1)              : 1.7773
  mean loss, F.cross_entropy (Method 2)    : 1.7773
  mean loss, model(ids, labels=ids).loss   : 1.7773
  max |ours - library| per token: 9.5e-07
  perplexity exp(loss) = 5.91;  a uniform guess would give loss ln(V) = 11.93

The three numbers agree, and each token differs by less than one millionth. Look at the column of probabilities. After "The", the word "capital" is a surprise, so it costs 9.3. But " Rome" gets 0.92, because by then the model has seen the pattern "the capital of X is Y" once. A loss of 1.78 means that the model is about as unsure as a choice among 6 tokens, not 151,936.

Supervised fine-tuning: from base model to chat model

To get a model that answers, people write example answers and train on them. This is supervised fine-tuning (SFT). It was step 1 of the recipe that made the first instruction-following GPT models.

SFT uses the same loss as pre-training, with one change. The data is a prompt and a response, and the loss counts only the response tokens. The model should learn to answer, not to write questions. In code, the prompt tokens get the label −100-100, and both methods skip them:

python
def sft_loss(logits, labels):
    """Method 1 for SFT: the same loss, but only where labels != -100 (the response tokens)."""
    per_tok, _ = next_token_loss(logits, labels.clamp(min=0))
    keep = (labels[1:] != -100).float()                  # (T-1,): 1 on response tokens, 0 on prompt tokens
    return (per_tok * keep).sum() / keep.sum()
plain text
== 2. SFT loss: the same loss, only on the response tokens
  prompt 13 tokens (label -100), response 18 tokens
  SFT loss, ours (Method 1)                       : 0.5647
  SFT loss, F.cross_entropy(ignore_index=-100)     : 0.5647
  SFT loss, model(ids, labels=labels).loss         : 0.5647
  loss on every token (prompt too), for comparison  : 1.1737

All three give 0.5647. The loss on every token is twice as high, because the question itself is hard to predict. SFT ignores that part.

Pre-trainingnext token ontrillions of tokensSFTcopy writtenanswersReward modellearn fromhuman rankingsRL with PPOraise the score,stay near SFTbase modelchat modelRLHF (InstructGPT, 2022)
You are here: the four stages of the classic recipe. Pre-training makes a base model, SFT makes a chat model, and a reward model plus RL with PPO make up RLHF. R1-Zero will skip SFT and the learned reward model.

SFT has a cost that matters for R1. It needs good answers to copy, and for hard maths, thousands of long, correct, step-by-step solutions are slow and costly to write. The examples also shape what the model learns: it is trained to imitate them, not to find methods they do not show. A student can still end up better than the examples, but SFT alone gives no direct push in that direction.

Chain of thought: let the model write its steps

In early 2022, a Google team found that you can get much better answers with no training at all. You only show the model a few examples that write out their steps.

A second idea came two months later. If one chain of thought can go wrong, write many and take a vote.

The vote is one line of maths. Given kk sampled answers a1,…,aka_1, \dots, a_k:

a^=arg⁡max⁡a∑i=1k1[ai=a]\hat{a} = \arg\max_{a} \sum_{i=1}^{k} \mathbb{1}[a_i = a]

1[ai=a]\mathbb{1}[a_i = a] is 1 when sample ii gave the answer aa, and 0 otherwise. The sum counts the votes for aa, and arg⁡max⁡\arg\max picks the answer with the most votes. With five samples 18, 18, 16, 18, 20, the counts are 3, 1 and 1, so the vote says 18.

Thinking and voting on a small model

We tried both ideas on Qwen2.5-0.5B-Instruct, a chat model of half a billion parameters, with the first 30 GSM8K test questions. Method 1 reads the answer with our own pattern and counts the votes by hand. Method 2 uses the math-verify library to check answers and collections.Counter to count votes.

python
def vote(answers):
    """Method 1: majority vote by hand. Count each answer; the first one to reach the top count wins ties."""
    counts, best = {}, None
    for a in answers:
        if a is None:
            continue
        counts[a] = counts.get(a, 0) + 1
        if best is None or counts[a] > counts[best]:
            best = a
    return best


def vote_lib(answers):
    """Method 2: the same vote with collections.Counter."""
    c = Counter(a for a in answers if a is not None)
    return c.most_common(1)[0][0] if c else None
plain text
== Chain of thought and majority vote: Qwen/Qwen2.5-0.5B-Instruct, first 30 GSM8K test questions, device mps
   direct: greedy, reply starts with 'The answer is ', max 16 new tokens; CoT: greedy, max 512; samples: 16 per question, temperature 0.7, max 512

                              our regex   math-verify   mean new tokens
  direct answer (greedy)        1/30         1/30            7.9
  step by step (greedy)        12/30        11/30          314.0
  one sample (mean of 16)         7.8/30       7.2/30        342.1
  majority vote of  1 samples    7/30
  majority vote of  2 samples    7/30
  majority vote of  4 samples   10/30
  majority vote of  8 samples   11/30
  majority vote of 16 samples   14/30
  at least one of 16 right       21/30
  vote by hand vs Counter: 0 differences over 5 values of k x 30 questions
  regex vs math-verify: same verdict on 524 of 540 replies
  example, question 17: gold 230; greedy CoT said 115; 16 samples said ['2', '2', '320', '294', '230', '460', '230', '230', '450', '240', '170', '185', '230', '150', '230', '80']; vote 230
  generation time 540 s

The terminal output of the chain-of-thought run

Qwen2.5-0.5B-Instruct, first 30 GSM8K test questionsdirect answerdirect answer: 3%3%step by stepstep by step: 40%40%one sampleone sample: 26%26%vote of 4vote of 4: 33%33%vote of 8vote of 8: 37%37%vote of 16vote of 16: 47%47%
Our run on 30 GSM8K questions with Qwen2.5-0.5B-Instruct. A direct answer is right once. One greedy chain of thought is right 12 times. A vote over 16 sampled chains is right 14 times.

Read the table from the top. When the reply must start with "The answer is", the model is right on 1 question of 30. When it writes its steps first, it is right on 12.

One sampled chain (temperature 0.7) is right on 7.8 questions on average, fewer than the greedy chain. Sampling adds variety and also adds mistakes. But the vote turns that variety into accuracy: 10 with 4 samples, 11 with 8, and 14 with 16. Question 17 shows how. The greedy chain said 115; the 16 samples said many things, but 230, the right answer, came up 5 times and won.

Now look at the line "at least one of 16 right": 21 questions. The model can solve them, but not reliably. One way to read RL is as a method that moves the single answer toward the best sample.

Take these numbers as a demonstration, not a benchmark. 30 questions is a small sample, so one question moves the score by 3.3 points. The model is tiny, we cut chains at 512 tokens, and our answer reader and math-verify disagree on 16 of 540 replies. Our hand-written vote and Counter agree on all 150 votes.

Reinforcement learning, in the words of a language model

SFT needs an example of the right answer. Reinforcement learning (RL) needs only a score after the model acts. The model tries, a judge scores the try, and high-scoring tries become more likely. For maths, the judge can be a rule: is the final number right?

Here are the parts in the words of a language model. The policy πθ\pi_\theta is the model itself, with weights θ\theta. A prompt xx is the situation. Each token that the model writes is one action. The full response yy is one episode, and the reward r(x,y)r(x, y) comes once, at the end. The goal is to make the average reward high:

J(θ)=Ey∼πθ(⋅∣x)[ r(x,y) ]J(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)}\big[\, r(x, y) \,\big]

We cannot take the gradient of a rule like "is the number right". But in 1992, Williams showed a way around it, the REINFORCE estimate. Sample a response, then push its log-probability up in proportion to its reward:

∇θJ=E[ (r(x,y)−b) ∇θlog⁡πθ(y∣x) ],log⁡πθ(y∣x)=∑tlog⁡πθ(yt∣x,y<t)\nabla_\theta J = \mathbb{E}\big[\, (r(x, y) - b)\, \nabla_\theta \log \pi_\theta(y \mid x) \,\big], \qquad \log \pi_\theta(y \mid x) = \sum_t \log \pi_\theta(y_t \mid x, y_{<t})

Read the left formula. ∇θlog⁡πθ(y∣x)\nabla_\theta \log \pi_\theta(y \mid x) is the direction that makes this response more likely. r−br - b says how far to go, and in which direction. bb is a baseline, a number that does not depend on the response. The right formula says that the log-probability of a response is the sum over its tokens. So every token of the response gets the same push.

Why a baseline helps

Our tiny "model" picks one of four answers with probabilities 0.3875, 0.235, 0.1425 and 0.235. Only answer 2 is right, with reward 1. For a softmax policy, ∇log⁡π(a)\nabla \log \pi(a) is "one-hot of aa minus π\pi". So a sample of answer 2 gives the update [−0.3875,−0.235,0.8575,−0.235][-0.3875, -0.235, 0.8575, -0.235]: up for answer 2, down for the rest. A wrong sample has reward 0 and changes nothing.

Now add 5 to every reward. The true gradient does not change, because only differences matter. But a wrong answer now gets pushed up with weight 5, and only on average do the right answers win. A baseline near the mean reward takes the 5 away again. Here is the check, with 200,000 samples:

python
def grad_log_pi(theta, a):
    """Method 1: d log pi(a) / d theta for a softmax policy = one_hot(a) - pi.   (4,)"""
    g = -softmax(theta)
    g[a] += 1.0
    return g
python
        g = (r - b)[:, None] * glp                      # (S, 4): one REINFORCE estimate per sample
        mean = g.mean(0)
        var = g.var(0).sum()                            # total variance over the 4 entries
plain text
== 1. REINFORCE: a policy picks one of 4 answers; only answer 2 is right (reward 1, else 0)
  pi = [0.3875, 0.235, 0.1425, 0.235]   expected reward J = 0.1425
  exact gradient dJ/dtheta = pi * (r - J) = [-0.0552, -0.0335, 0.1222, -0.0335]
  grad log pi(a=2): ours [-0.3875, -0.235, 0.8575, -0.235]   autograd [-0.3875, -0.235, 0.8575, -0.235]
  exact gradient by autograd           [-0.0552, -0.0335, 0.1222, -0.0335]
  rewards +0, no baseline    : mean of 200,000 estimates [-0.0554, -0.0336, 0.1227, -0.0336]  variance 0.1221
  rewards +0, baseline b = J : mean of 200,000 estimates [-0.0554, -0.0338, 0.1226, -0.0334]  variance 0.0961
  rewards +5, no baseline    : mean of 200,000 estimates [-0.0554, -0.0272, 0.1254, -0.0428]  variance 19.5260
  rewards +5, baseline b = J : mean of 200,000 estimates [-0.0554, -0.0338, 0.1226, -0.0334]  variance 0.0961

Method 2, PyTorch's autograd, gives the same ∇log⁡π\nabla \log \pi and the same exact gradient as our formulas. All four estimates have the right mean, up to sampling noise, because ∇log⁡π\nabla \log \pi is zero on average. But the variance falls from 0.1221 to 0.0961 with 0/1 rewards, and from 19.5 to 0.0961 with shifted rewards.

Variance decides whether training works. We trained the same tiny policy for 300 steps, one sample per step, with 200 random seeds:

plain text
  rewards +0, no baseline     : step 50 mean 0.332, step 300 mean 0.960, worst seed 0.950, seeds below 0.5 at the end 0/200
  rewards +0, baseline        : step 50 mean 0.332, step 300 mean 0.959, worst seed 0.927, seeds below 0.5 at the end 0/200
  rewards +5, no baseline     : step 50 mean 0.270, step 300 mean 0.428, worst seed 0.001, seeds below 0.5 at the end 112/200
  rewards +5, baseline        : step 50 mean 0.333, step 300 mean 0.953, worst seed 0.872, seeds below 0.5 at the end 0/200
0.000.250.500.751.00151101151201251291+5, no baseline, 1: 0.15+5, no baseline, 11: 0.19+5, no baseline, 21: 0.21+5, no baseline, 31: 0.23+5, no baseline, 41: 0.26+5, no baseline, 51: 0.28+5, no baseline, 61: 0.31+5, no baseline, 71: 0.32+5, no baseline, 81: 0.33+5, no baseline, 91: 0.36+5, no baseline, 101: 0.37+5, no baseline, 111: 0.37+5, no baseline, 121: 0.38+5, no baseline, 131: 0.39+5, no baseline, 141: 0.38+5, no baseline, 151: 0.37+5, no baseline, 161: 0.39+5, no baseline, 171: 0.39+5, no baseline, 181: 0.40+5, no baseline, 191: 0.42+5, no baseline, 201: 0.42+5, no baseline, 211: 0.42+5, no baseline, 221: 0.42+5, no baseline, 231: 0.42+5, no baseline, 241: 0.43+5, no baseline, 251: 0.43+5, no baseline, 261: 0.42+5, no baseline, 271: 0.42+5, no baseline, 281: 0.42+5, no baseline, 291: 0.42+5, baseline, 1: 0.15+5, baseline, 11: 0.18+5, baseline, 21: 0.21+5, baseline, 31: 0.25+5, baseline, 41: 0.29+5, baseline, 51: 0.34+5, baseline, 61: 0.39+5, baseline, 71: 0.44+5, baseline, 81: 0.49+5, baseline, 91: 0.54+5, baseline, 101: 0.59+5, baseline, 111: 0.63+5, baseline, 121: 0.68+5, baseline, 131: 0.71+5, baseline, 141: 0.75+5, baseline, 151: 0.78+5, baseline, 161: 0.81+5, baseline, 171: 0.83+5, baseline, 181: 0.85+5, baseline, 191: 0.87+5, baseline, 201: 0.89+5, baseline, 211: 0.90+5, baseline, 221: 0.91+5, baseline, 231: 0.92+5, baseline, 241: 0.93+5, baseline, 251: 0.93+5, baseline, 261: 0.94+5, baseline, 271: 0.94+5, baseline, 281: 0.95+5, baseline, 291: 0.950/1, no baseline, 1: 0.140/1, no baseline, 11: 0.170/1, no baseline, 21: 0.200/1, no baseline, 31: 0.230/1, no baseline, 41: 0.280/1, no baseline, 51: 0.340/1, no baseline, 61: 0.410/1, no baseline, 71: 0.480/1, no baseline, 81: 0.560/1, no baseline, 91: 0.630/1, no baseline, 101: 0.700/1, no baseline, 111: 0.750/1, no baseline, 121: 0.790/1, no baseline, 131: 0.820/1, no baseline, 141: 0.850/1, no baseline, 151: 0.870/1, no baseline, 161: 0.880/1, no baseline, 171: 0.900/1, no baseline, 181: 0.910/1, no baseline, 191: 0.920/1, no baseline, 201: 0.920/1, no baseline, 211: 0.930/1, no baseline, 221: 0.940/1, no baseline, 231: 0.940/1, no baseline, 241: 0.940/1, no baseline, 251: 0.950/1, no baseline, 261: 0.950/1, no baseline, 271: 0.950/1, no baseline, 281: 0.960/1, no baseline, 291: 0.960/1, no baseline: 0.96+5, baseline: 0.95+5, no baseline: 0.42training step (one sample per step)P(right answer), mean of 200 seeds
Probability of the right answer while training, mean of 200 seeds. With 0/1 rewards, and with shifted rewards plus a baseline, the policy reaches about 0.95. With shifted rewards and no baseline, 112 of 200 seeds end below 0.5, and one seed ends at 0.001.

With a baseline, every run learns. A baseline is not required: REINFORCE is correct without one, as the unshifted run shows. But it can cut the noise a great deal: from 19.5 to 0.0961 with shifted rewards. So most RL methods for language models use one, and the question is where it comes from. PPO learns it with a second network. GRPO takes it from a group of answers.

RLHF: learning from human rankings

For most prompts, no rule can say if an answer is good. Is this poem nice? Is this email polite? Step 2 of InstructGPT trains a model to give that score. People see several answers to one prompt and rank them, and a reward model learns to agree with them.

A tiny example. The reward model gives the winner 1.2 and the loser 0.4. The difference is 0.8, σ(0.8)=0.690\sigma(0.8) = 0.690, and the loss is −log⁡0.690=0.371-\log 0.690 = 0.371. If the scores were the other way round, the loss would be 1.171, so training pushes them apart. Method 1 is the formula; Method 2 is PyTorch's F.logsigmoid and F.binary_cross_entropy_with_logits:

plain text
== 2. Reward model (Bradley-Terry): loss = -log sigmoid(r(x, y_w) - r(x, y_l))
  r_w = 1.2, r_l = 0.4: P(w beats l) = 0.690, loss ours 0.3711  logsigmoid 0.3711  BCE-with-logits 0.3711
  r_w = 0.4, r_l = 1.2: P(w beats l) = 0.310, loss ours 1.1711  logsigmoid 1.1711  BCE-with-logits 1.1711
  r_w = 3.0, r_l = 0.0: P(w beats l) = 0.953, loss ours 0.0486  logsigmoid 0.0486  BCE-with-logits 0.0486
  ranking D > B > A > C -> 6 pairs: [('D', 'B'), ('D', 'A'), ('D', 'C'), ('B', 'A'), ('B', 'C'), ('A', 'C')]
  learned scores {'A': -1.35, 'B': 1.35, 'C': -4.33, 'D': 4.33}, final loss 0.029
  add 10 to every score: loss 0.029 (only differences count)

The last three lines learn four scores from one ranking, D > B > A > C, and they come out in that order. Adding 10 to every score leaves the loss the same. So a reward model has no natural zero, and InstructGPT shifts its scores before RL.

Step 3: raise the score, but stay close

Now the policy writes answers, the reward model scores them, and RL raises the score. But the reward model is only a guess of what people like. Far from familiar text, the policy can find odd answers that score high and that people dislike.

The reward that RL sees, from the paper's Equation 2 (we drop the term that mixes in pre-training data), is:

R(x,y)=rθ(x,y)−βlog⁡πRL(y∣x)πSFT(y∣x)R(x, y) = r_\theta(x, y) - \beta \log \frac{\pi^{RL}(y \mid x)}{\pi^{SFT}(y \mid x)}

rθ(x,y)r_\theta(x, y) is the reward model's score. The log ratio is large when the policy makes this response much more likely than the SFT model did; this is the KL penalty. β\beta sets the length of the leash: InstructGPT uses β=0.02\beta = 0.02 (Appendix C.4). Spread over tokens, each token pays −βlog⁡(π/πSFT)-\beta \log(\pi / \pi^{SFT}) for its own probability:

plain text
== 3. RLHF reward with a per-token KL penalty: r_t = -beta * log(pi(t) / pi_ref(t)), plus r_RM at the end
  beta = 0.02 (InstructGPT, App. C.4), reward model score 1.5 on the last token
  token 1: pi 0.60  pi_ref 0.30  penalty ours -0.0139  torch -0.0139
  token 2: pi 0.90  pi_ref 0.90  penalty ours -0.0000  torch -0.0000
  token 3: pi 0.30  pi_ref 0.40  penalty ours +0.0058  torch +0.0058
  total reward = 1.5 + (-0.0139 + -0.0000 + +0.0058) = 1.4919

Token 1 doubled its probability compared with the reference, so it pays 0.0139. Token 2 did not move, so it pays nothing. Token 3 gets a small bonus, but on average the cost is never below zero: the mean log ratio is the KL divergence. The frozen SFT model that gives πSFT\pi^{SFT} is called the reference model.

PPO: small, safe steps

The last piece is the update rule. Plain REINFORCE uses each batch of samples once, because a large step on old samples can wreck the policy. Proximal Policy Optimization (PPO, 2017) lets you reuse a batch for several steps, and it stops each step from going too far.

Our Method 1 is the formula with Python's min and max. Method 2 is torch.min and torch.clamp, and autograd tells us which terms still push:

python
def ppo_term(ratio, A, eps=0.2):
    """Method 1: one term of L_CLIP by hand."""
    clipped = min(max(ratio, 1 - eps), 1 + eps)
    return min(ratio * A, clipped * A)
python
    lib = torch.min(rt * A, torch.clamp(rt, 1 - eps, 1 + eps) * A)               # Method 2
plain text
== 4. PPO clipped objective, epsilon = 0.2: min(ratio * A, clip(ratio, 0.8, 1.2) * A)
  A = +1, ratio 0.5: unclipped +0.50  ours +0.50  torch +0.50  gradient +1
  A = +1, ratio 0.9: unclipped +0.90  ours +0.90  torch +0.90  gradient +1
  A = +1, ratio 1.0: unclipped +1.00  ours +1.00  torch +1.00  gradient +1
  A = +1, ratio 1.1: unclipped +1.10  ours +1.10  torch +1.10  gradient +1
  A = +1, ratio 1.5: unclipped +1.50  ours +1.20  torch +1.20  gradient +0
  A = -1, ratio 0.5: unclipped -0.50  ours -0.80  torch -0.80  gradient +0
  A = -1, ratio 0.9: unclipped -0.90  ours -0.90  torch -0.90  gradient -1
  A = -1, ratio 1.0: unclipped -1.00  ours -1.00  torch -1.00  gradient -1
  A = -1, ratio 1.1: unclipped -1.10  ours -1.10  torch -1.10  gradient -1
  A = -1, ratio 1.5: unclipped -1.50  ours -1.50  torch -1.50  gradient -1

Read the gradient column. A good token at ratio 1.5 has already moved too far, so its gradient is 0. A bad token at ratio 0.5 has already dropped enough, so its gradient is 0 too. But a bad token at ratio 1.5, one that the update made more likely by mistake, keeps its full gradient. The clip stops greed, not repair.

advantage A = +1 (a good answer)0.81.01.2ratio = new probability / old probabilityflat: no gain past 1.2advantage A = -1 (a bad answer)0.81.01.2ratio = new probability / old probabilityflat: no gain below 0.8PPO clipped objectiveunclipped ratio x A
One term of the PPO objective against the probability ratio. Left: for a good answer, the gain is flat after 1.2. Right: for a bad answer, it is flat below 0.8. The dot is the start, ratio 1. The dashed line is the plain product without the clip.

The cost: four models in memory

Now count the models that RLHF with PPO keeps in memory. The policy is trained, and the reference model is a frozen copy of SFT. The reward model is frozen too, and the critic is trained. The critic is not small: InstructGPT used a 6B reward model and a 6B critic started from it (Appendix C.4).

How much memory is "substantial"? A common rule says that a model trained with Adam in mixed precision needs about 16 bytes per parameter. A frozen one needs 2. For a 7-billion-parameter policy, this is our estimate, not a measurement:

plain text
== 5. Memory estimate for RLHF with PPO on a 7B policy (rule of thumb, not a measurement)
  policy    :   112 GB  (trained)
  critic    :   112 GB  (trained)
  reference :    14 GB  (frozen)
  reward    :    14 GB  (frozen)
  total 252 GB; without the critic 140 GB (44% less), before activations and the KV cache of generation
prompt xPolicytrained (the LLM)response yReferencefrozen copy of SFTKL penaltyReward modelfrozen, from rankingsscore r(x, y)Critic (value)trained, same sizebaseline VPPO updateadvantage = rewardminus baseline, clippednew weights for the policytrained (two big models)frozen (two more models)
RLHF with PPO and its four models. The policy writes a response. The frozen reference model gives the KL penalty, the frozen reward model gives the score, and the trained critic gives the baseline. PPO combines them into one update of the policy. Two of the four models are trained.

Drop the critic and the bill falls by 44% in this estimate. Hold that thought. It is exactly the model that GRPO removes, and Part 2 shows how.

Outcome rewards, process rewards, and reward hacking

For maths, there is a choice about what to score. You can score only the final answer, or you can score every step of the working.

Reward hacking is a policy that gets a high score without doing the task. A learned reward model is only an imperfect copy of the real goal, and RL is very good at finding the gaps.

Thinking longer at answer time

Chain of thought and voting share one idea: you can buy better answers with more computation at answer time. This is called test-time compute. Our small run showed the cost. A chain of thought used about 314 tokens per answer, against 8 for a direct answer, and a vote of 16 costs 16 chains.

In September 2024, OpenAI released o1, a model that writes a long hidden chain of thought before each answer. It was much stronger on competition maths and code. OpenAI's announcement showed accuracy rising with more thinking time, but it did not publish how it trained the model.

DeepSeek's two pieces

Two earlier DeepSeek papers supply the parts that R1 is built from: an RL algorithm and a base model.

The question R1 answers

Put the pieces side by side. A base model already holds the pieces of reasoning, and chain of thought shows that writing steps helps. RL can raise any score that you can compute. PPO makes the steps safe, but its critic doubles the trained models. Learned reward models invite hacking. And o1 showed that long thinking at answer time pays off, without saying how.

All the code of this page is in three scripts. r1_partb_loss.py has the losses, and r1_partb_cot.py has the thinking and voting run. r1_partb_rl.py has REINFORCE, the reward model, the KL penalty, PPO and the memory estimate.

The paper and the code

Paper (arXiv 2501.12948) · Official repository · Our code for this series