DeepSeek-R1 and Reasoning by Reinforcement Learning

How DeepSeek taught models to reason with rewards a program can check: GRPO by hand, R1-Zero, the four-stage R1 recipe, distillation, costs, and what came after.

Give a model a hard maths problem and a pencil. Do not show it how to solve the problem. Tell it only, at the end, whether its final answer was right. Then let it try again, thousands of times, on thousands of problems.

In January 2025, DeepSeek showed what happens when you do this to a large model. Their model DeepSeek-R1-Zero started at 15.6% on AIME 2024, a hard competition for school students. After pure reinforcement learning, with no worked examples from people, it scored 77.9% (the figure in the Nature version of the paper). Along the way, its answers grew from a few hundred tokens to many thousands. It started to check its own work, and to write "Wait" when it found a mistake.

Part 3 showed how DeepSeek pre-trained V3, the base model under all of this. This part shows what happened after pre-training. It builds the method, GRPO, from a group of four answers you can score by hand. Then it follows R1 from the first experiment to the model people use, and on to V3.2, V4 and V4.1.

From predicting the next token to earning a reward

A pre-trained model has one skill. It continues text. Give it the start of a maths solution and it writes a likely next line, because it saw billions of similar lines. Nothing in that training rewards a right answer over a fluent wrong one.

The usual next step is supervised fine-tuning (SFT). People write good answers, and the model learns to copy them. SFT teaches a format and a style. It cannot teach the model to do better than the people who wrote the examples. It also cannot show the model the hundred wrong paths that lead to one right answer.

Reinforcement learning works the other way round. The model writes its own answers. A reward scores each answer. Training then makes high-scoring answers more likely and low-scoring answers less likely. Nobody has to write the right path. The model has to find it, and the reward tells it when it did.

This is how the assistants of 2022 to 2024 were trained, with a reward model that learned people's preferences and an algorithm called PPO. The page How models are trained today tells that history. This part starts where DeepSeek changed it.

PPO and its value model

Picture one answer of 600 tokens that gets a reward of 1 at the end. Which of the 600 tokens deserve the credit? Maybe token 212 was the clever step. Maybe token 540 fixed an error. RL needs some way to give each token its share.

PPO, the standard algorithm, answers this with a second network, the value model. At every token it predicts the reward that the answer will get in the end. If the prediction jumps up after token 212, token 212 was a good move. The difference between what happened and what the value model expected is the advantage.

That design works, but for a large language model it is expensive in two ways. DeepSeek said so plainly when they proposed the alternative.

PPO: four modelsGRPO: three, or two plus rulespolicytrainedreferencefrozenreward modelfrozenvalue modeltrained, same sizepolicytrainedreferencefrozenreward: model or rulesR1: answer checker + format checkAdvantage of each token comes from thevalue model (GAE). The value model mustguess the final reward from a half answer.Advantage of each answer comes from itsgroup: sample G answers to one question,compare each reward with the group mean.
PPO trains two large networks, the policy and a value model of about the same size. GRPO keeps only the trained policy, a frozen reference copy, and a reward, which can be a set of rules.

GRPO: let the group be the baseline

Think of a golf course. A score of 5 on a hole means nothing by itself. On a short, flat hole it is a bad score. On a long hole into the wind, where most players took 6, it is a good one. You do not need a model of the hole to judge the 5. You need the scores of the other players on the same hole, on the same day.

GRPO uses the same idea. For one question, it samples a group of answers from the current model. It scores all of them. Each answer is then judged against the others: better than the group's average is good, worse is bad. The group's average replaces the value model.

The advantage, by hand

The advantage of answer ii in a group is its reward, minus the group's mean reward, divided by the group's standard deviation:

A^i=ri−mean⁡(r1,…,rG)std⁡(r1,…,rG)\hat{A}_i = \frac{r_i - \operatorname{mean}(r_1, \ldots, r_G)}{\operatorname{std}(r_1, \ldots, r_G)}

Read it left to right. A^i\hat{A}_i is the advantage of answer ii. rir_i is its reward. The mean is the group's average reward, so the top of the fraction says "how much better than the group". The standard deviation is the typical spread of rewards in the group.

Dividing by it puts every group on the same scale, whether its rewards spread over 0 to 1 or 0 to 10. In outcome-only RL, every token of answer ii gets the same A^i\hat{A}_i.

Now a group of four. The question is "What is 7 x 8?". The reward follows R1-Zero. An answer gets 1 point if the boxed answer is right. It gets 1 more point if the reasoning sits inside think tags and the answer inside answer tags. Here are four sampled answers and their scores.

Question: What is 7 x 8? Four sampled answers, scored by rulesaccuracyformatrewardadvantageo1think tags, boxed 561+1=2+1.414o2think tags, boxed 540+1=10.000o3think tags, boxed 630+1=10.000o4"The answer is 56."0+0=0-1.414mean = 1, std = 0.707 (divide by G = 4). A = (reward - mean) / stdo4 holds the right number but gets the lowest score: no box, no tags, nothing a rule can check.
Our group of four. The rewards are 2, 1, 1 and 0, so the mean is 1. The right, well-formatted answer gets a positive advantage, the two wrong ones get zero, and the unformatted one gets a negative advantage.

Check it with a pencil. The mean is (2+1+1+0)/4=1(2 + 1 + 1 + 0) / 4 = 1. The squared distances from the mean are 1, 0, 0 and 1, which average to 0.5. The standard deviation is 0.5=0.707\sqrt{0.5} = 0.707. So answer 1 gets (2−1)/0.707=+1.414(2 - 1) / 0.707 = +1.414 and answer 4 gets −1.414-1.414. Answers 2 and 3 sit exactly at the mean and get 0.

Answer 4 holds the right number, 56, and still gets the worst score. It wrote no box and no tags, so no rule can check it. This is not a bug in the toy. A rule-based reward pays only for answers it can verify, and the model learns that format quickly.

Here is the same computation in code, two ways. Method 1 is plain Python for one group. Method 2 is PyTorch for a whole batch of groups at once, which is how a real trainer does it.

python
def advantages_scratch(r, eps=1e-6):
    """Method 1: plain Python. Population std (divide by G), as in the formula's usual reading."""
    G = len(r)
    mean = sum(r) / G
    std = math.sqrt(sum((x - mean) ** 2 for x in r) / G)
    return mean, std, [(x - mean) / (std + eps) for x in r]


def advantages_torch(R, eps=1e-6, unbiased=False):
    """Method 2: a whole batch of groups at once. R has shape (B, G): B questions, G answers each."""
    mean = R.mean(dim=1, keepdim=True)                       # (B, 1)
    std = R.std(dim=1, keepdim=True, unbiased=unbiased)      # (B, 1)
    return (R - mean) / (std + eps)                          # (B, G)

The output of part4_grpo.py (CPU, PyTorch, a few seconds):

plain text
Method 1 (from scratch): rewards [2.0, 1.0, 1.0, 0.0]  mean 1.0000  std 0.7071
  advantages: +1.4142, +0.0000, +0.0000, -1.4142
Method 2 (torch, batch of 4 groups), first row: +1.4142, +0.0000, +0.0000, -1.4142
max |Method 1 - Method 2| = 5.08e-08

The small eps stops a division by zero. When does that happen? When every answer in the group gets the same reward. The script runs three more groups to show what the advantage does in each case:

plain text
our group [2,1,1,0]      -> +1.414, +0.000, +0.000, -1.414
one right [1,0,0,0]      -> +1.732, -0.577, -0.577, -0.577
three right [1,1,1,0]    -> +0.577, +0.577, +0.577, -1.732
all right [1,1,1,1]      -> +0.000, +0.000, +0.000, +0.000

Look at the second and third rows. When one answer in four is right, that rare success gets a big push, +1.732. When three are right, the one failure gets a big push down. The method spends its effort on the surprise.

The last row is the important limit: a group where every answer is right (or every answer is wrong) has zero advantage everywhere. That question teaches the model nothing in this step. Questions must be hard enough that the model sometimes fails, and within reach often enough that it sometimes succeeds.

The full objective: a clip and a leash

The advantage says which answers to make more likely. The objective says how far to move in one update. GRPO keeps two safety devices from PPO. A clip limits how much any token's probability may change per update. A KL penalty keeps the model near a reference copy of itself.

Here is the objective, written out:

J(θ)=1G∑i=1G1∣oi∣∑t=1∣oi∣{min⁡[ρi,tA^i,t, clip⁡(ρi,t,1−ε,1+ε) A^i,t]−β DKL[πθ ∥ πref]}\mathcal{J}(\theta) = \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\Big\{\min\big[\rho_{i,t}\hat{A}_{i,t},\ \operatorname{clip}(\rho_{i,t},1-\varepsilon,1+\varepsilon)\,\hat{A}_{i,t}\big]-\beta\,\mathbb{D}_{KL}\big[\pi_\theta\,\|\,\pi_{ref}\big]\Big\} ρi,t=πθ(oi,t∣q,oi,<t)πθold(oi,t∣q,oi,<t)\rho_{i,t} = \frac{\pi_\theta(o_{i,t}\mid q, o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}\mid q, o_{i,<t})}

Read the symbols. θ\theta are the weights being trained, and πθ\pi_\theta is the model with those weights. qq is the question and oio_i the ii-th answer; oi,to_{i,t} is its token number tt, and oi,<to_{i,<t} the tokens before it.

πθold\pi_{\theta_{old}} is the model that sampled the group, and πref\pi_{ref} the frozen reference. ρi,t\rho_{i,t} (rho) is the probability ratio. ε\varepsilon (epsilon) sets the clip range, and β\beta (beta) the strength of the KL penalty. Training changes θ\theta to make J\mathcal{J} larger.

The clip

Take one token of answer 1, with advantage +1.414, and ε=0.2\varepsilon = 0.2. If the update makes it 10% more likely (ratio 1.1), it adds 1.1×1.414=1.5561.1 \times 1.414 = 1.556. At ratio 1.2 it adds 1.697. At 1.5 the clip holds the ratio at 1.2, so it still adds only 1.697. Pushing that token further earns nothing, so the gradient stops pushing it. Our script prints the full table:

plain text
ratio   A=+1.414   A=-1.414
 0.50     +0.707     -1.131
 0.80     +1.131     -1.131
 1.00     +1.414     -1.414
 1.10     +1.556     -1.556
 1.20     +1.697     -1.697
 1.50     +1.697     -2.121
 2.00     +1.697     -2.828
What one token adds to the objective, as its probability ratio moves (eps = 0.2)-2-10+1+20.50.81.01.21.52.0trust regionA = +1: flatabove 1.2A = -1: keeps fallingas the ratio grows past 1.2ratio = new probability / old probability of the sampled token
The clipped term for one token. With a positive advantage the gain stops at ratio 1.2. With a negative advantage the gain stops at 0.8, but the penalty keeps growing if a bad token became more likely.

The min makes the clip one-sided in a careful way. A good token gets no extra credit for moving more than 20%. A bad token gets no extra credit for being pushed down more than 20%. But if an update made a bad token more likely (ratio 1.5 with a negative advantage), the min keeps the full penalty, −2.121. The objective is always the more pessimistic of the two readings. This keeps each update small, so the samples from the old model stay a fair guide to the new one.

The KL leash

The second device is the KL term. GRPO estimates it per token with a formula that needs only the probabilities of the sampled token:

DKL[πθ ∥ πref]≈πref(oi,t)πθ(oi,t)−log⁡πref(oi,t)πθ(oi,t)−1\mathbb{D}_{KL}\big[\pi_\theta\,\|\,\pi_{ref}\big] \approx \frac{\pi_{ref}(o_{i,t})}{\pi_\theta(o_{i,t})} - \log\frac{\pi_{ref}(o_{i,t})}{\pi_\theta(o_{i,t})} - 1

Call the ratio x=πref/πθx = \pi_{ref}/\pi_\theta. The formula is x−log⁡x−1x - \log x - 1. It is 0 when x=1x = 1 (the two models agree on that token) and positive for every other xx, because log⁡x≤x−1\log x \le x - 1 always. This estimator is often called k3, from John Schulman's note on KL approximations. A pencil check: if the trained model gives a token 0.5 and the reference gave it 0.4, then x=0.8x = 0.8 and the term is 0.8+0.223−1=0.0230.8 + 0.223 - 1 = 0.023.

The script checks the claim that matters: averaged over tokens sampled from the trained model, k3 equals the true KL.

plain text
pi_theta 0.5  pi_ref 0.4  ->  k3 = 0.0231
pi_theta 0.9  pi_ref 0.1  ->  k3 = 1.3083
pi_theta 0.1  pi_ref 0.9  ->  k3 = 5.8028
exact KL(pi_theta || pi_ref) = 0.05143; mean of k3 over 200,000 samples = 0.05156

The last line is the point. One token's k3 can be large or small, but the average over 200,000 samples lands within 0.0002 of the exact value. Note the third line too: a token the trained model almost never picks (0.1) but the reference liked (0.9) gives a large penalty. V3.2 later found this case caused trouble, as we will see.

One more design choice. PPO for language models usually subtracts the KL from the reward at every token. GRPO puts it straight into the loss instead, so the advantage stays a clean group comparison. The Nature version of the R1 paper adds a reason that matters for long reasoning.

The whole loss in code

Now the full objective on a toy group: four answers of 5, 3, 4 and 2 tokens, padded to 5. Method 1 follows equation (3) with loops. Method 2 is the vectorised form, with a mask for padding.

python
def grpo_loss_torch(logp_new, logp_old, logp_ref, A, mask, eps=EPS, beta=BETA):
    """Method 2: the same, vectorised. Shapes: logp_* and mask (G, T), A (G,)."""
    ratio = (logp_new - logp_old).exp()                                  # (G, T)
    surr = torch.minimum(ratio * A[:, None], ratio.clamp(1 - eps, 1 + eps) * A[:, None])
    x = (logp_ref - logp_new).exp()                                      # pi_ref / pi_theta
    kl = x - (logp_ref - logp_new) - 1                                   # k3, always >= 0
    per_token = (surr - beta * kl) * mask
    per_answer = per_token.sum(dim=1) / mask.sum(dim=1)                  # 1/|o_i| sum_t
    return -per_answer.mean()                                            # 1/G sum_i, negated
plain text
Method 1 (loops) loss = -0.054389
Method 2 (torch) loss = -0.054389   difference 2.2e-09
gradient of the loss summed over each answer (negative = that answer is pushed up): -0.199, +0.003, -0.004, +0.171

The two methods agree to nine decimal places. The last line shows the direction of learning. The optimizer moves against the gradient, so answer 1 (advantage +1.414) is pushed up and answer 4 (−1.414) is pushed down. Answers 2 and 3 have zero advantage. Their tiny gradients come only from the KL term.

Rewards a program can check

Every RL method depends on its reward. For a chat assistant, the reward is usually a reward model: a network trained on people's choices between two answers. It can judge anything, but it can also be fooled.

For maths and code there is a better judge: a program. A maths answer can be compared with the known answer. Code can be run against tests. Such a reward cannot be talked into a high score. This approach is now often called RL with verifiable rewards.

Here are our two reward functions, and what they gave the four answers:

python
def accuracy_reward(text, truth):
    """1 if the last \\boxed{...} holds the right answer, else 0. No box = no checkable answer = 0."""
    boxes = re.findall(r'\\boxed\{([^}]*)\}', text)
    return 1.0 if boxes and boxes[-1].strip() == truth else 0.0


def format_reward(text):
    """1 if the reasoning sits inside <think>...</think> and the answer inside <answer>...</answer>."""
    return 1.0 if re.fullmatch(r'\s*<think>.+?</think>\s*<answer>.+?</answer>\s*', text, flags=re.S) else 0.0
plain text
o1: accuracy 1 + format 1 = reward 2   | <think>7 x 8 = 7 x 4 x 2 = 28 x 2 = 56</think> <answer>\boxed{56}</answer>
o2: accuracy 0 + format 1 = reward 1   | <think>7 x 8 = 54, I think</think> <answer>\boxed{54}</answer>
o3: accuracy 0 + format 1 = reward 1   | <think>7 x 9 = 63</think> <answer>\boxed{63}</answer>
o4: accuracy 0 + format 0 = reward 0   | The answer is 56.

The R1 paper's checkers are more careful than our regular expressions. Its RL infrastructure runs a code executor, an answer matcher and a format checker as separate services (Nature version, Supplementary B.1). The idea is the same.

Watching it learn

Before the real thing, a toy run. Our "policy" here is not a language model. It is a choice between the four answers above, with a probability for each. At the start, it prefers the unformatted answer, as a base model with no training might. Each step samples 8 answers, scores them with the rules, computes group advantages and takes two clipped updates.

plain text
step  0: p(o1..o4) = 0.126, 0.229, 0.229, 0.417   expected reward 0.709
step  5: p(o1..o4) = 0.373, 0.270, 0.270, 0.087   expected reward 1.286
step 10: p(o1..o4) = 0.757, 0.117, 0.090, 0.036   expected reward 1.721
step 20: p(o1..o4) = 0.952, 0.023, 0.017, 0.008   expected reward 1.944
step 30: p(o1..o4) = 0.970, 0.014, 0.010, 0.006   expected reward 1.964
step 40: p(o1..o4) = 0.970, 0.014, 0.010, 0.006   expected reward 1.964
Toy GRPO run: probability of each answer to 7 x 8 (our run)0.000.250.500.751.00010203040o1 right + format: 0.970o2 wrong (54): 0.014o3 wrong (63): 0.010o4 right, no format: 0.006GRPO step (8 samples per step, eps 0.2)
Our toy GRPO run. The unformatted answer falls first, the right and formatted answer rises, and after step 30 nothing changes.

Two things to notice. First, the unformatted answer o4 falls fastest, even though its number was right. The format lesson comes first, just as in real runs.

Second, the curve stops at 0.970. From step 30, every group of 8 samples was eight copies of o1. All rewards equal, all advantages zero, no update. This is the "all right" row from before, in action. Real training keeps a stream of harder questions for exactly this reason.

R1-Zero: RL straight from the base model

Now the real experiment. DeepSeek took V3-Base, the 671B pre-trained model from Part 3, and applied GRPO with rule-based rewards. There was no SFT first. The only instruction was a template: think inside think tags, then answer inside answer tags.

One step of DeepSeek-R1-Zero (numbers from the Nature version)32 questionsmaths, code, logicpolicyV3-Base at step 016 answers each512 in totalanswer checkerformat checkerrewards -> group advantagesreward = accuracy + format, A = (r - mean)/stdGRPO update of the policylr 3e-6, KL weight 0.001, clip ratioRepeat 10,400 steps. Every 400 steps the reference model becomes a copy of the policy.Answer length cap: 32,768 tokens, raised to 65,536 at step 8,200.
One step of R1-Zero training: 32 questions, 16 sampled answers each, checks by rules, group advantages, and a GRPO update of the policy.

The score, and which number to quote

The headline result has two versions, and the press mixed them up. Here is what each one says.

R1-Zero on AIME 2024: which number, from which versionstart of RLboth versions15.6%R1-Zero, arXiv v1 (Jan 2025)pass@171.0%R1-Zero, Nature / arXiv v2pass@177.9%R1-Zero, majority votecons@16 (v2), cons@64 (v1)86.7%
R1-Zero on AIME 2024. Both versions start at 15.6%. arXiv v1 reports 71.0% pass@1, the Nature version reports 77.9%, and majority voting gives 86.7% in both.

Longer answers, and the word "wait"

Nobody told R1-Zero to think longer. The reward did not pay for length. Yet the length of its answers grew almost steadily through training.

The checking turned into a recognisable habit. DeepSeek called it the "aha moment".

The Nature version made the moment measurable instead of anecdotal.

There was a price. R1-Zero's thinking was often hard to read. It mixed languages, "occasionally combining English and Chinese within a single chain-of-thought response" (Nature version). The supplement suggests why: V3-Base was trained mostly on Chinese and English, and nothing in the reward asked for one language. The reward paid only for the final box, so the scratch work drifted into whatever helped.

What RL adds, and what it does not

Is the model learning new skills, or getting more reliable at skills it had? DeepSeekMath had already asked this in 2024.

From R1-Zero to R1: four stages

R1-Zero proved a point. It was not a product. Its thinking mixed languages and lacked structure, and it trained only on questions with checkable answers. DeepSeek built R1 in four stages to keep the reasoning and fix the rest.

DeepSeek-R1: four stages, each starting from a checkpoint1. cold start SFTthousands of long, readable CoTV3-Base -> Dev12. reasoning RLrules + language consistencyDev1 -> Dev23. rejection sampling + SFT600K reasoning + 200K otherfresh V3-Base -> Dev34. RL for all scenariosrules + reward modelsDev3 -> R1Dev2 writes the600K reasoningsamplesStage 3 trains a fresh copy of V3-Base for two epochs on about 800K samples.The same 800K samples later train the distilled Qwen and Llama models.
The four stages of DeepSeek-R1, and the checkpoint each one produces. Stage 3 starts again from V3-Base, with data written by the Dev2 checkpoint.

The four stages, with their numbers (arXiv v1 unless noted):

  1. Cold start. "Thousands" of long, readable chains of thought fine-tune V3-Base. The format ends each answer with a summary. The Nature version's supplement says the motivation "is primarily product-driven".
  2. Reasoning RL. The R1-Zero recipe, plus a language consistency reward. In the Nature version this stage uses a GRPO clip ratio ε\varepsilon of 10, very large next to the usual 0.2. The paper says only that the value "plays a crucial role in training".
  3. Rejection sampling and SFT. Dev2 writes answers; only correct ones are kept, and mixed-language or messy ones are filtered out. That gives about 600K reasoning samples. With about 200K non-reasoning samples (writing, factual questions, translation), V3-Base is fine-tuned for two epochs on about 800K samples.
  4. RL for all scenarios. Rule rewards for reasoning, reward models for helpfulness and harmlessness. The helpfulness model judges only the final summary; the harmlessness model judges the whole response, thinking included.

The last stage shows where reward models still cause trouble.

What each stage buys

The Nature version measured every checkpoint. This is the clearest picture of the recipe.

What each stage buys (Nature version, Table 3)0255075100R1-ZeroDev1Dev2Dev3R1AIME 2024, R1-Zero: 77.9AIME 2024, Dev1: 59.0AIME 2024, Dev2: 74.0AIME 2024, Dev3: 78.1AIME 2024, R1: 79.8IF-Eval, R1-Zero: 46.6IF-Eval, Dev1: 71.7IF-Eval, Dev2: 72.0IF-Eval, Dev3: 78.1IF-Eval, R1: 83.3AlpacaEval 2.0, R1-Zero: 24.7AlpacaEval 2.0, Dev1: 50.1AlpacaEval 2.0, Dev2: 55.8AlpacaEval 2.0, Dev3: 62.1AlpacaEval 2.0, R1: 87.6AlpacaEval 2.0: 87.6IF-Eval: 83.3AIME 2024: 79.8The cold start (Dev1) costs AIME points; reasoning RL (Dev2) wins them back.
Benchmark scores at each checkpoint (Nature version, Table 3). The cold start lifts instruction following but costs AIME points. Reasoning RL recovers them. The final RL stage mostly lifts the general chat benchmark.
BenchmarkR1-ZeroDev1Dev2Dev3R1
AIME 2024 (pass@1)77.959.074.078.179.8
MATH-50095.994.295.995.497.3
GPQA Diamond75.866.170.771.271.5
LiveCodeBench50.057.563.564.665.9
IF-Eval (prompt strict)46.671.772.078.183.3
AlpacaEval 2.0 (LC win rate)24.750.155.862.187.6
ArenaHard53.677.073.275.692.3

Source: DeepSeek-R1, Nature version (arXiv v2), Table 3, page 9.

Read down the AIME row. The cold start drops it from 77.9 to 59.0: a few thousand human-friendly examples pulled the model away from its own long style. Reasoning RL brings it back to 74.0. Note GPQA too: R1-Zero's 75.8 is never reached again. R1 is not better than R1-Zero at everything. It is a model people can use.

Distillation: small models learn faster by copying

The 800K samples from stage 3 had a second use. DeepSeek fine-tuned six open models on them, with SFT only and no RL: Qwen2.5-Math-1.5B and 7B, Qwen2.5-14B and 32B, Llama-3.1-8B and Llama-3.3-70B-Instruct.

ModelAIME 2024 pass@1MATH-500GPQA DiamondLiveCodeBenchCodeforces rating
R1-Distill-Qwen-1.5B28.983.933.816.9954
R1-Distill-Qwen-7B55.592.849.137.61189
R1-Distill-Llama-8B50.489.149.039.61205
R1-Distill-Qwen-14B69.793.959.153.11481
R1-Distill-Qwen-32B72.694.362.157.21691
R1-Distill-Llama-70B70.094.565.257.51633
GPT-4o-0513 (for scale)9.374.649.932.9759

Source: R1 arXiv v1, Table 5 (identical in the Nature version, Table 15).

A 1.5B model at 28.9% on AIME, three times GPT-4o's 9.3%, from SFT on a teacher's answers. But the more important experiment asked a sharper question. Could a 32B model reach the same level with its own large-scale RL, like R1-Zero did?

Same Qwen2.5-32B base, two ways to teach it to reasonlarge-scale RL, over 10K stepsSFT on 800K R1 samples (distilled)AIME 202447.072.6MATH-50091.694.3GPQA Diamond55.062.1LiveCodeBench40.257.2Pass@1 in %, arXiv v1 Table 6 (same numbers in the Nature version, Table 16).
The same Qwen2.5-32B base, trained two ways. Distillation from R1 wins on all four benchmarks.

The Nature version adds one more test. DeepSeek ran RL on Qwen2-Math-7B, a base released in August 2024, before OpenAI's o1 and its long reasoning outputs were public. After about 10,000 steps it reached 22.3% on AIME 2024, against 7.9% for its instruct version. The point is that RL alone can grow reasoning, even in a base that never saw reasoning traces from another model.

What did not work

The R1 paper has a short section that many papers leave out: two ideas that failed.

The Nature version: cost, data and safety

The arXiv preprint of January 2025 said nothing about cost. The Nature paper of September 2025, with its supplement, did. These numbers are the most quoted, and the most misquoted, part of the R1 story.

GPU hours (thousands of H800 hours), at the papers' $2 per hourR1-Zero RL101K = $202KSFT data creation5K = $10KR1 (all its stages)41K = $82KR1 total147K = $294KV3 base model, for scale2,788K = $5.576MR1 rows: Nature version, Supplementary Table 7. V3 row: V3 report, Table 1. Neither counts research runs.
R1's reinforcement learning and SFT data cost 147K GPU hours. The V3 base model under it cost 2,788K. Neither figure counts research runs.
plain text
R1-Zero: 512 H800 x 198 h = 101,376 GPU hours (table: 101K)
R1:      512 H800 x  80 h = 40,960 GPU hours (table: 41K)
SFT data creation: 5,000 GPU hours (table: 5K)
total as printed: 147,000 GPU hours x $2 = $294,000
V3 base model (V3 report, Table 1): 2,788,000 GPU hours = $5,576,000
R1 on top of V3, as a share of the V3 figure: 5.3%  (our arithmetic)

The Nature version also disclosed things the preprint did not. V3-Base was trained on "plain web pages and e-books, without incorporating any synthetic data", though DeepSeek notes that some web pages hold OpenAI-generated answers. The RL data is described by domain. It holds 26K maths questions, 17K programming problems plus 8K bug fixes, 22K STEM multiple-choice questions and 15K logic problems. General helpfulness and harmlessness prompts complete it.

Maths decontamination removed about six million pre-training texts. A safety section compares R1 with other models and describes the extra risk-control system that DeepSeek runs in its service.

After R1: thinking modes, bigger RL, and distillation again

R1 was one model with one mode: always think. The models that followed changed three things. They merged thinking and non-thinking into one model. They spent far more compute on RL. And then, in V4, they replaced the final RL stage with distillation.

How DeepSeek post-training changed, 2024 to 2026Feb 2024DeepSeekMathGRPO: no value modelJan 2025R1-Zero and R1pure RL; four-stage pipelineAug 2025V3.1thinking + non-thinking in one modelNov 2025DeepSeekMath-V2a verifier as the rewardDec 2025V3.2, V3.2-SpecialeRL > 10% of pre-training costApr 2026V4mixed RL replaced by on-policy distillationSep 2026V4.1-Flashsame algorithms; better data and environments
Two and a half years of DeepSeek post-training, from GRPO in a 7B maths model to V4.1-Flash.

R1-0528 and V3.1: one model, two modes

In May 2025, R1-0528 updated R1 with "increased computational resources" and "algorithmic optimization mechanisms" during post-training, by its model card. AIME 2025 rose from 70.0% to 87.5%. There was no paper.

In August 2025, V3.1 put both behaviours into one set of weights. Its card says: "One model supports both thinking mode and non-thinking mode by changing the chat template." The thinking mode reached answer quality "comparable" to R1-0528 "while responding more quickly". The card gives no RL details, so the recipe behind hybrid thinking is not public.

V3.2: RL at more than 10% of pre-training

The V3.2 paper (December 2025) is mostly about sparse attention, which Part 5 covers. Its RL sections are just as important.

V3.2 also changed the shape of the pipeline. First, specialist models for maths, programming, logic, agents, agentic coding and agentic search are each trained with RL. Their outputs become training data for one model. Then a single RL stage trains reasoning, agents and human alignment together.

DeepSeekMath-V2 (November 2025) supplied that proof reward. It trains a verifier that grades whole proofs and a second check on the verifier itself, then rewards the prover with the verifier's score. It is the first DeepSeek reward for answers that are not a single number. Part 8 covers it with the other specialised models.

Four changes that keep big RL stable

Scaling RL ten times broke things that small runs never showed. V3.2 changes the advantage and adds four fixes, all built on GRPO: an unbiased KL estimate, off-policy sequence masking, Keep Routing and Keep Sampling Mask.

plain text
V3.2 mean-only advantage (no division by std), our group: +1.000, +0.000, +0.000, -1.000
V3.2 mean-only, one right of four:                         +0.750, -0.250, -0.250, -0.250

The other three fixes deal with a quieter problem. The samples come from a fast inference engine, while the updates run in a training framework. The two never compute exactly the same numbers.

The first of them, Off-Policy Sequence Masking, drops answers that have a negative advantage and have drifted far from the current model, measured by KL. The paper's reasoning: a model "benefits the most by learning from its own mistakes", but mistakes made by a noticeably different model can mislead it. Answers with positive advantage are never masked.

Keep Routing: train the same experts that wrote the answersampling enginetoken01234567training, routes recomputedtoken01234567training with Keep Routingtoken01234567expert 5 picked: its weights produced the tokentiny number differences pick expert 6 insteadthe recorded route is replayed: same experts
Keep Routing. The router in the sampling engine picked experts 2 and 5. Recomputing the route in training can pick a different expert. Keep Routing replays the route that was used.

Agentic tasks: building the exam

R1 trained on maths, code and logic, where answers can be checked. V3.2 wanted agents that use tools. That needs tasks with tools, environments, and a way to check the result. There were not enough of them, so DeepSeek made them.

Task typeTasksEnvironmentPrompts
Code agent24,667realextracted
Search agent50,275realsynthesized
General agent4,417synthesizedsynthesized
Code interpreter5,908realextracted

Source: DeepSeek-V3.2, Table 1, page 10. The four rows add up to 85,267, the paper's "85,000 complex prompts".

V4: on-policy distillation from many teachers

V4 (April 2026) kept most of V3.2's pipeline and changed its last step.

V4: many specialists teach one student, on the student's own answersmathscodeagentsinstructions...each specialist: SFT, then GRPO on its own domain (more than ten teachers in total)studentwrites its own answerloss: sum of w_i x KL(student || teacher_i)full vocabulary, every tokenteachers score every tokenof the student's answer
V4's post-training: specialists trained with SFT and GRPO become teachers. The student writes its own answers, and the loss pulls its token distribution toward each teacher's.

V4.1: the algorithm is not the point

The V4.1-Flash paper (September 2026) makes a statement that sums up the whole arc.

Benchmarks across the line

Scores from different papers use different settings, so compare within a row only with care. Every number below comes from the source named in its column.

BenchmarkV3 (V3 report)R1-Zero (R1 Nature, Table 3)R1 (R1 v1, Table 4)R1-0528 (card)V3.1 Thinking (card)V3.2 Thinking (V3.2, Table 2)V3.2-Speciale (V3.2, Table 3)
AIME 202439.277.979.891.493.1--
AIME 2025---87.588.493.196.0
GPQA Diamond59.175.871.581.080.182.485.7
LiveCodeBench40.550.065.973.374.883.388.7
Codeforces rating-144420291930209123862701
SWE Verified42.043.249.257.6-73.1-

LiveCodeBench uses different date windows in different reports, and the R1-0528 and V3.1 Codeforces numbers are for Div1. V3's report limits output to 8K tokens; R1 allowed 32K and R1-0528 64K. The trend is still clear: each generation spent more on post-training and got more from it.

Research notes: GRPO's sequence form in the Nature version, and other details
  • Sequence-level ratio. DeepSeekMath writes GRPO per token: a ratio and a clip for each token oi,to_{i,t}, averaged over 1/∣oi∣1/|o_i|. The Nature version of R1 writes it per answer: πθ(oi∣q)/πθold(oi∣q)\pi_\theta(o_i \mid q) / \pi_{\theta_{old}}(o_i \mid q), with no sum over tokens. It does not discuss the difference. V3.2 goes back to the per-token form. Our toy run, where each "answer" is one choice, is the sequence form by construction.
  • Clip ratio 10. The Nature version sets "the GRPO clip ratio ε to 10" for R1's first RL stage. With ε = 10 the clip range is [−9, 11], so it almost never binds on the low side. We quote the number as printed and do not interpret it further.
  • Reward models in R1. The helpfulness reward model was trained on 66,000 preference pairs, each judged by DeepSeek-V3 four times with the order swapped. The safety model was trained point-wise on 106,000 prompts labelled safe or unsafe. Both were trained for one epoch at learning rate 6e-6, batch 256.
  • RL infrastructure. The Nature supplement describes four modules: rollout on vLLM workers with expert parallelism and MTP for self-speculative decoding, an inference module for the reward and reference models, rule-based reward services run asynchronously, and training.
  • R1-Zero's AIME curve. The 86.7% majority-vote figure is cons@64 in arXiv v1 (Table 2) and is described as self-consistency in the Nature text; the Nature Figure 1 legend says cons@16.
  • Not public. The RL recipes of R1-0528 and V3.1, and the GPU hours of any post-training after R1.

Next: Part 5, sparse attention and V3.2: how DeepSeek made long contexts cheap enough for this much reasoning. A lightning indexer picks 2,048 tokens out of 128K for each query.

References

  1. Z. Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. 2024. GRPO, §4.1.
  2. DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv v1 (January 2025) and v2 (January 2026, the Nature text with supplement).
  3. D. Guo et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645, 633-638 (2025).
  4. DeepSeek-AI. DeepSeek-V3.2. 2025. §3 on scaling GRPO and agentic tasks.
  5. DeepSeek-AI. DeepSeek-V4. 2026. §5 on specialists and on-policy distillation.
  6. DeepSeek-AI. DeepSeek-V4.1-Flash. 2026. §5.1.
  7. Z. Shao et al. DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning. 2025.
  8. J. Schulman et al. Proximal Policy Optimization Algorithms. 2017.
  9. J. Schulman. Approximating KL Divergence. 2020. The k3 estimator.
  10. Model cards: DeepSeek-R1-0528, DeepSeek-V3.1, DeepSeek-V3.2-Speciale.
  11. Our scripts: code/deepseek/part4_grpo.py (GRPO by hand and in PyTorch), figs_part4.py, shots_part4.py.