DeepSeek-R1 and Reasoning by Reinforcement Learning
How DeepSeek taught models to reason with rewards a program can check: GRPO by hand, R1-Zero, the four-stage R1 recipe, distillation, costs, and what came after.
Give a model a hard maths problem and a pencil. Do not show it how to solve the problem. Tell it only, at the end, whether its final answer was right. Then let it try again, thousands of times, on thousands of problems.
In January 2025, DeepSeek showed what happens when you do this to a large model. Their model DeepSeek-R1-Zero started at 15.6% on AIME 2024, a hard competition for school students. After pure reinforcement learning, with no worked examples from people, it scored 77.9% (the figure in the Nature version of the paper). Along the way, its answers grew from a few hundred tokens to many thousands. It started to check its own work, and to write "Wait" when it found a mistake.
Part 3 showed how DeepSeek pre-trained V3, the base model under all of this. This part shows what happened after pre-training. It builds the method, GRPO, from a group of four answers you can score by hand. Then it follows R1 from the first experiment to the model people use, and on to V3.2, V4 and V4.1.
From predicting the next token to earning a reward#
A pre-trained model has one skill. It continues text. Give it the start of a maths solution and it writes a likely next line, because it saw billions of similar lines. Nothing in that training rewards a right answer over a fluent wrong one.
The usual next step is supervised fine-tuning (SFT). People write good answers, and the model learns to copy them. SFT teaches a format and a style. It cannot teach the model to do better than the people who wrote the examples. It also cannot show the model the hundred wrong paths that lead to one right answer.
Reinforcement learning works the other way round. The model writes its own answers. A reward scores each answer. Training then makes high-scoring answers more likely and low-scoring answers less likely. Nobody has to write the right path. The model has to find it, and the reward tells it when it did.
This is how the assistants of 2022 to 2024 were trained, with a reward model that learned people's preferences and an algorithm called PPO. The page How models are trained today tells that history. This part starts where DeepSeek changed it.
Picture one answer of 600 tokens that gets a reward of 1 at the end. Which of the 600 tokens deserve the credit? Maybe token 212 was the clever step. Maybe token 540 fixed an error. RL needs some way to give each token its share.
PPO, the standard algorithm, answers this with a second network, the value model. At every token it predicts the reward that the answer will get in the end. If the prediction jumps up after token 212, token 212 was a good move. The difference between what happened and what the value model expected is the advantage.
That design works, but for a large language model it is expensive in two ways. DeepSeek said so plainly when they proposed the alternative.
PPO trains two large networks, the policy and a value model of about the same size. GRPO keeps only the trained policy, a frozen reference copy, and a reward, which can be a set of rules.
Think of a golf course. A score of 5 on a hole means nothing by itself. On a short, flat hole it is a bad score. On a long hole into the wind, where most players took 6, it is a good one. You do not need a model of the hole to judge the 5. You need the scores of the other players on the same hole, on the same day.
GRPO uses the same idea. For one question, it samples a group of answers from the current model. It scores all of them. Each answer is then judged against the others: better than the group's average is good, worse is bad. The group's average replaces the value model.
The advantage of answer i in a group is its reward, minus the group's mean reward, divided by the group's standard deviation:
A^i=std(r1,…,rG)ri−mean(r1,…,rG)
Read it left to right. A^i is the advantage of answer i. ri is its reward. The mean is the group's average reward, so the top of the fraction says "how much better than the group". The standard deviation is the typical spread of rewards in the group.
Dividing by it puts every group on the same scale, whether its rewards spread over 0 to 1 or 0 to 10. In outcome-only RL, every token of answer i gets the same A^i.
Now a group of four. The question is "What is 7 x 8?". The reward follows R1-Zero. An answer gets 1 point if the boxed answer is right. It gets 1 more point if the reasoning sits inside think tags and the answer inside answer tags. Here are four sampled answers and their scores.
Our group of four. The rewards are 2, 1, 1 and 0, so the mean is 1. The right, well-formatted answer gets a positive advantage, the two wrong ones get zero, and the unformatted one gets a negative advantage.
Check it with a pencil. The mean is (2+1+1+0)/4=1. The squared distances from the mean are 1, 0, 0 and 1, which average to 0.5. The standard deviation is 0.5=0.707. So answer 1 gets (2−1)/0.707=+1.414 and answer 4 gets −1.414. Answers 2 and 3 sit exactly at the mean and get 0.
Answer 4 holds the right number, 56, and still gets the worst score. It wrote no box and no tags, so no rule can check it. This is not a bug in the toy. A rule-based reward pays only for answers it can verify, and the model learns that format quickly.
Here is the same computation in code, two ways. Method 1 is plain Python for one group. Method 2 is PyTorch for a whole batch of groups at once, which is how a real trainer does it.
python
def advantages_scratch(r, eps=1e-6): """Method 1: plain Python. Population std (divide by G), as in the formula's usual reading.""" G = len(r) mean = sum(r) / G std = math.sqrt(sum((x - mean) ** 2 for x in r) / G) return mean, std, [(x - mean) / (std + eps) for x in r]def advantages_torch(R, eps=1e-6, unbiased=False): """Method 2: a whole batch of groups at once. R has shape (B, G): B questions, G answers each.""" mean = R.mean(dim=1, keepdim=True) # (B, 1) std = R.std(dim=1, keepdim=True, unbiased=unbiased) # (B, 1) return (R - mean) / (std + eps) # (B, G)
The output of part4_grpo.py (CPU, PyTorch, a few seconds):
The small eps stops a division by zero. When does that happen? When every answer in the group gets the same reward. The script runs three more groups to show what the advantage does in each case:
plain text
our group [2,1,1,0] -> +1.414, +0.000, +0.000, -1.414one right [1,0,0,0] -> +1.732, -0.577, -0.577, -0.577three right [1,1,1,0] -> +0.577, +0.577, +0.577, -1.732all right [1,1,1,1] -> +0.000, +0.000, +0.000, +0.000
Look at the second and third rows. When one answer in four is right, that rare success gets a big push, +1.732. When three are right, the one failure gets a big push down. The method spends its effort on the surprise.
The last row is the important limit: a group where every answer is right (or every answer is wrong) has zero advantage everywhere. That question teaches the model nothing in this step. Questions must be hard enough that the model sometimes fails, and within reach often enough that it sometimes succeeds.
The advantage says which answers to make more likely. The objective says how far to move in one update. GRPO keeps two safety devices from PPO. A clip limits how much any token's probability may change per update. A KL penalty keeps the model near a reference copy of itself.
Read the symbols. θ are the weights being trained, and πθ is the model with those weights. q is the question and oi the i-th answer; oi,t is its token number t, and oi,<t the tokens before it.
πθold is the model that sampled the group, and πref the frozen reference. ρi,t (rho) is the probability ratio. ε (epsilon) sets the clip range, and β (beta) the strength of the KL penalty. Training changes θ to make J larger.
Take one token of answer 1, with advantage +1.414, and ε=0.2. If the update makes it 10% more likely (ratio 1.1), it adds 1.1×1.414=1.556. At ratio 1.2 it adds 1.697. At 1.5 the clip holds the ratio at 1.2, so it still adds only 1.697. Pushing that token further earns nothing, so the gradient stops pushing it. Our script prints the full table:
The clipped term for one token. With a positive advantage the gain stops at ratio 1.2. With a negative advantage the gain stops at 0.8, but the penalty keeps growing if a bad token became more likely.
The min makes the clip one-sided in a careful way. A good token gets no extra credit for moving more than 20%. A bad token gets no extra credit for being pushed down more than 20%. But if an update made a bad token more likely (ratio 1.5 with a negative advantage), the min keeps the full penalty, −2.121. The objective is always the more pessimistic of the two readings. This keeps each update small, so the samples from the old model stay a fair guide to the new one.
Call the ratio x=πref/πθ. The formula is x−logx−1. It is 0 when x=1 (the two models agree on that token) and positive for every other x, because logx≤x−1 always. This estimator is often called k3, from John Schulman's note on KL approximations. A pencil check: if the trained model gives a token 0.5 and the reference gave it 0.4, then x=0.8 and the term is 0.8+0.223−1=0.023.
The script checks the claim that matters: averaged over tokens sampled from the trained model, k3 equals the true KL.
The last line is the point. One token's k3 can be large or small, but the average over 200,000 samples lands within 0.0002 of the exact value. Note the third line too: a token the trained model almost never picks (0.1) but the reference liked (0.9) gives a large penalty. V3.2 later found this case caused trouble, as we will see.
One more design choice. PPO for language models usually subtracts the KL from the reward at every token. GRPO puts it straight into the loss instead, so the advantage stays a clean group comparison. The Nature version of the R1 paper adds a reason that matters for long reasoning.
Now the full objective on a toy group: four answers of 5, 3, 4 and 2 tokens, padded to 5. Method 1 follows equation (3) with loops. Method 2 is the vectorised form, with a mask for padding.
Method 1 (loops) loss = -0.054389Method 2 (torch) loss = -0.054389 difference 2.2e-09gradient of the loss summed over each answer (negative = that answer is pushed up): -0.199, +0.003, -0.004, +0.171
The two methods agree to nine decimal places. The last line shows the direction of learning. The optimizer moves against the gradient, so answer 1 (advantage +1.414) is pushed up and answer 4 (−1.414) is pushed down. Answers 2 and 3 have zero advantage. Their tiny gradients come only from the KL term.
Every RL method depends on its reward. For a chat assistant, the reward is usually a reward model: a network trained on people's choices between two answers. It can judge anything, but it can also be fooled.
For maths and code there is a better judge: a program. A maths answer can be compared with the known answer. Code can be run against tests. Such a reward cannot be talked into a high score. This approach is now often called RL with verifiable rewards.
Here are our two reward functions, and what they gave the four answers:
python
def accuracy_reward(text, truth): """1 if the last \\boxed{...} holds the right answer, else 0. No box = no checkable answer = 0.""" boxes = re.findall(r'\\boxed\{([^}]*)\}', text) return 1.0 if boxes and boxes[-1].strip() == truth else 0.0def format_reward(text): """1 if the reasoning sits inside <think>...</think> and the answer inside <answer>...</answer>.""" return 1.0 if re.fullmatch(r'\s*<think>.+?</think>\s*<answer>.+?</answer>\s*', text, flags=re.S) else 0.0
plain text
o1: accuracy 1 + format 1 = reward 2 | <think>7 x 8 = 7 x 4 x 2 = 28 x 2 = 56</think> <answer>\boxed{56}</answer>o2: accuracy 0 + format 1 = reward 1 | <think>7 x 8 = 54, I think</think> <answer>\boxed{54}</answer>o3: accuracy 0 + format 1 = reward 1 | <think>7 x 9 = 63</think> <answer>\boxed{63}</answer>o4: accuracy 0 + format 0 = reward 0 | The answer is 56.
The R1 paper's checkers are more careful than our regular expressions. Its RL infrastructure runs a code executor, an answer matcher and a format checker as separate services (Nature version, Supplementary B.1). The idea is the same.
Before the real thing, a toy run. Our "policy" here is not a language model. It is a choice between the four answers above, with a probability for each. At the start, it prefers the unformatted answer, as a base model with no training might. Each step samples 8 answers, scores them with the rules, computes group advantages and takes two clipped updates.
Our toy GRPO run. The unformatted answer falls first, the right and formatted answer rises, and after step 30 nothing changes.
Two things to notice. First, the unformatted answer o4 falls fastest, even though its number was right. The format lesson comes first, just as in real runs.
Second, the curve stops at 0.970. From step 30, every group of 8 samples was eight copies of o1. All rewards equal, all advantages zero, no update. This is the "all right" row from before, in action. Real training keeps a stream of harder questions for exactly this reason.
Now the real experiment. DeepSeek took V3-Base, the 671B pre-trained model from Part 3, and applied GRPO with rule-based rewards. There was no SFT first. The only instruction was a template: think inside think tags, then answer inside answer tags.
One step of R1-Zero training: 32 questions, 16 sampled answers each, checks by rules, group advantages, and a GRPO update of the policy.
The headline result has two versions, and the press mixed them up. Here is what each one says.
R1-Zero on AIME 2024. Both versions start at 15.6%. arXiv v1 reports 71.0% pass@1, the Nature version reports 77.9%, and majority voting gives 86.7% in both.
Nobody told R1-Zero to think longer. The reward did not pay for length. Yet the length of its answers grew almost steadily through training.
The checking turned into a recognisable habit. DeepSeek called it the "aha moment".
The Nature version made the moment measurable instead of anecdotal.
There was a price. R1-Zero's thinking was often hard to read. It mixed languages, "occasionally combining English and Chinese within a single chain-of-thought response" (Nature version). The supplement suggests why: V3-Base was trained mostly on Chinese and English, and nothing in the reward asked for one language. The reward paid only for the final box, so the scratch work drifted into whatever helped.
R1-Zero proved a point. It was not a product. Its thinking mixed languages and lacked structure, and it trained only on questions with checkable answers. DeepSeek built R1 in four stages to keep the reasoning and fix the rest.
The four stages of DeepSeek-R1, and the checkpoint each one produces. Stage 3 starts again from V3-Base, with data written by the Dev2 checkpoint.
The four stages, with their numbers (arXiv v1 unless noted):
Cold start. "Thousands" of long, readable chains of thought fine-tune V3-Base. The format ends each answer with a summary. The Nature version's supplement says the motivation "is primarily product-driven".
Reasoning RL. The R1-Zero recipe, plus a language consistency reward. In the Nature version this stage uses a GRPO clip ratio ε of 10, very large next to the usual 0.2. The paper says only that the value "plays a crucial role in training".
Rejection sampling and SFT. Dev2 writes answers; only correct ones are kept, and mixed-language or messy ones are filtered out. That gives about 600K reasoning samples. With about 200K non-reasoning samples (writing, factual questions, translation), V3-Base is fine-tuned for two epochs on about 800K samples.
RL for all scenarios. Rule rewards for reasoning, reward models for helpfulness and harmlessness. The helpfulness model judges only the final summary; the harmlessness model judges the whole response, thinking included.
The last stage shows where reward models still cause trouble.
The Nature version measured every checkpoint. This is the clearest picture of the recipe.
Benchmark scores at each checkpoint (Nature version, Table 3). The cold start lifts instruction following but costs AIME points. Reasoning RL recovers them. The final RL stage mostly lifts the general chat benchmark.
Benchmark
R1-Zero
Dev1
Dev2
Dev3
R1
AIME 2024 (pass@1)
77.9
59.0
74.0
78.1
79.8
MATH-500
95.9
94.2
95.9
95.4
97.3
GPQA Diamond
75.8
66.1
70.7
71.2
71.5
LiveCodeBench
50.0
57.5
63.5
64.6
65.9
IF-Eval (prompt strict)
46.6
71.7
72.0
78.1
83.3
AlpacaEval 2.0 (LC win rate)
24.7
50.1
55.8
62.1
87.6
ArenaHard
53.6
77.0
73.2
75.6
92.3
Source: DeepSeek-R1, Nature version (arXiv v2), Table 3, page 9.
Read down the AIME row. The cold start drops it from 77.9 to 59.0: a few thousand human-friendly examples pulled the model away from its own long style. Reasoning RL brings it back to 74.0. Note GPQA too: R1-Zero's 75.8 is never reached again. R1 is not better than R1-Zero at everything. It is a model people can use.
Distillation: small models learn faster by copying#
The 800K samples from stage 3 had a second use. DeepSeek fine-tuned six open models on them, with SFT only and no RL: Qwen2.5-Math-1.5B and 7B, Qwen2.5-14B and 32B, Llama-3.1-8B and Llama-3.3-70B-Instruct.
Model
AIME 2024 pass@1
MATH-500
GPQA Diamond
LiveCodeBench
Codeforces rating
R1-Distill-Qwen-1.5B
28.9
83.9
33.8
16.9
954
R1-Distill-Qwen-7B
55.5
92.8
49.1
37.6
1189
R1-Distill-Llama-8B
50.4
89.1
49.0
39.6
1205
R1-Distill-Qwen-14B
69.7
93.9
59.1
53.1
1481
R1-Distill-Qwen-32B
72.6
94.3
62.1
57.2
1691
R1-Distill-Llama-70B
70.0
94.5
65.2
57.5
1633
GPT-4o-0513 (for scale)
9.3
74.6
49.9
32.9
759
Source: R1 arXiv v1, Table 5 (identical in the Nature version, Table 15).
A 1.5B model at 28.9% on AIME, three times GPT-4o's 9.3%, from SFT on a teacher's answers. But the more important experiment asked a sharper question. Could a 32B model reach the same level with its own large-scale RL, like R1-Zero did?
The same Qwen2.5-32B base, trained two ways. Distillation from R1 wins on all four benchmarks.
The Nature version adds one more test. DeepSeek ran RL on Qwen2-Math-7B, a base released in August 2024, before OpenAI's o1 and its long reasoning outputs were public. After about 10,000 steps it reached 22.3% on AIME 2024, against 7.9% for its instruct version. The point is that RL alone can grow reasoning, even in a base that never saw reasoning traces from another model.
The arXiv preprint of January 2025 said nothing about cost. The Nature paper of September 2025, with its supplement, did. These numbers are the most quoted, and the most misquoted, part of the R1 story.
R1's reinforcement learning and SFT data cost 147K GPU hours. The V3 base model under it cost 2,788K. Neither figure counts research runs.
plain text
R1-Zero: 512 H800 x 198 h = 101,376 GPU hours (table: 101K)R1: 512 H800 x 80 h = 40,960 GPU hours (table: 41K)SFT data creation: 5,000 GPU hours (table: 5K)total as printed: 147,000 GPU hours x $2 = $294,000V3 base model (V3 report, Table 1): 2,788,000 GPU hours = $5,576,000R1 on top of V3, as a share of the V3 figure: 5.3% (our arithmetic)
The Nature version also disclosed things the preprint did not. V3-Base was trained on "plain web pages and e-books, without incorporating any synthetic data", though DeepSeek notes that some web pages hold OpenAI-generated answers. The RL data is described by domain. It holds 26K maths questions, 17K programming problems plus 8K bug fixes, 22K STEM multiple-choice questions and 15K logic problems. General helpfulness and harmlessness prompts complete it.
Maths decontamination removed about six million pre-training texts. A safety section compares R1 with other models and describes the extra risk-control system that DeepSeek runs in its service.
After R1: thinking modes, bigger RL, and distillation again#
R1 was one model with one mode: always think. The models that followed changed three things. They merged thinking and non-thinking into one model. They spent far more compute on RL. And then, in V4, they replaced the final RL stage with distillation.
Two and a half years of DeepSeek post-training, from GRPO in a 7B maths model to V4.1-Flash.
In May 2025, R1-0528 updated R1 with "increased computational resources" and "algorithmic optimization mechanisms" during post-training, by its model card. AIME 2025 rose from 70.0% to 87.5%. There was no paper.
In August 2025, V3.1 put both behaviours into one set of weights. Its card says: "One model supports both thinking mode and non-thinking mode by changing the chat template." The thinking mode reached answer quality "comparable" to R1-0528 "while responding more quickly". The card gives no RL details, so the recipe behind hybrid thinking is not public.
The V3.2 paper (December 2025) is mostly about sparse attention, which Part 5 covers. Its RL sections are just as important.
V3.2 also changed the shape of the pipeline. First, specialist models for maths, programming, logic, agents, agentic coding and agentic search are each trained with RL. Their outputs become training data for one model. Then a single RL stage trains reasoning, agents and human alignment together.
DeepSeekMath-V2 (November 2025) supplied that proof reward. It trains a verifier that grades whole proofs and a second check on the verifier itself, then rewards the prover with the verifier's score. It is the first DeepSeek reward for answers that are not a single number. Part 8 covers it with the other specialised models.
Scaling RL ten times broke things that small runs never showed. V3.2 changes the advantage and adds four fixes, all built on GRPO: an unbiased KL estimate, off-policy sequence masking, Keep Routing and Keep Sampling Mask.
plain text
V3.2 mean-only advantage (no division by std), our group: +1.000, +0.000, +0.000, -1.000V3.2 mean-only, one right of four: +0.750, -0.250, -0.250, -0.250
The other three fixes deal with a quieter problem. The samples come from a fast inference engine, while the updates run in a training framework. The two never compute exactly the same numbers.
The first of them, Off-Policy Sequence Masking, drops answers that have a negative advantage and have drifted far from the current model, measured by KL. The paper's reasoning: a model "benefits the most by learning from its own mistakes", but mistakes made by a noticeably different model can mislead it. Answers with positive advantage are never masked.
Keep Routing. The router in the sampling engine picked experts 2 and 5. Recomputing the route in training can pick a different expert. Keep Routing replays the route that was used.
R1 trained on maths, code and logic, where answers can be checked. V3.2 wanted agents that use tools. That needs tasks with tools, environments, and a way to check the result. There were not enough of them, so DeepSeek made them.
Task type
Tasks
Environment
Prompts
Code agent
24,667
real
extracted
Search agent
50,275
real
synthesized
General agent
4,417
synthesized
synthesized
Code interpreter
5,908
real
extracted
Source: DeepSeek-V3.2, Table 1, page 10. The four rows add up to 85,267, the paper's "85,000 complex prompts".
V4 (April 2026) kept most of V3.2's pipeline and changed its last step.
V4's post-training: specialists trained with SFT and GRPO become teachers. The student writes its own answers, and the loss pulls its token distribution toward each teacher's.
Scores from different papers use different settings, so compare within a row only with care. Every number below comes from the source named in its column.
Benchmark
V3 (V3 report)
R1-Zero (R1 Nature, Table 3)
R1 (R1 v1, Table 4)
R1-0528 (card)
V3.1 Thinking (card)
V3.2 Thinking (V3.2, Table 2)
V3.2-Speciale (V3.2, Table 3)
AIME 2024
39.2
77.9
79.8
91.4
93.1
-
-
AIME 2025
-
-
-
87.5
88.4
93.1
96.0
GPQA Diamond
59.1
75.8
71.5
81.0
80.1
82.4
85.7
LiveCodeBench
40.5
50.0
65.9
73.3
74.8
83.3
88.7
Codeforces rating
-
1444
2029
1930
2091
2386
2701
SWE Verified
42.0
43.2
49.2
57.6
-
73.1
-
LiveCodeBench uses different date windows in different reports, and the R1-0528 and V3.1 Codeforces numbers are for Div1. V3's report limits output to 8K tokens; R1 allowed 32K and R1-0528 64K. The trend is still clear: each generation spent more on post-training and got more from it.
Research notes: GRPO's sequence form in the Nature version, and other details
Sequence-level ratio. DeepSeekMath writes GRPO per token: a ratio and a clip for each token oi,t, averaged over 1/∣oi∣. The Nature version of R1 writes it per answer: πθ(oi∣q)/πθold(oi∣q), with no sum over tokens. It does not discuss the difference. V3.2 goes back to the per-token form. Our toy run, where each "answer" is one choice, is the sequence form by construction.
Clip ratio 10. The Nature version sets "the GRPO clip ratio ε to 10" for R1's first RL stage. With ε = 10 the clip range is [−9, 11], so it almost never binds on the low side. We quote the number as printed and do not interpret it further.
Reward models in R1. The helpfulness reward model was trained on 66,000 preference pairs, each judged by DeepSeek-V3 four times with the order swapped. The safety model was trained point-wise on 106,000 prompts labelled safe or unsafe. Both were trained for one epoch at learning rate 6e-6, batch 256.
RL infrastructure. The Nature supplement describes four modules: rollout on vLLM workers with expert parallelism and MTP for self-speculative decoding, an inference module for the reward and reference models, rule-based reward services run asynchronously, and training.
R1-Zero's AIME curve. The 86.7% majority-vote figure is cons@64 in arXiv v1 (Table 2) and is described as self-consistency in the Nature text; the Nature Figure 1 legend says cons@16.
Not public. The RL recipes of R1-0528 and V3.1, and the GPU hours of any post-training after R1.
Next:Part 5, sparse attention and V3.2: how DeepSeek made long contexts cheap enough for this much reasoning. A lightning indexer picks 2,048 tokens out of 128K for each query.