A model that learned to think step by step from one signal: was the final answer right. We read the title, the abstract, Figure 1 and §1, and run the released 1.5B model next to its base model.
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and others. arXiv 2025 (Nature 2025), 2025. arXiv:2501.12948
In January 2025, a lab in Hangzhou released a model that solved competition maths as well as the best closed model of the time. It also published how, and the method was the surprise. Its first model, DeepSeek-R1-Zero, got no supervised fine-tuning on worked reasoning before reinforcement learning. Two rules scored it: was the final answer right, and was the reasoning written inside the right tags. From that alone it found long, careful reasoning by itself. The released model, DeepSeek-R1, then adds thousands of worked examples (the paper's "cold-start data") and more training stages on top.
Each piece below shows the exact lines of the paper, then explains them.
These three sentences describe the whole paper. The picture below puts them on one map. Each later part of this series zooms into one box.
The paper in one picture. Road 1: reinforcement learning straight on the base model gives DeepSeek-R1-Zero. Road 2: four stages, two of SFT and two of RL, give DeepSeek-R1. Road 3: about 800,000 samples from R1 fine-tune six small models. The part of this series that explains each step is under it.
The six benchmarks test different skills. AIME 2024 is the American Invitational Mathematics Examination: 30 hard competition problems, each with a whole-number answer from 0 to 999. MATH-500 is 500 problems from a large set of school competition maths. GPQA Diamond has 198 multiple-choice science questions written by experts so that a web search does not help. MMLU is multiple-choice general knowledge in 57 subjects.
Codeforces is a site for programming contests. SWE-bench Verified asks the model to fix real bugs in real Python projects.
Here is the same figure, redrawn from the paper's numbers, so you can hover each bar:
Figure 1 of the paper, redrawn from its numbers. R1 and o1-1217 are within about 4 points of each other on every benchmark. DeepSeek-V3, from the same base model, is far behind on AIME 2024 (39.2) and Codeforces (58.7), but close on MMLU (88.5).
The weak group is SWE-bench Verified. R1 fixes 49.2% of the bugs, and V3 already fixed 42.0%. Section 5 of the paper gives the reason. RL on software tasks is slow, because each test of a fix takes a long time. So R1 got little RL on this kind of task.
The AIME 2024 numbers of the introduction next to the models it compares with. Reinforcement learning alone takes R1-Zero from 15.6% to 71.0%, and a vote of 64 answers takes it to 86.7%, above the 83.3% of o1-0912 with the same vote.
The smallest student, DeepSeek-R1-Distill-Qwen-1.5B, is small enough for a laptop. Its base model is Qwen2.5-Math-1.5B (§2.4). So we can do something that is hard with the 671B models. We ask the same question to the base model and to the reasoning model, on one machine, and look at every token. The script is r1_part1.py; everything runs on an Apple laptop GPU (mps) in float32.
plain text
== 1. The two models (float32 on mps) deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B params 1,777,088,000 layers 28 d_model 1536 heads 12 vocab 151,936 Qwen/Qwen2.5-Math-1.5B params 1,543,714,304 layers 28 d_model 1536 heads 12 vocab 151,936 same shapes: True; mean |difference| of one weight matrix (layer 0 up_proj): 0.00406
The two models have the same 28 layers, the same width and the same vocabulary. Every weight matrix has the same shape. The parameter counts differ only because the base model shares one matrix between its input and output layers, and the distilled release stores two copies. The weights themselves moved only a little: in one matrix, the mean change is 0.004. All the difference in behaviour below comes from these small changes, made by SFT on R1's answers.
A language model writes one token at a time. At each step it gives a score, a logit, to every token of its vocabulary: 151,936 scores here. A decoding rule turns the scores into one chosen token. The paper's evaluation (§3, page 12) samples with "a temperature of 0.6 and a top-p value of 0.95". We use the same rule everywhere on this page.
Temperature divides every logit by a number T before the softmax:
pi=∑jexp(zj/T)exp(zi/T)
Read it from left to right. zi is the logit of token i, and pi is its probability. The sum in the denominator runs over all tokens, so the probabilities add up to 1. With T=1 this is the plain softmax. With T=0.6, the gaps between the logits grow by a factor of 1/0.6≈1.67. So the top token gets more likely, and the rare ones get rarer.
Top-p then sorts the tokens by probability and keeps the smallest group at the top whose probabilities add up to at least p. It throws away the rest and rescales the kept ones so that they add up to 1 again. A long tail of thousands of unlikely tokens can still hold a few percent in total. Top-p cuts that tail, so the model does not pick a strange token by bad luck.
Here is Method 1, our own Python. Each function does one step:
python
def softmax(z): """z: (V,) logits -> (V,) probabilities that add up to 1.""" e = torch.exp(z - z.max()) # subtract the max so exp never overflows return e / e.sum()def temperature_probs(logits, temperature): """Divide the logits by the temperature, then softmax. T < 1 makes the top token more likely.""" return softmax(logits / temperature) # (V,)def top_p_filter(p, top_p): """Keep the smallest set of top tokens whose probabilities add up to at least top_p; renormalise. p: (V,).""" order = torch.argsort(p, descending=True) # (V,) token ids, most likely first sorted_p = p[order] before = torch.cumsum(sorted_p, 0) - sorted_p # mass of the tokens ranked above each one keep_sorted = before < top_p # keep a token while the set is still short of top_p keep = torch.zeros_like(p, dtype=torch.bool) keep[order[keep_sorted]] = True q = torch.where(keep, p, torch.zeros_like(p)) return q / q.sum(), int(keep.sum())
The line with before is the one to read twice. For each token in the sorted list, it is the total probability of the tokens ranked above it. A token stays while that total is still below 0.95. So the token that crosses 0.95 stays too, and every token after it goes.
Method 2 is the library. The transformers package has the same two steps as TemperatureLogitsWarper and TopPLogitsWarper, the classes that model.generate uses. We run both on four toy logits, and then on the real logits of the distilled model for its first token:
plain text
== 2. Temperature and top-p on four toy logits token logit p (T=1) p (T=0.6) after top-p 0.95 library "18" 2.00 0.609 0.783 0.787 0.787 "9" 1.00 0.224 0.148 0.149 0.149 "16" 0.50 0.136 0.064 0.065 0.065 "2" -1.00 0.030 0.005 0.000 0.000 top-p keeps 3 of 4 tokens; max |ours - library| = 6.0e-08 real logits: the first token the distilled model writes after the GSM8K prompt (vocabulary 151,936) 'Okay' p (T=0.6) = 0.7923 'Alright' p (T=0.6) = 0.2069 'First' p (T=0.6) = 0.0006 'Hmm' p (T=0.6) = 0.0001 'To' p (T=0.6) = 0.0000 top-p 0.95 keeps 2 tokens of 151,936; max |ours - library| = 0.0e+00
Check the toy row by hand.
With T=1, the weight of "18" is e2=7.39, and the four weights add up to 7.39+2.72+1.65+0.37=12.13. So p=7.39/12.13=0.609, as printed. With T=0.6 the same token gets 0.783. Then top-p adds up 0.783, 0.148 and 0.064. The sum 0.995 is past 0.95, so the fourth token goes, and the three that stay are scaled up a little. Our numbers and the library's agree to 10−7.
The real step is more extreme. Of 151,936 tokens, only two survive top-p: "Okay" and "Alright". The distilled model almost always opens its thinking with one of these two words. It learned this habit from R1's answers.
One decoding step on four toy tokens, with the numbers printed by our script. The softmax turns logits into probabilities; temperature 0.6 sharpens them; top-p 0.95 cuts the last token and rescales the other three.
Greedy decoding has no randomness: it always takes the token with the highest logit. So it is a clean test of our own generation loop. If our loop and model.generate both decode greedily, they must write exactly the same tokens.
python
@torch.no_grad()def our_generate(model, ids, max_new, eos_id, greedy=False, seed=0): """Method 1: our own token-by-token loop. The first call reads the whole prompt and fills the KV cache; every later call feeds one token and reuses the cache. ids: (1, n) prompt token ids.""" gen = torch.Generator().manual_seed(seed) out = model(input_ids=ids, use_cache=True) # prefill: logits (1, n, V) past, logits = out.past_key_values, out.logits[0, -1] new = [] for _ in range(max_new): tok = pick(logits, gen, greedy) new.append(tok) if tok == eos_id: break out = model(input_ids=torch.tensor([[tok]], device=ids.device), past_key_values=past, use_cache=True) past, logits = out.past_key_values, out.logits[0, -1] # (V,) scores for the next token return new
The first call reads the whole prompt at once and keeps the keys and values of every layer. This is the KV cache, which LLM Inference, Part 2 explains. Each later call feeds only the newest token. pick is either argmax or our temperature and top-p functions followed by one random draw.
plain text
== 3. Greedy decoding, 128 new tokens: our loop (KV cache) against model.generate our loop 128 tokens, generate 128 tokens, equal at 128 of 128 positions: True first 12 tokens: ['Okay', ',', 'Ġso', 'ĠI', 'Ġneed', 'Ġto', 'Ġfigure', 'Ġout', 'Ġhow', 'Ġmuch', 'Ġmoney', 'ĠJanet']
All 128 tokens are the same. (The mark "Ġ" means "a space comes before this token".) With sampling, the two methods write different tokens, even with the same seed. They draw their random numbers in a different order. They draw from the same distribution, which the check in step 2 shows.
Now the real question, the first problem of the GSM8K test set, a set of school word problems. We add the instruction that the official repository recommends for maths, "Please reason step by step, and put your final answer within \boxed{}."
The model's chat format wraps the question in special tokens and ends the prompt with <think>, so the model starts inside its thinking. Our own sampling loop writes at most 2,048 new tokens. Here is the start of what it wrote (the full text is in results/r1_part1_stdout.txt):
plain text
== 4. GSM8K test question 0, distilled model, our own sampling loop (T 0.6, top-p 0.95, seed 0, max 2,048 new tokens) | Okay, so Janet has ducks that lay 16 eggs every day. She eats three eggs for breakfast each morning and bakes muffins for her friends with four eggs each day. Then, she sells the remaining eggs at the farmers' market for $2 each. I need to figure out how much money she makes every day from selling these eggs. ... | So, she eats 3 eggs every morning. There are 24 hours in a day, but since she eats breakfast every morning, which is 12 times a day, right? Because a day has 24 hours, so 24 divided by 2 is 12. So, she eats 3 eggs 12 times a day. Let me calculate how many eggs she eats for breakfast in a day. | That would be 3 eggs/morning * 12 mornings/day = 36 eggs. | Wait, hold on. That can't be right because she only has 16 eggs a day. If she eats 36 eggs for breakfast, that's way more than she has. That must be a mistake in my reasoning. | Let me rethink that. She eats 3 eggs for breakfast every morning. Since a day has 24 hours, but she only eats breakfast once a day. So, she eats 3 eggs once a day. So, that's 3 eggs. ... ... | Then, she sells these 9 eggs at the farmers' market for $2 each. So, she makes 9 * $2 = $18 per day. | | Wait, that seems like a lot. Let me confirm each step again.
This is a small model, and it makes a silly mistake: twelve breakfasts a day. But look at what it does next. "Wait, hold on. That can't be right." It checks the result against the problem, 36 eggs eaten from 16 laid, sees the contradiction, and starts again. Nobody wrote this check into the model. It came from R1's answers, and R1 got it from RL. This is the self-verification of §1.1, in a 1.5B model on a laptop.
Then the model keeps checking. It finds 18 dollars early, and then it checks again, and again. The end of its thinking and the start of its answer:
plain text
| **Final Answer** | \boxed{18} | </think> | | Janet’s ducks lay 16 eggs per day. She eats 3 eggs for breakfast every morning and 4 eggs for muffins every day. The remaining eggs are sold at the farmers' market for $2 each. ... new tokens 2048 in 134.3 s (15.2 tokens/s); thinking tokens 1907; answer tokens 140; closed </think>: True boxed answer None, gold '18'; correct (our checker) False; correct (math-verify) False
The thinking took 1,907 tokens for a problem that needs two subtractions and one product.
It ended with \boxed{18} inside the thinking. Then the model began a clean answer for the reader. Our limit of 2,048 tokens cut it off after 140 tokens, before its box. So the code marks it wrong. The reasoning was right; the budget was too small. The paper allows 32,768 new tokens for every benchmark (§3).
The answer of our own sampling loop, split at the closing think tag: 1,907 thinking tokens, then 140 tokens of the answer before the 2,048-token limit cut it off.
How does the code check an answer? It takes the content of the last \boxed{...} in the answer and compares it with the right answer. Method 1 is our own checker: remove spaces, dollar signs and units, then compare the strings. Method 2 is math-verify, a library that parses both answers as maths, so "0.5" and "\frac{1}{2}" count as equal. A rule that a program can check, with no human and no judge model, is exactly the kind of reward that R1-Zero trained on.
python
def check(pred, gold): """Return (ours, math_verify): is the boxed answer equal to the gold answer?""" ours = pred is not None and norm(pred) == norm(gold) mv = pred is not None and bool(verify(parse('