DeepSeek-R1, explained
The January 2025 paper that taught a language model to reason with reinforcement learning and two simple rules for reward. Eight parts, with the paper's own text, worked equations, GRPO written from scratch in Python, and real runs on the released distilled model.
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and others. arXiv 2025 (Nature 2025), 2025. arXiv:2501.12948
What you will learn
- What a reasoning model is, and why longer thinking at answer time gives better answers
- How GRPO works: sample a group of answers, score them, and push the model toward the better ones without a critic model
- Why two rule-based rewards (is the answer right, is the format right) were enough, and why the authors avoided a neural reward model
- How DeepSeek-R1-Zero learned to think longer and to check its own work, and what the aha moment really shows
- The four stages that turn R1-Zero's raw skill into DeepSeek-R1, a model people can read and use
- How 800,000 examples from R1 teach a small model to reason, and why distillation beat RL on the same small model
- What the results tables show, what failed (process rewards and tree search), and what R1 still cannot do
- How to write GRPO yourself in Python and train a small model to reason on your own laptop
Why this paper
Large language models were good at writing, but weak at hard multi-step problems such as competition maths. In 2024 one company showed that a model gets much better when it thinks for a long time before it answers. It did not say how it trained that model.
This paper shows one way, in the open. The authors took a base model and gave it only reinforcement learning, with two simple rules for reward: is the final answer right, and is the thinking inside the right tags. No human wrote an example of good reasoning. The model learned to think longer, to check its own steps and to try a new approach when the first one failed. The paper then turns this raw skill into DeepSeek-R1, a model that matched the best closed reasoning model of the time, and releases the weights.
The paper also shows that the reasoning of a big model can be copied into small models with plain fine-tuning. A 1.5-billion-parameter model trained this way runs on a laptop, and we use it in this series.
How the eight parts are organised
Before Part 1 (optional): The Road to R1. How language models are trained, chain-of-thought, reinforcement learning from human feedback, PPO, and thinking at answer time.
Each part follows the paper's own sections:
- The Big Idea: the title, the abstract, Figure 1 and §1. Reasoning by reinforcement learning alone, and the three things the paper releases.
- GRPO: §2.1 and §2.2.1, Equations 1 to 3. The learning algorithm, term by term, written from scratch and checked against a library.
- Rewards and the Template: §2.2.2, §2.2.3 and Table 1. The accuracy reward, the format reward, and the one-paragraph prompt that starts it all.
- R1-Zero and the Aha Moment: §2.2.4, Table 2, Figures 2 and 3, Table 3. What pure RL achieved, why answers got longer, and the moment the model said "Wait".
- The R1 Pipeline: §2.3. Cold-start data, reasoning RL with a language reward, rejection sampling, and the last RL stage for all tasks.
- Distillation: §2.4, §3.2, §4.1, Tables 5 and 6. Teaching small models with R1's answers, and why this beat RL on the small model.
- Results, Failures and Limits: §3, §3.1, §4.2, §5 and Table 4. How the models are scored, what the results show, what did not work, and what is still open.
- Train a Reasoner from Scratch: the whole method in one short Python package. We write GRPO and the rewards ourselves and train a small model to reason on a laptop.
How to read the boxes
- A teal "From the paper" box shows the exact lines of the paper as a highlighted screenshot. Click the screenshot to see it at full size. Under it, What it says gives the main words of the paper. What it means then explains those lines, and what the authors mean but do not write. When the box shows a different paper, the box gives the name of that paper.
- A yellow definition box explains a technical word the first time it occurs. If you know the word, skip the box.
- The small Paper §2.2 tag above a heading tells you which section of the paper you read.
- A green "Key takeaways" box ends each part with the facts to remember.
Code blocks show real code, and the output under a code block is what that code printed. The scripts are in code/papers/deepseek-r1.
We follow the first version of the paper (arXiv v1, 22 January 2025), the version most people read. A longer version came out later with the Nature article; where it adds a useful fact, the part says so.
What you need to know first
You need to know what a language model does: it reads text and predicts the next token. Every other term is explained when it first occurs. The Transformer series explains the model inside, and DeepSeek, from the Ground Up places R1 among DeepSeek's other models.
Parts
- Background: The Road to R1
How a language model learns, how it was taught to show its steps, and how reinforcement learning with human feedback works. The ideas the R1 paper assumes, with small runs you can check.
- Part 1: The Big Idea: Reasoning from a Reward
A model that learned to think step by step from one signal: was the final answer right. We read the title, the abstract, Figure 1 and §1, and run the released 1.5B model next to its base model.
- Part 2: GRPO: Learning from a Group of Answers
The learning rule behind DeepSeek-R1, term by term: sample a group of answers, score them, and push the model toward the better ones without a critic. We write GRPO from scratch, match it with TRL, and run one real step on a small model.
- Part 3: Rewards and the Template
How R1-Zero scores its own answers with two simple rules, why the authors refused a learned reward model, and the one-paragraph prompt that starts training. We write each reward two ways, game a weak reward on purpose, and test two base models in the real template.
- Part 4: R1-Zero and the Aha Moment
What pure RL gave DeepSeek-R1-Zero, why its answers grew from hundreds to thousands of tokens, and the moment it wrote Wait. We test the released 1.5B distill.
- Part 5: The R1 Pipeline: Four Stages
How R1-Zero's raw skill became DeepSeek-R1: a cold start, reasoning RL with a language reward, rejection sampling into 800k examples, and a last RL stage. We run each idea in Python.
- Part 6: Distillation: Small Models That Reason
How 800k answers written by R1 teach six small open models to reason with plain fine-tuning, and why this beat RL on the same model. We write the loss by hand and test a distilled 1.5B model.
- Part 7: Results, Failures and Limits
How R1 is scored, where it wins and where it loses, the two ideas that failed, and what R1 still cannot do. We write pass@k and majority voting in Python, and test greedy decoding and few-shot prompts on the 1.5B model.
- Part 8: Train a Reasoner from Scratch: R1-Zero's Recipe in 330 Lines
We write the whole DeepSeek-R1-Zero method in a small Python package: the template, the two reward rules and GRPO. We check it against TRL and train a 0.5B model on a laptop.