DeepSeek-R1, read beside the paper
The paper and its explanation side by side.
- Part 1: The Big Idea: Reasoning from a Reward
A model that learned to think step by step from one signal: was the final answer right. We read the title, the abstract, Figure 1 and §1, and run the released 1.5B model next to its base model.
- Part 2: GRPO: Learning from a Group of Answers
The learning rule behind DeepSeek-R1, term by term: sample a group of answers, score them, and push the model toward the better ones without a critic. We write GRPO from scratch, match it with TRL, and run one real step on a small model.
- Part 3: Rewards and the Template
How R1-Zero scores its own answers with two simple rules, why the authors refused a learned reward model, and the one-paragraph prompt that starts training. We write each reward two ways, game a weak reward on purpose, and test two base models in the real template.
- Part 4: R1-Zero and the Aha Moment
What pure RL gave DeepSeek-R1-Zero, why its answers grew from hundreds to thousands of tokens, and the moment it wrote Wait. We test the released 1.5B distill.
- Part 5: The R1 Pipeline: Four Stages
How R1-Zero's raw skill became DeepSeek-R1: a cold start, reasoning RL with a language reward, rejection sampling into 800k examples, and a last RL stage. We run each idea in Python.
- Part 6: Distillation: Small Models That Reason
How 800k answers written by R1 teach six small open models to reason with plain fine-tuning, and why this beat RL on the same model. We write the loss by hand and test a distilled 1.5B model.
- Part 7: Results, Failures and Limits
How R1 is scored, where it wins and where it loses, the two ideas that failed, and what R1 still cannot do. We write pass@k and majority voting in Python, and test greedy decoding and few-shot prompts on the 1.5B model.
- Part 8: Train a Reasoner from Scratch: R1-Zero's Recipe in 330 Lines
We write the whole DeepSeek-R1-Zero method in a small Python package: the template, the two reward rules and GRPO. We check it against TRL and train a 0.5B model on a laptop.