How R1-Zero's raw skill became DeepSeek-R1: a cold start, reasoning RL with a language reward, rejection sampling into 800k examples, and a last RL stage. We run each idea in Python.
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and others. arXiv 2025 (Nature 2025), 2025. arXiv:2501.12948
R1-Zero could reason, but its thinking was often hard to read, and it switched between English and Chinese in the middle of a thought. Section 2.3 is the recipe that kept the skill and fixed the rest, in four stages and two pages of text.
The whole recipe of §2.3. DeepSeek-V3-Base enters twice: at stage 1, and again at stage 3. The dashed arrow carries data, not weights.
Part 4 ended with R1-Zero's two problems: poor readability and language mixing. Section 2.3 opens with two questions that come straight from them.
The later long version of the paper (arXiv v2, January 2026, the text behind the Nature article) draws the same pipeline as a picture. It also names the checkpoints between the stages: Dev-1, Dev-2 and Dev-3. We use these names in this part.
The paper says R1 avoids "the early unstable cold start phase of RL training from the base model". To see why that phase is unstable, recall how GRPO learns (Part 2). For each question it samples a group of answers and scores them. Each answer's advantage is its reward minus the group mean, divided by the group's standard deviation.
Now think of a base model at step 0. It was trained to continue web text, not to answer in a format. Most of its answers to a hard question are wrong, or they never close the </think> tag. If all eight answers in a group get reward 0, then every answer equals the mean. Every advantage is 0, and the group teaches nothing. The model learns only from the rare groups where at least one answer is right.
So the first part of training is slow and noisy. The few useful signals come from lucky answers, and the model's style drifts in whatever direction those lucky answers point. R1-Zero lived through this phase, and its odd style (mixed languages, no summary) is partly a memory of it.
The name comes from engines. A cold engine is hard to start, so old diesel engines have a glow plug that heats the chamber first. A gas heater has a pilot light: a small flame that is always on, so the main burner lights at once. The cold-start data is R1's pilot light. A few thousand good examples make sure that, from the first RL step, some answers in each group are already readable and right.
The later long version describes the recipe in more detail. The words below are from that version, not from v1.
We can see this format in the released model. The chat template of DeepSeek-R1-Distill-Qwen-1.5B is a short Jinja program inside its tokenizer. We rewrote it in Python and compared the two on a two-turn chat.
Method 1: our own function.
python
def r1_prompt(messages, add_generation_prompt=False): """Build the prompt text the way the released template does (system, user and assistant turns; no tools).""" system = ''.join(m['content'] for m in messages if m['role'] == 'system') out = BOS + system # the system text goes first, with no tag for m in messages: if m['role'] == 'user': out += USER + m['content'] elif m['role'] == 'assistant': content = m['content'].split('</think>')[-1] # an old answer keeps only its summary out += ASSISTANT + content + EOS if add_generation_prompt: out += ASSISTANT + '<think>\n' # the model's turn opens its thinking return out
Method 2: the tokenizer's template. We call tok.apply_chat_template(chat, tokenize=False, add_generation_prompt=True) on the same chat. The first answer in the chat has a thinking part and a summary. The output (results/r1_part5_template_stdout.txt):
plain text
Method 1 (our function):'<|begin▁of▁sentence|><|User|>What is 15% of 80?<|Assistant|>\n\nThe answer is **12**.<|end▁of▁sentence|><|User|>And 15% of 60?<|Assistant|><think>\n'Method 2 (tokenizer.apply_chat_template):'<|begin▁of▁sentence|><|User|>What is 15% of 80?<|Assistant|>\n\nThe answer is **12**.<|end▁of▁sentence|><|User|>And 15% of 60?<|Assistant|><think>\n'same string: Truethe thinking of the first answer is in the prompt: False '<think>' -> token ids [151648] '</think>' -> token ids [151649] '<|User|>' -> token ids [151644] '<|Assistant|>' -> token ids [151645] '<|end▁of▁sentence|>' -> token ids [151643]
Three facts come out of this. First, <think> and </think> are single tokens (151648 and 151649), so they are the paper's "|special_token|". Second, the template itself writes <think>\n at the start of the model's turn. The model never has to decide whether to think; it is already inside the thinking part when it writes its first token. Third, an old answer in the chat history keeps only its summary. The template cuts everything up to </think>, so the next turn never sees old reasoning.
So the summary is the part meant for people, and also the part that stays in the conversation.
The v1 text gives this reward in words. The long version writes it as Equation 7:
Rlanguage=Num(Words)Num(Wordstarget)
Read it left to right. Rlanguage is the language reward, a number from 0 to 1. The top, Num(Wordstarget), counts the words of the chain of thought that are in the target language. The bottom, Num(Words), counts all its words. Only the thinking part is counted, not the summary.
The paper does not say how it counts words. English has spaces between words; Chinese has none. We made one choice and wrote it down: a run of Latin letters is one English word, and each Chinese character is one word. Digits, symbols and punctuation are not words. Here is a tiny example: an English question, "What is 15% of 80?", and a thinking part that drifts into Chinese.
Our word count on a thinking part that drifts into Chinese. Six of the eleven words are English, so the language reward is 6 / 11 = 0.545.
Method 1: our own word splitter. Python's unicodedata module knows the official Unicode name of every character. A Latin letter's name starts with "LATIN"; a Chinese character's name starts with "CJK UNIFIED IDEOGRAPH".
python
def script_of(ch): """'en' for a Latin letter, 'zh' for a CJK ideograph, None for digits, spaces, symbols and punctuation.""" if not ch.isalpha(): return None name = unicodedata.name(ch, '') if name.startswith('LATIN'): return 'en' if name.startswith('CJK UNIFIED IDEOGRAPH') or name.startswith('CJK COMPATIBILITY IDEOGRAPH'): return 'zh' return 'other' # Greek, Cyrillic, ... count as words of another languagedef words_scratch(text): """Split text into (word, language) pairs. A run of Latin letters is one English word. Chinese is written without spaces, so each Han character counts as one word.""" out, cur, cur_lang = [], '', None for ch in text: s = script_of(ch) if s in ('en', 'other'): # letters that build a word if cur and cur_lang != s: # a change of script ends the current word out.append((cur, cur_lang)); cur = '' cur, cur_lang = cur + ch, s continue if cur: # a space, digit, symbol or Han character ends it out.append((cur, cur_lang)); cur = '' if s == 'zh': out.append((ch, 'zh')) # one Han character = one word if cur: out.append((cur, cur_lang)) return outdef lang_reward_scratch(cot, target): """Reward_language = Num(Words_target) / Num(Words). A text with no words at all gets 0.""" ws = words_scratch(cot) if not ws: return 0.0, 0, 0 n_target = sum(1 for _, lang in ws if lang == target) return n_target / len(ws), n_target, len(ws)
Method 2: the regex library. It understands Unicode script classes: \p{Han} matches a Chinese character and \p{Latin} a Latin letter. One pattern finds all the words at once.
python
WORD = regex.compile(r'\p{Han}|\p{Latin}+|[^\P{L}\p{Latin}\p{Han}]+') # one Han char, a Latin run, or a run of other lettersdef lang_reward_regex(cot, target): ws = WORD.findall(cot) if not ws: return 0.0, 0, 0 pat = {'en': r'\p{Latin}+', 'zh': r'\p{Han}'}[target] n_target = sum(1 for w in ws if regex.fullmatch(pat, w)) return n_target / len(ws), n_target, len(ws)
We ran both on four thinking parts: English for English, English drifting into Chinese, Chinese with two English words, and English for a Chinese question. The output (results/r1_part5_lang_stdout.txt):
plain text
tiny example: how Method 1 splits case B words: First/en of/en is/en 然/zh 后/zh 是/zh 所/zh 以/zh Let/en me/en check/enA: English question, English thinking target=en method 1: 13/13 = 1.000 method 2: 13/13 = 1.000 agree: TrueB: English question, thinking drifts into Chinese target=en method 1: 6/11 = 0.545 method 2: 6/11 = 0.545 agree: TrueC: Chinese question, Chinese thinking with two English terms target=zh method 1: 12/14 = 0.857 method 2: 12/14 = 0.857 agree: TrueD: Chinese question, thinking all in English target=zh method 1: 0/9 = 0.000 method 2: 0/9 = 0.000 agree: Trueboth methods agree on all 4 strings: True
Case D is the interesting one. The thinking is perfect English and fully correct, but the question was in Chinese, so the reward is 0. The reward is not "prefer English". It is "answer in the language you were asked in". The paper's later limitations section notes that R1 is tuned for Chinese and English only. A question in a third language can still get English thinking.
Here is the sum as an equation, as v1 describes it:
ri=riacc+Rlanguage(oi)
Here ri is the reward of answer oi, the i-th answer in a group. The accuracy part riacc is 1 if its final answer is right and 0 if not. The last term is the language reward of its thinking. So the reward now ranges from 0 to 2.
Why does a small extra number change anything? Because GRPO compares answers inside a group. We took four answers to "What is 15% of 80?": right in English, right but mixed, wrong in English, wrong and mixed. Then we computed the GRPO advantage of Part 2 twice: with the summed reward and with accuracy alone.
plain text
final reward of §2.3.2 for a group of 4 answers to the English question (accuracy 1 or 0, plus language) right, English accuracy=1 language=1.000 reward=2.000 advantage=+1.097 right, mixed accuracy=1 language=0.545 reward=1.545 advantage=+0.449 wrong, English accuracy=0 language=1.000 reward=1.000 advantage=-0.328 wrong, mixed accuracy=0 language=0.375 reward=0.375 advantage=-1.219 accuracy alone gives advantages +0.866, +0.866, -0.866, -0.866 : the two right answers look the same (advantages from group_advantages() of r1_part2_grpo.py: std over G - 1, eps 1e-4)
The advantages come from the same group_advantages() function as in Part 2, so the standard deviation divides by G−1. That is why accuracy alone gives ±0.866 here and not ±1.
With accuracy alone, the clean answer and the mixed answer get the same push, +0.866. With the sum, the clean one gets +1.097 and the mixed one only +0.449. In this group both are still pushed up, because both are above the group's mean reward. That is not guaranteed. In a group where most answers are right and clean, a right but mixed answer can fall below the mean and get a negative advantage. The model now learns a little more from the clean one, step after step.
Note also what the sum does to right and wrong answers. Accuracy adds 1, and the language reward adds at most 1. So a right answer never gets a lower reward than a wrong one, but the two can tie. A right answer with language reward 0 gets a reward of 1. So does a wrong answer with language reward 1.
The long version gives the settings of this stage. They are not in v1.
The name comes from statistics. Suppose you want random points from a strange shape, but you only know how to draw points from a simple box around it. Throw points into the box at random. Keep the points that land inside the shape, and reject the rest. What you keep is a fair sample of the shape. John von Neumann described this "acceptance-rejection" method in 1951, and it is still a basic tool of Monte Carlo methods.
Language models use the same idea with answers instead of points. The model is the box: it is easy to sample from. "Good answers" are the shape: we cannot sample them directly, but we can check them. So sample many answers, keep the ones that pass the check, and throw away the rest. Then fine-tune on what you kept, and the model learns from its own best work.
Meta used this for Llama 2 in 2023, and DeepSeek's own maths work used it too.
We ran the whole stage-3 recipe at toy size on the released DeepSeek-R1-Distill-Qwen-1.5B. It is the smallest model that R1's own 800k samples trained (Part 6), so its answers have R1's shape. We took 10 problems: 5 from the GSM8K test set and 5 from MATH-500, one of each difficulty level, picked with a fixed seed. For each problem we sampled 8 answers with the model card's advice: temperature 0.6 and top-p 0.95. The prompt ends with the line "Please reason step by step, and put your final answer within \boxed{}." Each answer could use at most 2,048 new tokens. The run used a Mac with an Apple GPU (MPS).
python
with torch.no_grad(): out = model.generate(ids, do_sample=True, temperature=TEMP, top_p=TOP_P, max_new_tokens=MAX_NEW, num_return_sequences=N_SAMPLES, pad_token_id=tok.eos_token_id)
First, keep only the correct answers. Method 1 is our own check: find the last \boxed{...}, match its braces by hand, clean both answers, and compare.
python
def last_boxed(text): """The content of the last \\boxed{...}, with nested braces matched by hand.""" k = text.rfind('\\boxed{') if k < 0: return None i, depth, start = k + len('\\boxed{'), 1, k + len('\\boxed{') while i < len(text) and depth: depth += {'{': 1, '}': -1}.get(text[i], 0) i += 1 return text[start:i - 1] if depth == 0 else Nonedef correct_scratch(text, gold): pred = last_boxed(text) if pred is None: return None, False a, b = norm(pred), norm(gold) if a == b: return pred, True try: return pred, abs(float(a) - float(b)) < 1e-6 except ValueError: return pred, False
Method 2 is the math-verify library, which parses both answers into maths and compares them as expressions. The funnel uses Method 2, and the script prints every answer where the two disagree.