DeepSeek-R1, explained · Part 5 of 8 · Covers §2.3, §2.3.1 to §2.3.4

The R1 Pipeline: Four Stages

How R1-Zero's raw skill became DeepSeek-R1: a cold start, reasoning RL with a language reward, rejection sampling into 800k examples, and a last RL stage. We run each idea in Python.

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and others. arXiv 2025 (Nature 2025), 2025. arXiv:2501.12948

R1-Zero could reason, but its thinking was often hard to read, and it switched between English and Chinese in the middle of a thought. Section 2.3 is the recipe that kept the skill and fixed the rest, in four stages and two pages of text.

DeepSeek-V3-BaseDeepSeek-V3-Basea fresh start1 Cold startSFT of V3-Base onthousands of longreadable CoT examplesno reward (SFT)2 Reasoning RLGRPO, as for R1-Zeromaths, code, science,logic promptsrules + language3 Rejection sampling600k reasoning +200k non-reasoning= 800k; SFT of V3-Basefilters, 2 epochs4 RL, all scenariosreasoning: rulesgeneral: rewardmodelshelpful + harmlessits answers, not its weightsDeepSeek-R1
The whole recipe of §2.3. DeepSeek-V3-Base enters twice: at stage 1, and again at stage 3. The dashed arrow carries data, not weights.

The pipeline in one paragraph

The introduction gives the whole recipe before §2.3 explains it. Read it once now, so the four stages below have a place to go.

Two questions after R1-Zero

Part 4 ended with R1-Zero's two problems: poor readability and language mixing. Section 2.3 opens with two questions that come straight from them.

The later long version of the paper (arXiv v2, January 2026, the text behind the Nature article) draws the same pipeline as a picture. It also names the checkpoints between the stages: Dev-1, Dev-2 and Dev-3. We use these names in this part.

Stage 1: Cold start

DeepSeek-V3-BaseDeepSeek-V3-Basea fresh start1 Cold startSFT of V3-Base onthousands of longreadable CoT examplesno reward (SFT)2 Reasoning RLGRPO, as for R1-Zeromaths, code, science,logic promptsrules + language3 Rejection sampling600k reasoning +200k non-reasoning= 800k; SFT of V3-Basefilters, 2 epochs4 RL, all scenariosreasoning: rulesgeneral: rewardmodelshelpful + harmlessits answers, not its weightsDeepSeek-R1you are here
Stage 1 fine-tunes DeepSeek-V3-Base on a few thousand long, readable chains of thought. There is no reward yet; the model copies examples.

Why RL from a raw base model starts badly

The paper says R1 avoids "the early unstable cold start phase of RL training from the base model". To see why that phase is unstable, recall how GRPO learns (Part 2). For each question it samples a group of answers and scores them. Each answer's advantage is its reward minus the group mean, divided by the group's standard deviation.

Now think of a base model at step 0. It was trained to continue web text, not to answer in a format. Most of its answers to a hard question are wrong, or they never close the </think> tag. If all eight answers in a group get reward 0, then every answer equals the mean. Every advantage is 0, and the group teaches nothing. The model learns only from the rare groups where at least one answer is right.

So the first part of training is slow and noisy. The few useful signals come from lucky answers, and the model's style drifts in whatever direction those lucky answers point. R1-Zero lived through this phase, and its odd style (mixed languages, no summary) is partly a memory of it.

The name comes from engines. A cold engine is hard to start, so old diesel engines have a glow plug that heats the chamber first. A gas heater has a pilot light: a small flame that is always on, so the main burner lights at once. The cold-start data is R1's pilot light. A few thousand good examples make sure that, from the first RL step, some answers in each group are already readable and right.

The later long version describes the recipe in more detail. The words below are from that version, not from v1.

The readable format

We can see this format in the released model. The chat template of DeepSeek-R1-Distill-Qwen-1.5B is a short Jinja program inside its tokenizer. We rewrote it in Python and compared the two on a two-turn chat.

Method 1: our own function.

python
def r1_prompt(messages, add_generation_prompt=False):
    """Build the prompt text the way the released template does (system, user and assistant turns; no tools)."""
    system = ''.join(m['content'] for m in messages if m['role'] == 'system')
    out = BOS + system                                       # the system text goes first, with no tag
    for m in messages:
        if m['role'] == 'user':
            out += USER + m['content']
        elif m['role'] == 'assistant':
            content = m['content'].split('</think>')[-1]    # an old answer keeps only its summary
            out += ASSISTANT + content + EOS
    if add_generation_prompt:
        out += ASSISTANT + '<think>\n'                       # the model's turn opens its thinking
    return out

Method 2: the tokenizer's template. We call tok.apply_chat_template(chat, tokenize=False, add_generation_prompt=True) on the same chat. The first answer in the chat has a thinking part and a summary. The output (results/r1_part5_template_stdout.txt):

plain text
Method 1 (our function):
'<|begin▁of▁sentence|><|User|>What is 15% of 80?<|Assistant|>\n\nThe answer is **12**.<|end▁of▁sentence|><|User|>And 15% of 60?<|Assistant|><think>\n'
Method 2 (tokenizer.apply_chat_template):
'<|begin▁of▁sentence|><|User|>What is 15% of 80?<|Assistant|>\n\nThe answer is **12**.<|end▁of▁sentence|><|User|>And 15% of 60?<|Assistant|><think>\n'
same string: True

the thinking of the first answer is in the prompt: False

  '<think>'                    -> token ids [151648]
  '</think>'                   -> token ids [151649]
  '<|User|>'                   -> token ids [151644]
  '<|Assistant|>'              -> token ids [151645]
  '<|end▁of▁sentence|>'        -> token ids [151643]

Three facts come out of this. First, <think> and </think> are single tokens (151648 and 151649), so they are the paper's "|special_token|". Second, the template itself writes <think>\n at the start of the model's turn. The model never has to decide whether to think; it is already inside the thinking part when it writes its first token. Third, an old answer in the chat history keeps only its summary. The template cuts everything up to </think>, so the next turn never sees old reasoning.

So the summary is the part meant for people, and also the part that stays in the conversation.

Better than R1-Zero

Stage 2: Reasoning-oriented RL

DeepSeek-V3-BaseDeepSeek-V3-Basea fresh start1 Cold startSFT of V3-Base onthousands of longreadable CoT examplesno reward (SFT)2 Reasoning RLGRPO, as for R1-Zeromaths, code, science,logic promptsrules + language3 Rejection sampling600k reasoning +200k non-reasoning= 800k; SFT of V3-Basefilters, 2 epochs4 RL, all scenariosreasoning: rulesgeneral: rewardmodelshelpful + harmlessits answers, not its weightsDeepSeek-R1you are here
Stage 2 runs the R1-Zero recipe on the cold-started model, with one new reward for language consistency.

The language consistency reward

The v1 text gives this reward in words. The long version writes it as Equation 7:

Rlanguage=Num(Wordstarget)Num(Words)R_{\text{language}} = \frac{\text{Num}(\text{Words}_{\text{target}})}{\text{Num}(\text{Words})}

Read it left to right. RlanguageR_{\text{language}} is the language reward, a number from 0 to 1. The top, Num(Wordstarget)\text{Num}(\text{Words}_{\text{target}}), counts the words of the chain of thought that are in the target language. The bottom, Num(Words)\text{Num}(\text{Words}), counts all its words. Only the thinking part is counted, not the summary.

The paper does not say how it counts words. English has spaces between words; Chinese has none. We made one choice and wrote it down: a run of Latin letters is one English word, and each Chinese character is one word. Digits, symbols and punctuation are not words. Here is a tiny example: an English question, "What is 15% of 80?", and a thinking part that drifts into Chinese.

English question: "What is 15% of 80?" The thinking, split into words:Firstofis然后是所以LetmecheckEnglish (the target): 6 wordsChinese: 5 words (one per character)Reward_language = 6 / 11 = 0.545Digits, %, +, = and . are not words, so they do not count either way.
Our word count on a thinking part that drifts into Chinese. Six of the eleven words are English, so the language reward is 6 / 11 = 0.545.

Method 1: our own word splitter. Python's unicodedata module knows the official Unicode name of every character. A Latin letter's name starts with "LATIN"; a Chinese character's name starts with "CJK UNIFIED IDEOGRAPH".

python
def script_of(ch):
    """'en' for a Latin letter, 'zh' for a CJK ideograph, None for digits, spaces, symbols and punctuation."""
    if not ch.isalpha():
        return None
    name = unicodedata.name(ch, '')
    if name.startswith('LATIN'):
        return 'en'
    if name.startswith('CJK UNIFIED IDEOGRAPH') or name.startswith('CJK COMPATIBILITY IDEOGRAPH'):
        return 'zh'
    return 'other'                                   # Greek, Cyrillic, ... count as words of another language


def words_scratch(text):
    """Split text into (word, language) pairs. A run of Latin letters is one English word.
    Chinese is written without spaces, so each Han character counts as one word."""
    out, cur, cur_lang = [], '', None
    for ch in text:
        s = script_of(ch)
        if s in ('en', 'other'):                     # letters that build a word
            if cur and cur_lang != s:                 # a change of script ends the current word
                out.append((cur, cur_lang)); cur = ''
            cur, cur_lang = cur + ch, s
            continue
        if cur:                                       # a space, digit, symbol or Han character ends it
            out.append((cur, cur_lang)); cur = ''
        if s == 'zh':
            out.append((ch, 'zh'))                    # one Han character = one word
    if cur:
        out.append((cur, cur_lang))
    return out


def lang_reward_scratch(cot, target):
    """Reward_language = Num(Words_target) / Num(Words). A text with no words at all gets 0."""
    ws = words_scratch(cot)
    if not ws:
        return 0.0, 0, 0
    n_target = sum(1 for _, lang in ws if lang == target)
    return n_target / len(ws), n_target, len(ws)

Method 2: the regex library. It understands Unicode script classes: \p{Han} matches a Chinese character and \p{Latin} a Latin letter. One pattern finds all the words at once.

python
WORD = regex.compile(r'\p{Han}|\p{Latin}+|[^\P{L}\p{Latin}\p{Han}]+')   # one Han char, a Latin run, or a run of other letters


def lang_reward_regex(cot, target):
    ws = WORD.findall(cot)
    if not ws:
        return 0.0, 0, 0
    pat = {'en': r'\p{Latin}+', 'zh': r'\p{Han}'}[target]
    n_target = sum(1 for w in ws if regex.fullmatch(pat, w))
    return n_target / len(ws), n_target, len(ws)

We ran both on four thinking parts: English for English, English drifting into Chinese, Chinese with two English words, and English for a Chinese question. The output (results/r1_part5_lang_stdout.txt):

plain text
tiny example: how Method 1 splits case B
  words: First/en of/en is/en 然/zh 后/zh 是/zh 所/zh 以/zh Let/en me/en check/en

A: English question, English thinking
  target=en  method 1: 13/13 = 1.000   method 2: 13/13 = 1.000   agree: True
B: English question, thinking drifts into Chinese
  target=en  method 1: 6/11 = 0.545   method 2: 6/11 = 0.545   agree: True
C: Chinese question, Chinese thinking with two English terms
  target=zh  method 1: 12/14 = 0.857   method 2: 12/14 = 0.857   agree: True
D: Chinese question, thinking all in English
  target=zh  method 1: 0/9 = 0.000   method 2: 0/9 = 0.000   agree: True

both methods agree on all 4 strings: True

The terminal output of r1_part5_lang.py: both methods agree on all four strings, and the summed rewards and advantages of a group of four answers

Case D is the interesting one. The thinking is perfect English and fully correct, but the question was in Chinese, so the reward is 0. The reward is not "prefer English". It is "answer in the language you were asked in". The paper's later limitations section notes that R1 is tuned for Chinese and English only. A question in a third language can still get English thinking.

Summing the two rewards

Here is the sum as an equation, as v1 describes it:

ri=riacc+Rlanguage(oi)r_i = r_i^{\text{acc}} + R_{\text{language}}(o_i)

Here rir_i is the reward of answer oio_i, the ii-th answer in a group. The accuracy part riaccr_i^{\text{acc}} is 1 if its final answer is right and 0 if not. The last term is the language reward of its thinking. So the reward now ranges from 0 to 2.

Why does a small extra number change anything? Because GRPO compares answers inside a group. We took four answers to "What is 15% of 80?": right in English, right but mixed, wrong in English, wrong and mixed. Then we computed the GRPO advantage of Part 2 twice: with the summed reward and with accuracy alone.

plain text
final reward of §2.3.2 for a group of 4 answers to the English question (accuracy 1 or 0, plus language)
  right, English  accuracy=1  language=1.000  reward=2.000  advantage=+1.097
  right, mixed    accuracy=1  language=0.545  reward=1.545  advantage=+0.449
  wrong, English  accuracy=0  language=1.000  reward=1.000  advantage=-0.328
  wrong, mixed    accuracy=0  language=0.375  reward=0.375  advantage=-1.219
  accuracy alone gives advantages +0.866, +0.866, -0.866, -0.866 : the two right answers look the same
  (advantages from group_advantages() of r1_part2_grpo.py: std over G - 1, eps 1e-4)

The advantages come from the same group_advantages() function as in Part 2, so the standard deviation divides by G−1G - 1. That is why accuracy alone gives ±0.866\pm 0.866 here and not ±1\pm 1.

With accuracy alone, the clean answer and the mixed answer get the same push, +0.866. With the sum, the clean one gets +1.097 and the mixed one only +0.449. In this group both are still pushed up, because both are above the group's mean reward. That is not guaranteed. In a group where most answers are right and clean, a right but mixed answer can fall below the mean and get a negative advantage. The model now learns a little more from the clean one, step after step.

Note also what the sum does to right and wrong answers. Accuracy adds 1, and the language reward adds at most 1. So a right answer never gets a lower reward than a wrong one, but the two can tie. A right answer with language reward 0 gets a reward of 1. So does a wrong answer with language reward 1.

The long version gives the settings of this stage. They are not in v1.

Stage 3: Rejection sampling and SFT

DeepSeek-V3-BaseDeepSeek-V3-Basea fresh start1 Cold startSFT of V3-Base onthousands of longreadable CoT examplesno reward (SFT)2 Reasoning RLGRPO, as for R1-Zeromaths, code, science,logic promptsrules + language3 Rejection sampling600k reasoning +200k non-reasoning= 800k; SFT of V3-Basefilters, 2 epochs4 RL, all scenariosreasoning: rulesgeneral: rewardmodelshelpful + harmlessits answers, not its weightsDeepSeek-R1you are here
Stage 3 turns the RL checkpoint into a writer of data, then fine-tunes a fresh copy of DeepSeek-V3-Base on 800k samples.

What rejection sampling means

The name comes from statistics. Suppose you want random points from a strange shape, but you only know how to draw points from a simple box around it. Throw points into the box at random. Keep the points that land inside the shape, and reject the rest. What you keep is a fair sample of the shape. John von Neumann described this "acceptance-rejection" method in 1951, and it is still a basic tool of Monte Carlo methods.

Language models use the same idea with answers instead of points. The model is the box: it is easy to sample from. "Good answers" are the shape: we cannot sample them directly, but we can check them. So sample many answers, keep the ones that pass the check, and throw away the rest. Then fine-tune on what you kept, and the model learns from its own best work.

Meta used this for Llama 2 in 2023, and DeepSeek's own maths work used it too.

Our rejection sampling run

We ran the whole stage-3 recipe at toy size on the released DeepSeek-R1-Distill-Qwen-1.5B. It is the smallest model that R1's own 800k samples trained (Part 6), so its answers have R1's shape. We took 10 problems: 5 from the GSM8K test set and 5 from MATH-500, one of each difficulty level, picked with a fixed seed. For each problem we sampled 8 answers with the model card's advice: temperature 0.6 and top-p 0.95. The prompt ends with the line "Please reason step by step, and put your final answer within \boxed{}." Each answer could use at most 2,048 new tokens. The run used a Mac with an Apple GPU (MPS).

python
with torch.no_grad():
    out = model.generate(ids, do_sample=True, temperature=TEMP, top_p=TOP_P, max_new_tokens=MAX_NEW,
                         num_return_sequences=N_SAMPLES, pad_token_id=tok.eos_token_id)

First, keep only the correct answers. Method 1 is our own check: find the last \boxed{...}, match its braces by hand, clean both answers, and compare.

python
def last_boxed(text):
    """The content of the last \\boxed{...}, with nested braces matched by hand."""
    k = text.rfind('\\boxed{')
    if k < 0:
        return None
    i, depth, start = k + len('\\boxed{'), 1, k + len('\\boxed{')
    while i < len(text) and depth:
        depth += {'{': 1, '}': -1}.get(text[i], 0)
        i += 1
    return text[start:i - 1] if depth == 0 else None


def correct_scratch(text, gold):
    pred = last_boxed(text)
    if pred is None:
        return None, False
    a, b = norm(pred), norm(gold)
    if a == b:
        return pred, True
    try:
        return pred, abs(float(a) - float(b)) < 1e-6
    except ValueError:
        return pred, False

Method 2 is the math-verify library, which parses both answers into maths and compares them as expressions. The funnel uses Method 2, and the script prints every answer where the two disagree.

python
def correct_mv(text, gold):
    from math_verify import parse, verify
    try:
        return bool(verify(parse('
  

 + gold + '
  

) or parse(gold), parse(text)))
    except Exception:
        return False

Then the three filters, on the thinking part only. The mixed-language filter reuses the language reward from stage 2. The paper gives no thresholds, so these are ours. A language reward below 0.99 is "mixed", and a paragraph of more than 120 words is "long".

python
def paragraph_words(thinking):
    return [len(p.split()) for p in re.split(r'\n\s*\n', thinking) if p.strip()]


def filters(thinking):
    lc, _, _ = lang_reward_scratch(thinking, 'en')
    longest = max(paragraph_words(thinking) or [0])
    return dict(lang=round(lc, 4), longest_para=longest, code='```' in thinking)

The run took 11.6 minutes for 80 answers. Here is the score of each group (results/r1_part5_reject_stdout.txt):

plain text
per problem: correct answers out of 8 (math-verify), mean length in tokens
  gsm8k-1275             correct 8/8   finished 8/8   mean tokens 979
  gsm8k-523              correct 7/8   finished 8/8   mean tokens 1258
  gsm8k-734              correct 4/8   finished 4/8   mean tokens 1695
  gsm8k-1085             correct 8/8   finished 8/8   mean tokens 1429
  gsm8k-59               correct 7/8   finished 7/8   mean tokens 1099
  math500-L1-362         correct 7/8   finished 7/8   mean tokens 1313
  math500-L2-120         correct 0/8   finished 0/8   mean tokens 2048
  math500-L3-396         correct 0/8   finished 0/8   mean tokens 2048
  math500-L4-71          correct 0/8   finished 0/8   mean tokens 2048
  math500-L5-166         correct 0/8   finished 0/8   mean tokens 2048
groups where all 8 rewards are equal (GRPO advantage 0 for every answer): 6 of 10

Our limit of 2,048 tokens is the main filter here. On the four harder MATH-500 problems, not one of the 32 answers closed its thinking in time. That is not a fault of the model: the paper lets R1 think for up to 32,768 tokens. It is the price of a toy run on a laptop.

The last line connects back to the cold start. In 6 of our 10 groups, all eight answers got the same reward: two groups all right, four all wrong. GRPO learns nothing from such a group. A raw base model at step 0 is in the "all wrong" case for almost every hard question, which is why a cold start helps.

Now the funnel. Each line is the number of answers left after one more sieve.

plain text
the funnel
   80  sampled   (10 problems)
   42  finished (</think> and an end token)   (6 problems)
   41  correct (math-verify)   (6 problems)
   41  no mixed language (language reward >= 0.99)   (6 problems)
   40  no long paragraph (<= 120 words)   (6 problems)
   39  no code block   (6 problems)

among the 41 correct answers: language reward min 1.000; longest paragraph max 122 words; code blocks 1
paragraphs in their thinking: 923, median 26 words, over 120 words: 1
sampled (10 problems x 8)sampled (10 problems x 8): 8080finished thinkingfinished thinking: 4242correctcorrect: 4141no mixed languageno mixed language: 4141no long paragraphno long paragraph: 4040no code block (kept)no code block (kept): 3939
Our rejection-sampling funnel on DeepSeek-R1-Distill-Qwen-1.5B: 80 answers sampled, 42 finished their thinking, 41 were correct, and 39 passed the three readability filters.

The two answer checks did not fully agree: they agree on 35 of the 42 finished answers. All seven differences are on one problem:

plain text
  methods disagree on math500-L1-362 #0: ours=False math-verify=True pred='\\dfrac{1}{4}' gold='\\frac14'

The model wrote \dfrac{1}{4} and the reference says \frac14. Both mean one quarter. Our string cleaner turns \dfrac into \frac, but it does not know that \frac14 is short for \frac{1}{4}. The library math-verify parses both into the number 0.25 and sees they are equal. This is why real pipelines use a parser; the long version names sympy. It is also why the paper adds DeepSeek-V3 as a judge for answers that rules cannot compare.

The readability filters removed only two answers. Every correct answer had a language reward of 1.000: this distilled model was trained on R1's cleaned data, so it already thinks in one language. One answer had a paragraph of 122 words, and one had a code block. The paper's filters bite much harder on a raw RL checkpoint like Dev-2 than on a model that already learned from filtered data.

The kept answers become a tiny SFT file, one JSON record per line, in the chat format that the template of stage 1 reads. Here is the shortest record, with the middle of the thinking cut for space:

plain text
{
 "id": "gsm8k-59-7",
 "messages": [
  {
   "role": "user",
   "content": "A raspberry bush has 6 clusters of 20 fruit each and 67 individual fruit scattered across the bush. How many raspberries are there total?\nPlease reason step by step, and put your final answer within \\boxed{}."
  },
  {
   "role": "assistant",
   "content": "<think>\nOkay, so I have this problem about a raspberry bush, and I need to figure out how many raspberries there are in total. Let me read it again: \"A raspberry bush has 6 clusters of 20 fruit each and 67 individual fruit scattered across the bush. How many r [...]  \\times 20 \\text{ raspberries per cluster} = 120 \\text{ raspberries} \\]\n\nAdditionally, there are 67 individual fruits scattered across the bush, each of which is a raspberry. Therefore, the number of raspberries from the scattered fruits is:\n\n\\[ 67 \\text{ individual fruits} \\]\n\nAdding these together gives the total number of raspberries:\n\n\\[ 120 + 67 = 187 \\]\n\nThus, the total number of raspberries is \\(\\boxed{187}\\)."
  }
 ]
}

Look at its shape. The thinking opens with <think>, reads the problem again, and works in short paragraphs. After </think> comes a clean summary with the boxed answer. That is the format of stage 1, written by the model itself. DeepSeek ran this same loop at a scale of about 600k kept answers; we kept 39.

The terminal output of r1_part5_reject.py: the per-problem scores, the disagreements of the two answer checks, and the funnel

The data mix: 600k + 200k = 800k

The long version gives the split of the 800k by domain. Maths and code are three quarters of it.

Why stage 3 starts again from V3-Base

V3-Basecold-start SFTR1 Dev-1reasoning RLR1 Dev-2weights that learnsample many answers, keepthe correct, readable onesDev-2 stops hereV3-BaseSFT, 2 epochsR1 Dev-3RL, all scenariosDeepSeek-R1a second start from the same base600k + 200k from V3 = 800k samples
The weights of the RL checkpoint Dev-2 stop at stage 2. Dev-2 only writes data. A fresh copy of DeepSeek-V3-Base learns from that data, together with 200k samples from DeepSeek-V3. Checkpoint names are from the long version.

This is the most surprising choice in the section. Dev-2 is the best reasoner so far. Why not fine-tune Dev-2 on the 800k samples and continue? The paper does not say. Here is our reading, built from facts in the paper.

The data carries the skill. The 600k reasoning samples are Dev-2's best answers, checked and cleaned. A model that learns to write them has learned what Dev-2 can do, without Dev-2's weights. Part 6 proves that this works: the same data turns a 1.5B Qwen model into a strong reasoner with SFT alone. If the data can carry the skill into a different model, it can carry it into a fresh copy of the same base.

The weights carry side effects. Dev-2 spent thousands of RL steps on maths, code, science and logic prompts only. RL also pushes a model's style in odd directions, and stage 2 paid "a slight degradation" for the language reward. Fine-tuning Dev-2 would keep all of that and then add writing on top. Starting from the base gives one clean training run in which reasoning and general skills learn side by side, from data chosen on purpose.

A clean mix is easier to control. With one SFT run on 800k known samples, the authors decide the exact mix. They choose how much maths, how much writing, and which answers have no thinking at all. Each fine-tuning stage on top of another can erase earlier skills. One run from the base avoids that chain of stages.

A teacher analogy helps. Dev-2 is a brilliant but odd student who has only done exam problems for a year. Instead of sending that student to writing classes, you ask the student to write a clean answer book. Then a new student learns from the answer book and from a writing course at the same time. The second student ends up with the skill and none of the odd habits. Table 3 above supports this: Dev-3 scores 78.1 on AIME, above Dev-2's 74.0, and 62.1 on AlpacaEval 2.0, above Dev-2's 55.8.

Stage 4: RL for all scenarios

DeepSeek-V3-BaseDeepSeek-V3-Basea fresh start1 Cold startSFT of V3-Base onthousands of longreadable CoT examplesno reward (SFT)2 Reasoning RLGRPO, as for R1-Zeromaths, code, science,logic promptsrules + language3 Rejection sampling600k reasoning +200k non-reasoning= 800k; SFT of V3-Basefilters, 2 epochs4 RL, all scenariosreasoning: rulesgeneral: rewardmodelshelpful + harmlessits answers, not its weightsDeepSeek-R1you are here
Stage 4 runs RL on a mix of prompts: rules grade the reasoning prompts, and reward models grade the general ones.

What a preference reward model is

A preference reward model is a language model with one extra output: a single number, the score of an answer. It is trained on pairs. People (or a strong model) see two answers to one prompt and pick the better one.

The classic training rule is the Bradley-Terry model from 1952. The chance that answer A beats answer B is σ(rA−rB)\sigma(r_A - r_B). Here σ\sigma is the sigmoid, and rA,rBr_A, r_B are the two scores. The Background page shows where this came from in RLHF.

The long version tells how these reward models were built. The details below are from that version.

What did the last stage buy? The long version's Table 3 (above) shows it. From Dev-3 to R1, AIME moves only from 78.1 to 79.8, but AlpacaEval 2.0 goes from 62.1 to 87.6 and ArenaHard from 75.6 to 92.3. Reasoning was mostly done by stage 3. Stage 4 made R1 pleasant to talk to.

The paper and the code

Paper (arXiv 2501.12948) · Official repository · Our scripts for this series