DeepSeek-R1, explained · Part 4 of 8 · Covers §2.2.4, Table 2, Figure 2, Figure 3, Table 3

R1-Zero and the Aha Moment

What pure RL gave DeepSeek-R1-Zero, why its answers grew from hundreds to thousands of tokens, and the moment it wrote Wait. We test the released 1.5B distill.

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and others. arXiv 2025 (Nature 2025), 2025. arXiv:2501.12948

Parts 2 and 3 built the machine: GRPO, two rule-based rewards and a one-paragraph template. This part switches it on. Out comes a model that scores close to OpenAI's o1. Its answers grow from hundreds to thousands of tokens. And one day it stops in the middle of its algebra to write "Wait, wait. Wait."

DeepSeek-V3-Basepre-trained, no reasoning SFTRL with GRPOrule rewards + template (Parts 2, 3)DeepSeek-R1-Zerothis part: what came outDeepSeek-R1 (Part 5)fixes readabilityYou are here: the result of pure RL on the base model
Where this part sits. DeepSeek-V3-Base goes through RL with GRPO, the two rule-based rewards and the template of Parts 2 and 3. What comes out is DeepSeek-R1-Zero. Its problems with readability lead to DeepSeek-R1 in Part 5.

Performance of DeepSeek-R1-Zero

How the scores are counted

The 71.0% is not one run of the model. It is an average over many answers. The rule is in the evaluation section of the paper.

A tiny example with two questions and k=4k = 4 answers each makes the two numbers concrete.

questionright answerthe 4 final answersp1…p4p_1 \dots p_4pass@1votevote right?
A4242, 42, 17, 421, 1, 0, 13/4 = 0.7542yes
B55, 7, 9, 71, 0, 0, 01/4 = 0.257no
mean0.501/2 = 0.50

Question B shows that a vote can also lose. The model found the right answer once, but its wrong answers agreed with each other, and the wrong value won. A vote helps when the right answer is the most common one, even if it is not the majority of all answers.

Method 1: from scratch. The paper's formula, written as a loop, and a vote over our own normalised answer strings:

python
def pass_at_1_m1(groups):
    """The paper's formula, page 12: pass@1 = (1/k) * sum_i p_i for each question, then the mean over questions."""
    per_q = []
    for g in groups:                              # g: the k correctness values (0 or 1) of one question
        per_q.append(sum(g) / len(g))
    return sum(per_q) / len(per_q)


def cons_m1(answers, gold):
    """Majority vote: the most common normalised answer wins (ties: the one seen first). None votes do not count."""
    votes = [normalise(a) for a in answers if a is not None]
    if not votes:
        return False, None
    counts = {}
    for v in votes:
        counts[v] = counts.get(v, 0) + 1
    best = max(counts, key=lambda v: (counts[v], -votes.index(v)))
    return best == normalise(gold), best

pass_at_1_m1 takes, for each question, the list of kk correctness values and averages it. Then it averages over the questions. cons_m1 counts how often each final answer occurs and keeps the most common one. normalise removes spaces, \left, \dfrac and similar, so that \dfrac{1}{4} and \frac14 count as one answer.

Method 2: with libraries. NumPy does the mean in one line. For the vote, two answers join one class when the math-verify library says they are equal as mathematics. It parses LaTeX with SymPy, so 0.25 and \frac{1}{4} vote together:

python
def pass_at_1_m2(groups):
    return float(np.mean(np.array(groups, dtype=float)))      # [questions, k] -> one number


def cons_m2(answers, gold):
    """Majority vote where two answers are one class if math-verify says they are equal."""
    classes = []                                  # [representative, count]
    for a in answers:
        if a is None:
            continue
        for c in classes:
            if same_m2(c[0], a):
                c[1] += 1
                break
        else:
            classes.append([a, 1])
    if not classes:
        return False, None
    best = max(classes, key=lambda c: c[1])[0]     # max keeps the first class on a tie
    return correct_m2(best, gold), best

The grading of each answer is also done twice: our own string compare, and math-verify. One policy decides what we grade, and Part 6 uses the same one.

An answer that hits the 2,048-token limit has no final answer, so it counts as wrong. For the distill, we grade only the text after </think>: the summary it writes once it has finished thinking. A distill answer that never closes </think> counts as wrong. In a finished answer, the final answer is the last \boxed{}. Here is what the run printed (the full log is results/r1_part4_stdout.txt):

plain text
20 MATH-500 problems x 8 samples, temperature 0.6, top-p 0.95, max_new_tokens 2048

== Qwen2.5-Math-1.5B (base, R1-Zero template): 160 answers
grading: ours and math-verify agree on 153/160; ours says 94 correct, math-verify 101 (we use math-verify)
   differ: problem 388 gold '\\frac{11}{2}' pred '5.5' ours False math-verify True
   differ: problem 388 gold '\\frac{11}{2}' pred '5.5' ours False math-verify True
   differ: problem 388 gold '\\frac{11}{2}' pred '5.5' ours False math-verify True
   differ: problem 388 gold '\\frac{11}{2}' pred '5.5' ours False math-verify True
   differ: problem 388 gold '\\frac{11}{2}' pred '5.5' ours False math-verify True
   differ: problem 456 gold '-2,1' pred '1, -2' ours False math-verify True
   differ: problem 456 gold '-2,1' pred '1, -2' ours False math-verify True
truncated at 2048 tokens: 12/160 (scored as wrong); finished but no final answer found: 11/160
answers that finished inside 2048 tokens: 148/160, of which correct: 101/148 = 0.682
pass@1 (mean of 8 samples per problem): Method 1 0.6312, Method 2 numpy 0.6312, agree: True
cons@8 (majority vote): Method 1 0.7000 (14/20), Method 2 0.7500 (15/20)
pass@8 (at least one of 8 correct): 0.8000 (16/20)
correct answers out of 8, per problem: 8 6 8 6 6 7 6 7 1 8 6 6 0 7 0 8 5 0 0 6

== DeepSeek-R1-Distill-Qwen-1.5B: 160 answers
grading: ours and math-verify agree on 160/160; ours says 61 correct, math-verify 61 (we use math-verify)
truncated at 2048 tokens: 96/160 (scored as wrong); finished but no final answer found: 0/160
answers that finished inside 2048 tokens: 64/160, of which correct: 61/64 = 0.953
pass@1 (mean of 8 samples per problem): Method 1 0.3812, Method 2 numpy 0.3812, agree: True
cons@8 (majority vote): Method 1 0.6000 (12/20), Method 2 0.6000 (12/20)
pass@8 (at least one of 8 correct): 0.6000 (12/20)
correct answers out of 8, per problem: 5 7 3 0 0 8 3 2 0 8 2 6 0 8 0 5 0 0 0 4

Terminal output of r1_part4_analyze.py with the grading, pass@1 and vote lines of both models

Start with the checks. Pass@1 from the loop and from NumPy agree to the last digit for both models. The two graders agree on 153 of 160 base answers and on all 160 distill answers.

Every difference is a case our strings miss and math-verify gets right: 5.5 against the reference \frac{11}{2}, and 1, -2 against -2,1. The same split explains why the two votes differ for the base model (14 against 15 problems). Our string vote treats 5.5 and \frac{11}{2} as two different answers. We keep the math-verify grades.

Now the surprise. The base model scores 0.631 and the reasoning model only 0.381. The reason is our length limit. The distill hit 2,048 tokens on 96 of its 160 answers, and a cut-off answer has no final answer, so it counts as wrong. Of the 64 distill answers that did finish, 61 were right: 95.3%. The base model writes short answers (median 518 tokens) and finished 148 of 160.

The paper gives this same distill 83.9% pass@1 on MATH-500 (Table 5, page 14), with a limit of 32,768 tokens. So our low number is not a fault of the model. It is a lesson about reasoning models: they need room to think. Cutting one off at 2,048 tokens is like taking the exam paper away halfway through. It is also why the paper sets its limit so high.

The vote helps both models. For the distill, pass@1 is 0.381 but the vote over 8 answers is right on 12 of 20 problems (0.600). When the distill can solve a problem, it usually solves it in a few of its 8 tries. The right answer then wins the vote. Keep the sample size in mind: with 20 problems, one problem moves a score by 5 points.

Table 2, column by column

AIME pass@1o1-mini, AIME pass@1: 63.663.6o1-minio1-0912, AIME pass@1: 74.474.4o1-0912R1-Zero, AIME pass@1: 71.071.0R1-ZeroAIME cons@64o1-mini, AIME cons@64: 80.080.0o1-minio1-0912, AIME cons@64: 83.383.3o1-0912R1-Zero, AIME cons@64: 86.786.7R1-ZeroMATH-500 pass@1o1-mini, MATH-500 pass@1: 90.090.0o1-minio1-0912, MATH-500 pass@1: 94.894.8o1-0912R1-Zero, MATH-500 pass@1: 95.995.9R1-ZeroGPQA Diamondo1-mini, GPQA Diamond: 60.060.0o1-minio1-0912, GPQA Diamond: 77.377.3o1-0912R1-Zero, GPQA Diamond: 73.373.3R1-ZeroLiveCodeBencho1-mini, LiveCodeBench: 53.853.8o1-minio1-0912, LiveCodeBench: 63.463.4o1-0912R1-Zero, LiveCodeBench: 50.050.0R1-ZeroCodeforces ratingo1-mini, Codeforces rating: 18201820o1-minio1-0912, Codeforces rating: 18431843o1-0912R1-Zero, Codeforces rating: 14441444R1-ZeroTable 2 redrawn: higher is better in every panel; the best value of each panel is bold
Table 2 redrawn as bars. R1-Zero (blue) is best in two panels: AIME with a vote of 64 (86.7) and MATH-500 (95.9). It is a little below o1-0912 on AIME pass@1 and GPQA Diamond, and far below both o1 models on LiveCodeBench (50.0) and the Codeforces rating (1444).

Two of the columns are new in this series.

Now the six columns, against o1-0912:

  • AIME pass@1: 71.0 against 74.4. R1-Zero is 3.4 points below. It is above o1-mini (63.6).
  • AIME cons@64: 86.7 against 83.3. With a vote over 64 answers, R1-Zero is 3.4 points above. Both models use the same vote here, so this is a fair comparison.
  • MATH-500: 95.9 against 94.8. A small win, on easier problems where both models are close to the top.
  • GPQA Diamond: 73.3 against 77.3. Science questions that need knowledge as well as reasoning. R1-Zero is 4.0 points below, but far above o1-mini (60.0).
  • LiveCodeBench: 50.0 against 63.4. The largest gap: 13.4 points. R1-Zero is also below o1-mini (53.8).
  • Codeforces: 1444 against 1843. About 400 rating points below, and 376 below o1-mini (1820).

The v1 paper does not explain the weak code scores of R1-Zero. We can only point to two facts. The finished DeepSeek-R1 does much better on the same two tests: 65.9 on LiveCodeBench and a rating of 2029 (Table 4, page 13). And the conclusion of the paper says that RL was not yet used much for software engineering tasks, because testing code is slow (page 16). Both facts fit the reading that R1-Zero's RL mostly sharpened mathematics, but the paper does not state it.

Figure 2: the training curve

Why 16 answers per problem? With 30 problems, one answer per problem gives a very noisy score. Our script prints how noisy:

plain text
AIME decoding noise on 30 fixed questions (each question p(1-p) <= 0.25): 1 sample per question, standard error at most 9.1 points; 16 independent samples per question, at most 2.3 points (an upper bound; it leaves out the noise of which 30 questions were chosen)

A score that moves 9 points by chance cannot show a curve. The 30 questions are fixed, and the 16 answers to one question are independent draws from the model. Each question has some true rate of success pp, and one draw has variance p(1−p)p(1-p), which is never more than 0.25. So with 16 answers per question, the noise from sampling is at most 2.3 points, four times less than with one answer.

This number covers only the dice rolls of decoding. It says nothing about a second kind of noise: a different set of 30 questions would give a different score. More answers per question do not reduce that noise.

0%20%40%60%80%100%pass@1 at the start: 15.6%pass@1 at the end: 71.0%majority vote at the end: 86.7%o1-0912 pass@1 74.4%o1-0912 cons@64 83.3%RL training steps (start to end, thousands of steps)AIME 2024 accuracy
The shape of Figure 2, drawn only from numbers in the text: pass@1 starts at 15.6% and ends at 71.0%; the vote ends at 86.7%; the dashed lines are o1-0912 at 74.4% and 83.3% from Table 2. The straight line only joins the two stated points. The real, bumpy curve is in the screenshot above.

The self-evolution process

hundreds of tokensthousands of tokensRL training stepsaverage length of a responseno length reward, no instruction to think longer
The shape of Figure 3 in the words of the text: from hundreds of tokens to thousands of tokens, with no reward for length. The curve is drawn by hand; the screenshot above has the real one.

Why a correctness reward makes answers longer

Nobody rewarded length, yet length grew. Why? Think of a student whose tests are graded only on the final answer, with no marks for working. One week she checks her sums before she hands in. Her grades go up. Nobody told her to write more, but checking is now a habit, and her papers get longer.

The paper's reading is that RL does the same thing, with numbers. GRPO samples a group of answers to each question (Part 2). Some answers check a step or try a second method, and so they are longer. On a hard question, those are more often right. GRPO gives each answer an advantage: its reward minus the group mean, divided by the group's standard deviation (equation 3).

answer 1answer 1: 300 tokens, reward 0300 tokensreward 0A = -1made less likelyanswer 2answer 2: 450 tokens, reward 0450 tokensreward 0A = -1made less likelyanswer 3answer 3: 1200 tokens, reward 11200 tokensreward 1A = +1made more likelyanswer 4answer 4: 1500 tokens, reward 11500 tokensreward 1A = +1made more likelyOne question, four sampled answers (made-up lengths)mean reward 0.5, standard deviation 0.5, so A = (r - 0.5) / 0.5 = +1 or -1 (equation 3)
One question, four sampled answers with made-up lengths. The two short answers are wrong (reward 0) and the two long ones are right (reward 1). The mean reward is 0.5 and the standard deviation 0.5 (dividing by G), so the advantages are -1, -1, +1, +1. The code in Part 2 divides by G-1, as PyTorch does by default, which gives -0.866 and +0.866; the direction of the update is the same. The update makes the long, careful answers more likely and the short guesses less likely.

Work the tiny example by hand. The rewards are 0, 0, 1, 1, with mean 0.5. Each reward is 0.5 away from the mean, so the standard deviation is 0.5.

The advantages are (0−0.5)/0.5=−1(0 - 0.5)/0.5 = -1 for the two wrong answers and (1−0.5)/0.5=+1(1 - 0.5)/0.5 = +1 for the two right ones. The update raises the probability of every token of the right answers. That includes the tokens "let me verify this" and the second method. It lowers the probability of the quick guesses.

One step changes the model very little. But the change has a direction: toward the kind of answer that checks. After thousands of steps the typical answer checks a lot, and it is long. Nothing pushes the other way. The reward has no cost per token. The only limit is the maximum length of an answer.

Three honest notes complete the picture. The first one comes from DeepSeek itself.

The second note is about the optimizer. In the GRPO of DeepSeekMath, the loss of each answer is averaged over its own tokens. A later study found that this average has a side effect.

The third note is our own data. We can ask the question of the tiny example on real answers. Inside one group, are the right answers longer or shorter than the wrong ones?

plain text
== Qwen2.5-Math-1.5B (base, R1-Zero template)
length (new tokens): mean 687, median 518; correct answers mean 502 (n=101), wrong answers mean 1005 (n=59), wrong and finished mean 739 (n=47)
length histogram, bins of 256 tokens: correct [23, 38, 18, 14, 7, 1, 0, 0], wrong [8, 11, 7, 9, 5, 4, 0, 15]; numpy agrees: True
GRPO view, groups with mixed results: positive-advantage answers mean length 609 (n=69), negative-advantage answers 785 (n=27)

== DeepSeek-R1-Distill-Qwen-1.5B
length (new tokens): mean 1834, median 2048; correct answers mean 1497 (n=61), wrong answers mean 2042 (n=99), wrong and finished mean 1865 (n=3)
length histogram, bins of 256 tokens: correct [0, 0, 0, 3, 13, 20, 10, 15], wrong [0, 0, 0, 0, 0, 0, 1, 98]; numpy agrees: True
GRPO view, groups with mixed results: positive-advantage answers mean length 1605 (n=37), negative-advantage answers 2048 (n=35)

Our data does not show "longer is better" inside a group. It shows the opposite.

In both models the right answers are shorter on average than the wrong ones. The base model: 502 against 1,005 tokens. The distill: 1,497 against 2,042. In the groups with mixed results, the answers that GRPO would reward are shorter too (609 against 785, and 1,605 against 2,048). One GRPO step on these groups would push toward shorter answers, not longer.

Why does this not contradict Figure 3? Look at where the wrong answers are. Many of them are long because the model got lost: it went round in circles until our limit cut it off. Even the base model's wrong answers that did finish are longer than its right ones (739 against 502 tokens). A model that does not know the way talks the longest. These long wrong answers are exactly the ones that the length bias of Liu et al. punishes too gently.

So our snapshot cannot tell which way a training run would move. It shows that length and correctness are tangled: careful checking makes an answer longer, and so does being lost. The paper reports the result of thousands of steps on harder problems, with a much higher limit. The story "checking pays, and checking takes tokens" is the authors' reading, and it is plausible. But a laptop run of two fixed models cannot prove it.

0204060801000 to 256 tokens, correct: 00 to 256 tokens, wrong: 0256 to 512 tokens, correct: 0256 to 512 tokens, wrong: 0512 to 768 tokens, correct: 0512 to 768 tokens, wrong: 0768 to 1024 tokens, correct: 33768 to 1024 tokens, wrong: 01024 to 1280 tokens, correct: 13131024 to 1280 tokens, wrong: 01280 to 1536 tokens, correct: 20201280 to 1536 tokens, wrong: 01536 to 1792 tokens, correct: 10101536 to 1792 tokens, wrong: 111792 to 2048 tokens, correct: 15151792 to 2048 tokens, wrong: 989802565127681,0241,2801,5361,7922,048answer length in new tokens (the last bin ends at our 2,048 limit)DeepSeek-R1-Distill-Qwen-1.5B, 160 answerscorrect (61)wrong (99)
Lengths of the distill's 160 answers in bins of 256 tokens. 98 of the 99 wrong answers sit in the last bin, and 96 of them were cut off by our 2,048-token limit. The 61 right answers spread from 768 tokens to the limit.
0102030400 to 256 tokens, correct: 23230 to 256 tokens, wrong: 88256 to 512 tokens, correct: 3838256 to 512 tokens, wrong: 1111512 to 768 tokens, correct: 1818512 to 768 tokens, wrong: 77768 to 1024 tokens, correct: 1414768 to 1024 tokens, wrong: 991024 to 1280 tokens, correct: 771024 to 1280 tokens, wrong: 551280 to 1536 tokens, correct: 111280 to 1536 tokens, wrong: 441536 to 1792 tokens, correct: 01536 to 1792 tokens, wrong: 01792 to 2048 tokens, correct: 01792 to 2048 tokens, wrong: 151502565127681,0241,2801,5361,7922,048answer length in new tokens (the last bin ends at our 2,048 limit)Qwen2.5-Math-1.5B with the R1-Zero template, 160 answerscorrect (101)wrong (59)
Lengths of the base model's 160 answers with the R1-Zero template. 61 of the 101 right answers are shorter than 512 tokens. The 15 wrong answers in the last bin include the 12 that our limit cut off.

Reflection and new approaches

The v1 paper describes these behaviours but does not count them. The later version does.

Our run counts four phrases in every answer of both small models. Method 1 uses regular expressions; Method 2 splits the text into words and counts them with collections.Counter:

python
REFLECT = {                                   # the four phrases we count; lower case, word boundaries
    'wait': r'\bwait\b',
    'let me check': r'\blet me check\b',
    'alternatively': r'\balternatively\b',
    're-evaluate': r'\bre-?evaluat\w*',
}


def reflect_m1(text):
    t = text.lower()
    return {k: len(re.findall(p, t)) for k, p in REFLECT.items()}


def reflect_m2(text):
    """Split into lower-case word tokens and count with Counter; two-word phrases from pairs of tokens."""
    toks = re.findall(r"[a-z]+(?:-[a-z]+)?", text.lower())
    c1 = Counter(toks)
    c2 = Counter(zip(toks, toks[1:], toks[2:]))
    return {'wait': c1['wait'], 'let me check': c2[('let', 'me', 'check')], 'alternatively': c1['alternatively'],
            're-evaluate': sum(n for w, n in c1.items() if w.startswith(('reevaluat', 're-evaluat')))}

\b in a regular expression means "word boundary", so \bwait\b counts "Wait," and "wait." but not "awaiting". Method 2 gets the same effect a different way: it cuts the text into words first, so "awaiting" is one word and never equals "wait". The two methods must agree on every total.

plain text
== Qwen2.5-Math-1.5B (base, R1-Zero template)
reflection words, total over 160 answers: Method 1 {'wait': 0, 'let me check': 0, 'alternatively': 0, 're-evaluate': 0}
                                              Method 2 {'wait': 0, 'let me check': 0, 'alternatively': 0, 're-evaluate': 0}, agree: True
per answer: wait 0.00, let me check 0.00, alternatively 0.00, re-evaluate 0.00
per 1,000 tokens: wait 0.00, let me check 0.00, alternatively 0.00, re-evaluate 0.00
answers with at least one "wait": 0/160
"wait" per answer: correct answers 0.00, wrong answers 0.00

== DeepSeek-R1-Distill-Qwen-1.5B
reflection words, total over 160 answers: Method 1 {'wait': 719, 'let me check': 77, 'alternatively': 208, 're-evaluate': 0}
                                              Method 2 {'wait': 719, 'let me check': 77, 'alternatively': 208, 're-evaluate': 0}, agree: True
per answer: wait 4.49, let me check 0.48, alternatively 1.30, re-evaluate 0.00
per 1,000 tokens: wait 2.45, let me check 0.26, alternatively 0.71, re-evaluate 0.00
answers with at least one "wait": 159/160
"wait" per answer: correct answers 3.00, wrong answers 5.41
"wait"base, wait: 0.00 per answer0.00distill, wait: 4.49 per answer4.49"let me check"base, let me check: 0.00 per answer0.00distill, let me check: 0.48 per answer0.48"alternatively"base, alternatively: 0.00 per answer0.00distill, alternatively: 1.30 per answer1.30"re-evaluate"base, re-evaluate: 0.00 per answer0.00distill, re-evaluate: 0.00 per answer0.00Qwen2.5-Math-1.5B (base)R1-Distill-Qwen-1.5Bmean count per answer, same 20 problems x 8 samples for each model
Reflection phrases per answer on the same 160 prompts. The base model with the R1-Zero template wrote none of the four phrases. The distill wrote "wait" 4.49 times per answer and "alternatively" 1.30 times. Neither model wrote "re-evaluate".

The two methods agree on every total. The contrast is total. In 160 answers, the base model never wrote "wait", "let me check" or "alternatively". The distill wrote "wait" 719 times, and 159 of its 160 answers contain at least one. Per 1,000 tokens that is 2.45 "wait"s, so the count is not only a side effect of longer answers.

The distill learned this habit by copying R1's answers, and R1 inherited it from the RL of R1-Zero. So this is the behaviour of Table 3, carried down into a model small enough for a laptop. It is not the training curve: we see the end point, not the climb.

One more line is worth a look. The distill's wrong answers contain more "wait"s than its right ones: 5.41 against 3.00 per answer. A model says "wait" most when it struggles, and the struggling answers are also the long ones that run out of room. More reflection does not mean more right answers here, which matches the finding of Liu et al. in the next section.

The aha moment

Solve it yourself

The model's first path hides a trap, and it helps to see it. Here are the steps, with the reason for each.

1. squarea - √(a+x) = x²needs x ≥ 02. square againa + x = (a - x²)²needs a - x² ≥ 03. a quarticx⁴-2ax²-x+a²-a = 0no x³ term: 4 roots sum to 04. factor(x²+x+1-a)(x²-x-a) = 0four candidates5. check eachin the first equationonly one survives6. answerx = (√(4a-3) - 1)/2the sum is this one root
The six steps: square, square again, get a quartic with no x-cubed term, factor it into two quadratics, test the four candidates in the first equation, and keep the one that works: x = (sqrt(4a - 3) - 1) / 2.

Step 1: square once. A square root is never negative, so the right side xx must be at least 0. Squaring gives

a−a+x=x2⟹a+x=a−x2.a - \sqrt{a + x} = x^2 \quad\Longrightarrow\quad \sqrt{a + x} = a - x^2 .

The left side is again a square root, so we also need a−x2≥0a - x^2 \ge 0.

Step 2: square again.

a+x=(a−x2)2=a2−2ax2+x4⟹x4−2ax2−x+(a2−a)=0.a + x = (a - x^2)^2 = a^2 - 2a x^2 + x^4 \quad\Longrightarrow\quad x^4 - 2a x^2 - x + (a^2 - a) = 0 .

This is the quartic of Table 3. The model got this far.

Step 3: the trap. The quartic has no x3x^3 term. By Vieta's rule, the four roots of x4+c3x3+⋯=0x^4 + c_3 x^3 + \dots = 0 add up to −c3-c_3. Here c3=0c_3 = 0, so the four roots add up to 0. A hurried solver writes "the sum is 0" and stops. That answer is wrong, because squaring adds false roots.

Step 4: factor. Read the quartic as a quadratic in aa, not in xx:

a2−(2x2+1) a+(x4−x)=0.a^2 - (2x^2 + 1)\,a + (x^4 - x) = 0 .

Its discriminant is (2x2+1)2−4(x4−x)=4x2+4x+1=(2x+1)2(2x^2 + 1)^2 - 4(x^4 - x) = 4x^2 + 4x + 1 = (2x + 1)^2, a perfect square. So a=x2+x+1a = x^2 + x + 1 or a=x2−xa = x^2 - x, and the quartic splits into two quadratics in xx:

(x2+x+1−a) (x2−x−a)=0.(x^2 + x + 1 - a)\,(x^2 - x - a) = 0 .

Step 5: check the four candidates. The second factor gives x2=x+ax^2 = x + a, so a−x2=−xa - x^2 = -x. Step 1 needs a−x2≥0a - x^2 \ge 0, so x≤0x \le 0, and also x≥0x \ge 0. Only x=0x = 0 is left, and then a=0a = 0, which breaks a>1a > 1. Both roots of the second factor are false.

The first factor gives x=−1±4a−32x = \frac{-1 \pm \sqrt{4a - 3}}{2}. The minus sign gives a negative xx, which is false. The plus sign gives a positive xx, because 4a−3>14a - 3 > 1 when a>1a > 1.

Step 6: the answer. There is exactly one real solution, so the "sum" is that one root:

x=4a−3−12.x = \frac{\sqrt{4a - 3} - 1}{2} .

Check it with a=3a = 3: x=(9−1)/2=1x = (\sqrt{9} - 1)/2 = 1, and 3−3+1=3−2=1\sqrt{3 - \sqrt{3 + 1}} = \sqrt{3 - 2} = 1. It works.

Method 1: by hand in Python. Build the four candidates from the two quadratics, then keep only those that pass the first equation:

python
def candidates(a):
    """The quartic is a quadratic in a:  a^2 - (2x^2 + 1) a + (x^4 - x) = 0.
    Its discriminant is (2x^2 + 1)^2 - 4(x^4 - x) = 4x^2 + 4x + 1 = (2x + 1)^2, a perfect square, so
    a = x^2 + x + 1  or  a = x^2 - x.  Each one is a quadratic in x with two roots: four candidates in all."""
    out = []
    d1 = 4 * a - 3                                   # x^2 + x + (1 - a) = 0
    if d1 >= 0:
        out += [(-1 + math.sqrt(d1)) / 2, (-1 - math.sqrt(d1)) / 2]
    d2 = 4 * a + 1                                   # x^2 - x - a = 0
    if d2 >= 0:
        out += [(1 + math.sqrt(d2)) / 2, (1 - math.sqrt(d2)) / 2]
    return out


def satisfies_original(a, x, tol=1e-9):
    """Plug x into sqrt(a - sqrt(a + x)) = x. Both square roots must exist and the result must equal x."""
    if a + x < 0:
        return False
    inner = a - math.sqrt(a + x)
    if inner < 0:
        return False
    return abs(math.sqrt(inner) - x) < tol

Method 2: with SymPy and NumPy. SymPy expands and factors the quartic for any aa, and solves the first equation for a=3a = 3. numpy.roots finds the four roots of the quartic numerically. Then SymPy checks the answer for every a>1a > 1 by substitution. We write aa through u=4a−3−1>0u = \sqrt{4a - 3} - 1 > 0, so that every square root becomes a plain expression.

plain text
equation: sqrt(a - sqrt(a + x)) = x, with a > 1
after squaring twice (sympy): Eq(a**2 - 2*a*x**2 - a + x**4 - x, 0)
factored (sympy):             Eq((-a + x**2 - x)*(-a + x**2 + x + 1), 0)
sum of the 4 quartic roots (Vieta, minus the x^3 coefficient): 0  <- the tempting wrong answer
a = 3: Method 1 candidates [1.0, -2.0, 2.3028, -1.3028], kept [1.0]
        numpy.roots [2.3028, 1.0, -1.3028, -2.0], sum +0.0000; formula (sqrt(4a-3)-1)/2 = 1.0000
a = 7: Method 1 candidates [2.0, -3.0, 3.1926, -2.1926], kept [2.0]
        numpy.roots [3.1926, 2.0, -2.1926, -3.0], sum -0.0000; formula (sqrt(4a-3)-1)/2 = 2.0000
a = 13: Method 1 candidates [3.0, -4.0, 4.1401, -3.1401], kept [3.0]
        numpy.roots [4.1401, 3.0, -3.1401, -4.0], sum +0.0000; formula (sqrt(4a-3)-1)/2 = 3.0000
a = 2.5: Method 1 candidates [0.8229, -1.8229, 2.1583, -1.1583], kept [0.8229]
        numpy.roots [2.1583, 0.8229, -1.1583, -1.8229], sum -0.0000; formula (sqrt(4a-3)-1)/2 = 0.8229
sympy solve for a = 3: [1]
sympy: a + X = (u + 2)**2/4  so sqrt(a + X) = u/2 + 1 ; a - sqrt(a + X) = u**2/4  so the left side = u/2  and X = u/2
sympy check: left side - X = 0  for every a > 1
answer: the only real solution is x = (sqrt(4a - 3) - 1)/2, so the sum is (sqrt(4a - 3) - 1)/2

Terminal output of r1_part4_aha_math.py: the quartic, its factors, the candidates and the one kept root

Read the lines for a=3a = 3. The quartic has four real roots: 2.3028, 1, -1.3028 and -2. NumPy says they add up to 0, exactly as Vieta says. Only 1 passes the first equation. The same happens for a=7a = 7 (kept: 2) and a=13a = 13 (kept: 3), as the formula says.

The last SymPy line shows that the left side minus our answer is 0 for every a>1a > 1. So the right answer is 4a−3−12\frac{\sqrt{4a - 3} - 1}{2}, and "0" was the trap.

This trap may be what the model sensed, but the paper does not say so. The excerpt in Table 3 stops at the quartic. It does not show the model using Vieta, writing a sum of 0, or finding the false roots. All we can say is this: squaring leads to a tidy quartic with an easy wrong sum. At that point, the model chose to go back and check its steps.

We do not know how the answer ended, because the table cuts it. Doubt at that point was a good idea.

A real "Wait" from our run

Here is one real "Wait" from the distill on our MATH-500 problems, copied from its output:

plain text
== a real "Wait" (distill, problem 20: 'Evaluate $(1+2i)6-3i$.', sample 2, 1579 tokens, final answer '6 + 9i', correct True)
Another thought: if I were to write this out in terms of real and imaginary parts, the expression is (1*6 - 2i*6) - 3i, which is (6 - 12i) - 3i. Wait, hold on, that's a different approach. Is that correct?

Wait, no, that's not right. Because when you have (1 + 2i)*6, it's 6*(1) + 6*(2i) = 6 + 12i. Then subtracting 3i, so 6 + 12i - 3i. So, that's 6 + (12i - 3i) = 6 + 9i. But if I distribute the multiplication first, maybe I can see another way. Let's think about it: 1*6 is 6, 2i*6 is 12i, so that's 6 + 12i. Then subtract 3i, so 6 + 12i - 3i is 6 + 9i.

The problem is one line: evaluate (1+2i)6−3i(1+2i)6 - 3i. The model already had the right answer, 6+9i6 + 9i, a few paragraphs earlier. Here it tries "another thought" and makes a sign error: it writes (1⋅6−2i⋅6)(1 \cdot 6 - 2i \cdot 6) where a plus belongs. The next sentence doubts the step. The one after rejects it ("that's not right") and redoes the product.

That is reflection in miniature: an error, a doubt, a repair. It also shows the cost. This answer used 1,579 tokens for a sum a person does in one line. The habit that saves the model on hard problems also runs on easy ones.

Was the aha moment new?

In the months after the paper, several groups studied R1-Zero-style training in the open. The most direct test came from Sea AI Lab.

Our small run adds one data point. With the R1-Zero template, temperature 0.6 and 2,048 tokens, the base model Qwen2.5-Math-1.5B wrote "wait" in none of 160 answers. The distill wrote it in 159. Liu et al. report some self-reflection for Qwen2.5-Math-1.5B too, counted over 500 questions and several temperatures, with keywords and a language-model check. So the habit can exist in the base model and still be rare enough to miss in 160 samples. Either way, after training it is everywhere.

Drawback of DeepSeek-R1-Zero

We looked for language mixing in our run and found none: 0 of 160 answers from either model contain a Chinese, Japanese or Korean character. That is expected. The distill learned from R1's answers, and for those the authors "filtered out chain-of-thought with mixed languages" (page 10). The base model had no RL, so it had no chance to drift. R1-Zero's mixing came from RL with a reward that never reads the thinking, and our run cannot reproduce that.

The paper and the code

Paper (arXiv 2501.12948) · Official repository · Our scripts for this part