DeepSeek-R1, explained · Part 7 of 8 · Covers §3, §3.1, Table 4, §4.2, §5

Results, Failures and Limits

How R1 is scored, where it wins and where it loses, the two ideas that failed, and what R1 still cannot do. We write pass@k and majority voting in Python, and test greedy decoding and few-shot prompts on the 1.5B model.

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and others. arXiv 2025 (Nature 2025), 2025. arXiv:2501.12948

A score of 79.8 on a maths test sounds like one clean number. Behind it are choices: which test, which prompt, how many tries, how hot the sampling, and what counts as right. This part reads those choices first. Then it reads the results, the two ideas that did not work, and the four limits the authors list.

Where this part sits in the R1 pipelineV3-Basecold start SFTreasoning RLreject + SFTRL, all tasksR1evaluateyou are hereOne question, scored the way §3 scores itquestionanswer 1: 12rightanswer 2: 7wronganswer 3: 12rightanswer 4: 12rightk samples (4 to 64)T = 0.6, top-p = 0.95pass@1 = mean of p_i = 3/4 = 0.75cons@k: majority answer 12, right = 1average over all answersvote first, then grade once
Where this part sits, and how §3 scores one question. R1 writes several answers with sampling. pass@1 is the share of right answers; cons@k grades only the most common answer.

Benchmarks

GroupBenchmarksWhat a question looks like
KnowledgeMMLU, MMLU-Redux, MMLU-Pro, GPQA DiamondA multiple-choice question from school or university subjects. GPQA Diamond is graduate-level science, written so that a web search does not help.
FactsSimpleQA, C-SimpleQA (Chinese)A short question with one fact as the answer, such as a date or a name.
ReadingDROP, FRAMESAnswer from a long text or several documents: find the numbers, then compute.
Following instructionsIF-Eval"Write 3 paragraphs, no commas." A program checks the format.
Open-ended writingAlpacaEval 2.0, ArenaHardA free request, like "plan a trip". Another model judges the answer.
CodeLiveCodeBench, Codeforces, SWE Verified, Aider-PolyglotContest problems with hidden tests, or a real bug to fix in a real repository.
MathsAIME 2024, MATH-500, CNMO 2024A competition problem with one exact answer.
ChineseCLUEWSC, C-Eval, C-SimpleQAChinese versions of reading, knowledge and facts.

Evaluation prompts

Baselines

Evaluation setup

pass@1=1k∑i=1kpi\text{pass@1} = \frac{1}{k}\sum_{i=1}^{k} p_i

Read it from left to right. kk is the number of sampled answers to one question. pip_i is 1 or 0 for the ii-th answer. The sum counts the right answers, and 1k\frac{1}{k} turns the count into a share. A tiny example: the model answers one question 8 times and gets p=(1,0,1,1,0,1,1,0)p = (1, 0, 1, 1, 0, 1, 1, 0). Five are right, so pass@1 is 5/8=0.6255/8 = 0.625.

Why is this better than one greedy answer? First, the two measure different things. Greedy accuracy scores one fixed rule: always take the most likely token. Sampled pass@1 scores the chance that one answer, sampled at temperature 0.6 and top-p 0.95, is right. The two numbers need not be equal.

Second, the sampled score can be measured precisely. For one question, one sample can only say 0 or 1; eight samples can say 0.625, and 64 samples can say it more precisely still. That is why the paper uses more samples on small test sets such as AIME, where each question moves the score a lot.

pass@k, the general form

The paper reports only pass@1, but it cites pass@k from Chen et al. (2021). With kk tries, how likely is at least one right answer? That chance is pass@k. This is the natural score for code, where you can run the tests and keep any answer that passes.

pass@k=1−(n−ck)(nk)\text{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}

Read it for one question. nn is the number of samples we drew, cc how many of them are right, and kk the number of tries we allow. (nk)\binom{n}{k}, read "n choose k", is the number of ways to pick kk items from nn. The fraction is the share of picks that hold only wrong answers. The score of a benchmark is the mean of this over all questions.

A tiny example you can check by hand: n=5n = 5 samples, c=2c = 2 right, k=2k = 2 tries. Call the right samples R1 and R2 and the wrong ones W1, W2, W3. There are (52)=10\binom{5}{2} = 10 pairs. The pairs with no right sample are W1-W2, W1-W3 and W2-W3: (32)=3\binom{3}{2} = 3 of them. So pass@2 is 1−3/10=0.71 - 3/10 = 0.7. Seven of the ten pairs hold a right answer.

With k=1k = 1 the formula becomes 1−n−cn=cn1 - \frac{n-c}{n} = \frac{c}{n}, the share of right samples. That is exactly the paper's pass@1. So the paper's formula is the k=1k = 1 case of Chen et al.'s.

The binomials get huge fast. (200100)\binom{200}{100} has 59 digits. Chen et al. avoid them with a product. Write out both binomials and cancel the common factors:

(n−ck)(nk)=∏i=n−c+1n(1−ki)\frac{\binom{n-c}{k}}{\binom{n}{k}} = \prod_{i=n-c+1}^{n} \left(1 - \frac{k}{i}\right)

The product has cc factors, one per right sample, and each is a number between 0 and 1. No big numbers appear. For the tiny example, n=5n = 5 and c=2c = 2 give the factors i=4i = 4 and i=5i = 5: (1−2/4)(1−2/5)=0.5×0.6=0.3(1 - 2/4)(1 - 2/5) = 0.5 \times 0.6 = 0.3. One minus 0.3 is 0.7 again.

pass@k in code

Method 1: our own estimator, the product form. One loop over the cc factors.

python
def pass_at_k_product(n, c, k):
    """Method 1, our own. n samples, c correct, budget k.
    P(all k picks wrong) = C(n-c, k) / C(n, k) = prod_{i=n-c+1}^{n} (1 - k/i)  (no big numbers)."""
    if n - c < k:                               # fewer than k wrong answers: every k-subset holds a correct one
        return 1.0
    all_wrong = 1.0
    for i in range(n - c + 1, n + 1):           # c factors, one per correct sample
        all_wrong *= 1 - k / i
    return 1 - all_wrong

The first test handles a corner case. If there are fewer than kk wrong samples, you cannot pick kk wrong ones, so the chance of at least one right answer is 1. Then the loop multiplies the cc factors (1−k/i)(1 - k/i), which is the chance that all kk picks are wrong.

Method 2: the formula as written, with exact integers, and the numpy one-liner that Chen et al. print in their Figure 3. A brute-force check lists every subset.

python
def pass_at_k_comb(n, c, k):
    """Method 2a: the formula as written, with exact integers from math.comb."""
    return 1 - math.comb(n - c, k) / math.comb(n, k)   # math.comb(a, b) = 0 when b > a


def pass_at_k_numpy(n, c, k):
    """Method 2b: the numpy script of Chen et al., Figure 3."""
    if n - c < k:
        return 1.0
    return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))


def pass_at_k_brute(n, c, k):
    """Check: list every way to pick k of the n samples and count the picks that hold a correct one."""
    samples = [1] * c + [0] * (n - c)
    picks = list(itertools.combinations(range(n), k))
    good = sum(any(samples[j] for j in pick) for pick in picks)
    return good / len(picks), good, len(picks)

We ran the three formulas on every nn up to 40, every cc and every kk, and the brute force up to n=10n = 10:

plain text
== 2. the unbiased pass@k estimator (Chen et al. 2021)
worked example: n=5 samples, c=2 correct, k=2
  all 10 pairs listed, 7 hold a correct answer  ->  7/10 = 0.700
  formula 1 - C(3,2)/C(5,2) = 1 - 3/10 = 0.700
  product 1 - (1-2/4)(1-2/5) = 1 - 0.50*0.60 = 0.700
all n <= 40, every c and k: max |product - comb| or |product - numpy| = 2.2e-16 (brute force for n <= 10)
Chen et al. use n=200: C(200,100) has 59 digits; pass@100 with c=3: comb 0.876884, product 0.876884

n = 16 samples per question, c correct: pass@k
   c | pass@1   pass@2   pass@4   pass@8   pass@16 
   0 |    0.000    0.000    0.000    0.000    0.000
   1 |    0.062    0.125    0.250    0.500    1.000
   2 |    0.125    0.242    0.450    0.767    1.000
   4 |    0.250    0.450    0.728    0.962    1.000
   8 |    0.500    0.767    0.962    1.000    1.000
  12 |    0.750    0.950    0.999    1.000    1.000
  16 |    1.000    1.000    1.000    1.000    1.000

== 3. the shortcut 1 - (1 - c/n)^k is biased: simulation
true success rate per sample q = 0.1, n = 16, k = 8, 200,000 simulated questions
  true pass@8 = 1 - (1-q)^8 = 0.5695
  mean of unbiased estimator     = 0.5685
  mean of 1 - (1 - c/n)^k        = 0.4898

== 4. cons@k: majority vote over k sampled answers
  plurality right        pass@1 0.500  cons (tie=fraction) 1.00  cons (tie=first) 1   own == lib: True
  plurality wrong        pass@1 0.250  cons (tie=fraction) 0.00  cons (tie=first) 0   own == lib: True
  right but only 3 of 8  pass@1 0.375  cons (tie=fraction) 1.00  cons (tie=first) 1   own == lib: True
  tie 2-2                pass@1 0.500  cons (tie=fraction) 0.50  cons (tie=first) 1   own == lib: True
  tie 2-2, wrong first   pass@1 0.500  cons (tie=fraction) 0.50  cons (tie=first) 0   own == lib: True
  unfinished answers     pass@1 0.250  cons (tie=fraction) 1.00  cons (tie=first) 1   own == lib: True

The ten pairs of the tiny example give 0.700 by listing, by the formula and by the product. Over all the cases, the methods differ by at most 2.2⋅10−162.2 \cdot 10^{-16}, the rounding error of a float. The table is the useful part. With 16 samples of which only 2 are right, pass@1 is 0.125, but pass@8 is already 0.767.

0.000.250.500.751.00124816c = 1 of 16, 1: 0.06c = 1 of 16, 2: 0.12c = 1 of 16, 3: 0.19c = 1 of 16, 4: 0.25c = 1 of 16, 5: 0.31c = 1 of 16, 6: 0.38c = 1 of 16, 7: 0.44c = 1 of 16, 8: 0.50c = 1 of 16, 9: 0.56c = 1 of 16, 10: 0.62c = 1 of 16, 11: 0.69c = 1 of 16, 12: 0.75c = 1 of 16, 13: 0.81c = 1 of 16, 14: 0.88c = 1 of 16, 15: 0.94c = 1 of 16, 16: 1.00c = 2 of 16, 1: 0.12c = 2 of 16, 2: 0.24c = 2 of 16, 3: 0.35c = 2 of 16, 4: 0.45c = 2 of 16, 5: 0.54c = 2 of 16, 6: 0.62c = 2 of 16, 7: 0.70c = 2 of 16, 8: 0.77c = 2 of 16, 9: 0.82c = 2 of 16, 10: 0.88c = 2 of 16, 11: 0.92c = 2 of 16, 12: 0.95c = 2 of 16, 13: 0.97c = 2 of 16, 14: 0.99c = 2 of 16, 15: 1.00c = 2 of 16, 16: 1.00c = 4 of 16, 1: 0.25c = 4 of 16, 2: 0.45c = 4 of 16, 3: 0.61c = 4 of 16, 4: 0.73c = 4 of 16, 5: 0.82c = 4 of 16, 6: 0.88c = 4 of 16, 7: 0.93c = 4 of 16, 8: 0.96c = 4 of 16, 9: 0.98c = 4 of 16, 10: 0.99c = 4 of 16, 11: 1.00c = 4 of 16, 12: 1.00c = 4 of 16, 13: 1.00c = 4 of 16, 14: 1.00c = 4 of 16, 15: 1.00c = 4 of 16, 16: 1.00c = 8 of 16, 1: 0.50c = 8 of 16, 2: 0.77c = 8 of 16, 3: 0.90c = 8 of 16, 4: 0.96c = 8 of 16, 5: 0.99c = 8 of 16, 6: 1.00c = 8 of 16, 7: 1.00c = 8 of 16, 8: 1.00c = 8 of 16, 9: 1.00c = 8 of 16, 10: 1.00c = 8 of 16, 11: 1.00c = 8 of 16, 12: 1.00c = 8 of 16, 13: 1.00c = 8 of 16, 14: 1.00c = 8 of 16, 15: 1.00c = 8 of 16, 16: 1.00c = 1 of 16: 1.00c = 2 of 16: 1.00c = 4 of 16: 1.00c = 8 of 16: 1.00k (the number of tries allowed)pass@k
pass@k from our estimator, for 16 samples of which c are right. Even one right sample in 16 gives pass@8 = 0.5: a model that rarely finds the answer still finds it often when it may try many times.

The last two blocks of the output test two more claims. The simulation gives each of 200,000 questions a true success rate of 0.1 per sample, draws 16 samples, and estimates pass@8. The true pass@8 is 1−0.98=0.56951 - 0.9^8 = 0.5695. Our estimator averages 0.5685, close to the truth. The shortcut 1−(1−c/n)k1 - (1 - c/n)^k averages 0.4898, far below. This is the bias Chen et al. warn about.

The second test is majority voting. The paper does not say how it breaks a tie, so we wrote two rules. "Fraction" gives half a point when the right answer ties with one wrong answer. "First" lets the answer seen first win.

The third case is the reason cons@64 can beat pass@1 by a lot. The right answer "12" appears in only 3 of 8 samples, so pass@1 is 0.375. But the wrong answers are all different, so "12" still wins the vote and cons@8 is 1.

python
def cons_at_k_own(answers, gold, ties='fraction'):
    """Method 1, our own. answers: k final answers (None = no answer). Count each answer, take the most frequent.
    ties='fraction': a tie between t answers gives 1/t credit if gold is among them; 'first': the earliest wins."""
    counts, first_seen = {}, {}
    for i, a in enumerate(answers):
        if a is None:
            continue                            # an answer that never finished does not vote
        counts[a] = counts.get(a, 0) + 1
        first_seen.setdefault(a, i)
    if not counts:
        return 0.0
    top = max(counts.values())
    winners = [a for a in counts if counts[a] == top]
    if ties == 'first':
        return float(min(winners, key=lambda a: first_seen[a]) == gold)
    return (gold in winners) / len(winners)

Method 2 does the same with collections.Counter(votes).most_common(1) for the "first" rule, and numpy.unique(..., return_counts=True) for the "fraction" rule. The two methods agreed on all six cases. This is how R1-Zero went from 71.0 pass@1 to 86.7 cons@64 on AIME (Part 4). Its wrong answers scatter, and its right answers agree.

Terminal output of r1_part7_passk.py: pass@1 of eight answers, the worked pass@k example, the agreement check, the table for 16 samples, the bias simulation and six majority-vote cases
The output of r1_part7_passk.py, as it ran for this article.

Greedy against sampling, on the 1.5B model

The paper's claim about greedy decoding is easy to test on the small model. We took the first 8 problems of MATH-500 that have no drawing. For each one, the released DeepSeek-R1-Distill-Qwen-1.5B wrote one greedy answer and 4 sampled answers at the paper's settings. The prompt is the one the model card recommends. It is the problem, then "Please reason step by step, and put your final answer within \boxed{}". The run used a Mac (mps, bfloat16) and a limit of 2,048 new tokens per answer, not the paper's 32,768.

The only difference between the two modes is one line:

python
    kw = dict(do_sample=True, temperature=TEMP, top_p=TOP_P) if sample else dict(do_sample=False, temperature=None, top_p=None)

math-verify grades the text after </think>. An answer that never closes its thinking has no final answer and counts as wrong. We measure repetition three ways. The repeated 4-gram rate is the share of 4-token windows that already appeared earlier in the same answer. A loop is a tail of 512 tokens that repeats with a fixed period. The last measure is the longest block of tokens that an answer writes twice.

python
def rep_rate(ids, n=4):
    """Repeated n-gram rate: share of the n-token windows that already appeared earlier in the same answer."""
    grams = [tuple(ids[i:i + n]) for i in range(len(ids) - n + 1)]
    return 1 - len(set(grams)) / len(grams) if grams else 0.0


def loop_period(ids, tail=512, need=0.95):
    """Smallest p <= 256 such that the last `tail` tokens repeat with period p (95% of positions agree)."""
    t = ids[-tail:]
    if len(t) < tail:
        return None
    for p in range(1, 257):
        same = sum(t[i] == t[i - p] for i in range(p, len(t))) / (len(t) - p)
        if same >= need:
            return p
    return None

The output, with the few-shot rows left for §5:

plain text
model deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B, device mps, bfloat16
8 MATH-500 problems (the first 8 without a drawing), max_new_tokens 2048
greedy: 1 answer each; sampling: 4 answers each at T=0.6, top-p=0.95

== greedy
  problem              lvl  gold            tokens  hit-limit  rep4   tail-rep4  loop  correct
  test/precalculus/807   2  \left( 3, \f...   850    0/1      0.13   0.11       0/1   1/1
  test/intermediate_al   5  p - q            2048    1/1      0.43   0.28       0/1   0/1
  test/algebra/2584.js   3  \frac{14}{3}     1444    0/1      0.32   0.35       0/1   1/1
  test/number_theory/5   3  9                1874    0/1      0.23   0.11       0/1   1/1
  test/prealgebra/1622   2  42                891    0/1      0.22   0.16       0/1   1/1
  test/number_theory/5   3  27               2048    1/1      0.34   0.18       0/1   0/1
  test/precalculus/927   4  90^\circ         2048    1/1      0.35   0.15       0/1   0/1
  test/algebra/2036.js   3  3\sqrt{13}       1059    0/1      0.15   0.14       0/1   1/1
  total: 8 answers, correct 5/8 = 0.625, hit the 2048-token limit 3/8, loop at the end 0/8, mean tokens 1533
  mean repeated 4-gram rate 0.272, in the last 512 tokens 0.183; correct among finished answers 5/5

== sample
  problem              lvl  gold            tokens  hit-limit  rep4   tail-rep4  loop  correct
  test/precalculus/807   2  \left( 3, \f...   869    0/4      0.15   0.12       0/4   4/4
  test/intermediate_al   5  p - q            2048    4/4      0.42   0.33       0/4   0/4
  test/algebra/2584.js   3  \frac{14}{3}     1635    1/4      0.33   0.28       0/4   3/4
  test/number_theory/5   3  9                1639    0/4      0.21   0.16       0/4   4/4
  test/prealgebra/1622   2  42                914    0/4      0.25   0.19       0/4   4/4
  test/number_theory/5   3  27               2048    4/4      0.30   0.17       0/4   0/4
  test/precalculus/927   4  90^\circ         2048    2/4      0.26   0.15       0/4   2/4
  test/algebra/2036.js   3  3\sqrt{13}       1841    0/4      0.17   0.14       0/4   4/4
  total: 32 answers, correct 21/32 = 0.656, hit the 2048-token limit 11/32, loop at the end 0/32, mean tokens 1608
  mean repeated 4-gram rate 0.261, in the last 512 tokens 0.192; correct among finished answers 21/21

== sampled answers: pass@1, pass@k (unbiased, n = 4) and cons@4 (fraction credit on ties)
  sample   pass@1 0.656  pass@2 0.729  pass@4 0.750  cons@4 0.750
  fewshot  pass@1 0.469  pass@2 0.583  pass@4 0.625  cons@4 0.625
  greedy   accuracy 0.625 (one answer per problem)

== longest block of tokens written twice in one answer
  greedy   mean   29.4 tokens, max   52, answers with a block of 50+ tokens repeated: 1/8
  sample   mean   23.9 tokens, max   48, answers with a block of 50+ tokens repeated: 0/32
  fewshot  mean   22.7 tokens, max   52, answers with a block of 50+ tokens repeated: 1/32
  greedy, problem test/precalculus/927.json: a block of 52 tokens at token 811 comes again at token 912:
  | - \(12 \times \frac{1}{6} = 2\)
  | - \(12 \times (-1) = -12\)
  | - \(12 \times (-\frac{1}{4}) = -3\)
Our run: R1-Distill-Qwen-1.5B on 8 MATH-500 problemsgreedy, 1 answersampled, zero-shotsampled, few-shotanswers correctgreedy, 1 answer, answers correct: 5 of 85/8sampled, zero-shot, answers correct: 21 of 3221/32sampled, few-shot, answers correct: 15 of 3215/32hit the 2,048-token limitgreedy, 1 answer, hit the 2,048-token limit: 3 of 83/8sampled, zero-shot, hit the 2,048-token limit: 11 of 3211/32sampled, few-shot, hit the 2,048-token limit: 17 of 3217/32repeat a 50+ token blockgreedy, 1 answer, repeat a 50+ token block: 1 of 81/8sampled, zero-shot, repeat a 50+ token block: 0 of 320/32sampled, few-shot, repeat a 50+ token block: 1 of 321/32
Our run on 8 MATH-500 problems with a 2,048-token limit. Greedy and zero-shot sampling score about the same, and every wrong answer was one cut by the limit. The few-shot prompt is tested in §5 below.

The honest result first: we did not see the repetition the paper describes. No greedy answer ended in a loop. The repeated 4-gram rate is 0.272 for greedy and 0.261 for sampling, almost the same. The longest block written twice is 52 tokens, and it is not a loop. The model recomputes a short list of products while it checks its work.

Our test is small. The paper's claim is about outputs of up to 32,768 tokens from a far larger model, and a loop needs room to form. A 2,048-token limit cuts most long answers before that could happen.

What we did see is the cut itself. Greedy hit the limit on 3 of 8 problems, sampling on 11 of 32 answers. Every answer that finished was right: 5 of 5 for greedy, 21 of 21 for sampling. So on these problems the limit, not the reasoning, decides the score. That is why the paper allows 32,768 tokens.

We also saw the variance argument at work. On the problem with gold answer 90∘90^\circ, greedy gives 0, but 2 of 4 samples are right, so pass@1 for that problem is 0.5. On the 143\frac{14}{3} problem, greedy gives 1 and sampling gives 3 of 4. Greedy rounds each problem to 0 or 1; sampling keeps the fraction. Over 8 problems, greedy scores 0.625 and the sampled pass@1 is 0.656.

DeepSeek-R1 evaluation

Table 4 by group: R1, o1-1217 and V3 (score out of 100)DeepSeek-R1OpenAI-o1-1217DeepSeek-V3020406080100EnglishMMLUMMLU, DeepSeek-V3: 88.5MMLU, OpenAI-o1-1217: 91.8MMLU, DeepSeek-R1: 90.8MMLU-ReduxMMLU-Redux, DeepSeek-V3: 89.1MMLU-Redux, DeepSeek-R1: 92.9MMLU-ProMMLU-Pro, DeepSeek-V3: 75.9MMLU-Pro, DeepSeek-R1: 84.0DROPDROP, DeepSeek-V3: 91.6DROP, OpenAI-o1-1217: 90.2DROP, DeepSeek-R1: 92.2IF-EvalIF-Eval, DeepSeek-V3: 86.1IF-Eval, DeepSeek-R1: 83.3GPQA DiamondGPQA Diamond, DeepSeek-V3: 59.1GPQA Diamond, OpenAI-o1-1217: 75.7GPQA Diamond, DeepSeek-R1: 71.5SimpleQASimpleQA, DeepSeek-V3: 24.9SimpleQA, OpenAI-o1-1217: 47.0SimpleQA, DeepSeek-R1: 30.1FRAMESFRAMES, DeepSeek-V3: 73.3FRAMES, DeepSeek-R1: 82.5AlpacaEval 2.0AlpacaEval 2.0, DeepSeek-V3: 70.0AlpacaEval 2.0, DeepSeek-R1: 87.6ArenaHardArenaHard, DeepSeek-V3: 85.5ArenaHard, DeepSeek-R1: 92.3CodeLiveCodeBenchLiveCodeBench, DeepSeek-V3: 36.2LiveCodeBench, OpenAI-o1-1217: 63.4LiveCodeBench, DeepSeek-R1: 65.9Codeforces pctCodeforces pct, DeepSeek-V3: 58.7Codeforces pct, OpenAI-o1-1217: 96.6Codeforces pct, DeepSeek-R1: 96.3SWE VerifiedSWE Verified, DeepSeek-V3: 42.0SWE Verified, OpenAI-o1-1217: 48.9SWE Verified, DeepSeek-R1: 49.2Aider-PolyglotAider-Polyglot, DeepSeek-V3: 49.6Aider-Polyglot, OpenAI-o1-1217: 61.7Aider-Polyglot, DeepSeek-R1: 53.3MathAIME 2024AIME 2024, DeepSeek-V3: 39.2AIME 2024, OpenAI-o1-1217: 79.2AIME 2024, DeepSeek-R1: 79.8MATH-500MATH-500, DeepSeek-V3: 90.2MATH-500, OpenAI-o1-1217: 96.4MATH-500, DeepSeek-R1: 97.3CNMO 2024CNMO 2024, DeepSeek-V3: 43.2CNMO 2024, DeepSeek-R1: 78.8ChineseCLUEWSCCLUEWSC, DeepSeek-V3: 90.9CLUEWSC, DeepSeek-R1: 92.8C-EvalC-Eval, DeepSeek-V3: 86.5C-Eval, DeepSeek-R1: 91.8C-SimpleQAC-SimpleQA, DeepSeek-V3: 68.0C-SimpleQA, DeepSeek-R1: 63.7bold names: R1 is behind V3 or o1-1217
Table 4 of the paper as a dot chart, by group. Each row joins the lowest and highest of the three scores. R1 (blue) is far right of V3 (green) on maths and competition code, and close to o1-1217 (orange) where both are reported. Bold names are the rows where R1 is behind.

The dot chart shows the pattern at a glance. The longest lines are in maths and code. AIME goes from 39.2 for V3 to 79.8 for R1, and CNMO from 43.2 to 78.8. The Codeforces percentile goes from 58.7 to 96.3.

These are the tasks that reasoning RL trained on, with rewards a program can check. The English knowledge rows move less: MMLU from 88.5 to 90.8. The model already knew those facts; reasoning helps it use them a little better.

The Codeforces rows show a second effect: a rating and a percentile measure the same skill on different scales. R1's rating is 2,029 and o1-1217's is 2,061, a gap of 32 points. Their percentiles are 96.3 and 96.6, a gap of only 0.3. V3's rating of 1,134 gives 58.7. Few human players have high ratings, so near the top a large step in rating moves the percentile only a little. Compare ratings when two strong models are close; the percentile hides their gap.

Unsuccessful attempts

Process reward models

A PRM is an attractive idea. With an outcome reward, a long solution that goes wrong in step 9 of 10 gets the same zero as nonsense. A teacher who marks each step can say "the first eight steps were good". That is a richer signal, and it should make learning faster.

The second failed idea comes from game-playing programs. To see why it was tempting and why it failed, we need the background first.

Where it comes from. In October 2015, DeepMind's AlphaGo beat the European Go champion by 5 games to 0, the first win of a program over a human professional. The paper appeared in 2016.

Go was the hard case of board games, because the number of possible games is far too large to search. AlphaGo combined two neural networks with a search method called Monte Carlo Tree Search (MCTS). A policy network suggests promising moves. A value network looks at a board and estimates who will win.

AlphaZero (2017) then learned chess, shogi and Go from self-play alone, with no human games.

How MCTS works, in plain words. Picture a tree. The root is the current board, each branch is a move, and each node is the board after that move. The program cannot grow the whole tree, so it grows it where it looks promising. Each round has four steps, shown below. After many rounds, the move from the root with the most visits is played.

One round of Monte Carlo Tree Search1. selectfollow the best-looking child2. expandadd a new child3. evaluatevalue model scores itv = 0.74. back upupdate every parent+0.7+0.7
One round of Monte Carlo Tree Search. Select walks down the tree along promising children; expand adds one new position; the value model scores it; back-up adds the score to every node on the path, so good branches get more visits next round.

The appeal for language models is clear. Treat the reasoning as a game. A node is a partial solution, a move is the next piece of text, and a win is a right final answer. Search the tree at answer time, and use the best paths found by search as training data. The model then gets better, its search gets better, and the loop repeats, as in AlphaZero.

How large is the tree for a real model? A full vocabulary is the wrong count, because no policy considers every token. A better count asks the model itself. At each step of an answer, how many tokens does top-p sampling at the paper's settings actually allow? That gives the branching factor at each step of the one path the model actually took.

We ran one forward pass over each of the 32 zero-shot sampled answers. At every step we sort the next-token probabilities and count how many tokens it takes to reach 0.95:

python
                    for T, store in ((TEMP, s_a), (1.0, s1_a)):
                        pr = torch.softmax(logits / T, -1)
                        srt = pr.sort(-1, descending=True).values.cumsum(-1)
                        store += ((srt < TOP_P).sum(-1) + 1).clamp(max=pr.shape[-1]).tolist()   # tokens to reach 0.95

Then we multiply the nucleus sizes along one answer, two ways. Method 1 adds their base-10 logarithms. Method 2 multiplies them as exact Python integers and counts the digits:

python
m1 = sum(math.log10(s) for s in one['sizes'])
prod = 1
for s in one['sizes']:
    prod *= s
digits = str(prod)
plain text
== branching factor of a search over tokens (51,452 steps of 32 sampled answers)
  vocabulary: 151,665 tokens in the tokenizer (151,936 rows in the output layer)
  top-p 0.95 nucleus at T=0.6: median 1, mean 1.59, 90th pct 3, 99th pct 7, max 22, size 1 at 71.9% of steps
  top-p 0.95 nucleus at T=1.0: median 1, mean 2.64, 90th pct 6, 99th pct 21, max 78, size 1 at 60.9% of steps
  mean entropy at T=0.6: 0.233 nats

== how many different answers could top-p sampling write? one answer (test/precalculus/927.json), 1997 tokens
  Method 1 (sum of log10 of nucleus sizes):  10^182.1
  Method 2 (exact product, count the digits): 183 digits = 10^182.1
  Go, b = 250, d = 150:   250^150 = 10^359.7  (exact: 360 digits)
  chess, b = 35, d = 80:  35^80 = 10^123.5  (exact: 124 digits)
  whole vocabulary, 1997 tokens: 151,665^1997 = 10^10346
  a tree of depth 150 with our median nucleus at T=0.6: 1^150 = 10^0.0
How many tokens are in the top-p 0.95 nucleus at each step?T = 0.6 (the paper)T = 1.00%20%40%60%80%nucleus of 1 tokens, T = 0.6: 71.9% of stepsnucleus of 1 tokens, T = 1.0: 60.9% of steps1nucleus of 2 tokens, T = 0.6: 15.1% of stepsnucleus of 2 tokens, T = 1.0: 14.6% of steps2nucleus of 3 tokens, T = 0.6: 5.8% of stepsnucleus of 3 tokens, T = 1.0: 7.0% of steps3nucleus of 4 tokens, T = 0.6: 3.2% of stepsnucleus of 4 tokens, T = 1.0: 4.1% of steps4nucleus of 5 tokens, T = 0.6: 1.6% of stepsnucleus of 5 tokens, T = 1.0: 2.8% of steps5nucleus of 6 tokens, T = 0.6: 0.9% of stepsnucleus of 6 tokens, T = 1.0: 2.0% of steps6nucleus of 7 tokens, T = 0.6: 0.6% of stepsnucleus of 7 tokens, T = 1.0: 1.5% of steps7nucleus of 8 tokens, T = 0.6: 0.3% of stepsnucleus of 8 tokens, T = 1.0: 1.1% of steps8nucleus of 9 tokens, T = 0.6: 0.2% of stepsnucleus of 9 tokens, T = 1.0: 1.0% of steps9nucleus of 10 tokens, T = 0.6: 0.1% of stepsnucleus of 10 tokens, T = 1.0: 0.7% of steps10nucleus of 11+ tokens, T = 0.6: 0.3% of stepsnucleus of 11+ tokens, T = 1.0: 4.2% of steps11+tokens in the nucleus (the choices sampling can make at that step)
Nucleus sizes over 51,452 steps of our sampled answers. At the paper's temperature of 0.6, 71.9% of steps allow only one token. The rest allow a few, and the largest nucleus we saw had 22 tokens.

The numbers hold a surprise. At T=0.6T = 0.6 the median nucleus has one token, and 71.9% of the steps allow no choice at all. The model is sure of most tokens: the end of a word, the next symbol of an equation. Choices appear at a few places, like which method to try or whether to write "Wait". So a model-guided tree is narrow, not wide.

It is also deep. Take one finished answer of 1,997 tokens and multiply its nucleus sizes. The product is 10182.110^{182.1}, and both methods agree to the digit count of 183. Read this number with care.

It is the size of a hypothetical tree in which every branch copies the branching schedule of this one answer. Step 1 is as wide as our step 1, step 2 as wide as our step 2, and so on. It is not a count of the real tree. A different early token changes every later distribution. So the other branches have their own nucleus sizes and their own lengths. The script's heading ("how many different answers could top-p sampling write?") is only a rough guide to that question.

As an order of magnitude, the number still makes the point. A tree with this schedule is smaller than Go's 10359.710^{359.7}, but no computer can search 1018210^{182} paths either. The unrestricted space, all 151,665 tokens at all 1,997 steps, has 101034610^{10346} paths. That is the "exponentially larger search space" of the paper.

Size of a full search tree, as a power of ten (log scale)chess, 35^80chess, 35^80: 10^12410^124Go, 250^150Go, 250^150: 10^36010^360one answer's schedule (1997 tok)one answer's schedule (1997 tok): 10^18210^182all 151,665 tokens (1997 tok)all 151,665 tokens (1997 tok): 10^1034610^10,346bar length = number of digits of the count
The size of four search trees, as the number of digits. Chess and Go come from the AlphaGo paper. The two blue bars are for one of our 1,997-token answers: a hypothetical tree that repeats this answer's nucleus size at every step, and the paths through the whole vocabulary.

The count also shows the deeper trouble. In Go, every move is a real decision. In text, the decisions hide among long stretches of forced tokens, and nothing marks where one step ends. That is why the authors made the model write tags at its reasoning steps, and why they capped the children per node. The search still needs a value model that can judge half a solution, and that is the second problem below.

Conclusion, limitations, and future work

We tested the advice on the same 8 problems. The few-shot prompt puts two short solved examples before the problem: the sum of the first 10 integers, and a linear equation. Everything else stays the same: 4 samples per problem at T=0.6T = 0.6, top-p 0.95, 2,048 new tokens.

plain text
== fewshot
  problem              lvl  gold            tokens  hit-limit  rep4   tail-rep4  loop  correct
  test/precalculus/807   2  \left( 3, \f...   884    0/4      0.13   0.11       0/4   4/4
  test/intermediate_al   5  p - q            2048    4/4      0.37   0.34       0/4   0/4
  test/algebra/2584.js   3  \frac{14}{3}     2048    2/4      0.35   0.25       0/4   2/4
  test/number_theory/5   3  9                1376    2/4      0.24   0.19       0/4   2/4
  test/prealgebra/1622   2  42                946    1/4      0.25   0.20       0/4   3/4
  test/number_theory/5   3  27               2048    4/4      0.29   0.16       0/4   0/4
  test/precalculus/927   4  90^\circ         2048    4/4      0.27   0.15       0/4   0/4
  test/algebra/2036.js   3  3\sqrt{13}       1227    0/4      0.17   0.11       0/4   4/4
  total: 32 answers, correct 15/32 = 0.469, hit the 2048-token limit 17/32, loop at the end 0/32, mean tokens 1649
  mean repeated 4-gram rate 0.259, in the last 512 tokens 0.188; correct among finished answers 15/15

== sampled answers: pass@1, pass@k (unbiased, n = 4) and cons@4 (fraction credit on ties)
  sample   pass@1 0.656  pass@2 0.729  pass@4 0.750  cons@4 0.750
  fewshot  pass@1 0.469  pass@2 0.583  pass@4 0.625  cons@4 0.625
  greedy   accuracy 0.625 (one answer per problem)

The few-shot prompt lowered pass@1 from 0.656 to 0.469 and cons@4 from 0.750 to 0.625. The direction matches the paper's "consistently degrades". The size of the drop means little: it is 6 answers out of 32 on 8 problems.

Look at where the drop comes from. Every finished few-shot answer was right too, 15 of 15. The few-shot answers simply hit the limit more often, 17 of 32 against 11 of 32. Our run cannot tell why. It shows only that the examples did not make the answers shorter or better, and on these problems they made them worse.

Terminal output of r1_part7_gen.py: greedy, zero-shot and few-shot runs on 8 MATH-500 problems, pass@k and cons@4, the longest repeated blocks, nucleus sizes and the size of the search tree
The output of r1_part7_gen.py, as it ran for this article (the first 60 lines).

What came after

The paper was later revised and published in Nature, received on 14 February 2025 and published online on 17 September 2025. That version keeps the method and the main tables, and adds detail. Its Table 2 scores every stage of the pipeline.

For example, IF-Eval rose from 46.6 for R1-Zero to 83.3 for R1, and AlpacaEval 2.0 from 24.7 to 87.6. The last stages bought the general skills. Its supplement gives the cost: 147,000 H800 GPU hours in all. At an assumed rental price of 2 US dollars per GPU hour, that is 294,000 US dollars.

The rest of the story is in DeepSeek-R1 and reasoning by reinforcement learning, part 4 of our DeepSeek series. It follows the line from R1 to R1-0528, V3.1 with its thinking and non-thinking modes, V3.2, which trained agents that call tools, and V4. Several limits of §5 became the goals of these later models.

The paper and the code

Paper (arXiv 2501.12948) · official repository · our scripts for this series