DeepSeek-R1, explained · Part 8 of 8 · Covers §2.2 (the whole R1-Zero recipe)

Train a Reasoner from Scratch: R1-Zero's Recipe in 330 Lines

We write the whole DeepSeek-R1-Zero method in a small Python package: the template, the two reward rules and GRPO. We check it against TRL and train a 0.5B model on a laptop.

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and others. arXiv 2025 (Nature 2025), 2025. arXiv:2501.12948

Parts 2 and 3 took R1-Zero's recipe apart, one equation and one rule at a time. Here we put it back together in about 330 lines of Python. The code takes a model, asks it maths questions, and makes it better at them with only two rule checks. Is the final answer right? Is it in the requested shape?

The proof has three steps. First, our loss gives the same number as the loss code of TRL, a widely used library, on the same tensors. Second, our accuracy rule gives the same verdicts as math_verify, a library built to compare maths answers. Our format rule agrees with an independent second check. Third, we run the recipe on a laptop and report what happened, including what did not.

The recipe in one paragraph

The package lives in code/papers/deepseek-r1/minir1. It uses PyTorch and the transformers library to load the model and to sample text. Every step of the method itself is our own code.

template.pyTable 1: the promptquestion → textgrpo.py: sample_groupG answers per questionfrom π_oldrewards.pyaccuracy + formatrules, no networkgrpo.py: Eq. 3, 1, 2advantages, clipped ratio,KL to π_ref → losstrain.pythe loop, the optimiser,one log line per stepeval.py200 held-out questionsbefore and afternext step
The minir1 package as one loop. A question goes through the template, the sampler writes G answers, the rules score them, GRPO turns the scores into a loss, and train.py repeats this. eval.py measures held-out questions before and after.
FileLinesWhat it holdsPaper
template.py22The prompt of Table 1§2.2.3
rewards.py81The accuracy rule and the format rule§2.2.2
grpo.py87Sampling, log-probabilities, Eq. 3, Eq. 1 and Eq. 2§2.2.1
train.py109The data, the loop and the log§2.2
eval.py28Held-out questions, scored with the same rules§2.2.4

The page walks through the files in the order a question meets them. For each piece you see the lines of the paper, the code, and a check.

The template

template.pyTable 1: the promptquestion → textyou are heregrpo.py: sample_groupG answers per questionfrom π_oldrewards.pyaccuracy + formatrules, no networkgrpo.py: Eq. 3, 1, 2advantages, clipped ratio,KL to π_ref → losstrain.pythe loop, the optimiser,one log line per stepeval.py200 held-out questionsbefore and afternext step
You are here: template.py turns a question into the text the model reads.

Here is the whole file:

python
"""Table 1 of the paper: the only instruction DeepSeek-R1-Zero gets. It fixes the shape of the answer
(<think> ... </think> <answer> ... </answer>) and says nothing about how to reason."""

TEMPLATE = (
    "A conversation between User and Assistant. The user asks a question, and the Assistant solves it. "
    "The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. "
    "The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, "
    "respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer>. "
    "User: {prompt}. Assistant:"
)
# Our one addition: the prompt ends with the opening tag, so a 0.5B model starts inside <think> (see the text).
PREFILL = ' <think>'


def make_prompt(question):
    """Put one question into the template. The text is fed to the model as is: no chat template, no examples."""
    return TEMPLATE.format(prompt=question.strip().rstrip('.')) + PREFILL


def full_completion(generated):
    """What the rewards see: the prefilled tag plus the generated text."""
    return '<think>' + generated

The constant is Table 1 word for word. We made one addition, and the code says so: the prompt ends with the opening tag <think>. The model then starts inside the thinking block.

Why the prefill? Without it, our 0.5B models rarely passed the format check. We sampled 4 answers to each of 24 training questions, at temperature 1, with at most 300 new tokens:

plain text
Qwen/Qwen2.5-0.5B: 96 samples, accuracy reward 0.010, format reward 0.021, mean length 198 tokens, 45 s
Qwen/Qwen2.5-0.5B-Instruct: 96 samples, accuracy reward 0.031, format reward 0.000, mean length 215 tokens, 44 s

With the plain template, the base model passed the format check on 2 of 96 answers (0.021) and the instruction-tuned model on none. With the prefill, both start to pass it:

plain text
Qwen/Qwen2.5-0.5B: 96 samples, accuracy reward 0.021, format reward 0.115, mean length 137 tokens, 45 s
Qwen/Qwen2.5-0.5B-Instruct: 96 samples, accuracy reward 0.052, format reward 0.427, mean length 141 tokens, 43 s

The base model now passes on 11 of 96 answers (0.115) and the Instruct model on 41 of 96 (0.427). GRPO's reward signal comes only from differences inside a group. If every answer in a group gets the same reward, the rewards push nothing up or down (we show why in the GRPO section). A start of 2% leaves almost every group with equal rewards. A start of 10% to 40% gives many groups a mix of good and bad answers.

The prefill does not tell the model how to reason, so it keeps the spirit of §2.2.3. The rewards still see the full text: full_completion puts the tag back in front of what the model wrote.

The rewards

template.pyTable 1: the promptquestion → textgrpo.py: sample_groupG answers per questionfrom π_oldrewards.pyaccuracy + formatrules, no networkyou are heregrpo.py: Eq. 3, 1, 2advantages, clipped ratio,KL to π_ref → losstrain.pythe loop, the optimiser,one log line per stepeval.py200 held-out questionsbefore and afternext step
You are here: rewards.py turns each answer into a number.

Here is the whole file:

python
"""The two rule-based rewards of §2.2.2. No neural reward model: a few lines of Python decide each reward."""
import re
from fractions import Fraction

ANSWER_RE = re.compile(r'<answer>(.*?)</answer>', re.DOTALL)
TAG_RE = re.compile(r'</?(think|answer)>')
# One number: '-3', '12.5', '1/2', '-7/4', '1e3'. The lookarounds stop it from starting or ending inside a word.
NUMBER_RE = re.compile(r'(?<![\w.])(-?\d+(?:\.\d+)?(?:[eE][-+]?\d+)?)(?:\s*/\s*(\d+(?:\.\d+)?))?(?!\w|\.\d)')
OPERATOR_RE = re.compile(r'\d\s*(?:[-+*×÷^])\s*\(?-?\d')     # '10 - 2', '3 * 4', '2^3': an expression, not a number


def extract_answer(completion):
    """The text inside the last <answer> ... </answer> pair, or None when there is no such pair."""
    found = ANSWER_RE.findall(completion)
    return found[-1].strip() if found else None


def extract_boxed(text):
    """What is inside the last \\boxed{...}, counting braces so that '\\boxed{\\frac{1}{2}}' works. None if unclosed."""
    start = text.rfind('\\boxed')
    i = text.find('{', start) if start >= 0 else -1
    if i < 0:
        return None
    depth = 0
    for j in range(i, len(text)):
        depth += {'{': 1, '}': -1}.get(text[j], 0)
        if depth == 0:                                   # the brace that closes the box
            return text[i + 1:j]
    return None


def to_number(text):
    """The final value of an answer as an exact Fraction, or None when it is not one number.
    '$1,250.00' -> 1250, 'x = -3' -> -3, '3 + 4 = 7' -> 7, '1/2' -> 1/2, '\\boxed{36/2}' -> 18, '1e3' -> 1000,
    '10 - 2' -> None (an unworked expression), '2^3' -> None, 'none' -> None."""
    if '\\boxed' in text:
        text = extract_boxed(text)                       # the box is the final answer
        if text is None:
            return None
    text = text.split('=')[-1]                           # the final expression: what comes after the last '='
    text = re.sub(r'\\[dt]?frac\{([^{}]*)\}\{([^{}]*)\}', r'\1/\2', text)     # \frac{a}{b} -> a/b
    text = re.sub(r'\\text\{[^{}]*\}|\\[$%,!]|[$%]', ' ', text)               # unit text, dollar and per cent signs
    while re.search(r'\d,\d{3}(?!\d)', text):
        text = re.sub(r'(\d),(\d{3})(?!\d)', r'\1\2', text)                    # 1,250,000 -> 1250000
    if OPERATOR_RE.search(text):
        return None                                      # a sum or a power left unworked: no single number
    found = NUMBER_RE.findall(text)
    if not found:
        return None
    a, b = found[-1]                                     # the last number of the final expression
    value = Fraction(float(a)) if 'e' in a.lower() else Fraction(a)
    return None if b and float(b) == 0 else (value / Fraction(b) if b else value)


def accuracy_reward(completion, gold):
    """1.0 when the number in the <answer> tags equals the known answer, else 0.0 (the 'accuracy reward')."""
    answer = extract_answer(completion)
    if answer is None:
        return 0.0                                   # no answer tags: nothing to check
    value = to_number(answer)
    return float(value is not None and value == Fraction(gold))


def format_reward(completion):
    """1.0 when the completion is exactly <think> ... </think> <answer> ... </answer>: each tag once, in this order,
    both blocks non-empty, nothing before, between or after them (the 'format reward'). Part 3's tag parser."""
    tags = list(TAG_RE.finditer(completion))
    if [t.group(0) for t in tags] != ['<think>', '</think>', '<answer>', '</answer>']:
        return 0.0                                   # a tag is missing, repeated or out of order
    t1, t2, a1, a2 = tags
    if completion[:t1.start()].strip() or completion[t2.end():a1.start()].strip() or completion[a2.end():].strip():
        return 0.0                                   # text before, between or after the two blocks
    if not completion[t1.end():t2.start()].strip() or not completion[a1.end():a2.start()].strip():
        return 0.0                                   # an empty block
    return 1.0


def total_reward(completion, gold):
    """The reward GRPO sees. Version 1 of the paper names the two rules but not how they combine; the later version
    adds them with the same weight (its Eq. 4). So one answer earns 0, 1 or 2."""
    return accuracy_reward(completion, gold) + format_reward(completion)

extract_answer takes the last answer pair, so an answer that changes its mind is graded on its final word.

to_number reads the final value of the answer. If there is a \boxed{...}, it takes what is inside the last box, counting braces. It keeps only what comes after the last =, so "3 + 4 = 7" gives 7. It removes dollar signs and the commas of thousands, so a dollar price of 1,250.00 written with its sign gives 1250. A fraction counts as a fraction and 1e3 as a thousand. A sum or a power that was never worked out, such as "10 - 2", gives no number at all, and so no reward.

format_reward is the tag parser of Part 3. It finds the four tags and asks for exactly <think>, </think>, <answer>, </answer>, once each and in this order. Both blocks must hold some text, and nothing may stand before, between or after them.

Here are the fixed rules on those hand-made cases:

plain text
accuracy, answer             '18' gold   18: value    18  reward 1
accuracy, answer      '$1,250.00' gold 1250: value  1250  reward 1
accuracy, answer      '3 + 4 = 7' gold    7: value     7  reward 1
accuracy, answer            '1/2' gold    2: value   1/2  reward 0
accuracy, answer            '1e3' gold    3: value  1000  reward 0
accuracy, answer  '\\boxed{36/2}' gold   18: value    18  reward 1
accuracy, answer         '10 - 2' gold    8: value  None  reward 0
accuracy, answer            '2^3' gold    8: value  None  reward 0
format 1: '<think> 2 + 3 = 5 </think> <answer> 5 </answer>'
format 0: '<think> </think><answer> </answer>'
format 0: '<think> a </think><think> b </think><answer> 5 </answer>'
format 0: '<think> a </think><answer> 5 </answer> more'
format 0: '<think> a <answer> 5 </answer>'

Version 1 of the paper names the two rewards but does not say how they combine. The later, longer version adds them with the same weight (its Eq. 4). We do the same, so one answer earns 0, 1 or 2.

Method 2, with libraries. The math_verify library parses two maths expressions and checks if they are equal. It knows fractions, LaTeX and units.

We ran it and our accuracy rule on every answer of the held-out evaluation. Both read the text inside the last answer tags and compare it with the GSM8K answer. For the format rule we wrote a second, independent check. It is one regular expression: neither block may contain a tag, and each block must hold a non-space character.

plain text
before: our accuracy rule and math_verify agree on 200 of 200 answers (both read the text inside the last answer tags)
before: our format parser and an independent regular expression agree on 200 of 200 answers
after: our accuracy rule and math_verify agree on 200 of 200 answers (both read the text inside the last answer tags)
after: our format parser and an independent regular expression agree on 200 of 200 answers

Both checks agree on all 400 answers of the Instruct run (200 before training, 200 after). They also agree on the 400 answers of the base run.

Read these numbers for what they are. The math_verify check covers only the accuracy rule, and both sides read the same text: what extract_answer found inside the last answer tags. It does not test the extraction itself, and it says nothing about format. The format check is the separate second line. GSM8K answers are also whole numbers, which keeps the comparison easy: the fraction and power cases above barely occur here. For MATH or AIME problems, use math_verify, or a parser like it, as the accuracy rule.

Group Relative Policy Optimization

Now the heart of the method. GRPO comes from an earlier DeepSeek paper on maths, and Part 2 explains each equation term by term. Here we write each equation as one short function.

Step 1: sample a group

template.pyTable 1: the promptquestion → textgrpo.py: sample_groupG answers per questionfrom π_oldyou are hererewards.pyaccuracy + formatrules, no networkgrpo.py: Eq. 3, 1, 2advantages, clipped ratio,KL to π_ref → losstrain.pythe loop, the optimiser,one log line per stepeval.py200 held-out questionsbefore and afternext step
You are here: sample_group writes G answers to each question with the current model.
python
@torch.no_grad()
def sample_group(model, tok, prompts, G, max_new_tokens, temperature=1.0):
    """Sample G completions for each prompt from the current policy (the 'old policy' of Eq. 1)."""
    texts = [p for p in prompts for _ in range(G)]               # B = len(prompts) * G, groups side by side
    enc = tok(texts, return_tensors='pt', padding=True, padding_side='left').to(model.device)
    out = model.generate(**enc, do_sample=True, temperature=temperature, top_p=1.0, top_k=0,
                         repetition_penalty=1.0, max_new_tokens=max_new_tokens,
                         stop_strings=['</answer>'], tokenizer=tok, pad_token_id=tok.pad_token_id)
    prompt_ids, completion_ids = enc.input_ids, out[:, enc.input_ids.size(1):]          # (B, P), (B, T)
    lengths = completion_lengths(completion_ids.tolist(), tok, end_token_ids(model, tok))
    T = completion_ids.size(1)
    completion_mask = (torch.arange(T)[None, :] < torch.tensor(lengths)[:, None]).long().to(model.device)  # (B, T)
    texts = [full_completion(tok.decode(ids[:n], skip_special_tokens=True))            # only the real tokens
             for ids, n in zip(completion_ids.tolist(), lengths)]
    return dict(prompt_ids=prompt_ids, prompt_mask=enc.attention_mask,
                completion_ids=completion_ids, completion_mask=completion_mask, texts=texts)

The function repeats each prompt G times, so the G answers to one question sit side by side. Padding on the left lines up the ends of the prompts, which is where generation starts. We sample at temperature 1 with no top-p or top-k filter, so the answers come from the model's real distribution. The later version of the paper also uses temperature 1.

Generation stops at the string </answer>, or at an end-of-text token, or at the length limit. When one answer stops early, generate fills the rest of its row with padding. For Qwen the padding token is the same token as end-of-text, so the ids alone cannot tell a sampled end from a filler.

completion_lengths decides from how the row stopped. An answer that stopped on </answer> ends with the token that completes the string. An answer that stopped on a sampled end token keeps that token, because the model chose it. An answer that hit the limit keeps everything.

plain text
mask, stopped on </answer>, then 2 padding tokens: tokens [' ', '5', ' </', 'answer', '>', '<|endoftext|>', '<|endoftext|>'] -> [1, 1, 1, 1, 1, 0, 0]
mask, stopped on a sampled <|im_end|>, then 1 padding token: tokens [' ', '5', '<|im_end|>', '<|endoftext|>'] -> [1, 1, 1, 0]
mask, hit the length limit: tokens [' ', '5', ' is', ' the', ' answer'] -> [1, 1, 1, 1, 1]

The mask is 1 for the tokens the model really wrote and 0 after. Every later step uses it, so padding never counts. The evaluation uses the same function for its lengths.

Step 2: the log-probability of each token

To change the probability of an answer, we need the probability of each of its tokens. One forward pass over prompt plus answer gives them all.

python
def per_token_logps(model, batch, temperature=1.0):
    """log pi(o_t | q, o_<t) for every completion token: the numbers that go into the ratio of Eq. 1."""
    ids = torch.cat([batch['prompt_ids'], batch['completion_ids']], 1)                  # (B, P + T)
    mask = torch.cat([batch['prompt_mask'], batch['completion_mask']], 1)               # (B, P + T)
    T = batch['completion_ids'].size(1)
    # The logits at position t predict token t + 1, so keep the T + 1 last positions and drop the very last one.
    logits = model(input_ids=ids, attention_mask=mask, logits_to_keep=T + 1).logits[:, :-1].float()   # (B, T, V)
    logits = logits / temperature
    chosen = logits.gather(-1, batch['completion_ids'].unsqueeze(-1)).squeeze(-1)      # (B, T): score of the token
    return chosen - torch.logsumexp(logits, -1)                                         # (B, T): log softmax

The logits at position tt predict the token at position t+1t+1. So for TT answer tokens we keep the last T+1T+1 positions and drop the very last one. gather picks the logit of the token that was actually written. Subtracting logsumexp over the vocabulary turns that logit into a log-probability, without building the full table of 151,936 probabilities per position a second time.

The paper writes πθ(oi∣q)\pi_\theta(o_i \mid q) for a whole answer. The probability of an answer is the product of the probabilities of its tokens, so its logarithm is a sum over tokens. The code keeps one number per token, because Eq. 1 works on each token, as the next box shows.

Step 3: the group advantage (Eq. 3)

template.pyTable 1: the promptquestion → textgrpo.py: sample_groupG answers per questionfrom π_oldrewards.pyaccuracy + formatrules, no networkgrpo.py: Eq. 3, 1, 2advantages, clipped ratio,KL to π_ref → lossyou are heretrain.pythe loop, the optimiser,one log line per stepeval.py200 held-out questionsbefore and afternext step
You are here: the update half of grpo.py, from rewards to a loss.
Ai=ri−mean(r1,…,rG)std(r1,…,rG)A_i = \frac{r_i - \text{mean}(r_1, \ldots, r_G)}{\text{std}(r_1, \ldots, r_G)}

Read it left to right. The advantage of answer ii is its reward rir_i, minus the mean reward of its group. That difference is divided by the standard deviation of the group's rewards. The paper does not say which standard deviation. We use the sample one (divide by G−1G-1), as TRL does. We also add 10−410^{-4} below the line, so a group of equal rewards does not divide by zero.

python
def group_advantages(rewards, G):
    """Eq. 3: A_i = (r_i - mean(r_1..r_G)) / std(r_1..r_G), each answer against its own group."""
    r = rewards.view(-1, G)                                                             # (questions, G)
    mean, std = r.mean(1, keepdim=True), r.std(1, keepdim=True)                         # std with n - 1, as TRL
    return ((r - mean) / (std + 1e-4)).view(-1)                                         # (B,): 0 when all equal

A worked example with G=4G = 4. The rewards are 2 (right and well formed), 0, 1 (well formed, wrong) and 0. The mean is 0.75.

The deviations are 1.25, -0.75, 0.25 and -0.75. Their squares add up to 2.75, and 2.75/3=0.957\sqrt{2.75/3} = 0.957. So the advantages are 1.25/0.957=1.311.25/0.957 = 1.31, then -0.78, 0.26 and -0.78. Our function and Python's statistics module agree:

plain text
Eq. 3, rewards of one group: [2.0, 0.0, 1.0, 0.0]
  mean 0.75  std (n - 1) 0.9574
  advantages (ours)      : [1.3054, -0.7833, 0.2611, -0.7833]
  advantages (statistics): [1.3054, -0.7833, 0.2611, -0.7833]
  all four the same reward: [0.0, 0.0, 0.0, 0.0]
  tokens clipped, ours 0.0202, TRL clip_ratio/region_mean 0.0202
  mean KL per token, ours 0.273503, TRL kl 0.273503
  tokens clipped, ours 0.1254, TRL clip_ratio/region_mean 0.1254
  mean KL per token, ours 0.273503, TRL kl 0.273503
  tokens clipped, ours 0.0202, TRL clip_ratio/region_mean 0.0202
  mean KL per token, ours 0.273503, TRL kl 9.148457

The last line is the case of equal rewards: four zero advantages. Such a group gives no reward signal. It is not completely silent, though. The KL term of the next steps still pulls each token toward the reference model. With A=0A = 0, πθ=0.6\pi_\theta = 0.6, πref=0.5\pi_{ref} = 0.5 and β=0.04\beta = 0.04, our loss still has a gradient:

plain text
A = 0, pi_theta 0.6, pi_ref 0.5, beta 0.04: gradient of the loss on log pi_theta = 0.006667

That is β(1−πref/πθ)=0.04×(1−0.5/0.6)\beta(1 - \pi_{ref}/\pi_\theta) = 0.04 \times (1 - 0.5/0.6): a small pull back toward the reference.

Step 4: the clipped ratio (Eq. 1)

L=−1G∑i=1G1∣oi∣∑t=1∣oi∣[min⁡(ρi,tAi, clip(ρi,t,1−ε,1+ε)Ai)−β Di,t],ρi,t=πθ(oi,t∣q,oi,<t)πθold(oi,t∣q,oi,<t)L = -\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\left[\min\left(\rho_{i,t}A_i,\ \text{clip}(\rho_{i,t}, 1-\varepsilon, 1+\varepsilon)A_i\right) - \beta\, D_{i,t}\right], \qquad \rho_{i,t} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t} \mid q, o_{i,<t})}

This is the loss our code minimises. ρi,t\rho_{i,t} is the ratio for token tt of answer ii. Di,tD_{i,t} is the KL term of the next step, for the same token. The minus sign in front turns "maximise the objective" into "minimise the loss", which is what an optimiser does.

Four tokens show what the min and the clip do, with ε=0.2\varepsilon = 0.2:

plain text
Eq. 1 term, A = +1, ratio 1.5: min(+1.50, +1.20) = +1.20
Eq. 1 term, A = +1, ratio 0.7: min(+0.70, +0.80) = +0.70
Eq. 1 term, A = -1, ratio 0.5: min(-0.50, -0.80) = -0.80
Eq. 1 term, A = -1, ratio 1.3: min(-1.30, -1.20) = -1.30

With a positive advantage, a ratio of 1.5 is cut to 1.2: the token has already gained enough, so this batch stops pushing it. A ratio of 0.7 is kept: the token lost ground, and it may climb back. With a negative advantage the mirror holds. A ratio of 0.5 counts as 0.8, flat, so the push down stops. A ratio of 1.3 keeps its full value: the bad token became more likely, and the loss keeps pushing it down.

So a token counts as clipped only in two cases: ratio below 1−ε1 - \varepsilon with a negative advantage, or above 1+ε1 + \varepsilon with a positive one. Only then does the min pick the flat branch. Our loss counts clipped tokens this way, the same way as TRL's clip_ratio/region_mean:

plain text
  clipped? A = +1, ratio 1.5: yes
  clipped? A = +1, ratio 0.7: no
  clipped? A = -1, ratio 0.5: yes
  clipped? A = -1, ratio 1.3: no

The ratios 0.7 and 1.3 lie outside the interval, but they are not clipped. Our first version counted every ratio outside the interval, so it reported these two as clipped too. That made the clipped share in our first logs too high.

The ratio is not 1 only when the model changed since sampling. With one optimiser step per batch of answers, it is always exactly 1 and the clip never acts. The later version of the paper splits each batch into 16 mini-batches. We split ours into 2, so the second half of each step meets a real ratio.

Step 5: the KL leash (Eq. 2)

With x=πref/πθx = \pi_{ref}/\pi_\theta for one token:

plain text
Eq. 2, pi_ref 0.5, pi_theta 0.5: 1.0000 - ln 1.0000 - 1 = 0.00000
Eq. 2, pi_ref 0.5, pi_theta 0.6: 0.8333 - ln 0.8333 - 1 = 0.01565
Eq. 2, pi_ref 0.5, pi_theta 0.4: 1.2500 - ln 1.2500 - 1 = 0.02686
Eq. 2, pi_ref 0.5, pi_theta 0.9: 0.5556 - ln 0.5556 - 1 = 0.14334

When the two models agree, the cost is 0. Moving a probability from 0.5 to 0.6 costs 0.016. Moving it from 0.5 to 0.9 costs 0.143, nine times more for four times the move. Small drifts are cheap and big ones are expensive.

The whole loss, in one function

python
def grpo_loss(logps, old_logps, ref_logps, advantages, mask, eps=0.2, beta=0.04):
    """Eq. 1 with Eq. 2, per token, as a loss to minimise (the paper maximises J, so the sign flips)."""
    ratio = torch.exp(logps - old_logps)                         # pi_theta / pi_theta_old, (B, T)
    clipped = torch.clamp(ratio, 1 - eps, 1 + eps)               # clip(ratio, 1 - eps, 1 + eps)
    A = advantages.unsqueeze(1)                                  # (B, 1): one advantage for every token of answer i
    surrogate = torch.min(ratio * A, clipped * A)                # the pessimistic one of the two
    log_r = ref_logps - logps                                    # log(pi_ref / pi_theta)
    kl = torch.exp(log_r) - log_r - 1                            # Eq. 2, always >= 0
    per_token = -(surrogate - beta * kl)                         # (B, T)
    m = mask.float()
    per_answer = (per_token * m).sum(1) / m.sum(1).clamp(min=1)  # 1/|o_i| sum over the tokens of answer i
    loss = per_answer.mean()                                     # 1/G sum over answers (and over questions)
    with torch.no_grad():   # sums and a token count, so the loop can take one token-weighted mean over a whole step
        # A token is clipped when the min of Eq. 1 picks the flat branch: ratio < 1 - eps with A < 0, or
        # ratio > 1 + eps with A > 0 (TRL's clip_ratio/region_mean counts the same tokens).
        is_clipped = ((ratio < 1 - eps) & (A < 0)) | ((ratio > 1 + eps) & (A > 0))
        stats = dict(kl_sum=(kl * m).sum().item(), clipped_sum=(is_clipped.float() * m).sum().item(),
                     tokens=m.sum().item())
    return loss, stats

Each line is one piece of the equations. ratio is ρi,t\rho_{i,t}. torch.min(ratio * A, clipped * A) is the min of Eq. 1. kl is Eq. 2 per token. Then the mask removes padding, the inner mean is the 1/∣oi∣1/|o_i| sum, and the outer mean is the 1/G1/G sum.

For the log, the function returns sums, not means: the total KL, the number of clipped tokens and the number of real tokens. The loop adds them over all slices of a step and divides once. Slices hold different numbers of tokens, so an average of slice means would weight them wrongly, and our first version did exactly that.

Method 2, with TRL. TRL's GRPOTrainer has a method, _compute_loss, that takes the same tensors: prompt and answer tokens, masks, advantages, old and reference log-probabilities. We built a trainer around the same model and called that method directly, so this is TRL's own code.

The batch is 8 real answers of Qwen2.5-0.5B-Instruct to 2 GSM8K questions. Real rewards of 8 short answers are often all equal, and then every advantage is 0. Fixed rewards turn every term on. In training, the three policies differ, so we took two optimiser steps first. The reference is the start, the old policy comes after one step, and the current one after two.

plain text
A = 0, pi_theta 0.6, pi_ref 0.5, beta 0.04: gradient of the loss on log pi_theta = 0.006667
== (b) our loss against TRL on one fixed batch
batch: 2 questions x 4 completions, completion tensor (8, 96), 694 real completion tokens
fixed rewards used for the check: [2.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 2.0]
advantages: [1.3054, -0.7833, 0.2611, -0.7833, 0.0, 0.0, -1.2246, 1.2246]
epsilon 0.2, beta 0.04, TRL use_bias_correction_kl=False:
  tokens clipped, ours 0.0202, TRL clip_ratio/region_mean 0.0202
  mean KL per token, ours 0.273503, TRL kl 0.273503
  loss, ours: +64.69831085
  loss, TRL 1.18.2 GRPOTrainer._compute_loss: +64.69834900
  |difference|: loss 3.81e-05, per-token log-probs 1.30e-06, gradient of lm_head 1.29e-02
  relative difference of the two losses: 5.9e-07
  our loss on the log-probs TRL computed: +64.69834900, |difference| from TRL 0.00e+00
epsilon 0.02, beta 0.04, TRL use_bias_correction_kl=False:
  tokens clipped, ours 0.1254, TRL clip_ratio/region_mean 0.1254
  mean KL per token, ours 0.273503, TRL kl 0.273503
  loss, ours: +64.70719147
  loss, TRL 1.18.2 GRPOTrainer._compute_loss: +64.70722961
  |difference|: loss 3.81e-05, per-token log-probs 1.30e-06, gradient of lm_head 8.20e-03
  relative difference of the two losses: 5.9e-07
  our loss on the log-probs TRL computed: +64.70722961, |difference| from TRL 0.00e+00
epsilon 0.2, beta 0.04, TRL use_bias_correction_kl=True:
  tokens clipped, ours 0.0202, TRL clip_ratio/region_mean 0.0202
  mean KL per token, ours 0.273503, TRL kl 9.148457
  loss, ours: +64.69831085
  loss, TRL 1.18.2 GRPOTrainer._compute_loss: +65.01915741
  |difference|: loss 3.21e-01, per-token log-probs 1.30e-06, gradient of lm_head 1.50e+02
  relative difference of the two losses: 4.9e-03
  our loss on the log-probs TRL computed: +64.69834900, |difference| from TRL 3.21e-01

Terminal output of r1_part8_check.py: the equations on small numbers, then our loss against TRL's loss

Read the first two settings, which use the KL of Eq. 2. Our share of clipped tokens equals TRL's clip_ratio/region_mean: 0.0202 at ε=0.2\varepsilon = 0.2 and 0.1254 at ε=0.02\varepsilon = 0.02. Our mean KL per token equals TRL's kl: 0.273503.

The two losses differ by 3.81×10−53.81 \times 10^{-5} on a loss of about 64.7, a relative difference of 5.9×10−75.9 \times 10^{-7}. Where does that come from? Not from the formula.

TRL computes the log-probabilities itself, through its own path to the output layer, and they differ from ours by at most 1.3×10−61.3 \times 10^{-6}: float32 rounding. The ratio is an exponential of a log-probability difference, and a few ratios here are huge, so a tiny rounding difference becomes a visible one. The last line of each setting settles it. Fed with the log-probabilities TRL computed, our grpo_loss gives exactly TRL's loss: the difference is 0.00e+00.

The loss of about 64.7 looks large, and it is real. A few tokens were sampled with a very low probability, and two optimiser steps made them much more likely: their ratio is huge. Where the advantage is negative, the min of Eq. 1 keeps the huge value, as in the last case of the four-token example. Those few tokens dominate the mean.

The third line is a surprise worth knowing. TRL 1.18 has a setting, use_bias_correction_kl, that is on by default. It multiplies each token's KL term by the ratio, a change that comes from the later DeepSeek-V3.2 paper. With it on, TRL's loss is 65.019 and ours is 64.698, and TRL's logged KL is 9.148457 instead of 0.273503. That is not Eq. 2 of the R1 paper, so for R1 we turn it off. A library default can quietly change the method you think you are running.

The loop

template.pyTable 1: the promptquestion → textgrpo.py: sample_groupG answers per questionfrom π_oldrewards.pyaccuracy + formatrules, no networkgrpo.py: Eq. 3, 1, 2advantages, clipped ratio,KL to π_ref → losstrain.pythe loop, the optimiser,one log line per stepyou are hereeval.py200 held-out questionsbefore and afternext step
You are here: train.py repeats sample, score, compare and update, and writes one log line per step.

The settings of our run, as the script printed them:

plain text
config: {"model": "Qwen/Qwen2.5-0.5B-Instruct", "steps": 400, "minutes": 80.0, "questions": 4, "G": 8, "max_new_tokens": 300, "temperature": 1.0, "lr": 3e-06, "eps": 0.2, "beta": 0.04, "minibatches": 2, "micro": 8, "seed": 0}

Here is the loop. It is the five steps above, in order:

python
def train(cfg, policy, tok, data, log, jsonl_path, samples_path):
    random.seed(cfg['seed']); torch.manual_seed(cfg['seed'])
    ref = copy.deepcopy(policy).eval().requires_grad_(False)           # pi_ref: frozen copy of the start
    opt = torch.optim.AdamW(policy.parameters(), lr=cfg['lr'], betas=(0.9, 0.99), weight_decay=0.0)
    G, start, samples = cfg['G'], time.time(), []
    open(jsonl_path, 'w').close()
    log(f'memory at the start: {memory_gb():.1f} GB')
    for step in range(1, cfg['steps'] + 1):
        qs = random.sample(data, cfg['questions'])
        # 1. Sample a group of G answers per question from the current policy (pi_old).
        policy.eval()
        batch = sample_group(policy, tok, [make_prompt(q['question']) for q in qs], G,
                             cfg['max_new_tokens'], cfg['temperature'])
        # 2. Score every answer with the two rules, then compare it with its own group (Eq. 3).
        gold = [q['gold'] for q in qs for _ in range(G)]
        acc = torch.tensor([accuracy_reward(t, g) for t, g in zip(batch['texts'], gold)])
        fmt = torch.tensor([format_reward(t) for t in batch['texts']])
        rewards = acc + fmt                                              # (B,)
        adv = group_advantages(rewards, G).to(policy.device)           # (B,)
        # 3. Log-probs of pi_old and pi_ref, once, before any update of this step.
        B = rewards.numel()
        with torch.no_grad():
            old = torch.cat([per_token_logps(policy, rows(batch, slice(i, i + cfg['micro'])), cfg['temperature'])
                             for i in range(0, B, cfg['micro'])])
            refl = torch.cat([per_token_logps(ref, rows(batch, slice(i, i + cfg['micro'])), cfg['temperature'])
                              for i in range(0, B, cfg['micro'])])
        # 4. Split the answers into mini-batches; one optimiser step each. From the second one on, pi_theta != pi_old.
        policy.train()
        order, stats = torch.randperm(B), []
        for mb in order.chunk(cfg['minibatches']):
            opt.zero_grad()
            for idx in mb.split(cfg['micro']):
                sub = rows(batch, idx.to(policy.device))
                logps = per_token_logps(policy, sub, cfg['temperature'])
                loss, st = grpo_loss(logps, old[idx], refl[idx], adv[idx], sub['completion_mask'],
                                     cfg['eps'], cfg['beta'])
                (loss * len(idx) / len(mb)).backward()                  # mean over the mini-batch
                stats.append(st)                                        # plain floats, no tensors kept
                del sub, logps, loss
            torch.nn.utils.clip_grad_norm_(policy.parameters(), 1.0)
            opt.step()
        # 5. Log the step.
        # KL and clipped share: one mean over every real token of the step (each slice has a different token count).
        lengths = batch['completion_mask'].sum(1).float()
        tokens = sum(s['tokens'] for s in stats)
        row = dict(step=step, reward=rewards.mean().item(), acc=acc.mean().item(), fmt=fmt.mean().item(),
                   length=lengths.mean().item(), kl=sum(s['kl_sum'] for s in stats) / tokens,
                   clipped=sum(s['clipped_sum'] for s in stats) / tokens,
                   zero_std_groups=(rewards.view(-1, G).std(1) == 0).float().mean().item(),
                   seconds=round(time.time() - start, 1))
        texts = batch['texts']
        del batch, old, refl, adv
        free_memory(policy.device)
        row['rss_gb'] = round(memory_gb(), 2)
        with open(jsonl_path, 'a') as f:
            f.write(json.dumps(row) + '\n')
        log(f"step {step:4d}  reward {row['reward']:.3f}  acc {row['acc']:.3f}  format {row['fmt']:.3f}  "
            f"length {row['length']:6.1f}  kl {row['kl']:.5f}  clipped {row['clipped']:.4f}  "
            f"memory {row['rss_gb']:.1f} GB  {row['seconds']:.0f} s")
        if step in (1, 2, 3) or step % 25 == 0:
            samples.append(dict(step=step, question=qs[0]['question'], gold=qs[0]['gold'],
                                answers=[dict(text=t, acc=a.item(), fmt=f.item())
                                         for t, a, f in zip(texts[:G], acc[:G], fmt[:G])]))
            with open(samples_path, 'w') as f:
                json.dump(samples, f, indent=1)
        if time.time() - start > cfg['minutes'] * 60:
            log(f'time budget of {cfg["minutes"]} minutes reached after step {step}')
            break
    return policy

Before the first step, copy.deepcopy makes the reference model, and requires_grad_(False) freezes it. Each step then does the following:

  1. Pick 4 random training questions and sample 8 answers to each with the current model (one rollout).
  2. Score the 32 answers with the two rules, and turn the scores into advantages, group by group.
  3. Compute the log-probabilities of the old policy and of the reference, once, without gradients. They do not change during the step.
  4. Shuffle the 32 answers into 2 mini-batches of 16. For each, compute the loss in slices of 8 to save memory and add up the gradients. Then clip their total size to 1 and take one AdamW step.
  5. Write one line of JSON with the mean rewards, the mean length, the KL, the share of clipped tokens and the memory the process uses. The KL and the clipped share are averaged over every real token of the step. Then free the memory the step left behind.

Every 25 steps it also saves the 8 answers to the first question, so we can read what the model writes.

The evaluation

template.pyTable 1: the promptquestion → textgrpo.py: sample_groupG answers per questionfrom π_oldrewards.pyaccuracy + formatrules, no networkgrpo.py: Eq. 3, 1, 2advantages, clipped ratio,KL to π_ref → losstrain.pythe loop, the optimiser,one log line per stepeval.py200 held-out questionsbefore and afteryou are herenext step
You are here: eval.py answers 200 held-out questions and scores them with the same rules.
python
"""Held-out check: greedy answers to GSM8K test questions, scored with the same two rules as in training."""
import torch
from .template import make_prompt, full_completion
from .rewards import accuracy_reward, format_reward
from .grpo import end_token_ids, completion_lengths


@torch.no_grad()
def evaluate(model, tok, problems, max_new_tokens=300, batch=32):
    model.eval()
    end_ids = end_token_ids(model, tok)
    out = []
    for i in range(0, len(problems), batch):
        chunk = problems[i:i + batch]
        enc = tok([make_prompt(p['question']) for p in chunk], return_tensors='pt', padding=True,
                  padding_side='left').to(model.device)
        # Greedy, with no repetition penalty: the Instruct model's own generation config would add one of 1.1.
        gen = model.generate(**enc, do_sample=False, repetition_penalty=1.0, max_new_tokens=max_new_tokens,
                             stop_strings=['</answer>'], tokenizer=tok, pad_token_id=tok.pad_token_id)
        rows = gen[:, enc.input_ids.size(1):].tolist()
        for p, ids, n in zip(chunk, rows, completion_lengths(rows, tok, end_ids)):   # n: tokens really written
            text = full_completion(tok.decode(ids[:n], skip_special_tokens=True))
            out.append(dict(question=p['question'], gold=p['gold'], text=text, length=n,
                            acc=accuracy_reward(text, p['gold']), fmt=format_reward(text)))
    n = len(out)
    summary = dict(n=n, acc=sum(o['acc'] for o in out) / n, fmt=sum(o['fmt'] for o in out) / n,
                   length=sum(o['length'] for o in out) / n)
    return summary, out

The evaluation uses the first 200 questions of the GSM8K test split. The model never trains on them. It answers each one greedily, the most likely token every time, with the same limit of 300 new tokens. We set repetition_penalty=1.0 on purpose.

The Instruct model's own generation settings add a repetition penalty of 1.1, which would quietly change "the most likely token" into "the most likely token after a penalty". Our first evaluation had that penalty; this one does not. The paper reports pass@1 from several samples at temperature 0.6. One greedy answer is cheaper, and it gives the same result every time.

The run: first try, the base model

R1-Zero starts from a base model, so we did too: Qwen2.5-0.5B, a model of 0.5 billion parameters that was never tuned to follow instructions. Before training, it answered 10% of the 200 held-out questions correctly, and 35.5% of its greedy answers passed the format parser:

plain text
BEFORE training: accuracy 0.100  format 0.355  mean length 98.9 tokens  (102 s)
before: our accuracy rule and math_verify agree on 200 of 200 answers (both read the text inside the last answer tags)
before: our format parser and an independent regular expression agree on 200 of 200 answers

We gave this run 20 minutes, to keep most of our time budget for the second model. It made 35 steps. In windows of 10 steps:

plain text
== base model: 35 steps in 20 minutes
steps    accuracy  format  length  KL      clipped  equal-reward groups  seconds per step  psutil RSS GB
  1- 10     0.044   0.762    47.5   2.654   0.0104                 0.35              34.3            1.5
 11- 20     0.050   0.894    54.2   0.209   0.0014                 0.42              36.8            1.3
 21- 30     0.041   0.872    43.7   0.224   0.0006                 0.28              37.4            1.3
 31- 35     0.031   0.944    34.1   0.241   0.0016                 0.50              27.9            1.3
base model, largest KL of one step: 24.257 at step 1; median KL 0.204
00.250.500.751.001102030format reward, 1: 0.16format reward, 2: 0.36format reward, 3: 0.46format reward, 4: 0.53format reward, 5: 0.59format reward, 6: 0.65format reward, 7: 0.70format reward, 8: 0.74format reward, 9: 0.75format reward, 10: 0.76format reward, 11: 0.84format reward, 12: 0.87format reward, 13: 0.89format reward, 14: 0.91format reward, 15: 0.92format reward, 16: 0.91format reward, 17: 0.90format reward, 18: 0.90format reward, 19: 0.90format reward, 20: 0.89format reward, 21: 0.87format reward, 22: 0.86format reward, 23: 0.86format reward, 24: 0.85format reward, 25: 0.85format reward, 26: 0.86format reward, 27: 0.86format reward, 28: 0.85format reward, 29: 0.86format reward, 30: 0.87format reward, 31: 0.89format reward, 32: 0.92format reward, 33: 0.93format reward, 34: 0.94format reward, 35: 0.93accuracy reward, 1: 0.00accuracy reward, 2: 0.08accuracy reward, 3: 0.05accuracy reward, 4: 0.05accuracy reward, 5: 0.04accuracy reward, 6: 0.04accuracy reward, 7: 0.04accuracy reward, 8: 0.03accuracy reward, 9: 0.05accuracy reward, 10: 0.04accuracy reward, 11: 0.05accuracy reward, 12: 0.03accuracy reward, 13: 0.04accuracy reward, 14: 0.05accuracy reward, 15: 0.06accuracy reward, 16: 0.05accuracy reward, 17: 0.05accuracy reward, 18: 0.06accuracy reward, 19: 0.05accuracy reward, 20: 0.05accuracy reward, 21: 0.05accuracy reward, 22: 0.06accuracy reward, 23: 0.05accuracy reward, 24: 0.05accuracy reward, 25: 0.04accuracy reward, 26: 0.04accuracy reward, 27: 0.04accuracy reward, 28: 0.03accuracy reward, 29: 0.04accuracy reward, 30: 0.04accuracy reward, 31: 0.03accuracy reward, 32: 0.03accuracy reward, 33: 0.03accuracy reward, 34: 0.04accuracy reward, 35: 0.03format reward: 0.93accuracy reward: 0.03training stepmean reward (10-step average)
The base model run, 10-step averages. The format reward climbs to about 0.9 within 15 steps. The accuracy reward stays near 0.04 the whole time.
0801601102030mean length (tokens), 1: 140mean length (tokens), 2: 93mean length (tokens), 3: 78mean length (tokens), 4: 64mean length (tokens), 5: 56mean length (tokens), 6: 50mean length (tokens), 7: 46mean length (tokens), 8: 43mean length (tokens), 9: 44mean length (tokens), 10: 48mean length (tokens), 11: 40mean length (tokens), 12: 42mean length (tokens), 13: 43mean length (tokens), 14: 46mean length (tokens), 15: 48mean length (tokens), 16: 50mean length (tokens), 17: 53mean length (tokens), 18: 55mean length (tokens), 19: 57mean length (tokens), 20: 54mean length (tokens), 21: 55mean length (tokens), 22: 54mean length (tokens), 23: 52mean length (tokens), 24: 52mean length (tokens), 25: 52mean length (tokens), 26: 50mean length (tokens), 27: 49mean length (tokens), 28: 49mean length (tokens), 29: 45mean length (tokens), 30: 44mean length (tokens), 31: 39mean length (tokens), 32: 36mean length (tokens), 33: 35mean length (tokens), 34: 34mean length (tokens), 35: 35mean length (tokens): 35training steptokens per answer (10-step average)
Answer length in the base model run, 10-step averages. Answers get shorter, not longer: they drop from about 140 tokens at step 1 to between about 30 and 60 tokens.

The format reward was 0.762 in the first 10 steps and 0.944 in the last 5. The accuracy reward stayed between 0.031 and 0.050. And the answers got much shorter, from 139.9 tokens at step 1 to 34.1 tokens on average in steps 31 to 35. The held-out test after training shows the same thing on new questions:

plain text
AFTER training: accuracy 0.040  format 0.985  mean length 37.3 tokens  (51 s)
after: our accuracy rule and math_verify agree on 200 of 200 answers (both read the text inside the last answer tags)
after: our format parser and an independent regular expression agree on 200 of 200 answers

Format went from 35.5% to 98.5%, but accuracy fell from 10% to 4%, and the mean answer shrank from 98.9 to 37.3 tokens. On this model and in this short run, the rules taught the shape and nothing else.

The samples show what the model writes. Here are three answers at step 2 and three at step 25:

plain text
== base model, training step 2, gold answer 10
Q: At the burger hut, you can buy a burger for $5, french fries for $3, and a soft drink for $3.  If you order a special burger meal, you get all 3 of th...
  [accuracy 0, format 1] '<think> reasoning process here </think>\n<answer> Boone has eaten all but 1 of his burger meals </answer>'
  [accuracy 0, format 0] '<think> Burger-Broblems </think>. <answer> Joe eats at an awesome burger shack </answer>'
  [accuracy 0, format 0] "<think> calculating with burger marketing </think> <answer> (burger meals relate relationship 1 )<item> 1'* burger meals are 2 *$ 5 = $10 </item> <item> 1*\\$3 + 1*\\$3 = $6 </item> <item> 1*burger meals total = $12 </item> <think> assuming 1 * burger meals are equal to 1 * food items </think> <item> 6 * $2 = $12 </item> <item> 10 + $12 = $22 </item> <answer> savings = food items - burger meals = $12 - $10 = $2 </answe"
== base model, training step 25, gold answer 8000
Q: Mr. Wells has a garden of flowers with 50 rows. If each row has 400 flowers and Mr. Wells cuts 60% of the flowers, how many flowers are remaining in t...
  [accuracy 0, format 1] "<think> First, let's figure out the total number of flowers Mr. Wells has. </think> <answer> 50 x 400 = 20,000 flowers </answer>"
  [accuracy 0, format 1] '<think> Subtract 60% of total flowers </think> <answer> 40*50 - 0.6*50 = 1600 </answer>'
  [accuracy 0, format 1] '<think> Call us!</think> <answer> 400 </answer>'

At step 2 one answer copies "reasoning process here", the placeholder text of Table 1, into the think tags. The other two put a few unrelated words in the think tags and an answer with no number at all. By step 25 the answers are short and tidy: a sentence of thinking, then an expression or a number. The format rule cannot tell real thinking from a copied placeholder, so a short well-formed answer is the cheapest way to earn one point.

This is the problem behind "equal-reward groups" in the table. Suppose all 8 answers to a question have the right shape and all 8 are wrong. Then they all get a reward of 1, and every advantage is 0. The group gives no reward signal; only the KL term still acts on it. In steps 31 to 35, half of the groups were like that. The model had learned the shape, and it almost never solved a problem, so there was little left for the rewards to teach.

The table has one more surprise: a KL of 24.257 at step 1, against a median of 0.204. After a single optimiser step, the second mini-batch already met a few tokens whose probability had fallen far below the reference. We do not know which tokens they were. The KL was back at 0.143 by step 2.

Our plan was to try the base model first. If it earned almost no accuracy reward, we would switch to the instruction-tuned model. It earned almost none, and its held-out accuracy fell.

The run: second try, the Instruct model

Qwen2.5-0.5B-Instruct has the same size, but it was fine-tuned to follow instructions and to explain its steps. That is a long way from R1-Zero's pure start, and we say so. It is closer to DeepSeek-R1, which starts RL from a fine-tuned model (Part 5).

Before training, its greedy answers were right 16.5% of the time, and 55% had the right shape:

plain text
BEFORE training: accuracy 0.165  format 0.550  mean length 115.1 tokens  (90 s)
before: our accuracy rule and math_verify agree on 200 of 200 answers (both read the text inside the last answer tags)
before: our format parser and an independent regular expression agree on 200 of 200 answers

Then 122 steps of our GRPO, the whole 80-minute budget. Here are the first steps and the last steps of the log:

plain text
step    1  reward 0.500  acc 0.062  format 0.438  length  132.2  kl 1.16192  clipped 0.0395  memory 1.4 GB  43 s
step    2  reward 0.344  acc 0.062  format 0.281  length  158.5  kl 0.09710  clipped 0.0079  memory 1.4 GB  90 s
step    3  reward 0.406  acc 0.062  format 0.344  length  123.7  kl 0.13512  clipped 0.0134  memory 1.4 GB  134 s
step    4  reward 0.469  acc 0.000  format 0.469  length  183.0  kl 0.07868  clipped 0.0024  memory 1.4 GB  177 s
step    5  reward 0.625  acc 0.031  format 0.594  length  144.8  kl 0.08281  clipped 0.0026  memory 1.4 GB  223 s
step  118  reward 1.188  acc 0.281  format 0.906  length  165.5  kl 0.11107  clipped 0.0002  memory 1.1 GB  4672 s
step  119  reward 1.375  acc 0.406  format 0.969  length  141.3  kl 0.10981  clipped 0.0007  memory 1.1 GB  4705 s
step  120  reward 1.312  acc 0.344  format 0.969  length  158.0  kl 0.09827  clipped 0.0006  memory 1.1 GB  4745 s
step  121  reward 1.188  acc 0.281  format 0.906  length  158.6  kl 0.09319  clipped 0.0008  memory 1.1 GB  4787 s
step  122  reward 1.188  acc 0.344  format 0.844  length  187.2  kl 0.10839  clipped 0.0005  memory 1.1 GB  4829 s

And the 10-step windows:

plain text
== instruct model: 122 steps in 80 minutes
steps    accuracy  format  length  KL      clipped  equal-reward groups  seconds per step  psutil RSS GB
  1- 10     0.041   0.656   114.7   0.217   0.0077                 0.23              40.2            1.4
 11- 20     0.075   0.966    78.9   0.120   0.0021                 0.42              36.4            1.4
 21- 30     0.131   0.931   125.3   0.088   0.0003                 0.20              40.4            1.5
 31- 40     0.113   0.944   117.3   0.106   0.0007                 0.42              35.7            1.5
 41- 50     0.338   0.953   133.5   0.109   0.0012                 0.33              39.6            1.5
 51- 60     0.206   0.950   140.2   0.103   0.0006                 0.33              40.3            1.4
 61- 70     0.122   0.963   108.3   0.143   0.0005                 0.47              41.5            1.2
 71- 80     0.259   0.972   124.6   0.121   0.0006                 0.28              38.1            1.1
 81- 90     0.297   0.959   148.0   0.122   0.0006                 0.28              41.6            1.1
 91-100     0.303   0.953   138.0   0.137   0.0008                 0.12              42.0            1.1
101-110     0.250   0.978   143.4   0.122   0.0002                 0.30              40.7            1.2
111-120     0.359   0.959   147.1   0.111   0.0004                 0.12              38.1            1.2
121-122     0.312   0.875   172.9   0.101   0.0006                 0.00              42.0            1.1

Terminal output of the Instruct run: evaluation before, 122 GRPO steps, evaluation after

00.250.500.751.001255075100format reward, 1: 0.44format reward, 5: 0.42format reward, 9: 0.64format reward, 13: 0.83format reward, 17: 0.94format reward, 21: 0.98format reward, 25: 0.96format reward, 29: 0.93format reward, 33: 0.93format reward, 37: 0.95format reward, 41: 0.95format reward, 45: 0.95format reward, 49: 0.95format reward, 53: 0.93format reward, 57: 0.93format reward, 61: 0.95format reward, 65: 0.96format reward, 69: 0.96format reward, 73: 0.98format reward, 77: 0.99format reward, 81: 0.97format reward, 85: 0.95format reward, 89: 0.97format reward, 93: 0.95format reward, 97: 0.96format reward, 101: 0.95format reward, 105: 0.97format reward, 109: 0.98format reward, 113: 0.98format reward, 117: 0.96format reward, 121: 0.95format reward, 122: 0.94accuracy reward, 1: 0.06accuracy reward, 5: 0.04accuracy reward, 9: 0.04accuracy reward, 13: 0.03accuracy reward, 17: 0.05accuracy reward, 21: 0.09accuracy reward, 25: 0.12accuracy reward, 29: 0.15accuracy reward, 33: 0.12accuracy reward, 37: 0.09accuracy reward, 41: 0.12accuracy reward, 45: 0.24accuracy reward, 49: 0.33accuracy reward, 53: 0.30accuracy reward, 57: 0.23accuracy reward, 61: 0.22accuracy reward, 65: 0.18accuracy reward, 69: 0.13accuracy reward, 73: 0.17accuracy reward, 77: 0.23accuracy reward, 81: 0.26accuracy reward, 85: 0.26accuracy reward, 89: 0.31accuracy reward, 93: 0.32accuracy reward, 97: 0.29accuracy reward, 101: 0.27accuracy reward, 105: 0.23accuracy reward, 109: 0.24accuracy reward, 113: 0.31accuracy reward, 117: 0.33accuracy reward, 121: 0.36accuracy reward, 122: 0.35format reward: 0.94accuracy reward: 0.35training stepmean reward (10-step average)
The Instruct model run, 10-step averages. The format reward rises from below 0.5 to above 0.9 within about 15 steps. The accuracy reward rises from about 0.05 to about 0.3, and it moves a lot from step to step.

Three things happened. The format was learned first: the format reward was 0.656 in the first 10 steps and 0.966 in the next 10. The accuracy reward rose more slowly. It was 0.041 in the first 10 steps, 0.338 in steps 41 to 50, and 0.359 in steps 111 to 120. On the way it dipped as low as 0.122.

With only 4 questions per step, one step can show 0.000 and two steps later 0.250, so read the trend, not single steps. And the answers did not collapse. They dipped to about 79 tokens on average in steps 11 to 20, then grew back to about 140.

The held-out test confirms the trend on questions the model never trained on:

plain text
AFTER training: accuracy 0.390  format 0.980  mean length 159.8 tokens  (92 s)
after: our accuracy rule and math_verify agree on 200 of 200 answers (both read the text inside the last answer tags)
after: our format parser and an independent regular expression agree on 200 of 200 answers
0.165before0.390afteraccuracy0.550before0.980afterformat
The 200 held-out GSM8K questions, greedy answers, before and after 122 steps of GRPO. Accuracy goes from 0.165 to 0.390, and the share of well-formed answers from 0.550 to 0.980.

Accuracy went from 16.5% to 39.0%: 33 right answers before, 78 after. 196 of the 200 answers now have the right shape. On 200 questions a gap of 22.5 points is far larger than the noise of a single evaluation. But it is one run with one seed, so we do not claim a precise size.

Here are four held-out questions, before and after. The text is cut at 500 characters:

plain text
== held-out question 0, gold answer 18
Q: Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the rem...
  before [accuracy 1, format 1, 128 tokens] "<think> First, we need to calculate the total number of eggs laid by the ducks per day. Janet's ducks lay 16 eggs per day. She eats 3 eggs for breakfast every morning, and she bakes muffins for her friends every day, which means she eats 3 + 4 = 7 eggs for her friends. So, the total number of eggs laid per day is 16 - 7 = 9 eggs. </think> <answer> Janet makes $2 per fresh duck egg, so she makes 9 x $2 = $18 every day at the farmers' market. </answer>"
  after  [accuracy 0, format 1, 183 tokens] "<think> First, we need to calculate the total number of eggs laid by Janet's ducks per day. Since Janet's ducks lay 16 eggs per day, and she eats 3 eggs for breakfast every morning, she will eat 3 eggs every day. She also bakes 4 muffins every day, which means she will bake 4 eggs every day. So, the total number of eggs laid by her ducks every day is 3 + 4 = 7 eggs. \n\nNext, we need to calculate how much money Janet makes every day from selling the eggs. Since she sells the eggs at $2 per fresh d"
== held-out question 1, gold answer 3
Q: A robe takes 2 bolts of blue fiber and half that much white fiber.  How many bolts in total does it take?
  before [accuracy 1, format 1, 83 tokens] '<think> To find the total number of bolts needed, we need to add the number of blue and white bolts together. We know that the number of blue bolts is 2 and the number of white bolts is half of that, which is 2/2 = 1. So, the total number of bolts needed is 2 + 1 = 3. </think> <answer> 3 </answer>'
  after  [accuracy 1, format 1, 76 tokens] '<think> First, we need to identify the amount of blue and white fibers that each robe takes. According to the question, a robe takes 2 bolts of blue fiber and half that much white fiber. So, each robe takes 2 blue fibers and 2/2 = 1 white fiber. </think> <answer> 2 + 1 = 3 </answer>'
== held-out question 2, gold answer 70000
Q: Josh decides to try flipping a house.  He buys a house for $80,000 and then puts in $50,000 in repairs.  This increased the value of the house by 150%...
  before [accuracy 0, format 1, 36 tokens] '<think> To calculate the profit, we need to subtract the initial cost from the final value of the house. </think> <answer> $30,000 </answer>'
  after  [accuracy 1, format 1, 234 tokens] '<think> First, we need to calculate the total cost of the house. Josh spent $80,000 on the house and $50,000 on repairs, so the total cost was $80,000 + $50,000 = $130,000. Next, we need to calculate the increase in the value of the house. The increase in value was 150% of the original value, which means the increase was $80,000 * 1.5 = $120,000. The increase in value is $120,000, so the total increase in value is $120,000 + $80,000 = $200,000. The profit is the increase in value minus the cost '
== held-out question 3, gold answer 540
Q: James decides to run 3 sprints 3 times a week.  He runs 60 meters each sprint.  How many total meters does he run a week?
  before [accuracy 0, format 1, 76 tokens] '<think> To calculate the total distance James runs in a week, we need to multiply the number of sprints he runs each week by the distance he runs each sprint. So, 3 sprints * 60 meters per sprint = 180 meters per week. </think> <answer> James runs a total of 180 meters a week. </answer>'
  after  [accuracy 0, format 1, 63 tokens] '<think> First, we need to calculate the total distance James runs each sprint. Since he runs 60 meters each sprint and he runs 3 times a week, he runs 60 * 3 = 180 meters each week. </think> <answer> 180 meters </answer>'

Training did not fix every answer, and it broke some. Question 2 went from wrong to right: the trained model now works out the cost, the new value and the profit step by step. Question 0 went the other way: right before, wrong after. Question 3 is wrong both times, with the same mistake (it multiplies only two of the three numbers). A held-out accuracy is a count of such wins and losses, and on these 200 questions the wins were many more.

The KL stayed small

The windows table shows a mean KL between 0.088 and 0.217 in every window. The largest KL of a single step was at step 1:

plain text
instruct model, largest KL of one step: 1.162 at step 1; median KL 0.112

Eq. 2 shows how one step could jump much higher. For one token, the term is x−ln⁡x−1x - \ln x - 1 with x=πref/πθx = \pi_{ref}/\pi_\theta. If the model has made a token very unlikely that the reference still likes, xx is huge, and the term grows like xx itself. A token with πθ\pi_\theta 100 times smaller than πref\pi_{ref} costs 100−ln⁡100−1=94.4100 - \ln 100 - 1 = 94.4. A few such tokens are enough to lift the mean of a whole step.

Our first Instruct run had such a step, with a mean KL far above every other step. The fixed Instruct run has none, but the fixed base run has one at step 1, as we saw above. So such spikes are not only a product of the old padding bug, and we did not find their cause.

What did not happen: no Figure 3

0901801255075100mean length (tokens), 1: 132mean length (tokens), 5: 148mean length (tokens), 9: 115mean length (tokens), 13: 98mean length (tokens), 17: 72mean length (tokens), 21: 78mean length (tokens), 25: 107mean length (tokens), 29: 124mean length (tokens), 33: 124mean length (tokens), 37: 114mean length (tokens), 41: 117mean length (tokens), 45: 131mean length (tokens), 49: 135mean length (tokens), 53: 140mean length (tokens), 57: 145mean length (tokens), 61: 136mean length (tokens), 65: 122mean length (tokens), 69: 111mean length (tokens), 73: 108mean length (tokens), 77: 108mean length (tokens), 81: 127mean length (tokens), 85: 139mean length (tokens), 89: 144mean length (tokens), 93: 147mean length (tokens), 97: 143mean length (tokens), 101: 139mean length (tokens), 105: 142mean length (tokens), 109: 142mean length (tokens), 113: 138mean length (tokens), 117: 146mean length (tokens), 121: 149mean length (tokens), 122: 156mean length (tokens): 156training steptokens per answer (10-step average)
Answer length in the Instruct model run, 10-step averages. It dips to about 80 tokens around step 15, then climbs back to between about 125 and 150 tokens.

Our answers did get somewhat longer. On the held-out questions the mean went from 115 tokens before training to 160 after. During training, the 10-step mean was 115 in the first window. After a dip in steps 11 to 20, it stayed between 138 and 148 from step 81 to step 120.

That is a rise of about 40%, not the twenty-fold growth of Figure 3. We cannot tell from one run why it happened. Longer answers may simply carry more of the arithmetic written out.

Our setting is far from R1-Zero's in every way the box above lists, so we did not expect a Figure 3. Our limit is 300 new tokens, about 3% of R1-Zero's final mean length, so a long answer is cut off and earns nothing. We ran 122 steps, not 10,400. And GSM8K problems need only a few arithmetic steps. These are differences, not a tested explanation: we did not run the experiment that would separate them.

We did not see an aha moment either. The 0.5B model writes simple, linear arithmetic. It does not stop to say "Wait" and check itself. We did reproduce the first part of the story. Rules alone, with no example of good reasoning, taught the model the format in about 15 steps. They also raised its accuracy on new questions.

What the laptop could and could not do

The run used an Apple laptop with 64 GB of memory, through PyTorch's mps device. The policy, the frozen reference and the optimiser state are all full float32 copies of the model.

Our first run did not get far. After about 30 steps its steps slowed down sharply, because the process kept growing until the machine started to swap. Every step writes answers of a different length, so the mps allocator keeps freed blocks of many sizes and the cache grows.

The fix is two lines in free_memory: drop the step's tensors, collect garbage and call torch.mps.empty_cache() after every step. The resident size that psutil reports does not see mps buffers: it stayed near 1.5 GB. So a small script, r1_part8_memwatch.py, read the process footprint once a minute. That is the number Activity Monitor shows:

plain text
minute   0.0  after step    2  footprint  30.0 GB
minute   1.0  after step    4  footprint  22.0 GB
minute   2.0  after step    5  footprint  27.0 GB
minute  40.1  after step   64  footprint  27.0 GB
minute  77.1  after step  120  footprint  26.0 GB
minute  78.1  after step  121  footprint  29.0 GB
minute  79.1  after step  122  footprint  29.0 GB

Over the 80 readings of the fixed run, the footprint stayed between 20 and 34 GB, with no upward trend. It moves from reading to reading because a reading can land anywhere inside a step. The logits of 8 answers over a vocabulary of 151,936 tokens take a lot of memory while they exist. The step time stayed flat too, as the seconds-per-step column of the windows table shows. So the run used its whole 80 minutes for 122 steps instead of stalling.

Generation is the slow part of RL. Each of our rollouts writes 32 answers of up to 300 tokens before the model can learn anything, and then takes 2 optimiser steps. In the later paper, each R1-Zero rollout wrote 8,192 answers of up to 32,768 tokens (65,536 after step 8,200). Then it took 16 optimiser steps.

Compare the most tokens one rollout may write: 8,192×32,7688{,}192 \times 32{,}768 against 32×30032 \times 300, almost 28,000 times more. That is a ratio of maximum rollout budgets, not of tokens actually written per optimiser step. Our run had 122 rollouts and 244 optimiser steps. The method is the same; the scale is not.

Method 2: the same recipe with TRL

TRL's GRPOTrainer does every step of our loop: sampling, rewards, advantages, the loss and the optimiser. We give it the same template and the same two rules. The learning rate, ε\varepsilon, β\beta and G are the same too. We run it for 4 steps only, to show the route. It is a demonstration, not a copy of our run. Each TRL step here uses 2 questions, not 4, and takes one optimiser step, not two.

python
def accuracy(completions, gold, **kw):               # TRL passes the dataset columns (here: gold) as keyword arguments
    return [accuracy_reward(full_completion(c), g) for c, g in zip(completions, gold)]   # put the prefilled tag back


def fmt(completions, **kw):
    return [format_reward(full_completion(c)) for c in completions]


with common.heavy('part8'):
    from datasets import Dataset
    from transformers import AutoModelForCausalLM, AutoTokenizer
    from trl import GRPOConfig, GRPOTrainer
    name = 'Qwen/Qwen2.5-0.5B-Instruct'
    train, _ = load_gsm8k()
    ds = Dataset.from_list([dict(prompt=make_prompt(q['question']), gold=q['gold']) for q in train[:64]])
    args = GRPOConfig(
        output_dir=tempfile.mkdtemp(), max_steps=4, learning_rate=3e-6, lr_scheduler_type='constant', loss_type='grpo', beta=0.04, epsilon=0.2,
        use_bias_correction_kl=False,                           # the KL of R1's Eq. 2, as in our package
        num_generations=8, per_device_train_batch_size=16,      # 16 completions per step = 2 questions x G = 8
        max_completion_length=300, temperature=1.0, top_p=1.0, top_k=0, repetition_penalty=1.0,
        generation_kwargs=dict(stop_strings=['</answer>']),      # stop where our sampler stops
        logging_steps=1, report_to='none',
        bf16=False, save_strategy='no', seed=0)
    model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float32)
    tok = AutoTokenizer.from_pretrained(name)
    model.generate = functools.partial(model.generate, tokenizer=tok)   # generate() needs the tokenizer for a stop string
    trainer = GRPOTrainer(model=model, reward_funcs=[accuracy, fmt], args=args, train_dataset=ds,
                          processing_class=tok)
    t = common.timer()
    trainer.train()

TRL passes the extra columns of the dataset, here gold, to every reward function as keyword arguments. The reward functions put the prefilled <think> back in front, as sample_group does.

Three settings matter. lr_scheduler_type='constant' keeps the learning rate fixed, as in our loop. use_bias_correction_kl=False gives the KL of Eq. 2, as the loss check above showed. And generation must stop at </answer>, as ours does. TRL passes generation_kwargs to transformers, but a stop string also needs the tokenizer, which TRL does not pass. So we bind the tokenizer to model.generate with functools.partial.

Without the stop string, TRL would keep sampling after </answer>. Those extra tokens would enter its loss, its KL and its lengths, while the rewards might see only the cut text. That is a different objective.

With the stop string, one small difference is left. After a stop on </answer>, TRL's mask keeps the first padding token. For Qwen, padding is the end-of-text token, and TRL reads it as a sampled end. Our completion_lengths leaves it out. So TRL's completions/mean_length counts one token more for each answer that stopped on </answer>.

TRL logs the same quantities as our loop:

plain text
step 1: reward 0.8125  rewards/accuracy/mean 0.0625  rewards/fmt/mean 0.75  completions/mean_length 109.7  kl -8.389e-09  clip_ratio/region_mean 0  frac_reward_zero_std 0
step 2: reward 0.6875  rewards/accuracy/mean 0.0625  rewards/fmt/mean 0.625  completions/mean_length 175.8  kl 0.01768  clip_ratio/region_mean 0  frac_reward_zero_std 0
step 3: reward 1  rewards/accuracy/mean 0.0625  rewards/fmt/mean 0.9375  completions/mean_length 138.1  kl 0.05312  clip_ratio/region_mean 0  frac_reward_zero_std 0
step 4: reward 1.438  rewards/accuracy/mean 0.5  rewards/fmt/mean 0.9375  completions/mean_length 142.3  kl 0.05238  clip_ratio/region_mean 0  frac_reward_zero_std 0
4 steps in 96 s on mps

Four steps of 2 questions prove nothing about learning, and we do not read the rewards as a trend. One column is worth a look. clip_ratio/region_mean is 0 at every step. Here TRL takes one optimiser step per batch of answers, so the ratio is exactly 1, as Step 4 predicts.

The library is the right tool for real work: it adds vLLM for fast sampling, many GPUs and many loss variants. Our package is the right tool to see what happens inside. The check above shows that, on the same tensors, they compute the same loss.

Run it yourself

The code needs Python, torch, transformers and datasets, plus trl and math-verify for the checks. The training run needs a machine with about 64 GB of memory, or a GPU.

bash
git clone https://github.com/ishwar6/ishwar-books
cd ishwar-books/code/papers/deepseek-r1
python r1_part8_probe.py prefill      # do the models earn any reward before training?
python r1_part8_check.py              # the equations on small numbers, and our loss against TRL
python r1_part8_train.py Qwen/Qwen2.5-0.5B-Instruct 400 80   # before eval, 80 minutes of GRPO, after eval
python r1_part8_train.py Qwen/Qwen2.5-0.5B 400 20            # the base model, 20 minutes
python r1_part8_memwatch.py <pid> instruct   # optional, in a second terminal: memory once a minute
python r1_part8_trl.py                # Method 2: 4 steps of TRL's GRPOTrainer
python r1_part8_stats.py              # the 10-step windows of the logs

Some good next experiments:

  • Give the format reward a weight of 0.5, so a right answer counts more than a tidy one. Does the base model learn more?
  • Raise the limit to 1,000 tokens and train on harder questions, such as MATH, to give length a reason to grow.
  • Set minibatches to 1 and check that the share of clipped tokens drops to 0, as Step 4 predicts.

The paper and the code

Paper (arXiv 2501.12948) · Official repository · Our scripts and the minir1 package