DeepSeek-R1, explained · Part 8 of 8 · Covers §2.2 (the whole R1-Zero recipe)
Train a Reasoner from Scratch: R1-Zero's Recipe in 330 Lines
We write the whole DeepSeek-R1-Zero method in a small Python package: the template, the two reward rules and GRPO. We check it against TRL and train a 0.5B model on a laptop.
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and others. arXiv 2025 (Nature 2025), 2025. arXiv:2501.12948
Parts 2 and 3 took R1-Zero's recipe apart, one equation and one rule at a time. Here we put it back together in about 330 lines of Python. The code takes a model, asks it maths questions, and makes it better at them with only two rule checks. Is the final answer right? Is it in the requested shape?
The proof has three steps. First, our loss gives the same number as the loss code of TRL, a widely used library, on the same tensors. Second, our accuracy rule gives the same verdicts as math_verify, a library built to compare maths answers. Our format rule agrees with an independent second check. Third, we run the recipe on a laptop and report what happened, including what did not.
The recipe in one paragraph
The package lives in code/papers/deepseek-r1/minir1. It uses PyTorch and the transformers library to load the model and to sample text. Every step of the method itself is our own code.
| File | Lines | What it holds | Paper |
|---|---|---|---|
template.py | 22 | The prompt of Table 1 | §2.2.3 |
rewards.py | 81 | The accuracy rule and the format rule | §2.2.2 |
grpo.py | 87 | Sampling, log-probabilities, Eq. 3, Eq. 1 and Eq. 2 | §2.2.1 |
train.py | 109 | The data, the loop and the log | §2.2 |
eval.py | 28 | Held-out questions, scored with the same rules | §2.2.4 |
The page walks through the files in the order a question meets them. For each piece you see the lines of the paper, the code, and a check.
The template
Here is the whole file:
"""Table 1 of the paper: the only instruction DeepSeek-R1-Zero gets. It fixes the shape of the answer
(<think> ... </think> <answer> ... </answer>) and says nothing about how to reason."""
TEMPLATE = (
"A conversation between User and Assistant. The user asks a question, and the Assistant solves it. "
"The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. "
"The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, "
"respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer>. "
"User: {prompt}. Assistant:"
)
# Our one addition: the prompt ends with the opening tag, so a 0.5B model starts inside <think> (see the text).
PREFILL = ' <think>'
def make_prompt(question):
"""Put one question into the template. The text is fed to the model as is: no chat template, no examples."""
return TEMPLATE.format(prompt=question.strip().rstrip('.')) + PREFILL
def full_completion(generated):
"""What the rewards see: the prefilled tag plus the generated text."""
return '<think>' + generatedThe constant is Table 1 word for word. We made one addition, and the code says so: the prompt ends with the opening tag <think>. The model then starts inside the thinking block.
Why the prefill? Without it, our 0.5B models rarely passed the format check. We sampled 4 answers to each of 24 training questions, at temperature 1, with at most 300 new tokens:
Qwen/Qwen2.5-0.5B: 96 samples, accuracy reward 0.010, format reward 0.021, mean length 198 tokens, 45 s
Qwen/Qwen2.5-0.5B-Instruct: 96 samples, accuracy reward 0.031, format reward 0.000, mean length 215 tokens, 44 sWith the plain template, the base model passed the format check on 2 of 96 answers (0.021) and the instruction-tuned model on none. With the prefill, both start to pass it:
Qwen/Qwen2.5-0.5B: 96 samples, accuracy reward 0.021, format reward 0.115, mean length 137 tokens, 45 s
Qwen/Qwen2.5-0.5B-Instruct: 96 samples, accuracy reward 0.052, format reward 0.427, mean length 141 tokens, 43 sThe base model now passes on 11 of 96 answers (0.115) and the Instruct model on 41 of 96 (0.427). GRPO's reward signal comes only from differences inside a group. If every answer in a group gets the same reward, the rewards push nothing up or down (we show why in the GRPO section). A start of 2% leaves almost every group with equal rewards. A start of 10% to 40% gives many groups a mix of good and bad answers.
The prefill does not tell the model how to reason, so it keeps the spirit of §2.2.3. The rewards still see the full text: full_completion puts the tag back in front of what the model wrote.
The rewards
Here is the whole file:
"""The two rule-based rewards of §2.2.2. No neural reward model: a few lines of Python decide each reward."""
import re
from fractions import Fraction
ANSWER_RE = re.compile(r'<answer>(.*?)</answer>', re.DOTALL)
TAG_RE = re.compile(r'</?(think|answer)>')
# One number: '-3', '12.5', '1/2', '-7/4', '1e3'. The lookarounds stop it from starting or ending inside a word.
NUMBER_RE = re.compile(r'(?<![\w.])(-?\d+(?:\.\d+)?(?:[eE][-+]?\d+)?)(?:\s*/\s*(\d+(?:\.\d+)?))?(?!\w|\.\d)')
OPERATOR_RE = re.compile(r'\d\s*(?:[-+*×÷^])\s*\(?-?\d') # '10 - 2', '3 * 4', '2^3': an expression, not a number
def extract_answer(completion):
"""The text inside the last <answer> ... </answer> pair, or None when there is no such pair."""
found = ANSWER_RE.findall(completion)
return found[-1].strip() if found else None
def extract_boxed(text):
"""What is inside the last \\boxed{...}, counting braces so that '\\boxed{\\frac{1}{2}}' works. None if unclosed."""
start = text.rfind('\\boxed')
i = text.find('{', start) if start >= 0 else -1
if i < 0:
return None
depth = 0
for j in range(i, len(text)):
depth += {'{': 1, '}': -1}.get(text[j], 0)
if depth == 0: # the brace that closes the box
return text[i + 1:j]
return None
def to_number(text):
"""The final value of an answer as an exact Fraction, or None when it is not one number.
'$1,250.00' -> 1250, 'x = -3' -> -3, '3 + 4 = 7' -> 7, '1/2' -> 1/2, '\\boxed{36/2}' -> 18, '1e3' -> 1000,
'10 - 2' -> None (an unworked expression), '2^3' -> None, 'none' -> None."""
if '\\boxed' in text:
text = extract_boxed(text) # the box is the final answer
if text is None:
return None
text = text.split('=')[-1] # the final expression: what comes after the last '='
text = re.sub(r'\\[dt]?frac\{([^{}]*)\}\{([^{}]*)\}', r'\1/\2', text) # \frac{a}{b} -> a/b
text = re.sub(r'\\text\{[^{}]*\}|\\[$%,!]|[$%]', ' ', text) # unit text, dollar and per cent signs
while re.search(r'\d,\d{3}(?!\d)', text):
text = re.sub(r'(\d),(\d{3})(?!\d)', r'\1\2', text) # 1,250,000 -> 1250000
if OPERATOR_RE.search(text):
return None # a sum or a power left unworked: no single number
found = NUMBER_RE.findall(text)
if not found:
return None
a, b = found[-1] # the last number of the final expression
value = Fraction(float(a)) if 'e' in a.lower() else Fraction(a)
return None if b and float(b) == 0 else (value / Fraction(b) if b else value)
def accuracy_reward(completion, gold):
"""1.0 when the number in the <answer> tags equals the known answer, else 0.0 (the 'accuracy reward')."""
answer = extract_answer(completion)
if answer is None:
return 0.0 # no answer tags: nothing to check
value = to_number(answer)
return float(value is not None and value == Fraction(gold))
def format_reward(completion):
"""1.0 when the completion is exactly <think> ... </think> <answer> ... </answer>: each tag once, in this order,
both blocks non-empty, nothing before, between or after them (the 'format reward'). Part 3's tag parser."""
tags = list(TAG_RE.finditer(completion))
if [t.group(0) for t in tags] != ['<think>', '</think>', '<answer>', '</answer>']:
return 0.0 # a tag is missing, repeated or out of order
t1, t2, a1, a2 = tags
if completion[:t1.start()].strip() or completion[t2.end():a1.start()].strip() or completion[a2.end():].strip():
return 0.0 # text before, between or after the two blocks
if not completion[t1.end():t2.start()].strip() or not completion[a1.end():a2.start()].strip():
return 0.0 # an empty block
return 1.0
def total_reward(completion, gold):
"""The reward GRPO sees. Version 1 of the paper names the two rules but not how they combine; the later version
adds them with the same weight (its Eq. 4). So one answer earns 0, 1 or 2."""
return accuracy_reward(completion, gold) + format_reward(completion)extract_answer takes the last answer pair, so an answer that changes its mind is graded on its final word.
to_number reads the final value of the answer. If there is a \boxed{...}, it takes what is inside the last box, counting braces. It keeps only what comes after the last =, so "3 + 4 = 7" gives 7. It removes dollar signs and the commas of thousands, so a dollar price of 1,250.00 written with its sign gives 1250. A fraction counts as a fraction and 1e3 as a thousand. A sum or a power that was never worked out, such as "10 - 2", gives no number at all, and so no reward.
format_reward is the tag parser of Part 3. It finds the four tags and asks for exactly <think>, </think>, <answer>, </answer>, once each and in this order. Both blocks must hold some text, and nothing may stand before, between or after them.
Here are the fixed rules on those hand-made cases:
accuracy, answer '18' gold 18: value 18 reward 1
accuracy, answer '$1,250.00' gold 1250: value 1250 reward 1
accuracy, answer '3 + 4 = 7' gold 7: value 7 reward 1
accuracy, answer '1/2' gold 2: value 1/2 reward 0
accuracy, answer '1e3' gold 3: value 1000 reward 0
accuracy, answer '\\boxed{36/2}' gold 18: value 18 reward 1
accuracy, answer '10 - 2' gold 8: value None reward 0
accuracy, answer '2^3' gold 8: value None reward 0
format 1: '<think> 2 + 3 = 5 </think> <answer> 5 </answer>'
format 0: '<think> </think><answer> </answer>'
format 0: '<think> a </think><think> b </think><answer> 5 </answer>'
format 0: '<think> a </think><answer> 5 </answer> more'
format 0: '<think> a <answer> 5 </answer>'Version 1 of the paper names the two rewards but does not say how they combine. The later, longer version adds them with the same weight (its Eq. 4). We do the same, so one answer earns 0, 1 or 2.
Method 2, with libraries. The math_verify library parses two maths expressions and checks if they are equal. It knows fractions, LaTeX and units.
We ran it and our accuracy rule on every answer of the held-out evaluation. Both read the text inside the last answer tags and compare it with the GSM8K answer. For the format rule we wrote a second, independent check. It is one regular expression: neither block may contain a tag, and each block must hold a non-space character.
before: our accuracy rule and math_verify agree on 200 of 200 answers (both read the text inside the last answer tags)
before: our format parser and an independent regular expression agree on 200 of 200 answers
after: our accuracy rule and math_verify agree on 200 of 200 answers (both read the text inside the last answer tags)
after: our format parser and an independent regular expression agree on 200 of 200 answersBoth checks agree on all 400 answers of the Instruct run (200 before training, 200 after). They also agree on the 400 answers of the base run.
Read these numbers for what they are. The math_verify check covers only the accuracy rule, and both sides read the same text: what extract_answer found inside the last answer tags. It does not test the extraction itself, and it says nothing about format. The format check is the separate second line. GSM8K answers are also whole numbers, which keeps the comparison easy: the fraction and power cases above barely occur here. For MATH or AIME problems, use math_verify, or a parser like it, as the accuracy rule.
Group Relative Policy Optimization
Now the heart of the method. GRPO comes from an earlier DeepSeek paper on maths, and Part 2 explains each equation term by term. Here we write each equation as one short function.
Step 1: sample a group
@torch.no_grad()
def sample_group(model, tok, prompts, G, max_new_tokens, temperature=1.0):
"""Sample G completions for each prompt from the current policy (the 'old policy' of Eq. 1)."""
texts = [p for p in prompts for _ in range(G)] # B = len(prompts) * G, groups side by side
enc = tok(texts, return_tensors='pt', padding=True, padding_side='left').to(model.device)
out = model.generate(**enc, do_sample=True, temperature=temperature, top_p=1.0, top_k=0,
repetition_penalty=1.0, max_new_tokens=max_new_tokens,
stop_strings=['</answer>'], tokenizer=tok, pad_token_id=tok.pad_token_id)
prompt_ids, completion_ids = enc.input_ids, out[:, enc.input_ids.size(1):] # (B, P), (B, T)
lengths = completion_lengths(completion_ids.tolist(), tok, end_token_ids(model, tok))
T = completion_ids.size(1)
completion_mask = (torch.arange(T)[None, :] < torch.tensor(lengths)[:, None]).long().to(model.device) # (B, T)
texts = [full_completion(tok.decode(ids[:n], skip_special_tokens=True)) # only the real tokens
for ids, n in zip(completion_ids.tolist(), lengths)]
return dict(prompt_ids=prompt_ids, prompt_mask=enc.attention_mask,
completion_ids=completion_ids, completion_mask=completion_mask, texts=texts)The function repeats each prompt G times, so the G answers to one question sit side by side. Padding on the left lines up the ends of the prompts, which is where generation starts. We sample at temperature 1 with no top-p or top-k filter, so the answers come from the model's real distribution. The later version of the paper also uses temperature 1.
Generation stops at the string </answer>, or at an end-of-text token, or at the length limit. When one answer stops early, generate fills the rest of its row with padding. For Qwen the padding token is the same token as end-of-text, so the ids alone cannot tell a sampled end from a filler.
completion_lengths decides from how the row stopped. An answer that stopped on </answer> ends with the token that completes the string. An answer that stopped on a sampled end token keeps that token, because the model chose it. An answer that hit the limit keeps everything.
mask, stopped on </answer>, then 2 padding tokens: tokens [' ', '5', ' </', 'answer', '>', '<|endoftext|>', '<|endoftext|>'] -> [1, 1, 1, 1, 1, 0, 0]
mask, stopped on a sampled <|im_end|>, then 1 padding token: tokens [' ', '5', '<|im_end|>', '<|endoftext|>'] -> [1, 1, 1, 0]
mask, hit the length limit: tokens [' ', '5', ' is', ' the', ' answer'] -> [1, 1, 1, 1, 1]The mask is 1 for the tokens the model really wrote and 0 after. Every later step uses it, so padding never counts. The evaluation uses the same function for its lengths.
Step 2: the log-probability of each token
To change the probability of an answer, we need the probability of each of its tokens. One forward pass over prompt plus answer gives them all.
def per_token_logps(model, batch, temperature=1.0):
"""log pi(o_t | q, o_<t) for every completion token: the numbers that go into the ratio of Eq. 1."""
ids = torch.cat([batch['prompt_ids'], batch['completion_ids']], 1) # (B, P + T)
mask = torch.cat([batch['prompt_mask'], batch['completion_mask']], 1) # (B, P + T)
T = batch['completion_ids'].size(1)
# The logits at position t predict token t + 1, so keep the T + 1 last positions and drop the very last one.
logits = model(input_ids=ids, attention_mask=mask, logits_to_keep=T + 1).logits[:, :-1].float() # (B, T, V)
logits = logits / temperature
chosen = logits.gather(-1, batch['completion_ids'].unsqueeze(-1)).squeeze(-1) # (B, T): score of the token
return chosen - torch.logsumexp(logits, -1) # (B, T): log softmaxThe logits at position predict the token at position . So for answer tokens we keep the last positions and drop the very last one. gather picks the logit of the token that was actually written. Subtracting logsumexp over the vocabulary turns that logit into a log-probability, without building the full table of 151,936 probabilities per position a second time.
The paper writes for a whole answer. The probability of an answer is the product of the probabilities of its tokens, so its logarithm is a sum over tokens. The code keeps one number per token, because Eq. 1 works on each token, as the next box shows.
Step 3: the group advantage (Eq. 3)
Read it left to right. The advantage of answer is its reward , minus the mean reward of its group. That difference is divided by the standard deviation of the group's rewards. The paper does not say which standard deviation. We use the sample one (divide by ), as TRL does. We also add below the line, so a group of equal rewards does not divide by zero.
def group_advantages(rewards, G):
"""Eq. 3: A_i = (r_i - mean(r_1..r_G)) / std(r_1..r_G), each answer against its own group."""
r = rewards.view(-1, G) # (questions, G)
mean, std = r.mean(1, keepdim=True), r.std(1, keepdim=True) # std with n - 1, as TRL
return ((r - mean) / (std + 1e-4)).view(-1) # (B,): 0 when all equalA worked example with . The rewards are 2 (right and well formed), 0, 1 (well formed, wrong) and 0. The mean is 0.75.
The deviations are 1.25, -0.75, 0.25 and -0.75. Their squares add up to 2.75, and . So the advantages are , then -0.78, 0.26 and -0.78. Our function and Python's statistics module agree:
Eq. 3, rewards of one group: [2.0, 0.0, 1.0, 0.0]
mean 0.75 std (n - 1) 0.9574
advantages (ours) : [1.3054, -0.7833, 0.2611, -0.7833]
advantages (statistics): [1.3054, -0.7833, 0.2611, -0.7833]
all four the same reward: [0.0, 0.0, 0.0, 0.0]
tokens clipped, ours 0.0202, TRL clip_ratio/region_mean 0.0202
mean KL per token, ours 0.273503, TRL kl 0.273503
tokens clipped, ours 0.1254, TRL clip_ratio/region_mean 0.1254
mean KL per token, ours 0.273503, TRL kl 0.273503
tokens clipped, ours 0.0202, TRL clip_ratio/region_mean 0.0202
mean KL per token, ours 0.273503, TRL kl 9.148457The last line is the case of equal rewards: four zero advantages. Such a group gives no reward signal. It is not completely silent, though. The KL term of the next steps still pulls each token toward the reference model. With , , and , our loss still has a gradient:
A = 0, pi_theta 0.6, pi_ref 0.5, beta 0.04: gradient of the loss on log pi_theta = 0.006667That is : a small pull back toward the reference.
Step 4: the clipped ratio (Eq. 1)
This is the loss our code minimises. is the ratio for token of answer . is the KL term of the next step, for the same token. The minus sign in front turns "maximise the objective" into "minimise the loss", which is what an optimiser does.
Four tokens show what the min and the clip do, with :
Eq. 1 term, A = +1, ratio 1.5: min(+1.50, +1.20) = +1.20
Eq. 1 term, A = +1, ratio 0.7: min(+0.70, +0.80) = +0.70
Eq. 1 term, A = -1, ratio 0.5: min(-0.50, -0.80) = -0.80
Eq. 1 term, A = -1, ratio 1.3: min(-1.30, -1.20) = -1.30With a positive advantage, a ratio of 1.5 is cut to 1.2: the token has already gained enough, so this batch stops pushing it. A ratio of 0.7 is kept: the token lost ground, and it may climb back. With a negative advantage the mirror holds. A ratio of 0.5 counts as 0.8, flat, so the push down stops. A ratio of 1.3 keeps its full value: the bad token became more likely, and the loss keeps pushing it down.
So a token counts as clipped only in two cases: ratio below with a negative advantage, or above with a positive one. Only then does the min pick the flat branch. Our loss counts clipped tokens this way, the same way as TRL's clip_ratio/region_mean:
clipped? A = +1, ratio 1.5: yes
clipped? A = +1, ratio 0.7: no
clipped? A = -1, ratio 0.5: yes
clipped? A = -1, ratio 1.3: noThe ratios 0.7 and 1.3 lie outside the interval, but they are not clipped. Our first version counted every ratio outside the interval, so it reported these two as clipped too. That made the clipped share in our first logs too high.
The ratio is not 1 only when the model changed since sampling. With one optimiser step per batch of answers, it is always exactly 1 and the clip never acts. The later version of the paper splits each batch into 16 mini-batches. We split ours into 2, so the second half of each step meets a real ratio.
Step 5: the KL leash (Eq. 2)
With for one token:
Eq. 2, pi_ref 0.5, pi_theta 0.5: 1.0000 - ln 1.0000 - 1 = 0.00000
Eq. 2, pi_ref 0.5, pi_theta 0.6: 0.8333 - ln 0.8333 - 1 = 0.01565
Eq. 2, pi_ref 0.5, pi_theta 0.4: 1.2500 - ln 1.2500 - 1 = 0.02686
Eq. 2, pi_ref 0.5, pi_theta 0.9: 0.5556 - ln 0.5556 - 1 = 0.14334When the two models agree, the cost is 0. Moving a probability from 0.5 to 0.6 costs 0.016. Moving it from 0.5 to 0.9 costs 0.143, nine times more for four times the move. Small drifts are cheap and big ones are expensive.
The whole loss, in one function
def grpo_loss(logps, old_logps, ref_logps, advantages, mask, eps=0.2, beta=0.04):
"""Eq. 1 with Eq. 2, per token, as a loss to minimise (the paper maximises J, so the sign flips)."""
ratio = torch.exp(logps - old_logps) # pi_theta / pi_theta_old, (B, T)
clipped = torch.clamp(ratio, 1 - eps, 1 + eps) # clip(ratio, 1 - eps, 1 + eps)
A = advantages.unsqueeze(1) # (B, 1): one advantage for every token of answer i
surrogate = torch.min(ratio * A, clipped * A) # the pessimistic one of the two
log_r = ref_logps - logps # log(pi_ref / pi_theta)
kl = torch.exp(log_r) - log_r - 1 # Eq. 2, always >= 0
per_token = -(surrogate - beta * kl) # (B, T)
m = mask.float()
per_answer = (per_token * m).sum(1) / m.sum(1).clamp(min=1) # 1/|o_i| sum over the tokens of answer i
loss = per_answer.mean() # 1/G sum over answers (and over questions)
with torch.no_grad(): # sums and a token count, so the loop can take one token-weighted mean over a whole step
# A token is clipped when the min of Eq. 1 picks the flat branch: ratio < 1 - eps with A < 0, or
# ratio > 1 + eps with A > 0 (TRL's clip_ratio/region_mean counts the same tokens).
is_clipped = ((ratio < 1 - eps) & (A < 0)) | ((ratio > 1 + eps) & (A > 0))
stats = dict(kl_sum=(kl * m).sum().item(), clipped_sum=(is_clipped.float() * m).sum().item(),
tokens=m.sum().item())
return loss, statsEach line is one piece of the equations. ratio is . torch.min(ratio * A, clipped * A) is the min of Eq. 1. kl is Eq. 2 per token. Then the mask removes padding, the inner mean is the sum, and the outer mean is the sum.
For the log, the function returns sums, not means: the total KL, the number of clipped tokens and the number of real tokens. The loop adds them over all slices of a step and divides once. Slices hold different numbers of tokens, so an average of slice means would weight them wrongly, and our first version did exactly that.
Method 2, with TRL. TRL's GRPOTrainer has a method, _compute_loss, that takes the same tensors: prompt and answer tokens, masks, advantages, old and reference log-probabilities. We built a trainer around the same model and called that method directly, so this is TRL's own code.
The batch is 8 real answers of Qwen2.5-0.5B-Instruct to 2 GSM8K questions. Real rewards of 8 short answers are often all equal, and then every advantage is 0. Fixed rewards turn every term on. In training, the three policies differ, so we took two optimiser steps first. The reference is the start, the old policy comes after one step, and the current one after two.
A = 0, pi_theta 0.6, pi_ref 0.5, beta 0.04: gradient of the loss on log pi_theta = 0.006667
== (b) our loss against TRL on one fixed batch
batch: 2 questions x 4 completions, completion tensor (8, 96), 694 real completion tokens
fixed rewards used for the check: [2.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 2.0]
advantages: [1.3054, -0.7833, 0.2611, -0.7833, 0.0, 0.0, -1.2246, 1.2246]
epsilon 0.2, beta 0.04, TRL use_bias_correction_kl=False:
tokens clipped, ours 0.0202, TRL clip_ratio/region_mean 0.0202
mean KL per token, ours 0.273503, TRL kl 0.273503
loss, ours: +64.69831085
loss, TRL 1.18.2 GRPOTrainer._compute_loss: +64.69834900
|difference|: loss 3.81e-05, per-token log-probs 1.30e-06, gradient of lm_head 1.29e-02
relative difference of the two losses: 5.9e-07
our loss on the log-probs TRL computed: +64.69834900, |difference| from TRL 0.00e+00
epsilon 0.02, beta 0.04, TRL use_bias_correction_kl=False:
tokens clipped, ours 0.1254, TRL clip_ratio/region_mean 0.1254
mean KL per token, ours 0.273503, TRL kl 0.273503
loss, ours: +64.70719147
loss, TRL 1.18.2 GRPOTrainer._compute_loss: +64.70722961
|difference|: loss 3.81e-05, per-token log-probs 1.30e-06, gradient of lm_head 8.20e-03
relative difference of the two losses: 5.9e-07
our loss on the log-probs TRL computed: +64.70722961, |difference| from TRL 0.00e+00
epsilon 0.2, beta 0.04, TRL use_bias_correction_kl=True:
tokens clipped, ours 0.0202, TRL clip_ratio/region_mean 0.0202
mean KL per token, ours 0.273503, TRL kl 9.148457
loss, ours: +64.69831085
loss, TRL 1.18.2 GRPOTrainer._compute_loss: +65.01915741
|difference|: loss 3.21e-01, per-token log-probs 1.30e-06, gradient of lm_head 1.50e+02
relative difference of the two losses: 4.9e-03
our loss on the log-probs TRL computed: +64.69834900, |difference| from TRL 3.21e-01Read the first two settings, which use the KL of Eq. 2. Our share of clipped tokens equals TRL's clip_ratio/region_mean: 0.0202 at and 0.1254 at . Our mean KL per token equals TRL's kl: 0.273503.
The two losses differ by on a loss of about 64.7, a relative difference of . Where does that come from? Not from the formula.
TRL computes the log-probabilities itself, through its own path to the output layer, and they differ from ours by at most : float32 rounding. The ratio is an exponential of a log-probability difference, and a few ratios here are huge, so a tiny rounding difference becomes a visible one. The last line of each setting settles it. Fed with the log-probabilities TRL computed, our grpo_loss gives exactly TRL's loss: the difference is 0.00e+00.
The loss of about 64.7 looks large, and it is real. A few tokens were sampled with a very low probability, and two optimiser steps made them much more likely: their ratio is huge. Where the advantage is negative, the min of Eq. 1 keeps the huge value, as in the last case of the four-token example. Those few tokens dominate the mean.
The third line is a surprise worth knowing. TRL 1.18 has a setting, use_bias_correction_kl, that is on by default. It multiplies each token's KL term by the ratio, a change that comes from the later DeepSeek-V3.2 paper. With it on, TRL's loss is 65.019 and ours is 64.698, and TRL's logged KL is 9.148457 instead of 0.273503. That is not Eq. 2 of the R1 paper, so for R1 we turn it off. A library default can quietly change the method you think you are running.
The loop
The settings of our run, as the script printed them:
config: {"model": "Qwen/Qwen2.5-0.5B-Instruct", "steps": 400, "minutes": 80.0, "questions": 4, "G": 8, "max_new_tokens": 300, "temperature": 1.0, "lr": 3e-06, "eps": 0.2, "beta": 0.04, "minibatches": 2, "micro": 8, "seed": 0}Here is the loop. It is the five steps above, in order:
def train(cfg, policy, tok, data, log, jsonl_path, samples_path):
random.seed(cfg['seed']); torch.manual_seed(cfg['seed'])
ref = copy.deepcopy(policy).eval().requires_grad_(False) # pi_ref: frozen copy of the start
opt = torch.optim.AdamW(policy.parameters(), lr=cfg['lr'], betas=(0.9, 0.99), weight_decay=0.0)
G, start, samples = cfg['G'], time.time(), []
open(jsonl_path, 'w').close()
log(f'memory at the start: {memory_gb():.1f} GB')
for step in range(1, cfg['steps'] + 1):
qs = random.sample(data, cfg['questions'])
# 1. Sample a group of G answers per question from the current policy (pi_old).
policy.eval()
batch = sample_group(policy, tok, [make_prompt(q['question']) for q in qs], G,
cfg['max_new_tokens'], cfg['temperature'])
# 2. Score every answer with the two rules, then compare it with its own group (Eq. 3).
gold = [q['gold'] for q in qs for _ in range(G)]
acc = torch.tensor([accuracy_reward(t, g) for t, g in zip(batch['texts'], gold)])
fmt = torch.tensor([format_reward(t) for t in batch['texts']])
rewards = acc + fmt # (B,)
adv = group_advantages(rewards, G).to(policy.device) # (B,)
# 3. Log-probs of pi_old and pi_ref, once, before any update of this step.
B = rewards.numel()
with torch.no_grad():
old = torch.cat([per_token_logps(policy, rows(batch, slice(i, i + cfg['micro'])), cfg['temperature'])
for i in range(0, B, cfg['micro'])])
refl = torch.cat([per_token_logps(ref, rows(batch, slice(i, i + cfg['micro'])), cfg['temperature'])
for i in range(0, B, cfg['micro'])])
# 4. Split the answers into mini-batches; one optimiser step each. From the second one on, pi_theta != pi_old.
policy.train()
order, stats = torch.randperm(B), []
for mb in order.chunk(cfg['minibatches']):
opt.zero_grad()
for idx in mb.split(cfg['micro']):
sub = rows(batch, idx.to(policy.device))
logps = per_token_logps(policy, sub, cfg['temperature'])
loss, st = grpo_loss(logps, old[idx], refl[idx], adv[idx], sub['completion_mask'],
cfg['eps'], cfg['beta'])
(loss * len(idx) / len(mb)).backward() # mean over the mini-batch
stats.append(st) # plain floats, no tensors kept
del sub, logps, loss
torch.nn.utils.clip_grad_norm_(policy.parameters(), 1.0)
opt.step()
# 5. Log the step.
# KL and clipped share: one mean over every real token of the step (each slice has a different token count).
lengths = batch['completion_mask'].sum(1).float()
tokens = sum(s['tokens'] for s in stats)
row = dict(step=step, reward=rewards.mean().item(), acc=acc.mean().item(), fmt=fmt.mean().item(),
length=lengths.mean().item(), kl=sum(s['kl_sum'] for s in stats) / tokens,
clipped=sum(s['clipped_sum'] for s in stats) / tokens,
zero_std_groups=(rewards.view(-1, G).std(1) == 0).float().mean().item(),
seconds=round(time.time() - start, 1))
texts = batch['texts']
del batch, old, refl, adv
free_memory(policy.device)
row['rss_gb'] = round(memory_gb(), 2)
with open(jsonl_path, 'a') as f:
f.write(json.dumps(row) + '\n')
log(f"step {step:4d} reward {row['reward']:.3f} acc {row['acc']:.3f} format {row['fmt']:.3f} "
f"length {row['length']:6.1f} kl {row['kl']:.5f} clipped {row['clipped']:.4f} "
f"memory {row['rss_gb']:.1f} GB {row['seconds']:.0f} s")
if step in (1, 2, 3) or step % 25 == 0:
samples.append(dict(step=step, question=qs[0]['question'], gold=qs[0]['gold'],
answers=[dict(text=t, acc=a.item(), fmt=f.item())
for t, a, f in zip(texts[:G], acc[:G], fmt[:G])]))
with open(samples_path, 'w') as f:
json.dump(samples, f, indent=1)
if time.time() - start > cfg['minutes'] * 60:
log(f'time budget of {cfg["minutes"]} minutes reached after step {step}')
break
return policyBefore the first step, copy.deepcopy makes the reference model, and requires_grad_(False) freezes it. Each step then does the following:
- Pick 4 random training questions and sample 8 answers to each with the current model (one rollout).
- Score the 32 answers with the two rules, and turn the scores into advantages, group by group.
- Compute the log-probabilities of the old policy and of the reference, once, without gradients. They do not change during the step.
- Shuffle the 32 answers into 2 mini-batches of 16. For each, compute the loss in slices of 8 to save memory and add up the gradients. Then clip their total size to 1 and take one AdamW step.
- Write one line of JSON with the mean rewards, the mean length, the KL, the share of clipped tokens and the memory the process uses. The KL and the clipped share are averaged over every real token of the step. Then free the memory the step left behind.
Every 25 steps it also saves the 8 answers to the first question, so we can read what the model writes.
The evaluation
"""Held-out check: greedy answers to GSM8K test questions, scored with the same two rules as in training."""
import torch
from .template import make_prompt, full_completion
from .rewards import accuracy_reward, format_reward
from .grpo import end_token_ids, completion_lengths
@torch.no_grad()
def evaluate(model, tok, problems, max_new_tokens=300, batch=32):
model.eval()
end_ids = end_token_ids(model, tok)
out = []
for i in range(0, len(problems), batch):
chunk = problems[i:i + batch]
enc = tok([make_prompt(p['question']) for p in chunk], return_tensors='pt', padding=True,
padding_side='left').to(model.device)
# Greedy, with no repetition penalty: the Instruct model's own generation config would add one of 1.1.
gen = model.generate(**enc, do_sample=False, repetition_penalty=1.0, max_new_tokens=max_new_tokens,
stop_strings=['</answer>'], tokenizer=tok, pad_token_id=tok.pad_token_id)
rows = gen[:, enc.input_ids.size(1):].tolist()
for p, ids, n in zip(chunk, rows, completion_lengths(rows, tok, end_ids)): # n: tokens really written
text = full_completion(tok.decode(ids[:n], skip_special_tokens=True))
out.append(dict(question=p['question'], gold=p['gold'], text=text, length=n,
acc=accuracy_reward(text, p['gold']), fmt=format_reward(text)))
n = len(out)
summary = dict(n=n, acc=sum(o['acc'] for o in out) / n, fmt=sum(o['fmt'] for o in out) / n,
length=sum(o['length'] for o in out) / n)
return summary, outThe evaluation uses the first 200 questions of the GSM8K test split. The model never trains on them. It answers each one greedily, the most likely token every time, with the same limit of 300 new tokens. We set repetition_penalty=1.0 on purpose.
The Instruct model's own generation settings add a repetition penalty of 1.1, which would quietly change "the most likely token" into "the most likely token after a penalty". Our first evaluation had that penalty; this one does not. The paper reports pass@1 from several samples at temperature 0.6. One greedy answer is cheaper, and it gives the same result every time.
The run: first try, the base model
R1-Zero starts from a base model, so we did too: Qwen2.5-0.5B, a model of 0.5 billion parameters that was never tuned to follow instructions. Before training, it answered 10% of the 200 held-out questions correctly, and 35.5% of its greedy answers passed the format parser:
BEFORE training: accuracy 0.100 format 0.355 mean length 98.9 tokens (102 s)
before: our accuracy rule and math_verify agree on 200 of 200 answers (both read the text inside the last answer tags)
before: our format parser and an independent regular expression agree on 200 of 200 answersWe gave this run 20 minutes, to keep most of our time budget for the second model. It made 35 steps. In windows of 10 steps:
== base model: 35 steps in 20 minutes
steps accuracy format length KL clipped equal-reward groups seconds per step psutil RSS GB
1- 10 0.044 0.762 47.5 2.654 0.0104 0.35 34.3 1.5
11- 20 0.050 0.894 54.2 0.209 0.0014 0.42 36.8 1.3
21- 30 0.041 0.872 43.7 0.224 0.0006 0.28 37.4 1.3
31- 35 0.031 0.944 34.1 0.241 0.0016 0.50 27.9 1.3
base model, largest KL of one step: 24.257 at step 1; median KL 0.204The format reward was 0.762 in the first 10 steps and 0.944 in the last 5. The accuracy reward stayed between 0.031 and 0.050. And the answers got much shorter, from 139.9 tokens at step 1 to 34.1 tokens on average in steps 31 to 35. The held-out test after training shows the same thing on new questions:
AFTER training: accuracy 0.040 format 0.985 mean length 37.3 tokens (51 s)
after: our accuracy rule and math_verify agree on 200 of 200 answers (both read the text inside the last answer tags)
after: our format parser and an independent regular expression agree on 200 of 200 answersFormat went from 35.5% to 98.5%, but accuracy fell from 10% to 4%, and the mean answer shrank from 98.9 to 37.3 tokens. On this model and in this short run, the rules taught the shape and nothing else.
The samples show what the model writes. Here are three answers at step 2 and three at step 25:
== base model, training step 2, gold answer 10
Q: At the burger hut, you can buy a burger for $5, french fries for $3, and a soft drink for $3. If you order a special burger meal, you get all 3 of th...
[accuracy 0, format 1] '<think> reasoning process here </think>\n<answer> Boone has eaten all but 1 of his burger meals </answer>'
[accuracy 0, format 0] '<think> Burger-Broblems </think>. <answer> Joe eats at an awesome burger shack </answer>'
[accuracy 0, format 0] "<think> calculating with burger marketing </think> <answer> (burger meals relate relationship 1 )<item> 1'* burger meals are 2 *$ 5 = $10 </item> <item> 1*\\$3 + 1*\\$3 = $6 </item> <item> 1*burger meals total = $12 </item> <think> assuming 1 * burger meals are equal to 1 * food items </think> <item> 6 * $2 = $12 </item> <item> 10 + $12 = $22 </item> <answer> savings = food items - burger meals = $12 - $10 = $2 </answe"
== base model, training step 25, gold answer 8000
Q: Mr. Wells has a garden of flowers with 50 rows. If each row has 400 flowers and Mr. Wells cuts 60% of the flowers, how many flowers are remaining in t...
[accuracy 0, format 1] "<think> First, let's figure out the total number of flowers Mr. Wells has. </think> <answer> 50 x 400 = 20,000 flowers </answer>"
[accuracy 0, format 1] '<think> Subtract 60% of total flowers </think> <answer> 40*50 - 0.6*50 = 1600 </answer>'
[accuracy 0, format 1] '<think> Call us!</think> <answer> 400 </answer>'At step 2 one answer copies "reasoning process here", the placeholder text of Table 1, into the think tags. The other two put a few unrelated words in the think tags and an answer with no number at all. By step 25 the answers are short and tidy: a sentence of thinking, then an expression or a number. The format rule cannot tell real thinking from a copied placeholder, so a short well-formed answer is the cheapest way to earn one point.
This is the problem behind "equal-reward groups" in the table. Suppose all 8 answers to a question have the right shape and all 8 are wrong. Then they all get a reward of 1, and every advantage is 0. The group gives no reward signal; only the KL term still acts on it. In steps 31 to 35, half of the groups were like that. The model had learned the shape, and it almost never solved a problem, so there was little left for the rewards to teach.
The table has one more surprise: a KL of 24.257 at step 1, against a median of 0.204. After a single optimiser step, the second mini-batch already met a few tokens whose probability had fallen far below the reference. We do not know which tokens they were. The KL was back at 0.143 by step 2.
Our plan was to try the base model first. If it earned almost no accuracy reward, we would switch to the instruction-tuned model. It earned almost none, and its held-out accuracy fell.
The run: second try, the Instruct model
Qwen2.5-0.5B-Instruct has the same size, but it was fine-tuned to follow instructions and to explain its steps. That is a long way from R1-Zero's pure start, and we say so. It is closer to DeepSeek-R1, which starts RL from a fine-tuned model (Part 5).
Before training, its greedy answers were right 16.5% of the time, and 55% had the right shape:
BEFORE training: accuracy 0.165 format 0.550 mean length 115.1 tokens (90 s)
before: our accuracy rule and math_verify agree on 200 of 200 answers (both read the text inside the last answer tags)
before: our format parser and an independent regular expression agree on 200 of 200 answersThen 122 steps of our GRPO, the whole 80-minute budget. Here are the first steps and the last steps of the log:
step 1 reward 0.500 acc 0.062 format 0.438 length 132.2 kl 1.16192 clipped 0.0395 memory 1.4 GB 43 s
step 2 reward 0.344 acc 0.062 format 0.281 length 158.5 kl 0.09710 clipped 0.0079 memory 1.4 GB 90 s
step 3 reward 0.406 acc 0.062 format 0.344 length 123.7 kl 0.13512 clipped 0.0134 memory 1.4 GB 134 s
step 4 reward 0.469 acc 0.000 format 0.469 length 183.0 kl 0.07868 clipped 0.0024 memory 1.4 GB 177 s
step 5 reward 0.625 acc 0.031 format 0.594 length 144.8 kl 0.08281 clipped 0.0026 memory 1.4 GB 223 s
step 118 reward 1.188 acc 0.281 format 0.906 length 165.5 kl 0.11107 clipped 0.0002 memory 1.1 GB 4672 s
step 119 reward 1.375 acc 0.406 format 0.969 length 141.3 kl 0.10981 clipped 0.0007 memory 1.1 GB 4705 s
step 120 reward 1.312 acc 0.344 format 0.969 length 158.0 kl 0.09827 clipped 0.0006 memory 1.1 GB 4745 s
step 121 reward 1.188 acc 0.281 format 0.906 length 158.6 kl 0.09319 clipped 0.0008 memory 1.1 GB 4787 s
step 122 reward 1.188 acc 0.344 format 0.844 length 187.2 kl 0.10839 clipped 0.0005 memory 1.1 GB 4829 sAnd the 10-step windows:
== instruct model: 122 steps in 80 minutes
steps accuracy format length KL clipped equal-reward groups seconds per step psutil RSS GB
1- 10 0.041 0.656 114.7 0.217 0.0077 0.23 40.2 1.4
11- 20 0.075 0.966 78.9 0.120 0.0021 0.42 36.4 1.4
21- 30 0.131 0.931 125.3 0.088 0.0003 0.20 40.4 1.5
31- 40 0.113 0.944 117.3 0.106 0.0007 0.42 35.7 1.5
41- 50 0.338 0.953 133.5 0.109 0.0012 0.33 39.6 1.5
51- 60 0.206 0.950 140.2 0.103 0.0006 0.33 40.3 1.4
61- 70 0.122 0.963 108.3 0.143 0.0005 0.47 41.5 1.2
71- 80 0.259 0.972 124.6 0.121 0.0006 0.28 38.1 1.1
81- 90 0.297 0.959 148.0 0.122 0.0006 0.28 41.6 1.1
91-100 0.303 0.953 138.0 0.137 0.0008 0.12 42.0 1.1
101-110 0.250 0.978 143.4 0.122 0.0002 0.30 40.7 1.2
111-120 0.359 0.959 147.1 0.111 0.0004 0.12 38.1 1.2
121-122 0.312 0.875 172.9 0.101 0.0006 0.00 42.0 1.1Three things happened. The format was learned first: the format reward was 0.656 in the first 10 steps and 0.966 in the next 10. The accuracy reward rose more slowly. It was 0.041 in the first 10 steps, 0.338 in steps 41 to 50, and 0.359 in steps 111 to 120. On the way it dipped as low as 0.122.
With only 4 questions per step, one step can show 0.000 and two steps later 0.250, so read the trend, not single steps. And the answers did not collapse. They dipped to about 79 tokens on average in steps 11 to 20, then grew back to about 140.
The held-out test confirms the trend on questions the model never trained on:
AFTER training: accuracy 0.390 format 0.980 mean length 159.8 tokens (92 s)
after: our accuracy rule and math_verify agree on 200 of 200 answers (both read the text inside the last answer tags)
after: our format parser and an independent regular expression agree on 200 of 200 answersAccuracy went from 16.5% to 39.0%: 33 right answers before, 78 after. 196 of the 200 answers now have the right shape. On 200 questions a gap of 22.5 points is far larger than the noise of a single evaluation. But it is one run with one seed, so we do not claim a precise size.
Here are four held-out questions, before and after. The text is cut at 500 characters:
== held-out question 0, gold answer 18
Q: Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the rem...
before [accuracy 1, format 1, 128 tokens] "<think> First, we need to calculate the total number of eggs laid by the ducks per day. Janet's ducks lay 16 eggs per day. She eats 3 eggs for breakfast every morning, and she bakes muffins for her friends every day, which means she eats 3 + 4 = 7 eggs for her friends. So, the total number of eggs laid per day is 16 - 7 = 9 eggs. </think> <answer> Janet makes $2 per fresh duck egg, so she makes 9 x $2 = $18 every day at the farmers' market. </answer>"
after [accuracy 0, format 1, 183 tokens] "<think> First, we need to calculate the total number of eggs laid by Janet's ducks per day. Since Janet's ducks lay 16 eggs per day, and she eats 3 eggs for breakfast every morning, she will eat 3 eggs every day. She also bakes 4 muffins every day, which means she will bake 4 eggs every day. So, the total number of eggs laid by her ducks every day is 3 + 4 = 7 eggs. \n\nNext, we need to calculate how much money Janet makes every day from selling the eggs. Since she sells the eggs at $2 per fresh d"
== held-out question 1, gold answer 3
Q: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?
before [accuracy 1, format 1, 83 tokens] '<think> To find the total number of bolts needed, we need to add the number of blue and white bolts together. We know that the number of blue bolts is 2 and the number of white bolts is half of that, which is 2/2 = 1. So, the total number of bolts needed is 2 + 1 = 3. </think> <answer> 3 </answer>'
after [accuracy 1, format 1, 76 tokens] '<think> First, we need to identify the amount of blue and white fibers that each robe takes. According to the question, a robe takes 2 bolts of blue fiber and half that much white fiber. So, each robe takes 2 blue fibers and 2/2 = 1 white fiber. </think> <answer> 2 + 1 = 3 </answer>'
== held-out question 2, gold answer 70000
Q: Josh decides to try flipping a house. He buys a house for $80,000 and then puts in $50,000 in repairs. This increased the value of the house by 150%...
before [accuracy 0, format 1, 36 tokens] '<think> To calculate the profit, we need to subtract the initial cost from the final value of the house. </think> <answer> $30,000 </answer>'
after [accuracy 1, format 1, 234 tokens] '<think> First, we need to calculate the total cost of the house. Josh spent $80,000 on the house and $50,000 on repairs, so the total cost was $80,000 + $50,000 = $130,000. Next, we need to calculate the increase in the value of the house. The increase in value was 150% of the original value, which means the increase was $80,000 * 1.5 = $120,000. The increase in value is $120,000, so the total increase in value is $120,000 + $80,000 = $200,000. The profit is the increase in value minus the cost '
== held-out question 3, gold answer 540
Q: James decides to run 3 sprints 3 times a week. He runs 60 meters each sprint. How many total meters does he run a week?
before [accuracy 0, format 1, 76 tokens] '<think> To calculate the total distance James runs in a week, we need to multiply the number of sprints he runs each week by the distance he runs each sprint. So, 3 sprints * 60 meters per sprint = 180 meters per week. </think> <answer> James runs a total of 180 meters a week. </answer>'
after [accuracy 0, format 1, 63 tokens] '<think> First, we need to calculate the total distance James runs each sprint. Since he runs 60 meters each sprint and he runs 3 times a week, he runs 60 * 3 = 180 meters each week. </think> <answer> 180 meters </answer>'Training did not fix every answer, and it broke some. Question 2 went from wrong to right: the trained model now works out the cost, the new value and the profit step by step. Question 0 went the other way: right before, wrong after. Question 3 is wrong both times, with the same mistake (it multiplies only two of the three numbers). A held-out accuracy is a count of such wins and losses, and on these 200 questions the wins were many more.
The KL stayed small
The windows table shows a mean KL between 0.088 and 0.217 in every window. The largest KL of a single step was at step 1:
instruct model, largest KL of one step: 1.162 at step 1; median KL 0.112Eq. 2 shows how one step could jump much higher. For one token, the term is with . If the model has made a token very unlikely that the reference still likes, is huge, and the term grows like itself. A token with 100 times smaller than costs . A few such tokens are enough to lift the mean of a whole step.
Our first Instruct run had such a step, with a mean KL far above every other step. The fixed Instruct run has none, but the fixed base run has one at step 1, as we saw above. So such spikes are not only a product of the old padding bug, and we did not find their cause.
What did not happen: no Figure 3
Our answers did get somewhat longer. On the held-out questions the mean went from 115 tokens before training to 160 after. During training, the 10-step mean was 115 in the first window. After a dip in steps 11 to 20, it stayed between 138 and 148 from step 81 to step 120.
That is a rise of about 40%, not the twenty-fold growth of Figure 3. We cannot tell from one run why it happened. Longer answers may simply carry more of the arithmetic written out.
Our setting is far from R1-Zero's in every way the box above lists, so we did not expect a Figure 3. Our limit is 300 new tokens, about 3% of R1-Zero's final mean length, so a long answer is cut off and earns nothing. We ran 122 steps, not 10,400. And GSM8K problems need only a few arithmetic steps. These are differences, not a tested explanation: we did not run the experiment that would separate them.
We did not see an aha moment either. The 0.5B model writes simple, linear arithmetic. It does not stop to say "Wait" and check itself. We did reproduce the first part of the story. Rules alone, with no example of good reasoning, taught the model the format in about 15 steps. They also raised its accuracy on new questions.
What the laptop could and could not do
The run used an Apple laptop with 64 GB of memory, through PyTorch's mps device. The policy, the frozen reference and the optimiser state are all full float32 copies of the model.
Our first run did not get far. After about 30 steps its steps slowed down sharply, because the process kept growing until the machine started to swap. Every step writes answers of a different length, so the mps allocator keeps freed blocks of many sizes and the cache grows.
The fix is two lines in free_memory: drop the step's tensors, collect garbage and call torch.mps.empty_cache() after every step. The resident size that psutil reports does not see mps buffers: it stayed near 1.5 GB. So a small script, r1_part8_memwatch.py, read the process footprint once a minute. That is the number Activity Monitor shows:
minute 0.0 after step 2 footprint 30.0 GB
minute 1.0 after step 4 footprint 22.0 GB
minute 2.0 after step 5 footprint 27.0 GB
minute 40.1 after step 64 footprint 27.0 GB
minute 77.1 after step 120 footprint 26.0 GB
minute 78.1 after step 121 footprint 29.0 GB
minute 79.1 after step 122 footprint 29.0 GBOver the 80 readings of the fixed run, the footprint stayed between 20 and 34 GB, with no upward trend. It moves from reading to reading because a reading can land anywhere inside a step. The logits of 8 answers over a vocabulary of 151,936 tokens take a lot of memory while they exist. The step time stayed flat too, as the seconds-per-step column of the windows table shows. So the run used its whole 80 minutes for 122 steps instead of stalling.
Generation is the slow part of RL. Each of our rollouts writes 32 answers of up to 300 tokens before the model can learn anything, and then takes 2 optimiser steps. In the later paper, each R1-Zero rollout wrote 8,192 answers of up to 32,768 tokens (65,536 after step 8,200). Then it took 16 optimiser steps.
Compare the most tokens one rollout may write: against , almost 28,000 times more. That is a ratio of maximum rollout budgets, not of tokens actually written per optimiser step. Our run had 122 rollouts and 244 optimiser steps. The method is the same; the scale is not.
Method 2: the same recipe with TRL
TRL's GRPOTrainer does every step of our loop: sampling, rewards, advantages, the loss and the optimiser. We give it the same template and the same two rules. The learning rate, , and G are the same too. We run it for 4 steps only, to show the route. It is a demonstration, not a copy of our run. Each TRL step here uses 2 questions, not 4, and takes one optimiser step, not two.
def accuracy(completions, gold, **kw): # TRL passes the dataset columns (here: gold) as keyword arguments
return [accuracy_reward(full_completion(c), g) for c, g in zip(completions, gold)] # put the prefilled tag back
def fmt(completions, **kw):
return [format_reward(full_completion(c)) for c in completions]
with common.heavy('part8'):
from datasets import Dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import GRPOConfig, GRPOTrainer
name = 'Qwen/Qwen2.5-0.5B-Instruct'
train, _ = load_gsm8k()
ds = Dataset.from_list([dict(prompt=make_prompt(q['question']), gold=q['gold']) for q in train[:64]])
args = GRPOConfig(
output_dir=tempfile.mkdtemp(), max_steps=4, learning_rate=3e-6, lr_scheduler_type='constant', loss_type='grpo', beta=0.04, epsilon=0.2,
use_bias_correction_kl=False, # the KL of R1's Eq. 2, as in our package
num_generations=8, per_device_train_batch_size=16, # 16 completions per step = 2 questions x G = 8
max_completion_length=300, temperature=1.0, top_p=1.0, top_k=0, repetition_penalty=1.0,
generation_kwargs=dict(stop_strings=['</answer>']), # stop where our sampler stops
logging_steps=1, report_to='none',
bf16=False, save_strategy='no', seed=0)
model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float32)
tok = AutoTokenizer.from_pretrained(name)
model.generate = functools.partial(model.generate, tokenizer=tok) # generate() needs the tokenizer for a stop string
trainer = GRPOTrainer(model=model, reward_funcs=[accuracy, fmt], args=args, train_dataset=ds,
processing_class=tok)
t = common.timer()
trainer.train()TRL passes the extra columns of the dataset, here gold, to every reward function as keyword arguments. The reward functions put the prefilled <think> back in front, as sample_group does.
Three settings matter. lr_scheduler_type='constant' keeps the learning rate fixed, as in our loop. use_bias_correction_kl=False gives the KL of Eq. 2, as the loss check above showed. And generation must stop at </answer>, as ours does. TRL passes generation_kwargs to transformers, but a stop string also needs the tokenizer, which TRL does not pass. So we bind the tokenizer to model.generate with functools.partial.
Without the stop string, TRL would keep sampling after </answer>. Those extra tokens would enter its loss, its KL and its lengths, while the rewards might see only the cut text. That is a different objective.
With the stop string, one small difference is left. After a stop on </answer>, TRL's mask keeps the first padding token. For Qwen, padding is the end-of-text token, and TRL reads it as a sampled end. Our completion_lengths leaves it out. So TRL's completions/mean_length counts one token more for each answer that stopped on </answer>.
TRL logs the same quantities as our loop:
step 1: reward 0.8125 rewards/accuracy/mean 0.0625 rewards/fmt/mean 0.75 completions/mean_length 109.7 kl -8.389e-09 clip_ratio/region_mean 0 frac_reward_zero_std 0
step 2: reward 0.6875 rewards/accuracy/mean 0.0625 rewards/fmt/mean 0.625 completions/mean_length 175.8 kl 0.01768 clip_ratio/region_mean 0 frac_reward_zero_std 0
step 3: reward 1 rewards/accuracy/mean 0.0625 rewards/fmt/mean 0.9375 completions/mean_length 138.1 kl 0.05312 clip_ratio/region_mean 0 frac_reward_zero_std 0
step 4: reward 1.438 rewards/accuracy/mean 0.5 rewards/fmt/mean 0.9375 completions/mean_length 142.3 kl 0.05238 clip_ratio/region_mean 0 frac_reward_zero_std 0
4 steps in 96 s on mpsFour steps of 2 questions prove nothing about learning, and we do not read the rewards as a trend. One column is worth a look. clip_ratio/region_mean is 0 at every step. Here TRL takes one optimiser step per batch of answers, so the ratio is exactly 1, as Step 4 predicts.
The library is the right tool for real work: it adds vLLM for fast sampling, many GPUs and many loss variants. Our package is the right tool to see what happens inside. The check above shows that, on the same tensors, they compute the same loss.
Run it yourself
The code needs Python, torch, transformers and datasets, plus trl and math-verify for the checks. The training run needs a machine with about 64 GB of memory, or a GPU.
git clone https://github.com/ishwar6/ishwar-books
cd ishwar-books/code/papers/deepseek-r1
python r1_part8_probe.py prefill # do the models earn any reward before training?
python r1_part8_check.py # the equations on small numbers, and our loss against TRL
python r1_part8_train.py Qwen/Qwen2.5-0.5B-Instruct 400 80 # before eval, 80 minutes of GRPO, after eval
python r1_part8_train.py Qwen/Qwen2.5-0.5B 400 20 # the base model, 20 minutes
python r1_part8_memwatch.py <pid> instruct # optional, in a second terminal: memory once a minute
python r1_part8_trl.py # Method 2: 4 steps of TRL's GRPOTrainer
python r1_part8_stats.py # the 10-step windows of the logsSome good next experiments:
- Give the format reward a weight of 0.5, so a right answer counts more than a tidy one. Does the base model learn more?
- Raise the limit to 1,000 tokens and train on harder questions, such as MATH, to give length a reason to grow.
- Set
minibatchesto 1 and check that the share of clipped tokens drops to 0, as Step 4 predicts.
The paper and the code
Paper (arXiv 2501.12948) · Official repository · Our scripts and the minir1 package

