DeepSeek-R1, explained · Part 3 of 8 · Covers §2.2.2, §2.2.3, Table 1
Rewards and the Template
How R1-Zero scores its own answers with two simple rules, why the authors refused a learned reward model, and the one-paragraph prompt that starts training. We write each reward two ways, game a weak reward on purpose, and test two base models in the real template.
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and others. arXiv 2025 (Nature 2025), 2025. arXiv:2501.12948
Reinforcement learning only sees one number per answer, so whoever writes that number decides what the model becomes. R1-Zero's number comes from two short rules and one paragraph of prompt, and this part takes both apart line by line.
Reward modeling
Background: the reward model of RLHF
To see why "rule-based" is a choice, look at what came before. The assistants of 2022 to 2024 were trained with reinforcement learning from human feedback (RLHF). There is no answer key for "write a kind email", so people compare two answers and pick the better one. A second network, the reward model, learns from many such comparisons. Then RL trains the language model to get high scores from the reward model.
The reward model learns with one equation. Give it a prompt , the answer people chose and the answer they rejected . The network gives each answer a score, and the loss pushes the chosen score above the rejected one:
Read it from the inside. is the score that the network with weights gives answer . The difference is the margin between the chosen and the rejected answer. The sigmoid turns the margin into a probability that a person prefers . The minus log makes the loss small when that probability is near 1.
A tiny example. The network scores the chosen answer 1.2 and the rejected one 0.4. The margin is 0.8, and . The loss is . If the scores were the other way round, the loss would be 1.171, and training would move the weights to fix the order. Our script checks this by hand and with PyTorch's logsigmoid:
== 0. reward model loss for one human comparison (Bradley-Terry)
r(chosen)=+1.2 r(rejected)=+0.4 P(chosen preferred)=0.690 loss ours=0.371 torch=0.371
r(chosen)=+0.4 r(rejected)=+1.2 P(chosen preferred)=0.310 loss ours=1.171 torch=1.171
r(chosen)=+3.0 r(rejected)=-3.0 P(chosen preferred)=0.998 loss ours=0.002 torch=0.002Note the last sentence of that box. The reward model is a stand-in. RL then pushes hard on that stand-in for thousands of steps. That is where the trouble starts, as the section on reward hacking shows.
DeepSeek's own earlier work used this kind of network for maths too. GRPO comes from the DeepSeekMath paper, and there the score came from a trained reward model.
Accuracy rewards
As an equation, the accuracy reward of an output for a question with the right answer is
Read it from the inside. The function takes the text inside the last \boxed{...} of the output. The function rewrites an answer into one standard spelling, so that and look the same. The reward is 1 when the two spellings match and 0 otherwise. There is no partial credit and no score for nice working.
The equation hides two hard parts, and both are about strings. First, find the box. Second, decide when two different strings are the same number.
Method 1: our own maths checker
Finding the box looks like a one-line regular expression. It is not. LaTeX answers contain their own braces, as in \boxed{\frac{\sqrt{3}}{2}}. A pattern that stops at the first } cuts the answer in half. Our function counts braces instead, like a person checking brackets in a long formula:
def extract_boxed(text):
"""Return what is inside the LAST \\boxed{...} of text, or None. Counts braces, so nested braces work:
'\\boxed{\\frac{\\sqrt{3}}{2}}' -> '\\frac{\\sqrt{3}}{2}'."""
start = text.rfind('\\boxed') # the last box is the final answer
if start < 0:
return None
i = text.find('{', start) # the opening brace of the box
if i < 0:
return None
depth = 0
for j in range(i, len(text)):
if text[j] == '{':
depth += 1 # one level deeper
elif text[j] == '}':
depth -= 1 # one level up
if depth == 0: # back to the level of the box: this brace closes it
return text[i + 1:j]
return None # the box never closes: no answer
def naive_boxed(text):
"""The one-line regex many people write first. It stops at the FIRST closing brace."""
m = re.findall(r'\\boxed\{(.*?)\}', text)
return m[-1] if m else NoneHere are both on four outputs:
== 1a. extract the last box: a naive regex vs our brace counter
'so the answer is \\boxed{18}.' naive regex: '18' ours: '18'
'Thus $\\boxed{\\frac{\\sqrt{3}}{2}}$.' naive regex: '\\frac{\\sqrt{3' ours: '\\frac{\\sqrt{3}}{2}'
'First \\boxed{4}, no wait: \\boxed{\\dfrac{3}{4}}' naive regex: '\\dfrac{3' ours: '\\dfrac{3}{4}'
'The answer is \\boxed{\\frac{1}{2}' naive regex: '\\frac{1' ours: NoneThe regex fails on every answer with a fraction. In RL that bug is not a small error. Every right fraction would get reward 0, and the model would learn to avoid fractions. The third line shows why we take the last box: the model changed its mind, and the last box holds its final answer. The fourth line has a box that never closes, so our checker gives no answer at all.
Next, the spelling. The function normalise removes one harmless difference per line. Then to_number reads the result as an exact fraction when it can, so and become the same Fraction(1, 2):
def normalise(s):
"""Turn an answer string into one standard spelling. Each line removes one harmless difference."""
s = s.strip()
s = re.sub(r'\\text\{\s*([^{}]*?)\s*\}', r' \1', s) # \text{ cm} -> ' cm' (keep the words, drop the macro)
s = s.replace('\\dfrac', '\\frac').replace('\\tfrac', '\\frac') # display and text fractions are fractions
s = s.replace('\\left', '').replace('\\right', '') # bracket sizes do not change the value
s = s.replace('\\