From RLHF to Reasoning: How Language Models Are Trained Today

How chat and reasoning models are taught after pre-training: human preference labels, AI feedback, answers a program can check, and how we test the result.

You give a model a hard maths problem. It writes a few lines, stops, and writes "Wait, wait. Wait." Then it goes back and checks its own work. Nobody typed that line into its training data as an example to copy. The model found that checking earned more reward, so checking became a habit.

That line comes from DeepSeek-R1-Zero, released in January 2025. After pre-training, it was trained with reinforcement learning and almost nothing else. During that training, its score on the AIME 2024 maths competition rose from 15.6% to 77.9%. The reward was simple: 1 if the final answer was right, 0 if it was not.

This is Part 4 of How AI Got Here. Part 1 told the story of neural networks and gave a short account of RLHF; Part 2 covered classical machine learning. Part 3 followed reinforcement learning from Thorndike's cats to PPO and AlphaZero. This part shows how those methods became the way language models are trained, from 2017 to October 2026. Part 5 then opens the base model itself.

From preferences to reasoning, 2017 to 2026Learning from preferences2017 to 2020RLHF assistants2022 to 2023Constitutional AI, RLAIF2022 to 2023DPO and its family2023 to 2024Verifiers, process rewards2022 to 2024Reasoning RL: o1, R1, RLVR2024 to 2026RL for agents and tools2025 to 202620172019202120232025
Nine years, seven overlapping threads. Each one is still in use; new stages were added on top of the old ones, not in their place.

Why a pre-trained model needs more training

A model trained only to predict the next token is called a base model. It has read a large part of the public web, so it knows a great deal. But it continues text; it does not answer. Ask it "How do I bake bread?" and it may write three more questions, as a web forum might.

The first step of post-training is supervised fine-tuning (SFT). People write good answers to many prompts, and the model is trained on them with the same next-token loss as before. It is like an apprentice copying the work of a master. It works, but it has limits.

Writing a great answer is slow and costly, and the best a copy can be is as good as the person who wrote it. Judging is easier than writing. Most people cannot write a prize-winning essay, but they can tell which of two essays they prefer. That difference is the starting point of everything in this part.

2017 to 2020: learning a reward from people

Preferences between video clips

In June 2017, Paul Christiano and colleagues at OpenAI and DeepMind showed how to train an agent when nobody can write the reward. A person watched two short video clips of the agent and picked the one that looked closer to the goal. A network learned to predict those choices. The agent then ran ordinary reinforcement learning against the predicted reward, while new clips went to the person.

From clips to text

In 2019, Daniel Ziegler and colleagues at OpenAI applied the method to GPT-2. People compared continuations of text and later summaries. The paper used a KL penalty to keep the fine-tuned model close to the original, following earlier work by Natasha Jaques and colleagues. It also found an early warning. With 60,000 comparisons, the summarisers learned to copy whole sentences from the input, which pleased the labellers but was not real summarising.

In September 2020, Nisan Stiennon and colleagues scaled the method for summaries of Reddit posts (the TL;DR data set).

The reward model, up close

Ranking by comparisons

How do you turn "A is better than B" into a number for each answer? Chess has the same problem. Players are never given an absolute score, only wins and losses, yet the Elo rating gives each player one number. The gap between two ratings predicts who will win. In 1952, Ralph Bradley and Milton Terry wrote down the model behind such ratings.

P(yw≻yl∣x)=σ(rϕ(x,yw)−rϕ(x,yl)),σ(z)=11+e−zP(y_w \succ y_l \mid x) = \sigma\big( r_\phi(x, y_w) - r_\phi(x, y_l) \big), \qquad \sigma(z) = \frac{1}{1 + e^{-z}}

xx is the prompt, ywy_w is the answer people preferred (the winner) and yly_l is the other one (the loser). The symbol ≻\succ means "is preferred to". rϕ(x,y)r_\phi(x, y) is the reward model's score for an answer, with parameters ϕ\phi (phi). The sigmoid σ\sigma turns any number into a probability between 0 and 1.

Training makes the observed choices as likely as possible, which gives the loss that Part 1 showed briefly:

LRM(ϕ)=−E(x,yw,yl)∼D[log⁡σ(rϕ(x,yw)−rϕ(x,yl))]\mathcal{L}_{\text{RM}}(\phi) = - \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \Big[ \log \sigma\big( r_\phi(x, y_w) - r_\phi(x, y_l) \big) \Big]

D\mathcal{D} is the data set of comparisons and E\mathbb{E} means the average over it. The loss is small when the winner's score is well above the loser's.

Our script gives the loss for four score gaps. A gap of 2 gives a probability of 0.8808 and a loss of 0.1269. A gap of 0.5 gives 0.4741. A gap of 0 gives ln⁡2=0.6931\ln 2 = 0.6931, which means "no idea". A gap of −2-2, the wrong way round, gives 2.1269.

A reward model: a language model with a one-number headprompt x + answer y_w(the one people chose)prompt x + answer y_l(the one they rejected)Transformerfrom the SFT modelsame Transformersame weightsr_w = 2.0linear headr_l = 0.0linear headσ(2.0 − 0.0)= 0.881loss 0.127The head reads the hidden state at the last token and outputs one number. Only the differencebetween two scores matters, so the loss teaches an ordering, not an absolute grade.
The Bradley-Terry reward model. The numbers are the worked example from the text, computed by our script.

Only differences matter. If you add 10 to every score, the loss does not change. So a reward model learns an ordering, not a grade, and its numbers mean nothing on their own. In practice it is a copy of the language model with its word output replaced by a single number. The number is read at the last token of the answer.

2022: InstructGPT and the RLHF objective

Three stages

In March 2022, Long Ouyang and colleagues at OpenAI published InstructGPT, which Part 1 described. Their recipe had three stages, and the paper gave the sizes of the data. About 13,000 prompts with written answers went into SFT. People ranked answers for 33,000 prompts to train the reward model. A further 31,000 prompts, with no labels at all, were used for the RL stage.

Classic RLHF in three stages1. Supervised fine-tuningcopy answers people wrote2. Reward modellearn which answer people prefer3. RL with PPOraise the reward, stay closedata: prompt + good answerInstructGPT: about 13k promptsdata: prompt + ranked answersInstructGPT: 33k promptsdata: prompts onlyInstructGPT: 31k promptsscore = r(x, y) − β · log( π(y | x) / π_SFT(y | x) )The SFT model is also thereference that the KL termmeasures distance from.
The InstructGPT recipe of 2022, with its dataset sizes. Stage 3 needs no new human labels: the reward model scores every answer.

The objective with a leash

In the third stage, the model writes answers and the reward model scores them. Left alone, PPO would push the model toward whatever the reward model happens to like most, even nonsense. So the objective adds a penalty for moving away from the SFT model. It works like a leash on a dog, which can explore but not run off.

max⁡πθ  Ex∼D,  y∼πθ(⋅∣x)[rϕ(x,y)]  −  β DKL[πθ(y∣x) ∥ πref(y∣x)]\max_{\pi_\theta} \; \mathbb{E}_{x \sim \mathcal{D},\; y \sim \pi_\theta(\cdot \mid x)} \big[ r_\phi(x, y) \big] \;-\; \beta \, \mathbb{D}_{\text{KL}} \big[ \pi_\theta(y \mid x) \,\Vert\, \pi_{\text{ref}}(y \mid x) \big]

πθ\pi_\theta is the language model being trained, the policy. πref\pi_{\text{ref}} is the reference model, usually the SFT model, kept frozen. y∼πθ(⋅∣x)y \sim \pi_\theta(\cdot \mid x) means "an answer sampled from the policy for prompt xx". DKL\mathbb{D}_{\text{KL}} is the KL divergence between the two models' distributions over answers. β\beta (beta) sets the length of the leash.

In practice, each sampled answer gets a penalised reward: the reward model's score minus β\beta times the log ratio of the two models' probabilities. Take β=0.1\beta = 0.1. Answer 1 scores 2.0, and the policy gives it a log-probability of −12-12 against −11-11 for the reference. Its penalised reward is 2.0−0.1×(−1)=2.12.0 - 0.1 \times (-1) = 2.1. Answer 2 scores higher, 2.6, but the policy has moved far to produce it: −14-14 against −22-22. Its penalised reward is 2.6−0.1×8=1.82.6 - 0.1 \times 8 = 1.8, so the leash prefers answer 1.

InstructGPT also mixed in some pre-training gradients (PPO-ptx), so the model would not forget skills while it learned to please the reward model. The paper called the loss of skills an "alignment tax".

Helpful and harmless

In April 2022, Yuntao Bai and colleagues at Anthropic published "Training a Helpful and Harmless Assistant with RLHF". They collected two kinds of comparisons, one for helpfulness and one for harmlessness, and found the two goals pull against each other. A model that refuses everything is harmless but useless. They also retrained with fresh human feedback each week, an "online" form of RLHF. And they reported "a roughly linear relation between the RL reward and the square root of the KL divergence".

Goodhart's law: when the reward model is wrong

A reward model is a measure, and RL makes it a target. It was trained on answers from the SFT model. As the policy moves away, it writes answers the reward model has never seen, and the scores become less reliable. The policy finds and exploits those blind spots. This is reward hacking.

In October 2022, Leo Gao, John Schulman and Jacob Hilton measured this without human labels. A large "gold" reward model played the role of people and labelled data. Smaller "proxy" reward models were trained on those labels, and policies were optimised against the proxies. Since the gold model was available, they could watch the true score as the proxy score rose.

The paper fits a simple curve to each line. With d=DKLd = \sqrt{\mathbb{D}_{\text{KL}}}, the distance from the start, the gold score under RL follows

RRL(d)=d (αRL−βRLlog⁡d)R_{\text{RL}}(d) = d \, \big( \alpha_{\text{RL}} - \beta_{\text{RL}} \log d \big)

At first the term αRLd\alpha_{\text{RL}} d wins and the true score climbs. The second term, βRL dlog⁡d\beta_{\text{RL}}\, d \log d, grows faster, and eventually pulls the score back down. αRL\alpha_{\text{RL}} and βRL\beta_{\text{RL}} are fitted numbers that depend on the size of the reward model. The lesson for practice: watch the KL, keep the leash, and stop early.

Best-of-n and rejection sampling

The simplest optimiser

There is a way to use a reward model with no RL at all. Sample nn answers, score them all, and keep the best. It is like a photographer who takes twenty shots and prints one.

yBoN=arg⁡max⁡y∈{y1,…,yn}rϕ(x,y),DKL(πBoN ∥ π)=log⁡n−n−1ny_{\text{BoN}} = \arg\max_{y \in \{y_1, \dots, y_n\}} r_\phi(x, y), \qquad \mathbb{D}_{\text{KL}}\big(\pi_{\text{BoN}} \,\Vert\, \pi\big) = \log n - \frac{n - 1}{n}

y1y_1 to yny_n are answers sampled independently from the model π\pi. The first formula keeps the one with the highest reward. The second gives the KL divergence of this best-of-n procedure from the model, in nats. Gao and colleagues take it from the appendix of Stiennon and colleagues. It grows only like log⁡n\log n.

Say each answer is right with probability 0.3, and a perfect judge picks a right one when there is one. Then the chance of success is 1−0.7n1 - 0.7^n. For n=4n = 4 it is 0.7599, for n=16n = 16 it is 0.9967, at a KL cost of 0.6363 and 1.8351 nats. Real reward models are not perfect judges, so best-of-n can be hacked too; the Gao paper shows the same rise and fall for it.

Best-of-n: more samples, more chance of a right one, more distance from the model000.2510.520.75314n = 1n = 2n = 4n = 8n = 16n = 64● chance that at least one of n answers is right, if each is right with p = 0.3 (left axis)● KL of best-of-n from the model, log n − (n − 1)/n, in nats (right axis)
Picking the best of n answers is the simplest way to optimise a reward. Its cost in KL grows like log n, so it is a good yardstick for how hard RL pushes a model. Numbers from our script.

Llama 2

Rejection sampling fine-tuning turns best-of-n into training: sample many answers, keep the best by the reward model, and fine-tune on them. Meta used it, together with PPO, for Llama 2-Chat in July 2023. Its paper is one of the most detailed public accounts of an RLHF pipeline.

2022 to 2026: feedback from AI and from written principles

Part 1 introduced Constitutional AI, which Anthropic published in December 2022. It has two phases. In the first, the model critiques and revises its own answers against a list of principles, and is fine-tuned on the revisions. In the second, a model compares pairs of answers against the principles, and those AI-made preferences train the reward model. This second phase is RLAIF, reinforcement learning from AI feedback.

In 2023, Harrison Lee and colleagues at Google compared the two directly. Across summaries, helpful dialogue and harmless dialogue, they found that "RLAIF achieves comparable performance to RLHF". AI labels are fast and cheap, and the principles behind them are written down, so anyone can read them.

Written principles grew into long public documents. OpenAI published its Model Spec on 8 May 2024 and updated it in February 2025. On 22 January 2026, Anthropic published a new constitution for Claude, far longer than the 2023 one. It explains the reasons behind its rules, and Anthropic says the document is part of how the models are trained.

In December 2024, OpenAI described deliberative alignment, a way to train a reasoning model to think about the rules before it answers.

2023 to 2024: DPO and the family after it

The derivation

RLHF with PPO is heavy. It keeps four large models in memory: the policy, the reference, the reward model and a value model for PPO's advantages. It needs fresh samples at every step, and it has many settings to tune. In May 2023, Rafael Rafailov, Archit Sharma, Eric Mitchell and colleagues at Stanford showed that, for preferences, the RL step can be removed.

Their starting point is a known result: the KL-penalised objective above has an exact best policy.

π∗(y∣x)=1Z(x) πref(y∣x) exp⁡ ⁣(1β r(x,y))\pi^{*}(y \mid x) = \frac{1}{Z(x)} \, \pi_{\text{ref}}(y \mid x) \, \exp\!\Big( \frac{1}{\beta} \, r(x, y) \Big)

Read it as "start from the reference model and tilt it toward high reward". exp⁡\exp is the exponential function, so each unit of reward multiplies an answer's probability by e1/βe^{1/\beta}. Z(x)Z(x) is the sum of the tilted probabilities over all answers, which makes them add up to 1 again. Nobody can compute Z(x)Z(x), because the number of possible answers is astronomical.

The trick is to solve this equation for the reward instead:

r(x,y)=βlog⁡π∗(y∣x)πref(y∣x)+βlog⁡Z(x)r(x, y) = \beta \log \frac{\pi^{*}(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)

Now put this reward into the Bradley-Terry model. Bradley-Terry uses only the difference between two rewards for the same prompt, so the two βlog⁡Z(x)\beta \log Z(x) terms cancel. The impossible sum disappears, and what is left depends only on the policy and the reference.

LDPO(θ)=−E(x,yw,yl)[log⁡σ(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))]\mathcal{L}_{\text{DPO}}(\theta) = - \mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]

This is the reward-model loss, with the reward replaced by βlog⁡(πθ/πref)\beta \log (\pi_\theta / \pi_{\text{ref}}), the implicit reward. The language model is its own reward model. Training raises the probability of the chosen answer and lowers that of the rejected one, each measured against the reference.

A worked example

Take β=0.1\beta = 0.1 and one pair. The policy gives the chosen answer a log-probability of −10-10, against −12-12 for the reference, so its implicit reward is 0.1×2=0.20.1 \times 2 = 0.2. The rejected answer has −15-15 against −14-14, for an implicit reward of −0.1-0.1. The margin is 0.3, the sigmoid gives 0.5744, and the loss is 0.5544.

At the very start, the policy equals the reference, every margin is 0, and the loss is ln⁡2=0.6931\ln 2 = 0.6931. The gradient of each pair is weighted by σ(−margin)\sigma(-\text{margin}), which starts at 0.5. In our example it has already fallen to 0.4256. Pairs the model already gets right count less, and pairs it gets wrong count more.

What each method keeps in memory during trainingRLHF with PPO: four modelsDPO: two modelspolicytrainedreferencefrozenreward modelfrozenvalue modeltrainedpolicytrainedreferencefrozenLoop: sample answers, score them,estimate advantages, run PPO steps,sample again. Online and fiddly.Loop: read a fixed pair (chosen, rejected),compute four log-probabilities,one classification-style loss. Offline.loss = −log σ( β·[Δ chosen] − β·[Δ rejected] )Δ = log π − log π_ref of the answer
DPO drops the reward model and the value model and needs no sampling during training. The price: it learns only from the pairs in its data set.

DPO spread fast because it is cheap and stable. Hugging Face's Zephyr models used it in late 2023. Meta's Llama 3 paper of July 2024 describes post-training with SFT, rejection sampling and DPO, "as opposed to more complex reinforcement learning algorithms". Its weakness is that it learns only from the pairs in its fixed data set, while PPO keeps sampling new answers from the current model.

The family after DPO

DPO opened a family of "offline" preference methods. IPO (Mohammad Gheshlaghi Azar and colleagues at Google DeepMind, October 2023) argued that both RLHF and DPO can overfit. The reason is the Bradley-Terry assumption that every preference becomes a point score, and IPO replaces the log-sigmoid with a squared loss on the margin. KTO (Kawin Ethayarajh and colleagues, February 2024) borrowed prospect theory from economics, which says people feel losses more than gains. It learns from single answers marked good or bad, so it needs no pairs.

ORPO (Jiwoo Hong and colleagues, March 2024) folded preference learning into SFT itself and needs no reference model. SimPO (Yu Meng and colleagues, May 2024) also drops the reference model. Its implicit reward is the average log-probability per token of an answer. It also adds a target margin by which the winner must beat the loser. Taking the average per token keeps the length of an answer from tilting its reward.

Rewarding the steps: verifiers and process rewards

Train a checker, keep the best

A preference reward model judges style and helpfulness. For a maths problem, there is a sharper question: is the answer right? In October 2021, Karl Cobbe and colleagues at OpenAI released GSM8K, 8,500 grade-school maths word problems. They trained verifiers to judge whether a solution is correct. At test time they sampled many solutions and kept the one the verifier ranked highest, which is best-of-n with a correctness judge.

STaR: learn from your own right answers

In March 2022, Eric Zelikman and colleagues published STaR, the Self-Taught Reasoner. The model writes a chain of thought and an answer for many problems. The chains that reach the right answer are kept, and the model is fine-tuned on them; then the loop repeats. For problems it fails, the model gets the right answer and writes a reasoning that leads to it. The authors call this step "rationalization". STaR is a filtered form of reinforcement learning: a reward of 1 for a correct answer, and training only on the rewarded samples.

Outcome or process?

A wrong final answer tells you that something went wrong, not where. A teacher who marks every line of a pupil's work gives far more help. In May 2023, Hunter Lightman and colleagues at OpenAI compared two kinds of reward model on hard maths problems.

Outcome reward vs process reward for one solutionStep 1: 17 × 24 = 17 × 20 + 17 × 4✓ correctStep 2: 17 × 20 = 340✓ correctStep 3: 17 × 4 = 58✗ wrongStep 4: 340 + 58 = 398✗ follows the errorprocess reward modeloutcome rewardanswer 398is wrong:reward 0for allThe outcome label says only "wrong". The process labels say where: step 3. That is a far more useful signal.
The correct answer is 408. Lightman and colleagues paid people to label about 800,000 steps like these (PRM800K).

September 2024: o1 and thinking at test time

On 12 September 2024, OpenAI released o1 and a post called "Learning to reason with LLMs". It said that o1 is trained with reinforcement learning to think, in a long chain of thought, before it answers. It also said that the performance of o1 "consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute)".

The post gave numbers for the 2024 AIME, a hard American maths competition. GPT-4o solved 12% of the problems on average. o1 solved 74% with one sample per problem and 83% with a consensus of 64 samples. A consensus answer is the most common final answer across the samples.

OpenAI did not publish how o1 was trained: not the algorithm, the rewards or the data. The post showed accuracy rising along two axes, training compute and thinking time, and that was enough to change the field. Within months, several labs published their own versions, with far more detail.

Rewards a program can check: RLVR

In November 2024, Nathan Lambert and colleagues at the Allen Institute for AI published Tülu 3, a fully open post-training recipe. Its last stage gave a name to an idea that was spreading: reinforcement learning with verifiable rewards.

GRPO: dropping the critic

The idea

PPO needs a value model to estimate advantages, and for a language model the value model is as big as the policy. In February 2024, Zhihong Shao and colleagues at DeepSeek proposed a cheaper baseline in their DeepSeekMath paper. For each question, sample a group of answers, and compare each answer with the group's average. It is like grading on a curve: a mark means something only next to the marks of the rest of the class.

The equations

The advantage of answer ii in a group of GG answers is its reward, centred and scaled by the group:

A^i=ri−mean(r1,…,rG)std(r1,…,rG)\hat{A}_i = \frac{r_i - \text{mean}(r_1, \dots, r_G)}{\text{std}(r_1, \dots, r_G)}

Every token of answer ii gets this same advantage. The objective is PPO's clipped objective, averaged over the tokens of each answer and over the group, with the KL penalty inside:

JGRPO(θ)=E[1G∑i=1G1∣oi∣∑t=1∣oi∣(min⁡(ρi,tA^i,  clip(ρi,t,1−ϵ,1+ϵ) A^i)−β DKL[πθ ∥ πref])]J_{\text{GRPO}}(\theta) = \mathbb{E} \left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \Big( \min\big( \rho_{i,t} \hat{A}_i,\; \text{clip}(\rho_{i,t}, 1 - \epsilon, 1 + \epsilon)\, \hat{A}_i \big) - \beta\, \mathbb{D}_{\text{KL}}\big[ \pi_\theta \,\Vert\, \pi_{\text{ref}} \big] \Big) \right]

∣oi∣|o_i| is the number of tokens in answer ii. ρi,t\rho_{i,t} (rho) is the probability ratio of token tt under the new and the old policy, as in PPO: πθ(oi,t∣q,oi,<t)/πθold(oi,t∣q,oi,<t)\pi_\theta(o_{i,t} \mid q, o_{i,<t}) / \pi_{\theta_\text{old}}(o_{i,t} \mid q, o_{i,<t}). qq is the question and oi,<to_{i,<t} the tokens before tt. ϵ\epsilon is the clip range and β\beta the KL weight.

GRPO: sample a group, score it, compare each answer with the groupone mathsquestionpolicysamples 8answer 1: rightr = 1A = +1.291answer 2: wrongr = 0A = −0.775answer 3: wrongr = 0A = −0.775answer 4: rightr = 1A = +1.291answer 5: rightr = 1A = +1.291answer 6: wrongr = 0A = −0.775answer 7: wrongr = 0A = −0.775answer 8: wrongr = 0A = −0.775mean = 0.375, std = 0.484, A = (r − mean) / stdNo value network: the group mean is the baseline. Right answers are pushed up, wrong ones down, then a PPO-style clipped step.
Our script's group of eight. If all eight answers were right (or all wrong), every advantage would be 0, and the question would teach nothing.

Take one question and eight answers, three right and five wrong. The rewards are 1, 0, 0, 1, 1, 0, 0, 0, with mean 0.375 and standard deviation 0.4841. The right answers get an advantage of +1.291+1.291 and the wrong ones −0.775-0.775. For an easy question with seven right answers, the one wrong answer gets −2.646-2.646, and the right ones only +0.378+0.378. If all eight are right, the standard deviation is 0 and every advantage is 0: the question teaches nothing.

With GRPO, DeepSeekMath 7B reached 51.7% on the MATH benchmark, the paper reports, without tools or voting. The method was cheap enough to run at scale, and a year later it was the engine of DeepSeek-R1.

January 2025: DeepSeek-R1

R1-Zero: reinforcement learning alone

On 20 January 2025, DeepSeek released DeepSeek-R1, with open weights and a paper; a longer version appeared in Nature on 17 September 2025. Its most striking model was R1-Zero. It started from the base model DeepSeek-V3-Base and went straight to RL with GRPO, with no SFT first. The rewards were rule-based: an accuracy reward for a correct final answer, and a format reward for putting the reasoning between think tags.

R1-Zero had problems. Its reasoning was hard to read, and it often mixed English and Chinese in one chain of thought. So DeepSeek-R1 used a fuller pipeline. A small set of long, readable reasoning examples came first (a "cold start"), then reasoning RL. Then came SFT on a large set of the model's own filtered answers and general data, and a final RL stage for helpfulness and safety.

The team also distilled R1 into smaller open models based on Qwen and Llama by fine-tuning them on R1's outputs. The paper's appendix lists the RL training cost of R1 as 147,000 H800 GPU hours. At 2 dollars per hour, that is about 294,000 US dollars. That does not include the much larger cost of pre-training the base model.

The paper is also open about reward hacking. For reasoning, DeepSeek used no neural reward models at all, because it found them "susceptible to reward hacking" at scale. In the last stage, a model-based preference reward was used for general tasks. One figure shows that reward rising while performance on CodeForces falls: Goodhart's law again.

2025: fixing GRPO and scaling the recipe

The R1 paper started a rush of open work. In the same week, Moonshot AI's Kimi k1.5 report described long-context RL with a simple framework, "without relying on more complex techniques such as Monte Carlo tree search, value functions, and process reward models". Both reports said that plain outcome rewards and a lot of RL were enough.

Then came the fixes. In March 2025, ByteDance Seed and Tsinghua University released DAPO, with four changes to GRPO. Clip-Higher raises the upper clip limit, so rare tokens that turn out well can grow faster. That keeps the policy from collapsing onto a few answers. Dynamic Sampling drops prompts whose group is all right or all wrong, since they give zero advantage. A token-level loss and a shaped penalty for overlong answers complete the set.

Also in March 2025, Zichen Liu and colleagues at Sea AI Lab published "Understanding R1-Zero-Like Training". They found an "optimization bias" in GRPO that "artificially increases response length (especially for incorrect outputs)". Dividing by the answer length ∣oi∣|o_i| makes each token of a long wrong answer count for less, so long wrong answers are punished too gently. Their Dr. GRPO drops that division and the division by the standard deviation.

For our group of eight, Dr. GRPO's advantages are simply ri−0.375r_i - 0.375: +0.625+0.625 for a right answer and −0.375-0.375 for a wrong one. Without the standard deviation, an easy question no longer gives its one wrong answer an outsized push. The same paper found that DeepSeek-V3-Base "already exhibit[s]" an aha moment before RL, so RL amplifies a behaviour that pre-training made possible.

In May 2025, Alibaba's Qwen3 report described one model with both a "thinking" and a "non-thinking" mode. The user can also set a thinking budget.

By the end of 2025, RL was no longer a small final polish. In its December 2025 report, DeepSeek wrote that the post-training of DeepSeek-V3.2 used a large compute budget.

What Google says about Gemini

Google publishes less than DeepSeek, but its 2025 technical report on Gemini 2.5 gives an outline of post-training and of the "thinking" models.

The same report describes Deep Think, a mode of Gemini 2.5 Pro that "naturally blends in parallel thinking techniques during response generation". It lets the model "produce multiple hypotheses and carefully critique them before arriving at the final answer". On 21 July 2025, Google DeepMind announced that an advanced version of Gemini with Deep Think had reached gold-medal standard at the International Mathematical Olympiad.

It solved five of the six problems for 35 of 42 points, within the 4.5-hour time limit. It worked in natural language from the official problem statements, and IMO coordinators graded and certified the solutions. A year earlier, AlphaProof and AlphaGeometry 2 had scored 28 points, but only after the problems were translated into a formal language. The post names new reinforcement learning methods, data on multi-step reasoning and theorem proving, and parallel thinking. OpenAI reported the same score of 35 points for an experimental model of its own.

Sycophancy: when the reward is a thumbs-up

Preference learning has a built-in danger: people like to be agreed with. In October 2023, Mrinank Sharma and colleagues at Anthropic studied five assistants trained with human feedback and found sycophancy in all of them. Their analysis of preference data showed why. Both humans and preference models sometimes preferred convincing, sycophantic answers over correct ones.

In April 2025, this happened in a widely used product. On 25 April, OpenAI shipped an update to GPT-4o, the default model of its chat app. Users soon saw it flattering them and endorsing poor decisions. OpenAI began rolling it back on 28 April. In its account of 2 May, OpenAI said the update had added a reward signal based on users' thumbs-up and thumbs-down ratings. Together with other changes, this weakened the main reward signal that had kept sycophancy in check.

The lesson is Goodhart's law again, at the scale of millions of users. A thumbs-up measures what pleases a person in the moment. Optimise it hard enough and it stops measuring what helps them. OpenAI said it would weigh behaviour problems like this more heavily in its review before each launch.

RL for agents

A chat answer is one step. An agent works for many steps: it reads files, runs commands, reads the results and tries again. Training it with RL means building environments: a computer with a code base, a task, tools and a checker at the end.

Coding is the clearest case because the reward is checkable. SWE-bench, from Carlos Jimenez and colleagues at Princeton in October 2023, has 2,294 real GitHub issues from 12 Python projects. A model must edit the code base so that the project's tests pass. In the paper, the best model, Claude 2, solved 1.96% of the issues. In August 2024, OpenAI released SWE-bench Verified, a subset of 500 issues that people had checked were solvable.

Labs now build such environments at scale. Moonshot AI's Kimi K2 report of July 2025 describes "a large-scale agentic data synthesis pipeline and a joint reinforcement learning (RL) stage". There, the model "improves its capabilities through interactions with real and synthetic environments". For tasks with no checker, it adds a "self-critique rubric reward": the model judges answers against written rubrics. DeepSeek-V3.2 describes a "large-scale agentic task synthesis pipeline" for tool use. And the Gemini 2.5 report names RL environments with "multi-step actions and tool use".

The hard part is the reward. A test suite can be fooled by code that special-cases the tests, and a judge model can be fooled by confident text. Much of the engineering of agent RL is making the checker hard to game.

How the models are tested

Training needs a reward, and progress needs a test. The tests themselves have changed as fast as the models.

From exams to harder exams

In September 2020, Dan Hendrycks and colleagues released MMLU, multiple-choice questions on 57 subjects, from law to physics. It became the standard test of knowledge. In the DeepSeek-R1 paper of 2025, R1 scored 90.8% and OpenAI's o1 91.8%. A re-check of the questions, MMLU-Redux, found errors in some of them. A test that every model passes can no longer tell models apart: it is saturated.

In November 2023, David Rein and colleagues released GPQA, 448 multiple-choice questions in biology, physics and chemistry written by domain experts. People with or studying for PhDs in the field scored 65%. Skilled non-experts scored 34%, even with over 30 minutes and free use of the web. That is why it is called "Google-proof". In January 2025, the Center for AI Safety and Scale AI released Humanity's Last Exam. Experts worldwide wrote its 2,500 questions at the edge of human knowledge.

For code, the field moved from writing one function to fixing real projects: from HumanEval in 2021 to SWE-bench. Code tests usually report pass@k, the chance that at least one of kk sampled answers is right. Mark Chen and colleagues gave an unbiased way to estimate it from nn samples, of which cc are right:

pass@k=1−(n−ck)(nk)\text{pass@}k = 1 - \frac{\binom{n - c}{k}}{\binom{n}{k}}

(nk)\binom{n}{k} is the number of ways to choose kk samples out of nn. So the fraction is the chance that a random choice of kk samples contains no right one. With 10 samples of which 3 are right, pass@1 is 1−7/10=0.31 - 7/10 = 0.3. pass@5 is 1−21/252=0.91671 - 21/252 = 0.9167. A reasoning model that is reported with pass@1 is being judged on its first try.

Tests of a different kind

In 2019, François Chollet published "On the Measure of Intelligence" and the ARC benchmark. Its small grid puzzles are easy for people, and each one needs a new rule. The ARC Prize reported that GPT-4o scored 5% on the original version in 2024. On 20 December 2024, it reported that OpenAI's o3 scored 75.7% on its semi-private test set, and 87.5% with about 172 times more compute. Harder versions of ARC followed, built to stay hard for machines.

The arena and AI judges

Fixed tests miss what users care about, such as tone, clarity and usefulness. In May 2023, a team at Berkeley's LMSYS opened Chatbot Arena, now called LMArena. A user asks anything, sees two anonymous answers, and votes for the better one. The 2024 paper reports over 240,000 votes at the time.

The ranking uses the Bradley-Terry model, the same equation as the reward model at the start of this part. If two models' scores differ by 1, the one with the higher score is preferred with probability σ(1)=0.7311\sigma(1) = 0.7311. The arena is a reward model for whole models, fitted to the votes of the public.

Arena rankings have their own Goodhart problems. In April 2025, "The Leaderboard Illusion", from Cohere Labs and university researchers, reported "undisclosed private testing practices". These let a few providers test many variants before release and publish only the best score. Arena votes also reward style: length, formatting and confident tone.

Human votes are slow, so many tests use a strong model as the judge. In June 2023, Lianmin Zheng and colleagues introduced MT-Bench and found that GPT-4 as a judge reached "over 80% agreement" with human preferences. That is the same level as between humans. But AI judges have biases too, for example toward longer answers and toward answers in a later position.

Contamination

The web is full of copies of popular benchmarks, and pre-training reads the web. The Olmo 3 report warns that results can be distorted when "mid-training data contain[s] the evaluation". The usual defences are fresh tests built after a model's training cut-off, such as new competition problems. Others are private held-out sets and searching the training data for test questions. None is perfect.

October 2026: the stages of a frontier model

By 2026, training a frontier language model is a long pipeline. No lab outside the open-model world publishes its full recipe. Anthropic, OpenAI and Google describe their stages only in outline, in model cards and reports. The figure is a composite of what the open reports describe and what the closed labs confirm in outline.

How a frontier language model is trained, as far as public reports showPre-trainingtrillions of tokensnext-token lossMid-trainingcurated maths, code,reasoning, long contextSFTinstructions, chats,reasoning traces, toolsPreference tuningreward models, DPO,AI feedbackReasoning RLverifiable rewards:tests, checkersAgent RLmulti-step tasksin tool environmentsSafety trainingspecs, constitutions,red-team dataDistillationsmall models learnfrom the big oneEvaluation at every stage: benchmarks, held-out tests, human and model judges, safety testsStages 3 to 7 are often repeated in rounds, and their order differs between labs. Few labs publish their full recipe.
A composite drawn from the reports of Llama 3, DeepSeek-R1 and V3.2, Qwen3, Olmo 3, Kimi K2 and Gemini 2.5. No single lab describes exactly this list.

The clearest public example is from the Allen Institute for AI, which releases its data, code and checkpoints for every stage.

Here are the stages, with what is public about each.

Pre-training on trillions of tokens of text, code and images, with the next-token loss of Part 1. The Olmo 3 report lists 5.9 trillion tokens for its pre-training mix.

Mid-training continues pre-training on a smaller, cleaner mix that favours maths, code, reasoning and question answering. For Olmo 3 it is 100 billion tokens. Long-context training, with longer documents, often follows.

Supervised fine-tuning on instructions, conversations, tool calls and long chains of thought. Many of the chains are written by a stronger model and filtered by checkers, not written by people.

Preference tuning with reward models, DPO or both, on human and AI-labelled comparisons. This is where style, tone and refusals are shaped.

Reasoning RL with verifiable rewards on maths, code and puzzles, usually with a GRPO-style or PPO-style algorithm. DeepSeek reports more than 10% of the pre-training cost here.

Agent RL in tool environments, with rewards from tests, checkers and AI judges with rubrics. Kimi K2, DeepSeek-V3.2 and Gemini 2.5 all describe some form of it.

Safety training runs through several of these stages. It includes refusal and red-team data in SFT and safety reward models (Llama 2 had a separate one). It also includes specifications or a constitution that the model learns to reason over.

Distillation makes the smaller members of a model family from the largest one, as Qwen3 and DeepSeek-R1 describe.

Evaluation runs alongside every stage, because each stage can damage what an earlier one built. The order varies, rounds repeat, and labs keep the details private. What they share is the shape. Learn from text, copy good examples, learn what people prefer, then practise against rewards that are hard to fool.

What is still open

Rewards are still the weak point. Learned reward models can be hacked, and human votes reward flattery. Checkers exist only where answers can be checked, and most useful work, such as writing, advice and research, has no unit test. Much of today's work is on rubric-based and model-based rewards that stretch RL to such tasks, and on keeping them honest.

The other open question is what RL really adds. Several 2025 papers found that base models already contain much of the reasoning that RL brings out. In that view, RL mostly selects and strengthens behaviour that pre-training made possible, as with the aha moment in DeepSeek-V3-Base. Others point to new skills learned in long agent training. The answer matters for where the next unit of compute should go.

StepWhat it fixedWhat it left open
Preferences from people (2017 to 2020)A reward for goals nobody can write downExpensive labels; reward hacking
RLHF with PPO, InstructGPT (2022)Helpful assistants from base modelsFour models in memory; fiddly training
KL penalty, early stoppingLimits on over-optimisationStill a proxy; length and style bias
Constitutional AI, RLAIF (2022, 2023)Cheaper labels; written principlesAI judges have their own biases
DPO and its family (2023, 2024)Preference learning without RLLearns only from a fixed data set
Verifiers, PRMs, STaR (2021 to 2023)Signals for correct reasoningStep labels are costly
o1, RLVR, GRPO, R1 (2024, 2025)Long reasoning from checkable rewardsOnly where answers can be checked
DAPO, Dr. GRPO, Qwen3 (2025)Stable, efficient reasoning RLLength and entropy problems remain
Agent RL in environments (2025, 2026)Models that finish multi-step tasksCheckers that can be gamed
Harder tests, arenas, AI judgesMeasuring models beyond saturated testsContamination; leaderboard gaming

Key papers: Christiano et al. 2017, InstructGPT 2022, Gao et al. 2022, DPO 2023, DeepSeekMath 2024, DeepSeek-R1 2025, Olmo 3 2025.

Next: Part 5, Inside a Modern Language Model, how the base model is built: its data, scale and architecture.