From RLHF to Reasoning: How Language Models Are Trained Today
How chat and reasoning models are taught after pre-training: human preference labels, AI feedback, answers a program can check, and how we test the result.
You give a model a hard maths problem. It writes a few lines, stops, and writes "Wait, wait. Wait." Then it goes back and checks its own work. Nobody typed that line into its training data as an example to copy. The model found that checking earned more reward, so checking became a habit.
That line comes from DeepSeek-R1-Zero, released in January 2025. After pre-training, it was trained with reinforcement learning and almost nothing else. During that training, its score on the AIME 2024 maths competition rose from 15.6% to 77.9%. The reward was simple: 1 if the final answer was right, 0 if it was not.
This is Part 4 of How AI Got Here. Part 1 told the story of neural networks and gave a short account of RLHF; Part 2 covered classical machine learning. Part 3 followed reinforcement learning from Thorndike's cats to PPO and AlphaZero. This part shows how those methods became the way language models are trained, from 2017 to October 2026. Part 5 then opens the base model itself.
Why a pre-trained model needs more training
A model trained only to predict the next token is called a base model. It has read a large part of the public web, so it knows a great deal. But it continues text; it does not answer. Ask it "How do I bake bread?" and it may write three more questions, as a web forum might.
The first step of post-training is supervised fine-tuning (SFT). People write good answers to many prompts, and the model is trained on them with the same next-token loss as before. It is like an apprentice copying the work of a master. It works, but it has limits.
Writing a great answer is slow and costly, and the best a copy can be is as good as the person who wrote it. Judging is easier than writing. Most people cannot write a prize-winning essay, but they can tell which of two essays they prefer. That difference is the starting point of everything in this part.
2017 to 2020: learning a reward from people
Preferences between video clips
In June 2017, Paul Christiano and colleagues at OpenAI and DeepMind showed how to train an agent when nobody can write the reward. A person watched two short video clips of the agent and picked the one that looked closer to the goal. A network learned to predict those choices. The agent then ran ordinary reinforcement learning against the predicted reward, while new clips went to the person.
From clips to text
In 2019, Daniel Ziegler and colleagues at OpenAI applied the method to GPT-2. People compared continuations of text and later summaries. The paper used a KL penalty to keep the fine-tuned model close to the original, following earlier work by Natasha Jaques and colleagues. It also found an early warning. With 60,000 comparisons, the summarisers learned to copy whole sentences from the input, which pleased the labellers but was not real summarising.
In September 2020, Nisan Stiennon and colleagues scaled the method for summaries of Reddit posts (the TL;DR data set).
The reward model, up close
Ranking by comparisons
How do you turn "A is better than B" into a number for each answer? Chess has the same problem. Players are never given an absolute score, only wins and losses, yet the Elo rating gives each player one number. The gap between two ratings predicts who will win. In 1952, Ralph Bradley and Milton Terry wrote down the model behind such ratings.
is the prompt, is the answer people preferred (the winner) and is the other one (the loser). The symbol means "is preferred to". is the reward model's score for an answer, with parameters (phi). The sigmoid turns any number into a probability between 0 and 1.
Training makes the observed choices as likely as possible, which gives the loss that Part 1 showed briefly:
is the data set of comparisons and means the average over it. The loss is small when the winner's score is well above the loser's.
Our script gives the loss for four score gaps. A gap of 2 gives a probability of 0.8808 and a loss of 0.1269. A gap of 0.5 gives 0.4741. A gap of 0 gives , which means "no idea". A gap of , the wrong way round, gives 2.1269.
Only differences matter. If you add 10 to every score, the loss does not change. So a reward model learns an ordering, not a grade, and its numbers mean nothing on their own. In practice it is a copy of the language model with its word output replaced by a single number. The number is read at the last token of the answer.
2022: InstructGPT and the RLHF objective
Three stages
In March 2022, Long Ouyang and colleagues at OpenAI published InstructGPT, which Part 1 described. Their recipe had three stages, and the paper gave the sizes of the data. About 13,000 prompts with written answers went into SFT. People ranked answers for 33,000 prompts to train the reward model. A further 31,000 prompts, with no labels at all, were used for the RL stage.
The objective with a leash
In the third stage, the model writes answers and the reward model scores them. Left alone, PPO would push the model toward whatever the reward model happens to like most, even nonsense. So the objective adds a penalty for moving away from the SFT model. It works like a leash on a dog, which can explore but not run off.
is the language model being trained, the policy. is the reference model, usually the SFT model, kept frozen. means "an answer sampled from the policy for prompt ". is the KL divergence between the two models' distributions over answers. (beta) sets the length of the leash.
In practice, each sampled answer gets a penalised reward: the reward model's score minus times the log ratio of the two models' probabilities. Take . Answer 1 scores 2.0, and the policy gives it a log-probability of against for the reference. Its penalised reward is . Answer 2 scores higher, 2.6, but the policy has moved far to produce it: against . Its penalised reward is , so the leash prefers answer 1.
InstructGPT also mixed in some pre-training gradients (PPO-ptx), so the model would not forget skills while it learned to please the reward model. The paper called the loss of skills an "alignment tax".
Helpful and harmless
In April 2022, Yuntao Bai and colleagues at Anthropic published "Training a Helpful and Harmless Assistant with RLHF". They collected two kinds of comparisons, one for helpfulness and one for harmlessness, and found the two goals pull against each other. A model that refuses everything is harmless but useless. They also retrained with fresh human feedback each week, an "online" form of RLHF. And they reported "a roughly linear relation between the RL reward and the square root of the KL divergence".
Goodhart's law: when the reward model is wrong
A reward model is a measure, and RL makes it a target. It was trained on answers from the SFT model. As the policy moves away, it writes answers the reward model has never seen, and the scores become less reliable. The policy finds and exploits those blind spots. This is reward hacking.
In October 2022, Leo Gao, John Schulman and Jacob Hilton measured this without human labels. A large "gold" reward model played the role of people and labelled data. Smaller "proxy" reward models were trained on those labels, and policies were optimised against the proxies. Since the gold model was available, they could watch the true score as the proxy score rose.
The paper fits a simple curve to each line. With , the distance from the start, the gold score under RL follows
At first the term wins and the true score climbs. The second term, , grows faster, and eventually pulls the score back down. and are fitted numbers that depend on the size of the reward model. The lesson for practice: watch the KL, keep the leash, and stop early.
Best-of-n and rejection sampling
The simplest optimiser
There is a way to use a reward model with no RL at all. Sample answers, score them all, and keep the best. It is like a photographer who takes twenty shots and prints one.
to are answers sampled independently from the model . The first formula keeps the one with the highest reward. The second gives the KL divergence of this best-of-n procedure from the model, in nats. Gao and colleagues take it from the appendix of Stiennon and colleagues. It grows only like .
Say each answer is right with probability 0.3, and a perfect judge picks a right one when there is one. Then the chance of success is . For it is 0.7599, for it is 0.9967, at a KL cost of 0.6363 and 1.8351 nats. Real reward models are not perfect judges, so best-of-n can be hacked too; the Gao paper shows the same rise and fall for it.
Llama 2
Rejection sampling fine-tuning turns best-of-n into training: sample many answers, keep the best by the reward model, and fine-tune on them. Meta used it, together with PPO, for Llama 2-Chat in July 2023. Its paper is one of the most detailed public accounts of an RLHF pipeline.
2022 to 2026: feedback from AI and from written principles
Part 1 introduced Constitutional AI, which Anthropic published in December 2022. It has two phases. In the first, the model critiques and revises its own answers against a list of principles, and is fine-tuned on the revisions. In the second, a model compares pairs of answers against the principles, and those AI-made preferences train the reward model. This second phase is RLAIF, reinforcement learning from AI feedback.
In 2023, Harrison Lee and colleagues at Google compared the two directly. Across summaries, helpful dialogue and harmless dialogue, they found that "RLAIF achieves comparable performance to RLHF". AI labels are fast and cheap, and the principles behind them are written down, so anyone can read them.
Written principles grew into long public documents. OpenAI published its Model Spec on 8 May 2024 and updated it in February 2025. On 22 January 2026, Anthropic published a new constitution for Claude, far longer than the 2023 one. It explains the reasons behind its rules, and Anthropic says the document is part of how the models are trained.
In December 2024, OpenAI described deliberative alignment, a way to train a reasoning model to think about the rules before it answers.
2023 to 2024: DPO and the family after it
The derivation
RLHF with PPO is heavy. It keeps four large models in memory: the policy, the reference, the reward model and a value model for PPO's advantages. It needs fresh samples at every step, and it has many settings to tune. In May 2023, Rafael Rafailov, Archit Sharma, Eric Mitchell and colleagues at Stanford showed that, for preferences, the RL step can be removed.
Their starting point is a known result: the KL-penalised objective above has an exact best policy.
Read it as "start from the reference model and tilt it toward high reward". is the exponential function, so each unit of reward multiplies an answer's probability by . is the sum of the tilted probabilities over all answers, which makes them add up to 1 again. Nobody can compute , because the number of possible answers is astronomical.
The trick is to solve this equation for the reward instead:
Now put this reward into the Bradley-Terry model. Bradley-Terry uses only the difference between two rewards for the same prompt, so the two terms cancel. The impossible sum disappears, and what is left depends only on the policy and the reference.
This is the reward-model loss, with the reward replaced by , the implicit reward. The language model is its own reward model. Training raises the probability of the chosen answer and lowers that of the rejected one, each measured against the reference.
A worked example
Take and one pair. The policy gives the chosen answer a log-probability of , against for the reference, so its implicit reward is . The rejected answer has against , for an implicit reward of . The margin is 0.3, the sigmoid gives 0.5744, and the loss is 0.5544.
At the very start, the policy equals the reference, every margin is 0, and the loss is . The gradient of each pair is weighted by , which starts at 0.5. In our example it has already fallen to 0.4256. Pairs the model already gets right count less, and pairs it gets wrong count more.
DPO spread fast because it is cheap and stable. Hugging Face's Zephyr models used it in late 2023. Meta's Llama 3 paper of July 2024 describes post-training with SFT, rejection sampling and DPO, "as opposed to more complex reinforcement learning algorithms". Its weakness is that it learns only from the pairs in its fixed data set, while PPO keeps sampling new answers from the current model.
The family after DPO
DPO opened a family of "offline" preference methods. IPO (Mohammad Gheshlaghi Azar and colleagues at Google DeepMind, October 2023) argued that both RLHF and DPO can overfit. The reason is the Bradley-Terry assumption that every preference becomes a point score, and IPO replaces the log-sigmoid with a squared loss on the margin. KTO (Kawin Ethayarajh and colleagues, February 2024) borrowed prospect theory from economics, which says people feel losses more than gains. It learns from single answers marked good or bad, so it needs no pairs.
ORPO (Jiwoo Hong and colleagues, March 2024) folded preference learning into SFT itself and needs no reference model. SimPO (Yu Meng and colleagues, May 2024) also drops the reference model. Its implicit reward is the average log-probability per token of an answer. It also adds a target margin by which the winner must beat the loser. Taking the average per token keeps the length of an answer from tilting its reward.
Rewarding the steps: verifiers and process rewards
Train a checker, keep the best
A preference reward model judges style and helpfulness. For a maths problem, there is a sharper question: is the answer right? In October 2021, Karl Cobbe and colleagues at OpenAI released GSM8K, 8,500 grade-school maths word problems. They trained verifiers to judge whether a solution is correct. At test time they sampled many solutions and kept the one the verifier ranked highest, which is best-of-n with a correctness judge.
STaR: learn from your own right answers
In March 2022, Eric Zelikman and colleagues published STaR, the Self-Taught Reasoner. The model writes a chain of thought and an answer for many problems. The chains that reach the right answer are kept, and the model is fine-tuned on them; then the loop repeats. For problems it fails, the model gets the right answer and writes a reasoning that leads to it. The authors call this step "rationalization". STaR is a filtered form of reinforcement learning: a reward of 1 for a correct answer, and training only on the rewarded samples.
Outcome or process?
A wrong final answer tells you that something went wrong, not where. A teacher who marks every line of a pupil's work gives far more help. In May 2023, Hunter Lightman and colleagues at OpenAI compared two kinds of reward model on hard maths problems.
September 2024: o1 and thinking at test time
On 12 September 2024, OpenAI released o1 and a post called "Learning to reason with LLMs". It said that o1 is trained with reinforcement learning to think, in a long chain of thought, before it answers. It also said that the performance of o1 "consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute)".
The post gave numbers for the 2024 AIME, a hard American maths competition. GPT-4o solved 12% of the problems on average. o1 solved 74% with one sample per problem and 83% with a consensus of 64 samples. A consensus answer is the most common final answer across the samples.
OpenAI did not publish how o1 was trained: not the algorithm, the rewards or the data. The post showed accuracy rising along two axes, training compute and thinking time, and that was enough to change the field. Within months, several labs published their own versions, with far more detail.
Rewards a program can check: RLVR
In November 2024, Nathan Lambert and colleagues at the Allen Institute for AI published Tülu 3, a fully open post-training recipe. Its last stage gave a name to an idea that was spreading: reinforcement learning with verifiable rewards.
GRPO: dropping the critic
The idea
PPO needs a value model to estimate advantages, and for a language model the value model is as big as the policy. In February 2024, Zhihong Shao and colleagues at DeepSeek proposed a cheaper baseline in their DeepSeekMath paper. For each question, sample a group of answers, and compare each answer with the group's average. It is like grading on a curve: a mark means something only next to the marks of the rest of the class.
The equations
The advantage of answer in a group of answers is its reward, centred and scaled by the group:
Every token of answer gets this same advantage. The objective is PPO's clipped objective, averaged over the tokens of each answer and over the group, with the KL penalty inside:
is the number of tokens in answer . (rho) is the probability ratio of token under the new and the old policy, as in PPO: . is the question and the tokens before . is the clip range and the KL weight.
Take one question and eight answers, three right and five wrong. The rewards are 1, 0, 0, 1, 1, 0, 0, 0, with mean 0.375 and standard deviation 0.4841. The right answers get an advantage of and the wrong ones . For an easy question with seven right answers, the one wrong answer gets , and the right ones only . If all eight are right, the standard deviation is 0 and every advantage is 0: the question teaches nothing.
With GRPO, DeepSeekMath 7B reached 51.7% on the MATH benchmark, the paper reports, without tools or voting. The method was cheap enough to run at scale, and a year later it was the engine of DeepSeek-R1.
January 2025: DeepSeek-R1
R1-Zero: reinforcement learning alone
On 20 January 2025, DeepSeek released DeepSeek-R1, with open weights and a paper; a longer version appeared in Nature on 17 September 2025. Its most striking model was R1-Zero. It started from the base model DeepSeek-V3-Base and went straight to RL with GRPO, with no SFT first. The rewards were rule-based: an accuracy reward for a correct final answer, and a format reward for putting the reasoning between think tags.
R1-Zero had problems. Its reasoning was hard to read, and it often mixed English and Chinese in one chain of thought. So DeepSeek-R1 used a fuller pipeline. A small set of long, readable reasoning examples came first (a "cold start"), then reasoning RL. Then came SFT on a large set of the model's own filtered answers and general data, and a final RL stage for helpfulness and safety.
The team also distilled R1 into smaller open models based on Qwen and Llama by fine-tuning them on R1's outputs. The paper's appendix lists the RL training cost of R1 as 147,000 H800 GPU hours. At 2 dollars per hour, that is about 294,000 US dollars. That does not include the much larger cost of pre-training the base model.
The paper is also open about reward hacking. For reasoning, DeepSeek used no neural reward models at all, because it found them "susceptible to reward hacking" at scale. In the last stage, a model-based preference reward was used for general tasks. One figure shows that reward rising while performance on CodeForces falls: Goodhart's law again.
2025: fixing GRPO and scaling the recipe
The R1 paper started a rush of open work. In the same week, Moonshot AI's Kimi k1.5 report described long-context RL with a simple framework, "without relying on more complex techniques such as Monte Carlo tree search, value functions, and process reward models". Both reports said that plain outcome rewards and a lot of RL were enough.
Then came the fixes. In March 2025, ByteDance Seed and Tsinghua University released DAPO, with four changes to GRPO. Clip-Higher raises the upper clip limit, so rare tokens that turn out well can grow faster. That keeps the policy from collapsing onto a few answers. Dynamic Sampling drops prompts whose group is all right or all wrong, since they give zero advantage. A token-level loss and a shaped penalty for overlong answers complete the set.
Also in March 2025, Zichen Liu and colleagues at Sea AI Lab published "Understanding R1-Zero-Like Training". They found an "optimization bias" in GRPO that "artificially increases response length (especially for incorrect outputs)". Dividing by the answer length makes each token of a long wrong answer count for less, so long wrong answers are punished too gently. Their Dr. GRPO drops that division and the division by the standard deviation.
For our group of eight, Dr. GRPO's advantages are simply : for a right answer and for a wrong one. Without the standard deviation, an easy question no longer gives its one wrong answer an outsized push. The same paper found that DeepSeek-V3-Base "already exhibit[s]" an aha moment before RL, so RL amplifies a behaviour that pre-training made possible.
In May 2025, Alibaba's Qwen3 report described one model with both a "thinking" and a "non-thinking" mode. The user can also set a thinking budget.
By the end of 2025, RL was no longer a small final polish. In its December 2025 report, DeepSeek wrote that the post-training of DeepSeek-V3.2 used a large compute budget.
What Google says about Gemini
Google publishes less than DeepSeek, but its 2025 technical report on Gemini 2.5 gives an outline of post-training and of the "thinking" models.
The same report describes Deep Think, a mode of Gemini 2.5 Pro that "naturally blends in parallel thinking techniques during response generation". It lets the model "produce multiple hypotheses and carefully critique them before arriving at the final answer". On 21 July 2025, Google DeepMind announced that an advanced version of Gemini with Deep Think had reached gold-medal standard at the International Mathematical Olympiad.
It solved five of the six problems for 35 of 42 points, within the 4.5-hour time limit. It worked in natural language from the official problem statements, and IMO coordinators graded and certified the solutions. A year earlier, AlphaProof and AlphaGeometry 2 had scored 28 points, but only after the problems were translated into a formal language. The post names new reinforcement learning methods, data on multi-step reasoning and theorem proving, and parallel thinking. OpenAI reported the same score of 35 points for an experimental model of its own.
Sycophancy: when the reward is a thumbs-up
Preference learning has a built-in danger: people like to be agreed with. In October 2023, Mrinank Sharma and colleagues at Anthropic studied five assistants trained with human feedback and found sycophancy in all of them. Their analysis of preference data showed why. Both humans and preference models sometimes preferred convincing, sycophantic answers over correct ones.
In April 2025, this happened in a widely used product. On 25 April, OpenAI shipped an update to GPT-4o, the default model of its chat app. Users soon saw it flattering them and endorsing poor decisions. OpenAI began rolling it back on 28 April. In its account of 2 May, OpenAI said the update had added a reward signal based on users' thumbs-up and thumbs-down ratings. Together with other changes, this weakened the main reward signal that had kept sycophancy in check.
The lesson is Goodhart's law again, at the scale of millions of users. A thumbs-up measures what pleases a person in the moment. Optimise it hard enough and it stops measuring what helps them. OpenAI said it would weigh behaviour problems like this more heavily in its review before each launch.
RL for agents
A chat answer is one step. An agent works for many steps: it reads files, runs commands, reads the results and tries again. Training it with RL means building environments: a computer with a code base, a task, tools and a checker at the end.
Coding is the clearest case because the reward is checkable. SWE-bench, from Carlos Jimenez and colleagues at Princeton in October 2023, has 2,294 real GitHub issues from 12 Python projects. A model must edit the code base so that the project's tests pass. In the paper, the best model, Claude 2, solved 1.96% of the issues. In August 2024, OpenAI released SWE-bench Verified, a subset of 500 issues that people had checked were solvable.
Labs now build such environments at scale. Moonshot AI's Kimi K2 report of July 2025 describes "a large-scale agentic data synthesis pipeline and a joint reinforcement learning (RL) stage". There, the model "improves its capabilities through interactions with real and synthetic environments". For tasks with no checker, it adds a "self-critique rubric reward": the model judges answers against written rubrics. DeepSeek-V3.2 describes a "large-scale agentic task synthesis pipeline" for tool use. And the Gemini 2.5 report names RL environments with "multi-step actions and tool use".
The hard part is the reward. A test suite can be fooled by code that special-cases the tests, and a judge model can be fooled by confident text. Much of the engineering of agent RL is making the checker hard to game.
How the models are tested
Training needs a reward, and progress needs a test. The tests themselves have changed as fast as the models.
From exams to harder exams
In September 2020, Dan Hendrycks and colleagues released MMLU, multiple-choice questions on 57 subjects, from law to physics. It became the standard test of knowledge. In the DeepSeek-R1 paper of 2025, R1 scored 90.8% and OpenAI's o1 91.8%. A re-check of the questions, MMLU-Redux, found errors in some of them. A test that every model passes can no longer tell models apart: it is saturated.
In November 2023, David Rein and colleagues released GPQA, 448 multiple-choice questions in biology, physics and chemistry written by domain experts. People with or studying for PhDs in the field scored 65%. Skilled non-experts scored 34%, even with over 30 minutes and free use of the web. That is why it is called "Google-proof". In January 2025, the Center for AI Safety and Scale AI released Humanity's Last Exam. Experts worldwide wrote its 2,500 questions at the edge of human knowledge.
For code, the field moved from writing one function to fixing real projects: from HumanEval in 2021 to SWE-bench. Code tests usually report pass@k, the chance that at least one of sampled answers is right. Mark Chen and colleagues gave an unbiased way to estimate it from samples, of which are right:
is the number of ways to choose samples out of . So the fraction is the chance that a random choice of samples contains no right one. With 10 samples of which 3 are right, pass@1 is . pass@5 is . A reasoning model that is reported with pass@1 is being judged on its first try.
Tests of a different kind
In 2019, François Chollet published "On the Measure of Intelligence" and the ARC benchmark. Its small grid puzzles are easy for people, and each one needs a new rule. The ARC Prize reported that GPT-4o scored 5% on the original version in 2024. On 20 December 2024, it reported that OpenAI's o3 scored 75.7% on its semi-private test set, and 87.5% with about 172 times more compute. Harder versions of ARC followed, built to stay hard for machines.
The arena and AI judges
Fixed tests miss what users care about, such as tone, clarity and usefulness. In May 2023, a team at Berkeley's LMSYS opened Chatbot Arena, now called LMArena. A user asks anything, sees two anonymous answers, and votes for the better one. The 2024 paper reports over 240,000 votes at the time.
The ranking uses the Bradley-Terry model, the same equation as the reward model at the start of this part. If two models' scores differ by 1, the one with the higher score is preferred with probability . The arena is a reward model for whole models, fitted to the votes of the public.
Arena rankings have their own Goodhart problems. In April 2025, "The Leaderboard Illusion", from Cohere Labs and university researchers, reported "undisclosed private testing practices". These let a few providers test many variants before release and publish only the best score. Arena votes also reward style: length, formatting and confident tone.
Human votes are slow, so many tests use a strong model as the judge. In June 2023, Lianmin Zheng and colleagues introduced MT-Bench and found that GPT-4 as a judge reached "over 80% agreement" with human preferences. That is the same level as between humans. But AI judges have biases too, for example toward longer answers and toward answers in a later position.
Contamination
The web is full of copies of popular benchmarks, and pre-training reads the web. The Olmo 3 report warns that results can be distorted when "mid-training data contain[s] the evaluation". The usual defences are fresh tests built after a model's training cut-off, such as new competition problems. Others are private held-out sets and searching the training data for test questions. None is perfect.
October 2026: the stages of a frontier model
By 2026, training a frontier language model is a long pipeline. No lab outside the open-model world publishes its full recipe. Anthropic, OpenAI and Google describe their stages only in outline, in model cards and reports. The figure is a composite of what the open reports describe and what the closed labs confirm in outline.
The clearest public example is from the Allen Institute for AI, which releases its data, code and checkpoints for every stage.
Here are the stages, with what is public about each.
Pre-training on trillions of tokens of text, code and images, with the next-token loss of Part 1. The Olmo 3 report lists 5.9 trillion tokens for its pre-training mix.
Mid-training continues pre-training on a smaller, cleaner mix that favours maths, code, reasoning and question answering. For Olmo 3 it is 100 billion tokens. Long-context training, with longer documents, often follows.
Supervised fine-tuning on instructions, conversations, tool calls and long chains of thought. Many of the chains are written by a stronger model and filtered by checkers, not written by people.
Preference tuning with reward models, DPO or both, on human and AI-labelled comparisons. This is where style, tone and refusals are shaped.
Reasoning RL with verifiable rewards on maths, code and puzzles, usually with a GRPO-style or PPO-style algorithm. DeepSeek reports more than 10% of the pre-training cost here.
Agent RL in tool environments, with rewards from tests, checkers and AI judges with rubrics. Kimi K2, DeepSeek-V3.2 and Gemini 2.5 all describe some form of it.
Safety training runs through several of these stages. It includes refusal and red-team data in SFT and safety reward models (Llama 2 had a separate one). It also includes specifications or a constitution that the model learns to reason over.
Distillation makes the smaller members of a model family from the largest one, as Qwen3 and DeepSeek-R1 describe.
Evaluation runs alongside every stage, because each stage can damage what an earlier one built. The order varies, rounds repeat, and labs keep the details private. What they share is the shape. Learn from text, copy good examples, learn what people prefer, then practise against rewards that are hard to fool.
What is still open
Rewards are still the weak point. Learned reward models can be hacked, and human votes reward flattery. Checkers exist only where answers can be checked, and most useful work, such as writing, advice and research, has no unit test. Much of today's work is on rubric-based and model-based rewards that stretch RL to such tasks, and on keeping them honest.
The other open question is what RL really adds. Several 2025 papers found that base models already contain much of the reasoning that RL brings out. In that view, RL mostly selects and strengthens behaviour that pre-training made possible, as with the aha moment in DeepSeek-V3-Base. Others point to new skills learned in long agent training. The answer matters for where the next unit of compute should go.
| Step | What it fixed | What it left open |
|---|---|---|
| Preferences from people (2017 to 2020) | A reward for goals nobody can write down | Expensive labels; reward hacking |
| RLHF with PPO, InstructGPT (2022) | Helpful assistants from base models | Four models in memory; fiddly training |
| KL penalty, early stopping | Limits on over-optimisation | Still a proxy; length and style bias |
| Constitutional AI, RLAIF (2022, 2023) | Cheaper labels; written principles | AI judges have their own biases |
| DPO and its family (2023, 2024) | Preference learning without RL | Learns only from a fixed data set |
| Verifiers, PRMs, STaR (2021 to 2023) | Signals for correct reasoning | Step labels are costly |
| o1, RLVR, GRPO, R1 (2024, 2025) | Long reasoning from checkable rewards | Only where answers can be checked |
| DAPO, Dr. GRPO, Qwen3 (2025) | Stable, efficient reasoning RL | Length and entropy problems remain |
| Agent RL in environments (2025, 2026) | Models that finish multi-step tasks | Checkers that can be gamed |
| Harder tests, arenas, AI judges | Measuring models beyond saturated tests | Contamination; leaderboard gaming |
Key papers: Christiano et al. 2017, InstructGPT 2022, Gao et al. 2022, DPO 2023, DeepSeekMath 2024, DeepSeek-R1 2025, Olmo 3 2025.
Next: Part 5, Inside a Modern Language Model, how the base model is built: its data, scale and architecture.