How Models Are Trained

From next-word predictor to assistant: pretraining, SFT, RLHF, DPO and RL for reasoning

How a language model goes from predicting the next word to following instructions, holding a conversation and reasoning step by step. The history from 2017 to today, the research papers as highlighted screenshots, every equation explained symbol by symbol, and real code run on real models, in plain English for beginners.

A model that has only read the internet can finish your sentence, but it cannot answer your question. Something happens between the raw model and the assistant you talk to, and that something has a history, a set of equations and a lot of practical tricks. This book explains all of it, one step at a time.

It is written for beginners, in simple English. Every idea comes with the research paper that introduced it (shown as highlighted screenshots), a simple picture, the maths explained one symbol at a time with real numbers, and small pieces of real code that you can run yourself, with every line explained.

Part 1, the big picture, covers what training actually changes inside a model, how the methods began and evolved, and how anyone can tell whether a model got better.

Part 2, from base model to instruction follower, covers pretraining (the data, the compute arithmetic, scaling laws, and a tiny GPT trained from scratch) and supervised fine-tuning (instruction datasets, chat templates, the masked loss, LoRA, and a real fine-tune of a small open model).

Part 3, learning from feedback, covers reinforcement learning from scratch, reward models and RLHF with PPO.

Chapters

  • Chapter 1 · From next word to assistant

    What a language model computes (tokens, next-token probabilities, the chain rule, softmax, cross-entropy, perplexity), the full training pipeline from pretraining to RL with verifiable rewards, and hands-on experiments with Qwen2.5-0.5B base and instruct: the chat template, how few tokens post-training really changes, and a tiny SFT loop.

  • Chapter 2 · How it started, and how it evolved

    The history of how language models learned to follow instructions, told as one story where each step fixes a problem left by the step before: from Shannon's n-grams and the Transformer, through pretraining, learning from human preferences and instruction tuning, to InstructGPT, ChatGPT, DPO, GRPO and DeepSeek-R1. With the papers' own figures, the four key equations worked on real numbers, and small experiments you can run.

  • Chapter 3 · How we measure a model

    How language models are measured, and how far to trust the numbers: held-out loss, perplexity and bits per byte; the history of benchmarks from GLUE to GPQA; how a multiple-choice question is really scored; confidence intervals; pass@k; contamination; human preference, Elo and Bradley-Terry ratings; model judges and their biases; reward-model evaluation and reward hacking. With real evaluations of Qwen2.5-0.5B base and Instruct, and the papers' own tables and figures.

  • Chapter 4 · Pretraining: learning from the whole internet

    Pretraining from the inside: how web text is collected, filtered and deduplicated, how it is tokenized, what training costs (C = 6ND), what scaling laws say about model size and data, the learning-rate schedule, batch size and mixed precision, and a real 12M-parameter GPT pretrained from random weights.

  • Chapter 5 · Supervised fine-tuning: teaching it to follow instructions

    Supervised fine-tuning from the inside: the same loss on different data with the prompt masked out, the history of instruction datasets from FLAN to Tulu 3, chat templates, a worked masked loss on real tokens, hyperparameters and packing, LoRA and QLoRA with their maths, catastrophic forgetting, and a real LoRA fine-tune of Qwen2.5-0.5B.

  • Chapter 6 · Reinforcement learning, from scratch

    Reinforcement learning built up from nothing for language models: the vocabulary mapped onto text, return and discount, the policy gradient derived step by step with the log-derivative trick, REINFORCE, why variance is the enemy, baselines with a proof that they cost no bias, value functions, advantages, TD errors and GAE with its lambda trade-off, and the per-token KL penalty as reward shaping. Hands-on: a bandit over 200 seeds, a token world where every value is exact, and REINFORCE on GPT-2 with and without a KL penalty.

  • Chapter 7 · Reward models: turning preferences into a number

    Reward models from the inside: why RLHF learns a reward instead of asking people every time, ratings against comparisons, the Bradley-Terry model derived from a noisy-judge story, the reward model loss and its gradient worked by hand, rankings, label noise and normalisation, the architecture (a language model with a one-number head), a real LoRA reward model trained on UltraFeedback, its accuracy, calibration and length bias, best-of-n overoptimisation against a stronger gold reward model, margins and ensembles, and process rewards.

  • Chapter 8 · RLHF with PPO: the full loop

    RLHF with PPO from the inside: why plain REINFORCE wastes samples and takes unsafe steps, trust regions in plain words, the probability ratio and PPO's clipped objective worked by hand, the value loss and the entropy bonus, GAE on a real reply, the four models of RLHF and what they cost in memory, the implementation details that make PPO work for language models, and a from-scratch PPO run on Qwen2.5-0.5B-Instruct against the Chapter 7 reward model, checked by a stronger gold judge.