LLM from Scratch

Tokenizer to GRPO, one working model

Build a language model end to end (tokenizer, data pipeline, architecture, pretraining, fine-tuning, mixture of experts, reasoning and RL with GRPO), then learn how frontier labs do the same thing at scale.

Nineteen chapters that take a model from an empty directory to a trained, evaluated checkpoint, and explain every decision along the way, including the ones frontier labs make differently and why.

Chapters

Chapters are being edited for publication.