Topic map

Attention and Transformers

Pre-training (BERT)

  • BERT

    A Transformer encoder that reads text in both directions, pre-trained once and fine-tuned for any task.

  • Masked language modelling

    Hide 15% of the words and train the model to guess them from both sides.

  • Pre-train, then fine-tune

    Train once on a huge dataset, then adapt a copy to each task with one small new layer.

  • The [CLS] / [class] token

    An extra token whose output vector stands for the whole input. BERT invented it; ViT reused it.

Vision (ViT)

  • Vision Transformer (ViT)

    Cut a picture into 16×16 patches, treat them as words, and feed them to an unchanged Transformer encoder.

  • Images as patches

    A 224×224 picture becomes 196 patches of 768 numbers each: the image's tokens.

  • Inductive bias versus data

    Built-in assumptions help with little data; with 300 million images, learning from data wins.

  • Convolutional networks (ResNet)

    The design that ruled computer vision from 2012 to 2020: small filters slid across the picture, with residual shortcuts.

LLM inference

  • Prefill and decode

    How an LLM answers: read the whole prompt at once, then write one token at a time.

  • KV cache

    The keys and values of every past token, kept in memory so they are not recomputed. It is the biggest memory bill.

  • vLLM: paging and batching

    Serving thousands of requests by paging the KV cache like an operating system and batching continuously.

  • Speculative decoding

    A small model drafts several tokens; the big model checks them in one step. Same words, fewer steps.

  • Prefix caching

    Requests that start the same way reuse the saved KV cache, so the shared start is not computed again.

  • SGLang and RadixAttention

    A radix tree of cached prefixes, compared step by step with vLLM's hashed blocks.

  • Cache-aware scheduling

    Serving requests that share a cached start together raises reuse, at the risk of making others wait.

  • Faster structured output

    When the JSON format fixes the next tokens, write them at once instead of one step at a time.

  • Tensor parallelism

    Split every layer across GPUs, columns then rows: two all-reduces per layer, checked on a real two-process run.

  • All-reduce and other collectives

    All-reduce, all-gather, reduce-scatter and all-to-all, the ring traced step by step, and its cost formula.

  • Pipeline parallelism

    Give each GPU a slice of the layers: little traffic across slow links, more throughput, no lower latency.

  • Expert parallelism (MoE)

    Spread experts over GPUs with all-to-all, measured routing imbalance, and redundant hot experts as in DeepSeek-V3.

  • Disaggregated prefill and decode

    Run the two phases on different GPUs and move the KV cache between them: Splitwise, DistServe, Mooncake, and when it pays.

  • Latency SLOs and goodput

    TTFT, ITL and end-to-end latency as percentiles, and goodput: the requests per second served inside the SLO.

  • Queueing and the latency cliff

    Little's law and why waiting explodes near full load, checked on a simulated batching engine.

  • Capacity planning and cost per token

    From model size, KV bytes and traffic to a number of GPUs, and why a generated token costs more than a prompt token.

  • Quantization for serving

    INT8, INT4, FP8 and an FP8 KV cache: memory and speed gained, quality lost, measured on a real model.

  • Routing across replicas

    Sending a request to the replica that holds its prefix, without overloading it.

  • Multi-LoRA serving

    Many fine-tuned adapters on one base model, batched together in one step.

  • Autoscaling and failure modes

    Cold starts, retry storms, preemption storms and noisy neighbours, and the metrics that reveal them.

Retrieval and RAG

  • RAG

    Retrieve the right passages first, then let the model answer from them. A full book, from first principles.

  • Embeddings

    Meaning as geometry: texts become vectors, and similar meanings land close together.

  • Vector search and HNSW

    Finding the nearest vectors among millions fast, with a layered graph of neighbours.

  • BM25 and hybrid search

    Keyword scores and vector scores, fused, beat either alone.

  • Rerankers

    A second, slower model (often BERT-like) re-orders the top results more carefully.

  • Evaluating RAG

    Every retrieval and generation metric, and how to measure a pipeline honestly.

Agents

GPUs and hardware

Training LLMs

  • How models are trained

    A book on how a chatbot is made: pretraining, fine-tuning, reward models and RLHF, with real runs on a small model.

  • Base model to assistant

    The three stages every chat model goes through: pretrain on the web, fine-tune on examples, then learn from feedback.

  • Cross-entropy and perplexity

    The single number a language model is trained to lower, and the easier-to-read number made from it.

  • Benchmarks and evals

    How a benchmark score is really computed, and how sure we can be about it.

  • LLM-as-a-judge

    Using one model to grade another, and the biases that come with it.

  • Pretraining

    Predicting the next token over trillions of tokens. Where a model gets almost everything it knows.

  • Scaling laws

    How loss falls with more parameters and more data, and how to split a fixed compute budget between them.

  • Supervised fine-tuning

    Training on example conversations so the base model answers like an assistant. Only the reply tokens count in the loss.

  • LoRA

    Freeze the model and train two small matrices per layer instead. A few million numbers in place of half a billion.

  • Policy gradient (REINFORCE)

    Make rewarded replies more likely and bad ones less likely. The simplest way to learn from a score.

  • Advantage and GAE

    How much better an action was than expected, estimated with a dial between low noise and low bias.

  • KL penalty

    A leash that keeps the trained model close to where it started, so it cannot drift into nonsense that scores well.

  • Reward models

    A model trained on pairs of answers to score which one people prefer. It stands in for the human during RLHF.

  • Reward overoptimisation

    Push too hard on a learned reward and the real quality stops rising, then falls.

  • PPO clipping

    Reuse each batch several times, but clip the update so the model never moves too far from where the batch came from.

  • RLHF with four models

    The policy, a frozen reference, the reward model and a value model, working together in one training loop.

Interpretability

  • Transformer circuits (paper)

    A 2021 paper that takes small attention-only transformers apart, piece by piece, to see the algorithms inside.

  • Residual stream

    The running total every layer reads from and adds to. Because it is a plain sum, the model can be split into parts.

  • Virtual weights

    Multiply one layer's write matrix by a later layer's read matrix to see how they talk, even with layers in between.

  • Zero-layer model (bigrams)

    Embed then unembed with nothing in between: the model can only learn which token usually follows which.

  • Heads add up independently

    Stacking heads and multiplying by one output matrix is the same as letting each head write on its own and adding the results.

  • QK and OV circuits

    Split a head into where it looks (QK) and what it moves (OV), and read both straight from the weights as token-to-token tables.