Topic map
Attention and Transformers
- Self-attention
Every token looks at every other token and mixes in what it finds. The one idea everything else here builds on.
- Multi-head attention
Several attention heads run in parallel, each looking for a different kind of relation.
- MQA, GQA, MLA
Ways to share keys and values across heads so the KV cache gets smaller.
- Sliding-window and sparse attention
Let each token look only at a window or a pattern of others, to cut the quadratic cost.
- Linear attention and hybrids
Attention rewritten so its cost grows linearly, and models that mix it with the standard kind.
- The Transformer: encoder and decoder
The 2017 design: an encoder that reads and a decoder that writes, built only from attention and small MLPs.
- Position embeddings
Attention has no sense of order, so a learned vector per position is added to every token.
Pre-training (BERT)
- BERT
A Transformer encoder that reads text in both directions, pre-trained once and fine-tuned for any task.
- Masked language modelling
Hide 15% of the words and train the model to guess them from both sides.
- Pre-train, then fine-tune
Train once on a huge dataset, then adapt a copy to each task with one small new layer.
- The [CLS] / [class] token
An extra token whose output vector stands for the whole input. BERT invented it; ViT reused it.
Vision (ViT)
- Vision Transformer (ViT)
Cut a picture into 16×16 patches, treat them as words, and feed them to an unchanged Transformer encoder.
- Images as patches
A 224×224 picture becomes 196 patches of 768 numbers each: the image's tokens.
- Inductive bias versus data
Built-in assumptions help with little data; with 300 million images, learning from data wins.
- Convolutional networks (ResNet)
The design that ruled computer vision from 2012 to 2020: small filters slid across the picture, with residual shortcuts.
LLM inference
- Prefill and decode
How an LLM answers: read the whole prompt at once, then write one token at a time.
- KV cache
The keys and values of every past token, kept in memory so they are not recomputed. It is the biggest memory bill.
- vLLM: paging and batching
Serving thousands of requests by paging the KV cache like an operating system and batching continuously.
- Speculative decoding
A small model drafts several tokens; the big model checks them in one step. Same words, fewer steps.
- Prefix caching
Requests that start the same way reuse the saved KV cache, so the shared start is not computed again.
- SGLang and RadixAttention
A radix tree of cached prefixes, compared step by step with vLLM's hashed blocks.
- Cache-aware scheduling
Serving requests that share a cached start together raises reuse, at the risk of making others wait.
- Faster structured output
When the JSON format fixes the next tokens, write them at once instead of one step at a time.
- Tensor parallelism
Split every layer across GPUs, columns then rows: two all-reduces per layer, checked on a real two-process run.
- All-reduce and other collectives
All-reduce, all-gather, reduce-scatter and all-to-all, the ring traced step by step, and its cost formula.
- Pipeline parallelism
Give each GPU a slice of the layers: little traffic across slow links, more throughput, no lower latency.
- Expert parallelism (MoE)
Spread experts over GPUs with all-to-all, measured routing imbalance, and redundant hot experts as in DeepSeek-V3.
- Disaggregated prefill and decode
Run the two phases on different GPUs and move the KV cache between them: Splitwise, DistServe, Mooncake, and when it pays.
- Latency SLOs and goodput
TTFT, ITL and end-to-end latency as percentiles, and goodput: the requests per second served inside the SLO.
- Queueing and the latency cliff
Little's law and why waiting explodes near full load, checked on a simulated batching engine.
- Capacity planning and cost per token
From model size, KV bytes and traffic to a number of GPUs, and why a generated token costs more than a prompt token.
- Quantization for serving
INT8, INT4, FP8 and an FP8 KV cache: memory and speed gained, quality lost, measured on a real model.
- Routing across replicas
Sending a request to the replica that holds its prefix, without overloading it.
- Multi-LoRA serving
Many fine-tuned adapters on one base model, batched together in one step.
- Autoscaling and failure modes
Cold starts, retry storms, preemption storms and noisy neighbours, and the metrics that reveal them.
Retrieval and RAG
- RAG
Retrieve the right passages first, then let the model answer from them. A full book, from first principles.
- Embeddings
Meaning as geometry: texts become vectors, and similar meanings land close together.
- Vector search and HNSW
Finding the nearest vectors among millions fast, with a layered graph of neighbours.
- BM25 and hybrid search
Keyword scores and vector scores, fused, beat either alone.
- Rerankers
A second, slower model (often BERT-like) re-orders the top results more carefully.
- Evaluating RAG
Every retrieval and generation metric, and how to measure a pipeline honestly.
Agents
- LangGraph and agentic RAG
RAG as a graph of steps that can loop, branch and call tools.
- Multi-agent systems and guardrails
Several agents working together, and the checks that keep them in bounds.
- Jev: a model that decides
A video on an AI model that does not talk, it decides.
- Agentic Systems (book)
A book on agent patterns, evals, tracing and guardrails, with real runs and what companies learned in production.
- What an agent is
An LLM in a loop that picks tools and acts until the task is done, and when a plain workflow is the better choice.
- Model, tools, memory and state
The five parts every agent is made of, and how they connect.
- Context engineering
Choosing what goes into the model's window at each step, and what stays out.
- ReAct and reasoning loops
Think, act, look at the result, repeat. Measured with an exact grader and an ablation of what really helps.
- Reflection and verifiers
Self-review rarely fixes answers; a real check, such as a test suite, does.
- Chaining, routing and parallel calls
Fixed workflows where your code holds the control flow: prompt chains, routers and fan-out.
- Orchestrators and evaluator loops
One model splits the work for others, and a checker loop refines an answer until it passes.
GPUs and hardware
- Why GPUs exist
Thousands of simple cores doing the same arithmetic at once: exactly what matrix multiplication needs.
- The CUDA programming model
Threads, blocks and grids: how work is handed to a GPU.
- CPU parallelism
What a CPU can and cannot do in parallel, and why that is not enough.
- NVLink, PCIe and InfiniBand
How fast GPUs talk inside a server and between servers, and why a small message costs a fixed delay.
Training LLMs
- How models are trained
A book on how a chatbot is made: pretraining, fine-tuning, reward models and RLHF, with real runs on a small model.
- Base model to assistant
The three stages every chat model goes through: pretrain on the web, fine-tune on examples, then learn from feedback.
- Cross-entropy and perplexity
The single number a language model is trained to lower, and the easier-to-read number made from it.
- Benchmarks and evals
How a benchmark score is really computed, and how sure we can be about it.
- LLM-as-a-judge
Using one model to grade another, and the biases that come with it.
- Pretraining
Predicting the next token over trillions of tokens. Where a model gets almost everything it knows.
- Scaling laws
How loss falls with more parameters and more data, and how to split a fixed compute budget between them.
- Supervised fine-tuning
Training on example conversations so the base model answers like an assistant. Only the reply tokens count in the loss.
- LoRA
Freeze the model and train two small matrices per layer instead. A few million numbers in place of half a billion.
- Policy gradient (REINFORCE)
Make rewarded replies more likely and bad ones less likely. The simplest way to learn from a score.
- Advantage and GAE
How much better an action was than expected, estimated with a dial between low noise and low bias.
- KL penalty
A leash that keeps the trained model close to where it started, so it cannot drift into nonsense that scores well.
- Reward models
A model trained on pairs of answers to score which one people prefer. It stands in for the human during RLHF.
- Reward overoptimisation
Push too hard on a learned reward and the real quality stops rising, then falls.
- PPO clipping
Reuse each batch several times, but clip the update so the model never moves too far from where the batch came from.
- RLHF with four models
The policy, a frozen reference, the reward model and a value model, working together in one training loop.
Interpretability
- Transformer circuits (paper)
A 2021 paper that takes small attention-only transformers apart, piece by piece, to see the algorithms inside.
- Residual stream
The running total every layer reads from and adds to. Because it is a plain sum, the model can be split into parts.
- Virtual weights
Multiply one layer's write matrix by a later layer's read matrix to see how they talk, even with layers in between.
- Zero-layer model (bigrams)
Embed then unembed with nothing in between: the model can only learn which token usually follows which.
- Heads add up independently
Stacking heads and multiplying by one output matrix is the same as letting each head write on its own and adding the results.
- QK and OV circuits
Split a head into where it looks (QK) and what it moves (OV), and read both straight from the weights as token-to-token tables.