llmdeepdive
PT

Type to search

    Curriculum

    Learning tracks

    A connected path from first principles to modern language-model systems.

    01

    foundations · 8 available lessons

    Orientation & Foundations

    Build an accurate mental model of language models and the mathematics needed for everything that follows.

    Open track
    1. 01.01What is a language model, really?
    2. 01.02A 60-second history: n-grams to the GPT-5 era
    3. 01.03Probability you actually need
    4. 01.04Vectors, matrices, and dot products
    5. 01.05Matrix multiplication as transformation
    6. 01.06Derivatives and gradients, geometrically
    7. 01.07Python, NumPy, and PyTorch in 20 minutes
    8. 01.08How to read an ML paper
    02

    foundations · 7 available lessons

    From Text to Tensors

    See how text becomes tokens, vectors, and numerical structures a neural network can process.

    Open track
    1. 02.01Why computers can't read
    2. 02.02Tokenization I: characters, words, and subwords
    3. 02.03Tokenization II: BPE, step by step
    4. 02.04Tokenization III: SentencePiece, byte-level, and why token counts bite
    5. 02.05Embeddings: meaning as geometry
    6. 02.06word2vec, GloVe, and the analogy trick
    7. 02.07Entropy, perplexity, and what 'good' means
    03

    foundations · 9 available lessons

    Neural Network Fundamentals

    Learn how neural networks represent functions, measure error, propagate gradients, and improve through optimization.

    Open track
    1. 03.01The perceptron
    2. 03.02Multilayer perceptrons and non-linearity
    3. 03.03Activation functions: ReLU, GELU, and SwiGLU
    4. 03.04Loss functions and cross-entropy
    5. 03.05Backpropagation, visualized
    6. 03.06Gradient descent and the loss landscape
    7. 03.07Optimizers: SGD to Momentum to Adam to AdamW
    8. 03.08Initialization, normalization, and residuals
    9. 03.09Overfitting, regularization, and the bitter lesson
    04

    core · 5 available lessons

    Sequence Models

    Follow the path from recurrent state to encoder-decoder models and attention, the immediate ancestors of the Transformer.

    Open track
    1. 04.01Modelling sequences: the setup
    2. 04.02RNNs and the vanishing gradient
    3. 04.03LSTM and GRU: gates that remember
    4. 04.04Seq2seq, encoder-decoder, and the bottleneck
    5. 04.05Bahdanau attention: the idea that changed everything
    05

    core · 14 available lessons

    The Transformer

    Assemble the architecture behind modern LLMs, beginning with self-attention and its query, key, and value operations.

    Open track
    1. 05.01"Attention Is All You Need" in context
    2. 05.02Self-attention from first principles: Q, K, V
    3. 05.03Scaled dot-product attention & the √d_k
    4. 05.04Causal masking & why order matters
    5. 05.05Multi-head attention
    6. 05.06Positional encoding I: sinusoidal & learned
    7. 05.07Positional encoding II: RoPE
    8. 05.08Positional encoding III: ALiBi & relative bias
    9. 05.09The feed-forward block & where knowledge lives
    10. 05.10Residuals, pre-norm vs post-norm
    11. 05.11The full block, assembled
    12. 05.12Encoder-only, decoder-only, encoder–decoder
    13. 05.13Build a GPT from scratch, annotated
    14. 05.14Reading real weights: what a trained model looks like
    06

    advanced · 13 available lessons

    Pretraining at Scale

    Design the objectives, data, optimization, parallelism, and cost model behind a serious language-model pretraining run.

    Open track
    1. 06.01Objectives: causal LM, MLM, span corruption, FIM
    2. 06.02Data: sourcing, filtering, dedup, decontamination
    3. 06.03Training your own tokenizer
    4. 06.04Scaling laws I: Kaplan
    5. 06.05Scaling laws II: Chinchilla & compute-optimality
    6. 06.06Scaling laws III: inference-aware & over-training
    7. 06.07Data parallelism, ZeRO & FSDP
    8. 06.08Tensor, pipeline & sequence parallelism
    9. 06.09Mixed precision: fp16, bf16, fp8
    10. 06.10Gradient checkpointing & memory maths
    11. 06.11Learning-rate schedules & warmup
    12. 06.12When training goes wrong: spikes, divergence, NaNs
    13. 06.13What a real pretraining run costs
    07

    advanced · 11 available lessons

    Post-training & Alignment

    Turn pretrained predictors into useful assistants through instruction tuning, preference learning, reinforcement learning, and distillation.

    Open track
    1. 07.01Why base models aren't assistants
    2. 07.02Supervised fine-tuning & instruction data
    3. 07.03LoRA, QLoRA & PEFT
    4. 07.04Reward models & human preference data
    5. 07.05RLHF with PPO
    6. 07.06DPO: skipping the reward model
    7. 07.07ORPO, KTO, SimPO & the alignment zoo
    8. 07.08GRPO & RL on verifiable rewards
    9. 07.09Reasoning models: test-time compute & long CoT
    10. 07.10Constitutional AI & RLAIF
    11. 07.11Distillation: making small models punch up
    08

    advanced · 12 available lessons

    Inference & Efficiency

    Understand how trained models generate text and how decoding choices trade off diversity, coherence, latency, and cost.

    Open track
    1. 08.01What actually happens when you hit send
    2. 08.02The KV cache
    3. 08.03Prefill vs decode: two different machines
    4. 08.04Sampling: temperature, top-k, top-p, and min-p
    5. 08.05Beam search, speculative decoding & Medusa
    6. 08.06MQA, GQA & MLA
    7. 08.07FlashAttention & IO-awareness
    8. 08.08PagedAttention & continuous batching
    9. 08.09Quantization I: int8, int4, the basics
    10. 08.10Quantization II: GPTQ, AWQ, GGUF, QAT
    11. 08.11Serving stacks: vLLM, SGLang, TensorRT-LLM
    12. 08.12Running models locally