llmdeepdive
PT

Type to search

    Learning track

    The Transformer

    Assemble the architecture behind modern LLMs, beginning with self-attention and its query, key, and value operations.

    05

    core · 14 available lessons

    The Transformer

    Assemble the architecture behind modern LLMs, beginning with self-attention and its query, key, and value operations.

    1. 05.01"Attention Is All You Need" in context
    2. 05.02Self-attention from first principles: Q, K, V
    3. 05.03Scaled dot-product attention & the √d_k
    4. 05.04Causal masking & why order matters
    5. 05.05Multi-head attention
    6. 05.06Positional encoding I: sinusoidal & learned
    7. 05.07Positional encoding II: RoPE
    8. 05.08Positional encoding III: ALiBi & relative bias
    9. 05.09The feed-forward block & where knowledge lives
    10. 05.10Residuals, pre-norm vs post-norm
    11. 05.11The full block, assembled
    12. 05.12Encoder-only, decoder-only, encoder–decoder
    13. 05.13Build a GPT from scratch, annotated
    14. 05.14Reading real weights: what a trained model looks like