llmdeepdive

Type to search

    The anatomy explorer

    Anatomy of an LLM

    A transformer block, drawn to scale and taken apart. Click any marker to open a component — or use the arrow keys.

    Diagram of a transformer block, from embeddings up to the sampler

    3D SPECIMEN · CLICK A DOT TO EXPLORE

    Input encoding

    Tokenizer

    Text into integers

    Splits text into subword units and maps each to an integer. Every downstream cost — context length, price per call, throughput — is denominated in these units.

    Key facts

    Parameters
    None (a lookup table, not learned weights)
    Cost
    Negligible at inference
    Introduced
    BPE: Sennrich et al., 2016
    Variants
    BPE · SentencePiece · byte-level BPE · Unigram
    View lesson

    The maths

    Across architectures

    A token's journey

    Failure modes

    Where it sits