Learning track
Inference & Efficiency
Understand how trained models generate text and how decoding choices trade off diversity, coherence, latency, and cost.
advanced · 12 available lessons
Inference & Efficiency
Understand how trained models generate text and how decoding choices trade off diversity, coherence, latency, and cost.
- What actually happens when you hit send
- The KV cache
- Prefill vs decode: two different machines
- Sampling: temperature, top-k, top-p, and min-p
- Beam search, speculative decoding & Medusa
- MQA, GQA & MLA
- FlashAttention & IO-awareness
- PagedAttention & continuous batching
- Quantization I: int8, int4, the basics
- Quantization II: GPTQ, AWQ, GGUF, QAT
- Serving stacks: vLLM, SGLang, TensorRT-LLM
- Running models locally