Learn Inference Engineering

Introduces the three layers of inference: runtime, infrastructure, and tooling. A map of the entire book.

Use case definition, latency and cost budgeting, model selection and evaluation, and fine-tuning for quality.

Technical architecture of LLMs and diffusion models — transformers, attention, MoE, and inference bottlenecks.

GPU architecture, compute and memory, NVIDIA generations (Hopper, Blackwell, Rubin), instances, and alternatives.

CUDA kernels, PyTorch, model formats, inference engines (vLLM, SGLang, TensorRT-LLM), NVIDIA Dynamo, and benchmarking.

Quantization, speculative decoding, KV cache re-use, model parallelism, and disaggregation in practice.

Vision language models, embeddings, ASR, TTS, image generation, and video generation inference.

Containerization, autoscaling, multi-cloud, deployment, observability, and client code for production inference.