Production Serving & Inference 60–75 mins read Advanced • vLLM, PagedAttention & Quantization

LLM Deployment & Inference Optimization: Quantization, KV Cache, Batching & vLLM

Training or fine-tuning a model is only half the battle. When real users hit your endpoint concurrently, naive execution collapses under memory bandwidth starvation, Out-of-Memory (OOM) crashes, and astronomical cloud bills. Learn how to transform raw transformer weights into a lightning-fast, production-grade serving infrastructure using Quantization (INT4/AWQ/GPTQ), KV Cache budgeting, Continuous Batching, PagedAttention, and vLLM.

What You Will Master

Inference Mechanics: The fundamental divide between Compute-bound Prefill and Memory-bandwidth-bound Decode phases.

Latency & Throughput Metrics: Deconstructing Time to First Token (TTFT), Time per Output Token (TPOT), and Aggregate Throughput.

VRAM Budgeting & KV Cache: Exact mathematical formulation of weight storage, KV cache explosion, and multi-tenant memory exhaustion.

Post-Training Quantization: INT8, INT4, FP8, GPTQ, and AWQ precision-loss trade-offs, scaling factors, and hardware kernel acceleration.

Continuous Batching & PagedAttention: Eliminating straggler bubbles and physical memory fragmentation with virtual memory paging.

Production Architecture: High-throughput serving with vLLM, Tensor/Pipeline Parallelism, autoscaling signals, and Server-Sent Events (SSE) streaming.

Introduction: From Trained LLM to Production AI

In the previous lessons, we explored fine-tuning with LoRA/QLoRA, preference alignment via RLHF and DPO, and rigorous multi-dimensional evaluation. Now you possess an aligned, highly accurate model checkpoint.

However, the moment you transition that model to a production environment where hundreds or thousands of concurrent users submit requests simultaneously, you encounter a brutal operational reality:

Latency Spikes: Simple questions take 10+ seconds to respond because the GPU cannot juggle concurrent requests.
VRAM Exhaustion (OOM): Long conversations eat up memory rapidly, crashing the server process with CUDA out of memory errors.
Astronomical Cloud Costs: Renting underutilized A100 or H100 GPUs at $3.00–$5.00/hour drains company runway when throughput sits at a miserable 15 tokens/sec.
Straggler Bottlenecks: When one user requests a 1,000-token summary and another asks for a 5-word answer, traditional batching locks both in lockstep, wasting GPU compute cycles.

Deploying large language models requires a distinct engineering discipline known as Inference Optimization. In this lesson, we will dissect the physics of GPU computation, the mathematics of the Key-Value (KV) cache, state-of-the-art quantization techniques, and modern inference serving engines like vLLM.

What Is LLM Inference & How Does It Differ from Training?

LLM Inference is the forward-pass execution of a pre-trained neural network to generate predictions (tokens) from novel user inputs. While training is about gradient descent and parameter updates, inference is strictly about forward propagation, speed, and cost efficiency.

User Prompt → [ Tokenizer ] → Token IDs → [ LLM Inference Engine (GPU) ] → Predicted Token IDs → [ Detokenizer ] → Response Text

Dimension	Training Phase	Inference Phase
Core Objective	Optimize loss; update model weights via backpropagation.	Generate tokens sequentially with minimal latency and maximal throughput.
Weights State	Dynamic; modified every optimizer step.	Static and frozen (read-only in VRAM).
Gradients & Optimizer	Requires Adam states (8–16 bytes per param) + backward gradients.	Zero gradients. No optimizer states stored in memory.
Memory Footprint	Weights + Gradients + Optimizer States + Activations (~16–20x model size).	Model Weights + KV Cache + Activation buffers (~1.2–2x model size).
Computational Pass	Both Forward Pass and Backward Pass.	Forward Pass only (autoregressive loop).
Critical Engineering Bottleneck	Total compute FLOPS and training run wall-clock days.	Memory Bandwidth (HBM to SRAM transfer) and Serving Cost.

Autoregressive Generation: Why Language Models Are Inherently Sequential

Modern decoder-only foundation models (GPT-4, Llama 3, Mistral, Qwen) generate text autoregressively. Each newly generated token depends conditionally on all prior tokens:

\[ P(x_1, x_2, \dots, x_T) = \prod_{t=1}^T P(x_t \mid x_1, x_2, \dots, x_{t-1}) \]

To emit token $x_t$, the network must evaluate the entire preceding sequence $x_{<t}$. Once $x_t$ is sampled from the softmax vocabulary distribution, it is appended to the context, and the process repeats for $x_{t+1}$ until an End-of-Sequence (<|endoftext|>) token is emitted or maximum tokens is reached.

This creates an inherent architectural consequence: You cannot parallelize the generation of future tokens across time. Token 50 cannot be computed until Token 49 is finished.

Prefill and Decode Phases: The Two Personalities of Inference

Every LLM inference request consists of two fundamentally distinct operational regimes:

Phase 1: Prefill (Prompt Processing)

All prompt tokens $x_1, \dots, x_N$ are known in advance. The model computes the self-attention of the entire prompt simultaneously in parallel using matrix-matrix multiplications (GEMM).

• Hardware Regime: Compute-bound. High Arithmetic Intensity (FLOPS / byte).
• Activity: Saturates Tensor Cores, populates Key-Value Cache for prompt tokens.
• Metric Impact: Governs Time to First Token (TTFT).

Phase 2: Decode (Token-by-Token Generation)

Tokens are emitted one by one. For each single token, the GPU must load all model weights from High Bandwidth Memory (HBM) into on-chip cache just to process a single vector (GEMV).

• Hardware Regime: Memory-bandwidth-bound. Extremely low Arithmetic Intensity.
• Activity: Tensor Cores sit largely idle waiting for memory transfers from HBM.
• Metric Impact: Governs Time per Output Token (TPOT).

The Memory Bandwidth Trap in Decode Phase:

Consider an unquantized 70B parameter model in FP16 (140 GB weights). To generate just one single token for batch size 1, the GPU memory controller must transfer 140 gigabytes across the memory bus! On an A100 GPU with 2.0 TB/s memory bandwidth, reading 140 GB takes $\frac{140}{2000} = 0.07$ seconds (70 ms). Thus, the physical hardware ceiling for generating a single token is roughly 14 tokens/second, regardless of how many TFLOPS of compute power the GPU has!

Critical Serving Metrics: TTFT, TPOT, Latency & Throughput

In traditional REST APIs, performance is measured simply by round-trip latency. In LLM serving, single latency metrics hide critical user experience bottlenecks. Production LLM systems are governed by four distinct metrics:

1. Time to First Token (TTFT) Responsiveness

\[ \text{TTFT} = T_{\text{first\_token} } - T_{\text{request\_received} } \]

The elapsed time between when a client submits a prompt and when the very first token appears on screen. Dictated by network transit, queueing delay, tokenization, and Prefill execution.

2. Time per Output Token (TPOT) Perceived Speed

\[ \text{TPOT} = \frac{T_{\text{last\_token} } - T_{\text{first\_token} } }{N_{\text{output\_tokens} } - 1} \]

The average time required to generate each subsequent token during the Decode phase. Dictated by GPU memory bandwidth and concurrent batch size. Typical human reading speed is ~200 words/min (~5 tokens/sec or TPOT ≤ 200ms).

3. End-to-End Latency Total Duration

\[ T_{\text{total} } = \text{TTFT} + (N_{\text{output} } - 1) \times \text{TPOT} \]

Total wall-clock duration to fulfill the complete request. Critical for non-streaming batch background jobs, document extraction, and offline pipelines.

4. Aggregate Throughput System Capacity

\[ \text{Throughput} = \frac{\sum N_{\text{tokens\_processed} } }{\Delta t} \quad \left(\frac{\text{tokens} }{\text{sec} }\right) \]

The total volume of tokens processed and generated across all simultaneous users per second. Determines how many requests a single server can absorb before needing autoscaling.

The Inherent Latency vs. Throughput Trade-off: As you increase batch size to process more users simultaneously (maximizing aggregate throughput and amortizing weight-loading overhead), each individual user's TPOT deteriorates due to VRAM contention and larger matrix dimensions. Engineering production inference is the art of tuning batch size to satisfy your target Service Level Agreement (SLA) (e.g., TTFT < 500ms, TPOT < 50ms).

GPU Memory Architecture: Estimating Model Weight VRAM

To avoid Out-of-Memory (OOM) failures, serving engineers must calculate precise VRAM allocations before deploying. The raw memory required to store static model parameters in GPU memory is:

\[ M_{\text{weights} } = P \times \frac{B}{8} \]

Where $P$ is the parameter count and $B$ is the precision bits per parameter.

Data Type / Format	Bits / Element	Bytes / Param	7B Model Weights	70B Model Weights
FP32 (Single Precision)	32 bits	4.0 bytes	28.0 GB	280.0 GB
FP16 / BF16 (Half Precision)	16 bits	2.0 bytes	14.0 GB	140.0 GB
INT8 / FP8 (8-bit Quantized)	8 bits	1.0 byte	7.0 GB	70.0 GB
INT4 (AWQ / GPTQ)	4 bits	0.5 bytes	3.5 GB	35.0 GB

Runtime Overhead Rule: In production, you must budget an extra 15–20% overhead on top of raw weights to account for CUDA context buffers, PyTorch/driver allocations, and activation tensor workspace.

Quantization: Compressing Precision to Unlock Hardware Acceleration

Quantization maps continuous floating-point weights ($x \in \mathbb{R}$) into lower-precision discrete integer or micro-float bins. It directly slashes the memory bandwidth footprint during the memory-bound Decode phase!

\[ q = \operatorname{clip}\left( \operatorname{round}\left( \frac{x}{s} \right), q_{\min}, q_{\max} \right), \quad x \approx s \cdot q \]

Where $s$ is a dynamically calculated scaling factor: $s = \frac{\max(|x|)}{q_{\max} }$.

FP8 (Floating Point 8)

Native hardware support on NVIDIA Ada Lovelace (RTX 4090) and Hopper (H100) architectures. Employs E4M3 or E5M2 exponent-mantissa splits. Preserves near 100% of FP16 accuracy with 2x memory reduction and 2x faster Tensor Core GEMM.

GPTQ (Post-Training Quantization)

A second-order error minimization technique based on the Optimal Brain Surgeon algorithm. Quantizes layer by layer using inverse Hessian matrices. Compresses weights to 4-bit while dequantizing on the fly in GPU registers during forward passes.

AWQ (Activation-aware Weight Quantization)

Recognizes that not all weights are equally vital: protecting the top 1% of salient weights that correspond to large activation magnitudes prevents perplexity degradation. Highly resilient and dominant for 4-bit vLLM serving.

KV Cache Optimization: Why Key-Value Memory Becomes the Real Bottleneck

In multi-head attention, every token projects into Query ($Q$), Key ($K$), and Value ($V$) vectors. To calculate attention between the new token and all preceding tokens:

\[ \text{Attention}(Q, K, V) = \operatorname{softmax}\left( \frac{Q K^T}{\sqrt{d_k} } \right) V \]

Without caching, generating token 1,001 would require recomputing the $K$ and $V$ matrices for tokens 1 through 1,000 from scratch. This would cause $O(N^2)$ cumulative compute complexity!

The KV Cache stores the computed Key and Value vectors for all past tokens in GPU memory. When generating token $t$, the model only computes $Q_t, K_t, V_t$, appends $K_t, V_t$ to the cache, and computes attention against the cached history.

Exact KV Cache Memory Formula:

\[ M_{\text{KV} } = 2 \times L \times B \times S \times H_{\text{KV} } \times D \times b \]

• $2$: Key and Value vectors

• $L$: Number of layers

• $B$: Concurrent batch size

• $S$: Sequence context length

• $H_{\text{KV} }$: Key-Value heads

• $D$: Head dimension ($d / H$)

• $b$: Bytes per element (FP16 = 2)

The Multi-Tenant VRAM Explosion:

Consider Llama-3-8B ($L=32, H_{\text{KV} }=8, D=128, b=2$). For 1 request at 4,096 tokens, KV cache is ~0.5 GB. But if 64 concurrent users are chatting at 8,192 context length:

\[ M_{\text{KV} } = 2 \times 32 \times 64 \times 8192 \times 8 \times 128 \times 2 = 68,719,476,736 \text{ bytes} \approx 64.0 \text{ GiB}! \]

The KV cache alone exceeds the entire memory capacity of an NVIDIA A100-40GB GPU! This is why naive LLM deployment crashes with OOM under load.

Static Batching vs. Continuous Batching: Eliminating the Straggler Problem

In traditional machine learning, models employ Static Batching: a batch of $B$ requests is assembled, padded to the longest sequence, and executed together until all requests finish.

// The Straggler Effect in Static Batching

Request A (Short): [== 20 tokens ==] [ ....... IDLE STRAGGLER BUBBLE (WASTED GPU) ....... ] Done

Request B (Long): [=========================== 600 tokens ===========================] Done

In Static Batching, Request A finishes after 20 tokens, but its slot cannot be reused until Request B completes 600 tokens! The GPU wastes memory and compute on padding tokens.

Continuous Batching (Iteration-Level Scheduling): Pioneered by Orca and standardized by vLLM. Instead of scheduling at the batch level, scheduling occurs at each iteration step. As soon as Request A emits an EOS token at step 20, it is immediately evicted from the batch, and a waiting request from the queue enters the batch at step 21 to utilize the compute resources.

Continuous batching increases aggregate serving throughput by 2.5x to 4x over static batching!

PagedAttention: Virtual Memory Management for Large Language Models

Continuous batching solved scheduling, but a severe memory problem remained: KV Cache Fragmentation.

Traditional serving engines pre-allocate contiguous VRAM chunks for the maximum possible sequence length (e.g., 4,096 tokens) for each request. Because most requests only use a fraction of their allocated capacity, 60% to 80% of GPU memory is wasted on internal and external fragmentation.

PagedAttention (vLLM): Inspired by virtual memory and paging in Operating Systems. PagedAttention divides each sequence's KV cache into fixed-size physical blocks (e.g., 16 tokens per block).

How PagedAttention Eliminates Memory Waste:

Non-contiguous Allocation: Logical sequence blocks are mapped to non-contiguous physical memory blocks across GPU VRAM via a Block Table.
On-Demand Allocation: A new physical block is allocated only when the current block is completely filled with 16 tokens.
Near-Zero Fragmentation: Memory waste is limited strictly to the tail of the final block (< 4% waste).
Memory Sharing (Parallel Sampling & Prefix Caching): Multiple requests sharing common system prompts or branching outputs share the identical physical memory blocks in VRAM without duplicating weights!

The Modern Serving Engine Landscape: vLLM, TGI, TensorRT-LLM & llama.cpp

Serving Engine	Primary Target	Key Strengths	Best Fit Use Case
vLLM	Multi-tenant Cloud GPUs (NVIDIA/AMD)	PagedAttention, continuous batching, chunked prefill, OpenAI API drop-in compatibility.	Production web services, high-concurrency APIs, general serving.
TensorRT-LLM	NVIDIA Enterprise GPUs (A100, H100)	Deep kernel fusion, in-flight batching, FP8 hardware optimization.	Ultra-low latency mission-critical enterprise environments.
Hugging Face TGI	Cloud Kubernetes Clusters	Safetensors loading, FlashAttention-2, token streaming, Prometheus metrics.	Cloud deployments tightly integrated with Hugging Face Hub.
llama.cpp	Edge Devices, Apple Silicon & CPUs	Pure C/C++ with zero dependencies, GGUF quantization (2-bit to 8-bit), Metal/AVX2 acceleration.	On-premise edge hardware, local development, consumer laptops.
SGLang	Complex Structured Generation	RadixAttention (deep automatic prefix caching across multi-turn calls).	Autonomous agent workflows, few-shot prompting, JSON schemas.

Scaling Strategy: Model Replication vs. Model Sharding

When workload exceeds a single GPU's capacity, you must decide whether to scale horizontally via replication or vertically via parallel sharding:

1. Model Replication (Data Parallelism)

Run complete copies of the entire model across independent GPUs or nodes. Incoming requests are distributed via round-robin or least-connections load balancers.

• When to use: When the model fits entirely inside 1 GPU (e.g., 7B model in 24 GB VRAM).
• Advantage: Zero inter-GPU communication latency; linear aggregate throughput scaling.

2. Model Sharding (Tensor / Pipeline Parallelism)

Split the weights of a single model across multiple GPUs connected via high-speed NVLink interconnects.

• Tensor Parallelism (TP): Slices weight matrices across GPUs (e.g., Megatron-LM); requires high-bandwidth All-Reduce per layer.
• Pipeline Parallelism (PP): Divides transformer layers sequentially across GPUs.
• When to use: When model parameters exceed single-GPU VRAM (e.g., 70B in FP16 requires 4x or 8x A100s).

Production Serving with vLLM: Setup & OpenAI Compatibility

Deploying an OpenAI-compatible HTTP serving endpoint with vLLM requires a single production-hardened CLI command:

# Start production vLLM OpenAI-compatible server vllm serve Qwen/Qwen2.5-7B-Instruct \ --host 0.0.0.0 \ --port 8000 \ --tensor-parallel-size 1 \ --max-model-len 8192 \ --gpu-memory-utilization 0.90 \ --quantization awq \ --api-key production_secret_token

Clients interact with the server using standard OpenAI client libraries with streaming enabled:

from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="production_secret_token" ) # Low-latency streaming chat completion response_stream = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct", messages=[ {"role": "system", "content": "You are an expert AI deployment engineer."}, {"role": "user", "content": "Explain why Decode phase is memory-bandwidth bound."} ], temperature=0.7, max_tokens=300, stream=True ) for chunk in response_stream: token = chunk.choices[0].delta.content if token: print(token, end="", flush=True)

Production Observability, Autoscaling & Cost Modeling

In cloud infrastructure, idle GPUs incinerate capital. Serving costs are calculated as:

\[ \text{Cost} = G \times H \times R \]

Where $G$ is GPU count, $H$ is operational hours, and $R$ is hourly cloud rate ($/GPU-hr).

Four Essential Production Telemetry Signals:

1. Pending Queue Length Requests waiting in gateway queue. Direct indicator of saturation and the primary signal to trigger auto-scale out.

2. P95 & P99 TTFT Percentile tail latency for prompt processing. Spikes indicate queueing congestion or oversized prompt payloads.

3. KV Cache Usage % Active physical blocks in PagedAttention. When > 90%, preemption or request rejection begins.

4. Token Generation Rate Aggregate output tokens/sec per GPU. Validates that GPUs are operating near memory bandwidth capacity.

Top-10 Production LLM Deployment Anti-Patterns

Anti-Pattern	Dangerous Consequence	Production Remedy
1. Ignoring KV Cache Budget	Server crashes with sudden OOM when users submit long prompts.	Pre-calculate KV cache limits; set `--max-model-len` and strict token bounds.
2. Using Raw Hugging Face Pipeline	Sequential execution wastes 80% of GPU compute and memory bandwidth.	Deploy dedicated high-performance serving engines like vLLM or TGI.
3. Assuming Quantization Always Accelerates	Quantized models on unsupported GPUs run slower due to runtime dequantization overhead.	Benchmark AWQ/GPTQ kernels on your exact hardware target before shipping.
4. Omitting Streaming for Web UIs	Users perceive 8-second total latency as a broken or frozen system.	Enable Server-Sent Events (SSE) streaming for real-time token rendering.
5. Oversizing Static Batches	Fast requests stall behind massive stragglers, degrading TTFT.	Adopt Continuous Batching with iteration-level request eviction.
6. Relying on Arithmetic Mean Latency	Hides severe 15-second P99 tail latency stalls that alienate enterprise users.	Monitor P50, P95, and P99 percentiles for TTFT and TPOT independently.
7. Autoscaling Solely on CPU/GPU %	Cold model loading delays spin-up by 2–5 minutes, missing traffic spikes.	Autoscale on pending queue depth and active sequence counts.
8. Missing Admission Control	Traffic surges cause cascading timeouts and server-wide collapse.	Implement API gateway rate limits, concurrency gates, and fast HTTP 429 shedding.
9. Unbounded Maximum Output Tokens	Runaway hallucinated loops exhaust VRAM and tie up worker threads.	Enforce strict `max_tokens` ceilings on all completion requests.
10. Skipping Quantization Accuracy Audits	Silent degradation in math reasoning, formatting, or Bengali generation.	Run automated evaluation benchmarks (MMLU, GSM8K) against the quantized checkpoint.

Interactive Serving & Inference Optimization Studio

Explore real-time simulations of GPU VRAM budgeting, Prefill vs Decode latency dynamics, Continuous Batching scheduling, and Production Autoscaling economics.

Hardware & Model Configuration

Model Parameters:

Weight Precision:

Concurrent Users (Batch Size): 16

Context Length (Tokens): 4,096

Target GPU Hardware:

Model Weights: 4.0 GB

KV Cache Allocation: 8.0 GB

Total VRAM Demand: 13.5 GB

VRAM Headroom: +10.5 GB Free

// GPU VRAM Memory Allocation Breakdown Weights vs KV Cache vs Headroom

Coming Up: Lesson 15.41 — LLM Production Architecture: RAG, Caching, Guardrails & AI Gateway

Now that your inference engine is blazing fast and memory-optimized, you must wrap it in a resilient production enterprise architecture. In Lesson 15.41, we will explore multi-tier Semantic Caching, production RAG retrieval pipelines, Guardrails (input/output sanitization), automated fallback routing, rate-limiting AI Gateways, and end-to-end distributed LLM tracing.

Previous: Lesson 15.39 (LLM Evaluation & Benchmarking) Next: Lesson 15.41 (LLM Production Architecture)