Lesson 15.54: AI Infrastructure & Scaling — GPU Architecture, Distributed Inference, Model Serving, Autoscaling & Cost Stacks

1 Beyond the Model: The 10,000-User Scaling Shock

In previous lessons, we mastered fine-tuning LLMs, building autonomous multi-agent loops, constructing enterprise RAG graphs, evaluating guardrails, and quantizing weights. However, a catastrophic reality awaits engineering teams when transitioning from local prototyping to commercial production: A functional neural model is not a scalable software product.

The 1-User Development Sandbox

A single researcher runs a Jupyter notebook or simple FastAPI wrapper. A single prompt arrives, passes into PyTorch via Hugging Face Transformers, runs a forward pass, and outputs streaming text in 1.2 seconds. Everything seems functional, simple, and stable.

The 10,000-User Production Reality

10,000 concurrent enterprise users dispatch heterogeneous requests: short 10-token queries, 32k-token legal briefs, real-time code autocomplete, and multimodal documents. Without dedicated infrastructure, GPU VRAM explodes with CUDA Out-Of-Memory (OOM) errors, latency skyrockets from 800ms to 45 seconds, and monthly cloud GPU bills spiral into hundreds of thousands of dollars.

Senior AI Infrastructure Engineers must solve questions that have nothing to do with training loss or prompt wording, but everything to do with distributed systems and hardware physics:

Hardware Sizing: How many A100s, H100s, or L40Ss are needed for 50,000 peak tokens/sec?
Memory Allocation: How much VRAM is allocated to static weights versus dynamic Key-Value (KV) cache?
Scheduling & Batching: How can incoming requests be continuously merged without penalizing in-flight generation?
Resilience & Failover: How does the system handle sudden 10x traffic spikes, GPU hardware crashes, and worker OOMs without dropping user state?

2 Modern AI Infrastructure Blueprint & Layered Architecture

A production AI inference infrastructure is a multi-tier distributed system decoupled across control-plane and data-plane components. The system separates general CPU network orchestration from specialized GPU matrix acceleration.

Figure 55.1: End-to-End Enterprise AI Infrastructure Topology Production Blueprint

flowchart TD
    Users["Client Applications & SDKs"] --> Cloudflare["CDN / Edge API Gateway (WAF, TLS)"]
    Cloudflare --> AuthRate["Auth, Tenant Isolation & Token Bucket Rate Limiter"]
    AuthRate --> AIGateway["AI Semantic Gateway & Observability Tracer"]
    
    subgraph RoutingAndCaching["Routing & Caching Tier"]
        AIGateway --> RespCache{"Exact / Semantic Cache Hit?"}
        RespCache -- Hit --> CachedOut["Instant Response (< 25ms, $0 GPU cost)"]
        RespCache -- Miss --> AIRouter["AI Model & Semantic Complexity Router"]
    end

    subgraph QueuesAndAdmission["Buffer & Admission Control"]
        AIRouter --> PriorityQ["Priority Request Queue (Redis / NATS JetStream)"]
        PriorityQ --> AdmissionCtrl["Admission Controller & Load Shedding (Backpressure)"]
    end

    subgraph WorkerPools["GPU Compute Clusters (Autoscaling Pools)"]
        AdmissionCtrl --> LBPool["Load Balancer (Latency & KV-aware)"]
        LBPool --> SmallPool["SLM Pool (8B Models / 2x L4 GPUs)"]
        LBPool --> MidPool["Mid-Tier Pool (70B Models / 4x A100-80G TP=4)"]
        LBPool --> LargePool["Flagship Pool (405B Models / 8x H100-SXM TP=8)"]
    end

    subgraph ServingEngine["Model Serving Engines (vLLM / TensorRT-LLM)"]
        SmallPool & MidPool & LargePool --> Scheduler["Continuous Iteration Scheduler"]
        Scheduler --> PagedAttn["PagedAttention KV Cache Memory Manager"]
        PagedAttn --> EngineOut["SSE Streaming Token Pipeline"]
    end

    EngineOut --> StreamOut["Response Stream to Clients"]

Infrastructure Tier	Core Function	Key Technologies	Primary Performance Target
Edge & Gateway	TLS termination, WAF, tenant authentication, rate limiting	Kong, Envoy, Cloudflare Workers, FastAPI	< 5ms overhead, 100k+ concurrent connections
Semantic Cache	Eliminating redundant inference on exact/semantic prompts	Redis Stack, Qdrant, Milvus, GPTCache	> 25% cache hit rate, < 20ms lookup latency
Buffering & Queuing	Spike absorption, priority tiering (Free vs Enterprise), backpressure	Redis Streams, RabbitMQ, Kafka, NATS	Zero message loss, instant admission rejection under load
Model Serving Engine	Continuous batching, PagedAttention, KV-cache management, token streaming	vLLM, TensorRT-LLM, Triton Inference Server, TGI	Max throughput (tok/s/GPU) under P95 TTFT SLO
Autoscaling Controller	Dynamic node provisioning based on queue depth & in-flight tokens	KEDA, Kubernetes HPA, Karpenter, Ray Serve	Zero flapping, warm standby spin-up in < 90s

3 CPU vs GPU Architecture & The Physics of Parallel Matrix Compute

To build high-performance AI infrastructure, engineers must understand why Central Processing Units (CPUs) falter at deep learning inference and how Graphics Processing Units (GPUs) fundamentally operate.

Modern CPU: Latency-Optimized Architecture

Modern CPUs (e.g., AMD EPYC 9654, Intel Xeon Platinum) feature between 16 and 96 complex, powerful physical cores. Each core is engineered to minimize instruction latency on complex branchy execution paths.

• Massive Out-of-Order (OoO) speculative execution engines.
• Multi-level hierarchical L1/L2/L3 caches (up to 384 MB).
• Sophisticated branch prediction units for operating system tasks, networking, and serialization.
• DRAM Bandwidth: DDR5 achieves ~300–460 GB/s across 8 channels.

Modern GPU: Throughput-Optimized Architecture

Modern datacenter GPUs (e.g., NVIDIA H100 SXM, A100, B200) contain tens of thousands of lightweight Arithmetic Logic Units (ALUs) grouped into Streaming Multiprocessors (SMs).

• 132–144 Streaming Multiprocessors with specialized 4th Gen Tensor Cores.
• Hardware Single Instruction Multiple Threads (SIMT) scheduling.
• Massive warp schedulers hiding memory access latency through thread-level parallelism.
• HBM3 Memory Bandwidth: High-Bandwidth Memory (HBM) delivers up to 3,350 GB/s (3.35 TB/s).

The Fundamental Transformer Operation: General Matrix Multiply (GEMM)

During autoregressive Transformer inference, computing Self-Attention and Multi-Layer Perceptron (MLP) feed-forward projections boils down to massive matrix multiplications:

$$Y = \text{Activation}(X \cdot W^T)$$

Where $X \in \mathbb{R}^{B \times d_{\text{model}}}$ represents the batch of token representations, and $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{model}}}$ represents the weight matrix. While a CPU must execute nested vector loops, a GPU distributes the matrix tiles across thousands of Tensor Cores concurrently, executing billions of Multiply-Accumulate (MAC) operations in single clock cycles.

4 Rigorous GPU VRAM Planning: Weights, KV Cache & Memory Bandwidth

A common junior engineering mistake is believing that if a model has 7 billion parameters in FP16 precision ($7\text{B} \times 2\text{ bytes} = 14\text{ GB}$), it can safely run production traffic on a 16 GB GPU. In reality, the server will crash with a CUDA Out-of-Memory exception almost immediately under concurrent load.

The Complete Production VRAM Allocation Formula

Total GPU Video RAM consumption ($VRAM_{\text{total}}$) is composed of four distinct components:

$$VRAM_{\text{total}} = M_{\text{weights}} + M_{\text{KV\_cache}} + M_{\text{activations}} + M_{\text{runtime\_context}}$$

1. Model Weights Memory ($M_{\text{weights}}$)

The static weight tensors loaded into GPU memory at initialization:

• FP16 / BF16 (16-bit): $P \times 2\text{ bytes}$
• INT8 / FP8 (8-bit): $P \times 1\text{ byte}$
• INT4 / AWQ / GPTQ (4-bit): $P \times 0.5\text{ bytes} + \text{scales}$

For Llama-3-70B in FP16: $70 \times 2 = 140\text{ GB}$. In 4-bit AWQ: $70 \times 0.5 \approx 35\text{ GB}$.

2. Key-Value (KV) Cache Memory ($M_{\text{KV\_cache}}$)

The dynamic memory storing projected Keys and Values across all attention layers for all active tokens to avoid recalculating past context during autoregressive generation:

$$\text{Bytes/Token} = 2 \times n_{\text{layers}} \times n_{\text{kv\_heads}} \times d_{\text{head}} \times b_{\text{precision}}$$

For Llama-3-8B ($n_{\text{layers}}=32, n_{\text{kv\_heads}}=8, d_{\text{head}}=128$, FP16):
$2 \times 32 \times 8 \times 128 \times 2 = 131,072\text{ bytes} = 128\text{ KB/token}$.
At 8,192 context length and 32 concurrent requests:
$32 \times 8,192 \times 128\text{ KB} = \mathbf{33.55\text{ GB}}$ of KV cache alone!

3. Activation Memory ($M_{\text{activations}}$)

Intermediate tensors generated during forward passes across normalization layers, softmax attention scores, and feed-forward intermediate projections. Scales proportionally with batch size and sequence length during the Prefill phase.

4. CUDA Runtime & Safety Headroom ($M_{\text{runtime}}$)

PyTorch/CUDA context overhead, memory allocator fragmentation, NCCL communication buffers for distributed tensor parallelism, and kernels require between 1.5 GB to 3.5 GB of unfragmented VRAM.

5 Production Model Serving: Continuous Batching, vLLM & PagedAttention

Deploying raw Hugging Face `pipeline()` or naive PyTorch script in a web server is an anti-pattern. Enterprise model serving architectures rely on specialized engines designed around high-throughput batching and memory management.

The Evolution of Inference Batching Algorithms

1. Static Batching

Collects a fixed number of requests (e.g. $B=8$) or waits for a fixed timeout (e.g. 100ms).

Fatal Flaw: If request A finishes in 20 tokens while request B requires 500 tokens, the GPU slots for request A sit completely idle, wasted by zero-padding until request B terminates.

2. Dynamic Batching

Collects arriving requests up to a variable window. Employs padding tensors to match sequence length.

Limitation: Still couples requests across the generation step. Short requests suffer latency degradation waiting for straggler sequences.

3. Continuous / In-Flight Batching

Iterative token-level scheduling (introduced by Orca and popularized by vLLM).

Industry Standard: After every single decode token iteration, terminated sequences are immediately evicted, and arriving requests are dynamically inserted into the batch. Delivers 4x to 8x throughput gains.

PagedAttention: Solving the 80% KV Cache Waste

In traditional deep learning frameworks, KV cache memory is pre-allocated as contiguous blocks for the maximum possible sequence length (e.g., 8,192 tokens). Because actual user prompts vary wildly, between 60% to 80% of allocated VRAM was wasted on unused reservations and internal fragmentation.

Virtual Memory Paging: vLLM translates operating system virtual memory principles into GPU DRAM. KV cache is divided into non-contiguous physical blocks (e.g., 16 tokens per block).

Block Tables & Zero Waste: Memory is allocated strictly on-demand as new tokens are decoded. Multiple requests sharing a common system prompt or prefix point to the same physical memory blocks via Copy-on-Write (RadixAttention).

vllm_production_engine.py High-Throughput Async Engine Setup

from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams
import asyncio

# Production-grade AsyncEngineArgs configuration
engine_args = AsyncEngineArgs(
    model="meta-llama/Meta-Llama-3-70B-Instruct",
    tensor_parallel_size=4,           # Distributed across 4x A100-80GB GPUs
    gpu_memory_utilization=0.90,       # Allocate 90% of VRAM to weights + KV cache
    max_model_len=8192,                # Context limit
    max_num_batched_tokens=8192,       # Continuous batching token budget per step
    max_num_seqs=256,                  # Concurrency limit (in-flight sequences)
    kv_cache_dtype="auto",             # FP16 or FP8 KV cache quantization
    enable_prefix_caching=True,        # Automatic RadixAttention prefix cache
    trust_remote_code=True,
    disable_log_requests=False
)

engine = AsyncLLMEngine.from_engine_args(engine_args)

async def stream_inference(prompt: str, request_id: str):
    sampling_params = SamplingParams(
        temperature=0.7,
        top_p=0.95,
        max_tokens=512,
        stop=["<|eot_id|>"]
    )
    
    # Asynchronous token streaming iterator
    results_generator = engine.generate(prompt, sampling_params, request_id)
    async for request_output in results_generator:
        # Stream the most recent delta token to the client via SSE
        yield request_output.outputs[0].text

6 Prefill vs Decode Phases & Latency SLOs: TTFT vs TPOT

Inference on autoregressive Large Language Models is fundamentally divided into two distinct computational phases with radically different hardware performance profiles:

Phase 1: Prefill (Prompt Processing)

The model processes all input tokens in the user's prompt simultaneously in a single forward pass.

• Hardware Regime: Compute-Bound ($TFLOPs$). High Arithmetic Intensity.

• Operation: Highly parallel GEMM across all input tokens. High GPU Core Utilization.

• Directly Dictates: TTFT (Time To First Token).

Phase 2: Decode (Token Generation)

The model generates one output token at a time autoregressively, feeding each generated token back into the input.

• Hardware Regime: Memory Bandwidth-Bound ($GB/s$). Low Arithmetic Intensity.

• Operation: Must reload entire model weights and KV cache from HBM for every single generated token.

• Directly Dictates: TPOT (Time Per Output Token).

Core Latency Metrics for AI Production SLOs

1. TTFT (Time to First Token)

Latency from HTTP request arrival until the first token streams out to client.

User Perception: Responsiveness

2. TPOT (Time Per Output Token)

Mean duration to generate each incremental token ($1 / \text{Tokens Per Second}$).

Target: < 30–50 ms/tok (20+ tok/s)

3. E2E Latency (End-to-End)

$$\text{E2E} = \text{TTFT} + (N_{\text{output}} \times \text{TPOT}) + \text{Queue\_Wait}$$

Total Completion Duration

7 Queues, Backpressure & Admission Control Architecture

In conventional web architectures, databases and APIs can handle thousands of concurrent TCP sockets. In AI infrastructure, dispatching 1,000 unthrottled requests directly into GPU memory causes immediate system failure via VRAM exhaustion and unrecoverable tail latency.

The Danger of Unbounded Queues: Cascading Degradation

If an infrastructure design incorporates an unlimited queue (e.g. `maxsize=float('inf')`), during a sudden 5x traffic surge requests accumulate into thousands of waiting items. By the time a GPU worker pulls a request from the queue, 30 seconds have passed: the user has already closed their browser or triggered a frontend timeout retry. The GPU expends expensive compute generating tokens for abandoned connections, exacerbating the queue backlog until the entire cluster fails.

1. Strict Admission Control & Load Shedding

The API Gateway evaluates cluster health prior to accepting any request:

• If $\text{Queue\_Depth} > \text{Threshold}_{\text{max}}$, immediately reject with HTTP `429 Too Many Requests` or `503 Service Unavailable`.
• Include `Retry-After: 5` headers with exponential backoff and jitter instructions.
• Implement CoDel (Controlled Delay) to drop expired requests before compute is wasted.

2. Multi-Tenant Priority Queues

Tiered priority dispatching prevents paying enterprise customers from starving behind batch free-tier tasks:

• Priority 0 (Enterprise Tier): Dedicated GPU capacity reservation with instantaneous preemption.
• Priority 1 (Standard Paid): Shared warm pool with guaranteed concurrency quotas.
• Priority 2 (Free / Batch Background): Processed only during GPU idle cycles with strict rate caps.

admission_controller.py Redis Sorted-Set Priority Queue & Admission Logic

import redis
import time
import json
from fastapi import HTTPException

r = redis.Redis(host='localhost', port=6379, db=0)

MAX_QUEUE_DEPTH = 500  # Hard circuit-breaker limit

def enqueue_inference_request(tenant_id: str, priority_tier: int, payload: dict) -> str:
    current_depth = r.zcard("inference_priority_queue")
    
    # 1. Backpressure Admission Control Check
    if current_depth >= MAX_QUEUE_DEPTH:
        if priority_tier > 1:  # Reject non-enterprise traffic immediately
            raise HTTPException(
                status_code=503, 
                detail="Inference cluster saturated. Load shedding active. Retry with backoff."
            )
            
    # 2. Priority Score: Higher priority (lower int) + Arrival Timestamp for FIFO within tier
    request_id = f"{tenant_id}:{int(time.time()*1000)}"
    # Score = priority * 1e12 + timestamp (ensures enterprise requests jump ahead)
    score = (priority_tier * 1000000000000) + time.time()
    
    r.zadd("inference_priority_queue", {json.dumps(payload): score})
    return request_id

def pop_next_batch(batch_size: int = 16):
    # Atomic worker pop using Redis Sorted Set
    items = r.zpopmin("inference_priority_queue", count=batch_size)
    return [json.loads(item[0]) for item in items]

8 Distributed Inference: Tensor Parallelism vs Pipeline Parallelism vs Data Parallelism

When models exceed the physical VRAM capacity of a single GPU (such as 70B models requiring 140 GB in FP16 or 405B models requiring 810 GB), or when ultra-low latency requires splitting matrix computations across hardware, Distributed Inference becomes mandatory.

Figure 55.2: Distributed Model Partitioning Paradigms Parallelism Topologies

flowchart LR
    subgraph DP["Data Parallelism (DP)"]
        DP_Req1["Request 1"] --> DP_GPU1["GPU 1: Full Model Replica"]
        DP_Req2["Request 2"] --> DP_GPU2["GPU 2: Full Model Replica"]
    end

    subgraph TP["Tensor Parallelism (TP) - Intra-Node"]
        TP_Req["Single Request"] --> TP_Split["Split Attention & MLP Layers"]
        TP_Split --> TP_GPU1["GPU 1: Column Slice W1"]
        TP_Split --> TP_GPU2["GPU 2: Column Slice W2"]
        TP_GPU1 & TP_GPU2 --> AllReduce["All-Reduce Sync (NVLink 900 GB/s)"]
    end

    subgraph PP["Pipeline Parallelism (PP) - Inter-Node"]
        PP_Req["Single Request"] --> PP_GPU1["GPU 1: Layers 1-20"]
        PP_GPU1 --> PP_GPU2["GPU 2: Layers 21-40"]
        PP_GPU2 --> PP_GPU3["GPU 3: Layers 41-60"]
    end

Parallelism Paradigm	Partitioning Strategy	Interconnect Requirement	Latency vs Throughput Impact	Primary Use Case
Data Parallelism (DP)	Duplicate complete model across independent GPUs; each serves distinct requests.	Standard Ethernet / No GPU-to-GPU sync during inference.	Maximizes aggregate cluster throughput; does not reduce single-request latency.	Models that fit entirely on 1 GPU (e.g. 8B models on A10G/L4/RTX 4090).
Tensor Parallelism (TP)	Splits individual weight matrices (Linear & MLP projections) column-wise and row-wise (Megatron-LM).	Extremely High: Requires intra-node NVLink (600–900 GB/s). Unviable over PCIe/Ethernet.	Drastically reduces single-request latency by parallelizing matrix ops.	Large models (70B, 405B) requiring low TTFT/TPOT within a single multi-GPU node.
Pipeline Parallelism (PP)	Divides transformer layers sequentially across GPUs (e.g., GPU 0 gets layers 1–20, GPU 1 gets 21–40).	Moderate bandwidth (InfiniBand 400 Gbps or 100GbE). Only activations pass between stages.	Suffers from "pipeline bubble" idle time during decode unless micro-batched heavily.	Massive models exceeding a single 8-GPU node (e.g. 405B across multiple servers).

9 Dynamic Autoscaling, Cooldown & Cold Start Mitigation

In conventional microservices, autoscaling on CPU > 70% is standard. In deep learning infrastructure, CPU utilization is completely decoupled from GPU inference saturation. Autoscaling an AI cluster requires dedicated metrics and hysteresis controllers.

Correct Scaling Signals for AI Clusters

• Queue Depth / Latency in Queue: If requests wait > 200ms in queue, compute capacity is exhausted.
• Active In-Flight Tokens: $\sum \text{Prompt\_Tokens} + \text{Output\_Tokens}$ tracking real GPU saturation.
• KV Cache Memory Utilization: If vLLM reports > 85% KV block allocation, scale out immediately to prevent request eviction.
• P95 Time To First Token (TTFT): Breaches of latency SLO directly trigger scaling alarms.

The Flapping Problem & Cooldown Hysteresis

Because GPU provisioning is expensive ($1.50–$4.00/hr per GPU) and initialization takes 60–180 seconds, rapid scale-up and scale-down oscillations ("flapping") destroy economics and introduce latency spikes:

• Scale-Up Trigger: Fast response (e.g. sustained high load for 30 seconds).
• Scale-Down Cooldown: Conservative hysteresis window (e.g. maintain idle instances for 10–15 minutes before terminating).
• Warm Standby Pool: Maintain $N=1$ or $2$ pre-warmed nodes with weights loaded in memory ready to accept traffic within milliseconds.

10 Capacity Planning Math & Economic Infrastructure Modeling

A Senior AI Infrastructure Engineer must translate business Service Level Objectives (SLOs) into concrete hardware provisioning budgets.

The Enterprise Capacity Planning Formula

Given peak traffic of $RPS_{\text{peak}}$ (requests/second), average input tokens $T_{\text{in}}$, average output tokens $T_{\text{out}}$, and a single GPU benchmarked sustainable throughput $TH_{\text{gpu}}$ (tokens/second):

$$\text{Total Token Demand Rate} = RPS_{\text{peak}} \times (T_{\text{in}} + T_{\text{out}})$$

$$\text{Minimum Theoretical GPUs} = \frac{\text{Total Token Demand Rate}}{TH_{\text{gpu}}}$$

$$\mathbf{\text{Production Provisioned GPUs}} = \left\lceil \text{Minimum GPUs} \times (1 + \text{Headroom}) \right\rceil$$

Where enterprise safety headroom is strictly kept between 30% to 50% ($1.3 - 1.5$) to absorb sudden flash crowds and permit background rolling updates without breaching P95 latency SLOs.

Hands-On Simulation Workbench

Enterprise AI Infrastructure Simulation & Planning Suite

Zero-Quiz interactive engineering laboratory: calculate VRAM footprints, simulate continuous batching timelines, analyze distributed topologies, and stress-test autoscaling controllers.

Workload & Architecture Inputs

Model Parameter Scale

Weight Precision

Concurrent Active Requests: 32

Max Context Length per Request (tokens): 4096

Dynamic Memory Footprint Breakdown

Weight Memory 35.0 GB

KV Cache Pool 16.8 GB

Activations & Runtime 3.5 GB

Total Required VRAM 55.3 GB

Hardware Recommendation: 1x A100-80GB or 1x H100-80GB (Sufficient single-node VRAM with 30.8% headroom).

Enterprise Production Readiness Evaluation Rubric

Evaluate your AI infrastructure against these 16 rigorous production criteria across four architectural pillars.

Pillar 1: Hardware & VRAM Architecture

• VRAM Margin: Static weights + KV cache allocate ≤ 90% of total VRAM, leaving unfragmented headroom.
• Precision Standardization: Quantization (AWQ/FP8) applied strategically without degrading reasoning benchmarks.
• Interconnect Validation: Tensor Parallelism deployed strictly over NVLink buses; zero inter-node TP across raw PCIe.
• Thermal & ECC Monitoring: Hardware telemetry captures throttling events and GPU uncorrectable memory errors.

Pillar 2: Serving & Low-Latency Scheduling

• Continuous Batching: In-flight iteration-level scheduling active; zero static padding waste.
• Paged Memory: Virtual memory paging (PagedAttention) eliminates KV cache memory fragmentation.
• Radix Prefix Caching: Common system prompts and few-shot demonstration blocks cached with zero duplicate prefill.
• Async Streaming: First token dispatched over Server-Sent Events (SSE) within target TTFT SLO.

Pillar 3: Autoscaling & Resilience

• Token-Based Metrics: Scaling triggers driven by queue depth and active in-flight tokens, not CPU utilization.
• Hysteresis & Cooldown: 10–15 minute scale-down cooldown prevents costly provisioning flapping.
• Warm Standby Capacity: At least 1 pre-warmed instance maintained to absorb instantaneous flash spikes.
• Active Load Shedding: Strict queue depth limits trigger HTTP 429/503 before latency spirals.

Pillar 4: Cost Governance & Observability

• Tail Latency Observability: P50, P95, and P99 TTFT and TPOT tracked independently.
• Cost Accounting: Infrastructure spend tracked per tenant, per model pool, and per generated token.
• Multi-Tier Semantic Cache: Response cache absorbs redundant FAQ queries at < 20ms with zero GPU cost.
• Model Cascades: Lightweight queries routed to 8B SLMs; expensive 70B/405B models reserved for complex tasks.

Senior Architectural Decision Scenarios

Examine real-world production crises and architectural resolutions encountered by principal infrastructure engineers.

Scenario 1: Sudden 15x Traffic Spike & Cascading KV Cache OOM Failures

Context: During a national marketing campaign, an AI assistant experiences an instantaneous 15x traffic surge. Arriving prompts contain 16k tokens of uploaded PDFs. GPU workers crash sequentially with CUDA OOM errors, and restarted nodes crash immediately upon receiving retransmitted client requests.

Architectural Resolution: (1) Deploy an immediate admission control circuit breaker at the API gateway: hard-cap maximum concurrent requests and reject excess traffic with HTTP 503 + Retry-After. (2) Enforce an immediate context admission limit: truncate or reject single prompts exceeding 8k tokens during surge mode. (3) Configure vLLM with `enable_chunked_prefill=True` to divide massive prefill tokens across multiple scheduling iterations, preventing activation spikes from exhausting VRAM. (4) Direct client SDKs to employ randomized exponential backoff to eliminate the thundering herd.

Scenario 2: Serving 70B Model under < 1.2s P95 TTFT SLO on a Strict Budget

Context: An enterprise legal SaaS requires Llama-3-70B for contract QA. Customers demand a P95 Time To First Token under 1,200ms on 4k-token context windows. The finance team caps monthly infrastructure budget at $12,000.

Architectural Resolution: (1) Convert the 70B model from unquantized FP16 (requiring 4x A100-80GB at $14,000/mo) to 4-bit AWQ or FP8 (fitting comfortably across 2x L40S 48GB or 4x A10G 24GB at $5,200/mo). (2) Deploy Tensor Parallelism (TP=2) across high-bandwidth NVLink to accelerate matrix multiplication during the prefill phase. (3) Enable vLLM RadixAttention prefix caching: since 80% of customer prompts share identical 3k-token contract boilerplate clauses, prefix cache hits bypass prefill compute entirely, reducing TTFT from 1,800ms to 45ms.

Scenario 3: Mixed Workload Collocation (Embeddings + 8B SLM + 70B Reasoning)

Context: A developer attempts to host a 384-dimensional embedding model, an 8B chat model, and a 70B reasoning model inside a single shared GPU server to "maximize utilization." Under load, chat latency spikes unpredictably from 400ms to 9 seconds.

Architectural Resolution: (1) Decouple compute pools strictly by hardware characteristics. Embedding models are compute-light and memory-compact: offload them to CPU instances (e.g. c6i.4xlarge with ONNX Runtime/OpenVINO) or lightweight T4/L4 GPUs. (2) Place 8B interactive chat models into a dedicated pool optimized for low TTFT and continuous batching. (3) Dedicate high-memory multi-GPU nodes (TP=4) exclusively to 70B deep reasoning queries. Heterogeneous workloads sharing a single CUDA context suffer from memory lock contention and unpredictable queue head-of-line blocking.

Scenario 4: Self-Hosted Bare Metal vs Hyperscaler Managed Inference TCO

Context: An engineering director debates whether to continue paying $0.80 per million tokens on managed inference APIs or lease dedicated 8x H100 GPU servers at $24,000/month.

Architectural Resolution: Model the Total Cost of Ownership (TCO) break-even curve: $$\text{Break-Even Tokens/Month} = \frac{\text{Monthly Fixed GPU Lease} + \text{Eng DevOps Overhead}}{\text{Managed Cost per Token}}$$ If monthly volume is < 20 million tokens or highly bursty with 80% night-time idle hours, managed serverless APIs are vastly more cost-effective. If traffic exceeds 150 million tokens/month with predictable 24/7 baseline demand, self-hosted bare-metal clusters lower unit inference costs by over 65%.

Key Takeaways: Principles of Production AI Infrastructure

• Inference is Physics: Throughput is bound by memory bandwidth ($TB/s$) during decoding and compute cores ($TFLOPs$) during prefill.

• Memory Footprint Math: Model weight size is only half the equation; dynamic KV cache grows rapidly with concurrency and sequence length.

• Continuous Batching is Essential: Never deploy static or naive batching; iteration-level scheduling (vLLM) provides 4x–8x throughput improvements.

• Queues Protect Hardware: Unbounded queues guarantee system failure; combine strict admission control, backpressure, and priority tiering.

• Interconnect Dictates Topology: Tensor Parallelism requires high-speed intra-node NVLink; inter-node communication requires Data or Pipeline Parallelism.

• Useful Throughput over Raw Utilization: Maximize useful tokens generated under strict latency SLOs, not idle GPU core metrics.

AI Infrastructure & Scaling

Table of Contents & Engineering Map