Lesson 16.03: The Modern AI Stack — Providers, Models & Runtimes

1 Introduction: 3 Distinct Paths to Running AI Models

Imagine you are building an enterprise AI SaaS platform. The application must answer customer queries, retrieve internal company documentation, extract structured data from invoices, and execute audited transactions against a SQL database.

As a software architect, your first foundational decision is: Which AI model should we select, and how will it be executed within our software infrastructure?

In modern software engineering, there are 3 distinct execution paths available:

2 The 6 Core Components of the Modern AI Stack

To architect AI systems effectively, engineers must maintain clear boundaries between the following 6 distinct layers and concepts:

3 Complete AI Stack Connectivity Architecture

                   +----------------------------+
                   |       AI Application       |
                   +--------------+-------------+
                                  |
                                  v
                   +----------------------------+
                   |    Model Integration Layer |
                   +--------------+-------------+
                                  |
                 +----------------+----------------+
                 |                                 |
                 v (Managed Cloud Path)            v (Self-Hosted Path)
        +------------------+             +-------------------+
        | Provider API     |             | Your GPU Server   |
        | (OpenAI/Anthropic|             | (AWS EC2 / RunPod)|
        +--------+---------+             +---------+---------+
                 |                                 |
                 v                                 v
        +------------------+             +-------------------+
        | Cloud Model      |             | Model Runtime     |
        | (Proprietary)    |             | (vLLM / Ollama)   |
        +------------------+             +---------+---------+
                                                   |
                                                   v
                                         +-------------------+
                                         | Open Model Weights|
                                         | (Llama 3 / Mistral|
                                         +-------------------+

4 The Automobile Analogy & Conceptual Comparison Table

To build intuition across non-AI team members, consider this real-world analogy:

5 How Managed AI APIs Work in Enterprise Systems

With Managed APIs, the provider maintains massive clusters of high-end GPUs (Nvidia H100s/B200s). Your backend communicates purely over encrypted HTTPS. However, calling an API is only one small step in an enterprise system.

6 Open-Weight Models & Self-Hosted AI

An Open-Weight Model is a neural network whose trained parameter files are publicly downloadable (e.g., Meta's Llama 3.1, Mistral NeMo, Qwen 2.5, DeepSeek). You can deploy these weights onto private servers or on-premises GPU hardware.

7 Open-Weight vs. Open Source: Demystifying the Distinction

8 OpenAI, Anthropic & Google Gemini — Strategic Provider Comparison

9 What is a Model Hub? (Hugging Face Hub)

A Model Hub (predominantly Hugging Face Hub) serves as the GitHub of the machine learning ecosystem. It hosts hundreds of thousands of open-weight models, datasets, benchmark leaderboards, and parameter checkpoints.

10 What is Ollama? (Local Model Runtime)

Ollama is a streamlined CLI utility and background service that enables running open-weight models locally with zero setup overhead. Built on top of `llama.cpp`, it bundles model management, quantization, and an OpenAI-compatible REST server into a single binary.

11 What is vLLM? (High-Throughput Production Inference)

vLLM is an open-source, production-grade LLM inference and serving engine designed for high-concurrency enterprise workloads. By implementing the revolutionary PagedAttention algorithm (which manages Key-Value attention cache memory similar to virtual memory paging in operating systems), vLLM virtually eliminates GPU VRAM fragmentation and boosts throughput by 2x to 4x compared to traditional serving systems.

12 Managed API vs. Self-Hosted Model: Real-World Cost Modeling

13 When to Choose Managed Cloud APIs

• You are building an MVP or newly launched product requiring rapid iteration and market validation.
• Your engineering organization lacks dedicated MLOps, GPU infrastructure, or Kubernetes specialists.
• Workloads experience high burstiness or variable traffic patterns where idle GPU reservations would waste capital.
• The product demands peak cognitive reasoning capability (e.g., frontier coding or multi-step legal analysis).

14 When to Consider Self-Hosted Production Models

• Strict data sovereignty, regulatory compliance, or on-premises mandates (defense, healthcare, sovereign banking).
• Low latency demands: internal LAN or VPC service communication must eliminate public WAN round-trips.
• High sustained volume: millions of continuous queries daily where GPU fixed costs break even against token bills.
• Custom fine-tuned weights or proprietary LoRA adapters that must remain confidential within private infrastructure.

15 Hybrid Architecture: The Intelligent Model Router Pattern

Production architectures rarely adhere to an all-or-nothing binary choice. Modern systems employ an Intelligent Model Router to dispatch incoming requests dynamically:

                        Incoming User Request
                                  |
                                  v
                        Intelligent AI Router
                                  |
                +-----------------+-----------------+
                | (Simple Task / High Volume)       | (Complex Reasoning / Low Volume)
                v                                   v
       Self-Hosted / Small Model            Frontier Managed API
     (Llama 3.1 8B via vLLM / Ollama)      (Claude 3.5 Sonnet / GPT-4o)
                |                                   |
                +-----------------+-----------------+
                                  |
                                  v
                        Consolidated Application

For instance, 80% of repetitive, routine classification requests are routed to a cheap self-hosted 8B model, while the 20% requiring nuanced multi-step legal reasoning are routed to Claude 3.5 Sonnet or GPT-4o.

16 The 9-Point Model Selection Checklist

17 Preventing Provider Lock-in with Model Abstraction

Scattering vendor-specific SDK calls across your controller layer tightly couples your application to a single commercial provider. To maintain strategic architectural flexibility, implement the Adapter Design Pattern:

from typing import Protocol

class TextModelInterface(Protocol):
    def generate(self, prompt: str) -> str:
        ...

# Concrete OpenAI Implementation
class OpenAIAdapter:
    def generate(self, prompt: str) -> str:
        # Executes OpenAI SDK logic
        return "response"

# Concrete Anthropic Implementation
class AnthropicAdapter:
    def generate(self, prompt: str) -> str:
        # Executes Anthropic Messages SDK logic
        return "response"

Your core application controllers invoke `TextModelInterface`, remaining completely agnostic of which backend provider fulfills the inference contract.

18 Safe 6-Step Model Migration Pipeline in Production

Never switch a production foundation model globally in a single atomic deployment. Always validate with dark traffic or canary increments.

19 Hands-On Architecture Exercise: Infrastructure Selection for an AI Invoice Assistant