Lesson 15.73: AI Security Architecture & Threat Modeling

The moment an AI system moves into production, the fundamental nature of application security changes. In a traditional web application, engineers defend against well-defined deterministic threats: Authentication, Authorization, SQL Injection, XSS, CSRF, Session Security, API Security, and Network Security. In a modern AI platform, however, the attack surface spans the entire reasoning and execution pipeline: User Input → Prompt → RAG → Documents → Agent → Tools → External APIs → Model → Business Systems.

Unlike a passive web page, a production AI system does not merely read user input—it can retrieve private records, call internal and external APIs, execute database mutations, send emails, process uploaded files, trigger financial workflows, and make autonomous multi-step decisions. A compromised AI system does not merely return a wrong answer; it can directly damage real business operations.

                                       User
                                         │
                                         ▼
                                  Edge / WAF / CDN
                                         │
                                         ▼
                                    API Gateway
                                         │
                          ┌──────────────┼──────────────┐
                          ▼              ▼              ▼
                   Authentication   Rate Limit      Validation
                          │              │              │
                          └──────────────┼──────────────┘
                                         ▼
                                    AI Gateway
                                 ┌───────┴────────┐
                                 ▼                ▼
                              Policy           Context
                                 └───────┬────────┘
                                         ▼
                                   AI Runtime
                            ┌────────────┼────────────┐
                            ▼            ▼            ▼
                           RAG         Agents       Models
                            │            │            │
                            ▼            ▼            ▼
                        Knowledge      Tools      Model Gateway
                            │            │            │
                            └────────────┼────────────┘
                                         ▼
                                   Domain Services
                                         │
                                         ▼
                                  Business Systems
   [Surrounding Rings: Secrets Management • Audit • Observability • Threat Detection • Evaluation • Governance • Incident Response]

The Fundamental Principle of AI Security (Zero Trust Classification)

The single most important principle in AI security is: Never trust the model, user input, retrieved content, tool output, or external content by default. LLM intelligence is never a security boundary.

Threat Modeling is the systematic engineering discipline of analyzing which components of a system can be attacked, who the threat actors are, what impact a breach causes, and which deterministic controls reduce residual risk: System → Assets → Actors → Trust Boundaries → Threats → Impact → Controls → Residual Risk.

While a traditional endpoint like POST /orders has deterministic behavior, an AI Agent executes a probabilistic loop (Goal → Reason → Tool → Observation → Reason → Tool), expanding the attack surface across Input + Context + Model + Tools + Actions.

Mapping STRIDE to AI Systems & Quantitative Threat Scoring

We map Microsoft's classic STRIDE framework directly to AI platforms and prioritize remediation using quantitative risk scoring ($\text{Risk} = \text{Likelihood} \times \text{Impact}$, e.g., $\text{Likelihood}=4 \times \text{Impact}=5 \Rightarrow \text{Risk}=20$, augmented by exploitability, detectability, and exposure):

Prompt Injection attempts to manipulate the model's instruction hierarchy so it deviates from system policy, whereas a Jailbreak attempts to bypass the model's built-in safety alignment (e.g., adversarial roleplay).

Strict Instruction Hierarchy & Layered Prompt Injection Defense

A single prompt injection classifier is never sufficient because classifiers suffer from both False Positives (blocking legitimate users) and False Negatives (missing novel encodings). Remember the iron law of RAG security: Retrieved knowledge is evidence, not authority.

STRICT 5-LEVEL INSTRUCTION HIERARCHY               6-LAYER PROMPT INJECTION & JAILBREAK DEFENSE
====================================               ============================================
1. SYSTEM POLICY (Highest Authority)               Input Detection & Moderation
         ↓                                                      ↓
2. APPLICATION POLICY                              Strict Instruction / Data Separation (XML/Delimiter Framing)
         ↓                                                      ↓
3. TENANT POLICY                                   Least-Privilege Tool Restrictions
         ↓                                                      ↓
4. USER INPUT (Untrusted Request)                  Deterministic Policy Engine Enforcement
         ↓                                                      ↓
5. RETRIEVED DATA / TOOL OUTPUT                    Domain Service Authorization (RBAC/ABAC)
   (Evidence Only — Zero Authority!)                            ↓
                                                   Output Validation & PII Redaction

In agentic architectures, Excessive Agency occurs when an agent is granted broad write/mutation tools (`send_email()`, `delete_customer()`, `refund_order()`, `create_purchase_order()`) and autonomously executes high-impact business mutations when a user merely asks “Help me resolve this issue.”

SSRF Defense, Web-Browsing Agents & Sandboxed Code Execution

Document ingestion pipelines and multi-tenant stores represent high-value targets for resource exhaustion, cross-tenant leakage, and chained data exfiltration:

Secrets must never enter the LLM prompt or agent context. If an API key (`OPENAI_API_KEY`, Stripe secret, or database password) is placed in the context window, any prompt injection or log leak compromises the credential.

WRONG (Secret Inside Model Context)               RIGHT (Application-Controlled Secret Injection)
===================================               ===============================================
LLM Context:                                      Agent proposes tool call (zero secrets in context)
  OPENAI_API_KEY = "sk-..."                                ↓
  STRIPE_SECRET  = "rk_live_..."                  Tool Gateway / Application Layer
         ↓                                                 ↓ Fetches from Vault / AWS KMS / Secret Manager
Prompt Injection → Exfiltrates Key!               Scoped Credential Injected at HTTP Call Site Only

• Minimal IAM Credential Scope: Never attach full cloud account permissions to an AI worker; grant only path-scoped permissions (e.g., read:s3://tenant-bucket/tenant-a/*).
• Per-Tenant Integration Key Isolation: Store external integration tokens (`Tenant A → Shopify Credential A`, `Tenant B → Shopify Credential B`) in separate encrypted vault paths so one tenant's tool execution can never decrypt another tenant's token.
• Automated Key Rotation Lifecycle: Enforce short-lived tokens and automated rotation (Create → Use → Rotate → Revoke) rather than static long-lived keys.

An AI system's software supply chain extends far beyond npm or pip packages—it includes Model Weights (safetensors vs. unsafe pickle), Tokenizers, Embedding Models, Rerankers, Python ML Frameworks (PyTorch, Transformers), Agent Tools, Plugins, and Datasets.

Download Artifact (Prefer .safetensors over pickle)
  ↓
SHA-256 Checksum Validation
  ↓
Cryptographic Signature Verification (Cosign / Sigstore)
  ↓
Malware & Serialization Security Scan
  ↓
Promote to Internal Approved Model Registry
  ↓
Signed Container Deployment

Public Zone (WAF / CDN)
  ↓
API Zone (API Gateway / Laravel Auth)
  ↓
AI Zone (RAG Service / Agent Runtime / Model Server)
  ↓  [Agents NEVER get direct DB subnet access!]
Data Zone (Primary DB / Vector DB / Object Storage)

Container & GPU Node Hardening: Run model inference containers as non-root with a read-only root filesystem, minimal distroless base images, dropped Linux capabilities (`CAP_NET_RAW`), signed images, private GPU subnets, and identity-aware mTLS access.

Zero Trust AI dictates that no request is trusted simply because it originated inside the internal network. Every internal call (`Laravel ↔ mTLS ↔ Python AI Runtime`) must execute: Every Request → Authenticate → Authorize → Validate → Policy Check → Execute.

Why Guardrails Alone Fail — The 10-Layer Defense-in-Depth Stack

Input, Tool, and Output Guardrails (PII detection, injection classifiers, format validators) are valuable, but attackers routinely bypass standalone guardrail models. Production AI security requires 10 independent layers of Defense in Depth:

Security decisions must be enforced by deterministic systems outside the model—never by model compliance alone. Even if an LLM reasons “The customer appears eligible for a refund,” the deterministic Policy Engine and Domain Service evaluate if today - purchase_date <= 14: allow else: deny. If the Policy Engine says DENIED, the Policy Engine always wins.

                   User → Model → Proposed Action
                                         │
                                         ▼
                              DETERMINISTIC POLICY ENGINE
                           ┌─────────────┼─────────────┐
                           ▼             ▼             ▼
                         ALLOW          DENY     APPROVAL_REQUIRED
                           │                           │
                           │                           ▼
                           │                   Human / Dual Control
                           └─────────────┬─────────────┘
                                         ▼
                            Domain API (Business Rules)

Training Data Governance, Model Extraction & AI DoS / Loop Protection

• Data Provenance & Training Isolation: Track source, author, tenant, timestamp, version, approval for every artifact. Never feed raw production customer data directly into training (`Customer Data → Consent/Policy → Sanitization → Anonymization → Approved Dataset → Tenant-Isolated Fine-Tuning`).
• Model Extraction & Membership Privacy Defense: Mitigate model distillation/extraction and training data memorization leakage via output rate limits, query anomaly detection, watermarking, and data minimization.
• Model DoS, 100K Context Abuse, Agent Loop & Billing Abuse Controls: Prevent resource exhaustion and stolen API key spikes by enforcing max_input_tokens, max_output_tokens, max_steps, max_tool_calls, max_cost, max_runtime, daily tenant spend caps, and automatic anomaly suspension.

Security by Design & Secure-by-Default: Security is never bolted on after deployment (Requirements → Threat Model → Architecture → Implementation → Security Testing → Deployment). By default: No permission → No access; No approval → No high-risk action; No trusted source → No authority; No quota → No execution.

6-Stage AI Security Incident Response & Containment Playbook

Every code change, prompt change, or model upgrade (`v1 → v2`) must pass an automated AI Security CI/CD Release Pipeline before reaching production:

Below is the production Python reference implementation for Byteion's Deterministic Tool Gateway, Approval-Based Refund Schema, and Anti-SSRF URL Validator:

# 1. Deterministic Domain-Enforced Refund Tool (Model cannot bypass 14-day business policy!)
def refund_order(*, tenant_id: int, user_id: int, order_id: str, today_days_since_purchase: int):
    # Step 1: Server-side RBAC/ABAC authorization (never trust model reasoning)
    authorize(
        tenant_id=tenant_id,
        user_id=user_id,
        action="refund_order",
    )

    # Step 2: Tenant-scoped order fetch (prevents BOLA / cross-tenant mutation)
    order = orders.get(tenant_id=tenant_id, order_id=order_id)

    # Step 3: Deterministic business rule outside the LLM
    if today_days_since_purchase > 14:
        raise PolicyDeniedError("Refund denied: deterministic 14-day window exceeded.")

    # Step 4: Route high-risk mutation to approval workflow instead of direct execution
    return orders.request_refund_approval(order=order, requested_by=user_id)


# 2. Least-Privilege Support Agent Tool Schema (Exposes approval request, NOT direct refund!)
SUPPORT_AGENT_REFUND_SCHEMA = {
    "name": "request_refund_approval",
    "description": "Create a refund approval request for an authorized order",
    "parameters": {
        "type": "object",
        "properties": {
            "order_id": {"type": "string"},
            "reason": {"type": "string"}
        },
        "required": ["order_id", "reason"]
    }
}


# 3. Anti-SSRF & Anti-DNS-Rebinding Fetch Guard for Web-Capable Agents
import ipaddress, socket
from urllib.parse import urlparse

ALLOWED_EGRESS_HOSTS = {"docs.byteion.com", "status.stripe.com"}

def validate_agent_url(url: str) -> str:
    parsed = urlparse(url)
    if parsed.scheme not in ("https",) or parsed.hostname not in ALLOWED_EGRESS_HOSTS:
        raise SecurityViolationError(f"SSRF Blocked: Host {parsed.hostname} not in egress allowlist.")
    resolved_ip = ipaddress.ip_address(socket.gethostbyname(parsed.hostname))
    if resolved_ip.is_private or resolved_ip.is_loopback or resolved_ip.is_link_local:
        raise SecurityViolationError(f"SSRF Blocked: Private/Metadata IP {resolved_ip} forbidden.")
    return url

In Byteion's Zero Trust AI Architecture, five explicit separation laws govern the entire platform: Model ≠ Authorization, Agent ≠ Business Logic, RAG ≠ Trusted Instructions, User Input ≠ Policy, and Tool Schema ≠ Permission.

                                     BYTEION ZERO TRUST AI
                                               │
                                    WAF / Edge Security / CDN
                                               │
                                               ▼
                                          API Gateway
                              (Authentication • Rate Limit • Validation)
                                               │
                                               ▼
                                     Laravel Core Service
                               (Server Tenant Resolution • RBAC/ABAC)
                                               │
                                               ▼
                                          AI Gateway
                                  (Policy • Guardrail • Quota)
                                               │
                                               ▼
                                          AI Runtime
                              ┌────────────────┼────────────────┐
                              ▼                ▼                ▼
                        Scoped RAG          Agents        Model Gateway
                              │                │                │
                              │                ▼                │
                              │          Tool Gateway           │
                              │    (Auth + Schema/Biz Valid)    │
                              └────────────────┼────────────────┘
                                               ▼
                                        Domain Services
                                  (Commerce • CRM • Inventory)
                                               │
                                               ▼
                                   Tenant-Scoped Databases

Byteion 10-Row Security Control Matrix Across Core Assets

Use this 16-point maturity rubric to benchmark any AI platform's security architecture across four progressive stages:

Simulate Direct/Indirect Prompt Injection and RAG Poisoning against a 10-layer defense stack, test the Deterministic Tool Gateway against Excessive Agency and SSRF, calculate quantitative STRIDE threat scores, and audit your Zero Trust AI posture across 4 interactive tabs: