Lesson 16.02: AI Product Requirements & Architecture Decisions

1 Introduction: What to Do Before Writing Code?

Imagine an executive or enterprise stakeholder approaches you with enthusiasm:

Stakeholder Brief:

“We want an AI system that automatically reads all incoming supplier invoices and enters the data straight into our accounting software!”

A junior developer might immediately begin brainstorming implementation syntax:

• Which LLM should I call? • FastAPI or Express? • PostgreSQL tables? • Tesseract OCR? • Prompt template? • OpenAI API key?

A Senior AI Engineer, however, never rushes into code. Instead, they step back and interrogate the problem across 11 diagnostic dimensions:

11 Diagnostic Questions Every AI Engineer Asks Before Coding:

1. Problem Definition: What is the exact operational bottleneck—speed of entry or frequency of data entry errors?

2. Current Baseline: How is the workflow handled today, and how many human hours does it consume?

3. Workload Volume: How many invoices arrive per month? (100 or 100,000?)

4. Ingestion Formats: Are files clean digital PDFs, phone camera photos, or degraded fax scans?

5. Layout Diversity: Do invoices come from 3 predictable suppliers or thousands of unstructured layouts?

6. Required Accuracy: What accuracy is acceptable? (95% field-level or 99.9% exact match?)

7. Blast Radius of Errors: What financial or legal damage occurs if an invoice total is miscalculated?

8. Data Sensitivity: Do invoices contain supplier bank accounts, tax IDs, or confidential pricing?

9. Latency SLA: Must parsing complete in 5 seconds synchronously, or can it run in a background queue?

10. Automation Level: Should the system post directly into the ledger, or requires human review?

11. Unit Economics: What is the maximum acceptable compute and API cost per processed document?

Without answers to these questions, selecting an architecture is mere guesswork. Multiple viable architectures exist for every problem; the right choice is dictated strictly by constraints, risk, and unit economics.

Core Tenet: Never start with the technology. Start with the problem, analyze the constraints, and let the requirements dictate the architecture.

2 From Business Problem to Technical Solution: The 5 Layers

Building an enterprise AI application requires transitioning sequentially through 5 distinct layers of abstraction:

Business Problem (The pain point)
       ↓
Business Requirements (Desired outcomes & ROI)
       ↓
Technical Requirements (System specs & behavior)
       ↓
Architecture Decisions (Component topologies & patterns)
       ↓
Implementation (Code, models, testing & deployment)

1. Business Problem:

What organizational bottleneck exists? Example: Accounts payable employees spend 3 hours daily manually typing invoice figures, causing billing delays and duplicate payment errors.

2. Business Requirements:

What measurable metrics define success? Example: Reduce manual entry time by 80%, shorten processing turnaround from 24 hours to 10 minutes, and keep data entry discrepancy rates below 1%.

3. Technical Requirements:

What must the software system do? Example: Ingest PDFs/images, extract structured fields into strict schemas, validate arithmetic relationships, and persist approved records in PostgreSQL.

4. Architecture Decisions:

How will the components be arranged? Example: Secure S3 bucket, Redis message queue, background Celery workers, multimodal LLM extraction, deterministic validator, and human review split-screen.

5. Implementation:

Executing the blueprint. Example: FastAPI backend, Next.js frontend, Pydantic schema validation, OpenTelemetry distributed tracing, Docker containers, and CI/CD pipelines.

3 Should You Use AI in the First Place? (The Decision Framework)

No! AI is not the solution for every software problem. Forcing an LLM or neural network onto a problem that standard code can solve introduces unnecessary latency, financial waste, and non-deterministic fragility.

A senior engineer always matches the task to the most efficient and reliable computational mechanism:

Problem Type	Optimal Solution	Rationale
Tax calculations or currency sums	Standard Code (Python / JS)	Math is 100% deterministic; native code runs in microseconds with zero hallucinations.
User permission / RBAC verification	Authorization Middleware	Security requires strict boolean guarantees; never delegate authorization to probabilistic prompts.
Querying a customer order by ID	Indexed SQL Database Query	Database queries return exact records in 1ms with ACID guarantees.
Filtering server logs for error codes	Keyword / Regex Search	Exact string matching is trivial and free; using vector embeddings or LLMs is wasteful.
Extracting data from unpredictable layouts	OCR + AI Vision Extraction	Document layouts vary endlessly; neural vision models excel at contextual comprehension.
Synthesizing answers across corporate handbooks	RAG (Hybrid Search + LLM)	Synthesizing natural language summaries with exact source citations requires language models.
Dynamic multi-step goal execution	AI Agent / State Machine	Autonomous tool selection and recovery from unexpected tool responses require agentic loops.
Forecasting next month's sales revenue	Statistical ML (LightGBM/ARIMA)	Numeric time-series forecasting is best solved by classical gradient boosted trees or statistical models.

5-Question Decision Framework

Question 1: Can this be solved with deterministic business rules?

If yes → Write standard procedural code. Never use AI where deterministic logic works.

Question 2: Is this essentially a structured database lookup?

If key-value or indexed lookup suffices → Query SQL or Redis directly.

Question 3: Does the task require interpreting unstructured language or vision?

If parsing human text, semantic meaning, or layout images → AI models are the right tool.

Question 4: Does the answer depend on dynamic, proprietary enterprise context?

If grounded in private company handbooks → Integrate a Retrieval-Augmented Generation (RAG) pipeline.

Question 5: Does the workflow involve multi-step branching and dynamic tool calling?

If steps are fixed → Use deterministic state machines. If runtime decision-making is necessary → Employ an AI agent.

4 The 4 Categories of Requirements

Production AI engineering demands categorizing specifications across 4 distinct layers:

4.1 Business Requirements

Focus: ROI & Outcomes

The core economic rationale. Example: Reduce manual entry hours by 70%, lower processing turnaround from 24 hours to 5 minutes, and ensure operating costs stay within defined budget caps.

4.2 Functional Requirements (FRs)

Focus: System Actions & Behaviors

FR-01: Authenticated users can upload PDF and image invoice files.

FR-02: The system validates file MIME types and rejects unsupported extensions immediately.

FR-03: The system extracts Vendor Name, Invoice Number, Date, Tax, and Total Amount.

FR-04: Extracted data is validated against pre-defined schemas and arithmetic rules.

FR-05: Users can review and edit extracted fields alongside a live PDF preview.

FR-06: Approved records are persisted to PostgreSQL and pushed to the ERP accounting ledger.

4.3 Non-Functional Requirements (NFRs)

Focus: Performance, Security & Reliability

NFR-01 (Latency): Background processing P95 latency must remain under 15 seconds per single-page invoice.

NFR-02 (Security): Invoices must be encrypted at rest (AES-256) and access-restricted via RBAC.

NFR-03 (Reliability): Provider API outages must trigger exponential backoff retries and secondary fallbacks.

NFR-04 (Cost): Total average API and compute cost must remain below $0.03 per processed invoice.

4.4 AI-Specific Requirements

Focus: Quality, Validation & Safety

AI-01: Field-level extraction accuracy must reach at least 95% on golden evaluation sets.

AI-02: Model outputs must strictly adhere to the Pydantic JSON schema.

AI-03: The model must not hallucinate missing fields; ambiguous fields must be flagged as "MISSING".

AI-04: Invoices with confidence scores below 0.85 must automatically route to human review.

5 User Stories & Acceptance Criteria

To bridge human user intent with technical execution, software engineering utilizes User Stories paired with verifiable Acceptance Criteria.

Standard User Story Format:

As a [user role],
I want [feature / capability],
So that [business benefit].

Concrete Example: “As an accounts payable clerk, I want to drag and drop PDF invoices onto the portal, so that I don't have to manually re-type invoice line items into the ERP system.”

Gherkin Acceptance Criteria (Given-When-Then)

// Feature: Invoice Upload and Background Processing
Scenario: Successful file upload and worker dispatch
  Given an authenticated user with accounting permissions
  When they submit a valid PDF invoice under 10MB
  Then the system saves the file to encrypted object storage
  And issues a unique `job_id` with state "QUEUED"
  And dispatches a background task for AI extraction.

Defensive AI Acceptance Rule: Acceptance criteria must never state “The model will be 100% accurate.” Instead, specify how the application handles model failures, schema errors, and low-confidence edge cases.

6 Measuring Quality in an AI System (The 6 Core Metrics)

“The app works well” is not an engineering metric. Operating AI systems in production demands continuous measurement across 6 concrete KPIs:

6.1 Field-Level Accuracy Extraction Metric

The percentage of target fields extracted correctly across an evaluation batch.

Field Accuracy = (Correctly Extracted Fields ÷ Total Evaluated Fields) × 100

6.2 Exact Match Rate Strict Precision

For critical identifiers (Invoice Number, Tax ID), partial semantic similarity is worthless; outputs must match character-for-character.

Expected: INV-901 | Actual: INV-901 → PASS Expected: INV-901 | Actual: INV-902 → FAIL

6.3 End-to-End Latency (P50, P95, P99) Speed Metric

Total elapsed time from file submission to finalized database record. Measure total user experience, not just the isolated LLM inference duration.

6.4 Cost per Successful Task (FinOps) Economics Metric

Total blended infrastructure spend per successfully processed invoice.

Cost per Task = (Total AI API + OCR + Compute Spend) ÷ Total Successful Invoices

6.5 Human Review Escalation Rate Automation Efficiency

The fraction of tasks requiring manual intervention. (Target: under 10–15% for scalable systems).

6.6 Failure Taxonomy Rate Reliability Metric

Classifying errors into root causes: (1) Provider timeouts/outages, (2) Schema validation errors, (3) Corrupt/unreadable source documents.

7 System Constraints: Boundaries of Viability

Every software architecture operates within strict boundary conditions:

1. Latency Constraints

Real-time chat requires sub-second token streaming; background document extraction easily tolerates 15–30 seconds.

2. Volume & Scale Constraints

50 invoices daily run fine on a single server; 100,000 invoices demand distributed message queues and auto-scaling worker nodes.

3. Cost & Unit Economics

If an automated invoice saves $0.20 of human labor but costs $0.25 in API tokens, the application is economically unviable.

4. Data Heterogeneity

Digital vector PDFs, handwritten receipts, multilingual tables, and low-res mobile scans require resilient preprocessing pipelines.

5. Security, Privacy & Compliance Constraints

Adhering to strict enterprise data residency rules, Zero Data Retention (ZDR) agreements, and rigorous multi-tenant data isolation.

8 Architectural Archetypes for AI Products (3 Core Patterns)

In real-world systems, requirements and scale dictate which of 3 primary architectural topologies to implement:

Option A: Synchronous Request-Response

Low Volume / Simple Tools

User → Frontend → Backend API → AI API → Validation → Database → Response

Best For: Fast inferences (< 2s), simple prototypes, and low-concurrency internal utilities.

Bottlenecks: Long operations trigger gateway timeouts (HTTP 504); traffic spikes easily overwhelm backend processes.

Option B: Queue-Based Asynchronous Architecture

Mid to High Scale

User → Frontend → Backend API → Message Queue (Redis) → Background Workers → AI API → DB → SSE/Polling

Best For: Heavy files, non-deterministic latency, automatic worker retries, and high-concurrency ingestion.

Trade-offs: Operational complexity of queue brokers, worker pool monitoring, and async UI polling.

Option C: Document Pipeline with Human Review

Enterprise Standard

Upload → S3 Storage → Queue → OCR/Parse → AI Extraction → Pydantic Schema → Math Check → Human Review Screen → ERP DB

Best For: Financial, legal, and medical domains where errors carry severe consequences. High-confidence extractions pass automatically; ambiguous records pause for side-by-side human validation.

9 Architecture Decision Matrix

Evaluation Criteria	Option A: Synchronous	Option B: Queue-Based	Option C: Pipeline + Review
Initial Setup Complexity	Minimal (1–2 days)	Moderate (1–2 weeks)	High (3–4 weeks)
Heavy Files / Long Inference	Poor (Timeout risk)	Excellent	Excellent
Background Retries & Resilience	Negligible	Native Support	Built-in Circuit Breakers
Status Tracking & Progress	Browser Spinner	Polling / WebSockets	Comprehensive Audit Trail
Human-in-the-Loop Safety	Unsuitable	Requires Custom Addons	Core Architectural Pillar

10 MVP vs. Production System: The Maturation Curve

An MVP (Minimum Viable Product) validates core technical hypotheses with minimal code. An MVP, however, is not a hardened production platform.

Prototype → MVP → Validated Product → Hardened Production → Scaled Platform

What Belongs in an MVP

Basic file upload form and simple UI.
Direct backend-to-model API calls.
Core Pydantic schema validation.
Single relational database storage.
Basic try-catch exception handling.

What Production Demands

Encrypted S3 object storage with RBAC controls.
Asynchronous queue workers with exponential backoff.
FinOps token metering and real-time cost tracking.
OpenTelemetry distributed tracing and Slack alerting.
Automated CI evaluation suites on golden datasets.
Rate limit throttling and multi-provider failovers.

11 Architecture Decision Records (ADRs)

Six months after launch, a new engineer might ask: “Why did we introduce Redis queues instead of calling the LLM directly in the HTTP handler?” An Architecture Decision Record (ADR) preserves that historical context.

ADR-001: Background Queue Architecture for Long-Running Invoice Extraction Status: ACCEPTED

Context:

Multi-page PDF parsing, OCR, and multimodal LLM inference take between 8 and 30 seconds. Synchronous HTTP requests risk cloud load balancer timeouts (HTTP 504) and cause severe thread starvation under concurrent traffic.

Decision:

We will decouple file submission from processing by issuing a unique `job_id` immediately from FastAPI and offloading execution to Redis and Celery/BullMQ workers.

Positive Consequences:

Client requests return in under 200ms; zero dropped connections; workers automatically retry failed API calls with exponential backoff.

Trade-offs & Overhead:

Requires managing a Redis cluster and implementing frontend job status polling or Server-Sent Events.

12 Comprehensive Case Study: Enterprise AI Invoice Processing Architecture

Below is the end-to-end architecture dataflow for an enterprise processing 10,000 monthly supplier invoices:

// Enterprise AI Invoice Processing Architecture Dataflow

                       Accounting Employees
                                 |
                                 v
                    Frontend (Next.js Dashboard)
                                 |  HTTPS Upload
                                 v
                     Backend API (FastAPI)
                                 |
                +----------------+----------------+
                |                                 |
                v                                 v
          Authentication (RBAC)            S3 Object Storage
          Session / Audit Log                     | (File Saved)
                                                  v
                                          Redis Task Queue
                                                  |
                                                  v
                                        Background Workers
                                                  |
                                        PDF Text / OCR Engine
                                                  |
                                        Structured AI Extraction
                                         (GPT-4o / Claude Vision)
                                                  |
                                        Pydantic Schema Check
                                                  |
                                        Math Rules Check
                                      (Subtotal + Tax == Total)
                                                  |
                         +------------------------+------------------------+
                         | (Validation PASSED)                             | (Validation FAILED)
                         v                                                 v
                 Automatic Approval                               Human Review Screen
                         |                                        (Side-by-side Diff)
                         |                                                 |
                         +------------------------+------------------------+
                                                  |
                                                  v
                                      PostgreSQL ERP Database
                                                  |
                                                  v
                                      External Accounting System

13 Hands-on Exercise: Blueprint Your Own AI Product Requirements

Apply the 6-part specification framework to design your chosen application (e.g., Customer Support Copilot, Legal Research Assistant, or Coding Agent):

Part A: Problem Definition — Identify target users, current operational pain points, and specific value added by AI.

Part B: Functional Requirements — Define 5 distinct user-facing capabilities (FR-01 through FR-05).

Part C: Non-Functional Constraints — Target P95 latency, maximum monthly token budgets, and security controls.

Part D: AI-Specific Requirements — Accuracy benchmarks, schema strictness, and human escalation criteria.

Part E: Architecture Diagram — Diagram dataflow across UI, backend, AI gateway, vector DB, and workers.

Part F: Architecture Decision (ADR) — Author an ADR comparing synchronous vs queue-based execution for your product.

14 Self-Assessment: Knowledge Check

Question 1: Why is analyzing requirements before writing code paramount in AI engineering?

A. To avoid writing code permanently.

B. To understand the root problem, desired outcomes, and system constraints to select the right architecture.

C. To justify using microservices in every scenario.

D. To immediately select the most expensive model available.

Correct Answer: B — Coding without clear requirements leads to fragile architectures and costly rewrites.

Question 2: Which of the following is a Functional Requirement?

A. System latency must remain under 5 seconds.

B. Sensitive financial data must be encrypted.

C. Users can upload PDF invoices through the portal.

D. Monthly infrastructure costs must not exceed $100.

Correct Answer: C — Functional requirements define what capabilities the system offers to users. Speed, security, and cost are non-functional.

Question 3: Which of the following is a Non-Functional Requirement (NFR)?

A. Users can generate support tickets.

B. The system extracts supplier names from invoices.

C. The system supports automated task recovery with backoff upon model API outages.

D. Users can reset passwords.

Correct Answer: C — System resilience, fault tolerance, and recovery mechanisms are quality constraints (NFRs).

Question 4: How should an engineer decide whether AI is necessary for a problem?

A. Deploying AI indiscriminately across all workflows.

B. Following social media trends.

C. Evaluating whether deterministic code suffices, analyzing unstructured data needs, and comparing alternative solutions.

D. Building multi-agent swarms immediately.

Correct Answer: C — If standard code, SQL, or regex solves the problem, avoid using AI models.

Question 5: What is the primary purpose of Acceptance Criteria?

A. Choosing color schemes for the UI.

B. Establishing unambiguous, verifiable conditions that must be satisfied for a feature to be complete.

C. Wiping database tables.

D. Halting model evaluations.

Correct Answer: B — Acceptance criteria align developers, testers, and product managers on expected feature behavior.

Question 6: Why are Architecture Decision Records (ADRs) maintained?

A. Backing up line-by-line code comments.

B. Documenting the context, rationale, and trade-offs behind significant architectural choices for future maintainers.

C. Modifying model neural weights.

D. Bypassing automated testing.

Correct Answer: B — ADRs preserve historical context, preventing future developers from undoing sound architectural decisions.

Lesson Summary & Key Takeaways

✓ Problem First, Technology Second: Clarify business pain points and answer 11 diagnostic questions before writing code.
✓ Deterministic vs. AI: Keep arithmetic and permissions in standard code; deploy AI only for unstructured language and vision tasks.
✓ 4-Tier Requirements: Frame Business, Functional, Non-Functional, and AI-Specific requirements with verifiable metrics.
✓ Architecture Selection: Evaluate synchronous, queue-based, and human-in-the-loop review topologies against volume and risk constraints.
✓ ADR Stewardship: Institutionalize technical choices and trade-offs to ensure long-term architectural integrity.

Core Takeaway

“Never start with the technology—start with the requirements. First understand what must be built, why it must be built, what constraints exist, and how success will be measured. Only then select the architecture and models.”

Lesson 16.02: AI Product Requirements & Architecture Decisions — From Problem to System Design

What You Will Master