AI Agent Memory & Context Engineering: Short-Term, Long-Term, Episodic & Semantic Retrieval

Building autonomous systems reveals a fundamental engineering reality: LLMs are inherently stateless compute functions. Every HTTP request dispatched to an inference endpoint begins with a blank slate. Without an external persistence harness, an agent cannot maintain relationships, remember user preferences, learn from historical failures, or track multi-day objectives.

However, solving this does not mean simply dumping every historical conversation into the prompt window. Production systems must answer eight critical architectural questions:

The most common novice mistake in AI engineering is conflating Chat History with Agent Memory. They serve radically different operational purposes in production infrastructure:

[
  {"role": "user", "content": "Hey, can you export that report?"},
  {"role": "assistant", "content": "Sure, which format do you prefer?"},
  {"role": "user", "content": "I like PDF best, thanks! Also my cat is named Luna."}
]

{
  "entity_id": "usr_9981",
  "category": "user_preference",
  "key": "preferred_export_format",
  "value": "PDF",
  "confidence": 0.98,
  "provenance": "explicit_user_statement",
  "updated_at": "2026-10-08T09:30:00Z"
}

Inspired by cognitive neuroscience and distributed systems engineering, production agents categorize memory into five discrete, specialized subsystems:

Working memory is the agent's active CPU cache. It lives in fast key-value memory (such as Redis or local thread state) during the lifecycle of an ongoing workflow. The agent updates this structure after every observation:

{
  "task_id": "tsk_8471b",
  "goal": "Reconcile Q3 European churn against Competitor X pricing",
  "plan_dag": [
    {"step": 1, "action": "sql_query_churn", "status": "COMPLETED"},
    {"step": 2, "action": "scrape_competitor_pricing", "status": "COMPLETED"},
    {"step": 3, "action": "compute_differential_math", "status": "IN_PROGRESS"},
    {"step": 4, "action": "generate_executive_pdf", "status": "PENDING"}
  ],
  "active_step": 3,
  "observations": {
    "churn_rate": "14.2%",
    "competitor_price": "€49/month",
    "internal_price": "€69/month"
  },
  "token_budget_consumed": 4210,
  "loop_iteration": 4,
  "circuit_breaker_threshold": 10
}

When Step 4 completes and the user receives their report, this working state is either evicted or handed off to the Memory Consolidation Pipeline to extract durable insights.

Consider how human memory operates: you might not remember every single email you replied to on March 14 (episodic detail), but your brain consolidates that repeated interaction into a durable semantic principle: “Client Alpha requires approval from Sarah before shipping.”

The Consolidation Pipeline runs as an asynchronous background cron job. It scans recent episodic interaction clusters, performs deduplication, clusters repeated behavioral patterns, and generates durable semantic entities while pruning redundant narrative logs.

How does candidate information become permanent memory? A production architecture enforces a multi-tier formation gate:

In standard RAG, vector similarity alone frequently suffices. In Agent Memory, pure vector similarity is a guaranteed path to failure. A high-similarity semantic match from three years ago will override a fresh preference stated five minutes ago unless temporal and provenance weights are integrated.

The enterprise standard for memory retrieval is the Hybrid Composite Scoring Formula:

What happens when memories collide? In January, the user specified: “Export all reports to Excel.” In October, they announce: “I now strictly require PDF reports.”

{
  "id": "mem_oct_08",
  "user_id": "usr_4401",
  "key": "report_format",
  "value": "PDF",
  "status": "ACTIVE",
  "supersedes_id": "mem_jan_15",
  "created_at": "2026-10-08T10:00:00Z",
  "ttl_days": 365,
  "decay_rate_lambda": 0.005
}

Agent memory represents persistent, stateful data. Consequently, it inherits the full threat surface of enterprise databases, plus AI-specific adversarial vectors:

Production architectures deploy a Dual-Store Architecture: combining relational databases (PostgreSQL) for transactional integrity and access control, with vector extensions (pgvector / Qdrant) for semantic search:

CREATE TABLE agent_memories (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id VARCHAR(64) NOT NULL,
    user_id VARCHAR(64) NOT NULL,
    category VARCHAR(32) NOT NULL, -- 'preference', 'semantic_fact', 'episodic_event'
    key VARCHAR(128) NOT NULL,
    content TEXT NOT NULL,
    embedding vector(1536),
    importance FLOAT DEFAULT 0.5,
    confidence FLOAT DEFAULT 0.8,
    provenance VARCHAR(64) NOT NULL,
    superseded_by UUID REFERENCES agent_memories(id),
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),
    expires_at TIMESTAMPTZ
);

CREATE INDEX idx_memories_tenant_user ON agent_memories(tenant_id, user_id);
CREATE INDEX idx_memories_vector ON agent_memories USING ivfflat (embedding vector_cosine_ops);

A decoupled Memory Manager Subsystem exposes clean internal APIs for storing, searching, and managing memory records:

class MemoryManager:
    async def store(self, tenant_id: str, user_id: str, key: str, content: str, provenance: str) -> MemoryRecord:
        # Embeds content, applies validation gates, and persists to PostgreSQL
        pass

    async def search(self, tenant_id: str, user_id: str, query: str, limit: int = 5) -> list[MemoryRecord]:
        # Executes tenant-isolated hybrid retrieval scoring formula
        pass

    async def invalidate(self, memory_id: str, replacement_id: str) -> None:
        # Marks old memory as superseded and shifts active lineage pointer
        pass

Context Engineering is the art and science of curating the exact prompt payload delivered to an LLM. More context is not better context. Excessive context dilutes attention mechanisms (“lost in the middle” phenomenon), spikes latency, increases token expenses, and introduces conflicting signals.

Dynamic Token Budgeting: Rather than hardcoding prompt lengths, the Context Builder dynamically budgets tokens based on task complexity. For analytical reporting: 35% RAG chunks, 20% tools, 15% memory, 10% system, 20% generation output.

When a conversation spans 50 turns, keeping verbatim transcripts costs tens of thousands of tokens per request. Context Compression condenses the dialogue into a compact, structured snapshot:

[ACTIVE ARCHITECTURAL STATE]
- Project: Analytics Dashboard
- Frontend: React (Confirmed)
- Database: PostgreSQL (Approved)
- Active Step: API Schema Definition
- Unresolved Issues: None

The compression engine operates on a rolling window: retaining the last 3 turns verbatim for immediate conversational fluidity, while summarizing turns $1 \dots N-3$ into structured state attributes.