Forem: Amar Dhillon

Agent Memory Architecture 🧠

Amar Dhillon — Thu, 23 Apr 2026 15:44:46 +0000

If you are building agentic AI systems then memory management quickly becomes one of the most confusing parts (You'll soon realize)

Not because the idea is hard but because there are too many ways to think about it

People mix up sessions, context, history, embeddings, user preferences, and logs. Everything starts getting labeled as “memory”

So I wanted to break this down in a simple way using a practical architecture that I’ve been using

The First Principle: Memory is Not One Thing 🤔

When we say “agent memory”, we are actually talking about multiple layers working together

At a high level, you can think of memory as two major categories

Short-term memory
Long-term memory

both serve very different purposes as shown in below diagram

Memory ID: The Logical Vault 🏦

Before diving into types of memory, there is one important concept

Memory ID

Think of it as a logical vault that holds everything related to a user or system . Inside this vault, memory is organized in a structured way

memoryId = the memory store / vault

├── short-term memory
     └── actorId → sessionId → events

└── long-term memory
      └── namespace → memory records

This abstraction is powerful because it decouples memory from the agent

Keep in mind that memory is not tied to a specific agent, it is tied to an actor

Short-Term Memory: Covers what Just Happened 🧩

Short-term memory is all about the current interaction

It captures the flow of a conversation or task in real time

At its core, it has a simple hierarchy

Actor ID → identifies the user or system
Session ID → one continuous interaction
Events → the smallest unit of memory

An event can be anything like

user prompt
tool invocation
assistant response So a session becomes a sequence of events

What About Branching? 🌿

Things get interesting when you introduce branching . Instead of having a single linear flow, you can fork memory into multiple branches

For example

main conversation
flight agent path (me going from Ottawa to NZ)
hotel agent path (Finding a cool spot in Queentown)

Each branch can evolve independently while still being tied to the same session. This becomes very useful in multi-agent systems where different agents explore different reasoning paths

Long-Term Memory: Covers what Should Be Remembered 📚

Short-term memory is temporary

Long-term memory is intentional

This is where you decide what is worth keeping . Tt is not just one thing. Long-term memory has multiple types

1. Semantic Memory (Facts as Vectors) 🔎

This is your classic vector database layer . You store facts, embeddings, and retrievable knowledge

Examples

user preferences
structured facts
extracted knowledge

A system like OpenSearch or any vector DB works well here

This enables similarity search and contextual recall

2. Episodic Memory (What Happened Over Time) 🕰️

This is about history . Instead of storing raw conversations forever, you summarize them

You keep

summarized conversations
logs
historical patterns

Typically stored in something like S3 or object storage

This helps the agent recall past interactions without loading everything into context

3. Procedural Memory (How the System Behaves) ⚙️

This is often overlooked but very important . Procedural memory defines how the agent operates

It includes

tool definitions
policies
system rules
configurations

Think of this as the “operating manual” for the agent

Connecting Short-Term and Long-Term Memory 🔄

Now the real question

How do these two layers work together???

There are two main flows

Persist Flow (Short → Long)

Not everything from short-term memory should be stored

You apply strategies like

Summarization
Extraction
Filtering

Only important signals get promoted to long-term memory

Recall Flow (Long → Short)

When a new request comes in, then agent retrieves relevant long-term memory and injects it into the current context

This is how personalization and continuity happen

Why Actor-Centric Memory Matters ? 👤

One subtle but important design choice . Memory is attached to actorId, not agent

This means

one user can interact with multiple agents
memory remains consistent across all of them

It also means your system is agent-agnostic

Whether you have

1 agent
10 agents
or no agent (just APIs) The memory model still works

Practical Example 🛫

Let’s say Amar (user) is booking travel

Short-term memory tracks

current booking session
user inputs
tool calls

Long-term memory stores

Preference for window seat and business class
Vegetarian meals
Past trips

When the user comes back later

The system does not start from scratch, It recalls preferences and continues seamlessly

Key Takeaways I want you to remember from this blog 💡

Memory in agent systems is not just about storing chat history . It is about structuring information in a way that supports reasoning, recall and personalization

Short-term memory handles the now
Long-term memory handles the forever

The real power comes from how you connect the two

Shadow Deployments for AI Agents: Test in Prod without breaking anything 🚀

Amar Dhillon — Thu, 23 Apr 2026 05:39:39 +0000

If you’ve worked with AI agents in production, you already know one thing. Deploying a new version is not the same as deploying traditional software

With non AI systems, you push code and then run tests. If everything looks fine then you go live

With agents, things get messy. The same input can produce slightly different outputs. Improvements in reasoning might come with unexpected side effects. Sometimes a “better” model performs worse in edge cases that actually matter

So the real challenge is not building a better agent. The challenge is proving that it’s better before users see it 🔍

Why Traditional Deployment Fails for Agents 🤔

The core issue is that agent behavior is not deterministic. You can’t rely on a handful of test cases and assume production will behave the same way. Even if your offline evaluations look great then real users can bring unpredictable inputs, messy context and ambiguous intent

This means a direct rollout is risky. If something goes wrong, it’s not always obvious. it can give:

Slightly worse answers
Slightly more hallucinations
Slightly longer responses that annoy users

By the time you notice, the damage is already done 😬

The Idea Behind Shadow Deployments 🧠

As shown in the above diagram, instead of replacing your current agent (V1) you run the new version (V2) alongside it

The user sends a request and your system (Orchestrator in this case) does something interesting behind the scenes

The stable agent handles the request as usual and returns the response to the user
At the same time, the new agent (V2) receives the exact same input but its output is never shown to the user. It just runs quietly in the background 🏃🏻‍♂️

This is what I call a shadow path 👻

You are effectively replaying real production traffic through your new agent without exposing any risk. The _user experience _remains unchanged but you now have a way to observe how the new version behaves under real conditions

What Actually Happens Under the Hood? ⚙️

At the center of this setup is an orchestrator; It takes incoming requests and sends them down 2 paths

The first path is the live path, which goes to your stable agent. This is the version you trust. It produces the response that the user sees

The second path is the shadow path. This goes to your canary agent which is the version you’re testing. It receives the same input often with the** same context and knowledge sources** but its output is held back

Its important to note that, to make this comparison meaningful, both agents typically rely on the same knowledge base. If one agent had access to different data, you wouldn’t know whether the difference in output came from better reasoning or just better information. Keeping the data layer consistent ensures you are comparing apples to apples 🍎

Comparing Outputs Is Where the Magic Happens ⚖️

Now comes the tricky part. How do you decide which output is better?

You could try to define strict rules, but language is messy. Quality is subjective. What looks better to one evaluator might not look better to another

This is where the idea of using an LLM-as-a-judge comes in. A reasoning model can evaluate both responses and decide which one is more accurate or more aligned with the user’s intent

Over time, you start collecting signals

Maybe the new agent wins 65% of the time
Maybe it’s more accurate but slightly slower
Maybe it handles complex queries better but struggles with short factual ones

All of this gets logged and analyzed 📊

Turning Observations Into Decisions 🔁

After running this setup for a while, patterns begin to emerge. You can see latency differences, cost implications and even qualitative improvements in reasoning.

At this point, promoting the canary is no longer a risky move;It becomes a controlled decision

If the new agent consistently performs better and meets your criteria, you promote it to production. The canary becomes the new stable version and the cycle continues

Things That Still Need Careful Thought ⚠️

Shadow deployments are powerful but they are not free
Running two agents in parallel increases cost, so many teams sample traffic instead of shadowing everything
Latency also needs to be isolated so the shadow path never slows down the user response
Evaluation quality is another challenge. LLM-as-a-judge works well, but it can be inconsistent. Many teams improve this by combining automated evaluation with occasional human review
Observability becomes critical. You need to track inputs, outputs, context, and decisions in a structured way. Without that, you are just collecting noise

The Bigger Picture 🧩

If you are serious about building production-grade AI agents this is not just a nice-to-have pattern

It’s one of the foundational pieces that makes everything else possible 🚀