Forem: Varsha Das

AI Agents Don't Crash. They Drift. Here's the Framework to See It.

Varsha Das — Wed, 20 May 2026 18:03:41 +0000

The scariest AI agent failures don't trigger alerts. They look like success. Here's a 7-dimension resilience framework for building trust in agentic systems — based on the AWS Architecture Blog's approach to resilient generative AI agents.

💡 What this post covers: Why code getting cheap now creates a very big trust crisis, why the resilience patterns we have built for decades don't work for AI agents, and the 7-dimension framework I use to reason about trust in agentic systems.

A few months ago, a developer told me a story that I haven't stopped thinking about.

Her team had shipped an AI agent built in some 2 weeks, basically — that processed customer support tickets, classified them by urgency, and routed them to the right team. The demo was great. Stakeholders loved it. It went to production.

Two weeks later, someone on the receiving end asked:

"Hey, has something changed with the routing? I'm getting tickets that make no sense for my queue."

They checked their dashboards. Everything was green.

But something was wrong. The agent had been confidently routing tickets to the wrong teams for days.

Not all of them, just enough to confuse, but not enough to trigger an alarm.

That story just got me thinking so much that when I dug into it, there was no way of knowing how long the drift had been happening.

Yes, the drift, that itself is the main caveat.

The system looked healthy. The output was broken. And there was no framework for how to think or anticipate this kind of failure.

This blog is about that framework.

Let's dive right in……

The Tax on Ideas Just Hit Zero

For most of the history of software, there was and always has been a "significant" tax on ideas. You had an idea, and then you spent days or weeks or months turning it into working code.

The tax was high enough that most ideas died in a backlog.

You triaged ruthlessly.

You picked the three things that mattered most and let everything else pile up in the JIRA boards. (Much to the dismay of the Jira board owners, haha)

So that tax? It just hit zero.

AI agents can generate dozens of PRs overnight — building code, features, and entire systems. The gap between having an idea and seeing it built has effectively collapsed.

When code generation becomes nearly free, the bottleneck shifts:

from implementation to orchestration,

from writing to judgment,

from building to operating.

But here's what nobody tells you: when you can build code at the speed of thought, deploying that code to production becomes the bottleneck.

A system can be assembled at the speed of thought.

Trust is earned at a different pace entirely.

When Systems Fail Without Breaking

Last month, a Cursor agent deleted a company's entire production database.

This agent running Anthropic's Claude Opus 4.6 deleted PocketOS's entire production database — plus all backups — in nine seconds.

The agent was working on a routine task in a test environment when it hit a credentials problem. Instead of stopping, it found an API token in an unrelated file, a token that carried full account-wide permissions including destructive operations, and issued a single command that wiped everything.

No confirmation prompt. No warning. No check that it was targeting production instead of test.

Railway's backup model stored volume-level backups inside the same volume — so when the volume went, the backups went with it. The most recent recoverable backup was three months old.

When the founder asked the agent to explain itself, it produced what he called a "written confession": "I guessed instead of verifying. I ran a destructive action without being asked. I didn't understand what I was doing before doing it."

Two layers of guardrails — Cursor's published safety rules and the company's internal safety instructions — both told the agent not to do exactly what it did. Both failed at the same time.

The internet blamed the AI. But the real failure was an over-permissioned token sitting in a file the agent could read, paired with infrastructure that collapsed when the volume did.

That was a loud failure. Dramatic. Viral. Obvious.

The scarier ones? They're silent.

Consider an enterprise AI assistant designed to summarise regulatory updates for financial analysts. Every morning, this assistant retrieves documents from internal repositories, synthesizes them using a language model, and distributes summaries across internal channels. Technically, everything works.

But over time, something slips.

An updated document repository hasn't been added to the retrieval pipeline.

The assistant keeps producing summaries that are coherent and internally consistent — but they're increasingly based on obsolete information.

Nothing crashes. No alerts fire. Every component behaves as designed.

The problem is that the overall result is wrong.

From the outside, the system looks operational. All your monitoring dashboards read "healthy." Latency is fine. Error rates are zero.

Analysts are making decisions based on outdated regulatory information, and nobody knows. Catastrophic disaster for the business.

When humans wrote all the code, they at least understood what they shipped.

When agents generate it, the gap between "it works" and "I understand why it works" becomes the attack surface.

I've started calling these "green-dashboard failures." The kind where every metric says you're fine while the system is quietly betraying the people who depend on it.

Why the Patterns We Know Don’t Work Here

To understand why this is such a big threat, I need to take you back to how we've always built resilient systems.

Because the patterns we know, the ones we've built entire engineering practices around, they kind of break down here.

Over decades, we built resilience into three layers:

Infrastructure resilience. We deploy across multiple availability zones, auto-scale on demand, and load balance traffic — so if hardware fails, the system stays up.

Data resilience. We use read replicas, automated failover, and connection pooling — so if a database goes down, we don't lose data or availability.

Application resilience. We write circuit breakers, retry logic, and graceful degradation — so if a service fails, the app handles it predictably instead of crashing.

These patterns assume something fundamental: failures are binary.

A service is working or it's broken.

A sensor responds or it doesn't.

A constraint is met or it triggers a shutdown.

But AI agents don't crash. They degrade silently. They hallucinate confidently.

They might drift without a single metric turning red.

Autonomous Systems Behave Differently

While building and observing agentic systems for the past year, I see three things that make them fundamentally different from the software we've built for decades:

1. Continuous reasoning loop. They reason in loops, not steps. Unlike traditional request-response software, agents observe, think, and act in an ongoing cycle — always changing their own context.

2. Contextual inappropriateness. They produce output that is syntactically perfect but semantically wrong for the situation. A hallucinated paragraph looks like a real answer. A wrong tool call looks like a right one — until you trace what happened downstream.

3. Behavioral drift without errors. Small mistakes compound. The system gradually moves away from correct behaviour without any single step triggering an alarm.

It's not a cliff — it's a slow incline you don't notice until you're in the wrong valley.

This is why traditional resilience patterns break down.

So, we need a new framework.

The 7-Dimension Resilience Framework

Here's how we should think about building trust in agentic systems.

There are 7 dimensions you need to reason about and for each one, you ask: which failure modes apply here?

Foundation Models — Your LLM choice: self-hosted (you handle failover), managed or serverless. Each shifts resilience responsibility differently.

Something very basic like — If your model provider has a bad day, does your entire system go dark?
Agent Orchestration — The conductor. How agents coordinate, select tools, and escalate to humans. This is the brain — and if the brain makes a bad decision, the hands execute it perfectly.
Infrastructure — Where agents run: EC2, ECS, or a managed runtime like Bedrock AgentCore. If a container crashes, this layer handles the restart. The boring stuff that isn't boring when it fails.
Knowledge Base — Vector DBs, embeddings, RAG pipelines. If retrieval fails, your agent is answering questions without being able to look anything up. It doesn't know it's blind. It just confabulates.
Agent Tools — External dependencies: APIs, MCP servers, memory, prompt caching. What happens when that inventory API goes down? Does your agent wait forever, or does it move on?
Security & Compliance — Auth, guardrails, content validation. Prevents your agent from doing things it shouldn't — like leaking customer data or executing destructive actions without human approval.
Observability — Metrics, traces, reasoning logs. If you can't see why your agent made a decision, you can't fix it when it goes wrong.

That's the framework. 7 dimensions. Each one a surface where your agent can silently fail.

But knowing where things can break is only half the picture. The other half is knowing how they break — the specific failure modes, what they look like at 3 AM, and how to defend against each one.

In Part 2, I break down all 5 silent failure modes — with real-world case studies (including an agent that deleted a production database in 9 seconds) and the exact defenses for each.

This post is based on and extends the resilience framework from the AWS Architecture Blog: Build Resilient Generative AI Agents.

📺 More from the series:

Have you seen your agent "drift" without any metric catching it? How long before someone noticed? Drop it in the comments — I'll respond to every one.

🚀 How Developers Can Stop Pretending to Understand AI Buzzwords

Varsha Das — Thu, 11 Dec 2025 19:02:59 +0000

> If you can't explain it simply, you don't understand it well enough.
 - Albert Einstein

You know that feeling when someone starts talking about "agentic AI workflows with RAG pipelines and vector embeddings" and everyone's nodding like they totally get it? Yeah, I was that developer pretending to understand while feeling so lost within.

A few months ago, I hit my breaking point. Every dev thread, every tech talk—just buzzwords with zero actual clarity. So I stopped faking it and decided to actually learn things from scratch.

Plot twist: turns out most people are faking it too.

The Research Paper Rabbit Hole

My first move? Dive into IBM research papers. Because that's what good developers do, right? Don’t get me wrong — they’re good. Like, really good. I referred to them while writing this because of the kind of thorough, well-researched content that builds solid foundational knowledge. But, they’re dense. My brain started to explode after reading 1 paper.

Next stop: YouTube. Surely someone had figured out how to explain this without requiring a PhD? And yes, there's brilliant content out there. But here's the problem: you watch one video on transformers, another on embeddings, then someone casually mentions "attention mechanisms" and suddenly you’re like “wait, how does this connect to what I learned yesterday?”

I am pretty sure many of you would have been there.

And somewhere in the middle of all this chaos, I just thought: “Can someone PLEASE just give me one clean map? Like, all of it. In one place. That actually makes sense?”

So… I made one.

A humble attempt to build one, I would say.

Key Takeaways

You'll finally get a plain-talk view of AI terms that often feel too dense or too "expert-only."
You'll learn how the basics link together — models, prompts, safety, and the stuff that holds the whole AI stack in place.
You'll understand why prompts matter so much, why they sometimes go wrong, and what people do to keep them on track.
You'll get a sense of how machines learn, how they pull info, and how this leads to better answers.
You'll see the flow from simple chat systems to tools, tasks, and full-on AI helpers that can act on your behalf.
You'll be able to read AI threads, posts, papers or videos without feeling lost or drained.

So yeah… sit with a paper and pen, take notes if you want, maybe read this on your laptop, and slowly absorb. No rush.

Before We Start

If you're completely new, just make sure you've heard of these concepts:

Core Concepts

Neural Networks — The brain-inspired structure that powers modern AI, consisting of interconnected nodes that process information
Deep Learning — Using many layers of neural networks to learn complex patterns from large amounts of data
Natural Language Processing (NLP) — Teaching computers to understand, interpret, and generate human language
Machine Learning — The broader field where computers learn patterns from data without being explicitly programmed for every scenario
Training Data — The collection of examples used to teach AI models patterns and relationships
Model — The trained AI system that has learned patterns and can make predictions or generate outputs
Algorithm — The mathematical rules and procedures that guide how a model learns from data
Pattern Recognition — The ability of AI systems to identify recurring structures, relationships, and trends in data
Prediction — How trained models generate outputs by using learned patterns to make informed guesses about what comes next
Inference — The process of using a trained model to generate outputs or make decisions on new, unseen data

The Four Phase Journey

The Four-Phase Learning Framework

Instead of drowning in terminology, here's how AI concepts actually connect:

🎯 Phase 1: The Foundation — How AI Learns

First, the LLM must learn through training. This happens in three fundamental ways:

Supervised learning — labeled examples
Self-supervised learning — predicting missing pieces in unlabeled data (how modern LLMs are trained)
Reinforcement learning — trial-and-error with feedback

The Training Pipeline

During this training phase, the model processes massive amounts of text by:

Breaking it into tokens
Converting tokens into embeddings
Using attention mechanisms to understand which parts matter most
Building patterns across transformer layers that capture complex relationships

Tokenization breaks text into processable units, embeddings convert those tokens into numerical vectors in high-dimensional space, and self-attention mechanisms within transformer architectures determine which tokens matter most for context.

To make models production-ready, we use:

Distillation — shrinking big models into smaller ones
Quantization — reducing numerical precision from 32-bit to 8-bit or 4-bit to run faster on resource-constrained devices

🔍 Phase 2: Knowledge Retrieval — Bridging Training and Real-Time Access

Once trained, models need efficient ways to access information during inference. This is where semantic search and vector databases become critical.

How Semantic Search Works

Unlike traditional keyword matching, semantic search understands meaning:

Searching "smartphone" also retrieves "cellphone" and "mobile devices"
These concepts live close together in vector space

Vector Databases

Vector databases store data as high-dimensional numerical arrays, enabling lightning-fast similarity searches essential for real-time AI applications.

This retrieval capability forms the bridge between what models learned during training and what they can access when answering your questions—the foundation for everything that follows.

💬 Phase 3: User Interaction — Prompts, Safety, and Inference

Prompts are your interface for communicating with AI. When you submit a prompt, the model:

Tokenizes it
Converts tokens to embeddings
Generates responses one token at a time through inference
Calculates probabilities for potential next tokens
Outputs the most likely one

Prompt Engineering Techniques

Zero-shot — no examples provided
Few-shot — providing sample outputs
Chain-of-thought — step-by-step reasoning

Safety Considerations

However, prompts introduce risks:

Hallucinations — fabricated responses not grounded in training data
Prompt injection — malicious instructions disguised as user input

This is why guardrails—safeguards operating across data, models, applications, and workflows—are essential to keep AI systems safe, responsible, and within defined boundaries in production environments.

🤖 Phase 4: Advanced Applications — RAG, MCP, and Autonomous Agents

Now everything converges into autonomous systems.

RAG (Retrieval-Augmented Generation)

RAG solves the knowledge cutoff problem by:

Converting your question into vector embeddings
Performing semantic search across vector databases
Feeding retrieved information to the LLM as additional context

This enables AI to work with proprietary data or recent information without expensive retraining.

MCP (Model Context Protocol)

MCP provides the universal language for AI-tool communication, standardizing how agents:

Discover tools
Request data access
Execute actions safely
Receive results

Think of it like REST APIs but designed specifically for AI systems.

AI Agents & Multi-Agent Systems

AI Agents are autonomous systems that:

Break complex goals into subtasks (planning)
Use external tools to gather missing information (reasoning via RAG and MCP)
Make decisions and take actions independently
Learn from past interactions

Agentic AI is the umbrella term for this paradigm shift—AI that exhibits agency, acting independently rather than just answering questions.

Multi-agent systems represent the cutting edge: multiple specialized agents working together, each handling specific roles like input validation, business logic, data operations, and system monitoring.

🔄 The Complete Flow

Learn (Foundation) → Store (Vector Databases) → Retrieve (Semantic Search) → Apply (RAG + Prompts) → Act (MCP + Agents)

This isn't just terminology—it's the architectural pattern production AI systems are built on today, where models evolve from statistical predictors into autonomous problem-solvers that reason, plan, and execute complex workflows.

How Each Phase Connects

Phase 1 establishes the learning foundation (tokenization → embeddings → attention → transformers, plus the three learning types and optimization techniques)
Phase 2 bridges training to real-time access (semantic search and vector databases as the retrieval layer)
Phase 3 covers the interaction layer (prompts, inference, safety concerns, and guardrails)
Phase 4 brings it all together (RAG, MCP, and agents as the autonomous execution layer)

Each phase flows naturally into the next, creating a comprehensive understanding of how modern AI systems work from training to autonomous execution.

Stop Pretending. Start Understanding.

No more nodding along in meetings. No more feeling lost in AI discussions. Get the complete picture that connects every dot from tokens to autonomous agents.

This overview gives you the mental map, but each phase has layers of complexity that make the difference between "getting it" and actually understanding it.

I've written a 22-min comprehensive guide that breaks down every concept in more details and shows how they interconnect.

Read the full guide on Medium →

Stop pretending you understand AI buzzwords. Get the complete picture NOW!!

Found this helpful? Follow me for more developer-friendly AI content that actually makes sense.