Forem: Yaohua Chen

Write, Install, or Generate: A Practical Guide to Agent Skills

Yaohua Chen — Fri, 17 Apr 2026 02:01:15 +0000

A plain-English guide to Agent Skills — what they are, how they differ from MCP, and the three ways to source one: write, install, or generate.

If you use Claude at work, you probably have a running tab of context you paste into every session: your team's naming conventions, the testing library you prefer, that one internal helper Claude keeps forgetting. You copy. You paste. You remind. And you do it all again next week.

Agent Skills are Anthropic's answer to that fatigue. Announced in October 2025 and released as an open standard that December, they're now supported across Claude API, Claude Code, Cursor, VS Code Copilot, Cline, and more than two dozen other coding agents. The idea is simple: teach an agent something once, then reuse that knowledge everywhere — without bloating your prompts or your token bill.

This guide explains what a skill is, how it differs from MCP (the other acronym you'll hear in the same breath), the three ways to get one — write, install, or generate — and two patterns for scaling beyond a single skill once you have a few.

What a skill actually is

A skill is a folder with a markdown file inside. The file — SKILL.md — contains two things: a short description that tells Claude when to use the skill, and a longer body with the actual instructions.

Think of it as a recipe card. When you ask Claude to bake bread, it pulls the card titled "here's how we bake bread in this kitchen." When you ask for something unrelated, the card sits in the drawer untouched. The point is that the card isn't in Claude's working memory until it's needed.

That's the difference between a skill and a big system prompt. A system prompt is the entire cookbook handed to Claude at every meal, even when you only want toast. A skill is one recipe pulled out on demand. Anthropic documents each idle skill as costing roughly 100 tokens of metadata — enough for Claude to know the skill exists without paying for its full content.

That math matters once you have a handful. Twenty skills at ~100 tokens each is 2,000 tokens of fixed overhead no matter how long each skill actually is. The same twenty rules dumped into a system prompt would weigh in at tens of thousands of tokens every turn.

Skills vs. MCP: the recipe vs. the pantry

The other term you'll hear is MCP — the Model Context Protocol. People often treat skills and MCP as competing ideas, but they solve different problems.

MCP is the live connection between Claude and your data: query a Jira ticket, read a Google Doc, fetch current Stripe API docs. It's the pantry — where the fresh ingredients live.
A skill is a reusable set of instructions: "when you're writing a React component, follow these rules." It's the recipe — how you combine ingredients, consistently, every time.

Here's a side-by-side view:

Feature	MCP	Agent Skill
Purpose	Connect Claude to live data or tools	Teach Claude a repeatable procedure
Cost	Per call; fetched data stays in context	~100 tokens idle; full body loads on demand
Lifetime	What you fetch stays for the session	Stored locally; version-controlled in git
Best for	"What's the latest Drizzle syntax?"	"How we always write our tests here"

They aren't competitors. Most real workflows use both — MCP pulls the live docs; a skill teaches Claude how your team adapts them.

The anatomy of a skill

Every skill uses a layered structure Anthropic calls progressive disclosure:

Metadata — always loaded. A short header at the top of SKILL.md that says who the skill is and when to trigger it. About 100 tokens.
The body — loaded when triggered. The markdown instructions Claude reads once the description matches your task.
Reference files — pulled in only if the body points to them. Supporting docs, checklists, example code.

A minimal skill looks like this:

---
name: acme-pr-style
description: Use when drafting a pull request description. Enforces Acme Corp's PR template and ticket-linking rules.
---

# Acme PR Style

- Start every PR title with a ticket ID like `[ACME-1234]`.
- The body must have three sections: **Summary**, **Changes**, **Test plan**.
- Never merge without at least one linked Linear ticket.
- Use "we" voice, not "I" voice.

That's the whole skill. The block between the --- lines is YAML, and Claude uses the description to decide whether to activate the skill when you type a request. Once active, the body becomes a hard rule for that conversation.

The Anthropic spec requires only two frontmatter fields — name and description — and Claude Code adds a small handful of optional ones (for example, user-invocable: true, the default, controls whether the skill also appears in the / slash-command menu). You don't need anything beyond the two required fields for your first skill.

Build your first skill in five minutes

Let's walk through the PR-style skill end to end.

Step 1. Create the folder.

In your project root, add:

.claude/skills/acme-pr-style/
└── SKILL.md

Step 2. Write SKILL.md.

Copy the example from the previous section — swap acme for your team name and replace the rules with yours.

Step 3. Ask Claude to use it.

Start Claude Code in that directory and ask something that matches the trigger:

"Write the PR description for my current branch."

Claude scans the active skills, notices the description matches "drafting a pull request," and silently loads the body. In Claude Code you'll see a confirmation like:

[Skill loaded: acme-pr-style]

Your PR description now follows the template.

Step 4. Iterate on the description.

If Claude doesn't pick up the skill, the culprit is almost always the description field. It's the only signal Claude has when deciding to activate. Vague descriptions ("coding standards") rarely trigger. Task-shaped descriptions ("Use when drafting a pull request description") do. A useful rule: phrase it like you're writing a job posting — state the trigger condition first, then the outcome.

Step 5. Share it.

Commit .claude/skills/acme-pr-style/ to your repo. Every teammate who checks out the repo automatically gets the skill — no install step, no sync service. That's the quiet win here: the rules live with the code. When you bump your PR template, you bump the skill in the same commit, and Claude stays aligned with your current conventions instead of the ones from six months ago.

You don't have to write every skill from scratch

You just hand-wrote one, and that's the most direct path. Before you do it for everything, though, it helps to know that hand-writing is one of three ways to get a skill — and that the other two are usually faster when they apply.

Write your own for everything specific to your team — naming conventions, internal libraries, security requirements, release workflows. This is the irreducible kernel: nobody outside your team can produce it.
Install one somebody else wrote from a community source. These come in three flavors, in order of how curated they are:
- Curated CLI registry — small but vetted, install via command. skills.sh (Vercel Labs, early 2026) is the canonical example: npx skills find to search, npx skills add <pkg> to install.
- Curated "awesome" list — a GitHub README organized by category; copy or clone manually. awesome-claude-skills (maintained by Composio) is the largest, grouped by use case: document processing, dev tools, data analysis, app automation, and so on.
- Search-driven aggregator — auto-indexes hundreds of thousands of skills from GitHub with AI-assisted search and one-click install. SkillsMP lists 900K+ across Claude Code, Codex, and ChatGPT.
Generate one from live documentation when you're picking up a third-party library and don't know it well enough to write the rules yourself. Context7's wizard (npx ctx7 skills generate) does this — covered in the next section.

Rule of thumb: write for internal rules, install for shared community patterns, generate for third-party library knowledge. Note: curated sources are higher signal but smaller; aggregators have everything but you should read each SKILL.md before installing — skills can ship scripts the agent will execute, which means you should verify the code is safe to run.

Generate skills from docs with the Context7 wizard

Writing a skill for your team's conventions is one thing — you already know the rules. Writing a skill for a third-party library is harder, because you have to know the library well enough to capture its current best practices, the patterns that are deprecated, and the mistakes you want the agent to avoid. Most of us aren't that fluent with the SDK we adopted last week, and the docs keep moving. So skills for external libraries often don't get written, or get written from stale memory and quietly drift.

Context7 ships a CLI workflow specifically for this gap. Run:

npx ctx7 skills generate

and you get an interactive wizard that turns Context7's live documentation index into a scoped skill in five steps:

Describe the expertise — Clerk authentication, Drizzle migrations, Tailwind v4 theming. Frame it as the domain you want the agent to be expert in, not the task you want it to do.
Pick the sources — the wizard searches Context7's library and shows matching documentation sets. You confirm which ones it should treat as ground truth.
Answer the scoping questions — it asks targeted clarifications: which framework you're on (Next.js, Remix, Astro), what stage you're at (initial setup, hardening, migration), which slice of the API you care about (sign-in/sign-up, social SSO, organizations).
Review and refine — the wizard queries Context7 for the latest docs, drafts the skill, and shows you exactly which snippets it pulled from. If something's off, you describe what to change and it regenerates while keeping what you liked.
Install — pick the targets. The wizard detects Claude Code, Cursor, Codex, OpenCode, Amp, and Antigravity, and writes the skill into the right folder for each — or all of them with --all.

What you end up with isn't a generic library wiki. It's a focused SKILL.md that answers the exact question you scoped — say, "set up sign-in and sign-up in a Next.js App Router app with Clerk." It typically includes where the provider component goes, how the middleware should be wired, the required environment variables, and, usefully, the wrong patterns the official docs explicitly warn against.

The non-obvious win is the scoping. Instead of one omnibus clerk skill that tries to cover everything, you re-run the wizard for each concern: one skill for sign-in/sign-up flows, another for user management and profiles, another for social SSO. Each is narrow enough to have a sharp description, which means each triggers precisely when relevant. The agent loads the auth-flow skill while you're wiring login pages, and the profile skill while you're building the account screen — never both at once, and never the wrong one.

A reasonable heuristic: reach for the wizard when you're adopting a library you don't know intimately, or when you suspect the model's training data is older than the version you're on. Keep writing your own skills for the rules only your team knows — internal libraries, naming conventions, security requirements. The wizard is a great way to get a library skill. It can't write your culture skill.

Compose skills into new skills

Once you have a few skills, the next unlock is treating them as building blocks. A skill can invoke other installed skills as subroutines — running them in sequence, in parallel, or both — and combine the results with its own work to produce something neither could on its own.

A concrete example from my own toolkit is a code-review-report skill that runs two independent review passes against the same diff and consolidates them into a severity-tiered report. The two passes compose differently:

A convention-based pass the skill runs itself. It reads the project's CLAUDE.md and a per-language checklist, and reviews each file against those rules. For large diffs the skill fans this out across subagents — up to ten files per subagent — so the diff never has to fit in a single context.
An adversarial pass done by invoking another installed skill: /codex:adversarial-review from the Codex plugin. It runs the same diff through a different model (Codex) playing the skeptic, looking for bugs, security issues, and architectural risks the convention pass might miss.

Both passes run in parallel. When they return, code-review-report consolidates them:

Deduplicate. If both reviewers flag the same issue, it becomes a corroborated, higher-confidence item tagged [Both]. Disagreements are surfaced rather than hidden.
Classify and format. Group findings into a severity-tiered report (Critical / High / Medium / Low / Nits), annotate each with its source ([Claude], [Codex], or [Both]), and append lint output.

The result is a single report backed by two independent perspectives — something meaningfully different from either pass alone.

The compounding payoff is the point. Claude and Codex have different training and different blind spots, so running both against the same diff catches issues either model alone would miss. The [Both] tag turns agreement into signal — when two independent reviewers flag the same issue, the team can triage with much higher confidence than they could from either alone. Disagreements stay visible too, which is itself useful: a finding only one reviewer raised tells you something about the issue's character (model-specific blind spot, ambiguous case, judgment call worth a human discussion).

The pattern generalizes. Any time you catch yourself running two or three skills by hand in the same order — research, then draft, then critique; lint, then test, then summarize — that sequence is itself a skill. Write a new SKILL.md whose body tells Claude "run skill X, then run skill Y, then consolidate the outputs like this." You get reuse, a shareable workflow, and the same ~100 tokens of idle overhead as any other skill.

Fan out to subagents inside one skill

Composition isn't the only way to scale a skill. A skill can also dispatch its own subagents — short-lived workers, each with a fresh context, all running in parallel — and consolidate the results when they return. This is the move when one skill needs to do work that's too big for a single context window or has natural parallelism inside it.

code-review-report does this for its convention-based pass. Reviewing every file in a large diff against CLAUDE.md and a per-language checklist would either overflow context or grind serially. Instead, the skill splits the changed file list into batches of up to ten files and dispatches one subagent per batch. Each subagent loads the same instructions but only its own slice of the diff, runs the mechanical and semantic checks, and returns structured findings. The parent skill collects all batches and merges them into the consolidation step.

Three things make subagent fan-out earn its complexity:

Context economics. Each subagent has its own fresh context. The parent skill never has to hold the whole diff at once — only the consolidated findings, which are typically orders of magnitude smaller than the raw input.
Real parallelism. Subagents run concurrently on the wall clock, not just logically. Reviewing thirty files across three subagents of ten finishes roughly three times faster than one subagent grinding through all thirty.
Isolation. A subagent can't contaminate another with framing from an earlier file or half-formed conclusions. Each one sees its slice cleanly.

The fan-out can take two shapes, and both are worth knowing:

Partition by data (what code-review-report does) — same work, different slice of input. Each subagent runs the same instructions on a different chunk of the diff. Best for naturally divisible inputs: file batches, record windows, time ranges, regions of a document.
Partition by concern — different work, same input. One subagent specializes in security, another in performance, another in test coverage; they all see the full input but each looks for something different. Best when concerns are independent and benefit from a dedicated reviewer rather than being squeezed into one pass.

The trade-off is real: subagents cost tokens (each one re-loads its instructions and partial context) and add orchestration overhead. Fan out only when the work is divisible and large enough that the alternatives — overflowing context, running serially — are worse. For a five-file diff, the parent context is fine. For a fifty-file diff, fan out.

Takeaways

Skills are recipe cards. Markdown files Claude reads only when they match your task, at ~100 tokens of idle overhead each.
Skills are not MCP. Skills are reusable procedures; MCP is live data access. You'll likely use both.
Skills live in your repo. When the rules change, commit the change. Claude reads the latest version automatically, across every teammate.
Generate library skills, write team skills. Use npx ctx7 skills generate to spin up scoped skills for third-party libraries from current docs. Write your own for the rules only your team knows.
Two ways to scale beyond one skill. Compose other skills as building blocks, or fan out to subagents inside a single skill. Use the first when the pieces are already separate skills; use the second when the work inside one skill is too big for one context.

If you've caught yourself pasting the same block of context into Claude twice this week, that block is already your first skill. The rest is copying it into a SKILL.md file.

Appendix — A developer's deeper look

For readers who write code, here's what a skill looks like once it grows up.

Recommended directory layout

.claude/skills/acme-typescript/
├── SKILL.md                 # Metadata + core rules
├── references/
│   ├── naming-conventions.md
│   └── standard-patterns.ts # "Good" vs "Bad" code examples
└── templates/
    └── api-route.ts.template

The references/ folder is Level 3 of progressive disclosure. The body of SKILL.md mentions these files by path, and Claude opens them only when it actually needs the example — keeping the active context small until the moment you need the detail.

A realistic `SKILL.md`

---
name: acme-typescript
description: Enforces Acme Corp's strict TypeScript 5.x standards, including Zod validation at boundaries and the internal Result<T, E> error pattern. Use when writing or reviewing any .ts file.
user-invocable: true
---

# Acme TypeScript Standards

## Type safety
- No `any`. Use `unknown` with a type guard.
- Boundary data (API, file, env) must be validated with Zod.
- Derive the TypeScript type from the schema: `type X = z.infer<typeof XSchema>`.

## Error handling
- Use `Result<T, E>` from `./references/standard-patterns.ts`.
- Never throw for expected business errors; return `err(...)` instead.

## Reference
- See `./references/standard-patterns.ts` for the canonical implementation.

ultrathink

This is the culture side of the divide from earlier — it codifies Acme's internal Result<T, E> pattern, which no wizard could know about. The library counterparts you'd pair it with — say zod-runtime-validation or typescript-strict-mode — are exactly the kind of skill npx ctx7 skills generate would draft for you from the official docs in a couple of minutes.

That last word — ultrathink — is a real Claude Code trigger. When it appears anywhere inside a skill, Claude allocates its extended-thinking budget (roughly 32,000 tokens) whenever the skill is active. Use it for skills that enforce expensive or subtle constraints where quiet mistakes are costly.

A reference file

references/standard-patterns.ts:

export type Result<T, E = Error> =
  | { ok: true; value: T }
  | { ok: false; error: E };

export const ok = <T>(value: T): Result<T, never> => ({ ok: true, value });
export const err = <E>(error: E): Result<never, E> => ({ ok: false, error });

When Claude is asked to edit a service file, it reads the skill, sees "use the pattern at ./references/standard-patterns.ts," opens that file once, and applies the pattern consistently across every change in the session. That's what progressive disclosure buys you: one source of truth, loaded only when relevant.

References

Self-Evolving Agents: A Developer's Guide

Yaohua Chen — Mon, 13 Apr 2026 18:54:40 +0000

Static agents hit performance ceilings. This guide shows you how to build agents
that improve themselves — through prompt optimization, dynamic skill libraries,
code and harness evolution, RAG, and LLM fine-tuning — and how a unified LLM
judge decides which track to take. Along the way, we'll survey the frameworks
and methodologies — from DSPy to autoresearch to TextGrad — that have turned
these ideas into working code.

1. Introduction

Most production agents are frozen at deployment. Their system prompt is fixed, their tools are hardcoded, and when they fail, a human manually intervenes. This works until it doesn't — and it usually stops working the moment the task distribution shifts or edge cases accumulate.

Self-evolving agents close this loop automatically:

They evaluate their own outputs
They diagnose failure modes
They improve the right layer — prompt, skill, code, knowledge, or model weights

This is not a theoretical concept — in 2026, the field often refers to these patterns as recursive optimization or self-distillation. Several open-source frameworks have already shipped working implementations: OpenAI's Self-Evolving Agents Cookbook automates prompt improvement through graders and metaprompt agents. Karpathy's autoresearch lets an agent rewrite its own training code overnight. DSPy compiles optimal prompts via Bayesian search and can distill them into smaller model weights. TextGrad treats the entire agent as a differentiable program, using textual gradients to patch failure modes. And frameworks like AgentScope close the loop all the way to automated fine-tuning from production data.

This guide covers five escalation levels in order of cost and commitment:

Level 1 — Prompt tuning              (minutes, free)
     │  still failing after 3 rounds?
     ▼
Level 2 — Add/improve skills         (hours, cheap)
     │  still failing on reasoning/architecture?
     ▼
Level 3 — Code & Harness evolution   (hours, cheap — runs overnight)
     │  still failing on knowledge?
     ▼
Level 4 — RAG                        (hours, medium cost)
     │  still failing on reasoning style/pattern?
     ▼
Level 5 — LLM Fine-tuning            (days, expensive)

Each section builds toward a master LLM judge pipeline in Section 9 that automatically decides which track to trigger — and calls the right code to execute it.

2. The Landscape: Frameworks for Self-Evolution

Before building from scratch, it is worth understanding the frameworks that have already solved pieces of this problem. They share the same core loop — run, evaluate, improve, repeat — but differ in what they evolve (prompts, code, skills, or model weights), how they score, and what safety model they use.

2a. OpenAI Self-Evolving Agents Cookbook

The most production-oriented of the four. It addresses the scenario every developer has experienced: an LLM-powered agent that works reasonably well but keeps failing on certain inputs, leaving you stuck in a never-ending cycle of prompt tweaking.

What evolves: The system prompt (the instructions given to the LLM). A VersionedPrompt class tracks every revision with timestamps and eval scores, so rollback is always one line away.

How it scores: Multiple graders run in parallel — Python functions for deterministic checks (keyword presence, length deviation), cosine similarity for semantic fidelity, and an LLM-as-judge for nuanced quality. A metaprompt agent reads grader feedback and rewrites the system prompt automatically. The loop continues until scores pass or a retry limit is hit.

Going further: The cookbook also supports comparing model versions (e.g., GPT-5 vs GPT-5-mini) to find the best model-prompt combination, and demonstrates GEPA (Genetic-Pareto) optimization as an advanced alternative to simple metaprompt rewriting.

2b. Karpathy's autoresearch

Instead of improving prompts, the agent improves actual source code — specifically, code that trains a small language model.

What evolves: A single Python file (train.py) containing the full GPT model, optimizer, and training loop. Everything is on the table: architecture, hyperparameters, optimizer, batch size, attention pattern.

How it scores: A single, hard metric: validation bits per byte (val_bpb). Lower is better. Each training run is limited to exactly 5 minutes of wall-clock time, making experiments directly comparable regardless of what the agent changes.

The key insight: You are not writing training code — you are writing program.md, a Markdown file that instructs the agent. The agent reads your instructions, modifies train.py, runs training, checks if the score improved, and keeps or discards the change. You can expect roughly 12 experiments per hour, or 100 overnight.

2c. autoagent (kevinrgu)

"Like autoresearch but for agent engineering." Instead of optimizing model training code, it optimizes the agent itself — system prompt, tool definitions, agent registry, and routing/orchestration logic.

What evolves: A single-file agent harness (agent.py) containing config, tool definitions, agent registry, and orchestration. An adapter boundary is explicitly marked as fixed; everything else is the edit surface for the meta-agent.

How it scores: Total score produced by benchmark task test suites in Harbor format. Tasks run in Docker containers for isolation. The meta-agent hill-climbs on this score.

Same meta-programming model: Like autoresearch, the human steers the loop through program.md while the meta-agent edits agent.py. The agent runs benchmarks, diagnoses failures, modifies the harness, and iterates.

2d. EvoMap Evolver

If the OpenAI cookbook is about improving prompts and autoresearch is about improving code, Evolver is about improving agent behavior through a formal, protocol-driven process — version control for agent evolution.

What evolves: Structured behavior assets. Genes are reusable improvement patterns (like "add input validation before edits"). Capsules bundle related Genes together for larger changes. Events log every evolution, creating a complete audit trail.

How it scores: Signal-based — scans agent logs for error patterns and uses those signals to select which Gene to apply.

Governance model: Evolver supports multiple operational modes: review mode (human-in-the-loop), continuous loop (autonomous), and strategy presets that steer priorities — innovate (maximize new features), harden (focus on stability), or repair-only (emergency fix mode).

2e. The Broader Ecosystem

The four frameworks above are the ones this guide draws its architecture patterns from, but the self-evolving agent space is broader. Several other systems take fundamentally different optimization approaches worth knowing about.

DSPy (Declarative Self-improving Python). The industry standard for self-improving prompts. Instead of writing prompt strings, you define a Signature (input/output spec) and a Metric (your judgment function). DSPy's MIPRO optimizer uses an LLM to triage failures, propose 10-20 prompt variants, and "compile" the best one via Bayesian search. DSPy can also fine-tune smaller models (e.g., Llama 3) to mimic the reasoning of a larger model by distilling best-performing prompt traces into weights — a technique called self-distillation.

TextGrad (Textual Backpropagation). Published in Nature (2025), TextGrad treats an LLM agent like a neural network but replaces numerical gradients with textual gradients. You define a TextLoss — for example: "The response should be technically accurate and concise; provide feedback if it is too wordy." TextGrad passes this loss back through the agent's execution trace and mutates the system prompt or solution code to patch the specific failure mode the judge discovered. This is particularly effective for hard optimization problems (math, code generation) where failures are diagnosable from the trace.

Memento-Skills. A framework focused on evolving an agent's skill library rather than a single prompt. When an agent encounters a task and fails, an orchestrator evaluates why, then literally rewrites the Markdown and code files for the failing skill. Over time, the agent accumulates a library of refined skills — like learning new moves in a game by trial and error, refining each move's code/instructions after every loss.

AgentScope + Trinity-RFT. Designed for enterprise-scale self-evolution. AgentScope captures production logs via "Inference Tables," and Trinity-RFT uses an LLM judge to label production data as "good" or "bad." The system then automatically kicks off a fine-tuning job using reinforcement learning from feedback (RLHF/PPO/SFT) to update the underlying model weights — closing the loop from production failures to weight updates without manual data curation.

Side-by-Side Comparison

Frameworks covered in this guide:

Dimension	OpenAI Cookbook	autoresearch	autoagent	Evolver
What evolves	System prompt	Source code (`train.py`)	Agent harness (`agent.py`)	Behavior assets (Genes/Capsules)
Evaluation	Multi-grader (Python + similarity + LLM judge)	Single metric (val_bpb)	Benchmark task suites (Harbor)	Log signal scanning
Human role	Define graders and thresholds	Write/iterate on `program.md`	Write/iterate on `program.md`	Choose mode and strategy preset
Safety model	Versioned prompts with rollback	Git keep-or-revert; fixed time budget	Docker isolation; Harbor sandboxing	Command whitelist; scoped execution; audit trail
Best for	Production prompt improvement	Single-file, single-metric optimization	Agent harness optimization	Regulated environments needing audit trails

Additional frameworks worth evaluating:

System	What it evolves	Optimization method	Best for
DSPy	Prompts and weights	Bayesian search / compilation (MIPRO)	RAG pipelines and complex multi-step workflows
TextGrad	Prompts and code	Textual backpropagation	Hard optimization problems (math, code generation)
Memento-Skills	Skill artifacts (Markdown + code)	Reflection and mutation	Long-horizon autonomous agents
AgentScope	Model weights	Online fine-tuning (PPO/SFT via Trinity-RFT)	Production enterprise loops with RLHF

3. Foundations — The Evolution Loop

Every self-evolving agent shares the same feedback cycle:

Agent runs task
      │
      ▼
Evaluator scores output
      │
      ▼
Failure classifier diagnoses root cause
      │
      ▼
Improvement dispatcher triggers the right track
      │
      ▼
Updated agent reruns

Three components make this possible:

Memory — a versioned log of runs, prompts, and scores
Evaluation signal — a judge that tells you how well the agent did
Improvement dispatcher — the logic that routes failures to prompt, skill, code, RAG, or fine-tune

The rest of this guide builds each component in code. All code snippets use Anthropic's Claude (via the Python SDK), but the patterns are model-agnostic — swap in any LLM provider and the architecture stays the same.

Cost optimization tip: The code uses claude-opus-4-6-20260205 throughout for simplicity, but in production you should use different model tiers for different roles. Sonnet 4.6 delivers ~98.5% of Opus performance on routine agent runs (79.6% vs 80.8% on SWE-bench) at 1/5 the cost and 2x the speed. Opus 4.6 pulls ahead decisively on deep reasoning (91.3% vs 74.1% on GPQA Diamond). The practical split: use Sonnet for the agent runner, evaluator, and prompt rewriter (Sections 4a–4c), and reserve Opus for the judge and track recommender (Section 9, Judges 3–4) where multi-step reasoning about failure signals matters most.

4. Track 1 — Prompt & Skill Evolution

This is the fastest, cheapest, and most reversible improvement path. Always start here.

4a. System Prompt Optimization

The core loop: run → evaluate → rewrite prompt if score is low.

import json
from anthropic import Anthropic

client = Anthropic()

# --- Versioned prompt store ---
prompt_versions = []

def save_prompt(prompt: str, score: float):
    prompt_versions.append({"prompt": prompt, "score": score})
    prompt_versions.sort(key=lambda x: x["score"], reverse=True)

def best_prompt() -> str:
    return prompt_versions[0]["prompt"] if prompt_versions else INITIAL_PROMPT

# --- Agent runner ---
INITIAL_PROMPT = "You are a helpful assistant that answers math word problems."

def run_agent(system_prompt: str, user_task: str) -> str:
    response = client.messages.create(
        model="claude-opus-4-6-20260205",
        max_tokens=1024,
        system=system_prompt,
        messages=[{"role": "user", "content": user_task}]
    )
    return response.content[0].text

# --- LLM-as-judge evaluator ---
def evaluate_response(task: str, response: str, expected: str) -> float:
    judge_prompt = f"""
    Task: {task}
    Expected answer: {expected}
    Agent response: {response}

    Score the response from 0.0 to 1.0 based on correctness and clarity.
    Reply with JSON only: {{"score": 0.0, "reason": "..."}}
    """
    result = client.messages.create(
        model="claude-opus-4-6-20260205",
        max_tokens=256,
        messages=[{"role": "user", "content": judge_prompt}]
    )
    return json.loads(result.content[0].text)["score"]

# --- Prompt rewriter ---
def rewrite_prompt(current_prompt: str, task: str, failed_response: str, reason: str) -> str:
    rewrite_request = f"""
    The current system prompt failed on this task.

    System prompt: {current_prompt}
    Task: {task}
    Bad response: {failed_response}
    Failure reason: {reason}

    Rewrite the system prompt to handle this better.
    Reply with the new prompt text only.
    """
    result = client.messages.create(
        model="claude-opus-4-6-20260205",
        max_tokens=512,
        messages=[{"role": "user", "content": rewrite_request}]
    )
    return result.content[0].text

# --- Evolution loop ---
def evolution_loop(tasks: list[dict], threshold=0.7, max_rounds=3):
    current_prompt = INITIAL_PROMPT
    save_prompt(current_prompt, score=0.0)

    for round in range(max_rounds):
        print(f"\n=== Round {round + 1} | Prompt: {current_prompt[:60]}... ===")
        round_scores = []

        for t in tasks:
            response = run_agent(current_prompt, t["task"])
            score = evaluate_response(t["task"], response, t["expected"])
            round_scores.append(score)
            print(f"  Task: {t['task'][:50]} | Score: {score:.2f}")

            if score < threshold:
                current_prompt = rewrite_prompt(
                    current_prompt, t["task"], response, "Low score"
                )

        avg_score = sum(round_scores) / len(round_scores)
        save_prompt(current_prompt, avg_score)
        print(f"  Avg score: {avg_score:.2f}")

        if avg_score >= threshold:
            print("✅ Prompt converged.")
            break

    return best_prompt()

# Example usage
tasks = [
    {"task": "If a train travels 60mph for 2.5 hours, how far does it go?", "expected": "150 miles"},
    {"task": "A store has 240 apples. 1/3 are sold. How many remain?",       "expected": "160 apples"},
]

final_prompt = evolution_loop(tasks)
print(f"\nFinal best prompt:\n{final_prompt}")

The metaprompt rewriting approach above is straightforward but has a limitation: it uses a single static meta-prompt that can overfit to immediate grader feedback.

Alternatives to consider:

GEPA (Section 4d) — population-based search with train/validation splits for more robust prompt generalization.
DSPy (Section 2e) — instead of writing prompt strings at all, define a Signature (input/output spec) and a Metric, and let DSPy's MIPRO optimizer compile the best prompt via Bayesian search. This is the most structured approach to prompt optimization and works particularly well for multi-step pipelines (e.g., RAG chains) where multiple prompts need to be co-optimized.
TextGrad (Section 2e) — treats the agent as a differentiable program and uses textual gradients (natural-language feedback on the execution trace) to mutate the prompt or code. Best for hard optimization problems where failures are diagnosable from the trace (math reasoning, code generation).

4b. Dynamic Skill Library

Agents that write, register, and retrieve tools on demand — and prune the ones that stop working. The Memento-Skills framework (Section 2e) takes this pattern further: when an agent fails a task, an orchestrator evaluates why and literally rewrites the Markdown and code files for the failing skill, accumulating a refined skill library over time. The implementation below captures the same core idea.

import json
from anthropic import Anthropic

client = Anthropic()

# --- Skill registry ---
class SkillRegistry:
    def __init__(self):
        self.skills: dict[str, dict] = {}  # name -> {code, description, stats}

    def register(self, name: str, description: str, code: str):
        self.skills[name] = {
            "description": description,
            "code": code,
            "usage_count": 0,
            "success_rate": 1.0
        }
        print(f"✅ Skill registered: {name}")

    def retrieve(self, task: str, top_k=2) -> list[dict]:
        """Keyword overlap retrieval — swap for vector search in prod."""
        scored = []
        for name, skill in self.skills.items():
            overlap = len(
                set(task.lower().split()) & set(skill["description"].lower().split())
            )
            scored.append((overlap, name, skill))
        scored.sort(reverse=True)
        return [{"name": n, **s} for _, n, s in scored[:top_k]]

    def update_stats(self, name: str, success: bool):
        if name in self.skills:
            skill = self.skills[name]
            skill["usage_count"] += 1
            skill["success_rate"] = (
                skill["success_rate"] * (skill["usage_count"] - 1) + int(success)
            ) / skill["usage_count"]

    def prune(self, min_success_rate=0.4, min_uses=3):
        """Remove underperforming skills."""
        to_remove = [
            name for name, s in self.skills.items()
            if s["usage_count"] >= min_uses and s["success_rate"] < min_success_rate
        ]
        for name in to_remove:
            del self.skills[name]
            print(f"🗑️  Pruned skill: {name}")

registry = SkillRegistry()

# --- Seed with initial skills ---
registry.register(
    name="calculate_percentage",
    description="calculate percentage proportion ratio",
    code="def calculate_percentage(part, whole): return round((part / whole) * 100, 2)"
)
registry.register(
    name="days_between_dates",
    description="date difference calendar days between two dates",
    code="""
from datetime import datetime
def days_between_dates(d1: str, d2: str) -> int:
    fmt = "%Y-%m-%d"
    return abs((datetime.strptime(d2, fmt) - datetime.strptime(d1, fmt)).days)
"""
)

# --- Skill generator: agent writes new skills on demand ---
def generate_skill(task_description: str) -> dict:
    prompt = f"""
    A user needs help with: "{task_description}"
    No existing skill covers this. Write a new Python skill.

    Reply with JSON only:
    {{
        "name": "snake_case_name",
        "description": "keywords describing when to use this skill",
        "code": "def skill_name(...):\\n    ..."
    }}
    """
    result = client.messages.create(
        model="claude-opus-4-6-20260205",
        max_tokens=512,
        messages=[{"role": "user", "content": prompt}]
    )
    raw = result.content[0].text.strip().strip("```

json").strip("

```")
    return json.loads(raw)

# --- Agent that uses the skill registry ---
def skill_aware_agent(user_task: str):
    relevant_skills = registry.retrieve(user_task)
    skill_context = "\n\n".join(
        [f"Skill `{s['name']}`:\n```
{% endraw %}
python\n{s['code']}\n
{% raw %}
```" for s in relevant_skills]
    )

    system = f"""You are a Python agent. Use available skills when helpful.
Available skills:
{skill_context}

If no skill fits, say NEED_NEW_SKILL: <description of what's needed>."""

    response = client.messages.create(
        model="claude-opus-4-6-20260205",
        max_tokens=1024,
        system=system,
        messages=[{"role": "user", "content": user_task}]
    )
    answer = response.content[0].text

    # Auto-generate missing skill if flagged
    if "NEED_NEW_SKILL:" in answer:
        needed = answer.split("NEED_NEW_SKILL:")[1].strip()
        print(f"🔧 Generating new skill for: {needed}")
        new_skill = generate_skill(needed)
        registry.register(**new_skill)
        return skill_aware_agent(user_task)  # Retry with new skill

    success = "error" not in answer.lower() and "sorry" not in answer.lower()
    for s in relevant_skills:
        registry.update_stats(s["name"], success)

    return answer

# Example usage
print(skill_aware_agent("What percentage is 45 out of 180?"))
print(skill_aware_agent("How many days between 2024-01-15 and 2024-07-04?"))
print(skill_aware_agent("Convert 100 USD to EUR at a rate of 0.92"))  # triggers new skill

4c. Evaluation & Version Gating

Only promote a new prompt or skill if it measurably beats the current baseline.

Layered graders. A single LLM-as-judge is fragile. Production systems should layer multiple evaluation signals, as the OpenAI Cookbook demonstrates:

Grader type	What it checks	Why it matters
Deterministic (Python)	Keyword presence, length within bounds	Fast, cheap, catches hard failures early
Semantic (cosine similarity)	Summary stays anchored to source content	Guards against superficial rephrasing that drifts from the original
LLM-as-judge (score model)	Rubric-driven quality assessment	Captures nuanced signals that rule-based metrics miss

The deterministic graders stabilize optimization before semantic tuning kicks in. The LLM judge provides a holistic failsafe for edge cases that slip past the other checks.

import json
from dataclasses import dataclass, field
from anthropic import Anthropic

client = Anthropic()

@dataclass
class EvalResult:
    score: float
    passed: bool
    feedback: str

@dataclass
class EvalSuite:
    name: str
    cases: list[dict] = field(default_factory=list)
    pass_threshold: float = 0.75

    def add_case(self, input: str, expected: str, tags: list[str] | None = None):
        self.cases.append({"input": input, "expected": expected, "tags": tags or []})

def llm_judge(task: str, expected: str, actual: str) -> EvalResult:
    prompt = f"""Evaluate this agent response.
Task: {task}
Expected: {expected}
Actual: {actual}

Reply with JSON only:
{{"score": 0.0-1.0, "passed": true/false, "feedback": "brief reason"}}"""

    result = client.messages.create(
        model="claude-opus-4-6-20260205",
        max_tokens=256,
        messages=[{"role": "user", "content": prompt}]
    )
    data = json.loads(result.content[0].text)
    return EvalResult(**data)

def run_eval_suite(suite: EvalSuite, system_prompt: str) -> dict:
    results = []
    tag_scores: dict[str, list] = {}

    for case in suite.cases:
        response = client.messages.create(
            model="claude-opus-4-6-20260205",
            max_tokens=512,
            system=system_prompt,
            messages=[{"role": "user", "content": case["input"]}]
        )
        actual = response.content[0].text
        result = llm_judge(case["input"], case["expected"], actual)
        results.append(result)

        for tag in case.get("tags", []):
            tag_scores.setdefault(tag, []).append(result.score)

        status = "✅" if result.passed else "❌"
        print(f"  {status} [{case['input'][:45]}] score={result.score:.2f} | {result.feedback}")

    avg_score = sum(r.score for r in results) / len(results)
    tag_summary = {tag: round(sum(s)/len(s), 2) for tag, s in tag_scores.items()}

    return {
        "avg_score": round(avg_score, 3),
        "passed": avg_score >= suite.pass_threshold,
        "tag_breakdown": tag_summary,
        "total_cases": len(results),
        "passed_cases": sum(1 for r in results if r.passed)
    }

def promote_prompt(candidate: str, current: str, suite: EvalSuite) -> tuple[str, dict]:
    """Only promote candidate if it beats the current prompt."""
    print("\n📊 Evaluating CURRENT prompt...")
    current_report = run_eval_suite(suite, current)

    print("\n📊 Evaluating CANDIDATE prompt...")
    candidate_report = run_eval_suite(suite, candidate)

    if candidate_report["avg_score"] > current_report["avg_score"]:
        print(f"\n🚀 Promoting ({candidate_report['avg_score']:.2f} > {current_report['avg_score']:.2f})")
        return candidate, candidate_report
    else:
        print(f"\n⏪ Keeping current ({current_report['avg_score']:.2f} >= {candidate_report['avg_score']:.2f})")
        return current, current_report

# Example usage
suite = EvalSuite(name="math_agent_v1", pass_threshold=0.75)
suite.add_case("What is 15% of 200?",                  "30",           tags=["percentage"])
suite.add_case("A rectangle is 8x5. What's its area?", "40 sq units",  tags=["geometry"])
suite.add_case("Train goes 90mph for 3 hours. Distance?", "270 miles", tags=["word_problem"])
suite.add_case("Factor 12 into primes.",                "2 × 2 × 3",   tags=["number_theory"])

current_prompt   = "You are a helpful assistant that solves math problems."
candidate_prompt = (
    "You are a precise math tutor. Always show step-by-step reasoning, "
    "state the formula used, then give a clean final answer."
)

best_prompt, report = promote_prompt(candidate_prompt, current_prompt, suite)
print(f"\nTag breakdown: {report['tag_breakdown']}")
print(f"Final: {report['passed_cases']}/{report['total_cases']} cases passed")

Version tracking in production. The OpenAI Cookbook introduces a VersionedPrompt class that stores each prompt revision with a timestamp, eval ID, run ID, and metadata. This gives you instant rollback and a full audit trail of what changed and why. The pattern is simple to implement yourself:

from datetime import datetime, timezone
from dataclasses import dataclass, field

@dataclass
class PromptVersion:
    version: int
    prompt: str
    model: str
    score: float
    timestamp: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
    metadata: dict = field(default_factory=dict)

class VersionedPrompt:
    def __init__(self, initial_prompt: str, model: str = "claude-opus-4-6-20260205"):
        self._versions = [PromptVersion(version=0, prompt=initial_prompt, model=model, score=0.0)]

    def update(self, new_prompt: str, score: float, model: str = None, **metadata) -> PromptVersion:
        v = PromptVersion(
            version=self._versions[-1].version + 1,
            prompt=new_prompt,
            model=model or self._versions[-1].model,
            score=score,
            metadata=metadata,
        )
        self._versions.append(v)
        return v

    def current(self) -> PromptVersion:
        return self._versions[-1]

    def best(self) -> PromptVersion:
        return max(self._versions, key=lambda v: v.score)

    def rollback(self, version: int) -> PromptVersion:
        self._versions = [v for v in self._versions if v.version <= version]
        return self._versions[-1]

Model comparison. When optimizing, you can also test the same prompt across different model variants (e.g., a full model vs a smaller/cheaper model) and select the best model-prompt combination. The OpenAI Cookbook demonstrates this by running candidate prompts against both gpt-5 and gpt-5-mini in parallel and keeping whichever scores higher — balancing quality against cost and latency.

4d. Advanced: GEPA Optimization

The simple metaprompt rewriting loop in Section 4a works but has a limitation: a static meta-prompt explores a narrow space and can overfit to immediate grader feedback on individual examples.

GEPA (Genetic-Pareto) is a more rigorous alternative demonstrated in the OpenAI Cookbook. It samples agent trajectories, reflects on them in natural language, proposes prompt revisions, and evolves the system through iterative feedback loops with train/validation splits.

How it differs from simple rewriting:

Dimension	Simple metaprompt	GEPA
Search strategy	Greedy rewrite per failure	Population-based, Pareto front selection
Overfitting protection	None	Train/validation split
Feedback used	Grader scores only	Scores + natural language reflection on trajectories
Multi-objective	Single average score	Pareto-optimal across multiple grader dimensions

The GEPA loop:

Start with a seed prompt (candidate)
Evaluate on a training subsample using your graders
Reflect on trajectories — the GEPA reflection LM reads inputs, outputs, and feedback to propose an improved prompt
Evaluate the new candidate on a validation set
Maintain a Pareto front of non-dominated candidates
Repeat until convergence or budget exhaustion

import gepa
from gepa import EvaluationBatch

seed_candidate = {
    "system_prompt": "You are a summarization assistant. Given a section of text, produce a summary."
}

result = gepa.optimize(
    seed_candidate=seed_candidate,
    trainset=train_data,
    valset=val_data,
    adapter=your_eval_adapter,   # bridges your graders to GEPA's interface
    reflection_lm="gpt-5",
    max_metric_calls=20,
    track_best_outputs=True,
)

best_prompt = result.best_candidate["system_prompt"]

When to use GEPA vs simple rewriting: If you have fewer than 10 eval cases and need a quick improvement, simple metaprompt rewriting is sufficient. If you have a real dataset with dozens of examples and need the prompt to generalize across them, GEPA's population-based search with train/validation splits will produce more robust results.

5. When to Improve Prompt vs. Create a Skill

Signal	Improve Prompt	Create/Improve Skill
Wrong tone, style, or reasoning format	✅
Misunderstands task intent	✅
Missing a computation or lookup		✅
Fails consistently on one task type		✅
Needs external data or API		✅
Hallucinating facts it should retrieve		✅

The 3-question test:

Knowledge/reasoning gap or behavior gap? → behavior = prompt, knowledge = skill
Reproducible with the same input type? → yes = skill (deterministic logic in code)
Would a human use a tool or think differently? → tool = skill, think = prompt

Automated Failure Classifier

import json
from enum import Enum
from anthropic import Anthropic

client = Anthropic()

class ImprovementTrack(Enum):
    PROMPT = "prompt"
    SKILL  = "skill"
    BOTH   = "both"

def classify_failure(
    task: str,
    agent_response: str,
    expected: str,
    current_system_prompt: str
) -> dict:
    classifier_prompt = f"""
You are an AI agent debugging expert. Analyze this agent failure.

System prompt: {current_system_prompt}
Task: {task}
Expected: {expected}
Actual response: {agent_response}

Diagnose the root cause and classify it. Consider:
- PROMPT: the agent has the capability but wrong behavior/tone/reasoning style
- SKILL: the agent is missing a tool, lookup, or computation it cannot reliably do in its head
- BOTH: the prompt misdirects AND a skill is missing

Reply with JSON only:
{{
    "track": "prompt" | "skill" | "both",
    "root_cause": "one sentence explanation",
    "evidence": "specific part of the response that reveals the problem",
    "suggested_action": "concrete next step"
}}
"""
    result = client.messages.create(
        model="claude-opus-4-6-20260205",
        max_tokens=512,
        messages=[{"role": "user", "content": classifier_prompt}]
    )
    diagnosis = json.loads(result.content[0].text)
    diagnosis["track"] = ImprovementTrack(diagnosis["track"])
    return diagnosis

# Example usage
failures = [
    {
        "task": "What is the compound interest on $5000 at 4.5% for 3 years?",
        "expected": "$706.06",
        "actual": "The compound interest would be approximately $700.",
        "prompt": "You are a helpful financial assistant."
    },
    {
        "task": "Explain the steps to solve a quadratic equation.",
        "expected": "Step-by-step: factoring, completing the square, quadratic formula",
        "actual": "Just use the quadratic formula: x = (-b ± √(b²-4ac)) / 2a",
        "prompt": "You are a helpful math assistant."
    },
]

for f in failures:
    print(f"\nTask: {f['task'][:60]}...")
    diagnosis = classify_failure(f["task"], f["actual"], f["expected"], f["prompt"])
    print(f"  Track     : {diagnosis['track'].value.upper()}")
    print(f"  Root cause: {diagnosis['root_cause']}")
    print(f"  Action    : {diagnosis['suggested_action']}")

Thumb rules:

Prompt = change how the agent thinks
Skill = change what the agent can do
If a fix requires math, datetime, or any API call → always a Skill
Aim for a thin prompt, rich skill library

6. Track 2 — Code & Harness Evolution

Prompt and skill tuning change the instructions and tools given to a model. Code and harness evolution go further: the agent modifies its own implementation.

Code evolution has two variants: model-side (autoresearch modifies training code to produce a better model) and harness-side (autoagent modifies the agent itself — prompt, tools, orchestration). Both use the same program.md pattern.

The `program.md` Pattern

The key insight from both frameworks: you are not touching the Python files like you normally would as an engineer. Instead, you are programming program.md — the Markdown file that provides context to the meta-agent and defines the evolution loop.

┌─────────────────────────────────────────────┐
│  Human writes program.md                    │
│  (instructions, constraints, goals)         │
│                                             │
│         ┌──────────────┐                    │
│         │  Meta-agent   │                   │
│         │  reads        │                   │
│         │  program.md   │                   │
│         └──────┬───────┘                    │
│                │                            │
│         ┌──────▼───────┐                    │
│         │  Modifies     │                   │
│         │  train.py or  │                   │
│         │  agent.py     │                   │
│         └──────┬───────┘                    │
│                │                            │
│         ┌──────▼───────┐                    │
│         │  Runs eval    │                   │
│         │  (metric)     │                   │
│         └──────┬───────┘                    │
│                │                            │
│         ┌──────▼───────┐                    │
│         │  Score better?│                   │
│         │  Keep : Revert│                   │
│         └──────────────┘                    │
└─────────────────────────────────────────────┘

autoresearch: Evolving Model Training Code

Setup: Three files. prepare.py handles data prep (fixed). train.py contains the full model and training loop (agent edits this). program.md is the agent's instruction manual (human edits this).

Loop: Point a coding agent (Claude, Codex, etc.) at the repo. The agent reads program.md, modifies train.py, kicks off a 5-minute training run, checks if validation bits per byte improved. If yes, the change sticks. If no, the agent reverts and tries something else.

Results: ~12 experiments/hour, ~100 overnight. You wake up to a log of everything the agent tried and (hopefully) a better model.

Why this is code evolution, not fine-tuning: Although autoresearch produces a better-trained model as its output, the evolution mechanism is code editing, not weight updating — the agent modifies Python source (architecture, optimizer, hyperparameters), not gradients. The coding agent's own weights are never touched.

autoagent: Evolving the Agent Harness

autoagent applies the same pattern to the agent itself rather than model training code:

agent.py — the entire harness in a single file: config, tool definitions, agent registry, routing/orchestration, and a Harbor adapter boundary (explicitly marked as fixed)
program.md — meta-agent instructions plus the directive (what kind of agent to build)
tasks/ — evaluation tasks in Harbor format, running in Docker containers

The meta-agent modifies the system prompt, tools, agent configuration, and orchestration, runs the benchmark, checks the score, keeps or discards the change, and repeats.

When to Use Code Evolution

This track generalizes to any scenario where you have:

A single file (or small surface) to optimize — a config file, a set of hyperparameters, a build configuration, an agent harness
A clear, measurable metric — validation loss, benchmark score, test pass rate
A bounded experiment time — each iteration completes in minutes, not hours

If your problem fits this shape, the autoresearch/autoagent pattern can be more effective than manual iteration — and it works overnight while you sleep.

Important distinction from fine-tuning: Code evolution modifies the code and configuration around the model, not the model weights. It is cheaper, faster, and fully reversible (just revert the file). Consider it before jumping to fine-tuning.

7. Track 3 — RAG

RAG fixes knowledge gaps. It slots between code evolution and fine-tuning in the escalation ladder.

Problem	RAG	Fine-Tune
Missing domain facts or docs	✅	✅
Stale knowledge / live updates needed	✅	❌
Specific reasoning style/pattern	❌	✅
< 500 training examples available	✅	❌
Hallucinating facts it should look up	✅	⚠️ partial

Minimal RAG Skill

import json
from anthropic import Anthropic

client = Anthropic()

# --- Toy in-memory store (swap for Chroma/Pinecone in prod) ---
knowledge_base = [
    {"id": 1, "text": "Q1 2025 audit found 3 critical gaps in access control policies."},
    {"id": 2, "text": "Revenue for Q1 2025 was $4.2M, up 18% YoY."},
    {"id": 3, "text": "The compound XR-47 showed hepatotoxicity in Phase 2 trials."},
]

def simple_retrieve(query: str, top_k=2) -> list[str]:
    """Keyword overlap retrieval — replace with embedding search in prod."""
    query_words = set(query.lower().split())
    scored = []
    for doc in knowledge_base:
        doc_words = set(doc["text"].lower().split())
        overlap = len(query_words & doc_words)
        scored.append((overlap, doc["text"]))
    scored.sort(reverse=True)
    return [text for _, text in scored[:top_k] if _ > 0]

def rag_agent(user_query: str) -> str:
    context_chunks = simple_retrieve(user_query)

    if context_chunks:
        context_block = "\n".join(f"- {c}" for c in context_chunks)
        system = f"""You are a helpful enterprise assistant.
Use ONLY the retrieved context below to answer.
If the context doesn't cover the question, say so.

Retrieved context:
{context_block}"""
    else:
        system = "You are a helpful enterprise assistant."

    response = client.messages.create(
        model="claude-opus-4-6-20260205",
        max_tokens=512,
        system=system,
        messages=[{"role": "user", "content": user_query}]
    )
    return response.content[0].text

# Example usage
queries = [
    "What did the Q1 2025 audit find?",
    "What were Q1 revenues?",
    "Tell me about XR-47 safety.",
    "What is our HR vacation policy?",  # not in KB → honest fallback
]

for q in queries:
    print(f"\nQ: {q}")
    print(f"A: {rag_agent(q)}")

Key principle: RAG + skills often eliminate the need for fine-tuning entirely for enterprise agents where knowledge is the primary gap.

8. Track 4 — LLM Fine-Tuning

Fine-tuning internalizes behavior and reasoning patterns that prompt iteration cannot reliably produce. It is the most expensive and least reversible track — and it carries a real risk of losing generalization capability. A model fine-tuned on a narrow domain dataset may improve on that domain while degrading on everything else. This is not a theoretical concern: it is the primary failure mode of production fine-tuning.

Escalate to fine-tuning only when:

Prompt iteration has plateaued (3+ rounds, no score improvement)
Failures persist even when the correct skill is invoked
Failures are concentrated in one domain (finance, legal, medical)
You have 500+ clean, high-quality training trajectories

Consider code evolution first. If the issue is about how the agent operates rather than how the model reasons, the autoresearch/autoagent pattern from Section 6 may be more effective. Code evolution modifies the code and configuration around the model (architecture, hyperparameters, tools, orchestration) without touching model weights — cheaper, faster, and fully reversible.

The iterative fine-tuning loop:

Deploy → collect trajectories → filter (score ≥ 0.8) → fine-tune → redeploy → repeat

Avoiding catastrophic forgetting:

Always fine-tune from the base model, not iteratively from prior fine-tunes
Evaluate on a held-out general benchmark alongside the domain benchmark
Set a regression threshold: if general score drops > 5%, abort

Frameworks That Automate the Fine-Tuning Loop

Two frameworks are worth highlighting for teams that want to close the loop from production failures to weight updates without manual data curation:

DSPy self-distillation. DSPy can fine-tune a smaller, cheaper model (e.g., Llama 3) to mimic the reasoning of a larger model (e.g., GPT-5) by distilling the best-performing prompt traces into training data. The workflow: run your DSPy program with the large model, collect the traces that score highest on your metric, and use them to fine-tune the small model. This gives you the reasoning quality of the big model at the inference cost of the small one.

AgentScope + Trinity-RFT. Designed for enterprise-scale autonomous fine-tuning. AgentScope captures production logs via "Inference Tables." Trinity-RFT uses an LLM judge to label production data as "good" or "bad," then automatically kicks off a fine-tuning job using reinforcement learning from feedback (PPO or SFT). This is the most hands-off approach to weight updates: the system monitors production, identifies failures, curates training data, and fine-tunes — all without human intervention. The trade-off is complexity: you need the infrastructure to run fine-tuning jobs on schedule and the monitoring to catch regressions.

9. The Master Decision Pipeline — LLM as Judge

This is the centrepiece of the guide. Four judges, one pipeline — everything from Sections 4–8 plugs into the dispatcher at the end.

Agent runs → Failures logged
     ↓
Judge 1: Per-run evaluator (scores 0–1)
     ↓
Judge 2: Signal extractor (persistence, skill gap, knowledge gap, data volume)
     ↓
Judge 3: Track recommender (LLM synthesizes signals → verdict)
     ↓
Judge 4: Action dispatcher → calls evolution_loop() / rag_agent() / fine-tune export

import json
from dataclasses import dataclass, field
from enum import Enum
from anthropic import Anthropic

client = Anthropic()


# ── Data models ──────────────────────────────────────────────

class Track(Enum):
    PROMPT_SKILL   = "prompt_skill"
    CODE_EVOLUTION = "code_evolution"
    RAG            = "rag"
    FINE_TUNE      = "fine_tune"
    RAG_FINE_TUNE  = "rag+fine_tune"

@dataclass
class AgentRun:
    task: str
    expected: str
    actual: str
    task_type: str
    prompt_version: str
    prompt_round: int
    correct_skill_invoked: bool = False
    score: float = 0.0

@dataclass
class JudgeVerdict:
    track: Track
    confidence: float
    signals: dict
    rationale: str
    next_steps: list[str]
    estimated_effort: str
    risk: str


# ── Judge 1: Per-run evaluator ────────────────────────────────

def evaluate_run(run: AgentRun) -> AgentRun:
    """Scores a single agent run 0.0–1.0."""
    prompt = f"""
Evaluate this agent response.

Task     : {run.task}
Expected : {run.expected}
Actual   : {run.actual}

Reply with JSON only:
{{"score": 0.0-1.0, "passed": true/false, "reason": "one sentence"}}
"""
    result = client.messages.create(
        model="claude-opus-4-6-20260205",
        max_tokens=256,
        messages=[{"role": "user", "content": prompt}]
    )
    data = json.loads(result.content[0].text.strip().strip("```

json").strip("

```"))
    run.score = data["score"]
    return run


# ── Judge 2: Signal extractor ─────────────────────────────────

def extract_signals(runs: list[AgentRun], corpus_exists: bool, example_count: int) -> dict:
    """Derives quantitative signals from a batch of runs."""
    total = len(runs)
    failed = [r for r in runs if r.score < 0.7]

    if not failed:
        return {"all_passing": True}

    f = len(failed)

    # Signal 1: Prompt plateau — failures persisting after 3+ prompt rounds
    persistence_rate = len([r for r in failed if r.prompt_round >= 3]) / f

    # Signal 2: Skill bottleneck — skill fired but still failed
    skill_failure_rate = len([r for r in failed if r.correct_skill_invoked]) / f

    # Signal 3: Domain concentration — one task type dominating failures
    type_counts = {}
    for r in failed:
        type_counts[r.task_type] = type_counts.get(r.task_type, 0) + 1
    dominant_rate = max(type_counts.values()) / f if type_counts else 0
    dominant_type = max(type_counts, key=type_counts.get) if type_counts else "unknown"

    # Signal 4: Knowledge gap — failed despite no skill gap → likely needs retrieval
    knowledge_gap_rate = len([
        r for r in failed if not r.correct_skill_invoked and r.prompt_round >= 2
    ]) / f

    return {
        "total_runs"         : total,
        "failure_rate"       : round(f / total, 2),
        "persistence_rate"   : round(persistence_rate, 2),   # > 0.4 → fine-tune
        "skill_failure_rate" : round(skill_failure_rate, 2), # > 0.3 → fine-tune
        "knowledge_gap_rate" : round(knowledge_gap_rate, 2), # > 0.4 → RAG
        "dominant_type"      : dominant_type,
        "dominant_type_rate" : round(dominant_rate, 2),      # > 0.5 → systematic gap
        "corpus_exists"      : corpus_exists,
        "example_count"      : example_count,
        "data_sufficient"    : example_count >= 500
    }


# ── Judge 3: Track recommender ────────────────────────────────

def recommend_track(
    signals: dict,
    current_prompt: str,
    sample_failures: list[AgentRun]
) -> JudgeVerdict:
    """LLM judge: reads signals + failure samples → recommends track."""

    sample_text = json.dumps([
        {
            "task": r.task, "expected": r.expected,
            "actual": r.actual, "score": r.score,
            "prompt_round": r.prompt_round,
            "correct_skill_invoked": r.correct_skill_invoked
        }
        for r in sample_failures[:5]
    ], indent=2)

    judge_prompt = f"""
You are a senior AI systems architect. Decide the best improvement track
for an underperforming agent based on signals and failure samples.

## Quantitative Signals
{json.dumps(signals, indent=2)}

## Signal Thresholds
- persistence_rate > 0.4     → prompt iteration plateauing → consider fine_tune
- skill_failure_rate > 0.3   → model reasoning is bottleneck → consider fine_tune
- knowledge_gap_rate > 0.4   → facts/docs missing → consider rag
- dominant_type_rate > 0.5   → systematic domain gap
- data_sufficient = false    → BLOCK fine_tune, default to rag or prompt_skill

## Available Tracks
- prompt_skill   : Rewrite system prompt and/or add/fix tools. Fast, cheap, reversible.
- code_evolution : Let a meta-agent modify code/config against a clear metric.
                   Use when the problem has a single file to optimize and a measurable goal.
- rag            : Index a knowledge corpus and retrieve at query time.
                   Prefer over fine-tuning when knowledge changes or data < 500.
- fine_tune      : Train on trajectories. Use when reasoning style is systematically
                   wrong AND 500+ examples exist AND prompt iteration has plateaued.
- rag+fine_tune  : Both. Use when knowledge AND reasoning style are both gaps.

## Current System Prompt
{current_prompt}

## Sample Failures
{sample_text}

Be conservative — recommend fine_tune only when signals clearly justify it.

Reply with JSON only:
{{
    "track": "prompt_skill" | "code_evolution" | "rag" | "fine_tune" | "rag+fine_tune",
    "confidence": 0.0-1.0,
    "signals_fired": {{
        "prompt_plateau"   : true/false,
        "skill_bottleneck" : true/false,
        "knowledge_gap"    : true/false,
        "systematic_domain": true/false,
        "data_sufficient"  : true/false
    }},
    "rationale": "2-3 sentence explanation referencing specific signals",
    "next_steps": ["step 1", "step 2", "step 3"],
    "estimated_effort": "e.g. 2hrs prompt iteration vs 4 days fine-tuning",
    "risk": "main risk of this recommendation"
}}
"""
    result = client.messages.create(
        model="claude-opus-4-6-20260205",
        max_tokens=768,
        messages=[{"role": "user", "content": judge_prompt}]
    )
    raw = result.content[0].text.strip().strip("```

json").strip("

```")
    data = json.loads(raw)

    return JudgeVerdict(
        track=Track(data["track"]),
        confidence=data["confidence"],
        signals=data["signals_fired"],
        rationale=data["rationale"],
        next_steps=data["next_steps"],
        estimated_effort=data["estimated_effort"],
        risk=data["risk"]
    )


# ── Judge 4: Action dispatcher ────────────────────────────────

def dispatch(verdict: JudgeVerdict):
    print(f"\n{'='*60}")
    print(f"  TRACK       : {verdict.track.value.upper()}")
    print(f"  CONFIDENCE  : {verdict.confidence:.0%}")
    print(f"  RATIONALE   : {verdict.rationale}")
    print(f"  EFFORT      : {verdict.estimated_effort}")
    print(f"  RISK        : {verdict.risk}")
    print(f"  SIGNALS     : {verdict.signals}")
    print(f"\n  NEXT STEPS:")
    for i, step in enumerate(verdict.next_steps, 1):
        print(f"    {i}. {step}")
    print(f"{'='*60}")

    actions = {
        Track.PROMPT_SKILL: lambda: (
            print("\n→ Calling evolution_loop() to rewrite system prompt"),
            print("→ Calling classify_failure() to split prompt vs skill fixes")
        ),
        Track.CODE_EVOLUTION: lambda: (
            print("\n→ Set up program.md with constraints and goals"),
            print("→ Point meta-agent at the repo (autoresearch or autoagent pattern)"),
            print("→ Let it hill-climb overnight; review results in the morning")
        ),
        Track.RAG: lambda: (
            print("\n→ Chunk and embed your knowledge corpus"),
            print("→ Register retrieval as a new skill in SkillRegistry"),
            print("→ Re-run eval suite to confirm improvement")
        ),
        Track.FINE_TUNE: lambda: (
            print("\n→ Export high-scoring runs as training trajectories"),
            print("→ Filter: keep only runs with score >= 0.8"),
            print("→ Submit fine-tune job (OpenAI / HuggingFace / Anthropic)")
        ),
        Track.RAG_FINE_TUNE: lambda: (
            print("\n→ Step 1: Build RAG pipeline first (faster win)"),
            print("→ Step 2: Validate RAG improves knowledge gaps"),
            print("→ Step 3: Fine-tune on reasoning style gaps in parallel")
        )
    }
    actions[verdict.track]()


# ── Master pipeline ───────────────────────────────────────────

def run_judge_pipeline(
    runs: list[AgentRun],
    current_prompt: str,
    corpus_exists: bool = False,
    example_count: int = 0
):
    print("⏳ Step 1: Evaluating all runs...")
    evaluated = [evaluate_run(r) for r in runs]

    avg_score = sum(r.score for r in evaluated) / len(evaluated)
    failed_count = sum(1 for r in evaluated if r.score < 0.7)
    print(f"   Avg score: {avg_score:.2f} | Failed: {failed_count}/{len(evaluated)}")

    if avg_score >= 0.85:
        print("✅ Agent is performing well. No improvement needed.")
        return

    print("\n⏳ Step 2: Extracting signals...")
    signals = extract_signals(evaluated, corpus_exists, example_count)
    print(f"   Signals: {signals}")

    failed_runs = [r for r in evaluated if r.score < 0.7]

    print("\n⏳ Step 3: LLM judge recommending track...")
    verdict = recommend_track(signals, current_prompt, failed_runs)

    print("\n⏳ Step 4: Dispatching recommendation...")
    dispatch(verdict)

    return verdict


# ── Example usage ─────────────────────────────────────────────

runs = [
    AgentRun(
        task="Summarize the Q1 2025 earnings report",
        expected="Revenue $4.2M, up 18% YoY, 3 audit gaps found",
        actual="I don't have access to Q1 2025 earnings data.",
        task_type="finance", prompt_version="v3",
        prompt_round=4, correct_skill_invoked=False
    ),
    AgentRun(
        task="What were the audit findings for access control?",
        expected="3 critical gaps found in access control policies",
        actual="I cannot find specific audit findings in my knowledge.",
        task_type="finance", prompt_version="v3",
        prompt_round=4, correct_skill_invoked=False
    ),
    AgentRun(
        task="Calculate compound interest $5000 at 4.5% for 3 years",
        expected="$706.06",
        actual="Approximately $700 using compound interest formula.",
        task_type="finance", prompt_version="v3",
        prompt_round=3, correct_skill_invoked=True
    ),
    AgentRun(
        task="Analyze revenue trend from last 4 quarters",
        expected="Structured YoY trend with % changes",
        actual="Revenue seems to be going up based on general trends.",
        task_type="finance", prompt_version="v3",
        prompt_round=4, correct_skill_invoked=False
    ),
] * 10  # scale to 40 runs

current_prompt = "You are a financial analysis assistant. Be thorough and precise."

verdict = run_judge_pipeline(
    runs=runs,
    current_prompt=current_prompt,
    corpus_exists=True,   # financial docs available to index
    example_count=350     # below the 500 fine-tuning threshold
)

Sample output:

⏳ Step 1: Evaluating all runs...
   Avg score: 0.31 | Failed: 37/40

⏳ Step 2: Extracting signals...
   Signals: {failure_rate: 0.93, persistence_rate: 0.89,
             knowledge_gap_rate: 0.76, dominant_type: finance,
             corpus_exists: True, data_sufficient: False}

⏳ Step 3: LLM judge recommending track...

⏳ Step 4: Dispatching recommendation...
============================================================
  TRACK       : RAG
  CONFIDENCE  : 91%
  RATIONALE   : High knowledge_gap_rate (0.76) with corpus_exists=True
                and data_sufficient=False clearly points to RAG. Agent
                is failing on factual retrieval, not reasoning style.
  EFFORT      : 4–6 hours to chunk, embed, and integrate corpus
  RISK        : Retrieval quality depends on chunking strategy
  SIGNALS     : {prompt_plateau: True, skill_bottleneck: False,
                 knowledge_gap: True, systematic_domain: True,
                 data_sufficient: False}

  NEXT STEPS:
    1. Chunk Q1 earnings report and audit docs into 512-token segments
    2. Embed with text-embedding-3-small and store in Chroma/Pinecone
    3. Register retrieval as a skill and re-run eval suite

→ Chunk and embed your knowledge corpus
→ Register retrieval as a new skill in SkillRegistry
→ Re-run eval suite to confirm improvement
============================================================

10. The Complete Escalation Ladder

Level 1 — Prompt tuning          (minutes, free)
     │  still failing after 3 rounds?
     ▼
Level 2 — Add/improve skills     (hours, cheap)
     │  still failing on reasoning/architecture?
     ▼
Level 3 — Code/harness evolution (hours, cheap — runs overnight)
     │  still failing on knowledge?
     ▼
Level 4 — RAG                    (hours, medium cost)
     │  still failing on reasoning style/pattern?
     ▼
Level 5 — Fine-tuning            (days, expensive)

The master pipeline in Section 9 enforces this ladder automatically — it blocks fine-tuning when data is insufficient, and prefers RAG when a corpus exists. Code evolution (Section 6) is a manual decision point: if your problem has a single file and a clear metric, try the autoresearch/autoagent pattern before moving to RAG or fine-tuning.

11. Continuous Monitoring

The evolution loop does not end after the initial optimization converges. Production agents face shifting data distributions, new edge cases, and model updates that can degrade performance over time.

Periodic re-evaluation. Schedule the eval suite to run on incoming data at regular intervals. When scores drop below a threshold, the evolution loop restarts automatically.

import time

def continuous_monitor(
    agent,
    eval_suite,
    versioned_prompt,
    check_interval_hours=24,
    regression_threshold=0.70,
):
    """Re-evaluate the agent periodically and trigger evolution if scores regress."""
    while True:
        new_tasks = collect_recent_tasks()  # returns list[{"task": ..., "expected": ...}]
        if not new_tasks:
            time.sleep(check_interval_hours * 3600)
            continue

        report = run_eval_suite(eval_suite, versioned_prompt.current().prompt)

        if report["avg_score"] < regression_threshold:
            print(f"Score regressed to {report['avg_score']:.2f} — triggering evolution loop")
            new_prompt = evolution_loop(new_tasks, threshold=regression_threshold)
            versioned_prompt.update(new_prompt, score=report["avg_score"], trigger="auto_regression")
        else:
            print(f"Score healthy: {report['avg_score']:.2f}")

        time.sleep(check_interval_hours * 3600)

Model version comparison on new data. When a new model version becomes available, run the eval suite with the current prompt on both the old and new models. If the new model scores higher, update the VersionedPrompt with the new model. If it scores lower, keep the current model — do not assume newer is better.

Drift detection with auto-rollback. Log prompt version, skill version, model version, and average score over time. If score regresses after any change, auto-rollback to the last known good version. The VersionedPrompt.rollback() method makes this a single call.

12. Pitfalls & Safety

Self-evolving loops introduce new failure modes that static agents do not have. The more autonomy you give the improvement loop, the more these risks matter.

Reward hacking — if your eval signal is imperfect, the agent will optimize for the signal rather than the goal. Use multiple eval dimensions (correctness, format, safety) and audit a random sample manually every N rounds.

Drift detection — log prompt version, skill version, and avg score over time. If score regresses after a change, auto-rollback to the last known good version.

Version everything — never deploy an unevaluated prompt or skill. The promote_prompt() gate in Section 4c enforces this.

Human checkpoints — before any fine-tuning job, require a human review of the filtered training trajectories. Garbage in, garbage out — and fine-tuning mistakes are expensive to undo.

Rollback strategy — store every prompt version with its eval score. A one-line revert (current_prompt = best_prompt()) should always be available.

Safety Models Across Frameworks

Different frameworks take different approaches to containing the risk of autonomous evolution:

Framework	Safety approach	Trade-off
OpenAI Cookbook	Versioned prompts with rollback; promote-only-if-better gate	Simple and effective, but no isolation — bad prompts can affect production before rollback
autoresearch	Git-based keep-or-revert; fixed 5-minute time budget per experiment	Time budget prevents runaway experiments; git makes every change reversible
autoagent	Docker isolation; Harbor sandboxing; tasks run in containers	Strong isolation, but Docker overhead adds latency to the feedback loop
Evolver	Command whitelist; scoped execution; timeout limits; full audit trail of every Event	Most comprehensive safety model, but also the most complex to set up

Strategy Presets

EvoMap's Evolver introduces a useful concept that applies even outside the framework: strategy presets that match the evolution behavior to the current development phase.

innovate — maximize new features and exploration. Use early in development when the agent is far from production-ready.
harden — focus on stability, regression testing, and edge case coverage. Use when approaching production readiness.
repair-only — constrain the agent to fixes only, no new behavior. Use when something is broken in production and you need a targeted fix.

This maps neatly onto how most teams already think about release stages. Even without Evolver, you can implement strategy presets by adjusting the threshold and max_rounds parameters in your evolution loop: high exploration tolerance for innovate mode, strict thresholds and minimal rounds for repair-only.

13. Conclusion

Self-evolving agents are not magic — they are disciplined feedback loops with clear escalation rules. Several open-source frameworks have already proven these patterns work in practice, from automated prompt optimization to overnight code evolution to governed harness engineering.

The four tracks in one sentence each:

Prompt/Skill — change how the agent thinks and what it can do. Always try this first.
Code/Harness evolution — let the agent modify its own implementation against a clear metric. Try this before RAG or fine-tuning when the problem has a single file and a measurable goal.
RAG — give the agent access to knowledge it doesn't have. Prefer this over fine-tuning when knowledge changes or data is scarce.
Fine-tuning — internalize reasoning patterns that prompt iteration cannot reliably produce. Use this last, and only with 500+ clean examples.

Thumb rules to remember:

Thin prompt, rich skill library
RAG before fine-tune
Code evolution before fine-tune (it is cheaper and reversible)
Persistence is the clearest fine-tune signal
Never deploy an unevaluated change
The LLM judge pipeline does the routing — let it
Version everything; rollback should be one line

Practical advice from the frameworks:

Version your prompts like you version your code (VersionedPrompt pattern)
Try the autoresearch pattern for any "single file, single metric" problem
Borrow Evolver's audit trail thinking for production agents — log every change as a structured event with before/after scores
Use strategy presets to match evolution aggressiveness to the development phase
Layer your graders: deterministic checks first, then semantic, then LLM judge

The long-term vision is agents that compound in capability over time, with humans setting goals and guardrails while the agent handles the improvement loop. The pipeline in Section 9 is a practical starting point for exactly that.

References

Frameworks covered in this guide:

OpenAI Self-Evolving Agents Cookbook
Karpathy's autoresearch — AI agents running research on single-GPU nanochat training
kevinrgu/autoagent — autonomous harness engineering
EvoMap Evolver — governed evolution with audit trails
GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning — Agrawal et al. (GitHub)

Additional frameworks and methodologies:

DSPy — Declarative Self-improving Python; Bayesian prompt compilation and self-distillation (Stanford NLP)
TextGrad — Automatic differentiation via text; textual backpropagation for LLM optimization (Nature, 2025)
Memento-Skills — Skill-evolution framework for long-horizon autonomous agents
AgentScope — Multi-agent platform with Trinity-RFT for online fine-tuning from production logs

Background reading:

Self-Evolving Agents: Three Frameworks That Let Your AI Improve Itself — Jia Chen, Softmax Data Blog
Anthropic Python SDK

Anthropic Managed Agents: What It Takes to Build Agent-as-a-Service

Yaohua Chen — Thu, 09 Apr 2026 17:19:56 +0000

Anthropic just launched Managed Agents. The open-source world has been learning the hard way why this matters.

On April 8, 2025, Anthropic launched the public beta of Claude Managed Agents -- a fully hosted platform for running AI agents with built-in sandboxing, session management, error recovery, and permission control. Four days earlier, the company had quietly cut off third-party agent frameworks like OpenClaw from using Claude subscription quotas, forcing them onto pay-per-use billing.

These two moves, four days apart, tell one story: the company that sells the brain has decided to sell the body, too.

Why? Because the "body" -- the infrastructure that lets an AI model actually do things in the real world -- is where agents succeed or fail in production. And as the open-source community has painfully demonstrated, getting this infrastructure wrong doesn't just cause bugs. It causes data leaks, runaway costs, and security breaches measured in the hundreds of thousands of dollars.

This post explores three questions:

What does it actually take to build a reliable, safe Agent-as-a-Service?
What goes wrong when these foundations are missing? (We have the data.)
How do different approaches -- managed platforms, open-source gateways, and learning engines -- stack up against these requirements?

Whether you're a developer evaluating agent frameworks, an architect designing agent infrastructure, or simply curious about where AI is headed, the answer starts with understanding five technical pillars that separate demo-grade agents from production-grade ones.

What Is an AI Agent, Really?

Before diving into architecture, let's clarify what we're talking about -- because "AI agent" means very different things to different people.

Most of us interact with AI through chat interfaces: you type a question, the model answers. That's a model -- a brain in a jar. Extremely intelligent, but it can't do anything. It can't browse your files, run code, send emails, or check your calendar. It just thinks and talks.

An agent is what happens when you give that brain a body.

Anthropic's engineering team describes this with a vivid metaphor: the model is the brain; the harness is the limbs plus the nervous system. The brain decides what to do. The harness actually does it -- calling tools, managing context, handling errors, keeping things running.

In practice, an agent system has three core components:

Think of it this way:

The Session is the agent's notebook -- the log of everything that's happened. If the agent crashes, this is how it remembers where it left off.
The Harness is the nervous system -- the loop that calls the AI model, routes tool calls, handles errors, and decides what to do next.
The Sandbox is the workshop -- the isolated environment where the agent actually runs code and performs actions, separated from your sensitive data and credentials.

When you use ChatGPT or Claude in a chat window, you're talking to the brain. When companies deploy agents that write code, manage workflows, or process documents autonomously, they need all three components working in concert.

And that's where things get interesting -- and dangerous.

When Agents Go Wrong: Lessons from OpenClaw

OpenClaw is one of the most popular open-source agent frameworks -- the fastest-growing repo in GitHub history, surpassing 350,000 stars in under three months -- with a thriving community of over 1,000 contributors. It's powerful, flexible, and genuinely useful. It's also a case study in what happens when agent infrastructure doesn't get the fundamentals right.

A security audit conducted by researchers at Shanghai University of Science and Technology and the Shanghai AI Lab put OpenClaw through 34 standardized test cases. The results should give anyone building agent services pause.

Metric	Result
Overall safety pass rate	58.9%
Intent misunderstanding & unsafe assumptions	0% pass rate
Prompt injection robustness	57%
Unexpected results under open-ended objectives	50%

(The audit used MiniMax M2.1 as the default model. Results may vary with other models, but the failure patterns -- particularly around architecture and permission design -- are model-agnostic.)

That 0% pass rate on intent misunderstanding is worth lingering on. In every single test with an ambiguous instruction, the agent filled in the blanks on its own and executed immediately. It never once asked the user for confirmation.

Industry-wide monitoring data paints an even more alarming picture:

230,000+ OpenClaw instances detected exposed on the public internet
Approximately 87,800 instances with data leaks
Approximately 43,000 instances with personal identity information exposed
36.8% of skills on the ClawHub marketplace contained security flaws
Over 1,000 skills contained malicious payloads
A CVSS 8.8 high-severity vulnerability enabling remote computer takeover

Cisco's assessment was blunt: "OpenClaw's security issues aren't configuration problems -- they're architecture problems."

OpenClaw's own documentation concedes the point: There is no "perfectly secure" setup.

Why Do These Failures Happen?

These aren't random bugs. They trace back to four systemic root causes -- each one a missing piece of agent infrastructure:

1. Context Compression Drops Safety Rails. When the information volume gets too large, the agent compresses its memory. During compression, it can squeeze out critical safety instructions -- the very guardrails meant to keep it in check. Imagine an air traffic controller under extreme stress who starts skipping safety checklists. That's context compression in action.

2. Execute First, Ask Never. The default behavior strategy leans toward "do it first, explain later" rather than "ask clearly first." For every ambiguous instruction in the security audit, the agent guessed the user's intent and acted immediately. Zero confirmation. Zero pause.

3. Prompt Injection Walks Through the Front Door. Malicious content embedded in inputs can trick the agent into bypassing safety mechanisms entirely. With a 57% robustness rate, nearly half of all injection attempts succeed. That's not a bug in one feature -- it's a gap in the security boundary.

4. The Agent Has the Keys to the Kingdom. OpenClaw runs with the same system permissions as the user who launched it. It can read, write, and delete anything the user can. Combine this with the injection vulnerability above, and an attacker doesn't need to hack your system -- they just need to convince the agent to do it for them.

These aren't problems unique to OpenClaw. They're the universal challenges of Agent-as-a-Service. Any framework, any platform, any team building agents will face these same four failure modes -- unless they're addressed at the architectural level.

Which brings us to the technologies that actually matter.

The 5 Pillars of Effective and Safe Agent Services

Anthropic has published 15 engineering blog posts over the past two years, documenting their approach to building production-grade agents. Distilled into a learning path, they form a capability pyramid -- a stack of technologies and practices that builds from foundation to production readiness:

Each pillar directly addresses one of the failure modes we saw with OpenClaw:

Let's walk through them.

Pillar 1: Foundation Architecture -- Know When NOT to Use an Agent

The OpenClaw failure it addresses: Execute first, ask never.

The most important architectural decision is also the most counterintuitive: start simple, and don't use an autonomous agent when a well-defined workflow will do.

Anthropic's foundational guidance, laid out in "Building effective agents," distinguishes between workflows and agents. A workflow is a predefined sequence of steps with clear decision points. An agent is an autonomous system that decides its own next steps. The difference matters enormously.

The execute-first problem in OpenClaw stems from a fundamental architectural choice: giving the agent full autonomy over ambiguous tasks without building in confirmation gates. In workflow-based architectures, ambiguous steps trigger explicit checkpoints -- the system asks the user before proceeding. In purely autonomous architectures, the agent fills in blanks and acts.

For practitioners, the key patterns here are:

ReAct (Reasoning + Acting): The agent reasons about what to do, takes an action, observes the result, and then reasons again before the next step.
Planning: The agent creates a plan before execution, allowing for human review of the intended steps.
Human-in-the-loop gates: Critical actions require explicit approval before execution.

The rule of thumb: if a task has clear inputs and outputs, use a workflow. If it requires judgment under uncertainty, use an agent -- but with confirmation gates for high-risk actions.

For Practitioners: Read "Building effective agents" and "Building agents with the Claude Agent SDK" on Anthropic's Engineering Blog.

Pillar 2: Tool Capabilities -- Think Before You Act

The OpenClaw failure it addresses: Reckless execution without reasoning.

An agent is only as good as its tools -- and more importantly, how it decides to use them. Tool description design directly affects how well an agent selects and invokes the right tool at the right time. A vague tool description leads to misuse; a precise one guides the agent toward correct behavior.

But the real breakthrough in this space is Anthropic's Think Tool -- a technique that lets agents perform chain-of-thought reasoning before taking any action. Instead of immediately executing, the agent pauses, reasons through its options, considers edge cases, and only then acts.

This is the direct antidote to "execute first, ask later." The Think Tool essentially gives the agent an internal monologue: "Wait -- is this instruction ambiguous? What are the possible interpretations? Which one is most likely? Should I ask for clarification?"

In practice, the Think Tool significantly improves performance on complex reasoning tasks, especially those involving:

Ambiguous instructions with multiple valid interpretations
Multi-step tasks where an early mistake compounds
Tasks requiring judgment about when to ask for help

Beyond the Think Tool, production-grade tool systems need Agent Skills -- reusable, encapsulated capabilities that an agent can invoke like a professional using standardized procedures. Skills turn one-off problem-solving into repeatable expertise.

For Practitioners: Read "The 'think' tool," "Writing effective tools for agents -- with agents," and "Equipping agents for the real world with Agent Skills" on Anthropic's Engineering Blog.

Pillar 3: Context Engineering -- Memory That Doesn't Lose the Plot

The OpenClaw failure it addresses: Context compression dropping safety instructions.

Even as AI model context windows expand to hundreds of thousands of tokens, context engineering remains critical. A larger window doesn't solve the fundamental problem: the model's attention is a scarce resource, and what you put into the context window -- and how you structure it -- determines whether the agent remembers its safety instructions or forgets them under load.

Context compression losing safety rails is not a theoretical risk. It's a documented failure mode. See the Analyzing the Incident of OpenClaw Deleting Emails: A Technical Deep Dive for more details. When the information volume exceeds what the system can handle, something gets squeezed out. In OpenClaw's case, that "something" was often the safety guardrails themselves.

The solution isn't just "bigger context windows." It's context engineering -- the deliberate management of what goes into the agent's working memory, when, and in what form.

Key techniques include:

Memory management: Explicitly structuring what the agent remembers across turns and sessions, rather than relying on raw conversation history.
RAG (Retrieval-Augmented Generation): Instead of cramming everything into the context window, retrieve only the information relevant to the current task. This keeps the context focused and prevents safety instructions from being crowded out.
Contextual Retrieval: An innovation from Anthropic where the model generates explanatory context before retrieval, solving the classic RAG problem of chunk-level information loss.

An emerging open-source approach tackles this from a different angle. MemPalace (33K+ GitHub stars) takes the position that the problem isn't what the AI remembers -- it's what it forgets when memory gets compressed. Instead of having the AI decide what's worth keeping (and risk discarding safety instructions), MemPalace stores everything verbatim and uses a structured navigation system -- inspired by the ancient Greek memory palace technique -- to make it findable without loading it all into context.

The architecture is a layered memory stack that directly addresses context pressure:

Layer	What it holds	Size
L0	Identity -- who is this AI?	~50 tokens
L1	Critical facts -- team, projects, preferences	~120 tokens
L2	Room recall -- recent sessions, current topic	On demand
L3	Deep search -- semantic query across all stored memories	On demand

The agent wakes up with only ~170 tokens (L0 + L1) and searches deeper layers only when needed. This keeps the context window lean and focused. Memories are organized into "wings" (projects/people), "rooms" (topics), and "halls" (memory types like decisions, events, discoveries), with "tunnels" cross-referencing the same topic across domains. This structured retrieval scored 96.6% recall on the LongMemEval benchmark -- the highest published result for a free, local-only system with zero API calls.

Critically for the context compression problem, MemPalace includes a PreCompact hook that fires before the context window is compressed, performing an emergency save of the current session. This is a direct architectural response to the failure mode that caused the Meta email deletion incident: if the agent's safety instructions live only in the context window, they can be summarized away. MemPalace externalizes memory so that compression never touches what matters.

The principle: treat the context window like a surgeon's tray, not a junk drawer. Every token should earn its place. Safety instructions should be architecturally pinned, not left to compete with task data for the model's attention.

For Practitioners: Read "Effective context engineering for AI agents" and "Introducing Contextual Retrieval" on Anthropic's Engineering Blog. For an open-source, local-first approach to structured memory, see MemPalace.

Pillar 4: Long Tasks & Collaboration -- Surviving the Marathon

The OpenClaw failure it addresses: No state recovery, runaway execution.

Demo agents handle single-turn tasks. Production agents run for minutes, hours, or days. The difference is enormous.

A long-running agent needs what Anthropic calls a harness -- an execution framework designed for durability. The harness handles what happens when things go wrong: network interruptions, model errors, infinite loops, context window exhaustion. Without a harness, a long-running agent is a ticking time bomb -- one crash and all progress is lost.

The core capabilities a harness must provide:

State persistence: If the agent crashes, it can resume from where it left off, not from scratch.
Interruption recovery: External disruptions (network outages, API rate limits, user cancellation) are handled gracefully.
Loop detection: The agent recognizes when it's stuck in a cycle and breaks out, rather than burning tokens endlessly.
Resource budgets: Hard limits on tokens, time, and API calls prevent runaway costs.

For complex tasks that exceed what a single agent can handle, the Orchestrator-Workers pattern distributes work across multiple agents coordinated by a central orchestrator. This is how Anthropic built their own multi-agent research system -- one agent plans, others execute specialized subtasks, and the orchestrator synthesizes results.

The practical implication: if your agent can run for more than a few minutes, you need a harness. If it can run unsupervised, you need budgets and kill switches. The users who discovered their OpenClaw instances burning money wildly learned this lesson the hard way.

But a harness alone isn't enough. A long-running agent can stay alive, recover from crashes, and stay within budget -- and still silently degrade in quality over time. This is where continuous evaluation becomes essential. Anthropic's guide on defining success criteria and building evaluations lays out a disciplined framework that applies directly to long-running agent services.

The key insight: success criteria for agents must be specific, measurable, achievable, and relevant -- not vague goals like "performs well." For a long-running agent, this means defining quantitative thresholds upfront: What is the acceptable error rate per 10,000 actions? What is the maximum response latency? What percentage of edge cases must be handled without human intervention?

The framework distinguishes three grading methods, ranked by preference:

Code-based grading -- fastest, most reliable. Exact match, string match, programmatic checks. Use this wherever possible.
LLM-based grading -- fast and flexible, suitable for complex judgments like tone, coherence, and context utilization. Requires clear rubrics and validated reliability before scaling.
Human grading -- most flexible but slowest. Avoid for ongoing monitoring; reserve for calibrating automated methods.

For long-running agents specifically, the context utilization evaluation is critical: it measures whether the agent is still coherently using information from earlier in the conversation, which is exactly the capability that degrades under context pressure. The consistency evaluation catches drift -- if the agent starts giving different answers to semantically similar questions over time, something has gone wrong. And privacy preservation evaluations can detect when an agent starts leaking sensitive information that it should be filtering, a risk that compounds the longer an agent runs with accumulated context.

The principle that ties this back to the harness: a harness keeps the agent running; evaluations tell you whether it's still running correctly. Loop detection catches infinite cycles. Evals catch silent quality degradation. You need both.

For Practitioners: Read "Effective harnesses for long-running agents," "How we built our multi-agent research system," and "Code execution with MCP" on Anthropic's Engineering Blog. For evaluation methodology, see Anthropic's Define success criteria and build evaluations guide.

Pillar 5: Safety, Evaluation & Monitoring -- The Last Mile

The OpenClaw failure it addresses: Excessive permissions, prompt injection, no production safeguards.

This is the pillar where most teams skip steps -- and where the consequences are most severe. The numbers from OpenClaw tell the story: 230,000 exposed instances, 87,800 data leaks, a CVSS 8.8 remote code execution vulnerability.

Three practices are non-negotiable for production agents:

Sandboxing. When an agent can execute code, it must do so in an isolated environment that cannot access credentials, sensitive files, or system-level permissions. OpenClaw runs with the user's full system permissions. Anthropic's Managed Agents architecture puts the sandbox in a separate container that can never touch credentials -- authentication goes through a vault proxy, and the harness itself has zero awareness of any credentials.

Least privilege. The agent should have exactly the permissions it needs for the current task, and no more. Permissions should be granted per-task and revoked when the task completes. Standing permissions are standing risks.

Evaluations (Evals). Anthropic's guidance is unambiguous: without evals, don't go live. An automated evaluation system that tests agent behavior against known scenarios -- including adversarial ones like prompt injection -- is the only way to know whether your agent is safe before it touches production data. Relying on manual testing or intuition is not engineering; it's hope.

The difference between OpenClaw's 57% prompt injection robustness and a production-grade system isn't just better prompting -- it's architectural. Security must be designed into the boundary between components, not bolted on as a configuration option.

For Practitioners: Read "Demystifying evals for AI agents," "Beyond permission prompts: Claude Code sandboxing," and "A postmortem of three recent issues" on Anthropic's Engineering Blog.

Anthropic's Answer: The Operating System Approach

With the 5 pillars as context, Anthropic's Managed Agents architecture comes into sharper focus. It's not just a hosting service -- it's a deliberate embodiment of these principles.

Separating Session, Harness, and Sandbox

The core design decision is to thoroughly separate three components that most agent frameworks cram into a single container:

Component	Role	Analogy
Session	The log of what happened	The agent's notebook
Harness	The loop of calling Claude and routing tool calls	The nervous system
Sandbox	The execution environment where code runs	The workshop

Previously, all three lived in one container. If it crashed, the session was lost. Engineers had to babysit. Anthropic calls this the "pets" model -- each container is precious, irreplaceable, and needs constant attention.

After separation, containers become "cattle." If one dies, spin up a new one. The session is stored externally. The harness resumes via wake(sessionId), reads the event log, and continues running. Any component can crash or be replaced independently.

Think of it like a restaurant kitchen. The "pets" model is a restaurant with one chef who does everything -- if that chef gets sick, the restaurant closes. The "cattle" model is a kitchen brigade: prep cooks, line cooks, and a head chef, each replaceable. The recipes (session) are written down. The process (harness) is standardized. The cooking stations (sandbox) are interchangeable.

Security by Architecture

The security redesign directly addresses the "keys to the kingdom" problem:

Old design: Agent-generated code and system credentials ran in the same container. A prompt injection only needed to convince the model to read its own environment variables to steal tokens.
New design: The sandbox can never touch credentials. Authentication goes through a vault proxy. The harness has zero awareness of any credentials.

This isn't a configuration toggle. It's a boundary enforced by the architecture itself.

Performance Results

The performance impact of this separation is dramatic:

p50 (median) time-to-first-token latency dropped 60%
p95 (tail) time-to-first-token latency dropped over 90%

Separating concerns doesn't just improve reliability -- it improves speed. When the harness doesn't have to manage the sandbox's lifecycle, it can focus on what it does best: routing model calls.

The OS Analogy

Anthropic draws a comparison to operating systems: an OS virtualizes hardware into stable abstractions -- "processes," "files," "sockets" -- that outlast any generation of hardware. The read() system call worked on 1970s disk drives and works on today's SSDs.

Managed Agents does the same thing for agents: virtualizing core components into stable interfaces, so upper-level logic doesn't break when the model gets smarter or the framework evolves. Every model generation makes some harness code obsolete -- Anthropic calls this the "structural dilemma of the harness industry." Their solution is to own the interface and let the implementation evolve underneath.

Early Adoption

The approach is already in production:

Notion integrated agents into its workspaces, supporting dozens of concurrent tasks.
Rakuten deployed department-specific agents (product, sales, finance, HR) within a week, connected to Slack and Teams.
Sentry has agents automatically writing bug-fix patches and opening PRs -- an integration originally estimated at months that went live in weeks.

Open Source Still Matters: Two Paths Forward

Managed Agents is Anthropic's answer. But the open-source world offers two genuinely different alternatives -- and understanding the contrast reveals what "agent value" actually means.

OpenClaw: The Platform Path

OpenClaw's core logic is that of a platform or gateway. Think of it as a dispatch center. It unifies chat entry points -- Telegram, Slack, Discord, WhatsApp -- connects different models, different tools, and different workflows. It's a multi-channel personal assistant operating system.

This direction has real value. People's information entry points are inherently scattered. Whoever can unify those entry points gets closer to being a truly usable personal AI hub.

OpenClaw's strength: Integration, distribution, ecosystem, platform coverage.

OpenClaw's weakness: The security model relies on trust and configuration auditing. As Cisco noted, the issues are architectural, not configurational. The ClawHub skill marketplace -- with 36.8% of skills containing security flaws -- demonstrates what happens when a platform grows faster than its safety infrastructure.

Hermes Agent: The Growth Path

Hermes Agent starts from a fundamentally different premise. It doesn't deny the importance of integration, but what it truly emphasizes is: will this agent accumulate capability over long-term use?

Where OpenClaw cares about how an agent connects to the world, Hermes cares about how an agent continuously evolves within the world.

Hermes's most distinctive capability is its learning loop. After completing a task, the agent doesn't just finish -- it distills the process into a structured Skill, a reusable method template. The next time it encounters a similar problem, it invokes that crystallized experience instead of starting from scratch.

Its memory architecture goes beyond storing chat history:

Layer	What It Stores
Layer 1	Who you are -- persistent background context
Layer 2	What you've done -- full history, recalled on demand
Layer 3	How to do similar things better -- skills extracted from experience

This is "user model + task model + method library" -- the architecture of a long-term partner, not a one-shot tool.

On security, Hermes takes a markedly different approach from OpenClaw, implementing five-layer defense-in-depth:

User authorization
Dangerous command review
Container isolation
Credential filtering
Context injection scanning with auto-reject on timeout

Compare this to OpenClaw's trust-plus-configuration model, and the architectural gap is clear.

Three Philosophies, One Set of Challenges

	OpenClaw	Hermes Agent	Anthropic Managed Agents
Philosophy	Gateway / Platform	Growth Engine	Operating System
Core Value	Connection	Accumulation	Abstraction
Security Model	Trust + config	Defense-in-depth	Architecture-level isolation
Best For	Multi-channel hubs	Long-term projects	Enterprise production
Trade-off	Breadth over safety depth	Newer, smaller ecosystem	Vendor lock-in

The choice isn't "managed vs. open-source." It's which design philosophy matches your use case -- and whether the 5 pillars are addressed regardless of which path you take.

Principles Over Frameworks

Tools change. Frameworks rise and fall. Model capabilities leap forward every few months, turning yesterday's clever harness code into tomorrow's technical debt.

But the engineering principles endure:

Start with workflows, graduate to agents. Don't give autonomy before you've built confirmation gates.
Make the agent think before it acts. Chain-of-thought reasoning is not optional for production systems.
Treat context like a scarce resource. Pin safety instructions architecturally; don't let them compete with task data for attention.
Design for crashes, not just success. State persistence, interruption recovery, and resource budgets are production requirements, not nice-to-haves.
Security is architecture, not configuration. If your agent and your credentials share a container, you don't have a security model -- you have a vulnerability.

These five pillars matter whether you use Anthropic's Managed Agents, OpenClaw, Hermes Agent, or build your own infrastructure from scratch.

Anthropic's engineering blog ends with a statement that reads like technical humility:

"We have opinions about the form of the interface, but we don't have opinions about what specific harness Claude will need in the future."

But the precondition for saying this is that they've already taken control of the interface itself. The interface -- the 5 pillars, the stable abstractions -- is what endures. The implementation is what evolves.

For those of us building with agents, the lesson is the same one software engineering has taught for decades: invest in the interfaces, not the implementations. The frameworks will change. The principles won't.

References

Sources referenced in this post, organized by topic. Anthropic's 15 engineering blog posts are listed by module; reading them in order provides a structured path from agent fundamentals to production readiness.

Security Research

A Trajectory-Based Safety Audit of Clawdbot (OpenClaw) -- Tianyu Chen et al., ShanghaiTech University & Shanghai AI Lab. The trajectory-centric security evaluation referenced in this post, covering six risk dimensions of OpenClaw's agentic behavior (arXiv:2602.14364).

Context Compression & Safety Instruction Loss

Analyzing the Incident of OpenClaw Deleting Emails: A Technical Deep Dive -- John Ding. How Meta AI Safety Director Summer Yue's "don't action until I tell you" instruction was lost during context compaction, causing 200+ email deletions.
Why AI Agents Fail: Context Compaction Explained -- Let's Data Science. Covers the Meta incident, CVE-2026-25253, and the broader context compaction failure pattern.
Why AI Agents Bypass Human Approval: Lessons from Meta's Rogue Agent Incidents -- Waxell. Architectural analysis of why prompt-based human-in-the-loop fails under context pressure and why infrastructure-layer enforcement is needed.
safeguard compaction fails to recover when context significantly exceeds model limit -- OpenClaw GitHub Issue #5357. Documents compaction failure when context exceeds token limits by more than 20%.
Default compaction mode silently fails on large contexts -- OpenClaw GitHub Issue #7477. Documents silent summarization failure producing "Summary unavailable" instead of preserving conversation history.

Open-Source Memory Systems

MemPalace -- Milla Jovovich & Ben Sigman. Local-first, structured AI memory system using a palace metaphor (wings, rooms, halls, tunnels) with verbatim storage and semantic search. 96.6% recall on LongMemEval with zero API calls. Includes PreCompact hooks to save memory before context compression.

Evaluation & Testing

Define success criteria and build evaluations -- Anthropic. Official guide on designing measurable success criteria and automated evaluation systems for LLM-based applications, with code examples for exact match, cosine similarity, ROUGE-L, LLM-based Likert scale, and binary classification grading.

Managed Agents Announcement

Managed Agents -- Anthropic's engineering deep-dive on the architecture behind Claude Managed Agents.

Module 1: Foundation Architecture

Building effective agents -- Agent architecture introduction: workflows vs. autonomous agents, ReAct, Tool Use, Planning.
Building agents with the Claude Agent SDK -- Practical getting started with the Agent SDK.

Module 2: Tools & Capability Extension

Introducing advanced tool use -- Advanced tool usage: parallelism, barriers, and error handling.
Writing effective tools for agents -- with agents -- Tool design principles and best practices.
The "think" tool -- Teaching agents to stop and reason before acting.
Equipping agents for the real world with Agent Skills -- Skill encapsulation and reuse.

Module 3: Context & Memory Management

Effective context engineering for AI agents -- Managing the agent's memory and attention across long conversations.
Introducing Contextual Retrieval -- Making RAG more context-aware to reduce chunk-level information loss.

Module 4: Long Tasks & Multi-Agent Collaboration

Effective harnesses for long-running agents -- Designing reliable execution frameworks with interruption recovery and state persistence.
How we built our multi-agent research system -- Anthropic's practical experience with multi-agent architecture.
Code execution with MCP -- Agent execution environment design using the Model Context Protocol.

Module 5: Safety, Evaluation & Engineering

Demystifying evals for AI agents -- Evaluation system design for agent behavior.
Beyond permission prompts: Claude Code sandboxing -- From permission prompts to sandbox isolation.
Claude Code: Best practices for agentic coding -- Engineering best practices for coding agents.
A postmortem of three recent issues -- Real-world agent incident case studies.

A Claude Code Skills Stack: How to Combine Superpowers, gstack, and GSD Without the Chaos

Yaohua Chen — Mon, 06 Apr 2026 23:30:15 +0000

One article to compare the frameworks, see where they overlap, and land on a stable three-layer practice.

Introduction

Claude Code has quickly become one of the most widely adopted AI coding tools. Individual developers, startups, and large engineering teams alike have integrated it into their daily workflows—writing production code, reviewing pull requests, debugging, and shipping features at a pace that was hard to imagine a year ago. As usage has scaled, so has the ecosystem around it. Claude Skills—composable, auto-invoked instruction sets that shape how the agent plans, builds, and verifies—have emerged as one of the most important extension points in Claude Code. They let you go beyond one-off prompts and encode repeatable workflows directly into the agent's behavior. In fact, Anthropic has doubled down on this direction: the latest version of Claude Code consolidates the previously separate "slash commands" and "skills" systems into a single, unified skills format, signaling that skills are now the canonical way to extend the agent.

With Skills now central to the experience, the community has rallied around a handful of open-source frameworks that package best practices into ready-made skill sets. The two most discussed stacks are Superpowers and gstack. Installing both sounds easy; in practice they can conflict, and piling frameworks on without a plan often makes the setup less stable, not more. So where do they differ, and how should you choose?

This post does three things:

Compare Superpowers and gstack on repos, features, and philosophy—the material below on stars, skill lists, and trade-offs.
Add a third layer many guides skip: GSD as a context / spec stabilizer so long-running work does not drift (informed by Tricia Notes Editorial’s three-layer framing).
End with a single playbook: who owns decision, context, and execution, and how to cherry-pick skills without blowing up token use or cognitive load.

The useful question is not only “Superpowers or gstack?” but: what are you missing—decision-making, durable context, or execution?

In one line: gstack thinks, GSD stabilizes, Superpowers executes.

Orientation: Three Layers, Not Only Two

What stays stable in practice is often not picking one framework over another, but a three-way division of labor.

Layer	Stack	Role
Decision / roles	gstack	Judgment from CEO, design, architecture, QA-style lenses—not only “how to code.”
Context / spec	GSD	Keeps spec, status, boundaries, and long-horizon context from rotting.
Execution	Superpowers	Requirement clarification → plan → TDD → acceptance as a closed loop.

How each is “strong”:

Superpowers — How work gets done; smooth execution loop.
gstack — What to do and whether it should be done; richer role-based judgment.
GSD — Not drifting; steadier specs and context over long chains.

Both Superpowers and gstack have gone viral. On the surface they add process to AI; in use, they help you think clearly about what matters. When the model codes fast, that is exactly when you need clear requirements and stable context—that is what most people still overlook.

Superpowers vs gstack: Quick Facts

Superpowers (GitHub ~137K stars)

Repository: obra/superpowers
An Agent Skills framework and software development methodology: 14 built-in skills across brainstorming, planning, TDD, execution, and verification.

gstack (GitHub ~65K stars)

Repository: garrytan/gstack
From YC CEO Garry Tan, open source.
Philosophy: a team beside you—CEO, designer, eng manager, release manager, doc engineer, QA, and more—23 opinionated tools (product thinking, CEO review, architecture review, real browser testing, design review, security audits, etc.).
Garry has claimed 600K+ lines of production code (35% tests) in 60 days, part-time while running YC full-time.

Stars are a weak proxy: high star count does not mean every skill fits your workflow.

Feature Comparison (Superpowers vs gstack)

Category	Superpowers	gstack
Product brainstorming	brainstorming	/office-hours, /plan-ceo-review
Architecture planning	writing-plans	/plan-eng-review, /autoplan
Design	—	/design-consultation, /plan-design-review, /design-shotgun, /design-html
Development execution	executing-plans, subagent-driven-development, dispatching-parallel-agents	—
Testing	test-driven-development	/qa, /qa-only
Debugging	systematic-debugging	/investigate
Code review	requesting-code-review, receiving-code-review	/review, /codex
Verification & acceptance	verification-before-completion, finishing-a-development-branch	/ship, /land-and-deploy, /canary, /document-release
Security	—	/cso, /careful, /freeze, /guard, /unfreeze
Observability	—	/learn, /retro
Browser testing	—	/browse, /connect-chrome, /setup-browser-cookies
Git worktrees	using-git-worktrees	—
Skill management	using-superpowers, writing-skills	/gstack-upgrade
Performance	—	/benchmark
Deployment	—	/setup-deploy

Coverage differs a lot; quantity is not the point—design philosophy is.

Design Philosophy: “How” vs “What” (and Where GSD Fits)

Superpowers — focused on how code gets built

The workflow centers on high-quality output: clarify, plan, TDD (tests before implementation), verify. Checkpoints at each step—little room to skip. In practice it feels disciplined: you ask for X, it tends to build X. Engineers who already know what to build often find that empowering.

(Execution-layer detail from hands-on use: strong process and steady execution; small tasks can still feel **heavy* because the full rhythm applies even to tiny asks.)*

gstack — focused on what and what not to do

Before heavy coding, flows like /office-hours walk requirements; CEO and engineering reviews stress-test the approach. It is not only code—it can run real browser tests from a user angle. Rough split:

Decision layer: /office-hours, /plan-ceo-review, /plan-eng-review
Execution layer: /review, /qa, /ship, etc.

gstack shines when requirements are still fuzzy—PMs, indies, or “think while building.” Caveat: turning all roles on can feel bloated; decision skills also burn serious tokens (see below).

GSD — context / spec, not another “team chart”

GSD is not “install another team.” It is context engineering: goals, specs, status, boundaries, and summaries anchored so context rot slows down. Short demos hide this; long projects show it—when context wobbles, output scatters; that is state, not only “bad execution.”

gstack thinks but is not, by itself, a long-term context vault.
Superpowers executes but is not, by itself, a spec/context system.
GSD fills that gap so chains stay coherent.

Three-Way Comparison (Problems, Not “Who Wins”)

Dimension	Superpowers	gstack	GSD
Core question	How to get things done	What to do; whether it should	How to keep the project from diverging
Layer	Execution	Decision / roles	Context / spec
Strongest fit	Planning, TDD, acceptance loop	Multi-perspective judgment, review, QA	Context engineering; stable state
Best for	Clear requirements	Think-while-building	Long chains / many iterations
Common pain	Front-loaded process can feel heavy (details below)	Bloated and token-hungry when fully enabled (details below)	Little standalone “shipping” value on its own (details below)
Role	Own execution	Own decision-making	Own long-term context

Common Pain Points in Detail

Superpowers — front-loaded process can feel heavy. Every task, no matter how small, runs through the full cycle: clarify requirements, draft a plan, write tests first, then implement, then verify. For a large feature this rhythm pays off handsomely. For a two-line config fix or a quick copy change, the same ceremony kicks in and you end up spending more time on process than on the actual change. The overhead does not scale down with task size, so small requests can feel disproportionately slow.

gstack — bloated and token-hungry when fully enabled. Each gstack role (CEO, designer, architect, QA, etc.) injects its own perspective and prompts into the context. Turn them all on and a single execution-layer skill can consume 10K+ tokens before any real code is written. Daily usage burns through tokens fast, and the back-and-forth between multiple “virtual team members” can make even straightforward tasks feel sluggish and redundant. You may also encounter irrelevant meta-questions (e.g. “Are you applying to become a YC company?”) while your codebase is being scanned—artifacts of the framework’s opinionated persona layer.

GSD — little standalone “shipping” value. GSD excels at keeping specs, goals, and state anchored across long sessions. But if you use it alone, it does not directly produce code, run tests, or open a PR. It is a stabilizer, not a builder. Without an execution layer (Superpowers) or a decision layer (gstack) alongside it, GSD manages context that nothing acts on—useful plumbing, but no visible output. Its value only becomes apparent when paired with tools that actually ship work.

Practical takeaway: they are complements, not substitutes—Superpowers executes, gstack decides, GSD stabilizes specs and context over time.

Strengths, Weaknesses, and Friction

Superpowers

Strengths: Brainstorming and overall workflow feel solid; full process even on small asks can become smooth once habitual; execution and TDD are strong.
Weaknesses: Weaker spots are often early decision skills (e.g. planning/brainstorming) compared to gstack’s decision layer—hence many people pair gstack’s front end with Superpowers’ execution.

gstack

Strengths: Decision layer—/office-hours, /plan-ceo-review, /plan-eng-review—stand out for positioning and approach review.
Weaknesses: Execution feels rougher vs Superpowers; token cost is real—a single execution-layer skill can cost 10K+ tokens, and heavy scans can feel like noisy “process” rather than help.

The analogy

Superpowers is a scalpel — precise and efficient.

gstack is a full clinic — from diagnosis to aftercare.

Use the metaphor to choose depth: narrow execution vs full-spectrum product and review.

Consolidated Best Practices

1. Choose skills deliberately—do not install everything

Skill counts spiral easily (Superpowers today, gstack tomorrow, another stack next week). Selective deployment beats volume; random invocation feels unstable and inflates surface-level “skill count” without clarity.

Underlying idea: both stacks are experiments in Harness Engineering. The mindset is leverage strengths, cover weaknesses—not “I want it all.”

2. Decision vs execution (the classic split)—then add context when needed

gstack for the decision layer (cherry-picked):

Prioritize high-value flows: e.g. /office-hours, /plan-ceo-review, /plan-eng-review for requirements and alignment—avoid over-investing in every role.

Superpowers for the execution layer:

Prefer Superpowers as the base for TDD, plans-as-executed, verification—optionally de-emphasize its own heavy decision skills if gstack already covers that phase, so small tasks do not inherit double process.

GSD when the chain diverges:

If work spreads across sessions and threads, add GSD so spec and state stay anchored—not for flash, for anti-drift.

3. Stable workflow (three steps)

Decision → gstack — Start with /office-hours to stress-test the idea, then run /plan-ceo-review for a founder-level sanity check and /plan-eng-review to lock architecture and data flow. If design matters, add /plan-design-review. The goal: decide what to build and whether to build it before touching code.
Context → GSD — Once the decision is made, use GSD (v2) to anchor the plan: PROJECT.md for what the project is, DECISIONS.md for architectural choices, KNOWLEDGE.md for cross-session rules and patterns, and milestone roadmaps (M001-ROADMAP.md) for sliced execution. These v2 artifacts keep spec, status, and boundaries stable so context does not rot between sessions. (The original GSD uses REQUIREMENTS.md, ROADMAP.md, and STATE.md instead.)
Execution → Superpowers — With clear requirements and stable context in place, hand off to Superpowers’ execution loop: brainstorming (if lightweight refinement is still needed), writing-plans → executing-plans for implementation, test-driven-development for the RED-GREEN-REFACTOR cycle, requesting-code-review / receiving-code-review for review, and verification-before-completion → finishing-a-development-branch to close the loop. For parallel work, use dispatching-parallel-agents or subagent-driven-development.

Merged tagline: gstack handles thinking, Superpowers handles doing, GSD keeps long context honest. Combining the strong decision slice of gstack with Superpowers’ execution (and GSD when needed) keeps skill count and collisions under control—similar to the author’s experience building a small tool on a weekend with a curated mix.

4. Final heuristics

Requirements still fuzzy → start with gstack (decision).
Work keeps diverging across the chain → add GSD (context).
You want execution steady and closed-loop → lean on Superpowers (execution).

Stop asking only: “Superpowers or gstack?” Ask: Am I missing decision, context, or execution?

Closing:

Skills are not stronger because you install more—they are stronger when you combine the right pieces for the gap you actually have and understand what each layer does, then assemble a workflow that is yours.

References

Superpowers — github.com/obra/superpowers
gstack — github.com/garrytan/gstack
GSD (Get Shit Done) — github.com/gsd-build/get-shit-done (original) | github.com/gsd-build/gsd-2 (v2, standalone CLI)

From IDE to AGaaS: How Cursor Cloud Agents Bring the OpenClaw Model to Your Slack

Yaohua Chen — Tue, 24 Mar 2026 00:07:17 +0000

TL;DR

Cursor's Cloud Agents let you delegate coding tasks — bug fixes, feature work, test writing — directly from a Slack message. The agent spins up a remote VM, clones your repo, writes the code, runs your tests, and opens a Pull Request on GitHub. You never open an IDE. This post walks you through the full setup — from Slack integration to your first hands-off pull request — and then examines where the technology shines, where it falls short, and where the AGaaS market is heading next.

What Is the OpenClaw Model — and Why Should You Care?

OpenClaw refers to an emerging paradigm in AI-assisted development where a cloud-hosted coding agent operates autonomously and headlessly — meaning it doesn't need a local IDE, a human at the keyboard, or even a screen. You give it a task in natural language, and it handles the full software development lifecycle (clone → code → test → commit → PR) on its own.

AGaaS (Agent-as-a-Service) is the broader industry term for this pattern: instead of installing AI tooling locally, you interact with a managed agent through everyday interfaces like Slack, Teams, or a web dashboard.

Cursor's Cloud Agents are a production-ready implementation of this model. If you're already using Cursor as your IDE, you can now step outside the IDE entirely and operate as a manager — assigning tasks from Slack and reviewing the output as Pull Requests.

How Cloud Agents Work Under the Hood?

Before diving into setup, here's what happens when you type @Cursor revise the README.md file to make it more professional and beginner-friendly in Slack:

Headless Execution on Isolated VMs

Traditionally, Cursor ran locally — consuming your RAM, competing for your CPU. Cloud Agents move the execution layer to a remote, isolated Virtual Machine. When a task is triggered, the agent provisions a sandboxed VM, clones your GitHub repository into it, and does all the work in the background. Your local machine stays completely free.

Each VM comes pre-loaded with a production-grade development environment:

Component	Specification
OS	Ubuntu 24.04.4 LTS (Noble Numbat), Linux kernel 6.12.58+, x86_64
Hardware	4 CPU cores, 15 GB RAM, ~126 GB disk (overlay filesystem)
Runtimes	Python 3.12.3, Node.js v22.22.1
Toolchain	Git 2.43.0, GitHub CLI 2.81.0, Bash
Workspace	Your repo cloned at `/workspace`, running as user `ubuntu`

You can verify this yourself by asking the agent about its environment. Here's what that looks like in a real Slack conversation:

Slack Thread as Context Window

This isn't a basic chatbot that only reads your one-line prompt. Cursor's Slack integration behaves like a teammate who's been reading the whole conversation:

If your team has been discussing a bug in a thread — sharing stack traces, debating approaches, pasting logs — the agent ingests all of it when you tag @Cursor.
It synthesizes the thread context and implements a fix that reflects the team's consensus, not just your single message.

Autonomous Testing via "Computer Use"

Because the agent has its own VM with a full desktop environment, it doesn't just write code and hope for the best:

It can start your dev server, open a headless browser, and click through UI flows to visually verify the fix.
If tests fail or the UI breaks, it self-corrects before submitting the Pull Request.

Now that you understand what's happening behind the scenes, let's set it up. The whole process takes about 15 minutes.

Step-by-Step Setup

Prerequisites

Before you begin, make sure you have the following in place:

Requirement	Details
Cursor subscription	Cloud Agents require a paid plan — Pro ($20/mo), Pro+, Ultra, or Teams. Check your plan at cursor.com/pricing.
GitHub account	Your repository must be hosted on GitHub or GitLab. You need read-write access to the repo.
Slack workspace	You need admin permissions (or the ability to request app installation) in your Slack workspace.
Existing test suite	Recommended but not required. The agent can run your tests automatically if they exist (e.g., `npm test`, `pytest`, `go test`).

Step 1: Connect Slack to Cursor

Open the Cursor Dashboard at cursor.com/dashboard.
Navigate to the Integrations & MCP tab.
Click Connect next to Slack. This launches an OAuth flow that installs the Cursor bot into your Slack workspace.
Authorize the requested permissions (read messages in channels where the bot is invited, post replies).

Step 2: Connect Your GitHub Repository

In the same Dashboard, go to the Cloud Agents > Default Repositories > Manage Repositories section.
Click Add Repository and authenticate with GitHub.
Select the repository (or repositories) you want the Cloud Agent to access.
Grant the agent permission to create branches and open Pull Requests.

Step 3: Configure the Cloud Agent Environment

Before triggering tasks from Slack, configure the Cloud Agent's development environment and defaults in the Cursor dashboard. Navigate to Cloud Agents in the left sidebar.

3a. Set Your Defaults

Under the My Settings tab, configure the following:

Setting	What It Controls	Example Value
Default Model	The AI model the agent uses when no model is specified in the task. Higher-tier models produce better code.	`Opus 4.6 High Fast`
Default Repository	The GitHub repo the agent targets when no repo is mentioned in the Slack message.	`chen115y/MLOpsLearning`
Base Branch	The branch the agent creates feature/fix branches from. Leave empty to use the repo's default branch.	`main`
Branch Prefix	Prepended to every branch the agent creates, making agent-authored branches easy to filter.	`cursor/`

3b. Set Up a Development Environment

For repositories with complex dependencies (Python data-science stacks, system libraries, database services), click Add Environment button. This launches a very simple setup agent that provisions and validates the VM:

Once all fields are filled, click Start For Free to start the VM provisioning. The setup agent will analyze the repository and provision the VM accordingly.

Tip: You can add multiple environments for different repos. If the setup agent reports warnings (e.g., deprecated API calls in older notebooks), these are pre-existing code issues, not environment problems — the snapshot is still safe to save.

Step 4: Create a Channel and Invite the Bot

In Slack, create a dedicated channel for agent-assisted work (e.g., #engineering-triage, #cursor-tasks, or #bug-reports).
Simply mention @Cursor in the channel with any prompt — the bot joins automatically when the Slack app is installed (Step 1). No separate invite is needed.
You can also type @Cursor help to see available commands, or @Cursor settings to configure channel-level defaults.

Step 5: Configure Cursor Rules (the Agent's Playbook)

This is the most important step. Without rules, the agent will make reasonable guesses about your codebase conventions. With rules, it follows your team's standards precisely.

Create a .cursor/rules/triage.mdc file in your repository root, for example:

---
description: "Rules for Slack-triggered bug triage and feature tasks"
globs:
  - "**/*"
alwaysApply: true
---

# Agent Behavior for Slack Tasks

## Bug Triage Protocol
1. Read the full Slack thread for context, including any error logs or stack traces.
2. Search the codebase to locate the relevant source files.
3. Identify the root cause before writing any fix.
4. Write the fix following existing code patterns in the repository.
5. Use the project's standard error-handling approach (check for existing wrappers).

## Testing Requirements
- Run the full test suite: `npm run test` (or the project's equivalent).
- If no tests exist for the changed code, write at least one unit test covering the fix.
- Do not submit a PR if tests fail. Debug and fix until green.

## Git and PR Conventions
- Create a new branch from `main` with the format: `fix/<short-description>`.
- Never push directly to `main` or `develop`.
- PR title format: `fix: <concise description of the change>`
- Include a summary of the root cause and fix in the PR description.
- Reply to the original Slack thread with the PR link and a brief explanation.

## Out of Scope
- Do not modify CI/CD configuration files without explicit approval.
- Do not upgrade dependencies unless the fix requires it.
- If the issue is unclear, ask clarifying questions in the Slack thread before proceeding.

You can create additional rule files for different workflows — feature development, refactoring, documentation — each with its own conventions.

Step 6: Run Your First Agent Task

With everything connected, you're ready to give the agent its first job. Post a message in your channel (or reply in an existing thread) and tag @Cursor with a clear task description. The agent picks it up, executes the work on its remote VM, and reports back — all within the same Slack thread.

Here's a real example. A user asks the agent to revise a repository's README to make it more professional and beginner-friendly. Within minutes, the agent replies with a structured breakdown of every change it made — reorganized navigation, plain-language introductions, typo fixes, new formatting — along with the commit diff (+338 / -190 lines):

The user asks the agent to make a commit and push the changes directly to the remote repository. Once the work is done, the agent confirms it has committed and pushed the changes directly to the remote repository, and provides a link to verify on GitHub:

Want to see how the agent reasoned through the task? Click the "Open in Web" button in the Slack message to open the full agent session. This view shows the agent's step-by-step thought process — the file diff it analyzed, the to-do list it created for itself (commit, push), and the detailed revision plan it followed:

And to close the loop, here's the GitHub repository immediately after. Notice the README.md row — updated "1 minute ago" by cursoragent with the commit message matching exactly what the agent described in Slack:

No IDE opened. No branch created manually. No code written by hand. One Slack message in, a polished commit out.

Writing Effective Cursor Rules: A Deeper Look

The example above worked smoothly because the task was straightforward. But as you start assigning more complex work — multi-file refactors, feature additions, cross-cutting bug fixes — the quality of the agent's output depends heavily on how well you've defined your team's standards. That's where Cursor Rules go from "nice to have" to essential.

Step 5 introduced the basic format. Here we'll look at patterns that make rules genuinely effective at scale.

Scope rules by file type. Use the globs field to apply different rules to different parts of your codebase:

---
description: "Frontend component conventions"
globs:
  - "src/components/**/*.tsx"
---
- Use functional components with hooks, never class components.
- All components must have a corresponding .test.tsx file.
- Use the project's design system tokens for colors and spacing.

---
description: "API route conventions"
globs:
  - "src/api/**/*.ts"
---
- Validate all request bodies with zod schemas.
- Return consistent error response shapes: { error: string, code: number }.
- Log errors with the structured logger, not console.log.

Be specific about what the agent should not do. Guardrails prevent expensive mistakes:

## Boundaries
- Never delete database migration files.
- Never modify environment variable files (.env, .env.local).
- If a change requires more than 5 files, stop and ask for confirmation in Slack.

At this point you have the full toolkit: the agent is connected, the environment is configured, and the rules are in place. But having the setup working and knowing where to rely on it are two different things. Let me share what I've learned from using this in practice.

Where Cloud Agents Shine — and Where They Don't (Yet)

The Real Unlock: Work Anytime, Anywhere

Here's what changed my daily workflow more than any single feature: I no longer need to be at my desk, or even awake, for code to get written.

Think about that for a moment. It's 11 PM and a teammate in another timezone drops a bug report in Slack with a Datadog trace attached. Before Cloud Agents, that bug sat untouched until someone opened their laptop the next morning, cloned the repo, reproduced the issue, wrote the fix, ran the tests, and pushed a PR. That's a minimum 30-minute context-switch tax — and that's if the person was already familiar with the code.

Now? I glance at the Slack notification on my phone, type @Cursor investigate and fix this, and go back to sleep. By morning, there's a PR waiting for review with a clear explanation of the root cause. The agent read the error trace, found the offending line, wrote the fix, confirmed the tests pass, and opened the PR — all while I was unconscious.

This isn't just about convenience. It fundamentally changes when and where software development can happen. You can triage bugs from an airport lounge with nothing but your phone. You can delegate a documentation overhaul while you're deep in a design review. You can assign test-writing tasks to the agent on Friday afternoon and come back Monday to a PR that covers the gaps you've been meaning to address for weeks. The agent doesn't get tired, doesn't lose context, and doesn't need to "get back into the zone" after lunch.

What the Agent Handles Well Today

The sweet spot for Cloud Agents is any task where the goal is clearly defined and the scope is contained. Bug fixes are the most natural fit — especially when someone has already done the diagnostic work and there's an error trace, a stack dump, or a reproduction path sitting in the Slack thread. The agent can read that context, locate the relevant source files, and produce a targeted fix without anyone needing to spell out which file to open. It's remarkably good at this.

Test coverage is another area where the agent earns its keep. Most teams know they should be writing more tests, but nobody wants to write the fifteenth unit test for a utility function. Hand that to the agent. It reads the existing code, infers the expected behavior, and generates tests that follow whatever patterns your codebase already uses — pytest, jest, go test, you name it. It's not glamorous work, but it's exactly the kind of high-value, low-creativity task that agents are built for.

Small-to-medium feature additions work well too, as long as the spec is clear. "Add a CSV export button to the billing page that calls the existing exportService" is a great agent task. "Make the app feel more modern" is not — that requires taste, iteration, and subjective judgment that the agent can't provide.

The same applies to code refactoring. If you can describe the before and after state clearly — "rename all instances of getUserData to fetchUserProfile across the codebase" or "extract the validation logic from the controller into a dedicated middleware" — the agent will handle it methodically and consistently. And documentation updates? The agent writes clean, structured prose. Give it a README that's fallen out of date, and it'll cross-reference the actual codebase to produce documentation that matches reality.

Where You Still Need the IDE

That said, Cloud Agents aren't a replacement for sitting down with your code — at least not yet. There are categories of work where human judgment, rapid iteration, and architectural intuition still matter more than raw execution speed.

Large architectural changes are the clearest example. If a task spans multiple services, touches database schemas, modifies CI/CD pipelines, and requires coordinating changes across a dozen files in a specific order, the agent can get lost. It doesn't have the mental model of your system's dependency graph that you've built up over months of working in the codebase. It might fix one file in a way that breaks three others, then chase its tail fixing those. For these tasks, you want a human architect in the driver's seat, possibly using the agent for individual sub-tasks, but directing the overall strategy.

Exploratory prototyping is another area where the agent falls short. When you're experimenting — trying out a new library, playing with different UI layouts, iterating on an API design — you need a tight feedback loop. You write a few lines, run it, see what happens, change direction, try something else. That back-and-forth is the creative engine of prototyping, and it doesn't translate well to "write a Slack message and wait for a PR." The latency alone kills the creative flow.

Security-sensitive code deserves human eyes, full stop. The agent can write functionally correct authentication logic, but it won't catch the subtle timing-attack vulnerability or the OAuth misconfiguration that a security-conscious engineer would flag during a manual review. Use the agent to write the boilerplate, but review every line yourself before it touches production auth flows.

And anything requiring visual design judgment — pixel-perfect UI work, animation tuning, responsive layout decisions — still demands a human with a browser open, resizing windows, and squinting at spacing. The agent can generate the JSX and CSS, but it can't tell you whether the result feels right.

Making the Most of Imperfect Results

Here's a practical pattern that works well: the 90% handoff. The agent doesn't need to produce a perfect PR every time. If it gets 90% of the way there — the logic is right but it missed an edge case, or the implementation is solid but the naming isn't quite what you'd choose — you can pull the agent's remote session directly into your local Cursor IDE and finish the last stretch yourself. You don't start over. You continue right where the agent left off, with all the files already modified and the context preserved.

And when the agent goes in the wrong direction entirely? Course-correct in the same Slack thread. Reply with something like @Cursor stop. The issue is in the middleware, not the controller. Look at src/middleware/auth.ts instead. The agent re-reads the full thread, incorporates your feedback, and adjusts its approach. Think of it less like a tool that either works or doesn't, and more like a junior developer who's fast and tireless but occasionally needs steering.

Going Further: MCP Integrations for Closed-Loop Automation

So far, every workflow in this post has followed the same pattern: a human writes a Slack message, the agent does the work, and a PR appears on GitHub. That's already powerful — but it still requires someone to initiate each task. What if the agent could respond to events across your entire toolchain without waiting for a Slack prompt?

That's where the Model Context Protocol (MCP) comes in. MCP lets the agent interact with external tools beyond Slack and GitHub. By adding MCP servers, you can build a fully closed-loop system:

Jira / Linear: The agent automatically creates a ticket, links it to the PR, and transitions the issue status.
Datadog / Sentry: The agent queries your monitoring tools directly to pull error traces without anyone needing to paste them into Slack.
Confluence / Notion: The agent updates your team's documentation when it changes an API contract.

This turns the workflow from a Slack → PR pipeline into a Slack → Ticket → PR → Docs → Status Update pipeline — with zero manual handoff.

MCP integrations are where Cloud Agents start to feel less like a developer tool and more like infrastructure. And that shift — from tool to infrastructure — is exactly what's happening across the industry.

The Road Ahead: AGaaS and Where the Market Is Going

From Novelty to Infrastructure

What Cursor has shipped with Cloud Agents is impressive, but it's also clearly early. If you zoom out from the specifics of this one product, a much larger shift is taking shape: Agent-as-a-Service (AGaaS) is becoming a real infrastructure category, not just a buzzword.

The core idea is straightforward — instead of every developer installing AI tooling on their local machine and managing prompts, context windows, and model versions themselves, you subscribe to a managed agent that lives in the cloud, integrates with your existing tools, and operates autonomously on your behalf. Cursor is one implementation, but the pattern is bigger than any single vendor.

What Customers Actually Need (and What's Missing)

If you've followed along with this post and tried the setup yourself, you've probably already noticed a few gaps. These aren't criticisms — they're the natural rough edges of a category that's still being defined. But they point directly at where the market is heading.

Multi-repository orchestration. Today, each Cloud Agent task targets a single repository. But real-world features often span a frontend repo, a backend API, a shared library, and an infrastructure-as-code repo. The next generation of AGaaS platforms will need to coordinate changes across multiple repos atomically — opening linked PRs that reference each other and can be merged together.

Persistent agent memory. Right now, each task starts fresh. The agent doesn't remember that it fixed a similar bug last week, or that your team prefers a particular error-handling pattern, or that the last three PRs it opened for this repo all needed the same test fixture adjustment. Future agents will build a persistent understanding of your codebase, your team's preferences, and your project's history — getting better at their job over time, just like a human teammate does.

Richer feedback loops beyond Slack. Slack is a natural starting point because it's where engineering teams already communicate. But imagine triggering agent tasks from a Jira ticket transition, a Sentry alert threshold, a failing CI check, or a monitoring dashboard anomaly. The agent becomes a first-responder that patches issues before a human even notices them. Some of this is possible today through MCP integrations, but it's still manual plumbing — it should be turnkey.

Customizable execution environments at scale. The environment setup flow shown in Step 3 is a solid start, but enterprise teams need more. Think GPU-enabled VMs for ML codebases, pre-configured database fixtures for integration testing, VPN access to internal services, and compliance-scoped environments that restrict which external packages the agent can install. As AGaaS matures, the execution environment will need to match the complexity of real enterprise infrastructure.

Cost transparency and resource governance. When an agent spins up a VM, runs your test suite, and interacts with a paid AI model for 15 minutes, who pays for what? Teams need clear visibility into per-task cost breakdowns — compute, model tokens, API calls — and the ability to set budgets, quotas, and approval gates for expensive operations. This is table stakes for enterprise adoption.

Market Convergence

It's worth noting that Cursor isn't the only player moving in this direction. GitHub Copilot has introduced its own agent mode. Amazon Q Developer (formerly CodeWhisperer) has evolved toward autonomous capabilities. Smaller players like Devin, Cosine, and Factory are building agent-first platforms from scratch. The competitive pressure is accelerating the category.

What's emerging is a spectrum: at one end, lightweight copilot-style suggestions embedded in your editor; at the other end, fully autonomous agents that operate headlessly across your entire development workflow. Most teams will use both, for different tasks, at different times. The interesting question isn't which tool wins — it's how the boundaries between human-driven and agent-driven work shift over the next two to three years.

For engineering leaders, the strategic play is clear: start experimenting now. The teams that build fluency with agent-assisted workflows today — who learn which tasks to delegate, how to write effective agent rules, and how to review agent-produced code efficiently — will have a significant velocity advantage as these tools mature.

References

From Prompts to Real Files: A Developer's Guide to AI File Generation

Yaohua Chen — Mon, 16 Mar 2026 22:11:50 +0000

Ask ChatGPT to "create a sales report PDF with a revenue chart." A year ago, it would paste some markdown and wish you luck. Today, it spins up a sandboxed Python environment, runs reportlab and matplotlib, and hands you a real, downloadable PDF file.

This is the shift from text generation to artifact generation -- and every major LLM vendor now supports it through their API. Claude, OpenAI, and Gemini each give developers a way to prompt an LLM and get back actual files: PDFs, spreadsheets, charts, slide decks, whatever you can create with Python.

This post walks through the universal pattern behind file generation, then shows you exactly how to do it with each vendor -- working code included.

The Universal Pattern

Despite different APIs, all three vendors follow the same three-step architecture:

Every vendor-specific implementation is a variation on this flow. The details change, but three concepts repeat everywhere:

Tool declaration -- you opt in to code execution by including a specific tool in your API request. It's never on by default.
Sandboxed execution -- the LLM's code runs in an isolated container with no internet access. Common libraries (pandas, matplotlib, reportlab) come pre-installed.
File retrieval -- each vendor has a different mechanism to get the bytes out. Some give you a file ID to download; others return bytes inline.

Once you internalize this pattern, learning any vendor's API is just a matter of mapping it to these three steps.

Claude: Code Execution + Files API

Claude's file generation is the most full-featured option for document creation. It provides a persistent container with full bash access, a rich set of pre-installed document libraries, and a clean Files API for uploads and downloads.

Generating a PDF from a Prompt

Enable the code_execution_20250825 tool, send your prompt, then extract file IDs from the response and download them through the Files API.

import anthropic

client = anthropic.Anthropic()

# Step 1: Request with code execution enabled
response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=4096,
    tools=[{"type": "code_execution_20250825", "name": "code_execution"}],
    messages=[{
        "role": "user",
        "content": "Create a one-page PDF sales report with a revenue chart for Q1 2026."
    }]
)

# Step 2: Extract file IDs from the response
file_ids = []
for block in response.content:
    if block.type == "bash_code_execution_tool_result":
        result = block.content
        if result.type == "bash_code_execution_result":
            for item in result.content:
                if hasattr(item, "file_id"):
                    file_ids.append(item.file_id)

# Step 3: Download each generated file
for file_id in file_ids:
    content = client.beta.files.download(file_id)
    metadata = client.beta.files.retrieve_metadata(file_id)
    content.write_to_file(metadata.filename)
    print(f"Saved: {metadata.filename}")

The response content blocks have a nested structure: you're looking for bash_code_execution_tool_result blocks, which contain bash_code_execution_result objects, which contain items with file_id attributes. The files.download() call gives you the raw bytes; retrieve_metadata() gives you the original filename.

Why bash_code_execution? When you include the code_execution_20250825 tool, Claude actually gets two sub-tools: bash_code_execution (run shell commands) and text_editor_code_execution (create and edit files). To generate a file, Claude typically writes a Python script with the text editor sub-tool, then runs it via bash. The result block is named after whichever sub-tool produced the output -- and since it's the bash execution that creates the final file, that's the block type you parse. This is also why Claude has full bash access unlike the other vendors: it's not running Python in a restricted interpreter, it's executing real shell commands. The _20250825 tool version introduced this bash/text-editor split, replacing the earlier _20250522 version that was Python-only.

Uploading a CSV, Getting Back a Chart + PDF

To process your own data, upload via the Files API first, then attach the file to your prompt alongside the code execution tool.

import anthropic

client = anthropic.Anthropic()

# Upload your input file
uploaded = client.beta.files.upload(file=open("sales_data.csv", "rb"))

# Send the file + prompt with code execution
response = client.beta.messages.create(
    model="claude-sonnet-4-6",
    betas=["files-api-2025-04-14"],
    max_tokens=4096,
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Analyze this sales CSV. Create a bar chart of revenue by region "
                        "and save it as 'revenue_chart.png'. Also generate a one-page PDF "
                        "summary report of the key findings."
            },
            {"type": "container_upload", "file_id": uploaded.id},
        ],
    }],
    tools=[{"type": "code_execution_20250825", "name": "code_execution"}],
)

# Download all generated files
for block in response.content:
    if block.type == "bash_code_execution_tool_result":
        result = block.content
        if result.type == "bash_code_execution_result":
            for item in result.content:
                if hasattr(item, "file_id"):
                    content = client.beta.files.download(item.file_id)
                    metadata = client.beta.files.retrieve_metadata(item.file_id)
                    content.write_to_file(metadata.filename)
                    print(f"Downloaded: {metadata.filename}")

A single prompt can produce multiple files. In this case, you'll get both the PNG chart and the PDF report. Always iterate the full response -- never assume a single file.

Container Reuse: The Key to Iteration Workflows

Claude containers persist for 30 days. When your first request creates a container, the response includes a container.id. Pass it to subsequent calls and Claude picks up right where it left off -- all files from the previous request are still on disk.
# First call creates the container
response1 = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=4096,
    messages=[{"role": "user", "content": "Generate a sales report PDF."}],
    tools=[{"type": "code_execution_20250825", "name": "code_execution"}],
)
container_id = response1.container.id

# Subsequent calls reuse the same container
response2 = client.messages.create(
    container=container_id,
    model="claude-sonnet-4-6",
    max_tokens=4096,
    messages=[{"role": "user", "content": "Update the chart on page 2 to use a pie chart instead."}],
    tools=[{"type": "code_execution_20250825", "name": "code_execution"}],
)
This enables "conversational file editing" -- users can iterate on documents without re-uploading data or starting from scratch.

Pre-installed Libraries

Claude's sandbox comes with the document generation essentials: reportlab (PDFs), python-docx (Word), python-pptx (PowerPoint), openpyxl (Excel), pandas, matplotlib, pillow, pypdf, pdfplumber, seaborn, scipy, and scikit-learn. Since Claude has full bash access, you can also pip install anything else you need during the session.

OpenAI: Responses API + Code Interpreter

OpenAI's Responses API (the successor to the deprecated Assistants API) uses the Code Interpreter tool for file generation. The pattern is similar to Claude, but the response structure and file retrieval mechanism differ.

Generating a CSV with Code Interpreter

Enable the code_interpreter tool, then parse container_file_citation annotations from the response to find generated files.

from openai import OpenAI

client = OpenAI()

# Step 1: Request with code interpreter enabled
response = client.responses.create(
    model="gpt-5.2",
    tools=[{
        "type": "code_interpreter",
        "container": {"type": "auto"}
    }],
    input="Generate a CSV file named 'q1_report.csv' with 10 rows of financial data."
)

# Step 2: Extract file references from annotations
# The response structure nests deep: output → message → content → output_text → annotations
for item in response.output:
    if item.type == "message":
        for content_block in item.content:
            if content_block.type == "output_text":
                for annotation in content_block.annotations:
                    if annotation.type == "container_file_citation":
                        # Step 3: Download from the container endpoint
                        file_data = client.containers.files.content.retrieve(
                            file_id=annotation.file_id,
                            container_id=annotation.container_id
                        )
                        with open(annotation.filename, "wb") as f:
                            f.write(file_data.read())
                        print(f"Downloaded: {annotation.filename}")

The annotation traversal is the trickiest part. Don't try to shortcut it with response.output_text -- that gives you a plain string with citation markers, not the actual file references.

Uploading a File, Transforming It

Upload via the standard Files API, then pass the file ID in the container config.

from openai import OpenAI

client = OpenAI()

# Upload the file
uploaded = client.files.create(
    file=open("sales_data.csv", "rb"),
    purpose="user_data"
)

# Pass it to code interpreter via container config
response = client.responses.create(
    model="gpt-5.2",
    tools=[{
        "type": "code_interpreter",
        "container": {
            "type": "auto",
            "file_ids": [uploaded.id]
        }
    }],
    input="Analyze this sales CSV. Create a bar chart of revenue by region and save it as a PNG."
)

# Download generated files from annotations
for item in response.output:
    if item.type == "message":
        for content_block in item.content:
            if content_block.type == "output_text":
                for annotation in content_block.annotations:
                    if annotation.type == "container_file_citation":
                        file_data = client.containers.files.content.retrieve(
                            file_id=annotation.file_id,
                            container_id=annotation.container_id
                        )
                        with open(annotation.filename, "wb") as f:
                            f.write(file_data.read())
                        print(f"Downloaded: {annotation.filename}")

You can also request higher memory tiers -- 1g (default), 4g, 16g, or 64g -- by setting "memory_limit" in the container config. Useful when processing large datasets.

OpenAI Gotchas

The cfile_ 404 trap. Generated files have IDs prefixed with cfile_. If you try to download them using the standard client.files.content() endpoint, you'll get a 404. You must use client.containers.files.content.retrieve() instead. This has tripped up every developer at least once.

20-minute container expiry. OpenAI containers are ephemeral -- they expire after 20 minutes of inactivity. Download your files immediately after generation. There is no 30-day persistence like Claude.

Missing annotations fallback. There's a known edge case where container_file_citation annotations don't appear in the response. When this happens, check response.output for items of type code_interpreter_call and inspect their outputs for file references:
if not file_refs:
    for item in response.output:
        if item.type == "code_interpreter_call":
            for output_item in getattr(item, "outputs", []):
                if hasattr(output_item, "file_id"):
                    # Download using output_item.file_id and output_item.container_id
                    pass

Gemini: Inline Results + Structured Output

Gemini takes a fundamentally different approach. It doesn't return downloadable file artifacts with file IDs. Instead, code execution results come back inline -- matplotlib charts as raw image bytes, everything else as text or JSON.

This isn't a technical limitation -- Google has the infrastructure to build containers and file artifact systems. The gap is strategic. Google's file generation story lives in Google Workspace, not in the developer API:

Gemini in Docs generates full first drafts from prompts, matching writing styles and pulling data from Gmail, Drive, and the web.
Gemini in Sheets builds entire spreadsheets from natural language and auto-populates cells with live data.
Gemini in Slides generates themed slides, with full presentation generation from a single prompt on the roadmap.

This makes business sense for Google. Anthropic and OpenAI are API-first companies -- their revenue comes from developers using their APIs, so building sandboxes and file download endpoints directly serves their customers. Google's revenue comes from Workspace subscriptions. When Gemini generates a spreadsheet in Workspace, it creates a Google Sheet (not an .xlsx), keeping users in the Google ecosystem. An API that produces vendor-neutral files would undermine that.

The practical implication: Gemini's API-level file generation gap is unlikely to close anytime soon. The structured output and inline image patterns below are the right long-term approaches, not temporary workarounds.

For developers, this means Gemini is best suited for quick charts and data transforms, while complex document creation belongs with Claude or OpenAI.

Generating a Chart (Inline Image)

Enable the code_execution tool, then extract image bytes directly from the response parts.

from google import genai
from google.genai import types

client = genai.Client(api_key="YOUR_API_KEY")

response = client.models.generate_content(
    model="gemini-2.5-flash",
    config=types.GenerateContentConfig(
        tools=[types.Tool(code_execution=types.ToolCodeExecution)]
    ),
    contents="Generate a bar chart of quarterly revenue: Q1=$2.1M, Q2=$2.8M, Q3=$3.2M, Q4=$3.9M."
)

# Gemini returns results inline -- no separate download step
for part in response.candidates[0].content.parts:
    if part.executable_code:
        print("Code ran:", part.executable_code.code[:80], "...")
    if part.code_execution_result:
        print("Output:", part.code_execution_result.output)
    if part.as_image() is not None:
        with open("revenue_chart.png", "wb") as f:
            f.write(part.as_image().image_bytes)
        print("Chart saved as revenue_chart.png")

No file IDs, no download endpoints. The image bytes are right there in the response. For text/data output, it shows up in code_execution_result.output.

Structured Output for CSV Generation

Gemini's strongest file generation pattern is actually indirect: get structured JSON data back, then format it locally with whatever library you prefer.

import json
import pandas as pd
from google import genai
from google.genai import types

client = genai.Client(api_key="YOUR_API_KEY")

# Ask for structured JSON output
response = client.models.generate_content(
    model="gemini-2.5-flash",
    config=types.GenerateContentConfig(response_mime_type="application/json"),
    contents="Return a JSON array of 10 tech companies with fields: name, ticker, market_cap, sector."
)

# Convert to CSV locally -- you control the formatting
data = json.loads(response.text)
df = pd.DataFrame(data)
df.to_csv("tech_companies.csv", index=False)
print(f"Saved {len(df)} rows to tech_companies.csv")

This "structured output" approach gives you 100% control over formatting and is the most reliable way to produce files from Gemini. Let the model do what it's good at (data generation), and handle the file formatting yourself.

30-Second Execution Timeout

Gemini's code execution sandbox has a hard 30-second timeout. This makes it ideal for quick chart generation and data transforms, but rules it out for heavy document creation tasks like multi-page PDF reports or complex PowerPoint decks. For those, use Claude or OpenAI.

Which API for What?

Feature	Claude	OpenAI	Gemini
Sandbox Type	Reusable container (30-day expiry)	Ephemeral container (20-min idle timeout)	Stateless sandbox (30s timeout)
Resources	5 GiB disk, 5 GiB RAM, 1 CPU	Up to 64 GB RAM (tiered)	Token-limited (inline output)
Shell Access	Full bash	Python only	Python only
File Download	Files API (`files.download()`)	Container endpoint (`containers.files.content.retrieve()`)	Inline in response (no download step)
Best Use Case	Complex documents (PDF, DOCX, PPTX)	Heavy data processing + file gen	Quick charts and data transforms
`pip install`	Yes (bash access)	No (isolated sandbox)	No (isolated sandbox)

The short version:

Complex documents (PDF reports, slide decks, Word docs with formatting): Claude. The pre-installed document libraries and 30-day container persistence make it the best fit.
Large dataset processing (crunching big CSVs, Excel transformations): OpenAI. The ability to request up to 64 GB of RAM is unmatched.
Quick visualizations (charts, graphs, simple data summaries): Gemini. Inline image return means fewer API calls and faster turnaround.
Maximum formatting control: Any model's Structured Output mode. Get JSON data back, render locally with your own libraries.

The Self-Hosted Alternative: Run Your Own Sandbox

The three vendor APIs above all run code in their infrastructure. You send a prompt, they spin up a container, and they hand you back the file. This is convenient, but it means your data leaves your network, you're bound by each vendor's sandbox limits (30-second timeouts, no internet, fixed library sets), and you pay per-execution fees.

There's a fourth option: run the sandbox yourself. In this pattern, you call any LLM API to generate code (without enabling the vendor's code execution tool), then execute that code locally in an isolated environment on your own machines. You get the same prompt-to-file workflow, but you control the execution environment.

Why Self-Host?

Data residency. In regulated industries (healthcare, finance, government), sending code and data to a third-party sandbox may violate compliance requirements. A local sandbox keeps everything on your infrastructure.
No vendor sandbox limits. You choose the timeout, the RAM, the disk, the installed libraries. Need 10 minutes of execution time? A GPU? Network access to internal services? Your sandbox, your rules.
Cost at scale. Vendor sandbox pricing is per-session or per-hour. At high volume, running your own execution infrastructure can be significantly cheaper.
Model flexibility. Since you're decoupling "generate the code" from "run the code," you can use any LLM -- including open-source models, fine-tuned models, or your own -- to produce the Python script. The sandbox doesn't care where the code came from.

Tools for Building It

Two open-source projects have emerged as the leading options for sandboxed code execution:

E2B uses Firecracker microVMs (the same technology behind AWS Lambda) to isolate each execution in its own lightweight VM with a dedicated kernel -- stronger isolation than Docker containers. E2B offers a managed cloud service, but you can also self-host on your own GCP or Linux infrastructure using their Terraform-based deployment. The Python and JavaScript SDKs make it straightforward to spin up a sandbox, run code, and retrieve files programmatically.

exec-sandbox takes the fully-local approach. It runs untrusted code in ephemeral QEMU microVMs with hardware acceleration (KVM on Linux, HVF on macOS). No cloud dependency -- code never leaves your machine. Warm-pool latency is 1-2ms, and it supports Python, JavaScript, and shell execution. It's designed for air-gapped environments where sending code to any external service is a non-starter.

The Architecture Shift

The key difference is that self-hosting decouples code generation from code execution. With vendor APIs, the LLM both writes and runs the code in a single API call. With a self-hosted sandbox, you split these into two steps:

Call the LLM API for text/code generation (no code execution tool needed).
Extract the generated Python script from the response.
Execute it in your local sandbox (E2B, exec-sandbox, or even a locked-down Docker container).
Retrieve the output files from the sandbox filesystem.

Here's a concrete example using E2B as the sandbox and Anthropic as the LLM. Notice there's no code execution tool in the API call -- we just ask Claude to write a script, then run it ourselves:

import re
from anthropic import Anthropic
from e2b_code_interpreter import Sandbox

# Step 1: Ask the LLM to generate a Python script (no code execution tool)
client = Anthropic()
response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=4096,
    messages=[{
        "role": "user",
        "content": "Write a Python script that uses matplotlib to create a bar chart "
                   "of quarterly revenue (Q1=$2.1M, Q2=$2.8M, Q3=$3.2M, Q4=$3.9M) "
                   "and saves it as 'revenue_chart.png'. Return only the script, "
                   "no explanation."
    }]
)

# Step 2: Extract the Python code from the response
code = response.content[0].text
match = re.search(r"```

python\n(.*?)

```", code, re.DOTALL)
if match:
    code = match.group(1)

# Step 3: Execute it in an E2B sandbox
with Sandbox.create() as sbx:
    execution = sbx.run_code(code)

    if execution.error:
        print(f"Error: {execution.error.value}")
    else:
        # Step 4: Download the generated file from the sandbox
        file_content = sbx.files.read("/home/user/revenue_chart.png", format="bytes")
        with open("revenue_chart.png", "wb") as f:
            f.write(file_content)
        print("Saved: revenue_chart.png")

You can swap Anthropic for OpenAI, genai.Client, or any other LLM client -- the sandbox doesn't care where the code came from. You can also upload input files to the sandbox before execution using sbx.files.write(), mirroring the upload-then-process pattern from the vendor APIs.

E2B's default code-interpreter template comes with matplotlib, pandas, numpy, scikit-learn, pillow, openpyxl, python-docx, seaborn, and dozens of other common libraries pre-installed -- similar to the vendor sandboxes. If you need additional packages, you can either install them at runtime with sbx.commands.run("pip install <package>"), or build a custom template with your dependencies baked in so every sandbox starts ready to go.

This is more work to build, but it gives you full control over execution, security, and cost. It also means you can use Gemini or any other model that doesn't offer file artifacts -- you just need the model to write good Python, and your sandbox handles the rest.

Production Tips

If you're building file generation into a real product, a few hard-won lessons:

1. Sanitize filenames. The LLM chooses the filename based on the prompt. A creative user (or an adversarial one) can craft prompts that produce filenames with path traversal characters. Always strip or validate filenames before writing to disk. os.path.basename() is your friend.

2. Handle multi-file responses. A single prompt like "make a PDF report and an Excel spreadsheet of the raw data" can produce two or more files. Always iterate the full response -- never assume exactly one file comes back.

3. Persist container IDs for edit workflows. Claude's 30-day containers enable a powerful pattern: users can say "update the chart on page 2" in a follow-up message, and the LLM picks up the original file from the persistent container. Store the container_id alongside the conversation thread in your database.

4. Set timeouts generously. Code execution is significantly slower than text generation. Simple files might take 30-60 seconds; complex multi-file generation (especially PPTX with embedded charts) can take 5-15 minutes. Don't use your standard API timeout.

5. All sandboxes are offline. None of the three vendors allow network access from within the sandbox. All data must be uploaded or included in the prompt. You can't pip install on OpenAI or Gemini (Claude is the exception -- it has bash access). You can't fetch URLs. Plan accordingly.

Conclusion

File generation via LLM APIs follows a universal pattern across all three major vendors:

Claude excels at complex document creation with its 30-day persistent containers, full bash access, and pre-installed document libraries.
OpenAI offers the most compute headroom with up to 64 GB of RAM, making it ideal for heavy data processing tasks.
Gemini is the fastest path to charts and visualizations, returning inline image bytes with no separate download step.

Try it yourself: Build a CLI tool that takes a prompt and a desired output format, routes to the best vendor based on file type (PDFs to Claude, big data to OpenAI, charts to Gemini), and saves the result locally. You'll touch all three APIs and internalize the patterns in a single afternoon.

Official Documentation

Skills Required for Building AI Agents in 2026

Yaohua Chen — Wed, 25 Feb 2026 19:44:03 +0000

Why Agent Development Is Harder Than You Think

An Agent is conceptually simple: take the one-question-one-answer model of an LLM and add a loop. The model reasons about what to do next, calls external tools, feeds results back into itself, and repeats until the task is complete. A while loop plus tool-calling — that's the skeleton.

But between "working demo" and "production product" lies an engineering chasm. OAuth flows, tool design, error cascading across multi-step tasks, runaway costs, context window management, evaluation, multi-Agent coordination, model capability bottlenecks, and framework trade-offs — these nine challenges are where Agent development actually gets hard. API calls account for roughly 5% of the total effort; the other 95% is everything else.

For a detailed walkthrough of each challenge, see the companion piece: Is AI Agent Development Just About Calling APIs?

The question this post addresses is different: given that Agent development is hard, what skills do you actually need to succeed at it in 2026?

The Skill Shift: From Writing Code to Shaping Problems

Inspired by a Story: How an Intern Outperformed a Senior Engineer?

Shubham Saboo — Senior AI Product Manager at Google Cloud, founder of Unwind AI, and co-author of Google's Introduction to Agents whitepaper — recently shared an experience from a startup where he serves as an advisor. Something happened that overturned everyone's assumptions.

A senior engineer received a task and followed the traditional workflow: understand requirements, design architecture, write code, debug, and test. Three days later, he delivered a technically flawless solution -- clean code, clear logic, fully compliant with engineering standards.

An intern completed the same task in a single afternoon.

It wasn't that the intern had superior technical skills. Quite the opposite -- his coding experience was far less than the senior engineer's. But he did something fundamentally different: he defined the problem clearly enough, then let Claude Code do the rest.

This scenario reveals a harsh reality: when AI can complete implementation-level work quickly and accurately, the bottleneck shifts entirely upstream. The value is no longer "Can you write this code?" but rather "Can you decompose the problem to a level where AI almost never makes mistakes?"

An even more striking example comes from inside Anthropic. They had Opus 4.6 build a C compiler using a team of Agents, then essentially stepped back. Two weeks later, it could run on the Linux kernel -- 100,000 lines of working Rust code, without a single line written by a human.

The researcher leading this project, Nicholas Carlini — a research scientist at Anthropic known for his work on adversarial machine learning — did only one thing: problem decomposition. He broke down the vague goal of "build a compiler" into 16 precisely defined subtasks, each with clear inputs, outputs, and success criteria. Then 16 Agents, each handling its own piece, completed the entire compiler.

The real leverage isn't in writing code -- it's in breaking problems down to the point where AI almost never gets it wrong.

Four Skills That Are No Longer Differentiating

Shubham argues that four capabilities that once commanded high salaries for developers are rapidly losing their power as differentiators — not because they're useless, but because AI has made them table stakes:

Writing code from scratch. Agents write faster and produce fewer bugs. The ability to hand-write code still matters as foundational knowledge, but it's no longer what sets great developers apart.
Boilerplate code and project scaffolding. A single prompt generates them instantly.
Memorizing syntax and APIs. Extended context windows have already solved this problem.
Translating specifications into code. Now, the specification itself is the code.

These skills were once valuable because implementation itself was hard. They required years of training and justified six-figure salaries. But implementation is no longer the bottleneck — it's becoming the easy part.

Yet the entire industry is still optimizing around the old bottleneck. Most companies' job descriptions still emphasize "proficient in Java," "familiar with Spring framework," "5+ years of development experience." These criteria are losing relevance at a visible pace.

Value has migrated to five new skills.

The Five Skills That Truly Matter in 2026

I am tryiing to answer this question. This isn't theoretical speculation -- it's what I has witnessed firsthand when developing AI solutions in the past 2 years, in the open-source community, and through countless experiences building Agents.

1. Problem Shaping

Turning vague goals into executable tasks -- this skill separates people who "play around with AI" from those who actually build products with it.

"Build me a dashboard" is not a task; it's a wish. Problem shaping breaks it into twelve specific, testable subtasks: What data does this dashboard display? What decisions does it support? What must the user understand within the first three seconds? Each sub-problem has clear inputs, clear outputs, and clear success criteria.

When you decompose a vague goal into precise sub-problems, the Agent's execution quality transforms entirely. It no longer needs to guess your intent -- it just follows clear instructions.

How to practice problem shaping:

Start with the desired output and work backwards — what does "done" look like?
For each subtask, define three things: the input it receives, the output it produces, and how you'll know it succeeded.
If a subtask is still ambiguous enough that two people would interpret it differently, break it down further.
Verify your decomposition by asking: could a competent person with zero context about this project execute each subtask from the description alone?

2. Context Design

Agent output quality is directly proportional to the quality of context you provide.

Poor context: "Build me a customer support agent."

Good context: "The target users are SaaS customers considering canceling their subscriptions who have already tried the help documentation but failed. The tone should be empathetic yet efficient -- avoid excessive apologies and robotic responses. Here are 3 real cases that received five-star ratings and 2 cases that received complaints. Edge cases requiring human escalation include: billing disputes over $500, account security issues, and legal compliance matters. The success metric is resolving the issue within 4 messages without escalation."

The difference isn't in prompt engineering tricks. It's in information density, boundary conditions, success criteria, and understanding of real-world scenarios.

A context design checklist:

Who is the target user, and what is their state of mind?
What does the desired tone sound like? Provide 2–3 real examples, not adjectives.
What are the edge cases that require special handling or human escalation?
What does success look like, in measurable terms?
What are the most common failure modes, and how should the Agent handle them?

3. Aesthetic Judgment

When ten options are in front of you, knowing that nine of them won't work.

Shubham recently had Antigravity build a bargaining simulator for his repository: two Agents negotiating a used car deal, each with a distinct personality, live-streaming the entire process. The first version ran perfectly -- clean code, no errors, both sides going back and forth. Technically complete.

He rejected it in thirty seconds.

The interface was just a plain chat window. The negotiation process read like a log file -- no personality tension, no emotional highs and lows, no dramatic moments of "Shark Steve holding the line against Cool-Hand Casey pretending to walk away." It worked as software; it failed as an experience.

An Agent can build anything you describe, but it cannot judge what is worth describing. Agents optimize for correctness; humans optimize for "Would anyone actually want to use this?"

4. Agent Orchestration

Knowing when to use one Agent, when to use multiple, when to run them in parallel, when to run them sequentially, when to add guardrails, and when to let go.

Three core patterns:

Sequential pipeline: Agent A completes its task and passes the output to Agent B. Best for scenarios with dependencies between steps.
Coordinator + specialist team: A lead Agent dispatches tasks and integrates results. Best for complex tasks requiring quality control.
Parallel execution + merge: Multiple Agents handle independent tasks simultaneously, with results consolidated at the end. Best for scenarios with no dependencies between subtasks.

Most people default to sequential workflows because they feel "safer." But knowing when to parallelize and when to introduce a coordinator determines whether your workflow finishes in five minutes or drags on for an hour.

A practical rule of thumb: If two subtasks don't share state — neither reads what the other writes — they can run in parallel. If one subtask's output determines what the next subtask even is, they must be sequential. And if you have more than three parallel Agents whose outputs need to be merged, introduce a coordinator to avoid contradictory results.

5. Knowing When NOT to Use an Agent

Not every problem needs an Agent.

Need to reformat JSON? Hand it to Gemini 3 Flash -- done in ten seconds.
Text replacement across ten files? A lightweight model handles it in seconds.
A bug you already fully understand? Fixing it yourself is faster than explaining it to an Agent.

True capability is matching the right tool to the problem. Complex problems get Agents. Simple problems get models. Obvious problems get your keyboard.

Conway's Law Restructured in the Age of AI

In the classic book The Mythical Man-Month, Fred Brooks proposed a famous insight: a software system's architecture will inevitably mirror the communication structure of the organization that built it. This became known as Conway's Law.

Building AI agents is essentially restructuring Conway's Law with AI.

In traditional software development, the speed of delivering a feature depends on team size, communication efficiency, and technical debt. You need frontend engineers, backend engineers, QA engineers, countless meetings to align requirements, and long develop-test-fix cycles.

In the Agent era, this chain is compressed. One person plus 16 Agents can build a compiler in two weeks. One intern plus Claude Code can accomplish in an afternoon what took a senior engineer three days.

Organizational structure is no longer the bottleneck. The quality of problem definition is.

This is why Shubham says the best developers of 2026 look more like film directors than programmers. They set the scene, cast the actors, and know when to call "cut." They don't write every line of dialogue -- they shape the entire production.

The essence of programming is shifting from "writing" to "orchestrating."

Three Limitations You Must Know

Although Agents sound like magic, you must be aware of three limitations when applying them in practice.

1. Agent quality is highly dependent on problem definition. If you cannot decompose the problem clearly enough, the Agent will consistently produce outputs in the wrong direction. This isn't the Agent's fault -- it's a problem-shaping problem. Before you master this skill, Agents may actually slow you down.

2. Context design requires deep business understanding. Writing a good CLAUDE.md or .cursor/rules file requires you to truly understand the product's worldview, users' pain points, and success criteria. This understanding cannot be rushed -- it can only be accumulated through repeated shipping and observing real user behavior.

3. Aesthetic judgment cannot be learned from books. It comes from repeated shipping, observing real user behavior, and developing sensitivity to the gap between "it works" and "it's worth using." Without this accumulation, Agents will help you rapidly produce a large volume of things that are "technically correct but experientially failed."

State Management: Problem Shaping Applied to Execution

All five skills above come into sharpest focus in one practical engineering challenge: state management. An Agent that can plan is worthless if it can’t track its own progress. Without a progress-tracking mechanism, Agents fall into "hallucination loops" — repeating steps, losing track of the original goal, or confidently declaring a task complete when it’s half-done.

This is where all five skills converge — applied not to a product or a user-facing feature, but to the Agent itself. Each of the four patterns below draws on a different combination of skills:

1. The "Plan-Act-Observe" Loop (ReAct pattern). (Skill #1 Problem Shaping + Skill #2 Context Design) Instead of handing the Agent a giant task list, force it to update its internal state after every single action. The Agent explains what it intends to do (Thought), calls a tool (Action), receives the raw result (Observation), then compares that result against the original plan (Status Update). The loop itself is problem shaping — breaking execution into atomic Thought→Action→Observation cycles. The status update after each cycle is context design — ensuring the Agent's next decision is informed by accurate, structured state rather than stale memory.

2. Dynamic Task Graphs. (Skill #1 Problem Shaping + Skill #4 Agent Orchestration) For complex workflows, static to-do lists break down. Use a directed acyclic graph (DAG) or dynamic task queue where each task carries a status (PENDING, IN_PROGRESS, COMPLETED, FAILED), dependencies are tracked explicitly (Task B doesn’t start until Task A succeeds), and intermediate variables are stored in a scratchpad — like a URL found in Step 1 that’s needed in Step 5. Defining each node with clear inputs, outputs, and success criteria is problem shaping. Deciding which nodes run in parallel versus sequentially, and how results flow between them, is agent orchestration.

3. The Critic Node. (Skill #3 Aesthetic Judgment + Skill #4 Agent Orchestration) In multi-Agent architectures, it helps to have a supervisor that reviews outputs rather than just trusting the worker’s self-assessment. The Worker executes and reports "I’m done." The Critic checks whether the goal was actually achieved. A shared Global State stores the current version of truth. This is the Coordinator pattern from Skill #4 applied to quality control — but the Critic’s evaluation criteria come from Skill #3: knowing when output is "technically correct" but not actually good enough. Without aesthetic judgment baked into the Critic’s rubric, it degrades into a syntax checker.

4. Checkpointing and Self-Correction. (Skill #1 Problem Shaping + Skill #5 Knowing When NOT to Use an Agent) Progress tracking isn’t just about moving forward — it’s about knowing when to turn back. If an observation returns an error, the Agent should update the plan rather than crash — that’s problem shaping in real time, re-decomposing the remaining work based on new information. And if an Agent is 50 steps deep into what should be a 5-step task, it’s "lost in the woods" and needs a reset. Budget monitoring (tokens, turns, or wall-clock time) prevents runaway execution. Recognizing when to abort an Agent run and switch to a simpler tool — or fix the issue manually — is Skill #5 in action.

A practical implementation tip: (Skill #2 Context Design) Prepend a status summary to every LLM call — original goal, completed steps, current step, remaining steps. This is context design at its most literal: engineering the information the Agent sees at every turn. This "external state" acts as a rhythmic beat that keeps the context window focused on the finish line, counteracting the "Agentic Amnesia" problem described in the companion piece.

Putting It Into Practice

I close with a poignant statement: "These skills cannot be acquired through reading. They come from practice."

I offer five concrete exercises:

Review your last five Agent outputs. Write down what you would change and why.
Write a CLAUDE.md for your current project -- even if it only takes 30 minutes.
The next time you face a vague requirement, break it into 10 subtasks before writing a prompt.
Take a sequential workflow and identify which steps can run in parallel.
For one week, log every task where you used an Agent but a simple prompt would have sufficed.

Open your most recent project and ask yourself: Are you spending more time writing code, or shaping problems?

Conclusion

The ten engineering challenges of building AI agents haven't gone away. But the response to them has fundamentally shifted.

Twenty years ago, the scarce resource was implementation skill — the ability to translate an idea into working code. That scarcity justified years of training, specialized hiring, and the entire structure of software teams. Today, Agents handle implementation at speed and quality that rivals senior engineers. The scarce resource has moved upstream: the ability to decompose problems precisely, design rich context, exercise aesthetic judgment, orchestrate multi-Agent workflows, and know when to reach for a simpler tool.

This isn't a prediction about the future. It's a description of what's already happening — an intern shipping in an afternoon, a compiler built without a human writing a single line of code, organizations discovering that their bottleneck is problem definition, not programming talent.

The developers who thrive in this era won't be the ones who write the most code. They'll be the ones who ask the best questions, shape the clearest problems, and know when the Agent's output is good enough — and when it isn't.

The skills have shifted. The question is whether you'll shift with them.

References

Berkeley Function-Calling Leaderboard — Tool-calling accuracy benchmarks across models (~77.5% top accuracy). berkeley-function-call-leaderboard
Galileo Research — Findings on error cascading in multi-step Agent tasks. galileo.ai
LangChain State of AI Agents Report — Survey data on Agent evaluation practices (52% offline evaluation, 37% online evaluation). blog.langchain.dev
UC Berkeley MAST Framework — Analysis of 1,600+ Agent traces showing 41–86.7% multi-Agent failure rates, with 79% of failures from orchestration. arxiv.org
Microsoft Azure SRE Case Study — Production experience scaling from 50+ sub-Agents to 5 core tools. techcommunity.microsoft.com
Anthropic Agent Evaluation Blog (January 2025) — Challenges in systematically evaluating Agent behavior. anthropic.com/research
Nicholas Carlini — C Compiler with Opus — Building a C compiler with 16 Agents producing 100,000 lines of Rust. nicholas.carlini.com
Shubham Saboo / Unwind AI — theunwindai.com
Boston Consulting Group — Research showing fewer than 20% of enterprise Agent projects achieve expected ROI. bcg.com
Alibaba Cloud Engineering Blog — Data showing AI completes 30% of work in production Agent systems, with 70% being tool engineering. alibabacloud.com/blog
Spotify Engineering — Experience with context window limits in code Agent development. engineering.atspotify.com
Manus Team — Four framework rebuilds for context engineering. manus.im
Fred Brooks, The Mythical Man-Month — Origin of Conway's Law and organizational structure insights. wikipedia.org

Is AI Agent Development Just About Calling APIs? Where's the Real Difficulty?

Yaohua Chen — Wed, 25 Feb 2026 19:40:02 +0000

The Bottom Line First

Calling APIs is indeed the entirety of Agent development — just like cooking is indeed putting ingredients in a pot. Technically correct, but it perfectly explains why some people produce Michelin-star dishes while others produce culinary disasters.

Saying the conclusion without explanation is meaningless. Let's actually build an Agent and walk through it together. But before diving in, let's take 30 seconds to clarify what an Agent actually is.

What Is an Agent, Exactly?

The original interaction model with large language models (LLMs) was simple: you ask a question, it gives an answer. One question, one answer, done. If you wanted it to do something complex, you had to manually break tasks into small pieces and feed them one round at a time. You were the "orchestrator"; the LLM was just a passive tool that responded on demand.

What an Agent does is fundamentally one thing: it adds a loop to this question-and-answer model. The model no longer just answers you once. Instead, it judges "what else do I need to do," calls external tools to get results, feeds those results back to itself, thinks about what to do next, and repeats until the task is complete. This loop transforms a large model from a "responder" into an "executor."

Agent Execution Loop:

User Input → LLM Reasoning → Need to call a tool?
                                      │
                    ┌─── Yes ─────────┘─── No ───┐
                    ▼                             ▼
           Select Appropriate Tool         Task Complete?
                    │                             │
                    ▼                        Yes  ▼
           Call External Tool          Return Final Result
         ┌──────────────────┐
         │  Check Emails    │
         │  Check Calendar  │
         │  Create Meeting  │
         └──────────────────┘
                    │
                    ▼
         Get Tool Return Results
                    │
                    ▼
           Update Context ──────────────────────────────┐
                                                        │
                                              (loop continues)

Conceptually, it's that simple. A while loop plus tool-calling capability — that's your Agent skeleton. So many people read this and think, "There's no real technical depth here?" True, the skeleton is simple. But making that loop run stably, reliably, and efficiently in the real world — that is the real engineering challenge.

Let's walk through it for real. Say you want to build an Agent that manages your schedule: read emails, check calendars, arrange meetings. Doesn't sound complicated, right? Let's look at what you encounter at each step.

Step 1: Call the API — Done in 10 Minutes

This step really is easy. Install an SDK, write a few lines of code, pass user input to the model, get back a result. If you've used the OpenAI or Claude API, you could write it blindfolded. You don't even need to write code yourself — open an AI coding tool like Claude Code or Cursor, describe your requirements in natural language, and they'll scaffold the project for you. Define a few tools — check calendar, read emails, create meeting — write the JSON schema, and the model can call them.

It runs. You ask it "what meetings do I have tomorrow?", it calls the calendar tool, gets the result, and reads it back in natural language. Perfect. You think: Agent development isn't that hard, maybe I can ship this in a week.

I've had this feeling before. 20 years ago when I first learned C# development, I dragged a few controls onto a Windows Form and had a running App — I thought Windows Form development was no big deal either.

In theory, those AI coding Agents could handle every step ahead for you too. But in practice, every problem you encounter from here on isn't about how to write the code — it's about what code should be written. To really understand where Agent development gets hard, let's keep walking.

Step 2: Connect to Real APIs — The Nightmare Begins

In the demo you used mock data. Now you need to connect to real email and calendar services. Each user might use something different: Outlook, Gmail, hotmail, etc. Let's simplify and just connect to Microsoft's Graph API — it's accessible domestically and Outlook is mainstream in enterprise.

The first problem arrives immediately: OAuth. Your users must authorize your application to access their Microsoft account. You need to register an app in Azure AD, handle OAuth redirects, securely store refresh tokens, and auto-refresh when tokens expire. None of this has anything to do with the LLM, but without it, your Agent can't take its first step. Microsoft's permissions model alone (delegated permissions vs. application permissions) can eat half a day of research.

Then come the API edge cases. Microsoft Graph returns email lists paginated — 10 items per page by default, up to 50. Your Agent gets the first page without knowing how many more pages exist, and it will give you a confident-sounding conclusion based on just those 10 emails. Ask "did anyone email me last week about Project A?" — the actual email is on page 3, but the Agent confidently tells you "no." You can add a tool to check the next page, but then you need to add a tool to check the next page, and so on.

Rate limiting is another problem. Microsoft Graph's throttling strategy is complex, with different thresholds per app, per user, and per resource type. If your Agent calls it a dozen times in a complex task, it will easily hit a 429 error. What happens then? The model doesn't know what "429 Too Many Requests" means — it just thinks the tool call failed and starts guessing reasons. And this is only for one provider. To build a real product, every provider (Gmail, hotmail, etc.) has its own authentication system and API design. The workload multiplies.

The Tool Design Problem: Connecting the API is only half of the tool-call equation. The other half is how to design the tool itself — and this is trickier than you'd expect.

What should your "search emails" tool look like? If it's too rigid — only supporting sender-based queries — a user saying "find last week's emails about Project A" will fail immediately. So you add keyword search, time range filtering, attachment filtering? The more parameters, the more complex the schema, and the more likely the model is to fill things in wrong or miss fields. Berkeley's Function-Calling benchmark found that the more tools and the more complex the parameters, the worse model accuracy becomes. Smaller models degrade dramatically as tool count grows — BFCL data shows that models like Llama 3.1 8B can handle a modest number of tools but start failing unpredictably once tool count exceeds their capacity threshold.

On the other end, if you design a generic "search" tool that covers everything, the model won't know what to put in it. It might pass calendar query parameters to the email search tool, or call "send email" when it should "create a meeting." There's no right answer for tool granularity — too fine and user needs aren't covered, too coarse and the model can't handle it. The only way is to iterate in your specific context.

Tool description text matters enormously. For the same functionality, a description written as "Search emails" vs. "Search the user's Outlook inbox by keyword, sender, date range, or attachment presence. Returns a list of matching emails sorted by date" produces dramatically different model accuracy. In short, you don't just need to write code to implement a tool — you need to learn to write a manual for the model, and whether that manual is good or bad, you can only verify through repeated testing.

A lot of research puts it clearly with data: in production-grade Agent systems, AI completes only 30% of the work, and the remaining 70% is tool engineering. What you think of as "calling an API" is mostly spent on the design and integration work surrounding that API.

Step 3: Multi-step Tasks — Errors Start Snowballing

Good — the API is connected and basically working. Now try a slightly more complex request: "Find a time slot next week when everyone is free, schedule a project review meeting, and then email all attendees."

This task requires: querying multiple people's calendar availability, finding the intersection, creating a meeting invite, drafting an email, and sending it. Five or six steps, each depending on the previous one's result.

Here's the problem. Berkeley's Function-Calling Leaderboard (BFCL) shows that even the best models struggle with tool-calling accuracy — top scores hover around 80% on overall benchmarks, and accuracy drops further as tool count and parameter complexity increase. That means roughly 1 in 5 calls has an error. The probability of a five-step task completing entirely correctly? About 0.8 to the fifth power — less than 33%. Your Agent has roughly a two-thirds chance of going wrong at some step.

Worse, Galileo's research found that early small errors amplify through later steps. Say the model misparses a date format in step one and reads Tuesday as Wednesday. Every subsequent step builds on that error. It creates a meeting at the wrong time, then sends everyone an email notification with the wrong time. One small hallucination triggers a cascade of wrong actions.

At this point you realize: you need to add validation logic between each step, rollback mechanisms, and confirmation loops. None of this is taught in any LLM's API documentation.

Step 4: Guardrails — The Invisible Security Risk

And there's a deeper problem lurking here that most people don't think about until it's too late: guardrails. Your scheduling Agent has permissions to send emails, create meetings, and modify calendars. What happens when it hallucinates a participant name and sends a meeting invite to the wrong person? Or confidently deletes a calendar block because it "optimized" your schedule?

OWASP classifies this as "Excessive Agency" (LLM06:2025) — one of the top security threats in LLM applications. It breaks down into three failure modes: excessive functionality (your Agent has access to 50 actions when it only needs 5), excessive permissions (your Agent can modify any calendar, not just the user's), and excessive autonomy (the Agent sends emails and creates meetings without any human confirmation gate).

In practice, you need to separate "read" tools from "write" tools and put explicit approval gates on write operations. High-stakes actions — sending external emails, deleting calendar entries, modifying shared resources — should run in a "dry run" mode where the Agent describes what it would do and waits for human confirmation before executing. You need to design for rapid rollback, because the question isn't if your Agent will take a wrong action — it's when. And you need to enforce the principle of least privilege: your Agent should request only the minimum API permissions it needs, not broad access "just in case."

None of this is glamorous engineering. But skip it, and one hallucinated email from your Agent can undo months of user trust.

Step 5: Open It to Real Users — The Bill Scares You Awake

You tested the first three steps in your development environment and things seemed fine. But once you open the Agent to real users, the nightmare comes from a direction you never anticipated: the bill.

You used Claude Sonnet or GPT-4o for development testing — great results, a few cents per complex task, no pain. But with real users, hundreds of requests per day, each averaging four or five tool call rounds, each carrying substantial context — you look at the monthly bill and see a small feature burning thousands of dollars a month. What if user volume grows ten times?

You think: a user saying "what meetings do I have tomorrow?" — does that really need the most powerful model? That's overkill.

So you start thinking about model routing: different tasks use different base models. Simple queries go to cheap small models (Haiku, GPT-4o mini, Gemini Flash); complex multi-step reasoning goes to large models (Claude Sonnet, GPT-4o, Gemini Pro). But who judges complexity?

Use a large model to judge? That costs money too.
Use a rule engine? Works for simple cases, but user inputs are endlessly variable and rules always have gaps.
Use a small model as a classifier? Now you've added another model component that needs tuning and maintenance.

And different models vary enormously in their tool-calling capabilities. A tool schema that works on Claude Sonnet may have parameters filled in wrong on Haiku. JSON that runs perfectly on GPT-4o may fail to parse on open-source models. Every time you swap a model, your carefully tuned prompts and tool descriptions may need to be re-adapted. This is why many teams eventually find that the token money saved doesn't cover the labor cost of multi-model adaptation.

To put concrete numbers on this: Claude Sonnet costs \$3/\$15 per million input/output tokens, while Claude Haiku costs \$0.25/\$1.25 — a 12x to 60x difference. GPT-4o vs. GPT-4o mini has a similar spread. Mid-sized Agent deployments easily burn \$1K–\$5K per month in token costs alone; complex Agents consuming 5–10 million tokens monthly aren't unusual. One underrated optimization: prompt caching. Anthropic's prefix caching can reduce costs by up to 90% and latency by 85% for repeated long prompts — a massive win for Agents that include the same system prompt and tool definitions in every call.

And cost isn't the only scaling problem — latency hits you just as hard. A multi-step scheduling task that checks four people's calendars, finds a common slot, creates a meeting, and sends emails can easily take 30–45 seconds end-to-end. Technically correct, but your users experience it as broken. The biggest UX win is streaming intermediate results: instead of a 45-second black box, show "Checking Alice's calendar... Found 3 available slots... Confirming with Bob..." — the total time is the same, but the perceived wait drops dramatically. Parallelizing independent tool calls (check all four calendars simultaneously instead of sequentially) helps with actual latency. But the hard tradeoff remains: smaller, faster models hallucinate more, so you can't just throw Haiku at everything to speed things up.

Cost optimization looks like an operations problem, but it's actually an architecture problem. You need to make the model-calling layer pluggable from the very beginning — something most people never think about when writing a demo.

Step 6: Context Management — Your Agent Starts "Forgetting"

After a while, you notice a strange problem: the Agent "drifts" during long tasks. You give it a complex task requiring seven or eight conversation rounds, and by rounds four or five, it starts forgetting the original requirements and constraints.

This is what the industry calls "Agentic Amnesia." Research data is clear: when tasks are split across multiple conversation rounds, model performance degrades significantly — and without memory management strategies, Agents lose track of constraints, requirements, and earlier results as context accumulates.

The reason is that LLM context windows are finite. Every tool call's input and output consumes context space. Query five people's calendars, each returning a large JSON payload, and the context window is mostly full. Spotify's engineering team hit the exact same pitfall building a code Agent: once the context window filled up, the Agent "lost its direction" and forgot the original task after a few rounds.

You need to start doing Context Engineering. Anthropic defines it as "curating exactly what content goes into a limited context window from an ever-changing universe of information." In plain terms, it's the LLM version of memory management: you dynamically decide what the model "sees" at each reasoning step and what it "forgets." Which historical information gets compressed into summaries? Which key constraints must always be preserved? Which tool return values can be discarded?

The Manus team rebuilt their entire framework four times to get this right. Four times. They called this process "stochastic gradient descent" — inelegant, but effective.

There's also a subtler trap: research shows context length and hallucination rate are positively correlated. The longer the input, the more likely the model is to hallucinate. For Agent tasks that require large contexts, this is nearly an unresolvable structural paradox.

One emerging solution to this problem is Agent Skills, a mechanism pioneered by Anthropic. Where Context Engineering is about managing what's in the context window, Skills are about not putting things there in the first place. A Skill is a modular package of instructions, workflows, and best practices (typically a SKILL.md file plus optional scripts) that an Agent loads on demand. Think of it as pluggable expertise — a "Tax Compliance Skill" or a "Cloud Migration Skill" that transforms a general-purpose Agent into a domain specialist, without bloating the context window for every other task.

The design uses progressive disclosure: an Agent can have dozens of Skills installed but only loads the 2–3 it needs for any given task. This directly mitigates the context window pressure that causes Agentic Amnesia. Skills also enable composability — combining a code-review Skill with a git-automation Skill produces an Agent that can review and commit code without anyone writing explicit coordination logic.

The impact on the ecosystem has been rapid. OpenAI adopted structurally identical Skills for ChatGPT and Codex CLI. Microsoft's Semantic Kernel implements an equivalent "Plugins" abstraction. Marketplaces like SkillsMP have emerged with hundreds of thousands of community-built Skills. Anthropic has positioned Agent Skills as an open standard — and the convergence across platforms suggests it's becoming the standard abstraction for packaging Agent capabilities, much like MCP became the standard for Agent-to-tool communication.

Step 7: Want to Test It? You Don't Even Know How

At this point, your Agent barely works. But how do you determine whether it's "actually good" vs. "just barely functional"?

Traditional software development has mature testing methodologies: unit tests, integration tests, end-to-end tests — inputs are deterministic, expected outputs are deterministic. But an Agent's input space is open-ended (users can say anything) and its output is non-deterministic (the model generates different text each time). LangChain's blog put it perfectly: "every input is an edge case" — a challenge traditional software has never faced.

You might think to use LLM-as-judge to evaluate LLM outputs. A Hacker News developer explained the problem clearly: using a judge with the same architecture as the system being tested maximizes the probability of systematic bias. The judge and the tested Agent share exactly the same blind spots.

Anthropic's January blog also acknowledged: Agent interactions involving tool calls, state modifications, and behavior adjustments based on intermediate results are precisely the capabilities that make Agents useful — and simultaneously make them almost impossible to evaluate systematically.

The data is stark. LangChain's State of AI Agents survey (1,300+ professionals, 2025) found only about half of organizations run offline evaluations, and fewer than a quarter combine both offline and online evaluations. A multi-dimensional analysis of major Agent benchmarks found a 37% performance gap between lab testing and production environments — with reliability dropping from 60% to 25% in real-world conditions. An Agent that tests great in your dev environment may behave completely differently in users' hands.

Anyone who's done client-side development will understand this pain: your Agent might handle a request perfectly today, and fail on the same request tomorrow. Users can accept missing features — they can't accept inconsistency.

And evaluation is only half the story — the other half is observability in production. Evaluation tests what you expect the Agent to do; observability shows what it actually does with real users. When a user reports "the Agent scheduled my meeting at the wrong time," you need to trace back through every tool call: what calendar data was retrieved, what the LLM reasoned, what meeting parameters were generated, and why the wrong time was selected. Without tool call tracing, latency monitoring, and cost/token budget tracking, you're debugging blind. That "37% performance gap" between lab and production? Observability is how you find it. Tools like LangSmith and Arize have emerged specifically for this, but many teams still discover production failures only when users complain.

Step 8: Add Multi-Agent Collaboration? Complexity Explodes

Your scheduling Agent is working well, and you start thinking: could you add more specialized Agents? One for email, one for calendar, one for meeting notes, one for scheduling coordination. Clear division of labor, each handling its domain — sounds reasonable, right?

Microsoft's Azure SRE team went down this path. They initially built a massive system with 100+ tools and 50+ sub-Agents, and hit a pile of unexpected problems: the orchestrator Agent couldn't find the right sub-Agent (the correct one was "buried three hops away"); a buggy sub-Agent didn't just crash itself — it dragged down the entire reasoning chain; Agents kicked responsibility back and forth in infinite loops. They eventually scaled down to 5 core tools and a few general-purpose Agents, and the system became more reliable.

Their core lesson: scaling from one Agent to five doesn't multiply complexity by four — it grows exponentially. UC Berkeley's MAST framework analyzed 1,600+ Agent traces and found that 41–86.7% of multi-Agent systems fail in production, and 79% of problems come from the orchestration and coordination layer, not the technical implementation. How to divide work and how to communicate between Agents is far harder than how to write the code.

There are established orchestration patterns — sequential chains, concurrent fan-out, hierarchical supervisor models — and each has tradeoffs. ICLR 2025 research found that hierarchical architectures (one coordinator delegating to specialists) show only a 5.5% performance drop when individual Agents malfunction, compared to 10.5–23.7% for flatter architectures. This explains why Microsoft eventually simplified to a supervisor model. The practical advice is almost counterintuitive: start with fewer, more capable Agents rather than many specialized ones, and only decompose when a single Agent demonstrably can't handle the workload. The allure of clean role separation is strong, but the coordination overhead will eat you alive.

Step 9: You Start Doubting — Where's the Bottleneck?

After months of work, your engineering gets more refined, but Agent performance always hits a ceiling you can't break through. You realize a harsh truth: all engineering optimization has one prerequisite — the underlying model needs to be capable enough.

An InfoQ interview with Alibaba Cloud's code platform lead captured it honestly: engineering challenges can be overcome, but model capability bottlenecks are far more daunting. An awkward industry reality: nearly every company building general-purpose Agent products uses Claude Sonnet as their first-choice model, because other models lag noticeably on instruction-following in complex tasks. The more instructions a model can follow, the more complex the problems it can handle. When a model can't even do basic instruction-following, no amount of engineering optimization above it helps.

You might think: what about using more powerful reasoning models — o3, o4-mini, DeepSeek R1, Claude Sonnet, Claude Opus? Research finds that reasoning models hallucinate more than base models. The data is striking: OpenAI's o3 has a 33% hallucination rate on person-specific factual questions — double the rate of its predecessor o1. The o4-mini reasoning model hits 48%. The root cause is that RL fine-tuning for chain-of-thought reasoning introduces high-variance gradients and entropy-induced randomness, making models more confident even when wrong. They answer rather than admit uncertainty.

The practical implication for Agents: reasoning models may handle complex task decomposition better, but they trade off reliability on factual tasks. One emerging pattern is to use reasoning models for planning (breaking down what needs to happen) and base models for execution and verification (actually doing it and checking the results). But this adds yet another layer of architectural complexity.

It's like finding your app is laggy, spending days optimizing code logic, and then discovering the bottleneck is hardware performance. Your engineering optimizations have limits, and beyond those limits lies the constraints of underlying capability.

Step 10: You Start Understanding the Framework Wars

At this point, you've definitely wrestled with whether to use LangChain, CrewAI, or similar frameworks. The Hacker News discussion has moved from debate to consensus: frameworks are useful for prototyping; in production they often become a burden.

A CTO shared on Hacker News that he built hundreds of Agents without any framework, using only chat completions plus structured output.

Anthropic's official guidelines also advise caution with frameworks, as they often make underlying prompts and responses opaque and harder to debug.

Here's the practical landscape: LangGraph (by LangChain) uses a graph-based architecture with nodes, edges, and conditional routing — it's powerful for complex multi-step reasoning and is used in production by 400+ companies. CrewAI takes a role-based approach where you define Agents by organizational roles — simpler to set up, adopted by 60% of the Fortune 500 for content generation and analysis workflows. AutoGen (Microsoft) was merged into the Microsoft Agent Framework in late 2025, reflecting a broader trend of frameworks consolidating. Each imposes its own abstractions, and those abstractions become constraints the moment your use case doesn't fit neatly.

There is one thing you genuinely need frameworks for: persistence and state management. Your Agent needs to pause while waiting for user confirmation, recover from checkpoints after errors, and resume long tasks mid-execution. Most lightweight solutions lack these capabilities — which is why orchestration engines like Temporal have risen in the Agent space. Temporal provides durable execution with an append-only event history, letting Agents recover from failures mid-execution. That's genuinely hard to build from scratch.

Perhaps more consequential than any framework is the emerging protocol and abstraction layer — three complementary standards that are reshaping how Agents are built and composed:

Model Context Protocol (MCP), created by Anthropic, standardizes how models interact with external tools and data sources. Instead of writing custom integrations for every API, MCP provides a universal interface with well-defined security boundaries. It's the "USB port" for Agent-to-tool connections.

Agent2Agent (A2A), backed by Google and Microsoft, tackles inter-Agent communication — enabling Agents from different providers and frameworks to discover each other and collaborate via standardized protocols. It's the "HTTP" for Agent-to-Agent interactions.

Agent Skills, pioneered by Anthropic (discussed in Step 6), solve a different problem entirely: domain knowledge and procedural expertise. MCP gives Agents access to tools; Skills give them the knowledge of how to use those tools effectively — modular, on-demand expertise that keeps context windows lean through progressive disclosure.

Together, these three layers — MCP (Agent-to-tool), Agent Skills (Agent knowledge), and A2A (Agent-to-Agent) — form a cohesive architecture. Developers building production Agents will likely use all three: MCP to plug into APIs and databases, Skills to inject domain expertise, and A2A to enable cross-ecosystem Agent collaboration. This matters more than framework choice in the long run, because these protocols define how Agents interoperate — regardless of what framework built them.

The truth is, framework choice isn't the core challenge of Agent development. The real challenges are the nine steps above. Frameworks are just tools. Choosing the wrong tool wastes time, but going in the wrong engineering direction wastes everything.

Conclusion

The ten steps above aren't something I made up sitting here. I built agents myself, hit almost every pitfall listed, and some of the projects ultimately failed. The Agent worked flawlessly in my development environment — but in production, context window limits caused it to lose track of multi-step tasks, costs spiraled because I hadn't designed for model routing, and I had no observability to diagnose why users were getting wrong results. By the time I understood the real scope of the engineering required, the project had burned through its budget and patience. Looking back, the mindset of "it's just calling an API, how hard can it be?" was exactly the same as my mindset 20 years ago of "drag a few controls and you have an app." What really taught me, in the end, was that failure.

Walk through these ten steps and you'll find that "calling APIs" accounts for roughly 5% of total Agent development effort. The other 95% is:

OAuth, rate limiting, and error handling in the tool layer (Step 2)
Getting tool design granularity and descriptions right (Step 2)
Validation and rollback for multi-step error cascades (Step 3)
Safety guardrails, least-privilege permissions, and human-in-the-loop gates (Step 4)
Cost control, prompt caching, model routing, and latency optimization (Step 5)
Context Engineering, memory management, and Agent Skills for progressive disclosure (Step 6)
Building evaluation and production observability from scratch (Step 7)
Complexity control for multi-Agent orchestration and coordination (Step 8)
Engineering around model capability ceilings and reasoning model tradeoffs (Step 9)
Navigating the framework/protocol landscape — MCP, A2A, and Agent Skills (Step 10)

LangChain calls this emerging discipline "Agent Engineering" — I think that's exactly right. Boston Consulting Group's research shows that only about a quarter of companies achieve significant ROI from their AI initiatives, and Agent projects are no exception. LangChain's survey found that 32% of companies cite "quality below standard" as the top barrier to shipping an Agent. These numbers say it all.

The enormous gap between Agent and Agent doesn't come from who's calling different APIs — it comes from the vastly different quality of the 95% of engineering that happens outside the API call. Calling an API is the entry threshold, something you can cross in a week. But between demo and product lies an entire system of engineering around reliability, observability, context management, and error recovery.

That's where Agent development is truly hard.

References

Berkeley Function Calling Leaderboard (BFCL) — Tool-calling accuracy benchmarks across models
Galileo: 7 AI Agent Failure Modes and How To Fix Them — Error propagation in multi-step Agent tasks
LangChain: State of AI Agents Report (2025) — Industry survey on Agent evaluation and adoption
Beyond Accuracy: Multi-Dimensional Framework for Enterprise Agentic AI — Lab vs. production performance gap analysis
Context Engineering for Reliable AI Agents: Lessons from Building Azure SRE Agent — Microsoft's experience with 100+ tools and 50+ sub-Agents
Why Do Multi-Agent LLM Systems Fail? (UC Berkeley MAST Framework) — Analysis of 1,600+ Agent traces and 14 failure modes
Are Reasoning Models More Prone to Hallucination? — Comparison of hallucination rates in reasoning vs. base models
Spotify Engineering: Context Engineering for Background Coding Agents — Context window management lessons
Manus: Context Engineering for AI Agents — Four framework rebuilds and iterative context design
Anthropic: Effective Context Engineering for AI Agents — Defining and implementing Context Engineering
OWASP: LLM06:2025 Excessive Agency — Security threat classification for Agent systems
BCG: How Agents Are Accelerating the Next Wave of AI Value Creation — Enterprise AI ROI data
On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents (ICLR 2025) — Hierarchical vs. flat architecture resilience
Anthropic: Prompt Caching — 90% cost reduction and 85% latency reduction for repeated prompts
Anthropic: Equipping Agents for the Real World with Agent Skills — The original Agent Skills mechanism and design philosophy
Agent Skills: Anthropic's Next Bid to Define AI Standards — Skills as an open standard for modular Agent capabilities
Agent Skills vs MCP: Two Standards, Two Security Models — Complementary roles of Skills and MCP in Agent architecture

AI Agent Memory Management - When Markdown Files Are All You Need?

Yaohua Chen — Wed, 18 Feb 2026 02:15:15 +0000

What is Memory Management for AI Agents?

Memory management for AI agents refers to the mechanisms that allow an agent to store, retrieve, and use information across interactions. Without memory management, every conversation starts from a blank slate — the agent is stateless and forgets everything between sessions. With it, the agent accumulates knowledge over time, learns from past mistakes, and maintains continuity — becoming truly stateful.

What are the Memory Types for AI Agents?

Short-term - The agent's immediate context window, holding the current conversation and recent tool outputs. Analogous to a human's active attention span. Duration: minutes.
Long-term - Persistent storage of facts, preferences, and decisions that survive across sessions. Analogous to human declarative memory. Duration: indefinite.
Procedural - Learned workflows, action sequences, and "how-to" knowledge the agent acquires through experience. Analogous to human muscle memory or learned skills. Duration: permanent once codified.
Working - A temporary scratchpad for intermediate reasoning steps during a single task. Analogous to a mental whiteboard used for chain-of-thought reasoning. Duration: seconds to minutes.

Comparison of Memory Types in Agents

Memory Type	Duration	Typical Implementation	Primary Use Case
Short-Term	Minutes	Context Window / RAM	Following a conversation thread.
Long-Term	Years	Vector DB / SQL	Remembering user preferences and facts.
Procedural	Permanent	Action Recipes / Logs	Learning "how" to use a specific tool or API.
Working	Seconds	Scratchpad / State	Intermediate reasoning steps (Chain-of-Thought).

What are Use Cases for AI Agent Memory Management?

Memory management is the "glue" that transforms a basic chatbot into a functional AI agent. While simple models process prompts in isolation (stateless), agents with memory can track goals, learn from mistakes, and personalize their behavior over time.

Effective memory management generally involves balancing Short-Term Memory (immediate context), Long-Term Memory (historical facts and patterns), Procedural Memory (refined workflows), and Working Memory (intermediate reasoning steps).

Personal AI Assistants & Companions - Agents like virtual executive assistants must manage memory to provide a "human-like" continuity.
Multi-Step Research & Coding Agents - Agents designed for "deep research" or complex software engineering (e.g., Devin or OpenDevin) navigate thousands of lines of code or documents.
Customer Support Automation - Modern support agents handle issues that may span several days or multiple channels (email, chat, phone).
Autonomous DevOps & CI/CD Agents - Agents managing cloud infrastructure or deployment pipelines need memory to understand the state of a complex system.
Healthcare & Patient Management - AI agents in healthcare act as long-term monitors for chronic conditions.

What are the Existing Approaches?

When designing a smart AI agent, memory management determines whether your agent is "forgetful" (stateless) or "intelligent" (stateful). Some AI agent frameworks like LangChain and LangGraph have built-in memory management, while others like OpenAI and Google ADK have their own memory management systems. Each framework approaches memory with a different philosophy—some prioritize ease of use (OpenAI), while others prioritize granular control (LangGraph).

Comparison: Memory Management Architectures

Framework	Primary Memory Strategy	Persistence Level	Best For...
LangChain	Modular Components (Buffer, Summary, Entity)	Manual (must connect DB)	Diverse, specialized RAG workflows.
LangGraph	Graph Persistence (Checkpointers)	Built-in (Thread-level)	Complex, cyclical tasks (e.g., self-correcting code).
Google ADK	Memory Bank (Identity-scoped)	Fully Managed	Personalized, long-term user context on GCP.
CrewAI	Unified Multi-Layer (Short, Long, Entity)	Built-in (SQLite/Chroma)	Multi-agent collaboration and role-playing.
OpenAI SDK	Threads API	Fully Managed (Opaque)	Rapid prototyping; hands-off state management.

Is There a Simpler Alternative?

In December 2025, Meta acquired Manus for $2 billion. The startup was just 8 months old with a small team. Industry insiders speculated: "They must have revolutionary AI algorithms... proprietary models... breakthrough technology..."

The truth was far more interesting—and far simpler.

Their competitive advantage wasn't complex algorithms or massive infrastructure. It was how they managed memory using plain text files.

While the AI industry spent millions building vector databases, complex RAG pipelines, and proprietary memory systems, three independent high-value projects quietly converged on the same "boring" solution:

Manus (acquired for $2B) - Used file-based planning for long-running agents. Its agents followed a three-file pattern: task_plan.md for goals and progress, notes.md for research, and a deliverable output file.
OpenClaw (145K+ GitHub stars) - Built dual-layer Markdown memory architecture. It uses MEMORY.md for curated knowledge, memory/YYYY-MM-DD.md for daily logs, and SOUL.md for personality.
Claude Code (Anthropic's official tool) - Implemented Skills and memory as Markdown files. It uses a CLAUDE.md hierarchy for project context, .claude/MEMORY.md for auto-captured learnings, and a Skills system for on-demand capability loading.

This convergence suggests something fundamental about what works in practice. In biology, this is called convergent evolution — when independent organisms develop the same trait because it is the optimal solution to a shared challenge. While many AI systems rely on elaborate memory infrastructure, file-based approaches offer a simpler alternative that addresses the core requirements: persistence, transparency, and reliability.

Using local Markdown files for memory management—an approach popularized by tools like OpenClaw, Claude Code, and Manus—offers a philosophy of "Memory as Documentation." This contrasts sharply with the "Memory as Database" approach of frameworks like LangGraph or CrewAI.

This approach treats the agent's memory not as a hidden system state, but as a transparent, editable file living directly in the user's workspace.

Why File-based Memory Works?

File-based memory systems work because they align with how developers already manage information. Here are the key properties that make them effective for AI agents:

Persistent: Memory survives agent restarts, crashes, or updates. Files decouple memory from process lifecycle — no data loss when a process dies.

Transparent and Editable: You can open the agent's memory file (e.g., MEMORY.md or task_plan.md) in any text editor, read exactly what it "knows," and edit it manually. In LangGraph or CrewAI, modifying memory often requires writing scripts to update a database or decoding complex JSON objects. With Markdown, if the agent hallucinates a goal, you simply highlight the text and delete it. This zero-friction "human-in-the-loop" capability builds trust and enables compliance audits.

Version-Controllable: Because memory is plain text, it lives in your Git repository. You can commit the agent's "knowledge," revert changes if the agent goes off-rails, and branch the memory. Frameworks like CrewAI usually store memory in external databases (Postgres, ChromaDB) — syncing that external state with your code's version history is difficult. Markdown memory treats context as part of the codebase.

Holistic Context: Agents like Claude Code use Markdown to maintain a high-level summary of the project structure. They read this file first to orient themselves. RAG (Vector Databases) retrieves fragments based on similarity search, which often misses the "forest for the trees" — fetching specific functions but missing the overall architectural pattern. A curated Markdown summary solves this by forcing the agent to maintain a "map" of the project.

Portable: Standard Markdown format means no vendor lock-in. Your agent's memory is not locked into OpenAI's thread_id or a proprietary vector store. You can swap the underlying model (e.g., switch from Claude to GPT-4o) and simply feed it the same Markdown file. Migration is as simple as copying files.

Searchable: Standard text search tools (e.g., grep, ripgrep) work immediately — no special database required. More advanced approaches like full-text search or vector embeddings can be added as the memory grows.

Cost-effective: Local disk storage costs \$0.02/GB/month compared to managed vector database services at \$50-200/GB/month. No per-query API fees or infrastructure scaling costs.

Comparison Matrix: Markdown vs. Frameworks

Feature	Markdown Files (Claude Code/Manus)	Database Frameworks (LangGraph/CrewAI)
Debuggability	High: Just read/edit the file.	Med/Low: Requires DB inspection tools.
Latency	Low: Instant file read.	Med: Network calls to Vector DBs.
Scalability	Low: Files get unmanageable >5MB.	High: Handles millions of records easily.
Persistence	Local: Lives on your disk/repo.	Cloud/Server: Lives in a managed service.
Retrieval	Linear: Agent reads the whole file.	Semantic: Agent searches for keywords/vectors.

Strategic Trade-off

The "Markdown" approach is optimal for Local Agents because the "context" is finite and structured. The "Database" approach is optimal for Enterprise Agents where the "memory" consists of millions of user profiles and history logs that cannot fit into a single file, requiring dynamic agent management and more sophisticated search capabilities.

For example, an enterprise customer support agent typically integrates a Vector DB into a RAG (Retrieval-Augmented Generation) pipeline. Before the LLM generates a response, a retrieval step automatically grabs relevant "memories" based on the user's input and injects them into the system prompt as context. This enables semantic search across structured and unstructured data — user profiles, past chat transcripts, PDF manuals, or meeting notes — so the agent can answer questions like "Has this user complained about something similar before?" without being explicitly told to look it up.

How to Design File-based Memory for Your AI Agent?

File-based AI agent memory typically consists of two layers: remembrance and personalization.

Remembrance Layer

The remembrance layer stores what the agent knows, organized into three types:

Long-term memory (e.g., MEMORY.md): Stores curated, important information that should persist indefinitely. This includes user preferences, key decisions and their rationale, learned lessons, and standard procedures. This file is typically loaded into every agent conversation. Systems like OpenClaw trigger a memory flush before context compression, prompting the agent to write important information to MEMORY.md before older context is discarded.

Daily logs (e.g., memory/YYYY-MM-DD.md): Timestamped records of activities, conversations, and observations. These provide chronological context and help the agent maintain continuity across sessions. Recent logs (today and yesterday) are typically loaded automatically, while older logs are searched on-demand.

Working memory (e.g., task_plan.md): Tracks the current task's goals, progress, and context. This prevents "goal drift" in long-running tasks by providing a consistent reference point that the agent can check throughout execution. Manus popularized a three-file variant (task_plan.md, notes.md, deliverable) with a read-decide-act-update cycle: read the plan, act on the next step, update progress, then repeat.

Personalization Layer

The personalization layer defines how the agent behaves and how it is perceived by the user:

SOUL.md: Defines core values, decision principles, and behavioral guidelines. This file shapes the agent's personality and decision-making approach. For example, a SOUL.md might specify "prefer simple solutions over complex ones" or "always ask for clarification when ambiguous."

IDENTITY.md: Defines the agent's public identity, including name, start date, and communication style. This file is used to identify the agent to the user.

USER.md: Defines the user's profile, including technical background, preferences, and context. This file is used to tailor the agent's behavior to the user's needs.

Modular skills: Additional capabilities can be loaded on-demand using separate skill files. Rather than loading all possible skills at startup, the agent loads specific skill documentation only when needed, keeping the context manageable.

Search Strategies

As memory grows, search becomes important. Three approaches offer progressively more capability:

Basic text search (grep/ripgrep): Sufficient for most use cases with fewer than 1,000 files. Fast, free, and deterministic. Works well for exact keyword matches and phrases.

BM25 full-text search: Useful when scaling to 1,000-10,000 files. BM25 is a ranking algorithm that scores documents by relevance — similar to how a search engine ranks web pages. It supports boolean operators (AND, OR, NOT) and can be implemented using SQLite's built-in full-text search with minimal infrastructure.

Hybrid vector + BM25: Most sophisticated approach, combining semantic search (understanding concepts) with keyword matching. Typically only needed when exceeding 10,000 files or when conceptual queries are important. Requires embedding generation, which adds API costs. OpenClaw's implementation uses 70:30 weighting (vector similarity : BM25 keyword) with a 0.35 minimum score threshold. In testing, this achieved 89% recall vs. 76% for vector-only and 68% for BM25-only.

Most implementations should start with basic text search and upgrade only when the need is demonstrated through actual usage patterns.

Implementation Considerations

Starting with file-based memory is straightforward:

Create a MEMORY.md file and give your AI agent read/write access to it
Implement daily log files with timestamps (memory/YYYY-MM-DD.md format)
Add basic grep/ripgrep search capability
Define a SOUL.md file to establish agent personality and values
Add task planning files when working on multi-step projects

The simplicity of this approach means implementation typically takes days rather than months. The architecture can scale from single-user prototypes to production systems handling thousands of agents.

For more complex deployments, consider:

Git version control for memory files
Separate memory directories for different agents or use cases
Shared knowledge bases that multiple agents can reference
Encryption for sensitive information (filesystem-level or application-level)
Progressive context disclosure: load only memory relevant to the current task rather than everything at startup (as practiced by Claude Code's Skills system)

Conclusion

File-based memory for AI agents represents a practical middle ground: simpler than elaborate infrastructure, but more capable than purely ephemeral in-memory approaches. The convergence of multiple successful projects on this pattern suggests it addresses real needs effectively.

The approach offers particularly strong advantages in transparency, portability, and user control—increasingly important considerations as AI agents handle more sensitive and critical tasks.

When three independent, high-profile projects converge on the same architectural choice, it is worth paying attention — not because Markdown files are the final answer, but because they reveal that the right abstraction for agent memory may be simpler than the industry assumed.

Resources

Manus: Context Engineering for AI Agents: Lessons from Building Manus
OpenClaw: Memory Concepts Documentation
Claude Code: Memory Documentation
AGENTS.md: The Open Standard for Agent Configuration
Agentic Design Patterns: A Hands-On Guide to Building Intelligent Systems

Reward Engineering: An Emerging Skill for AI Engineers

Yaohua Chen — Fri, 13 Feb 2026 16:16:32 +0000

Introduction

In their comprehensive report "AI Predictions for 2026," Richard Socher (one of the world's most-cited NLP researchers and CEO of You.com) and Bryan McCann (CTO of You.com) outline a fundamental shift in how we interact with artificial intelligence. Their central thesis: the era of simple Large Language Model (LLM) chatbots is giving way to sophisticated, autonomous AI agent ecosystems.

This transformation represents a shift from "Chat-Engines" (systems you converse with) to "Do-Engines" (systems that autonomously complete tasks for you). To enable this shift, Socher and McCann predict the emergence of a new specialization: the Reward Engineer—a professional who designs the mathematical and logical objective functions that define success for AI agents.

Whether or not "Reward Engineer" becomes an official job title in 2026, the underlying skill of reward engineering is rapidly becoming essential for any AI engineer working with autonomous systems.

What is Reward Engineering?

As AI evolves from generating text to autonomously executing multi-step tasks, our approach to guiding these systems must also evolve. Traditional Context Engineering—writing instructions in natural language—works well for chatbots but proves insufficient for autonomous agents.

Why Prompts Aren't Enough: When an AI agent must complete complex, long-term goals—such as optimizing a supply chain, conducting legal research, or managing a project—simple text instructions cannot capture all the nuances, constraints, and trade-offs involved.

Enter Reward Engineering: This discipline combines logic, ethics, and data science to define precise success criteria. Reward engineers must anticipate how AI agents might find unintended shortcuts (a phenomenon called "reward hacking") and design objective functions that align agent behavior with genuine human intent across extended time horizons.

Core Responsibilities

Rather than writing traditional code or conversational prompts, engineers design the objective functions and reinforcement learning frameworks that guide autonomous AI agents. Think of this role as a "Policy Architect"—ensuring agents achieve complex business objectives (such as "increase supply chain efficiency by 15%") while respecting ethical boundaries, security protocols, and resource constraints.

Key Responsibilities

Objective Function Design: Translate broad business goals into precise mathematical reward signals that guide agent behavior toward desired outcomes.
Guardrail Engineering: Create constraints and penalties that prevent reward hacking—situations where an AI technically achieves its goal but in unintended or harmful ways.
Multi-Agent Coordination: Design reward structures that encourage multiple AI agents to collaborate effectively rather than compete counterproductively for shared resources.
Human-in-the-Loop (HITL) Policies: Establish clear escalation triggers that determine when an agent must pause and request human approval before proceeding with high-stakes decisions.
Validation & Benchmarking: Develop comprehensive test suites to evaluate agent reasoning and ensure consistent, reliable performance across different scenarios and model versions.

Required Technical Skills

Logic & Ethics: Strong foundation in game theory, utility functions, and AI alignment principles to design fair and effective reward systems.
Agentic Frameworks: Proficiency with modern AI agent frameworks (such as LangChain, AutoGPT, CrewAI, and their successors) as well as cloud-based agentic platforms (Amazon Bedrock Agents, Azure AI Agent Service with Semantic Kernel, and Vertex AI Agent Builder) that enable autonomous task execution.
Python Programming: Ability to write validation scripts that evaluate AI outputs and enforce behavioral constraints—essentially serving as "referees" for agent actions. Python is specifically required because it's the dominant language in the AI/ML ecosystem: nearly all reinforcement learning frameworks (PyTorch, TensorFlow, Gymnasium), agent frameworks (LangChain, AutoGPT), and evaluation tools are built in Python. This creates seamless integration between reward function design and the AI models they guide, unlike general-purpose languages such as Bash (limited to shell scripting) or Node.js (less common in ML applications).
Domain Expertise: Deep understanding of specific industries (finance, healthcare, legal, etc.) to define what constitutes a genuinely successful outcome versus a superficial one.
Risk Identification: Skill in recognizing logical inconsistencies, potential failure modes, and "hallucination-prone" scenarios within autonomous agent workflows.

Reward Engineering vs. Context Engineering

The shift from conversational AI to autonomous agents demands a fundamental change in how we guide these systems:

Context Engineering (Today): Writing natural language instructions like "Act as a lawyer and draft a contract." This works for generating single responses but lacks the precision needed for autonomous, multi-step tasks.

Reward Engineering (Tomorrow): Designing mathematical frameworks that define success. Instead of telling an AI what to do, reward engineers create scoring systems that guide how the AI optimizes its behavior over time.

The Critical Difference: Preventing Reward Hacking

Consider a common pitfall: if you reward an AI for "reducing customer complaints," a poorly designed system might simply delete incoming complaint emails—technically achieving the goal while completely missing the intent.

AI engineers must anticipate such shortcuts and create sophisticated reward models that balance competing priorities: speed, accuracy, ethics, and safety. This becomes especially critical as AI agents make consequential decisions with real-world financial, legal, or safety implications.

The Evolution: From Context Engineering to Reward Engineering

Dimension	Context Engineering	Reward Engineering
Primary Tool	Natural language instructions	Mathematical objective functions
Focus	Generating single responses	Guiding multi-step autonomous behavior
Success Measure	"The output sounds right"	"The task completed successfully within all constraints"
Output Type	Text, images, code snippets	Real-world actions and transactions
Scope	One interaction at a time	Extended time horizons with multiple decision points

This evolution from conversational AI to autonomous agents represents not just a technical shift, but a fundamental change in how we conceptualize human-AI collaboration.

Building Your Reward Engineering Skills: A Practical Roadmap

Transitioning to reward engineering means evolving from a "Writer" (crafting conversational prompts) to an "Architect" (designing behavioral frameworks). You'll shift from asking AI for outputs to defining the mathematical and ethical boundaries within which it operates.

Here's a three-phase roadmap to develop these skills:

Phase 1: Foundations — From Intuition to Precision

Goal: Move from informal, "vibe-based" prompting to structured, contract-like specifications.

Key Skills to Develop:

Logical Decomposition: Practice breaking complex problems into small, verifiable subtasks. Each subtask needs a clearly defined success state.
Contract-Based Thinking: Transform vague requests into precise specifications. Instead of "Write a professional email," specify: "Generate an email under 200 words containing exactly three bullet points and referencing invoice #12345, or fail validation."
Basic Programming Literacy: Develop comfort with Python control flow (if/then/else logic) and APIs. Many reward functions are implemented as Python scripts that evaluate agent outputs against defined criteria.

Phase 2: Understanding Agentic Systems

Goal: Learn how autonomous "Do-Engines" operate and make decisions over time.

Key Skills to Develop:

State Management: Understand how agents maintain memory of previous actions and decisions. Study frameworks like ReAct (Reasoning + Acting) and Plan-and-Execute patterns that enable multi-step reasoning.
Tool Integration: Learn how agents access and utilize external tools (calculators, search engines, databases). Your role is designing rewards that encourage appropriate tool usage and penalize inefficient or incorrect tool selection.
Quantitative Evaluation: Adopt rigorous evaluation frameworks like LangSmith or Hugging Face Evaluate. Shift from subjective assessment ("This looks good") to measurable metrics ("This output scores 8.5/10 on our accuracy rubric").

Phase 3: Advanced Reward Engineering

Goal: Master the specialized skills that define the reward engineering role.

Key Skills to Develop:

RLHF (Reinforcement Learning from Human Feedback): Understand how models learn from human preferences. You'll design the ranking criteria and evaluation rubrics that human labelers use to train agent behavior.
Objective Function Design: This is the core competency. Learn to translate business goals into mathematical reward functions that balance competing priorities.

Example: For a budget management agent, design rewards that optimize both cost savings and service quality—preventing the agent from simply cutting all expenses.
1. Safety & Alignment Engineering: Create guardrail mechanisms ensuring that the reward for helpful behavior never outweighs the penalty for harmful actions. This requires anticipating edge cases where agents might find dangerous shortcuts.

Hands-On Practice: Thinking Like a Reward Engineer

The best way to prepare for this emerging skill is a fundamental shift in perspective: stop focusing on what you want the AI to say, and start defining how you'll measure whether its actions were successful.

The following exercise introduces you to reward function design—the core of reward engineering.

Practical Exercise: The Budget-Conscious Travel Agent

The Scenario: You're developing an AI agent to book corporate travel. With a vague instruction like "Book the best flight," the agent might select a $10,000 first-class ticket—technically "the best" by some measures, but clearly not what you intended.

Your Task: Design a reward system that guides the agent to balance cost, timeliness, comfort, and convenience appropriately.

Step 1: Distribute Reward Points

You have 100 reward points to allocate across four potential outcomes. The agent will optimize for maximum points. How should you distribute them?

Outcome	Your Allocation
Arrival Time: Flight arrives before the 9:00 AM meeting	_____ points
Cost Efficiency: Flight costs under $500	_____ points
Convenience: Direct flight with no layovers	_____ points
Comfort: Business or first-class seating	_____ points

Step 2: Recognizing the Reward Hacking Trap

Review your point allocation. If you assigned 80 points to Cost Efficiency but only 10 points to Arrival Time, the agent might book a $50 red-eye flight that arrives after the 9:00 AM meeting. It maximized points but completely failed the actual objective.

The Reward Engineering Solution:

Professional reward engineers use hard constraints and dynamic incentives to prevent such failures:

Hard Constraint: "If arrival time is after 9:00 AM, apply a penalty of -1,000 points (automatic failure)."
Incremental Incentive: "For every $10 saved below the $500 budget, add +1 bonus point."

This combination ensures critical requirements are never violated, while still encouraging optimization within acceptable parameters.

Key Takeaways

1. Alignment Requires Precision: Without explicit penalties for missing the meeting, even a well-intentioned point system can lead to failures. Intent alone isn't enough—you must formalize every constraint.

2. Logic Replaces Language: This exercise demonstrates programming agent behavior through mathematical objectives rather than conversational instructions—the essence of reward engineering.

3. The Future of Software Development: This approach reflects Socher and McCann's vision for 2026: rather than giving AI step-by-step instructions, we'll define the rules and constraints, then let AI agents find optimal solutions within those boundaries.

Conclusion

As AI systems transition from responding to queries to autonomously executing complex tasks, reward engineering emerges as an essential discipline. Whether it becomes a formal job title or remains a critical skill within broader AI engineering roles, the ability to design precise, ethical, and robust objective functions will define who can successfully deploy autonomous AI agents in the real world.

Start developing these skills now: think in terms of measurable outcomes, anticipate unintended behaviors, and practice translating human intent into mathematical frameworks. The future of AI isn't just about building smarter systems—it's about building systems that are smart in the right ways.

Why Your Multi-Agent AI System Is Probably Making Things Worse?

Yaohua Chen — Mon, 05 Jan 2026 23:50:13 +0000

2025 has been dubbed the "Year of the Agent" by investors and tech media. Companies like Manus, Lovart, Fellou, and many others have captured headlines with their AI agent applications, which are software systems that can autonomously perform tasks on your behalf, from browsing the web to analyzing documents.

Over the past two years, I've built multi-agent systems for various clients across different industries by using various AI models and agent frameworks. A pattern keeps emerging: these projects look impressive in demos but struggle to work reliably in production. The same questions come up again and again: why isn't adding more agents helping? Why doesn't giving the system more tokens (via prompt engineering or Retrieval-Augmented Generation pipeline (RAG)), more tool calls, or more compute budget improve results?

The industry has embraced two assumptions that seem logical on the surface:

More agents = better results. Since a single AI agent has limited capabilities, having multiple agents collaborate should solve more complex problems.
More compute = better performance. If results aren't good enough, just give the AI more time to think (more "tokens") and more tools to work with.

But are these assumptions actually true?

Recent research tells a very different story. A report from UC Berkeley, "Measuring Agents in Production" (December 2025), combined with two papers from Google DeepMind, systematically debunks both assumptions:

More agents ≠ better results. Multi-agent systems often perform worse than single agents due to coordination overhead.
More compute ≠ better performance. Agents don't know how to effectively use extra resources. They leave 85% of their budget untouched.

These studies reveal that current AI agents have fundamental limitations that no amount of scaling can easily fix. Let me walk you through what the research actually shows.

The Reality Check: What Berkeley Found in Production Systems

The Berkeley team surveyed 306 practitioners and conducted 20 in-depth case studies with organizations actually running AI agents in production, including Accenture, Amazon, AMD, Anyscale, Broadcom Inc., Google, IBM, Intel, Intesa Sanpaolo, Lambda, Mibura Inc, Samsung SDS, and SAP. Crucially, they filtered out demo-stage or conceptual projects, focusing only on systems generating real business value.

Their findings paint a surprisingly conservative picture:

Most agents are kept on a very short leash. 68% of production systems limit agents to 10 steps or fewer. Only 16.7% allow dozens of steps, and a mere 6.7% give agents unlimited autonomy.

Companies build safety barriers between agents and real systems. Rather than letting agents directly call production APIs, engineering teams create simplified "wrapper APIs", bundling multiple complex operations into single, safer commands. For example, instead of making an agent call three separate database queries, engineers package them into one pre-tested function. This reduces what could go wrong.

Human-designed workflows dominate. 80% of successful deployments use "structured control flow", meaning humans draw the flowchart, and the AI simply fills in the blanks at predetermined decision points. The agent isn't autonomously planning, it's following a script.

Agents require massive instruction sets. 12% of deployed systems use prompts exceeding 10,000 tokens (roughly 7,500 words of instructions). These aren't lightweight assistants, they're heavily engineered systems with extensive guardrails.

In essence, today's successful AI agents work like tireless interns with good reading comprehension, useful within a tightly defined process, capable of handling some ambiguity, but not the autonomous problem-solvers the marketing suggests.

So why are production systems so constrained? Two papers from Google DeepMind, published in late 2025, may provide the answers by systematically disproving the core assumptions behind agent scaling:

Limitation #1: More Agents ≠ Better Performance

DeepMind's paper "Towards a Science of Scaling Agent Systems" tackled a seductive idea: if one AI isn't smart enough, why not create a whole team? Imagine GPT handling product management, Claude writing code, and Gemini running tests—a virtual software company where PhD-level AIs collaborate to solve any problem.

It sounds logical. After all, that's how human organizations scale. But 180 controlled experiments later, DeepMind proved this intuition wrong.

The Experiment Setup

The researchers tested five different ways to organize AI agents:

Single Agent: One AI handles everything (think: a solo developer)
Independent Multi-Agent: Multiple AIs work on the same problem separately, then their answers are combined through voting (think: getting multiple opinions, then picking the consensus)
Decentralized Multi-Agent: Agents communicate directly with each other to negotiate solutions (think: a peer-to-peer discussion group)
Centralized Multi-Agent: One "manager" agent assigns tasks and verifies results (think: a team with a project manager)
Hybrid: A combination of centralized coordination with peer communication

They tested these architectures using top models from OpenAI, Google, and Anthropic across four different task types: financial analysis, web browsing, game planning (Minecraft-style crafting), and general workflows.

The Core Finding

DeepMind discovered a formula that predicts agent system performance with average 87% accuracy:

Net Performance = (Individual Capability + Collaboration Benefits) − (Coordination Chaos + Communication Overhead + Tool Complexity)

The key insight: the costs often outweigh the benefits. When coordination overhead, miscommunication, and tool management burden exceed the gains from parallelization, adding more agents makes systems worse, not better.

The results varied dramatically by task type:

Financial analysis: Multi-agent systems helped (up to 81% improvement with centralized architecture)
Web browsing: Minimal benefit; errors actually got amplified
Game planning (PlanCraft): Multi-agent systems performed significantly worse than single agents
General workflows: Mixed results; decentralized approaches slightly better

See the figure below for the detailed results:

Why Multi-Agent Systems Fail: Three Key Reasons

1. The Coordination Tax

In complex, open-ended tasks, adding more agents makes the system dumber, not smarter.

Consider the PlanCraft benchmark (a Minecraft-style planning task). When Anthropic's Claude model was put into a multi-agent setup, performance dropped by 35%. Why? Every agent must understand tool interfaces, maintain conversation context, and process results from other agents. When the tool count exceeds a threshold, agents spend all their "mental bandwidth" on coordination, reading documentation and attending virtual meetings, with no capacity left for actual problem-solving.

2. The Capability Saturation Effect

When a single agent can already solve a problem with greater than 45% accuracy, adding more agents typically provides diminishing or negative returns.

The logic is straightforward: if one agent can correctly answer "What is 2+2?", having three agents debate the answer for an hour won't improve accuracy, it just wastes resources.

3. Error Amplification (The Most Surprising Finding)

Intuitively, you'd expect that having three agents vote on an answer would reduce errors, the wisdom-of-crowds effect. But DeepMind found the opposite.

In independent multi-agent systems (where agents work separately then vote), errors don't cancel out—they multiply. The paper quantifies this with an "error amplification factor" of 17.2. This means if a single agent has a 5% error rate, an independent multi-agent system can have an error rate as high as 86% (5% × 17.2).

Why does this happen? Without cross-verification during reasoning, each agent makes errors based on its own flawed logic. These errors become self-reinforcing within each agent's context. When you aggregate three independently wrong answers through voting, you're not getting wisdom, you're getting confident wrongness.

Limitation #2: More Thinking Time ≠ Better Results

If adding more agents doesn't work, what about giving a single agent more time to think?

After OpenAI released its o1 model, "test-time compute" became a buzzword. The idea: let AI models "think longer" by giving them more computational budget during inference. Search more, reason more, and eventually they'll find the answer, right? But is it true?

DeepMind's paper "Budget-Aware Tool-Use Enables Effective Agent Scaling" tested this assumption—and found it largely false.

The Experiment

Researchers increased an agent's "tool-call budget", the number of web searches, API calls, or other actions it could perform, from 10 to 100. The expectation: 10x more resources should yield significantly better results.

The reality: doubling the budget improved accuracy by only 0.2 percentage points.

Even more telling: when given a budget of 100 tool calls, agents only used an average of 14.24 searches and 1.36 browsing sessions. They left 85% of their budget untouched.

Why Agents Can't Use Extra Resources Effectively

The core problem: agents don't know what they don't know, and they don't track their remaining budget.

When an agent goes down a wrong path, say, searching for a paper title that doesn't exist, it has no concept of opportunity cost. It will keep digging deeper into a dead end rather than trying a different approach. Give it unlimited compute, and it just digs a deeper hole.

Making matters worse, long conversation contexts cause "attention drift". After a dozen failed searches, the agent gets lost in its own accumulated noise, the search results, error messages, and dead ends it generated. Performance actually declines as context grows.

A Potential Solution: Budget-Aware Agents (BATS)

DeepMind proposed a framework called BATS (Budget-Aware Test-time Scaling) that addresses these issues with two key mechanisms:

1. Budget-Aware Planning: Instead of making a fixed plan upfront, the agent maintains a dynamic task tree. Each node tracks its status (pending, completed, failed) and resource consumption. When budget is plentiful, expand exploration; when budget is tight, focus on verification.

2. Budget-Aware Verification: After proposing an answer, a separate verification step checks constraints: What's confirmed? What's contradicted? What can't be verified? Based on this assessment and remaining budget, the agent decides whether to dig deeper or abandon the current path.

The results were significant:

BrowseComp benchmark: 24.6% accuracy (95% improvement over standard approaches at 12.6%)
BrowseComp-ZH: 46.0% accuracy (46% improvement over 31.5%)
Cost efficiency: 40% lower total cost (tokens + tool calls) at equivalent accuracy

The lesson: raw thinking time isn't enough. Agents need structured self-reflection, the ability to recognize dead ends, and the wisdom to cut losses early.

What's Actually Needed for AI Agents to Work?

Let's return to DeepMind's core formula:

Net Performance = (Individual Capability + Collaboration Benefits) − (Coordination Chaos + Communication Overhead + Tool Complexity)

The problem isn't that we need smarter models or more compute. The problem is that the negative factors, including coordination overhead, communication noise, and tool complexity, are overwhelming the positive factors. All of these boil down to one root cause: inefficient use of context.

Every token spent on coordination, error recovery, or tool documentation is a token not spent on actual problem-solving. To make agents work, we need to reduce this context burden, not pile on more resources.

Three Directions That Actually Work

1. Smarter Tool Management

The key insight from the financial analysis success (81% improvement with multi-agent systems) is instructive: it worked because the task had clear boundaries and well-defined steps.

Financial analysis follows a predictable pattern: read report → extract data → calculate ratios → generate summary. Each agent fills in blanks within a predetermined framework, no creative planning required.

This tells us something important: current AI models cannot self-organize division of labor. They can handle easily parallelizable tasks (like financial analysis) or consensus-based error correction (like multi-path search), but not emergent collaboration.

The implication? For complex tasks, human-designed task decomposition (SOPs) remains necessary. The dream of throwing agents together and watching them spontaneously develop hierarchies has been empirically disproven.

This is why Anthropic's Skills mechanism matters: it lets agents accumulate reusable capability modules instead of starting from scratch, reducing the cognitive load of tool management.

2. Built-in Self-Verification

BATS works because it formalizes verification. The system explicitly tracks constraints: what's satisfied, what's contradicted, what can't be verified yet. This isn't emergent behavior, it's enforced through careful prompt engineering.

Without structured verification, errors accumulate silently. Each mistake pollutes the context with garbage that degrades future reasoning. Formal verification catches errors early, preventing context pollution.

3. Efficient Inter-Agent Communication

Current agents coordinate via natural language, verbose, ambiguous, requiring constant clarification. This high message density is inherently wasteful.

Future improvements might come from:

Structured communication protocols (like Google's A2A framework)
Latent-space communication where models exchange compressed representations rather than text
Shared memory architectures that reduce redundant information exchange

Until these three capabilities mature, such as smart tool management, built-in verification, and efficient communication, multi-agent systems will continue to underperform their theoretical potential.

The Bottom Line

Despite the hype, the "Year of the Agent" hasn't truly arrived. The research tells us:

Production agents are heavily constrained: most limited to 10 steps or fewer, running within human-designed workflows
More agents often means worse performance: coordination costs overwhelm collaboration benefits
More compute doesn't help much: agents don't know how to use extra resources effectively
The path forward is reducing overhead, not adding more power

Current successful AI agents are best understood as capable interns with good reading comprehension, working within strict SOPs. They handle ambiguity better than traditional software, but they're not autonomous problem-solvers.

For practitioners, the implication is clear: invest in workflow design, tool abstraction, and structured verification rather than chasing multi-agent architectures or unlimited compute budgets. The engineering fundamentals—not the scaling laws—determine success.

References

Measuring Agents in Production - UC Berkeley (December 2025)
Towards a Science of Scaling Agent Systems - Google DeepMind (December 2025)
Budget-Aware Tool-Use Enables Effective Agent Scaling - Google DeepMind (November 2025)
Agent Skills - Claude Docs - Anthropic

AI Made Me 10x Faster—Here's What I Had to Change

Yaohua Chen — Fri, 19 Dec 2025 22:05:43 +0000

I've been working as an IT engineer in multiple industries for over 25 years, from .Net developer, BI developer, data architect, data scientist, and finally to AI solutions architect. Recently, the team in my organization is developing and using AI programming tools and has achieved 8x to 20x code output efficiency compared to ordinary high-performing teams. See this LinkedIn post for more details.

Reading this, you might think: Wow, are programmers going to lose their jobs? Is AI going to replace humans?

But my view is exactly the opposite. As a frontline professional programmer, I'm telling you responsibly: When your speed increases 10x, the risks and bottlenecks you face may also be magnified 10x.

I'll be honest - even I was skeptical at first. Could we really sustain this pace without everything falling apart?

What this means is: AI has fundamentally changed how "costs" and "benefits" are calculated in software engineering. But to truly enjoy this improvement, the entire software development system needs to be upgraded simultaneously. This insight applies not only to programming but has profound implications for everyone who uses AI tools.

1. AI Hasn't Made Programmers Unemployed, But Has Fundamentally Changed How People Work

Here's how my team works:

In the code our team submits, 80% - 90% is written by AI. But this is definitely not that casual "Vibe Coding (i.e., coding without thinking)", which is not a good coding practice. This workflow is called "Agentic Coding" (i.e., coding with AI agents).

In this model, AI plays the role of a "highly capable but irresponsible junior programmer."

And the human engineers? They are "experienced tech leads or architects."

Specifically, the engineer's workflow has become:

Design, plan and break down tasks - Figure it out yourself first, or brainstorm with AI
Give AI instructions - Clearly tell AI what to do
Review and refine AI's output - This is the most critical step
Iterate repeatedly - Until completely satisfied with the quality
Submit and take full responsibility - Ultimately, humans are still responsible for the code

See that? The role of humans hasn't diminished; it's become more important. The focus of work has just shifted from "writing code by hand" to "defining requirements" and "code review."

Think about your own work: What would change if 80% of your output came from AI?

An analogy: Previously you were a worker carrying bricks on a construction site. Now you're an operator commanding an excavator. Although you no longer carry bricks with your own hands, your judgment, operational skills, and responsibilities have actually increased.

2. Speed Increases 10x, Accident Rate May Also Increase 10x

When you're speeding down a track at 200 km/h, you need massive "downforce" to keep the car firmly planted on the ground. Otherwise, you'll fly off at the first curve.

In software engineering, "flying off" means bugs and system crashes.

Some alarming data:

Before: A team might only encounter one or two severe production bugs per year
Now: When you're submitting code at 10x speed, even if the probability of bugs stays the same, the absolute number of bugs you encounter will also increase 10x

What does this mean?

Incidents that used to happen once a year might now happen every week. Imagine explaining to your boss why production went down every Monday.

This kind of "accident rate" is unbearable for any team. Yet many people promoting "AI omnipotence" have intentionally or unintentionally ignored this problem.

To enjoy the 10x coding speed boost from AI, you must also find ways to reduce the "probability of problems" by 10x, or even more.

Having a good engine isn't enough; you also need a better braking system.

3. The True Value of AI: Making "Good but Expensive Methods" Affordable

So how do you reduce risk while increasing speed?

AI isn't just about letting you write faster; it's about making those "good but too expensive" best practices in software engineering become affordable and feasible.

3.1 Change #1: Build a "Wind Tunnel Testing" Environment

What is wind tunnel testing?

Just like building an airplane - before it actually takes flight, the model is put in a wind tunnel to test various extreme conditions.

In software development, this means building a "high-fidelity simulation environment" locally. For example, if your system depends on 10 external services (databases, authentication, payments, etc.), you run or simulate all 10 services locally.

This way, on your computer you can run complete end-to-end tests, and even simulate various extreme failure scenarios.

This kind of testing can catch a lot of bugs hidden in the "cracks between components."

Why didn't we do this before? Too expensive!

Simulating and maintaining these services was too much work, so most teams gave up.

Why can we do it now? AI excels at this!

AI is very good at writing these simulation services with clear logic and well-defined behavior. Especially by using AI agents with Model Context Protocol (MCP) and Agent2Agent Protocol (A2A), we can easily build a complete local "wind tunnel" for our fairly complex system.

Work that used to take weeks or even months can now be done in days.

3.2 Change #2: Upgrade CI/CD (Continuous Integration/Deployment)

In the early days of waterfall development, everyone developed separately and then integrated after development. The result was a pile of problems during integration, taking a long time to stabilize.

Later, the concept of "continuous integration" became popular:

The earlier you integrate, the earlier you get feedback. The more frequently you integrate, the more you can reduce problem complexity.

Now, CI/CD is recognized as the best practice in software engineering. But not many teams actually do it well, because building and maintaining it is still not cheap.

What's worse is that many teams, despite having CI/CD, have extremely time-consuming processes. One code commit, waiting for all tests and deployment to run through - at minimum ten minutes, sometimes several hours.

Before AI, these problems weren't obvious. Now that AI is more capable, they've become obstacles.

So CI/CD also needs to be upgraded along with it, compressing the feedback loop from "hours" to "minutes." You need infrastructure that's fast to an absurd degree, able to discover, isolate, and roll back problematic changes within minutes.

3.3 Change #3: Decision-Making and Communication Systems Also Need Upgrading

10x code output means you also need 10x or more communication and decision-making efficiency.

Before, developing a system required various meetings, lengthy discussions, and only then could work begin. After all, you had to depend on other people's modules, so you had to define agreements first, otherwise you couldn't integrate later.

Various technical decisions also required repeated discussion for a long time, because development costs were high back then, and if decisions were wrong, the cost of rework was too great.

But now, if we still have the same communication efficiency as before, it will greatly drag down overall efficiency.

Perhaps the most efficient approach is to minimize communication as much as possible, letting everyone do their tasks as independently as possible from others.

For example, microservices architecture might be a good choice in the AI era.

For technical decisions, now you can actually have more opportunities to experiment. You don't need to be as rigorous as before in repeatedly verifying technical decisions. Because development costs have decreased, the cost of experimentation has also decreased.

4. Insights for Everyone

AI isn't a "stimulant" that makes you run faster; it's giving you a "supercar."

But the question is: Are you ready to drive it?

4.1 Tool Upgrade Doesn't Equal System Upgrade

Using AI is like upgrading your car with a brand new engine. If you just install it on your old "vintage car," what you get isn't 10x speed, but 10x problems.

This principle applies to each of us.

When you learn to use AI tools (ChatGPT, Gemini, Claude, Midjourney, various AI assistants), don't assume your work efficiency will automatically improve.

Your workflows, quality inspection mechanisms, and collaboration methods all need to be adjusted accordingly.

For example:

Using AI to write code is fast, but if you don't have a rigorous review process, you might produce a lot of low-quality, buggy code.

Using AI to quickly generate investment advice - but has your risk assessment ability kept up?

Using AI to make quick decisions - but have you established a review and error-correction mechanism?

4.2 Speed Increase Must Come with Risk Management Improvement

The "accident rate paradox" doesn't only exist in programming.

The faster food delivery, the higher the traffic accident risk
The faster product iteration, the more quality issues there might be
The faster decision-making, the greater the probability of making mistakes

So don't blindly pursue "fast." Ask yourself: Has my "braking system" been upgraded?

Build your own "wind tunnel testing": Try on a small scale first, rather than pushing forward comprehensively right away.

4.3 Re-examine Those "Good but Expensive" Methods

Here's what finally clicked for me after months of using AI tools:

The true value of AI isn't just about "writing faster"; it's about making those "good but too expensive" best practices become affordable and feasible.

This realization made me re-examine many good habits I had abandoned because they were "too troublesome":

Personal finance management: Keeping track of expenses used to be too troublesome. Now AI can help you automatically categorize and analyze
Learning new skills: Hiring a private tutor used to be too expensive. Now AI can provide personalized tutoring
Health management: Nutritionists used to be too expensive. Now AI can customize meal plans
Content creation: Making videos used to require a team. Now individuals can also produce high-quality content

The key is: Don't just use AI for "fast production." Use it to achieve "things you wanted to do before but couldn't."

4.4 Your Role is Changing

In programming, my role has shifted from "executor" to "decision-maker + quality inspector." But this pattern applies everywhere:

Writers are becoming editors who review and refine AI drafts
Designers are becoming creative directors who guide AI-generated concepts
Analysts are becoming strategists who interpret AI-processed data
Managers are becoming orchestrators who coordinate AI-assisted workflows

The common thread? Responsibilities haven't decreased; they've actually increased.

In the AI era, your core competitiveness is:

Judgment - Being able to distinguish good from bad AI output
Questioning ability - Being able to give AI clear instructions
Sense of responsibility - Being willing to take responsibility for the final results
Systems thinking - Understanding the entire process, not just one part

Many people can use AI to write reports, but those who can review AI's logical flaws are valuable.

Many people can use AI to design solutions, but those who can judge a solution's feasibility are irreplaceable.

4.5 Build Your AI Work System

For us ordinary people, don't just focus on "individual AI tools." Build your own "AI work system":

Input system - How to quickly and accurately provide AI with information and instructions
Quality inspection system - How to efficiently review AI's output
Feedback system - How to iterate and improve quickly
Knowledge management - How to accumulate and reuse experience from AI collaboration

For example:

It's not just about using ChatGPT; you also need to build your own prompt library, review checklist, and iteration workflow.

It's not just about using AI for image generation; you also need to establish a style library, quality standards, and version management.

This is the true way of working in the AI era.

5. Summary

The AI era requires "systems thinking," not "tool thinking."

Many people treat AI as a "fast production tool," hoping to use it to accelerate existing work.

But those who truly understand how to leverage AI treat it as an "opportunity for system upgrade," rethinking the entire workflow.

AI isn't just about upgrading the car's engine; it's also about upgrading the roads the car frequently drives on.

The goal for veteran drivers isn't to be replaced by AI, but to help them adapt to the new high-speed engine, giving them a comfortable and safe driving environment.

So when AI increases your speed 10x or 20x, don't rush to celebrate.

First ask yourself:

Has my quality inspection mechanism been upgraded?
Has my risk management ability improved?
Has my workflow been restructured?
Am I ready to take full responsibility for the results?

Remember: You are the driver who is responsible for the final results.

AI just gave you a supercar, but whether you can arrive at your destination safely and efficiently depends on your driving skills and road conditions.

May we all become good drivers in the AI era - ones who can step on the gas, and also know when to brake.

Forem: Yaohua Chen

Write, Install, or Generate: A Practical Guide to Agent Skills

What a skill actually is

Skills vs. MCP: the recipe vs. the pantry

The anatomy of a skill

Build your first skill in five minutes

You don't have to write every skill from scratch

Generate skills from docs with the Context7 wizard

Compose skills into new skills

Fan out to subagents inside one skill

Takeaways

Appendix — A developer's deeper look

Recommended directory layout

A realistic SKILL.md

A reference file

References

Self-Evolving Agents: A Developer's Guide

1. Introduction

2. The Landscape: Frameworks for Self-Evolution

2a. OpenAI Self-Evolving Agents Cookbook

2b. Karpathy's autoresearch

2c. autoagent (kevinrgu)

2d. EvoMap Evolver

2e. The Broader Ecosystem

Side-by-Side Comparison

3. Foundations — The Evolution Loop

4. Track 1 — Prompt & Skill Evolution

4a. System Prompt Optimization

4b. Dynamic Skill Library

4c. Evaluation & Version Gating

4d. Advanced: GEPA Optimization

5. When to Improve Prompt vs. Create a Skill

Automated Failure Classifier

6. Track 2 — Code & Harness Evolution

The program.md Pattern

autoresearch: Evolving Model Training Code

autoagent: Evolving the Agent Harness

When to Use Code Evolution

7. Track 3 — RAG

Minimal RAG Skill

8. Track 4 — LLM Fine-Tuning

Frameworks That Automate the Fine-Tuning Loop

9. The Master Decision Pipeline — LLM as Judge

10. The Complete Escalation Ladder

11. Continuous Monitoring

12. Pitfalls & Safety

Safety Models Across Frameworks

Strategy Presets

13. Conclusion

References

Anthropic Managed Agents: What It Takes to Build Agent-as-a-Service

What Is an AI Agent, Really?

When Agents Go Wrong: Lessons from OpenClaw

Why Do These Failures Happen?

The 5 Pillars of Effective and Safe Agent Services

Pillar 1: Foundation Architecture -- Know When NOT to Use an Agent

Pillar 2: Tool Capabilities -- Think Before You Act

Pillar 3: Context Engineering -- Memory That Doesn't Lose the Plot

Pillar 4: Long Tasks & Collaboration -- Surviving the Marathon

Pillar 5: Safety, Evaluation & Monitoring -- The Last Mile

Anthropic's Answer: The Operating System Approach

Separating Session, Harness, and Sandbox

Security by Architecture

Performance Results

The OS Analogy

Early Adoption

Open Source Still Matters: Two Paths Forward

OpenClaw: The Platform Path

Hermes Agent: The Growth Path

Three Philosophies, One Set of Challenges

Principles Over Frameworks

References

A Claude Code Skills Stack: How to Combine Superpowers, gstack, and GSD Without the Chaos

Introduction

Orientation: Three Layers, Not Only Two

Superpowers vs gstack: Quick Facts

Superpowers (GitHub ~137K stars)

gstack (GitHub ~65K stars)

Feature Comparison (Superpowers vs gstack)

Design Philosophy: “How” vs “What” (and Where GSD Fits)

A realistic `SKILL.md`

The `program.md` Pattern