Forem: Roman Belov

Prompt Engineering System: Managing 50+ Prompts in Production

Roman Belov — Fri, 10 Apr 2026 04:04:41 +0000

The average LLM project in production uses 20–50 prompts. Classification, summarization, data extraction, response generation, quality evaluation. Each prompt requires iteration, and each iteration can break something that was working. At 50 prompts, managing them manually becomes chaos: who changed the classifier prompt? Why did summarizer accuracy drop? Which version is in production right now?

This article covers how to build a prompt management system that scales from 5 to 500 prompts.

Why You Can't Store Prompts in Code

A prompt looks like a string. Developers store it in code, next to the call logic. This works fine when there are only a few prompts and iterations are infrequent.

Problems start at scale:

Changing a prompt requires deploying the app. The prompt is hardcoded. To fix a single word in a system prompt, you need a PR, review, merge, deploy. Iteration cycle: hours instead of minutes.

No versioning. Git stores history, but a diff on a 2,000-character prompt is unreadable. There's no fast path to roll back a prompt to a previous version without rolling back the entire app.

No link between version and metrics. Prompt changed, quality dropped. Connecting a specific prompt version to specific metrics is manual work when the prompt lives in code.

Cross-team chaos. The product manager wants to adjust the tone. The ML engineer is optimizing tokens. The developer is refactoring the template. All three are editing the same file, and the outcome is unpredictable.

Anatomy of a Prompt Engineering System

A mature prompt management system has four layers:

┌─────────────────────────────────────────────────┐
│              Prompt Engineering System          │
├────────────┬────────────┬────────────┬──────────┤
│  Registry  │  Testing   │  Deploy    │ Monitor  │
│            │            │            │          │
├────────────┼────────────┼────────────┼──────────┤
│ Storage    │ Pre-deploy │ Canary /   │ Metrics  │
│ + versions │ eval       │ A/B rollout│ + alerts │
└────────────┴────────────┴────────────┴──────────┘

Registry — a centralized prompt store with versioning, metadata, and access control.

Testing — automated quality evaluation of a prompt against test datasets before deploying to production.

Deploy — a mechanism to push a new prompt version to production without deploying the application.

Monitor — tracking quality metrics tied to specific prompt versions.

You don't need to build all four layers at once. A minimum viable system is registry + deploy. Without testing and monitoring, you're flying blind.

Registry: Centralized Prompt Storage

The registry solves the basic problem: a single source of truth for all prompts. Two approaches.

Approach 1: Langfuse Prompt Management

Langfuse provides prompt management out of the box. Each prompt is a named entity with versions, labels, and variables.

from langfuse import Langfuse

langfuse = Langfuse()

# Get the production version of a prompt
prompt = langfuse.get_prompt(
    name="ticket-classifier",
    label="production"  # or "staging", "latest"
)

# Prompt with variables
system_message = prompt.compile(
    categories="billing,technical,general,urgent",
    language="en"
)

Prompt structure in Langfuse:

Field	Purpose	Example
`name`	Unique identifier	`ticket-classifier`
`version`	Auto-increment	`14`
`label`	Environment / status	`production`, `staging`
`type`	Format	`text` or `chat`
`config`	Model parameters	`{"model": "gpt-4o-mini", "temperature": 0}`

The prompt is decoupled from code. A product manager edits the prompt in the UI, assigns the staging label, tests it, and switches to production. The application code stays the same.

Approach 2: Prompts-as-Code

For teams that prefer Git as the single source of truth:

prompts/
├── ticket-classifier/
│   ├── prompt.yaml
│   ├── config.yaml
│   └── tests/
│       ├── dataset.jsonl
│       └── eval.py
├── summarizer/
│   ├── prompt.yaml
│   ├── config.yaml
│   └── tests/
└── prompt_registry.py

# prompts/ticket-classifier/prompt.yaml
name: ticket-classifier
type: chat
model: gpt-4o-mini
temperature: 0
messages:
  - role: system
    content: |
      You are a support ticket classifier.
      Categories: {{categories}}.
      Return JSON: {"category": "...", "confidence": 0.0-1.0, "reasoning": "..."}
      Response language: {{language}}.
  - role: user
    content: "{{ticket_text}}"
variables:
  categories: "billing,technical,general,urgent"
  language: "en"

# prompt_registry.py
import yaml
from pathlib import Path

class PromptRegistry:
    def __init__(self, prompts_dir: str = "prompts"):
        self.prompts_dir = Path(prompts_dir)
        self._cache = {}

    def get(self, name: str) -> dict:
        if name not in self._cache:
            prompt_path = self.prompts_dir / name / "prompt.yaml"
            with open(prompt_path) as f:
                self._cache[name] = yaml.safe_load(f)
        return self._cache[name]

    def compile(self, name: str, **variables) -> list[dict]:
        prompt = self.get(name)
        messages = []
        for msg in prompt["messages"]:
            content = msg["content"]
            for key, value in {**prompt.get("variables", {}), **variables}.items():
                content = content.replace(f"{{{{{key}}}}}", str(value))
            messages.append({"role": msg["role"], "content": content})
        return messages

Both approaches support a hybrid variant: prompts live in Git, and CI/CD syncs them to Langfuse on every merge to main.

# ci/sync_prompts.py — called in CI pipeline
from langfuse import Langfuse
from prompt_registry import PromptRegistry

langfuse = Langfuse()
registry = PromptRegistry()

for prompt_name in ["ticket-classifier", "summarizer", "response-generator"]:
    prompt_data = registry.get(prompt_name)
    langfuse.create_prompt(
        name=prompt_name,
        prompt=prompt_data["messages"],
        config={"model": prompt_data["model"], "temperature": prompt_data["temperature"]},
        labels=["production"],
    )

Testing: Eval Before Deploying a Prompt

A prompt without tests is a gamble. Every change can silently break edge cases. Automated evaluation before deployment catches regressions before they reach users.

Datasets: The Gold Standard

Every prompt needs a test dataset. Minimum size: 20–30 examples covering the main scenarios and edge cases.

{"input": "Can't process payment, card is being declined", "expected": {"category": "billing", "confidence_min": 0.8}}
{"input": "App crashes when opening the chat", "expected": {"category": "technical", "confidence_min": 0.8}}
{"input": "I want to delete my account and all my data", "expected": {"category": "general", "confidence_min": 0.7}}
{"input": "URGENT! Server is down, customers can't log in", "expected": {"category": "urgent", "confidence_min": 0.9}}

Dataset sources:

Production logs. Real requests with labeled responses. The most valuable source.
Manual labeling. For new prompts with no production data yet.
Synthetic data. An LLM generates variations of existing examples. Useful for expanding edge case coverage.

Eval Pipeline

import json
from openai import OpenAI
from prompt_registry import PromptRegistry

client = OpenAI()
registry = PromptRegistry()

def evaluate_prompt(prompt_name: str, dataset_path: str, threshold: float = 0.85):
    """Evaluate a prompt against a dataset. Return pass/fail."""
    with open(dataset_path) as f:
        examples = [json.loads(line) for line in f]

    correct = 0
    total = len(examples)
    failures = []

    for example in examples:
        messages = registry.compile(prompt_name, ticket_text=example["input"])
        response = client.chat.completions.create(
            model=registry.get(prompt_name)["model"],
            messages=messages,
            temperature=0,
        )

        result = json.loads(response.choices[0].message.content)

        if result["category"] == example["expected"]["category"]:
            if result["confidence"] >= example["expected"]["confidence_min"]:
                correct += 1
            else:
                failures.append({
                    "input": example["input"],
                    "reason": f"low confidence: {result['confidence']}",
                })
        else:
            failures.append({
                "input": example["input"],
                "reason": f"wrong category: {result['category']}",
            })

    accuracy = correct / total
    passed = accuracy >= threshold

    return {
        "accuracy": accuracy,
        "threshold": threshold,
        "passed": passed,
        "failures": failures,
    }

For complex cases, LLM-as-Judge fits well. A judge model evaluates response quality against defined criteria: relevance, completeness, tone.

CI/CD Integration

# .github/workflows/prompt-eval.yml
name: Prompt Evaluation
on:
  pull_request:
    paths:
      - 'prompts/**'

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install dependencies
        run: pip install openai langfuse pyyaml

      - name: Run prompt evaluations
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python ci/eval_prompts.py --changed-only

      - name: Comment PR with results
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const results = JSON.parse(fs.readFileSync('eval_results.json'));
            let body = '## Prompt Eval Results\n\n';
            for (const [name, result] of Object.entries(results)) {
              const status = result.passed ? '✅' : '❌';
              body += `| ${name} | ${status} | ${result.accuracy.toFixed(2)} | ${result.threshold} |\n`;
            }
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body
            });

Every PR touching prompts automatically runs the eval pipeline and posts results as a comment.

Deploy: Shipping Prompts Without Deploying Code

Three strategies for delivering a new prompt version to production.

Instant Switch

The simplest option. Flip the production label to a new prompt version.

# In Langfuse UI: assign label "production" to prompt v14
# The app picks it up automatically on the next request

prompt = langfuse.get_prompt(
    name="ticket-classifier",
    label="production",
    cache_ttl_seconds=300,  # 5-minute cache
)

Good for non-critical prompts and quick fixes. Risk: 100% of traffic immediately hits the new version.

Canary Deploy

Gradual traffic shift: 5% → 25% → 50% → 100%.

import random

def get_prompt_with_canary(
    name: str,
    canary_percentage: int = 10,
) -> tuple[dict, str]:
    """Return a prompt and its version (production or canary)."""
    if random.randint(1, 100) <= canary_percentage:
        prompt = langfuse.get_prompt(name=name, label="canary")
        return prompt, "canary"
    else:
        prompt = langfuse.get_prompt(name=name, label="production")
        return prompt, "production"

Canary and production metrics are compared in real time. If canary degrades — automatic rollback.

Feature Flags

For teams with an existing feature flag system (LaunchDarkly, Unleash, or homegrown):

def get_prompt_version(name: str, user_id: str) -> str:
    """Determine the prompt version via feature flag."""
    flag = feature_flags.get(f"prompt_{name}_version")

    if flag.is_enabled(user_id):
        return flag.get_variant(user_id)  # "v14", "v15"
    return "production"

You can also target specific users, segments, or regions.

Monitor: Tying Metrics to Prompt Versions

Monitoring without version context is useless. Quality dropped — but what broke: the prompt, the model, the data?

Tracing with Prompt Version

Every LLM call should include the prompt version in metadata:

trace = langfuse.trace(
    name="ticket-classification",
    metadata={
        "prompt_name": "ticket-classifier",
        "prompt_version": prompt.version,      # 14
        "prompt_label": "production",
        "model": "gpt-4o-mini",
    },
)

generation = trace.generation(
    name="classify",
    model="gpt-4o-mini",
    prompt=prompt,  # Langfuse automatically links the version
    input=messages,
    output=response,
)

Version Dashboard

Key metrics to monitor:

Metric	What it shows	Alert when
Accuracy	Fraction of correct responses	< threshold for prompt
Latency p95	Response time	> 2x baseline
Token usage	Token consumption	> 1.5x vs previous version
Error rate	Fraction of invalid responses	> 5%
Cost per request	Cost per call	> budget

# Example: automatic comparison of two prompt versions
def compare_prompt_versions(
    prompt_name: str,
    version_a: int,
    version_b: int,
    metric: str = "accuracy",
) -> dict:
    """Compare metrics for two prompt versions from Langfuse."""
    traces_a = langfuse.fetch_traces(
        name=f"{prompt_name}-eval",
        metadata={"prompt_version": version_a},
        limit=1000,
    )
    traces_b = langfuse.fetch_traces(
        name=f"{prompt_name}-eval",
        metadata={"prompt_version": version_b},
        limit=1000,
    )

    scores_a = [t.scores[metric] for t in traces_a if metric in t.scores]
    scores_b = [t.scores[metric] for t in traces_b if metric in t.scores]

    return {
        "version_a": {"version": version_a, "mean": sum(scores_a) / len(scores_a)},
        "version_b": {"version": version_b, "mean": sum(scores_b) / len(scores_b)},
        "diff": (sum(scores_b) / len(scores_b)) - (sum(scores_a) / len(scores_a)),
    }

Regression Alerts

# Check metrics every 15 minutes (cron job or Langfuse webhook)
def check_prompt_regression(prompt_name: str):
    current_version = langfuse.get_prompt(name=prompt_name, label="production").version
    recent_scores = get_recent_scores(prompt_name, current_version, hours=1)

    baseline = get_baseline_scores(prompt_name, current_version)

    if recent_scores["accuracy"] < baseline["accuracy"] * 0.9:  # > 10% degradation
        alert(
            channel="slack",
            message=f"Regression detected: {prompt_name} v{current_version}. "
                    f"Accuracy: {recent_scores['accuracy']:.2f} "
                    f"(baseline: {baseline['accuracy']:.2f})",
        )
        # Automatic rollback to previous version
        rollback_prompt(prompt_name, to_version=current_version - 1)

Prompt Organization Patterns

Composition Over Monoliths

A 3,000-token monolithic prompt is hard to test and maintain. Break it into components:

# prompts/components/output-format.yaml
name: output-format-json
content: |
  Respond STRICTLY in JSON. No text before or after the JSON.
  If you cannot determine the answer, return {"error": "unable to classify"}.

# prompts/components/language-rules.yaml
name: language-rules
content: |
  Response language: {{language}}.
  Do not translate proper nouns or technical terms.

def compose_prompt(*component_names: str, **variables) -> str:
    """Assemble a prompt from components."""
    parts = []
    for name in component_names:
        component = registry.get(f"components/{name}")
        content = component["content"]
        for key, value in variables.items():
            content = content.replace(f"{{{{{key}}}}}", str(value))
        parts.append(content)
    return "\n\n".join(parts)

# Usage
system_prompt = compose_prompt(
    "ticket-classifier-core",
    "output-format-json",
    "language-rules",
    categories="billing,technical,general",
    language="en",
)

Naming Convention

At 50+ prompts, consistent naming matters:

{domain}-{task}-{variant}

ticket-classifier-v2
ticket-classifier-multilingual
order-summarizer-short
order-summarizer-detailed
response-generator-formal
response-generator-casual
quality-judge-relevance
quality-judge-toxicity

Prompt Metadata

Each prompt should carry metadata for auditing:

name: ticket-classifier
metadata:
  owner: ml-team
  created: 2026-01-15
  last_tested: 2026-03-20
  model_compatibility:
    - gpt-4o-mini
    - claude-3-5-sonnet-20241022
  avg_tokens: 450
  cost_per_call_usd: 0.002
  test_accuracy: 0.92
  dataset_size: 150

Scaling: From 5 to 500 Prompts

How the system evolves as the number of prompts grows:

Scale	Registry	Testing	Deploy	Monitor
5–10 prompts	YAML in Git	Manual eval	Instant switch	Logs
10–50 prompts	Langfuse + Git sync	CI eval pipeline	Canary	Version dashboard
50–200 prompts	Langfuse + RBAC	CI + LLM-as-Judge	Feature flags	Alerts + auto-rollback
200+ prompts	Custom registry	Eval platform	Progressive rollout	ML monitoring

Key thresholds:

10 prompts — you need a registry. Prompts in code become unmanageable.

30 prompts — you need CI eval. Manual testing doesn't scale; regressions slip through.

50 prompts — you need RBAC. Different teams own different prompts; access control becomes non-optional.

100 prompts — you need auto-rollback. Humans can't respond to regressions fast enough in real time.

Prompt Management Tools

Tool	Type	Strengths
Langfuse	Open-source	Prompt management + tracing + evals in one. Self-hostable
PromptLayer	SaaS	Specialized in prompt management. Good UI
Humanloop	SaaS	Prompt management + eval + annotation. Enterprise
Pezzo	Open-source	Prompt management. Lightweight
Custom	Custom	Git + YAML + CI scripts. Maximum control

Langfuse covers most scenarios: registry with versioning, prompt-to-trace linking, dataset-based evals, MCP server for IDE management. Detailed walkthrough in the Langfuse guide.

Common Mistakes

Prompts in .env or config files. No versioning, no testing, no connection to metrics. Fine for prototypes, falls apart in production.

Testing on three examples. The prompt passes three tests and ships to production. A week later you discover it breaks on long inputs or edge case categories.

No baseline. The new prompt version "works well." Without a baseline, there's nothing to compare against. The previous version may have been better.

Optimizing tokens at the expense of quality. Prompt reduced from 800 to 300 tokens. Cost drops 60%. Accuracy drops from 0.94 to 0.81. Saving $50/month costs dozens of wrong responses every day.

Context Engineering for Prompts

A prompt doesn't exist in isolation. Quality depends on what's fed alongside it: context engineering determines which data enters the context window and in what order.

Three rules for production prompts:

Variables instead of hardcoded values. Anything that might change (categories, languages, formats) goes into variables. The prompt stays stable.
Few-shot examples at the end. Models "see" the end of the context more clearly. Placing examples after instructions improves accuracy.
Minimal context. Every extra token in the prompt dilutes the model's attention. If an instruction doesn't affect quality — remove it.

Where to Start

Week 1. Inventory. Collect all prompts from your codebase into one place — YAML files in Git or Langfuse. Standardize the format: name, version, model, messages, variables.

Week 2. Datasets. For each prompt, collect 20–30 test examples from production logs. Label the expected output.

Week 3. Eval pipeline. A script that runs the prompt against the dataset and outputs accuracy. Triggered in CI when prompts change.

Week 4. Monitoring. Prompt version in every trace's metadata. Dashboard with metrics per version. Alert on > 10% degradation.

After a month — a working system where every prompt change is tested, versioned, and monitored. No chaos, no regressions, no "who changed this prompt?"

Context Engineering: How to Manage Context for AI Models and Agents

Roman Belov — Thu, 09 Apr 2026 04:33:23 +0000

Claude's context window holds 200,000 tokens. Gemini's stretches to two million. But response quality starts degrading long before the window fills up. Window size doesn't solve the context problem — it masks it.

Prompt engineering teaches you how to ask. Context engineering teaches you what to feed the model before asking. And the second one shapes the answer more than the first.

Andrej Karpathy put it this way: "Context engineering — the delicate art and science of filling the context window with just the right information for the next step." Tobi Lütke, CEO of Shopify, popularized the term itself, and Gartner declared in July 2025: "Context engineering is in, and prompt engineering is out."

This piece covers concrete techniques, models, and patterns. Things that actually work when you're using AI agents in development every day.

Prompt vs Context: Where the Line Falls

Here's an analogy that works: you're hiring an expert consultant.

Prompt — your question: "What should I do?"
Context — the briefing you hand them before the question.

You can phrase the question perfectly, but if the briefing contains 500 pages of irrelevant documents, even a strong expert will get lost. Flip it around: hand them exactly the 2 pages they need, and even a simple question yields a precise answer.

Prompt engineering answers questions like: how to frame the task, what role to assign the model, what output format to request. Context engineering answers different ones: feed 100 reviews or pick 15 representative ones? The entire 500-line file or just lines 45–80? All the documentation or extract the facts?

A more technical analogy drives it home. The LLM is a CPU. The context window is RAM. You're the operating system deciding what gets loaded into working memory. The goal: load exactly the data needed for the current operation.

Why More Context Is Worse

This is counterintuitive, but backed by research.

Context Rot

Chroma's research (2025) showed that LLM accuracy drops as the token count in context grows — even when the window is far from full.

The mechanism: attention is a fixed resource. Weights always sum to 1. More tokens means less attention per fragment. Think of a flashlight — the wider the beam, the dimmer the light at any point. And the harder the task, the steeper the drop.

Lost in the Middle

A study found a specific pattern: LLM performance drops 30%+ when critical information sits in the middle of a long context. Beginning and end? Fine. The middle is a blind spot.

Practical takeaway: put the important stuff at the beginning or end. System prompt up top. Few-shot examples at the bottom.

Economics

Every token costs money, and the model rereads the entire context on every request (LLMs are stateless):

Input: ~$3 per 1M tokens (Claude Sonnet)
100K context × 100 requests/day = ~$30/day = $900/month

Context engineering is budget engineering too.

Hallucinations from Overload

With a bloated context, the model tries to "use everything" and starts inventing connections between unrelated parts. Data about Company A gets attributed to Company B. Functions that don't exist get "recalled" from similar code that drifted into the context twenty screens back.

Six Layers of Context

Structure context like an onion — six layers, each with a specific job. This fights degradation by placing the most important information at the beginning and end, instead of spreading it across the middle.

┌─────────────────────────────────────────┐
│  1. SYSTEM — who you are & how to act   │  ← Permanent (beginning)
├─────────────────────────────────────────┤
│  2. PROJECT — project context           │  ← Semi-permanent
├─────────────────────────────────────────┤
│  3. TASK — the specific task            │  ← Per task
├─────────────────────────────────────────┤
│  4. DIFF / CODE — relevant fragments    │  ← Per task
├─────────────────────────────────────────┤
│  5. ACCEPTANCE CRITERIA — exit criteria │  ← Per task
├─────────────────────────────────────────┤
│  6. EXAMPLES (Few-shot) — samples       │  ← Optional (end)
└─────────────────────────────────────────┘

System: Role and Behavior

Who the model is and how it should behave. Always at the very beginning.

You are an experienced backend developer working with Python and FastAPI.
Keep answers concise. Use type hints. Don't add dependencies without asking.

This is where the role, response style, and constraints go. Task details do not belong here — that's layer 3.

Project: Project Context

Tech stack, structure, architecture decisions, code conventions. This layer gets reused across tasks. In Claude Code, it lives in the CLAUDE.md file — the agent reads it automatically on every launch.

Task: What to Do

A clear description of what to do, why, and — this gets forgotten constantly — what not to do.

A good example:

Task: Add rate limiting to /users.
Context: Endpoint is unprotected, bots are overloading it.
Requirements: 100 req/min per IP, Redis for counters, 429 on exceeded.
Out of scope: Changing endpoint logic, adding authorization.

A bad example: "Add rate limiting."

Diff/Code: Only What's Relevant

Provide only the code fragments that relate to the task. Not the entire file. Specify path and lines: app/api/users.py, lines 45–60.

Acceptance Criteria: When to Stop

Clear, verifiable conditions. The model only knows when to stop if you tell it. Skip these and you'll get either a half-finished answer or something wildly overengineered.

- [ ] Return 429 status when limit is exceeded
- [ ] Include Retry-After header in the response
- [ ] Unit tests cover edge cases

Examples (Few-shot): At the End

For nonstandard output formats or a specific style. Place them at the end of the context — the model "sees" the finale better.

What Hurts to Put in Context

A few anti-patterns that will reliably tank your results:

The entire project codebase — signal drowns in noise
Contradictory instructions — "use Redux" + "use Context API" = the model gets confused
Outdated examples — code with deprecated APIs gets reproduced verbatim
Vague phrasing — "make it better" gives the model no direction

Four Strategies for Managing Context

LangChain and Anthropic propose a framework: all context work boils down to four actions.

Strategy	What It Does	Example
Write	Persist externally	Scratchpads, MEMORY.md, progress files
Select	Extract only what's relevant	RAG, grep, code search
Compress	Compact	Compaction, summarization, tool result cleanup
Isolate	Isolate tasks	Subagents with clean context

Everything described below is a specific case of one of these four.

Persistence: Bridging Sessions

Every session with an AI agent starts from scratch. New context window, zero memory of previous work. Anthropic calls it the "shift engineer" problem: each new engineer coming on shift remembers nothing of the previous one's work. No notes left behind? Start over.

Plain Files

The most basic form of memory — markdown notes the agent writes for its future self. Claude Code uses MEMORY.md for this: the agent automatically records project patterns, decisions, and architectural notes.

Git as Memory

Commits with meaningful messages form a changelog and restore points. The agent can experiment freely, knowing it can always roll back.

Structured Notes

Plain files evolve. Instead of a flat log, the agent maintains a structured knowledge base. The pattern: write_to_notes(topic, content) + read_from_notes(topic) — an external hard drive for memory.

An example from Anthropic: an agent playing Pokemon recorded "trained Pikachu 1234 steps, 8 out of 10 levels." After a context reset, it read its own notes and picked up right where it left off.

Scratchpad

Working memory within the current session. The agent "thinks out loud" — storing intermediate results, hypotheses, a plan. Scratchpad is RAM; files are disk.

Simple thought, but it changes everything: stop making the model remember. Give it a notebook.

Context Compaction

When the context fills up, compress it. The model gets the full history and produces a summary. Old conversation gets tossed, compressed version goes at the start of the new context.

Manual compaction at logical breakpoints (after finishing a feature) beats automatic. There's also a lighter variant: cleaning up tool results — strip the verbose command outputs from history, keep just the fact that they ran.

Task Trackers

For long-running projects, the "Initializer + Executor" pattern works well. The first agent doesn't write code — it creates a structured task list in JSON: description, status, dependencies. Each subsequent agent reads the list, picks a pending task, completes it, updates the status, and commits.

Subagents: Isolation as Strategy

The main agent can delegate a subtask to a subagent — a separate process with its own clean context window. Like a manager asking a database specialist to optimize a query: hand them the schema and the slow query, not the entire month's email thread.

Three wins:

Context purity. The subagent isn't polluted by the main agent's history. The main agent might have 85% of its window occupied — the subagent starts at 5%.
Specialization. You can use different models or system prompts for different subagents.
Parallelization. Multiple subagents can work simultaneously.

In Claude Code, subagents are launched via the Task tool. The main agent describes the task, the subagent receives it in a clean context, does the work, and returns a structured result. The main agent's context cost is minimal.

MCP and the Tool Tax

MCP (Model Context Protocol) is an open standard defining how AI agents discover and call tools. Each MCP server adds its tool descriptions to the context. Every description costs tokens.

You feel it the moment you start working for real: connect 5–10 MCP servers (GitHub, Slack, database, analytics, monitoring) and tens of thousands of tokens in tool descriptions land in every request, even when none of them get called.

The fix is lazy loading. Claude Code uses Tool Search: tool descriptions load on demand, only when the agent decides it might need one. Saves around 85% of tokens. Other agents have similar tricks: lazy-mcp, MetaMCP.

Tool design principles:

Self-sufficiency: the description contains everything needed for use. The model doesn't read your README.
Unambiguity: user_email instead of data, validate_payment instead of process.
Minimalism: one tool = one atomic operation. If the description exceeds 200 words, the tool does too much.

Memory Hierarchy

Context in production tools isn't a single file — it's a multi-level system. Claude Code's docs lay out the hierarchy:

System prompt — base instructions (always loaded)
Settings — user preferences
CLAUDE.md — project instructions (loaded from the repository root)
Rules — modular instructions, can be path-specific (loaded only when working with certain files)
Skills — entire folders of instructions and scripts the agent loads at its own discretion
Auto Memory — memory the agent forms for itself

Martin Fowler proposes a useful distinction: Instructions (orders — "write a test for this function") vs Guidance (general rules — "all tests must be independent of each other"). CLAUDE.md and rules are mostly Guidance. Chat prompts are Instructions.

Working with Large Documents

You can't just dump a 50-page PDF into the model. You need a strategy.

Chunking

Break it into pieces of 1,500–3,000 tokens with 10–20% overlap. Semantic chunking (by chapters and sections) works noticeably better than chopping at fixed lengths.

Contextual Retrieval from Anthropic tackles the ripped-from-context problem: before indexing, each fragment gets a description of where it came from and what the section covers. Result: at least 35% fewer retrieval failures, up to 67% with reranking.

Fact Extraction

Skip the full text. Pull a structured list of facts and figures from each chunk instead. Smaller footprint, better accuracy for analysis.

Map-Reduce

For very large documents: split into chunks, summarize each (MAP), assemble the mini-summaries into a final one (REDUCE). The MAP phase can be parallelized — speedup scales with the number of workers.

RAG vs Long Context

With windows getting bigger (Gemini 2M), the question keeps coming up: do we still need RAG? Research (arXiv:2501.01880) says it depends on the task.

RAG wins: the corpus is huge (> 1M tokens), freshness matters, budget is limited.

Long context wins: you need synthesis across sections, structural understanding, document < 200K.

Hybrid (the way to go): RAG for selection, long context for analysis. The cost gap is real: full 2M context on every request runs an order of magnitude more than RAG selection + 50K of relevant context.

Where This Doesn't Work

Wouldn't be honest to stop at the upsides.

Context engineering won't fix a bad model

If the model can't write Rust, no amount of context will help. Context engineering works within what the model can already do. If the task is too hard for the current generation, break it into subtasks or try a different angle.

Preparation overhead

Assembling a perfect six-layer context package for every request takes time. For quick questions ("how does this function work?") it's overkill. Context engineering pays off on repeatable tasks and with agents that chain dozens of operations.

Compaction loses information

Compression is a tradeoff. The model picks what to keep and what to toss. Sometimes it tosses what matters. Manual compaction at logical breakpoints is safer, but needs the operator paying attention.

Lost in the Middle works both ways

You can get so focused on "important stuff at the beginning and end" that the middle turns into a junk drawer. Better to cut the context down than hope positioning saves you.

Subagents add latency

Delegating to a subagent means a separate API call with its own context. On a complex task, one subagent fires dozens of requests. For anything real-time, that's too slow.

Lazy tool loading isn't free

Tool Search saves context but adds a search step. If the agent hunts for a tool before every action, that's extra requests and wasted time. Balancing tools-in-context against search frequency takes tuning.

Common mistakes

Three that come up more than anything else:

Copying an entire file instead of the relevant fragment. The model gets 500 lines when it needed lines 45–60. The other 440 lines are pure noise.
Not saying what NOT to do. Without constraints, the model refactors the whole file when you asked it to fix one function.
Skipping acceptance criteria. The model doesn't know when to stop. You get either undercooked or overcomplicated output.

Checklist

Run through this before every serious request to a model.

Before the request:

Is there a source of truth (docs, code, data) in the context?
Is the task clearly described?
Is the output format specified?
Is what NOT to do specified?

In the prompt:

"Answer only based on the provided context"
"If you don't know — say you don't know"

After the response:

Are the facts verified?
Do the referenced functions and libraries actually exist?
Were characteristics of one object attributed to another?

Five takeaways

Less = better. Quality and relevance of context matter more than quantity. The goal is the smallest set of tokens with the strongest signal.
Structure it. Six layers: System, Project, Task, Code, Criteria, Examples. Important stuff at the beginning and end.
Persist it. Persistence = bridge between sessions. State files, structured notes, git.
Isolate it. Subagents with clean context for specialized tasks.
Compress it. Compaction and tool result cleanup when the context grows.

Start small: assemble a six-layer context package for one typical task and compare the result to what you get from pasting code into the chat. The difference tends to be obvious on the first try.

FAQ

At what token count does context rot become practically noticeable, and is there a threshold to monitor?

Multiple benchmarks (including studies by Chroma and others) show measurable accuracy degradation starting around 20–30K tokens for complex reasoning tasks, with a steeper drop past 50K. For simpler extraction tasks the threshold is higher — around 80–100K. A practical monitoring rule: if your average context exceeds 40K tokens per request and you're seeing inconsistent output quality, context size is the first variable to investigate. The $900/month calculation in the article assumes 100K tokens — most production agents can cut that by 60–70% through selective RAG retrieval without measurable quality loss.

How does lazy tool loading in Claude Code achieve 85% token savings, and what is the actual mechanism?

Without lazy loading, every MCP server's full tool schema is injected into the system prompt on every request — 10 servers with 5 tools each at ~200 tokens per tool description equals 10,000 tokens of overhead per call, regardless of which tools actually get used. Tool Search defers schema injection: the agent first sends a semantic search query to find relevant tool names (~50 tokens), then loads only the matching tool descriptions (~400 tokens for 2 tools). The 85% savings comes from eliminating the full schema dump for 8–9 unused tools per typical request.

When should you use manual context compaction versus automatic, and what information is typically lost?

Manual compaction at logical breakpoints (end of a feature, after a passing test suite) is safer because you control what the summary captures. Automatic compaction triggers on window fill and summarizes whatever is current — which may include half-finished reasoning, temporary debugging state, or contradictory instructions from mid-session pivots. The most common loss is architectural decisions made conversationally: "let's not use Redux here because X" survives a manual summary but gets dropped by automatic compaction which treats it as transient chat rather than binding constraint.

I Built a Lie Detector for AI Coding Agents

Roman Belov — Mon, 09 Mar 2026 13:59:51 +0000

The problem nobody talks about

AI coding agents lie. Not on purpose - they hallucinate.

Claude Code tells you "All tests pass!" when tests were never executed. It says "I updated the file" when the content is byte-for-byte identical. It sneaks in git commit --no-verify to skip the hooks that would catch its mistakes.

This isn't rare. It's a documented bug and it hits every serious Claude Code user. System prompts don't help - the agent just ignores them when it "decides" something is done.

I spent a couple weeks building a fix.

Don't ask the agent to be honest - verify it

That's the whole idea. Claude Code has a hooks API - before and after every tool call, it can run your scripts. Those scripts inspect what actually happened and block the agent if the results don't match the claims.

Agent claims: "I updated utils.ts"
    |
[PostToolUse hook]
    |
Compare SHA256 before/after -> IDENTICAL
    |
BLOCKED: "File was not actually modified. Checksum unchanged."

Can't argue with a checksum. This isn't a prompt the agent can ignore. It's a gate.

Six hooks, zero fluff

Hook	What it catches
Dangerous command blocker	`--no-verify`, `--force push`; warns on `reset --hard`, `clean -f`
Pre-commit test runner	Auto-detects your framework, runs tests before every commit
File checksum recorder	Saves SHA256 before file edit
Exit code verifier	Command failed (exit 1) but agent claims success
Phantom edit detector	File unchanged after a claimed "edit"
Commit verification reminder	Makes the agent prove the fix works before claiming "done"

Two days of real use

I ran TruthGuard on a production Flutter project:

5 commits blocked - agent kept trying to commit with failing tests
3 dangerous commands caught - 2x git push --force, 1x git commit --no-verify
0 false positives

The pre-commit test hook alone stopped me from shipping broken code five times in two days. Five times.

Pre-commit testing is the killer feature

When Claude runs git commit, TruthGuard intercepts it. Detects your project type, runs the right test command, and blocks the commit if tests fail. Simple as that.

# Auto-detection:
# pubspec.yaml     -> flutter test
# package.json     -> npm test
# Cargo.toml       -> cargo test
# go.mod           -> go test ./...
# pyproject.toml   -> python -m pytest

Override if you want:

# .truthguard.yml
test_command: "npm run test:unit"
skip_on_no_tests: false

The subtler problem: wrong fixes

After building the basic hooks, I ran into something trickier. Claude makes real changes, tests pass, but the fix doesn't actually solve the original problem. It genuinely thinks it's done.

So I added a post-commit reminder. After every successful commit:

"You just committed code. STOP and verify: did you actually confirm the fix works?"

A nudge, basically. But it makes the agent pause instead of rushing to "Done."

Install

npx truthguard install
cd your-project
npx truthguard init

Copies scripts to ~/.truthguard/, adds hooks to .claude/settings.json. Restart Claude Code and that's it.

Homebrew works too:

brew tap spyrae/truthguard && brew install truthguard

Agent-agnostic

Scripts read JSON from stdin, write JSON to stdout. Same scripts power both Claude Code and Gemini CLI. Supporting another agent means writing a config file, not rewriting hooks.

What's next

This is the free local-only version. No backend, no telemetry, everything on your machine.

Some ideas I'm thinking about:

A second LLM that checks whether the diff actually solves the described problem
Team dashboard with honesty stats
VS Code extension for Cursor and Copilot users

Links

GitHub: github.com/spyrae/truthguard
npm: npmjs.com/package/truthguard

If your agent lies in ways I haven't covered - open an issue and I'll write a hook for it.