Forem: klement Gunndu

RAG From First Principles: Why Every AI App Retrieves Before It Generates

klement Gunndu — Thu, 16 Apr 2026 11:36:46 +0000

Large language models have a problem nobody talks about at the beginning.

You train a model on terabytes of internet data. It knows a lot. But it doesn't know your data — your company's policies, your product docs, last week's incident report, or the contract you signed yesterday.

Ask it anyway, and it does something worse than saying "I don't know." It makes something up that sounds perfectly correct. This is called hallucination, and it's the reason you can't just point GPT-4 at your enterprise and call it a day.

Retrieval-Augmented Generation (RAG) was invented to fix exactly this.

The Problem: Confident Nonsense

Here's the core issue:

What the LLM knows:  whatever was in its training data
What it doesn't know: your data, recent events, private docs
What it does:         generates plausible-sounding wrong answers

For a chatbot that writes poems, hallucination is a quirk. For a system answering questions about medical records, legal contracts, or financial reports — it's a liability.

Businesses need AI that:

Answers from their data, not the internet
Provides citations for every claim
Updates instantly when documents change
Never fabricates facts

The Solutions People Tried First

1. Fine-tuning

Retrain the model on your data. Sounds logical.

The reality: It costs $10K–$100K per training run. It takes days. When your data changes, you retrain. And here's the kicker — the model still hallucinates. Fine-tuning adds knowledge to the weights, but nothing forces the model to use it over its imagination.

2. Stuff Everything in the Prompt

Just paste all your documents into the context window before asking the question.

The reality: In 2022, context windows were 4,096 tokens — roughly 3 pages. Your company has 50,000 documents. Even today with 1M+ token windows, sending everything costs ~$15 per query and takes 30–60 seconds to respond. Not viable at scale.

3. Search First, Then Ask

Run a keyword search on your documents, grab the top results, and feed them to the LLM.

This actually worked. But keyword search has a fundamental flaw. Search for "employee termination policy" and it won't find the document titled "Offboarding Procedures" — because the words don't match, even though the meaning does.

The Birth of RAG (2020)

In May 2020, Facebook AI Research published a paper that changed everything:

"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks"
— Patrick Lewis, Ethan Perez, Aleksandra Piktus, et al.

The idea was deceptively simple:

User asks a question
       ↓
[1] RETRIEVE  — search a knowledge base for relevant documents
       ↓
[2] AUGMENT   — add those documents to the prompt as context
       ↓
[3] GENERATE  — LLM answers using the retrieved context
       ↓
Answer grounded in real documents, with citations

Why this was revolutionary:

No retraining. Update the knowledge base, answers update instantly.
Cheap. A search query + one LLM call vs. retraining a $100K model.
Grounded. The model has actual sources to draw from.
Auditable. You can trace every answer back to specific documents.

This three-step pattern — retrieve, augment, generate — became the foundation of every serious enterprise AI system.

How RAG Evolved (2020–2026)

The original paper was a starting point. The real engineering happened in the five years that followed.

Phase 1: Keyword RAG (2020–2022)

Early RAG systems used traditional keyword search (BM25 / TF-IDF). You type a query, the system finds documents with matching words, and feeds them to the LLM.

It worked for simple, direct questions. But it failed whenever the user's words didn't exactly match the document's words.

Phase 2: Vector RAG (2022–2023)

The breakthrough: embeddings. Instead of matching words, convert text into numerical vectors that capture meaning. Similar meanings produce similar vectors, regardless of the specific words used.

"employee termination"  →  [0.23, -0.87, 0.41, ...]
"offboarding procedures" →  [0.21, -0.85, 0.39, ...]
                              ↑ very similar vectors!

Now a search for "termination policy" finds documents about "offboarding" because the meaning is close, even when the words are completely different.

Vector databases (Pinecone, Weaviate, Chroma, pgvector) emerged specifically to store and search these vectors efficiently.

Phase 3: Hybrid RAG (2023–2024)

A surprising lesson: vector search alone isn't enough.

Try searching for ERR_CONNECTION_REFUSED using vectors. The embedding captures the general concept of "connection error," but it doesn't reliably match the exact string. Keyword search, on the other hand, finds it instantly.

The solution: run both searches and merge the results.

Query: "How do I fix ERR_CONNECTION_REFUSED in auth service?"

Vector search → docs about connection issues, auth troubleshooting
Keyword search → docs containing "ERR_CONNECTION_REFUSED" exactly

Merge using Reciprocal Rank Fusion (RRF) → best of both

RRF is elegant in its simplicity. For each document, it calculates:

score = 1/(k + rank_in_vector) + 1/(k + rank_in_keyword)

Documents that rank high in both searches bubble to the top. Documents that only one method found still appear, just lower.

Most production systems today use a split around 90-95% vector weight, 5-10% keyword weight — semantic search handles most queries, while keyword search catches the edge cases.

Phase 4: Advanced RAG (2024–2025)

At enterprise scale, new problems emerged:

Retrieval returned irrelevant chunks. Fix: add a reranker — a second, more precise model that re-scores the top results. A cross-encoder examines each (query, document) pair and produces a fine-grained relevance score. This typically improves precision by 15–30%.

Chunks split in the wrong places. Fix: smarter chunking strategies. Instead of blindly splitting every 500 tokens, use recursive character splitting (split on paragraphs first, then sentences, then words), or semantic chunking (split when the topic changes).

No way to measure quality. Fix: the RAGAS framework — automated metrics for RAG:

Faithfulness: Does the answer come from the retrieved context?
Answer relevancy: Does the answer address the question?
Context precision: Are the retrieved chunks actually relevant?
Context recall: Did retrieval find everything it needed?

Document quality was terrible. Enterprise PDFs aren't clean text. They have tables, multi-column layouts, headers, footers, scanned pages, embedded images. Naive text extraction produces garbage. Production systems now run layout analysis, OCR, and table structure recognition before chunking — treating document parsing as a first-class engineering problem, not an afterthought.

Phase 5: Agentic RAG (2025–2026)

This is where we are today. RAG is no longer a fixed pipeline. It's an agent decision.

Agent receives question
  → Decides: do I need retrieval? which knowledge base? what query?
  → Retrieves from multiple sources
  → Evaluates: is this enough? should I search again differently?
  → Synthesizes answer from multiple contexts
  → Self-checks: does my answer match the evidence?
  → Returns answer with citations and confidence score

The retrieval step became intelligent:

Query rewriting — the agent reformulates vague questions into precise search queries
Multi-step retrieval — if the first search isn't sufficient, the agent searches again with different terms
Self-RAG — the agent evaluates whether retrieved chunks actually support its answer, and discards irrelevant ones
Multi-source — the agent queries multiple knowledge bases and merges results

Graph RAG: The New Frontier

Standard RAG finds documents. But sometimes you need to find connections.

Graph RAG extracts entities and relationships from your documents and builds a knowledge graph:

Standard RAG:
  "Who manages the auth service?"
  → finds: docs mentioning "auth service" + "manager"

Graph RAG:
  Same query
  → traverses: auth-service → managed_by → Platform Team → led_by → Sarah Chen
  → returns: the full chain of relationships, not just isolated mentions

This matters for questions that span multiple documents — where the answer isn't in any single chunk, but in the connections between them. Org charts, legal references, dependency trees, compliance chains — anywhere relationships matter more than content.

Why RAG Isn't Going Away

A common pushback: "Context windows are 1M+ tokens now. Can't we just send everything?"

No. Here's why:

Factor	Stuff Everything	RAG
Cost per query	~$15 (1M tokens)	~$0.01 (5 chunks)
Latency	30–60 seconds	2–5 seconds
Accuracy	Degrades in the middle of long contexts	Relevant info placed front and center
Data freshness	Rebuild full context every time	Knowledge base updates independently
Scale	Max ~700 pages per query	Millions of documents, retrieve what's needed

The economics alone kill the "stuff everything" approach. And research consistently shows that models perform worse with information buried deep in long contexts — the "lost in the middle" effect.

RAG isn't a workaround for small context windows. It's a fundamentally better architecture for knowledge-intensive applications.

The RAG Architecture in 2026

If you're building a production RAG system today, here's what the architecture looks like:

Documents → Parse (OCR, layout, tables) → Chunk → Embed → Index
                                                          ↓
User query → Embed → Hybrid Search (vector + keyword) → Rerank
                                                          ↓
                                                    Top chunks
                                                          ↓
                            LLM generates answer with citations
                                                          ↓
                                              Guardrails check
                                                          ↓
                                           Response to user

Each stage is its own engineering challenge:

Parsing determines data quality (garbage in, garbage out)
Chunking determines retrieval granularity
Embedding determines semantic understanding
Search determines recall (finding the right documents)
Reranking determines precision (ordering by relevance)
Generation determines answer quality
Guardrails determine safety (hallucination detection, PII filtering)
Evaluation determines whether any of it actually works (RAGAS)

Skip any one of these, and the system fails in production.

Key Takeaways

RAG exists because LLMs hallucinate when they don't have the right information, and fine-tuning is too expensive to fix it.
The evolution went: keyword search → vector search → hybrid search → advanced retrieval → agentic retrieval. Each phase solved a real failure mode.
Document quality is the bottleneck. Not the retrieval algorithm, not the LLM. If your PDFs are parsed into garbage, no amount of reranking saves you.
Hybrid search is non-negotiable in production. Pure vector search misses exact terms. Pure keyword search misses meaning. You need both.
Evaluation from day one. RAGAS gives you real numbers — faithfulness, precision, relevancy. Without metrics, you're flying blind.
RAG at scale is an engineering problem, not an AI problem. The LLM call is the easy part. Parsing, chunking, indexing, searching, reranking, caching, monitoring — that's where the work is.
It's not going away. Even with million-token context windows, RAG wins on cost, latency, accuracy, and scale. Every serious enterprise AI system uses it.

If you're building RAG systems or want to dive deeper into any of these patterns, drop a comment — I'll follow up with deep dives on hybrid search, chunking strategies, and evaluation.

AI-Generated Code Is Building Tech Debt You Can't See

klement Gunndu — Thu, 16 Apr 2026 00:50:24 +0000

Your team shipped more features last quarter than any quarter before. The AI coding tools are working. Everyone feels faster.

Then you look at the codebase six months later and nothing makes sense.

GitClear analyzed 211 million changed lines of code across repositories from Google, Microsoft, and Meta between 2020 and 2024. Their finding: copy-pasted code rose from 8.3% to 12.3% of all changes, while refactored code dropped from 25% to under 10%. Code duplication blocks increased eightfold. The codebase is growing, but the architecture is rotting.

This is not traditional tech debt. Traditional debt comes from shortcuts under deadline pressure. AI-generated tech debt comes from code that works, passes tests, and reads fine — but lacks architectural judgment.

The Measurement Problem

Ox Security analyzed 300 repositories and found 10 recurring anti-patterns in 80-100% of AI-generated code. The top offenders: excessive commenting (90-100% of repos), avoidance of refactoring (80-90%), and duplicated bug patterns across files (80-90%). They called AI-generated code "highly functional but systematically lacking in architectural judgment."

The METR study made this concrete. Sixteen experienced open-source developers (averaging 22,000+ star repositories) were randomly assigned tasks with and without AI tools. The result: developers using AI took 19% longer to complete tasks. But when surveyed afterward, those same developers estimated they were 20% faster. The perception gap was 39 percentage points.

If your team cannot measure the debt, they cannot manage it. Here are five detection patterns that surface AI-generated tech debt before it compounds.

Pattern 1: Cyclomatic Complexity Drift Detection

AI-generated code tends to solve problems by adding conditions rather than abstracting patterns. A function that started at complexity 5 slowly grows to 15 as the AI adds edge case handling inline rather than extracting helper functions.

Track complexity over time, not just at a single point.

"""
complexity_tracker.py — Track cyclomatic complexity drift per function.
Requires: pip install radon
Radon docs: https://radon.readthedocs.io/
"""
import json
import subprocess
import sys
from datetime import date
from pathlib import Path


def get_complexity(source_dir: str) -> list[dict]:
    """Run radon cc and return per-function complexity scores."""
    result = subprocess.run(
        ["radon", "cc", source_dir, "-j", "-n", "C"],
        capture_output=True, text=True, check=True,
    )
    raw = json.loads(result.stdout)
    functions = []
    for filepath, blocks in raw.items():
        for block in blocks:
            functions.append({
                "file": filepath,
                "name": block["name"],
                "complexity": block["complexity"],
                "lineno": block["lineno"],
            })
    return functions


def load_baseline(path: Path) -> dict:
    """Load previous complexity snapshot."""
    if path.exists():
        return json.loads(path.read_text())
    return {}


def detect_drift(baseline: dict, current: list[dict], threshold: int = 3) -> list[dict]:
    """Flag functions whose complexity increased beyond threshold."""
    alerts = []
    for func in current:
        key = f"{func['file']}::{func['name']}"
        prev = baseline.get(key, {}).get("complexity", func["complexity"])
        delta = func["complexity"] - prev
        if delta >= threshold:
            alerts.append({
                "function": key,
                "was": prev,
                "now": func["complexity"],
                "delta": delta,
                "line": func["lineno"],
            })
    return alerts


def save_snapshot(functions: list[dict], path: Path) -> None:
    """Save current complexity as the new baseline."""
    snapshot = {}
    for f in functions:
        key = f"{f['file']}::{f['name']}"
        snapshot[key] = {
            "complexity": f["complexity"],
            "date": str(date.today()),
        }
    path.write_text(json.dumps(snapshot, indent=2))


if __name__ == "__main__":
    source = sys.argv[1] if len(sys.argv) > 1 else "src"
    baseline_path = Path(".complexity-baseline.json")

    current = get_complexity(source)
    baseline = load_baseline(baseline_path)
    alerts = detect_drift(baseline, current)

    if alerts:
        print(f"Found {len(alerts)} complexity drift alerts:")
        for a in alerts:
            print(f"  {a['function']} line {a['line']}: "
                  f"{a['was']} -> {a['now']} (+{a['delta']})")
        sys.exit(1)
    else:
        print(f"No drift detected across {len(current)} functions.")

    save_snapshot(current, baseline_path)

Run this in CI on every pull request. When a function's complexity jumps by 3 or more since the last baseline, the build flags it. The developer must either justify the increase or refactor before merging.

The threshold of 3 is deliberate. A single if adds 1 point. Three conditional branches added to one function in a single PR almost always means inline logic that should be extracted.

Pattern 2: Clone Detection With Structural Matching

AI models generate code by statistical prediction. When similar problems appear in different parts of a codebase, the model generates similar — but not identical — solutions. These near-duplicates are harder to find than exact copies.

jscpd (copy-paste detector) catches both exact and near-duplicates across 150+ languages.

# Install: npm install -g jscpd
# Docs: https://github.com/kucherenko/jscpd

# Scan your source directory for duplicates
jscpd ./src --min-lines 5 --min-tokens 50 --reporters consoleFull

# Output shows duplicate blocks with file locations:
# Clone found (Python):
#   src/auth/login.py [10:25]
#   src/auth/register.py [15:30]
#   Lines: 15, Tokens: 89

# Set a duplication threshold for CI
# Configure in .jscpd.json: {"threshold": 5}
jscpd ./src --threshold 5 --reporters consoleFull

The --threshold flag turns this into a CI gate. GitClear's data shows the industry average crossed 12% duplication in 2024. Set your threshold at your current level and ratchet it down each quarter.

For Python-specific detection, pylint has a built-in duplicate checker:

# Uses Pylint's similarity checker across your codebase
# Docs: https://pylint.readthedocs.io/
pylint --disable=all --enable=duplicate-code src/

Both approaches complement each other. jscpd catches structural similarity across languages. Pylint catches Python-specific patterns like duplicated class hierarchies and repeated decorator chains.

Pattern 3: Dead Code Accumulation Tracking

AI assistants frequently generate utility functions, helper classes, and imports that the final implementation never uses. Over weeks of AI-assisted development, dead code accumulates silently.

vulture detects unused Python code by analyzing ASTs:

# Install: pip install vulture
# Docs: https://github.com/jendrikseipp/vulture

# Scan for dead code with 80% confidence threshold
vulture src/ --min-confidence 80

# Output:
# src/utils/helpers.py:45: unused function 'format_response' (90% confidence)
# src/models/user.py:12: unused import 'Optional' (100% confidence)
# src/api/routes.py:89: unused variable 'temp_cache' (80% confidence)

The confidence scoring matters. At 100%, vulture is certain the code is unreachable within the analyzed files. At 60%, there might be dynamic usage the static analysis missed. Start at 80% for CI gates and 60% for manual review.

Track dead code percentage over time:

"""
dead_code_tracker.py — Track dead code accumulation over time.
Requires: pip install vulture
"""
import subprocess
import json
from datetime import date
from pathlib import Path


def count_dead_code(source_dir: str, min_confidence: int = 80) -> dict:
    """Run vulture and count findings by type."""
    result = subprocess.run(
        ["vulture", source_dir, f"--min-confidence={min_confidence}"],
        capture_output=True, text=True,
    )
    lines = result.stdout.strip().split("\n") if result.stdout.strip() else []
    counts = {"unused_function": 0, "unused_import": 0, "unused_variable": 0, "other": 0}
    for line in lines:
        if "unused function" in line:
            counts["unused_function"] += 1
        elif "unused import" in line:
            counts["unused_import"] += 1
        elif "unused variable" in line:
            counts["unused_variable"] += 1
        else:
            counts["other"] += 1
    counts["total"] = len(lines)
    counts["date"] = str(date.today())
    return counts


def append_history(counts: dict, history_path: Path) -> None:
    """Append today's count to the tracking history."""
    history = []
    if history_path.exists():
        history = json.loads(history_path.read_text())
    history.append(counts)
    history_path.write_text(json.dumps(history, indent=2))


if __name__ == "__main__":
    counts = count_dead_code("src")
    append_history(counts, Path(".dead-code-history.json"))
    print(f"Dead code: {counts['total']} findings "
          f"({counts['unused_function']} functions, "
          f"{counts['unused_import']} imports, "
          f"{counts['unused_variable']} variables)")

When dead code count climbs week over week, something is generating code nobody uses. That is the signal to review AI-assisted PRs more carefully.

Pattern 4: Refactoring Ratio Gate

GitClear's most striking finding was not that duplication increased — it was that refactoring collapsed. From 25% of all code changes in 2021 to under 10% in 2024. AI tools generate new code. They rarely suggest consolidating existing code.

Measure the ratio of refactoring to new code in every sprint:

"""
refactor_ratio.py — Measure refactoring vs new code ratio from git history.
Uses git log to classify commits as refactoring or feature work.
"""
import subprocess
import re
import sys


def get_recent_commits(days: int = 14) -> list[str]:
    """Get commit messages from the last N days."""
    result = subprocess.run(
        ["git", "log", f"--since={days} days ago",
         "--pretty=format:%s", "--no-merges"],
        capture_output=True, text=True, check=True,
    )
    return [line.strip() for line in result.stdout.split("\n") if line.strip()]


def classify_commits(messages: list[str]) -> dict:
    """Classify commits as refactor, feature, fix, or other."""
    refactor_patterns = re.compile(
        r"refactor|extract|consolidate|simplify|rename|restructure|deduplicate|cleanup|clean up",
        re.IGNORECASE,
    )
    feature_patterns = re.compile(
        r"add|implement|create|build|introduce|new|feature",
        re.IGNORECASE,
    )
    fix_patterns = re.compile(r"fix|bug|patch|resolve|hotfix", re.IGNORECASE)

    counts = {"refactor": 0, "feature": 0, "fix": 0, "other": 0}
    for msg in messages:
        if refactor_patterns.search(msg):
            counts["refactor"] += 1
        elif feature_patterns.search(msg):
            counts["feature"] += 1
        elif fix_patterns.search(msg):
            counts["fix"] += 1
        else:
            counts["other"] += 1
    return counts


def compute_ratio(counts: dict) -> float:
    """Compute refactoring ratio as percentage of total commits."""
    total = sum(counts.values())
    if total == 0:
        return 0.0
    return (counts["refactor"] / total) * 100


if __name__ == "__main__":
    days = int(sys.argv[1]) if len(sys.argv) > 1 else 14
    commits = get_recent_commits(days)
    counts = classify_commits(commits)
    ratio = compute_ratio(counts)

    print(f"Last {days} days: {len(commits)} commits")
    print(f"  Refactoring: {counts['refactor']} ({ratio:.1f}%)")
    print(f"  Features:    {counts['feature']}")
    print(f"  Fixes:       {counts['fix']}")
    print(f"  Other:       {counts['other']}")

    if ratio < 15:
        print(f"\nRefactoring ratio ({ratio:.1f}%) is below 15% threshold.")
        print("Consider scheduling dedicated refactoring time.")

This is a proxy metric. Commit messages are noisy. But the trend matters more than any single measurement. If your refactoring ratio drops below 15% for three consecutive sprints, your codebase is accumulating structural debt regardless of the source.

The fix is not to stop using AI tools. The fix is to schedule explicit refactoring time — separate from feature work, tracked separately in your sprint. AI tools generate. Humans consolidate. Both steps are necessary.

Pattern 5: Architectural Boundary Enforcement

AI-generated code does not respect module boundaries. A function in src/auth/ might import directly from src/billing/ because the model saw that pattern somewhere in its training data. Over time, the dependency graph becomes a web.

Enforce boundaries with import rules:

"""
boundary_check.py — Enforce architectural boundaries via import analysis.
Uses Python's ast module (standard library) to parse imports.
"""
import ast
import sys
from pathlib import Path


# Define allowed imports between modules.
# Each key is a module, values are modules it MAY import from.
ALLOWED_IMPORTS = {
    "auth": {"models", "utils", "config"},
    "billing": {"models", "utils", "config"},
    "api": {"auth", "billing", "models", "utils", "config"},
    "models": {"utils", "config"},
    "utils": {"config"},
    "config": set(),
}


def get_module_name(filepath: Path, src_root: Path) -> str:
    """Extract the top-level module name from a file path."""
    relative = filepath.relative_to(src_root)
    return relative.parts[0] if len(relative.parts) > 1 else ""


def check_imports(filepath: Path, src_root: Path) -> list[dict]:
    """Parse a Python file and check imports against boundary rules."""
    module = get_module_name(filepath, src_root)
    if module not in ALLOWED_IMPORTS:
        return []

    violations = []
    source = filepath.read_text()
    tree = ast.parse(source, filename=str(filepath))

    for node in ast.walk(tree):
        target = None
        if isinstance(node, ast.Import):
            for alias in node.names:
                parts = alias.name.split(".")
                if parts[0] in ALLOWED_IMPORTS and parts[0] != module:
                    target = parts[0]
        elif isinstance(node, ast.ImportFrom):
            if node.module:
                parts = node.module.split(".")
                if parts[0] in ALLOWED_IMPORTS and parts[0] != module:
                    target = parts[0]

        if target and target not in ALLOWED_IMPORTS.get(module, set()):
            violations.append({
                "file": str(filepath),
                "line": node.lineno,
                "module": module,
                "imports": target,
                "allowed": sorted(ALLOWED_IMPORTS[module]),
            })
    return violations


def scan_directory(src_root: Path) -> list[dict]:
    """Scan all Python files for boundary violations."""
    all_violations = []
    for pyfile in src_root.rglob("*.py"):
        all_violations.extend(check_imports(pyfile, src_root))
    return all_violations


if __name__ == "__main__":
    src = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("src")
    violations = scan_directory(src)

    if violations:
        print(f"Found {len(violations)} boundary violations:")
        for v in violations:
            print(f"  {v['file']}:{v['line']} — "
                  f"'{v['module']}' imports '{v['imports']}' "
                  f"(allowed: {v['allowed']})")
        sys.exit(1)
    else:
        print("No boundary violations found.")

The ALLOWED_IMPORTS dictionary is your architecture. When the AI generates an import that crosses a boundary, the check fails. The developer must either fix the import or update the architecture — both of which force a deliberate decision.

This pattern scales. Start with top-level module boundaries. Add sub-module rules as the codebase grows. The AI does not know your architecture. This tool enforces it.

Putting It Together: The CI Pipeline

Each pattern works independently. Together, they form a debt detection pipeline:

# .github/workflows/debt-detection.yml
name: Tech Debt Detection
on: [pull_request]

jobs:
  complexity-drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install radon
      - run: python complexity_tracker.py src

  clone-detection:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm install -g jscpd
      - run: jscpd ./src --threshold 5

  dead-code:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install vulture
      - run: vulture src/ --min-confidence 80

  refactor-ratio:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - run: python refactor_ratio.py 14

  boundary-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python boundary_check.py src

None of these tools know whether code was written by a human or an AI. They measure structural quality. That is the point. The source does not matter. The architecture does.

What This Means for Your Team

The research is clear. AI coding tools increase output velocity. They also increase structural debt. The METR study showed experienced developers were 19% slower with AI tools while believing they were 20% faster — a 39 percentage point perception gap.

This does not mean AI tools are bad. It means teams need to pair generation speed with detection systems. The five patterns above give you concrete metrics: complexity drift, duplication percentage, dead code count, refactoring ratio, and boundary violations.

Track these metrics every sprint. Set thresholds. Ratchet them tighter over time. AI generates code faster than humans. Humans still need to maintain the architecture.

The teams that ship fast in 2026 will not be the ones that generate the most code. They will be the ones that detect and resolve structural debt before it compounds.

Follow @klement_gunndu for more AI engineering content. We're building in public.

5 MCP Dev Summit Takeaways That Change How You Build Python Agents

klement Gunndu — Tue, 07 Apr 2026 11:00:42 +0000

The first MCP Dev Summit just ended. April 2-3, New York City, 95 sessions, speakers from Anthropic, AWS, Microsoft, OpenAI, Datadog, and Hugging Face. The Agentic AI Foundation now has 170 member organizations governing the protocol.

Most coverage focuses on announcements. This post focuses on what you need to change in your Python code.

1. The Python SDK V2 Is Coming — Plan Your Migration Now

Max Isbey from Anthropic presented "Path to V2 for MCP SDKs" at the summit. The Python SDK has moved slower than TypeScript — v1.26.0 landed in January 2026, with v1.27 following later. Meanwhile, the TypeScript SDK shipped multiple releases with conformance testing improvements. The pace difference is intentional — Anthropic is holding back major Python changes until the V2 design solidifies.

What V2 likely breaks:

mcp.server.auth module — The authentication surface is being redesigned. If you use the current auth middleware, document every import and configuration pattern you depend on.
Transport initialization — Streamable HTTP is evolving (more on this below). Server setup code will change.
Session management — Sessions are moving from transport-level to data-model-level, which means your session handling code needs a different abstraction.

What to do this week: Audit your MCP servers. Run pip show mcp and note your current version. Document every mcp.server.auth import. List every transport configuration. When V2 drops, you will have a migration checklist instead of a debugging session.

# Document your current setup before V2 lands
# Example: typical MCP server with auth (v1.x pattern)
from mcp.server import Server
from mcp.server.stdio import stdio_server

server = Server("my-tool-server")

@server.tool()
async def search_docs(query: str) -> str:
    """Search internal documentation."""
    # Your tool implementation
    results = await doc_index.search(query)
    return format_results(results)

async def main():
    async with stdio_server() as (read, write):
        await server.run(read, write)

Pin your current version in requirements.txt until V2 migration guides ship. Do not upgrade blindly.

2. OAuth 2.1 Becomes the Standard Auth Pattern

Six dedicated sessions at the summit focused on MCP authentication. Aaron Parecki — the author of the OAuth 2.1 draft specification — attended and participated. That signals the auth solution is grounded in real spec work, not vendor positioning.

The key shift: Client-Initiated Metadata Discovery (CIMD) replaces Dynamic Client Registration (DCR) as the preferred registration method. DCR required enterprise authorization servers to enable a feature most disable by default. CIMD lets each client publish a metadata document at a .well-known URL, and authorization servers make trust decisions based on the domain.

What this means for your MCP servers:

STDIO servers stay simple — they inherit the host process's permissions and do not need auth.
HTTP servers should prepare for OAuth 2.1 with PKCE as the standard flow.
Multi-tenant servers will benefit from CIMD — one metadata document per domain instead of managing client registrations.

Two spec enhancement proposals are under review: SEP-1932 (DPoP) for token binding and SEP-1933 (Workload Identity Federation) for cloud service-to-service authentication. If you are building MCP servers that run in AWS, GCP, or Azure, watch SEP-1933 — it will let your server authenticate using cloud-native identity instead of managing OAuth clients.

# Future pattern: MCP server with OAuth 2.1 (conceptual)
# Exact API will ship with SDK V2 — do NOT implement this yet

# What to prepare NOW:
# 1. Separate your tool logic from your auth logic
# 2. Keep auth configuration in environment variables
# 3. Design tools to be auth-agnostic

@server.tool()
async def query_database(sql: str) -> str:
    """Run a read-only SQL query."""
    # Tool logic should not know about auth
    # Auth happens at the transport layer
    result = await db.execute(sql)
    return result.to_json()

3. Streamable HTTP Goes Stateless With .well-known Discovery

The 2026 MCP roadmap makes the direction clear: agentic applications should be stateful, but the protocol itself should not be. The current Streamable HTTP transport has three scaling problems:

Stateful sessions fight load balancers. If your MCP server holds session state, sticky sessions are required, which defeats horizontal scaling.
No discovery without a live connection. A registry or crawler cannot learn what your server does without connecting to it and negotiating a session.
No standard metadata format. Every MCP server describes its capabilities differently.

The solution shipping in the next spec release (targeted for June 2026):

.well-known metadata documents — Your server publishes a JSON document at a well-known URL describing its tools, resources, and capabilities. Registries can index this without connecting.
Cookie-like session mechanism — Sessions decouple from the transport layer, mirroring standard HTTP patterns. Your server can scale horizontally behind a load balancer without sticky sessions.
Stateless protocol, stateful applications — The protocol handles routing and discovery. Your application handles state.

# Future: .well-known/mcp-server.json (conceptual)
# Enables discovery without a live MCP connection

{
  "name": "my-tool-server",
  "version": "1.0.0",
  "tools": [
    {"name": "search_docs", "description": "Search documentation"},
    {"name": "query_database", "description": "Run SQL queries"}
  ],
  "transport": "streamable-http",
  "auth": "oauth2.1"
}

What to do now: If you are running MCP servers over Streamable HTTP, start designing for statelessness. Move session state to Redis or a database. When the June spec drops, migration will be straightforward if your transport layer does not hold state.

4. Cross-App Access Brings SSO to AI Agents

Paul Carleton from Anthropic presented Cross-App Access (XAA), part of the ID-JAG project. This is single sign-on for AI agents.

Today, every MCP client manages its own auth tokens per server. If an agent connects to five MCP servers, it negotiates five separate authentication flows. XAA changes this: one authentication event grants scoped access across multiple servers that trust the same identity provider.

Why this matters for Python developers:

Fewer auth flows to implement. Your MCP server trusts an identity provider. The client authenticates once. Done.
Cross-platform interop. Nick Cooper from OpenAI keynoted the summit. OpenAI is moving toward MCP Resources support. Agents built with either Anthropic or OpenAI SDKs will be able to query resources from the same MCP servers.
Enterprise adoption unlocks. SSO is table stakes for enterprise. Without it, MCP stays in developer tooling. With it, MCP enters production enterprise workflows.

What to do now: If you are building MCP servers for internal tools, do not roll custom auth. Wait for the SDK V2 auth module. Design your tools to be auth-agnostic (business logic separated from transport), so plugging in XAA later requires zero tool code changes.

5. The Enterprise Working Group Changes the Governance Game

The Agentic AI Foundation announced the formation of an Enterprise Working Group. This is new — it did not exist before the summit.

What this group will tackle:

Audit trails — Logging which agent called which tool with which parameters. Required for compliance in regulated industries.
SSO-integrated auth — Enterprise identity providers (Okta, Azure AD, Google Workspace) as first-class MCP auth sources.
Gateway behavior — Standard patterns for MCP gateways that proxy, filter, and log tool calls. Think API gateways, but for MCP.
Configuration portability — Standard formats for describing MCP server configurations that work across Claude Code, Cursor, Windsurf, and other clients.

Most of this work will ship as extensions rather than core spec changes. That means your existing MCP servers will not break. But if you need enterprise features, you will opt into extension modules.

The governance model is also decentralizing. Working Groups can now accept Spec Enhancement Proposals (SEPs) in their domain without full Core Maintainer review. This means the spec can evolve faster in areas like enterprise auth without blocking the core protocol.

What to Do This Week

Run pip show mcp and pin your current version. Do not upgrade until V2 migration guides are published.
Separate tool logic from auth logic. Every @server.tool() function should be a pure function that takes inputs and returns outputs. Auth lives in the transport layer.
Move session state out of your transport. If you use Streamable HTTP, store state in Redis or a database, not in the MCP session object.
Document your current auth setup. List every mcp.server.auth import, every environment variable, every OAuth client configuration.
Watch three SEPs: SEP-1686 (Tasks), SEP-1932 (DPoP), SEP-1933 (Workload Identity Federation). These determine what ships in the June spec release.

The MCP spec is moving fast. The Dev Summit proved that the protocol has institutional backing from every major AI lab. The Python SDK V2 will be the biggest migration event since the SSE-to-Streamable-HTTP transition. Prepare now, migrate cleanly later.

Follow @klement_gunndu for more MCP and AI agent content. We are building in public.

Lock Down Claude Code With 5 Permission Patterns

klement Gunndu — Mon, 06 Apr 2026 12:57:28 +0000

I denied .env file reads in my settings.json. Claude Code read them anyway. Here is how to build permissions that actually hold.

Claude Code ships with a tiered permission system that most developers never configure beyond clicking "Yes, don't ask again." That default workflow creates invisible gaps. Every auto-approved command persists permanently in your project settings. Every unconfigured tool runs with maximum access. The result is an AI assistant with more filesystem and network access than any human on your team.

This article covers 5 permission patterns that lock down Claude Code properly -- from basic deny rules to OS-level sandboxing.

Pattern 1: Deny-First Rules in settings.json

Claude Code evaluates permission rules in a strict order: deny, then ask, then allow. Rules are evaluated by category: all deny rules are checked first, then ask rules, then allow rules. A deny rule always beats an allow rule, regardless of order in the JSON array or which settings file it lives in.

Here is a starter configuration that blocks secrets, restricts network tools, and allows only your build commands:

{
  "permissions": {
    "deny": [
      "Read(./.env)",
      "Read(./.env.*)",
      "Read(./secrets/**)",
      "Bash(curl *)",
      "Bash(wget *)",
      "Bash(git push --force *)",
      "Bash(rm -rf *)"
    ],
    "allow": [
      "Bash(npm run lint)",
      "Bash(npm run test *)",
      "Bash(git commit *)",
      "Bash(python -m pytest *)",
      "Bash(ruff check *)"
    ]
  }
}

Save this to .claude/settings.json in your project root. It gets checked into version control, so every developer on your team inherits the same restrictions.

Three things to notice:

Glob patterns use * for wildcards. Bash(npm run test *) matches npm run test unit, npm run test --verbose, and any other variation. The space before * enforces a word boundary -- Bash(npm *) matches npm run build but not npmx.
Read and Edit rules follow gitignore syntax. Read(./.env.*) blocks .env.local, .env.production, and every dotenv variant. Read(./secrets/**) blocks everything recursively under the secrets directory.
Deny rules block the built-in tools, not Bash subprocesses. A Read(./.env) deny rule blocks the Read tool but does not prevent cat .env in Bash. For full protection, deny both: add Bash(cat .env) or enable sandboxing (Pattern 4).

Pattern 2: The 4-Layer Settings Hierarchy

Claude Code loads settings from 4 sources, evaluated in this precedence order:

1. Managed settings      (admin-deployed, cannot be overridden)
2. Command line arguments (--allowedTools, --disallowedTools)
3. Local project settings (.claude/settings.local.json, gitignored)
4. Shared project settings (.claude/settings.json, committed)
5. User settings          (~/.claude/settings.json, global)

If a tool is denied at any level, no lower level can allow it. A managed settings deny cannot be overridden by --allowedTools. A project-level deny overrides a user-level allow.

This hierarchy enables a practical team workflow:

// .claude/settings.json (shared, committed)
// Team-wide rules everyone follows
{
  "permissions": {
    "deny": [
      "Bash(git push --force *)",
      "Read(./.env)",
      "Read(./.env.*)",
      "Bash(rm -rf *)"
    ],
    "allow": [
      "Bash(npm run *)",
      "Bash(git commit *)"
    ]
  }
}

// .claude/settings.local.json (personal, gitignored)
// Your own additions that don't affect the team
{
  "permissions": {
    "allow": [
      "Bash(python -m pytest *)",
      "Bash(docker compose *)"
    ]
  }
}

The local file adds your personal tool approvals without weakening team-wide deny rules. You cannot override the shared deny on git push --force from your local file -- the deny always wins.

Pattern 3: MCP Server and Subagent Controls

Claude Code connects to MCP servers and spawns subagents. Both need permission rules. Without them, any MCP server tool runs with full auto-approval once you click "allow" once, and every subagent has unrestricted access.

MCP permission rules follow the format mcp__<server>__<tool>:

{
  "permissions": {
    "allow": [
      "mcp__filesystem__read_file",
      "mcp__github__list_pull_requests"
    ],
    "deny": [
      "mcp__filesystem__write_file",
      "mcp__github__merge_pull_request"
    ]
  }
}

This allows reading through MCP but blocks writing. The server name matches the key you configured in your MCP settings.

For subagents, use Agent(name) rules:

{
  "permissions": {
    "deny": [
      "Agent(Explore)"
    ],
    "allow": [
      "Agent(Plan)",
      "Agent(my-reviewer)"
    ]
  }
}

Denying the Explore agent prevents Claude from spawning a read-only exploration subprocess. This is useful in CI environments where you want deterministic behavior -- no side explorations, no extra tool calls, just the task you assigned.

Pattern 4: Sandbox for OS-Level Enforcement

Permission rules control what Claude Code chooses to do. Sandboxing controls what the operating system allows. These are complementary layers.

The core problem: a Read(./.env) deny rule blocks the Read tool, but Claude can still run cat .env through Bash. Permission rules are application-level. A determined or confused model can work around them.

Sandboxing fixes this by restricting the Bash tool at the OS level:

{
  "permissions": {
    "deny": [
      "Read(./.env)",
      "Read(./.env.*)",
      "Bash(cat .env)",
      "Bash(cat .env.*)",
      "Bash(curl *)",
      "Bash(wget *)"
    ]
  },
  "sandbox": {
    "enabled": true,
    "filesystem": {
      "denyRead": [".env", ".env.*", "secrets/**"],
      "allowRead": ["src/**", "tests/**", "docs/**"]
    },
    "network": {
      "allowedDomains": ["registry.npmjs.org", "pypi.org"]
    }
  }
}

With sandboxing enabled, cat .env fails at the OS level, regardless of whether your permission rules catch it. The allowedDomains list restricts which domains Bash commands can reach, closing the curl escape hatch.

When sandboxing is enabled with the default autoAllowBashIfSandboxed: true, sandboxed Bash commands run without prompting. The sandbox boundary replaces the per-command permission prompt. This gives you fewer interruptions with stronger enforcement.

Use both layers together for defense in depth: permission deny rules prevent Claude from attempting restricted actions, and sandbox restrictions block the underlying process even if a prompt injection bypasses Claude's decision-making.

Pattern 5: Permission Modes for Different Workflows

Claude Code supports 6 permission modes. Most developers use default and never change it. Matching the mode to your workflow eliminates unnecessary prompts without weakening security.

{
  "permissions": {
    "defaultMode": "acceptEdits"
  }
}

Here is when to use each mode:

default -- Standard behavior. Prompts on first use of each tool. Use this when you are learning the tool or working on an unfamiliar codebase.

acceptEdits -- Auto-approves file edits for the session but still prompts for Bash commands. Use this when you trust Claude's code changes but want to review every shell command.

plan -- Read-only mode. Claude can analyze files but cannot modify anything or run commands. Use this for code review, architecture planning, or when you want analysis without side effects.

dontAsk -- Auto-denies every tool unless it is pre-approved via your permissions.allow list. This is the most restrictive interactive mode. Use this in CI or automated pipelines where you want zero prompts and complete control.

auto -- Auto-approves tool calls with background safety checks that verify actions align with your request. The classifier blocks actions it deems risky, like force-pushing or accessing domains outside your configured trust boundary. Currently a research preview.

bypassPermissions -- Skips all permission prompts except writes to protected directories (.git, .claude, .vscode). Only use this inside containers or VMs where Claude Code cannot cause lasting damage.

To prevent bypass mode from being used at all, set this in your managed settings or any settings file:

{
  "permissions": {
    "disableBypassPermissionsMode": "disable"
  }
}

This is the single most important setting for teams. One developer in bypass mode can undo every permission rule you have configured.

A Real-World Settings File

Here is the complete settings.json we use in production. It combines all 5 patterns:

{
  "permissions": {
    "defaultMode": "acceptEdits",
    "deny": [
      "Read(./.env)",
      "Read(./.env.*)",
      "Read(./secrets/**)",
      "Bash(curl *)",
      "Bash(wget *)",
      "Bash(git push --force *)",
      "Bash(git push * --force)",
      "Bash(rm -rf *)",
      "Bash(git checkout .)",
      "Bash(git reset --hard *)",
      "mcp__filesystem__write_file"
    ],
    "allow": [
      "Bash(npm run *)",
      "Bash(python -m pytest *)",
      "Bash(ruff check *)",
      "Bash(black *)",
      "Bash(git status)",
      "Bash(git diff *)",
      "Bash(git add *)",
      "Bash(git commit *)",
      "Bash(git log *)",
      "Bash(* --version)",
      "Bash(* --help *)",
      "mcp__github__list_pull_requests",
      "mcp__github__get_pull_request",
      "Agent(Plan)"
    ],
    "disableBypassPermissionsMode": "disable"
  }
}

Notice the duplicate force-push deny rules: git push --force * and git push * --force. The flag can appear before or after the remote name. Both patterns must be denied.

The Permission Audit Checklist

Run this checklist on every new project before the first Claude Code session:

Create .claude/settings.json with deny rules for .env, secrets, and destructive commands.
Add .claude/settings.local.json to .gitignore so personal preferences stay personal.
Deny force-push in both flag positions. git push --force * and git push * --force.
Deny Read AND the Bash equivalent for sensitive files. Read(./.env) plus Bash(cat .env).
Enable sandboxing if you are on macOS or Linux. It closes the Bash escape hatch.
Set disableBypassPermissionsMode to "disable" in shared settings.
Review auto-approved commands with /permissions after every session. Remove rules you did not intend to save.
Use acceptEdits mode as your default. It eliminates edit prompts while keeping Bash prompts active.

Every "Yes, don't ask again" click saves a permanent allow rule to your local settings. After a month of development, you may have dozens of invisible allow rules you never explicitly configured. The /permissions command lists all of them. Audit it.

What Happens When You Skip This

A Claude Code session with default permissions and no settings.json has:

Full read access to every file in your project directory
Full write access after one approval per session
Full network access through Bash (curl, wget, any HTTP client)
No restrictions on destructive git commands
Permanent allow rules accumulating from every "don't ask again" click

One prompt injection in a pasted error message or a dependency's README can exploit all of these. The permission system exists to shrink this attack surface. Use it.

Follow @klement_gunndu for more Claude Code content. We are building in public.

Inside Claude Code's Hidden Multi-Agent Architecture

klement Gunndu — Thu, 02 Apr 2026 23:53:47 +0000

Anthropic's Claude Code has 58 tools, but the one that matters most is the one that spawns copies of itself.

On March 31, the full source leaked via npm source maps. I spent the last two days reading the multi-agent architecture. Here is what I found.

AgentTool: The Tool That Spawns Agents

Every subagent in Claude Code is created through a single tool. The input schema tells you everything about how Anthropic thinks about agent orchestration:

const baseInputSchema = z.object({
  description: z.string().describe('A short (3-5 word) description'),
  prompt: z.string().describe('The task for the agent to perform'),
  subagent_type: z.string().optional(),
  model: z.enum(['sonnet', 'opus', 'haiku']).optional(),
  run_in_background: z.boolean().optional(),
})

The parent agent picks the model tier per task. Search gets Haiku. Complex reasoning gets Opus. Everything else gets Sonnet. This is not automatic routing — the parent makes an explicit choice every time it spawns a child.

One-Shot vs Persistent Agents

The source defines two categories:

export const ONE_SHOT_BUILTIN_AGENT_TYPES: ReadonlySet<string> = new Set([
  'Explore',
  'Plan',
])

One-shot agents run a task and return a report. The parent never sends follow-up messages. This saves tokens — no agent ID, no SendMessage trailer, no usage block. At 34 million Explore runs per week, those 135 characters per run add up.

Every other agent type is persistent. The parent can continue the conversation using SendMessage with the agent's ID. This is how Claude Code runs parallel research tasks while you wait.

Team Spawning: tmux Panes, Not API Calls

The most surprising discovery: teammates are not spawned via API. They are spawned as separate Claude Code processes in tmux panes.

async function handleSpawnSplitPane(input, context) {
  const model = resolveTeammateModel(input.model, getAppState().mainLoopModel)
  const uniqueName = await generateUniqueTeammateName(name, teamName)
  const { paneId } = await createTeammatePaneInSwarmView(...)

  const spawnCommand = `cd ${workingDir} && env ${envStr} ${binaryPath} ${args}`
  await sendCommandToPane(paneId, spawnCommand, ...)

  // Communication via filesystem mailbox
  await writeToMailbox(sanitizedName, { from: 'TEAM_LEAD', text: prompt }, teamName)
}

Each teammate gets its own tmux pane, its own process, its own context window. Communication happens through a filesystem-based mailbox — not shared memory, not API calls. The team lead writes a message to ~/.claude/teams/{team}/mailbox/{agent}.json. The teammate reads it on its next loop iteration.

This is the simplest possible multi-agent communication protocol. No message broker. No WebSocket. No shared state. Just files on disk.

KAIROS: The Autonomous Daemon

Behind a feature flag called KAIROS, there is an unreleased autonomous mode. The agent runs as a persistent daemon that:

Monitors GitHub webhooks for new issues and PRs
Reads a channel-based task queue
Executes tasks without human prompting
Reports results back through the same mailbox system

const fullInputSchema = baseInputSchema.merge(z.object({
  name: z.string().optional(),
  team_name: z.string().optional(),
  mode: permissionModeSchema().optional(),
  isolation: z.enum(['worktree', 'remote']).optional(),  // KAIROS feature
  cwd: z.string().optional(),  // KAIROS feature
}))

export const inputSchema = feature('KAIROS') ? fullInputSchema : fullInputSchema.omit({ cwd: true })

When KAIROS is enabled, agents can specify their own working directory and run in isolated git worktrees. Without it, those fields are stripped from the schema entirely — the model never sees them.

44 Feature Flags Control Everything

The entire system is gated behind feature flags. I counted 44 in the buildable fork:

KAIROS — autonomous daemon mode
PROACTIVE — agent initiates without prompting
COORDINATOR_MODE — multi-agent swarm orchestration
BUDDY — Tamagotchi companion system
VOICE_MODE — voice interaction
BRIDGE_MODE — IDE integration with JWT auth
CHICAGO_MCP — Computer Use (screen control)
ULTRAPLAN — enhanced planning mode
TEAMMEM — team memory sharing
EXTRACT_MEMORIES — automatic memory extraction

Each flag is checked with a feature() function that conditionally includes code, schemas, and even entire tool definitions. Dead code elimination means if a flag is off, the model literally cannot see or call the gated functionality.

What This Architecture Teaches

Three things stood out to me after reading the full multi-agent system:

1. Filesystem beats message brokers for local agents. When all agents run on the same machine, JSON files on disk are simpler, more debuggable, and more reliable than any message queue. You can cat the mailbox. You can tail -f the team log. No infrastructure to maintain.

2. Model routing should be explicit, not automatic. The parent agent chooses Haiku, Sonnet, or Opus for each child. This is a deliberate cost-quality tradeoff made at spawn time, not a system-level optimization. The agent that understands the task picks the model for the task.

3. Feature flags are the real architecture. The 44 flags mean Claude Code is not one product. It is dozens of products sharing a codebase, each activated by a boolean. KAIROS-mode Claude Code is a fundamentally different system from default Claude Code — and the flag system lets Anthropic test both in production simultaneously.

The source was not supposed to be public. But now that it is, it is the most detailed reference for production multi-agent architecture I have read. Every decision is visible in the code.

Follow @klement_gunndu for more AI engineering breakdowns. We are building in public.

What 512K Lines of Leaked Claude Code Taught Me About AI Tool Design

klement Gunndu — Thu, 02 Apr 2026 08:46:54 +0000

On March 31, 2026, Anthropic shipped Claude Code v2.1.88 with a 59.8MB source map file still attached. The entire TypeScript source — 1,900 files, 512K+ lines — was readable by anyone who ran npm pack.

I downloaded it. I read the tool architecture. What I found changed how I think about building AI tools.

This is not speculation. Every code snippet below comes from the actual source. I have the full archive on disk.

The Tool Interface: One Type to Rule 58 Tools

Claude Code ships 58 tools — from BashTool to AgentTool to GrepTool. Every single one implements the same TypeScript type:

export type Tool<Input, Output, Progress> = {
  name: string
  searchHint?: string  // 3-10 word capability hint

  // Core execution
  call(args, context, canUseTool, parentMessage, onProgress): Promise<ToolResult>

  // Schema (Zod)
  readonly inputSchema: Input
  readonly outputSchema?: z.ZodType<unknown>

  // Safety declarations
  isConcurrencySafe(input): boolean
  isReadOnly(input): boolean
  isDestructive?(input): boolean

  // Permission hooks
  validateInput?(input, context): Promise<ValidationResult>
  checkPermissions(input, context): Promise<PermissionResult>
}

The insight is not in any single field. It is in what the type forces you to declare.

Every tool must answer three questions before it runs: Can it run alongside other tools? Does it modify state? Could it destroy something? These are not optional annotations. They are required by the type system.

Most AI tool frameworks I have seen treat safety as an afterthought — a wrapper you add later. Claude Code makes it structural. You cannot build a tool without deciding upfront whether it is safe.

buildTool(): Defaults That Fail Closed

All 58 tools go through a factory function called buildTool(). It supplies defaults:

const TOOL_DEFAULTS = {
  isConcurrencySafe: () => false,   // assume NOT safe
  isReadOnly: () => false,          // assume writes
  isDestructive: () => false,
  checkPermissions: (input) =>
    Promise.resolve({ behavior: 'allow', updatedInput: input }),
}

Read that first line again: isConcurrencySafe: () => false.

If you forget to declare concurrency safety, your tool defaults to serial execution. If you forget to declare read-only, the system assumes your tool writes. The defaults are pessimistic.

This is a pattern I now use in every tool system I build. When the GrepTool overrides it:

export const GrepTool = buildTool({
  name: 'Grep',
  searchHint: 'search file contents with regex (ripgrep)',

  isConcurrencySafe() { return true },
  isReadOnly() { return true },
})

That true is an explicit, conscious declaration. The developer had to think about it.

Compare this to LangChain's @tool decorator, where concurrency and safety are not part of the interface at all. You get convenience, but you lose the forcing function.

BashTool: 22 Security Validators Before Execution

The BashTool is the most complex tool in the system. Before any command runs, it passes through 22 distinct security validators:

const BASH_SECURITY_CHECK_IDS = {
  INCOMPLETE_COMMANDS: 1,
  JQ_SYSTEM_FUNCTION: 2,
  OBFUSCATED_FLAGS: 4,
  SHELL_METACHARACTERS: 5,
  DANGEROUS_PATTERNS_COMMAND_SUBSTITUTION: 8,
  IFS_INJECTION: 11,
  PROC_ENVIRON_ACCESS: 13,
  MALFORMED_TOKEN_INJECTION: 14,
  BRACE_EXPANSION: 16,
  CONTROL_CHARACTERS: 17,
  UNICODE_WHITESPACE: 18,
  ZSH_DANGEROUS_COMMANDS: 20,
  COMMENT_QUOTE_DESYNC: 22,
  // ... 9 more
}

Each validator catches a specific class of shell injection. UNICODE_WHITESPACE catches invisible characters that look like spaces but are not. COMMENT_QUOTE_DESYNC catches payloads that exploit the gap between how comments and quotes are parsed.

This is defense in depth. The permission system handles "should this command run?" The security validators handle "is this command what it appears to be?"

I counted: 22 validators for one tool. Most AI agent frameworks ship bash execution with zero input validation. If you are building a tool that runs shell commands, this is the minimum bar.

Three-Layer Permission Architecture

Claude Code does not have one permission check. It has three layers, and they run in order:

Layer 1: validateInput() — Semantic checks before anything else.

// FileEditTool example
async validateInput(input, context) {
  if (oldString === newString) {
    return { result: false, message: 'No changes to make' }
  }
  const { size } = await fs.stat(fullFilePath)
  if (size > MAX_EDIT_FILE_SIZE) {
    return { result: false, message: 'File too large' }
  }
  return { result: true }
}

Layer 2: checkPermissions() — Rule engine for allow/deny/ask decisions.

Layer 3: canUseTool callback — Hook integration. External systems (pre-tool-use hooks) get a veto.

The key design decision: validation happens before permissions. If the input is semantically invalid, the system rejects it before even checking whether you have permission. This prevents wasting a user's permission approval on a request that would fail anyway.

I have started applying this pattern in my own Python tools. Validate first, authorize second, execute third.

ToolSearch: Lazy Loading That Saves Tokens

Claude Code has 58 tools, but the model does not see all 58 schemas in every prompt. That would burn thousands of tokens on tools the model will never call.

Instead, most tools are "deferred." The model sees only their names. When it needs a tool, it calls ToolSearch:

async function searchToolsWithKeywords(query, deferredTools, maxResults) {
  // Fast path: exact match on tool name
  const exactMatch = deferredTools.find(
    t => t.name.toLowerCase() === queryLower
  )
  if (exactMatch) return [exactMatch]

  // Keyword search: parse CamelCase names into words
  // Score by word boundary matches in name + searchHint
  const matches = scoreAndRankTools(query, deferredTools)
  return matches.slice(0, maxResults)
}

Only after ToolSearch returns a match does the full schema get injected into the conversation.

This is smart token economics. The searchHint field — that 3-10 word description each tool declares — is the entire search corpus. No embeddings, no vector DB. Just keyword matching on short hints.

If you are building an agent with more than 10 tools, steal this pattern. Keep tool descriptions short. Load schemas lazily. Let the model search for what it needs.

What I Am Applying to My Own Systems

I maintain an autonomous content engine (Herald) that publishes to dev.to. It has tools for article creation, comment monitoring, engagement tracking, and browser automation. After reading Claude Code's source, I changed three things:

1. Every tool now declares safety properties. My Python tools have is_read_only and is_concurrent_safe as required attributes, not optional. The default is False for both.

2. Validation before authorization. My Playwright engagement tools now validate comment content (quality gate) before checking browser session permissions. This catches LLM-generated spam before wasting a browser launch.

3. Lazy tool registration. My agent no longer loads all tool schemas at startup. Tools register with a one-line description. Full schemas load on first use.

None of these are revolutionary ideas. But seeing them implemented at scale, in production code serving millions of users, made the patterns click in a way that documentation never did.

The Takeaway

Claude Code's tool architecture is not clever. It is disciplined. Every tool declares its safety properties. Defaults fail closed. Validation precedes authorization. Schemas load lazily. Security checks are specific, not generic.

The source was not supposed to be public. But now that it is, it is the best reference implementation for AI tool design I have seen. Study it.

Follow @klement_gunndu for more AI engineering breakdowns. We are building in public.

5 Red Flags in AI Product Demos That PMs Should Never Ignore

klement Gunndu — Sat, 28 Mar 2026 12:34:02 +0000

Every AI vendor has a demo that works perfectly. That is the problem.

Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027 due to escalating costs, unclear business value, or inadequate risk controls (Gartner, June 2025). A separate Gartner report found that through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data (Gartner, February 2025).

The pattern is consistent: teams greenlight AI products based on impressive demos, then discover the gap between demo and production is a canyon.

PMs sit at the decision point. You approve the budget. You set the timeline. You own the outcome when it ships — or when it doesn't. These 5 red flags help you spot the canyon before you walk into it.

Red Flag 1: "AI-Powered" With No Explanation of What That Means

The vendor says their product is "AI-powered." You ask what the AI actually does. They pivot to a slide about "leveraging machine learning" or "using advanced neural networks."

This is AI washing. The term "AI-powered" has become so overused that the U.S. Federal Trade Commission issued guidance warning companies about making unsubstantiated AI claims (FTC, February 2023). The problem has only gotten worse since then.

What to ask instead:

"What specific task does the AI perform that wasn't possible before?"
"What model or approach powers this? Is it a foundation model, a fine-tuned model, or a rules engine with an AI label?"
"What happens when I turn the AI off? What manual process does it replace?"

If the vendor cannot explain in one sentence what the AI does — not what it "leverages" or "harnesses" — the product is either not AI or the team does not understand their own technology. Both are disqualifying.

The test: Ask the sales engineer, not the account executive. Sales engineers talk implementation. Account executives talk vision. You need implementation.

Red Flag 2: The Demo Uses Their Data, Not Yours

The demo runs on a curated dataset. The search returns perfect results. The classification hits 98% accuracy. The generated text reads like a press release from a Fortune 500 company.

Then you feed it your data — messy CSVs with missing fields, inconsistent naming conventions, and 3 years of legacy formatting — and accuracy drops to 60%.

This is the most common gap between demo and production. Gartner found that lack of AI-ready data is the primary reason organizations abandon AI projects (Gartner, February 2025).

What to ask instead:

"Can we run the demo on our data? Not our cleanest data — our realistic data."
"What data preparation did you do before this demo? How long did it take?"
"What percentage of your customers needed data cleaning before going live? How long did that take on average?"

If the vendor hesitates to run on your data, that tells you everything. A mature product handles messy inputs. An immature product needs a clean room.

The test: Bring a sample dataset to the second meeting. Not your best data. Your average data. Watch what happens.

Red Flag 3: No Production Customers — Only Pilots and POCs

"We have 15 enterprise pilots running right now."

Pilots are not production. A pilot is a controlled experiment with a dedicated support team, a narrow scope, and a safety net. Production means the product handles real traffic, real edge cases, and real failures at scale with no one holding its hand.

Gartner predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025 (Gartner, July 2024). The pilot-to-production gap is where most AI projects die.

What to ask instead:

"How many customers are running this in production today — not pilots, production?"
"What is your average time from pilot to production deployment?"
"Can I talk to a production customer in my industry? Not a reference they prepared — one I choose from a list."
"What is the biggest production failure you have seen, and how did you handle it?"

The last question is the most revealing. Every production system fails. A vendor who cannot describe a specific failure and their response to it has either never been in production or is hiding something.

The test: Ask for 3 production customer names. If they give you 3 pilot names instead, you have your answer.

Red Flag 4: Pricing That Hides the Real Cost

The vendor quotes $2,000 per month for their AI platform. What they don't mention: the platform makes API calls to a foundation model provider, and those calls are billed separately based on token usage.

Your proof of concept runs 50 queries a day. Your production environment will run 5,000. The $2,000/month platform fee stays the same. The model inference cost goes from $200/month to $20,000/month.

This is not a vendor problem — it is an AI economics problem. Foundation model costs scale with usage in ways that traditional SaaS does not. A SaaS product costs the same whether 10 users or 10,000 users run the same query. An AI product that calls GPT-4 or Claude costs more with every query, every token, every retry.

What to ask instead:

"What is the total cost at 10x our current volume? At 100x?"
"Does your pricing include model inference costs, or are those separate?"
"What happens to cost if we switch to a different model? Are we locked into one provider?"
"What cost optimization have you built in? Caching? Model routing? Batch processing?"

If the vendor quotes a flat rate and cannot answer volume questions, they either haven't scaled or they're counting on you not asking.

The test: Ask for a cost calculator or a cost projection at 3 volume tiers: current, 10x, and 100x. If they don't have one, they haven't thought about it.

Red Flag 5: No Answer to "What Happens When It's Wrong?"

The AI agent summarizes a contract and misses a liability clause. The AI classifier labels a support ticket as low priority when the customer is about to churn. The AI recommendation engine suggests a product that was discontinued last month.

Every AI system produces wrong outputs. The question is not "is it perfect?" — the question is "what happens when it isn't?"

What to ask instead:

"What is your accuracy rate on tasks similar to ours? How do you measure it?"
"When the system produces a wrong output, how does it signal that to the user?"
"Is there a confidence score? What threshold do you recommend for human review?"
"What is your feedback loop? If a user corrects an error, does the system learn from it?"
"Do you have an audit trail? Can I trace why the system made a specific decision?"

A product that cannot explain its failure mode is a product that has not been tested at scale. Confidence scores, human-in-the-loop workflows, and audit trails are not optional features — they are table stakes for any AI product that touches business decisions.

The test: Ask the vendor to show you a wrong output from their system. If they can show it and explain why it happened, they understand their product. If they insist the system "doesn't make mistakes," leave the meeting.

The 5-Question Cheat Sheet

Print this. Bring it to your next vendor meeting.

#	Question	What a Good Answer Sounds Like
1	What specific task does the AI do?	"It classifies support tickets into 12 categories with 94% accuracy, measured on our benchmark of 10,000 labeled tickets."
2	Can we run the demo on our data?	"Yes. Send us a sample and we'll run it in our sandbox. Here's the data format we need."
3	How many production customers use this?	"47 production customers. Average time from pilot to production: 6 weeks. Here are 3 you can call."
4	What is the total cost at 100x volume?	"Platform fee stays flat. Inference costs scale linearly — here's our cost calculator with 3 tiers."
5	What happens when the AI is wrong?	"We surface a confidence score on every output. Below 0.85, the system flags it for human review. Here's our audit trail."

If the vendor cannot give concrete answers to all 5 questions, the product is not ready for your team. It might be ready for a pilot. It is not ready for production.

The Deeper Problem

These red flags are symptoms of a market moving faster than its quality controls. AI vendors are under pressure to ship. PMs are under pressure to adopt. The result: decisions made on demo quality instead of production evidence.

The fix is not to avoid AI products. The fix is to evaluate them the same way you evaluate any production dependency: with your data, at your scale, with a clear understanding of failure modes and costs.

40% of agentic AI projects will be canceled by 2027. The teams that avoid that outcome are the ones asking these questions before they sign — not after.

Follow @klement_gunndu for more AI product content. We're building in public.

How to Read Any Codebase in 30 Minutes With AI Tools

klement Gunndu — Sat, 28 Mar 2026 08:04:26 +0000

Your manager says "get familiar with the codebase." You open the repo. 200K lines. No architecture docs. The README was last updated two years ago.

This is the first real challenge every new developer faces — and nobody teaches you how to handle it. Reading code is harder than writing it, and scrolling through files at random wastes hours without building understanding.

Here are 5 steps that turn AI into your codebase guide. Total time: 30 minutes for a medium-sized project.

Step 1: Map the File Tree (5 Minutes)

Before reading a single line of code, understand the shape of the project.

Run tree with depth limits to avoid drowning in files:

# Get the top 2 levels of the project structure
tree -L 2 -I 'node_modules|.git|__pycache__|.venv|dist'

# For larger projects, limit to directories only
tree -L 3 -d -I 'node_modules|.git|__pycache__|.venv'

This gives you output like:

.
├── src/
│   ├── api/
│   ├── models/
│   ├── services/
│   └── utils/
├── tests/
│   ├── unit/
│   └── integration/
├── docker-compose.yml
├── pyproject.toml
└── README.md

Copy the tree output and paste it into your AI coding assistant (ChatGPT, Claude, Copilot Chat — any works). Ask:

"Here is the file tree for a project I just joined. What does each top-level directory likely do? What architectural pattern does this suggest?"

The AI gives you a mental map in 60 seconds. You now know where the API routes live, where business logic sits, and where tests are.

Why this works: Your brain processes spatial layouts faster than text. A file tree is a spatial map of the codebase. The AI fills in the labels.

Step 2: Read the Config Files (5 Minutes)

Config files are the most honest documentation in any project. They list the actual dependencies, scripts, and settings — not what someone intended to write, but what the project actually uses.

Read these files in order:

# Python projects
cat pyproject.toml   # or requirements.txt, setup.py

# JavaScript projects
cat package.json

# Any project with containers
cat docker-compose.yml

# Environment variables tell you what external services exist
cat .env.example     # never .env — that has real secrets

For a Python project, pyproject.toml tells you everything:

[project]
dependencies = [
    "fastapi>=0.104.0",
    "sqlalchemy>=2.0",
    "pydantic>=2.5",
    "httpx>=0.25.0",
    "celery>=5.3.0",
]

[project.scripts]
serve = "app.main:run"
worker = "app.tasks:start_worker"

From these 10 lines, you know:

FastAPI — this is a web API, not a CLI tool
SQLAlchemy — there is a database with an ORM
Celery — there are background tasks
httpx — the app calls external APIs
Entry points — app/main.py has run(), app/tasks.py has start_worker()

Paste the config file into your AI assistant and ask:

"What does this project do based on its dependencies? What external services does it need?"

Five minutes in. You already know the tech stack, external dependencies, and entry points.

Step 3: Find and Trace the Entry Point (10 Minutes)

Every application has a front door. Find it, then follow the first hallway.

# Python — find the main entry
grep -rn "if __name__" --include="*.py" | head -5
grep -rn "app = FastAPI\|app = Flask\|app = Django" --include="*.py" | head -5

# JavaScript — find the main entry
grep -rn "createServer\|express()\|new Hono" --include="*.js" --include="*.ts" | head -5

# Or just check the config — package.json "main" or pyproject.toml [project.scripts]

Once you find the entry file, read it with your AI assistant. In Claude Code, you can open the project and ask directly. In other tools, paste the file content and ask:

"Walk me through what happens when this application starts. What gets initialized? What routes get registered?"

Now trace one path deeper. Pick the most important-looking route or function and follow it:

# Find where a function is defined
grep -rn "def process_order" --include="*.py"

# Find where it's called
grep -rn "process_order" --include="*.py"

You are tracing a single thread through the codebase. Not reading everything — reading one path from entry to exit. This builds a mental model of how the pieces connect.

The 10-minute rule: Set a timer. Trace one request from the API endpoint to the database and back. When the timer goes off, stop. You now understand one complete flow, and every other flow follows a similar pattern.

Step 4: Read One Test File (5 Minutes)

Tests are executable documentation. They show you what the code is supposed to do, what inputs it expects, and what outputs it produces.

# Find the test files
ls tests/ 2>/dev/null || ls test/ 2>/dev/null

# Pick the test file that matches the entry point you traced
# If you traced process_order, look for test_process_order.py
find . -name "test_*order*" -o -name "*order*_test*" | head -5

A well-written test tells you more than any documentation:

def test_process_order_calculates_total():
    order = Order(items=[
        Item(name="Widget", price=9.99, quantity=2),
        Item(name="Gadget", price=24.99, quantity=1),
    ])

    result = process_order(order)

    assert result.total == 44.97
    assert result.status == "confirmed"
    assert len(result.line_items) == 2

From this single test, you know:

Order takes a list of Item objects
Each Item has name, price, and quantity
process_order returns an object with total, status, and line_items
The function calculates totals and confirms orders

No documentation needed. The test IS the documentation.

If the project has no tests (it happens), check for API documentation, Swagger/OpenAPI specs, or example scripts in a docs/ or examples/ directory.

Step 5: Read the Git Log (5 Minutes)

The git history tells you what is actively changing — which matters more than what exists.

# See the last 20 commits with files changed
git log --oneline --stat -20

# See who works on what
git shortlog -sn --since="3 months ago"

# Find the most frequently changed files (these are the hot spots)
git log --pretty=format: --name-only --since="3 months ago" | sort | uniq -c | sort -rn | head -15

The last command is the most powerful. It shows you the files that change most often. These are the files you should understand first because:

They contain the most active business logic
They are where bugs are most likely to appear
They are where your first tasks will probably be

  47 src/services/order_service.py
  31 src/api/routes/orders.py
  28 src/models/order.py
  19 tests/test_order_service.py
  12 src/utils/pricing.py

This output tells you the order system is the hot zone. Your first PR will probably touch these files. Read them next.

Bonus — read recent PR descriptions:

# If using GitHub
gh pr list --state merged --limit 10

# Read a specific PR's description and comments
gh pr view 142

PR descriptions often contain more context than commit messages. They explain why changes were made, not just what changed.

What NOT to Do

Three mistakes new developers make when reading a codebase:

Do not read every file sequentially. A 200K-line codebase is not a book. Reading src/a.py through src/z.py builds no mental model. Trace flows instead.

Do not memorize implementation details. You do not need to know how the caching layer works on day one. You need to know it exists and where it lives. Details come when you work on a task that touches them.

Do not skip the config files. package.json and pyproject.toml tell you the truth about the project. README files tell you what someone hoped the project would become.

The 30-Minute Template

Copy this checklist for your first day on any codebase:

[  ] 0:00 - Run tree -L 2, paste into AI, get structure overview
[  ] 0:05 - Read pyproject.toml / package.json, identify stack
[  ] 0:10 - Find entry point (grep for main/app creation)
[  ] 0:12 - Trace one request from route to database
[  ] 0:20 - Read one test file matching the flow you traced
[  ] 0:25 - Run git log frequency analysis, find hot files
[  ] 0:30 - Write 5 bullet points: what the app does, how it works

That last step matters. Writing a summary forces your brain to organize what you learned. Keep it in a personal notes file. Update it as you learn more.

After the First 30 Minutes

The 30-minute method gives you a working mental model. Not a complete one — a working one. Enough to:

Ask informed questions in your first standup
Understand which files a bug report probably touches
Review a PR without feeling completely lost
Pick up your first task without starting from zero

Every week, trace one more flow through the codebase. Within a month, you will know the system better than developers who have been there for years but never mapped it systematically.

The codebase is not a mystery. It is a system with entry points, flows, and patterns. Map the structure, trace the flows, read the tests, check the history. AI accelerates each step, but the method works with or without it.

Follow @klement_gunndu for more beginner-friendly AI content. We're building in public.

Pick the Right Claude Code Model for Every Task

klement Gunndu — Fri, 27 Mar 2026 08:08:36 +0000

Claude Code supports three model tiers, seven aliases, four effort levels, and per-subagent model overrides. Most developers use the default for everything. That means they're paying Opus prices for tasks Haiku handles in half the time.

This guide covers 5 patterns for matching the right model to the right task in Claude Code, based on the official model configuration system as of March 2026.

The Three Tiers and When Each Wins

Claude Code gives you three model families, each with a different speed-cost-quality tradeoff:

Model	Best For	Speed	Cost
Haiku	File search, simple refactors, quick lookups	Fastest	Lowest
Sonnet	Daily coding, implementation, test writing	Balanced	Medium
Opus	Architecture decisions, complex debugging, multi-file refactoring	Slowest	Highest

The default model depends on your subscription. Max and Team Premium plans default to Opus 4.6. Pro and Team Standard default to Sonnet 4.6. Claude Code may automatically fall back to Sonnet if you hit a usage threshold with Opus.

Knowing this matters because many developers on Max plans run Opus for every task, including tasks where Haiku would finish 3x faster with identical results.

Pattern 1: Switch Models Mid-Session With /model

You don't need to restart Claude Code to change models. The /model command switches instantly:

# Planning a complex migration? Switch to Opus
/model opus

# Now implementing the plan? Switch to Sonnet
/model sonnet

# Quick file search or simple rename? Switch to Haiku
/model haiku

You can also start a session with a specific model:

# Start with Haiku for a quick task
claude --model haiku

# Start with Opus for a design session
claude --model opus

Or set it permanently in your settings file (~/.claude/settings.json):

{
  "model": "sonnet"
}

The priority order is: /model during session > --model flag > ANTHROPIC_MODEL environment variable > settings file.

This alone saves significant time. If you're in the middle of implementing a feature and need to look up how a module works, switching to Haiku for that exploration means you get answers faster and spend less context on a task that doesn't need deep reasoning.

Pattern 2: Use opusplan for Automatic Routing

The most underused model alias in Claude Code is opusplan. It uses Opus when you're in plan mode and automatically switches to Sonnet for execution:

claude --model opusplan

Here's why this matters. Planning requires the model to reason about architecture, weigh tradeoffs, and consider edge cases. That's where Opus excels. But once the plan is set, implementation is largely mechanical: write the code, follow the patterns, run the tests. Sonnet handles that efficiently.

With opusplan, you get Opus-quality planning and Sonnet-speed execution without manually switching. The transition happens automatically when you exit plan mode.

To enter plan mode during a session, use:

/plan

Claude Code switches to read-only exploration mode. If you're on opusplan, the model upgrades to Opus for this phase. When you approve the plan and exit plan mode, it drops back to Sonnet for implementation.

This is the closest thing to automatic model routing in Claude Code today, and it works well for the most common workflow: think first, then build.

Pattern 3: Control Effort Levels

Model selection isn't the only lever. Effort levels control how much thinking the model does per turn, independent of which model you're using.

Three levels persist across sessions: low, medium, and high. A fourth level, max, is available on Opus 4.6 only and does not persist.

# Quick formatting fix? Low effort
/effort low

# Standard coding work? Medium (default)
/effort medium

# Hard debugging or architecture? High effort
/effort high

# The hardest problem in the codebase? Max (Opus only)
/effort max

You can also set effort at startup:

claude --effort high

Or via environment variable:

export CLAUDE_CODE_EFFORT_LEVEL=high

Or in your settings file:

{
  "effortLevel": "medium"
}

The key insight: medium effort is the recommended default for most coding tasks. Higher effort levels can cause the model to overthink routine work, actually making it slower without improving quality.

Reserve high or max for tasks that genuinely benefit from deeper reasoning: tracing a complex bug across multiple files, designing a new system architecture, or debugging a race condition.

For one-off deep reasoning without changing your session setting, include "ultrathink" in your prompt. This triggers high effort for that single turn.

The combination of model tier and effort level gives you a 2D control surface. Haiku at low effort is near-instant for simple lookups. Opus at max effort is the deepest reasoning available, but takes longer and costs more. Matching both dimensions to the task is what separates efficient Claude Code usage from wasteful usage.

Pattern 4: Route Models to Subagents

This is where model routing gets powerful. Claude Code subagents can each run on a different model, set in their definition file.

Here's a practical example. You want a code review agent that runs on Sonnet (good enough for pattern matching and style checks) and a research agent that runs on Haiku (fast lookups, no need for deep reasoning):

Create .claude/agents/code-reviewer.md:

---
name: code-reviewer
description: Reviews code for quality and best practices. Use proactively after code changes.
tools: Read, Grep, Glob, Bash
model: sonnet
---

You are a senior code reviewer. When invoked:
1. Run git diff to see recent changes
2. Focus on modified files
3. Review for clarity, security, error handling, and test coverage

Provide feedback organized by priority:
- Critical issues (must fix)
- Warnings (should fix)
- Suggestions (consider improving)

Create .claude/agents/researcher.md:

---
name: researcher
description: Fast codebase exploration and documentation lookup.
tools: Read, Grep, Glob
model: haiku
---

You are a fast research assistant. Find files, search code,
and answer questions about the codebase structure.
Return concise summaries, not full file contents.

Now your main session can run on Opus for complex work, while delegating reviews to Sonnet and lookups to Haiku. Each subagent runs in its own context window, so the verbose output from exploration doesn't consume your main conversation's context.

Claude Code's built-in subagents already follow this pattern. The Explore agent runs on Haiku for fast, read-only codebase searches. The Plan agent inherits your session model for research during plan mode. The general-purpose agent also inherits your model for complex multi-step tasks.

You can also set a global default for all subagents via environment variable:

export CLAUDE_CODE_SUBAGENT_MODEL=haiku

This overrides the model for every subagent invocation, regardless of what's set in their definition files. Useful for cost control during development.

The model resolution order for subagents is:

CLAUDE_CODE_SUBAGENT_MODEL environment variable
Per-invocation model parameter (set by Claude when delegating)
Subagent's model frontmatter field
Main conversation's model

Pattern 5: Pin Models for Team Consistency

When multiple developers share a project, model inconsistency becomes a real problem. One developer tests on Opus, another on Haiku. Code that works with Opus-level reasoning might fail when a teammate runs it with Haiku.

Pin models with environment variables to ensure everyone uses the same configuration:

# In your team's .envrc or shell profile
export ANTHROPIC_DEFAULT_OPUS_MODEL="claude-opus-4-6"
export ANTHROPIC_DEFAULT_SONNET_MODEL="claude-sonnet-4-6"
export ANTHROPIC_DEFAULT_HAIKU_MODEL="claude-haiku-4-5-20251001"

For enterprise deployments on AWS Bedrock or Google Vertex AI, this is critical. Without pinning, Claude Code uses aliases that resolve to the latest version. When Anthropic releases a new model, users whose accounts don't have the new version enabled will break silently.

You can also restrict which models your team can select:

{
  "availableModels": ["sonnet", "haiku"]
}

This prevents developers from switching to models outside the approved list via /model, --model, or environment variables. The default model for their subscription tier always remains available.

For full control, combine both settings:

{
  "model": "sonnet",
  "availableModels": ["sonnet", "haiku"]
}

This ensures everyone starts on Sonnet and can only switch between Sonnet and Haiku. No surprise Opus bills. No inconsistent behavior across the team.

The Decision Framework

Here's how to decide which model and effort to use for common tasks:

Task	Model	Effort	Why
Find a file or function	Haiku	Low	Pure lookup, no reasoning needed
Rename a variable across files	Haiku	Low	Mechanical, well-defined scope
Write a new function	Sonnet	Medium	Needs context awareness, not deep reasoning
Write tests for existing code	Sonnet	Medium	Pattern matching against existing code
Debug a failing test	Sonnet	High	Needs reasoning about causality
Design a new module architecture	Opus	High	Multi-factor tradeoff analysis
Refactor across 10+ files	Opus	High	Needs to hold full dependency graph in context
Plan then implement a feature	opusplan	Auto	Opus for the plan, Sonnet for the code

The 1M context window option (opus[1m] or sonnet[1m]) is available for long sessions with large codebases. On Max, Team, and Enterprise plans, Opus automatically gets 1M context. For Sonnet or other plans, it requires extra usage billing.

# Explicitly request 1M context
/model opus[1m]
/model sonnet[1m]

Start Here

If you're currently using the default model for everything, start with one change: switch to opusplan. It gives you the best of both tiers with zero manual switching.

Then add effort levels. Most of your work is /effort medium. When you hit a hard problem, /effort high gives you deeper reasoning without changing models.

Finally, create subagent definitions for your most common delegated tasks. A Haiku-powered explorer and a Sonnet-powered reviewer cover 80% of subagent use cases.

The goal isn't to use the cheapest model everywhere. It's to use the right model for each task, so you get faster answers on simple work and deeper reasoning on hard problems.

Follow @klement_gunndu for more Claude Code content. We're building in public.

Ask Your CSV Anything: Build a Data Analysis Agent in Python

klement Gunndu — Thu, 26 Mar 2026 12:37:45 +0000

Every data scientist has the same ritual. Load a CSV. Write df.describe(). Squint at column names. Write 14 lines of pandas to answer one question. Repeat.

What if an LLM could write those 14 lines for you — and run them?

That is exactly what data analysis agents do. You ask a question in English. The agent writes pandas code, executes it in a sandboxed Python environment, reads the output, and returns a plain-language answer. No manual wrangling. No copy-pasting between notebook cells.

Here are 4 patterns that make this work in production — from a 5-line quick start to a fully custom analysis pipeline.

Pattern 1: The Pandas DataFrame Agent

LangChain's create_pandas_dataframe_agent is the fastest path from CSV to answers. It wraps your DataFrame in a tool-calling agent that writes and executes pandas code through a sandboxed Python REPL.

Install the dependencies:

pip install langchain-experimental langchain-openai pandas

Load your data and create the agent:

import pandas as pd
from langchain_openai import ChatOpenAI
from langchain_experimental.agents.agent_toolkits import (
    create_pandas_dataframe_agent,
)

df = pd.read_csv("sales_data.csv")

llm = ChatOpenAI(model="gpt-4o", temperature=0)

agent = create_pandas_dataframe_agent(
    llm,
    df,
    agent_type="tool-calling",
    verbose=True,
    allow_dangerous_code=True,
)

The allow_dangerous_code=True flag is mandatory. The agent executes real Python code against your DataFrame, so it must be opted into explicitly. Run this only in a sandboxed environment — never on a production server with sensitive data.

Now ask questions:

result = agent.invoke("What are the top 5 products by total revenue?")
print(result["output"])

The agent inspects the DataFrame's columns and dtypes, writes a pandas expression like df.groupby('product')['revenue'].sum().nlargest(5), executes it, and returns the result as natural language.

Multi-DataFrame support. Pass a list of DataFrames to compare datasets:

df_2024 = pd.read_csv("sales_2024.csv")
df_2025 = pd.read_csv("sales_2025.csv")

agent = create_pandas_dataframe_agent(
    llm,
    [df_2024, df_2025],
    agent_type="tool-calling",
    verbose=True,
    allow_dangerous_code=True,
)

result = agent.invoke(
    "Which products grew more than 20% between 2024 and 2025?"
)

When to use this pattern: Small to medium datasets (under 1 million rows) that fit in memory. Quick exploratory analysis. Prototyping.

Limitation: The agent sends the first few rows and column names to the LLM as context. For wide DataFrames with 100+ columns, this context can overwhelm the model's attention. For large datasets, use Pattern 2.

Pattern 2: The SQL Agent for Larger Datasets

When your data outgrows a DataFrame — millions of rows, multiple related tables, or anything that benefits from indexing — convert it to SQLite and let a SQL agent handle the queries.

pip install langchain-community langchain-openai sqlalchemy

Load your CSV into SQLite and create the agent:

import pandas as pd
import sqlalchemy
from langchain_community.utilities import SQLDatabase
from langchain_community.agent_toolkits import create_sql_agent
from langchain_openai import ChatOpenAI

engine = sqlalchemy.create_engine("sqlite:///analysis.db")

df_orders = pd.read_csv("orders.csv")
df_customers = pd.read_csv("customers.csv")
df_orders.to_sql("orders", engine, if_exists="replace", index=False)
df_customers.to_sql("customers", engine, if_exists="replace", index=False)

db = SQLDatabase(engine)
print(f"Tables: {db.get_usable_table_names()}")

llm = ChatOpenAI(model="gpt-4o", temperature=0)

agent = create_sql_agent(
    llm,
    db=db,
    agent_type="tool-calling",
    verbose=True,
)

Now ask questions that span multiple tables:

result = agent.invoke(
    "What is the average order value per customer segment?"
)
print(result["output"])

The agent inspects the database schema, writes a SQL query with the appropriate JOINs, executes it, and translates the result into English.

Why SQL beats pandas at scale:

Indexing. SQLite creates indexes that make filtering and joining fast. Pandas scans the full DataFrame.
Memory. SQLite reads from disk. Pandas loads everything into RAM.
Joins. SQL JOINs are declarative and optimized. Pandas merges require explicit column mapping and can be memory-intensive.
Audit trail. Every query the agent writes is a valid SQL statement you can review and rerun.

Safety note: The SQL agent generates and executes queries against your database. Use a read-only connection or a copy of your data for analysis. Never point it at a production database with write access.

engine = sqlalchemy.create_engine(
    "sqlite:///analysis.db",
    connect_args={"check_same_thread": False},
)

# For read-only protection, use a copy of the database
# or set PRAGMA query_only = ON
with engine.connect() as conn:
    conn.execute(sqlalchemy.text("PRAGMA query_only = ON"))

Pattern 3: Custom Analysis Tools With Tool-Calling

The pre-built agents handle common cases. But real data analysis needs domain-specific operations — statistical tests, time series decomposition, custom visualizations. Build a custom tool-calling agent that exposes exactly the operations your analysts need.

pip install langchain-openai langgraph scipy

Define your analysis tools as Python functions with clear docstrings — the LLM reads these to decide which tool to call:

import pandas as pd
import json
from scipy import stats
from langchain_openai import ChatOpenAI
from langchain_core.tools import tool

df = pd.read_csv("experiment_results.csv")


@tool
def describe_dataset(column: str) -> str:
    """Get summary statistics for a specific column in the dataset.
    Returns count, mean, std, min, 25%, 50%, 75%, max."""
    if column not in df.columns:
        return f"Column '{column}' not found. Available: {list(df.columns)}"
    return df[column].describe().to_json()


@tool
def compare_groups(
    group_column: str, value_column: str, group_a: str, group_b: str
) -> str:
    """Run a two-sample t-test comparing two groups on a numeric column.
    Returns t-statistic, p-value, and whether the difference is
    statistically significant at alpha=0.05."""
    a = df[df[group_column] == group_a][value_column].dropna()
    b = df[df[group_column] == group_b][value_column].dropna()
    t_stat, p_value = stats.ttest_ind(a, b)
    significant = p_value < 0.05
    return json.dumps({
        "group_a_mean": round(a.mean(), 4),
        "group_b_mean": round(b.mean(), 4),
        "t_statistic": round(t_stat, 4),
        "p_value": round(p_value, 6),
        "significant_at_0.05": significant,
        "n_a": len(a),
        "n_b": len(b),
    })


@tool
def correlation_matrix(columns: list[str]) -> str:
    """Calculate pairwise Pearson correlations between specified columns.
    Returns a correlation matrix as JSON."""
    valid = [c for c in columns if c in df.columns]
    if not valid:
        return f"No valid columns found. Available: {list(df.columns)}"
    corr = df[valid].corr()
    return corr.to_json()


@tool
def filter_and_aggregate(
    filter_column: str,
    filter_value: str,
    agg_column: str,
    agg_function: str,
) -> str:
    """Filter the dataset by a column value, then aggregate another column.
    Supported agg_function values: mean, sum, count, min, max, median."""
    valid_aggs = {"mean", "sum", "count", "min", "max", "median"}
    if agg_function not in valid_aggs:
        return f"Invalid agg_function. Use one of: {valid_aggs}"
    filtered = df[df[filter_column] == filter_value]
    result = getattr(filtered[agg_column], agg_function)()
    return json.dumps({
        "filter": f"{filter_column} == {filter_value}",
        "rows_matched": len(filtered),
        "aggregation": f"{agg_function}({agg_column})",
        "result": round(float(result), 4),
    })

Wire the tools into a tool-calling agent:

from langchain_core.messages import SystemMessage

tools = [describe_dataset, compare_groups, correlation_matrix, filter_and_aggregate]

llm = ChatOpenAI(model="gpt-4o", temperature=0)
llm_with_tools = llm.bind_tools(tools)

system_prompt = """You are a data analysis assistant. You have access to a dataset
with these columns: {columns}

When the user asks a question:
1. Identify which tools can answer it
2. Call the appropriate tool(s)
3. Interpret the results in plain language
4. Note statistical significance, sample sizes, and caveats

Always mention sample sizes when reporting statistical results.""".format(
    columns=list(df.columns)
)

from langgraph.prebuilt import create_react_agent

agent = create_react_agent(llm, tools)

Ask domain-specific questions:

result = agent.invoke({
    "messages": [
        {"role": "system", "content": system_prompt},
        {
            "role": "user",
            "content": "Is there a significant difference in conversion rate between the control and treatment groups?",
        },
    ]
})

print(result["messages"][-1].content)

Why custom tools beat the generic pandas agent:

Controlled operations. The agent can only call functions you defined. No arbitrary code execution.
Domain language. Tool descriptions use your team's terminology. The LLM maps natural language to the right operation.
Safety. No Python REPL. No allow_dangerous_code. Each tool validates its inputs and returns structured output.
Reproducibility. Every tool call is logged with exact parameters. You can replay any analysis.

Pattern 4: Output Validation and Guardrails

An analysis agent that returns wrong numbers is worse than no agent at all. These guardrails catch errors before they reach your stakeholders.

Guardrail 1: Schema validation on tool inputs.

Use Pydantic to enforce that tool arguments match expected types and ranges:

from pydantic import BaseModel, field_validator


class AggregationRequest(BaseModel):
    filter_column: str
    filter_value: str
    agg_column: str
    agg_function: str

    @field_validator("agg_function")
    @classmethod
    def validate_agg(cls, v: str) -> str:
        allowed = {"mean", "sum", "count", "min", "max", "median"}
        if v not in allowed:
            raise ValueError(f"Must be one of {allowed}, got '{v}'")
        return v

    @field_validator("filter_column", "agg_column")
    @classmethod
    def validate_columns(cls, v: str) -> str:
        if v not in df.columns:
            raise ValueError(
                f"Column '{v}' not in dataset. "
                f"Available: {list(df.columns)}"
            )
        return v

Guardrail 2: Result sanity checks.

Wrap your agent's output with automatic validation:

def validate_numeric_result(result: dict, column: str) -> dict:
    """Check that a computed result falls within the column's actual range."""
    col_min = float(df[column].min())
    col_max = float(df[column].max())
    value = result.get("result")

    if value is not None and not (col_min <= value <= col_max):
        result["warning"] = (
            f"Result {value} is outside the column range "
            f"[{col_min}, {col_max}]. Verify the calculation."
        )
    return result

Guardrail 3: Row count assertions.

Catch empty results before they produce misleading statistics:

def safe_aggregate(
    filtered_df: pd.DataFrame,
    column: str,
    operation: str,
    min_rows: int = 5,
) -> dict:
    """Aggregate with a minimum sample size requirement."""
    n = len(filtered_df)
    if n == 0:
        return {"error": "Filter returned zero rows. Check filter criteria."}
    if n < min_rows:
        return {
            "warning": f"Only {n} rows matched (minimum {min_rows}). "
            "Results may not be statistically meaningful.",
            "result": getattr(filtered_df[column], operation)(),
            "n": n,
        }
    return {
        "result": getattr(filtered_df[column], operation)(),
        "n": n,
    }

Guardrail 4: Query logging for audit.

Log every tool call so analysts can verify what the agent actually computed:

import logging
from datetime import datetime, timezone

logger = logging.getLogger("analysis_agent")
handler = logging.FileHandler("agent_audit.log")
handler.setFormatter(
    logging.Formatter("%(asctime)s | %(message)s")
)
logger.addHandler(handler)
logger.setLevel(logging.INFO)


def log_tool_call(tool_name: str, params: dict, result: dict):
    logger.info(
        f"TOOL={tool_name} | PARAMS={params} | "
        f"RESULT_KEYS={list(result.keys())} | "
        f"TIMESTAMP={datetime.now(timezone.utc).isoformat()}"
    )

These guardrails are not optional extras. In any workflow where analysis drives decisions — A/B test results, revenue forecasts, user segmentation — a wrong number costs more than a slow answer.

Which Pattern Fits Your Data

Scenario	Pattern	Why
Quick CSV exploration	1 (Pandas Agent)	Five lines to first answer
Multiple related tables, millions of rows	2 (SQL Agent)	Indexing, joins, disk-based
Domain-specific analysis (stats, experiments)	3 (Custom Tools)	Controlled operations, no REPL
Production pipeline with stakeholders	3 + 4 (Custom + Guardrails)	Validated, audited, reproducible

Start with Pattern 1 to prototype. Move to Pattern 2 when data gets large. Move to Pattern 3 when you need control. Add Pattern 4 when wrong answers have consequences.

What to Watch Out For

Code execution risks. Patterns 1 and 2 execute generated code. Run them in Docker containers or sandboxed environments — never on the same machine as production data without isolation.

LLM hallucination in analysis. The agent can write syntactically valid pandas code that answers the wrong question. Always log the generated code (verbose=True) and spot-check against manual queries.

Token costs scale with data context. The pandas agent sends column names, dtypes, and sample rows to the LLM on every call. Wide DataFrames with 100+ columns burn tokens fast. Use Pattern 3's targeted tools to control exactly what context the LLM sees.

Statistical literacy. The LLM knows how to call scipy.stats.ttest_ind. It does not always know when a t-test is the wrong test for your data distribution. Domain expertise still matters — the agent accelerates the workflow, it does not replace the analyst.

Every analyst spends more time wrangling data than analyzing it. These 4 patterns shift that ratio. The LLM handles the pandas syntax, the SQL joins, and the boilerplate. You handle the questions that matter.

Follow @klement_gunndu for more AI engineering content. We're building in public.

Your MCP Server Has No Tests. Here Are 4 Patterns to Fix That.

klement Gunndu — Thu, 26 Mar 2026 08:06:13 +0000

Most MCP servers ship with exactly one test: a developer typing a prompt into Claude and checking if the output looks right.

That is not testing. That is hoping. And it breaks the moment you change a tool signature, add a parameter, or update your database schema.

Why MCP Servers Are Hard to Test

MCP servers sit between deterministic code and non-deterministic LLMs. Your tools are pure functions — they take inputs, return outputs. But the consumers of those tools are language models that interpret schemas, pick tools based on descriptions, and pass arguments based on inference.

This creates a testing gap. Traditional unit tests cover your business logic. LLM integration tests are slow, expensive, and non-deterministic. The four patterns below close that gap using real MCP protocol interactions — without calling an LLM.

Pattern 1: In-Memory Unit Tests With FastMCP Client

FastMCP 2.x includes a Client class that connects directly to your server in-memory. No subprocess. No network. No LLM. You test your actual server logic through the real MCP protocol — in milliseconds.

Install the testing dependency:

pip install pytest-asyncio

Configure pytest to handle async automatically in pyproject.toml:

[tool.pytest.ini_options]
asyncio_mode = "auto"

Here is a server with two tools:

# server.py
from fastmcp import FastMCP

mcp = FastMCP("WeatherServer")

@mcp.tool()
def get_forecast(city: str, days: int = 3) -> dict:
    """Get weather forecast for a city."""
    if days < 1 or days > 14:
        raise ValueError(f"Days must be 1-14, got {days}")
    return {
        "city": city,
        "days": days,
        "forecast": [{"day": i + 1, "temp_c": 20 + i} for i in range(days)],
    }

@mcp.tool()
def get_alerts(region: str) -> list[str]:
    """Get active weather alerts for a region."""
    alerts_db = {"northwest": ["Wind advisory until 6 PM"]}
    return alerts_db.get(region.lower(), [])

Now write the tests. Create a pytest fixture that wraps your server in a Client:

# test_server.py
import pytest
from fastmcp import Client
from server import mcp  # your FastMCP server instance

@pytest.fixture
async def client():
    async with Client(transport=mcp) as c:
        yield c

async def test_forecast_returns_correct_days(client):
    result = await client.call_tool("get_forecast", {"city": "Seattle", "days": 5})
    data = result.data
    assert data["city"] == "Seattle"
    assert len(data["forecast"]) == 5

async def test_forecast_default_days(client):
    result = await client.call_tool("get_forecast", {"city": "Portland"})
    assert len(result.data["forecast"]) == 3

async def test_forecast_invalid_days_raises(client):
    with pytest.raises(Exception):
        await client.call_tool("get_forecast", {"city": "Seattle", "days": 30})

async def test_alerts_existing_region(client):
    result = await client.call_tool("get_alerts", {"region": "Northwest"})
    assert len(result.data) == 1
    assert "Wind advisory" in result.data[0]

async def test_alerts_unknown_region(client):
    result = await client.call_tool("get_alerts", {"region": "Antarctica"})
    assert result.data == []

Run with pytest -v. Every test executes in milliseconds because it uses in-memory transport. No HTTP, no subprocess, no LLM calls.

This pattern catches three categories of bugs immediately: broken tool logic, wrong return types, and missing error handling.

Pattern 2: Schema Validation Tests

Your MCP tools expose JSON schemas to LLMs. If the schema drifts from your implementation — a renamed parameter, a changed type, a missing description — the LLM picks the wrong tool or passes wrong arguments. Schema tests lock this contract down.

async def test_tool_registry_completeness(client):
    """Verify all expected tools are registered."""
    tools = await client.list_tools()
    tool_names = {t.name for t in tools}
    assert tool_names == {"get_forecast", "get_alerts"}

async def test_forecast_schema_has_required_params(client):
    """Verify the forecast tool schema matches expectations."""
    tools = await client.list_tools()
    forecast = next(t for t in tools if t.name == "get_forecast")

    schema = forecast.inputSchema
    assert "city" in schema["properties"]
    assert schema["properties"]["city"]["type"] == "string"
    assert "city" in schema.get("required", [])

async def test_all_tools_have_descriptions(client):
    """LLMs select tools based on descriptions. Missing = broken routing."""
    tools = await client.list_tools()
    for tool in tools:
        assert tool.description, f"Tool '{tool.name}' has no description"
        assert len(tool.description) > 10, (
            f"Tool '{tool.name}' description too short: '{tool.description}'"
        )

Schema tests catch a specific class of failure that unit tests miss entirely: your code works, but the LLM cannot use it. This happens more often than you think. A developer renames a parameter from city to location. The tool still works. The schema updates automatically. But every prompt template, every LLM workflow, and every agent that was trained on the old schema now sends city and gets a validation error.

Schema tests make this failure loud and immediate. When the parameter name changes, test_forecast_schema_has_required_params fails in CI. The developer sees the break before it ships.

If your server exposes resources alongside tools, test those registrations too:

async def test_resources_are_registered(client):
    """Verify static resources are accessible."""
    resources = await client.list_resources()
    assert len(resources) > 0, "Server exposes no resources"

Run schema tests in CI on every commit. Schema drift is silent and devastating — these tests make it visible.

Pattern 3: Parameterized Edge Case Testing

MCP tools receive arguments from LLMs. LLMs send unexpected inputs — empty strings, extreme values, wrong types. Parameterized tests cover these systematically.

@pytest.mark.parametrize(
    "city, days, expected_count",
    [
        ("Seattle", 1, 1),
        ("Tokyo", 7, 7),
        ("São Paulo", 14, 14),
        ("New York", 3, 3),
    ],
)
async def test_forecast_valid_ranges(client, city, days, expected_count):
    result = await client.call_tool("get_forecast", {"city": city, "days": days})
    assert len(result.data["forecast"]) == expected_count

@pytest.mark.parametrize(
    "invalid_days",
    [0, -1, 15, 100],
)
async def test_forecast_rejects_invalid_days(client, invalid_days):
    with pytest.raises(Exception):
        await client.call_tool(
            "get_forecast", {"city": "Seattle", "days": invalid_days}
        )

@pytest.mark.parametrize(
    "region, has_alerts",
    [
        ("Northwest", True),
        ("northwest", True),
        ("NORTHWEST", True),
        ("southeast", False),
        ("", False),
    ],
)
async def test_alerts_case_insensitive(client, region, has_alerts):
    result = await client.call_tool("get_alerts", {"region": region})
    if has_alerts:
        assert len(result.data) > 0
    else:
        assert len(result.data) == 0

For complex return structures, the inline-snapshot library eliminates manual assertion writing. Install it with pip install inline-snapshot, then write:

from inline_snapshot import snapshot

async def test_forecast_structure(client):
    result = await client.call_tool("get_forecast", {"city": "Seattle", "days": 2})
    assert result.data == snapshot()

Run pytest --inline-snapshot=fix,create once. The library fills in the expected value automatically from the actual output. On subsequent runs, it asserts against the stored snapshot. When your output changes intentionally, run --inline-snapshot=fix to update.

Pattern 4: Interactive Testing With MCP Inspector

Unit tests verify code paths. But sometimes you need to see what the LLM sees — the exact schemas, the raw responses, the protocol messages. MCP Inspector is the official visual testing tool for this.

Launch it against your server:

# For a Python MCP server
npx @modelcontextprotocol/inspector uv --directory ./my-server run my-server

# For a published PyPI package
npx @modelcontextprotocol/inspector uvx mcp-server-git --repository ~/code/repo.git

This opens a web UI at http://localhost:6274 that connects to your MCP server through a local proxy. From the Inspector, you can:

Browse all tools — see the JSON schema exactly as an LLM receives it
Call any tool — fill in parameters through a form, see the raw response
Inspect resources — view static context your server exposes
Test prompts — verify prompt templates render correctly

MCP Inspector serves a different purpose than pytest. Use it for:

Exploratory testing during development — try edge cases manually
Schema review — verify descriptions are clear enough for LLM tool selection
Debugging failures — reproduce exact inputs that caused production issues
Demo and documentation — show stakeholders what your server exposes

The Inspector does not replace automated tests. It complements them. Write pytest tests for regression coverage. Use Inspector for exploration and debugging.

A Practical Test Strategy

Combine all four patterns into a layered strategy:

Layer 1: In-memory unit tests (Pattern 1)
  → Run on every save. Sub-second feedback. Catch logic bugs.

Layer 2: Schema validation tests (Pattern 2)
  → Run in CI on every commit. Catch contract drift.

Layer 3: Parameterized edge cases (Pattern 3)
  → Run in CI. Catch boundary failures and type handling.

Layer 4: MCP Inspector (Pattern 4)
  → Use during development. Manual exploration and debugging.

Your pyproject.toml test configuration:

[tool.pytest.ini_options]
asyncio_mode = "auto"
markers = [
    "schema: schema validation tests",
    "edge: edge case and boundary tests",
]

Run fast tests during development:

pytest -v -m "not schema and not edge"

Run everything in CI:

pytest -v --tb=short

What This Costs You

Setting up in-memory MCP tests takes about 30 minutes for an existing server. The fixture is 5 lines. Each test is 3-6 lines. You get sub-second feedback on every change.

Compare that to the alternative: a user discovers your tool returns the wrong type, the LLM hallucinates a workaround, and you spend two hours debugging a production trace.

Thirty minutes of test setup prevents hours of production debugging. That trade is worth it every time.

The MCP ecosystem is growing fast. As of March 2026, thousands of MCP servers exist on npm and PyPI, and the number is accelerating. Most of them have zero automated tests. If you ship yours with a proper test suite, you are already ahead of 90% of the ecosystem. More importantly, your users will trust your server because it actually works when they upgrade.

Follow @klement_gunndu for more MCP and AI engineering content. We're building in public.

Vibe Coding Got You Started. These 5 Skills Keep You Employed.

klement Gunndu — Wed, 25 Mar 2026 12:34:19 +0000

Vibe coding got 92% of US developers using AI tools daily. It also got 40% of junior developers deploying code they don't fully understand.

Both numbers come from 2026 developer surveys. Both are true at the same time. And if you're switching careers into tech right now, that gap is the most important thing you need to understand.

Vibe coding — describing what you want in plain English and letting AI write the code — is a legitimate entry point. It lowers the barrier. It gets you building. Nobody should gatekeep that.

But every hiring manager I've talked to asks the same follow-up question: "What happens when the AI is wrong?"

If you can't answer that with specifics, you're competing against every other candidate who can also type a prompt. These five skills separate developers who get hired from developers who stay hired.

1. Read Code You Didn't Write

The most underrated skill in 2026 is reading. Not writing. Reading.

When AI generates 200 lines of code for your feature, you become a code reviewer — whether you wanted to or not. GitClear analyzed 211 million lines of code and found that AI-assisted development led to 4x more code cloning and a drop in refactoring from 25% of changed lines in 2021 to under 10% by 2024. That means more duplicated, unreviewed code shipping to production every day.

Here's a practical exercise. Take this AI-generated Python function:

def get_user_data(user_id):
    import requests
    response = requests.get(f"https://api.example.com/users/{user_id}")
    data = response.json()
    return {
        "name": data["name"],
        "email": data["email"],
        "created": data["created_at"]
    }

Before you use it, answer five questions:

What happens if the API returns a 404?
What happens if data doesn't have a "name" key?
Is import inside the function a good idea?
What if the network is down?
Is the URL hardcoded? Should it be?

If you can answer all five, you're reading code. If you can't, the AI wrote code you don't own yet.

How to practice: Pick any open-source Python project on GitHub. Read one file per day. Write a comment explaining what each function does. Start with small utilities — not frameworks.

2. Debug Without AI

AI tools are excellent at generating code. They are unreliable at diagnosing why code fails. When you paste an error message back into the chat, you get a plausible-sounding fix. Sometimes it works. Sometimes it introduces a new bug that surfaces three deployments later.

Debugging is the skill that separates someone who can build from someone who can maintain.

Start with tracebacks. Every Python error tells you exactly where it broke:

Traceback (most recent call last):
  File "app.py", line 42, in process_order
    total = calculate_total(items)
  File "app.py", line 18, in calculate_total
    price = item["price"] * item["quantity"]
KeyError: 'quantity'

This traceback says: line 18, inside calculate_total, tried to access item["quantity"] but the key doesn't exist. The fix isn't to add a try/except everywhere. The fix is to find where items is constructed and figure out why "quantity" is missing.

Three debugging techniques that don't require AI:

# 1. Print the actual data shape
print(type(items), len(items))
print(items[0].keys())  # What keys does each item actually have?

# 2. Use breakpoint() to pause execution
def calculate_total(items):
    breakpoint()  # Drops you into pdb — inspect items interactively
    return sum(item["price"] * item["quantity"] for item in items)

# 3. Check assumptions with assert
assert all("quantity" in item for item in items), f"Missing quantity in: {[i for i in items if 'quantity' not in i]}"

The rule: Before asking AI to fix an error, read the traceback yourself first. Identify the file, line number, and the variable that failed. Then decide if you need AI or if the fix is obvious.

3. Use Git Like a Professional

Every development team uses Git. No exceptions. If you can vibe-code a feature but can't create a branch, you'll fail your first code review.

You don't need to memorize every Git command. You need these six:

# Start work on a new feature
git checkout -b feature/add-search

# Check what you changed
git status
git diff

# Save your work
git add search.py test_search.py
git commit -m "Add search endpoint with input validation"

# Push to remote for review
git push -u origin feature/add-search

The habits that matter more than commands:

Commit messages describe why, not what. Not "updated file" — "Add input validation to prevent empty search queries." Your commit message is a note to Future You, and Future You has no context.

Never commit to main directly. Always branch. Always open a pull request. Even on personal projects. Build the muscle memory now so it's automatic when you join a team.

Small commits beat large commits. One commit that changes 50 files is impossible to review. Five commits that each change 10 files tell a story.

# Good: each commit is one logical change
git commit -m "Add User model with email validation"
git commit -m "Add /users endpoint with POST handler"
git commit -m "Add tests for user creation and validation"

# Bad: one commit with everything
git commit -m "Added user feature"

4. Write Tests for AI-Generated Code

AI co-authored code contains 1.7x more major issues than human-written code, according to CodeRabbit's analysis of 470 open-source GitHub pull requests. That number isn't going down. If anything, as more code is AI-generated, the testing gap widens.

You don't need to become a testing expert. You need one pattern: write a test that proves the code does what you think it does.

# The AI wrote this function
def calculate_discount(price, tier):
    if tier == "gold":
        return price * 0.8
    elif tier == "silver":
        return price * 0.9
    return price

# You write these tests
def test_gold_gets_20_percent_off():
    assert calculate_discount(100, "gold") == 80

def test_silver_gets_10_percent_off():
    assert calculate_discount(100, "silver") == 90

def test_unknown_tier_gets_no_discount():
    assert calculate_discount(100, "bronze") == 100

def test_zero_price_returns_zero():
    assert calculate_discount(0, "gold") == 0

def test_negative_price_is_handled():
    # Does the function handle this? Should it?
    result = calculate_discount(-50, "gold")
    # If this surprises you, the function needs a guard

The last test is the important one. It forces you to ask: "What should happen with invalid input?" AI rarely generates edge case tests because edge cases require understanding the business logic — not just the syntax.

Run tests with pytest:

pip install pytest
pytest test_discount.py -v

test_discount.py::test_gold_gets_20_percent_off PASSED
test_discount.py::test_silver_gets_10_percent_off PASSED
test_discount.py::test_unknown_tier_gets_no_discount PASSED
test_discount.py::test_zero_price_returns_zero PASSED
test_discount.py::test_negative_price_is_handled PASSED

Five tests. Two minutes to write. They'll catch every regression when the function changes later.

The habit: Every time AI generates a function, write at least three tests: one happy path, one edge case, one invalid input.

5. Ask Better Questions

The difference between a junior and senior developer using AI isn't the tool. It's the prompt.

A junior prompt:

Write a function that gets user data from an API

A senior prompt:

Write a Python function that fetches user data from a REST API.
Requirements:
- Accept user_id as parameter
- Handle HTTP errors (4xx, 5xx) with specific exceptions
- Timeout after 5 seconds
- Return a typed dict with name, email, and created_at fields
- Log failed requests with the status code

The second prompt produces code you can actually ship. It includes error handling, timeouts, typing, and logging — the things that break in production.

Three patterns for better prompts:

1. Specify the failure modes: "What should happen when the API is down?" forces the AI to generate error handling instead of just the happy path.

2. Ask for the tests alongside the code: "Write the function and the pytest tests for it" gives you verification built in.

3. Ask why, not just what: "Explain why you chose requests.Session() over requests.get()" teaches you the trade-offs. If the AI can't explain its choice clearly, it probably made an arbitrary one.

This isn't prompt engineering in the abstract. This is learning to specify requirements — the same skill that separates junior from senior developers regardless of AI.

The Floor, Not the Ceiling

Vibe coding lowered the entry barrier to software development. That's good. More people building means more problems getting solved.

But the barrier to staying in software development hasn't moved. Teams still need people who can read a codebase they didn't write. Debug a failure at 2 AM without an AI assistant. Push clean commits that tell a story. Write tests that catch the bugs AI introduces. Specify requirements precisely enough that the generated code actually works in production.

These five skills aren't advanced. They're foundational. You can learn each one in a week of deliberate practice:

Week 1: Read one open-source file per day. Write comments explaining each function.
Week 2: Debug three errors using only tracebacks and print(). No AI.
Week 3: Use Git branches for every change in a personal project. Write descriptive commit messages.
Week 4: Write three tests for every AI-generated function before using it.
Week 5: Rewrite your AI prompts with explicit requirements, error handling specs, and typing.

Vibe coding is the door. These skills are the floor.

Follow @klement_gunndu for more AI engineering content. We're building in public.