Forem: ShipWithAI

Beyond CLAUDE.md: 5 Layers Your AI Agent Harness Is Missing

ShipWithAI — Wed, 22 Apr 2026 01:00:00 +0000

Most developers stop at CLAUDE.md. That's layer 1. A production Claude Code harness needs 5 layers: memory, tools, permissions, hooks, and observability. Here's the full setup guide.

Claude Code harness has 5 layers:

Memory — CLAUDE.md, MEMORY.md, .claude/commands/
Tools — MCP servers (sweet spot: 2–3)
Permissions — settings.json allow/deny lists
Hooks — PreToolUse/PostToolUse verification
Observability — Decision logging, cost tracking, anomaly detection

Most developers only have layer 1. Setup order: 1→4→2→3→5 (guardrails before capabilities).

Why? Because LangChain gained +13.7 benchmark points from harness changes alone — jumping from 52.8% to 66.5% on the same model.

Layer 1: Memory (The Foundation)

Your CLAUDE.md is the project rules file. Claude reads it every prompt and follows it consistently.

What goes in memory:

CLAUDE.md — 40–60 lines max. Project context, conventions, constraints.
MEMORY.md — Long-term learning. "We discovered X fails without Y."
.claude/commands/ — Reusable prompt templates as commands.

The ETH Zurich finding: CLAUDE.md alone caps improvement at ~4%. It's necessary but not sufficient.

The HumanLayer benchmark: Teams keeping CLAUDE.md under 60 lines saw better compliance than those writing 200-line manifestos. Shorter = clearer.

# Example CLAUDE.md structure

## Project Identity
- Framework: Next.js 15 + TypeScript
- Package manager: pnpm
- Architecture: API routes + React components

## You Are
- A full-stack developer shipping features
- Opinionated about patterns: prefer hooks > HOCs
- Balancing speed with maintainability

## Rules
1. Always include tests when modifying /lib
2. Use conventional commits for all commits
3. If suggesting breaking changes, warn first
4. Database migrations need rollback logic

## Code Conventions
- Folder structure: /pages, /components, /lib, /styles
- Component naming: PascalCase for React files
- API routes: camelCase for endpoint handlers

## What NOT to do
- Don't refactor without atomic commits
- Don't add dependencies without checking bundle impact
- Don't commit .env files

Layer 2: Tools (Adding Capability)

Tools are MCP servers. Claude uses them to read files, run commands, query databases.

The HumanLayer finding: Too many tools cause agent confusion. Each tool is context overhead. Sweet spot: 2–3 MCP servers per project.

Not 20. Not "all available servers."

Which 2–3 tools?

Filesystem tool — read/write/execute (almost always)
One domain-specific tool — database, API, CLI
Optional: Observability tool — logs, metrics

Example for a Next.js project:

Filesystem (built-in)
PostgreSQL client (query → fix migrations)
GitHub API (check PR status → adjust approach)

More tools = more tokens + more decision fatigue for Claude.

Layer 3: Permissions (The Guardrails)

Permissions live in settings.json. Specify exactly what Claude is allowed to do.

Allowlist over denylist. It's safer to say "Claude can only modify these files" than "Claude cannot do X."

{
  "permissions": {
    "filesystem": {
      "allow": [
        "/src/**",
        "/public/**",
        "*.config.js",
        ".env.local"
      ],
      "deny": [
        "/node_modules/**",
        "/.git/**",
        "/build/**",
        ".env"
      ]
    },
    "execution": {
      "allow": ["npm run test", "npm run build"],
      "deny": ["rm -rf", "sudo *"]
    }
  }
}

Why this matters:

Claude won't accidentally delete node_modules (been there)
Can't run destructive commands without review
Enforced at runtime, not a suggestion

Check settings.json into git. This becomes part of your project's DNA.

Layer 4: Hooks (Deterministic Enforcement)

Hooks are the most powerful layer. They run before and after Claude uses tools.

PreToolUse hook: Intercept tool calls, validate them, reject bad ones.
PostToolUse hook: Inspect results, catch anomalies, trigger alerts.

Boris Cherny, Anthropic, calls verification "the most important thing" for quality. Hooks are that verification.

#!/bin/bash
# Runs before every tool use

TOOL=$1
PARAMS=$2

case $TOOL in
  "filesystem_write")
    if echo "$PARAMS" | grep -E "(node_modules|\.git|\.env)" > /dev/null; then
      echo "REJECTED: Protected path"
      exit 1
    fi
    ;;
  "command_execute")
    if echo "$PARAMS" | grep -E "(rm -rf|:(){ :|:)" > /dev/null; then
      echo "REJECTED: Dangerous command"
      exit 1
    fi
    ;;
esac

echo "APPROVED"
exit 0

#!/bin/bash
# Runs after every tool use

TOOL=$1
RESULT=$2
DURATION=$3

if (( DURATION > 30 )); then
  echo "⚠️  Slow tool: $TOOL took ${DURATION}s"
fi

if echo "$RESULT" | grep -i "error\|failed\|undefined"; then
  echo "🔴 Tool failed: $(echo $RESULT | head -20)"
fi

Where to set hooks:

.claude/hooks/pre-tool-use.sh
.claude/hooks/post-tool-use.sh

Hooks are not bypassed. They're enforcement.

Layer 5: Observability (Learning from Decisions)

Observability means: logging decisions, tracking costs, detecting anomalies.

What to log:

Which tools Claude called and why
Tokens used per session (cost tracking)
Time spent on each decision
Failures and retries

The HumanLayer insight: Surface only failures, not 4,000 lines of passing tests.

Most developers log everything. Better: log strategically.

#!/bin/bash
# Log Claude's decisions

echo "$(date '+%Y-%m-%d %H:%M:%S') | Tool: $TOOL | Status: $STATUS | Tokens: $TOKENS | Duration: ${DURATION}s" >> .claude/logs/decisions.log

TOTAL_COST=$(grep "Tokens:" .claude/logs/decisions.log | awk '{sum+=$NF} END {print sum}')
if (( $(echo "$TOTAL_COST > 5.00" | bc -l) )); then
  echo "💰 Cost alert: $TOTAL_COST USD today"
fi

ERROR_RATE=$(grep "FAILED" .claude/logs/decisions.log | wc -l)
if (( ERROR_RATE > 5 )); then
  echo "🚨 High error rate detected: $ERROR_RATE failures in last hour"
fi

Setup Order Matters: 1 → 4 → 2 → 3 → 5

Why not 1 → 2 → 3 → 4 → 5?

Wrong order: Capabilities before guardrails

Build CLAUDE.md ✅
Add 10 MCP servers ⚠️
Grant all permissions ⚠️
No hooks (too late, broke things already)
Now add observability (chaos already happened)

Right order: Guardrails first

Build CLAUDE.md ✅ (memory/rules)
Add hooks ✅ (enforcement before tools exist)
Add 2–3 MCP servers ✅ (now hooks guard them)
Restrict permissions ✅ (layered safety)
Add observability ✅ (track what's working)

Adding hooks after tools is like adding seatbelts after the crash.

Production-Ready Harness: 10-Item Checklist

[ ] CLAUDE.md exists, 40–60 lines, checked into git
[ ] MEMORY.md setup with "lessons learned"
[ ] .claude/commands/ has 3+ reusable prompts
[ ] Max 3 MCP servers chosen and documented
[ ] settings.json has allowlist (filesystem, execution)
[ ] .claude/hooks/pre-tool-use.sh validates calls
[ ] .claude/hooks/post-tool-use.sh inspects results
[ ] .claude/logs/ directory exists + observability hook running
[ ] Cost tracking implemented (tokens/session)
[ ] Team knows where each file lives + how to update it

FAQ

Which layer do I need first?
Layer 1 (CLAUDE.md). Everything depends on clear memory. Start there.

Does this harness slow down Claude Code?
No. Hooks add ~100–300ms per tool use. Worth it for the safety. Observability has negligible cost.

What are the most important hooks?
PreToolUse (validation) and PostToolUse (anomaly detection). Those two prevent 80% of issues.

How many MCP servers is "too many"?
More than 5 becomes noise. More than 3 means you're probably adding tools you won't use. Start with 1–2, add more only when they solve a real workflow problem.

Can I skip permissions and just use hooks?
Technically yes, but no. Permissions are defense-in-depth. Hooks catch mistakes. Permissions prevent them.

How do I update CLAUDE.md over time?
Document it in MEMORY.md. "We added this rule because X failed." Over time, CLAUDE.md stabilizes.

Originally published on ShipWithAI. I write about Claude Code workflows, AI-assisted development, and building production systems with AI. Full blog + templates at shipwithai.io.

What's your harness score? Drop it in the comments. Do you have all 5 layers, or are you still at layer 1?

Harness Engineering: Why the System Around AI Matters More Than the AI Itself

ShipWithAI — Mon, 20 Apr 2026 12:03:07 +0000

Harness engineering is everything around your AI agent except the model: memory, tools, permissions, hooks, observability. LangChain gained 13.7 benchmark points by changing only the harness (52.8% to 66.5%, same model). Most developers only have Layer 1 (CLAUDE.md). Production needs all 5.

Two lines of config. Same AI model. Completely different reliability:

# CLAUDE.md approach (can be ignored)
"Never delete production database tables."
# Claude reads this, weighs it against 200K tokens of context, may ignore it.

# Hook approach (always enforced)
# PreToolUse hook: command contains "DROP TABLE" + env=production → exit 2 → BLOCKED.

The first is advice. The second is enforcement.

One lives in a markdown file that competes with thousands of other tokens for the model's attention. The other is a shell script that runs before every command and cannot be bypassed. The gap between these two approaches is the gap most teams don't know exists.

That gap has a name now: harness engineering.

What is harness engineering? (And why prompt engineering isn't enough)

Harness engineering is the discipline of building constraints, tools, feedback loops, and observability around an AI agent to make it reliable in production. The formula, popularized by LangChain and refined on Martin Fowler's site:

Agent = Model + Harness

The model is a commodity. The harness is your competitive advantage.

Mitchell Hashimoto, creator of Terraform and Ghostty, defined the core idea: anytime you find an agent makes a mistake, you engineer a solution so the agent never makes that mistake again. In Ghostty's repository, each line in the AGENTS.md file corresponds to a specific past agent failure that's now prevented.

The industry has moved through three distinct eras:

Era	Years	Focus	Key Question	Limitation
Prompt Engineering	2022-2024	Crafting better instructions	"How do I phrase this?"	Instructions get diluted in long contexts
Context Engineering	2025	Curating what the model sees	"What information does it need?"	Knowing isn't doing
Harness Engineering	2026	Building systems around the agent	"What can it do, and what can't it?"	Emerging discipline

Prompt engineering shapes what the agent tries. Context engineering shapes what the agent knows. Harness engineering shapes what the agent can and cannot do.

How did LangChain gain 13.7 benchmark points without changing the model?

By improving three harness components, LangChain jumped from 52.8% to 66.5% on Terminal Bench 2.0 (a benchmark of 89 real-world terminal tasks) while keeping the same model, gpt-5.2-codex. They went from Top 30 to Top 5. No fine-tuning. No model swap. Just harness changes.

Here are the three changes:

1. Context injection. LangChain's LocalContextMiddleware maps the environment upfront and injects it directly into the agent's context. Before this change, the agent wasted steps trying to understand its surroundings.

2. Self-verification loops. After each action, the agent verifies its output against task-specific criteria before moving on. Not just "run the tests." The agent checks whether the output matches what the task actually asked for.

3. Compute allocation. This one is counterintuitive: running at maximum reasoning budget (xhigh) scored only 53.9%, while the high setting scored 63.6%. More compute caused timeouts that hurt overall performance.

Setting	Score	Notes
Before harness changes	52.8%	Baseline, Top 30
After harness changes (high reasoning)	66.5%	Top 5, +13.7pp
Max reasoning (xhigh)	53.9%	Worse than baseline, timeouts

If you're evaluating AI coding tools by comparing model benchmarks alone, you're measuring the wrong variable.

What are the 5 layers of an AI agent harness?

A production harness has five layers. Most developers I talk to in the Claude Code community have Layer 1 and maybe part of Layer 2. That leaves three layers of reliability on the table.

Layer	What It Is	Problem It Solves	Claude Code Implementation
1. Memory	Persistent context across sessions	Agent "forgets" your conventions every session	CLAUDE.md, MEMORY.md, .claude/commands/
2. Tools	Extended capabilities beyond built-ins	Agent can't access your APIs, databases, or services	MCP servers, custom tools
3. Permissions	What the agent is allowed to do	Agent edits sensitive files or runs dangerous commands	settings.json allow/deny lists
4. Hooks	Automated enforcement at lifecycle points	Instructions get ignored under context pressure	PreToolUse/PostToolUse hooks
5. Observability	Knowing what the agent actually did	No visibility into agent decisions or cost	Session logs, cost tracking, action audit

Think of it like your CI/CD pipeline. You built that infrastructure once, and the whole team benefits on every push. A harness works the same way for AI agent sessions.

OpenAI demonstrated this at scale. Their Codex team shipped roughly one million lines of production code, with zero lines written by human hands, over five months. Their harness included AGENTS.md files, reproducible dev environments, and mechanical invariants in CI. Development throughput was roughly one-tenth the time a human team would have needed.

Where is your harness right now?

Run this checklist:

#	Question	Layer
1	Do you have a CLAUDE.md with project conventions and constraints?	Memory
2	Do you have MCP servers connecting Claude Code to external tools?	Tools
3	Do you have settings.json with explicit allow/deny lists?	Permissions
4	Do you have at least one PreToolUse hook that blocks dangerous actions?	Hooks
5	Can you see what Claude did in each session and how much it cost?	Observability

Your score:

1/5: You're in the majority. Most developers stop at CLAUDE.md.
2-3/5: Ahead of most. You've started building real infrastructure.
4-5/5: Production-ready. You're doing harness engineering whether you knew the name or not.

Be honest about question 4. If the answer is no, your agent can still rm -rf your project directory. CLAUDE.md says "don't do that." A hook actually prevents it.

Here's why this matters: an ETH Zurich study (Feb 2026) tested context files across 138 real-world tasks from 12 Python repositories. Human-written context files improved agent success by only about 4%. LLM-generated ones actually reduced success by ~3% while increasing inference costs by over 20%. Instructions alone aren't enough. You need enforcement layers.

How do you start building a harness today?

You don't need all 5 layers at once. Start with three high-impact changes that take less than 30 minutes total.

Quick Win 1: Create a MEMORY.md (5 minutes)

MEMORY.md is a lightweight index that points to where knowledge lives in your project. Unlike CLAUDE.md (which holds static rules), MEMORY.md tracks evolving state: recent decisions, architectural changes, active work.

- [Auth](src/lib/auth/) — Clerk, not NextAuth. Migrated March 2026.
- [DB](prisma/schema.prisma) — PostgreSQL on Supabase. All queries via Prisma.
- [Deploy](docs/deploy.md) — Vercel preview for PRs, production on main.
- [Testing](vitest.config.ts) — Vitest unit, Playwright E2E. Min 80% coverage.
- [API](src/app/api/) — Server Actions preferred over API routes for mutations.

Quick Win 2: Add one PreToolUse guardrail hook (15 minutes)

This hook blocks Claude Code from editing sensitive files. Copy-paste ready:

#!/bin/bash
# .claude/hooks/block-sensitive-files.sh
# Blocks edits to .env, credentials, and CI config

INPUT=$(cat)
FILE_PATH=$(echo "$INPUT" | jq -r '.tool_input.file_path // empty')

SENSITIVE=('.env' 'credentials' '.github/workflows' 'secrets')

for pattern in "${SENSITIVE[@]}"; do
  if [[ "$FILE_PATH" == *"$pattern"* ]]; then
    echo "BLOCKED: Cannot edit sensitive file: $FILE_PATH" >&2
    exit 2
  fi
done

exit 0

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Edit|Write",
        "hooks": [
          {
            "type": "command",
            "command": "bash .claude/hooks/block-sensitive-files.sh"
          }
        ]
      }
    ]
  }
}

Quick Win 3: Enable cost awareness (10 minutes)

Track what each session costs so you notice anomalies early. Boris Cherny, creator of Claude Code, calls verification "probably the most important thing" for quality:

"Give Claude a way to verify its work. If Claude has that feedback loop, it will 2-3x the quality of the final result."

Start simple: review ~/.claude/projects/ after each session to check what Claude did and how much it cost.

FAQ

What is the difference between harness engineering and prompt engineering?

Prompt engineering shapes what the agent tries. Context engineering shapes what the agent knows. Harness engineering shapes what the agent can and cannot do. They're not replacements — they're layers. A production AI workflow uses all three, but harness engineering provides the strongest reliability guarantees because it uses enforcement (hooks, permissions) rather than suggestions (prompts, context).

Do I need harness engineering for Claude Code?

Yes. Claude Code is itself a harness that Anthropic built around their model. But it's the inner harness. You need an outer harness tailored to your project: CLAUDE.md for conventions, hooks for guardrails, MCP servers for tools, permissions for safety boundaries, and observability for cost control.

Is harness engineering only for Claude Code?

No. The principles apply to any AI coding agent: Cursor, GitHub Copilot, OpenAI Codex, Windsurf, Cline. Claude Code happens to offer the most programmable harness surface (17 hook events, MCP protocol, skills system), which is why examples here use it. The concepts transfer directly to other tools.

Try it now: Pick one quick win above and implement it before your next Claude Code session. Quick Win 2 is copy-paste ready and takes 3 minutes.

What's your harness score right now? Drop it in the comments — I'm curious how many devs have gone beyond Layer 1.

Originally published on ShipWithAI. I write about Claude Code workflows, AI-assisted development, and shipping software faster with structured AI.