Forem: Webmaster Ramos

One Nav, Two Stacks: A Microfrontend Between Magento and Laravel Without Replatforming

Webmaster Ramos — Thu, 30 Apr 2026 21:04:50 +0000

A working reference implementation on two production-grade stacks (Magento 2.4 + Laravel 11), with the host integration shape shown below and a server-rendered nav skeleton shipped on day one - not retrofitted after GSC panic.

TL;DR

Mid-market ecommerce rarely lives on one stack. The industry answer - "replatform everything onto one stack" - is a $100-500k, 6-12 month project most of them cannot afford.

I shipped a smaller answer on a real client stack: a 15-20 kb Preact microfrontend that mounts into both Magento 2.4 and Laravel 11 via a manifest. This is not a Module Federation hello-world - it is two real host integrations, around 120 lines of PHP on Magento and around 90 lines on Laravel, with one pnpm build and both sites updating in under a minute.

The opinionated part: microfrontends failed as a product architecture but work as a repair strategy. Greenfield product teams drown in coordination cost. Repair across inherited stacks is a different problem - and the pattern solves it cleanly, if you get the SEO contract right before shipping.

The proof point is deliberately concrete: a before/after crawler diff on identical URLs and user-agents. Not a modelled SEO score, not a Lighthouse proxy, but raw HTML facts - bytes, anchor counts, and whether the navigation exists in initial markup for non-rendering crawlers.

The problem nobody names out loud

A mid-market ecommerce group with multiple brands, accrued over years:

A Magento 2.4 storefront - catalogue, cart, checkout.
A Laravel 11 marketing site - brand story, awards programme, editorial.
A handful of single-purpose SPAs on top.

Each stack has its own header and footer. When marketing adds a top-level category, it ships in one stack in a week and in the other in three. When design changes the logo, it takes two sprints to roll out across everything.

The cost is not engineering hours. The cost is that the brand is visibly inconsistent to customers, the teams know it, and every cross-team sync about the nav takes an hour.

Why "just consolidate on one stack" is not the answer

The standard advice is a monorepo or a headless rewrite. Both are correct on paper and wrong in the field.

Monorepos assume teams that want to converge. Inherited teams - Magento folks who have been on that stack for seven years, a Laravel team that came with an acquisition - do not want to converge. They have skill investment, release cadences, and on-call rotations built around their stack. A monorepo migration is a political project before it is an engineering one, and most mid-market companies cannot push one through.

Headless replatforming is the same project in a different wrapper. Twelve-month runway, executive buy-in, and a new front end that has to outpace the rate at which the old ones break the business.

A shared nav microfrontend does not compete with monorepo architecturally. It competes with doing nothing - which is what the organisation was going to do for the next two years anyway.

Why repair is different from design

Spotify publicly rolled back its extreme squad autonomy. The failure mode is always the same: teams own product-level vertical slices, those slices compose into one surface, coordination cost explodes, UX inconsistency becomes a feature of the architecture.

That is a real lesson. It does not mean no microfrontend is ever right.

Repair is a different problem than design. You are not building the surface - the surface already exists, in two incompatible implementations. You are installing the narrowest possible shared layer that brings them back into visual alignment. The nav is exactly that narrow: no business logic, no routing, no data dependencies beyond a flat link tree.

Everything the microfrontend critique flags - duplicate runtime, fragmented UX ownership, coordination overhead - either does not apply to a 15 kb shell (runtime is negligible) or applies less than the status quo (UX is already fragmented; we are reducing coordination by centralising the decision).

Shell architecture: 15-20 kb, one build, one file

A Preact tree built with Vite's library mode into one IIFE script and one CSS file, both with content-hashed filenames. A manifest.json maps logical names to hashed URLs.

Preact over React - ~10 kb gzipped vs ~45 kb. Non-negotiable at a 15-20 kb budget.
IIFE over ES modules - works in Magento's RequireJS environment without extra config, and in any <script> tag on any stack.
cssCodeSplit: false - one file, one request, no FOUC.
Tailwind with a prefix - scoped classes, no collision with host CSS.
Content-hashed URLs via manifest - immutable caching. Hosts read the manifest at render time and emit <link href="/nav/shared-nav.abc123.css">.

pnpm build takes ~8 seconds. Hosts pick up new hashes within their cache TTL. One bugfix lands on both sites in ~1 minute.

Host integration: Magento 2.4

Around 120 lines of new PHP, three files:

Acme\Theme\Model\SharedNavManifest (~85 lines) - HTTP-fetches the manifest with Magento's cache backend, falls back to a non-hashed shared-nav.iife.js on fetch failure so the nav never disappears, only loses cache-busting.
Acme\Theme\ViewModel\SharedNavAssets (~26 lines) - the ViewModel that phtml templates talk to. CSS goes through a ViewModel rather than static <css> layout XML because the URL has a hash in it.
Acme\Theme\etc\frontend\di.xml (~7 lines) - wires the manifest URL through deploy config.

Two phtml partials - header and footer - emit the mount divs and asset tags. Included from default.xml, so every page type inherits the shared nav.

Host integration: Laravel 11

Around 90 lines. Smaller because the service container carries more weight.

App\Services\SharedNavManifest (~65 lines) - HTTP-fetches the manifest, caches via Cache::remember('shared_nav.manifest', 60, ...), logs and falls back to the unhashed bundle on fetch failure.
config/services.php - three lines exposing services.shared_nav.manifest_url as env-driven config.
Two Blade layouts - public and a secondary layout for older marketing pages - emit <link> and <script> tags from the manifest service.

The 60-second cache TTL controls how fast a pnpm build propagates - aggressive enough for release cadence, conservative enough that manifest fetches are one request per minute per worker.

Representative code shape (abridged)

The full production classes are client code, so I am not publishing them verbatim here. But the integration should not stay abstract either. This is the shape of the two host adapters - abridged to show the contract rather than every guardrail and framework detail.

A note on the manifest keys: Vite indexes manifest.json by entry source path and asset name - src/main.tsx and style.css in our build - not by the output filename. The host lookups use those keys; the unhashed filenames (shared-nav.iife.js, shared-nav.css) are only used as fallbacks when the manifest fetch fails.

Magento 2.4 - manifest service shape:

class SharedNavManifest
{
    public function getJsUrl(): string
    {
        return '/nav/' . ($this->manifest()['src/main.tsx']['file'] ?? $this->fallbackJs);
    }

    public function getCssUrl(): string
    {
        return '/nav/' . ($this->manifest()['style.css']['file'] ?? $this->fallbackCss);
    }

    // SSR fallback: fetched once from the shell, cached in Magento's cache
    // backend, and inlined into the mount div at render time.
    public function getHeaderHtml(): string
    {
        return $this->snapshotHtml('header.html');
    }

    public function getFooterHtml(): string
    {
        return $this->snapshotHtml('footer.html');
    }

    private function manifest(): array
    {
        // 1) read cached manifest
        // 2) on miss, fetch remote manifest URL
        // 3) cache parsed JSON
        // 4) on failure, log and fall back to unhashed asset names
    }

    private function snapshotHtml(string $key): string
    {
        // 1) read cached snapshot for $key
        // 2) on miss, fetch rendered HTML from the shell (e.g. /nav/header.html)
        // 3) cache body with a short TTL
        // 4) on failure, return '' so the shell still hydrates later
    }
}

Laravel 11 - manifest service shape:

class SharedNavManifest
{
    public function manifest(): array
    {
        return Cache::remember('shared_nav.manifest', 60, function () {
            // GET config('services.shared_nav.manifest_url'), parse JSON.
            // On failure, log and return [] so fallback filenames kick in.
        });
    }

    public function jsUrl(): string
    {
        return '/nav/' . ($this->manifest()['src/main.tsx']['file'] ?? 'shared-nav.iife.js');
    }

    public function cssUrl(): string
    {
        return '/nav/' . ($this->manifest()['style.css']['file'] ?? 'shared-nav.css');
    }

    public function headerHtml(): string
    {
        return $this->snapshotHtml('header.html');
    }

    public function footerHtml(): string
    {
        return $this->snapshotHtml('footer.html');
    }

    private function snapshotHtml(string $key): string
    {
        return Cache::remember("shared_nav.snapshot:{$key}", 60, function () use ($key) {
            // fetch rendered HTML from the shell (e.g. /nav/header.html)
            // return '' on failure so the shell still hydrates later
        });
    }
}

That is why the line counts matter. The host code is small enough to be reviewable, boring enough to be supportable, and explicit enough that another PHP team can own it without learning a front-end platform first.

The host <-> shell contract

The two sides agree on a tiny surface:

Host provides:

<link rel="stylesheet" href="{{ $nav->cssUrl() }}">
<div id="sa-header" style="min-height: 80px;">{!! $nav->headerHtml() !!}</div>
<!-- page body -->
<div id="sa-footer">{!! $nav->footerHtml() !!}</div>
<script defer src="{{ $nav->jsUrl() }}"></script>

Shell provides:

A nav-fallback.html emitted at build time, split into header and footer snippets the host inlines into the mount divs (the SSR fallback).
Client-side mount into #sa-header and #sa-footer that replaces the SSR snapshot with the interactive tree (dropdowns, mobile menu, state).
One CSS file, one JS file, no global pollution (IIFE scope).
No knowledge of Magento or Laravel. No runtime config, no feature flags.

Everything else - routing, authentication, cart state, checkout - stays on the host. The nav does not know the host exists. The host does not know the nav is Preact. That is the whole integration.

The min-height: 80px on the header mount is anti-CLS insurance - the slot reserves its space before hydration, so Core Web Vitals do not punish the deferred render.

The SEO question, answered honestly

This is the part every microfrontend post skips or hand-waves. I will not.

Also, this section is intentionally based on observable crawler facts, not modelled SEO metrics. I am not claiming a ranking uplift from a synthetic score. I am showing what a crawler can and cannot see in the initial HTML before and after the fallback ships.

Without a fallback, initial HTML is two empty divs:

<div id="sa-header" style="min-height: 80px;"></div>
<div id="sa-footer"></div>

Googlebot renders JavaScript (eventually) and sees the nav - with a delay measured in days. But GPTBot, ClaudeBot, and PerplexityBot do not render JavaScript. They see the empty divs. As far as AI search is concerned, the site has no nav.

I measured this before shipping the SSR fallback. Three pages, five user-agents, identical curl invocations. Same URLs, same crawl method, same parsing rule - only the fallback changed.

Before SSR fallback:

Metric	Homepage	/about	/portfolio
Bytes	35,050	35,050	97,521
`<a href>` total	12	12	12
Anchors from nav	0	0	0

Twelve anchors per page, none of them structural. Every page - no matter how deep - exposed the same twelve inline body links to a non-rendering crawler. Sitemap.xml covered URL discovery, but not the four things nav does beyond discovery:

Link equity - a multi-level nav is hundreds of internal links per page pointing at categories. Without it, category pages lose authority.
Crawl budget - Googlebot prioritises pages by incoming-link density. Sitemap-only pages get crawled less often.
Topic hierarchy - sitemap is flat. Nav signals semantic structure ("Shop -> Men -> Shoes").
AI assistant context - ChatGPT and Perplexity build mental models from HTML, often ignoring sitemaps. Without nav in HTML, AI knows your URLs but not your structure.

The three-level mitigation ladder:

<noscript> fallback with critical links inside the mount div (hours of work).
SSR skeleton - Vite emits a nav-fallback.html at build time; hosts inline it into the mount divs before hydration replaces it (a day or two).
Full SSR service - a Node process renders each nav request server-side (a week, plus a new production dependency).

Level 2 is the sweet spot for an ecommerce group this size. We shipped it before the first production release. Same curl invocations, four days later:

After SSR fallback:

Metric	Homepage	/about	/portfolio
Bytes	98,881	98,881	161,348
`<a href>` total	112	112	112
Anchors from nav (`#sa-header`)	31	31	31
Anchors from footer (`#sa-footer`)	69	69	69

All five user-agents received byte-identical HTML (the only per-request variance is the Laravel CSRF meta token). The nav and footer tree are in initial HTML - 100 additional anchors per page, constant across every page, visible to every crawler that can parse HTML.

That matters methodologically. A crawler can disagree with my interpretation of the SEO impact, but it cannot disagree with 35,050 -> 98,881 bytes or 12 -> 112 anchors under the same crawl conditions. This is a reusable audit method, not a one-off anecdote.

The gap closed on release day. No retroactive GSC panic, no "we measured a drop and here's how we fixed it" narrative. The honest framing is "we knew the risk, we closed it before shipping".

What this article proves today - and what it does not yet

This article proves three things:

The integration pattern is real on two production-grade PHP stacks.
The SEO risk is real if the shell ships with empty mount points only.
A Level 2 fallback closes that crawler-visibility gap on day one.

What it does not prove yet is a 90-day business outcome story. I do not have a "three months later, here are CrUX and GSC deltas" chart in this draft, because that would require waiting for the post-release window to mature. I would rather publish the implementation pattern and the crawler evidence honestly than pretend I have impact numbers I do not have yet.

That makes this a build-and-ship case study, not a finished growth narrative. When post-launch search-console and field-performance data are mature enough to be worth showing, they belong in a follow-up article.

What this pattern does not solve

Not overselling: the shared nav is the minimum viable shared surface - that is its strength and its ceiling.

Primary page content still diverges. Magento renders products; Laravel renders marketing copy.
Shared checkout - not solved. Checkout lives in Magento; marketing links into it via cookies on a common parent domain.
Shared authentication - not solved. Cookies, redirects, OAuth handshakes - all host-specific.
Shared search - could be built on top of the shell, but we did not. Search UX is coupled to Magento-native catalogue data.

A shared nav is not a distributed-front-end strategy. It is a band-aid across a healed fracture. If you need a distributed front end, you need a different architecture.

When this pattern fits

Short checklist. If you check fewer than three boxes, do something else.

You have two or more existing stacks with established teams you cannot realistically move.
There is no budget or appetite for a full front-end unification right now.
The primary pain is UX inconsistency, not performance or architectural debt.
Nobody on the executive side is willing to own a "unified portal" programme.
You need to be AI-agent ready - which means the nav must be in initial HTML, not only after JS runs.

If all five apply, the pattern pays for itself in weeks, not quarters.

What's next

The same shell is about to land on two more stacks in the same group - a greenfield Magento storefront rewrite and a full Laravel marketing rewrite. Both will consume the existing manifest.json unchanged. Zero additional shell work, the same integration footprint per host. That is the portability proof.

If the pattern looks like it might fit your stack, the interesting conversation is not "how do I build a shell" - Vite's library mode docs will get you there in an afternoon. The interesting conversation is the SEO contract and shipping Level 2 on day one instead of retrofitting it after Google Search Console punishes you.

For a mid-market team, that is usually the real decision framework:

Level 1: <noscript> links when the risk window is small and the nav is shallow.
Level 2: build-time SSR fallback when you need full crawler-visible structure without adding a Node service.
Level 3: full SSR service when the nav is dynamic enough that static fallback HTML becomes a maintenance problem.

That is where outside perspective usually helps, and where I spend most of my consulting time.

MCP vs CLI for AI Agents: A Real AWS Benchmark (and Why the Popular Narrative Asks the Wrong Question)

Webmaster Ramos — Tue, 21 Apr 2026 19:12:16 +0000

Full code, aggregated numbers (n=10 across 5 tasks and 5 transports), and a curated selection of 8 hand-picked runs live in the mcp-vs-cli-aws-benchmark repo. This article is a dense version of docs/findings.md from the same repo, rewritten for a reader who doesn't have an hour to study the test harness.

TL;DR

The question in the title is wrong. "MCP or CLI?" assumes they have the same use case and one of them is objectively better. In reality it's a trade-off between two currencies: engineering time vs. input tokens per run, and you need both numbers to decide.

I compared raw aws CLI against the official awslabs.aws-api-mcp-server on five read-only tasks against a real production AWS account. The model is Claude Sonnet 4.6, direct Anthropic API, my own minimal agent loop (no Claude Code and no claude-agent-sdk, to avoid poisoning the context). Ground truth is collected via boto3, verification is automatic. n=10 per (task, transport) cell.

Result: a well-designed CLI tool beats awslabs MCP by 43-60% on input tokens on every one of the five tasks, at equal success rate. But it takes half a day of engineering work per service.

If you run 200 agent invocations a day - put MCP in and forget about it. If you run 200 thousand - sit down and write your own tool wrapper following the checklist at the end of the article.

Where this whole debate comes from

Since February 2026, dev Twitter and dev.to have been flooded with posts carrying the same message: "MCP loses to CLI, here are the numbers". Titles like «Why CLI Tools Are Beating MCP for AI Agents», «MCP vs CLI: Benchmarking AI Agent Cost & Reliability», «Why CLI is the New MCP for AI Agents». They all cite the same Scalekit benchmark, which reported:

MCP is 10-32x more expensive than CLI on input tokens.
Reliability: CLI 100%, MCP 72% (the cause of all 28% of failures is TCP timeouts connecting to the GitHub Copilot MCP server).
Example: a simple "what language is this repo?" query - CLI 1,365 tokens, MCP 44,026 tokens.

The authors' explanation: schema dump. The GitHub Copilot MCP server dumps descriptions of all 43 of its tools into the model's context on startup, and 42 of them are unused in any given query.

The problem is that this benchmark is n=1 on a single service, with one kind of MCP server ("fat", per-resource). From that, people draw "MCP loses" conclusions - that's roughly like measuring internet speed on a single website and concluding "IPv6 is slower than IPv4". There is a useful signal, but no grounds for generalisation.

I decided to reproduce the comparison on a different service (AWS), with a larger n, and in a setting where the MCP server is not designed as a "fat" directory.

AWS has already done its homework

The first thing I found when I went to look at awslabs/mcp was not what I had expected. Following the Scalekit GitHub Copilot MCP analogy, I was expecting to see dozens of per-resource MCP servers: awslabs/ec2, awslabs/s3, awslabs/iam, each with their own 20-30 tools (describe_instances, run_instances, terminate_instances, modify_instance_attribute...). That would have been a clean schema dump in the context of a single task.

In reality, the main AWS MCP server - awslabs.aws-api-mcp-server - is built very differently. It exposes three tools:

call_aws - takes an aws CLI command string (or an array of up to 20 commands for batch mode) and runs it.
suggest_aws_commands - natural language to a list of candidate aws CLI commands. The authors explicitly mark it as FALLBACK.
get_execution_plan - multi-step plans, experimental, gated behind an environment variable.

By default two are published (without get_execution_plan). And there is a built-in READ_OPERATIONS_ONLY=true switch - you can tell the server "describe/list/get only" and it will cut everything else off at its own level.

This is an important engineering choice: AWS itself acknowledged the schema-dump problem and opted out of a fat MCP server in favour of a wrapper over the CLI living under the MCP protocol. Comparing such a wrapper against "raw CLI" is a far more honest experiment than repeating Scalekit on the GitHub MCP.

Methodology

The details (runner code, ground-truth script, whitelist) are in the repo. Here - compressed.

5 read-only tasks against a production-like AWS account:

ID	Category	Task	What it tests
`ec2_running`	simple	List running EC2 in `us-west-2`	One API call + filtering
`s3_bucket_policy`	edge	Bucket policy for a single bucket	Handling of an optional resource
`s3_bucket_regions`	chained	All S3 buckets + region of each	List + per-item lookup
`iam_admin_roles`	filter	IAM roles with `AdministratorAccess` policy	Pagination + content filtering
`ec2_cpu_last_hour`	chained	CloudWatch CPU over 60 min for running EC2	Composition + time windows

The correct answer for iam_admin_roles in my account is an empty list. A separate honesty test: will the model make up role names.

Model: Claude Sonnet 4.6, direct Anthropic API, my own minimal agent loop (~150 lines). Why not claude-agent-sdk or Claude Code - see the "methodology notes" section below, this choice cost me a day and a half.
Transports: CLI - subprocess.run(['aws', ...]) behind a whitelist. MCP - the mcp python lib, which boots awslabs.aws-api-mcp-server via uvx stdio and performs a real MCP handshake.
Safety: a dedicated IAM user mcp-benchmark with ReadOnlyAccess + a local command whitelist. Two lines of defense - in case the model tries to break something.
Verification: a boto3 script captures ground truth before the benchmark, a verifier compares the model's JSON response automatically.
n=10 per cell, median on the main metrics.

First attempt: CLI loses everywhere

Spoiler for anyone who won't read to the end: everything you are about to see - CLI failing two tasks, 60% success rate, a naive strategy with 36 tool calls - turned into the opposite result three days later: a CLI that beats MCP by 43-60% on tokens. But to get there I had to walk through five failed hypotheses and one bug in my own code. This part of the article is here for the detective story, not for the numbers. The numbers are at the end.

On the pilot run with three transports (plain cli, cli with an enriched description, mcp) the picture looked like a confirmation of the Scalekit narrative. On iam_admin_roles:

cli plain: 36 tool calls, 20k input tokens, 68 seconds. Strategy: list-roles + list-attached-role-policies for each of the 34 roles in the account.
mcp: 1 tool call, 5k input tokens, 4 seconds. One command: iam list-entities-for-policy --policy-arn ... --entity-filter Role.

The same model on the same prompt made a different command choice. On MCP - perfect; on CLI - the naive, linear-complexity path.

Even scarier was ec2_cpu_last_hour. CLI failed in 60% of cases: it hit the max_turns limit trying to guess the correct timestamp for CloudWatch get-metric-statistics. I looked at the logs and saw commands with --start-time 2025-05-16T..., --start-time 2025-07-14T... - the model clearly had no idea what year it was.

MCP in the same conditions made 3 calls, always with correct 2026 timestamps, 100% success.

This looked like a ready-made "CLI loses" article. Good thing I didn't stop there.

Five hypotheses, five ablation experiments

Before publishing results like that, I wanted to understand why. "MCP is smarter" is not an explanation, it's a description. Sonnet 4.6 has no way to know which transport it's using to talk to AWS: the agent loop is the same, the prompt is the same. Something structural in the MCP transport was making the model behave differently.

What follows is five controlled experiments. Each time I took the CLI transport and added one trait from the MCP world to test an isolated hypothesis.

Hypothesis 1: tool description length and structure. awslabs's call_aws description is ~3000 characters with examples and best practices. My aws_cli was ~500. I wrote tools_cli_rich.py with a description of the same length, including a direct hint: "For 'find roles attached to policy X', use iam list-entities-for-policy --policy-arn ... --entity-filter Role instead of listing every role and inspecting each one."

Result on iam_admin_roles: 37 tool calls, the same naive strategy. The model read the description (you can tell by the input tokens: they grew), but didn't follow it.

Hypothesis 2: the presence of a second "hinter" tool. Besides call_aws, awslabs exposes suggest_aws_commands, whose description includes an example: "List all IAM users who have AdministratorAccess policy". Maybe the mere presence of this description in context works as "scaffolding", even if the model never actually calls suggest_aws_commands itself?

I made tools_cli_with_fake_suggest.py: a second tool that returns an error when called, with a verbatim copy of awslabs's suggest_aws_commands description. Result: 35 tool calls, the same naive strategy. The model did not call the fake suggest_aws_commands (because the description says in black and white "use only when uncertain") - it just read it. And that didn't help.

Hypothesis 3: tool and parameter names. awslabs's tool is called call_aws with a cli_command parameter. Mine was aws_cli with a command parameter. Maybe "call_aws" semantically nudges the model towards "API-style" thinking, while "aws_cli" nudges it towards "shell-style"?

tools_cli_renamed.py: renamed everything, even added a max_results parameter for full parity. Result: 39 tool calls, naive strategy. This hypothesis was a miss too.

Hypothesis 4: MCP capabilities / prompts / resources. Maybe the MCP server passes something beyond the tool list to the model? The protocol has three other channels: prompts (system prompts from the server), resources (documents for RAG) and instructions (system-level instructions).

I wrote a diagnostic script and asked the server directly:

capabilities: experimental={} logging=LoggingCapability()
              prompts=PromptsCapability(listChanged=False)
              resources=ResourcesCapability(subscribe=False, listChanged=False)
              tools=ToolsCapability(listChanged=True)
instructions: None
prompts: 0
resources: 0

The server declares the capability but publishes nothing. instructions is None. It really does send the model only the tool list and nothing else.

Hypothesis 5: runtime context in the system prompt. This was the most productive one. I made a cli-ctx transport - the same aws_cli, but with four extra lines in the system prompt:

Runtime context (provided by the runner, not by the tool):
- Current UTC time: 2026-04-08T23:06:57Z
- Default AWS region: us-west-2
- This account is real and live; commands return real data.

Four lines. 118 tokens.

And here is what happened on ec2_cpu_last_hour, n=3:

Variant	Calls	Input tokens	Wall	Success
`cli` plain	13-15	26-55k	50-70s	50%
`mcp`	3	13.4k	14s	100%
`cli-ctx`	2	4.1k	10s	100%

cli-ctx didn't just catch up with MCP - it beat it. Three times fewer input tokens and faster wall-clock.

Where did the effect come from? I went into the MCP server logs and looked at what exactly it returns to the model in each tool result. And here's what was in the very first call_aws response:

"ResponseMetadata": {
  "RequestId": "...",
  "HTTPStatusCode": 200,
  "HTTPHeaders": {
    "date": "Wed, 08 Apr 2026 00:15:21 GMT",
    ...
  }
}

The awslabs MCP server passes the full HTTP headers from the AWS API back, including date. Raw aws CLI v2 returns only the response body without headers. The model on MCP knows, from the very first tool call, what today's date is; the model on raw CLI does not, because its training cutoff is somewhere in 2025, and it honestly assumes it's still 2025.

The entire gap on ec2_cpu_last_hour was explained by an HTTP Date header leaking through the MCP abstraction. Four lines in the system prompt reproduce the effect for free.

That was the moment I rethought all the previous results.

Three mechanisms I found and closed

The first mechanism - effect A, HTTP metadata - is already covered in the previous section. Runtime context in the system prompt closed the failures on ec2_cpu_last_hour, and that's the most important of the three effects. But on iam_admin_roles (36 vs 1) and s3_bucket_regions (16 vs 2) the gap remained. So there had to be at least one more thing going on.

Effect B: batch calling

On s3_bucket_regions in the MCP run I looked at the second tool call and saw this:

call_aws(cli_command=[
  "aws s3api get-bucket-location --bucket bucket-1",
  "aws s3api get-bucket-location --bucket bucket-2",
  ... (15 items total)
])

An array of 15 commands. In a single call. I went to the call_aws description and found this section:

Batch Running: The tool can also run multiple independent commands at the same time. Call this tool with multiple CLI commands whenever possible. You can call at most 20 CLI commands in batch mode.

So cli_command accepts anyOf string | array of strings, and the server executes them in parallel inside its own process, returning the results together. The model reads this and uses it.

My original aws_cli accepted only a string. I wrote tools_cli_v2.py: added batch support to the input schema, rewrote the description following the same structure as awslabs's, and added parallel execution via asyncio.gather.

On s3_bucket_regions this instantly cut the tool call count from 16 to 2 - exactly like MCP.

Effect C: "smart" command choice - turned out to be a benchmark bug

But on iam_admin_roles the effect remained. The model on cli-v2 kept doing 36 calls. I was convinced this was some subtle feature of how the model models command selection, and I was preparing an "unexplained mystery" section for the article.

Then I ran cli-v2 iam_admin_roles again and carefully looked at the raw trace instead of the aggregated numbers. Here is the first tool call:

1. aws_cli (0ms, error=True)
   aws iam list-entities-for-policy --policy-arn arn:aws:iam::aws:policy/AdministratorAccess
     --entity-filter Role --output json

Execution time 0ms. error=True. The model immediately tried the right command - exactly the same one MCP uses. And got an error. Not from AWS - the error never reached AWS. The error came from my own safety.py:

ALLOWED = {
    "iam": {
        "list-roles",
        "list-attached-role-policies",
        # list-entities-for-policy WAS NOT IN THIS LIST
    },
    ...
}

I wrote the whitelist based on how I pictured this task being solved. And I put in exactly the commands needed for the naive path. The model on CLI tried the optimal command, got rejected, fell back to the naive path and conscientiously walked through all 36 roles.

The awslabs MCP server has its own allowlist - significantly broader. And list-entities-for-policy is allowed there.

This was a benchmark bug, not a property of MCP. I added one line to the whitelist:

"iam": {
    "list-roles",
    "list-attached-role-policies",
    "list-entities-for-policy",  # <- this one
},

And re-ran cli-v2 iam_admin_roles:

Variant	Calls	Input tokens	Wall
`cli` plain	36	20k	68s
`mcp`	1	5k	4s
`cli-v2` (whitelist fixed)	1	2.8k	4s

Exactly one tool call. And at the same time fewer input tokens than MCP, because we have one tool description of ~3000 characters and MCP has two descriptions totalling ~5800 characters.

This is a methodologically important point for anyone who wants to reproduce a benchmark like this: your own whitelist can silently determine the outcome. If the allowlist only covers the commands needed for the naive strategy, you aren't measuring the transport, you're measuring your whitelist.

Final table: cli-full vs mcp at n=10

cli-full is the union of all three improvements in a single transport:

Batch input (cli-v2 tool spec).
Rich tool description with batch examples and best practices (cli-v2).
Runtime context in the system prompt (cli-ctx).
Broad whitelist with list-entities-for-policy and everything else needed for the optimal path.

At n=10 per cell, median:

Task	cli-full input	mcp input	Δ input	cli-full calls	mcp calls	cli-full ok%	mcp ok%
`ec2_running`	3,053	5,368	-43%	1	1	90%*	100%
`s3_bucket_policy`	2,975	5,425	-45%	1	1	100%	100%
`s3_bucket_regions`	5,801	14,317	-60%	2	2	100%	100%
`iam_admin_roles`	2,934	5,213	-44%	1	1	100%	100%
`ec2_cpu_last_hour`	5,345	9,461	-44%	2	2	100%	100%

* the single failure on ec2_running cli-full #9 was an HTTP 529 Overloaded from the Anthropic API. That's infrastructure noise, not a transport problem. I deliberately did not retry failed runs to avoid masking real failures - and this lone 529 made it into the stats as 90%. MCP could just as easily have caught the same 529; it just got lucky.

cli-full beats MCP on input tokens on every one of the five tasks, 43-60%. Success rate - parity.

On wall clock MCP wins on 4 of 5 tasks. Reason: wall clock is dominated by AWS API call time, not by model turn time. Tokens don't translate directly into seconds. The only wall-clock win for CLI is s3_bucket_regions, where MCP spends time marshalling a 15-item batch through its protocol layer, and my asyncio.gather does not.

The right question: how much is your engineering time worth

This is where the popular "CLI is better than MCP" narrative breaks.

My cli-full is a few hundred lines of code and half a day of debugging. A tool wrapper with a whitelist, a rich description copied from awslabs best practices, batch support via asyncio.gather, a system prompt with runtime context, verify + ground truth for a specific task. And that's only for AWS. For GCP, for Linear, for Notion - everything from scratch.

awslabs.aws-api-mcp-server is one command (uvx awslabs.aws-api-mcp-server@latest) and one environment variable. Works with every AWS service, not with five tasks. Best practices are already baked in by the authors (who know AWS better than I do). Updates come with @latest. Read-only mode is an environment variable.

MCP pays with service knowledge, CLI pays with engineering labour. It's a question of which currency you pay for your agent in: person-hours or tokens.

When to choose MCP

High velocity, low QPS. New project, the agent has to work tomorrow. MCP installs in 30 seconds and covers everything.
Broad surface. The agent pokes at EC2, S3, IAM, Lambda, CloudWatch, RDS, ECS. Writing a CLI wrapper for each service is an unrealistic budget.
Polyglot environment. AWS today, GCP tomorrow, Notion the day after. Per-service CLI wrappers don't scale; one MCP server per service does.
You're not an expert on the service. You don't know by heart that list-entities-for-policy is more efficient than list-attached-role-policies in a loop. The awslabs authors do. You reuse their knowledge by paying a few extra tokens.
Low QPS. A few hundred agent invocations a day. Saving 8k tokens per request is a few dollars a month. Engineering time costs more.

When to choose a purpose-built CLI

High-QPS production. A million calls a day x 8k extra tokens x $3/M input = $24/day on top. That's $8k a year, which is enough to hire a contractor to write the tool wrapper once.
Narrow, stable task set. The agent does five specific things. A narrow whitelist and a short description will be more compact than any universal MCP server.
Full control over the context. Every token in the system prompt and tool description is yours. No ~3KB of hidden awslabs guidance, no update surprises, no external dependency that might suddenly change.
Compliance / audit. Every tool call is visible, every input is validated by your code, every failure mode is known. MCP adds a protocol layer between you and the AWS API that some audits won't accept.
You already have the knowledge. If you know how to work with the service efficiently, you can bake that knowledge into the tool description once and reuse it forever.

Checklist: how to build a cli-full equivalent

If after all this you've decided your use case is CLI, here are six items that turn "raw subprocess.run" into something that beats awslabs MCP.

1. Accept batch input. Tool input schema:

"cli_command": {
  "anyOf": [
    {"type": "string"},
    {"type": "array", "items": {"type": "string"}}
  ]
}

When the model passes an array, the runner executes the commands in parallel via asyncio.gather (or equivalent) and returns the results in list order with index headers [1/15], [2/15]... Saves 10-20x on tool calls for tasks where one command has to be run with different parameters.

2. Put runtime context in the system prompt. Minimum - four lines:

Runtime context (provided by the runner, not by the tool):
- Current UTC time: <now>
- Default region: <region>
- Identity: <arn>
- This account is real and live; commands return real data.

This closes a whole class of problems where the model gets confused about dates, regions, or thinks it's working against documentation rather than production.

3. Write a rich tool description. Aim for 2500-3000 characters. A structure that works (copying awslabs):

Short tool description (1 sentence).
Key constraints (allowed commands, region defaults, auth model).
A "Best practices" section - how to pick commands, when to use batch, when to use --query and --filters.
An "Anti-patterns" section - an explicit "don't list-then-iterate if there's a more specific operation".
2-3 concrete examples covering different task categories.
Restrictions: no shell pipes, no --profile, no substitution.

The model reads this as a cookbook. A badly written description means the model writes naive commands.

4. The whitelist must cover the **optimal commands, not just the "obvious" ones.** This is the point that cost me half a day. Ask yourself: "what would a senior AWS engineer write for this task?" - and make sure that command is in the whitelist. Not just the commands needed for the naive strategy.

5. Return structured output, not prose. Always --output json + truncate to a fixed byte budget with an explicit truncation marker. The model has to know that the response was truncated.

6. Forward tool errors to the model verbatim. When a command fails with [exit=N] <stderr>, return it to the model as-is. It can self-correct on the next turn. Silent failures waste turns for nothing.

Following these six rules turns the CLI wrapper from a parody of a tool into something that actually beats awslabs MCP on tokens. Takes half a day per service.

Methodology notes

Three things I spent time on and which are worth knowing if you want to reproduce a benchmark like this.

First: claude-agent-sdk and Claude Code poison the context. For the first two days I was measuring CLI vs MCP through claude-agent-sdk, and the numbers were wild. 30k input tokens on a "how many running EC2" task. I thought for a long time that it was protocol overhead, but no - it was Claude Code through the SDK dragging my entire user-level ~/.claude.json into the context: figma MCP, pencil MCP, PubMed MCP, Gmail, Calendar, Bash, Edit, Read... 40+ tools from other servers I hadn't asked for. I rewrote the runner onto the direct Anthropic API - cache_read dropped from 30k to 0, input tokens dropped to "normal" 2k on a simple task. If you are benchmarking agents through someone else's ready-made harness, check with your own eyes what exactly goes into the model on the first system turn.

Second: your own whitelist is an invisible benchmark variable. I already wrote about this in the "effect C" section. I'll repeat: any safety / security / validation layer between the model and the real service is part of what you are measuring, even if you don't consciously think of it that way. If your whitelist forces the model into a narrow path, you are measuring the model's behaviour in that narrow path, not the model's behaviour in general.

Third: success_rate and retry policy. One of my cli-full ec2_running runs fell over with an HTTP 529 Overloaded from the Anthropic API. In the stats that's 90% success rate, even though it's not a transport issue. I decided not to retry, because then the risk of masking real problems is too high. The article has to mention that 529 explicitly - otherwise the reader will compare 100% MCP against "90%" CLI and draw the wrong conclusion. Retry policy is yet another invisible variable the benchmark has to state out loud.

Reproducibility

Everything is in a public repo: github.com/webmaster-ramos/mcp-vs-cli-aws-benchmark.

What's in there:

src/agent_loop.py - ~150 lines of a self-contained agent loop on the direct Anthropic API.
src/tools_cli.py, tools_cli_v2.py, tools_mcp.py - CLI and MCP transports. Plus the ablation variants (tools_cli_rich.py, tools_cli_renamed.py, tools_cli_with_fake_suggest.py) from the "five hypotheses" section.
src/runner.py - CLI for running --tasks <ids> --transports <ids> --n <N>.
src/aggregate.py - medians + IQR + success rate from raw JSONL.
src/safety.py - whitelist + injection guard.
src/ground_truth.py - a boto3 script that captures ground truth from a live account (parameterised via BENCH_S3_BUCKET).
results/scrubbed/final_summary.json - aggregated numbers at n=10 across all (task, transport) cells. These are the same numbers as in the tables above, in machine-readable form.
results/scrubbed/sample_runs.jsonl - 8 hand-curated runs, one per key storyline in the article: naive CLI on iam_admin_roles (36 calls), MCP on the same task (1 call), cli-full (1 call); CLI failure on ec2_cpu_last_hour due to 2025 timestamps vs the cli-ctx fix; naive CLI on s3_bucket_regions (16 calls) vs MCP with batch (2 calls) vs cli-full with batch (2 calls). All role, bucket and instance names are replaced with role-N, bucket-N, i-instanceNN. Metrics and full model response text are preserved.
docs/findings.md - extended analytical notes, part of which went into this article.

Why there are no full 250 raw runs in the repo: the raw JSONL files contain real IAM role names, S3 bucket names and EC2 instance IDs from my AWS account, woven into free-form text of model responses and batch commands. They can't be auto-scrubbed without a manual mapping for every name, and one missed line is a leak. So the repo only includes what I reviewed by eye: the aggregated final_summary.json and 8 curated sample runs. If you want to see a full dataset, the best way to get a correct one is to run the benchmark on your own account in ~20 minutes (see below).

To run the benchmark under your own account:

Create a dedicated IAM user with the ReadOnlyAccess policy + any extra grants for your tasks.
cp .env.example .env, fill in AWS_PROFILE, AWS_REGION, ANTHROPIC_API_KEY, BENCH_S3_BUCKET (the name of any bucket in your account for the bucket-policy task).
python -m src.ground_truth - captures ground truth for your account.
python -m src.runner --n 10 - runs the full series, ~15-20 minutes, ~$5-10 on the Anthropic API.
python -m src.aggregate results/raw/*.jsonl - prints the table.

If you repeat this on your own stack and get different numbers - drop me a line, I'd love to compare.

Conclusions

The popular "MCP loses to CLI" narrative rests on a single benchmark (Scalekit, n=1, GitHub Copilot MCP). It is correct in its own conditions, but generalising from it to "MCP is bad" is a mistake.
AWS has already solved the schema-dump problem in awslabs.aws-api-mcp-server. Their flagship MCP server is essentially the CLI with two tools, and that's a fair benchmark partner for raw aws CLI.
On a fair 5-task series at n=10, cli-full beats MCP on input tokens by 43-60% on every task. But that takes writing a tool wrapper, a whitelist, a system prompt, a rich description. Half a day of engineering per service.
The real question isn't "MCP or CLI" but "how much does your engineering time cost vs how much do your tokens cost". MCP wins on velocity, broad surface, polyglot, low-QPS. CLI wins on high-QPS, narrow task set, compliance, and when best-practice knowledge already lives in your head.
All three gap mechanisms - HTTP metadata, batch calling, a broad allowlist - are reproducible in a CLI tool via 4 lines in the system prompt, anyOf string | array in the input schema, and one line in the whitelist. None of them is a structural property of the MCP protocol.
Methodologically - check with your own eyes what goes into the model's context, treat your own whitelist as a benchmark variable, and state your retry policy explicitly when reporting success rate.

If after all this you look at your own use case and decide you want a well-designed CLI tool, the six-item checklist is above. If you decide you want MCP - uvx awslabs.aws-api-mcp-server@latest and you're in the game.

Both options are correct answers to different questions.

YAML vs Markdown vs JSON vs TOON: Which Format Is Most Efficient for the Claude API

Webmaster Ramos — Tue, 14 Apr 2026 20:22:36 +0000

My own benchmark across three Claude tiers (Haiku, Sonnet, Opus): 120 data files, 8 real-world scenarios, 5 formats. Tokens, cost, and accuracy – numbers, not opinions.

You Are Overpaying for Prompts

Every time you send data to the Claude API, the format of that data determines how many tokens you spend. The same 200-product catalog in JSON costs 15,879 tokens. In Markdown, it costs 7,814. In TOON, 6,088. That is a 62% difference.

A 120-task list? JSON consumes 8,500 tokens. TOON uses 2,267. Savings: 73%.

The problem is that every existing benchmark focuses on GPT, Gemini, and Llama. There has not been a public benchmark for Claude. I decided to fix that.

I ran 450 API calls on Claude Haiku 4.5, tested Sonnet 4.6 and Opus 4.6, and counted tokens across 120 files using Anthropic’s production tokenizer. Eight real-world scenarios, five formats. In this article – the results, the conclusions, and specific recommendations.

Five Formats at a Glance

JSON (JavaScript Object Notation)

Year created: 2001; ECMA-404 standard (2013)
Author: Douglas Crockford
Primary use case: APIs, data exchange between systems, configuration files
Key characteristic: strict typing, nesting via {} and [], mandatory quotes

JSON is the lingua franca of programmatic interfaces. Every API speaks JSON, and every language can parse it. But that universality comes at a price in an LLM context: quotes, braces, and commas all consume tokens. They carry syntactic weight, but not semantic meaning.

{"products": [{"id": 1, "name": "Mouse", "price": 29.99, "in_stock": true}]}

YAML (YAML Ain't Markup Language)

Year created: 2001; YAML 1.2 standard (2009)
Authors: Clark Evans, Ingy döt Net, Oren Ben-Kiki
Primary use case: configuration files (Docker Compose, Kubernetes, GitHub Actions)
Key characteristic: indentation-based structure, minimal punctuation

YAML is the de facto standard of the DevOps world. It reads like pseudocode and usually does not require quotes. The trade-off is that repeating keys for every array item eats up much of the punctuation savings.

products:
  - id: 1
    name: Mouse
    price: 29.99
    in_stock: true

Markdown

Year created: 2004
Author: John Gruber (with Aaron Swartz)
Primary use case: documentation, READMEs, blogs, wikis
Key characteristic: human-first syntax – headings #, tables |, lists -

Markdown is the most “native” format for LLMs. Models have been trained on billions of READMEs and wiki pages. GitHub, Notion, Obsidian – all rely on Markdown. It is a communication format, not a data format.

## Products

| ID | Name  | Price | In Stock |
|----|-------|-------|----------|
| 1  | Mouse | 29.99 | Yes      |

Plain Text

Primary use case: human communication – emails, notes, instructions
Key characteristic: no syntax, no markup, maximum flexibility

Plain text with no markup. It minimizes token overhead, but it provides no explicit structure for programmatic data extraction.

Products: Mouse (ID 1, $29.99, in stock)

TOON (Token-Oriented Object Notation)

Year created: 2025 (v1.0 – November 2025, MIT license)
Author: open-source community (GitHub)
Primary use case: token optimization in LLM prompts, replacing JSON in AI workflows
Key characteristic: a YAML + CSV hybrid (indentation for objects, row-style encoding for arrays)

The newest format in this comparison. TOON was created for one purpose: minimize tokens while preserving lossless JSON round-tripping. For arrays of homogeneous objects, field names are declared once and values are written as CSV-style rows. On GPT-5 Nano, it showed 99.4% accuracy with 46% token savings. Before this benchmark, it had not been tested on Claude.

products[1]{id,name,price,in_stock}:
1,Mouse,29.99,true

Methodology

What I Tested

Eight scenarios, each in three sizes (S / M / L), each in five formats. Total: 120 data files.

#	Scenario	Data type	S	M	L
1	System prompt / instructions	Rules, sections	10 rules	30 rules	60 rules
2	Product catalog	Tabular data	20 products	100 products	200 products
3	Roadmap / tasks	Statuses, dependencies	15 tasks	50 tasks	120 tasks
4	Business rules	Conditional logic	8 rules	25 rules	50 rules
5	Few-shot classification	Input-output examples	5 examples	15 examples	40 examples
6	Organizational hierarchy	3 levels of nesting	12 people	60 people	150 people
7	API documentation	Endpoints, parameters	5 endpoints	15 endpoints	30 endpoints
8	Output format	Requesting data in a given format	10 countries	50 countries	100 countries

Few-shot (scenario 5) is a prompting technique in which several “input → output” examples are included directly in the prompt so the model can infer the task from a pattern. For example: "Great product!" → positive, "Terrible quality" → negative, then the question "Love it!" → ?. Zero examples is zero-shot, one example is one-shot, several examples is few-shot. The format of those examples directly affects cost: 40 pairs in JSON take 2,131 tokens; in TOON, 996.

For scenarios 2, 3, 6, and 7, I prepared questions with precomputed correct answers (ground truth). For scenarios 1, 4, and 5, scoring was manual and rubric-based. For scenario 8, I measured output tokens and format compliance.

Models and Pricing

Model	Tier	Input ($/1M)	Output ($/1M)
Claude Haiku 4.5	Fast	$0.80	$4
Claude Sonnet 4.6	Mid	$3	$15
Claude Opus 4.6	Premium	$15	$75

Accuracy was measured across all three tiers. Sizes S and M were tested for accuracy. L-size was used only for token counts.

Clean-Test Principle

All requests were sent directly via the anthropic Python SDK: plain client.messages.create() with temperature=0. No MCP servers, IDE plugins, or agent frameworks.

Token counting was done with client.messages.count_tokens() – Anthropic’s production tokenizer, i.e. the same numbers used for billing. The tokenizer is the same across all Claude tiers – so the token-count data applies to all Claude models.

Benchmark code: github.com/webmaster-ramos/yaml-vs-md-benchmark

Input-Token Efficiency

These numbers apply to all Claude tiers – Haiku, Sonnet, and Opus all use the same tokenizer. The only cost difference comes from the price per token.

Summary Table: Average Input Tokens Across All Scenarios

Format	Average tokens	vs JSON
JSON	3,252	baseline
YAML	2,208	-32%
Markdown	1,514	-53%
Plain Text	1,391	-57%
TOON	1,226	-62%

TOON saves 62% of input tokens on average versus JSON. Markdown saves 53%. YAML, despite its minimal punctuation, saves only 32% – because of repeated keys and indentation overhead.

Breakdown by Scenario (% Savings vs JSON, L-size)

Scenario	YAML	MD	TXT	TOON
Instructions	-22%	-29%	-24%	-24%
Products	-29%	-51%	-53%	-62%
Tasks	-35%	-63%	-69%	-73%
Business Rules	-28%	-52%	-48%	-63%
Few-shot	-31%	-45%	-37%	-53%
Hierarchy	-37%	-61%	-67%	-68%
API Docs	-35%	-45%	-59%	-53%

YAML Savings vs JSON (%, L-size)

MD Savings vs JSON (%, L-size)

TXT Savings vs JSON (%, L-size)

TOON Savings vs JSON (%, L-size)

Detailed Charts by Scenario

Input tokens by scenario: Instructions

)

Input tokens by scenario: Products

Input tokens by scenario: Tasks

Input tokens by scenario: Rules

Input tokens by scenario: Few-shot

Input tokens by scenario: Hierarchy

Input tokens by scenario: API Docs

Key Observations

TOON is the clear leader for tabular data. Product catalogs, task lists, few-shot examples – anything that looks like an array of homogeneous objects. Savings: 62–73% versus JSON.
Markdown is the best all-purpose format. A stable 50–65% reduction across all data types. It is the only format that performs consistently well across tables, instructions, and hierarchies.
YAML is underwhelming. Many people expect YAML to be much more compact than JSON. In practice, the savings are only 14–41%. The reason is repeated keys for every array element.
Plain Text wins on API docs. For technical specifications, plain text is more efficient than TOON (59% vs 53%). Without extra syntax, descriptive text compresses better.
Scale barely affects the percentage savings. The difference between S and L is under 2 percentage points. Format drives efficiency more than data volume does.

Haiku 4.5: When Format Matters

Haiku is the most format-sensitive tier. In 35% of questions, it produced different answers depending on the input format. Accuracy spread reached as high as 36 percentage points between the best and worst format within the same scenario.

Accuracy by Scenario

Accuracy Haiku: Products (product catalog)

Accuracy Haiku: Tasks (tasks / roadmap)

Accuracy Haiku: Hierarchy (organizational hierarchy)

Accuracy Haiku: API Docs (documentation)

Scenario	JSON	YAML	MD	TXT	TOON	Best
Products	63.4%	61.4%	69.2%	70.2%	66.2%	TXT
Tasks	71.0%	65.7%	66.7%	56.7%	65.3%	JSON
Hierarchy	85.7%	92.9%	85.7%	78.2%	85.7%	YAML
API Docs	85.7%	85.7%	57.1%	78.6%	85.7%	JSON/YAML/TOON

Hierarchy shows the sharpest gap: YAML (92.9%) vs Markdown (57.1%) – a 36-point difference. Tree-like structures are clearly easier for Haiku to parse in an indentation-based format.

API Docs: Markdown performs unexpectedly poorly – 57.1% vs 85.7% for JSON. For technical specifications with parameters and types, explicit structure matters more than compactness.

Accuracy by Size (Haiku)

Size	Accuracy
S (small data)	80.3%
M (medium data)	67.2%

Scale matters more than format. Accuracy drops by 13 points when moving from S to M – more than the average difference between formats (5.7 points). The implication is straightforward: reduce data volume first, then optimize format.

Cost: Haiku

Format	Avg tokens	Cost / request	100K requests / month
JSON	3,252	$0.0026	$260
YAML	2,208	$0.0018	$177
MD	1,514	$0.0012	$121
TXT	1,391	$0.0011	$111
TOON	1,226	$0.0010	$98
JSON -> TOON	-	-62%	$162/month

Output Format: Haiku

Output tokens: S-size (10 countries) – Haiku, Sonnet, Opus

Output tokens: M-size (50 countries) – Haiku, Sonnet, Opus

Requested format	S (10 countries)	M (50 countries)	Savings vs JSON
JSON	465	1,985	baseline
YAML	296	1,352	-32..36%
Markdown	165	1,125	-43..65%
Plain Text	294	1,381	-30..37%
TOON	342	1,369	-26..31%

Markdown is the cheapest output format on Haiku. 165 vs 465 tokens on S-size – a 65% reduction. At $4 per 1M output tokens, that matters.

Important: TOON loses on output. Haiku does not know the TOON format and, instead of producing compact CSV-like rows, tends to emit verbose plain text that only vaguely resembles TOON. A few-shot example improves TOON output quality, but it still trails Markdown in efficiency.

Output-Format Choice: Technical Requirements

Output cost is not the only thing that matters. Often, Claude’s response must be processed programmatically – parsed, inserted into a database, or passed to another service. The best output format depends on who or what is going to read it.

Usage scenario	Recommendation	Why
User-facing answer in UI	Markdown	Renders natively, lowest token cost
Backend parsing	JSON	Reliable, universal, guaranteed structure
Config / YAML pipeline	YAML	Human-readable + machine-parsable
Rows for CSV / spreadsheet	TXT	Minimal overhead, structure via delimiters
Compact output for TOON SDK	TOON	Only if using Opus, or with a few-shot example

Rule of thumb: if a human reads the output, use Markdown. If code reads it, use JSON or YAML. Do not optimize output cost at the expense of parsing reliability in production.

Recommendations for Haiku

Data type	Best input	Accuracy	Best output
System prompts	MD	stable	MD
Catalogs, lists	TXT	70.2%	MD
Tasks / roadmap	JSON	71.0%	MD or JSON
Hierarchies	YAML	92.9%	YAML
API documentation	JSON or YAML	85.7%	JSON
Few-shot examples	TOON	65.3% (-0.5% vs JSON)	MD

On Haiku, format matters – especially for hierarchies and API documentation. Use TOON on input where token savings are worth a small accuracy trade-off, but do not use TOON on output without a few-shot example.

Sonnet 4.6: Format Affects Cost, Not Quality

Sonnet 4.6 produced identical answers across all five formats. In 100% of questions, the result was the same regardless of how the data was represented. For Sonnet, format optimization is pure cost reduction with no quality trade-off.

Accuracy: Format-Invariant

Accuracy by model and format

Format	Sonnet 4.6
JSON	89.4%
YAML	89.4%
Markdown	89.4%
Plain Text	89.4%
TOON	89.4%

The answers are completely identical across all formats. Switching from JSON to TOON saves 62% of input tokens while preserving the same output.

Cost: Sonnet

Format	Avg tokens	Cost / request	100K requests / month
JSON	3,252	$0.0098	$975
YAML	2,208	$0.0066	$663
MD	1,514	$0.0045	$454
TXT	1,391	$0.0042	$417
TOON	1,226	$0.0037	$368
JSON -> TOON	-	-62%	$607/month

At 100K requests per month, switching from JSON to TOON saves $607/month. On Sonnet, output costs $15 per 1M tokens, so output optimization also matters.

Output Format: Sonnet

Output tokens for Sonnet (estimated as characters ÷ 3.5 chars/token):

Format	S (10 countries)	M (50 countries)
JSON	~210	~1,120
YAML	~195	~1,023
Markdown	~143	~746
Plain Text	~103	~549
TOON	~86	~414

Comparison of output tokens across all three models (S-size):

M-size (50 countries):

On Sonnet, TOON output requires a few-shot example. Without extra context, Sonnet interprets “TOON format” literally – as an abbreviation connected to cartoons – and returns an irrelevant answer. With a format example in the prompt, it generates correct TOON.

Technical requirements for output on Sonnet are the same as on Haiku: if a downstream system parses the response programmatically, use JSON or YAML. If a human is going to read it, use Markdown.

Recommendations for Sonnet

On Sonnet, format choice is a pure cost optimization. The logic is simple:

Input data: use TOON (for tables) or MD (for instructions / hierarchies)
Human-readable output: Markdown (-65% vs JSON)
Machine-parsed output: JSON (most reliable) or YAML (more compact, still parseable)
TOON output: add a few-shot example to the prompt; otherwise the answer may be incorrect

Optimal prompt design: MD for instructions + TOON for data + a request for MD/JSON output.

Opus 4.6: Maximum Capability, Also Format-Invariant

Opus 4.6 is the strongest model and the most expensive one. Like Sonnet, it is completely insensitive to input format. But Opus has one unique advantage: it knows TOON “out of the box.”

Accuracy: Format-Invariant

Format	Opus 4.6
JSON	93.5%
YAML	93.5%
Markdown	93.5%
Plain Text	93.5%
TOON	93.5%

The answers are 100% identical across all formats. Changing format affects only cost.

Cost: Opus

Format	Avg tokens	Cost / request	100K requests / month
JSON	3,252	$0.0488	$4,878
YAML	2,208	$0.0331	$3,312
MD	1,514	$0.0227	$2,271
TXT	1,391	$0.0209	$2,087
TOON	1,226	$0.0184	$1,839
JSON -> TOON	-	-62%	$3,039/month

On Opus, switching from JSON to TOON saves over $3,000/month at 100K requests. Output costs $75 per 1M tokens – so format optimization has the largest financial impact here.

Output Format: Opus

Output tokens for Opus (estimated as characters ÷ 3.5 chars/token):

Format	S (10 countries)	M (50 countries)
JSON	~254	~1,271
YAML	~286	~1,414
Markdown	~177	~814
Plain Text	~194	~986
TOON	~106	~543

Comparison of output tokens across all three models (S-size):

M-size (50 countries):

Opus generates TOON without hints. That is the key difference from Sonnet and Haiku. Opus knows the format and produces valid TOON output on the first try.

Can Claude generate valid TOON output?

Model	Without example in prompt	With few-shot example
Opus 4.6	Valid TOON	Valid TOON
Sonnet 4.6	Cartoon / irrelevant	Valid TOON
Haiku 4.5	Verbose plain text	Closer to TOON, but still inaccurate

In practical terms, this means: if you need TOON output and want it to work reliably without prompt scaffolding, use Opus.

Technical Requirements for Output: When Parsing Matters More Than Cost

On Opus, output costs $75 per 1M tokens – so output-format savings are highly relevant. But the requirements of the downstream system still take priority:

Scenarios where output must be parsed programmatically:

The response goes into a database or structured store – use JSON
Another LLM or service consumes the response through an API – use JSON or YAML
The response is part of a pipeline (the next step processes the data) – use JSON
The response is rendered in the UI as text or a document – use Markdown (lowest token cost)
You need compact machine-readable output and already have a TOON SDK – use TOON (only Opus works reliably without prompt help)

The key point: output on Opus costs $75 per 1M – five times more than input. A 65% output reduction (Markdown vs JSON) can matter even more than input savings. But do not trade away parse reliability just to cut cost.

Recommendations for Opus

Input: TOON for tabular data (-62%), MD for instructions (-53%)
Human-readable output: Markdown (-65% output tokens)
Machine-parsed output: JSON – reliable and universal
TOON output: works without few-shot – Opus’s unique advantage
Do not use JSON on input: it is the most expensive format with no accuracy benefit

Summary Results

Accuracy Across All Models and Formats

Format	Haiku 4.5	Sonnet 4.6	Opus 4.6
JSON	75.3%	89.4%	93.5%
YAML	75.1%	89.4%	93.5%
Markdown	69.6%	89.4%	93.5%
Plain Text	70.6%	89.4%	93.5%
TOON	74.8%	89.4%	93.5%

For Sonnet and Opus, format does not affect accuracy. For Haiku, it matters materially – especially for hierarchies and documentation.

Decision Matrix: Input Format

Data type	Haiku	Sonnet / Opus
System prompts / instructions	MD (-29%)	TOON or MD
Catalogs, lists	TXT (70.2%)	TOON (-62%)
Tasks / roadmap	JSON (71.0%)	TOON (-73%)
Business rules	JSON (stable)	TOON (-63%)
Few-shot examples	TOON (≈JSON)	TOON (-53%)
Hierarchies	YAML (92.9%)	TOON or MD
API documentation	JSON/YAML (85.7%)	TXT (-59%)

Decision Matrix: Output Format

Output consumer	Recommendation	Haiku	Sonnet	Opus
UI / end user	Markdown	native	native	native
API / JSON parser	JSON	reliable	reliable	reliable
YAML pipeline	YAML	reliable	reliable	reliable
TOON SDK	TOON	with few-shot*	with few-shot*	native
CSV / spreadsheet	TXT	with template	with template	with template

*Requires a few-shot example in the prompt

Benchmark Limitations

Accuracy was measured only on S+M sizes. L-size includes token counts only. Accuracy may degrade more sharply on larger data.
The data is synthetic. Catalogs and tasks were script-generated. Real-world data may be messier (missing fields, Unicode, long descriptions).
Automatic scoring covers 4 of 8 cases. Cases 1, 4, and 5 require rubric-based evaluation. The accuracy numbers here cover cases 2, 3, 6, and 7.
Sonnet / Opus were tested via subscription (subagents). Output-token counts are estimated, not directly measured. Haiku was tested via API.
No A/B test on live traffic. This is a laboratory benchmark. The impact on a production product must be validated separately.

The code and data are open – reproduce it, extend it, challenge it.

What Surprised Me

Opus and Sonnet are completely insensitive to format. I expected a 3–5% gap. I got 0%. For the higher tiers, format is pure cost optimization.
YAML is not as efficient as many assume. The expectation is usually “YAML is more compact than JSON.” In practice, the savings are only 32%. Repeated keys wipe out much of the benefit from removing braces.
TOON works on Claude without special training. Claude may not have seen much TOON in training data, yet all three tiers parse it correctly – essentially on par with JSON.
Opus knows TOON; Sonnet does not. Opus generates valid TOON output without hints. Sonnet interpreted “TOON format” as “cartoon” and produced an irrelevant answer. With a few-shot example, both work correctly.
Markdown is the best output format. The gap in output tokens between JSON and Markdown is 65%. At $75 per 1M on Opus, that is significant. It is also the only format every tier generates natively without extra prompting.
On Haiku, scale matters more than format. Accuracy drops from 80.3% (S) to 67.2% (M) – a 13-point drop. The average difference between formats is 5.7 points. On Sonnet and Opus, scale is much less of an issue.

FAQ

Q: Do these results apply to other models (GPT, Gemini)?

The trends are similar, but the numbers differ. Every model has its own tokenizer. On GPT-5 Nano, YAML shows 62% accuracy on nested data (ImprovingAgents); on Claude Haiku, it reaches 93%. Use these results for Claude, and other benchmarks for other models.

Q: How were tokens counted?

Using client.messages.count_tokens() – the standard Anthropic SDK method and production tokenizer. These are the same numbers used for billing. The tokenizer is the same across all tiers.

Q: Why not test XML?

XML is rarely used in modern LLM workflows. Existing benchmarks (ShShell) suggest that XML is significantly more expensive than Markdown in token terms, with comparable or worse accuracy.

Q: Is TOON a serious format or just hype?

TOON v1.0 was released in November 2025 under MIT, and there are SDKs in 6+ languages. For tabular data, the savings are real – 62% on Claude with JSON-level accuracy. Opus generates TOON output without prompting. Other tiers require a few-shot example.

Q: Does the input format affect the output format?

Partially. If you provide data in YAML, Claude is more likely to structure its answer with indentation. But an explicit instruction such as “Return as a Markdown table” overrides that tendency.

Q: Is it worth converting all prompts away from JSON?

At 100K requests/month on Sonnet, moving from JSON to TOON saves $607/month. On Opus, it saves $3,039/month. For hobby projects with 1K requests, the difference is around $6. Run the math on your own usage.

Q: Can you combine formats in one prompt?

Yes – and that is usually the recommended approach. Markdown for instructions + TOON for data + a request for output in the format you need. Claude handles multi-format prompts well.

Q: Where is the benchmark source code?

github.com/webmaster-ramos/yaml-vs-md-benchmark. All 120 data files, 51 questions, ground truth, runner, and scorer are open for reproduction.

Conclusion

Data format in a prompt is not a cosmetic choice. On the Claude API, the gap between JSON and TOON is 62% on input tokens. Markdown saves 65% on output tokens. At 100K requests/month on Opus, that means $3,039 saved on input and even more on output.

But the main finding is not about tokens. Claude Sonnet 4.6 and Opus 4.6 are completely insensitive to format. They produced 100% identical answers on JSON, YAML, Markdown, Plain Text, and TOON. For the higher tiers, format optimization is pure savings with no quality trade-off.

Only Haiku 4.5 is meaningfully format-sensitive – and only there does the choice of format affect accuracy (by up to 36 percentage points). On Haiku, format should be matched to data type: YAML for hierarchies, JSON for tasks with dependencies.

Beyond cost, there are technical requirements: if the output must be parsed programmatically, JSON is more reliable than Markdown. If a human reads the answer, Markdown is cheaper. Opus is the only tier that generates TOON natively; Sonnet and Haiku require a few-shot example.

TL;DR by tier:

	Haiku 4.5	Sonnet 4.6	Opus 4.6
Does format affect accuracy?	Yes, by up to 36 points	No	No
Best input (data)	YAML/JSON/TXT by data type	TOON	TOON
Best input (instructions)	MD	MD	MD
Best output (human-readable)	MD	MD	MD
Best output (parsing)	JSON	JSON	JSON
TOON output without prompt help	No	No	Yes
JSON -> TOON savings	$162 / 100K	$607 / 100K	$3,039 / 100K

Benchmark run in April 2026 on Claude Opus 4.6, Sonnet 4.6, and Haiku 4.5.
120 data files, 8 scenarios, 3 sizes, 5 formats, 3 models.
All code and data: github.com/webmaster-ramos/yaml-vs-md-benchmark