How do you benchmark an MCP server you built?

Luc B. Perussault-Diallo — Fri, 15 May 2026 13:34:59 +0000

I built a code-intelligence MCP server. Then I built a benchmark for code-intelligence MCP servers. Then my tool placed first on every scenario.

I didn't believe it.

So I threw the harness out and rewrote it from scratch. Same result. I built three held-out scenarios with hand-graded reference scores, ran each iteration of the bench against them, and only trusted a number when it correlated above 0.85 with my own judgment.

That's the part I want to write about. Not which tool won. The methodology decisions you have to make when you are also one of the contestants.

Benchmarking AI tools is harder than benchmarking models because the variable is the tool, not the model.

Disclosure first. I am the author of Sense, one of four MCP servers in the bench, scored alongside a no-MCP baseline (Claude Code with grep, find, and Read). Everything below is in the open: code, scenarios, rubrics, transcripts, judge prompts, and the analysis of where my own tool loses. Repo here. Don't take my word for any of it.

The result

Rank	Tool	Fairness	Quality	Tokens	Time	Cost
#1	sense	81.3%	85.4%	10,896	141s	$6.22
#2	probe	77.7%	84.8%	12,119	162s	$6.23
#3	baseline	77.2%	84.2%	12,716	185s	$7.57
#4	serena	75.2%	83.4%	14,800	191s	$7.57
#5	gitnexus	74.9%	84.5%	12,964	173s	$6.87

Two of the four MCP servers scored below the no-MCP baseline. Serena and Gitnexus, by default, made the agent worse than just letting Claude use grep, find, and Read.

Why an eval framework wouldn't cut it

That result is the question worth defending. Two MCP servers below baseline is a claim I need to back up, which is why most of my time went into the bench's methodology, not the bench's tools.

I looked at the obvious frameworks first (Inspect AI from UK AISI is the closest fit for agent evals; Promptfoo, DeepEval, OpenAI Evals all have their place). They handle maybe a third of the work: the run loop, the LLM-as-judge calls, the report rendering. The other two-thirds had to be custom.

What a framework would handle	What I had to build
Run loop, prompt construction	MCP server orchestration per cell
LLM-as-judge calls	Citation grounding vs pinned repo
Report rendering	`answer_text` / `tool_input` split
	Fairness vs adoption layering
	Held-out + Spearman anchor
	Judge variance characterization

The reason is shape. An eval framework is built around "model produces output, output gets graded against expected output." That's a single-turn paradigm. What I needed was: given an MCP server attached to Claude Code, does an agent reach a useful answer faster, cheaper, and with fewer hallucinated file paths than the same agent with no MCP?

The variable isn't the model. The model is fixed (Opus 4.7, 1M context). The variable is the tool the model has access to. None of those custom pieces exist as off-the-shelf primitives.

1. Citation grounding against a pinned commit

Most evals grade text against text. I needed to grade text against a filesystem.

When an AI agent says "the dispatch logic lives at app.py:1625," that claim is either true or it isn't. The number is either inside the file or beyond EOF. The file is either at that path or it isn't.

So every file:line and file:Symbol reference in the answer gets extracted by regex, then verified against the repo at the pinned commit. Three buckets:

Grounded: the file exists and the line is in range, or the symbol resolves within ±5 lines of the cited line
Unresolved: the file is not at the cited path (usually a basename-only reference where the agent dropped the directory)
Hallucinated: the file exists, but the line is past EOF. Outright fabrication.

Hallucinated is the hard signal. When you flag a line that doesn't exist in a file that does, the agent isn't paraphrasing. It's inventing.

The numbers across all five were stark:

Tool	Citation grounding
sense	89.2%
baseline	80.8%
gitnexus	76.9%
probe	72.8%
serena	61.9%

That gap between 89% and 62% is the difference between trusting the answer and manually verifying every line reference. The LLM-as-judge alone would never have surfaced this. The judge reads the answer; it doesn't crawl the repo.

The naive extractor handles ~95% of cases at a fraction of the complexity. It misses some edge cases (basename-only paths it correctly cannot resolve, symbols at unusual offsets) but it doesn't lie.

2. The `answer_text` vs `tool_input` split

This one almost killed the bench before it was real.

A keyword check asks: did the answer mention TopicCreator? Pre-fix, the scorer searched the entire transcript, including tool calls. So when the agent ran Grep("TopicCreator"), the keyword TopicCreator got a "hit" inside the grep invocation. Even if grep returned nothing.

That's not a measurement. That's a tax on tools that don't use grep. Sense, which uses semantic search instead of grep, would lose keyword points purely because its tool calls didn't contain English-language symbol names. Probe, baseline, anything grep-flavored would win keyword points just for grepping for the right string.

The fix sounds simple: keyword checks search the assistant's prose only. Tool inputs and results live in a separate audit_text field, available for diagnostics but never scored against.

# Before: scorer searched the whole transcript
hits = count_keyword_in(transcript_text, keyword)

# After: scorer searches assistant prose only
hits = count_keyword_in(answer_text, keyword)
# Tool calls live in audit_text, never scored

A bench cannot be honest if its first instinct rewards using the tools the bench's author dislikes.

3. Fairness vs adoption: two layers, never folded

Two questions that look similar are not the same question:

Did the developer get a better answer? That's fairness.
Did the agent fluently use the MCP tools available? That's adoption.

If you measure how often the agent uses an MCP server, Sense looks great. But baseline (no MCP attached) gets a structural zero on that axis. Through no fault of its own. The agent literally cannot call a server that isn't there.

So if "adoption" feeds into the headline score, the headline answers the second question, not the first. That's not a benchmark. That's an MCP-adoption survey.

The two-layer model:

fairness  = 0.10 * keyword_coverage
          + 0.55 * llm_quality
          + 0.15 * citation_grounding
          + 0.20 * efficiency

adoption  = 0.60 * tool_fluency
          + 0.40 * discoverability

Adoption is computed, reported, and never folded into fairness. It's there for code-intel-vs-code-intel comparisons only. The headline number is fairness, and baseline can beat any MCP server on it.

Two of the four MCP servers I benchmarked did score below baseline on fairness. They added friction without offsetting it with answer quality.

If adoption had been in the formula, that finding would have been invisible.

4. Held-out scenarios + Spearman correlation

The anti-Goodhart move.

A self-improving benchmark can drift away from human judgment. You tune the rubric, the rubric tunes the scores, the scores look better, and at some point you're optimizing the metric instead of the underlying capability. Goodhart's Law applies to your own measurement.

The defense is an anchor the loop cannot touch.

Three held-out scenarios, frozen. Hand-graded reference scores in gold.json. SHA256-pinned in a lockfile:

# bench/locked/held-out.lock
flask-blueprints:
  transcript_sha256: 7f3a...
  rubric_sha256: 91e2...
  gold_sha256: 8d4c...

The improvement loop refuses to start if any hash has drifted. Each iteration re-judges the frozen transcripts against the current rubric, then computes Spearman correlation between the current llm_quality and the gold scores.

Drop below 0.85, convergence fails, the loop must stop or be re-anchored.

It's the line between we tuned the bench and we tuned the bench until it agrees with our own grading. The smallest part of the codebase. Possibly the most important.

5. Judge variance: what's reliable, what isn't

You don't get to claim the LLM-as-judge is consistent. You measure it.

I ran the judge twice over the same 12 transcripts. Same prompt. Same model. Default sampling.

Layer	Max stdev	Target	Verdict
Per-criterion (raw 0–1 scores)	0.071	<0.05	Fails
Per-step (4-criterion weighted sum)	0.048	<0.05	Passes
Per-scenario (mean of 4 steps)	max \|Δ\| 0.014	<0.05	Rock-solid

The judge is jittery at the criterion level, especially on a fuzzy criterion like "uncertainty." It averages down at the step level. At the scenario level, the number that actually matters for ranking tools, it's stable enough.

So I use scenario_quality to rank. I don't gate decisions on a single criterion-level delta of 0.05 between two tools. That's inside the noise floor. Per-criterion rationales are commentary, not data.

The win here isn't proving the judge is perfect. It's knowing where it's reliable and where it isn't, so I know which numbers I can defend and which ones I shouldn't quote at all.

Reproduce it yourself

The bench is open. Adding a new MCP server to test is one shell script in bench/tools/.

git clone https://github.com/luuuc/sense
cd sense/bench

# Run the full bench (12 sessions = 5 tools × 6 repos, sense + baseline at minimum)
bash bench/bench.sh

# Or run a single (tool, repo) cell
bash bench/run.sh --tool sense --repo flask

Cost: ~$40 in Opus 4.7 tokens for the full 5×6 matrix. Time: ~20 minutes wall-clock. Every transcript ends up under bench/results/<tool>/<repo>/ with transcript.json, scored.json, judged.json, and run_meta.json pinned to the commit the agent worked against.

To add a new MCP server:

# 1. Drop a config script
cat > bench/tools/yourtool.sh <<'EOF'
#!/bin/bash
# MCP server invocation for yourtool
exec your-mcp-binary --your-flags
EOF
chmod +x bench/tools/yourtool.sh

# 2. Run it
bash bench/run.sh --tool yourtool

If you find a methodology hole, the bench is open and replayable. I'll patch what's real.

What the bench does not claim

Not a benchmark of AI model quality. The model is fixed.
Not a real-world end-to-end task benchmark. Each scenario is bounded, scripted, rubric-anchored.
Not a cost benchmark in production terms. Cost is computed from public API pricing for comparability, not from anyone's actual invoice.
Not a measure of MCP-tool fluency in isolation. Adoption is a secondary signal, not the headline.

It is, narrowly, this: given a six-scenario, four-step exploration script across six real repos, how does each tool affect the agent's answer quality, citation grounding, and efficiency.

A smaller claim than "this is the best code-intel MCP." Also a more useful one.

PS: The hardest part wasn't building the tool. It was building a benchmark I'd still trust after it favored me. Most benchmarks die at that step.

Links:

Bench (code, scenarios, rubrics, replay): github.com/luuuc/sense/tree/main/bench
Full results report: report-for-humans.md
Sense: github.com/luuuc/sense

I built AI tools that roast your startup ideas so your investors don't

Luc B. Perussault-Diallo — Sun, 22 Feb 2026 09:50:46 +0000

I got tired of pitching bad ideas to polite friends.

You know the drill, you share your latest "revolutionary" concept, and everyone nods. "Sounds cool." "Interesting." "You should totally build that." Then six months later you realize nobody ever told you it was a terrible idea.

So I built StartupCheck - a set of free AI tools that deliver the painfully honest feedback founders rarely get.

What it does

Five tools, each targeting a different founder blind spot:

Idea Funeral
Submit your startup idea. Get an honest eulogy instead of polite encouragement.

Corporate Translator
Paste your corporate speak. Find out what it actually means. ("Right-sizing the organization" = layoffs start next week.)

Startup Therapist
Share your founder optimism. Get the reality check your co-founder is too nice to give you.

Pitch Without Slides
Deliver your pitch in plain text. See if it holds up without pretty graphs and stock photos.

Build or Die
Describe your project scope. Find out if your "2-week MVP" is actually a 6-month odyssey.

Why I built it

Every founder has blind spots. The best feedback I've ever received was the most uncomfortable. But most people won't give you that feedback unprompted, it's socially awkward, it risks the relationship, and honestly, it's easier to just nod.

AI has no social incentive to spare your feelings. That's a feature, not a bug.

Details

Free to start, no signup required. Builder plan at 7 EUR/month for higher limits
Built with AI that has zero reason to be polite

Check it out: star.tupcheck.me

Would love to hear your thoughts, and yes, you're welcome to roast StartupCheck using its own tools. Fair game.

Forem: Luc B. Perussault-Diallo