I built a Lighthouse for MCP tools — it scores your tool definitions on every PR

Hiroki Honda — Mon, 30 Mar 2026 10:55:43 +0000

The problem

AI agents choose between tools based on one thing: the quality of their descriptions.

Research shows 97% of MCP tool descriptions have quality defects (arXiv 2602.14878), and optimized tools get selected 3.6x more often (arXiv 2602.18914).

Most MCP developers don't know their tool definitions are broken until an agent silently ignores them.

What I built

ToolRank scores MCP tool definitions across 4 dimensions:

Findability (25pts) — Can agents discover your tool?
Clarity (35pts) — Can agents understand what it does?
Precision (25pts) — Is the input schema complete?
Efficiency (15pts) — Is it token-efficient?

It's like Lighthouse, but for MCP tools.

GitHub Action — score on every PR

Today I published ToolRank Score on GitHub Marketplace.

Add this to your repo:

name: ToolRank Score
on:
  pull_request:
    paths: ['**/*.json']

permissions:
  pull-requests: write

jobs:
  score:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: imhiroki/toolrank-action@v1

On every PR that touches tool definitions, you get a comment like:

🟣 mcp.json — 95/100 (Dominant, top 3%)

Tool Score F C P E

search_web 95 25 33 23 14

Top issues:

search_web: No usage context — Add 'Use this when...' (+5pt)

Scored by ToolRank v1.0.0

Tool	Score	F	C	P	E
`search_web`	95	25	33	23	14

You can also set a minimum score and fail PRs that don't meet it:

- uses: imhiroki/toolrank-action@v1
  with:
    min-score: 70
    fail-on-low: true

REST API

If you want to integrate scoring into your own workflow:

curl -X POST https://mcp.toolrank.dev/api/score \
  -H "Content-Type: application/json" \
  -d '{"tools": [{"name": "search", "description": "Searches things"}]}'

Returns score, level, percentile, dimensions, and specific fix suggestions.

Ecosystem data

We scan 4,000+ MCP servers daily from Smithery and the Official MCP Registry. Some findings:

374 servers have scorable tool definitions
73% of MCP servers have zero tool definitions (invisible to agents)
Average score: 85.7/100
Servers scoring 85+ get selected ~85% of the time in competitive scenarios

Check where your server ranks: toolrank.dev/ranking

What's next

Layer 2: Real LLM selection testing (Q2 2026)
Agent framework integrations (LangChain, CrewAI)
Registry partnerships

The scoring engine is fully open source. Star it if this is useful.

Try it

Score your tools — paste JSON or enter your Smithery server name
Add the GitHub Action — 30 seconds to set up
Check the ranking — see where you stand

I'm building ToolRank as the quality standard for the MCP ecosystem. If you're building MCP tools, I'd love to hear what you think.

We scanned 4,162 MCP servers. 73% are invisible to AI agents.

Hiroki Honda — Sun, 29 Mar 2026 00:46:39 +0000

There are 4,162 MCP servers registered on Smithery right now. The Python and TypeScript SDKs see 97 million monthly downloads. Every major AI provider has adopted MCP.

But nobody had measured the quality of these tools.

We built ToolRank, an open-source scoring engine that analyzes MCP tool definitions across four dimensions. Then we pointed it at the entire Smithery registry.

The biggest finding wasn't about quality. It was about visibility.

73% of MCP servers are invisible

Out of 4,162 registered servers, only 1,122 expose tool definitions that agents can read.

The remaining 3,040 — 73% — have no tool definitions at all. They're registered, but when an AI agent searches for tools, these servers don't exist. They have no name, no description, no schema. They are invisible.

This is the equivalent of having a website with no indexable content. Google can't rank what it can't read. Agents can't select what they can't see.

Among the visible: average score 84.7/100

For the 1,122 servers that do expose tool definitions, we scored each one across four dimensions:

Dimension	Weight	What it measures
Findability (25%)	Can agents discover you?	Registry presence, naming
Clarity (35%)	Can agents understand you?	Description quality, purpose, context
Precision (25%)	Is your schema precise?	Types, enums, required fields
Efficiency (15%)	Are you token-efficient?	Definition size, tool count

Average ToolRank Score: 84.7/100

Level	Score	Count	%
Dominant	85-100	677	60%
Preferred	70-84	406	36%
Selectable	50-69	39	3.5%
Visible	25-49	0	0%
Absent	0-24	0	0%

The average is higher than expected. But there's a survivorship bias: servers that bother to expose tool definitions tend to be better maintained overall. The real quality problem is the 73% that expose nothing.

Top scoring servers

Rank	Server	Score
1	microsoft/learn_mcp	96.5
2	docfork/docfork	96.5
3	brave (Brave Search)	94.7
4	LinkupPlatform/linkup-mcp-server	93.5
5	smithery-ai/national-weather-service	93.3

What do top servers do differently? They start descriptions with clear action verbs. They include usage context ("Use this when..."). They define required fields, enums, and defaults. They keep tool count under 15.

The most common defects

Among the 1,122 scored servers, the most frequent issues:

Missing usage context — Description says what the tool does, but not when to use it. Agents need "Use this when..." to decide between competing tools.
No return value described — Agents can't predict what they'll get back. This leads to incorrect downstream handling.
Missing parameter descriptions — Schema has types but no explanations. An agent sees "query": {"type": "string"} but doesn't know what format the query should be in.
No required fields defined — Agents guess which parameters are mandatory, leading to failed executions.
Description too short — Under 50 characters. Not enough information for reliable selection.

What does this mean?

Academic research backs up why this matters. A study of 10,831 MCP servers (arXiv 2602.18914) found that tools with quality-compliant descriptions achieve 72% selection probability vs 20% for non-compliant ones. That's a 3.6x advantage.

The fixes are trivial. Adding "Use this when..." to a description. Defining required fields. Starting the description with a verb. These are 5-minute changes with measurable impact.

We're calling this ATO

ATO (Agent Tool Optimization) is the practice of optimizing tools so AI agents autonomously discover, select, and execute them.

SEO optimized for search engines. LLMO optimized for LLM citations. ATO optimizes for agent tool selection.

The key difference: SEO and LLMO result in mentions and links. ATO results in your API being called and transactions occurring. LLMO is Stage 1 of ATO — necessary but not sufficient.

Full ATO framework →

Try it yourself

Score your tools: toolrank.dev/score — paste your tool definition JSON. Includes "Try example" buttons so you can see how scoring works instantly.

Ecosystem ranking: toolrank.dev/ranking — live rankings updated weekly.

Scoring engine: github.com/imhiroki/toolrank — fully open source. The scoring logic is transparent.

Methodology

Data source: Smithery Registry API (registry.smithery.ai)
Scan date: March 28, 2026
Servers in registry: 4,162
Servers with tool definitions: 1,122 (27%)
Scoring level: Level A (rule-based, 14 checks, zero LLM cost)
Request interval: 2 seconds (respectful to Smithery infrastructure)
Full scan time: ~2 hours
Code: Open source

Data updates weekly via automated scanning. Daily differential scans catch new servers.

Built by @imhiroki. Questions, feedback, or want to improve your score? Open an issue on GitHub.

Forem: Hiroki Honda

I built a Lighthouse for MCP tools — it scores your tool definitions on every PR

The problem

What I built

GitHub Action — score on every PR

🟣 `mcp.json` — 95/100 (Dominant, top 3%)

REST API

Ecosystem data

What's next

Try it

We scanned 4,162 MCP servers. 73% are invisible to AI agents.

73% of MCP servers are invisible

Among the visible: average score 84.7/100

Top scoring servers

The most common defects

What does this mean?

We're calling this ATO

Try it yourself

Methodology

Forem: Hiroki Honda

I built a Lighthouse for MCP tools — it scores your tool definitions on every PR

The problem

What I built

GitHub Action — score on every PR

🟣 mcp.json — 95/100 (Dominant, top 3%)

REST API

Ecosystem data

What's next

Try it

We scanned 4,162 MCP servers. 73% are invisible to AI agents.

73% of MCP servers are invisible

Among the visible: average score 84.7/100

Top scoring servers

The most common defects

What does this mean?

We're calling this ATO

Try it yourself

Methodology

🟣 `mcp.json` — 95/100 (Dominant, top 3%)