Forem: Anton Gulin

I Ate My Own Dog Food: How I Benchmarked AI Skills and Proved Eval-Driven Development Works

Anton Gulin — Wed, 15 Apr 2026 18:20:22 +0000

I built a tool to test AI skills. Then I used it on my own project. The benchmarks shocked even me.

As a QA architect, I've spent my career building systems that verify software works correctly. At Apple, we tested everything — every interaction, every edge case, every regression. At CooperVision, I built a Playwright/TypeScript framework from scratch that grew test coverage by 300%.

So when I started working with AI agent skills, I noticed something: nobody was testing them.

You write a SKILL.md file. You try it manually once. Maybe it works for your prompt. You ship it.

There's no automated test suite. No regression testing. No CI pipeline that catches when a description change breaks triggering.

That's a QA problem. I built opencode-skill-creator to solve it.

Then I dogfooded it on a real project. Here's what happened.

The Project: AdLoop Skills for Google Ads

AdLoop is a Google Ads MCP (Model Context Protocol) integration — it connects AI agents to Google Ads and GA4 data through a set of tools.

I created 4 skills for AdLoop using opencode-skill-creator:

adloop-planning — Keyword research, competition analysis, and budget forecasting
adloop-read — Performance analysis, campaign reporting, and conversion diagnostics
adloop-write — Campaign creation, ad management, keyword bidding, and budget changes (spends real money)
adloop-tracking — GA4 event validation, conversion tracking diagnosis, and code generation

The Benchmark Results

opencode-skill-creator's benchmark runs each skill through its eval queries in two configurations:

With skill loaded — the AI has full domain knowledge, safety rules, and orchestration patterns
Without skill — the AI only has bare MCP tool names and descriptions

Skill	Evals	With Skill	Without Skill	Improvement
adloop-write	8	100%	17%	+83pp
adloop-planning	6	100%	21%	+79pp
adloop-read	8	100%	27%	+73pp
adloop-tracking	6	100%	33%	+67pp

But the raw numbers only tell part of the story. The failures without skills aren't just wrong answers — they're dangerous actions.

The Scariest Failure: Real Money at Stake

adloop-write manages campaigns, ads, keywords, and budgets — operations that spend real money. Without the skill:

Added BROAD match keywords to MANUAL_CPC campaigns — the #1 cause of wasted ad spend
Set budget above safety caps ($100 when max is $50) — no guardrail
Deleted campaigns irreversibly without warning — no confirmation, no pause alternative
Batched multiple changes in one call — bypassing review steps

This isn't about "better answers." This is about preventing real financial harm.

GDPR ≠ Broken Tracking

A common scenario: 500 clicks in Google Ads, 180 sessions in GA4. "Is my tracking broken?"

Without the skill, AI diagnosed this as a tracking issue and offered to investigate.

With the skill, AI recognized: "A 2.8:1 ratio is normal with GDPR consent banners. Google Ads counts all clicks. GA4 only counts consenting users. Your tracking is fine."

The #1 false positive in digital marketing analytics, prevented by domain knowledge in the skill.

Don't Trust Google Blindly

Without the skill, AI endorsed Google's recommendations at face value: "Raise budget" with zero conversions. "Add BROAD match" without Smart Bidding.

The skill explicitly states: "Google recommendations optimize for Google's revenue, not yours." It cross-references against conversion data first. The 73% improvement comes from teaching critical thinking, not compliance.

Why This Matters

The same AI model. The same tools. The same prompts. The only variable: whether the skill is loaded. The difference is 67–83 percentage points.

Skills do three things bare tool access doesn't:

Inject domain expertise — GDPR mechanics, budget rules, competition levels
Enforce safety guardrails — budget caps, deletion warnings, one-change-at-a-time
Provide orchestration patterns — when to call which tool, in what order, with what validation

Try It Yourself

npx opencode-skill-creator install --global

Free, open source (Apache 2.0). Works with any of OpenCode's 300+ supported models. Pure TypeScript, zero Python dependency.

→ github.com/antongulin/opencode-skill-creator

Skills are software. Software should be tested.

Anton Gulin is the AI QA Architect — the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET, current Lead Software Engineer in Test. Find him at anton.qa or on LinkedIn.

Eval-Driven Development for AI Agent Skills

Anton Gulin — Wed, 15 Apr 2026 17:58:00 +0000

The Problem with Writing Skills by Hand

You've written a skill for your AI coding agent. It's got clear instructions, proper formatting, a good description. You test it in a session — it works. Ship it, right?

Not so fast.

Skills trigger based on their description field — a 1-2 sentence summary in the SKILL.md frontmatter. And here's the thing: descriptions that seem crystal clear to humans often trigger wrong. Too specific, and the skill never activates when it should. Too broad, and it fires on unrelated prompts.

The result: skills that feel right in theory but fail unpredictably in practice. And there's no systematic way to measure whether a skill is getting better or worse across iterations.

This is the same problem software engineering solved decades ago with automated testing. Skills are software. They need testing too.

What Is Eval-Driven Development?

Eval-driven development is the practice of:

Writing test cases that define expected behavior
Running those tests automatically to measure actual vs. expected outcomes
Using the results to improve iteratively, with quantifiable evidence

For AI agent skills, this means:

Generating test prompts (should-trigger and should-not-trigger queries)
Running each prompt with and without the skill
Comparing outputs to see if the skill actually improves results
Optimizing the description so the skill triggers on the right prompts

The Skill Creation Lifecycle

opencode-skill-creator implements eval-driven development as a structured lifecycle:

Create → Evaluate → Optimize → Benchmark → Install
   ↑                                      |
   └──────────── Iterate ─────────────────┘

1. Create

Start with an intake interview. The skill-creator asks 3-5 targeted questions:

What should this skill enable the agent to do?
When should it trigger?
What output format is expected?
What workflow steps must be preserved exactly?

This captures intent before writing any code.

2. Evaluate

Auto-generate eval test sets — realistic prompts categorized as should-trigger or should-not-trigger. Run each test case twice:

With skill: The agent has the skill loaded
Without skill: The agent runs without it (baseline)

This measures whether the skill actually improves the output for relevant prompts.

3. Optimize

The description optimization loop treats triggering accuracy as a search problem:

For each iteration (up to 5):
  1. Evaluate current description on train set (60%)
  2. Analyze failure patterns
  3. LLM proposes improved description
  4. Evaluate on both train AND test (40%) sets
  5. Select best description by test score

The 60/40 train/test split prevents overfitting. An description that works perfectly on train queries but fails on held-out test queries is overfit — it's memorized specific prompts rather than learning the general pattern.

4. Benchmark

Run the full eval suite across multiple iterations with variance analysis. This answers:

Is the skill getting consistently better?
Are there eval cases where the skill never triggers correctly?
How much variance is there across runs?

The benchmark includes:

Pass rates (with-skill vs. baseline)
Timing data (tokens, duration)
Mean ± standard deviation for each metric

5. Install

Install the final validated skill to project-level (.opencode/skills/) or global (~/.config/opencode/skills/). Only the final version gets installed — eval artifacts stay in the staging directory.

Why This Works

Skills are software

They have inputs (prompts), outputs (agent behavior), and a triggering mechanism (the description). Just like any software, they need testing.

Manual testing doesn't scale

You can test a skill manually in a session, but that's one prompt, one run, no measurement. Eval-driven development gives you 20+ test cases, multiple runs per case, and quantitative metrics.

Description optimization is more impactful than skill content

The description field is the primary triggering mechanism. A perfectly-written skill with a poor description won't trigger. An average skill with an optimized description will trigger reliably. The optimization loop focuses effort where it matters most.

Train/test splits prevent overfitting

If you only test on the same queries you optimize for, descriptions become overfit — they work on those specific prompts but fail on real-world usage. The 60/40 split keeps you honest.

Human review catches what automation misses

The visual eval viewer puts outputs side by side so you can see with your own eyes whether the skill is producing good results. Quantitative metrics tell you if it's triggering correctly; human review tells you if the output is actually useful.

Getting Started

npx opencode-skill-creator install --global

Then ask OpenCode to create or improve a skill. The eval-driven workflow starts automatically.

Apache 2.0, free, open source. Works with any of OpenCode's supported models.

GitHub: https://github.com/antongulin/opencode-skill-creator

opencode-skill-creator is free and open source (Apache 2.0). Star it on GitHub. Install: npx opencode-skill-creator install --global

How to Create Custom OpenCode Skills (Step-by-Step Guide)

Anton Gulin — Sun, 12 Apr 2026 18:52:31 +0000

Why Custom Skills Matter

Out-of-the-box AI coding agents are powerful, but they don't know your team's conventions, your deployment process, or your documentation style. Skills let you encode that knowledge so the agent follows your workflows every time.

But creating skills has been guesswork. You write a SKILL.md file, test it manually in a session, maybe tweak the description, and hope it works. There's no feedback loop, no measurement, no way to know if a change actually improved things.

opencode-skill-creator changes this by providing a structured workflow for the full skill lifecycle: create, evaluate, optimize, benchmark, and install.

Prerequisites

OpenCode installed and configured
Node.js 18+ (for the npm package)
5 minutes

Step 1: Install

One command:

npx opencode-skill-creator install --global

This adds the plugin to your global OpenCode config. Restart OpenCode to activate it.

Verify the install:

ls ~/.config/opencode/skills/skill-creator/SKILL.md

Then ask OpenCode: Create a skill that helps with Docker compose files

You should see it use the skill-creator workflow and tools.

Step 2: Describe What You Want

The skill-creator starts with an intake interview. It asks 3-5 targeted questions about what your skill should do:

What should this skill enable OpenCode to do end-to-end?
When should this skill trigger?
What output format and quality bar are expected?
What workflow steps must be preserved vs. where can the agent improvise?

Don't skip this. The interview captures your intent before any code is written. Think of it as shadowing a teammate — you're the domain expert, the agent is the new hire learning your workflow.

Step 3: Review the Skill Draft

Based on your interview, the skill-creator produces a draft SKILL.md with:

Proper YAML frontmatter (name and description)
Markdown instructions for the agent
Optional supporting files (references, agents, templates)

The draft goes to a staging directory (outside your repo) so your project stays clean:

/tmp/opencode-skills/your-skill-name/
├── SKILL.md
├── agents/
├── references/
└── templates/

Review this draft. Make sure the description is accurate (it's the primary triggering mechanism) and the instructions reflect your actual workflow.

Step 4: Generate Eval Test Cases

The skill-creator automatically generates test cases — realistic prompts that an OpenCode user would actually type:

{
  "skill_name": "docker-compose",
  "evals": [
    {
      "id": 1,
      "prompt": "help me set up a compose file for my Node app with a Postgres database",
      "expected_output": "Skill triggers and provides Docker compose guidance",
      "should_trigger": true
    },
    {
      "id": 2,
      "prompt": "explain how Kubernetes deployments work",
      "should_trigger": false
    }
  ]
}

Good eval queries are realistic and specific — not abstract like "help with containers" but concrete like "ok so my boss just sent me this xlsx file (its in my downloads, called something like 'Q4 sales final FINAL v2.xlsx')..."

Review the eval set. Add or modify test cases that reflect your real usage.

Step 5: Run Evals

The eval system runs each test case twice — once with the skill and once without (baseline). This measures whether the skill actually improves the output.

For each test case:

OpenCode runs with the skill loaded
OpenCode runs without the skill
Both outputs are saved for comparison

Timing data (tokens used, duration) is captured automatically.

Step 6: Review Results Visually

The skill-creator launches an HTML eval viewer:

Call skill_serve_review with:
  workspace: /tmp/opencode-skills/your-skill-name-workspace/iteration-1
  skillName: "your-skill-name"

The viewer shows:

Outputs tab: Each test case with with-skill and without-skill outputs side by side
Benchmark tab: Quantitative metrics — pass rates, timing, token usage
Feedback fields: Leave comments on each test case

Review the outputs. Give specific feedback on what's working and what's not. Empty feedback means "looks good."

Step 7: Iterate and Improve

Based on your feedback, the skill-creator improves the skill:

Applies your feedback
Reruns all test cases (new iteration)
Launches the reviewer with previous iteration for comparison
You review again

Repeat until you're satisfied or feedback is all empty.

Step 8: Optimize the Description

Even with perfect skill instructions, the skill won't trigger correctly if the description field isn't right. The description is what OpenCode reads to decide whether to load your skill.

The optimization loop:

Generates 20 eval queries (should-trigger and should-not-trigger)
Splits them 60/40 into train/test
Evaluates each query 3 times for statistical reliability
Analyzes failure patterns
LLM proposes improved descriptions
Re-evaluates on both train and test
Selects the best description by test score
Repeats up to 5 iterations

# Tell OpenCode:
"Optimize the description of my docker-compose skill"

This takes some time — grab a coffee while it runs.

Step 9: Install the Final Skill

Once you're satisfied with the skill and its description:

Project-level: .opencode/skills/your-skill-name/SKILL.md — available only in this project
Global: ~/.config/opencode/skills/your-skill-name/SKILL.md — available everywhere

# Project-level install
cp -r /tmp/opencode-skills/your-skill-name/ .opencode/skills/your-skill-name/

# Global install
cp -r /tmp/opencode-skills/your-skill-name/ ~/.config/opencode/skills/your-skill-name/

Only the final validated skill gets installed. All eval artifacts stay in the staging directory.

Real-World Example: Docker Compose Skill

Here's what the full workflow looks like in practice:

Ask OpenCode: "Create a skill that helps with Docker compose files"
Interview: The skill-creator asks about your conventions (multi-service vs. single container, development vs. production, preferred base images)
Draft: Produces a SKILL.md with Docker compose best practices, service configuration patterns, volume mount strategies
Eval: Generates test cases like "my api keeps crashing on startup, can you help me debug my compose file" (should trigger) and "what's the difference between Docker and Podman" (should not trigger)
Review: You look at the outputs, give feedback: "the skill should prioritize security configurations in production compose files"
Iterate: Improved skill draft, better outputs
Optimize: Description goes from "Help with Docker compose files" to something much more specific that triggers reliably
Install: Copy to ~/.config/opencode/skills/docker-compose/

Tips for Great Skills

Be specific in the intake interview: The more context you give, the better the draft
Don't skip evals: They catch triggering issues you'd never find manually
Use realistic test prompts: Write them the way you'd actually type them, typos and all
Iterate at least twice: First drafts are rarely perfect
Optimize the description: It's the #1 factor in whether your skill triggers correctly
Install globally for general skills, project-level for specific ones

Getting Started

npx opencode-skill-creator install --global

Then ask OpenCode to create a skill. That's it.

GitHub: https://github.com/antongulin/opencode-skill-creator
npm: https://www.npmjs.com/package/opencode-skill-creator

opencode-skill-creator is free and open source (Apache 2.0). Star it on GitHub. Install: npx opencode-skill-creator install --global