TraceMind v2 — I added hallucination detection and A/B testing to my open-source LLM eval platform

Aayush kumarsingh — Tue, 14 Apr 2026 05:39:42 +0000

What changed since v1

When I posted the first version of TraceMind, I got one clear piece of feedback: "this is useful but I need to know if my AI is making things up, not just scoring low."

So I built hallucination detection. Then while building it I realized I needed a way to compare prompts systematically. So I built A/B testing too.

Here's what's new and how I built it.

The original problem (unchanged)

I was building a multi-agent orchestration system. Three days after deploying, I changed a system prompt. Quality dropped from 84% to 52%. I found out 11 days later from a user complaint.

TraceMind was built to catch this on day zero.

What's new in v2

Hallucination detection

The endpoint takes a question, the AI's response, and optional ground truth context. It extracts individual claims from the response, checks each one against the context, and returns a structured result:

{
  "has_hallucinations": True,
  "overall_risk": "high",
  "claims": [
    {
      "claim": "We offer 60-day refunds",
      "verdict": "hallucination",
      "reason": "Context says 30-day refunds only"
    }
  ]
}

The key architectural decision: claim extraction and verification are separate LLM calls. The first call extracts atomic claims. The second verifies each claim against ground truth. This is more reliable than asking one model to do both.

Prompt A/B testing

You give it two system prompts and a dataset. It runs both prompts against every test case and compares results.

The interesting part is the statistical layer. A naive implementation would just compare average scores. But with small datasets (5-20 cases),average score differences are often noise. I added Mann-Whitney U test and Cohen's d to give a confidence score on whether prompt B is actually better or just randomly different.

{
  "prompt_a_score": 6.2,
  "prompt_b_score": 8.1,
  "winner": "B",
  "confidence": "high",
  "cohen_d": 1.4,
  "p_value": 0.03
}

Verification suite

I built a 44-test verification script covering all 11 feature areas. Running python verify_all.py hits every endpoint end-to-end against a real running server and reports pass/fail. This was more useful than unit tests for catching integration issues.

What I'd still do differently

The same things from v1, plus one new one: the hallucination detection is synchronous. For production use it should be a background job like span scoring. A user with 1000 traces would need to wait for each one — that doesn't scale.

Try it

GitHub: https://github.com/Aayush-engineer/tracemind

pip install tracemind
from tracemind import TraceMind
tm = TraceMind(api_key="...", project="my-app",
               base_url="https://tracemind.onrender.com")

@tm.trace("llm_call")
def your_function(msg): ...  # unchanged

Self-hosted, free, no vendor lock-in.

If you're building with LLMs — I'd genuinely love to know
what breaks when you try it.

I built an open-source LLM eval platform with a ReAct agent that diagnoses quality regressions

Aayush kumarsingh — Thu, 09 Apr 2026 11:45:30 +0000

The problem that made me build this

I was building a multi-agent orchestration system. It worked great
in testing. I deployed it. Three days later I changed a system prompt.
Quality dropped from 84% to 52%. I found out 11 days later when a
user complained.

This is the most common failure mode in LLM applications. Unlike
traditional software where a bug throws an exception, bad LLM outputs
look like valid responses. They just happen to be wrong, unhelpful,
or unsafe. You need systematic measurement to catch this.

I looked for existing tools. Langfuse is good but expensive at scale for self-hosted teams.
Braintrust doesn't have a free self-hosted option. Helicone doesn't do
evals. I built TraceMind.

What TraceMind does

Three things:

1. Automatic quality scoring
Every LLM response is scored 1-10 by another LLM acting as judge
(LLM-as-judge pattern). I use Groq's free tier — llama-3.1-8b-instant
for fast scoring, llama-3.3-70b for deep analysis. The score runs in
the background, never blocking your application.

2. Golden dataset evals
You define expected behaviors once:

ds = tm.dataset("support-v1")
ds.add("I want a refund", expected="acknowledge and ask for order number")
ds.push()

result = tm.run_eval("support-v1", function=your_agent.run)
result.wait()
print(f"Pass rate: {result.pass_rate:.0%}")  # Pass rate: 87%

3. AI agent that diagnoses regressions
This is the part I'm most proud of. You can ask:

"Why did quality drop yesterday?"
"What are the most common failure patterns?"
"Generate test cases for billing question failures"

The agent implements the ReAct pattern with 6 tools and 4 memory types.

The architecture decisions that matter

Parallel eval execution with asyncio.Semaphore

The naive approach runs LLM judge calls sequentially.
For 100 test cases at 500ms each = 50 seconds.

I use asyncio.Semaphore(3) to run 3 evaluations concurrently:

semaphore = asyncio.Semaphore(max_concurrent)
tasks = [run_case(ex, system_fn, criteria, semaphore) for ex in examples]
for coro in asyncio.as_completed(tasks):
    result = await coro

100 cases now takes ~17 seconds. The semaphore limit exists because
Groq's free tier has rate limits — I tuned it to stay under the threshold.

The ReAct agent with semantic memory

The agent has 4 memory types:

In-context: conversation history within the session
External KV: project config from database
Semantic: past failures in ChromaDB with sentence-transformers embeddings
Episodic: past agent run results in SQLite

When you ask "why did quality drop?", the agent:

Searches ChromaDB semantically for similar past failures
Fetches recent low-scoring traces from the database
Runs a targeted eval on the failure category
Uses Opus-equivalent model to analyze root cause
Generates new test cases to prevent future recurrence

I intentionally avoided LangChain. The ReAct loop is 80 lines of
readable Python. When something breaks at 3am, you want to read
your own code.

Background worker for async scoring

The HTTP ingestion endpoint returns in <10ms regardless of batch size.
Scoring runs in a background worker that polls every 10 seconds:

async def _score_unscored_spans(self):
    spans = fetch_unscored(limit=20)
    for span in spans:
        score = await self._score_span(span.input, span.output)
        save_score(span.id, score)

The worst thing an observability tool can do is slow down the system
it's monitoring. Scoring is completely decoupled from ingestion.

Local embeddings — no OpenAI dependency

I use sentence-transformers all-MiniLM-L6-v2 for ChromaDB embeddings.
It runs locally, downloads once (~90MB), works offline, zero API cost.
This was a deliberate choice — I wanted the tool to work completely
free with no external dependencies beyond Groq for LLM calls.

What I'd do differently in production

Multi-tenancy: Row-level security instead of project-level isolation
Celery + Redis instead of asyncio background worker for horizontal scaling
Streaming eval results via WebSocket — see case-by-case progress in real time
Alembic migrations from day one (I added these later)

Try it

Live demo: https://tracemind.vercel.app
GitHub: https://github.com/Aayush-engineer/tracemind

3-line setup:

pip install tracemind
from tracemind import TraceMind
tm = TraceMind(api_key="...", project="my-app", 
               base_url="https://tracemind.onrender.com")

@tm.trace("llm_call")
def your_function(msg): ...  # your code unchanged

If you're building with LLMs and want to know if they're actually
working — I'd love feedback.

Forem: Aayush kumarsingh

TraceMind v2 — I added hallucination detection and A/B testing to my open-source LLM eval platform

What changed since v1

The original problem (unchanged)

What's new in v2

Hallucination detection

Prompt A/B testing

Verification suite

What I'd still do differently

Try it

I built an open-source LLM eval platform with a ReAct agent that diagnoses quality regressions

The problem that made me build this

What TraceMind does

The architecture decisions that matter

Parallel eval execution with asyncio.Semaphore

The ReAct agent with semantic memory

Background worker for async scoring

Local embeddings — no OpenAI dependency

What I'd do differently in production

Try it