Forem: Abhi Chatterjee

Building AI Evaluation Pipelines: Automating LLM Testing from Dataset to CI/CD

Abhi Chatterjee — Thu, 30 Apr 2026 14:23:00 +0000

Part 2 of a series on testing AI systems in production

In Part 1, we explored why testing AI systems is fundamentally different from traditional software.

We talked about non-determinism, prompt sensitivity, and why unit tests aren’t enough.

Now let’s move from theory to practice.

How do you actually build a system to test AI reliably?

This post walks through a practical approach to building an AI evaluation pipeline—from dataset creation to CI/CD integration.

What is an AI Evaluation Pipeline?

At a high level, an evaluation pipeline looks like this:

Dataset → System → Evaluation → Metrics → Analysis

More concretely:

You define a dataset of test cases
Run them through your AI system
Evaluate outputs using defined metrics
Store and analyze results over time

This becomes your source of truth for system quality.

Step 1: Build a High-Quality Evaluation Dataset

Your evaluation pipeline is only as good as your dataset.

Where data comes from:

Production logs (most valuable)
Synthetic examples (for coverage)
Edge cases and failure scenarios

Example structure:

{
  "input": "What is the refund policy?",
  "expected": "Answer should mention 30-day refund window",
  "context": "Optional (for RAG systems)",
  "metadata": {
    "type": "faq",
    "difficulty": "easy"
  }
}

What makes a good dataset:

Represents real user behavior
Includes edge cases
Covers known failure modes

Insight: Most teams underestimate this step. Dataset quality matters more than model choice in many cases.

Step 2: Define Evaluation Metrics

Unlike traditional systems, correctness isn’t always binary.

You’ll need a mix of evaluation strategies.

Common approaches:

1. Exact match (for structured tasks)

Useful for classification or JSON outputs

2. Semantic similarity

Measures meaning, not exact wording

3. LLM-as-a-judge

Uses a model to evaluate output quality

4. Task success (for agents)

Did the system complete the objective?

Tradeoffs:

Exact match → precise but brittle
Semantic → flexible but fuzzy
LLM judge → scalable but imperfect

The key is combining multiple signals.

Step 3: Run Evaluations

At this stage, you execute your system against the dataset.

A simple evaluation loop might look like this:

results = []

for sample in dataset:
    output = system.run(sample["input"])

    score = evaluator(
        output=output,
        expected=sample.get("expected"),
        context=sample.get("context")
    )

    results.append({
        "input": sample["input"],
        "output": output,
        "score": score
    })

Keep it simple at first. Complexity can come later.

Step 4: Store Results and Enable Debugging

Raw scores are not enough. You need visibility.

Store:

Inputs
Outputs
Scores
Metadata

Add:

Failure tagging
Error categories (hallucination, formatting, etc.)
Trace logs (especially for agents)

This is what allows you to answer:

Why did the system fail?

Without this layer, debugging becomes guesswork.

Step 5: Track Changes Over Time

An evaluation pipeline is not a one-time exercise.

You should be able to answer:

Did the latest change improve performance?
Did hallucination rates increase?
Did a prompt tweak break edge cases?

Track metrics like:

Accuracy
Hallucination rate
Task success rate

Version your datasets and compare results across runs.

Step 6: Integrate with CI/CD

This is where evaluation becomes part of engineering discipline.

Run evaluations when:

Prompts change
Models are updated
Retrieval logic is modified

Example workflow:

Code Change → Run Evals → Compare Metrics → Pass/Fail

You can define thresholds like:

Fail if accuracy drops below X%
Fail if hallucination rate increases

This prevents silent regressions.

End-to-End Flow

Putting it all together:

Dataset
   ↓
Run System
   ↓
Evaluate Outputs
   ↓
Store Results
   ↓
Compare with Previous Runs
   ↓
Trigger Alerts / Decisions

This is your AI quality control loop.

Real-World Example

Let’s say you’re testing a support chatbot.

Before pipeline:

Manual testing
Inconsistent results
Hard to track improvements

After pipeline:

~200 real queries as dataset
Automated evaluation on every update
Clear metrics (correctness, grounding)

Outcome:

Faster iteration
Reduced hallucinations
Better confidence in releases

Common Pitfalls

Even with a pipeline, teams run into issues:

Overfitting to the evaluation dataset
Blind trust in LLM-as-a-judge
Not updating datasets with real usage
Lack of dataset versioning

Avoid treating evals as static—they should evolve with your system.

What’s Next

In the next part of this series, I’ll go deeper into:

Evaluating RAG systems (retrieval + generation)
Measuring context relevance and faithfulness
Common failure patterns in retrieval pipelines

Final Thoughts

AI systems don’t fail loudly—they drift.

An evaluation pipeline gives you a way to detect, measure, and control that drift.

It’s not just about testing once.
It’s about building a system that continuously tells you:

Is my AI still working as expected?

Testing AI Systems in Production: From LLM Evals to Agent Reliability

Abhi Chatterjee — Tue, 21 Apr 2026 16:34:27 +0000

Testing AI Systems in Production: From LLM Evals to Agent Reliability

Practical strategies to evaluate LLMs, RAG pipelines, and AI agents in real-world systems

Most AI systems don’t fail in development — they fail quietly in production.

Not with crashes, but with subtle errors: hallucinations, incorrect tool usage, or inconsistent outputs that slip past traditional tests.

The root problem is simple: we are still trying to test probabilistic systems using deterministic testing strategies.

This is Part 1 of a series on testing AI systems in production.
In this post, we’ll build a practical mental model and testing strategy.
In upcoming parts, I’ll go deeper into evaluation pipelines, RAG testing, and agent-level reliability.

Why Traditional Testing Breaks for AI

In traditional software, a given input maps to a predictable output.

That assumption breaks with AI systems.

Key differences:

Outputs are non-deterministic
Correctness is often subjective
Ground truth is hard to define
Behavior can shift with small prompt changes

This means unit tests alone are not enough. You need layered evaluation strategies.

The AI Testing Stack (A Practical Mental Model)

Think of AI testing as a stack rather than a single technique:

+--------------------------------------------------+
| Agent / Workflow Testing (multi-step reasoning)   |
+--------------------------------------------------+
| System Testing (RAG, tools, memory)              |
+--------------------------------------------------+
| Prompt Testing (instructions, few-shot behavior) |
+--------------------------------------------------+
| Model Evaluation (benchmarks, accuracy)          |
+--------------------------------------------------+

Each layer introduces different failure modes — and requires different testing approaches.

1. Model-Level Evaluation

This is the foundation: evaluating raw model capability.

Typical techniques:

Benchmark datasets (task-specific)
Accuracy, precision/recall (structured outputs)
BLEU / ROUGE (for text similarity)

But strong benchmark performance does not guarantee real-world reliability.

Example:
A model performing well on QA benchmarks may still hallucinate on domain-specific queries.

Takeaway: Model evals are necessary, but insufficient.

2. Prompt-Level Testing

Prompts are effectively your “programming layer” — and they are fragile.

What to test:

Consistency across paraphrased inputs
Sensitivity to prompt changes
Instruction adherence
Edge cases and adversarial phrasing

Example test case:

Input: "Summarize this document in 3 bullet points"
Variation: "Give me a short summary in bullets"
Expected: Similar structure and quality

Small wording changes shouldn’t break behavior — but often do.

Approach:

Maintain a golden dataset
Run regression tests when prompts change

3. System-Level Testing (RAG, Tools, Pipelines)

Once you introduce retrieval or external tools, complexity increases.

Typical components:

Retrieval (vector DB / search)
Context construction
Tool/API calls
Output formatting

Common failure modes:

Irrelevant retrieval results
Missing critical context
Incorrect tool selection
Hallucinated answers despite available data

Example RAG flow:

User Query
    ↓
Retriever → Context
    ↓
LLM → Response

What to evaluate:

Context relevance — Did we fetch the right data?
Faithfulness — Did the model use the context?
Answer correctness

4. Agent-Level Testing (Where Things Get Hard)

Agents introduce multi-step reasoning, planning, and state.

Example loop:

User Goal
   ↓
Plan → Tool Call → Observe → Repeat
   ↓
Final Answer

Common failures:

Infinite loops
Wrong tool usage
Partial task completion
Confident but incorrect outputs

How to test agents:

1. Scenario-based testing

Define end-to-end tasks
Measure success rate and correctness

2. Simulation environments

Mock tools and external dependencies

3. Trace inspection

Log actions, inputs, outputs
Analyze decision paths

This is essential for debugging complex failures.

Core Testing Techniques That Work

1. Golden Datasets

Curate:

Real user queries
Edge cases
Known failure scenarios

This becomes your most valuable testing asset.

2. LLM-as-a-Judge

Use a model to evaluate outputs.

Example:

"Is this answer correct and grounded in the context?"

Pros:

Scalable
Flexible

Cons:

Can be biased
Requires validation

3. Regression Testing

Every change should trigger evaluation:

Prompt updates
Model changes
Retrieval modifications

Track:

Accuracy
Hallucination rate
Task success

4. Red Teaming

Actively try to break the system:

Prompt injection
Jailbreak attempts
Malicious inputs

Critical for production readiness.

A Practical Testing Workflow

Define Metrics
     ↓
Build Eval Dataset
     ↓
Run Automated Evals
     ↓
Analyze Failures
     ↓
Fix (Prompt / System / Model)
     ↓
Repeat (CI/CD Integration)

In practice:

Version control your eval datasets
Automate evaluations in CI/CD
Track performance over time

Real-World Example: Support Chatbot

Scenario:

A chatbot answering queries from a knowledge base.

Issues:

Hallucinated responses
Ignoring retrieved context
Inconsistent tone

Solution:

Built dataset (~200 real queries)
Added evaluation metrics (correctness, grounding)
Introduced regression testing
Added adversarial test cases

Result:

Reduced hallucinations
Improved consistency
Faster iteration

Key Challenges (That Don’t Go Away)

Non-determinism
Expensive evaluations
Limited ground truth
Continuous model drift

The goal isn’t perfection — it’s controlled reliability.

What’s Next

In the next parts of this series, I’ll go deeper into:

Building automated evaluation pipelines
Testing RAG systems (metrics + pitfalls)
Agent evaluation and tracing strategies
Tooling and implementation patterns

Final Thoughts

AI testing is not a single technique — it’s a discipline.

The teams that succeed:

Test at multiple layers
Build strong evaluation datasets
Automate aggressively
Continuously learn from failures

Because in AI systems, what you don’t test is exactly where things break.