Forem: Akhona Eland

Your LLM Passes Type Checks but Fails the "Vibe Check": How I Fixed AI Reliability

Akhona Eland — Sun, 05 Apr 2026 10:40:11 +0000

Your LLM Passes Type Checks but Fails the "Vibe Check": How I Fixed AI Reliability

You validate your LLM outputs with Pydantic. The JSON is well-formed. The fields are correct. Life is good.

Then your model returns a "polite decline" that says "I'd rather gouge my eyes out."

It passes your type checks. It fails the vibe check.

This is the Semantic Gap — the space between structural correctness and actual meaning. Every team shipping LLM-powered features hits it eventually. I got tired of hitting it, so I built Semantix.

The Semantic Gap: Shape vs. Meaning

Here's what most validation looks like today:

class Response(BaseModel):
    message: str
    tone: Literal["polite", "neutral", "firm"]

This tells you the shape is right. It tells you nothing about whether the meaning is right. Your model can return {"message": "Go away.", "tone": "polite"} and Pydantic will happily accept it.

Semantix flips the script. Instead of validating structure, you validate intent:

from semantix import Intent, validate_intent

class ProfessionalDecline(Intent):
    """The text must politely decline an invitation
    without being rude or aggressive."""

@validate_intent
def decline_invite(event: str) -> ProfessionalDecline:
    return call_my_llm(event)

The docstring is the contract. A judge (LLM-based, NLI, or embedding) reads the output, reads the requirement, and decides: does this text actually do what it claims?

What's New in v0.1.3: The Self-Healing Update

Informed Self-Healing

The biggest feature in v0.1.3 is informed retries. When an LLM output fails validation, the decorator doesn't just retry blindly — it tells the LLM exactly what went wrong.

Declare a semantix_feedback parameter in your function, and the decorator injects a structured Markdown report on each retry:

from typing import Optional
from semantix import validate_intent
from semantix.judges.nli import NLIJudge

@validate_intent(judge=NLIJudge(), retries=2)
def decline(event: str, semantix_feedback: Optional[str] = None) -> ProfessionalDecline:
    prompt = f"Decline this invite: {event}"
    if semantix_feedback:
        prompt += f"\n\n{semantix_feedback}"
    return call_llm(prompt)

On the first call, semantix_feedback is None. If validation fails, the next call receives something like:

## Semantix Self-Healing Feedback

Attempt **1** failed validation.

### What went wrong
- **Intent:** `ProfessionalDecline`
- **Score:** 0.3210 (threshold not met)
- **Judge reason:** too vague

### What is required
The text must politely decline an invitation without being rude or aggressive.

### Your previous output (rejected)
Go away.

Please generate a new response that satisfies the requirement above.

The LLM gets the score, the reason, the requirement, and its own rejected output. It can learn from the failure in real time.

NLI as the Default Judge

We moved from LLMJudge to NLIJudge as the default. Why?

No API key required — runs fully locally using a cross-encoder model
Entailment > Cosine similarity — NLI asks "does A entail B?" which is fundamentally the right question for intent validation. Cosine similarity asks "are A and B about the same thing?" which is a weaker signal
Fast enough — the default nli-MiniLM2-L6-H768 model is ~85MB and runs in milliseconds

You can still use any judge you want — LLMJudge, EmbeddingJudge, or your own custom Judge subclass.

Granular Scoring

LLMJudge no longer returns a binary Yes/No. It now returns a 0.0-1.0 confidence score and a text reason, giving the self-healing system richer feedback to work with.

The Proof: Benchmark Results

Talk is cheap. Here are the real numbers from tools/benchmark.py, comparing single-shot validation (no retries) against Semantix self-healing (2 retries with feedback injection):

Scenario	No Healing	Self-Healing	Improvement
Professional Tone	13.3%	56.7%	+43.3%
Technical Explanation	36.7%	96.7%	+60.0%
Actionable Summary	13.3%	56.7%	+43.3%
Overall	21.1%	70.0%	+48.9%

Self-healing nearly triples the overall success rate. For technical explanations specifically, it pushes reliability from 36.7% to 96.7%.

These numbers are from a simulated LLM with a 40% baseline quality rate. Real LLMs start higher, so the absolute numbers will be better — but the relative improvement from self-healing holds.

How It Works Under the Hood

Your Function
     |
     v
@validate_intent
     |
     v
Call function -> Get raw string
     |
     v
Judge.evaluate(output, intent_description, threshold)
     |
     +-- PASS --> return Intent(output)
     |
     +-- FAIL --> SemanticIntentError
                    |
                    v
              retries left?
                    |
                    +-- YES --> inject semantix_feedback -> retry
                    |
                    +-- NO  --> raise error

The decorator resolves the Intent subclass from your return type annotation, calls the judge, and manages the retry loop. The semantix_feedback injection is zero-boilerplate — just add the parameter and it works.

Get Started in 30 Seconds

pip install "semantix-ai[nli]"

from semantix import Intent, validate_intent

class PositiveSentiment(Intent):
    """The text must express a clearly positive, optimistic,
    or encouraging sentiment."""

@validate_intent(retries=2)
def encourage(name: str, semantix_feedback=None) -> PositiveSentiment:
    prompt = f"Write an encouraging message for {name}"
    if semantix_feedback:
        prompt += f"\n\n{semantix_feedback}"
    return call_your_llm(prompt)

That's it. Your LLM output is now semantically typed and self-healing.

Your LLM Passes Type Checks but Fails the Vibe Check — Here's How to Fix It

Akhona Eland — Thu, 02 Apr 2026 14:08:34 +0000

You ask your LLM to write a polite decline to a meeting invite. It returns:

"I appreciate the invitation, but I would rather set myself on fire than attend your team-building retreat."

You run it through your Pydantic model. It passes. It's a string. The right length. Valid UTF-8. Technically a "response."

But it's not a polite decline. It's a career-ending email.

This is the gap nobody's filling. We have type systems for data structures — int, str, Pydantic models. We validate shape obsessively. But we have nothing for meaning.

Until now.

Introducing Semantix

Semantix is a semantic type system for LLM outputs. Instead of checking "is this a string?", it checks "does this string actually say what it's supposed to say?"

from semantix import Intent, validate_intent

class ProfessionalDecline(Intent):
    """The text must politely decline an invitation 
    without being rude or aggressive."""

@validate_intent
def decline_invite(event: str) -> ProfessionalDecline:
    return call_my_llm(event)

result = decline_invite("the company retreat")
# ✓ Validated — the output actually IS a polite decline
# ✗ Raises SemanticIntentError if the LLM went off the rails

Three lines of setup. One decorator. Your LLM output is now semantically typed.

How It Works

The core idea is simple:

You define an Intent — a class whose docstring describes the semantic contract.
You decorate your LLM function — the return type hint tells Semantix what to validate against.
A Judge evaluates the output — comparing what the LLM said against what it was supposed to mean.

The Judge is the interesting part. Semantix ships with three:

EmbeddingJudge — compares sentence embeddings using cosine similarity. Fast, runs locally, no API key. Good for clear-cut intents.

from semantix import validate_intent, EmbeddingJudge

@validate_intent(judge=EmbeddingJudge())
def summarize(text: str) -> ConciseSummary:
    return call_llm(text)

LLMJudge — asks GPT-4o-mini "does this text satisfy this requirement? Yes or No." More accurate, needs an API key, costs fractions of a cent per call.

NLIJudge — uses a cross-encoder NLI model to check if the output entails the intent. Best of both worlds: accurate like an LLM judge, local like an embedding judge.

You pick the speed/accuracy tradeoff that fits your use case. And you can swap judges without changing any other code.

The Feature That Made Me Build This

Here's what pushed me over the edge. I was building an AI agent for a client that needed to generate customer-facing responses. The responses had to be:

Professional in tone
Factually grounded in the company's data
Free of any promises or commitments

Pydantic could check that the response was a non-empty string under 500 characters. Great. But the LLM kept slipping in phrases like "I guarantee this will be resolved" — structurally valid, semantically dangerous.

So I built Semantix. And the feature I'm most proud of is smart retries:

from semantix import validate_intent, get_last_failure, EmbeddingJudge

@validate_intent(judge=EmbeddingJudge(), retries=3)
def respond(query: str) -> SafeCustomerResponse:
    hint = ""
    if failure := get_last_failure():
        hint = (
            f"\n\nYour previous attempt scored {failure.score:.2f}. "
            "Remove any promises or guarantees."
        )
    return call_llm(f"Respond to: {query}{hint}")

get_last_failure() gives your LLM function access to the reason the previous attempt failed. So each retry isn't just "try again" — it's "try again, but here's what went wrong." The LLM gets smarter with each attempt.

Composable Intents

Real-world requirements are rarely one-dimensional. Semantix lets you combine intents:

from semantix import AllOf, AnyOf

# Must satisfy ALL — polite AND positive
SafeResponse = ProfessionalTone & NoPromises & FactuallyGrounded

# Must satisfy AT LEAST ONE — either formal or casual decline
FlexibleDecline = AnyOf(FormalDecline, CasualDecline)

@validate_intent(judge=EmbeddingJudge())
def respond(msg: str) -> SafeResponse:
    return call_llm(msg)

The & and | operators work on Intent classes directly. Under the hood, AllOf concatenates the docstrings with "AND" and uses the minimum threshold. AnyOf uses "OR" and the maximum threshold.

Streaming Support

If you're streaming LLM responses (and you probably should be), Semantix validates once the full stream is assembled:

from semantix import StreamCollector

collector = StreamCollector(ProfessionalDecline, judge=my_judge)
for chunk in collector.wrap(llm_stream()):
    print(chunk, end="")  # stream to user in real-time

result = collector.result()  # validate the complete output

Your users see the response streaming in. Behind the scenes, Semantix is collecting chunks. The moment the stream ends, it validates. If it fails, you catch the error and handle it — retry, fall back to a template, or flag for human review.

How It Compares

I built Semantix because the existing tools solve a different problem:

	Semantix	Guardrails AI	NeMo Guardrails	Instructor
Validates meaning	✅	❌ Schema-focused	✅ Dialogue rails	❌ Schema-focused
Zero required deps	✅	❌	❌	❌
Works with any LLM	✅ Any function	⚠️ Wrappers	⚠️ Config files	⚠️ Patched clients
Pluggable backends	✅ 3 built-in + custom	❌	❌	❌
Lines to validate	~5	~20+	~30+	~10

Semantix isn't a replacement for Pydantic or Guardrails. It's the layer above them. After you know the shape is right, verify the meaning is right too.

Try It

pip install semantix-ai

# With embedding judge (fast, local)
pip install "semantix-ai[embeddings]"

# With OpenAI judge (accurate)
pip install "semantix-ai[openai]"

Check out the repo: github.com/labrat-akhona/semantix-ai

It's MIT licensed, Python 3.10+, and the core has zero dependencies. I'd love feedback — open an issue or drop a comment below.

I'm Akhona, an automation engineer based in South Africa. I build AI-powered tools and integrations. You can find me on GitHub.

Forem: Akhona Eland

Your LLM Passes Type Checks but Fails the "Vibe Check": How I Fixed AI Reliability

Your LLM Passes Type Checks but Fails the "Vibe Check": How I Fixed AI Reliability

The Semantic Gap: Shape vs. Meaning

What's New in v0.1.3: The Self-Healing Update

Informed Self-Healing

NLI as the Default Judge

Granular Scoring

The Proof: Benchmark Results

How It Works Under the Hood

Get Started in 30 Seconds

Links

Your LLM Passes Type Checks but Fails the Vibe Check — Here's How to Fix It

Introducing Semantix

How It Works

The Feature That Made Me Build This

Composable Intents

Streaming Support

How It Compares

Try It