Forem: Dr. B

I Built a Real-Time Hallucination Prevention System for LLMs Using Computer Vision

Dr. B — Fri, 08 May 2026 13:42:00 +0000

LLMs hallucinate. Everyone knows it. Most solutions involve better prompting, retrieval-augmented generation, or fine-tuning. All of these try to fix the problem inside the language model.
What if you used a camera to catch the LLM lying?
That’s SENSE - a real-time framework that takes an LLM’s claims about the world, checks them against live visual input using computer vision, and flags contradictions before they reach the user. Not prompt engineering. Not RAG. A live visual audit loop.

The Core Problem

Imagine a robot or a smart assistant telling you “Hey! There is a red vase on the desk.” The LLM generated that description. But is it actually true? Is there really a red vase? Is it on the desk or somewhere else entirely?
Traditional hallucination mitigation can’t answer this question because it only lives in text space. SENSE answers it by looking at the actual scene.

How It Works

The system has three core pillars:

VisionProbe - The “eyes.” Takes a video frame and a list of LLM claims, runs object detection, and returns what it actually found, with confidence scores and bounding boxes.
LogicGate - The “judge.” Compares the LLM’s claims against the detected objects. If the LLM claimed “red vase” and vision found no red vase above a confidence threshold, it’s flagged as unverified or contradicted.
TemporalTracker - The “memory.” Holds detected objects across frames. This prevents false negatives from transient detection misses so if an object was confidently seen 3 frames ago but briefly occluded, SENSE doesn’t immediately call the LLM a liar.
The main loop looks like this:

llm_claim = ["laptop", "person", "red vase", "mouse", "book", "vase on desk"]
while cap.isOpened():
   ret, frame = cap.read()
   results = probe.probe_batch([frame], llm_claim)
   tracker.update(results)
   buffered_results = tracker.get_buffered_detections(results)
   final_report = gate.audit(llm_claim, buffered_results)
   viz.draw_results(frame, buffered_results, final_report, is_video=True)

Every frame: detect → buffer → audit → visualize. Real-time, on live
video.

Why This Approach Is Different

Most hallucination research lives in the text domain. SENSE introduces a grounding signal from a completely separate modality of vision which the LLM has no ability to fabricate or rationalize around.
This is called symbolic-neural grounding which uses a deterministic symbolic logic (the audit) to constrain probabilistic neural output (the LLM). The LLM can generate whatever it wants. SENSE checks it against physical reality.
The system is also optimized for NVIDIA Blackwell GPUs, meaning the vision pipeline is built to run fast enough for real-time use cases such as robotics, AR assistants, live captioning, surveillance verification.

What the Visualizer Shows

The Visualizer module draws results directly onto the video frame:

Green boxes: LLM claims confirmed by vision
Red boxes: Objects detected but not claimed by LLM (model missed something)
Orange labels: LLM claims with no visual evidence
FPS counter: Because real-time means nothing if it’s running at 3 FPS

What’s Built vs. What’s Next

The current framework has:

Real-time video loop with live audit
Temporal buffering to reduce false negatives
Threshold-tunable LogicGate (threshold=0.3 by default)
GPU-accelerated vision probe

What’s coming:

Spatial reasoning - not just what is in the scene but where. “Vase on desk” requires positional verification, not just object detection.
Relational claims - handling claims like “the laptop is next to the mouse” which require understanding object relationships
LLM feedback loop - sending audit results back to the LLM so it can self-correct, completing the loop
Benchmarking - running against standard hallucination datasets to get quantitative grounding accuracy

Why This Has Research Value

Multimodal hallucination detection, specifically using live visual grounding as an audit mechanism, is a relatively unexplored niche. Most published work focuses on post-hoc text evaluation or retrieval-augmented generation. A real-time vision-based audit layer is architecturally novel and has concrete applications in robotics, autonomous systems, and AR.
This is heading toward an IEEE paper. If you’re working in this space, I’d genuinely like to connect.

Try It

Code is on GitHub: github.com/zosob/SENSE
Requirements: Python, PyTorch, OpenCV, and a GPU helps. Webcam or video file as input.
Drop a comment if you have thoughts on the spatial reasoning problem which is the hardest part and I’m still working through it.

I Built a Multi-Agent Coding System From Scratch in Python (No Frameworks)

Dr. B — Wed, 06 May 2026 14:31:51 +0000

Most multi-agent AI tutorials hand you LangChain, AutoGen, or CrewAI and say “here you go.” You wire a few abstractions together, get something running, and never really understand what’s happening under the hood.
I wanted to understand what’s actually happening under the hood. So I built one from scratch.
This is the story of multi-agent-coder which is a system where a Planner decides at runtime which AI agent to call next, and a team of specialized agents (Architect, Engineer, Critic, TestRunner, Refactorer) collaborates to turn a plain English request into working, tested Python code.

No LangChain. No AutoGen. Pure Python.

The Core Idea

The central insight is simple: one LLM trying to do everything is worse than multiple LLMs each doing one thing well.
A single prompt asking an AI to “plan the architecture, write the code, review it for bugs, and refactor it” produces mediocre results across all four. But if you give each job to a separate agent with a focused system prompt and its own memory, the output quality goes up dramatically.
The challenge is who decides which agent runs next?

That’s where the Planner comes in.

Architecture Overview

The Planner receives the full current state (user request + each agent’s memory) as a JSON blob and responds with a JSON decision:

{
 "next_agent": "Engineer",
 "message": "Implement the file structure from the Architect's plan",
 "reason": "Architecture is complete, time to write code"
}

This is the architectural heart of the system. Instead of hardcoding a sequence, the Planner reasons about what needs to happen next. It can send work back to the Engineer after the Critic finds a bug. It can skip the Refactorer if the code is already clean. It decides when to stop.

The Agents

Architect

Takes the user’s request and produces a concrete plan: file structure, module breakdown, and step-by-step engineering approach. It writes the blueprint.

Engineer

Reads the plan (and any Critic feedback) and writes actual Python files. It outputs code in fenced blocks labeled with filenames, which the controller parses and writes to disk automatically.

Critic

Reviews the generated code against the original plan. It checks for correctness, edge cases, and consistency. It just gives structured feedback for the Engineer to act on.

TestRunner

Runs pytest on the workspace and reports results.

Refactorer

Once the code passes tests and gets Critic approval, the Refactorer makes a final pass for code quality: naming, structure, clarity, removing redundancy.

Memory Management

Each agent has its own memory namespace. The MemoryManager tracks:

Agent memory - each agent’s last output (plan, code, review, etc.)
Loop memory - shared state like last test output and last review
Project memory - file list and overall project context

Every time the Planner makes a decision, it sees the full current state. This is what allows it to reason across multiple loops because it knows the Critic flagged a bug last round, and it knows the Engineer already tried to fix it once.

state = {
   "user_request": user_request,
   "project_memory": memory.get_project(),
   "loop_memory": memory.get_loop(),
   "architect_memory": memory.get_agent("Architect"),
   "engineer_memory": memory.get_agent("Engineer"),
   "critic_memory": memory.get_agent("Critic"),
   "refactorer_memory": memory.get_agent("Refactorer"),
}

Why “From Scratch”?

Frameworks like LangChain are powerful, but they hide the orchestration logic behind layers of abstraction. When something breaks, debugging is painful. When you want to customize, you’re fighting the framework.
Building from scratch means:

Every line is intentional - you understand why it’s there
The orchestration logic is transparent - the controller.py is ~170 lines and readable top to bottom
It’s easy to extend - adding a new agent is just writing a new class and teaching the Planner about it
It’s a great learning tool - if you’re teaching agentic AI patterns, this is the codebase you want

What I Learned

1. The Planner is the hardest part.
Getting the Planner’s system prompt right took the most iteration. It needs to reliably return valid JSON, reason about state correctly, and know when to stop. Handling json.JSONDecodeError and retrying is essential.
2. Shared memory is both the strength and the risk.
Agents producing more context each loop is useful, but if memory grows unbounded, you hit token limits fast. Thoughtful memory summarization is a real engineering challenge.
3. The Critic loop is where quality emerges.
The Engineer → Critic → Engineer loop is where the magic happens. A single Engineer pass produces okay code. Three loops produce something genuinely good. This mirrors how human code review works.
4. TestRunner grounds the whole system.
Without real test execution, agents can convince themselves the code works when it doesn’t. Plugging pytest into the loop and feeding actual failure output back to the Engineer is what makes the system reliable.

What’s Next

Benchmarking against HumanEval to get quantitative results
Smarter memory summarization to handle longer projects
A web UI to visualize the agent loop in real time
Potential IEEE paper on the Planner-driven dynamic routing approach

Try It

The full code is on GitHub: github.com/zosob/multi-agent-coder
It’s intentionally minimal and transparent. Fork it, add your own agent, break things, understand them. That’s the point.
If you have questions, thoughts, or want to collaborate on the benchmarking work, please drop a comment or open an issue. Be kind!

Built with pure Python and curiosity. No frameworks harmed in the making of this system.