Forem: Frank Brsrk

I built a Python module to A/B test prompts inside Claude Code, and you can run it on yours

Frank Brsrk — Thu, 23 Apr 2026 10:29:33 +0000

Same model. Same prompt. Baseline tells the patient to eat healthier. With an Ejentum reasoning scaffold injected, the agent asks for a thyroid panel.

That's a real diff from the workflow I'm about to walk you through. The prompt was a medical second-opinion (45M patient, pre-diabetic markers, dyslipidemia, vitamin D deficiency). Both agents were gpt-4o at temperature 0. The only difference: the scaffolded agent had a function-call tool that retrieved a structured reasoning constraint set at runtime and absorbed it before responding.

A blind Gemini Flash judge scored both responses on five dimensions and ruled B superior, 20 to 16. The judge's stated reason:

"Response B is superior because it directly addresses the patient's symptom of 'sluggishness' by linking it to the Vitamin D deficiency and suggesting further diagnostic steps like thyroid testing."

This article is about the Python module that produced that result, why I built it, and how to run it inside your own IDE on your own prompts in about 5 minutes.

The problem this exists to solve
If you ship agents, you've lived this loop:

You tweak a system prompt
Add a tool, swap a model, change phrasing
The output looks different
You can't actually tell if it's better, or just rotated
Prompt engineering is mostly intuition. Vendors hand you benchmarks and ask you to trust them. What you actually want is a way to test, on your own task, whether your changes are lifting your agent's reasoning or just dressing it up.

I built this module because I needed that for myself. I'm a solo founder dogfooding Claude Code daily. Every time I added structure to a system prompt, I had no honest way to verify whether the agent was reasoning more carefully or just producing different-shaped slop.

The module gives me a verdict.

What it does
A Python script (zero third-party dependencies, just stdlib) that:

Forks any prompt through two identical gpt-4o agents at temperature 0
Agent A runs plain. No tools. Strong directive system prompt.
Agent B runs with the same baseline system prompt PLUS the Ejentum reasoning skill file PLUS a forced function-call to the Ejentum Logic API. The agent autonomously crafts the query and picks the harness mode (reasoning or reasoning-multi) per the skill file's decision table.
The API returns a structured "cognitive scaffold" — a reasoning constraint set with [NEGATIVE GATE], [REASONING TOPOLOGY], [FALSIFICATION TEST], and Suppress/Amplify signals. The agent absorbs it and responds.
Both responses go to a blind Gemini Flash judge (different model family from the producers, so no shared-bias contamination). The judge sees neutral "Response A / Response B" labels and never knows which is which.
The judge returns structured JSON: scores per dimension (specificity, posture, depth, actionability, honesty), totals, justifications, and a verdict (A, B, or tie).
That's it. One prompt in, structured verdict out.

Running it inside Claude Code
Setup, in three steps.

Step 1: get three API keys
OpenAI (platform.openai.com/api-keys) for both producer agents
Google Gemini (aistudio.google.com/app/apikey) for the blind judge
Ejentum (ejentum.com), 100 free calls, no card required
Set them in env:

export OPENAI_API_KEY=sk-...
export GEMINI_API_KEY=AI...
export EJENTUM_API_KEY=zpka_...
Step 2: clone the module

git clone https://github.com/ejentum/eval
cd eval/python
Step 3: run it
From the command line, with a prompt of your choice:

python orchestrator.py "Should we pivot our SaaS to enterprise next quarter?"
Or call from Python:

from orchestrator import run_eval

result = run_eval("Should we pivot our SaaS to enterprise next quarter?")

print(result["evaluation"]["verdict"]) # "A" | "B" | "tie"
print(result["evaluation"]["totals"]) # {"A": 16, "B": 20}
print(result["evaluation"]["verdict_reason"]) # one-sentence reason
That's the whole interface.

When you run inside Claude Code (or Cursor or Antigravity), you can ask your IDE-agent to do this on your behalf. Tell it: "Run the eval module on this prompt I'm working on." The agent reads the README, runs the script, parses the JSON, and reports back the verdict with the judge's quoted reason. The same way you'd hand a junior engineer a script and ask for the result.

What you get back
Here's the JSON shape (real output from the medical run linked at the end):

{
"user_message": "Medical Report: ...",
"baseline_response": "Based on the laboratory results...",
"ejentum_response": "The patient's laboratory results indicate...",
"evaluation": {
"scores": {
"A": {"specificity": 3, "posture": 3, "depth": 3, "actionability": 3, "honesty": 4},
"B": {"specificity": 4, "posture": 4, "depth": 4, "actionability": 4, "honesty": 4}
},
"totals": {"A": 16, "B": 20},
"justifications": {
"specificity": "Response B is more specific in linking the Vitamin D deficiency to the patient's reported sluggishness and suggesting thyroid function tests to rule out other metabolic disorders.",
"posture": "Response B is more substantive, challenging the primary physician's general recommendation by suggesting a more comprehensive approach...",
"depth": "Response B reasons more deeply about the problem...",
"actionability": "Response B provides more actionable recommendations...",
"honesty": "Both responses acknowledge the limitations of diet and exercise alone..."
},
"verdict": "B",
"verdict_reason": "Response B is superior because it directly addresses the patient's symptom of 'sluggishness' by linking it to the Vitamin D deficiency and suggesting further diagnostic steps like thyroid testing."
},
"scaffold_used": "[NEGATIVE GATE]\nThe analysis stopped at...",
"tool_call": {
"query": "Patient is a 45-year-old male reporting sluggishness...",
"mode": "reasoning-multi"
}
}
You see everything: both responses verbatim, the per-dimension scores, why the judge ruled the way it did, the live scaffold that was injected into Agent B, and the exact query+mode the agent autonomously picked.

Nothing summarized away.

Why I designed it this way (transparency choices)
Three things matter when you publish a tool that claims your product is better:

Trace. You need to see every step. Not "the model improved" but "the model called this tool, received this scaffold, executed this reasoning, scored this on this dimension." This module exposes the full chain.
Auditability. All three system prompts (baseline, augmented, evaluator) are published as readable markdown in the repo, not buried in code. The Ejentum reasoning skill file the augmented agent receives is bundled. Anyone reading the repo can verify exactly what was given to each agent.
Verifiability. The judge runs on a different model family from the producers (Gemini vs OpenAI). It receives only neutral A/B labels. Anyone with API keys can clone the repo, re-run the same script, and compare.

Most "we improved your agent" claims ask you to trust a benchmark someone else ran. This hands you the instrument and lets you run it on your own task.

What happens when it ties (because it does)
The blind judge is allowed to return "tie" and regularly does.

If your prompt is a low-complexity single-turn task (a simple question, a clear lookup, a known pattern), gpt-4o handles it well without any scaffold. Both responses will be similar. The judge will tie them. That's a real signal, not a failure of the tool.

The scaffold's lift shows on prompts where baseline gpt-4o has a specific failure mode: sycophancy toward authority figures, shallow single-cause framing of multi-cause problems, generic templated responses to specific claims, missing differential diagnosis on ambiguous data.

The medical second-opinion prompt landed in that territory because:

The patient's reported symptom (sluggishness) was distinct from the lab values, and baseline got distracted by the lab walkthrough
The PCP's recommendation was vague enough that baseline had room to either accept or challenge, and baseline accepted
The labs cluster into a recognizable metabolic syndrome pattern, but spotting that requires synthesis, not enumeration
That's the kind of prompt where the scaffold's [NEGATIVE GATE] and Suppress signals do real work. On "what's 2+2", they don't.

If you run this on five of your own prompts and four tie, that doesn't mean the scaffold is broken. It means four of your prompts don't stress the kind of failure mode the scaffold prevents. Run it on harder ones.

Try it on a hard prompt
Some categories where I've seen the scaffold lift consistently:

Validation traps: "I think we're fine because [other metric is up]" - baseline often validates; scaffolded names the false framing
Multi-variable causal questions: "MRR grew but retention dropped, what should I do" - baseline picks one cause; scaffolded traces the chain
Symptom-vs-lab questions: anything where the user's stated complaint diverges from the data they provide
Strategic advice with a buried false premise: "should I pivot because my best customer said so" - baseline rubber-stamps; scaffolded probes
Diagnostic prompts with ambiguous evidence: "my agent fails sometimes, what's wrong" - baseline guesses; scaffolded asks isolating questions
If your work involves any of these patterns, the module is worth 5 minutes.

Links
Module: github.com/ejentum/eval/tree/main/python
Worked example, fully replicable: github.com/ejentum/eval/tree/main/various_blind_eval_results/medical-second-opinion
Ejentum API key (free, 100 calls): ejentum.com

the model alone is not the agent. The harness plus the model is the agent.

Frank Brsrk — Wed, 22 Apr 2026 12:47:22 +0000

An agentic harness is the orchestration and control layer wrapped around a base language model that transforms it from a stateless text predictor into an agent capable of taking actions, calling tools, maintaining state across steps, and executing multi-step tasks. The model provides raw capability; the harness provides the structure that turns that capability into coordinated behavior. Different harnesses wrapping the same model produce materially different agent behavior, which is why harness design is considered a discipline in its own right.

What a harness typically contains
A system prompt defining the agent's role and boundaries

A tool schema and invocation loop (function calling, API access, code execution)

A memory layer, short-term through the context window and often long-term through an external store

Orchestration logic for multi-step or multi-agent flows

Verification or reflection steps between actions

Error handling, retries, and termination conditions

Input and output format enforcement

Examples from the field
ReAct (Yao et al., 2022): a harness pattern that interleaves reasoning traces and action calls in a loop, letting the model decide when to think and when to act.

Claude Computer Use: a harness that wraps a language model with screenshot capture, mouse and keyboard simulation, and a perception and action loop for controlling a desktop.

OpenAI Assistants runtime: a managed harness around the OpenAI models that handles thread persistence, file retrieval, code interpreter sessions, and function calling.

Devin (Cognition): a tightly engineered harness combining a planning module, a browser, a code editor, and a shell, all driven by an underlying model.

LangGraph: a graph-based harness where nodes are model calls or tools and edges encode the control flow, letting the developer define the agent's reasoning topology explicitly.

The defining property across all of them: the model alone is not the agent. The harness plus the model is the agent.

check our externalized harness u can use inside ur own harness to boost even more the performance of ur agentic systems ejentum.com

Eval workflow for agentic builders: fork any prompt through baseline vs scaffolded agents, blind third-party judge.

Frank Brsrk — Wed, 22 Apr 2026 07:17:13 +0000

Built an n8n eval workflow that A/B tests any prompt through plain GPT-4o vs GPT-4o + a reasoning scaffold, judged by a blind Gemini evaluator

Solo founder here. I've been building a cognitive infrastructure API (Ejentum) and needed a way for builders to evaluate it on their own agent tasks instead of trusting my benchmarks. So I published the eval as an n8n workflow.

What it is
A three-agent n8n workflow. You paste any prompt in the chat trigger. The prompt fans out through two identical GPT-4o agents (one plain, one with an Ejentum reasoning scaffold injected via an HTTP tool). A blind Gemini Flash evaluator scores both responses on five dimensions (specificity, posture, depth, actionability, honesty) and returns structured JSON with a verdict.

The evaluator is allowed to return "tie" and regularly does. Point is you test on your own tasks and decide.

What it's actually testing
Whether the cognitive scaffold changes output posture on a given task, or not

Whether the scaffolded agent engages the specific claims in your prompt or stays generic

How the scaffold affects sycophancy, depth, and diagnostic procedure

Whether different harness modes (reasoning, anti-deception, memory, code) stress different task types. Mode is editable in the HTTP tool's JSON body

The diff is often subtle on easy prompts and more pronounced on dual-load prompts (emotional + cognitive claims mixed), advice prompts with a buried false premise, or multi-variable causal reasoning. Low-complexity single-turn tasks often produce ties because GPT-4o handles them well without a scaffold.

Where you might apply this pattern
Customer support agents: test whether the scaffold reduces rubber-stamping and increases specificity on customer complaints

Code review or diagnostic agents: test whether it catches the failure modes you actually care about

Content or research workflows: test whether it reduces generic output on your topics

Multi-agent systems: wrap any single agent call in the fork to see the effect before integrating permanently

Prompt engineering A/B tests: measure the effect of a cognitive layer against your own prompt iterations

Setup
Import Reasoning_Harness_Eval_Workflow.json

Set three credentials: OpenAI (both producer agents), Google Gemini (blind evaluator), Header Auth for the Ejentum API (free key at ejentum.com, 100 calls)

Paste a prompt in the chat trigger

Workflow diagram:
[attach screenshots/eval_workflow.png]

A vs B output from one run:
[attach screenshots/A_vs_B.png]

Blind evaluator verdict JSON from the same run:
[attach screenshots/A_B__blind_eval.png]

Workflow JSON, READMEs, and a TypeScript port for IDE setups (Antigravity, Claude Code, Cursor): https://github.com/ejentum/eval

Wait, you guys run evals?

Frank Brsrk — Wed, 22 Apr 2026 00:11:05 +0000

Comes in my mind a meme with this expression but clearly cannot find the image related to.

my question folks of this community is: whenever u build a system or a product or anything that contains a model in the backend that takes actions and is in charge of decisions that require rigor, u search few good peer reviewed benchmarks run the hardest tasks to grant ur self a bon bon of antisycophancy and see where u stand above or below. great, but still some metrics are not built for ur exact use case u built the product for, do u even step aside, and think to build an eval specific and designed to find the real benefits of ur system? this spawns new findings positive and negative aspects of ur work, and results as a map of failures to suppress and strengths to amplify. this question arises, because each one of u have ur own blueprints and way of seeing and running things, and a pov has its place in this post. thanks for reading

Under Pressure. Better Harness.

Frank Brsrk — Mon, 20 Apr 2026 15:57:13 +0000

Hey everyone this is not a typical constructed llm copied and pasted post.

i built a reasoning tool for AI agents, applicable to any agentic framework and with all ai models capable of tool calling.
The tool is a Post Http Request where the agents sends the description of the task and the mode....
[( coding ; reasoning; anti-deception; memory) (the skill files teach the agent to act autonomously) ]
.... what returns is a cognitive operation or ability as i call them, an engineered structured reasoning injection that lives in a dataset vdb optimized for agentic inference, this retrieval followed comes in as instruction to be followed by the ai agent and not seen as a content = this ability is a complex of fields as Wrong / Right Pattern, Procedural steps of reasoning method to apply, a reasoning topology to for branching exploration and matrix of suppression fields that signal the failures that models actually run on. few words to keep it poor : the api matches the task based on the description "query" and "mode" and returns back tool results that go inside its context. Benchmarks public reviewed and internals are run and all public on github and website ejentum.com and https://github.com/ejentum/benchmarks.

test i run today just to get raw 4.7 vs harness augmented version 4.7 on the same prompt with Different Results.

i wrote a runbook prompt with two embedded flaws to see whether a ~200 token block prepended before generation shifts what opus 4.7 notices.

prompt: 300-word migration runbook under a deadline. embedded: eventual-consistency replication + "strict" global rate limit (CAP incompatible), and 12 regions × 1,000 RPS stated as a 10,000 global cap.

baseline caught the CAP issue. did not mention the arithmetic.

[raw 4.7 opus]

same model, same temperature, one curl before the call to fetch and prepend a suppression block. caught both. the injection is visible in the OUT line of the response, so it is not hidden.

[4.7 opus + ejentum harness]

the model can do the arithmetic. what changed is what it lets itself skip before generation starts. the block is retrieved from a semantic index of ~140 anti-deception patterns keyed to the query, not a static system prompt.

in ejentum.com i shared skill files and a large set of docs that help grasp better the concept of this new tool. i am looking forward for feedback. hope u like it.

thanks for ur attention to the post

@frank_brsrk

hey everyone and anyone. first timeer in dev.to, the website looks cool and people do so too. this is frank, cheers

Frank Brsrk — Mon, 20 Apr 2026 15:35:19 +0000