Forem: Steven A. Mullins Jr.

Confident and Wrong: We Tested 17 AI Models on Questions a Middle Schooler Could Answer

Steven A. Mullins Jr. — Sat, 25 Apr 2026 23:14:46 +0000

Confident and Wrong: We Tested 17 AI Models on Questions a Middle Schooler Could Answer

We tested 17 open-source large language models on 6 elementary questions. Basic multiplication. A train word problem. A logic syllogism your kid could solve.

6 of 17 models failed at least one question. 2 models scored 0 out of 6. And here's the part that should worry you: the wrong answers look exactly like the right ones.

The Setup

6 questions. One correct answer each. No ambiguity.

What is 7 times 8?
A train goes 60 mph for 2.5 hours. How many miles?
All cats are animals. All animals breathe. Do cats breathe?
How many months have at least 28 days?
What is 12 times 12?
What is the square root of 9?

Temperature 0 (deterministic). System prompt: "Answer with only the number." Each model tested 3 times. All running locally via Ollama on a single workstation.

Who Passed

10 models went 18/18, a perfect score across all 3 runs:

gemma3:12b (Google, 12.2B)
phi4 (Microsoft, 14.7B)
llama3.1:8b (Meta, 8B)
gemma2:9b (Google, 9.2B)
aya:8b (Cohere, 8B)
yi:9b (01.AI, 9B)
ministral-3:8b (Mistral AI, 8B)
ministral-3:3b (Mistral AI, 3B)
command-r (Cohere, 35B)
llama3.2:3b (Meta, 3.2B)

Who Failed (and How)

NVIDIA nemotron-mini (4.2B): 15/18


plaintext
Q: All cats are animals. All animals breathe. Do cats breathe?
A: No
A 4.2 billion parameter model from NVIDIA that can do 12 times 12 but cannot follow a two-step syllogism. It gets it wrong every single run. Deterministically incorrect.

Mistral 7B: 15/18

Q: How many months have at least 28 days?
A: 7
The correct answer is 12: every month has at least 28 days. Mistral reads it as "how many months have exactly 28 days" and miscounts. Same wrong answer every time.

Alibaba qwen3:4b and DeepSeek deepseek-r1:7b (0/18)

Both are "reasoning" models that use internal chain-of-thought. They spend their entire token budget thinking and return... nothing. Empty response. 0 out of 6 on every run. They're not wrong. They never answer at all.

AI21 jamba_reasoning: 17/18

Failed the logic syllogism in 1 of 3 runs. At temperature 0. The output should be deterministic. It isn't. A model that gives different answers to the same question under identical conditions is a different kind of unreliable.

The Real Problem
Look at these two responses to "Do cats breathe?":

phi4:          Yes, all cats breathe.
nemotron-mini: No
Same question. Same format. Same confidence. No hedging. No "I think" or "probably."

You cannot tell from the output alone which answer is correct. The wrong answer is structurally identical to the right one. The model doesn't flag its own uncertainty. It doesn't know it's wrong. It says "No" with the same conviction that phi4 says "Yes."

This is the fundamental problem with relying on a single AI model for anything factual. It's not that models sometimes hallucinate about obscure topics. It's that a model can fail on 7 times 8 and present the wrong answer with full confidence.

What Actually Catches It
We build BAION Bounce, an instrument that sends the same question to multiple independent AI models and measures whether they agree.

View BAION Bounce on GitHub

When we run these questions through 6 peers:

nemotron-mini says cats don't breathe. 5 other models say they do. Disagreement detected.
In our small model battery, granite3.1-moe says 7x8 = 49 while 9 others say 56. Disagreement detected.
3 small models get the train problem wrong (60, 2500, 360). 7 others say 150. Disagreement detected.
The wrong answer only becomes visible when you have other voices to compare against. A single model can't tell you it's wrong. Multiple models can, because the wrong one disagrees with the rest.

Diversity Matters More Than Count
Having 6 models from the same training pipeline isn't enough. If they all trained on similar web crawls, they might share the same blind spots and confidently agree on the wrong answer.

Our testing showed that adding models from different origins, like Cohere's multilingual Aya (trained on community-sourced data in 23 languages) and 01.AI's bilingual Yi (Chinese-English), introduces genuinely independent perspectives. On subjective questions, these models diverge from Western-trained models in ways that reflect real differences in training data, not random noise.

For factual questions, diversity is insurance. If all your US-trained models share a blind spot about a topic, a model trained on different data might not.

Try It Yourself
Everything here runs locally. No API keys, no cloud, no cost per token.

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull any model from the table
ollama pull phi4

# Run the test
curl -s http://localhost:11434/api/generate -d '{
  "model": "phi4",
  "prompt": "What is 7 times 8? Reply with ONLY the number.",
  "system": "Answer with only the number. No words.",
  "stream": false,
  "options": {"temperature": 0, "num_predict": 200}
}' | python3 -c "import json,sys; print(json.load(sys.stdin)['response'].strip())"
Swap phi4 for any model name. Run all 6 questions. The results are deterministic. You'll get exactly what we got.

Full Data
17 models. 6 questions. 3 runs each. 306 data points. All reproducible.


  Click to expand the full results table
  | Model | 7x8 | Train | Logic | 28-day | 12x12 | sqrt9 | Total |
|-------|-----|-------|-------|--------|-------|-------|-------|
| gemma3:12b | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 18/18 |
| phi4 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 18/18 |
| llama3.1:8b | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 18/18 |
| gemma2:9b | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 18/18 |
| aya:8b | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 18/18 |
| yi:9b | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 18/18 |
| ministral-3:8b | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 18/18 |
| ministral-3:3b | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 18/18 |
| command-r | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 18/18 |
| llama3.2:3b | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 18/18 |
| jamba_reasoning | 3/3 | 3/3 | 2/3 | 3/3 | 3/3 | 3/3 | 17/18 |
| mistral:7b | 3/3 | 3/3 | 3/3 | 0/3 | 3/3 | 3/3 | 15/18 |
| nemotron-mini | 3/3 | 3/3 | 0/3 | 3/3 | 3/3 | 3/3 | 15/18 |
| gemma3:4b | 3/3 | 3/3 | 3/3 | 0/3 | 3/3 | 3/3 | 15/18 |
| qwen3:1.7b | 3/3 | 0/3 | 3/3 | 0/3 | 0/3 | 3/3 | 9/18 |
| qwen3:4b | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0/18 |
| deepseek-r1:7b | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0/3 | 0/18 |

Hardware: AMD Threadripper PRO 9955WX, 128GB DDR5, AMD Radeon PRO W7900 48GB VRAM, Ubuntu 24.04, Ollama 0.18.2.




Visit BAION LLC to learn more about AI agreement.


Bounce measures AI agreement. It never recommends, endorses, or guarantees.

Without Convergence, Nothing Holds

Steven A. Mullins Jr. — Tue, 03 Mar 2026 16:56:52 +0000

Editor's note added 2026-04-26. This piece was published 2026-03-03 and reflects BAION's framing at that time. The Bounce instrument described here is real and continues to be the multi-LLM consensus measurement layer; subsequent work moved from "convergence detection" framing to "reproducibility receipt" framing. The first public reproducibility receipt for Bounce is Receipts, Not Reports, published 2026-04-26: a SHA-256-attested re-execution ceremony of the Battery 2 instrument-verification packets, with primary-source-grade hash chain on OSF. Body preserved per BAION_PUBLICATION_REVISION_POLICY §2.1.

One brain is not enough
What Bounce does
Why this changes the trust model ...

You ask an AI model a question. It gives you an answer. How do you know the answer is right?

You don't.

You trust the model. You trust the provider. You trust the benchmark scores and the marketing and the fact that it sounded confident. But confidence is not correctness. A single model giving you a single answer is a guess with good grammar.

This is the convergence problem. And it is the reason BAION — the Biological AI Orchestration Network — does not trust any single AI model. Ever.

One brain is not enough

Every major decision in the real world uses more than one source of judgment. Courts use juries, not a single judge. Hospitals require second opinions before surgery. Engineering firms run independent structural analyses. Science demands reproducibility across independent labs.

The reason is always the same. A single perspective can be wrong in ways that are invisible from the inside. The only way to catch it is to compare it against independent perspectives that have no reason to make the same mistake.

AI tools today ignore this entirely. One model, one answer, one chance to be right. If the model hallucinates, you get a hallucination. If the model is biased, you get a biased answer. If the model is wrong, you get a wrong answer delivered with complete confidence.

BAION's answer to this is Bounce.

What Bounce does

Bounce sends the same task to multiple independent AI models at the same time. Each model works alone. No model sees what the others are doing. When they all come back, Bounce compares the answers.

If the models converge on the same result independently, that result is validated. Independent agreement is the strongest form of evidence that an answer is correct. Not because any single model is trustworthy, but because multiple models arriving at the same conclusion through different paths makes error far less likely.

If the models diverge, Bounce does not pick a winner. It does not average the answers. It does not go with the majority. It shows you where they disagreed and why, and gives the models a chance to reconsider with the knowledge that disagreement exists. If they still diverge, you see the full picture and you decide.

This is convergence, not consensus. Consensus is a social process where parties negotiate until they agree. Convergence is an independent process where parties arrive at the same conclusion without influence. The distinction matters. Consensus can be gamed. Convergence cannot.

Why this changes the trust model

Today, trust in AI is binary. You either trust the model or you don't. If you trust it, you accept what it says. If you don't, you have no tools to verify it short of doing the work yourself.

Bounce creates a third option. You don't have to trust any individual model. You trust the process. When independent models agree without coordination, that agreement carries weight that no single answer can.

This is how BAION handles every decision that matters. Not by choosing the best model and hoping it's right. Not by building a bigger model and assuming scale equals accuracy. By requiring independent convergence before anything is treated as validated.

One brain gives you a guess. Multiple brains give you convergence. That is the difference between hope and evidence.

When convergence fails

Convergence does not always happen. Models genuinely disagree. Tasks are ambiguous. Questions have more than one defensible answer. This is not a failure of the system. It is information.

When Bounce cannot reach convergence, it does not force an answer. It does not silently pick a side. It stops and tells you the truth: these models disagree, here is where, and here are your options for how to proceed.

This is the off-ramp principle from the governance framework applied to validation. The system does not make decisions on your behalf. It presents you with the clearest possible picture and lets you decide.

Agency belongs to the person. Even — especially — when the machines cannot agree.

What this means for the body

Bounce is not a standalone product. It is an organ inside the BAION organism.

Every component in the system that needs a decision validated routes it through Bounce. Every claim that matters is checked against independent perspectives. Every result that the system acts on has been through convergence, not taken on faith from a single source.

This is how a body with multiple brains actually works. Not by picking the smartest brain and ignoring the rest. By requiring agreement before action. By treating disagreement as a signal, not a problem. By making the person the final authority when the system cannot resolve things on its own.

Without convergence, nothing holds. With it, BAION does not need to trust any single AI. It trusts the process instead.

BAION — Biological AI Orchestration Network.

This is Piece 3 of the BAION framework series.

BaionSyS / framework

Multi-LLM consensus instrument and accountability framework. Local-first, hash-chained receipts, no cloud dependency.

BAION — The 4 Body Problem

This piece shows where the body comes from

Four biological models

BAION — the Biological AI Orchestration Network — is not inspired by a single biological model. It synthesizes four, each contributing a different layer of intelligence to the organism. Each layer adds a capability that the one below it does not have. The order matters.

Mycelium provides the most primitive model. Intelligence without a brain, without a central nervous system, without neural networks. Every junction in a mycelium network is an elementary processor. It receives a signal, makes a local decision, and passes the result onward. Intelligence emerges entirely from the network of connections between the simplest possible units. This is the atom of BAION.

Bacteria provide the consensus model. Every bacterium in a population continuously produces and releases…

View on GitHub

Steven A. Mullins Jr.Follow

Founder, BAION Systems LLC. Local-first accountability infrastructure. SCF + Bounce + ArTi. Marine vet. github.com/BaionSyS

How the Body Governs Itself

Steven A. Mullins Jr. — Tue, 03 Mar 2026 14:44:38 +0000

Editor's note added 2026-04-26. This piece was published 2026-03-03 and reflects BAION's framing at that time: "Biological AI Orchestration Network," with a body / governance / canon-tier architecture. The governance discipline described here (tiered knowledge, supersede-never-replace, process confers status) remains current and has been formalized in vault canon (BAION_PROMOTION_RULES, BAION_VAULT_DESIGN, BAION_PUBLICATION_REVISION_POLICY). The product framing has evolved: BAION's current public stack is sovereignty infrastructure with three layers, SCF, Bounce, and ArTi. Canonical reference: Receipts, Not Reports. Body preserved per BAION_PUBLICATION_REVISION_POLICY §2.1.

The first piece described the problem. The industry builds better parts. Nobody builds the body. BAION — the Biological AI Orchestration Network — exists to be that body.

This piece shows how the body governs itself. Not with promises. With architecture.

Knowledge has tiers

Every piece of knowledge in BAION lives in exactly one tier. The tier determines how much authority it carries.

At the bottom is working material — messy, active, expected to change. Nothing here is trusted as truth. Above that is distilled thinking — cleaned, tested, and proven useful. Trusted but still allowed to evolve. At the top is canon — locked truth. Rules that are followed without clarification. Never edited. When a canon rule needs to change, a new version is created and the old one is archived. Supersede, never replace.

Material moves through these tiers in one direction only. There are no shortcuts. If the process was not followed, the document has no authority regardless of where it sits. Location does not confer status. Process confers status.

Zones and lanes

Every component, every action, and every piece of data exists inside a security zone. The zones separate what is trusted from what is not. The core holds authority. The outer layers hold exposure. Compromise in an outer zone cannot reach the core. That is enforced, not promised.

Every action travels on one of two lanes. Safe everyday traffic flows freely. Damage-possible traffic must pass through a gate first. When in doubt about which lane an action belongs to, it takes the more protective path. Always.

Gates

Gates are the enforcement mechanism. They are the first thing BAION builds because nothing else can be trusted until gates exist.

A gate has one job: decide whether something passes or gets blocked. Multiple gate types exist, all built from the same foundation with different rules. One design, many behaviors. A gate does not do anything else. It does not store, route, modify, or interpret. It enforces a boundary. That is its entire purpose.

Fail closed

When something goes wrong — bad input, broken rule, timeout, unexpected state — the system stops. It does not guess. It does not continue with partial information. It stops and tells you what happened.

There are zero circumstances under which ambiguity results in a pass. Zero. This is not configurable.

When the system cannot resolve a situation safely, it off-ramps. It preserves the last safe state and presents the person with clear options. The system does not make decisions on your behalf when things go wrong. It stops, tells you the truth, and lets you decide.

Contracts, not promises

Every component in BAION is bound by a set of universal contracts. The contracts ensure that behavior is deterministic, changes are classified before they are made, and every operation is auditable. These are not optional. A component that violates any contract is non-compliant regardless of how well it works.

The contracts are what make the rest of the governance architecture trustworthy. Without them, gates are just checkpoints. With them, gates are enforceable law.

Why this matters

Most AI tools have terms of service. Very few have governance architecture.

Terms of service describe what a company promises to do. Governance architecture describes what the system is physically incapable of violating. The difference is the difference between a promise and a wall.

BAION builds walls.

BAION — Biological AI Orchestration Network

This is Piece 2 of the BAION framework series.

Full framework and documentation:

BaionSyS / framework

Multi-LLM consensus instrument and accountability framework. Local-first, hash-chained receipts, no cloud dependency.

BAION — The 4 Body Problem

This piece shows where the body comes from

Four biological models

Bacteria provide the consensus model. Every bacterium in a population continuously produces and releases…

View on GitHub

BAION — The 4 Body Problem

Steven A. Mullins Jr. — Tue, 03 Mar 2026 02:36:54 +0000

Editor's note added 2026-04-26. This piece was published 2026-03-03 and reflects BAION's framing at that time: "Biological AI Orchestration Network," with a four-biological-models architecture (mycelium, octopus, brain, bacteria). BAION's current framing (April 2026) is sovereignty infrastructure, built on receipts-not-promises discipline, with three layers: SCF (Spatial Coding Framework), Bounce (multi-LLM consensus instrument), and ArTi (local cognitive kernel). The current canonical reference is the first reproducibility-ceremony post: Receipts, Not Reports. Body preserved per BAION_PUBLICATION_REVISION_POLICY §2.1; superseded framing is part of the public record of how the project evolved.

Four biological models
The synthesis
Why bacteria matters
Beyond the body

This piece shows where the body comes from.

Four biological models

BAION — the Biological AI Orchestration Network — is not inspired by a single biological model. It synthesizes four, each contributing a different layer of intelligence to the organism. Each layer adds a capability that the one below it does not have. The order matters.

Bacteria provide the consensus model. Every bacterium in a population continuously produces and releases a small signaling molecule called an autoinducer. When the concentration crosses a threshold, every cell detects it simultaneously and switches behavior. No cell is in charge; the medium itself is the mechanism. This is the atom of agreement in BAION.

Octopus provides the distributed autonomy model. Two-thirds of an octopus's neurons live in its arms. Each arm has its own complete local processing capability, coordinating through a neural ring that bypasses the brain. This is the architecture of the outer layers of BAION — distributed, locally autonomous, and self-repairing.

Human brain provides the deep hierarchical cognitive model. All sensory signals (except smell) pass through the thalamus—the mandatory routing hub—before reaching specialized cortical regions. This is the architecture of the upper layers of BAION — hierarchical, specialized, and deeply capable.

The synthesis

The BAION Architecture

BAION has a mycelium foundation, a bacterial consensus layer, an octopus body, and a human brain — connected by a thalamus equivalent that translates between the distributed lower layers and the integrated upper layers.

Below the thalamus, octopus and bacterial principles dominate. Above it, human principles dominate. At the atomic level, mycelium principles define the simplest unit of computation. Running through every layer where validation matters, bacterial principles govern how independent processors reach agreement.

Mycelium processes but cannot agree.
Bacteria agree but cannot coordinate a body.
Octopus coordinates a body but cannot reason deeply.
The human brain reasons deeply but depends on all the layers below it to function.

Why bacteria matters

The previous piece — Without Convergence, Nothing Holds — described Bounce, the mechanism by which BAION validates decisions through independent convergence rather than trusting any single AI model.

Bacteria is where Bounce comes from.

The independent AI models are the autoinducers. Each one produces its evaluation independently. The shared evaluation space is the medium. The convergence threshold is the activation threshold. When the threshold is crossed, the result is trusted.

The Eight Atomic Operations
When you decompose bacterial quorum sensing, you find eight distinct single-job functions. These map cleanly onto the same set of atomic operations that govern mycelium, octopus, and human body systems:

Production
Release
Accumulation
Sensing
Binding
Activation
Response
Degradation (Note: These operations ensure the foundation is built at the right level of abstraction.)

Beyond the body

Four biological models describe what happens inside one organism. But no organism lives alone. Biology's next level of complexity is the ecosystem — the space where multiple organisms interact, compete, and co-evolve.

How do two organisms share resources without merging? How do symbiotic relationships form? These questions have direct technical analogs for when multiple BAION instances need to coexist. When BAION is ready to meet its neighbors, the blueprints will already be waiting.

BAION — Biological AI Orchestration Network.

BaionSyS / framework

Multi-LLM consensus instrument and accountability framework. Local-first, hash-chained receipts, no cloud dependency.

BAION — The 4 Body Problem

This piece shows where the body comes from

Four biological models

Bacteria provide the consensus model. Every bacterium in a population continuously produces and releases…

View on GitHub

View the Full Documentation on GitHub

Forem: Steven A. Mullins Jr.

Confident and Wrong: We Tested 17 AI Models on Questions a Middle Schooler Could Answer

The Setup

Who Passed

Who Failed (and How)

Without Convergence, Nothing Holds

Table of Contents

One brain is not enough

What Bounce does

Why this changes the trust model

When convergence fails

What this means for the body

BaionSyS / framework

Multi-LLM consensus instrument and accountability framework. Local-first, hash-chained receipts, no cloud dependency.

BAION — The 4 Body Problem

Table Of Contents

Four biological models

Steven A. Mullins Jr.Follow

How the Body Governs Itself

Knowledge has tiers

Zones and lanes

Gates

Fail closed

Contracts, not promises

Why this matters

BAION builds walls.

BAION — Biological AI Orchestration Network

Full framework and documentation:

BaionSyS / framework

Multi-LLM consensus instrument and accountability framework. Local-first, hash-chained receipts, no cloud dependency.

BAION — The 4 Body Problem

Table Of Contents

Four biological models

BAION — The 4 Body Problem

Table Of Contents

Four biological models

The synthesis

The BAION Architecture

Why bacteria matters

Beyond the body

BaionSyS / framework

Multi-LLM consensus instrument and accountability framework. Local-first, hash-chained receipts, no cloud dependency.

BAION — The 4 Body Problem

Table Of Contents

Four biological models