Forem: System Rationale

Part 3 — Making Gemma 4 Agents Production-Ready: Guardrails, Structured Outputs, and Self-Healing Systems

System Rationale — Fri, 10 Apr 2026 02:58:00 +0000

The uncomfortable truth about AI agents

By the time most teams reach this stage, they’ve already built:
• a multi-step workflow
• a supervisor + worker setup
• integration with tools and APIs

And yet, the system still fails in production.

Not because the model is weak.

But because the system is non-deterministic.

⸻

Where reliability actually breaks

In real deployments, failures don’t come from “bad reasoning”.

They come from:
• malformed outputs (invalid JSON, missing fields)
• inconsistent decisions across steps
• uncontrolled retries and loops
• unsafe or duplicated side effects

You can’t patch these with better prompts.

You need contracts, validation, and control layers.

⸻

From probabilistic outputs → deterministic contracts

The first shift is simple but critical:

Treat every model output as untrusted input

Instead of accepting free-form text, define strict schemas using
Pydantic or
PydanticAI.

⸻

Example: Root Cause Contract

class RootCause(BaseModel):
service: str
confidence: float
error_type: Literal["OOM", "MemoryLeak", "Config", "Network"]
evidence: list[str]
next_steps: list[str]

This does three things:
1. Forces the model into a structured format
2. Enables automatic validation
3. Creates a stable interface between system components

⸻

What this looks like in practice

A production pipeline becomes:

LLM Output → Schema Validation → Accept / Reject → Retry / Escalate

This is no longer “AI responding”.

It’s a controlled data pipeline.

⸻

The self-healing loop

Validation is only half the system.

The real reliability comes from how you handle failure.

⸻

Controlled retry pattern
1. Generate output
2. Validate against schema
3. Capture validation error
4. Feed error back into model
5. Retry with constraints
6. Stop after N attempts

⸻

Example failure feedback

Instead of:

“Try again”

You send:

“Field confidence must be a float between 0 and 1.
error_type must be one of [OOM, MemoryLeak, Config, Network].
Fix the JSON.”

This transforms the model into a self-correcting system.

⸻

Why Gemma 4 fits this model well

With Gemma 4, this loop becomes practical at scale.

Because:
• thinking mode improves structured reasoning
• MoE architecture reduces cost per retry
• long context allows passing validation history
• tool calling aligns with structured outputs

This is critical.

Self-healing systems require multiple attempts.
Cost-efficient inference makes that viable.

⸻

Guardrails are not optional

Without guardrails, your system will eventually:
• loop indefinitely
• call the wrong tools
• execute unsafe actions

⸻

Minimum guardrail layer

You should implement:

Step limits
• Hard cap on number of node executions
Error classification
• Retry: timeouts, rate limits
• Fail: schema errors, auth issues
Circuit breakers
• Stop calling failing dependencies
Human-in-the-loop
• Required for destructive actions

⸻

Visualizing guardrails in the system

Think of your system as:

State Machine
↓
Validation Layer
↓
Guardrails
↓
Execution

Each layer reduces uncertainty.

⸻

Going beyond validation: adaptive systems with DSPy

Validation ensures correctness.

But how do you improve the system over time?

⸻

Enter DSPy

DSPy treats your pipeline as a program:
• inputs → outputs
• defined signatures
• measurable metrics

It allows you to:
• run evaluation datasets
• measure output quality
• optimize prompts automatically

⸻

What this unlocks

Instead of manual tuning:
• the system detects failures
• adjusts prompts / examples
• improves over time

This is the missing layer in most agent systems.

⸻

Combining everything: the deterministic stack

A production-ready Gemma 4 system looks like:

State Graph (LangGraph)
↓
Supervisor (Gemma 4 thinking mode)
↓
Workers (task-specific agents)
↓
Pydantic Validation
↓
Guardrails
↓
DSPy Evaluation + Optimization

Each layer solves a specific failure mode.

⸻

Real-world application: autonomous DevOps agent

Example workflow:

Trace
• collect logs, metrics, events

RootCause
• detect anomalies (OOMKilled, memory leaks)

Plan
• decide corrective action

Fix
• restart pods, scale services, or open PR

Verify
• confirm system recovery

⸻

Why this works

Because:
• every step is validated
• every action is controlled
• every failure is recoverable

This is not an “AI agent”.

It’s a deterministic system with AI inside it.

⸻

Practical implementation stack

If you’re building this today:
• Model: Gemma 4 (26B MoE)
• Orchestration: LangGraph
• Validation: Pydantic / PydanticAI
• Guardrails: custom + middleware
• Evaluation: DSPy

⸻

Resources

Core
• https://github.com/google-deepmind/gemma
• https://github.com/google/gemma_pytorch

Orchestration
• https://github.com/langchain-ai/langgraph
• https://github.com/langchain-ai/langgraph-example

Validation & Guardrails
• https://github.com/pydantic/pydantic-ai
• https://github.com/jagreehal/pydantic-ai-guardrails

Evaluation & Optimization
• https://github.com/stanfordnlp/dspy
• https://github.com/Scale3-Labs/dspy-examples

Real-world systems
• https://github.com/qicesun/SRE-Agent-App

⸻

Final perspective

Most teams are still chasing:
• better prompts
• better models
• better outputs

That’s not where reliability comes from.

⸻

Reliability comes from:
• explicit state
• strict contracts
• controlled execution
• continuous evaluation

Designing Multi-Agent Systems with Gemma 4: Supervisor and Worker Pattern

System Rationale — Wed, 08 Apr 2026 02:49:00 +0000

Most agent implementations fail for a simple reason:

They try to make one model do everything.

That approach does not scale.

⸻

The limitation of single-agent systems

When one agent is responsible for:
• understanding context
• making decisions
• calling tools
• validating outputs
• executing actions

you introduce uncontrolled complexity.

The result is:
• inconsistent behavior
• hallucinated decisions
• poor failure recovery

This is not a model limitation. It’s a design issue.

⸻

The correct pattern: separation of responsibilities

A more stable architecture separates concerns into two layers:

Worker agents

Each worker is narrowly scoped:
• log analysis
• root cause detection
• code or PR generation
• infrastructure interaction

Workers should be predictable and task-specific.

⸻

Supervisor agent

The supervisor coordinates the system.

With Gemma 4, this becomes significantly more powerful due to its thinking mode.

The supervisor:
• reads the global system state
• decides which worker to invoke
• validates outputs before progressing
• handles retries and escalation

⸻

Why thinking mode matters

Gemma 4 introduces structured reasoning behavior, often referred to as a “thinking” phase.

In practice, this allows the supervisor to:
1. evaluate multiple possible actions
2. internally reason about risks and outcomes
3. select the next state transition

This creates a separation between:
• internal reasoning
• external actions

That separation is critical for reliability.

⸻

Putting it together: state-driven execution

A typical flow looks like this:
• Trace — collect logs, metrics, events
• RootCause — identify likely issue
• Plan — decide next action
• Fix / Escalate — execute or request approval
• Verify — confirm resolution

Each step is a node in a state machine.

The supervisor controls transitions between nodes.

⸻

What this architecture fixes

This approach eliminates common issues:
• uncontrolled loops → bounded by state transitions
• inconsistent decisions → centralized in supervisor
• retry chaos → handled explicitly in graph
• unclear execution → traceable at each node

⸻

What most teams still get wrong

Even with this architecture, many implementations fail because they:
• skip output validation
• allow unlimited retries
• treat tool calls as always safe
• don’t distinguish between reversible and irreversible actions

These are not optional concerns.

They define whether your system is production-ready.

⸻

Resources
• https://github.com/langchain-ai/langgraph
• https://github.com/emarco177/langgraph-course
• https://codelabs.developers.google.com/aidemy-multi-agent/instructions

⸻

In the final part:

How to make Gemma 4 agents deterministic using structured outputs, guardrails, and self-healing pipelines

Gemma 4 MoE: frontier quality at 1/10th the API cost

System Rationale — Tue, 07 Apr 2026 02:43:00 +0000

Gemma 4 MoE: frontier quality at 1/10th the API cost

gemma4 #moe #llm #openweights #aiinfra

Continuing from Part 1 — once you have a proper state machine architecture, the next question is: which model runs inside it?

For high-volume agent workloads, my pick is Gemma 4 26B MoE.

Here's the actual reasoning.

What MoE means (no marketing)

Most LLMs are dense. A 30B dense model activates 30B parameters per token — every single one, every single call.

Mixture-of-Experts works differently:

Total parameters: ~25B
Active parameters per token: ~3.8B
A router picks 8 experts out of 128 per token

Near-30B quality. ~4B compute per token.

Not a trick. Just a better architecture for inference-heavy workloads.

The real cost math

GPT-4o: $2.50 per 1M input tokens, $10 per 1M output tokens.

Gemma 4 is open-weight. Host it yourself on an A100. At volume — thousands of agent runs per day — the math flips hard in your favor.

This matters specifically for agents because agents are token-heavy. One agent run might involve 5–20 LLM calls, each with a full context window. At GPT-4o pricing, that adds up fast. On self-hosted Gemma 4, it stays manageable.

What Gemma 4 gives you specifically for agents

256K context window — feed full log files, traces, conversation history in one shot
Native function calling — no wrapper hacks for tool use
Thinking mode — model reasons privately before acting (critical for Supervisor agents — Part 3)
Multimodal input — pass Grafana screenshots directly to it

When GPT-4o still wins

Being honest here:

Need sub-second latency, don't control infra → GPT-4o
Need best reasoning with zero setup → GPT-4o
Running under 10k tokens/day → pricing doesn't matter, use anything

Gemma 4 wins when:

You need cost control at volume
Data can't leave your infra (regulated, private)
You're comfortable with GPU infra or a cloud GPU provider

Getting started

ollama pull gemma4:26b

Local testing done. For production throughput, pair with vLLM.

Part 3 is the architecture — Supervisor + Worker agents using Gemma 4's thinking mode inside a LangGraph state machine. That's where 99.9% reliability actually becomes achievable.

— System Rationale

Gemma 4 on Mobile: Which Model to Load (E2B vs E4B) + Real Implementation Guide

System Rationale — Mon, 06 Apr 2026 18:09:21 +0000

Hey devs 👋
I’ve been hands-on with Gemma 4 since it dropped 4 days ago and honestly — the E2B and E4B variants are the first models that actually feel practical for real mobile apps.
Here’s the no-BS guide I wish I had: which model to load for your use case + exactly how to load it on Android, iOS, React Native, and web.

Which Gemma 4 model should you actually load?

E2B (≈5.1B total params, only 2.3B active thanks to Per-Layer Embeddings)
→ Your default for phones.
Use cases: offline tutor, smart replies, chat rephrasing, note summarization, safety filters, anything battery/RAM sensitive.
Cold start is fast, runs smooth on mid-range devices.
E4B (≈8B total, 4.5B effective)
→ Sweet spot for flagship phones or when you need noticeably better reasoning + native audio + image understanding.
Use cases: multimodal (photo → description), longer context tasks, or when E2B feels a bit “light”.
26B A4B MoE or 31B
→ Skip these on mobile. Only for laptops, desktops, or server-side heavy lifting.

Rule of thumb I use: start with E2B. Only bump to E4B if users complain about quality or you need audio/image input.

How to actually load the model (the part that matters) Android

Easiest path: AICore Developer Preview (system-wide Gemma 4, zero weights to ship).
Just call the ML Kit GenAI Prompt API — Google handles hardware delegation (NPU/GPU).
For full control in your app: LiteRT-LM
Download the quantized .task file (4-bit) from HF
Use on-demand Play Asset Delivery so your APK stays <100 MB
Load in background with Coroutines → never block UI
Use streaming callback so tokens appear live

iOS

MediaPipe LLM Inference API is the official way.
Convert to MediaPipe task format → memory-map the weights → Metal/MPS acceleration.
Warm up the model during app idle time so first token feels instant.

React Native

Native TurboModule (Kotlin + Swift) is non-negotiable.
Keep the entire model + inference in native code.
Expose only generateResponse(prompt, options) and onToken events back to JS.
Never run inference on the JS thread — you will OOM and crash.

Web

MediaPipe + WebGPU (works surprisingly well in Chrome).

Universal tips that saved my ass:

Always use 4-bit quantized version (Q4_K_M or LiteRT equivalent)
Never bundle the full model in the APK/IPA — download on first user opt-in
Cap context at 4K–8K for mobile (128K is possible but eats RAM)
Stream tokens. Always. Users hate staring at a blank screen.

Security bonus: because E2B/E4B run 100% offline, user data (exam answers, private notes, photos) never touches your servers. Huge privacy win.
I’m using this exact stack right now for an offline-first tutor app and it’s buttery smooth.
Drop your use case below and I’ll tell you which variant + exact loading path I’d pick for it.
Useful resources (all fresh as of April 2026):

Official Gemma 4 announcement: https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/
Model card + sizes: https://ai.google.dev/gemma/docs/core/model_card_4
Full model overview (E2B/E4B details): https://ai.google.dev/gemma/docs/core
Android AICore + ML Kit guide: https://android-developers.googleblog.com/2026/04/AI-Core-Developer-Preview.html
LiteRT-LM mobile deployment: https://ai.google.dev/edge/litert-lm
Hugging Face E2B/E4B quantized models: https://huggingface.co/google/gemma-4-E2B-it

Who’s actually shipping Gemma 4 on device right now? Show me your stack 🙌

Why your LLM agent fails at 3 AM (and how state machines fix it)

System Rationale — Mon, 06 Apr 2026 09:35:26 +0000

Why your LLM agent fails at 3 AM (and how state machines fix it)

agents #llm #langgraph #systemdesign #aiinfra

I've been reading postmortems from teams running LLM agents in production.

Same failure every time.

Not model quality. Not prompt engineering. The architecture.

Most AI agents today still look like this:

User Input → LLM Call → Tool Call → LLM Call → Output

A chain. Linear. Stateless. Hopeful.

Works great in a notebook. Breaks under real load.

The 4 ways chains die in production

1. Infinite loops
Agent calls a tool → tool fails → agent retries → tool fails → agent retries.
No exit condition. You're burning tokens at 3 AM while sleeping.

2. No checkpoint on failure
Step 7 of 10 fails. You restart from step 1. Every. Single. Time.
Duplicate side effects — emails, API writes, deploys — retried blindly.

3. Opaque debugging
You see the final error. Not which step poisoned the state.
No trace. No replay. Just vibes.

4. Mixed mutation semantics
Read-only and write steps treated identically.
A retry re-applies a deployment or a payment. You've now deployed twice.

The mental model shift

Stop thinking: "prompt chain"
Start thinking: "distributed system with state"

A state machine models your workflow as:

States — Idle, Planning, Executing, Validating, Recovering
Transitions — conditional, guarded, audited
Persisted state — survives crashes, enables checkpointing, replay

LangGraph made this practical. Every node writes to a shared state object. Every edge is conditional.

If a node fails → resume from the last checkpoint. Not from scratch.

What this actually looks like

Chain: A → B → C → D → Error (restart from A)

Graph: A → B → C → Error → Retry(C) → D
↓
HumanApproval → D

The graph knows where it failed. It knows what to do next.
The chain just panics.

This is Part 1 of a series on building deterministic, production-grade multi-agent systems.

Next up: Why I'm using Gemma 4 26B MoE as the reasoning engine — and how it compares to GPT-4o on real cost.

If you're building AI systems that need to work under an SLA — follow along.

— System Rationale