Forem: Tijo Gaucher

Running Gemma 4 next to your agent runtime: notes from a small shop

Tijo Gaucher — Mon, 06 Apr 2026 08:04:51 +0000

My brother Brandon and I run RapidClaw. Most days it's just the two of us, a handful of customers, and a few agents chugging along in production. A few months ago we started putting small open-weight models on the same box as the agent runtime — mostly Gemma 4, a bit of Phi-4 for comparison, some Qwen. This is a short write-up of what's actually worked and what hasn't.

Nothing revolutionary here. I'm writing it because I searched for "agent + local Gemma" a bunch of times last quarter and mostly found benchmark posts, not lived-experience notes.

The thing we noticed

The newest small models are small enough that they fit on the same machine as the agent loop. That's the whole observation. Gemma 4 4B runs fine on a 24 GB GPU next to a Node process running our agent code. Phi-4 14B is tight but works. A year ago you needed a separate inference box, which meant a network hop, which meant we just paid a hosted API and moved on.

Now the tradeoff is different. You can keep the hosted model for the hard stuff and quietly route the cheap, high-volume calls to the local model. Hybrid, not replacement.

What we actually do

We have four agents running in production right now. One of them — the one that classifies incoming support messages and decides which of the other agents to hand off to — used to make a hosted-model call per message. That single agent was roughly 80% of our inference spend because it ran on every message, even the obvious ones.

We moved that classifier to Gemma 4 4B on the same box. The agent framework is unchanged, it just points at a local OpenAI-compatible endpoint (we're using Ollama for now, llama.cpp's server also works). The other three agents still call the hosted models when they need to reason about something real.

That's it. One local model, four agents, one box. No Kubernetes, no model router, no fancy fallback chain.

Numbers from our box

Single machine, RTX 4090, one of our production workers. Measured over a week in March on real traffic, not a synthetic benchmark.

Path	Median latency	p95	Cost per 1k calls
Hosted Sonnet-class	1.8s	4.2s	~$4.50
Hosted mini/flash-class	0.9s	2.1s	~$0.60
Gemma 3 4B, local, same box	0.25s	0.6s	~$0.04*

*Local cost is amortized GPU + power on a box we were already paying for. If you had to rent a GPU just for this, the numbers flip hard — more on that below.

For the classifier workload specifically, Gemma 4 is good enough. It's not as sharp as the big hosted models, but "is this message a billing question or a bug report" doesn't need the big hosted models. We compared a week of its outputs against the hosted model's outputs on the same messages — they agreed on about 94% of them. The 6% where they disagreed were mostly ambiguous messages where the hosted model wasn't obviously right either.

Gotchas we hit

Cold starts are real. First request after the model unloads was 8–15 seconds. We pin the model in memory with a keepalive. Obvious in hindsight.

VRAM math is tighter than you think. Gemma 4 4B at Q4, plus an 8k context window, plus our Node process, plus the occasional burst of parallel requests: we hit OOM twice in the first week. We now cap concurrent local calls at 3 and queue the rest. Nothing fancy.

Prompt formats drift. A prompt that worked cleanly on the hosted model produced mush on Gemma. Small models are less forgiving of vague instructions. We ended up maintaining two prompt versions — one terse and explicit for Gemma, one more conversational for the hosted model. Not ideal but it's only two prompts.

Eval is annoying but necessary. You can't just swap models and hope. We built a small eval set (about 200 labeled messages) and run it whenever we change the local model or the prompt. Takes five minutes. Worth it.

When not to bother

Honestly, most people reading this probably shouldn't do this yet. A few cases where it doesn't make sense:

Low volume. If you're making under ~10k inference calls a day, the hosted APIs are cheaper than any GPU you'd rent. Local only wins at volume.
You don't already have a box. If you're renting a GPU purely to run Gemma 3, the math only works if you're saturating it. We could do this because we already had machines running the agent runtime with idle GPU capacity.
The task actually needs the big model. If you're doing code generation or multi-step planning, Gemma 4 4B will frustrate you. Use the hosted model and stop fighting it.
You're early. If you're pre-product-market-fit, every hour spent on inference optimization is an hour not spent on the thing users actually care about. We only did this after the classifier bill started showing up in the monthly.

What I'd try next

Phi-4 14B for one of the agents that does light reasoning over structured data. We haven't moved it yet because the quality bar is higher and I haven't built the eval set for it. Probably in April.

Also curious about Qwen 2.5 for a multilingual case we have, but that's further out.

That's the whole post. Nothing dramatic — a classifier moved, a bill went down, we learned some boring operational lessons. Small open-weight models finally being small enough to share a box with the agent runtime is, for us, the thing that made any of this viable.

Tijo Bear runs RapidClaw (rapidclaw.dev) with his brother Brandon — managed hosting for AI agents. If you're running agents and curious about hybrid local/hosted setups, the site has more.

Token Cost Optimization for AI Agents: 7 Patterns That Cut Our Bill by 73%

Tijo Gaucher — Mon, 06 Apr 2026 07:15:49 +0000

Token Cost Optimization for AI Agents: 7 Patterns That Cut Our Bill by 73%

Six months ago our monthly LLM bill at RapidClaw hit a number I'd rather not print. We were running production AI agents across customer workloads, and every "let's just add one more tool call" was quietly compounding into a four-figure surprise on the invoice.

I'm Tijo Bear, founder of RapidClaw. We build infrastructure for teams who want to ship AI agents without becoming full-time prompt engineers. After spending a quarter obsessing over our own token economics, we cut spend by 73% — without degrading agent quality. Here are the seven patterns that mattered most.

1. Prompt caching is the cheapest 90% win you'll ever ship

If you're sending the same system prompt, tool definitions, or RAG context on every turn, you're paying full freight for tokens the model has already seen. Anthropic, OpenAI, and most major providers now support prompt caching with cache hits priced at roughly 10% of normal input tokens.

# Before: 4,200 input tokens/turn at full price
messages = [
    {"role": "system", "content": LARGE_SYSTEM_PROMPT + TOOL_DEFINITIONS},
    {"role": "user", "content": user_input}
]

# After: same prompt, marked cacheable
messages = [
    {
        "role": "system",
        "content": [
            {
                "type": "text",
                "text": LARGE_SYSTEM_PROMPT + TOOL_DEFINITIONS,
                "cache_control": {"type": "ephemeral"}
            }
        ]
    },
    {"role": "user", "content": user_input}
]

One line of config. ~85% cost reduction on the cached portion. There's no excuse not to ship this today.

2. Route by complexity, not by habit

Not every task needs your most expensive model. We built a tiny router that classifies incoming agent requests into three buckets and dispatches them to the cheapest model that can plausibly handle the job:

def route_model(task_type: str, context_size: int) -> str:
    if task_type in ("classify", "extract", "format"):
        return "haiku"          # ~$0.25/M input
    if context_size > 50_000 or task_type == "reason":
        return "sonnet"         # ~$3/M input
    return "haiku"              # default to cheap

We escalate to the bigger model only when the cheap one returns low confidence or fails validation. Roughly 68% of our agent calls now resolve on the small model. That alone moved the needle more than any other optimization.

3. Trim your tool definitions ruthlessly

Tool/function schemas are tokens too. We audited ours and found 11 tools with descriptions averaging 180 tokens each, half of which were redundant explanation the model didn't actually need.

Cut every tool description down to its single most informative sentence. Move worked examples into a separate retrievable doc the agent can fetch only when it needs guidance. We saved ~1,400 tokens per turn just by editing JSON.

4. Stop re-feeding the entire conversation history

The naive agent loop ships the full message history on every turn. By turn 12 you're paying for turns 1–11 again. Three things help:

Sliding window — keep only the last N turns verbatim
Summary compaction — once history exceeds a threshold, ask a cheap model to summarize older turns into a 200-token recap
Memory extraction — pull stable facts (user prefs, project IDs, decisions) into a structured memory store, then inject only the relevant rows

def compact_history(messages, threshold=20):
    if len(messages) < threshold:
        return messages
    old, recent = messages[:-10], messages[-10:]
    summary = cheap_summarize(old)
    return [{"role": "system", "content": f"Earlier context: {summary}"}] + recent

5. Cap your tool-call loops

The single biggest money pit in agent systems isn't the model — it's the runaway loop. An agent that retries a flaky tool 14 times will quietly burn through more budget than 200 normal sessions.

Hard cap iterations. Add exponential backoff. Surface a clear error to the user instead of letting the model keep paying to re-try. Our default is 8 tool calls per turn with a budget guardrail that aborts the session if input tokens exceed a configured ceiling. You can read more about how we handle this in our agent runtime docs.

6. Stream and short-circuit

If your agent's output gets parsed and acted on, you don't need to wait for the full completion. Stream the response and short-circuit as soon as you've got the structured field you need. We saved roughly 22% of output tokens on long-form generations by stopping early when a <done> sentinel was emitted.

async for chunk in client.messages.stream(...):
    buffer += chunk.text
    if "<done>" in buffer:
        break  # stop paying for more tokens

7. Self-host the cheap stuff

Not every step in an agent pipeline needs a frontier model. Embeddings, classification, reranking, simple extraction — these run beautifully on small open models you can deploy on a single GPU box for a fixed monthly cost.

We moved embeddings and intent classification onto a self-hosted setup and the marginal cost dropped to effectively zero. The frontier model still handles the hard reasoning, but the surrounding plumbing now runs on infrastructure we control. If you're curious how we deploy and scale these, we wrote up the full architecture on the RapidClaw blog.

The numbers

Stacked together, here's what each pattern contributed to our 73% cut:

Pattern	Contribution
Prompt caching	31%
Model routing	19%
Self-hosting plumbing	11%
History compaction	6%
Tool definition trim	3%
Loop caps + budget guard	2%
Stream short-circuit	1%

The lesson isn't that any single trick is magical — it's that token economics is additive. Five mediocre optimizations beat one heroic one, and they're far easier to ship.

What I'd do first if I were starting over

If I had to rebuild this from scratch tomorrow with one week to optimize, I'd ship in this order: prompt caching → loop caps → model routing → history compaction. Those four alone get you to roughly 60% savings and require no infrastructure changes.

Everything else is polish.

If you're building production agents and want a runtime that bakes these patterns in by default, that's exactly what we're building at RapidClaw. I'd love to hear how you're handling token economics in your own stack — drop a comment or hit me up.

— Tijo

I replaced myself with AI agents and now my startup runs 60% faster

Tijo Gaucher — Mon, 30 Mar 2026 02:45:03 +0000

So about 6 months ago I was basically drowning. Solo founder, trying to build an AI platform, doing everything myself — investor outreach, pitch decks, dev work, customer support, content, SEO... you know the drill. I was working 14 hour days and still falling behind.

Then I started using AI agents for real. Not just ChatGPT for writing emails — I mean actual autonomous agents that handle entire workflows end to end. And honestly it kinda changed everything about how I run my startup.

what actually happened

I'm building Rapid Claw — its a platform for deploying and managing OpenClaw AI agents. OpenClaw is an open-source AI co-founder framework, and we make it stupid easy to spin up instances and manage them without dealing with all the infra headaches.

But before we had the platform ready, I was running these agents manually. Setting up servers, configuring environments, managing state, handling crashes at 2am... it was a mess. I was spending more time babysitting the agents than actually building my product.

The irony of building an agent hosting platform while struggling to host your own agents is not lost on me lol.

the numbers that made me rethink everything

Here's roughly what I was spending per month running agents the "hard way":

VPS instances: ~$400/mo (3 servers on Hetzner)
API costs (OpenAI + Anthropic): ~$800/mo
My time on devops/firefighting: ~25 hrs/mo (thats worth... a lot when you're a solo founder)
Random tools and services: ~$200/mo

Total: ~$1,400/mo + 25 hours of my life

After I dogfooded our own platform and moved everything to managed Rapid Claw instances, it dropped to about $600/mo total and I spend maybe 3-4 hours a month on agent ops. The rest of that time goes into actually building features and talking to users.

what the agents actually do for me

I'm not just using agents for one thing — I have multiple OpenClaw instances handling different parts of the business:

Research agent — scrapes competitor pricing, tracks Product Hunt launches in my space, monitors relevant subreddits. Used to spend 5+ hours a week on this manually.

Content agent — drafts blog posts, helps with SEO research, generates social media content. I still edit everything but starting from a solid draft vs a blank page saves me hours.

Dev assistant agent — reviews PRs, writes tests, handles repetitive code tasks. This one alone probably saves me 10 hours a week.

Outreach agent — personalizes cold emails for investor outreach, researches potential partners. Way better than the generic templates I was sending before.

the part nobody warns you about

The thing that surprised me most wasnt the cost savings or the time savings. It was how much mental energy it freed up.

When you're a solo founder, context switching is the real killer. Going from writing code to researching competitors to drafting emails to fixing a server — your brain never gets to go deep on anything.

Having agents handle the repetitive stuff means I can actually focus on the 2-3 things that matter most each day. Thats honestly been the biggest win.

why I built rapid claw

After going through all this pain myself, I realized other founders are dealing with the exact same thing. Everyone wants to use AI agents but nobody wants to deal with:

Setting up and maintaining servers
Managing multiple agent instances
Handling security and permissions (you do NOT want an agent with unrestricted access to your systems btw)
Monitoring and logging
Scaling up when things get busy

So thats basically why Rapid Claw exists. You pick your OpenClaw agent template, configure it, deploy it, and we handle all the infra. We've got this permission firewall thing that lets you control exactly what each agent can access, which honestly should be table stakes for anyone running agents in production but most people just... don't do it.

if you're thinking about trying agents

Few things I wish someone told me when I started:

Start with one workflow. Don't try to automate everything at once. Pick the most repetitive task you do and agent-ify that first. For me it was competitor research.
Expect to iterate. Your first agent config will suck. Thats fine. The second one will be way better. By the third you'll have a solid sense of what works.
Don't give agents more access than they need. Seriously. An agent with write access to your production database is a disaster waiting to happen. Principle of least privilege, always.
Track the time savings. Its easy to underestimate how much time agents save you. I started logging it and was genuinely surprised — went from ~60 hrs/week of work to about 35 hrs for the same output.

wrapping up

I went from being a burned out solo founder working insane hours to actually having time to think strategically about my business. The agents aren't perfect and they definitely need supervision, but they've basically become my team.

If you're a founder or indie hacker whos been curious about agents but hasn't taken the plunge — just start. Even a basic research agent will change how you work.

Happy to answer questions about my setup in the comments. Been running this way for a few months now and have learned a ton about what works and what definitely doesnt.

btw if you want to try running your own OpenClaw agents without the infra pain, check out rapidclaw.dev — we're still early but the free tier is enough to get started.