I built an AI Gateway with no technical background. Here's where I'm stuck.

Juan — Wed, 08 Apr 2026 11:51:19 +0000

I'm a solo founder from Argentina. I'm not an engineer — I built the backend of NeuralRouting.io almost entirely with Claude.

The problem

Most teams send every LLM request to GPT-4 even when a smaller model would return the same quality answer. The cost difference between a $30/M token model and a $0.50/M model is massive at scale.

What NeuralRouting does

It sits between your app and LLM providers. Each request gets a complexity score, and the router picks the cheapest model that can handle it.

It also has:

Dual-layer semantic cache — similar queries get served from cache instead of hitting the API again
Shadow Engine — runs cheaper models in parallel to benchmark quality over time
PII filtering and rate limiting
Agent loop detection

Where it's at

Early. I only support OpenAI and Groq right now. Zero users. I built too much before talking to anyone and I'm fixing that now.

If you work with LLMs and want to try it, I'm looking for honest feedback: neuralrouting.io

I was mass-sending everything to GPT-4. Here's what I changed.

Juan — Tue, 07 Apr 2026 01:19:39 +0000

I'm a solo dev from Argentina building AI tools. For months I was doing what most of us do — every API call went straight to GPT-4 (now GPT-4o). Summarization? GPT-4. Formatting a JSON? GPT-4. Answering "what's 2+2"? GPT-4.

Then I looked at my bill and did some math.

The numbers that made me stop

Here's what the main LLM providers charge per 1M tokens right now:

Model	Input	Output
GPT-4o	$2.50	$10.00
GPT-4o-mini	$0.15	$0.60
Llama 3.1 8B (via Groq)	$0.05	$0.05
Claude 3.5 Sonnet	$3.00	$15.00

Look at that gap between GPT-4o and Llama. That's a 50x price difference on input tokens.

And here's the thing — for probably 70% of what I was sending to GPT-4o, Llama would've given me the same answer.

What I tried first

The obvious solution: just add some if/else logic.

if is_simple(prompt):
    model = "llama-3.1-8b-instant"
else:
    model = "gpt-4o"

Sounds easy. It's not.

What's "simple"? How do you define that? Token count? Keywords? And then you need different API clients for OpenAI vs Groq. Different error handling. Fallback logic when one provider goes down. Rate limiting per provider.

It turned into spaghetti real fast.

What I actually ended up building

I spent a few months building a proxy that handles all of this automatically. You point your OpenAI SDK at it, and it figures out which model to use per request.

The routing logic is basically:

Classify the prompt — is it casual chat, coding, analysis, math, translation? Each has a different complexity baseline.
Check complexity — token count, multi-step signals, risk level.
Route — simple stuff goes to Llama (nearly free), complex stuff goes to GPT-4o.
Validate — a background process compares cheap vs premium responses to catch quality drops.

The integration looks like this:

from openai import OpenAI

# just change the base_url, that's it
client = OpenAI(
    base_url="https://your-proxy-url/v1",
    api_key="your-routing-key"
)

# use it exactly like before
response = client.chat.completions.create(
    model="gpt-4o",  # the router overrides this
    messages=[{"role": "user", "content": "hello"}]
)

One line change. Everything else stays the same.

What I learned

A few things that surprised me:

Most classification doesn't need AI. I started with GPT-4o-mini classifying each prompt (ironic, paying for AI to decide if I should pay for AI). Switched to regex + heuristics. Zero cost, runs in <1ms.
Fallbacks matter more than routing. If Groq goes down and you don't have a fallback, your users don't care about your 80% savings. They care that the app is broken.
Quality validation is the hard part. Routing is easy. Knowing if the cheap model gave a good enough answer — that's the real problem. I built a shadow engine that samples responses and compares them. Still not perfect.

Where I'm at now

$0 MRR. Zero paying customers. The product works — I use it myself every day. But I'm at the "talking to people" stage now, which is way harder than building.

If you're spending $100+/mo on LLM APIs and want to try it, the project is called NeuralRouting. Free tier available, no credit card.

But honestly I'm more interested in hearing: how are you handling multi-model routing? Roll your own? Using a gateway? Just sending everything to one model and accepting the cost?

First post here. Building in public from Corrientes, Argentina. 🇦🇷

Forem: Juan