How multi-provider LLM routers silently fail

eleata team — Fri, 08 May 2026 14:58:04 +0000

How multi-provider LLM routers silently fail

A failure mode common to several Python LLM routers: a 429 caused by an
exhausted long-period quota is treated identically to a 429 caused by a
transient per-minute rate limit. The cooldown TTL ends up applied to
both, and one of the two cases is wrong by orders of magnitude.

This essay describes the failure mode in concrete terms and outlines the
small fix.

The shape of the failure

Most LLM routers track a single "this provider is unhealthy until X"
field per deployment. When a request fails, the router sets X = now + cooldown, and is_call_allowed() returns False until then.

That works perfectly for a transient per-minute rate limit, where
"cooldown" is correctly measured in seconds. It works very badly for a
monthly quota cap, where the provider only resumes serving requests
when its billing period rolls over — possibly weeks away.

A trace, with cooldown = 60s, against a provider that exhausted its
monthly token budget at t=0:

t=0       Provider returns 429 (body: "monthly quota exhausted")
          Router sets unhealthy_until = t+60s
t=60      Router's is_call_allowed() → True → call → 429 again
          Router sets unhealthy_until = t+120s
t=120     ... same cycle ...
... (1440 more retries in 24h, all fail) ...
t=86400   Period rolls over, provider works

That's roughly 1440 wasted round trips per day per provider, until the
billing period ends. No exception is raised — the router believes it
is working as designed.

How current routers handle this

I read the routing layers of three projects in the space at the time
of writing. Specific code references below; behavior may change.

LiteLLM (BerriAI/litellm). The Router class exposes
cooldown_time (single TTL) and allowed_fails (count). When a
deployment exceeds allowed_fails, it goes into the CooldownCache
for cooldown_time. Operators can configure both per deployment, but
the body of the failure isn't inspected. A 429 caused by a quota cap
gets the same TTL as a 429 caused by RPM throttling.

ClawRouter (BlockRunAI/ClawRouter). The categorizeError function
in proxy.ts classifies failures. For HTTP 403, the body is matched
against /plan.*limit|quota.*exceeded|subscription|allowance/i and, on
a hit, returns quota_exceeded. For HTTP 429, the function returns
rate_limited regardless of body. The cooldown for rate_limited is
RATE_LIMIT_COOLDOWN_MS = 60_000 — a hardcoded 60 seconds. Several
providers I tested while developing this library return 429 (not 403)
for some long-period quota failures, so the quota-exceeded branch
never fires for those.

OmniRoute (diegosouzapw/OmniRoute). A textbook circuit breaker
FSM in circuitBreaker.ts: failureThreshold (count) and
resetTimeout (single TTL). Custom isFailure(error) callback is
allowed, but the default treats any exception equally.

None of these is a defect of negligence. The failure mode requires
parsing the response body on a status code most projects map to a
single category. The fix is small but easy to miss.

Three states, not one

The fix is to track three orthogonal pieces of state per
(provider, model, credential_alias):

rate_limit_state — short-lived TTL from Retry-After headers, capped at 1 hour by default. Per-dimension (RPM/TPM/RPD).
quota_state — long-period (daily or monthly) consumption. exhausted=True is set only when the response body matches a known quota keyword. The TTL is the time until the period rolls over.
circuit_state — closed/open/half-open from generic 5xx and timeout streaks. Independent of the other two: a provider can be healthy on rate/quota but flaky on the network path, or vice versa.

Pre-call, the routing decision becomes:

if circuit_open(retry_at > now):    skip until retry_at
if circuit_open(retry_at is None):  skip indefinitely (auth failure)
if rate_limit_active:               skip 60s–1h
if quota_exhausted:                 skip until period_end
if quota_near_cap:                  skip until period_end
otherwise:                          allow

Post-call, the failure dispatch becomes:

status 429 + body matches quota keyword:    update quota_state
status 429 + Retry-After (no quota body):   update rate_limit_state
status 401 / 403 (auth):                     open circuit indefinitely
status 5xx / timeout (transient):            increment circuit error_streak
status 200:                                  bump quota counters; close circuit

The state contracts are simple enough that in-memory, SQLite, and
Postgres backends share the same Protocol. The implementation is small
relative to the failure mode it addresses; the difficulty is mostly in
the failure-classification regex and in keeping the increment + status
decision atomic under concurrent failures.

Two bugs the adversarial review caught in my own first draft

Before publishing the implementation I ran it through an adversarial
review. Two HIGH-severity bugs were caught that I would have shipped
otherwise.

Auth circuit not respected. A 401 / 403 should keep the circuit
open indefinitely — there's no point retrying a rejected API key. My
first draft set retry_at=None to mean "never retry", but the guard
function checked if retry_at and retry_at > now, which is falsy
when retry_at is None. Net effect: traffic kept flowing on a key the
provider had already rejected. Every call would re-fail with the same
401.

Race in circuit opening. The original code did read-decide-write
across two awaits:

streak = (await get_circuit()).error_streak
if streak + 1 >= threshold:
    await upsert(status='open', ...)
else:
    await upsert(status='closed', error_streak=streak+1)

Two concurrent failures could both read streak=2, both decide
will_open=False, and both write status='closed' with their
respective increments. The post-condition: streak=4 >= threshold=3
but status='closed'. The fix is to do increment + status decision in
one atomic SQL statement:

UPDATE circuits SET
  error_streak = error_streak + 1,
  status = CASE
      WHEN error_streak + 1 >= ? THEN 'open'
      ELSE status
  END
WHERE ...

Both bugs now have regression tests. Both are exactly the kind of bug
single-process tests on a quiet branch don't catch.

What this is and isn't

It's a small Python library, MIT-licensed, with three backends
(in-memory, SQLite, Postgres) and zero required runtime dependencies.
It does state — it doesn't make HTTP calls or wrap a client. You use
your existing OpenAI/Anthropic/etc. SDK and call guard() before each
request and record_outcome() after.

It is alpha. It does what it says, and the failure mode it addresses
is real, but I would not bet a production system on a one-week-old
library written by one person. The point of writing this essay
alongside the code is so that even if this particular implementation
goes nowhere, the pattern is documented and someone else's router can
adopt it.

Repo

Code: https://github.com/eleata/resilient-llm-router
Persistence demo (no deps, no API keys): python examples/persistence_demo.py
Caps catalog (5 providers seeded): src/resilient_llm_router/caps.toml

If you maintain or use a router that has the failure mode described
here, I'd be glad to compare notes.

How to validate Peppol BIS 3 invoices in 5 lines of Python (or Node, or Go)

eleata team — Tue, 05 May 2026 01:23:49 +0000

Starting September 2026 every B2B invoice in France must be e-invoiced (Peppol/Factur-X). Germany has mandated XRechnung for B2G since 2020. Italy has been on FatturaPA since 2019. Spain's Verifactu rolls out 2025-2026.

If you build accounting software, an ERP, or any e-commerce flow that touches EU customers, you'll likely need to validate these invoice formats at some point. The two main open-source Schematron engines are phive (by Philip Helger of the EN16931 working group) and Mustang (by Jochen Stärk and the FNFE-MPE working group). Both are great. Both ship as Java JARs or web forms — no API, no SDKs.

I built a thin REST API on top of Mustang 2.23.0 that gives you what's missing: SDKs, GitHub Action, and async batch.

The 5-line version

# pip install eleata-peppol
from eleata_peppol import Client

client = Client(api_key=os.environ["ELEATA_KEY"])
result = client.validate(format="peppol-bis-3", xml=open("invoice.xml", "rb").read())
print(result.valid, result.errors)

That's it. Same Schematron rules CEN/OpenPeppol publish — equivalent accuracy to the phive reference web form.

Node version

import { Eleata } from "@eleata/peppol";
import fs from "node:fs";

const client = new Eleata({ apiKey: process.env.ELEATA_KEY! });
const result = await client.validate({
  format: "xrechnung-2.x",
  xml: fs.readFileSync("./invoice.xml"),
});

if (!result.valid) console.error(result.errors);

In CI (GitHub Action)

- uses: eleata/validate-xrechnung-action@v1
  with:
    files: ./invoices/**/*.xml
    api-key: ${{ secrets.ELEATA_KEY }}

Add a validates-in-CI badge to your README. Break the build before you ship a broken invoice.

What's in the box

POST /v1/validate (sync) and /v1/validate/batch (async + webhook callback)
4 formats: Peppol BIS 3, XRechnung 2.x, Factur-X 1.07.2 (PDF/A-3 hybrid extracted natively), UBL 2.1
Public shareable validation reports at /r/{id}
50 anonymized real EU invoice fixtures (CC0)
Free 200 validations/month, no credit card

Honest about scope

This is not a full Peppol Access Point — Ecosio, B2BRouter, Pagero do that. We are only the validator-as-an-API layer. If you need to send invoices through the Peppol network, you still need an Access Point.

Repos public on github.com/eleata. Live demo at peppol.eleata.io.

Curious to hear from anyone using a different validator (xrechnung-tooling, validator-fhir, InvoiceNavigator). What pain points hit you first when integrating?

Forem: eleata team

How multi-provider LLM routers silently fail

How multi-provider LLM routers silently fail

The shape of the failure

How current routers handle this

Three states, not one

Two bugs the adversarial review caught in my own first draft

What this is and isn't

Repo

How to validate Peppol BIS 3 invoices in 5 lines of Python (or Node, or Go)

The 5-line version

Node version

In CI (GitHub Action)

What's in the box

Honest about scope