Forem: Pranay Batta

Best AI Gateway to Route Codex CLI to Any Model

Pranay Batta — Thu, 16 Apr 2026 10:05:16 +0000

Codex CLI is OpenAI's terminal-based coding agent that runs entirely in your shell. It reads your codebase, proposes changes, runs commands, and writes code. Solid tool. One problem: it only talks to OpenAI by default.

I wanted to route Codex CLI through an AI gateway so I could use Claude Sonnet, Gemini 2.5 Pro, Mistral, and others without switching tools. I tested a few options. Bifrost worked best. Open-source, written in Go, 11 microsecond overhead. Here is exactly how I set it up and what I found.

Why Route Codex CLI Through an AI Gateway

Codex CLI sends requests to OpenAI's API. That is fine until you need something else. Maybe Claude Sonnet handles your refactoring tasks better. Maybe Gemini's context window fits your monorepo. Maybe you want automatic failover when OpenAI rate limits you mid-session.

An AI gateway sits between Codex CLI and your providers. It translates requests, routes traffic, and handles failures. You configure it once and Codex CLI does not know the difference.

Without a gateway, your options are:

Stick with OpenAI only (no routing, no failover, no cost tracking)
Manually swap API keys and base URLs every time you want a different model

Neither scales.

Setting Up Bifrost for Codex CLI

Bifrost exposes an OpenAI-compatible endpoint. Codex CLI connects to it like it would connect to OpenAI directly. Full Codex CLI integration docs here.

Install Bifrost

npx -y @maximhq/bifrost

That starts the gateway locally. The setup guide has the full walkthrough.

The OAuth Gotcha

This one tripped me up. Codex CLI always prefers OAuth authentication over custom API keys. If you have previously logged in with OpenAI, Codex CLI will ignore your custom OPENAI_API_KEY entirely.

Fix: Run /logout inside Codex CLI before configuring Bifrost. Without this step, your gateway config will be silently bypassed.

Configure Codex CLI to Use Bifrost

Set your environment variables:

export OPENAI_API_KEY=bifrost_virtual_key
export OPENAI_BASE_URL=http://localhost:8080/openai/v1

Or add it to your codex.toml:

[auth]
api_key = "bifrost_virtual_key"

[network]
openai_base_url = "http://localhost:8080/openai/v1"

The OPENAI_API_KEY here is a Bifrost virtual key. Your actual provider keys live in the Bifrost config.

Done. Every Codex CLI request now flows through Bifrost.

Routing Codex CLI to Any Model

This is the core use case. Configure multiple providers in Bifrost, and route Codex CLI traffic however you want. Bifrost uses the provider/model-name format for cross-provider routing:

accounts:
  - id: "codex-dev"
    providers:
      - id: "claude-primary"
        type: "anthropic"
        api_key: "${ANTHROPIC_API_KEY}"
        model: "anthropic/claude-sonnet-4-5-20250929"
        weight: 60
      - id: "gemini-secondary"
        type: "gemini"
        api_key: "${GEMINI_API_KEY}"
        model: "gemini/gemini-2-5-pro"
        weight: 25
      - id: "openai-fallback"
        type: "openai"
        api_key: "${OPENAI_API_KEY}"
        model: "gpt-4o"
        weight: 15

60% of requests go to Claude Sonnet. 25% to Gemini. 15% to GPT-4o. Weights auto-normalise, so use any numbers.

I ran this for a week. Claude Sonnet handled tool-heavy refactoring better. Gemini was faster on large context reads. GPT-4o was solid as a fallback. The routing docs cover all configuration options.

Other providers you can route to: Mistral, Groq, Cerebras, Cohere, Perplexity. All via the same provider/model-name format.

Can You Use Codex CLI with Non-OpenAI Models?

Yes. That is exactly what this setup does. Bifrost translates the OpenAI-format requests from Codex CLI into whatever format each provider expects. Codex CLI thinks it is talking to OpenAI. Bifrost handles the rest.

Critical requirement: non-OpenAI models must support tool use. Codex CLI relies on function calling for file operations, terminal commands, and code editing. If a model does not support tools, it will break on anything beyond simple chat.

Automatic Failover

Provider outages are inevitable. Bifrost sorts providers by weight and retries on failure. If Claude goes down, Gemini picks up. Gemini fails, falls back to OpenAI. Your Codex CLI session never interrupts.

The failover docs explain the retry logic in detail.

Comparison: AI Gateway Options for Codex CLI

Feature	Bifrost	LiteLLM	Direct API
Language	Go	Python	N/A
Routing overhead	11 microseconds	~8 milliseconds	0
Weighted routing	Yes	Yes	No
Automatic failover	Yes	Yes	No
Budget controls	4-tier hierarchy	Basic	No
Semantic caching	Yes	No	No
Self-hosted	Yes	Yes	N/A
Codex CLI compatible	Yes	Yes	Default

LiteLLM works as a proxy for Codex CLI, but the Python runtime adds measurable latency. When every Codex CLI request goes through the gateway, those milliseconds compound. For a tool sitting in the critical path of your coding workflow, overhead matters.

How to Route Codex CLI Through an AI Gateway?

Three steps:

Start Bifrost (npx -y @maximhq/bifrost)
Run /logout in Codex CLI to clear OAuth
Set OPENAI_API_KEY and OPENAI_BASE_URL to point at Bifrost

That is it. Configure your providers in the Bifrost config, and Codex CLI routes to any model you specify.

Budget and Observability

Once all Codex CLI traffic flows through Bifrost, you get cost controls and logging for free. The four-tier budget hierarchy lets you cap spend at the virtual key, team, or provider level.

budgets:
  - level: "virtual_key"
    id: "codex-cli-dev"
    limit: 150
    period: "monthly"

The observability layer logs every request: latency, tokens, cost, which provider handled it. When you are routing across three providers, this data tells you exactly where your money goes and which model performs best for your tasks.

Semantic caching also helps. Repeated or similar queries hit the cache instead of the provider. Cuts both cost and latency for common operations.

Honest Trade-offs

OAuth quirk is easy to miss. If you skip the /logout step, Codex CLI silently ignores your gateway config. There is no error. It just routes to OpenAI directly. I lost an hour to this before checking the docs.

Tool use is non-negotiable. Not every model supports function calling well enough for Codex CLI. Stick to models with solid tool use: Claude Sonnet, GPT-4o, Gemini 2.5 Pro. Smaller or older models may fail on file operations.

Self-hosted only. You run and maintain the gateway. No managed cloud version for the open-source release. The governance layer helps with access control, but ops is on you.

Extra hop. One more process in the chain. The 11 microsecond overhead is negligible, but it is still something to keep running.

Quick Start

# 1. Start Bifrost
npx -y @maximhq/bifrost

# 2. Logout from OpenAI OAuth in Codex CLI
# Inside Codex CLI, run: /logout

# 3. Point Codex CLI at Bifrost
export OPENAI_API_KEY=bifrost_virtual_key
export OPENAI_BASE_URL=http://localhost:8080/openai/v1

# 4. Use Codex CLI normally - it routes through Bifrost

GitHub | Docs | Website

If you are using Codex CLI for real work, routing through an AI gateway gives you model flexibility, failover, and cost visibility that you cannot get from a single provider. I benchmarked the performance and the overhead is genuinely negligible.

Open an issue on the repo if you run into anything.

What is LLM Orchestration and How AI Gateways Enable It

Pranay Batta — Wed, 15 Apr 2026 17:37:58 +0000

Most teams start with one LLM provider. Then they add a second for cost reasons. Then a third for latency. Six months in, they have a tangled mess of provider-specific SDKs, manual failover logic, and zero visibility into what anything costs. That mess is the problem LLM orchestration solves.

I evaluated how teams handle multi-model routing at scale. Custom code, orchestration frameworks, AI gateways. Here is what works and what just adds overhead.

What is LLM Orchestration?

LLM orchestration is the practice of managing multiple LLM providers, models, and configurations through a unified control layer. Instead of hard-coding provider logic into your application, you route, balance, cache, and monitor all LLM traffic from one place.

Think of it like a load balancer, but purpose-built for AI workloads. It handles which model gets which request, what happens when a provider goes down, how costs are tracked, and where the logs go.

The core components of LLM orchestration:

Routing - Deciding which model handles each request based on weight, cost, or capability
Failover - Automatically switching to a backup provider when one fails
Load balancing - Distributing requests across providers to avoid rate limits
Cost governance - Enforcing budgets per team, project, or API key
Caching - Avoiding duplicate calls for identical or semantically similar prompts
Observability - Tracking latency, tokens, costs, and errors across every request

Why Do You Need LLM Orchestration?

If you are calling one model from one provider, you do not. The moment any of these are true, you do:

Multiple models (GPT-4o, Claude, Gemini) serving different use cases
Multiple teams sharing the same providers
Budget limits per team or per project
Uptime requirements that demand automatic failover
Cost optimisation that requires routing cheaper queries to cheaper models

I measured what happens without orchestration. Teams I evaluated had 15-30% higher LLM costs from duplicate calls, no failover causing multi-minute outages during provider incidents, and zero per-team cost attribution.

How Does an AI Gateway Handle Orchestration?

An AI gateway is the infrastructure layer that makes LLM orchestration practical. Without a gateway, you are building every orchestration component yourself. With one, you configure it.

Here is the comparison:

Orchestration Feature	DIY (Custom Code)	AI Gateway
Multi-model routing	Custom SDK per provider, manual selection	Config-based weighted routing, auto-normalised
Failover	Try/catch with manual retry logic	Automatic, sorted by weight, instant retry
Load balancing	Custom queue + rate tracking	Built-in weighted distribution
Cost governance	Manual token counting + billing integration	Budget hierarchy with auto-enforcement
Caching	Redis/Memcached with custom key logic	Semantic + exact-match, built-in
Observability	Custom logging + dashboards	Real-time streaming, filters, sub-millisecond
Maintenance	Ongoing engineering effort	Configuration changes

The DIY approach works for prototypes. For production with multiple teams and providers, it becomes a full-time job.

Setting Up LLM Orchestration with Bifrost

I tested several gateways for model orchestration. Bifrost stood out on performance: 11 microsecond overhead, 5000 RPS throughput, written in Go. That matters because your orchestration layer should not become a bottleneck.

Start the gateway:

npx -y @maximhq/bifrost

Full setup guide here.

Weighted Routing Config

This is where LLM orchestration starts. You define providers with weights, and Bifrost routes traffic accordingly. Weights are auto-normalised to 1.0, and it follows a deny-by-default model.

accounts:
  - id: "production"
    providers:
      - id: "openai-primary"
        type: "openai"
        model: "gpt-4o"
        weight: 70
      - id: "anthropic-fallback"
        type: "anthropic"
        model: "claude-sonnet-4-20250514"
        weight: 30

70% of traffic goes to GPT-4o. 30% to Claude. If OpenAI goes down, Bifrost automatically fails over to the next provider sorted by weight. No code changes. No redeployment.

Budget Governance

Bifrost uses a four-tier budget hierarchy: Customer > Team > Virtual Key > Provider Config. Each tier can have independent spend limits and rate limits.

virtual_keys:
  - id: "ml-team-key"
    budget:
      max_spend: 500
      reset_duration: "1M"
    rate_limit:
      max_requests: 10000
      reset_duration: "1d"

When a team hits their budget, requests are denied. Not throttled. Denied. That is real cost governance, not just monitoring.

Semantic Caching

Bifrost runs dual-layer caching: exact hash matching for identical prompts, plus semantic similarity for prompts that mean the same thing but are worded differently. Both layers reduce redundant API calls without any application code changes.

Observability

Every request is logged with less than 0.1ms overhead. 14+ filters for slicing data. WebSocket-based live streaming so you can watch requests in real time. No separate logging pipeline needed.

What About MCP Workloads?

If you are running MCP servers with 500+ tools, orchestration gets expensive fast. Every tool definition eats tokens. Bifrost's MCP Code Mode achieves 92% token reduction by encoding tool schemas efficiently. That is a direct cost saving on top of the orchestration layer.

Honest Trade-offs

No tool is perfect. Here is where to be careful:

Bifrost is self-hosted only. You run it in your infrastructure. If you want a fully managed SaaS gateway, this is not it. For teams with compliance requirements, self-hosted is actually a benefit.
Go-based, not Python. If your team needs to extend gateway logic in Python, the codebase will be unfamiliar. The upside is the 11 microsecond latency that Python gateways cannot match.
Configuration over code. Bifrost favours YAML/UI config over programmatic SDKs. If you need deeply custom routing logic (like routing based on prompt content analysis), you will need to handle that at the application layer.
Alternatives exist. For simple single-provider setups, a gateway is overkill. If you are only using OpenAI and do not need failover or budgets, just call the API directly.

FAQ

What is the difference between LLM orchestration and LLM routing?

LLM routing is one component of orchestration. Routing decides which model handles a request. Orchestration includes routing plus failover, caching, budgets, load balancing, and observability. Multi-model routing is necessary but not sufficient for production AI workloads.

Can I do LLM orchestration without a gateway?

Technically, yes. You can build routing, failover, caching, and observability yourself. Practically, I have seen teams spend 2-3 engineering months building what a gateway provides out of the box. And then they still need to maintain it.

How does an AI gateway compare to LangChain for orchestration?

LangChain is a framework for building LLM applications. An AI gateway is infrastructure for managing LLM traffic. They solve different problems. You can use both: LangChain for application logic, and a gateway like Bifrost for orchestration underneath. Bifrost is a drop-in replacement for OpenAI's API format, so integration is straightforward.

Bottom Line

LLM orchestration is not optional once you are running multiple models in production. The question is whether you build it or use a gateway. I have tested both paths. The gateway approach - specifically Bifrost at 11 microsecond overhead - saves engineering time and gives you better observability from day one.

Star the repo: https://git.new/bifrost
Docs: https://getmax.im/bifrostdocs

MCP at Scale: Access Control, Cost Governance, and 92% Lower Token Costs

Pranay Batta — Tue, 14 Apr 2026 08:44:38 +0000

The Hidden Tax on Every MCP Request

Here is something nobody talks about when they demo MCP integrations: token costs at scale.

I have been running MCP setups with increasing numbers of connected servers. The pattern is always the same. You connect a few servers, everything works brilliantly. You connect a dozen, costs start climbing. You connect sixteen servers with 500+ tools, and suddenly your token budget is gone before the model even starts thinking about your actual query.

Why? Every tool definition from every connected server gets injected into the model's context on every single request. 150+ tool definitions can consume the majority of your token budget. And there is zero access control. Any consumer can call any tool. No cost tracking at tool level.

This is unsustainable for production deployments.

I Tested Bifrost's Code Mode Approach

Bifrost takes a fundamentally different approach to this problem. Instead of dumping all tool definitions into the context window, it exposes a virtual filesystem of Python stub files. The model discovers tools on-demand through four meta-tools:

listToolFiles - discover available servers and tools
readToolFile - load specific function signatures
getToolDocs - fetch detailed documentation only when needed
executeToolCode - run scripts in a sandboxed Starlark interpreter

The key insight: the model only loads what it actually needs for the current query. If you ask it to read a file, it does not need to know about your Slack, GitHub, Jira, and database tools all at once.

Here is what a typical tool discovery flow looks like:

# Model calls listToolFiles to see available servers
available = listToolFiles()
# Returns: ["filesystem/", "github/", "slack/", "jira/", ...]

# Model identifies it needs filesystem tools for this query
tools = readToolFile("filesystem/read.py")
# Returns only the function signature for filesystem_read

# Model fetches docs only if needed
docs = getToolDocs("filesystem", "read")

# Executes with full sandboxing
result = executeToolCode("filesystem/read.py", {"path": "/src/main.go"})

This is lazy loading for LLM tool contexts. Simple idea. Massive impact.

Benchmark Results: 3 Controlled Rounds

I ran three controlled rounds, scaling from 6 servers to 16 servers. Every round maintained a 100% task pass rate. The model completed every task correctly while using dramatically fewer tokens.

Round	Tools	Servers	Token Reduction	Cost Savings
1	96	6	58.2%	55.7%
2	251	11	84.5%	83.4%
3	508	16	92.8%	92.2%

At roughly 500 tools, Code Mode reduces per-query token usage by about 14x. From 1.15M tokens down to 83K. That is not an incremental improvement. That is a different cost structure entirely.

The savings compound non-linearly. As you add more tools, the percentage saved increases because Code Mode's overhead stays roughly constant while traditional mode scales linearly with tool count.

For full benchmark methodology, check the benchmarking docs.

Access Control That Actually Works

Token savings are great, but production MCP deployments need governance. Bifrost handles this through two mechanisms.

Virtual Keys let you create scoped credentials per user, team, or customer. You can scope at the tool level:

virtual_key:
  name: "data-team-key"
  allowed_tools:
    - database_read
    - database_query
  blocked_tools:
    - database_delete
    - filesystem_write

Allow filesystem_read, block filesystem_write. Allow database_query, block database_delete. Fine-grained, declarative, no code changes needed.

MCP Tool Groups are named collections of tools from multiple servers. You create a group, attach it to keys, teams, or users. No database queries at resolve time. This is important when you are running at 5000 RPS and cannot afford lookup latency.

Per-Tool Observability

Every tool execution gets logged with:

Tool name and server source
Arguments passed and results returned
Execution latency
Virtual key that initiated the call
Parent LLM request context

You can track cost at the tool level alongside LLM token costs. This matters when your finance team asks why the AI bill doubled last month. You can point to exactly which tools, which teams, and which queries drove the spend.

Budget and limits let you set spending caps per virtual key, so no single team can blow through the monthly allocation.

Connection Flexibility

Bifrost supports four MCP connection types: STDIO, HTTP, SSE, and in-process via the Go SDK. OAuth 2.0 with PKCE and automatic token refresh is built in. Health monitoring with automatic reconnects keeps things running without manual intervention.

You can run it in manual approval mode where a human reviews tool calls, or in autonomous agent loop mode where the model chains tool calls independently.

For Claude Code and Cursor users, the /mcp endpoint integrates directly. Setup takes minutes.

Honest Trade-offs

No tool is perfect. Here is what I noticed:

Learning curve for Code Mode. The virtual filesystem abstraction is elegant, but it is a new mental model. Teams used to traditional MCP tool injection will need to understand why their tools are now "files" the model reads on demand.

Meta-tool overhead on simple queries. If you only have 10-20 tools, the overhead of the four meta-tools (listToolFiles, readToolFile, etc.) might not save you much. The real wins kick in above 50-100 tools. Below that threshold, traditional mode works fine.

Starlark sandbox limitations. The sandboxed Starlark interpreter is secure by design, but it means tool code runs in a restricted environment. Complex tool implementations may need adjustments.

Dependency on gateway availability. Adding a gateway layer means one more component to monitor. Bifrost's 11 microsecond latency and Go-based architecture make this a non-issue in practice, but it is still an additional piece of infrastructure.

Who Should Care

If you are running fewer than 50 MCP tools, you probably do not need Code Mode yet. Traditional tool injection works fine at that scale.

If you are running 100+ tools across multiple servers, or if you need per-team access control, or if your CFO is asking questions about AI infrastructure costs, this is worth evaluating.

The 92% cost reduction at 500+ tools is the headline number, but the governance features (virtual keys, tool groups, audit logging) are what make it production-ready.

Try It

Bifrost is open-source and written in Go.

GitHub repo - star it if this is useful
MCP documentation - full setup guide
Governance docs - virtual keys, tool groups, budgets
Getting started - up and running in minutes

I have been testing a lot of MCP tooling lately. Bifrost's approach to the context window problem is the most practical solution I have seen. The lazy loading pattern for tool definitions should honestly be how all MCP gateways work.

Check the docs and give it a spin. Happy to discuss benchmarks or setup in the comments.

How to Track LLM Costs and Rate Limits on AWS Bedrock with an AI Gateway

Pranay Batta — Mon, 13 Apr 2026 10:21:38 +0000

Running LLM workloads on AWS is easy. Knowing what they cost is not. You spin up Bedrock, call Claude or Mistral a few thousand times, and the bill shows up three days later as a single line item. No breakdown by team. No per-model cost tracking. No rate limits unless you build them yourself.

I spent the last two weeks evaluating how teams can get proper cost governance over LLM usage on AWS. Native tools, third-party gateways, open-source options. Here is what I found.

The Problem with AWS Native Cost Tracking

AWS gives you CloudWatch and Cost Explorer. Both are built for general AWS resource monitoring. They work fine for EC2, Lambda, S3. For LLM workloads on Bedrock, they fall short.

What you get from CloudWatch + Cost Explorer:

Aggregate Bedrock spend per region
Invocation counts at the service level
Basic alarms on total spend thresholds

What you do not get:

Per-model token-level cost breakdowns
Team or project-level budget enforcement
Rate limiting by user, team, or API key
Real-time cost tracking per request
Automatic routing away from providers that exceed limits

If you are running one model for one team, native tools are fine. The moment you have multiple teams, multiple models, or need to enforce granular budgets, you are building custom infrastructure.

The Gateway Approach

An LLM gateway sits between your application and Bedrock. Every request passes through it. That gives you a single place to track costs, enforce rate limits, and control routing.

I tested three approaches:

Feature	AWS Native (CloudWatch + Cost Explorer)	LiteLLM	Bifrost
LLM-specific cost tracking	Aggregate only	Per-request, per-model	Per-request, per-model
Budget hierarchy	Account-level billing alerts	Basic budget controls	4-tier: Customer > Team > Virtual Key > Provider
Rate limiting	No native LLM rate limits	Basic rate limiting	VK + Provider Config level, token and request limits
Reset durations	N/A	Limited options	1m, 5m, 1h, 1d, 1w, 1M, 1Y (calendar-aligned UTC)
Bedrock support	Native	Yes	Yes (provider type "bedrock")
Overhead	None	~8ms (Python)	11 microseconds (Go)
Deployment	N/A	Self-hosted or cloud	Self-hosted (runs in your VPC)
Language	N/A	Python	Go

The numbers tell the story. For teams that need real LLM cost governance on AWS, a dedicated gateway is the right call.

Setting Up Bifrost with AWS Bedrock

Bifrost runs in your VPC alongside Bedrock. No data leaves your infrastructure. That matters for teams with compliance requirements.

Start the gateway:

npx -y @maximhq/bifrost

Full setup guide here.

Configure Bedrock as a provider:

accounts:
  - id: "ml-team"
    providers:
      - id: "bedrock-claude"
        type: "bedrock"
        region: "us-east-1"
        model: "anthropic.claude-sonnet-4-20250514-v1:0"
        weight: 80
      - id: "bedrock-mistral"
        type: "bedrock"
        region: "us-west-2"
        model: "mistral.mistral-large-2407-v1:0"
        weight: 20

Weighted routing across models. 80% of requests go to Claude Sonnet on Bedrock, 20% to Mistral. Both running through your AWS account. The provider configuration docs cover all Bedrock model formats and region options.

Four-Tier Budget Hierarchy

This is where Bifrost separates itself from everything else I tested. The budget system has four levels: Customer, Team, Virtual Key, and Provider Config. All four must pass for a request to go through.

budgets:
  customer:
    - id: "acme-corp"
      limit: 5000
      period: "1M"

  team:
    - id: "ml-engineering"
      customer_id: "acme-corp"
      limit: 2000
      period: "1M"

  virtual_key:
    - id: "staging-key"
      team_id: "ml-engineering"
      limit: 500
      period: "1w"

  provider_config:
    - id: "bedrock-claude"
      limit: 1000
      period: "1M"

Customer gets $5,000/month. ML Engineering team gets $2,000 of that. The staging key is capped at $500/week. And the Bedrock Claude provider itself is capped at $1,000/month. If any tier hits its limit, the request is blocked.

Cost is calculated from provider pricing, token usage, request type, cache status, and batch operations. Not estimated. Calculated from actual usage data.

The governance docs have the full breakdown.

Rate Limiting That Actually Works for LLMs

AWS does not give you LLM-specific rate limits. Bedrock has service quotas, but those are blunt instruments. You cannot limit a specific team to 100 requests per minute or cap token consumption per API key.

Bifrost handles rate limiting at two levels: Virtual Key and Provider Config. You can set both request limits (calls per duration) and token limits (tokens per duration).

rate_limits:
  virtual_key:
    - id: "staging-key"
      requests:
        limit: 100
        duration: "1h"
      tokens:
        limit: 50000
        duration: "1h"

  provider_config:
    - id: "bedrock-claude"
      requests:
        limit: 500
        duration: "1h"

Reset durations: 1m, 5m, 1h, 1d, 1w, 1M, 1Y. The daily, weekly, monthly, and yearly resets are calendar-aligned in UTC. So "1d" resets at midnight UTC, not 24 hours from first request.

Here is the clever part: if a provider config exceeds its rate limit, that provider gets excluded from routing. But other providers in the account remain available. Traffic shifts automatically. No downtime, no manual intervention.

Observability at Sub-Millisecond Overhead

Every request through Bifrost is captured: tokens used, latency, cost, response status. The observability layer adds less than 0.1ms of overhead. Storage backend is SQLite or PostgreSQL.

What makes this useful for AWS teams:

14+ API filter options for querying logs. Filter by model, provider, team, cost range, status code, time window.
WebSocket live updates. Watch requests flow through in real time. Useful during load testing or incident debugging.
Single pane across providers. If you are running Bedrock plus OpenAI or Gemini as failover, all logs are in one place.

Compare that to checking CloudWatch for Bedrock, then the OpenAI dashboard for your fallback, then manually correlating timestamps. The centralised view saves real time.

Honest Trade-offs

No tool solves everything. Here is what to know:

Bifrost is self-hosted only. You run it, you maintain it. For teams already on AWS with VPC infrastructure, this is straightforward. For smaller teams without DevOps, it is extra work.

LiteLLM has broader provider coverage. 100+ providers out of the box. If you need niche providers, LiteLLM may have them. Bifrost focuses on major providers but adds the Go performance advantage and deeper governance features.

AWS native tools have zero overhead. If all you need is aggregate cost visibility and basic billing alerts, CloudWatch is already there. No extra infrastructure.

Go vs Python matters at scale. Bifrost's 11 microsecond overhead versus LiteLLM's ~8ms becomes significant when you are processing thousands of requests per minute. At low volume, both are fine. At scale, the difference compounds. The benchmarks back this up: 5,000 RPS on a single instance.

Bifrost is a newer project. The community is growing but smaller than LiteLLM's. Documentation is solid. Edge cases may require checking GitHub issues.

When to Use What

Stick with AWS native tools if: You have one team, one model, and just need billing alerts.

Consider LiteLLM if: You need maximum provider coverage and are comfortable with Python-based overhead.

Use Bifrost if: You need granular cost governance, multi-tier budgets, LLM-specific rate limiting, and minimal latency on AWS. Especially if you are already running in a VPC and want semantic caching and automatic failover alongside cost controls.

Quick Start

# 1. Start Bifrost in your VPC
npx -y @maximhq/bifrost

# 2. Configure Bedrock providers in bifrost.yaml

# 3. Set budget and rate limit tiers

# 4. Point your application at the gateway
export ANTHROPIC_BASE_URL=http://localhost:8080/anthropic
export ANTHROPIC_API_KEY=your-bifrost-virtual-key

Every Bedrock request now has cost tracking, rate limiting, and observability built in.

GitHub | Docs | Website

AWS makes it easy to run LLM workloads. It does not make it easy to govern them. If your team is scaling Bedrock usage and needs real cost controls, a dedicated LLM gateway fills the gap that CloudWatch and Cost Explorer leave open.

Check the repo if you want to dig into the source.

Best Claude Code Gateway for Multi-Model Routing

Pranay Batta — Fri, 10 Apr 2026 22:29:45 +0000

Claude Code is great until you need more than one model. You hit a rate limit on Anthropic, want Gemini for long context, or need GPT-4o for a specific task. The default setup gives you no way to route across providers.

I spent a week testing gateways that sit between Claude Code and LLM providers. The goal was simple: configure multiple models, set routing weights, get automatic failover, and keep Claude Code working normally.

Bifrost was the clear winner. Open-source, written in Go, 11 microsecond overhead per request. Here is how I set up multi-model routing and what I learned.

Why Multi-Model Routing Matters

Different models are good at different things. Claude Sonnet handles tool use well. GPT-4o is strong at certain code generation tasks. Gemini 2.5 Pro handles massive context windows. Using one model for everything means you are leaving performance on the table.

Multi-model routing lets you:

Split traffic across providers by weight
Fail over automatically when a provider goes down
Pin specific models for specific tasks
Control costs by routing cheaper models for simpler operations

The problem: Claude Code talks to api.anthropic.com by default. No native multi-model support. You need a gateway.

The Setup: Bifrost as a Claude Code Gateway

Bifrost exposes an Anthropic-compatible endpoint. Claude Code does not know a gateway exists. It sends standard requests, and Bifrost translates and routes them to whatever provider you configure.

Full Claude Code integration docs here.

Install and Connect

npx -y @maximhq/bifrost

That starts the gateway locally. Setup guide has the details.

Point Claude Code at Bifrost:

export ANTHROPIC_BASE_URL=http://localhost:8080/anthropic
export ANTHROPIC_API_KEY=your-bifrost-virtual-key

The ANTHROPIC_API_KEY here is a Bifrost virtual key, not your actual Anthropic key. Provider keys live in the Bifrost config. This is a drop-in replacement for the Anthropic API.

Done. Every Claude Code request now flows through Bifrost.

Weighted Routing Configuration

This is the core of multi-model routing. You assign weights to providers, and Bifrost distributes traffic accordingly. Weights auto-normalize to sum 1.0, so you can use any numbers.

Here is a config that splits traffic between GPT-4o and Claude Sonnet:

accounts:
  - id: "dev-team"
    providers:
      - id: "openai-primary"
        type: "openai"
        api_key: "${OPENAI_API_KEY}"
        model: "gpt-4o"
        weight: 70
      - id: "anthropic-secondary"
        type: "anthropic"
        api_key: "${ANTHROPIC_API_KEY}"
        model: "claude-sonnet-4-20250514"
        weight: 30

70% of requests go to GPT-4o. 30% to Claude Sonnet. I used this to compare output quality across providers in real coding sessions without manually switching anything.

The routing docs cover all the configuration options.

Important detail: cross-provider routing does not happen automatically. You must explicitly configure each provider in your config. Bifrost does not guess or infer routing rules.

Automatic Failover

Weighted routing is useful. Automatic failover is essential. Providers go down. Rate limits hit. You do not want your Claude Code session to break mid-task.

Bifrost sorts providers by weight and retries on failure. If the primary provider fails, the next one picks up the request.

accounts:
  - id: "dev-team"
    providers:
      - id: "openai-primary"
        type: "openai"
        api_key: "${OPENAI_API_KEY}"
        model: "gpt-4o"
        weight: 80
      - id: "gemini-fallback"
        type: "gemini"
        api_key: "${GEMINI_API_KEY}"
        model: "gemini-2.5-pro"
        weight: 15
      - id: "anthropic-fallback"
        type: "anthropic"
        api_key: "${ANTHROPIC_API_KEY}"
        model: "claude-sonnet-4-20250514"
        weight: 5

OpenAI goes down, Bifrost retries with Gemini. Gemini fails, falls back to Anthropic. My coding session never interrupts.

Model Pinning for Bedrock and Vertex AI

If your team uses AWS Bedrock or Google Vertex AI, you can pin specific models directly:

# Bedrock
export ANTHROPIC_DEFAULT_SONNET_MODEL="bedrock/global.anthropic.claude-sonnet-4-6"

# Vertex AI
export ANTHROPIC_DEFAULT_SONNET_MODEL="vertex/claude-sonnet-4-6"

You can also override the model mid-session using the --model flag or the /model command inside Claude Code. Useful when you want to switch between models for different parts of a task. Start with Sonnet for scaffolding, switch to GPT-4o for a tricky implementation, then back again. The gateway handles the translation layer for each provider.

This is one area where the Anthropic SDK compatibility matters. Bifrost maintains full compatibility with the Anthropic message format, so model pinning and switching work without any client-side changes.

Provider configuration docs list all supported providers and model formats.

Budget Controls Across Providers

Once all traffic flows through one gateway, cost management becomes straightforward. Bifrost has a four-tier budget hierarchy: Customer, Team, Virtual Key, Provider Config.

budgets:
  - level: "virtual_key"
    id: "claude-code-dev"
    limit: 200
    period: "monthly"

Set a limit. When it is reached, requests get blocked. No surprise bills from a runaway Claude Code session.

The full governance layer handles rate limiting, access control, and spend management across all configured providers.

Observability Across All Providers

Every request through Bifrost gets logged: latency, token count, cost, provider used, response status. The observability layer gives you a single view across all providers.

This is particularly useful with multi-model routing. You can see exactly which provider handled each request, compare response times across models, and track per-provider costs. When I was running 70/30 weighted routing between GPT-4o and Claude Sonnet, the observability data showed me exactly how each model performed on real coding tasks. Response times, token consumption, and cost per request, all in one place.

Without centralized logging, you are checking multiple provider dashboards and guessing which model handled what. That is not sustainable when you are running multiple providers through Claude Code daily.

Honest Trade-offs

No tool is perfect. Here is what I found:

OpenRouter streaming limitation. OpenRouter does not stream function call arguments properly. This causes file operation failures in Claude Code. If you use OpenRouter as a provider, expect issues with tool use.

Non-Anthropic model requirements. Any non-Anthropic model you route through must support tool use. Claude Code relies heavily on function calling. Models without proper tool support will fail on file operations, search, and other agent tasks.

Self-hosted only. The open-source version requires you to run and maintain the gateway. There is no managed cloud offering. That means monitoring, updating, and debugging are on you.

Newer project. Bifrost's community is growing but still smaller than older alternatives. Documentation is solid, but edge cases may require digging through issues on GitHub.

Extra hop. You are adding a process between Claude Code and your provider. The 11 microsecond overhead is negligible, but it is one more thing in the chain to keep running.

Performance

I ran benchmarks matching the benchmarking guide. The numbers held up: 11 microseconds of routing overhead, 5,000 requests per second on a single instance. The Go implementation makes a real difference. Python-based gateways I tested added significantly more latency.

For a gateway that sits in the critical path of every LLM call, low overhead matters.

Quick Start Summary

# 1. Start Bifrost
npx -y @maximhq/bifrost

# 2. Configure providers in bifrost.yaml (weighted routing + failover)

# 3. Point Claude Code at Bifrost
export ANTHROPIC_BASE_URL=http://localhost:8080/anthropic
export ANTHROPIC_API_KEY=your-bifrost-virtual-key

# 4. Use Claude Code normally

That is it. Your Claude Code session now routes across multiple models with automatic failover and budget controls.

GitHub | Docs | Website

If you are running Claude Code for real work, multi-model routing is not optional. Single-provider setups break at the worst times. A gateway that handles routing, failover, and cost controls in one place saves hours of debugging and thousands in unexpected spend.

Open an issue on the repo if you run into anything.

Best MCP Gateway for 50% Token Cost Savings

Pranay Batta — Tue, 07 Apr 2026 08:32:48 +0000

TL;DR: Classic MCP dumps 100+ tool definitions into every LLM call. Bifrost's Code Mode generates TypeScript declarations instead, cutting token usage by 50%+ and latency by 40-50%. If you are running 3 or more MCP servers, this is the single biggest cost lever you have.

The Problem with Classic MCP

I have been testing MCP setups for a few months now. The standard approach is simple. You connect your MCP servers, and every tool definition gets sent to the LLM as part of the context window. Every single call.

With 3 MCP servers, you might have 30-40 tools. With 10 servers, easily 100+. Each tool definition includes the name, description, input schema, and parameter types. That is a lot of tokens. And you are paying for every single one of them on every request.

The math is straightforward. If your average tool definition is 200 tokens, and you have 50 tools, that is 10,000 tokens of overhead per call. At scale, this adds up fast.

How Bifrost Code Mode Changes This

Bifrost takes a different approach with its Code Mode. Instead of exposing raw tool definitions to the LLM, it generates TypeScript declaration files (.d.ts) for all connected MCP tools.

The LLM then writes TypeScript code to orchestrate multiple tools in a restricted sandbox environment. Instead of the model making 5 separate tool calls (each requiring a round trip), it writes one code block that handles all 5 operations.

Here is what this means in practice:

Token reduction: 50%+ compared to classic MCP. The TypeScript declarations are more compact than full JSON schemas, and the model makes fewer round trips.
Latency reduction: 40-50% compared to classic MCP. Fewer round trips means faster overall execution.
Recommended when: You are using 3 or more MCP servers.

What Code Mode Actually Does

The execution model is restricted by design. Here is what is available in the sandbox:

Available: ES5.1+ JavaScript, async/await, TypeScript, console.log/error/warn, JSON.parse/stringify, and all MCP tool bindings as globals.

Not available: ES Modules, Node.js APIs, browser APIs, DOM, timers (setTimeout/setInterval), network access.

This is not a general-purpose runtime. It is a controlled environment where the LLM can orchestrate tools safely. No arbitrary code execution, no network calls outside of the tool bindings.

You can configure tool bindings at the server level or tool level, depending on how granular you need the control to be. The docs cover the binding configuration in detail.

The Latency Numbers

Bifrost itself adds 11 microseconds of latency overhead per request. It is written in Go and handles 5,000 RPS sustained throughput. That is roughly 50x faster than Python-based alternatives.

For MCP-specific operations:

Sub-3ms MCP latency overall
InProcess connections: ~0.1ms
STDIO connections: ~1-10ms
HTTP connections: ~10-500ms (network dependent)

The MCP tool discovery is cached after the first request, so subsequent calls hit ~100-500 microseconds for discovery and ~50-200 nanoseconds for tool filtering.

Agent Mode: The Other Side

Bifrost also has an Agent Mode that turns the gateway into an autonomous agent runtime. You configure which tools are auto-approved via tools_to_auto_execute, set a max_depth to prevent infinite loops, and let the agent handle iterative execution.

This is a different use case from Code Mode. Agent Mode is for workflows where you want the LLM to act autonomously within boundaries. Code Mode is for when you want to reduce token costs on tool-heavy operations.

Deployment

Setup is zero-config. You can start with npx or Docker. The gateway supports 19+ providers out of the box (OpenAI, Anthropic, Azure, Bedrock, Gemini, Mistral, Cohere, Groq, and others), all through an OpenAI-compatible API format.

# npx
npx -y @maximhq/bifrost

# Docker
docker run -p 8080:8080 maximhq/bifrost

Who Should Use Code Mode

If you are running fewer than 3 MCP servers, classic mode is probably fine. The overhead is manageable.

If you are running 3+, especially with 50+ tools across those servers, Code Mode is worth testing. The 50%+ token savings are significant at scale, and the 40-50% latency improvement compounds across multi-step agent workflows.

I tested this on a setup with 5 MCP servers and 80+ tools. The token savings were immediately visible in the cost dashboard. The reduced round trips also made the overall agent response noticeably faster.

GitHub: git.new/bifrost
Docs: getmax.im/bifrostdocs
Website: getmax.im/bifrost-home

How to Connect Any Model with Gemini CLI Using Bifrost AI Gateway

Pranay Batta — Mon, 06 Apr 2026 10:12:17 +0000

TL;DR: Gemini CLI works with Google's models out of the box. But if you want to route requests through multiple providers, add failover, or track costs, you can point Gemini CLI at Bifrost. One config change. Every model available through a single endpoint.

The Problem with Single-Provider CLI Tools

Gemini CLI connects to Google's Generative AI API. That is fine if you only use Gemini models. But most production setups involve multiple providers. OpenAI for some tasks. Anthropic for others. Maybe a local Ollama instance for development.

Switching between CLIs and API keys for each provider gets old fast.

I tested Bifrost, an open-source LLM gateway written in Go, as a unified routing layer for Gemini CLI. The setup took about 5 minutes.

How Bifrost Works with Gemini CLI

Bifrost exposes a fully Google GenAI compatible endpoint:

http://localhost:8080/genai/v1beta/models/{model}/generateContent

This means Gemini CLI can talk to Bifrost without any code changes. Just point the base URL to your Bifrost instance.

Bifrost then routes the request to whatever provider and model you specify. OpenAI, Anthropic, Vertex AI, Bedrock, Groq, Ollama. All through the same endpoint.

Setup

Step 1: Install Bifrost

npx @anthropic-ai/bifrost@latest

Zero config. Starts on port 8080 by default.

Step 2: Configure Providers

Add your provider keys to the config:

{
  "providers": {
    "openai": {
      "keys": [{"name": "openai-1", "value": "env.OPENAI_API_KEY", "models": ["gpt-4o", "gpt-4o-mini"], "weight": 1.0}]
    },
    "anthropic": {
      "keys": [{"name": "anthropic-1", "value": "env.ANTHROPIC_API_KEY", "models": ["claude-sonnet-4-20250514"], "weight": 1.0}]
    },
    "gemini": {
      "keys": [{"name": "gemini-1", "value": "env.GEMINI_API_KEY", "models": ["gemini-2.5-flash"], "weight": 1.0}]
    }
  }
}

Step 3: Point Gemini CLI to Bifrost

Set the base URL to your Bifrost instance:

export GEMINI_API_BASE_URL=http://localhost:8080

Now every request from Gemini CLI goes through Bifrost. You can target any provider using the provider-prefixed model format:

gemini/gemini-2.5-flash      → Google Gemini
openai/gpt-4o                → OpenAI
anthropic/claude-sonnet-4-20250514  → Anthropic
vertex/gemini-pro             → Vertex AI

What You Get

Multi-Provider Routing

One CLI, every model. No more switching between tools or managing separate API keys per provider.

Automatic Failover

Set up fallback chains in your requests. If Gemini is rate-limited, the request goes to OpenAI. If OpenAI is down, it goes to Anthropic. Each fallback is a fresh request. All plugins still run.

Budget Controls

Bifrost has a four-tier budget hierarchy: Customer, Team, Virtual Key, and Provider Config. Set a monthly spending cap on your Virtual Key. When it is hit, the gateway stops routing to paid providers. Your local Ollama instance can serve as the fallback.

Cost Tracking

Every request is logged with token counts and cost calculations. The Model Catalog tracks pricing across all providers automatically.

Performance

I ran 1,000 requests through Bifrost targeting Gemini models. The gateway adds 11µs of overhead per request. At 5,000 RPS sustained throughput, the bottleneck is always the provider, never the gateway. That is 50x faster than Python-based alternatives like LiteLLM.

The Overhead Question

The concern with adding a proxy layer is always latency. In practice, LLM API calls take 500ms to 5 seconds depending on the model and prompt. An 11µs gateway overhead is invisible.

The semantic caching layer (currently Weaviate-backed) can actually reduce latency for repeated queries by serving cached responses instead of hitting the provider again.

When This Makes Sense

This setup is useful if you:

Use Gemini CLI but also need OpenAI or Anthropic models
Want failover so your workflow does not break during provider outages
Need to track costs across providers in one place
Want to set budget limits so you do not get surprise bills

If you only use Gemini models and do not care about failover or cost tracking, direct connection is fine. The gateway adds value when you are working across providers.

Top 5 Enterprise AI Gateways for Dynamic Routing in 2026

Pranay Batta — Fri, 03 Apr 2026 05:42:24 +0000

If you are running multiple LLM providers in production, routing logic becomes a critical infrastructure decision. Send everything to one provider and you get single points of failure. Hardcode routing rules and you lose flexibility when latency spikes or rate limits hit.

I spent the last few weeks evaluating five AI gateways specifically for their dynamic routing capabilities. The criteria: latency overhead, failover behaviour, weighted distribution, and how much config it takes to get routing working in production.

The short version: Bifrost came out on top for raw performance and routing flexibility. 11 microsecond latency overhead, written in Go, with weighted routing and automatic failover built in. You can run it right now with npx -y @maximhq/bifrost. Full docs here.

Why dynamic routing matters

Static routing is fine for prototypes. Pick a model, call the API, ship it.

Production is different. You need:

Failover: When OpenAI returns 429s or 500s, traffic should automatically shift to Anthropic or another provider. No manual intervention.
Weighted distribution: Split traffic 70/30 across providers for cost optimization or A/B testing model quality.
Latency-based routing: Send requests to whichever provider responds fastest at that moment.
Budget-aware routing: Stop sending traffic to a provider when your spend cap is hit.

The gateway layer is the right place to handle this. Application code should not care which provider serves a request.

The five gateways I tested

1. Bifrost

Language: Go | Overhead: 11 microseconds | Throughput: 5,000 RPS sustained

Bifrost is the fastest gateway I have tested. The 11 microsecond overhead is not a typo. That is roughly 50x faster than Python-based alternatives like LiteLLM, which adds around 8ms per request.

Routing configuration is declarative and clean. Here is what weighted routing across two providers looks like:

# bifrost-config.yaml
providers:
  - name: openai-primary
    provider: openai
    model: gpt-4o
    weight: 70
    api_key: ${OPENAI_API_KEY}
  - name: anthropic-fallback
    provider: anthropic
    model: claude-sonnet-4-20250514
    weight: 30
    api_key: ${ANTHROPIC_API_KEY}

routing:
  strategy: weighted
  fallback:
    enabled: true
    max_retries: 2

That splits 70% of traffic to OpenAI and 30% to Anthropic. If OpenAI fails, requests automatically fall back to Anthropic.

What I like: the governance layer ties routing to budgets. You can set a four-tier budget hierarchy (Customer, Team, Virtual Key, Provider Config) and routing decisions respect those limits. When a provider budget is exhausted, traffic shifts automatically.

Setup is genuinely fast. One command to start:

npx -y @maximhq/bifrost

Or Docker:

docker run -p 8080:8080 maximhq/bifrost

The setup guide covers both approaches. Provider configuration takes a few minutes.

Other features worth noting: semantic caching with dual-layer support (exact hash + semantic similarity), observability built in, MCP support with sub-3ms latency and 50%+ token reduction in Code Mode, and a drop-in replacement endpoint for the Anthropic SDK so you can migrate without changing application code. Anthropic SDK integration docs here.

Check the benchmarks if you want to verify the numbers yourself.

2. LiteLLM

Language: Python | Overhead: ~8ms | Providers: 100+

LiteLLM has the widest provider coverage I have seen. Over 100 providers through a unified interface. If you need to call a niche model API, LiteLLM probably supports it.

Routing is available through the proxy server. You can configure fallbacks and load balancing across models. The configuration is YAML-based and straightforward.

model_list:
  - model_name: gpt-4
    litellm_params:
      model: openai/gpt-4
      api_key: sk-xxx
  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4
      api_key: sk-yyy

router_settings:
  routing_strategy: least-busy
  num_retries: 3

The trade-off is performance. At ~8ms overhead per request, you are adding meaningful latency at high throughput. For applications doing thousands of requests per second, that adds up. The Python runtime is the bottleneck.

Credit where it is due: LiteLLM's provider coverage is unmatched and the community is active. For teams that prioritize breadth over speed, it is a solid choice.

3. Kong AI Gateway

Language: Lua/C (OpenResty) | Type: Enterprise, plugin-based

Kong is a well-established API gateway that added AI capabilities through plugins. If your organization already runs Kong for general API management, adding AI routing is incremental.

The AI plugin supports multiple providers and basic routing. Rate limiting, authentication, and logging come from Kong's mature plugin ecosystem.

The limitation: AI-specific routing features require the enterprise tier. The open-source version gives you basic proxying, but weighted routing, advanced failover, and AI-specific analytics are paid features. Configuration is also more complex because you are working within Kong's plugin architecture rather than a purpose-built AI gateway.

Credit: Kong's plugin ecosystem is mature and battle-tested for general API management.

4. Cloudflare AI Gateway

Type: Managed service | Setup: Minutes

Cloudflare AI Gateway is the easiest to set up on this list. If you are already on Cloudflare, you can enable it from the dashboard and start routing requests through their edge network.

It provides caching, rate limiting, and basic analytics out of the box. The managed nature means zero infrastructure to maintain.

The limitation: routing flexibility is constrained compared to self-hosted options. Custom routing strategies, weighted distribution, and provider-level budget controls are limited. You also depend on Cloudflare's edge network for all LLM traffic, which may not work for teams with data residency requirements.

Credit: For teams that want AI gateway functionality without managing infrastructure, Cloudflare delivers the simplest path to production.

5. Azure API Management

Type: Enterprise, Azure-native | Setup: Hours to days

Azure APIM is the default choice for organizations already invested in Azure. It supports routing to Azure OpenAI endpoints with built-in integration, and you can configure policies for retry, circuit breaking, and load balancing.

The routing configuration uses Azure's policy XML, which is verbose but powerful. You get deep integration with Azure Monitor, Key Vault, and other Azure services.

The limitation: it is Azure-native. If you are multi-cloud or use non-Azure LLM providers, the integration story gets complicated. Routing to Anthropic or other providers requires custom policy work. Setup is also significantly more complex than purpose-built AI gateways.

Credit: For Azure-first organizations, the deep integration with the Azure ecosystem and enterprise compliance features are genuinely valuable.

Comparison table

Feature	Bifrost	LiteLLM	Kong AI	Cloudflare AI	Azure APIM
Latency overhead	11 microseconds	~8ms	Low (Lua/C)	Varies (edge)	Varies
Language	Go	Python	Lua/C	Managed	Managed
Weighted routing	Yes	Yes	Enterprise only	Limited	Via policy
Automatic failover	Yes	Yes	Yes	Basic	Via policy
Budget-aware routing	Yes (4-tier)	Basic	No	No	No
Semantic caching	Yes (dual-layer)	Basic	No	Yes	No
Provider count	Growing	100+	Major providers	Major providers	Azure-focused
Open source	Yes	Yes	Partial	No	No
Self-hosted	Yes	Yes	Yes	No	No
Setup time	Minutes	Minutes	Hours	Minutes	Hours to days

Honest trade-offs

No tool is perfect. Here is what I found lacking in each.

Bifrost: Provider count is still growing. If you need a niche provider that is not yet supported, you will need to check the docs or request it. The project is newer than LiteLLM, so community resources are still building up.

LiteLLM: Performance at scale is the main concern. The ~8ms overhead is fine for low-throughput applications, but at 5,000+ RPS, you are looking at significant cumulative latency. Memory usage also climbs with the Python runtime under load.

Kong AI Gateway: The AI features feel bolted on rather than native. If you are not already a Kong customer, adopting the full Kong stack just for AI routing is overkill. Enterprise pricing for AI-specific features is a barrier.

Cloudflare AI Gateway: Limited control. You cannot implement custom routing strategies or complex failover logic. Data flows through Cloudflare's network, which is a non-starter for some compliance requirements.

Azure APIM: Vendor lock-in is real. Multi-provider routing outside Azure requires significant custom work. Configuration through XML policies is tedious compared to YAML-based alternatives.

Which one should you pick

Pick Bifrost if performance and routing flexibility are your top priorities. The 11 microsecond overhead and built-in governance features (budget-aware routing, weighted distribution, automatic failover) make it the strongest option for high-throughput production workloads. Star it on GitHub or check the docs to get started.

Pick LiteLLM if you need the widest provider coverage and performance is not your bottleneck.

Pick Kong if your organization already runs Kong and wants to add AI routing incrementally.

Pick Cloudflare if you want zero infrastructure overhead and can live with limited routing customization.

Pick Azure APIM if you are fully committed to the Azure ecosystem.

For most teams building production AI infrastructure, routing is a gateway-level concern that should not leak into application code. The right gateway depends on your throughput requirements, provider mix, and how much control you need over routing logic.

I would start with Bifrost. One command to run, sub-microsecond overhead, and routing that actually works at scale. Docs are here.

How to Connect Non-Anthropic Models to Claude Code with Bifrost AI Gateway

Pranay Batta — Wed, 01 Apr 2026 13:41:26 +0000

I tested five different LLM gateways to route non-Anthropic models through Claude Code. Bifrost was the fastest by a wide margin. 11 microseconds of overhead per request. 50x faster than the Python-based alternatives I benchmarked.

Here is exactly how I set it up, what worked, and where each feature matters.

TL;DR: Bifrost is an open-source Go gateway that exposes an Anthropic-compatible endpoint, letting you route Claude Code requests to GPT-4o, Gemini, Bedrock, or any supported provider by changing one environment variable. You get multi-provider failover, budget controls, and semantic caching at 11 microseconds of overhead per request.

This post assumes you are familiar with Claude Code and have used at least one LLM API.

Bifrost on GitHub -- open-source, written in Go, handles 5,000 RPS on a single instance.

The Problem

Claude Code locks you into api.anthropic.com. No native way to swap providers. You cannot route to GPT-4o, Gemini, or Bedrock models without building your own proxy or switching tools entirely.

I needed:

GPT-4o for certain coding tasks
Gemini 2.5 Pro for long context
Automatic failover when a provider goes down
One place to track costs across all models

Building a custom proxy was not worth the maintenance burden. So I went looking for something production-ready.

What Bifrost Does

Bifrost exposes an Anthropic-compatible endpoint at /anthropic. Claude Code sends standard Anthropic-format requests. Bifrost translates and routes them to whatever provider you configure -- OpenAI, Bedrock, Vertex AI, Gemini, others.

It is a drop-in replacement. Change one URL. No SDK modifications. No wrapper code.

Claude Code -> Bifrost (/anthropic) -> Any LLM Provider

The Anthropic SDK integration page has the full compatibility details.

Setup: 3 Minutes

Step 1: Install

npx -y @maximhq/bifrost

That starts the gateway locally. Full setup instructions here.

Step 2: Configure a Provider

Create bifrost.yaml. This routes everything to GPT-4o:

accounts:
  - id: "my-account"
    providers:
      - id: "openai-primary"
        type: "openai"
        api_key: "${OPENAI_API_KEY}"
        model: "gpt-4o"
        weight: 100

See the provider configuration docs for all supported providers and options.

Step 3: Point Claude Code at Bifrost

export ANTHROPIC_BASE_URL=http://localhost:8080/anthropic

Done. Claude Code now sends requests through Bifrost, which translates them to OpenAI format and forwards to GPT-4o. Zero code changes.

Multi-Provider Routing

This is where it gets interesting. I configured weighted routing across two providers:

accounts:
  - id: "my-account"
    providers:
      - id: "openai-primary"
        type: "openai"
        api_key: "${OPENAI_API_KEY}"
        model: "gpt-4o"
        weight: 80
      - id: "anthropic-fallback"
        type: "anthropic"
        api_key: "${ANTHROPIC_API_KEY}"
        model: "claude-sonnet-4-20250514"
        weight: 20

80% of traffic goes to GPT-4o. 20% to Claude. Useful when you want to compare output quality across models in real usage.

Automatic Failover

This was the feature that sold me. Failover configuration took five minutes. If GPT-4o goes down, Bifrost tries the next provider automatically:

accounts:
  - id: "my-account"
    failover:
      enabled: true
    providers:
      - id: "openai-primary"
        type: "openai"
        api_key: "${OPENAI_API_KEY}"
        model: "gpt-4o"
        priority: 1
      - id: "gemini-secondary"
        type: "gemini"
        api_key: "${GEMINI_API_KEY}"
        model: "gemini-2.5-pro"
        priority: 2
      - id: "anthropic-tertiary"
        type: "anthropic"
        api_key: "${ANTHROPIC_API_KEY}"
        model: "claude-sonnet-4-20250514"
        priority: 3

OpenAI fails, Bifrost tries Gemini. Gemini fails, falls back to Anthropic. My Claude Code session never breaks. No retry logic on my side.

Bedrock and Vertex AI

I also tested with AWS Bedrock and Vertex AI. Same pattern:

providers:
  - id: "bedrock-claude"
    type: "bedrock"
    region: "us-east-1"
    model: "anthropic.claude-sonnet-4-20250514-v2:0"
    priority: 1
  - id: "vertex-gemini"
    type: "vertex"
    project_id: "my-gcp-project"
    region: "us-central1"
    model: "gemini-2.5-pro"
    priority: 2

Same Anthropic-compatible endpoint. Claude Code does not know which provider is behind Bifrost. That is the point.

Features Worth Mentioning

Routing alone is useful. But once all requests flow through one gateway, you get access to several other capabilities I found genuinely practical.

Budget Enforcement

Bifrost has a four-tier budget hierarchy: Customer, Team, Virtual Key, Provider Config. I set team-level limits:

budgets:
  - level: "team"
    id: "engineering"
    limit: 500
    period: "monthly"

Budget runs out, requests get blocked. No surprise bills. The governance docs cover the full hierarchy.

Semantic Caching

This cut my costs noticeably. Bifrost supports dual-layer caching: exact hash matching plus semantic similarity. If I have already asked a similar question, it returns the cached response instead of hitting the provider.

Supported vector stores: Weaviate, Redis, Qdrant.

Observability

Every request gets logged with latency, tokens, cost, and provider information. The observability layer gives you full visibility into what is happening across all your providers.

MCP Support

Bifrost also works as an MCP server. I tested Code Mode -- it reduced tokens by over 50% and latency by 40-50%. Agent Mode is available for more complex workflows. Useful if you are connecting to Claude Desktop or other MCP-compatible clients.

Benchmarks

I ran my own tests and the numbers matched what is documented. 11 microseconds overhead. 5,000 RPS on a single instance. The Go implementation makes a real difference compared to Python gateways I tested.

The benchmarking guide explains how to reproduce these numbers yourself.

Trade-offs and Limitations

Worth being upfront about the downsides:

Relatively new project. Bifrost does not have the years of battle-testing that older proxies have. The community is growing but smaller than established alternatives.
Self-hosted only. The open-source version has no managed cloud offering. You run and maintain the infrastructure yourself.
Extra operational overhead. You are running a separate process between Claude Code and your LLM provider. That is one more thing to monitor, update, and debug compared to direct API calls.
Provider coverage is expanding but not exhaustive. Some niche providers or model variants may not be supported yet. Check the docs before committing.

Quick Recap

Install: npx -y @maximhq/bifrost
Configure providers in bifrost.yaml
Set ANTHROPIC_BASE_URL=http://localhost:8080/anthropic
Use Claude Code normally. Bifrost routes to whatever model you configured.

I tried building custom proxies before. I tried other gateways. This is the fastest option I found, and the setup takes minutes not hours.

GitHub repo | Docs | Website

If you run into issues or want a specific provider supported, open an issue on the repo.

How Bifrost Reduces GPT Costs and Response Times with Semantic Caching

Pranay Batta — Wed, 01 Apr 2026 05:51:15 +0000

TL;DR

Every GPT API call costs money and takes time. If your app sends the same (or very similar) prompts repeatedly, you are paying full price each time for answers you already have. Bifrost, an open-source LLM gateway, ships with a semantic caching plugin that uses dual-layer caching: exact hash matching plus vector similarity search. Cache hits cost zero. Semantic matches cost only the embedding lookup. This post walks you through how it works and how to set it up.

The cost problem with GPT API calls

If you are building anything production-grade with GPT-4, GPT-4o, or any OpenAI model, you already know that API costs add up fast. Token-based pricing means every request burns through your budget, whether it is a fresh question or something your system answered three minutes ago.

Here is the thing: in most real applications, a significant portion of requests are either identical or semantically similar to previous ones. Think about it. Customer support bots get asked the same questions in slightly different words. Code assistants receive near-identical prompts from different users. RAG pipelines retrieve similar context and ask similar follow-ups.

Without caching, you pay full model cost for every single one of those requests. You also wait for the full round-trip to the provider each time, adding latency that your users notice.

The obvious fix is caching. But traditional exact-match caching has a big limitation: it only works when the prompt is character-for-character identical. Change one word, add a comma, rephrase slightly, and you get a cache miss. That is where semantic caching changes the game.

What semantic caching is and how it differs from exact-match caching

Exact-match caching hashes the entire request and looks up that hash. If the hash matches a stored response, you get a cache hit. If even one character is different, it is a miss. This works well for automated pipelines where prompts are templated and predictable. It falls apart for user-facing applications where people phrase things differently.

Semantic caching converts the request into a vector embedding and searches for similar embeddings in a vector store. If a stored request is semantically similar enough (above a configurable threshold), the cached response is returned. This means "How do I reset my password?" and "What are the steps to change my password?" can both hit the same cache entry.

Bifrost combines both approaches in a dual-layer architecture, giving you the speed of exact matching with the intelligence of semantic similarity as a fallback.

How Bifrost implements dual-layer caching

Bifrost's semantic cache plugin uses a two-step lookup process for every request that has a cache key:

Layer 1: Exact hash match. The plugin hashes the request and checks for a direct match. This is the fastest path. If it hits, you get the cached response with zero additional cost. No embedding generation, no vector search, no provider call.

Layer 2: Semantic similarity search. If the exact match misses, Bifrost generates an embedding for the request and searches the vector store for semantically similar entries. If a match is found above the similarity threshold (default 0.8), the cached response is returned. The only cost here is the embedding generation.

If both layers miss, the request goes to the LLM provider as normal. The response is then stored in the vector store with its embedding for future lookups.

You can also control which layer to use per request. If you know your use case only needs exact matching (templated prompts), you can skip the semantic layer entirely. If you want semantic-only, that is an option too. The default is both, with direct matching first and semantic as fallback.

Here is how the cost breaks down:

Scenario	LLM API Cost	Embedding Cost	Total Cost
Exact cache hit	Zero	Zero	Zero
Semantic cache hit	Zero	Embedding only	Minimal
Cache miss	Full model cost	Embedding generation	Full + embedding

Bifrost also handles cost calculation natively through CalculateCostWithCacheDebug, which automatically accounts for cache hits, semantic matches, and misses in your cost tracking. All pricing data is cached in memory for O(1) lookup, so the cost calculation itself adds no overhead.

Check out the full Bifrost documentation for the complete API reference.

Setting it up

Follow the setup guide to get Bifrost running, then configure two things: a vector store and the semantic cache plugin.

Step 1: Configure the vector store

Bifrost uses Weaviate as its vector store. You can run Weaviate locally with Docker or use Weaviate Cloud.

Local setup with Docker:

docker run -d \
  -p 8080:8080 \
  -p 50051:50051 \
  -e PERSISTENCE_DATA_PATH='/var/lib/weaviate' \
  semitechnologies/weaviate:latest

config.json (local Weaviate):

{
  "vector_store": {
    "enabled": true,
    "type": "weaviate",
    "config": {
      "host": "localhost:8080",
      "scheme": "http"
    }
  }
}

config.json (Weaviate Cloud):

{
  "vector_store": {
    "enabled": true,
    "type": "weaviate",
    "config": {
      "host": "your-cluster.weaviate.network",
      "scheme": "https",
      "api_key": "your-weaviate-api-key"
    }
  }
}

Step 2: Configure the semantic cache plugin

Add the plugin to your Bifrost config:

{
  "plugins": [
    {
      "enabled": true,
      "name": "semantic_cache",
      "config": {
        "provider": "openai",
        "embedding_model": "text-embedding-3-small",
        "ttl": "5m",
        "threshold": 0.8,
        "conversation_history_threshold": 3,
        "exclude_system_prompt": false,
        "cache_by_model": true,
        "cache_by_provider": true,
        "cleanup_on_shutdown": true
      }
    }
  ]
}

A few things to note about these settings:

threshold: The similarity score (0 to 1) required for a semantic match. 0.8 is a good starting point. Higher means stricter matching, fewer false positives, but more cache misses.
conversation_history_threshold: Defaults to 3. If a conversation has more messages than this, caching is skipped. Long conversations have high probability of false positive semantic matches due to topic overlap, and they rarely produce exact hash matches anyway.
ttl: How long cached responses stay valid. Accepts duration strings like "30s", "5m", "1h", or numeric seconds.
cache_by_model and cache_by_provider: When true, cache entries are isolated per model and provider combination. A GPT-4 response will not be returned for a GPT-3.5-turbo request.

Step 3: Trigger caching per request

Caching is opt-in per request. You need to set a cache key, either via the Go SDK or HTTP headers:

HTTP API:

# This request WILL be cached
curl -H "x-bf-cache-key: session-123" \
     -H "Content-Type: application/json" \
     -d '{"model": "gpt-4", "messages": [{"role": "user", "content": "What is semantic caching?"}]}' \
     http://localhost:8080/v1/chat/completions

Go SDK:

ctx = context.WithValue(ctx, semanticcache.CacheKey, "session-123")
response, err := client.ChatCompletionRequest(ctx, request)

Without the cache key, requests bypass caching entirely. This gives you fine-grained control over what gets cached and what does not.

Per-request overrides (HTTP):

curl -H "x-bf-cache-key: session-123" \
     -H "x-bf-cache-ttl: 30s" \
     -H "x-bf-cache-threshold: 0.9" \
     http://localhost:8080/v1/chat/completions

Cache type control:

# Direct hash matching only (fastest, no embedding cost)
curl -H "x-bf-cache-key: session-123" \
     -H "x-bf-cache-type: direct" ...

# Semantic similarity search only
curl -H "x-bf-cache-key: session-123" \
     -H "x-bf-cache-type: semantic" ...

# Default: both (direct first, semantic fallback)
curl -H "x-bf-cache-key: session-123" ...

You can also use no-store mode to read from cache without storing the response:

curl -H "x-bf-cache-key: session-123" \
     -H "x-bf-cache-no-store: true" ...

When semantic caching helps vs when it does not

Semantic caching is not a universal solution. Here is where it works well and where it does not.

Good fit:

Customer support bots where users ask the same questions in different words
FAQ-style applications with predictable query patterns
RAG pipelines where similar contexts produce similar queries
Internal tools where multiple team members ask overlapping questions
Any high-volume application with repetitive prompt patterns

Not a good fit:

Conversations that are heavily context-dependent and unique every time
Long multi-turn conversations (the conversation_history_threshold exists for this reason, as longer conversations create false positive matches)
Applications where responses must reflect real-time data that changes frequently
Creative generation tasks where you want varied outputs for similar inputs

The key insight is that semantic caching works best when your application naturally produces clusters of similar requests. If every request is genuinely unique, caching of any kind will not help much.

Other performance details worth knowing

Beyond semantic caching, Bifrost caches aggressively at multiple levels:

Tool discovery is cached after the first request, bringing subsequent lookups down to roughly 100-500 microseconds.
Health check results are cached at approximately 50 nanoseconds.
All pricing data is cached in memory for O(1) lookups during cost calculations.

Cache entries use namespace isolation. Each Bifrost instance gets its own vector store namespace to prevent conflicts. When the Bifrost client shuts down (with cleanup_on_shutdown set to true), all cache entries and the namespace itself are cleaned up. You can also programmatically clear cache by key or clear cache by request ID via the API.

Cache metadata is automatically added to responses via response.ExtraFields.CacheDebug, so you can inspect whether a response came from direct cache, semantic match, or a fresh provider call. You can also use the log statistics API for deeper observability into your cache performance.

Wrapping up

If your GPT-powered application handles any volume of requests, there is a good chance a meaningful portion of those requests are semantically similar. Paying full API cost for every one of them does not make sense.

Bifrost's semantic cache plugin gives you dual-layer caching with exact matching and vector similarity search, opt-in per request, configurable thresholds, and built-in cost tracking. It is open source, written in Go, and designed for production workloads.

Check out the GitHub repo to get started, read the docs for the full configuration reference, or visit the Bifrost website to learn more about the gateway.

LLM Cost Tracking and Spend Management for Engineering Teams

Pranay Batta — Wed, 01 Apr 2026 05:43:30 +0000

Your team ships a feature using GPT-4, it works great in staging, and then production traffic hits. Suddenly you are burning through API credits faster than anyone expected. Multiply that across three providers, five teams, and a few hundred thousand requests per day. Good luck figuring out where the money went.

We built Bifrost, an open-source LLM gateway in Go, and cost tracking was one of the first problems we had to solve properly. This post covers what we learned, how we designed spend management into the gateway layer, and what the alternatives look like. You can get started with the setup guide in under a minute.

TL;DR: Bifrost gives you per-request cost logging, four-tier budget hierarchies (Customer, Team, Virtual Key, Provider Config), auto-synced model pricing, and cache-aware cost calculations. All at 11 microsecond latency overhead. You can run it right now with npx -y @maximhq/bifrost. Full docs here.

The actual problem with LLM costs

Cloud compute costs are predictable. You pick an instance type, you know the hourly rate, you can forecast monthly spend within a few percent.

LLM costs are nothing like that.

A single API call costs somewhere between $0.0001 and $0.50 depending on the model, the input length, the output length, whether you are sending images or audio, and whether the context crosses the 128k token threshold (where pricing tiers change). That is per request.

Now add multi-provider routing. Your app might use OpenAI for chat, Anthropic for analysis, and a smaller model for classification. Each provider has different pricing structures, different token counting methods, and different billing cycles.

The result: engineering teams have no idea what they are spending until the invoice arrives.

What cost tracking actually requires

Most teams start with "we will check the provider dashboard." That breaks down fast for three reasons.

Per-request granularity. You need to know the cost of every single API call, tied to which customer, which team, and which feature triggered it. Provider dashboards give you aggregate numbers, not per-request attribution.

Real-time budget enforcement. Knowing you overspent last month does not help. You need the system to reject requests when a budget limit is hit, before the money is gone.

Multi-modal cost calculation. If your app sends images, audio, or very long contexts, the cost calculation is not a simple token multiplication. You need tiered pricing support, per-image costs, per-second audio costs, and character-based pricing for certain models.

How we built cost tracking in Bifrost

We wanted cost management to be a gateway-level concern, not something each application team has to implement. Here is how the pieces fit together.

Model Catalog with auto-synced pricing

The Model Catalog is the foundation. It maintains pricing data for every supported model across all providers. You can also force a pricing sync at any time via the API.

On startup, Bifrost downloads the latest pricing sheet and loads it into memory. When a ConfigStore (SQLite or PostgreSQL) is available, it also persists the data and re-syncs every 24 hours automatically. All lookups are O(1) from memory.

The pricing data covers multiple modalities:

Text: token-based and character-based pricing for chat completions, text completions, and embeddings
Audio: token-based and duration-based pricing for speech synthesis and transcription
Images: per-image costs with tiered pricing for high-token contexts
Tiered pricing: automatic rate changes above 128k tokens, reflecting actual provider pricing

This means cost calculation is accurate for every request type, not an approximation based on token count alone.

Four-tier budget hierarchy

This is where spend management happens. Bifrost supports budgets at four levels:

Customer - set a spending cap for an entire customer account
Team - limit spend per team within a customer
Virtual Key - control costs per API key (useful for per-feature or per-environment budgets)
Provider Config - cap total spend on a specific provider

Each budget has a max_limit, a reset_duration (daily, weekly, monthly), and tracks current_usage in real time.

Here is what creating a customer with a budget looks like via the API:

curl --request POST \
  --url http://localhost:8080/api/governance/customers \
  --header 'Content-Type: application/json' \
  --data '{
    "name": "acme-corp",
    "budget": {
      "max_limit": 500,
      "reset_duration": "monthly"
    }
  }'

The response includes the budget object with current_usage tracked automatically:

{
  "customer": {
    "id": "cust-abc123",
    "name": "acme-corp",
    "budget": {
      "id": "bdgt-xyz",
      "max_limit": 500,
      "reset_duration": "monthly",
      "current_usage": 0
    }
  }
}

When current_usage hits max_limit, requests are rejected. No surprises on the invoice.

LogStore: per-request cost audit trail

Every request that passes through Bifrost gets logged with full cost data. The LogStore captures:

Provider and model used
Input tokens, output tokens, total tokens
Calculated cost (broken down into input cost, output cost, request cost, total cost)
Latency
Status (success or error)
Timestamps
Full input/output content (serialized as JSON)

You can query this data with filters. Want to see all requests to OpenAI that cost more than $0.10 in the last hour? That is a single API call.

curl --request POST \
  --url http://localhost:8080/api/logs/search \
  --header 'Content-Type: application/json' \
  --data '{
    "filters": {
      "providers": ["openai"],
      "min_cost": 0.10,
      "start_time": "2026-03-31T00:00:00Z"
    },
    "pagination": {
      "limit": 50,
      "sort_by": "cost",
      "order": "desc"
    }
  }'

The response includes aggregated stats alongside individual logs: total requests, success rate, average latency, total tokens, and total cost for the query. This is the data you need for cost attribution and chargeback.

Getting started

You can have this running in under a minute:

npx -y @maximhq/bifrost

Or with Docker if you prefer containerized deployment. Then point your LLM calls at the Bifrost endpoint instead of directly at the provider — it works as a drop-in replacement for the OpenAI SDK, Anthropic SDK, and Bedrock SDK. Cost tracking, budget enforcement, and logging happen automatically.

Check the setup docs for configuration details.

Cache-aware cost tracking

This is a detail that matters more than you would expect.

Bifrost includes a dual-layer semantic cache (exact hash matching + semantic similarity via Weaviate). When a request hits the cache, the cost calculation changes:

Direct cache hit (exact match): zero cost. The response comes from cache, no provider API call is made.
Semantic cache hit (similar query found): the cost is the embedding generation cost only. No model inference cost.
Cache miss with storage: the cost is the base model usage plus the embedding generation cost for storing the result.

If you are not tracking cache-aware costs, your cost reports will overcount. Every cache hit that gets reported at full model price inflates your numbers and hides the ROI of caching.

How other tools handle cost tracking

Credit where it is due. There are several tools in this space, and they each take a different approach.

Helicone is a proxy-based observability platform. It logs requests and provides cost analytics through a dashboard. The cost tracking is solid, with per-request granularity. Where it differs from Bifrost: Helicone is primarily an observability tool. Budget enforcement and cache-aware cost calculations are not its focus. It is a good choice if you want analytics without gateway-level controls.

OpenRouter acts as a unified API layer across multiple LLM providers. It handles routing and gives you a single bill, which simplifies accounting. However, OpenRouter is a hosted proxy — your requests pass through their infrastructure. There is no self-hosted option, no budget enforcement at the gateway level, and no per-customer or per-team spend hierarchy. If you need cost attribution beyond "which model was called," you will need to build that yourself on top of their logs.

AWS API Gateway + Bedrock is what many AWS-native teams reach for. You get IAM-based access control and CloudWatch metrics. The limitation is that cost tracking is coarse-grained — you get aggregate billing through AWS Cost Explorer, not per-request cost breakdowns tied to your internal teams or customers. Building a four-tier budget hierarchy on top of AWS services means stitching together Lambda, DynamoDB, and custom billing logic. It works but it is a lot of glue code.

Kong AI Gateway and Cloudflare AI Gateway both provide rate limiting and basic analytics for AI API traffic. Kong gives you plugin-based extensibility, and Cloudflare gives you edge caching and DDoS protection. Neither provides built-in per-request cost calculation with multi-modal pricing awareness, and neither offers the kind of budget hierarchy where you can set spending caps at the customer, team, and key level with automatic enforcement.

LiteLLM is the most well-known Python-based proxy. It supports cost tracking and has a wide model coverage. The trade-off is performance. LiteLLM adds roughly 8ms of latency overhead per request. Bifrost adds 11 microseconds, which is about 50x faster. At 5,000 RPS, that difference compounds. If your use case is low-throughput internal tooling, LiteLLM works fine. If you are running production workloads at scale, the latency overhead matters.

The math is straightforward: at 5,000 requests per second, 8ms overhead means 40 seconds of cumulative latency overhead per second of wall time. At 11 microseconds, it is 0.055 seconds.

What we learned building this

A few things surprised us during development.

Pricing data goes stale fast. Providers update pricing regularly. We started with a static pricing file and quickly realized it needed to be auto-synced. The 24-hour sync interval with O(1) memory lookups was the balance we settled on. You can also trigger a manual pricing sync via POST /api/pricing/force-sync if a provider drops prices and you want immediate accuracy.

Budget enforcement needs to be in the hot path. We tried implementing budgets as an async check initially. The problem: by the time the async check ran, the request was already sent to the provider and the cost was incurred. Budget checks have to happen before the request goes upstream. That is why Bifrost handles it at the gateway layer with in-memory state.

Multi-modal cost calculation is harder than it looks. Text-only cost is straightforward: multiply tokens by price per token. But when a request includes images, the cost depends on the image resolution and the token context length. Audio adds per-second pricing. Some models charge per character instead of per token. The Model Catalog handles all of this, but getting it right required modelling each provider's pricing structure individually.

Cost attribution needs hierarchy. Flat per-key budgets are not enough for real organizations. An engineering team needs to know: "How much is Customer X spending? How much of that is Team Y? Which virtual key is burning through budget?" That is why we built the four-tier hierarchy (Customer, Team, Virtual Key, Provider Config). You can create virtual keys via the API and attach budgets to each level.

Wrapping up

LLM cost management is not optional for production systems. If you are routing requests across multiple providers without per-request cost tracking, budget enforcement, and cache-aware calculations, you are flying blind. For enterprise teams, Bifrost also supports audit logs, log exports, and intelligent load balancing.

Bifrost is open-source, written in Go, and runs with a single command. It handles cost tracking at the gateway layer so your application code does not have to.

If you are dealing with LLM spend management, give it a try and let us know what is missing. We are actively building based on what teams actually need.

LiteLLM vs Bifrost: Why the Supply Chain Attack Changes Everything for LLM Gateways

Pranay Batta — Sat, 28 Mar 2026 05:10:03 +0000

If you're running LiteLLM in production, the March 2026 supply chain attack probably got your attention. Mine too. I spent the past few days digging into what happened, why it happened, and what it means for anyone choosing an LLM gateway in 2026.

This is not a hit piece. LiteLLM is a solid project with massive adoption. But this incident exposed something structural that every engineering team needs to think about. And it happens to make the case for Bifrost, a Go-based alternative, in ways that go beyond the usual performance benchmarks.

Let's break it all down.

TL;DR

Two backdoored versions of LiteLLM (1.82.7, 1.82.8) were published to PyPI on March 24, 2026, via stolen credentials.
The malware stole SSH keys, AWS/GCP/Azure credentials, and Kubernetes secrets. It used Python's .pth persistence mechanism to survive across interpreter restarts.
DSPy, MLflow, CrewAI, OpenHands, and Arize Phoenix all pulled the compromised version.
Bifrost is a Go-based LLM gateway that compiles to a single binary. The attack vector that hit LiteLLM simply does not exist in its architecture.
Beyond security, Bifrost adds 11 microseconds of overhead per request vs LiteLLM's roughly 8ms, supports 20+ providers, offers semantic caching via Weaviate, and has a four-tier budget hierarchy.

What Happened: The Full Attack Chain

Here's the sequence of events, based on Snyk's detailed investigation.

Step 1: The Trivy GitHub Action was compromised. A group called TeamPCP tampered with the widely-used Trivy security scanner GitHub Action.

Step 2: LiteLLM's CI/CD pipeline pulled the compromised Trivy Action. Because LiteLLM's workflow used an unpinned version of the Trivy GitHub Action (not pinned to a specific SHA), the compromised version ran inside LiteLLM's CI environment.

Step 3: The malicious Trivy Action exfiltrated LiteLLM's PYPI_PUBLISH token. With that token, the attackers could publish any package version to PyPI under LiteLLM's name.

Step 4: Two backdoored versions (1.82.7, 1.82.8) were published to PyPI. These looked like normal LiteLLM updates. Anyone running pip install --upgrade litellm got them.

Step 5: The malware deployed a .pth persistence file. This is the part that needs explaining.

What are .pth files?

If you're not deep into Python internals, .pth files might be new to you. They live in Python's site-packages directory and get executed automatically every time the Python interpreter starts up. Not when you import a specific package. Every single time Python runs. Anything.

The attackers placed a .pth file that loaded their malware on every Python interpreter startup. It did not matter whether your code imported litellm or not. If the package was installed in the environment, the malware was active.

What the malware stole:

SSH private keys
AWS, GCP, and Azure credentials
Kubernetes secrets
Crypto wallet keys

Step 6: The attackers used 73 compromised GitHub accounts to spam the disclosure issue with noise and eventually closed it using stolen maintainer credentials, trying to suppress the report.

The backdoored versions were live on PyPI for approximately 3 hours. LiteLLM has 3.4 million+ daily downloads. You can do the math on the blast radius.

Why Architecture Matters More Than You Think

Let's talk about why this specific attack cannot happen to Bifrost.

It is not just "Bifrost is written in Go, so it's safe." That would be a lazy argument. The actual reasons are architectural, and they matter.

No site-packages directory

Python packages install into site-packages. That directory is a shared space where any installed package can drop files, including .pth files that execute on interpreter startup. This is the mechanism the LiteLLM attackers exploited.

Go compiles to a single static binary. There is no site-packages equivalent. There is no shared directory where a compromised dependency could drop a persistence mechanism. The binary is the binary.

No .pth hook mechanism

Python's .pth file execution is a feature, not a bug. It exists for legitimate reasons (configuring import paths, running initialization code). But it also means any package you install can run arbitrary code on every Python startup without your knowledge or consent.

Go has no equivalent mechanism. When you compile a Go binary, what goes in is what comes out. There is no startup hook that third-party code can inject into after compilation.

No transitive pip dependency chain

LiteLLM has a substantial dependency tree. Each of those dependencies has its own dependencies. Each one is a potential attack surface. When you pip install litellm, you're trusting not just the LiteLLM maintainers but every maintainer of every transitive dependency.

Bifrost ships as a compiled binary via npx -y @maximhq/bifrost or Docker (docker pull maximhq/bifrost). Dependencies are resolved and compiled at build time by the Bifrost team. You're running a single binary, not managing a dependency tree.

The CI/CD surface area is smaller

The LiteLLM attack started with a compromised GitHub Action in CI/CD. Go binaries distributed via npm or Docker reduce the CI/CD surface area because the compilation and dependency resolution happen upstream, not in your pipeline.

This is not about Go being "more secure" than Python as a language. It's about the deployment model. A compiled binary distributed as a single artifact has a fundamentally smaller attack surface than a package installed via a package manager with a transitive dependency tree and runtime hook mechanisms.

Side-by-Side Feature Comparison

Here's an honest look at both gateways.

Feature	LiteLLM	Bifrost
Language	Python	Go
Deployment	`pip install`, Docker	`npx`, Docker, Go binary
Provider support	100+ providers	20+ providers (OpenAI, Anthropic, Bedrock, Azure, Gemini, Vertex AI, Groq, Mistral, Cohere, xAI, and more) + custom providers
Overhead per request	~8ms	11 microseconds
Throughput	Varies (Python GIL limits)	5,000 RPS sustained
Caching	Redis-based key-value	Weaviate-powered dual-layer semantic caching
Budget management	Basic spend tracking	Four-tier hierarchy (Customer > Team > Virtual Key > Provider Config)
MCP support	Limited	Full MCP gateway with four connection types, sub-3ms latency
Web UI	Dashboard available	Built-in Web UI for visual setup, monitoring, and governance
OpenAI compatibility	Yes	Yes (drop-in replacement, single URL change)
Supply chain surface	PyPI + transitive deps + .pth hooks	Single compiled binary
Configuration	Config files, environment variables	Zero-config start, Web UI, API, or config.json

Let me be upfront: LiteLLM's provider count is significantly higher. If you need access to 100+ providers through a single gateway, that is a real advantage. Bifrost supports 20+ providers natively with the ability to add custom providers, which covers most production use cases, but it is not the same breadth.

Performance Deep Dive: What the Numbers Actually Mean

You'll see "11 microseconds vs 8 milliseconds" in Bifrost's benchmarks. That's roughly a 50x difference. But what does it mean in practice?

Let's do the math at different scales.

At 10,000 requests per day:

LiteLLM overhead: 10,000 x 8ms = 80 seconds of cumulative gateway latency
Bifrost overhead: 10,000 x 11 microseconds = 0.11 seconds

At 100,000 requests per day:

LiteLLM overhead: 100,000 x 8ms = 800 seconds (~13.3 minutes)
Bifrost overhead: 100,000 x 11 microseconds = 1.1 seconds

At 1,000,000 requests per day:

LiteLLM overhead: 1,000,000 x 8ms = 8,000 seconds (~2.2 hours)
Bifrost overhead: 1,000,000 x 11 microseconds = 11 seconds

At low volume, the difference doesn't matter much. Your LLM provider's response time (hundreds of milliseconds to seconds) dwarfs the gateway overhead either way.

But at scale, the difference becomes real. 13 minutes of cumulative latency at 100K requests/day isn't catastrophic, but it adds up across your user base. And 2.2 hours at a million requests/day starts affecting tail latencies and user experience, especially for streaming responses where gateway overhead is felt on every chunk.

The 5,000 RPS sustained throughput from Bifrost also matters. Python's GIL (Global Interpreter Lock) creates a concurrency ceiling that Go simply doesn't have. If you're running high-concurrency workloads, this is a material difference.

Here's What This Means for Your Stack

If you're evaluating LLM gateways right now, the LiteLLM incident should change your evaluation criteria. Not because LiteLLM is bad software, but because it highlighted a category of risk that most teams weren't thinking about.

Questions to ask about any LLM gateway:

What's the dependency footprint? How many transitive dependencies does it pull in? Each one is a potential attack surface.
What's the deployment model? Is it a package you install into your environment, or a standalone binary/container?
Does it have runtime hook mechanisms? Can dependencies execute code at startup without explicit imports?
How is it distributed? Via a package manager with mutable versions, or via immutable artifacts?
What's in the CI/CD chain? Are GitHub Actions pinned by SHA? Are publish tokens scoped and rotated?

These aren't questions most teams were asking about their LLM gateway a month ago. They should be now.

When LiteLLM Still Makes Sense

I want to be honest about this. There are real scenarios where LiteLLM is the better choice.

You need access to 100+ providers. LiteLLM's provider breadth is unmatched. If you're working with niche or specialized providers that Bifrost doesn't support yet, LiteLLM gets you there faster.
Your entire stack is Python and you want deep integration. LiteLLM plays well with the Python ML ecosystem. If you're already in that world and need tight integration with LangChain, LlamaIndex, or similar frameworks, LiteLLM fits naturally.
You need it as a library, not a gateway. LiteLLM can be imported and used as a Python library within your application code. Bifrost is a standalone gateway service.

If any of these are your primary requirement, LiteLLM may still be right for you. Just audit your versions, pin your dependencies, and check for .pth files in your site-packages directory.

When Bifrost Is the Better Choice

Bifrost wins when your priorities look like this:

Security surface area matters to you. If you're in a regulated industry, handle sensitive data, or simply don't want to worry about Python supply chain attacks in your infrastructure layer, a compiled Go binary is a different risk profile entirely.
Performance at scale. If you're pushing high request volumes and need minimal gateway overhead, 11 microseconds vs 8 milliseconds is not a rounding error.
You want governance out of the box. Bifrost's four-tier budget hierarchy (Customer > Team > Virtual Key > Provider Config) with independent budget checking at each level gives you cost control that's built into the gateway, not bolted on.
Semantic caching. Bifrost's Weaviate-powered dual-layer caching understands the meaning of requests, not just exact matches. Similar queries hit the cache even if they're worded differently.
MCP gateway support. If you're building agentic applications, Bifrost has native MCP support with four connection types and sub-3ms tool execution latency.
Zero-config setup. Run npx -y @maximhq/bifrost and you have a working gateway with a Web UI at localhost:8080. No config files, no environment variables, no setup ceremony.

The Bigger Question

Should your LLM gateway be a Python package at all?

This isn't Python-bashing. Python is great for ML research, data science, prototyping, and application-level code. But your LLM gateway sits in the critical path of every AI request your application makes. It's infrastructure.

Infrastructure components have different requirements than application code. They need to be fast, stable, have minimal dependencies, and present the smallest possible attack surface. This is why web servers, databases, load balancers, and message queues are almost never written in Python. They're written in C, C++, Go, or Rust.

The LiteLLM incident didn't happen because of a bug in LiteLLM's code. It happened because of a structural property of the Python packaging ecosystem. That's a different kind of risk, and it's one that applies to any Python package in your infrastructure layer.

Action Items

If you're currently using LiteLLM:

Check your installed version immediately. Versions 1.82.7 and 1.82.8 are compromised.
Search for .pth files in your Python site-packages directories.
Rotate all credentials that were accessible from environments where LiteLLM was installed (SSH keys, cloud provider credentials, Kubernetes secrets).
Pin your GitHub Actions by SHA, not by tag.
Evaluate whether a compiled gateway is a better fit for your security posture.

If you're evaluating LLM gateways:

Try Bifrost: npx -y @maximhq/bifrost (takes 30 seconds)
Check out the GitHub repo to see the codebase
Read the docs for the full feature set
Visit the website for architecture details

The LLM gateway space is going to look different after this incident. Supply chain security just became an evaluation criterion, and compiled gateways have a structural advantage that no amount of Python dependency scanning can match.