It's the AI Era — Why Are You Still Using nginx for Your LLM Gateway?

tao zeng — Fri, 23 Jan 2026 02:48:05 +0000

Introduction

2024 was the year LLM applications exploded. Countless companies began integrating ChatGPT, Claude, Gemini, and other LLM services into their products. As a backend architect, you probably instinctively reached for your old friend nginx — configure an upstream, add a rate-limiting module, write a few lines of Lua script — done!

But soon you'll discover things aren't that simple:

• Streaming responses lag: Users stare at "Thinking..." for 5 seconds before seeing the first character
• Token metering is inaccurate: SSE streams get truncated by nginx buffering, usage stats don't add up
• Rate limiting fails: Limiting by request count? A single streaming conversation can run for 30 seconds, and your concurrency explodes
• Costs spiral out of control: One backend goes down, traffic doesn't intelligently failover, retry errors burn through your quota

This isn't because nginx is bad — it simply wasn't designed for the LLM era.

nginx's "Original Sin": General-Purpose Proxy vs. Specialized Use Cases

nginx was born in 2004, created to solve the C10K problem as a high-performance HTTP server. Its design philosophy is generality:

• Static file serving
• Reverse proxy and load balancing
• HTTP caching and compression
• SSL/TLS termination

But LLM API gateway requirements are completely different:

Zero-Latency Streaming vs. Buffer Optimization

nginx buffers upstream responses by default (proxy_buffering on). Even if you disable buffering, its event loop and memory management aren't optimized for "byte-by-byte passthrough." The result:
LLMProxy was designed from day one with TTFT (Time To First Token) optimization as its first principle. It uses a zero-buffer architecture where every SSE event is flushed to the client immediately upon arrival, introducing no additional latency.

Token-Level Metering vs. Request-Level Statistics

nginx's logging and statistics unit is the request, but LLM billing is based on tokens:
nginx
To implement token statistics in nginx, you need to:

Parse the complete response body with lua-nginx-module
Extract the usage field from JSON (in streaming responses, it's in the final SSE event)
Write to a database or call a webhook
Handle stream disconnections, parsing failures, race conditions...

This logic is async and out-of-the-box in LLMProxy, supporting database, webhook, and Prometheus reporting methods, with the ability to integrate with authentication systems for quota deduction.

Intelligent Routing vs. Simple Round-Robin

nginx's load balancing strategy:
nginx
Looks great, but in reality:
• api1 has 200ms latency, api2 has 50ms → nginx doesn't know, keeps round-robin
• api1 returns a 429 rate limit error → nginx marks it as failed, but user quota was already consumed
• Want to use Claude as a fallback for GPT → nginx's backup only kicks in when ALL primary nodes are down

LLMProxy's intelligent router can do this:
yaml
Every request dynamically selects the optimal backend based on real-time health, historical latency, and error rates. Failures auto-retry without double-billing.

Modern Authentication vs. Hand-Written Lua Scripts

To implement "check Redis for API Key + verify quota + IP allowlist" authentication logic in nginx, you need:
lua
LLMProxy's authentication pipeline is declarative:
yaml
Each provider can have custom Lua scripts for fine-grained control, but infrastructure (connection pools, error recovery, performance metrics) is managed by the framework.

Performance Comparison: Let the Data Speak

We tested nginx (OpenResty) and LLMProxy on identical hardware in a typical LLM proxy scenario:
| Metric | nginx + lua | LLMProxy | Notes |
| --------------------------- | ----------- | ----------- | ----------------------------- |
| TTFT | 120-300ms | 5-15ms | Time to first token (p50) |
| Streaming throughput | ~800 req/s | ~2400 req/s | 100 concurrent, streaming |
| Memory usage | 1.2GB | 380MB | Steady state, 10k connections |
| Token metering accuracy | 93%* | 99.99% | Requires custom Lua logic |
| **Health check overhead* | ±50ms/req | <1ms/req | Backend health probe impact |
Test environment: 4 cores, 8GB RAM, backend simulating 20ms latency streaming LLM API

Why is it so much faster?

Go's concurrency model: goroutines + channels are naturally suited for many long-lived connections
Zero-copy streaming: Read directly from upstream socket → write to downstream, no intermediate buffering
Dedicated metrics collection: Async goroutines handle metering without blocking the hot path
Smart connection reuse: Maintains long-lived connection pools to backend LLM APIs, avoiding TLS handshake overhead

Real-World Scenario: The Painful Story of an AI Customer Service System

An online education company deployed an AI customer service bot with nginx. It crashed the day after launch:

Problem 1: Poor streaming experience, users complaining about "lag"
• nginx config: proxy_buffering off; proxy_cache off;
• Root cause: nginx's event loop reads at most 8KB at a time, streaming responses get chunked
• LLMProxy solution: Zero-buffer engine + immediate flush, SSE events delivered to client in <5ms

Problem 2: Costs exploded, token usage 40% higher than expected
• nginx symptom: Couldn't accurately track per-session token consumption
• Investigation revealed: Retry logic had a bug, same request was billed multiple times; usage lost on stream disconnect
• LLMProxy solution: Async metering with at-least-once semantics, database + webhook dual-write prevents data loss

Problem 3: Peak hour avalanche, entire site crashed after one API endpoint returned 429
• nginx behavior: Upstream 429 counted as "failure," triggered circuit breaker; backup node instantly got 10x the load
• LLMProxy solution: 429 triggers automatic fallback to backup model, with separate rate limiting per backend URL

Final results:
• After switching to LLMProxy, P99 latency dropped from 8 seconds to 1.2 seconds
• Token metering accuracy improved from 85% to 99%+
• Monthly API costs decreased 30% (reduced invalid retries and quota waste)

"But I Already Have nginx — Why Switch?"

This is the most common question. The answer depends on your stage:

If you're still in POC/MVP stage
• nginx is fine, don't over-engineer
• But leave room for upgrades (don't hardcode business logic into nginx.conf)

If you already have real user traffic
You've probably already encountered these pain points:
• ✅ Users complaining about slow, laggy responses
• ✅ Token metering doesn't match, finance can't reconcile the books
• ✅ Want A/B testing, model fallback, but nginx config is unmaintainable
• ✅ Monitoring only shows request count/bandwidth, not business metrics (tokens/cost/model distribution)

At this point, LLMProxy lets you accomplish in half a day what nginx + Lua takes a week to implement.

If you're building an enterprise LLM platform
You don't need a proxy — you need an LLM gateway:
• Multi-tenant isolation and quota management
• Fine-grained cost accounting and billing
• Compliance auditing (logging all prompts and completions)
• Model routing and canary deployments
• Custom request/response transformations

Can nginx do all this? Theoretically yes, but you'll need:
• OpenResty + extensive custom Lua modules
• External services (auth center, quota management, audit logging...)
• Your own complex configuration and operations toolchain

LLMProxy provides a complete solution out of the box — you just write YAML config and business logic Lua scripts.

Quick Start: Migrate from nginx to LLMProxy in 15 Minutes
bash
Production migration checklist:

[ ] Deploy LLMProxy alongside nginx, route 10% traffic for validation
[ ] Set up monitoring (Prometheus + Grafana) to compare against nginx metrics
[ ] Migrate authentication logic (API key validation, quota management)
[ ] Configure intelligent routing and fallback rules
[ ] Full cutover, keep nginx as backup
[ ] One week observation period, then decommission nginx

Conclusion: Use the Right Tool for the Job

nginx is great software. It has proven its dominance in web serving over 20 years. But LLM API gateway is an entirely new domain that requires:

• Microsecond-level streaming optimization
• Async token-level metering and cost accounting
• Business-semantic intelligent routing
• Flexible multi-tenancy and quota management

Implementing these requirements with a general-purpose HTTP proxy is like using a screwdriver to hammer nails — theoretically possible, practically painful.

LLMProxy's mission is to make LLM API gateways simple again:
• Developers write YAML config, not Lua scripts
• Ops see business metrics (tokens/cost/models), not just QPS/latency
• Product managers do canary releases and A/B tests without waiting for engineering sprints

Project: GitHub - LLMProxy

Documentation: Complete architecture design, configuration reference, best practices

Community: Issues and Discussions welcome for questions and contributions

In the AI era, your gateway should be AI-native too. What's your choice?

Forem: tao zeng

It's the AI Era — Why Are You Still Using nginx for Your LLM Gateway?