Forem: Portkey

Claude Code agents: what they are, how they work, and how to scale them

Drishti Shah — Mon, 09 Mar 2026 14:57:31 +0000

Claude Code is now the most widely used AI coding agent. We all know what it does. The harder question is what happens when you roll it out across a team of 20, 50, or 200 engineers, each running agentic loops that spawn subagents, call MCP tools, and burn through tokens autonomously.

This post covers the agent architecture inside Claude Code, what breaks when you take it to production, and how to add the governance and observability layer that enterprises need.

How Claude Code agents work under the hood

Claude Code runs a straightforward agent loop: the model produces a message, and if it includes a tool call, the tool executes and results feed back into the model. No tool call means the loop stops and the agent waits for input. It ships with around 14 tools spanning file operations, shell commands, web access, and control flow.

What makes the system powerful is the layered agent architecture sitting on top of that loop.

Subagents

Claude Code delegates to specialized subagents that each run in their own context window. The built-in ones include Explore (read-only codebase search), Plan (research for planning mode), and a general-purpose agent for complex multi-step tasks. Each subagent has restricted tool access and independent permissions.

The real value here is context management. Agentic tasks fill up the context window fast. Subagents keep exploration and implementation out of the main conversation, preserving the primary context for decision-making.

You can also define custom subagents as markdown files with YAML frontmatter, scoped to specific tools, models, and system prompts. A read-only code reviewer, a documentation generator, a security auditor, each with exactly the permissions it needs.

Agent teams

Subagents work within a single session. Agent teams coordinate across separate sessions, enabling parallel workflows. Think: one agent building a backend API while another builds the frontend, each in an isolated Git worktree.

Anthropic's own 2026 trends report names multi-agent coordination as a top priority for engineering teams this year.

Skills, hooks, and the Agent SDK

Skills are reusable instruction packages that Claude loads automatically when it encounters a matching task. Hooks trigger actions at specific workflow points (run tests after changes, lint before commits). The Claude Agent SDK gives teams the same underlying tools and permissions framework to build custom agent experiences outside the terminal entirely.

What breaks at scale

None of these capabilities ship with the operational scaffolding that production environments demand. Here is where teams consistently run into trouble.

Runaway costs with no attribution

A single complex task can spawn multiple subagents, each making dozens of tool calls across extended agentic loops. Multiply that by a team of engineers, and token spend becomes unpredictable. Claude Code has no native cost tracking by user, team, or project. The first monthly bill is usually the wake-up call.

Zero observability

There is no built-in logging, tracing, or audit trail. You cannot see what prompts were run, which subagents were spawned, how long requests took, or where errors occurred. For debugging, optimization, and compliance, this is a non-starter.

Single-provider fragility

Claude Code ties you to one provider. If that provider hits rate limits, has an outage, or is unavailable in a region you need, your workflows stop. Switching providers requires manual reconfiguration.

No access control

There is no centralized way to manage API keys, restrict access by role or team, enforce rate limits, or set budget caps. Every developer's setup is independent, which makes governance at scale nearly impossible.

Ungoverned MCP tool access

Claude Code supports MCP for connecting to external tools like GitHub, Slack, databases, and internal APIs. When agents can take real actions through these tools, the question is not whether MCP works but whether it should be allowed, for which agent, with what permissions, and who is watching. Without a governance layer, MCP adoption stalls at experimentation.

Adding governance with an AI gateway

An AI gateway sits between Claude Code and the model providers, adding observability, access control, and reliability without changing how developers use the tool. Portkey is built for exactly this.

Team-level governance

Isolate teams with separate workspaces, each with its own budget, rate limits, and access controls. Manage API keys centrally instead of distributing raw keys to individual developers. Enforce org-wide policies without building custom wrappers.

Multi-provider routing

Route Claude Code requests across Anthropic, Bedrock, and Vertex AI through a single endpoint. Define fallback logic, add load balancing or define conditions using meta data for smarter routing. Developers change nothing in their workflow. The gateway handles it.

Just add this to settings.json

{
  "env": {
    "ANTHROPIC_BASE_URL": "https://api.portkey.ai",
    "ANTHROPIC_AUTH_TOKEN": "YOUR_PORTKEY_API_KEY",
    "ANTHROPIC_CUSTOM_HEADERS": "x-portkey-api-key: YOUR_PORTKEY_API_KEY\nx-portkey-provider: @anthropic-prod"
  }
}

Cost tracking and budget controls

Every request is logged with token usage and cost, broken down by provider, team, user, and project. Set hard budget limits per team or developer. Get alerts when spend crosses thresholds. Route to cheaper providers when cost matters more than latency.

For teams running parallel agent fleets, this is the difference between controlled scaling and a surprise invoice.

Full observability

Portkey's OTEL-compliant observability captures every request with metadata: latency, tokens, error rates, provider, route, and custom tags. Filter and search across all Claude Code usage in one dashboard. Debug failed agent loops, monitor MCP tool calls, and track performance trends over time.

Guardrails

Enforce PII detection, content filtering, prompt injection protection, and token limits before requests reach the model. This matters especially for agentic workflows processing sensitive codebases or interacting with production systems through MCP.

MCP Gateway

Portkey's MCP Gateway provides centralized authentication, role-based access control, and full observability for MCP tool connections. Agents authenticate once through Portkey. The gateway handles credential injection, permission checks, and request logging for every configured MCP server.

Platform teams control exactly which teams can access which MCP servers and tools. Every tool call is logged with full context: who accessed what, when, and with what parameters. This is the layer that makes MCP safe for production use.

Putting it together

The shift happening right now is clear. Claude Code agents are moving from individual developer tools to team-level infrastructure. Subagents, agent teams, skills, MCP integrations, and orchestration layers are making agents dramatically more capable. But capability without governance is a liability.

The teams scaling Claude Code successfully are treating it as an infrastructure problem: routing through a gateway, tracking every request, setting budget boundaries, and governing tool access centrally. The developers keep typing claude in their terminal. The platform team gets the visibility and control they need.

To run Claude Code agents with governance at scale, get started with Portkey or book a demo for a walkthrough.

LLM Deployment Pipeline Explained Step by Step

Drishti Shah — Fri, 27 Feb 2026 11:41:00 +0000

LLM deployment is the process of taking a trained language model and converting it into a production service that can handle live user requests reliably and at scale.

In practice, a truly production-ready LLM system is shaped by five interconnected layers: containerization, infrastructure and GPU allocation, the API and serving layer, autoscaling, and monitoring. These layers keep performance stable, costs predictable, and outputs trustworthy as real traffic flows in.

💡 Gartner estimates that more than 80% of enterprises will have generative AI applications in production by 2026, yet most online tutorials still stop at “get it running” and ignore what happens after.

Don’t worry, though, because this article covers the full lifecycle teams struggle with post-launch, including scaling predictably under real demand, monitoring probabilistic outputs that can fail silently, and controlling costs that compound with every request.

Cloud APIs, self-hosted, or on-premises

Every LLM deployment begins with a foundational architectural decision: consume models via cloud APIs, run them yourself on cloud GPUs, or operate fully on-premises. The right choice depends on how you balance speed, control, cost, and compliance.

Architecture path

What it offers

Tradeoffs

Typical use case

|
|

Cloud APIs (OpenAI, Anthropic, Microsoft Azure)

Fastest route to production, no infrastructure management, elastic scaling

Pay-per-token costs compound, vendor lock-in risk, data residency constraints

Prototyping and early production

|
|

Self-hosted on cloud GPUs

Full control over open models (Llama, Mistral), performance tuning

You own reliability, scaling, and ops complexity

Teams needing flexibility and customization

|
|

On-premises

Strong compliance posture (HIPAA, GDPR), predictable cost at high utilization

Hardware expense, deep infrastructure expertise required

Regulated or high-volume workloads

For self-hosting, infrastructure costs typically range from about $0.75/hour for an L4 GPU up to $3.25/hour for an H100, before orchestration and redundancy. On-premises shifts that spend into upfront hardware investment that only pays off with sustained utilization.

Across all three paths, data residency and security requirements often become the deciding factor.

Also, keep in mind that according to Portkey’s LLMs in Prod 2025, 40% of teams now use multiple LLM providers, up from 23% ten months earlier. As a result, you have to carefully consider how your architectural choice will affect portability and long-term leverage to avoid the risk of vendor lock-in. You can also use Portkey’s AI Gateway to mitigate that risk by routing across 1,600+ models through a single API, so switching providers becomes a configuration change rather than a rewrite.

GPU selection and infrastructure sizing

LLMs only run fast when everything fits inside GPU memory (VRAM). That includes not just the model itself, but also the temporary memory it uses to hold conversation context while generating responses, called the KV cache. If either spills into normal system memory, performance drops sharply.

This is why sizing GPUs by model size alone often fails in production. For example:

A 70B model (FP16) needs about 140GB for weights. If you add a standard 8k context window and batch headroom, total VRAM reaches ~161GB, requiring two 80GB GPUs (like the H100) or a single H200.
An 8B model (FP16) uses roughly 16GB for weights and ~18.4GB with context overhead – an ideal fit for one 24GB L4 GPU.

So, always budget VRAM for both the model and its live working memory.

Inference frameworks and API endpoint design

The efficiency of an LLM deployment is defined by the request lifecycle:

During prefill, the full prompt is processed in parallel and is primarily compute-bound.
During decode, tokens are generated one by one and become memory-bandwidth bound.

For this, you’ll need an inference framework to manage the KV cache so the GPU never sits idle between these phases.

Framework

Key innovation

2026 performance edge

Best for

|
|

vLLM

PagedAttention

Industry standard; highest stability for general APIs.

High-concurrency production APIs.

|
|

SGLang

RadixAttention

29% higher throughput than vLLM; instant prefix caching.

Agentic and RAG workloads.

|
|

TensorRT-LLM

Compiled kernels

Lowest latency (up to 2x faster TTFT on NVIDIA).

Latency-critical systems.

|
|

LMDeploy

TurboMind C++ engine

Rivals SGLang; excellent 4-bit/8-bit throughput

High-volume batch inference.

The hierarchy of metrics

First is TTFT (Time to First Token), which is how quickly users see the first word appear – the most critical UX signal. With superior cache reuse, SGLang achieves roughly 3.7× faster TTFT at low concurrency, making it ideal for highly interactive agents.

Next is TPS (Tokens Per Second), which defines total system capacity. While vLLM is commonly the default choice, SGLang can deliver about 33% higher TPS in multi-turn conversations by avoiding repeated context processing.

Then comes KV cache utilization , which determines stability. Modern inference engines target around 80% GPU memory usage to balance throughput and safety, while pushing toward 95% can increase the risk of instability and trigger crashes during graph capture and runtime spikes.

API design for production UX

Streaming is mandatory. Total generation time matters less than TTFT – users tolerate a 10-second response if the first token appears in ~200ms.

Also, continuous batching prevents queue bottlenecks. Your framework must allow new requests to join an active decode loop, rather than waiting in a strict first-in-first-out pipeline.

For RAG systems, prefix caching is critical. By caching the system prompt and retrieved documents, you can reduce prefill latency and cost by up to 90% for returning users, dramatically improving both responsiveness and efficiency.

Scaling strategies that actually work for LLMs

Traditional autoscaling based on CPU or memory utilization fails for LLM workloads. Inference is primarily memory-bandwidth and I/O bound, so scaling decisions must reflect user-facing latency and GPU saturation, not generic resource metrics.

For Kubernetes deployments, configure the Horizontal Pod Autoscaler (HPA) around three signals:

Queue depth (num_requests_waiting) – the most responsive signal: Best practice today is to trigger scale-out when just 3–5 requests begin to queue, preventing latency spikes before users feel them.
P90 TTFT (Time to First Token) – the SLA-level experience metric: If the 90th percentile drifts beyond roughly 200–500ms, new replicas should spin up to preserve a fast, conversational feel.
GPU KV cache utilization (gpu_cache_usage_perc) – shows physical saturation: High cache usage combined with a growing queue means existing pods can’t accept more tokens without instability or crashes.

Additionally, cold starts were once the biggest constraint, with model loads taking 30–120 seconds. However, modern streaming loaders have reset that baseline. Using NVIDIA’s Run:ai Model Streamer, a 7B model can stream from object storage into GPU memory in 5–15 seconds. When paired with frameworks like vLLM, total Time to Ready (container spin-up plus engine initialization) is now under 25 seconds, reducing the need for costly warm pools.

In practice, Kubernetes users should avoid default CPU-based scaling rules. Instead, configure the HPA to scale using queue depth and TTFT, since those directly reflect user experience and system saturation.

Scaling at the infrastructure layer is only part of the solution. Above it, the AI Gateway adds production safeguards such as automatic retries, exponential backoff, circuit breakers, and provider failover. These mechanisms prevent temporary overloads or upstream failures from cascading into user-visible downtime.

With the Gateway, a leading online food delivery platform successfully handled a 3,100× traffic increase, peaking at 1,800 requests per second, while maintaining 99.99% uptime by combining HPA-based scaling with gateway-level resilience controls.

Monitoring LLMs in production

LLMs often fail silently, returning a successful status code (200 OK) while producing hallucinated or misleading outputs. Monitoring and observability are essential here: monitoring confirms the system is running, while observability confirms the response is actually correct. For this, production teams need a dual-layer telemetry approach.

Layer 1: Operational metrics

Operational metrics (the heartbeat of your inference stack) ensure the fleet is responsive and stable under load. The most important are:

P95 TTFT (Time to First Token) – the gold standard for user experience: In 2026, enterprise targets have tightened to 200–500ms to preserve a natural, conversational feel.
TPOT (Time Per Output Token): This measures streaming smoothness, with a goal of under 50ms so responses outpace human reading speed.
KV cache saturation – an early warning system: Rising cache usage paired with growing queues signals an impending generation stall or crash long before errors appear.

Layer 2: Semantic observability

Semantic observability evaluates whether answers are actually correct and safe. Using LLM-as-a-Judge techniques, teams can detect quality drift that traditional APM tools miss:

Faithfulness and grounding verify that responses truly exist in retrieved documents, forming the core guardrail for RAG systems.
Cost harvesting attacks track abusive prompts designed to explode token usage and drain budgets – a growing 2026 threat.
Tone and bias drift monitor whether the model’s personality or alignment shifts subtly across millions of requests.

Modern platforms now unify these layers. For instance, Portkey provides observability directly within its AI Gateway, capturing over 40 data points per request across cost, performance, and accuracy – earning recognition as a Gartner Cool Vendor in LLM Observability (2025).

👀 Typical production SLOs in 2026 include:

99.99% availability.
P95 TTFT of 200–500ms.
P50 end-to-end RAG latency under 1.5 seconds.
Hallucination rates below 1%, depending on domain risk tolerance.

Controlling costs as you scale

LLM costs grow in a fundamentally different way than traditional infrastructure. Instead of paying mainly for reserved compute, spending increases with usage volume, since every request consumes tokens. What looks like a few cents per call can quickly become a major expense at millions of requests per month.

To manage this, teams must start with cost attribution – tracking spend by feature, team, and user. Without this visibility, optimization happens too late, after budgets have already been exceeded.

Once costs are visible, three levers drive meaningful savings:

Semantic caching avoids recomputing responses for repeated or similar queries. For instance, semantic cache delivers around 20% average cache hit rates in early production and up to 60% in focused RAG systems, with roughly 20× faster responses at near-zero cost on cached requests.
Intelligent routing sends simpler requests to cheaper, faster models while reserving premium models for complex reasoning.
Batch processing , such as OpenAI’s Batch API, reduces costs for large, non-urgent workloads by grouping requests into lower-priced bulk jobs with flexible completion windows.

If you’re looking for a way to maintain control over costs, opt for Portkey’s AI Gateway. One delivery platform saved $500,000 through optimized routing and caching via Portkey while maintaining 99.99% uptime across billions of requests.

Building your deployment pipeline

As you’ve seen, LLM deployment is an operational pipeline made up of infrastructure, serving, scaling, monitoring, and cost control, with each layer compounding the reliability and efficiency of the next.

If your models are already running but the operational layer is holding you back, this is where an AI gateway like Portkey fits.

Portkey sits above your inference stack, providing intelligent routing across providers, built-in observability, semantic caching, cost controls, and automatic failover, so you can scale reliably without building custom infrastructure.

Ready to move from “model deployed” to truly production-ready? Try Portkey’s AI gateway now and turn your LLMs into reliable, cost-efficient production systems!

Full Width CTA<br>
* {<br>
margin: 0;<br>
padding: 0;<br>
box-sizing: border-box;<br>
}</p>
<div class="highlight"><pre class="highlight plaintext"><code> body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, sans-serif;
line-height: 1.6;
color: #333;
background: #f5f5f5;
padding: 40px 20px;
display: flex;
align-items: center;
justify-content: center;
min-height: 100vh;
}

.container {
    max-width: 1200px;
    width: 100%;
}

/* Full Width CTA */
.cta-full {
    text-align: center;
    padding: 60px 40px;
    background: #000;
    color: white;
    border-radius: 12px;
}

.cta-full h2 {
    font-size: 36px;
    margin-bottom: 16px;
}

.cta-full p {
    font-size: 18px;
    margin-bottom: 32px;
    opacity: 0.9;
}

/* Button Styles */
.btn-primary {
    display: inline-block;
    background: white;
    color: #000;
    padding: 14px 32px;
    border-radius: 6px;
    text-decoration: none;
    font-weight: 500;
    transition: all 0.2s ease;
    border: none;
    cursor: pointer;
    font-size: 16px;
}

.btn-primary:hover {
    background: #f0f0f0;
    transform: translateY(-1px);
    box-shadow: 0 4px 12px rgba(255,255,255,0.15);
}

.btn-secondary {
    display: inline-block;
    background: transparent;
    color: white;
    padding: 14px 32px;
    border-radius: 6px;
    text-decoration: none;
    font-weight: 500;
    transition: all 0.2s ease;
    border: 2px solid white;
    cursor: pointer;
    font-size: 16px;
}

.btn-secondary:hover {
    background: white;
    color: #000;
}

.btn-group {
    display: flex;
    gap: 16px;
    flex-wrap: wrap;
    justify-content: center;
}

@media (max-width: 768px) {
    .cta-full h2 {
        font-size: 28px;
    }

    .cta-full p {
        font-size: 16px;
    }

    .btn-group {
        flex-direction: column;
        width: 100%;
        max-width: 300px;
        margin: 0 auto;
    }

    .btn-primary,
    .btn-secondary {
        width: 100%;
        text-align: center;
    }
}

</style>
</code></pre></div><h2>
<a name="ship-faster-with-ai" href="#ship-faster-with-ai" class="anchor">
</a>
Ship Faster with AI
</h2>

<p>Everything you need to build, deploy, and scale AI applications</p>

<p><a href="#">Get Started</a><a href="#">Book a Demo</a></p>

The best approach to compare LLM outputs

Drishti Shah — Tue, 24 Feb 2026 13:45:39 +0000

Once LLMs are in production, output quality stops being a subjective question and becomes an operational one. Teams are no longer asking whether they need to evaluate outputs, but how to do it reliably as systems evolve.

Production systems can change frequently. Prompts are iterated on, models are swapped, routing logic is adjusted, and traffic patterns shift. In this environment, point-in-time judgments of quality are not useful. What matters is whether output quality is stable, improving, or degrading across these changes.

A repeatable approach to measurable output quality gives teams a way to:

Establish baselines that survive prompt and model updates
Compare outputs across versions, providers, and configurations
Detect regressions before they reach users
Reason about trade-offs between quality, latency, and cost

Is manual review and one-off prompting enough to compare LLM outputs?

Manual review and ad-hoc prompting still play a role in mature LLM systems, but they stop being sufficient once scale and change are introduced.

Spot-checking outputs is inherently narrow. Reviewers see a small slice of traffic, often curated or synthetic, and their judgments are shaped by context that may not exist in real usage. As prompts and models evolve, those reviews quickly become outdated. What was “approved” last week may no longer reflect what users are seeing today.

One-off prompting has similar limitations. Testing a handful of inputs against a new model or prompt can reveal obvious failures, but it does not capture variance. Non-determinism means two runs of the same prompt can produce meaningfully different outputs. Without repetition and aggregation, it’s impossible to tell whether a change actually improved quality or simply produced a better-looking example.

At scale, manual review also introduces inconsistency. Different reviewers apply different standards, and those standards shift over time. There is no durable baseline, no reliable way to compare runs across days or deployments, and no systematic way to surface regressions early.

Manual review remains useful for:

Calibrating evaluation criteria
Investigating edge cases
Providing high-quality labels for difficult tasks

But as the primary mechanism for comparing LLM outputs, it breaks down. Production systems need methods that are repeatable, aggregatable, and resilient to change.

Metrics to consider when comparing LLM outputs

The core dimensions:

Effective LLM evaluation requires metrics that map to the actual risks and goals of your application. Most production systems care about three primary dimensions:

_1. whether the output is factually grounded (hallucination detection),

whether it addresses what the user actually asked (relevance and correctness), and
whether it meets safety and quality standards (toxicity, coherence, formatting)._

These dimensions are not interchangeable. A response can be highly relevant but factually wrong, or factually correct but unhelpful. Choosing which metrics to track depends on what failure modes matter most for your use case.

Deterministic vs. LLM-based metrics:

Metrics fall into two categories: deterministic and model-based.

Deterministic metrics like regex matching, JSON validation, and keyword presence are fast, cheap, and predictable. They work well for structured outputs or hard constraints.

Model-based metrics use an LLM to judge another LLM's output, which is necessary for evaluating qualities like coherence, completeness, or whether an answer is actually helpful. LLM-as-a-judge approaches are more expensive and introduce their own variance, but they scale where human review cannot. In practice, most teams use both: deterministic checks for objective criteria, model-based evaluation for subjective quality.

Choosing the right granularity:

Metrics also differ in scope.

Span-level evaluation scores individual LLM calls, useful for isolating which step in a pipeline is underperforming.
Trace-level evaluation looks at the full chain of reasoning, which matters when correctness depends on multiple steps working together.
Session-level evaluation captures user experience across a conversation, surfacing issues like frustration or confusion that only emerge over time.

Comparing outputs effectively means instrumenting at the right level and aggregating results in ways that surface actionable patterns, not just individual failures.

How Arize approaches evals

The evaluation loop:

Arize treats evaluation as an operational loop, not a one-time gate. Evaluations attach directly to traces, so every LLM call can be scored and explained in context. This means teams can run evaluations offline during development, testing prompt changes against datasets before deployment, and then transition the same evaluators to run online against production traffic. The result is a continuous feedback mechanism: evaluations surface regressions, traces provide the context to diagnose them, and experiments validate fixes before they ship.

Pre-built and custom evaluators:

The platform includes pre-built LLM-as-a-judge evaluators for common concerns: hallucination, relevance, toxicity, summarization quality, code correctness, and user frustration, among others. These templates are benchmarked against golden datasets with known precision and recall, so teams can deploy them with confidence. For domain-specific needs, custom evaluators can be defined in plain language, describing what "good" looks like in the same way you would brief a human reviewer, or implemented as deterministic code for objective criteria. Evaluators are versioned and reusable across projects, which keeps evaluation criteria consistent as systems evolve.

Explainability and action:

Every evaluation produces not just a label or score, but an explanation. This matters because knowing that an output was marked "hallucinated" is less useful than understanding why: which claim was unsupported, which context was missing. Explanations make evaluations actionable: they point teams toward specific retrieval failures, prompt gaps, or model limitations. Combined with trace-level observability, this turns evaluation from a reporting mechanism into a diagnostic tool that accelerates iteration.

Completing the loop with Portkey: routing and orchestration for evaluations

Running meaningful LLM evaluations requires more than scoring outputs in isolation. Teams need a reliable way to orchestrate how different models, prompts, and configurations are exercised so that comparisons are intentional and repeatable.

By acting as a routing layer for LLM traffic, Portkey's AI Gatewaymakes this orchestration straightforward. Multiple models and providers can be invoked through a single, consistent API, allowing teams to evaluate alternatives side by side without changing application code. The same inputs can be routed to different models, prompt versions, or configurations as part of structured evaluation runs.

This becomes especially useful when evaluations move closer to production. Instead of maintaining separate pipelines for testing and live traffic, teams can reuse the same routing logic to:

Compare models under identical conditions
Test prompt changes against real inputs
Run controlled experiments alongside production traffic

Observability ensures that these evaluations remain interpretable. Every request captures the context needed to explain outcomes: which model was used, how it was routed, which prompt version was applied, and how the system performed in terms of latency and cost.

Together, routing and observability turn evaluations into an operational loop. Results from Arize evaluations can be traced back to the exact model and routing decisions that produced them, making it easier to act on those insights and iterate with confidence.

To try it out, get started with Portkey for free (Portkey is open source!)

Book a demo with our experts if you're looking to deploy AI Agents in production!

Open AI Responses API vs. Chat Completions vs. Anthropic Messages API

Drishti Shah — Mon, 23 Feb 2026 14:27:32 +0000

The LLM API landscape has never been more fragmented, or more consequential. As teams move from prototypes to production, the choice of which API format to build on shapes your vendor flexibility, your codebase complexity, and how quickly you can swap models when something better comes along.

Today, three API formats dominate how AI Agents talk to LLMs:

OpenAI's Chat Completions API — the de facto standard, universally supported
OpenAI's Responses API — the newer, agent-oriented evolution with built-in tools and state management
Anthropic's Messages API — Claude's native interface, with capabilities like extended thinking and prompt caching

Each was designed with different goals in mind. Understanding the differences affects how you build, how you scale, and how locked in you are to a single provider.

Portkey supports all three natively, and that's where standardization starts to matter.

Open AI Responses API vs. Chat Completions vs. Messages API: At a glance

	Chat Completions	Responses API	Messages API
Provider	OpenAI	OpenAI	Anthropic
Endpoint	`POST /v1/chat/completions`	`POST /v1/responses`	`POST /v1/messages`
Design goal	Stateless text generation	Agentic workflows with built-in tools	Claude-native capabilities
State management	Manual	Optional server-side (with `store: true`)	Manual
Streaming	✅	✅	✅
Tool / function calling	✅	✅ (with built-in tools)	✅
Built-in web search	❌	✅	✅ (via server tools)
Extended thinking	❌	❌	✅ (Claude only)
Prompt caching	❌	❌	✅ (with `cache_control`)
Computer use	❌	✅	✅
Ecosystem compatibility	Widest	Growing	Claude-specific

What makes each endpoint different

Chat Completions: the universal standard

Chat Completions (POST /v1/chat/completions) is where everything started. You send an array of messages, each with a role (system, user, assistant, or tool for tool call results) and the model replies. It's stateless by design, so you own the conversation history and pass it with every request.

This simplicity is its biggest strength. Because the model has no memory between calls, you have full control over what context it sees. And because practically every major provider has adopted this format, code written against Chat Completions works across OpenAI, Anthropic (via adapters), Gemini, Mistral, Bedrock, and other models with minimal changes.

What it does well:

Widest ecosystem of tools, frameworks, and libraries
Predictable, well-understood response format
Easiest path to switching providers or running multi-provider setups

What it doesn't do:

No built-in tools. Web search, code execution, file search all need external orchestration
No native support for extended reasoning or prompt caching
No server-side state, you manage conversation history entirely

The response object returns choices, where each choice contains a message with role: "assistant" and content. Tool calls come back in tool_calls. Clean and predictable, which is why it became the lingua franca of LLM APIs.

When to use it: When your use case is primarily text generation i.e., chatbots, summarization, classification, content generation, Q&A. It's the right default if you're using frameworks like LangChain or LlamaIndex that abstract over providers, or if cross-provider portability matters.

Responses API: built for agents

The Responses API (POST /v1/responses) takes a different approach. It's designed to run agentic loops as the model can call multiple built-in tools (web search, file search, code interpreter, computer use, remote MCP servers) within a single API request, without you orchestrating each step.

State management comes in two forms. With previous_response_id, you chain responses by referencing a prior response ID and the model picks up context without you resending the full history, but you're still tracking the ID yourself. The newer Conversations API goes further, maintaining a durable conversation object server-side that automatically accumulates turns across sessions.

What it does well:

Built-in tools that run within a single request, no external orchestration needed
previous_response_id chains turns without resending prior tokens
Designed for multi-step agentic workflows where context and tool results accumulate
Better cache utilization compared to Chat Completions for repeated context

What it doesn't do:

Natively available on OpenAI models only (though Portkey makes it work across providers)
More complex response structure, output is an array of typed items rather than a single message
Overkill for simple single-turn completions

When to use it: When you're building autonomous agents that use built-in tools, or multi-turn workflows where you want to reduce token overhead across turns. It's the right choice when agentic behavior and tool use are core to your application.

Messages API: Claude's native interface

Anthropic's Messages API (POST /v1/messages) is designed around how Claude works. While it shares surface similarities with Chat Completions, it exposes capabilities that are specific to Claude and don't exist in OpenAI's formats.

What it does well:

Extended thinking: Claude returns type: "thinking" content blocks before the final answer, exposing its reasoning process.
Prompt caching: Fine-grained cache_control lets you cache specific content blocks (with 5-minute or 1-hour TTLs), reducing latency and cost significantly for repeated context
Rich content blocks: The content array supports text, images, PDFs, tool use, thinking blocks, and citations pointing to source documents
Stop reason granularity: stop_reason can be end_turn, max_tokens, stop_sequence, tool_use, pause_turn, or refusal
Native web search: Pass {"type": "web_search_20250305", "name": "web_search"} in the tools array and Claude handles execution server-side, returning results in a server_tool_use response block

What it doesn't do:

No server-side state management (you manage history yourself)
Not natively compatible with non-Anthropic providers without a translation layer

The response object returns a content array of typed blocks. A single response might include a thinking block, a text block, and a tool_use block in sequence. Citations on text blocks tell you exactly which document or character range the model drew from.

When to use it: When you're building specifically on Claude and need extended thinking for complex reasoning, prompt caching for document-heavy workloads, or reasoning transparency in your application.

How Portkey supports all three

Most teams end up needing more than one of these formats. You might use Chat Completions for a general assistant, the Responses API for an autonomous agent, and Claude's Messages API for a document reasoning pipeline, all in the same AI Agent.

Building direct integrations with each provider means separate SDKs, separate observability, and code that breaks every time you want to try a new model.

Portkey sits between your application and every provider, handling the translation so you don't have to.

The best part: you can use any of the three API formats with any provider and model.

Want to use the Messages API format but route to a Gemini model? Portkey handles the transformation. Want Chat Completions format but call a Claude model? Same thing.

Beyond format flexibility, Portkey adds what direct API access can't give you:

Observability: Every request logged, traced, and searchable across all providers and API formats
Fallbacks and load balancing: Route to a backup provider if your primary is down or rate-limited
Prompt management: Version, test, and deploy prompts centrally
Cost tracking: Unified spend view across providers and models
Governance: Enterprise controls over which teams access which models

Making calls with each API through Portkey

Chat Completions

from portkey_ai import Portkey

portkey = Portkey(api_key="PORTKEY_API_KEY")

response = portkey.chat.completions.create(
    model="@openai-provider/gpt-5.2",
    messages=[{"role": "user", "content": "Explain quantum computing in simple terms"}]
)

print(response.choices[0].message.content)

Responses API

from portkey_ai import Portkey

portkey = Portkey(api_key="PORTKEY_API_KEY")

response = portkey.responses.create(
    model="@openai-provider/gpt-4o",
    input="Explain quantum computing in simple terms"
)

print(response.output_text)

Messages API

import anthropic

client = anthropic.Anthropic(
    api_key="PORTKEY_API_KEY",
    base_url="https://api.portkey.ai"
)

message = client.messages.create(
    model="@anthropic-provider/claude-sonnet-4-5-20250514",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Explain quantum computing in simple terms"}]
)

print(message.content[0].text)

Switching providers without changing your code

The real payoff is when you want to swap providers.

To move from OpenAI to Claude on Responses, all you need to do is change the provider name and model:

from portkey_ai import Portkey

portkey = Portkey(api_key="PORTKEY_API_KEY")

response = portkey.responses.create(
    model="@anthropic-provider/claude-sonnet-4-5-20250514",
    input="Explain quantum computing in simple terms"
)

print(response.output_text)

Your application code, your observability, your fallback logic, none of it changes. The format stays the same and Portkey's universal API handles the translation to whichever provider you're routing to.

Getting started

All three API formats work through Portkey with a single configuration change, pointing your SDK's base URL at Portkey's gateway. From there, routing, translation, observability, and reliability are handled for you.

Get started with Portkey | Read the docs

To see how explore how Portkey can support your AI strategy, book a demo here.

Introducing the MCP Gateway!

Drishti Shah — Wed, 21 Jan 2026 15:14:37 +0000

Your interns need three approvals to touch production. Your AI agents? Zero.

With MCP, agents can take real action – ✅ connect to databases, ✅ trigger workflows, ✅ access internal systems. The protocol just works.

What's missing is the operational layer that lets platform teams say yes without losing control.

Why Everyone’s Obsessed With MCP Right Now

byu/According-Site9848 inautomation

Over the last year, that early experimentation has turned into meaningful adoption.

MCP servers are now a core part of how modern agents interact with tools and data sources. The number of publicly available and deployable MCP servers has grown. So has the adoption.

More importantly, this adoption is happening in enterprise-aligned scenarios where teams are connecting dozens or hundreds of MCP servers in real deployments.

The pattern is clear across early adopters: internal MCP servers exposing company systems, third-party vendor servers, and open-source tools, all flowing through the same agent workflows.

Why operating MCP at scale introduces new requirements

A typical MCP setup today includes:

Third-party MCP servers maintained outside the organization
Internal MCP servers owned by platform or infrastructure teams
Agents consuming both as part of a single workflow

Using multiple types of MCP servers changes the operational equation.

Authentication stops being uniform. Third-party servers use OAuth. Internal servers integrate with Okta or Entra ID. Agents now need credentials for both and the list keeps growing.

Access control gets complex fast. Some tools should be broadly available. Others need tight scoping by team, agent, or environment. Managing this at the agent or server level doesn't scale.

Visibility fragments by default. Usage data lives with each MCP server. Understanding which tools are actually being used, by which agents, and with what impact requires stitching together multiple sources.

Operational changes become risky. Rotating credentials, updating access, or enforcing new policies means touching individual servers or redeploying agents.

How Portkey's MCP Gateway works

The gateway sits between agents and MCP servers. It moves operational concerns out of individual servers and agent runtimes, and into a shared control plane managed by platform teams.

Unified authentication across MCP servers

Yes.

In ad-hoc setups, each MCP server handles auth independently. Agents juggle credentials. Platform teams can't rotate keys without touching every deployment.

The gateway centralizes authentication. Servers are authenticated once. The gateway handles downstream auth with each MCP server say OAuth flows, API keys, custom headers, whatever that server requires.

For internal MCP servers: Connect without distributing credentials. Access is managed at the organization, workspace, or team level.

For enterprise identity: Integrate directly with Okta, Entra ID, or any OIDC/OAuth-compliant provider. Users authenticate with existing enterprise credentials.

For third-party servers: OAuth flows and API keys are managed at the gateway level, not scattered across agent configs.

For existing APIs: The gateway can auto-generate MCP tools from OpenAPI specs, bringing them into the same auth model without building custom servers.

Fine-grained authorization and access control

Authentication gets agents in the door. Authorization determines what they can actually do.

The gateway enforces fine-grained access control across:

Organizations, workspaces, and teams
Individual users or agents
Specific MCP servers
Individual tools within those servers

Internal systems stay tightly scoped. Common utilities are available broadly. Permissions are policy decisions managed at the gateway—not implementation details buried in agent code.

Change access controls without redeploying agents or touching MCP servers.

Centralized MCP server and tool registry

Once authentication is centralized, access needs to be explicit.

Portkey’s MCP Gateway introduces a centralized registry that makes MCP servers and tools explicit, discoverable, and manageable. Every MCP server is registered once and becomes part of a controlled inventory rather than a hidden dependency embedded in agent runtimes.

The registry provides a clear view of:

Internal, external, and third-party MCP servers
The tools each server exposes
Ownership and intended scope
Usage across agents and teams

Platform teams can see which tools are being used, by which agents, how frequently, and in what context. This makes it possible to identify critical dependencies, spot unused or risky tools, and understand how MCP capabilities are actually being consumed.

MCP server usage data

By attaching usage data directly to MCP servers and tools, the registry supports informed decisions around access, policy, and lifecycle management.

Policy enforcement at the access layer

The gateway applies policies at the access layer, before requests reach LLMs or MCP servers:

Centralized guardrails: Enforce PII redaction, content filtering, and compliance policies once. They apply uniformly across every agent and workflow.

Block risky tool calls: Define which tools can take which actions. Prevent unauthorized invocations before they execute.

Consistent policies across LLMs and MCP: Use a single control plane to govern both model interactions and tool usage.

End-to-end observability for MCP usage

Portkey’s MCP Gateway provides end-to-end observability across the full agent execution path.

Every MCP interaction is captured in context:

Which agent made the request
Which MCP server and tool were invoked
What parameters were passed
How the call behaved and whether it succeeded or failed

This MCP-level visibility is correlated with LLM activity, giving platform teams a single, coherent view of how agents reason, act, and interact with tools. When something goes wrong, it is possible to trace failures back to the exact tool call or policy decision that caused them.

Combined with the MCP registry and policy enforcement, this visibility allows teams to operate MCP with the same rigor as any other production system.

What this enables for platform teams

With the gateway in place, MCP becomes infrastructure you can operate deliberately:

Safe sharing of MCP servers and tools across teams
Clear ownership and lifecycle management for MCP capabilities
Centralized enforcement of security, compliance, and usage policies
Faster iteration without embedding control logic into agents or servers

Most importantly: MCP can move from isolated usage to organization-wide adoption without increasing operational risk.

Built on production infrastructure

The gateway runs on the same infrastructure that processes 640 billion tokens monthly for Fortune 50 companies including DoorDash and Roche.

It integrates directly with Portkey's AI Gateway. Manage models, MCP servers, tools, policies, and observability through one control plane. Use them together or independently.

Also, the gateway is fully open source.

Inspect the code. Run it in your environment. Extend it for your use case. No opaque infrastructure in critical paths. No vendor lock-in.

This is infrastructure you can trust because you can verify it.

Speed up your MCP adoption today

MCP has changed how agents access tools. What’s changing now is how teams operate MCP once it becomes shared infrastructure.

As adoption accelerates, the challenge is no longer protocol support. It’s access control, policy enforcement, visibility, and security. These are platform problems, and they require platform-level solutions.

Portkey’s MCP Gateway is open source and ready to run.

Explore the repository, review the architecture, and try it with your existing MCP servers and agents.

If you’re operating MCP today or planning to roll it out across your organization, get started or book a demo with us!

MCP tool discovery for autonomous LLM agents

Drishti Shah — Fri, 26 Dec 2025 12:51:38 +0000

Large language model agents are increasingly expected to work with real systems like files, APIs, databases, execution environments through tools. As the number of available tools grows, however, most agent architectures struggle to scale in a clean, autonomous way.

The paper “MCP-Zero: Active Tool Discovery for Autonomous LLM Agents” introduces a different approach to tool use: instead of giving models a fixed set of tools up front, it lets agents actively request the tools they need, when they need them. (Source).

At a high level, MCP-Zero reframes tool usage as a capability discovery problem rather than a retrieval or prompt-injection problem.

Why current tool selection approaches break down

Today, most LLM agents use one of these two patterns for tool access:

System-prompt injection , where all available tool schemas are included in the prompt
Retrieval-based selection , where tools are selected once based on the user’s initial query

But these have limitations at scale.

Injecting full tool schemas = extreme context overhead. Even a single MCP server can require thousands of tokens just to describe its tools, and large MCP ecosystems can exceed 200k tokens in total. This forces models into a passive role, scanning oversized prompts instead of reasoning about the task.

Retrieval-based approaches reduce prompt size, but they still assume that tool needs are static. Many agent tasks are multi-step. So, new requirements emerge as the agent reads files, inspects outputs, or encounters errors. Selecting tools once, just based on the initial user query, often results in missing or incorrect tools later in the task.

The key issue is autonomy: in both cases, the agent does not control how it acquires new capabilities.

How's MCP-Zero different?

MCP-Zero starts from a simple shift in responsibility: the agent itself decides when it needs a tool.

Instead of being handed a predefined list of tools, the model is prompted to explicitly declare missing capabilities as they arise during task execution. When the agent realizes it cannot proceed with its own knowledge, it gives a structured tool request that describes:

the server domain (platform or permission scope)
the tool operation it needs to perform

This request is generated by the model, not inferred from the user query. That distinction matters. Model-generated requests tend to align much more closely with tool documentation than raw user input, which improves matching accuracy downstream.

Importantly, the agent can make multiple tool requests over the course of a single task , rather than being limited to a single selection step at the beginning.

Once the agent gives a tool request, MCP-Zero uses a hierarchical semantic routing process to locate relevant tools efficiently.

This happens in two stages:

Server-level filtering The system first matches the requested server domain against MCP server descriptions (and enriched summaries) to narrow the search space.
Tool-level ranking Within the selected servers, individual tools are ranked based on semantic similarity between the requested operation and the tool descriptions.

By separating server selection from tool selection, MCP-Zero avoids scanning thousands of tools at once while preserving precision.

MCP-Zero’s iterative active invocation (Source)

A key difference between MCP-Zero and prior work is that tool discovery is iterative.

After receiving a tool, the agent evaluates whether it is sufficient for the current subtask. If it is not, the agent can refine its request and trigger another retrieval cycle. This creates a natural feedback loop:

analyze the task
request a tool
use the tool
reassess what’s missing

This is especially important for multi-step workflows such as debugging code, where tools from different domains (filesystem, code editing, execution) are needed in sequence.

What this means for MCP-based agent systems

As the MCP ecosystem grows with hundreds of servers & thousands of tools, the cost of naïvely exposing everything to an agent becomes unsustainable. Prompt injection does not scale, and retrieval-only approaches still treat agents as passive consumers of tools.

MCP-Zero shows a different path: treat tool discovery as an agent capability, not an external preprocessing step.

MCP-Zero reframes tool usage as an active, agent-driven process rather than a static configuration problem. By allowing agents to request tools on demand, route them efficiently, and refine their capabilities over time, it addresses both the scalability and autonomy limits of existing approaches.

For MCP-based agent systems, this points toward a more sustainable design pattern, where agents stay lightweight, adaptive, and in control of how they extend their own capabilities.

Source: arXiv:2506.01056 [cs.AI]

Enterprise MCP access control: managing tools, servers, and agents

Drishti Shah — Tue, 23 Dec 2025 10:31:32 +0000

MCP has moved fast.

Source

What started as a convenient way to plug tools into an AI workflow is now being used as shared infrastructure across teams.

What began as a way to expose tools to a single agent is now being used as shared infrastructure across teams and applications. It’s common to see multiple MCP servers running in parallel: some backing internal tools, others connecting to external services, and some dedicated to specific environments like development or production.

As MCP adoption grows, teams also start centralizing discovery. Instead of hardcoding MCP endpoints, they publish servers and tools through an MCP registry so agents can dynamically discover what’s available. This makes MCP easier to adopt and scale, but it also changes how access is managed.

At this stage, MCP starts to resemble other pieces of shared infrastructure. Multiple clients, agents, and workloads rely on the same servers and tools, often across organizational boundaries.

This shift sets the stage for why MCP access control becomes necessary as it moves from experimentation to sustained use.

Why MCP access control matters now

Teams are standing up MCP servers for internal services. Tools are getting published through registries so agents can discover them dynamically. Multiple users, applications, and environments begin connecting to the same MCP endpoints.

In early setups, this risk is easy to ignore. Once MCP servers are shared and discoverable, access is no longer implicit. Connecting to an MCP server means gaining visibility into the tools it exposes, and tools represent real capabilities: reading data, modifying systems, or triggering downstream actions.

Without access control, any connected client or agent can see and attempt to use every exposed tool. In practice, this leads to over-permissioned agents, unclear ownership, and difficulty reasoning about who can do what.

In production environments, the consequences are more serious: accidental writes, unintended data access, or misuse of high-cost tools.

This is the point where MCP access control stops being optional. If MCP servers are treated as shared infrastructure, they need the same kind of access boundaries that other internal platforms rely on.

What MCP access control actually means

MCP access control is often misunderstood as a simple authentication problem. In practice, authentication only answers one question: who is connecting. Access control answers the harder one: _ what that client or agent is allowed to do once connected._

At first glance, MCP access control can look like an authentication problem. You might assume that once a client or agent is identified, access is implicitly safe. In reality, authentication is only the starting point.

Authentication identifies who is connecting to an MCP server.

Authorization defines which MCP servers and tools a client or an agent is allowed to access.

MCP authorization is the harder problem. A single MCP server can expose multiple tools with very different risk profiles. Some tools are read-only, others modify state or trigger workflows. Treating all authenticated clients as equally trusted ignores these differences.

MCP server access control needs to operate beyond simple allow-or-deny at the connection level. It must define which tools are visible, which actions are permitted, and which are explicitly forbidden, often based on context like environment or workload.

When handled correctly, access control becomes an explicit policy layer rather than scattered logic inside individual servers. That clarity is what makes enterprise MCP access control possible as usage continues to grow.

How AI agents change the access control model

AI agents interact with MCP very differently from traditional API clients. Instead of calling a single known endpoint with a fixed purpose, agents explore. They discover available tools, reason about which ones might help, and invoke them in sequence based on intermediate outcomes.

This behavior changes how access control needs to be designed. An agent may enumerate all exposed tools on an MCP server, attempt multiple variations of an action, or chain tools together in ways that weren’t explicitly anticipated.

In MCP environments without strong access boundaries, agents tend to inherit more capability than they need. A tool exposed for internal maintenance might be visible to a customer-facing agent. Over time, access becomes implicit and difficult to reason about.

This is why MCP access control for agents needs to be explicit and restrictive by default. Authorization decisions must account for the agent’s role, environment, and workload context, not just its identity.

What good MCP access control looks like

Policy-driven authorization Access decisions are defined declaratively, not hardcoded inside MCP servers. Policies clearly specify who can connect, which tools are visible, and which actions are allowed or denied.
Clear separation of server-level and tool-level access Server access determines who can connect at all, while tool-level permissions restrict what can actually be invoked.
Least-privilege by default Clients and agents only see the tools they are meant to use, reducing overexposure as MCP environments grow.
Explicit deny boundaries Sensitive, destructive, or high-cost tools are intentionally restricted, even as new servers or agents are added.
Centralized and auditable rules Access intent is easy to review and reason about, without tracing through server code or scattered configurations.
Extensible as MCP usage evolves New MCP servers, tools, and environments inherit consistent access behavior without reimplementing permission logic.

Making MCP access control practical at scale

As MCP becomes a shared layer across agents, tools, and teams, access control can’t live inside individual servers or evolve informally. It needs to be enforced consistently, audited easily, and extended as MCP usage grows.

This is where a centralized control plane matters. Portkey’s MCP gateway brings MCP servers under the same governance layer teams already use for AI model access. Instead of managing access rules separately for models and tools, platform teams can apply unified policies across both, with clear boundaries, environment awareness, and auditability built in.

By treating MCP access control as part of your broader AI infrastructure, you avoid fragmented permissions, reduce operational overhead, and make MCP safe to run in production environments.

If you’re starting to use MCP beyond local or experimental setups, this is the right point to introduce structured access control. To see how MCP gateway can help you govern MCP servers and tools as first-class infrastructure, book a demo with us!

What is a virtual MCP server: Need, benefits, use cases

Drishti Shah — Mon, 22 Dec 2025 13:55:09 +0000

MCP adoption inside organizations has moved quickly beyond single-developer setups.

What started as a convenient way to connect an AI client to a handful of tools is now being used across:

Multiple internal teams
Shared agent platforms
SaaS and third-party MCP servers
Central MCP registries and hubs

Each team brings its own MCP servers. Each server exposes its own set of tools. And each client or agent ends up wiring directly into multiple MCP sources.

The need for a virtual MCP server

When MCP servers multiply across teams and providers, the growing challenge is it’s composition and provisioning.

MCP registries help you find available servers. But they don’t help you answer questions like:

Which tools should this agent actually see?
How do we group tools that belong together?
How do we provision MCP access consistently across teams?

The virtual MCP server sits one layer above the MCP registry to solve this.

What is a virtual MCP server?

A virtual MCP Server lets you create a combine a set of MCP tools from multiple MCP servers.

From the client’s point of view, there is just one MCP server to connect to.

Behind the scenes, that virtual server maps to:

Multiple underlying MCP servers
A curated subset of tools from each
A predefined structure that matches how teams and agents actually work

This changes how MCP is provisioned. Rather than wiring every client to many MCP servers, platform teams can:

Create virtual MCP servers per app, agent, or workflow
Group related tools together once
Reuse those groupings across environments and teams

Common use cases for virtual MCP servers

Virtual MCP servers are typically adopted when MCP usage needs to be simplified, scoped, or standardized across teams and environments.

Task-specific tool bundles for agents Expose only the exact tools an agent needs, even when those tools come from multiple MCP servers.

Source

Simplified onboarding for internal teams Give teams a single MCP endpoint with a curated tool set instead of asking them to wire multiple MCP servers manually.
Environment-specific MCP configurations Use different virtual MCP servers for development, staging, and production without changing client or agent code.
Safer consumption of third-party MCP servers Consume external or community MCP servers through a controlled layer that limits exposed tools.
Stable MCP interfaces despite upstream changes Pin tool definitions and MCP server versions to avoid breaking agents when source MCPs evolve.

Virtual MCP servers still need governance

As virtual MCP servers become the primary way teams consume MCP, they naturally turn into a critical control point.

They sit between agents and real systems, aggregating tools from internal and third-party MCP servers and exposing them through a single interface. At this layer, questions around access, visibility, and safety start to matter more than simple connectivity.

Teams need to know which agents or applications can use a given virtual MCP, how those tools are being invoked in practice, and whether usage is behaving as expected. Without governance, virtual MCP servers risk becoming another blind spot powerful, but hard to reason about once they’re in production.

Where platforms like Portkey come in

Platforms like Portkey provide the infrastructure needed to operate virtual MCP servers safely at scale.

Portkey’s MCP Gateway acts as a control plane for MCP usage, giving teams visibility into tool calls, latency, and failures while enforcing consistent access boundaries across environments. Instead of managing MCP composition in isolation, platform teams can treat virtual MCP servers as part of their broader AI platform, alongside models, routing, and observability.

This makes it possible to roll out MCP across teams without losing control or adding operational overhead.

If you’re already working with multiple MCP servers, the next step is treating MCP as platform infrastructure.

To see how Portkey’s MCP Gateway helps teams run and govern MCP servers, book a demo with us.

Understanding MCP Authorization

Drishti Shah — Thu, 18 Dec 2025 07:35:10 +0000

MCP makes it possible for AI models and agents to interact with external tools, APIs, and data sources through a standardized interface. As MCP moves from local experimentation to shared servers, registries, and production deployments, one question becomes unavoidable: who is allowed to do what, and under which conditions?

This is where authorization comes in.

Why MCP needs authorization

MCP introduces a powerful execution layer between AI clients and real systems. Once a client is connected to an MCP server, it can discover tools, invoke actions, and access data dynamically often without hard-coded call paths or predefined workflows.

That flexibility changes the security model.

MCP turns intent into capability

In traditional applications, developers explicitly wire which APIs can be called and under what conditions. With MCP, an AI client can decide at runtime which tools to invoke based on model outputs.

Without authorization controls, this effectively means:

Every connected client can attempt to use every exposed tool
Tool access is governed by availability, not permission
Risk scales with the number of tools, not the number of users

Early MCP setups often relied on implicit trust. A local MCP server, a single user, and a small set of non-destructive tools can make broad access feel acceptable during experimentation. In these environments, the cost of a mistake is low, and security boundaries are often assumed rather than enforced.

In shared development environments, internal platforms serving multiple teams, central MCP registries, or production agents with real side effects, implicit trust no longer holds. Simply knowing that a client connected to a server is not sufficient.

At that point, systems must explicitly define which tools a client is allowed to invoke, which resources it can access, and which actions are prohibited altogether.

AI agents further amplify authorization risk because they behave very differently from traditional API consumers. Rather than following fixed call paths, agents explore their environment. They may enumerate available tools, retry failed actions with modified inputs, or chain multiple tools together in ways the developer did not anticipate.

This exploratory behavior is useful for capability discovery, but it also increases the likelihood of unintended outcomes.

Without clear authorization boundaries, that exploration can easily cross into unsafe territory. Agents may access data they were not meant to see, perform unintended writes or deletions, or overuse high-privilege or high-cost tools.

Authorization is what constrains this behavior, ensuring that even when agents explore, they do so within well-defined and enforceable limits.

How MCP authorization works

MCP authorization is built around a simple but strict principle: the server is the final authority. Clients may discover capabilities and attempt tool calls, but it is always the MCP server that decides whether a request is allowed to proceed.

Authorization is evaluated at request time, not assumed at connection time. This distinction matters because MCP clients, especially agents, can change behavior dynamically.

In practice, authorization decisions are derived from identity and context. The server receives an authenticated request, extracts information about the caller (such as user, service, workspace, or environment), and evaluates that against its authorization rules. Those rules determine which tools are visible, which resources can be accessed, and which operations are explicitly denied.

Unlike static API integrations, MCP authorization is not just about endpoint access. It often operates at a finer granularity: tool-level permissions, action-level constraints, and contextual limits based on metadata. This allows the same MCP server to safely serve multiple clients with different levels of access, without duplicating infrastructure or exposing unnecessary capabilities.

Another important aspect is that authorization in MCP is continuous. Permission is not granted once and assumed forever. Each tool invocation can be independently authorized, logged, and audited. This makes it possible to revoke access, tighten policies, or introduce new constraints without redeploying clients or retraining agents.

Taken together, MCP authorization acts as the control plane that sits between AI decision-making and real-world execution. It ensures that no matter how flexible or autonomous a client becomes, its actions remain bounded by explicit, enforceable rules.

Authorization models and patterns in MCP

MCP does not mandate a single authorization mechanism, but most real-world deployments converge on a few common patterns. These patterns are shaped by two constraints:

MCP servers need strong control over tool execution,
and clients—especially agents—must be able to operate without embedding long-lived secrets.

One widely adopted approach is token-based authorization , often aligned with OAuth 2.1 semantics for HTTP-based MCP servers. In this model, a client authenticates through an identity provider and receives a short-lived access token. That token encodes scopes or claims describing what the client is allowed to do. Each MCP request carries the token, and the server evaluates it before allowing any tool invocation.

Another common pattern is scoped capability access. Instead of granting blanket access to an MCP server, tokens or credentials are limited to a specific subset of tools or actions. For example, a client may be allowed to call read-only tools but not tools that mutate data or trigger external side effects.

In multi-tenant or platform scenarios, role-based and attribute-based authorization becomes important. Rather than authorizing individual clients one by one, servers define roles or policies tied to attributes such as workspace, environment, team, or workload type. When a request arrives, the server evaluates these attributes and applies the appropriate policy.

Some deployments also separate authentication from execution authority. A client may authenticate as a known identity, but receive a more limited execution token when interacting with specific MCP servers. This avoids passing high-privilege credentials downstream and ensures that even trusted clients operate under constrained permissions when invoking tools.

Best practices for MCP authorization

Apply least privilege by default Expose only the tools and actions a client truly needs. High-impact or side-effecting tools should always require stricter permissions.
Use short-lived, scoped tokens Avoid long-lived secrets. Expiring tokens with explicit scopes reduce blast radius and support easy revocation.
Authorize using context, not just identity Enforce policies based on workspace, environment, tenant, workload type, and request metadata—not just who the client is.
Enforce authorization on every tool call Do not rely on connect-time checks. Evaluate permissions per invocation to contain agent exploration.
Delegate access safely Mint constrained execution tokens for agents instead of passing upstream credentials through them.
Make authorization auditable Log who invoked which tool, under what policy, and why the request was allowed or denied.

Conclusion

MCP makes it easy for AI clients and agents to interact with real systems, but that flexibility comes with risk. Authorization is what turns MCP from a powerful abstraction into a production-ready interface.

Strong MCP authorization means enforcing clear, contextual permissions at the server boundary, so tools are accessible without being overexposed, and agents can operate without inheriting excessive privilege. This is especially important as MCP moves into shared platforms, registries, and production workloads.

Platforms like Portkey’s MCP Gateway build these controls in by default, combining MCP connectivity with scoped access, centralized policy enforcement, and full auditability.

The result is an MCP setup that supports experimentation and autonomy, while still meeting the security and governance expectations of real-world deployments. If you're looking to get started with MCP in your organization, connect with Portkey today!

LLM routing techniques for high-volume applications

Drishti Shah — Fri, 05 Dec 2025 12:31:01 +0000

High-volume AI applications don't depend on a single model or provider.

Routing steps in as the control layer that keeps applications stable under load. For teams serving millions of requests, this flexibility is the difference between smooth operation and unpredictable outages.

In this blog, we'll discuss different LLM routing techniques and how you can apply them to your AI workloads.

How routing differs from simple model selection

Most teams begin with static model selection: pick one model for a task, wire the endpoint, and ship. This works at small scale but breaks down quickly under real production traffic. High-volume systems need something more adaptive, and that’s where LLM routing techniques come in.

Model selection is a one-time architectural decision. Routing is continuous decision-making. Instead of binding every request to a predetermined model, you can evaluate each request against real-time signalsand choose the best model for that moment.

Static selection assumes the environment is stable. Routing assumes things will vary. And at scale, they always do. Traffic surges, providers throttle, latency drifts regionally, or a newer model becomes more cost-efficient. Routing absorbs this volatility so applications don’t have to.

Challenges in high-volume workloads that routing solves

As applications scale, the behavior of LLMs becomes increasingly uneven. Providers operate under fluctuating load, pricing structures shift, and latency patterns vary across regions and times of day. LLM routing techniques exist precisely to handle these challenges.

Unpredictable latency variance

Even the best models can swing from 200ms to 800ms. When every millisecond matters—search, chat, support automation—these spikes degrade user experience. Routing helps maintain consistency by choosing the fastest healthy endpoint in real time.

Rate limits and throttling

Providers impose hard and soft limits that are easy to hit during traffic surges. Retry storms make this worse. Effective routing distributes load across models or providers to avoid saturating any single endpoint.

Cost instability at scale

What seems affordable at 10,000 requests becomes unsustainable at 10 million. Always using frontier models creates unnecessary spend for routine or low-sensitivity tasks. Routing enables cheaper paths without compromising output quality where it matters.

Provider degradation or outages

Models occasionally slow down, return elevated error rates, or fail entirely. Without routing, these incidents become application-wide outages. With routing, systems can detect degradation and shift traffic within seconds.

High-volume environments make these challenges unavoidable but with the right LLM routing techniques, they become manageable, predictable, and often invisible to users.

LLM routing techniques

Conditional routing

Latency-based routing

Some applications need consistently fast responses—search, support chat, and consumer-facing assistants. Latency-based routing directs traffic to the model or provider that historically responds the fastest for the given region or workload.

Cost-based routing

High-volume workloads amplify every pricing difference. Instead of sending all requests to a frontier model, cost-based routing classifies traffic and steers low-sensitivity or repetitive tasks to cheaper models.

Examples:

Content rewrites → economical models
Complex planning → high-accuracy models
Internal automation → mid-tier models

This is one of the most impactful LLM routing techniques for keeping spend predictable at scale.

Region-aware routing

Routing requests to the closest deployment or provider region improves latency and reduces cross-border data movement. For regulated industries or global applications, geo-location based routing is essential for both speed and compliance.

Semantic routing

Semantic routing classifies the intent of the request and routes it to the model best suited for that task. This can be embedding-based, rule-based, or classifier-driven.

Examples:

Coding tasks → code-optimized models
Summaries → small, fast models
Reasoning tasks → frontier models

This approach dramatically improves quality-to-cost ratio.

Metadata-based routing

Traffic often varies by team, tier, or workload type. Metadata-based routing uses contextual information to determine which model a request should hit.

This allows organizations to enforce:

different cost ceilings per team
stricter models for sensitive workloads
lightweight models for experimentation

Metadata becomes a simple but powerful dimension in LLM routing techniques, especially inside large organizations.

Performance-based routing techniques

While conditional routing relies on predefined rules, performance-based routing reacts to what’s happening right now. These LLM routing techniques use real-time signals like latency, error rates, throughput, and health checks to make decisions dynamically.

Load-based routing

LLM providers often enforce strict rate limits. When traffic spikes, a single endpoint can saturate quickly, leading to throttling or elevated error rates. Load-based routing allows you to distribute requests across multiple models or providers to keep every endpoint below its capacity threshold.

This is especially important for:

consumer-scale applications with bursty traffic
agent workloads generating many parallel requests
batch processing pipelines with high concurrency

By smoothing load across the system, it prevents cascading failures and maintains consistent throughput.

Fallback routing

Providers can degrade without warning from latency spikes, error codes rise, or throughput collapses. Fallback routing ensures that when a primary model fails or slows down, traffic automatically switches to a secondary or tertiary model.

Fallback chains are critical for uptime-sensitive workloads because they remove the single point of failure inherent in relying on any one provider.

Circuit breakers

Circuit breakers temporarily “trip” a failing route to protect the system. When error rates or timeouts cross a threshold, the route is disabled for a cooldown period while traffic is diverted elsewhere.

This prevents the system from repeatedly calling a degraded endpoint, avoids retry storms, and stabilizes behavior during provider outages or maintenance windows.

Canary testing

Before adopting a new model or upgrading to a new version, canary routing sends a small percentage of traffic—1–5%—to the candidate model. Teams can observe quality, latency, and cost metrics in production before rolling out fully.

This is one of the safest LLM routing techniques for introducing new models without risking production regressions.

Observability requirements for effective routing

Routing only works as well as the signals you feed into it. High-volume systems need continuous visibility into model behavior so that routing decisions remain accurate, timely, and cost-aware. Without LLM observability, even the most advanced LLM routing techniques become guesswork.

Real-time latency and throughput metrics

Latency isn’t static. Providers fluctuate throughout the day, and different regions behave differently. Routing engines need a live view of:

p50 / p95 latency
token throughput (tokens/sec)
variance across regions and providers

Error rates and provider health

Errors tend to spike before full outages. Tracking failure patterns helps performance-based routing decide when to apply fallbacks or trip circuit breakers. Key indicators include:

timeout frequency
429 and 5xx spikes
connection or rate-limit failures

Cost and token usage visibility

Routing decisions, especially cost-based ones, depend on knowing token consumption and effective cost per request. High-volume teams need granular insight into:

token usage per model
cost per route
cost drift over time

Model comparison insights

To refine routing rules, teams need to know how models actually perform under real workloads. Observability should surface:

quality deltas across models
model regressions after version changes
output differences for similar inputs

Feedback loops

The best routing setups continuously learn. Observability closes the loop between offline evaluation and online performance, allowing teams to adjust conditions, thresholds, and fallback priorities as the system evolves.

With proper observability, LLM routing techniques become predictable, measurable, and aligned with real user experience.

How Portkey supports advanced LLM routing at scale

High-volume teams need routing that adapts instantly to shifting latency, cost, or provider performance. Portkey was built as an AI Gateway to give teams this exact control layer without building it themselves. It brings both conditional and performance-based LLM routing techniques into a single, production-grade system.

Multi-provider routing with one API

Portkey connects to 1,600+ models across leading providers. This makes routing decisions fully transparent and manageable through one unified interface.

Dynamic per-request routing

Routing rules can be applied using real-time context:

budget ceilings or cost thresholds
metadata (team, workload, risk level)
rate limits
guardrail checks

This means each request can take the most efficient path without manual orchestration.

Built-in performance protection: fallbacks, circuit breakers, and load management

Portkey continuously monitors provider health and automatically redirects traffic when a model slows, errors spike, or rate limits are reached. This protects applications from upstream volatility and prevents outages from spreading.

Budgets, guardrails, and semantic caching for cost control

High-volume workloads often face cost drift. Portkey pairs routing with budget policies, guardrails, and semantic caching so teams can keep spend predictable even as usage scales.

End-to-end observability

Latency, token usage, errors, cost, and routing decisions are all captured automatically. This visibility helps teams tune routing strategies, evaluate model performance, and understand how decisions impact reliability and cost.

If you want to adopt intelligent, multi-provider LLM routing techniques without building the infrastructure yourself, Portkey’s AI Gateway gives you a ready-to-use routing layer that scales reliably with your traffic.

If you'd like to apply these LLM routing techniques across providers, you can try it out with Portkey or book a demo with the team to learn more.

LLM access control in multi-provider environments

Drishti Shah — Wed, 03 Dec 2025 08:01:21 +0000

Most organizations have already adopted a mix of AI providers and open source models. This flexibility has become the norm, but it also brings a new governance challenge: every provider comes with different tokens, permissions, limits, and safety settings.

As adoption spreads across departments, organizations need a defined access control layer. Strong LLM access control brings order to this complexity, keeping usage safe, predictable, and aligned with institutional standards while still supporting experimentation and development.

What LLM access control means in a multi-provider world

LLM access control is the set of policies and permissions that determine who can use which models, under what conditions , and with what safeguards. In a single-provider setup, this is often limited to API keys and basic permissions. But in a multi-provider environment, access control becomes far more layered.

Different providers expose different capabilities, cost structures, context limits, and safety defaults. Teams also bring their own tools i.e., agents, notebooks, apps, and integrations, each with its own access patterns. Without a consistent control layer, every team ends up managing permissions differently, which leads to policy drift, duplicated credentials, and uneven security across the organization.

In this environment, access control must span several levels at once:

Provider-level access : deciding who can use OpenAI, Anthropic, Azure, Google, or local models.
Account and subscription access : handling enterprise subscriptions, project accounts, and shared keys.
Model-level access : allowing or blocking specific models depending on role, risk, or cost.
User and team-level permissions : mapping people and departments to the right capabilities.
Application-level enforcement : ensuring internal tools, agents, and services follow the same policies.

Together, these levels form the baseline for predictable, governed LLM usage across providers.

Roles and permissions: who can access what

Role-based access control (RBAC) is the foundation of LLM access control. It ensures that people only use the models, providers, and capabilities that match their responsibilities and that these permissions stay consistent across every AI provider and internal tool.

At the top level, organization administrators manage global access. They decide which providers are available, what enterprise accounts are connected, and which high-cost or sensitive models need restricted access.

Project or department owners handle access within their own scope. They assign roles to team members, choose which models support their work, and apply local policies that fit their department’s goals. This matters especially in research and education environments where different groups operate with different levels of risk tolerance and funding.

End-users i.e., developers, researchers, analysts, or students inherit access based on their role. Their permissions determine which models they can call, how much they can consume, and the guardrails applied to their requests.

RBAC becomes far more important in multi-provider setups. Access to GPT-4.1, Claude Opus, Gemini 1.5 Pro, or a specialized open-source model may be limited to specific teams. Some users may only work with smaller, lower-cost models. These distinctions keep usage predictable and prevent unnecessary spend or unintended exposure of sensitive data.

Identity systems like SSO, SCIM, and directory groups tie everything together. When a user joins, leaves, or changes roles, their LLM permissions follow automatically without anyone needing to manage individual API keys.

Budgets and consumption controls

LLM usage can grow faster than expected, especially when multiple teams work across different providers. LLM access control keeps that usage predictable. They define how much a department, project, or individual user can spend, and they ensure that consumption doesn’t outpace governance.

Budgets often operate at several levels. Organizations can set department-wide allocations, while project owners can apply more specific limits that match their workloads. Individual users may have personal quotas to prevent runaway scripts, aggressive experimentation, or unexpected spikes.

Administrators can also apply usage limit policies, which add flexibility beyond static budgets. These policies enforce limits based on dynamic conditions such as API keys, metadata, workspace, environment, or user role.

Multi-provider setups increase the importance of these controls. Each provider has different pricing structures, context window sizes, and token costs. A model that feels affordable for early experimentation may become expensive as workloads grow. Consumption controls help teams choose the right models for the right jobs and prevent hidden costs from accumulating.

Real-time tracking and enforcement complete the picture.

When a user or team approaches a limit, the system can automatically route to lower-cost or lower-capacity models. This ensures consistent access while keeping spending aligned with organizational expectations.

Rate limits and operational safeguards

Rate limits are another essential part of LLM access control. They determine how frequently users or applications can call a model and help keep workloads stable across different teams and providers. Without rate limits, a single script, agent loop, or high-volume application can saturate capacity, causing slowdowns or blocking access for others.

Rate limits operate at multiple layers. Providers impose their own limits per account or per model. Organizations then apply internal limits to make sure usage is distributed fairly across departments and that production workloads aren’t affected by experimentation or testing. These internal limits often vary by user role, environment, or model sensitivity.

Operational safeguards build on these limits by automating responses when the system is under load. For example, requests can be queued, throttled, or AI gateway to automatically routed to alternate providers when one endpoint is saturated.

These safeguards keep critical applications responsive and protect shared enterprise accounts from unpredictable demand.

In multi-provider environments, rate limits also help balance traffic across different models and clouds. This prevents accidental over-reliance on a single provider and ensures continuity even when a provider enforces sudden throttling or undergoes downtime.

Together, rate limits and safeguards provide strong LLM access control and the operational stability needed to support both high-volume production use cases and day-to-day exploration across an organization.

Guardrails as part of LLM access control

Guardrails extend access control beyond “who can call what” to “what is allowed within those calls.” They create an operational safety layer that shapes the inputs, outputs, and behaviors of LLMs, ensuring that usage stays within institutional policies even when users have broad access.

At the input level, guardrails can enforce data rules. Sensitive information like PII, PHI, or research data can be redacted, masked, or blocked before reaching a model. This matters in environments where students, researchers, or distributed teams work with diverse datasets, not all of which should be exposed to every provider.

Output guardrails help maintain policy alignment. They can filter or transform responses that violate content guidelines, prevent potentially harmful instructions, or restrict the model from generating certain types of information. This keeps results consistent with institutional standards and reduces manual moderation.

Guardrails also apply to capabilities , especially in agentic systems. Administrators can allow or deny access to specific tools, APIs, or file operations, and constrain what an agent is permitted to execute. This is critical when multiple teams build custom workflows, automation, or research assistants.

In multi-provider environments, guardrails ensure consistency. Different providers offer varying safety settings and defaults. A central guardrail layer enforces the same policies across all models regardless of their vendor, size, or capabilities, so that teams don’t need to manage separate rule sets.

By combining access rules, consumption controls, and guardrails, organizations can give users more flexibility without losing oversight or exposing themselves to unnecessary risk.

What good LLM access control looks like

Effective LLM access control is simple to understand but difficult to implement without a unified layer. At its core, a well-designed system includes:

Clear identities and roles

Every user is mapped through SSO, directory groups, or SCIM, and roles determine which models and providers they can use.

Consistent model and provider permissions

No shadow keys, no mismatched configurations. Access rules apply uniformly across OpenAI, Anthropic, Azure, Google, and any internal or open-source models.

Budgets and usage limits that reflect real needs

Teams have predictable consumption, with dynamic limits that adjust for roles, environments, and metadata.

Rate limits that protect shared workloads

Critical applications stay stable even during spikes, experimentation, or heavy agent activity.

Guardrails that shape allowed behavior

Both inputs and outputs follow institutional guidelines, and agent tools are restricted to what’s safe and necessary.

Real-time enforcement

Policies activate at the moment of the request, not after the fact.

Unified visibility and auditability

LLM observability allows teams can monitor usage, policy hits, and provider performance in one place, without stitching together logs from multiple clouds.

When these elements come together, organizations get a controlled, predictable, and safe AI environment without blocking innovation or slowing teams down.

How Portkey helps

Portkey's AI Gateway brings all of these controls into a single platform that works across every model and provider. You can set roles, model permissions, budgets, rate limits, and guardrails once, and Portkey enforces them consistently across OpenAI, Anthropic, Azure, Google, and on-prem models.

Policies operate in real time, so overspend, policy violations, and unsafe requests are caught before they reach a model.

Portkey also centralizes identity, usage visibility, audit logs, and provider monitoring. Teams get a governed environment they can rely on, while still having the freedom to experiment, evaluate new models, and run production workloads without managing scattered keys or inconsistent policies.

If you’d like to exercise LLM access control in your organisation, you can book a demo with our team.

Buyer’s guide to LLM observability tools

Drishti Shah — Thu, 27 Nov 2025 12:18:51 +0000

Observability for LLM systems has evolved from a debugging utility to a business function. When AI applications scale from prototypes to production, every model call represents real cost, latency, and reliability risk. Teams now need more than basic logs, they need full visibility into how models perform, drift, and behave in real-world conditions.

But with this shift comes a question every AI platform team faces sooner or later:

Should we build our own observability stack or buy a specialized tool?

The visibility gap and what LLM observability should cover

While the adoption of LLMs has experienced a significant boost, running LLM applications in production has proven to be more challenging compared to traditional ML applications. This difficulty arises from the massive model sizes, intricate architecture, and non-deterministic outputs of LLMs. Furthermore, troubleshooting issues originating in LLM applications is a time-consuming and resource-intensive task due to the black-box nature of their decision-making processes.

Regular observability tools are insufficient for modern LLM apps or agents due to their complex multi-provider and agentic flow, as well as constant hallucinations, compliance gaps, cost spikes, and changing output quality, which makes LLM observability a core business function of AI. Here we are listing down the six core pillars your observability system must be able to do:

Handle runtime metrics such as latency, success rates, error codes, and costs incurred by token usage.
Ensure quality by checking accuracy, detecting hallucinations, eval coverage, and grounding.
Maintain ongoing governance, including logs, access controls, guardrails, PII checks, and audit trails.
Track how much the provider, model, user, and workspace have spent with built-in budgets and alerts.
In case of failures, it must monitor slowdowns, retries, failovers, and behavioral changes over time.
As the AI’s outputs are influenced by prompts, data, and agent actions, they guide the AI's behavior, contextual understanding, and ability to perform tasks. Thus, LLM observability must also trace agents, data sources, and user prompts.

All of these factors make observability an important business function for AI stacks, as it provides a holistic view of IT systems and enables learning about the current state based on the environment and the data it generates.

Enterprises must decide whether to invest time making a custom observability stack or use a third-party tool that already has the necessary features. By examining both options, you can determine which one suits your needs in terms of time, money, and features. In the following sections, we will explore this. Let's get started.

The “Build” path: When teams try to create LLM observability in-house

Building LLM observability in-house enables teams to tailor monitoring to their specific needs using tools like OpenTelemetry for traces and logs, as well as Grafana, ELK, or BI platforms for dashboards.

Building your own LLM observability systems provides your team with strong control over your data and security, keeping all sensitive information within your infrastructure. This is especially important for industries with strict compliance requirements, such as healthcare, finance, or government, where maintaining privacy and meeting regulatory standards are crucial.

This also provides your team with a customised solution, enabling you to tailor features such as security rules, integrations, dashboards, and model tuning to match the exact needs of your internal systems. This flexibility ensures the system grows and adapts as your organization evolves, enabling smoother workflows and a unified internal ecosystem.

Additionally, an in-house solution lets you scale instantly without waiting for vendor approvals or external resource allocation.

The challenges of building an LLM observability In-house

Companies starting with only a few models may consider building an in-house monitoring tool as a cost-effective way to test out the value of their models.

While this approach offers flexibility and full control, it often requires significant development effort, such as logging and tracing schemas, ongoing maintenance of custom SDKs, normalizing metrics across different models, providers, and building pipelines to track lineage, drift, and guardrails.

Even with this, the dashboards require constant monitoring across all AI workflows as the complexity and quantity of models increase, and an in-house monitoring tool may no longer be sufficient with the limited coverage for specific LLM components, such as prompt evals, hallucination tracking, and cost attribution.

Most organizations spend 6–12 months just to create a basic version, and ongoing updates are also necessary to accommodate the evolving models and workloads. Hallucination detection and agent tracing also require enterprise-level AI expertise, making it tedious.

The “Buy” path: What off-the-shelf solutions offer

A ready LLM observability solution provides teams with a turnkey solution to monitor, analyze, and optimize AI models. By eliminating the need for in-house infrastructure, it enables faster deployment, seamless scalability, and comprehensive insights into model performance, reliability, and usage—allowing teams to focus on improving AI outcomes rather than managing tooling with fewer resources and lower cost.

Such platforms work across models and comes with a purpose-built telemetry for LLMs and agents. It offers ready dashboards for latency, cost, drift, and quality, along with built-in eval frameworks for hallucinations, grounding, and other LLM-specific behaviors.

These platforms integrate seamlessly with LLM gateways, routers, and model providers, enabling smarter routing decisions based on real performance data such as costs and governance. This leads to more efficient usage, better quality outputs, and a reliable LLM stack.

It can be set up in days instead of months. It evolves with automatic updates, with new models and capabilities. This requires less ongoing maintenance, versioning, and compatibility checks while you still own the entire system.

What to consider when buying an LLM observability solution

When buying an observability tool, it's important to evaluate how the solution fits your specific environment, whether you need open-source or commercial options, and your organization's future goals.

Key dimensions for evaluation

Metrics:

Data types: Does it support the essential pillars of observability (logs, metrics, and traces)?
Querying & visualization: How easy is it to query data, create meaningful dashboards, and set up alerts effectively?

Integrations:

Ecosystem compatibility: Does it offer out-of-the-box integrations for your existing infrastructure, languages, databases, and third-party services?
Open standards: Does it support open standards like OpenTelemetry for vendor-neutral data collection?

Scale:

Performance: Can the system handle your current data volume and projected future growth without performance degradation?
Data retention: Does it provide flexible data retention policies to meet both compliance requirements and performance needs?

Governance:

Data quality & control: Are there mechanisms for tagging, sampling, and controlling the quality and volume of data ingested to avoid "garbage in, garbage out"?
Access control: Does it offer robust role-based access control (RBAC) to manage who can view, edit, or delete data and configurations?

Costs:

Pricing model: Is the pricing predictable (e.g., based on data volume, hosts, or usage), and are there transparent cost-management tools?
Total cost of ownership (TCO): Beyond the sticker price, consider the operational costs of maintenance, training, and potential overages.

Privacy and Security:

Compliance: Does the solution help you meet relevant industry and regulatory compliance standards (e.g., GDPR, HIPAA, SOC 2)?
Data handling: How are sensitive data and PII handled, masked, or encrypted both in transit and at rest?

Deployment Options:

Flexibility: Does it offer options for SaaS, self-hosted, or hybrid deployment models to match your operational preferences and security requirements?
Cloud compatibility: If cloud-native, is it compatible with your specific cloud provider (e.g., AWS, Azure, GCP)?

Platforms like Portkey are adaptable and can support your future technology stack. They also provide flexible deployments, open standards support, and enterprise-grade security, making it a reliable choice for scaling AI applications.

Build or buy an LLM observability platform

Factor

Build if...

Buy if...

|
|

Control

You need full internal control or have strict privacy constraints

You prefer managed, compliant oversight

|
|

Resources

You have a strong platform/infrastructure team for handling AI operations

You want a fast setup without engineering overhead

|
|

Complexity

You only need basic latency or cost metrics

You need deep evaluations, lineage, and governance

|
|

Scale

Traffic is low and predictable

You run high-volume, multi-model workflows and multiple team setups

|
|

Future readiness

Your team can maintain and update observability and make quick iterations when needed.

You want ongoing updates and a new model, as well as existing ecosystem support.

Why Portkey is a strong option for enterprises looking for LLM observability

Recognised as Cool Vendor in LLM Observability, Portkey provides visibility above the AI gateway.

Portkey's Observability exists for teams that want enterprise-grade observability without taking on the engineering overhead of building and scaling it internally.

It provides:

Unified tracing across models, providers, and agents
Structured logs that capture prompts, completions, token counts, cost, latency, and guardrail outcomes
Built-in quality, safety, and reliability signals
Team and department-level governance with budgets, rate limits, RBAC

Teams using Portkey don’t have to stitch together custom pipelines, dashboards, and schemas or maintain compatibility across changing model APIs. Instead, they start with a production-ready observability layer that evolves with the ecosystem.

Software Engineer in the Software Industry gives Portkey 4/5 Rating in Gartner Peer Insights™ Generative AI Engineering Market. Read the full review here: https://gtnr.io/vbcumMl7N #gartnerpeerinsights

If you’re exploring LLM observability, Portkey can help you get there faster, with less operational overhead.

Want to see how this works in practice? Book a demo with our team.