Forem: Arindam Majumder

Claude’s new Advisor Strategy is pretty interesting

Arindam Majumder — Wed, 15 Apr 2026 20:07:17 +0000

A lot of people building AI agents run into the same problem sooner or later.

If you run the entire agent on a powerful model, it works well but the costs grow quickly. If you run everything on a cheaper model, the system stays fast and affordable but it sometimes makes weak decisions, especially when planning complex tasks or choosing tools.

Anthropic recently introduced something called Advisor Strategy that tries to solve this in a simple way.

Instead of using one model for everything, the agent runs on a smaller executor model like Sonnet or Haiku. That model handles the normal workflow such as calling tools, executing steps, and moving the task forward. When the agent reaches something more complex, it can consult a stronger model like Opus for guidance. The advisor reads the full context, suggests what to do next, and the executor continues the workflow.

So most of the work stays cheap and fast, but the agent can still get strong reasoning when it actually needs it. It feels a lot like how a junior engineer works most of the time but occasionally asks a senior engineer for advice.

I found this architecture interesting because it pushes agent systems toward multi-model setups instead of relying on a single model for everything, which seems like a direction many frameworks will probably move toward.

I made a short video breaking down how the advisor strategy works and how developers can implement it in their own agents

Production-Aware AI: Giving LLMs Real Debugging Context

Arindam Majumder — Thu, 09 Apr 2026 05:22:32 +0000

TL;DR

Large language models struggle with production debugging because they do not have visibility into how code actually executes at runtime.
Inputs such as logs, stack traces, and metrics provide incomplete signals, which often cause confident but incorrect conclusions about root causes.
When AI reasoning is grounded in function-level runtime data collected from production systems, debugging becomes accurate, explainable, and reliable.

Introduction

Large language models are increasingly used by developers to understand code, analyze failures, and assist during incident response. In controlled environments, they are effective at explaining logic and suggesting fixes. In production systems, however, their usefulness often drops sharply.

A recent survey of developers found that a quarter of developers spend more time debugging than writing code each week. The same survey reported that bugs and tooling failures cost teams nearly 20 working days per year in lost productivity. These numbers reflect a reality most engineering teams already experience.

Production debugging takes time because failures depend on runtime factors such as traffic patterns, concurrency, queue depth, and system state that are absent in non-production environments. Most AI systems do not observe these execution conditions. They analyze code structure and reported symptoms, rather than the runtime behavior that caused the failure.

In this article, we will discuss why production context is critical for AI debugging, what production-aware AI really means, and how runtime intelligence enables more accurate and trustworthy debugging outcomes.

Why Production Issues Cannot Be Understood from Code Alone

Code defines control flow and data handling, but production behavior is determined by runtime conditions such as traffic volume, concurrency, and system state.

In production, requests arrive concurrently and compete for shared resources. As traffic increases, queues begin to accumulate work, caches evolve, and external dependencies respond with variable latency or partial failures. Together, these factors influence execution order, timing, and resource contention in ways that are not visible when reading code or running isolated tests.

Many production failures arise only when specific runtime conditions are met. Race conditions appear under concurrent access. Performance regressions surface under sustained or uneven load. Retry mechanisms can magnify transient upstream failures into system-wide impact. In each case, the logic itself may be correct, while the observed failure is a result of how that logic behaves under real execution pressure.

This leads to a common outcome during incident response. The code appears correct because the failure is not caused by a logical error. The root cause exists in how the code executes under real production conditions, not in how it reads in isolation.

How LLMs Debug Today: Strengths and Structural Limits

Large language models assist debugging by analyzing text. They infer intent, recognize common patterns, and map symptoms to known classes of problems. This makes them effective for code review, error explanation, and reasoning about familiar failure modes.

However, their understanding is entirely constrained by the inputs they receive. Without access to runtime execution data, their conclusions are based on probability rather than evidence.

Aspect	What LLMs Do Well	Structural Limitation
Code understanding	Explain logic, control flow, and common anti patterns	Cannot observe how code executes under real load
Input analysis	Reason over logs, stack traces, and snippets	Inputs represent symptoms, not full execution context
Pattern matching	Identify known bug patterns and typical fixes	Fails when failures are novel or environment specific
Root cause analysis	Propose plausible explanations	Cannot validate causality without runtime signals
Decision making	Rank likely fixes based on training data	Relies on probabilistic inference when facts are missing

Without visibility into execution order, timing, frequency, and state, LLMs are forced to guess. The results may sound correct, but they are not grounded in how the system actually behaved.

Hallucinations Are Caused by Missing Runtime Evidence

Hallucinations in AI-assisted debugging usually appear when the system does not have enough information about what actually happened during execution. This is common in production, where AI is asked to explain failures using logs, stack traces, or small pieces of code that describe symptoms but not runtime behavior.

Recent research on AI reliability shows that incorrect answers increase when important contextual details are missing. In debugging scenarios, these details include execution order, timing, system state, and how frequently specific code paths were executed. Without this information, AI systems infer causes based on likelihood rather than evidence.

The same pattern appears in studies on AI-driven debugging and code repair. When models are given execution traces or feedback from real runs, fault localization and fix accuracy improve. When this runtime information is absent, models often produce explanations and fixes that appear reasonable but fail to address the real cause of the issue.

Prompt refinement does not address this limitation. Clearer prompts help structure responses, but they do not introduce new facts. If execution data is missing, the model still reasons without evidence about how the system behaved.

In production debugging, hallucinations are therefore expected. They occur when AI systems are asked to explain failures they cannot observe, not because the reasoning process is flawed, but because the necessary runtime evidence is absent.

The Missing Context in AI Debugging Workflows

Most AI debugging workflows rely on the same signals engineers have used for years. These signals are useful, but they describe outcomes, not execution, which creates a gap between what failed and why it failed.

What AI usually receives today

Logs: Logs capture messages emitted by code paths that were explicitly instrumented. They are selective, often incomplete, and rarely reflect execution order, frequency, or timing across concurrent requests.
Stack traces: Stack traces show where an error surfaced, not how the system reached that state. They lack information about prior execution paths, state changes, and interactions with other components.
Metrics: Metrics summarize system behavior at an aggregate level. They indicate that something is slow or failing, but they do not identify which functions caused the issue or how behavior changed over time.

What is missing

Function level execution behavior: Which functions ran, how often they executed, and how long they took under real load conditions.
Runtime performance characteristics: Execution timing, concurrency effects, retries, and resource contention that emerge only during live operation.
Connection between user impact and code: Clear linkage between affected endpoints or workflows and the exact functions responsible for the observed behavior.

When AI reasons over incomplete signals, it cannot establish causality. Proposed fixes are derived from statistical patterns rather than observed execution, which often results in changes that compile or deploy successfully but do not resolve the underlying issue. Effective debugging requires visibility into execution behavior, not only error reports or surface-level symptoms.

Defining Production-Aware AI

Consider a common production incident. An API endpoint becomes slow after a deployment. Logs show no errors. Metrics show increased latency. The code itself looks unchanged or correct. An AI system reviewing this information can suggest several possible causes, such as a database query, a cache miss, or an external dependency. Each suggestion sounds reasonable, but none is confirmed.

This is where production awareness matters. A production-aware AI does not rely only on aggregated metrics or isolated log lines. It reasons using information about how the system actually executed under real traffic. It can see which functions ran more often than before, where execution time increased, and which code paths were exercised during the slowdown.

Production-aware AI is defined by the context it uses. It grounds reasoning in runtime behavior rather than static structure. It focuses on how functions are executed, how often they ran, and how their performance changes over time, instead of relying only on what the code looks like or what developers expect it to do.

This approach changes the quality of debugging. Instead of proposing likely explanations, the AI reasons from observed execution evidence.

Why Function-Level Runtime Intelligence Changes AI Debugging

Function-level runtime intelligence gives AI direct visibility into how software behaves while it is running. This visibility changes debugging from interpreting symptoms to analyzing execution.

Instead of inferring behavior from secondary signals, AI can reason using execution facts collected in real time.

Function-level data as the missing signal: Function-level data shows which functions executed, how frequently they ran, and how long they took under real load. This information allows AI to identify abnormal behavior at the exact point where performance or correctness changed.
Linking endpoints to execution paths: Runtime intelligence connects external symptoms to internal execution. When an HTTP endpoint slows down, or a queue backs up, AI can trace the issue to the specific functions involved, rather than reasoning only at the service or request level.
Temporal awareness across deployments: By comparing runtime behavior before and after a deployment, AI can identify which functions changed execution characteristics. This makes regressions visible without relying on alerts or manual comparison.

How Hud Enables Production-Aware AI

Hud captures function-level execution behavior directly from production systems. Instead of relying on aggregated metrics, sampled traces, or predefined alert rules, it observes how individual functions execute under real traffic, including errors and performance changes.

This execution data can be consumed directly by engineers and AI systems to reason about production behavior based on observed runtime evidence.

Below are the core capabilities that allow Hud to provide production-aware runtime context for AI-assisted debugging.

Runtime code sensing at the function level: Hud acts as a runtime code sensor. You get continuous function-level execution data from production, without manual instrumentation or ongoing maintenance. This data reflects how code actually runs under real traffic.
Automatic detection of errors and slowdowns: Hud automatically detects errors and performance degradations based on changes in runtime behavior, not static rules.
Linking user impact to code: When an endpoint slows down, or a queue backs up, Hud connects that business-level symptom directly to the functions responsible. You can see which parts of the code caused the impact, not just where it surfaced.
Post-deployment behavior comparison: Hud automatically detects deployments and compares function behavior across versions. You can see what changed in production after a release and identify regressions without manual diffing.
Runtime context for AI debugging: Hud provides a full forensic runtime context that you can use inside the IDE or pass to AI agents through its MCP server. This allows AI to reason from execution evidence instead of guessing from partial signals.

Key Takeaways

Without visibility into how code actually ran in production, AI systems reason over symptoms instead of causes, which leads to incorrect or incomplete fixes. Production systems demand runtime grounded reasoning, where function-level behavior, execution timing, and real traffic conditions are first-class inputs.

When AI is given this level of visibility, hallucination decreases, and confidence aligns with correctness. Production-aware AI is therefore not an optimization, but a requirement for reliable debugging.

Hud gives you function-level runtime visibility directly from production, with no configuration and no maintenance. Explore how Hud works, read the documentation, or book a demo to see how production-aware debugging changes the way you and your AI systems understand failures.

I built a local dashboard to inspect Claude Code sessions, tokens, and costs

Arindam Majumder — Thu, 02 Apr 2026 07:58:55 +0000

I’ve been using Claude Code heavily over the last few weeks and started wondering where my tokens were actually going.

Claude stores everything locally in ~/.claude/, which is great, but the data mostly sits in JSON logs. If you want to understand session usage, token costs, tool calls, or activity patterns, you basically end up digging through raw files.

So I built a small tool called cc-lens.

I built a local dashboard to inspect Claude Code sessions, tokens, and costs
It’s a local-first dashboard that reads your Claude Code session files and turns them into something you can actually explore.

It runs entirely on your machine. It doesn't have any cloud sync, sign-ups, or telemetry.

Some things it shows:

• Usage overview: sessions, messages, tokens, estimated cost
• Per-project breakdown: see which repos are burning the most tokens
• Full session replay: inspect conversations turn-by-turn with token counts and tool calls
• Cost & cache analytics: stacked charts by model and cache usage
• Activity heatmap: GitHub-style view of when you’re using Claude the most
• Memory & plan explorer: browse/edit Claude memory files and saved plans
• Export/import: move dashboards across machines

You can run it instantly with:

npx cc-lens
(or clone the repo if you prefer).

Here's the Github Repo, if you want to try it out!

[Boost]

Arindam Majumder — Sat, 28 Mar 2026 13:37:31 +0000

Arindam Majumder for Studio1

Mar 17

Running LLM Applications Across Providers with Bifrost

#ai #llm #proxy #litellm

Comments

5 min read

Build a Semantic Movie Discovery App with Claude Code and Weaviate Agent Skills

Arindam Majumder — Fri, 27 Mar 2026 20:45:45 +0000

Introduction

Versatility in agentic coding is increasing as new tools such as Model Context Protocol (MCP) servers and Agent Skills become more common. At the same time, many developers ask the same question when building AI applications: should they use MCP servers or Agent Skills? The important thing is understanding what each approach does well and choosing the one that fits your use case.

In this post, we’ll explain what MCP servers and Agent Skills are and how they differ, including architecture diagrams and technical details. In the later sections, we’ll also walk through how to use Weaviate Agent Skills with Claude Code to build a “Semantic Movie Discovery” application with several useful features.

Let’s get started!

Understanding MCP

The Model Context Protocol (MCP) is an open standard introduced by Anthropic that enables Large Language Models (LLMs) to interact with external systems such as data sources, APIs and services. MCP provides a structured way for an AI agent to connect to compliant tools through a single interface instead of requiring custom integrations for each service.

MCP Architecture

The MCP system operates on a client–server model and consists of three main components.

Host: the application that runs the AI model and provides the environment where the agent operates.
Client: the protocol connector inside the host that handles communication between the model and MCP servers.
Server: an external service that exposes tools, resources, or prompts that the agent can access.

MCP and Agentic Coding

Before MCP, each AI tool required custom integrations for every external service it wanted to connect to. MCP simplifies this process by introducing a shared protocol that multiple agents and tools can use.

Developers can now expose capabilities through an MCP server once and allow any compatible agent to access them without building separate integrations for each system.

Understanding Agent Skills

Agent Skills, also introduced by Anthropic, provide developers with a simple way to extend AI coding agents without running MCP servers. An Agent Skill is a structured configuration file, usually written as markdown files with YAML metadata that defines capabilities, parameter schemas and natural-language instructions describing how the agent should use those capabilities.

AI tools such as Claude Code read these files at session start and load the skills directly into the agent's working context without requiring an additional runtime.

How Agent Skills Work

When Claude Code detects a skill file in the project directory (typically under .claude/skills/), it loads the manifest into the agent's context at the beginning of the session.
The skill definition describes available capabilities, how to invoke them correctly and when to prefer one approach over another. Because the instructions are written in natural language alongside parameter schemas, the agent can reason about how to use the skill.
Skills are portable across repositories. If a developer commits a skill file to a repository, any collaborator who clones the project and opens it in Claude Code automatically gains access to the same capabilities without additional setup.

MCP and Agent Skills solve different problems in agent systems. MCP provides a standardized way for AI agents to connect to external tools, APIs, databases and services through a client–server architecture with structured schemas. Agent Skills extend the agent’s capabilities through configuration files that define workflows, instructions and parameter schemas without requiring a running server.

In simple terms, MCP enables agents to access external systems, while Agent Skills define how agents perform tasks or workflows within their environment.

Weaviate Agent Skills

Weaviate has released an official set of Agent Skills designed for use with Claude Code and other compatible agent-based development environments like Cursor, Antigravity, Windsurf and more. These skills provide structured access to Weaviate vector databases, allowing agents to perform common operations such as search, querying, schema inspection, data exploration and collection management.

The repository includes ready-to-use skill definitions for tasks like semantic, hybrid and keyword search, along with natural language querying through the Query Agent. It also supports workflows such as creating collections, importing data and fetching filtered results, and cookbooks. This enables agents to interact/build with Weaviate and perform multi-step retrieval and agentic tasks more effectively.

Agent Skills and Vector Databases

AI coding agents face difficulties when working with vector databases. Vector database APIs provide extensive capabilities, including basic “key–value” retrieval, single-vector near-text searches, multimodal near-image searches, hybrid BM25-plus-vector search, generative modules and multi-tenant system support. Without structured guidance, even a capable coding agent may produce suboptimal queries: correct syntax but the wrong search strategy, missing parameters or failure to use powerful features like the Weaviate Query Agent.

Weaviate Agent Skills address this by providing correct usage patterns, parameter recommendations and decision logic, enabling coding agents to generate production-ready code from their initial attempts.

The Weaviate Agent Skills repository is organized into two main parts:

Weaviate 𝗦𝗸𝗶𝗹𝗹 (skills/weaviate): Focused scripts for tasks such as schema inspection, data ingestion and vector search. Agents use these while writing application logic or backend code.
𝗖𝗼𝗼𝗸𝗯𝗼𝗼𝗸𝘀 Skill (skills/weaviate-cookbooks): End-to-end project examples that combine tools such as FastAPI, Next.js and Weaviate to demonstrate full application workflows.

Weaviate Agent Skills work with several development environments, including Claude Code, Cursor, GitHub Copilot, VS Code and Gemini CLI. When connected to a Weaviate Cloud instance, agents can directly interact with database modules and perform search, data management and retrieval tasks.

To evaluate how effective Weaviate Agent Skills really are, let’s build a small project and see how they accelerate RAG and agentic application development with Claude Code.

Building the Semantic Movie Discovery Application

We will build a Movie Discovery App that takes a natural-language description and returns the most semantically similar movies from a Weaviate collection. In the process, we will explore Weaviate capabilities such as multimodal storage, named vector search, generative AI (RAG) and the Query Agent in action with Claude Code, showing how these Agentic tools help you build applications faster.

Prerequisites

Python 3.10 or higher
Weaviate Cloud – Create a free cluster and obtain an API key.
TMDB API key – Used to fetch movie metadata
OpenAI API key – Required for RAG features.
Access to Claude Code
Node.js 18+ and npm – Required to run the Next.js frontend

Step 1: Project Setup

Create a movie-discovery-app folder

mkdir movie-discovery-app

Create and activate a Python virtual environment in the folder

cd movie-discovery-app py -m venv venv && source venv\Scripts\activate.bat

Install Python dependencies

pip install weaviate-client==4.20.1 fastapi uvicorn[standard] openai weaviate-agents>=1.3.0 requests python-dotenv

Install Node.js dependencies for the frontend

cd frontend && npm install

Now, create a .env file at the project root. Add the following parameters to configure Weaviate Agent Skills with Claude Code, along with your OpenAI API key and TMDB API key.

WEAVIATE_URL=your-cluster-host-without-https
WEAVIATE_API_KEY=your-api-key
OPENAI_API_KEY=your-openai-key
TMDB API key=your-tmdb-api-key

After signing up for Weaviate, click the Create Cluster button to start a new cluster for your use.

Click “How to Connect” to view the required Weaviate connection parameters.

Now that everything is set up, we can connect Weaviate Cloud with Claude Code by running claude in your project terminal:

Use the following prompt in your Claude terminal:

Write and run `check_modules.py` that connects using `weaviate.connect_to_weaviate_cloud`with `skip_init_checks=True`, loads credentials from `.env` with `python-dotenv`,
and prints the full JSON list of enabled Weaviate modules.
Run it with `venv/Scripts/python check_modules.py`."

Step 2: Create A Weaviate Collection and Import Sample Movie Data

In this step, we create a Weaviate collection and import the movie dataset into Weaviate. The dataset contains movie metadata sourced from the TMDB API. Each entry includes: title, overview, release_date, poster_url, popularity, and other important movie fields. You can import a JSON or CSV dataset directly into Weaviate.

Run this prompt to retrieve the dataset from the TMDB API and save it to a file named movies.json.

Create a TMDB dataset JSON file, movies.json, that contains 100 movie metadata and poster URLs directly from the TMDB API.

Afterwards, Weaviate Import Skills creates a Weaviate collection and imports the data from movies.json into the Weaviate database. Claude code activates Weaviate to perform this action when prompted with:

Import `movie.json` into a new Weaviate collection called Movie

Then the data is imported

Step 3: Building the FastAPI Backend and Next.js Frontend with Weaviate Cookbooks

Weaviate cookbooks enable the app to use a two-layer architecture: a FastAPI backend that exposes REST endpoints and a Next.js frontend that renders the UI. The backend connects directly to Weaviate Cloud and the Weaviate Query Agent. Weaviate cookbooks also include some frontend guidelines to communicate with the Weaviate backend over HTTP.

The app is organized into two views accessed via a collapsible sidebar:

Search view: performs semantic search and RAG using Weaviate named vectors.
Chat view: handles multi-turn conversations through the Weaviate Query Agent.

Our app includes the following features:

Layer	Component	Role
Backend	backend.py (FastAPI) - REST API on port 8000/docs	Routes: GET /health, GET /search, POST /ai/explain, POST /ai/plan, POST /chat
Frontend	Next.js + TypeScript (port 3000)	Single-page app with sidebar navigation
	SearchView.tsx	Semantic search (near_text), AI explanations (single_prompt), Movie Night Planner (grouped_task)
	MovieCard.tsx	Renders base64 poster inline, watchlist add/remove button
	ChatView.tsx	Multi-turn Query AI Agent chat
	AppSidebar.tsx	Navigation (Search/Chat), Weaviate logo + feature summary, watchlist manager with ‘.txt’ export

Use the following prompts with Claude Code to generate the backend and frontend:

Backend Prompt:

/weaviate cookbooks 

Create `backend.py`: a FastAPI app with CORS enabled for localhost:3000.
Connect to Weaviate Cloud using credentials from .env with skip_init_checks=True.
The /search endpoint should return genre and vote_average alongside title, description, release_year, and poster.
Implement these routes:  

- GET  /health                  → {"status": "ok"}  
- GET  /search?q=...&limit=3    → near_text on text_vector, return title/description/release_year/poster  
- POST /ai/explain              → generate.near_text with single_prompt  
- POST /ai/plan                 → generate.near_text with grouped_task  
- POST /chat                    → QueryAgent.ask() with full message history

Frontend Prompt:

Using Weaviate cookbooks frontend reference, create a Next.js TypeScript app in the frontend/ folder.
MovieCard.tsx should display a star rating (vote_average) and genre tag beneath the movie title. 

Components needed:  

- page.tsx        — SidebarProvider layout, view state (search | chat)  
- SearchView.tsx  — search input, MovieCard grid, AI explain and plan buttons  
- MovieCard.tsx   — poster image, title, year, description, watchlist button  
- ChatView.tsx    — message bubbles, source citations, clear chat  
- AppSidebar.tsx  — navigation, Weaviate logo + feature list, watchlist + exportBackend base URL from NEXT_PUBLIC_BACKEND_HOST env var (default localhost:8000)

Run backend and frontend servers with: uvicorn backend:app --reload --port 800 and npm run dev

After this, Claude Code will automatically build the app by adding relevant files and start both servers. You can start using the application immediately.

The FastAPI backend runs at http://localhost:8000/docswhile the frontend app is available at http://localhost:3000

You can also manually start both processes in separate terminals:

# Terminal 1 — Backend 
uvicorn backend:app --reload --port 8000
# Terminal 2 — Frontend
cd frontend && npm run dev

Congratulations! You’ve completed the project without needing to do much manual configuration or coding. 🔥

Demo

So far, we have used Weaviate Agent Skills with Claude Code to build a Semantic Movie Discovery Application powered by an OpenAI API key, a TMDB API key, and Weaviate.

The Movie Discovery app we built includes the following features:

Semantic search: Describe a mood or theme and retrieve matching movies using vector-based search (near_text).
AI explanations: Generate per-movie summaries using RAG with single_prompt.
Movie Night Planner: Create a viewing order, snack pairings and a theme summary using grouped_task.
Conversational chat: Ask questions about the movie collection through a chat interface powered by the Weaviate Query Agent, with source citations.
Watchlist: Save movies during your session and export the list as a .txt file.

What’s Next?

You could add image-based search to find similar movies and better meet your movie choices. You could also include a hybrid search feature that incorporates keyword-heavy queries and image search.

You can take your app even further by getting up to speed with Weaviate’s latest releases and becoming familiar with features such as server-side batching, async replication improvements, Object TTL and many more.

To explore further, check out the latest Weaviate releases and join the discussion on the community forum.

Weaviant Agent Skills in Action

Weaviate modules were used in the application:

Text2vec-weaviate: Responsible for text embeddings.
Multi2multivec-weaviate: Responsible for embedding images.
Generative-openai: Integrates GPT directly into the query workflow.
Weaviate Skill: Creates a collection and imports data.
Weaviate Cookbooks Skill: For defining the app’s logic.
Weaviate Query Agent: A higher-level abstraction that accepts natural language queries, decides the best query method, executes queries, synthesizes results and returns answers.

Weaviate Agent Skills help in shipping faster and more accurate RAG applications. Backend development tasks such as schema inspection, data ingestion and search operations are automated and optimized. Ultimately, this helps developers save valuable development time.

Conclusion

Both MCP servers and Agent Skills provide useful patterns for building AI-powered applications. MCP servers are well-suited for exposing external tools and services through a standardized interface, while Agent Skills focus on guiding coding agents with structured workflows and best practices.

In this tutorial, we demonstrated how Weaviate Agent Skills can simplify development by helping Claude Code generate correct database queries, ingestion pipelines and search logic. By combining vector search, multimodal storage and generative capabilities, we built a semantic movie discovery application with minimal manual setup.

As agentic development environments continue to evolve, tools like MCP servers and Agent Skills will likely be used together. The key is understanding where each approach fits and selecting the one that best supports your application architecture.

Happy building.

Resources

Model Context Protocol
Weaviate Agent Skills
Claude Code
GitHub Repository for the Movie Discovery App

We Cut Our MCP Token Spend in Half. Here's the Architecture

Arindam Majumder — Wed, 25 Mar 2026 19:04:52 +0000

When we started scaling our MCP workflows, token usage was something we barely tracked. The system worked well, responses were accurate, and adding more tools felt like the right next step. Over time, the cost began rising in ways that did not align with how much the system was actually used.

At first, we assumed this was due to higher usage or more complex queries. The data showed something else. Even simple requests were using more tokens than expected. This led us to ask a basic question. What exactly are we sending to the LLM on every call?

A closer look made things clearer. The issue came from how the system was built. We handled context, tool definitions, and execution flow by adding extra tokens at every step.

This article explains how we found the root cause and redesigned the architecture to fix it. The changes cut our MCP token usage by nearly half and gave us better control over how the system behaves.

Understanding Token Usage in MCP Systems

Once we started examining token usage, a clear pattern showed up. The LLM was receiving far more context than most requests actually needed. A large part of this came from tool definitions being sent repeatedly on every call.

Each request included the full list of tools, even when only one or two were needed. On top of that, earlier outputs and intermediate results were passed back into the model. The context kept growing, even for simple queries.

The execution flow added to the problem. The LLM would choose a tool, call it, process the result, and then repeat the same cycle if another step was needed. Each step added more tokens, and the same data often appeared many times across calls.

This setup worked at a small scale. As the number of tools increased, the cost grew quickly. More tools meant more context. More steps meant repeated processing. The system was doing extra work without adding real value. At this point, the cause was clear. Token usage came from how the system handled context and execution. The design itself was driving the overhead.

Introducing Bifrost

We started looking for a way to change how the system handled tool execution. The goal was simple. Reduce the amount of context sent to the LLM and avoid repeated processing across steps.

During this process, we came across Bifrost, an open source MCP gateway. It works between the application, the model, and the tools. It brings structure for how tools are discovered and executed, so the LLM receives only what is needed on each call.

This changed how we thought about the system. Tool access became more controlled. Context stayed limited to what was required for each request. The overall flow of execution became easier to follow and reason about.

These changes directly addressed the issues we were seeing. Tool definitions were sent only when required. Repeated decision loops were reduced. The system handled execution in a more controlled and predictable way.

From here, the focus moved away from adjusting prompts and toward changing how the system runs end-to-end.

Architectural Changes with Bifrost Code Mode

The main change came from how execution was handled inside Bifrost. Code Mode is a Bifrost feature that changes how the LLM interacts with MCP tools. Earlier, the LLM handled both planning and step-by-step tool interaction. Each step required another call, and each call carried a growing context.

Code Mode separates these responsibilities. The LLM focuses on planning. It generates executable code that defines the full workflow for a task.

Code Mode works best when multiple MCP servers are involved, workflows have several steps, or tools need to share data. For simpler setups with one or two tools, Classic MCP works well.

A mixed setup also works. Use Code Mode for heavier workflows like search or databases, and keep simple tools as direct calls.

This includes:

Selecting the right tools
Passing data between tools
Defining how the final output is produced

The system exposes a minimal interface to the LLM. It can list available tools, read tool details, and, when required, understand how each tool works. Tool definitions are accessed on demand, which keeps the initial context small.

Once the plan is generated, execution moves to a runtime environment. The code runs in a sandbox and interacts directly with tools. All intermediate steps, tool responses, and data transformations stay within this layer.

This removes the need for repeated LLM calls during execution. The workflow runs in one pass, guided by the generated code. The LLM is involved mainly at the planning stage and for producing the final response if required.

The flow becomes more structured. A request comes in, relevant tools are identified, code is generated, and execution happens in a controlled environment. The system handles state and intermediate data outside the LLM.

This approach improves clarity in how tasks are executed. The generated code can be inspected, debugged, and understood directly. Each request follows a defined path, which makes behavior easier to track and reason about.

Using Bifrost CLI in Our Workflow

Getting started required two commands. First, start the gateway:

npx -y @maximhq/bifrost

Then launch the CLI from a separate terminal:

npx -y @maximhq/bifrost-cli

MCP servers are registered once through the API. The key flag is is_code_mode_client, which tells Bifrost to handle that server through Code Mode instead of sending its tool definitions on every request:

curl -X POST http://localhost:8080/api/mcp/client \
  -H "Content-Type: application/json" \
  -d '{
    "name": "youtube",
    "connection_type": "http",
    "connection_string": "http://localhost:3001/mcp",
    "tools_to_execute": ["*"],
    "is_code_mode_client": true
  }'

Once registered, the LLM discovers tools on demand using listToolFiles and readToolFile, then submits a full execution plan through executeToolCode. A workflow that previously took six LLM turns now completes in three to four.

Bifrost organizes tool definitions using two binding levels. Server-level (default) groups all tools from a server into one .pyi file. Tool-level gives each tool its own file — better for servers with 30+ tools. Set it once in config.json:

{
  "mcp": {
    "tool_manager_config": {
      "code_mode_binding_level": "server"
    }
  }
}

Debugging became simpler because the generated code is the execution plan. When something went wrong, the issue was visible directly in the code rather than buried in prompt chains. This setup also made execution easier to inspect.

results = youtube.search(query="AI infrastructure", maxResults=5)
titles = [item["snippet"]["title"] for item in results["items"]]
result = {"titles": titles, "count": len(titles)}

The execution runs in a Starlark interpreter, a restricted subset of Python. A few constraints to keep in mind:

No import statements, file I/O, or network access
Classes are not supported, use dictionaries
Tool calls run synchronously; async handling is not required
Each tool call has a default timeout of 30 seconds

Code Mode also works with Agent Mode for automated workflows. The listToolFiles and readToolFile tools are always auto-executable since they are read-only.

The executeToolCode tool only auto-executes if every tool call within the generated code is on the approved list. If any call falls outside that list, Bifrost returns it to the user for approval before running.

Impact on Token Usage and System Efficiency

The reduction in token usage came from four specific changes:

Tool schemas were sent only when required
Intermediate outputs stayed within the execution layer
Repeated context across steps was removed
Fewer LLM calls were needed, since execution moved to a sandbox and ran in a single flow

These changes had a clear effect. Token usage dropped by nearly half. Latency reduced along with it. Execution became more predictable, since each request followed a defined path with fewer moving parts.

The broader takeaway is clear. Token cost comes from system design. Small changes in prompts or outputs help at the edges. The main overhead comes from the system's structure.

LLMs work best when they focus on planning. Managing execution through repeated loops adds cost and introduces variability. A separate execution layer keeps the flow stable and easier to understand. Context also needs careful control. It should be built for each request with only the required information. Letting it grow across steps results in unnecessary overhead and increased token usage.

Conclusion

Token inefficiency in MCP workflows comes from system design. Bifrost and Code Mode introduced a clear separation between planning and execution. The LLM handles planning, and the runtime handles execution. This brought immediate and measurable improvements in both cost and system behavior.

If you are working with MCP workflows at scale, Bifrost is worth exploring. The documentation provides a good starting point to set up the gateway, connect servers, and run workflows using Code Mode.

Composer 2 is controversial, but my actual experience was solid

Arindam Majumder — Sat, 21 Mar 2026 06:55:09 +0000

I tried Composer 2 properly today, and honestly, if you put all the controversy aside for a second, the model itself is not bad at all.

In fact, my first impression is that it’s a real upgrade over Composer 1 and 1.5. I gave it a pretty solid test. I asked it to build a full-stack Reddit clone and deploy it too.

On the first go, it handled most of the work surprisingly well. The deployment also worked, which was a good sign. The main thing that broke was authentication.

Then on the second prompt, I asked it to fix that, and it actually fixed the auth issue and redeployed the app.

That said, it was not perfect. There were still some backend issues left that it could not fully solve. So I would not say it is at the level of Claude Opus 4.6 or GPT-5.4 for coding quality.

But speed-wise, it felt much faster. For me, it was around 5 to 7x faster than Opus 4.6 / GPT-5.4 in actual workflow, and it also feels much more cost-effective.

That combination matters a lot.

Because even if the raw coding quality is still below Opus 4.6 / GPT-5.4, the overall experience was smoother than I expected. It gets you from idea to working product much faster, and for a lot of people that tradeoff will be worth it.

My current take is:

Better than Composer 1 / 1.5 by a clear margin
Fast enough to change how often I’d use it
Good at getting most of the app done quickly
Still weak enough in backend reliability that I would not fully trust it yet for complex production work
Not as strong as Opus 4.6 / GPT-5.4 in coding depth, but still very usable

So yeah, I agree with the criticism that it is not on the same level as Opus 4.6 / GPT-5.4 for hard-coding tasks. ( may be because the base model is Kimi K2.5)

But I also think some people are dismissing it too quickly. If you judge it as a fast, cheaper, improved Composer, it is genuinely solid.

I shared a longer breakdown here with the exact build flow, where it got things right, and where it still fell short, in case anyone wants more context

Building an AI-Powered Content Moderation API with InsForge Edge Functions

Arindam Majumder — Fri, 20 Mar 2026 09:55:13 +0000

Introduction

Modern applications rely on user-generated content such as comments, reviews, and messages. Platforms must moderate this content to enforce safety policies and maintain compliance. Manual moderation does not scale, so production systems typically rely on automated moderation pipelines powered by AI.

Traditional implementations require multiple backend services. Developers often provision servers, integrate AI APIs, manage databases, and configure storage separately. This fragmented setup increases operational overhead and slows development.

InsForge simplifies this architecture by combining Edge Functions, PostgreSQL Database, Storage, and Model Gateway in a single platform. Benchmarks also show that it can deliver ~1.6× faster responses and 2.4x lower token usage compared to fragmented integrations.

In this tutorial, we will build a production-ready AI moderation API that runs entirely within InsForge.

What We Are Building

Here are the tools that we will be using to build a simple backend moderation workflow using InsForge core services:

AI Moderation API Endpoint: We will create an API endpoint using Edge Functions that accepts user-submitted text content and processes moderation requests.
AI-Powered Content Evaluation: The API will use Model Gateway to access an AI model that classifies submitted content as SAFE or UNSAFE.
Database Storage for Approved Content: Approved comments will be stored in a PostgreSQL Database managed by InsForge.
Attachment Handling with Storage: Optional user attachments will be uploaded and stored using Storage Buckets.
Automated Moderation Response: Unsafe content will be rejected immediately, and the API will return a structured moderation response.
Production-Ready Backend Workflow: The moderation pipeline will run entirely within InsForge using Database, Edge Functions, Model Gateway, and Storage, without external servers or additional infrastructure.

Project Setup and Repository Structure

Before configuring the backend resources, clone the project repository and review the project structure.

Clone the repository:

git clone https://github.com/Studio1HQ/Content-moderation-Insforge
cd content-moderation-app

Install dependencies:

npm install

The repository contains both the Next.js frontend and the InsForge Edge Function used for moderation.

Repository Structure

Folder	Purpose
`src/app`	Next.js application pages and layouts
`src/components`	UI components such as the moderation form
`src/lib`	Client utilities for connecting to InsForge APIs
`insforge-functions/moderate-comment`	Edge Function implementation for moderation
`handler.ts`	Serverless function that processes moderation requests

This structure keeps the frontend and backend logic organized within the same project while allowing the Edge Function to be deployed independently.

After cloning the repository, proceed with configuring the backend resources in InsForge.

Note: You can set up this backend in two ways. Follow the manual steps in this tutorial to create the database, storage bucket, and Edge Function using the dashboard and CLI. Alternatively, you can use InsForge MCP with your AI coding agent to provision the same resources using a single prompt. See the MCP section at the end of the article for the prompt template and instructions.

Step 1: Setting Up the Database

InsForge provides a managed PostgreSQL Database that you can configure directly from the dashboard.

Open the Tables Section

Open your project in the InsForge Dashboard.
In the left sidebar, select Tables.
Click the + icon next to Tables.

Create the following columns.

Column	Type	Description
`id`	`uuid`	Primary key for each comment
`content`	`string`	User submitted comment text
`attachment_url`	`string`	URL for uploaded file (optional)
`status`	`string`	Moderation result (`approved` or `rejected`)
`created_at`	`timestamp`	Time when the comment was created

Save the Table

Click Create Table to apply the schema.
The comments table will appear in the Tables panel.

Step 2: Creating the Edge Function

Next, create the serverless API that will process moderation requests.

InsForge Edge Functions allow you to run backend logic without managing servers. In this tutorial, the function receives user content, evaluates it using AI, and stores approved results in the database.

Navigate to the Edge Function directory in the repository:

insforge-functions/moderate-comment/

Inside this folder, there will be a file named:

handler.ts

This file will contain the moderation logic executed by the Edge Function.

The Edge Function performs the following tasks:

Accept a POST request containing user content.
Send the content to the AI model through Model Gateway.
Classify the content as SAFE or UNSAFE.
Upload attachments to Storage if present.
Insert approved content into the comments table.
Return a structured moderation response.

All moderation logic runs inside the Edge Function, keeping the backend workflow centralized within InsForge.

Deploy the function using the InsForge CLI:

insforge functions deploy moderate-comment--file ./insforge-functions/moderate-comment/handler.ts

Once deployed, the function becomes available as a backend API endpoint that the frontend application can call.

Step 3: AI Integration Inside the Function

The moderation logic inside the Edge Function uses Model Gateway, which provides unified access to multiple AI models directly within InsForge.

Model Gateway allows Edge Functions to call AI models without configuring external API clients or managing provider-specific integrations.

Open the Model Gateway section in the InsForge dashboard and enable a model for the project.

For this tutorial, enable:

openai/gpt-4o-mini

This model will be used to classify incoming content during moderation.

Use the CLI to send a test request to the moderation API.

insforge functions invoke moderate-comment--data"{\"content\":\"This community platform is very helpful.\"}"

This command sends a JSON payload containing the content field to the Edge Function.

The Edge Function also inserts the approved comment into the comments table in the database.

Step 4: Configuring Insforge Storage

The moderation workflow also supports optional file uploads using InsForge Storage. Storage provides an S3-compatible object storage system that integrates directly with Edge Functions and the database.

When a user submits a comment with an attachment, the Edge Function uploads the file to a storage bucket before inserting the comment into PostgreSQL.

Create a Storage Bucket

Open the Storage section in the InsForge dashboard.

Navigate to Storage in the sidebar.
Click Create Bucket.
Name the bucket: attachments

This bucket will store files uploaded with moderated comments.

The upload operation returns a public file URL, which is stored in the attachment_url column of the comments table.

The moderation function processes attachments as follows:

The user submits content with an optional file.
The Edge Function evaluates the text using AI moderation.
If the content is classified as SAFE, the file is uploaded to the attachments bucket.
The returned file URL is stored in the comments table.
If the content is UNSAFE, the function rejects the request and no file is uploaded.

This ensures that only approved content and attachments are stored, keeping the storage system aligned with the moderation rules.

Step 5: Building the Next.js UI

The repository already includes a Next.js application that provides a simple interface for interacting with the moderation API.

Navigate to the frontend code inside the src directory.

Key UI Files

File / Folder	Purpose
`src/app/page.tsx`	Main page that renders the moderation interface
`src/components`	Reusable UI components for the moderation workflow
`src/lib/insforge.ts`	Utility for connecting the frontend to the InsForge backend

The UI includes a form where users submit content for moderation.

The form collects:

Text content entered by the user
Optional file attachment
Submit an action that triggers the moderation request

When the user submits the form, the application sends a POST request to the Edge Function endpoint.

The UI handles the API response and updates the interface accordingly.

Approved comments appear in the moderation results section.
Rejected content displays an error message.
Approved entries are also visible in the comments database table.

This setup creates a complete workflow where the Next.js UI communicates with the InsForge Edge Function to perform moderation in real time.

Using an AI Agent to Build the UI

You can also accelerate this step using an AI coding agent (such as Cursor, Claude Code, or other agent-based tools). Instead of manually writing the UI components, the agent can generate the form, API calls, and component structure based on a prompt.

Example prompt:

Create a Next.js page for a content moderation demo.

Requirements:
- A form with a textarea for user comments
- An optional file upload input
- A submit button
- Send a POST request to the InsForge Edge Function endpoint for moderation
- Display the moderation result (approved or rejected) in the UI
- Use React state to handle form submission and responses

Step 6: Testing the API Endpoint

After deploying the Edge Function and setting up the UI, test the moderation workflow to verify that the API behaves correctly.

Submit Safe Content

Enter a comment through the UI and submit the form.

Expected behavior:

The Edge Function sends the content to the AI moderation model.
The model classifies the text as SAFE.
The function inserts the comment into the comments table in PostgreSQL.
If an attachment is included, the file is uploaded to the attachments storage bucket.
The API returns an approved response to the frontend.

Next, test a rejection case.

Expected behavior:

The Edge Function sends the text to the AI moderation model.
The model classifies the content as UNSAFE.
The function immediately returns a rejection response.
No entry is inserted into the comments table.
No file is uploaded to Storage.

The table in your Insforge dashboard also reflects the results:

Step 6: Deployment Using InsForge

Once the function and UI are ready, deploy the backend using the InsForge CLI. This publishes the Edge Function and connects it to the project environment.

Refer to the deployment guide here

Authenticate the CLI with your InsForge account.

insforge auth login

Complete the authentication process in the browser. Link the local project directory to your InsForge backend.

insforge link

Select the project created earlier in the InsForge dashboard. This connects the CLI to the correct backend workspace.

Deploy the Next.js application while passing the required environment variable.

insforge deployments deploy .--env"{\"NEXT_PUBLIC_INSFORGE_BASE_URL\":\"https://your-project.insforge.app\"}"

This environment variable allows the frontend to communicate with the deployed Edge Function.

Verify the Deployment

After deployment, the application becomes accessible via the InsForge-hosted domain.

Access the live demo here

Using MCP to Accelerate Development

Instead of manually creating tables, storage buckets, and Edge Functions, you can also configure the backend using Remote MCP (Model Context Protocol).

MCP exposes InsForge backend capabilities as tools that an AI coding agent can call to provision resources automatically. With a single prompt, the agent can generate the database schema, configure storage, and deploy the moderation function.

Example prompt used to create this backend workflow:

Create backend resources for a content moderation application using InsForge.

Requirements:

1. Create a PostgreSQL table named "comments" with fields:
   id (UUID primary key)
   content (text)
   attachment_url (text, nullable)
   status (text)
   created_at (timestamp)

2. Create a storage bucket named "attachments" for storing uploaded files.

3. Create an Edge Function named "moderate-comment" that:
   - accepts POST requests with comment text
   - sends the text to an AI model
   - classifies the content as SAFE or UNSAFE
   - uploads attachments to storage if present
   - inserts approved content into the database

Using MCP, developers can provision backend resources and deploy functions directly from prompts, significantly accelerating backend setup while keeping the same architecture described in this tutorial.

Refer to the quick demo here.

Conclusion

In this tutorial, we built a content moderation API using InsForge Edge Functions, integrated AI-powered classification through Model Gateway, stored approved results in PostgreSQL, and handled optional file uploads with Storage. The entire workflow runs inside InsForge, without external servers or fragmented infrastructure.

This approach demonstrates how developers can combine Edge Functions, AI integration, database services, and storage to implement production-ready backend APIs with minimal operational overhead.

If your application relies on user-generated content, moderation pipelines, or AI-assisted workflows, this architecture provides a straightforward and scalable foundation.

Ready to simplify your backend stack? Explore InsForge’s Edge Functions, Model Gateway, PostgreSQL database, and Storage services to build intelligent APIs without managing infrastructure.

Try InsForge
Quickstart guide here

Cursor Composer 2: Features, Pricing, Benchmarks, and Initial Impressions

Arindam Majumder — Thu, 19 Mar 2026 20:25:28 +0000

Introduction

Cursor has released Composer 2, the latest version of its in-house coding model.

The announcement is focused and fairly easy to summarize. Cursor is making three main claims:

Composer 2 is frontier-level at coding
it is materially better than previous Composer versions on Cursor’s published benchmarks
it is priced aggressively enough to be practical for everyday use

That combination makes the release worth paying attention to. In this post, I’ll walk through what Composer 2 is, what Cursor says improved, how the benchmark results look, what the pricing means, and my initial take on the release.

What Is Composer 2?

Composer 2 is Cursor’s latest in-house coding model.

Cursor describes it as frontier-level at coding and positions it as a better cost-performance option for agentic software work. The model is now available in Cursor, and the announcement puts most of the emphasis on three areas:

stronger coding performance
improved long-horizon task handling
lower cost than many competing fast models

Unlike some model launches that bundle a large number of product features together, this one is mostly about the model itself. Cursor is not presenting Composer 2 as a general platform shift. It is presenting it as a more capable and more economical coding model.

Composer 2 Key Features

The Composer 2 announcement is short, but there are still a few important takeaways.

Better coding performance

Cursor says Composer 2 delivers large improvements on all of the benchmarks it tracks, including Terminal-Bench 2.0 and SWE-bench Multilingual.

That matters because it suggests the gains are not limited to one internal evaluation. Cursor is showing improvement across several coding-oriented benchmarks rather than relying on a single headline number.

Continued pretraining

One of the most notable details in the post is that these improvements come from Cursor’s first continued pretraining run.

This is important because continued pretraining is often what gives a model a stronger base before more specialized post-training methods are applied. Cursor is explicitly saying that Composer 2 starts from a better foundation than earlier Composer versions.

Reinforcement learning for long-horizon tasks

Cursor also says it trains Composer 2 on long-horizon coding tasks using reinforcement learning.

This is probably the most interesting technical claim in the announcement. Cursor says Composer 2 can solve challenging tasks requiring hundreds of actions. That implies the model is being optimized for sustained multi-step software tasks, not just short code completions or simple edits.

A fast variant with the same intelligence

Cursor also introduces a faster Composer 2 variant and says it has the same intelligence.

That is a useful product choice. Instead of forcing users to pick between a “smart” model and a “fast” model family, Cursor is presenting speed as a deployment option on top of the same underlying capability level.

Composer 2 Benchmarks

Cursor publishes three benchmark comparisons in the announcement:

Model	CursorBench	Terminal-Bench 2.0	SWE-bench Multilingual
Composer 2	61.3	61.7	73.7
Composer 1.5	44.2	47.9	65.9
Composer 1	38.0	40.0	56.9

These gains are large enough to be meaningful.

The biggest point here is not just that Composer 2 is ahead of Composer 1 and 1.5, but that the improvements show up consistently across all three benchmarks. That gives the release more credibility than a single isolated result would.

Terminal-Bench 2.0 is especially relevant because Cursor frames it as an evaluation for agentic terminal use. If Composer 2 is genuinely stronger there, that supports Cursor’s claim that the model is getting better at longer, more interactive coding tasks.

SWE-bench Multilingual is also worth noting because it suggests broader coding competence beyond narrow English-only setups.

Still, these are vendor-published numbers, so the right takeaway is measured optimism rather than certainty.

How Composer 2 Is Priced

Cursor says Composer 2 is priced at:

$0.50 per million input tokens
$2.50 per million output tokens

The faster variant is priced at:

$1.50 per million input tokens
$7.50 per million output tokens

Cursor also says the fast variant has lower cost than other fast models and that fast will be the default option.

This part of the announcement is more important than it looks. Model releases are usually judged on benchmark quality first, but pricing determines whether a model becomes part of normal daily use or gets reserved for occasional high-value tasks. Cursor is clearly trying to push Composer 2 into the first category.

On individual plans, Composer usage is part of a standalone usage pool with generous usage included.

Composer 2 vs Earlier Composer Versions

Based on Cursor’s published table, Composer 2 is a clear step up from Composer 1.5 and Composer 1.

The improvement is visible across all the benchmarks included in the post, and Cursor attributes that jump to a combination of:

a stronger base model from continued pretraining
reinforcement learning on long-horizon coding tasks

That is a sensible recipe for a coding model. Better base training improves general capability, while long-horizon RL helps the model stay coherent over extended multi-step tasks.

From the announcement alone, Composer 2 looks like a real model upgrade rather than a minor iteration.

Initial Impressions

My first impression is that this is a disciplined release.

Cursor is not trying to claim that Composer 2 changes everything. The message is narrower and more believable: the model is better, it handles long-horizon coding tasks more effectively, and it is priced aggressively enough to be useful in regular workflows.

The long-horizon point is the one I would pay most attention to. A lot of coding models can produce a good patch in one pass. Fewer models stay reliable across a task that unfolds over many actions. If Composer 2 is genuinely stronger there, that is a meaningful improvement.

The pricing is the other major strength. A coding model can be strong on benchmarks and still be awkward in practice if the economics are wrong. Cursor seems to understand that and is making cost a central part of the launch rather than an afterthought.

At the same time, this is still an announcement built around Cursor’s own evaluation framing. The benchmark gains look strong, but the real test will be whether Composer 2 feels materially better in day-to-day software work.

Final Thoughts

Composer 2 looks like a meaningful upgrade to Cursor’s coding model stack.

The release is compelling for three reasons:

the benchmark gains are substantial
the training story is technically coherent
the pricing is practical

If you already use Cursor, Composer 2 is worth trying.

If you evaluate coding models more broadly, this release is notable because it tries to improve both capability and economics at the same time. That is the right combination to optimize for.

Running LLM Applications Across Providers with Bifrost

Arindam Majumder — Tue, 17 Mar 2026 16:15:23 +0000

Many modern applications include AI features that rely on large language models accessed through APIs. When an application sends a prompt to a model and receives a response, that request usually goes through an external service.

Getting access to different LLM models is easier today. Providers such as OpenAI and Anthropic provide model APIs, and platforms like Amazon Bedrock and Google Vertex AI give access to several models from one place. Because of this, many applications connect to more than one provider to compare models, manage cost, or keep a backup option if one service fails.

But each provider works a little differently. Authentication methods, rate limits, and request formats are not the same. Managing these differences inside an application can slowly add complexity to the system. In this article, let us explore Bifrost, an open-source LLM gateway that provides a single layer to route requests and manage interactions with multiple model providers.

The Hidden Cost of Provider Integrations

Connecting to several LLM providers may look simple at the start. Adding another provider can feel like just integrating one more API.

That situation changes once the application runs in production. Requests may need to go to different models based on cost, response quality, or latency. If a provider slows down or becomes unavailable, the system must redirect requests to another provider and keep the service running.

Handling these situations introduces additional logic into the codebase. The application needs to manage how requests are routed between models. It must also include retry logic for failed calls, fallback providers during outages, and tracking for how requests are distributed across models.

Each of these responsibilities adds extra work to the system. Over time, operational logic becomes part of the application and increases maintenance effort. This overhead becomes the hidden cost of working directly with multiple model providers.

Introducing Bifrost: A Gateway for LLM Infrastructure

Bifrost is an open-source LLM and MCP gateway designed to manage interactions between applications and model providers. It sits between the application and the LLM services and acts as a central layer that controls how requests move between systems.

Applications often connect directly to each provider they use. Bifrost adds a gateway layer between the application and the providers, so requests pass through a single entry point before reaching the model services.

This structure separates provider management from the application. The application sends requests to one endpoint, and the gateway manages communication with different model providers. Provider configuration and request handling stay inside the gateway layer, reducing provider-specific logic in the application code.

Core Infrastructure Capabilities

Bifrost provides several infrastructure capabilities for managing LLM interactions across providers. These capabilities move provider-specific handling out of the application and into the gateway layer.

Multi-provider routing: Bifrost supports multiple AI providers through a single API interface. Applications send requests to one endpoint, and the gateway routes each request to the configured provider or model.
Load balancing: When multiple providers or API keys are configured, Bifrost distributes requests across them based on defined rules. Traffic spreads across providers and reduces the chance of hitting rate limits on a single service.
Automatic fallback: When a provider returns an error or becomes unavailable, Bifrost sends the request to another configured provider.
Semantic caching: Bifrost stores responses and returns them for similar prompts. Prompt comparison uses semantic similarity. This reduces repeated API calls and improves response time.

Platform Support and Integrations

Bifrost fits environments where applications use multiple models and providers. The gateway exposes an OpenAI-compatible API, so applications that already use OpenAI SDKs can connect with minimal changes and send requests through a single endpoint.

Bifrost works with several LLM providers, such as OpenAI, Anthropic, Amazon Bedrock, Google Vertex AI, Cohere, and Mistral. Applications can reach these providers through the same gateway interface.

The gateway also supports the Model Context Protocol (MCP). Systems that use MCP can connect tools and external services through the same layer used for model requests. Bifrost also includes a plugin system for adding custom behavior such as request validation, logging, or request transformation.

Bifrost can run using tools such as NPX or Docker and can operate in local setups or production environments. The project is open source under the MIT license and can run across different infrastructure environments.

Gateway Performance and Benchmark

A gateway processes every request sent to a model provider. The performance of this layer becomes important in systems that handle a large number of AI requests.

Bifrost is written in Go, a language often used for backend services that process many requests simultaneously. The system focuses on keeping the extra processing time very small.

Benchmark tests show that Bifrost adds about 11 microseconds of latency at 5,000 requests per second. One microsecond equals 0.001 milliseconds, so 11 microseconds equals 0.011 milliseconds, which means the delay introduced by the gateway remains extremely small.

The published benchmarks were executed on AWS EC2 t3.medium and t3.large instances. These are cloud virtual machines with moderate CPU and memory resources that are commonly used to run backend services and APIs.

Bifrost also provides a public benchmarking repository with the scripts and setup used in the tests. Anyone can run the same tests or perform custom benchmarking based on their own infrastructure, traffic patterns, or model providers.

Getting Started with Bifrost

Bifrost is designed for quick setup and can run locally or in a server environment. The gateway can start in a few steps and begin routing LLM requests through a single endpoint.

One way to start Bifrost is by using NPX:

npx -y @maximhq/bifrost

Bifrost can also run using Docker, which allows the gateway to start inside a container environment:

docker run -p 8080:8080 maximhq/bifrost

After the gateway starts, applications can send LLM requests to the Bifrost endpoint. The gateway then routes the requests to the configured model providers.

Configuration options allow the gateway to define providers, API keys, routing rules, caching behavior, and fallback settings. These configurations control how requests move between different LLM providers.

Closing

Managing several LLM providers inside an application can introduce extra operational logic and maintenance effort. A gateway layer offers a cleaner structure for handling these interactions.

Bifrost provides this layer by placing a gateway between applications and model providers. Requests go through one endpoint, and the gateway manages routing and provider communication.

This approach keeps provider integrations outside the core application code and places request management in a separate infrastructure layer.

To explore configuration options, deployment steps, and additional features, refer to the official Bifrost documentation.

5 OpenClaw Plugins That Actually Make It Production-Ready

Arindam Majumder — Fri, 13 Mar 2026 15:19:20 +0000

Introduction

There is a certain point every serious OpenClaw user reaches. The agent is running, the setup works, and then slowly, almost without noticing, the cracks show up. A workflow that should take seconds starts requiring three follow-up prompts. The context window fills up with things the agent should already know. The API bill at the end of the month is higher than expected, and there is no clear answer for why.

Most people at this point start tweaking their skills, adjusting prompts, or switching models, but the problem is usually none of those things.

OpenClaw's default configuration is designed to get you started, not to match how you actually use it. The real power that makes it suitable for daily professional use lies in the plugin layer, yet most OpenClaw users have never explored it.

In this post, we are covering five OpenClaw Plugins, and each solves a different problem, each adding a layer that the default setup simply does not have. But before getting into the plugins themselves, it is worth understanding what separates a plugin from a skill.

What Are OpenClaw Plugins (And Why They're Different from Skills)

If you have spent any time in the OpenClaw community, you have probably seen both terms used interchangeably. They are not the same thing, and the distinction matters more than it seems.

A skill is a markdown file, specifically a SKILL.md that gets injected into the agent's context at inference time. It shapes how the agent thinks, what tone it uses, and what steps it follows. Every time the agent runs, that file loads into the prompt. Skills are useful for behavior, but they come at a cost: they consume tokens on every single request, whether or not they are relevant to what you asked.

A plugin is fundamentally different. It is a standalone executable that runs as a separate process alongside OpenClaw. Instead of loading into context, it exposes a set of tools through a defined interface that the agent can call when it actually needs them. OpenClaw loads plugins once at startup and calls into them only when a task requires it. No tokens consumed just by existing.

How to Install a Plugin

Installing any plugin follows the same pattern:

openclaw plugins install <plugin-name>

That command downloads the plugin, registers it in your OpenClaw configuration at ~/.openclaw/openclaw.json, and makes its tools available the next time the agent starts. You can open that file at any time to see which plugins are currently registered and adjust their individual configurations.

To confirm a plugin is active after installation, run:

openclaw plugins list

This returns all registered plugins and their current status. If something is not showing up, a full restart of the OpenClaw daemon is usually all it takes.

With that covered, here are the five plugins worth adding to your setup.

1. Manifest

When you configure OpenClaw, you pick a default model. Claude Opus, GPT-4, whatever you prefer. From that point on, every request, regardless of its type, goes to that model. Asking the agent to list files in a directory costs the same as asking it to debug a race condition across three services. The model does not know the difference, and OpenClaw does not try to make one.

This is where most API bills quietly spiral. Not from one expensive task, but from hundreds of simple ones hitting a premium model they never needed.

Manifest sits between OpenClaw and your LLM providers. Every request passes through it before reaching a model. It reads the request, classifies the task complexity, and routes it to the cheapest model capable of handling it. Simple lookups go to lighter models. Reasoning-heavy tasks escalate to whatever model can actually handle them. Routing occurs in milliseconds and is invisible to the agent; it only sees a response.

The cost difference compounds fast. Users running OpenClaw through Manifest have reported up to 70% reduction in monthly API spend, not by doing less, but by stopping the habit of paying Opus prices for Haiku-level work. The Manifest dashboard makes this visible: you can see cost broken down per session, per tool call, and per model, so you know exactly where your spend is going and whether the routing decisions are working as expected.

Installing Manifest:

openclaw plugins install manifest

Once installed, Manifest registers itself as the default routing layer. You can configure routing thresholds and model preferences in ~/.openclaw/openclaw.json under the manifest plugin entry.

Manifest makes the biggest difference in setups where the agent runs long sessions, handles multi-step tasks, or operates overnight without supervision. The more requests flow through OpenClaw, the more the routing logic saves, because the inefficiency it fixes is not a one-time cost; it is per-request.

2. Composio

Out of the box, OpenClaw cannot reach your Gmail, Slack, GitHub, or Notion. Not because the agent is incapable, but because every external service requires OAuth authentication, token management, and refresh handling, none of which OpenClaw sets up for you. Most people work around this by manually generating API keys, pasting them into configuration files, and hoping the tokens don't expire mid-session. It works until it does not.

Composio solves this at the authentication layer. It runs as an MCP server that sits between OpenClaw and every external app you want the agent to reach. You connect your accounts once through the Composio dashboard, and from that point on, OpenClaw talks to Composio, and it handles everything else. Token refresh, OAuth flows, rate limits, API versioning. None of that touches your OpenClaw config directly.

Each app connection runs in an isolated MCP session. If one integration fails or a token expires, it does not affect the others. The agent continues operating normally while Composio handles the reconnection in the background.

Installing Composio:

openclaw plugins install composio

After installation, connect your apps through the Composio dashboard and add the plugin entry to ~/.openclaw/openclaw.json:

{
  "plugins": {
    "composio": {
      "apiKey": "your-composio-api-key"
    }
  }
}

In practice, what this unlocks is straightforward. A single prompt like "summarize my unread emails, open a GitHub issue for anything that needs follow-up, and post a summary to the team Slack channel" now executes end to end, no switching tabs, no copying API keys, no manual auth setup. The agent has the required access, and Composio ensures it remains valid.

With 850+ supported apps, Composio covers most of what a professional OpenClaw setup would ever need to reach.

3. Hyperspell

OpenClaw's default memory is a MEMORY.md file. It grows with every session, gets compacted when it reaches a limit, loses information in the process, and reloads entirely on every turn, whether the content is relevant or not. For occasional use, this is fine, but for anyone relying on OpenClaw daily, it becomes a real problem fast.

Hyperspell replaces this layer entirely. It indexes your connected data sources, emails, documents, and past conversations into a knowledge graph, then injects only the relevant slice of that graph before each agent turn. The agent gets what it needs, not everything it has ever seen.

Memory also becomes sharper over time. Every query refines how the knowledge graph is indexed, so context recall improves the more you use it. An agent running Hyperspell can reference a decision you made three weeks ago without you having to bring it up.

Installing Hyperspell:

openclaw plugins install @hyperspell/openclaw-hyperspell

Connect your data sources through the Hyperspell dashboard, then add your API key under the hyperspell entry in ~/.openclaw/openclaw.json. Context injection is automatic from there.

4. OpenClaw Foundry

Most workflows repeat. You run the same sequence of tasks every morning, follow the same steps every time a PR needs review, and ask the agent the same three things before a meeting. OpenClaw handles all of these, but it handles them the same way every time, waiting for you to prompt it from scratch. It does not recognize the pattern. It does not try to make things easier on its own.

Foundry fixes this. It sits in the background during your sessions, watches what you ask for, and, when it detects a recurring pattern, writes a new tool definition into itself. That tool becomes part of the agent's available toolkit the next time you start a session, no manual configuration required.

What makes this different from writing a skill is the output. A skill adds behavioral instructions to the agent's context. Foundry creates an executable tool that the agent can call, with its own inputs and outputs, registered in the tool registry and available on demand.

Installing Foundry:

openclaw plugins install @getfoundry/foundry-openclaw

This downloads the plugin from npm, extracts it to ~/.openclaw/extensions/foundry/, enables it automatically, and restarts the gateway. After that, add the following to ~/.openclaw/openclaw.json:

{
  "plugins": {
    "entries": {
      "foundry": {
        "enabled": true,
        "config": {
          "autoLearn": true,
          "sources": {
            "docs": true,
            "experience": true,
            "arxiv": false,
            "github": false
          },
          "marketplace": {
            "autoPublish": false
          }
        }
      }
    }
  }
}

autoLearn: true is the key setting, it tells Foundry to continuously learn from your sessions without requiring you to trigger it manually. The sources block controls where Foundry pulls additional context when writing new tools: OpenClaw's own documentation, your past session experience, arXiv papers, and public GitHub repos. For most setups, keeping docs and experience enabled is enough to start.

5. Opik

Multi-step agent runs fail in non-obvious ways. A tool call returns incorrect output, a sub-agent silently errors out, and a model call takes 12 seconds on a task that should take 2 seconds. Without structured tracing, you are left reading raw logs and guessing. That gets old fast.

Opik is an open-source LLM and agent observability platform built by Comet ML. The OpenClaw plugin hooks into the gateway process and exports a structured trace for every run, LLM request and response spans, tool call inputs and outputs, sub-agent lifecycle events, latency at each step, and token usage with cost. Every event that matters has a corresponding span in the Opik dashboard.

Image reference: https://github.com/comet-ml/opik-openclaw

This is a different layer from what Manifest covers. Manifest tells you how much a request costs and which model handled it. Opik tells you what the agent actually did inside that request, which tools it called, in what order, what each one returned, and where the run slowed down or failed. Both answer different questions and neither replaces the other.

Installing Opik:

Requirements: OpenClaw >=2026.3.2, Node.js >=22.12.0

openclaw plugins install @opik/opik-openclaw

After installation, restart the gateway, then run the setup wizard:

openclaw opik configure

This validates your endpoint and API key and automatically writes the config. To verify everything is connected:

openclaw opik status

The recommended config in ~/.openclaw/openclaw.json looks like this:

{
  "plugins": {
    "entries": {
      "opik-openclaw": {
        "enabled": true,
        "config": {
          "apiKey": "your-api-key",
          "apiUrl": "https://www.comet.com/opik/api",
          "projectName": "openclaw",
          "workspaceName": "default"
        }
      }
    }
  }
}

For teams that cannot send trace data to a third party, Opik is fully self-hostable. Replace apiUrl with your own instance endpoint, and nothing else changes.

Where to Go From Here

Each plugin owns a distinct layer. Hyperspell handles context before the request starts. Manifest handles model routing during it. Composio handles external reach when the agent needs to act. Foundry watches for patterns across sessions and builds tools from them. Opik traces everything after the fact, so you know exactly what happened and why.

None of them overlap, and none of them are doing the same job twice. You can start with just one, whichever layer is causing the most friction in your current setup, and layer in the rest as your workflow grows.

Each plugin has its own documentation to read before configuring anything: Manifest, Composio, Hyperspell, OpenClaw Foundry, and Opik. The OpenClaw plugin docs cover the installation system in full if you want to go deeper on how plugins interact with the gateway.

[Boost]

Arindam Majumder — Sat, 07 Mar 2026 08:35:19 +0000

Arindam Majumder

Mar 6

What is LLM Observability? The Complete Guide (2026)

#ai #llm #observability #programming

Comments 2

15 min read