Forem: Dhananjay Lakkawar

The Open-Source Alternative to Oracle 26ai: Why PostgreSQL is All You Need

Dhananjay Lakkawar — Thu, 02 Apr 2026 20:03:14 +0000

The database industry is currently undergoing a massive identity crisis. Driven by the Generative AI boom, legacy database vendors are rushing to reinvent themselves as the ultimate "all-in-one" AI platforms.

The most recent, and perhaps most aggressive, example of this is Oracle AI Database 26ai.

With the launch of 26ai, Oracle has made a very clear architectural statement: The database should be the center of gravity for enterprise AI. They have embedded LLMs directly into the database engine, introduced native vector storage, and built the "Oracle Unified Memory Core" to provide persistent state for AI agents. They converge JSON, graph, vector, and relational data into a single, highly governed monolith.

If you are a legacy enterprise with two decades of PL/SQL technical debt and heavy regulatory requirements, this makes a lot of sense.

But if you are a startup founder, a scale-up CTO, or a cloud-native engineering team, adopting a monolithic, proprietary "AI Database" is a fast track to severe vendor lock-in and catastrophic licensing costs.

As a cloud architect, I have a completely different philosophy. You do not need a proprietary AI database. You just need PostgreSQL, pgvector, and scalable AWS cloud primitives.

Here is why PostgreSQL is the only AI database you actually need, and how to architect the open-source alternative to Oracle 26ai on AWS.

The Myth of the "AI-Native" Monolith

Oracle 26ai pushes the idea of running AI models and agentic workflows directly inside the database container to eliminate data movement and avoid the "integration tax" of modern AI stacks.

From an engineering perspective, this violates one of the core principles of modern system design: the separation of compute and storage.

Coupling unpredictable, highly intensive LLM inference compute with your mission-critical transactional database is an operational risk. If an AI agent hallucinates or gets stuck in a reasoning loop, you do not want it consuming the CPU cycles required to process your core user transactions.

Instead, we can use Amazon Aurora PostgreSQL paired with Amazon Bedrock to achieve the exact same "converged" AI capabilities, but with a decoupled, modular, and infinitely more cost-effective architecture.

Architectural Comparison: Monolithic vs. Composable

Deconstructing 26ai Features with PostgreSQL

Let’s break down the major selling points of proprietary AI databases and look at how the open-source ecosystem handles them natively today.

1. Vector Search & Similarity

The Proprietary Claim: You need a specialized engine or a massive vendor upgrade to handle vector search securely alongside relational data.
The PostgreSQL Reality: The open-source pgvector extension has already won the vector database war. Running on Amazon Aurora, pgvector utilizes Hierarchical Navigable Small World (HNSW) indexing to execute sub-millisecond similarity searches across millions of embeddings. You can join your vectors against standard relational tables in a single SQL query—no expensive licensing required.

2. Multi-Model Data (JSON, Graph, Relational)

The Proprietary Claim: Modern apps need a single engine that syncs JSON documents, graphs, and relational tables.
The PostgreSQL Reality: PostgreSQL has been doing this for a decade. The JSONB data type handles unstructured document data with indexing capabilities that rival dedicated NoSQL databases. If you need graph capabilities, Apache AGE brings graph queries directly into Postgres. It is the ultimate converged database.

3. In-Database AI & Agent Orchestration

The Proprietary Claim: Running LLMs inside the database natively is faster and more secure.
The PostgreSQL Reality: If you really want your database to invoke AI models without moving data, Amazon Aurora PostgreSQL provides the aws_ml extension. This allows you to write standard SQL queries that securely invoke Amazon Bedrock directly from the database engine.

However, in 90% of real-world use cases, you shouldn't do this. It is architecturally safer to keep your agentic orchestration in a stateless compute layer (like AWS Lambda or Step Functions) and treat PostgreSQL strictly as your robust, highly-available storage engine.

Building the Composable RAG Architecture on AWS

When you decouple your AI from your database, your Retrieval-Augmented Generation (RAG) architecture becomes incredibly flexible. You aren't locked into Oracle's specific LLM partnerships or pricing models. You can swap out a Claude 3.5 model for a Llama 3 model in Amazon Bedrock with a single line of code, while your PostgreSQL database remains completely untouched.

Here is what the standard production RAG flow looks like on AWS:

The CTO Perspective: Build vs. Buy and the Economics of AI

As a technology leader, choosing your database is the most consequential decision you will make. It dictates your hiring, your hosting costs, and your long-term agility.

Proprietary AI databases operate on the "convenience tax" model. They promise to reduce the complexity of wiring together different AI components, but the tradeoff is total vendor capture.

Here is why building on open-source PostgreSQL is the only logical choice for cloud-native teams:

1. Talent Density

Every competent backend engineer knows Postgres. You don't need to hire specialized, highly-paid DBAs to manage proprietary AI syntax.

2. True Cloud Economics

With Amazon Aurora Serverless v2, your database automatically scales up during high-traffic AI inference events and scales down to practically nothing at midnight.

3. Future-Proofing

The AI landscape changes every three weeks. By keeping your data in standard, open-source PostgreSQL and handling AI via Amazon Bedrock, you can rapidly adopt next month's breakthrough model without needing a database migration.

4. The "Lock-in" Economic Risk

Architectural decisions are ultimately about leverage.

The Oracle Cost Risk: If Oracle increases its "AI Option" license fee by 20% next year, you are trapped. Migrating a monolithic database containing your vectors, agents, and relational data is a multi-year, multi-million dollar project.
The AWS Composable Risk: If Amazon Bedrock becomes too expensive, you simply point your Lambda function to OpenAI, Anthropic, or a self-hosted Llama 3 model on an EC2 instance. Your database (Postgres) remains unchanged. You retain price leverage over your AI providers.

Summary Table: Estimated Monthly Spend (Mid-Sized App)

To put this in perspective, here is a rough look at the unit economics of a mid-sized production application running a monolithic proprietary stack vs. an open-source composable stack on AWS:

Component	Oracle 26ai	AWS Composable Stack
Database License	$2,000+ (Subscription)	$0 (Open Source)
Compute/Instance	$800 (Fixed)	$200 (Aurora Serverless avg)
AI Inference	Included in Compute	$100 (Token-based)
Orchestration	In-DB (Fixed)	$10 (Lambda/Step Functions)
Total Est. Monthly	$2,800/mo	$310/mo

Final Verdict: Beware the Gold-Plated Handcuffs

The AWS Architecture described in this blog is approximately 80-90% more cost-effective for new builds, startups, and scale-ups.

Oracle 26ai only becomes "cost-effective" when the cost of migrating away from an existing Oracle ecosystem exceeds the exorbitant licensing fees a situation often referred to in enterprise IT as the "Gold-Plated Handcuffs."

Oracle 26ai is an impressive piece of engineering designed to keep enterprise data exactly where it is. But for teams building the next generation of software, AI does not need to be a proprietary database feature.

By combining the rock-solid reliability of PostgreSQL with the raw power of AWS cloud primitives, you can build massively scalable, AI-native applications without ever sacrificing your budget or your architectural freedom.

Are you running your vector workloads inside PostgreSQL, or did you adopt a dedicated vector database? Let's discuss the tradeoffs in the comments below!

The 15-Millisecond AI: Building "Pre-Cognitive" Edge Caching on AWS

Dhananjay Lakkawar — Sun, 29 Mar 2026 19:17:07 +0000

If you want to watch a product manager's soul leave their body, sit in on a live demo of a Generative AI feature where the model takes 12 seconds to generate a response.

Typing... typing... typing...

In the world of AI product development, latency is the ultimate UX killer. You can have the smartest prompt and the most expensive foundational model in the world, but if your users have to stare at a spinning loading wheel for 10 seconds every time they click a button, they will abandon your app.

Most engineering teams try to solve this by streaming tokens to the frontend or switching to smaller, less capable models. But as a cloud architect, I prefer a different approach.

What if we stopped waiting for the user to ask the question?

What if we used the user's application state to predict what they are going to ask, generated the answer in the background, and pushed it to a CDN edge location before their mouse even hovers over the button?

When I sketch this out for engineering leaders, the reaction is almost always the same: "Wait, we can pre-generate AI responses in the background and cache them at the CDN level to completely bypass inference latency?"

Yes. Here is how to build a "Pre-Cognitive" AI architecture using AWS Step Functions, Amazon Bedrock, and Amazon CloudFront with Lambda@Edge.

The Concept: From Reactive AI to Proactive Caching

Think about your favorite SaaS dashboard. When a user logs in on Monday morning, their "next best actions" are highly predictable.

They are going to ask for a summary of weekend alerts.
They are going to ask for the status of their latest deployment.
They are going to ask for a draft reply to their most urgent ticket.

Instead of waiting for the user to click "Summarize Alerts" and forcing them to wait 8 seconds for an LLM to read the data, we move the LLM inference out of the synchronous request path and into an asynchronous background job.

We generate the responses, store them as key-value pairs, and push them to the network edge. When the user finally clicks the button, the response loads in 15 milliseconds. It feels like magic.

The Architecture: Phase 1 (Background Generation)

To make this work without slowing down the initial user login, we decouple the generation using an event-driven flow.

1. The Trigger: When the user logs in (or enters a specific workflow), your backend fires an event to AWS EventBridge.
2. The Orchestrator: AWS Step Functions takes over. It acts as the background traffic cop, ensuring your API doesn't hang.
3. The Inference: A Lambda function analyzes the user's state, grabs the required context, and fires off 3 concurrent prompts to Amazon Bedrock (using a fast, cheap model like Claude 3 Haiku).
4. The Edge Push: Once Bedrock returns the generated text, Lambda pushes these pre-computed AI responses into Amazon CloudFront KeyValueStore (a globally distributed datastore designed specifically for edge functions) keyed by UserID_ActionID.

The Architecture: Phase 2 (The 15ms Delivery)

Now, the user is looking at their dashboard. They see a button that says "✨ Generate Morning Briefing." They click it.

Because we are using CloudFront and Lambda@Edge (or CloudFront Functions), the request never even reaches your primary backend servers in us-east-1.

1. The Interception: The user's HTTPS request hits the closest AWS Edge location (e.g., a server in London, Tokyo, or New York). Lambda@Edge intercepts the request.
2. The Edge Lookup: Lambda@Edge checks the attached CloudFront KeyValueStore for the user's pre-generated response.
3. Instant Delivery: If the response is there, it is returned instantly. The user experiences sub-20ms latency for a complex Generative AI task.
4. The Fallback: If the user asks a completely custom question that we didn't predict, Lambda@Edge simply forwards the request to your standard API Gateway/Bedrock backend to generate the response synchronously.

The CTO Perspective: Tradeoffs and Reality Checks

As a technology strategist, I will be the first to tell you that "magic" always comes with an engineering invoice. You should only use this pattern if you understand the tradeoffs.

1. The Cost of Wasted Compute

By predicting 3 things the user might ask, you are generating tokens that might never be read. You are trading compute cost for user experience.
The Mitigation: Only use this pattern with ultra-cheap, highly efficient models like Claude 3 Haiku or Llama 3 8B. Do not use Claude 3 Opus or GPT-4o for speculative background generation, or you will torch your AWS bill.

2. State Invalidation

What happens if you pre-generate a "Deployment Summary" at 9:00 AM, but at 9:05 AM a deployment fails, and the user clicks the button at 9:06 AM? The cached AI response is now lying to them.
The Mitigation: Tie your cache invalidation to your application's critical state changes. If a critical DB row updates, fire an EventBridge rule that immediately deletes the stale key from the CloudFront KeyValueStore.

3. Build Complexity vs. Product Value

Don't build this for a general-purpose chatbox. Humans are too unpredictable. Build this for highly structured, high-value UX checkpoints—like daily briefings, code review summaries, or personalized dashboard greetings.

The Bottom Line

When we build AI applications, we often forget that the rules of distributed systems still apply. You don't have to accept the latency of a foundational model as a fixed constraint.

By aggressively predicting user intent and leveraging AWS edge networking primitives like CloudFront and Lambda@Edge, you can completely mask LLM latency.

It takes your application from feeling like a "cool AI wrapper" to feeling like a deeply integrated, hyper-responsive superpower.

Have you struggled with GenAI latency in your production applications? Are you using streaming, or have you started exploring asynchronous generation? Let me know your architecture in the comments below.

The $50,000 Chat History Problem: Building Event-Driven AI Memory on AWS

Dhananjay Lakkawar — Fri, 27 Mar 2026 17:46:37 +0000

It was 11:00 PM on a Tuesday when my friend startup's CTO dropped a screenshot of their monthly cloud bill into the engineering Slack channel.

The AWS infrastructure costs were flat. But their LLM inference API bill looked like a hockey stick pointing straight up.

"Why are we burning thousands of dollars a day on Claude 3 Opus?" she asked.

The lead engineer replied: "Because to make the AI assistant feel 'smart' and remember the user, we have to pass their entire conversation history into the context window for every single message. If they've been using the app for a month, we are passing 80,000 tokens just so the bot remembers their dog's name when they say 'hello'."

They had fallen into the classic Generative AI trap: Treating the LLM's context window as a database.

As a cloud architect, I love when we can take "boring" cloud primitives and combine them with AI to create something that feels like magic but is actually just brilliant, highly-scalable engineering. If you want to make a CTO stop in their tracks, rethink their architecture, and say, "Wait, is this actually possible?" you need to move away from standard chatbots.

Here is an architectural pivot that radically changes how an AI application scales, operates, and spends money: Event-Driven AI Memory using AWS EventBridge.

The Pivot: From Context Windows to a "Neural Memory Bus"

The traditional approach to AI memory is brute force: stuff conversational history into giant, expensive LLM context windows, or build complex Retrieval-Augmented Generation (RAG) pipelines over raw chat logs.

Both approaches are slow, expensive, and prone to losing important details in the noise.

Instead of keeping a running transcript of everything the user has ever said, what if we decoupled "memory" from the "chat interface" entirely? What if we treated user actions as asynchronous events?

The Architecture: Building the "Fact Store"

We can achieve this by combining AWS EventBridge, AWS Lambda, Amazon DynamoDB, and a hyper fast, cheap LLM like Claude 3 Haiku via Amazon Bedrock.

Here is how the event-driven memory pipeline works:

Step 1: The Event Bus
Route every user action in your app not just chat messages, but button clicks, page views, and settings changes through AWS EventBridge as standard JSON events.

Step 2: The Memory Extractor (Async)
Have a lightweight AWS Lambda function subscribe to these events. When an event fires, the Lambda function passes the event payload to a fast, cheap model like Claude Haiku.

The system prompt is simple: "You are a background observer. Review this user event. Extract any permanent, highly relevant facts about this user. Output as a JSON array. If nothing is relevant, return an empty array."

Step 3: The Fact Store (DynamoDB)
If Haiku detects a fact (e.g., User is building a SaaS, User prefers Python, User operates in the EU), the Lambda function upserts that key-value pair into an Amazon DynamoDB table keyed by the UserID. This is your "Fact Store" a living, breathing profile of the user.

The "Aha!" Moment: Querying the AI

Now, let's go back to that expensive chat interface.

When the user asks a complex question, you do not query a massive chat history. You don't pass 80,000 tokens of past transcripts.

Instead, your backend does a sub-millisecond GetItem lookup against DynamoDB for that user's Fact Profile. You take those concentrated facts and inject them into the system prompt of your heavy-lifting model (like Claude 3.5 Sonnet or Opus).

The CTO’s Reaction: Why This Pattern Wins

When you explain this architecture to engineering leaders, the reaction is almost always the same: "Wait, we can use EventBridge as a global 'neural memory bus' for our AI?"

Yes. And here is why this tradeoff makes sense for scaling startups:

1. Massive Cost Reduction

You are swapping synchronous, high-token inference on your most expensive model for asynchronous, low-token inference on your cheapest model. A 1,000-token prompt to Claude Haiku costs fractions of a cent. Querying a DynamoDB table costs practically nothing. You drop your token consumption by 90%.

2. Infinite Scale and Speed

DynamoDB delivers single-digit millisecond performance at any scale. Because you are only injecting a condensed JSON object of "Facts" into your final chat prompt, your time-to-first-token (TTFT) drops drastically. The AI responds faster because it has less text to read.

3. Omnichannel Intelligence

Because the memory is tied to EventBridge not the chat window the AI learns from the user's actions, not just their words. If a user struggles with a dashboard and triggers three "Error 500" events, the Fact Store updates. When they finally open the support chatbot, the AI already knows they are frustrated and exactly which error they hit.

The Bottom Line

We need to stop treating Large Language Models as databases. They are reasoning engines.

By leveraging standard, highly scalable cloud primitives like AWS EventBridge and DynamoDB, we can offload the burden of memory from the LLM context window into actual infrastructure.

It feels like AI magic to the user, but under the hood? It’s just brilliant, boring, beautiful engineering.

Have you hit the "context window cost wall" in your generative AI applications yet? Let me know in the comments how your team is managing AI memory at scale.

# Treating Prompts Like Code: Building CI/CD for LLM Workflows on AWS

Dhananjay Lakkawar — Tue, 24 Mar 2026 14:31:00 +0000

If you look at the codebase of an early-stage AI startup, you will almost always find a file named utils.py or constants.js containing massive blocks of hardcoded text.

These are the LLM system prompts.

When a model hallucination occurs in production, a developer goes into the code, tweaks a few sentences in the prompt, runs a quick manual test, and pushes the change to production.

This works for prototypes, but for production systems, this is a massive operational risk.

"Prompt drift" is real. A small change designed to fix an edge case can unintentionally break the formatting, tone, or logic for dozens of other use cases. If you want to build reliable AI systems, you have to stop treating prompts like magical incantations and start treating them like code.

Here is how a modern engineering team architects an automated, version-controlled CI/CD pipeline for LLM prompts using GitHub Actions, AWS CodePipeline, and AWS Systems Manager (SSM) Parameter Store.

The Core Problem: Tightly Coupled AI

When you hardcode prompts into your application logic (e.g., inside an AWS Lambda function), you tightly couple your application release cycle with your AI tuning cycle.

To fix a typo in a prompt, you have to redeploy the entire application.
You have no historical record of why a prompt changed and how it affected output quality.
You have no automated gate preventing a "bad" prompt from reaching production.

The solution is to decouple the prompt from the code, version it in Git, evaluate it automatically, and inject it at runtime.

The Serverless Prompt Pipeline Architecture

To bring engineering rigor to our AI workflows, we need three distinct layers: Storage, Evaluation, and Runtime Injection.

1. The Git & Evaluation Flow

Instead of hardcoding strings, developers maintain a prompts.json or prompts.yaml file in their repository. When a pull request is opened, it triggers an evaluation pipeline.

2. Runtime Injection (AWS SSM Parameter Store)

Once the CI/CD pipeline validates that the new prompt doesn't break existing functionality, it uses the AWS CLI/SDK to push the updated prompt string into AWS SSM Parameter Store (e.g., under the path /prod/llm/customer_service_prompt).

When your application (running on AWS Lambda, ECS, or EKS) is invoked, it dynamically fetches the prompt from SSM.

The CTO Perspective: Why Architect It This Way?

Building this pipeline requires upfront engineering effort. Here is why it is worth it for scaling teams:

1. Zero-Downtime Prompt Updates

Because the Lambda function fetches the prompt from SSM at runtime, your product managers or AI engineers can deploy prompt improvements instantly without requiring a full backend deployment or passing through a lengthy code build process.

2. Guarding Against Regression

The "Automated Evaluation Gate" is the most critical piece of this architecture. You maintain a "Golden Dataset" of 50-100 real user inputs and expected outputs.
During the CI phase, you run the proposed prompt against this dataset using an "LLM-as-a-judge" pattern. If the new prompt causes the model to start hallucinating or dropping required JSON keys, the pipeline fails the build automatically.

3. Auditability and Rollbacks

Because SSM Parameter Store supports versioning, you get an automatic audit trail. If Version 14 of your prompt causes issues in production, rolling back is simply a matter of reverting to Version 13 via the AWS Console or CLI.

Engineering Tradeoffs & Best Practices

If you implement this architecture tomorrow, keep these real-world constraints in mind:

SSM API Limits: AWS SSM Parameter Store has API rate limits. If you have a high-traffic API (e.g., hundreds of requests per second), fetching the prompt from SSM on every single invocation will result in ThrottlingException errors.
- The Fix: Implement caching inside your Lambda execution environment (e.g., caching the prompt in memory outside the handler function for 5 minutes), or use AWS AppConfig, which is explicitly designed for high-throughput dynamic configuration.
Evaluation Costs: Running 100 tests through Claude 3.5 Sonnet on every single Git commit will spike your Amazon Bedrock bill.
- The Fix: Run the full evaluation suite only on merges to the main branch, or use a smaller, cheaper model (like Claude 3 Haiku) to run quick sanity checks on feature branches.
String Limits: Standard SSM parameters have a 4KB size limit. If you are using massive few-shot prompts with thousands of tokens, you will need to use the Advanced Parameter tier (up to 8KB) or store the prompt in an S3 bucket and store the S3 URI in SSM.

The Bottom Line

Generative AI is shifting from an experimental feature to a core architectural component of modern applications. If you wouldn't deploy database schema changes without testing and version control, you shouldn't deploy prompt changes without them either.

By combining GitOps, AWS CodePipeline, and SSM Parameter Store, you bridge the gap between AI experimentation and reliable software engineering.

How does your team currently manage LLM prompts? Are they hardcoded, stored in a database, or managed via an external tool? Let's discuss in the comments.

Routing LLM Traffic on AWS: How to Build a Cost-Optimized Multi-Model API Router

Dhananjay Lakkawar — Wed, 18 Mar 2026 11:05:06 +0000

When engineering teams first integrate Generative AI into their products, they usually make a rational, but ultimately expensive, decision: they pick the smartest model available and send every single query to it.

Using Claude 3 Opus or GPT-4o for everything is the fastest way to get to market. But as your user base grows, your inference costs will scale linearly or worse, exponentially, if your context windows are expanding.

The reality of production AI is this: You don't need a PhD-level reasoning engine to summarize a 3-paragraph email. Claude 3 Haiku or Llama 3 can handle 80% of standard production workloads at a fraction of the cost and with much lower latency.

To protect your startup's runway and optimize your cloud economics, you need to stop hardcoding a single LLM into your backend. Instead, you need to build a Multi-Model API Router.

Here is how to architect a dynamic LLM router using Amazon API Gateway, AWS Lambda, and Amazon Bedrock to reduce your inference costs by up to 60%.

The Concept: Dynamic Prompt Routing

Think of an LLM router like an API load balancer, but instead of routing based on server capacity, it routes based on cognitive complexity.

When a prompt arrives, a lightweight heuristic evaluates the request. Simple tasks (summarization, formatting, basic entity extraction) slide down a "green pipe" to a fast, cheap model. Complex reasoning tasks (coding, deep analysis, complex multi-step logic) slide down a "purple pipe" to a high-end model.

The AWS Architecture

We can build this entirely using primitives on AWS. Because Amazon Bedrock acts as a unified API for multiple foundation models, we don't have to manage different API keys or deal with diverse SDKs for Claude, Llama, or Mistral. Bedrock normalizes the invocation.

Here is the underlying AWS infrastructure:

1. Amazon API Gateway (The Entry Point)

We use API Gateway to expose a unified REST or WebSocket API to our front end. The front end doesn't know which model is being used; it simply sends the payload to /api/v1/generate.

2. AWS Lambda (The Routing Engine)

This is where the brain of your application lives. The Lambda function receives the payload and applies a set of routing rules to determine the destination.

3. Amazon Bedrock (The Execution Layer)

Based on Lambda's decision, it uses the AWS SDK (boto3 in Python or the AWS SDK for Node.js) to invoke the specific Bedrock model ARN.

3 Strategies for Building the Router Logic

How exactly does the Lambda function know where to send the prompt? There are three ways to approach this, ranging from simple to advanced.

Strategy A: Deterministic Heuristics (Fastest & Cheapest)

You don't always need AI to route AI. You can use standard code logic.

System Prompts: If the user is hitting the "Summarize" button in your UI, your frontend passes a task_type="summarize" flag. Lambda reads the flag and instantly routes to Haiku.
Token Count: If the prompt length is under 500 tokens, send it to a smaller model. If it's a massive 50k-token document, route it to a model with a larger, highly-capable context window like Claude 3.5 Sonnet.

Strategy B: The "LLM-as-a-Judge" Router

For unstructured user inputs (like a chatbot), use a fast, ultra-cheap model (like Haiku) to read the prompt and classify its intent.

Prompt to Haiku: "Is the following user request a basic factual question (Return 1) or a complex reasoning task (Return 2)?"
Lambda reads the 1 or 2 and routes the actual query accordingly. (Note: This adds a slight latency overhead, usually ~200-400ms).

Strategy C: The Cascading Fallback (Highest Reliability)

If you want to maximize cost savings while guaranteeing high quality, you implement a Cascade. You send the prompt to a cheap model first. If the cheap model fails, hallucinates, or outputs bad JSON, Lambda catches the error and retries with the expensive model.

The CTO Perspective: Tradeoffs to Consider

As a technology strategist, I always emphasize that architectural decisions are about balancing tradeoffs. A Multi-Model Router is not a silver bullet.

1. Latency vs. Cost If you use LLM-based routing (Strategy B) or Cascading (Strategy C), you are introducing multiple network hops and inference cycles. For an internal tool or asynchronous data processing, this latency is fine. For a real-time conversational voice bot, adding 500ms of routing latency will ruin the user experience. Choose deterministic heuristics (Strategy A) for real-time apps.

2. Maintenance Complexity Prompt engineering is hard enough for one model. When you route across three different models (e.g., Claude, Llama, and Amazon Titan), you must maintain different system prompts optimized for each model's specific quirks. Bedrock's Converse API makes standardizing the payload easier, but the prompt wording still requires tuning per model.

3. Build vs. Buy There are specialized third-party tools (like Portkey or Langfuse) that handle LLM routing as a managed service. However, building this inside AWS via API Gateway and Lambda keeps your data entirely within your VPC and avoids adding another vendor to your billing stack. For most startups, a 150-line Lambda function is perfectly sufficient for the first year of scale.

The Bottom Line

Scaling an AI product doesn't mean your AWS bill has to scale at the exact same rate. By treating LLMs as interchangeable utility endpoints rather than monolithic brains, you can ruthlessly optimize your unit economics.

Route the heavy lifting to the expensive models, let the cheap models handle the busywork, and let AWS handle the infrastructure.

The full Lambda implementation with both strategies, the fallback chain, and task type buckets is below copy it, drop it into your Lambda function, wire up API Gateway, and you're routing.

AWS Lamda Code

How is your team handling LLM costs in production? Are you defaulting to the largest models, or have you started implementing routing architectures? Let me know in the comments!

Stop Overpaying for VectorDBs: Architecting Serverless RAG on AWS

Dhananjay Lakkawar — Mon, 16 Mar 2026 13:26:09 +0000

Building a Retrieval-Augmented Generation (RAG) prototype takes a weekend. Taking that prototype to production without burning through your infrastructure budget is a completely different engineering challenge.

One of the most common pitfalls I see founders and engineering teams fall into is the Vector Database Cost Trap.

To get their MVP out the door, teams spin up provisioned vector databases or run dedicated EC2 instances 24/7. It works brilliantly for the first 100 users. But as you scale or worse, when traffic is unpredictable paying for idle compute to keep a vector index in memory becomes a massive drain on your runway.

If you want to build a highly scalable AI product while protecting your startup's runway, you need to shift from provisioned infrastructure to an event-driven, serverless architecture.

The Shift: Serverless RAG

Traditional RAG architecture requires you to provision database nodes, manage cluster scaling, and pay for peak capacity even at 3 AM.

By moving to a serverless model, we separate the storage of our vectors from the compute required to query them, and we rely on AWS to scale the ingestion and retrieval layers on demand.

1. The Ingestion Pipeline

Trigger (Amazon S3): A new document (PDF, TXT, JSON) is dropped into an S3 bucket.
Compute (AWS Lambda): An S3 event triggers a Lambda function to chunk the text.
Embedding (Amazon Bedrock): Lambda calls Bedrock (e.g., Titan Embeddings) to convert text to vectors.
Indexing (Amazon OpenSearch Serverless): Lambda writes the vectors/metadata into an OpenSearch Serverless Vector Search collection.

2. The Retrieval Flow

User Query: Arrives via API Gateway.
Embed Query: Lambda calls Bedrock to embed the search string.
Similarity Search: Lambda queries OpenSearch Serverless (k-NN) to find relevant chunks.
Generation: Lambda sends the context + prompt to an LLM (e.g., Claude 3.5 Sonnet) via Bedrock.

Why This Works for Startups

Zero Infrastructure Management: No patching nodes or managing shards.
Event-Driven: The pipeline only runs when a document arrives. Zero ingestion = zero cost.
Decoupled Scaling: If a user uploads 10,000 documents, Lambda fans out to process them concurrently without impacting search performance.

A CTO's Perspective: The Economics

You could build your own vector index using pgvector on RDS. If your dataset is tiny, that works. But if search latency and scale are critical, a dedicated vector engine is necessary.

With OpenSearch Serverless, AWS recently lowered the minimum capacity to 0.5 OCUs (OpenSearch Compute Units). This brings the base cost of a highly available, scalable vector database down to a startup-friendly level, with the peace of mind that it will auto-scale if your app goes viral.

The Tradeoffs (Know Before You Build)

As an architect, I don't believe in silver bullets. Design for these constraints:

Cold Starts: If your RAG app requires sub-second latency for the first request after inactivity, you may need Lambda Provisioned Concurrency.
Scaling Lag: OpenSearch Serverless auto-scales, but it isn't instantaneous for massive, sudden spikes. Configure your max OCUs properly and load test your scaling behavior.
Vendor Lock-in: You are utilizing AWS primitives. However, because you are using standard frameworks (HTTP requests to Bedrock and standard OpenSearch APIs), migrating your application logic later is feasible.

Final Thoughts

The era of overpaying for oversized, underutilized vector databases just to validate an AI product is over. By leveraging Amazon Bedrock, Lambda, and OpenSearch Serverless, you can build an enterprise-grade, event-driven AI architecture from Day 1.

I originally published this on my Hashnode blog: HASHNODE_LINK

Have you made the switch to serverless vector databases yet? Let me know your experience with cold starts and latency in the comments!