Forem: Venkat Ambati

Memory Palace Part 2: Agentic RAG, Chrome Extension, and Making AI Actually Understand You 🧠✨

Venkat Ambati — Fri, 02 Jan 2026 12:36:11 +0000

From "dumb search" to intelligent reasoning — plus save anything with one click

Previously on Memory Palace...

A few weeks ago, I shared how I built Memory Palace — a RAG-powered knowledge management system that handles both external research (Pockets) and personal thoughts (Memories).

The feedback was amazing. But two things kept coming up:

"This is great, but sometimes the answers don't quite get what I'm asking..."

"I don't want to copy-paste URLs into a web app. Can I just... click a button?"

So I rebuilt the entire RAG pipeline. And built a Chrome Extension.

What's New in Part 2?

🧠 Agentic RAG Pipeline — AI That Actually Thinks

The biggest upgrade isn't visible — it's in how the system thinks. We went from "dumb retrieval" to a multi-step reasoning pipeline.

🔌 Chrome Extension — Save Anything With One Click

A full-featured browser extension that brings Memory Palace to every webpage.

Let's dive into both.

Part 1: The Agentic RAG Pipeline

The original RAG was simple:

Embed query → Vector search → Get chunks → Send to LLM → Done

It worked. But it was... dumb. It didn't understand intent. It couldn't tell if you were asking a follow-up question. It treated "hello" the same as "compare the methodologies in my research papers."

The New Pipeline: 7 Steps of Intelligence

┌─────────────────────────────────────────────────────────────────────────────┐
│                        AGENTIC RAG PIPELINE                                 │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│   ┌─────────────┐     ┌─────────────┐     ┌─────────────┐                   │
│   │  1. Query   │     │  2. Adaptive │     │  3. Context │                   │
│   │   Router    │────▶│  Retrieval   │────▶│   Rewrite   │                   │
│   │             │     │   Params     │     │             │                   │
│   └─────────────┘     └─────────────┘     └─────────────┘                   │
│         │                                       │                           │
│         ▼                                       ▼                           │
│   ┌─────────────┐     ┌─────────────┐     ┌─────────────┐                   │
│   │  Skip RAG?  │     │  4. Multi   │     │  5. Hybrid  │                   │
│   │  (Greeting) │     │   Query     │────▶│   Search    │                   │
│   └─────────────┘     │  Generation │     │             │                   │
│                       └─────────────┘     └─────────────┘                   │
│                                                 │                           │
│                                                 ▼                           │
│                       ┌─────────────┐     ┌─────────────┐                   │
│                       │  7. Answer  │     │  6. CRAG    │                   │
│                       │  Synthesis  │◀────│  Grading    │                   │
│                       │  + Stream   │     │             │                   │
│                       └─────────────┘     └─────────────┘                   │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Let me explain each step:

Step 1: Query Router — Intent Classification

Before doing anything, we ask: "What kind of question is this?"

type QueryIntent =
    | "no_retrieval" // "Hello!" - no sources needed
    | "simple_lookup" // "What is X?" - direct fact lookup
    | "comparison" // "Compare A and B" - needs multiple sources
    | "summarization" // "Summarize..." - needs aggregation
    | "analytical" // "Why does..." - deep reasoning required
    | "follow_up"; // "Tell me more" - needs conversation context

Why it matters: If someone says "Hi, how are you?", we don't need to search 1000 chunks. We can respond directly.

if (routerResult.skipRetrieval) {
    // Just respond, no RAG needed
    sendEvent({ type: "token", payload: "Hello! How can I help you today?" });
    return;
}

Step 2: Adaptive Retrieval Parameters

Different questions need different retrieval strategies:

function getAdaptiveRetrievalParams(
    intent: QueryIntent
): AdaptiveRetrievalParams {
    switch (intent) {
        case "comparison":
            return {
                chunkCount: 20, // Need more chunks for comparison
                vectorWeight: 0.5, // Balance semantic + keyword
                ftsWeight: 0.5,
                expansionQueries: 5, // Generate more query variations
            };

        case "simple_lookup":
            return {
                chunkCount: 5, // Few chunks, high precision
                vectorWeight: 0.7, // Lean into semantic
                ftsWeight: 0.3,
                expansionQueries: 2,
            };

        case "analytical":
            return {
                chunkCount: 15, // Lots of context for analysis
                vectorWeight: 0.6,
                ftsWeight: 0.4,
                expansionQueries: 4,
            };
        // ...
    }
}

The insight: "Compare Apple and Google's AI strategy" needs way more chunks than "What is Apple's market cap?"

Step 3: Context-Aware Query Rewriting

Follow-up questions are the hardest. When you ask "What about their revenue?", what does "their" mean?

We rewrite ambiguous queries using conversation history:

const rewrittenQuery = await rewriteQueryWithContext(
    "What about their revenue?", // Original query
    [
        { role: "user", content: "Compare Apple and Google AI" },
        { role: "assistant", content: "Apple focuses on..." },
    ]
);

// Result:
// {
//   original: "What about their revenue?",
//   rewritten: "What is Apple and Google's revenue?",
//   extractedEntities: ["Apple", "Google", "revenue"],
//   needsContext: true
// }

Now the search actually finds what you meant.

Step 4: Multi-Query Generation

One query isn't enough. We generate variations to catch different phrasings in your sources:

User asks: "What are the risks of AI?"

We search for:

"What are the risks of AI?"
"AI dangers and downsides"
"Negative impacts of artificial intelligence"
"AI safety concerns"
"Problems with AI adoption"

const searchQueries = await generateSearchQueriesStream(
    effectiveQuery,
    retrievalParams.expansionQueries // 2-5 queries based on intent
);

Result: 40% better recall on average. We find chunks that matter even if they don't use your exact words.

Step 5: Hybrid Search

Vector search is great for semantics. But sometimes you need exact matches.

We combine both:

-- Hybrid search: vector + full-text with adaptive weights
SELECT
  chunk_id,
  text,
  (
    ({vectorWeight} * (1 - (embedding <=> query_embedding))) +
    ({ftsWeight} * ts_rank(search_vector, websearch_to_tsquery(query)))
  ) as combined_score
FROM chunks
ORDER BY combined_score DESC
LIMIT {chunkCount};

Example: If you search for "NVIDIA earnings Q3", vector search finds semantically similar chunks. Full-text search finds chunks with those exact words. Combined = best results.

Step 6: CRAG — Corrective RAG (Chunk Grading)

Here's the innovation: not all retrieved chunks are relevant.

Before sending chunks to the LLM, we grade each one:

interface GradedChunk {
    chunk: any;
    relevance: "relevant" | "partially_relevant" | "irrelevant";
    score: number; // 0-1
    reasoning: string;
}

const cragResult = await gradeChunksRelevance(query, chunks);

// Result:
// {
//   decision: 'sufficient' | 'needs_expansion' | 'no_relevant_sources',
//   avgRelevanceScore: 0.73,
//   relevantChunks: [...],  // Only the good ones
// }

Three outcomes:

sufficient — Good chunks found, proceed to answer
needs_expansion — Chunks are borderline, try broader search
no_relevant_sources — Nothing relevant, tell user honestly

Why this matters: Without CRAG, the LLM gets noisy context and hallucinates. With CRAG, it only sees relevant chunks.

Step 7: Answer Synthesis with Streaming

Finally, we generate the answer with real-time streaming:

// Stream tokens as they're generated
for await (const token of chatGen) {
    sendEvent({ type: "token", payload: token });
}

// Include citations
sendEvent({
    type: "done",
    payload: {
        answer,
        citations: sources.map((s) => ({ id: s.source_id, title: s.title })),
        intent: routerResult.intent,
    },
});

Real-time status updates throughout:

[Status] Analyzing query intent...
[Routing] Intent: comparison, Confidence: 0.89
[Status] Rewriting query with context...
[Rewriting] "What about their approach?" → "What is Apple and Google's approach to AI?"
[Status] Generating 4 search queries...
[Queries] ["Apple Google AI approach", "tech giants AI strategy", ...]
[Status] Searching 4 queries...
[Status] Grading chunk relevance...
[Grading] Decision: sufficient, Avg Score: 0.78, 12/15 chunks relevant
[Sources] Found 3 relevant sources
[Status] Generating answer...
[Token] Apple...
[Token] 's approach...

Part 2: The Chrome Extension

Now onto the second major feature: save anything from anywhere.

Features

One-click save — Save any webpage as a published memory instantly
Built-in chat — Ask questions about your memories without leaving the page
Smart extraction — Pulls full content from any website
Secure login — Uses the same Supabase auth as the web app

The Architecture

┌─────────────────────────────────────────────────────────────────┐
│                     Your Browser                                │
│  ┌─────────────────┐     ┌─────────────────┐                    │
│  │   Popup UI      │     │  Content Script │                    │
│  │  - Login        │     │  - Extracts DOM │                    │
│  │  - Chat         │     │  - Full content │                    │
│  │  - Save button  │     │  - No limits    │                    │
│  └────────┬────────┘     └────────┬────────┘                    │
│           │                       │                              │
│           ▼                       ▼                              │
│  ┌─────────────────────────────────────────┐                    │
│  │         Background Service Worker       │                    │
│  └────────────────────┬────────────────────┘                    │
│                       │                                          │
└───────────────────────┼──────────────────────────────────────────┘
                        │
                        ▼
          ┌─────────────────────────┐
          │    Memory Palace API    │
          │      (Railway)          │
          └─────────────────────────┘

The Game Changer: Unlike the worker which fetches URLs and parses HTML, the extension runs directly in your browser with full DOM access:

No content limits — Get the entire article, not just 50KB
JavaScript-rendered content — Works on SPAs and dynamic sites
Bypasses bot detection — You're a real browser, not a scraper
Sees what you see — If you can read it, you can save it

Smart Content Extraction

We don't just grab document.body.innerText. We intelligently extract content:

Platform-specific selectors:

const mainSelectors = [
    // Medium
    "article[data-testid='post']",
    ".meteredContent",

    // Substack
    ".post-content",
    ".available-content",

    // WordPress
    ".entry-content",
    ".article-body",

    // News sites
    ".story-body",
    ".article__body",

    // Dev blogs
    ".markdown-body",
    ".prose",

    // Generic fallbacks
    "article",
    "main",
    '[role="main"]',
];

Find the best element (most content):

let mainElement = null;
let maxContentLength = 0;

for (const selector of mainSelectors) {
    const element = document.querySelector(selector);
    if (element && element.innerText.length > maxContentLength) {
        mainElement = element;
        maxContentLength = element.innerText.length;
    }
}

Aggressive cleanup:

const removeSelectors = [
    "script",
    "style",
    "noscript",
    "iframe",
    "nav",
    "header",
    "footer",
    ".sidebar",
    ".comments",
    ".advertisement",
    ".ad",
    ".social-share",
    ".related-posts",
    ".newsletter",
    ".cookie-notice",
    "button",
    "form",
];

Structure-preserving extraction:

const walkNode = (node) => {
    if (node.nodeType === Node.TEXT_NODE) {
        content += node.textContent;
    } else if (node.nodeType === Node.ELEMENT_NODE) {
        if (["p", "div", "h1", "h2", "li"].includes(tag)) content += "\n";
        if (["h1", "h2", "h3"].includes(tag)) content += "\n## ";
        if (tag === "li") content += "• ";

        for (const child of node.childNodes) walkNode(child);
    }
};

Result: Clean, formatted text with headings and bullet points preserved.

Auto-Publish: Skip the Draft Stage

Previously: Create draft → Review → Publish → Chunk → Embed

Now: Save from extension → Immediately published and searchable

// Extension sends:
body: JSON.stringify({
    org_id: orgId,
    title,
    content,
    status: "published", // Skip the draft!
});

// API automatically queues for chunking:
if (body.status === "published") {
    await queue.add("chunk-memory", { memoryId: memory.id });
}

Result: Save an article → Immediately searchable in chat. No extra clicks.

GitHub Repositories

vedha-pocket-extension — Chrome Extension for one-click saves
👉 https://github.com/venki0552/vedha-pocket-extension ← NEW!

The Funny Bits (More Lessons Learned)

1. The "Why Is Everything Relevant?" Disaster

First version of CRAG graded everything as "relevant" because the prompt was too lenient. LLM was like "well, it could be related..."

Fix: Added strict grading criteria and asked for reasoning before the score.

2. The Query Router That Said "Hello" To Everything

Intent classification kept detecting greetings in legitimate questions because the prompt prioritized "be friendly."

Before: "Hello, can you compare the AI strategies?" → intent: 'no_retrieval'

After: Only no_retrieval for actual greetings with no substantive question.

3. The Timeout Cascade

Agentic RAG has 7 LLM calls. At 10 seconds each, that's 70 seconds worst case. Original code had no timeouts. Users waited. Forever.

Fix: 10-second timeout per step, graceful fallbacks:

const routerResult = await fetchWithTimeout(
    url,
    options,
    LLM_TIMEOUT_MS // 10 seconds max
);

4. The "Multi-Query Made It Worse" Mystery

More queries = more results. But more results = more noise. CRAG was filtering out 90% of chunks.

Insight: Generate fewer, better queries. 3-5 is the sweet spot.

5. The "Why Is It All On One Line" Formatting Bug

Extension content extraction used:

content = content.replace(/\s+/g, " ");

Except \s+ matches newlines too. Every article became one giant paragraph.

Fix:

content = content.replace(/ +/g, " "); // Only spaces, preserve newlines

Updated Full Architecture

┌─────────────────┐
│                 │
│Chrome Extension │─────────────────────────────┐
│  - Save pages   │                             │
│  - Chat         │                             │
│                 │                             │
└─────────────────┘                             │
                                                ▼
┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│                 │     │                 │     │                 │
│   Next.js Web   │────▶│   Fastify API   │────▶│   BullMQ Worker │
│   (Vercel)      │     │   (Railway)     │     │   (Railway)     │
│                 │     │  + Agentic RAG  │     │                 │
│                 │     │                 │     │                 │
└─────────────────┘     └────────┬────────┘     └────────┬────────┘
                                 │                       │
                                 ▼                       ▼
                        ┌─────────────────┐     ┌─────────────────┐
                        │                 │     │                 │
                        │    Supabase     │     │    OpenRouter   │
                        │  (PostgreSQL +  │     │   (LLM + Embed) │
                        │    pgvector)    │     │                 │
                        └─────────────────┘     └─────────────────┘

What's Next?

Completed ✅

[x] Agentic RAG Pipeline — Query routing, CRAG, adaptive retrieval
[x] Chrome Extension — Save pages with one click
[x] Auto-publish from extension
[x] Real-time status streaming

Coming Soon 🚀

[ ] Self-reflective answer grading — Retry if answer has hallucinations
[ ] Firefox Extension
[ ] Keyboard shortcuts — Ctrl+Shift+S to save
[ ] Offline queue — Save when offline, sync when online

Try It Yourself

Web App: https://vedha-pocket-web.vercel.app
Extension: Clone from GitHub
API: https://vedha-api-production.up.railway.app

All open source. All self-hostable.

Final Thoughts

Part 1 was about making RAG work. Part 2 is about making it smart.

The difference between "good enough" and "actually useful" is in the details:

Understanding what kind of question you're asking
Knowing when retrieval isn't needed
Filtering noise before it reaches the LLM
Meeting users where they are (in the browser)

The biggest insight: RAG isn't one thing. It's a pipeline. And every step in that pipeline is an opportunity to add intelligence.

Next up: making the system learn from feedback. If you downvote an answer, it should remember why.

Built with ❤️, even more ☕, and a deep appreciation for graceful timeouts.

— Venkat

Part 1: I Built a RAG-Powered Second Brain

GitHub Repos:

Tags: #AI #AgenticRAG #CRAG #ChromeExtension #OpenSource #Supabase #TypeScript #RAG #KnowledgeManagement

I Built a RAG-Powered “Second Brain” and Accidentally Created My Personal Research Assistant

Venkat Ambati — Wed, 31 Dec 2025 08:18:58 +0000

The Problem: Information Overload Is Real

You know that feeling when you're in a meeting and someone asks, "Didn't we read something about that last month?" And you're like... "Yeah... somewhere... maybe... it was in a PDF... or a URL... or perhaps I dreamed it?"

Or when you have a brilliant insight in the shower, promise yourself you'll remember it, and by the time you're dressed, it's gone forever?

That's been my life for years. I have bookmarks dating back to 2015 that I'll "definitely read later," and random thoughts scattered across 17 different note apps.

So I built Memory Palace — a personal knowledge management system that handles BOTH your external research (PDFs, articles, documents) AND your internal thoughts (notes, ideas, memories). Think of it as NotebookLM meets Obsidian, but open source, self-hostable, and powered by whatever LLM you want (local or cloud).

What Is Memory Palace?

Memory Palace is a RAG (Retrieval-Augmented Generation) powered knowledge management system with a dual brain:

📚 Pockets (External Knowledge):

Save external sources — URLs, PDFs, Word docs, text files
Auto-processing — Extracts text, chunks it, creates embeddings
Ask research questions — "What did that paper say about momentum investing?"
Get cited answers — No hallucinations, just facts from YOUR sources

🧠 Memories (Internal Knowledge):

Save personal thoughts — Notes, ideas, insights, reflections
Rich formatting — Markdown support with colors and tags
Ask about your thinking — "What was my take on that marketing strategy?"
Connect the dots — AI helps link your thoughts across time

It's like having both a research assistant who has actually read all your documents AND a personal memory that never forgets what you were thinking.

Real-World Use Cases

For Research (Pockets):

Researchers: Save papers, ask "Compare the methodologies across my sources"
Students: Upload course materials, ask "Explain quantum entanglement using my lecture notes"
Professionals: Save industry reports, ask "What are the key trends from Q3 reports?"
Content Creators: Save research, ask "Give me statistics about AI adoption"

For Personal Knowledge (Memories):

Entrepreneurs: Save business ideas, ask "What were my thoughts on the subscription model?"
Writers: Save story concepts, ask "Find all my character development notes"
Learners: Save insights, ask "What did I learn from that productivity experiment?"
Decision Makers: Save pros/cons lists, ask "Why did I decide against that opportunity?"
Everyone: Finally remember those shower thoughts and 3am insights!

Combined Power:

Ask "How does this research paper relate to my previous thoughts on AI ethics?"
Connect external knowledge with personal insights seamlessly

The Tech Stack (For Nerds)

Here's what powers this beast:

Frontend: Next.js 14 + Tailwind CSS — Fast, pretty, server components
API: Fastify + TypeScript — Speed demon API framework
Worker: BullMQ + Playwright — Background processing with headless browser
Database: Supabase (PostgreSQL + pgvector) — Vector search + SQL = ❤️
LLM: OpenRouter (or local via Ollama) — Model flexibility
Hosting: Railway + Vercel — Deploy in minutes

Architecture Overview

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│                 │     │                 │     │                 │
│   Next.js Web   │────▶│   Fastify API   │────▶│   BullMQ Worker │
│   (Vercel)      │     │   (Railway)     │     │   (Railway)     │
│                 │     │                 │     │                 │
└─────────────────┘     └────────┬────────┘     └────────┬────────┘
                                 │                       │
                                 ▼                       ▼
                        ┌─────────────────┐     ┌─────────────────┐
                        │                 │     │                 │
                        │    Supabase     │     │    OpenRouter   │
                        │  (PostgreSQL +  │     │   (LLM + Embed) │
                        │    pgvector)    │     │                 │
                        └─────────────────┘     └─────────────────┘

GitHub Repositories

Here are the three repos that make up Memory Palace:

vedha-pocket-web — Next.js frontend with dual chat UI (pockets + memories)
👉 https://github.com/venki0552/vedha-pocket-web

vedha-pocket-api — Fastify API with RAG pipeline for both document and memory search
👉 https://github.com/venki0552/vedha-pocket-api

vedha-pocket-worker — Background worker with Playwright for document processing
👉 https://github.com/venki0552/vedha-pocket-worker

Key Features

🔍 Hybrid Search (Vector + Full-Text)

Not just semantic search. We combine vector similarity with PostgreSQL full-text search for the best of both worlds. Works across both documents and memories.

🌊 Streaming Responses

Answers stream in real-time, token by token. No waiting 10 seconds staring at a spinner. Real-time status updates during search.

📚 Multi-Query RAG

The system generates multiple search queries from your question for better retrieval. Ask "What are the risks?" and it also searches for "dangers," "downsides," and "limitations."

🧠 Dual Knowledge Base

Pockets for external sources (PDFs, URLs) and Memories for personal notes. Ask questions across both or search each separately.

🎨 Rich Memory Management

Markdown rendering with bold, italics, lists, links
Color coding for visual organization
Tag system for categorization
Resizable interface - adjust memory/chat panel sizes

🤖 Bot Detection

The worker detects when websites block scrapers (CloudFlare, access denied, etc.) and fails gracefully instead of saving junk.

📄 PDF Page-Level Chunking

PDFs are chunked by page with page numbers preserved. Citations tell you exactly which page to check.

🔐 Multi-Tenant Security

Full Row Level Security (RLS) with organizations and memberships. Share pockets with your team. Complete privacy with encrypted API keys.

🏠 Local LLM Support

Set LLM_BASE_URL to your Ollama/LM Studio/vLLM instance and use local models. Full control over your AI.

⚡ Performance & UX

Fallback model support - automatic retry with different models
Free model defaults - uses Google Gemma for cost-effective operation
Vercel Analytics - privacy-focused performance tracking
Responsive design - works great on all devices

How to Run Locally

Prerequisites

Node.js 20+
Docker (for Redis)
Supabase account
OpenRouter API key (or local LLM)

1. Clone the Repos

git clone https://github.com/venki0552/vedha-pocket-web.git
git clone https://github.com/venki0552/vedha-pocket-api.git
git clone https://github.com/venki0552/vedha-pocket-worker.git

2. Start Redis

docker run -d --name redis -p 6379:6379 redis:alpine

3. Set Up Environment Variables

vedha-pocket-api/.env

PORT=3001
NODE_ENV=development

# Supabase
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_ANON_KEY=your-anon-key
SUPABASE_SERVICE_ROLE_KEY=your-service-role-key

# Redis
REDIS_URL=redis://localhost:6379

# OpenRouter
OPENROUTER_API_KEY=your-openrouter-key
OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
OPENROUTER_EMBED_MODEL=openai/text-embedding-3-large
OPENROUTER_CHAT_MODEL=anthropic/claude-sonnet-4

# For local LLM (optional)
# LLM_BASE_URL=http://localhost:11434/v1

vedha-pocket-worker/.env

SUPABASE_URL=https://your-project.supabase.co
SUPABASE_SERVICE_ROLE_KEY=your-service-role-key
REDIS_URL=redis://localhost:6379
OPENROUTER_API_KEY=your-openrouter-key
OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
OPENROUTER_EMBED_MODEL=openai/text-embedding-3-large
PLAYWRIGHT_ENABLED=true
WORKER_CONCURRENCY=5

vedha-pocket-web/.env.local

NEXT_PUBLIC_SUPABASE_URL=https://your-project.supabase.co
NEXT_PUBLIC_SUPABASE_ANON_KEY=your-anon-key
NEXT_PUBLIC_API_URL=http://localhost:3001

4. Install Dependencies & Run

# Terminal 1 - API
cd vedha-pocket-api
npm install
npm run dev

# Terminal 2 - Worker
cd vedha-pocket-worker
npm install
npx playwright install chromium
npm run dev

# Terminal 3 - Web
cd vedha-pocket-web
npm install
npm run dev

5. Open the App

Visit http://localhost:3000 — Sign up, create a pocket, add some sources!

Deployment Guide

Step 1: Supabase Setup

Create a new project at supabase.com
Enable the Vector extension:

CREATE EXTENSION IF NOT EXISTS vector;

Run the database migrations. Here's the core schema:

-- Organizations
CREATE TABLE orgs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name TEXT NOT NULL,
    slug TEXT UNIQUE,
    owner_id UUID REFERENCES auth.users(id) NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Memberships
CREATE TABLE memberships (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID REFERENCES orgs(id) ON DELETE CASCADE NOT NULL,
    user_id UUID REFERENCES auth.users(id) NOT NULL,
    role TEXT NOT NULL DEFAULT 'member',
    created_at TIMESTAMPTZ DEFAULT NOW(),
    UNIQUE(org_id, user_id)
);

-- Pockets (Knowledge Collections)
CREATE TABLE pockets (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID REFERENCES orgs(id) ON DELETE CASCADE NOT NULL,
    name TEXT NOT NULL,
    description TEXT,
    created_by UUID REFERENCES auth.users(id) NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- Pocket Members
CREATE TABLE pocket_members (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    pocket_id UUID REFERENCES pockets(id) ON DELETE CASCADE NOT NULL,
    org_id UUID REFERENCES orgs(id) ON DELETE CASCADE NOT NULL,
    user_id UUID REFERENCES auth.users(id) NOT NULL,
    role TEXT NOT NULL DEFAULT 'viewer',
    created_at TIMESTAMPTZ DEFAULT NOW(),
    UNIQUE(pocket_id, user_id)
);

-- Sources (URLs, PDFs, etc.)
CREATE TABLE sources (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID REFERENCES orgs(id) ON DELETE CASCADE NOT NULL,
    pocket_id UUID REFERENCES pockets(id) ON DELETE CASCADE NOT NULL,
    type TEXT NOT NULL,
    title TEXT NOT NULL,
    url TEXT,
    storage_path TEXT,
    mime_type TEXT NOT NULL,
    size_bytes BIGINT,
    status TEXT NOT NULL DEFAULT 'queued',
    error_message TEXT,
    created_by UUID REFERENCES auth.users(id) NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- Chunks (Text segments with embeddings)
CREATE TABLE chunks (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID REFERENCES orgs(id) ON DELETE CASCADE NOT NULL,
    pocket_id UUID REFERENCES pockets(id) ON DELETE CASCADE NOT NULL,
    source_id UUID REFERENCES sources(id) ON DELETE CASCADE NOT NULL,
    idx INTEGER NOT NULL,
    page INTEGER,
    text TEXT NOT NULL,
    content_hash TEXT NOT NULL,
    embedding VECTOR(3072),
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Conversations
CREATE TABLE conversations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID REFERENCES orgs(id) ON DELETE CASCADE NOT NULL,
    pocket_id UUID REFERENCES pockets(id) ON DELETE CASCADE NOT NULL,
    created_by UUID REFERENCES auth.users(id) NOT NULL,
    title TEXT,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- Messages
CREATE TABLE messages (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID REFERENCES orgs(id) ON DELETE CASCADE NOT NULL,
    pocket_id UUID REFERENCES pockets(id) ON DELETE CASCADE NOT NULL,
    conversation_id UUID REFERENCES conversations(id) ON DELETE CASCADE NOT NULL,
    role TEXT NOT NULL,
    content TEXT NOT NULL,
    citations JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Memories (Personal Notes/Thoughts)
CREATE TABLE memories (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID REFERENCES orgs(id) ON DELETE CASCADE NOT NULL,
    user_id UUID REFERENCES auth.users(id) NOT NULL,
    title TEXT NOT NULL,
    content TEXT NOT NULL,
    content_html TEXT,
    color TEXT DEFAULT 'default',
    tags TEXT[] DEFAULT '{}',
    status TEXT NOT NULL DEFAULT 'active',
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- Memory Chunks (For AI search)
CREATE TABLE memory_chunks (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID REFERENCES orgs(id) ON DELETE CASCADE NOT NULL,
    memory_id UUID REFERENCES memories(id) ON DELETE CASCADE NOT NULL,
    user_id UUID REFERENCES auth.users(id) NOT NULL,
    idx INTEGER NOT NULL,
    text TEXT NOT NULL,
    content_hash TEXT NOT NULL,
    embedding VECTOR(1536),
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- General Conversations (For Memory Chat)
CREATE TABLE general_conversations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID REFERENCES orgs(id) ON DELETE CASCADE NOT NULL,
    user_id UUID REFERENCES auth.users(id) NOT NULL,
    title TEXT,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- General Messages (For Memory Chat)
CREATE TABLE general_messages (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID REFERENCES orgs(id) ON DELETE CASCADE NOT NULL,
    user_id UUID REFERENCES auth.users(id) NOT NULL,
    conversation_id UUID REFERENCES general_conversations(id) ON DELETE CASCADE NOT NULL,
    role TEXT NOT NULL,
    content TEXT NOT NULL,
    citations JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- User Settings
CREATE TABLE user_settings (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID REFERENCES auth.users(id) UNIQUE NOT NULL,
    theme TEXT DEFAULT 'system',
    llm_preference TEXT DEFAULT 'openrouter',
    openrouter_api_key_encrypted TEXT,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- Indexes for performance
CREATE INDEX idx_chunks_pocket_id ON chunks(pocket_id);
CREATE INDEX idx_chunks_source_id ON chunks(source_id);
CREATE INDEX idx_chunks_embedding ON chunks USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
CREATE INDEX idx_chunks_text_fts ON chunks USING gin(to_tsvector('english', text));

-- Memory indexes
CREATE INDEX idx_memory_chunks_memory_id ON memory_chunks(memory_id);
CREATE INDEX idx_memory_chunks_user_id ON memory_chunks(user_id);
CREATE INDEX idx_memory_chunks_embedding ON memory_chunks USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
CREATE INDEX idx_memory_chunks_text_fts ON memory_chunks USING gin(to_tsvector('english', text));
CREATE INDEX idx_memories_user_id ON memories(user_id);
CREATE INDEX idx_memories_status ON memories(status) WHERE status = 'active';

-- Hybrid Search Function
CREATE OR REPLACE FUNCTION hybrid_search(
    query_embedding VECTOR(3072),
    query_text TEXT,
    target_pocket_id UUID,
    match_count INT DEFAULT 10,
    vector_weight FLOAT DEFAULT 0.7,
    fts_weight FLOAT DEFAULT 0.3
)
RETURNS TABLE (
    id UUID,
    chunk_id UUID,
    source_id UUID,
    source_title TEXT,
    source_type TEXT,
    page INT,
    text TEXT,
    similarity FLOAT,
    combined_score FLOAT
)
LANGUAGE plpgsql
AS $$
BEGIN
    RETURN QUERY
    WITH vector_results AS (
        SELECT
            c.id,
            c.source_id,
            s.title as source_title,
            s.type as source_type,
            c.page,
            c.text,
            1 - (c.embedding <=> query_embedding) as similarity
        FROM chunks c
        JOIN sources s ON s.id = c.source_id
        WHERE c.pocket_id = target_pocket_id
        ORDER BY c.embedding <=> query_embedding
        LIMIT match_count * 2
    ),
    fts_results AS (
        SELECT
            c.id,
            c.source_id,
            s.title as source_title,
            s.type as source_type,
            c.page,
            c.text,
            ts_rank(to_tsvector('english', c.text), plainto_tsquery('english', query_text)) as rank
        FROM chunks c
        JOIN sources s ON s.id = c.source_id
        WHERE c.pocket_id = target_pocket_id
        AND to_tsvector('english', c.text) @@ plainto_tsquery('english', query_text)
        ORDER BY rank DESC
        LIMIT match_count * 2
    ),
    combined AS (
        SELECT
            COALESCE(v.id, f.id) as id,
            COALESCE(v.source_id, f.source_id) as source_id,
            COALESCE(v.source_title, f.source_title) as source_title,
            COALESCE(v.source_type, f.source_type) as source_type,
            COALESCE(v.page, f.page) as page,
            COALESCE(v.text, f.text) as text,
            COALESCE(v.similarity, 0) as similarity,
            (COALESCE(v.similarity, 0) * vector_weight) +
            (COALESCE(f.rank, 0) * fts_weight) as combined_score
        FROM vector_results v
        FULL OUTER JOIN fts_results f ON v.id = f.id
    )
    SELECT
        combined.id,
        combined.id as chunk_id,
        combined.source_id,
        combined.source_title,
        combined.source_type,
        combined.page,
        combined.text,
        combined.similarity,
        combined.combined_score
    FROM combined
    ORDER BY combined_score DESC
    LIMIT match_count;
END;
$$;

-- Memory Hybrid Search Function
CREATE OR REPLACE FUNCTION memory_hybrid_search(
    query_embedding VECTOR(1536),
    query_text TEXT,
    target_user_id UUID,
    match_count INT DEFAULT 10,
    vector_weight FLOAT DEFAULT 0.7,
    fts_weight FLOAT DEFAULT 0.3
)
RETURNS TABLE (
    id UUID,
    memory_id UUID,
    memory_title TEXT,
    text TEXT,
    similarity FLOAT,
    combined_score FLOAT
)
LANGUAGE plpgsql
AS $$
BEGIN
    RETURN QUERY
    WITH vector_results AS (
        SELECT
            mc.id,
            mc.memory_id,
            m.title as memory_title,
            mc.text,
            1 - (mc.embedding <=> query_embedding) as similarity
        FROM memory_chunks mc
        JOIN memories m ON m.id = mc.memory_id
        WHERE mc.user_id = target_user_id
        AND m.status = 'active'
        ORDER BY mc.embedding <=> query_embedding
        LIMIT match_count * 2
    ),
    fts_results AS (
        SELECT
            mc.id,
            mc.memory_id,
            m.title as memory_title,
            mc.text,
            ts_rank(to_tsvector('english', mc.text), plainto_tsquery('english', query_text)) as rank
        FROM memory_chunks mc
        JOIN memories m ON m.id = mc.memory_id
        WHERE mc.user_id = target_user_id
        AND m.status = 'active'
        AND to_tsvector('english', mc.text) @@ plainto_tsquery('english', query_text)
        ORDER BY rank DESC
        LIMIT match_count * 2
    ),
    combined AS (
        SELECT
            COALESCE(v.id, f.id) as id,
            COALESCE(v.memory_id, f.memory_id) as memory_id,
            COALESCE(v.memory_title, f.memory_title) as memory_title,
            COALESCE(v.text, f.text) as text,
            COALESCE(v.similarity, 0) as similarity,
            (COALESCE(v.similarity, 0) * vector_weight) +
            (COALESCE(f.rank, 0) * fts_weight) as combined_score
        FROM vector_results v
        FULL OUTER JOIN fts_results f ON v.id = f.id
    )
    SELECT
        combined.id,
        combined.memory_id,
        combined.memory_title,
        combined.text,
        combined.similarity,
        combined.combined_score
    FROM combined
    ORDER BY combined_score DESC
    LIMIT match_count;
END;
$$;

-- Auto-create org on signup
CREATE OR REPLACE FUNCTION handle_new_user()
RETURNS TRIGGER AS $$
DECLARE
    v_org_id UUID;
BEGIN
    INSERT INTO public.orgs (name, slug, owner_id)
    VALUES ('Personal', 'personal-' || substring(NEW.id::text from 1 for 8), NEW.id)
    RETURNING id INTO v_org_id;

    INSERT INTO public.memberships (org_id, user_id, role)
    VALUES (v_org_id, NEW.id, 'owner');

    RETURN NEW;
EXCEPTION
    WHEN OTHERS THEN
        RAISE WARNING 'Failed to create org for user %: %', NEW.id, SQLERRM;
        RETURN NEW;
END;
$$ LANGUAGE plpgsql SECURITY DEFINER;

CREATE TRIGGER on_auth_user_created
    AFTER INSERT ON auth.users
    FOR EACH ROW
    EXECUTE FUNCTION handle_new_user();

Enable Row Level Security (RLS) on all tables
Create a storage bucket called sources for file uploads

Step 2: OpenRouter Setup

Create account at openrouter.ai
Add credits ($5 gets you pretty far)
Create an API key
Note: Embedding costs ~$0.0001 per 1K tokens, Chat varies by model

Step 3: Railway Deployment

Railway makes deploying Docker containers dead simple.

Create account at railway.app
Create a new project
Add a Redis service (click "+ New" → "Database" → "Redis")

Deploy the API:

Click "+ New" → "GitHub Repo"
Select vedha-pocket-api
Add environment variables (from .env above)
Railway auto-detects Dockerfile and deploys

Deploy the Worker:

Click "+ New" → "GitHub Repo"
Select vedha-pocket-worker
Add environment variables
Deploy!

Step 4: Vercel Deployment

Connect your GitHub to Vercel
Import vedha-pocket-web
Add environment variables:
- NEXT_PUBLIC_SUPABASE_URL
- NEXT_PUBLIC_SUPABASE_ANON_KEY
- NEXT_PUBLIC_API_URL (your Railway API URL)
Deploy!

Docker Compose (One-Click Local Setup)

Want to run everything locally with one command? Here's a docker-compose file:

# docker-compose.yml
version: "3.8"

services:
  redis:
    image: redis:alpine
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data

  api:
    build: ./vedha-pocket-api
    ports:
      - "3001:3001"
    environment:
      - PORT=3001
      - NODE_ENV=production
      - SUPABASE_URL=${SUPABASE_URL}
      - SUPABASE_ANON_KEY=${SUPABASE_ANON_KEY}
      - SUPABASE_SERVICE_ROLE_KEY=${SUPABASE_SERVICE_ROLE_KEY}
      - REDIS_URL=redis://redis:6379
      - OPENROUTER_API_KEY=${OPENROUTER_API_KEY}
      - OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
      - OPENROUTER_EMBED_MODEL=openai/text-embedding-3-large
      - OPENROUTER_CHAT_MODEL=anthropic/claude-sonnet-4
    depends_on:
      - redis

  worker:
    build: ./vedha-pocket-worker
    environment:
      - NODE_ENV=production
      - SUPABASE_URL=${SUPABASE_URL}
      - SUPABASE_SERVICE_ROLE_KEY=${SUPABASE_SERVICE_ROLE_KEY}
      - REDIS_URL=redis://redis:6379
      - OPENROUTER_API_KEY=${OPENROUTER_API_KEY}
      - OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
      - OPENROUTER_EMBED_MODEL=openai/text-embedding-3-large
      - PLAYWRIGHT_ENABLED=true
      - WORKER_CONCURRENCY=5
    depends_on:
      - redis

volumes:
  redis_data:

Run it:

# Create .env file with your secrets
cp .env.example .env
# Edit .env with your values

# Start everything
docker-compose up -d

# Check logs
docker-compose logs -f

The Funny Bits (Lessons Learned)

1. The Bot Detection Saga

First version of the scraper successfully saved... CloudFlare challenge pages. Very useful. "According to your sources, JavaScript is required to view this content."

Now we check for phrases like "Please enable JavaScript" and "Access Denied" before declaring victory.

2. The Infinite Recursion Incident

Row Level Security is great until your policies check table A which checks table B which checks table A. PostgreSQL was NOT amused. Spent 3 hours debugging "infinite recursion detected in policy."

Moral: Don't let your security policies play ping-pong.

3. The UUID Citation Disaster

LLM was citing sources like: "According to [0d1ca55c-8278-44b3-aedf-a5fada07bb97]..."

Very helpful for robots. Less so for humans. Fixed it to use [Source 1] instead.

4. The "Why Won't It Scroll" Mystery

Chat messages wouldn't scroll. Tried everything. Turns out flex and overflow have a complicated relationship. CSS: keeping developers humble since 1996.

What's Next?

Some ideas brewing:

For Memories:

[ ] Daily/Weekly Reviews — AI-powered reflection prompts
[ ] Memory Connections — Auto-suggest related thoughts
[ ] Voice Memos — Speak your thoughts, auto-transcribed
[ ] Memory Maps — Visual connections between ideas

For Pockets:

[ ] Chrome Extension — Save pages with one click
[ ] Smart Summaries — Auto-generate document summaries
[ ] Multi-modal — Images, audio, video support

For Both:

[ ] Mobile App — Capture thoughts and research on the go
[ ] Slack Integration — Ask questions in Slack
[ ] Team Collaboration — Share memories and pockets
[ ] Automated Insights — Weekly "what you learned" reports

Try It Out!

🌐 Live Demo: https://vedha-pocket-web.vercel.app/

⚠️ Note: This is a BYOK (Bring Your Own Key) app. After signing up, you'll need to add your OpenRouter API key in Settings to use AI features. No subscription required — you only pay for what you use!

🆓 Free tier available: The app uses free models by default (Google Gemma), so you can try it out with minimal costs.

The project is fully open source. Clone it, deploy it, break it, fix it, make it yours.

GitHub Repos:

Tech Stack:

Frontend: Next.js 14, Tailwind CSS, shadcn/ui
Backend: Fastify, TypeScript, BullMQ
Database: Supabase (PostgreSQL + pgvector)
LLM: OpenRouter (Claude, GPT-4, etc.) or local (Ollama)

Final Thoughts

Building Memory Palace taught me that RAG isn't magic — it's chunking, embedding, and good prompts. But adding personal memory management revealed new challenges:

Technical challenges:

Extracting clean text — PDFs are still nightmares
Chunking intelligently — Too small = no context, too big = irrelevant
Not hallucinating — LLMs love to make stuff up
Making it fast — Streaming and multi-query search save UX

UX challenges:

Dual context switching — Users need to think "documents" vs "memories"
Information architecture — How do you organize both research and thoughts?
Privacy concerns — Personal memories are more sensitive than research papers
Cross-pollination — Connecting external knowledge with internal insights

The most surprising insight? People don't just want to remember what they read — they want to remember what they thought about what they read. That's where the real value lies.

If you build something cool with this, let me know! And if you find bugs, well... that's what issues are for. 😄

Built with ❤️, ☕, and way too many hours debugging RLS policies.

— Venkat