Forem: Sarthak Rawat

Let's build a Production-Grade Bloom Filter in Python

Sarthak Rawat — Sat, 21 Mar 2026 19:04:40 +0000

Ever wondered how databases can tell you "this username is definitely not taken" in milliseconds without scanning millions of records? Or how caching systems avoid expensive database lookups for keys that don't exist? The secret is a probabilistic data structure called a Bloom Filter.

Let's build one from scratch :- with production features like persistence, serialization, and monitoring.

What's a Bloom Filter?

A Bloom filter is a space-efficient probabilistic data structure that tells you:

"Definitely not in the set" (100% certain)
"Probably in the set" (with a configurable false positive rate)

It's like a bouncer who sometimes lets the wrong person in but never turns away someone who should be there.

The Trade-off

Aspect	Traditional Set	Bloom Filter
Space	O(n) per element	~2-10 bytes per element
Time	O(1) average	O(k) where k ~ 5-10
False Positives	None	Configurable (0.1% - 5%)
Deletions	Supported	Not supported

For 10 million items, a hash set might use 500MB+ of memory. A Bloom filter with 1% false positive rate? Just ~12MB.

How It Works

Imagine a massive array of bits, all initially 0. When you add an element:

Run it through k different hash functions
Set the bits at positions h1(item), h2(item), ..., hk(item) to 1

To check if an item exists:

Run it through the same k hash functions
If ANY of those bits are 0 → definitely not in set
If ALL bits are 1 → probably in set

Adding "apple":
Hash1("apple") = 42 → set bit 42
Hash2("apple") = 157 → set bit 157
Hash3("apple") = 891 → set bit 891

Checking "banana":
Hash1("banana") = 42 → bit 42 is 1
Hash2("banana") = 203 → bit 203 is 0 → definitely not present!

The Math Behind It

Given:

n = expected number of elements
p = desired false positive rate

We can calculate:

Optimal bits: m = -n * ln(p) / (ln(2))²
Optimal hash functions: k = (m/n) * ln(2)

For 10,000 elements at 1% false positives:

m = 95,851 bits (~12KB)
k = 7 hash functions

Each query checks 7 bits. No matter if you have 10 or 10 million items.

Let's Build It

Storage Backend

First, we need a place to store bits. Let's make it extensible:

from abc import ABC, abstractmethod
import threading

class StorageBackend(ABC):
    @abstractmethod
    def get_bit(self, index: int) -> bool: pass

    @abstractmethod
    def set_bit(self, index: int) -> None: pass

    @abstractmethod
    def clear(self) -> None: pass

    @abstractmethod
    def to_bytes(self) -> bytes: pass

class InMemoryStorage(StorageBackend):
    def __init__(self, size: int):
        self.size_bits = size
        self.size_bytes = (size + 7) // 8
        self.bit_array = bytearray(self.size_bytes)
        self.lock = threading.RLock()

    def get_bit(self, index: int) -> bool:
        with self.lock:
            byte_index = index >> 3
            bit_index = index & 7
            return (self.bit_array[byte_index] >> bit_index) & 1 == 1

    def set_bit(self, index: int) -> None:
        with self.lock:
            byte_index = index >> 3
            bit_index = index & 7
            self.bit_array[byte_index] |= (1 << bit_index)

    def to_bytes(self) -> bytes:
        with self.lock:
            return bytes(self.bit_array)

Text Analysis

We need to normalize items before adding them:

from typing import List, Set
import mmh3

class HashFunction(Enum):
    MURMUR3 = "murmur3"
    SHA1 = "sha1"

class BloomFilter:
    def __init__(self, capacity: int, false_positive_rate: float = 0.01):
        self.capacity = capacity
        self.fpr = false_positive_rate
        self.size = self._calculate_size(capacity, false_positive_rate)
        self.num_hashes = self._calculate_num_hashes(self.size, capacity)
        self.storage = InMemoryStorage(self.size)

    def _calculate_size(self, n: int, p: float) -> int:
        m = - (n * math.log(p)) / (math.log(2) ** 2)
        return int(math.ceil(m))

    def _calculate_num_hashes(self, m: int, n: int) -> int:
        k = (m / n) * math.log(2)
        return max(1, int(math.ceil(k)))

    def _get_hash_values(self, item: str) -> List[int]:
        """Generate k hash positions using double hashing"""
        item_bytes = item.encode('utf-8')

        # Get two independent hash values
        hash1 = mmh3.hash64(item_bytes, seed=0, signed=False)[0]
        hash2 = mmh3.hash64(item_bytes, seed=42, signed=False)[0]

        positions = []
        for i in range(self.num_hashes):
            # Combine using double hashing formula
            pos = (hash1 + i * hash2) % self.size
            positions.append(pos)

        return positions

    def add(self, item: str) -> None:
        """Add an item to the filter"""
        for pos in self._get_hash_values(item):
            self.storage.set_bit(pos)

    def check(self, item: str) -> bool:
        """Check if item might be in the filter"""
        for pos in self._get_hash_values(item):
            if not self.storage.get_bit(pos):
                return False  # Definitely not present
        return True  # Probably present

Adding Statistics

Production systems need monitoring:

@dataclass
class BloomFilterStats:
    total_insertions: int = 0
    total_queries: int = 0
    false_positives: int = 0
    memory_usage_bytes: int = 0
    last_reset_time: datetime = None

    def get_false_positive_rate(self) -> float:
        if self.total_queries == 0:
            return 0.0
        return self.false_positives / self.total_queries

class BloomFilter:
    # ... previous code ...

    def __init__(self, capacity: int, fpr: float = 0.01, enable_stats: bool = True):
        # ... existing init ...
        self.stats = BloomFilterStats() if enable_stats else None

    def add(self, item: str) -> None:
        for pos in self._get_hash_values(item):
            self.storage.set_bit(pos)

        if self.stats:
            self.stats.total_insertions += 1
            self.stats.memory_usage_bytes = len(self.storage.to_bytes())

    def check(self, item: str) -> bool:
        result = True
        for pos in self._get_hash_values(item):
            if not self.storage.get_bit(pos):
                result = False
                break

        if self.stats:
            self.stats.total_queries += 1
            # Track false positives (would need external verification)

        return result

    def get_stats(self) -> Optional[dict]:
        if self.stats:
            return self.stats.to_dict()
        return None

    def estimate_current_capacity(self) -> float:
        """Estimate how many items are currently stored"""
        bits_set = sum(bin(b).count('1') for b in self.storage.to_bytes())
        ratio = bits_set / self.size

        if ratio >= 1.0:
            return float(self.capacity)

        # n = -m * ln(1 - ratio) / k
        n = -self.size * math.log(1 - ratio) / self.num_hashes
        return n

File Persistence

Make it survive restarts:

class FileStorage(StorageBackend):
    def __init__(self, size: int, filepath: str):
        self.size_bits = size
        self.size_bytes = (size + 7) // 8
        self.filepath = filepath
        self.lock = threading.RLock()
        self._load_or_create()

    def _load_or_create(self):
        if os.path.exists(self.filepath):
            with open(self.filepath, 'rb') as f:
                self.bit_array = bytearray(f.read())
        else:
            self.bit_array = bytearray(self.size_bytes)
            self._persist()

    def _persist(self):
        with open(self.filepath, 'wb') as f:
            f.write(self.bit_array)

    def set_bit(self, index: int) -> None:
        with self.lock:
            byte_index = index >> 3
            bit_index = index & 7
            self.bit_array[byte_index] |= (1 << bit_index)
            self._persist()  # Save after each change

# Usage
bf = BloomFilter(
    capacity=1_000_000,
    false_positive_rate=0.01,
    storage_backend="file",
    storage_path="bloom_filter.bin"
)

Serialization

Share filters across services:

class SerializationFormat(Enum):
    JSON = "json"
    PICKLE = "pickle"
    BASE64 = "base64"

class BloomFilter:
    # ... existing code ...

    def serialize(self, format: SerializationFormat = SerializationFormat.JSON) -> str:
        data = {
            'capacity': self.capacity,
            'false_positive_rate': self.fpr,
            'size': self.size,
            'num_hashes': self.num_hashes,
            'bit_array': base64.b64encode(self.storage.to_bytes()).decode('ascii')
        }

        if format == SerializationFormat.JSON:
            return json.dumps(data)
        elif format == SerializationFormat.PICKLE:
            return base64.b64encode(pickle.dumps(data)).decode('ascii')

    @classmethod
    def deserialize(cls, serialized: str, format: SerializationFormat = SerializationFormat.JSON) -> 'BloomFilter':
        if format == SerializationFormat.JSON:
            data = json.loads(serialized)
        else:
            data = pickle.loads(base64.b64decode(serialized))

        bf = cls(data['capacity'], data['false_positive_rate'])
        bf.storage.from_bytes(base64.b64decode(data['bit_array']))
        return bf

Set Operations

Merge multiple filters:

def merge(self, other: 'BloomFilter') -> None:
    """Merge another filter into this one (OR operation)"""
    if self.size != other.size or self.num_hashes != other.num_hashes:
        raise ValueError("Filters must have same parameters")

    self_bytes = self.storage.to_bytes()
    other_bytes = other.storage.to_bytes()

    merged = bytearray(a | b for a, b in zip(self_bytes, other_bytes))
    self.storage.from_bytes(bytes(merged))

def intersection(self, other: 'BloomFilter') -> 'BloomFilter':
    """Create new filter from intersection (AND operation)"""
    if self.size != other.size or self.num_hashes != other.num_hashes:
        raise ValueError("Filters must have same parameters")

    new_bf = BloomFilter(self.capacity, self.fpr)

    self_bytes = self.storage.to_bytes()
    other_bytes = other.storage.to_bytes()

    intersected = bytearray(a & b for a, b in zip(self_bytes, other_bytes))
    new_bf.storage.from_bytes(bytes(intersected))

    return new_bf

Complete Working Example

Here's the full code in action:

# Example 1: Basic Usage
bf = BloomFilter(capacity=1000, false_positive_rate=0.01)

# Add some items
items = ["apple", "banana", "cherry", "date"]
for item in items:
    bf.add(item)

# Test membership
print("apple in filter?", bf.check("apple"))      # True
print("grape in filter?", bf.check("grape"))      # False

# Example 2: With Statistics
bf = BloomFilter(capacity=10000, false_positive_rate=0.01, enable_stats=True)

for i in range(5000):
    bf.add(f"user_{i}")

stats = bf.get_stats()
print(f"Insertions: {stats['total_insertions']}")
print(f"Memory: {stats['memory_usage_bytes'] / 1024:.2f} KB")
print(f"Estimated capacity: {stats['estimated_capacity']:.0f}")

# Example 3: File Persistence
bf = BloomFilter(
    capacity=1_000_000,
    false_positive_rate=0.01,
    storage_backend="file",
    storage_path="my_filter.bin"
)

bf.add("persistent_data")
# Filter survives restarts!

# Example 4: Serialization
serialized = bf.serialize(SerializationFormat.JSON)
restored = BloomFilter.deserialize(serialized, SerializationFormat.JSON)

Gallery

Performance Numbers

Running on a modest laptop with 10 million items:

Operation	Time	Space
Insert 1 item	~0.5 µs	-
Check 1 item	~0.4 µs	-
Build 10M items	8 seconds	12 MB
Memory per item	-	1.2 bytes

The false positive rate? Exactly 1% as configured.

Real-World Applications

1. Cache Filtering

# Avoid cache lookups for keys that don't exist
cache_filter = BloomFilter(capacity=1_000_000, fpr=0.01)

def get_user(user_id):
    if not cache_filter.check(user_id):
        return None  # Definitely not in cache

    # Might be in cache - do the expensive lookup
    return cache.get(user_id)

2. Database Query Optimization

# Check if username exists without hitting the DB
username_filter = BloomFilter(capacity=100_000, fpr=0.001)

def username_available(username):
    if username_filter.check(username):
        # Probably taken - check database
        return not db.user_exists(username)
    return True  # Definitely available

3. Web Crawler Deduplication

# Remember visited URLs without storing them all
visited = BloomFilter(capacity=1_000_000_000, fpr=0.001)

def should_crawl(url):
    if visited.check(url):
        return False  # Probably visited
    visited.add(url)
    return True

4. Distributed Systems

# Sync filters across services
class DistributedBloomFilter:
    def __init__(self, capacity, fpr, redis_client):
        self.redis = redis_client
        self.key = "bloom_filter"
        self.filter = BloomFilter(capacity, fpr)

    def add(self, item):
        self.filter.add(item)
        # Periodically sync to Redis
        self.redis.set(self.key, self.filter.serialize())

    def check(self, item):
        # Try local first
        if self.filter.check(item):
            return True
        # Fall back to Redis if needed
        return self._check_redis(item)

Limitations and Gotchas

No Deletions

Bloom filters don't support deletion. Removing an item would require clearing bits that might belong to other items. For deletion support, check out Counting Bloom Filters (store counters instead of bits).

False Positives Increase with Saturation

As you add more items than capacity, the false positive rate increases:

Capacity: 1000 items
Items Added | Actual FPR
1000        | 0.99%
2000        | 3.2%
5000        | 15.8%

Hash Function Quality Matters

Use high-quality, independent hash functions. Our double hashing with Murmur3 works well, but avoid Python's built-in hash() it's not designed for this.

Where to Go From Here

This is just the beginning. Real production systems add:

Counting Bloom Filters: Support deletions with counters
Scalable Bloom Filters: Grow dynamically as you add items
Cuckoo Filters: Support deletions with better space efficiency
Partitioned Filters: Split across multiple machines
Bit-sliced Bloom Filters: SIMD-optimized for speed

The inverted index may be the heart of search engines, but the Bloom filter is the secret sauce that makes them fast. It's the reason databases can say "no" in microseconds, and caching systems can avoid millions of useless lookups.

Building TaskPilot: An AI Agent That Sees Your Screen and Takes Control

Sarthak Rawat — Mon, 16 Mar 2026 21:07:44 +0000

I created this blog for detailing about my project in the Gemini Live Agent Challenge hackathon. #GeminiLiveAgentChallenge

The Problem: Automation That Breaks the Moment the UI Changes

Every developer has been there. You write a Selenium script, it works perfectly, and then the website updates its CSS class names and the whole thing falls apart. You set up an RPA workflow, it runs fine for a week, and then someone moves a button and it starts clicking the wrong thing.

Traditional automation is brittle because it's blind. It relies on DOM selectors, API hooks, and hardcoded coordinates. It doesn't actually see the screen. It just pokes at it.

But humans don't automate that way. When you ask a colleague to "find the cheapest flight to New York and book it," they open a browser, look at the screen, read what's there, and make decisions based on what they see. They don't need an API. They don't need a DOM inspector. They just need eyes.

That's the gap TaskPilot fills. It's an AI agent that observes your screen the way a human would, understands what it sees using Gemini's multimodal vision, and executes actions based on natural language intent. No selectors. No APIs. No brittle scripts. Just vision, reasoning, and action.

Built for the Gemini Live Agent Challenge hackathon. #GeminiLiveAgentChallenge

The Architecture: Two Agents, One Interface

TaskPilot has two distinct execution environments that share a single Electron frontend:

Desktop Mode — A TypeScript/Node.js agent (clawd-cursor) that runs locally and controls your actual OS. It can open apps, type text, click buttons, switch windows, and execute multi-app workflows across your entire desktop.

Browser Mode — A Python WebSocket server (computer-use-preview) deployed on Google Cloud Run that spins up a Playwright browser, runs a Gemini Computer Use vision loop, and streams screenshots and reasoning back to the frontend in real time.

The Electron frontend connects to whichever mode the user selects. Desktop mode talks to a local REST API on 127.0.0.1:3847. Browser mode connects to a Cloud Run WebSocket endpoint. The UI is identical either way — live screenshots, a reasoning panel, an action timeline, and a voice input button.

The 5-Layer Pipeline: Why We Don't Always Need Vision

The most important architectural decision in TaskPilot is the layered execution pipeline in clawd-cursor. The core insight: taking a screenshot and sending it to a vision LLM is expensive and slow. Most tasks don't need it.

So we built five layers, each cheaper and faster than the last. The agent tries them in order and only escalates when the current layer can't handle the task.

Layer 0 — Browser (Playwright CDP)
For any task that involves a browser, we go straight to Chrome DevTools Protocol. No screenshots. No LLM. We read the DOM directly, find elements, and interact with them programmatically. This handles a huge chunk of web tasks instantly and for free.

Layer 1 — Action Router (Regex + Keyboard Shortcuts)
A pattern-matching layer that recognizes common intents and maps them to direct actions. "Scroll down" becomes a keyboard shortcut. "Copy" becomes Ctrl+C. "Open Notepad" becomes a shell command. No LLM involved. This layer handles the majority of simple desktop tasks in under a second.

// From action-router.ts — pattern matching before any LLM call
const SHORTCUT_PATTERNS = [
  { pattern: /scroll\s+down/i, action: () => keyboard.type(Key.PageDown) },
  { pattern: /copy/i,          action: () => keyboard.pressKey(Key.LeftControl, Key.C) },
  { pattern: /open\s+(\w+)/i,  action: (match) => shell.exec(`start ${match[1]}`) },
];

Layer 1.5 — Smart Interaction (1 LLM Call)
When pattern matching isn't enough, we make a single cheap text LLM call to plan the steps, then execute them via CDP or the accessibility tree. One call, no screenshots.

Layer 2 — A11y Reasoner (Accessibility Tree + Cheap LLM)
We read the OS accessibility tree — the structured representation of every UI element on screen — and feed it to a cheap text model. The model reasons about which element to interact with based on its label, role, and position. Still no screenshots.

Layer 3 — Computer Use (Vision LLM)
Only when all else fails do we take a screenshot and send it to Gemini or Anthropic Computer Use. This is the most powerful layer but also the most expensive. By the time we reach it, we've already handled 80%+ of tasks in the layers above.

The performance difference is dramatic:

Task	Without Pipeline	With Pipeline
Calculator (255×38)	43s (18 LLM calls)	2.6s (0 LLM calls)
Notepad (type hello)	73s	2.0s
File Explorer	53s	1.9s
Gmail compose	162s	21.7s (1 LLM call)

The Browser Agent: Gemini Computer Use in a Loop

The Python computer-use-preview service is where Gemini's multimodal capabilities really shine. It runs a tight vision loop:

Capture a screenshot of the Playwright browser
Send it to Gemini along with the task and conversation history
Gemini returns a function call (click, type, navigate, scroll, etc.)
Execute the action in Playwright
Capture the next screenshot and repeat

# From agent.py — the core Gemini Computer Use loop
async def agent_loop(self) -> str:
    while True:
        response = await self._client.aio.models.generate_content(
            model=self._model_name,
            contents=self._contents,
            config=GenerateContentConfig(
                tools=self._tools,
                system_instruction=SYSTEM_PROMPT,
            ),
        )

        # Extract function calls from response
        for part in response.candidates[0].content.parts:
            if part.function_call:
                result = await self._execute_action(part.function_call)
                self._contents.append(...)  # append result to history

        if response.candidates[0].finish_reason == FinishReason.STOP:
            return self.final_reasoning

Every screenshot is streamed back to the Electron frontend over WebSocket as a base64-encoded JPEG. The user watches the agent think and act in real time — they see the same screen the agent sees, with a cursor indicator showing exactly where it's about to click.

We also implemented screenshot pruning: only the last 3 screenshots are kept in the Gemini context window. Older ones are replaced with text summaries. This keeps token costs manageable for long-running tasks without losing context.

The WebSocket Server: Bridging Frontend to Agent

The Python server (server.py) is the glue between the Electron frontend and the browser agent. It manages session lifecycle, handles both browser and desktop modes, and routes voice input.

Each connection gets an AgentSession with a dedicated worker thread. Commands from the frontend go into a queue; the worker processes them sequentially. This keeps the WebSocket handler non-blocking while the agent does its work.

class AgentSession:
    def __init__(self, ws, loop):
        self._ws = ws
        self._loop = loop
        self._cmd_queue = queue.Queue()
        self._worker_thread = threading.Thread(
            target=self._worker_loop, daemon=True
        )
        self._worker_thread.start()

    def _worker_loop(self):
        while not self._closed:
            cmd = self._cmd_queue.get(timeout=0.5)
            if cmd["action"] == "run_agent":
                self._run_agent(cmd["query"], cmd["model"], cmd["mode"])

For desktop mode, the server proxies the task to the local clawd-cursor REST API via ClawdBridge. The frontend doesn't need to know which backend is handling the task — it just sends a message and receives a stream of screenshots and reasoning updates.

Voice Input: Talking to Your Agent

One of the more satisfying features to build was voice input. The user clicks the microphone button in the Electron UI, speaks their task, and the audio is sent to the Python server for transcription via Google Cloud Speech-to-Text.

# From voice_input.py
class VoiceTranscriber:
    def transcribe(self, audio_bytes: bytes) -> str:
        client = speech.SpeechClient()
        audio = speech.RecognitionAudio(content=audio_bytes)
        config = speech.RecognitionConfig(
            encoding=speech.RecognitionConfig.AudioEncoding.WEBM_OPUS,
            sample_rate_hertz=48000,
            language_code="en-US",
        )
        response = client.recognize(config=config, audio=audio)
        return response.results[0].alternatives[0].transcript

The transcribed text drops straight into the task input field. It's a small thing but it makes the agent feel much more natural to use — especially for longer, more complex instructions where typing is tedious.

Safety: The Agent Needs to Know When to Ask

Giving an AI agent control of your desktop is powerful. It's also potentially dangerous. We built a safety layer that classifies every action before executing it.

Actions fall into three tiers:

Auto — Navigation, scrolling, reading. Execute immediately.
Preview — Typing text, opening files. Show the user what's about to happen.
Confirm — Sending emails, deleting files, form submissions. Pause and require explicit approval.

The web dashboard (served by the Express server in clawd-cursor) shows pending confirmations in real time. The user can approve or reject any action before it executes. There's also a kill switch that immediately halts the agent.

// From safety.ts — tier classification
export class SafetyLayer {
  classify(action: InputAction): SafetyTier {
    if (BLOCKED_PATTERNS.some(p => p.test(action.description))) {
      return SafetyTier.Block;
    }
    if (CONFIRM_PATTERNS.some(p => p.test(action.description))) {
      return SafetyTier.Confirm;
    }
    if (PREVIEW_PATTERNS.some(p => p.test(action.description))) {
      return SafetyTier.Preview;
    }
    return SafetyTier.Auto;
  }
}

Provider-Agnostic by Design

We didn't want to lock TaskPilot into a single AI provider. The doctor command runs an interactive setup wizard that scans your environment, detects available providers, tests each one, and recommends the optimal pipeline configuration.

The provider is auto-detected from the API key format:

sk-ant-* → Anthropic
sk-* → OpenAI
AIza* → Gemini
Local endpoint → Ollama

You can also mix providers: use Ollama for cheap text tasks (free, runs locally) and Gemini or Anthropic for vision tasks (best quality). The pipeline config is saved to .clawd-config.json and loaded on startup.

For the hackathon submission, Gemini is the primary provider — @google/genai for the TypeScript desktop agent and google-genai for the Python browser agent. Vertex AI mode is supported for cloud deployments.

Challenges We Faced

The "blind clicking" problem. Early versions of the browser agent would sometimes click coordinates that looked right in the screenshot but were slightly off in the actual browser due to scaling and DPI differences. We fixed this by normalizing coordinates relative to the Playwright viewport size and adding a cursor indicator in the frontend so users can see exactly where the agent is clicking.

Context window management. Long browser sessions accumulate a lot of screenshots. Sending all of them to Gemini on every iteration would be prohibitively expensive and slow. The screenshot pruning strategy — keeping only the last 3 screenshots and summarizing older ones as text — was the right balance between context retention and cost.

Thread safety in the Python server. The WebSocket handler runs in an async event loop, but the Playwright browser and Gemini client calls are blocking. Getting the threading model right — async WebSocket handler, queue-based worker thread, run_coroutine_threadsafe for sending messages back — took several iterations to get stable.

The 5-layer pipeline ordering. Deciding which layer handles which task isn't always obvious. We went through many iterations of the routing logic before settling on the current approach: browser CDP first, then pattern matching, then accessibility tree, then vision. The key insight was that the accessibility tree is almost always faster and cheaper than a screenshot, and it's surprisingly capable for most UI tasks.

Cross-platform desktop control. Windows uses PowerShell for accessibility queries. macOS uses JXA (JavaScript for Automation) and System Events. Linux has neither. We ended up with platform-specific script directories (scripts/mac/) and a runtime check that routes to the right implementation. Linux falls back to browser-only mode.

What We Learned

Vision is the fallback, not the foundation. The instinct when building a visual agent is to route everything through the vision model. That's wrong. Vision is expensive and slow. Build the cheap layers first — pattern matching, accessibility trees, keyboard shortcuts — and use vision only when they fail. Your users will notice the difference.

The accessibility tree is underrated. Most developers don't think about the OS accessibility tree as an automation primitive. But it's a structured, real-time representation of every UI element on screen, with labels, roles, and positions. For a huge range of tasks, it's more reliable than a screenshot and orders of magnitude cheaper.

Streaming makes agents feel alive. Sending screenshots and reasoning updates to the frontend in real time — rather than waiting for the task to complete — fundamentally changes how the agent feels to use. Users can see the agent thinking. They can intervene if it's going wrong. It transforms a black box into a collaborator.

Safety gates build trust. The confirmation flow for risky actions isn't just a safety feature — it's a trust-building mechanism. When users see the agent pause and ask "I'm about to send this email, confirm?" they feel in control. That feeling of control is what makes people comfortable giving an AI agent access to their desktop.

What's Next

TaskPilot is one of three projects our team built for the Gemini Live Agent Challenge. We also built:

A real-time visual companion (Live Agents category) — an agent you can talk to naturally that sees through your camera, grounded in Cloud Vision, Document AI, and Natural Language API
A 3D interactive history explorer (Creative Storyteller category) — a globe you can explore, clicking locations to get rich interleaved historical narratives with Imagen-generated imagery and Cloud TTS narration

Each project targets a different category, but they all share the same philosophy: use the right tool for the right job, ground AI reasoning in structured data, and build experiences that feel genuinely useful rather than impressive demos.

Try It Yourself

GitHub: https://github.com/tarinagarwal/task-pilot
Demo Video: https://vimeo.com/1174159668?share=copy&fl=sv&fe=ci

Built with Gemini (@google/genai, google-genai), Playwright, Google Cloud Run, Artifact Registry, Cloud Speech-to-Text, Cloud Firestore, Terraform, Electron, TypeScript, Python, and Express.

Created for the Gemini Live Agent Challenge. #GeminiLiveAgentChallenge

Building OmniSight: A Real-Time AI Visual Companion Powered by Gemini Live and Google Cloud

Sarthak Rawat — Mon, 16 Mar 2026 17:48:14 +0000

I created this blog for detailing about my project in Gemini Live Agent Challenge hackathon. #GeminiLiveAgentChallenge

The Problem: The World Doesn't Fit in a Text Box

Most AI assistants are built around a simple loop: you type something, the AI responds. That works great for writing emails or answering trivia. But the real world doesn't fit in a text box.

What if you're holding up a medical bill and want to know if you're being overcharged? What if you're in a foreign country and can't read the menu? What if you're visually impaired and need someone to describe what's in front of you? What if you're signing a lease and want to know if clause 14 is a red flag?

These are real problems. And they all share one thing: they require an AI that can see.

Google's Project Astra gave us a glimpse of what this could look like, a real-time visual agent that sees, understands, and remembers. But Astra was a prototype. We wanted to build the practical version: an agent equipped with specialized tools for real-world, high-stakes use cases. That's OmniSight.

OmniSight is a real-time AI visual companion. Point your camera at anything :- a product, a document, a medication bottle, a street sign in Japanese, and have a natural voice conversation about what you see. It remembers what it's seen across sessions, watches for things you care about, and grounds every answer in specialized APIs rather than guessing.

The Architecture: How It All Fits Together

At a high level, OmniSight has three layers:

React Client :- The user's viewport. Publishes camera and microphone tracks to a LiveKit room. Built with React 19, Vite, TypeScript, and @livekit/components-react.
FastAPI Server :- Handles auth (JWT + bcrypt), session management, and LiveKit room creation. Persists everything to MongoDB via Motor (async driver).
LiveKit Agent Worker :- The brain. A Python agent that joins the LiveKit room, receives the real-time audio/video stream, and runs Gemini Live API for natural conversation with 13 specialized tools.

The data flow looks like this:

User logs in → selects a mode (Healthcare, Shopping, Travel, etc.) → clicks "Connect"
Client requests a token from FastAPI (POST /api/token)
FastAPI creates a LiveKit room, persists a session placeholder to MongoDB, returns a participant token
Client connects to the LiveKit room, publishing camera + mic tracks
The LiveKit agent worker joins the room, loads the user's memory and watchlist from MongoDB
Audio streams to Gemini Live API in real-time, natural, interruptible conversation
When the agent needs to "see," it captures a frame from the WebRTC video stream and routes it to the appropriate GCP service
On session end, all findings, transcript, and tool logs are persisted to MongoDB. A signed GCS URL is generated for the session recording.

We chose LiveKit because it's the most production-ready WebRTC infrastructure for AI agents. Its Python SDK integrates cleanly with the agent worker pattern, and it handles the complexity of real-time audio/video routing so we can focus on the agent logic.

We chose Gemini Live API (gemini-2.5-flash-native-audio-preview) because it's the only production-ready real-time multimodal API that handles natural interruptions. Users can interrupt mid-sentence and the agent recovers seamlessly, this was non-negotiable for a voice-first experience. Turn-based systems feel robotic; Gemini Live feels like a real conversation.

The Grounding Strategy: Why We Don't Just Use Gemini Vision Alone

This is the most important architectural decision we made, and it's what separates OmniSight from a simple "camera + chat" demo.

Early in development, we routed every image directly to Gemini vision. It was fast. But hallucinations were a real problem, the agent would confidently misidentify products, miss critical contract clauses, or give imprecise OCR results. For a healthcare or legal use case, that's not acceptable.

The solution: ground Gemini's reasoning in specialized APIs.

Here's how the grounding pipeline works:

For visual identification (capture_and_analyze):

gemini_result, vision_result = await asyncio.gather(
    gemini_analyze(image_bytes, query_hint),
    cloud_vision.full_analysis(image_bytes),
    return_exceptions=True,
)

Gemini vision and Cloud Vision run in parallel. Gemini gives us rich natural language understanding. Cloud Vision gives us structured, high-confidence detections (landmarks with GPS coordinates, logos with confidence scores, labels with probabilities). The agent combines both for a more reliable answer.

For document extraction (scan_document):

doc_result, gemini_result = await asyncio.gather(
    document_ai.process_document(image_bytes, "image/jpeg"),
    _gemini_document_analyze(image_bytes, doc_type),
    return_exceptions=True,
)

Google Cloud Document AI handles structured extraction, tables, form fields, key-value pairs, with accuracy that generic vision models can't match. Gemini then provides the plain-language summary and risk assessment on top of that structured data.

For entity extraction (extract_entities):
After Document AI extracts the text, we run it through Cloud Natural Language API to pull out people, organizations, dates, and monetary amounts with entity-level sentiment. This is critical for contracts and medical bills.

The confidence module:
We built a grounding module (tools/grounding.py) that checks confidence scores from every analysis and flags low-confidence results:

def check_visual_confidence(result: dict, thresholds: dict | None = None) -> dict:
    # Checks landmark, logo, and label confidence scores
    # Returns is_confident: bool + specific warnings
    ...

When confidence is low, the agent proactively asks the user for more context rather than guessing. This builds trust, users know OmniSight won't make things up.

The 13 Tools: A Specialized Toolkit for the Real World

The agent is equipped with 13 tools across 7 categories:

Vision & Reality

capture_and_analyze :- Gemini + Cloud Vision in parallel. The go-to for any identification task.
multi_frame_capture :- Captures 2-5 frames over a set interval. Used for multi-page documents, different angles, or when one frame isn't enough. Includes a consistency check across frames.
detect_text :- Precise OCR via Cloud Vision. 200+ languages.
detect_landmark :- Identifies monuments and notable buildings with GPS coordinates.
detect_logo :- Brand identification from logos.
visual_search :- Reverse image search via Cloud Vision web detection.

Document Intelligence

scan_document :- Document AI + Gemini in parallel. Handles contracts, receipts, bills, forms.
extract_entities :- Natural Language API entity extraction on scanned text.
analyze_clauses :- Gemini-powered clause analysis for contracts and legal documents.

Search & Pricing

web_search :- Real-time web search via Tavily.
compare_prices :- Product price comparison across retailers.
check_pricing :- Fair pricing check for medical bills, invoices, repair quotes.

Health & Safety

read_medication :- Scans prescription bottles. Returns name, dosage, instructions, warnings, expiry.
scan_food :- Food label analysis with allergen detection against the user's profile.
analyze_ingredients :- Deep health assessment of food ingredients.

Location (Google Maps)

find_nearby :- Google Places API. Find restaurants, pharmacies, ATMs within a radius.
get_directions :- Google Routes API. Walking, driving, transit, cycling.
geocode :- Address to coordinates conversion.

Memory (Cross-Session)

remember / recall :- Persistent MongoDB memory across sessions. The agent remembers products, prices, documents, and preferences from previous sessions.

Smart Alerts / Watchlist

add_alert / remove_alert / list_alerts / check_alerts :- Users set conditions ("alert me if you see Nike shoes under $50", "watch for peanut ingredients"). After every analysis, the agent checks observations against the watchlist and proactively notifies the user on a match.

Export

generate_session_report :- Aggregates all session findings into a structured HTML report, saved to MongoDB and accessible from the History page.

Key Technical Decisions

Singleton Clients

Every GCP client and Gemini client is initialized once as a module-level singleton:

_genai_client: genai.Client | None = None

def _get_genai_client() -> genai.Client:
    global _genai_client
    if _genai_client is None:
        _genai_client = genai.Client(api_key=GOOGLE_API_KEY)
    return _genai_client

This avoids recreating clients on every tool call, reducing connection overhead and latency significantly. The same pattern is applied to Cloud Vision, Document AI, Natural Language, and Google Maps clients.

Async-First with `run_in_executor`

All GCP client libraries are synchronous. We wrap every blocking call in asyncio.get_running_loop().run_in_executor() to avoid blocking the event loop:

async def full_analysis(image_bytes: bytes) -> dict:
    return await asyncio.get_running_loop().run_in_executor(
        None, partial(_sync_full_analysis, image_bytes)
    )

This keeps the agent responsive during long-running API calls.

Parallel API Calls

Wherever possible, we run multiple API calls in parallel using asyncio.gather:

gemini_result, vision_result = await asyncio.gather(
    gemini_analyze(image_bytes, query_hint),
    cloud_vision.full_analysis(image_bytes),
    return_exceptions=True,
)

This cuts latency roughly in half for the most common tool calls.

Centralized Configuration

All environment variables are managed in a single config.py module. No scattered os.environ calls across the codebase. This makes deployment and environment management clean and predictable.

Graceful Error Handling

Every tool returns a structured error dict rather than raising exceptions. The agent handles low-confidence results, API failures, and invalid inputs gracefully, always giving the user a useful response rather than crashing.

Google Cloud Services: The Full Stack

OmniSight is deeply integrated with Google Cloud. Here's every service we use and why:

Service	Role	Why Google
Gemini Live API (`gemini-2.5-flash-native-audio-preview`)	Real-time voice + vision	Only production-ready API with natural interruption handling
Gemini Vision (`gemini-2.5-flash`)	Image analysis, document understanding, clause analysis	Rich natural language output, multimodal reasoning
Cloud Vision API	OCR, landmark detection, logo detection, web detection	Structured, high-confidence detections with confidence scores
Document AI	Structured document extraction (forms, tables, key-value pairs)	Industry-leading accuracy for contracts, receipts, bills
Natural Language API	Entity extraction, sentiment analysis	Reliable entity detection with salience scores
Google Cloud Storage	Session recording archives	Seamless LiveKit egress integration, signed URL generation
Google Maps Places API	Nearby place search	Comprehensive POI database, real-time open/closed status
Google Maps Routes API	Turn-by-turn directions	Multi-modal routing (walking, driving, transit, cycling)
Google Maps Geocoding API	Address to coordinates	Accurate global geocoding

The integration between these services is what makes OmniSight reliable. We're not asking Gemini to guess what a document says, we're using Document AI for structured extraction and Gemini for reasoning on top of that structure. We're not asking Gemini to identify a landmark, we're using Cloud Vision for detection and Gemini for context. Every tool uses the right service for the right job.

The 7 Modes: Context-Aware Behavior

One of OmniSight's most practical features is its mode system. Before starting a session, users select a context:

General :- All-purpose visual companion
Accessibility :- Spatial descriptions, safety warnings, proactive hazard detection for visually impaired users
Education :- Patient tutor mode, step-by-step reasoning, checking questions
Healthcare :- Medication safety, allergen detection, medical document analysis
Shopping :- Price comparison, deal spotting, product identification
Travel :- Translation, landmark context, navigation, cultural tips
Professional :- Contract analysis, invoice extraction, compliance flags

Each mode injects a specialized behavioral prompt into the agent's system instructions. The same underlying tools are available in every mode, but the agent's priorities, communication style, and proactive behavior adapt to the context.

For example, in Healthcare mode, the agent always includes a "consult a doctor" disclaimer and proactively flags expired medications. In Accessibility mode, it uses spatial language ("3 feet ahead, on your left") and prioritizes safety hazards. In Professional mode, it's concise and flags legal/financial concerns immediately.

Challenges We Faced

Real-time latency: Capturing a frame, running it through Cloud Vision and Gemini in parallel, and returning a response, all while maintaining a live voice conversation, requires careful async management. The run_in_executor pattern and parallel asyncio.gather calls were essential here.

Hallucination prevention: Early versions relied too heavily on Gemini vision alone. Adding Document AI and Cloud Vision as structured validators dramatically improved reliability for high-stakes use cases. The confidence module was a late addition that made a big difference, the agent now knows when it doesn't know.

Multi-frame consistency: When capturing multiple frames for document scanning or multi-angle analysis, frames can be inconsistent (user moves the camera, lighting changes). We built a consistency checker that compares entity sets across frames using Jaccard similarity and warns the agent when frames diverge.

Cross-session memory: Designing a memory system that's useful without being noisy was tricky. We settled on a category-based approach (product, document, price, landmark, translation, preference) with keyword search, loading the 20 most recent memories at session start as context.

Results and Learnings

Building OmniSight taught us a few things about real-time multimodal agents:

Grounding matters more than model quality. A well-grounded Gemini Flash response is more reliable than an ungrounded Gemini Pro response. Specialized APIs exist for a reason, use them.

Proactive behavior is hard to tune. The agent needs to speak up when it notices something relevant, but not be annoying. The system prompt does a lot of work here, and it took many iterations to get the balance right.

The voice experience changes everything. When users can interrupt naturally and the agent recovers seamlessly, the interaction feels fundamentally different from a chat interface. Gemini Live API makes this possible in a way that turn-based systems simply can't replicate.

Cross-session memory is underrated. Users love when the agent says "Last time we looked at this TV, the best price was $749 at Amazon." It transforms a tool into a companion.

What's Next

OmniSight is one of three projects our team built for the Gemini Live Agent Challenge. We also built:

A computer control agent (UI Navigator category), an agent that observes your screen and executes actions based on natural language instructions
A 3D interactive history explorer (Creative Storyteller category), a globe you can explore, clicking locations to get rich, interleaved historical narratives with generated imagery and narration

Try It Yourself

GitHub: https://github.com/SarthakRawat-1/omni-sight
Demo Video: https://vimeo.com/1174104635?share=copy&fl=sv&fe=ci

Built with Gemini Live API, Gemini Vision, Cloud Vision, Document AI, Natural Language API, Google Cloud Storage, Google Maps API, LiveKit, FastAPI, React 19, and MongoDB.

Created for the Gemini Live Agent Challenge. #GeminiLiveAgentChallenge

Building Tapestry: How We Taught an AI to Tell History Like a Documentary

Sarthak Rawat — Mon, 16 Mar 2026 15:31:47 +0000

I created this blog for detailing about my project in Gemini Live Agent Challenge hackathon. #GeminiLiveAgentChallenge

The Problem: History Is Everywhere, But It's Buried

Drop a pin anywhere on a map. There's a story there.

The field where a decisive battle was fought. The street corner where a movement began. The mountain that an entire civilization built its mythology around. The port city that connected two worlds for five hundred years.

Most of us walk past these places — or scroll past them on a map — without any idea what happened there. And when we do get curious, we hit a wall: Wikipedia gives us a list of facts. Search gives us ten blue links. None of it feels like a story. None of it makes you feel the weight of what happened.

We built Tapestry to fix that.

Tapestry is a location-based historical research agent. You click anywhere on a globe, and Tapestry generates a rich, multi-layered historical narrative about that place — complete with AI-generated imagery, a narrated audio track, a chronological timeline, and four distinct ways to experience the story. It's not a search engine. It's a storyteller.

The Core Insight: History Has a Shape

Before we wrote a single line of code, we spent time thinking about what makes a great historical narrative. Not a Wikipedia article — a story.

Great historical storytelling has a consistent shape:

An opening — set the scene, establish the place, make the reader feel present
Origins and discovery — where did this place come from? Who was here first?
Pivotal moments — the events that changed everything
The human layer — the people, not just the events. The lives lived here.
The present day — what remains? What was lost? What does it mean now?
A closing — bring it home. Leave the reader with something to carry.

This six-act structure became the backbone of Tapestry's research schema. Every piece of content the AI generates maps to one of these stages. The result is a narrative that has rhythm and arc — not just a dump of historical facts.

The Architecture: A Research Pipeline That Thinks Like a Writer

Tapestry's backend is a multi-stage pipeline, each stage building on the last:

Stage 1: Deep Research

When a user selects a location, we kick off a parallel research sweep using Tavily (web search) and Gemini (synthesis). Tavily pulls current, grounded web sources. Gemini synthesizes them into structured research notes organized by our six-act schema.

This two-step approach is deliberate. We don't ask Gemini to invent history — we ask it to synthesize from real sources. Every claim is grounded in web-retrieved content. The result is a ResearchOutput object: a typed schema with sections, timeline entries, location coordinates, and source citations.

// The research schema that drives everything
interface ResearchSection {
  stage: 'opening' | 'discovery' | 'key_events' | 'human_layer' | 'today' | 'closing';
  title: string;
  blocks: ContentBlock[]; // text, image, map, diagram
}

Stage 2: Visual Storytelling (Interleaved Output)

This is where Tapestry goes beyond research. Once the text narrative is complete, a second Gemini pass generates the Visual Story — an interleaved sequence of narration passages and AI-generated image prompts, woven together into a single cinematic flow.

The key word is interleaved. We're not generating text and then separately generating images. We're generating a unified narrative where text and imagery are designed together — each image prompt is written to complement the passage it follows, creating a coherent visual arc across the whole story.

// Interleaved output structure
type InterleavedPart =
  | { type: 'text'; text: string }        // narration passage
  | { type: 'image'; imageUrl: string }   // AI-generated illustration

The image generation uses Google Imagen via Vertex AI. Each prompt is crafted by Gemini to match the tone and period of the narrative — a battle scene gets dramatic lighting, a founding moment gets a sense of dawn and possibility, a human story gets intimacy and warmth.

Stage 3: Audio Narration (Streaming TTS)

The text narrative doesn't just sit on the page — it speaks.

Every research output can be narrated using Google Cloud Text-to-Speech with the en-US-Journey-D voice — a natural, documentary-style voice tuned with a slightly deeper pitch (-1.0 semitones) and a measured speaking rate (0.95x) that fits long-form historical content. This is a deliberate choice: the voice sounds like a documentary narrator, not a virtual assistant.

We stream the audio in chunks: the text is split into ~50-word segments at sentence boundaries, each synthesized independently and sent to the client as a Server-Sent Event. The client starts playing the first chunk while the rest are still being generated. The result is near-instant audio playback — users hear the narration within seconds of clicking "Listen", with no waiting for the full script to synthesize.

The audio layer is what completes the interleaved experience. The Visual Story gives you text and images woven together. The narration adds a third dimension — you can read along, look at the imagery, and listen simultaneously. All three modalities telling the same story at the same time.

Stage 4: Persistence and Sharing

Everything is persisted to MongoDB: the full research output, the interleaved narrative parts, hero images, sources, and metadata. Users can return to any past research through a history sidebar. Any research can be shared via a public link with a unique token — the share page reconstructs the full experience from the stored data.

The Four Display Modes: One Story, Four Experiences

This is the part of Tapestry we're most proud of. The same historical content can be experienced in four completely different ways, each with its own visual identity and interaction model.

1. Ancient Scroll — Documentary Scroll

The default mode. The content unrolls like an ancient parchment scroll, complete with wooden rollers at the top and bottom, aged paper texture, SVG displacement filters for the wavy torn-edge effect, and parallax stain overlays that move as you scroll. Drop caps on opening paragraphs. Ornamental flourishes between sections. A ❧ divider that feels like it belongs in a medieval manuscript.

This mode is designed for reading — long-form, immersive, unhurried.

2. Stone Tablet — Timeline Explorer

A vertical chronological timeline rendered as carved stone. Dark granite background (#2a2218), chiseled text with textShadow drop and highlight, Roman numerals alongside year labels, and — the detail we spent the most time on — genuinely rough-hewn edges.

The edges use multi-layer clip-path polygons with dozens of irregular vertices, an SVG displacement filter applied on top, and a fracture highlight line at the break. The result looks like someone actually chipped this out of rock, not like a CSS border-radius.

Events animate into view with whileInView as you scroll down, with alternating left/right layout on desktop.

3. Celluloid Film — Visual Story

The interleaved narrative rendered as a vintage film strip. Warm amber-brown base (#2a1f0e), translucent sprocket holes with inset shadows, sepia-toned text, and a film grain SVG filter layered over everything.

Text passages are styled as screenplay intertitles with scene numbers. Images are styled as projected slides with light leak, vignette, and slight desaturation — like frames from an old documentary. A parallax grain overlay moves at a different rate than the content as you scroll.

This mode only activates when the Visual Story has been generated. If it hasn't, it falls back gracefully to a condensed text view.

4. Hardcover Book — Flipbook

The most technically complex mode. The content is split into pages — title page, one page per section, a timeline page, a sources page — and presented as an open hardcover book.

The left side is a leather spine with gold ornaments and the title printed vertically in writing-mode: vertical-rl. The right side is a cream page with paper grain texture, running headers, and corner page numbers.

The page-turn animation uses two stacked layers: the destination page sits underneath, and the current page folds away using rotateY from the right edge (or left edge for backward navigation), revealing the next page beneath. The back face of the folding page shows slightly darker paper — the reverse side of the leaf. A pulsing corner triangle hints that the page is turnable.

The Interleaved Output: Text + Image + Audio as One

The hackathon's Creative Storyteller category asks for agents that "think and create like a creative director, seamlessly weaving together text, images, audio, and video in a single, fluid output stream."

Tapestry's answer is three modalities working together — not sequentially, not independently, but as a unified experience.

Text + Image: Designed Together

The key is that Gemini doesn't generate text and images separately. It generates a unified creative brief — a sequence of narrative beats where each beat is either a passage of prose or an image that illustrates it. The image prompts are written with the surrounding text in mind. The prose is written knowing an image will follow. They're designed together.

Here's a simplified version of the prompt structure:

You are a documentary filmmaker and writer. Generate an interleaved narrative 
about [location] that alternates between:
- Narration passages (2-3 sentences, cinematic, present-tense)
- Image descriptions (vivid, specific, painterly — describe what the camera sees)

The narrative should follow this arc: [six-act structure]
Each image should visually complement the passage before it.
Output as a JSON array of { type: "text"|"image", content: string } objects.

The result is a story where you read a passage about the moment a city was founded, then see an AI-generated image of that dawn — torches, mud walls, the first stones being laid. Then you read about the first traders arriving, and see the harbor as it might have looked. Text and image, woven together, each making the other more powerful.

Audio: The Third Pillar

The visual story doesn't just read and show — it speaks.

Hit the narration button and the entire research output is converted to audio using Google Cloud Text-to-Speech with the en-US-Journey-D voice — a natural, documentary-style voice with a slightly deeper pitch and measured pace, tuned specifically for long-form narration. This isn't a robotic screen reader. It sounds like a BBC documentary.

The audio is synchronized with the written narrative. As the voice reads through the opening, you're looking at the same prose on screen. As it moves into the pivotal moments section, the imagery from that section is in view. The three modalities — prose, image, voice — are all telling the same story at the same time, each reinforcing the others.

Critically, the audio streams in chunks rather than waiting for the full narration to synthesize. The text is split into ~50-word segments at sentence boundaries, each synthesized independently and sent to the client as a Server-Sent Event. Playback starts within seconds of clicking the button — the first chunk plays while the rest are still being generated. This is what makes it feel live rather than like a file download.

User clicks "Listen" →
  Chunk 1 synthesized → sent → starts playing immediately
  Chunk 2 synthesized → queued → plays when chunk 1 ends
  Chunk 3 synthesized → queued → ...
  [audio plays continuously, seamlessly, while generation is still running]

The result is a genuinely immersive experience: you can read along, look at the images, and listen to the narration simultaneously. That's text, image, and audio interleaved — not three separate features, but one coherent story told across three senses.

Google Cloud Services: The Full Stack

Service	Role
Gemini 2.0 Flash	Research synthesis, narrative generation, interleaved storytelling
Google Imagen (Vertex AI)	AI image generation for the Visual Story mode
Google Cloud Text-to-Speech	Streaming audio narration (`en-US-Journey-D`)
Google Cloud Translate	Full research translation into 15+ languages
Google Maps Embed API	Interactive maps embedded in content blocks
Google Cloud Storage	Hero image and generated image hosting
MongoDB Atlas	Research persistence, history, share tokens
Next.js on Cloud Run	Frontend and API routes, server-side rendering

The pipeline is entirely Google Cloud native. Research, synthesis, image generation, audio, translation — every AI capability runs through a Google service. This isn't a wrapper around OpenAI with a Google Maps widget bolted on. It's built on the Google AI stack from the ground up.

Key Technical Decisions

Streaming Everything

Tapestry streams at every layer. The research pipeline sends Server-Sent Events as each stage completes — the client shows a live pipeline progress indicator as research, storytelling, and image generation happen in sequence. The TTS narration streams chunk by chunk so audio starts playing within seconds. The user never stares at a loading spinner waiting for everything to finish.

Typed Research Schema

The ResearchOutput schema is the contract between the AI and the UI. Every display mode reads from the same typed structure. This means we can add a new display mode without touching the research pipeline, and improve the research pipeline without touching the display modes. The schema enforces that every section has a stage, every block has a type, every source has a url.

Grounded Research — No Hallucinations by Design

This is the most important reliability decision we made, and it directly addresses the judging criterion: "Does the agent avoid hallucinations? Is there evidence of grounding?"

The answer is structural, not just a prompt instruction.

Step 1: Retrieve before generating. Before Gemini writes a single sentence, Tavily fetches real web sources for the location — Wikipedia articles, historical society pages, academic references, news archives. These are passed directly into the Gemini prompt as source material.

Step 2: Synthesize, don't invent. The research prompt explicitly instructs Gemini to synthesize from the provided sources, not to draw on training data alone. If a claim isn't supported by the retrieved sources, it shouldn't appear in the narrative.

Step 3: Cite everything. Every source retrieved by Tavily is stored in the ResearchOutput alongside a grounded: boolean flag. Sources that were directly retrieved from the web are marked grounded: true. These citations appear in the Sources page of the Flipbook and the Sources section of the Scroll — visible to the user, not hidden in metadata.

Step 4: Structured output enforces coherence. Gemini outputs to a strict JSON schema. If the output doesn't conform — wrong stage names, missing required fields, malformed blocks — the pipeline rejects it and retries. The schema is the guardrail. A hallucinated "stage" that doesn't exist in the schema simply can't make it into the final output.

The result: Tapestry's narratives are grounded in real sources, cited transparently, and structurally validated before they ever reach the user. It's not foolproof — no LLM system is — but the architecture makes hallucination significantly harder than a naive "ask Gemini about this place" approach.

Dark Mode Throughout

Every display mode has a carefully designed dark variant. The parchment scroll becomes deep amber-on-dark. The stone tablet stays dark but the chiseled text brightens. The film strip deepens to near-black with warm amber accents. The flipbook pages go to a warm near-black (#1e1c18). Dark mode isn't an afterthought — it's a first-class experience.

Challenges We Faced

Interleaved output consistency: Getting Gemini to reliably output a well-structured interleaved JSON array — with the right balance of text and image beats, and image prompts that actually match the surrounding prose — required significant prompt engineering. The final prompt includes explicit examples, length constraints per beat, and a reminder that image prompts should describe what the camera sees, not abstract concepts.

Image generation latency: Imagen takes 3-8 seconds per image. With 6-10 images per story, sequential generation would take a minute. We generate images in parallel where possible and stream them to the client as they complete, so the Visual Story mode populates progressively rather than all at once.

Page-fold animation: The Flipbook's page-turn effect went through three complete rewrites. The first used react-pageflip (which takes over the DOM with absolute positioning and breaks Tailwind's fluid layout). The second used framer-motion AnimatePresence with rotateY (which looked like a slide, not a fold). The final version uses two stacked layers — destination page underneath, current page folding away on top — with backfaceVisibility: hidden on both faces so you see the front going away and the back briefly before it's gone. That's what makes it feel like a real page turning.

TTS streaming state machine: The audio player has four states (idle, loading, playing, paused) and a chunk queue that fills asynchronously while playback is already running. The original implementation had a stale closure bug — playNextChunk was defined with empty useCallback deps, so it captured a frozen reference to the queue. We fixed it by storing the function in a ref that's updated every render, so async callbacks always call the current implementation.

Stone tablet edges: Getting the timeline's edges to look genuinely rough — not like a CSS clip-path — required two overlapping clip-path polygon layers (outer chip + inner fracture), an SVG displacement filter applied on top, and a fracture highlight line at the break. The final polygon has 40+ vertices per edge.

What We Learned

The schema is the product. The ResearchOutput schema — six stages, typed blocks, timeline entries — is what makes Tapestry work. Every display mode, every AI prompt, every database query is built around it. Getting the schema right early saved us from a dozen refactors later. The schema is also what makes grounding tractable: when the output is structured, you can validate it. When you can validate it, you can reject malformed or hallucinated content before it reaches the user.

True interleaved output is qualitatively different from "generate text, then add images." When Gemini writes the prose and the image prompts in the same pass — knowing that an image will follow each passage — the result is a coherent visual arc, not a illustrated article. The images feel like they were directed, not retrieved. That's the difference between a creative director and a search engine.

Audio completes the triad. Text and images alone are a rich experience. Add narration and it becomes something else — something that doesn't require active reading, that you can close your eyes and listen to. The en-US-Journey-D voice with its documentary cadence turns a history article into something that feels like it belongs on a streaming platform. The three modalities together are more than the sum of their parts.

Grounding is an architecture decision, not a prompt instruction. You can tell Gemini "don't make things up" in the system prompt. That helps. But what actually prevents hallucinations is retrieving real sources first, passing them as context, enforcing a strict output schema, and citing everything. The prompt instruction is the last line of defense. The architecture is the first.

Visual identity matters for immersion. We could have made all four display modes look the same — a clean card with a white background. Instead, each mode has its own material metaphor: parchment, stone, celluloid, paper. Users notice. The stone tablet feels ancient. The film strip feels like a documentary. The aesthetic isn't decoration — it's part of the storytelling. A story about ancient Rome told on a stone tablet lands differently than the same story in a sans-serif card.

Streaming changes the experience. When the research pipeline streams its progress in real time — "Researching sources... Generating narrative... Creating visual story... Generating images...", users feel like they're watching something being made. It's more engaging than a spinner. It builds anticipation. And when the audio starts playing within two seconds of clicking "Listen" — before the full narration has even finished synthesizing — it feels instant. Latency is a UX problem as much as a technical one.

Try It Yourself

GitHub: https://github.com/SarthakRawat-1/tapestry
Demo Video: https://vimeo.com/1174045515?share=copy&fl=sv&fe=ci

Built with Gemini 2.0 Flash, Google Imagen, Google Cloud Text-to-Speech, Google Cloud Translate, Google Maps, Google Cloud Storage, Next.js, Framer Motion, MongoDB, and Tailwind CSS — all running on Google Cloud.

Created for the Gemini Live Agent Challenge. #GeminiLiveAgentChallenge

I Built a React Library That Looks Like It Was Scribbled in a Notebook

Sarthak Rawat — Mon, 09 Mar 2026 01:45:00 +0000

The Problem With Modern UI

We've all been there. You're building a side project at 2 AM, everything's working fine, but then you step back and look at your creation. It's functional. It's clean. It's also... painfully generic.

Another white card. Another blue button with perfect 4px border radius. Another system font.

Don't get me wrong, I love Tailwind. I use it daily. But sometimes you want your project to feel alive. Like a human actually touched it.

My "Aha" Moment

4 months ago, I was drawing diagrams in my exam book. My pen lines were crooked. The borders wobbled. Little doodles appeared in the margins when I got bored.

And I thought: why don't web interfaces look like this?

Not the crooked part, nobody wants broken layouts. But the warmth. The imperfection. The sense that someone cared enough to draw this by hand rather than drag it from a Figma template.

So I started building Sketchbook UI.

What Makes It Different

Most "alternative" UI libraries go for glassmorphism or neumorphism or whatever the latest Dribbble trend is. They still feel digital.

Sketchbook UI uses SVG paths that look like Bézier curves drawn by someone who's had too much coffee. Every component has:

Wobbly borders - Not broken, just... organic
Paper texture overlays - Subtle noise patterns you notice when you look closely
Hand-drawn focus rings - Dashed lines that look like they were sketched with a pencil
The Caveat font - If you haven't seen it, imagine a designer's handwriting at 3 AM

The Button component alone has 335 lines of code. Most of that is SVG path data. I spent an embarrassing amount of time tweaking control points to make the corners look "accidentally perfect."

"But Is It Production-Ready?"

Fair question. I asked myself this a lot while building it.

Here's the thing, it's not trying to be Ant Design. It won't power your enterprise dashboard (though honestly, it could). It's built for:

Side projects you actually want to show people
Landing pages that need to stand out in a sea of sameness
Creative tools where the interface should feel approachable
Portfolios of designers who want to practice what they preach

It's fully typed. Tree-shakeable. Under 70KB gzipped. It has Storybook docs, keyboard navigation, and respects prefers-reduced-motion.

But more importantly? It has personality.

The Technical Stuff (For The Nerds)

Under the hood, it's SVG-heavy. Every component measures itself with ResizeObserver and generates paths dynamically. The noise textures use SVG filters with feTurbulence, that fractal noise thing that gives you grain without bitmap images.

The color system is simple: paper background (#faf7f0), pencil strokes (#2a2a2a), and overlays for depth. But every component accepts a colors prop if you want to go wild with purple paper and green ink.

Oh, and the components has a hidden feature: it measures your text content and expands beyond its minimum width if needed. So "Submit" looks compact, but "Create New Repository" gets the space it deserves without breaking the sketch aesthetic.

Real Talk: Who Should Use This?

If you're building the next Stripe dashboard, maybe stick to their design system.

But if you're making:

A personal blog that doesn't look like every other Ghost template
A creative tool for writers or designers
A landing page for your indie hacker project
Literally anything where you want someone to say "oh, this is cute"

Then yeah. This is for you.

Check this video to see how each components look like :- Youtube Link

Try It. Or Don't. But Maybe Do?

I'm not going to hard sell you. The code is open source. The Storybook docs are live. You can play with every component in your browser right now without installing anything.

If you like it, ⭐ the repo. If you love it, use it in your next project. If you hate it... well, you probably stopped reading four paragraphs ago.

Live demo: sarthakrawat-1.github.io/sketchbook-ui

GitHub: github.com/SarthakRawat-1/sketchbook-ui

Thanks for Reading

Let's build a Full Text Search Engine in Python

Sarthak Rawat — Thu, 26 Feb 2026 16:02:36 +0000

Ever wondered how Google finds what you're looking for in milliseconds? Or how Wikipedia's search instantly surfaces the right article? It's all powered by full-text search — a technique that transforms messy, unstructured text into something computers can query efficiently. Let's build one from scratch.

How Search Actually Works

At its heart, every search engine does two things: it pre-processes documents once (indexing), then answers queries super fast using that pre-built index. The trick is doing the heavy lifting upfront so searches feel instant.

Turning Text Into Searchable Tokens

Try searching for "running cats" in a document that says "The cat runs fast." A simple string match would fail — "running" ≠ "runs" and "cats" ≠ "cat". We need to normalize text so semantically similar words match.

Here's the pipeline we use:

Stage	What Happens	Example
Tokenization	Split into words	"The cat runs fast." → ["the", "cat", "runs", "fast"]
Lowercasing	Make it case-insensitive	["the", "cat", "runs", "fast"]
Stopword Removal	Drop common words	["cat", "runs", "fast"]
Stemming	Reduce to root form	["cat", "run", "fast"]

Now "running cats" and "The cat runs fast" both become ["run", "cat"] — they match!

The Magic: Inverted Index

Instead of storing documents as-is, we flip the relationship. Instead of "Document 1 contains [cat, dog]", we store:

"cat" → [1, 3, 7, 12]    (documents containing "cat")
"dog" → [2, 3, 9]        (documents containing "dog")

Now finding documents with "cat" is just a hash lookup — O(1) instead of scanning everything.

Multi-word searches? Easy. For "cat AND dog":

Get both lists: [1, 3, 7, 12] and [2, 3, 9]
Find common IDs: [3]
Document 3 has both words

Let's Build It

Data Model

from dataclasses import dataclass

@dataclass
class Document:
    title: str
    url: str
    text: str
    id: int = 0

Text Analysis

import string
from typing import List, Set
from collections import defaultdict

try:
    from nltk.stem import PorterStemmer
    nltk_available = True
except ImportError:
    nltk_available = False

class TextAnalyzer:
    STOP_WORDS: Set[str] = {
        'a', 'an', 'and', 'be', 'have', 'i', 'in', 'of', 'that', 'the', 'to',
        'is', 'it', 'for', 'with', 'as', 'on', 'at', 'by', 'from'
    }

    def __init__(self):
        self.stemmer = PorterStemmer() if nltk_available else None

    def tokenize(self, text: str) -> List[str]:
        text = text.lower()
        for char in string.punctuation:
            text = text.replace(char, ' ')
        return [t for t in text.split() if t]

    def analyze(self, text: str) -> List[str]:
        tokens = self.tokenize(text)
        tokens = [t for t in tokens if t not in self.STOP_WORDS]
        if self.stemmer:
            tokens = [self.stemmer.stem(t) for t in tokens]
        return tokens

The stemmer here is the classic Porter algorithm from 1980 — it strips suffixes like "-ing", "-ed", "-s" to get word roots.

The Index Itself

from typing import Dict, List, Optional, Set

class InvertedIndex:
    def __init__(self):
        self.index: Dict[str, List[int]] = defaultdict(list)
        self.analyzer = TextAnalyzer()
        self.documents: Dict[int, Document] = {}

    def add_documents(self, documents: List[Document]):
        for doc in documents:
            self.documents[doc.id] = doc
            tokens = self.analyzer.analyze(doc.text)

            for token in tokens:
                if not self.index[token] or self.index[token][-1] != doc.id:
                    self.index[token].append(doc.id)

    def search(self, query: str) -> List[int]:
        """AND search - all terms must match"""
        tokens = self.analyzer.analyze(query)
        if not tokens:
            return []

        result = None
        for token in tokens:
            if token not in self.index:
                return []
            if result is None:
                result = self.index[token][:]
            else:
                result = self._intersection(result, self.index[token])

        return result if result else []

    def _intersection(self, a: List[int], b: List[int]) -> List[int]:
        """Two-pointer merge for O(n+m) intersection"""
        result = []
        i = j = 0
        while i < len(a) and j < len(b):
            if a[i] < b[j]:
                i += 1
            elif a[i] > b[j]:
                j += 1
            else:
                result.append(a[i])
                i += 1
                j += 1
        return result

Loading Real Wikipedia Data

Wikipedia publishes "stub articles" — files with all 7+ million article titles and metadata, but without the full article text. Perfect for building a title search engine, and much smaller than full dumps (~600MB vs 50GB+).

Grab one of the 27 parts:

wget https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-stub-articles14.xml.gz
gunzip enwiki-latest-stub-articles14.xml.gz

These files have a quirk: the <text> element is empty. The actual content lives in external storage. So we index article titles instead — which works great for finding articles by name.

import xml.etree.ElementTree as ET
from typing import List

def load_documents(path: str) -> List[Document]:
    tree = ET.parse(path)
    root = tree.getroot()

    # Handle XML namespace - required for Wikipedia dumps
    ns = {'mw': 'http://www.mediawiki.org/xml/export-0.11/'}
    pages = root.findall('mw:page', ns) or root.findall('.//page')

    documents = []
    for i, page in enumerate(pages):
        title = page.findtext('mw:title', '', ns) or page.findtext('title', '')
        url = f"https://en.wikipedia.org/wiki/{title.replace(' ', '_')}"

        # Stub articles have no content text - use title as searchable content
        documents.append(Document(
            title=title,
            url=url,
            text=title,
            id=i
        ))

    return documents

Performance

Running this on enwiki-latest-stub-articles14.xml (~600MB, ~780K articles):

Operation	Time
XML Parsing	~28 seconds
Index Building	~1.6 seconds
Search	<1 millisecond
Memory (index)	~50MB

That sub-millisecond search time holds even as you add more documents — we're only intersecting posting lists for query terms, not scanning everything.

Full Working Code

import xml.etree.ElementTree as ET
import time
import sys
from dataclasses import dataclass
from typing import List, Dict, Optional
from collections import defaultdict

try:
    from nltk.stem import PorterStemmer
    nltk_available = True
except ImportError:
    nltk_available = False

@dataclass
class Document:
    title: str
    url: str
    text: str
    id: int = 0

class TextAnalyzer:
    STOP_WORDS = {'the', 'a', 'an', 'and', 'or', 'but', 'in', 'on', 'at', 'to', 'for', 'of', 'with', 'by'}

    def __init__(self):
        self.stemmer = PorterStemmer() if nltk_available else None

    def analyze(self, text: str) -> List[str]:
        text = text.lower()
        for char in '.,!?;:"()[]{}':
            text = text.replace(char, ' ')
        tokens = [t for t in text.split() if t and t not in self.STOP_WORDS]
        if self.stemmer:
            tokens = [self.stemmer.stem(t) for t in tokens]
        return tokens

class InvertedIndex:
    def __init__(self):
        self.index: Dict[str, List[int]] = defaultdict(list)
        self.analyzer = TextAnalyzer()
        self.documents: Dict[int, Document] = {}

    def add_documents(self, documents: List[Document]):
        for doc in documents:
            self.documents[doc.id] = doc
            for token in self.analyzer.analyze(doc.text):
                if not self.index[token] or self.index[token][-1] != doc.id:
                    self.index[token].append(doc.id)

    def search(self, query: str) -> List[int]:
        tokens = self.analyzer.analyze(query)
        if not tokens:
            return []

        result = None
        for token in tokens:
            if token not in self.index:
                return []
            if result is None:
                result = self.index[token][:]
            else:
                result = [d for d in result if d in set(self.index[token])]
        return result if result else []

    def get_document(self, doc_id: int) -> Optional[Document]:
        return self.documents.get(doc_id)

def load_documents(path: str) -> List[Document]:
    tree = ET.parse(path)
    root = tree.getroot()
    ns = {'mw': 'http://www.mediawiki.org/xml/export-0.11/'}
    pages = root.findall('mw:page', ns) or root.findall('.//page')

    documents = []
    for i, page in enumerate(pages):
        title = page.findtext('mw:title', '', ns) or page.findtext('title', '')
        url = f"https://en.wikipedia.org/wiki/{title.replace(' ', '_')}"
        documents.append(Document(title=title, url=url, text=title, id=i))

    return documents

def main():
    if len(sys.argv) != 2:
        print("Usage: python search.py <path-to-wikipedia-xml>")
        print("\nDownload Wikipedia stub articles:")
        print("  wget https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-stub-articles14.xml.gz")
        print("  gunzip enwiki-latest-stub-articles14.xml.gz")
        sys.exit(1)

    xml_path = sys.argv[1]
    print(f"Loading documents from: {xml_path}")

    start = time.time()
    documents = load_documents(xml_path)
    print(f"Loaded {len(documents)} documents in {time.time()-start:.2f}s")

    print("Building index...")
    start = time.time()
    index = InvertedIndex()
    index.add_documents(documents)
    print(f"Index built in {time.time()-start:.2f}s")
    print(f"Index contains {len(index.index)} unique tokens")

    while True:
        query = input("\nSearch (or 'quit'): ").strip()
        if query.lower() == 'quit':
            break

        start = time.time()
        results = index.search(query)
        elapsed = (time.time() - start) * 1000

        print(f"Found {len(results)} results in {elapsed:.2f}ms")
        for doc_id in results[:10]:
            doc = index.get_document(doc_id)
            print(f"  [{doc_id}] {doc.title}")

if __name__ == "__main__":
    main()

Running It

pip install nltk
wget https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-stub-articles14.xml.gz
gunzip enwiki-latest-stub-articles14.xml.gz
python search.py enwiki-latest-stub-articles14.xml

Gallery

Where to Go From Here

This is Boolean search — results either match or they don't. Real search engines add ranking:

TF-IDF: Score by how rare/important words are
BM25: Probabilistic scoring used by Elasticsearch
Vector search: Semantic similarity using embeddings

But the inverted index remains the foundation everything else builds on.

Forem: Sarthak Rawat

Let's build a Production-Grade Bloom Filter in Python

What's a Bloom Filter?

The Trade-off

How It Works

The Math Behind It

Let's Build It

Storage Backend

Text Analysis

Adding Statistics

File Persistence

Serialization

Set Operations

Complete Working Example

Gallery

Performance Numbers

Real-World Applications

1. Cache Filtering

2. Database Query Optimization

3. Web Crawler Deduplication

4. Distributed Systems

Limitations and Gotchas

No Deletions

False Positives Increase with Saturation

Hash Function Quality Matters

Where to Go From Here

Building TaskPilot: An AI Agent That Sees Your Screen and Takes Control

The Problem: Automation That Breaks the Moment the UI Changes

The Architecture: Two Agents, One Interface

The 5-Layer Pipeline: Why We Don't Always Need Vision

The Browser Agent: Gemini Computer Use in a Loop

The WebSocket Server: Bridging Frontend to Agent

Voice Input: Talking to Your Agent

Safety: The Agent Needs to Know When to Ask

Provider-Agnostic by Design

Challenges We Faced

What We Learned

What's Next

Try It Yourself

Building OmniSight: A Real-Time AI Visual Companion Powered by Gemini Live and Google Cloud

The Problem: The World Doesn't Fit in a Text Box

The Architecture: How It All Fits Together

The Grounding Strategy: Why We Don't Just Use Gemini Vision Alone

The 13 Tools: A Specialized Toolkit for the Real World

Vision & Reality

Document Intelligence

Search & Pricing

Health & Safety

Location (Google Maps)

Memory (Cross-Session)

Smart Alerts / Watchlist

Export

Key Technical Decisions

Singleton Clients

Async-First with run_in_executor

Parallel API Calls

Centralized Configuration

Graceful Error Handling

Google Cloud Services: The Full Stack

The 7 Modes: Context-Aware Behavior

Challenges We Faced

Results and Learnings

What's Next

Try It Yourself

Building Tapestry: How We Taught an AI to Tell History Like a Documentary

The Problem: History Is Everywhere, But It's Buried

The Core Insight: History Has a Shape

The Architecture: A Research Pipeline That Thinks Like a Writer

Stage 1: Deep Research

Stage 2: Visual Storytelling (Interleaved Output)

Stage 3: Audio Narration (Streaming TTS)

Stage 4: Persistence and Sharing

The Four Display Modes: One Story, Four Experiences

1. Ancient Scroll — Documentary Scroll

2. Stone Tablet — Timeline Explorer

3. Celluloid Film — Visual Story

4. Hardcover Book — Flipbook

The Interleaved Output: Text + Image + Audio as One

Text + Image: Designed Together

Audio: The Third Pillar

Async-First with `run_in_executor`