Forem: Chidari Sandeep

Why I rewrote my 90+ Engine Meta-Search in Rust 🦀

Chidari Sandeep — Wed, 06 May 2026 10:13:15 +0000

Just testing out my automated dev-log pipeline for SearchWala. Moving from Python to Rust dropped my RAM from 512 MB → 38 MB and made cold starts nearly instant. Here's the short version of why and how.

The Problem

SearchWala aggregates results from 90+ search engines — Google, Bing, DuckDuckGo, Brave, Mojeek, and dozens of niche/academic sources. The original Python stack (FastAPI + asyncio + BeautifulSoup) worked, but:

RAM hungry: Each worker held parsed DOM trees in memory. Under load, a single instance ate ~512 MB.
Cold start pain: On a fresh container, Python import chains + dependency init took 4-6 seconds.
GIL bottleneck: True parallelism across 90 engines was faked with async I/O, but CPU-bound parsing still serialized.

The Rust Rewrite

I rewrote the core in Rust using tokio for async, reqwest for HTTP, and scraper for HTML parsing. The results:

Metric	Python	Rust
RAM (idle)	512 MB	38 MB
Cold start	4.2s	0.3s
P95 latency (90 engines)	2.8s	0.9s
Binary size	~180 MB (venv)	12 MB

The dual-path LLM synthesis pipeline (lite mode for speed, research mode for depth) stayed as a sidecar microservice, but all search orchestration, ranking (BM25 + Reciprocal Rank Fusion), and content extraction now run natively in Rust.

Key Takeaway

If your I/O-heavy Python service is eating memory and you need predictable latency — Rust with tokio is the move. Not everything needs a rewrite, but the hot path absolutely does.

Check out the full source code and drop a star on GitHub: SearchWala on GitHub

Building a 90+ engine meta-search in Rust [TEST-0x8A7B]

Chidari Sandeep — Wed, 06 May 2026 08:57:53 +0000

Just testing out my automated dev-log pipeline for SearchWala. Moving from Python to Rust dropped my RAM usage significantly.

Building a 90+ engine meta-search in Rust

Chidari Sandeep — Wed, 06 May 2026 08:50:29 +0000

I just set up an automated dev

SearchWala: I Built a Blazing-Fast Meta-Search Engine in Rust That Queries 90+ Engines Simultaneously

Chidari Sandeep — Tue, 05 May 2026 12:10:48 +0000

Hey devs! I want to share a project I have been working on called SearchWala (Swift-Search-RS).

What is it?

SearchWala is an open-source meta-search engine written in Rust that queries 90+ search engines simultaneously including Google, Bing, DuckDuckGo, Brave, Yandex, Baidu, and many more. It aggregates the results, scrapes the actual content, and can synthesize an AI-powered answer using any LLM provider you bring (Groq, OpenAI, Cerebras, etc).

Key Features

90+ Search Engines: Google (15 regional variants), Bing, DuckDuckGo, Brave, Yahoo, Qwant, Mojeek, Startpage, Wikipedia, and 60+ more
Dual LLM Pipeline: Lite mode (fast BM25-ranked answer) and Research mode (iterative deep analysis across 200+ sources)
5-Tier Content Extraction: Intelligent article extraction that goes beyond simple scraping
Built in Rust: Blazing fast with async Tokio runtime, 24 concurrent scrapers
BYOK (Bring Your Own Key): Works with any OpenAI-compatible LLM provider
Privacy First: No tracking, no ads, self-hostable
Docker Ready: Multi-stage build producing a tiny 15MB image

Performance

Scrapes and ranks 420 URLs in under 10 seconds
BM25 ranking with phrase match bonuses
Browser fingerprint rotation across 12 profiles to avoid blocks

Try It

GitHub: https://github.com/SandeepAi369/Swift-Search-Rs

Would love to hear your feedback! Drop a comment if you have questions.