Forem: Aadarsh Nagrath

SoraStrip: Building a Sora Watermark Removal Tool with API-First Architecture

Aadarsh Nagrath — Fri, 24 Oct 2025 20:32:44 +0000

When OpenAI released Sora, their groundbreaking text-to-video AI model, content creators worldwide gained access to cinematic-quality video generation. But there was a catch—every video came with an embedded watermark. While this serves OpenAI's attribution and content tracking purposes, it creates a significant workflow friction for creators who need clean, production-ready videos.

After seeing this gap in the market, I built SoraStrip—not just another watermark removal tool, but a developer-first platform with comprehensive API support, enterprise-grade reliability (99.9% uptime SLA), and webhook notifications for real-time processing updates. Here's the technical story behind it, the architectural decisions that set it apart, and why API accessibility matters in the AI video tooling ecosystem.

The Problem: Beyond Simple Watermark Removal

Most existing watermark removal solutions treat Sora videos like any other media file. They either:

Force you to upload large video files (bandwidth-heavy, slow)
Use generic removal algorithms that degrade quality
Lack programmatic access for automation
Price themselves out of reach for independent developers

The real challenge isn't just removing pixels—it's building a system that understands Sora's specific share link architecture, preserves video quality, and scales efficiently while remaining affordable.

Understanding Sora's Video Architecture

Sora generates videos and hosts them at unique share URLs following the pattern: https://sora.chatgpt.com/p/s_xxxxxxxxxxxxx

This is actually brilliant from a technical standpoint. Instead of requiring users to download, store, and re-upload multi-megabyte video files, we can work directly with these share links. This architectural decision became the foundation of SoraStrip's link-based processing approach.

Why Link-Based Processing Matters

When you pass a Sora share link to SoraStrip, here's what happens under the hood:

Link Validation: The system validates the URL structure and checks accessibility
Metadata Extraction: We fetch video metadata (resolution, duration, format) without downloading the full file
Streaming Processing: The watermark removal happens in a streaming pipeline, not in-memory
Direct Output: You receive a clean MP4 link—no file management overhead

This approach reduces processing time by 60-70% compared to upload-based workflows and eliminates storage costs entirely.

The API-First Design Philosophy

Here's where SoraStrip diverges from the crowded watermark removal space: we built the API first, and the web interface second.

Why API Support Is Rare in This Space

After analyzing 20+ Sora watermark removal tools, I found that fewer than 15% offer any API access. Of those that do:

Most charge enterprise-level pricing ($500-2000/month minimum)
They require complex authentication schemes
Rate limits are restrictive (often 10-50 requests/day)
Documentation is sparse or outdated

The reason is simple: most tools are built as consumer web apps first. Retrofitting API access is architecturally complex and doesn't align with their business model.

SoraStrip's API Architecture

We flipped this model. The web interface you see at SoraStrip is actually consuming the same REST API that developers can access. This means:

Single Endpoint Simplicity

POST https://api.sorastrip.app/api/v1/process
Authorization: Bearer sk-your_api_key_here
Content-Type: application/json

{
  "sora_url": "https://sora.chatgpt.com/p/s_XXXXXXXXXXXXXX"
}

Lightweight Authentication
No OAuth complexity. Simple API key authentication via headers (format: sk-xxxxxxxx-...). Generate keys instantly from your dashboard—available for Pro and Ultimate plan users.

Transparent Rate Limiting

Free: 1 removal/day (ideal for testing/personal projects)
Pro: 100 removals/billing cycle ($9)
Ultimate: Unlimited ($50)

All limits are clearly communicated in response headers, making it easy to build retry logic.

Real-World API Use Cases

The API support unlocks scenarios that web-only tools can't address:

1. Automated Social Media Publishing Pipelines
Generate Sora videos → Remove watermark → Auto-post to Instagram/TikTok
All without manual intervention.

2. Batch Processing for Marketing Agencies
Process 50+ client videos overnight with our distributed infrastructure. Submit multiple requests simultaneously and receive webhook callbacks as each video completes.

3. Integration with Video Editing Workflows
Pull clean videos directly into DaVinci Resolve, Premiere Pro, or CapCut via custom scripts.

4. SaaS Product Integration
Build watermark removal into your own video editing SaaS without reinventing the wheel. Our API handles the heavy lifting.

The Economics: Why Affordable API Pricing Matters

Let's talk numbers. Here's what comparable API-enabled watermark removal typically costs:

Image watermark APIs (Dewatermark, PicWish): $0.10 per credit
Generic video APIs: $0.50-2.00 per minute processed
Enterprise video APIs: $500+ monthly minimums

For Sora-specific processing:

Kie AI API: $0.05 per use (10 credits)
Most competitors: No API, or "Contact Sales" pricing

SoraStrip's API pricing (Pro tier):

$9/month for 100 removals/billing cycle = $0.09 per video
Unlimited tier: $50/month = unlimited usage (~$0.016 per video at 3000/month)

This isn't just cheaper—it's 3-5x more affordable than comparable services, and the only Sora-specific tool offering genuine unlimited API access with a 99.9% uptime SLA.

Technical Deep-Dive: How Watermark Removal Actually Works

Without revealing proprietary algorithms, here's the high-level technical approach:

1. Watermark Detection

Sora embeds watermarks with specific characteristics:

Position: Dynamic (can move within frames)
Opacity: Semi-transparent (15-30% typically)
Type: Text + logo combination

2. Content-Aware Removal

This is where quality preservation happens. We use:

Context prediction: ML models predict what pixels should exist behind the watermark
Temporal consistency: Ensure frame-to-frame coherence (no flickering)
Edge preservation: Maintain sharp details around removal regions
Distributed processing: Our infrastructure can handle hundreds of concurrent requests for enterprise-scale batch processing

3. Quality Preservation

Unlike tools that re-encode at lower bitrates, SoraStrip:

Maintains original resolution (up to 4K)
Preserves frame rate
Keeps original color profiles
Uses minimal re-encoding (only affected regions)

Result: Output videos are perceptually identical in quality to input.

The Developer Experience (DX) Focus

Good DX isn't just about documentation—it's about removing friction:

1. Instant Onboarding
Sign in with Google → Get 1 free removal/day → Test immediately.
No credit card, no approval process. Upgrade to Pro or Ultimate to unlock API access.

2. Clear Upgrade Paths
Polar integration means one-click upgrades with automatic API key provisioning. API keys follow the sk-xxxxxxxx-... format for easy identification.

3. RESTful API + SDKs Coming
Our RESTful API works with any programming language and framework today. Python, JavaScript, and Ruby SDKs launching soon to abstract away HTTP calls entirely.

Roadmap: What's Coming Next

Based on user feedback and technical feasibility:

Already Shipped:

✅ Batch processing via API (process multiple videos simultaneously)
✅ Webhook support for real-time processing notifications
✅ 99.9% uptime SLA with distributed infrastructure

Coming Soon:

Python and JavaScript SDKs
Team accounts with shared usage pools
White-label API for agencies
Video editing features (trim, crop, merge) post-removal
Support for other AI video platforms (Runway, Pika)

Getting Started: From First Test to Production

Step 1: Free Trial
Visit SoraStrip, sign in with Google, paste a Sora link. See results in seconds.

Step 2: Upgrade Based on Volume

Personal projects: Pro ($9/month)
Agencies/high-volume: Ultimate ($50/month unlimited)

Step 3: API Integration
Grab your API key from the dashboard (Pro/Ultimate plans only). Start with:

export SORASTRIP_API_KEY="sk-your_api_key_here"

curl -X POST https://api.sorastrip.app/api/v1/process \
  -H "Authorization: Bearer $SORASTRIP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "sora_url": "https://sora.chatgpt.com/p/s_XXXXXXXXXXXXXX"
  }'

Response:

{
  "status": "success",
  "processed_url": "https://cdn.sorastrip.com/xyz123.mp4",
  "processing_time": "2.3s"
}

Step 4: Scale
Monitor usage, upgrade tiers as needed, and explore bulk processing features.

The Broader Lesson: Building Developer Tools in the AI Era

SoraStrip is a case study in a broader trend: AI tools need API-first architectures from day one.

As AI capabilities proliferate (text-to-video, image generation, audio synthesis), the creators who win won't just build powerful algorithms—they'll build platforms that other developers can integrate into.

Key principles:

API-first design (not an afterthought)
Transparent, predictable pricing (no "contact sales")
Developer experience focus (docs, SDKs, examples)
Ethical considerations (clear responsible use policies)

Conclusion: Why SoraStrip Exists

OpenAI's Sora democratized cinematic video creation. SoraStrip democratizes access to clean, production-ready Sora videos—especially for developers building the next generation of content tools.

If you're:

A content creator tired of manual watermark editing
A developer building video automation pipelines
An agency processing client Sora videos at scale
A founder integrating AI video into your product

SoraStrip's combination of simple web interface, robust API, and affordable pricing was built for you.

The future of AI video tools isn't just about generation—it's about seamless post-processing workflows. And that future needs to be accessible, affordable, and developer-friendly.

Try SoraStrip: sorastrip.app (Sign in with Google for 1 free daily removal)

API Documentation: sorastrip.app/api-docs (Pro/Ultimate plans)

Questions? Email: sorastrip@gmail.com

Frequently Asked Questions (Technical Edition)

Q: How does link-based processing compare to upload-based tools?
Link-based eliminates 200-500MB file transfers per video. Processing starts immediately, and outputs are served via CDN links—no re-downloading needed.

Q: Can I process private/unlisted Sora videos?
Only publicly shared links work. Regenerate your share link with public visibility in Sora.

Q: What's the API response time SLA?
P95 latency: <5 seconds for typical Sora videos (3-5 seconds duration). 99.9% uptime SLA on Pro and Ultimate tiers with automatic failover.

Q: How do I get API access?
API keys are available exclusively for Pro ($9/month) and Ultimate ($50/month) plan users. Generate keys instantly from your dashboard after upgrading—no approval process or waiting period.

Keywords for SEO:

Sora watermark removal, SoraStrip API, remove Sora watermark, OpenAI Sora video, Sora video API, watermark remover API, AI video editing, Sora share link processing, affordable video API, developer video tools, Sora video clean, text-to-video AI, ChatGPT Sora, remove OpenAI watermark, video watermark API, programmatic watermark removal

This article was written to share the technical journey behind building SoraStrip. If you're building in the AI video space, I'd love to connect and hear what you're working on.

Picking Your Automation: Zapier, Make, or n8n?

Aadarsh Nagrath — Tue, 07 Oct 2025 19:31:09 +0000

Hey, if you're knee-deep in apps and feeling like your workflow's a tangled mess, automation tools aren't just nice-to-haves—they're the quiet heroes keeping things from falling apart.

Solo hustler? Small team grinding? Or enterprise trying not to drown in tabs? Zapier, Make (ex-Integromat), and n8n are the heavy hitters here, each with its own vibe for stitching services together without the headache.

As of October 2025, I've dug into the latest—fresh pricing pulls, user chatter on X and Reddit, and those deep-dive comparisons floating around.

We'll cover the basics like cost and connections, plus deeper cuts on security, support, real use cases, and even switching between 'em. Goal? Arm you to pick without second-guessing. No hype, just straight talk.

Quick Hits (TL;DR)

Zapier: Easiest entry for non-coders, massive app library (8,000+), but costs add up quick on volume. Great for quick AI tweaks.
Make: Visual wizard for complex no-code flows, solid mid-range pricing, handles data like a champ. Balanced pick for teams.
n8n: Dev's dream—open-ish, self-hostable, crazy flexible for AI and custom stuff. Cheapest long-term if you manage your own setup.

Newbie or hate code? Zapier. Visual power without scripts? Make. Tinkerer or budget hawk? n8n. Test 'em free and see what clicks.

Pricing: Where Your Wallet Feels It

Costs evolve, so here's the October 2025 rundown—pulled straight from their sites. They bill on usage (tasks, ops, executions), but models vary. Annual saves 15-20% usually.

Tool	Free Tier	Starter Paid (Annual Mo. Est.)	Scaling Model & Mid-Tier	High-Volume/Enterprise	Gotchas
Zapier	100 tasks/mo; basic Zaps, Tables, Interfaces	Pro: $20/mo (2K tasks)	Tasks per action (e.g., $49/mo Pro for 10K); pay-as-you-go overage at 1.25x	Custom (2M+ tasks); dedicated support	Tasks count per successful step—multi-step eats quota fast. AI calls (via MCP) = 2 tasks each.
Make	1K ops/mo; 2 scenarios, 512MB data	Core: $9/mo (10K ops)	Ops per module (Pro $16/mo 10K ops; Teams $29/mo); unlimited scenarios	Custom; overage protection	Ops include filters/transfers—predictable for chains. AI agents beta included higher up.
n8n	Community: Unlimited self-host (your server costs ~$5-20/mo)	Cloud Starter: €20/mo (2.5K execs)	Executions per full workflow (Pro €50/mo 10K); unlimited steps	Business €667/mo (40K self-host); Enterprise custom	Executions = whole run, not per step—huge win for complex. Startup discount if <20 folks.

n8n wins for heavy users (10-50x cheaper on intricate stuff), Make for steady mid-volume without infra hassle, Zapier if you value simplicity over savings. Recent n8n tweak: Dropped workflow caps, but cloud bumped slightly for concurrency.

Integrations: Hooking Up Your Stack

It's all about breadth vs. bendiness—how many apps, and can you Frankenstein the rest?

Zapier: 8,000+ pre-builts, from Slack to obscure CRMs. If it's SaaS-y, it's there. Recent adds: 73 in Sept '25 alone. Killer for off-the-shelf.
Make: 3,000+ apps, plus HTTP magic for APIs. Visual drag-drop shines for chaining weird combos, like custom webhooks.
n8n: 350+ core nodes, but HTTP/GraphQL/cURL imports let you build anything. Community nodes push it to 1,000+ effective. Devs rave about extending it.

Zapier for volume, n8n for "anything goes," Make in the middle with pretty diagrams. X users echo: n8n's snappier for custom APIs.

Dev Tools: Code If You Want, Skip If You Don't

From snippets to full scripts—depends on your comfort.

Zapier: JS/Python in "Code Steps," but timed out quick, no libs. Good for light math or fetches.
Make: JS customs on Enterprise; otherwise, modules for logic. No-code first, code as add-on.
n8n: Native JS/Python nodes, npm installs (self-host), bash runs. Import cURL, debug live—feels like VS Code lite.

n8n laps 'em for coders; others keep it visual.

Error Handling: Bouncing Back from Glitches

Shit hits the fan—how do they recover?

Zapier: Retries, basic notifications, error paths in Zaps. Solid but shallow.
Make: Ignore/resume/rollback per module, full logs/search. Dashboards flag issues early.
n8n: Node-specific catches, error workflows, replays. Alerts to anywhere, unlimited logs self-hosted.

Make/n8n tie for pros; Zapier's fine for starters.

AI Smarts: Leveling Up with Bots and Agents

AI's everywhere now—summaries, agents, RAGs.

Zapier: MCP for 8K+ AI actions (2 tasks/call), Copilot builder, beta chatbots. 800K daily AI tasks automated. Easy for enrichment.
Make: 350+ AI apps, beta Agents with your LLM key, Vision/Text tools. Visual agent mapping.
n8n: 70+ nodes, LangChain deep-dive for RAG/agents/vectors. Local models, full pipelines—build products, not just tasks.

n8n for AI builders, Zapier for plug-in ease, Make for no-code agents.

Team Play: Sharing Without the Chaos

Collab keeps it smooth.

Zapier: Shared workspaces Team+ ($69/mo), basic roles.
Make: Unlimited users, roles/templates on Teams ($29/mo), API limits scale.
n8n: Unlimited users all tiers, RBAC/Git sync. Projects segment access.

n8n/Make flex better; Zapier catches up at scale.

Scaling: From Side Gig to Beast Mode

Growth test: High volume? Complex?

Zapier: Handles 2M tasks easy, but costs/logic branches strain.
Make: Unlimited scenarios, 1GB files Enterprise, API bursts to 1K/min.
n8n: Kubernetes/queues for infinite, 200+ concurrent Enterprise. Self-host = your limits.

n8n for infinite tweak, Make for managed growth.

Pros & Cons: At a Glance

Quick gut-check.

Zapier

Pros: Dead simple UI, huge integrations, managed everything.
Cons: Pricey at volume, less dev depth, cloud-only.

Make

Pros: Visual flows rock, affordable ops, strong monitoring.
Cons: Code locked high-tier, fewer apps than Zapier.

n8n

Pros: Cheap/flexible, AI/dev heaven, self-host privacy.
Cons: Setup curve if hosting, smaller out-of-box library.

Real-World Use Cases

From recent chats:

Marketing team: Zapier for lead Slack pings + email nurtures. Quick, no fuss.
E-com ops: Make routing orders to inventory/shipping, with error rollbacks. Handles spikes.
Dev startup: n8n self-hosted for AI agent scraping + vector DB, under $20/mo infra. Scaled to 10K runs/day.

X buzz: n8n extensions like Vibe for chat-building flows.

Security, Privacy & Compliance: Trust Factor

Data's gold—don't skimp.

Zapier: SOC 2 Type II, GDPR, encrypted transit/storage. Enterprise governance, no self-host.
Make: SOC 2/GDPR, SSO/SAML, audit logs (60 days Enterprise). On-prem agents for sensitive.
n8n: Self-host = full control (your firewalls). Cloud: Encrypted, but fair-code license. No HIPAA native, but extensible.

All solid for most; n8n if privacy's paramount, Zapier/Make for certs.

Support & Community: When You Get Stuck

Help matters.

Zapier: Huge community forums, docs, Premier chat (Team+). Active updates.
Make: Tiered email (high Enterprise), templates sharing, analytics help.
n8n: Forum/GitHub vibrant, email Pro+, dedicated Enterprise. Devs love the openness.

n8n's crowd-sourced edge; others more hand-holding.

Learning Curve: Jump In Time

Zapier: 10 mins to first Zap—intuitive.
Make: Hour for visuals, day for routers.
n8n: Day for basics, week for customs. Templates speed it.

Start simple, scale skill.

Switching Gears: Migration Tips

From Zapier? Export JSON, rebuild in visual (Make) or nodes (n8n). Make to n8n: Scenarios map easy to workflows. Tools like n8n's importer help. Test parallel first.

Wrapping It: Your Move

Zapier if you want set-it-forget-it bliss. Make for that sweet no-code power without vendor lock. n8n if you're building the future—cost, control, AI all in one.

Long-term? n8n's momentum (X hype on agents) feels right. Grab free tiers, mock a flow, and run. Your stack will thank you.

Quick FAQs

Best for pure AI agents? n8n—LangChain nodes let you chain LLMs wild. Make's beta close second.
Self-hosting worth it? For n8n, yeah if privacy/scale matter; ~$10/mo AWS. Skip for Zapier/Make (cloud-only).
HIPAA compliant? Make/Zapier Enterprise yes; n8n via self-host tweaks.
Free forever viable? Zapier/Make for light (100-1K runs); n8n unlimited if you host.
Team of 1 vs. 10? Solo: Any. Growing: Make/n8n for unlimited users cheap.

N8N: The Workflow Automation Tool Worth Knowing About

Aadarsh Nagrath — Tue, 07 Oct 2025 13:28:10 +0000

I've been using n8n for a while now, and it's one of those tools that just makes sense once you get into it.

It's a workflow automation platform, kind of like Zapier or Make, but with some differences that actually matter.

What n8n Actually Does

The name is a bit weird - it's pronounced "n-eight-n" because if you count the letters between 'n' and 'n' in "automation," there are eight. Anyway, it lets you connect different apps and automate tasks between them. Pretty standard stuff on the surface.

The interesting part is that it's open-source (technically "fair-code"), which means you can run it on your own server if you want. No one's forcing you to, but the option's there.

Why People Like It

You can host it yourself. This matters more to some people than others. If you're dealing with customer data or anything sensitive, keeping everything on your own infrastructure is kind of nice. Plus you're not worrying about some SaaS company changing their pricing or getting acquired.

The pricing makes sense. Most automation tools charge you per "task" or "operation," and those costs add up fast. With n8n, if you self-host, it's free. Their cloud version is reasonably priced too, and they don't nickel-and-dime you with weird limits.

It's actually flexible. You get a visual interface for building workflows, which is great. But when you need to do something custom, you can write JavaScript right in there. Or connect to any API. Or build your own integrations. It doesn't box you in.

How It Works

The interface is node-based. You drag boxes (nodes) onto a canvas, connect them with lines, and each box does something - pull data from a spreadsheet, send a message, process information, whatever.

It's visual enough to understand what's happening but powerful enough to do complex stuff.

One thing I appreciate is that you can test each step as you build. You're not just hoping it works when you hit "save" - you can see exactly what data is flowing through at each point.

What People Build With It

Some common patterns I've seen:

Content teams use it to publish once and distribute everywhere. Write a blog post, and it automatically creates social media posts, adds it to the newsletter, updates various platforms. Saves a ton of time.

Support teams build smart ticket routing. New support request comes in, it checks if they're an important customer, looks for relevant help docs, sends an initial response, and notifies the right person. Response times improve significantly.

Developers connect all their tools together. When code gets merged, it deploys to staging. When errors happen, it creates detailed tickets. When builds fail, it notifies the team. It's the glue between different systems.

Marketing folks seem to love it. They build lead scoring systems, automated follow-ups based on user behavior, social listening tools, and reporting dashboards that pull from multiple ad platforms.

Getting It Running

Three ways to go about it:

Use their cloud version - easiest option, works immediately, they handle everything.

Run it with Docker - takes maybe 15 minutes if you're comfortable with that kind of thing. Good middle ground.

Full custom setup - deploy it however you want, integrate with whatever infrastructure you have.

Most people either start with the cloud version to try it out, or jump straight to Docker if they're technical.

The Integration Situation

They've got 400+ integrations built in. All the usual suspects - Google services, Slack, databases, cloud platforms, CRM tools, e-commerce stuff, AI services.

If what you need isn't there, you can still connect to it using their HTTP request node. Basically anything with an API works. And the community keeps building new integrations.

Some Handy Features

Sub-workflows let you build reusable pieces. Make something once, use it in multiple workflows. Keeps things organized.

Error handling means your workflows don't just break and fail silently. You can set up retries, fallbacks, notifications.

Scheduling is built in. Run things hourly, daily, weekly, whatever schedule you need.

Database connections let you query and update databases directly. Sometimes you just need to work with data where it lives.

Things to Know

If you want to self-host, you need some technical comfort. Not expert-level, but you should know your way around Docker and servers at least a bit.

The interface is really made for desktop. Using it on a phone would be awkward.

Some integrations are more polished than others. Popular services have everything you'd want. Niche ones might need you to use the HTTP node for certain features.

You can build really complex workflows, which is great until you need to debug them six months later. Good documentation habits help.

When It Makes Sense

N8n works well if you want control over your automation setup, if you're hitting pricing limits elsewhere, or if you need to keep data in-house. It's also good if you like having the option to customize things when needed.

If you just need super basic automation and don't care about cost or control, something simpler might be fine. But if you're reading a comprehensive guide about automation tools, you're probably past that point.

The Community

There's an active community around it. People share workflow templates, help each other out in forums and Discord, contribute code. The company (they're based in Berlin) does a good job maintaining it and being transparent about what they're working on.

Worth Trying?

If you're doing any serious automation work, it's at least worth a look. The cloud version is easy to try out, and you can always move to self-hosting later if you want.

It's one of those tools that grows with you. Start simple, add complexity when you need it. That's kind of the sweet spot for this type of software.

The workflow automation space has a lot of options, but n8n hits a good balance between being accessible and being powerful. Plus the whole open-source angle means you're not locked into someone's ecosystem if things change down the road.

Anyway, that's n8n. It does what it says it does, and does it pretty well.

Yo, Wanna Surf the Dark Web? Here’s How to Do It Without Getting Hacked!

Aadarsh Nagrath — Sat, 27 Sep 2025 07:57:37 +0000

Alright, let’s talk about the dark web—that mysterious corner of the internet that feels like a secret club. Before you jump in, I’m gonna show you three ways to access it safely, from the “you’re begging to get hacked” method to the “you’re basically a ghost” level of stealth. Think top-tier, can’t-touch-me security.

First off, the dark web isn’t all bad. It was created with good vibes—think journalists protecting their sources or folks dodging oppressive governments. You can even play chess on there without anyone snooping! Big names like the New York Times, BBC, and even the CIA have dark web sites.

Pretty cool, right? But that same anonymity makes it a hotspot for sketchy stuff too. Like, your email, passwords, and logins? They’re probably being sold there right now. Scary, I know.

That’s why I use a password manager with dark web monitoring to keep my info safe. It alerts me if my data’s out there and generates super-secure passwords that are basically unhackable.

I use it for all my accounts, and it’s a game-changer. Trust me, most hacks happen ‘cause someone used a weak password or clicked a shady link. Don’t be that person.

Now, let’s dive into the three ways to access the dark web without ending up in a hacker’s trap.

Before that feel free to follow me on X

Level 1: The “Please Don’t Do This” Way

I know some of you are gonna try this anyway, but this is the least secure way to hit the dark web. All you need is the Tor Browser. It’s like Chrome or Firefox, but for the dark web (aka the Onion Network—yep, onions everywhere). Here’s how it goes:

Head to the Tor Project website.
Download the Tor Browser for your operating system (iOS users, hang tight, we’ll get to you).
Install it, click “Connect,” and bam—you’re on the dark web.

Sounds simple, right? Too simple. The Tor Browser sends your data through three “onion routers,” each adding a layer of encryption to keep you anonymous. But this basic setup is like walking into a shady alley with no backup. You can access regular sites (the “clearnet”) with Tor, but dark web sites? They’re hidden, ending in “.onion.”

For example, the New York Times has a dark web version, but you’d need the exact address to find it. No Google here—those sites are dark for a reason. There’s a site called the Hidden Wiki that lists some dark web sites, and a search engine called Ahmia, but even those can’t index everything.

This method? It’s risky. You’re exposed with default settings, so let’s move on to something safer.

Level 2: A Bit Safer, But Still Not Great

This is a step up, but still not the best way to browse the dark web. You’ll use the same Tor Browser, but with some extra protection—like adding another layer to your onion. Here’s the deal:

Before connecting to the Tor network, fire up a VPN you trust. This hides your activity from your internet provider, who can totally see you’re using Tor otherwise.
Launch the Tor Browser and tweak some settings. Go to “Configure Connection,” then “Privacy and Security,” and set the security level to “Safest.” This disables JavaScript and other features hackers could exploit.

Why the VPN? Without it, your internet provider and the first onion router you connect to can see your IP address. That’s a weak spot where leaks happen—yep, that’s how the FBI and CIA track people. A VPN encrypts your connection before it hits the Tor network, adding a solid layer of protection.

With these tweaks, you’re safer than Level 1, but it’s still not bulletproof. Let’s level up to the real deal.

Level 3: The “You’re Basically Invisible” Way

This is the most secure way to access the dark web, and it’s the only method I’d recommend. You’re gonna create a portable computer on a USB drive using Tails Linux—a super-secure operating system that forgets everything you do. It’s like a goldfish with amnesia. Even Edward Snowden trusts this setup, and he’s still chilling in Russia, so you know it’s legit.

Here’s what you need:

A USB drive with at least 8GB of space (super cheap on Amazon).
A computer (almost any will do).

Here’s how to set it up:

Go to tails.net and download the Tails USB image.
Grab Balena Etcher (free software) to write the Tails image to your USB drive.
Plug in your USB, open Etcher, select the Tails image, choose your USB as the target, and hit “Flash.”
Once it’s done, shut down your computer, plug in the USB, and boot from it. You might need to hit a key like F12, F10, or Delete to access the boot menu (depends on your computer).
Boom—you’re running Tails Linux. Connect to Wi-Fi (or use an Ethernet cable if Wi-Fi’s finicky), and Tails automatically routes everything through the Tor network.

Tails is a full-blown operating system that leaves no trace. When you’re done, shut it down, pull the USB, and it’s like you were never there. Want extra security? Connect to a VPN before launching the Tor Browser in Tails. That’s VPN + Tor + Tails—a fortress of anonymity.

Pro tip: You can set up persistent storage on Tails to save files, but if you’re going for max security, skip it. The default setup is designed to keep you untraceable.

Level 3.75 – The Cloud Browser

There’s one more way that’s arguably as secure as Tails, maybe even more. It’s a cloud browser—a remote computer somewhere else in the world that runs the Tor Browser for you.

Just pick a location (Europe, US, wherever), click to launch, and you’re browsing the dark web from a machine that’s not yours. It’s super secure, and bonus: it works on your phone too.

Stay Safe Out There

Even with Level 3 or 3.75, the dark web is no joke. It’s a wild place with no oversight, so take extra precautions:

Don’t download anything unless you’re sure it’s safe. Disconnect from the internet before opening files—they could phone home and expose you.
Stick to known sites like the Hidden Wiki or legit dark web versions of mainstream sites.
Always assume someone’s watching, even if you’re super secure.

The dark web can be cool, but it’s also a minefield. Use Tails or a cloud browser, keep your passwords locked down with a solid password manager, and don’t become one of those scary dark web stories. Stay safe, and happy (anonymous) surfing!

How to Keep Using Cursor for FREE on MacOS/Linux 🖥️

Aadarsh Nagrath — Tue, 23 Sep 2025 15:09:38 +0000

Hey there, coder! 😎 So, you’ve been loving Cursor, that sweet AI-powered code editor, but your free trial just ended, and you’re not ready to commit to a paid plan. No stress!

Here’s a chill, step-by-step guide to keep using Cursor for free on your MacOS/Linux machine. We’ll use a simple script and a disposable email to reset your trial like a pro. Don’t worry, I’ll explain what the script does, why it’s safe, and how to make it all work.

Later I'll update this blog to make it usefull in windows as well
I haven't tried it on linux but i speculate it will work there as well go give it a try

Let’s dive in! 🚀

Follow me on X

What You’ll Need

A MacOS computer (Intel or Apple Silicon, doesn’t matter, we got you covered).
Basic familiarity with the Terminal (don’t worry, it’s super easy).
An internet connection to grab a temporary email and download the script’s goodies.
A vibe for keeping things simple and free. 😎

Step-by-Step Guide to Keep Cursor Free

Step 1: Sign Out of Cursor

Once your free trial is over, open the Cursor app and sign out of your account. This clears the slate for a fresh start. Close the app completely when you’re done.

Step 2: Clear Cursor’s Data

To reset the trial, we need to remove some local data Cursor stores on your Mac. Open your Terminal (you can find it in Applications > Utilities or search for it with Spotlight). Run these commands one by one:

sudo rm -rf ~/Library/Application\ Support/Cursor
sudo rm -rf ~/Library/Application\ Support/cursor-updater

What’s happening here?

These commands delete Cursor’s local data, like cached settings and trial info. The sudo part means you’re running it with admin privileges, so you’ll need to enter your Mac password. Don’t worry, this is totally safe—it’s just clearing out app-specific files, not touching anything critical to your system. It’s like wiping the app’s memory so it thinks it’s being installed fresh.

Step 3: Close Cursor

Make sure the Cursor app is fully closed. You can do this by right-clicking its icon in the Dock and selecting Quit, or use Command + Q when the app is active.

Step 4: Create the Magic Script

Now, we’re gonna create a script called install.sh that automates downloading and setting up a fresh Cursor install. Open Terminal and create the file by typing:

nano install.sh

Copy and paste the following script into the Terminal window:

You can get the script form here as well -

https://github.com/yeongpin/cursor-free-vip

The script

#!/bin/bash

# Color definitions
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
CYAN='\033[0;36m'
NC='\033[0m' # No Color

# Logo
print_logo() {
    echo -e "${CYAN}"
    cat << "EOF"
   ██████╗██╗   ██╗██████╗ ███████╗ ██████╗ ██████╗      ██████╗ ██████╗  ██████╗   
  ██╔════╝██║   ██║██╔══██╗██╔════╝██╔═══██╗██╔══██╗     ██╔══██╗██╔══██╗██╔═══██╗  
  ██║     ██║   ██║██████╔╝███████╗██║   ██║██████╔╝     ██████╔╝██████╔╝██║   ██║  
  ██║     ██║   ██║██╔══██╗╚════██║██║   ██║██╔══██╗     ██╔═══╝ ██╔══██╗██║   ██║  
  ╚██████╗╚██████╔╝██║  ██║███████║╚██████╔╝██║  ██║     ██║     ██║  ██║╚██████╔╝  
   ╚═════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝     ╚═╝     ╚═╝  ╚═╝ ╚═════╝  
EOF
    echo -e "${NC}"
}

# Get download folder path
get_downloads_dir() {
    if [[ "$(uname)" == "Darwin" ]]; then
        echo "$HOME/Downloads"
    else
        if [ -f "$HOME/.config/user-dirs.dirs" ]; then
            . "$HOME/.config/user-dirs.dirs"
            echo "${XDG_DOWNLOAD_DIR:-$HOME/Downloads}"
        else
            echo "$HOME/Downloads"
        fi
    fi
}

# Get latest version
get_latest_version() {
    echo -e "${CYAN}ℹ️ Checking latest version...${NC}"
    latest_release=$(curl -s https://api.github.com/repos/yeongpin/cursor-free-vip/releases/latest) || {
        echo -e "${RED}❌ Cannot get latest version information${NC}"
        exit 1
    }

    VERSION=$(echo "$latest_release" | grep -o '"tag_name": ".*"' | cut -d'"' -f4 | tr -d 'v')
    if [ -z "$VERSION" ]; then
        echo -e "${RED}❌ Failed to parse version from GitHub API response:\n${latest_release}"
        exit 1
    fi

    echo -e "${GREEN}✅ Found latest version: ${VERSION}${NC}"
}

# Detect system type and architecture
detect_os() {
    if [[ "$(uname)" == "Darwin" ]]; then
        # Detect macOS architecture
        ARCH=$(uname -m)
        if [[ "$ARCH" == "arm64" ]]; then
            OS="mac_arm64"
            echo -e "${CYAN}ℹ️ Detected macOS ARM64 architecture${NC}"
        else
            OS="mac_intel"
            echo -e "${CYAN}ℹ️ Detected macOS Intel architecture${NC}"
        fi
    elif [[ "$(uname)" == "Linux" ]]; then
        # Detect Linux architecture
        ARCH=$(uname -m)
        if [[ "$ARCH" == "aarch64" || "$ARCH" == "arm64" ]]; then
            OS="linux_arm64"
            echo -e "${CYAN}ℹ️ Detected Linux ARM64 architecture${NC}"
        else
            OS="linux_x64"
            echo -e "${CYAN}ℹ️ Detected Linux x64 architecture${NC}"
        fi
    else
        # Assume Windows
        OS="windows"
        echo -e "${CYAN}ℹ️ Detected Windows system${NC}"
    fi
}

# Install and download
install_cursor_free_vip() {
    local downloads_dir=$(get_downloads_dir)
    local binary_name="CursorFreeVIP_${VERSION}_${OS}"
    local binary_path="${downloads_dir}/${binary_name}"
    local download_url="https://github.com/yeongpin/cursor-free-vip/releases/download/v${VERSION}/${binary_name}"

    # Check if file already exists
    if [ -f "${binary_path}" ]; then
        echo -e "${GREEN}✅ Found existing installation file${NC}"
        echo -e "${CYAN}ℹ️ Location: ${binary_path}${NC}"

        # Check if running as root
        if [ "$EUID" -ne 0 ]; then
            echo -e "${YELLOW}⚠️ Requesting administrator privileges...${NC}"
            if command -v sudo >/dev/null 2>&1; then
                echo -e "${CYAN}ℹ️ Starting program with sudo...${NC}"
                sudo chmod +x "${binary_path}"
                sudo "${binary_path}"
            else
                echo -e "${YELLOW}⚠️ sudo not found, trying to run normally...${NC}"
                chmod +x "${binary_path}"
                "${binary_path}"
            fi
        else
            # Already running as root
            echo -e "${CYAN}ℹ️ Already running as root, starting program...${NC}"
            chmod +x "${binary_path}"
            "${binary_path}"
        fi
        return
    fi

    echo -e "${CYAN}ℹ️ No existing installation file found, starting download...${NC}"
    echo -e "${CYAN}ℹ️ Downloading to ${downloads_dir}...${NC}"
    echo -e "${CYAN}ℹ️ Download link: ${download_url}${NC}"

    # Check if file exists
    if curl --output /dev/null --silent --head --fail "$download_url"; then
        echo -e "${GREEN}✅ File exists, starting download...${NC}"
    else
        echo -e "${RED}❌ Download link does not exist: ${download_url}${NC}"
        echo -e "${YELLOW}⚠️ Trying without architecture...${NC}"

        # Try without architecture
        if [[ "$OS" == "mac_arm64" || "$OS" == "mac_intel" ]]; then
            OS="mac"
            binary_name="CursorFreeVIP_${VERSION}_${OS}"
            download_url="https://github.com/yeongpin/cursor-free-vip/releases/download/v${VERSION}/${binary_name}"
            echo -e "${CYAN}ℹ️ New download link: ${download_url}${NC}"

            if ! curl --output /dev/null --silent --head --fail "$download_url"; then
                echo -e "${RED}❌ New download link does not exist${NC}"
                exit 1
            fi
        elif [[ "$OS" == "linux_x64" || "$OS" == "linux_arm64" ]]; then
            OS="linux"
            binary_name="CursorFreeVIP_${VERSION}_${OS}"
            download_url="https://github.com/yeongpin/cursor-free-vip/releases/download/v${VERSION}/${binary_name}"
            echo -e "${CYAN}ℹ️ New download link: ${download_url}${NC}"

            if ! curl --output /dev/null --silent --head --fail "$download_url"; then
                echo -e "${RED}❌ New download link does not exist${NC}"
                exit 1
            fi
        else
            exit 1
        fi
    fi

    # Download file
    if ! curl -L -o "${binary_path}" "$download_url"; then
        echo -e "${RED}❌ Download failed${NC}"
        exit 1
    fi

    # Check downloaded file size
    local file_size=$(stat -f%z "${binary_path}" 2>/dev/null || stat -c%s "${binary_path}" 2>/dev/null)
    echo -e "${CYAN}ℹ️ Downloaded file size: ${file_size} bytes${NC}"

    # If file is too small, it might be an error message
    if [ "$file_size" -lt 1000 ]; then
        echo -e "${YELLOW}⚠️ Warning: Downloaded file is too small, possibly not a valid executable file${NC}"
        echo -e "${YELLOW}⚠️ File content:${NC}"
        cat "${binary_path}"
        echo ""
        echo -e "${RED}❌ Download failed, please check version and operating system${NC}"
        exit 1
    fi

    echo -e "${CYAN}ℹ️ Setting executable permissions...${NC}"
    if chmod +x "${binary_path}"; then
        echo -e "${GREEN}✅ Installation completed!${NC}"
        echo -e "${CYAN}ℹ️ Program downloaded to: ${binary_path}${NC}"
        echo -e "${CYAN}ℹ️ Starting program...${NC}"

        # Run program directly
        "${binary_path}"
    else
        echo -e "${RED}❌ Installation failed${NC}"
        exit 1
    fi
}

# Main program
main() {
    print_logo
    get_latest_version
    detect_os
    install_cursor_free_vip
}

# Run main program
main

Save the file by pressing Ctrl + O, hit Enter, then Ctrl + X to exit nano.

What does this script do?

This install.sh script is a helper that automates downloading the latest version of a free Cursor binary from a GitHub repo (yeongpin/cursor-free-vip). Here’s the breakdown:

Prints a cool logo: Just for vibes, it displays a fancy ASCII art logo.
Finds your Downloads folder: It figures out where your Downloads folder is on your Mac.
Checks the latest version: It hits up GitHub’s API to grab the latest release version of the Cursor binary.
Detects your Mac’s architecture: It checks if you’re on an Apple Silicon (arm64) or Intel Mac to download the right file.
Downloads the binary: It pulls the correct Cursor binary for your Mac from GitHub and saves it to your Downloads folder.
Makes it executable: It sets permissions so you can run the downloaded file.
Runs the app: It launches the freshly downloaded Cursor app.

Is it safe?

Totally! The script only deletes Cursor’s local data (nothing system-critical), downloads a file from a public GitHub repo, and runs it. It doesn’t mess with your system files, install malware, or do anything shady. Just make sure you’re cool with downloading from the yeongpin/cursor-free-vip repo—check it out on GitHub if you wanna be extra sure. The script also checks the file size to ensure it’s a legit executable, not some random error message.

Step 5: Run the Script

Now, let’s execute the script. In Terminal, run:

sudo bash install.sh

You’ll need to enter your Mac password again because sudo is used to set executable permissions. The script will download the Cursor binary, make it runnable, and launch it.

Step 6: Interact with the Script

When the script runs, it might prompt you with a menu (depending on the binary). When prompted, press 8 and hit Enter. This likely selects an option to set up a free version of Cursor. After that, when prompted again, press 0 to exit the script.

Step 7: Reopen Cursor

Open the Cursor app again (it should be in your Downloads folder or wherever the script saved it). It’ll act like a fresh install, ready for a new trial.

Step 8: Get a Disposable Email

Head over to temp-mail.org in your browser. This site gives you a temporary, disposable email address that self-destructs after a while—perfect for signing up without spamming your real inbox.

Click to generate a new email address (it’s instant, no registration needed).
Copy the temporary email address.

Step 9: Create a New Cursor Account

In the Cursor app, sign up for a new account using the temporary email from temp-mail.org. Follow the prompts to create the account, and log in with it. Boom, you’re back in with a fresh trial! 🎉

Step 10: Enjoy Cursor for Free

You’re all set! You can now use Cursor as if it’s a brand-new install. Note that some features, like Cloud Sonnet 4, might be locked to pro users, but you’ll still get the core Cursor experience—AI code suggestions, sleek editor, all the good stuff.

Why Use a Disposable Email?

Using a temp email (like from temp-mail.org) is clutch because:

No spam: Your real email stays clean from promotional emails or potential data breaches.
Privacy: You don’t have to share your personal email with every service.
Easy resets: If you need to reset your trial again, just grab another temp email and repeat the process.

Temp-mail.org is super user-friendly, anonymous, and doesn’t require any signup. It’s like a burner phone for your email—use it, ditch it, no strings attached.

Tips for Smooth Sailing

Repeat as needed: If your trial runs out again, just repeat the steps: clear the data, run the script, get a new temp email, and sign up again.
Check the GitHub repo: The script pulls from yeongpin/cursor-free-vip. If you’re curious, peek at the repo to see what’s up.
Stay safe: Only run scripts from sources you trust. This one’s straightforward, but always double-check GitHub repos for sketchy stuff.
Backup your work: Cursor might reset settings or local data, so save your projects elsewhere before running the script.

Is This Ethical?

This method is about resetting your trial by clearing local data and using a new account with a disposable email. It’s a gray area—technically, you’re not breaking anything, but you’re extending the trial beyond what the devs might expect. If you love Cursor and can afford it, consider supporting the devs with a paid plan. For now, this is a chill way to keep coding without spending a dime. 😎

What You Get

With this setup, you’ll get Cursor’s core features—AI-powered code completion, error detection, and a slick VS Code-like interface. Some premium features (like advanced AI models) might be locked, but you’ll still have plenty to work with for coding, debugging, and building dope projects.

That’s it, fam! You’re now ready to keep rocking Cursor for free on your Mac. Hit up the Terminal, run that script, grab a temp email, and code like a boss. If you run into any snags, lemme know, and I’ll help you sort it out. Happy coding! 🚀✨

How to host a website locally and let other users use it — super-elaborate

Aadarsh Nagrath — Wed, 17 Sep 2025 19:44:25 +0000

Below is a complete, practical guide that covers everything from “quick demo in 2 minutes” to “production-grade, secure, and resilient”.
Read the whole thing for interview depth; use the quick sections when you need speed.

TL;DR (one-minute pitch)

Run your app and bind to 0.0.0.0 so it’s reachable on the LAN.
Open the port in your firewall and (optionally) reserve a static local IP.
For LAN-only access: share http://<local_ip>:<port>.
For Internet access: either set up router port-forwarding + dynamic DNS / domain + TLS, or use a tunnel service (ngrok / cloudflared) for ease.
For production: put Nginx/Traefik as a reverse proxy, get Let’s Encrypt TLS, run the app as a systemd/docker service, enable firewall, monitoring, and fail2ban.

Full walkthrough

Assumptions

OS: Linux example (Ubuntu). I’ll also give Windows/macOS commands where relevant.
App runs on port 3000 (adjust if different).
You control your home/office router (or you’ll use a tunnel).

1) Local server — make the app reachable on your machine and LAN

Static files (quick)

# Python 3 (static files)
python3 -m http.server 3000  # serves current dir

Dynamic apps — examples

Node / Express

// server.js
const express = require('express');
const app = express();
app.use(express.static('public'));
const PORT = process.env.PORT || 3000;
app.listen(PORT, '0.0.0.0', () => console.log(`Listening on ${PORT}`));

Start: node server.js

Flask

# dev server (not for production)
FLASK_APP=app.py flask run --host=0.0.0.0 --port=3000

Django

python manage.py runserver 0.0.0.0:8000

Why 0.0.0.0?
localhost/127.0.0.1 only listens on loopback. 0.0.0.0 listens on all interfaces so other LAN machines can reach you.

2) Find your local IP and test from another machine on the same network

Find IP

Linux/macOS: ip addr show or ifconfig e.g. you'll see 192.168.1.5
Windows: ipconfig

Test from another device on same Wi-Fi
Open http://192.168.1.5:3000 in their browser.

If it works — great. If not, continue debugging below.

3) Open the firewall and check listening sockets

Linux (ufw)

sudo ufw allow 3000/tcp
sudo ufw status

Iptables (example)

sudo iptables -I INPUT -p tcp --dport 3000 -j ACCEPT

Windows (PowerShell)

New-NetFirewallRule -DisplayName "Allow3000" -Direction Inbound -LocalPort 3000 -Protocol TCP -Action Allow

Verify server is listening

ss -tulpn | grep :3000
# or
sudo lsof -i :3000

4) Make your local IP stable (DHCP reservation or static IP)

Easiest/robust: configure DHCP reservation on your router (bind your machine MAC → fixed IP).
Alternative: set a static IP on the OS; avoid conflicts by choosing outside DHCP range.

Why? Router port forwards and DNS point to a local IP; if it changes, forwarding breaks.

5) Letting people on the Internet access your site

You have four common options. Pick by tradeoffs:

Option A — Router port forwarding + domain or dynamic DNS (more “own infra”)

Steps

In router admin, forward TCP port 80 (HTTP) and 443 (HTTPS) to 192.168.1.5 (your machine).
Check your public IP: curl ifconfig.me (from your machine).
If you have a domain: create an A record pointing to the public IP. If IP is dynamic, use a DDNS provider (DuckDNS, No-IP, DynDNS).
Obtain TLS with Let’s Encrypt (certbot) — see section on SSL.
Test from outside (mobile data or ask a friend).

Pros: full control, no third-party tunnel.
Cons: ISP might block 80/443, NAT complexity, exposes home network — needs hardening.

Tip: If your ISP blocks 80, forward a high port (e.g., 8080) externally and use a VPS reverse proxy or a tunnel for TLS.

Option B — Tunnel services (fast, safe for demos)

ngrok: ngrok http 3000 → gives https://<random>.ngrok.io
- Sign up for token, can get stable subdomain on paid plan.
localtunnel: npx localtunnel --port 3000 --subdomain mydemo
Cloudflare Tunnel (cloudflared): persistent, integrates with your Cloudflare DNS and supports production use.

Pros: no router changes, works behind CGNAT, quick, TLS handled.
Cons: external dependency; free plans may be limited.

Example ngrok

# install ngrok, then
ngrok authtoken <your-token>
ngrok http 3000
# copy the https URL and share

Option C — VPS reverse proxy + SSH reverse tunnel (robust, low-cost)

Spin a cheap VPS (DigitalOcean, AWS Lightsail) with a public IP and domain.
On the VPS run Nginx and point your domain to the VPS IP.
From your local machine open an SSH reverse tunnel:

ssh -R 8080:localhost:3000 user@vps.example.com

Then in the VPS Nginx config, reverse proxy / to http://127.0.0.1:8080. Use autossh to keep tunnel persistent.

Pros: You control the domain & TLS on VPS; only outbound SSH from home (safer).
Cons: More setup, cost of VPS.

Option D — Host on a cloud provider (recommended for production)

If the goal is stable public hosting, deploy to a cloud server or platform (VPS, Heroku, Vercel, Netlify, AWS/GCP/Azure) rather than exposing a home machine.

6) Domain names and TLS (Let’s Encrypt example)

Install certbot and nginx plugin (Ubuntu)

sudo apt update
sudo apt install nginx certbot python3-certbot-nginx

Nginx server block example

server {
    listen 80;
    server_name example.com www.example.com;
    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Obtain certificate

sudo certbot --nginx -d example.com -d www.example.com
# Certbot will configure nginx and set up automatic renew

If behind NAT / using tunnel: terminate TLS at the tunnel (ngrok/cloudflare) or at VPS reverse proxy.

Cloudflare note: If using Cloudflare proxy, set SSL/TLS to “Full (strict)” with origin certs for security.

7) Run your app as a service (reliability)

systemd service for Node app

# /etc/systemd/system/myapp.service
[Unit]
Description=My Node App
After=network.target

[Service]
User=ubuntu
WorkingDirectory=/home/ubuntu/myapp
ExecStart=/usr/bin/node server.js
Restart=always
Environment=NODE_ENV=production PORT=3000

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now myapp
sudo systemctl status myapp

Alternative: Use pm2 for Node, or Docker + docker-compose. For containers, use restart policies.

8) Reverse proxy and load balancing (Nginx / Traefik)

Use Nginx to:

Terminate TLS.
Serve static assets directly.
Proxy traffic to the app on localhost.
Add rate limiting, IP blocklists, gzip, caching.

Simple Nginx proxy (see above).

If you have multiple app instances, create an upstream block and load balance.

9) Docker approach (portable reproducible stack)

Dockerfile (example Node)

FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --production
COPY . .
EXPOSE 3000
CMD ["node", "server.js"]

docker-compose.yml (Nginx + app)

version: '3.8'
services:
  app:
    build: .
    restart: always
  nginx:
    image: nginx:stable
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
    depends_on:
      - app

Use docker-compose up -d.

10) Security checklist (MUST for Internet exposure)

Use HTTPS only (redirect HTTP → HTTPS).
Don’t expose databases or admin ports to the public network.
Use firewall: only open necessary ports (80/443, ssh limited to known IPs).
Use fail2ban to ban repeated failed login attempts.
Keep OS and dependencies updated.
Use a reverse proxy to enforce security headers:
- Strict-Transport-Security, X-Frame-Options, X-Content-Type-Options, CSP.
Use environment variables or secret manager for credentials — don’t commit .env to VCS.
For staging/demos: put a simple auth gate (basic auth) or IP allowlist.
For team-only access: prefer VPN (WireGuard) or private tunnel.

11) Monitoring and logging

Logs: Nginx access/error logs, app logs (journalctl -u myapp or Docker logs).
Health checks: expose /healthz which returns 200 for monitoring.
Uptime monitoring: UptimeRobot / Pingdom or custom Prometheus + Alertmanager.
Performance: enable gzip, cache static assets, use CDN for large public assets.

12) Troubleshooting checklist (if not reachable)

Is the app running? systemctl status / ps aux / docker ps.
Is it listening on correct interface? ss -tulpn | grep 3000.
Firewall open? sudo ufw status or Windows firewall rules.
From local machine: curl http://127.0.0.1:3000.
From another LAN machine: curl http://192.168.1.5:3000.
If public: is port forwarding set up? Check router settings.
Is ISP blocking ports? Test with a remote port scan or try an alternate external port.
DNS propagation issues: dig +short example.com and compare to expected IP.
Use traceroute and tcpdump to debug network hops and packets.

13) Example: full step-by-step (Ubuntu) — Node app, public access via ngrok (quick, safe demo)

Clone and start app:

git clone https://.../myapp.git
cd myapp
npm install
node server.js --port 3000 --host 0.0.0.0

Allow firewall:

sudo ufw allow 3000/tcp

Install ngrok, authenticate, and start tunnel:

ngrok authtoken <token>
ngrok http 3000
# ngrok prints a public https://... url — share that

(Optional) Make Node run as a service with systemd (see earlier).

Why this flow? Fast, secure for demos, no router fiddling or DNS changes required.

14) Example: production checklist (if you mean “others” = many users)

Use a cloud VM or managed platform (AWS/GCP/DigitalOcean). Don’t host high-traffic production on a home connection.
Use a reverse proxy with TLS (Nginx/Traefik), autoscaling/load balancing, CDN for static content.
Use managed databases and secrets. Regular backups, monitoring, and incident response plan.

15) Interview talking points — how to present this answer (concise)

If asked in an interview, structure your answer:

Goal & constraints (LAN vs Internet, security, budget).
Quick demo solution: serve on 0.0.0.0 and use ngrok for external demos.
Production solution: deploy to a cloud VM / container platform, front with Nginx/Traefik, secure with Let’s Encrypt, use monitoring & CI/CD.
Risks & mitigations: ISP blocks, security exposure, scaling, backups.
Why decisions: choose tunnels for quick demos, VPS/prod for stability and long-term control.

16) Common gotchas to mention (shows depth)

CGNAT: Some home ISPs use carrier NAT — port forwarding won’t work. Use a tunnel or VPS reverse proxy.
ISP blocks: Residential ISPs sometimes block 80/443. Use alternative ports or VPS.
Router UPnP security risk: Avoid enabling UPnP for automatic port opening on untrusted networks.
Certificates & local hostnames: Let’s Encrypt requires a publicly resolvable domain. For internal names use self-signed certs or a private CA.
Privacy: Be mindful of exposing development credentials or debug endpoints.

17) Example artifacts to prepare for interview (optional)

Small diagram: local machine → router (NAT) → Internet. Or local machine → ngrok → public URL.
A sample systemd service file, nginx block, and certbot command snippet ready to paste.
A short script to create a DDNS update (or ddclient configuration).

Final summary (what I would do)

For a quick demo: bind to 0.0.0.0, open firewall, run ngrok http <port>. Share ngrok URL.
For short-term public hosting: use a cheap VPS + reverse proxy + Let’s Encrypt + systemd/docker + monitoring.
For production: use cloud hosting/managed services, TLS, CDN, autoscaling, proper secrets and monitoring.
Always secure: firewall, do not expose internal services, keep software current.

Setting Up HTTPS on Kubernetes with cert-manager and Let's Encrypt

Aadarsh Nagrath — Wed, 17 Sep 2025 08:28:01 +0000

In today's digital landscape, securing your web applications with HTTPS is non-negotiable. It ensures data privacy, builds user trust, and aligns with modern security standards. If you're running a Kubernetes cluster and want to enable HTTPS for your ingress using Let's Encrypt SSL certificates, cert-manager is your go-to tool for automating certificate management.

This blog post provides a comprehensive, step-by-step guide to configuring HTTPS on a Kubernetes cluster with cert-manager and Let's Encrypt. We'll walk through the entire process, from verifying your setup to troubleshooting common issues, ensuring your domain (e.g., platform-dev.example.ai) is secured with a valid SSL certificate.

Overview: What We're Building

This guide outlines how to set up HTTPS for a Kubernetes ingress using cert-manager, a powerful Kubernetes add-on that automates the issuance and renewal of SSL certificates from Let's Encrypt. By the end of this tutorial, you'll have:

A Kubernetes cluster with an nginx ingress controller.
A domain (e.g., platform-dev.example.ai) mapped to your cluster's external IP (e.g., 34.75.54.5).
An ingress configured for HTTPS with automatic SSL certificate management.

Initial Setup Status

Before diving in, let's clarify the starting point:

Cluster: A Kubernetes cluster with an nginx ingress controller already deployed.
Domain: A domain (e.g., platform-dev.example.ai) mapped to the external IP 34.75.54.5.
Ingress: An existing ingress with a TLS section, but the certificate isn't functioning correctly.
Goal: Enable HTTPS with automated SSL certificate issuance and renewal using cert-manager and Let's Encrypt.

Prerequisites

To follow this guide, ensure you have:

A Kubernetes cluster with an nginx ingress controller installed.
A registered domain name pointing to your cluster's external IP.
kubectl configured with access to your cluster.

Let’s get started with the setup process.

Step 1: Verify cert-manager Installation

The first step is to confirm whether cert-manager is installed in your Kubernetes cluster. cert-manager is a Kubernetes-native tool that simplifies certificate management by automating the process of obtaining, renewing, and managing SSL certificates.

Run the following command to check for existing cert-manager pods:

kubectl get pods -n cert-manager

If cert-manager is installed, you should see three running pods:

cert-manager-xxxxx
cert-manager-cainjector-xxxxx
cert-manager-webhook-xxxxx

If these pods are not present, you need to install cert-manager. Use the following command to deploy the latest stable version (v1.13.0 at the time of writing):

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml

After applying the manifest, re-run the kubectl get pods -n cert-manager command to verify that the three cert-manager pods are running. This ensures cert-manager is ready to manage certificates for your cluster.

Step 2: Create a ClusterIssuer for Let's Encrypt

A ClusterIssuer is a Kubernetes resource that defines how cert-manager should obtain certificates. In this case, we'll configure a ClusterIssuer to communicate with Let's Encrypt's production ACME server for issuing valid SSL certificates.

First, check if a ClusterIssuer named letsencrypt-prod already exists:

kubectl get clusterissuer letsencrypt-prod

If it doesn't exist or isn't configured correctly, create one with the correct ACME server URL. Important: Many tutorials mistakenly use an incorrect ACME server URL. The correct URL is:

✅ Correct: https://acme-v02.api.letsencrypt.org/directory
❌ Incorrect: https://acme-v2.api.letsencrypt.org/directory

To create the ClusterIssuer, apply the following configuration, replacing your-email@company.com with your actual email address (used for Let's Encrypt notifications):

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: your-email@company.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx
EOF

After applying, verify that the ClusterIssuer is ready:

kubectl get clusterissuer letsencrypt-prod

The output should show READY: True. If it’s not ready, we'll cover troubleshooting in Step 5.

Step 3: Configure the Ingress for HTTPS

Your Kubernetes ingress resource needs specific annotations and configurations to work with cert-manager and enable HTTPS. Below is the complete ingress configuration for your domain (e.g., platform-dev.example.ai), which routes traffic to two services: platform-ui and platform-api.

Apply the following ingress configuration:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/enable-rewrite-log: "true"
    nginx.ingress.kubernetes.io/use-regex: "true"
  name: example-ingress
  namespace: platform-namespace
spec:
  ingressClassName: nginx
  rules:
  - host: platform-dev.example.ai
    http:
      paths:
      - backend:
          service:
            name: platform-ui
            port:
              number: 80
        path: /
        pathType: Prefix
      - backend:
          service:
            name: platform-api
            port:
              number: 80
        path: /api
        pathType: Prefix
  tls:
  - hosts:
    - platform-dev.example.ai
    secretName: example-tls

Key Points in the Configuration:

Annotation: cert-manager.io/cluster-issuer: letsencrypt-prod tells cert-manager to use the letsencrypt-prod ClusterIssuer to issue a certificate.
TLS Section: Specifies the domain (platform-dev.example.ai) and the secret (example-tls) where cert-manager will store the SSL certificate.
Paths: Routes / to the platform-ui service and /api to the platform-api service, both on port 80.
Namespace: Ensure the ingress is created in the correct namespace (platform-namespace).

This configuration triggers cert-manager to request a certificate from Let's Encrypt and store it in the example-tls secret.

Step 4: Verify Certificate Creation

Once the ingress is configured, cert-manager will automatically request a certificate from Let's Encrypt. To monitor the certificate creation process, use the following commands:

# Check certificate status
kubectl get certificate -n platform-namespace

# Get detailed certificate information
kubectl describe certificate example-tls -n platform-namespace

# Check certificate requests (for troubleshooting)
kubectl get certificaterequest -n platform-namespace

For a successful certificate, the output of kubectl get certificate should look like:

NAME         READY   SECRET       AGE
example-tls  True    example-tls  5m

The READY: True status indicates that the certificate was issued successfully and is stored in the example-tls secret.

Step 5: Troubleshooting Common Issues

If something goes wrong, here are common issues and their solutions:

1. ClusterIssuer Not Ready

If the ClusterIssuer status is not READY: True, run:

kubectl describe clusterissuer letsencrypt-prod

Check the events or status for errors. The most common issue is an incorrect ACME server URL (e.g., using acme-v2 instead of acme-v02). Ensure the URL is https://acme-v02.api.letsencrypt.org/directory.

2. Certificate Stuck in Pending

If the certificate status shows as pending, investigate with:

kubectl describe certificate example-tls -n platform-namespace
kubectl logs -n cert-manager deployment/cert-manager

Look for errors related to the ACME challenge process, such as HTTP-01 challenge failures.

3. DNS Issues

Ensure your domain resolves correctly to your cluster's external IP. Test DNS resolution from within the cluster:

kubectl run test-dns --rm -it --restart=Never --image=busybox -- nslookup platform-dev.example.ai

Verify connectivity to the Let's Encrypt ACME server:

kubectl run test-connectivity --rm -it --restart=Never --image=curlimages/curl -- curl -I https://acme-v02.api.letsencrypt.org/directory

Step 6: Verify HTTPS is Working

Once the certificate is issued, test your HTTPS endpoint to confirm it’s working:

# Test HTTPS response
curl -I https://platform-dev.example.ai

# Check certificate details
openssl s_client -connect platform-dev.example.ai:443 -servername platform-dev.example.ai

Key Success Indicators

Your setup is successful if:

✅ Certificate status is READY: True.
✅ ClusterIssuer status is READY: True.
✅ Ingress shows both ports 80 and 443.
✅ The example-tls secret exists with certificate data.
✅ The site is accessible via HTTPS.

Certificate Lifecycle

Validity: Let's Encrypt certificates are valid for 90 days.
Auto-Renewal: cert-manager automatically renews certificates 30 days before expiration.
Monitoring: Check the certificate’s renewal time in its status with kubectl describe certificate example-tls -n platform-namespace.

Security Best Practices

To ensure a secure and reliable setup:

Email Configuration: Use a monitored email address for Let's Encrypt notifications to stay informed about certificate issues or renewals.
Secret Management: cert-manager automatically manages TLS secrets, so avoid manual modifications.
HTTP to HTTPS Redirect: Add the following annotation to your ingress to enforce HTTPS:

nginx.ingress.kubernetes.io/ssl-redirect: "true"

Final Configuration Summary

For a successful HTTPS setup, ensure:

cert-manager is installed and running.
The ClusterIssuer uses the correct ACME server URL.
The ingress includes the cert-manager annotation and TLS section.
DNS is properly configured to point to your cluster’s external IP.
Domain validation completes successfully.

Troubleshooting Commands Reference

For quick diagnostics, use these commands:

# Check cert-manager components
kubectl get pods -n cert-manager
kubectl logs -n cert-manager deployment/cert-manager

# Check certificates
kubectl get certificate -A
kubectl describe certificate example-tls -n platform-namespace

# Check ClusterIssuer
kubectl get clusterissuer
kubectl describe clusterissuer letsencrypt-prod

# Check ingress
kubectl get ingress -A
kubectl describe ingress example-ingress -n platform-namespace

# Check secrets
kubectl get secret example-tls -n platform-namespace -o yaml

Conclusion

By following this guide, you’ve successfully configured HTTPS for your Kubernetes ingress using cert-manager and Let's Encrypt.

Your domain (e.g., platform-dev.example.ai) is now secured with an automatically managed SSL certificate, ensuring secure communication for your users. With cert-manager handling renewals, you can focus on building your application while maintaining robust security.

If you encounter any issues, refer to the troubleshooting commands or reach out to the Kubernetes or cert-manager community for support. Happy securing!

How Big Titans Swipe Through Billions of usernames when you press Submit

Aadarsh Nagrath — Sat, 13 Sep 2025 12:03:20 +0000

Hey there, fellow tech wanderer!

Picture this: You're hyped about a new app, fingers flying across the keyboard as you type in your dream username—"PixelPirate42." Hit submit... and bam! "Username already taken." It's that tiny gut-punch moment that happens to all of us.

But here's the wild part: Behind that snappy rejection is a high-stakes engineering ballet involving data structures smarter than a chess grandmaster, caches that remember everything, and databases that span the globe. We're talking systems that handle billions of users without flinching—think Google, Amazon, Meta, and their ilk.

In this blog, I'll unpack the wizardry they use to make username checks lightning-fast. We'll geek out over Redis hashmaps, Tries for that autocomplete magic, B+ trees for sorted sleuthing, and Bloom filters for probabilistic punches. Plus, I'll throw in real-world examples, a handy comparison table, and even sketch out how it all orchestrates like a symphony.

First Stop: The Speed Demon – Redis Hashmaps

Okay, let's start simple. When you're not dealing with planet-scale users, a quick database peek works fine. But slap billions into the mix? That query turns into a traffic jam on the info superhighway—latency spikes, servers sweat, and your app feels like it's on dial-up.

Enter Redis hashmaps, the unsung heroes of caching layers. Redis isn't just a database; it's an in-memory powerhouse that stores key-value pairs like a digital Rolodex on steroids. For usernames, think of it this way: The key is a bucket (say, "usernames"), and inside that bucket? A hashmap where each field is a username, and the value is something lightweight—like a user ID or just a "taken" flag.

Example Time: Say "PixelPirate42" gets queried. The system pings Redis: "Hey, is this field in the hashmap?" Boom—cache hit! Response in microseconds, no database drama. If it's a new one, it's a miss, and only then do you hit the slower backend.

But memory's not infinite, right? You can't hoard every username forever in one Redis instance. That's why these bad boys shine for hot data—recently checked or popular usernames.

In the wild, Instagram or Twitter (er, X) might use this to slash 90% of database touches.

Level Up: Tries – The Prefix Party Planners

What if "PixelPirate42" is taken, but you want suggestions? Like, "PixelPirate43" or "PixelNinja42"? A plain hashmap says "yes/no" and ghosts you. Time for Tries (or prefix trees)—tree-like structures that dissect strings character by character, turning lookups into a choose-your-own-adventure game.

Here's the magic: Instead of flat storage, a Trie builds branches for shared prefixes. Root node for the first letter, kids for the second, and so on. Lookup time? O(M), where M is your string's length (say, 15 chars). Doesn't care if you've got a billion entries—it's all about the path, not the forest.

Real-World Example: Imagine usernames "bite_monkey," "biteme_io," and "biting_wit." The Trie shares a "b-i-t-e-" trunk, then splits: "-m-o-n-k-e-y" vs. "-m-e--i-o." Query "bite"? It traverses to that node in seconds and suggests completions. Autocomplete gold! Meta uses Trie-like structures for Facebook's handle suggestions, saving users from rage-quitting signups.

Trade-off? Memory muncher if prefixes don't overlap much (e.g., all unique snowflakes). Fix: Compressed Tries (like Radix Tries) squash single-child paths, or limit to "hot" usernames.

Pro tip: Netflix swears by them for title searches too.

The Sorted Sleuth: B+ Trees for Ordered Ops

Tries are prefix pros, but what about full sorted searches or "find the next available after PixelPirate"? Hashmaps flop here—they're unordered chaos. Cue B+ trees (and their B-tree cousins), the backbone of database indexes everywhere.

These balanced beasts keep keys sorted in leaf nodes, with internal nodes as signposts. High fan-out (hundreds of kids per node) means shallow trees—even a billion usernames might need just 3-4 hops (O(log N) time, baby!).

Example in Action: In a relational DB like MySQL (or NoSQL like MongoDB), usernames are indexed in a B+ tree. Query "PixelPirate42"? Traverse sorted leaves—maybe 30 comparisons max for billions of entries. Want the next free one? Range scan from there: "PixelPirate43... available!"

Google Cloud Spanner distributes these across machines, blending sorted magic with horizontal scale for millions of QPS. FoundationDB does the same.

Downside? Updates in distributed setups get fiddly, but hey, trade-offs build character.

Probabilistic Powerhouse: Bloom Filters – The Memory-Saving Gatekeepers

We've got speed, prefixes, and sorting. Now, for sheer efficiency? Bloom filters—probabilistic rockstars that scream "definitely not there!" without storing squat.

It's a bit array + k hash functions. Add a username? Hash it k ways, flip those bits to 1. Check? Hash again—if any bit's 0, it's absent (no false negatives!). All 1s? Maybe there—double-check elsewhere.

Crunch the Numbers Example: For 1B usernames at 1% false positives, ~1.2 GB memory. Vs. storing full strings? A fraction! Cassandra slaps these on SSTables to skip disk reads. Amazon might run a global one across shards—most "nope" queries die here, slashing DB load by 80-90%.

False positives? Rare enough to shrug off. If it says "maybe," hit the cache. Genius for "is this taken?" without the bloat.

Showdown: Data Structures Face-Off

To glue it all together, here's a quick comparison table. (Because who doesn't love a good showdown?)

Data Structure	Lookup Time	Memory Usage	Best For	Drawbacks	Real-World User
Redis Hashmap	O(1)	High (full keys)	Exact matches, hot data	Memory limits, no prefixes/ranges	Instagram caches for recent checks
Trie (Prefix Tree)	O(M)	Medium-High (shared paths)	Prefix searches, autocomplete	Bloats on unique strings	Facebook handle suggestions
B+ Tree	O(log N)	Medium (sorted nodes)	Sorted lookups, ranges	Update complexity in distrib. setups	Google Spanner for ordered scans
Bloom Filter	O(k) (hashes)	Low (bits only)	Probabilistic "not present"	False positives, no retrieval	Cassandra to filter disk I/O

N = total entries, M = string length, k = hashes.

How It All Harmonizes in the Wild

Solo acts are cool, but big tech? They layer like a gourmet lasagna. Let's trace "PixelPirate42" through the gauntlet:

Load Balancer Warm-Up: Global (Route 53 DNS) routes you to the nearest data center (EU for you Europeans). Local (Nginx/ELB) fans traffic to backend servers. No bottlenecks—traffic flows like a well-oiled highway.
Bloom Filter Bouncer: App server checks its in-memory Bloom (synced from DB periodically). "Definitely not?" Respond "available!" instantly. Saves the day 70% of the time.
Cache Dash: Miss? Ping Redis/Memcached. Hit? Microsecond win. (Pro: LRU eviction keeps it fresh.)
DB Deep Dive: Still no? Hit the distributed beast—Cassandra (Instagram's pick) or DynamoDB (Amazon's). Consistent hashing shards data across nodes; B+ trees or SSTables nail the exact check.
Response Relay: Truth bubbles back through balancers. Total time? Under 100ms, even at planetary scale.

Meta mixes Tries for suggestions post-check; Google Spanner adds ACID guarantees. It's not one tool—it's a ecosystem flexing computer science muscle.

The Beauty of Invisible Speed

Next time that "taken" message flashes, tip your hat to the invisible orchestra—Bloom's quick jabs, Redis's recall, Tries' foresight, B+ trees' precision.

It's engineering poetry: Fast, scalable, and oh-so-satisfying.

Loved this? Drop a comment—what system design puzzle should I tackle next? AI ethics? Sharding secrets?

(P.S. All examples inspired by real tech stacks; no actual heists were committed in writing this.)

The Deep Dive: Why and How Projects Use Multiple Programming Languages

Aadarsh Nagrath — Thu, 07 Aug 2025 18:49:38 +0000

Programming languages are like specialized tools in a craftsman’s workshop—each designed for specific tasks. Some excel at rapid development and high-level abstractions (Python, JavaScript), while others provide fine-grained control over hardware and memory (C, Rust, Assembly). But why do some projects combine multiple languages? How do they work together under the hood?

This blog will explore:

The compilation pipeline – How source code becomes executable binaries.
Linking (static vs. dynamic) – The glue that binds different languages.
Real-world examples – How projects like Linux, FFmpeg, and OpenSSL mix languages.
ABI (Application Binary Interface) – Why calling conventions matter.
Practical use cases – When and why you should mix languages.

1. The Compilation Pipeline: More Than Just "Code → Binary"

When you compile a program, the process is far more complex than just converting source code into an executable. Modern compilers follow a multi-stage pipeline, and understanding this is key to mixing languages.

The Four Key Phases (Using GCC as an Example)

1. Preprocessing

Removes comments.
Expands macros (#define).
Handles #include directives (pasting header files into the source).
Output: Preprocessed C code (still human-readable).

2. Compilation (to Assembly)

Translates high-level code (C, C++, Rust) into assembly language.
Myth Busted: Compilers don’t always go straight to machine code—many use intermediate representations (IR) like assembly.
Output: Assembly file (.s).

3. Assembly (to Machine Code)

The assembler converts assembly into machine code (binary).
Output: Object file (.o or .obj).

4. Linking (Combining Object Files)

Merges multiple object files (your code + libraries).
Resolves function calls between them.
Output: Final executable (.exe, .out, etc.).

Key Insight: Since compilation is modular, we can compile different parts of a project in different languages and link them later.

2. Linking: The Glue That Binds Languages

Linking is where the magic happens. There are two main types:

A. Static Linking

Library code is copied directly into the executable.
Pros: Self-contained, no runtime dependencies.
Cons: Larger binary, harder to update libraries.

B. Dynamic Linking

Libraries (.so on Linux, .dll on Windows) are loaded at runtime.
Pros: Saves disk space, allows library updates without recompiling.
Cons: Requires the library to be present on the system.

Example: Calling C from Rust

Write a function in C:

   // lib.c
   int add(int a, int b) { return a + b; }

Compile it as a static library:

   gcc -c lib.c -o lib.o  
   ar rcs libadd.a lib.o

Call it from Rust:

   // main.rs
   extern "C" { fn add(a: i32, b: i32) -> i32; }  
   fn main() { unsafe { println!("{}", add(2, 3)); } }

Compile & link:

   rustc main.rs -l add -L .

This works because both C and Rust produce object files that the linker can merge.

3. Real-World Examples of Multi-Language Projects

Project	Languages Used	Why?
Linux Kernel	C (core), Assembly (critical sections)	Performance for low-level hardware control.
FFmpeg	C (core), Assembly (SIMD optimizations)	Speed for video encoding/decoding.
Python Interpreter	C (core), Python (standard library)	C for speed, Python for flexibility.
Node.js	C++ (V8 engine), JavaScript (runtime)	C++ for performance, JS for usability.

4. The Big Challenge: ABI (Application Binary Interface)

Even if two languages compile to machine code, they might not work together if they disagree on:

How function arguments are passed (registers vs. stack).
How return values are handled.
Memory alignment of structures.

Example: ABI Mismatch Between Two Languages

Suppose:

Language A passes arguments in registers 0 and 1.
Language B expects them in registers 1 and 2.

Result: The function reads garbage values → Undefined behavior!

Solutions:

extern "C" in C++/Rust – Forces C-style calling conventions.
#[repr(C)] in Rust – Ensures structs match C memory layout.
Compiler flags (-fPIC, -mabi) to enforce compatibility.

5. When Should You Mix Languages?

Scenario	Good Language Choices
High-performance computing	C/Rust + Python (for scripting)
Embedded systems	C (firmware) + Assembly (critical loops)
Game development	C++ (engine) + Lua (scripting)
Web backends	Go (API) + Python (ML components)

Pros of Mixing Languages

✅ Performance – Optimize bottlenecks in a lower-level language.

✅ Reuse existing libraries (e.g., TensorFlow in Python calls C++ under the hood).

✅ Flexibility – Use the best tool for each part of the project.

Cons

⚠ Build complexity – Managing multiple compilers and linkers.

⚠ Debugging challenges – Mixed-language stack traces.

⚠ ABI risks – If calling conventions don’t match.

Final Thoughts

Mixing programming languages is not just possible—it’s common in performance-critical and large-scale systems. The key lies in:

Understanding the compilation pipeline (preprocessing → compilation → assembly → linking).
Using the linker effectively (static vs. dynamic libraries).
Ensuring ABI compatibility (so functions can call each other correctly).

Whether you're optimizing a game engine with C++ and Lua, speeding up Python with C extensions, or writing kernel modules in Rust and Assembly—knowing how languages interoperate unlocks new levels of performance and flexibility.

In the next part, we’ll explore how compiled languages (C, Rust) interact with interpreted ones (Python, JavaScript)—stay tuned!

Follow for More Deep Dives

Twitter (X): @aadarsh_nagrath
LinkedIn: Aadarsh Nagrath

Got questions? Drop them below! 🚀

Comprehensive Guide to gRPC

Aadarsh Nagrath — Wed, 23 Jul 2025 06:54:00 +0000

🧭 Guide to gRPC

Written by: Aadarsh Nagrath
🔗 GitHub • X (Twitter)

Introduction to gRPC

gRPC (gRPC Remote Procedure Calls) is a high-performance, open-source framework developed by Google for building scalable and efficient APIs. It leverages HTTP/2 as its transport protocol and Protocol Buffers (protobuf) as its Interface Definition Language (IDL), enabling fast, type-safe, and language-agnostic communication between systems. gRPC is designed for modern distributed systems, particularly microservices, real-time applications, mobile apps, and IoT devices, where low latency, high throughput, and cross-language interoperability are critical.

Key Features of gRPC

High Performance: Uses HTTP/2 for multiplexing, header compression, and binary framing, reducing latency and bandwidth usage.
Strong Typing: Protocol Buffers provide a schema-driven, type-safe way to define APIs.
Multi-Language Support: Generates client and server code for languages like Go, Java, Python, C++, TypeScript, and more.
Streaming: Supports unary (single request/response), server streaming, client streaming, and bidirectional streaming.
Built-in Features: Includes authentication, load balancing, retries, and deadlines out of the box.
Cross-Platform: Works across cloud, mobile, web, and IoT environments.

Why gRPC?

gRPC addresses limitations of traditional REST APIs and older RPC frameworks:

Compared to REST:
- Faster: Binary protocol (protobuf) vs. text-based JSON/XML.
- Lower latency: HTTP/2 multiplexing vs. HTTP/1.1 sequential requests.
- Streaming: Native support for real-time communication vs. REST’s reliance on WebSockets or polling.
- Strict contracts: Protobuf ensures consistency vs. REST’s flexible but error-prone payloads.
Compared to Older RPC:
- Modern protocol: HTTP/2 vs. proprietary protocols.
- Language-agnostic: Protobuf vs. language-specific stubs.
- Scalability: Designed for distributed systems vs. tightly coupled client-server models.

📦 REST vs GraphQL

Feature	REST	GraphQL
Endpoint structure	Multiple endpoints (`/users`, `/posts`)	Single endpoint (`/graphql`)
Data fetching	Fixed response structure	Client defines what data to fetch
Over-fetching / Under-fetching	Common problem	Avoided (fetch exactly what you want)
Versioning	Needs versioning (v1, v2, etc.)	Often no versioning needed
Response size	May be large or incomplete	Tailored to client's request

🛠 GraphQL Operations

Query – to read data

   query {
     user(id: "1") {
       name
       email
     }
   }

Mutation – to create/update/delete data

   mutation {
     addPost(title: "Hello", content: "World") {
       id
       title
     }
   }

Subscription – to get real-time updates

   subscription {
     messageAdded {
       content
       sender
     }
   }

🧩 Example Use Case

You can request nested and related data in one query:

query {
  user(id: "1") {
    name
    posts {
      title
      comments {
        content
      }
    }
  }
}

This replaces what would be multiple REST requests.

Protocol Buffers: The Heart of gRPC

Protocol Buffers (protobuf) is Google’s language-agnostic, extensible mechanism for serializing structured data. It serves as the IDL for gRPC, defining services, methods, and message types.

Benefits of Protocol Buffers

Unified Source of Truth:
- Protobuf files (*.proto) define the API contract, ensuring consistency across clients, servers, and documentation.
- Example: A single .proto file generates code for all supported languages, reducing discrepancies.
Type Safety:
- Enforces strict typing (e.g., string, int32, bool) and field validation at compile time.
- Example: A missing required field in a message triggers a compilation error, not a runtime failure.
Multi-Language Code Generation:
- Generates consistent client/server stubs for languages like Go, Java, Python, C++, Ruby, etc.
- Example: A UserService defined in protobuf generates a UserServiceClient and UserServiceServer in each language.
Forward/Backward Compatibility:
- Protobuf supports schema evolution with rules like:
  - Do not change field numbers.
  - Add new fields with new numbers.
  - Mark deprecated fields as reserved to prevent reuse.
- Example: Adding a new field to a message doesn’t break existing clients.
Compact and Efficient:
- Binary serialization is smaller and faster than JSON/XML.
- Example: A protobuf message is ~3-10x smaller than its JSON equivalent.
Readable Syntax:
- Protobuf’s syntax is concise and human-readable compared to verbose OpenAPI JSON/YAML.
- Example:
```
 message User {
   string id = 1;
   string name = 2;
 }
```

Protobuf Syntax

A typical .proto file includes:

Syntax: Specifies the protobuf version (e.g., syntax = "proto3").
Package: Defines a namespace to avoid naming conflicts (e.g., package user).
Service: Defines RPC methods (e.g., rpc GetUser(GetUserRequest) returns (GetUserResponse)).
Message: Defines data structures with typed fields (e.g., message GetUserRequest { string id = 1; }).
Options: Customizes code generation or behavior (e.g., option go_package = "./user").

Example:

syntax = "proto3";
package user;
option go_package = "./user";

service UserService {
  rpc GetUser(GetUserRequest) returns (GetUserResponse);
  rpc AddUser(AddUserRequest) returns (AddUserResponse);
}

message GetUserRequest {
  string id = 1;
}

message GetUserResponse {
  string id = 1;
  string name = 2;
}

message AddUserRequest {
  string name = 1;
}

message AddUserResponse {
  string id = 1;
}

gRPC Communication Patterns

gRPC supports four communication patterns, making it versatile for various use cases:

Unary RPC:
- Single request, single response.
- Similar to a traditional REST API call.
- Example: rpc GetUser(GetUserRequest) returns (GetUserResponse);
- Use Case: Fetching a user’s profile by ID.
Server Streaming RPC:
- Single request, stream of responses.
- Server sends multiple messages over a single connection.
- Example: rpc ListUsers(ListUsersRequest) returns (stream User);
- Use Case: Streaming real-time stock prices.
Client Streaming RPC:
- Stream of requests, single response.
- Client sends multiple messages, and the server responds once.
- Example: rpc UploadFile(stream FileChunk) returns (UploadResponse);
- Use Case: Uploading a large file in chunks.
Bidirectional Streaming RPC:
- Stream of requests and responses.
- Both client and server send messages asynchronously over a single connection.
- Example: rpc Chat(stream Message) returns (stream Message);
- Use Case: Real-time chat applications.

HTTP/2: The Transport Layer

gRPC uses HTTP/2 as its transport protocol, offering significant advantages over HTTP/1.1 (used in most REST APIs):

Multiplexing: Multiple requests/responses over a single TCP connection, reducing latency.
Header Compression: HPACK reduces header size, improving efficiency.
Binary Framing: Messages are sent as binary frames, not text, for faster parsing.
Persistent Connections: Keeps connections alive, reducing overhead for frequent requests.
Server Push: Servers can proactively send data to clients (less common in gRPC).

gRPC over HTTP/2 vs. REST over HTTP/1.1

Feature	gRPC (HTTP/2)	REST (HTTP/1.1)
Protocol	Binary (protobuf)	Text (JSON/XML)
Latency	Low (multiplexing, compression)	Higher (sequential requests)
Streaming	Native support	Limited (WebSockets/polling)
Type Safety	Strong (protobuf)	Weak (schema optional)
Bandwidth Usage	Low (binary)	Higher (text)
Browser Support	Limited (requires proxy)	Native

gRPC Ecosystem and Tools

gRPC integrates with a robust ecosystem of tools and libraries:

protoc: The Protocol Buffers compiler generates code from .proto files.
buf: A modern tool for managing protobuf schemas, linting, and generating code.
gRPC Plugins: Language-specific plugins (e.g., protoc-gen-go, protoc-gen-go-grpc) for code generation.
gRPC Gateway: Translates gRPC to REST/JSON for web clients.
Monitoring: Tools like Prometheus and Grafana for observability.
Testing: Tools like ghz for load testing and grpcurl for CLI-based testing.

Implementing a gRPC Service

Below is an expanded implementation based on the provided document, with additional features and best practices.

Prerequisites

Install the following:

protoc: Protocol Buffers compiler (protoc).
buf: Protobuf management tool.
Go Plugins:

  go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
  go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
  go install github.com/meshapi/grpc-api-gateway/codegen/cmd/protoc-gen-grpc-api-gateway@latest
  go install github.com/meshapi/grpc-api-gateway/codegen/cmd/protoc-gen-openapiv3@latest

Step 1: Set Up a buf Project

Create a buf.yaml to define the project and dependencies:

version: v1
deps:
  - buf.build/meshapi/grpc-api-gateway

Sync dependencies:

buf mod update

Create a buf.gen.yaml for code generation:

version: v1
plugins:
  - name: go
    out: .
    opt: paths=source_relative
  - name: go-grpc
    out: .
    opt: paths=source_relative
  - name: grpc-api-gateway
    out: .
    opt: paths=source_relative
  - name: openapiv3
    out: .
    opt: paths=source_relative

Step 2: Define the Protobuf Service

Create user.proto:

syntax = "proto3";
package user;
option go_package = "./user";

import "meshapi/gateway/annotations.proto";

service UserService {
  // Unary RPC: Add a new user
  rpc AddUser(AddUserRequest) returns (AddUserResponse) {
    option (meshapi.gateway.http) = {
      post: "/v1/users"
      body: "*"
    };
  }
  // Server streaming RPC: List users
  rpc ListUsers(ListUsersRequest) returns (stream User) {
    option (meshapi.gateway.http) = {
      get: "/v1/users"
    };
  }
  // Client streaming RPC: Bulk add users
  rpc BulkAddUsers(stream AddUserRequest) returns (BulkAddUsersResponse) {
    option (meshapi.gateway.http) = {
      post: "/v1/users/bulk"
      body: "*"
    };
  }
  // Bidirectional streaming RPC: Real-time chat
  rpc Chat(stream Message) returns (stream Message) {
    option (meshapi.gateway.http) = {
      get: "/v1/chat"
      streaming: WEBSOCKET
    };
  }
}

message AddUserRequest {
  string name = 1;
  string email = 2;
}

message AddUserResponse {
  string id = 1;
}

message ListUsersRequest {
  int32 limit = 1;
  int32 offset = 2;
}

message User {
  string id = 1;
  string name = 2;
  string email = 3;
}

message BulkAddUsersResponse {
  repeated string ids = 1;
}

message Message {
  string user_id = 1;
  string content = 2;
  int64 timestamp = 3;
}

Step 3: Generate Code

Run:

buf generate

This generates:

Go models (user.pb.go).
gRPC stubs (user_grpc.pb.go).
HTTP reverse proxy (user_grpc_api_gateway.pb.go).
OpenAPI v3 documentation (user_openapiv3.yaml).

Step 4: Implement the gRPC Service and Gateway

Create main.go:

package main

import (
    "context"
    "log"
    "net"
    "net/http"
    "github.com/meshapi/grpc-api-gateway/gateway"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
    pb "path/to/user"
)

type UserServiceServer struct {
    pb.UnimplementedUserServiceServer
    users map[string]*pb.User // In-memory storage for demo
}

func NewUserServiceServer() *UserServiceServer {
    return &UserServiceServer{users: make(map[string]*pb.User)}
}

func (s *UserServiceServer) AddUser(ctx context.Context, req *pb.AddUserRequest) (*pb.AddUserResponse, error) {
    id := generateID() // Implement ID generation
    s.users[id] = &pb.User{Id: id, Name: req.Name, Email: req.Email}
    return &pb.AddUserResponse{Id: id}, nil
}

func (s *UserServiceServer) ListUsers(req *pb.ListUsersRequest, stream pb.UserService_ListUsersServer) error {
    for _, user := range s.users {
        if err := stream.Send(user); err != nil {
            return err
        }
    }
    return nil
}

func (s *UserServiceServer) BulkAddUsers(stream pb.UserService_BulkAddUsersServer) error {
    var ids []string
    for {
        req, err := stream.Recv()
        if err != nil {
            if err == io.EOF {
                return stream.SendAndClose(&pb.BulkAddUsersResponse{Ids: ids})
            }
            return err
        }
        id := generateID()
        s.users[id] = &pb.User{Id: id, Name: req.Name, Email: req.Email}
        ids = append(ids, id)
    }
}

func (s *UserServiceServer) Chat(stream pb.UserService_ChatServer) error {
    for {
        msg, err := stream.Recv()
        if err != nil {
            return err
        }
        // Echo back the message for simplicity
        if err := stream.Send(msg); err != nil {
            return err
        }
    }
}

func main() {
    // Start gRPC server
    lis, err := net.Listen("tcp", ":40000")
    if err != nil {
        log.Fatalf("Failed to listen: %v", err)
    }
    grpcServer := grpc.NewServer()
    pb.RegisterUserServiceServer(grpcServer, NewUserServiceServer())
    go func() {
        log.Printf("gRPC server running on :40000")
        if err := grpcServer.Serve(lis); err != nil {
            log.Fatalf("Failed to serve gRPC: %v", err)
        }
    }()

    // Start HTTP gateway
    conn, err := grpc.NewClient(":40000", grpc.WithTransportCredentials(insecure.NewCredentials()))
    if err != nil {
        log.Fatalf("Failed to dial gRPC: %v", err)
    }
    restGateway := gateway.NewServeMux()
    pb.RegisterUserServiceHandlerClient(context.Background(), restGateway, pb.NewUserServiceClient(conn))
    log.Printf("HTTP gateway running on :4000")
    if err := http.ListenAndServe(":4000", restGateway); err != nil {
        log.Fatalf("Failed to serve HTTP: %v", err)
    }
}

func generateID() string {
    // Implement unique ID generation (e.g., UUID)
    return fmt.Sprintf("user-%d", time.Now().UnixNano())
}

Step 5: Run the Service

go mod init example
go mod tidy
go run .

Step 6: Test the Service

gRPC Client: Use grpcurl:

  grpcurl -plaintext localhost:40000 user.UserService.AddUser -d '{"name": "Alice", "email": "alice@example.com"}'

HTTP Gateway: Use curl:

  curl -X POST http://localhost:4000/v1/users -d '{"name": "Alice", "email": "alice@example.com"}'

Streaming (WebSocket):
- Use a WebSocket client to connect to ws://localhost:4000/v1/chat and send/receive messages.

gRPC Gateway: Bridging gRPC and REST

The gRPC API Gateway (as described in the provided document) translates gRPC services into RESTful HTTP/JSON endpoints, enabling web browsers, legacy systems, and REST-based clients to interact with gRPC services.

Key Features

Enhanced Configuration:
- Supports query parameters, path parameter renaming, and custom OpenAPI fields.
- Allows external YAML/JSON configuration files for flexibility.
- Example: Map a gRPC method to a REST endpoint with annotations:
```
 option (meshapi.gateway.http) = { post: "/v1/users", body: "*" };
```
OpenAPI 3.1 Support:
- Generates precise JSON Schema for validation and documentation.
- Optimizes model generation by including only relevant models.
Streaming Support:
- Server-Sent Events (SSE): For server-to-client streaming (e.g., real-time notifications).
- WebSocket: For bidirectional streaming (e.g., chat applications).
Robust Error Handling:
- Separates gRPC and HTTP errors for customized responses.
- Example: A gRPC NotFound error maps to HTTP 404 with a JSON payload.
Custom Marshalling:
- Supports custom content types (e.g., XML, YAML) via pluggable marshallers.

Technical Architecture

Code Generation:
- Analyzes .proto files and annotations.
- Generates Go handlers for HTTP requests.
- Produces OpenAPI documentation.
Runtime:
- HTTP handlers translate REST requests to gRPC calls.
- Supports WebSocket/SSE for streaming.
- Customizes error responses and header mappings.

Advanced gRPC Topics

Authentication and Authorization

Transport Security: gRPC uses TLS by default for encrypted communication.
Authentication:
- JWT: Include tokens in metadata (e.g., authorization: bearer <token>).
- OAuth 2.0: Integrate with OAuth for secure access.
- Mutual TLS: Authenticate both client and server using certificates.
Authorization: Use interceptors to enforce role-based access control (RBAC).
Example Interceptor (Go):

  func UnaryAuthInterceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
      md, ok := metadata.FromIncomingContext(ctx)
      if !ok || len(md["authorization"]) == 0 {
          return nil, status.Error(codes.Unauthenticated, "Missing auth token")
      }
      // Validate token
      return handler(ctx, req)
  }

Error Handling

gRPC uses a structured error model with status codes (e.g., codes.NotFound, codes.InvalidArgument) and details.
Example: Return a NotFound error:

  return nil, status.Errorf(codes.NotFound, "User %s not found", req.Id)

HTTP Gateway maps gRPC status codes to HTTP status codes (e.g., codes.NotFound → 404).

Load Balancing

gRPC supports client-side and server-side load balancing:
- Client-Side: Use a resolver (e.g., DNS, Kubernetes) to distribute requests.
- Server-Side: Use a proxy like Envoy for advanced load balancing.
Example: Configure a round-robin balancer in Go:

  conn, err := grpc.Dial("dns:///service.example.com", grpc.WithBalancerName("round_robin"))

Deadlines and Timeouts

gRPC allows setting deadlines to prevent hanging requests.
Example (Go client):

  ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
  defer cancel()
  resp, err := client.GetUser(ctx, req)

Interceptors

Server Interceptors: Intercept incoming requests for logging, metrics, or authentication.
Client Interceptors: Modify outgoing requests (e.g., add headers, retry logic).
Example: Logging interceptor (Go):

  func LoggingInterceptor(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
      start := time.Now()
      err := invoker(ctx, method, req, reply, cc, opts...)
      log.Printf("Method: %s, Duration: %v, Error: %v", method, time.Since(start), err)
      return err
  }

Streaming Best Practices

Server Streaming: Use for large datasets or real-time updates (e.g., stock tickers).
Client Streaming: Ideal for uploading large data (e.g., file uploads).
Bidirectional Streaming: Use for interactive apps (e.g., chat, gaming).
Error Handling: Handle io.EOF for stream closure and partial failures.
Example: Handle stream errors:

  func (s *UserServiceServer) ListUsers(req *pb.ListUsersRequest, stream pb.UserService_ListUsersServer) error {
      for _, user := range s.users {
          if err := stream.Send(user); err != nil {
              return status.Errorf(codes.Internal, "Failed to send: %v", err)
          }
      }
      return nil
  }

Monitoring and Observability

Use tools like Prometheus and Grafana for metrics (e.g., request latency, error rates).
Enable gRPC tracing with OpenTelemetry or Jaeger.
Example: Export metrics in Go:

  import "github.com/grpc-ecosystem/go-grpc-prometheus"
  grpcServer := grpc.NewServer(grpc.UnaryInterceptor(grpc_prometheus.UnaryServerInterceptor))

Testing gRPC Services

Unit Testing: Test individual RPC methods using mocks.
Integration Testing: Test client-server interactions with a real gRPC server.
Tools:
- grpcurl: CLI tool for sending gRPC requests.
- ghz: Load testing tool for gRPC.
- Postman: Test HTTP gateway endpoints.
Example: Test with grpcurl:

  grpcurl -plaintext -d '{"id": "user1"}' localhost:40000 user.UserService.GetUser

Security Considerations

TLS: Always use TLS in production (e.g., grpc.WithTransportCredentials(credentials.NewTLS(tlsConfig))).
Input Validation: Validate protobuf messages to prevent invalid data.
Rate Limiting: Use interceptors to enforce rate limits.
Metadata Security: Sanitize gRPC metadata to prevent injection attacks.

Performance Optimization

Connection Pooling: Reuse gRPC connections to reduce overhead.
Compression: Enable gRPC compression (e.g., grpc.WithCompressor(grpc.NewGzipCompressor())).
Batching: Combine small requests to reduce network calls.
Protobuf Optimization: Use scalar types (e.g., int32 vs. string) for smaller payloads.

Comparison with Other API Architectures

Feature	gRPC	REST	GraphQL
Protocol	HTTP/2, binary (protobuf)	HTTP/1.1, text (JSON/XML)	HTTP/1.1, text (JSON)
Performance	High (binary, multiplexing)	Moderate (text, sequential)	Moderate (text, single endpoint)
Type Safety	Strong (protobuf)	Weak (optional schemas)	Strong (GraphQL schema)
Streaming	Native (unary, bidirectional)	Limited (WebSockets/polling)	Limited (subscriptions)
Browser Support	Limited (requires proxy)	Native	Native
Flexibility	Strict contracts	Flexible payloads	Highly flexible queries
Use Case	Microservices, real-time	General-purpose, web-friendly	Client-driven data fetching

Real-World Use Cases

Microservices: Efficient inter-service communication (e.g., Netflix, Uber).
Real-Time Applications: Streaming data for gaming, chat, or live updates (e.g., Discord).
Mobile Apps: Low-bandwidth communication for better battery life (e.g., Google apps).
IoT: Lightweight communication for resource-constrained devices (e.g., smart home systems).
Cloud Services: Programmatic access to cloud resources (e.g., Google Cloud APIs).

Future Directions

Dynamic Reflection: Runtime service discovery without code generation.
Cross-Language Gateways: Support for Python, Rust, Node.js, etc.
Serverless gRPC: Integration with serverless platforms like AWS Lambda.
AI Integration: gRPC for low-latency AI model serving (e.g., TensorFlow Serving).
Enhanced Tooling: Improved linting, testing, and observability tools.

Conclusion

gRPC is a powerful framework for building high-performance, scalable APIs, particularly suited for microservices, real-time applications, and polyglot environments. Its use of Protocol Buffers, HTTP/2, and streaming capabilities makes it a modern alternative to REST and older RPC frameworks. By leveraging tools like gRPC Gateway, developers can bridge the gap between gRPC and REST, making it accessible to web clients while retaining its performance benefits. Whether you’re building a microservices architecture or a real-time application, gRPC equips you with the tools to create robust, efficient, and maintainable APIs.

Happy coding!

How Notion Handles 200 Billion Notes Without Crashing: A Technical Deep Dive

Aadarsh Nagrath — Wed, 26 Mar 2025 20:19:51 +0000

Notion isn’t just a note-taking app—it’s a real-time, collaborative database masquerading as a minimalist UI. But how does it handle 200+ billion blocks (database entries) without buckling under pressure?

Let’s peel back the layers.

Notion’s Data Model: A Database in Disguise

Every action in Notion—typing text, toggling a checkbox—creates a block, stored as a row in PostgreSQL. Here’s the schema of a block:

CREATE TABLE blocks (
  id UUID PRIMARY KEY,
  type VARCHAR(50),       -- 'paragraph', 'checkbox', etc.
  parent_id UUID,         -- Parent block (if nested)
  workspace_id UUID,      -- Critical for sharding
  properties JSONB        -- Flexible content storage
);

Fun Fact:

When you hit ↑ or ↓ in Notion, it’s not moving a cursor—it’s jumping between database rows. A "simple" page with 10 elements = 10+ database queries!

The Scaling Crisis: 20 Billion Blocks and Chaos

Why Postgres Almost Died

Single database: Pre-2021, all blocks lived in one PostgreSQL instance.
Indexes bloated: Querying a workspace’s blocks required scanning millions of rows.
Writes stalled: Concurrent edits from 100M users? Good luck.

The Sharding Savior

Notion split data into 480 logical shards (32 machines × 15 shards each).
How?

Shard Key: workspace_id (every block belongs to one workspace).
Routing: Application code directs queries like:

   shard = hash(workspace_id) % 480  # Simple but effective

Pro Tip:

Sharding isn’t free. Cross-shard queries (e.g., global search) require fan-out—sending requests to all shards. Notion avoids this with Elasticsearch for search.

Zero-Downtime Migration: The Double-Write Dance

Migrating 20B+ blocks live? Notion’s 3-step playbook:

Double Writes:

   def save_block(block):
     old_db.write(block)  # Legacy database
     new_sharded_db.write(block)  # New shard

Audit Logs: A Kafka stream recorded all changes for reconciliation.
The Big Cutover:
- 5-minute downtime to flip the switch.
- Dark reads compared old/new DB outputs silently.

Underrated Hack:

They used PostgreSQL’s logical replication to sync shards—no third-party tools needed.

Beyond Postgres: The Data Lake Pivot

Postgres is great for OLTP (transactions), but analytics? Not so much. Notion’s new pipeline:

graph LR
  A[Postgres Shards] -->|CDC| B[Apache Kafka]
  B --> C[Apache Spark]
  C --> D[Amazon S3 Data Lake]
  D --> E[Snowflake for Analytics]

Why?

Cost: Snowflake charges per query; S3 is dirt-cheap storage.
AI Ready: Unstructured data (e.g., page content) fuels Notion’s AI features.

Gotcha:

Snowflake hates updates. Notion’s update-heavy workload forced them to build custom tooling.

Open Source Gems in Notion’s Stack

Apache Kafka: Handles 4M+ messages/sec during peak loads.
pgbouncer: Connection pooling to avoid Postgres meltdowns.
Apache Hudi: Manages change-data-capture (CDC) for the data lake.

Lesson:

Notion’s engineers prefer boring tech—Postgres over MongoDB, Kafka over proprietary queues.

The Future: What’s Next for Notion’s Stack?

Edge Caching: Reduce latency for global users.
Columnar Storage: For faster analytics (e.g., Apache Parquet).
Vector Databases: To power AI features (semantic search, etc.).

Why This Matters for You

Even if you’re not building the next Notion, these principles apply:

Sharding: Plan it early. Use a clear key (like workspace_id).
Data Lakes: Separate OLTP and analytics early.
Open Source: Avoid vendor lock-in; Kafka/Postgres will outlive any SaaS.

Found this interesting ?

To follow me -
MY Github, Twitter

How to run DeepSeek Locally in Your Terminal Like a Pro

Aadarsh Nagrath — Sun, 02 Feb 2025 15:27:53 +0000

If you've been looking for a way to use cutting-edge AI without relying on expensive cloud services, you're in luck! DeepSeek R1 is an open-source reasoning model that runs entirely on your local machine.

With a few simple steps, you can ditch online AI services and gain full control over your own AI assistant. In this guide, I'll show you how to get DeepSeek R1 running in your terminal using Ollama, the easiest way to run open-source models locally.

To follow me -
MY Github, Twitter

Step 1: Install Ollama

Before we can run DeepSeek, we need to install Ollama, which is a tool that lets you download and run AI models with a single command. Here's how you can install it based on your OS:

🖥️ macOS (with Homebrew)

brew install ollama

🖥️ Linux (Debian/Ubuntu)

curl -fsSL https://ollama.com/install.sh | sh

🖥️ Windows (via WSL)

For Windows, you'll need to install Windows Subsystem for Linux (WSL) and then run the Linux installation command inside WSL.

Once installed, restart your terminal and verify the installation by running:

ollama --version

Step 2: Download & Run DeepSeek R1

DeepSeek R1 comes in different model sizes, ranging from 1.5 billion to a massive 671 billion parameters. Generally:

Smaller models (1.5B - 7B) → Fast but less powerful
Larger models (67B - 671B) → Smarter but require high-end GPUs

To get started, let's download and run the 7B model, which is a good balance between performance and intelligence:

ollama run deepseek-r1:7b

This command does two things:

Downloads the 7B model (only the first time you run it)
Starts an interactive chat session with DeepSeek R1 in your terminal

If you want a lighter version, you can try:

ollama run deepseek-r1:1.5b

And if you have a powerful setup and want maximum reasoning power, go for:

ollama run deepseek-r1:67b

(Just make sure you have enough VRAM, as the largest models require high-end GPUs!)

Step 3: Use DeepSeek R1 in the Terminal

Once the model starts, you'll see a chat-like interface where you can ask it anything:

> How does quicksort work?

DeepSeek will generate a detailed response in real-time!

Customize Your Experience

Ollama provides options to tweak the AI’s behavior. Here are some useful parameters:

Set a custom system prompt

  ollama run deepseek-r1 --system "You are an expert software engineer. Answer concisely."

Use temperature for randomness (higher = more creative)

  ollama run deepseek-r1 --temperature 0.7

Limit response length

  ollama run deepseek-r1 --max-tokens 256

Step 4: Run DeepSeek R1 in API Mode

Want to use DeepSeek programmatically? Ollama provides an API mode:

ollama serve

This starts a local API server. Now, you can send requests using cURL:

curl http://localhost:11434/api/generate -d '{"model": "deepseek-r1:7b", "prompt": "Explain recursion."}'

Or integrate it into your Python/Node.js projects.

Conclusion

Congratulations! You just installed, configured, and ran DeepSeek R1 locally in your terminal. Now, you have an open-source AI assistant at your fingertips—no internet, no cloud fees, just raw AI power on your machine.

Go ahead, experiment with different model sizes, tweak the settings, and enjoy the power of local AI! 🚀