Forem: Dhawal Kumar Singh

Message Queues Explained: RabbitMQ vs Kafka vs SQS — When to Use What

Dhawal Kumar Singh — Fri, 24 Apr 2026 08:20:34 +0000

Why Message Queues?

Imagine Service A needs to send emails after a user signs up. If it calls the email service directly:

What if the email service is down? The signup fails.
What if it's slow? The user waits.
What if 10,000 users sign up at once? Everything crashes.

A message queue solves all three problems. Service A drops a message in the queue and moves on. The email service picks it up when it's ready.

Core Concepts

Producer → Queue → Consumer

The basic pattern is simple:

Producer publishes a message to the queue
Queue stores the message durably
Consumer pulls the message and processes it

The producer and consumer are completely decoupled — they don't know about each other.

Key Benefits

Decoupling: Services evolve independently
Buffering: Handle traffic spikes gracefully
Reliability: Messages persist even if consumers are down
Scalability: Add more consumers to process faster

The Big Three: RabbitMQ vs Kafka vs SQS

RabbitMQ

What it is: A traditional message broker implementing AMQP protocol.

Best for:

Task distribution (send emails, process images)
Complex routing (route messages based on headers/topics)
Request-reply patterns
When you need message-level acknowledgments

How it works:

Messages are pushed to consumers
Once acknowledged, messages are deleted
Supports exchanges for flexible routing (direct, topic, fanout)

Apache Kafka

What it is: A distributed event streaming platform.

Best for:

Event sourcing and event-driven architecture
Real-time analytics and data pipelines
Log aggregation
When you need to replay events

How it works:

Messages are appended to an immutable log (topic partitions)
Consumers pull messages and track their own offset
Messages are retained for a configurable period (not deleted after consumption)
Partitions enable parallel processing

AWS SQS

What it is: A fully managed message queue service.

Best for:

Simple async processing on AWS
When you want zero operational overhead
Decoupling Lambda functions and microservices

How it works:

Standard queues: at-least-once delivery, best-effort ordering
FIFO queues: exactly-once processing, strict ordering
Fully serverless — no clusters to manage

Quick Comparison

Feature	RabbitMQ	Kafka	SQS
Delivery	Push	Pull	Pull
Ordering	Per-queue	Per-partition	Best-effort (FIFO available)
Retention	Until acknowledged	Time-based	4 days (configurable)
Throughput	Moderate	Very high	High
Replay	No	Yes	No
Operations	Self-managed	Self-managed	Fully managed

Common Patterns

1. Point-to-Point

One message → one consumer. Classic task queue.

2. Pub/Sub

One event → many subscribers. Each subscriber gets a copy.

3. Dead Letter Queue

Messages that fail processing N times go to a separate DLQ for investigation.

4. Competing Consumers

Multiple consumer instances read from the same queue — work gets distributed automatically.

When to Use What

"I need to send background jobs" → RabbitMQ or SQS
"I need to stream events to multiple consumers" → Kafka
"I want zero ops and I'm on AWS" → SQS
"I need to replay past events" → Kafka
"I need complex message routing" → RabbitMQ

Learn More

Full guide at swehelper.com/blog/message-queues. Also check out the Kafka deep dive and Pub/Sub patterns.

Plus 119+ free developer tools at swehelper.com/tools — no signup required.

Rate Limiting in Distributed Systems: Algorithms Every Backend Developer Should Know

Dhawal Kumar Singh — Tue, 21 Apr 2026 08:32:31 +0000

Rate limiting is one of those things you don't think about until your API gets hammered at 3 AM.

Whether it's a DDoS attack, a buggy client sending 10,000 requests per second, or a partner integration gone wrong — rate limiting is your first line of defense.

Why Rate Limiting Matters

Without rate limiting:

A single client can starve others of resources
Your database gets overwhelmed during traffic spikes
You can't enforce fair usage across API consumers
Cost of serving requests spirals out of control

The 4 Core Algorithms

1. Token Bucket

The most widely used algorithm. Think of it as a bucket that fills with tokens at a steady rate. Each request removes a token. No tokens? Request denied.

Why it works: Allows short bursts while enforcing long-term rate. Used by AWS, Stripe, and most major APIs.

2. Leaky Bucket

Requests enter a queue (the bucket) and are processed at a fixed rate. If the queue is full, new requests are dropped.

Why it works: Provides a perfectly smooth output rate. Great for downstream services that can't handle bursts.

3. Fixed Window Counter

Divide time into fixed windows (e.g., 1 minute). Count requests per window. Reset at the boundary.

Why it works: Dead simple to implement. But has the "boundary problem" — 2x the allowed rate can hit at window edges.

4. Sliding Window Log/Counter

Tracks the exact timestamp of each request (log) or uses a weighted combination of current and previous windows (counter).

Why it works: Most accurate. No boundary problem. But uses more memory.

Quick Comparison

Algorithm	Burst Handling	Memory	Accuracy
Token Bucket	✅ Allows bursts	Low	Good
Leaky Bucket	❌ Smooths all	Low	Good
Fixed Window	⚠️ Boundary issue	Very Low	Fair
Sliding Window	✅ Accurate	Higher	Best

Where to Rate Limit

API Gateway — First line of defense (per-client limits)
Application layer — Business logic limits (e.g., 5 password attempts)
Database layer — Connection pooling and query limits

What Happens When Limits Are Hit?

Return 429 Too Many Requests with these headers:

X-RateLimit-Limit — Max requests allowed
X-RateLimit-Remaining — Requests left in window
Retry-After — Seconds until the client can retry

Deep Dive

I wrote a comprehensive guide covering distributed rate limiting, Redis implementations, and real-world patterns used by companies like Stripe and Cloudflare:

👉 Rate Limiting: The Complete Guide

This is part of my system design series at SWE Helper — free tools, guides, and interview prep for software engineers.

SQL vs NoSQL — A Decision Framework That Actually Works

Dhawal Kumar Singh — Mon, 20 Apr 2026 11:19:37 +0000

Every developer argues about SQL vs NoSQL. Most of the advice online is surface-level.

Here's a practical decision framework based on how real systems are built 👇

The Core Difference

SQL databases store data in structured tables with predefined schemas. Think rows and columns with relationships enforced by the database itself.

NoSQL databases store data in flexible formats — documents, key-value pairs, wide columns, or graphs. Schema is defined by the application, not the database.

When to Choose SQL

PostgreSQL, MySQL, SQL Server

Pick SQL when you need:

ACID transactions — money transfers, order processing, inventory management
Complex queries — multi-table JOINs, aggregations, reporting
Data integrity — foreign keys, constraints, cascading deletes
Mature tooling — ORMs, migration tools, monitoring

Real examples:

Stripe uses PostgreSQL for payment processing
GitHub uses MySQL for repository metadata
Shopify uses MySQL for order management

When to Choose NoSQL

MongoDB, Cassandra, DynamoDB, Redis

Pick NoSQL when you need:

Flexible schemas — user profiles with varying fields, product catalogs
Horizontal scaling — write throughput across geographic regions
High availability — systems that can never go down
Specific access patterns — key-value lookups, time-series data

Real examples:

Netflix uses Cassandra for streaming metadata (high availability, global scale)
Uber uses Cassandra for real-time GPS tracking (massive write throughput)
Discord uses Cassandra for message storage (billions of messages)

The 4-Question Decision Framework

When designing a system, ask:

1. Do I need transactions across multiple entities?
→ Yes? Start with SQL. Multi-document transactions in NoSQL exist but add complexity.

2. Will my schema evolve frequently?
→ Yes? NoSQL handles schema changes gracefully. SQL migrations can be painful at scale.

3. Do I need to scale writes globally?
→ Yes? NoSQL databases like Cassandra and DynamoDB are built for this. SQL write-scaling requires sharding, which is hard.

4. Do I need complex aggregation queries?
→ Yes? SQL is purpose-built for this. NoSQL requires you to pre-compute or use separate analytics tools.

What Senior Engineers Actually Do: Polyglot Persistence

Most production systems use both:

Company	SQL (Transactions)	NoSQL (Scale)
Uber	MySQL (trips, billing)	Cassandra (GPS, real-time)
Netflix	PostgreSQL (billing)	Cassandra + DynamoDB (content)
Airbnb	MySQL (bookings)	Redis (caching) + Elasticsearch (search)

The answer isn't "SQL or NoSQL." It's "which one, where."

CAP Theorem Connection

Your database choice is fundamentally tied to the CAP theorem:

SQL databases tend to be CP — they prioritize consistency, may become unavailable during partitions
NoSQL databases tend to be AP — they prioritize availability, may serve slightly stale data

Understanding this trade-off is what separates junior from senior engineers in system design interviews.

Wrapping Up

SQL for transactions, integrity, and complex queries
NoSQL for scale, flexibility, and availability
Most real systems use both (polyglot persistence)
The right answer depends on your access patterns, not the technology trend

📚 Full guide with comparison tables: swehelper.com/blog/sql-vs-nosql
🔧 Free dev tools (120+): swehelper.com

CAP Theorem — What Every Developer Gets Wrong

Dhawal Kumar Singh — Sun, 19 Apr 2026 17:41:10 +0000

If you've ever been in a system design interview, you've heard: "Explain the CAP theorem."

Most developers say: "You can only pick two out of three — Consistency, Availability, Partition Tolerance."

That's technically correct but misses the real point. Let me explain 👇

🔍 The Three Properties

Consistency (C)
Every read receives the most recent write.

You write balance = $100. Someone immediately reads → they get $100. Not $90. Not stale data.

Availability (A)
Every request gets a response — even if it's not the latest data.

The system never says "try later."

Partition Tolerance (P)
The system keeps working even when the network connection between nodes breaks.

🚨 The Big Misconception

Most people think you're choosing 2 from a menu:

Choice	What happens	Practical?
CA	Consistent + Available, no partition handling	❌ Not in distributed systems
CP	Consistent, may reject requests during partitions	✅ Yes
AP	Available, may serve stale data during partitions	✅ Yes

Here's what people get wrong:

You don't "choose" partition tolerance. Network partitions WILL happen — they're not optional.

So the real choice is:

🎯 During a network partition, do you want Consistency or Availability?

That's it. That's CAP.

🏢 Real-World Examples

CP Systems — Choose Consistency

System	Behavior	Best for
MongoDB (majority write concern)	Minority side rejects writes during partition	Financial data
HBase	Strong consistency, may become unavailable	Analytics platforms

Use case: Banking — wrong data is worse than no data.

AP Systems — Choose Availability

System	Behavior	Best for
Cassandra	Always accepts writes, resolves conflicts later (last-write-wins)	Social feeds
DynamoDB	Returns data even if slightly stale	Shopping carts

Use case: Social media feeds — slightly stale data is totally fine.

The "CA" Myth

Traditional single-node PostgreSQL is consistent AND available — but it's not distributed.

The moment you scale to multiple nodes, you must handle partitions. CA doesn't exist in the real world of distributed systems.

🧠 Beyond CAP: The PACELC Theorem

CAP only describes behavior during partitions. But what about normal operation? That's where PACELC comes in.

PACELC says:

If there's a Partition → choose Availability or Consistency
Else (normal operation) → choose Latency or Consistency

System	During Partition	Normal Operation
DynamoDB (PA/EL)	Available	Low Latency
HBase (PC/EC)	Consistent	Consistent
Cassandra (PA/EC)	Available	Consistent (tunable)

This is honestly more useful than CAP for real-world system design decisions.

💡 Interview Tip

When asked about CAP in an interview, don't just recite the definition. Say something like:

"Network partitions are inevitable in distributed systems, so the real trade-off is between consistency and availability during partitions. In practice, I'd use PACELC to also consider the latency-consistency trade-off during normal operation.

For example, for a payment system I'd choose a CP system like PostgreSQL with synchronous replication. But for a social media feed, I'd choose an AP system like Cassandra where eventual consistency is acceptable."

That answer shows you understand it, not just memorized it.

Wrapping Up

CAP isn't really "pick 2 of 3" — it's "pick C or A during a partition"
PACELC is the practical extension you should actually use
Real systems often have tunable consistency (Cassandra lets you choose per query)
In interviews, always explain the trade-off, not just the acronym

If you found this useful, I write about system design and distributed systems regularly. I also maintain 119+ free browser-based dev tools for developers.

📚 Full article with more examples: swehelper.com/blog/cap-theorem
🔧 Free dev tools: swehelper.com