Forem: Sibasish Mohanty

Real-Time Systems: Making Data Flow Live

Sibasish Mohanty — Mon, 15 Sep 2025 07:51:38 +0000

In the final part of our System Design series, we’re focusing on how to deliver real-time data effectively, balancing complexity, latency, and reliability.

We’ll cover:

Serialization – JSON, Avro, Proto
Real-Time Delivery – Polling, SSE, WebSockets

1. Serialization

TL;DR: Choose the right format to balance speed, size, and compatibility.

JSON: Easy to debug, human-readable, but large and slow to parse.
Avro: Compact, schema evolution support, good for Kafka.
Protobuf: Highly efficient, strict schema, widely used in gRPC.

👉 Example: Use Protobuf for high-throughput internal RPC between microservices, JSON for external APIs.

👉 Interview tie-in: "Why not use JSON everywhere?" — Size and parsing speed trade-offs.

2. Real-Time Delivery

TL;DR: Choose the right mechanism based on use case and scale.

Polling: Simple but inefficient; periodic client requests.
Server-Sent Events (SSE): Server pushes updates over HTTP, uni-directional.
WebSockets: Full-duplex communication, ideal for bidirectional interaction.

👉 Example: Stock trading platform uses WebSockets for real-time price updates. Simple dashboards use SSE for infrequent updates.

👉 Interview tie-in: "When would you use SSE over WebSockets?" — SSE for simple streaming, WebSockets for interactive apps.

✅ Takeaways

Pick serialization format based on throughput, compatibility, and schema evolution.
Use polling only if absolutely necessary; prefer SSE or WebSockets for real-time.

💡 Practice Question:

"Design a live sports score app that supports thousands of concurrent users. How do you handle real-time updates efficiently and ensure message order?"

From Deployment to Architecture: Designing for Change and Scale

Sibasish Mohanty — Mon, 15 Sep 2025 07:48:09 +0000

In part 8 of our System Design series, we’ll explore the practices and patterns that help systems evolve, stay resilient, and simplify operations over time.

We’ll cover:

Service Discovery – Dynamic endpoints, central config
Deployment Strategies – Canary, blue/green, rollbacks
Monolith vs Microservices – Split only if ops cost justified
Distributed Transactions – Use sagas, avoid 2PC
Event Sourcing & CQRS – Append-only logs, splitting models

1. Service Discovery

TL;DR: Don’t hardcode service endpoints; let services find each other dynamically.

Central Registry (e.g., Consul, Eureka): Services register and clients query the registry.
Client-side Discovery: Clients load balance by querying registry.
Server-side Discovery: Load balancer handles discovery logic.

👉 Example: Kubernetes services use DNS and service mesh proxies (e.g., Envoy).

👉 Interview tie-in: "How does a service find other services in a dynamic environment?" — Central registry approach.

2. Deployment Strategies

TL;DR: Minimize risk during releases.

Blue/Green Deployment: Deploy to parallel environment, then switch traffic.
Canary Deployment: Gradually expose new version to a subset of users.
Rollbacks: Always have a plan to revert to stable version.

👉 Example: New feature rolled out to 5% of users first, monitor errors, then full rollout.

👉 Interview tie-in: "How do you minimize deployment risk in production?" — Canary or blue/green deployment.

3. Monolith vs Microservices

TL;DR: Don’t microservice just because it sounds trendy.

Monolith: Simple to develop but harder to scale as code grows.
Microservices: Independent scaling and deployments but added complexity.

👉 Example: Start monolithic, split when ops bottlenecks emerge.

👉 Interview tie-in: "When would you split a monolith?" — Ops complexity or team size grows.

4. Distributed Transactions

TL;DR: Avoid 2PC; prefer sagas.

2PC (Two-Phase Commit): Strong consistency, high latency.
Sagas: Local transactions with compensating actions.

👉 Example: Order triggers inventory decrement and payment; if payment fails, inventory rollback occurs.

👉 Interview tie-in: "How do you handle cross-service transactions?" — Sagas.

5. Event Sourcing & CQRS

TL;DR: Append-only logs with separate read/write models.

Event Sourcing: Store all events immutably.
CQRS: Separate models for reads and writes.

👉 Example: Event store keeps actions history; read model serves queries.

👉 Interview tie-in: "Why use Event Sourcing over CRUD?" — Auditability and state rebuilds.

✅ Takeaways

Use service discovery for dynamic environments
Deploy safely with blue/green or canary strategies
Microservices make sense only if ops cost justified
Implement sagas over 2PC for distributed transactions
Use Event Sourcing & CQRS when audit and performance matter

💡 Practice Question:

"Design a booking system where inventory, payment, and notification services coordinate. How do you ensure eventual consistency without 2PC?"

Event-Driven Systems: Building for Scale and Decoupling

Sibasish Mohanty — Thu, 11 Sep 2025 08:34:53 +0000

In part 7 of our System Design series, we’re focusing on asynchronous communication patterns that help systems scale, remain resilient, and avoid tight coupling.

We’ll cover:

Queues & Streams – Kafka, RabbitMQ; smoothing spikes
Backpressure – Managing slow consumers and load shedding
Delivery Semantics – At-most-once, at-least-once, exactly-once

1. Queues & Streams

TL;DR: Decouple producers from consumers to handle traffic spikes and processing delays.

Message Queues: RabbitMQ, SQS – good for task-based workloads.
Event Streams: Kafka, Kinesis – append-only logs, useful for event sourcing and replay.

👉 Example: Order placement publishes events to a queue; different consumers handle billing, inventory update, and notification separately.

👉 Interview tie-in: "Why use a queue instead of direct HTTP calls?" — Decoupling improves resilience and scales better under spikes.

2. Backpressure

TL;DR: Don’t let slow consumers cause system meltdown.

Load Shedding: Drop low-priority messages when overwhelmed.
Buffer Limits: Fixed queue sizes prevent uncontrolled memory growth.
Consumer Throttling: Consumers signal readiness to receive data.

👉 Example: Kafka pauses sending data to a consumer when it’s overloaded until the consumer catches up.

👉 Interview tie-in: "How do you handle a slow consumer in a messaging system?" — Implement backpressure strategies.

3. Delivery Semantics

TL;DR: Pick the right guarantee based on business needs.

At-most-once: Risk of message loss, but no duplicates (good for non-critical notifications).
At-least-once: Duplicates possible; consumers must be idempotent (e.g., order processing).
Exactly-once: Hard to implement; requires complex coordination (Kafka Streams offers it in some cases).

👉 Example: Payment processing must be exactly-once; use unique transaction IDs to avoid duplication.

👉 Interview tie-in: "Which delivery guarantee would you pick for sending email notifications?" — At-most-once is acceptable since duplicates are harmless.

✅ Takeaways

Use queues and streams to decouple and scale
Implement backpressure to prevent system overload
Choose the right delivery semantics based on business criticality

💡 Practice Question:

"Design a video processing pipeline where users upload videos and get them transcoded. How do you handle spikes and ensure each video is processed exactly once?"

Don’t Trust, Just Verify: Auth, Faults, and Monitoring

Sibasish Mohanty — Thu, 11 Sep 2025 08:32:48 +0000

In part 6 of our System Design series, we’ll tackle the critical pillars of system reliability and security that keep services running smoothly and securely.

We’ll cover:

AuthN & AuthZ – OAuth2, JWT, RBAC/ABAC
Resilience – Circuit breakers, timeouts, retries
Observability – Logs, metrics, traces, SLI/SLO
Health Checks – Detect failures, auto-replace
Redundancy – Avoid single points of failure (SPOFs); multi-AZ/region

1. AuthN & AuthZ

TL;DR: Authentication proves identity, Authorization checks permissions.

OAuth2: Delegated authorization, useful for third-party apps.
JWT (JSON Web Tokens): Stateless, signed tokens for identity verification.
RBAC (Role-Based Access Control): Permissions based on roles.
ABAC (Attribute-Based Access Control): Fine-grained control based on attributes.

👉 Example: User logs in → receives a JWT → Access to /orders endpoint is verified using claims in the JWT.

👉 Interview tie-in: "How would you design auth for a multi-tenant SaaS app?" — Use OAuth2 + scoped JWTs.

2. Resilience

TL;DR: Systems must fail gracefully.

Circuit Breakers: Stop calling a failing service to prevent cascading failure.
Timeouts: Don’t wait forever for a response.
Retries: Retry transient failures with exponential backoff.

👉 Example: Payment service calls third-party gateway; use circuit breaker to avoid repeated failures.

👉 Interview tie-in: "What happens when a downstream service is down?" — Circuit breaker opens, fallback or error is returned.

3. Observability

TL;DR: If you can’t measure it, you can’t improve it.

Logs: For detailed debugging
Metrics: Quantitative measures (e.g., QPS, error rate)
Traces: End-to-end request flow (distributed tracing)
SLI (Service Level Indicator): Measured metric (e.g., 99.9% requests < 200ms)
SLO (Service Level Objective): Target (e.g., 99.9% of requests < 200ms)

👉 Example: Grafana dashboard shows error rates and latency percentiles.

👉 Interview tie-in: "How do you monitor a microservice-based architecture?" — Combine logs, metrics, and tracing.

4. Health Checks

TL;DR: Detect problems early and recover automatically.

Liveness Check: Is the process alive?
Readiness Check: Can it handle requests?

👉 Example: Kubernetes probes services every few seconds and restarts if unhealthy.

👉 Interview tie-in: "How do you prevent traffic to unhealthy services?" — Use readiness probes + service discovery.

5. Redundancy

TL;DR: No SPOFs. Multi-AZ or Multi-Region deployments for resilience.

👉 Example: Primary DB in us-east-1, replica in us-west-1.

👉 Interview tie-in: "How do you handle data center failure?" — Multi-region replication, failover mechanisms.

✅ Takeaways

Design auth flows with security and scalability in mind
Build resilience using circuit breakers, timeouts, retries
Implement observability: logs, metrics, traces, SLI/SLO
Health checks prevent serving requests from unhealthy instances
Use redundancy to avoid SPOFs and enable disaster recovery

💡 Practice Question:

"Design an auth system for an API gateway in a microservices architecture. How do you enforce per-service access control and trace requests end-to-end?"

Systems Under Pressure: Race Conditions, Rate Limits, and Recovery

Sibasish Mohanty — Wed, 10 Sep 2025 06:55:55 +0000

In part 5 of our System Design series, we’re focusing on how systems behave under load and concurrency challenges. These are critical topics that often trip up engineers in interviews and production.

We’ll cover:

Concurrency – Locks, MVCC, retries
Multithreading – Pools, contention, switching
Idempotency – Same request, same effect
Rate Limiting – Fairness and protection

1. Concurrency

TL;DR: Multiple operations happening at once? Protect shared data and avoid races.

Locks (mutexes): Simple but can cause deadlocks and performance issues.
MVCC (Multi-Version Concurrency Control): Every write creates a new version; readers never block writers.
Retries: Optimistic concurrency by retrying on conflict.

👉 Example: Bank transfer between accounts uses locks to prevent double spending.

👉 Interview tie-in: "How do you handle concurrent writes to the same data?" — Answer with locking strategies or MVCC.

2. Multithreading

TL;DR: Parallelism needs control.

Thread pools: Pre-allocated worker threads to handle tasks.
Contention: Avoid too many threads competing for the same resource.
Context switching: Expensive if overused.

👉 Example: Web server uses a thread pool to handle incoming HTTP requests, limiting concurrency to prevent overload.

👉 Interview tie-in: "Why not just spawn a new thread per request?" — Explain thread pool and resource limits.

3. Idempotency

TL;DR: Repeating the same request should have the same effect.

Why important: Retries due to network errors should not cause duplicate side effects.

👉 Example: HTTP PUT /users/123/profile is idempotent; POST /orders is not, unless you provide a unique order ID.

👉 Interview tie-in: "Design a payment API that handles retries safely." — Focus on idempotent design using unique transaction IDs.

4. Rate Limiting

TL;DR: Prevent abuse and manage fair usage.

Strategies:
- Fixed window (e.g., 100 requests per minute)
- Sliding window
- Token bucket (allows bursts)

👉 Example: Twitter limits API usage per developer key.

👉 Interview tie-in: "How do you prevent a single client from overloading your system?" — Implement rate limiting with sliding window or token bucket.

✅ Takeaways

Protect shared data using locks or MVCC
Use thread pools to control parallelism
Ensure APIs are idempotent to handle retries
Implement rate limiting to avoid abuse and spikes

💡 Practice Question:

"Design an API for sending SMS messages. How do you ensure idempotency and rate limits are enforced?"

From Simplicity to Complexity: My Love-Hate Journey with Next.js

Sibasish Mohanty — Tue, 09 Sep 2025 03:13:41 +0000

I remember the day I started using Next.js like it was yesterday. It was version 8. And honestly? It was love at first sight. I bootstraped our new project in Nextjs despite my CTOs hesitation and made my teammates who never wrote a single line of code in React, learn Nextjs and ship with confidence.

File-based routing.
SSR. SSG. ISR.
Zero config.
You slapped a file in the pages/ folder, and boom—you had a route.
Simple, elegant, almost… magical.

// pages/about.js
export default function About() {
  return <div>About us</div>
}

It did what React should have done out of the box but didn’t. I was sold.
Deploy it, forget about webpack configs, no client-side routing hacks. Just productivity.

The Shift: Enter Next.js 13 & The App Router Madness

Then Next.js 13 hit, and things changed.
Suddenly, pages/ was optional. Enter the app/ directory. Layouts. Templates. Nested routing. React Server Components. use client directives.

It felt… like betrayal.

// app/layout.js
export default function Layout({ children }) {
  return <div className="layout">{children}</div>
}

Why?
Because the mental model shifted from “Page = Route” to “Segment = Route + Layout + Data Layer + Responsibility Split.”
Now I had to think in terms of route segments and where data should live: server component vs client component vs fetch cache vs what the hell.

The Mental Model Shift Nobody Talks About

Before, I thought:
“I add a file, it’s a route. Done.”

Now I have to ask:

Is this a server component or client component?
Should data fetching happen in generateStaticParams or inline in the component?
Where does this layout go?
Is this a template or a layout?

It’s not just about writing code anymore.
It’s about architecting for Next.js’s new ecosystem.

// app/products/[id]/page.js
export default function ProductPage({ params }) {
  const { id } = params
  // Do I fetch here? Should this be a server component?
}

Fun, right?

Love, But With Conditions

Let me be clear: I love the power.
Server Components are a huge win.
Nested layouts? Clean.
Improved routing flexibility? Great.
Performance optimizations baked in? Finally.

But…

It feels overengineered.
Every simple use case now requires reading three different RFCs.
Constantly wondering if you’re doing it “the Next Way™” or just making a hack that’ll break in the next release.

Should You Care?

If you’re building large, scalable apps or deeply care about performance, yes.
Next.js App Router is awesome when you’ve got the bandwidth to understand it.

If you just want to ship stuff, fast and simple…
Stick to pages/. Forget Server Components. Forget complicated layouts.
Use Next.js like the framework it used to be.

My Takeaway

Next.js is amazing at scale.
Next.js is terrible when you want simplicity.

It’s like loving a Swiss Army knife, but sometimes you just need a plain old screwdriver.
And now, the screwdriver has a dozen attachments you don’t care about.

Still, I’m in.
Because I know the payoff exists…
If I survive the learning curve.

Distributed Systems Without the Buzzwords

Sibasish Mohanty — Mon, 08 Sep 2025 05:50:24 +0000

In part 4 of our System Design series, we’re exploring the building blocks of distributed systems. These are the principles that separate toy apps from production-grade platforms.

We’ll cover:

CDN & Edge – serve static content near users
Sharding – hash, range, geo; handling hot keys
Replication – sync/async, leader/follower, read replicas
Consistency Models – strong, eventual, causal
CAP Theorem – in partition, pick A or C

1. CDN & Edge

TL;DR: Put content closer to users to cut latency.

CDN (Content Delivery Network): Caches static assets (images, CSS, JS) at edge locations worldwide.
Edge computing: Pushes logic (e.g., auth, personalization) closer to users.

👉 Example: Netflix streams video chunks from servers near your city, not across the ocean.

👉 Interview tie-in: “How would you reduce latency for global users?” — CDN is step 1.

2. Sharding

TL;DR: Split data across multiple nodes for scale.

Hash-based: Even distribution, but rebalancing is tricky.
Range-based: Easy for range queries, but hotspots form.
Geo-sharding: Route users to nearest region.

👉 Problem: Hot keys — a celebrity account with 10M followers can overload one shard.
👉 Solution: Key-splitting, consistent hashing, or virtual shards.

👉 Interview tie-in: “How do you handle scale in a key-value store?” — talk sharding.

3. Replication

TL;DR: Keep multiple copies of data for durability & scale.

Leader/follower (primary/replica): One leader handles writes, replicas handle reads.
Synchronous: Safer, but higher latency.
Asynchronous: Faster, but risk of data loss on failure.

👉 Example: MySQL read replicas to scale reads, leader for writes.

👉 Interview tie-in: “How do you scale database reads?” — use replicas.

4. Consistency Models

TL;DR: Tradeoff between user experience and performance.

Strong consistency: Every read sees the latest write.
Eventual consistency: Reads may lag but converge over time.
Causal consistency: Preserves cause-effect order.

👉 Example: Banking needs strong consistency. Social feeds tolerate eventual.

👉 Interview tie-in: Expect “Would you use strong or eventual consistency here?”

5. CAP Theorem

TL;DR: In a network partition, you must choose between Consistency (C) and Availability (A).

CP systems: Choose consistency, sacrifice availability (e.g., Zookeeper).
AP systems: Choose availability, tolerate stale reads (e.g., DynamoDB).

👉 Interview tie-in: They love asking “Which side of CAP would you pick?” — always tie your answer back to the product requirements.

✅ Takeaways

Use CDN/edge to cut latency for global users
Shard for scale, but handle hot keys carefully
Replicate for durability and read scaling
Know when to pick strong vs eventual consistency
CAP isn’t academic — it’s the heart of distributed tradeoffs

💡 Practice Question:

"Design a globally available chat app. How would you shard user messages, and what consistency model would you choose?"

My Recent Debugging Adventure: The Case of the Disappearing Stylesheets

Sibasish Mohanty — Thu, 04 Sep 2025 14:32:03 +0000

You know those days where a tiny bug eats half your brainpower, and then when you finally solve it, you feel like both an idiot and a wizard? Yeah. This is one of those stories.

The Setup: Performance Optimization Gone Wrong

Like any good frontend engineer, I wanted to optimize stylesheet loading. So instead of letting stylesheets block render, I tagged them like this:

<link rel="stylesheet" media="print" href="/assets/css/base.css" />

The trick here: browsers skip applying media="print" styles until you explicitly flip it to all. That way, the stylesheet downloads in the background, but doesn’t block rendering.

And then I had a little script to flip the switch:

document
  .querySelectorAll('link[rel="stylesheet"][media="print"]')
  .forEach(link => {
    const enableMedia = () => {
      link.media = 'all'
      link.removeEventListener('load', enableMedia)
    }
    if (link.sheet) {
      enableMedia()
    } else {
      link.addEventListener('load', enableMedia)
    }
  })

Beautiful, right? Except … nothing happened. Stylesheets never applied. My page looked naked. Not in a good way 😜.

The Bug: CSP Meets Timing

First instinct: CSP. I had a pretty strict Content Security Policy with script-src 'self' 'strict-dynamic'. Inline hacks were out, so the loader script was external.

But then I noticed something curious: my script was being deferred.

<script defer src="/static/js/stylesheetloader.js"></script>

defer means “wait until HTML parsing is done before running this script”. By then, the browser had already seen my <link media="print"> tags, queued them, and … didn’t trigger load the way I expected. My script showed up too late to catch the party.

The Fix: Async to the Rescue

The fix was almost laughably simple:

<script async src="/static/js/stylesheetloader.js"></script>

Switching defer to async made the script run as soon as it was available, before the render pipeline locked things down. Boom 💥. Stylesheets applied perfectly.

Sometimes all it takes is one attribute flip to go from “WTF why is CSS broken” to “I am a genius.”

Why This Works

Let’s zoom out:

defer scripts wait until after HTML parsing is complete, keeping order intact.
async scripts run as soon as they’re downloaded, ignoring order.

In my case, I didn’t care about execution order. I just needed the loader to run ASAP. Async was the perfect fit.

Lessons Learned

Performance hacks are sharp knives. Using media="print" for async stylesheets works, but you need to manage timing carefully.
CSP adds invisible complexity. With inline scripts off the table, your timing depends entirely on how you load external JS.
defer vs async actually matters. Don’t just cargo-cult defer everywhere. Think about when you need the script to run.

Generalizing the Problem

This isn’t just about my stylesheet loader. It’s a broader lesson about script execution timing under CSP:

If you have critical scripts that need to act before rendering decisions (like stylesheet toggles, dynamic meta updates, etc.), use async or preload strategies.
If execution order matters (like multiple polyfills), then defer is safer.
If you’re battling CSP, externalize your hacks, don’t rely on inline event handlers (onload="...").

Alternatives Worth Considering

<link rel="preload">: preload stylesheets and apply later.
Modern bundlers (Rsbuild, Vite): can inline critical CSS and lazy-load the rest.
JS-based style injection: less common, but some teams prefer runtime style injection for full control.

Wrap-up

Sometimes, a bug isn’t about broken code. It’s about tiny mismatches in timing and browser behavior. This one was a perfect storm of CSP, defer, and stylesheet loading tricks.

I wasted hours chasing ghosts, but in the end, flipping one attribute fixed everything. The kind of debugging adventure that keeps you humble … and slightly amused.

So next time your CSS looks naked, check your <script> attributes. 😉

Cache Me If You Can: Design Patterns for Performance

Sibasish Mohanty — Wed, 03 Sep 2025 04:00:23 +0000

In part 3 of our System Design series, we’re tackling caching and load balancing — the unsung heroes of performance. Without them, systems crumble under scale.

We’ll cover:

Caching – App/DB/CDN; write-through/write-back, TTLs
Cache Invalidation – TTLs, versioning, stampede protection
Load Balancing – L4/L7, round-robin, least-connections, hashing

1. Caching

TL;DR: Caching is your first lever for scale. Use it everywhere, but know the trade-offs.

App cache: In-memory (Redis, Memcached). Ultra-fast but volatile.
DB cache: Query or object cache to offload hot queries.
CDN cache: Push static assets near users.

Strategies:

Write-through: Write to cache + DB simultaneously (safe, consistent, slower writes)
Write-back: Write to cache first, sync to DB later (fast, risky if cache crashes)
TTL (Time To Live): Expire stale data automatically

👉 Example: A news homepage caches top stories for 30s — thousands of requests saved.

👉 Interview tie-in: “How would you scale a read-heavy service?” — caching is the first answer.

2. Cache Invalidation

TL;DR: The hardest part of caching isn’t caching — it’s invalidation.

TTL: Safe default, but may serve stale data.
Versioning: Change cache key when data updates (e.g., user:v2:123)
Stampede protection: Use locking or request coalescing so multiple clients don’t hammer the DB when cache expires.

👉 Example: If 1M users refresh when a cache expires, that’s a cache stampede. Use jittered TTLs or async refresh.

👉 Interview tie-in: They’ll ask “What’s the hardest part about caching?” — answer: invalidation and consistency.

3. Load Balancing

TL;DR: Load balancers spread requests across servers and hide failures.

L4 (Transport): Balances based on IP/port. Simple, fast.
L7 (Application): Smarter — routes based on headers, cookies, paths.

Algorithms:

Round Robin: Even distribution
Least Connections: Send to the server with fewest active requests
Hashing: Sticky sessions (e.g., same user → same server)

👉 Example: E-commerce app uses L7 LB to route /images → CDN, /checkout → payment cluster.

👉 Interview tie-in: “How do you handle uneven traffic across servers?” — least-connections or weighted load balancing.

✅ Takeaways

Cache where it hurts most: hot queries, static assets, read-heavy endpoints
Invalidation is the real challenge; plan strategies upfront
Load balancing is critical for fairness, resilience, and routing logic

💡 Practice Question:

"Design the caching strategy for a Twitter timeline. How would you avoid cache stampede during trending events?"

Data Design That Doesn't Crumble at Scale

Sibasish Mohanty — Thu, 28 Aug 2025 05:34:24 +0000

The second part of our System Design series dives into data fundamentals. Most systems fail not because of code, but because of bad data modeling decisions. Let’s break down four pillars:

SQL vs NoSQL
Data Modeling
Indexing
Normalization

1. SQL vs NoSQL

TL;DR: Pick based on access patterns, not hype.

SQL (Relational): Strong consistency, structured schema, powerful joins. Great for transactions (e.g., payments).
NoSQL: Flexible schema, horizontal scale, eventual consistency. Great for unstructured or high-write workloads (e.g., social feeds).

👉 Rule of thumb: If you need joins and ACID, use SQL. If you need scale and flexibility, lean toward NoSQL.

👉 Interview tie-in: “Design Twitter feed storage.” Likely NoSQL, with denormalized fan-out.

2. Data Modeling

TL;DR: Always model data around your hot queries.

Think backwards: What queries will dominate traffic?
Denormalize or restructure tables/collections for those.
Optimize for reads if the system is read-heavy.

👉 Example: In an e-commerce app, Orders table needs fast lookup by userId. That index matters more than less-frequent joins.

👉 Interview tie-in: They want to see that you don’t design schemas in a vacuum — you design for usage.

3. Indexing

TL;DR: The right index makes queries fly; the wrong one makes writes crawl.

Primary index: Natural row identifier (e.g., id)
Secondary indexes: Speed up queries but slow down writes
Composite indexes: Useful when filtering on multiple fields

👉 Example: WHERE userId = 123 AND status = 'ACTIVE' → composite index on (userId, status)

👉 Interview tie-in: Expect questions like “Why is this query slow despite indexing?” — hint: wrong index or low selectivity.

4. Normalization

TL;DR: Normalize for writes, denormalize for reads.

Normalization: Reduce redundancy, avoid anomalies (great for OLTP)
Denormalization: Duplicate data to speed up queries (great for OLAP)

👉 Example: User profile data normalized in Users table. For faster feed rendering, duplicate username and avatar in Posts.

👉 Interview tie-in: “Would you normalize user data in a news feed?” — no, you’d denormalize for speed.

✅ Takeaways

Choose SQL vs NoSQL based on workload, not dogma
Model around hot queries, not theoretical schemas
Use indexes wisely — they’re a double-edged sword
Normalize for writes, denormalize for reads

💡 Practice Question:

"Design the database schema for Instagram posts, likes, and comments. Which parts would you normalize, and which would you denormalize?"

Scaling Isn’t Magic: Fundamentals They Expect You to Know

Sibasish Mohanty — Tue, 26 Aug 2025 15:51:46 +0000

System Design interviews can feel overwhelming, but they always boil down to a few first principles. In this post, we’ll cover four foundational topics:

Scalability – Vertical vs Horizontal
Latency & Throughput – P50, P95, P99
Capacity Estimation – QPS, storage, bandwidth
Networking Basics – TCP, HTTP/HTTPS, TLS, UDP, DNS

1. Scalability: Vertical vs Horizontal

TL;DR: Scale vertically until you hit diminishing returns, then go horizontal. Default to stateless services for easy horizontal scaling.

Vertical scaling: Bigger machine (more CPU, RAM, disk). Easy but capped. Expensive and risk of single point of failure.
Horizontal scaling: More machines (stateless servers behind a load balancer). Harder to manage but scales better.

👉 Interview tie-in: They’ll ask “How would you scale this system to handle 10x users?” Answer with horizontal scaling + stateless services.

2. Latency & Throughput

TL;DR: Latency = how long one request takes. Throughput = how many requests per unit time.

Latency percentiles:
- P50 (median): What most users see
- P95: The "tail" – 5% of users worse than this
- P99: Worst case for 1% of users, often drives design
Tradeoff: Optimizing tail latency may hurt throughput, and vice versa.

👉 Analogy: Coffee shop.

Latency = how long it takes to serve one customer.
Throughput = how many coffees per minute.

👉 Interview tie-in: "We want 95% of requests under 200ms." That’s P95 latency target.

3. Capacity Estimation

TL;DR: Always estimate load before designing. Even rough math is better than guessing.

QPS (Queries Per Second): How many requests per second your service must handle.
Storage: How much data accumulates per day/month/year.
Bandwidth: How much data moves across the network.

👉 Example:

1M daily active users
Each makes 10 requests/day → ~115 QPS
Each request ~1KB → ~10MB/day traffic

👉 Interview tie-in: They’ll push you to "estimate scale" before proposing an architecture. Show your math, even if it’s rough.

4. Networking Basics

TL;DR: Most systems fail because people ignore networking bottlenecks.

TCP: Reliable, ordered, connection-based
UDP: Fast, connectionless, may drop packets (used for video/games)
HTTP/HTTPS: Request-response on top of TCP; HTTPS adds TLS encryption
DNS: Resolves google.com → IP address
TLS handshake: Extra latency on first connection

👉 Interview tie-in: Expect questions like "Why use UDP for video streaming instead of TCP?"

✅ Takeaways

Prefer stateless horizontal scaling
Measure both latency (P95) and throughput
Always estimate capacity before proposing a design
Know your networking trade-offs

💡 Practice Question:

"Design a URL shortener for 50M users. How many requests/sec do you expect? What’s your scaling plan?"

Stop Fetching Data in useEffect: Redux Edition

Sibasish Mohanty — Tue, 29 Jul 2025 06:45:20 +0000

In Part 1, we talked about why fetching data directly inside useEffect is a flawed approach — tightly coupling component lifecycle to data-fetching logic, leading to unmanageable side effects, broken UX, and stale state bugs. We compared it to jQuery’s imperative data flows and showed how React in 2025 demands a more declarative mental model.

But the story doesn’t end there.

In this second part, we tackle an even more common anti-pattern, especially in Redux-heavy codebases:
Dispatching async thunks from useEffect.

useEffect(() => {
  dispatch(fetchUserData(userId))
}, [userId])

It might look cleaner than raw fetch(), but it inherits the same rot underneath.

Why This Pattern Still Exists

Before React Hooks, data fetching in Redux apps happened in componentDidMount(), with connected components via connect() HOCs. That lifecycle-to-dispatch model was normative.

When hooks arrived with useEffect, developers naturally ported this over. This became the de facto pattern. Even the official React-Redux docs show this for simple use cases.

It was a one-to-one migration. And at the time, it made sense.

You needed to kick off a thunk or saga somewhere, and useEffect was the hammer for all side-effect nails.

But Redux evolved.

As Mark Erikson (Redux maintainer) has repeatedly emphasized:

"You probably don’t need useEffect. And if you're dispatching from it, it's likely a code smell."

Yet, many codebases (even recent ones) still cling to the old dispatch-in-useEffect dance. Let’s unpack why that’s a problem.

The Core Problem

This pattern:

Delays data availability until after first render
Forces you to manage loading and error state manually
Ties async logic to the lifecycle, not the state graph
Doesn’t support SSR or Suspense
Is verbose, hard to test, and easy to misuse
Encourages repetition (every page/component reimplements fetch logic)

Worse, this makes your components non-composable. You can’t reuse this logic in another component without copy-pasting or lifting.

You're not modeling data — you're modeling timing.

But Isn’t Redux Still Useful?

It is — but not for remote server data.

Redux is great for:

Global, non-remote state (auth, UI toggles, preferences)
Workflow state (multi-step forms, undo/redo, complex user flows)
Cross-cutting logic (feature flags, permission checks)

But for:

Fetching lists
User profiles
Paginated feeds
Notifications
Dashboard metrics

...classic Redux is too much ceremony.

You don’t need reducers, actions, thunks, and selectors just to load some damn data.

You need a query layer — not a state machine factory.

Is dispatch() in useEffect() Always Wrong?

Not strictly.

Sometimes you genuinely need to dispatch on mount, maybe to initialize a websocket, fire a metrics event, or pre-warm a cache.

But when you're fetching render-critical data, dispatching inside useEffect means you're reacting too late.

You’ve already rendered once.
You’ve already missed the chance to prefetch or SSR.
You’ve already broken the declarative contract.

If You Have to Use Classic Redux

Encapsulate the dispatch logic in a custom hook to decouple data orchestration from your component:

function useUserData(userId) {
  const dispatch = useDispatch()
  const data = useSelector(state => state.users[userId])
  const loading = useSelector(state => state.loading.users[userId])

  useEffect(() => {
    dispatch(fetchUserData(userId))
  }, [userId])

  return { data, loading }
}

Then use declaratively:

const { data, loading } = useUserData(userId)

This way:

Your component stays declarative
The effect logic is testable and reusable
You reduce boilerplate inside the view layer

Still far from ideal — but survivable.

So What Should You Do Instead?

Data fetching deserves better than a useEffect hack job. If your mental model is still stuck in 2019, it's time for a reset.

In Part 3, we’ll explore a *declarative routing-first approach * to data fetching using loader, useLoaderData, and compare that with popular solutions like RTK Query, TanStack Query, swr and even emerging Suspense APIs. We'll also address how to gracefully migrate away from this legacy pattern in large codebases.

And yes — there’s a way to do all this without ditching Redux if you don’t want to.

TL;DR

useEffect is not a data-fetching tool
Dispatching async thunks in useEffect is an outdated, fragile pattern
This made sense in pre-RTK Redux, but should be retired
Redux already solved this. It’s called RTK Query.
RTK Query, TanStack Query, and Loader APIs offer better abstractions
Declarative data > lifecycle-driven orchestration

If you’re still writing async thunks just to get a list of users, it’s time to ask yourself why.

You’re still not writing jQuery.
And now — you shouldn't be writing it with Redux either.