Forem: CodeWithDhanian

What is Horizontal vs Vertical Scaling?

CodeWithDhanian — Fri, 08 May 2026 08:03:13 +0000

Scaling is the fundamental process of increasing a system’s capacity to handle greater workloads, more users, or higher traffic without compromising performance. In system design, two primary strategies exist: vertical scaling and horizontal scaling. Each approach addresses growth differently, carries unique architectural implications, and demands distinct engineering considerations. Understanding both is essential for building systems that remain reliable, cost-effective, and performant as demand evolves.

Understanding Vertical Scaling

Vertical scaling, also known as scaling up, involves enhancing the capabilities of a single server or instance by adding more resources to it. This typically means increasing CPU cores, RAM, storage capacity, or network bandwidth on the existing machine.

The process is straightforward. Consider a web server running on a machine with 4 CPU cores and 8 GB of RAM. When traffic grows, the operations team upgrades that same machine to 16 CPU cores and 64 GB of RAM. No additional servers are introduced; the workload continues to run on the upgraded hardware.

Vertical scaling shines in scenarios where the application is monolithic or tightly coupled to a single process. Databases often benefit from this approach during early growth phases because a larger instance can process more queries per second without requiring data partitioning logic.

Advantages of vertical scaling include simplicity of implementation, lower operational overhead, and minimal changes to application code. Latency between components remains low since everything runs within one machine. Management is easier because there is only a single instance to monitor, backup, and secure.

However, vertical scaling has hard physical limits. Hardware vendors offer only finite maximum configurations for any server type. Beyond a certain point, upgrading becomes prohibitively expensive or technically impossible. A single point of failure exists: if that upgraded machine crashes, the entire system goes down. Upgrades frequently require downtime while the instance is stopped, resized, and restarted. In cloud environments, this translates to higher costs for larger instance types that may be over-provisioned during low-traffic periods.

Understanding Horizontal Scaling

Horizontal scaling, also known as scaling out, involves adding more servers or instances to distribute the workload across multiple machines. Instead of making one server more powerful, the system grows by increasing the number of identical servers working together.

A load balancer sits in front of the fleet of servers and routes incoming requests intelligently across them. As demand increases, new instances are spun up automatically or manually, and traffic is spread evenly. This approach aligns naturally with cloud-native architectures and microservices.

Horizontal scaling provides virtually unlimited growth potential because additional machines can be added indefinitely. It delivers built-in fault tolerance: if one server fails, the remaining servers continue serving traffic. Cost efficiency improves because smaller, commodity instances are cheaper than a single massive machine. Upgrades can occur without downtime by adding new instances before removing old ones.

The trade-offs are significant. Horizontal scaling introduces complexity in areas such as data synchronization, session management, and inter-service communication. Network latency between machines becomes a factor. Applications must be designed to be stateless or use external shared stores for state. Distributed system challenges like consistency, leader election, and failure detection emerge. Debugging across multiple nodes is more difficult than on a single machine.

Comparing Vertical and Horizontal Scaling in Practice

The choice between vertical scaling and horizontal scaling depends on the application’s architecture, expected growth curve, team expertise, and budget.

Vertical scaling suits early-stage startups, legacy monolithic applications, or workloads with heavy in-memory computations where splitting data is impractical. Horizontal scaling becomes necessary when traffic exceeds what any single machine can handle or when high availability is non-negotiable.

Real-world systems frequently combine both strategies. A database might use vertical scaling for its primary instance while employing horizontal scaling through read replicas or sharded clusters for read-heavy workloads.

Designing Applications for Horizontal Scaling: A Practical Example

To succeed with horizontal scaling, applications must be stateless whenever possible. The following complete code example illustrates the difference.

Stateful example (problematic for horizontal scaling)

from flask import Flask

app = Flask(__name__)
counter = 0  # Global variable stored in memory

@app.route('/increment')
def increment():
    global counter
    counter += 1
    return f"Counter: {counter}"

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

If this service runs on multiple instances behind a load balancer, each instance maintains its own counter. Users hitting different instances receive inconsistent values. This breaks correctness.

Stateless example (ready for horizontal scaling)

from flask import Flask
import redis

app = Flask(__name__)
redis_client = redis.Redis(host='redis-shared-store', port=6379, db=0)

@app.route('/increment')
def increment():
    counter = redis_client.incr('global_counter')
    return f"Counter: {counter}"

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

All instances share the same Redis store. The counter remains consistent regardless of which instance processes the request. This design scales horizontally without modification.

Implementing Horizontal Scaling with Nginx Load Balancer

A complete Nginx configuration demonstrates how to distribute traffic across multiple application instances.

# /etc/nginx/nginx.conf
user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    upstream backend {
        server app-instance-1:5000;
        server app-instance-2:5000;
        server app-instance-3:5000;
        # Add more servers here as you scale out
        least_conn;  # Distribute to the least busy server
    }

    server {
        listen 80;
        server_name example.com;

        location / {
            proxy_pass http://backend;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            # Health checks for automatic removal of unhealthy instances
            proxy_next_upstream error timeout http_500 http_502 http_503 http_504;
        }
    }
}

This configuration defines an upstream block listing all application instances. The least_conn directive ensures intelligent load distribution. Proxy headers preserve original client information. As demand grows, simply add more server lines or use orchestration tools to dynamically update the upstream list. Reload Nginx with nginx -s reload to apply changes without downtime.

When to Choose Each Strategy

Start with vertical scaling when the system is small, the team is focused on rapid feature delivery, and the application does not yet require distributed coordination. Transition to horizontal scaling when traffic patterns show sustained growth, when downtime during upgrades becomes unacceptable, or when cloud costs for larger instances exceed the expense of multiple smaller ones.

Horizontal scaling is the foundation of modern resilient systems. It forces thoughtful design decisions that pay dividends in reliability and flexibility far beyond raw capacity.

If you found this deep dive into horizontal versus vertical scaling valuable and want the complete professional treatment of all 100 system design concepts with diagrams, real-world architectures, and production-ready patterns, grab the full system design ebook at https://codewithdhanian.gumroad.com/l/urcjee. If this content helped you, consider buying me a coffee at https://ko-fi.com/codewithdhanian to support more free in-depth resources like this.

Visualizing the Concept

What is Load Balancing?

CodeWithDhanian — Fri, 08 May 2026 07:43:53 +0000

Load balancing is the fundamental technique used in modern distributed systems to distribute incoming network traffic across multiple backend servers or resources in order to ensure no single server becomes overwhelmed, thereby improving responsiveness, availability, and scalability. At its core, a load balancer acts as a traffic cop that sits between clients and the actual application servers, intelligently routing each request to the most appropriate server based on predefined rules and real-time conditions.

Why Load Balancing Is Essential in System Design

In any production-grade application that serves millions of users, relying on a single server is impractical and risky. A sudden spike in traffic, such as during a flash sale or viral event, can cause that server to slow down, crash, or become unresponsive. Load balancing solves this by enabling horizontal scaling — the ability to add more servers dynamically — while maintaining a seamless user experience. It also provides fault tolerance: if one server fails, the load balancer automatically stops sending traffic to it and redirects requests to healthy servers. This ensures the system remains highly available even during hardware failures, maintenance windows, or unexpected load surges.

Core Components of a Load Balancer

A typical load balancer consists of the following essential elements:

Frontend Listener: The entry point that accepts incoming client requests on specific ports (usually 80 for HTTP or 443 for HTTPS).
Backend Pool: A group of healthy application servers (often called targets or origins) that actually process the requests.
Health Check Mechanism: Continuous monitoring that probes each backend server to verify it is responding correctly. A failed health check removes the server from the active pool until it recovers.
Routing Engine: The brain that applies load balancing algorithms and rules to decide which server receives each request.
Session Persistence Layer (optional): Ensures that a user’s subsequent requests are routed to the same server when necessary (also known as sticky sessions).

How Load Balancing Works Step by Step

A client (browser, mobile app, or another service) sends a request to the public IP or domain of the load balancer.
The load balancer inspects the request headers, source IP, or other metadata.
Using its configured algorithm and current server metrics (CPU load, active connections, response time), the load balancer selects the optimal backend server.
The request is forwarded (proxied) to the chosen server.
The backend server processes the request and sends the response back through the load balancer to the client.
The load balancer may also perform TLS termination, compression, or request rewriting before forwarding.

Types of Load Balancers

Load balancers are broadly classified into two layers of the OSI model:

Layer 4 (Transport Layer) Load Balancers: Operate at the TCP/UDP level. They forward packets based on IP address and port without inspecting the actual content of the request. Examples include AWS Network Load Balancer and HAProxy in TCP mode. They are extremely fast and suitable for high-throughput scenarios but cannot make routing decisions based on HTTP headers or URL paths.
Layer 7 (Application Layer) Load Balancers: Operate at the HTTP/HTTPS level. They can read the full request, including URL, headers, cookies, and method. This allows advanced routing such as sending image requests to one pool and API requests to another. Examples include AWS Application Load Balancer, NGINX, and Envoy. They support content-based routing, rate limiting, and header manipulation but introduce slightly higher latency due to inspection.

Load balancers can also be deployed as:

Hardware appliances (F5 BIG-IP, Citrix ADC) — expensive but offer high performance and specialized ASIC chips.
Software solutions (NGINX, HAProxy, Traefik) — run on commodity servers or containers.
Cloud-managed services (AWS ELB, Google Cloud Load Balancing, Azure Load Balancer) — fully managed with auto-scaling built in.

Popular Load Balancing Algorithms

The choice of algorithm directly impacts system performance. Here are the most widely used ones with detailed explanations:

Round Robin: Requests are distributed sequentially across the backend servers in a cyclic order. Simple and fair when all servers have identical capacity.
Weighted Round Robin: Each server is assigned a weight based on its capacity. A more powerful server receives proportionally more requests.
Least Connections: The load balancer routes the next request to the server currently handling the fewest active connections. Excellent for uneven workloads.
Least Response Time: Routes to the server with the lowest average response time, combining connection count and latency.
IP Hash: Uses the client’s IP address to consistently route requests to the same server. Useful for session persistence without cookies.
Random: Selects a server at random. Surprisingly effective and simple to implement.

Complete NGINX Configuration Example

Below is a production-ready NGINX configuration that demonstrates load balancing with health checks, weighted round robin, and session persistence. Every line is explained in detail.

# Global settings
worker_processes auto;
events {
    worker_connections 1024;
}

http {
    # Define the upstream (backend pool)
    upstream backend_servers {
        # Least Connections algorithm with weights
        least_conn;

        server app-server-1.example.com:8080 weight=3 max_fails=3 fail_timeout=30s;
        server app-server-2.example.com:8080 weight=2 max_fails=3 fail_timeout=30s;
        server app-server-3.example.com:8080 weight=1 max_fails=3 fail_timeout=30s;

        # Health check (requires nginx-plus or open-source module)
        # In open-source NGINX, use external tools like consul-template
        keepalive 32;
    }

    server {
        listen 80;
        listen 443 ssl;
        server_name myapp.com;

        # SSL termination happens here
        ssl_certificate /etc/nginx/ssl/fullchain.pem;
        ssl_certificate_key /etc/nginx/ssl/privkey.pem;

        location / {
            # Forward request to the upstream pool
            proxy_pass http://backend_servers;

            # Preserve original host and client IP
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            # Enable session persistence using cookies
            proxy_cookie_path / "/; secure; HttpOnly";

            # Timeout settings for reliability
            proxy_connect_timeout 5s;
            proxy_send_timeout 10s;
            proxy_read_timeout 10s;
        }
    }
}

Explanation of key directives:

upstream backend_servers defines the pool of servers.
least_conn activates the Least Connections algorithm.
weight=3 gives app-server-1 three times more traffic than app-server-3.
max_fails=3 fail_timeout=30s removes a server after three consecutive failures for 30 seconds.
proxy_pass http://backend_servers forwards traffic to the chosen server.
Header directives ensure the backend knows the original client information.

Advanced Load Balancing Concepts

Consistent Hashing is often combined with load balancing to minimize disruption when servers are added or removed. Instead of rehashing everything, only a small portion of traffic is affected.

Global Server Load Balancing (GSLB) extends the concept across multiple data centers using DNS-based routing (Anycast or GeoDNS) to direct users to the nearest healthy region.

Auto-scaling integration allows the load balancer to dynamically register new instances launched by Kubernetes Horizontal Pod Autoscaler or AWS Auto Scaling Groups.

If you found this deep dive into load balancing valuable and want to master the remaining 99 system design concepts with equally detailed explanations, code examples, and diagrams, grab the complete System Design eBook at https://codewithdhanian.gumroad.com/l/urcjee. You can also support the creation of more high-quality technical content by buying me a coffee at https://ko-fi.com/codewithdhanian.

Retry & Exponential Backoff in System Design

CodeWithDhanian — Sun, 05 Apr 2026 10:17:33 +0000

In distributed systems and microservices architectures, transient failures are common. Network glitches, temporary service overloads, brief database contention, or momentary unavailability of third-party APIs frequently resolve themselves within seconds. The Retry mechanism combined with Exponential Backoff provides a fundamental resilience strategy that intelligently re-attempts failed operations instead of failing immediately. This pattern significantly improves overall system reliability and user experience by handling flaky conditions gracefully without overwhelming the failing service.

Retry & Exponential Backoff forms one of the core building blocks of fault-tolerant design, often used alongside the Circuit Breaker Pattern, timeouts, idempotency, and bulkhead isolation. When implemented correctly, it reduces unnecessary errors while protecting downstream services from retry storms that could lead to cascading failures.

Understanding Retry Mechanisms

A retry is simply the act of re-executing a failed operation after a short delay. Not every failure deserves a retry. Only idempotent operations or those that are safe to repeat should be retried. Non-idempotent operations require careful handling, often through idempotency keys or unique transaction identifiers to prevent duplicate effects.

Common transient failure scenarios suitable for retries include:

Network timeouts or connection resets
HTTP 503 Service Unavailable or 429 Too Many Requests
Temporary database deadlocks or lock contention
Rate limiting responses from external services
Brief unavailability during scaling events or deployments

Permanent failures such as validation errors (HTTP 400), authentication failures (401/403), or business logic errors should not trigger retries.

Exponential Backoff Strategy

Simple fixed-delay retries can create thundering herd problems where many clients retry simultaneously, overwhelming the recovering service. Exponential Backoff solves this by increasing the wait time between retries exponentially. The delay typically follows the formula:

delay = base_delay × 2^retry_attempt

To prevent synchronization of retries across clients, jitter (random variation) is added to the calculated delay.

Full delay formula with jitter:
delay = min(cap, base_delay × 2^retry_attempt) + random(0, jitter)

Common variations include:

Full Jitter: Random delay between 0 and the computed exponential value
Equal Jitter: Computed delay minus a random portion
Decorrelated Jitter: Next delay based on previous delay with randomness

Exponential Backoff with Jitter dramatically improves system stability under load by spreading retry attempts over time.

Detailed Implementation of Retry with Exponential Backoff

Production-grade implementations must handle concurrency safely, respect maximum retry limits, support different backoff strategies, and integrate with logging and monitoring.

Pseudocode for Retry with Exponential Backoff

class RetryWithBackoff {
    int maxAttempts;
    long baseDelayMs;
    long maxDelayMs;
    double jitterFactor;

    Object executeWithRetry(Callable operation) {
        Exception lastException;

        for (int attempt = 0; attempt < maxAttempts; attempt++) {
            try {
                return operation.call();
            } catch (TransientException e) {
                lastException = e;
                if (attempt == maxAttempts - 1) {
                    break;  // Final attempt failed
                }
                long delay = calculateDelay(attempt);
                sleep(delay);
            } catch (PermanentException e) {
                throw e;  // Do not retry
            }
        }
        throw lastException;  // Propagate after exhausting retries
    }

    private long calculateDelay(int attempt) {
        long exponentialDelay = baseDelayMs * (1L << attempt);  // 2^attempt
        long cappedDelay = min(exponentialDelay, maxDelayMs);

        // Add full jitter
        long jitter = random(0, (long)(cappedDelay * jitterFactor));
        return cappedDelay + jitter;
    }
}

Complete Python Implementation

import time
import random
from typing import Callable, Any, Type

class TransientError(Exception):
    pass

def retry_with_exponential_backoff(
    max_attempts: int = 5,
    base_delay: float = 0.1,      # 100ms
    max_delay: float = 10.0,      # 10 seconds
    jitter: bool = True,
    backoff_factor: float = 2.0
):
    def decorator(func: Callable) -> Callable:
        def wrapper(*args, **kwargs) -> Any:
            last_exception = None

            for attempt in range(max_attempts):
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    last_exception = e

                    # Check if error is transient (custom logic)
                    if not is_transient_error(e):
                        raise  # Permanent error - do not retry

                    if attempt == max_attempts - 1:
                        break  # Last attempt failed

                    # Calculate exponential backoff
                    delay = base_delay * (backoff_factor ** attempt)
                    delay = min(delay, max_delay)

                    if jitter:
                        delay += random.uniform(0, delay * 0.1)  # 10% jitter

                    time.sleep(delay)

                    # Optional: log retry attempt
                    # logger.warning(f"Retry {attempt+1}/{max_attempts} after {delay:.2f}s")

            raise last_exception  # Re-raise after all retries exhausted

        return wrapper
    return decorator

# Example usage
@retry_with_exponential_backoff(max_attempts=4, base_delay=0.2, max_delay=5.0)
def call_external_api(user_id: str):
    # Simulate network call that may fail transiently
    response = requests.get(f"https://api.example.com/users/{user_id}")
    response.raise_for_status()
    return response.json()

Java Conceptual Structure (Resilience4j Style)

RetryConfig config = RetryConfig.custom()
    .maxAttempts(5)
    .waitDuration(Duration.ofMillis(100))
    .retryOnException(e -> e instanceof TransientException)
    .intervalFunction(IntervalFunction.ofExponentialBackoff(100, 2.0))
    .build();

Retry retry = Retry.of("externalService", config);

Callable<String> retryableCall = Retry.decorateCallable(retry, () -> callExternalService());

String result = Try.ofCallable(retryableCall)
    .recover(this::fallbackResponse)
    .get();

These implementations demonstrate key elements: configurable attempt limits, proper classification of transient versus permanent errors, exponential delay calculation, jitter for load distribution, and clean separation of concerns.

Best Practices for Retry & Exponential Backoff

Effective use of this pattern requires attention to several critical details:

Idempotency: Always ensure retried operations are idempotent or use idempotency keys (unique request identifiers stored server-side) to prevent duplicate side effects.
Timeout Integration: Combine retries with appropriate per-attempt timeouts to avoid hanging requests.
Circuit Breaker Synergy: Use circuit breakers to stop retries entirely when a service is confirmed unhealthy.
Monitoring & Observability: Track retry counts, success-after-retry rates, and backoff delays using tools like Prometheus and Grafana.
Maximum Delay Caps: Prevent excessively long waits by capping delays.
Client-Specific Backoff: Different clients or services may need tailored backoff parameters based on their importance and load characteristics.
Avoid Retry Storms: Jitter and randomized delays are essential in large-scale systems with thousands of instances.

In event-driven architectures using message queues like Kafka or RabbitMQ, retries are often handled through dead-letter queues and delayed message redelivery rather than in-process loops.

Real-World Considerations

In high-scale systems, Retry & Exponential Backoff must be applied judiciously. Overly aggressive retries can still contribute to overload. Many modern service meshes (such as Istio) and API gateways provide built-in retry capabilities at the infrastructure layer, allowing application code to focus on business logic.

The combination of Retry with Exponential Backoff remains one of the simplest yet most powerful techniques for improving resilience in distributed systems. When paired with proper idempotency, timeouts, and circuit breakers, it enables applications to withstand transient issues while maintaining high availability and responsive user experiences.

System Design Handbook

For more in-depth insights and comprehensive coverage of system design topics, consider purchasing the System Design Handbook at https://codewithdhanian.gumroad.com/l/ntmcf. It will equip you with the knowledge to master complex distributed systems.

Buy me coffee to support my content at: https://ko-fi.com/codewithdhanian

Circuit Breaker Pattern in System Design

CodeWithDhanian — Fri, 03 Apr 2026 08:07:13 +0000

In distributed systems and microservices architectures, failures are inevitable. Network latency, service overload, database slowdowns, or third-party API outages can quickly cascade into widespread system instability. The Circuit Breaker Pattern serves as a critical resilience mechanism that prevents these cascading failures by intelligently isolating faulty components. Inspired by electrical circuit breakers that interrupt current flow during overloads, the software version acts as a protective proxy around remote calls, allowing systems to fail fast, degrade gracefully, and recover automatically.

This pattern is essential for building robust, highly available applications where services depend on each other across network boundaries. By monitoring failure rates and response times, the circuit breaker stops repeated attempts to reach an unhealthy service, giving it time to recover while providing immediate feedback or fallback responses to callers.

Understanding the Circuit Breaker Pattern

The Circuit Breaker Pattern functions as a stateful wrapper around operations that interact with external services or resources. Instead of allowing every request to reach a failing downstream service—which could overwhelm it further and degrade the entire system—the circuit breaker tracks metrics such as error counts, latency, or exceptions. When failure thresholds are breached, it “trips” and redirects traffic away from the problematic service.

Key benefits include:

Prevention of cascading failures
Reduction in resource consumption on both caller and callee sides
Faster response times through immediate failure detection
Graceful degradation via fallback mechanisms
Automatic recovery without manual intervention

The pattern works best when combined with complementary techniques such as retries with exponential backoff, timeouts, rate limiting, and bulkhead isolation.

The Three States of a Circuit Breaker

A circuit breaker maintains one of three distinct states, each dictating how incoming requests are handled. These states form a finite state machine that transitions based on observed behavior and configurable thresholds.

Closed State:

This is the normal operating state. All requests pass through to the protected service. The circuit breaker monitors outcomes, counting failures within a sliding time window or consecutive failure count. If the failure rate or count exceeds a predefined threshold (for example, 50% errors in the last 10 seconds or 5 consecutive failures), the breaker transitions to the Open state. Successes reset or decrement failure counters.

Open State:

When the circuit is open, the breaker immediately rejects all requests without forwarding them to the downstream service. This prevents further load on the failing component and avoids long timeouts or resource exhaustion. Instead, the caller receives an immediate exception or a fallback response. A timeout timer (reset timeout) starts, after which the breaker moves to the Half-Open state to test recovery.

Half-Open State:

This transitional state allows a limited number of test requests (often just one or a small configurable count) to reach the service. If these probe requests succeed, the circuit breaker assumes recovery and returns to the Closed state, resetting failure counters. If any test fails, the breaker reverts to the Open state and restarts the timeout period. This cautious probing ensures the service has truly stabilized before resuming full traffic.

These state transitions enable self-healing while protecting system stability.

Detailed Implementation of the Circuit Breaker Pattern

Implementing a circuit breaker from scratch requires careful handling of concurrency, metrics tracking, and state persistence. In production, developers typically use battle-tested libraries such as Resilience4j (Java), Hystrix (legacy Java), Polly (.NET), or pybreaker (Python). Below are complete, illustrative code structures.

Pseudocode for a Generic Circuit Breaker

class CircuitBreaker {
    enum State { CLOSED, OPEN, HALF_OPEN }

    State currentState = CLOSED;
    int failureCount = 0;
    int successCount = 0;
    long lastFailureTime = 0;
    Configuration config;  // failureThreshold, timeout, successThreshold, etc.

    Object execute(Callable operation) {
        if (currentState == OPEN) {
            if (isTimeoutExpired()) {
                transitionTo(HALF_OPEN);
            } else {
                return invokeFallback();  // or throw CircuitOpenException
            }
        }

        try {
            Object result = operation.call();
            onSuccess();
            return result;
        } catch (Exception e) {
            onFailure(e);
            return invokeFallback();
        }
    }

    private void onSuccess() {
        failureCount = 0;
        successCount++;
        if (currentState == HALF_OPEN && successCount >= config.successThreshold) {
            transitionTo(CLOSED);
        }
    }

    private void onFailure(Exception e) {
        failureCount++;
        lastFailureTime = currentTime();
        if (failureCount >= config.failureThreshold || currentState == HALF_OPEN) {
            transitionTo(OPEN);
        }
    }

    private boolean isTimeoutExpired() {
        return (currentTime() - lastFailureTime) > config.resetTimeout;
    }

    private void transitionTo(State newState) {
        currentState = newState;
        // Log state change, notify monitoring system
        if (newState == HALF_OPEN) {
            successCount = 0;
        }
    }

    private Object invokeFallback() {
        // Execute fallback logic, e.g., return cached data or default value
        return defaultResponse();
    }
}

Java Example Using Resilience4j Style (Conceptual Full Structure)

// Configuration
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    .failureRateThreshold(50)           // Open if failure rate > 50%
    .waitDurationInOpenState(Duration.ofSeconds(30))  // Reset timeout
    .permittedNumberOfCallsInHalfOpenState(3)         // Test calls
    .slidingWindowSize(10)              // Window for metrics
    .build();

CircuitBreaker circuitBreaker = CircuitBreaker.of("paymentService", config);

// Decorator usage
Supplier<String> decoratedSupplier = CircuitBreaker.decorateSupplier(
    circuitBreaker, 
    () -> callPaymentService()  // remote call
);

// With fallback
String result = Try.ofSupplier(decoratedSupplier)
    .recover(throwable -> fallbackPaymentResponse())
    .get();

Python Example Using a Simple Custom Implementation

import time
from enum import Enum
from typing import Callable, Any

class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

class CircuitBreaker:
    def __init__(self, failure_threshold: int = 5, reset_timeout: int = 30, success_threshold: int = 2):
        self.state = CircuitState.CLOSED
        self.failure_threshold = failure_threshold
        self.reset_timeout = reset_timeout
        self.success_threshold = success_threshold
        self.failure_count = 0
        self.success_count = 0
        self.last_failure_time = 0

    def call(self, func: Callable, *args, **kwargs) -> Any:
        if self.state == CircuitState.OPEN:
            if time.time() - self.last_failure_time > self.reset_timeout:
                self.state = CircuitState.HALF_OPEN
                self.success_count = 0
            else:
                raise CircuitBreakerOpenException("Circuit breaker is OPEN")

        try:
            result = func(*args, **kwargs)
            self._on_success()
            return result
        except Exception as e:
            self._on_failure()
            raise e  # or handle with fallback

    def _on_success(self):
        self.failure_count = 0
        self.success_count += 1
        if self.state == CircuitState.HALF_OPEN and self.success_count >= self.success_threshold:
            self.state = CircuitState.CLOSED

    def _on_failure(self):
        self.failure_count += 1
        self.last_failure_time = time.time()
        if self.failure_count >= self.failure_threshold or self.state == CircuitState.HALF_OPEN:
            self.state = CircuitState.OPEN

class CircuitBreakerOpenException(Exception):
    pass

These implementations highlight essential elements: configurable thresholds, state management, fallback execution, and safe transitions. In real systems, thread-safety (using locks or atomic operations) and integration with monitoring tools like Prometheus are mandatory.

When and How to Use the Circuit Breaker Pattern

Apply the Circuit Breaker Pattern to any synchronous or asynchronous call to external services, databases, or third-party APIs where failure could propagate. Common scenarios include microservices communication, payment gateways, inventory checks, or recommendation engines.

Best practices:

Combine with timeouts to avoid indefinite waits.
Implement meaningful fallbacks—cached data, default values, or queued operations.
Monitor state transitions and metrics for observability.
Tune thresholds based on service characteristics and traffic patterns.
Ensure idempotency for operations that may be retried.

The pattern shines in high-traffic environments but adds slight overhead in normal operation due to metric collection. For extremely latency-sensitive paths, evaluate whether the protection justifies the cost.

Mastering the Circuit Breaker Pattern equips system designers with a powerful tool to build resilient, fault-tolerant distributed systems that maintain availability even when individual components fail.

Buy me coffee to support my content at: https://ko-fi.com/codewithdhanian

Rate Limiting & Throttling in System Design

CodeWithDhanian — Fri, 03 Apr 2026 07:54:26 +0000

In large-scale distributed systems and microservices architectures, uncontrolled incoming traffic can quickly lead to resource exhaustion, degraded performance, or complete service outages. Rate limiting and throttling serve as critical defensive mechanisms that protect backend services, ensure fair usage among clients, prevent abuse, and maintain overall system stability under varying load conditions. These techniques control the flow of requests to APIs, databases, or other resources, allowing systems to operate reliably even during traffic spikes or malicious attacks.

Understanding Rate Limiting

Rate limiting is a technique that enforces a strict upper bound on the number of requests a client, user, IP address, or API key can make within a defined time window. The primary goals include protecting against DDoS attacks, ensuring fair resource allocation, enforcing business quotas, and preventing any single client from monopolizing shared resources.

When a request exceeds the allowed limit, the system typically rejects it immediately and returns an HTTP 429 Too Many Requests status code, often accompanied by headers such as Retry-After to inform the client when it may retry.

Rate limiting operates at multiple layers: at the API gateway, within individual microservices, at the load balancer, or even at the edge using content delivery networks. In distributed environments, the rate limiter must maintain consistent state across multiple nodes, typically using a centralized store such as Redis.

Understanding Throttling

Throttling differs from rate limiting by focusing on controlling the processing speed or flow of requests rather than imposing a hard rejection limit. Instead of outright denying excess requests, throttling slows down, queues, or paces the handling of requests to maintain a steady load on the system.

While rate limiting answers the question “Is this request allowed?”, throttling addresses “How fast should this request be processed?”. Throttling is particularly useful for smoothing bursty traffic, protecting downstream services with their own rate limits, or gracefully handling temporary overload without dropping legitimate requests.

Common throttling strategies include introducing artificial delays, queuing requests in message queues, or dynamically reducing the processing rate based on current system metrics such as CPU usage or queue length.

Key Differences Between Rate Limiting and Throttling

Rate limiting provides a hard cap and immediate rejection for excess requests, making it ideal for quota enforcement and abuse prevention. Throttling prioritizes smoothing traffic and improving user experience by avoiding abrupt denials, often at the cost of increased latency for some requests. Many production systems combine both: rate limiting at the entry point for protection and throttling internally for traffic shaping.

Common Rate Limiting Algorithms

Several well-established algorithms exist for implementing rate limiting, each offering different trade-offs in terms of burst tolerance, accuracy, memory usage, and implementation complexity.

Token Bucket Algorithm

The token bucket algorithm is one of the most widely adopted approaches due to its flexibility and ability to handle controlled bursts. It models capacity as a bucket that accumulates tokens at a constant refill rate up to a maximum capacity. Each incoming request consumes one token. If tokens are available, the request is allowed; otherwise, it is rejected.

Key parameters:

Refill rate (r): Tokens added per unit time (e.g., 10 tokens per second).
Bucket capacity (b): Maximum number of tokens the bucket can hold, determining burst size.

This algorithm allows short bursts up to the bucket capacity while enforcing the long-term average rate. It is particularly suitable for public APIs where users may send occasional bursts of requests after periods of inactivity.

Complete Token Bucket Implementation Example Using Redis (Lua Script for Atomicity)

-- Token Bucket Lua Script for Redis
local key = KEYS[1]                  -- e.g., "rate:limit:user:123"
local now = tonumber(ARGV[1])        -- current timestamp in seconds
local refill_rate = tonumber(ARGV[2]) -- tokens per second
local capacity = tonumber(ARGV[3])   -- max bucket size
local tokens_requested = tonumber(ARGV[4]) or 1

-- Get current tokens and last refill time
local last_refill = tonumber(redis.call("HGET", key, "last_refill") or now)
local tokens = tonumber(redis.call("HGET", key, "tokens") or capacity)

-- Calculate new tokens to add
local elapsed = now - last_refill
local new_tokens = math.floor(elapsed * refill_rate)
tokens = math.min(tokens + new_tokens, capacity)

-- Check if enough tokens available
if tokens >= tokens_requested then
    tokens = tokens - tokens_requested
    redis.call("HSET", key, "tokens", tokens)
    redis.call("HSET", key, "last_refill", now)
    redis.call("EXPIRE", key, 3600)  -- expire after 1 hour for cleanup
    return {1, tokens}               -- allowed, remaining tokens
else
    return {0, tokens}               -- rejected, remaining tokens
end

This Lua script ensures atomic execution, preventing race conditions in distributed systems. The client calls this script via EVAL or EVALSHA commands in Redis.

Leaky Bucket Algorithm

The leaky bucket algorithm treats requests as water pouring into a bucket with a small hole at the bottom. Requests enter the bucket and are processed (leaked) at a constant fixed rate. If the bucket overflows, incoming requests are rejected or queued.

Leaky bucket excels at smoothing traffic to a steady output rate, making it ideal for scenarios requiring predictable load, such as payment processing or integration with external services that have strict rate limits. Unlike token bucket, it does not permit large bursts; excess requests are either delayed or dropped.

Fixed Window Counter Algorithm

The fixed window algorithm divides time into fixed intervals (e.g., one minute or one hour) and counts the number of requests within each window. A counter is incremented for every allowed request. When the counter exceeds the limit for the current window, further requests are rejected until the next window begins.

This approach is simple and memory-efficient but suffers from the boundary burst problem: clients can send twice the allowed rate at window edges (e.g., 100 requests at the end of one minute and another 100 immediately at the start of the next).

Simple Fixed Window Pseudocode

function isAllowed(clientId, limit, windowSeconds):
    currentWindow = floor(currentTime / windowSeconds)
    counterKey = "rate:" + clientId + ":" + currentWindow
    count = redis.INCR(counterKey)
    if count == 1:
        redis.EXPIRE(counterKey, windowSeconds)
    return count <= limit

Sliding Window Algorithms

Sliding window approaches provide higher accuracy by using a continuously moving time frame instead of rigid boundaries.

Sliding Window Log: Maintains a sorted list or set of timestamps for every request made by a client within the window. On each request, remove old timestamps outside the window and check if the remaining count is below the limit. This offers precise control but consumes significant memory for high-traffic clients.

Sliding Window Counter: A hybrid that combines fixed windows with mathematical adjustment. It tracks counts in the current and previous windows and calculates a weighted count for the sliding period. This balances accuracy and memory usage effectively.

Distributed Rate Limiting Considerations

In microservices or multi-node deployments, a single in-memory rate limiter is insufficient. Designers must ensure consistency across instances using a shared distributed cache such as Redis, Memcached, or a dedicated rate-limiting service.

Consistent hashing can route requests for the same client to the same shard, while Lua scripts or atomic operations guarantee correctness under concurrency. For extremely high scale, consider Redis Cluster or consistent hashing combined with local caching for hot clients.

Idempotency and proper error handling are essential: clients should receive clear rate limit headers (X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset) to adjust their behavior gracefully.

Best Practices for Implementing Rate Limiting & Throttling

Apply rate limiting at multiple levels: edge (CDN or API gateway), service level, and database level. Choose the algorithm based on requirements — token bucket for burst-tolerant APIs, leaky bucket for traffic shaping, and sliding window counter for strict fairness with good performance.

Use Redis with Lua scripts for atomicity in distributed setups. Always return informative headers and consider adaptive rate limiting that dynamically adjusts limits based on system load. Combine with circuit breakers, bulkheads, and monitoring (Prometheus, Grafana) to detect and respond to abuse patterns.

For throttling, integrate with message queues (Kafka, RabbitMQ) to queue excess requests or apply exponential backoff and jitter on retries.

Rate limiting and throttling form foundational resilience patterns in system design. Proper implementation protects services, improves user experience, and enables sustainable scaling of distributed systems.

Buy me coffee to support my content at: https://ko-fi.com/codewithdhanian

Idempotency in System Design

CodeWithDhanian — Fri, 03 Apr 2026 07:05:37 +0000

In the realm of distributed systems and microservices architectures, ensuring that operations produce the same result regardless of how many times they are executed stands as a fundamental requirement for building reliable and fault-tolerant applications. Idempotency addresses this need by guaranteeing that repeated invocations of the same operation yield identical outcomes without causing unintended side effects. This concept becomes especially critical when dealing with network failures, retries, message queues, and distributed transactions, where the same request may arrive multiple times due to timeouts, duplicate deliveries, or client retries.

Understanding Idempotency

Idempotency derives from mathematics, where a function is idempotent if applying it multiple times produces the same result as applying it once. In system design, an idempotent operation ensures that executing the same request several times has the same effect as executing it exactly once.

Key characteristics of idempotent operations include:

Safe repetition: Repeating the request does not create duplicate resources, charge a customer multiple times, or alter system state unexpectedly.
No cumulative side effects: The system state after N identical requests equals the state after a single request.
Predictable outcomes: Clients can safely retry failed operations without fear of inconsistency.

Common examples of idempotent operations include:

Retrieving a resource via GET in REST APIs
Updating a resource with a complete replacement (PUT)
Deleting a resource (DELETE)
Processing a payment with a unique idempotency key

Non-idempotent operations, such as creating a new resource with POST or incrementing a counter, require special handling to prevent duplicates.

Why Idempotency Matters in Distributed Systems

Distributed systems face inherent unreliability due to network partitions, service failures, and timeout issues. Clients and intermediaries often implement retry mechanisms with exponential backoff. Without idempotency, these retries can lead to serious problems:

Duplicate orders in e-commerce platforms
Multiple charges on customer payment methods
Inconsistent inventory levels
Corrupted data in financial ledgers

Idempotency serves as a critical defense mechanism that allows safe retries, simplifies error recovery, and supports at-least-once delivery semantics commonly found in message queues like Kafka, RabbitMQ, and SQS. It forms an essential building block alongside patterns such as the Circuit Breaker, Retry & Exponential Backoff, and Saga for distributed transactions.

Implementing Idempotency with Idempotency Keys

The most robust approach to achieving idempotency involves the use of idempotency keys. A client generates a unique identifier for each logical operation and includes it in every request. The server stores the result associated with this key and reuses it for subsequent identical requests.

Core Components of Idempotency Key Implementation

Idempotency Key: A unique string or UUID generated by the client, typically tied to a specific business operation.
Storage Layer: A durable store (database table, Redis cache, or dedicated service) that records processed keys along with their outcomes.
Validation Logic: Server-side checks to detect duplicate requests and return cached responses.
Expiration Mechanism: Optional time-based cleanup of old keys to prevent unbounded storage growth.

Complete Implementation Example: Idempotent Payment API

Consider a payment processing endpoint that must remain idempotent even under heavy retry scenarios.

Server-Side Controller (Idempotent Endpoint)

@RestController
@RequestMapping("/payments")
class PaymentController {

    PaymentService paymentService;
    IdempotencyStore idempotencyStore;  // Backed by Redis or Database

    @PostMapping
    ResponseEntity<PaymentResponse> processPayment(
            @RequestHeader("Idempotency-Key") String idempotencyKey,
            @RequestBody PaymentRequest request) {

        // Step 1: Check for existing result
        Optional<PaymentResponse> existingResult = idempotencyStore.getResult(idempotencyKey);
        if (existingResult.isPresent()) {
            return ResponseEntity.ok(existingResult.get());  // Return cached response
        }

        // Step 2: Validate key format and business rules
        if (!isValidIdempotencyKey(idempotencyKey)) {
            throw new InvalidIdempotencyKeyException();
        }

        try {
            // Step 3: Execute the actual business logic
            PaymentResult result = paymentService.processPayment(request);

            PaymentResponse response = mapToResponse(result);

            // Step 4: Store the result atomically with the key
            idempotencyStore.storeResult(idempotencyKey, response, Duration.ofHours(24));

            return ResponseEntity.ok(response);

        } catch (Exception e) {
            // Store failure state to prevent partial retries
            idempotencyStore.storeFailure(idempotencyKey, e.getMessage());
            throw e;
        }
    }
}

Idempotency Store Implementation (Using Redis for High Performance)

class RedisIdempotencyStore implements IdempotencyStore {

    RedisTemplate<String, Object> redisTemplate;

    Optional<PaymentResponse> getResult(String key) {
        String stored = (String) redisTemplate.opsForValue().get("idempotency:" + key);
        if (stored == null) {
            return Optional.empty();
        }
        return Optional.of(deserializeResponse(stored));
    }

    void storeResult(String key, PaymentResponse response, Duration ttl) {
        String serialized = serializeResponse(response);
        redisTemplate.opsForValue().set(
            "idempotency:" + key, 
            serialized, 
            ttl
        );
    }

    void storeFailure(String key, String errorMessage) {
        redisTemplate.opsForValue().set(
            "idempotency:" + key + ":failure", 
            errorMessage, 
            Duration.ofHours(1)
        );
    }
}

This implementation ensures that even if the same request with the identical idempotency key arrives multiple times—due to network retry, load balancer duplication, or client-side retry logic—the server processes the payment operation only once and returns the same response for all subsequent calls.

Best Practices for Idempotency

Client Responsibilities:

Generate a unique idempotency key (UUID v4 recommended) per logical operation before sending the request.
Store the key locally and reuse it for all retries of the same operation.
Include the key in a standard header such as Idempotency-Key.

Server Responsibilities:

Reject requests missing a valid idempotency key for non-safe operations.
Make storage and business logic execution atomic where possible.
Use short TTLs on stored results to manage storage growth.
Design all compensating actions in Saga patterns to also be idempotent.

Idempotency in Message-Driven Systems:
When consuming from message queues, attach idempotency keys to messages. Consumers should check the key before processing and acknowledge only after successful storage of the result. This pattern supports at-least-once delivery while preventing duplicate side effects.

Idempotency in Broader System Design Patterns

Idempotency integrates deeply with other critical concepts:

In distributed transactions using the Saga pattern, every compensating transaction must be idempotent to handle repeated failure recoveries safely.
API Gateways and service meshes can enforce idempotency at the infrastructure layer.
Event-Driven Architectures rely on idempotent event handlers to maintain consistency across services.
Retry & Exponential Backoff strategies become safe only when paired with proper idempotency controls.

In high-scale systems, idempotency often combines with rate limiting, circuit breakers, and distributed caching to create resilient request pipelines.

Mastering idempotency enables system designers to build applications that gracefully handle the realities of distributed environments while maintaining data integrity and providing consistent user experiences.

Buy me coffee to support my content at: https://ko-fi.com/codewithdhanian

Distributed Transactions (2PC, Saga) in System Design

CodeWithDhanian — Fri, 03 Apr 2026 06:52:41 +0000

In the complex landscape of modern distributed systems, maintaining data consistency across multiple independent services and databases presents one of the most challenging problems in system design. Distributed transactions provide the foundation for ensuring that operations spanning several resources either succeed completely or fail entirely, preserving the ACID properties of atomicity, consistency, isolation, and durability. This article explores two primary approaches to handling distributed transactions: the Two-Phase Commit protocol, commonly known as 2PC, and the Saga pattern. Each method addresses the coordination of long-running transactions in environments where traditional single-database transactions fall short.

What Are Distributed Transactions

A distributed transaction involves multiple participating resources, such as separate databases, microservices, or external systems, that must coordinate to achieve a unified outcome. Unlike local transactions confined to a single resource, distributed transactions must manage cross-service consistency while dealing with network latency, partial failures, and independent scaling of components.

The core requirement remains the same as in monolithic systems: the entire operation must appear atomic to the end user. If any part fails, all changes must be undone. However, achieving this in a distributed environment introduces significant complexity because each participant operates autonomously, and communication occurs over unreliable networks. System designers must therefore select protocols that balance strong consistency with availability and performance.

The Two-Phase Commit Protocol (2PC)

The Two-Phase Commit protocol, or 2PC, stands as the classic solution for achieving strong consistency in distributed transactions. Introduced in the 1970s, 2PC relies on a central coordinator and multiple participants to ensure all-or-nothing semantics across heterogeneous resources.

Core Components of 2PC

Coordinator: The central authority responsible for driving the transaction. It receives the initial transaction request and manages the voting and decision process.
Participants: The individual resources (databases or services) that perform local work and respond to the coordinator's instructions.
Transaction Manager: Often implemented using standards such as XA (eXtended Architecture) for database interactions.

Phases of the 2PC Protocol

2PC operates in two distinct phases, ensuring safety before any permanent changes occur.

Prepare Phase (Voting Phase):

The coordinator sends a prepare message to all participants. Each participant performs the necessary local operations, acquires locks, writes changes to a durable log, and responds with either ready (vote yes) or abort (vote no). If any participant votes no or fails to respond, the coordinator decides to abort.

Commit Phase (Decision Phase):

If all participants vote ready, the coordinator logs the global commit decision and sends commit messages to every participant. Each participant then applies the changes permanently and releases locks. If the decision is to abort, the coordinator sends rollback messages, and participants undo their local changes using the prepared log entries.

Pseudocode Implementation of 2PC Coordinator

class TwoPhaseCommitCoordinator {
    List<Participant> participants;
    TransactionLog log;

    void beginTransaction(Transaction tx) {
        log.write("BEGIN_TX", tx.id);
        boolean allReady = true;

        // Prepare Phase
        for each participant in participants {
            Response response = participant.prepare(tx);
            if (!response.isReady()) {
                allReady = false;
                break;
            }
        }

        // Decision
        if (allReady) {
            log.write("GLOBAL_COMMIT", tx.id);
            for each participant in participants {
                participant.commit(tx);
            }
        } else {
            log.write("GLOBAL_ABORT", tx.id);
            for each participant in participants {
                participant.rollback(tx);
            }
        }
    }
}

Pseudocode for a 2PC Participant

class DatabaseParticipant implements Participant {
    LocalDatabase db;
    UndoLog undoLog;

    Response prepare(Transaction tx) {
        try {
            db.acquireLocks(tx.operations);
            db.executeOperations(tx.operations);  // tentative changes
            undoLog.recordUndoInfo(tx);
            return new Response(true, "READY");
        } catch (Exception e) {
            return new Response(false, "ABORT");
        }
    }

    void commit(Transaction tx) {
        db.makeChangesPermanent(tx);
        db.releaseLocks(tx);
        undoLog.clear(tx);
    }

    void rollback(Transaction tx) {
        db.applyUndo(undoLog.getUndoInfo(tx));
        db.releaseLocks(tx);
        undoLog.clear(tx);
    }
}

These code structures illustrate the blocking nature of 2PC: participants hold locks from the prepare phase until the final decision arrives. The coordinator must persist its decision durably before proceeding, ensuring recoverability after crashes.

Limitations of 2PC

While 2PC guarantees strong consistency, it suffers from several critical drawbacks. The coordinator becomes a single point of failure and a performance bottleneck. The protocol is blocking—if the coordinator fails after the prepare phase, participants remain locked indefinitely until recovery. Network partitions can cause prolonged unavailability. In high-throughput microservices environments, these issues make 2PC impractical for long-lived operations.

The Saga Pattern

The Saga pattern offers a fundamentally different approach to distributed transactions by embracing eventual consistency instead of immediate strong consistency. Originally described in the 1980s for handling long-lived transactions, a Saga decomposes a large distributed transaction into a sequence of smaller, local transactions. Each local transaction has an associated compensating transaction that undoes its effects if later steps fail.

Key Principles of Saga

Local Transactions: Each service executes its part independently and commits immediately.
Compensating Transactions: Reversible operations that restore the system to a consistent state without global rollback.
No Global Lock: Resources remain available throughout the process.
Eventual Consistency: The system converges to a consistent state over time rather than instantly.

Two Implementation Styles of Saga

Choreography-Based Saga:

Services communicate directly through events. Each service listens for events from previous steps and publishes its own events upon completion or failure. No central controller exists. This style promotes loose coupling but can become difficult to trace as the number of services grows.

Orchestration-Based Saga:

A central Saga Orchestrator coordinates the flow by sending commands to services and reacting to their responses or events. The orchestrator maintains the overall state and decides the next step or triggers compensation. This approach provides clearer visibility into the transaction flow and simplifies error handling.

Complete Orchestration-Based Saga Example: E-commerce Order Processing

Consider an online store where placing an order involves three services: Order Service, Payment Service, and Inventory Service. The Saga ensures that if payment fails, inventory is not deducted, or if inventory is unavailable, payment is refunded.

Saga Orchestrator Pseudocode (Full Structure)

class OrderSagaOrchestrator {
    OrderService orderService;
    PaymentService paymentService;
    InventoryService inventoryService;
    SagaStateRepository stateRepo;

    void startOrderSaga(OrderRequest request) {
        SagaInstance saga = new SagaInstance(request.orderId);
        stateRepo.save(saga);

        // Step 1: Create Order (local transaction)
        Order order = orderService.createOrder(request);
        saga.updateStep("ORDER_CREATED", order);

        try {
            // Step 2: Process Payment
            Payment payment = paymentService.processPayment(order);
            saga.updateStep("PAYMENT_SUCCESS", payment);

            // Step 3: Reserve Inventory
            InventoryReservation reservation = inventoryService.reserveInventory(order);
            saga.updateStep("INVENTORY_RESERVED", reservation);

            saga.complete();
            return;

        } catch (PaymentFailedException e) {
            // Compensation: Cancel Order
            orderService.cancelOrder(order);
            saga.fail("PAYMENT_FAILED");
        } catch (InventoryUnavailableException e) {
            // Compensation Chain
            paymentService.refundPayment(payment);
            orderService.cancelOrder(order);
            saga.fail("INVENTORY_FAILED");
        }
    }

    // Compensating transaction examples
    void compensatePayment(Payment payment) {
        paymentService.refundPayment(payment);  // idempotent refund
    }

    void compensateOrder(Order order) {
        orderService.cancelOrder(order);  // releases any reservations
    }
}

Service-Level Local Transaction Example (Inventory Service)

class InventoryService {
    InventoryRepository repo;

    InventoryReservation reserveInventory(Order order) {
        // Local transaction - fully committed immediately
        return repo.withinTransaction(() -> {
            Stock stock = repo.findStock(order.productId);
            if (stock.quantity < order.quantity) {
                throw new InventoryUnavailableException();
            }
            stock.quantity -= order.quantity;
            repo.save(stock);
            return new InventoryReservation(order.orderId, order.quantity);
        });
    }

    // Compensating transaction - public and idempotent
    void releaseInventory(InventoryReservation reservation) {
        repo.withinTransaction(() -> {
            Stock stock = repo.findStock(reservation.productId);
            stock.quantity += reservation.quantity;
            repo.save(stock);
        });
    }
}

This full code structure demonstrates how the orchestrator drives the Saga while each service remains responsible only for its local ACID transaction and its compensating transaction. Idempotency keys should be included in every command and compensation to handle retries safely after network failures.

Advantages of the Saga Pattern

Saga excels in microservices because it avoids long-held locks, improves availability, and scales horizontally. Failures trigger targeted compensations rather than global aborts. The pattern naturally fits event-driven architectures and works seamlessly with message queues such as Kafka or RabbitMQ for reliable event delivery.

Choosing Between 2PC and Saga

2PC suits scenarios demanding immediate strong consistency, such as financial systems where partial states are unacceptable. Saga fits better for business processes that tolerate temporary inconsistencies, prioritize high availability, and involve long-running workflows across many services. In practice, many system designs combine both: 2PC for critical synchronous steps within a bounded context and Saga for cross-context orchestration.

The Saga pattern, supported by modern frameworks, has become the de facto standard for distributed transactions in cloud-native microservices due to its resilience and performance characteristics. Proper implementation requires careful design of compensating transactions, idempotency, and comprehensive monitoring of Saga instances to detect and resolve stuck workflows.

Distributed Transactions (2PC, Saga) in System Design remains a cornerstone topic that every professional system designer must master to build reliable, scalable, and maintainable large-scale applications.

Buy me coffee to support my content at: https://ko-fi.com/codewithdhanian

Event-Driven Architecture in System Design

CodeWithDhanian — Fri, 03 Apr 2026 06:34:44 +0000

Event-Driven Architecture represents a foundational paradigm in modern system design where the entire flow of processing is triggered and governed by events rather than direct synchronous calls between components. In this approach, systems detect meaningful changes in state, encapsulate them as events, and propagate them asynchronously across decoupled services. This enables real-time responsiveness, loose coupling, and high scalability while allowing individual components to evolve independently without breaking the overall system.

What is Event-Driven Architecture

Event-Driven Architecture shifts away from the traditional request-response model commonly seen in RESTful APIs or monolithic applications. Instead of one service directly invoking another and waiting for a reply, producers emit events that signal something has occurred. Consumers then react to these events at their own pace. This asynchronous nature eliminates blocking calls, reduces latency bottlenecks, and supports massive concurrency.

At its core, Event-Driven Architecture treats events as first-class citizens. An event is an immutable record of a fact that happened in the past, carrying both metadata and payload data. Examples include OrderPlaced, UserRegistered, or PaymentProcessed. These events drive the behavior of the entire distributed system without requiring tight integration between services.

Core Components of Event-Driven Architecture

Every Event-Driven Architecture relies on four essential building blocks that work together to ensure reliable event flow.

Events

An event is the fundamental unit of communication. It must be immutable, idempotent, and self-contained. Each event typically includes a unique event ID, timestamp, event type, source, and a payload with domain-specific data. Events are never altered after creation; instead, new events represent subsequent state changes.

Producers

Producers are the components responsible for detecting state changes and publishing events to the central event broker. A producer can be a microservice, a database trigger, or an external system. Upon generating an event, the producer serializes it into a standard format such as JSON or Avro and sends it reliably to the broker.

Consumers

Consumers subscribe to one or more event streams and execute business logic when matching events arrive. A single event can be consumed by multiple consumers simultaneously, enabling parallel processing. Consumers can be grouped into consumer groups to achieve load balancing and horizontal scaling.

Event Broker

The event broker serves as the reliable messaging backbone. It receives events from producers, persists them durably, and delivers them to consumers. Popular event brokers include Apache Kafka for high-throughput streaming and RabbitMQ for flexible routing. The broker guarantees at-least-once, exactly-once, or at-most-once delivery semantics depending on configuration.

How Event-Driven Architecture Works in Practice

Consider an e-commerce platform built with Event-Driven Architecture. When a customer places an order, the Order Service acts as a producer and publishes an OrderPlaced event. This event flows to the event broker and is immediately available to three independent consumers:

The Inventory Service subtracts stock and publishes an InventoryUpdated event.
The Payment Service processes the transaction and publishes a PaymentProcessed event.
The Notification Service sends an email confirmation to the customer.

None of these services call each other directly. They remain completely decoupled, allowing each to scale, fail, or be updated independently while the event broker ensures reliable delivery.

Key Patterns in Event-Driven Architecture

Several proven patterns elevate Event-Driven Architecture from basic messaging to sophisticated system design solutions.

Event Sourcing

Event Sourcing stores the complete history of events rather than just the current state of an entity. The current state of any object is reconstructed by replaying the sequence of events from the beginning. This pattern provides perfect auditability, time-travel debugging, and easy recovery from failures.

Command Query Responsibility Segregation (CQRS)

CQRS separates the write path (commands) from the read path (queries). Commands generate events that update the write model. A separate read model is kept synchronized through event subscriptions. This allows optimized data structures for reads while maintaining strong consistency on writes.

Saga Pattern

The Saga Pattern orchestrates long-running distributed transactions without relying on traditional two-phase commit. Each step in a business process publishes a completion event or a compensating event on failure. For example, if an order fails payment, a CancelOrder event triggers compensating actions across services to maintain overall consistency.

Implementation Example Using Apache Kafka

Apache Kafka is the industry-standard event broker for high-scale Event-Driven Architecture. Below are complete, production-ready code snippets in Python using the official confluent-kafka library.

Kafka Producer Implementation

from confluent_kafka import Producer
import json
import socket

def delivery_callback(err, msg):
    if err:
        print(f"Message delivery failed: {err}")
    else:
        print(f"Message delivered to {msg.topic()} [{msg.partition()}]")

conf = {
    'bootstrap.servers': 'kafka-broker-1:9092,kafka-broker-2:9092',
    'client.id': socket.gethostname(),
    'acks': 'all',                    # Wait for all in-sync replicas
    'enable.idempotence': True        # Prevent duplicate events
}

producer = Producer(conf)

# Publish an OrderPlaced event
event_data = {
    "event_id": "evt-1234567890",
    "event_type": "OrderPlaced",
    "timestamp": "2026-04-03T06:18:00Z",
    "payload": {
        "order_id": "ORD-98765",
        "user_id": "USR-54321",
        "items": [{"product_id": "PROD-111", "quantity": 2}],
        "total_amount": 149.99
    }
}

producer.produce(
    topic='orders',
    value=json.dumps(event_data).encode('utf-8'),
    key=event_data["payload"]["order_id"],  # Ensures ordering per order
    callback=delivery_callback
)

producer.flush()  # Ensure all messages are sent before exit

This producer guarantees exactly-once semantics through idempotence and waits for full replication before considering the event published.

Kafka Consumer Implementation

from confluent_kafka import Consumer, KafkaError
import json

conf = {
    'bootstrap.servers': 'kafka-broker-1:9092,kafka-broker-2:9092',
    'group.id': 'inventory-service-group',
    'auto.offset.reset': 'earliest',
    'enable.auto.commit': False,      # Manual commit for exactly-once
    'isolation.level': 'read_committed'
}

consumer = Consumer(conf)
consumer.subscribe(['orders'])

while True:
    msg = consumer.poll(timeout=1.0)
    if msg is None:
        continue
    if msg.error():
        if msg.error().code() == KafkaError._PARTITION_EOF:
            continue
        else:
            print(f"Error: {msg.error()}")
            break

    event = json.loads(msg.value().decode('utf-8'))

    if event["event_type"] == "OrderPlaced":
        # Process inventory deduction
        print(f"Processing inventory for order {event['payload']['order_id']}")
        # ... business logic here ...

        # Publish follow-up event
        # (In real systems this would use a separate producer)

        # Manual commit only after successful processing
        consumer.commit(msg)

The consumer belongs to a consumer group, processes events in order within each partition, and commits offsets only after successful business logic execution.

Challenges in Event-Driven Architecture

While powerful, Event-Driven Architecture introduces specific complexities that must be addressed:

Eventual Consistency: Data across services may temporarily differ until all events propagate.
Event Ordering: Guaranteeing strict chronological order requires careful partitioning and key selection.
Idempotency: Consumers must handle duplicate events gracefully using event IDs or deduplication tables.
Debugging Distributed Flows: Tracing a single business transaction across dozens of events requires distributed tracing tools.
Schema Evolution: Events must support forward and backward compatibility through schema registries.

Best Practices for Event-Driven Architecture

To build robust Event-Driven Architecture systems, always:

Design events as facts about the past, never as commands.
Use Avro or Protobuf with a schema registry for type safety.
Implement dead-letter queues for failed events.
Monitor event lag, throughput, and consumer health continuously.
Version events explicitly and maintain backward compatibility.
Combine Event-Driven Architecture with CQRS and Event Sourcing only when business requirements justify the added complexity.

Event-Driven Architecture empowers system design teams to create resilient, scalable, and maintainable distributed systems that respond instantly to real-world changes.

System Design Handbook

To master every concept in system design including Event-Driven Architecture, purchase the complete System Design Handbook at https://codewithdhanian.gumroad.com/l/ntmcf.

Buy me coffee to support my content at https://ko-fi.com/codewithdhanian.

Message Queues (Kafka, RabbitMQ, SQS) in System Design

CodeWithDhanian — Fri, 03 Apr 2026 06:09:40 +0000

Introduction to Message Queues in System Design

Message queues serve as the backbone of asynchronous communication in distributed systems. They enable producers to send messages that consumers process independently, without requiring both parties to be available simultaneously. This decoupling eliminates tight temporal dependencies, buffers traffic during spikes, and provides fault tolerance by allowing retries and dead-letter queues for failed messages.

In system design, message queues address critical challenges such as scalability, reliability, and throughput. They support patterns including event-driven architecture, microservices communication, task queues, and log aggregation. By persisting messages durably, they ensure data survives failures and can be replayed when necessary. Message queues come in two broad categories: traditional task queues focused on routing and delivery, and event streams optimized for high-volume, ordered, replayable data.

Three prominent implementations dominate modern system design: Apache Kafka for high-throughput event streaming, RabbitMQ for flexible routing and messaging patterns, and Amazon SQS for fully managed, serverless queue operations. Each offers distinct strengths in architecture, delivery guarantees, and operational model.

Apache Kafka: Distributed Event Streaming Platform

Apache Kafka functions as a distributed event streaming platform rather than a simple message queue. It excels in scenarios demanding massive throughput, durability, and replayability across thousands of producers and consumers.

Core Architecture of Kafka

A Kafka cluster consists of multiple brokers that store and serve data. Topics act as logical categories for messages, each divided into partitions for parallelism. Every partition is an ordered, immutable log stored on a broker.

Replication ensures fault tolerance: each partition has a leader handling reads and writes, plus followers (in-sync replicas) that copy data. If the leader fails, a follower is elected. Producers write to topics, while consumers read from partitions and commit offsets to track progress. Consumer groups enable load balancing, with each partition assigned to exactly one consumer in the group.

Kafka achieves exactly-once semantics through idempotent producers (using sequence numbers to deduplicate retries) and transactions (atomic produce-and-commit operations). Modern deployments use KRaft mode for metadata management via the Raft consensus protocol, eliminating external coordination dependencies.

Kafka delivers high throughput via batching, compression, and zero-copy transfers. It supports log compaction for stateful streams and integrates seamlessly with stream processing frameworks.

Producer and Consumer Implementation in Kafka

Here is a complete Python implementation using the kafka-python library for a basic producer and consumer. This demonstrates idempotent production and offset management.

from kafka import KafkaProducer, KafkaConsumer
from kafka.errors import KafkaError
import json
import time

# Producer configuration
producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    value_serializer=lambda v: json.dumps(v).encode('utf-8'),
    acks='all',                    # Wait for all in-sync replicas
    retries=3,                     # Retry on transient failures
    enable_idempotence=True,       # Prevent duplicates on retries
    batch_size=16384,              # Batch records for efficiency
    linger_ms=5                    # Wait briefly to fill batches
)

# Send messages
for i in range(100):
    message = {'event_id': i, 'data': f'payload-{i}'}
    future = producer.send('user-events', value=message, key=str(i).encode())
    try:
        record_metadata = future.get(timeout=10)
        print(f"Message sent to topic {record_metadata.topic} partition {record_metadata.partition} offset {record_metadata.offset}")
    except KafkaError as e:
        print(f"Failed to send: {e}")
producer.flush()
producer.close()

Explanation of producer code: The bootstrap_servers connects to the cluster. acks='all' ensures durability by waiting for replication. enable_idempotence=True guarantees no duplicates. Batching via batch_size and linger_ms maximizes throughput. The send method returns a future for asynchronous confirmation.

# Consumer configuration
consumer = KafkaConsumer(
    'user-events',
    bootstrap_servers=['localhost:9092'],
    group_id='event-processors',
    value_deserializer=lambda x: json.loads(x.decode('utf-8')),
    auto_offset_reset='earliest',   # Start from beginning if no offset
    enable_auto_commit=False        # Manual control for exactly-once
)

for message in consumer:
    print(f"Consumed: {message.value} from partition {message.partition} offset {message.offset}")
    # Process message here
    # On success: consumer.commit() for manual offset commit

Explanation of consumer code: The group_id joins a consumer group for parallel processing. auto_offset_reset controls initial position. Manual commits allow transactional exactly-once processing when combined with producer transactions. Offsets are committed only after successful handling.

RabbitMQ: Flexible Messaging Broker with Advanced Routing

RabbitMQ implements the AMQP protocol as a robust message broker optimized for complex routing and reliable delivery. It suits task queues, work distribution, and scenarios requiring sophisticated message patterns.

Core Architecture of RabbitMQ

RabbitMQ uses exchanges to route messages to queues based on bindings and routing keys. Producers publish to exchanges; consumers pull from queues.

Four main exchange types exist:

Direct: Routes based on exact routing key match.
Fanout: Broadcasts to all bound queues (ignores key).
Topic: Uses pattern matching on routing keys (e.g., user.*.event).
Headers: Routes based on message header attributes.

Queues can be durable (survive restarts), exclusive, or auto-delete. Acknowledgments ensure reliable delivery: consumers explicitly acknowledge after processing. Prefetch limits unacknowledged messages per consumer for flow control. Dead-letter queues capture failed messages.

RabbitMQ supports clustering for high availability and mirrored queues for replication. It provides low-latency delivery and works across multiple protocols.

Producer and Consumer Implementation in RabbitMQ

Here is a complete Python implementation using the pika library.

import pika
import json

# Establish connection and channel
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

# Declare durable queue and direct exchange
channel.exchange_declare(exchange='user-events', exchange_type='direct', durable=True)
channel.queue_declare(queue='event-processor', durable=True)
channel.queue_bind(exchange='user-events', queue='event-processor', routing_key='user.created')

# Producer: publish message
for i in range(100):
    message = {'event_id': i, 'type': 'user.created', 'data': f'payload-{i}'}
    channel.basic_publish(
        exchange='user-events',
        routing_key='user.created',
        body=json.dumps(message),
        properties=pika.BasicProperties(
            delivery_mode=2,  # Persistent
        )
    )
    print(f"Published event {i}")

connection.close()

Explanation of producer code: The exchange and queue are declared durable for persistence. basic_publish with delivery_mode=2 ensures the message survives broker restarts. Routing key determines delivery via the binding.

# Consumer setup
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

channel.queue_declare(queue='event-processor', durable=True)

def callback(ch, method, properties, body):
    message = json.loads(body)
    print(f"Processed: {message}")
    ch.basic_ack(delivery_tag=method.delivery_tag)  # Manual ack

channel.basic_qos(prefetch_count=1)  # Fair dispatch
channel.basic_consume(queue='event-processor', on_message_callback=callback)

print("Waiting for messages...")
channel.start_consuming()

Explanation of consumer code: basic_qos with prefetch_count=1 prevents overload. Manual basic_ack confirms successful processing; unacknowledged messages return to the queue. This pattern supports idempotency and retry logic.

Amazon SQS: Fully Managed Serverless Queues

Amazon SQS provides a fully managed message queue service that removes infrastructure overhead. It focuses on simplicity and seamless integration within cloud-native architectures.

Core Architecture of SQS

SQS offers two queue types:

Standard queues: Deliver at-least-once with high throughput and scalability. Messages may arrive out of order or duplicated.
FIFO queues: Guarantee exactly-once processing and strict ordering within message groups.

Producers send messages via API; consumers poll for messages. Visibility timeout hides a received message temporarily, preventing concurrent processing. If not deleted within the timeout, the message reappears. Long polling waits up to 20 seconds for messages, reducing empty responses. Dead-letter queues capture messages failing after a configurable receive count.

SQS handles replication, encryption, and scaling automatically. It integrates natively with other cloud services for event-driven workflows.

Producer and Consumer Implementation in SQS

Here is a complete Python implementation using boto3.

import boto3
import json

sqs = boto3.client('sqs')
queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue'  # Replace with actual URL

# Producer: send message
for i in range(100):
    message = {'event_id': i, 'data': f'payload-{i}'}
    response = sqs.send_message(
        QueueUrl=queue_url,
        MessageBody=json.dumps(message),
        MessageAttributes={
            'EventType': {
                'DataType': 'String',
                'StringValue': 'user.created'
            }
        }
    )
    print(f"Sent message ID: {response['MessageId']}")

Explanation of producer code: send_message accepts body and optional attributes for filtering. SQS handles durability and distribution automatically.

# Consumer: receive and process
while True:
    response = sqs.receive_message(
        QueueUrl=queue_url,
        MaxNumberOfMessages=1,
        WaitTimeSeconds=20,          # Long polling
        VisibilityTimeout=30         # Hide for 30 seconds
    )

    if 'Messages' in response:
        message = response['Messages'][0]
        body = json.loads(message['Body'])
        print(f"Received and processing: {body}")

        # Process logic here

        # Delete after success
        sqs.delete_message(
            QueueUrl=queue_url,
            ReceiptHandle=message['ReceiptHandle']
        )
    else:
        print("No messages available")

Explanation of consumer code: Long polling via WaitTimeSeconds improves efficiency. Visibility timeout prevents duplicate processing. delete_message removes the message permanently using the receipt handle.

Choosing the Right Message Queue

Kafka shines for event streaming with replayability and massive scale. RabbitMQ excels in complex routing and traditional task queues. SQS offers zero-ops management for simple decoupling in cloud environments. Evaluate based on throughput needs, ordering requirements, operational burden, and delivery semantics when designing systems.

To master these and more concepts in system design, consider purchasing the System Design Handbook at https://codewithdhanian.gumroad.com/l/ntmcf. Buy me coffee to support my content at https://ko-fi.com/codewithdhanian.

API Design (REST, GraphQL, gRPC) in System Design

CodeWithDhanian — Fri, 03 Apr 2026 05:47:43 +0000

In the intricate landscape of system design, API design serves as the foundational communication layer that enables different components, services, and clients to interact seamlessly at scale. The choice of API paradigm directly influences performance, maintainability, scalability, and developer experience. Three dominant approaches dominate modern distributed systems: REST, GraphQL, and gRPC. Each addresses distinct challenges in data exchange, efficiency, and real-time capabilities while fitting specific architectural patterns such as microservices, event-driven architectures, and high-throughput applications.

REST API Design

REST, or Representational State Transfer, is an architectural style introduced by Roy Fielding that emphasizes a stateless, client-server model built on standard HTTP protocols. In system design, REST APIs excel when simplicity, cacheability, and broad compatibility are paramount. The core constraints of REST include client-server separation, statelessness, cacheability, uniform interface, layered system, and code on demand.

A REST API treats data as resources identified by URIs. Operations on these resources map to HTTP methods: GET for retrieval, POST for creation, PUT for full updates, PATCH for partial updates, and DELETE for removal. This mapping aligns with CRUD operations while enforcing idempotency where applicable—PUT and DELETE are idempotent, whereas POST is not.

HTTP status codes provide explicit feedback: 2xx for success, 4xx for client errors, and 5xx for server errors. REST leverages HTTP headers for metadata, such as Content-Type, Authorization, and Cache-Control. Versioning is typically handled via URI paths (/v1/users), headers, or query parameters to maintain backward compatibility in evolving systems.

Here is a complete, production-ready REST API implementation using Node.js and Express to illustrate resource management for a user service in a microservices environment:

const express = require('express');
const app = express();
app.use(express.json());

// In-memory store for demonstration (replace with database in production)
let users = [{ id: 1, name: 'Alice', email: 'alice@example.com' }];

// GET all users - retrieval with optional pagination and filtering
app.get('/api/v1/users', (req, res) => {
  const { page = 1, limit = 10 } = req.query;
  const startIndex = (page - 1) * limit;
  const paginatedUsers = users.slice(startIndex, startIndex + parseInt(limit));
  res.status(200).json({
    data: paginatedUsers,
    meta: { total: users.length, page: parseInt(page), limit: parseInt(limit) }
  });
});

// GET single user by ID - resource-specific endpoint
app.get('/api/v1/users/:id', (req, res) => {
  const user = users.find(u => u.id === parseInt(req.params.id));
  if (!user) return res.status(404).json({ error: 'User not found' });
  res.status(200).json(user);
});

// POST create user - non-idempotent creation with validation
app.post('/api/v1/users', (req, res) => {
  const { name, email } = req.body;
  if (!name || !email) return res.status(400).json({ error: 'Name and email required' });
  const newUser = { id: users.length + 1, name, email };
  users.push(newUser);
  res.status(201).json(newUser);
});

// PUT full update - idempotent
app.put('/api/v1/users/:id', (req, res) => {
  const userIndex = users.findIndex(u => u.id === parseInt(req.params.id));
  if (userIndex === -1) return res.status(404).json({ error: 'User not found' });
  users[userIndex] = { id: parseInt(req.params.id), ...req.body };
  res.status(200).json(users[userIndex]);
});

// PATCH partial update
app.patch('/api/v1/users/:id', (req, res) => {
  const user = users.find(u => u.id === parseInt(req.params.id));
  if (!user) return res.status(404).json({ error: 'User not found' });
  Object.assign(user, req.body);
  res.status(200).json(user);
});

// DELETE user - idempotent removal
app.delete('/api/v1/users/:id', (req, res) => {
  const userIndex = users.findIndex(u => u.id === parseInt(req.params.id));
  if (userIndex === -1) return res.status(404).json({ error: 'User not found' });
  users.splice(userIndex, 1);
  res.status(204).send();
});

app.listen(3000, () => console.log('REST API running on port 3000'));

This snippet demonstrates stateless design—each request contains all necessary information. In a real system, integrate database indexing, caching with Redis, and rate limiting at the API gateway level for horizontal scaling.

GraphQL API Design

GraphQL is a query language and runtime for APIs created by Facebook that allows clients to request exactly the data they need in a single round trip, eliminating over-fetching and under-fetching common in REST. In system design, GraphQL shines in complex, hierarchical data models and client-driven microservices where flexibility is essential.

The GraphQL schema defines types, queries, mutations, and subscriptions using Schema Definition Language (SDL). A single endpoint (/graphql) handles all operations via POST requests containing a query or mutation document. Resolvers map fields to data-fetching logic, enabling nested queries without multiple HTTP calls.

GraphQL supports introspection for self-documentation and subscriptions over WebSockets for real-time updates. Batching and caching at the resolver level prevent N+1 query problems using tools like DataLoader.

Here is a complete GraphQL setup using Node.js with Apollo Server and an in-memory store, showcasing a user service with nested relationships:

const { ApolloServer, gql } = require('apollo-server');

// Schema definition
const typeDefs = gql`
  type User {
    id: ID!
    name: String!
    email: String!
    posts: [Post!]!
  }
  type Post {
    id: ID!
    title: String!
    content: String!
    author: User!
  }
  type Query {
    users: [User!]!
    user(id: ID!): User
  }
  type Mutation {
    createUser(name: String!, email: String!): User!
  }
  type Subscription {
    userCreated: User!
  }
`;

// Resolvers with DataLoader pattern for N+1 prevention (simplified)
const users = [{ id: '1', name: 'Alice', email: 'alice@example.com', posts: ['101'] }];
const posts = [{ id: '101', title: 'GraphQL Basics', content: 'Deep dive...', authorId: '1' }];

const resolvers = {
  Query: {
    users: () => users,
    user: (_, { id }) => users.find(u => u.id === id)
  },
  Mutation: {
    createUser: (_, { name, email }) => {
      const newUser = { id: String(users.length + 1), name, email, posts: [] };
      users.push(newUser);
      return newUser;
    }
  },
  User: {
    posts: (parent) => posts.filter(p => parent.posts.includes(p.id))
  }
};

const server = new ApolloServer({ typeDefs, resolvers });
server.listen().then(({ url }) => console.log(`GraphQL server ready at ${url}`));

This structure centralizes data fetching while giving clients precise control. In distributed systems, GraphQL integrates with API gateways for schema stitching across microservices and rate limiting per operation complexity.

gRPC API Design

gRPC is a high-performance, open-source RPC framework developed by Google that uses HTTP/2 for transport and Protocol Buffers for binary serialization. In system design, gRPC is preferred for internal service-to-service communication in polyglot microservices due to its low latency, built-in streaming, and strong typing.

gRPC contracts are defined in .proto files, which generate client and server stubs in multiple languages. It supports four communication patterns: unary, server streaming, client streaming, and bidirectional streaming. HTTP/2 multiplexing enables concurrent requests over a single connection, with automatic flow control and header compression.

Here is a complete gRPC definition and implementation example using a .proto file for a user service, followed by a Python server snippet:

syntax = "proto3";

package users;

service UserService {
  rpc GetUser (UserRequest) returns (UserResponse);
  rpc ListUsers (stream Pagination) returns (stream UserResponse);
  rpc CreateUser (UserCreateRequest) returns (UserResponse);
  rpc StreamUserUpdates (stream UserUpdate) returns (stream UserResponse);
}

message UserRequest {
  string id = 1;
}

message Pagination {
  int32 page = 1;
  int32 limit = 2;
}

message UserCreateRequest {
  string name = 1;
  string email = 2;
}

message UserResponse {
  string id = 1;
  string name = 2;
  string email = 3;
}

message UserUpdate {
  string id = 1;
  string field = 2;
  string value = 3;
}

Corresponding Python server using grpcio (full implementation):

import grpc
from concurrent import futures
import users_pb2
import users_pb2_grpc

class UserService(users_pb2_grpc.UserServiceServicer):
    def GetUser(self, request, context):
        # Simulate database lookup
        return users_pb2.UserResponse(id=request.id, name="Alice", email="alice@example.com")

    def ListUsers(self, request_iterator, context):
        for pagination in request_iterator:
            # Stream users with pagination
            for i in range(pagination.limit):
                yield users_pb2.UserResponse(id=str(i), name=f"User{i}", email=f"user{i}@example.com")

    def CreateUser(self, request, context):
        return users_pb2.UserResponse(id="new123", name=request.name, email=request.email)

    def StreamUserUpdates(self, request_iterator, context):
        for update in request_iterator:
            yield users_pb2.UserResponse(id=update.id, name="Updated", email="updated@example.com")

server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
users_pb2_grpc.add_UserServiceServicer_to_server(UserService(), server)
server.add_insecure_port('[::]:50051')
server.start()
server.wait_for_termination()

gRPC leverages binary format for smaller payloads and native streaming for real-time scenarios, making it ideal for low-latency distributed systems. Service discovery and load balancing integrate naturally with Kubernetes and Istio.

Choosing the Right API Design in System Design

REST prioritizes simplicity and caching for public-facing APIs where broad client support matters. GraphQL solves data flexibility challenges in frontend-heavy applications and complex data graphs. gRPC delivers superior performance and strict contracts for internal microservices communication, especially under high load or with polyglot teams.

Trade-offs include REST’s verbosity versus GraphQL’s query complexity management and gRPC’s ecosystem maturity. In a complete system, hybrid approaches are common: REST for external clients, GraphQL for mobile/web, and gRPC for backend service meshes. Security considerations—OAuth, JWT, mTLS—apply uniformly, while observability tools like distributed tracing ensure reliability across paradigms.

For a comprehensive guide that builds upon these foundations and covers the full spectrum of system design principles, purchase the System Design Handbook at https://codewithdhanian.gumroad.com/l/ntmcf. Your purchase directly supports the creation of in-depth technical content. Buy me a coffee to support my content at: https://ko-fi.com/codewithdhanian.

Consistency Patterns (Strong, Eventual, Weak) in System Design

CodeWithDhanian — Thu, 02 Apr 2026 20:18:47 +0000

Understanding Consistency in Distributed Systems

In distributed systems, consistency defines how and when updates to data become visible across multiple nodes or replicas. When a client performs a write operation on one node, the system must decide whether subsequent read operations on any other node will immediately reflect that change or tolerate some delay. This decision directly influences availability, latency, throughput, and overall system behavior under network partitions or failures.

Consistency patterns provide structured guarantees that help architects balance these competing requirements. The three primary patterns—Strong Consistency, Eventual Consistency, and Weak Consistency—form a spectrum from the strictest guarantees to the most relaxed. Each pattern addresses different real-world demands, from financial accuracy to massive-scale user-generated content.

Strong Consistency

Strong Consistency, also known as linearizability, guarantees that once a write operation completes successfully, every subsequent read operation—regardless of which replica or client issues it—will return the most recent write value or a newer one. There is no window for stale data. All operations appear to execute in a single, global, sequential order as if on a single atomic copy of the data.

This pattern enforces immediate visibility of updates. If Client A writes a value and receives confirmation, Client B reading immediately afterward will always see the updated value. The system achieves this through tight synchronization mechanisms such as quorum-based replication, consensus protocols like Paxos or Raft, or two-phase commit (2PC) for distributed transactions.

How Strong Consistency Works in Practice

A typical architecture uses a primary node (or leader) that coordinates writes. When a write arrives:

The primary applies the change locally.
It replicates the update synchronously to a quorum of replicas (for example, a majority of nodes in a five-node cluster requires acknowledgment from at least three).
Only after the quorum confirms does the primary acknowledge success to the client.
Any read request, whether directed to the primary or a replica, is served only after verifying it reflects the latest committed state.

This ensures linearizability but introduces latency because writes block until synchronization completes. Failures or network partitions can temporarily reduce availability until consensus is restored.

Complete Implementation Structure and Code Example

Consider a simplified Python simulation of a strongly consistent key-value store using threading locks and a central coordinator to mimic quorum behavior. This demonstrates the core structure used in production systems like Google Spanner or etcd.

import threading
import time
from typing import Dict, Optional

class StrongConsistencyStore:
    def __init__(self, num_replicas: int = 3):
        self.data: Dict[str, str] = {}          # Shared data store
        self.versions: Dict[str, int] = {}      # Version tracking for linearizability
        self.lock = threading.Lock()            # Global lock simulating quorum coordination
        self.replicas = num_replicas
        self.quorum = (self.replicas // 2) + 1  # Majority quorum

    def write(self, key: str, value: str) -> bool:
        with self.lock:  # Simulate synchronous quorum acknowledgment
            current_version = self.versions.get(key, 0)
            self.data[key] = value
            self.versions[key] = current_version + 1
            print(f"Write committed: {key} = {value} (version {self.versions[key]})")
            # In real systems, this would wait for quorum replicas to acknowledge
            time.sleep(0.05)  # Simulate network round-trip latency
            return True

    def read(self, key: str) -> Optional[str]:
        with self.lock:  # All reads go through coordinated check
            if key in self.data:
                value = self.data[key]
                version = self.versions[key]
                print(f"Read returned: {key} = {value} (version {version})")
                return value
            return None

# Usage demonstration
if __name__ == "__main__":
    store = StrongConsistencyStore()
    store.write("account_balance", "1500.00")
    print("Client A writes balance")
    # Immediate read from any "replica" (simulated)
    print("Client B reads balance:", store.read("account_balance"))
    store.write("account_balance", "1400.00")
    print("Client C reads updated balance:", store.read("account_balance"))

This code enforces strong consistency by serializing all operations under a single lock, ensuring every read sees the latest write. In a real distributed deployment, the lock would be replaced by a consensus algorithm that requires quorum acknowledgments from live replicas. The version counter prevents stale reads even if network delays occur.

Use Cases for Strong Consistency

Strong Consistency is essential in domains where correctness cannot be compromised:

Financial systems and banking applications where account balances must reflect every transaction instantly to prevent double-spending.
Inventory management in e-commerce to ensure stock counts are accurate across all regional warehouses.
Reservation systems such as airline seat booking or hotel room allocation.

Trade-offs of Strong Consistency

While it provides perfect data accuracy, Strong Consistency sacrifices availability during partitions (per the CAP theorem) and increases latency due to synchronization overhead. Systems may return errors or block operations rather than serve potentially inconsistent data.

Eventual Consistency

Eventual Consistency relaxes the immediate guarantee. It promises that if no new writes occur to a data item, all replicas will eventually converge to the same latest value after some unspecified but finite time. Temporary divergence is allowed, and reads may return stale data during the propagation window.

This pattern relies on asynchronous replication. Writes are acknowledged quickly after being applied to a single node or a minimal set, then propagated in the background through mechanisms like gossip protocols, anti-entropy, read repair, or hinting handoff.

How Eventual Consistency Works in Practice

A write succeeds as soon as it reaches one or more nodes (often a single node for maximum availability). Background processes then push the update to other replicas. Conflict resolution strategies—such as last-write-wins (LWW) based on timestamps or vector clocks—resolve any concurrent updates when replicas reconcile.

Clients may see different values briefly, but the system self-heals without manual intervention.

Complete Implementation Structure and Code Example

Below is a full Python implementation simulating an eventually consistent store using asynchronous queues and background reconciliation threads. This mirrors the architecture of Amazon DynamoDB (default mode) or Apache Cassandra.

import threading
import time
import queue
from typing import Dict, Optional

class EventualConsistencyStore:
    def __init__(self, num_replicas: int = 3):
        self.replicas: Dict[int, Dict[str, str]] = {i: {} for i in range(num_replicas)}
        self.versions: Dict[int, Dict[str, int]] = {i: {} for i in range(num_replicas)}
        self.write_queue = queue.Queue()
        self.replica_count = num_replicas
        self.lock = threading.Lock()
        # Start background propagation thread
        self.propagator = threading.Thread(target=self._propagate_updates, daemon=True)
        self.propagator.start()

    def write(self, key: str, value: str, replica_id: int = 0) -> bool:
        with self.lock:
            current_version = self.versions[replica_id].get(key, 0)
            self.replicas[replica_id][key] = value
            self.versions[replica_id][key] = current_version + 1
            self.write_queue.put((key, value, self.versions[replica_id][key]))
            print(f"Write acknowledged on replica {replica_id}: {key} = {value}")
            return True

    def read(self, key: str, replica_id: int = 0) -> Optional[str]:
        value = self.replicas[replica_id].get(key)
        version = self.versions[replica_id].get(key, 0)
        print(f"Read from replica {replica_id}: {key} = {value} (version {version})")
        return value

    def _propagate_updates(self):
        while True:
            try:
                key, value, version = self.write_queue.get(timeout=1)
                with self.lock:
                    for rid in range(self.replica_count):
                        current_version = self.versions[rid].get(key, 0)
                        if version > current_version:
                            self.replicas[rid][key] = value
                            self.versions[rid][key] = version
                time.sleep(0.2)  # Simulate network propagation delay
            except queue.Empty:
                continue

# Usage demonstration
if __name__ == "__main__":
    store = EventualConsistencyStore()
    store.write("user_post", "Hello world", replica_id=0)
    print("Client A reads immediately (may be stale):", store.read("user_post", replica_id=1))
    time.sleep(1)  # Allow propagation
    print("Client B reads after convergence:", store.read("user_post", replica_id=1))

The background thread ensures eventual convergence. In production, vector clocks or CRDTs would replace simple version numbers for more sophisticated conflict handling.

Use Cases for Eventual Consistency

Eventual Consistency excels in high-scale, high-availability scenarios:

Social media feeds and like counters where slight delays in visibility are acceptable.
Content delivery networks (CDN) and caching layers.
DNS systems and email propagation.

Weak Consistency

Weak Consistency provides the fewest guarantees. After a write, subsequent reads may or may not see the update, and there is no assurance that replicas will ever converge. Divergence can persist indefinitely unless explicitly resolved.

This model prioritizes raw performance and availability above all else. Updates are fire-and-forget, with no automatic synchronization or conflict resolution built into the core protocol.

How Weak Consistency Works in Practice

Writes are applied locally to whichever node receives them. Reads return whatever local state exists at that moment. Reconciliation, if any, happens only through application-level logic or manual intervention. There are no background propagators or quorum requirements.

Complete Implementation Structure and Code Example

Here is a complete Python simulation of a weakly consistent store with zero synchronization:

import threading
from typing import Dict, Optional

class WeakConsistencyStore:
    def __init__(self, num_replicas: int = 3):
        self.replicas: Dict[int, Dict[str, str]] = {i: {} for i in range(num_replicas)}
        self.lock = threading.Lock()  # Only for internal safety, not for consistency

    def write(self, key: str, value: str, replica_id: int = 0) -> bool:
        with self.lock:
            self.replicas[replica_id][key] = value
            print(f"Write applied only to replica {replica_id}: {key} = {value}")
            return True

    def read(self, key: str, replica_id: int = 0) -> Optional[str]:
        value = self.replicas[replica_id].get(key)
        print(f"Read from replica {replica_id}: {key} = {value} (no convergence guarantee)")
        return value

# Usage demonstration
if __name__ == "__main__":
    store = WeakConsistencyStore()
    store.write("game_score", "150", replica_id=0)
    print("Client reads from replica 1:", store.read("game_score", replica_id=1))  # Likely None
    store.write("game_score", "200", replica_id=2)
    print("Client reads from replica 0:", store.read("game_score", replica_id=0))  # Still 150

No propagation occurs. Each replica remains isolated unless the application adds custom logic.

Use Cases for Weak Consistency

Weak Consistency suits applications where freshness is secondary to speed:

Multiplayer game leaderboards where occasional staleness does not affect gameplay.
Non-critical caching layers or analytics dashboards.
Highly partitioned sensor networks that tolerate data loss.

Comparing the Patterns

Strong Consistency delivers immediate correctness at the cost of higher latency and reduced availability during failures. Eventual Consistency trades immediate accuracy for superior scalability and availability, relying on time for convergence. Weak Consistency maximizes performance by removing all guarantees, making it suitable only when temporary or permanent divergence is tolerable.

Each pattern aligns with specific positions on the CAP spectrum: Strong Consistency typically favors CP systems, while Eventual and Weak Consistency enable AP designs that remain responsive even under partitions.

The choice depends entirely on the application's tolerance for stale data, required throughput, and acceptable failure modes.

System Design Handbook

Master the complete System Design interview process with real-world architectures, deep-dive explanations, and battle-tested patterns used at top tech companies.

Buy it now at: https://codewithdhanian.gumroad.com/l/ntmcf

Buy me coffee to support my content at: https://ko-fi.com/codewithdhanian

SQL vs. NoSQL in System Design

CodeWithDhanian — Thu, 02 Apr 2026 19:51:30 +0000

Foundations of SQL Databases

SQL databases, also known as relational databases, organize data into structured tables with predefined schemas. Each table consists of rows and columns, where columns enforce specific data types and constraints. Relationships between tables are established through foreign keys, enabling complex joins to retrieve interconnected data efficiently. This model follows the principles of normalization to minimize data redundancy and ensure data integrity.

The core strength of SQL lies in its adherence to ACID properties: Atomicity, Consistency, Isolation, and Durability. These guarantees make SQL ideal for applications requiring strict transactional consistency, such as financial systems or e-commerce platforms where partial failures cannot occur.

A complete SQL schema for an e-commerce system illustrates this structure. Consider the following full code snippet using PostgreSQL syntax:

-- Complete SQL schema for e-commerce system
CREATE TABLE users (
    user_id SERIAL PRIMARY KEY,
    username VARCHAR(50) UNIQUE NOT NULL,
    email VARCHAR(100) UNIQUE NOT NULL,
    password_hash TEXT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE products (
    product_id SERIAL PRIMARY KEY,
    name VARCHAR(200) NOT NULL,
    description TEXT,
    price DECIMAL(10,2) NOT NULL,
    stock_quantity INTEGER DEFAULT 0,
    category_id INTEGER REFERENCES categories(category_id)
);

CREATE TABLE categories (
    category_id SERIAL PRIMARY KEY,
    name VARCHAR(100) NOT NULL
);

CREATE TABLE orders (
    order_id SERIAL PRIMARY KEY,
    user_id INTEGER REFERENCES users(user_id) ON DELETE CASCADE,
    order_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    total_amount DECIMAL(10,2) NOT NULL,
    status VARCHAR(20) DEFAULT 'pending'
);

CREATE TABLE order_items (
    order_item_id SERIAL PRIMARY KEY,
    order_id INTEGER REFERENCES orders(order_id) ON DELETE CASCADE,
    product_id INTEGER REFERENCES products(product_id),
    quantity INTEGER NOT NULL,
    price_at_purchase DECIMAL(10,2) NOT NULL
);

-- Example transaction ensuring ACID compliance
BEGIN;
INSERT INTO orders (user_id, total_amount, status) 
VALUES (1, 99.99, 'pending');
INSERT INTO order_items (order_id, product_id, quantity, price_at_purchase) 
VALUES (currval('orders_order_id_seq'), 5, 2, 49.99);
UPDATE products SET stock_quantity = stock_quantity - 2 WHERE product_id = 5;
COMMIT;

This structure enforces referential integrity through foreign keys and cascading deletes. The transaction block guarantees that either all operations succeed or none do, maintaining consistency even under concurrent access. In system design, such SQL setups are typically deployed with master-slave replication for read scalability and vertical scaling by upgrading hardware on a single server.

Foundations of NoSQL Databases

NoSQL databases, or non-relational databases, reject the rigid table structure in favor of flexible data models. They store data in formats such as documents, key-value pairs, wide-column stores, or graphs. Schemas are often dynamic or schema-less, allowing applications to evolve without downtime for migrations. This flexibility prioritizes scalability and performance over strict consistency.

NoSQL systems typically follow BASE principles: Basically Available, Soft state, and Eventual consistency. They excel in horizontal scaling across distributed clusters, making them suitable for high-throughput applications like social media feeds, real-time analytics, or content management systems.

NoSQL encompasses four primary types:

Document stores (e.g., MongoDB) store self-contained JSON-like documents.
Key-value stores (e.g., Redis) provide ultra-fast lookups.
Column-family stores (e.g., Cassandra) handle massive sparse data.
Graph databases (e.g., Neo4j) optimize for relationship-heavy queries.

A complete NoSQL implementation example uses MongoDB for the same e-commerce scenario. The following full code snippet demonstrates document-based storage and operations:

// Complete MongoDB setup and operations for e-commerce system
const { MongoClient } = require('mongodb');
const uri = "mongodb://localhost:27017";
const client = new MongoClient(uri);

async function run() {
  try {
    await client.connect();
    const database = client.db("ecommerce");

    // Collections are created implicitly on first insert
    const usersCollection = database.collection("users");
    const productsCollection = database.collection("products");
    const ordersCollection = database.collection("orders");

    // Insert a user document (schema-less)
    const newUser = {
      username: "johndoe",
      email: "john@example.com",
      passwordHash: "hashedpassword123",
      createdAt: new Date(),
      addresses: [  // Embedded array for flexibility
        { street: "123 Main St", city: "Nairobi", country: "Kenya" }
      ]
    };
    const userResult = await usersCollection.insertOne(newUser);

    // Insert product with dynamic fields
    const newProduct = {
      name: "Wireless Headphones",
      description: "Noise-cancelling over-ear",
      price: 99.99,
      stockQuantity: 50,
      category: "Electronics",
      tags: ["audio", "wireless"]  // Flexible array
    };
    await productsCollection.insertOne(newProduct);

    // Complete order as a single document with embedded items (denormalized for speed)
    const newOrder = {
      userId: userResult.insertedId,
      orderDate: new Date(),
      totalAmount: 99.99,
      status: "pending",
      items: [
        {
          productId: "productObjectIdHere",
          name: "Wireless Headphones",
          quantity: 1,
          priceAtPurchase: 99.99
        }
      ],
      // Embedded shipping info without separate table
      shippingAddress: {
        street: "123 Main St",
        city: "Nairobi",
        country: "Kenya"
      }
    };
    await ordersCollection.insertOne(newOrder);

    // Query example with aggregation for analytics
    const analytics = await ordersCollection.aggregate([
      { $match: { status: "completed" } },
      { $group: { _id: null, totalRevenue: { $sum: "$totalAmount" } } }
    ]).toArray();

    console.log("Analytics:", analytics);
  } finally {
    await client.close();
  }
}
run().catch(console.dir);

This code demonstrates denormalization by embedding related data directly into documents, eliminating the need for joins. Operations occur atomically within a single document, and the system scales horizontally by adding more nodes to the replica set or sharded cluster. In system design, MongoDB would use consistent hashing for data partitioning and replication for high availability.

Comparative Analysis in System Design

SQL and NoSQL differ fundamentally in how they handle data, scale, and guarantee consistency, directly impacting architectural decisions.

Aspect	SQL Databases	NoSQL Databases
Data Model	Relational tables with fixed schema	Flexible documents, key-value, etc.
Schema	Rigid and enforced	Dynamic or schema-less
Query Language	Standardized SQL with joins	Database-specific (e.g., MongoDB Query Language)
Consistency	ACID (strong guarantees)	BASE (eventual consistency)
Scalability	Vertical scaling preferred	Horizontal scaling across clusters
Use Case Fit	Transactions, complex relationships	High volume, unstructured data, real-time

SQL shines when data integrity and complex queries are paramount. NoSQL excels when velocity, variety, and volume dominate, as in big data pipelines or global user bases.

In a distributed system design, a hybrid approach often emerges. User authentication and financial transactions might reside in PostgreSQL for ACID compliance, while user activity logs and product recommendations use Cassandra or MongoDB for horizontal scaling and eventual consistency. Data partitioning via sharding in NoSQL contrasts with partitioning strategies in SQL that rely more on read replicas.

Practical Implementation in Distributed Systems

System designers must evaluate trade-offs when selecting databases. For a high-traffic social platform, SQL might handle user profiles with strong consistency via two-phase commits in rare cross-service transactions. NoSQL would manage feeds using event-driven architecture with message queues publishing changes for eventual propagation.

Leader election and consensus algorithms like Raft ensure NoSQL clusters remain available during node failures. SQL clusters often employ multi-master replication with careful conflict resolution.

The choice influences microservices architecture: each service owns its database, enforcing database-per-service patterns. API gateways route requests, while circuit breakers and retry mechanisms handle transient failures across database boundaries.

This comprehensive understanding equips system designers to architect resilient, scalable solutions tailored to specific requirements.

To help visualize the concepts discussed, here is one complete image:

System Design Handbook

To master these concepts and many more, purchase the complete System Design Handbook at https://codewithdhanian.gumroad.com/l/ntmcf.

Buy me coffee to support my content at: https://ko-fi.com/codewithdhanian