Forem: Gabriel Anhaia

Request Hedging: The Tail-at-Scale Technique Most Teams Skip

Gabriel Anhaia — Sun, 24 May 2026 20:30:28 +0000

Book: System Design Pocket Guide: Fundamentals — Core Building Blocks for Scalable Systems
Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Your p99 is 2 seconds. Your p50 is 80 milliseconds. That gap is what wakes you up. It's also what your SLO actually measures, and the tricks you've been reaching for (bigger pods, more replicas, a fatter cache) barely move it.

Jeff Dean and Luiz Barroso wrote about this in 2013 in a paper called The Tail at Scale (CACM, Vol. 56, No. 2). The headline finding: in a fanout system where every request touches 100 backends, a per-backend p99 of 10ms produces an end-to-end p99 closer to 140ms. Tail latency multiplies. You can't outrun that by tuning the average.

One of the techniques they shipped at Google to fix it (request hedging) is dead simple, well understood, and somehow still missing from most production codebases in 2026. So let's put it back on the table.

The tail-at-scale problem still applies

The math hasn't changed since 2013. If a single backend has a 1% chance of being slow on any given request, a request that fans out to 100 backends has a 1 - (0.99)^100 ≈ 63% chance of hitting at least one slow backend. Your end-to-end latency is the max across all backends, not the average.

The same effect shows up in non-fanout systems too. A user-facing request that depends on three serial calls (auth, profile, recommendations) sees its p99 stack. Each hop has its own bad day, and the bad days line up more often than your intuition expects.

Causes haven't changed either: GC pauses, hot keys, kernel scheduling glitches, contended locks, noisy neighbors on shared hardware, NIC queue backpressure. You can fix individual sources, but you'll never zero them out. The tail is structural.

That's why Dean and Barroso framed the solution as latency-tolerant techniques rather than latency-reduction tricks. Hedging is the one you should ship first.

What hedging actually is

The pattern: send the request to backend A. If you haven't gotten a response by the time you'd expect a slow-but-not-broken reply (say, your p95), send the same request to backend B. Take whichever response comes back first. Cancel the loser.

That's it. No retries on failure. No queue. You're betting that the slow tail of one backend isn't correlated with the slow tail of another, which is usually true when the slowness comes from GC, kernel, or local hot-spot causes.

The Dean/Barroso paper reports that hedging at the 95th percentile reduced their BigTable lookup p99 from 1800ms to 74ms, while only inflating total backend work by about 2%. Those numbers depend on your workload, but the shape holds.

The cost of hedging

You're sending extra requests, so you're doing extra work. The question is how much.

If you hedge at the p95, then by definition only 5% of requests trigger a second call. Of those, the second call usually completes faster than the first would have, but you still pay for one extra request 5% of the time. That's where the 5–10% overhead figure comes from, and it assumes you cancel the loser fast enough that the loser stops working as soon as the winner returns.

If you hedge at the p50, you've doubled your backend traffic. Don't do that.

If you hedge at the p99 you're too late. The user has already noticed.

The threshold matters. Measure it from production data, not from a load test.

A working Go implementation

The minimum viable Go version uses two goroutines, a buffered channel, and context.WithCancel. Real, no foo/bar:

package hedged

import (
    "context"
    "errors"
    "time"
)

type Result struct {
    Body []byte
    Err  error
}

// Fetch issues the primary request, then fires a hedge after p95 latency.
// The first non-error response wins. The loser is cancelled.
func Fetch(parent context.Context, p95 time.Duration, call func(context.Context) ([]byte, error)) ([]byte, error) {
    ctx, cancel := context.WithCancel(parent)
    defer cancel() // always cancel the loser, plus tear down on parent cancel

    results := make(chan Result, 2) // buffered so the loser's send never blocks

    go func() {
        body, err := call(ctx)
        results <- Result{body, err}
    }()

    hedgeTimer := time.NewTimer(p95)
    defer hedgeTimer.Stop()

    select {
    case r := <-results:
        // primary returned before the hedge fired
        return r.Body, r.Err
    case <-hedgeTimer.C:
        // primary is slow; fire the hedge
    case <-ctx.Done():
        return nil, ctx.Err()
    }

    go func() {
        body, err := call(ctx)
        results <- Result{body, err}
    }()

    // take the first response that isn't a context-cancel artefact
    for i := 0; i < 2; i++ {
        select {
        case r := <-results:
            if r.Err == nil {
                return r.Body, nil
            }
            // loser may have errored because we cancelled it; ignore and wait
            if errors.Is(r.Err, context.Canceled) {
                continue
            }
            return nil, r.Err
        case <-parent.Done():
            return nil, parent.Err()
        }
    }
    return nil, errors.New("hedged: both attempts failed")
}

The defer cancel() line does the heavy lifting. The moment the function returns (winner found, error raised, or parent cancelled), the still-in-flight loser sees its context die and should abort whatever it was doing. If call is a net/http request built with http.NewRequestWithContext, the TCP socket closes within microseconds. That's how you keep the overhead at single-digit percent instead of double.

One subtle bit: the results channel is buffered to 2 so the loser's send doesn't deadlock when nobody's reading. Skip that and you leak goroutines.

A working Python implementation

Same pattern in asyncio. The trick is asyncio.wait with FIRST_COMPLETED, then explicit cancellation of the pending task:

import asyncio
from typing import Awaitable, Callable, TypeVar

T = TypeVar("T")

async def hedged_call(
    p95: float,
    call: Callable[[], Awaitable[T]],
) -> T:
    primary = asyncio.create_task(call())

    # wait up to p95 for the primary
    done, _pending = await asyncio.wait({primary}, timeout=p95)
    if primary in done:
        return primary.result()

    # primary is slow; fire the hedge
    hedge = asyncio.create_task(call())
    done, pending = await asyncio.wait(
        {primary, hedge},
        return_when=asyncio.FIRST_COMPLETED,
    )

    # cancel the loser; await it so we don't leak the task
    for task in pending:
        task.cancel()
    for task in pending:
        try:
            await task
        except (asyncio.CancelledError, Exception):
            pass  # loser exceptions are uninteresting

    winner = next(iter(done))
    return winner.result()

If call() wraps an httpx.AsyncClient.get, task.cancel() propagates down and closes the connection. If it wraps a requests.get running in a thread executor, cancellation is advisory and the thread keeps working, which is exactly the case where hedging makes your load worse, not better. Use async clients all the way down or skip hedging at this layer.

Hedging at the load balancer

You don't have to write any of the above if your traffic flows through Envoy. The hedge_policy on a route does it for you:

route_config:
  name: profile_service
  virtual_hosts:
    - name: profile
      domains: ["profile.internal"]
      routes:
        - match: { prefix: "/" }
          route:
            cluster: profile_cluster
            timeout: 1s
            retry_policy:
              retry_on: "5xx,reset"
              num_retries: 2
              per_try_timeout: 0.4s
            hedge_policy:
              initial_requests: 1
              additional_request_chance:
                numerator: 100
                denominator: HUNDRED
              hedge_on_per_try_timeout: true

The hedge_on_per_try_timeout: true flag is the one that matters. When the per-try timeout (400ms here) fires, Envoy issues a second request to a different upstream and races them. Pair this with a per_try_timeout set to your measured p95 and you've got hedging without writing a line of Go or Python.

Istio inherits this through its Envoy dataplane. If you're on a service mesh you may already have the primitive sitting there, unused.

The application-layer version is more flexible. You can hedge selectively based on request type, vary the threshold per endpoint, or hedge against a different backend entirely. The mesh version is easier to ship and harder to get wrong. Pick based on whether you need that flexibility.

When NOT to hedge

A short list of places hedging will hurt you:

Non-idempotent operations. POST /payments, PUT /counter/increment, anything that mutates state. Two requests means two payments. There is no clever idempotency-key trick that makes this safe by default. You have to engineer the dedup explicitly and you usually shouldn't bother.
Expensive backends. If each call costs you GPU seconds or a $0.02 LLM token bill, hedging means paying twice for 5% of requests. Do the math on your unit economics.
Backends with their own cascading retries. If service B retries service C three times before returning, your hedge fires a second tree of retries. The amplification gets ugly fast.
Stateful sessions. WebSockets, gRPC streaming, anything sticky. Hedging assumes the request is a pure function of its input.

The rule of thumb: hedge read paths that hit shared backends. Don't hedge anything else.

The outage gotcha: pair it with a circuit breaker

This is the part everyone misses, and it's the part that turns hedging from a tail-latency win into an outage amplifier.

When a backend gets unhealthy, every request crosses your p95 threshold. Every request triggers a hedge. You've just doubled your traffic to a backend that was already failing. The hedges hit the same dying instances. They time out too. You retry. The cluster dies faster.

Dean and Barroso flagged this in the paper. The mitigation is non-optional: hedging must be coupled with a circuit breaker that opens when the failure rate or hedge rate crosses a ceiling.

Concretely:

Track your hedge rate. If it exceeds, say, 15% over a 10-second window, stop hedging entirely until it drops below 10%. The percentages are workload-specific; pick numbers that mean "this isn't tail latency, this is an outage."
Track downstream failure rate. Standard circuit-breaker behaviour: open on errors, half-open to probe, close on recovery. The hedge logic should only fire when the breaker is closed.
Cap concurrent hedged requests. A bounded semaphore that drops the hedge attempt (not the primary) when saturated.

In code, the simplest form wraps the hedge call in a breaker check:

func FetchSafe(ctx context.Context, p95 time.Duration, call func(context.Context) ([]byte, error)) ([]byte, error) {
    if !breaker.Allow() {
        // circuit open: skip hedging entirely, just run the primary
        return call(ctx)
    }
    body, err := Fetch(ctx, p95, call)
    breaker.Record(err == nil)
    return body, err
}

Envoy's hedge policy doesn't pair this for you automatically. You configure the outlier detection separately, and you need both: outlier detection ejects bad upstreams, hedging covers the tail of the healthy ones.

What to ship this week

Measure your real p95 per endpoint. Pick the three highest-fanout read paths in your system. Add hedging at the p95 threshold with a circuit breaker. Watch your p99 drop and your overhead stay under 10%.

If you're on Envoy or Istio, ship the hedge_policy first — it's a config change, no code. If you're in a service that calls a database or a downstream API directly, ship the application-layer version with proper context cancellation. Either way, instrument the hedge rate as a first-class metric. The day it spikes is the day you'll be glad you wired up the breaker.

The 2013 paper called this "good enough" engineering against unavoidable variability. Twelve years on, that's still the right framing. You can't make the tail go away. You can race it.

What's the p99/p50 gap on your hottest read path right now, and what's stopping you from shipping a hedge against it this week?

If this was useful

Hedging sits in a family of patterns (timeouts, retries, circuit breakers, load shedding, backpressure) that decide whether your system survives its own scale. The System Design Pocket Guide: Fundamentals walks through the trade-offs in the latency and reliability chapters, with the same level of "show me the actual config" that this post tried to hit.

Pull-Based vs Push-Based Architecture: The Choice That Decides Your Reliability Story

Gabriel Anhaia — Sun, 24 May 2026 20:30:04 +0000

Book: System Design Pocket Guide: Fundamentals — Core Building Blocks for Scalable Systems
Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Pull-based and push-based aren't just delivery styles. They're reliability stories. One absorbs producer spikes; the other absorbs consumer outages. Picking the wrong one means your most predictable failure shape becomes your worst.

Most teams pick by accident. Someone says "we need webhooks," everyone nods, and three months later a marketing campaign fires a million events at a receiver that was sized for a quiet Wednesday. Or someone picks Kafka because it's on the resume bingo card, and now a tiny webhook integration runs through a 7-broker cluster with a 4-week retention policy.

The right way to pick is to ask: which failure shape can I afford?

Push and pull, in failure-shape terms

A push system has the producer drive the data. The producer decides when to send, how much to send, and where to send it. The consumer is the passenger. When the producer is calm, life is easy. When the producer panics (a viral campaign, a backfill job, a fan-out from a single upstream event) the consumer absorbs the punch.

A pull system inverts the relationship. The consumer drives. It decides when to fetch, how much to fetch, and how fast to drain. The producer writes to a buffer (a log, a queue, a table) and walks away. When the consumer is slow, work piles up in the buffer. When the consumer dies, work waits.

That's the whole tradeoff. Push gives you low latency and amplifies producer spikes. Pull gives you natural backpressure and survives consumer outages.

Everything else in this post is a footnote on that one sentence.

Push: webhooks, server-sent events, server-initiated jobs

The push family is broader than people think. Webhooks are push. Server-sent events are push. WebSocket fan-out is push. A cron job firing HTTP POSTs at downstream services is push. AWS SNS topics that fan out to HTTPS endpoints are push.

What unites them: the producer decides the timing, and the network round-trip happens at production time.

The good news is latency. A push event lands at the consumer milliseconds after it's produced. That's why payment processors use webhooks for payment.succeeded. The merchant wants to update the order page right now, not in 30 seconds when a poller wakes up.

The bad news is that the consumer has no say. If the producer fires 50,000 webhooks per second and the consumer can absorb 5,000, the producer doesn't know or care. The 45,000 overflow becomes failed deliveries, retries, or both. Stripe, for example, retries failed webhooks with exponential backoff for up to 3 days, but most teams' receivers fall over long before the retries help.

There's no shared buffer between producer and consumer in pure push. The network is the buffer, and the network is bad at buffering.

Pull: polling consumers, Kafka-style log readers, batch jobs

Pull means the consumer initiates. A SQS consumer that calls ReceiveMessage every second. A Kafka consumer group that calls poll() and gets a batch back. A nightly ETL job that scans a transactions table where created_at > last_run. All pull.

The key property: there's a buffer between producer and consumer, and the consumer chooses its rate.

Here's the minimum viable Kafka consumer in Python. Production code, not pseudocode:

from confluent_kafka import Consumer, KafkaError
import json
import logging

log = logging.getLogger(__name__)

def make_consumer(group_id: str) -> Consumer:
    return Consumer({
        "bootstrap.servers": "kafka-1:9092,kafka-2:9092",
        "group.id": group_id,
        "enable.auto.commit": False,        # we commit after work succeeds
        "auto.offset.reset": "earliest",
        "max.poll.interval.ms": 300_000,    # 5 min — kicks slow consumers out
        "session.timeout.ms": 45_000,
    })

def run(topic: str, group_id: str, handle):
    c = make_consumer(group_id)
    c.subscribe([topic])
    try:
        while True:
            msg = c.poll(timeout=1.0)
            if msg is None:
                continue
            if msg.error():
                # partition EOF is fine, anything else we log and continue
                if msg.error().code() != KafkaError._PARTITION_EOF:
                    log.error("poll error: %s", msg.error())
                continue
            try:
                handle(json.loads(msg.value()))
                c.commit(message=msg, asynchronous=False)
            except Exception:
                # don't commit — message will be redelivered after rebalance
                log.exception("handler failed for offset %d", msg.offset())
    finally:
        c.close()

Notice what this consumer controls: poll timing, batch size (via the broker config), commit timing, and what "processed" means. The producer doesn't see any of this. If the consumer dies for 6 hours, Kafka holds the messages (default retention is a week, but you can keep them for years). When the consumer comes back, it picks up at the last committed offset and drains the backlog.

That's the magic of pull. The buffer absorbs the consumer's downtime.

How they handle producer spikes (pull absorbs, push amplifies)

This is the failure mode that surprises teams most.

Imagine a checkout system that emits an order.placed event for every order. On a normal day, 100 orders per second. Black Friday morning, 8,000 per second for 90 seconds, then back to normal.

In a pull system with Kafka or SQS in front of the consumers, the buffer takes the spike. The consumer group sees the backlog rising and drains it at whatever rate it can sustain. End-to-end latency goes from 50ms to maybe 30 seconds during the spike. Nothing breaks. No alerts.

In a push system, the producer tries to POST 8,000 webhooks per second at the consumer's /webhook endpoint. The consumer's connection pool runs out. New connections queue. The load balancer's queue fills. Requests start timing out at 30 seconds. The producer's retry logic kicks in and now we have 8,000 original requests plus retries, hammering an already-overloaded receiver. This is the classic retry storm, and it's how short producer spikes turn into multi-hour outages.

The fix in push systems is rate limiting at the producer (Stripe caps webhook concurrency per endpoint), backoff schedules, and dead letter queues for permanently-failed deliveries. All real, all expensive to build, all things teams discover after their first incident.

Pull doesn't need any of that. The buffer is the rate limiter.

How they handle consumer outages (push loses, pull catches up)

The mirror image of the spike scenario is the outage scenario.

Your consumer is down for 2 hours. Database migration, deploy bug, whatever.

In a pull system, nothing happens to the producer. It keeps writing to Kafka. Messages accumulate in the partitions. When the consumer comes back, it sees the unprocessed offsets and works through them. If you provisioned enough partitions to allow parallelism, you can catch up in minutes by scaling the consumer group temporarily.

In a push system, every webhook delivery during those 2 hours fails. The producer's retry policy decides what happens next. Stripe retries for 3 days; AWS SNS retries with limits configurable per subscription; GitHub gives up after a few attempts and writes the failure to an admin page nobody looks at. If your retry window is shorter than your outage, those events are gone unless the producer is willing to backfill on request — and most aren't.

This is why "we use webhooks for everything" usually becomes "we use webhooks plus a daily reconciliation job that pulls the truth from the source-of-record API." Teams end up building pull on top of push because push alone loses events.

Backpressure mechanisms per style

Pull has backpressure for free. The consumer doesn't poll faster than it can process. The buffer grows during slow periods and shrinks during fast ones. The producer never knows.

Push needs explicit backpressure, and it's awkward.

The HTTP-level mechanism is 429 Too Many Requests plus a Retry-After header. The producer is supposed to read this and back off. Some producers respect it (Stripe, GitHub). Many don't (your in-house service that fires HTTP from a cron job). And 429 only works if your service can still respond to the producer with a 429 — which means it's not actually overloaded, just selectively rejecting.

The transport-level mechanism is connection limits. Set your reverse proxy to accept N concurrent connections, no more. Past that, requests get rejected at the LB. The producer sees connection failures and (if well-behaved) retries with backoff.

Neither of these is as clean as "the queue grew by 10,000 messages." That's why senior engineers reach for pull whenever the work doesn't strictly need millisecond latency.

Real systems are usually hybrid: push for notifications, pull for processing

Once you've stared at the tradeoff long enough, you stop picking one. You pick both.

The pattern: push a tiny notification, pull the heavy data.

# producer side — the push is a 200-byte heads-up
def on_order_placed(order_id: str):
    db.insert("orders", order_id=order_id, status="placed", ...)
    notify_webhook(
        url=subscriber.webhook_url,
        body={"event": "order.placed", "order_id": order_id},
    )

# consumer side — push handler just records intent, pulls the rest
@app.post("/webhooks/orders")
async def receive(req: Request):
    event = await req.json()
    # idempotent enqueue keyed on order_id
    await tasks.enqueue("fetch_order", order_id=event["order_id"])
    return {"ok": True}, 202

# worker — pulls from the queue, fetches the full record at its own pace
async def fetch_order(order_id: str):
    order = await api.get(f"/orders/{order_id}")  # producer's read API
    await db.upsert("local_orders", **order)

Webhook delivery is now a 200-byte ping. The receiver's job is to write the event ID into a queue and return 202 in under 50ms. The actual processing happens on a worker pool that drains the queue at whatever rate is healthy. If the worker pool goes down, the queue grows; no webhook deliveries fail.

Stripe's docs explicitly recommend this pattern. The webhook is a notification, not a payload. The full source of truth is the producer's read API, which you can call at your own cadence.

This is also how Kafka Connect, Debezium CDC pipelines, and most production event systems work. The CDC connector pushes a small notification to Kafka; consumers pull. Nobody pushes a 10MB payload through 14 hops of HTTP.

The gotcha: webhooks need retry + idempotency on the receiver; most teams under-engineer this

Here's where most teams ship the bug that bites them six months in.

Webhooks retry. Stripe retries. GitHub retries. SNS retries. Your in-house webhook producer retries because you copied the pattern from Stripe's docs. Retries mean the same logical event arrives at your receiver multiple times. If your handler isn't idempotent, you charge the customer twice, you send two emails, you double the inventory decrement.

The minimum viable receiver looks like this:

from fastapi import FastAPI, Request, HTTPException
import hmac, hashlib, asyncpg

app = FastAPI()

@app.post("/webhooks/payments")
async def receive(req: Request):
    body = await req.body()
    sig = req.headers.get("Stripe-Signature", "")

    # 1. verify signature — reject forgeries before any DB touch
    if not verify(body, sig, WEBHOOK_SECRET):
        raise HTTPException(401, "bad signature")

    event = json.loads(body)
    event_id = event["id"]   # Stripe gives you a unique id per event

    # 2. idempotency — write the event id first, in its own txn
    async with pool.acquire() as conn:
        try:
            await conn.execute(
                "INSERT INTO processed_events (id, received_at) VALUES ($1, NOW())",
                event_id,
            )
        except asyncpg.UniqueViolationError:
            # already processed — return 200 so the producer stops retrying
            return {"status": "duplicate"}

    # 3. enqueue actual work — don't process inline
    await queue.enqueue("handle_payment_event", event_id=event_id, payload=event)

    # 4. ack within the producer's timeout (usually 30s)
    return {"status": "queued"}

Three things every webhook receiver needs and most don't have:

Signature verification before anything else. The receiver is on the public internet. Anyone can POST to it. If you write to the DB before checking the signature, you've built a public RCE on your event handler.

Idempotency on the event ID, persisted in your own store. Not Redis (it evicts), not in-memory (it resets on deploy). A real table with a unique constraint. The INSERT ... ON CONFLICT DO NOTHING pattern or a try-catch on the unique violation. The point is: the second delivery of the same event should be a no-op.

Decouple receipt from processing. Return 2xx fast, do the work async. If you process inline, a slow downstream call makes the producer think you failed and retry, turning one event into ten attempts. The receiver becomes a 50ms-budget endpoint whose only job is to validate, dedupe, and enqueue.

The thing that keeps biting people: idempotency keys on Stripe's side don't help here. Those protect your API calls to Stripe from being processed twice. They do nothing for webhook deliveries flowing the other way. Different problem, different key.

Picking the right side

A small checklist for your next system.

Pick push when: latency under one second matters, payloads are small, the consumer is reachable and well-sized, you control or trust the producer's retry behavior, and you're willing to invest in idempotency + signature + DLQ on the receiver.

Pick pull when: throughput is bursty, the consumer might be slow or down, you want replay, you want fan-out to multiple consumer groups at different speeds, or latency above a few seconds is acceptable.

Pick hybrid when: you want push's latency but pull's reliability, or when the payload is large and only some consumers care about the full record. This is the production default for a reason.

The mistake to avoid is treating delivery style as a taste preference. It's a reliability decision. The day your traffic doubles or your consumer crashes, the architecture you picked will either absorb the event or become the headline of the incident report.

If this was useful

Pull vs push is one of those choices that looks like plumbing and turns out to be load-bearing. The System Design Pocket Guide: Fundamentals walks through the same tradeoff lens for the rest of the core building blocks (queues, caches, replication, consistency) so the next time you sketch a system on a whiteboard, you're picking failure shapes on purpose instead of by accident.

What's the worst push-vs-pull mismatch you've inherited, and what did the cleanup actually look like?

Service Mesh in 2026: When Istio Is Overkill, When It's the Right Answer

Gabriel Anhaia — Sun, 24 May 2026 15:21:06 +0000

Book: System Design Pocket Guide: Fundamentals — Core Building Blocks for Scalable Systems
Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

A platform team I talked to last quarter spent four months rolling out Istio across a 30-service estate. They wanted three things: encrypted service-to-service traffic, retries handled outside application code, and a single dashboard showing who calls whom. Six months after go-live, they ripped it out. Not because it didn't work. Because the win was a fraction of the operational cost they paid for it.

That story isn't rare. It's the modal outcome for teams under a hundred services who adopt a heavyweight mesh because a conference talk made it sound mandatory. Service mesh is a real architectural pattern with a real ROI shape. In 2026 the answer to "do we need one" got both clearer and more interesting, because the alternatives finally caught up.

What a service mesh actually does

Strip away the marketing and a mesh is four features bundled into one data plane:

Mutual TLS between services. Every pod gets an identity, every connection is encrypted, certificate rotation happens automatically.
Traffic policy. Timeouts, retries, circuit breakers, weighted routing for canaries, all declared once and enforced at the proxy.
L7 observability. Per-RPC latency, status codes, request rate, broken down by source and destination service, without any app-side instrumentation.
Multi-cluster gateway logic. Cross-cluster service discovery and routing without bespoke gateway code in each app.

You get those features by injecting a proxy (Envoy, in most cases) next to every workload. That sidecar is doing the work. The control plane (Istio's istiod, Linkerd's controller) configures the sidecars and rotates certs.

The cost is the sidecar itself. Memory per pod doubles or worse. Network hops add latency. The control plane has its own SRE story. And every CRD-laden Helm chart is a thing your team now operates.

That cost is what makes the question "do we need this" harder than the conference talks admit.

Istio in 2026: what changed

Istio's biggest 2024–2025 story was ambient mode going GA. Ambient swaps the per-pod Envoy sidecar for a per-node L4 proxy called ztunnel, plus an optional per-namespace L7 proxy (waypoint). The net effect: mTLS and basic identity are cheap, only namespaces that need L7 policy pay for an Envoy.

The pre-ambient Istio model was the operational equivalent of a tank: capable, heavy, and the reason most teams shipped Linkerd instead. Ambient changes that calculus. If you're starting fresh in 2026 and you do want Istio, ambient is the default mode you should be evaluating, not the legacy sidecar mode.

What ambient doesn't fix: the CRD surface area, the rate of breaking changes between minor versions, and the operational expertise required to debug an Envoy config that misbehaves under load. Istio is more approachable than it was. It isn't yet what anyone calls easy.

The alternatives, briefly

Linkerd is the answer for most teams that decide they need a mesh but don't want to operate Istio. The proxy is written in Rust, the control plane is small, the default configuration is sane. The Buoyant team's stance is that simplicity is a feature, and the project's history shows it. You give up some of Istio's L7 routing expressiveness; you get an operational story most platform teams can actually staff.

A minimal Linkerd route policy looks like this:

apiVersion: policy.linkerd.io/v1beta3
kind: HTTPRoute
metadata:
  name: checkout-canary
  namespace: payments
spec:
  parentRefs:
    - name: checkout-svc
      kind: Service
      group: core
      port: 8080
  rules:
    - matches:
        - headers:
            - name: x-release-channel
              value: canary
      backendRefs:
        - name: checkout-svc-v2
          port: 8080
          weight: 100
    - backendRefs:
        - name: checkout-svc-v1
          port: 8080
          weight: 100

---

apiVersion: policy.linkerd.io/v1alpha1
kind: HTTPLocalRateLimitPolicy
metadata:
  name: checkout-limits
  namespace: payments
spec:
  targetRef:
    group: core
    kind: Service
    name: checkout-svc
  total:
    requestsPerSecond: 500
  overrides:
    - requestsPerSecond: 50
      clientRefs:
        - kind: ServiceAccount
          name: marketing-batch
          namespace: growth

Read that and the mental model is the same as Kubernetes' own Gateway API: parents, matches, backends. No EnvoyFilter escape hatches, no second config language to learn. That is the whole Linkerd pitch.

Cilium service mesh is the option that changes the architecture, not just the implementation. Cilium uses eBPF in the kernel to do what an Envoy sidecar does in user space. mTLS, L7 policy, observability, all without a sidecar process per pod. For teams already running Cilium as their CNI, turning on mesh features is a config flag, not a new deployment topology.

The Cilium trade is real: your mesh logic now lives in the kernel and depends on a recent kernel version, your debugging story shifts from "exec into the sidecar" to "read cilium monitor", and the L7 features still aren't as deep as Istio's. But if your platform team already knows eBPF, Cilium is the lowest-overhead path to mesh features.

Consul Connect is the right answer if you already run Consul for service discovery. Otherwise it's rarely the first choice in a Kubernetes-native shop.

No mesh, with Envoy at the edge. This is what most teams should actually consider first. An ingress gateway running Envoy or Contour gives you north-south policy. Your apps emit OpenTelemetry traces. mTLS, if you need it, comes from cert-manager + an SDK like SPIFFE/SPIRE. You pay zero per-pod sidecar tax. The ceiling is lower, but most teams never approach the ceiling.

The three questions that decide it

When a team asks me whether they should adopt a service mesh, the conversation collapses to three questions. Two yeses is the bar. One yes means you're reaching for a mesh because of a different unsolved problem.

Question	If yes, why a mesh helps	If no, what to do instead
Is mTLS between services a hard requirement (regulator, security review, zero-trust mandate)?	The mesh issues per-workload identities and rotates certs automatically. Doing this yourself with cert-manager + SPIRE works but the operational cost is non-trivial.	Stick with TLS at the edge and network policies. Per-pod mTLS without an external requirement is a tax most teams don't recover.
Do you need cross-team traffic policies (rate limits, retries, timeouts) enforced uniformly without trusting every team to ship them in code?	The mesh moves the contract out of app code into a platform layer the SRE team owns. This is the killer feature for orgs with 50+ services and a platform team.	Document timeouts in the SDK, fail PR review when they're missing. Cheaper than running a mesh until you scale past trust.
Are you running multi-cluster with cross-cluster service-to-service traffic that needs identity-aware routing?	The mesh's gateway and identity model handles this without you writing cluster-aware DNS hacks.	A regional gateway + a service registry is enough for most setups. Multi-cluster mesh is the heaviest deployment of any of these tools.

Two yeses justifies a mesh. Three makes the cost-benefit obvious. One yes means you have a specific gap you can probably close with a smaller tool.

The false positive that traps everyone

The single most common reason teams reach for a mesh is observability. "We can't see what our services are doing to each other." Then they install Istio, get a Kiali dashboard, and feel productive.

Here's the problem with that path: a mesh gives you L7 telemetry at the proxy. That telemetry is per-service, not per-request-flow. You see that service A called service B 1,200 times in the last minute with p99 latency of 380ms. You can't, from mesh telemetry alone, tell which user request that traffic served, which downstream calls fanned out from it, or which code path was responsible.

The thing you actually want is distributed tracing. That is OpenTelemetry, not a mesh. Instrument your apps with the OTel SDKs, propagate the W3C traceparent header through your HTTP and gRPC calls, ship spans to your tracing backend. A mesh can decorate those spans with mTLS identity, but the trace pipeline is what answers the questions you actually have.

If your reason for wanting a mesh is "I want to see what's happening," the cheaper, better answer is OpenTelemetry first. Adopt a mesh later, if and when the three questions above shift from no to yes.

When ambient mode changes the math

Ambient Istio is interesting because it decouples the cost of mTLS from the cost of L7 features. With sidecar Istio, you paid the full Envoy memory and CPU bill for every workload, even ones that only needed encrypted transport and basic telemetry. With ambient, the per-node ztunnel handles mTLS and L4 telemetry at a fraction of the cost. Waypoint proxies (per namespace, opt-in) handle L7 policy only where you actually need it.

If the answer to question one (mTLS required) is yes but the answer to question two (cross-team policy) is no, ambient Istio becomes a viable shape where full sidecar Istio wasn't. You get the encryption-and-identity story without paying the full sidecar tax across every pod in the cluster.

Linkerd's lighter footprint already addressed this trade for many teams. Ambient closes the gap from Istio's side. If you're reaching for Istio specifically and have an ambient-capable kernel, evaluate ambient first.

The operational reality nobody puts on the slides

Two failure modes you'll hit, regardless of which mesh you pick:

Sidecars and init order. Workloads that need to make outbound calls during startup race with the sidecar. The classic symptom is your migration job that runs at pod start failing with a connection refused, because the Envoy sidecar isn't ready yet. Kubernetes' native sidecar containers (stable since 1.29) fix this if you're on a recent enough cluster. If not, you need holdApplicationUntilProxyStarts in Istio or equivalent in Linkerd. This bites every team eventually.

Upgrade choreography. A mesh control plane upgrade can cascade into every sidecar in the cluster restarting. You'll discover, the first time you do this in production, that some of your pods don't tolerate a sidecar restart as gracefully as you assumed. Plan upgrades for low-traffic windows, drain the cluster's most sensitive workloads first, and have a tested rollback. The mesh vendor docs underplay this; the on-call rotation doesn't.

These aren't deal-breakers. They're the kind of cost the conference talks gloss over and the procurement deck doesn't surface.

A practical decision tree

You have under 30 services and a platform team of two people. Don't run a mesh. OpenTelemetry, ingress Envoy, cert-manager, network policies. Revisit in a year.

You have 30 to 100 services, a real platform team, and one of the three justification questions is a hard yes. Run Linkerd. Smallest surface area, fastest time to value, easiest to debug.

You have 100+ services across multiple clusters and at least two of the three questions are yes. Ambient Istio is the modern default. Pay the operational tax knowingly. Staff for it.

You already run Cilium as your CNI and your platform team is comfortable with eBPF. Cilium service mesh is your shortest path; the kernel-level data plane wins on overhead.

The operational complexity of any mesh outweighs the technical win for teams under fifty services. That isn't snark; it's the conclusion the platform team I opened with reached after running Istio in production for half a year. Their advice, retroactively: solve the actual problem (mTLS for one regulator-facing service path, observability across the whole estate) with the smallest tool that solves it. Add the mesh later, if the answers to the three questions change.

The thing that flipped in 2026 is the bottom of the toolbox got better. Ambient Istio lowers the floor. Linkerd's policy model matured. Cilium gives you a sidecar-free path if your infrastructure cooperates. The case for adopting a mesh is now stronger when it applies and weaker when it doesn't. That asymmetry is the win.

What pushed your team toward a mesh, or kept you off one? Curious which of the three questions actually moved the decision.

If this was useful

Service mesh is one of the higher-stakes architectural choices a platform team makes. Get it wrong and you carry the operational cost for years. The chapter on networking and service-to-service communication in the System Design Pocket Guide: Fundamentals walks through the same trade lens applied to load balancers, ingress, and inter-service protocols, with the goal of making "do we need this layer" answerable on the back of an envelope.

Caching Layers in 2026: CDN, App, DB, Query: What Goes Where

Gabriel Anhaia — Sun, 24 May 2026 15:20:39 +0000

Book: System Design Pocket Guide: Fundamentals — Core Building Blocks for Scalable Systems
Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Four cache layers sit between a user's request and the row they're asking about. Most teams use two of them (usually Redis in front of the DB plus a CDN for static assets) and treat the other two as someone else's problem.

That's how you end up with a Redis tier doing the work the CDN should be doing, a database that's silently using its plan cache as a quality cushion, and a stampede every time a hot key expires. Each layer answers a different question. Pick the wrong one and you pay for the layer you didn't pick.

Why caching is layered, not picked

The pattern matters because the questions stack:

The CDN answers "can we avoid the origin entirely?"
The application cache answers "can we avoid the DB round trip?"
The database cache answers "can we avoid recomputing the result?"
The query cache answers "can we avoid parsing and planning the statement?"

Each "yes" you can get short-circuits the layers below. Each "no" passes the request down. A request that hits all four layers shouldn't feel different from one that hits none, except in cost and latency.

The mistake is collapsing this into a single layer. Teams shove everything into Redis because it's the layer they already have. Redis is a fine application cache. It's a terrible CDN, a worse materialised view, and it can't help your prepared-statement planner at all.

Layer 1: CDN edge cache

The CDN sits closest to the user. It's the cheapest place to serve a request because the request never reaches your servers. The trick in 2026 is that CDNs cache more than .jpg files. They cache API responses too, if you tell them how.

Here's a Cloudflare cache rule for a public product-listing endpoint:

// Cloudflare Worker - cache /api/products/* responses for 60s at edge
export default {
  async fetch(request, env, ctx) {
    const url = new URL(request.url);
    const cache = caches.default;

    // only cache GETs for the public catalog
    if (request.method !== "GET" ||
        !url.pathname.startsWith("/api/products/")) {
      return fetch(request);
    }

    // strip auth-affecting query params from the cache key
    url.searchParams.delete("trace_id");
    const cacheKey = new Request(url.toString(), request);

    let response = await cache.match(cacheKey);
    if (response) {
      return response; // edge hit, never touches origin
    }

    response = await fetch(request);
    if (response.status === 200) {
      const cached = new Response(response.body, response);
      cached.headers.set("Cache-Control", "public, max-age=60, s-maxage=60");
      cached.headers.set("CDN-Cache-Control", "max-age=60");
      ctx.waitUntil(cache.put(cacheKey, cached.clone()));
      return cached;
    }
    return response;
  },
};

The s-maxage is the shared-cache directive. It tells the CDN how long to hold the response, separate from what browsers do. The two Cache-Control directives let you give the CDN 60 seconds while telling the browser something different (often max-age=0, must-revalidate).

What goes here: public GETs, anonymous responses, things that look the same for thousands of users. Product listings, marketing pages, public profile pages, search-results-without-personalisation, OG image endpoints, sitemap.xml. Anything that personalises per-user is wrong here unless you cache-key on a user ID.

What doesn't: anything with Set-Cookie in the response, anything authenticated, anything with a write side effect. If your CDN hit rate on /api/* is above 5%, you're already doing better than most teams.

Layer 2: Application cache (Redis)

The application cache is what people mean when they say "cache." It sits in your service process or in Redis next to it. It catches the requests the CDN couldn't (because they're authenticated or personalised) and serves them without a DB round trip.

The pattern that ships well:

import json
import redis
from typing import Optional

r = redis.Redis(host="cache.internal", port=6379, decode_responses=True)

def get_user_profile(user_id: str) -> dict:
    key = f"user:profile:{user_id}"

    # try cache first
    cached = r.get(key)
    if cached is not None:
        return json.loads(cached)

    # cache miss - hit the DB
    profile = db.fetch_one(
        "SELECT id, name, plan, created_at FROM users WHERE id = $1",
        user_id,
    )

    # set with a TTL so stale data times out even if invalidation fails
    r.setex(key, 300, json.dumps(profile, default=str))
    return profile

def invalidate_user_profile(user_id: str) -> None:
    # called from any writer that mutates the user row
    r.delete(f"user:profile:{user_id}")

Two things people skip: the TTL safety net, and the write-side invalidation. TTL alone leaks stale reads for up to 5 minutes after a write. Invalidation alone leaks forever when a network blip drops the DEL. You need both. Belt, suspenders, and a third hand on the belt.

What goes here: per-user data, session state, computed aggregates that are expensive to rebuild, anything you'd hate to recompute on every request but that changes infrequently. The 80/20 rule: pick the 20% of queries that account for 80% of your DB load and cache those first.

What doesn't: anything that needs to be strongly consistent (a bank balance someone is about to spend). Anything written more often than read. Anything where a stale read is worse than a 50ms delay.

Layer 3: Database cache (materialised views)

Materialised views are the layer most teams forget exists. They sit inside the database and pre-compute the result of an expensive query. The DB stores the result like a table. Reads are O(1) lookups instead of seven joins and a window function.

Postgres example. A per-day, per-account revenue rollup that would otherwise scan a fact table on every dashboard load:

CREATE MATERIALIZED VIEW account_revenue_daily AS
SELECT
    account_id,
    date_trunc('day', created_at)::date AS day,
    sum(amount_cents) AS revenue_cents,
    count(*) AS txn_count
FROM transactions
WHERE status = 'settled'
GROUP BY account_id, day;

CREATE UNIQUE INDEX ON account_revenue_daily (account_id, day);

-- refresh policy: every 10 minutes via pg_cron
-- CONCURRENTLY needs the unique index above to work
SELECT cron.schedule(
    'refresh-account-revenue',
    '*/10 * * * *',
    $$REFRESH MATERIALIZED VIEW CONCURRENTLY account_revenue_daily$$
);

REFRESH ... CONCURRENTLY is the one nobody reads about. Without it, the refresh takes an ACCESS EXCLUSIVE lock that blocks reads. With it, the refresh writes to a shadow copy and swaps atomically. You pay slightly more disk during the swap; you stop blocking your dashboard.

What goes here: aggregations, joins across 4+ tables, anything where the underlying data changes slower than you query it. Per-day rollups, leaderboards, search facets, anything an analyst would write a CTE for.

What doesn't: results that need to be real-time. Materialised views are stale by definition between refreshes. If a user expects to see their action reflected immediately, this layer isn't the answer.

The gotcha here is silent: materialised views age. A team ships one, hits the dashboard latency target, moves on. Six months later the underlying table has tripled in size, the refresh runs for 12 minutes, and the view is more often stale than fresh. Audit refresh durations like you audit query plans.

Layer 4: Query cache (prepared-statement plan cache)

The deepest layer is also the most invisible. Every time your driver sends a SQL statement, the database has to parse it, plan it, and execute it. The first two steps can be cached if you use prepared statements.

Most ORMs do this poorly because they ship a fresh statement string for every query (WHERE id = 1 vs WHERE id = 2), defeating the cache. The fix is parameter binding:

# bad - new statement every call, plan cache miss every time
def get_order_bad(order_id: int):
    return db.execute(f"SELECT * FROM orders WHERE id = {order_id}")

# good - same statement text, only parameters change, plan reused
def get_order_good(order_id: int):
    return db.execute(
        "SELECT * FROM orders WHERE id = $1",
        order_id,
    )

In Postgres, you can see what's getting cached:

-- requires pg_stat_statements extension
SELECT
    query,
    calls,
    mean_exec_time,
    rows
FROM pg_stat_statements
WHERE query LIKE 'SELECT % FROM orders WHERE id = $1'
ORDER BY calls DESC
LIMIT 10;

If you see the same logical query showing up dozens of times with different literal values instead of $1, your ORM is bypassing the plan cache. Fix the ORM config (Eloquent's DB::statement vs DB::select with bindings, GORM's Raw vs Where, etc.) before you tune anything else.

What goes here: hot OLTP queries. The lookup-by-id, the insert-into-orders, the update-customer-last-seen. The win per query is small (a millisecond, maybe two). The win at 50,000 queries per second is the difference between two database instances and ten.

What doesn't: queries with variable shapes. If your WHERE clause changes structure based on user input, you can't reuse the plan, and that's fine.

The decision matrix

The shortcut for "which layer does this data belong in":

Data shape	CDN	App cache	Materialised view	Plan cache
Static asset, anonymous	yes	no	no	no
Public GET, no auth	yes	maybe	no	no
Per-user profile, hot	no	yes	no	yes
Hourly analytics rollup	no	maybe (TTL)	yes	no
OLTP id lookup	no	yes	no	yes
Strongly consistent read	no	no	no	yes
Real-time write feedback	no	no	no	yes

The "maybe" rows are where teams argue in design reviews. The honest answer is "measure first." If your hourly rollup gets hit 200 times per minute, a materialised view earns its keep. If it gets hit 3 times per day, a Redis cache with a 30-minute TTL is fine and the materialised view is over-engineering.

Invalidation strategy per layer

Each layer wants a different invalidation story.

CDN. TTL is your only realistic option for most teams. You can purge specific URLs via API, but at high request rates the purge takes seconds to propagate and you can't rely on it for correctness. Use short TTLs (60s for hot endpoints, 5 min for catalog data) and accept the staleness. For long-TTL assets, version the URL (/static/app.a3f7b2.js) so a new version is a new key, not an invalidation.

App cache. Event-driven invalidation. Every write path that mutates a row calls r.delete(key) for every cached derivative of that row. This works until you have 12 places that write to users and someone adds the 13th without remembering to invalidate. Centralise it: every write goes through a repository that fires invalidations as part of its post-commit hook.

Materialised view. Scheduled refresh. Decide your acceptable staleness (10 minutes? 1 hour?) and set a cron. For lower latency on the staleness window, layer event-driven REFRESH ... CONCURRENTLY triggers, but be careful: refresh is itself an expensive operation that you don't want firing 100 times a minute.

Plan cache. You don't invalidate it. The database manages it. Your job is to write queries that look the same to the planner.

The gotcha: stampedes across layers

The bug that kills the layered design: when a hot cache entry expires across multiple layers at once, every concurrent request misses every layer simultaneously, and they all stampede the origin together.

A real version: a CDN cache for /api/homepage expires at 12:00:00. A thousand concurrent requests miss the CDN, hit your app, miss the app cache (which also expired at 12:00:00 because both TTLs were 60s and both were set at 11:59:00), hit the database, all trigger the same materialised-view rebuild, and the database falls over.

Two patterns prevent this. The first is singleflight: collapse N identical concurrent requests into one and broadcast the result:

import (
    "context"
    "encoding/json"
    "github.com/redis/go-redis/v9"
    "golang.org/x/sync/singleflight"
)

var sf singleflight.Group

func GetHomepage(ctx context.Context, rdb *redis.Client) ([]byte, error) {
    key := "homepage:v3"

    if cached, err := rdb.Get(ctx, key).Bytes(); err == nil {
        return cached, nil
    }

    // singleflight: only the first concurrent miss recomputes;
    // every other caller waits on the same future
    result, err, _ := sf.Do(key, func() (interface{}, error) {
        fresh, err := buildHomepageFromDB(ctx)
        if err != nil {
            return nil, err
        }
        payload, _ := json.Marshal(fresh)
        // SETEX with a small jitter so 1000 keys don't expire on the same second
        rdb.SetEx(ctx, key, payload, ttlWithJitter(60))
        return payload, nil
    })
    if err != nil {
        return nil, err
    }
    return result.([]byte), nil
}

The second is lock + serve-stale. Hold the expired value in cache past its TTL, return it while one request goes to rebuild. Pseudo-pattern:

def get_with_stale(key: str, fetch_fn, fresh_ttl=60, stale_ttl=600):
    payload = r.get(key)
    meta = r.get(f"{key}:meta")

    if payload and meta and meta == "fresh":
        return json.loads(payload)

    # try to grab the rebuild lock; if we get it, rebuild
    lock_key = f"{key}:lock"
    got_lock = r.set(lock_key, "1", nx=True, ex=10)

    if got_lock:
        try:
            fresh = fetch_fn()
            r.setex(key, stale_ttl, json.dumps(fresh))
            r.setex(f"{key}:meta", fresh_ttl, "fresh")
            return fresh
        finally:
            r.delete(lock_key)

    # we didn't get the lock - serve stale if we have it
    if payload:
        return json.loads(payload)

    # no stale, no lock - wait briefly and retry the read
    time.sleep(0.05)
    payload = r.get(key)
    return json.loads(payload) if payload else fetch_fn()

Now a thousand concurrent requests trigger one rebuild. The other 999 either get the slightly-stale value (acceptable) or wait 50ms and pick up the freshly written one. The database sees one extra query instead of a thousand.

Apply this pattern at every layer. CDNs do it natively with stale-while-revalidate directives. Application caches need you to wire it. Materialised views are inherently stale-while-revalidate by design. Plan caches don't have the problem.

What to do Monday

Walk a request through your system and label which layer each piece of data should live in. Most teams discover they have one layer doing the job of three, or three layers all caching the same thing with different TTLs that don't agree.

Pick the layer with the highest miss-to-cost ratio. Add it. Measure. Repeat. The goal is not to use every layer. It's to put each piece of data in the layer that answers its specific question, then make sure none of them stampede.

Which layer is doing the most work in your current system, and which one is silently absent?

If this was useful

The four-layer cache stack is the kind of thing a system-design interview will ask about, and the kind of thing your ops team will thank you for shipping. The System Design Pocket Guide: Fundamentals covers caching in the chapter on "performance primitives," alongside the load-balancing, queueing, and replication patterns that decide whether your design holds up at scale. If this post helped, the book is the same voice across 250+ pages of the building blocks.

Design a Multi-Device Authentication Service (Sessions vs JWT vs Passkeys)

Gabriel Anhaia — Sun, 24 May 2026 15:20:18 +0000

Book: System Design Pocket Guide: Interviews — 15 Real System Designs, Step by Step
Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Sessions, JWTs, refresh tokens, OAuth, magic links, passkeys. Your authentication service has to support all of them, on any device, with a "log out everywhere" button that actually works. Most candidates pick one mechanism and freeze. The interviewer wants to see you weigh the tradeoff and design the seams.

Here are the five decisions that frame the whole thing, the three implementations of the log-out button (cheap to bulletproof), and the gotcha that quietly breaks production at month four.

The five decisions

Before you draw a single box, write these on the whiteboard:

Token shape: opaque session ID or self-contained JWT?
Stateful or stateless: does verification hit a store on every request?
Revocation model: how do you kill a credential before it expires?
Device tracking: one session per user, or one per device?
MFA gating: always-on, risk-based, or step-up on sensitive actions?

Each decision constrains the next. Pick stateless JWTs and revocation becomes a research project. Pick per-user sessions and "log out my laptop only" becomes impossible. The interviewer is watching you handle the constraint propagation, not memorize a stack.

Session model: server-side store, easy revocation

The classic shape. Login succeeds, the server creates a session row and returns an opaque ID via cookie. Every request hits a store (Redis, Postgres, whatever) to validate.

CREATE TABLE sessions (
  id            BYTEA PRIMARY KEY,          -- 256-bit random
  user_id       UUID NOT NULL,
  device_id     UUID NOT NULL,
  created_at    TIMESTAMPTZ NOT NULL,
  last_seen_at  TIMESTAMPTZ NOT NULL,
  expires_at    TIMESTAMPTZ NOT NULL,
  ip            INET,
  user_agent    TEXT,
  mfa_verified  BOOLEAN NOT NULL DEFAULT false
);

CREATE INDEX sessions_user_id_idx ON sessions (user_id);
CREATE INDEX sessions_expires_idx ON sessions (expires_at);

Revocation is a DELETE. Log out everywhere is DELETE FROM sessions WHERE user_id = $1. Device-tracking is free because every row already names a device. Auditing is free too.

The cost is a round trip per request. At 50k RPS, that's 50k Redis hits per second. Doable, but you'll cache the session in the gateway's local memory for 5-10 seconds to absorb the load, and you'll size Redis for the working set of active sessions, not the total count.

The hidden cost: every service that wants to know "who is this" needs the store. Pick this model and your auth service becomes a hard dependency for the whole platform. Plan for it.

JWT model: stateless, fast, revocation is the hard problem

A JWT carries its own claims, signed by the auth service. Any service with the public key can validate it without calling back. No store lookup, no round trip. Beautiful, until you need to kick someone out.

// Verifying a JWT on a downstream service
import { jwtVerify } from 'jose';

const { payload } = await jwtVerify(token, publicKey, {
  issuer: 'auth.example.com',
  audience: 'api.example.com',
});

// payload.sub is the user id. No DB call.

The token's claims are frozen at issue time. If you change a user's role, demote them, or notice their token leaked, the existing JWT keeps working until it expires. That's the whole catch. Every revocation strategy is a workaround for this one property.

Three workarounds, in order of pain:

Short expiry + refresh tokens. Issue JWTs that live 5-15 minutes. Compromise window is small. Refresh tokens live in a server store and can be revoked.
Revocation list (denylist). Maintain a set of jti claims that should be rejected. Every verifier consults the list. You just put the store back; the win is partial (only revoked tokens need a lookup).
Versioned claims. Embed a token_version claim. Bump the version in the user record on revocation. Verifiers compare to the user's current version. Cheap to write, expensive to read (still a store hit).

Notice the pattern: every workaround re-introduces the state the JWT was meant to avoid. Stateless verification is the wrong goal. The right goal is cheap verification, and short JWTs + refresh tokens gets you there.

Refresh tokens: the bridge

Refresh tokens are the seam where session-like state meets stateless verification. Issue a short JWT (15 min) plus a long refresh token (30 days). The refresh token sits in a server store and looks like a session row.

CREATE TABLE refresh_tokens (
  id              BYTEA PRIMARY KEY,
  user_id         UUID NOT NULL,
  device_id       UUID NOT NULL,
  family_id       UUID NOT NULL,            -- rotation lineage
  issued_at       TIMESTAMPTZ NOT NULL,
  expires_at      TIMESTAMPTZ NOT NULL,
  revoked_at      TIMESTAMPTZ,
  replaced_by     BYTEA REFERENCES refresh_tokens(id)
);

When the JWT expires, the client posts the refresh token to /auth/refresh. The server validates it, rotates it (issues a new one and revokes the old), and returns a fresh JWT + fresh refresh token. The family_id tracks the rotation lineage.

The rotation matters for one reason: replay detection. If a refresh token gets used twice (once by the legit client, once by an attacker who stole it), the server sees a token in family_id = X being used after it was rotated. That's a hard signal. Revoke the entire family and force re-login.

def refresh(token_id: bytes) -> NewTokenPair:
    row = db.fetch_one(
        "SELECT * FROM refresh_tokens WHERE id = $1", token_id
    )
    if row is None:
        raise InvalidToken()
    if row.revoked_at is not None:
        # Replay detected. Kill the whole family.
        db.execute(
            "UPDATE refresh_tokens SET revoked_at = now() "
            "WHERE family_id = $1 AND revoked_at IS NULL",
            row.family_id,
        )
        raise InvalidToken()
    if row.expires_at < now():
        raise ExpiredToken()

    new_refresh = issue_refresh(row.user_id, row.device_id, row.family_id)
    db.execute(
        "UPDATE refresh_tokens SET revoked_at = now(), replaced_by = $1 "
        "WHERE id = $2",
        new_refresh.id, token_id,
    )
    return NewTokenPair(jwt=issue_jwt(row.user_id), refresh=new_refresh)

This is the architecture worth defending in the interview. Short JWTs for cheap verification, refresh tokens for revocation, families for replay detection. Almost every modern auth service ships this shape.

Passkeys (WebAuthn): the 2026 default

Passkeys replaced "password + TOTP" as the recommended default for new sign-in flows. They're public-key credentials stored in the user's device keychain (or roaming via iCloud Keychain, Google Password Manager, 1Password). The server never sees a secret.

Registration:

Server sends a challenge plus user info.
Browser asks the authenticator (Touch ID, Windows Hello, security key) to generate a keypair.
Authenticator returns the public key plus an attestation. Server stores the public key against the user.

Authentication:

Server sends a challenge.
Authenticator signs it with the matching private key.
Server verifies the signature against the stored public key.

The server changes you need:

CREATE TABLE webauthn_credentials (
  credential_id   BYTEA PRIMARY KEY,
  user_id         UUID NOT NULL,
  public_key      BYTEA NOT NULL,
  sign_count      BIGINT NOT NULL DEFAULT 0,
  transports      TEXT[],                   -- ['internal','hybrid','usb']
  aaguid          UUID,                     -- authenticator model
  backup_eligible BOOLEAN NOT NULL,
  backup_state    BOOLEAN NOT NULL,
  created_at      TIMESTAMPTZ NOT NULL,
  last_used_at    TIMESTAMPTZ
);

sign_count is a monotonically increasing counter the authenticator returns. If it ever goes backwards, the credential was cloned. backup_eligible plus backup_state tell you whether the passkey is synced (iCloud) or device-bound (a YubiKey).

Once a user signs in with a passkey, your service issues the same JWT + refresh token pair as before. WebAuthn replaces the credential check, not the session layer. That's a useful framing for the interview: passkeys are an authentication factor, the rest of the architecture stays.

The one new failure mode: a user loses every device with the passkey. Your account-recovery flow has to exist and has to not be a password reset to email (defeats the point). Use a recovery code printed at enrollment, a verified backup device, or a delegated recovery contact.

"Log out everywhere": three implementations

This is the question that separates candidates. Three answers, cheap to bulletproof.

1. Delete sessions / refresh tokens. Works if you're session-based or short-JWT + refresh. One SQL statement:

DELETE FROM refresh_tokens WHERE user_id = $1;
DELETE FROM sessions       WHERE user_id = $1;

The catch: existing JWTs keep working until they expire. If your JWT lives 15 minutes, the user's old devices have up to 15 minutes of grace. For consumer apps that's usually fine. For a banking app it isn't.

2. Bump a token version. Add token_version to the user row and to every JWT claim. On verify, compare. Mismatch = reject.

def verify(token: str) -> Claims:
    claims = jwt.decode(token, public_key)
    user_version = cache.get_user_token_version(claims['sub'])
    if claims['ver'] != user_version:
        raise Revoked()
    return claims

Log out everywhere becomes UPDATE users SET token_version = token_version + 1 WHERE id = $1. Every existing JWT is now invalid. Verifiers pay one cache hit per request (cache the version aggressively, invalidate on bump).

3. Per-credential revocation broadcast. The bulletproof option. Maintain a denylist of jti claims with TTLs matching the JWT's exp. Push revocations to every verifier via Redis pub/sub, NATS, or a streaming bus. Verifiers keep a local Bloom filter, fall back to the central store on a hit.

def verify(token: str) -> Claims:
    claims = jwt.decode(token, public_key)
    if revocation_bloom.might_contain(claims['jti']):
        if redis.sismember('revoked', claims['jti']):
            raise Revoked()
    return claims

This combines low-latency verification (Bloom filter is in-process) with hard revocation guarantees. The cost is operational: a streaming bus to maintain, a Bloom filter to size, and a denylist that has to expire entries.

For most products, option 1 plus short JWTs is the right answer. Option 2 if your JWTs live longer than a minute and you can't tolerate that grace. Option 3 if you're under regulatory pressure (PCI, HIPAA, SOC 2 with specific revocation SLAs).

Device tracking: sessions per device

A row per (user, device) is what makes "log out my laptop only" possible. The device_id is generated by the client on first auth, persisted in secure storage (Keychain on iOS, Credential Manager on Android, IndexedDB + a fingerprint on web). It survives reinstalls if you ship a recovery hint, and it survives nothing if you don't.

Track three things per device row:

last_seen_at: update on every refresh, drives "active devices" UI.
ip and a geo lookup: drives "new sign-in from Brazil" emails.
user_agent plus a friendly device label ("Pixel 9 Pro", "Chrome on macOS"): what the user actually recognizes in the settings page.

Avoid hard fingerprinting (canvas, WebGL, font enumeration). It's brittle, it breaks privacy modes, and the GDPR conversation is uncomfortable. A self-generated device ID covers 95% of the use cases without the legal surface area.

MFA gating: risk-based vs always-on

Always-on MFA is the safe answer. Every login asks for a second factor. Users hate it. They also enable it and forget about it, which is the point.

Risk-based MFA is the better UX answer. Score each login attempt: new device, new geo, unusual hour, password reset within 24h, impossible travel since last session. Above a threshold, demand a second factor. Below, skip.

def risk_score(attempt: LoginAttempt) -> int:
    score = 0
    if attempt.device_id not in known_devices(attempt.user_id):
        score += 40
    if attempt.geo != last_geo(attempt.user_id):
        score += 20
    if impossible_travel(attempt):
        score += 60
    if attempt.password_changed_within(hours=24):
        score += 20
    if attempt.hour not in usual_hours(attempt.user_id):
        score += 10
    return score

A score above 50 triggers a step-up: passkey, TOTP, push approval. The implementation lives in the auth service, the rules live in config so you can tune them without a deploy.

Step-up MFA is the related pattern: sensitive actions (change password, add a new payment method, delete account) trigger MFA even mid-session. Implement it as a mfa_verified_at timestamp on the session. Within 5 minutes counts, older requires a re-prompt.

The 90-second answer

If the interviewer asks for the elevator pitch:

Short-lived JWTs (15 min) for fast, stateless verification at every service. Long-lived refresh tokens (30 days, rotated on use, families tracked for replay detection) backed by a Postgres or Redis store. That's the revocation surface. Per-device session rows so the user can see and kill individual devices. Passkeys (WebAuthn) as the primary factor for new accounts, password-plus-TOTP as the legacy fallback. Risk-based step-up MFA for sensitive actions. "Log out everywhere" bumps a token_version on the user record so existing JWTs fail their next verify, and deletes all refresh tokens so they can't bootstrap a new JWT. Device-tracking via a self-generated client ID, not browser fingerprinting.

That covers the five decisions, names the rotation pattern, includes a real revocation strategy, and namechecks the modern default. It's about 90 seconds spoken.

The gotcha: JWT revocation lists grow without bound

The trap I see in code reviews more than any other: someone implements a JWT denylist as a Redis set keyed on user ID, with no TTL on the entries.

# Don't do this
redis.sadd(f'revoked:{user_id}', token_jti)

A year later, that set has 50k JTIs in it, half of them long expired, every verify is shipping a 200KB set membership check. Eventually the verify path dominates your CPU and someone deletes the set, which silently un-revokes everything.

Two fixes:

Store JTIs with TTLs matching the JWT exp. Use individual keys, not sets. SET revoked:<jti> 1 EX <seconds-until-exp>. Redis evicts on its own.
Use short JWTs. If your JWT lives 5 minutes, you barely need a denylist. Revoke the refresh token, wait 5 minutes, done. The denylist becomes a defense-in-depth layer for the worst case, not the primary mechanism.

Or, put bluntly: if your JWTs live an hour and you need real revocation, your JWTs are too long.

The interview takeaway: revocation drives the architecture. Pick a token lifetime that matches your revocation SLA. Five minutes if you can absorb the refresh chatter, fifteen if you can't, never an hour for anything that matters. The rest of the design (refresh families, passkeys, device tracking, MFA gating) falls out of that one decision.

What's your team's revocation SLA, and does your current JWT lifetime actually match it?

If this was useful

Auth services are one of the 15 walkthroughs in the System Design Pocket Guide: Interviews. The chapter walks the same five decisions in interview-pace, plus the variants the interviewer is likely to push you on (OAuth provider integration, mobile vs web token storage, the multi-tenant case). It's the book I'd hand a friend the week before an L5 system-design loop.

Design a Payment Ledger: Idempotent, Audit-Compliant, Reconciles to the Cent

Gabriel Anhaia — Sun, 24 May 2026 13:47:58 +0000

Book: System Design Pocket Guide: Interviews — 15 Real System Designs, Step by Step
Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

"Design a payment ledger" is the prompt that ends interview rounds. Candidates draw a payments table, an update balance arrow, and a Redis cache. Twenty minutes later the interviewer has asked about duplicate charges, partial refunds, and what happens when the processor times out. The whiteboard is a mess.

A payment ledger isn't a CRUD app. It's an append-only event store with two-sided arithmetic, a reconciliation job that runs at 03:00, and a regulator who wants to see every byte you ever wrote. Get four properties right and the rest of the design draws itself.

Why a ledger isn't a CRUD app

The instinct is wrong from the first table. CRUD systems mutate rows. Ledgers don't mutate anything. Every event (charge, refund, chargeback, fee, FX adjustment) is a new entry. The current balance is a projection over the entries, not a value you store and update.

The reason isn't ideology. It's audit. A regulator (or an angry customer, or your CFO) eventually asks: "what was account 7218's balance at 14:32 on March 4th?" If you've been updating a balance column, you can't answer. If you've been appending entries, you replay events up to that timestamp and the answer falls out.

The four properties that separate ledgers from CRUD:

Double-entry: every transaction touches at least two accounts, and the sum is zero.
Idempotency per attempt: the same payment attempt, retried, doesn't double-charge.
Append-only: entries are never updated or deleted, only reversed by new entries.
Reconcilable: your numbers match the payment processor's numbers, to the cent, every day.

Miss any one and you're shipping a bug factory.

Double-entry bookkeeping in code

Double-entry has been around since the 15th century. Pacioli wrote it down in 1494. Every accountant on Earth uses it. Software engineers keep reinventing single-entry ledgers and discovering, three years later, why that's bad.

The rule: every transaction has two sides, and they sum to zero. A customer pays you $50. The customer's liability account goes down by $50; your cash account goes up by $50. Net change: zero. You haven't created or destroyed money. You've moved it.

In SQL, that looks like this:

CREATE TABLE accounts (
    id              BIGSERIAL PRIMARY KEY,
    owner_type      TEXT NOT NULL,     -- 'customer', 'platform', 'processor'
    owner_id        TEXT NOT NULL,
    currency        CHAR(3) NOT NULL,  -- ISO 4217
    account_type    TEXT NOT NULL,     -- 'asset', 'liability', 'revenue', 'expense'
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE (owner_type, owner_id, currency, account_type)
);

CREATE TABLE transactions (
    id              UUID PRIMARY KEY,
    idempotency_key TEXT NOT NULL UNIQUE,
    description     TEXT NOT NULL,
    occurred_at     TIMESTAMPTZ NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE entries (
    id              BIGSERIAL PRIMARY KEY,
    transaction_id  UUID NOT NULL REFERENCES transactions(id),
    account_id      BIGINT NOT NULL REFERENCES accounts(id),
    direction       CHAR(1) NOT NULL CHECK (direction IN ('D','C')), -- Debit / Credit
    amount_minor    BIGINT NOT NULL CHECK (amount_minor > 0),         -- cents, always positive
    currency        CHAR(3) NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- The invariant that every code review should check for:
-- SUM(D) - SUM(C) = 0 for every transaction_id, per currency.

Two things to notice. amount_minor is always positive; direction lives in its own column. And entries carries no updated_at. The table is append-only; there's no UPDATE entries SET ... anywhere in the codebase. If you find one in code review, that's a blocker.

The balance projection is a query, not a stored value:

SELECT
    a.id,
    a.currency,
    COALESCE(SUM(CASE WHEN e.direction = 'D' THEN  e.amount_minor
                       WHEN e.direction = 'C' THEN -e.amount_minor END), 0) AS balance_minor
FROM accounts a
LEFT JOIN entries e ON e.account_id = a.id
WHERE a.id = $1
GROUP BY a.id, a.currency;

For hot accounts this gets too slow to run on every request. You materialize the projection into a balances table with last_entry_id, and a worker incrementally folds new entries in. The projection is a cache. If it goes corrupt, you drop it and rebuild from entries. The append-only log is the source of truth.

A single payment writes one transaction and two entries:

BEGIN;

INSERT INTO transactions (id, idempotency_key, description, occurred_at)
VALUES ('a3f...', 'pay_2026_05_24_attempt_1_user_7218', 'Order 9921 capture', now());

INSERT INTO entries (transaction_id, account_id, direction, amount_minor, currency) VALUES
  ('a3f...', 4501, 'D', 5000, 'USD'),  -- cash (asset) up
  ('a3f...', 7218, 'C', 5000, 'USD');  -- customer liability up (we owe them goods/refund-right)

COMMIT;

The BEGIN/COMMIT matters. The two entries must land atomically, or you have a half-applied transaction and the invariant is broken. Postgres gives you this for free. If you're sharding across databases, you need a two-phase commit, or better, keep both legs of any transaction in the same shard. Sharding a ledger is its own essay; the short version is: shard by owner_id, route both legs to the same shard, and accept that some cross-account transfers will need a saga.

Idempotency keys: per attempt, not per customer

The most common bug in payment systems: idempotency keyed on the wrong thing.

A customer clicks "pay" and the request times out. The mobile app retries. Without idempotency, you charge twice. With idempotency keyed on customer_id + order_id, the second attempt sees the existing transaction and returns the old result. Perfect.

Now the customer cancels and pays again for the same order. Different attempt, same key. You return the cached failure. The new payment never happens.

The fix is in the title: idempotency keys are per attempt, not per customer or per order. The client generates a fresh UUID for each payment attempt, sends it on every retry of that attempt, and discards it once the attempt completes.

import uuid

def charge_customer(client, order_id: str, amount_minor: int) -> str:
    # ONE attempt = ONE key. Retries of THIS attempt reuse it.
    # If the user clicks "pay" again later, the client generates a new one.
    idempotency_key = f"pay_{order_id}_{uuid.uuid4()}"

    for retry in range(3):
        try:
            return client.post(
                "/v1/payments",
                json={"order_id": order_id, "amount_minor": amount_minor},
                headers={"Idempotency-Key": idempotency_key},
                timeout=10,
            )
        except (TimeoutError, ConnectionError):
            # Same key, server returns the previously committed result if it landed
            continue
    raise PaymentFailed("3 retries exhausted")

Server side, the key is enforced by the UNIQUE constraint on transactions.idempotency_key. Two requests with the same key race; one wins the insert, the other gets a unique-violation and reads back the existing transaction.

def create_payment(db, idempotency_key: str, payload: dict) -> Transaction:
    try:
        with db.begin():
            txn = db.execute(
                "INSERT INTO transactions (id, idempotency_key, description, occurred_at) "
                "VALUES (%s, %s, %s, %s) RETURNING id",
                (uuid.uuid4(), idempotency_key, payload["description"], payload["occurred_at"]),
            ).fetchone()
            insert_entries(db, txn.id, payload["entries"])
            return txn
    except UniqueViolation:
        # Another request with the same key already committed
        return db.execute(
            "SELECT * FROM transactions WHERE idempotency_key = %s", (idempotency_key,)
        ).fetchone()

The UniqueViolation branch is the idempotency contract. It says: "I already did this exact thing, here's the result." That's why the key must scope to one attempt. If it scoped to the order, the retry of a second legitimate attempt would silently return the first attempt's result and the user would never get charged.

One more wrinkle: store the request body's hash alongside the key. If a client reuses an idempotency key with a different body (different amount, different currency), reject it with 409 Conflict. Stripe does this. It catches integration bugs early.

The append-only event log

The entries table is your event log. Treat it like Kafka. New events are appended, existing events are immutable, downstream projections are rebuilt from the log.

This buys you three things:

Time-travel: you can answer "what was the balance at any past moment" by replaying entries up to that timestamp.
Disaster recovery: if your balances materialized table is corrupted (a bad migration, a bug in the projector), you drop it and rebuild from entries.
Audit: the log is what regulators read. They don't want your balances table; they want the chronological record of every event.

The projector worker is straightforward:

def project_balances(db, batch_size: int = 1000):
    while True:
        new_entries = db.execute(
            "SELECT * FROM entries WHERE id > (SELECT COALESCE(MAX(last_entry_id), 0) FROM balances) "
            "ORDER BY id LIMIT %s",
            (batch_size,),
        ).fetchall()

        if not new_entries:
            time.sleep(1)
            continue

        with db.begin():
            for e in new_entries:
                delta = e.amount_minor if e.direction == "D" else -e.amount_minor
                db.execute(
                    "INSERT INTO balances (account_id, balance_minor, last_entry_id) "
                    "VALUES (%s, %s, %s) "
                    "ON CONFLICT (account_id) DO UPDATE SET "
                    "  balance_minor = balances.balance_minor + EXCLUDED.balance_minor, "
                    "  last_entry_id = EXCLUDED.last_entry_id",
                    (e.account_id, delta, e.id),
                )

If this worker falls behind, balances are stale but never wrong. If the worker dies and you restart it, it picks up from last_entry_id and catches up. If you find a bug in projection logic, you TRUNCATE balances and restart; the log rebuilds it.

Reconciliation: daily checksum against the processor

Every payment processor (Stripe, Adyen, PayPal, your acquiring bank) sends a settlement report. It lists what they think they collected, what they took in fees, and what they're going to deposit. Your job is to make their report match your ledger, to the cent.

The reconciliation job runs nightly:

def reconcile_day(db, processor, date: datetime.date):
    # Pull processor's view
    processor_charges = processor.list_charges(date=date)   # external truth

    # Pull our ledger's view of charges from that processor account
    our_charges = db.execute("""
        SELECT t.id, t.idempotency_key, e.amount_minor, e.currency
        FROM transactions t
        JOIN entries e ON e.transaction_id = t.id
        WHERE e.account_id = (SELECT id FROM accounts WHERE owner_id = %s)
          AND t.occurred_at::date = %s
          AND e.direction = 'D'
    """, (processor.account_id, date)).fetchall()

    # Match on processor's transaction reference, which we store in transactions.description or a side table
    discrepancies = compare(processor_charges, our_charges)

    if discrepancies:
        # Don't auto-fix. File a discrepancy record, page on-call.
        for d in discrepancies:
            db.execute(
                "INSERT INTO reconciliation_discrepancies "
                "(date, kind, processor_ref, processor_amount, our_amount, details) "
                "VALUES (%s, %s, %s, %s, %s, %s)",
                (date, d.kind, d.processor_ref, d.processor_amount, d.our_amount, d.details),
            )
        alert.page("Reconciliation failed", count=len(discrepancies), date=date)

The discrepancies are usually one of four shapes:

Processor has, we don't: a charge succeeded at the processor but the success webhook never reached us. Reach out, replay it.
We have, processor doesn't: we recorded a charge that never actually settled. Bug in our state machine.
Amount mismatch: usually rounding (more on that), or a fee we recorded against the wrong account.
Currency mismatch: an FX conversion happened differently on the processor side than we modeled.

Auto-fixing reconciliation discrepancies is a trap. You think you're cleaning up; you're actually papering over the bug that produced the discrepancy. File the discrepancy, page a human, fix the root cause. That's the only way the gap doesn't widen.

Dispute handling: chargebacks as reverse entries

A customer disputes a charge two months after it happened. The processor reverses the $50 and takes a $15 fee for their trouble. You do not go back and delete the original transaction. You write three new entries:

-- The chargeback reversal (mirrors the original charge, opposite direction)
INSERT INTO transactions (id, idempotency_key, description, occurred_at)
VALUES ('b88...', 'chargeback_a3f', 'Chargeback of txn a3f', '2026-07-20 10:00:00+00');

INSERT INTO entries (transaction_id, account_id, direction, amount_minor, currency) VALUES
  ('b88...', 4501, 'C', 5000, 'USD'),  -- cash down
  ('b88...', 7218, 'D', 5000, 'USD');  -- customer liability down

-- The processor's chargeback fee
INSERT INTO transactions (id, idempotency_key, description, occurred_at)
VALUES ('b89...', 'chargeback_fee_a3f', 'Chargeback fee for txn a3f', '2026-07-20 10:00:01+00');

INSERT INTO entries (transaction_id, account_id, direction, amount_minor, currency) VALUES
  ('b89...', 4501, 'C', 1500, 'USD'),  -- cash down
  ('b89...', 9100, 'D', 1500, 'USD');  -- 'dispute fees' expense account up

The original transaction a3f is still there. Its idempotency key (chargeback_a3f) refers back to it. The history reads: charged on March 4th, reversed on July 20th, fee posted on July 20th. A regulator or auditor can see the whole story. A DELETE FROM transactions WHERE id = 'a3f' would have destroyed it.

The rule is brutal and simple: never delete, always reverse. If you find code that runs DELETE against transactions or entries, it's a bug. Add a CI check.

Multi-currency, FX, and the precision problem

Money is not a float. This is the single most expensive bug in payment systems, and you find it in year 2 when you reconcile $0.03 against the processor and have no idea where it came from.

Floats lose precision. 0.1 + 0.2 != 0.3 in IEEE 754. Run it in any language:

>>> 0.1 + 0.2
0.30000000000000004

Multiply that across a million transactions and the rounding error becomes real money. Worse, the rounding is non-deterministic across operations: sum-then-multiply gives a different answer from multiply-then-sum. You can't reconcile against a processor that does the math in integers when your code does it in floats.

The fix is the boring one. Store money as integer minor units (cents for USD, satoshis for BTC, the smallest denomination your currency supports). The schema above already does this: amount_minor BIGINT. A $50.00 charge is 5000. A €19.99 charge is 1999. A ¥500 charge is 500 (JPY has no minor unit, so the conversion factor is 1 instead of 100).

# Wrong
amount_usd = 49.99
total = amount_usd * 1.07  # tax. Hello, float drift.

# Right
amount_minor = 4999          # $49.99
tax_minor = round(amount_minor * 7 / 100)   # 350 cents = $3.50
total_minor = amount_minor + tax_minor
# Bankers' rounding for half-cent cases:
from decimal import Decimal, ROUND_HALF_EVEN
tax_minor = int(
    (Decimal(amount_minor) * Decimal("0.07")).quantize(Decimal("1"), rounding=ROUND_HALF_EVEN)
)

FX is its own pit. When you accept a payment in EUR and settle in USD, you do the conversion once, store both legs in their native currencies, and record the FX rate as metadata. Don't average rates, don't recompute historical FX with today's rate, don't lose the original currency. A regulator will ask, and "we converted everything to USD at ingest" is the wrong answer.

ALTER TABLE transactions ADD COLUMN fx_rate NUMERIC(20, 10);
ALTER TABLE transactions ADD COLUMN fx_source_currency CHAR(3);
ALTER TABLE transactions ADD COLUMN fx_source_amount_minor BIGINT;

The cardinal sin is rounding at the wrong place. Round when the customer sees a number. Don't round inside the ledger. Every intermediate calculation runs at full precision; the rounding happens once, at the boundary, with ROUND_HALF_EVEN (bankers' rounding) to keep long-run drift symmetric.

The 90-second answer

When the interviewer asks "design a payment ledger," the answer that wins the round:

A payment ledger is an append-only event store with double-entry bookkeeping. Every transaction has two or more entries that sum to zero, stored in an immutable entries table. Current balances are a projection over the log, materialized into a balances table by a worker that folds new entries in incrementally.

Idempotency is per payment attempt, not per customer or order. The client generates a UUID for each attempt and reuses it on retries. The server enforces uniqueness with a UNIQUE constraint on transactions.idempotency_key, and stores a hash of the request body to reject reuse with a different payload.

Refunds and chargebacks never delete entries; they write reversing entries that point back to the original transaction by idempotency key. This preserves the audit trail that regulators require.

A nightly reconciliation job compares our ledger against the payment processor's settlement report, files discrepancies, and pages on-call. Reconciliation never auto-fixes; humans investigate.

Money is stored as integer minor units (cents). Never floating point. FX conversions store both source and target currency plus the rate, computed once at ingest and never recomputed.

The properties that make this different from CRUD: double-entry invariant per transaction, idempotency per attempt, append-only log as source of truth, daily reconciliation to the cent.

That's the 90 seconds. The interviewer asks follow-ups (sharding, partial refunds, multi-currency settlement, what happens during a processor outage), and the design above gives you a vocabulary to answer them without contradiction.

The gotcha: floating-point money is the bug you find at year 2

Every ledger that uses float or double for money eventually fails a reconciliation. Usually around year 2, when you have enough volume that a tenth of a cent of drift per transaction shows up as a dollar a day, and a few months of those add up to a number your CFO notices.

The fix is to use integers (cents) from day one. The refactor at year 2, when you have millions of rows in entries, is doable but expensive. You convert column types, rewrite every query, recompute every historical balance, and run for months in parallel comparing old and new sums. It works. It's also six months of work you didn't need.

If you take one thing from this post: amount_minor BIGINT, on every monetary column, from day one. The interview answer mentions it. The codebase enforces it. The CI check rejects any new NUMERIC or DOUBLE PRECISION column on a financial table.

The other lessons are reversible. This one isn't.

What's the worst ledger bug you've shipped to prod? Float drift, a missing idempotency key, a DELETE that should've been a reverse entry? Drop the story in the comments; I want to read it.

If this was useful

The full version of this design (sharding strategy, saga compensation for cross-shard transfers, the projection table schema in detail, and the regulator export format) is the payment-ledger chapter in the System Design Pocket Guide: Interviews. The book walks 15 system designs at this depth, each ending with a 90-second answer template. If you've got an interview coming up where "design Stripe" or "design Robinhood" might land, the ledger chapter is the one to read first.

Design a Real-Time Collaboration Backend (OT vs CRDT, Step by Step)

Gabriel Anhaia — Sun, 24 May 2026 13:47:32 +0000

Book: System Design Pocket Guide: Interviews — 15 Real System Designs, Step by Step
Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

"Design Google Docs" is the prompt that exposes whether a candidate has actually thought about real-time collaboration, or whether they've memorized a list of buzzwords. The trap is that OT and CRDT are not interchangeable. Pick the wrong one and your sync engine fights itself for the rest of its life.

Most candidates pick CRDT because it sounds fancier. Most production systems that ship still use OT. Both are right answers depending on the question, and the interviewer is watching to see if you know which question is being asked.

The interview-grade definition of "real-time collaboration"

When the interviewer says "real-time collaboration," they mean four properties at once:

Two or more clients edit the same document. Concurrent edits don't lose each other.
Edits propagate in under ~100ms perceived latency.
Every client converges to the same final state, eventually.
The system survives a client going offline mid-edit and reconnecting later.

That's it. Notice what's missing: nothing about locking, nothing about "the server decides who wins." A collaboration system that locks rows is a CRUD app with extra steps. The hard part is that two users typed at the same position at the same time and both edits have to land.

The two families of algorithms that solve this are Operational Transformation (OT) and Conflict-free Replicated Data Types (CRDT). They solve the same problem with opposite philosophies.

OT: the Google Docs lineage

Operational Transformation came out of academic CSCW research in the late 1980s. Jupiter and the GROVE editor refined it through the 1990s. Google Docs picked it up around 2010 and made it the household name.

The core idea: every edit is an op (operation) with a position. When two ops happen concurrently, a central server transforms one against the other so they apply in a consistent order on every client.

A minimal OT op looks like:

type Op =
  | { type: "insert"; pos: number; char: string }
  | { type: "delete"; pos: number; len: number };

// Client A sees: "hello"
// A inserts "X" at pos 2 → "heXllo"
// Concurrently, B deletes 1 char at pos 0 → "ello"
// Server receives both. It transforms A's op against B's:
//   A.pos -= B.len  →  insert "X" at pos 1
// Final state on every client: "eXllo"

The transform function is the heart of the algorithm. For a real editor you have transforms for every pair: insert × insert, insert × delete, delete × delete, plus formatting ops, attribute changes, and embedded objects. Get one transform wrong and two users with the same op history end up with different text. Debugging that bug in production is the kind of thing that turns engineers into woodworkers.

What OT requires: a central, authoritative server. Every client sends ops to the server, the server orders them, transforms them, and broadcasts the transformed versions back. You can't run OT peer-to-peer without electing one peer as the server, which defeats the point.

What you get in return: small ops on the wire (a single insert is a handful of bytes), simple client-side state (just the current document plus a small pending-op queue), and a decades-deep body of academic work that explains every edge case.

OT in the wild: Google Docs, Google Slides, ShareDB, Apache Wave (RIP), CodeMirror's @codemirror/collab package.

CRDT: the Figma / Linear / Notion direction

A CRDT replaces "transform ops on a central server" with "ops carry enough metadata to be commutative and idempotent." If every replica applies every op exactly once, in any order, the replicas converge. No transform needed. No central server needed.

For text, the most common CRDT family is RGA-style (Replicated Growable Array) or YATA, used by Yjs. Every character gets a unique ID (usually (clientID, clock)) and a parent pointer to the character it was inserted after. Insertion order is reconstructed from the IDs, not from positions.

// Conceptual shape of a Yjs-style insert.
// Every character carries its own identity.
type YItem = {
  id: { client: number; clock: number };  // unique per character
  origin: ItemID | null;                   // the char it was inserted after
  rightOrigin: ItemID | null;              // tie-breaker for concurrent inserts
  content: string;
  deleted: boolean;                        // tombstone, never removed eagerly
};

The wire format and tombstones are bigger than OT ops. But the math works out: any two replicas that have seen the same set of ops are byte-identical. No transform table. No central order.

Yjs is the de-facto library for the JavaScript world. Automerge is the Rust/JS sibling with a JSON-document-shaped API. Both are production-grade.

A side-by-side of the same edit pattern looks like this:

// Yjs: insert "X" at position 2 in a shared text doc.
import * as Y from "yjs";

const doc = new Y.Doc();
const ytext = doc.getText("body");
ytext.insert(0, "hello");
ytext.insert(2, "X");                     // "heXllo"

// The encoded update is what you send on the wire.
// Doesn't matter if 7 peers receive these in different orders.
const update = Y.encodeStateAsUpdate(doc);

// Automerge: same insert, JSON-document API.
import * as A from "@automerge/automerge";

let doc = A.from({ body: "hello" });
doc = A.change(doc, (d) => {
  d.body = d.body.slice(0, 2) + "X" + d.body.slice(2);   // "heXllo"
});

const changes = A.getChanges(A.init(), doc);
// Ship `changes` to any peer; merge order doesn't matter.

The trade: bigger payloads, heavier client memory, but you can ship a peer-to-peer editor over WebRTC, support offline-first natively, and never write a transform function in your life.

CRDT in the wild: Figma (custom CRDT, not Yjs), Linear (custom, ops + snapshots), Notion (CRDT-ish for blocks), Apple Notes sync, Roam Research clones, AppFlowy, the Y-collab ecosystem (Tldraw, Hocuspocus, Liveblocks, ProseMirror with y-prosemirror).

Choosing: three questions

Forget "which is better." The choice falls out of three questions:

1. Server-authoritative or peer-to-peer? If the server has to enforce permissions, redact content, or be the legal source of truth (medical records, financial documents, regulated content), OT fits the mental model. The server already orders everything. With CRDT you can still enforce permissions, but the server is one peer among many and the model leaks.

2. Offline-first? If users edit on a plane for six hours and expect their work to merge back in cleanly, CRDT wins. OT's offline story exists, but the transform queues get hairy after a long disconnect.

3. What's your operational complexity budget? OT means you own the transform function. Every new op type is a quadratic explosion of transform pairs you have to test. CRDT means you adopt Yjs or Automerge and accept their wire format, their memory shape, their idiosyncrasies. Neither is free.

A rough heuristic: if the answer to "where does the document live?" is "in our database, and the client is just a view," OT. If the answer is "on every device, and the server is just one replica," CRDT.

Server architecture: what you actually deploy

Either algorithm, the deployment shape is similar. Three components.

                ┌───────────────────────┐
   clients ──── │  WebSocket gateway    │  (sticky to doc shard)
                └──────────┬────────────┘
                           │
                ┌──────────▼────────────┐
                │  Doc shard (per doc)  │  ── in-memory authoritative state
                │  - applies ops        │     - OT: transform & broadcast
                │  - keeps clients map  │     - CRDT: merge & broadcast
                └──────────┬────────────┘
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
         ops log    snapshots     presence service
         (append)  (periodic)     (Redis pub/sub)

WebSocket gateway. Terminates client connections. Routes by doc_id to the right shard. Sticky session: once a client lands on a shard, all its messages go there. Use consistent hashing on doc_id so adding shards doesn't reshuffle everything.

Doc shard. One process (or actor, or goroutine, or Erlang process) per active document. Holds the current state in memory, applies incoming ops, broadcasts to every connected client for that doc. Dies when nobody's connected and the shard manager evicts it. Cold-start the next time someone opens the doc by replaying the ops log over the last snapshot.

Presence service. Who's online, where their cursor is, what color they get. This is ephemeral and high-frequency (cursor moves at 60Hz). Don't put it through the same path as document ops, or it'll drown the ops log. Redis pub/sub or a separate WebSocket channel.

For OT, the doc shard also holds the transform queue: ops the server has accepted but not yet acknowledged to the originating client. For CRDT, the shard is mostly a relay. It merges incoming ops into its replica and broadcasts to other peers, and the merge math handles the rest.

Persistence: ops log + snapshots

This is the part that's the same for both families, and the part that interviewers almost always want you to draw on the whiteboard.

You never want to write the full document to disk on every keystroke. Even a moderate-size doc is 50KB and you'd be doing 50KB writes 30 times a second per active editor. So:

Append-only ops log. Every accepted op gets written to a log keyed by (doc_id, op_seq). Postgres, DynamoDB, Cassandra, Kafka: anything that does cheap appends and ordered scans by doc. The log is the source of truth. If everything else burns down, you can rebuild any document from its log.

CREATE TABLE doc_ops (
  doc_id      uuid    NOT NULL,
  op_seq      bigint  NOT NULL,
  client_id   text    NOT NULL,
  op          jsonb   NOT NULL,
  created_at  timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (doc_id, op_seq)
);

-- Snapshots: a materialized state at a known op_seq.
CREATE TABLE doc_snapshots (
  doc_id      uuid    NOT NULL,
  op_seq      bigint  NOT NULL,
  state       bytea   NOT NULL,           -- serialized doc (OT) or Y.encodeStateAsUpdate
  created_at  timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (doc_id, op_seq)
);

Periodic snapshots. Every N ops (N = 1000 is a fine starting point), serialize the current document state and write it as a snapshot. Cold-start = load the latest snapshot + replay the ops since then. Without snapshots, a hot doc with a year of history takes minutes to open.

For Yjs specifically, snapshots are just Y.encodeStateAsUpdate(doc), the encoded merge of every update so far. Restore with Y.applyUpdate(newDoc, snapshot). Automerge has A.save(doc) / A.load(bytes) for the same purpose.

A practical pattern: ship snapshots out-of-band on a background job, not in the hot path. The doc shard accepts an op, writes it to the log, broadcasts to clients, and returns. A separate process tails the log and rolls up snapshots every N ops or every M minutes.

Undo/redo

Both OT and CRDT handle undo, but the semantics differ.

In OT, undo is generating an inverse op (delete the inserted character, re-insert the deleted run) and applying it as a fresh op. The transform machinery does the rest. If someone else edited around your op in the meantime, the inverse gets transformed and lands cleanly.

In CRDT (Yjs in particular), undo is done by an UndoManager that tracks which ops belong to the local user and inverts them on demand. The trick is that the inverse has to respect concurrent edits. If someone replied to your sentence, undoing your sentence doesn't take their reply with it.

Both libraries ship reasonable defaults. The interview answer: "Per-user undo stacks. The inverse is generated against the current state, not the original state, so concurrent edits survive."

Conflict UI: what the user actually sees

True conflicts in text are rarer than you'd think. Two people typing at adjacent positions isn't a conflict, since both insertions land and the order is determined by the algorithm. A real conflict in a rich-text doc shows up at the semantic layer:

Two users edit the same cell of a table.
One user deletes a paragraph another user is editing.
Concurrent format changes (one user bolds a run, another italicizes the overlapping run).

The honest UI answer is: most of the time, you don't show anything. The doc converges, the user reads the result. For the cases that matter, like a deleted block someone was typing into, surface a non-blocking notice ("Someone deleted this section while you were editing. Restore?") and offer one-click revert. Don't put a modal in the user's face.

Figma's approach is the gold standard: their conflict-on-property-change leaves both edits applied in op order, and you can scrub the version history to see what happened.

The 90-second answer

When the interviewer asks "design Google Docs," this is the shape:

"I'd start with the algorithm. OT if the server is authoritative: Google Docs, anything with strict permissions or compliance. CRDT if I need offline-first or peer-to-peer: Figma, Linear, anything where the client is a real replica.

Either way the architecture is the same: WebSocket gateway, sticky-routed by doc ID to a per-document shard process that holds in-memory state. The shard accepts ops, applies them (transforms for OT, merges for CRDT), broadcasts to other connected clients, and appends to a persistent ops log. Periodic snapshots roll the log up so cold-start is fast.

Presence (cursors, selections, who's online) goes through a separate channel. Redis pub/sub. Don't pollute the ops log with cursor moves.

Undo is per-user inverse ops generated against the current state. Conflict UI is mostly invisible, since the algorithm converges. For semantic conflicts (deleted-while-editing), surface a non-blocking notice with a restore option.

The gotcha I'd call out: CRDT state grows. Every tombstone is forever unless you have a garbage-collection story. Yjs and Automerge both ship one. I'd budget for it from day one."

That's 90 seconds. It hits algorithm choice, server shape, persistence, presence, undo, conflict UI, and one operational risk. Interviewer follow-ups will dig into whichever of those you said with the least confidence.

The gotcha: CRDT garbage collection

Here's the bug that hits CRDT systems in production around month 18: the document size on disk keeps growing even when the user perception is that the doc shrank.

CRDTs work by keeping tombstones. When you delete a character, the character object stays in the structure with deleted: true. The tombstone has to exist so that concurrent edits referencing that character (an insert "after this character") have something to anchor to. If you eagerly remove the character, a delayed peer with an old reference can't merge.

So your "empty" doc with one paragraph of text is actually carrying every character that's ever been typed, deleted, and re-typed. A doc that's been edited heavily for a year is comically larger than the visible text.

Yjs ships a garbage collector that runs when you call Y.encodeStateAsUpdate(doc): it collapses tombstones whose causal context is no longer reachable. Automerge has A.save which produces a compacted form. Both work, both have caveats:

GC requires that all peers are caught up past the GC point. A peer that's been offline for two weeks and tries to merge against a GC'd state will fail or produce a corrupted merge. The library raises this, but you have to handle it, usually by forcing the lagging peer to re-fetch a fresh snapshot instead of merging old updates.
GC is expensive on big docs. Don't run it in the request path. Run it in a background job, ideally during low-traffic windows.
The compacted form is not byte-identical across runs of the GC. Don't use document bytes as a cache key after compaction.

In Yjs the practical setup looks like this:

import * as Y from "yjs";

// On the server, every N ops or every M minutes:
async function compactDoc(docId: string) {
  const doc = await loadDoc(docId);

  // Encode + decode with GC enabled. This is the compaction step.
  const compacted = Y.encodeStateAsUpdate(doc);

  // Write the compacted bytes as the new snapshot.
  await saveSnapshot(docId, compacted);

  // Truncate the ops log up to the snapshot point.
  // Any peer ahead of this point must be told to refetch.
  await truncateOpsLog(docId, doc.store.clientStates);
}

The interview-flavored version of this gotcha: "CRDT state grows because tombstones are forever. You need a GC pass that runs in the background, and a story for peers that have fallen behind the GC horizon, typically forcing a fresh snapshot pull instead of an incremental merge."

OT doesn't have this problem because the server can prune ops freely once everyone has acknowledged them. The trade is that OT puts the burden on the transform function instead.

If this was useful

If you're prepping for system design interviews and want this same step-by-step treatment for the other 14 designs interviewers actually ask (sharded counters, payment ledgers, rate limiters, news feeds), that's the spine of my System Design Pocket Guide: Interviews — 15 Real System Designs, Step by Step. The collaboration chapter goes deeper on the OT transform table and walks through a Yjs server you can actually run.

Which side do you fall on for your own product: OT because the server is the source of truth, or CRDT because the client has to keep working when the network doesn't?

Design a Job Scheduler at 10M Jobs/Day: 4 Components, 3 Failure Modes

Gabriel Anhaia — Sun, 24 May 2026 13:41:48 +0000

Book: System Design Pocket Guide: Interviews — 15 Real System Designs, Step by Step
Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Cron is fine until it isn't. One box, a flat file, a tab character that nobody can type from memory. Works great for the nightly backup. Then somebody asks you to fire ten million scheduled jobs a day, across timezones, with sub-minute accuracy, and the interviewer wants to know what k8s CronJob controllers and Airflow schedulers do under the hood.

This is one of the cleanest system design prompts to practice on because the failure modes are real, the components are small, and the wrong answer is loud. Let's build it.

What "scheduler" actually means

A scheduler is four things glued together. People conflate them and the design suffers.

A schedule store: the durable truth about what should run and when.
A tick coordinator: the thing that wakes up every second (or 100ms, or 10ms), reads the store, and decides "these N jobs are due now."
A dispatcher: the thing that hands due jobs to a queue. Owns idempotency keys.
An executor: the worker pool that actually runs the job, with timeouts and retries.

Confuse "dispatcher" with "executor" and you get a design where the same box reads the schedule and runs the user's code. That's cron. It doesn't scale past one machine because the work the executor does (run arbitrary user code for arbitrary lengths of time) has wildly different failure modes from the work the dispatcher does (publish a tiny message to a queue, deterministically).

Keep these separated. The interviewer will nod.

Component 1: the schedule store

This is the part most candidates get wrong because they reach for Redis. Redis is fine for the due-now queue. It is not the source of truth for "every Tuesday at 14:30 Europe/Berlin, run job 47 forever."

The store needs three things: durability, an index on the next firing time, and the ability to update both atomically when a job runs. Postgres works at this scale. So does CockroachDB or any decent OLTP database with a proper index.

CREATE TABLE scheduled_jobs (
    job_id           UUID PRIMARY KEY,
    tenant_id        UUID NOT NULL,
    name             TEXT NOT NULL,
    payload          JSONB NOT NULL,
    -- cron expression OR a one-shot timestamp; never both
    cron_expr        TEXT,
    timezone         TEXT NOT NULL DEFAULT 'UTC',
    -- the calculated next firing time, always in UTC
    next_run_at      TIMESTAMPTZ NOT NULL,
    last_run_at      TIMESTAMPTZ,
    -- versioned so dispatcher can do optimistic locking
    version          BIGINT NOT NULL DEFAULT 0,
    enabled          BOOLEAN NOT NULL DEFAULT true,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    CONSTRAINT one_of_schedule
      CHECK ((cron_expr IS NULL) <> (next_run_at IS NOT NULL))
);

-- the only index the tick coordinator hits
CREATE INDEX scheduled_jobs_due_idx
  ON scheduled_jobs (next_run_at)
  WHERE enabled = true;

-- audit table, append-only, never updated
CREATE TABLE scheduled_runs (
    run_id           UUID PRIMARY KEY,
    job_id           UUID NOT NULL REFERENCES scheduled_jobs(job_id),
    scheduled_for    TIMESTAMPTZ NOT NULL,
    dispatched_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
    -- the idempotency key the executor will use
    idempotency_key  TEXT NOT NULL UNIQUE,
    status           TEXT NOT NULL  -- 'dispatched','succeeded','failed','skipped'
);

A few things to call out before the interviewer asks.

next_run_at is always UTC. The timezone column exists so you can re-compute the next firing time correctly when DST hits. Storing local time in the index is how you end up firing twice in November and zero times in March.

The version column is the optimistic lock the dispatcher uses to claim a job. No SELECT ... FOR UPDATE row locks held across a tick; that doesn't scale. The dispatcher reads candidates, then issues an UPDATE ... WHERE version = $expected and only proceeds if the row count is 1.

The partial index on enabled = true is a real win at 10M rows. You almost never query disabled jobs, and the index stays small.

The scheduled_runs audit table is append-only. This is the table you'll be glad you had at 03:00 when someone asks "did job 47 actually run on Tuesday or did we just say it did."

A back-of-envelope sanity check: 10M jobs/day is ~116 inserts/second sustained on scheduled_runs. A single Postgres primary handles that without sweating. The schedule store itself is mostly read traffic.

Component 2: the tick coordinator

The tick coordinator is the dangerous part. One job here: every second, find the jobs whose next_run_at <= now() and hand them to the dispatcher.

Single instance is a single point of failure. Multiple instances without coordination is duplicate dispatch. The textbook answer is leader election: run N instances, exactly one is leader, the rest are warm standbys.

// the leader-election loop, etcd-style
func RunCoordinator(ctx context.Context, etcd *clientv3.Client) error {
    session, err := concurrency.NewSession(etcd, concurrency.WithTTL(10))
    if err != nil { return err }
    defer session.Close()

    election := concurrency.NewElection(session, "/scheduler/leader")

    // blocks until we win the election
    if err := election.Campaign(ctx, hostname()); err != nil {
        return err
    }

    log.Info("became leader, starting tick loop")
    return tickLoop(ctx, session.Done())
}

func tickLoop(ctx context.Context, lost <-chan struct{}) error {
    // 1-second ticks. Don't try to be cleverer than this.
    ticker := time.NewTicker(time.Second)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-lost:
            // we lost the lease; step down immediately
            return fmt.Errorf("leader lease lost")
        case now := <-ticker.C:
            // hand the tick to the dispatcher; don't block here
            if err := dispatchDue(ctx, now); err != nil {
                log.Error("dispatch failed", "err", err)
            }
        }
    }
}

Two non-obvious things here.

The <-lost channel matters more than the Campaign call. etcd's lease can be lost without you noticing if you only listen for context cancellation. If you keep dispatching after losing the lease, the new leader is also dispatching, and you've recreated the duplicate-dispatch problem you were trying to avoid. Step down the moment the session ends.

Don't tick faster than your dispatch can finish. If dispatchDue takes longer than 1 second sometimes (it will, under load), tickers in Go won't queue; the next tick is dropped. That's the behavior you want. What you don't want is goroutine-per-tick where you fan out and overlap dispatches. Serialize the tick. Parallelism happens inside dispatchDue, where it's bounded.

For 10M jobs/day, peak rate matters more than average. If 60% of jobs are on 0 0 * * * (midnight UTC), you're dispatching 6M jobs in a 1-minute window. The tick coordinator doesn't run those; it hands them off. So the question becomes how fast the dispatcher can drain that backlog.

Component 3: the dispatcher

The dispatcher reads due jobs, publishes to a queue (SQS, Kafka, RabbitMQ; pick one), updates the schedule store with the next firing time, and writes an entry to scheduled_runs. All under an idempotency key.

func dispatchDue(ctx context.Context, tickAt time.Time) error {
    // batch read, process in chunks so a single tick can drain a spike
    const batchSize = 1000
    for {
        rows, err := db.QueryContext(ctx, `
            SELECT job_id, tenant_id, name, payload,
                   cron_expr, timezone, next_run_at, version
            FROM scheduled_jobs
            WHERE enabled = true
              AND next_run_at <= $1
            ORDER BY next_run_at
            LIMIT $2
        `, tickAt, batchSize)
        if err != nil { return err }

        jobs, err := scanJobs(rows)
        if err != nil { return err }
        if len(jobs) == 0 { return nil }

        for _, j := range jobs {
            // idempotency key = job + scheduled instant.
            // Same key for retries; never collides across schedules.
            idem := fmt.Sprintf("%s:%d", j.JobID, j.NextRunAt.Unix())

            nextRun := computeNext(j.CronExpr, j.Timezone, tickAt)

            // claim-and-advance in one statement
            res, err := db.ExecContext(ctx, `
                UPDATE scheduled_jobs
                SET next_run_at = $1,
                    last_run_at = $2,
                    version     = version + 1
                WHERE job_id = $3 AND version = $4
            `, nextRun, j.NextRunAt, j.JobID, j.Version)
            if err != nil { return err }

            n, _ := res.RowsAffected()
            if n == 0 {
                // another dispatcher (shouldn't happen with leader
                // election, but belt-and-braces) already claimed it
                continue
            }

            if err := publish(ctx, j, idem); err != nil {
                // queue down; the job will be picked up next tick
                // because we'll revert the version on a retry tick
                log.Warn("publish failed", "job", j.JobID, "err", err)
                continue
            }

            _, _ = db.ExecContext(ctx, `
                INSERT INTO scheduled_runs
                  (run_id, job_id, scheduled_for, idempotency_key, status)
                VALUES ($1, $2, $3, $4, 'dispatched')
            `, uuid.New(), j.JobID, j.NextRunAt, idem)
        }

        if len(jobs) < batchSize { return nil }
    }
}

The idempotency key is job_id + scheduled_instant_unix. Two retries of the same scheduled fire share the key. Two different schedulings of the same job (today's midnight and tomorrow's midnight) get different keys because the timestamp differs. The executor uses this key to enforce at-most-once execution at the application layer if the user opted in.

Note the at-most-once vs at-least-once choice lives here, not in the queue. The queue is at-least-once (because SQS, Kafka, RabbitMQ all are). The dispatcher's INSERT into scheduled_runs happens after publish, so if the publish succeeds and the insert fails, you get a duplicate dispatch on the next tick. Flip the order and you can lose dispatches. Pick one, document the failure mode loudly. Most real schedulers pick at-least-once because losing jobs is worse than running them twice.

Component 4: the executor

The executor is a worker pool consuming from the queue. Standard stuff: pull a message, run the user's job with a timeout, ack on success, nack on failure with a retry policy.

import asyncio
import signal
from contextlib import asynccontextmanager

class Executor:
    def __init__(self, queue, registry, *, concurrency=64,
                 default_timeout=300):
        self.queue = queue
        self.registry = registry          # name -> handler
        self.sem = asyncio.Semaphore(concurrency)
        self.default_timeout = default_timeout
        self._running = True

    async def run(self):
        while self._running:
            msg = await self.queue.receive()
            if msg is None:
                continue
            asyncio.create_task(self._handle(msg))

    async def _handle(self, msg):
        async with self.sem:
            handler = self.registry.get(msg.job_name)
            if handler is None:
                # unknown job; quarantine, don't drop silently
                await self.queue.dead_letter(msg, reason="unknown_handler")
                return

            timeout = msg.timeout_s or self.default_timeout
            try:
                async with asyncio.timeout(timeout):
                    await handler(msg.payload, idempotency_key=msg.idem)
                await self.queue.ack(msg)
            except TimeoutError:
                # log scheduled_for, not now(): the audit trail wants the
                # original tick, not the time the worker happened to drain it
                await self.queue.nack(msg, retryable=True,
                                      reason="timeout")
            except Exception as e:
                await self.queue.nack(msg, retryable=is_retryable(e),
                                      reason=str(e))

Two things to highlight for the interviewer.

The default timeout is per-job, not global. A schedule that runs send_daily_email and one that runs rebuild_search_index have different tolerances. Storing timeout_s on the job and shipping it with the message means workers don't need to know about job semantics.

The bounded semaphore is what keeps a single worker from blowing out memory when a spike arrives. Without it, every received message creates a task immediately and you fan out to whatever the queue gave you. 10k pending tasks waiting on shared resources is how an executor box dies.

Failure mode 1: missed ticks during failover

The leader dies. The standby takes over. The transition isn't instant; etcd's TTL is 10 seconds in the code above, so up to 10 seconds of ticks didn't fire.

What did you miss? Anything with next_run_at between the leader's death and the new leader's first tick.

The fix is the catch-up window. The dispatcher query is next_run_at <= $1 where $1 is the current tick instant, not next_run_at = $1. So when the new leader fires its first tick, it picks up everything that was due during the gap. Late, but fired.

For schedules where lateness matters (cron, but not "the next-run-at" semantics of a one-shot job), this is the right tradeoff. For schedules where firing late is actively wrong (a stock-market open trigger at 09:30:00 doesn't want a 09:30:09 firing), you need a different model. The job carries a "skip if older than" tolerance, and the dispatcher checks now() - next_run_at < tolerance before publishing.

The catch-up window also covers the case where the dispatcher itself is slow. If a tick takes 8 seconds to drain a spike, the next tick fires immediately when the previous one returns, and it sees everything that became due during the slow batch.

What you do not want is the new leader rewinding its clock and re-firing already-dispatched ticks. The version check in the UPDATE plus the audit table's unique constraint on idempotency_key are the belt and braces against that.

Failure mode 2: clock skew across executors

Your tick coordinator's clock is 400ms ahead of one executor box. The executor box's clock is 600ms behind a third box. A job named mark_subscription_expired runs at 00:00:00 UTC against a row that has expires_at = 00:00:00.000. The executor's clock says it's 23:59:59.4. The row isn't expired yet.

This isn't a thought experiment. It's a class of bug that gets shipped quarterly somewhere in the industry.

Two rules. UTC everything. No localtime in the schedule store, no localtime in the queue message, no localtime in the executor's comparisons. The user's timezone is metadata used to compute next_run_at once, not a thing you carry around at dispatch time.

NTP is non-negotiable. Every box runs chronyd or systemd-timesyncd against a known-good pool. AWS EC2 has its time-sync endpoint at 169.254.169.123; GCP has Google's NTP servers; on-prem teams run their own stratum-2. The interviewer wants to hear that you'd alert on chrony tracking reporting more than 100ms of offset.

A third belt-and-braces move: include scheduled_for in every queue message and have the executor compare its own clock to that timestamp on receipt. If the executor sees scheduled_for = 00:00:00 and its own clock says 23:59:59, it knows to either wait or refuse, and either is better than running the job against a state that isn't yet the state the schedule was designed for.

Failure mode 3: long jobs blocking the next tick

The dispatcher publishes to a queue. The executor pulls from the queue. A job that takes 4 hours doesn't block the dispatcher, it blocks an executor worker. This is the right separation, and it's exactly the reason dispatcher and executor are different components.

The failure shape: a job takes 4 hours. The schedule fires every 1 hour. By hour 5, you have five concurrent runs of the same job, all working on overlapping data, racing each other to corrupt state.

The schedule store needs one more field for this.

ALTER TABLE scheduled_jobs ADD COLUMN concurrency_policy TEXT
    NOT NULL DEFAULT 'allow'
    CHECK (concurrency_policy IN ('allow','forbid','replace'));

Three policies, taken straight from k8s CronJob because Kubernetes already solved this and there's no point re-inventing the vocabulary.

allow: fire it, let them overlap. Default for stateless jobs.
forbid: skip the new run if the previous run hasn't completed.
replace: cancel the previous run and start the new one.

The dispatcher enforces this by checking the most recent scheduled_runs row for the job before publishing. forbid means: if there's a dispatched row without a succeeded/failed row, write a skipped row and don't publish. replace means: send a cancellation message for the in-flight run, then publish the new one.

Queue depth isn't execution depth. A backed-up queue and a backed-up executor pool look the same to a naive dashboard. You want metrics for both, and the alert that matters is executions_in_flight > expected_max for any job tagged forbid. That's the one that catches the silent overlap.

The 90-second answer

You'll be asked to summarize at the end. Here's the version that fits in the time you'll be given:

"Four components, kept separate. A schedule store in Postgres with the cron expression, the next-run-at as UTC, and a partial index on the next-run-at where enabled is true. A tick coordinator running with etcd leader election so exactly one instance is firing per second; standbys are warm. A dispatcher that reads due rows in batches, advances the next-run-at with an optimistic version check, then publishes to SQS with an idempotency key that's job_id + scheduled_unix. An executor worker pool consuming from the queue, with per-job timeouts and a bounded concurrency semaphore. The audit table scheduled_runs is append-only and tells operators exactly what was dispatched and when.

Three failure modes I'd call out. Missed ticks during failover: the dispatcher query is next_run_at <= now(), not equality, so the catch-up window absorbs the leader-transition gap. Clock skew: UTC stored, NTP enforced via chronyd, alert on >100ms offset, and scheduled_for ships in the message so the executor can sanity-check its own clock. Long jobs overlapping their own next run: concurrency_policy column with allow/forbid/replace, dispatcher checks the latest scheduled_runs row before publishing. Queue depth and execution depth are different metrics; alerting on one without the other hides the overlap."

That's the answer. 90 seconds, hits every component, names the failure modes by their actual shapes, and shows you know that cron and Airflow and k8s CronJob all converge on roughly this architecture for the same reasons.

If this was useful

This walk-through is the exact shape the System Design Pocket Guide: Interviews uses across all 15 designs in the book: components first, failure modes second, a 90-second summary you can actually deliver in the room. The job-scheduler chapter goes deeper into the catch-up-window math and the cron-expression edge cases (DST, leap seconds, the L and W operators most parsers get wrong). If you liked this one, that's the chapter to start with.

What part of the scheduler do you spend the most time arguing about in interviews: the leader-election story, the at-least-once vs at-most-once tradeoff, or the concurrency-policy column? Drop your take in the comments.

Design a Feature Flag Service: 100k SDK Clients and the SSE Protocol Reframe

Gabriel Anhaia — Sun, 24 May 2026 12:05:35 +0000

Book: System Design Pocket Guide: Interviews — 15 Real System Designs, Step by Step
Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

"Design a feature flag service" sounds soft. It's the kind of prompt candidates think they can wing because they've used LaunchDarkly and Unleash. Then the interviewer asks the follow-up: "Your SDK ships in 100,000 production processes. How do they know when a flag changes?"

The naive answer hits the load balancer first. The right answer reframes the protocol before drawing a single box.

The interviewer's hidden question: how do you scale READS?

The shape of the system is asymmetric. Writes are rare: a product manager toggles a flag a few times a day. Reads are everywhere. Every request your fleet handles asks "is this flag on for this user?" at least once, sometimes dozens of times.

100,000 SDK clients is not your user count. It's your process count. If your app runs on 5,000 pods and each pod is a separate SDK instance, you're at 5,000. If your mobile app has a million daily actives and each device runs the SDK, you're at a million. The interviewer says "100k" to anchor you somewhere realistic for a mid-size SaaS. Push them on it. Ask whether the SDKs live in your backend fleet or on end-user devices. The protocol changes.

The real question under the prompt: given that flag config is small (kilobytes) and changes rarely, how do you make every SDK in the world see the new value in under a second without melting your origin?

If you start sketching MySQL and a GET /flags/:key endpoint, you've answered the wrong question. The interviewer wants you to notice that this is a read-distribution problem with a publish-subscribe shape, not a CRUD app.

Naive design: REST polling, and why it dies at 100k clients

The first instinct is GET /api/flags. The SDK polls every 30 seconds. Toggle latency is bounded by the poll interval. Done.

Walk the math on the whiteboard. 100,000 clients, 30-second poll, that's 3,333 requests per second on average. Sustainable. Except every SDK starts at process boot and processes boot in waves: a deploy of 5,000 pods finishes in 90 seconds and you've stacked thousands of polls on top of each other. Now your p99 latency on the flag endpoint spikes and your SDKs time out at startup, which means your app boots without flags, which means defaults everywhere, which means a silent incident.

Lower the poll interval to 5 seconds and you're at 20,000 req/s sustained. You can cache aggressively at the edge, but you've also turned a flag toggle into a 5-second worst-case propagation. Not interview-grade.

Raise the interval to 5 minutes and your incident-response story collapses. The SRE flips the kill switch and waits five minutes for the bad code path to stop firing. The PM who shipped the broken experiment is already on a call.

The pattern: polling forces a tradeoff between propagation latency and origin load that gets worse linearly with client count. There is no value of pollInterval that wins. The protocol itself is wrong.

The reframe: SSE push from server, clients hold the connection

Server-Sent Events. One long-lived HTTP connection per SDK, opened at startup, held open by the server, written to only when a flag changes. The flag toggle becomes O(N) writes across N open sockets instead of O(N) polls per interval.

Why SSE over WebSocket: unidirectional fits the problem (server tells client, client never tells server), it's plain HTTP so corporate proxies don't choke, browsers and most HTTP libraries support it natively, and reconnect-with-Last-Event-ID is part of the spec. WebSocket is fine if you also need client-to-server messages, but for flag distribution you don't.

Here's the wire-level protocol an SDK should implement. Real SSE, not pseudo-code:

import requests
import json
import time
from typing import Iterator

class FlagStream:
    def __init__(self, base_url: str, sdk_key: str, env: str):
        self.base_url = base_url
        self.sdk_key = sdk_key
        self.env = env
        self.last_event_id: str | None = None
        self.flags: dict[str, dict] = {}

    def stream(self) -> Iterator[dict]:
        # exponential backoff on disconnect — flags service is best-effort
        backoff = 1.0
        while True:
            try:
                headers = {
                    "Authorization": f"Bearer {self.sdk_key}",
                    "Accept": "text/event-stream",
                    "Cache-Control": "no-cache",
                }
                if self.last_event_id:
                    headers["Last-Event-ID"] = self.last_event_id

                url = f"{self.base_url}/sdk/stream?env={self.env}"
                # stream=True is the whole point — don't buffer the response
                with requests.get(url, headers=headers, stream=True,
                                  timeout=(5, None)) as resp:
                    resp.raise_for_status()
                    backoff = 1.0  # reset on successful connect
                    yield from self._parse_events(resp)
            except (requests.RequestException, ConnectionError):
                # keep last-known flags; never block app startup on this
                time.sleep(min(backoff, 30))
                backoff *= 2

    def _parse_events(self, resp) -> Iterator[dict]:
        event_type = "message"
        data_buffer = []
        event_id = None

        for raw in resp.iter_lines(decode_unicode=True):
            if raw == "":
                # blank line = dispatch the event
                if data_buffer:
                    payload = json.loads("\n".join(data_buffer))
                    if event_id:
                        self.last_event_id = event_id
                    yield {"type": event_type, "data": payload}
                event_type = "message"
                data_buffer = []
                event_id = None
                continue

            if raw.startswith(":"):
                continue  # SSE comment / keepalive
            field, _, value = raw.partition(":")
            value = value.lstrip(" ")
            if field == "event":
                event_type = value
            elif field == "data":
                data_buffer.append(value)
            elif field == "id":
                event_id = value

The server side is symmetrically simple. On connect, send the full flag snapshot as a put event. On every subsequent change, send a patch event with just the diff.

# server-side SSE handler — FastAPI / Starlette
from fastapi import FastAPI, Request
from sse_starlette.sse import EventSourceResponse
import asyncio

app = FastAPI()

@app.get("/sdk/stream")
async def stream(request: Request, env: str, last_event_id: str | None = None):
    async def event_generator():
        # 1) snapshot — bring the client to current state
        snapshot, version = await flag_store.get_snapshot(env)
        yield {
            "event": "put",
            "id": str(version),
            "data": json.dumps({"flags": snapshot, "version": version}),
        }

        # 2) live patches — pubsub fan-out from the write path
        async for change in flag_pubsub.subscribe(env, since=version):
            if await request.is_disconnected():
                break
            yield {
                "event": "patch",
                "id": str(change.version),
                "data": json.dumps(change.diff),
            }

    return EventSourceResponse(
        event_generator(),
        ping=15,  # send :keepalive every 15s for proxy timeouts
    )

Capacity changes character. A modern Linux box holds 200k+ open SSE connections with a sane file-descriptor limit and a non-blocking server. Five such gateway nodes cover the 100k fleet with 5x headroom. The hard part stops being throughput and becomes connection lifecycle: graceful drain on deploy, half-open detection, idle-killer proxies in front of you.

Edge caching: flag evaluations at the CDN edge for read-anywhere clients

For browser SDKs and mobile SDKs, even SSE is too chatty. Every cold start opens a connection, downloads the snapshot, then holds an idle socket. On a flaky mobile network you'd rather not.

The reframe again: push the evaluation to the edge. Flag config is small enough that it fits in a CloudFlare Worker, a Fastly Compute@Edge function, or a Lambda@Edge handler. The SDK calls one HTTP endpoint, the edge worker has the flag rules cached locally, and the answer comes back from the nearest PoP in 30ms.

// CloudFlare Worker — evaluates a single flag at the edge
// flag config is hydrated from KV (CF's edge KV store)

export default {
  async fetch(req, env) {
    const url = new URL(req.url);
    const flagKey = url.pathname.split("/").pop();
    const ctx = await req.json(); // { userId, attrs }

    // KV hit is sub-ms at the edge; miss falls through to origin
    const cfgRaw = await env.FLAGS_KV.get(`flag:${flagKey}`, "json");
    if (!cfgRaw) {
      return new Response(JSON.stringify({ value: null, reason: "UNKNOWN" }),
        { status: 404 });
    }

    const result = evaluate(cfgRaw, ctx);

    // 60s edge cache, but vary on the bucket — not the userId itself
    // (don't blow up the cache key space)
    const bucket = stickyBucket(ctx.userId, flagKey);
    const headers = new Headers({
      "Content-Type": "application/json",
      "Cache-Control": "public, s-maxage=60",
      "X-Flag-Bucket": String(bucket),
    });
    return new Response(JSON.stringify(result), { headers });
  },
};

function evaluate(cfg, ctx) {
  // 1) kill switch — fast path
  if (!cfg.enabled) {
    return { value: cfg.offVariation, reason: "OFF" };
  }

  // 2) targeting rules — explicit user/segment overrides
  for (const rule of cfg.rules ?? []) {
    if (matches(rule.clauses, ctx.attrs)) {
      return { value: rule.variation, reason: "TARGET_MATCH" };
    }
  }

  // 3) percentage rollout — bucket the user, compare to threshold
  const bucket = stickyBucket(ctx.userId, cfg.salt);
  for (const variant of cfg.rollout) {
    if (bucket < variant.cumulativeWeight) {
      return { value: variant.value, reason: "ROLLOUT" };
    }
  }
  return { value: cfg.fallthrough, reason: "FALLTHROUGH" };
}

Edge cache invalidation runs off the same pubsub channel the SSE gateway uses. When a flag changes, you push a KV update to every edge PoP and the next request reads the new config. Propagation is dominated by KV replication time, which on CloudFlare KV is sub-second globally.

The gotcha: edge caching only works for flag values that don't depend on per-request secrets. If the flag rules reference ctx.attrs.email to do regex matching, you can't cache the response; the cache key would explode. Restrict edge evaluation to flags with bucket-based rollouts and named-segment matches; route attribute-heavy evaluations back to origin.

Flag-config distribution: durable store, snapshot to object storage, push diffs over pubsub

Behind the gateway and the edge sits the source of truth. The write path is low-traffic: a dashboard call writes a new flag version to a relational store (Postgres, simple), bumps a monotonic version counter, and publishes a change event to Redis pubsub or NATS.

The read path is everything. Gateway pods don't read Postgres on every SDK request; they hold the flag set in process memory and subscribe to the same pubsub channel. On boot, a gateway reads a snapshot from S3 (refreshed by a background job every 30s) so a cold restart of the entire fleet doesn't herd against Postgres.

S3 plus CloudFront is also the SDK-side fallback channel. If the SSE connection won't establish (corporate proxy strips the connection, mobile network is hostile, the gateway is down), the SDK falls back to a 60-second polled GET of flags-{env}-{version}.json from a public CloudFront URL. Slower, lossier, but the application boots with real flag values instead of compiled-in defaults.

The pattern to name in the interview: write to a durable store, snapshot to object storage for cold-start, push diffs over pubsub for hot fan-out, fall back to polled snapshots for hostile networks. Three independent paths, ranked by cost and latency.

SDK-side caching with TTL + fallback (offline-safe)

Every flag evaluation must answer in microseconds. The SDK keeps the full flag set in memory and serves evaluations locally. The SSE stream keeps the in-memory copy fresh.

When the connection drops, the SDK keeps serving the last-known values. That's the offline-safe property. No timeout, no fallback default unless the SDK has never received a snapshot. Add a lastSyncedAt field on the SDK that your monitoring scrapes; a process that's been disconnected from the flag service for 10 minutes is a real signal, but it shouldn't crash the request path.

Compiled-in defaults belong in the application code, not the SDK. The contract is: if the SDK has no value for this key, return the default that the calling code supplied. The application owner decides what "off" looks like for that specific flag, not the flag platform.

Targeting and rules engine: boolean predicates, rollout percentages, sticky bucketing

Three primitives, in order: kill switch, targeting rules, percentage rollout. Already shown in the edge evaluator. Worth saying out loud in the interview because it answers "what's a flag actually evaluating?"

Sticky bucketing is the load-bearing piece. When you say "5% rollout", a given user must always land in the same bucket. Otherwise the user flips between treatment and control across requests, ruins your experiment, and corrupts your analytics.

# the bucketing hash — identical implementation on every SDK,
# every edge worker, every backend evaluator. one source of truth.
import hashlib

def sticky_bucket(user_id: str, flag_salt: str, total_buckets: int = 10_000) -> int:
    # SHA-1, not for security — for stable, uniform distribution across languages.
    # Every SDK ships the same impl. If you swap algorithms you re-bucket
    # every user mid-experiment, which is a silent data corruption bug.
    h = hashlib.sha1(f"{flag_salt}.{user_id}".encode("utf-8")).digest()
    # take 4 bytes, big-endian, mod the bucket count
    n = int.from_bytes(h[:4], "big")
    return n % total_buckets

Notice the salt. Without it, the same user lands in bucket 4,217 for every flag, so a user in the 5% rollout of flag A is also in the 5% rollout of flag B, C, and D. Correlation across experiments destroys your stats. The salt is usually the flag key itself plus a per-flag random string set at flag creation.

Why total_buckets=10_000: lets you express rollout in basis points (0.01% granularity), enough precision for ramp schedules and small canary groups.

The 90-second answer that wins the round

When the interviewer drops the prompt, talk for 90 seconds before drawing anything:

"Feature flags are a read-heavy, write-rare distribution problem. The naive GET /flags design fails at 100k SDK clients because polling forces a bad tradeoff between propagation latency and origin load. I'd reframe to a push protocol: SSE from a stateless gateway tier, with each SDK holding one long-lived connection that receives a full snapshot on connect and patches on every flag change. The gateways subscribe to a pubsub channel (Redis or NATS) that the write path publishes to. The source of truth is Postgres for writes and S3-plus-CDN for cold-start snapshots, so a fleet restart doesn't herd. For browser and mobile SDKs I'd push evaluation to the edge via CloudFlare Workers or Lambda@Edge, with flag config replicated to edge KV; that gives 30ms response from the nearest PoP. SDKs cache locally and evaluate in microseconds. Sticky bucketing uses a salted SHA-1 hash with the same implementation in every SDK, edge worker, and backend evaluator. The whole system is offline-safe because the SDK keeps serving last-known values when the stream drops."

That's the answer. Now they ask follow-ups and you draw boxes.

The gotcha: sticky bucketing requires a deterministic hash, and every SDK must agree

The single failure mode that sinks real flag platforms: hash drift across SDKs.

Your Node SDK uses MurmurHash3 because someone copy-pasted from LaunchDarkly's old open-source SDK. Your Go SDK uses FNV-1a because Go's stdlib has it. Your Python SDK uses MD5 because that's what the first engineer reached for. The same user gets bucketed three different ways. You roll out a flag to 10% and you actually hit 27% of users because each SDK has independent randomness.

Worse: the bug is invisible. Aggregate counts look right (10% of evaluations across the fleet return the new variant), but per-user consistency is gone. An experiment that should detect a 2% conversion lift sees noise. A canary that should affect 5% of traffic affects different 5%-slices in different SDKs.

The fix is governance, not code: one bucketing spec, written down, with test vectors. Every SDK ships a test_sticky_bucket.py (or _test.go, etc.) with at least 20 (user_id, flag_salt) -> expected_bucket pairs. CI fails if any SDK disagrees with the canonical vectors. When you change the algorithm, you bump a bucketingVersion field on every flag and run the old and new algorithms in parallel during the cutover.

If the interviewer is sharp they'll ask about this. Bring it up unprompted and you've shown you've actually shipped one of these systems.

If this was useful

This pattern (protocol reframe, push beats poll, edge evaluation, deterministic bucketing) is one of fifteen full system designs walked end-to-end in System Design Pocket Guide: Interviews. The feature-flag design lives next to the rate-limiter, the URL shortener, and the notification system; each one structured around the 90-second answer and the follow-up questions that actually decide the round.

What's the gnarliest follow-up you've been asked on this kind of design? Drop it in the comments and I'll work through it.

Design a Feature Flag Service: 100k SDK Clients and the SSE Protocol Reframe

Gabriel Anhaia — Sun, 24 May 2026 12:05:34 +0000

Book: System Design Pocket Guide: Interviews — 15 Real System Designs, Step by Step
Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

The naive answer hits the load balancer first. The right answer reframes the protocol before drawing a single box.

The interviewer's hidden question: how do you scale READS?

Naive design: REST polling, and why it dies at 100k clients

The first instinct is GET /api/flags. The SDK polls every 30 seconds. Toggle latency is bounded by the poll interval. Done.

The reframe: SSE push from server, clients hold the connection

Here's the wire-level protocol an SDK should implement. Real SSE, not pseudo-code:

import requests
import json
import time
from typing import Iterator

class FlagStream:
    def __init__(self, base_url: str, sdk_key: str, env: str):
        self.base_url = base_url
        self.sdk_key = sdk_key
        self.env = env
        self.last_event_id: str | None = None
        self.flags: dict[str, dict] = {}

    def stream(self) -> Iterator[dict]:
        # exponential backoff on disconnect — flags service is best-effort
        backoff = 1.0
        while True:
            try:
                headers = {
                    "Authorization": f"Bearer {self.sdk_key}",
                    "Accept": "text/event-stream",
                    "Cache-Control": "no-cache",
                }
                if self.last_event_id:
                    headers["Last-Event-ID"] = self.last_event_id

                url = f"{self.base_url}/sdk/stream?env={self.env}"
                # stream=True is the whole point — don't buffer the response
                with requests.get(url, headers=headers, stream=True,
                                  timeout=(5, None)) as resp:
                    resp.raise_for_status()
                    backoff = 1.0  # reset on successful connect
                    yield from self._parse_events(resp)
            except (requests.RequestException, ConnectionError):
                # keep last-known flags; never block app startup on this
                time.sleep(min(backoff, 30))
                backoff *= 2

    def _parse_events(self, resp) -> Iterator[dict]:
        event_type = "message"
        data_buffer = []
        event_id = None

        for raw in resp.iter_lines(decode_unicode=True):
            if raw == "":
                # blank line = dispatch the event
                if data_buffer:
                    payload = json.loads("\n".join(data_buffer))
                    if event_id:
                        self.last_event_id = event_id
                    yield {"type": event_type, "data": payload}
                event_type = "message"
                data_buffer = []
                event_id = None
                continue

            if raw.startswith(":"):
                continue  # SSE comment / keepalive
            field, _, value = raw.partition(":")
            value = value.lstrip(" ")
            if field == "event":
                event_type = value
            elif field == "data":
                data_buffer.append(value)
            elif field == "id":
                event_id = value

The server side is symmetrically simple. On connect, send the full flag snapshot as a put event. On every subsequent change, send a patch event with just the diff.

# server-side SSE handler — FastAPI / Starlette
from fastapi import FastAPI, Request
from sse_starlette.sse import EventSourceResponse
import asyncio

app = FastAPI()

@app.get("/sdk/stream")
async def stream(request: Request, env: str, last_event_id: str | None = None):
    async def event_generator():
        # 1) snapshot — bring the client to current state
        snapshot, version = await flag_store.get_snapshot(env)
        yield {
            "event": "put",
            "id": str(version),
            "data": json.dumps({"flags": snapshot, "version": version}),
        }

        # 2) live patches — pubsub fan-out from the write path
        async for change in flag_pubsub.subscribe(env, since=version):
            if await request.is_disconnected():
                break
            yield {
                "event": "patch",
                "id": str(change.version),
                "data": json.dumps(change.diff),
            }

    return EventSourceResponse(
        event_generator(),
        ping=15,  # send :keepalive every 15s for proxy timeouts
    )

Edge caching: flag evaluations at the CDN edge for read-anywhere clients

For browser SDKs and mobile SDKs, even SSE is too chatty. Every cold start opens a connection, downloads the snapshot, then holds an idle socket. On a flaky mobile network you'd rather not.

// CloudFlare Worker — evaluates a single flag at the edge
// flag config is hydrated from KV (CF's edge KV store)

export default {
  async fetch(req, env) {
    const url = new URL(req.url);
    const flagKey = url.pathname.split("/").pop();
    const ctx = await req.json(); // { userId, attrs }

    // KV hit is sub-ms at the edge; miss falls through to origin
    const cfgRaw = await env.FLAGS_KV.get(`flag:${flagKey}`, "json");
    if (!cfgRaw) {
      return new Response(JSON.stringify({ value: null, reason: "UNKNOWN" }),
        { status: 404 });
    }

    const result = evaluate(cfgRaw, ctx);

    // 60s edge cache, but vary on the bucket — not the userId itself
    // (don't blow up the cache key space)
    const bucket = stickyBucket(ctx.userId, flagKey);
    const headers = new Headers({
      "Content-Type": "application/json",
      "Cache-Control": "public, s-maxage=60",
      "X-Flag-Bucket": String(bucket),
    });
    return new Response(JSON.stringify(result), { headers });
  },
};

function evaluate(cfg, ctx) {
  // 1) kill switch — fast path
  if (!cfg.enabled) {
    return { value: cfg.offVariation, reason: "OFF" };
  }

  // 2) targeting rules — explicit user/segment overrides
  for (const rule of cfg.rules ?? []) {
    if (matches(rule.clauses, ctx.attrs)) {
      return { value: rule.variation, reason: "TARGET_MATCH" };
    }
  }

  // 3) percentage rollout — bucket the user, compare to threshold
  const bucket = stickyBucket(ctx.userId, cfg.salt);
  for (const variant of cfg.rollout) {
    if (bucket < variant.cumulativeWeight) {
      return { value: variant.value, reason: "ROLLOUT" };
    }
  }
  return { value: cfg.fallthrough, reason: "FALLTHROUGH" };
}

Flag-config distribution: durable store, snapshot to object storage, push diffs over pubsub

SDK-side caching with TTL + fallback (offline-safe)

Every flag evaluation must answer in microseconds. The SDK keeps the full flag set in memory and serves evaluations locally. The SSE stream keeps the in-memory copy fresh.

Targeting and rules engine: boolean predicates, rollout percentages, sticky bucketing

# the bucketing hash — identical implementation on every SDK,
# every edge worker, every backend evaluator. one source of truth.
import hashlib

def sticky_bucket(user_id: str, flag_salt: str, total_buckets: int = 10_000) -> int:
    # SHA-1, not for security — for stable, uniform distribution across languages.
    # Every SDK ships the same impl. If you swap algorithms you re-bucket
    # every user mid-experiment, which is a silent data corruption bug.
    h = hashlib.sha1(f"{flag_salt}.{user_id}".encode("utf-8")).digest()
    # take 4 bytes, big-endian, mod the bucket count
    n = int.from_bytes(h[:4], "big")
    return n % total_buckets

Why total_buckets=10_000: lets you express rollout in basis points (0.01% granularity), enough precision for ramp schedules and small canary groups.

The 90-second answer that wins the round

When the interviewer drops the prompt, talk for 90 seconds before drawing anything:

"Feature flags are a read-heavy, write-rare distribution problem. The naive GET /flags design fails at 100k SDK clients because polling forces a bad tradeoff between propagation latency and origin load. I'd reframe to a push protocol: SSE from a stateless gateway tier, with each SDK holding one long-lived connection that receives a full snapshot on connect and patches on every flag change. The gateways subscribe to a pubsub channel (Redis or NATS) that the write path publishes to. The source of truth is Postgres for writes and S3-plus-CDN for cold-start snapshots, so a fleet restart doesn't herd. For browser and mobile SDKs I'd push evaluation to the edge via CloudFlare Workers or Lambda@Edge, with flag config replicated to edge KV; that gives 30ms response from the nearest PoP. SDKs cache locally and evaluate in microseconds. Sticky bucketing uses a salted SHA-1 hash with the same implementation in every SDK, edge worker, and backend evaluator. The whole system is offline-safe because the SDK keeps serving last-known values when the stream drops."

That's the answer. Now they ask follow-ups and you draw boxes.

The gotcha: sticky bucketing requires a deterministic hash, and every SDK must agree

The single failure mode that sinks real flag platforms: hash drift across SDKs.

If the interviewer is sharp they'll ask about this. Bring it up unprompted and you've shown you've actually shipped one of these systems.

If this was useful

What's the gnarliest follow-up you've been asked on this kind of design? Drop it in the comments and I'll work through it.

JSONB vs Relational in 2026: 5 Query Shapes, 5 Verdicts

Gabriel Anhaia — Sun, 24 May 2026 12:05:00 +0000

Book: Database Playbook: Choosing the Right Store for Every System You Build
Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

JSONB landed in Postgres 9.4 back in 2014. A decade later your team is still arguing about it. Half of them want every new table to have a data jsonb column "for flexibility." The other half think every JSONB column is a future migration tax. Both camps are right about half the time, which is the most unhelpful version of being right.

So let's stop arguing in the abstract. There are five query shapes that cover roughly 90% of what apps actually do against Postgres. Each one has a clear winner. The row count where the verdict flips is also knowable. Here it is.

When JSONB actually ships

Before the verdicts, a sanity check. JSONB is the right call when you genuinely don't know the shape yet, when the shape is polymorphic per row, or when you need to store an opaque blob that you'll read back as-is.

Three patterns that age well:

Audit logs: event_payload jsonb because every event type has different fields and you want to dump the whole envelope. Nobody queries individual keys at scale.
Rapid prototyping: week one of a product, you have no idea what fields the integration partner will send. Catch it in JSONB, see what the data looks like, extract columns in week three.
Polymorphic per-row data: a settings column where a kafka integration has different keys than a slack one. Forcing both into a shared schema costs more than it saves.

Outside those three, the default should be columns. The five query shapes below show why.

Shape 1: Equality on a known field (relational wins by a lot)

The most common app query in existence: "find the user with email x." If email lives inside a JSONB column, this is what it looks like:

-- email lives in profile->>'email'
SELECT id FROM users
WHERE profile->>'email' = 'sarah@example.com';

You can index it with an expression index:

CREATE INDEX users_profile_email_idx
  ON users ((profile->>'email'));

That works. It's also the moment you realize you've reinvented a column with extra steps. Every query that filters by email now has to spell the JSON path correctly, the index won't apply if anyone writes profile -> 'email' instead of profile ->> 'email', and the planner's row estimates for expression indexes are routinely worse than for plain columns.

Benchmarked on 5M rows with the email present on every row, the column version is consistently 2–4x faster on cold cache and emits a saner plan. The verdict here doesn't flip with size. If you're filtering on a known scalar field, that field belongs in a column.

Shape 2: Existence check (`?` operator), where JSONB with `jsonb_path_ops` is fine

Now the actual JSONB use case: "find every row whose features JSON has a beta_billing key."

SELECT id FROM accounts
WHERE features ? 'beta_billing';

Without an index, Postgres scans every row and parses every JSONB document. On 5M rows that's not a query you want on a hot path. The fix is a GIN index, and this is where the index choice matters more than people realise:

-- the default GIN class indexes everything: keys, values, paths.
-- bigger, supports more operators (@>, ?, ?|, ?&, @?, @@).
CREATE INDEX accounts_features_gin
  ON accounts USING gin (features);

-- jsonb_path_ops indexes only the hashed path-value pairs.
-- about 30% smaller, faster builds, but only supports @>.
CREATE INDEX accounts_features_gin_pathops
  ON accounts USING gin (features jsonb_path_ops);

If your workload is pure @> containment lookups, jsonb_path_ops is the right pick: smaller index, faster builds, faster lookups. If you need ? existence checks too, you're stuck with the default opclass. Or you keep both, which I've seen teams do when the containment index is the hot path and the existence one runs on admin pages.

A nasty subtlety: ? checks for a key at the top level only. WHERE features ? 'beta_billing' won't find {"flags": {"beta_billing": true}}. People learn this the hard way at 2am. For nested keys you need @? with a jsonpath, or restructure.

Shape 3: Aggregation across a JSONB field (relational wins below ~1M rows)

"Sum amount_cents from every order placed last week."

If amount_cents lives inside order_data jsonb, every aggregation row has to extract and cast:

SELECT SUM((order_data->>'amount_cents')::bigint)
FROM orders
WHERE created_at >= NOW() - INTERVAL '7 days';

The ->> operator returns text. The cast to bigint happens for every row in the scan. On a bigint column, the same query reads a fixed-width number directly from the tuple. On a 500k-row weekly window, the column version finishes in tens of milliseconds, while the JSONB version takes seconds. Easily a 20–40x gap on typical hardware.

The crossover point is somewhere between 100k and 1M rows depending on payload size and how warm the cache is. Below that you'll mostly notice it on slow endpoints. Above that, aggregation queries start showing up in your slow log as the dominant cost. Hot numeric fields you aggregate on belong in columns. Full stop.

Shape 4: Containment query (`@>`), where JSONB shines

"Find every event whose payload contains {"source": "stripe", "type": "invoice.paid"}." Try expressing that with relational columns and you end up with N filter predicates per query plus index combinations the planner has to choose between. With JSONB it's one operator:

SELECT id FROM events
WHERE payload @> '{"source": "stripe", "type": "invoice.paid"}'::jsonb;

With the jsonb_path_ops GIN index from Shape 2, this is fast. Really fast. The index stores hashed path-value pairs, so the lookup is essentially a hash probe against a posting list. On a 50M-row event table, this returns in a couple of milliseconds for selective predicates.

This is the shape JSONB was built for. Multi-key containment against semi-structured data with an unbounded set of possible keys. If your app does a lot of "find the subset of records whose JSON matches this prototype," JSONB plus jsonb_path_ops is genuinely the right answer and there's no relational shape that beats it without a denormalisation explosion.

Shape 5: Sparse / polymorphic schema, where JSONB is the right answer

"Store user-defined custom fields per account, where account A has {vat_id, billing_contact} and account B has {purchase_order_number, accounting_email, ap_phone}."

The relational shape is the EAV anti-pattern: a custom_fields table with (account_id, field_name, field_value) and a million JOINs to assemble one record. That works at small scale and rots at large scale. The other relational option is nullable columns for every possible custom field, which means a schema migration every time a customer asks for a new one.

JSONB makes this trivial:

CREATE TABLE accounts (
  id         bigserial PRIMARY KEY,
  name       text NOT NULL,
  custom     jsonb NOT NULL DEFAULT '{}'::jsonb
);

-- queryable when you need it
CREATE INDEX accounts_custom_gin
  ON accounts USING gin (custom jsonb_path_ops);

You get type-flexible storage, indexed containment lookups, and no schema churn. The trade is that you have no schema enforcement at the column level. That responsibility moves to the app layer, ideally with a versioned validator (JSON Schema, a Pydantic/Zod model, whatever your stack uses). When the schema is genuinely open-ended, this trade is good.

The migration path: extract hot keys to columns, keep cold ones in JSONB

The common mistake is treating this as binary. It isn't. A table can have a relational core and a JSONB tail. The pattern that ages best in production:

Start with data jsonb when shape is unclear.
Run for a few weeks. Look at your slow query log. The fields showing up in WHERE and ORDER BY are the hot ones.
Extract those fields into typed columns. Keep the long tail in JSONB.

Here's a migration script for that extraction, written so you can paste it as a template:

BEGIN;

-- 1. add the new typed columns, nullable so the backfill can run hot.
ALTER TABLE orders
  ADD COLUMN customer_id bigint,
  ADD COLUMN amount_cents bigint,
  ADD COLUMN currency char(3),
  ADD COLUMN placed_at timestamptz;

-- 2. backfill from the JSONB blob. batch for big tables.
UPDATE orders
SET
  customer_id  = (data->>'customer_id')::bigint,
  amount_cents = (data->>'amount_cents')::bigint,
  currency     = data->>'currency',
  placed_at    = (data->>'placed_at')::timestamptz
WHERE customer_id IS NULL;

-- 3. enforce NOT NULL once the backfill is verified.
ALTER TABLE orders
  ALTER COLUMN customer_id  SET NOT NULL,
  ALTER COLUMN amount_cents SET NOT NULL,
  ALTER COLUMN currency     SET NOT NULL,
  ALTER COLUMN placed_at    SET NOT NULL;

-- 4. drop the now-redundant keys from the JSONB blob to save space.
--    skip this step if anything still reads them via the JSON path.
UPDATE orders SET data = data
  - 'customer_id' - 'amount_cents' - 'currency' - 'placed_at';

-- 5. indexes appropriate for the new columns.
CREATE INDEX orders_customer_id_idx ON orders (customer_id);
CREATE INDEX orders_placed_at_idx   ON orders (placed_at);

COMMIT;

For tables above ~5M rows, do step 2 in batches with LIMIT + a WHERE primary_key BETWEEN clause, and run them in a loop outside a single transaction. A single UPDATE over 50M rows holds a row exclusive lock long enough to wreck your p99 latency, even with the rest of the workload otherwise idle.

Also worth saying: dropping keys from JSONB in step 4 rewrites every row. On a big table, that's a lot of bloat and a vacuum after. Skip it if disk is cheap and reads through the JSON path are still happening anywhere in the codebase. The schema-as-documentation value isn't worth a multi-hour VACUUM FULL.

The gotcha: JSONB columns can't be a foreign key target

This is the trap that catches teams who push too hard into JSONB. Foreign keys can only reference columns. Not JSONB paths. Not expressions.

So this fails:

-- doesn't work. there's no syntax for "FK targeting a JSONB path."
ALTER TABLE invoices
  ADD CONSTRAINT invoices_customer_fk
  FOREIGN KEY ((data->>'customer_id'))
  REFERENCES customers(id);
-- ERROR: syntax error at or near "("

You can't fake it with a check constraint, either. Check constraints in Postgres can't run subqueries, so CHECK (EXISTS (SELECT 1 FROM customers WHERE id = (data->>'customer_id')::bigint)) is rejected. You're stuck with either trigger-based enforcement (slow and easy to bypass) or no referential integrity (and the orphaned-record problems that follow).

The escape hatch is exactly what the migration script above does: extract the FK target to a real column.

ALTER TABLE invoices
  ADD COLUMN customer_id bigint;

UPDATE invoices
SET customer_id = (data->>'customer_id')::bigint;

ALTER TABLE invoices
  ALTER COLUMN customer_id SET NOT NULL,
  ADD CONSTRAINT invoices_customer_fk
    FOREIGN KEY (customer_id) REFERENCES customers(id);

Now you have a real FK with ON DELETE behaviour, plan-time row estimates, and the rest of what relational integrity gives you. The JSONB column can still hold the long tail.

If you find yourself wanting FKs from three different JSONB keys, that's the database telling you those keys should have been columns from day one. Listen to it.

The five verdicts on one card

Query shape	Winner	Crossover row count
Equality on a known scalar	Relational column	Never flips. Column wins at all sizes
Existence check (`?`)	JSONB + GIN	Below ~10k rows the seq scan is fine; above, you need GIN
Numeric aggregation	Relational column	Flips around 100k–1M rows; columns win above
Containment (`@>`) on semi-structured data	JSONB + `jsonb_path_ops`	JSONB wins at all sizes when the schema is open
Polymorphic / per-row sparse keys	JSONB	JSONB wins at all sizes; alternative is EAV pain

The decision isn't "JSONB or relational." It's "which fields per table." Hot, typed, queried-by-equality, foreign-keyed: column. Cold, sparse, polymorphic, containment-searched: JSONB. The migration path between them is well-trodden. Use it.

What's the worst JSONB-vs-column call you've seen ship to prod, and how long did it take to undo?

If this was useful

Picking the right storage shape is the chapter most teams skim and the one that costs them six months later. The Database Playbook: Choosing the Right Store for Every System You Build walks the JSONB-versus-relational decision in more depth — the GIN opclass trade-offs, the schema-evolution patterns that don't paint you into a corner, and the queries you can't write because you picked the wrong store. If you spend any of your week tuning Postgres or arguing about it in PRs, it's probably worth the read.

Logical Replication for Migrations: Zero-Downtime Postgres Upgrades in 2026

Gabriel Anhaia — Sun, 24 May 2026 12:04:34 +0000

Book: Database Playbook: Choosing the Right Store for Every System You Build
Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Major Postgres upgrades used to mean a maintenance window. With logical replication, they're zero-downtime, if you set it up right. There are three traps that turn a 20-minute cutover into a 6-hour incident, and none of them are in the official upgrade docs.

This is the playbook a team I work with uses for 13 to 16 to 17 jumps on databases that can't take a banner. Real SQL, the sequence script that always gets forgotten, and the verification step that catches the silent data drift before users do.

Why pg_upgrade is the old way

pg_upgrade is fast. On the same machine, with hard links, it can move a 500GB cluster in under a minute. The catch is everything around it.

You stop the database. You run the upgrade. You restart it. You analyze statistics so the planner doesn't fall over on the first query. If anything fails partway, you're recovering from backup while your status page bleeds. And pg_upgrade doesn't help at all if you're moving across machines: different host, different storage class, different cloud region.

A team I talked to last quarter ran pg_upgrade on a 2TB cluster as part of a 13 to 16 jump. The binary swap took 90 seconds. The ANALYZE on the upgraded cluster took 47 minutes. During those 47 minutes, the application was up but every query that touched a non-trivial index ran a sequential scan. Their p99 went from 80ms to 14 seconds. Customers noticed.

Logical replication sidesteps all of that. You build the new cluster while the old one is still serving traffic. You analyze on the new cluster while nobody's watching it. You cut over at a moment of your choosing, with seconds of write pause instead of minutes of downtime.

Logical replication primer

Logical replication moves row-level changes from a publisher to a subscriber over a normal Postgres connection. It's been stable since Postgres 10 and has gotten meaningfully better every release since.

Three pieces you need to understand:

Publication: a named set of tables on the source. Created with CREATE PUBLICATION.
Subscription: a connection from the target back to the source that pulls changes for a publication. Created with CREATE SUBSCRIPTION.
Replication slot: a server-side bookmark on the publisher that tracks how far the subscriber has consumed. The slot keeps WAL files around until the subscriber acks them, which is why a stuck subscriber can fill your disk.

The flow: the publisher decodes its WAL into row changes, ships them over the wire, the subscriber applies them with regular SQL. Because it's logical (rows, not pages), the publisher and subscriber can run different Postgres versions. That's what makes this whole thing work.

The 5-step zero-downtime upgrade

The whole sequence assumes you have a target cluster running the new Postgres version, reachable from the source over the network, with enough disk for the data plus 30% headroom for the initial sync.

Step 1. Prepare the schema on the target

Logical replication does not copy the schema. You have to dump it yourself.

# on a host that can reach the source
pg_dump \
  --host=postgres-13.internal \
  --port=5432 \
  --username=migrator \
  --dbname=orders \
  --schema-only \
  --no-owner \
  --no-acl \
  --file=orders-schema.sql

# apply it to the target
psql \
  --host=postgres-16.internal \
  --port=5432 \
  --username=migrator \
  --dbname=orders \
  --file=orders-schema.sql

Disable foreign key checks on the subscriber? No, leave them on. Logical replication applies changes in commit order, so FK ordering works out. What you do need to disable, for the initial sync, is any trigger-based logic that would double-fire on rows that already happened upstream. The subscriber respects session_replication_role = replica, but only for triggers explicitly marked ENABLE REPLICA TRIGGER. By default, your application triggers sit out during replication apply. Good.

Step 2. Create the publication on the source

-- on postgres-13.internal
CREATE PUBLICATION orders_migration FOR ALL TABLES;

FOR ALL TABLES is the right call for a full-cluster migration. If you want a subset, list them: FOR TABLE customers, orders, line_items. Either way, this is cheap. It's metadata, no data movement yet.

Confirm the publisher's wal_level is logical. If it's not, you're restarting the source to flip it, which is the one downtime moment in this whole plan that you can't avoid. Check ahead of time:

SHOW wal_level;
-- if 'replica', set 'logical' in postgresql.conf and restart

Step 3. Create the subscription on the target

-- on postgres-16.internal
CREATE SUBSCRIPTION orders_migration_sub
  CONNECTION 'host=postgres-13.internal port=5432 dbname=orders user=replicator password=...'
  PUBLICATION orders_migration
  WITH (
    copy_data = true,
    create_slot = true,
    slot_name = 'orders_migration_slot',
    streaming = 'parallel',
    binary = true
  );

A few choices worth knowing:

copy_data = true triggers the initial table sync: every existing row gets copied before incremental changes start applying. For a 2TB cluster expect hours.
streaming = 'parallel' (Postgres 16+) lets large transactions stream as they happen instead of waiting for commit. Big win for write-heavy workloads.
binary = true is faster but requires matching column types on both sides. If you changed a type during this migration, leave it false.

Watch the sync progress:

SELECT
  subname,
  pid,
  received_lsn,
  latest_end_lsn,
  pg_size_pretty(
    pg_wal_lsn_diff(latest_end_lsn, received_lsn)
  ) AS lag_bytes
FROM pg_stat_subscription;

When lag_bytes is 0 bytes and stays there during normal write load, you're caught up.

Step 4. Cut over

The cutover is where teams get nervous. Here's the actual sequence:

Set the application to read-only mode (or pause writes entirely). One feature flag, one deploy.
Wait 5 seconds for in-flight transactions to commit on the source.
Confirm lag_bytes = 0 from the subscription stat query above.
Bump sequences (next section: the trap that bites everyone).
Flip DNS / connection string / PgBouncer config to point at the new cluster.
Take the application out of read-only mode.

Total write pause: 10 to 30 seconds depending on how fast your config rollout is. That's the "zero-downtime": reads kept serving from a read replica throughout, writes paused for under a minute.

Step 5. Decommission

Don't drop the old cluster yet. Run the new one as the source of truth for at least 24 hours, with the old cluster still up but no traffic. If something goes wrong, you can flip back.

After 24 hours of clean operation:

-- on postgres-16.internal
DROP SUBSCRIPTION orders_migration_sub;

-- on postgres-13.internal
DROP PUBLICATION orders_migration;

-- check the slot is gone (DROP SUBSCRIPTION should have cleaned it,
-- but verify because a leaked slot will fill disk)
SELECT slot_name, active, pg_size_pretty(
  pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
) AS retained_wal
FROM pg_replication_slots;

Then snapshot the old cluster and shut it down.

Trap 1. Sequences don't replicate

This is the one. Logical replication copies row changes. Sequences are not rows; they're a separate object Postgres bumps independently. When you cut over, every sequence on the new cluster is at whatever value the schema dump set it to (probably 1), while your application is about to insert a row that conflicts with an existing primary key.

You have to bump every sequence to match the source, right before the cutover. Here's the script that does it:

-- run on the SOURCE to generate the bump statements
SELECT format(
  'SELECT setval(%L, %s, true);',
  schemaname || '.' || sequencename,
  last_value
)
FROM pg_sequences
WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
ORDER BY schemaname, sequencename;

That returns one line per sequence, like:

SELECT setval('public.orders_id_seq', 8472913, true);
SELECT setval('public.customers_id_seq', 412877, true);
SELECT setval('public.line_items_id_seq', 31204882, true);

You collect the output, then during step 4 of the cutover (between the write pause and the DNS flip), you run those statements on the target. The true flag means "the next call to nextval returns a value greater than this," which is what you want.

A safer variant, if you don't trust your write-pause window: add a buffer.

SELECT format(
  'SELECT setval(%L, %s, true);',
  schemaname || '.' || sequencename,
  last_value + 10000  -- buffer for in-flight writes
)
FROM pg_sequences
WHERE schemaname NOT IN ('pg_catalog', 'information_schema');

You lose 10,000 IDs per sequence. That's nothing. What you gain is bulletproof: even if a write slipped through your read-only flag, the new cluster will not collide.

Trap 2. DDL doesn't replicate

Logical replication moves row data, not schema changes. If you ALTER TABLE orders ADD COLUMN canceled_at TIMESTAMPTZ on the source while replication is active, the subscriber will start failing on the next insert that includes that column, with an error like:

ERROR: logical replication target relation "public.orders" is missing
replicated column: "canceled_at"

Replication halts. The slot keeps growing. You're now in an incident.

The fix is process, not SQL. During the migration window, you freeze DDL. No schema changes, no new columns, no index rebuilds that drop and recreate. If you absolutely must ship a schema change mid-migration:

Apply the DDL on the subscriber first.
Apply the DDL on the publisher second.
Use only additive changes (ADD COLUMN NULL, ADD INDEX CONCURRENTLY). Never drop, rename, or change a type.

For longer migrations where you can't freeze DDL for days, look at the pglogical extension or Postgres 17's experimental DDL replication. For a 1–2 day window, just freeze.

Trap 3. Large objects and unlogged tables get skipped

Two categories of data that logical replication silently ignores:

Large objects (the pg_largeobject system table, accessed via lo_* functions). If your app stores PDFs or images using the LO API, those bytes are not in your subscription. You'll have an empty file table on the new cluster and not know it.

Unlogged tables. These are tables created with CREATE UNLOGGED TABLE, used for ephemeral data because they skip the WAL. Logical replication reads from the WAL. No WAL, no replication.

Workarounds:

For large objects: use pg_dump --large-objects separately, restore into the target after the initial sync, then accept that any LOs written during the cutover window will need a delta sync. The cleaner long-term answer is to move away from large objects to bytea columns or external object storage. Postgres 16 marked the LO API as effectively legacy.
For unlogged tables: dump and restore them at cutover time, accepting that data written between dump and cutover is lost. If the table genuinely doesn't matter (cache, session store), that's fine. If it does matter, it shouldn't be unlogged.

There's a third quiet skip worth knowing: tables without a REPLICA IDENTITY. By default that's the primary key. If you have a table with no PK and no REPLICA IDENTITY FULL set, updates and deletes will fail to replicate with cannot update table without primary key. The check:

SELECT n.nspname, c.relname, c.relreplident
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE c.relkind = 'r'
  AND c.relreplident = 'd'  -- default = primary key
  AND NOT EXISTS (
    SELECT 1 FROM pg_index i
    WHERE i.indrelid = c.oid AND i.indisprimary
  );

Any row that comes back needs a REPLICA IDENTITY FULL set before you start.

Verifying the cutover

Row counts are not enough. A table can have the right count and the wrong data. You want checksums.

-- run on both source and target, compare results
SELECT
  'orders' AS table_name,
  COUNT(*) AS row_count,
  md5(string_agg(
    md5(t::text),
    ',' ORDER BY id
  )) AS table_checksum
FROM orders t;

That builds a per-row hash, concatenates them in primary-key order, and hashes the result. Two clusters with identical data return identical checksums. Two clusters off by one row return wildly different hashes.

For tables with hundreds of millions of rows, this gets slow. Sample instead:

SELECT
  'orders' AS table_name,
  COUNT(*) AS sample_count,
  md5(string_agg(md5(t::text), ',' ORDER BY id)) AS checksum
FROM orders t TABLESAMPLE BERNOULLI (1)  -- 1% sample
WHERE id BETWEEN 1000000 AND 2000000;     -- bounded range

Run the same sample bounds on both sides. If checksums match on three or four random ranges, you're good.

The 60-second smoke test for the application: hit your top 5 read paths, hit your top 3 write paths, then run a query you know exact-counts the result for (a small reference table, an enum lookup). If all three pass, you're shipping.

The gotcha. Replication lag during cutover

Right when you cut over, replication lag tends to spike. The subscriber suddenly stops receiving the load it was streaming, the source is finishing in-flight transactions, and pg_stat_subscription can briefly show non-zero lag_bytes even after you've paused writes.

If your application has a read replica pattern, keep reads going to the old primary's read replica for a 5-minute window after cutover. Writes go to the new cluster, reads to the old one. Once lag_bytes has been zero for 60 seconds straight, flip reads over too.

This sounds like overkill. It isn't. The class of bug it prevents (a user submits a form, the write hits the new cluster, the immediate GET reads from a stale read replica) is exactly the kind of "ghost data" issue that gets logged as "intermittent UI bug" and lives in your backlog for months.

Zero-downtime is doable. It is not effortless. The five-step playbook gets you there; the three traps and the checksum verification keep you out of the incident channel.

What's the worst cutover you've shipped, and which of these traps bit you? Drop it in the comments.

If this was useful

This kind of migration sits in the gap between "knows Postgres" and "knows Postgres in production." If you want the longer treatment (picking the right replication tool for your topology, when to choose physical over logical, what happens to your stats and autovacuum after a major version jump), that's chapter 7 of the Database Playbook.