Forem: Wahyu Tricahyo

Active-Passive vs Active-Active: How to Know Which Pattern Fits Your Database

Wahyu Tricahyo — Thu, 19 Feb 2026 11:36:11 +0000

OpenAI recently shared how they scale PostgreSQL for ChatGPT's 800 million weekly users. The part worth paying attention to wasn't the scale itself, it was the decision. They chose Active-Passive over Active-Active, and they didn't shard. For a product growing that fast, that's a bet most teams wouldn't make.

But the real question isn't "what did OpenAI do?" It's: how do you figure out which pattern is right for your own workload?

That's what this post is about. OpenAI's setup is a reference point, but the focus is on giving you the tools and framework to make this decision yourself.

First: Measure Before You Architect

Teams often jump to Active-Active or sharding because their user count "feels" big enough. The better approach is to let your actual data tell you what to do.

Check Your Read/Write Ratio in PostgreSQL

This is the single most important number for deciding your replication pattern.

Database-Level: `pg_stat_database`

SELECT
  datname,
  tup_returned + tup_fetched AS total_reads,
  tup_inserted + tup_updated + tup_deleted AS total_writes,
  ROUND(
    (tup_returned + tup_fetched)::numeric /
    NULLIF((tup_returned + tup_fetched + tup_inserted + tup_updated + tup_deleted), 0) * 100, 2
  ) AS read_percentage,
  ROUND(
    (tup_inserted + tup_updated + tup_deleted)::numeric /
    NULLIF((tup_returned + tup_fetched + tup_inserted + tup_updated + tup_deleted), 0) * 100, 2
  ) AS write_percentage
FROM pg_stat_database
WHERE datname = 'your_database_name';

Quick reference: tup_returned = rows from sequential scans, tup_fetched = rows from index scans, and the write fields are self-explanatory. Note that tup_returned includes internal operations (e.g., sequential scans by autovacuum), which can inflate your read count. Treat the ratio as directional, not exact. These are cumulative since the last stats reset. To measure a specific window (peak hours vs off-peak), reset first with SELECT pg_stat_reset(); and query again after.

Per-Table: `pg_stat_user_tables`

The database-level number hides a lot. A database can be 85% reads overall while two tables are 95% writes. Those tables are the ones that matter.

SELECT
  schemaname,
  relname AS table_name,
  seq_tup_read + idx_tup_fetch AS total_reads,
  n_tup_ins + n_tup_upd + n_tup_del AS total_writes,
  ROUND(
    (seq_tup_read + idx_tup_fetch)::numeric /
    NULLIF((seq_tup_read + idx_tup_fetch + n_tup_ins + n_tup_upd + n_tup_del), 0) * 100, 2
  ) AS read_percentage
FROM pg_stat_user_tables
ORDER BY (n_tup_ins + n_tup_upd + n_tup_del) DESC
LIMIT 20;

This is the query that actually drives decisions. It shows which tables are write-heavy and might need to live somewhere else, while the rest stay on a single-primary Postgres.

Per-Query: `pg_stat_statements`

For deeper analysis, pg_stat_statements shows exactly which queries hit the database hardest:

CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

-- Top write queries by call count
SELECT query, calls, ROUND(total_exec_time / calls, 2) AS avg_time_ms
FROM pg_stat_statements
WHERE query ILIKE '%INSERT%' OR query ILIKE '%UPDATE%' OR query ILIKE '%DELETE%'
ORDER BY calls DESC
LIMIT 20;

-- Top read queries by call count
SELECT query, calls, ROUND(total_exec_time / calls, 2) AS avg_time_ms
FROM pg_stat_statements
WHERE query ILIKE 'SELECT%'
ORDER BY calls DESC
LIMIT 20;

This also catches problematic queries before they cause incidents. OpenAI found an ORM-generated query joining 12 tables that caused multiple high-severity outages. That's not a scaling problem, it's a query problem. pg_stat_statements would have surfaced it immediately.

A note on the ILIKE filters above: string matching on %INSERT% can catch queries that merely mention INSERT in comments or subqueries. For more precise filtering, check the query plan or use pg_stat_statements' queryid grouping. These queries are a starting point, not a production monitoring setup.

What Your Numbers Mean

Your Read %	Likely Pattern	Notes
90%+ reads	Active-Passive	Same territory as ChatGPT. Add replicas, not complexity
70-90% reads	Active-Passive (probably)	Check per-table breakdown. Offload the write-heavy outliers
50-70% reads	It depends	Look at where writes concentrate. Selective sharding might be enough
Below 50% reads	Active-Active or Sharding	A single primary will become the bottleneck at scale

These percentages assume writes are spread across many tables. If your writes are concentrated in two or three tables, the database-level ratio can be misleading. A database that's 70% reads overall might work fine on Active-Passive if the write-heavy tables can be offloaded. Always cross-check with the per-table query.

Important caveat: These percentages reflect steady-state traffic. Spikes matter too.

OpenAI's workload is heavily read-dominant, but during the ImageGen launch, 100 million users signed up in a single week and writes spiked 10x overnight. That spike caused their only SEV-0 Postgres incident in twelve months. The primary got overwhelmed, requests timed out, and users retried, creating a vicious cycle where retry traffic pushed the already-drowning database into full outage. They recovered by design (hot standby failover, rate limiting kicked in), but the lesson is clear: a read-heavy ratio can temporarily flip during a viral launch.

Measure steady-state, but also look at the worst traffic day in the last year. If writes can spike 5-10x during launches, the architecture needs headroom for that.

The Two Patterns: What Actually Matters

Active-Passive: The Gotchas

The selling point is simplicity: one source of truth, no conflict resolution, scale reads by adding replicas. But:

Failover isn't instant. Promoting a replica means brief downtime. Without recent testing, "brief" can stretch into minutes. OpenAI runs a hot standby matching the primary's specs, paying double for the writer tier as insurance.
Replicas have diminishing returns. 50 replicas doesn't mean 50x read capacity. Each one adds WAL streaming overhead on the primary. After ~15-20, the primary's CPU budget for writes starts getting eaten. OpenAI is hitting this now, hence cascading replication. There's also a cost question: 50 read replicas plus a hot standby can exceed the cost of a smaller Active-Active cluster with fewer nodes that handle both reads and writes. The math depends on instance sizes and cloud provider, but it's worth running before committing to either pattern.
Replication lag is invisible until it isn't. Usually milliseconds. During write spikes, lag balloons, and replicas serve stale data. Without routing recent-write reads to the primary, users see inconsistencies they can't explain. Monitor it with pg_stat_replication on the primary:

SELECT
  client_addr,
  state,
  sent_lsn,
  replay_lsn,
  replay_lag
FROM pg_stat_replication;

If replay_lag is consistently above a few hundred milliseconds, replicas are falling behind. Set up alerts for this, not just dashboards.

Active-Active: The Complexity Tax

Source: pgEdge

The selling point is resilience and global write latency. But:

Conflict resolution leaks into application code. "Last-write-wins" sounds fine until two users update the same record simultaneously and one silently loses their change. Especially nasty for account balances, permissions, or anything where silent overwrites cause real harm.
Testing is exponentially harder. Every code path needs testing against the possibility of concurrent conflicting writes across regions.
It couples database choice to architecture. PostgreSQL doesn't natively support multi-master. Active-Active means adding third-party tooling (pgEdge, BDR) or switching databases entirely (CockroachDB, YugabyteDB). That's a bigger commitment than adding read replicas.

Side-by-Side

Factor	Active-Passive	Active-Active
Write throughput	Limited to one primary	Scales with nodes
Read throughput	Scales with replicas	Scales with nodes
Consistency	Simple, one source of truth	Needs conflict resolution
Write latency	Higher for distant regions	Low everywhere
Operational complexity	Lower	Significantly higher
Failover	Promote a replica (brief downtime)	Other masters keep serving
Best for	Read-heavy workloads	Write-heavy or geo-distributed writes

How OpenAI Applies This

OpenAI describes their workload as heavily read-dominant. Active-Passive was the obvious call.

The setup: one primary PostgreSQL on Azure Flexible Server, ~50 read replicas across regions, millions of QPS, low double-digit millisecond p99 latency, five-nines availability.

The interesting part isn't the pattern, it's what keeps it alive at this scale.

Operational Defenses

A single-primary setup at 800M users is fragile without these layers. Most of these apply regardless of replication pattern:

PgBouncer for connection pooling. Dropped connection setup from 50ms to 5ms. At millions of QPS, this is the difference between a system that works and one that doesn't. Note that PgBouncer handles pooling, not read/write routing. Sending reads to replicas and writes to the primary requires something else: HAProxy, pgpool-II, or application-level routing logic.
Cache locking. When a cache key expires, one request fetches from Postgres. Everyone else waits for the cache to repopulate instead of all hammering the database at once.
Workload isolation. Low-priority and high-priority traffic route to separate replica instances. A background analytics job can't slow down user-facing reads.
Multi-layer rate limiting. At application, pooler, proxy, and query levels. All of them, not just one.

Moving Write-Heavy Tables, Not the Whole Database

Instead of sharding Postgres or migrating to a distributed database, OpenAI identified specific write-heavy workloads and moved only those to Cosmos DB. Everything else stays on Postgres.

The rule: no new tables in the main Postgres. New features needing heavy writes go to sharded systems from day one. Existing write-heavy tables get migrated out when possible.

This maps directly to the per-table query above. Run it, find the top write-heavy tables, and ask: can these live somewhere else?

Cascading Replication: Scaling Past the Replica Ceiling

With ~50 replicas, the primary streams WAL to every single one, and each replica adds CPU and network overhead. Cascading replication solves this by introducing intermediate replicas: the primary streams to a handful of intermediates, and each intermediate relays WAL to a group of downstream replicas. This means the primary only maintains a few replication connections instead of fifty. The tradeoff is slightly more lag for downstream replicas (they're one hop further from the source) and more complex failover, since promoting an intermediate requires repointing its downstream replicas. OpenAI is testing this with Azure but hasn't deployed it to production yet.

The Hybrid Approach

This is the option most teams overlook, and probably the right answer for most:

Keep the main database Active-Passive
Identify write-heavy outliers with pg_stat_user_tables
Move those specific workloads to a system built for writes
Don't touch anything that doesn't need to be touched

Full Active-Active is expensive in complexity. Full Active-Passive might have a few tables that don't fit. The hybrid gets 90% of the benefit with 10% of the migration pain.

The Real Lesson

The most common mistake is choosing a replication pattern based on the user count a team hopes to reach, not the workload they actually have. Active-Active gets adopted for a scale problem that doesn't exist yet, and the team spends two years debugging the complexity.

Meanwhile, the problems that actually cause outages are mundane: an unoptimized ORM query, missing connection pooling, a background job competing with user-facing reads. None of these require a different replication pattern. They require running pg_stat_statements and paying attention.

Run the queries. Look at the ratio. If it says Active-Passive, trust it, even if it feels too simple. Complexity can always be added later. It can never easily be removed.

When to revisit the decision:

Write latency on the primary is consistently above your SLA and you've already optimized queries and offloaded what you can
Replication lag (replay_lag) stays above acceptable thresholds during normal traffic, not just spikes
Failover tests are taking longer than your downtime budget allows
You're expanding to regions where write latency to the primary is unacceptable for the user experience

If none of those are true, the current pattern is working. Keep measuring.

References

Redis Advanced Techniques: Beyond Simple Key-Value Storage

Wahyu Tricahyo — Mon, 16 Feb 2026 02:32:39 +0000

Redis gets introduced as a caching layer, and for many teams, that's where the story ends. But Redis is far more than a cache. It's a data structure server with capabilities that can replace entire categories of infrastructure if you know how to use it. In this post, I'll cover advanced techniques that unlock Redis's full potential, with all examples using TypeScript and ioredis.

Data Structures You're Probably Not Using

Most developers stick to strings and maybe hashes. Redis offers much more, and choosing the right structure for your use case can eliminate complexity from your application layer entirely.

Sorted Sets are one of Redis's most underrated features. They store members with a score, keeping everything ordered automatically. This makes them perfect for leaderboards, priority queues, rate limiters, and time-based feeds.

async function leaderboardExample(): Promise<void> {
  await redis.zadd("leaderboard", 1500, "player:alice");
  await redis.zadd("leaderboard", 2300, "player:bob");
  await redis.zadd("leaderboard", 1800, "player:carol");

  // Top 10 players (highest scores first)
  const topPlayers = await redis.zrevrange("leaderboard", 0, 9, "WITHSCORES");
  // Returns: ["player:bob", "2300", "player:carol", "1800", "player:alice", "1500"]

  // Parse into a usable format
  const parsed: { member: string; score: number }[] = [];
  for (let i = 0; i < topPlayers.length; i += 2) {
    parsed.push({ member: topPlayers[i], score: Number(topPlayers[i + 1]) });
  }

  // Players scoring between 1000 and 2000
  const midRange = await redis.zrangebyscore("leaderboard", 1000, 2000, "WITHSCORES");
  console.log("Mid-range players:", midRange);
}

HyperLogLog solves the problem of counting unique items in a dataset without storing every item. It uses roughly 12KB of memory regardless of how many elements you add, with a standard error of about 0.81%. If you need to count unique visitors, unique search queries, or distinct events, HyperLogLog gives you an approximate count that's good enough for analytics at a fraction of the memory cost.

async function hyperLogLogExample(): Promise<void> {
  const today = "2025-02-14";

  await redis.pfadd(`unique_visitors:${today}`, "user:1001", "user:1002", "user:1003");
  await redis.pfadd(`unique_visitors:${today}`, "user:1001"); // duplicate, won't increase count

  const count = await redis.pfcount(`unique_visitors:${today}`);
  console.log(`Unique visitors today: ${count}`); // ~3

  // Merge multiple days for weekly count
  await redis.pfmerge(
    "unique_visitors:week",
    "unique_visitors:2025-02-10",
    "unique_visitors:2025-02-11"
  );
  const weeklyCount = await redis.pfcount("unique_visitors:week");
  console.log(`Unique visitors this week: ${weeklyCount}`);
}

Streams give you an append-only log structure with consumer group support. Think of them as a lightweight alternative to Kafka for many use cases. They support blocking reads, message acknowledgment, and automatic ID generation, making them a solid choice for event sourcing, task queues, and inter-service communication.

async function streamProducerExample(): Promise<void> {
  const id1 = await redis.xadd("orders", "*", "product", "laptop", "quantity", "1", "customer", "alice");
  const id2 = await redis.xadd("orders", "*", "product", "phone", "quantity", "2", "customer", "bob");
  console.log(`Added orders: ${id1}, ${id2}`);
}

async function streamConsumerExample(): Promise<void> {
  // Create a consumer group (wrap in try/catch since it throws if group exists)
  try {
    await redis.xgroup("CREATE", "orders", "processing", "$", "MKSTREAM");
  } catch (err) {
    // Group already exists, safe to ignore
  }

  // Consumer reads from the group with a 2-second block
  const messages = await redis.xreadgroup(
    "GROUP", "processing", "worker-1",
    "COUNT", "5",
    "BLOCK", 2000,
    "STREAMS", "orders", ">"
  );

  if (messages) {
    for (const [stream, entries] of messages) {
      for (const [id, fields] of entries) {
        console.log(`Processing message ${id}:`, fields);
        // Acknowledge after processing
        await redis.xack("orders", "processing", id);
      }
    }
  }
}

Lua Scripting: Atomic Operations on Steroids

When you need multiple Redis commands to execute atomically, Lua scripting is the answer. Scripts run on the server side with no interruption, which means no race conditions and no need for distributed locks in many cases.

A classic example is a rate limiter. You need to check the current count, compare it to a limit, and either increment or reject, all without another request sneaking in between steps.

const rateLimiterScript = `
  local key = KEYS[1]
  local limit = tonumber(ARGV[1])
  local window = tonumber(ARGV[2])

  local current = redis.call("INCR", key)
  if current == 1 then
    redis.call("EXPIRE", key, window)
  end
  if current > limit then
    return 0
  end
  return 1
`;

// Define custom commands for reuse
redis.defineCommand("rateLimit", {
  numberOfKeys: 1,
  lua: rateLimiterScript,
});

// Extend the Redis type so TypeScript knows about our custom command
declare module "ioredis" {
  interface RedisCommander<Context> {
    rateLimit(key: string, limit: string, window: string): Promise<number>;
  }
}

async function checkRateLimit(userId: string): Promise<boolean> {
  // Allow 100 requests per 60 seconds
  const allowed = await redis.rateLimit(`ratelimit:${userId}`, "100", "60");
  return allowed === 1;
}

async function handleRequest(userId: string): Promise<void> {
  const isAllowed = await checkRateLimit(userId);
  if (!isAllowed) {
    console.log("Rate limited!");
    return;
  }
  console.log("Request processed");
}

Keep your Lua scripts short and focused. Long-running scripts block the entire Redis instance since Redis is single-threaded. If your script takes more than a few milliseconds, rethink your approach.

Pipelining and Transactions

Pipelining is one of the simplest ways to boost Redis performance. Instead of sending commands one at a time and waiting for each response, you batch them together. This cuts down on network round trips dramatically. In high-throughput scenarios, pipelining alone can give you a 5-10x improvement.

async function pipelineExample(): Promise<void> {
  const pipeline = redis.pipeline();

  for (let i = 0; i < 1000; i++) {
    pipeline.set(`key:${i}`, `value:${i}`);
  }

  const results = await pipeline.exec();
  // One round trip instead of 1000
  console.log(`Executed ${results?.length} commands in a single round trip`);
}

// You can also read results from a pipeline
async function pipelineReadExample(): Promise<void> {
  const pipeline = redis.pipeline();

  pipeline.get("user:1001:name");
  pipeline.get("user:1001:email");
  pipeline.zrevrange("leaderboard", 0, 4, "WITHSCORES");

  const results = await pipeline.exec();
  // results is [[null, "Alice"], [null, "alice@example.com"], [null, [...]]]
  // Each entry is [error, result]

  if (results) {
    const [nameErr, name] = results[0];
    const [emailErr, email] = results[1];
    const [leaderboardErr, topPlayers] = results[2];
    console.log(`User: ${name}, Email: ${email}`);
  }
}

Transactions with MULTI/EXEC guarantee that a batch of commands runs sequentially without interleaving from other clients. Combined with watch, you get optimistic locking for check-and-set operations.

async function transactionExample(): Promise<boolean> {
  // Optimistic locking: decrement inventory only if it hasn't changed
  await redis.watch("inventory:laptop");

  const currentStock = await redis.get("inventory:laptop");
  const stock = Number(currentStock);

  if (stock <= 0) {
    await redis.unwatch();
    console.log("Out of stock");
    return false;
  }

  const result = await redis
    .multi()
    .set("inventory:laptop", String(stock - 1))
    .exec();

  // result is null if WATCH detected a change
  if (!result) {
    console.log("Conflict detected, retry needed");
    return false;
  }

  console.log("Purchase successful");
  return true;
}

The important distinction: pipelining reduces latency by batching network calls, while transactions ensure atomicity. You can combine them, but they solve different problems.

Pub/Sub and Beyond

Redis Pub/Sub gives you simple, fire-and-forget messaging. It's great for broadcasting events like cache invalidation signals, live notifications, or real-time updates. But it comes with a major caveat: messages are not persisted. If a subscriber is disconnected when a message is published, that message is lost forever.

// Important: ioredis requires a separate connection for subscriptions because
// once a connection enters subscriber mode, it can only run subscribe/unsubscribe
// commands — all other commands like GET or SET will be rejected.
const subscriber = new Redis();
const publisher = new Redis();

interface NotificationPayload {
  type: string;
  from: string;
  message: string;
}

// Subscriber
subscriber.subscribe("notifications:user:1001", (err, count) => {
  if (err) console.error("Subscribe error:", err);
  console.log(`Subscribed to ${count} channel(s)`);
});

subscriber.on("message", (channel: string, message: string) => {
  const payload: NotificationPayload = JSON.parse(message);
  console.log(`[${channel}] ${payload.type} from ${payload.from}: ${payload.message}`);
});

// Publisher (from another part of your app)
async function sendNotification(userId: string, payload: NotificationPayload): Promise<void> {
  await publisher.publish(`notifications:${userId}`, JSON.stringify(payload));
}

// Pattern-based subscriptions
subscriber.psubscribe("notifications:*", (err) => {
  if (err) console.error("Pattern subscribe error:", err);
});

subscriber.on("pmessage", (pattern: string, channel: string, message: string) => {
  console.log(`[pattern: ${pattern}] [channel: ${channel}] ${message}`);
});

For cases where you need message durability, use Streams instead. Streams retain messages, support consumer groups for load balancing, and let you replay history. Pub/Sub is for ephemeral, real-time broadcasting; Streams are for reliable message processing.

Keyspace Notifications are a specialized form of Pub/Sub that lets you subscribe to events happening inside Redis itself. You can get notified when keys expire, when a key is deleted, or when any command modifies data. This is useful for building reactive systems, cache invalidation callbacks, or session expiry handling.

async function keyspaceNotifications(): Promise<void> {
  const sub = new Redis();

  // Enable keyspace notifications for expired events
  await redis.config("SET", "notify-keyspace-events", "Ex");

  // Subscribe to expiration events on db 0
  sub.subscribe("__keyevent@0__:expired", (err) => {
    if (err) console.error("Keyspace subscribe error:", err);
  });

  sub.on("message", (channel: string, expiredKey: string) => {
    console.log(`Key expired: ${expiredKey}`);
    // Handle session cleanup, cache refresh, etc.
  });

  // Test it: set a key that expires in 5 seconds
  await redis.set("session:temp", "data", "EX", 5);
}

Memory Optimization Techniques

Redis keeps everything in memory, so efficiency matters. Here are strategies to reduce your memory footprint without sacrificing functionality.

Use hashes for small objects. Redis has a special memory optimization where small hashes (under hash-max-ziplist-entries and hash-max-ziplist-value thresholds, renamed to hash-max-listpack-entries / hash-max-listpack-value in Redis 7+) are stored as a compact structure instead of a full hash table. Grouping related keys into hashes can cut memory usage significantly.

interface UserProfile {
  name: string;
  email: string;
  plan: string;
}

// Instead of multiple individual keys (memory-heavy):
async function inefficientStorage(userId: string, profile: UserProfile): Promise<void> {
  await redis.set(`user:${userId}:name`, profile.name);
  await redis.set(`user:${userId}:email`, profile.email);
  await redis.set(`user:${userId}:plan`, profile.plan);
}

// Use a single hash (memory-efficient):
async function efficientStorage(userId: string, profile: UserProfile): Promise<void> {
  await redis.hset(`user:${userId}`, profile);
}

async function getProfile(userId: string): Promise<UserProfile | null> {
  const data = await redis.hgetall(`user:${userId}`);
  if (!data || Object.keys(data).length === 0) return null;
  return data as unknown as UserProfile;
}

// Partial reads and updates
async function updatePlan(userId: string, newPlan: string): Promise<void> {
  await redis.hset(`user:${userId}`, "plan", newPlan);
}

async function getEmail(userId: string): Promise<string | null> {
  return redis.hget(`user:${userId}`, "email");
}

Set appropriate TTLs on everything. Stale data accumulating without expiration is the most common cause of unexpected memory growth.

// Set TTL at write time
async function cacheWithTTL(key: string, value: string, ttlSeconds: number): Promise<void> {
  await redis.set(key, value, "EX", ttlSeconds);
}

// Helper: cache with automatic JSON serialization
async function cacheJSON<T>(key: string, data: T, ttlSeconds: number): Promise<void> {
  await redis.set(key, JSON.stringify(data), "EX", ttlSeconds);
}

async function getCachedJSON<T>(key: string): Promise<T | null> {
  const raw = await redis.get(key);
  if (!raw) return null;
  try {
    return JSON.parse(raw) as T;
  } catch {
    // Corrupted or unexpected data - delete the bad key and return a cache miss
    await redis.del(key);
    return null;
  }
}

Use object and memory commands to understand how Redis is actually storing your data.

async function inspectKey(key: string): Promise<void> {
  const encoding = await redis.object("ENCODING", key);
  const memoryUsage = await redis.memory("USAGE", key);
  const ttl = await redis.ttl(key);

  console.log(`Key: ${key}`);
  console.log(`  Encoding: ${encoding}`);     // "ziplist", "hashtable", "embstr", etc.
  console.log(`  Memory: ${memoryUsage} bytes`);
  console.log(`  TTL: ${ttl === -1 ? "no expiry" : `${ttl}s`}`);
}

Redis Cluster and High Availability

For production workloads, a single Redis instance is a single point of failure. ioredis has first-class support for both Sentinel and Cluster modes.

// Sentinel setup for automatic failover
const sentinelRedis = new Redis({
  sentinels: [
    { host: "sentinel-1", port: 26379 },
    { host: "sentinel-2", port: 26379 },
    { host: "sentinel-3", port: 26379 },
  ],
  name: "mymaster",
});

// Cluster setup for horizontal scaling
const cluster = new Redis.Cluster([
  { host: "node-1", port: 6380 },
  { host: "node-2", port: 6381 },
  { host: "node-3", port: 6382 },
], {
  redisOptions: {
    password: "your_password",
  },
  scaleReads: "replica", // Read from replicas to distribute load
});

Hash tags ensure related keys land on the same node, which is required for multi-key operations in cluster mode.

async function clusterSafeOperations(userId: string): Promise<void> {
  const pipeline = cluster.pipeline();

  pipeline.set(`{user:${userId}}:profile`, JSON.stringify({ name: "Alice" }));
  pipeline.set(`{user:${userId}}:sessions`, JSON.stringify(["sess_abc"]));
  pipeline.set(`{user:${userId}}:preferences`, JSON.stringify({ theme: "dark" }));

  await pipeline.exec();

  // Multi-key operations work because all keys share the same hash tag
  const results = await cluster.mget(
    `{user:${userId}}:profile`,
    `{user:${userId}}:sessions`,
    `{user:${userId}}:preferences`
  );
  console.log("All user data:", results);
}

Cross-slot operations fail in cluster mode. You cannot run mget, sunion, or Lua scripts across keys on different nodes. Design your key structure around hash tags from the start if you plan to use cluster mode. Retrofitting this later is painful.

Persistence: RDB vs AOF

Understanding Redis persistence options prevents data loss surprises.

RDB snapshots create point-in-time dumps at configured intervals. They're compact, fast to load, and great for backups, but you can lose data between snapshots. AOF (Append Only File) logs every write operation, giving you much better durability at the cost of larger files and slightly more I/O.

The recommended production setup is to use both: AOF for durability with appendfsync everysec (a good compromise between safety and performance), and RDB for fast restarts and backups. Redis 7+ supports Multi-Part AOF, which eliminates the need for periodic AOF rewrites and simplifies operations.

# redis.conf
save 900 1          # RDB: snapshot after 900s if at least 1 key changed
save 300 10         # RDB: snapshot after 300s if at least 10 keys changed
appendonly yes      # Enable AOF
appendfsync everysec # Fsync once per second

You can also trigger persistence operations programmatically through ioredis if needed for backup workflows.

async function triggerBackup(): Promise<void> {
  // Trigger a background RDB snapshot
  await redis.bgsave();

  // Or trigger an AOF rewrite
  await redis.bgrewriteaof();

  // Check last save time
  const lastSave = await redis.lastsave();
  console.log(`Last RDB save: ${new Date(lastSave * 1000).toISOString()}`);
}

Wrapping Up

Redis is deceptively simple on the surface, but there's enormous depth once you move beyond basic string operations. Sorted sets, streams, and Lua scripting alone can replace entire microservices worth of logic. Combine those with proper memory optimization, pipelining, and a well-planned cluster topology, and you have an infrastructure component that punches far above its weight.

Pick one technique here that applies to a problem you're currently solving. Implement it, benchmark it, and you'll likely find that Redis was already capable of something you were building from scratch elsewhere.

MongoDB Advanced Techniques: Going Beyond the Basics

Wahyu Tricahyo — Sat, 14 Feb 2026 05:36:43 +0000

This post covers the MongoDB patterns and strategies I reach for most often when building applications that need to handle real traffic. Aggregation pipelines, schema design patterns with working examples, index strategies that survive production, and the operational features that tie it all together.

What's covered:

The Aggregation Pipeline
Schema Design Patterns - Bucket, Computed, Outlier
Time-Series Collections
Index Strategies - ESR rule, partial indexes, wildcard indexes
Change Streams
Transactions - write concerns, retryable writes, session handling
Query Performance

The Aggregation Pipeline: Your Most Powerful Tool

The aggregation pipeline is where MongoDB truly shines. Think of it as a series of stages that your data flows through, each one transforming it along the way. While most developers know $match and $group, the real power comes from combining lesser-known stages.

$facet lets you run multiple aggregation pipelines in parallel on the same set of input documents. This is incredibly useful when you need to compute different summaries from the same dataset - say, a total count, a breakdown by category, and an average price - all in a single query instead of three separate round trips.

db.products.aggregate([
  { $facet: {
    totalCount: [{ $count: "count" }],
    byCategory: [{ $group: { _id: "$category", total: { $sum: 1 } } }],
    priceStats: [{ $group: { _id: null, avgPrice: { $avg: "$price" }, maxPrice: { $max: "$price" } } }]
  }}
]);

$lookup with pipelines is another game-changer. Instead of a basic left outer join, you can embed a full sub-pipeline inside the lookup. This means you can filter, project, and even sort the joined documents before they're merged into your results. It's the closest thing MongoDB has to a SQL subquery, and it's incredibly flexible.

db.orders.aggregate([
  { $lookup: {
    from: "products",
    let: { productIds: "$items.productId" },
    pipeline: [
      { $match: { $expr: { $in: ["$_id", "$$productIds"] } } },
      { $project: { name: 1, price: 1 } }
    ],
    as: "productDetails"
  }}
]);

Schema Design Patterns That Scale

MongoDB's flexibility with schema is a double-edged sword. Without discipline, your database can become a mess. Here are patterns that hold up under pressure.

The Bucket Pattern works wonders for time-series data. Instead of storing one document per event, you group events into buckets - say, one document per hour or per day. This reduces the total document count, makes range queries faster, and plays nicely with indexes. If you're logging sensor data, analytics events, or user activity, this pattern can cut your storage and query times dramatically.

// Instead of one document per sensor reading (millions of docs):
// { sensorId: "temp-01", value: 22.5, timestamp: ISODate("2025-02-14T10:00:01Z") }
// { sensorId: "temp-01", value: 22.7, timestamp: ISODate("2025-02-14T10:00:02Z") }

// Bucket them into hourly documents:
{
  sensorId: "temp-01",
  date: ISODate("2025-02-14T10:00:00Z"),
  count: 120,
  readings: [
    { value: 22.5, timestamp: ISODate("2025-02-14T10:00:01Z") },
    { value: 22.7, timestamp: ISODate("2025-02-14T10:00:02Z") },
    // ... up to a cap (e.g., 200 readings per bucket)
  ],
  summary: { min: 21.8, max: 23.4, avg: 22.6 }
}

// Insert a new reading into the current bucket
db.sensorData.updateOne(
  { sensorId: "temp-01", date: ISODate("2025-02-14T10:00:00Z"), count: { $lt: 200 } },
  {
    $push: { readings: { value: 22.9, timestamp: new Date() } },
    $inc: { count: 1 },
    $min: { "summary.min": 22.9 },
    $max: { "summary.max": 22.9 },
  },
  { upsert: true }
);

// Query an entire day with just 24 documents instead of thousands
db.sensorData.find({
  sensorId: "temp-01",
  date: { $gte: ISODate("2025-02-14"), $lt: ISODate("2025-02-15") }
});

The Computed Pattern is all about precomputing values that are expensive to calculate on the fly. If your application frequently displays a product's average rating, don't aggregate all reviews every time. Instead, update a ratingSummary field whenever a new review is added. You trade slightly more work on writes for significantly faster reads.

// Product document with precomputed rating data
{
  _id: ObjectId("..."),
  name: "Wireless Headphones",
  price: 79.99,
  ratingSummary: {
    average: 4.3,
    count: 256,
    total: 1100,         // sum of all ratings, used to recalculate average
    distribution: { 1: 8, 2: 12, 3: 30, 4: 86, 5: 120 }
  }
}

// When a new 5-star review comes in, recalculate the average atomically
// using a pipeline-style update (note the array wrapper - this is required
// to use aggregation expressions like $divide and $add inside updateOne)
db.products.updateOne(
  { _id: productId },
  [
    {
      $set: {
        "ratingSummary.count": { $add: ["$ratingSummary.count", 1] },
        "ratingSummary.total": { $add: ["$ratingSummary.total", 5] },
        "ratingSummary.distribution.5": { $add: ["$ratingSummary.distribution.5", 1] },
        "ratingSummary.average": {
          $round: [
            { $divide: [{ $add: ["$ratingSummary.total", 5] }, { $add: ["$ratingSummary.count", 1] }] },
            1
          ]
        }
      }
    }
  ]
);

// Now displaying a product page requires zero aggregation - just read the document
db.products.findOne({ _id: productId }, { name: 1, price: 1, ratingSummary: 1 });

The key detail here is the array wrapper around the update document. A regular update ({ $inc: ..., $set: ... }) doesn't support aggregation operators like $divide or $add. Wrapping the update in an array tells MongoDB to treat it as an aggregation pipeline, which unlocks the full expression language. Without the array, those expressions get stored as literal objects instead of being evaluated.

The Outlier Pattern handles the reality that not all documents are created equal. Most users might have 10 orders, but a handful might have 10,000. Embedding all orders in the user document works for the majority, but blows up for the outliers. The solution: embed up to a threshold, then overflow into a separate collection with a reference back to the parent.

// User document with embedded orders (works for 99% of users)
{
  _id: "user:1001",
  name: "Alice",
  hasOverflow: false,
  orders: [
    { orderId: "ord-001", total: 49.99, date: ISODate("2025-01-15") },
    { orderId: "ord-002", total: 129.00, date: ISODate("2025-02-01") },
    // embedded up to a threshold, say 50 orders
  ]
}

// When a user exceeds the threshold, flip the flag and overflow to a separate collection
const MAX_EMBEDDED = 50;

async function addOrder(userId, order) {
  const result = await db.users.updateOne(
    { _id: userId, hasOverflow: false, [`orders.${MAX_EMBEDDED - 1}`]: { $exists: false } },
    { $push: { orders: order } }
  );

  // If update didn't match, the user has hit the threshold
  if (result.matchedCount === 0) {
    // Mark the user as overflowed
    await db.users.updateOne({ _id: userId }, { $set: { hasOverflow: true } });
    // Store in the overflow collection
    await db.orderOverflow.insertOne({ userId, ...order });
  }
}

// Note: there's a small race window between the matchedCount check and setting
// hasOverflow - a concurrent write could push past the threshold before the flag
// is set. In practice this means a user might end up with slightly more than
// MAX_EMBEDDED orders before overflowing. For most applications this is harmless.
// If you need strict enforcement, wrap the check-and-set in a transaction or use
// a findOneAndUpdate with the same filter to make the overflow decision atomic.

// Querying: check the flag to decide where to look
async function getUserOrders(userId) {
  const user = await db.users.findOne({ _id: userId });

  if (!user.hasOverflow) {
    return user.orders; // Fast path: everything is embedded
  }

  // Combine embedded orders with overflow
  const overflowOrders = await db.orderOverflow.find({ userId }).toArray();
  return [...user.orders, ...overflowOrders];
}

A note on the addOrder function: there's a subtle race condition between checking matchedCount === 0 and setting hasOverflow: true. If two writes land at the same time when the user is at exactly the threshold, both could fail the initial updateOne, and one could insert into the overflow collection before the flag is set. In practice this is rarely an issue since hitting the exact threshold concurrently is uncommon, and the hasOverflow flag is only used as a read-path optimization, not a consistency guarantee. If you need stronger guarantees, wrapping the overflow transition in a transaction eliminates the race entirely.

Time-Series Collections: The Native Approach

If you're considering the bucket pattern for time-series data, it's worth knowing that MongoDB 5.0+ provides native time-series collections. These handle bucketing, compression, and optimized storage automatically under the hood, so you don't have to manage bucket boundaries or summary fields yourself.

// Create a time-series collection
db.createCollection("sensorReadings", {
  timeseries: {
    timeField: "timestamp",       // required: the field containing the datetime
    metaField: "sensorId",        // optional: groups measurements from the same source
    granularity: "minutes"        // hint for internal bucketing: "seconds", "minutes", or "hours"
  },
  expireAfterSeconds: 2592000     // optional: auto-delete data older than 30 days
});

// Inserts look like regular documents - no manual bucketing needed
db.sensorReadings.insertMany([
  { sensorId: "temp-01", timestamp: new Date(), value: 22.5, unit: "celsius" },
  { sensorId: "temp-01", timestamp: new Date(), value: 22.7, unit: "celsius" },
  { sensorId: "humidity-01", timestamp: new Date(), value: 65.2, unit: "percent" },
]);

// Queries and aggregations work as expected
db.sensorReadings.aggregate([
  { $match: { sensorId: "temp-01", timestamp: { $gte: ISODate("2025-02-14"), $lt: ISODate("2025-02-15") } } },
  { $group: { _id: { $hour: "$timestamp" }, avgTemp: { $avg: "$value" }, count: { $sum: 1 } } },
  { $sort: { _id: 1 } }
]);

The manual bucket pattern still has its place - it gives you full control over bucket size, lets you store precomputed summaries, and works on older MongoDB versions. But for most time-series workloads on MongoDB 5.0+, the native collections are less code, better compression, and one fewer thing to maintain.

Index Strategies That Actually Matter

Indexes are easy to create and easy to get wrong. Here's what I've learned the hard way.

Compound index order matters more than you think. The ESR rule - Equality, Sort, Range - should guide how you order fields in a compound index. Fields used in equality matches come first, followed by sort fields, then range conditions. Getting this order wrong can be the difference between a millisecond query and a full collection scan.

// If your query is: { status: "active", createdAt: { $gte: lastWeek } } sorted by { score: -1 }
// The optimal index is:
db.posts.createIndex({ status: 1, score: -1, createdAt: 1 });
// Equality (status) -> Sort (score) -> Range (createdAt)

Partial indexes save you space and speed by only indexing documents that match a filter expression. If you only query active users, there's no reason to index the inactive ones. This is especially valuable in large collections where a significant portion of documents are irrelevant to your hot queries.

db.users.createIndex(
  { email: 1 },
  { partialFilterExpression: { isActive: true } }
);

Wildcard indexes are useful when you have dynamic or unpredictable field names - common in IoT data or user-defined metadata. Rather than creating an index for every possible field, a wildcard index covers them all.

// Index all fields inside the metadata subdocument
db.devices.createIndex({ "metadata.$**": 1 });

// Query any dynamic field under metadata
db.devices.find({ "metadata.firmware_version": "2.1.0" });
db.devices.find({ "metadata.region": "us-east-1" });

There are real tradeoffs to understand before using them. Every write to an indexed field path creates or updates index entries for each matching field, so write-heavy workloads will see higher I/O and CPU overhead compared to a targeted compound index. Wildcard indexes also consume more memory since they maintain entries across all indexed field paths. Most importantly, they can't support compound queries - a wildcard index on metadata.$** can serve a filter on metadata.region or metadata.firmware_version individually, but not both in the same query. If you know your access patterns, a compound index on the specific fields will always outperform a wildcard. Reserve wildcards for the genuinely unpredictable cases where you can't anticipate which fields will be queried.

Change Streams: Real-Time Without the Complexity

Change streams let you watch for real-time changes to your data without polling. They sit on top of MongoDB's oplog and give you a cursor that emits events whenever documents are inserted, updated, replaced, or deleted.

The practical applications are everywhere: syncing data to a search engine, triggering notifications, maintaining materialized views, or feeding an event-driven architecture. Unlike tailing the oplog directly, change streams are resumable - if your application crashes, you can pick up exactly where you left off using a resume token.

Every change event includes an _id field that acts as the resume token. By saving it to a dedicated collection after processing each event and passing it back as resumeAfter on startup, you create a checkpoint that survives crashes and restarts. Keep in mind that resume tokens are only valid while the oplog still contains the corresponding entry - typically 24-72 hours depending on your cluster. If your application is down longer than that, the token expires and you'll need to handle a full re-sync. This gives you at-least-once delivery, so if your processing isn't idempotent, deduplicate on the consumer side.

const RESUME_TOKEN_COLLECTION = "changeStreamTokens";

async function saveResumeToken(streamName, token) {
  await db.collection(RESUME_TOKEN_COLLECTION).updateOne(
    { _id: streamName },
    { $set: { token, updatedAt: new Date() } },
    { upsert: true }
  );
}

async function getResumeToken(streamName) {
  const doc = await db.collection(RESUME_TOKEN_COLLECTION).findOne({ _id: streamName });
  return doc?.token ?? null;
}

async function watchTickets() {
  const pipeline = [{ $match: { "fullDocument.priority": "high" } }];
  const options = { fullDocument: "updateLookup" };

  const resumeToken = await getResumeToken("tickets-high-priority");
  if (resumeToken) {
    options.resumeAfter = resumeToken;
    console.log("Resuming change stream from stored token");
  }

  const changeStream = db.collection("tickets").watch(pipeline, options);

  changeStream.on("change", async (event) => {
    console.log("High-priority ticket changed:", event.fullDocument);
    // Persist the token after successful processing
    await saveResumeToken("tickets-high-priority", event._id);
  });

  changeStream.on("error", (err) => {
    console.error("Change stream error:", err);
    // Reconnect after a delay - the saved token lets us pick up where we left off
    setTimeout(() => watchTickets(), 5000);
  });
}

Transactions: When You Actually Need Them

Multi-document transactions in MongoDB exist, but they shouldn't be your default approach. If you find yourself needing transactions frequently, it might be a signal that your schema design needs rethinking. That said, there are legitimate cases - financial operations, inventory management, or any workflow where partial updates would leave your data in an inconsistent state.

const session = client.startSession();
try {
  session.startTransaction({
    readConcern: { level: "snapshot" },
    writeConcern: { w: "majority" },
    readPreference: "primary"
  });

  await db.accounts.updateOne({ _id: from }, { $inc: { balance: -amount } }, { session });
  await db.accounts.updateOne({ _id: to }, { $inc: { balance: amount } }, { session });
  await session.commitTransaction();
} catch (error) {
  await session.abortTransaction();
  throw error;
} finally {
  session.endSession();
}

Read and write concerns matter inside transactions. The readConcern: "snapshot" ensures the transaction sees a consistent point-in-time view of the data, so reads within the transaction won't see writes from other concurrent operations. The writeConcern: { w: "majority" } guarantees that committed data has been acknowledged by a majority of replica set members before the transaction reports success. Without majority write concern, a committed transaction could be rolled back if the primary fails before replicating the writes. For financial or critical operations, these aren't optional.

Retryable writes and the retry loop. MongoDB supports retryable writes (retryWrites=true in your connection string, which is the default in modern drivers). This handles transient network errors on individual write operations automatically. But transactions need their own retry logic because the entire transaction can fail due to write conflicts or transient errors, not just a single operation within it. The recommended pattern wraps the transaction in a loop:

async function transferWithRetry(client, from, to, amount, maxRetries = 3) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const session = client.startSession();
    try {
      session.startTransaction({
        readConcern: { level: "snapshot" },
        writeConcern: { w: "majority" },
      });

      const sender = await db.accounts.findOne({ _id: from }, { session });
      if (sender.balance < amount) {
        await session.abortTransaction();
        throw new Error("Insufficient funds");
      }

      await db.accounts.updateOne({ _id: from }, { $inc: { balance: -amount } }, { session });
      await db.accounts.updateOne({ _id: to }, { $inc: { balance: amount } }, { session });
      await session.commitTransaction();
      return; // success

    } catch (error) {
      await session.abortTransaction();

      // TransientTransactionError: safe to retry the whole transaction
      if (error.hasOwnProperty("errorLabels") && error.errorLabels.includes("TransientTransactionError") && attempt < maxRetries - 1) {
        console.log(`Transient error on attempt ${attempt + 1}, retrying...`);
        continue;
      }
      throw error;
    } finally {
      session.endSession();
    }
  }
}

A few ground rules for transactions: keep them under a second if possible - long-running transactions hold WiredTiger snapshots open, increasing cache pressure and blocking other operations. Don't use them across collections that live on different shards unless absolutely necessary, as distributed transactions carry substantially more latency. And if your application is using transactions to coordinate writes across three or more collections regularly, that's usually a sign to revisit whether those collections should be restructured.

Query Performance: Reading `explain()` Like a Pro

Every developer should be comfortable reading the output of explain("executionStats"). The key metrics to watch are totalDocsExamined versus nReturned. If you're examining 100,000 documents to return 10 results, something is wrong with your indexing strategy.

Look for COLLSCAN in the winning plan - that's a collection scan, and outside of very small collections or analytics workloads, it's usually a red flag. You want to see IXSCAN followed by a FETCH, or ideally a covered query where the index alone satisfies the request without touching the documents at all.

db.orders.find({ status: "pending", total: { $gt: 100 } })
  .sort({ createdAt: -1 })
  .explain("executionStats");

Wrapping Up

Each section here solves a specific class of problem: the aggregation pipeline replaces what would otherwise be multi-query application logic, schema patterns prevent document bloat before it starts, proper indexing keeps reads fast without tanking writes, and transactions with the right concerns protect data integrity when single-document atomicity isn't enough.

If any of these sections felt unfamiliar, the best next step is to run explain() on your slowest queries and see what comes back. That tends to reveal which of these techniques your application actually needs.

Forem: Wahyu Tricahyo

Active-Passive vs Active-Active: How to Know Which Pattern Fits Your Database

First: Measure Before You Architect

Check Your Read/Write Ratio in PostgreSQL

Database-Level: pg_stat_database

Per-Table: pg_stat_user_tables

Per-Query: pg_stat_statements

What Your Numbers Mean

The Two Patterns: What Actually Matters

Active-Passive: The Gotchas

Active-Active: The Complexity Tax

Side-by-Side

How OpenAI Applies This

Operational Defenses

Moving Write-Heavy Tables, Not the Whole Database

Cascading Replication: Scaling Past the Replica Ceiling

The Hybrid Approach

The Real Lesson

References

Further Reading

Redis Advanced Techniques: Beyond Simple Key-Value Storage

Data Structures You're Probably Not Using

Lua Scripting: Atomic Operations on Steroids

Pipelining and Transactions

Pub/Sub and Beyond

Memory Optimization Techniques

Redis Cluster and High Availability

Persistence: RDB vs AOF

Wrapping Up

MongoDB Advanced Techniques: Going Beyond the Basics

The Aggregation Pipeline: Your Most Powerful Tool

Schema Design Patterns That Scale

Time-Series Collections: The Native Approach

Index Strategies That Actually Matter

Change Streams: Real-Time Without the Complexity

Transactions: When You Actually Need Them

Query Performance: Reading explain() Like a Pro

Wrapping Up

Database-Level: `pg_stat_database`

Per-Table: `pg_stat_user_tables`

Per-Query: `pg_stat_statements`

Query Performance: Reading `explain()` Like a Pro