Forem: Steven Leggett

Debug Real Production Incidents in Your Browser Before They Happen to You

Steven Leggett — Tue, 24 Mar 2026 20:12:26 +0000

The Situation Room - a 3D war room with live architecture, cascading alerts, and a ticking clock

The Idea: A Flight Simulator for On-Call Engineers

Here's the thing about learning incident response: you learn it by breaking things in production, on a clock, while people are angry. There is no training simulator. You get one real scenario at a time, with real consequences, and you either figure it out or you don't.

Flight simulators exist so pilots can experience a landing gear failure, a stall at 30,000 feet, a hydraulics problem - without dying. Musicians practice scales before performing. Athletes train before competing. Surgeons use simulation before cutting into a real person.

But SREs? You get dropped into a live incident and told to figure it out. Maybe your company has a "Wheel of Misfortune" session occasionally, but realistically you just wait for the pager to go off and hope the scenario isn't too bad.

I wanted to build something different. A place where you could practice the muscle memory of incident response without the 3 AM wake-up call. Where you could learn what a connection pool exhaustion actually looks like in the logs before you encounter it at 2 AM with a VP watching over your shoulder.

The tagline came quickly: Wordle meets Hack The Box for DevOps.

Wordle because it needed to be quick, daily, shareable, and satisfying
Hack The Box because it needed to be genuinely technical and respect your intelligence
For DevOps because SREs, platform engineers, and backend developers are the audience

I wanted the game to feel like you were actually sitting at a terminal in a crisis. Not a quiz. Not a multiple choice test. A real investigation.

Building the Thing

Picking the Stack

I wanted to go from zero to playable as fast as possible, and I wanted the infrastructure costs to scale from "$0 at launch" to "reasonable at 10,000 users." That meant serverless-first.

Next.js 16 was a given. App Router, React Server Components, edge runtime, first-class Vercel support. I've used it for years and I can move fast in it.

Turso for the database. This was the interesting choice. Turso is SQLite at the edge - it gives you sub-10ms reads globally, scales to zero when nobody is playing, and the free tier covers a lot of users. I was already using Drizzle ORM and had a schema ready. The alternative was Supabase's Postgres, but for a read-heavy app (leaderboards, profile lookups) Turso made more sense and the economics were better at launch.

Supabase Auth for authentication. Free up to 50,000 monthly active users. GitHub OAuth and Google OAuth both built in. It handles email verification, password resets, and OAuth flows out of the box. When you're building solo, you don't want to think about auth.

Tailwind v4 for styling. The v4 release with its CSS-first configuration cleaned up a lot of the config overhead I used to fight with.

Zustand for client state. The game state during an active incident - your command history, your elapsed time, your discovered clues, your score - lives in a Zustand store. Simple, no boilerplate.

Anime.js v4 for animations. This was the fun one.

The Game Engine

Clicking a service node in the architecture map reveals alerts and investigation options

The core of the whole thing is IncidentEngine.ts - a TypeScript class that runs each scenario.

Every incident is defined as a TypeScript object: it has metadata (title, difficulty, time limit), a scenario description, an environment with simulated logs and metrics, a list of commands the player can run, diagnosis options, and a solution with hints.

The engine does a few things:

Manages the game timer and emits events at warning thresholds (30% time remaining, 10% time remaining, time expired)
Processes commands by pattern-matching against the incident's defined commands, then checking shared easter egg commands
Tracks which clues have been discovered as the player runs investigative commands
Handles diagnosis submission - correct gets you to the "fixing" phase, incorrect costs you points
Validates fix commands with flexible matching so you don't have to type a 200-character shell command perfectly
Calculates a final score based on time taken vs par time, whether you diagnosed correctly on the first try, and how many commands you used vs the optimal path

The command system was one of the more interesting design challenges. Real terminal commands have arguments, flags, and variations. I couldn't enumerate every possible way someone might run df -h vs df -H vs df --human-readable. The solution was pattern matching with some tolerance - commands match if the input starts with the pattern, or if it matches a regex, or if it contains enough of the key terms.

The clue discovery system is what makes the game feel like an investigation rather than a quiz. Running df -h reveals the clue "var-full." Running du -sh /var/log/* reveals "analytics-logs." Each clue unlocked drives you deeper into the diagnosis. The engine emits a clue_discovered event that triggers a satisfying UI animation.

What I Learned Building It

The balance between educational and fun is harder than it sounds.

Every scenario has to be realistic enough that a working engineer recognizes the pattern from their own experience. But it also has to be solvable in a few minutes by someone who hasn't been doing this for ten years. Beginner scenarios need to feel approachable. Advanced ones need to feel like a genuine challenge.

I spent more time on scenario design than on any other part of the project. The disk-full scenario alone went through several iterations before it felt right. The key insight was: start with real postmortems. The disk-full debug-logging story is a thing that actually happens. The DB connection pool exhaustion that has a leak in the error handling path - real. The K8s crash loop from a process trying to bind to port 80 as a non-root user - real. The DNS failure that's half because of propagation and half because of TTL misconfiguration - real.

Scenarios based on real patterns teach real skills. Made-up scenarios feel like trivia.

Scoped animations in React are worth the setup cost.

Anime.js v4's scoped animation system was unfamiliar at first. The v3 API was simpler. But the v4 approach where you attach animations to a DOM ref and call scope.revert() on cleanup eliminated an entire class of bugs I was hitting - animations targeting elements that no longer existed in the DOM, intervals that kept firing after a game ended. The useAnime hook is 20 lines of code and everything downstream just works.

TypeScript for game data is underrated.

Defining incidents as typed TypeScript objects gives you autocomplete on every field, type errors if you forget a required property, and no JSON parsing issues. The alternative would be a database of scenario definitions or a YAML/JSON format. Those have their place, but for this project where the scenario logic is complex and needs to be right, having the TypeScript compiler check your work is worth it.

Write the easter eggs.

I almost cut them because they felt like scope creep. I'm glad I didn't. The game without easter eggs is a quiz. With them, it's something you want to share. The chatgpt easter egg - where ChatGPT confidently gives you unhelpful advice and then asks if maybe you're using MongoDB - has gotten more reactions in playtesting than any feature I actually planned.

SQLite at the edge is genuinely good now.

Turso was a gamble as the database choice. SQLite has a reputation for being a toy. But Turso's implementation is production-ready. The Drizzle integration is clean. The free tier is generous. For a read-heavy app with global users, edge replication means reads that actually are fast regardless of where you are. I haven't had a single database-related issue since launch.

The Situation Room

The full war room mid-incident - architecture map, live metrics, alert feed, and terminal all running simultaneously

The Situation Room - a 3D interactive war room. Live architecture visualization with traffic flows and service dependencies rendered in real time. Cascading alerts. Streaming logs. A countdown clock as revenue drops by the second. Countdown music that kicks in at 45 seconds and genuinely stresses me out even though I wrote the thing.

The first Situation Room scenario is a Cyber Monday DNS failure. Your payment provider becomes unreachable at peak traffic. You have the full service map in front of you - every node clickable, every service inspectable. Four difficulty modes from guided (5 services, extra hints) to Hardcore with permadeath and a 30-service enterprise architecture full of red herrings.

Nobody has earned the "Nerves of Steel" badge yet. That's the one for beating the Situation Room on Hardcore.

The Result

10+ scenarios are live at youbrokeprod.com, free to play, no credit card required.

Beginner:

Disk full from debug logs left on in production (the 3 AM story above)
Expired SSL certificate (everyone's had this one)
DB connection pool exhaustion from a connection leak in the error path

Intermediate:

K8s CrashLoopBackOff from a port binding permission error
Mobile API breaking change that someone forgot to version
DNS failure in the Situation Room - the new 3D war room mode

Advanced:

Memory leak that only reproduces under production traffic patterns
Redis thundering herd when all connections in the pool expire simultaneously
Lambda cold start cascade turning a routine deploy into a 10-minute outage
The terraform destroy scenario based on the real DataTalksClub incident (685K+ views, 9% win rate)

Each scenario has a scoring system (time, accuracy, efficiency), progressive hints, easter eggs, and a result card you can share. The Situation Room adds live architecture visualization, post-game postmortems with real-world incidents (Dyn 2016, Fastly 2021, Cloudflare 2020), and difficulty modes that scale the architecture complexity.

The game is free. Pro unlocks harder difficulty modes and deep-dive postmortems with actual monitoring configs and prevention playbooks you can take back to work. For now I just want engineers to play it, get better at incident response, and maybe think about setting up disk space monitoring before they need it at 3 AM.

If you've ever sat at a terminal at 2 AM and typed "why is everything broken" into a command prompt as if the computer would answer you directly - this game is for you.

Traffic flowing through the architecture before the incident hits

Play it at youbrokeprod.com. Try the Situation Room if you want the full 3D war room experience, or start with the disk-full scenario if you want to ease in. Either way, the countdown is ticking.

I built a training simulator for the dev skills nobody teaches

Steven Leggett — Tue, 17 Mar 2026 18:21:00 +0000

Pop quiz. What's wrong with this code?

router.post('/login', async (req, res) => {
  const { email, password } = req.body;

  logger.info('Login attempt', {
    email,
    password,           // LINE 5
    ip: req.ip
  });

  const user = await db.query(
    'SELECT * FROM users WHERE email = $1',
    [email]
  );
  const u = user.rows[0];

  // ... auth check ...

  analytics.track('user_login', {
    email: u.email,
    ssn: u.ssn_last4,        // LINE 22
    creditScore: u.credit_score,
  });

  return res.json({
    token,
    user: {
      email: u.email,
      passwordHash: u.password_hash,  // LINE 30
      ssn: u.ssn_last4,
      creditScore: u.credit_score,
    }
  });
});

There are 7 issues in there. How many did you spot? Did you catch that line 5 writes plaintext passwords to your log aggregator? That line 22 sends SSN data to a third-party analytics service, violating GDPR Article 28? That line 30 returns the password hash in the API response?

This is a real scenario from LearningTo.co - a training platform I built for the dev skills that CS programs don't teach.

The problem

I've been hiring and mentoring developers for years and I keep seeing the same pattern. Junior devs come in knowing algorithms and data structures but have never:

Reviewed a pull request and caught a real bug
Identified PII leaking through logs or API responses
Recovered from a force-push that wiped a teammate's work
Looked at AI-generated code and said "this looks right but it's wrong"

These aren't edge cases. This is Tuesday on a real engineering team. And almost nobody teaches them.

What I built

LearningTo.co is scenario-based training. You get dropped into a realistic situation - a PR review, a failing pipeline, a suspicious config - and you have to investigate, flag issues, and explain your reasoning.

Each scenario gives you context like a real team would: "You've just joined a fintech startup. Your tech lead drops a PR review in your queue." Then you're evaluated on three things: what you find, how you explain it, and whether you can prioritize severity.

Here's what gameplay looks like

You see the code. A timer's running. You click lines to flag them. Then you write your analysis explaining what's wrong and why it matters.

The key: reasoning quality is scored, not just finding the line numbers. Saying "line 6 has a bug" gets you nothing. Saying "line 6 logs the plaintext password to the log aggregator, which means anyone with log access can see user credentials" gets you points.

After you submit, you get a full debrief: what you found, what you missed, and why each issue matters in production. A score, a grade, and a ship/block verdict.

The AI Code Review course (free, 8 chapters)

This is the part I'm most proud of. GitHub teaches you how to use Copilot. Nobody teaches you how to review what it produces.

Eight hands-on chapters, each a real scenario:

Copilot's First CRUD - it compiles, it runs, but should it ship? (SQL injection, no auth, swallowed errors)
The Hallucinated Import - half the imports point to packages that don't exist on npm
The 100% Coverage Lie - AI-generated tests show perfect coverage but test nothing meaningful
The Confident Refactor - AI "improved readability" but silently removed business rules and compliance checks
Audit the Agent - an AI support agent makes tool calls to external APIs. What could go wrong?
MCP Server Lockdown - an MCP server gives an AI assistant access to your filesystem and database
The Destructive Migration - AI-generated schema migration. Your DBA is on vacation. You're the last line of defense.
CI Pipeline from Copilot - it builds, tests, and deploys. Find the supply chain and injection risks.

That screenshot above is Chapter 2 - "The Hallucinated Import." Can you spot which of those imports point to real npm packages and which were completely fabricated by Copilot? (@react-toolkit/data-grid doesn't exist. Neither does use-smart-fetch. And next/analytics is not a thing.)

The entire course is free. No paywall, no credit card.

What's live

4 categories, 17 scenarios:

Category	Scenarios	What you practice
PII & Security (`/ntrol`)	4	API key exposure, login endpoint bugs, webhook handling, user profile data leaks
Git & Gitflow (`/mmit`)	3	Merge conflicts, force-push recovery, conventional commits
Code Reasoning (`/de`)	2	Auditing AI-generated services, spotting subtle bugs
AI Code Review (`/view`)	8	The full course above

The URL taxonomy is a nerdy Easter egg - every slug completes the phrase "learning to..." (learning to /mmit, learning to /ntrol, learning to /de, learning to /view).

7 more categories are coming: Observability, Debugging, Code Review, CI/CD, Performance, Incident Ownership, and Testing.

Who it's for

CS students about to start their first internship or job
Bootcamp grads who can build features but haven't done team-based dev work
Junior devs who want to level up on the skills that get noticed in code reviews
Senior devs - try the AI Code Review course. I guarantee Copilot has snuck something past you that you didn't catch.

Try it

Head to learningto.co and start with The Login Endpoint - it has 7 PII issues hidden in a fintech auth route. See how many you can find in 10 minutes.

Or jump straight to the AI Code Review course if you want to see how good you really are at catching Copilot's mistakes.

Everything live is free. Sign up takes 10 seconds with GitHub or Google.

I'd love to hear: what dev skills do you wish someone had taught you before your first job? Drop a comment - it'll probably become a scenario.

Setting Up CocoIndex with Docker and pgvector - A Practical Guide

Steven Leggett — Tue, 17 Mar 2026 18:20:44 +0000

Setting Up CocoIndex with Docker and pgvector - A Practical Guide

CocoIndex is a data transformation framework for AI that handles indexing with incremental processing. It uses a Rust engine with Python bindings, which means it's fast, but the setup has a few gotchas that aren't obvious from the docs. The project is open source on GitHub.

I spent an afternoon getting it running locally and hit every sharp edge so you don't have to. Here's what actually works.

What You'll Build

A pipeline that reads markdown files, chunks them, generates vector embeddings using sentence-transformers, and stores them in PostgreSQL with pgvector for semantic similarity search.

Prerequisites

Python 3.11 to 3.13 (officially supported - 3.14 works but isn't listed yet)
Docker
About 10 minutes

Step 1: PostgreSQL with pgvector (not plain Postgres)

This is the first thing that will bite you. CocoIndex requires the vector extension for HNSW indexes. Plain postgres:16 or postgres:17 will fail with extension "vector" is not available.

CocoIndex provides a docker compose config you can use directly:

docker compose -f <(curl -L https://raw.githubusercontent.com/cocoindex-io/cocoindex/refs/heads/main/dev/postgres.yaml) up -d

Or run the container manually:

docker run -d --name cocoindex-postgres \
  -e POSTGRES_USER=cocoindex \
  -e POSTGRES_PASSWORD=cocoindex \
  -e POSTGRES_DB=cocoindex \
  -p 5432:5432 \
  pgvector/pgvector:pg17

If port 5432 is already in use, pick a different host port (e.g., -p 5450:5432) and adjust your connection string.

Port tip: Before picking a port, check nothing else is listening there. SSH tunnels can silently bind to the same port as Docker, causing misleading "password authentication failed" errors even when your credentials are correct. Verify with:

lsof -i :5432

You should only see Docker's com.docker process, not SSH or anything else.

Step 2: Python Environment

mkdir cocoindex-quickstart && cd cocoindex-quickstart
python3 -m venv .venv
source .venv/bin/activate
pip install -U 'cocoindex[embeddings]'

The [embeddings] extra pulls in sentence-transformers and torch. It's a big download but gives you local embeddings with no API key needed.

Step 3: Configure the Database Connection

Create a .env file in your project root. CocoIndex reads it automatically via python-dotenv:

echo 'COCOINDEX_DATABASE_URL=postgresql://cocoindex:cocoindex@localhost:5432/cocoindex' > .env

Adjust the port if you mapped to something other than 5432 in Step 1.

Step 4: Write the Pipeline

Create main.py:

import cocoindex

@cocoindex.flow_def(name="TextEmbedding")
def text_embedding_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope):
    data_scope["documents"] = flow_builder.add_source(
        cocoindex.sources.LocalFile(path="markdown_files"))

    doc_embeddings = data_scope.add_collector()

    with data_scope["documents"].row() as doc:
        doc["chunks"] = doc["content"].transform(
            cocoindex.functions.SplitRecursively(),
            language="markdown", chunk_size=2000, chunk_overlap=500)

        with doc["chunks"].row() as chunk:
            chunk["embedding"] = chunk["text"].transform(
                cocoindex.functions.SentenceTransformerEmbed(
                    model="sentence-transformers/all-MiniLM-L6-v2"
                )
            )
            doc_embeddings.collect(
                filename=doc["filename"],
                location=chunk["location"],
                text=chunk["text"],
                embedding=chunk["embedding"],
            )

    doc_embeddings.export(
        "doc_embeddings",
        cocoindex.storages.Postgres(),
        primary_key_fields=["filename", "location"],
        vector_indexes=[
            cocoindex.VectorIndexDef(
                field_name="embedding",
                metric=cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY,
            )
        ],
    )

Step 5: Add Some Content

mkdir markdown_files

Drop some markdown files in there. For testing, even a couple of files work. The pipeline will chunk them, embed each chunk, and store the vectors.

Step 6: Run the Indexer

cocoindex update main.py

It will show you the tables it needs to create and ask for confirmation. Type yes.

You'll see it load the sentence-transformers model (first run downloads it from HuggingFace), create the pgvector extension, build the HNSW index, and process your files. Output looks like:

TextEmbedding.documents (batch update): 2/2 source rows: 2 added

Step 7: Query with Semantic Search

Install psycopg2 and create a simple query script:

pip install psycopg2-binary

# query.py
import sys
from sentence_transformers import SentenceTransformer
import psycopg2

DB_URL = "postgresql://cocoindex:cocoindex@localhost:5432/cocoindex"

def search(query: str, top_k: int = 3):
    model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
    embedding = model.encode(query)
    vec_str = "[" + ",".join(str(x) for x in embedding) + "]"

    conn = psycopg2.connect(DB_URL)
    cur = conn.cursor()
    cur.execute("""
        SELECT filename, left(text, 200),
               1 - (embedding <=> %s::vector) as similarity
        FROM textembedding__doc_embeddings
        ORDER BY embedding <=> %s::vector
        LIMIT %s
    """, (vec_str, vec_str, top_k))

    results = cur.fetchall()
    cur.close()
    conn.close()
    return results

if __name__ == "__main__":
    query = " ".join(sys.argv[1:]) or "What is incremental processing?"
    print(f"\nQuery: {query}\n")
    for filename, text, score in search(query):
        print(f"Score: {score:.4f} | {filename}")
        print(f"  {text.strip()[:150]}...\n")

python query.py "which embedding models are popular?"

Things That Tripped Me Up

Use pgvector/pgvector, not postgres. This is the number one issue. The plain Postgres Docker image doesn't include the vector extension. You need pgvector/pgvector:pg17 (or pg16). CocoIndex will fail at table creation without it.

Table naming is lowercase. Your flow is named TextEmbedding but the table is textembedding__doc_embeddings. CocoIndex lowercases the flow name. Keep this in mind when writing direct SQL queries.

The old main_fn API is gone. If you see examples using cocoindex.main_fn(), that's outdated. The current API (v0.3.36+) uses the cocoindex CLI command directly.

Docker volume persistence. If you change Postgres env vars (user/password) but reuse the container volume, the old credentials persist. Use docker rm -v to remove the volume when recreating.

The .env file wins. CocoIndex loads .env automatically via python-dotenv. If you set COCOINDEX_DATABASE_URL in your shell but have a different value in .env, the file takes precedence. This caught me when debugging connection issues.

Port conflicts with SSH tunnels. If you're forwarding database ports over SSH (common with remote dev setups), an SSH tunnel can bind to the same port as Docker. The connection goes to the wrong Postgres, and you get auth failures that look like a password problem. Always verify the port with lsof.

Using CocoIndex with Claude Code

If you're using Claude Code, there are a couple of integrations worth knowing about.

Claude Code Skill

CocoIndex provides an official Claude Code skill that gives Claude built-in knowledge about CocoIndex's API, so it can help you write pipelines, create custom functions, and run CLI commands correctly. This would have saved me from hitting the deprecated main_fn API issue.

Install it from within Claude Code:

/plugin marketplace add cocoindex-io/cocoindex-claude
/plugin install cocoindex-skills@cocoindex

Once installed, Claude Code understands CocoIndex's current API and can generate correct pipeline code without relying on outdated examples.

MCP for Code Search

cocoindex-code is a lightweight MCP server for semantic code search:

pipx install cocoindex-code
claude mcp add cocoindex-code -- ccc mcp

It uses SQLite locally and runs its own embeddings - no Postgres required. It's a separate tool from the main CocoIndex library, focused on searching your codebase.

MCP for Postgres-backed Indexes

There is no official MCP server for the Postgres-backed pipeline we built in this guide. The main cocoindex library has a built-in HTTP server (cocoindex server main.py) that exposes REST APIs, but it uses a proprietary protocol for their CocoInsight UI, not the MCP standard.

If you need MCP access to your pgvector index, you'd need to write a thin wrapper. The query.py script above is essentially all the logic you need - wrap it in an MCP server and you're there. That's a good project for a follow-up post.

The full pipeline takes about 10 minutes to set up once you know the gotchas. The incremental processing means subsequent runs only reprocess changed files, which is where CocoIndex really shines over rebuilding indexes from scratch.

5 Production Incidents Every DevOps Engineer Should Know How to Debug

Steven Leggett — Fri, 13 Mar 2026 02:53:21 +0000

It's 2 AM. Your phone is screaming. The dashboard is red. Users are tweeting.

You have been on call long enough to know that the gap between "I think I know what's wrong" and "I know exactly what's wrong" can cost your company thousands of dollars per minute. The engineers who close that gap fast are not smarter than everyone else. They have just seen these patterns before.

Here are 5 production incidents that every DevOps engineer will encounter at some point - what they look like, why they happen, and how to debug them.

1. "No Space Left on Device"

The story

A developer was chasing a gnarly bug in production. To get more visibility, they temporarily cranked the application log level to DEBUG. They fixed the bug, merged the PR, and completely forgot to revert the log level setting.

Three weeks later, at 3 AM on a Tuesday, your monitoring fires. Every service on that host is returning 500s. The database is refusing writes. Nothing makes sense until you SSH in and run the one command that tells you everything:

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        80G   80G     0 100% /

Full. The disk is completely full. /var/log has grown to 64GB of verbose debug output nobody was watching.

Why it happens

Debug logging is chatty by design. It logs every function call, every query parameter, every header. In a high-traffic service, debug logs can generate gigabytes per hour. Combine that with a missing or misconfigured log rotation policy and you have a slow-motion disaster playing out in the background while everyone is focused on feature work.

How to debug it

# Step 1: Confirm the problem
df -h

# Step 2: Find the culprit
du -sh /var/log/*
du -sh /var/log/nginx/*

# Step 3: Immediate relief - clear old compressed logs
find /var/log -name "*.gz" -mtime +7 -delete

# Step 4: Truncate (don't delete) the active log file
truncate -s 0 /var/log/myapp/app.log

# Step 5: Check logrotate config
cat /etc/logrotate.d/myapp

The fix

Set the log level back to INFO or WARN. Fix your logrotate config to enforce retention limits. Then add a disk space alert at 80% - not 95%. By the time you hit 95%, you probably have minutes, not hours.

Key takeaway

Your logging infrastructure needs to be monitored too. The tool you use to diagnose outages can itself cause outages if you ignore it.

2. Database Connection Pool Exhaustion

The story

Traffic is normal. CPU is normal. The database server is completely idle. But your application is throwing errors that look like this:

Error: Cannot acquire connection from pool
TimeoutError: timeout of 5000ms exceeded

And users are getting 503s.

This one is maddening the first time you see it because every instinct tells you to look at the database. The database is fine. The problem is how your application is managing its connection to the database.

Why it happens

Most database drivers give you a connection pool - a fixed set of reusable connections shared across your application's threads or async workers. When a request needs to run a query, it borrows a connection from the pool. When it's done, it returns the connection.

The failure mode that is easy to miss: what happens when a request throws an exception before it returns the connection?

// This is a leak
async function getUser(id) {
  const conn = await pool.acquire();
  const result = await db.query('SELECT * FROM users WHERE id = $1', [id]);
  // If the query throws, conn is never released
  conn.release();
  return result;
}

// This is correct
async function getUser(id) {
  const conn = await pool.acquire();
  try {
    return await db.query('SELECT * FROM users WHERE id = $1', [id]);
  } finally {
    conn.release(); // Always runs, even on error
  }
}

Under normal traffic, the leak is slow enough that the pool replenishes. Under higher load, or when errors spike, the pool drains faster than it fills. Then everything queues up waiting for a connection that never comes back.

How to debug it

-- On PostgreSQL: see all active connections
SELECT state, count(*)
FROM pg_stat_activity
GROUP BY state;

-- See who is holding connections longest
SELECT pid, now() - pg_stat_activity.query_start AS duration, query, state
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY duration DESC;

Look for connections stuck in idle in transaction state. That is almost always a leak. The connection was borrowed, a transaction started, and it was never committed or rolled back.

The fix

Audit your error handling paths. Every connection acquire must have a matching release in a finally block or equivalent. Set a connectionTimeoutMillis on your pool so leaked connections get reclaimed automatically. Add an alert when active connection count exceeds 80% of pool size.

Key takeaway

Low database CPU during an "outage" is a red flag pointing to connection management, not query performance. Always check pg_stat_activity before assuming the database is healthy.

3. Kubernetes CrashLoopBackOff - The Missing Secret

The story

You deploy a new version of your application to Kubernetes. Instead of the pods coming up healthy, you see this in kubectl get pods:

NAME                          READY   STATUS             RESTARTS   AGE
myapp-7d9f8b-xkj2p            0/1     CrashLoopBackOff   4          3m

The pod starts, crashes almost immediately, Kubernetes restarts it, it crashes again, and the backoff timer grows exponentially. Within 10 minutes the pod is waiting 5 minutes between restart attempts.

Why it happens

This specific variant is one of the more frustrating ones: the app crashes on startup because it cannot find a required configuration value. It is looking for a secret - maybe a database password, maybe an API key - via an environment variable mounted from a Kubernetes Secret.

But the Secret does not exist in this namespace.

kubectl logs myapp-7d9f8b-xkj2p

Error: Required environment variable DATABASE_PASSWORD is not set
Process exited with code 1

kubectl describe pod myapp-7d9f8b-xkj2p

Events:
  Warning  Failed    2m   kubelet  Error: secret "myapp-credentials" not found

The deployment referenced a Secret that was never created in the target namespace. It exists in staging. It does not exist in production. The deployment YAML was copy-pasted and nobody noticed.

How to debug it

# Step 1: Get the actual error
kubectl logs <pod-name>
kubectl logs <pod-name> --previous  # Logs from the crashed instance

# Step 2: Describe the pod for Kubernetes-level events
kubectl describe pod <pod-name>

# Step 3: Check if the secret exists
kubectl get secrets -n <namespace>

# Step 4: Verify the secret has the expected keys
kubectl describe secret myapp-credentials

The fix

Create the missing Secret in the correct namespace. For the longer-term fix, use a tool like helm diff, kubectl diff, or a GitOps pipeline that validates all referenced resources exist before allowing a deployment to proceed.

Key takeaway

CrashLoopBackOff means the pod keeps dying. kubectl logs --previous shows why it died. kubectl describe pod shows what Kubernetes tried and failed to do. Always check both.

4. The Node.js Memory Leak (WebSocket Edition)

The story

Your Node.js service is running fine after deployment. Memory usage is at 200MB, which is normal. Over the next 18 hours, you watch it creep up. 300MB. 400MB. 600MB. Then the process gets OOMKilled by the container runtime and restarts. The whole cycle starts again.

You check your code for obvious leaks - giant arrays, global caches growing unbounded. Nothing jumps out. This one hides.

Why it happens

The classic Node.js memory leak pattern that catches even experienced engineers: adding event listeners inside a function that gets called repeatedly, without removing them.

// This leaks memory on every new WebSocket connection
function setupWebSocket(socket) {
  // This listener is added fresh on every call
  // But the reference to process.on keeps the socket alive
  // even after the connection closes
  process.on('SIGTERM', () => {
    socket.close();
  });

  socket.on('message', handleMessage);
}

Every time a new WebSocket connection comes in, a new listener is added to process. When the connection closes, the listener is not removed. The listener holds a reference to the socket object. The socket object cannot be garbage collected. After thousands of connections, you have thousands of dead socket references sitting in memory.

Node.js will even warn you about this - but the warning often gets lost in log noise:

MaxListenersExceededWarning: Possible EventEmitter memory leak detected.
11 SIGTERM listeners added to [process]. Use emitter.setMaxListeners() to increase limit

That warning is not something to suppress. It is a canary telling you there is a leak.

How to debug it

# Get a heap snapshot from a running Node.js process
kill -USR2 <pid>

# Or via the Node.js inspector
node --inspect app.js
# Then open chrome://inspect and take a heap snapshot

Look for object types with counts growing over time. In this case you would see Socket instances accumulating far beyond the number of active connections.

The clinic.js tool is excellent for this:

npx clinic heapprofiler -- node app.js

The fix

Always clean up listeners when the associated resource goes away:

function setupWebSocket(socket) {
  const cleanup = () => socket.close();
  process.on('SIGTERM', cleanup);

  socket.on('close', () => {
    // Remove the listener when the connection closes
    process.removeListener('SIGTERM', cleanup);
  });
}

Key takeaway

Memory leaks in Node.js are almost always about retaining references longer than necessary. Event listeners are the most common culprit. Take MaxListenersExceededWarning seriously - it is not noise.

5. Cache Stampede After Redis Restart

The story

Your Redis cache went down for planned maintenance. You brought it back up. Simple, right?

Sixty seconds later your database server is on fire. CPU is pegged at 100%. Query latency went from 5ms to 8 seconds. The database is drowning.

What happened? Every single cache key expired at the same moment - because they all had the same TTL set from the last cache warming cycle - and every single application server tried to rebuild the cache simultaneously by hitting the database.

This is a cache stampede, also called a thundering herd.

Why it happens

Consider what happens when your cache is empty after a restart and you have 50 application servers:

Request comes in for /api/products
All 50 servers check the cache - cache miss
All 50 servers query the database for product data
All 50 servers write the result back to cache
49 of those database queries were wasted
Under high traffic, "50 servers" becomes "50,000 requests per second"

The database - which normally handles 200 queries per second because the cache absorbs the rest - suddenly receives 20,000 queries per second. It collapses.

How to debug it

The diagnosis is usually visible in the metrics:

Cache hit rate: dropped from 95% to 0%
Database connections: spiked from 50 to 800 in 30 seconds
Database CPU: 100%
API P99 latency: 50ms -> 12,000ms

Correlate the timeline with the Redis restart event. If the metrics cliff happened right when Redis came back up, you have your answer.

The fix

Several strategies exist, and production systems often use multiple in combination:

Cache locking (mutex pattern): Only one process populates a cache key. Others wait.

import redis
import time

def get_with_lock(key, fetch_fn, ttl=300):
    r = redis.Redis()
    value = r.get(key)
    if value:
        return value

    # Try to acquire a lock
    lock_key = f"lock:{key}"
    if r.set(lock_key, "1", nx=True, ex=10):
        # We got the lock - populate the cache
        try:
            value = fetch_fn()
            r.setex(key, ttl, value)
            return value
        finally:
            r.delete(lock_key)
    else:
        # Someone else has the lock - wait briefly and retry
        time.sleep(0.1)
        return r.get(key)

TTL jitter: Add random variance to cache expiration times so keys do not all expire simultaneously.

import random

base_ttl = 300
jitter = random.randint(-30, 30)
r.setex(key, base_ttl + jitter, value)

Probabilistic early expiration: Proactively refresh cache entries before they expire, based on how expensive the recomputation is.

Key takeaway

A cache restart is not a safe non-event. Treat cache warming as part of your maintenance procedure. Use TTL jitter by default - it costs nothing and prevents a whole class of stampede failures.

The Pattern Across All Five

Look at what these incidents have in common:

The symptoms lied. Disk full causing database errors. Connection pool causing "database" problems. A cache issue causing what looks like a database overload.
The actual cause was one layer removed from where the pain was visible.
All five are preventable with the right monitoring thresholds, code patterns, and configuration choices.
All five are faster to debug if you have seen them before.

That last point is the crux of it. Incident response speed is largely pattern recognition. The engineer who has seen a connection pool exhaustion before spots the idle database CPU and goes straight to pg_stat_activity. The one who has not seen it spends an hour tuning query indexes that are not the problem.

Practice Before the Pager Goes Off

If you want to build that pattern recognition without waiting for production to teach you the hard way, I built youbrokeprod.com - a free browser game where you investigate production outages step by step.

Each scenario drops you into a live incident: you run commands, read logs, check metrics, and work toward a diagnosis. No signup required to try it. The game currently has 10 scenarios across beginner, intermediate, and advanced difficulty - including all five incidents described in this post.

The goal is simple: make the muscle memory of incident debugging feel familiar before it is your on-call rotation on the line.

What production incidents have scarred you the most? Drop them in the comments - there are 44 scenarios in the backlog and the most painful real-world ones make the best levels.