Our Production System Went Down at 2:13AM — Here’s Exactly What Happened

arnoldbaraka — Fri, 20 Mar 2026 16:12:25 +0000

At 2:13AM, production went down.

No warning. No gradual degradation. Just alerts firing everywhere.

CPU was fine. Memory was fine. Nodes were healthy.

But users?
Nothing was working.

—

We traced it to Kubernetes.

Pods were restarting.
CrashLoopBackOff.

But logs?
Almost useless.

No clear error. Just silence… and restarts.

—

After digging deeper, we found it:

An image pull issue.

The cluster couldn’t pull from ECR.

Not because the image didn’t exist.
Not because of network.

But because of authentication.

Expired credentials.

—

Here’s what made it worse:

• CI/CD pipeline was green

• Deployment succeeded

• No alerts for registry auth failures

• Monitoring didn’t catch it early

Everything looked healthy.

It wasn’t.

—

What this incident taught me:

—

Fix:

• Implemented registry auth monitoring

• Added image pull failure alerts

• Rotated credentials with automation

• Improved logging visibility

—

DevOps isn’t about tools.

It’s about understanding failure.

And failure doesn’t announce itself.

It hides.

—

I’ll be sharing more real production incidents like this.

No theory. No fluff.

Just what actually happens in the trenches.

Forem: arnoldbaraka