Forem: Samson Tanimawo

Kubernetes Network Policies: Lessons from Production Incidents

Samson Tanimawo — Tue, 05 May 2026 14:33:42 +0000

Why Default Kubernetes Networking Is Wrong

Fresh Kubernetes cluster:

Every pod can talk to every other pod
Across namespaces, across services, across environments
No egress restrictions
No ingress restrictions

This is a lateral movement attack waiting to happen. One compromised pod = entire cluster.

Network Policies fix this. Most teams ignore them until the first security audit.

A Simple Rule That Breaks Things

Start with: "deny all traffic by default, explicitly allow what you need."

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: production
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress

Apply this to a running namespace, and everything breaks:

Pods can't reach DNS (kube-dns is in kube-system)
Pods can't reach the API server
Metrics scraping fails
Service mesh control plane loses connectivity

This is not a bug. This is the point. You have to explicitly allow everything.

The Allow-List Pattern

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: api-server
namespace: production
spec:
podSelector:
matchLabels:
app: api-server
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- protocol: TCP
port: 8080
egress:
# DNS
- to:
- namespaceSelector:
matchLabels:
name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
# Database
- to:
- podSelector:
matchLabels:
app: postgres
ports:
- protocol: TCP
port: 5432

Every allowed flow has to be declared. No exceptions.

Production Incidents We've Debugged

Incident 1: The invisible DNS failure

Symptoms: pods running but inter-service calls failing intermittently. Logs showed DNS resolution timeouts.

Root cause: a recently applied Network Policy forgot to allow UDP port 53 to kube-dns. About 50% of DNS queries were failing.

Fix: always include DNS egress in the base template.

egress:
- to:
- namespaceSelector:
matchLabels:
name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53 # Some clients fall back to TCP

Incident 2: Metrics gap after namespace migration

Symptoms: Prometheus stopped scraping metrics for services in the payments namespace after a security audit applied strict policies.

Root cause: the scraper pod in the monitoring namespace couldn't reach the metrics port on payments pods. Network Policy blocked it.

Fix:

ingress:
- from:
- namespaceSelector:
matchLabels:
name: monitoring
podSelector:
matchLabels:
app: prometheus
ports:
- protocol: TCP
port: 9090

Incident 3: External API calls blocked

Symptoms: a service that integrates with Stripe started returning 503s after a deploy.

Root cause: the new Network Policy allowed egress to internal services but didn't allow egress to external IPs. Stripe calls failed.

Fix:

egress:
# Allow to all external IPs on HTTPS
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8 # Block private ranges
- 172.16.0.0/12
- 192.168.0.0/16
ports:
- protocol: TCP
port: 443

Namespace-Level vs Pod-Level

Namespace-level policies are easier but coarse:

# Allow all pods in "api" namespace to reach "db" namespace
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
name: db

Pod-level policies are harder but more secure:

# Only the "orders" service can reach the "orders_db"
spec:
podSelector:
matchLabels:
app: orders
egress:
- to:
- podSelector:
matchLabels:
app: orders-db

Start namespace-level. Tighten to pod-level for sensitive services (payments, auth, PII stores).

The CNI Matters

Not all CNIs support all Network Policy features:

Cilium: full support, plus advanced L7 (HTTP method filtering)
Calico: full support for v1 spec
Weave Net: basic support
Flannel: NONE need to pair with Calico

If you're on Flannel without Calico, your Network Policies are being silently ignored. Check your CNI.

The Rollout Strategy

Don't apply strict policies to production on day one. You will cause an outage.

Phase 1 (week 1): Deploy in dev cluster only, break things, fix things
Phase 2 (week 2): Apply "log only" mode in staging (Cilium supports this)
Phase 3 (week 3-4): Apply in staging as enforced, watch for issues
Phase 4 (week 5+): Apply in production during a quiet window, have rollback ready

Total time: 4-6 weeks for a full rollout. Faster rollouts cause faster outages.

Testing Network Policies

Before deploying:

# Test from inside the cluster
kubectl run test-pod --image=busybox --rm -it -- sh
wget -qO- http://api-service:8080

# Verify denied flows fail
wget -qO- http://admin-service:8080 # Should timeout

CI idea: spin up a cluster, apply your policies, run a test suite that asserts which pods can reach which others. Fails on policy regressions.

Observability Into Policies

Cilium provides Hubble UI for real-time network flows:

See which pods are talking
See which flows are denied
Visualize policy coverage

Without something like Hubble, debugging Network Policy issues is archaeology. Invest in it.

The Minimum Viable Policy

If you're starting from zero, here's the minimum:

# 1. Default deny ingress across all namespaces
# 2. Allow DNS egress (kube-dns)
# 3. Allow same-namespace pod-to-pod
# 4. Explicit cross-namespace allows for specific services
# 5. Deny egress to private IP ranges (except approved)
# 6. Allow egress to 0.0.0.0/0 on 443 for external APIs

This covers 80% of the security benefit for 20% of the complexity. Tighten from there over time.

Common Mistakes

Skipping DNS everything breaks
Forgetting metrics scrape paths Prometheus goes blind
Not testing in staging first production outage on day one
Using a CNI that doesn't support policies silent failure
Applying policies before installing them chicken and egg
No rollback plan panic mode when things break

Network Policies are one of the highest-leverage security controls in Kubernetes. They're also one of the easiest ways to cause a self-inflicted outage. Treat them with respect.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Reducing Toil: The Google SRE Book Applied to Startups

Samson Tanimawo — Mon, 04 May 2026 14:28:17 +0000

The Google Rule That Breaks at Startups

Google's SRE book says: SRE time should be no more than 50% toil. The other 50% must go to engineering work that reduces toil.

At a 10-person startup, your "SRE team" is one overworked engineer. They're already at 95% toil. There is no slack to reduce it.

So you have to be ruthless about what work is worth automating and what work is worth eliminating entirely.

Defining Toil Precisely

Google's definition:

Manual
Repetitive
Automatable
Tactical (not strategic)
Lacks enduring value
Scales linearly with service growth

If a task checks all six boxes, it's toil. If it checks some but not all, it might be legitimate engineering work.

Example: responding to an alert is tactical and lacks enduring value, but if it's not repetitive, it's not toil.

Example: writing a new runbook is manual, but it's strategic and has enduring value, so it's not toil.

The Startup-Sized Toil Audit

Track for 2 weeks. Every 30 minutes, write down:

What am I doing right now?
Is it toil (manual, repetitive, automatable)?
How long have I been doing it this week?

At the end of 2 weeks, you'll have a toil ranking. Pick the top 3.

Typical top offenders:

Manually running deploys (30+ min/week)
Responding to known false-positive alerts (3+ hours/week)
Provisioning new dev environments (1+ hour per request)
Checking on flaky CI runs (2+ hours/week)
Writing the same runbook context in every incident (1+ hour/incident)

Rule 1: Eliminate Before Automating

The best toil is toil you don't do at all.

Before automating the deploy process, ask: why do we deploy manually?

If the answer is "we don't trust our tests" → fix the tests
If the answer is "we need human approval" → build a self-serve approval flow
If the answer is "production is scary" → build better rollback, then trust the automation

Before automating alert response, ask: why is the alert firing?

If it's a false positive → fix the alert
If it's a symptom of something deeper → fix the root cause
If it's expected behavior → delete the alert

Automating false positives is worse than doing them manually it hides the underlying problem.

Rule 2: Automate the Second-Most Common Task

Counter-intuitive but works:

The most common manual task is usually the one you've already optimized manually. You've gotten fast at it.

The second-most common task is where you're slow, it's still frequent, and automation has the highest ROI.

Example: you spend 3 hours/week on deploys (already optimized with scripts). You spend 2 hours/week manually provisioning dev environments (still done via the UI). Automate the environments first.

Rule 3: Measure Before and After

toil_metrics:
deploy_manual_time: 35 min/week
deploy_automated_time: 5 min/week # Saves 30 min/week

alert_response_time: 180 min/week
alert_response_time_after_tuning: 45 min/week # Saves 135 min/week

If you can't measure the savings, you don't know if the automation worked.

Rule: if automating a toil task takes 10x longer than the toil itself saves per year, don't automate. Eliminate.

Rule 4: Self-Service Is the Force Multiplier

At scale, toil scales linearly with team size. Self-service breaks the linear relationship.

Examples:

Instead of SRE provisioning dev environments → Terraform module + docs + CI approval
Instead of SRE running database queries → read-only proxy with query approval flow
Instead of SRE creating alerts → YAML templates engineers can copy

Rule of thumb: if three different engineers have asked you to do the same thing, build it as self-service.

Rule 5: Runbooks Are Temporary Debt

A runbook says "here's the manual procedure." A good runbook is instructions for a bot, not a human.

Every runbook should have an expiration date:

runbook: restart_stuck_worker
created: 2024-01-15
expires: 2024-04-15
automation_ticket: #4872

If a runbook is still manual after 3 months, either:

It's rare enough to not need automation
We've failed to allocate time for automation
The underlying issue should be fixed instead

Either way, reconsider.

The 80/20 Rule for Startup SRE

At Google scale, you can justify building a platform team. At startup scale, you can't. So you apply Pareto ruthlessly.

What to automate (20% effort, 80% value):

Deploys (frees 30+ min/week, prevents errors)
Dev environment provisioning (frees 2+ hrs/week)
Known-cause alert response (frees 3+ hrs/week)
Secret rotation
Backup verification

What not to automate (80% effort, 20% value):

Provisioning one-off infrastructure
Debugging novel issues
Writing custom dashboards
Responding to security incidents (needs judgment)

Save your engineering cycles for the high-leverage automation.

The Monthly Toil Review

Every month, ask:

What toil do I do now that I didn't do last month?
What toil did I eliminate in the last month?
Is my total toil going up or down?

If toil is going up and you're not seeing a plan to reduce it, that's your biggest reliability problem. Not the outages. Not the alerts. The toil.

Because toil crowds out the time you need to fix the underlying systems.

The Hardest Part

The hardest part of toil reduction isn't technical. It's psychological.

Toil feels productive. You finish tasks. You feel needed. You're the hero who fixed the broken thing.

Engineering work to eliminate toil feels slow. You build for weeks before seeing results. Nobody pages you for doing it.

Resist the dopamine of toil. The goal is to make yourself less needed, not more.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Incident Severity Levels: SEV-1 to SEV-5 Calibration

Samson Tanimawo — Sun, 03 May 2026 14:27:49 +0000

Why Severity Is Broken at Most Companies

Everyone has severity levels. Almost nobody agrees on what they mean.

Ask ten engineers what SEV-2 means and you'll get eight different answers. This causes:

Under-paged incidents (people thought SEV-3 meant "no rush")
Over-paged incidents (everything is SEV-1)
Exhausted on-call (false alarms)
Missed SLOs (incidents not escalated in time)

Calibration matters. Here's a definition that actually works.

The Five Levels

SEV-1: Critical

Primary product is completely down for all users
Active data loss
Security breach in progress
Core business stopped (can't process payments, can't log in)

Target response: 5 minutes
Escalation: Immediate, all hands
Post-mortem: Required, public within 5 days

SEV-2: High

Primary product is degraded for most users
Core feature unavailable for a subset
Significant customer impact but workaround exists
Performance significantly degraded (>5x normal latency)

Target response: 15 minutes
Escalation: Page primary on-call, notify secondary
Post-mortem: Required, internal within 5 days

SEV-3: Medium

Non-critical feature broken
Affects a small percentage of users
Degraded performance within tolerance
Bug in new feature rollout

Target response: 1 hour
Escalation: Page during business hours, ticket overnight
Post-mortem: Recommended

SEV-4: Low

Minor bug with workaround
Internal tooling broken
Non-customer-facing issue
Cosmetic problems

Target response: 1 business day
Escalation: Ticket only
Post-mortem: Not required

SEV-5: Informational

Not actually broken
Preemptive warning
"This might become a problem"
Observed anomaly without impact

Target response: Backlog
Escalation: None
Post-mortem: Not required

The Calibration Problem

Levels written on paper are useless. What matters is consistent application.

Run this exercise: take your last 50 incidents. Ask three SRE leads to independently assign severity levels. Compare.

If more than 20% disagree by at least one level, your definitions aren't calibrated. Run training.

The "When In Doubt" Rules

When severity is ambiguous, default to higher severity and downgrade if wrong.

Better to over-escalate and apologize than under-escalate and miss a SEV-1 for 4 hours.

Specific rules:

User data loss → always SEV-1 or SEV-2, never lower
Security issue → always SEV-1 or SEV-2
Revenue impact → SEV-2 minimum if measurable
Uncertain scope → start at higher severity, downgrade when scope is clear

Customer Impact Matrix

For fast calibration, use a matrix:

| <1% users | 1-10% users | 10-50% | >50% users
Product Down | SEV-2 | SEV-1 | SEV-1 | SEV-1
Major Degraded | SEV-3 | SEV-2 | SEV-2 | SEV-1
Minor Degraded | SEV-4 | SEV-3 | SEV-2 | SEV-2
Workaround Exists| SEV-4 | SEV-4 | SEV-3 | SEV-2

This gives you a fast severity assignment without relying on intuition.

Time-Based Escalation

Severity isn't fixed for the incident lifetime. It escalates:

sev_2:
auto_escalate_to_sev_1:
- if_not_resolved_in: 60_minutes
- if_user_impact_grows: above_10_percent
- if_revenue_loss_exceeds: $10000/hour

Start at SEV-2, auto-escalate if things worsen. Don't let an incident linger at the same severity if the impact is growing.

The Downgrade Rule

Downgrading is allowed but must be justified in writing in the incident channel.

"Downgrading from SEV-1 to SEV-2 at 10:23. Initial reports of
total outage were incorrect. Real impact is ~5% of users in
us-west-2 only. Ticket: INC-1234"

This prevents silent downgrades that understate severity for retro analysis.

SLO Integration

Your SLOs and severity levels should align:

SLO: 99.95% availability (21.6 min/month budget)

If this month's error budget burned:
<25% → normal operations
25-50% → no SEV-3 burn-down deploys
50-75% → SEV-2 threshold lowered
>75% → any degradation is SEV-1

When you're running low on error budget, everything gets more severe.

Practical Incident Categories

Beyond numeric severity, label incidents by type:

INCIDENT_TYPES:
- infrastructure (AWS, networking)
- application (code bug)
- deployment (bad release)
- capacity (scaling failure)
- data (corruption, loss)
- security (breach, exposure)
- external (3rd-party dependency)

Severity tells you how urgent. Type tells you who to page.

The Monthly Review

Once a month, review:

All SEV-1s and SEV-2s
Any SEV-3 that should have been SEV-2
Any SEV-2 that should have been SEV-3
Average time from incident open to correct severity assignment

Adjust the definitions based on what you learn. Severity is a living standard.

Common Mistakes

Pet severity every team invents their own. Standardize company-wide.
SEV-0 don't add levels above SEV-1. Just use "SEV-1 all hands."
Severity inflation if every incident is SEV-2, nobody takes SEV-2 seriously
Severity deflation pressure to avoid post-mortems leads to fake SEV-4s
Unchanging severity escalation is a tool, use it

The Goal

Severity should mean the same thing to every person in the org. Engineers, PMs, execs, customer support.

When someone says "SEV-1," everyone should know what that means, how urgent it is, and what the response looks like.

When you achieve that, incident response gets dramatically better.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Memory Leak Detection in Long-Running Services

Samson Tanimawo — Sat, 02 May 2026 14:27:34 +0000

The Slowest Incident to Diagnose

Memory leaks are sneaky. The service runs fine for hours. Then, slowly, it gets worse. Slower responses, more GC pauses, eventual OOM kills.

And when you look at the first 30 minutes of metrics, everything looks normal.

The Three Flavors of Memory Growth

1. True leaks

Objects allocated but never freed
Classic in C/C++, rare in Go/Java with GC
Grows linearly forever until OOM

2. Unbounded caches

Cache adds entries but never evicts
Common in Node.js, Python, Go
Grows until memory pressure triggers other issues

3. Memory fragmentation

Heap is large but not usable
Happens in long-running Java, Go,.NET services
Not really a "leak" but behaves like one

All three cause the same symptom: memory grows over time. Treatment is different for each.

Detection Without Heap Dumps

Before you reach for pprof or heap dumps, the fastest diagnosis is graph-watching:

# Is memory growing linearly over the last 24 hours?
deriv(container_memory_working_set_bytes{service="api"}[24h]) > 0

# Is GC pause time increasing?
rate(jvm_gc_pause_seconds_sum[1h]) > 0.05

If memory is growing by ~500MB/day and GC pauses are increasing, you have a leak. Diagnosis complete.

The question is where.

Go Memory Profiling

Go makes this relatively easy:

import _ "net/http/pprof"

// In main():
go func() {
http.ListenAndServe(":6060", nil)
}()

Then:

# Get a heap profile
go tool pprof http://localhost:6060/debug/pprof/heap

# In the pprof shell:
(pprof) top
(pprof) list suspiciousFunction
(pprof) web # generates a SVG callgraph

Look for:

Objects with high inuse_space
Objects with growing counts over time
Unexpected large maps or slices

Key trick: take two heap profiles 1 hour apart and diff them:

go tool pprof -base heap1.pprof heap2.pprof

What shows up as "new" allocations in the diff is almost certainly your leak.

Java Memory Profiling

Java is harder because the JVM adds layers:

# Dump the heap
jmap -dump:format=b,file=heap.hprof <pid>

# Analyze with Eclipse MAT or JVisualVM

In MAT, look for:

Leak Suspects report (automatic)
Dominator tree (what's holding the most memory)
GC roots path (what's preventing garbage collection)

Common Java culprits:

Static collections (especially static Map)
ThreadLocal values without cleanup
Listeners/callbacks registered but never unregistered
finalize() methods delaying collection

Node.js Memory Profiling

// Enable the inspector
node --inspect app.js

// Then in Chrome DevTools → Memory → Heap Snapshot
// Take 3 snapshots: baseline, after 10 min, after 20 min
// Compare to find retained objects

Common Node culprits:

Event emitter listeners that accumulate
Closures holding references to large objects
Unbounded caches (remember, Node has no built-in LRU)
Stream buffers not being drained

Python Memory Profiling

import tracemalloc
tracemalloc.start()

#... run the leaky operation...

snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
for stat in top_stats[:10]:
print(stat)

Or use memory_profiler:

from memory_profiler import profile

@profile
def suspect_function():
# code here

Common Python culprits:

Global lists/dicts growing unbounded
Reference cycles with __del__ methods
C extensions leaking (hardest to find)
Pandas DataFrames kept around too long

The Cache Leak Special Case

The most common "leak" isn't a leak at all. It's a cache without eviction.

# BAD: unbounded
cache = {}
def get_user(user_id):
if user_id not in cache:
cache[user_id] = fetch_from_db(user_id)
return cache[user_id]

# GOOD: bounded LRU
from functools import lru_cache

@lru_cache(maxsize=10000)
def get_user(user_id):
return fetch_from_db(user_id)

Always bound your caches. Always.

Fragmentation in Go

Go's garbage collector can leave the heap fragmented. You see:

Runtime memory is high
Heap profile shows low allocations
runtime.GC() doesn't reduce usage much

Solution: tune GOGC or force memory release:

import "runtime/debug"
debug.SetGCPercent(20) // More aggressive GC
debug.FreeOSMemory() // Return memory to OS

The Long-Running Service Pattern

Services that run for weeks without restart accumulate cruft. Even without leaks.

We use this pattern:

deployment_policy:
max_uptime: 7d
restart_schedule: "rolling restart every 7 days"

Every pod gets restarted weekly during a quiet window. Eliminates slow memory growth as a class of problem.

This isn't defeat. It's acknowledging that long-running processes in any language eventually accumulate state you don't want.

The Diagnostic Checklist

When a service is suspected of leaking:

Is memory growing linearly or logarithmically? (linear = real leak)
Is GC frequency/duration increasing? (yes = real pressure)
Are request rates growing proportionally? (yes = normal growth, not leak)
Take heap profile, save baseline
Wait 1 hour, take second profile, diff
Look for unexpected high-count objects
Trace back to allocation site
Fix the leak, deploy, watch metrics for 24h

Rinse and repeat. Memory leaks are annoying but systematically fixable.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

CI/CD Reliability: When Your Deploy Pipeline is Your SPOF

Samson Tanimawo — Fri, 01 May 2026 14:23:02 +0000

The Invisible SPOF

Every engineering org has a single point of failure that nobody lists on their risk registry: the deploy pipeline itself.

When CI/CD breaks, you can't ship features. You can't deploy hotfixes. You can't roll back a broken release. Your production doesn't go down, but your ability to fix production does.

We had a 4-hour outage last year caused by a GitHub Actions incident. Not a single server went down. We just couldn't deploy the fix.

Categorizing the Risk

Your pipeline consists of:

source_control: # GitHub, GitLab, Bitbucket
failure_mode: "can't merge PRs"

ci_runners: # GitHub Actions, CircleCI, self-hosted
failure_mode: "builds don't run"

artifact_storage: # ECR, Artifactory, S3
failure_mode: "images don't build or push"

deployment_controller: # ArgoCD, Flux, Spinnaker
failure_mode: "deploys don't happen"

cluster_api: # k8s API, cloud provider API
failure_mode: "resources don't change"

Each layer is a failure domain. A serious pipeline needs fallback plans for each.

The Manual Escape Hatch

Rule #1: You must have a documented path to deploy manually.

Not for daily use for emergencies. Every team should know:

How to build the image locally
How to push to the registry
How to update the cluster without the normal pipeline
Who has permission to do this in production

We test this quarterly. Every SRE must deploy one service manually, end-to-end, in under 10 minutes, without the pipeline.

Hardening the Pipeline Itself

1. Pin your dependencies

# BAD
uses: actions/checkout@main

# GOOD
uses: actions/checkout@v4.1.1

If actions/checkout@main breaks, your deploys break. Pin to versions.

2. Cache everything locally

registry:
primary: ghcr.io/yourorg
fallback: ecr.amazonaws.com/yourorg

When the primary registry is down, you need a mirror. Every production image should exist in at least two registries.

3. Monitor the pipeline

You probably monitor your services. Do you monitor your CI?

pipeline_metrics:
- build_success_rate (target: >99%)
- deploy_duration_p99 (target: <5 min)
- time_to_rollback_p99 (target: <2 min)
- runner_queue_depth (target: <5)

Alert on these the same way you'd alert on a service.

4. Test disaster modes

Can you ship if GitHub Actions is down?
Can you ship if the main registry is unreachable?
Can you ship if ArgoCD is down?

If the answer is "no", you have undocumented SPOFs.

The Rollback Rule

Every deploy must be reversible in under 2 minutes. No exceptions.

time_to_deploy: 15 minutes
time_to_rollback: 90 seconds

If your rollback takes longer than your deploy, your pipeline is backwards.

How to achieve fast rollbacks:

Keep the previous image running in parallel during deploys
Use traffic-shifting deploys (ALB weights, Istio)
Label every image with the git commit
Never deploy untested rollback paths

The Deploy Freeze

Some teams never deploy on Fridays. This is cargo culting.

The real rules:

Don't deploy when the on-call person is asleep
Don't deploy during peak traffic windows
Don't deploy major changes during holidays
DO deploy hotfixes anytime

If Friday at 5pm is the only time you can deploy a fix, you deploy. The alternative is customers suffering all weekend.

A reliable pipeline makes any-time deploys safe. Banning Friday deploys means your pipeline isn't reliable enough.

Multi-Provider Strategy

Big-ticket item: run critical workloads on CI from a different vendor than your code host.

Code: GitHub
CI: CircleCI (not GitHub Actions)

When GitHub Actions is down (it happens twice a year), your builds still run. When CircleCI is down, you can fall back to GitHub Actions.

This doubles your CI bill but removes a major SPOF.

The "Break Glass" Deploy

Every pipeline should have an emergency bypass:

# Normal deploy (takes 15 minutes, runs all tests)
./deploy.sh

# Break-glass deploy (skips tests, full audit log, Slack alert)
./deploy.sh --break-glass --reason "Fixing P1 incident #1234"

The break-glass path:

Requires written justification
Skips long-running tests
Notifies the whole team
Writes to a permanent audit log
Can only be used with incident in progress

Used maybe 3-5 times a year. Without it, your 2-hour deploy pipeline becomes a bottleneck when every minute matters.

The Metric That Matters Most

Mean Time to Deploy a Hotfix (MTTDHF)

From "we need to fix this" to "fix is in production" how long?

Good: under 30 minutes
Great: under 10 minutes
Unicorn: under 5 minutes

Track this. Optimize it. It's the most important reliability metric nobody talks about.

The Takeaway

Your pipeline is production infrastructure. Treat it with the same respect.

Monitor it
Back it up
Test failure modes
Document manual paths
Never let it become a SPOF

When it breaks during an incident, you'll be very glad you did.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Multi-Region Failover: Lessons from Running It Hot

Samson Tanimawo — Thu, 30 Apr 2026 14:47:08 +0000

Why "Hot" Matters

Three multi-region strategies:

Cold: Backup region is off. You start it when primary fails. RTO: hours.

Warm: Backup region runs on minimum capacity. Scale up on failover. RTO: 15-30 minutes.

Hot: Both regions serve live traffic simultaneously. RTO: seconds.

If you need under 15 minutes RTO, you need hot. Everything else is marketing copy.

The Illusion of Warm Failover

Warm sounds great on paper. In practice, on the day you need it:

The warm region has never seen real load
DNS cache propagation takes 5-15 minutes
Autoscaling lags because it's starting cold
Your team has never run on the warm region
Half your connection strings are hardcoded to the primary

Warm failover works in tabletop exercises. It does not work during real incidents under pressure. We learned this the hard way.

Running It Hot: The Architecture

┌─────────────┐
│ DNS / CDN │
└──────┬──────┘
│
┌──────────┴──────────┐
▼ ▼
┌──────────┐ ┌──────────┐
│ Region A │ │ Region B │
│ 50% TX │ │ 50% TX │
└────┬─────┘ └─────┬────┘
│ │
└──────┬──────┬────────┘
▼ ▼
┌─────────────┐
│ Shared DB │
│ (writer + │
│ replicas) │
└─────────────┘

Both regions always serve traffic. Split is usually 50/50 but can shift.

The Hard Parts

1. Database replication

This is where multi-region gets hard. Three options:

Single writer, multi-region readers: simplest, but writes pay cross-region latency
Multi-master: complex, but truly hot requires conflict resolution
Region-sharded: users pinned to a region for writes, simplest if your data model allows it

We use region-sharded for user-scoped data and single-writer for global config.

2. Session stickiness

If a user's session is in Region A, and their next request goes to Region B, things break.

Solutions:

JWT tokens with no server state
Session data in a globally replicated store (DynamoDB Global Tables, CockroachDB)
Cookie routing that pins a user to a region

3. Cache coherence

Region A's cache doesn't know when Region B updates the database. Options:

Short TTLs (1-5 minutes) and accept the inconsistency
Pub/sub cache invalidation across regions (complex)
Read-through caches only, never write-through

The Failover Mechanics

When Region A dies:

Health checks detect failure route53/ALB removes Region A from DNS
Traffic shifts to Region B already warm, already running
Autoscaling kicks in Region B doubles capacity
User sessions degrade gracefully re-authentication, cache warmup
Monitoring reports the shift team gets paged, not customers

Target: customer-facing latency spike of under 30 seconds, full recovery under 5 minutes.

Testing It Monthly

If you don't test failover monthly, you don't have failover. You have hope.

We do this:

First Tuesday of every month, 10 AM
Route100% of traffic to Region B for 30 minutes
Watch dashboards, fix anything that degrades
Route back to 50/50
Document any issues, fix them
Repeat next month with the other region

Cost Reality Check

Running hot doubles your compute cost. For most companies, that's $10K-$100K/month.

The question is: what's your revenue per hour of downtime?

Company A: $100K/hr revenue → 99.99% target → hot is worth it
Company B: $1K/hr revenue → 99.9% target → warm is fine

Do the math. Don't copy FAANG patterns if your revenue doesn't justify them.

The Operational Complexity Tax

Running hot costs more than money. It costs:

More runbooks (one per region)
More monitoring (cross-region latency, replication lag)
Harder debugging ("which region was this request in?")
More compliance surface (data residency, each region)
More deployment pipelines (usually)

Budget 20% more engineering time for multi-region from day one.

Common Mistakes

Single point of failure in DNS config your DNS provider becomes the new SPOF
Testing only with healthy traffic test with 2x normal load during drills
Forgetting about databases DB failover is the hardest part
Using regions as backup, not active never tested until crisis
Not planning for split-brain what if both regions think they're primary?

The Minimum Viable Hot Setup

Two regions, stateless app tier, 50/50 traffic
Database: multi-AZ primary, cross-region async replica
CDN/DNS: health-check-based routing
Session: JWT-based (stateless)
Monthly failover drills
Runbooks tested in last 90 days

Start there. Layer in complexity as you need it.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Multi-Region Failover: Lessons from Running It Hot

Samson Tanimawo — Thu, 30 Apr 2026 14:08:47 +0000

Why "Hot" Matters

Three multi-region strategies:

Cold: Backup region is off. You start it when primary fails. RTO: hours.

Warm: Backup region runs on minimum capacity. Scale up on failover. RTO: 15-30 minutes.

Hot: Both regions serve live traffic simultaneously. RTO: seconds.

If you need under 15 minutes RTO, you need hot. Everything else is marketing copy.

The Illusion of Warm Failover

Warm sounds great on paper. In practice, on the day you need it:

The warm region has never seen real load
DNS cache propagation takes 5-15 minutes
Autoscaling lags because it's starting cold
Your team has never run on the warm region
Half your connection strings are hardcoded to the primary

Warm failover works in tabletop exercises. It does not work during real incidents under pressure. We learned this the hard way.

Running It Hot: The Architecture

┌─────────────┐
│ DNS / CDN │
└──────┬──────┘
│
┌──────────┴──────────┐
▼ ▼
┌──────────┐ ┌──────────┐
│ Region A │ │ Region B │
│ 50% TX │ │ 50% TX │
└────┬─────┘ └─────┬────┘
│ │
└──────┬──────┬────────┘
▼ ▼
┌─────────────┐
│ Shared DB │
│ (writer + │
│ replicas) │
└─────────────┘

Both regions always serve traffic. Split is usually 50/50 but can shift.

The Hard Parts

1. Database replication

This is where multi-region gets hard. Three options:

Single writer, multi-region readers: simplest, but writes pay cross-region latency
Multi-master: complex, but truly hot requires conflict resolution
Region-sharded: users pinned to a region for writes, simplest if your data model allows it

We use region-sharded for user-scoped data and single-writer for global config.

2. Session stickiness

If a user's session is in Region A, and their next request goes to Region B, things break.

Solutions:

JWT tokens with no server state
Session data in a globally replicated store (DynamoDB Global Tables, CockroachDB)
Cookie routing that pins a user to a region

3. Cache coherence

Region A's cache doesn't know when Region B updates the database. Options:

Short TTLs (1-5 minutes) and accept the inconsistency
Pub/sub cache invalidation across regions (complex)
Read-through caches only, never write-through

The Failover Mechanics

When Region A dies:

Health checks detect failure route53/ALB removes Region A from DNS
Traffic shifts to Region B already warm, already running
Autoscaling kicks in Region B doubles capacity
User sessions degrade gracefully re-authentication, cache warmup
Monitoring reports the shift team gets paged, not customers

Target: customer-facing latency spike of under 30 seconds, full recovery under 5 minutes.

Testing It Monthly

If you don't test failover monthly, you don't have failover. You have hope.

We do this:

First Tuesday of every month, 10 AM
Route100% of traffic to Region B for 30 minutes
Watch dashboards, fix anything that degrades
Route back to 50/50
Document any issues, fix them
Repeat next month with the other region

Cost Reality Check

Running hot doubles your compute cost. For most companies, that's $10K-$100K/month.

The question is: what's your revenue per hour of downtime?

Company A: $100K/hr revenue → 99.99% target → hot is worth it
Company B: $1K/hr revenue → 99.9% target → warm is fine

Do the math. Don't copy FAANG patterns if your revenue doesn't justify them.

The Operational Complexity Tax

Running hot costs more than money. It costs:

More runbooks (one per region)
More monitoring (cross-region latency, replication lag)
Harder debugging ("which region was this request in?")
More compliance surface (data residency, each region)
More deployment pipelines (usually)

Budget 20% more engineering time for multi-region from day one.

Common Mistakes

Single point of failure in DNS config your DNS provider becomes the new SPOF
Testing only with healthy traffic test with 2x normal load during drills
Forgetting about databases DB failover is the hardest part
Using regions as backup, not active never tested until crisis
Not planning for split-brain what if both regions think they're primary?

The Minimum Viable Hot Setup

Two regions, stateless app tier, 50/50 traffic
Database: multi-AZ primary, cross-region async replica
CDN/DNS: health-check-based routing
Session: JWT-based (stateless)
Monthly failover drills
Runbooks tested in last 90 days

Start there. Layer in complexity as you need it.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Disaster Recovery Drills That Actually Work

Samson Tanimawo — Wed, 29 Apr 2026 15:45:56 +0000

Most DR Drills Are Theater

Someone schedules a meeting. A few senior engineers walk through a runbook. Everyone agrees "yes, we could do this" and marks it complete.

Then the real disaster hits and nobody remembers the procedure, the runbook is 2 years out of date, and half the backup systems don't work.

Real DR drills test whether your team can actually recover, not whether they can talk about recovery.

The Three Levels of DR Testing

Level 1: Tabletop

Walk through a scenario on paper
Identify missing runbooks
Find ownership gaps
Useful for: New team members, initial gap analysis
Limits: Doesn't prove anything actually works

Level 2: Partial Failure Test

Actually fail one component in staging
Watch recovery happen with real tools
Time the full recovery
Useful for: Validating specific runbooks
Limits: Staging ≠ production

Level 3: Full Production Drill

Actually fail a real production component
Customer-facing (announce a maintenance window if needed)
Full team responds as if it's real
Useful for: Proving you can recover
Limits: Scary, high-coordination

Most teams stop at Level 1. Good teams do Level 2 quarterly. The best teams do Level 3 twice a year.

A Real DR Drill Scenario

Scenario: Primary database becomes unreachable

Setup (48 hours before):

Schedule window with product team
Pick a time when customer impact is minimal
Brief the team: "Something will fail tomorrow, respond as normal"
Pre-position the incident commander

Execution:

At T+0, block network access to primary database via iptables
Start stopwatch
Watch the team respond
Do NOT intervene or give hints
Document every action, every decision, every delay

Metrics to capture:

Time to detection (first alert fire)
Time to engagement (first human acknowledges)
Time to diagnosis ("we know what's wrong")
Time to mitigation (customer impact stops)
Time to recovery (fully restored)

Scoring the Drill

Good scores:

Detection: under 2 minutes
Engagement: under 5 minutes
Diagnosis: under 15 minutes
Mitigation: under 30 minutes (for DR scenarios)
Recovery: depends on scenario

If any of these are 5x longer than target, you have a real problem.

What Always Goes Wrong

In every DR drill we run:

Runbook is out of date. The one that worked 6 months ago has wrong commands now.
Credentials don't work. The service account was rotated, nobody updated the runbook.
Backup is untested. The restore fails because the backup is corrupted.
Escalation paths are stale. The "DBA on-call" has left the company.
Dependencies are missing. The recovery playbook assumes Service X is up, but Service X depends on the failed component.

Every drill uncovers 3-5 of these. Fix them, then drill again.

Chaos Engineering vs. DR Drills

These are different. Chaos engineering is continuous (daily/weekly) and usually automated. DR drills are intentional and large-scale.

Chaos engineering answers: "Can we survive small failures routinely?"

DR drills answer: "Can we survive catastrophic failures at all?"

You need both.

The Blame-Free Rule

DR drills expose weaknesses. Those weaknesses are process problems, not people problems.

The rules:

No firing based on drill performance
No promotions based on "being the hero"
Focus on process gaps, not individual failures
The post-drill retrospective is 90% about fixing systems, 10% about training people

If the team is afraid of the drill, you'll never learn anything real.

Frequency That Actually Works

Level 1 (Tabletop): Monthly, 1 hour
Level 2 (Partial): Quarterly, 4 hours including retro
Level 3 (Full Production): Twice a year, full day

Also: after every major infrastructure change, drill the affected components.

The Hardest Lesson

The drill is the easy part. The hard part is making the fixes from the drill a priority when there's feature pressure.

We track "DR drill remediation items" as a standing OKR. If after two quarters the same items are still open, the SRE team has authority to freeze feature work until they're fixed.

Starting Point

If you've never done a DR drill:

Pick one scenario (database failure, region outage, API gateway down)
Schedule it for a quiet hour
Run a tabletop first find the obvious gaps
Fix those gaps
Run a partial failure test in staging
Measure everything
Run a retro focused on process
Schedule the next one

Do this for three scenarios, and you'll have a DR program. Do it for ten, and you'll have a resilient company.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Disaster Recovery Drills That Actually Work

Samson Tanimawo — Wed, 29 Apr 2026 14:41:48 +0000

Most DR Drills Are Theater

Someone schedules a meeting. A few senior engineers walk through a runbook. Everyone agrees "yes, we could do this" and marks it complete.

Then the real disaster hits and nobody remembers the procedure, the runbook is 2 years out of date, and half the backup systems don't work.

Real DR drills test whether your team can actually recover, not whether they can talk about recovery.

The Three Levels of DR Testing

Level 1: Tabletop

Walk through a scenario on paper
Identify missing runbooks
Find ownership gaps
Useful for: New team members, initial gap analysis
Limits: Doesn't prove anything actually works

Level 2: Partial Failure Test

Actually fail one component in staging
Watch recovery happen with real tools
Time the full recovery
Useful for: Validating specific runbooks
Limits: Staging ≠ production

Level 3: Full Production Drill

Actually fail a real production component
Customer-facing (announce a maintenance window if needed)
Full team responds as if it's real
Useful for: Proving you can recover
Limits: Scary, high-coordination

Most teams stop at Level 1. Good teams do Level 2 quarterly. The best teams do Level 3 twice a year.

A Real DR Drill Scenario

Scenario: Primary database becomes unreachable

Setup (48 hours before):

Schedule window with product team
Pick a time when customer impact is minimal
Brief the team: "Something will fail tomorrow, respond as normal"
Pre-position the incident commander

Execution:

At T+0, block network access to primary database via iptables
Start stopwatch
Watch the team respond
Do NOT intervene or give hints
Document every action, every decision, every delay

Metrics to capture:

Time to detection (first alert fire)
Time to engagement (first human acknowledges)
Time to diagnosis ("we know what's wrong")
Time to mitigation (customer impact stops)
Time to recovery (fully restored)

Scoring the Drill

Good scores:

Detection: under 2 minutes
Engagement: under 5 minutes
Diagnosis: under 15 minutes
Mitigation: under 30 minutes (for DR scenarios)
Recovery: depends on scenario

If any of these are 5x longer than target, you have a real problem.

What Always Goes Wrong

In every DR drill we run:

Runbook is out of date. The one that worked 6 months ago has wrong commands now.
Credentials don't work. The service account was rotated, nobody updated the runbook.
Backup is untested. The restore fails because the backup is corrupted.
Escalation paths are stale. The "DBA on-call" has left the company.
Dependencies are missing. The recovery playbook assumes Service X is up, but Service X depends on the failed component.

Every drill uncovers 3-5 of these. Fix them, then drill again.

Chaos Engineering vs. DR Drills

These are different. Chaos engineering is continuous (daily/weekly) and usually automated. DR drills are intentional and large-scale.

Chaos engineering answers: "Can we survive small failures routinely?"

DR drills answer: "Can we survive catastrophic failures at all?"

You need both.

The Blame-Free Rule

DR drills expose weaknesses. Those weaknesses are process problems, not people problems.

The rules:

No firing based on drill performance
No promotions based on "being the hero"
Focus on process gaps, not individual failures
The post-drill retrospective is 90% about fixing systems, 10% about training people

If the team is afraid of the drill, you'll never learn anything real.

Frequency That Actually Works

Level 1 (Tabletop): Monthly, 1 hour
Level 2 (Partial): Quarterly, 4 hours including retro
Level 3 (Full Production): Twice a year, full day

Also: after every major infrastructure change, drill the affected components.

The Hardest Lesson

The drill is the easy part. The hard part is making the fixes from the drill a priority when there's feature pressure.

We track "DR drill remediation items" as a standing OKR. If after two quarters the same items are still open, the SRE team has authority to freeze feature work until they're fixed.

Starting Point

If you've never done a DR drill:

Pick one scenario (database failure, region outage, API gateway down)
Schedule it for a quiet hour
Run a tabletop first find the obvious gaps
Fix those gaps
Run a partial failure test in staging
Measure everything
Run a retro focused on process
Schedule the next one

Do this for three scenarios, and you'll have a DR program. Do it for ten, and you'll have a resilient company.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Feature Flags as a Reliability Tool, Not Just an A/B Platform

Samson Tanimawo — Tue, 28 Apr 2026 14:11:47 +0000

Most Teams Use Feature Flags Wrong

They wire up LaunchDarkly or Unleash, use it for two A/B tests, then forget about it.

Meanwhile, their production is full of if (isNewCheckoutEnabled) blocks that nobody remembers how to toggle.

Feature flags are not primarily an experimentation tool. They're a reliability tool.

The Real Value

Feature flags let you separate deploy from release. You ship code to production cold, then turn it on gradually for real users.

When things break, you flip the switch back in 10 seconds. No rollback, no redeploy, no PR reverts.

The Four Reliability Patterns

1. Kill Switches

Every risky new feature ships behind a kill switch:

if (featureFlags.isEnabled('new_payment_flow', userId)) {
return newPaymentFlow();
}
return legacyPaymentFlow();

When the new flow has a bug, you don't rollback. You flip the flag.

2. Gradual Rollouts

new_search_algorithm:
rollout_percentage: 1 # Start at 1% of users
rules:
- if: "user.tier == 'internal'"
enabled: true # Internal users always see it

Deploy to 1%, watch metrics, go to 5%, watch, 25%, 50%, 100%. Takes 2-4 hours per rollout instead of a single risky deploy.

3. Circuit Breakers

external_recommendations_service:
enabled: true
automatic_disable_if:
error_rate_above: 5%
for_minutes: 5

If a downstream service starts failing, the flag auto-disables that feature. Your product degrades gracefully instead of crashing.

4. Load Shedding

expensive_realtime_dashboard:
enabled_when:
cpu_utilization_below: 70%
active_users_below: 50000

Under load, disable non-critical features to preserve the critical path.

The Anti-Pattern: Permanent Flags

After a feature is 100% rolled out, the flag should be deleted within 2 weeks. Every flag left in the codebase is technical debt.

Flag hygiene rules:

- Every flag has an expiration date (90 days max)
- Every flag has an owner in CODEOWNERS
- CI fails if a flag is older than 180 days
- Monthly flag cleanup is part of standard operations

We track "flag count" as a reliability metric. If it grows unbounded, we're doing it wrong.

The Architecture

A solid feature flag system has three parts:

1. Definition store

Source of truth for all flags
Versioned in Git or a managed service (LaunchDarkly, Unleash, GrowthBook)
Audit log for every change

2. Client SDK

In-app flag evaluation
Falls back to defaults if the service is unreachable
Caches decisions for 60 seconds
Emits telemetry for flag usage

3. Admin interface

Change flags without deploying code
See current state across environments
Role-based access (not everyone can flip prod flags)
Approval workflow for high-risk flags

Evaluating at the Right Layer

Flags can live at multiple layers:

CDN edge use for marketing experiments
Load balancer use for blue/green deploys
App server use for feature experiments
Database use for schema migrations

The deeper the layer, the faster the rollout. CDN flags flip in seconds. Database flags take minutes to propagate.

The Reliability Metric

Track: mean time to mitigate (MTTM).

If your team can mitigate an incident in under 30 seconds via a feature flag flip, that's a win. If you have to redeploy to mitigate, your reliability is bottlenecked by deploy time.

Good teams: MTTM under 60 seconds
Great teams: MTTM under 15 seconds

Common Gotchas

Stale flags skew A/B results clean them up after experiments
Flags without defaults cause prod outages every flag must have a safe fallback
Flag flips mid-request cause weird bugs evaluate at request start, cache for the request lifetime
Nested flags (flags inside flags) are impossible to reason about avoid

A Reliability-First Flag Strategy

Start simple:

Every new feature ships behind a kill switch
Gradual rollouts for anything touching the critical path
Circuit breakers for external dependencies
Flag cleanup is a monthly ritual
Track MTTM and optimize it

Feature flags are the most underrated reliability tool in modern engineering. Treat them that way.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

eBPF for SREs: Observability Without Agents

Samson Tanimawo — Mon, 27 Apr 2026 14:11:20 +0000

The Agent Problem

Traditional monitoring means shipping an agent with every service. That agent:

Adds memory overhead
Needs to be updated
Gets out of date
Breaks with kernel upgrades
Needs instrumentation code

eBPF says: what if the kernel itself could emit observability data?

What eBPF Actually Is

eBPF (extended Berkeley Packet Filter) lets you run sandboxed programs inside the Linux kernel without recompiling or loading modules. It was originally for packet filtering. Now it powers Cilium, Pixie, Falco, and dozens of other tools.

From an SRE perspective: you get deep visibility into syscalls, network traffic, process behavior, and filesystem operations with zero code changes to your applications.

What You Can Observe

network:
- every TCP connection (src, dst, bytes, duration)
- DNS queries and response times
- TLS handshake failures
- HTTP request/response cycles

application:
- function call latencies (uprobes)
- memory allocations
- lock contention
- GC pauses

security:
- syscall audit trails
- privilege escalations
- suspicious file access
- container escape attempts

performance:
- CPU scheduling delays
- I/O wait time per process
- disk latency histograms
- page fault patterns

All of this without modifying your application code.

A Practical Example: Detecting Slow HTTP Requests

Traditional approach: instrument your HTTP framework with OpenTelemetry, deploy a collector, ship traces.

eBPF approach:

# Install bpftrace
sudo apt install bpftrace

# Trace every HTTP response larger than 1MB
sudo bpftrace -e '
uprobe:/usr/lib/libssl.so:SSL_write {
@http_writes[pid] = count();
@http_bytes[comm] = sum(arg2);
}
'

No code changes. No restarts. Real-time visibility.

Tools Worth Knowing

1. Pixie (now part of New Relic)

Auto-instruments every service in your K8s cluster
No code changes, no sidecars
Full HTTP, MySQL, Postgres, DNS tracing
Open source

2. Cilium

Network observability + security policy enforcement
Replaces kube-proxy
Hubble UI for service-to-service traffic visualization

3. Falco

Runtime security detection
"Alert if a process inside a container spawns a shell"
Writes rules in YAML

4. Parca

Continuous profiling via eBPF
See CPU flame graphs across your entire fleet
Identify the most expensive code paths

5. Tracee

Security-focused eBPF tracing
Detects privilege escalations, cryptojacking, suspicious syscalls

The Tradeoffs

Pros:

Zero app code changes
Near-zero overhead (kernel-level efficiency)
Unified view across languages (Go, Python, Java, Rust, all seen the same way)
No agent lifecycle to manage

Cons:

Requires Linux 4.14+ (5.0+ preferred)
Steep learning curve for custom probes
Limited visibility into in-process logic (you see syscalls, not business logic)
eBPF verifier rejects programs for subtle reasons

When eBPF Shines

Network debugging: "Why is service A slow to reach service B?"
Security auditing: "What containers are making unexpected syscalls?"
Performance profiling: "Where is the cluster CPU time actually going?"
Incident forensics: "Reconstruct the syscall timeline during the outage"

When eBPF Is Wrong

Business logic observability you still need OpenTelemetry for spans
Application errors your logs and exception tracking still matter
Multi-region correlation eBPF is node-local

Use eBPF for infrastructure and network. Use OpenTelemetry for application logic. They complement each other.

Getting Started

Deploy Pixie in a dev cluster (1-line install)
Open the UI, watch real-time HTTP traffic
Try a bpftrace one-liner to trace a specific syscall
Read the Cilium + Hubble docs
Replace one agent-based tool with its eBPF equivalent

The future of observability is kernel-native. Agent-based tools will still exist, but the gap will keep shrinking.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Observability as Code: Managing Dashboards and Alerts with Terraform

Samson Tanimawo — Sun, 26 Apr 2026 14:11:06 +0000

The Problem with Click-Ops Dashboards

Your team has 200 dashboards. You don't know who owns them. Half are broken. The rest show yesterday's reality.

This is click-ops debt, and it compounds faster than code debt.

Observability as Code

Every dashboard, alert, and SLO definition should live in a Git repository alongside your service code.

resource "datadog_dashboard" "api_gateway" {
title = "API Gateway - Golden Signals"
description = "Owner: @platform-team"
layout_type = "ordered"

widget {
timeseries_definition {
title = "Request Rate (per second)"
request {
q = "sum:api.requests{service:gateway}.as_rate()"
}
}
}

widget {
timeseries_definition {
title = "P99 Latency"
request {
q = "max:api.latency{service:gateway}.as_count()"
}
}
}
}

This lives next to main.tf for your service. When you deploy the service, you deploy the observability.

Benefits That Compound

1. Ownership is clear. The file has a CODEOWNERS entry. PRs require review.

2. Dashboards auto-update. Renaming a service? Terraform refactor propagates to all dashboards.

3. Drift detection. Someone clicked "save as" in the UI and now that dashboard is out of sync. terraform plan catches it.

4. Review before production. Alert changes go through PR review. No more "who set this threshold?"

Tooling by Platform

datadog:
provider: DataDog/datadog
resources: datadog_monitor, datadog_dashboard, datadog_slo

grafana:
provider: grafana/grafana
resources: grafana_dashboard, grafana_alert_rule

prometheus:
approach: YAML files in Git, deployed by ArgoCD
resources: alert rules, recording rules

new_relic:
provider: newrelic/newrelic
resources: newrelic_alert_policy, newrelic_dashboard

Pick one source of truth. Don't mix.

A Real Example

We have a module that takes a service name and generates a complete observability stack:

module "service_observability" {
source = "./modules/observability"

service_name = "payment-processor"
team_slack = "#payments"
severity_map = {
error_rate_pct = 1.0
p99_latency_ms = 500
saturation_pct = 80
}

slo_targets = {
availability = 0.9995
latency_p99 = 0.99
}
}

One module call creates: 3 dashboards, 8 alerts, 2 SLOs, a Slack channel binding, and a PagerDuty escalation policy.

The Hardest Part

The code is easy. The hard part is:

Migrating existing click-ops dashboards budget 2 weeks
Getting engineers to edit YAML/HCL instead of the UI budget 3 months of reminders
Blocking UI edits some tools let you set dashboards to read-only
Reviewing alert changes PR reviewers need context

The Anti-Pattern to Avoid

Don't write Terraform for every custom chart an engineer wants. That leads to 500-line dashboard modules nobody understands.

Instead, define standard dashboards (golden signals, RED/USE, SLO burn rate) as modules. Let engineers add their own custom dashboards in the UI if they want, but mark them as "explore-only" (not alert-worthy).

Core observability = code. Experimental exploration = UI.

Migration Strategy

Week 1: Pick 1 service, convert its dashboards to Terraform
Week 2: Add alerts + SLOs to Terraform
Week 3: Delete the UI versions
Week 4: Create a module from the patterns
Month 2: Roll out to 10 more services
Month 3: Require all new services to use the module

Six months in, your click-ops debt is gone and your observability is reproducible.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com