Forem: Samson Tanimawo

The Golden Signals: A Practical Implementation Guide

Samson Tanimawo — Thu, 16 Apr 2026 08:12:08 +0000

Four Metrics to Rule Them All

Google's SRE book introduced the four golden signals: Latency, Traffic, Errors, and Saturation. Simple concept, but I've seen teams struggle with implementation.

Here's a practical guide from someone who's implemented them across 50+ services.

Signal 1: Latency

Not all latency is equal. You need to track successful requests and error requests separately.

# Bad: Average latency
latency = total_request_time / total_requests  # Useless

# Good: Percentile latency, separated by status
from prometheus_client import Histogram

REQUEST_LATENCY = Histogram(
    'http_request_duration_seconds',
    'Request latency',
    ['method', 'endpoint', 'status_class'],
    buckets=[.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]
)

@app.middleware
async def track_latency(request, call_next):
    start = time.time()
    response = await call_next(request)
    duration = time.time() - start
    status_class = f"{response.status_code // 100}xx"
    REQUEST_LATENCY.labels(
        method=request.method,
        endpoint=request.url.path,
        status_class=status_class
    ).observe(duration)
    return response

Alert on p99, not p50. Your happiest users don't need help.

- alert: HighLatencyP99
  expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 0.5
  for: 5m
  labels:
    severity: warning

Signal 2: Traffic

Traffic tells you "is this normal?" It's the context for every other signal.

# Current request rate
rate(http_requests_total[5m])

# Compare to same time last week
rate(http_requests_total[5m]) 
  / 
rate(http_requests_total[5m] offset 7d)

# Alert on sudden drops (possible outage nobody noticed)
- alert: TrafficDrop
  expr: >
    rate(http_requests_total[5m]) 
    < 
    (rate(http_requests_total[5m] offset 1h) * 0.5)
  for: 10m
  annotations:
    summary: "Traffic dropped >50% compared to 1 hour ago"

Traffic drops are often more concerning than traffic spikes.

Signal 3: Errors

Track error rate as a percentage, not absolute count:

# Error rate percentage
(
  sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
) * 100

But also track error types separately:

error_categories:
  - 5xx: "Server errors (our fault)"
  - 4xx_excluding_404: "Client errors (possible API issue)"
  - timeout: "Request timeouts"
  - circuit_breaker: "Dependency failures"

Signal 4: Saturation

The most underrated signal. Saturation answers: "how close are we to full?"

# CPU saturation
process_cpu_seconds_total / container_spec_cpu_quota

# Memory saturation
container_memory_working_set_bytes / container_spec_memory_limit_bytes

# Connection pool saturation
active_connections / max_connections

# Queue saturation (the one everyone forgets)
message_queue_depth / message_queue_capacity

Alert before you hit 100%. I use 80% as the threshold for warning and 95% for critical.

Putting It All Together

Every service gets a standard dashboard with four rows:

Row 1: Latency   [p50] [p90] [p99] [error latency]
Row 2: Traffic    [rate] [vs last week] [by endpoint]
Row 3: Errors     [rate %] [by type] [by endpoint]
Row 4: Saturation [CPU] [Memory] [Connections] [Queue]

This fits on one screen. No scrolling. Any engineer can assess service health in 10 seconds.

The Anti-Pattern

Don't build a golden signals dashboard per service manually. Template it:

{
  "dashboard": {
    "title": "Golden Signals: {{ service_name }}",
    "templating": {
      "list": [
        { "name": "service", "type": "query" },
        { "name": "environment", "type": "custom", "options": ["prod", "staging"] }
      ]
    }
  }
}

One template, 50 dashboards. Update once, apply everywhere.

If you want golden signal monitoring that sets itself up automatically, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

The Golden Signals: A Practical Implementation Guide

Samson Tanimawo — Thu, 16 Apr 2026 07:30:02 +0000

Four Metrics to Rule Them All

Google's SRE book introduced the four golden signals: Latency, Traffic, Errors, and Saturation. Simple concept, but I've seen teams struggle with implementation.

Here's a practical guide from someone who's implemented them across 50+ services.

Signal 1: Latency

Not all latency is equal. You need to track successful requests and error requests separately.

# Bad: Average latency
latency = total_request_time / total_requests  # Useless

# Good: Percentile latency, separated by status
from prometheus_client import Histogram

REQUEST_LATENCY = Histogram(
    'http_request_duration_seconds',
    'Request latency',
    ['method', 'endpoint', 'status_class'],
    buckets=[.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]
)

@app.middleware
async def track_latency(request, call_next):
    start = time.time()
    response = await call_next(request)
    duration = time.time() - start
    status_class = f"{response.status_code // 100}xx"
    REQUEST_LATENCY.labels(
        method=request.method,
        endpoint=request.url.path,
        status_class=status_class
    ).observe(duration)
    return response

Alert on p99, not p50. Your happiest users don't need help.

- alert: HighLatencyP99
  expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 0.5
  for: 5m
  labels:
    severity: warning

Signal 2: Traffic

Traffic tells you "is this normal?" It's the context for every other signal.

# Current request rate
rate(http_requests_total[5m])

# Compare to same time last week
rate(http_requests_total[5m]) 
  / 
rate(http_requests_total[5m] offset 7d)

# Alert on sudden drops (possible outage nobody noticed)
- alert: TrafficDrop
  expr: >
    rate(http_requests_total[5m]) 
    < 
    (rate(http_requests_total[5m] offset 1h) * 0.5)
  for: 10m
  annotations:
    summary: "Traffic dropped >50% compared to 1 hour ago"

Traffic drops are often more concerning than traffic spikes.

Signal 3: Errors

Track error rate as a percentage, not absolute count:

# Error rate percentage
(
  sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
) * 100

But also track error types separately:

error_categories:
  - 5xx: "Server errors (our fault)"
  - 4xx_excluding_404: "Client errors (possible API issue)"
  - timeout: "Request timeouts"
  - circuit_breaker: "Dependency failures"

Signal 4: Saturation

The most underrated signal. Saturation answers: "how close are we to full?"

# CPU saturation
process_cpu_seconds_total / container_spec_cpu_quota

# Memory saturation
container_memory_working_set_bytes / container_spec_memory_limit_bytes

# Connection pool saturation
active_connections / max_connections

# Queue saturation (the one everyone forgets)
message_queue_depth / message_queue_capacity

Alert before you hit 100%. I use 80% as the threshold for warning and 95% for critical.

Putting It All Together

Every service gets a standard dashboard with four rows:

Row 1: Latency   [p50] [p90] [p99] [error latency]
Row 2: Traffic    [rate] [vs last week] [by endpoint]
Row 3: Errors     [rate %] [by type] [by endpoint]
Row 4: Saturation [CPU] [Memory] [Connections] [Queue]

This fits on one screen. No scrolling. Any engineer can assess service health in 10 seconds.

The Anti-Pattern

Don't build a golden signals dashboard per service manually. Template it:

{
  "dashboard": {
    "title": "Golden Signals: {{ service_name }}",
    "templating": {
      "list": [
        { "name": "service", "type": "query" },
        { "name": "environment", "type": "custom", "options": ["prod", "staging"] }
      ]
    }
  }
}

One template, 50 dashboards. Update once, apply everywhere.

If you want golden signal monitoring that sets itself up automatically, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

The Golden Signals: A Practical Implementation Guide

Samson Tanimawo — Thu, 16 Apr 2026 07:11:12 +0000

Four Metrics to Rule Them All

Google's SRE book introduced the four golden signals: Latency, Traffic, Errors, and Saturation. Simple concept, but I've seen teams struggle with implementation.

Here's a practical guide from someone who's implemented them across 50+ services.

Signal 1: Latency

Not all latency is equal. You need to track successful requests and error requests separately.

# Bad: Average latency
latency = total_request_time / total_requests  # Useless

# Good: Percentile latency, separated by status
from prometheus_client import Histogram

REQUEST_LATENCY = Histogram(
    'http_request_duration_seconds',
    'Request latency',
    ['method', 'endpoint', 'status_class'],
    buckets=[.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]
)

@app.middleware
async def track_latency(request, call_next):
    start = time.time()
    response = await call_next(request)
    duration = time.time() - start
    status_class = f"{response.status_code // 100}xx"
    REQUEST_LATENCY.labels(
        method=request.method,
        endpoint=request.url.path,
        status_class=status_class
    ).observe(duration)
    return response

Alert on p99, not p50. Your happiest users don't need help.

- alert: HighLatencyP99
  expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 0.5
  for: 5m
  labels:
    severity: warning

Signal 2: Traffic

Traffic tells you "is this normal?" It's the context for every other signal.

# Current request rate
rate(http_requests_total[5m])

# Compare to same time last week
rate(http_requests_total[5m]) 
  / 
rate(http_requests_total[5m] offset 7d)

# Alert on sudden drops (possible outage nobody noticed)
- alert: TrafficDrop
  expr: >
    rate(http_requests_total[5m]) 
    < 
    (rate(http_requests_total[5m] offset 1h) * 0.5)
  for: 10m
  annotations:
    summary: "Traffic dropped >50% compared to 1 hour ago"

Traffic drops are often more concerning than traffic spikes.

Signal 3: Errors

Track error rate as a percentage, not absolute count:

# Error rate percentage
(
  sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
) * 100

But also track error types separately:

error_categories:
  - 5xx: "Server errors (our fault)"
  - 4xx_excluding_404: "Client errors (possible API issue)"
  - timeout: "Request timeouts"
  - circuit_breaker: "Dependency failures"

Signal 4: Saturation

The most underrated signal. Saturation answers: "how close are we to full?"

# CPU saturation
process_cpu_seconds_total / container_spec_cpu_quota

# Memory saturation
container_memory_working_set_bytes / container_spec_memory_limit_bytes

# Connection pool saturation
active_connections / max_connections

# Queue saturation (the one everyone forgets)
message_queue_depth / message_queue_capacity

Alert before you hit 100%. I use 80% as the threshold for warning and 95% for critical.

Putting It All Together

Every service gets a standard dashboard with four rows:

Row 1: Latency   [p50] [p90] [p99] [error latency]
Row 2: Traffic    [rate] [vs last week] [by endpoint]
Row 3: Errors     [rate %] [by type] [by endpoint]
Row 4: Saturation [CPU] [Memory] [Connections] [Queue]

This fits on one screen. No scrolling. Any engineer can assess service health in 10 seconds.

The Anti-Pattern

Don't build a golden signals dashboard per service manually. Template it:

{
  "dashboard": {
    "title": "Golden Signals: {{ service_name }}",
    "templating": {
      "list": [
        { "name": "service", "type": "query" },
        { "name": "environment", "type": "custom", "options": ["prod", "staging"] }
      ]
    }
  }
}

One template, 50 dashboards. Update once, apply everywhere.

If you want golden signal monitoring that sets itself up automatically, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Kubernetes Observability: What to Monitor and Why

Samson Tanimawo — Thu, 16 Apr 2026 00:19:43 +0000

The Kubernetes Monitoring Maze

Kubernetes gives you a thousand metrics out of the box. Most teams monitor all of them and understand none of them.

After running K8s in production for four years, here's what actually matters.

The Three Layers

Kubernetes observability has three distinct layers, and you need different strategies for each:

Layer 1: Cluster Health (infrastructure)
Layer 2: Workload Health (your apps)
Layer 3: Application Performance (user experience)

Layer 1: Cluster Health

These are your "is the platform working?" metrics:

critical_cluster_metrics:
  nodes:
    - node_ready_status        # Are all nodes healthy?
    - node_cpu_utilization     # Alert at 85%
    - node_memory_utilization  # Alert at 90%
    - node_disk_pressure       # Boolean alert
    - node_pid_pressure        # Rarely fires, always critical

  control_plane:
    - apiserver_request_latency_p99  # Alert > 1s
    - etcd_disk_wal_fsync_duration   # Alert > 100ms
    - scheduler_pending_pods         # Alert if > 0 for 5min
    - controller_manager_queue_depth # Alert if growing

Pro tip: Don't alert on individual node CPU. Alert on cluster-level capacity:

# Alert when cluster is 80% utilized
(
  sum(node_cpu_seconds_total{mode!="idle"}) 
  / 
  sum(node_cpu_seconds_total)
) > 0.80

Layer 2: Workload Health

This is where most teams get it wrong. They monitor pods instead of workloads.

critical_workload_metrics:
  deployments:
    - available_replicas < desired_replicas  # For > 5min
    - deployment_generation != observed_generation  # Stuck rollout

  pods:
    - restart_count increasing       # CrashLoopBackOff detection
    - container_oom_killed            # Memory limits too low
    - pod_pending_duration > 2min     # Scheduling issues

  hpa:
    - current_replicas == max_replicas  # Scale ceiling hit
    - cpu_utilization_vs_target         # Consistently above target

The most valuable alert I ever wrote:

# Detect pods stuck in CrashLoopBackOff
alert: PodCrashLooping
expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
for: 15m
labels:
  severity: warning
annotations:
  summary: "Pod {{ $labels.pod }} is crash-looping"
  description: "{{ $labels.pod }} in {{ $labels.namespace }} has restarted {{ $value }} times in 15 minutes"

Layer 3: Application Performance

This is what your users actually care about:

application_metrics:
  red_method:  # Rate, Errors, Duration
    - request_rate_per_second
    - error_rate_percentage        # Alert > 1%
    - request_duration_p99         # Alert > 500ms

  use_method:  # Utilization, Saturation, Errors
    - cpu_request_vs_limit_ratio
    - memory_request_vs_limit_ratio
    - network_receive_bytes_rate

The Dashboard That Saves Us

We built a single "K8s Health" dashboard with four panels:

Cluster capacity — CPU/Memory/Disk utilization per node pool
Workload status — Table of all deployments with health status
Error rates — All services, sorted by error rate
Recent events — K8s events filtered to warnings and errors

This one dashboard answers 90% of "is something wrong?" questions.

Common Mistakes

Monitoring pods instead of services — Pods are ephemeral, services are what matter
Not setting resource requests — Without requests, your metrics are meaningless
Alerting on resource usage instead of SLOs — High CPU isn't a problem if latency is fine
Ignoring the control plane — An unhealthy API server affects everything

If you want unified Kubernetes observability without the complexity, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Kubernetes Observability: What to Monitor and Why

Samson Tanimawo — Wed, 15 Apr 2026 23:29:48 +0000

The Kubernetes Monitoring Maze

Kubernetes gives you a thousand metrics out of the box. Most teams monitor all of them and understand none of them.

After running K8s in production for four years, here's what actually matters.

The Three Layers

Kubernetes observability has three distinct layers, and you need different strategies for each:

Layer 1: Cluster Health (infrastructure)
Layer 2: Workload Health (your apps)
Layer 3: Application Performance (user experience)

Layer 1: Cluster Health

These are your "is the platform working?" metrics:

critical_cluster_metrics:
  nodes:
    - node_ready_status        # Are all nodes healthy?
    - node_cpu_utilization     # Alert at 85%
    - node_memory_utilization  # Alert at 90%
    - node_disk_pressure       # Boolean alert
    - node_pid_pressure        # Rarely fires, always critical

  control_plane:
    - apiserver_request_latency_p99  # Alert > 1s
    - etcd_disk_wal_fsync_duration   # Alert > 100ms
    - scheduler_pending_pods         # Alert if > 0 for 5min
    - controller_manager_queue_depth # Alert if growing

Pro tip: Don't alert on individual node CPU. Alert on cluster-level capacity:

# Alert when cluster is 80% utilized
(
  sum(node_cpu_seconds_total{mode!="idle"}) 
  / 
  sum(node_cpu_seconds_total)
) > 0.80

Layer 2: Workload Health

This is where most teams get it wrong. They monitor pods instead of workloads.

critical_workload_metrics:
  deployments:
    - available_replicas < desired_replicas  # For > 5min
    - deployment_generation != observed_generation  # Stuck rollout

  pods:
    - restart_count increasing       # CrashLoopBackOff detection
    - container_oom_killed            # Memory limits too low
    - pod_pending_duration > 2min     # Scheduling issues

  hpa:
    - current_replicas == max_replicas  # Scale ceiling hit
    - cpu_utilization_vs_target         # Consistently above target

The most valuable alert I ever wrote:

# Detect pods stuck in CrashLoopBackOff
alert: PodCrashLooping
expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
for: 15m
labels:
  severity: warning
annotations:
  summary: "Pod {{ $labels.pod }} is crash-looping"
  description: "{{ $labels.pod }} in {{ $labels.namespace }} has restarted {{ $value }} times in 15 minutes"

Layer 3: Application Performance

This is what your users actually care about:

application_metrics:
  red_method:  # Rate, Errors, Duration
    - request_rate_per_second
    - error_rate_percentage        # Alert > 1%
    - request_duration_p99         # Alert > 500ms

  use_method:  # Utilization, Saturation, Errors
    - cpu_request_vs_limit_ratio
    - memory_request_vs_limit_ratio
    - network_receive_bytes_rate

The Dashboard That Saves Us

We built a single "K8s Health" dashboard with four panels:

Cluster capacity — CPU/Memory/Disk utilization per node pool
Workload status — Table of all deployments with health status
Error rates — All services, sorted by error rate
Recent events — K8s events filtered to warnings and errors

This one dashboard answers 90% of "is something wrong?" questions.

Common Mistakes

Monitoring pods instead of services — Pods are ephemeral, services are what matter
Not setting resource requests — Without requests, your metrics are meaningless
Alerting on resource usage instead of SLOs — High CPU isn't a problem if latency is fine
Ignoring the control plane — An unhealthy API server affects everything

If you want unified Kubernetes observability without the complexity, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Kubernetes Observability: What to Monitor and Why

Samson Tanimawo — Wed, 15 Apr 2026 23:18:54 +0000

The Kubernetes Monitoring Maze

Kubernetes gives you a thousand metrics out of the box. Most teams monitor all of them and understand none of them.

After running K8s in production for four years, here's what actually matters.

The Three Layers

Kubernetes observability has three distinct layers, and you need different strategies for each:

Layer 1: Cluster Health (infrastructure)
Layer 2: Workload Health (your apps)
Layer 3: Application Performance (user experience)

Layer 1: Cluster Health

These are your "is the platform working?" metrics:

critical_cluster_metrics:
  nodes:
    - node_ready_status        # Are all nodes healthy?
    - node_cpu_utilization     # Alert at 85%
    - node_memory_utilization  # Alert at 90%
    - node_disk_pressure       # Boolean alert
    - node_pid_pressure        # Rarely fires, always critical

  control_plane:
    - apiserver_request_latency_p99  # Alert > 1s
    - etcd_disk_wal_fsync_duration   # Alert > 100ms
    - scheduler_pending_pods         # Alert if > 0 for 5min
    - controller_manager_queue_depth # Alert if growing

Pro tip: Don't alert on individual node CPU. Alert on cluster-level capacity:

# Alert when cluster is 80% utilized
(
  sum(node_cpu_seconds_total{mode!="idle"}) 
  / 
  sum(node_cpu_seconds_total)
) > 0.80

Layer 2: Workload Health

This is where most teams get it wrong. They monitor pods instead of workloads.

critical_workload_metrics:
  deployments:
    - available_replicas < desired_replicas  # For > 5min
    - deployment_generation != observed_generation  # Stuck rollout

  pods:
    - restart_count increasing       # CrashLoopBackOff detection
    - container_oom_killed            # Memory limits too low
    - pod_pending_duration > 2min     # Scheduling issues

  hpa:
    - current_replicas == max_replicas  # Scale ceiling hit
    - cpu_utilization_vs_target         # Consistently above target

The most valuable alert I ever wrote:

# Detect pods stuck in CrashLoopBackOff
alert: PodCrashLooping
expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
for: 15m
labels:
  severity: warning
annotations:
  summary: "Pod {{ $labels.pod }} is crash-looping"
  description: "{{ $labels.pod }} in {{ $labels.namespace }} has restarted {{ $value }} times in 15 minutes"

Layer 3: Application Performance

This is what your users actually care about:

application_metrics:
  red_method:  # Rate, Errors, Duration
    - request_rate_per_second
    - error_rate_percentage        # Alert > 1%
    - request_duration_p99         # Alert > 500ms

  use_method:  # Utilization, Saturation, Errors
    - cpu_request_vs_limit_ratio
    - memory_request_vs_limit_ratio
    - network_receive_bytes_rate

The Dashboard That Saves Us

We built a single "K8s Health" dashboard with four panels:

Cluster capacity — CPU/Memory/Disk utilization per node pool
Workload status — Table of all deployments with health status
Error rates — All services, sorted by error rate
Recent events — K8s events filtered to warnings and errors

This one dashboard answers 90% of "is something wrong?" questions.

Common Mistakes

Monitoring pods instead of services — Pods are ephemeral, services are what matter
Not setting resource requests — Without requests, your metrics are meaningless
Alerting on resource usage instead of SLOs — High CPU isn't a problem if latency is fine
Ignoring the control plane — An unhealthy API server affects everything

If you want unified Kubernetes observability without the complexity, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Kubernetes Observability: What to Monitor and Why

Samson Tanimawo — Wed, 15 Apr 2026 23:08:21 +0000

The Kubernetes Monitoring Maze

Kubernetes gives you a thousand metrics out of the box. Most teams monitor all of them and understand none of them.

After running K8s in production for four years, here's what actually matters.

The Three Layers

Kubernetes observability has three distinct layers, and you need different strategies for each:

Layer 1: Cluster Health (infrastructure)
Layer 2: Workload Health (your apps)
Layer 3: Application Performance (user experience)

Layer 1: Cluster Health

These are your "is the platform working?" metrics:

critical_cluster_metrics:
  nodes:
    - node_ready_status        # Are all nodes healthy?
    - node_cpu_utilization     # Alert at 85%
    - node_memory_utilization  # Alert at 90%
    - node_disk_pressure       # Boolean alert
    - node_pid_pressure        # Rarely fires, always critical

  control_plane:
    - apiserver_request_latency_p99  # Alert > 1s
    - etcd_disk_wal_fsync_duration   # Alert > 100ms
    - scheduler_pending_pods         # Alert if > 0 for 5min
    - controller_manager_queue_depth # Alert if growing

Pro tip: Don't alert on individual node CPU. Alert on cluster-level capacity:

# Alert when cluster is 80% utilized
(
  sum(node_cpu_seconds_total{mode!="idle"}) 
  / 
  sum(node_cpu_seconds_total)
) > 0.80

Layer 2: Workload Health

This is where most teams get it wrong. They monitor pods instead of workloads.

critical_workload_metrics:
  deployments:
    - available_replicas < desired_replicas  # For > 5min
    - deployment_generation != observed_generation  # Stuck rollout

  pods:
    - restart_count increasing       # CrashLoopBackOff detection
    - container_oom_killed            # Memory limits too low
    - pod_pending_duration > 2min     # Scheduling issues

  hpa:
    - current_replicas == max_replicas  # Scale ceiling hit
    - cpu_utilization_vs_target         # Consistently above target

The most valuable alert I ever wrote:

# Detect pods stuck in CrashLoopBackOff
alert: PodCrashLooping
expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
for: 15m
labels:
  severity: warning
annotations:
  summary: "Pod {{ $labels.pod }} is crash-looping"
  description: "{{ $labels.pod }} in {{ $labels.namespace }} has restarted {{ $value }} times in 15 minutes"

Layer 3: Application Performance

This is what your users actually care about:

application_metrics:
  red_method:  # Rate, Errors, Duration
    - request_rate_per_second
    - error_rate_percentage        # Alert > 1%
    - request_duration_p99         # Alert > 500ms

  use_method:  # Utilization, Saturation, Errors
    - cpu_request_vs_limit_ratio
    - memory_request_vs_limit_ratio
    - network_receive_bytes_rate

The Dashboard That Saves Us

We built a single "K8s Health" dashboard with four panels:

Cluster capacity — CPU/Memory/Disk utilization per node pool
Workload status — Table of all deployments with health status
Error rates — All services, sorted by error rate
Recent events — K8s events filtered to warnings and errors

This one dashboard answers 90% of "is something wrong?" questions.

Common Mistakes

Monitoring pods instead of services — Pods are ephemeral, services are what matter
Not setting resource requests — Without requests, your metrics are meaningless
Alerting on resource usage instead of SLOs — High CPU isn't a problem if latency is fine
Ignoring the control plane — An unhealthy API server affects everything

If you want unified Kubernetes observability without the complexity, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

On-Call Wellness: Protecting Your Engineers from Burnout

Samson Tanimawo — Wed, 15 Apr 2026 15:21:46 +0000

The On-Call Burnout Epidemic

I watched three senior SREs leave our team in six months. Exit interviews all said the same thing: on-call was unsustainable.

We were spending $500K+ recruiting replacements for a problem that could have been fixed with $0 and better practices.

The Warning Signs

Before someone quits, they show these signals:

Cynicism in post-mortems — "This will never get fixed"
Alert numbness — Slow to respond, missed pages
Vacation avoidance — "I can't take time off, who would cover?"
Scope creep rejection — "That's not my problem"
Meeting silence — Previously engaged, now checked out

If you see three or more of these in someone on your team, they're already halfway out the door.

What We Changed

1. Hard Cap on Pages

We set a maximum of 2 pages per 8-hour on-call shift. If someone gets paged more than that, the secondary automatically takes over and the incident is escalated as a process failure.

on_call_policy:
  max_pages_per_shift: 2
  shift_duration_hours: 8
  overflow_action: "escalate_to_secondary"
  overflow_review: "weekly_ops_review"

2. Follow-the-Sun Rotation

We stopped asking people to be on-call at 3am. With team members across US timezones, we created overlapping business-hours shifts:

Shift A (Eastern):  6am - 2pm ET
Shift B (Central):  11am - 7pm CT  
Shift C (Pacific):  2pm - 10pm PT
Overnight:          Managed by alert automation + escalation

Nobody gets paged between 10pm and 6am unless it's a true P1.

3. On-Call Compensation

We implemented:

$500 flat fee per on-call week
$200 per off-hours page
Comp day after any overnight incident > 30 minutes
On-call swaps require zero management approval

4. The "Toil Budget"

Each engineer gets a toil budget: maximum 30% of their time on operational work. If toil exceeds 30%, they're pulled from on-call until the team automates the excess.

Weekly toil tracking:
  Alert response:     4 hours
  Manual deployments: 2 hours  
  Config updates:     1 hour
  Ad-hoc debugging:   3 hours
  ─────────────────────────
  Total:              10 hours (25% of 40hr week) ✓

5. Quarterly On-Call Reviews

Every quarter, we review:

Pages per person
Off-hours disruptions
Toil percentages
Team sentiment survey
Attrition risk signals

The Results

Metric	Before	After (6 months)
Attrition rate	40%/year	8%/year
Pages per shift	4.7	1.2
Off-hours pages	12/week	2/week
Team NPS	-15	+45
Recruitment cost saved	-	~$400K/year

The Key Insight

On-call wellness isn't a perk. It's a business decision. Replacing a senior SRE costs $150-200K in recruiting, onboarding, and lost productivity. Preventing burnout costs almost nothing.

If you're looking to reduce on-call toil and protect your team from burnout, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

On-Call Wellness: Protecting Your Engineers from Burnout

Samson Tanimawo — Wed, 15 Apr 2026 15:06:15 +0000

The On-Call Burnout Epidemic

I watched three senior SREs leave our team in six months. Exit interviews all said the same thing: on-call was unsustainable.

We were spending $500K+ recruiting replacements for a problem that could have been fixed with $0 and better practices.

The Warning Signs

Before someone quits, they show these signals:

Cynicism in post-mortems — "This will never get fixed"
Alert numbness — Slow to respond, missed pages
Vacation avoidance — "I can't take time off, who would cover?"
Scope creep rejection — "That's not my problem"
Meeting silence — Previously engaged, now checked out

If you see three or more of these in someone on your team, they're already halfway out the door.

What We Changed

1. Hard Cap on Pages

We set a maximum of 2 pages per 8-hour on-call shift. If someone gets paged more than that, the secondary automatically takes over and the incident is escalated as a process failure.

on_call_policy:
  max_pages_per_shift: 2
  shift_duration_hours: 8
  overflow_action: "escalate_to_secondary"
  overflow_review: "weekly_ops_review"

2. Follow-the-Sun Rotation

We stopped asking people to be on-call at 3am. With team members across US timezones, we created overlapping business-hours shifts:

Shift A (Eastern):  6am - 2pm ET
Shift B (Central):  11am - 7pm CT  
Shift C (Pacific):  2pm - 10pm PT
Overnight:          Managed by alert automation + escalation

Nobody gets paged between 10pm and 6am unless it's a true P1.

3. On-Call Compensation

We implemented:

$500 flat fee per on-call week
$200 per off-hours page
Comp day after any overnight incident > 30 minutes
On-call swaps require zero management approval

4. The "Toil Budget"

Each engineer gets a toil budget: maximum 30% of their time on operational work. If toil exceeds 30%, they're pulled from on-call until the team automates the excess.

Weekly toil tracking:
  Alert response:     4 hours
  Manual deployments: 2 hours  
  Config updates:     1 hour
  Ad-hoc debugging:   3 hours
  ─────────────────────────
  Total:              10 hours (25% of 40hr week) ✓

5. Quarterly On-Call Reviews

Every quarter, we review:

Pages per person
Off-hours disruptions
Toil percentages
Team sentiment survey
Attrition risk signals

The Results

Metric	Before	After (6 months)
Attrition rate	40%/year	8%/year
Pages per shift	4.7	1.2
Off-hours pages	12/week	2/week
Team NPS	-15	+45
Recruitment cost saved	-	~$400K/year

The Key Insight

On-call wellness isn't a perk. It's a business decision. Replacing a senior SRE costs $150-200K in recruiting, onboarding, and lost productivity. Preventing burnout costs almost nothing.

If you're looking to reduce on-call toil and protect your team from burnout, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Post-Mortem Best Practices That Actually Drive Change

Samson Tanimawo — Wed, 15 Apr 2026 07:37:37 +0000

The Post-Mortem Nobody Learns From

I've sat through hundreds of post-mortems. Most follow the same pattern: something breaks, someone writes a Google Doc, we have a meeting, we list action items, nobody follows up, the same thing happens again in 3 months.

Here's how to break the cycle.

The Blameless Culture Trap

"Blameless" doesn't mean "actionless." The biggest failure mode I see is teams that use blameless culture as an excuse to avoid accountability.

Blameless means: we don't punish the person who pushed the bad deploy.
Blameless does NOT mean: nobody is responsible for fixing the systemic issue.

My Post-Mortem Template

# Incident: [SERVICE] [SYMPTOM] on [DATE]

## Impact
- Duration: X minutes
- Users affected: N
- Revenue impact: $X
- SLO budget consumed: X%

## Timeline (UTC)
- HH:MM - First alert fired
- HH:MM - On-call acknowledged
- HH:MM - Root cause identified
- HH:MM - Fix deployed
- HH:MM - Service recovered
- HH:MM - All-clear declared

## Root Cause
[2-3 sentences. Technical but readable.]

## Contributing Factors
1. [Factor that made the incident possible]
2. [Factor that made detection slow]
3. [Factor that made resolution slow]

## What Went Well
- [Something that worked]
- [Something that helped]

## What Went Wrong
- [Process failure]
- [Technical gap]

## Action Items
| Action | Owner | Priority | Due Date | Status |
|--------|-------|----------|----------|--------|
| ...    | ...   | P1/P2/P3 | ...      | Open   |

## Lessons Learned
[1-2 paragraphs of genuine insight]

The Action Item Problem

Action items from post-mortems have a 30% completion rate industry-wide. That's terrible. Here's why:

Too many items (I've seen post-mortems with 15 action items)
No clear ownership
No deadline
No follow-up mechanism
Competing with feature work

The Fix: Three Rules

Rule 1: Maximum 3 action items per post-mortem.

If you can't narrow it to 3, you haven't identified the real problems.

Rule 2: Every action item gets a JIRA ticket linked to the next sprint.

Not "someday." Not "backlog." Next sprint. If it's not important enough for next sprint, it's not an action item.

Rule 3: Review completion in the next post-mortem.

Start every post-mortem meeting by reviewing open action items from previous incidents. This creates accountability without blame.

# Post-mortem meeting agenda

1. Review open action items (10 min)
   - Incident #42: "Add circuit breaker" — DONE
   - Incident #43: "Add canary deploys" — IN PROGRESS (blocked on CI)
   - Incident #44: "Fix retry logic" — NOT STARTED (reassigning)

2. Current incident review (30 min)
   - Timeline walkthrough
   - Contributing factors
   - Action items (max 3)

3. Pattern analysis (10 min)
   - Any recurring themes?
   - Systemic issues to address?

The Metric That Matters

Track Repeat Incident Rate: what percentage of incidents have the same root cause as a previous incident?

When we started tracking this, our repeat rate was 45%. After implementing the three rules above, it dropped to 12% over six months.

That's the real measure of whether your post-mortems are working.

If you're looking for better incident learning loops and pattern detection across your post-mortems, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Post-Mortem Best Practices That Actually Drive Change

Samson Tanimawo — Wed, 15 Apr 2026 07:27:49 +0000

The Post-Mortem Nobody Learns From

Here's how to break the cycle.

The Blameless Culture Trap

"Blameless" doesn't mean "actionless." The biggest failure mode I see is teams that use blameless culture as an excuse to avoid accountability.

Blameless means: we don't punish the person who pushed the bad deploy.
Blameless does NOT mean: nobody is responsible for fixing the systemic issue.

My Post-Mortem Template

# Incident: [SERVICE] [SYMPTOM] on [DATE]

## Impact
- Duration: X minutes
- Users affected: N
- Revenue impact: $X
- SLO budget consumed: X%

## Timeline (UTC)
- HH:MM - First alert fired
- HH:MM - On-call acknowledged
- HH:MM - Root cause identified
- HH:MM - Fix deployed
- HH:MM - Service recovered
- HH:MM - All-clear declared

## Root Cause
[2-3 sentences. Technical but readable.]

## Contributing Factors
1. [Factor that made the incident possible]
2. [Factor that made detection slow]
3. [Factor that made resolution slow]

## What Went Well
- [Something that worked]
- [Something that helped]

## What Went Wrong
- [Process failure]
- [Technical gap]

## Action Items
| Action | Owner | Priority | Due Date | Status |
|--------|-------|----------|----------|--------|
| ...    | ...   | P1/P2/P3 | ...      | Open   |

## Lessons Learned
[1-2 paragraphs of genuine insight]

The Action Item Problem

Action items from post-mortems have a 30% completion rate industry-wide. That's terrible. Here's why:

Too many items (I've seen post-mortems with 15 action items)
No clear ownership
No deadline
No follow-up mechanism
Competing with feature work

The Fix: Three Rules

Rule 1: Maximum 3 action items per post-mortem.

If you can't narrow it to 3, you haven't identified the real problems.

Rule 2: Every action item gets a JIRA ticket linked to the next sprint.

Not "someday." Not "backlog." Next sprint. If it's not important enough for next sprint, it's not an action item.

Rule 3: Review completion in the next post-mortem.

Start every post-mortem meeting by reviewing open action items from previous incidents. This creates accountability without blame.

# Post-mortem meeting agenda

1. Review open action items (10 min)
   - Incident #42: "Add circuit breaker" — DONE
   - Incident #43: "Add canary deploys" — IN PROGRESS (blocked on CI)
   - Incident #44: "Fix retry logic" — NOT STARTED (reassigning)

2. Current incident review (30 min)
   - Timeline walkthrough
   - Contributing factors
   - Action items (max 3)

3. Pattern analysis (10 min)
   - Any recurring themes?
   - Systemic issues to address?

The Metric That Matters

Track Repeat Incident Rate: what percentage of incidents have the same root cause as a previous incident?

When we started tracking this, our repeat rate was 45%. After implementing the three rules above, it dropped to 12% over six months.

That's the real measure of whether your post-mortems are working.

If you're looking for better incident learning loops and pattern detection across your post-mortems, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Runbook Automation: From 45-Minute Fixes to 90-Second Recoveries

Samson Tanimawo — Wed, 15 Apr 2026 03:21:55 +0000

The Runbook Nobody Reads

We had runbooks. Beautiful, detailed, Google-Docs runbooks. 47 pages long. Nobody read them at 3am.

The problem isn't the documentation. The problem is expecting a sleep-deprived human to follow a 47-step procedure correctly.

The Automation Ladder

I think about runbook automation as a ladder:

Level 0: No runbook (tribal knowledge)
Level 1: Written runbook (Google Doc)
Level 2: Structured runbook (checklist format)
Level 3: Semi-automated (scripts for each step)
Level 4: Fully automated (one-click remediation)
Level 5: Self-healing (no human needed)

Most teams are at Level 1-2. The goal is Level 4-5 for your top 10 incidents.

Identifying Automation Candidates

Not everything should be automated. Start with high-frequency, well-understood procedures:

-- Query your incident database
SELECT 
  root_cause_category,
  COUNT(*) as frequency,
  AVG(resolution_time_minutes) as avg_mttr,
  COUNT(*) * AVG(resolution_time_minutes) as total_impact_minutes
FROM incidents
WHERE created_at > NOW() - INTERVAL '6 months'
GROUP BY root_cause_category
ORDER BY total_impact_minutes DESC
LIMIT 10;

For us, the top 5 were:

Disk full on log volumes (2x/week)
Memory leak requiring pod restart (1x/week)
Certificate expiry (1x/month, but high impact)
Database connection pool exhaustion (1x/week)
Stuck deployment (2x/week)

Example: Disk Full Auto-Remediation

Before (Level 1 — runbook):

1. SSH to the affected host
2. Run df -h to confirm
3. Check /var/log for large files
4. Run logrotate manually
5. If still full, find and remove old files
6. If still full, expand the volume
7. Verify service recovered

After (Level 5 — self-healing):

#!/bin/bash
# disk-remediation.sh — triggered by monitoring alert

HOST=$1
THRESHOLD=90

USAGE=$(ssh $HOST "df /var/log --output=pcent | tail -1 | tr -dc '0-9'")

if [ "$USAGE" -gt "$THRESHOLD" ]; then
  echo "[Auto-Remediation] Disk at ${USAGE}% on ${HOST}"

  # Step 1: Rotate logs
  ssh $HOST "sudo logrotate -f /etc/logrotate.conf"

  # Step 2: Clean old logs (>7 days)
  ssh $HOST "find /var/log -name '*.gz' -mtime +7 -delete"

  # Step 3: Clean temp files
  ssh $HOST "find /tmp -mtime +3 -delete 2>/dev/null"

  # Verify
  NEW_USAGE=$(ssh $HOST "df /var/log --output=pcent | tail -1 | tr -dc '0-9'")

  if [ "$NEW_USAGE" -lt "$THRESHOLD" ]; then
    echo "[Auto-Remediation] Resolved. ${USAGE}% -> ${NEW_USAGE}%"
    notify_slack "Disk full on ${HOST} auto-resolved (${USAGE}% -> ${NEW_USAGE}%)"
  else
    echo "[Auto-Remediation] Still at ${NEW_USAGE}%. Escalating."
    page_oncall "Disk full on ${HOST} - auto-remediation failed. Manual intervention needed."
  fi
fi

The Results

Incident Type	Before (MTTR)	After (MTTR)	Automation Level
Disk full	25 min	90 sec	Self-healing
Memory leak	15 min	45 sec	One-click
Cert expiry	45 min	0 (prevented)	Proactive
DB conn pool	20 min	60 sec	Self-healing
Stuck deploy	30 min	2 min	One-click

Total monthly incident time: 14 hours → 45 minutes.

The Golden Rule

If you've fixed the same incident three times manually, it's time to automate. The third time pays for the automation effort. Everything after that is pure savings.

If you're tired of repetitive incident response and want to automate your runbooks with AI, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com