Forem: Neeraja Khanapure

Something I wish someone had told me five years earlier:

Neeraja Khanapure — Fri, 03 Apr 2026 09:39:52 +0000

LinkedIn Draft — Insight (2026-04-03)

Something I wish someone had told me five years earlier:

Zero-downtime deployments: what 'zero' actually requires most teams don't have

Most teams say they do zero-downtime deploys and mean 'we haven't gotten a complaint in a while.' Actually measuring it reveals the truth: connection drops, in-flight request failures, and cache invalidation spikes during rollouts that nobody's tracking because nobody defined what zero means.

What 'zero downtime' actually requires:

✓ Health checks reflect REAL readiness (not just 'process started')
✓ Graceful shutdown drains in-flight requests (SIGTERM handling)
✓ Connection draining at the load balancer (not just the pod)
✓ Rollback faster than the deploy (< 5 min, automated)
✓ SLI measurement during the rollout window (not just after)

Missing any one of these = not zero downtime. Just unmonitored downtime.

The non-obvious part:
→ The most common failure mode is passing health checks before the app is actually ready — DB connections not pooled, caches not warm, background workers not started. The pod is 'Ready' and the app is still initializing. Users see errors. Nobody's dashboard shows it because nobody's measuring error rate during the rollout window.

My rule:
→ Define 'zero downtime' with a measurable SLI: error rate < 0.1% during any 5-minute deploy window. Validate this in staging before calling it done. Measure it in production on every release.

Worth reading:
▸ Kubernetes deployment strategies — rolling, blue/green, canary with traffic splitting
▸ AWS ALB / GCP Cloud Load Balancing — connection draining configuration and health check tuning

https://neeraja-portfolio-v1.vercel.app/insights/zero-downtime-deployments-what-zero-actually-requires-most-teams-dont-have

If you're a manager reading this — it's worth asking your team where they are on this.

devops #sre #observability #platformengineering

End of week. Here's the thing I kept coming back to:

Neeraja Khanapure — Thu, 02 Apr 2026 14:42:22 +0000

LinkedIn Draft — Insight (2026-04-02)

End of week. Here's the thing I kept coming back to:

SLOs work when they create conversations, not when they create compliance

Most SLOs are set once, filed in a doc, and forgotten until an incident. The teams getting real value from error budgets use them as a weekly forcing function — a number that makes the reliability vs. velocity tradeoff visible to engineers and product managers in the same room.

SLO as compliance (common):     SLO as conversation (effective):

Set SLO ──▶ Monitor              Set SLO ──▶ Weekly budget review
     │                                │          │
  Incident ──▶ Check SLO         Budget OK  Budget low
     │              │                │          │
   Blame       Finger-pointing    Ship fast  Freeze features
                                   │          │
                               Engineering + Product aligned

The non-obvious part:
→ An SLO that's never violated is almost always a problem. Either it's too loose (you're over-investing in reliability) or it's not being measured honestly. Both cost money in different ways. The goal is a number that occasionally creates productive tension.

My rule:
→ Review error budgets in sprint planning alongside features. If engineering and product aren't having an uncomfortable conversation once a quarter, your SLO isn't tight enough.

Worth reading:
▸ Alex Hidalgo — 'Implementing Service Level Objectives' (O'Reilly, 2020)
▸ Google SRE Workbook — Alerting on SLOs (ch. 5, free at sre.google)

https://neeraja-portfolio-v1.vercel.app/insights/slos-work-when-they-create-conversations-not-when-they-create-compliance

What's the version of this that your org gets wrong? Drop it below.

devops #sre #observability #platformengineering

This pattern has saved production twice in the last year:

Neeraja Khanapure — Tue, 31 Mar 2026 09:44:19 +0000

LinkedIn Draft — Workflow (2026-03-31)

This pattern has saved production twice in the last year:

Service mesh adoption: the operational debt lands before the value does

Service meshes promise mTLS, traffic splitting, and deep observability. What arrives first is a new category of production failures your team has never debugged before.

Adoption curve reality:

Value
  │                              ╱ mTLS + traffic control
  │                         ╱
  │              ╱╲  complexity trough
  │         ╱╲╱
  │    ╱╲╱   ← sidecar failures, upgrade pain
  │╱
  └──────────────────────────────▶ Time
     Week 1     Month 3     Month 9

Where it breaks:
▸ Sidecar injection failures look like app bugs — hours spent debugging the wrong layer.
▸ mTLS policy rollout in a live cluster requires namespace-by-namespace phasing — one mistake stops traffic.
▸ Mesh upgrades require coordinated sidecar restarts across the cluster — on large deployments, that's everything.

The rule I keep coming back to:
→ Start mesh in observability-only mode (no policy enforcement). Prove value in one namespace first. Earn the rollout, don't mandate it.

How I sanity-check it:
▸ Linkerd for latency-sensitive workloads — lower resource overhead than Istio's Envoy per sidecar.
▸ Namespace-level feature flags for mesh policy — lets you roll back one team without affecting others.

The difference between a senior engineer and a principal is knowing which guardrails to build before you need them.

Deep dive: https://neeraja-portfolio-v1.vercel.app/workflows/service-mesh-adoption-the-operational-debt-lands-before-the-value-does

If this triggered a war story, I'd genuinely love to hear it.

kubernetes #devops #sre #platformengineering

Something every senior engineer learns the expensive way:

Neeraja Khanapure — Sat, 28 Mar 2026 20:48:45 +0000

LinkedIn Draft — Workflow (2026-03-28)

Something every senior engineer learns the expensive way:

Terraform DAGs at scale: when the graph becomes the hazard

Terraform's dependency graph is elegant at small scale. At 500+ resources across a mono-repo, it becomes the most dangerous part of your infrastructure.

SAFE (small module):              DANGEROUS (at scale):

[vpc] ──▶ [subnet] ──▶ [ec2]     [shared-net] ──▶ [team-a-infra]
                                          │         [team-b-infra]
                                          │         [team-c-infra]
                                          │         [data-layer]
                                  One change → fan-out destroy/create

Where it breaks:
▸ Implicit ordering assumptions survive until a refactor exposes them — usually as an unplanned destroy chain in prod.
▸ Fan-out graphs make blast radius review near-impossible. 'What does this change affect?' has no fast answer.
▸ depends_on papering over bad module interfaces — it fixes the symptom and couples the modules permanently.

The rule I keep coming back to:
→ If a module needs depends_on to be safe, the module boundary is wrong. Redesign the interface — don't paper over it.

How I sanity-check it:
▸ terraform graph | dot -Tsvg > graph.svg — visualize fan-out and cycles before every major refactor.
▸ Gate all applies with OPA/Conftest + mandatory human review on any planned destroy operations.

The difference between a senior engineer and a principal is knowing which guardrails to build before you need them.

Deep dive: https://neeraja-portfolio-v1.vercel.app/workflows/terraform-dags-at-scale-when-the-graph-becomes-the-hazard

Curious what guardrails you've built around this. Drop your pattern below.

terraform #iac #devops #sre

A hard-earned rule from incident retrospectives:

Neeraja Khanapure — Sat, 28 Mar 2026 17:11:20 +0000

LinkedIn Draft — Workflow (2026-03-28)

A hard-earned rule from incident retrospectives:

GitOps drift: the silent accumulation that makes clusters unmanageable

GitOps promises Git as the source of truth. The reality: every manual kubectl during an incident is a lie you told your cluster and forgot to retract.

GitOps truth gap over time:

Week 1:  Git ══════════ Cluster  (clean)
Week 4:  Git ══════╌╌╌╌ Cluster  (2 manual patches)
Week 12: Git ════╌╌╌╌╌╌╌╌╌╌╌╌╌  (drift accumulates)
                         Cluster  (unknown state)

Where it breaks:
▸ Manual patches during incidents create cluster state Git doesn't know about — Argo/Flux will overwrite it silently.
▸ Secrets managed outside GitOps (sealed-secrets, Vault agent) drift independently — invisible in sync status.
▸ Multi-cluster setups multiply drift: each cluster diverges at its own pace once human intervention happens.

The rule I keep coming back to:
→ Treat every manual cluster change as a 5-minute loan. Commit it back to Git before the incident closes — or it's gone.

How I sanity-check it:
▸ Argo CD drift detection dashboard — surface out-of-sync resources before they become incident contributors.
▸ Weekly diff job: live cluster state vs Git. Opens a PR for anything untracked. Makes drift visible before it's painful.

The best platform teams I've seen measure success by how rarely product teams have to think about infrastructure.

Deep dive: https://neeraja-portfolio-v1.vercel.app/workflows

Curious what guardrails you've built around this. Drop your pattern below.

gitops #kubernetes #devops #platformengineering

One insight that changed how I design systems:

Neeraja Khanapure — Sat, 28 Mar 2026 16:48:55 +0000

LinkedIn Draft — Insight (2026-03-28)

One insight that changed how I design systems:

Runbook quality decays silently — and that decay kills MTTR

Runbooks that haven't been run recently are wrong. Not outdated — wrong. The service changed. The tool was deprecated. The endpoint moved. Nobody updated the doc because nobody reads it until 3am. And at 3am, a wrong runbook is worse than no runbook — it sends engineers down confident paths that dead-end.

Runbook decay curve:

Quality
  │▓▓▓▓▓▓▓▓▓▓
  │         ▓▓▓▓▓
  │              ▓▓▓▓
  │                  ▓▓▓▓▓
  │                       ▓▓▓░░░░░░░
  │                             ░░░░░░░░ ← "last validated 8 months ago"
  └────────────────────────────────────▶
  Write   Month 1  Month 3  Month 6  Month 9

The non-obvious part:
→ The highest-leverage runbook improvement isn't better writing — it's a validation date and a quarterly review reminder. A runbook with 'last validated: 2 weeks ago' that's 70% accurate is worth more than a beautifully written one from 8 months ago that's 40% accurate.

My rule:
→ Every runbook gets a 'last validated' date. Anything older than 3 months is assumed broken until proven otherwise. Review is part of the on-call rotation, not optional.

Worth reading:
▸ PagerDuty Incident Response guide — runbook standards and validation cadence
▸ Post-incident review template: 'Did the runbook help, mislead, or was it missing?' (standard question)

https://neeraja-portfolio-v1.vercel.app/insights/runbook-quality-decays-silently-and-that-decay-kills-mttr

What's the version of this that your org gets wrong? Drop it below.

devops #sre #observability #platformengineering

Insight of the Week

Neeraja Khanapure — Sat, 28 Mar 2026 16:06:06 +0000

LinkedIn Draft — Insight (2026-03-28)

Observability is a label strategy problem disguised as a tooling problem

You can’t debug what you can’t slice. Most “noisy dashboards” are really missing ownership labels, consistent dimensions, and SLI intent.

What I’ve seen go wrong:

{{pitfall1}}
{{pitfall2}}

My rule:
Define SLIs first, then design labels that let you isolate (service, env, version, tenant) without blowing up cardinality.

If you want to go deeper: https://neeraja-portfolio-v1.vercel.app/projects/prometheus-scaling

devops #sre #observability #platformengineering

Insight of the Week

Neeraja Khanapure — Fri, 27 Mar 2026 09:39:22 +0000

LinkedIn Draft — Insight (2026-03-27)

Observability is a label strategy problem disguised as a tooling problem

You can’t debug what you can’t slice. Most “noisy dashboards” are really missing ownership labels, consistent dimensions, and SLI intent.

What I’ve seen go wrong:

{{pitfall1}}
{{pitfall2}}

My rule:
Define SLIs first, then design labels that let you isolate (service, env, version, tenant) without blowing up cardinality.

If you want to go deeper: https://neeraja-portfolio-v1.vercel.app/projects/prometheus-scaling

devops #sre #observability #platformengineering

Workflow Deep Dive

Neeraja Khanapure — Tue, 24 Mar 2026 09:36:48 +0000

LinkedIn Draft — Workflow (2026-03-24)

End‑to‑end MLOps retraining loop: reliability is in the guardrails

Auto‑retraining is easy to wire. Making it safe in production is the hard part: data drift, silent label shifts, and rollback semantics.

What usually bites later:

A “better” offline model can degrade live KPIs due to skew (training vs serving features) and traffic shift.
Unversioned data/labels make incident RCA impossible — you can’t reproduce what trained the model.
Promotion without canary + rollback turns retraining into a weekly outage generator.

My default rule:
No model ships without: dataset/version lineage, shadow/canary evaluation, and a one‑click rollback path.

When I’m sanity-checking this, I usually do:

Track dataset + features with DVC/LakeFS + model registry (MLflow/SageMaker Registry) for auditable promotion.
Monitor drift + performance slices with Prometheus/Grafana + alert on trend, not single spikes.

Deep dive (stable link): https://neeraja-portfolio-v1.vercel.app/workflows/resilient-architecture

mlops #aiops #automation #python

Insight of the Week

Neeraja Khanapure — Fri, 20 Mar 2026 09:25:33 +0000

LinkedIn Draft — Insight (2026-03-20)

CI/CD isn’t speed — it’s predictable change under load

Most pipelines fail not because tests are slow, but because rollout risk isn’t modeled (blast radius, rollback, and observability gates).

What I’ve seen go wrong:

{{pitfall1}}
{{pitfall2}}

My rule:
If you can’t explain rollback + SLO gates in one slide, the pipeline is not production‑ready.

If you want to go deeper: https://neeraja-portfolio-v1.vercel.app/insights/cicd-isnt-speed-its-predictable-change-under-load

devops #sre #observability #platformengineering

Claude on AWS Bedrock was throttling requests and the billing dashboard showed zero issues

Neeraja Khanapure — Mon, 16 Mar 2026 18:36:21 +0000

Most teams running Claude on Bedrock watch latency and cost.
Neither shows you when you are about to get throttled.
Claude Sonnet output tokens cost 5x more compute to generate than input tokens to process. AWS counts them at 5x against your TPM quota. Your bill charges for real tokens. Your quota gate reflects real compute.
Your bill shows 100 tokens.
Bedrock counted 500 against your limit.
Throttling hits. Dashboard looks clean.
What AWS just shipped
AWS just released two CloudWatch metrics that fix this blind spot. Both are free, automatic, and already in your AWS/Bedrock CloudWatch namespace. No code changes. No opt-in.
EstimatedTPMQuotaUsage
Real quota consumed per request, burndown multipliers included. Not what you were billed. What Bedrock actually counted against your limit.
TimeToFirstToken
Server side metric. Measures time from request to first Claude response token. Tells you if slowness lives in Bedrock or your own stack. Stops the guessing. Narrows the debug in seconds.
3 alarms worth setting today
80% of TPM limit on EstimatedTPMQuotaUsage
Warning before throttle hits. You get runway instead of a surprise.
P95 threshold on TimeToFirstToken
Catch Claude response degradation before users feel it.
Compare TimeToFirstToken vs InvocationLatency
If TTFT is fine but total latency is high, the problem is output generation not model startup. Narrows the debug surface immediately.
Were you tracking billing tokens thinking that was your quota? Most teams are.
Source: https://aws.amazon.com/blogs/machine-learning/improve-operational-visibility-for-inference-workloads-on-amazon-bedrock-with-new-cloudwatch-metrics-for-ttft-and-estimated-quota-consumption/[](url)

Kubernetes rollouts: promote on SLOs, not on "pods are Ready"

Neeraja Khanapure — Sat, 14 Mar 2026 15:57:32 +0000

Readiness is a local signal. Production impact is global.
Pods can be Ready while your SLO window is already burning.
The failure chain
Rollout shifts traffic fast.
New pods saturate before HPA reacts.
HPA scrape window is 15 to 30 seconds minimum.
P95 latency climbs.
Error rate ticks up.
SLI degrades.
Everything looks healthy. The error budget is draining quietly.
Why "pods are Ready" lies to you
Ready means the container started and passed a health check.
It says nothing about P95 latency, error rate, or whether your SLO slice is holding.
Canary gets stuck green because metrics are too coarse.
No labels, no slices, blast radius stays invisible.
Three resolvers
Pre-scale before the first canary step
Bump replicas before traffic shifts.
HPA catches up from a safe baseline instead of a saturated one.
Match step interval to your HPA scaleUp window
Default stabilization window is 3 minutes.
Check yours with:
bashkubectl get hpa -o yaml
Promoting before that window closes is promoting blind.
Gate steps on SLI health
Wire an AnalysisRun in Argo Rollouts that checks error rate and P95 latency are within SLO bounds before promoting.
If the SLI is still recovering, promotion waits.
The rule
Promote only when the canary holds the SLO slice that matters for a fixed window.
Anything outside that window triggers auto-rollback.
Rollout speed and autoscaler reaction time are tuned independently.
That gap is where error budget burns before anyone pages.

Kubernetes rollouts: promote on SLOs, not on "pods are Ready" | Neeraja Khanapure

Pods are Ready. P95 is climbing. Error rate is ticking. HPA has not moved. The rollout looks healthy. The SLO window is already burning. The exact failure chain and three resolvers that actually work. Pre-scaling, matching step interval to your HPA stabilization window, and gating promotion on SLI health instead of pod status. #kubernetes #sre #devops #platformengineering

linkedin.com

Deep dive: https://neeraja-portfolio-v1.vercel.app/workflows/kubernetes-rollouts-promote-on-slos-not-on-pods-are-ready
What is the step interval on your rollouts right now?