Forem: varun varde

Building Production-Grade Observability: OpenTelemetry + Grafana Stack

varun varde — Tue, 14 Apr 2026 09:05:05 +0000

Stop guessing what's broken in production. Here's a complete, deploy-it-this-week observability stack built on OpenTelemetry and Grafana — the same stack I've deployed for three clients in the last 18 months.

This isn't a toy setup. This is production-grade: traces, metrics, and logs unified under a single pane of glass, with auto-instrumentation for the most common runtimes, alerting that pages on symptoms not causes, and dashboards your non-SRE teammates can actually read.

What you'll build:

OpenTelemetry Collector (gateway mode) for vendor-agnostic telemetry collection
Grafana Tempo for distributed tracing
Prometheus + Grafana Mimir for metrics at scale
Loki for structured log aggregation
Grafana dashboards with pre-built SLO panels
AlertManager rules tied to error budgets

Prerequisites: Kubernetes 1.25+, Helm 3, basic familiarity with YAML. Estimated time: 3–5 hours end to end.

Why OpenTelemetry? The vendor-lock argument settled once and for all

You’ve heard it before: “Just use Datadog.” Then the bill arrives. Or “Use Prometheus alone.” Then you lose traces.

OpenTelemetry (OTel) is the single CNCF standard for generating and exporting telemetry data. Here’s why it wins:

One instrumentation, many backends: Instrument your app once with OTel SDKs. Send to Tempo, Jaeger, Datadog, or New Relic simultaneously.
No vendor lock-in: Your telemetry data remains in your control (S3 for traces, block storage for metrics).
Automatic context propagation: Trace IDs flow seamlessly across services, even across different languages (Java → Python → Node.js).
Future-proof: New backends emerge? Point your OTel Collector there. No code changes.

The bottom line: OTel is the USB-C of observability. Stop writing custom exporters.

Architecture overview: Collector, Backends, Visualization

Here’s what you’re deploying:

[Your App] --(OTLP)--> [OTel Collector (Gateway)] --+--> [Tempo] (traces)
                                                      +--> [Mimir] (metrics)
                                                      +--> [Loki] (logs)
                                                              |
                                                         [Grafana] (visualization)
                                                              |
                                                       [AlertManager] (paging)

OTel Collector (Gateway mode): Receives OTLP from all services. Validates, batches, and routes telemetry. Single ingress point.
Tempo: Object-storage-backed tracing. Cheap, scalable, no indexing costs.
Mimir: Horizontally scalable Prometheus-compatible metrics store.
Loki: Log aggregation with low-cost object storage.
Grafana: Unified UI with Explore, dashboards, and alerting.
AlertManager: Deduplicates, groups, and routes alerts to PagerDuty/Slack.

Storage requirements (minimal): 50GB for Loki, 100GB for Tempo (can use S3/GCS/MinIO), 50GB for Mimir.

Installing the OTel Collector (gateway mode Helm values)

Create otel-collector-values.yaml

mode: deployment   # gateway mode (as opposed to daemonset for agent mode)

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

  processors:
    batch:
      timeout: 1s
      send_batch_size: 1024
    memory_limiter:
      check_interval: 1s
      limit_mib: 512
    attributes:
      actions:
        - key: environment
          value: production
          action: upsert

  exporters:
    otlp/tempo:
      endpoint: "tempo-distributor:4317"
      tls:
        insecure: true
    prometheusremotewrite/mimir:
      endpoint: "http://mimir-distributor:8080/api/v1/push"
    loki:
      endpoint: "http://loki-gateway:3100/loki/api/v1/push"

  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [memory_limiter, batch, attributes]
        exporters: [otlp/tempo]
      metrics:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [prometheusremotewrite/mimir]
      logs:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [loki]

Deploy

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm upgrade --install otel-collector open-telemetry/opentelemetry-collector -f otel-collector-values.yaml

Auto-instrumentation: Java, Python, Node.js, Go

No code changes for traces/metrics/logs. Use OTel's auto-instrumentation agents.

Java (Spring Boot, any JVM app)

ENV JAVA_TOOL_OPTIONS="-javaagent:/otel/opentelemetry-javaagent.jar"
ENV OTEL_SERVICE_NAME=payment-service
ENV OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317

Python (Django, Flask, FastAPI)

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
otel-instrument \
  --service_name checkout-service \
  --exporter_otlp_endpoint http://otel-collector:4317 \
  python app.py

Node.js (Express, NestJS)

npm install @opentelemetry/auto-instrumentations-node
npx opentelemetry-instrument \
  --service_name=api-gateway \
  --exporter_otlp_endpoint=http://otel-collector:4317 \
  node server.js

Go (manual instrumentation required, but minimal)

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
)

func initTracer() {
    exporter, _ := otlptracegrpc.New(ctx, 
        otlptracegrpc.WithEndpoint("otel-collector:4317"),
        otlptracegrpc.WithInsecure())
    // ... standard setup (5 lines)
}

Verify: Check Collector logs for TraceID spans.

Deploying Tempo for distributed tracing

Tempo is designed for cost-effective tracing. It stores traces in object storage (S3/MinIO) and indexes only by trace ID.

tempo-values.yaml

tempo:
  storage:
    trace:
      backend: s3
      s3:
        bucket: tempo-traces
        endpoint: minio.minio:9000
        access_key: "minioadmin"
        secret_key: "minioadmin"
        insecure: true
      pool:
        max_workers: 100
        queue_depth: 10000

  overrides:
    defaults:
      ingestion:
        rate_limit_bytes: 15000000   # 15MB/s
        burst_size_bytes: 20000000

distributor:
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: "0.0.0.0:4317"

Deploy

helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install tempo grafana/tempo -f tempo-values.yaml

Query Tempo from Grafana: Add data source → Tempo → URL: http://tempo-query-frontend:16686

Prometheus + Mimir for long-term metrics storage

Mimir replaces single-instance Prometheus. It provides horizontal scaling, replication, and long-term retention.

mimir-values.yaml

mimir:
  structuredConfig:
    blocks_storage:
      backend: s3
      s3:
        endpoint: minio.minio:9000
        bucket_name: mimir-blocks
        access_key_id: "minioadmin"
        secret_access_key: "minioadmin"
        insecure: true
    ingester:
      ring:
        replication_factor: 3   # for HA
    ruler:
      rule_path: /data/rules
      alertmanager_url: http://alertmanager:9093

  ingester:
    replicas: 3
  distributor:
    replicas: 2
  querier:
    replicas: 2

Deploy

helm upgrade --install mimir grafana/mimir -f mimir-values.yaml

Migrate existing Prometheus data

promtool tsdb create-blocks-from-rules --rules-file=recording-rules.yaml data/

Then point Prometheus remote write to http://mimir-distributor:8080/api/v1/push.

Loki for log aggregation with structured querying

Loki is like Prometheus for logs. It indexes only labels, not full text, making it cheap at scale.

loki-values.yaml

loki:
  storage:
    type: s3
    s3:
      endpoint: minio.minio:9000
      bucketnames: loki-chunks
      access_key_id: "minioadmin"
      secret_access_key: "minioadmin"
      s3forcepathstyle: true
      insecure: true

  schemaConfig:
    configs:
      - from: 2024-01-01
        store: boltdb-shipper
        object_store: s3
        schema: v12
        index:
          prefix: loki_index_
          period: 24h

  limits_config:
    ingestion_rate_mb: 10
    ingestion_burst_size_mb: 20
    max_global_streams_per_user: 10000

  chunk_store_config:
    max_look_back_period: 672h  # 28 days

Deploy

helm upgrade --install loki grafana/loki -f loki-values.yaml

Query example (LogQL)

{namespace="production", app="payment-service"} |= "error" 
| json 
| latency_ms > 500 
| line_format "{{.trace_id}} - {{.message}}"

Grafana: Connecting all three data sources

grafana-values.yaml

datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Prometheus-Mimir
        type: prometheus
        url: http://mimir-query-frontend:8080/prometheus
        access: proxy
        isDefault: true

      - name: Tempo
        type: tempo
        url: http://tempo-query-frontend:16686
        access: proxy
        jsonData:
          tracesToLogs:
            datasourceUid: 'loki'
            tags: ['service.name', 'pod']
          serviceMap:
            enabled: true

      - name: Loki
        type: loki
        url: http://loki-gateway:3100
        access: proxy
        jsonData:
          derivedFields:
            - name: trace_id
              matcherRegex: 'trace_id=(\w+)'
              url: '$${__value.raw}'
              datasourceUid: 'tempo'

dashboardProviders:
  dashboardproviders.yaml:
    apiVersion: 1
    providers:
      - name: 'slo'
        orgId: 1
        folder: 'SLO Dashboards'
        type: file
        options:
          path: /var/lib/grafana/dashboards

Deploy

helm upgrade --install grafana grafana/grafana -f grafana-values.yaml

Test correlation: In Loki, find a log with trace_id=abc123. Click it → jumps to Tempo trace. In Tempo, see affected service → jumps to Mimir metrics for that service.

Building your first SLO dashboard (template included)

Save as slo-dashboard.json and mount into Grafana

{
  "title": "SLO Dashboard - Payment Service",
  "panels": [
    {
      "title": "Availability (30d SLI)",
      "targets": [{
        "expr": "sum(rate(http_requests_total{status!~'5..'}[$__range])) / sum(rate(http_requests_total[$__range]))",
        "legendFormat": "Availability SLI"
      }],
      "thresholds": [
        {"color": "red", "value": null, "op": "lt", "valueType": "absolute", "value": 0.995},
        {"color": "yellow", "value": null, "op": "lt", "valueType": "absolute", "value": 0.999},
        {"color": "green", "value": null, "op": "gte", "valueType": "absolute", "value": 0.999}
      ]
    },
    {
      "title": "Error Budget Remaining (30d)",
      "targets": [{
        "expr": "(1 - (sum(rate(http_requests_total{status=~'5..'}[30d])) / sum(rate(http_requests_total[30d])))) / 0.999",
        "legendFormat": "Budget remaining"
      }],
      "fieldConfig": {
        "defaults": {
          "unit": "percentunit",
          "min": 0,
          "max": 1,
          "color": {"mode": "thresholds"},
          "thresholds": [
            {"color": "red", "value": null, "op": "lt", "value": 0.7},
            {"color": "yellow", "value": null, "op": "lt", "value": 0.9},
            {"color": "green", "value": null, "op": "gte", "value": 0.9}
          ]
        }
      }
    },
    {
      "title": "Latency P99 (30d SLI)",
      "targets": [{
        "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[$__range])) by (le))",
        "legendFormat": "P99 latency"
      }]
    }
  ]
}

SLO math explained

Availability target: 99.9% → error budget = 0.1% of requests can fail.
Budget remaining: (actual_availability - target) / (1 - target) → 1.0 means on track, 0 means exhausted.

AlertManager: Alerting on symptoms, not causes

Bad alert: "CPU on pod payment-7d8f9 is 92%" (cause)
Good alert: "Payment service error budget exhausted" (symptom)

alertmanager-config.yaml

route:
  group_by: ['alertname', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  receiver: 'pagerduty-critical'
  routes:
    - match:
        severity: critical
      receiver: pagerduty-critical
      continue: false
    - match:
        severity: warning
      receiver: slack-warnings

receivers:
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: <your-pd-key>
        severity: critical
  - name: 'slack-warnings'
    slack_configs:
      - api_url: <webhook>
        channel: '#alerts-warning'

Prometheus alerting rule example (slo-alerts.yaml)

groups:
  - name: slo
    rules:
      - alert: ErrorBudgetExhausted
        expr: |
          (1 - (sum(rate(http_requests_total{status=~"5.."}[30d])) 
          / sum(rate(http_requests_total[30d])))) / 0.999 < 0.2
        for: 5m
        labels:
          severity: critical
          service: "{{$labels.service}}"
        annotations:
          summary: "Error budget for {{$labels.service}} is below 20%"
          description: "Remaining budget: {{$value | humanizePercentage}}"

Deploy

kubectl create configmap alertmanager-config --from-file=alertmanager.yaml=alertmanager-config.yaml
helm upgrade --install prometheus prometheus-community/prometheus \
  --set alertmanager.enabled=true \
  --set alertmanager.configFromSecret=alertmanager-config

The 3 dashboards every on-call engineer needs

Stop building 50-panel dashboards. Start with these three.

Dashboard 1: Service Health (RED method)

Rate (requests per second) per endpoint
Errors (5xx rate, grouped by status code)
Duration (P50, P95, P99 latency)
Saturation (CPU/memory per pod, queue depth)

PromQL snippets

# Rate
sum(rate(http_requests_total[1m])) by (service, endpoint)

# Error ratio
sum(rate(http_requests_total{status=~"5.."}[1m])) / sum(rate(http_requests_total[1m]))

# P99 latency
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))

Dashboard 2: Trace Explorer

Top 10 slowest traces in last hour
Trace heatmap (duration vs. timestamp)
Service dependency graph (from Tempo service graph)
High-error traces panel (filter by status.error=true)

Dashboard 3: The "Burndown" Chart

Error budget remaining (daily trend line)
SLO burn rate (1h, 6h, 24h windows)
Multi-burn alert status (green/yellow/red)
Top offending services by error budget consumption

Why this works: On-call opens Dashboard 1 → sees elevated latency → clicks a trace in Dashboard 2 → finds slow database query → checks Dashboard 3 to decide if paging SREs is urgent.

Final checklist for production readiness

Before you sleep soundly:

Ingestion testing: curl a test span/metric/log through the Collector.
Retention: Set Mimir 30d, Tempo 14d, Loki 30d (adjust to compliance).
Auth: Add Grafana OAuth (Google/GitHub) and basic auth for Mimir/Loki ingesters.
Backups: Object storage (MinIO/S3) should have versioning enabled.
Alert testing: Silence a service, verify PagerDuty gets the page.
Runbook: Link each alert to a Confluence doc (e.g., "ErrorBudgetExhausted → https://wiki/runbooks/slo").

What’s next? Add OpenTelemetry for your database (PostgreSQL, Redis, MongoDB) using OTel collector receivers. Or add synthetic monitoring with Blackbox exporter.

You now have the same stack that cost my clients $0/month (excluding storage) instead of $15k/month for Datadog. Ship it.

From GitHub Actions to ArgoCD: A Complete GitOps Migration Guide

varun varde — Tue, 07 Apr 2026 13:33:40 +0000

I've done this migration three times now. Each time, I learn something new about what NOT to do. This guide is the documentation I wish I'd had the first time: the complete migration from a multi-stage GitHub Actions deployment pipeline to a fully declarative GitOps workflow with ArgoCD including the parts every other tutorial glosses over.

What you'll build: Production ready ArgoCD installation, ApplicationSet for multi cluster management, Image Updater for automated rollouts, RBAC configuration for multi-team environments, Rollback procedures that work under pressure.

Prerequisites: Kubernetes cluster (1.24+), kubectl, Helm 3, existing GitHub Actions CI pipeline. Estimated time: 4-6 hours for learning setup (30 days for production migration).

Phase 1: Assessment — Mapping Your Existing GitHub Actions Pipeline

Every successful migration begins with clarity. Before introducing GitOps, the existing CI/CD topology must be dissected—methodically, not casually.

Start by inventorying workflows

# .github/workflows/deploy.yml
name: Deploy Service

on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: docker build -t myapp .

  deploy:
    needs: build
    steps:
      - run: kubectl apply -f k8s/

This pipeline reveals implicit assumptions:

Deployment is tightly coupled with CI
Kubernetes manifests are applied imperatively
No state reconciliation exists

Map each component

Component	Purpose
Build Step	Artifact creation
Deploy Step	Cluster mutation
Secrets	Runtime configuration

The goal is not replication but transformation. GitOps demands declarative intent, not procedural execution.

Phase 2: ArgoCD Installation (Production-Grade Helm Values)

A default installation is rarely sufficient. Production demands rigor secure ingress, RBAC, and resource constraints.

Install ArgoCD using Helm

helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd -n argocd --create-namespace -f values.yaml

A hardened values.yaml might resemble

server:
  service:
    type: ClusterIP

  ingress:
    enabled: true
    hosts:
      - argocd.example.com

configs:
  params:
    server.insecure: false

  cm:
    url: https://argocd.example.com

redis:
  enabled: true

controller:
  resources:
    limits:
      cpu: 500m
      memory: 512Mi

Security is not ornamental. It is foundational. Misconfigured access at this stage invites future instability.

Phase 3: Structuring Your GitOps Repository

Git becomes the single source of truth. Structure, therefore, becomes paramount.

A canonical layout

gitops-repo/
├── apps/
│   ├── payments/
│   ├── auth/
├── environments/
│   ├── dev/
│   ├── staging/
│   └── prod/
└── infrastructure/
    ├── ingress/
    └── monitoring/

Each directory encapsulates intent.

Example application manifest

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-service
spec:
  source:
    repoURL: https://github.com/org/gitops-repo
    path: apps/payments
    targetRevision: HEAD
  destination:
    server: https://kubernetes.default.svc
    namespace: payments
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Declarative. Predictable. Auditable.

Phase 4: ApplicationSet Configuration for Multi-Team

At scale, manually defining applications becomes untenable. ApplicationSet introduces dynamic orchestration.

A Git generator example

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: team-apps
spec:
  generators:
    - git:
        repoURL: https://github.com/org/gitops-repo
        revision: HEAD
        directories:
          - path: apps/*
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/org/gitops-repo
        targetRevision: HEAD
        path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{path.basename}}'

Teams simply add a folder. The system reacts autonomously.

This is where GitOps begins to feel almost sentient responsive, adaptive, yet deterministic.

Phase 5: Migrating Your First Service (with Rollback Plan)

Start small. Select a non-critical service. Minimize risk while maximizing insight.

Step 1: Remove Deploy Step from CI

# Keep build, remove kubectl apply
- run: docker build -t myapp .

Step 2: Push Kubernetes Manifests to GitOps Repo

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 2

Step 3: Let ArgoCD Sync

ArgoCD reconciles desired state with actual state.

Rollback Strategy

Rollback becomes trivial

git revert <commit-hash>
git push origin main

No manual intervention. No frantic patching. Just version control.

Phase 6: Image Updater for CI Integration

CI still builds images. But deployment is decoupled.

ArgoCD Image Updater bridges the gap

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: myapp=repo/myapp
    argocd-image-updater.argoproj.io/update-strategy: latest

Pipeline pushes image

docker push repo/myapp:1.2.3

Image Updater modifies Git

image: repo/myapp:1.2.3

ArgoCD syncs automatically.

Elegant. Autonomous.

Phase 7: RBAC and Multi-Tenant Configuration

As teams proliferate, governance becomes essential.

Define roles

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
data:
  policy.csv: |
    p, role:dev, applications, get, */*, allow
    p, role:dev, applications, sync, */*, allow
    g, team-dev, role:dev

Namespace isolation

spec:
  destination:
    namespace: team-a

Each team operates within boundaries autonomous yet controlled.

Phase 8: Observability for Your GitOps Pipelines

Visibility transforms guesswork into certainty.

Enable metrics

controller:
  metrics:
    enabled: true

Scrape with Prometheus

- job_name: 'argocd'
  static_configs:
    - targets: ['argocd-metrics:8082']

Key metrics

Sync status
Reconciliation duration
Drift detection

Logs provide narrative

kubectl logs deployment/argocd-application-controller

Without observability, GitOps becomes a black box. With it, a glass box.

The 5 Things No Tutorial Mentions (Learned the Hard Way)

1. Git Becomes Your Bottleneck

Frequent commits can overwhelm repositories. Optimize branching strategies. Avoid unnecessary churn.

2. Drift Happens More Than Expected

Manual changes in clusters still occur. ArgoCD corrects them—but not always instantly. Expect transient inconsistencies.

3. Secrets Are a Persistent Challenge

Plain YAML is insufficient. Integrate tools like sealed secrets or external secret managers.

4. ApplicationSet Can Spiral into Complexity

Dynamic generation is powerful—but dangerous. Poor templates create cascading misconfigurations.

5. Human Behavior Is the Hardest Problem

Engineers accustomed to imperative deployments resist change. Training is not optional. It is essential.

Migrating to ArgoCD is not merely a tooling shift. It is a philosophical transition—from imperative control to declarative intent.

The rewards are substantial. Stability. Traceability. Confidence.

But the journey demands discipline. And a willingness to relinquish old habits in favor of something far more resilient.

Zero Downtime Cloud Migration: The 6-Phase Playbook

varun varde — Thu, 26 Mar 2026 08:27:40 +0000

Phase 1 is never 'lift and shift.' Here's the framework that keeps production stable throughout the entire move.

After leading migrations for organizations ranging from 200 to 4,000 engineers, I've distilled it into 6 phases that keep production alive and teams sane.

Phase 1: Discovery & Dependency Mapping
Before touching a single VM, audit everything.

Inventory all services: apps, databases, middleware, integrations
Map inter service dependencies including the undocumented ones (ask the engineers who've been there longest)
Tag everything by criticality, data sensitivity, migration complexity
Identify the "spiders in the web" services everything else depends on

Tools: AWS Application Discovery Service, Cloudamize, or a structured spreadsheet + interviews.

Phase 2: Cloud Foundation & Landing Zone
Build the platform before you migrate anything.

Set up your VPC architecture (hub spoke or flat decide now)
Implement IAM roles, SCPs, and guardrails
Deploy centralized logging, monitoring, and alerting
Establish your IaC standard (Terraform modules, Pulumi stacks — standardize before day one)
Set cost budgets and alerts BEFORE anything runs
Never skip this. Teams that migrate first and "sort out governance later" always regret it.

Phase 3: Pilot Migration (Noncritical workloads)
Start small. Build confidence.

Choose a non-critical, low traffic service
Run full migration lifecycle: move → validate → monitor → optimize
Document everything. This becomes your runbook for every subsequent migration
Identify gaps in tooling and process while the stakes are low

Phase 4: Wave Based Migration
Organize workloads into migration waves by complexity:

Wave 1: Stateless apps (easiest)
Wave 2: Stateful apps with managed DB alternatives
Wave 3: Complex integrations and legacy services
Wave 4 (optional): Services that require refactoring before migration

Run each wave over 2–4-week sprints. Include a 2-week stabilization window after each wave before proceeding.

Phase 5: Cutover & Traffic Management
Zero-downtime means dual running, not big-bang.

Use feature flags and DNS-based traffic shifting (Route 53 weighted routing or equivalent)
Implement a circuit breaker to instantly roll back traffic if error rates spike
Keep on prem running in parallel for 30–60 days post cutover
Don't decommission anything until you've run at least one full billing cycle on cloud

Phase 6: Optimize & Decommission
The migration isn't over when the apps are running.

Right-size instances based on real usage data (wait at least 2 weeks)
Implement autoscaling where you were running fixed capacity
Set up FinOps dashboards cloud spend will be surprising at first
Decommission on prem systematically, not in a rush
Run a migration retrospective with every team involved

The Rule I Never Break
No service goes to production without observability in place (logs, metrics, traces), a documented runbook, an on-call rotation assigned, and a tested rollback procedure.

Save for your next migration kickoff

Terraform Modules That Actually Scale: Patterns from 20 Years

varun varde — Wed, 18 Mar 2026 08:47:03 +0000

Provisioning was once an artisanal craft defined by shell scripts and fragile golden images. We've since pivoted to the declarative, idempotent paradigm of Terraform. But scaling isn't just about resource count it’s about managing the cognitive load of massive environments. At some point, the question stops being how do I write HCL? and becomes how do I stop this complexity from collapsing under its own weight?

The Monolith Trap: Why Large State Files Kill Velocity

The Mega State is the ultimate architectural debt. Dumping your VPC, EKS cluster, and RDS instances into one root module creates a catastrophic blast radius. One syntax error in a security group shouldn't lock the state file for your entire production database. Scaling demands decoupling. Infrastructure components must be treated as independent lifecycles. Networking is slow and steady app pods are ephemeral. Isolate them to minimize plan times and risk.

Strict Typing and Input Orthogonality

Ditch type = any. It’s lazy and dangerous. High utility modules require strict object validation to fail fast. If the calling code sends a malformed schema, the plan should die immediately not thirty minutes into a deployment.

variable "cluster_config" {
  description = "EKS spec"
  type = object({
    version    = string
    vpc_id     = string
    subnet_ids = list(string)
    node_groups = map(object({
      instance_types = list(string)
      capacity_type  = string
    }))
  })

  validation {
    condition     = contains(["1.28", "1.29", "1.30"], var.cluster_config.version)
    error_message = "K8s version restricted by organizational compliance."
  }
}

Compositional Strategy: Favoring Layering over Nesting

Nesting modules inside modules leads to Prop Drilling a nightmare where a variable is passed through five layers of code just to reach a single resource. It's brittle.

Layering is the professional choice. Keep modules flat and use a Composition Root to stitch outputs to inputs. This keeps the logic readable and the dependencies explicit.

Cross Account Orchestration via Provider Aliasing

Enterprise scale means multiple regions and hundreds of AWS accounts. Never hardcode provider blocks inside a module; it makes them non portable. Use Provider Aliases. This allows you to instantiate the same module across different regions within a single run.

# Instance in US-East
module "s3_east" {
  source = "./modules/s3"
  providers = {
    aws = aws.us-east-1
  }
}

# Instance in US-West
module "s3_west" {
  source = "./modules/s3"
  providers = {
    aws = aws.us-west-2
  }
}

Immutable Versioning and Private Registries

Local file paths are for hobbyists. In production, referencing a local path means every save on your laptop could theoretically break a CI/CD pipeline. Scaling requires immutable versioning. Use Git tags or a Private Registry. It’s the only way to ensure Production stays on v1.2.0 while Dev experiments with v2.0.0 beta.

Automated Validation: Terratest and OPA Integration

If you aren't testing, you're guessing. For core modules, Terratest is the standard. It spins up real resources, pings them to ensure they work, and tears them down. Couple this with Open Policy Agent (OPA) to programmatically block anyone from creating an unencrypted volume or a public S3 bucket before the code even leaves the PR.

Day 2 Operations: Refactoring with moved Blocks

Refactoring used to mean terraform state mv commands that risked corrupting the remote backend. Now, we have the moved block. It allows you to move resources into modules without triggering a "destroy and recreate" cycle.

moved {
  from = aws_instance.legacy_app
  to   = module.app_cluster.aws_instance.this[0]
}

This is how you turn a tangled mess of HCL into a robust ecosystem. It’s about predictability, isolation, and strict interfaces.