Forem: InstaDevOps

AWS VPC Networking: Transit Gateway, Peering & PrivateLink

InstaDevOps — Wed, 06 May 2026 13:48:03 +0000

AWS VPC Networking: Subnets, NAT Gateways, Transit Gateway, and PrivateLink

AWS networking is the foundation that everything else sits on, yet it is the area where most teams accumulate the most technical debt. A poorly designed VPC leads to security gaps, connectivity issues, and painful migrations later. Getting your network architecture right from the start - proper CIDR planning, subnet tiers, and connectivity patterns - saves enormous headaches as you scale.

Every production VPC should have three subnet tiers across multiple availability zones: public subnets for load balancers and bastion hosts, private subnets for application workloads, and isolated subnets for databases with no internet access. NAT Gateways provide outbound internet access for private subnets - deploy one per AZ for high availability, but be aware they are one of the most expensive networking components. Use VPC endpoints (Gateway endpoints for S3/DynamoDB, Interface endpoints for other services) to keep traffic on the AWS backbone and reduce NAT Gateway costs.

For multi-VPC architectures, Transit Gateway replaces the mesh of VPC peering connections that becomes unmanageable beyond 3-4 VPCs. Transit Gateway acts as a hub that all VPCs connect to, with route tables controlling which VPCs can communicate. PrivateLink exposes services across VPCs or to customers without traversing the public internet - essential for SaaS architectures and shared services platforms. Plan your CIDR ranges carefully to avoid overlaps, and use AWS RAM to share subnets across accounts in an AWS Organizations setup.

Need help designing your AWS network? InstaDevOps architects production-grade VPC layouts for startups and scale-ups. Book a free consultation.

Terraform State Management: Remote Backends, Locking & Recovery

InstaDevOps — Tue, 05 May 2026 13:47:55 +0000

Terraform State Management: Remote Backends, Locking, and State Surgery

Terraform state is the single most critical file in your infrastructure-as-code workflow. It maps your Terraform configuration to real-world resources, and corrupting or losing it means Terraform loses track of everything it manages. Yet many teams start with local state files, no locking, and no backup strategy - a recipe for disaster the moment two people run terraform apply simultaneously.

Remote backends solve the fundamentals: store state in S3 with DynamoDB locking, or use Terraform Cloud for managed state with built-in versioning. Every remote backend should have encryption at rest, versioning enabled for rollback, and a DynamoDB table for state locking to prevent concurrent modifications. Structure your state files by environment and component - a single monolithic state file for your entire infrastructure creates blast radius problems and slows down every plan and apply.

State surgery is the skill you need when things go wrong. terraform state mv renames resources without destroying them. terraform state rm removes resources from state without deleting them from your cloud provider. terraform import brings existing infrastructure under Terraform management. These commands are dangerous - always back up your state file before running them. For large-scale refactoring, terraform state pull and terraform state push let you manipulate state as JSON, but treat this as a last resort.

Need help with your Terraform setup? InstaDevOps specializes in production-grade infrastructure as code. Book a free consultation to discuss your IaC strategy.

Kubernetes Autoscaling: HPA, VPA, and KEDA Deep Dive

InstaDevOps — Mon, 04 May 2026 13:47:52 +0000

Kubernetes Autoscaling Deep Dive: HPA, VPA, KEDA, and Cluster Autoscaler

Kubernetes autoscaling is not a single feature - it is a layered system where four components work together to match your infrastructure to actual demand. Getting autoscaling wrong means either wasting money on idle resources or dropping traffic during load spikes. Understanding how HPA, VPA, KEDA, and the Cluster Autoscaler interact is essential for any production Kubernetes deployment.

The Horizontal Pod Autoscaler (HPA) scales the number of pod replicas based on CPU, memory, or custom metrics. The Vertical Pod Autoscaler (VPA) adjusts resource requests and limits for individual containers based on observed usage - critical for right-sizing workloads you have not profiled yet. KEDA (Kubernetes Event-Driven Autoscaling) extends HPA with scalers for external event sources like SQS queue depth, Kafka consumer lag, or Prometheus queries, enabling scale-to-zero for workloads that do not need to run continuously. The Cluster Autoscaler adds or removes nodes when pods cannot be scheduled due to insufficient cluster capacity.

The key to effective autoscaling is layering these components correctly. Run VPA in recommendation mode first to establish baseline resource requests. Configure HPA with appropriate metrics and thresholds - CPU utilization of 60-70% is a good starting point. Use KEDA for event-driven workloads like queue processors. And ensure Cluster Autoscaler is configured with appropriate node groups and scale-down policies to avoid unnecessary churn.

Struggling with Kubernetes scaling? InstaDevOps helps teams implement autoscaling strategies that balance performance and cost. Book a free consultation.

HashiCorp Vault: Production Secrets Management Guide

InstaDevOps — Sun, 03 May 2026 13:47:49 +0000

HashiCorp Vault for DevOps: Dynamic Secrets, PKI, and Zero-Trust Infrastructure

Static secrets are a ticking time bomb. Hardcoded API keys, long-lived database passwords stored in environment variables, and shared credentials passed around in Slack channels - these are the security gaps that attackers exploit every day. HashiCorp Vault solves this by providing a centralized secrets management platform that generates short-lived, dynamic credentials on demand.

Vault's power lies in its secrets engines. The database secrets engine generates unique credentials for each application instance with automatic expiration. The PKI engine issues TLS certificates programmatically, eliminating manual certificate management. The AWS engine creates temporary IAM credentials scoped to exactly the permissions each service needs. Combined with Vault Agent for automatic secret injection and renewal, you can build infrastructure where no secret lives longer than it needs to.

Implementing zero-trust with Vault means every service authenticates independently - whether through Kubernetes service accounts, AWS IAM roles, or AppRole for CI/CD pipelines. Vault's audit log tracks every secret access, giving you a complete trail for compliance. The transit engine handles encryption-as-a-service without exposing keys to applications. For teams running Kubernetes, the Vault CSI provider and sidecar injector make integration seamless.

Need help securing your infrastructure? At InstaDevOps, we implement production-grade secrets management with HashiCorp Vault. Book a free consultation to discuss your security posture.

Microservices Communication: gRPC vs REST, Message Queues, and Sagas

InstaDevOps — Sat, 02 May 2026 13:47:45 +0000

Introduction

The moment you split a monolith into microservices, communication becomes your biggest challenge. In a monolith, a function call is nanoseconds, type-safe, and transactional. In microservices, every interaction is a network call that can fail, timeout, arrive out of order, or silently duplicate. The communication patterns you choose determine your system's reliability, latency, and operational complexity.

This guide covers the practical decisions you face when designing microservice communication: synchronous vs asynchronous, gRPC vs REST, choosing the right message broker, implementing the saga pattern for distributed transactions, and building circuit breakers to prevent cascade failures.

Synchronous vs Asynchronous Communication

The first architectural decision is whether services communicate synchronously (request-response) or asynchronously (event-driven). Most systems need both.

Synchronous (request-response): Service A sends a request to Service B and waits for a response. Use for operations where the caller needs an immediate answer.

User → API Gateway → Order Service → Inventory Service (check stock)
                                    ← stock confirmed
                     ← order created
← 201 Created

Asynchronous (event-driven): Service A publishes an event and moves on. Other services consume the event whenever they are ready. Use for operations where the caller does not need an immediate answer, or when you want to decouple services.

Order Service → publishes "OrderCreated" event → Message Queue
                                                    ├→ Payment Service (processes payment)
                                                    ├→ Notification Service (sends email)
                                                    └→ Analytics Service (records metric)

When to use synchronous:

User-facing requests that need an immediate response
Data reads where you need the latest state
Simple request-response queries between two services

When to use asynchronous:

Operations that can happen in the background (email, analytics, audit logs)
When you need to fan out to multiple consumers
When services have different availability requirements
When you need to handle traffic spikes (the queue acts as a buffer)

The biggest mistake teams make is going 100% synchronous. If your order service synchronously calls payment, inventory, notification, and analytics services in sequence, a slowdown in any one of them degrades the entire order flow. Move non-critical steps to async processing.

gRPC vs REST

For synchronous service-to-service communication, you are choosing between REST (HTTP/JSON) and gRPC (HTTP/2, Protocol Buffers).

REST is the default because it is simple, well-understood, and debuggable with curl:

// Express.js REST endpoint
app.get('/api/inventory/:productId', async (req, res) => {
  const stock = await db.query(
    'SELECT quantity FROM inventory WHERE product_id = $1',
    [req.params.productId]
  );
  res.json({ productId: req.params.productId, quantity: stock.rows[0]?.quantity || 0 });
});

// Calling it from another service
const response = await fetch('http://inventory-service:3000/api/inventory/SKU-123');
const data = await response.json();

gRPC uses Protocol Buffers for serialization and HTTP/2 for transport, giving you strong typing, smaller payloads, and bidirectional streaming:

// inventory.proto
syntax = "proto3";

package inventory;

service InventoryService {
  rpc CheckStock (StockRequest) returns (StockResponse);
  rpc WatchStock (StockRequest) returns (stream StockUpdate);  // Server streaming
}

message StockRequest {
  string product_id = 1;
}

message StockResponse {
  string product_id = 1;
  int32 quantity = 2;
  bool in_stock = 3;
}

message StockUpdate {
  string product_id = 1;
  int32 quantity = 2;
  string timestamp = 3;
}

// gRPC server (Node.js)
const grpc = require('@grpc/grpc-js');
const protoLoader = require('@grpc/proto-loader');

const packageDef = protoLoader.loadSync('inventory.proto');
const proto = grpc.loadPackageDefinition(packageDef);

const server = new grpc.Server();
server.addService(proto.inventory.InventoryService.service, {
  checkStock: async (call, callback) => {
    const stock = await db.query(
      'SELECT quantity FROM inventory WHERE product_id = $1',
      [call.request.product_id]
    );
    callback(null, {
      product_id: call.request.product_id,
      quantity: stock.rows[0]?.quantity || 0,
      in_stock: (stock.rows[0]?.quantity || 0) > 0,
    });
  },
});

server.bindAsync('0.0.0.0:50051', grpc.ServerCredentials.createInsecure(), () => {
  console.log('gRPC server running on port 50051');
});

Choose REST when:

External-facing APIs (browsers, mobile apps, third parties)
Simple CRUD operations
You want maximum debuggability
Team is not familiar with protobuf

Choose gRPC when:

Internal service-to-service communication with high call volume
You need streaming (real-time feeds, file uploads)
Latency-sensitive paths where the ~30% serialization speedup matters
You want strict API contracts enforced at compile time

In practice, many teams use REST for their public API and gRPC for internal service mesh communication.

Choosing a Message Broker

For asynchronous communication, you need a message broker. The three most common choices are SQS, RabbitMQ, and Kafka, and they serve different use cases.

Amazon SQS - Managed queue, simplest to operate, no infrastructure to manage:

const { SQSClient, SendMessageCommand, ReceiveMessageCommand, DeleteMessageCommand } = require('@aws-sdk/client-sqs');

const sqs = new SQSClient({ region: 'us-east-1' });
const QUEUE_URL = 'https://sqs.us-east-1.amazonaws.com/123456789/order-events';

// Producer: publish event
await sqs.send(new SendMessageCommand({
  QueueUrl: QUEUE_URL,
  MessageBody: JSON.stringify({
    eventType: 'OrderCreated',
    orderId: 'ord_abc123',
    userId: 'usr_456',
    total: 99.99,
    timestamp: new Date().toISOString(),
  }),
  MessageGroupId: 'ord_abc123',  // FIFO queue: ensures ordering per order
  MessageDeduplicationId: `order-created-ord_abc123`,  // Prevents duplicates
}));

// Consumer: poll for events
async function pollMessages() {
  while (true) {
    const response = await sqs.send(new ReceiveMessageCommand({
      QueueUrl: QUEUE_URL,
      MaxNumberOfMessages: 10,
      WaitTimeSeconds: 20,  // Long polling
      VisibilityTimeout: 60,
    }));

    for (const message of response.Messages || []) {
      const event = JSON.parse(message.Body);
      await processEvent(event);

      await sqs.send(new DeleteMessageCommand({
        QueueUrl: QUEUE_URL,
        ReceiptHandle: message.ReceiptHandle,
      }));
    }
  }
}

RabbitMQ - Feature-rich broker with flexible routing (exchanges, bindings, dead letter queues):

Best when you need complex routing patterns (topic-based, headers-based), priority queues, or delayed messages. More operational overhead than SQS since you manage the broker yourself.

Apache Kafka - Distributed log for high-throughput event streaming:

Best when you need event replay (consumers can re-read history), very high throughput (millions of events/second), or multiple consumer groups reading the same stream independently. Most complex to operate.

Decision matrix:

Requirement	SQS	RabbitMQ	Kafka
Zero ops overhead	Yes	No	No
Message ordering	FIFO queues	Per queue	Per partition
Event replay	No	No	Yes
Complex routing	No	Yes	No (use streams)
Throughput	High	Medium	Very high
Latency	~20ms	~1ms	~5ms
Cost at low volume	Cheapest	Moderate	Expensive

For most startups: start with SQS. You can always migrate to Kafka later when you actually need event replay or million-message-per-second throughput.

The Saga Pattern for Distributed Transactions

In a monolith, creating an order means a single database transaction: deduct inventory, charge payment, create order - all or nothing. In microservices, each step is a different service with its own database. You cannot use a distributed transaction (2PC) because it does not scale and couples all services together.

The saga pattern breaks a distributed transaction into a sequence of local transactions, each publishing an event that triggers the next step. If any step fails, compensating transactions undo the previous steps.

Choreography-based saga (each service reacts to events):

OrderService                PaymentService              InventoryService
    │                            │                            │
    ├─ Create order (PENDING)    │                            │
    ├─ Publish "OrderCreated" ──►│                            │
    │                            ├─ Charge payment            │
    │                            ├─ Publish "PaymentCharged" ─►│
    │                            │                            ├─ Reserve stock
    │                            │                            ├─ Publish "StockReserved"
    │◄───────────────────────────┼────────────────────────────┤
    ├─ Update order → CONFIRMED  │                            │
    │                            │                            │
    │   --- If payment fails --- │                            │
    │                            ├─ Publish "PaymentFailed" ──►│
    │◄───────────────────────────┤                            ├─ Release stock
    ├─ Update order → CANCELLED  │                            │

Orchestration-based saga (a central coordinator manages the flow):

// saga-orchestrator.js
class CreateOrderSaga {
  constructor(orderService, paymentService, inventoryService) {
    this.steps = [
      {
        execute: (data) => inventoryService.reserveStock(data.items),
        compensate: (data) => inventoryService.releaseStock(data.items),
      },
      {
        execute: (data) => paymentService.charge(data.userId, data.total),
        compensate: (data) => paymentService.refund(data.paymentId),
      },
      {
        execute: (data) => orderService.confirmOrder(data.orderId),
        compensate: (data) => orderService.cancelOrder(data.orderId),
      },
    ];
  }

  async execute(orderData) {
    const completedSteps = [];

    for (const step of this.steps) {
      try {
        const result = await step.execute(orderData);
        orderData = { ...orderData, ...result };
        completedSteps.push(step);
      } catch (error) {
        console.error(`Saga step failed: ${error.message}. Compensating...`);
        // Compensate in reverse order
        for (const completed of completedSteps.reverse()) {
          try {
            await completed.compensate(orderData);
          } catch (compError) {
            console.error(`Compensation failed: ${compError.message}`);
            // Alert on-call - manual intervention needed
          }
        }
        throw new Error(`Order saga failed: ${error.message}`);
      }
    }

    return orderData;
  }
}

Choreography vs orchestration: Use choreography when you have 2-3 services in the saga and the flow is simple. Use orchestration when you have 4+ services or complex branching logic. Orchestration is easier to understand, debug, and monitor.

Circuit Breakers

When a downstream service is failing, you do not want every request to wait for a timeout. A circuit breaker detects failures and short-circuits requests to the failing service, returning an error immediately or falling back to a cached response.

// circuit-breaker.js
class CircuitBreaker {
  constructor(options = {}) {
    this.failureThreshold = options.failureThreshold || 5;
    this.resetTimeout = options.resetTimeout || 30000;  // 30 seconds
    this.failureCount = 0;
    this.lastFailureTime = null;
    this.state = 'CLOSED';  // CLOSED = normal, OPEN = failing, HALF_OPEN = testing
  }

  async execute(fn, fallback) {
    if (this.state === 'OPEN') {
      if (Date.now() - this.lastFailureTime > this.resetTimeout) {
        this.state = 'HALF_OPEN';
      } else {
        if (fallback) return fallback();
        throw new Error('Circuit breaker is OPEN');
      }
    }

    try {
      const result = await fn();
      if (this.state === 'HALF_OPEN') {
        this.state = 'CLOSED';
        this.failureCount = 0;
      }
      return result;
    } catch (error) {
      this.failureCount++;
      this.lastFailureTime = Date.now();

      if (this.failureCount >= this.failureThreshold) {
        this.state = 'OPEN';
        console.warn('Circuit breaker tripped to OPEN');
      }

      if (fallback) return fallback();
      throw error;
    }
  }
}

// Usage
const inventoryBreaker = new CircuitBreaker({
  failureThreshold: 3,
  resetTimeout: 15000,
});

async function checkStock(productId) {
  return inventoryBreaker.execute(
    // Primary: call inventory service
    () => fetch(`http://inventory-service:3000/api/stock/${productId}`).then(r => {
      if (!r.ok) throw new Error(`Inventory service returned ${r.status}`);
      return r.json();
    }),
    // Fallback: return cached/default value
    () => ({ productId, quantity: null, in_stock: true, source: 'fallback' })
  );
}

In production, use a library like opossum for Node.js or rely on your service mesh (Istio, Linkerd) to handle circuit breaking at the infrastructure level:

# Istio DestinationRule with circuit breaking
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: inventory-service
spec:
  host: inventory-service
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        h2UpgradePolicy: DEFAULT
        http1MaxPendingRequests: 50
        http2MaxRequests: 100
        maxRequestsPerConnection: 10
    outlierDetection:
      consecutive5xxErrors: 3
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

Observability for Distributed Communication

Debugging microservice communication without proper observability is like debugging a distributed system blindfolded. Implement these three pillars:

Distributed tracing - propagate trace IDs across service boundaries:

// Using OpenTelemetry
const { trace, context, propagation } = require('@opentelemetry/api');

// When making an outbound request, propagate context
async function callService(url, data) {
  const headers = {};
  propagation.inject(context.active(), headers);

  return fetch(url, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      ...headers,  // Includes traceparent header
    },
    body: JSON.stringify(data),
  });
}

Structured logging - include correlation IDs in every log line:

// Every log entry includes the trace/request ID
logger.info('Processing order', {
  traceId: span.spanContext().traceId,
  orderId: order.id,
  service: 'order-service',
  action: 'create_order',
  duration_ms: Date.now() - startTime,
});

Metrics - track request rate, error rate, and latency (the RED method):

// Prometheus metrics
const httpRequestDuration = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests',
  labelNames: ['method', 'route', 'status', 'target_service'],
  buckets: [0.01, 0.05, 0.1, 0.5, 1, 5],
});

Need Help with Your DevOps?

Designing and implementing microservices communication patterns - from choosing the right broker to implementing sagas and circuit breakers - requires deep infrastructure expertise. At InstaDevOps, we help startups and SMBs build reliable, scalable microservice architectures - starting at $2,999/mo.

Book a free 15-minute consultation to discuss your microservices architecture and communication challenges.

Backup and Disaster Recovery: RTO/RPO Planning and AWS Backup

InstaDevOps — Fri, 01 May 2026 13:47:40 +0000

Introduction

Every startup says they have backups. Very few have actually tested restoring from them. Even fewer have a documented disaster recovery plan with defined recovery objectives. Then something goes wrong - a developer runs a destructive migration against production, a region goes down, ransomware encrypts your EBS volumes - and the team discovers their "backup strategy" was an untested assumption.

Disaster recovery does not have to be complicated or expensive, especially on AWS. But it does have to be intentional. This guide walks through building a real DR strategy: defining your recovery objectives, implementing the 3-2-1 backup rule, configuring AWS Backup, setting up cross-region replication, and most importantly, testing that everything actually works.

Understanding RTO and RPO

Before you configure a single backup, you need two numbers:

Recovery Point Objective (RPO): How much data can you afford to lose? If your RPO is 1 hour, you need backups at least every hour. If your RPO is zero, you need synchronous replication.

Recovery Time Objective (RTO): How long can your service be down before the business impact becomes unacceptable? If your RTO is 4 hours, you need a recovery process that can restore full service within 4 hours.

These numbers vary by system. Here is a realistic example for a SaaS startup:

System	RPO	RTO	Strategy
Primary database (PostgreSQL)	5 minutes	30 minutes	Continuous WAL archiving + read replica
User uploads (S3)	0 (no loss)	1 hour	Cross-region replication
Application servers	N/A (stateless)	15 minutes	Auto-scaling group, multi-AZ
Redis cache	1 hour	15 minutes	Rebuild from DB if needed
Elasticsearch	24 hours	4 hours	Nightly snapshots to S3
Configuration/secrets	0 (no loss)	30 minutes	Versioned in AWS Secrets Manager

The key insight: not everything needs the same level of protection. Your production database needs 5-minute RPO. Your Elasticsearch cluster (which can be rebuilt from the database) can tolerate 24-hour RPO. Treating everything equally means overspending on DR for low-criticality systems.

The 3-2-1 Backup Rule

The 3-2-1 rule is decades old and still the best framework for backup strategy:

3 copies of your data
2 different storage media/types
1 copy offsite (different region or provider)

In AWS terms:

Copy 1: Production data (RDS instance, EBS volumes, S3 buckets)
Copy 2: AWS Backup vault in the same region (different storage from production)
Copy 3: Cross-region backup copy (different region entirely)

For truly critical data, consider a fourth copy outside AWS entirely (GCP bucket, Azure blob, or even Backblaze B2). This protects against the unlikely but possible scenario of an AWS account compromise.

# Example: backup RDS snapshot to an external provider using pg_dump
pg_dump -h your-rds-endpoint.region.rds.amazonaws.com \
  -U dbadmin -d production \
  --format=custom \
  --file=production-$(date +%Y%m%d).dump

# Encrypt and upload to external storage
gpg --symmetric --cipher-algo AES256 production-$(date +%Y%m%d).dump
aws s3 cp production-$(date +%Y%m%d).dump.gpg s3://offsite-backups/ --storage-class GLACIER_IR

Configuring AWS Backup

AWS Backup provides a centralized service to manage backups across RDS, EBS, EFS, DynamoDB, S3, and more. Here is a production-ready configuration using Terraform:

# Backup vault
resource "aws_backup_vault" "main" {
  name        = "production-vault"
  kms_key_arn = aws_kms_key.backup.arn

  tags = {
    Environment = "production"
  }
}

# Cross-region vault for DR
resource "aws_backup_vault" "dr" {
  provider = aws.dr_region
  name     = "production-vault-dr"
  kms_key_arn = aws_kms_key.backup_dr.arn
}

# Backup plan
resource "aws_backup_plan" "production" {
  name = "production-backup-plan"

  # Hourly backups retained for 24 hours
  rule {
    rule_name         = "hourly"
    target_vault_name = aws_backup_vault.main.name
    schedule          = "cron(0 * * * ? *)"
    start_window      = 60
    completion_window  = 120

    lifecycle {
      delete_after = 1  # days
    }
  }

  # Daily backups retained for 30 days
  rule {
    rule_name         = "daily"
    target_vault_name = aws_backup_vault.main.name
    schedule          = "cron(0 3 * * ? *)"
    start_window      = 60
    completion_window  = 180

    lifecycle {
      cold_storage_after = 7
      delete_after       = 30
    }

    # Copy to DR region
    copy_action {
      destination_vault_arn = aws_backup_vault.dr.arn
      lifecycle {
        delete_after = 30
      }
    }
  }

  # Weekly backups retained for 1 year
  rule {
    rule_name         = "weekly"
    target_vault_name = aws_backup_vault.main.name
    schedule          = "cron(0 4 ? * SUN *)"
    start_window      = 60
    completion_window  = 300

    lifecycle {
      cold_storage_after = 30
      delete_after       = 365
    }
  }
}

# Assign resources to backup plan
resource "aws_backup_selection" "production" {
  iam_role_arn = aws_iam_role.backup.arn
  name         = "production-resources"
  plan_id      = aws_backup_plan.production.id

  # Backup everything tagged with Backup=true
  selection_tag {
    type  = "STRINGEQUALS"
    key   = "Backup"
    value = "true"
  }
}

Tag your resources to include them in the backup plan:

resource "aws_db_instance" "production" {
  # ... other config
  tags = {
    Backup      = "true"
    Environment = "production"
  }
}

resource "aws_ebs_volume" "data" {
  # ... other config
  tags = {
    Backup      = "true"
    Environment = "production"
  }
}

Cross-Region Replication

For services where you need faster recovery than restoring from backups, set up active replication:

RDS Cross-Region Read Replica:

# Primary in us-east-1
resource "aws_db_instance" "primary" {
  identifier     = "production-primary"
  engine         = "postgres"
  engine_version = "16.2"
  instance_class = "db.r6g.xlarge"
  multi_az       = true

  backup_retention_period = 7
  backup_window           = "03:00-04:00"
}

# Cross-region replica in us-west-2
resource "aws_db_instance" "dr_replica" {
  provider = aws.dr_region

  identifier          = "production-dr-replica"
  replicate_source_db = aws_db_instance.primary.arn
  instance_class      = "db.r6g.large"  # Can be smaller to save cost

  # Can be promoted to primary during DR
  auto_minor_version_upgrade = true
}

S3 Cross-Region Replication:

resource "aws_s3_bucket" "primary" {
  bucket = "myapp-uploads-primary"
}

resource "aws_s3_bucket_versioning" "primary" {
  bucket = aws_s3_bucket.primary.id
  versioning_configuration {
    status = "Enabled"  # Required for replication
  }
}

resource "aws_s3_bucket_replication_configuration" "primary" {
  bucket = aws_s3_bucket.primary.id
  role   = aws_iam_role.replication.arn

  rule {
    id     = "replicate-all"
    status = "Enabled"

    destination {
      bucket        = aws_s3_bucket.dr.arn
      storage_class = "STANDARD_IA"
    }
  }
}

DynamoDB Global Tables:

resource "aws_dynamodb_table" "sessions" {
  name         = "sessions"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "session_id"

  attribute {
    name = "session_id"
    type = "S"
  }

  replica {
    region_name = "us-west-2"
  }
}

Testing Your DR Plan

This is where most organizations fail. A backup strategy without tested recovery procedures is not a strategy - it is a hope.

Quarterly DR drill checklist:

## DR Drill - Q2 2026

### Pre-drill
- [ ] Schedule 4-hour maintenance window
- [ ] Notify stakeholders
- [ ] Document current production state (record counts, latest transactions)

### Database Recovery
- [ ] Restore RDS from latest cross-region snapshot
- [ ] Verify restore completed without errors
- [ ] Compare row counts against production
- [ ] Run application smoke tests against restored DB
- [ ] Record actual restore time: _____ minutes (RTO target: 30 min)
- [ ] Record data loss window: _____ minutes (RPO target: 5 min)

### Application Recovery
- [ ] Deploy application stack in DR region using Terraform
- [ ] Verify DNS failover configuration works
- [ ] Run full integration test suite
- [ ] Record actual application recovery time: _____ minutes

### S3 Data Verification
- [ ] Compare object counts between primary and DR bucket
- [ ] Download and verify random sample of objects
- [ ] Verify replication lag: _____ seconds

### Post-drill
- [ ] Document any issues encountered
- [ ] Update runbooks based on findings
- [ ] File tickets for gaps identified
- [ ] Update RTO/RPO estimates based on actual times

Automate what you can. This script verifies your RDS backup is restorable:

#!/bin/bash
# dr-test-rds-restore.sh
set -euo pipefail

SNAPSHOT_ID=$(aws rds describe-db-snapshots \
  --db-instance-identifier production-primary \
  --query 'reverse(sort_by(DBSnapshots, &SnapshotCreateTime))[0].DBSnapshotIdentifier' \
  --output text)

echo "Restoring from snapshot: $SNAPSHOT_ID"

aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier dr-test-restore \
  --db-snapshot-identifier "$SNAPSHOT_ID" \
  --db-instance-class db.t3.medium \
  --no-multi-az

echo "Waiting for instance to be available..."
aws rds wait db-instance-available --db-instance-identifier dr-test-restore

echo "Running verification queries..."
ENDPOINT=$(aws rds describe-db-instances \
  --db-instance-identifier dr-test-restore \
  --query 'DBInstances[0].Endpoint.Address' \
  --output text)

RESTORED_COUNT=$(psql -h "$ENDPOINT" -U dbadmin -d production -tAc "SELECT count(*) FROM users")
PROD_COUNT=$(psql -h prod-endpoint -U dbadmin -d production -tAc "SELECT count(*) FROM users")

echo "Production users: $PROD_COUNT"
echo "Restored users: $RESTORED_COUNT"

# Cleanup
aws rds delete-db-instance \
  --db-instance-identifier dr-test-restore \
  --skip-final-snapshot

Common DR Mistakes to Avoid

1. Backing up data but not configuration. Your database backup is useless if you cannot recreate the VPC, security groups, IAM roles, and application configuration needed to run your service. Infrastructure as code (Terraform, CloudFormation) is a DR requirement, not a nice-to-have.

2. Not encrypting backups. Unencrypted backups in S3 are a security liability. Use KMS encryption on all backup vaults and S3 buckets.

3. Using the same KMS key for primary and DR. If your primary region's KMS key becomes unavailable, you cannot decrypt backups in the DR region. Use separate KMS keys in each region, or use multi-region KMS keys.

4. Ignoring DNS TTL. If you need to failover DNS to a DR region, a 24-hour TTL means clients will keep hitting the dead primary for up to 24 hours. Set TTLs to 60 seconds for any record that might need to change during DR.

5. No communication plan. When production goes down, people panic. Document who gets notified, how (Slack, PagerDuty, phone), and who has the authority to initiate DR procedures. Practice this too.

Need Help with Your DevOps?

Building a disaster recovery strategy that actually works when you need it requires planning, implementation, and regular testing. At InstaDevOps, we help startups and SMBs implement production-grade backup and DR infrastructure - starting at $2,999/mo.

Book a free 15-minute consultation to discuss your backup and disaster recovery needs.

ArgoCD GitOps Deployment Guide: App-of-Apps and Progressive Delivery

InstaDevOps — Thu, 30 Apr 2026 13:47:36 +0000

Introduction

GitOps is the practice of using Git as the single source of truth for your infrastructure and application configuration. ArgoCD is the most widely adopted GitOps operator for Kubernetes, and for good reason - it watches your Git repositories and automatically reconciles your cluster state to match what is defined in your manifests.

But installing ArgoCD is the easy part. The hard part is structuring your repositories, managing multi-environment deployments, implementing progressive delivery, and setting up proper RBAC so your platform team does not become a bottleneck. This guide covers all of that with production-tested patterns.

Installing and Configuring ArgoCD

Start with a production-ready ArgoCD installation using the HA manifest:

# Create namespace
kubectl create namespace argocd

# Install ArgoCD HA (recommended for production)
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/ha/install.yaml

# Wait for all pods to be ready
kubectl wait --for=condition=ready pod -l app.kubernetes.io/part-of=argocd -n argocd --timeout=300s

# Get initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Expose ArgoCD via an Ingress (assuming you have an ingress controller and cert-manager):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server
  namespace: argocd
  annotations:
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - argocd.yourcompany.com
      secretName: argocd-tls
  rules:
    - host: argocd.yourcompany.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: argocd-server
                port:
                  number: 443

Configure ArgoCD to connect to your Git repositories. For private repos, use SSH deploy keys:

argocd repo add git@github.com:yourorg/k8s-manifests.git \
  --ssh-private-key-path ~/.ssh/argocd_deploy_key

The App-of-Apps Pattern

The app-of-apps pattern is the standard way to manage multiple ArgoCD applications declaratively. Instead of manually creating each Application resource through the UI or CLI, you define a single root Application that points to a directory of Application manifests.

Repository structure:

k8s-manifests/
├── apps/                    # Root app-of-apps directory
│   ├── api.yaml             # Application manifest for API service
│   ├── frontend.yaml        # Application manifest for frontend
│   ├── worker.yaml          # Application manifest for worker
│   ├── redis.yaml           # Application manifest for Redis
│   └── monitoring.yaml      # Application manifest for monitoring stack
├── services/
│   ├── api/
│   │   ├── base/
│   │   │   ├── deployment.yaml
│   │   │   ├── service.yaml
│   │   │   └── kustomization.yaml
│   │   └── overlays/
│   │       ├── staging/
│   │       │   └── kustomization.yaml
│   │       └── production/
│   │           └── kustomization.yaml
│   ├── frontend/
│   │   └── ...
│   └── worker/
│       └── ...
└── infrastructure/
    ├── redis/
    └── monitoring/

The root application:

# root-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: git@github.com:yourorg/k8s-manifests.git
    targetRevision: main
    path: apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

An individual app manifest within the apps/ directory:

# apps/api.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "2"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: git@github.com:yourorg/k8s-manifests.git
    targetRevision: main
    path: services/api/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

When you add a new service, you just add a new YAML file to the apps/ directory and push to Git. ArgoCD picks it up automatically.

Sync Waves and Resource Ordering

Sync waves control the order in which ArgoCD applies resources. This is critical when you have dependencies - you need namespaces before deployments, CRDs before custom resources, and databases before applications.

# Wave -1: Namespaces and CRDs first
apiVersion: v1
kind: Namespace
metadata:
  name: production
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

---
# Wave 0: Infrastructure (databases, caches)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: redis
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  # ... Redis application config

---
# Wave 1: Shared services (service mesh, secrets)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  # ... External Secrets Operator config

---
# Wave 2: Application services
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  # ... API application config

Combine sync waves with resource hooks for even finer control:

# Run database migration before deploying new version
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: yourorg/api:latest
          command: ["node", "migrate.js"]
      restartPolicy: Never
  backoffLimit: 3

Progressive Delivery with Argo Rollouts

ArgoCD handles syncing manifests to your cluster, but it does not manage how traffic shifts to new versions. That is where Argo Rollouts comes in. It replaces the standard Kubernetes Deployment with a Rollout resource that supports canary and blue-green deployment strategies.

Install Argo Rollouts:

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

A canary rollout with automated analysis:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api
  namespace: production
spec:
  replicas: 5
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: yourorg/api:v2.1.0
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: 500m
              memory: 512Mi
  strategy:
    canary:
      canaryService: api-canary
      stableService: api-stable
      trafficRouting:
        nginx:
          stableIngress: api-ingress
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 30
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 60
        - pause: { duration: 5m }
        - setWeight: 100

The analysis template that gates each promotion step:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
    - name: success-rate
      interval: 60s
      count: 5
      successCondition: result[0] >= 0.99
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{app="api",status=~"2.."}[5m]))
            /
            sum(rate(http_requests_total{app="api"}[5m]))

If the success rate drops below 99% during any analysis phase, Argo Rollouts automatically rolls back to the stable version. No human intervention required at 3 AM.

RBAC and Multi-Tenancy

For teams with multiple projects or environments, ArgoCD's RBAC system controls who can see and sync what. Define projects to create boundaries:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-payments
  namespace: argocd
spec:
  description: "Payments team applications"
  sourceRepos:
    - 'git@github.com:yourorg/payments-*'
  destinations:
    - namespace: 'payments-*'
      server: https://kubernetes.default.svc
  clusterResourceWhitelist:
    - group: ''
      kind: Namespace
  namespaceResourceWhitelist:
    - group: '*'
      kind: '*'
  roles:
    - name: developer
      description: "Payments team developers"
      policies:
        - p, proj:team-payments:developer, applications, get, team-payments/*, allow
        - p, proj:team-payments:developer, applications, sync, team-payments/*, allow
      groups:
        - payments-team  # Maps to SSO group

Configure SSO integration (Dex with GitHub example):

# argocd-cm ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  url: https://argocd.yourcompany.com
  dex.config: |
    connectors:
      - type: github
        id: github
        name: GitHub
        config:
          clientID: $dex.github.clientID
          clientSecret: $dex.github.clientSecret
          orgs:
            - name: yourorg

Repository Structure Best Practices

After working with dozens of ArgoCD deployments, here are the patterns that hold up:

Separate app manifests from app source code. Keep your Kubernetes manifests in a dedicated repository, not alongside your application code. This gives you independent versioning, cleaner git history, and prevents application CI from triggering ArgoCD syncs.

Use Kustomize overlays for environments. Do not duplicate manifests for staging and production. Use a base with overlays:

# services/api/overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
patches:
  - patch: |-
      - op: replace
        path: /spec/replicas
        value: 5
    target:
      kind: Deployment
      name: api
images:
  - name: yourorg/api
    newTag: v2.1.0   # Updated by CI pipeline

Automate image tag updates. Your application CI pipeline should update the image tag in the manifests repo after a successful build. Use kustomize edit set image or a tool like ArgoCD Image Updater:

# In your application CI pipeline (GitHub Actions example)
- name: Update manifest repo
  run: |
    git clone git@github.com:yourorg/k8s-manifests.git
    cd k8s-manifests/services/api/overlays/production
    kustomize edit set image yourorg/api=yourorg/api:${{ github.sha }}
    git add .
    git commit -m "Update api image to ${{ github.sha }}"
    git push

Need Help with Your DevOps?

Implementing GitOps with ArgoCD properly - from repository structure to progressive delivery to RBAC - takes experience and planning. At InstaDevOps, we help startups and SMBs set up production-grade Kubernetes infrastructure and deployment pipelines - starting at $2,999/mo.

Book a free 15-minute consultation to discuss your Kubernetes and deployment challenges.

Elasticsearch Operations Guide: Cluster Sizing and Performance Tuning

InstaDevOps — Wed, 29 Apr 2026 13:47:33 +0000

Introduction

Elasticsearch is one of those tools that is easy to set up but notoriously difficult to operate at scale. A basic single-node cluster handles your first million documents fine, then things start breaking: slow queries, disk pressure alerts, unbalanced shards, and the dreaded cluster yellow or red status at 3 AM.

This guide covers the operational side of Elasticsearch - the things you learn after running it in production for years. We will go through cluster sizing, index lifecycle management (ILM), shard allocation strategies, snapshot and restore procedures, and performance tuning techniques that actually matter.

Cluster Sizing and Hardware Planning

The most common mistake in Elasticsearch sizing is starting with too few nodes and too little memory. Here are the numbers that matter:

Memory: Elasticsearch lives and dies by its JVM heap and filesystem cache. The rule of thumb: give Elasticsearch 50% of available RAM as JVM heap (capped at 31GB to stay under compressed oops threshold), and leave the other 50% for the OS filesystem cache.

# jvm.options
-Xms16g
-Xmx16g
# Never set heap above 31g - you lose compressed ordinary object pointers

For a production logging cluster handling ~100GB/day of logs:

Role	Nodes	CPU	RAM	Storage
Master-eligible	3	2 vCPU	8 GB	50 GB SSD
Data (hot)	3	8 vCPU	64 GB	2 TB NVMe SSD
Data (warm)	2	4 vCPU	32 GB	8 TB HDD
Coordinating	2	4 vCPU	16 GB	100 GB SSD

Dedicated master nodes are non-negotiable for any production cluster. Master nodes handle cluster state, shard allocation, and index creation. If your master nodes are also serving queries and indexing data, cluster instability follows.

# elasticsearch.yml - dedicated master node
node.roles: [master]
cluster.name: production-logs
node.name: master-1

# elasticsearch.yml - hot data node
node.roles: [data_hot, data_content, ingest]
node.attr.data_tier: hot
cluster.name: production-logs
node.name: hot-data-1

AWS-specific sizing: On EC2, use r6g instances for data nodes (memory-optimized, Graviton for cost savings) and m6g for master/coordinating nodes. Use gp3 EBS volumes - they offer 3000 baseline IOPS regardless of volume size, which is more cost-effective than gp2 for most Elasticsearch workloads.

Index Lifecycle Management (ILM)

ILM automates the progression of indices through hot, warm, cold, and delete phases. Without ILM, you end up with hundreds of indices eating resources forever, or someone writes a fragile cron job to delete old indices.

Here is a production ILM policy for application logs:

PUT _ilm/policy/logs-lifecycle
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_age": "1d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "2d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          },
          "allocate": {
            "require": {
              "data": "warm"
            }
          },
          "set_priority": {
            "priority": 50
          }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "allocate": {
            "require": {
              "data": "cold"
            }
          },
          "set_priority": {
            "priority": 0
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Apply the policy to an index template:

PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-lifecycle",
      "index.lifecycle.rollover_alias": "logs-write",
      "index.routing.allocation.require.data": "hot"
    }
  }
}

Key ILM tuning tips:

Rollover on shard size (50GB max primary shard) rather than document count - shard size is what affects performance
Force merge to 1 segment in the warm phase to reclaim disk and improve query performance on read-only indices
Shrink indices in warm phase if you started with multiple shards for write throughput but no longer need them

Shard Allocation and Rebalancing

Sharding is where most Elasticsearch performance problems originate. The guidelines are straightforward but frequently violated:

Rule 1: Keep shards between 10GB and 50GB. Shards under 1GB waste resources on overhead. Shards over 50GB make recovery slow and rebalancing painful.

Rule 2: Aim for fewer than 20 shards per GB of heap. A node with 16GB heap should host no more than ~300 shards. Each shard consumes memory for metadata, field data, and segment info regardless of whether it is being queried.

Check your current shard distribution:

# Shard count per node
curl -s localhost:9200/_cat/allocation?v

# Shards sorted by size
curl -s localhost:9200/_cat/shards?v\&s=store:desc

# Identify hot spots - nodes with too many shards
curl -s localhost:9200/_cat/nodes?v\&h=name,heap.percent,disk.used_percent,shards

When shards are unevenly distributed, you can adjust allocation awareness:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "zone",
    "cluster.routing.allocation.awareness.force.zone.values": "us-east-1a,us-east-1b,us-east-1c"
  }
}

This ensures that primary and replica shards land in different availability zones, giving you zone-failure resilience.

Snapshot and Restore

Snapshots are your safety net. Without them, a bad mapping change, accidental index deletion, or cluster corruption means data loss. Set up automated snapshots to S3:

PUT _snapshot/s3-backup
{
  "type": "s3",
  "settings": {
    "bucket": "my-es-snapshots",
    "region": "us-east-1",
    "base_path": "production",
    "max_snapshot_bytes_per_sec": "200mb",
    "max_restore_bytes_per_sec": "200mb"
  }
}

Create a snapshot lifecycle management (SLM) policy:

PUT _slm/policy/nightly-backup
{
  "schedule": "0 0 2 * * ?",
  "name": "<nightly-snap-{now/d}>",
  "repository": "s3-backup",
  "config": {
    "indices": ["logs-*", "metrics-*", "app-*"],
    "ignore_unavailable": true,
    "include_global_state": false
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 7,
    "max_count": 30
  }
}

Test your restores regularly. A snapshot you have never restored is a backup you do not have. Schedule a quarterly restore drill:

# Restore a specific index to verify backup integrity
curl -X POST "localhost:9200/_snapshot/s3-backup/nightly-snap-2026.04.01/_restore" \
  -H 'Content-Type: application/json' -d '{
    "indices": "logs-2026.03.31",
    "rename_pattern": "(.+)",
    "rename_replacement": "restored-$1"
  }'

# Verify document count matches
curl -s "localhost:9200/restored-logs-2026.03.31/_count"
curl -s "localhost:9200/logs-2026.03.31/_count"

Performance Tuning

Here are the performance levers that have the biggest real-world impact, ranked by effort-to-benefit ratio:

1. Use bulk indexing, always. Single-document indexing is orders of magnitude slower. Aim for bulk requests of 5-15MB.

# Bad: indexing one document at a time
curl -X POST "localhost:9200/logs/_doc" -d '{"message": "log entry"}'

# Good: bulk indexing
curl -X POST "localhost:9200/_bulk" -H 'Content-Type: application/x-ndjson' --data-binary @bulk-data.ndjson

2. Tune refresh interval for write-heavy indices. The default 1-second refresh creates a new Lucene segment every second, which is expensive. For logging use cases where near-real-time search is not critical:

PUT logs-write/_settings
{
  "index": {
    "refresh_interval": "30s"
  }
}

3. Use doc_values and avoid fielddata. For aggregations and sorting on text fields, use the keyword type or multi-field mapping. Never enable fielddata on analyzed text fields - it loads the entire inverted index into heap memory.

4. Profile slow queries. Enable slow log to catch queries that degrade cluster performance:

PUT logs-*/_settings
{
  "index.search.slowlog.threshold.query.warn": "5s",
  "index.search.slowlog.threshold.query.info": "2s",
  "index.search.slowlog.threshold.fetch.warn": "1s"
}

5. Disable swapping entirely. Swapped JVM heap is a death sentence for Elasticsearch performance. Configure this at the OS level:

# /etc/sysctl.conf
vm.swappiness = 1

# elasticsearch.yml
bootstrap.memory_lock: true

Monitoring and Alerting

At minimum, monitor these metrics (Prometheus with elasticsearch_exporter, Datadog, or Elastic's own monitoring):

Cluster health - yellow means missing replicas, red means missing primaries
JVM heap pressure - alert at 75%, panic at 85%
Disk watermark - ES stops allocating at 85% (low), marks read-only at 95% (flood)
Thread pool rejections - search, write, and bulk thread pool rejections indicate saturation
GC pause time - young GC pauses over 100ms or old GC pauses over 1s are problems

# Quick health check script
curl -s localhost:9200/_cluster/health | jq '{
  status,
  number_of_nodes,
  active_primary_shards,
  relocating_shards,
  unassigned_shards
}'

# Thread pool rejections
curl -s localhost:9200/_cat/thread_pool/search,write?v\&h=node_name,name,active,rejected,completed

Need Help with Your DevOps?

Running Elasticsearch in production requires ongoing attention to cluster health, capacity planning, and performance optimization. At InstaDevOps, we help startups and SMBs manage complex infrastructure like Elasticsearch clusters without the cost of a full-time hire - starting at $2,999/mo.

Book a free 15-minute consultation to discuss your Elasticsearch challenges.

Git Branching Strategies: Trunk-Based vs GitFlow vs GitHub Flow

InstaDevOps — Tue, 28 Apr 2026 13:47:28 +0000

Introduction

Your Git branching strategy directly impacts how fast you ship code, how often deployments break, and how much time your team wastes on merge conflicts. Yet most teams pick a branching model once and never revisit it, even as their team size, deployment frequency, and product complexity change.

In this guide, we will compare the three most popular branching strategies - trunk-based development, GitFlow, and GitHub Flow - with honest assessments of when each works and when each falls apart. We will also cover feature flags, release management, and monorepo-specific considerations that most branching guides skip entirely.

Trunk-Based Development

Trunk-based development (TBD) means everyone commits to a single branch (main/trunk), either directly or through very short-lived feature branches that last no more than a day or two. Google, Meta, and Netflix all practice trunk-based development at massive scale.

The core rules are simple:

The main branch is always deployable
Feature branches live less than 24-48 hours
Broken code is hidden behind feature flags, not long-lived branches
CI runs on every commit and blocks merges on failure

Here is a typical workflow:

# Start work
git checkout main
git pull origin main
git checkout -b feature/add-retry-logic

# Work for a few hours, then push
git add -A
git commit -m "Add exponential backoff to API client"
git push origin feature/add-retry-logic

# Open PR, get review, merge same day
# Delete branch immediately after merge

When it works: Teams with strong CI/CD pipelines, good test coverage (70%+ meaningful coverage), and engineers comfortable with feature flags. Ideal for continuous deployment where you ship multiple times per day.

When it breaks down: Teams without automated testing, regulated industries requiring formal release sign-off, or junior-heavy teams where code review bottlenecks cause branches to pile up.

The biggest misconception about trunk-based development is that it means no code review. You absolutely still review code - you just review smaller, more frequent changes rather than massive PRs that touch 50 files.

GitFlow

GitFlow uses long-lived branches to manage releases: main holds production code, develop holds the next release, feature/* branches come off develop, release/* branches stabilize a release, and hotfix/* branches patch production emergencies.

main ─────────────────────────────────────────────
  │                               ▲          ▲
  │                               │          │
  └──▶ develop ──────────────────────────────────
         │         ▲       │            ▲
         │         │       │            │
         └── feature/x ──┘  └── feature/y ──┘

A typical GitFlow release cycle:

# Start a feature
git checkout develop
git checkout -b feature/new-dashboard

# ... work for days/weeks ...

# Merge back to develop
git checkout develop
git merge --no-ff feature/new-dashboard

# Cut a release
git checkout -b release/2.4.0 develop

# Stabilize (bug fixes only)
git commit -m "Fix date formatting in dashboard"

# Finalize release
git checkout main
git merge --no-ff release/2.4.0
git tag -a v2.4.0 -m "Release 2.4.0"
git checkout develop
git merge --no-ff release/2.4.0

When it works: Teams shipping versioned software (desktop apps, mobile apps, SDKs, on-prem installations), teams with scheduled release cycles (bi-weekly, monthly), and teams that need to maintain multiple release versions simultaneously.

When it breaks down: Web applications deploying continuously, small teams (under 5 engineers) where the overhead is not justified, and teams that end up with feature branches living for weeks, creating merge hell.

The honest truth about GitFlow in 2026: it was designed for a world where shipping software meant burning a CD. If you deploy a web application, GitFlow is almost certainly more process than you need.

GitHub Flow

GitHub Flow is the middle ground. You have one long-lived branch (main), create feature branches off it, open pull requests, and merge back to main after review. Main is always deployable.

# Create feature branch from main
git checkout main
git pull origin main
git checkout -b add-health-check-endpoint

# Work, commit, push
git add src/health.ts
git commit -m "Add /health endpoint with dependency checks"
git push origin add-health-check-endpoint

# Open PR on GitHub
gh pr create --title "Add health check endpoint" \
  --body "Adds /health endpoint that checks DB and Redis connectivity"

# After review and CI passes, merge via GitHub UI
# Deploy automatically from main

When it works: Most web application teams. It is simple enough for small teams, scales well to medium teams (5-30 engineers), and pairs naturally with CI/CD pipelines that deploy on every merge to main.

When it breaks down: When feature branches live too long (the same problem as GitFlow but less structured), or when you need to maintain multiple production versions.

Feature Flags as a Branching Strategy

Feature flags are not a replacement for branching - they are a complement that makes trunk-based development and GitHub Flow viable for complex features that take weeks to build.

Here is a practical implementation using environment variables for simple flags and a feature flag service for anything more complex:

# Simple approach: environment-based flags
# docker-compose.yml
services:
  api:
    environment:
      - FEATURE_NEW_BILLING=false
      - FEATURE_V2_SEARCH=true

// Application code
function getSearchResults(query) {
  if (process.env.FEATURE_V2_SEARCH === 'true') {
    return searchV2(query);  // New implementation
  }
  return searchV1(query);    // Existing implementation
}

For production systems, use a proper feature flag service (LaunchDarkly, Unleash, Flagsmith, or even a simple Redis-backed solution):

// Using Unleash (open source)
const { initialize } = require('unleash-client');

const unleash = initialize({
  url: 'https://unleash.yourcompany.com/api',
  appName: 'api-service',
  customHeaders: { Authorization: process.env.UNLEASH_TOKEN },
});

app.get('/api/dashboard', async (req, res) => {
  if (unleash.isEnabled('new-dashboard', { userId: req.user.id })) {
    return res.json(await getNewDashboard(req.user));
  }
  return res.json(await getLegacyDashboard(req.user));
});

The critical rule with feature flags: clean them up. Every flag should have an expiration date. Dead flags are technical debt that makes code harder to reason about. Set a calendar reminder to remove each flag within 2 weeks of full rollout.

Release Management Patterns

Regardless of your branching strategy, you need a release management approach. Here are three patterns ranked by deployment frequency:

Pattern 1: Continuous Deployment (trunk-based or GitHub Flow)

Every merge to main deploys to production automatically. Requires:

Full automated tests
Feature flags for incomplete work
Canary or blue-green deployments
Fast rollback capability

# GitHub Actions - deploy on every merge to main
name: Deploy
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm test
      - run: npm run build
      - name: Deploy to production
        run: |
          aws ecs update-service \
            --cluster production \
            --service api \
            --force-new-deployment

Pattern 2: Release Trains (GitHub Flow with scheduled deploys)

Merge to main anytime, but deploy on a schedule (daily, twice a week). Good for teams building confidence in their pipeline.

Pattern 3: Versioned Releases (GitFlow)

Cut release branches, stabilize, tag, and deploy. Necessary for software with multiple live versions.

Monorepo Branching Considerations

Monorepos add complexity to any branching strategy because a single branch may contain changes to multiple services. Key considerations:

monorepo/
├── services/
│   ├── api/
│   ├── worker/
│   └── frontend/
├── packages/
│   ├── shared-types/
│   └── utils/
└── infrastructure/
    ├── terraform/
    └── k8s/

Use path-based CI triggers so changes to one service do not trigger builds for unrelated services:

# GitHub Actions with path filters
name: API CI
on:
  pull_request:
    paths:
      - 'services/api/**'
      - 'packages/shared-types/**'
      - 'packages/utils/**'

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: cd services/api && npm ci && npm test

For monorepos, trunk-based development or GitHub Flow works far better than GitFlow. Long-lived feature branches in a monorepo are a recipe for merge conflicts across service boundaries.

Choosing the Right Strategy for Your Team

Here is a decision framework:

Factor	Trunk-Based	GitHub Flow	GitFlow
Team size	Any	2-30	5-50+
Deploy frequency	Multiple/day	Daily-weekly	Weekly-monthly
Test coverage	High (required)	Medium-high	Any
Feature flag maturity	High	Low-medium	Not needed
Multiple live versions	No	No	Yes
Regulatory requirements	Works with flags	Works fine	Natural fit

For most startups and SMBs deploying web applications: Start with GitHub Flow. It has the lowest overhead and the fewest ways to shoot yourself in the foot. Move to trunk-based development when your CI/CD pipeline and test coverage are mature enough to support it.

Avoid GitFlow unless you are shipping versioned software (mobile apps, SDKs, on-prem products) or you have regulatory requirements that demand formal release branches.

Need Help with Your DevOps?

Setting up the right branching strategy, CI/CD pipeline, and deployment workflow is just the beginning. At InstaDevOps, we help startups and SMBs build production-grade DevOps infrastructure at a fraction of the cost of a full-time hire - starting at $2,999/mo.

Book a free 15-minute consultation to discuss your team's workflow and deployment challenges.

DNS and CDN Architecture: Building Fast, Resilient Global Infrastructure

InstaDevOps — Mon, 27 Apr 2026 13:47:26 +0000

Introduction

DNS is the most critical and least understood part of most infrastructure. When DNS is working, nobody thinks about it. When it breaks, everything breaks. Your application, your API, your email, your monitoring dashboards that would help you debug the DNS issue, all of them depend on DNS resolution.

CDN configuration is similarly important but often treated as an afterthought. Teams deploy CloudFront or Cloudflare, accept the defaults, and never think about cache hit ratios, edge computing, or failover behavior until there is an incident.

This guide covers the DNS and CDN architecture decisions that matter for production systems, from basic setup through multi-region failover, with specific configurations for Route 53, CloudFront, and Cloudflare.

DNS Fundamentals That Matter in Production

Before diving into architecture, let us establish the DNS concepts that directly affect your infrastructure decisions.

Record types you will use most:

Type	Purpose	Example
A	Maps domain to IPv4 address	`api.example.com → 203.0.113.50`
AAAA	Maps domain to IPv6 address	`api.example.com → 2001:db8::1`
CNAME	Alias to another domain	`www.example.com → example.com`
ALIAS/ANAME	CNAME-like at zone apex	`example.com → d1234.cloudfront.net`
MX	Mail server routing	`example.com → 10 mail.example.com`
TXT	Verification and policy	SPF, DKIM, DMARC records
NS	Nameserver delegation	`example.com → ns-1234.awsdns-12.org`

TTL (Time To Live) controls how long resolvers cache your records. Lower TTL means faster propagation of changes but more DNS queries hitting your nameservers. Higher TTL means better performance but slower failover.

# Check current TTL and propagation
dig +noall +answer api.example.com A

# Check from multiple locations
dig @8.8.8.8 api.example.com A     # Google DNS
dig @1.1.1.1 api.example.com A     # Cloudflare DNS
dig @208.67.222.222 api.example.com A  # OpenDNS

Production TTL recommendations:

Static content CDN domains: 86400 (24 hours)
API endpoints (stable): 300-600 (5-10 minutes)
API endpoints (failover-ready): 60 (1 minute)
During migrations: 60 or lower, set 24-48 hours before the change

Route 53 Configuration for Production

Route 53 is AWS's DNS service and the most common choice for teams running on AWS. Its killer feature is routing policies that go beyond simple DNS records.

Basic hosted zone setup:

# Create a hosted zone
aws route53 create-hosted-zone \
  --name example.com \
  --caller-reference "$(date +%s)"

# The output includes nameservers - update these at your domain registrar

Health checks and failover routing:

# Create a health check for the primary endpoint
aws route53 create-health-check --caller-reference "primary-api" \
  --health-check-config '{
    "IPAddress": "203.0.113.50",
    "Port": 443,
    "Type": "HTTPS",
    "ResourcePath": "/health",
    "RequestInterval": 10,
    "FailureThreshold": 3,
    "EnableSNI": true,
    "FullyQualifiedDomainName": "api.example.com"
  }'

// Primary record (us-east-1)
{
  "Name": "api.example.com",
  "Type": "A",
  "SetIdentifier": "primary",
  "Failover": "PRIMARY",
  "TTL": 60,
  "ResourceRecords": [{"Value": "203.0.113.50"}],
  "HealthCheckId": "abcd-1234-health-check-id"
}

// Secondary record (eu-west-1)
{
  "Name": "api.example.com",
  "Type": "A",
  "SetIdentifier": "secondary",
  "Failover": "SECONDARY",
  "TTL": 60,
  "ResourceRecords": [{"Value": "198.51.100.25"}]
}

When the primary health check fails (3 consecutive failures at 10-second intervals = 30 seconds), Route 53 automatically starts returning the secondary IP. With a 60-second TTL, most clients will failover within 90 seconds.

Latency-based routing sends users to the closest region:

// US East record
{
  "Name": "api.example.com",
  "Type": "A",
  "SetIdentifier": "us-east-1",
  "Region": "us-east-1",
  "TTL": 60,
  "AliasTarget": {
    "HostedZoneId": "Z35SXDOTRQ7X7K",
    "DNSName": "us-east-alb-1234.us-east-1.elb.amazonaws.com",
    "EvaluateTargetHealth": true
  }
}

// EU West record
{
  "Name": "api.example.com",
  "Type": "A",
  "SetIdentifier": "eu-west-1",
  "Region": "eu-west-1",
  "TTL": 60,
  "AliasTarget": {
    "HostedZoneId": "Z32O12XQLNTSW2",
    "DNSName": "eu-west-alb-5678.eu-west-1.elb.amazonaws.com",
    "EvaluateTargetHealth": true
  }
}

Route 53 determines the user's region based on the location of the DNS resolver they are using. This works well for most cases but can route incorrectly when users use public DNS resolvers (like 8.8.8.8) that are anycast and may resolve from a different location than the user.

CloudFront CDN Configuration

CloudFront is AWS's CDN with 600+ edge locations globally. For static websites and API acceleration, proper CloudFront configuration can reduce latency by 50-80% for geographically distributed users.

# Create a CloudFront distribution for a static website on S3
aws cloudfront create-distribution --distribution-config '{
  "CallerReference": "static-site-2026",
  "Origins": {
    "Quantity": 1,
    "Items": [{
      "Id": "S3-static-site",
      "DomainName": "my-site-bucket.s3.amazonaws.com",
      "S3OriginConfig": {
        "OriginAccessIdentity": ""
      },
      "OriginAccessControlId": "E2QWRUHEXAMPLE"
    }]
  },
  "DefaultCacheBehavior": {
    "TargetOriginId": "S3-static-site",
    "ViewerProtocolPolicy": "redirect-to-https",
    "CachePolicyId": "658327ea-f89d-4fab-a63d-7e88639e58f6",
    "Compress": true,
    "AllowedMethods": {
      "Quantity": 2,
      "Items": ["GET", "HEAD"]
    }
  },
  "Aliases": {
    "Quantity": 1,
    "Items": ["www.example.com"]
  },
  "ViewerCertificate": {
    "ACMCertificateArn": "arn:aws:acm:us-east-1:123456789012:certificate/abc123",
    "SSLSupportMethod": "sni-only",
    "MinimumProtocolVersion": "TLSv1.2_2021"
  },
  "HttpVersion": "http2and3",
  "DefaultRootObject": "index.html",
  "Enabled": true,
  "Comment": "Production static site"
}'

Cache policies control what CloudFront caches and for how long. The default policies work for most cases, but for APIs you need custom policies:

{
  "CachePolicyConfig": {
    "Name": "API-Cache-Policy",
    "DefaultTTL": 0,
    "MaxTTL": 86400,
    "MinTTL": 0,
    "ParametersInCacheKeyAndForwardedToOrigin": {
      "EnableAcceptEncodingGzip": true,
      "EnableAcceptEncodingBrotli": true,
      "HeadersConfig": {
        "HeaderBehavior": "whitelist",
        "Headers": {
          "Quantity": 1,
          "Items": ["Authorization"]
        }
      },
      "QueryStringsConfig": {
        "QueryStringBehavior": "all"
      },
      "CookiesConfig": {
        "CookieBehavior": "none"
      }
    }
  }
}

Origin failover provides automatic failover between origin servers:

{
  "OriginGroups": {
    "Quantity": 1,
    "Items": [{
      "Id": "api-origin-group",
      "FailoverCriteria": {
        "StatusCodes": {
          "Quantity": 4,
          "Items": [500, 502, 503, 504]
        }
      },
      "Members": {
        "Quantity": 2,
        "Items": [
          { "OriginId": "api-us-east-1" },
          { "OriginId": "api-eu-west-1" }
        ]
      }
    }]
  }
}

When CloudFront gets a 5xx from the primary origin, it automatically retries the request against the secondary origin. This failover is per-request and happens at the edge, so it is much faster than DNS failover.

Cloudflare as an Alternative

Cloudflare offers a compelling alternative to CloudFront, especially for teams that are not all-in on AWS. Its free tier is generous, the dashboard is more intuitive, and features like Workers, automatic image optimization, and DDoS protection are included.

Key Cloudflare configurations for production:

# Using Cloudflare API to configure DNS records
curl -X POST "https://api.cloudflare.com/client/v4/zones/{zone_id}/dns_records" \
  -H "Authorization: Bearer $CF_API_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{
    "type": "A",
    "name": "api.example.com",
    "content": "203.0.113.50",
    "ttl": 1,
    "proxied": true
  }'

When proxied: true, traffic flows through Cloudflare's network, enabling caching, DDoS protection, and WAF rules. When proxied: false (DNS-only), Cloudflare just serves DNS.

Page Rules and Cache Rules control caching behavior:

# Cache everything on the static assets subdomain
curl -X POST "https://api.cloudflare.com/client/v4/zones/{zone_id}/rulesets" \
  -H "Authorization: Bearer $CF_API_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{
    "name": "Static Assets Cache",
    "kind": "zone",
    "phase": "http_request_cache_settings",
    "rules": [{
      "expression": "(http.host eq \"static.example.com\")",
      "action": "set_cache_settings",
      "action_parameters": {
        "cache": true,
        "edge_ttl": {
          "mode": "override_origin",
          "default": 2592000
        },
        "browser_ttl": {
          "mode": "override_origin",
          "default": 86400
        }
      }
    }]
  }'

Cloudflare Workers run JavaScript at the edge, enabling powerful patterns like A/B testing, request routing, and API response transformation without touching your origin:

// worker.js - Edge-side API response caching with stale-while-revalidate
export default {
  async fetch(request, env, ctx) {
    const cacheKey = new Request(request.url, request);
    const cache = caches.default;

    let response = await cache.match(cacheKey);

    if (response) {
      // Return cached response immediately
      // Revalidate in the background if older than 60 seconds
      const age = Date.now() - new Date(response.headers.get('x-cache-time')).getTime();
      if (age > 60000) {
        ctx.waitUntil(revalidate(request, cacheKey, cache));
      }
      return response;
    }

    return await revalidate(request, cacheKey, cache);
  }
};

async function revalidate(request, cacheKey, cache) {
  const response = await fetch(request);
  const newResponse = new Response(response.body, response);
  newResponse.headers.set('x-cache-time', new Date().toISOString());
  newResponse.headers.set('Cache-Control', 'public, max-age=120');

  await cache.put(cacheKey, newResponse.clone());
  return newResponse;
}

Caching Headers and Cache Control Strategy

Your CDN is only as effective as your caching headers. If your origin sends Cache-Control: no-cache on everything, your CDN is just an expensive reverse proxy.

# Nginx configuration for optimal caching headers
server {
    listen 443 ssl http2;
    server_name api.example.com;

    # Static assets - cache aggressively (hashed filenames for cache busting)
    location ~* \.(js|css|png|jpg|jpeg|gif|svg|woff2|ico)$ {
        expires 1y;
        add_header Cache-Control "public, max-age=31536000, immutable";
        add_header Vary "Accept-Encoding";
    }

    # HTML pages - short cache, revalidate
    location ~* \.html$ {
        add_header Cache-Control "public, max-age=300, must-revalidate";
        add_header Vary "Accept-Encoding";
    }

    # API responses - no CDN cache, allow browser cache
    location /api/ {
        add_header Cache-Control "private, max-age=0, must-revalidate";
        add_header Vary "Authorization, Accept-Encoding";
    }

    # API responses that are cacheable (public, non-personalized)
    location /api/products {
        add_header Cache-Control "public, max-age=60, s-maxage=300";
        add_header Vary "Accept-Encoding";
    }
}

The distinction between max-age and s-maxage is important:

max-age controls browser cache duration.
s-maxage controls CDN/proxy cache duration (overrides max-age for shared caches).

This lets you cache at the CDN for 5 minutes while telling browsers to cache for only 1 minute, giving you a balance between performance and freshness.

Cache invalidation when you deploy:

# CloudFront invalidation
aws cloudfront create-invalidation \
  --distribution-id E1234ABCDEF \
  --paths "/*"

# Targeted invalidation (faster and cheaper)
aws cloudfront create-invalidation \
  --distribution-id E1234ABCDEF \
  --paths "/index.html" "/api/products*"

# Cloudflare cache purge
curl -X POST "https://api.cloudflare.com/client/v4/zones/{zone_id}/purge_cache" \
  -H "Authorization: Bearer $CF_API_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{"purge_everything": true}'

# Cloudflare targeted purge
curl -X POST "https://api.cloudflare.com/client/v4/zones/{zone_id}/purge_cache" \
  -H "Authorization: Bearer $CF_API_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{"files": ["https://example.com/styles.css", "https://example.com/app.js"]}'

Multi-Region Failover Architecture

For services that require high availability across regions, combine DNS routing with CDN origin failover for defense in depth.

A strong multi-region architecture:

User Request
  │
  ▼
Cloudflare/CloudFront (Global Edge)
  │
  ├─ Cache HIT → Return immediately
  │
  ├─ Cache MISS → Origin Request
  │     │
  │     ▼
  │   Route 53 Latency-Based Routing
  │     │
  │     ├─ US users → us-east-1 ALB
  │     │     │
  │     │     └─ Health check fails → eu-west-1 ALB (failover)
  │     │
  │     └─ EU users → eu-west-1 ALB
  │           │
  │           └─ Health check fails → us-east-1 ALB (failover)

Implementation checklist for multi-region failover:

DNS TTL at 60 seconds for all records involved in failover.
Health checks every 10 seconds with a failure threshold of 3 (30-second detection).
CDN origin failover for per-request resilience (faster than DNS failover).
Database replication across regions (Aurora Global Database or DynamoDB Global Tables).
Session management in a shared store (ElastiCache Global Datastore or DynamoDB).
Monitoring from multiple regions to distinguish between regional and global outages.

# Monitor DNS resolution from your CI/CD or monitoring system
# Check that failover is working by querying from multiple locations
for resolver in 8.8.8.8 1.1.1.1 208.67.222.222; do
  echo "Resolver $resolver:"
  dig +short @$resolver api.example.com A
done

# Monitor CDN cache hit ratio
aws cloudwatch get-metric-statistics \
  --namespace AWS/CloudFront \
  --metric-name CacheHitRate \
  --dimensions Name=DistributionId,Value=E1234ABCDEF \
  --start-time "$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --period 300 \
  --statistics Average

A healthy CDN should have a cache hit ratio above 80% for static content. If it is lower, review your caching headers and cache key configuration. Every cache miss is a request to your origin, which means more load on your infrastructure and higher latency for your users.

Need Help with Your DevOps?

DNS and CDN architecture is foundational infrastructure that directly impacts your application's performance, reliability, and user experience. At InstaDevOps, we design and implement global infrastructure architectures with multi-region failover, CDN optimization, and DNS strategies that keep your applications fast and available.

We offer fractional DevOps engineering starting at $2,999/month with no long-term contracts. Book a free 15-minute call to discuss your infrastructure needs: https://calendly.com/instadevops/15min

Load Testing with k6: From First Script to CI Pipeline Integration

InstaDevOps — Sun, 26 Apr 2026 13:47:22 +0000

Introduction

Most teams do not load test their applications until something breaks in production. A marketing campaign drives 10x traffic, the API starts returning 503s, and suddenly everyone is scrambling to figure out how many requests the system can actually handle.

k6 by Grafana Labs has become the go-to load testing tool for DevOps teams, and for good reason. It uses JavaScript for test scripts, runs efficiently on minimal hardware, has built-in support for protocols beyond HTTP (gRPC, WebSocket, browser), and integrates cleanly into CI/CD pipelines.

This guide takes you from writing your first k6 script to running automated performance tests in your CI pipeline with threshold-based pass/fail gates.

Writing Your First k6 Script

k6 scripts are JavaScript (ES6 modules) that define virtual user behavior. Each virtual user (VU) runs the default function repeatedly for the duration of the test.

// load-test.js
import http from 'k6/http';
import { check, sleep } from 'k6';

// Test configuration
export const options = {
  vus: 50,           // 50 virtual users
  duration: '2m',    // Run for 2 minutes
};

// Default function - each VU runs this in a loop
export default function () {
  // GET request
  const listRes = http.get('https://api.example.com/products');

  check(listRes, {
    'list status is 200': (r) => r.status === 200,
    'list response time < 500ms': (r) => r.timings.duration < 500,
    'list returns products': (r) => JSON.parse(r.body).length > 0,
  });

  // POST request with JSON body
  const payload = JSON.stringify({
    name: 'Test Product',
    price: 29.99,
    category: 'electronics',
  });

  const createRes = http.post('https://api.example.com/products', payload, {
    headers: {
      'Content-Type': 'application/json',
      'Authorization': 'Bearer ' + __ENV.API_TOKEN,
    },
  });

  check(createRes, {
    'create status is 201': (r) => r.status === 201,
    'create response time < 1000ms': (r) => r.timings.duration < 1000,
  });

  sleep(1); // Think time between iterations
}

Run it:

# Basic run
k6 run load-test.js

# Pass environment variables
k6 run -e API_TOKEN=your-token load-test.js

# Override options from CLI
k6 run --vus 100 --duration 5m load-test.js

The sleep(1) at the end simulates real user think time. Without it, each VU fires requests as fast as possible, which does not represent realistic traffic patterns and will skew your results.

Ramping Strategies for Realistic Load Profiles

Constant VU count is fine for quick smoke tests, but production load testing requires more sophisticated patterns. k6 supports multiple execution scenarios with different ramping strategies.

Ramp-up/ramp-down (most common for load tests):

export const options = {
  stages: [
    { duration: '2m', target: 50 },   // Ramp up to 50 VUs over 2 minutes
    { duration: '5m', target: 50 },   // Stay at 50 VUs for 5 minutes
    { duration: '2m', target: 100 },  // Ramp up to 100 VUs
    { duration: '5m', target: 100 },  // Stay at 100 VUs for 5 minutes
    { duration: '2m', target: 0 },    // Ramp down to 0
  ],
};

Spike test (sudden traffic surge):

export const options = {
  stages: [
    { duration: '1m', target: 10 },    // Normal load
    { duration: '30s', target: 500 },   // Spike to 500 VUs
    { duration: '2m', target: 500 },    // Sustain spike
    { duration: '30s', target: 10 },    // Return to normal
    { duration: '2m', target: 10 },     // Recovery period
  ],
};

Soak test (find memory leaks and degradation over time):

export const options = {
  stages: [
    { duration: '5m', target: 50 },    // Ramp up
    { duration: '4h', target: 50 },    // Sustained load for 4 hours
    { duration: '5m', target: 0 },     // Ramp down
  ],
};

Multiple scenarios (simulate different user types simultaneously):

export const options = {
  scenarios: {
    browse: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '2m', target: 100 },
        { duration: '5m', target: 100 },
        { duration: '2m', target: 0 },
      ],
      exec: 'browseProducts',
    },
    purchase: {
      executor: 'constant-arrival-rate',
      rate: 10,             // 10 iterations per timeUnit
      timeUnit: '1s',       // = 10 requests per second
      duration: '9m',
      preAllocatedVUs: 50,
      maxVUs: 200,
      exec: 'purchaseFlow',
    },
  },
};

export function browseProducts() {
  http.get('https://api.example.com/products');
  sleep(Math.random() * 3 + 1); // 1-4 second think time
}

export function purchaseFlow() {
  const products = http.get('https://api.example.com/products');
  const productId = JSON.parse(products.body)[0].id;

  http.post('https://api.example.com/cart', JSON.stringify({ productId }), {
    headers: { 'Content-Type': 'application/json' },
  });

  http.post('https://api.example.com/checkout', null, {
    headers: { 'Content-Type': 'application/json' },
  });
}

The constant-arrival-rate executor is important. Instead of controlling the number of VUs, it controls the request rate. This gives you a consistent load regardless of how fast or slow the server responds, which is better for finding the breaking point.

Threshold-Based SLOs

Thresholds are k6's most powerful feature for CI integration. They define pass/fail criteria for your test. If any threshold is violated, k6 exits with a non-zero code, which fails your CI pipeline.

export const options = {
  stages: [
    { duration: '2m', target: 50 },
    { duration: '5m', target: 50 },
    { duration: '2m', target: 0 },
  ],
  thresholds: {
    // 95th percentile response time must be under 500ms
    'http_req_duration': ['p(95)<500', 'p(99)<1000'],

    // Error rate must be under 1%
    'http_req_failed': ['rate<0.01'],

    // Custom metrics per endpoint
    'http_req_duration{name:list_products}': ['p(95)<300'],
    'http_req_duration{name:create_order}': ['p(95)<800'],

    // Check pass rate must be above 98%
    'checks': ['rate>0.98'],

    // Custom counter threshold
    'payment_errors': ['count<5'],
  },
};

Use tags to set different thresholds for different endpoints:

import { Counter } from 'k6/metrics';

const paymentErrors = new Counter('payment_errors');

export default function () {
  // Tag requests for per-endpoint thresholds
  const listRes = http.get('https://api.example.com/products', {
    tags: { name: 'list_products' },
  });

  const orderRes = http.post('https://api.example.com/orders', payload, {
    tags: { name: 'create_order' },
  });

  if (orderRes.status !== 201) {
    paymentErrors.add(1);
  }
}

Map your thresholds directly to your SLOs. If your SLO states that 99.9% of API requests must complete within 500ms, your k6 threshold should reflect that. This turns load testing from a manual exercise into an automated contract verification.

Integrating k6 into CI/CD Pipelines

The real value of k6 emerges when you run it on every deployment. Here is a complete GitHub Actions workflow:

# .github/workflows/load-test.yml
name: Load Test

on:
  deployment_status:
    # Run after successful deployment to staging
  workflow_dispatch:
    inputs:
      target_url:
        description: 'Target URL for load test'
        required: true

jobs:
  load-test:
    runs-on: ubuntu-latest
    if: github.event.deployment_status.state == 'success' || github.event_name == 'workflow_dispatch'

    steps:
      - uses: actions/checkout@v4

      - name: Install k6
        run: |
          sudo gpg -k
          sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
          echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list
          sudo apt-get update
          sudo apt-get install k6

      - name: Run load test
        env:
          API_TOKEN: ${{ secrets.STAGING_API_TOKEN }}
          TARGET_URL: ${{ github.event.inputs.target_url || 'https://staging-api.example.com' }}
        run: |
          k6 run \
            --out json=results.json \
            --env TARGET_URL=$TARGET_URL \
            --env API_TOKEN=$API_TOKEN \
            tests/load/api-load-test.js

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: k6-results
          path: results.json

      - name: Post results to PR
        if: failure() && github.event.pull_request
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: '## Load Test Failed\n\nPerformance thresholds were violated. Check the [workflow run](' + context.serverUrl + '/' + context.repo.owner + '/' + context.repo.repo + '/actions/runs/' + context.runId + ') for details.'
            });

For GitLab CI:

load_test:
  stage: test
  image: grafana/k6:latest
  script:
    - k6 run --out json=results.json tests/load/api-load-test.js
  artifacts:
    when: always
    paths:
      - results.json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Interpreting Results and Finding Bottlenecks

k6 outputs a summary at the end of each run. Here is how to read it:

http_req_duration..............: avg=245.3ms  min=12ms  med=198ms  max=4521ms  p(90)=412ms  p(95)=523ms
http_req_failed................: 0.34%   345 out of 100234
http_req_waiting...............: avg=240.1ms  min=10ms  med=193ms  max=4518ms  p(90)=407ms  p(95)=518ms
http_reqs......................: 100234  557.96/s
iteration_duration.............: avg=1.25s   min=1.01s  med=1.2s   max=5.52s   p(90)=1.41s  p(95)=1.53s
vus............................: 50      min=1       max=50

Key metrics to focus on:

p(95) and p(99) duration: These matter more than average. A 200ms average with a 5-second p99 means 1% of your users are having a terrible experience.
http_req_failed rate: Any non-zero failure rate under load deserves investigation.
http_req_waiting vs http_req_duration: The difference is the time spent receiving the response body. A large gap suggests large response payloads.
iteration_duration: If this is much higher than your request durations, check your think times and test logic.

When performance degrades under load, correlate k6 results with server-side metrics:

CPU saturation: If CPU hits 100%, you need horizontal scaling or code optimization.
Memory pressure: Climbing memory with eventual OOM kills points to memory leaks.
Database connections: Connection pool exhaustion causes sudden latency spikes.
Network I/O: Check for bandwidth limits between your services and dependencies.

Export results to Grafana for visualization:

# Stream results to Prometheus via remote write
k6 run --out experimental-prometheus-rw \
  --env K6_PROMETHEUS_RW_SERVER_URL=http://prometheus:9090/api/v1/write \
  load-test.js

# Or output to InfluxDB
k6 run --out influxdb=http://influxdb:8086/k6 load-test.js

Advanced Patterns and Tips

Parameterize with CSV data for realistic test data:

import papaparse from 'https://jslib.k6.io/papaparse/5.1.1/index.js';
import { SharedArray } from 'k6/data';

const users = new SharedArray('users', function () {
  return papaparse.parse(open('./test-users.csv'), { header: true }).data;
});

export default function () {
  const user = users[Math.floor(Math.random() * users.length)];

  const loginRes = http.post('https://api.example.com/login', JSON.stringify({
    email: user.email,
    password: user.password,
  }), {
    headers: { 'Content-Type': 'application/json' },
  });

  const token = JSON.parse(loginRes.body).token;

  http.get('https://api.example.com/dashboard', {
    headers: { 'Authorization': 'Bearer ' + token },
  });
}

Group related requests for logical organization:

import { group } from 'k6';

export default function () {
  group('User Login Flow', function () {
    http.get('https://app.example.com/login');
    http.post('https://api.example.com/auth/login', credentials);
  });

  group('Dashboard Load', function () {
    http.get('https://api.example.com/dashboard/summary');
    http.get('https://api.example.com/dashboard/notifications');
    http.get('https://api.example.com/dashboard/recent-activity');
  });
}

Test WebSocket connections:

import ws from 'k6/ws';

export default function () {
  const url = 'wss://api.example.com/ws';
  const res = ws.connect(url, {}, function (socket) {
    socket.on('open', () => {
      socket.send(JSON.stringify({ type: 'subscribe', channel: 'updates' }));
    });

    socket.on('message', (data) => {
      const msg = JSON.parse(data);
      check(msg, { 'received update': (m) => m.type === 'update' });
    });

    socket.setTimeout(function () {
      socket.close();
    }, 30000);
  });
}

Need Help with Your DevOps?

Performance testing is most valuable when it is part of your development workflow, not an afterthought before a launch. At InstaDevOps, we help teams build performance testing pipelines that catch regressions before they reach production, including k6 setup, realistic test scenarios, and CI integration.

We offer fractional DevOps engineering starting at $2,999/month with no long-term contracts. Book a free 15-minute call to discuss your performance testing needs: https://calendly.com/instadevops/15min

AWS Lambda Best Practices: Cold Starts, Memory Tuning, and Cost Optimization

InstaDevOps — Sat, 25 Apr 2026 13:47:17 +0000

Introduction

AWS Lambda changed how we think about infrastructure. No servers to manage, automatic scaling, pay-per-invocation pricing. But Lambda is not magic, and teams that treat it as a black box end up with slow, expensive, and hard-to-debug serverless applications.

After managing Lambda functions processing millions of invocations per day across multiple production environments, I have learned that the difference between a well-optimized Lambda and a naive one can be a 10x difference in both latency and cost. Cold starts that take 5 seconds can be reduced to 200ms. Monthly bills of $500 can drop to $50 with the same throughput.

This guide covers the practices that make the biggest difference in real-world Lambda deployments.

Understanding and Reducing Cold Starts

A cold start happens when Lambda needs to create a new execution environment for your function. This involves downloading your deployment package, starting the runtime, and running your initialization code. For Python and Node.js, cold starts typically add 200-500ms. For Java and .NET, they can exceed 5 seconds.

The factors that affect cold start duration, in order of impact:

Runtime choice: Node.js and Python cold-start fastest. Java and .NET are slowest.
Package size: Larger deployment packages take longer to download and extract.
Initialization code: Database connections, SDK clients, and config loading during init.
VPC configuration: Functions in a VPC used to add 8-10 seconds. Since AWS added Hyperplane ENIs, this is down to ~1 second on first invocation.
Memory allocation: More memory means more CPU, which speeds up initialization.

Practical steps to reduce cold starts:

// GOOD: Initialize SDK clients outside the handler (runs once per cold start)
const { DynamoDBClient } = require("@aws-sdk/client-dynamodb");
const { DynamoDBDocumentClient, GetCommand } = require("@aws-sdk/lib-dynamodb");

const client = new DynamoDBClient({ region: "us-east-1" });
const docClient = DynamoDBDocumentClient.from(client);

// GOOD: Reuse database connections across invocations
let dbConnection = null;
async function getDbConnection() {
  if (!dbConnection) {
    dbConnection = await createConnection({
      host: process.env.DB_HOST,
      // ... connection config
    });
  }
  return dbConnection;
}

exports.handler = async (event) => {
  // Handler code uses the pre-initialized clients
  const result = await docClient.send(new GetCommand({
    TableName: "users",
    Key: { id: event.pathParameters.userId }
  }));

  return {
    statusCode: 200,
    body: JSON.stringify(result.Item)
  };
};

For latency-sensitive functions, use Provisioned Concurrency. This keeps a specified number of execution environments warm and ready:

# Set provisioned concurrency on a Lambda alias
aws lambda put-provisioned-concurrency-config \
  --function-name my-api-handler \
  --qualifier prod \
  --provisioned-concurrent-executions 10

# Use Application Auto Scaling to adjust based on utilization
aws application-autoscaling register-scalable-target \
  --service-namespace lambda \
  --resource-id "function:my-api-handler:prod" \
  --scalable-dimension lambda:function:ProvisionedConcurrency \
  --min-capacity 5 \
  --max-capacity 50

Provisioned Concurrency costs money even when idle, so only use it for functions that are user-facing and latency-sensitive. For background processing, async invocations, and SQS consumers, cold starts do not matter.

Memory and CPU Tuning

Lambda allocates CPU proportional to memory. At 1,769 MB, you get one full vCPU. At 10,240 MB, you get six vCPUs. This means increasing memory does not just give you more RAM; it gives you more compute power, which can make CPU-bound functions faster and cheaper.

The AWS Lambda Power Tuning tool automates finding the optimal memory setting:

# Deploy the power tuning state machine
aws serverlessrepo create-cloud-formation-change-set \
  --application-id arn:aws:serverlessrepo:us-east-1:451282441545:applications/aws-lambda-power-tuning \
  --stack-name lambda-power-tuning \
  --capabilities CAPABILITY_IAM

# Run the tuning (tests your function at different memory levels)
aws stepfunctions start-execution \
  --state-machine-arn arn:aws:states:us-east-1:123456789012:stateMachine:powerTuningStateMachine \
  --input '{
    "lambdaARN": "arn:aws:lambda:us-east-1:123456789012:function:my-function",
    "powerValues": [128, 256, 512, 1024, 1769, 3008],
    "num": 50,
    "payload": "{}",
    "parallelInvocation": true,
    "strategy": "cost"
  }'

I have seen cases where increasing memory from 128 MB to 512 MB made a function 4x faster and actually cheaper because the per-millisecond cost increase was offset by the dramatic reduction in execution time. Always benchmark before assuming less memory means lower cost.

Lambda Powertools for Production Observability

AWS Lambda Powertools is a toolkit that should be in every Lambda function you deploy to production. It provides structured logging, distributed tracing, custom metrics, and parameter management with minimal boilerplate.

const { Logger } = require("@aws-lambda-powertools/logger");
const { Tracer } = require("@aws-lambda-powertools/tracer");
const { Metrics, MetricUnit } = require("@aws-lambda-powertools/metrics");
const { injectLambdaContext } = require("@aws-lambda-powertools/logger/middleware");
const { captureLambdaHandler } = require("@aws-lambda-powertools/tracer/middleware");
const { logMetrics } = require("@aws-lambda-powertools/metrics/middleware");
const middy = require("@middy/core");

const logger = new Logger({ serviceName: "payment-api" });
const tracer = new Tracer({ serviceName: "payment-api" });
const metrics = new Metrics({ namespace: "PaymentService" });

const lambdaHandler = async (event) => {
  // Structured logging with automatic Lambda context
  logger.info("Processing payment", {
    orderId: event.orderId,
    amount: event.amount,
    currency: event.currency
  });

  // Custom business metrics
  metrics.addMetric("PaymentProcessed", MetricUnit.Count, 1);
  metrics.addMetric("PaymentAmount", MetricUnit.None, event.amount);

  // Annotate traces for filtering in X-Ray
  tracer.putAnnotation("orderId", event.orderId);
  tracer.putMetadata("orderDetails", event);

  try {
    const result = await processPayment(event);
    metrics.addMetric("PaymentSucceeded", MetricUnit.Count, 1);
    return { statusCode: 200, body: JSON.stringify(result) };
  } catch (error) {
    metrics.addMetric("PaymentFailed", MetricUnit.Count, 1);
    logger.error("Payment failed", { error: error.message, orderId: event.orderId });
    throw error;
  }
};

// Middy middleware chain adds context to every invocation
exports.handler = middy(lambdaHandler)
  .use(injectLambdaContext(logger))
  .use(captureLambdaHandler(tracer))
  .use(logMetrics(metrics));

This gives you structured JSON logs that you can query in CloudWatch Logs Insights, X-Ray traces for distributed tracing, and CloudWatch custom metrics for business dashboards and alarms, all without writing any plumbing code.

Lambda Layers for Dependency Management

Lambda Layers let you package shared dependencies separately from your function code. This reduces deployment package size, speeds up deployments, and ensures consistent dependency versions across functions.

# Create a layer with shared Node.js dependencies
mkdir -p layer/nodejs
cd layer/nodejs
npm init -y
npm install @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb @aws-lambda-powertools/logger @aws-lambda-powertools/tracer

cd ..
zip -r shared-deps-layer.zip nodejs/

aws lambda publish-layer-version \
  --layer-name shared-deps \
  --description "Shared dependencies for backend Lambda functions" \
  --zip-file fileb://shared-deps-layer.zip \
  --compatible-runtimes nodejs20.x nodejs22.x

# Attach the layer to a function
aws lambda update-function-configuration \
  --function-name my-function \
  --layers arn:aws:lambda:us-east-1:123456789012:layer:shared-deps:3

A few rules for effective Layer usage:

Keep layers under 50 MB unzipped for fast cold starts.
Version your layers and pin functions to specific layer versions in production.
Use one layer for AWS SDK dependencies, another for your own shared utilities.
Do not put business logic in layers. Layers are for dependencies and utilities.

Event-Driven Architecture Patterns

Lambda functions are most effective as part of event-driven architectures. Instead of building monolithic Lambda functions that try to do everything, decompose your system into small, focused functions triggered by events.

# SAM template for an event-driven order processing pipeline
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:
  # API receives order, puts it on SQS queue
  OrderQueue:
    Type: AWS::SQS::Queue
    Properties:
      VisibilityTimeout: 300
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt OrderDLQ.Arn
        maxReceiveCount: 3

  OrderDLQ:
    Type: AWS::SQS::Queue
    Properties:
      MessageRetentionPeriod: 1209600  # 14 days

  # Process orders from queue (batch processing, auto-scaling)
  ProcessOrderFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: handlers/process-order.handler
      Runtime: nodejs20.x
      MemorySize: 512
      Timeout: 60
      Events:
        SQSEvent:
          Type: SQS
          Properties:
            Queue: !GetAtt OrderQueue.Arn
            BatchSize: 10
            MaximumBatchingWindowInSeconds: 5
            FunctionResponseTypes:
              - ReportBatchItemFailures  # Partial batch failure handling

  # On successful processing, emit event to EventBridge
  OrderEventBus:
    Type: AWS::Events::EventBus
    Properties:
      Name: orders

  # Send confirmation email on order.completed event
  SendConfirmationFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: handlers/send-confirmation.handler
      Runtime: nodejs20.x
      MemorySize: 256
      Timeout: 30
      Events:
        OrderCompleted:
          Type: EventBridgeRule
          Properties:
            EventBusName: !Ref OrderEventBus
            Pattern:
              detail-type:
                - "order.completed"

This pattern gives you several advantages: each function scales independently, failures in one step do not block others, and the DLQ catches any messages that fail processing three times for manual review.

Cost Optimization Strategies

Lambda pricing is based on three factors: number of invocations ($0.20 per million), duration (GB-seconds at $0.0000166667), and any provisioned concurrency. Most teams overpay because they have not optimized duration.

Quick wins for reducing Lambda costs:

Right-size memory: Use Power Tuning as described above. The optimal memory setting minimizes cost, not memory usage.

Reduce execution time: Every millisecond counts. Cache expensive lookups, use connection pooling, and avoid synchronous waits.

// BAD: Sequential external calls
const user = await getUser(userId);       // 50ms
const orders = await getOrders(userId);   // 80ms
const prefs = await getPreferences(userId); // 30ms
// Total: 160ms

// GOOD: Parallel external calls
const [user, orders, prefs] = await Promise.all([
  getUser(userId),       // 50ms
  getOrders(userId),     // 80ms  (runs in parallel)
  getPreferences(userId) // 30ms  (runs in parallel)
]);
// Total: 80ms (limited by slowest call)

Use ARM64 (Graviton2): Lambda functions on ARM64 are 20% cheaper and typically 10-20% faster. Switching is a one-line change in most cases:

aws lambda update-function-configuration \
  --function-name my-function \
  --architectures arm64

Batch SQS processing: Instead of invoking one Lambda per message, process batches. With ReportBatchItemFailures, failed messages get retried individually while successful ones are removed.

Review CloudWatch Logs costs: Lambda logging to CloudWatch can cost more than the Lambda invocations themselves at scale. Set log retention policies and filter what you log:

# Set log retention to 14 days (default is never expire)
aws logs put-retention-policy \
  --log-group-name /aws/lambda/my-function \
  --retention-in-days 14

Monitoring and Alerting Essentials

Every production Lambda function needs these CloudWatch alarms:

# CloudFormation for essential Lambda alarms
ErrorRateAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmName: !Sub "${FunctionName}-error-rate"
    MetricName: Errors
    Namespace: AWS/Lambda
    Statistic: Sum
    Period: 300
    EvaluationPeriods: 2
    Threshold: 5
    ComparisonOperator: GreaterThanThreshold
    Dimensions:
      - Name: FunctionName
        Value: !Ref ProcessOrderFunction

ThrottleAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmName: !Sub "${FunctionName}-throttles"
    MetricName: Throttles
    Namespace: AWS/Lambda
    Statistic: Sum
    Period: 60
    EvaluationPeriods: 1
    Threshold: 0
    ComparisonOperator: GreaterThanThreshold

DurationAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmName: !Sub "${FunctionName}-duration-p99"
    MetricName: Duration
    Namespace: AWS/Lambda
    ExtendedStatistic: p99
    Period: 300
    EvaluationPeriods: 3
    Threshold: 5000
    ComparisonOperator: GreaterThanThreshold

IteratorAgeAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmName: !Sub "${FunctionName}-iterator-age"
    MetricName: IteratorAge
    Namespace: AWS/Lambda
    Statistic: Maximum
    Period: 60
    EvaluationPeriods: 5
    Threshold: 60000
    ComparisonOperator: GreaterThanThreshold

The Iterator Age alarm is critical for functions consuming from Kinesis or DynamoDB Streams. A rising iterator age means your function is falling behind the event stream, which eventually leads to data loss.

Need Help with Your DevOps?

Lambda is deceptively simple to start with but requires real expertise to run efficiently at scale. At InstaDevOps, we help teams design and optimize serverless architectures that are fast, reliable, and cost-effective. From cold start optimization to event-driven architecture design, we have seen it all.

We offer fractional DevOps engineering starting at $2,999/month with no long-term contracts. Book a free 15-minute call to discuss your serverless infrastructure: https://calendly.com/instadevops/15min