Forem: InstaDevOps

Building a DevSecOps Pipeline: Shift Security Left Without Slowing Down

InstaDevOps — Thu, 16 Apr 2026 13:46:46 +0000

Introduction

Every week brings news of another data breach, supply chain attack, or misconfigured cloud resource exposing millions of records. The traditional approach of bolting security on at the end of the development cycle no longer works. By the time a penetration tester finds a vulnerability in staging, the code has been through weeks of development, and fixing it means expensive rework.

DevSecOps integrates security testing directly into your CI/CD pipeline so vulnerabilities are caught in minutes, not months. The goal is not to slow down development - it is to catch security issues at the same speed you catch bugs: automatically, on every commit.

This guide walks through building a practical DevSecOps pipeline with real tool configurations, covering static analysis, dependency scanning, container image scanning, infrastructure security, and policy-as-code.

The DevSecOps Pipeline Stages

A mature DevSecOps pipeline runs security checks at every stage:

Code Commit → SAST → Dependency Scan → Build → Container Scan → IaC Scan → DAST → Deploy → Runtime Protection

You do not need all of these on day one. Start with the highest-impact, lowest-friction stages and layer on additional checks as your team matures.

Stage 1: Static Application Security Testing (SAST)

SAST tools analyze your source code for vulnerabilities without executing it. They catch issues like SQL injection, cross-site scripting, hardcoded secrets, and insecure cryptographic patterns.

Semgrep for Custom and Community Rules

Semgrep is fast, supports 30+ languages, and lets you write custom rules in a simple YAML syntax:

# .github/workflows/sast.yml
name: SAST Scan
on: [pull_request]

jobs:
  semgrep:
    runs-on: ubuntu-latest
    container:
      image: semgrep/semgrep
    steps:
      - uses: actions/checkout@v4
      - run: semgrep scan --config auto --error --sarif --output semgrep.sarif .
      - uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: semgrep.sarif
        if: always()

The --config auto flag uses Semgrep's curated community ruleset. For stricter scanning, add specific rule packs:

semgrep scan \
  --config p/security-audit \
  --config p/secrets \
  --config p/owasp-top-ten \
  --error .

Writing Custom Security Rules

Create organization-specific rules for patterns your team should avoid:

# .semgrep/custom-rules.yml
rules:
  - id: no-raw-sql-queries
    patterns:
      - pattern: |
          db.query($SQL, ...)
      - pattern-not: |
          db.query($SQL, [...])
    message: "Use parameterized queries to prevent SQL injection"
    severity: ERROR
    languages: [javascript, typescript]

  - id: no-console-log-in-production
    pattern: console.log(...)
    message: "Remove console.log before merging to main"
    severity: WARNING
    languages: [javascript, typescript]
    paths:
      include:
        - src/

Stage 2: Dependency and Supply Chain Scanning

Your application's dependencies are a massive attack surface. A single vulnerable transitive dependency can compromise your entire system.

Scanning with Trivy

Trivy scans not just containers but also filesystems for vulnerable dependencies:

# .github/workflows/dependency-scan.yml
jobs:
  scan-dependencies:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Trivy filesystem scan
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: fs
          scan-ref: .
          severity: HIGH,CRITICAL
          exit-code: 1
          format: sarif
          output: trivy-fs.sarif

      - uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: trivy-fs.sarif
        if: always()

Software Bill of Materials (SBOM)

Generate an SBOM for compliance and vulnerability tracking:

# Generate SBOM with Syft
syft packages dir:. -o spdx-json > sbom.spdx.json

# Scan the SBOM for vulnerabilities with Grype
grype sbom:sbom.spdx.json --fail-on high

Automated Dependency Updates

Use Renovate or Dependabot to keep dependencies current:

// renovate.json
{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:recommended"],
  "vulnerabilityAlerts": {
    "enabled": true,
    "labels": ["security"]
  },
  "packageRules": [
    {
      "matchUpdateTypes": ["patch"],
      "automerge": true,
      "automergeType": "branch"
    },
    {
      "matchUpdateTypes": ["major"],
      "labels": ["breaking-change"],
      "assignees": ["team-lead"]
    }
  ]
}

Stage 3: Container Image Scanning

Container images often contain vulnerable OS packages, misconfigured permissions, and unnecessary tools that increase the attack surface.

Scanning During Build

jobs:
  build-and-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build image
        run: docker build -t myapp:${{ github.sha }} .

      - name: Scan image with Trivy
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: myapp:${{ github.sha }}
          severity: HIGH,CRITICAL
          exit-code: 1

      - name: Check for root user
        run: |
          USER=$(docker inspect --format='{{.Config.User}}' myapp:${{ github.sha }})
          if [ -z "$USER" ] || [ "$USER" = "root" ]; then
            echo "ERROR: Container runs as root. Add a USER directive to your Dockerfile."
            exit 1
          fi

Hardening Dockerfiles

A secure Dockerfile follows these patterns:

# Use specific version, not :latest
FROM node:20.11-alpine AS builder

WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

# Multi-stage build - no build tools in final image
FROM node:20.11-alpine

# Create non-root user
RUN addgroup -g 1001 -S appuser && \
    adduser -S appuser -u 1001 -G appuser

WORKDIR /app
COPY --from=builder /app/node_modules ./node_modules
COPY --chown=appuser:appuser . .

# Drop all capabilities
USER appuser

# Use dumb-init to handle PID 1 properly
RUN apk add --no-cache dumb-init
ENTRYPOINT ["dumb-init", "--"]
CMD ["node", "server.js"]

Stage 4: Infrastructure as Code Security

Misconfigured infrastructure is the leading cause of cloud breaches. Scan your Terraform, CloudFormation, and Kubernetes manifests before they reach production.

Checkov for IaC Scanning

jobs:
  iac-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Checkov
        uses: bridgecrewio/checkov-action@v12
        with:
          directory: terraform/
          framework: terraform
          soft_fail: false
          output_format: sarif
          output_file_path: checkov.sarif

      - uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: checkov.sarif
        if: always()

Checkov catches misconfigurations like:

S3 buckets without encryption
Security groups open to 0.0.0.0/0
RDS instances without encryption at rest
IAM policies with wildcard permissions
EBS volumes without encryption

tfsec for Terraform-Specific Scanning

# Run tfsec on your Terraform directory
tfsec terraform/ --format sarif --out tfsec.sarif

# Common findings:
# - aws-ec2-no-public-ingress-sgr
# - aws-s3-enable-bucket-encryption
# - aws-iam-no-policy-wildcards
# - aws-rds-encrypt-instance-storage-data

Stage 5: Dynamic Application Security Testing (DAST)

DAST tools test your running application by sending requests and analyzing responses for vulnerabilities. Run these against a staging environment after deployment.

OWASP ZAP Baseline Scan

jobs:
  dast:
    runs-on: ubuntu-latest
    needs: deploy-staging
    steps:
      - name: OWASP ZAP Baseline Scan
        uses: zaproxy/action-baseline@v0.12.0
        with:
          target: https://staging.example.com
          rules_file_name: zap-rules.tsv
          fail_action: true
          cmd_options: '-a -j'

DAST scans are slower (minutes to hours) and noisier than SAST, so run them post-deployment rather than on every commit.

Stage 6: Policy-as-Code with Open Policy Agent

Policy-as-Code enforces organizational rules programmatically. Open Policy Agent (OPA) lets you write policies in Rego that gate deployments:

# policy/kubernetes.rego
package kubernetes.admission

deny[msg] {
  input.request.kind.kind == "Deployment"
  container := input.request.object.spec.template.spec.containers[_]
  not container.resources.limits.memory
  msg := sprintf("Container '%v' must have memory limits set", [container.name])
}

deny[msg] {
  input.request.kind.kind == "Deployment"
  container := input.request.object.spec.template.spec.containers[_]
  container.image
  not contains(container.image, "@sha256:")
  not regex.match("^.*:[0-9]+\\.[0-9]+\\.[0-9]+$", container.image)
  msg := sprintf("Container '%v' must use a specific version tag or digest, not :latest", [container.name])
}

deny[msg] {
  input.request.kind.kind == "Deployment"
  container := input.request.object.spec.template.spec.containers[_]
  container.securityContext.privileged == true
  msg := sprintf("Container '%v' must not run in privileged mode", [container.name])
}

Conftest for CI Integration

Use Conftest to run OPA policies against Kubernetes manifests, Terraform plans, and Dockerfiles in CI:

# Test Kubernetes manifests
conftest test k8s/deployment.yaml --policy policy/

# Test Terraform plans
terraform plan -out=tfplan
terraform show -json tfplan > tfplan.json
conftest test tfplan.json --policy policy/terraform/

# Test Dockerfiles
conftest test Dockerfile --policy policy/docker/

Putting It All Together

Here is a complete pipeline that ties all stages together:

name: DevSecOps Pipeline
on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

jobs:
  # Fast checks first (< 2 minutes)
  secrets-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: trufflesecurity/trufflehog@main
        with:
          extra_args: --only-verified

  sast:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          docker run --rm -v "${PWD}:/src" semgrep/semgrep \
            semgrep scan --config auto --error /src

  # Medium checks (2-5 minutes)
  dependency-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aquasecurity/trivy-action@master
        with:
          scan-type: fs
          severity: HIGH,CRITICAL
          exit-code: 1

  iac-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: bridgecrewio/checkov-action@v12
        with:
          directory: terraform/

  # Build and container scan
  build:
    needs: [secrets-scan, sast, dependency-scan, iac-scan]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t app:${{ github.sha }} .
      - uses: aquasecurity/trivy-action@master
        with:
          image-ref: app:${{ github.sha }}
          severity: HIGH,CRITICAL
          exit-code: 1

  # Deploy to staging, then DAST
  deploy-staging:
    needs: build
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - run: echo "Deploy to staging"

  dast:
    needs: deploy-staging
    runs-on: ubuntu-latest
    steps:
      - uses: zaproxy/action-baseline@v0.12.0
        with:
          target: https://staging.example.com

Managing False Positives

Every security scanning tool produces false positives. If you do not manage them, your team will start ignoring security alerts entirely.

Create a baseline. On initial integration, capture existing findings as a baseline and only alert on new issues.

Use inline suppressions with justification. Every suppressed finding should include a reason:

# nosemgrep: python.lang.security.audit.insecure-hash.insecure-hash
# Justification: MD5 used for non-security cache key, not for cryptographic purposes
cache_key = hashlib.md5(data).hexdigest()

Triage weekly. Assign someone to review and triage findings weekly. Close false positives, create tickets for real issues, and tune rules that produce too much noise.

Security Metrics That Matter

Tracking the right metrics tells you whether your DevSecOps pipeline is actually improving security or just generating noise.

Mean Time to Remediate (MTTR)

How long does it take from when a vulnerability is detected to when it is fixed in production? Track this by severity:

Critical: Target under 24 hours
High: Target under 7 days
Medium: Target under 30 days
Low: Target under 90 days

Vulnerability Escape Rate

What percentage of vulnerabilities make it past your CI/CD pipeline into production? This is your pipeline's effectiveness metric. Track it by comparing CI findings against production security scans and penetration test results.

Coverage Percentage

What percentage of your repositories have security scanning enabled? In most organizations, the answer is disturbingly low. Aim for 100% of production services covered by at minimum dependency scanning and SAST.

Fix Rate vs Find Rate

If you are finding vulnerabilities faster than you are fixing them, your backlog will grow indefinitely. This signals that your pipeline is too noisy (tune it) or your team needs more security training.

Getting Started: A Phased Approach

Do not try to implement everything at once. Here is a realistic rollout plan:

Week 1-2: Secrets scanning and dependency scanning. These have the highest signal-to-noise ratio and catch the most impactful issues. Enable Dependabot or Renovate for automated updates.

Week 3-4: SAST with Semgrep. Start with the default auto config. Do not write custom rules yet. Let the team get comfortable with the findings.

Month 2: Container image scanning. Add Trivy to your Docker build pipeline. Fix the critical and high findings, suppress the false positives with documentation.

Month 3: IaC scanning with Checkov. Scan your Terraform and Kubernetes manifests. This catches misconfigurations before they reach production.

Month 4+: DAST and policy-as-code. These require more setup and tuning but provide the deepest security coverage.

Need Help with Your DevOps?

Building a DevSecOps pipeline that catches real vulnerabilities without drowning your team in false positives takes careful tuning. At InstaDevOps, we implement security-hardened CI/CD pipelines for startups and growing teams - baking security in from day one.

Plans start at $2,999/mo for a dedicated fractional DevOps engineer.

Book a free 15-minute consultation to discuss your security posture.

AWS ECS vs EKS: A Practical Decision Framework for Startups

InstaDevOps — Wed, 15 Apr 2026 13:46:43 +0000

Introduction

You have containerized your application and now you need to run it in production on AWS. The two main options stare back at you from the AWS console: Elastic Container Service (ECS) and Elastic Kubernetes Service (EKS). Both are fully managed, both run containers, and both integrate deeply with the AWS ecosystem.

So which one should you pick?

The answer is not "EKS because Kubernetes is the industry standard" and it is not "ECS because it is simpler." The right choice depends on your team size, operational maturity, workload characteristics, and growth trajectory. This article gives you a practical framework for making this decision, with real cost numbers and migration considerations.

ECS: The AWS-Native Path

ECS is AWS's proprietary container orchestration service. It is deeply integrated with the AWS ecosystem and was purpose-built to make running containers on AWS as straightforward as possible.

ECS Architecture

ECS has two launch types:

EC2 Launch Type: You manage the underlying EC2 instances. ECS handles container placement and scheduling.

Fargate Launch Type: Serverless containers. AWS manages the infrastructure entirely. You just define CPU, memory, and your container image.

{
  "family": "my-api",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "512",
  "memory": "1024",
  "containerDefinitions": [
    {
      "name": "api",
      "image": "123456789012.dkr.ecr.eu-west-1.amazonaws.com/my-api:latest",
      "portMappings": [
        {
          "containerPort": 3000,
          "protocol": "tcp"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/my-api",
          "awslogs-region": "eu-west-1",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ]
}

ECS Strengths

Zero Kubernetes knowledge required. Your team interacts with familiar AWS concepts: task definitions, services, target groups, IAM roles. There is no kubectl, no YAML manifests with 15 indentation levels, no CRDs.

Deep AWS integration. ECS natively integrates with ALB/NLB, CloudWatch, IAM task roles, Secrets Manager, Parameter Store, App Mesh, and CloudMap for service discovery. These integrations work out of the box with minimal configuration.

Lower operational overhead. Especially with Fargate, there are no nodes to patch, no cluster upgrades to manage, no etcd to worry about. AWS handles all of it.

Simpler networking. The awsvpc network mode gives each task its own ENI and security group. No overlay networks, no kube-proxy, no CNI plugin decisions.

ECS Limitations

Vendor lock-in to AWS
Smaller community and ecosystem compared to Kubernetes
Limited scheduling customization
No equivalent to Kubernetes operators or CRDs for extending the platform
Fewer third-party tools (no Helm, no ArgoCD, no Istio)

EKS: The Kubernetes Path

EKS is AWS's managed Kubernetes service. AWS manages the control plane (API server, etcd, scheduler), and you manage the worker nodes or use Fargate for serverless pods.

EKS Architecture

# Sample Kubernetes deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-api
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-api
  template:
    metadata:
      labels:
        app: my-api
    spec:
      serviceAccountName: my-api
      containers:
        - name: api
          image: 123456789012.dkr.ecr.eu-west-1.amazonaws.com/my-api:latest
          ports:
            - containerPort: 3000
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: 500m
              memory: 1Gi
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 10
          readinessProbe:
            httpGet:
              path: /ready
              port: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: my-api
spec:
  selector:
    app: my-api
  ports:
    - port: 80
      targetPort: 3000

EKS Strengths

Portability. Kubernetes manifests work on any cloud provider or on-premises cluster. If multi-cloud or hybrid-cloud is in your future, EKS keeps your options open.

Rich ecosystem. Helm charts, ArgoCD for GitOps, Istio or Linkerd for service mesh, Prometheus and Grafana for monitoring, cert-manager for TLS, external-dns for DNS automation. The Kubernetes ecosystem is enormous.

Advanced scheduling. Node affinities, taints and tolerations, pod topology spread constraints, priority classes, and preemption give you fine-grained control over workload placement.

Extensibility. Custom Resource Definitions and operators let you extend Kubernetes to manage databases, message queues, ML pipelines, and virtually anything else as native Kubernetes resources.

EKS Limitations

$0.10/hour ($73/month) control plane cost before any compute
Steep learning curve (expect 3-6 months for a team new to Kubernetes)
Cluster upgrades require planning and testing (new version every 4 months)
More moving parts: CNI plugins, ingress controllers, cluster autoscaler, DNS
Significantly higher operational burden

Cost Comparison

Let us compare the cost of running a typical startup workload: 3 services, each running 2 replicas with 0.5 vCPU and 1 GB RAM.

ECS Fargate

Per task: 0.5 vCPU + 1 GB RAM
Fargate pricing (eu-west-1):
  vCPU: $0.04048/hour
  Memory: $0.004445/GB/hour

Per task/hour: (0.5 * $0.04048) + (1 * $0.004445) = $0.02469
6 tasks total: $0.02469 * 6 = $0.14814/hour

Monthly: $0.14814 * 730 = $108.14/month

ECS EC2 (with Reserved Instances)

2x t3.medium (2 vCPU, 4 GB) with 1-year RI:
  On-demand: $0.0416/hour each
  1-year RI (no upfront): $0.027/hour each

Monthly: 2 * $0.027 * 730 = $39.42/month

EKS with Managed Node Group

EKS control plane: $0.10/hour = $73/month
2x t3.medium nodes (same as ECS EC2):
  1-year RI: 2 * $0.027 * 730 = $39.42/month

Monthly total: $73 + $39.42 = $112.42/month

EKS Fargate

EKS control plane: $73/month
Fargate pods (same as ECS Fargate): $108.14/month
Plus CoreDNS pods on Fargate: ~$15/month

Monthly total: $73 + $108.14 + $15 = $196.14/month

At small scale, ECS on EC2 is the cheapest option by far. EKS adds a fixed $73/month control plane cost that only becomes negligible at larger scales.

The Decision Framework

Choose ECS When

Your team is small (under 10 engineers). The operational overhead of Kubernetes is not justified when a few people can manage ECS with part-time attention.

You are all-in on AWS. If you have no plans for multi-cloud and are already using ALB, CloudWatch, and IAM extensively, ECS slots in naturally.

You want Fargate's simplicity. ECS Fargate is the easiest path from container image to running workload. No nodes, no cluster management, no upgrades.

Your workloads are straightforward. Web APIs, background workers, and scheduled tasks that do not need advanced scheduling or custom operators.

You need to ship fast. A team new to containers can have ECS running in production within a week. Kubernetes takes months to operate confidently.

Choose EKS When

Multi-cloud or hybrid is a real requirement. Not a hypothetical future possibility, but an actual business constraint.

You have Kubernetes experience on the team. If your engineers already know Kubernetes, EKS employs that knowledge. Forcing Kubernetes-experienced engineers to learn ECS is a lateral move with no payoff.

You need the ecosystem. If your architecture requires service mesh, GitOps (ArgoCD), advanced traffic management, or custom operators, the Kubernetes ecosystem is unmatched.

You are running 20+ services. At scale, Kubernetes' sophisticated scheduling, resource management, and namespace isolation become genuine advantages.

You are building a platform team. If part of your strategy is building an internal developer platform, Kubernetes provides the extensibility to build abstractions on top.

Migration Paths

ECS to EKS

If you outgrow ECS, migration is manageable:

Set up EKS cluster alongside existing ECS
Convert task definitions to Kubernetes deployments (the container config is similar)
Use AWS ALB Ingress Controller to maintain the same load balancer
Migrate services one at a time, shifting traffic gradually
Decommission ECS services after verification

EKS to ECS

Rarer, but it happens when teams realize they over-invested in Kubernetes:

Convert Kubernetes deployments to ECS task definitions
Replace Kubernetes services with ECS services + ALB target groups
Replace Helm/ArgoCD with AWS CodePipeline or similar
Migrate ConfigMaps/Secrets to AWS Parameter Store/Secrets Manager

A Pragmatic Recommendation

For most startups and SMBs running fewer than 15 services on AWS, start with ECS. Here is why:

You avoid 3-6 months of Kubernetes learning curve
You save $73/month in control plane costs (trivial individually, but symbolic of broader overhead)
You eliminate cluster upgrade maintenance windows
Your engineers focus on product features instead of infrastructure plumbing
If you outgrow ECS, the migration to EKS is well-understood

The exception: if you already have a team member who is deeply experienced with Kubernetes and can own the cluster operations, EKS is fine from day one.

Operational Overhead: What Nobody Tells You

The cost comparison above only covers compute. The real difference between ECS and EKS is operational overhead - the time your engineers spend managing the orchestration platform itself rather than building product features.

ECS Day-2 Operations

With ECS (especially Fargate), your ongoing operational tasks are minimal:

Task definition updates when you change container config (straightforward JSON/Terraform changes)
Service scaling policies adjustments based on traffic patterns
ALB health check tuning to match your application's startup time
Security group and IAM role updates as your network requirements change

Most of this is standard AWS work that any cloud engineer can handle.

EKS Day-2 Operations

EKS adds a significant layer of Kubernetes-specific operational work:

Cluster version upgrades every 4 months (Kubernetes drops support for old versions aggressively). Each upgrade requires testing all workloads, updating add-ons, and often updating manifests for deprecated APIs.
Add-on management: CoreDNS, kube-proxy, VPC CNI, cluster autoscaler, ingress controller, cert-manager, external-dns - each has its own version matrix and upgrade path.
Node group rotation when you change AMIs or instance types. Cordon, drain, replace.
RBAC policy management as teams and services grow.
etcd monitoring for the control plane (managed by AWS, but you still need to understand its implications for API server performance).
Debugging Kubernetes networking when services cannot reach each other (is it a NetworkPolicy? A CNI issue? A security group? A missing service account?).

A reasonable estimate: EKS adds 10-20 hours per month of platform engineering work compared to ECS. At a senior engineer's loaded cost, that is $3,000-6,000/month in operational overhead that does not show up on your AWS bill.

Real-World Scenarios

Scenario 1: 4-Person Startup, 2 Services

Choose ECS Fargate. You do not have the engineering bandwidth to learn and operate Kubernetes. ECS Fargate lets you deploy containers in an afternoon and forget about infrastructure management for the next 12 months.

Scenario 2: 25-Person Scale-Up, 12 Services, AWS Only

Choose ECS EC2. You are big enough that Fargate costs add up, but your services are straightforward enough that Kubernetes' advanced features are not needed. ECS on EC2 with Reserved Instances gives you the best cost efficiency.

Scenario 3: 50-Person Company, 30 Services, Multi-Cloud Mandate

Choose EKS. At this scale, you likely have or can hire a dedicated platform engineer. Kubernetes' portability and ecosystem justify the operational investment, and you will benefit from tools like ArgoCD, Istio, and custom operators.

Scenario 4: Regulated Industry, Compliance Requirements

EKS edges ahead. Kubernetes' namespace isolation, network policies, pod security standards, and OPA Gatekeeper provide a richer set of compliance controls. Combined with tools like Falco for runtime security, you can build a compliance story that auditors understand and respect.

Need Help with Your DevOps?

Choosing the right container orchestration platform is just the first decision. At InstaDevOps, we help startups set up production-grade container infrastructure on AWS - whether that is ECS, EKS, or a migration between the two.

Plans start at $2,999/mo for a dedicated fractional DevOps engineer.

Book a free 15-minute consultation to discuss your container strategy.

Advanced GitHub Actions: Matrix Builds, Reusable Workflows, and Self-Hosted Runners

InstaDevOps — Tue, 14 Apr 2026 13:46:40 +0000

Introduction

GitHub Actions has matured from a simple CI tool into a full-featured automation platform. Most teams start with a basic build-and-test workflow, but GitHub Actions offers far more powerful patterns that can dramatically reduce pipeline duplication, speed up builds, and cut CI costs.

This article covers advanced patterns that separate hobby-level GitHub Actions usage from production-grade CI/CD: matrix builds for testing across multiple environments, reusable workflows for eliminating duplication, self-hosted runners for cost and performance, and secrets management strategies that keep your pipelines secure.

If you are already comfortable writing basic GitHub Actions workflows, this guide will take you to the next level.

Matrix Builds: Test Everything in Parallel

Matrix builds let you run the same job across multiple combinations of variables - OS versions, language versions, dependency versions - without duplicating workflow code.

Basic Matrix Strategy

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [18, 20, 22]
        os: [ubuntu-latest, macos-latest]
      fail-fast: false
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm ci
      - run: npm test

Setting fail-fast: false ensures all matrix combinations run to completion even if one fails. This is critical for understanding the full scope of compatibility issues.

Dynamic Matrix Generation

For more complex scenarios, generate the matrix dynamically from a previous job:

jobs:
  detect-services:
    runs-on: ubuntu-latest
    outputs:
      services: ${{ steps.find.outputs.services }}
    steps:
      - uses: actions/checkout@v4
      - id: find
        run: |
          # Find all directories with a Dockerfile
          SERVICES=$(find services/ -name Dockerfile -exec dirname {} \; | jq -R -s -c 'split("\n")[:-1]')
          echo "services=$SERVICES" >> $GITHUB_OUTPUT

  build:
    needs: detect-services
    runs-on: ubuntu-latest
    strategy:
      matrix:
        service: ${{ fromJson(needs.detect-services.outputs.services) }}
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t ${{ matrix.service }}:latest ${{ matrix.service }}

This pattern is invaluable for monorepos: only build and test services that have changed, and automatically pick up new services without modifying the workflow.

Matrix Include and Exclude

Fine-tune your matrix with include and exclude rules:

strategy:
  matrix:
    os: [ubuntu-latest, windows-latest]
    node: [18, 20]
    include:
      # Add an experimental Node 22 build on Ubuntu only
      - os: ubuntu-latest
        node: 22
        experimental: true
    exclude:
      # Skip Node 18 on Windows (known incompatibility)
      - os: windows-latest
        node: 18

Reusable Workflows: Eliminate Duplication

If you have 10 microservices with nearly identical CI workflows, reusable workflows let you define the pipeline once and call it from each repository.

Creating a Reusable Workflow

Define a workflow with workflow_call trigger in a shared repository:

# .github/workflows/build-and-deploy.yml in org/shared-workflows repo
name: Build and Deploy Service
on:
  workflow_call:
    inputs:
      service-name:
        required: true
        type: string
      dockerfile-path:
        required: false
        type: string
        default: './Dockerfile'
      environment:
        required: true
        type: string
    secrets:
      AWS_ACCESS_KEY_ID:
        required: true
      AWS_SECRET_ACCESS_KEY:
        required: true
      ECR_REGISTRY:
        required: true

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: eu-west-1

      - name: Login to ECR
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build and push image
        run: |
          IMAGE="${{ secrets.ECR_REGISTRY }}/${{ inputs.service-name }}:${{ github.sha }}"
          docker build -f ${{ inputs.dockerfile-path }} -t $IMAGE .
          docker push $IMAGE

  deploy:
    needs: build
    runs-on: ubuntu-latest
    environment: ${{ inputs.environment }}
    steps:
      - name: Deploy to ECS
        run: |
          aws ecs update-service \
            --cluster ${{ inputs.environment }}-cluster \
            --service ${{ inputs.service-name }} \
            --force-new-deployment

Calling a Reusable Workflow

Each service repository calls the shared workflow:

# .github/workflows/ci.yml in service repo
name: CI/CD
on:
  push:
    branches: [main]

jobs:
  deploy-staging:
    uses: org/shared-workflows/.github/workflows/build-and-deploy.yml@v2.1.0
    with:
      service-name: payment-api
      environment: staging
    secrets:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      ECR_REGISTRY: ${{ secrets.ECR_REGISTRY }}

  deploy-production:
    needs: deploy-staging
    uses: org/shared-workflows/.github/workflows/build-and-deploy.yml@v2.1.0
    with:
      service-name: payment-api
      environment: production
    secrets:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      ECR_REGISTRY: ${{ secrets.ECR_REGISTRY }}

Pin the reusable workflow to a specific tag (@v2.1.0) so that upstream changes do not break your pipelines unexpectedly.

Self-Hosted Runners: Performance and Cost Control

GitHub-hosted runners are convenient but come with limitations: fixed hardware specs, cold start times, no persistent caches, and costs that scale linearly with usage. Self-hosted runners solve these problems.

When Self-Hosted Runners Make Sense

Large Docker builds that benefit from persistent layer caches
GPU workloads for ML model training or testing
Network-restricted environments where jobs need access to internal resources
High CI volume where GitHub-hosted runner costs exceed $1,000/month
Specialized hardware requirements

Setting Up Ephemeral Self-Hosted Runners on AWS

Use the actions-runner-controller (ARC) to autoscale runners on Kubernetes, or use EC2 with a simpler setup:

# Terraform for self-hosted runner ASG
resource "aws_launch_template" "runner" {
  name_prefix   = "github-runner-"
  image_id      = data.aws_ami.ubuntu.id
  instance_type = "c6i.2xlarge"

  user_data = base64encode(templatefile("runner-init.sh", {
    github_org    = var.github_org
    runner_token  = var.runner_registration_token
    runner_labels = "self-hosted,linux,x64,large"
  }))

  block_device_mappings {
    device_name = "/dev/sda1"
    ebs {
      volume_size = 100
      volume_type = "gp3"
    }
  }
}

resource "aws_autoscaling_group" "runners" {
  desired_capacity = 2
  max_size         = 10
  min_size         = 0

  launch_template {
    id      = aws_launch_template.runner.id
    version = "$Latest"
  }

  tag {
    key                 = "Purpose"
    value               = "github-actions-runner"
    propagate_at_launch = true
  }
}

Using Runner Labels for Job Routing

Target specific runners using labels:

jobs:
  unit-tests:
    # Fast tests on GitHub-hosted runners
    runs-on: ubuntu-latest
    steps:
      - run: npm test

  docker-build:
    # Heavy builds on self-hosted with Docker cache
    runs-on: [self-hosted, linux, large]
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t myapp:latest .

  gpu-tests:
    # ML tests on GPU runners
    runs-on: [self-hosted, gpu]
    steps:
      - run: python -m pytest tests/ml/

Secrets Management Patterns

Secrets handling in CI/CD is a top security concern. GitHub Actions provides several mechanisms, but you need to layer them correctly.

Environment-Scoped Secrets

Use GitHub environments to scope secrets to specific deployment targets:

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production  # Only secrets defined in "production" environment are available
    steps:
      - name: Deploy
        env:
          DB_PASSWORD: ${{ secrets.DB_PASSWORD }}  # Production-specific secret
        run: ./deploy.sh

Configure required reviewers on the production environment so deployments require manual approval.

OIDC Authentication (No Long-Lived Keys)

Eliminate static AWS credentials entirely using OpenID Connect:

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsRole
          aws-region: eu-west-1
          # No access key or secret needed - uses OIDC token exchange

The IAM role trust policy restricts which repositories and branches can assume it:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
      },
      "StringLike": {
        "token.actions.githubusercontent.com:sub": "repo:yourorg/yourrepo:ref:refs/heads/main"
      }
    }
  }]
}

Secrets Scanning and Rotation

Add a workflow that scans for accidentally committed secrets:

jobs:
  secrets-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: trufflesecurity/trufflehog@main
        with:
          extra_args: --only-verified

Caching Strategies for Faster Builds

Effective caching can cut build times by 50-80%.

Dependency Caching

- uses: actions/cache@v4
  with:
    path: |
      ~/.npm
      node_modules
    key: deps-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      deps-${{ runner.os }}-

Docker Layer Caching

- uses: docker/build-push-action@v5
  with:
    context: .
    push: true
    tags: myapp:latest
    cache-from: type=gha
    cache-to: type=gha,mode=max

Artifact Passing Between Jobs

Use artifacts to pass build outputs between jobs without rebuilding:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - run: npm run build
      - uses: actions/upload-artifact@v4
        with:
          name: build-output
          path: dist/
          retention-days: 1

  deploy:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: build-output
          path: dist/
      - run: ./deploy.sh dist/

Workflow Organization at Scale

Path-Based Triggers

Only run workflows when relevant files change:

on:
  push:
    branches: [main]
    paths:
      - 'services/payment-api/**'
      - '.github/workflows/payment-api.yml'

Concurrency Control

Prevent redundant workflow runs when commits are pushed rapidly:

concurrency:
  group: deploy-${{ github.ref }}
  cancel-in-progress: true

This cancels the previous run when a new commit is pushed to the same branch, saving runner minutes.

Composite Actions for Shared Steps

When full reusable workflows are overkill, composite actions bundle common steps:

# .github/actions/setup-project/action.yml
name: Setup Project
description: Install dependencies and configure environment
inputs:
  node-version:
    required: false
    default: '20'
runs:
  using: composite
  steps:
    - uses: actions/setup-node@v4
      with:
        node-version: ${{ inputs.node-version }}
    - uses: actions/cache@v4
      with:
        path: node_modules
        key: deps-${{ hashFiles('package-lock.json') }}
    - run: npm ci
      shell: bash

Use it in any workflow with a single line:

- uses: ./.github/actions/setup-project
  with:
    node-version: '20'

Monitoring and Debugging Workflows

When workflows fail in production, you need fast diagnosis. GitHub provides several tools for this.

Step-Level Timing Analysis

Every workflow run shows timing per step. If your pipeline is slow, look for steps that take disproportionate time. Common culprits:

Checkout with full history (fetch-depth: 0) on large repos - use shallow clones unless you need full history
Docker builds without caching - always use BuildKit cache with cache-from and cache-to
NPM/pip installs without cache - dependency caching alone can save 30-60 seconds per run

Workflow Run Logs and Annotations

Use ::warning and ::error workflow commands to surface issues directly in PR annotations:

- name: Check bundle size
  run: |
    SIZE=$(stat -f%z dist/bundle.js)
    if [ "$SIZE" -gt 500000 ]; then
      echo "::warning file=dist/bundle.js::Bundle size is ${SIZE} bytes (over 500KB limit)"
    fi

Reusable Debug Workflow

Create a debug composite action that dumps useful context when things go wrong:

- name: Debug info
  if: failure()
  run: |
    echo "## Environment" >> $GITHUB_STEP_SUMMARY
    echo "- Runner: ${{ runner.os }} ${{ runner.arch }}" >> $GITHUB_STEP_SUMMARY
    echo "- Node: $(node --version)" >> $GITHUB_STEP_SUMMARY
    echo "- Disk: $(df -h / | tail -1)" >> $GITHUB_STEP_SUMMARY
    echo "- Memory: $(free -h 2>/dev/null || vm_stat)" >> $GITHUB_STEP_SUMMARY

Common Pitfalls to Avoid

Not pinning action versions. Using actions/checkout@main means your workflow can break without any change on your side. Always pin to a specific version tag or commit SHA for third-party actions.

Leaking secrets in logs. GitHub automatically masks secrets in logs, but derived values (like a URL containing a token) are not masked. Use ::add-mask:: for any sensitive derived values.

Ignoring workflow permissions. The default GITHUB_TOKEN has broad permissions. Use the permissions key to restrict to only what each job needs:

permissions:
  contents: read
  pull-requests: write

Running everything on push and pull_request. This causes duplicate runs. Use push for deploys (main branch only) and pull_request for checks.

Not using job summaries. The $GITHUB_STEP_SUMMARY file lets you write rich Markdown summaries that appear in the workflow run UI - much better than scrolling through raw logs.

Need Help with Your DevOps?

Setting up production-grade CI/CD pipelines with proper security, caching, and scaling takes real-world experience. At InstaDevOps, we build and maintain CI/CD infrastructure for startups and growing teams so your developers can focus on shipping features.

Plans start at $2,999/mo for a dedicated fractional DevOps engineer.

Book a free 15-minute consultation to optimize your GitHub Actions workflows.

Terraform Modules Done Right: Mono-Repo, Versioning, and Registry Patterns

InstaDevOps — Mon, 13 Apr 2026 13:46:36 +0000

Introduction

As your Terraform codebase grows beyond a handful of resources, you inevitably face a structural decision: how do you organize reusable infrastructure components so that multiple teams can collaborate without stepping on each other's toes?

The answer is Terraform modules. But writing a module is easy - organizing, versioning, and distributing them at scale is where most teams struggle. Poorly structured modules lead to copy-paste drift, version conflicts, and infrastructure that nobody fully understands.

In this guide, we will walk through battle-tested patterns for organizing Terraform modules, choosing between mono-repo and multi-repo strategies, leveraging module registries, and implementing versioning that actually works in production.

What Makes a Good Terraform Module

Before discussing organization, let us establish what a well-designed module looks like. A good Terraform module follows these principles:

Single responsibility. Each module should manage one logical piece of infrastructure. A VPC module should not also create RDS instances.

Sensible defaults with full override capability. Provide defaults that work for 80% of use cases, but allow every significant parameter to be overridden:

variable "instance_type" {
  description = "EC2 instance type for the application servers"
  type        = string
  default     = "t3.medium"
}

variable "enable_enhanced_monitoring" {
  description = "Enable enhanced monitoring with 60-second granularity"
  type        = bool
  default     = true
}

variable "tags" {
  description = "Additional tags to apply to all resources"
  type        = map(string)
  default     = {}
}

Clear input/output contracts. Every variable should have a description, type constraint, and validation where appropriate. Every output that downstream consumers need should be explicitly exported:

variable "vpc_cidr" {
  description = "CIDR block for the VPC"
  type        = string

  validation {
    condition     = can(cidrhost(var.vpc_cidr, 0))
    error_message = "Must be a valid CIDR block."
  }
}

output "vpc_id" {
  description = "ID of the created VPC"
  value       = aws_vpc.main.id
}

output "private_subnet_ids" {
  description = "List of private subnet IDs"
  value       = aws_subnet.private[*].id
}

Minimal provider assumptions. Do not hardcode provider configurations inside modules. Let the caller configure the provider:

# Bad - hardcoded provider in module
provider "aws" {
  region = "us-east-1"
}

# Good - module relies on caller's provider configuration
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.0"
    }
  }
}

Mono-Repo vs Multi-Repo: Choosing the Right Strategy

This is the most debated structural decision in Terraform module management. Both approaches have legitimate trade-offs.

Mono-Repo Pattern

All modules live in a single repository with a directory structure like:

terraform-modules/
├── modules/
│   ├── vpc/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── README.md
│   ├── ecs-service/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── README.md
│   ├── rds-postgres/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── README.md
│   └── s3-bucket/
│       ├── main.tf
│       ├── variables.tf
│       ├── outputs.tf
│       └── README.md
├── examples/
│   ├── complete-vpc/
│   └── ecs-with-rds/
├── tests/
│   ├── vpc_test.go
│   └── ecs_test.go
└── .github/
    └── workflows/
        └── test.yml

Advantages:

Atomic cross-module changes in a single PR
Unified CI/CD pipeline for testing
Easier to maintain consistency across modules
Simpler onboarding for new team members
One place to search for all infrastructure patterns

Disadvantages:

Git tags version the entire repo, not individual modules
Large repos can slow down CI runs
Permission boundaries are harder (everyone can see everything)

When to use: Teams under 20 engineers, organizations with fewer than 30 modules, or when most modules are tightly coupled.

Multi-Repo Pattern

Each module gets its own repository:

terraform-module-vpc/          (v2.3.1)
terraform-module-ecs-service/  (v1.8.0)
terraform-module-rds-postgres/ (v3.1.0)
terraform-module-s3-bucket/    (v1.2.4)

Advantages:

Independent versioning per module using Git tags
Fine-grained access control per repository
Smaller, focused CI/CD pipelines
Clear ownership boundaries

Disadvantages:

Cross-module changes require multiple PRs
Dependency management becomes complex
More repositories to maintain
Harder to ensure consistency

When to use: Organizations with 50+ modules, multiple platform teams owning different modules, or strict compliance requirements needing audit trails per component.

The Hybrid Approach

Many mature organizations use a hybrid: a mono-repo for foundational modules maintained by the platform team, with separate repos for domain-specific modules owned by product teams:

platform-terraform-modules/     # Platform team owns VPC, IAM, networking
├── modules/vpc/
├── modules/iam-roles/
└── modules/cloudfront/

team-payments-terraform/        # Payments team owns their service modules
├── modules/payment-service/
└── modules/fraud-detection/

Module Versioning Strategies

Versioning is where most Terraform setups break down. Without proper versioning, a module change can silently break every environment that references it.

Semantic Versioning

Follow semver strictly for modules:

MAJOR (v2.0.0): Breaking changes (removed variables, renamed resources that force replacement)
MINOR (v1.3.0): New features, new optional variables with defaults
PATCH (v1.2.1): Bug fixes, documentation updates

Pinning Module Versions

Always pin module versions in your root configurations:

# Good - pinned to exact version
module "vpc" {
  source  = "git::https://github.com/yourorg/terraform-modules.git//modules/vpc?ref=v2.3.1"
}

# Acceptable - pinned to minor version range
module "vpc" {
  source  = "app.terraform.io/yourorg/vpc/aws"
  version = "~> 2.3"
}

# Bad - no version pin, uses latest
module "vpc" {
  source = "git::https://github.com/yourorg/terraform-modules.git//modules/vpc"
}

Automated Version Bumping

Use a CI workflow that automatically creates releases when module directories change:

# .github/workflows/release.yml
name: Release Modules
on:
  push:
    branches: [main]

jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      changed_modules: ${{ steps.changes.outputs.modules }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - id: changes
        run: |
          CHANGED=$(git diff --name-only HEAD~1 HEAD | grep '^modules/' | cut -d'/' -f2 | sort -u | jq -R -s -c 'split("\n")[:-1]')
          echo "modules=$CHANGED" >> $GITHUB_OUTPUT

  release:
    needs: detect-changes
    if: needs.detect-changes.outputs.changed_modules != '[]'
    runs-on: ubuntu-latest
    strategy:
      matrix:
        module: ${{ fromJson(needs.detect-changes.outputs.changed_modules) }}
    steps:
      - uses: actions/checkout@v4

      - name: Determine version bump
        id: version
        run: |
          CURRENT=$(git tag -l "modules/${{ matrix.module }}/v*" | sort -V | tail -1)
          # Parse commit messages for bump type
          if git log --oneline HEAD~1..HEAD | grep -q "BREAKING"; then
            BUMP="major"
          elif git log --oneline HEAD~1..HEAD | grep -q "feat"; then
            BUMP="minor"
          else
            BUMP="patch"
          fi
          echo "bump=$BUMP" >> $GITHUB_OUTPUT

      - name: Create release tag
        run: |
          # Bump version and create tag
          git tag "modules/${{ matrix.module }}/v${NEW_VERSION}"
          git push --tags

Using a Private Module Registry

A module registry provides a discoverable, versioned catalog of your organization's modules. You have several options.

Terraform Cloud / HCP Terraform Registry

The simplest option if you are already using Terraform Cloud:

module "vpc" {
  source  = "app.terraform.io/yourorg/vpc/aws"
  version = "2.3.1"

  cidr_block  = "10.0.0.0/16"
  environment = "production"
}

Publishing is automatic when you connect your VCS repository to the registry.

Self-Hosted with Artifactory or S3

For air-gapped or highly regulated environments, you can host modules on S3:

module "vpc" {
  source = "s3::https://my-terraform-modules.s3.amazonaws.com/vpc/v2.3.1.zip"
}

Pair this with a CI pipeline that packages and uploads module archives on release:

#!/bin/bash
MODULE=$1
VERSION=$2

cd modules/$MODULE
zip -r "/tmp/${MODULE}-${VERSION}.zip" .
aws s3 cp "/tmp/${MODULE}-${VERSION}.zip" \
  "s3://my-terraform-modules/${MODULE}/${VERSION}.zip"

GitHub Releases as a Registry

A lightweight approach using Git tags directly:

module "vpc" {
  source = "git::https://github.com/yourorg/terraform-modules.git//modules/vpc?ref=v2.3.1"
}

This works well for smaller organizations and avoids the overhead of a dedicated registry.

Module Testing and Validation

Modules without tests are modules you cannot trust. Here are the layers of testing you should implement.

Static Analysis

Run these on every PR:

# Format check
terraform fmt -check -recursive modules/

# Validation
for dir in modules/*/; do
  cd "$dir"
  terraform init -backend=false
  terraform validate
  cd ../..
done

# Security scanning with tfsec
tfsec modules/

# Linting with tflint
tflint --recursive

Integration Testing with Terratest

Write Go tests that actually provision and destroy infrastructure:

package test

import (
    "testing"
    "github.com/gruntwork-io/terratest/modules/terraform"
    "github.com/stretchr/testify/assert"
)

func TestVpcModule(t *testing.T) {
    t.Parallel()

    terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
        TerraformDir: "../modules/vpc",
        Vars: map[string]interface{}{
            "cidr_block":  "10.99.0.0/16",
            "environment": "test",
            "name":        "terratest-vpc",
        },
    })

    defer terraform.Destroy(t, terraformOptions)
    terraform.InitAndApply(t, terraformOptions)

    vpcId := terraform.Output(t, terraformOptions, "vpc_id")
    assert.NotEmpty(t, vpcId)

    privateSubnets := terraform.OutputList(t, terraformOptions, "private_subnet_ids")
    assert.Equal(t, 3, len(privateSubnets))
}

Example Configurations

Every module should ship with a working example in an examples/ directory:

modules/vpc/
├── main.tf
├── variables.tf
├── outputs.tf
├── README.md
└── examples/
    ├── simple/
    │   └── main.tf       # Minimal usage
    └── complete/
        └── main.tf       # All features enabled

Module Composition Patterns

Real infrastructure is built by composing modules together. Here are patterns that work well at scale.

The Root Module Pattern

Create environment-specific root modules that compose shared modules:

# environments/production/main.tf

module "network" {
  source  = "app.terraform.io/yourorg/vpc/aws"
  version = "2.3.1"

  cidr_block         = "10.0.0.0/16"
  availability_zones = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
  environment        = "production"
}

module "database" {
  source  = "app.terraform.io/yourorg/rds-postgres/aws"
  version = "3.1.0"

  vpc_id             = module.network.vpc_id
  subnet_ids         = module.network.private_subnet_ids
  instance_class     = "db.r6g.xlarge"
  multi_az           = true
  environment        = "production"
}

module "application" {
  source  = "app.terraform.io/yourorg/ecs-service/aws"
  version = "1.8.0"

  vpc_id             = module.network.vpc_id
  subnet_ids         = module.network.private_subnet_ids
  lb_target_group    = module.network.alb_target_group_arn
  database_endpoint  = module.database.endpoint
  desired_count      = 6
  environment        = "production"
}

The Terragrunt DRY Pattern

For organizations managing many environments, Terragrunt eliminates repetition:

# terragrunt.hcl (root)
remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket         = "myorg-terraform-state"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "eu-west-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
}

# environments/production/vpc/terragrunt.hcl
terraform {
  source = "git::https://github.com/yourorg/terraform-modules.git//modules/vpc?ref=v2.3.1"
}

inputs = {
  cidr_block  = "10.0.0.0/16"
  environment = "production"
}

Common Pitfalls and How to Avoid Them

Over-abstracting too early. Do not create a module until you have written the same Terraform code at least twice. Premature abstraction creates rigid modules that fight against real requirements.

Nested modules more than two levels deep. Module A calls module B which calls module C is already hard to debug. Keep your module hierarchy shallow.

Not using moved blocks during refactors. When restructuring modules, use moved blocks to prevent Terraform from destroying and recreating resources:

moved {
  from = aws_instance.web
  to   = module.application.aws_instance.web
}

Ignoring module documentation. Every module should have a README generated by terraform-docs:

# Generate docs automatically
terraform-docs markdown table modules/vpc/ > modules/vpc/README.md

Storing secrets in tfvars files. Use a secrets manager and data sources instead of committing sensitive values.

Need Help with Your DevOps?

Building and maintaining a well-structured Terraform module library takes experience. At InstaDevOps, we help startups and growing teams implement production-grade Infrastructure as Code from day one - so you can ship infrastructure changes with the same confidence as application code.

Plans start at $2,999/mo for a dedicated fractional DevOps engineer.

Book a free 15-minute consultation to discuss your Terraform architecture.

Infrastructure Testing: Terratest, Checkov, and Validating Your IaC

InstaDevOps — Sun, 12 Apr 2026 13:46:59 +0000

Introduction

Infrastructure as Code has transformed cloud management, but when a single Terraform apply can spin up hundreds of resources, testing becomes essential.

This article explores Terratest for functional testing and Checkov for security scanning.

Static Analysis with Checkov

pip install checkov
checkov -d ./terraform

Custom Policies

from checkov.terraform.checks.resource.base_resource_check import BaseResourceCheck
from checkov.common.models.enums import CheckCategories, CheckResult

class EC2HasEnvironmentTag(BaseResourceCheck):
    def __init__(self):
        name = "Ensure EC2 instances have environment tag"
        id = "CKV_CUSTOM_1"
        supported_resources = ['aws_instance']
        super().__init__(name=name, id=id,
                        supported_resources=supported_resources)

    def scan_resource_conf(self, conf):
        tags = conf.get('tags', [{}])[0]
        if isinstance(tags, dict) and 'Environment' in tags:
            return CheckResult.PASSED
        return CheckResult.FAILED

Functional Testing with Terratest

func TestS3Bucket(t *testing.T) {
    t.Parallel()

    terraformOptions := terraform.WithDefaultRetryableErrors(t,
        &terraform.Options{
            TerraformDir: "../modules/s3-bucket",
            Vars: map[string]interface{}{
                "bucket_name": "test-bucket-" + random.UniqueId(),
            },
        })

    defer terraform.Destroy(t, terraformOptions)
    terraform.InitAndApply(t, terraformOptions)

    bucketID := terraform.Output(t, terraformOptions, "bucket_id")
    actualStatus := aws.GetS3BucketVersioning(t, "us-east-1", bucketID)
    assert.Equal(t, "Enabled", actualStatus)
}

When to Use Each Tool

Aspect	Checkov	Terratest
Type	Static analysis	Functional testing
Speed	Seconds	Minutes to hours
Cost	Free	Incurs cloud costs
Coverage	Security, compliance	Actual functionality

CI/CD Integration

jobs:
  static-analysis:
    steps:
      - uses: bridgecrewio/checkov-action@master
        with:
          directory: terraform/

  terratest:
    needs: static-analysis
    steps:
      - run: |
          cd test
          go test -v -timeout 30m ./...

Conclusion

Layer your testing: Start with Checkov for fast feedback, add Terratest for critical modules, and run full integration tests before production.

Need Help with Your DevOps Infrastructure?

At InstaDevOps, we specialize in helping startups build production-ready infrastructure.

📅 Book a Free 15-Min Consultation

Originally published at instadevops.com

Kubernetes Networking Demystified: Services, Ingress, and Network Policies

InstaDevOps — Sat, 11 Apr 2026 13:46:57 +0000

Introduction

Kubernetes networking is often cited as one of the most challenging aspects of container orchestration. This guide covers the three pillars: Services, Ingress, and Network Policies.

Services: Stable Endpoints for Dynamic Pods

ClusterIP Services

apiVersion: v1
kind: Service
metadata:
  name: backend-api
spec:
  type: ClusterIP
  selector:
    app: backend
  ports:
  - port: 80
    targetPort: 8080

LoadBalancer Services

apiVersion: v1
kind: Service
metadata:
  name: public-api
spec:
  type: LoadBalancer
  selector:
    app: api
  ports:
  - port: 443
    targetPort: 8443

Ingress: HTTP/HTTPS Traffic Management

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - app.example.com
    secretName: app-tls-secret
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: backend-api
            port:
              number: 80

Network Policies: Securing Pod Communication

Default Deny All

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Allow Specific Ingress

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
spec:
  podSelector:
    matchLabels:
      app: backend
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

Troubleshooting

# Check endpoints
kubectl get endpoints my-service

# Test DNS
kubectl exec -it my-pod -- nslookup kubernetes.default

# Debug with netshoot
kubectl run netshoot --image=nicolaka/netshoot -it --rm -- bash

Conclusion

Master Services, Ingress, and Network Policies, and you'll have a solid foundation for building secure, scalable applications on Kubernetes.

Need Help with Your DevOps Infrastructure?

At InstaDevOps, we specialize in helping startups build production-ready infrastructure.

📅 Book a Free 15-Min Consultation

Originally published at instadevops.com

Container Registry Best Practices: ECR, Docker Hub, and Self-Hosted Options

InstaDevOps — Fri, 10 Apr 2026 13:46:54 +0000

Introduction

Container registries are the backbone of containerized application deployment. Choosing the right registry and implementing proper practices can mean the difference between smooth deployments and security nightmares.

Amazon ECR: AWS-Native Registry

# Create a repository
aws ecr create-repository \
    --repository-name my-app \
    --image-scanning-configuration scanOnPush=true

# Push an image
docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest

ECR Lifecycle Policies

{
  "rules": [{
    "rulePriority": 1,
    "description": "Keep last 10 production images",
    "selection": {
      "tagStatus": "tagged",
      "tagPrefixList": ["prod-"],
      "countType": "imageCountMoreThan",
      "countNumber": 10
    },
    "action": { "type": "expire" }
  }]
}

Docker Hub

Docker Hub remains the most widely used registry, hosting millions of public images.

Rate Limits: Anonymous pulls limited to 100 per 6 hours; authenticated free users get 200.

Self-Hosted: Harbor

helm install harbor harbor/harbor \
  --set expose.type=ingress \
  --set expose.ingress.hosts.core=registry.example.com \
  --set trivy.enabled=true

Security Best Practices

Enable image scanning
Implement least-privilege access
Sign your images with Cosign
Use immutable tags
Scan base images regularly

Image Tagging Strategies

VERSION="1.2.3"
GIT_SHA=$(git rev-parse --short HEAD)

docker build \
    -t my-app:${VERSION} \
    -t my-app:${VERSION}-${GIT_SHA} \
    -t my-app:${GIT_SHA} \
    .

Conclusion

Whether you choose ECR for AWS integration, Docker Hub for ubiquity, or Harbor for control, applying security best practices will keep your container infrastructure secure.

Need Help with Your DevOps Infrastructure?

At InstaDevOps, we specialize in helping startups build production-ready infrastructure.

📅 Book a Free 15-Min Consultation

Originally published at instadevops.com

Incident Management: Building Effective On-Call Rotations and Runbooks

InstaDevOps — Thu, 09 Apr 2026 20:46:50 +0000

Introduction

Every engineering team dreads the 3 AM page. How your team handles these moments defines your reliability as a service provider.

This guide covers on-call rotation design, runbook creation, incident response processes, and blameless post-mortems.

On-Call Best Practices

Designing Sustainable Rotations

Rotation Length: Weekly rotations work best
Minimum Team Size: Never fewer than 4 engineers
Compensation: Fair pay for on-call work

Reducing Alert Fatigue

Actionable alerts only
Consolidate related alerts
Regular alert reviews

Runbook Template Example

# Runbook: Database Connection Pool Exhausted

## Impact Assessment
- User Impact: New requests may fail
- Affected Services: All services using primary database

## Immediate Actions
1. Check current connections:
   SELECT count(*), state FROM pg_stat_activity GROUP BY state;

2. Identify connection hogs:
   SELECT usename, application_name, count(*)
   FROM pg_stat_activity GROUP BY usename, application_name;

## Resolution Steps
- If long-running query: pg_terminate_backend(<pid>)
- If traffic spike: Scale up read replicas
- If connection leak: Restart affected service

Incident Response Process

Phase 1: Detection and Triage (0-5 minutes)

Acknowledge the alert
Assess severity
Open relevant runbook

Phase 2: Response Coordination

Incident Commander: Coordinates response
Technical Lead: Hands-on debugging
Communications Lead: Updates stakeholders

Blameless Post-Mortems

Focus on systems and processes, not individuals. The goal is learning and improvement.

Conclusion

Effective incident management is built on preparation, not heroics. Well-designed rotations, comprehensive runbooks, and blameless post-mortems transform incident response from stress into a competitive advantage.

Need Help with Your DevOps Infrastructure?

At InstaDevOps, we specialize in helping startups build production-ready infrastructure.

📅 Book a Free 15-Min Consultation

Originally published at instadevops.com

SRE Fundamentals: Defining SLOs, SLIs, and Error Budgets That Actually Work

InstaDevOps — Thu, 09 Apr 2026 20:33:35 +0000

Introduction

Site Reliability Engineering (SRE) has transformed how organizations think about system reliability. Central to this framework are Service Level Objectives (SLOs), Service Level Indicators (SLIs), and Error Budgets.

This guide will walk you through defining SLOs, SLIs, and Error Budgets that actually drive meaningful improvements.

Understanding the Reliability Hierarchy

SLAs (Service Level Agreements): External contracts with customers specifying consequences for failures.

SLOs (Service Level Objectives): Internal targets your team commits to, stricter than SLAs.

SLIs (Service Level Indicators): The actual measurements determining whether you're meeting SLOs.

Error Budgets: How much unreliability you can tolerate while meeting your SLO.

Defining Meaningful SLIs

The Four Golden Signals

Latency: How long requests take
Traffic: Request volume
Errors: Rate of failed requests
Saturation: How "full" your service is

# Availability SLI
sum(rate(http_requests_total{status=~"2.."}[5m]))
/
sum(rate(http_requests_total[5m]))

# Latency SLI
sum(rate(http_request_duration_seconds_bucket{le="0.2"}[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))

Setting Realistic SLOs

Each additional "nine" dramatically reduces your error budget:

Availability	Monthly Downtime
99%	7.2 hours
99.9%	43.8 minutes
99.99%	4.38 minutes

Error Budgets: The Key to Balance

Error Budget = 1 - SLO

For a 99.9% SLO over 30 days:
Error Budget = 0.1% = 43.2 minutes of downtime

Conclusion

SLOs, SLIs, and Error Budgets aren't just metrics—they're a framework for making better decisions about reliability. The goal is appropriate reliability—enough to keep users happy while maintaining velocity.

Need Help with Your DevOps Infrastructure?

At InstaDevOps, we specialize in helping startups build production-ready infrastructure.

📅 Book a Free 15-Min Consultation

Originally published at instadevops.com

Disaster Recovery in the Cloud: RPO, RTO, and Building Resilient Systems

InstaDevOps — Wed, 08 Apr 2026 14:15:32 +0000

Introduction

Every minute of downtime costs money. For some enterprises, that figure reaches $5,600 per minute. But beyond financial impact, outages erode customer trust and can pose compliance risks.

This article explores disaster recovery fundamentals, specifically Recovery Point Objective (RPO) and Recovery Time Objective (RTO), and provides practical guidance for building resilient cloud systems.

Understanding RPO and RTO

Recovery Point Objective (RPO)

How much data can we afford to lose? RPO represents the maximum acceptable data loss measured in time.

Recovery Time Objective (RTO)

How quickly must we restore operations? RTO defines maximum acceptable downtime.

DR Strategy Tiers

Tier 1: Backup and Restore

RPO: Hours to days | RTO: Hours to days | Cost: Lowest

Tier 2: Pilot Light

RPO: Minutes to hours | RTO: Hours | Cost: Low to Medium

Keep core components synchronized, but application servers remain off until needed.

Tier 3: Warm Standby

RPO: Minutes | RTO: Minutes to hours | Cost: Medium to High

A scaled-down but fully functional environment runs continuously.

Tier 4: Multi-Region Active-Active

RPO: Near-zero | RTO: Near-zero | Cost: Highest

resource "aws_globalaccelerator_accelerator" "main" {
  name            = "production-global"
  ip_address_type = "IPV4"
  enabled         = true
}

Automating Failover

resource "aws_route53_health_check" "primary" {
  fqdn              = "api-primary.example.com"
  port              = 443
  type              = "HTTPS"
  failure_threshold = "3"
  request_interval  = "10"
}

Testing Your DR Plan

A DR plan that has never been tested is not a plan. Regular testing validates assumptions and trains your team.

Conclusion

Building resilient systems requires understanding business requirements, choosing appropriate DR strategies, and relentlessly testing. The cost of preparation is always less than the cost of recovery without a plan.

Need Help with Your DevOps Infrastructure?

At InstaDevOps, we specialize in helping startups build production-ready infrastructure.

Book a Free 15-Min Consultation

Originally published at instadevops.com

API Gateway Patterns: Kong, AWS API Gateway, and When to Use Each

InstaDevOps — Sun, 05 Apr 2026 13:48:01 +0000

Introduction

API gateways have become the backbone of modern microservices architecture. They act as the single entry point for all client requests, handling everything from routing and authentication to rate limiting and load balancing.

In this article, we'll compare two popular API gateway solutions: Kong (open-source and self-hosted) and AWS API Gateway (fully managed).

Understanding API Gateway Patterns

Edge Gateway Pattern: A single entry point at the network edge that handles all external traffic.

Backend for Frontend (BFF) Pattern: Separate gateways optimized for different client types.

Sidecar Pattern: A gateway deployed alongside each service in service mesh architectures.

Kong: The Open-Source Powerhouse

Kong is built on NGINX and offers both open-source and enterprise versions.

# kong.yaml - Declarative configuration
_format_version: "3.0"

services:
  - name: user-service
    url: http://user-api:3000
    routes:
      - name: user-routes
        paths:
          - /api/users
    plugins:
      - name: rate-limiting
        config:
          minute: 100
      - name: key-auth

AWS API Gateway

AWS API Gateway is fully managed and integrates deeply with AWS services.

resource "aws_api_gateway_rest_api" "main" {
  name        = "production-api"
  description = "Production API Gateway"
}

resource "aws_api_gateway_method" "get_users" {
  rest_api_id   = aws_api_gateway_rest_api.main.id
  resource_id   = aws_api_gateway_resource.users.id
  http_method   = "GET"
  authorization = "COGNITO_USER_POOLS"
}

Decision Framework

Choose Kong When:

Multi-cloud or hybrid deployments
Advanced customization needs
Cost optimization at scale
Avoiding vendor lock-in

Choose AWS API Gateway When:

AWS-native architecture
Minimal operational overhead
Rapid prototyping
Deep AWS integration needed

Conclusion

Both Kong and AWS API Gateway are excellent choices. Kong excels in flexibility and multi-cloud scenarios, while AWS API Gateway shines in AWS-native architectures.

Need Help with Your DevOps Infrastructure?

At InstaDevOps, we specialize in helping startups and scale-ups build production-ready infrastructure.

📅 Book a Free 15-Min Consultation

Originally published at instadevops.com

Kubernetes Cost Optimization: FinOps Strategies for K8s Clusters

InstaDevOps — Sat, 04 Apr 2026 13:47:59 +0000

Introduction

Kubernetes has become the de facto standard for container orchestration, but with great power comes great... cloud bills. According to recent studies, organizations waste an average of 30-35% of their Kubernetes spending on over-provisioned or idle resources. This isn't just a technical problem—it's a business problem that directly impacts your bottom line.

FinOps (Financial Operations) brings financial accountability to the variable spend model of cloud computing. When applied to Kubernetes, FinOps practices help teams understand, manage, and optimize their cluster costs while maintaining the performance and reliability their applications need.

In this guide, we'll explore practical strategies for optimizing Kubernetes costs, from right-sizing workloads to implementing automated scaling policies.

Understanding Kubernetes Cost Drivers

Before optimizing costs, you need to understand where your money goes. Kubernetes costs typically break down into several categories:

Compute Resources: The CPU and memory allocated to your nodes and pods represent the largest portion of most Kubernetes bills.

Storage: Persistent volumes, storage classes, and backup solutions add up quickly.

Networking: Data transfer between availability zones, regions, and the internet can surprise you with unexpected charges.

Control Plane: Managed Kubernetes services (EKS, GKE, AKS) charge for the control plane itself.

Right-Sizing Workloads with Resource Requests and Limits

The foundation of Kubernetes cost optimization is properly configuring resource requests and limits.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: api
        image: myapp/api:v1.2.3
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"

Implementing Vertical Pod Autoscaler (VPA)

VPA automatically adjusts resource requests for pods based on actual usage:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-server-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  updatePolicy:
    updateMode: "Auto"

Horizontal Pod Autoscaler for Demand-Based Scaling

HPA adjusts the number of replicas based on demand:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-server-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Leveraging Spot Instances

Spot instances offer 60-90% discounts for suitable workloads:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: batch-processor
spec:
  template:
    spec:
      nodeSelector:
        kubernetes.io/lifecycle: spot
      tolerations:
      - key: "kubernetes.io/lifecycle"
        operator: "Equal"
        value: "spot"
        effect: "NoSchedule"

Namespace Resource Quotas

Prevent runaway costs with namespace-level guardrails:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-alpha-quota
  namespace: team-alpha
spec:
  hard:
    requests.cpu: "20"
    requests.memory: "40Gi"
    limits.cpu: "40"
    limits.memory: "80Gi"

Best Practices Checklist

Set resource requests and limits on every container
Use VPA in recommendation mode first
Configure HPA with appropriate thresholds
Enable Cluster Autoscaler with 50% scale-down threshold
Use spot instances for fault-tolerant workloads
Implement namespace quotas for accountability
Deploy cost monitoring tools like Kubecost

Conclusion

Kubernetes cost optimization isn't a one-time project—it's an ongoing practice. Start with visibility, implement technical controls, and treat cost as a first-class metric alongside availability and performance.

Need Help with Your DevOps Infrastructure?

At InstaDevOps, we specialize in helping startups and scale-ups build production-ready infrastructure without the overhead of a full-time DevOps team.

Our Services:

🏗️ AWS Consulting - Cloud architecture, cost optimization, and migration
☸️ Kubernetes Management - Production-ready clusters and orchestration
🚀 CI/CD Pipelines - Automated deployment pipelines that just work
📊 Monitoring & Observability - See what's happening in your infrastructure

📅 Book a Free 15-Min Consultation

Originally published at instadevops.com