Forem: Matthew

Part 6: CI/CD Pipeline

Matthew — Wed, 15 Apr 2026 14:00:00 +0000

Part 6: CI/CD Pipeline — GitHub Actions, Trivy, Cosign, and ECR

Part of the series: Building a Production-Grade DevSecOps Pipeline on AWS

Introduction

The CI/CD pipeline is the gateway between a developer's git push and a running container in production. Every security control that can be automated should live here — not as an afterthought but as a first-class gate that blocks bad artifacts from ever reaching a cluster.

This pipeline enforces:

No HIGH or CRITICAL CVEs — Trivy blocks the build before the image is pushed
No static AWS credentials — GitHub OIDC exchanges JWT tokens for temporary STS creds
Cryptographic image provenance — Cosign signs every image with an AWS KMS key; Kyverno verifies the signature before admitting the pod
Immutable image tags — sha-<full-commit-sha>, never :latest
Audit trail — every push is logged to S3 with digest, timestamp, and caller identity

The Application (myapp)

A minimal Node.js/Express API that represents any real production service:

myapp/
├── src/
│   └── index.js        # Express app with /health and /metrics
├── Dockerfile
├── package.json
└── .github/
    └── workflows/
        └── ci.yaml

// src/index.js
const express = require('express');
const promClient = require('prom-client');

const app = express();
const register = new promClient.Registry();
promClient.collectDefaultMetrics({ register });

const httpRequestsTotal = new promClient.Counter({
  name: 'myapp_http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status_code'],
  registers: [register],
});

app.use((req, res, next) => {
  res.on('finish', () => {
    httpRequestsTotal.inc({
      method: req.method,
      route: req.path,
      status_code: res.statusCode,
    });
  });
  next();
});

app.get('/health', (req, res) => {
  res.json({ status: 'healthy', region: process.env.AWS_REGION || 'unknown' });
});

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

app.listen(8080, () => console.log('Listening on :8080'));

Dockerfile — Distroless Nonroot

# Stage 1: Build dependencies
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

# Stage 2: Runtime — distroless (no shell, no package manager)
FROM gcr.io/distroless/nodejs18-debian12:nonroot

WORKDIR /app

# Copy only what's needed — no node_modules dev dependencies, no source maps
COPY --from=builder /app/node_modules ./node_modules
COPY src/ ./src/
COPY package.json ./

# nonroot image runs as uid 65532 by default — no root, ever
EXPOSE 8080

CMD ["src/index.js"]

Why distroless?
The nonroot variant contains only the Node.js runtime. There is no:

/bin/sh or /bin/bash — an attacker with RCE cannot spawn an interactive shell
apt, apk, yum — cannot install additional tools
curl, wget — cannot exfiltrate data or download payloads
Any other utility that would help lateral movement

Falco still alerts on any unexpected syscalls, but the attack surface is dramatically reduced.

Pipeline Overview

The complete pipeline: no static AWS credentials anywhere. OIDC authenticates
GitHub Actions to AWS. Every image is signed with Cosign before Kyverno will
admit it to the cluster.

┌─────────────────────────────────────────────────────────────────────────┐
│  git push → main                                                        │
│                    │                                                    │
│                    ▼                                                    │
│         ┌──────────────────┐                                            │
│         │  Job 1: test     │  npm ci + npm test                         │
│         └────────┬─────────┘                                            │
│                  │ needs: test                                          │
│                  ▼                                                      │
│         ┌──────────────────┐                                            │
│         │  Job 2: scan     │  trivy image → fail on HIGH/CRITICAL       │
│         └────────┬─────────┘                                            │
│                  │ needs: scan                                          │
│                  ▼                                                      │
│         ┌──────────────────────────────────────────────┐                │
│         │  Job 3: build-push-sign                      │                │
│         │  ├─ OIDC → assume IAM role (no static keys)  │                │
│         │  ├─ docker build (distroless)                │                │
│         │  ├─ push → ECR us-east-1                     │                │
│         │  ├─ push → ECR us-west-2                     │                │
│         │  ├─ cosign sign (AWS KMS)                    │                │
│         │  └─ S3 audit log                             │                │
│         └────────┬─────────────────────────────────────┘                │
│                  │ needs: build-push-sign                               │
│                  ▼                                                      │
│         ┌──────────────────────────────────────────────┐                │
│         │  Job 4: update-gitops                        │                │
│         │  └─ patch image.tag in values-*.yaml → push  │                │
│         └──────────────────────────────────────────────┘                │
└─────────────────────────────────────────────────────────────────────────┘

Full GitHub Actions Workflow

# .github/workflows/ci.yaml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  ECR_REGISTRY_USE1: 206617159586.dkr.ecr.us-east-1.amazonaws.com
  ECR_REGISTRY_USW2: 206617159586.dkr.ecr.us-west-2.amazonaws.com
  IMAGE_NAME: myapp
  AWS_REGION_USE1: us-east-1
  AWS_REGION_USW2: us-west-2

permissions:
  id-token: write   # Required for OIDC
  contents: write   # Required for gitops update commit
  packages: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '18'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run tests
        run: npm test

  scan:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build image for scanning
        run: |
          docker build -t ${{ env.IMAGE_NAME }}:scan-${{ github.sha }} .

      - name: Trivy vulnerability scan
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ env.IMAGE_NAME }}:scan-${{ github.sha }}
          format: table
          exit-code: '1'              # Fail the pipeline on findings
          severity: 'HIGH,CRITICAL'   # Only fail on HIGH and CRITICAL
          ignore-unfixed: true        # Skip CVEs with no available fix

  build-push-sign:
    needs: scan
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'  # Only push on main branch, not PRs
    outputs:
      image-digest-use1: ${{ steps.push-use1.outputs.digest }}
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ vars.AWS_ROLE_ARN_USE1 }}
          aws-region: ${{ env.AWS_REGION_USE1 }}
          # No static access keys — OIDC exchanges GitHub JWT for STS temp creds

      - name: Login to ECR us-east-1
        uses: aws-actions/amazon-ecr-login@v2
        with:
          region: ${{ env.AWS_REGION_USE1 }}

      - name: Login to ECR us-west-2
        uses: aws-actions/amazon-ecr-login@v2
        with:
          region: ${{ env.AWS_REGION_USW2 }}

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build and push to us-east-1
        id: push-use1
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: |
            ${{ env.ECR_REGISTRY_USE1 }}/${{ env.IMAGE_NAME }}:sha-${{ github.sha }}
            ${{ env.ECR_REGISTRY_USE1 }}/${{ env.IMAGE_NAME }}:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Copy image to us-west-2
        run: |
          DIGEST="${{ steps.push-use1.outputs.digest }}"
          docker buildx imagetools create \
            --tag ${{ env.ECR_REGISTRY_USW2 }}/${{ env.IMAGE_NAME }}:sha-${{ github.sha }} \
            ${{ env.ECR_REGISTRY_USE1 }}/${{ env.IMAGE_NAME }}@${DIGEST}

      - name: Install Cosign
        uses: sigstore/cosign-installer@v3

      - name: Sign image with AWS KMS
        env:
          COSIGN_KEY: ${{ vars.COSIGN_KMS_KEY_ARN }}
        run: |
          DIGEST="${{ steps.push-use1.outputs.digest }}"
          IMAGE="${{ env.ECR_REGISTRY_USE1 }}/${{ env.IMAGE_NAME }}@${DIGEST}"

          # Sign the image — creates an OCI attestation artifact in ECR
          cosign sign --key awskms:///${COSIGN_KEY} \
            --yes \
            ${IMAGE}

          echo "Signed: ${IMAGE}"

      - name: Write S3 audit log
        run: |
          DIGEST="${{ steps.push-use1.outputs.digest }}"
          TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)
          CALLER=$(aws sts get-caller-identity --query Arn --output text)

          echo "{
            \"timestamp\": \"${TIMESTAMP}\",
            \"repo\": \"${{ github.repository }}\",
            \"sha\": \"${{ github.sha }}\",
            \"digest\": \"${DIGEST}\",
            \"pushed_by\": \"${CALLER}\",
            \"workflow\": \"${{ github.workflow }}\",
            \"run_id\": \"${{ github.run_id }}\"
          }" | aws s3 cp - \
            s3://${{ vars.AUDIT_BUCKET }}/ci-push-audit/${{ github.sha }}.json

  update-gitops:
    needs: build-push-sign
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - name: Checkout myapp-gitops
        uses: actions/checkout@v4
        with:
          repository: MatthewDipo/myapp-gitops
          token: ${{ secrets.GITOPS_PAT }}
          path: myapp-gitops

      - name: Update image tags
        run: |
          cd myapp-gitops
          NEW_TAG="sha-${{ github.sha }}"

          # Update all environment value files
          for FILE in apps/myapp/values-dev.yaml \
                      apps/myapp/values-staging.yaml \
                      apps/myapp/values-production.yaml; do
            sed -i "s|tag: sha-[a-f0-9]*|tag: ${NEW_TAG}|g" $FILE
            echo "Updated $FILE → ${NEW_TAG}"
          done

      - name: Commit and push
        run: |
          cd myapp-gitops
          git config user.email "ci@github.com"
          git config user.name "GitHub Actions"
          git add apps/myapp/values-*.yaml
          git commit -m "ci: update image tag to sha-${{ github.sha }}"
          git push origin main

GitHub Repository Variables (not secrets)

These are set in the GitHub UI under Settings → Secrets and variables → Actions → Variables:

Variable	Value	Why not a secret?
`AWS_ROLE_ARN_USE1`	`arn:aws:iam::206617159586:role/myapp-dev-use1-github-ci`	Not sensitive — it's a role ARN, useless without the OIDC token
`COSIGN_KMS_KEY_ARN`	`arn:aws:kms:us-east-1:206617159586:key/...`	Not sensitive — KMS key ID alone grants nothing
`AUDIT_BUCKET`	`myapp-ci-audit-206617159586`	Not sensitive

Only GITOPS_PAT (GitHub Personal Access Token to push to myapp-gitops) is a secret.

No AWS_ACCESS_KEY_ID. No AWS_SECRET_ACCESS_KEY. These should never appear in a modern CI pipeline.

Cosign Image Signing

Cosign attaches a cryptographic signature to the image in the same ECR repository. The signature is stored as a separate OCI artifact tagged with the image digest.

ECR Repository: myapp

sha-abc123def456...          ← Your application image
sha256-abc123def456...sig    ← Cosign signature (OCI artifact)
sha256-abc123def456...att    ← Cosign attestation (SBOM, etc.)

Verification (what Kyverno does at admission time):

cosign verify \
  --key awskms:///arn:aws:kms:us-east-1:206617159586:key/YOUR_KEY_ID \
  206617159586.dkr.ecr.us-east-1.amazonaws.com/myapp:sha-abc123

# Output if valid:
# Verification for 206617159586.dkr.ecr.us-east-1.amazonaws.com/myapp:sha-abc123
# The following checks were performed on each of these signatures:
#   - The cosign claims were validated
#   - The signatures were verified against the specified public key

Image Naming Convention

206617159586.dkr.ecr.us-east-1.amazonaws.com/myapp:sha-<full-40-char-git-sha>

Why sha-<full-sha> and not semantic versioning?

Traceability: Given any running pod, you can run git show <sha> to see the exact commit
Immutability: Two builds of the same SHA produce the same image (deterministic builds)
No tag collisions: v1.0.0 can be overwritten; a git SHA cannot
No :latest: :latest is the devil — it means different things at different times and breaks reproducibility

Pre-commit Hooks

Before code ever reaches GitHub, pre-commit hooks catch common issues locally:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-merge-conflict

  - repo: https://github.com/hadolint/hadolint
    rev: v2.12.0
    hooks:
      - id: hadolint-docker
        args: ['--ignore', 'DL3006']   # DL3006: always tag FROM image

  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.5.0
    hooks:
      - id: detect-secrets
        args: ['--baseline', '.secrets.baseline']

# Install pre-commit
pip install pre-commit
pre-commit install

# Run manually against all files
pre-commit run --all-files

Pipeline Security Properties

Property	How Achieved
No secrets in Git	detect-secrets pre-commit hook
No secrets in CI	OIDC replaces static AWS keys
No HIGH/CRITICAL CVEs	Trivy scan blocks pipeline
No unverified images	Kyverno admission webhook
Immutable artifacts	ECR `IMMUTABLE` tag mutability
Cryptographic provenance	Cosign + AWS KMS signing
Full audit trail	S3 + CloudTrail
Reproducible builds	Pinned base image SHA, `npm ci`

Summary

By the end of Part 6 you have:

✅ A secure Dockerfile using distroless/nonroot with multi-stage build
✅ GitHub Actions pipeline with 4 jobs: test → scan → build/push/sign → gitops update
✅ OIDC-based AWS authentication (zero static credentials)
✅ Trivy CVE scanning blocking HIGH/CRITICAL vulnerabilities
✅ Cosign image signing with AWS KMS
✅ Automatic image tag update in myapp-gitops triggering ArgoCD sync
✅ S3 audit log for every image push
✅ Pre-commit hooks catching issues before they reach CI

Screenshot Placeholders

SCREENSHOT: GitHub Actions — workflow run showing all 4 jobs passing (green checkmarks)

SCREENSHOT: ECR repository showing images with sha- tags and scan results (no HIGH/CRITICAL)

SCREENSHOT: GitHub Actions — Job 3 logs showing "Signed: ..." cosign output

Next: Part 7 — Secrets Management: AWS Secrets Manager + External Secrets Operator + IRSA

Follow the series — next part publishes next Wednesday.
Live system: https://www.matthewoladipupo.dev/health
Runbook: Operations Guide
Source code: myapp-infra | myapp-gitops | myapp

Part 5: GitOps with ArgoCD

Matthew — Thu, 09 Apr 2026 23:50:20 +0000

5: GitOps with ArgoCD — Hub-Spoke Model

Part of the series: Building a Production-Grade DevSecOps Pipeline on AWS

Introduction

GitOps flips the traditional CI/CD model. Instead of a pipeline pushing manifests into a cluster, the cluster pulls its desired state from Git. The result:

Audit trail built-in: every cluster change is a Git commit with author, timestamp, and diff
Self-healing: ArgoCD continuously reconciles — if someone kubectl applys something manually, ArgoCD reverts it within minutes
Rollback is git revert: no special tooling, no cluster access needed
Drift detection: ArgoCD shows you exactly when a cluster diverges from what's in Git

This pipeline uses ArgoCD in a hub-spoke topology: one ArgoCD installation on myapp-production-use1 manages all six clusters.

┌─────────────────────────────────────────────────────────────────┐
│          ArgoCD HUB: myapp-production-use1                      │
│                                                                 │
│  Watches: github.com/MatthewDipo/myapp-gitops (main branch)     │
│  Manages: 6 clusters via registered cluster credentials         │
│                                                                 │
│  ApplicationSets generate Applications per cluster:             │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │  environments/production/applicationset.yaml             │   │
│  │  → prometheus-myapp-production-use1                      │   │
│  │  → prometheus-myapp-production-usw2                      │   │
│  │  → prometheus-myapp-staging-use1    (staging project)    │   │
│  │  → prometheus-myapp-staging-usw2                         │   │
│  └──────────────────────────────────────────────────────────┘   │
└────────────────────────────┬────────────────────────────────────┘
         VPC Peering (private │ endpoints)
    ┌────────────────────────┤
    │    ┌───────────────────┤
    │    │    ┌──────────────┤──────────────┐
    ▼    ▼    ▼              ▼              ▼
prod-usw2  staging-use1  staging-usw2  dev-use1  dev-usw2

Why hub-spoke over running ArgoCD in every cluster?
One ArgoCD = one UI, one set of secrets, one audit log. Running six ArgoCD instances means six independent control planes to maintain, upgrade, and monitor. The operational overhead multiplies linearly with clusters.

Step 1: Install ArgoCD on the Hub

# Enable public endpoint on prod-use1 for bootstrapping
aws eks update-cluster-config \
  --name myapp-production-use1 \
  --region us-east-1 --profile myapp-prod-use1 \
  --resources-vpc-config endpointPublicAccess=true,endpointPrivateAccess=true,publicAccessCidrs="0.0.0.0/0"

sleep 180  # Wait for propagation

# Install ArgoCD
kubectl --context prod-use1 create namespace argocd

helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd \
  --namespace argocd \
  --version 6.7.3 \
  --set server.service.type=LoadBalancer \
  --set configs.params."server\.insecure"=true \
  --no-hooks \
  --wait --timeout 5m

# Get the LoadBalancer URL
kubectl --context prod-use1 get svc -n argocd argocd-server \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

# Get initial admin password
kubectl --context prod-use1 get secret -n argocd argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d && echo

# Login via CLI
argocd login <LB_HOSTNAME> --username admin --password <PASSWORD> --insecure

Step 2: Create AppProjects

AppProjects define what each group of Applications is allowed to do — which source repos, which destination clusters, and which namespaces. They are the RBAC boundary in ArgoCD.

# argocd/project-production.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: production
  namespace: argocd
spec:
  description: Production workloads and infrastructure

  sourceRepos:
    - https://github.com/MatthewDipo/myapp-gitops.git
    - https://prometheus-community.github.io/helm-charts
    - https://charts.external-secrets.io
    - https://aws.github.io/eks-charts
    - https://kyverno.github.io/kyverno
    - https://falcosecurity.github.io/charts
    - https://vmware-tanzu.github.io/helm-charts
    - https://charts.jetstack.io
    - https://grafana.github.io/helm-charts

  destinations:
    - server: "*"       # IMPORTANT: use server: "*" not name: "*"
      namespace: "*"    # ArgoCD resolves cluster name → server URL for permission checks

  clusterResourceWhitelist:
    - group: "*"
      kind: "*"

  namespaceResourceWhitelist:
    - group: "*"
      kind: "*"

Critical lesson: Use server: "*" not name: "*" in AppProject destinations. ArgoCD resolves cluster names to server URLs when enforcing project permissions. With name: "*" alone, syncs to named clusters fail with permission errors. With server: "*", it works correctly.

Apply all three projects:

kubectl --context prod-use1 apply -f argocd/project-dev.yaml
kubectl --context prod-use1 apply -f argocd/project-staging.yaml
kubectl --context prod-use1 apply -f argocd/project-production.yaml

Step 3: Add the Private GitOps Repo

ArgoCD needs credentials to pull from a private repository:

argocd repo add https://github.com/MatthewDipo/myapp-gitops.git \
  --username MatthewDipo \
  --password <GITHUB_PAT> \
  --insecure-skip-server-verification

# Verify
argocd repo list

Step 4: Register Spoke Clusters

Each spoke cluster is registered by running argocd cluster add with the cluster's kubeconfig context. This creates a ServiceAccount and ClusterRoleBinding in the spoke cluster that ArgoCD uses to manage resources.

# Dev clusters (public endpoints — accessible directly)
argocd cluster add dev-use1 --name myapp-dev-use1 --yes
argocd cluster add dev-usw2 --name myapp-dev-usw2 --yes

# Private clusters — must temporarily enable public endpoint first
for CLUSTER in myapp-staging-use1 myapp-staging-usw2 myapp-production-usw2; do
  REGION=$(echo $CLUSTER | grep -q "use1" && echo "us-east-1" || echo "us-west-2")
  ENV=$(echo $CLUSTER | cut -d- -f2)
  PROFILE="myapp-${ENV}-${REGION//us-east-1/use1}-${REGION//us-west-2/usw2}"

  echo "Enabling public endpoint on $CLUSTER..."
  aws eks update-cluster-config --name $CLUSTER --region $REGION \
    --profile myapp-${ENV}-$(echo $REGION | sed 's/us-east-1/use1/;s/us-west-2/usw2/') \
    --resources-vpc-config endpointPublicAccess=true,endpointPrivateAccess=true,publicAccessCidrs="0.0.0.0/0"
  sleep 180

  CTX=$(echo $CLUSTER | sed 's/myapp-//;s/-use1/-use1/;s/-usw2/-usw2/')
  argocd cluster add $CTX --name $CLUSTER --yes

  echo "Locking $CLUSTER back to private..."
  aws eks update-cluster-config --name $CLUSTER --region $REGION \
    --profile myapp-${ENV}-$(echo $REGION | sed 's/us-east-1/use1/;s/us-west-2/usw2/') \
    --resources-vpc-config endpointPublicAccess=false,endpointPrivateAccess=true
done

# Verify all clusters registered
argocd cluster list

Expected output:

SERVER                          NAME                      VERSION  STATUS
https://kubernetes.default.svc  in-cluster                1.29     Successful
<spoke-endpoint>                myapp-dev-use1            1.29     Successful
<spoke-endpoint>                myapp-dev-usw2            1.29     Successful
<spoke-endpoint>                myapp-staging-use1        1.29     Successful
<spoke-endpoint>                myapp-staging-usw2        1.29     Successful
<spoke-endpoint>                myapp-production-usw2     1.29     Successful

Solid lines = VPC Peering (private). Dashed lines = public endpoints (dev only).
The hub cluster runs ArgoCD and manages 47 Applications across all 6 clusters.

Step 5: ApplicationSet — The Core of Hub-Spoke GitOps

An ApplicationSet is a controller that generates ArgoCD Applications from a template and a generator. The list generator creates one Application per element in the list — one per cluster.

# environments/production/applicationset.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: myapp-production
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - cluster:    myapp-production-use1
            region:     us-east-1
            certArn:    "arn:aws:acm:us-east-1:591120834781:certificate/9ab022c9-..."
            wafAclArn:  "arn:aws:wafv2:us-east-1:591120834781:regional/webacl/..."
            irsaRoleArn: "arn:aws:iam::591120834781:role/myapp-production-use1-eso"
          - cluster:    myapp-production-usw2
            region:     us-west-2
            certArn:    "arn:aws:acm:us-west-2:591120834781:certificate/171cac9d-..."
            wafAclArn:  "arn:aws:wafv2:us-west-2:591120834781:regional/webacl/..."
            irsaRoleArn: "arn:aws:iam::591120834781:role/myapp-production-usw2-eso"

  template:
    metadata:
      name: "myapp-production-{{cluster}}"
      labels:
        component: application
        environment: production
        region: "{{region}}"
    spec:
      project: production
      sources:
        # Source 1: the Helm chart (from gitops repo)
        - repoURL:        https://github.com/MatthewDipo/myapp-gitops.git
          targetRevision: main
          ref: gitopsValues   # Named reference — used in source 2

        # Source 2: Helm chart with values from the gitops repo
        - repoURL:        https://github.com/MatthewDipo/myapp-gitops.git
          targetRevision: main
          path:           apps/myapp
          helm:
            valueFiles:
              - $gitopsValues/apps/myapp/values.yaml
              - $gitopsValues/apps/myapp/values-production.yaml
            parameters:
              - name:  "image.tag"
                value: "sha-45d92fc5ffd4555caf35b996ed1eec4e45152dce"
              - name:  "env.AWS_REGION"
                value: "{{region}}"
              - name:  "ingress.certArn"
                value: "{{certArn}}"
              - name:  "ingress.wafAclArn"
                value: "{{wafAclArn}}"
              - name:  "externalSecrets.irsaRoleArn"
                value: "{{irsaRoleArn}}"

      destination:
        name:      "{{cluster}}"
        namespace: myapp

      syncPolicy:
        automated:
          prune:    true    # Remove resources deleted from Git
          selfHeal: true    # Revert manual kubectl changes
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true
        retry:
          limit: 3
          backoff:
            duration:    30s
            maxDuration: 10m
            factor:      2

Key ApplicationSet concepts:

automated.prune: true — If you delete a file from Git, ArgoCD deletes the corresponding Kubernetes resource. Without this, deleted manifests leave orphaned resources.

automated.selfHeal: true — If someone runs kubectl edit and changes something, ArgoCD detects the drift and reverts it within ~3 minutes. This enforces Git as the only source of truth.

ServerSideApply: true — Required for CRDs and resources that multiple controllers manage (e.g., Prometheus operator patches its own webhook config). Without this, field ownership conflicts cause sync failures.

retry — Network blips or transient API errors shouldn't fail a deployment permanently. The retry policy handles intermittent failures automatically.

Step 6: Deploy the ApplicationSets

# Apply all ApplicationSets from the hub cluster
kubectl --context prod-use1 apply -f environments/dev/applicationset.yaml
kubectl --context prod-use1 apply -f environments/staging/applicationset.yaml
kubectl --context prod-use1 apply -f environments/production/applicationset.yaml

# Infrastructure ApplicationSets
kubectl --context prod-use1 apply -f infrastructure/eso/applicationset.yaml
kubectl --context prod-use1 apply -f infrastructure/kyverno/applicationset.yaml
kubectl --context prod-use1 apply -f infrastructure/falco/applicationset.yaml
kubectl --context prod-use1 apply -f infrastructure/monitoring/applicationset.yaml
kubectl --context prod-use1 apply -f infrastructure/logging/applicationset.yaml
kubectl --context prod-use1 apply -f infrastructure/velero/applicationset.yaml
kubectl --context prod-use1 apply -f infrastructure/karpenter/applicationset.yaml
kubectl --context prod-use1 apply -f infrastructure/argo-rollouts/applicationset.yaml

# Watch sync status
argocd app list

Sync Status and Known False Positives

After deploying, most apps show Synced/Healthy. A few will permanently show OutOfSync/Healthy — this is expected and safe:

App	Why OutOfSync	Safe?
ESO (`external-secrets-*`)	ESO writes `status.refreshTime` at runtime; ArgoCD sees this as drift	✅ Yes
`myapp-*` (with ESO)	Same ESO status drift propagates to the parent app	✅ Yes
`prometheus-*`	Prometheus operator patches its own ValidatingWebhookConfiguration at runtime	✅ Yes

Real sync failures (need action):

SyncFailed — usually a YAML error or missing CRD
Degraded — pods crashing; check kubectl describe pod
Missing — ArgoCD can't reach the cluster

Troubleshooting

Orphaned resources after a Helm release name change:

# Force prune all resources not in current Git state
argocd app sync argocd/karpenter-myapp-production-use1 --prune --force

App stuck in Progressing:

argocd app get argocd/myapp-production-myapp-production-use1
# Check the "Conditions" and "Events" sections

Cluster shows Unknown:

# Re-register the cluster (token may have expired)
argocd cluster add prod-usw2 --name myapp-production-usw2 --yes

AppProject permission denied:

# Ensure project destinations include server: "*"
# Ensure sourceRepos includes the chart repo URL
kubectl --context prod-use1 edit appproject production -n argocd

VPC Peering for Hub → Spoke Connectivity

ArgoCD hub reaches spoke private endpoints via VPC peering. Three peering connections:

prod-use1 VPC (10.20.0.0/16) ◄──────► prod-usw2 VPC   (10.21.0.0/16)
prod-use1 VPC (10.20.0.0/16) ◄──────► staging-use1 VPC (10.10.0.0/16)
prod-use1 VPC (10.20.0.0/16) ◄──────► staging-usw2 VPC (10.11.0.0/16)

After the peering is active, ArgoCD can communicate with spoke API servers on their private IPs (10.x.x.x:443) without any traffic touching the internet.

Summary

By the end of Part 5 you have:

✅ ArgoCD running on the hub cluster with a public LoadBalancer URL
✅ Three AppProjects (dev, staging, production) with correct source and destination restrictions
✅ All 5 spoke clusters registered (ArgoCD SA + ClusterRoleBinding installed in each)
✅ ApplicationSets generating Applications for all environments and infrastructure components
✅ GitOps loop closed: a commit to myapp-gitops triggers automatic sync to all clusters

Screenshot Placeholders

SCREENSHOT: ArgoCD UI — Applications view showing all apps, most Synced/Healthy
## Part

SCREENSHOT: ArgoCD UI — Cluster list showing all 6 clusters registered and Successful

SCREENSHOT: ArgoCD UI — One Application detail view showing resource tree

SCREENSHOT: GitOps Repo Structure on GitHub

Next: Part 6 — CI/CD Pipeline: GitHub Actions, Trivy, Cosign, ECR

Part 4: EKS Multi-Cluster Setup

Matthew — Wed, 01 Apr 2026 13:00:00 +0000

Part 4: EKS Multi-Cluster Setup — Six Clusters Across Two Regions

Part of the series: Building a Production-Grade DevSecOps Pipeline on AWS

Introduction

Why six clusters instead of one? The answer is isolation and resilience:

┌──────────────────────────────────────────────────────────────────────────┐
│  ONE CLUSTER (anti-pattern)                                              │
│                                                                          │
│  Dev pods → same etcd as Production pods                                 │
│  A misconfigured dev deployment can consume all cluster resources        │
│  Cluster upgrade = every environment goes down simultaneously            │
│  Cost visibility: impossible to attribute spend per environment          │
└──────────────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────────────────┐
│  SIX CLUSTERS (this guide)                                                 │
│                                                                            │
│  myapp-dev-use1      (us-east-1, public endpoint,  2 nodes)                │
│  myapp-dev-usw2      (us-west-2, public endpoint,  2 nodes)                │
│  myapp-staging-use1  (us-east-1, private endpoint, 2 nodes)                │
│  myapp-staging-usw2  (us-west-2, private endpoint, 2 nodes)                │
│  myapp-production-use1 (us-east-1, private endpoint, 2+ nodes + Karpenter) │
│  myapp-production-usw2 (us-west-2, private endpoint, 2+ nodes + Karpenter) │
│                                                                            │
│  Benefits:                                                                 │
│  ✓ Complete IAM isolation between environments                             │
│  ✓ Production upgrade independent of dev                                   │
│  ✓ Regional failover — us-east-1 outage → us-west-2 serves traffic         │
│  ✓ Clear cost attribution per cluster tag                                  │
└────────────────────────────────────────────────────────────────────────────┘

Cluster Overview

Cluster	Region	Endpoint	Nodes	Karpenter
`myapp-dev-use1`	us-east-1	Public	2 × t3.medium	No
`myapp-dev-usw2`	us-west-2	Public	2 × t3.medium	No
`myapp-staging-use1`	us-east-1	Private	2 × t3.medium	No
`myapp-staging-usw2`	us-west-2	Private	2 × t3.medium	No
`myapp-production-use1`	us-east-1	Private	2+ × t3.medium	Yes
`myapp-production-usw2`	us-west-2	Private	2+ × t3.medium	Yes

EKS Terraform Module

# _modules/eks/main.tf

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = var.cluster_name
  cluster_version = "1.29"

  vpc_id                   = var.vpc_id
  subnet_ids               = var.private_subnet_ids   # Nodes always in private subnets
  control_plane_subnet_ids = var.private_subnet_ids

  # Endpoint access: private always on, public only for dev
  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = var.public_api
  cluster_endpoint_public_access_cidrs = var.public_api ? ["0.0.0.0/0"] : ["0.0.0.0/0"]
  # Note: AWS rejects empty list when public is disabled — always set to 0.0.0.0/0

  # Without this the cluster creator IAM role (your Terragrunt role) can't kubectl
  enable_cluster_creator_admin_permissions = true

  # Encrypt Kubernetes secrets in etcd with KMS
  cluster_encryption_config = {
    provider_key_arn = var.kms_key_arn
    resources        = ["secrets"]
  }

  # EKS managed add-ons (AWS manages patching)
  cluster_addons = {
    coredns = {
      most_recent = true
    }
    kube-proxy = {
      most_recent = true
    }
    vpc-cni = {
      most_recent              = true
      service_account_role_arn = module.vpc_cni_irsa.iam_role_arn
    }
    aws-ebs-csi-driver = {
      most_recent              = true
      service_account_role_arn = module.ebs_csi_irsa.iam_role_arn
    }
  }

  eks_managed_node_groups = {
    main = {
      instance_types = var.instance_types   # ["t3.medium"]
      min_size       = var.min_nodes
      max_size       = var.max_nodes
      desired_size   = var.desired_nodes

      # IMPORTANT: name_prefix has a 38 character limit.
      # "myapp-production-use1-eks-node-group-" = 39 chars → FAILS.
      # Fix: use explicit role name (IAM limit is 64 chars).
      iam_role_name            = "${var.cluster_name}-node-group"
      iam_role_use_name_prefix = false

      # Nodes need these policies to pull from ECR, write to CloudWatch, etc.
      iam_role_additional_policies = {
        AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
      }

      labels = {
        "node-type" = "general"
      }

      # Karpenter discovery tag — only needed on production node groups
      # (Karpenter uses this to find the right security group)
      taints = var.karpenter_enabled ? [] : []
    }
  }

  # Node security group — allow Karpenter to manage nodes
  node_security_group_tags = var.karpenter_enabled ? {
    "karpenter.sh/discovery" = var.cluster_name
  } : {}
}

# IRSA for VPC CNI (pod networking)
module "vpc_cni_irsa" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.0"

  role_name             = "${var.cluster_name}-vpc-cni"
  attach_vpc_cni_policy = true
  vpc_cni_enable_ipv4   = true

  oidc_providers = {
    main = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["kube-system:aws-node"]
    }
  }
}

# IRSA for EBS CSI Driver (persistent volumes)
module "ebs_csi_irsa" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.0"

  role_name             = "${var.cluster_name}-ebs-csi"
  attach_ebs_csi_policy = true

  oidc_providers = {
    main = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["kube-system:ebs-csi-controller-sa"]
    }
  }
}

# _modules/eks/outputs.tf

output "cluster_name"                { value = module.eks.cluster_name }
output "cluster_endpoint"            { value = module.eks.cluster_endpoint }
output "cluster_certificate_authority_data" {
  value = module.eks.cluster_certificate_authority_data
}
output "oidc_provider_arn"           { value = module.eks.oidc_provider_arn }
output "oidc_provider"               { value = module.eks.oidc_provider }
output "node_security_group_id"      { value = module.eks.node_security_group_id }
output "node_subnet_ids"             { value = var.private_subnet_ids }

Per-Environment Terragrunt Configs

Dev (public endpoint):

# live/dev/us-east-1/eks/terragrunt.hcl

include "root" { path = find_in_parent_folders() }
terraform { source = "../../../../_modules/eks" }

dependency "vpc" {
  config_path = "../vpc"
  mock_outputs = {
    vpc_id             = "vpc-mock"
    private_subnet_ids = ["subnet-mock1", "subnet-mock2"]
  }
}

dependency "kms" {
  config_path = "../kms"
  mock_outputs = { key_arn = "arn:aws:kms:us-east-1:123456789:key/mock" }
}

inputs = {
  cluster_name       = "myapp-dev-use1"
  vpc_id             = dependency.vpc.outputs.vpc_id
  private_subnet_ids = dependency.vpc.outputs.private_subnet_ids
  kms_key_arn        = dependency.kms.outputs.key_arn
  public_api         = true    # Dev gets public endpoint for laptop + CI access
  min_nodes          = 2
  max_nodes          = 4
  desired_nodes      = 2
  instance_types     = ["t3.medium"]
  karpenter_enabled  = false
}

Production (private endpoint + Karpenter):

# live/production/us-east-1/eks/terragrunt.hcl

include "root" { path = find_in_parent_folders() }
terraform { source = "../../../../_modules/eks" }

dependency "vpc" { config_path = "../vpc" }
dependency "kms" { config_path = "../kms" }

inputs = {
  cluster_name       = "myapp-production-use1"
  vpc_id             = dependency.vpc.outputs.vpc_id
  private_subnet_ids = dependency.vpc.outputs.private_subnet_ids
  kms_key_arn        = dependency.kms.outputs.key_arn
  public_api         = false   # Private endpoint only — no public internet access
  min_nodes          = 2
  max_nodes          = 10
  desired_nodes      = 2
  instance_types     = ["t3.medium"]
  karpenter_enabled  = true    # Karpenter manages additional nodes beyond the initial 2
}

Bootstrapping Private Clusters

Staging and production clusters have endpointPublicAccess: false. This means kubectl from your laptop or CI cannot reach the API server directly. You must temporarily enable public access, bootstrap the cluster (install ArgoCD, register spokes, etc.), then lock it back down.

# Step 1: Temporarily enable public access
aws eks update-cluster-config \
  --name myapp-production-use1 \
  --region us-east-1 \
  --profile myapp-prod-use1 \
  --resources-vpc-config \
    endpointPublicAccess=true,\
    endpointPrivateAccess=true,\
    publicAccessCidrs="0.0.0.0/0"

# Step 2: Wait 3 minutes — AWS takes time to update the Elastic Network Interfaces
sleep 180

# Step 3: Verify access
kubectl --context prod-use1 get nodes

# Step 4: Bootstrap (install ArgoCD, apply ApplicationSets, etc.)
# ... your bootstrap commands ...

# Step 5: Lock back to private-only
aws eks update-cluster-config \
  --name myapp-production-use1 \
  --region us-east-1 \
  --profile myapp-prod-use1 \
  --resources-vpc-config \
    endpointPublicAccess=false,\
    endpointPrivateAccess=true

Do not forget Step 5. A production cluster with a public API endpoint is a security risk — the API server is internet-accessible, relying solely on authentication for protection.

kubectl Context Setup

After terragrunt apply completes for each cluster, add it to your kubeconfig:

# Update kubeconfig for all 6 clusters
aws eks update-kubeconfig \
  --name myapp-dev-use1 \
  --region us-east-1 \
  --alias dev-use1 \
  --profile myapp-dev-use1

aws eks update-kubeconfig \
  --name myapp-dev-usw2 \
  --region us-west-2 \
  --alias dev-usw2 \
  --profile myapp-dev-usw2

aws eks update-kubeconfig \
  --name myapp-staging-use1 \
  --region us-east-1 \
  --alias staging-use1 \
  --profile myapp-staging-use1

aws eks update-kubeconfig \
  --name myapp-staging-usw2 \
  --region us-west-2 \
  --alias staging-usw2 \
  --profile myapp-staging-usw2

aws eks update-kubeconfig \
  --name myapp-production-use1 \
  --region us-east-1 \
  --alias prod-use1 \
  --profile myapp-prod-use1

aws eks update-kubeconfig \
  --name myapp-production-usw2 \
  --region us-west-2 \
  --alias prod-usw2 \
  --profile myapp-prod-usw2

# Verify
kubectl config get-contexts

EKS Add-ons

EKS managed add-ons are maintained by AWS — they patch security vulnerabilities in CoreDNS, kube-proxy, and vpc-cni without you having to manage Helm releases.

kube-proxy     — handles iptables rules for Service routing
coredns        — in-cluster DNS resolution
vpc-cni        — AWS VPC networking for pods (each pod gets a real VPC IP)
aws-ebs-csi-driver — allows EKS to provision EBS volumes for PersistentVolumeClaims

Why IRSA for vpc-cni and ebs-csi?
These add-ons need to call AWS APIs (EC2 for ENI management, EC2 for EBS volume ops). Without IRSA they would use the node's EC2 instance profile — giving every pod on the node those permissions. With IRSA, only the specific add-on service account has the permissions.

Fixing kubectl 401 Errors

If kubectl get nodes returns HTTP 401 Unauthorized, the IAM role you're using is not in the cluster's access entries.

# List current access entries
aws eks list-access-entries \
  --cluster-name myapp-production-use1 \
  --region us-east-1 \
  --profile myapp-prod-use1

# If your OrganizationAccountAccessRole is missing, add it:
ROLE_ARN="arn:aws:iam::591120834781:role/OrganizationAccountAccessRole"

aws eks create-access-entry \
  --cluster-name myapp-production-use1 \
  --region us-east-1 \
  --profile myapp-prod-use1 \
  --principal-arn $ROLE_ARN

aws eks associate-access-policy \
  --cluster-name myapp-production-use1 \
  --region us-east-1 \
  --profile myapp-prod-use1 \
  --principal-arn $ROLE_ARN \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
  --access-scope type=cluster

The root cause: enable_cluster_creator_admin_permissions = true must be explicitly set in the EKS module. If it's missing, Terraform creates the cluster but the IAM role that ran Terraform doesn't get an access entry.

AWS Load Balancer Controller

The AWS Load Balancer Controller (LBC) runs in every cluster and watches for Ingress resources with ingressClassName: alb. When it sees one, it provisions an Application Load Balancer in AWS automatically.

Install via Helm (or ArgoCD) after the cluster is up:

# IRSA for LBC
eksctl create iamserviceaccount \
  --cluster myapp-production-use1 \
  --namespace kube-system \
  --name aws-load-balancer-controller \
  --attach-policy-arn arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess \
  --override-existing-serviceaccounts \
  --approve \
  --region us-east-1 \
  --profile myapp-prod-use1

# Install controller
helm repo add eks https://aws.github.io/eks-charts
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=myapp-production-use1 \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set region=us-east-1 \
  --set vpcId=<VPC_ID>

In this pipeline, the LBC is deployed via ArgoCD ApplicationSet — the Helm release is version-controlled in myapp-gitops/infrastructure/aws-lbc/.

StorageClass for EBS PVCs

kube-prometheus-stack needs persistent storage for Prometheus and Grafana data. With the EBS CSI driver installed, create a StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp2
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer   # Don't provision until pod is scheduled
reclaimPolicy: Retain                      # Don't delete EBS volume if PVC is deleted
parameters:
  type: gp2
  encrypted: "true"
  kmsKeyId: <your-kms-key-arn>

Node Security Group Tags

For Karpenter to manage node lifecycles, it needs to find the cluster's node security group. Tag it during EKS creation:

node_security_group_tags = {
  "karpenter.sh/discovery" = var.cluster_name
}

Similarly, private subnets need the discovery tag:

private_subnet_tags = {
  "karpenter.sh/discovery" = var.cluster_name
}

Verifying All Six Clusters

for CTX in dev-use1 dev-usw2 staging-use1 staging-usw2 prod-use1 prod-usw2; do
  echo "=== $CTX ==="
  kubectl --context $CTX get nodes -o wide
done

Expected output:

=== dev-use1 ===
NAME                           STATUS   ROLES    AGE   VERSION
ip-10-0-8-xx.ec2.internal      Ready    <none>   5d    v1.29.15-eks-ac2d5a0
ip-10-0-16-xx.ec2.internal     Ready    <none>   5d    v1.29.15-eks-ac2d5a0

=== prod-use1 ===
NAME                           STATUS   ROLES    AGE   VERSION
ip-10-20-8-xx.ec2.internal     Ready    <none>   5d    v1.29.15-eks-ac2d5a0
ip-10-20-16-xx.ec2.internal    Ready    <none>   5d    v1.29.15-eks-ac2d5a0

Summary

By the end of Part 4 you have:

✅ Six EKS clusters (Kubernetes 1.29) across three environments and two regions
✅ Private endpoints on staging and production (public on dev)
✅ KMS encryption for Kubernetes secrets in etcd
✅ IAM IRSA for VPC CNI and EBS CSI add-ons
✅ AWS Load Balancer Controller installed
✅ kubectl contexts configured for all six clusters
✅ Karpenter discovery tags on production node security groups and subnets

Screenshot Placeholders

SCREENSHOT: AWS EKS console showing 2 clusters running in production with ACTIVE status

SCREENSHOT: kubectl get nodes output for all 6 clusters

Next: Part 5 — GitOps with ArgoCD: Hub-Spoke Model

Part 3: Infrastructure as Code

Matthew — Wed, 25 Mar 2026 08:00:00 +0000

Part 3: Infrastructure as Code — Terraform Modules + Terragrunt

Part of the series: Building a Production-Grade DevSecOps Pipeline on AWS

Introduction

Plain Terraform works fine for a single environment. But this pipeline has 6 clusters across 3 environments and 2 regions — 18+ Terragrunt child directories. Without a DRY strategy, you end up copy-pasting the same provider, backend, and module blocks everywhere, and a single account ID change means updating 18 files.

Terragrunt solves this with two mechanisms:

include — child configs inherit the root config's provider generation and remote state
dependency — explicit ordering ensures VPC exists before EKS, KMS before EKS, etc.

The result: each child terragrunt.hcl is typically 10–30 lines of pure inputs, with all boilerplate generated automatically.

Repository Layout

myapp-infra/
├── _modules/                    # Reusable Terraform modules (no Terragrunt here)
│   ├── vpc/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── outputs.tf
│   ├── eks/
│   ├── kms/
│   ├── iam/
│   ├── ecr/
│   ├── waf/
│   ├── guardduty/
│   ├── eso-irsa/
│   ├── fluent-bit-irsa/
│   ├── karpenter/
│   └── velero/
│
└── live/                        # Terragrunt wrappers — one dir per resource per env/region
    ├── terragrunt.hcl           # ROOT config (provider + backend generation)
    ├── dev/
    │   ├── us-east-1/
    │   │   ├── vpc/
    │   │   │   └── terragrunt.hcl
    │   │   ├── kms/
    │   │   │   └── terragrunt.hcl
    │   │   ├── eks/
    │   │   │   └── terragrunt.hcl
    │   │   └── iam/
    │   │       └── terragrunt.hcl
    │   └── us-west-2/
    │       └── ... (mirror)
    ├── staging/
    │   └── ...
    └── production/
        └── ...

Key principle: modules in _modules/ are pure Terraform — no Terragrunt, no state config, no provider config. They are just reusable building blocks. The live/ tree contains nothing but thin Terragrunt wrappers that call those modules with environment-specific values.

Dependency Ordering

┌─────────────────────────────────────────────────────────────┐
│ APPLY ORDER (Terragrunt resolves this from dependency graph)│
│                                                             │
│  1. kms          (no dependencies)                          │
│  2. vpc          (no dependencies)                          │
│  3. eks          (depends on: vpc, kms)                     │
│  4. iam          (depends on: eks — needs OIDC provider URL)│
│  5. eso-irsa     (depends on: eks, iam)                     │
│  6. fluent-bit-irsa (depends on: eks)                       │
│  7. karpenter    (depends on: eks, iam)                     │
│  8. velero       (depends on: eks)                          │
│  9. waf          (no dependencies)                          │
│  10. guardduty   (no dependencies)                          │
└─────────────────────────────────────────────────────────────┘

Run everything in order automatically:

cd live/production/us-east-1
terragrunt run-all apply

Terragrunt reads all dependency blocks, builds a DAG, and applies in the correct order.

Root Terragrunt Config

# live/terragrunt.hcl

locals {
  path_parts = split("/", path_relative_to_include())
  env        = local.path_parts[0]   # dev | staging | production
  region     = local.path_parts[1]   # us-east-1 | us-west-2

  account_ids = {
    dev        = "557702566877"
    staging    = "YOUR_STAGING_ACCOUNT_ID"
    production = "591120834781"
  }
  account_id = local.account_ids[local.env]

  # Region short alias for naming (avoids long names hitting IAM limits)
  region_alias = local.region == "us-east-1" ? "use1" : "usw2"

  cluster_name = "myapp-${local.env}-${local.region_alias}"
}

# Auto-generate provider.tf in every child directory
generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<-EOF
    provider "aws" {
      region = "${local.region}"
      assume_role {
        role_arn = "arn:aws:iam::${local.account_id}:role/OrganizationAccountAccessRole"
      }
      default_tags {
        tags = {
          Environment = "${local.env}"
          Region      = "${local.region}"
          ManagedBy   = "Terraform"
          Project     = "myapp"
          Cluster     = "${local.cluster_name}"
        }
      }
    }
  EOF
}

# Auto-generate backend.tf — per-module state file in S3
remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket         = "myapp-terraform-state-${local.account_id}"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "us-east-1"   # State always in us-east-1 regardless of resource region
    encrypt        = true
    dynamodb_table = "myapp-terraform-locks"
    role_arn       = "arn:aws:iam::${local.account_id}:role/OrganizationAccountAccessRole"
  }
}

VPC Module

The VPC is the network foundation everything else sits in. Each environment gets its own VPC per region — 6 VPCs total.

CIDR allocation:
  dev     us-east-1:  10.0.0.0/16
  dev     us-west-2:  10.1.0.0/16
  staging us-east-1:  10.10.0.0/16
  staging us-west-2:  10.11.0.0/16
  prod    us-east-1:  10.20.0.0/16
  prod    us-west-2:  10.21.0.0/16

# _modules/vpc/main.tf

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = var.vpc_name
  cidr = var.vpc_cidr

  azs = [
    "${var.region}a",
    "${var.region}b",
    "${var.region}c"
  ]

  # Public subnets — for NAT Gateways, Internet-facing ALBs
  public_subnets = [
    cidrsubnet(var.vpc_cidr, 8, 0),   # x.x.0.0/24
    cidrsubnet(var.vpc_cidr, 8, 1),   # x.x.1.0/24
    cidrsubnet(var.vpc_cidr, 8, 2),   # x.x.2.0/24
  ]

  # Private subnets — EKS nodes, RDS, ElastiCache
  private_subnets = [
    cidrsubnet(var.vpc_cidr, 3, 1),   # x.x.8.0/21  (2048 IPs)
    cidrsubnet(var.vpc_cidr, 3, 2),   # x.x.16.0/21
    cidrsubnet(var.vpc_cidr, 3, 3),   # x.x.24.0/21
  ]

  enable_nat_gateway     = true
  single_nat_gateway     = var.single_nat_gateway   # true for dev (cost), false for prod (HA)
  enable_vpn_gateway     = false
  enable_dns_hostnames   = true
  enable_dns_support     = true

  # Required tags for AWS Load Balancer Controller to discover subnets
  public_subnet_tags = {
    "kubernetes.io/role/elb"                    = "1"
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb"           = "1"
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
    "karpenter.sh/discovery"                    = var.cluster_name  # Karpenter node discovery
  }

  # VPC Flow Logs for network traffic auditing
  enable_flow_log                      = true
  create_flow_log_cloudwatch_log_group = true
  create_flow_log_cloudwatch_iam_role  = true
  flow_log_max_aggregation_interval    = 60
}

# _modules/vpc/outputs.tf
output "vpc_id"              { value = module.vpc.vpc_id }
output "private_subnet_ids" { value = module.vpc.private_subnets }
output "public_subnet_ids"  { value = module.vpc.public_subnets }
output "vpc_cidr_block"     { value = module.vpc.vpc_cidr_block }

Terragrunt child config:

# live/production/us-east-1/vpc/terragrunt.hcl

include "root" {
  path = find_in_parent_folders()
}

terraform {
  source = "../../../../_modules/vpc"
}

inputs = {
  vpc_name          = "myapp-production-use1"
  vpc_cidr          = "10.20.0.0/16"
  region            = "us-east-1"
  cluster_name      = "myapp-production-use1"
  single_nat_gateway = false   # HA: one NAT GW per AZ
}

KMS Module

# _modules/kms/main.tf

# Handle the AWSServiceRoleForAutoScaling chicken-and-egg problem.
# In a fresh account this SLR doesn't exist yet, so we optionally create it.
resource "aws_iam_service_linked_role" "autoscaling" {
  count            = var.create_autoscaling_slr ? 1 : 0
  aws_service_name = "autoscaling.amazonaws.com"
}

# Wait 10s for IAM to propagate before referencing it in KMS key policy
resource "null_resource" "wait_for_slr" {
  count = var.create_autoscaling_slr ? 1 : 0
  depends_on = [aws_iam_service_linked_role.autoscaling]
  provisioner "local-exec" {
    command = "sleep 10"
  }
}

resource "aws_kms_key" "main" {
  depends_on              = [null_resource.wait_for_slr]
  description             = "${var.env}-${var.region}-main"
  deletion_window_in_days = 30
  enable_key_rotation     = true

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid      = "RootFullAccess"
        Effect   = "Allow"
        Principal = { AWS = "arn:aws:iam::${var.account_id}:root" }
        Action   = "kms:*"
        Resource = "*"
      },
      {
        Sid    = "AutoScalingSLR"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::${var.account_id}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        }
        Action   = ["kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*",
                    "kms:GenerateDataKey*", "kms:DescribeKey", "kms:CreateGrant"]
        Resource = "*"
      }
    ]
  })
}

resource "aws_kms_alias" "main" {
  name          = "alias/${var.env}-${var.region_alias}-main"
  target_key_id = aws_kms_key.main.key_id
}

Critical lesson: AWSServiceRoleForAutoScaling is an account-scoped IAM entity, not region-scoped. If you're deploying to two regions in the same account, only the first region should set create_autoscaling_slr = true. The second region's KMS config uses create_autoscaling_slr = false because the SLR already exists from the first apply.

# live/production/us-east-1/kms/terragrunt.hcl
include "root" { path = find_in_parent_folders() }
terraform { source = "../../../../_modules/kms" }
inputs = {
  env                    = "production"
  region                 = "us-east-1"
  region_alias           = "use1"
  account_id             = "591120834781"
  create_autoscaling_slr = false  # Already created by us-west-2 first apply
}

EKS Module (overview — full detail in Part 4)

# _modules/eks/main.tf (abbreviated)

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = var.cluster_name
  cluster_version = "1.29"

  vpc_id                   = var.vpc_id
  subnet_ids               = var.private_subnet_ids
  control_plane_subnet_ids = var.private_subnet_ids

  # Private endpoint — spokes only; dev gets public too
  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = var.public_api

  # Must be explicit — without this the creator role can't kubectl
  enable_cluster_creator_admin_permissions = true

  cluster_encryption_config = {
    provider_key_arn = var.kms_key_arn
    resources        = ["secrets"]
  }

  eks_managed_node_groups = {
    main = {
      instance_types = ["t3.medium"]
      min_size       = 2
      max_size       = 10
      desired_size   = 2

      # Workaround: name_prefix is limited to 38 chars.
      # Long cluster names (staging, production) overflow this limit.
      # Using explicit name bypasses the prefix (IAM name limit is 64 chars).
      iam_role_name            = "${var.cluster_name}-node-group"
      iam_role_use_name_prefix = false
    }
  }
}

VPC Peering (for ArgoCD hub-spoke)

ArgoCD on myapp-production-use1 needs to reach the private API endpoints of the 5 spoke clusters. VPC peering provides private connectivity without internet traversal.

prod-use1 (10.20.0.0/16)  ←──── VPC Peering ────► prod-usw2   (10.21.0.0/16)
prod-use1 (10.20.0.0/16)  ←──── VPC Peering ────► staging-use1 (10.10.0.0/16)
prod-use1 (10.20.0.0/16)  ←──── VPC Peering ────► staging-usw2 (10.11.0.0/16)

Dev clusters use public endpoints — no VPC peering needed.

# live/production/us-east-1/vpc-peering/terragrunt.hcl
# NOTE: vpc-peering configs CANNOT use include "root" with remote_state.
# They must define remote_state explicitly because the generate label
# conflicts with the parent. Define it inline instead.

locals {
  env    = "production"
  region = "us-east-1"
}

remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket         = "myapp-terraform-state-591120834781"
    key            = "production/us-east-1/vpc-peering/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "myapp-terraform-locks"
  }
}

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<-EOF
    provider "aws" {
      region = "us-east-1"
      assume_role {
        role_arn = "arn:aws:iam::591120834781:role/OrganizationAccountAccessRole"
      }
    }
    # Peer VPC is in the staging account — needs its own provider alias
    provider "aws" {
      alias  = "staging"
      region = "us-east-1"
      assume_role {
        role_arn = "arn:aws:iam::STAGING_ACCOUNT_ID:role/OrganizationAccountAccessRole"
      }
    }
  EOF
}

inputs = {
  requester_vpc_id = dependency.prod_use1_vpc.outputs.vpc_id
  accepter_vpc_id  = dependency.staging_use1_vpc.outputs.vpc_id
  # ... route table IDs, CIDR blocks, etc.
}

Running the Stack

# First apply: production us-east-1 (this region creates the AutoScaling SLR)
cd live/production/us-west-2
terragrunt run-all apply --terragrunt-non-interactive

# Second: production us-east-1 (SLR already exists, create_autoscaling_slr=false)
cd live/production/us-east-1
terragrunt run-all apply --terragrunt-non-interactive

# Check what changed before applying
cd live/staging/us-east-1
terragrunt run-all plan

# Destroy a specific module (e.g., for re-creating)
cd live/dev/us-east-1/eks
terragrunt destroy

State Management Best Practices

Each module has its own state file: {env}/{region}/{module}/terraform.tfstate.

Why not one big state file?

A corrupt or locked state file affects only one module, not the entire environment
terraform plan on EKS doesn't load/lock VPC state — faster, safer
Different engineers can work on different modules concurrently

State file key examples:

production/us-east-1/vpc/terraform.tfstate
production/us-east-1/eks/terraform.tfstate
production/us-east-1/iam/terraform.tfstate
production/us-west-2/vpc/terraform.tfstate

Common Pitfalls

Problem	Symptom	Fix
AutoScaling SLR doesn't exist	`MalformedPolicyDocumentException` on KMS create	Set `create_autoscaling_slr = true` in first region; `false` in subsequent
IAM name_prefix > 38 chars	`ValidationError: name_prefix` on node group create	Use `iam_role_name` + `iam_role_use_name_prefix = false`
VPC peering uses `include "root"`	`generate label already defined` error	Define `remote_state` block explicitly in vpc-peering configs
AWS SG description has Unicode	`Invalid description` on security group	Use plain ASCII only in SG descriptions — no arrows (→) or greater-than (>)

Summary

By the end of Part 3 you have:

✅ DRY Terragrunt root config (provider + backend auto-generated from path)
✅ VPC module with public/private subnets, NAT gateways, flow logs
✅ KMS module handling the AutoScaling SLR chicken-and-egg problem
✅ Dependency graph ensuring correct apply order
✅ Per-module S3 state isolation
✅ VPC peering between production hub and all spoke VPCs

Next: Part 4 — EKS Multi-Cluster: Six Clusters Across Two Regions

Part 2: AWS Foundation

Matthew — Wed, 18 Mar 2026 09:00:00 +0000

Part 2: AWS Foundation — Organizations, SSO, and Account Setup

Part of the series: Building a Production-Grade DevSecOps Pipeline on AWS

Introduction

The foundation of any serious AWS deployment is a well-structured multi-account setup. Running everything in one AWS account is the equivalent of putting all your files on a single server with no access controls — the blast radius of any mistake or breach is your entire infrastructure.

In this part we set up AWS Organizations with four accounts, configure AWS IAM Identity Center (SSO) for human access, and establish the IAM trust relationships that allow GitHub Actions to deploy to our clusters without static credentials.

Why Multi-Account?

┌─────────────────────────────────────────────────────────────────────┐
│  SINGLE ACCOUNT (anti-pattern)                                      │
│                                                                     │
│  Dev workloads ─────────────────────────────┐                       │
│  Staging workloads ─────────────────────────┤── Same IAM boundary   │
│  Production workloads ──────────────────────┘   Same VPC space      │
│                                                  Same billing       │
│  Risk: dev engineer accidentally deletes production RDS             │
│  Risk: security incident in dev reaches production secrets          │
└─────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────┐
│  MULTI-ACCOUNT (this guide)                                         │
│                                                                     │
│  Dev Account:         strong IAM isolation, cheap instance sizes    │
│  Staging Account:     production-like, but no real data             │
│  Production Account:  SCPs block destructive operations             │
│  Management Account:  no workloads, only ECR + SSO + billing        │
│                                                                     │
│  Benefit: IAM permissions are account-scoped                        │
│  Benefit: Service Control Policies (SCPs) protect production        │
│  Benefit: Separate billing per environment                          │
│  Benefit: VPC IP space per account (no CIDR conflicts)              │
└─────────────────────────────────────────────────────────────────────┘

Step 1: Create AWS Organizations

AWS Console → AWS Organizations → Create Organization

AWS Organizations gives you a single management pane for all accounts, consolidated billing, and — crucially — Service Control Policies (SCPs) that act as guardrails even for account root users.

After creating the organization, note your Organization ID (format: o-xxxxxxxxxx).

Step 2: Create Member Accounts

Navigate to AWS Organizations → AWS Accounts → Add an AWS Account.

Create three accounts:

Account Name	Purpose	Example ID
`myapp-dev`	Development environment	`557702566877`
`myapp-staging`	Staging environment	(your value)
`myapp-production`	Production environment	`591120834781`

Important: Use + email aliases to reuse your existing email. If your email is you@gmail.com, use you+aws-dev@gmail.com, you+aws-staging@gmail.com, etc. Gmail (and most providers) deliver these to the same inbox.

Tip: Record each account ID immediately. You will reference them throughout this series.

SCREENSHOT: AWS Organizations showing all 4 accounts in their OUs

Step 3: Organize Accounts into OUs

Organizational Units (OUs) let you apply different SCPs to groups of accounts.

AWS Console → AWS Organizations → AWS Accounts → Root

Create OUs:
  Root
  ├── Management (leave root account here)
  ├── Workloads
  │   ├── Dev        (move myapp-dev here)
  │   ├── Staging    (move myapp-staging here)
  │   └── Production (move myapp-production here)

Step 4: Apply Service Control Policies

SCPs are JSON IAM policies attached to OUs. They define the maximum permissions any principal in that OU can ever have — even the account root user cannot exceed them.

Apply this SCP to the Production OU to prevent accidental deletion of critical resources:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyDangerousEKSOperations",
      "Effect": "Deny",
      "Action": [
        "eks:DeleteCluster",
        "eks:DeleteNodegroup"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:PrincipalTag/AllowDestructive": "true"
        }
      }
    },
    {
      "Sid": "DenyRDSDelete",
      "Effect": "Deny",
      "Action": [
        "rds:DeleteDBInstance",
        "rds:DeleteDBCluster"
      ],
      "Resource": "*"
    },
    {
      "Sid": "RequireRegions",
      "Effect": "Deny",
      "NotAction": [
        "iam:*",
        "sts:*",
        "route53:*",
        "cloudfront:*",
        "waf:*",
        "acm:*",
        "support:*",
        "health:*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": [
            "us-east-1",
            "us-west-2"
          ]
        }
      }
    }
  ]
}

The region restriction SCP ensures no resources are accidentally created outside your approved regions. IAM, STS, Route53, and ACM are excluded because they are global services.

Step 5: Configure AWS IAM Identity Center (SSO)

AWS IAM Identity Center (formerly AWS SSO) lets your team log in with a single set of credentials across all accounts. It is far superior to creating IAM users in every account.

AWS Console → IAM Identity Center → Enable

Steps:
1. Choose identity source: "Identity Center directory" (built-in, no external IdP needed)
2. Create users for each team member
3. Create Permission Sets (these become IAM roles in each account)
4. Assign users to accounts with appropriate Permission Sets

Create two Permission Sets:

AdministratorAccess (for platform engineers):

Name: AdministratorAccess
Session duration: 8 hours
Managed policy: AdministratorAccess

ReadOnlyAccess (for developers / auditors):

Name: ReadOnlyAccess
Session duration: 8 hours
Managed policy: ReadOnlyAccess

Assign to accounts:

myapp-dev:         you → AdministratorAccess
myapp-staging:     you → AdministratorAccess
myapp-production:  you → AdministratorAccess
Management:        you → AdministratorAccess

Configure the AWS CLI for SSO:

# Run on your local machine
aws configure sso

# When prompted:
SSO session name: admin
SSO start URL: https://your-id.awsapps.com/start
SSO region: us-east-1
SSO registration scopes: sso:account:access

# After login, name each profile:
# Profile for dev/us-east-1: myapp-dev-use1
# Profile for dev/us-west-2: myapp-dev-usw2
# etc.

Your ~/.aws/config will look like:

[sso-session admin]
sso_start_url = https://your-id.awsapps.com/start
sso_region = us-east-1
sso_registration_scopes = sso:account:access

[profile myapp-prod-use1]
sso_session = admin
sso_account_id = 591120834781
sso_role_name = AdministratorAccess
region = us-east-1
output = json

[profile myapp-prod-usw2]
sso_session = admin
sso_account_id = 591120834781
sso_role_name = AdministratorAccess
region = us-west-2
output = json

[profile myapp-dev-use1]
sso_session = admin
sso_account_id = 557702566877
sso_role_name = AdministratorAccess
region = us-east-1
output = json

Authenticate:

aws sso login --sso-session admin
# Opens browser → log in → token saved locally for 8 hours

# Test:
aws sts get-caller-identity --profile myapp-prod-use1

SCREENSHOT: IAM Identity Center showing SSO portal with all accounts and permission sets

Step 6: GitHub OIDC — No Static AWS Keys in CI

This is one of the most important security decisions in the entire pipeline. Traditional CI/CD stores AWS access keys as GitHub Secrets. Those keys:

Never expire automatically
Are as powerful as the IAM user they belong to
Can be exfiltrated from logs if misconfigured

OIDC (OpenID Connect) eliminates static keys. GitHub Actions generates a short-lived JWT token for each workflow run. AWS validates this token cryptographically and exchanges it for a temporary STS credential that expires when the job ends.

┌─────────────────────────────────────────────────────────────────────┐
│  OIDC TOKEN FLOW                                                    │
│                                                                     │
│  GitHub Actions Job starts                                          │
│       │                                                             │
│       ▼                                                             │
│  GitHub generates JWT (signed by GitHub's OIDC provider)            │
│  Claims include:                                                    │
│    sub: repo:MatthewDipo/myapp:ref:refs/heads/main                  │
│    aud: sts.amazonaws.com                                           │
│       │                                                             │
│       ▼                                                             │
│  aws sts assume-role-with-web-identity                              │
│    --role-arn arn:aws:iam::ACCOUNT:role/ROLE                        │
│    --web-identity-token <JWT>                                       │
│       │                                                             │
│       ▼                                                             │
│  AWS validates JWT against GitHub's OIDC endpoint                   │
│  (https://token.actions.githubusercontent.com)                      │
│       │                                                             │
│       ▼                                                             │
│  Returns: AccessKeyId + SecretAccessKey + SessionToken              │
│  (valid for 1 hour maximum, then automatically expire)              │
└─────────────────────────────────────────────────────────────────────┘

Terraform to create the OIDC provider (management account, us-east-1 only — it is global):

# _modules/iam/main.tf

# Fetch GitHub's OIDC thumbprint automatically
data "tls_certificate" "github" {
  url = "https://token.actions.githubusercontent.com"
}

resource "aws_iam_openid_connect_provider" "github" {
  count = var.create_github_oidc ? 1 : 0

  url = "https://token.actions.githubusercontent.com"

  client_id_list = ["sts.amazonaws.com"]

  thumbprint_list = [
    data.tls_certificate.github.certificates[0].sha1_fingerprint
  ]
}

# IAM role for GitHub Actions CI — one per cluster
resource "aws_iam_role" "github_ci" {
  name = "${var.cluster_name}-github-ci"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = {
        Federated = var.github_oidc_provider_arn
      }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
        }
        StringLike = {
          # Only main branch of your specific repo can assume this role
          "token.actions.githubusercontent.com:sub" =
            "repo:${var.github_user}/${var.github_repo}:ref:refs/heads/main"
        }
      }
    }]
  })
}

resource "aws_iam_role_policy" "github_ci" {
  name = "github-ci-policy"
  role = aws_iam_role.github_ci.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "ECRAuth"
        Effect = "Allow"
        Action = ["ecr:GetAuthorizationToken"]
        Resource = "*"
      },
      {
        Sid    = "ECRPush"
        Effect = "Allow"
        Action = [
          "ecr:BatchCheckLayerAvailability",
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage",
          "ecr:InitiateLayerUpload",
          "ecr:UploadLayerPart",
          "ecr:CompleteLayerUpload",
          "ecr:PutImage",
          "ecr:DescribeImages",
          "ecr:ListImages"
        ]
        Resource = "arn:aws:ecr:*:${var.account_id}:repository/myapp"
      },
      {
        Sid    = "KMSCosignSign"
        Effect = "Allow"
        Action = [
          "kms:Sign",
          "kms:GetPublicKey",
          "kms:DescribeKey"
        ]
        Resource = var.cosign_kms_key_arn
      },
      {
        Sid    = "S3AuditWrite"
        Effect = "Allow"
        Action = ["s3:PutObject"]
        Resource = "${var.audit_bucket_arn}/ci-push-audit/*"
      }
    ]
  })
}

Key security insight: The Condition block in the trust policy is critical. It restricts role assumption to:

Only your specific repository (repo:MatthewDipo/myapp)
Only the main branch

A fork of your repository, or a branch other than main, cannot assume this role.

Step 7: ECR Repository in the Management Account

We store Docker images in the management account's ECR rather than per-environment accounts. This means one place to manage image lifecycle policies and one IAM policy for cross-account pull permissions.

# _modules/ecr/main.tf

resource "aws_ecr_repository" "app" {
  name                 = var.name
  image_tag_mutability = "IMMUTABLE"   # Tags cannot be overwritten

  image_scanning_configuration {
    scan_on_push = true   # ECR runs basic CVE scan on every push
  }

  encryption_configuration {
    encryption_type = "KMS"
    kms_key         = var.kms_key_arn
  }
}

resource "aws_ecr_lifecycle_policy" "app" {
  repository = aws_ecr_repository.app.name

  policy = jsonencode({
    rules = [
      {
        rulePriority = 1
        description  = "Keep last 30 tagged images"
        selection = {
          tagStatus     = "tagged"
          tagPrefixList = ["sha-"]
          countType     = "imageCountMoreThan"
          countNumber   = 30
        }
        action = { type = "expire" }
      },
      {
        rulePriority = 2
        description  = "Delete untagged images after 1 day"
        selection = {
          tagStatus   = "untagged"
          countType   = "sinceImagePushed"
          countUnit   = "days"
          countNumber = 1
        }
        action = { type = "expire" }
      }
    ]
  })
}

# Cross-account pull policy — allows dev/staging/prod accounts to pull images
resource "aws_ecr_repository_policy" "cross_account" {
  repository = aws_ecr_repository.app.name

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "CrossAccountPull"
        Effect = "Allow"
        Principal = {
          AWS = [
            "arn:aws:iam::${var.dev_account_id}:root",
            "arn:aws:iam::${var.staging_account_id}:root",
            "arn:aws:iam::${var.prod_account_id}:root"
          ]
        }
        Action = [
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage",
          "ecr:BatchCheckLayerAvailability"
        ]
      }
    ]
  })
}

IMMUTABLE tags mean that once you push sha-abc123, that tag forever points to that exact image digest. No one can silently overwrite an existing tag with a different image — a subtle but important supply chain security control.

SCREENSHOT: ECR repository showing images with sha- tags and scan results

Step 8: KMS Keys for Encryption at Rest

Each environment gets its own KMS key for encrypting:

EKS Kubernetes secrets (etcd encryption)
EBS volumes (Prometheus/Grafana/Velero PVCs)
ECR images
S3 buckets (Velero backups, CI audit logs)

# _modules/kms/main.tf

resource "aws_kms_key" "main" {
  description             = "${var.env}-${var.region}-main"
  deletion_window_in_days = 30
  enable_key_rotation     = true   # Rotate annually, automatically

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "Enable IAM User Permissions"
        Effect = "Allow"
        Principal = { AWS = "arn:aws:iam::${var.account_id}:root" }
        Action   = "kms:*"
        Resource = "*"
      },
      {
        Sid    = "Allow EKS"
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
        Action   = ["kms:Encrypt", "kms:Decrypt", "kms:GenerateDataKey*", "kms:DescribeKey"]
        Resource = "*"
      },
      {
        Sid    = "Allow AutoScaling"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::${var.account_id}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        }
        Action   = ["kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:DescribeKey", "kms:CreateGrant"]
        Resource = "*"
      }
    ]
  })
}

resource "aws_kms_alias" "main" {
  name          = "alias/${var.env}-${var.region}-main"
  target_key_id = aws_kms_key.main.key_id
}

Lesson learned: The AWSServiceRoleForAutoScaling must exist in the account before you can reference it in the KMS key policy. In fresh accounts, create this Service Linked Role first or AWS will reject the key policy with MalformedPolicyDocumentException. See Part 3 for the Terragrunt pattern that handles this.

Step 9: Terragrunt Root Configuration

Before writing any Terragrunt configs, establish the root terragrunt.hcl that all child configs inherit:

# live/terragrunt.hcl  (root)

locals {
  # Parse path to extract env and region
  # e.g., live/production/us-east-1/eks → env=production, region=us-east-1
  path_parts = split("/", path_relative_to_include())
  env        = local.path_parts[0]
  region     = local.path_parts[1]

  account_ids = {
    dev        = "557702566877"
    staging    = "STAGING_ACCOUNT_ID"
    production = "591120834781"
    management = "MGMT_ACCOUNT_ID"
  }

  account_id = local.account_ids[local.env]
}

# Generate provider.tf in every child directory
generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider "aws" {
  region = "${local.region}"
  assume_role {
    role_arn = "arn:aws:iam::${local.account_id}:role/OrganizationAccountAccessRole"
  }
  default_tags {
    tags = {
      Environment = "${local.env}"
      ManagedBy   = "Terraform"
      Project     = "myapp"
    }
  }
}
EOF
}

# Generate backend.tf — S3 state, DynamoDB lock table
remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket         = "myapp-terraform-state-${local.account_id}"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "myapp-terraform-locks"
    role_arn       = "arn:aws:iam::${local.account_id}:role/OrganizationAccountAccessRole"
  }
}

The generate "provider" block means you never write a provider.tf by hand. Every module automatically gets the correct AWS account and region based purely on its directory path. This is the key DRY benefit of Terragrunt.

Understanding `OrganizationAccountAccessRole`

When you create a member account via AWS Organizations, AWS automatically creates a role called OrganizationAccountAccessRole in that account. This role trusts your management account, allowing management account principals to assume it and perform actions in the member account.

This is how Terraform (running with management account credentials) deploys infrastructure into dev, staging, and production without needing separate credentials per account.

Management Account (your terminal / CI)
    │
    │ sts:AssumeRole
    ▼
arn:aws:iam::591120834781:role/OrganizationAccountAccessRole
    │
    │ (full AdministratorAccess in production account)
    ▼
Production Account resources

Step 10: Bootstrap S3 State Buckets

Before running any Terragrunt, each account needs its S3 bucket and DynamoDB table for Terraform state.

# Run this once per account (adjust account ID and profile)
for PROFILE in myapp-dev-use1 myapp-staging-use1 myapp-prod-use1; do
  ACCOUNT_ID=$(aws sts get-caller-identity --profile $PROFILE --query Account --output text)
  REGION="us-east-1"

  # Create state bucket
  aws s3api create-bucket \
    --bucket "myapp-terraform-state-${ACCOUNT_ID}" \
    --region $REGION \
    --profile $PROFILE

  # Enable versioning (lets you recover from bad applies)
  aws s3api put-bucket-versioning \
    --bucket "myapp-terraform-state-${ACCOUNT_ID}" \
    --versioning-configuration Status=Enabled \
    --profile $PROFILE

  # Enable encryption
  aws s3api put-bucket-encryption \
    --bucket "myapp-terraform-state-${ACCOUNT_ID}" \
    --server-side-encryption-configuration '{
      "Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]
    }' \
    --profile $PROFILE

  # Block public access
  aws s3api put-public-access-block \
    --bucket "myapp-terraform-state-${ACCOUNT_ID}" \
    --public-access-block-configuration \
      "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true" \
    --profile $PROFILE

  # Create DynamoDB lock table
  aws dynamodb create-table \
    --table-name myapp-terraform-locks \
    --attribute-definitions AttributeName=LockID,AttributeType=S \
    --key-schema AttributeName=LockID,KeyType=HASH \
    --billing-mode PAY_PER_REQUEST \
    --region $REGION \
    --profile $PROFILE

  echo "State backend ready for account $ACCOUNT_ID"
done

Summary

By the end of this part you have:

✅ AWS Organizations with four accounts (management, dev, staging, production)
✅ Service Control Policies protecting production from accidental destruction
✅ AWS SSO for human access (no IAM users with permanent credentials)
✅ GitHub OIDC provider enabling keyless CI authentication
✅ ECR repository with immutable tags, cross-account pull, and lifecycle policies
✅ KMS keys for encryption at rest in every environment
✅ Terragrunt root config that automatically derives account/region from directory path
✅ S3 + DynamoDB Terraform state backend per account

If this was useful, follow me on dev.to — I publish Part 3 next Wednesday covering the Infrastructure as Code — Terraform Modules + Terragrunt.

Questions? Drop them in the comments — I read and reply to every one.

Next: Part 3 — Infrastructure as Code: Terraform Modules + Terragrunt

Part 1: Architecture Overview

Matthew — Wed, 11 Mar 2026 09:24:20 +0000

Building a Production-Grade DevSecOps Pipeline on AWS: A Complete Guide

Series Overview: This 10-part series walks you through building a real-world, production-grade DevSecOps platform on AWS from scratch — the same architecture used at mature engineering organizations. By the end, you will have six EKS clusters, a GitOps delivery model, a hardened CI/CD pipeline, runtime security, full-stack observability, and automated disaster recovery.

Live system: Everything in this series is running in production right now.
Check it: curl https://www.matthewoladipupo.dev/health
→ {"status":"healthy","region":"us-east-1"}

This is Part 1 of a 10-part series. You can follow the full series here on dev.to.

Part 1: Architecture Overview & What We Are Building

Introduction

Most DevSecOps tutorials show you one piece of the puzzle — a Kubernetes cluster here, a CI pipeline there. This series is different. We build the entire platform end to end: infrastructure-as-code, multi-environment clusters, GitOps, security policy enforcement, runtime threat detection, secrets management, observability, canary deployments, autoscaling, and backup — all wired together the way a production engineering team would actually build it.

Every component in this guide is running live. The screenshots you will see throughout this series come from the actual deployment at matthewoladipupo.dev.

The complete system: GitHub → CI/CD → ECR → ArgoCD hub → 6 EKS clusters
across 3 environments and 2 AWS regions. Every component is covered in this
10-part series.

What you will build:

Layer	Tools
Infrastructure as Code	Terraform + Terragrunt
Cloud Platform	AWS (multi-account, multi-region)
Container Orchestration	Amazon EKS (6 clusters)
GitOps Delivery	ArgoCD (hub-spoke)
CI/CD Pipeline	GitHub Actions
Container Security	Trivy, Cosign, Distroless images
Policy Enforcement	Kyverno
Runtime Security	Falco
Secrets Management	AWS Secrets Manager + External Secrets Operator
Monitoring	Prometheus + Grafana (kube-prometheus-stack)
Logging	Fluent Bit → AWS CloudWatch
Canary Deployments	Argo Rollouts
Autoscaling	Karpenter + HPA
Backup & DR	Velero + S3
Web Application Firewall	AWS WAF v2
Threat Detection	AWS GuardDuty
DNS & TLS	Route53 + ACM (wildcard cert)
Load Balancing	AWS Load Balancer Controller

High-Level Architecture

The platform follows a hub-spoke GitOps model across three environments and two AWS regions.

┌─────────────────────────────────────────────────────────────────────────────────┐
│                            AWS ORGANIZATION                                     │
│                                                                                 │
│  ┌──────────────────┐    ┌──────────────────┐    ┌──────────────────────────┐   │
│  │  Management Acct │    │  Dev Account     │    │  Staging Account         │   │
│  │  (ECR, CI Audit) │    │  557702566877    │    │                          │   │
│  │                  │    │  ┌─────────────┐ │    │  ┌────────┐ ┌────────┐   │   │
│  │  ┌───────────┐   │    │  │ EKS use1    │ │    │  │EKS use1│ │EKS usw2│   │   │
│  │  │ECR: myapp │   │    │  │ (public ep) │ │    │  │        │ │        │   │   │
│  │  └───────────┘   │    │  └─────────────┘ │    │  └────────┘ └────────┘   │   │
│  └──────────────────┘    │  ┌─────────────┐ │    └──────────────────────────┘   │
│                          │  │ EKS usw2    │ │                                   │
│                          │  │ (public ep) │ │    ┌──────────────────────────┐   │
│                          │  └─────────────┘ │    │  Production Account      │   │
│                          └──────────────────┘    │  591120834781            │   │
│                                                  │                          │   │
│                                                  │  ┌────────┐ ┌────────┐   │   │
│                                                  │  │EKS use1│ │EKS usw2│   │   │
│                                                  │  │  HUB   │ │ Spoke  │   │   │
│                                                  │  │(ArgoCD)│ │        │   │   │
│                                                  │  └────────┘ └────────┘   │   │
│                                                  └──────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────────────────┘

Detailed Architecture Diagram

                              DEVELOPER WORKFLOW
                              ──────────────────
                              git push → GitHub

                                      │
                                      ▼
┌─────────────────────────────────────────────────────────────────────────────────┐
│                         GITHUB ACTIONS CI PIPELINE                              │
│                                                                                 │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────────┐   │
│  │  Lint +  │  │  Trivy   │  │  Docker  │  │  Cosign  │  │  Push to ECR     │   │
│  │  Test    │→ │  Scan    │→ │  Build   │→ │  Sign    │→ │  (multi-region)  │   │
│  └──────────┘  └──────────┘  └──────────┘  └──────────┘  └──────────────────┘   │
│                                                                    │            │
│                                              OIDC (no static keys) │            │
│                                              IAM Roles per cluster │            │
└────────────────────────────────────────────────────────────────────┼────────────┘
                                                                     │
                                                                     ▼
                                                             myapp-gitops repo
                                                           (image tag updated)
                                                                     │
                                                                     ▼
┌─────────────────────────────────────────────────────────────────────────────────┐
│                    ARGOCD HUB  (myapp-production-use1)                          │
│                                                                                 │
│   ApplicationSets (list generators per cluster)                                 │
│   ┌─────────────────────────────────────────────────────────────────────────┐   │
│   │  environments/  infrastructure/  argocd/                                │   │
│   │  ├─ dev         ├─ monitoring    └─ project-*.yaml                      │   │
│   │  ├─ staging     ├─ logging                                              │   │
│   │  └─ production  ├─ eso                                                  │   │
│   │                 ├─ kyverno                                              │   │
│   │                 ├─ falco                                                │   │
│   │                 ├─ velero                                               │   │
│   │                 ├─ karpenter                                            │   │
│   │                 └─ argo-rollouts                                        │   │
│   └─────────────────────────────────────────────────────────────────────────┘   │
│                                                                                 │
│   Syncs to ──────────────────────────────────────────────────────────────────►  │
└──────┬──────────────────────────────────────────────────────────────────────────┘
       │  VPC Peering (private endpoints)
       ├────────────────────────────────► myapp-production-usw2
       ├────────────────────────────────► myapp-staging-use1
       ├────────────────────────────────► myapp-staging-usw2
       ├────────────────────────────────► myapp-dev-use1
       └────────────────────────────────► myapp-dev-usw2

Per-Cluster Component Stack

Every cluster runs the same security and observability baseline. The diagram below shows what runs on each cluster after bootstrapping:

┌──────────────────────────────────────────────────────────────────────┐
│                    EKS CLUSTER (per-cluster stack)                   │
│                                                                      │
│  SYSTEM NAMESPACES                                                   │
│  ┌─────────────┐  ┌──────────────────┐  ┌────────────────────────┐   │
│  │  kube-system│  │  kyverno         │  │  falco                 │   │
│  │  (aws-lbc,  │  │  (policy engine) │  │  (runtime security)    │   │
│  │   coreDNS)  │  │                  │  │                        │   │
│  └─────────────┘  └──────────────────┘  └────────────────────────┘   │
│                                                                      │
│  ┌─────────────────────┐  ┌───────────────────────────────────────┐  │
│  │  external-secrets   │  │  monitoring                           │  │
│  │  (ESO operator)     │  │  Prometheus ── Grafana ── Alertmanager│  │
│  └─────────────────────┘  └───────────────────────────────────────┘  │
│                                                                      │
│  ┌───────────────────────┐  ┌──────────────────┐  ┌─────────────┐    │
│  │  logging              │  │  velero          │  │  karpenter  │    │
│  │  (Fluent Bit DS)      │  │  (backup)        │  │  (prod only)│    │
│  └───────────────────────┘  └──────────────────┘  └─────────────┘    │
│                                                                      │
│  ┌───────────────────────┐  ┌──────────────────────────────────────┐ │
│  │  argo-rollouts        │  │  myapp (prod) / myapp (dev/staging)  │ │
│  │  (canary controller)  │  │  Rollout ──► canary ──► stable       │ │
│  └───────────────────────┘  └──────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────┘

Network Architecture

                        AWS REGION: us-east-1
┌───────────────────────────────────────────────────────────┐
│  VPC: production-use1  (10.20.0.0/16)                     │
│                                                           │
│  ┌──────────────────────────┐  ┌──────────────────────┐   │
│  │  Public Subnets          │  │  Private Subnets     │   │
│  │  10.20.0.0/24 (us-east-1a)  10.20.8.0/21 (use1a)   │   │
│  │  10.20.1.0/24 (us-east-1b)  10.20.16.0/21 (use1b)  │   │
│  │  10.20.2.0/24 (us-east-1c)  10.20.24.0/21 (use1c)  │   │
│  │                          │  │                      │   │
│  │  NAT Gateways            │  │  EKS Node Groups     │   │
│  │  Internet Gateway        │  │  EKS API endpoint    │   │
│  │  ALB (internet-facing)   │  │  (private only)      │   │
│  └──────────────────────────┘  └──────────────────────┘   │
└───────────────────────────────────────────────────────────┘
          │                            │
          │ VPC Peering                │
          │ (private, encrypted)       │
          ▼                            ▼
┌────────────────────┐    ┌────────────────────────────────┐
│  VPC: prod-usw2    │    │  VPC: staging-use1             │
│  (10.21.0.0/16)    │    │  (10.10.0.0/16)                │
└────────────────────┘    └────────────────────────────────┘

Internet Traffic Flow:
User → Route53 (latency routing) → ALB → AWS WAF → ALB Target Group
     → Pod (via ALB target type: ip, directly to pod IP)
     → Response back to user

CI/CD Pipeline Flow

┌─────────────────────────────────────────────────────────────────────┐
│  GitHub: MatthewDipo/myapp                                          │
│                                                                     │
│  Developer: git push origin main                                    │
└──────────────────────────────┬──────────────────────────────────────┘
                               │ triggers
                               ▼
┌─────────────────────────────────────────────────────────────────────┐
│  GitHub Actions: .github/workflows/ci.yaml                          │
│                                                                     │
│  Job 1: lint-and-test                                               │
│    └─ npm ci && npm test                                            │
│                                                                     │
│  Job 2: scan (needs: lint-and-test)                                 │
│    └─ trivy image --severity HIGH,CRITICAL                          │
│                                                                     │
│  Job 3: build-push-sign (needs: scan)                               │
│    ├─ OIDC → assume IAM role (no static AWS keys)                   │
│    ├─ docker build (distroless:nonroot base)                        │
│    ├─ docker push → ECR us-east-1 (management account)              │
│    ├─ docker push → ECR us-west-2 (management account)              │
│    ├─ cosign sign --key awskms:// (KMS signing key)                 │
│    └─ Write S3 audit log (image digest + timestamp)                 │
│                                                                     │
│  Job 4: update-gitops (needs: build-push-sign)                      │
│    └─ Patch myapp-gitops/apps/myapp/values-*.yaml (image.tag)       │
└──────────────────────────────┬──────────────────────────────────────┘
                               │ git push to myapp-gitops
                               ▼
┌─────────────────────────────────────────────────────────────────────┐
│  GitHub: MatthewDipo/myapp-gitops                                   │
│  ArgoCD detects diff → triggers sync per cluster                    │
│                                                                     │
│  Production: Argo Rollouts Canary                                   │
│    ├─ Step 1: setWeight 20% (canary gets 20% traffic)               │
│    ├─ Step 2: pause 5 minutes                                       │
│    ├─ Step 3: AnalysisRun (check error rate < 1%)                   │
│    └─ Step 4: setWeight 100% (promote to stable)                    │
│                                                                     │
│  Dev/Staging: Rolling Update (immediate)                            │
└─────────────────────────────────────────────────────────────────────┘

Security Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                       SECURITY LAYERS                                   │
│                                                                         │
│  Layer 1: SUPPLY CHAIN SECURITY                                         │
│  ┌────────────────────────────────────────────────────────────────────┐ │
│  │  Source: GitHub branch protection + required reviews               │ │
│  │  Build:  Trivy CVE scan (fails pipeline on HIGH/CRITICAL)          │ │
│  │  Image:  Distroless base (no shell, no package manager)            │ │
│  │  Sign:   Cosign + AWS KMS (cryptographic attestation)              │ │
│  │  Verify: Kyverno policy blocks unsigned images at admission        │ │
│  └────────────────────────────────────────────────────────────────────┘ │
│                                                                         │
│  Layer 2: INFRASTRUCTURE SECURITY                                       │
│  ┌────────────────────────────────────────────────────────────────────┐ │
│  │  Network:  Private EKS endpoints (staging/prod)                    │ │
│  │            VPC peering (no internet traversal between clusters)    │ │
│  │            NetworkPolicies (deny-all default, allow explicitly)    │ │
│  │  IAM:      IRSA (pod-level IAM, no node-level credentials)         │ │
│  │            OIDC for GitHub Actions (no static AWS keys in CI)      │ │
│  │  Secrets:  AWS Secrets Manager (never stored in Git)               │ │
│  │  KMS:      Envelope encryption for EKS secrets + ECR + S3          │ │
│  └────────────────────────────────────────────────────────────────────┘ │
│                                                                         │
│  Layer 3: WORKLOAD ADMISSION CONTROL (Kyverno)                          │
│  ┌────────────────────────────────────────────────────────────────────┐ │
│  │  ✗ Block privileged containers                                     │ │
│  │  ✗ Block hostPath volume mounts                                    │ │
│  │  ✗ Block containers running as root (uid=0)                        │ │
│  │  ✗ Block images without valid Cosign signature                     │ │
│  │  ✗ Block missing resource limits                                   │ │
│  │  ✓ Allow myapp from ECR with valid signature                       │ │
│  └────────────────────────────────────────────────────────────────────┘ │
│                                                                         │
│  Layer 4: RUNTIME THREAT DETECTION (Falco)                              │
│  ┌────────────────────────────────────────────────────────────────────┐ │
│  │  Alert on: shell spawned in container                              │ │
│  │  Alert on: sensitive file read (/etc/shadow, /etc/passwd)          │ │
│  │  Alert on: unexpected outbound network connection                  │ │
│  │  Alert on: privilege escalation attempts                           │ │
│  │  Output: CloudWatch Logs (via Fluent Bit)                          │ │
│  └────────────────────────────────────────────────────────────────────┘ │
│                                                                         │
│  Layer 5: PERIMETER SECURITY                                            │
│  ┌────────────────────────────────────────────────────────────────────┐ │
│  │  AWS WAF v2: Managed rules (OWASP Top 10, SQL injection, XSS)      │ │
│  │  AWS GuardDuty: Account-level threat intelligence                  │ │
│  │  ACM: TLS 1.2+ enforced, HTTP → HTTPS redirect                     │ │
│  └────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘

GitOps Repository Structure

Two GitHub repositories drive everything after the CI pipeline builds the image:

MatthewDipo/myapp-gitops/
│
├── argocd/
│   ├── project-dev.yaml             # AppProject: dev clusters
│   ├── project-staging.yaml         # AppProject: staging clusters
│   └── project-production.yaml      # AppProject: production clusters
│
├── environments/
│   ├── dev/
│   │   └── applicationset.yaml      # Deploys myapp to dev-use1, dev-usw2
│   ├── staging/
│   │   └── applicationset.yaml      # Deploys myapp to staging-use1, staging-usw2
│   └── production/
│       └── applicationset.yaml      # Deploys myapp to prod-use1, prod-usw2
│
├── infrastructure/
│   ├── monitoring/
│   │   ├── applicationset.yaml      # kube-prometheus-stack (4 clusters)
│   │   ├── prometheus-values.yaml
│   │   └── alert-rules/
│   │       └── applicationset.yaml  # PrometheusRule CRDs
│   ├── logging/
│   │   └── applicationset.yaml      # Fluent Bit DaemonSet (6 clusters)
│   ├── eso/
│   │   └── applicationset.yaml      # External Secrets Operator (6 clusters)
│   ├── kyverno/
│   │   └── applicationset.yaml      # Kyverno + policies (6 clusters)
│   ├── falco/
│   │   └── applicationset.yaml      # Falco DaemonSet (6 clusters)
│   ├── velero/
│   │   └── applicationset.yaml      # Velero (6 clusters)
│   ├── karpenter/
│   │   ├── applicationset.yaml      # Karpenter controller (2 prod)
│   │   └── nodepools/
│   │       └── applicationset.yaml  # NodePool + EC2NodeClass CRDs
│   └── argo-rollouts/
│       └── applicationset.yaml      # Argo Rollouts controller (2 prod)
│
└── apps/
    └── myapp/                       # Helm chart for the application
        ├── Chart.yaml
        ├── values.yaml              # Default values
        ├── values-dev.yaml
        ├── values-staging.yaml
        ├── values-production.yaml
        └── templates/
            ├── deployment.yaml      # Deployment OR Rollout (conditional)
            ├── service.yaml
            ├── service-canary.yaml  # Canary service (prod only)
            ├── ingress.yaml
            ├── hpa.yaml
            ├── networkpolicy.yaml
            ├── serviceaccount.yaml
            ├── external-secret.yaml
            ├── servicemonitor.yaml  # Prometheus scraping
            └── analysis-template.yaml

Infrastructure Repository Structure

MatthewDipo/myapp-infra/
│
├── _modules/                        # Reusable Terraform modules
│   ├── vpc/
│   ├── eks/
│   ├── kms/
│   ├── iam/
│   ├── ecr/
│   ├── waf/
│   ├── guardduty/
│   ├── eso-irsa/
│   ├── fluent-bit-irsa/
│   ├── karpenter/
│   └── velero/
│
└── live/                            # Terragrunt configurations (per env/region)
    ├── terragrunt.hcl               # Root config (provider, remote state)
    ├── dev/
    │   ├── us-east-1/
    │   │   ├── vpc/terragrunt.hcl
    │   │   ├── kms/terragrunt.hcl
    │   │   ├── eks/terragrunt.hcl
    │   │   ├── iam/terragrunt.hcl
    │   │   └── fluent-bit-irsa/terragrunt.hcl
    │   └── us-west-2/
    │       └── ... (mirror of use1)
    ├── staging/
    │   ├── us-east-1/
    │   │   ├── vpc/ kms/ eks/ iam/
    │   │   ├── waf/terragrunt.hcl
    │   │   ├── guardduty/terragrunt.hcl
    │   │   ├── eso-irsa/terragrunt.hcl
    │   │   └── fluent-bit-irsa/terragrunt.hcl
    │   └── us-west-2/
    │       └── ...
    └── production/
        ├── us-east-1/
        │   ├── vpc/ kms/ eks/ iam/
        │   ├── waf/
        │   ├── guardduty/
        │   ├── eso-irsa/
        │   ├── fluent-bit-irsa/
        │   ├── karpenter/
        │   └── velero/
        └── us-west-2/
            └── ...

AWS Account Structure

┌──────────────────────────────────────────────┐
│         AWS Organizations Root               │
│                                              │
│  ┌───────────────────────────────────────┐   │
│  │  Management / Root Account            │   │
│  │  • AWS SSO (Identity Center)          │   │
│  │  • ECR repositories (shared)          │   │
│  │  • S3 CI audit bucket                 │   │
│  │  • GitHub OIDC provider               │   │
│  └───────────────────────────────────────┘   │
│           │            │            │        │
│           ▼            ▼            ▼        │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐    │
│  │   Dev    │  │ Staging  │  │Production│    │
│  │ Account  │  │ Account  │  │ Account  │    │
│  │          │  │          │  │          │    │
│  │ 2x EKS   │  │ 2x EKS   │  │ 2x EKS   │    │
│  │ VPCs     │  │ VPCs     │  │ VPCs     │    │
│  │ KMS keys │  │ KMS keys │  │ KMS keys │    │
│  │ Secrets  │  │ Secrets  │  │ Secrets  │    │
│  │ Manager  │  │ Manager  │  │ Manager  │    │
│  └──────────┘  └──────────┘  └──────────┘    │
└──────────────────────────────────────────────┘

Technology Decision Rationale

Understanding WHY each tool was chosen matters as much as knowing HOW to configure it.

Terragrunt over plain Terraform
Terragrunt provides DRY (Don't Repeat Yourself) configuration. Without it, you would have near-identical provider, backend, and module blocks repeated across 18+ directories. Terragrunt's include and dependency blocks eliminate 90% of that duplication while keeping each environment's overrides explicit and auditable.

ArgoCD hub-spoke over fleet management per cluster
Running ArgoCD on every cluster is operationally expensive. The hub-spoke model means one ArgoCD installation manages all six clusters via VPC peering. This single pane of glass dramatically simplifies debugging — you see all cluster states in one place.

Kyverno over OPA/Gatekeeper
Kyverno policies are written in YAML and operate on the same resource schema as Kubernetes objects. OPA/Gatekeeper requires learning Rego, a purpose-built policy language. For Kubernetes-native teams, Kyverno is faster to adopt and maintain.

External Secrets Operator over Sealed Secrets
Sealed Secrets encrypts secrets and commits them to Git — meaning the encrypted value is your source of truth. ESO keeps secrets out of Git entirely: the secret lives in AWS Secrets Manager, and ESO fetches it at runtime with IRSA credentials. This is a fundamentally stronger posture because a compromised Git repo never exposes secret material.

Argo Rollouts over plain Kubernetes rolling updates
Rolling updates are binary — you either roll forward or roll back. Argo Rollouts adds weighted traffic splitting between stable and canary versions, analysis runs (automated metric-based promotion gates), and pause steps for manual inspection. A canary deployment that automatically fails back on a rising error rate is far safer than a rolling update you monitor manually.

Distroless base images
The Google distroless nonroot image contains only the application runtime and its direct dependencies — no shell (sh, bash), no package manager (apt, apk), no curl or wget. If an attacker achieves code execution inside the container, they have almost no tools available to escalate or exfiltrate. Combined with Falco alerting on shell spawning, you get both prevention and detection.

Cost Estimate

These are rough estimates for running the full stack 24/7 in AWS us-east-1 + us-west-2. Production workloads should be sized to actual usage.

Component	Approx. Monthly Cost
6x EKS cluster control planes	~$216 ($0.10/hr × 6)
12x EC2 t3.medium nodes (2 per cluster)	~$300
6x EBS volumes (gp2, 50–100GB each)	~$60
NAT Gateways (2 per VPC, 6 VPCs)	~$250
ALBs (production + monitoring)	~$30
Route53 hosted zone + queries	~$5
ACM (free)	$0
ECR storage	~$5
CloudWatch logs (Fluent Bit)	~$15
S3 (Velero backups, CI audit)	~$10
AWS WAF	~$15
GuardDuty	~$10
Secrets Manager	~$2
Total	~$918/month

Cost reduction tip: For a demo/learning setup, use 1 node per cluster, t3.small instances, and skip the second region. This brings the cost to approximately $200–300/month.

Prerequisites

Before starting Part 2, ensure you have the following:

Tools to install:

# AWS CLI v2
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip && sudo ./aws/install

# Terraform 1.6+
brew install terraform   # or download from terraform.io

# Terragrunt 0.54+
brew install terragrunt  # or download from terragrunt.gruntwork.io

# kubectl
curl -LO "https://dl.k8s.io/release/$(curl -sL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

# Helm 3
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# ArgoCD CLI
curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
sudo install -m 555 argocd /usr/local/bin/argocd

# Cosign
curl -sSfL https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64 \
  -o cosign && sudo install -m 0755 cosign /usr/local/bin/cosign

# Trivy
sudo apt-get install wget apt-transport-https gnupg lsb-release
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
echo "deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" \
  | sudo tee /etc/apt/sources.list.d/trivy.list
sudo apt-get update && sudo apt-get install trivy

AWS accounts needed:

AWS Organization root account (or a management account)
Three member accounts: dev, staging, production
AWS SSO (IAM Identity Center) configured

GitHub:

Two repositories: myapp (application code) and myapp-gitops (manifests)
A Personal Access Token for ArgoCD to pull from the GitOps repo

Domain name:

A registered domain you control (we use matthewoladipupo.dev)
Nameservers pointed to Route53

Series Roadmap

Part	Title
Part 1	Architecture Overview (this article)
Part 2	AWS Foundation: Organizations, SSO, and Account Setup
Part 3	Infrastructure as Code: Terraform Modules + Terragrunt
Part 4	EKS Multi-Cluster: Six Clusters Across Two Regions
Part 5	GitOps with ArgoCD: Hub-Spoke Model
Part 6	CI/CD Pipeline: GitHub Actions, Trivy, Cosign, ECR
Part 7	Secrets Management: AWS Secrets Manager + ESO + IRSA
Part 8	Security Stack: Kyverno, Falco, WAF, GuardDuty
Part 9	Observability: Prometheus, Grafana, Fluent Bit, CloudWatch
Part 10	Resilience: Karpenter, HPA, Argo Rollouts, Velero
Live Data	Appendix
Codes	Runbook

Screenshot Placeholders

SCREENSHOT: ArgoCD UI showing all 6 clusters registered and all ApplicationSets synced

SCREENSHOT: Grafana dashboard — Node CPU/Memory overview across production clusters

SCREENSHOT: GitHub Actions workflow showing all steps passing (lint → scan → build → sign → push → gitops update)

SCREENSHOT: AWS ECR showing signed image with cosign attestation tag

If this was useful, follow me on dev.to — I will publish Part 2 next Wednesday covering the AWS Organizations + IAM Identity Center setup.
Questions? Drop them in the comments — I read and reply to every one.

Next up: Part 2 — AWS Foundation: Organizations, SSO, and Account Setup

Source code: myapp-infra | myapp-gitops | myapp
Runbook: Operations Guide — every operational procedure for this pipeline.

Forem: Matthew

Part 6: CI/CD Pipeline

Part 6: CI/CD Pipeline — GitHub Actions, Trivy, Cosign, and ECR

Introduction

The Application (myapp)

Dockerfile — Distroless Nonroot

Pipeline Overview

Full GitHub Actions Workflow

GitHub Repository Variables (not secrets)

Cosign Image Signing

Image Naming Convention

Pre-commit Hooks

Pipeline Security Properties

Summary

Screenshot Placeholders

Part 5: GitOps with ArgoCD

Introduction

Step 1: Install ArgoCD on the Hub

Step 2: Create AppProjects

Step 3: Add the Private GitOps Repo

Step 4: Register Spoke Clusters

Step 5: ApplicationSet — The Core of Hub-Spoke GitOps

Key ApplicationSet concepts:

Step 6: Deploy the ApplicationSets

Sync Status and Known False Positives

Troubleshooting

VPC Peering for Hub → Spoke Connectivity

Summary

Screenshot Placeholders

Part 4: EKS Multi-Cluster Setup

Part 4: EKS Multi-Cluster Setup — Six Clusters Across Two Regions

Introduction

Cluster Overview

EKS Terraform Module

Per-Environment Terragrunt Configs

Bootstrapping Private Clusters

kubectl Context Setup

EKS Add-ons

Fixing kubectl 401 Errors

AWS Load Balancer Controller

StorageClass for EBS PVCs

Node Security Group Tags

Verifying All Six Clusters

Summary

Screenshot Placeholders

Part 3: Infrastructure as Code

Part 3: Infrastructure as Code — Terraform Modules + Terragrunt

Introduction

Repository Layout

Dependency Ordering

Root Terragrunt Config

VPC Module

KMS Module

EKS Module (overview — full detail in Part 4)

VPC Peering (for ArgoCD hub-spoke)

Running the Stack

State Management Best Practices

Common Pitfalls

Summary

Part 2: AWS Foundation

Part 2: AWS Foundation — Organizations, SSO, and Account Setup

Introduction

Why Multi-Account?

Step 1: Create AWS Organizations

Step 2: Create Member Accounts

Step 3: Organize Accounts into OUs

Step 4: Apply Service Control Policies

Step 5: Configure AWS IAM Identity Center (SSO)

Step 6: GitHub OIDC — No Static AWS Keys in CI

Step 7: ECR Repository in the Management Account

Step 8: KMS Keys for Encryption at Rest

Step 9: Terragrunt Root Configuration

Understanding OrganizationAccountAccessRole

Step 10: Bootstrap S3 State Buckets

Summary

Part 1: Architecture Overview

Building a Production-Grade DevSecOps Pipeline on AWS: A Complete Guide

Part 1: Architecture Overview & What We Are Building

Introduction

High-Level Architecture

Understanding `OrganizationAccountAccessRole`