Forem: Glenn Gray

Stop Manually Updating Jira After Every PR Merge

Glenn Gray — Tue, 05 May 2026 12:31:48 +0000

Originally published on graycloudarch.com.

You just merged a PR. Now you open Jira, find the ticket, paste the PR link in a comment, transition the status to Done, and update the deployed field. Five minutes. Twenty times a week. That's 1,700 minutes per year per engineer — nearly 30 hours of pure mechanical overhead.

And that's assuming you remember. On one team I worked with, we audited the last three months of merged PRs. Thirty percent of tickets had no update after merge. No comment, no transition, no link. The ticket just sat in In Dev until someone noticed during sprint review.

The fix is two GitHub Actions workflows and a shared composite action. Here's exactly how to build it.

The Architecture

Two workflows, one shared extraction layer:

Workflow 1: Fires on PR creation — posts a Jira link comment to the PR so reviewers can navigate directly to the ticket.
Workflow 2: Fires on PR merge to main — posts a comment to the Jira ticket with the PR URL, commit SHA, and who merged it, then transitions the ticket to Done.

Both workflows need to find the Jira ticket ID. Instead of duplicating that logic, we extract it into a composite action.

Step 1: Composite Action for Ticket Extraction

Create .github/actions/extract-jira-ticket/action.yml.

The action checks four sources in priority order — easiest to fix first:

PR title (simplest for the developer to correct)
Commit messages
Branch name in standard format: PROJECT-123-description
Branch name with prefix: feat/PROJECT-123-description

name: Extract Jira Ticket
description: "Extracts Jira ticket from PR title, commits, or branch name"

inputs:
  jira-base-url:
    required: true
  jira-user-email:
    required: true
  jira-api-token:
    required: true

outputs:
  jira-key:
    value: ${{ steps.extract.outputs.jira_key }}
  found:
    value: ${{ steps.extract.outputs.found }}

runs:
  using: composite
  steps:
    - name: Extract ticket ID
      id: extract
      shell: bash
      run: |
        JIRA_KEY=""

        # Priority 1: PR title
        if [[ "${{ github.event.pull_request.title }}" =~ ([A-Z]+-[0-9]+) ]]; then
          JIRA_KEY="${BASH_REMATCH[1]}"
        fi

        # Priority 2: Branch name
        if [ -z "$JIRA_KEY" ]; then
          BRANCH="${{ github.head_ref }}"
          if [[ "$BRANCH" =~ ([A-Z]+-[0-9]+) ]]; then
            JIRA_KEY="${BASH_REMATCH[1]}"
          fi
        fi

        if [ -n "$JIRA_KEY" ]; then
          echo "jira_key=$JIRA_KEY" >> $GITHUB_OUTPUT
          echo "found=true" >> $GITHUB_OUTPUT
        else
          echo "found=false" >> $GITHUB_OUTPUT
        fi

The regex [A-Z]+-[0-9]+ matches any Jira ticket format: PROJ-1, IN-89, INFRA-1234. If you have tickets with lowercase project keys, adjust accordingly.

Step 2: PR Creation Workflow

Create .github/workflows/link-jira-on-pr.yml.

This fires when a PR is opened and posts a formatted comment with the Jira ticket link. If no ticket is found, it posts a warning so the author knows to add one — before review, not after.

name: Link Jira on PR
on:
  pull_request:
    types: [opened]

permissions:
  pull-requests: write

jobs:
  link-jira:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/extract-jira-ticket
        id: jira
        with:
          jira-base-url: ${{ secrets.JIRA_BASE_URL }}
          jira-user-email: ${{ secrets.JIRA_USER_EMAIL }}
          jira-api-token: ${{ secrets.JIRA_API_TOKEN }}

      - name: Post Jira link comment
        if: steps.jira.outputs.found == 'true'
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `📋 Jira: [${{ steps.jira.outputs.jira-key }}](${{ secrets.JIRA_BASE_URL }}/browse/${{ steps.jira.outputs.jira-key }})`
            })

      - name: Warn if no ticket found
        if: steps.jira.outputs.found == 'false'
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: '⚠️ No Jira ticket found. Add a ticket ID to the PR title (e.g., `PROJ-123: Your title`).'
            })

The warning step matters. It creates a feedback loop that trains the team to include ticket IDs upfront. Within a few weeks, the warning fires rarely.

Step 3: PR Merge Workflow

Create .github/workflows/update-jira-on-merge.yml.

This fires when a PR is closed against main. The if: github.event.pull_request.merged == true guard is important — the closed event also fires for PRs that are closed without merging.

name: Update Jira on Merge
on:
  pull_request:
    types: [closed]
    branches: [main]

jobs:
  update-jira:
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/extract-jira-ticket
        id: jira
        with:
          jira-base-url: ${{ secrets.JIRA_BASE_URL }}
          jira-user-email: ${{ secrets.JIRA_USER_EMAIL }}
          jira-api-token: ${{ secrets.JIRA_API_TOKEN }}

      - name: Post merge comment to Jira
        if: steps.jira.outputs.found == 'true'
        run: |
          HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
            -u "${{ secrets.JIRA_USER_EMAIL }}:${{ secrets.JIRA_API_TOKEN }}" \
            -H "Content-Type: application/json" \
            -X POST "${{ secrets.JIRA_BASE_URL }}/rest/api/2/issue/${{ steps.jira.outputs.jira-key }}/comment" \
            -d "{\"body\": \"PR merged: #${{ github.event.pull_request.number }} ${{ github.event.pull_request.html_url }}\nCommit: ${{ github.sha }}\nBy: ${{ github.event.pull_request.merged_by.login }}\"}")

          echo "Jira comment HTTP status: $HTTP_STATUS"
          [ "$HTTP_STATUS" -eq 201 ] && echo "✅ Comment posted" || echo "⚠️ Comment failed (non-critical)"

      - name: Transition ticket to Done
        if: steps.jira.outputs.found == 'true'
        run: |
          TRANSITION_ID="${{ secrets.JIRA_DONE_TRANSITION_ID }}"
          [ -z "$TRANSITION_ID" ] && echo "No transition ID configured, skipping" && exit 0

          curl -s -u "${{ secrets.JIRA_USER_EMAIL }}:${{ secrets.JIRA_API_TOKEN }}" \
            -H "Content-Type: application/json" \
            -X POST "${{ secrets.JIRA_BASE_URL }}/rest/api/2/issue/${{ steps.jira.outputs.jira-key }}/transitions" \
            -d "{\"transition\": {\"id\": \"$TRANSITION_ID\"}}"
          echo "✅ Transitioned to Done"

The comment step uses an HTTP status check rather than relying on curl's exit code. A failed comment doesn't fail the job — the PR already merged, and a missing notification shouldn't generate noise in CI. The transition step is fully optional: if JIRA_DONE_TRANSITION_ID isn't set, it skips silently. This lets you start with just comments and add transitions once you've verified the workflow runs cleanly.

Finding Your Transition IDs

Transition IDs are project-specific. There's no universal "Done" ID. Run this against any ticket in your project to find yours:

curl -u "$JIRA_EMAIL:$JIRA_API_TOKEN" \
  "$JIRA_BASE_URL/rest/api/2/issue/$TICKET_KEY/transitions" \
  | jq -r '.transitions[] | "ID: \(.id) | \(.name)"'

Example output:

ID: 91 | Done
ID: 31 | In Review
ID: 21 | In Progress

Set the Done ID as JIRA_DONE_TRANSITION_ID in your repository secrets.

A Note on Jira API Versions

Use the v2 API: /rest/api/2/. Some teams try v3 and get silent empty responses — {"errorMessages":[],"errors":{}} — that look exactly like auth failures. It's not auth. The v3 request body format changed, and error handling is poor. v2 is stable, well-documented, and works consistently.

Required Secrets

Add these to your GitHub repository secrets:

Secret	Value
`JIRA_BASE_URL`	`https://yourorg.atlassian.net`
`JIRA_USER_EMAIL`	The email address tied to your API token
`JIRA_API_TOKEN`	Generate at id.atlassian.com → Security → API tokens
`JIRA_DONE_TRANSITION_ID`	Optional — from the transitions API call above

For org-wide rollout, set these as organization secrets and restrict to relevant repositories.

Results

After rolling this out across a team of eight engineers:

Zero manual Jira updates after merge
Forgotten ticket updates dropped from 30% to 0%
Roughly 1,700 minutes per year recovered per engineer
Every merged PR has a complete audit trail: PR number, URL, commit SHA, who merged it

The composite action pattern also means when you need to extend this — adding a Slack notification on merge, posting to Confluence — you extend one file, not two.

If you're rolling this out and hitting edge cases — multi-project Jira setups, tickets that span repos, or teams that don't follow branch naming conventions — get in touch. The extraction logic and composite action pattern are straightforward to extend once the baseline is working.

What the first 24 hours of production CloudWatch data told us

Glenn Gray — Mon, 04 May 2026 18:43:32 +0000

Originally published on graycloudarch.com.

The morning after go-live, the first thing I looked at was CPU. One of the two delivery services was sitting at 99.8% average utilization across 9 tasks. P50 latency: 1,010ms.

We'd launched deliberately without autoscaling. The plan was to observe real traffic patterns before configuring a scaling policy — you can't tune a policy you haven't seen the workload demand yet. What we didn't know was that the workload would reveal something about the task itself before we'd had a chance to watch it for a week.

Thirty-six hours after go-live, we'd shipped right-sizing changes, a working autoscaling configuration, and a new observability source for ALB-layer signals. All of it came directly from what the first day of production data said. Here's how we read it.

What 99.8% CPU means at 0.5 vCPU

The service was allocated 512 ECS CPU units per task — half a vCPU. CloudWatch was telling us the tasks were spending essentially all of their scheduled CPU time working.

The first instinct in this situation is to add tasks. Scale out horizontally. But adding more 0.5 vCPU containers when each one is already saturated doesn't change the constraint. In ECS, the scheduler distributes tasks across hosts, but the per-task CPU ceiling is set in the task definition. More tasks at ceiling is not materially different from fewer tasks at ceiling — you're distributing the same undersized unit more widely.

The signal wasn't about count. It was about the unit itself.

At 99.8% utilization, any burst in per-request processing time — a downstream API call that's slow, a cache miss, a spike in concurrent requests — queues. The task has no headroom to absorb it. That's where the 1,010ms p50 comes from: not that individual requests are slow, but that tasks are scheduled tightly enough that requests wait before they even start processing.

Right-sizing the task before configuring the autoscaler

We doubled the CPU allocation: 512 → 1,024 units. The rationale is mechanical once you see it: you can't configure a useful CPU-based autoscaling policy on a task that's already running at ceiling. If 100% CPU is the baseline, the autoscaler has nothing to respond to — it would scale out immediately on creation and never scale in.

Target tracking at 70% CPU requires headroom. A 1 vCPU task running the same workload that previously pinned a 0.5 vCPU task will land around 50% utilization — below the target, room to absorb variance before triggering a scale-out, and enough signal for scale-in to be meaningful rather than noise.

The second service had a different profile: 12 tasks, 1 vCPU each, hitting 92% at peak. Not saturated the same way, but thin on headroom. We went to 2 vCPU there.

Two other services in the platform were running the opposite problem — allocated more memory than they'd ever used. Those went the other direction: overprovisioned memory cut back based on observed peaks. The same 24-hour data window showed both problems at once.

Sequencing matters: right-size the task before you configure the autoscaler. Otherwise you're teaching a scaling policy to respond to a signal that's already maxed out, and the first thing it does is scale out to a floor that's still running on undersized tasks.

Why we chose CPU tracking instead of request count

The obvious autoscaling metric for an HTTP service is ALBRequestCountPerTarget. The ALB knows the request rate per target group; scaling on that metric tracks load linearly and is highly predictable.

We couldn't use it.

The platform uses a cross-account Lambda to register ECS tasks with ALB target groups at boot. Because of how the registration bridge works, the ECS service resource is provisioned with target_group_arn = null — the target group lives in a different account, and the service module doesn't know its ARN. ALBRequestCountPerTarget requires the target group ARN to be known to the Application Auto Scaling policy. Without it, there's no way to wire the metric across accounts without building additional dependency plumbing.

CPU target tracking at 70% was the correct second choice. For a CPU-bound workload — which 99.8% utilization confirms this is — CPU is a meaningful proxy for load. The metric was there, it was clean, and the task was now sized to make it useful.

One thing worth noting: the cross-account registration bridge was the right architectural decision for the problem it solved. But it created a constraint three layers away in a scaling configuration we hadn't designed yet. Architecture decisions compound downstream. The fix here was straightforward; I've seen the same pattern take longer to untangle when the constraint wasn't recognized.

The observability gap app logs can't fill

Application logs were already flowing to BetterStack from both services. We had route-level latency, HTTP status codes, request counts, error breakdowns — everything that happens inside a container.

What the logs couldn't tell us was what happens above them. The ALB generates its own error signals: HTTPCode_ELB_5XX_Count for errors the load balancer generates before a request reaches a container, RejectedConnectionCount for connections refused at the ALB layer when backend capacity is exhausted, ActiveConnectionCount as a proxy for in-flight load per target group. None of this appears in application logs. If the ALB had been dropping connections during the 99.8% CPU period, we would have had no signal in our observability platform.

CloudWatch had the data. The gap was getting it into the same place as everything else.

A 60-second Lambda in the infrastructure account — where the ALB lives — calls GetMetricData and ships structured JSON to BetterStack. One EventBridge rule, no ECS changes, effectively zero cost (one CloudWatch API call per minute against Lambda's free tier). The metrics land alongside the application data and show the ALB layer that the app logs are blind to.

The design decision here was Lambda over an ECS sidecar. A sidecar would have run per-service, per-task, 24 hours a day, and required task definition changes across the platform. A single Lambda running once per minute in the account that owns the ALB costs nothing and touches no ECS configuration.

Autoscaling parameters worth explaining

For the higher-load service: min=9, max=20, CPU target=70%, scale-out cooldown=60s, scale-in cooldown=300s.

Setting min_capacity to 9 — the current running task count — was deliberate. We'd just established that 9 tasks was a functional floor for this workload at current traffic levels. An autoscaler configured with min=2 or min=4 would have attempted to scale in on the first quiet period, bringing the service back to a state we knew was already under-provisioned. Anchoring the floor to the observed stable-state count prevents that while we accumulate enough autoscaling history to set a meaningful long-term floor.

The asymmetric cooldowns — 60 seconds for scale-out, 5 minutes for scale-in — reflect the cost asymmetry of being wrong in each direction. Scaling out too slowly during a load spike means requests queue. Scaling in too aggressively during a brief quiet period means tasks are killed and restarted unnecessarily. The 5-minute scale-in cooldown is conservative; we'll revisit it once we have a week of data showing where the service naturally stabilizes.

What 24 hours of data drove

We launched expecting to spend the first week observing. What the data delivered instead was a complete picture of three distinct problems: a task sizing issue that was causing queuing, a scaling policy that needed the right foundation before it could be configured, and an observability gap for a class of signals that app logs fundamentally can't surface.

All three were solved from the same 24-hour data window. The pre-launch load testing hadn't revealed any of them — synthetic traffic and production ad-bidding traffic have different CPU profiles, and you don't know which until the real thing runs.

The thing I'd change if running this again: put a structured post-launch data review into the go-live plan, not the next morning's to-do list. Not a formal incident review — a deliberate hour with CloudWatch after the first day's traffic has run through. The data is there. The question is whether you've planned to look at it.

If you're planning a production go-live and want a structured approach to post-launch data review and stabilization — or you're staring at a service running at ceiling with no autoscaling — get in touch. This is the kind of platform work I do regularly, and the pattern here applies well beyond ad delivery.

Stop Managing EKS Add-ons by Hand

Glenn Gray — Sun, 05 Apr 2026 16:53:40 +0000

Originally published on graycloudarch.com.

I was preparing to upgrade a production EKS cluster to version 1.32 when I discovered a problem.

Four of our core cluster components—VPC CNI, CoreDNS, kube-proxy, and Metrics Server—were all running versions incompatible with EKS 1.32. I needed to update them before upgrading.

And I had no easy way to do it.

VPC CNI, CoreDNS, and kube-proxy had been installed automatically when the cluster was created, running in "self-managed" mode. Metrics Server was installed with kubectl apply -f metrics-server.yaml from some GitHub release page, months ago, by someone who is no longer on the team.

No version pinning. No history of what changed or when. No way to test the upgrade before applying it to production.

That's when I decided to stop managing EKS add-ons by hand.

The Problem with Self-Managed Add-ons

There are two categories of EKS add-ons, and most teams don't think about the distinction until they're stuck.

Self-managed: You're responsible for installation, updates, and compatibility. AWS won't help you troubleshoot them. When EKS releases a new version, you need to manually verify your add-ons still work, find compatible versions, and update them yourself.

EKS-managed: AWS handles the lifecycle. Compatible versions are tested and published for each EKS release. AWS Support can troubleshoot them. Security patches are available without you tracking CVEs.

If you created an EKS cluster without explicitly enabling managed add-ons, VPC CNI, CoreDNS, and kube-proxy are running in self-managed mode right now.

The fix is straightforward—migrate them to EKS-managed. But if you're also running kubectl-installed tools like Metrics Server, you have a second problem: those aren't managed by anything at all.

The Solution: One Terraform Module for All Six Add-ons

I built a single eks-addons Terraform module that manages everything:

EKS-managed (4):

VPC CNI — pod networking
EBS CSI Driver — persistent volumes (added this one while I was at it)
CoreDNS — DNS resolution
kube-proxy — network proxy

Helm-managed (2):

Metrics Server — resource metrics for kubectl top and HPA
Reloader — auto-restart pods when ConfigMaps or Secrets change

Why one module instead of six separate ones? All of these share the same dependency: the EKS cluster. Consolidating them means one terragrunt apply deploys everything, one terraform plan shows drift across all add-ons, and one PR updates any version.

The core Terraform for an EKS-managed add-on is minimal:

resource "aws_eks_addon" "vpc_cni" {
  count = var.enable_vpc_cni ? 1 : 0

  cluster_name                = var.cluster_name
  addon_name                  = "vpc-cni"
  addon_version               = var.vpc_cni_version
  resolve_conflicts_on_create = "OVERWRITE"
  resolve_conflicts_on_update = "OVERWRITE"
  preserve                    = true
}

Two things worth explaining:

resolve_conflicts = "OVERWRITE" tells Terraform it's the source of truth. Any manual changes in the cluster get overwritten on the next apply. This is what you want.

preserve = true means if you remove the resource from Terraform, the add-on stays in the cluster. Safety net during refactoring—you won't accidentally delete a running add-on.

EBS CSI Driver Needs an IAM Role

The EBS CSI Driver is the one add-on that requires extra work: it needs IAM permissions to create and attach EBS volumes. The right way to handle this is IRSA (IAM Roles for Service Accounts).

resource "aws_iam_role" "ebs_csi" {
  count = var.enable_ebs_csi ? 1 : 0
  name  = "${var.cluster_name}-ebs-csi-driver"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Federated = var.oidc_provider_arn }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${var.oidc_provider}:sub" = "system:serviceaccount:kube-system:ebs-csi-controller-sa"
          "${var.oidc_provider}:aud" = "sts.amazonaws.com"
        }
      }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "ebs_csi" {
  count      = var.enable_ebs_csi ? 1 : 0
  role       = aws_iam_role.ebs_csi[0].name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
}

No credentials in pods, automatic rotation, and a clean audit trail in CloudTrail. IRSA is the correct pattern for any AWS service that needs to call AWS APIs from inside Kubernetes.

Migrating Metrics Server from kubectl to Helm

This is the one step that requires manual cleanup before Terraform can take over.

The existing kubectl-installed Metrics Server needs to go first:

kubectl delete deployment metrics-server -n kube-system
kubectl delete service metrics-server -n kube-system
kubectl delete apiservice v1beta1.metrics.k8s.io

Then Terraform installs the Helm-managed version:

resource "helm_release" "metrics_server" {
  count      = var.enable_metrics_server ? 1 : 0
  name       = "metrics-server"
  repository = "https://kubernetes-sigs.github.io/metrics-server/"
  chart      = "metrics-server"
  version    = var.metrics_server_chart_version
  namespace  = "kube-system"

  values = [yamlencode({
    replicas = 2
    args = [
      "--kubelet-preferred-address-types=InternalIP",
      "--kubelet-insecure-tls"
    ]
    podDisruptionBudget = {
      enabled      = true
      minAvailable = 1
    }
  })]
}

Expected downtime: 2-3 minutes. Only kubectl top is unavailable during the transition. Running applications are not affected.

Deploying It

One thing that bit me: CI/CD doesn't pick up module changes automatically.

Our GitHub Actions workflow detects changes by looking for modified terragrunt.hcl files. When I changed files in common/modules/eks-addons/, the workflow triggered but found no stacks to deploy (no terragrunt.hcl changed), so nothing ran.

Module changes require a manual deploy:

cd workloads-nonprod/us-east-1/cluster-name/eks-addons
terragrunt init
terragrunt plan   # Review: should show ~10 resources to add
terragrunt apply

After apply, verify everything is healthy:

# Check EKS-managed add-on status
for addon in vpc-cni aws-ebs-csi-driver coredns kube-proxy; do
  aws eks describe-addon --cluster-name <cluster> --addon-name $addon \
    --query 'addon.[addonName,status]' --output text
done
# All should show: ACTIVE

# Verify Metrics Server
kubectl top nodes

What Changed

Before: four add-ons running in self-managed mode, one installed by kubectl, no version history, no drift detection.

After:

All six add-ons defined in code with pinned versions
terraform plan shows immediately if anything drifts from the declared state
Rollback is git revert + terragrunt apply
EKS cluster upgrade checklist is now: update four version strings in the Terragrunt config, open a PR, done

The cluster upgrade I was dreading took about 30 minutes instead of a day of manual compatibility checking.

Running into EKS add-on management problems? Reach out—this is the kind of operational work I do for platform teams.

Zero-Downtime AWS Transit Gateway Hub-Spoke Migration

Glenn Gray — Sat, 04 Apr 2026 23:36:57 +0000

Originally published on graycloudarch.com.

The request came from the security team: they needed network-level access from the nonprod account to the dev account so a vulnerability scanner could reach internal services. Simple enough on the surface. In practice, it exposed a gap we'd been living with for months — and forced us to fix the network architecture we'd been deferring.

We had three standalone Transit Gateways: one in each workload account, dev, nonprod, and prod. Completely isolated from each other. No cross-account connectivity at all. The security scanner couldn't reach its targets, and adding more point-to-point peering connections to fix it would have made everything worse.

But the TGW isolation was only part of the problem. We also had no inspection of traffic crossing our network boundary. Egress from workload pods went straight to the internet with no filtering. Ingress came through per-account load balancers with no centralized enforcement point. As the platform scaled toward additional workload accounts, this pattern was going to get expensive and hard to reason about.

So we didn't just fix the TGW. We rebuilt the network foundation: a centralized Inspection VPC with a Network Firewall inline, a single hub Transit Gateway shared across all accounts, and centralized security tooling (GuardDuty, CloudTrail, Security Hub) aggregated in a dedicated Security account. Two maintenance windows, a few weeks of module work, and the platform went from fragmented per-account networking to a coherent hub-spoke design with full traffic inspection.

The Architecture We Were Replacing

Before the migration, each workload account was self-contained. It had its own TGW, its own internet gateway, its own NAT gateways. Security tooling ran independently in each account with no aggregation. The management account had no single-pane visibility into what was happening across the environment.

The cost of running this way was about $150/month in TGW charges plus duplicated NAT gateway charges in each account. Every new workload account would multiply this cost again and add another independent security configuration.

The Target: Inspection VPC + Hub Transit Gateway

The target was AWS Security Reference Architecture Pattern B: an Inspection VPC that sits between the internet and all workload VPCs. All internet traffic — ingress and egress — flows through this VPC and through a Network Firewall before reaching any workload account.

Egress path: workload pod → TGW → Inspection VPC TGW subnets → Network Firewall → NAT Gateway → IGW → internet.

Ingress path: internet → IGW → centralized ALB (public subnet) → Network Firewall → TGW → workload VPC → pod.

Nothing crosses the network boundary without passing through the firewall. Workload accounts carry no internet-facing infrastructure at all — no IGW, no NAT gateways, no public load balancers.

Phase 1: Module Changes

All Terraform work happened before scheduling any maintenance. The goal was to reach a state where the migration itself was just running pre-staged plan files in a specific sequence.

Transit Gateway: add a conditional create flag

The existing network module always created a TGW. We needed spoke accounts to declare the same module without spinning up their own gateway:

variable "create_transit_gateway" {
  description = "Whether to create a Transit Gateway (false for hub-spoke spokes)"
  type        = bool
  default     = true
}

resource "aws_ec2_transit_gateway" "this" {
  count       = var.create_transit_gateway ? 1 : 0
  description = var.tgw_description
}

output "transit_gateway_id" {
  value = var.create_transit_gateway ? aws_ec2_transit_gateway.this[0].id : null
}

default = true means existing configurations need no changes. The flag only flips to false after the spoke attachment is confirmed working.

New module: vpc-attachment

The vpc-attachment module handles the spoke side of the hub relationship: create the TGW attachment, associate it to the hub's route table, and add routes to every private route table in the spoke VPC pointing at the hub TGW.

resource "aws_ec2_transit_gateway_vpc_attachment" "this" {
  transit_gateway_id = var.transit_gateway_id
  vpc_id             = var.vpc_id
  subnet_ids         = var.subnet_ids

  tags = merge(var.tags, {
    Name = "${var.name}-hub-attachment"
  })
}

resource "aws_ec2_transit_gateway_route_table_association" "this" {
  transit_gateway_attachment_id  = aws_ec2_transit_gateway_vpc_attachment.this.id
  transit_gateway_route_table_id = var.transit_gateway_route_table_id
}

resource "aws_route" "to_hub_tgw" {
  for_each               = toset(var.vpc_route_table_ids)
  route_table_id         = each.value
  destination_cidr_block = "10.0.0.0/8"
  transit_gateway_id     = var.transit_gateway_id
}

The 10.0.0.0/8 supernet covers all workload and Inspection VPC CIDRs without maintaining per-prefix route entries. It also covers the Inspection VPC CIDR (10.100.0.0/20) — that's how return traffic from the centralized ALB finds its way back to pods in workload VPCs.

The Terragrunt config for a spoke account reads VPC details from the existing network dependency and hardcodes the hub TGW identifiers:

dependency "network" {
  config_path = "../network"
  mock_outputs = {
    vpc_id                  = "vpc-mockid"
    private_subnet_ids      = ["subnet-mock1"]
    private_route_table_ids = ["rtb-mock1"]
  }
}

inputs = {
  transit_gateway_id             = "tgw-xxxxx"   # hub TGW, documented in runbook
  transit_gateway_route_table_id = "tgw-rtb-xxxxx"
}

We hardcoded the hub TGW and route table IDs rather than using cross-account data sources. The alternative — reading TGW details from the Infrastructure account at plan time — requires cross-account state access and adds complexity that isn't worth it for values that change maybe once in the platform's lifetime.

Hub route tables: workload isolation by default

A key design decision: workload accounts should not route to each other directly. Dev should not reach nonprod; nonprod should not reach prod. The hub TGW enforces this through route table structure:

default-association-rt: all workload attachments associate here. The only route is 0.0.0.0/0 → inspection attachment. Workloads can reach the internet via the Inspection VPC, but cannot reach other workload VPCs.
default-propagation-rt: the inspection attachment propagates workload CIDRs here for return traffic routing.

Inter-account communication is opt-in: you add an explicit route table entry for a specific attachment pair. By default, the architecture prevents lateral movement across workload accounts.

Inspection VPC subnet layout

The Inspection VPC has three tiers with carefully constructed route tables that force traffic through the firewall in both directions:

The asymmetric route table design ensures the firewall sees every packet crossing the network boundary, regardless of direction. Traffic entering from the internet hits the firewall before reaching workloads. Traffic from workloads hits the firewall before reaching the internet.

Security baseline: convert to delegated admin model

GuardDuty and CloudTrail were running independently per account. We added enable_guardduty and enable_cloudtrail boolean variables to the security-baseline module so workload accounts could switch from standalone to member without touching the module invocation itself.

In the Security account, we deployed:

GuardDuty as delegated admin with organization-level auto-enrollment. EKS Protection and S3 Protection enabled. All findings from all accounts visible in a single dashboard.
CloudTrail organization trail writing to a cross-account S3 bucket. Log file validation and KMS encryption enabled. Per-account trails archived after the cutover — not deleted, in case historical log formats differed.
Security Hub with CIS AWS Foundations Benchmark and AWS Foundational Security Best Practices enabled across the full organization.

Phase 2: Two Maintenance Windows

Window 1: Deploy the hub (~45 minutes, low risk)

With no existing attachments and no workload traffic, deploying the hub infrastructure carried minimal risk. We applied the Infrastructure account TGW and Inspection VPC in a single window. The Network Firewall takes 5–10 minutes to reach READY state after creation — account for that in your timing.

At the end of this window: hub TGW running, Inspection VPC active, Network Firewall endpoints healthy in both AZs, centralized ALB deployed. Nothing attached yet. We documented the TGW ID and route table IDs in the runbook before scheduling window 2.

Window 2: Spoke cutover (~2 hours)

The key insight for keeping applications running: create the hub attachment before destroying the standalone TGW. While both exist simultaneously, traffic continues flowing through the standalone path. The actual cutover is updating routes to point at the hub — that's a single terragrunt apply, not the destruction of the old TGW.

T+0 — Accept RAM share. Infrastructure account shares the hub TGW via Resource Access Manager. Workload accounts accept the share invitation. Pure metadata operation; zero network impact.

T+15 — Deploy VPC attachments. Apply the vpc-attachment module in each workload account. At this point each spoke VPC has two routes for 10.0.0.0/8: the existing one pointing at the standalone TGW, and the new one pointing at the hub. With identical prefix lengths, traffic still flows through the standalone path. Rollback at this stage is terragrunt destroy on the attachment module — under five minutes.

T+30 — Verify routes and test cross-account connectivity. Confirm hub routes are present in every private route table:

aws ec2 describe-route-tables \
  --filters "Name=vpc-id,Values=vpc-xxxxx" \
  --query 'RouteTables[*].Routes[?DestinationCidrBlock==`10.0.0.0/8`]'

Then test actual cross-account traffic: connect from a dev instance to a service in the nonprod VPC. The hub TGW and Inspection VPC should route it correctly. This also validates that the firewall rule groups are permitting expected traffic — catch any rule issues here, before cutting over production.

T+45 — Migrate security tooling. Apply the updated security-baseline to each workload account. GuardDuty converts from standalone admin to member; findings flow to the Security account delegated admin. CloudTrail local trail disabled; organization trail confirmed logging events from the account. Zero network impact.

# Verify GuardDuty membership
aws guardduty get-administrator-account --detector-id <id>
# Returns the Security account as administrator

# Verify organization trail is capturing events
# Make an API call, wait ~15 minutes, check the Security account's S3 bucket
aws s3 ls s3://<org-trail-bucket>/AWSLogs/<account-id>/

T+60 — Set create_transit_gateway = false in each spoke. This is the cutover. Run terraform plan first and confirm it shows only the TGW and its attached resources being destroyed — nothing else. Apply dev first, watch the destruction complete, confirm application traffic is flowing through the hub. Then apply nonprod. About 3 minutes per account.

T+90 — Health checks and close. Spot-check API endpoints, database connectivity, anything that traverses the network. Confirm egress traffic is hitting the firewall logs in the Infrastructure account. The maintenance window closed at the 90-minute mark; actual work was done by T+75. We kept the window open for the last 15 minutes as a buffer.

The parallel attachment approach ensured there was never a moment where a workload account had no routing path. Even if the hub TGW had been misconfigured, traffic would have continued flowing through the standalone gateway until we chose to destroy it.

What We Ended Up With

One TGW in the Infrastructure account with three spoke attachments. Route tables that allow workload→internet traffic while preventing workload→workload lateral movement by default.

One Inspection VPC with Network Firewall endpoints in two AZs. All egress inspected against stateful domain filter rules and stateless port rules. All ingress from the centralized ALB inspected. Firewall policy updates apply to all workload accounts simultaneously — no per-account changes needed.

One centralized ALB in the Infrastructure account, routing to EKS target groups in workload accounts via cross-account IAM role assumption. Workload accounts carry no public-facing load balancers.

One security console in the Security account. GuardDuty findings from all accounts in a single dashboard. CloudTrail logs from every account in one S3 bucket. Security Hub compliance posture for the full organization visible in one place.

Cost went from roughly $150–200/month (standalone TGWs, per-account NAT, independent security tooling) to approximately $50/month (single hub TGW plus attachment hours, shared NAT in the Inspection VPC, delegated security services). Cost savings validated against AWS Cost Explorer after 30 days.

The original security scanner request — cross-account access from nonprod to dev — was live the same day. The compliance team had a single GuardDuty and Security Hub dashboard the same week.

More importantly: adding a new workload account to this architecture now takes about an hour. Create the VPC, deploy the vpc-attachment module pointing at the documented hub TGW ID, invite the new account as a GuardDuty and Security Hub member, apply the security-baseline with enable_guardduty = false. Every new account inherits the full inspection and security posture without any per-account configuration. That's the actual value of a hub-spoke design — not the one-time cost savings, but the fact that account seven is as well-secured and as easy to audit as account two.

Working through a multi-account network redesign, or building the inspection layer on top of an existing Transit Gateway setup? Get in touch — this is the kind of platform architecture I work on regularly.

DNS Validation: From 15 Steps to Zero

Glenn Gray — Sat, 04 Apr 2026 22:30:30 +0000

Originally published on graycloudarch.com.

You know what's the worst part of launching a new site?

SSL certificate validation.

Not creating the cert—that's one click in AWS ACM. It's the validation dance:

AWS gives you a CNAME record: _abc123extremely-long-string-here.graycloudarch.com
The value is equally ridiculous: _xyz789another-massive-string.acm-validations.aws.
You copy it (pray you don't miss a character)
Switch to Cloudflare (or Route 53, or wherever)
Paste it in
Wait 5-10 minutes
Refresh AWS console
Still pending...
Refresh again
Finally validated!

Now do it again for www.graycloudarch.com.

And then repeat the whole thing for your second domain.

This is "DNS hell."

There's a Better Way

Terraform can read AWS validation records and create them in Cloudflare automatically.

Zero copy-paste. Zero browser tab switching. Zero waiting and refreshing.

Here's the whole thing:

# Request certificate
resource "aws_acm_certificate" "site" {
  domain_name       = "graycloudarch.com"
  validation_method = "DNS"
  subject_alternative_names = ["www.graycloudarch.com"]
}

# Create validation records in Cloudflare
resource "cloudflare_record" "cert_validation" {
  for_each = {
    for dvo in aws_acm_certificate.site.domain_validation_options :
    dvo.domain_name => {
      name   = dvo.resource_record_name
      value  = dvo.resource_record_value
      type   = dvo.resource_record_type
    }
  }

  zone_id = data.cloudflare_zone.site.id
  name    = each.value.name
  value   = each.value.value
  type    = each.value.type
  proxied = false  # Critical - ACM validation breaks with proxy
}

# Wait for validation
resource "aws_acm_certificate_validation" "site" {
  certificate_arn = aws_acm_certificate.site.arn
  validation_record_fqdns = [
    for record in cloudflare_record.cert_validation : record.hostname
  ]
}

Run terraform apply. Go make coffee. Come back to a validated certificate.

The Magic: for_each

The key is this part:

for_each = {
  for dvo in aws_acm_certificate.site.domain_validation_options :
  dvo.domain_name => { name = dvo.resource_record_name, ... }
}

AWS generates validation records dynamically (one for apex domain, one for www). Terraform reads them, loops over them, and creates each one in Cloudflare.

You never see the records. You never copy anything. It just works.

What I Screwed Up

First time I ran this, ACM validation timed out after 30 minutes.

The problem:

proxied = true  # Wrong!

Cloudflare's proxy rewrites DNS responses. ACM's validation servers hit Cloudflare's IP instead of seeing your validation record.

The fix:

proxied = false  # Correct

DNS-only mode. No proxy. ACM validation works.

Cost me 30 minutes of debugging. Now it's in code so I never hit it again.

Why This Matters

I'm running two brands: graycloudarch.com and cloudpatterns.io.

Manual approach: 15 steps per domain = 30 steps total. 30 minutes minimum. High chance of typos.

Terraform approach: One terraform apply. 5 minutes to write the code (once), 10 minutes for AWS to validate. Then copy-paste the pattern for the second domain.

When I launch my third brand (and I will), it'll take 5 minutes and one terraform apply.

That's the bet: upfront automation for long-term velocity.

The Part People Miss

Most Terraform tutorials stop at requesting the certificate. They don't show you the validation loop or the waiting resource.

Without aws_acm_certificate_validation, Terraform exits immediately after creating the cert. It's still "Pending Validation" in AWS. When you try to use it in CloudFront, it fails.

You'd have to run terraform apply again later, after manually checking that validation completed.

That's not automation—that's just documentation.

The waiting resource makes it truly hands-off.

Scaling It

Adding a second domain is 10 lines of code:

resource "aws_acm_certificate" "cloudpatterns" {
  domain_name       = "cloudpatterns.io"
  validation_method = "DNS"
  subject_alternative_names = ["www.cloudpatterns.io"]
}

resource "cloudflare_record" "cloudpatterns_validation" {
  for_each = { /* same pattern */ }
  # ...
}

resource "aws_acm_certificate_validation" "cloudpatterns" {
  # ...
}

Same pattern, different names. No clicking. No switching between consoles. No remembering which validation record goes where.

The Real Win

It's not the time savings (though 30 minutes per deployment adds up).

It's the mental overhead.

Manual DNS configuration requires focus. "Did I copy the whole string? Did I add the trailing dot? Is it DNS-only mode?"

Terraform requires running one command. That's it.

I get my focus back. I can write this blog post while Terraform validates certificates.

Want the full code? It's not open source (yet), but if you're building something similar and want to talk through it, reach out.

Or if you just want to tell me I'm overthinking this and should've clicked through Cloudflare like a normal person, that's cool too.

Building Multi-Account AWS Infrastructure with Terraform and ECP

Glenn Gray — Sat, 04 Apr 2026 22:30:25 +0000

Originally published on graycloudarch.com.

After years of building AWS infrastructure at scale, I've learned that multi-account strategy isn't just about security—it's about organizational clarity and cost management.

At a large podcast hosting platform, we implemented an Enterprise Control Plane (ECP) pattern using Terraform to manage 20+ AWS accounts. Here's what I learned:

The Problem with Single-Account AWS

Most companies start with one AWS account. Everything lives together: dev, staging, prod, data pipelines, security tools. It works... until it doesn't.

Problems emerge:

Blast radius: A misconfigured dev resource can affect production
IAM complexity: Permission boundaries become impossible to manage
Cost allocation: Finance can't track spending by team or project
Compliance: Auditors want logical separation between environments

The ECP Pattern

Enterprise Control Plane is an architectural pattern for managing multiple AWS accounts as a unified platform:

Organization Structure: AWS Organizations with OUs (Organizational Units) for different environments and teams
Centralized Networking: Transit Gateway connecting all accounts through hub-and-spoke model
Security Baseline: Service Control Policies (SCPs) enforcing guardrails at the organization level
Infrastructure as Code: Terraform/Terragrunt managing everything from a central repository

Key Design Decisions

Account Boundaries:

Production accounts: Isolated per application/team
Non-prod accounts: Shared dev/staging to reduce overhead
Platform accounts: Separate accounts for logging, monitoring, security tools
Data accounts: Isolated for compliance and access control

Network Architecture:

Hub account with Transit Gateway
VPC peering only where absolutely necessary
Private subnet defaults for everything
Centralized egress through NAT Gateway in hub

Security Model:

SCPs prevent account-level misconfigurations
IAM roles for cross-account access (no shared credentials)
CloudTrail logs aggregated to security account
GuardDuty and Security Hub in every account

Terraform Structure

We use Terragrunt to manage configurations across accounts:

ecp-ou-structure/     # Organization and account management
ecp-network/          # Transit Gateway, VPCs, networking
ecp-security/         # Security baseline, SCPs, IAM
tf-live-aws-*/        # Application-specific infrastructure

Lessons Learned

Start with security: SCPs first, then networking, then workloads
Automate account creation: Manual account provisioning doesn't scale
Document the why: Every architectural decision needs context
Plan for day 2: Operations matter more than initial setup

Results

After implementing ECP:

Reduced security incident blast radius by 90%
Finance can now track costs by team and project
New environments deploy in hours, not days
Passed SOC2 audit with zero infrastructure findings

Multi-account AWS isn't just best practice—it's how you scale infrastructure beyond the startup phase.

Stop Manually Updating Jira After Every PR Merge

Glenn Gray — Wed, 25 Mar 2026 07:10:48 +0000

This post was originally published on graycloudarch.com.

You just merged a PR. Now you open Jira, find the ticket, paste the
PR link in a comment, transition the status to Done, and update the
deployed field. Five minutes. Twenty times a week. That's 1,700 minutes
per year per engineer --- nearly 30 hours of pure mechanical overhead.

And that's assuming you remember. On one team I worked with, we
audited the last three months of merged PRs. Thirty percent of tickets
had no update after merge. No comment, no transition, no link. The
ticket just sat in In Dev until someone noticed during sprint
review.

The fix is two GitHub Actions workflows and a shared composite
action. Here's exactly how to build it.

The Architecture

Two workflows, one shared extraction layer:

Workflow 1: Fires on PR creation --- posts a Jira link comment to the PR so reviewers can navigate directly to the ticket.
Workflow 2: Fires on PR merge to main --- posts a comment to the Jira ticket with the PR URL, commit SHA, and who merged it, then transitions the ticket to Done.

Both workflows need to find the Jira ticket ID. Instead of
duplicating that logic, we extract it into a composite action.

Step 1: Composite Action for Ticket Extraction

Create
.github/actions/extract-jira-ticket/action.yml.

The action checks four sources in priority order --- easiest to fix
first:

PR title (simplest for the developer to correct)
Commit messages
Branch name in standard format: PROJECT-123-description
Branch name with prefix: feat/PROJECT-123-description

::: {#cb1 .sourceCode}

name: Extract Jira Ticket
description: Extracts Jira ticket from PR title, commits, or branch name

inputs:
  jira-base-url:
    required: true
  jira-user-email:
    required: true
  jira-api-token:
    required: true

outputs:
  jira-key:
    value: ${{ steps.extract.outputs.jira_key }}
  found:
    value: ${{ steps.extract.outputs.found }}

runs:
  using: composite
  steps:
    - name: Extract ticket ID
      id: extract
      shell: bash
      run: |
        JIRA_KEY=""

        # Priority 1: PR title
        if [[ "${{ github.event.pull_request.title }}" =~ ([A-Z]+-[0-9]+) ]]; then
          JIRA_KEY="${BASH_REMATCH[1]}"
        fi

        # Priority 2: Branch name
        if [ -z "$JIRA_KEY" ]; then
          BRANCH="${{ github.head_ref }}"
          if [[ "$BRANCH" =~ ([A-Z]+-[0-9]+) ]]; then
            JIRA_KEY="${BASH_REMATCH[1]}"
          fi
        fi

        if [ -n "$JIRA_KEY" ]; then
          echo "jira_key=$JIRA_KEY" >> $GITHUB_OUTPUT
          echo "found=true" >> $GITHUB_OUTPUT
        else
          echo "found=false" >> $GITHUB_OUTPUT
        fi

:::

The regex [A-Z]+-[0-9]+ matches any Jira ticket format:
PROJ-1, IN-89, INFRA-1234. If you
have tickets with lowercase project keys, adjust accordingly.

Step 2: PR Creation Workflow

Create .github/workflows/link-jira-on-pr.yml.

This fires when a PR is opened and posts a formatted comment with the
Jira ticket link. If no ticket is found, it posts a warning so the
author knows to add one --- before review, not after.

::: {#cb2 .sourceCode}

name: Link Jira on PR
on:
  pull_request:
    types: [opened]

permissions:
  pull-requests: write

jobs:
  link-jira:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/extract-jira-ticket
        id: jira
        with:
          jira-base-url: ${{ secrets.JIRA_BASE_URL }}
          jira-user-email: ${{ secrets.JIRA_USER_EMAIL }}
          jira-api-token: ${{ secrets.JIRA_API_TOKEN }}

      - name: Post Jira link comment
        if: steps.jira.outputs.found == 'true'
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `📋 Jira: [${{ steps.jira.outputs.jira-key }}](${{ secrets.JIRA_BASE_URL }}/browse/${{ steps.jira.outputs.jira-key }})`
            })

      - name: Warn if no ticket found
        if: steps.jira.outputs.found == 'false'
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: '⚠️ No Jira ticket found. Add a ticket ID to the PR title (e.g., `PROJ-123: Your title`).'
            })

:::

The warning step matters. It creates a feedback loop that trains the
team to include ticket IDs upfront. Within a few weeks, the warning
fires rarely.

Step 3: PR Merge Workflow

Create .github/workflows/update-jira-on-merge.yml.

This fires when a PR is closed against main. The
if: github.event.pull_request.merged == true guard is
important --- the closed event also fires for PRs that are
closed without merging.

::: {#cb3 .sourceCode}

name: Update Jira on Merge
on:
  pull_request:
    types: [closed]
    branches: [main]

jobs:
  update-jira:
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/extract-jira-ticket
        id: jira
        with:
          jira-base-url: ${{ secrets.JIRA_BASE_URL }}
          jira-user-email: ${{ secrets.JIRA_USER_EMAIL }}
          jira-api-token: ${{ secrets.JIRA_API_TOKEN }}

      - name: Post merge comment to Jira
        if: steps.jira.outputs.found == 'true'
        run: |
          HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}"
            -u "${{ secrets.JIRA_USER_EMAIL }}:${{ secrets.JIRA_API_TOKEN }}"
            -H "Content-Type: application/json"
            -X POST "${{ secrets.JIRA_BASE_URL }}/rest/api/2/issue/${{ steps.jira.outputs.jira-key }}/comment"
            -d "{\"body\": \"PR merged: #${{ github.event.pull_request.number }} ${{ github.event.pull_request.html_url }}\nCommit: ${{ github.sha }}\nBy: ${{ github.event.pull_request.merged_by.login }}\"}")

          echo "Jira comment HTTP status: $HTTP_STATUS"
          [ "$HTTP_STATUS" -eq 201 ] && echo "✅ Comment posted" || echo "⚠️ Comment failed (non-critical)"

      - name: Transition ticket to Done
        if: steps.jira.outputs.found == 'true'
        run: |
          TRANSITION_ID="${{ secrets.JIRA_DONE_TRANSITION_ID }}"
          [ -z "$TRANSITION_ID" ] && echo "No transition ID configured, skipping" && exit 0

          curl -s -u "${{ secrets.JIRA_USER_EMAIL }}:${{ secrets.JIRA_API_TOKEN }}"
            -H "Content-Type: application/json"
            -X POST "${{ secrets.JIRA_BASE_URL }}/rest/api/2/issue/${{ steps.jira.outputs.jira-key }}/transitions"
            -d "{\"transition\": {\"id\": \"$TRANSITION_ID\"}}"
          echo "✅ Transitioned to Done"

:::

The comment step uses an HTTP status check rather than relying on
curl's exit code. A failed comment doesn't fail the job --- the PR already
merged, and a missing notification shouldn't generate noise in CI. The
transition step is fully optional: if
JIRA_DONE_TRANSITION_ID isn't set, it skips silently. This
lets you start with just comments and add transitions once you've
verified the workflow runs cleanly.

Finding Your Transition IDs

Transition IDs are project-specific. There's no universal "Done" ID.
Run this against any ticket in your project to find yours:

::: {#cb4 .sourceCode}

curl -u "$JIRA_EMAIL:$JIRA_API_TOKEN"
  "$JIRA_BASE_URL/rest/api/2/issue/$TICKET_KEY/transitions"
  | jq -r '.transitions[] | "ID: \(.id) | \(.name)"'

:::

Example output:

ID: 91 | Done
ID: 31 | In Review
ID: 21 | In Progress

Set the Done ID as JIRA_DONE_TRANSITION_ID in your
repository secrets.

A Note on Jira API Versions

Use the v2 API: /rest/api/2/. Some teams try v3 and get
silent empty responses --- {"errorMessages":[],"errors":{}} ---
that look exactly like auth failures. It's not auth. The v3 request body
format changed, and error handling is poor. v2 is stable,
well-documented, and works consistently.

Required Secrets

Add these to your GitHub repository secrets:

Secret Value

JIRA_BASE_URL https://yourorg.atlassian.net
JIRA_USER_EMAIL The email address tied to your API token
JIRA_API_TOKEN Generate at id.atlassian.com → Security → API tokens
JIRA_DONE_TRANSITION_ID Optional --- from the transitions API call above

For org-wide rollout, set these as organization secrets and restrict
to relevant repositories.

Results

After rolling this out across a team of eight engineers:

Zero manual Jira updates after merge
Forgotten ticket updates dropped from 30% to 0%
Roughly 1,700 minutes per year recovered per engineer
Every merged PR has a complete audit trail: PR number, URL, commit SHA, who merged it

The composite action pattern also means when you need to extend this
--- adding a Slack notification on merge, posting to Confluence --- you
extend one file, not two.

If you're building out automation like this across your engineering
platform and want a second opinion on the design, I'm available for advisory engagements.

How I Manage Claude Code Context Across 20+ Repositories

Glenn Gray — Fri, 20 Mar 2026 06:57:13 +0000

This post was originally published on graycloudarch.com.

Three months ago I was re-explaining my Terragrunt state backend to
Claude for the third time in a week. Different session, same repo, same
repo I'd worked in the session before. Claude had no idea I was even in
the same project.

I run Claude Code daily across a 6-account AWS platform monorepo, a
personal consulting site, homelab infrastructure, and a handful of side
projects. Every session started with the same five minutes of "here's
the project, here are the conventions, here's the Jira workflow" --- and
still ended with Claude suggesting patterns that didn't fit the
environment, because I'd inevitably forgotten to mention something.

After three months of broken symlinks and abandoned experiments, I
landed on a three-tier context hierarchy that loads the right context
automatically depending on which directory I'm working in --- and I manage
all of it from a single dotfiles repo.

The Problem with Single-File Context

Claude Code loads CLAUDE.md from the current directory
(and parent directories, walking up to
~/.claude/CLAUDE.md). Most teams start with one file and
put everything in it.

That breaks down quickly:

Global preferences get mixed with project specifics. Your "use snake_case for variable names" preference shouldn't live next to your Terraform state bucket configuration.
Credentials and account IDs end up in files you accidentally commit. Put AWS account IDs in a shared CLAUDE.md, and someone will eventually push it.
You can't share patterns across repos without duplication. Every new repo gets a fresh copy of the same conventions, and updates never propagate.
Multi-employer context creates conflicts. Your consulting client's Jira workflow shouldn't contaminate your personal project sessions.

My first attempt at fixing this was a shared scripts directory. The
three-tier hierarchy came later, after I figured out what was actually
wrong with the simpler approach.

My First Attempt: A Shared Scripts Directory

Before landing on the three-tier system, I built something more
obvious: a ~/shared-claude-infra/ directory containing a
setup-project.sh script that initialized
.claude/ context for each new repo.

The script created the directory structure and symlinked a
rules/shared/ folder back to the shared repo:

mkdir -p "$PROJECT_DIR/.claude/rules"
ln -s ~/shared-claude-infra/rules "$PROJECT_DIR/.claude/rules/shared"

This worked for the first two repos I configured. Then the problems
compounded:

Manual per-project setup. Every new repo required running the script. Miss one, and that repo has no shared context.
Two repos to maintain. The shared infrastructure lived in its own git repo, separate from dotfiles. Two places to update when conventions changed, and they drifted.
Nested symlinks instead of directory-level symlinks. The rules/shared symlink lived deep inside the project's .claude/ tree. When the target moved, every project that had run the script got a broken symlink --- silently.
Hardcoded paths that drifted. The script referenced workspace paths from three months earlier. My actual directory layout had changed; the script still pointed at the old locations.

When I eventually deleted the shared directory, a quick
find confirmed broken symlinks scattered across every repo
that had run the setup script. The approach was inherently fragile
because it depended on every machine, every repo, and every workspace
path staying synchronized manually.

The fix isn't a smarter script. It's inverting the relationship:
instead of a script that runs once per project, use dotfiles that wire
context automatically based on what directories exist.

The Three-Tier Hierarchy

~/.claude/CLAUDE.md              ← Global: preferences, style, git workflow
~/work/{employer}/.claude/       ← Org: team structure, AWS accounts, Jira workflow
~/work/{employer}/{repo}/.claude/ ← Project: repo architecture, active tickets

Claude Code walks up the directory tree loading
CLAUDE.md files at each level. Each tier handles a specific
scope:

Global tier (~/.claude/): Everything
that applies across all work --- communication style, git commit format,
PR description templates, universal infrastructure patterns. No
credentials, no account IDs, nothing employer-specific.

Org tier (~/work/{employer}/.claude/):
Team structure, Jira project keys, AWS account layout, CI/CD pipeline
conventions. Sensitive patterns (account IDs, VPC IDs, state bucket
names) go in gitignored files within this directory. Reusable patterns
(CI/CD templates, AWS patterns without specifics) go in committed
files.

Project tier
(~/work/{employer}/{repo}/.claude/): Architecture decisions
for this specific repo, active tickets, ongoing work state. Always
gitignored --- this is ephemeral working context that changes
frequently.

Implementation: Symlinks from Dotfiles

The hierarchy only works if it's consistent across machines. I manage
all context files from a dotfiles repo using symlinks:

dotfiles/claude/
├── global/          → symlinked to ~/.claude/
├── {employer}/      → symlinked to ~/work/{employer}/.claude/
└── {personal}/      → symlinked to ~/personal/.claude/

install.sh wires these automatically:

# Global context
ln -sf "$DOTFILES/claude/global" "$HOME/.claude"

# Per-employer context
for employer in "${EMPLOYERS[@]}"; do
  WORK_DIR="$HOME/work/$employer"
  if [ -d "$WORK_DIR" ]; then
    ln -sf "$DOTFILES/claude/$employer" "$WORK_DIR/.claude"
  fi
done

Any machine that runs install.sh gets the same context
hierarchy. Changes committed to dotfiles propagate immediately.

What Each Level Contains

Global (`~/.claude/`)

~/.claude/
├── CLAUDE.md          # Preferences, active work summary
└── rules/
    ├── git-workflow.md
    ├── pr-patterns.md
    ├── infrastructure.md
    └── context-management.md

CLAUDE.md is short --- preferences and a pointer to where
active work lives. The heavy lifting goes in rules/ files
that Claude loads as supplementary context.

## Communication Style
- Be direct and technical — I understand infrastructure concepts
- Explain the "why" behind decisions
- Provide specific file paths and line numbers

## Git Workflow
- Branch format: feat/TICKET-123-description
- Commit format: [TICKET-123] Brief summary\n\nWhy this change...
- Never add Co-Authored-By trailers

Org Level (`~/work/{employer}/.claude/`)

~/work/{employer}/.claude/
├── {EMPLOYER}.md              # Team structure, Jira workflow — committed
└── rules/
    ├── cicd-patterns.md       # CI/CD conventions — committed
    ├── aws-patterns.md        # Account IDs, VPC IDs — GITIGNORED
    └── terraform-patterns.md  # State config, module paths — GITIGNORED

The privacy split matters. cicd-patterns.md contains
reusable GitHub Actions patterns --- fine to commit.
aws-patterns.md contains actual account IDs --- stays
local.

A typical employer context file:

## Team Structure
- Platform team: 5 engineers, all in Jira project IN
- AWS accounts: dev, nonprod, prod (+ 3 infra accounts network, security, management)
- Monorepo: ~/work/{employer}/iac — Terragrunt, 78 components

## Jira Workflow
- IN project (infrastructure): transition IDs 3=In Dev, 8=Needs Review, 9=Done
- Prefix all commits: [IT-XXX]
- API: REST v2 only — v3 silently returns empty responses

## CI/CD
- GitHub Actions with OIDC to AWS (no long-lived credentials)
- PR requires: terraform plan output posted as comment
- Merge to main triggers auto-deploy to nonprod

The gitignored aws-patterns.md contains account IDs and
specific ARNs that Claude needs for generating Terraform configurations
accurately but shouldn't be committed anywhere.

Project Level (`~/work/{employer}/{repo}/.claude/`)

Project context is ephemeral and always gitignored. It's the working
memory for an ongoing effort:

## Current State
- Branch: feat/IT-89-my-app-dev-ecr
- Active ticket: IT-89 — ECR in dev
- Next: IT-90 — ECS task definition

## Architecture Decisions
- ECR in dev only; cross-account pull policies for nonprod and prod
- Mutable tags in dev, immutable in nonprod/prod
- KMS key per environment, not per repository

## Blockers
- Waiting on network DNS zone creation before cutover can proceed

I update this file at the end of each session with current state so
the next session loads instantly without re-explaining where things
are.

The Privacy Model

The critical insight is that context files need two categories:
committed (shareable) and local-only (sensitive).

Content Location Committed?

Personal preferences ~/.claude/CLAUDE.md ✅
Git workflow rules ~/.claude/rules/git-workflow.md ✅
Team structure {employer}/.claude/{EMPLOYER}.md ✅ sanitized
CI/CD patterns {employer}/.claude/rules/cicd-patterns.md ✅
AWS account IDs {employer}/.claude/rules/aws-patterns.md ❌ gitignored
VPC IDs, state config {employer}/.claude/rules/terraform-patterns.md ❌ gitignored
Active ticket state {repo}/.claude/OVERRIDES.md ❌ gitignored

The .gitignore at the dotfiles level handles this
automatically by ignoring **/aws-patterns.md and
**/terraform-patterns.md across all employer
directories.

Custom Commands

Beyond context files, Claude Code supports custom
/commands --- reusable prompts stored as markdown files:

~/.claude/commands/
├── checkpoint.md       # Create context snapshot
├── sync-work.md        # Update active work status
└── pr-ready.md         # Generate PR description

A command file is just the prompt Claude should execute:

# checkpoint.md
Create a context checkpoint. Read the current git status across active repos,
summarize open PRs and their status, list active tickets with their current
state, and write a structured summary to ~/.claude/local.md. Include any
blocking issues and the next planned action.

Commands at the global level are available everywhere. Org-level
commands handle employer-specific workflows like Jira transitions.

What This Solves in Practice

Before this system: every new Claude session started with "here's the
project, here are the conventions, here's where things are." Five
minutes of ramp-up, inconsistent outputs because I'd forget to mention
something.

After: I cd into a repo and Claude already knows the
Jira workflow, the AWS account structure, the naming conventions, and
where the active work stands. When I start a session mid-ticket, the
project-level context tells Claude exactly what was in progress.

The bigger payoff is consistency. When Claude generates Terraform, it
generates it with the correct state backend. When it writes commit
messages, they follow the format reviewers expect. When it suggests
architecture, it fits the actual account model rather than a generic AWS
example.

Starting Point

If you're starting from scratch, work tier by tier:

Create ~/.claude/CLAUDE.md with your communication preferences and git conventions.
Add a rules/ directory with patterns you want loaded consistently.
Create an org-level directory when you start working with a specific employer or major project.
Add project-level context when you start a multi-session effort.

Don't try to build the whole system at once. The global tier alone
eliminates most of the per-session ramp-up. The org and project tiers
pay off as work gets more complex.

The thing that surprised me most wasn't the time saved on ramp-up. It
was how much the output quality improved. When Claude knows the actual
state backend, the actual account IDs, the actual PR format your
reviewers expect --- the suggestions it makes fit your environment. That
gap between "technically correct" and "actually usable" is where most of
the friction in AI-assisted infrastructure work lives. The context
hierarchy is mostly just closing that gap.

If you're setting up Claude Code for a platform team and want to talk
through the context design, I do advisory
engagements for teams getting serious about AI tooling in their
infrastructure workflow.

Great design, and easy to follow.

Glenn Gray — Wed, 18 Mar 2026 16:23:46 +0000

How to Implement AWS Network Firewall in a Multi-Account Architecture Using Transit Gateway

Cristhian Becerra ・ Oct 13 '25

#english #aws #networking #cybersecurity

Building Automated AWS Permission Testing Infrastructure for CI/CD

Glenn Gray — Wed, 18 Mar 2026 07:14:50 +0000

This post was originally published on graycloudarch.com.

I deployed a permission set for our data engineers five times before
it worked correctly.

The first deployment: S3 reads worked, Glue Data Catalog reads
worked. Athena queries failed --- the query engine needs KMS decrypt
through a service principal, and I'd missed the
kms:ViaService condition. Second deployment: Athena worked.
EMR Serverless job submission failed --- missing
iam:PassRole. Third deployment: EMR submission worked. Job
execution failed --- missing permissions on the EMR Serverless execution
role boundary. I kept deploying, engineers kept getting blocked, I kept
opening tickets.

Five iterations. Two weeks. Every failure meant a data engineer
opened a ticket instead of running their job.

The problem wasn't that IAM is complicated --- it is, but that's
expected. The problem was that I had no way to catch these issues before
deploying to the account where real engineers were trying to do real
work. Every bug was a production bug.

The "Access Denied" Debugging Loop

Here's what the reactive debugging cycle looks like from the
inside.

Engineer opens a ticket:
AccessDeniedException: User is not authorized to perform: s3:GetObject.
I add s3:GetObject to the permission set. Next day:
AccessDeniedException: s3:PutObject. I add
s3:PutObject. Day after: write succeeds but cleanup fails ---
s3:DeleteObject. At this point I've done four deployment
cycles and two days of work to get S3 read/write/delete working. If I'd
just added s3:* I'd be done, but that violates
least-privilege and opens the raw zone to write access, which we
explicitly don't want.

The deeper issue is that individual services don't fail atomically.
Athena requires athena:StartQueryExecution and
athena:GetQueryResults and
athena:GetQueryExecution, but it also requires KMS decrypt
through the Athena service principal to read encrypted S3 results. That
last piece isn't in the Athena docs --- you find it by failing in
production.

I wanted a way to find it before deploying.

What I Built

The testing framework has four components: per-persona permission set
templates, a Bash test library, per-service test scripts, and a GitHub
Actions workflow that runs everything on pull requests.

┌─────────────────────────────────────────────────┐
│  GitHub Pull Request (Permission Set Changes)   │
└───────────────────┬─────────────────────────────┘
                    │
         ┌──────────▼──────────┐
         │  CI/CD Workflow     │
         │  (GitHub Actions)   │
         └──────────┬──────────┘
                    │
    ┌───────────────┼───────────────┐
    ▼               ▼               ▼
┌───────┐      ┌──────────┐   ┌──────────┐
│ S3    │      │  Glue    │   │ Athena   │
│ Tests │      │  Tests   │   │  Tests   │
└───────┘      └──────────┘   └──────────┘
                    │
         ┌──────────▼──────────┐
         │  Test Report        │
         │  (Posted to PR)     │
         └─────────────────────┘

The workflow triggers on any pull request that modifies the
identity-center Terraform directory. Tests run against real AWS accounts
--- dev and nonprod --- using test credentials provisioned for that purpose.
Results post as a PR comment before anyone approves the change.

Phase 1: Pre-Validated Templates

Before I wrote a single test, I needed a starting point for
permission sets that captured the patterns I'd learned the hard way.
Templates that handle the non-obvious pieces --- zone-scoped S3 access,
KMS conditions tied to specific services, explicit denies for
destructive operations.

The AnalystAccess template is representative. Analysts
get read-only access to the curated zone of the data lake, Athena query
execution in the primary workgroup, and KMS decrypt --- but only when the
decrypt request originates from S3 or Athena, not from arbitrary API
calls:

inline_policy = jsonencode({
  Version = "2012-10-17"
  Statement = [
    {
      Sid    = "GlueCatalogReadOnly"
      Effect = "Allow"
      Action = ["glue:GetDatabase", "glue:GetTable", "glue:GetPartitions", "glue:SearchTables"]
      Resource = [
        "arn:aws:glue:*:*:catalog",
        "arn:aws:glue:*:*:database/curated_*",
        "arn:aws:glue:*:*:table/curated_*/*"
      ]
    },
    {
      Sid    = "S3CuratedReadOnly"
      Effect = "Allow"
      Action = ["s3:GetObject", "s3:ListBucket"]
      Resource = ["arn:aws:s3:::lake-bucket-*/curated/*", "arn:aws:s3:::lake-bucket-*"]
      Condition = { StringLike = { "s3:prefix" = ["curated/*"] } }
    },
    {
      Sid    = "AthenaQueryExecution"
      Effect = "Allow"
      Action = ["athena:StartQueryExecution", "athena:GetQueryExecution", "athena:GetQueryResults", "athena:StopQueryExecution"]
      Resource = "arn:aws:athena:*:*:workgroup/primary"
    },
    {
      Sid    = "KMSDecryptViaSvc"
      Effect = "Allow"
      Action = ["kms:Decrypt", "kms:DescribeKey"]
      Resource = "arn:aws:kms:*:*:key/*"
      Condition = {
        StringEquals = { "kms:ViaService" = ["s3.us-east-1.amazonaws.com", "athena.us-east-1.amazonaws.com"] }
      }
    },
    {
      Sid    = "DenyDestructiveOps"
      Effect = "Deny"
      Action = ["s3:DeleteObject", "s3:DeleteBucket", "glue:DeleteDatabase", "glue:DeleteTable"]
      Resource = "*"
    }
  ]
})

The kms:ViaService condition is the piece that took five
production failures to discover. KMS decrypt without that condition
allows an analyst to call kms:Decrypt directly from their
shell, which is not what we want. The condition locks decrypt to
requests that pass through S3 or Athena specifically.

The explicit deny block matters too. Without it, if someone later
grants broader S3 permissions to this persona for a different reason,
the curated zone protection evaporates. The deny creates a hard floor
regardless of what else gets added.

Phase 2: The Test Framework

I chose Bash over Python or a proper test framework deliberately. The
tests run in CI with no dependencies beyond the AWS CLI --- no package
installs, no virtual environments, no version pinning of test libraries.
The machines running these tests already have the AWS CLI.

The core library in lib/test-framework.sh:

::: {#cb3 .sourceCode}

declare -a TESTS_PASSED=()
declare -a TESTS_FAILED=()

run_test() {
  local test_name="$1"
  local test_command="$2"
  local description="$3"

  if eval "$test_command" &>/dev/null; then
    TESTS_PASSED+=("$test_name")
    echo "  ✅ PASS: $test_name"
  else
    TESTS_FAILED+=("$test_name")
    echo "  ❌ FAIL: $test_name"
  fi
}

generate_text_report() {
  echo "Total: $((${#TESTS_PASSED[@]} + ${#TESTS_FAILED[@]}))"
  echo "Passed: ${#TESTS_PASSED[@]}"
  echo "Failed: ${#TESTS_FAILED[@]}"
  [ ${#TESTS_FAILED[@]} -gt 0 ] && printf '  - %s\n' "${TESTS_FAILED[@]}"
}

:::

The most important design decision in the test scripts is testing
denials as carefully as allowances. Testing only what should succeed
tells you the permission set isn't obviously broken. Testing what should
fail tells you it's not accidentally too permissive.

::: {#cb4 .sourceCode}

# Test what should succeed
run_test "s3-list-curated"
  "aws s3 ls s3://lake-bucket-dev/curated/"
  "Analyst can list curated zone"

# Test what should fail (negative test)
run_test "s3-write-denied"
  "! aws s3 cp /tmp/test.txt s3://lake-bucket-dev/curated/test.txt 2>&1 | grep -q 'AccessDenied'"
  "Analyst cannot write to curated zone"

run_test "s3-raw-zone-denied"
  "! aws s3 ls s3://lake-bucket-dev/raw/ 2>&1 | grep -q 'AccessDenied'"
  "Analyst cannot access raw zone"

:::

Beyond service-level tests, I run persona tests that simulate
end-to-end workflows. An analyst's workflow isn't "call S3, then call
Athena separately" --- it's "run an Athena query that reads encrypted S3
data and writes results to the query results bucket." That integration
test catches failures that individual service tests miss. The original
five-iteration DataPlatformAccess failure? An individual S3 test would
have passed. A persona test running an actual Athena query against the
encrypted lake would have caught the KMS gap.

Phase 3: CI/CD Integration

The GitHub Actions workflow triggers on pull requests that touch the
identity-center Terraform directory, runs tests in a matrix against dev
and nonprod, and posts a summary comment to the PR.

::: {#cb5 .sourceCode}

on:
  pull_request:
    paths:
      - 'common/modules/identity-center/**/*.tf'

permissions:
  contents: read
  id-token: write
  pull-requests: write

jobs:
  test-permissions:
    strategy:
      matrix:
        environment: [workloads-dev, workloads-nonprod]
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::${{ matrix.environment.account }}:role/github-actions-role
      - run: ./scripts/test-permissions/run-permission-tests.sh --persona analyst

:::

The id-token: write permission is required for OIDC
authentication to AWS --- the workflow assumes a role in each account
rather than using long-lived credentials in GitHub Secrets. This is the
right pattern: credentials rotate automatically, and there's no secret
to rotate manually or accidentally expose.

The PR comment posts the full test output with pass/fail counts per
persona per account. A reviewer can look at the comment and immediately
see whether the permission change has test coverage and whether the
tests pass.

Three Things I Learned the Hard Way

First: test KMS decryption through each service separately.
kms:Decrypt via S3 and kms:Decrypt via Athena
are different IAM evaluation paths even though they're the same API
call. A test that puts an object and gets it back via S3 directly won't
catch a broken Athena KMS path.

Second: negative tests matter as much as positive ones. Before I had
the test framework, every permission set I wrote was tested only for
what it should allow. I had no systematic check that it didn't allow
more. The denial tests are what give security reviewers confidence.

Third: persona tests catch failures that service tests miss.
Individual service tests are fast to write and good for regression
coverage, but they test permissions in isolation. Real workflows cross
service boundaries. Build both.

What Changed

Before the framework: five iterations to get one permission set
right, every iteration a production impact. After: 95% of permission
issues caught at PR review time. Zero production impacts from permission
bugs since we shipped it. The templates reduced new permission set
creation time by about 70% --- instead of starting from scratch with the
IAM documentation, we start from a pre-validated base and modify from
there.

The time investment was about a week: two days for templates, two
days for the test framework and scripts, one day for CI/CD integration
and documentation. That investment paid back in the first sprint when
the analyst permission set for a new hire went out correct on the first
deployment.

Running into IAM permission debugging loops on your team? Reach out --- permission testing infrastructure is
one of the first things I build when joining a new platform team.

Zero-Downtime AWS Transit Gateway Hub-Spoke Migration

Glenn Gray — Thu, 12 Mar 2026 03:05:59 +0000

This post was originally published on graycloudarch.com.

The request came from the security team: they needed network-level\
access from the nonprod account to the dev account so a vulnerability\
scanner could reach internal services. Simple enough on the surface. In\
practice, it exposed a gap we'd been living with for months --- and forced\
us to fix the network architecture we'd been deferring.

We had three standalone Transit Gateways: one in each workload\
account, dev, nonprod, and prod. Completely isolated from each other. No\
cross-account connectivity at all. The security scanner couldn't reach\
its targets, and adding more point-to-point peering connections to fix\
it would have made everything worse.

But the TGW isolation was only part of the problem. We also had no\
inspection of traffic crossing our network boundary. Egress from\
workload pods went straight to the internet with no filtering. Ingress\
came through per-account load balancers with no centralized enforcement\
point. As the platform scaled toward additional workload accounts, this\
pattern was going to get expensive and hard to reason about.

So we didn't just fix the TGW. We rebuilt the network foundation: a\
centralized Inspection VPC with a Network Firewall inline, a single hub\
Transit Gateway shared across all accounts, and centralized security\
tooling (GuardDuty, CloudTrail, Security Hub) aggregated in a dedicated\
Security account. Two maintenance windows, a few weeks of module work,\
and the platform went from fragmented per-account networking to a\
coherent hub-spoke design with full traffic inspection.

The Architecture We Were Replacing

Before the migration, each workload account was self-contained. It\
had its own TGW, its own internet gateway, its own NAT gateways.\
Security tooling ran independently in each account with no aggregation.\
The management account had no single-pane visibility into what was\
happening across the environment.

The cost of running this way was about \$150/month in TGW charges plus\
duplicated NAT gateway charges in each account. Every new workload\
account would multiply this cost again and add another independent\
security configuration.

The Target: Inspection VPC + Hub Transit Gateway

The target was AWS Security Reference Architecture Pattern B: an\
Inspection VPC that sits between the internet and all workload VPCs. All\
internet traffic --- ingress and egress --- flows through this VPC and\
through a Network Firewall before reaching any workload account.

Egress path: workload pod → TGW → Inspection VPC TGW subnets →\
Network Firewall → NAT Gateway → IGW → internet.

Ingress path: internet → IGW → centralized ALB (public subnet) →\
Network Firewall → TGW → workload VPC → pod.

Nothing crosses the network boundary without passing through the\
firewall. Workload accounts carry no internet-facing infrastructure at\
all --- no IGW, no NAT gateways, no public load balancers.

Phase 1: Module Changes

All Terraform work happened before scheduling any maintenance. The\
goal was to reach a state where the migration itself was just running\
pre-staged plan files in a specific sequence.

Transit Gateway: add a conditional create flag

The existing network module always created a TGW. We needed spoke\
accounts to declare the same module without spinning up their own\
gateway:

::: {#cb1 .sourceCode}

variable "create_transit_gateway" {
  description = "Whether to create a Transit Gateway (false for hub-spoke spokes)"
  type        = bool
  default     = true
}

resource "aws_ec2_transit_gateway" "this" {
  count       = var.create_transit_gateway ? 1 : 0
  description = var.tgw_description
}

output "transit_gateway_id" {
  value = var.create_transit_gateway ? aws_ec2_transit_gateway.this[0].id : null
}

:::

default = true means existing configurations need no\
changes. The flag only flips to false after the spoke\
attachment is confirmed working.

New module: vpc-attachment

The vpc-attachment module handles the spoke side of the hub\
relationship: create the TGW attachment, associate it to the hub's route\
table, and add routes to every private route table in the spoke VPC\
pointing at the hub TGW.

::: {#cb2 .sourceCode}

resource "aws_ec2_transit_gateway_vpc_attachment" "this" {
  transit_gateway_id = var.transit_gateway_id
  vpc_id             = var.vpc_id
  subnet_ids         = var.subnet_ids

  tags = merge(var.tags, {
    Name = "${var.name}-hub-attachment"
  })
}

resource "aws_ec2_transit_gateway_route_table_association" "this" {
  transit_gateway_attachment_id  = aws_ec2_transit_gateway_vpc_attachment.this.id
  transit_gateway_route_table_id = var.transit_gateway_route_table_id
}

resource "aws_route" "to_hub_tgw" {
  for_each               = toset(var.vpc_route_table_ids)
  route_table_id         = each.value
  destination_cidr_block = "10.0.0.0/8"
  transit_gateway_id     = var.transit_gateway_id
}

:::

The 10.0.0.0/8 supernet covers all workload and\
Inspection VPC CIDRs without maintaining per-prefix route entries. It\
also covers the Inspection VPC CIDR (10.100.0.0/20) ---\
that's how return traffic from the centralized ALB finds its way back to\
pods in workload VPCs.

The Terragrunt config for a spoke account reads VPC details from the\
existing network dependency and hardcodes the hub TGW identifiers:

dependency "network" {
  config_path = "../network"
  mock_outputs = {
    vpc_id                  = "vpc-mockid"
    private_subnet_ids      = ["subnet-mock1"]
    private_route_table_ids = ["rtb-mock1"]
  }
}

inputs = {
  transit_gateway_id             = "tgw-xxxxx"   # hub TGW, documented in runbook
  transit_gateway_route_table_id = "tgw-rtb-xxxxx"
}

We hardcoded the hub TGW and route table IDs rather than using\
cross-account data sources. The alternative --- reading TGW details from\
the Infrastructure account at plan time --- requires cross-account state\
access and adds complexity that isn't worth it for values that change\
maybe once in the platform's lifetime.

Hub route tables: workload isolation by default

A key design decision: workload accounts should not route to each\
other directly. Dev should not reach nonprod; nonprod should not reach\
prod. The hub TGW enforces this through route table structure:

default-association-rt: all workload attachments\ associate here. The only route is\ 0.0.0.0/0 → inspection attachment. Workloads can reach the\ internet via the Inspection VPC, but cannot reach other workload\ VPCs.
default-propagation-rt: the inspection attachment\ propagates workload CIDRs here for return traffic routing.

Inter-account communication is opt-in: you add an explicit route\
table entry for a specific attachment pair. By default, the architecture\
prevents lateral movement across workload accounts.

Inspection VPC subnet layout

The Inspection VPC has three tiers with carefully constructed route\
tables that force traffic through the firewall in both directions:

The asymmetric route table design ensures the firewall sees every\
packet crossing the network boundary, regardless of direction. Traffic\
entering from the internet hits the firewall before reaching workloads.\
Traffic from workloads hits the firewall before reaching the\
internet.

Security baseline: convert to delegated admin model

GuardDuty and CloudTrail were running independently per account. We\
added enable_guardduty and enable_cloudtrail\
boolean variables to the security-baseline module so workload accounts\
could switch from standalone to member without touching the module\
invocation itself.

In the Security account, we deployed:

GuardDuty as delegated admin with\ organization-level auto-enrollment. EKS Protection and S3 Protection\ enabled. All findings from all accounts visible in a single\ dashboard.
CloudTrail organization trail writing to a\ cross-account S3 bucket. Log file validation and KMS encryption enabled.\ Per-account trails archived after the cutover --- not deleted, in case\ historical log formats differed.
Security Hub with CIS AWS Foundations Benchmark and\ AWS Foundational Security Best Practices enabled across the full\ organization.

Phase 2: Two Maintenance Windows

Window 1: Deploy the hub (~45 minutes, low risk)

With no existing attachments and no workload traffic, deploying the\
hub infrastructure carried minimal risk. We applied the Infrastructure\
account TGW and Inspection VPC in a single window. The Network Firewall\
takes 5--10 minutes to reach READY state after creation --- account for\
that in your timing.

At the end of this window: hub TGW running, Inspection VPC active,\
Network Firewall endpoints healthy in both AZs, centralized ALB\
deployed. Nothing attached yet. We documented the TGW ID and route table\
IDs in the runbook before scheduling window 2.

Window 2: Spoke cutover (~2 hours)

The key insight for keeping applications running: create the\
hub attachment before destroying the standalone TGW. While both\
exist simultaneously, traffic continues flowing through the standalone\
path. The actual cutover is updating routes to point at the hub --- that's\
a single terragrunt apply, not the destruction of the old\
TGW.

T+0 --- Accept RAM share. Infrastructure account\
shares the hub TGW via Resource Access Manager. Workload accounts accept\
the share invitation. Pure metadata operation; zero network impact.

T+15 --- Deploy VPC attachments. Apply the\
vpc-attachment module in each workload account. At this\
point each spoke VPC has two routes for 10.0.0.0/8: the\
existing one pointing at the standalone TGW, and the new one pointing at\
the hub. With identical prefix lengths, traffic still flows through the\
standalone path. Rollback at this stage is\
terragrunt destroy on the attachment module --- under five\
minutes.

T+30 --- Verify routes and test cross-account\
connectivity. Confirm hub routes are present in every private\
route table:

::: {#cb4 .sourceCode}

aws ec2 describe-route-tables \
  --filters "Name=vpc-id,Values=vpc-xxxxx" \
  --query 'RouteTables[*].Routes[?DestinationCidrBlock==`10.0.0.0/8`]'

:::

Then test actual cross-account traffic: connect from a dev instance\
to a service in the nonprod VPC. The hub TGW and Inspection VPC should\
route it correctly. This also validates that the firewall rule groups\
are permitting expected traffic --- catch any rule issues here, before\
cutting over production.

T+45 --- Migrate security tooling. Apply the updated\
security-baseline to each workload account. GuardDuty converts from\
standalone admin to member; findings flow to the Security account\
delegated admin. CloudTrail local trail disabled; organization trail\
confirmed logging events from the account. Zero network impact.

::: {#cb5 .sourceCode}

# Verify GuardDuty membership
aws guardduty get-administrator-account --detector-id <id>
# Returns the Security account as administrator

# Verify organization trail is capturing events
# Make an API call, wait ~15 minutes, check the Security account's S3 bucket
aws s3 ls s3://<org-trail-bucket>/AWSLogs/<account-id>/

:::

T+60 --- Set create_transit_gateway = false in\
each spoke. This is the cutover. Run\
terraform plan first and confirm it shows only the TGW and\
its attached resources being destroyed --- nothing else. Apply dev first,\
watch the destruction complete, confirm application traffic is flowing\
through the hub. Then apply nonprod. About 3 minutes per account.

T+90 --- Health checks and close. Spot-check API\
endpoints, database connectivity, anything that traverses the network.\
Confirm egress traffic is hitting the firewall logs in the\
Infrastructure account. The maintenance window closed at the 90-minute\
mark; actual work was done by T+75. We kept the window open for the last\
15 minutes as a buffer.

The parallel attachment approach ensured there was never a moment\
where a workload account had no routing path. Even if the hub TGW had\
been misconfigured, traffic would have continued flowing through the\
standalone gateway until we chose to destroy it.

What We Ended Up With

One TGW in the Infrastructure account with three\
spoke attachments. Route tables that allow workload→internet traffic\
while preventing workload→workload lateral movement by default.

One Inspection VPC with Network Firewall endpoints\
in two AZs. All egress inspected against stateful domain filter rules\
and stateless port rules. All ingress from the centralized ALB\
inspected. Firewall policy updates apply to all workload accounts\
simultaneously --- no per-account changes needed.

One centralized ALB in the Infrastructure account,\
routing to EKS target groups in workload accounts via cross-account IAM\
role assumption. Workload accounts carry no public-facing load\
balancers.

One security console in the Security account.\
GuardDuty findings from all accounts in a single dashboard. CloudTrail\
logs from every account in one S3 bucket. Security Hub compliance\
posture for the full organization visible in one place.

Cost went from roughly \$150--200/month (standalone TGWs, per-account\
NAT, independent security tooling) to approximately \$50/month (single\
hub TGW plus attachment hours, shared NAT in the Inspection VPC,\
delegated security services). Cost savings validated against AWS Cost\
Explorer after 30 days.

The original security scanner request --- cross-account access from\
nonprod to dev --- was live the same day. The compliance team had a single\
GuardDuty and Security Hub dashboard the same week.

More importantly: adding a new workload account to this architecture\
now takes about an hour. Create the VPC, deploy the vpc-attachment\
module pointing at the documented hub TGW ID, invite the new account as\
a GuardDuty and Security Hub member, apply the security-baseline with\
enable_guardduty = false. Every new account inherits the\
full inspection and security posture without any per-account\
configuration. That's the actual value of a hub-spoke design --- not the\
one-time cost savings, but the fact that account seven is as\
well-secured and as easy to audit as account two.

Working through a multi-account network redesign, or building the\
inspection layer on top of an existing Transit Gateway setup? Get in touch --- this is the kind of platform\
architecture I work on regularly.

Stop Managing EKS Add-ons by Hand

Glenn Gray — Wed, 11 Mar 2026 02:20:36 +0000

This post was originally published on graycloudarch.com.

I was preparing to upgrade a production EKS cluster to version 1.32\
when I discovered a problem.

Four of our core cluster components---VPC CNI, CoreDNS, kube-proxy, and\
Metrics Server---were all running versions incompatible with EKS 1.32. I\
needed to update them before upgrading.

And I had no easy way to do it.

VPC CNI, CoreDNS, and kube-proxy had been installed automatically\
when the cluster was created, running in "self-managed" mode. Metrics\
Server was installed with\
kubectl apply -f metrics-server.yaml from some GitHub\
release page, months ago, by someone who is no longer on the team.

No version pinning. No history of what changed or when. No way to\
test the upgrade before applying it to production.

That's when I decided to stop managing EKS add-ons by hand.

The Problem with Self-Managed Add-ons

There are two categories of EKS add-ons, and most teams don't think\
about the distinction until they're stuck.

Self-managed: You're responsible for installation,\
updates, and compatibility. AWS won't help you troubleshoot them. When\
EKS releases a new version, you need to manually verify your add-ons\
still work, find compatible versions, and update them yourself.

EKS-managed: AWS handles the lifecycle. Compatible\
versions are tested and published for each EKS release. AWS Support can\
troubleshoot them. Security patches are available without you tracking\
CVEs.

If you created an EKS cluster without explicitly enabling managed\
add-ons, VPC CNI, CoreDNS, and kube-proxy are running in self-managed\
mode right now.

The fix is straightforward---migrate them to EKS-managed. But if you're\
also running kubectl-installed tools like Metrics Server, you have a\
second problem: those aren't managed by anything at all.

The Solution: One Terraform Module for All Six Add-ons

I built a single eks-addons Terraform module that\
manages everything:

EKS-managed (4): -- VPC CNI --- pod networking -- EBS\
CSI Driver --- persistent volumes (added this one while I was at it) --\
CoreDNS --- DNS resolution -- kube-proxy --- network proxy

Helm-managed (2): -- Metrics Server --- resource\
metrics for kubectl top and HPA -- Reloader --- auto-restart\
pods when ConfigMaps or Secrets change

Why one module instead of six separate ones? All of these share the\
same dependency: the EKS cluster. Consolidating them means one\
terragrunt apply deploys everything, one\
terraform plan shows drift across all add-ons, and one PR\
updates any version.

The core Terraform for an EKS-managed add-on is minimal:

resource "aws_eks_addon" "vpc_cni" {
  count = var.enable_vpc_cni ? 1 : 0

  cluster_name                = var.cluster_name
  addon_name                  = "vpc-cni"
  addon_version               = var.vpc_cni_version
  resolve_conflicts_on_create = "OVERWRITE"
  resolve_conflicts_on_update = "OVERWRITE"
  preserve                    = true
}

Two things worth explaining:

resolve_conflicts = "OVERWRITE" tells Terraform it's the\
source of truth. Any manual changes in the cluster get overwritten on\
the next apply. This is what you want.

preserve = true means if you remove the resource from\
Terraform, the add-on stays in the cluster. Safety net during\
refactoring---you won't accidentally delete a running add-on.

EBS CSI Driver Needs an IAM Role

The EBS CSI Driver is the one add-on that requires extra work: it\
needs IAM permissions to create and attach EBS volumes. The right way to\
handle this is IRSA (IAM Roles for Service Accounts).

resource "aws_iam_role" "ebs_csi" {
  count = var.enable_ebs_csi ? 1 : 0
  name  = "${var.cluster_name}-ebs-csi-driver"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Federated = var.oidc_provider_arn }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${var.oidc_provider}:sub" = "system:serviceaccount:kube-system:ebs-csi-controller-sa"
          "${var.oidc_provider}:aud" = "sts.amazonaws.com"
        }
      }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "ebs_csi" {
  count      = var.enable_ebs_csi ? 1 : 0
  role       = aws_iam_role.ebs_csi[0].name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
}

No credentials in pods, automatic rotation, and a clean audit trail\
in CloudTrail. IRSA is the correct pattern for any AWS service that\
needs to call AWS APIs from inside Kubernetes.

Migrating Metrics Server from kubectl to Helm

This is the one step that requires manual cleanup before Terraform\
can take over.

The existing kubectl-installed Metrics Server needs to go first:

kubectl delete deployment metrics-server -n kube-system
kubectl delete service metrics-server -n kube-system
kubectl delete apiservice v1beta1.metrics.k8s.io

Then Terraform installs the Helm-managed version:

resource "helm_release" "metrics_server" {
  count      = var.enable_metrics_server ? 1 : 0
  name       = "metrics-server"
  repository = "https://kubernetes-sigs.github.io/metrics-server/"
  chart      = "metrics-server"
  version    = var.metrics_server_chart_version
  namespace  = "kube-system"

  values = [yamlencode({
    replicas = 2
    args = [
      "--kubelet-preferred-address-types=InternalIP",
      "--kubelet-insecure-tls"
    ]
    podDisruptionBudget = {
      enabled      = true
      minAvailable = 1
    }
  })]
}

Expected downtime: 2-3 minutes. Only kubectl top is\
unavailable during the transition. Running applications are not\
affected.

Deploying It

One thing that bit me: CI/CD doesn't pick up module changes\
automatically.

Our GitHub Actions workflow detects changes by looking for modified\
terragrunt.hcl files. When I changed files in\
common/modules/eks-addons/, the workflow triggered but\
found no stacks to deploy (no terragrunt.hcl changed), so\
nothing ran.

Module changes require a manual deploy:

cd workloads-nonprod/us-east-1/cluster-name/eks-addons
terragrunt init
terragrunt plan   # Review: should show ~10 resources to add
terragrunt apply

After apply, verify everything is healthy:

# Check EKS-managed add-on status
for addon in vpc-cni aws-ebs-csi-driver coredns kube-proxy; do
  aws eks describe-addon --cluster-name <cluster> --addon-name $addon \
    --query 'addon.[addonName,status]' --output text
done
# All should show: ACTIVE

# Verify Metrics Server
kubectl top nodes

What Changed

Before: four add-ons running in self-managed mode, one installed by\
kubectl, no version history, no drift detection.

After: -- All six add-ons defined in code with pinned versions --\
terraform plan shows immediately if anything drifts from\
the declared state -- Rollback is git revert +\
terragrunt apply -- EKS cluster upgrade checklist is now:\
update four version strings in the Terragrunt config, open a PR,\
done

The cluster upgrade I was dreading took about 30 minutes instead of a\
day of manual compatibility checking.

Running into EKS add-on management problems? Reach\
out---this is the kind of operational work I do for platform\
teams.

Forem: Glenn Gray

Stop Manually Updating Jira After Every PR Merge

The Architecture

Step 1: Composite Action for Ticket Extraction

Step 2: PR Creation Workflow

Step 3: PR Merge Workflow

Finding Your Transition IDs

A Note on Jira API Versions

Required Secrets

Results

What the first 24 hours of production CloudWatch data told us

What 99.8% CPU means at 0.5 vCPU

Right-sizing the task before configuring the autoscaler

Why we chose CPU tracking instead of request count

The observability gap app logs can't fill

Autoscaling parameters worth explaining

What 24 hours of data drove

Stop Managing EKS Add-ons by Hand

The Problem with Self-Managed Add-ons

The Solution: One Terraform Module for All Six Add-ons

EBS CSI Driver Needs an IAM Role

Migrating Metrics Server from kubectl to Helm

Deploying It

What Changed

Zero-Downtime AWS Transit Gateway Hub-Spoke Migration

The Architecture We Were Replacing

The Target: Inspection VPC + Hub Transit Gateway

Phase 1: Module Changes

Transit Gateway: add a conditional create flag

New module: vpc-attachment

Hub route tables: workload isolation by default

Inspection VPC subnet layout

Security baseline: convert to delegated admin model

Phase 2: Two Maintenance Windows

Window 1: Deploy the hub (~45 minutes, low risk)

Window 2: Spoke cutover (~2 hours)

What We Ended Up With

DNS Validation: From 15 Steps to Zero

There's a Better Way

The Magic: for_each

What I Screwed Up

Why This Matters

The Part People Miss

Scaling It

The Real Win

Building Multi-Account AWS Infrastructure with Terraform and ECP

The Problem with Single-Account AWS

The ECP Pattern

Key Design Decisions

Terraform Structure

Lessons Learned

Results

Stop Manually Updating Jira After Every PR Merge

The Architecture

Step 1: Composite Action for Ticket Extraction

Step 2: PR Creation Workflow

Step 3: PR Merge Workflow

Finding Your Transition IDs

A Note on Jira API Versions

Required Secrets

Results

How I Manage Claude Code Context Across 20+ Repositories

The Problem with Single-File Context

My First Attempt: A Shared Scripts Directory

The Three-Tier Hierarchy

Implementation: Symlinks from Dotfiles

What Each Level Contains

Global (~/.claude/)

Org Level (~/work/{employer}/.claude/)

Project Level (~/work/{employer}/{repo}/.claude/)

The Privacy Model

Custom Commands

What This Solves in Practice

Starting Point

Great design, and easy to follow.

How to Implement AWS Network Firewall in a Multi-Account Architecture Using Transit Gateway

Cristhian Becerra ・ Oct 13 '25

Building Automated AWS Permission Testing Infrastructure for CI/CD

The "Access Denied" Debugging Loop

What I Built

Global (`~/.claude/`)

Org Level (`~/work/{employer}/.claude/`)

Project Level (`~/work/{employer}/{repo}/.claude/`)