Forem: Glenn Gray

Building Automated AWS Permission Testing Infrastructure for CI/CD

Glenn Gray — Sun, 05 Apr 2026 16:53:43 +0000

Originally published on graycloudarch.com.

I deployed a permission set for our data engineers five times before it worked correctly.

The first deployment: S3 reads worked, Glue Data Catalog reads worked. Athena queries failed — the query engine needs KMS decrypt through a service principal, and I'd missed the kms:ViaService condition. Second deployment: Athena worked. EMR Serverless job submission failed — missing iam:PassRole. Third deployment: EMR submission worked. Job execution failed — missing permissions on the EMR Serverless execution role boundary. I kept deploying, engineers kept getting blocked, I kept opening tickets.

Five iterations. Two weeks. Every failure meant a data engineer opened a ticket instead of running their job.

The problem wasn't that IAM is complicated — it is, but that's expected. The problem was that I had no way to catch these issues before deploying to the account where real engineers were trying to do real work. Every bug was a production bug.

The "Access Denied" Debugging Loop

Here's what the reactive debugging cycle looks like from the inside.

Engineer opens a ticket: AccessDeniedException: User is not authorized to perform: s3:GetObject. I add s3:GetObject to the permission set. Next day: AccessDeniedException: s3:PutObject. I add s3:PutObject. Day after: write succeeds but cleanup fails — s3:DeleteObject. At this point I've done four deployment cycles and two days of work to get S3 read/write/delete working. If I'd just added s3:* I'd be done, but that violates least-privilege and opens the raw zone to write access, which we explicitly don't want.

The deeper issue is that individual services don't fail atomically. Athena requires athena:StartQueryExecution and athena:GetQueryResults and athena:GetQueryExecution, but it also requires KMS decrypt through the Athena service principal to read encrypted S3 results. That last piece isn't in the Athena docs — you find it by failing in production.

I wanted a way to find it before deploying.

What I Built

The testing framework has four components: per-persona permission set templates, a Bash test library, per-service test scripts, and a GitHub Actions workflow that runs everything on pull requests.

The workflow triggers on any pull request that modifies the identity-center Terraform directory. Tests run against real AWS accounts — dev and nonprod — using test credentials provisioned for that purpose. Results post as a PR comment before anyone approves the change.

Phase 1: Pre-Validated Templates

Before I wrote a single test, I needed a starting point for permission sets that captured the patterns I'd learned the hard way. Templates that handle the non-obvious pieces — zone-scoped S3 access, KMS conditions tied to specific services, explicit denies for destructive operations.

The AnalystAccess template is representative. Analysts get read-only access to the curated zone of the data lake, Athena query execution in the primary workgroup, and KMS decrypt — but only when the decrypt request originates from S3 or Athena, not from arbitrary API calls:

inline_policy = jsonencode({
  Version = "2012-10-17"
  Statement = [
    {
      Sid    = "GlueCatalogReadOnly"
      Effect = "Allow"
      Action = ["glue:GetDatabase", "glue:GetTable", "glue:GetPartitions", "glue:SearchTables"]
      Resource = [
        "arn:aws:glue:*:*:catalog",
        "arn:aws:glue:*:*:database/curated_*",
        "arn:aws:glue:*:*:table/curated_*/*"
      ]
    },
    {
      Sid    = "S3CuratedReadOnly"
      Effect = "Allow"
      Action = ["s3:GetObject", "s3:ListBucket"]
      Resource = ["arn:aws:s3:::lake-bucket-*/curated/*", "arn:aws:s3:::lake-bucket-*"]
      Condition = { StringLike = { "s3:prefix" = ["curated/*"] } }
    },
    {
      Sid    = "AthenaQueryExecution"
      Effect = "Allow"
      Action = ["athena:StartQueryExecution", "athena:GetQueryExecution", "athena:GetQueryResults", "athena:StopQueryExecution"]
      Resource = "arn:aws:athena:*:*:workgroup/primary"
    },
    {
      Sid    = "KMSDecryptViaSvc"
      Effect = "Allow"
      Action = ["kms:Decrypt", "kms:DescribeKey"]
      Resource = "arn:aws:kms:*:*:key/*"
      Condition = {
        StringEquals = { "kms:ViaService" = ["s3.us-east-1.amazonaws.com", "athena.us-east-1.amazonaws.com"] }
      }
    },
    {
      Sid    = "DenyDestructiveOps"
      Effect = "Deny"
      Action = ["s3:DeleteObject", "s3:DeleteBucket", "glue:DeleteDatabase", "glue:DeleteTable"]
      Resource = "*"
    }
  ]
})

The kms:ViaService condition is the piece that took five production failures to discover. KMS decrypt without that condition allows an analyst to call kms:Decrypt directly from their shell, which is not what we want. The condition locks decrypt to requests that pass through S3 or Athena specifically.

The explicit deny block matters too. Without it, if someone later grants broader S3 permissions to this persona for a different reason, the curated zone protection evaporates. The deny creates a hard floor regardless of what else gets added.

Phase 2: The Test Framework

I chose Bash over Python or a proper test framework deliberately. The tests run in CI with no dependencies beyond the AWS CLI — no package installs, no virtual environments, no version pinning of test libraries. The machines running these tests already have the AWS CLI.

The core library in lib/test-framework.sh:

declare -a TESTS_PASSED=()
declare -a TESTS_FAILED=()

run_test() {
  local test_name="$1"
  local test_command="$2"
  local description="$3"

  if eval "$test_command" &>/dev/null; then
    TESTS_PASSED+=("$test_name")
    echo "  ✅ PASS: $test_name"
  else
    TESTS_FAILED+=("$test_name")
    echo "  ❌ FAIL: $test_name"
  fi
}

generate_text_report() {
  echo "Total: $((${#TESTS_PASSED[@]} + ${#TESTS_FAILED[@]}))"
  echo "Passed: ${#TESTS_PASSED[@]}"
  echo "Failed: ${#TESTS_FAILED[@]}"
  [ ${#TESTS_FAILED[@]} -gt 0 ] && printf '  - %s\n' "${TESTS_FAILED[@]}"
}

The most important design decision in the test scripts is testing denials as carefully as allowances. Testing only what should succeed tells you the permission set isn't obviously broken. Testing what should fail tells you it's not accidentally too permissive.

# Test what should succeed
run_test "s3-list-curated" \
  "aws s3 ls s3://lake-bucket-dev/curated/" \
  "Analyst can list curated zone"

# Test what should fail (negative test)
run_test "s3-write-denied" \
  "! aws s3 cp /tmp/test.txt s3://lake-bucket-dev/curated/test.txt 2>&1 | grep -q 'AccessDenied'" \
  "Analyst cannot write to curated zone"

run_test "s3-raw-zone-denied" \
  "! aws s3 ls s3://lake-bucket-dev/raw/ 2>&1 | grep -q 'AccessDenied'" \
  "Analyst cannot access raw zone"

Beyond service-level tests, I run persona tests that simulate end-to-end workflows. An analyst's workflow isn't "call S3, then call Athena separately" — it's "run an Athena query that reads encrypted S3 data and writes results to the query results bucket." That integration test catches failures that individual service tests miss. The original five-iteration DataPlatformAccess failure? An individual S3 test would have passed. A persona test running an actual Athena query against the encrypted lake would have caught the KMS gap.

Phase 3: CI/CD Integration

The GitHub Actions workflow triggers on pull requests that touch the identity-center Terraform directory, runs tests in a matrix against dev and nonprod, and posts a summary comment to the PR.

on:
  pull_request:
    paths:
      - 'common/modules/identity-center/**/*.tf'

permissions:
  contents: read
  id-token: write
  pull-requests: write

jobs:
  test-permissions:
    strategy:
      matrix:
        environment: [workloads-dev, workloads-nonprod]
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::${{ matrix.environment.account }}:role/github-actions-role
      - run: ./scripts/test-permissions/run-permission-tests.sh --persona analyst

The id-token: write permission is required for OIDC authentication to AWS — the workflow assumes a role in each account rather than using long-lived credentials in GitHub Secrets. This is the right pattern: credentials rotate automatically, and there's no secret to rotate manually or accidentally expose.

The PR comment posts the full test output with pass/fail counts per persona per account. A reviewer can look at the comment and immediately see whether the permission change has test coverage and whether the tests pass.

Three Things I Learned the Hard Way

First: test KMS decryption through each service separately. kms:Decrypt via S3 and kms:Decrypt via Athena are different IAM evaluation paths even though they're the same API call. A test that puts an object and gets it back via S3 directly won't catch a broken Athena KMS path.

Second: negative tests matter as much as positive ones. Before I had the test framework, every permission set I wrote was tested only for what it should allow. I had no systematic check that it didn't allow more. The denial tests are what give security reviewers confidence.

Third: persona tests catch failures that service tests miss. Individual service tests are fast to write and good for regression coverage, but they test permissions in isolation. Real workflows cross service boundaries. Build both.

What Changed

Before the framework: five iterations to get one permission set right, every iteration a production impact. After: 95% of permission issues caught at PR review time. Zero production impacts from permission bugs since we shipped it. The templates reduced new permission set creation time by about 70% — instead of starting from scratch with the IAM documentation, we start from a pre-validated base and modify from there.

The time investment was about a week: two days for templates, two days for the test framework and scripts, one day for CI/CD integration and documentation. That investment paid back in the first sprint when the analyst permission set for a new hire went out correct on the first deployment.

Running into IAM permission debugging loops on your team? Reach out — permission testing infrastructure is one of the first things I build when joining a new platform team.

Stop Managing EKS Add-ons by Hand

Glenn Gray — Sun, 05 Apr 2026 16:53:40 +0000

Originally published on graycloudarch.com.

I was preparing to upgrade a production EKS cluster to version 1.32 when I discovered a problem.

Four of our core cluster components—VPC CNI, CoreDNS, kube-proxy, and Metrics Server—were all running versions incompatible with EKS 1.32. I needed to update them before upgrading.

And I had no easy way to do it.

VPC CNI, CoreDNS, and kube-proxy had been installed automatically when the cluster was created, running in "self-managed" mode. Metrics Server was installed with kubectl apply -f metrics-server.yaml from some GitHub release page, months ago, by someone who is no longer on the team.

No version pinning. No history of what changed or when. No way to test the upgrade before applying it to production.

That's when I decided to stop managing EKS add-ons by hand.

The Problem with Self-Managed Add-ons

There are two categories of EKS add-ons, and most teams don't think about the distinction until they're stuck.

Self-managed: You're responsible for installation, updates, and compatibility. AWS won't help you troubleshoot them. When EKS releases a new version, you need to manually verify your add-ons still work, find compatible versions, and update them yourself.

EKS-managed: AWS handles the lifecycle. Compatible versions are tested and published for each EKS release. AWS Support can troubleshoot them. Security patches are available without you tracking CVEs.

If you created an EKS cluster without explicitly enabling managed add-ons, VPC CNI, CoreDNS, and kube-proxy are running in self-managed mode right now.

The fix is straightforward—migrate them to EKS-managed. But if you're also running kubectl-installed tools like Metrics Server, you have a second problem: those aren't managed by anything at all.

The Solution: One Terraform Module for All Six Add-ons

I built a single eks-addons Terraform module that manages everything:

EKS-managed (4):

VPC CNI — pod networking
EBS CSI Driver — persistent volumes (added this one while I was at it)
CoreDNS — DNS resolution
kube-proxy — network proxy

Helm-managed (2):

Metrics Server — resource metrics for kubectl top and HPA
Reloader — auto-restart pods when ConfigMaps or Secrets change

Why one module instead of six separate ones? All of these share the same dependency: the EKS cluster. Consolidating them means one terragrunt apply deploys everything, one terraform plan shows drift across all add-ons, and one PR updates any version.

The core Terraform for an EKS-managed add-on is minimal:

resource "aws_eks_addon" "vpc_cni" {
  count = var.enable_vpc_cni ? 1 : 0

  cluster_name                = var.cluster_name
  addon_name                  = "vpc-cni"
  addon_version               = var.vpc_cni_version
  resolve_conflicts_on_create = "OVERWRITE"
  resolve_conflicts_on_update = "OVERWRITE"
  preserve                    = true
}

Two things worth explaining:

resolve_conflicts = "OVERWRITE" tells Terraform it's the source of truth. Any manual changes in the cluster get overwritten on the next apply. This is what you want.

preserve = true means if you remove the resource from Terraform, the add-on stays in the cluster. Safety net during refactoring—you won't accidentally delete a running add-on.

EBS CSI Driver Needs an IAM Role

The EBS CSI Driver is the one add-on that requires extra work: it needs IAM permissions to create and attach EBS volumes. The right way to handle this is IRSA (IAM Roles for Service Accounts).

resource "aws_iam_role" "ebs_csi" {
  count = var.enable_ebs_csi ? 1 : 0
  name  = "${var.cluster_name}-ebs-csi-driver"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Federated = var.oidc_provider_arn }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${var.oidc_provider}:sub" = "system:serviceaccount:kube-system:ebs-csi-controller-sa"
          "${var.oidc_provider}:aud" = "sts.amazonaws.com"
        }
      }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "ebs_csi" {
  count      = var.enable_ebs_csi ? 1 : 0
  role       = aws_iam_role.ebs_csi[0].name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
}

No credentials in pods, automatic rotation, and a clean audit trail in CloudTrail. IRSA is the correct pattern for any AWS service that needs to call AWS APIs from inside Kubernetes.

Migrating Metrics Server from kubectl to Helm

This is the one step that requires manual cleanup before Terraform can take over.

The existing kubectl-installed Metrics Server needs to go first:

kubectl delete deployment metrics-server -n kube-system
kubectl delete service metrics-server -n kube-system
kubectl delete apiservice v1beta1.metrics.k8s.io

Then Terraform installs the Helm-managed version:

resource "helm_release" "metrics_server" {
  count      = var.enable_metrics_server ? 1 : 0
  name       = "metrics-server"
  repository = "https://kubernetes-sigs.github.io/metrics-server/"
  chart      = "metrics-server"
  version    = var.metrics_server_chart_version
  namespace  = "kube-system"

  values = [yamlencode({
    replicas = 2
    args = [
      "--kubelet-preferred-address-types=InternalIP",
      "--kubelet-insecure-tls"
    ]
    podDisruptionBudget = {
      enabled      = true
      minAvailable = 1
    }
  })]
}

Expected downtime: 2-3 minutes. Only kubectl top is unavailable during the transition. Running applications are not affected.

Deploying It

One thing that bit me: CI/CD doesn't pick up module changes automatically.

Our GitHub Actions workflow detects changes by looking for modified terragrunt.hcl files. When I changed files in common/modules/eks-addons/, the workflow triggered but found no stacks to deploy (no terragrunt.hcl changed), so nothing ran.

Module changes require a manual deploy:

cd workloads-nonprod/us-east-1/cluster-name/eks-addons
terragrunt init
terragrunt plan   # Review: should show ~10 resources to add
terragrunt apply

After apply, verify everything is healthy:

# Check EKS-managed add-on status
for addon in vpc-cni aws-ebs-csi-driver coredns kube-proxy; do
  aws eks describe-addon --cluster-name <cluster> --addon-name $addon \
    --query 'addon.[addonName,status]' --output text
done
# All should show: ACTIVE

# Verify Metrics Server
kubectl top nodes

What Changed

Before: four add-ons running in self-managed mode, one installed by kubectl, no version history, no drift detection.

After:

All six add-ons defined in code with pinned versions
terraform plan shows immediately if anything drifts from the declared state
Rollback is git revert + terragrunt apply
EKS cluster upgrade checklist is now: update four version strings in the Terragrunt config, open a PR, done

The cluster upgrade I was dreading took about 30 minutes instead of a day of manual compatibility checking.

Running into EKS add-on management problems? Reach out—this is the kind of operational work I do for platform teams.

Building Apache Iceberg Lakehouse Storage with S3 Table Buckets

Glenn Gray — Sat, 04 Apr 2026 23:37:02 +0000

Originally published on graycloudarch.com.

The data platform team had a deadline and a storage decision to make. They'd committed to Apache Iceberg as the table format — open standard, time travel, schema evolution, the usual reasons. What they hadn't locked down was where the data was actually going to live, and whether the storage layer would hold up under the metadata-heavy access patterns Iceberg requires.

The default answer is regular S3. It works. Most Iceberg deployments run on it. But AWS launched S3 Table Buckets in late 2024, and they're purpose-built for exactly this workload: Iceberg metadata operations. The numbers made the decision easy — 10x faster metadata queries, 50% or more improvement in query planning time compared to standard S3. The gotcha worth knowing upfront: S3 Table Bucket support requires AWS Provider 5.70 or later. If your Terraform modules are pinned to an older provider version, that's your first upgrade.

We built the storage layer as a three-zone medallion architecture, fully managed with Terraform, with Intelligent-Tiering configured from day one. Here's how we did it.

The Medallion Architecture

Three zones, each with a clear contract about what data lives there and who owns it:

Raw is immutable. Once data lands there, it doesn't change — ETL failures don't corrupt the source record because the source record is untouched. Clean is normalized and domain-aligned, owned by data engineering. Curated is the analytics layer that BI tools, Athena queries, and QuickSight dashboards read from.

The naming convention we landed on was {zone}_{domain} for Glue databases — raw_crm, clean_customer, curated_sales_metrics. It looks minor, but it matters. When you're looking at a table in Athena or debugging a failed Glue job, the database name tells you exactly what tier you're in and what domain you're touching. Namespace collisions become impossible because the zone prefix scopes every domain. Data lineage is readable from table names alone.

Why Two Modules Instead of One

The first design question was whether to build a single composite module that creates the KMS key and the S3 Table Bucket together, or split them into separate modules. We split them.

The KMS key isn't just for the lake. It's used by five downstream services: Athena for query results, EMR for cluster encryption, MWAA for DAG storage, Kinesis for stream encryption, and Glue DataBrew for transform outputs. If we bundled the key into the lake storage module, every one of those services would need a dependency chain that eventually resolves back through lake storage just to get a KMS key ARN. Separate modules mean the key has one owner, and everything else declares a dependency on it independently.

The KMS module:

# kms-key/main.tf
resource "aws_kms_key" "this" {
  description             = var.description
  enable_key_rotation     = var.enable_key_rotation
  deletion_window_in_days = var.deletion_window_in_days

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "Enable IAM User Permissions"
        Effect = "Allow"
        Principal = { AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root" }
        Action   = "kms:*"
        Resource = "*"
      },
      {
        Sid    = "Allow Service Access"
        Effect = "Allow"
        Principal = { Service = var.service_principals }
        Action = ["kms:Decrypt", "kms:GenerateDataKey", "kms:CreateGrant"]
        Resource = "*"
      }
    ]
  })
}

The service_principals variable takes a list of service principal strings — ["athena.amazonaws.com", "glue.amazonaws.com"] and so on. Adding a new service that needs key access is one line in the Terragrunt config, no module change required.

The S3 Table Bucket Module

The table bucket itself is straightforward. The interesting part is Intelligent-Tiering:

# s3-table-bucket/main.tf
resource "aws_s3tables_table_bucket" "this" {
  name = var.bucket_name
}

resource "aws_s3_bucket_intelligent_tiering_configuration" "this" {
  count  = var.enable_intelligent_tiering ? 1 : 0
  bucket = aws_s3tables_table_bucket.this.name
  name   = "EntireBucket"

  tiering {
    access_tier = "ARCHIVE_ACCESS"
    days        = 90
  }
  tiering {
    access_tier = "DEEP_ARCHIVE_ACCESS"
    days        = 180
  }
}

We enable Intelligent-Tiering on the entire bucket from the start. The 90-day threshold for Archive Access and 180-day threshold for Deep Archive weren't arbitrary — they match the typical access patterns for a data lake: raw data is queried heavily during initial load and validation, then access drops off sharply once the clean layer is populated.

The reason Intelligent-Tiering beats manual lifecycle policies here is subtle but important. A manual lifecycle policy moves data based on age. Intelligent-Tiering moves data based on actual access patterns. If a dataset from eight months ago suddenly becomes relevant for a compliance audit, Intelligent-Tiering keeps it in a more accessible tier automatically. A manual policy would have moved it to Deep Archive on day 180 regardless. For a data lake, where access patterns are genuinely unpredictable, letting AWS monitor actual usage is worth the small monitoring fee.

The Terragrunt dependency chain wires the KMS key ARN into the table bucket configuration:

# lake-storage/terragrunt.hcl
dependency "kms" {
  config_path = "../kms-key"
}

inputs = {
  bucket_name = "company-lake-${local.environment}"
  kms_key_arn = dependency.kms.outputs.key_arn
}

Glue Data Catalog

We provisioned 12 Glue databases across the three zones — four domains per zone (CRM, customer, sales, operations). The Terraform for each database includes the Iceberg metadata parameters that enable Iceberg table format for all tables created in that database:

resource "aws_glue_catalog_database" "this" {
  name = "raw_crm"
  parameters = {
    "iceberg_enabled" = "true"
    "table_type"      = "ICEBERG"
    "format-version"  = "2"
  }
}

Format version 2 is the current Iceberg spec. It unlocks row-level deletes, which is required for GDPR compliance — when a user requests deletion, you can execute a targeted delete on the Iceberg table rather than rewriting entire Parquet partitions.

One thing that's easy to miss: Glue databases with Iceberg parameters set don't automatically create Iceberg tables. The database parameters act as defaults and metadata; actual table creation still happens via your ETL tooling (Glue jobs, Spark, Flink). What you get from Terraform is the catalog structure and the governance layer — databases, permissions, encryption settings — so that when the data engineering team writes their first Glue job, the infrastructure is already in place.

The Cost Model

When I presented this to the platform team's tech lead, the cost projection was what turned a "nice to have" into a "let's do this now."

For a 100TB lake:

Timeframe	Storage Tier	Monthly Cost
First 90 days	Standard	~$2,300
After 90 days	Archive Access	~$400
After 180 days	Deep Archive	~$100

That's roughly 80% savings once the bulk of the data ages past 90 days, and 95% savings at 180 days. The Intelligent-Tiering monitoring cost is $0.0025 per 1,000 objects — on a 100TB lake with typical Iceberg file sizes, that's a few dollars a month. Negligible.

The S3 Table Bucket metadata performance improvement compounds this. Faster query planning means less Athena scan time, which means lower query costs and faster results for analysts. The platform pays for itself in reduced query costs as the data volume grows.

Deployment Sequence

The deployment order is driven by dependencies: KMS must exist before S3 (bucket encryption needs the key ARN), and both must exist before Glue (catalog databases reference the bucket location).

In practice, across three environments (dev, nonprod, prod), the full deployment took about four hours. Most of that was Terragrunt apply time — the actual resource creation for each component is fast, but we ran plan, reviewed, applied, and verified before moving to the next environment.

One deployment note: the first time you run terragrunt plan on the Glue module in an account that hasn't had Glue configured before, you'll get an error about the Glue service-linked role not existing. Fix it by running aws iam create-service-linked-role --aws-service-name glue.amazonaws.com before the apply. It only needs to happen once per account.

What the Data Team Inherited

When we handed this over to the data engineering team, they had a fully provisioned catalog — three zones, twelve databases, Iceberg metadata configured, encryption enabled, Intelligent-Tiering active. They could start writing Glue jobs and creating tables immediately without worrying about storage configuration, access patterns, or cost optimization after the fact.

The Terraform modules are reusable. Adding a new domain (say, a finance domain across all three zones) is three database resource declarations and one pull request. The KMS key, bucket, and Intelligent-Tiering configuration don't change.

S3 Table Buckets are still relatively new, and the Terraform provider support came together in late 2024. If your team is planning an Iceberg migration and hasn't evaluated Table Buckets yet, the metadata performance gains and the cost trajectory make a strong case for starting there rather than retrofitting later.

Building out a data platform and figuring out the storage and catalog architecture? Get in touch — this kind of infrastructure design work is something I do regularly, whether you're starting from scratch or migrating an existing lake.

Zero-Downtime AWS Transit Gateway Hub-Spoke Migration

Glenn Gray — Sat, 04 Apr 2026 23:36:57 +0000

Originally published on graycloudarch.com.

The request came from the security team: they needed network-level access from the nonprod account to the dev account so a vulnerability scanner could reach internal services. Simple enough on the surface. In practice, it exposed a gap we'd been living with for months — and forced us to fix the network architecture we'd been deferring.

We had three standalone Transit Gateways: one in each workload account, dev, nonprod, and prod. Completely isolated from each other. No cross-account connectivity at all. The security scanner couldn't reach its targets, and adding more point-to-point peering connections to fix it would have made everything worse.

But the TGW isolation was only part of the problem. We also had no inspection of traffic crossing our network boundary. Egress from workload pods went straight to the internet with no filtering. Ingress came through per-account load balancers with no centralized enforcement point. As the platform scaled toward additional workload accounts, this pattern was going to get expensive and hard to reason about.

So we didn't just fix the TGW. We rebuilt the network foundation: a centralized Inspection VPC with a Network Firewall inline, a single hub Transit Gateway shared across all accounts, and centralized security tooling (GuardDuty, CloudTrail, Security Hub) aggregated in a dedicated Security account. Two maintenance windows, a few weeks of module work, and the platform went from fragmented per-account networking to a coherent hub-spoke design with full traffic inspection.

The Architecture We Were Replacing

Before the migration, each workload account was self-contained. It had its own TGW, its own internet gateway, its own NAT gateways. Security tooling ran independently in each account with no aggregation. The management account had no single-pane visibility into what was happening across the environment.

The cost of running this way was about $150/month in TGW charges plus duplicated NAT gateway charges in each account. Every new workload account would multiply this cost again and add another independent security configuration.

The Target: Inspection VPC + Hub Transit Gateway

The target was AWS Security Reference Architecture Pattern B: an Inspection VPC that sits between the internet and all workload VPCs. All internet traffic — ingress and egress — flows through this VPC and through a Network Firewall before reaching any workload account.

Egress path: workload pod → TGW → Inspection VPC TGW subnets → Network Firewall → NAT Gateway → IGW → internet.

Ingress path: internet → IGW → centralized ALB (public subnet) → Network Firewall → TGW → workload VPC → pod.

Nothing crosses the network boundary without passing through the firewall. Workload accounts carry no internet-facing infrastructure at all — no IGW, no NAT gateways, no public load balancers.

Phase 1: Module Changes

All Terraform work happened before scheduling any maintenance. The goal was to reach a state where the migration itself was just running pre-staged plan files in a specific sequence.

Transit Gateway: add a conditional create flag

The existing network module always created a TGW. We needed spoke accounts to declare the same module without spinning up their own gateway:

variable "create_transit_gateway" {
  description = "Whether to create a Transit Gateway (false for hub-spoke spokes)"
  type        = bool
  default     = true
}

resource "aws_ec2_transit_gateway" "this" {
  count       = var.create_transit_gateway ? 1 : 0
  description = var.tgw_description
}

output "transit_gateway_id" {
  value = var.create_transit_gateway ? aws_ec2_transit_gateway.this[0].id : null
}

default = true means existing configurations need no changes. The flag only flips to false after the spoke attachment is confirmed working.

New module: vpc-attachment

The vpc-attachment module handles the spoke side of the hub relationship: create the TGW attachment, associate it to the hub's route table, and add routes to every private route table in the spoke VPC pointing at the hub TGW.

resource "aws_ec2_transit_gateway_vpc_attachment" "this" {
  transit_gateway_id = var.transit_gateway_id
  vpc_id             = var.vpc_id
  subnet_ids         = var.subnet_ids

  tags = merge(var.tags, {
    Name = "${var.name}-hub-attachment"
  })
}

resource "aws_ec2_transit_gateway_route_table_association" "this" {
  transit_gateway_attachment_id  = aws_ec2_transit_gateway_vpc_attachment.this.id
  transit_gateway_route_table_id = var.transit_gateway_route_table_id
}

resource "aws_route" "to_hub_tgw" {
  for_each               = toset(var.vpc_route_table_ids)
  route_table_id         = each.value
  destination_cidr_block = "10.0.0.0/8"
  transit_gateway_id     = var.transit_gateway_id
}

The 10.0.0.0/8 supernet covers all workload and Inspection VPC CIDRs without maintaining per-prefix route entries. It also covers the Inspection VPC CIDR (10.100.0.0/20) — that's how return traffic from the centralized ALB finds its way back to pods in workload VPCs.

The Terragrunt config for a spoke account reads VPC details from the existing network dependency and hardcodes the hub TGW identifiers:

dependency "network" {
  config_path = "../network"
  mock_outputs = {
    vpc_id                  = "vpc-mockid"
    private_subnet_ids      = ["subnet-mock1"]
    private_route_table_ids = ["rtb-mock1"]
  }
}

inputs = {
  transit_gateway_id             = "tgw-xxxxx"   # hub TGW, documented in runbook
  transit_gateway_route_table_id = "tgw-rtb-xxxxx"
}

We hardcoded the hub TGW and route table IDs rather than using cross-account data sources. The alternative — reading TGW details from the Infrastructure account at plan time — requires cross-account state access and adds complexity that isn't worth it for values that change maybe once in the platform's lifetime.

Hub route tables: workload isolation by default

A key design decision: workload accounts should not route to each other directly. Dev should not reach nonprod; nonprod should not reach prod. The hub TGW enforces this through route table structure:

default-association-rt: all workload attachments associate here. The only route is 0.0.0.0/0 → inspection attachment. Workloads can reach the internet via the Inspection VPC, but cannot reach other workload VPCs.
default-propagation-rt: the inspection attachment propagates workload CIDRs here for return traffic routing.

Inter-account communication is opt-in: you add an explicit route table entry for a specific attachment pair. By default, the architecture prevents lateral movement across workload accounts.

Inspection VPC subnet layout

The Inspection VPC has three tiers with carefully constructed route tables that force traffic through the firewall in both directions:

The asymmetric route table design ensures the firewall sees every packet crossing the network boundary, regardless of direction. Traffic entering from the internet hits the firewall before reaching workloads. Traffic from workloads hits the firewall before reaching the internet.

Security baseline: convert to delegated admin model

GuardDuty and CloudTrail were running independently per account. We added enable_guardduty and enable_cloudtrail boolean variables to the security-baseline module so workload accounts could switch from standalone to member without touching the module invocation itself.

In the Security account, we deployed:

GuardDuty as delegated admin with organization-level auto-enrollment. EKS Protection and S3 Protection enabled. All findings from all accounts visible in a single dashboard.
CloudTrail organization trail writing to a cross-account S3 bucket. Log file validation and KMS encryption enabled. Per-account trails archived after the cutover — not deleted, in case historical log formats differed.
Security Hub with CIS AWS Foundations Benchmark and AWS Foundational Security Best Practices enabled across the full organization.

Phase 2: Two Maintenance Windows

Window 1: Deploy the hub (~45 minutes, low risk)

With no existing attachments and no workload traffic, deploying the hub infrastructure carried minimal risk. We applied the Infrastructure account TGW and Inspection VPC in a single window. The Network Firewall takes 5–10 minutes to reach READY state after creation — account for that in your timing.

At the end of this window: hub TGW running, Inspection VPC active, Network Firewall endpoints healthy in both AZs, centralized ALB deployed. Nothing attached yet. We documented the TGW ID and route table IDs in the runbook before scheduling window 2.

Window 2: Spoke cutover (~2 hours)

The key insight for keeping applications running: create the hub attachment before destroying the standalone TGW. While both exist simultaneously, traffic continues flowing through the standalone path. The actual cutover is updating routes to point at the hub — that's a single terragrunt apply, not the destruction of the old TGW.

T+0 — Accept RAM share. Infrastructure account shares the hub TGW via Resource Access Manager. Workload accounts accept the share invitation. Pure metadata operation; zero network impact.

T+15 — Deploy VPC attachments. Apply the vpc-attachment module in each workload account. At this point each spoke VPC has two routes for 10.0.0.0/8: the existing one pointing at the standalone TGW, and the new one pointing at the hub. With identical prefix lengths, traffic still flows through the standalone path. Rollback at this stage is terragrunt destroy on the attachment module — under five minutes.

T+30 — Verify routes and test cross-account connectivity. Confirm hub routes are present in every private route table:

aws ec2 describe-route-tables \
  --filters "Name=vpc-id,Values=vpc-xxxxx" \
  --query 'RouteTables[*].Routes[?DestinationCidrBlock==`10.0.0.0/8`]'

Then test actual cross-account traffic: connect from a dev instance to a service in the nonprod VPC. The hub TGW and Inspection VPC should route it correctly. This also validates that the firewall rule groups are permitting expected traffic — catch any rule issues here, before cutting over production.

T+45 — Migrate security tooling. Apply the updated security-baseline to each workload account. GuardDuty converts from standalone admin to member; findings flow to the Security account delegated admin. CloudTrail local trail disabled; organization trail confirmed logging events from the account. Zero network impact.

# Verify GuardDuty membership
aws guardduty get-administrator-account --detector-id <id>
# Returns the Security account as administrator

# Verify organization trail is capturing events
# Make an API call, wait ~15 minutes, check the Security account's S3 bucket
aws s3 ls s3://<org-trail-bucket>/AWSLogs/<account-id>/

T+60 — Set create_transit_gateway = false in each spoke. This is the cutover. Run terraform plan first and confirm it shows only the TGW and its attached resources being destroyed — nothing else. Apply dev first, watch the destruction complete, confirm application traffic is flowing through the hub. Then apply nonprod. About 3 minutes per account.

T+90 — Health checks and close. Spot-check API endpoints, database connectivity, anything that traverses the network. Confirm egress traffic is hitting the firewall logs in the Infrastructure account. The maintenance window closed at the 90-minute mark; actual work was done by T+75. We kept the window open for the last 15 minutes as a buffer.

The parallel attachment approach ensured there was never a moment where a workload account had no routing path. Even if the hub TGW had been misconfigured, traffic would have continued flowing through the standalone gateway until we chose to destroy it.

What We Ended Up With

One TGW in the Infrastructure account with three spoke attachments. Route tables that allow workload→internet traffic while preventing workload→workload lateral movement by default.

One Inspection VPC with Network Firewall endpoints in two AZs. All egress inspected against stateful domain filter rules and stateless port rules. All ingress from the centralized ALB inspected. Firewall policy updates apply to all workload accounts simultaneously — no per-account changes needed.

One centralized ALB in the Infrastructure account, routing to EKS target groups in workload accounts via cross-account IAM role assumption. Workload accounts carry no public-facing load balancers.

One security console in the Security account. GuardDuty findings from all accounts in a single dashboard. CloudTrail logs from every account in one S3 bucket. Security Hub compliance posture for the full organization visible in one place.

Cost went from roughly $150–200/month (standalone TGWs, per-account NAT, independent security tooling) to approximately $50/month (single hub TGW plus attachment hours, shared NAT in the Inspection VPC, delegated security services). Cost savings validated against AWS Cost Explorer after 30 days.

The original security scanner request — cross-account access from nonprod to dev — was live the same day. The compliance team had a single GuardDuty and Security Hub dashboard the same week.

More importantly: adding a new workload account to this architecture now takes about an hour. Create the VPC, deploy the vpc-attachment module pointing at the documented hub TGW ID, invite the new account as a GuardDuty and Security Hub member, apply the security-baseline with enable_guardduty = false. Every new account inherits the full inspection and security posture without any per-account configuration. That's the actual value of a hub-spoke design — not the one-time cost savings, but the fact that account seven is as well-secured and as easy to audit as account two.

Working through a multi-account network redesign, or building the inspection layer on top of an existing Transit Gateway setup? Get in touch — this is the kind of platform architecture I work on regularly.

DNS Validation: From 15 Steps to Zero

Glenn Gray — Sat, 04 Apr 2026 22:30:30 +0000

Originally published on graycloudarch.com.

You know what's the worst part of launching a new site?

SSL certificate validation.

Not creating the cert—that's one click in AWS ACM. It's the validation dance:

AWS gives you a CNAME record: _abc123extremely-long-string-here.graycloudarch.com
The value is equally ridiculous: _xyz789another-massive-string.acm-validations.aws.
You copy it (pray you don't miss a character)
Switch to Cloudflare (or Route 53, or wherever)
Paste it in
Wait 5-10 minutes
Refresh AWS console
Still pending...
Refresh again
Finally validated!

Now do it again for www.graycloudarch.com.

And then repeat the whole thing for your second domain.

This is "DNS hell."

There's a Better Way

Terraform can read AWS validation records and create them in Cloudflare automatically.

Zero copy-paste. Zero browser tab switching. Zero waiting and refreshing.

Here's the whole thing:

# Request certificate
resource "aws_acm_certificate" "site" {
  domain_name       = "graycloudarch.com"
  validation_method = "DNS"
  subject_alternative_names = ["www.graycloudarch.com"]
}

# Create validation records in Cloudflare
resource "cloudflare_record" "cert_validation" {
  for_each = {
    for dvo in aws_acm_certificate.site.domain_validation_options :
    dvo.domain_name => {
      name   = dvo.resource_record_name
      value  = dvo.resource_record_value
      type   = dvo.resource_record_type
    }
  }

  zone_id = data.cloudflare_zone.site.id
  name    = each.value.name
  value   = each.value.value
  type    = each.value.type
  proxied = false  # Critical - ACM validation breaks with proxy
}

# Wait for validation
resource "aws_acm_certificate_validation" "site" {
  certificate_arn = aws_acm_certificate.site.arn
  validation_record_fqdns = [
    for record in cloudflare_record.cert_validation : record.hostname
  ]
}

Run terraform apply. Go make coffee. Come back to a validated certificate.

The Magic: for_each

The key is this part:

for_each = {
  for dvo in aws_acm_certificate.site.domain_validation_options :
  dvo.domain_name => { name = dvo.resource_record_name, ... }
}

AWS generates validation records dynamically (one for apex domain, one for www). Terraform reads them, loops over them, and creates each one in Cloudflare.

You never see the records. You never copy anything. It just works.

What I Screwed Up

First time I ran this, ACM validation timed out after 30 minutes.

The problem:

proxied = true  # Wrong!

Cloudflare's proxy rewrites DNS responses. ACM's validation servers hit Cloudflare's IP instead of seeing your validation record.

The fix:

proxied = false  # Correct

DNS-only mode. No proxy. ACM validation works.

Cost me 30 minutes of debugging. Now it's in code so I never hit it again.

Why This Matters

I'm running two brands: graycloudarch.com and cloudpatterns.io.

Manual approach: 15 steps per domain = 30 steps total. 30 minutes minimum. High chance of typos.

Terraform approach: One terraform apply. 5 minutes to write the code (once), 10 minutes for AWS to validate. Then copy-paste the pattern for the second domain.

When I launch my third brand (and I will), it'll take 5 minutes and one terraform apply.

That's the bet: upfront automation for long-term velocity.

The Part People Miss

Most Terraform tutorials stop at requesting the certificate. They don't show you the validation loop or the waiting resource.

Without aws_acm_certificate_validation, Terraform exits immediately after creating the cert. It's still "Pending Validation" in AWS. When you try to use it in CloudFront, it fails.

You'd have to run terraform apply again later, after manually checking that validation completed.

That's not automation—that's just documentation.

The waiting resource makes it truly hands-off.

Scaling It

Adding a second domain is 10 lines of code:

resource "aws_acm_certificate" "cloudpatterns" {
  domain_name       = "cloudpatterns.io"
  validation_method = "DNS"
  subject_alternative_names = ["www.cloudpatterns.io"]
}

resource "cloudflare_record" "cloudpatterns_validation" {
  for_each = { /* same pattern */ }
  # ...
}

resource "aws_acm_certificate_validation" "cloudpatterns" {
  # ...
}

Same pattern, different names. No clicking. No switching between consoles. No remembering which validation record goes where.

The Real Win

It's not the time savings (though 30 minutes per deployment adds up).

It's the mental overhead.

Manual DNS configuration requires focus. "Did I copy the whole string? Did I add the trailing dot? Is it DNS-only mode?"

Terraform requires running one command. That's it.

I get my focus back. I can write this blog post while Terraform validates certificates.

Want the full code? It's not open source (yet), but if you're building something similar and want to talk through it, reach out.

Or if you just want to tell me I'm overthinking this and should've clicked through Cloudflare like a normal person, that's cool too.

Building Multi-Account AWS Infrastructure with Terraform and ECP

Glenn Gray — Sat, 04 Apr 2026 22:30:25 +0000

Originally published on graycloudarch.com.

After years of building AWS infrastructure at scale, I've learned that multi-account strategy isn't just about security—it's about organizational clarity and cost management.

At a large podcast hosting platform, we implemented an Enterprise Control Plane (ECP) pattern using Terraform to manage 20+ AWS accounts. Here's what I learned:

The Problem with Single-Account AWS

Most companies start with one AWS account. Everything lives together: dev, staging, prod, data pipelines, security tools. It works... until it doesn't.

Problems emerge:

Blast radius: A misconfigured dev resource can affect production
IAM complexity: Permission boundaries become impossible to manage
Cost allocation: Finance can't track spending by team or project
Compliance: Auditors want logical separation between environments

The ECP Pattern

Enterprise Control Plane is an architectural pattern for managing multiple AWS accounts as a unified platform:

Organization Structure: AWS Organizations with OUs (Organizational Units) for different environments and teams
Centralized Networking: Transit Gateway connecting all accounts through hub-and-spoke model
Security Baseline: Service Control Policies (SCPs) enforcing guardrails at the organization level
Infrastructure as Code: Terraform/Terragrunt managing everything from a central repository

Key Design Decisions

Account Boundaries:

Production accounts: Isolated per application/team
Non-prod accounts: Shared dev/staging to reduce overhead
Platform accounts: Separate accounts for logging, monitoring, security tools
Data accounts: Isolated for compliance and access control

Network Architecture:

Hub account with Transit Gateway
VPC peering only where absolutely necessary
Private subnet defaults for everything
Centralized egress through NAT Gateway in hub

Security Model:

SCPs prevent account-level misconfigurations
IAM roles for cross-account access (no shared credentials)
CloudTrail logs aggregated to security account
GuardDuty and Security Hub in every account

Terraform Structure

We use Terragrunt to manage configurations across accounts:

ecp-ou-structure/     # Organization and account management
ecp-network/          # Transit Gateway, VPCs, networking
ecp-security/         # Security baseline, SCPs, IAM
tf-live-aws-*/        # Application-specific infrastructure

Lessons Learned

Start with security: SCPs first, then networking, then workloads
Automate account creation: Manual account provisioning doesn't scale
Document the why: Every architectural decision needs context
Plan for day 2: Operations matter more than initial setup

Results

After implementing ECP:

Reduced security incident blast radius by 90%
Finance can now track costs by team and project
New environments deploy in hours, not days
Passed SOC2 audit with zero infrastructure findings

Multi-account AWS isn't just best practice—it's how you scale infrastructure beyond the startup phase.

Stop Manually Updating Jira After Every PR Merge

Glenn Gray — Wed, 25 Mar 2026 07:10:48 +0000

This post was originally published on graycloudarch.com.

You just merged a PR. Now you open Jira, find the ticket, paste the
PR link in a comment, transition the status to Done, and update the
deployed field. Five minutes. Twenty times a week. That's 1,700 minutes
per year per engineer --- nearly 30 hours of pure mechanical overhead.

And that's assuming you remember. On one team I worked with, we
audited the last three months of merged PRs. Thirty percent of tickets
had no update after merge. No comment, no transition, no link. The
ticket just sat in In Dev until someone noticed during sprint
review.

The fix is two GitHub Actions workflows and a shared composite
action. Here's exactly how to build it.

The Architecture

Two workflows, one shared extraction layer:

Workflow 1: Fires on PR creation --- posts a Jira link comment to the PR so reviewers can navigate directly to the ticket.
Workflow 2: Fires on PR merge to main --- posts a comment to the Jira ticket with the PR URL, commit SHA, and who merged it, then transitions the ticket to Done.

Both workflows need to find the Jira ticket ID. Instead of
duplicating that logic, we extract it into a composite action.

Step 1: Composite Action for Ticket Extraction

Create
.github/actions/extract-jira-ticket/action.yml.

The action checks four sources in priority order --- easiest to fix
first:

PR title (simplest for the developer to correct)
Commit messages
Branch name in standard format: PROJECT-123-description
Branch name with prefix: feat/PROJECT-123-description

::: {#cb1 .sourceCode}

name: Extract Jira Ticket
description: Extracts Jira ticket from PR title, commits, or branch name

inputs:
  jira-base-url:
    required: true
  jira-user-email:
    required: true
  jira-api-token:
    required: true

outputs:
  jira-key:
    value: ${{ steps.extract.outputs.jira_key }}
  found:
    value: ${{ steps.extract.outputs.found }}

runs:
  using: composite
  steps:
    - name: Extract ticket ID
      id: extract
      shell: bash
      run: |
        JIRA_KEY=""

        # Priority 1: PR title
        if [[ "${{ github.event.pull_request.title }}" =~ ([A-Z]+-[0-9]+) ]]; then
          JIRA_KEY="${BASH_REMATCH[1]}"
        fi

        # Priority 2: Branch name
        if [ -z "$JIRA_KEY" ]; then
          BRANCH="${{ github.head_ref }}"
          if [[ "$BRANCH" =~ ([A-Z]+-[0-9]+) ]]; then
            JIRA_KEY="${BASH_REMATCH[1]}"
          fi
        fi

        if [ -n "$JIRA_KEY" ]; then
          echo "jira_key=$JIRA_KEY" >> $GITHUB_OUTPUT
          echo "found=true" >> $GITHUB_OUTPUT
        else
          echo "found=false" >> $GITHUB_OUTPUT
        fi

:::

The regex [A-Z]+-[0-9]+ matches any Jira ticket format:
PROJ-1, IN-89, INFRA-1234. If you
have tickets with lowercase project keys, adjust accordingly.

Step 2: PR Creation Workflow

Create .github/workflows/link-jira-on-pr.yml.

This fires when a PR is opened and posts a formatted comment with the
Jira ticket link. If no ticket is found, it posts a warning so the
author knows to add one --- before review, not after.

::: {#cb2 .sourceCode}

name: Link Jira on PR
on:
  pull_request:
    types: [opened]

permissions:
  pull-requests: write

jobs:
  link-jira:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/extract-jira-ticket
        id: jira
        with:
          jira-base-url: ${{ secrets.JIRA_BASE_URL }}
          jira-user-email: ${{ secrets.JIRA_USER_EMAIL }}
          jira-api-token: ${{ secrets.JIRA_API_TOKEN }}

      - name: Post Jira link comment
        if: steps.jira.outputs.found == 'true'
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `📋 Jira: [${{ steps.jira.outputs.jira-key }}](${{ secrets.JIRA_BASE_URL }}/browse/${{ steps.jira.outputs.jira-key }})`
            })

      - name: Warn if no ticket found
        if: steps.jira.outputs.found == 'false'
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: '⚠️ No Jira ticket found. Add a ticket ID to the PR title (e.g., `PROJ-123: Your title`).'
            })

:::

The warning step matters. It creates a feedback loop that trains the
team to include ticket IDs upfront. Within a few weeks, the warning
fires rarely.

Step 3: PR Merge Workflow

Create .github/workflows/update-jira-on-merge.yml.

This fires when a PR is closed against main. The
if: github.event.pull_request.merged == true guard is
important --- the closed event also fires for PRs that are
closed without merging.

::: {#cb3 .sourceCode}

name: Update Jira on Merge
on:
  pull_request:
    types: [closed]
    branches: [main]

jobs:
  update-jira:
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/extract-jira-ticket
        id: jira
        with:
          jira-base-url: ${{ secrets.JIRA_BASE_URL }}
          jira-user-email: ${{ secrets.JIRA_USER_EMAIL }}
          jira-api-token: ${{ secrets.JIRA_API_TOKEN }}

      - name: Post merge comment to Jira
        if: steps.jira.outputs.found == 'true'
        run: |
          HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}"
            -u "${{ secrets.JIRA_USER_EMAIL }}:${{ secrets.JIRA_API_TOKEN }}"
            -H "Content-Type: application/json"
            -X POST "${{ secrets.JIRA_BASE_URL }}/rest/api/2/issue/${{ steps.jira.outputs.jira-key }}/comment"
            -d "{\"body\": \"PR merged: #${{ github.event.pull_request.number }} ${{ github.event.pull_request.html_url }}\nCommit: ${{ github.sha }}\nBy: ${{ github.event.pull_request.merged_by.login }}\"}")

          echo "Jira comment HTTP status: $HTTP_STATUS"
          [ "$HTTP_STATUS" -eq 201 ] && echo "✅ Comment posted" || echo "⚠️ Comment failed (non-critical)"

      - name: Transition ticket to Done
        if: steps.jira.outputs.found == 'true'
        run: |
          TRANSITION_ID="${{ secrets.JIRA_DONE_TRANSITION_ID }}"
          [ -z "$TRANSITION_ID" ] && echo "No transition ID configured, skipping" && exit 0

          curl -s -u "${{ secrets.JIRA_USER_EMAIL }}:${{ secrets.JIRA_API_TOKEN }}"
            -H "Content-Type: application/json"
            -X POST "${{ secrets.JIRA_BASE_URL }}/rest/api/2/issue/${{ steps.jira.outputs.jira-key }}/transitions"
            -d "{\"transition\": {\"id\": \"$TRANSITION_ID\"}}"
          echo "✅ Transitioned to Done"

:::

The comment step uses an HTTP status check rather than relying on
curl's exit code. A failed comment doesn't fail the job --- the PR already
merged, and a missing notification shouldn't generate noise in CI. The
transition step is fully optional: if
JIRA_DONE_TRANSITION_ID isn't set, it skips silently. This
lets you start with just comments and add transitions once you've
verified the workflow runs cleanly.

Finding Your Transition IDs

Transition IDs are project-specific. There's no universal "Done" ID.
Run this against any ticket in your project to find yours:

::: {#cb4 .sourceCode}

curl -u "$JIRA_EMAIL:$JIRA_API_TOKEN"
  "$JIRA_BASE_URL/rest/api/2/issue/$TICKET_KEY/transitions"
  | jq -r '.transitions[] | "ID: \(.id) | \(.name)"'

:::

Example output:

ID: 91 | Done
ID: 31 | In Review
ID: 21 | In Progress

Set the Done ID as JIRA_DONE_TRANSITION_ID in your
repository secrets.

A Note on Jira API Versions

Use the v2 API: /rest/api/2/. Some teams try v3 and get
silent empty responses --- {"errorMessages":[],"errors":{}} ---
that look exactly like auth failures. It's not auth. The v3 request body
format changed, and error handling is poor. v2 is stable,
well-documented, and works consistently.

Required Secrets

Add these to your GitHub repository secrets:

Secret Value

JIRA_BASE_URL https://yourorg.atlassian.net
JIRA_USER_EMAIL The email address tied to your API token
JIRA_API_TOKEN Generate at id.atlassian.com → Security → API tokens
JIRA_DONE_TRANSITION_ID Optional --- from the transitions API call above

For org-wide rollout, set these as organization secrets and restrict
to relevant repositories.

Results

After rolling this out across a team of eight engineers:

Zero manual Jira updates after merge
Forgotten ticket updates dropped from 30% to 0%
Roughly 1,700 minutes per year recovered per engineer
Every merged PR has a complete audit trail: PR number, URL, commit SHA, who merged it

The composite action pattern also means when you need to extend this
--- adding a Slack notification on merge, posting to Confluence --- you
extend one file, not two.

If you're building out automation like this across your engineering
platform and want a second opinion on the design, I'm available for advisory engagements.

How I Manage Claude Code Context Across 20+ Repositories

Glenn Gray — Fri, 20 Mar 2026 14:52:42 +0000

This post was originally published on graycloudarch.com.

Three months ago I was re-explaining my Terragrunt state backend to
Claude for the third time in a week. Different session, same repo, same
repo I'd worked in the session before. Claude had no idea I was even in
the same project.

I run Claude Code daily across a 6-account AWS platform monorepo, a
personal consulting site, homelab infrastructure, and a handful of side
projects. Every session started with the same five minutes of "here's
the project, here are the conventions, here's the Jira workflow" --- and
still ended with Claude suggesting patterns that didn't fit the
environment, because I'd inevitably forgotten to mention something.

After three months of broken symlinks and abandoned experiments, I
landed on a three-tier context hierarchy that loads the right context
automatically depending on which directory I'm working in --- and I manage
all of it from a single dotfiles repo.

The Problem with Single-File Context

Claude Code loads CLAUDE.md from the current directory
(and parent directories, walking up to
~/.claude/CLAUDE.md). Most teams start with one file and
put everything in it.

That breaks down quickly:

Global preferences get mixed with project specifics. Your "use snake_case for variable names" preference shouldn't live next to your Terraform state bucket configuration.
Credentials and account IDs end up in files you accidentally commit. Put AWS account IDs in a shared CLAUDE.md, and someone will eventually push it.
You can't share patterns across repos without duplication. Every new repo gets a fresh copy of the same conventions, and updates never propagate.
Multi-employer context creates conflicts. Your consulting client's Jira workflow shouldn't contaminate your personal project sessions.

My first attempt at fixing this was a shared scripts directory. The
three-tier hierarchy came later, after I figured out what was actually
wrong with the simpler approach.

My First Attempt: A Shared Scripts Directory

Before landing on the three-tier system, I built something more
obvious: a ~/shared-claude-infra/ directory containing a
setup-project.sh script that initialized
.claude/ context for each new repo.

The script created the directory structure and symlinked a
rules/shared/ folder back to the shared repo:

::: {#cb1 .sourceCode}

mkdir -p "$PROJECT_DIR/.claude/rules"
ln -s ~/shared-claude-infra/rules "$PROJECT_DIR/.claude/rules/shared"

:::

This worked for the first two repos I configured. Then the problems
compounded:

Manual per-project setup. Every new repo required running the script. Miss one, and that repo has no shared context.
Two repos to maintain. The shared infrastructure lived in its own git repo, separate from dotfiles. Two places to update when conventions changed, and they drifted.
Nested symlinks instead of directory-level symlinks. The rules/shared symlink lived deep inside the project's .claude/ tree. When the target moved, every project that had run the script got a broken symlink --- silently.
Hardcoded paths that drifted. The script referenced workspace paths from three months earlier. My actual directory layout had changed; the script still pointed at the old locations.

When I eventually deleted the shared directory, a quick
find confirmed broken symlinks scattered across every repo
that had run the setup script. The approach was inherently fragile
because it depended on every machine, every repo, and every workspace
path staying synchronized manually.

The fix isn't a smarter script. It's inverting the relationship:
instead of a script that runs once per project, use dotfiles that wire
context automatically based on what directories exist.

**Working through something similar?** I advise platform teams on AWS infrastructure â€" multi-account architecture, Transit Gateway, EKS, and Terraform IaC. [Let's talk.](https://graycloudarch.com/#contact)

The Three-Tier Hierarchy

~/.claude/CLAUDE.md              ← Global: preferences, style, git workflow
~/work/{employer}/.claude/       ← Org: team structure, AWS accounts, Jira workflow
~/work/{employer}/{repo}/.claude/ ← Project: repo architecture, active tickets

Claude Code walks up the directory tree loading
CLAUDE.md files at each level. Each tier handles a specific
scope:

Global tier (~/.claude/): Everything
that applies across all work --- communication style, git commit format,
PR description templates, universal infrastructure patterns. No
credentials, no account IDs, nothing employer-specific.

Org tier (~/work/{employer}/.claude/):
Team structure, Jira project keys, AWS account layout, CI/CD pipeline
conventions. Sensitive patterns (account IDs, VPC IDs, state bucket
names) go in gitignored files within this directory. Reusable patterns
(CI/CD templates, AWS patterns without specifics) go in committed
files.

Project tier
(~/work/{employer}/{repo}/.claude/): Architecture decisions
for this specific repo, active tickets, ongoing work state. Always
gitignored --- this is ephemeral working context that changes
frequently.

Implementation: Symlinks from Dotfiles

The hierarchy only works if it's consistent across machines. I manage
all context files from a dotfiles repo using symlinks:

dotfiles/claude/
├── global/          → symlinked to ~/.claude/
├── {employer}/      → symlinked to ~/work/{employer}/.claude/
└── {personal}/      → symlinked to ~/personal/.claude/

install.sh wires these automatically:

::: {#cb4 .sourceCode}

# Global context
ln -sf "$DOTFILES/claude/global" "$HOME/.claude"

# Per-employer context
for employer in "${EMPLOYERS[@]}"; do
  WORK_DIR="$HOME/work/$employer"
  if [ -d "$WORK_DIR" ]; then
    ln -sf "$DOTFILES/claude/$employer" "$WORK_DIR/.claude"
  fi
done

:::

Any machine that runs install.sh gets the same context
hierarchy. Changes committed to dotfiles propagate immediately.

What Each Level Contains

Global (`~/.claude/`)

~/.claude/
├── CLAUDE.md          # Preferences, active work summary
└── rules/
    ├── git-workflow.md
    ├── pr-patterns.md
    ├── infrastructure.md
    └── context-management.md

CLAUDE.md is short --- preferences and a pointer to where
active work lives. The heavy lifting goes in rules/ files
that Claude loads as supplementary context.

::: {#cb6 .sourceCode}

## Communication Style
- Be direct and technical — I understand infrastructure concepts
- Explain the "why" behind decisions
- Provide specific file paths and line numbers

## Git Workflow
- Branch format: feat/TICKET-123-description
- Commit format: [TICKET-123] Brief summary\n\nWhy this change...
- Never add Co-Authored-By trailers

:::

Org Level (`~/work/{employer}/.claude/`)

~/work/{employer}/.claude/
├── {EMPLOYER}.md              # Team structure, Jira workflow — committed
└── rules/
    ├── cicd-patterns.md       # CI/CD conventions — committed
    ├── aws-patterns.md        # Account IDs, VPC IDs — GITIGNORED
    └── terraform-patterns.md  # State config, module paths — GITIGNORED

The privacy split matters. cicd-patterns.md contains
reusable GitHub Actions patterns --- fine to commit.
aws-patterns.md contains actual account IDs --- stays
local.

A typical employer context file:

::: {#cb8 .sourceCode}

## Team Structure
- Platform team: 5 engineers, all in Jira project IN
- AWS accounts: dev, nonprod, prod (+ 3 infra accounts network, security, management)
- Monorepo: ~/work/{employer}/iac — Terragrunt, 78 components

## Jira Workflow
- IN project (infrastructure): transition IDs 3=In Dev, 8=Needs Review, 9=Done
- Prefix all commits: [IT-XXX]
- API: REST v2 only — v3 silently returns empty responses

## CI/CD
- GitHub Actions with OIDC to AWS (no long-lived credentials)
- PR requires: terraform plan output posted as comment
- Merge to main triggers auto-deploy to nonprod

:::

The gitignored aws-patterns.md contains account IDs and
specific ARNs that Claude needs for generating Terraform configurations
accurately but shouldn't be committed anywhere.

Project Level (`~/work/{employer}/{repo}/.claude/`)

Project context is ephemeral and always gitignored. It's the working
memory for an ongoing effort:

::: {#cb9 .sourceCode}

## Current State
- Branch: feat/IT-89-my-app-dev-ecr
- Active ticket: IT-89 — ECR in dev
- Next: IT-90 — ECS task definition

## Architecture Decisions
- ECR in dev only; cross-account pull policies for nonprod and prod
- Mutable tags in dev, immutable in nonprod/prod
- KMS key per environment, not per repository

## Blockers
- Waiting on network DNS zone creation before cutover can proceed

:::

I update this file at the end of each session with current state so
the next session loads instantly without re-explaining where things
are.

The Privacy Model

The critical insight is that context files need two categories:
committed (shareable) and local-only (sensitive).

Content Location Committed?

Personal preferences ~/.claude/CLAUDE.md ✅
Git workflow rules ~/.claude/rules/git-workflow.md ✅
Team structure {employer}/.claude/{EMPLOYER}.md ✅ sanitized
CI/CD patterns {employer}/.claude/rules/cicd-patterns.md ✅
AWS account IDs {employer}/.claude/rules/aws-patterns.md ❌ gitignored
VPC IDs, state config {employer}/.claude/rules/terraform-patterns.md ❌ gitignored
Active ticket state {repo}/.claude/OVERRIDES.md ❌ gitignored

The .gitignore at the dotfiles level handles this
automatically by ignoring **/aws-patterns.md and
**/terraform-patterns.md across all employer
directories.

Custom Commands

Beyond context files, Claude Code supports custom
/commands --- reusable prompts stored as markdown files:

~/.claude/commands/
├── checkpoint.md       # Create context snapshot
├── sync-work.md        # Update active work status
└── pr-ready.md         # Generate PR description

A command file is just the prompt Claude should execute:

::: {#cb11 .sourceCode}

# checkpoint.md
Create a context checkpoint. Read the current git status across active repos,
summarize open PRs and their status, list active tickets with their current
state, and write a structured summary to ~/.claude/local.md. Include any
blocking issues and the next planned action.

:::

Commands at the global level are available everywhere. Org-level
commands handle employer-specific workflows like Jira transitions.

What This Solves in Practice

Before this system: every new Claude session started with "here's the
project, here are the conventions, here's where things are." Five
minutes of ramp-up, inconsistent outputs because I'd forget to mention
something.

After: I cd into a repo and Claude already knows the
Jira workflow, the AWS account structure, the naming conventions, and
where the active work stands. When I start a session mid-ticket, the
project-level context tells Claude exactly what was in progress.

The bigger payoff is consistency. When Claude generates Terraform, it
generates it with the correct state backend. When it writes commit
messages, they follow the format reviewers expect. When it suggests
architecture, it fits the actual account model rather than a generic AWS
example.

Starting Point

If you're starting from scratch, work tier by tier:

Create ~/.claude/CLAUDE.md with your communication preferences and git conventions.
Add a rules/ directory with patterns you want loaded consistently.
Create an org-level directory when you start working with a specific employer or major project.
Add project-level context when you start a multi-session effort.

Don't try to build the whole system at once. The global tier alone
eliminates most of the per-session ramp-up. The org and project tiers
pay off as work gets more complex.

The thing that surprised me most wasn't the time saved on ramp-up. It
was how much the output quality improved. When Claude knows the actual
state backend, the actual account IDs, the actual PR format your
reviewers expect --- the suggestions it makes fit your environment. That
gap between "technically correct" and "actually usable" is where most of
the friction in AI-assisted infrastructure work lives. The context
hierarchy is mostly just closing that gap.

If you're setting up Claude Code for a platform team and want to talk
through the context design, I do advisory
engagements for teams getting serious about AI tooling in their
infrastructure workflow.

How I Manage Claude Code Context Across 20+ Repositories

Glenn Gray — Fri, 20 Mar 2026 06:57:13 +0000

This post was originally published on graycloudarch.com.

The Problem with Single-File Context

Claude Code loads CLAUDE.md from the current directory
(and parent directories, walking up to
~/.claude/CLAUDE.md). Most teams start with one file and
put everything in it.

That breaks down quickly:

Global preferences get mixed with project specifics. Your "use snake_case for variable names" preference shouldn't live next to your Terraform state bucket configuration.
Credentials and account IDs end up in files you accidentally commit. Put AWS account IDs in a shared CLAUDE.md, and someone will eventually push it.
You can't share patterns across repos without duplication. Every new repo gets a fresh copy of the same conventions, and updates never propagate.
Multi-employer context creates conflicts. Your consulting client's Jira workflow shouldn't contaminate your personal project sessions.

My first attempt at fixing this was a shared scripts directory. The
three-tier hierarchy came later, after I figured out what was actually
wrong with the simpler approach.

My First Attempt: A Shared Scripts Directory

The script created the directory structure and symlinked a
rules/shared/ folder back to the shared repo:

mkdir -p "$PROJECT_DIR/.claude/rules"
ln -s ~/shared-claude-infra/rules "$PROJECT_DIR/.claude/rules/shared"

This worked for the first two repos I configured. Then the problems
compounded:

Manual per-project setup. Every new repo required running the script. Miss one, and that repo has no shared context.
Two repos to maintain. The shared infrastructure lived in its own git repo, separate from dotfiles. Two places to update when conventions changed, and they drifted.
Nested symlinks instead of directory-level symlinks. The rules/shared symlink lived deep inside the project's .claude/ tree. When the target moved, every project that had run the script got a broken symlink --- silently.
Hardcoded paths that drifted. The script referenced workspace paths from three months earlier. My actual directory layout had changed; the script still pointed at the old locations.

The fix isn't a smarter script. It's inverting the relationship:
instead of a script that runs once per project, use dotfiles that wire
context automatically based on what directories exist.

The Three-Tier Hierarchy

~/.claude/CLAUDE.md              ← Global: preferences, style, git workflow
~/work/{employer}/.claude/       ← Org: team structure, AWS accounts, Jira workflow
~/work/{employer}/{repo}/.claude/ ← Project: repo architecture, active tickets

Claude Code walks up the directory tree loading
CLAUDE.md files at each level. Each tier handles a specific
scope:

Implementation: Symlinks from Dotfiles

The hierarchy only works if it's consistent across machines. I manage
all context files from a dotfiles repo using symlinks:

dotfiles/claude/
├── global/          → symlinked to ~/.claude/
├── {employer}/      → symlinked to ~/work/{employer}/.claude/
└── {personal}/      → symlinked to ~/personal/.claude/

install.sh wires these automatically:

# Global context
ln -sf "$DOTFILES/claude/global" "$HOME/.claude"

# Per-employer context
for employer in "${EMPLOYERS[@]}"; do
  WORK_DIR="$HOME/work/$employer"
  if [ -d "$WORK_DIR" ]; then
    ln -sf "$DOTFILES/claude/$employer" "$WORK_DIR/.claude"
  fi
done

Any machine that runs install.sh gets the same context
hierarchy. Changes committed to dotfiles propagate immediately.

What Each Level Contains

Global (`~/.claude/`)

~/.claude/
├── CLAUDE.md          # Preferences, active work summary
└── rules/
    ├── git-workflow.md
    ├── pr-patterns.md
    ├── infrastructure.md
    └── context-management.md

CLAUDE.md is short --- preferences and a pointer to where
active work lives. The heavy lifting goes in rules/ files
that Claude loads as supplementary context.

## Communication Style
- Be direct and technical — I understand infrastructure concepts
- Explain the "why" behind decisions
- Provide specific file paths and line numbers

## Git Workflow
- Branch format: feat/TICKET-123-description
- Commit format: [TICKET-123] Brief summary\n\nWhy this change...
- Never add Co-Authored-By trailers

Org Level (`~/work/{employer}/.claude/`)

~/work/{employer}/.claude/
├── {EMPLOYER}.md              # Team structure, Jira workflow — committed
└── rules/
    ├── cicd-patterns.md       # CI/CD conventions — committed
    ├── aws-patterns.md        # Account IDs, VPC IDs — GITIGNORED
    └── terraform-patterns.md  # State config, module paths — GITIGNORED

The privacy split matters. cicd-patterns.md contains
reusable GitHub Actions patterns --- fine to commit.
aws-patterns.md contains actual account IDs --- stays
local.

A typical employer context file:

## Team Structure
- Platform team: 5 engineers, all in Jira project IN
- AWS accounts: dev, nonprod, prod (+ 3 infra accounts network, security, management)
- Monorepo: ~/work/{employer}/iac — Terragrunt, 78 components

## Jira Workflow
- IN project (infrastructure): transition IDs 3=In Dev, 8=Needs Review, 9=Done
- Prefix all commits: [IT-XXX]
- API: REST v2 only — v3 silently returns empty responses

## CI/CD
- GitHub Actions with OIDC to AWS (no long-lived credentials)
- PR requires: terraform plan output posted as comment
- Merge to main triggers auto-deploy to nonprod

The gitignored aws-patterns.md contains account IDs and
specific ARNs that Claude needs for generating Terraform configurations
accurately but shouldn't be committed anywhere.

Project Level (`~/work/{employer}/{repo}/.claude/`)

Project context is ephemeral and always gitignored. It's the working
memory for an ongoing effort:

## Current State
- Branch: feat/IT-89-my-app-dev-ecr
- Active ticket: IT-89 — ECR in dev
- Next: IT-90 — ECS task definition

## Architecture Decisions
- ECR in dev only; cross-account pull policies for nonprod and prod
- Mutable tags in dev, immutable in nonprod/prod
- KMS key per environment, not per repository

## Blockers
- Waiting on network DNS zone creation before cutover can proceed

I update this file at the end of each session with current state so
the next session loads instantly without re-explaining where things
are.

The Privacy Model

The critical insight is that context files need two categories:
committed (shareable) and local-only (sensitive).

Content Location Committed?

The .gitignore at the dotfiles level handles this
automatically by ignoring **/aws-patterns.md and
**/terraform-patterns.md across all employer
directories.

Custom Commands

Beyond context files, Claude Code supports custom
/commands --- reusable prompts stored as markdown files:

~/.claude/commands/
├── checkpoint.md       # Create context snapshot
├── sync-work.md        # Update active work status
└── pr-ready.md         # Generate PR description

A command file is just the prompt Claude should execute:

# checkpoint.md
Create a context checkpoint. Read the current git status across active repos,
summarize open PRs and their status, list active tickets with their current
state, and write a structured summary to ~/.claude/local.md. Include any
blocking issues and the next planned action.

Commands at the global level are available everywhere. Org-level
commands handle employer-specific workflows like Jira transitions.

What This Solves in Practice

Starting Point

If you're starting from scratch, work tier by tier:

Create ~/.claude/CLAUDE.md with your communication preferences and git conventions.
Add a rules/ directory with patterns you want loaded consistently.
Create an org-level directory when you start working with a specific employer or major project.
Add project-level context when you start a multi-session effort.

Don't try to build the whole system at once. The global tier alone
eliminates most of the per-session ramp-up. The org and project tiers
pay off as work gets more complex.

Great design, and easy to follow.

Glenn Gray — Wed, 18 Mar 2026 16:23:46 +0000

How to Implement AWS Network Firewall in a Multi-Account Architecture Using Transit Gateway

Cristhian Becerra ・ Oct 13 '25

#english #aws #networking #cybersecurity

Building Automated AWS Permission Testing Infrastructure for CI/CD

Glenn Gray — Wed, 18 Mar 2026 07:14:50 +0000

This post was originally published on graycloudarch.com.

I deployed a permission set for our data engineers five times before
it worked correctly.

The first deployment: S3 reads worked, Glue Data Catalog reads
worked. Athena queries failed --- the query engine needs KMS decrypt
through a service principal, and I'd missed the
kms:ViaService condition. Second deployment: Athena worked.
EMR Serverless job submission failed --- missing
iam:PassRole. Third deployment: EMR submission worked. Job
execution failed --- missing permissions on the EMR Serverless execution
role boundary. I kept deploying, engineers kept getting blocked, I kept
opening tickets.

Five iterations. Two weeks. Every failure meant a data engineer
opened a ticket instead of running their job.

The problem wasn't that IAM is complicated --- it is, but that's
expected. The problem was that I had no way to catch these issues before
deploying to the account where real engineers were trying to do real
work. Every bug was a production bug.

The "Access Denied" Debugging Loop

Here's what the reactive debugging cycle looks like from the
inside.

Engineer opens a ticket:
AccessDeniedException: User is not authorized to perform: s3:GetObject.
I add s3:GetObject to the permission set. Next day:
AccessDeniedException: s3:PutObject. I add
s3:PutObject. Day after: write succeeds but cleanup fails ---
s3:DeleteObject. At this point I've done four deployment
cycles and two days of work to get S3 read/write/delete working. If I'd
just added s3:* I'd be done, but that violates
least-privilege and opens the raw zone to write access, which we
explicitly don't want.

The deeper issue is that individual services don't fail atomically.
Athena requires athena:StartQueryExecution and
athena:GetQueryResults and
athena:GetQueryExecution, but it also requires KMS decrypt
through the Athena service principal to read encrypted S3 results. That
last piece isn't in the Athena docs --- you find it by failing in
production.

I wanted a way to find it before deploying.

What I Built

The testing framework has four components: per-persona permission set
templates, a Bash test library, per-service test scripts, and a GitHub
Actions workflow that runs everything on pull requests.

┌─────────────────────────────────────────────────┐
│  GitHub Pull Request (Permission Set Changes)   │
└───────────────────┬─────────────────────────────┘
                    │
         ┌──────────▼──────────┐
         │  CI/CD Workflow     │
         │  (GitHub Actions)   │
         └──────────┬──────────┘
                    │
    ┌───────────────┼───────────────┐
    ▼               ▼               ▼
┌───────┐      ┌──────────┐   ┌──────────┐
│ S3    │      │  Glue    │   │ Athena   │
│ Tests │      │  Tests   │   │  Tests   │
└───────┘      └──────────┘   └──────────┘
                    │
         ┌──────────▼──────────┐
         │  Test Report        │
         │  (Posted to PR)     │
         └─────────────────────┘

The workflow triggers on any pull request that modifies the
identity-center Terraform directory. Tests run against real AWS accounts
--- dev and nonprod --- using test credentials provisioned for that purpose.
Results post as a PR comment before anyone approves the change.

Phase 1: Pre-Validated Templates

Before I wrote a single test, I needed a starting point for
permission sets that captured the patterns I'd learned the hard way.
Templates that handle the non-obvious pieces --- zone-scoped S3 access,
KMS conditions tied to specific services, explicit denies for
destructive operations.

The AnalystAccess template is representative. Analysts
get read-only access to the curated zone of the data lake, Athena query
execution in the primary workgroup, and KMS decrypt --- but only when the
decrypt request originates from S3 or Athena, not from arbitrary API
calls:

inline_policy = jsonencode({
  Version = "2012-10-17"
  Statement = [
    {
      Sid    = "GlueCatalogReadOnly"
      Effect = "Allow"
      Action = ["glue:GetDatabase", "glue:GetTable", "glue:GetPartitions", "glue:SearchTables"]
      Resource = [
        "arn:aws:glue:*:*:catalog",
        "arn:aws:glue:*:*:database/curated_*",
        "arn:aws:glue:*:*:table/curated_*/*"
      ]
    },
    {
      Sid    = "S3CuratedReadOnly"
      Effect = "Allow"
      Action = ["s3:GetObject", "s3:ListBucket"]
      Resource = ["arn:aws:s3:::lake-bucket-*/curated/*", "arn:aws:s3:::lake-bucket-*"]
      Condition = { StringLike = { "s3:prefix" = ["curated/*"] } }
    },
    {
      Sid    = "AthenaQueryExecution"
      Effect = "Allow"
      Action = ["athena:StartQueryExecution", "athena:GetQueryExecution", "athena:GetQueryResults", "athena:StopQueryExecution"]
      Resource = "arn:aws:athena:*:*:workgroup/primary"
    },
    {
      Sid    = "KMSDecryptViaSvc"
      Effect = "Allow"
      Action = ["kms:Decrypt", "kms:DescribeKey"]
      Resource = "arn:aws:kms:*:*:key/*"
      Condition = {
        StringEquals = { "kms:ViaService" = ["s3.us-east-1.amazonaws.com", "athena.us-east-1.amazonaws.com"] }
      }
    },
    {
      Sid    = "DenyDestructiveOps"
      Effect = "Deny"
      Action = ["s3:DeleteObject", "s3:DeleteBucket", "glue:DeleteDatabase", "glue:DeleteTable"]
      Resource = "*"
    }
  ]
})

The kms:ViaService condition is the piece that took five
production failures to discover. KMS decrypt without that condition
allows an analyst to call kms:Decrypt directly from their
shell, which is not what we want. The condition locks decrypt to
requests that pass through S3 or Athena specifically.

The explicit deny block matters too. Without it, if someone later
grants broader S3 permissions to this persona for a different reason,
the curated zone protection evaporates. The deny creates a hard floor
regardless of what else gets added.

Phase 2: The Test Framework

I chose Bash over Python or a proper test framework deliberately. The
tests run in CI with no dependencies beyond the AWS CLI --- no package
installs, no virtual environments, no version pinning of test libraries.
The machines running these tests already have the AWS CLI.

The core library in lib/test-framework.sh:

::: {#cb3 .sourceCode}

declare -a TESTS_PASSED=()
declare -a TESTS_FAILED=()

run_test() {
  local test_name="$1"
  local test_command="$2"
  local description="$3"

  if eval "$test_command" &>/dev/null; then
    TESTS_PASSED+=("$test_name")
    echo "  ✅ PASS: $test_name"
  else
    TESTS_FAILED+=("$test_name")
    echo "  ❌ FAIL: $test_name"
  fi
}

generate_text_report() {
  echo "Total: $((${#TESTS_PASSED[@]} + ${#TESTS_FAILED[@]}))"
  echo "Passed: ${#TESTS_PASSED[@]}"
  echo "Failed: ${#TESTS_FAILED[@]}"
  [ ${#TESTS_FAILED[@]} -gt 0 ] && printf '  - %s\n' "${TESTS_FAILED[@]}"
}

:::

The most important design decision in the test scripts is testing
denials as carefully as allowances. Testing only what should succeed
tells you the permission set isn't obviously broken. Testing what should
fail tells you it's not accidentally too permissive.

::: {#cb4 .sourceCode}

# Test what should succeed
run_test "s3-list-curated"
  "aws s3 ls s3://lake-bucket-dev/curated/"
  "Analyst can list curated zone"

# Test what should fail (negative test)
run_test "s3-write-denied"
  "! aws s3 cp /tmp/test.txt s3://lake-bucket-dev/curated/test.txt 2>&1 | grep -q 'AccessDenied'"
  "Analyst cannot write to curated zone"

run_test "s3-raw-zone-denied"
  "! aws s3 ls s3://lake-bucket-dev/raw/ 2>&1 | grep -q 'AccessDenied'"
  "Analyst cannot access raw zone"

:::

Beyond service-level tests, I run persona tests that simulate
end-to-end workflows. An analyst's workflow isn't "call S3, then call
Athena separately" --- it's "run an Athena query that reads encrypted S3
data and writes results to the query results bucket." That integration
test catches failures that individual service tests miss. The original
five-iteration DataPlatformAccess failure? An individual S3 test would
have passed. A persona test running an actual Athena query against the
encrypted lake would have caught the KMS gap.

Phase 3: CI/CD Integration

The GitHub Actions workflow triggers on pull requests that touch the
identity-center Terraform directory, runs tests in a matrix against dev
and nonprod, and posts a summary comment to the PR.

::: {#cb5 .sourceCode}

on:
  pull_request:
    paths:
      - 'common/modules/identity-center/**/*.tf'

permissions:
  contents: read
  id-token: write
  pull-requests: write

jobs:
  test-permissions:
    strategy:
      matrix:
        environment: [workloads-dev, workloads-nonprod]
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::${{ matrix.environment.account }}:role/github-actions-role
      - run: ./scripts/test-permissions/run-permission-tests.sh --persona analyst

:::

The id-token: write permission is required for OIDC
authentication to AWS --- the workflow assumes a role in each account
rather than using long-lived credentials in GitHub Secrets. This is the
right pattern: credentials rotate automatically, and there's no secret
to rotate manually or accidentally expose.

The PR comment posts the full test output with pass/fail counts per
persona per account. A reviewer can look at the comment and immediately
see whether the permission change has test coverage and whether the
tests pass.

Three Things I Learned the Hard Way

First: test KMS decryption through each service separately.
kms:Decrypt via S3 and kms:Decrypt via Athena
are different IAM evaluation paths even though they're the same API
call. A test that puts an object and gets it back via S3 directly won't
catch a broken Athena KMS path.

Second: negative tests matter as much as positive ones. Before I had
the test framework, every permission set I wrote was tested only for
what it should allow. I had no systematic check that it didn't allow
more. The denial tests are what give security reviewers confidence.

Third: persona tests catch failures that service tests miss.
Individual service tests are fast to write and good for regression
coverage, but they test permissions in isolation. Real workflows cross
service boundaries. Build both.

What Changed

Before the framework: five iterations to get one permission set
right, every iteration a production impact. After: 95% of permission
issues caught at PR review time. Zero production impacts from permission
bugs since we shipped it. The templates reduced new permission set
creation time by about 70% --- instead of starting from scratch with the
IAM documentation, we start from a pre-validated base and modify from
there.

The time investment was about a week: two days for templates, two
days for the test framework and scripts, one day for CI/CD integration
and documentation. That investment paid back in the first sprint when
the analyst permission set for a new hire went out correct on the first
deployment.

Running into IAM permission debugging loops on your team? Reach out --- permission testing infrastructure is
one of the first things I build when joining a new platform team.

Building Apache Iceberg Lakehouse Storage with S3 Table Buckets

Glenn Gray — Mon, 16 Mar 2026 23:28:31 +0000

This post was originally published on graycloudarch.com.

The data platform team had a deadline and a storage decision to make.
They'd committed to Apache Iceberg as the table format --- open standard,
time travel, schema evolution, the usual reasons. What they hadn't
locked down was where the data was actually going to live, and whether
the storage layer would hold up under the metadata-heavy access patterns
Iceberg requires.

The default answer is regular S3. It works. Most Iceberg deployments
run on it. But AWS launched S3 Table Buckets in late 2024, and they're
purpose-built for exactly this workload: Iceberg metadata operations.
The numbers made the decision easy --- 10x faster metadata queries, 50% or
more improvement in query planning time compared to standard S3. The
gotcha worth knowing upfront: S3 Table Bucket support requires AWS
Provider 5.70 or later. If your Terraform modules are pinned to an older
provider version, that's your first upgrade.

We built the storage layer as a three-zone medallion architecture,
fully managed with Terraform, with Intelligent-Tiering configured from
day one. Here's how we did it.

The Medallion Architecture

Three zones, each with a clear contract about what data lives there
and who owns it:

Raw is immutable. Once data lands there, it doesn't change --- ETL
failures don't corrupt the source record because the source record is
untouched. Clean is normalized and domain-aligned, owned by data
engineering. Curated is the analytics layer that BI tools, Athena
queries, and QuickSight dashboards read from.

The naming convention we landed on was {zone}_{domain}
for Glue databases --- raw_crm, clean_customer,
curated_sales_metrics. It looks minor, but it matters. When
you're looking at a table in Athena or debugging a failed Glue job, the
database name tells you exactly what tier you're in and what domain
you're touching. Namespace collisions become impossible because the zone
prefix scopes every domain. Data lineage is readable from table names
alone.

Why Two Modules Instead of One

The first design question was whether to build a single composite
module that creates the KMS key and the S3 Table Bucket together, or
split them into separate modules. We split them.

The KMS key isn't just for the lake. It's used by five downstream
services: Athena for query results, EMR for cluster encryption, MWAA for
DAG storage, Kinesis for stream encryption, and Glue DataBrew for
transform outputs. If we bundled the key into the lake storage module,
every one of those services would need a dependency chain that
eventually resolves back through lake storage just to get a KMS key ARN.
Separate modules mean the key has one owner, and everything else
declares a dependency on it independently.

The KMS module:

# kms-key/main.tf
resource "aws_kms_key" "this" {
  description             = var.description
  enable_key_rotation     = var.enable_key_rotation
  deletion_window_in_days = var.deletion_window_in_days

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "Enable IAM User Permissions"
        Effect = "Allow"
        Principal = { AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root" }
        Action   = "kms:*"
        Resource = "*"
      },
      {
        Sid    = "Allow Service Access"
        Effect = "Allow"
        Principal = { Service = var.service_principals }
        Action = ["kms:Decrypt", "kms:GenerateDataKey", "kms:CreateGrant"]
        Resource = "*"
      }
    ]
  })
}

The service_principals variable takes a list of service
principal strings ---
["athena.amazonaws.com", "glue.amazonaws.com"] and so on.
Adding a new service that needs key access is one line in the Terragrunt
config, no module change required.

::: {}
Working through something similar? I advise platform teams on AWS infrastructure â€" multi-account architecture, Transit Gateway, EKS, and Terraform IaC. Let's talk.
:::

The S3 Table Bucket Module

The table bucket itself is straightforward. The interesting part is
Intelligent-Tiering:

# s3-table-bucket/main.tf
resource "aws_s3tables_table_bucket" "this" {
  name = var.bucket_name
}

resource "aws_s3_bucket_intelligent_tiering_configuration" "this" {
  count  = var.enable_intelligent_tiering ? 1 : 0
  bucket = aws_s3tables_table_bucket.this.name
  name   = "EntireBucket"

  tiering {
    access_tier = "ARCHIVE_ACCESS"
    days        = 90
  }
  tiering {
    access_tier = "DEEP_ARCHIVE_ACCESS"
    days        = 180
  }
}

We enable Intelligent-Tiering on the entire bucket from the start.
The 90-day threshold for Archive Access and 180-day threshold for Deep
Archive weren't arbitrary --- they match the typical access patterns for a
data lake: raw data is queried heavily during initial load and
validation, then access drops off sharply once the clean layer is
populated.

The reason Intelligent-Tiering beats manual lifecycle policies here
is subtle but important. A manual lifecycle policy moves data based on
age. Intelligent-Tiering moves data based on actual access patterns. If
a dataset from eight months ago suddenly becomes relevant for a
compliance audit, Intelligent-Tiering keeps it in a more accessible tier
automatically. A manual policy would have moved it to Deep Archive on
day 180 regardless. For a data lake, where access patterns are genuinely
unpredictable, letting AWS monitor actual usage is worth the small
monitoring fee.

The Terragrunt dependency chain wires the KMS key ARN into the table
bucket configuration:

# lake-storage/terragrunt.hcl
dependency "kms" {
  config_path = "../kms-key"
}

inputs = {
  bucket_name = "company-lake-${local.environment}"
  kms_key_arn = dependency.kms.outputs.key_arn
}

Glue Data Catalog

We provisioned 12 Glue databases across the three zones --- four
domains per zone (CRM, customer, sales, operations). The Terraform for
each database includes the Iceberg metadata parameters that enable
Iceberg table format for all tables created in that database:

resource "aws_glue_catalog_database" "this" {
  name = "raw_crm"
  parameters = {
    "iceberg_enabled" = "true"
    "table_type"      = "ICEBERG"
    "format-version"  = "2"
  }
}

Format version 2 is the current Iceberg spec. It unlocks row-level
deletes, which is required for GDPR compliance --- when a user requests
deletion, you can execute a targeted delete on the Iceberg table rather
than rewriting entire Parquet partitions.

One thing that's easy to miss: Glue databases with Iceberg parameters
set don't automatically create Iceberg tables. The database parameters
act as defaults and metadata; actual table creation still happens via
your ETL tooling (Glue jobs, Spark, Flink). What you get from Terraform
is the catalog structure and the governance layer --- databases,
permissions, encryption settings --- so that when the data engineering
team writes their first Glue job, the infrastructure is already in
place.

The Cost Model

When I presented this to the platform team's tech lead, the cost
projection was what turned a "nice to have" into a "let's do this
now."

For a 100TB lake:

Timeframe Storage Tier Monthly Cost

First 90 days Standard ~\$2,300
After 90 days Archive Access ~\$400
After 180 days Deep Archive ~\$100

That's roughly 80% savings once the bulk of the data ages past 90
days, and 95% savings at 180 days. The Intelligent-Tiering monitoring
cost is \$0.0025 per 1,000 objects --- on a 100TB lake with typical Iceberg
file sizes, that's a few dollars a month. Negligible.

The S3 Table Bucket metadata performance improvement compounds this.
Faster query planning means less Athena scan time, which means lower
query costs and faster results for analysts. The platform pays for
itself in reduced query costs as the data volume grows.

Deployment Sequence

The deployment order is driven by dependencies: KMS must exist before
S3 (bucket encryption needs the key ARN), and both must exist before
Glue (catalog databases reference the bucket location).

In practice, across three environments (dev, nonprod, prod), the full
deployment took about four hours. Most of that was Terragrunt apply time
--- the actual resource creation for each component is fast, but we ran
plan, reviewed, applied, and verified before moving to the next
environment.

One deployment note: the first time you run
terragrunt plan on the Glue module in an account that
hasn't had Glue configured before, you'll get an error about the Glue
service-linked role not existing. Fix it by running
aws iam create-service-linked-role --aws-service-name glue.amazonaws.com
before the apply. It only needs to happen once per account.

What the Data Team Inherited

When we handed this over to the data engineering team, they had a
fully provisioned catalog --- three zones, twelve databases, Iceberg
metadata configured, encryption enabled, Intelligent-Tiering active.
They could start writing Glue jobs and creating tables immediately
without worrying about storage configuration, access patterns, or cost
optimization after the fact.

The Terraform modules are reusable. Adding a new domain (say, a
finance domain across all three zones) is three database
resource declarations and one pull request. The KMS key, bucket, and
Intelligent-Tiering configuration don't change.

S3 Table Buckets are still relatively new, and the Terraform provider
support came together in late 2024. If your team is planning an Iceberg
migration and hasn't evaluated Table Buckets yet, the metadata
performance gains and the cost trajectory make a strong case for
starting there rather than retrofitting later.

Building out a data platform and figuring out the storage and catalog
architecture? Get in touch --- this kind of
infrastructure design work is something I do regularly, whether you're
starting from scratch or migrating an existing lake.

Forem: Glenn Gray

Building Automated AWS Permission Testing Infrastructure for CI/CD

The "Access Denied" Debugging Loop

What I Built

Phase 1: Pre-Validated Templates

Phase 2: The Test Framework

Phase 3: CI/CD Integration

Three Things I Learned the Hard Way

What Changed

Stop Managing EKS Add-ons by Hand

The Problem with Self-Managed Add-ons

The Solution: One Terraform Module for All Six Add-ons

EBS CSI Driver Needs an IAM Role

Migrating Metrics Server from kubectl to Helm

Deploying It

What Changed

Building Apache Iceberg Lakehouse Storage with S3 Table Buckets

The Medallion Architecture

Why Two Modules Instead of One

The S3 Table Bucket Module

Glue Data Catalog

The Cost Model

Deployment Sequence

What the Data Team Inherited

Zero-Downtime AWS Transit Gateway Hub-Spoke Migration

The Architecture We Were Replacing

The Target: Inspection VPC + Hub Transit Gateway

Phase 1: Module Changes

Transit Gateway: add a conditional create flag

New module: vpc-attachment

Hub route tables: workload isolation by default

Inspection VPC subnet layout

Security baseline: convert to delegated admin model

Phase 2: Two Maintenance Windows

Window 1: Deploy the hub (~45 minutes, low risk)

Window 2: Spoke cutover (~2 hours)

What We Ended Up With

DNS Validation: From 15 Steps to Zero

There's a Better Way

The Magic: for_each

What I Screwed Up

Why This Matters

The Part People Miss

Scaling It

The Real Win

Building Multi-Account AWS Infrastructure with Terraform and ECP

The Problem with Single-Account AWS

The ECP Pattern

Key Design Decisions

Terraform Structure

Lessons Learned

Results

Stop Manually Updating Jira After Every PR Merge

The Architecture

Step 1: Composite Action for Ticket Extraction

Step 2: PR Creation Workflow

Step 3: PR Merge Workflow

Finding Your Transition IDs

A Note on Jira API Versions

Required Secrets

Results

How I Manage Claude Code Context Across 20+ Repositories

The Problem with Single-File Context

My First Attempt: A Shared Scripts Directory

The Three-Tier Hierarchy

Implementation: Symlinks from Dotfiles

What Each Level Contains

Global (~/.claude/)

Org Level (~/work/{employer}/.claude/)

Project Level (~/work/{employer}/{repo}/.claude/)

The Privacy Model

Custom Commands

What This Solves in Practice

Starting Point

How I Manage Claude Code Context Across 20+ Repositories

The Problem with Single-File Context

My First Attempt: A Shared Scripts Directory

The Three-Tier Hierarchy

Implementation: Symlinks from Dotfiles

What Each Level Contains

Global (`~/.claude/`)

Org Level (`~/work/{employer}/.claude/`)

Project Level (`~/work/{employer}/{repo}/.claude/`)

Global (`~/.claude/`)

Org Level (`~/work/{employer}/.claude/`)

Project Level (`~/work/{employer}/{repo}/.claude/`)