Forem: Dhruv Chaudhary

Guardrails for AI-Generated IaC: How MyCoCo Made Speed Sustainable

Dhruv Chaudhary — Sat, 06 Dec 2025 22:24:51 +0000

AI coding assistants promise to accelerate infrastructure delivery, but organizations are discovering a hidden cost: code that passes syntax validation often fails security audits. Recent research shows that while AI-generated infrastructure code may look correct, only 9% meets security compliance standards. When MyCoCo's platform team generated dozens of Terraform modules with AI assistance, a security scan revealed a sobering truth—speed without guardrails creates technical debt that compounds with every deployment.

TL;DR

The Problem: AI-generated Terraform passes terraform validate but fails organizational compliance—missing tags, overly permissive IAM, exposed resources.

The Solution: Implement OPA-based policy guardrails at the PR level that catch AI blind spots before code reaches production.

The Impact: MyCoCo reduced security findings from 47 to 3 per AI-generated module while retaining 70% of velocity gains.

Key Implementation: Custom OPA policies targeting common AI omissions: required tags, encryption enforcement, least-privilege IAM.

Bottom Line: AI accelerates IaC development, but only with organizational context injected through automated policy enforcement.

The Challenge: MyCoCo's AI Experiment

Jordan, MyCoCo's Platform Engineer, was convinced AI would transform their infrastructure delivery. With a major product launch approaching, the platform team faced an impossible timeline: 30 new Terraform modules in six weeks. Using GitHub Copilot and Claude, Jordan's team produced the modules in just two weeks.

"We were shipping infrastructure faster than ever. The AI understood Terraform syntax perfectly. Every module passed validation on the first try."

Then Maya, the Security Engineer, ran her pre-production Checkov scan.

The results stopped the launch cold: 47 security findings per module on average. S3 buckets without encryption. Lambda functions with wildcard IAM permissions. And the most painful discovery—not a single resource had MyCoCo's required Environment, Owner, or CostCenter tags.

"The AI wrote syntactically perfect Terraform. But it had no idea about our tagging policies, our naming conventions, or our security baseline. It generated code like we were a greenfield startup, not a company preparing for SOC 2."

Sam, the Senior DevOps Engineer, had warned the team from the start. The confidence gap was real—the team trusted AI-generated code more than manually written code, despite having less visibility into its logic.

Alex, VP of Engineering, faced a choice: delay the launch to manually fix every module, or find a way to make AI-generated code meet MyCoCo's standards automatically.

The Solution: OPA Guardrails for AI-Generated Code

MyCoCo's solution wasn't to abandon AI—it was to teach their pipeline what the AI didn't know. The team implemented a three-layer policy enforcement approach using Open Policy Agent (OPA) integrated with Conftest.

Layer 1: Required Tags Policy

The most common AI omission was resource tagging. MyCoCo created an OPA policy that blocks any PR missing required tags:

# policy/tags.rego
package terraform.tags

required_tags := ["Environment", "Owner", "CostCenter"]

deny[msg] {
    resource := input.resource_changes[_]
    resource.change.actions[_] == "create"

    tags := object.get(resource.change.after, "tags", {})
    missing := [tag | tag := required_tags[_]; not tags[tag]]

    count(missing) > 0
    msg := sprintf(
        "%s '%s' missing required tags: %v",
        [resource.type, resource.name, missing]
    )
}

Layer 2: Encryption Enforcement

AI-generated S3 buckets and RDS instances frequently lacked encryption configuration—a SOC 2 requirement:

# policy/encryption.rego
package terraform.encryption

deny[msg] {
    resource := input.resource_changes[_]
    resource.type == "aws_s3_bucket"
    resource.change.actions[_] == "create"

    # Check for server-side encryption configuration
    not has_encryption_config(resource.address)

    msg := sprintf(
        "S3 bucket '%s' must have encryption enabled",
        [resource.name]
    )
}

Layer 3: IAM Least Privilege

The most dangerous AI pattern was wildcard IAM permissions. This policy catches overly permissive policies before they reach production:

# policy/iam.rego
package terraform.iam

deny[msg] {
    resource := input.resource_changes[_]
    resource.type == "aws_iam_policy"

    policy_doc := json.unmarshal(resource.change.after.policy)
    statement := policy_doc.Statement[_]

    statement.Effect == "Allow"
    statement.Action[_] == "*"

    msg := sprintf(
        "IAM policy '%s' contains wildcard Action - use least privilege",
        [resource.name]
    )
}

Pipeline Integration

The team integrated these policies into their GitHub Actions workflow, running conftest against every Terraform plan:

- name: Policy Check
  run: |
    terraform plan -out=tfplan
    terraform show -json tfplan > tfplan.json
    conftest test tfplan.json --policy policy/

Any policy violation blocks the PR merge, with clear error messages explaining exactly what needs to be fixed. Jordan found that AI assistants could often fix the violations when given the specific error message—turning the guardrail into a feedback loop.

Results: MyCoCo's Transformation

Within three weeks of implementing OPA guardrails, MyCoCo's metrics shifted dramatically:

Security findings per AI-generated module: 47 → 3 (94% reduction)
Development velocity: Retained approximately 70% of the original speed gains
Unexpected benefit: The guardrails improved manually-written code too—engineers discovered their own modules had tagging gaps

"We stopped thinking of AI as a code generator and started thinking of it as a fast first draft. The guardrails aren't a speed bump—they're the quality gate that makes the speed sustainable."

Maya added the policies to MyCoCo's security documentation, creating an "AI-Generated Code Checklist" that new team members review before using coding assistants. The launch proceeded on schedule, with infrastructure that passed SOC 2 audit on the first attempt.

Key Takeaways

Syntax validity does not equal security compliance. AI-generated code that passes terraform validate may still fail 90%+ of security requirements.
AI lacks organizational context by design. Your tagging policies, naming conventions, and security baselines don't exist in training data. Guardrails inject that context automatically.
The confidence gap is dangerous. Teams often review AI-generated code less carefully than human-written code, despite it being more likely to have compliance gaps. Invert this assumption.
Guardrails create feedback loops. When AI assistants receive specific policy violation messages, they can often self-correct—making the guardrail an accelerator, not just a gate.
Start with the obvious omissions. Required tags, encryption, and least-privilege IAM catch the majority of AI blind spots with minimal policy complexity.

Conclusion

AI-generated infrastructure code isn't going away—it's too fast and too useful. But speed without guardrails creates security debt that compounds with every deployment. The solution isn't to abandon AI; it's to inject your organizational context through automated policy enforcement.

Start with three policies: required tags, encryption enforcement, and IAM least privilege. These catch the majority of AI blind spots and give you a foundation to build on.

Your infrastructure can be both fast and compliant. You just need the right guardrails.

What's your experience with AI-generated infrastructure code? Have you implemented guardrails, or are you still reviewing everything manually? Share your lessons learned in the comments below!

Terraform Stacks: MyCoCo's Landing Zone Dependencies Done Right

Dhruv Chaudhary — Sat, 29 Nov 2025 22:23:35 +0000

Elevator Pitch

Every growing platform team faces the same architectural challenge: shared infrastructure—networking, security, identity—must evolve independently from the applications that consume it, yet changes to these "landing zones" ripple unpredictably across downstream systems. MyCoCo's platform team discovered this the hard way when a routine networking update triggered a 47-minute production outage. Terraform Stacks, now generally available in HCP Terraform, transforms landing zone management from a coordination nightmare into an automatically-orchestrated dependency graph—making foundational infrastructure changes safe, visible, and automatically propagated to every consuming application.

TL;DR

The Problem: Landing zone changes—networking, security, IAM baselines—create invisible dependencies that break downstream applications when platform teams update shared infrastructure
The Solution: Terraform Stacks with Linked Stacks formalize landing zone relationships, automatically triggering downstream plans when foundational infrastructure changes
The Impact: MyCoCo eliminated surprise outages from landing zone updates and reduced cross-team coordination from hours of Slack messages to automatic HCP Terraform notifications
Key Implementation: Landing zone Stack publishes outputs via publish_output blocks; product Stacks consume them via upstream_input blocks with automatic change propagation
Bottom Line: If your platform team manages shared infrastructure that application teams consume, Linked Stacks turn that implicit dependency into an explicit, automatically-orchestrated relationship

The Challenge: When Landing Zone Updates Break Everything

MyCoCo's architecture followed the pattern every scaling platform team adopts: centralized landing zones managed by the platform team, consumed by product teams through Terraform data sources. Five product teams, three environments, two regions—on paper, clean separation of concerns. In practice? Invisible dependencies everywhere.

The breaking point came on a Friday afternoon. The platform team pushed a routine subnet reorganization. GitHub Actions ran successfully. Tests passed. Three hours later, the Support team deployed an application update referencing security group IDs that had been reorganized. Production was down for 47 minutes.

"The landing zone is supposed to be the stable foundation," Sam (Senior DevOps Engineer) said during the post-incident review. "Instead, it's a source of surprise outages because we have no way to know which product stacks depend on which outputs—until something breaks."

Jordan (Platform Engineer) had been tracking Terraform Stacks since the beta. The GA release included Linked Stacks—exactly what MyCoCo needed: landing zone dependencies that are explicit, visible, and automatically coordinated.

The Solution: Stacks for Landing Zone Architecture

Terraform Stacks address the landing zone problem through three interconnected concepts: Components for reusable infrastructure definitions, Deployments for environment-specific instantiation, and Linked Stacks for formalizing cross-stack dependencies.

Important: Stacks are exclusively an HCP Terraform feature—they're not available in open-source Terraform. With RUM-based pricing, each managed resource counts toward your bill. For teams already paying the coordination tax in engineering hours, Slack threads, and incident response, the trade-off often favors Stacks. But model your costs explicitly before migrating—especially for large landing zones with many downstream consumers.

Components: Reusable Landing Zone Modules

Components wrap Terraform modules into reusable building blocks. For landing zones, this means defining networking, security, and identity components once:

# landing-zone/components.tfcomponent.hcl
component "vpc" {
  source = "./modules/vpc"

  inputs = {
    environment = var.environment
    cidr_block  = var.cidr_block
    region      = var.region
  }

  providers = {
    aws = provider.aws.main
  }
}

component "security_baseline" {
  source = "./modules/security"

  inputs = {
    vpc_id      = component.vpc.vpc_id
    environment = var.environment
  }

  providers = {
    aws = provider.aws.main
  }
}

Components automatically resolve internal dependencies—security_baseline waits for vpc without orchestration scripts.

Deployments: Environment Parity Without the Drift

Maintaining landing zones across multiple environments traditionally forces an uncomfortable choice: copy-paste your Terraform code and modify it per environment (leading to inevitable drift), or manage complex variable files with conditional logic that becomes increasingly fragile.

Deployments solve this by letting you define your infrastructure once via Components, then create multiple instantiations with only the inputs changing:

# landing-zone/deployments.tfdeploy.hcl
deployment "development" {
  inputs = {
    environment = "dev"
    cidr_block  = "10.1.0.0/16"
    region      = "ca-central-1"
  }
}

deployment "production" {
  inputs = {
    environment = "prod"
    cidr_block  = "10.0.0.0/16"
    region      = "ca-central-1"
  }
}

Both deployments run the exact same VPC and security baseline components—just with different inputs. Each deployment maintains its own isolated state file, so changes to one environment never accidentally affect another.

The practical payoff: when your platform team updates the VPC component logic, that change rolls out consistently to both dev and prod. No more "works in dev, breaks in prod" because the configurations drifted over months of separate maintenance.

Linked Stacks: The Landing Zone Game-Changer

The killer feature for platform teams is Linked Stacks. Instead of product teams using fragile data sources to reference landing zone outputs, the relationship becomes explicit and automatically coordinated.

The landing zone Stack publishes specific outputs for downstream consumption—building on the production deployment we defined above:

# landing-zone/deployments.tfdeploy.hcl (continued)
publish_output "vpc_id" {
  description = "Production VPC for application stacks"
  value       = deployment.production.vpc_id
}

publish_output "private_subnet_ids" {
  description = "Private subnets for application workloads"
  value       = deployment.production.private_subnet_ids
}

Product Stacks declare explicit dependencies on the landing zone using upstream_input blocks:

# product-stack/deployments.tfdeploy.hcl
upstream_input "landing_zone" {
  type   = "stack"
  source = "app.terraform.io/mycoco/platform/landing-zone"
}

deployment "production" {
  inputs = {
    environment = "prod"
    vpc_id      = upstream_input.landing_zone.vpc_id
    subnet_ids  = upstream_input.landing_zone.private_subnet_ids
  }
}

The critical benefit: when the platform team updates the landing zone—adding subnets, modifying security groups, changing routing—HCP Terraform automatically triggers plans in every downstream product Stack. No more "who needs to know about this change?" conversations. No more forgotten dependencies causing Friday incidents.

Results: Landing Zone Changes Without Fear

Six weeks after migrating the landing zone to Stacks, the transformation was unmistakable.

Landing zone updates went from anxiety-inducing Slack announcements to routine operations. When networking changes are needed, HCP Terraform automatically queues plans for all five product Stacks—product teams see exactly what upstream changes triggered their plan. The "did anyone change the landing zone today?" messages disappeared entirely.

Environment drift hasn't occurred since the migration. The same components deploy identically across environments; when the platform team adds a subnet, it appears everywhere simultaneously.

"Friday deployments used to terrify me," Sam admitted. "Now the dependency graph isn't in my head anymore—it's in the configuration."

The team retired 400 lines of orchestration scripts and three GitHub Actions workflows. More importantly, they retired the on-call burden of being the human dependency resolver.

Key Takeaways

Start with your landing zone. If your platform team manages shared infrastructure consumed by application teams, migrate the landing zone first. The publish_output pattern immediately demonstrates value to downstream teams who see automatic plan triggers instead of surprise breakages.

Linked Stacks replace tribal knowledge. Every data source referencing another team's infrastructure is an implicit dependency waiting to cause an incident. upstream_input blocks make those relationships explicit, versioned, and automatically coordinated.

Automatic propagation changes team dynamics. When landing zone changes automatically trigger downstream plans, the platform team stops being a coordination bottleneck and starts being an infrastructure service provider.

Model your costs before migrating. Stacks are HCP Terraform-only. Calculate your RUM-based costs against the engineering hours you're currently spending on coordination—for most teams with complex landing zones, the math favors Stacks.

Migrate incrementally. Stacks coexist with traditional workspaces. Migrate your landing zone, connect one product Stack as a proof of concept, then expand. The upstream_input pattern works across the boundary—new Stacks can consume from existing ones.

For platform teams drowning in cross-repository dependencies and change coordination, Linked Stacks represent the most significant improvement to landing zone management since Terraform workspaces. The dependency graph you've been maintaining manually? It's finally infrastructure as code.

Ready to formalize your landing zone dependencies? Start by modeling your current architecture as Stacks Components, then connect one downstream consumer with upstream_input blocks.

Why Ephemeral Resources in Terraform Matter: How MyCoCo Eliminated Secrets from State Files

Dhruv Chaudhary — Fri, 19 Sep 2025 16:48:25 +0000

When organizations manage cloud infrastructure with code, sensitive passwords and access keys often get saved in files where anyone with access can see them—creating major security risks. Terraform v1.10's "ephemeral" features solve this by using secrets temporarily without saving them permanently. Here's how MyCoCo went from failing security audits to achieving SOC 2 compliance by eliminating all exposed passwords from their infrastructure files.

TL;DR

The Problem: Terraform state files store database passwords, API keys, and certificates in plaintext, creating major security and compliance risks.

The Solution: Terraform v1.10's ephemeral resources exist only during execution—never written to state or plan files. They use a unique lifecycle: open → use → close.

The Impact: MyCoCo went from failing security audits due to exposed secrets to achieving SOC 2 compliance by eliminating all sensitive data from state files. Zero secrets in state, improved developer experience, simplified disaster recovery.

Key Implementation: Use ephemeral "aws_secretsmanager_secret_version" to fetch secrets and secret_string_wo (write-only) arguments to store them without state persistence.

Bottom Line: If you're storing secrets in Terraform state files, ephemeral resources aren't optional—they're essential for enterprise security.

The Challenge: MyCoCo's Security Wake-Up Call

MyCoCo started like many tech companies—two DevOps engineers managing infrastructure for a growing startup. As they scaled from 5 to 50 engineers, their Terraform usage expanded rapidly. They were proud of their infrastructure-as-code approach, with everything properly versioned and state files safely stored in an encrypted S3 bucket.

Then came the compliance audit.

"Your Terraform state files contain database passwords in plaintext," the security consultant announced during their SOC 2 preparation. "Anyone with access to your state backend can see every database credential, API token, and private key you manage."

The MyCoCo team was stunned. They'd focused on encrypting state files at rest and restricting S3 bucket access, but they'd never considered what was inside those encrypted files. A quick investigation revealed the scope:

Database passwords for PostgreSQL and Redis instances
JWT signing keys for authentication
SSL certificate private keys
Third-party API tokens for monitoring services
SSH keys for bastion host access

Even worse, their disaster recovery procedures included copying state files to a secondary region, and their CI/CD system downloaded state files to runners for each deployment. Every copy contained the same sensitive data.

Their initial attempts to solve this failed. The sensitive argument only hides CLI output—data still goes to state files. External secret management tools proved complex to integrate with existing workflows.

The audit deadline was approaching, and MyCoCo needed a solution that would eliminate secrets from state files without a complete workflow overhaul.

The Solution: MyCoCo's Ephemeral Implementation

Terraform v1.10's ephemeral resources solved MyCoCo's problem. These resources exist only during execution—they're opened when needed, used during operations, then closed. Values never touch state or plan files.

Before: Passwords in State

# OLD APPROACH - Password stored in state
resource "random_password" "db_password" {
  length  = 16
  special = true
}

resource "aws_db_instance" "main" {
  password = random_password.db_password.result  # Goes to state!
}

After: Clean State Files

# NEW APPROACH - No passwords in state
ephemeral "random_password" "db_password" {
  length           = 16
  override_special = "!#$%&*()-_=+[]{}<>:?"
}

resource "aws_secretsmanager_secret_version" "db_password" {
  secret_id        = aws_secretsmanager_secret.db_password.id
  secret_string_wo = ephemeral.random_password.db_password.result
}

The key insight: secret_string_wo is a write-only argument that accepts ephemeral values but doesn't store them in state.

Retrieving Secrets Without Persistence

ephemeral "aws_secretsmanager_secret_version" "app_database" {
  secret_id = aws_secretsmanager_secret_version.db_password.secret_id
}

locals {
  db_credentials = jsondecode(
    ephemeral.aws_secretsmanager_secret_version.app_database.secret_string
  )
}

provider "postgresql" {
  host     = aws_db_instance.main.address
  username = local.db_credentials["username"]
  password = local.db_credentials["password"]
}

This pattern retrieves secrets during Terraform execution without persisting them in the infrastructure-as-code workflow.

Results: MyCoCo's Security Transformation

Complete Secret Elimination: State files went from containing dozens of sensitive values to zero. Security audits could focus on access controls rather than data exposure.

Simplified Compliance: SOC 2 compliance became straightforward when they could demonstrate no secrets persisted in infrastructure artifacts.

Improved Operations: Disaster recovery, state file backups, and vendor sharing became routine operations without security concerns.

Enhanced Developer Experience: Team members could examine state files for debugging without exposing production secrets.

The transition required understanding that ephemeral resources can't be referenced in contexts requiring persistence—but this constraint is the security feature, not a limitation.

Key Takeaways

Start with High-Value Secrets: Prioritize database passwords, API keys, and certificates for maximum security impact.
Understand Write-Only Arguments: These accept ephemeral values without storing them in state—the key to clean implementation.
Plan for Provider Differences: AWS, Azure, and GCP implement ephemeral resources differently based on their native secret management patterns.
Validate Implementation: Use terraform show to verify sensitive values no longer appear in state files.
Embrace the Constraint: Ephemeral resources can't persist data—this limitation is the security benefit.

Conclusion

For teams still storing secrets in state files, ephemeral resources represent a fundamental shift toward security-by-design in infrastructure-as-code. The investment in learning this approach pays dividends in compliance, security, and operational confidence.

The key is understanding that ephemeral resources solve a fundamental problem: how to use sensitive data in infrastructure code without creating persistent security risks. They're not just a feature—they're a security paradigm that should be standard practice for any organization handling sensitive infrastructure.

Ready to eliminate secrets from your state files? Start by identifying your highest-risk secrets and implementing ephemeral resources with Terraform v1.10.

What's your experience with secrets in Terraform state files? Have you implemented ephemeral resources or found other approaches to this security challenge? Share your strategies and lessons learned in the comments below!

Comprehensive Terraform State Security: MyCoCo's Journey from Public Exposure to Layered Protection

Dhruv Chaudhary — Fri, 19 Sep 2025 16:44:40 +0000

When organizations store Terraform state files, they're essentially creating blueprints of their entire infrastructure that hackers would love to access. Even encrypted storage isn't enough if the data inside reveals architectural vulnerabilities, database locations, and system dependencies. Here's how MyCoCo transformed a three-hour public exposure incident into enterprise-grade state security that passed rigorous compliance audits.

TL;DR

The Problem: Terraform state files expose infrastructure architecture, resource relationships, and system dependencies even without credential leaks—creating reconnaissance goldmines for attackers.

The Solution: Layered state security combining remote backends, encryption at rest/transit, access controls, and audit logging across multiple protection levels.

The Impact: MyCoCo went from accidental public exposure to enterprise-grade state security, achieving SOC 2 compliance and eliminating architectural intelligence gathering risks.

Key Implementation: S3 backend with KMS encryption, IAM policies with least privilege, and comprehensive audit trails.

Bottom Line: If your state files aren't comprehensively protected, you're giving attackers detailed blueprints of your infrastructure—encryption alone isn't enough.

The Challenge: MyCoCo's Security Wake-Up Call

During MyCoCo's SOC 2 preparation, Maya's security audit revealed multiple state file vulnerabilities beyond the secrets exposure issue. The immediate crisis came during a routine infrastructure update when Sam accidentally applied a Terraform configuration that modified their S3 bucket policy, making their state storage publicly readable for three hours before the monitoring system caught the misconfiguration.

Maya was immediately alerted to the bucket exposure. While investigating the incident, she discovered something that kept her awake that night: even without any credential leaks, the exposed state files revealed a detailed map of MyCoCo's entire infrastructure architecture.

"Look at this," Maya told the team during the post-incident review, pulling up the state file contents. "Someone could see our database instance types, VPC configurations, load balancer setups, and even our disaster recovery patterns. They'd know exactly how our systems connect and where our critical components live."

Alex, their VP of Engineering, realized the scope:

"This isn't just about encryption. Someone could use this architectural information to plan attacks, understand our scaling patterns, and identify potential weak points in our infrastructure."

The incident triggered an emergency security review that encompassed both state file access controls and the secrets exposure issues Maya had identified. While no credentials were compromised, the architectural exposure violated several SOC 2 requirements and could have enabled sophisticated attacks. MyCoCo needed comprehensive state security that protected both sensitive data and infrastructure intelligence.

The Solution: MyCoCo's Layered State Security Implementation

Maya designed a comprehensive state security strategy addressing multiple threat vectors beyond basic encryption. The approach required protecting state files from unauthorized access while maintaining team productivity and compliance requirements.

Layer 1: Secure Remote Backend Modernization

# Enhanced S3 backend configuration with native state locking
terraform {
  required_version = ">= 1.12"  # Latest stable version
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"  # Latest major version
    }
  }

  backend "s3" {
    bucket         = "mycoco-terraform-state-prod"
    key            = "infrastructure/terraform.tfstate"
    region         = "ca-central-1"
    encrypt        = true
    kms_key_id     = "arn:aws:kms:ca-central-1:123456789:key/terraform-state"
    use_lockfile   = true

    # Critical security configurations
    skip_region_validation      = false
    skip_credentials_validation = false
    skip_metadata_api_check     = false
  }
}

MyCoCo's existing S3 remote state setup had critical security gaps and used outdated DynamoDB locking. Layer 1 modernized their backend configuration with S3 native locking using use_lockfile = true, eliminating their DynamoDB dependency while adding proper security configurations. This foundational update secured their existing remote state infrastructure and established the baseline for all subsequent security controls.

Layer 2: Access Control and Authentication

# IAM policy for Terraform state access with S3 native locking support
data "aws_iam_policy_document" "terraform_state_access" {
  statement {
    effect = "Allow"
    actions = [
      "s3:GetObject",
      "s3:PutObject",
      "s3:DeleteObject"
    ]
    resources = [
      "${aws_s3_bucket.terraform_state.arn}/*",
      "${aws_s3_bucket.terraform_state.arn}/*.tflock"  # Lock file access
    ]

    condition {
      test     = "StringEquals"
      variable = "s3:x-amz-server-side-encryption"
      values   = ["aws:kms"]
    }

    condition {
      test     = "StringEquals"
      variable = "s3:x-amz-server-side-encryption-aws-kms-key-id"
      values   = ["${aws_kms_key.terraform_state.arn}"]
    }
  }

  statement {
    effect = "Allow"
    actions = ["s3:ListBucket"]
    resources = [aws_s3_bucket.terraform_state.arn]
  }

  statement {
    effect = "Allow"
    actions = [
      "kms:Decrypt",
      "kms:GenerateDataKey"
    ]
    resources = [aws_kms_key.terraform_state.arn]
  }
}

Layer 2 established granular control over who could access state files and under what conditions. The foundation began with defining precise IAM permissions that specified exactly which actions were allowed on state files and lock files, including the encryption requirements that must be met for any access attempts.

Layer 3: Monitoring and Audit Trail

# Configure AWS provider
provider "aws" {
  region = "ca-central-1"  # Primary region for Toronto-based MyCoCo
}

# CloudTrail for state access monitoring
resource "aws_cloudtrail" "terraform_state_audit" {
  name           = "terraform-state-access"
  s3_bucket_name = aws_s3_bucket.audit_logs.bucket
  s3_key_prefix  = "terraform-state-audit"

  event_selector {
    read_write_type           = "All"
    include_management_events = true

    data_resource {
      type   = "AWS::S3::Object"
      values = ["${aws_s3_bucket.terraform_state.arn}/*"]
    }
  }
}

# Real-time monitoring
resource "aws_cloudwatch_metric_alarm" "state_access_errors" {
  alarm_name          = "terraform-state-access-errors"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "ErrorCount"
  namespace           = "AWS/CloudTrail"
  period              = 300
  statistic           = "Sum"
  threshold           = 0
  alarm_description   = "Alert on S3 access errors for Terraform state files"
  alarm_actions       = [aws_sns_topic.state_security_alerts.arn]
}

Layer 3 provided comprehensive visibility into all state file access activities, enabling MyCoCo to detect security incidents, demonstrate compliance, and respond to suspicious activities. The code references existing infrastructure components (S3 buckets, KMS keys, SNS topics) that would be part of MyCoCo's broader security architecture.

Results: MyCoCo's Security Transformation

Complete Architectural Protection: State files became inaccessible to unauthorized users, eliminating reconnaissance risks and architectural intelligence gathering. Security audits could focus on access controls rather than exposure prevention.

Enterprise Compliance Achievement: SOC 2 Type II certification became straightforward when auditors could verify comprehensive state security controls, encryption standards, and access audit trails.

Operational Excellence: The security improvements enhanced rather than hindered operations. Automated monitoring provided early warning of issues, granular IAM policies provided appropriate access controls, and comprehensive audit trails simplified incident response and compliance reporting.

Zero Security Incidents: The comprehensive approach eliminated state-related security concerns. The team could focus on infrastructure innovation rather than security firefighting, and executive stakeholders gained confidence in MyCoCo's infrastructure governance.

The transition required understanding that state security extends beyond encryption to include access control, monitoring, and compliance integration—but these comprehensive protections became competitive advantages rather than operational overhead.

Key Takeaways

Start with Remote Backends: Local state files create uncontrollable security risks that grow with team size and infrastructure complexity.
Layer Security Controls: Encryption alone isn't sufficient—combine with access controls, monitoring, and audit trails for comprehensive protection.
Consider Multi-Region Deployments: For enhanced disaster recovery, consider deploying monitoring infrastructure across multiple regions.
Plan for Compliance: Enterprise customers increasingly require demonstrable state security controls as part of vendor security assessments.
Monitor Continuously: Automated monitoring and alerting enable rapid response to misconfigurations and unauthorized access attempts.
Implement Comprehensive Logging: Detailed audit trails and access monitoring provide the foundation for both security operations and compliance requirements.

Conclusion

For teams still using local state or basic remote backends, comprehensive state security represents a fundamental shift toward enterprise-ready infrastructure governance. The investment in proper state security pays dividends in compliance, security confidence, and operational reliability.

The key insight is that Terraform state files are more than just operational data—they're detailed architectural blueprints that must be protected with the same rigor as your production databases. Start with encrypted remote backends, then build layered access controls and monitoring around your infrastructure state.

What's your approach to Terraform state security? Have you dealt with state file exposure incidents or compliance requirements? Share your security strategies and lessons learned in the comments below!

Terraform Modules and Workspaces: How MyCoCo Scaled from Copy-Paste to DRY Infrastructure

Dhruv Chaudhary — Fri, 19 Sep 2025 16:42:24 +0000

When teams manage multiple environments in Terraform, they often duplicate entire configurations, creating a maintenance nightmare. Terraform workspaces combined with modules provide an elegant solution—one codebase serves all environments through workspace-aware configurations. Here's how MyCoCo transformed their infrastructure management using workspaces to achieve true DRY (Don't Repeat Yourself) principles.

TL;DR

The Problem: Copy-pasted Terraform configurations across dev/staging/production led to environment drift, deployment failures, and 20+ hours weekly maintenance overhead.

The Solution: Terraform workspaces for environment separation combined with modules for code reuse and workspace-specific variable files.

The Impact: MyCoCo reduced configuration management time by 70%, eliminated environment drift, and deployed new environments in hours instead of days.

Key Implementation: Use terraform.workspace with modular design patterns and structured terraform.workspace.tfvars files for environment-specific configurations.

Bottom Line: If you're maintaining separate Terraform directories per environment, workspaces + modules will transform your infrastructure management.

The Challenge: MyCoCo's Environment Sprawl

"Why do we have three different versions of our RDS configuration?" Jordan asked during his first week as MyCoCo's platform engineer. The answer revealed a deeper problem.

Sam, the senior DevOps engineer, explained their evolution:

"We started with one environment, then copied everything for staging, then copied again for production. Now we maintain three separate Terraform directories."

The structure told the story:

infrastructure/
├── dev/
│   ├── main.tf      # 500 lines
│   ├── rds.tf       # 200 lines
│   └── variables.tf # 150 lines
├── staging/
│   ├── main.tf      # 497 lines (slightly different)
│   ├── rds.tf       # 198 lines (minor variations)
│   └── variables.tf # 148 lines (different values)
└── production/
    ├── main.tf      # 502 lines (more differences)
    ├── rds.tf       # 205 lines (critical variations)
    └── variables.tf # 152 lines (production values)

Alex, VP of Engineering, quantified the impact:

"Every infrastructure change requires three pull requests, three code reviews, and three separate terraform apply commands. We're burning 20 hours per week just keeping these in sync."

The breaking point came when a critical security patch needed to be applied across all environments. What should have been a 30-minute task turned into a two-day project of carefully updating each environment, testing separately, and hoping nothing was missed.

The Solution: Workspaces + Modules Architecture

MyCoCo's transformation centered on two key Terraform features: workspaces for environment isolation and modules for code reuse.

Before: Duplicate Directories

# production/rds.tf - Copied and modified for each environment
resource "random_password" "db_password" {
  length  = 16
  special = true
}

# Note: Consider using password_wo instead of password in Terraform 1.11+
# for better security (write-only attribute that doesn't show in plans)
resource "aws_db_instance" "main" {
  identifier     = "mycoco-prod"
  engine         = "postgres"
  engine_version = "15.3"
  instance_class = "db.t3.large"  # Hard-coded per environment

  allocated_storage = 100
  storage_encrypted = true

  username = "dbadmin"
  password = random_password.db_password.result

  # 50+ more lines of mostly identical configuration...
}

After: Workspace-Aware Module

# main.tf - Single configuration for all environments
locals {
  workspace_config = {
    dev = {
      instance_class    = "db.t3.micro"
      allocated_storage = 20
      backup_retention  = 1
    }
    staging = {
      instance_class    = "db.t3.medium"
      allocated_storage = 50
      backup_retention  = 7
    }
    production = {
      instance_class    = "db.t3.large"
      allocated_storage = 100
      backup_retention  = 30
    }
  }

  env_config = local.workspace_config[terraform.workspace]
}

module "database" {
  source = "./modules/rds"

  environment       = terraform.workspace
  instance_class    = local.env_config.instance_class
  allocated_storage = local.env_config.allocated_storage
  backup_retention  = local.env_config.backup_retention

  # Common configuration for all environments
  engine         = "postgres"
  engine_version = "15.3"

  # Workspace-aware naming
  identifier = "mycoco-${terraform.workspace}"
}

The Module Structure

# modules/rds/variables.tf
variable "identifier" {
  description = "The name of the RDS instance"
  type        = string
}

variable "engine" {
  description = "The database engine"
  type        = string
}

variable "engine_version" {
  description = "The engine version to use"
  type        = string
}

variable "instance_class" {
  description = "The instance type of the RDS instance"
  type        = string
}

variable "allocated_storage" {
  description = "The allocated storage in gibibytes"
  type        = number
}

variable "backup_retention" {
  description = "The days to retain backups for"
  type        = number
  default     = 7
}

variable "environment" {
  description = "The deployment environment"
  type        = string
}

# modules/rds/main.tf
resource "random_password" "db_password" {
  length  = 16
  special = true
}

resource "aws_db_instance" "main" {
  identifier     = var.identifier
  engine         = var.engine
  engine_version = var.engine_version
  instance_class = var.instance_class

  allocated_storage       = var.allocated_storage
  max_allocated_storage   = var.allocated_storage * 2
  storage_encrypted       = true

  db_name  = "mycoco"
  username = "dbadmin"
  password = random_password.db_password.result

  backup_retention_period = var.backup_retention
  backup_window          = "03:00-04:00"

  # Production-only features
  deletion_protection = var.environment == "production" ? true : false
  skip_final_snapshot = var.environment != "production" ? true : false

  tags = {
    Environment = var.environment
    ManagedBy   = "terraform"
  }
}

# modules/rds/outputs.tf
output "endpoint" {
  description = "The connection endpoint"
  value       = aws_db_instance.main.endpoint
}

output "database_name" {
  description = "The database name"
  value       = aws_db_instance.main.db_name
}

output "instance_id" {
  description = "The RDS instance ID"
  value       = aws_db_instance.main.id
}

Environment-Specific Variables

# envs/production.tfvars
region               = "ca-central-1"
enable_monitoring    = true
enable_backups       = true
multi_az            = true
performance_insights = true

# envs/staging.tfvars
region               = "ca-central-1"
enable_monitoring    = true
enable_backups       = true
multi_az            = false
performance_insights = false

# envs/dev.tfvars
region               = "ca-central-1"
enable_monitoring    = false
enable_backups       = false
multi_az            = false
performance_insights = false

Workspace Management Workflow

# List available workspaces
$ terraform workspace list
  default
  dev
* staging
  production

# Switch to production
$ terraform workspace select production

# Deploy with workspace-specific config
$ terraform apply -var-file="envs/$(terraform workspace show).tfvars"

Results: MyCoCo's Workspace Transformation

The impact of adopting workspaces with modules was immediate:

70% Reduction in Maintenance Time: Changes that previously required updating three separate codebases now happened in one place. Sam noted:

"I make one change, test it in dev workspace, promote through staging, and apply to production—all with the same code."

Zero Environment Drift: With all environments using the same modules, configuration drift became impossible. The only differences were intentional, defined in workspace-specific variables.

Rapid Environment Provisioning: When MyCoCo needed a disaster recovery environment, Jordan created it in two hours: terraform workspace new dr-east && terraform apply. Previously, this would have taken days of copying and modifying configurations.

Clear Environment Isolation: Maya, the security engineer, appreciated the built-in separation:

"Each workspace has its own state file. There's no risk of accidentally modifying production while working in dev."

The real validation came during their next security audit. Instead of reviewing three sets of configurations, auditors examined one modular codebase with clear environment separation through workspaces.

Key Takeaways

Start with Workspaces: Before diving into complex module structures, implement workspaces to separate your environments while using the same code.
Use terraform.workspace: Reference the current workspace in your configurations for dynamic resource naming and conditional logic.
Combine with Modules: Workspaces handle environment separation; modules handle code reuse. Together, they eliminate duplication.
Workspace-Specific Variables: Store environment-specific values in separate tfvars files, loaded based on the current workspace.
State Isolation: Each workspace maintains its own state file, providing natural environment isolation without complex backend configurations.
Documentation is Critical: Each module needs clear documentation of required and optional variables, making adoption straightforward. Leverage terraform-docs to automate this process.
Consider Community Modules: For complex RDS setups, consider using established modules like terraform-aws-modules/rds which provides workspace-aware configurations out of the box.

Conclusion

For teams drowning in duplicate Terraform configurations, workspaces combined with modules offer a path to sanity. The investment in refactoring pays immediate dividends in reduced maintenance, eliminated drift, and happier engineers.

The key is starting simple: implement workspaces first, then gradually extract common patterns into reusable modules. Don't try to build the perfect module structure upfront—let your actual usage patterns guide the abstraction.

Ready to transform your multi-environment Terraform? Start with workspaces, add modules, and watch your infrastructure management complexity disappear.

What's your experience with Terraform workspaces and modules? Have you struggled with managing multiple environments or found better approaches? Share your infrastructure-as-code wins and challenges in the comments below!

Why Platform Teams Are Burning Out: How MyCoCo Escaped the Developer Experience Trap

Dhruv Chaudhary — Fri, 19 Sep 2025 16:38:15 +0000

Platform engineering promised to solve developer productivity challenges by creating self-service infrastructure. But at many organizations, platform teams are drowning in complexity they've absorbed from developers. MyCoCo discovered their platform team became the new bottleneck—working 60-hour weeks while being blamed for slowing everyone down. Here's how they transformed platform engineering from a burnout factory into a sustainable practice.

TL;DR

The Problem: Platform teams absorb all infrastructure complexity while becoming the scapegoat for deployment delays.

The Solution: Golden paths, Team Topologies, clear SLAs, and project inception frameworks.

The Impact: MyCoCo reduced platform team stress by 50% and increased deployment velocity 3x.

Key Implementation: Pre-approved Terraform patterns covering 80% of use cases with architectural education.

Bottom Line: Stop trying to be everything to everyone—constrain flexibility to preserve sanity.

The Challenge: MyCoCo's Platform Team Meltdown

Jordan had been leading MyCoCo's platform engineering initiative for nine months when they finally admitted defeat:

"We're not building a platform," they told Alex during their one-on-one. "We're running an IT helpdesk that happens to use Terraform."

The numbers told the story. MyCoCo's platform team of three engineers supported 50 developers across five product teams. They'd built what seemed like a modern setup: GitHub Actions for CI/CD, an LLM-based code reviewer for infrastructure PRs, and standardized Terraform modules for AWS deployments. Yet the team was drowning.

Every infrastructure PR became a potential disaster. One developer's "simple fix" to update a security group rule would have deleted and recreated the entire security group, causing a 20-minute production outage. Another's "minor IAM adjustment" affected 15 other services. The AI code reviewer caught syntax errors but couldn't understand business context or blast radius.

Worse, developers constantly bypassed the intake process. Jordan's Slack was a stream of "URGENT: Need this deployed NOW!" messages. Sprint planning became fiction—the team spent every day firefighting requests from DMs, hallway conversations, and executive escalations.

The breaking point came when a developer complained:

"I can deploy a Lambda in 5 minutes on my personal AWS account. Why does it take your team 3 days?"

They didn't see that their "simple" Lambda required updating three Terraform modules, modifying VPC configurations, creating new IAM patterns, updating monitoring dashboards, and committing to support this variation forever.

The Solution: From Chaos to Sustainable Platform Engineering

MyCoCo's transformation started with a hard truth: unlimited flexibility means unlimited support burden. Jordan's team implemented five key changes:

1. Golden Paths Over Infinite Options

Instead of supporting every possible infrastructure pattern, MyCoCo created pre-approved Terraform modules for common use cases. These "golden paths" covered 80% of developer needs while encoding all enterprise requirements automatically.

Rather than letting developers architect networking and compute from scratch, MyCoCo offered three standardized deployment patterns:

Standard VPC and Networking: Multi-AZ setup with public/private subnets, NAT gateways, VPC endpoints, and enterprise-compliant security groups
Standard Linux EC2: Application-ready instances with SSM access, CloudWatch agent, security patching, backup policies, and proper IAM roles
Standard ECS Serverless: Fargate deployments with ALB integration, service discovery, auto-scaling policies, and centralized logging

Each golden path included cost allocation tags, security scanning, compliance controls, and monitoring—all pre-configured. Developers got consistent, secure infrastructure while the platform team eliminated architectural debates and one-off configurations entirely.

2. Team Topologies for Clear Boundaries

Following Team Topologies principles, MyCoCo established clear interaction modes:

X-as-a-Service: Golden path modules with no customization
Collaboration: Scheduled pairing for complex requirements
Facilitation: Office hours for questions and guidance

No more random Slack interruptions. Developers knew exactly how and when to engage the platform team.

3. Transparent Service Level Expectations

Published SLAs set clear expectations:

Standard patterns (using golden paths): 2 business days
Custom configurations: 5 business days
New patterns requiring platform changes: 10 business days

"Urgent" requests dropped 80% once teams realized proper planning yielded faster results.

4. Project Inception Requirements

Borrowing from agile methodologies, any project requiring infrastructure now needed a kick-off with platform team involvement. This revealed infrastructure needs early, preventing mid-sprint surprises.

A simple template captured requirements:

What are you building?
Which golden paths fit your needs?
What's unique about your requirements?
When do you need infrastructure ready?

5. Infrastructure Architecture Reviews

The platform team instituted monthly "Infrastructure Deep Dives" where they walked through real production incidents and changes. Instead of abstract documentation, they showed actual scenarios:

Case Study Session Example:

"Here's last week's 'simple' Lambda request. Let me show you what actually happened behind the scenes..."

Step 1: New subnet allocation in three VPCs (dev, staging, prod)
Step 2: IAM role creation with 14 policy attachments for scoped permissions
Step 3: Updating shared security groups across 6 availability zones
Step 4: Modifying 3 Terraform modules used by 20 other services
Step 5: Cost allocation tag updates affecting monthly billing reports

Developers finally understood why their personal AWS experience didn't translate. One developer commented:

"I had no idea changing one Lambda meant touching 30 other resources. Now I get why you push golden paths so hard."

These sessions became so popular that developers started attending voluntarily, leading to better infrastructure proposals and fewer "urgent" requests.

Results: MyCoCo's Platform Renaissance

The transformation took three months, but the results were dramatic:

Quantitative Improvements

Platform team overtime dropped from 60 to 40 hours weekly
Standard deployments accelerated from 3 days to 4 hours
Urgent requests decreased by 80%
Developer satisfaction scores increased 35%

Qualitative Changes

Platform engineers felt valued instead of blamed
Developers understood infrastructure complexity better
Executive pressure shifted from "work faster" to "work smarter"

Most importantly, the platform team could finally focus on innovation instead of firefighting. They had time to improve golden paths, automate more processes, and even tackle technical debt from the early "build everything custom" days.

Sam, who'd been skeptical of platform engineering after years of manual DevOps work, admitted:

"I finally have time to think about improvements instead of just surviving each day."

Key Takeaways

Platform engineering isn't about building a portal or implementing the latest tools. It's about sustainable practices that balance developer autonomy with operational reality.

Start with Constraints: Less flexibility means less support burden. Your platform team can't be everything to everyone.
Invest in Education: Developers aren't trying to make your life difficult—they genuinely don't understand enterprise infrastructure complexity.
Set Clear Boundaries: Clear interaction patterns and SLAs prevent platform teams from becoming an always-on helpdesk.
Measure Team Health: Track platform team health alongside developer productivity metrics.
Focus on the 80%: Golden paths that handle most use cases are more valuable than infinite customization options.

Conclusion

Platform engineering success isn't measured by how many features you can build or how much customization you can support. It's measured by whether your team can sustainably deliver value without burning out.

The goal isn't to make developers happy in the short term—it's to create a system that works for everyone in the long term. Sometimes that means saying no to custom requests and guiding teams toward proven patterns instead.

Ready to rescue your platform team from burnout? Start by auditing how many infrastructure patterns you're supporting and ask: could 80% of these use a golden path instead?

What's your experience with platform team burnout? Have you successfully implemented golden paths or struggled with similar challenges? Share your platform engineering wins and failures in the comments below!

How MyCoCo Eliminated $180K in Zombie Infrastructure: The Framework That Transformed Their Resource Lifecycle Management

Dhruv Chaudhary — Fri, 19 Sep 2025 16:36:48 +0000

Growing technology companies often accumulate "zombie infrastructure"—resources that were provisioned for specific projects or experiments but never properly decommissioned when no longer needed. Without clear processes for tracking resource ownership and lifecycle, DevOps teams become reactive firefighters while costs spiral upward. MyCoCo's systematic approach to infrastructure lifecycle management eliminated $180,000 in annual waste while establishing sustainable cross-team collaboration that scales with business growth.

TL;DR

The Problem: MyCoCo had no systematic process for managing infrastructure lifecycle, leading to abandoned resources, unclear ownership, and reactive cost management as the company scaled from startup to enterprise.

The Solution: Implemented RACI accountability framework with weekly operational reviews, monthly strategic planning, and structured decommissioning processes coordinated between DevOps, Product, and Finance teams.

The Impact: MyCoCo eliminated $180K annually in unused resources, reduced infrastructure decision-making time by 60%, and transformed reactive operations into proactive lifecycle management.

Key Implementation: RACI matrices for resource ownership, comprehensive provider-level tagging with team-based identifiers, and automated discovery using Cloud Custodian policies.

Bottom Line: Systematic lifecycle management prevents infrastructure sprawl while enabling sustainable scaling—essential for companies transitioning from startup to enterprise operations.

The Challenge: MyCoCo's Infrastructure Sprawl Crisis

By late 2024, MyCoCo had reached 60+ employees across five product lines. What started as a simple project management tool had evolved into a comprehensive SaaS platform serving Fortune 500 customers across healthcare, finance, and retail verticals. But success brought unexpected challenges.

"Every week, I'm getting requests for new environments, new integrations, new regions," Sam reflected during a team retrospective. "But when was the last time anyone asked me to shut something down?"

The numbers told the story. Jordan's quarterly cost analysis revealed concerning trends: infrastructure spending had grown 300% over 18 months, but active application usage had only increased 150%. Somewhere in their AWS accounts, significant resources were running without clear business justification.

The breaking point came during a routine security audit when Maya discovered development environments from discontinued features still consuming production-grade resources. A proof-of-concept integration with a canceled partner was running three EC2 instances in multiple regions. Load testing infrastructure from six months ago was still provisioned "just in case."

"We're paying for the infrastructure equivalent of zombie apocalypse," Alex observed during the executive team meeting. "Resources that should be dead but keep consuming our budget."

The root problem wasn't technical—it was organizational. Product teams would request infrastructure for new features or experiments, but no clear process existed for determining when resources could be safely decommissioned. DevOps handled the technical provisioning, but Product Owners made business decisions about feature continuation. Finance tracked overall spending but couldn't map costs to specific business initiatives.

Without systematic lifecycle management, MyCoCo was hemorrhaging money on infrastructure that no longer served business purposes.

The Solution: MyCoCo's Systematic Lifecycle Framework

Rather than implementing expensive tooling, MyCoCo focused on organizational process improvements that would scale with their growing team. The solution centered on three core components: clear accountability, structured communication, and systematic decommissioning.

RACI Accountability for Infrastructure Decisions

MyCoCo established RACI (Responsible, Accountable, Consulted, Informed) matrices that eliminated confusion about infrastructure ownership. DevOps Engineers remained responsible for technical provisioning and maintenance while Product Owners became accountable for business decisions including feature lifecycle and resource needs. Finance teams owned cost optimization and budget compliance, with everyone staying informed about major changes.

Systematic resource tagging became essential for financial allocation, incident escalation, and business alignment. Without proper tagging, teams waste hours determining resource ownership and accountability while Finance struggles with accurate cost allocation.

Critical Enterprise Practice: MyCoCo used team-based and role-based identifiers rather than individual names for Owner, CreatedBy, and BusinessContact tags. Just like using service emails (devops@mycoco.com) instead of individual email addresses, this approach ensured tags remained valid when team members changed roles or left the organization.

MyCoCo's Essential Infrastructure Tags

Tag Name	Purpose	Example Values	Why Essential
Owner	Identifies the team responsible for ongoing maintenance and decisions	team-analytics, team-platform	Essential for escalation and accountability
CostCenter	Maps resources to budget allocation for financial tracking and chargeback	cost-center-analytics, cost-center-platform	Required for accurate cost allocation
Environment	Distinguishes dev/staging/production resources	dev, staging, production	Lifecycle management and risk assessment
BusinessUnit	Groups resources by organizational structure	product-eng, data-platform, security	Executive reporting and portfolio management
Application	Identifies specific applications or services	mycoco-projects, mycoco-analytics	Dependency mapping and impact analysis
Project	Links resources to business initiatives	q4-migration, customer-portal-v2	ROI tracking and project cost management
CreatedBy	Tracks which team provisioned the resource	devops-team, platform-team, security-team	Operational context and troubleshooting
BusinessContact	Provides service email for business decision escalation	analytics-team@mycoco.com, platform@mycoco.com	Business decision escalation when technical teams unavailable
DecommissionBy	Sets planned lifecycle end date	2024-12-31, 2025-06-15 (ISO 8601 format)	Proactive resource management and cost optimization
Criticality	Defines business impact level	critical, high, medium, low	Incident prioritization and maintenance planning
ManagedBy	Indicates management tool	terraform, manual, cloudformation	Operational context and change control procedures

MyCoCo implemented this comprehensive tagging through provider-level configuration, ensuring that every resource automatically inherited consistent ownership and lifecycle metadata:

# Provider-level tagging strategy ensuring ALL resources have ownership tracking
provider "aws" {
  region = "ca-central-1"

  default_tags {
    tags = {
      Owner           = var.default_owner
      CostCenter      = var.cost_center
      Environment     = var.environment
      BusinessUnit    = var.business_unit
      Application     = var.application_name
      Project         = var.project_name
      CreatedBy       = var.created_by_team
      BusinessContact = var.business_contact
      DecommissionBy  = var.default_sunset_date
      Criticality     = var.criticality_level
      ManagedBy       = "terraform"
    }
  }
}

This provider-level tagging strategy ensured that every resource including EC2 instances, RDS databases, S3 buckets, and Lambda functions automatically inherited consistent ownership and lifecycle metadata.

Stakeholder Visibility and Self-Service Access

To provide visibility for stakeholders, MyCoCo created an auditor role with readonly permissions accessed through SSO. This self-service approach eliminated the weekly "who owns this resource?" questions by giving Product Owners and Finance teams direct access to resource inventory and cost data.

Stakeholders use AWS Resource Groups and Cost Explorer with tag filters to view their specific resources and costs:

# Example: Analytics team viewing their resources via AWS CLI (readonly access)
aws resourcegroupstaggingapi get-resources \
  --tag-filters Key=Owner,Values=team-analytics Key=Project,Values=analytics \
  --resource-type-filters EC2 RDS S3

# Results show all resources owned by analytics team
# - i-0abc123 (EC2 instance, environment=production)
# - db-xyz789 (RDS instance, environment=staging)
# - analytics-data-bucket (S3 bucket, environment=production)

Structured Communication Cadences

MyCoCo implemented three meeting rhythms that transformed ad-hoc coordination into systematic planning:

Weekly operational reviews (30 minutes): utilization metrics, cost anomalies, and upcoming resource needs
Monthly strategic planning sessions (90 minutes): capacity planning, budget variance analysis, and technology roadmap alignment
Quarterly business reviews (half-day): lifecycle optimization and process improvements

The key insight was treating these meetings as investment reviews rather than status updates, focusing on ROI and business value rather than purely technical metrics.

Automated Resource Discovery and Systematic Decommissioning

MyCoCo developed a systematic approach to resource retirement that prevented both premature shutdowns and indefinite resource sprawl.

To automate the discovery of resources approaching their decommission dates, MyCoCo implemented Cloud Custodian - an open-source tool for managing cloud resources through policy-as-code:

# Cloud Custodian policy for automated discovery
policies:
  - name: identify-unused-instances
    resource: aws.ec2
    filters:
      - "tag:DecommissionBy": present
      - type: value
        key: "tag:DecommissionBy"
        value_type: date
        op: less-than
        value_type: age
        value: 7
    actions:
      - type: notify
        transport:
          type: sns
          topic: arn:aws:sns:us-east-1:123456789:infrastructure-alerts

When Cloud Custodian runs this policy, it generates output identifying specific resources that should be reviewed for decommissioning:

Found 12 resources for policy identify-unused-instances:
- i-0abc123def456789 (project-alpha-dev) - DecommissionBy: 2024-08-15
- i-0def456abc123789 (feature-beta-test) - DecommissionBy: 2024-08-12
- i-0789123abc456def (integration-poc) - DecommissionBy: 2024-08-10

Resources identified by Cloud Custodian are then discussed during the structured communication cadences, ensuring that business context and technical dependencies are properly evaluated before any decommissioning decisions are made.

Results: MyCoCo's Infrastructure Transformation

The systematic approach delivered immediate and sustained improvements:

Cost Reduction: MyCoCo eliminated $180,000 in annual infrastructure waste within the first quarter, primarily from development environments and discontinued feature infrastructure that had been running indefinitely.

Operational Efficiency: Infrastructure requests were resolved 60% faster through clear ownership and approval processes. Product teams gained visibility into infrastructure costs associated with their features, leading to more informed technical decisions and natural cost consciousness.

Team Impact: DevOps team stress decreased significantly as reactive "emergency" requests became planned initiatives coordinated through structured processes. Product Owners developed infrastructure awareness that improved their technical roadmap planning.

Most importantly, the framework scaled with business growth. As MyCoCo expanded into new markets and launched additional product lines, the lifecycle management processes handled increased complexity without breaking down.

Six months after implementation, Maya's security audits revealed zero zombie infrastructure, and Finance teams could accurately map 95% of infrastructure costs to specific business initiatives.

Key Takeaways

Start with Organizational Process: Technical tools cannot fix unclear accountability or poor communication. Establish RACI matrices and communication cadences before implementing automation.
Implement Provider-Level Tagging: Use Terraform default_tags to ensure ALL resources inherit ownership and lifecycle metadata automatically. This eliminates manual tagging gaps and enables comprehensive automated discovery.
Enable Stakeholder Self-Service: Provide readonly access to resource inventory and cost data through auditor roles, eliminating DevOps dependency for basic visibility while empowering teams to track their own infrastructure footprint.
Automate Discovery, Not Decisions: Use Cloud Custodian and tagging strategies to identify optimization opportunities, but maintain human oversight for business impact assessment and final approval.
Treat Infrastructure as Investment Portfolio: Regular review cycles with Finance, Product, and DevOps teams ensure resources align with business priorities and ROI expectations.
Scale Process with Team Growth: Systematic lifecycle management becomes more valuable as teams grow, preventing the organizational chaos that destroys productivity at enterprise scale.

Conclusion

For teams managing infrastructure across multiple product lines, lifecycle management transforms reactive operations into strategic capability that enables sustainable scaling. The key is starting with clear accountability and structured communication—the automation becomes straightforward once ownership is established.

Ready to eliminate infrastructure waste in your organization? Start with clear accountability frameworks and structured communication processes. Your bottom line will thank you.

What's your biggest challenge with infrastructure lifecycle management? Have you dealt with zombie resources at your company? Share your experiences and strategies in the comments below!

Breaking the Hero Culture: Why Sam's Burnout Nearly Killed MyCoCo's Growth

Dhruv Chaudhary — Fri, 19 Sep 2025 16:30:03 +0000

When your best engineer becomes your biggest bottleneck, company growth stalls and critical knowledge becomes dangerously concentrated. MyCoCo learned this lesson when their "DevOps hero" Sam—who single-handedly kept infrastructure running—burned out during a critical growth phase. By dismantling their hero culture and building sustainable team practices, they transformed from a fragile single-point-of-failure operation to a resilient engineering organization that could actually scale.

TL;DR

The Problem: Sam, MyCoCo's original DevOps engineer, had become the single point of failure—working 70-hour weeks, sole keeper of infrastructure knowledge, and the only person who could fix production issues.

The Crisis: Sam's burnout during a major product launch left MyCoCo unable to deploy fixes for 4 days, nearly losing their largest enterprise customer.

The Solution: Systematic knowledge distribution through documentation requirements, forced rotation of responsibilities, and cultural shift from rewarding heroes to building resilient teams.

The Impact: Reduced incident resolution time by 60%, eliminated single points of failure, and Sam actually took his first real vacation in 3 years.

Bottom Line: If one person's absence can cripple your operations, you're not building a company—you're building a house of cards.

The Challenge: Sam's Impossible Position

By late 2023, Sam had evolved into MyCoCo's superhero. As employee #3 and the original "DevOps person," he'd built every piece of infrastructure from scratch. He knew why that strange cron job ran at 3:47 AM (it avoided a rate limit issue from 2021). He understood the intricate dance of microservices that somehow kept MyCoCo Support running despite architectural decisions everyone now regretted.

"Sam will know" became the company motto. Mysterious production issue? Sam fixed it. Deployment failed? Sam had a workaround. Customer-specific infrastructure need? Sam would handle it over the weekend.

The warning signs were obvious in hindsight. Sam's Slack status permanently showed 🔥. He answered infrastructure questions at 11 PM while ostensibly watching Netflix. His "quick fixes" accumulated into an elaborate system only he understood. His vacation days rolled over year after year, unused because "what if something breaks?"

Maya (Security Engineer) noticed it during her security audit prep:

"Sam, who else knows how to rotate these certificates?" she asked. His pause said everything. "I've been meaning to document that..."

Alex (VP of Engineering) saw it in sprint planning. Every infrastructure task had Sam's name:

"We can't parallelize if everything goes through one person," he pointed out. Sam's response: "It's faster if I just do it myself than explain the whole history."

Then came Black Friday 2023. MyCoCo's biggest sales event. Sam had been working 16-hour days preparing. On Thursday night, exhausted and making mistakes, he pushed a config change that broke production deployments. Then his laptop died. The replacement wouldn't arrive until Monday.

For four days, MyCoCo couldn't deploy fixes. Customer tickets piled up. Their largest enterprise client threatened to leave. Jordan (Platform Engineer) and the platform team tried to help but couldn't decipher Sam's intricate web of bash scripts, environment variables, and "temporary" workarounds.

Drew (CTO) finally called an emergency leadership meeting:

"We've built our entire infrastructure on one person's brain. This isn't sustainable—it's negligent."

The Solution: Dismantling Hero Culture

The transformation started with an uncomfortable truth: MyCoCo had been rewarding the wrong behavior. They celebrated Sam's heroics in all-hands meetings. They praised his weekend work. They had inadvertently created a culture where being indispensable was valued over building resilient systems.

Phase 1: Immediate Knowledge Transfer (Week 1-2)

Alex instituted "Sam shadowing sessions." Every task Sam touched, someone watched and documented. Not detailed documentation—just enough to prevent total helplessness. Jordan paired with Sam on every production fix, asking "why" at each step.

Phase 2: Forced Distribution (Week 3-8)

The hardest change: Sam was banned from being first responder. When production issues arose, others had to try first. Sam could advise but not touch the keyboard. The first week was painful—a 15-minute Sam fix became a 2-hour team effort. But each incident built team knowledge.

They instituted "documentation-driven operations"—if it wasn't documented, it didn't exist. No more "Sam knows" or "ask Sam." Every system needed a runbook that a new engineer could follow.

Phase 3: Cultural Reset (Week 9-16)

Drew changed how they recognized work. Instead of celebrating weekend heroics, they celebrated knowledge sharing. The monthly MVP award went to whoever best distributed their expertise. On-call rotations became mandatory—everyone, including Alex, took shifts.

They implemented "chaos days"—randomly chosen team members had to handle infrastructure tasks they'd never done before, with documentation as their only guide. Gaps in knowledge became visible and fixable before they became crises.

Most importantly, they normalized boundaries. Sam was required to take vacation—a real one, with laptop left at home. The first time, he checked Slack every hour. By the third vacation, he actually relaxed.

Results: From Fragility to Resilience

Six months later, MyCoCo's infrastructure culture had transformed:

Incident Response: Mean time to resolution dropped from 2 hours to 45 minutes—because five people could investigate in parallel instead of waiting for Sam.

Deployment Velocity: Ship rate increased 40% as infrastructure work no longer bottlenecked through one person.

Team Growth: They successfully onboarded three new platform engineers who became productive within weeks, not months.

The Sam Factor: Sam rediscovered why he loved technology. Instead of fighting fires, he led architectural improvements. His stress levels dropped. He took a two-week vacation to visit family in Germany—and production didn't even hiccup.

The real test came during their Series B due diligence. When investors asked about infrastructure bus factor, Drew could confidently say:

"Any three of our eight platform engineers could rebuild our entire system from documentation."

Key Takeaways

Hero Culture is Organizational Debt: Every time you celebrate someone working weekends to save the day, you're borrowing against your future resilience.
Knowledge Hoarding Isn't Job Security: Sam thought being indispensable made him valuable. In reality, it trapped him in operational work instead of strategic growth.
Sustainable Pace Beats Heroic Sprints: MyCoCo now ships more with nobody working weekends than they did during Sam's 70-hour weeks.
Documentation is Cheaper Than Burnout: The time invested in runbooks and knowledge sharing paid for itself the first time someone other than Sam resolved a production issue.
Resilience Requires Redundancy: If your organization can't survive someone's two-week vacation, you're not building a company—you're managing a crisis waiting to happen.

Conclusion

Breaking hero culture isn't about diminishing individual contributions—it's about building systems that amplify everyone's impact. Sam didn't become less valuable when knowledge was distributed; he became free to work on problems that actually required his expertise.

Ready to break your own hero culture? Start by identifying your critical knowledge silos and implementing documentation requirements for every operational task. Your "heroes" will thank you, and your business will become truly resilient.

What's your experience with hero culture at your company? How did you handle knowledge silos? Share your thoughts in the comments below!

How AWS Nuke Saved MyCoCo $10K Monthly by Automating Resource Cleanup

Dhruv Chaudhary — Fri, 19 Sep 2025 16:19:33 +0000

When MyCoCo's AWS bill jumped from $18K to $28K monthly due to forgotten non-production resources and abandoned team sandbox experiments, they discovered that manual cleanup wasn't scalable for a growing engineering organization. AWS Nuke became their automated solution for systematically destroying unused infrastructure while protecting critical resources. Here's how they implemented safe, automated resource cleanup that reduced costs by over one-third without risking production.

TL;DR

The Problem: Development teams creating and forgetting AWS resources in non-production accounts (development and staging), plus abandoned team sandbox experiments, causing $10K monthly waste across 15 AWS accounts.

The Solution: AWS Nuke with strict configuration rules, account filtering, and automated scheduling to safely destroy unused resources in non-production and sandbox environments.

The Impact: MyCoCo reduced AWS costs by 36%, eliminated manual cleanup overhead, and improved non-production environment lifecycle management.

Key Implementation: Account-specific Nuke configurations with production safeguards, resource filtering, and CI/CD integration for scheduled cleanup across non-production and sandbox accounts.

Bottom Line: Automate resource destruction with the same rigor as resource creation - AWS Nuke makes infrastructure cleanup safe and systematic.

The Challenge: MyCoCo's Resource Sprawl Crisis

The wake-up call came during MyCoCo's monthly cost review. Sam (Senior DevOps Engineer) stared at an AWS bill that had grown by 56% in three months without any corresponding increase in customer traffic.

"We have hundreds of unattached EBS volumes accumulating across our accounts," Sam announced during the engineering meeting. "Our non-production environments have snapshots dating back months that no one remembers creating. I found 50GB of unattached storage that's been sitting unused for months in one of our development accounts."

But the non-production accounts were only part of the problem. MyCoCo had embraced a team-based sandbox account strategy where each product team received their own dedicated AWS account for experimentation and learning. While this improved security isolation and prevented accidental cross-contamination between teams, it also created a resource sprawl nightmare.

Jordan (Platform Engineer) dug deeper into the cost analysis:

"68% of our AWS spend is in non-production and sandbox accounts. The team sandbox accounts alone are costing us $7K monthly with forgotten compute resources. We're paying $1,800 monthly for RDS instances that haven't had a connection in weeks. There's a $600/month Elasticsearch cluster someone created for a POC that was abandoned. One team's sandbox account has been running a GPU instance for machine learning experiments since January - that's $2,400 we forgot about."

The sandbox problem was particularly challenging because these accounts were designed for team experimentation. Product teams would spin up expensive resources to test new technologies, complete their evaluation, and move on to other projects. Unlike non-production environments that followed project lifecycles, team sandbox experiments had no natural cleanup triggers.

The manual cleanup process was broken. Teams would create resources for testing, finish their work, and forget to delete them. The monthly "cleanup reminders" were ignored because manually identifying and deleting resources was time-consuming and error-prone. Worse, team members were afraid of deleting resources that might still be needed by teammates.

Maya (Security Engineer) raised the compliance concern:

"We have orphaned resources with overly permissive security groups sitting in forgotten accounts. These are security risks we're paying for."

The Solution: AWS Nuke with Surgical Precision

MyCoCo's solution centered on AWS Nuke - a tool designed to systematically delete AWS resources based on configurable rules. But implementing it safely required recognizing that different account types need fundamentally different approaches: non-production environments support ongoing project work and need persistence, while sandbox accounts are for experimentation and benefit from aggressive cleanup.

Step 1: Account Protection and Strategy

First, they established clear account boundaries with absolute protection for production:

# nuke-config.yaml - Account protection and strategy
regions:
  - ca-central-1
  - us-east-2
  - global

blocklist:
  - "111111111111" # Production account - absolute protection
  - "666666666666" # Shared services account

accounts:
  "222222222222": # Staging
    presets:
      - "non-prod-minimal-cleanup"
  "333333333333": # Development
    presets:
      - "non-prod-minimal-cleanup"
  "444444444444": # Team Sandbox A
    presets:
      - "sandbox-aggressive"
  "555555555555": # Team Sandbox B
    presets:
      - "sandbox-aggressive"

Step 2: Non-Production Account Strategy - Minimal Cleanup Approach

Non-production accounts support active projects that span weeks or months. Sam designed a minimal approach that protects all managed infrastructure while only cleaning up clearly abandoned manual storage:

presets:
  non-prod-minimal-cleanup:
    filters:
      __global__:
        # Manual protection tag for unmanaged resources that need retention
        - property: "tag:ignore-nuke"
          value: "true"
        # Protect all managed infrastructure
        - property: "tag:CreatedBy"
          value: "terraform"

      # For unmanaged EBS volumes: only delete if unattached AND older than 1 week
      EBSVolume:
        - property: "State"
          value: "attached" # Protect all attached volumes
        - property: "CreateTime"
          type: "dateOlderThan"
          value: "168h"
          invert: true # Protect volumes newer than 1 week

      # For unmanaged snapshots: only delete if older than 1 week
      EBSSnapshot:
        - property: "StartTime"
          type: "dateOlderThan"
          value: "168h"
          invert: true # Protect snapshots newer than 1 week

This approach protects all managed infrastructure (Terraform resources with the CreatedBy=terraform tag configured at the provider level) and all manually protected resources. Only unmanaged storage resources that are clearly abandoned get cleaned up - unattached EBS volumes older than 1 week and unmanaged snapshots older than 1 week.

Step 3: Sandbox Account Strategy - Aggressive Experimentation Cleanup

Sandbox accounts are different - they're designed for short-term experiments and POCs. Here, aggressive cleanup actually helps teams by removing the cognitive load of manual resource management:

presets:
  sandbox-aggressive:
    filters:
      # Only protect landing zone infrastructure and manually tagged experiments
      __global__:
        - property: "tag:landing-zone"
          value: "true"
        - property: "tag:ignore-nuke"
          value: "true"

This dramatically simpler configuration deletes everything except:

Landing zone infrastructure (VPCs, subnets, internet gateways) tagged during account setup
Resources manually protected by team members for longer experiments

Step 4: Automated Scheduling with Different Cadences

Jordan automated the cleanup process using separate GitHub Actions workflows for each cleanup strategy. The workflows authenticate to AWS using OpenID Connect (OIDC) with dedicated IAM roles that have the necessary permissions for AWS Nuke operations:

# .github/workflows/sandbox-cleanup.yml
name: Daily Sandbox Cleanup
on:
  schedule:
    - cron: "0 6 * * *" # Daily at 6 AM
  workflow_dispatch:

jobs:
  nuke-sandbox-accounts:
    permissions:
      id-token: write # Required for OIDC authentication
      contents: read
    strategy:
      matrix:
        account: ["444444444444", "555555555555"] # Team sandbox accounts
    steps:
      # Configure AWS credentials using OIDC
      # Download and execute AWS Nuke with configuration
      # See AWS and GitHub Actions documentation for implementation

# .github/workflows/non-prod-cleanup.yml
name: Weekly Non-Prod Cleanup
on:
  schedule:
    - cron: "0 6 * * SUN" # Weekly on Sunday at 6 AM
  workflow_dispatch:

jobs:
  nuke-non-prod-accounts:
    permissions:
      id-token: write # Required for OIDC authentication
      contents: read
    strategy:
      matrix:
        account: ["222222222222", "333333333333"] # Non-prod accounts
    steps:
      # Configure AWS credentials using OIDC
      # Download and execute AWS Nuke with configuration
      # See AWS and GitHub Actions documentation for implementation

Separate workflows provide better isolation, easier troubleshooting, and independent scheduling without complex conditional logic. The OIDC setup eliminates the need for long-lived AWS access keys while providing secure, temporary credentials for each workflow run.

Step 5: Cost Monitoring

Maya set up simple cost tracking using AWS Cost Explorer to monitor the effectiveness of their cleanup efforts, comparing monthly spending before and after implementing automated resource deletion.

To handle legitimate long-running experiments, they created a simple manual tagging strategy. Team members could simply add an ignore-nuke=true tag to any resource that needed to run longer than the standard cleanup cycle. Additionally, the landing zone infrastructure (VPCs, subnets, internet gateways, route tables) was pre-tagged with landing-zone=true during account setup to ensure basic networking remained intact. This gave teams a self-service way to protect their resources without requiring scripts or special permissions - just standard AWS resource tagging.

Results: MyCoCo's Automated Cleanup Success

The transformation was dramatic. Within three months, MyCoCo achieved unprecedented infrastructure consistency and cost control.

Financial Impact

AWS costs reduced from $28K to $18K monthly (36% reduction)
$10K in monthly savings from automated cleanup
Sandbox account costs dropped from $7K to $3K monthly
Immediate cost savings visible within the first cleanup cycle

Operational Improvements

The automated approach eliminated the manual cleanup burden entirely. Non-production teams never worried about their running infrastructure being touched - AWS Nuke only cleaned up forgotten storage. The protection tagging system gave teams control when they needed to keep resources longer.

The sandbox account cleanup was particularly transformative. The simplified approach of "delete everything except landing zone infrastructure" eliminated the complexity of managing different retention policies for different resource types. Daily automated cleanup meant teams could experiment freely without cost anxiety, knowing that only the basic networking foundation would persist.

"It's like having a responsible roommate who cleans up after everyone," Sam commented during their retrospective.

The daily sandbox cleanup cycles meant that experimental resource costs never accumulated beyond a day, while non-production environments had their compute resources completely protected with only storage waste removed weekly.

Development Velocity

Paradoxically, automated destruction improved development velocity. Teams felt more comfortable creating experimental resources knowing they would be automatically cleaned up. The fear of creating expensive infrastructure waste was eliminated.

Maya noted improved security posture:

"We went from having 150+ unknown resources scattered across accounts to maintaining clean, auditable environments. Every resource running has a purpose and an expiration date."

Key Takeaways

Use the Maintained Version: Always use the actively maintained ekristen/aws-nuke fork rather than the archived original repository. The maintained version includes security fixes and ongoing AWS service support.
Start with Account Protection: Use blocklists to absolutely protect production accounts. Never rely on filters alone to protect critical infrastructure.
Tailor Strategies by Account Type: Non-production accounts need minimal cleanup that only touches abandoned storage, while sandbox accounts benefit from aggressive "delete everything except landing zone" approaches.
Focus Age-Based Cleanup on Sandbox Accounts: Non-production environments often need to persist for ongoing project work. Reserve aggressive time-based cleanup for true sandbox/experimental accounts where resources are more likely to be forgotten.
Simple Tagging Works: Resource tagging provides comprehensive protection through three key tags:
- CreatedBy=terraform automatically protects all managed infrastructure
- ignore-nuke=true gives teams manual protection for unmanaged resources
- landing-zone=true preserves essential networking infrastructure
Monitor and Measure: Track cost savings and resource reduction to demonstrate value. Automated cleanup should show immediate cost benefits within the first cleanup cycle.
Test Extensively: Always start with dry runs and gradually implement automation. The tool's power makes careful testing essential.

Conclusion

AWS Nuke transforms resource cleanup from a manual chore into an automated process that keeps environments clean and costs predictable. The key is implementing it safely with comprehensive protection mechanisms and clear boundaries between environments.

Ready to implement automated AWS resource cleanup? Start with AWS Nuke's account protection features and gradually build comprehensive cleanup strategies for your non-production and sandbox environments.

What's your experience with AWS cost optimization? Have you tried automated resource cleanup tools? Share your thoughts in the comments below!