Forem: Santanu Das

Automating Externally-Managed Stateful Rule Groups in AWS Network Firewall Using Terraform (Part-2)

Santanu Das — Sun, 25 Jan 2026 14:26:49 +0000

In Part 1, we established the Terraform scaffolding for a resilient AWS Network Firewall, focusing on strict rule ordering and a decoupled parent-child module architecture. By defining clear boundaries between Terraform-owned and Automation-owned resources, we successfully eliminated the state drift tug-of-war for dynamic rule groups. Now, in Part 2, we move from infrastructure to orchestration to see how an event-driven, serverless pipeline manages high-frequency rule updates programmatically.

TL;DR
Externally managed stateful rule groups safely decoupled tenant-driven rule changes from Terraform-managed infrastructure, preserving IaC workflows and remote state — but automation is what makes them safe.

Architectural Overview
Engineering Walkthrough
Source Config
- Source of Truth
- S3 Event Bridge
- Cross-Account Trust
- Capturing Events
Target Config
- Custom Event Bus
- Actionable Changes
- Lambda Role/Policy
Runtime Environment
- Lambda function
- Explicit Invocation
- Pipeline wiring
The Firewall-Lambda
The Final Words

Architectural Overview

After analyzing the application’s update frequency and internal logic, it became clear that a traditional IaC approach would be insufficient. To support high-frequency rule updates without manual bottlenecks or the risk of Terraform state drift, an asynchronous, event-driven pipeline was the only scalable path forward.

This architecture creates a strict Separation of Concerns: it decouples the tenant-facing application logic from the core network security layer. By moving the heavy lifting to a dedicated automation plane, security constraints are enforced programmatically and independently of the application's lifecycle.

The following diagram illustrates the unidirectional flow from the tenant's request to the firewall's enforcement:

1️⃣ Tenant Self-Serviced Input:
Tenants submit hostname (new/update) via a their Admin UI. This is the entry point for all allowlist modifications.

2️⃣ Backend Application Processing:
The backend application first validates the input for format and ownership constraints. Once validated, it persists the record to the Application Database and immediately invokes Application Lambda function to handle the downstream synchronization.

3️⃣ Source-of-Truth Synchronization:
The application-lambda aggregates the active hosts and updates tenant_hosts.json file in one pre-configured Amazon S3 bucket. This file acts as the definitive Desired State for the firewall. The bucket is versioned to allow for immediate rollbacks and historical auditing.

4️⃣ S3 Event → EventBridge:
Critically, the application does not communicate with the firewall directly. Instead, the update to the S3 object emits an event to Amazon EventBridge. This event is filtered by bucket name and object key (tenant_hosts), ensuring the pipeline is only triggered by valid state changes.

5️⃣ Firewall Automation Lambda:
The EventBridge trigger invokes the Firewall Lambda, which serves as the authoritative control plane. This function:

Reads the updated tenant_hosts.json file from S3.
Generates the corresponding suricata-compatible rule syntax.
Performs a logical Diff against the active AWS Network Firewall configuration.
Executes an API update to the Externally Managed Rule Group only if actual changes are detected.

6️⃣ Human Visibility (ChatOps):
Once the operation is complete, the Lambda dispatches a success or failure notification to Microsoft Teams. This closes the feedback loop, providing the engineering team with real-time visibility into the automated security operations.

Engineering Walkthrough

As illustrated in the architectural diagram, this is a multi-account system. Both the Internal_Apps account and the External-Comm accounts must be configured in tandem to facilitate this cross-account event flow.

Note on Cross-Account Events: While the granular details of cross-account EventBridge bus configurations are out of scope for this article, you can find a comprehensive guide on Triggering Cross-Account Workflows here.

Source Side Config:

Generating the Source of Truth

The application-side implementation was moderately simple: Terraform provisions the application-lambda as part of the EKS-apps infrastructure and injects its metadata (ARN/ID) into the application pods as environment variables.

Whenever a tenant modifies their hostname, the backend invokes the Lambda using the ${lambda_function_arn}. The Lambda then aggregates the current state and updates the tenant_hosts.json file in the S3 bucket. This file serves as the immutable desired state for our firewall.

Configuring the S3 Event Bridge

To expose S3 activity to this automation pipeline, the bucket must be configured to publish events to Amazon EventBridge across the accounts. As S3 bucket operations are scoped to the service itself by default, S3 Event Notifications must be explicitly enabled for Amazon EventBridge.

Using aws_s3_bucket_notification resource, this can be enabled globally for the bucket. It will then allows S3 to publish Object Created events (from Internal-Apps account) directly to the EventBridge bus (in External-Comm account), eliminating the need for any intermediate Lambda functions or custom notification logic.

resource "aws_s3_bucket_notification" "eb_xacc" {
  bucket      = module.nfw_bucket.name
  eventbridge = true
  provider    = aws.eksapp #<-- required for cross-account
}

Establishing the Cross-Account Trust

For Amazon EventBridge to forward events across account boundaries, it requires explicit permission to hand-off the data. It can be achieved by provisioning a dedicated IAM Role within the Internal-Apps account that the EventBridge service is authorized to assume.

This role serves a single, highly-scoped purpose: it grants the local EventBridge service the events:PutEvents permission, specifically targeting the ARN of our central Event Bus in the External-Comm account. This setup ensures that as soon as S3 drops a notification into the local bus, EventBridge has the credentials necessary to push that event across the border to our automation pipeline.

resource "aws_iam_role" "eb_forward" {
  name     = "${local.template_name}-xacc-s3eb-Role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Effect    = "Allow",
      Principal = { "Service" : "events.amazonaws.com" },
      Action    = ["sts:AssumeRole"]
    }]
  })
  provider = aws.eksapp  #<-- if you doing cross-account
}

resource "aws_iam_role_policy" "eb_forward" {
  role     = aws_iam_role.eb_forward.id
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Effect   = "Allow",
      Action   = "events:PutEvents",
      Resource = aws_cloudwatch_event_bus.s3_trigger[each.value].arn
    }]
  })
  provider = aws.eksapp  #<-- if you doing cross-account
}

Capturing and Forwarding the Event(s)

With the permissions established, only thing was needed, a mechanism to intercept the specific S3 notifications and direct them to our cross-account destination. This is handled by the EventBridge Rule and its corresponding Target.

The EventBridge rule acts as a targeted filter on the application account’s default event bus. Instead of forwarding all S3 activities, it is configured with a highly specific event pattern that matches only Object Created events from the designated S3 bucket and restricts them further to the tenant_hosts key prefix. This precision filter prevents unnecessary Lambda invocations in the downstream account, keeping the overall design both efficient and cost-effective.

resource "aws_cloudwatch_event_rule" "eb_xacc" {
  for_each    = local.nfw_svc_enabled ? toset([var.service_name]) : []
  name        = "${local.template_name}-xacc"
  description = "Forward S3 Object Created events to EXC event bus"
  provider    = aws.eksapp  #<-- if you doing cross-account

  event_pattern = jsonencode({
    source      = ["aws.s3"],
    detail-type = ["Object Created"],
    detail = {
      bucket = {
        name = [local.xacc_s3_trigger.bucket]
      },
      object = {
        key = [
          { prefix = local.xacc_s3_trigger.object }
        ]
      }
    }
  })
}

The aws_cloudwatch_event_target then acts as the exit ramp. It connects the rule to our central Event Bus in the security account. By referencing the ${aws_iam_role.eb_forward} we created earlier, the target has the authority to cross the account boundary and deliver the payload to our central automation hub.

resource "aws_cloudwatch_event_target" "eb_xacc" {
  rule     = aws_cloudwatch_event_rule.eb_xacc.name
  arn      = aws_cloudwatch_event_bus.s3_trigger.arn
  role_arn = aws_iam_role.eb_forward.arn
  provider = aws.eksapp  #<-- if you doing cross-account
}

Target End Config:

Custom Event Bus and Permissions

In the External-Comm account, I provisioned a dedicated Custom Event Bus to isolate our firewall automation traffic from the general noise of the account, to keep the security events clean and auditable. It's as simple as defining this:

resource "aws_cloudwatch_event_bus" "s3_trigger" {
  name     = "${local.template_name}-nfw-s3-trigger"
}

However, a custom bus is closed by default. So, to receive events from the Internal-Apps account, I created a dedicated Custom EventBridge event bus in the External-Comm account. To enable cross-account delivery, I used the aws_cloudwatch_event_permission resource to create a resource-based policy (details below) to explicitly allow external-apps account (referenced via ${local.xacc_s3_trigger.acc_id}) to call PutEvents on this custom bus. In this way, the system can securely hand off S3 notifications across the account boundary without exposing the bus to the broader public.

resource "aws_cloudwatch_event_permission" "s3_trigger" {
  event_bus_name = aws_cloudwatch_event_bus.s3_trigger.name
  principal      = local.xacc_s3_trigger.acc_id
  action         = "events:PutEvents"
  statement_id   = "AllowIncToPutEvents"
}

Filtering for Actionable Changes

On this custom bus, an EventBridge rule is defined to listen specifically for S3 Object Created events, originating from the expected bucket and object key prefix. When a matching event is received, the rule triggers the firewall-lambda responsible for validation and rule group updates.

As soon as a new/updated tenant_hosts.json lands in S3 and crosses the border, this rule captures it and triggers the automation logic that keeps our firewall synchronized in real-time.

resource "aws_cloudwatch_event_rule" "s3_trigger" {
  name           = "${local.template_name}-s3-trigger"
  event_bus_name = aws_cloudwatch_event_bus.s3_trigger.name
  description    = "Trigger Lambda when object updated  in ${local.xacc_s3_trigger.bucket}"

  event_pattern = jsonencode({
    source      = ["aws.s3"],
    detail-type = ["Object Created"],
    detail = {
      bucket = { name = [local.xacc_s3_trigger.bucket] },
      object = {
        key = [{ prefix = local.xacc_s3_trigger.object }]
      }
    }
  })
}

Lambda IAM Role & Permissions

For the automation to function, the firewall-lambda needs to act as a privileged operator across a nmber of AWS services. This IAM role and policy document are specificially designed using the principle of least privilege, ensuring the function has exactly what it needs to perform the Diff and Update logic—and nothing more.

The most critical aspect of this policy is the AWS Network Firewall authority boundary. The UpdateRuleGroup and DescribeRuleGroup permissions are explicitly scoped to the A*RN of the externally managed stateful rule group* (${local.extl_suricata_rg_arn}) and nothing else. By restricting the access to a single resource, the automation pipeline is technically incapable of modifying any other Terraform-managed firewall rules or resources, even in a failure or misconfiguration scenario. Standard CloudWatch Logs permissions are also included to ensure every evaluation and decision made by the automation logic is fully auditable.

// IAM role for firewall-lambda
resource "aws_iam_role" "nfw_lambda" {
  name     = "${local.nfw_resource_prefix}-lambda-Role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = [
            "lambda.amazonaws.com"
          ]
        }
      },
    ]
  })
}

// Policy document for firewall-lambda role
data "aws_iam_policy_document" "nfw_lambda" {

  # S3 permission: read tenant-hosts.json
  statement {
    effect = "Allow"
    actions = [
      "s3:GetObject",
      "s3:GetObjectVersion",
      "s3:ListBucket"
    ]
    resources = [
      "arn:aws:s3:::${local.xacc_s3_trigger.bucket}",
      "arn:aws:s3:::${local.xacc_s3_trigger.bucket}/*",
    ]
  }

  # NFW permission: update the RSL rule group
  statement {
    effect = "Allow"
    actions = [
      "network-firewall:DescribeRuleGroup",
      "network-firewall:UpdateRuleGroup"
    ]
    resources = [
      local.extl_suricata_rg_arn
    ]
  }

  # Allow Lambda to kms:Decrypt
  statement {
    effect = "Allow"
    actions = [
      "kms:Decrypt",
      "kms:DescribeKey",
      "kms:GenerateDataKey*"
    ]
    resources = [module.nfw_bucket_key[each.value].kms_key_arn]
  }

  # CloudWatch Logs permission
  statement {
    effect = "Allow"
    actions = [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]
    resources = ["*"]
  }
}

Runtime Environment:

AWS Lambda Function

The dth_updater (Dynamic Tenant Hosts) Lambda acts as the control plane for firewall updates, with all of the required context at runtime. It runs on Python 3.12 for strong[er] Boto3 support and efficient string processing when handling Suricata rules. Critical configuration is passed via environment variables, including the ARN of the externally managed rule group, SOPS-encrypted Microsoft Teams webhook etc., enabling secure, real-time notifications without exposing sensitive endpoints.

resource "aws_lambda_function" "dth_updater" {
  filename         = data.archive_file.dthu_lambda_zip.output_path
  function_name    = "${local.template_name}-tenant-fw-updater"
  handler          = "nfw_tenant_hosts_updater.lambda_handler"
  role             = aws_iam_role.nfw_lambda.arn
  source_code_hash = data.archive_file.dthu_lambda_zip.output_base64sha256

  runtime = "python3.12"
  timeout = 30

  environment {
    variables = {
      DYNAMIC_SURICATA_RG_ARN = local.extl_suricata_rg_arn
      TEAMS_WEBHOOK_URL = lookup(
        data.sops_file.tenant_teams_wh.data,
        "teams_webhook_url",
        null
      )
      ENV_NAME     = var.aws_acc_name
      MSG_TITLE    = var.notify_teams["tenant_hosts"].title
      NOTIFY_TEAMS = var.notify_teams["tenant_hosts"].enabled
      SERVICE_NAME = var.tf_module_name
    }
  }

  depends_on = [
    aws_iam_role_policy_attachment.nfw_lambda
  ]
}

Explicit Invocation

Provisioning the Lambda and EventBridge rule alone is not sufficient; AWS services default to a zero-trust invocation model. An explicit Lambda permission is required to allow EventBridge to invoke the function.

By scoping the principal to events.amazonaws.com and restricting the source_arn to the specific S3-trigger rule, invocation is limited strictly to the events, originating from the custom event bus. This ensures the Lambda cannot be triggered accidentally or maliciously from other sources within the account.

resource "aws_lambda_permission" "s3_trigger" {
  statement_id  = "AllowEventsInvoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.dth_updater.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.s3_trigger.arn
}

Wiring the Pipeline

The final component is the EventBridge target, which connects the filtered rule to the firewall automation Lambda. The target references the dth_updater function ARN directly. I also added the explicit dependency on to ensure Lambda invoke permissions are fully propagated before the target becomes active. This avoids race conditions during initial deployment and guarantees the pipeline is ready and reliable from the first S3 object update onward.

resource "aws_cloudwatch_event_target" "s3_trigger" {
  rule           = aws_cloudwatch_event_rule.s3_trigger.name
  event_bus_name = aws_cloudwatch_event_bus.s3_trigger.name

  target_id = "tenant-fw-updater"
  arn       = aws_lambda_function.dth_updater.arn

  depends_on = [
    aws_lambda_permission.s3_trigger
  ]
}

Filewall-Lambda Function

Function Code

I have added the Lambda function shown below is provided purely as a reference implementation. The exact structure, validation logic, and rule-generation strategy will vary depending on individual requirements, tenant models, and security posture. This is just to show one of the ways of implementing an event-driven automation Lambda can safely control an externally managed Network Firewall rule group.

At a high level, this function performs the following steps:

Parses the incoming EventBridge S3 event to identify the updated object
Loads tenant-defined hostnames from S3 and expands them into fully qualified domains
Reads the current externally managed Suricata rule group from Network Firewall
Computes a deterministic diff between existing and desired state
Skips the update entirely if no effective change is detected
Rebuilds the Suricata ruleset and updates the rule group using an update token
Optional - Emits a structured notification to Microsoft Teams for visibility and auditability

This pattern keeps the Lambda focused on validation, diffing, and enforcement, while allowing upstream systems to express intent without direct access to the firewall control plane.

def lambda_handler(event, context):
    """
    Dynamic Suricata Rule-Group updater.
    Triggered via EventBridge when tenant-temp-hosts.json changes in S3.
    """

    result = {
        "stage": "start",
        "status": "ok",
        "added": [],
        "removed": [],
        "error": None,
    }

    try:
        logger.info("Event: %s", json.dumps(event))

        # 1. Parse S3 event (EventBridge style)
        # -----------------------------------------------
        result["stage"] = "parse_event"
        try:
            # This is very important to pick up the correct
            # S3 bucket and object from event JSON  
            bucket = event["detail"]["bucket"]["name"]
            key = event["detail"]["object"]["key"]
        except Exception as e:
            raise RuntimeError(f"Invalid EventBridge S3 event: {e}")

        logger.info("Processing file s3://%s/%s", bucket, key)

        # 2. Load tenants + expand to full FQDNs
        # -----------------------------------------------
        result["stage"] = "load_and_expand"
        host_map = load_domain_map(bucket, key)
        new_fqdns = expand_fqdns(host_map)

        logger.info("New expanded FQDNs: %d entries", len(new_fqdns))

        # 3. Read existing dynamic Suricata rule-group
        # -----------------------------------------------
        result["stage"] = "describe_rule_group"

        describe = nfw.describe_rule_group(
            RuleGroupArn=DYNAMIC_SURICATA_RG_ARN,
            Type="STATEFUL"
        )

        existing_desc = describe.get("RuleGroupResponse", {}).get("Description")
        rule_group_def = describe.get("RuleGroup", {})
        update_token   = describe["UpdateToken"]

        update_args = {
            "RuleGroupArn": DYNAMIC_SURICATA_RG_ARN,
            "Type": "STATEFUL",
            "UpdateToken": update_token,
            "RuleGroup": rule_group_def,
        }

        # Only include Description if AWS returned a valid string
        if isinstance(existing_desc, str) and existing_desc.strip():
            update_args["Description"] = existing_desc

        existing_rules_str = (
            rule_group_def
            .get("RulesSource", {})
            .get("RulesString", "")
        )

        old_fqdns = extract_fqdns_from_rules(existing_rules_str)

        # 4. The Diff
        # -----------------------------------------------
        result["stage"] = "diff"

        new_set = set(new_fqdns)
        added = sorted(new_set - old_fqdns)
        removed = sorted(old_fqdns - new_set)

        result["added"] = added
        result["removed"] = removed

        if not added and not removed:
            logger.info("No changes detected. RuleGroup remains unchanged.")
            result["stage"] = "no_change"
            return {
                "statusCode": 200,
                "body": json.dumps({
                    "message": "No change in tenant rules",
                    "changed": False,
                    "fqdn_count": len(new_fqdns),
                }),
            }

        logger.info("Changes detected: +%d, -%d", len(added), len(removed))

        # 5. Build new Suricata rules (stable hash SIDs)
        # -----------------------------------------------
        result["stage"] = "build_suricata_rules"
        new_rules_str = build_suricata_rules(new_fqdns)

        # 6. Insert updated rules back into RuleGroup dict
        # -----------------------------------------------
        result["stage"] = "prepare_update"
        if "RulesSource" not in rule_group_def:
            rule_group_def["RulesSource"] = {}

        rule_group_def["RulesSource"]["RulesString"] = new_rules_str

        # 7. Update rule group in Network Firewall
        # -----------------------------------------------
        result["stage"] = "update_rule_group"

        resp = nfw.update_rule_group(**update_args)
        logger.info("UpdateRuleGroup response: %s", json.dumps(resp, default=str))

        result["stage"] = "completed"

        return {
            "statusCode": 200,
            "body": json.dumps({
                "message": "Dynamic Suricata tenant allowlist updated",
                "added": added,
                "removed": removed,
                "fqdn_count": len(new_fqdns),
                "changed": True,
            }),
        }

    except Exception as e:
        logger.error("Lambda FAILED: %s", e, exc_info=True)
        result["status"] = "failure"
        result["error"] = str(e)
        raise

    finally:
        # 8. Teams notification (if enabled)
        # -----------------------------------------------
        if NOTIFY_TEAMS:
            try:
                env_name = os.environ.get("ENV_NAME", "unknown")
                svc = os.environ.get("SERVICE_NAME", "tenant-fw-updater")
                send_teams_update(
                    stage=result["stage"],
                    status=("success" if result["status"] == "ok" else "failure"),
                    added=result["added"],
                    removed=result["removed"],
                    env=env_name,
                    service_name=svc,
                    error=result["error"],
                )
            except Exception as te:
                logger.error("Teams notification failed: %s", te)

Helper Functions

For completeness, the following helper functions support the Lambda handler shown above. Python is not among my strongest skill-sets but worked for me handling domain normalization, deterministic rule generation, and extraction of existing state from the firewall rule group.

Treating S3 as the canonical desired-state source
Generating stable Suricata rule identifiers to avoid churn
Rebuilding rules deterministically from intent, rather than mutating in place

def load_domain_map(bucket: str, key: str) -> Dict[str, List[str]]:
    """Load JSON mapping of base_domain → list of tenant names."""
    obj = s3.get_object(Bucket=bucket, Key=key)
    text = obj["Body"].read().decode("utf-8")

    data = json.loads(text)
    if not isinstance(data, dict):
        raise ValueError("JSON must be a dict of { base_domain: [tenants...] }")

    out = {}
    for base, tenants in data.items():
        if not isinstance(tenants, list):
            raise ValueError(f"Value for '{base}' must be a list")
        cleaned = [t.strip().lower() for t in tenants if str(t).strip()]
        if cleaned:
            out[base.strip().lower()] = cleaned

    return out
#
def expand_fqdns(mapping: Dict[str, List[str]]) -> List[str]:
    """Expand tenant + base domain to FQDNs."""
    fqdns = set()
    for base, tenants in mapping.items():
        for t in tenants:
            fqdn = f"{t}.{base}"
            fqdns.add(fqdn)
    return sorted(fqdns)
#
def sid_for_fqdn(fqdn: str) -> int:
    """Stable hash-based SID."""
    base = 30_000_000
    h = zlib.crc32(fqdn.encode("utf-8")) & 0xFFFFFFFF
    return base + (h % 9_999_999)
#
def build_suricata_rules(fqdns: List[str]) -> str:
    """Generate Suricata rules for each FQDN."""
    rules = []
    for fqdn in fqdns:
        sid = sid_for_fqdn(fqdn)
        rule = (
            'pass tls $EXTERNAL_NET any -> $HOME_NET any ('
            f'tls.sni; content:"{fqdn}"; '
            'ssl_state:client_hello; '
            'msg:"Dynamic tenant allowed host"; '
            'flow:to_server, established; '
            'nocase; '
            f"sid:{sid}; rev:1;)"
        )
        rules.append(rule)

    return "\n".join(rules) + "\n"


CONTENT_RE = re.compile(r'content:"([^"]+)"')

#
def extract_fqdns_from_rules(rules: str) -> Set[str]:
    """Extract FQDNs from Suricata RulesString."""
    if not rules:
        return set()
    return {m.lower() for m in CONTENT_RE.findall(rules)}

At the Day's End

That's all for now, folks!

With the automated pipeline in place and cross-account EventBridge wiring established, I managed to turn our Network Firewall into a dynamic enforcement layer that responds safely to tenant-driven change. By delegating high-frequency updates to an event-driven Lambda workflow, we eliminated manual intervention and avoid the Terraform state drift that typically accompanies mutable firewall rules. As a result, now it takes only few secods to update the firewall (+ the time NFW takes to reload), compare to around 40 minitus with old HAProxy based system.

Let me know in the commanet section below, if anyone interested in the observability features (Microsoft Teams notification) that I implemented, failure handling or the long-term maintainability option of dynamic Suricata rule sets - I can make the final part of this series, focusing on those.

While I tried my best to ensure the accuracy of the code snippets provided; feel free to reach out in the comments below or via LinkedIn if you spot any discrepancies or have questions regarding the implementation.

Automating Externally-Managed Stateful Rule Groups in AWS Network Firewall Using Terraform (Part 1)

Santanu Das — Thu, 22 Jan 2026 14:37:35 +0000

TL;DR:
Managing AWS Network Firewall via Terraform often leads to a tug-of-war when external automation (like a Lambda) needs to update dynamic rules (e.g., FQDNs for ephemeral tenants).

In this post, I'll share an architecture that solves this by:

Decoupling Ownership: Splitting rule groups into "Self-Managed" (Terraform-owned) and "Custom-Managed" (Automation-owned).
Preventing State Drift: Using the lifecycle { ignore_changes = [...] } block on custom resources so Terraform ignores updates made by external APIs.
Enforcing Order: Implementing a stateful_rule_group_order list in the parent module to ensure a deterministic evaluation of Suricata, Domain Lists, and AWS-managed rules.
Scaling Gracefully: Using a placeholder strategy to initialize high-capacity rule groups that are ready for high-frequency updates without ever re-running a pipeline.

The Background
Architectural Requirements
Core Challenges
Engineering Overview
Rules of Engagement
Parent Module
Child Module
Rule Group Order
Lessons Learned
Part-2 Roadmap

Since migrating our edge security stack from a combination of F5, HAProxy, and standalone Suricata to AWS Network Firewall (NFW), the relationship has been a bit sweet-and-sour.

Having relied on Suricata for years, was excited to see that NFW supports Suricata-compatible IPS rules natively and based on that, our migration goals were very clear:

Retain Suricata Semantics: Full support for TLS SNI filtering, IDS/IPS signatures, and custom rulesets.
Preserve our Hostname Model: Maintain our existing user-defined hostname filtering logic without regression.
Pure GitOps: The entire stack must be Infrastructure-as-Code (IaC) via Terraform and Terragrunt.
Zero Manual Intervention: Eliminate "humans editing allow-lists" at all costs to ensure security and consistency.

However, the path to a fully automated, stateful firewall wasn't as linear as the documentation suggested. This article breaks down the architectural hurdles we encountered, the technical trade-offs we faced, and the specific automation patterns I used to reach the summit.

Background Overview

We operate a high-scale, multi-tenant SaaS platform. In our environment, customers can dynamically provision their own subdomains through a self-service user portal. The result is a constantly evolving list of hostnames, such as:

${TENANT_1}.examples.site
${TENANT_2}.examples.site
...

The backend application processes these tenant registrations and exposes them to the public internet as fully-fledged websites. The traffic flow follows a standard high-availability pattern: Internet → Network Load Balancer (NLB) → AWS Network Firewall (NFW).

Architectural Requirements

To support this model at scale, our firewall layer had to meet five set criteria:

L3/L4 Resilience: Robust protection against volumetric DDoS attacks.
Suricata-grade IDS/IPS: Deep packet inspection using industry-standard Suricata signatures.
Tenant-Specific Filtering: Granular hostname filtering to ensure traffic isolation and security.
Zero-Touch Onboarding: A fully automated pipeline where a new tenant registration triggers a firewall update without manual intervention.
In-line Propagation: Real-time availability for new sites to ensure a seamless user experience.

Beyond these dynamic, user-generated endpoints, the firewall must also manage a collection of static internal endpoints. These are traditional infrastructure dependencies that require a stable, code-defined allow-list within our primary firewall policy.

The Core Challenges: Dynamic Ingress at Scale

In our SaaS environment, hostnames are ephemeral — they are created (and removed) constantly. This means the ingress rule-set must be agile enough to reflect these changes in near real-time.

The hurdle here is one of context: The Infrastructure/Networking layer has no inherent knowledge of which hosts are being added or removed; that metadata lives exclusively within the Application layer. To bridge this gap, we needed a delivery mechanism that could update our Access Control Lists (ACLs) the moment a change occurred in the database.

The War: Declarative Tooling in Ephemeral Environments

Initially, the challenge was more engineering-focused than architectural. As Terraform is declarative, my first instinct was to trigger a CI/CD pipeline to run terraform apply every time a new hostname was added. This would rebuild the entire ruleset with the new entries.

However, I quickly realized this approach was unsustainable:

Latency: Running a full Terraform plan/apply cycle for every single tenant change was painfully slow.
Fragility: Frequent, automated applies increased the risk of state locks and race conditions, making the process highly error-prone

And Peace: A Hybrid Ownership Model

To solve this, I moved toward a partitioned management strategy:

Static Governance: Traditional Suricata rules (system-wide signatures and internal endpoints) remain strictly managed within Terraform.
External Agility: Dynamic tenant rules are offloaded to a separate, externally-managed stateful rule group.
Clean State: Safely automate that dynamic group via a Lambda or API call without Terraform ever knowing (or caring) that the rule content has changed.
Enforced Logic: The evaluation order of all groups — both static and dynamic — is centrally controlled via the Firewall Policy to ensure consistent security.

Engineering Overview

This workflow is based on a hybrid architecture that balances static infrastructure with dynamic security updates. The primary goal is to maintain a clean Terraform state while allowing external automation to handle high-frequency rule changes.

The diagram below is a conceptual view of the end-to-end process:

Key Architectural Concepts

Hybrid Ownership: As shown in the Management Plane, the module orchestrates four logical families of stateful rule groups. This allows for a mix of static, code-defined groups and the dynamic, externally-managed groups targeted by the Lambda Function.
Deterministic Evaluation: By enforcing STRICT_ORDER within the stateful_engine_options, we ensure that rules are evaluated in a specific sequence rather than the default Action Order.
Managed Safety Net: AWS-managed rule groups (e.g., threat signatures and botnet filters) are appended to the end of the evaluation chain, acting as a global security layer for traffic that passes our custom filters.
State Decoupling: To prevent tug-of-war deployments, Terraform initializes the dynamic groups but explicitly ignores subsequent rule changes, handing off control to external processes.

Policy Structure & Rule Families

The child module (tf-module-nfw) is responsible for building a prioritized stateful_rule_group_order. It categorizes rules into four distinct logical families, with the Evaluation Priority as listed below:

Family	Source	Ownership
`self_managed_domainlist`	FQDN Filter	Terraform: Managed via code
`self_managed_suricata`	Suricata IPS	Terraform: Managed via code
`extl_managed_suricata`	Suricata IPS	External: Initialized by TF; updated by Lambda
`extl_managed_domainlist`	FQDN Filter	External: Feed-driven updates
`aws_managed_groups`	Threat Intel	AWS: Automatically updated by AWS

This module concludes by constructing a flattened, ordered list of ARNs for the Firewall Policies, ensuring that the rules are applied in the exact sequence required by our security policy.

The Conflict Resolution

To prevent Terraform from overwriting updates made by external automation, I used a specific lifecycle strategy (as described below):

Initialize: Terraform creates the rule group shell.
Handoff: A dedicated Lambda (detailed in Part 2) updates the rules.
Ignore: Terraform is configured to ignore changes to the rules_source, ensuring our ${variable} definitions in external-system remain stable.

The Rules of Engagement

To ensure a resilient and scalable architecture, the module follows these five non-negotiable principles:

1️⃣ Terraform as the Immutable Source of Truth
The static Suricata rulesets originate within the configuration layer (Terragrunt), are rendered via Terraform, and persisted in S3 before deployment to the Network Firewall. To maintain architectural integrity, these core groups are treated as immutable; no external automation or manual intervention is permitted to modify them, ensuring a strictly code-defined security perimeter.

2️⃣ External Management for Dynamic Tenant Rules
Since tenant FQDNs are ephemeral, re-running a full Terragrunt pipeline for every hostname change is unacceptable. The dynamic rule groups must be designed for external updates (e.g., via Lambda) without triggering a conflict with the Terraform state.

3️⃣ Unified Ordering Mechanism
Firewall policy includes a diverse mix of rule types:

Static Suricata (Standard signatures)
Dynamic Suricata (Tenant-specific)
Domain List Groups (FQDN filtering)
AWS-Managed Groups (Global threat intel)

Regardless of the source, I required a single, centralized configuration to manage the evaluation order across all groups.

4️⃣ Strict Ordering and Explicit Ownership
For auditability and debugging, I enforced a clear Separation of Concerns:

rule_order = "STRICT_ORDER": Enforced for all stateful Suricata groups to ensure predictable packet flow.
Explicit Ownership: Every rule group is tagged as either Terraform-owned or Automation-owned, preventing overlapping authority.

5️⃣ Elimination of State Drift Tug-of-War
The most critical requirement is preventing Terraform from attempting to revert updates made by external automation. The module must be configured to explicitly ignore attributes that the Lambda is authorized to change, ensuring that terraform plan remains clean even after high-frequency rule updates.

While the architectural blueprint involves both IaC and external automation, the following implementation details focus exclusively on the Terraform orchestration logic.

The Parent Module: Orchestrating Rule Groups

The architectural blueprint is defined at the parent level. This is where I declared which rule families to include and precisely how I wanted them to be prioritized during packet inspection.

1️⃣ Defining the Evaluation Hierarchy:

The global ordering policy is injected as a simple list. This dictated the sequence in which the AWS Network Firewall engine processed traffic:

## Rule evaluation order for stateful rule groups
stateful_rule_group_order = [
  "self_managed_domainlist",
  "self_managed_suricata",
  "extl_managed_suricata",
]

The resulting evaluation logic is deterministic:

Priority 1: Any self_managed_domainlist groups are placed at the top of the stack.
Priority 2: All self_managed_suricata groups follow.
Priority 3: extl_managed_suricata groups are inserted next.
Baseline: Finally automatically appends AWS-managed groups to the end of the chain as a final safety layer.

By decoupling the evaluation order from the child module, the ability is achieved to adjust the security posture per environment (e.g., Dev vs. Prod) without modifying the underlying infrastructure code.

2️⃣ Static Rule Generator

Since static rules are predictable and fixed in nature, the module utilizes a Terraform template to generate the rule-set dynamically. This ensures that adding new AllowListed FQDNs is a matter of updating a configuration list rather than manually writing Suricata syntax.

The Rule Template
The following .tftpl file serves as the foundation. It combines global security alerts with a dynamic loop to generate pass rules for specific domains:

## Global rules
alert tcp any any -> any [443,465] (msg:"Detected non-TLS on TLS port"; flow:to_server; app-layer-protocol:!tls; threshold: type limit, track by_src, seconds 90, count 1; sid:10000001; rev:1;)
alert tcp any any -> any any (msg:"ALERT TCP Alert testing"; flow:to_server; sid:10000002; rev:1;)
alert http any any -> any any (msg:"ALERT HTTP Alert testing"; flow:to_server; sid:10000003; rev:1;)
alert tls any any -> any any (msg:"ALERT TLS Alert testing"; flow:to_server; sid:10000004; rev:1;)

## AllowListed FQDNs
%{~ for K1, V1 in rule_sets }
pass tls $EXTERNAL_NET any -> $HOME_NET any (tls.sni; content:"${V1.fqdn}"; ssl_state:client_hello; msg:"matching HTTPS allowListed FQDN"; flow:to_server, established; nocase; sid:${V1.indx}; rev:1;)
%{~ endfor }

The Input Data
It's a human-readable map of domains and their subdomains, presented in this form:

> local.nfw_suricata_pass_hosts
{
  "iba.mysite.com" = tolist([
    "api",
    "app",
  ])
  "iba.examples.site" = tolist([
    "sports",
    "news",
  ])
  ....
}

Data Transformation
To feed the template, a local block transforms this map into a structured object. This step is critical because it generates a unique, deterministic sid (Security ID) for every entry—a requirement for Suricata rules:

locals {
  ....
  nfw_suricata_pass_rules = merge([
    for K1, V1 in keys(local.nfw_suricata_pass_hosts) : {
      for K2, V2 in local.nfw_sctpass_hosts[V1] : "${K1}-${K2}-${V2}" => {
        indx = tonumber("11${K1}9989") + 11 + (K2 + 1),
        fqdn = "${V2}.${V1}",
      }
    }
  ]...)
}

The resulting state now looks like this, and ready for the template engine:

> local.nfw_suricata_pass_rules
{
  "0-0-api" = {
    "fqdn" = "api.iba.mysite.com"
    "indx" = 1110001
  }
  "0-1-app" = {
    "fqdn" = "app.iba.mysite.com"
    "indx" = 1110002
  }
  "1-0-sports" = {
    "fqdn" = "sports.iba.examples.site"
    "indx" = 1120001
  }
  ....
}

S3 as the Source of Truth
Persistence & the Source of Truth
The final ruleset is rendered into a local variable using the templatefile function. This step acts as the compilation phase, merging the dynamic host data with the static Suricata templates:

locals{
  nfw_rules_sets = templatefile(
    "${path.module}/templates/suricata_ingress_rules.tftpl", {
      rule_sets = local.nfw_suricata_pass_rules
    }
  )
}

Once rendered, the ruleset is persisted to an S3 bucket. This serves as the definitive source of truth for the firewall, providing an immutable version history and enabling offline auditing or forensic review:

// Save rule sets in S3 as source-of-truth
resource "aws_s3_object" "aso_sct_ingress" {
  for_each         = local.nfw_svc_enabled ? toset([var.service_name]) : []
  bucket           = module.nfw_bucket[var.service_name].name
  content_base64   = base64encode(local.nfw_rules_sets)
  content_type     = "text/plain"
  content_encoding = "utf-8"
  key              = "suricata_ingress.rules"
  kms_key_id       = local.remote_accs_data[var.eks_owner_acc].nfw_bucket_key_arn
  provider         = aws.eksapp
}

3️⃣ Defining Suricata Rule Groups:

A single list is used to define both the static and dynamic Suricata groups. To distinguish between them, I introduced a self_managed boolean flag, which is the primary pivot for how the child module handled the lifecycle of each group.

## Suricata Firewall Rule Groups
suricata_stateful_rule_group = [
  # 1) Static Suricata rule group (Terraform-owned)
  {
    name = format("${var.service_nfw.id}-suricata-%s-sf",
           substr(sha256(local.nfw_rules_sets), 0, 7))
    capacity       = 100
    description    = "Stateful Suricata rule-sets to allow static-hosts"
    rules_content  = base64decode(
      aws_s3_object.aso_sct_ingress[each.value].content_base64
    )
    rule_variables = local.nfw_rule_variables
    self_managed   = true
  },

  # 2) Dynamic Suricata rule group (externally-managed)
  {
    name           = "${var.service_nfw.id}-suricata-dynamic-sf"
    capacity       = 8000
    description    = "Stateful Suricata rule-sets for tenant-hosts"
    rules_file     = "${path.module}/templates/suricata_dynamic_placeholder"
    rule_variables = local.nfw_rule_variables
    self_managed   = false
  },
]

Key Implementation Details:

The Static Group (self_managed = true): This configuration utilizes a multi-stage pipeline where Terraform first renders the Suricata ruleset from a template and persists the output to S3. By pulling the rules_content directly from the S3 object, the firewall remains pinned to a versioned, verifiable source of truth. A sha256 hash is incorporated into the resource naming convention to ensure that any change in the S3 content triggers a clean, deterministic update of the firewall rule group.
The Dynamic Group (self_managed = false): For the tenant-specific rules, I pointed rules_file to a simple placeholder template. This allowed me to satisfy the AWS Network Firewall requirement of having a valid Suricata structure at creation time. I set the capacity significantly higher (8,000 units) to provide enough "headroom" for the Lambda to inject thousands of tenant hostnames later without hitting architectural limits.
The Logic Pivot: The self_managed flag was the brain of the operation. Inside the child module, I used this to separate groups that Terraform must protect from those that Terraform must ignore after the initial creation.

4️⃣ Implementing Domain List Rule Groups:

For the sake of completeness and architectural flexibility, I applied the exact same pattern to Domain List rule groups. While my primary focus was on Suricata for deep packet inspection and SNI filtering, I wanted the module to remain type-agnostic.

domain_stateful_rule_group = [
  {
    name           = "${local.template_name}-domains-sf"
    capacity       = 100
    description    = "Stateful rule with domain list option"
    actions        = "ALLOWLIST"
    protocols      = ["TLS_SNI"]
    domain_list    = ["placeholder.invalid"]
    rule_variables = local.nfw_rule_variables
    self_managed   = true
  }
]

Even though it relies heavily on Suricata signatures for our core logic, I included support for Domain Lists as a secondary mechanism. This allowed us to handle simple Allow-listing for standard HTTP/HTTPS traffic using AWS's native FQDN filtering when the full power of Suricata's regex and header inspection wasn't required.

5️⃣ Integrating AWS-Managed Rule Groups:

To round out our security posture, I wanted to leverage the threat intelligence provided by AWS. Instead of managing complex ARNs in the parent configuration, I designed the module to accept simple, human-readable names which I then programmatically converted into full ARNs.

aws_managed_rule_group_arns = [
  for name in var.aws_managed_rule_groups :
  "arn:aws:network-firewall:${var.region}:aws-managed:stateful-rulegroup/${name}"
]

The module was purposely structured to append these AWS-managed groups (such as AWSManagedRulesAnonymousIpList or ThreatSignaturesMalware) to the end of our evaluation chain.

By placing them after our custom Suricata and Domain List groups, I ensured that our specific business logic (our Allow rules) was evaluated first, while the AWS-managed groups acted as a global safety-net for any traffic that hadn't already been explicitly matched.

The Child Module: Orchestrating the Rule Groups

Once I had the parent module's requirements defined, the child module (tf-module-nfw) translates those lists into actual AWS resources. I decided to split the responsibilities into four distinct resource types to ensure different lifecycle rules and management logic can be applied to each.

Each rule family is mapped to a specific Terraform resource block:

self_managed_suricata → aws_networkfirewall_rule_group.sf_self_suricata
extl_managed_suricata → aws_networkfirewall_rule_group.sf_custom_suricata
self_managed_domainlist → aws_networkfirewall_rule_group.sf_self_domain_list
extl_managed_domainlist → aws_networkfirewall_rule_group.sf_custom_domain_list

By creating separate resource blocks for Self (Terraform-owned) vs Custom (Externally-owned), this design provides a granular control over how Terraform interacts with the AWS API. This was the foundation for solving the state-drift problem: I could tell Terraform to strictly enforce the rules for the Self groups while essentially looking the other way for the Custom ones.

1️⃣ Static Suricata — Terraform-owned:

For the Self-Managed resource, I designed a filter that only processes rule groups where the self_managed flag is set to true. This ensured that Terraform maintained absolute authority over these specific groups.

resource "aws_networkfirewall_rule_group" "sf_self_suricata" {
  for_each = {
    for rg in var.suricata_stateful_rule_group :
    rg.name => rg
    if rg.self_managed
  }

  type        = "STATEFUL"
  name        = each.value.name
  description = each.value.description
  capacity    = each.value.capacity

  rule_group {
    # Rule variables: HOME_NET, EXTERNAL_NET, etc. 
    rules_source {
      # Static Suricata rules come from S3 → Terraform → rules_content
      rules_string = each.value.rules_content
    }

    stateful_rule_options {
      rule_order = "STRICT_ORDER"
    }
  }

  lifecycle {
    create_before_destroy = true
  }

  tags = merge(var.tags)
}

In this block, Terraform owns everything: the rules, the metadata, and the resource lifecycle. This was my go-to choice for stable, platform-owned domains and internal network signatures.

The Power of for_each Filtering:
By using a conditional for_each loop, I kept the parent module interface clean while allowing the child module to selectively manage only the static groups.
Enforcing STRICT_ORDER:
I explicitly set the rule_order to STRICT_ORDER. This was critical because it allowed me to predict exactly how packets would traverse my custom rules versus the AWS-managed ones later in the chain.
Lifecycle Management:
create_before_destroy = true is exclusively added to prevent any security blind spots. If a static rule group needed a replacement (like a name change due to a new hash), the new group would be provisioned before the old one is torn down.

2️⃣ Dynamic Suricata — Externally-owned:

This is arguably the most important section of this entire process. For the dynamic rule groups, I implemented a resource block that specifically targets objects where self_managed is set to false. This is where the architectural hand-off occurs.

resource "aws_networkfirewall_rule_group" "sf_custom_suricata" {
  for_each = {
    for rg in var.suricata_stateful_rule_group :
    rg.name => rg
    if !rg.self_managed
  }

  type        = "STATEFUL"
  name        = each.value.name
  description = each.value.description
  capacity    = each.value.capacity

  rule_group {
    # Same rule_variables block as above (HOME_NET / EXTERNAL_NET etc.)

    rules_source {
      # Placeholder Suricata, just to create the group.
      rules_string = file(each.value.rules_file)
    }

    stateful_rule_options {
      rule_order = "STRICT_ORDER"
    }
  }

  lifecycle {
    create_before_destroy = true
    ignore_changes = [
      # Lambda will take ownership of the rules_string
      rule_group[0].rules_source[0].rules_string,
    ]
  }

  tags = merge(var.tags)
}

The Secret Sauce: lifecycle.ignore_changes
The real magic happens in the lifecycle block - configuring Terraform in this manner establishes a clear lifecycle for dynamic security rules.

Creation: Terraform provisions the rule group container using the high capacity that I defined (8,000 units) and populates it with a basic Suricata placeholder file to satisfy the AWS API.

Delegation: Once the resource exists, ownership of the rules_string attribute is effectively transferred to the external automation.

Coexistence: Because of the ignore_changes directive, Terraform will no longer attempt to correct or revert the rules during subsequent plan or apply cycles.

This approach is what allowed me to safely call UpdateRuleGroup from a Lambda function (which is covered in Part 2) without triggering a state-drift fight. Terraform manages the infrastructure shell, while the Lambda manages the security intelligence.

3️⃣ Domainlist Rule Groups:

The exact same blueprint has followed there for Domain List rule groups. Mirroring the logic used for Suricata ensures the module remains predictable and easy to maintain.

These are split into two distinct resources:

sf_self_domain_list: The Terraform-owned resource for static, code-defined FQDNs.
sf_custom_domain_list: The externally-owned resource for dynamic lists.

These are handled in an identical fashion to the Suricata groups—utilizing the self_managed flag to trigger either a strict management cycle or a handoff to external automation via ignore_changes.

The most important takeaway here isn't the resource type itself, but these groups are now prepared to participate in the unified ordering mechanism, which is the final piece of the puzzle.

Building the Final Stateful Rule Group Order

This approach transforms the design into an elegant, structured solution rather than an ad-hoc one. By leveraging Terraform locals, a scattered set of resources is transformed into a strictly ordered pipeline.

The child module utilizes a three-step logic:

Map the Categories: The ARNs of all created resources are grouped tinto a structured map.
Flatten by Priority: Those groups are then reordered and flattened based on the stateful_rule_group_order list provided by the parent.
Inject AWS Intelligence: The AWS-managed ARNs are appended as the final elements.

The Transformation Logic:

Here is how I handled that data transformation using HCL:

locals {
  # 1. Categorize rule-group ARNs into logical families
  sf_rule_group_arns = {
    self_managed_suricata = {
      for KV in var.suricata_stateful_rule_group :
      KV.name => aws_networkfirewall_rule_group.sf_self_suricata[KV.name].arn
      if KV.self_managed
    }

    extl_managed_suricata = {
      for KV in var.suricata_stateful_rule_group :
      KV.name => aws_networkfirewall_rule_group.sf_custom_suricata[KV.name].arn
      if !KV.self_managed
    }

    self_managed_domainlist = {
      for KV in var.domain_stateful_rule_group :
      KV.name => aws_networkfirewall_rule_group.sf_self_domain_list[KV.name].arn
      if KV.self_managed
    }

    extl_managed_domainlist = {
      for KV in var.domain_stateful_rule_group :
      KV.name => aws_networkfirewall_rule_group.sf_custom_domain_list[KV.name].arn
      if !KV.self_managed
    }
  }

# 2. Flatten categories using the parent’s
stateful_rule_group_order
  stateful_group_arns = concat(
    concat([
      for key in var.stateful_rule_group_order :
      values(local.sf_rule_group_arns[key])
    ]...),

    # then, tack on AWS-managed rule groups
    var.aws_managed_rule_group_arns,
  )
}

The Final Policy:

With a flat, ordered list of ARNs established, applying them to the firewall policy becomes a straightforward process. I then used a dynamic block to ensure the policy stayed updated whenever the list changed.

resource "aws_networkfirewall_firewall_policy" "this" {
  name = "${var.prefix}-policy-${var.firewall_name}"

  firewall_policy {
    stateless_default_actions          = ["aws:${var.stateless_default_actions}"]
    stateless_fragment_default_actions = ["aws:${var.stateless_fragment_default_actions}"]

    ## Stateless Rule Group Reference
    dynamic "stateless_rule_group_reference" {
      for_each = local.stateless_group_arns
      content {
        # Priority is sequentially as per index in stateless_rule_group list
        priority = index(
          local.stateless_group_arns,
          stateless_rule_group_reference.value
        ) + 11
        resource_arn = stateless_rule_group_reference.value
      }
    }
    # stateless config omitted
  }
}

The Resulting Impact:

This modular architecture yields three major wins:

Centralized Control: The parent module uses one simple variable to dictate the entire evaluation sequence.
True Reusability: The child module is now a generic engine that can be easily dropped into any service, account, or environment.
Future-Proofing: New categories (such as an external domain feed) can be added seamlessly without requiring a refactor of the parent module's core logic.

Lessons Learned (Part 1)

Building this architecture was a journey that reinforced several core principles of cloud networking and Infrastructure-as-Code. Here is what I took away from the experience:

1️⃣ Separation of Concerns is Mandatory
I realized early on that pushing every single rule through a Terraform pipeline is a recipe for frustration once you reach high-velocity, frequently changing lists. Splitting our firewall into Static (TF-owned) and Dynamic (Externally-owned) rule groups created a much healthier, more predictable management pattern.

2️⃣ The Danger of Joint Custody
One of my biggest takeaways was that Terraform and Lambda must never share ownership of a single attribute. If both tools think they are allowed to modify the rules_string, you will spend your life resolving state drift. By using separate resources (sf_self_* vs. sf_custom_*) and leveraging ignore_changes, I established a clean hand-off that kept our pipelines green.

3️⃣ Null Values are Not Empty
When defining input objects with optional fields like rules_content and rules_file, I found it's easy to accidentally treat a null value as "present." I had to ensure my module logic explicitly tested for null rather than just checking if a key existed, preventing several silent deployment failures.

4️⃣ Explicit Ordering is Worth the Overhead
The stateful_rule_group_order list might seem like extra configuration at first, but it completely removed the ambiguity from our security posture. In an environment that combines custom Suricata signatures, FQDN filters, and AWS-managed threat intel, knowing exactly which rule fires first is critical for both security and troubleshooting.

What’s Coming in Part-2

While we’ve built the infrastructure "shell" and enforced the ordering logic, the "Externally Managed" part of this architecture is where the automation truly comes to life.

In the next post, I’ll deep-dive into the Delivery Mechanism, featuring:

The Rule-Engine Lambda: How I built a function to read tenant_hosts.json from S3 and dynamically expand base domains into full Suricata SNI rules.
Concurrency & Safety: How to handle the UpdateRuleGroup API safely using the required UpdateToken to prevent race conditions.
Security & Identity: Managing cross-account access via S3 bucket policies and KMS key permissions.
Observability: Implementing Microsoft Teams notifications that include:
- Diff Analysis: Clearly showing which hosts were added or removed.
- Smart Truncation: Handling long lists gracefully so notifications remain readable.
- Status Cards: Visual Green (Success) and Red (Failure) indicators for immediate feedback.

That’s where the Self-Service aspect of our firewall becomes a reality.

While I tried my best to ensure the accuracy of the code snippets provided; feel free to reach out in the comments below or via LinkedIn if you spot any discrepancies or have questions regarding the implementation.

See you in Part 2!

BootStrapping Aurora RDS Databases using Lambda and Terraform (Part 3)

Santanu Das — Wed, 14 Jan 2026 12:20:43 +0000

TL;DR
We wrap our bootstrap logic in an AWS Step Functions workflow to transition from a manual script to an automated platform capability. By leveraging EventBridge and CloudTrail, we create an event-driven loop that detects infrastructure changes and reconciles the database state automatically—achieving a production-ready deployment with zero manual intervention.

The Architecture
Implementation
- Root Module
- Child Module
Validation
Conclusion

Orchestrating with AWS Step Functions

In Part 1, we identified the problem: Terraform is great for infrastructure, but poor for database state management. In Part 2, we built the solution: A "driver-less" Lambda function that uses the Aurora Data API to perform secure, idempotent bootstrap operations.

Now, we face the final hurdle - A Lambda function sitting in the console is just a tool: To make it a platform capability, we need to wrap it in a workflow that handles retries, provides visibility, and allows for controlled execution.

In this final part, we explore how to orchestrate that bootstrap logic using AWS Step Functions. It does not replace migrations, schema management, or application workflows — only control when the bootstrap Lambda needs to run.

The Architecture: The State Machine

I designed the state machine that treats the Lambda as a Worker. The State Machine itself acts as the Manager, responsible for:

Input Validation: Ensuring the target cluster exists.
Task Execution: Invoking the Bootstrap Lambda.
Error Handling: Managing retries and catching failures.

The Working-flow

It's an Event-Driven Architecture (EDA) that automatically triggers the bootstrap workflow whenever Terraform updates the Lambda function configuration or code, as below:

Terraform Apply
      ↓
AWS Lambda APIs
(CreateFunction / UpdateFunctionCode)
      ↓
AWS CloudTrail
(Records the Management Event)
      ↓
Amazon EventBridge
(Matches Rule: source="aws.lambda")
      ↓
EventBridge assumes the EventBridge IAM role
      ↓
AWS Step Functions
(Triggered via IAM Role)
      ↓
Bootstrap Lambda
(Executes Idempotent Logic)
   ↓
Bootstrap Lambda runs

For this to work effectively, EventBridge Rule pattern should look for specific API calls to avoid misfires. I filtered (using {prefix:..} matching) for:

CreateFunction20150331 (New deployments)
UpdateFunctionCode20150331v2 (Code changes)
UpdateFunctionConfiguration20150331v2 (Env var/Config changes)

This ensures the database bootstrap runs exactly when the code or configuration (like a new DB name in Env Vars) changes.

Implementation: Infrastructure as Code

Consistent with our IaC principles, the State Machine is provisioned via Terraform, while its workflow logic is defined using the Amazon States Language (ASL). The implementation follows a modular architecture, splitting responsibilities between the root (security & orchestration) and child (application logic) modules.

Terraform Resources - Root module

1️⃣ Creates one SFN execution role (states.amazonaws.com) with:

RDS cluster actions (start/stop/modify/describe/snapshot) scoped to module.db_cluster[KY].arn
lambda:InvokeFunction scoped to the installer Lambdas
CloudWatch Logs delivery permissions (CreateLogDelivery, PutResourcePolicy, etc.) on *

# ----------------------------------------------------------
# IAM role for Step-Function
# ----------------------------------------------------------
resource "aws_iam_role" "air_sfn" {
  count = length(local.sfn_db_engines) > 0 ? 1 : 0
  name  = "${local.template_name}-sfn-Role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Service = "states.amazonaws.com"
      }
      Action = "sts:AssumeRole"
    }]
  })

  tags = {
    Name     = "${local.template_name}-sfn-Role"
    Module   = var.tf_module_name,
    Resource = "aws_iam_role.air_sfn",
  }
}

data "aws_iam_policy_document" "aipd_sfn" {

  statement {
    sid    = "AllowLambdaInvoke"
    effect = "Allow"

    actions = [
      "lambda:GetFunction",
      "lambda:GetFunctionConfiguration",
      "lambda:InvokeFunction",
    ]

    resources = [
      for KY in local.sfn_db_engines : module.db_cluster[KY].lfn_resource_arn
    ]
  }

  statement {
    sid    = "AllowCloudWatchLogs"
    effect = "Allow"

    actions = [
      "logs:CreateLogDelivery",
      "logs:GetLogDelivery",
      "logs:UpdateLogDelivery",
      "logs:DeleteLogDelivery",
      "logs:DescribeLogGroups",
      "logs:DescribeResourcePolicies",
      "logs:GetLogDelivery",
      "logs:ListLogDeliveries",
      "logs:PutLogEvents",
      "logs:PutResourcePolicy",
      "logs:UpdateLogDelivery",
    ]
    resources = ["*"]
  }
}

resource "aws_iam_role_policy" "airp_sfn" {
  count  = length(local.sfn_db_engines) > 0 ? 1 : 0
  role   = aws_iam_role.air_sfn[0].id
  policy = data.aws_iam_policy_document.aipd_sfn.json
}

# ----------------------------------------------------------
# IAM role: EventBridge → Step Functions
# ----------------------------------------------------------
resource "aws_iam_role" "air_eb_sfn" {
  count = length(local.sfn_db_engines) > 0 ? 1 : 0
  name  = "${local.template_name}-eb-sfn-Role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Service = "events.amazonaws.com"
      }
      Action = "sts:AssumeRole"
    }]
  })

  tags = {
    Name     = "${local.template_name}-eb-sfn-Role"
    Module   = var.tf_module_name,
    Resource = "aws_iam_role.air_eb_sfn",
  }
}

// Policy: allow states:StartExecution
data "aws_iam_policy_document" "aipd_eb_sfn" {
  statement {
    effect = "Allow"
    actions = [
      "states:StartExecution"
    ]
    resources = [
      for KY in local.sfn_db_engines :
      module.sfn_machine[KY].arn
    ]
  }
}

resource "aws_iam_role_policy" "airp_eb_sfn" {
  count  = length(local.sfn_db_engines) > 0 ? 1 : 0
  role   = aws_iam_role.air_eb_sfn[0].id
  policy = data.aws_iam_policy_document.aipd_eb_sfn.json
}

2️⃣ Creates one EventBridge → SFN role (events.amazonaws.com) with:

states:StartExecution scoped to the child module SFN ARNs (module.sfn_machine[KY].arn)
Instantiates module sfn_machine per engine via local.sfn_db_engines

# ----------------------------------------------------------
# AWS Step-function
# ----------------------------------------------------------
module "sfn_machine" {
  for_each = toset(local.sfn_db_engines)
  source   = "./services//sfn"

  lambda_function_arn = module.db_cluster[each.key].lfn_resource_arn

  name_prefix = join("-", [
    replace(local.template_name, "/-[^-]+$/", ""),
    each.key
  ])

  eb_role_arn  = aws_iam_role.air_eb_sfn[0].arn
  sfn_role_arn = aws_iam_role.air_sfn[0].arn

  extra_tags = {
    Resource = "module.sfn_machine"
    Module   = var.tf_module_name
  }
}

Terraform Resources - Child module

1️⃣ Creates the state machine:

arn:aws:states:::lambda:invoke
FunctionName = var.lambda_function_arn
CW logs enabled to /aws/stepfunctions/${var.name_prefix}-bootstrap

# ----------------------------------------------------------
# Step Function Resource
# ----------------------------------------------------------
resource "aws_sfn_state_machine" "this" {
  name     = "${var.name_prefix}-bootstrap"
  role_arn = var.sfn_role_arn

  definition = jsonencode({
    Comment = "DB / User bootstrap orchestration"
    StartAt = "InvokeBootstrapLambda"
    States = {
      InvokeBootstrapLambda = {
        Type     = "Task"
        Resource = "arn:aws:states:::lambda:invoke"
        Parameters = {
          FunctionName = var.lambda_function_arn
          Payload      = {}
        }
        Retry = [{
          ErrorEquals = [
            "Lambda.ServiceException",
            "Lambda.TooManyRequestsException"
          ]
          IntervalSeconds = 5
          MaxAttempts     = 3
          BackoffRate     = 2.0
        }]
        End = true
      }
    }
  })

  logging_configuration {
    level                  = "ALL"
    include_execution_data = true
    log_destination        = "${aws_cloudwatch_log_group.sfn_logs.arn}:*"
  }

  tags = merge(
    var.extra_tags,
    {
      Name        = "${var.name_prefix}-bootstrap"
      SubResource = "aws_sfn_state_machine.db_bootstrap"
    }
  )
}

// CW logs
resource "aws_cloudwatch_log_group" "sfn_logs" {
  name              = "/aws/stepfunctions/${var.name_prefix}-bootstrap"
  retention_in_days = 14
}

2️⃣ Creates the EventBridge rule that works reliably:

source = ["aws.lambda"]
detail-type = ["AWS API Call via CloudTrail"]
eventName prefix match for both:
- UpdateFunctionCode*
- UpdateFunctionConfiguration*
Filters requestParameters.functionName with both forms:
- the full ARN: var.lambda_function_arn
- short name extracted via regex("[^:]+$", var.lambda_function_arn)
Creates the EventBridge target:
- Starts the SFN with an input_transformer payload containing account/region/event_id/time/function_name

# ----------------------------------------------------------
# EventBridge rule: detect Lambda code updates
# ----------------------------------------------------------
resource "aws_cloudwatch_event_rule" "lambda_code_update" {
  name        = "${var.name_prefix}-lambda-update"
  description = "Trigger SFN when bootstrap Lambda code is updated"

  event_pattern = jsonencode({
    source        = ["aws.lambda"]
    "detail-type" = ["AWS API Call via CloudTrail"]
    detail = {
      eventSource = ["lambda.amazonaws.com"]
      eventName = [
        { "prefix" : "UpdateFunctionCode" },
        { "prefix" : "UpdateFunctionConfiguration" }
      ]

      # without this, all the SFNs will keep triggering
      requestParameters = {
        functionName = [
          var.lambda_function_arn,
          regex("[^:]+$", var.lambda_function_arn),
        ]
      }
    }
  })
}

resource "aws_cloudwatch_event_target" "lambda_update_to_sfn" {
  rule      = aws_cloudwatch_event_rule.lambda_code_update.name
  target_id = "start-${var.name_prefix}-bootstrap"
  arn       = aws_sfn_state_machine.this.arn
  role_arn  = var.eb_role_arn

  retry_policy {
    maximum_retry_attempts       = 0
    maximum_event_age_in_seconds = 60
  }

  # Optional - To capture the "Silent" error 
  dead_letter_config {
    arn = aws_sqs_queue.eb_dlq.arn
  }

  input_transformer {
    input_paths = {
      update_fn = "$.detail.requestParameters.functionName"
      event_id  = "$.id",
      region    = "$.region"
      account   = "$.account"
      time      = "$.time"
    }

    input_template = <<-EOF
{
  "account": <account>,
  "action": "AUTO_BOOTSTRAP",
  "aws_region": <region>,
  "event_id": <event_id>,
  "event_time": <time>,
  "function_name": <update_fn>,
  "trigger": "lambda-code-update"
}
    EOF
  }
}

Setting up the automated trigger was a classic cat-and-mouse engineering challenge. The real complexity was correctly mapping the CloudTrail metadata into a format the State Machine understands. I spent significant time fine-tuning the input_transformer logic — navigating through the JSON paths to ensure that the specific Lambda updates correctly extracted the payload, needed to kick off the bootstrap. Those were the only parameters worked for me; anything more added to it, was stopping the trigger on the next run.

3️⃣ Adds an SQS DLQ and queue policy allowing EventBridge to SendMessage for that rule (optional)

# ----------------------------------------------------------
# SQS: Dead-letter queue redrive
# ----------------------------------------------------------
resource "aws_sqs_queue" "eb_dlq" {
  name = "${var.name_prefix}-eb-dlq"
}

resource "aws_sqs_queue_policy" "dlq_policy" {
  queue_url = aws_sqs_queue.eb_dlq.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "events.amazonaws.com" }
      Action    = "sqs:SendMessage"
      Resource  = aws_sqs_queue.eb_dlq.arn
      Condition = {
        ArnEquals = {
          "aws:SourceArn" : aws_cloudwatch_event_rule.lambda_code_update.arn
        }
      }
    }]
  })
}

Validation: The Orchestration in Action

Once the EventBridge rule triggers the workflow, the progress can be monitored in real-time. The Step Functions' Executions console provides the list of Failed and Successful executions, as shown below:

Beyond the high-level list, the console also allows us to drill down into the specific input and output of the bootstrap Lambda, where you can see exactly how the CloudTrail event was mapped into the parameters that drove the database creation.

Conclusion: An automated Database Platform

That brings us to the end of Part 3. We have successfully transformed our bootstrap Lambda into a robust, event-driven platform capability that triggers automatically whenever our infrastructure changes. By orchestrating this with AWS Step Functions, we’ve also gained the visibility, retries, and auditability needed for our environment. It's now a complete, driver-less solution that ensures our Aurora databases are initialized and ready the moment they are provisioned - and most importantly, without any manual intervention.

In This Series

Part 1: Architectural foundation and engineering basics.
Part 2: Deep dive into Lambda implementation and IAM-based access.
Part 3: Orchestrating workflows and Execution using AWS Step Functions.

BootStrapping Aurora RDS Databases using Lambda and Terraform (Part 2)

Santanu Das — Sun, 11 Jan 2026 14:25:10 +0000

TL;DR
We move from theory to implementation by building a zero-dependency Python Lambda function. By leveraging the Aurora Data API, we eliminate the need for heavy SQL drivers, VPC-peering, or static credentials.

The Solution
- Strategic Benefits
Core Design Concepts
Aurora Data API
The Code Structure
- Lambda Files
- Components
Implementation & Code
- Terraform Layer
- Lambda Finction-Code
Day-Zero Readiness
- Verification
- Validation

In Part-1, we explored the inherent challenges of managing post-deployment database state and user privileges within a declarative IaC workflow. In this article, we shift from problem to implementation, detailing a solution that provides a robust, scalable alternative for platform teams.

The Solution: Decoupling via Purpose-Built Lambda

The core of this architecture is the establishment of a clear execution boundary. By offloading database mutation logic to an AWS Lambda function, we decouple the state of the infrastructure from the state of the database.

Terraform’s Responsibility: Operates as the orchestrator, provisioning the Aurora cluster and Lambda function while injecting necessary metadata (endpoints, ARNs, and engine types).
Lambda’s Responsibility: Acts as the execution engine, performing engine-specific logic (MySQL/PostgreSQL), enforcing role-based access control (RBAC), and managing IAM-based authentication.

Strategic Benefits

True Idempotency: Unlike complex Terraform providers that can drift, the Lambda logic is procedural and testable, allowing for safe re-runs without impacting infrastructure integrity.
Hardened Security: The design eliminates the need for static administrative passwords, leveraging AWS IAM Database Authentication for all bootstrap operations.
Architectural Purity: Terraform remains purely declarative, focused on infrastructure lifecycle, while the Lambda handles the "Day 1" platform bootstrapping as a discrete capability.

This deep-dive focuses on the structural internals of the Lambda—exploring the design decisions that ensure maintainability and engine parity while keeping the footprint minimal.

Core Design Concepts & Requirements

To ensure a seamless handoff between infrastructure and application, the bootstrap Lambda was built around the following core requirements:

Passwordless Database Operations: Eliminates the need for static credentials or secret rotation by utilizing native AWS authentication.
IAM/Token-Based Access: Leverages short-lived IAM database tokens for all administrative actions, ensuring a secure and auditable connection lifecycle.
Zero Developer Friction: The entire process is transparent to application teams; databases and roles are provisioned automatically without manual intervention.
Multi-Tenant Support: A single execution can manage multiple database schemas and user roles, mapping complex requirements to a unified configuration.
Day-Zero Readiness: Ensures that the database environment is fully initialized and accessible the moment the Aurora cluster enters the available state.

The Secret Sauce: Aurora Data API

One of the biggest hurdles in building a lightweight Lambda for database bootstrapping is the dependency on database drivers. Python’s standard library does not include native drivers for MySQL or PostgreSQL.

Typically, this forces platform engineers into a dependency trap, implementing either:

Lambda Layers: Adding overhead for versioning and binary compatibility.

Container Images: Introducing complex build pipelines and increasing the security blast radius.

We solved this by enabling the Aurora Data API. This design choice is the cornerstone of the Lambda's simplicity:

Pure AWS SDK: The Lambda interacts with the database via standard boto3 calls. If the Lambda environment has the AWS SDK, it has everything it needs to execute SQL.
No Drivers Required: By sending SQL commands over HTTPS via the Data API, we eliminated the need for psycopg2, pymysql, or any other external binaries.
Simplified Networking: The Data API handles the connection pooling and overhead, allowing the Lambda to remain "stateless" and extremely fast to cold-start.

By treating the database as an API endpoint rather than a traditional socket connection, we achieved a zero-dependency deployment artifact. This makes the bootstrap process more auditable, easier to maintain, and significantly more resilient to environment drift.

Implementing this modern connectivity model required only a single configuration change within the chlid-module:

# Enabling native IAM authentication within the Aurora cluster
iam_database_authentication_enabled = true

The Modular Code Structure

The Lambda’s internal structure is designed for clarity and extensibility. Rather than a monolithic script, the codebase is partitioned into functional layers. This ensures that adding support for a new engine or modifying authentication logic doesn't disrupt the core execution flow.

File Structure

The project follows a flat, purposeful directory layout:

./lambda
 ├── __init__.py
 ├── db_config.py       # Environment-to-Runtime mapping
 ├── db_helper.py       # Core SQL logic and engine-specific functions
 ├── mysql_installer.py # MySQL execution workflow
 └── pgsql_installer.py # PostgreSQL execution workflow

Component Responsibilities

db_config.py (The Configuration Layer): This module acts as the bridge between Terraform and Python. It ingest cluster-specific environment variables — such as cluster identifiers, engine types, and database names—and validates them before execution begins.
db_helper.py (The Logic Library): This is the "brain" of the operation. It contains the granular, engine-specific functions required to verify and mutate state. By centralizing functions like mysql_user_exists and pgsql_create_db here, we ensure that the logic is reusable and easy to unit test.
*_installer.py (The Workflow Orchestrators): These files represent the high-level workflows for each engine. An installer imports the necessary primitives from the helper and executes them in a logical sequence.
e.g. Check if DB exists → Create if missing → Assign privileges.

Although the bootstrap flow is shared, MySQL and PostgreSQL differ fundamentally in ownership, role semantics, and grant models — which is why installers are intentionally split.

The helper functions are designed to accept a generic execute_fn. This allows us to swap the underlying execution engine — such as switching from a standard driver to the Aurora Data API — without rewriting the core business logic.

The Blueprint: Implementation & Code

With the architectural boundaries defined, now we will look at the implementation. This section provides the Copy-Paste ready code to stand up the bootstrap utility.

IaC: The Terraform Layer

1️⃣ Every AWS Lambda function needs permission to interact with other AWS infrastructure resources within your account. These permissions are set via an AWS IAM Role that required to interact with the Aurora Data API. The creation of the execution role and the specific IAM policies are part of the root-module; the rule ARN then supplied to the lambda-function.

# ----------------------------------------------------------
# IAM role Lambda-function
# ----------------------------------------------------------
resource "aws_iam_role" "air_lfn" {
  for_each = (
    anytrue(values(local.service_enabled))
    ? toset([var.service_name])
    : []
  )
  name = "${local.template_name}-lambda-Role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Service = "lambda.amazonaws.com"
      }
      Action = "sts:AssumeRole"
    }]
  })
}

# ----------------------------------------------------------
# Lambda role-policy document and attachment
# ----------------------------------------------------------
data "aws_iam_policy_document" "aipd_lfn" {
  for_each = (
    anytrue(values(local.service_enabled))
    ? toset([var.service_name])
    : []
  )

  statement {
    sid    = "RDSDataAPI"
    effect = "Allow"

    actions = [
      "rds-data:ExecuteStatement",
      "rds-data:BatchExecuteStatement"
    ]

    resources = ["*"]
  }

  statement {
    sid    = "DescribeCluster"
    effect = "Allow"

    actions = [
      "rds:DescribeDBClusters"
    ]

    resources = ["*"]
  }

  statement {
    sid    = "ReadDBSecret"
    effect = "Allow"

    actions = [
      "secretsmanager:GetSecretValue"
    ]

    resources = [
      for DB in keys(local.db_profiles) :
      module.db_cluster[DB].master_user_secret_arn
    ]
  }

  statement {
    sid    = "LambdaLogging"
    effect = "Allow"

    actions = [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]

    resources = ["arn:aws:logs:*:*:*"]
  }
}

resource "aws_iam_role_policy" "airp_lfn" {
  for_each = (
    anytrue(values(local.service_enabled))
    ? toset([var.service_name])
    : []
  )
  role   = aws_iam_role.air_lfn[each.value].id
  policy = data.aws_iam_policy_document.aipd_lfn[each.value].json
}

2️⃣ Next, we define AWS infrastructure resources, which I included as part of the child module. The following Terraform configuration automatically packages the Python source into a ZIP archive and provisions the Lambda function, along with its execution role and granular IAM policies.

# ---------------------------------------------------------
# Lambda function to manage BDs
# ---------------------------------------------------------
data "archive_file" "daf_lambda" {
  for_each = (
    var.enable_db_managment
    ? { (var.db_profile.engine_type) = true }
    : {}
  )
  type        = "zip"
  output_path = "${path.module}/${each.key}-lambda.zip"

  source {
    content  = file("${path.module}/lambda/__init__.py")
    filename = "__init__.py"
  }
  source {
    content  = file("${path.module}/lambda/db_config.py")
    filename = "db_config.py"
  }
  source {
    content  = file("${path.module}/lambda/db_helper.py")
    filename = "db_helper.py"
  }
  source {
    content  = file("${path.module}/lambda/${each.key}_installer.py")
    filename = "db_installer.py"
  }
}

resource "aws_lambda_function" "db_installer" {
  for_each = (
    var.enable_db_managment
    ? { (var.db_profile.engine_type) = true }
    : {}
  )
  function_name = "${var.name_prefix}-${each.key}-installer"
  role          = var.lambda_service_role  #<-- comes from root-module
  runtime       = "python3.12"
  handler       = "db_installer.handler"
  timeout       = 30
  memory_size   = 256

  filename         = data.archive_file.daf_lambda[each.key].output_path
  source_code_hash = data.archive_file.daf_lambda[each.key].output_base64sha256

  environment {    #<-- feeds db_config.py
    variables = {
      CLUSTER_ARN  = module.cluster.cluster_arn
      DATABASES    = jsonencode(var.db_profile.databases)
      DB_ENGINE    = var.db_profile.engine
      DB_USER_HOST = var.db_profile.dbs_user_host
      DB_USER_NAME = var.db_profile.database_user
      ENGINE_TYPE  = var.db_profile.engine_type
      SECRET_ARN   = local.secret_arn

      # default databases
      DEFAULT_DB = (
        var.db_profile.engine == "aurora-mysql" ? "mysql" : "postgres"
      )
    }
  }

  tags = merge(
    var.extra_tags,
    {
      Name        = "${var.name_prefix}-db-installer"
      SubResource = "aws_lambda_function.db_installer"
    }
  )
}

The Logic: Python Lambda Code

Here is how the Python components interact. The key is the db_helper.py, which leverages the Data API to execute SQL without traditional drivers.

1️⃣ db_config.py - The Configuration Layer

# db_config.py

import os
import json
from dataclasses import dataclass

def _require_env(name: str) -> str:
    try:
        return os.environ[name]
    except KeyError:
        raise RuntimeError(f"Missing required environment variable: {name}")

@dataclass(frozen=True)
class Config:
    cluster_arn: str
    databases: list[str]
    db_engine: str
    db_host: str
    db_user: str
    default_db: str
    engine_type: str
    secret_arn: str

CONFIG = Config(
    cluster_arn = _require_env("CLUSTER_ARN"),
    databases   = json.loads(_require_env("DATABASES")),
    db_engine   = _require_env("DB_ENGINE"),
    db_host     = _require_env("DB_USER_HOST"),
    db_user     = _require_env("DB_USER_NAME"),
    default_db  = os.environ.get("DEFAULT_DB", "postgres"),
    engine_type = _require_env("ENGINE_TYPE"),
    secret_arn  = _require_env("SECRET_ARN"),
)

2️⃣ db_helper.py - The Engine Logic

# db_helper.py

from db_config import CONFIG

db_user    = CONFIG.db_user
db_host    = CONFIG.db_host
default_db = CONFIG.default_db

# ==========================================================
# MySQL (IAM auth)
# ==========================================================

def mysql_user_exists(execute_fn) -> bool:
    resp = execute_fn(
        f"SELECT 1 FROM mysql.user WHERE user = '{db_user}'",
        default_db,
    )
    return len(resp.get("records", [])) > 0

def mysql_create_user(execute_fn):
    if mysql_user_exists(execute_fn):
        print(f"[SKIP] MySQL user exists: {db_user}@{db_host}")
        return

    print(f"[CREATE] MySQL user: {db_user}@{db_host}")

    execute_fn(
        f"""
        CREATE USER '{db_user}'@'{db_host}'
        IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS'
        """,
        default_db,
    )

def mysql_grant_privileges(execute_fn, db: str):
    print(f"[GRANT] MySQL privileges: {db_user}@{db_host} → {db}")

    execute_fn(
        f"GRANT ALL PRIVILEGES ON `{db}`.* TO '{db_user}'@'{db_host}'",
        default_db,
    )

def mysql_list_granted_dbs(execute_fn, user: str) -> set[str]:
    sql = """
        SELECT DISTINCT DB
        FROM mysql.db
        WHERE User = :user
    """

    resp = execute_fn(
        sql,
        database="mysql",
        params={"user": user},
    )

    records = resp.get("records", [])

    return {
        row[0]["stringValue"]
        for row in records
        if row and "stringValue" in row[0]
    }

def mysql_revoke_db(execute_fn, user: str, host: str, db: str):
    sql = f"REVOKE ALL PRIVILEGES ON `{db}`.* FROM '{user}'@'{host}'"
    execute_fn(sql, database="mysql")

# ==========================================================
# PostgreSQL (IAM auth)
# ==========================================================

def pgsql_db_exists(execute_fn, db: str, default_db: str) -> bool:
    resp = execute_fn(
        f"SELECT 1 FROM pg_database WHERE datname = '{db}'",
        default_db,
    )
    return len(resp.get("records", [])) > 0

def pgsql_create_app_role(execute_fn, app_role: str, default_db: str):
    resp = execute_fn(
        f"SELECT 1 FROM pg_roles WHERE rolname = '{app_role}'",
        default_db,
    )

    if len(resp.get("records", [])) == 0:
        print(f"[CREATE] PostgreSQL app role: {app_role}")
        execute_fn(
            f'CREATE ROLE "{app_role}" NOLOGIN',
            default_db,
        )
    else:
        print(f"[SKIP] PostgreSQL role exists: {app_role}")

    execute_fn(
        f'ALTER ROLE "{app_role}" LOGIN',
        default_db,
    )

    execute_fn(
        f'GRANT rds_iam TO "{app_role}"',
        default_db,
    )

def pgsql_create_db(execute_fn, db: str, default_db: str):
    execute_fn(
        f'CREATE DATABASE "{db}"',
        default_db,
    )

def pgsql_grant_privileges(
    execute_fn,
    db: str,
    app_role: str,
):
    print(f"[GRANT] PostgreSQL access: {app_role} → {db}")

    # DB-level access
    execute_fn(
        f'GRANT CONNECT, TEMP ON DATABASE "{db}" TO "{app_role}"',
        "postgres",
    )

    # Schema access
    execute_fn(
        f'GRANT USAGE ON SCHEMA public TO "{app_role}"',
        db,
    )

    # Explicit table grants (NO silent no-op)
    resp = execute_fn(
        """
        SELECT
            quote_ident(n.nspname) || '.' || quote_ident(c.relname)
        FROM pg_class c
        JOIN pg_namespace n ON n.oid = c.relnamespace
        WHERE n.nspname = 'public'
        AND c.relkind = 'r'
        """,
        db,
    )

    for r in resp.get("records", []):
        table = r[0]["stringValue"]
        execute_fn(
            f'GRANT SELECT, INSERT, UPDATE, DELETE ON TABLE {table} TO "{app_role}"',
            db,
        )

    # Sequences
    execute_fn(
        f'''
        GRANT USAGE, SELECT, UPDATE
        ON ALL SEQUENCES IN SCHEMA public
        TO "{app_role}"
        ''',
        db,
    )

    # Future objects (mandatory)
    execute_fn(
        f'''
        ALTER DEFAULT PRIVILEGES IN SCHEMA public
        GRANT SELECT, INSERT, UPDATE, DELETE
        ON TABLES TO "{app_role}"
        ''',
        db,
    )

    execute_fn(
        f'''
        ALTER DEFAULT PRIVILEGES IN SCHEMA public
        GRANT USAGE, SELECT, UPDATE
        ON SEQUENCES TO "{app_role}"
        ''',
        db,
    )

3️⃣ mysql_installer.py - mySQL Orchestrator

# mysql_installer.py

import boto3
from botocore.exceptions import ClientError

from db_config import CONFIG
from db_helper import (
    mysql_create_user,
    mysql_grant_privileges,
    mysql_list_granted_dbs,
    mysql_revoke_db,
)   #<-- loads only mySQL specific functions 

rds_data = boto3.client("rds-data")
rds_ctrl = boto3.client("rds")

def _assert_cluster_available():
    """
    Ensure the Aurora cluster exists and is in 'available' state.
    Step Function should normally guarantee this, but this keeps
    the Lambda safe when invoked manually.
    """
    cluster_id = CONFIG.cluster_arn.split(":")[-1]

    try:
        resp = rds_ctrl.describe_db_clusters(
            DBClusterIdentifier=cluster_id
        )
    except ClientError as e:
        raise RuntimeError(f"Failed to describe cluster: {e}")

    status = resp["DBClusters"][0]["Status"]
    if status != "available":
        raise RuntimeError(f"Cluster not ready (status={status})")

    print(f"Aurora cluster is available: {cluster_id}")

def _execute(sql: str, database: str, params: dict | None = None):
    kwargs = {
        "resourceArn": CONFIG.cluster_arn,
        "secretArn": CONFIG.secret_arn,
        "database": database,
        "sql": sql,
    }

    if params:
        kwargs["parameters"] = [
            {"name": k, "value": {"stringValue": str(v)}}
            for k, v in params.items()
        ]

    return rds_data.execute_statement(**kwargs)


# -------------------------
# MySQL helpers
# -------------------------
def _mysql_db_exists(db: str) -> bool:
    resp = _execute(
        f"SHOW DATABASES LIKE '{db}'",
        CONFIG.default_db,
    )
    return bool(resp.get("records"))

def _mysql_create_db(db: str):
    if _mysql_db_exists(db):
        print(f"[SKIP] MySQL database already exists: {db}")
        return

    _execute(
        f"CREATE DATABASE `{db}` CHARACTER SET utf8mb4",
        CONFIG.default_db,
    )
    print(f"[OK] Created MySQL database: {db}")

def _cleanup_stale_mysql_grants(execute_fn):
    desired = set(CONFIG.databases)
    granted = mysql_list_granted_dbs(execute_fn, CONFIG.db_user)

    stale = granted - desired

    for db in sorted(stale):
        print(
            f"[CLEANUP] Revoking stale grants: "
            f"{CONFIG.db_user}@{CONFIG.db_host} → {db}"
        )
        mysql_revoke_db(
            execute_fn,
            CONFIG.db_user,
            CONFIG.db_host,
            db,
        )


# -------------------------
# Lambda entrypoint
# -------------------------
def handler(event, context):
    print("Starting Aurora database bootstrap")
    print(f"Engine    : {CONFIG.db_engine}")
    print(f"Databases : {CONFIG.databases}")

    # Explicit no-op behavior
    if not CONFIG.databases:
        print("No databases defined — skipping bootstrap")
        return {
            "status": "skipped",
            "reason": "no_databases",
            "engine": CONFIG.db_engine,
        }

    _assert_cluster_available()

    if CONFIG.db_engine == "aurora-mysql":
        _cleanup_stale_mysql_grants(_execute)
        mysql_create_user(_execute)

        for db in CONFIG.databases:
            _mysql_create_db(db)
            mysql_grant_privileges(_execute, db)

    else:
        raise ValueError(f"Unsupported DB engine: {CONFIG.db_engine}")

    print("Database bootstrap completed successfully")

    return {
        "status": "ok",
        "engine": CONFIG.db_engine,
        "databases": CONFIG.databases,
    }

4️⃣ pgsql_installer.py - pgSQL Orchestrator

# pgsql_installer.py

import boto3
from botocore.exceptions import ClientError

from db_config import CONFIG
from db_helper import (
    pgsql_db_exists,
    pgsql_create_app_role,
    pgsql_create_db,
    pgsql_grant_privileges,
)   #<-- loads only pgSQL specific functions 

rds_data = boto3.client("rds-data")
rds_ctrl = boto3.client("rds")

# ==========================================================
# PgSQL helper functions
# ==========================================================
def _assert_cluster_available():
    try:
        resp = rds_ctrl.describe_db_clusters(
            DBClusterIdentifier=CONFIG.cluster_arn
        )
    except ClientError as e:
        raise RuntimeError(f"Failed to describe DB cluster: {e}")

    status = resp["DBClusters"][0]["Status"]
    if status != "available":
        raise RuntimeError(f"DB cluster not available (status={status})")

def _execute(sql: str, database: str):
    return rds_data.execute_statement(
        resourceArn=CONFIG.cluster_arn,
        secretArn=CONFIG.secret_arn,
        database=database,
        sql=sql,
    )
# ==========================================================
# Lambda handler entrypoint
# ==========================================================
def handler(event, context):
    print("Starting PostgreSQL database bootstrap")

    _assert_cluster_available()
    app_role   = CONFIG.db_user
    default_db = CONFIG.default_db

    # 1. App role (LOGIN + rds_iam)
    pgsql_create_app_role(_execute, app_role, default_db)

    # 2. Databases + grants
    for db in CONFIG.databases:
        if pgsql_db_exists(_execute, db, default_db):
            print(f"[SKIP] PostgreSQL DB already exists: {db}")
        else:
            print(f"[CREATE] PostgreSQL DB: {db}")
            pgsql_create_db(_execute, db, default_db)

        pgsql_grant_privileges(
            _execute,
            db,
            app_role,
        )

    print("PostgreSQL database bootstrap completed successfully")

    return {
        "status": "ok",
        "engine": "aurora-postgresql",
        "databases": CONFIG.databases,
    }

The Outcome: Day-Zero Readiness

With this implementation, we have successfully established a Day-Zero capability that operates as a native extension of the infrastructure pipeline. By offloading state mutation to a specialized Lambda function, we ensure that the database is not just available, but fully initialized and ready for application consumption the moment the Terraform run completes.

Deployment Verification

Following a successful terragrunt apply, the bootstrap functions are deployed as distinct platform primitives within your AWS environment. These functions remain idle until invoked by your orchestration layer, ensuring they consume no compute resources outside of active bootstrap windows.

Validating the Execution Flow

To verify the integration between the Lambda, the Aurora Data API, and the RDS cluster, you can perform a manual execution test. By invoking the function with a standard event payload, the Lambda will execute its "check-before-create" logic across the cluster.

A successful execution (as shown in the console logs below) confirms that the IAM permissions, networking via the Data API, and the engine-specific logic are all functioning as a cohesive unit.

In Part 3, we will explore how to integrate these functions into a controlled AWS Step Functions workflow. This move will provide us with platform-level visibility, auditable execution history, and the ability to run on-demand bootstraps—all while keeping our deployment pipeline entirely script-free.

In This Series

Part 1: Architectural foundation and engineering basics.
Part 2: Deep dive into Lambda implementation and IAM-based access.
Part 3: Orchestrating workflows and Execution using AWS Step Functions.

BootStrapping Aurora RDS Databases using Lambda and Terraform (Part 1)

Santanu Das — Sat, 10 Jan 2026 23:42:54 +0000

TL;DR
Terraform can create an Aurora cluster, but it cannot safely manage databases, users, or grants. This series shows how we decouple infrastructure provisioning from database bootstrapping using Lambda — and later, Step Functions — without breaking Terraform’s declarative model.

The Problems
Cluster Creation
- Child Module
- Root Module
- YAML DataSet
- Local Variables
- Input Variables

The Problems

Terraform’s Limitations

While Terraform is ideal for provisioning Aurora clusters, it is poorly suited for managing database-level objects because:

Limited Scope: It can only create one initial database; it cannot manage multiple databases, complex ownership models, or granular IAM-based access.
State Conflict: Managing SQL internals within Terraform leads to brittle plans, partial failures, and irreversible drift.
Risk: Running SQL during terraform apply creates tight coupling between infrastructure and runtime data, increasing operational risk.

The Manual Gap

Although it's able to provision the cluster and an initial database in minutes, it leaves behind an operational vacuum. Once the cluster is live, the engineering team is frequently forced into a manual waiting game:

Access Stagnation: The cluster exists, but there are no application-specific users.
Security Hurdles: A platform engineer must manually log in to configure IAM authentication and map roles to database users.
Permission Complexity: Defining granular Read/Write privileges, schema ownership, and grant structures becomes a ticket-driven manual task.
Developer Friction: Application developers sit idle until the tasks above are completed because the Initial Database provided by AWS lacks the schema-level management required for actual deployment.

This creates a bottleneck where infrastructure is automated, but the database remains unusable until a human intervenes to bridge the gap between a cloud resource and a functional data store.

This post outlines a platform engineering approach to bootstrapping Amazon Aurora databases, arguing that database internals should be decoupled from Terraform state.

Terraform’s responsibility ends when the cluster becomes reachable.
Everything beyond that point is runtime behavior, not infrastructure state.

The Cluster Basics

While this post isn't a deep dive into cluster creation itself, I will cover the essential infrastructure components to provide context for Part 2. My implementation utilizes the standard terraform-aws-rds-aurora module, wrapped within a custom child module to meet our specific requirements.

Child Module

module "cluster" {
  source  = "terraform-aws-modules/rds-aurora/aws"
  version = "10.0.2"  #<-- latest at the time of writing 

  name              = local.cluster_name
  engine            = var.db_profile.engine
  engine_mode       = "provisioned" #<-- NOT cluster-type
  engine_version    = var.db_profile.engine_version
  storage_encrypted = true

  enable_http_endpoint = anytrue([
    var.enable_db_managment, var.data_api_enabled
  ])

  master_username               = var.master_uname
  manage_master_user_password   = true
  master_user_secret_kms_key_id = var.kms_key_id

  manage_master_user_password_rotation              = true
  master_user_password_rotate_immediately           = true
  master_user_password_rotation_schedule_expression = "rate(21 days)"

  vpc_id               = var.vpc_id
  db_subnet_group_name = var.subnet_group_name

  security_group_ingress_rules = {
    for ci, dr in var.ingress_cidr_blocks :
    "cidr${ci}" => { cidr_ipv4 = dr }
  }

  cluster_monitoring_interval = 60

  cluster_parameter_group = {
    name        = local.cluster_name
    family      = local.db_engine_family
    description = "${local.cluster_name} cluster parameter group"
    parameters  = var.cluster_parameter_group_parameters
  }

  db_parameter_group = {
    name        = local.cluster_name
    family      = local.db_engine_family
    description = "${local.cluster_name} Database parameter group"
    parameters  = var.db_parameter_group_parameters
  }

  apply_immediately   = true
  deletion_protection = true
  skip_final_snapshot = true

  # ----------------------------------------------
  # Only required for Serverless V2 clusters
  # ----------------------------------------------
  serverlessv2_scaling_configuration = (
    var.cluster_type == "serverless"
    ) ? {
    min_capacity = 2
    max_capacity = 10
  } : null

  cluster_instance_class = (
    var.cluster_type == "serverless"
  ) ? "db.serverless" : var.db_profile.instance_class

  # ----------------------------------------------
  # Based on the number of AZs, it creates
  # 1x RW-instance & 1x RO-inatance per AZ
  # ----------------------------------------------
  instances = var.cluster_type == "serverless" ? {
    for ix, az in concat(var.aws_zones, [join("", var.aws_zones)]) :
    "${var.db_profile.engine_type}${ix + 1}" => {}
    } : merge(
    {
      "0rw" = {
        instance_class = lookup(
          var.db_profile, "rw_instance_class", var.db_profile.instance_class
        )
      }
    },
    {
      for ix, az in var.aws_zones :
      "${ix + 1}ro" => {
        instance_class = var.db_profile.instance_class
      }
    }
  )

  #create_cloudwatch_log_group = true
  enabled_cloudwatch_logs_exports = (
    var.db_profile.engine == "aurora-mysql" ? [
      "audit",
      "error",
      "general",
      "slowquery",
      ] : (
      var.db_profile.engine == "aurora-postgresql"
      ? ["postgresql"] : []
    )
  )

  iam_database_authentication_enabled = true #<-- MUST be true

  tags = merge(
    var.extra_tags,
    {
      Name        = local.cluster_name
      SubResource = "module.cluster"
    }
  )
}

Engine Mode Ambiguity: When deploying Aurora Serverless v2, it is important to note that the engine_mode must be set to provisioned. Unlike Serverless v1, v2 operates under the provisioned framework to allow for better scaling and feature parity.
Log Group Conflicts: Avoid setting create_cloudwatch_log_group = true within the module. In many Aurora configurations, AWS automatically creates the log group as soon as logging is enabled; if Terraform attempts to create it simultaneously, the deployment will fail due to a naming conflict.

Root Module

In the root-module, it just referrenced like this:

module "db_cluster" {
  for_each = anytrue(values(local.service_enabled)) ? tomap(local.db_profiles) : {}
  source   = "./services//aurora"

  aws_zones           = var.aws_zones
  cluster_type        = var.cluster_type
  db_profile          = each.value  #<-- details below
  ingress_cidr_blocks = distinct(local.static_route_cidrs)
  name_prefix         = replace(local.template_name, "/-[^-]+$/", "")
  kms_key_id          = var.kms_master_key_arn
  subnet_group_name   = aws_db_subnet_group.default[var.service_name].name
  vpc_id              = local.my_vpc_info.id

  cluster_parameter_group_parameters = lookup(
    local.cluster_parameters, each.key, []
  )

  db_parameter_group_parameters = lookup(
    local.db_parameters, each.key, []
  )

  dns_host_prefix     = var.dns_name_prefix
  dns_hosted_zone_id  = var.dns_hosted_zone_id
  enable_db_managment = each.value.enable_db_managment
  lambda_service_role = aws_iam_role.air_lfn[var.service_name].arn

  extra_tags = {
    Resource                = "module.db_cluster"
    Module                  = var.tf_module_name
    "${var.eks_access_tag}" = true
  }
  /*
  master_uname        = var.dbs_master_user
  */
}

While the name_prefix can be customized to any string, I utilize a standardized naming structure to maintain consistency across the platform. By using local.template_name, I construct a prefix following the pattern <env><acc>-<vpc_name>-<service_name>, which ensures resources are easily identifiable and organized.

Data set

I leverage Terragrunt to orchestrate our deployments. Below are the specific input variables and configurations I defined as the source of truth for the Aurora cluster:

cluster_type: serverless  #<-- or provisioned
db_clusters:
  - mysql
  - pgsql
# 
db_profile:
  default:
    allocated_storage: 20
    auto_stop: true
    autoscaling_enabled: true
    autoscaling_max_size: 3
    autoscaling_min_size: 1
    database_user: zenapp-iam-user
    enable_db_managment: true #<-- required for part2
    instance_class: db.t4g.small
    max_allocated_storage: 0
    multi_az: true
  mysql:
    enabled: true
    database_port: 3306
    engine: aurora-mysql
    engine_version: '8.0'
    databases:
      - my-database1
      - my-database2
  pgsql:
    enabled: true
    database_port: 5432
    engine: aurora-postgresql
    engine_version: 17
    databases:
      - pg-database1
      - pg-database2
#
cluster_parameter_group:
  mysql:
    family: aurora-mysql8.0
    description: 'MySQL Aurora Cluster Parameters'
    immediate:
      innodb_lock_wait_timeout: 60
      log_bin_trust_function_creators: 1
      long_query_time: 1
      max_allowed_packet: 67108864
      require_secure_transport: 'ON'
      slow_query_log: 1
      time_zone: UTC
      wait_timeout: 28800
    pending-reboot:
      aurora_parallel_query: 0
      binlog_checksum: NONE
      binlog_format: ROW
      log_output: FILE
      max_connections: LEAST({DBInstanceClassMemory/12582912},5000)
      tls_version: TLSv1.2
  pgsql:
    family: aurora-postgresql17
    description: 'PostgreSQL Aurora Cluster Parameters'
    immediate:
      deadlock_timeout: 1000
      idle_in_transaction_session_timeout: 600000
      log_lock_waits: 1
      log_min_duration_statement: 2000
      log_statement: ddl
    pending-reboot:
      log_line_prefix: '%t:%r:%u@%d:[%p]:'
      max_connections: LEAST({DBInstanceClassMemory/9531392},5000)
      rds.force_ssl: 1
      shared_preload_libraries: pg_stat_statements
      timezone: UTC
      track_activity_query_size: 2048
#
db_parameter_group:
  mysql:
    family: aurora-mysql8.0
    description: 'MySQL DB parameters'
    immediate:
      connect_timeout: 60
      general_log: 0
      log_bin_trust_function_creators: 1
      slow_query_log: 1
    pending-reboot:
      performance_schema: 1
  pgsql:
    family: postgresql16
    description: 'PostgreSQL DB parameters'
    immediate:
      default_statistics_target: 100
      maintenance_work_mem: 131072
      random_page_cost: 1.1
      temp_buffers: 16384
      work_mem: 8192

Aurora configuration updates are governed by two apply-methods: immediate and pending-reboot. These settings apply to both Cluster and DB parameter groups, but the engine — particularly PostgreSQL — is notoriously strict about how these classifications are handled. After extensive trial and error to avoid unexpected downtime or deployment friction, I’ve settled on the categorized list above as a stable, battle-tested baseline for platform environments.

Local Variables

I utilized locals {....} block to normalize the dataset, ensuring direct compatibility with the terraform-aws-rds-aurora module. The key transformations are outlined below:

locals {
  ....
  service_enabled = {
    for DB in var.db_clusters :
    DB => var.service_enabled && tobool(
      local.db_profiles[DB].enabled
    ) if contains(keys(local.db_profiles), DB)
  }

  # ---------------------------------------------
  # Decesion  making and conditional  resource
  # creation happens from the values from here
  # ---------------------------------------------
  db_profiles = {
    for DB in var.db_clusters : DB => merge(
      var.db_profile.default,
      {
        for K1, V1 in var.db_profile[DB] :
        K1 => V1
        if V1 != null
      },
      {
        dbs_user_host = var.eks_base_cidr != null ? join(".", [
          replace(var.eks_base_cidr, "/\\.[^./]+\\/\\d+$/", ""),
          "%",
        ]) : "%"
        engine_type = DB
      }
    )
    if contains(keys(var.db_profile), DB) && var.db_profile[DB].enabled
  }
  #
  db_parameters = {
    for DB in var.db_clusters : DB => flatten([
      for K1, V1 in {
        immediate = merge(
          try(var.db_parameter_group["default"].immediate, {}),
          try(var.db_parameter_group[DB].immediate, {})
        )
        pending-reboot = merge(
          try(var.db_parameter_group["default"]["pending-reboot"], {}),
          try(var.db_parameter_group[DB]["pending-reboot"], {})
        )
        } : [
        for K2, V2 in V1 : {
          name         = K2
          value        = try(tonumber(V2), V2)
          apply_method = K1
        }
      ]
    ])
    if contains(keys(local.db_profiles), DB)
  }
  #
  cluster_parameters = {
    for DB in var.db_clusters : DB => flatten([
      for K1, V1 in {
        immediate = merge(
          try(var.cluster_parameter_group["default"].immediate, {}),
          try(var.cluster_parameter_group[DB].immediate, {})
        )
        pending-reboot = merge(
          try(var.cluster_parameter_group["default"]["pending-reboot"], {}),
          try(var.cluster_parameter_group[DB]["pending-reboot"], {})
        )
        } : [
        for K2, V2 in V1 : {
          name         = K2,
          value        = try(tonumber(V2), V2),
          apply_method = K1
        }
      ]
    ])
    if contains(keys(local.db_profiles), DB)
  }
}

Input Variable Defination

Just for the completeness, these are the Terraform variables, to ingest the configuration from the Terragrunt YAML dataset:

variable "cluster_type" {
  type        = string
  default     = "serverless"
  description = <<-EOT
    The deployment mode for the Aurora Cluster
    Valid value is 'serverless' or 'provisioned'.
  EOT

  validation {
    condition = contains(
      ["serverless", "provisioned"], var.cluster_type
    )
    error_message = <<-EOT
      The cluster_type must be either 'serverless' or 'provisioned'.
    EOT
  }
}

variable "cluster_parameter_group" {
  type = map(object({
    family         = optional(string, null)
    description    = optional(string, null)
    immediate      = optional(map(string), {})
    pending-reboot = optional(map(string), {})
  }))
  default     = {}
  description = <<-EOT
    Configuration for RDS Cluster Parameter Groups (applies to the entire Aurora cluster).
    Use this to manage engine-specific settings that affect all instances in the cluster.

    Key attributes:
    - family: The engine family (e.g., 'aurora-postgresql15').
    - description: A custom description for the parameter group.
    - immediate: A map of parameters to apply immediately without a reboot (static and dynamic).
    - pending-reboot: A map of parameters that require a manual reboot to take effect.
  EOT
}

variable "db_parameter_group" {
  type = map(object({
    family         = optional(string, null)
    description    = optional(string, null)
    immediate      = optional(map(string), {})
    pending-reboot = optional(map(string), {})
  }))
  default     = {}
  description = <<-EOT
    Configuration for RDS DB Parameter Groups (applies to individual DB instances).
    Use this for settings that can vary between the primary and replica instances.

    Key attributes:
    - family: The engine family (e.g., 'aurora-postgresql15').
    - description: A custom description for the parameter group.
    - immediate: A map of parameters to apply immediately via the AWS API.
    - pending-reboot: A map of parameters that will only apply after the instance is rebooted.
  EOT
}

variable "db_profile" {
  type = map(object({
    allocated_storage     = optional(number, 20)
    auto_stop             = optional(bool, false)
    autoscaling_enabled   = optional(bool, true)
    autoscaling_max_size  = optional(number, 3)
    autoscaling_min_size  = optional(number, 1)
    database_port         = optional(number, 5432)
    database_user         = optional(string)
    enabled               = optional(bool, false)
    engine                = optional(string, null)
    engine_version        = optional(string, null)
    databases             = optional(list(string), [])
    enable_db_managment   = optional(bool, true)
    instance_class        = optional(string, "db.t4g.small")
    max_allocated_storage = optional(number, 0)
    multi_az              = optional(bool, true)
    rw_instance_class     = optional(string, "db.t4g.small")
  }))

  description = <<-EOT
    A map of database profile objects used to define one or more database clusters/instances.

    Key attributes:
    - allocated_storage: Initial disk size in GB (Default: 20).
    - auto_stop: Whether to automatically stop the instance during idle hours to save costs.
    - autoscaling_enabled: Enables horizontal or vertical scaling for the DB instances.
    - database_port: The network port for database connections (Default: 5432 for PostgreSQL).
    - database_user: The name of the application-user for the database.
    - engine/engine_version: Specific RDS engine type (e.g., 'aurora-postgresql') and its version.
    - databases: A list of specific database schemas to create within the cluster.
    - instance_class/rw_instance_class: The hardware specifications of DB instances (Default: db.t4g.small).
    - multi_az: If true, deploys a standby instance in a different Availability Zone for High Availability.
  EOT
}

This concludes the foundational setup for the Aurora cluster and our specific Terraform approach. With the infrastructure in place, we have a stable platform to build upon. In Part 2, we will dive into the Lambda implementation and explore how to consume the outputs from this module to drive the database bootstrapping process.

This approach is primarily aimed at platform and infrastructure teams managing shared Aurora clusters across multiple services and environments.

In This Series

Part 1: Architectural foundation and engineering basics.
Part 2: Deep dive into Lambda implementation and IAM-based access.
Part 3: Orchestrating workflows and Execution using AWS Step Functions.

Trigger AWS CodePipeline from a Cross-Account S3 Upload (Using EventBridge + IAM Roles)

Santanu Das — Fri, 14 Nov 2025 19:20:15 +0000

Overview
Source Account
Target Account
The Result
Conclusion
Upcoming

Being in a multi-account AWS environment, you quickly realize that while business domains are neatly isolated into separate accounts, real-world workloads often need to interact across those boundaries.

Recently, one of our app teams needed to trigger a platform pipeline whenever their EKS app uploaded a file to an S3 bucket. A simple ask - right? But in our world, that means messaging platform-team and waiting for them to push the magic “Run Pipeline” button, as there was no seamless way to trigger that workflow across accounts from S3. Hence, we are here today.

In this post, I’ll show how we eliminated that friction and built a clean, fully automated cross-account, S3 → CodePipeline trigger using AWS-native services.

Architectural Overview

This is the sample accounts scenario:

Account-A (source) - will call it ACA, has the S3 bucket
Account-B (Target) - will call it ACB, runs the PipeLine

Here, We will build a cross-account trigger, where:

ACA detects the S3 update event
ACA fires an EventBridge rule
EventBridge assumes a role in ACB
The role has permission to start the CodePipeline execution
ACB runs the pipeline

Here is the simple flow-diagram:

                   ┌──────────────────────────────────────────────────────────┐
                   │                    ACCOUNT A (Source)                    │
                   │                                                          │
                   │   ┌────────────────────────────────────────────────────┐ │
   CodeCommit /    │   │              EventBridge Rule                      │ │
   ECR / S3 →─────▶│   │ (Repo state change / image push / object put)      │ │
   etc.            │   └────────────────────────────────────────────────────┘ │
                   │                          │                               │
                   │                          ▼                               │
                   │        ┌──────────────────────────────────────┐          │
                   │        │ EventBridge Target                   │          │
                   │        │  - Uses IAM role in Account A        │          │
                   │        │    with sts:AssumeRole permission    │          │
                   │        └──────────────────────────────────────┘          │
                   │                          │  (sts:AssumeRole)             │
                   └──────────────────────────┼───────────────────────────────┘
                                              │
                                              ▼
                   ┌──────────────────────────────────────────────────────────┐
                   │                    ACCOUNT B (Pipeline)                  │
                   │                                                          │
                   │   ┌────────────────────────────────────────────────────┐ │
                   │   │  IAM Role: cross-account-pipeline-start            │ │
                   │   │  - Trusts Account A                                │ │
                   │   │  - Allows: codepipeline:StartPipelineExecution     │ │
                   │   └────────────────────────────────────────────────────┘ │
                   │                          │                               │
                   │                          ▼                               │
                   │   ┌────────────────────────────────────────────────────┐ │
                   │   │              AWS CodePipeline                      │ │
                   │   │            (Build/Test/Deploy Pipeline)            │ │
                   │   └────────────────────────────────────────────────────┘ │
                   │                                                          │
                   └──────────────────────────────────────────────────────────┘

There are resources to be created on the both sides. This can be done going to the individual account and deploy locally. Or, if there is any cross-account role configured, it can be done from one account as well.

In my case, as the S3 bucket (that used by EKS app to upload/update the file) was pre-created from ACB (as part of account vending process), I did everything from there. In that way, it was managed end-to-end from platform-managment side.

In Source Account - ACA

This where we will create the:

S3 bucket (with the Object/File inside),
Assume Role/policy for EventBridge to do PutEvents
EventBridge Rule and Target to forward the event to ACB bus

1️⃣ NOTE:
provider = aws.eksapp is optional.
As I’m running all ACA-side resources from the ACB account, so aws.eksapp is simply a provider alias pointing Terraform to the ACA account (where the EKS app actually runs).
It's not needed, if deploying from within ACA directly.

2️⃣ NOTE:
for_each = local.cdp_svc_enabled ? toset([var.service_name]) : [] is just an On/Off switch for this entire set of resources.
When local.cdp_svc_enabled is false, none of the cross-account trigger infrastructure gets created.
Handy for conditional deployments or environment-specific setups.

Now, tet’s break it down step-by-step.

1️⃣ Create the S3 bucket

This step is optional for this process; assuming you already have a bucket - mentioning here just for the completeness. Feel free to use your usual way of creating bucket.

2️⃣ Enable notification from S3 bucket

# ----------------------------------------------------------
# Ensure bucket sends events to EventBridge
# ----------------------------------------------------------
resource "aws_s3_bucket_notification" "eb_xacc" {
  for_each    = local.cdp_svc_enabled ? toset([var.service_name]) : []
  bucket      = module.app_bucket[each.value].name
  eventbridge = true
  provider    = aws.eksapp
}

3️⃣ Assume Role for EventBridge

# ----------------------------------------------------------
# Role for EB to assume to PutEvents to ACB bus
# ----------------------------------------------------------
resource "aws_iam_role" "eb_forward" {
  for_each = local.cdp_svc_enabled ? toset([var.service_name]) : []
  name     = "${local.template_name}-xacc-s3eb-Role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Effect    = "Allow",
      Principal = { "Service" : "events.amazonaws.com" },
      Action    = ["sts:AssumeRole"]
    }]
  })
  provider = aws.eksapp
}

resource "aws_iam_role_policy" "eb_forward" {
  for_each = local.cdp_svc_enabled ? toset([var.service_name]) : []
  role     = aws_iam_role.eb_forward[each.value].id

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Effect   = "Allow",
      Action   = "events:PutEvents",
      Resource = module.pipeline[each.value].event_bus.arn
    }]
  })
  provider = aws.eksapp
}

4️⃣ Capture the S3-bucket event, when file updates

# ----------------------------------------------------------
# Events Capture and forward to ACB bus
# ----------------------------------------------------------
resource "aws_cloudwatch_event_rule" "eb_xacc" {
  for_each    = local.cdp_svc_enabled ? toset([var.service_name]) : []
  name        = "${local.template_name}-xacc"
  description = "Forward S3 Object Created events to ACB event bus"
  provider    = aws.eksapp

  event_pattern = jsonencode({
    source      = ["aws.s3"],
    detail-type = ["Object Created"],
    detail = {
      bucket = {
        name = [module.app_bucket[each.value].name]
      },
      object = {
        key = [
          { prefix = data.aws_s3_object.aso_file[each.value].key }
        ]
      }
    }
  })
}

# ----------------------------------------------------------
# Target = ACB account event bus
# ----------------------------------------------------------
resource "aws_cloudwatch_event_target" "eb_xacc" {
  for_each = local.cdp_svc_enabled ? toset([var.service_name]) : []
  rule     = aws_cloudwatch_event_rule.eb_xacc[each.value].name
  arn      = module.pipeline[each.value].event_bus.arn
  role_arn = aws_iam_role.eb_forward[each.value].arn
  provider = aws.eksapp
}

In Target Account – ACB

This is where all the action actually happens — the CodePipeline lives here, the event bus lives here, and this is the account that ACA ultimately needs to poke to start the pipeline.

Here, we’ll configure:

Accept the forwarded EventBridge event
Create the IAM Role that EventBridge will assume
Give it permission to start the pipeline
Set up the CodePipeline start trigger
And wire everything to the event bus that ACA sends into

Let’s wlak it through step-by-step.

1️⃣ Create an EventBridge Event Bus

If you already have a centralized event bus — great!! You probably then have something like this:

event_bus = {
  arn  = aws_cloudwatch_event_bus.cdp_bus.arn
  name = aws_cloudwatch_event_bus.cdp_bus.name
}

If not, here’s the minimal setup to get one. I used a dedicated one.

# ----------------------------------------------------------
# Custom bus to receive events from INC account
# ----------------------------------------------------------
resource "aws_cloudwatch_event_bus" "s3_trigger" {
  name = "${var.name_prefix}-cdp-s3-trigger"
}

This is the bus ACA will forward events into.

2️⃣ Create the IAM Role that EventBridge will Assume

Unlike some other cross-account setup, where the other AWS account ID assumes a role, this uses the EventBridge service principal to assume this role.

# ----------------------------------------------------------
# EventBridge Assume-Role
# ----------------------------------------------------------
// Role-policy
data "aws_iam_policy_document" "s3_trigger" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["events.amazonaws.com"]
    }
  }
}

// Role
resource "aws_iam_role" "s3_trigger" {
  name               = "${var.name_prefix}-s3eb-Role"
  assume_role_policy = data.aws_iam_policy_document.s3_trigger.json
}

3️⃣ Give that role permission to start the CodePipeline

This is the second half of the trust chain — once EventBridge assumes the role above, it must be allowed to start the CodePipeline(s):

# ----------------------------------------------------------
# EventBridge start-pipeline policy
# ----------------------------------------------------------
resource "aws_iam_role_policy" "s3_trigger" {
  name = "${var.name_prefix}-s3eb-policy"
  role = aws_iam_role.s3_trigger.id
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Action = ["codepipeline:StartPipelineExecution"],
        Resource = [
          for br in local.repo_branches :
          module.auto_build[br].cdp_resource_arn
        ]
      }
    ]
  })
}

A few things to call out (for clarity):

It supports triggering multiple pipelines via local.repo_branches
The pipeline resource ARNs come from module.auto_build directly, which actually builds the pipeline, without using any wildcards
Allowed only very minimum IAM permission (StartPipelineExecution)

4️⃣ Allow the Trigger Account (ACA) to PutEvents

# ----------------------------------------------------------
# Allow ACA account to PutEvents on this bus
# ----------------------------------------------------------
resource "aws_cloudwatch_event_permission" "s3_trigger" {
  event_bus_name = aws_cloudwatch_event_bus.s3_trigger.name
  principal      = var.trigger_acc_id
  action         = "events:PutEvents"
  statement_id   = "AllowIncToPutEvents"
}

Here, var.trigger_acc_id is the ACA account ID that is allowed to send events into this bus.

5️⃣ EventBridge Rule + Target

# ----------------------------------------------------------
# EB rule: trigger CodePipeline on S3 Object Created
# ----------------------------------------------------------
resource "aws_cloudwatch_event_rule" "s3_trigger" {
  name           = "${var.name_prefix}-s3-trigger"
  event_bus_name = aws_cloudwatch_event_bus.s3_trigger.name
  description    = "Trigger CodePipeline when object is created in ${var.trigger_bucket}"

  event_pattern = jsonencode({
    source      = ["aws.s3"],
    detail-type = ["Object Created"],
    detail = {
      bucket = { name = [var.trigger_bucket] },
      object = {
        key = [{ prefix = var.trigger_prefix }]
      }
    }
  })
}

# ----------------------------------------------------------
# EB target: link S3-trigger to all pipelines
# ----------------------------------------------------------
resource "aws_cloudwatch_event_target" "s3_trigger" {
  for_each       = toset(local.repo_branches)
  rule           = aws_cloudwatch_event_rule.s3_trigger.name
  event_bus_name = aws_cloudwatch_event_bus.s3_trigger.name
  target_id      = "trigger-${each.key}"
  arn            = module.auto_build[each.key].cdp_resource_arn
  role_arn       = aws_iam_role.s3_trigger.arn
}

output "auto_build" {
  value = module.auto_build
}
output "event_bus" {
  value = {
    name     = aws_cloudwatch_event_bus.s3_trigger.name
    arn      = aws_cloudwatch_event_bus.s3_trigger.arn
    role_arn = aws_iam_role.s3_trigger.arn
  }
}

The rule matches the forwarded S3 events on the custom bus, and the target uses the s3_trigger role to start the appropriate pipeline for each branch.

The Result

And that’s it — the full cross-account flow stitched together end-to-end.
Here’s what the final architecture looks like, visually:

Conclusion

Cross-account automation is one of the trickiest parts of AWS CI/CD — especially in a multi-account setup where teams are isolated by design. But with this pattern in place, the entire S3 → EventBridge → CodePipeline workflow now flows cleanly across accounts:

ACA captures the S3 object update
ACA forwards the event to ACB
ACB’s custom event bus receives it
ACB’s EventBridge rule assumes a dedicated “start-pipeline” role
That role starts exactly the pipeline(s) it should — nothing more

And just like that, the whole process runs automatically, without Slack/Teams pings, without waiting for that to be happend, and most importantly — without the pipeline humans in the loop at all 😛

Event-driven. Cross-account. Zero friction.

Next: A Real-Life Example

In the next episode, we’ll revisit the real reason this whole setup was needed in the first place — and walk through how we used this exact cross-account trigger to solve a very real, very everyday operational problem.

Automating Executive-Ready AWS Cost Optimization FinOps Reports

Santanu Das — Fri, 07 Nov 2025 15:18:23 +0000

Overview
AWS Cost Reporter
Prerequisites
Dependencies
Getting Started
Running the Reporter
Inside the Report
Future Improvements
Closing Thoughts
Next Step

In Episode 1, we talked about how I came up with the idea of developing an automated pipeline that runs cost audits across AWS accounts and produces detailed JSON summaries. Those outputs are well-suited for automation and further analysis but are not ideal for executive or financial reviews, apart from those Top-5 Services Summary at the end.

🧭 Overview

This episode demonstrates how raw audit data can be converted into a structured, presentation-ready report that everyone can understand, using a Python-based automation tool that converts raw AWS Cost Audit (ACA) outputs into structured, executive-ready DOCX reports.
It demonstrates how ACR (AWS Cost Reporter) processes audit data, embeds cost-distribution charts, and generates comprehensive reports with a single command.
The post also outlines a macOS-safe installation workflow, integration with the main audit script, and a sample output report designed for leadership and FinOps reviews.
By the end, readers will understand how ACR extends the ACA toolkit from automated data collection to automated cost communication.

💡 Enter the AWS Cost Reporter

AWS Cost Reporter is a lightweight, finops-ready tool that consolidates cost drivers, EC2 rightsizing insights, storage and network findings, and visual summaries in a single, shareable document.
This is the final stage in our cost-optimization workflow:

AWS Cost Audit → gathers and analyzes usage.
AWS Cost Reporter → summarizes, visualizes, and delivers it.

The resulting report provides an at-a-glance view of cost distribution, optimization opportunities, and key recommendations, ready for presentation to stakeholders or leadership teams.

Automation delivers value only when insights can be clearly communicated.

🧱 Prerequisites

Before generating ACR (AWS Cost Reporter) report, ensure that the environment running the audit meets the following requirements:

AWS Cost Audit (ACA) Toolkit

ACA (AWS Cost Audit) v4.6.0 or above
A completed audit run producing output files such as:
- cost-by-service.json
- ec2-instances.json
- rds-instances.json
- ebs-volumes.json
- s3-*-size.json
- route53-cost.json
- elastic-ips.json
- and related data

System Requirements

Python 3.9 or higher (Python 3.12 recommended)
Approximately 200 MB of free disk space for dependencies and report generation
Internet access for the initial dependency installation

⚙️ Installing Dependencies

The AWS Cost Reporter (ACR) relies on a small set of Python libraries to process data, generate documents, and create visual summaries. These dependencies can be installed system-wide on macOS or Linux using either Homebrew Python, pip[3] or a virtual environment.

The following external libraries are used by The Reporter:

matplotlib – for chart creation
numpy – for numerical operations
pandas – for data parsing and aggregation
python-docx – for (.docx) report generation

ℹ️ As PEP-668 introduced externally managed environments to Python packaging and as it's now pretty much applied everywhere to stop the conflicts between OS package managers and Python-specific package management tools like pip, you may encounter an × This environment is externally managed message when using pip.

🐧 On Linux

Most (if not all) of the popular distributions comes with Python3 install as default, so just check the version to be sure.

1️⃣ Check Python3 version

On my fairly updated Ubuntu v24.04, I see this:

python3 --version
# Python 3.12.3

2️⃣ Install Core Libraries

If python virtual environment is not an option, then best is to use distribution's package-manager to install the library globally. On Ubuntu, this will do the job:

sudo apt install python3-{numpy,pandas,matplotlib}

🍎 On macOS

Because macOS also applies PEP 668, restrictions to protect system Python installations, Homebrew users may encounter the same externally managed environment message when using pip. The following approach installs the required libraries safely if using virtual environment not an option.

1️⃣ Install Python3 with Homebrew

(Purely optional, if you don't have it yet)

brew install python3

Confirm that the Homebrew version of Python is active:

which python3
# /opt/homebrew/bin/python3

2️⃣ Install Core Libraries

There are two different ways to install those dependencies in macOS.

Option 1: Allow pip to universally install user packages

mkdir -p ~/Library/Application\ Support/pip
echo "[global]\nbreak-system-packages = true\nuser = true" > ~/Library/Application\ Support/pip/pip.conf

Then use pip3 to install in the user-space:

pip3 install {matplotlib,numpy,pandas,python-docx} --user

Option 2: Temporary one-off (no config change)

pip3 install {matplotlib,numpy,pandas,python-docx} --user --break-system-packages

⚠️ You’ll need to re-add --break-system-packages every time you install or upgrade something.

🧪 Verify Installations

If all steps complete successfully, you can test the installation like this:

python3 -c "import docx as dx; print(dx.__version__)"
python3 -c "import numpy as ny; print(ny.__version__)"
python3 -c "import pandas as pd; print(pd.__version__)"
python3 -c "import matplotlib as mp; print(mp.__version__)"

Respective version numbers should be returned.

📦 Getting Started with ACR

The AWS Cost Reporter (ACR) is distributed as part of the AWS Cost Audit (ACA) toolkit starting from version v4.6.0.
The toolkit can be downloaded directly from GitHub as a compressed archive or cloned via Git.

1️⃣ Option A — Clone the Repository (recommended)

git clone https://github.com/dsantanu/aws-cost-audit.git
cd aws-cost-audit

This approach keeps the toolkit up to date and allows switching between tagged releases, e.g.:

git checkout v4.6.0

2️⃣ Option B — Download a Release Archive

If Git is not available, the toolkit can also be downloaded and extracted using curl:

curl -L https://github.com/dsantanu/aws-cost-audit/archive/refs/tags/v4.6.0.tar.gz -o aws-cost-audit-v4.6.0.tar.gz
tar -xzf aws-cost-audit-v4.6.0.tar.gz
cd aws-cost-audit-4.6.0

Once extracted, both aws-cost-audit.sh and aws_cost_reporter.py are available in the project directory.
The audit can then be executed directly from this location.

▶️ Running the Reporter

Once the dependencies are installed and an AWS Cost Audit (ACA) run has completed, the AWS Cost Reporter (ACR) can be executed directly to generate the final DOCX report.
The Reporter consumes the JSON outputs produced by the audit and compiles them into a well-structured document that summarizes service costs, resource utilization, and optimization opportunities.

1️⃣ Basic Usage (Standalone-run)

python3 aws_cost_reporter.py \
  --input ./json-output-dir \
  --output ./json-output-dir/audit_audit_report.docx \
  --charts \
  --org "Awesome Technologies" \
  --author "Awesome Cloud Architecture"

The above command reads the audit data from the specified directory (--input) and generates an audit_audit_report.docx file in the same location.
When the optional --charts flag is provided, the report also includes embedded PNG charts showing the Top 5 AWS Services by Cost* and Projected Savings by Optimization Category.

2️⃣ Command Parameters

Parameter	Description
`--input`	Path to the directory containing the AWS Cost Audit output files.
`--output`	Path and filename for the generated DOCX report.
`--charts`	Optional flag to embed cost-distribution and savings charts in the report.
`--org`	Organization name displayed in the report header.
`--author`	Preparer or team name shown in the report footer.

3️⃣ Integration with the AWS Cost Audit Toolkit

Beginning with ACA v4.6.0, the Reporter can also be invoked directly from the main audit script using the new --report flag.
This enables end-to-end automation, where the cost audit and Report generation occur within a single workflow.

bash aws-cost-audit.sh -p my_profile --report [--org <Organization_Name>] [--author <My_Name>]

When executed with the --report option, the audit script automatically runs the Reporter using the most recent audit output folder and generates AWS-Audit-Report-<aws_acc_id>-YYYYMMDD.docx file in the same location. --charts option is applied by default.
If the Reporter script is not present, a warning message is displayed without interrupting the audit process.

📊 Inside the Generated Report

The AWS Cost Reporter (ACR) produces a structured DOCX document designed for both technical and non-technical audiences.
Each section of the report highlights a specific dimension of cost optimization, using tables, charts, and clear textual summaries.

1️⃣ Report Structure Overview

Section	Description
Key Insights at a Glance	Summarizes the most significant findings such as top cost drivers, idle EC2 instances, unattached volumes, and S3 usage.
Executive Summary	Provides a narrative overview of overall cloud spend and optimization potential across services.
Compute (EC2) Analysis	Details EC2 utilization, rightsizing recommendations, and cost implications of downsizing.
Savings Plan Efficiency	Highlights coverage and utilization metrics if Savings Plan data is included in the audit.
Storage (EBS + S3)	Lists unattached EBS volumes, gp2 to gp3 migration candidates, and top S3 buckets by storage size.
Networking & Load Balancing	Summarizes NAT gateway usage, load balancer inventory, and optimization recommendations.
DNS & IP Optimization	Provides Route 53 and Elastic IP insights with duplicate records, TTL checks, and cost breakdowns.
Governance & Observability	Recommends tag enforcement, logging policies, and governance best practices.
Prioritized Remediation Plan	Presents a tabular action plan mapping potential savings against estimated implementation effort.
Appendix — Methodology & Data Validity	Documents how the data was collected and clarifies the boundaries of analysis.

2️⃣ Embedded Charts

When executed with the --charts flag, the report includes two embedded PNG visualizations:

Chart	Purpose
Top 5 Services by Cost (USD)	Displays a pie chart summarizing the five largest cost contributors in the account.
Projected Savings by Optimization Category	Shows a bar chart estimating savings potential from rightsizing, storage optimization, and governance improvements.

Both charts are placed contextually within the report:

The Top 5 Services chart appears immediately after the Key Insights at a Glance section.
The Projected Savings chart follows the Prioritized Remediation Plan, visually connecting recommended actions to potential impact.

3️⃣ Example Output Summary

An example excerpt from a generated report may include:

Key Insights at a Glance

Top Cost Driver: EC2 (~$5,298/month)

EC2 Fleet: 23 instances, 16 idle candidates

Storage: 31 gp2 → gp3 migrations recommended

RDS: 3 instances detected, none Multi-AZ

Networking: 5 NAT Gateways, 14 Elastic IPs (3 unattached)

This summary provides a quick snapshot of the organization’s cloud posture, highlighting both current spend and potential optimization opportunities.

4️⃣ Sample Report Preview

🚀 Future Improvements

The current release of the AWS Cost Reporter (v1.1.0) establishes a stable foundation for generating clear, data-driven cost optimization reports.
Future iterations will focus on enhancing customization, usability, and integration within the broader AWS Cost Audit (ACA) suite.

Planned areas of development include:

Enhanced Visual Styling
Improved font consistency, layout spacing, and visual alignment with Zenler’s brand palette for professional-grade presentations.
Branded Cover Page and Metadata
Optional front page featuring organization logo, report metadata, and generation parameters for executive distribution.
Configurable Report Sections
Ability to enable or disable specific sections (e.g., EC2, Storage, Networking) from the command line for faster targeted reporting.
Trend Visualization and Savings Tracking
Graphical representation of cost trends and realized savings across multiple audit periods.
Multi-Format Output
Support for generating reports in both DOCX and PDF formats directly from the CLI.
CSV/XLSX Export Integration
Parallel tabular data export to simplify follow-up analysis in Excel or BI dashboards.

These enhancements aim to make ACR not just a reporting utility but a complete FinOps communication layer—bridging raw analytics and executive insights.

🏁 Closing Thoughts

The AWS Cost Reporter (ACR) completes the final stage of the report generation pipeline introduced in Episode 1.
What began as a command-line audit tool now produces polished, presentation-ready reports that clearly communicate both cloud spend and optimization potential.

By turning complex JSON data into a single, structured document—with contextual charts, actionable insights, and clear recommendations—the Reporter bridges the gap between technical analysis and business visibility.
The result is a faster feedback loop for FinOps teams and decision-makers alike.

Organizations using the AWS Cost Audit (ACA) toolkit can now automate not only cost discovery but also cost storytelling—presenting findings that are ready for immediate executive review.

🪜 Next Step - Toward Full Automation

In the next installment, we will see how this toolkit can evolve toward a fully automated workflow, where audit execution, report generation, and distribution occur without manual intervention.
Future versions of the AWS Cost Audit (ACA) and AWS Cost Reporter (ACR) will include support for scheduled execution along with automated report delivery. These enhancements will enable organizations to keep stakeholders informed of their latest AWS cost insights—hands-free and on schedule.

Next up: Episode 3 — Distribute and Collaborate: Automating Cost Report Delivery

Automate Your AWS Cost Audits and Cut Cloud Waste

Santanu Das — Thu, 06 Nov 2025 12:50:14 +0000

The Problem
AWS Cost Audit
What It Does
Key Advantages
The Collections
Prerequisites
Getting Started
CLI Options
Usage Examples
Summary Report
Top 5 Services
FinOps Insight
Conclusion
Next Episode

The other day, a friend of mine was complaining that every month, their AWS bill is landing like a small novel: hundreds of line items, multiple accounts, and no clear picture of why costs changed.
Manually digging through the Cost Explorer, various EC2 dashboards, Route 53 pages etc. was slow and very error-prone.

So, I started thinking, what it takes to build a modular AWS Cost Audit script to collect and summarize cost and usage data automatically. That experiment grew into a reusable FinOps toolkit for cloud teams, which is lightweight, transparent, and CI/CD-friendly.

🧭 The Cost Problem in AWS

AWS gives incredible flexibility — but that also makes it easy to overspend. Idle EC2 instances, forgotten EBS volumes, over-provisioned RDS databases, and cross-region data transfers quietly add up. Even with native tools like Cost Explorer and AWS Budgets, many teams struggle to:

Continuously monitor all resources across accounts
Correlate costs with resource usage
Identify optimization opportunities fast enough

Cloud waste isn’t just a finance problem — it’s an engineering visibility shortfall.

💡 Enter the AWS Cost Audit

AWS Cost Audit is a lightweight, automation-ready toolkit that helps you take control of your AWS spend — without complex dashboards or heavy FinOps platforms.

It bridges the gap between AWS billing data and actionable cost insights, complements FinOps principles by enabling shared visibility and data-driven decisions.

It helps teams to:

Shift from reactive cost reviews to proactive optimization
Democratize cloud spending data — not just finance’s job
Build a culture of accountability across DevOps, engineering, and product

This isn’t about policing cost — it’s about empowering smarter usage.

⚙️ What It Does

Audits AWS environment for unused or idle resources
Summarizes spend by service, region, and tag
Highlights cost anomalies and trends
Generates clear, CLI-based reports for visibility or CI integration
Helps teams adopt FinOps practices without adding overhead

🚀 Key Advantages

Benefits	Helps with...
Immediate cost visibility	Understanding what’s costing you the most
Zero external dependencies	Works via CLI, with minimal setup
Multi-account awareness	Analyze organization-wide cost patterns
Automation-friendly	Integrate into CI/CD or cron jobs for continuous insights
FinOps-ready insights	Perfect for monthly cost reviews or reporting

📊 What this Script Collects

AWS Service	Data Collected	Why It Matters
EC2	Instances + 7-day CPU avg	Find underutilized compute
RDS	DB list + Multi-AZ flag	Detect over-redundant DBs
EBS	Volumes + size	Spot unattached/idle disks
S3	Bucket inventory + metrics	Track object-store sprawl
Route 53	Zones, records, cost	Simplify redundant DNS
Elastic IP	Allocation + attachment state	Catch idle IPs billed hourly
Tags	Key/value compliance	Attribute spend correctly
Compute Optimizer	Enrollment state	Enable rightsizing insights

All collectors talk directly to AWS CLI and write JSON into a timestamped output directory.

🧱 Prerequisites

An active AWS account with Cost Explorer enabled.
AWS CLI installed and configured with credentials that have the following permissions:
- ce:GetCostAndUsage*
- ce:GetSavingsPlansUtilization*
- ec2:Describe*
- rds:Describe*
- s3:List*
- cloudwatch:GetMetricStatistics
- tag:GetResources
Bash 3.2+, macOS or Linux
jq, wc, tar, date (GNU or BSD)
Optional (colors): tput

⚡ Getting Started

AWS Cost Audit is simple to try out — no dependencies, just a single script; can be done in one of these two ways.

1️⃣ Easy installation using `cURL`:

curl -sSL https://raw.githubusercontent.com/dsantanu/aws-cost-audit/main/aws-cost-audit.sh -o aws-cost-audit.sh
bash aws-cost-audit.sh -h

2️⃣ Or, if you prefer `git clone`:

git clone https://github.com/dsantanu/aws-cost-audit.git
cd aws-cost-audit
bash aws-cost-audit.sh -h

Git Repo: AWS Cost Audit

🧩 CLI Options

Usage: aws-cost-audit.sh [options]

General options:
  -p, --profile <name>   AWS CLI profile (default: default)
  -o, --out <file>       Output tar.gz filename
                         default: aws-cost-audit-YYYYMMDD.tgz
  -d, --dest <dir>       Output directory (default: ./)
  -r, --report           Generate executive FinOps report
  -s, --summary          Generate only text/JSON summary report
                         (Ignotes all other collector options)
  -u, --author           Name of the report generator
  -z, --org              Name of the Organization
  -h, --help             Show this help message

Selective collectors:
  --all         Run all collectors (default)
  --dns         Route 53 (zones/records/cost)
  --ec2         EC2 inventory + CPU metrics
  --eip         Elastic IPs (addresses + cost)
  --eks         EKs + NodeGroups
  --rds         RDS inventory
  --cost        Cost Explorer summaries
  --tags        Resource tags
  --network     EKS + networking resources
  --storage     EBS + S3
  --optimizer   Compute Optimizer enrollment check

Example:
  aws-cost-audit.sh -p prod -d outputs --ec2 --dns

🧪 Usage Examples

# Run full audit
bash aws-cost-audit.sh [--all]

# Use diffrent AWS profile
bash aws-cost-audit.sh -p <my-other-profile>

# Summary-only mode on existing data
bash aws-cost-audit.sh -s

# Generate results to diffrent location
bash aws-cost-audit.sh -d /tmp/aca-outputs-2025-11-06

# Focus on DNS + EIPs only
bash aws-cost-audit.sh --dns --eip

🧾 Summary Report

Here’s what AWS Cost Audit produces at the end of each run — a quick summary highlighting your top cost drivers:

📈 Top 5 Services Breakdown

Behind the scenes, this summary is generated from AWS Cost Explorer data using the ce:GetCostAndUsage API. It aggregates the last 30 days of spend, grouped by AWS service and account (if using AWS Organizations). The logic is straightforward:

Pull spend data by service.
Sort by descending cost.
Display the top 5 contributors to your total AWS bill.

This helps teams instantly see where their money is going — without logging into the console or building reports manually.

💰 Turning Data Into FinOps Insight

Insight	Example
Rightsizing	Identify EC2/RDS instances with low CPU or IOPS usage.
Idle cleanup	Find unattached EBS volumes or EIPs incurring hourly cost.
DNS optimization	Consolidate individual A-records via ALB aliases.
Tag hygiene	Detect untagged resources blocking cost allocation.

This bridges the gap between raw AWS data and actionable FinOps decisions — without external tools, which provides:

Visibility – instant service-level cost breakdowns.
Repeatability – same data set every run; perfect for monthly reviews.
Actionability – directly links metrics to optimization opportunities.
Autonomy – no third-party SaaS dependency or hidden API costs.

Integrate it into CI/CD or cron, and we have a continuous cloud-cost observability loop.

🏁 Conclusion

Cloud cost optimization doesn’t have to be complicated — it just needs visibility.

Automating cost visibility with AWS Cost Audit, teams can see where money is going, act quickly, and embed FinOps thinking into everyday operations.
With a few hundred lines of well-structured Bash and the AWS CLI, we now produce clear, auditable reports every run — no spreadsheets, no guesswork.

If your team wrestles with growing AWS costs, start small, automate what you can, and evolve like we did. Transparency is the first step to optimization.

🪜 Next Step - FinOps Report

In the next episode, we’ll explore how to take the generated audit reports and transform them into a beautifully formatted Microsoft Word report — ready for FinOps review meetings and executive summaries.

Next up: Episode 2 — Automating Executive-Ready AWS Cost Optimization FinOps Reports

Building a Scalable AWS Multi-Account Environment with Control Tower, Terraform AFT, and SCP Guardrails

Santanu Das — Tue, 28 Oct 2025 22:07:15 +0000

Overview
Organizational Structure
Regional Strategy
Conceptual Flow
Multi-Region Setup
CT & AFT Integration
Services Strategy
Governance & Security
SCP Guardrails
Control Layers
Deployment Workflow
Future Enhancements

Managing multiple AWS accounts securely and consistently is one of the hardest challenges in modern cloud architecture. In this guide, we’ll walk through how to build a scalable, governed AWS environment using Control Tower, Account Factory for Terraform (AFT), and Service Control Policies (SCPs), which is loosely based on Zenler's multi-account organization structure.

Overview

This document outlines the AWS account structure, governance, and control strategy used across our organization. We'll talk about how these three layers — Control Tower (govern), AFT (automate), and SCPs (enforce) — work together to provide an enterprise-grade foundation for compliance, security, and automated multi-account landing zone — aligning with AWS Well-Architected and CIS Benchmarks. This is the same model many large AWS customers use to keep their landing zones predictable and audit-ready while letting individual teams move fast with Terraform.

Below is an illustrative AWS Organization layout

Organizational Structure

OU	Description	Primary Accounts
Security OU	Core compliance and security monitoring	`Log Archive`, `Audit`
Internal OU	Shared platform services and IAM root	`Shared Services`, `IAM/Root`
NPR Networking OU	Non-Production networking environment	`Internal Workload`, `External Workload`
PRD Networking OU	Production networking environment	`Internal Workload`, `External Workload`
Deprecated OU	Legacy accounts (no new workloads)	Various (read-only)

Regional Strategy

Component	Region	Rationale
IAM Identity Center (SSO)	`us-east-1`	Global endpoint for AWS SSO and Organizations
Control Tower Management Account	`us-east-1` backend	Required by AWS
All Member Accounts	`eu-west-2` (London)	Primary data residency & workload region
Backup / DR	`eu-west-1` (Ireland)	Optional failover region

Conceptual Flow Diagram

                 ┌──────────────────────────────────────────────┐
                 │               Management Account             │
                 │ (AWS Organizations + Control Tower + AFT)    │
                 └──────────────────────────────────────────────┘
                                   │
                 ┌──────────────────────────────────────────────┐
                 │ Control Tower Landing Zone                   │
                 │ • Security OU (LogArchive + Audit)           │
                 │ • Creates baseline guardrails (AWS-managed)  │
                 │ • Delegates to AFT for account provisioning  │
                 └──────────────────────────────────────────────┘
                                   │ Management Acct (us-east-1)
              ┌────────────────────┴────────────────────┐
              │                                         │
    ┌──────────────────────────────┐          ┌──────────────────────────────┐
    │ Account Factory for TF (AFT) │          │ AWS Organizations (Org Root) │
    │ • GitOps: account requests   │          │ • OU hierarchy (SEC, INT,    │
    │ • Customizations pipelines   │          │   NPR, PRD, DEPRECATED)      │
    │ • Baselines, tagging, roles  │          │ • SCPs attached per OU       │
    └──────────────────────────────┘          └──────────────────────────────┘
              │                                         │
              │                                         │
              │              ┌────────────────────────────────────────┐
              │              │ SCP layer (Preventive Guardrails)      │
              │              │ • Enforced at Org root / OU level      │
              │              │ • Deny/allow APIs before IAM evaluated │
              │              │ • Prevents config drift or unsafe ops  │
              │              └────────────────────────────────────────┘
              │
     ┌────────┴─────────┐
     │ Enrolled Account │
     │ (e.g. Internal,  │
     │ NPR, PRD, etc.)  │
     └──────────────────┘

Why Setup Multi-Region

Like a lot of many others, we also started with everything in one region — but as soon as you deploy Control Tower, IAM Identity Center, and multi-account governance, regional separation becomes not just practical but recommended.

Here’s why I kept our management & identity layer in us-east-1 and our workloads in eu-west-2, following the best practice:

1️⃣ AWS Global Service Dependencies

Some services, like AWS Organizations, Control Tower, and IAM Identity Center (SSO) — are global by design but physically anchored in us-east-1. Running these components in their native region (if possible and allowed) ensures:

Full feature compatibility (some CT/AFT APIs only run in N. Virginia)
Stable upgrades when AWS releases new landing zone versions
Reduced orchestration latency for service-managed controls and SCPs

Control Tower’s orchestration backend always executes from us-east-1, even if your governed regions are elsewhere. Keeping the management plane in there prevents drift or control sync issues.

2️⃣ Regional Data Residency and Compliance

Our workloads stay in eu-west-2 (London), which:

Meets UK data sovereignty requirements
Keeps PII (Personally Identifiable Information), audit logs, and business data physically in-region
Aligns with GDPR and UK government guidelines for cloud workloads

This gives us the best of both worlds:

Control and governance (global layer in us-east-1)
Data and workload residency (EU region)

3️⃣ Reduced Blast Radius

By splitting control and data planes:

Security incidents in workload accounts cannot impact IAM or Organizations configuration.
SCPs and AFT pipelines stay isolated in the management plane, away from operational workloads.
Compliance tooling (Config, GuardDuty, Security Hub) in the Audit account can observe EU workloads without touching identity infrastructure.

4️⃣ Disaster Recovery and Fault Isolation

If the EU region faces an outage, our:

SSO / IAM Identity Center
Control Tower orchestration
AFT pipelines

…still remain operational in us-east-1, enabling us to recover or redeploy without waiting for the EU region to recover.

That’s a significant operational advantage for regulated workloads and CI/CD pipelines.

5️⃣ Aligned with AWS Multi-Account Reference Architecture

AWS explicitly recommends:

Global management and identity region (us-east-1)
Regional landing zones for workloads

This pattern is described in the AWS Multi-Account Landing Zone reference architecture, and most enterprise Control Tower deployments follow the same approach.

Control Tower & AFT Integration

Landing Zone

Deployed via AWS Control Tower in the management account.
Establishes:
- AWS Organizations
- Log Archive and Audit accounts
- Baseline guardrails (AWS-managed)

Account Factory for Terraform (AFT)

Provides GitOps-based account lifecycle management.
Each account is provisioned using Terraform pipelines that:
- Enroll the account in Control Tower
- Apply OU-specific baselines (e.g., Config, logging)
- Tag accounts automatically
AFT runs in its own AFT Management Account.

Account Hierarchy Diagram

Management Account
 ├── AWS Control Tower (Landing Zone)
 ├── AFT Pipelines (Account Factory for Terraform)
 └── AWS Organizations (Root OU)
      ├── Security OU
      ├── Internal OU
      ├── NPR Networking OU
      ├── PRD Networking OU
      └── Deprecated OU

Networking & Shared Services Strategy

Service	Owning Account	Scope	Sharing Mechanism
Transit Gateway (TGW)	Internal Comms (per env)	Environment-specific	AWS RAM (within the env)
Internet Gateway (IGW)	External Comms (per env)	Environment-specific	Single point of entry - not shared
NAT Gateway (NGW)	Shared Services	Cross-environment	Single point of exit - not shared
Network Firewall (NFW)	Shared Services	Cross-environment	AWS RAM + TGW routing
VPC Endpoint Services	Shared Services	Org-wide	Route53 Reslover

Governance & Security Controls

Governance is enforced using three layers:

Layer	Type	Enforcement Mechanism
Preventive	Service Control Policies (SCPs)	AWS Organizations
Detective	Config / Security Hub / GuardDuty	Audit account
Proactive	CloudFormation Hooks / AFT Customizations	Terraform baselines

Core SCP Pack (Preventive Guardrails)

1️⃣ Deny Root User Access

Prevents use of root credentials in any account.

{
  "Sid": "DenyRootUser",
  "Effect": "Deny",
  "Action": "*",
  "Resource": "*",
  "Condition": {
    "StringLike": {
      "aws:PrincipalArn": "arn:aws:iam::*:root"
    }
  }
}

2️⃣ Restrict Regions (EU + us-east-1)

{
  "Sid": "DenyOutsideApprovedRegionsExceptIdentity",
  "Effect": "Deny",
  "NotAction": [
    "iam:*","organizations:*","route53:*","sso:*","support:*"
  ],
  "Resource": "*",
  "Condition": {
    "StringNotEquals": {
      "aws:RequestedRegion": ["eu-west-1", "eu-west-2", "us-east-1"]
    }
  }
}

3️⃣ Deny Unapproved Network Creation

{
  "Sid": "DenyUnapprovedNetworking",
  "Effect": "Deny",
  "Action": [
    "ec2:CreateInternetGateway","ec2:AttachInternetGateway",
    "ec2:CreateNatGateway","ec2:CreateVpcPeeringConnection",
    "ec2:CreateTransitGateway","ec2:DeleteTransitGateway"
  ],
  "Resource": "*"
}

4️⃣ Restrict Service Creation

Resource	Allowed Account(s)	SCP Condition
TGW (Transit Gateway)	Internal Comms (PRD/NPR)	`aws:PrincipalAccount` = INC IDs
IGW (Internet Gateway)	External Comms (PRD/NPR)	`aws:PrincipalAccount` = EXC IDs
NFW (Network Firewall)	Shared Services	`aws:PrincipalAccount` = SSV ID
NGW (NAT Gateway)	Shared Services	`aws:PrincipalAccount` = SSV ID
VPC Endpoint Services	Shared Services	`aws:PrincipalAccount` = SSV ID

5️⃣ Tag Enforcement

{
  "Sid": "RequireStandardTags",
  "Effect": "Deny",
  "Action": ["ec2:RunInstances","rds:CreateDBInstance","s3:CreateBucket"],
  "Resource": "*",
  "Condition": {
    "Null": {
      "aws:RequestTag/Environment": "true",
      "aws:RequestTag/Owner": "true"
    }
  }
}

6️⃣ Deny Org Tampering

Protects core logging and control-plane resources

{
  "Sid": "DenyOrgTampering",
  "Effect": "Deny",
  "Action": [
    "organizations:LeaveOrganization",
    "cloudtrail:StopLogging",
    "config:StopConfigurationRecorder"
  ],
  "Resource": "*"
}

7️⃣ Deprecated OU Policy

{
  "Sid": "DenyNewCreates",
  "Effect": "Deny",
  "Action": ["*"],
  "Resource": "*",
  "Condition": {
    "StringEquals": {
      "aws:ResourceTag/Environment": "deprecated"
    }
  }
}

Control Layers Explained

Layer	Description	Managed By
Control Tower	Creates baseline governance (OU, guardrails, log archive, audit)	AWS
AFT	Automates account provisioning, tagging, baseline controls	Platform team
SCPs	Prevent actions that violate org standards (region, network, security)	Org root
Delegated Security Accounts	Detect & monitor compliance (Config, GuardDuty)	Security team

Deployment Workflow

Step	Action	Tool
1	Enable Control Tower (Landing Zone)	Console / CLI
2	Bootstrap AFT management	Terraform
3	Create OUs + SCPs	Terraform
4	Submit account requests via AFT	GitOps
5	Apply OU-specific baselines	Terraform
6	Validate security controls	AWS Config / Security Hub
7	Continuous compliance	Detective + Proactive guardrails

Future Enhancements

Add Policy Staging OU for testing new SCPs.
Integrate Proactive Controls (CloudFormation hooks).
Automate SCP compliance drift detection using AWS Config custom rules.
Add organizational backup plans (AWS Backup delegated admin).

Deploy FluxCD on Amazon EKS with IRSA for ECR Authentication (Full Terraform Example)

Santanu Das — Tue, 28 Oct 2025 01:45:16 +0000

The Goal
Architecture
End-to-End Flow
Deployment Overview
Terraform Implementation
Verification
Troubleshooting
Security
Change Log

The New Zenler is moving to a shiny new platform (at the time of this writing), where we decided to use FluxCD as our Continuous Delivery tool. The otherday, one of our developers reported that FluxCD suddenly stopped pulling the latest image from ECR and when I checked, I saw this error in the log:

failed to configure authentication options: operation error ECR: GetAuthorizationToken, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, canceled, context deadline exceeded

That moment I knew the reson for the FluxCD pulling-image failure: EC2 Instance Metadata Service (IMDS) wasn't being able to retrieve credentials via the instance profile on EC2 nodes; the code attempted the default credential chain (which includes IMDS) but IMDS wasn't available/accessible; therefore, Flux didn’t have the redentials to call ECR APIs and coundn’t get the token, hence the failure!!

As I was already using IRSA for CNI, CSI etc. I thought this is a great oppurtunity to move Flux from using IMDS to IRSA as well. This document captures my journey from the thought to the finishing-line, the issues I exprienced and how did I solve those to achieve the end goal.

🎯 Goal

Deploy FluxCD (Operator + Flux Instance) on Amazon EKS with IAM Roles for Service Accounts (IRSA), so Flux Image Reflector/Automation can authenticate to Amazon ECR without node credentials, and auto-deploy images from ECR.

🏗️ High-Level Architectural View

The diagram below shows how Bitbucket, FluxCD controllers, and AWS ECR interact: Bitbucket syncs manifests to Flux controllers, which pull images securely from ECR using IRSA

Component	Description
🧩 `Bitbucket Git Repo`	Holds our GitOps manifests, synced by Flux Source Controller
⚙️ `Flux Source Controller`	Pulls manifests from that Bitbucket repo
🧱 `Flux Kustomize Controller`	Applies manifests to our cluster
🔁 `Flux Image Automation Controller`	Watches image updates and commits manifest changes back to Git
🔍 `Flux Image Reflector Controller`	Scans ECR for image tags and metadata
🔐 `EKS OIDC Provider`	Issues tokens used by IRSA authentication
🧾 `STS AssumeRoleWithWebIdentity`	Exchanges OIDC token for temporary IAM credentials
🧠 `IRSA IAM Role`	Grants scoped ECR access (read + token retrieval)
🐳 `Amazon ECR Repository`	Stores application images, queried by Image Reflector

🔄 End-to-End Flow

Pod starts with SA image-reflector-controller.
EKS injects an OIDC web identity token.
AWS STS exchanges it for temporary IAM creds.
Flux Image Reflector uses them to call ECR APIs.
Reflector lists tags → Image Automation commits new manifests → Kustomize applies updates.

📦 Deployment Overview

IRSA IAM role for Flux image controllers:
- Module: terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts (v6.2.1)
- Trusted via your EKS OIDC provider
- Managed policy granting ECR read + GetAuthorizationToken
ServiceAccount annotations (Kubernetes) for:
- flux-system/image-reflector-controller
- flux-system/image-automation-controller
Flux Operator Helm chart (v0.32.0)
Flux Instance Helm chart (distribution v2.4.0), Git sync to your Bitbucket repo

⚙️ Terraform Implementation

1️⃣ IAM Policy for ECR (least privilege)

# ------------------------------------------------------
# IRSA policy document
# ------------------------------------------------------
data "aws_iam_policy_document" "flux_ecr_read" {
statement {
  sid       = "GetAuthTokenForECR"
  effect    = "Allow"
  actions   = ["ecr:GetAuthorizationToken"]
  resources = ["*"]
}

statement {
  sid     = "ReadOnlyAccessToECR"
  effect  = "Allow"
  actions = [
    "ecr:BatchCheckLayerAvailability",
    "ecr:GetDownloadUrlForLayer",
    "ecr:BatchGetImage",
    "ecr:DescribeImages",
    "ecr:DescribeRepositories",
    "ecr:ListImages",
    "ecr:DescribeRegistry",
  ]
  resources = [
  "arn:aws:ecr:${var.aws_region}:${var.aws_acc_id}:repository/*"
  ]
}

// Render the policy document
resource "aws_iam_policy" "flux_ecr_ro" {
  name   = "${var.name_prefix}-flux-ecr-ro"
  policy = data.aws_iam_policy_document.flux_ecr_read.json
}

📝 Note: ecr:GetAuthorizationToken must always be unscoped (Resource="*"), since this API isn’t repository-specific

⚠️ Common pitfall: Avoid something like: repository/<apps>/* — ECR isn’t hierarchical; that pattern doesn’t match e.g. apps/mainapp and causes AccessDeniedException.

2️⃣ IRSA Role for Flux Controllers

# ------------------------------------------------------
# Flux Image-Controller IRSA
# ------------------------------------------------------
module "flux_irsa" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts"
  version = "6.2.1"

  name = "${var.name_prefix}-flux-imgctrl-irsa"

  oidc_providers = {
    automation = {
      provider_arn               = var.eks_oidc_provider
      namespace_service_accounts = ["flux-system:image-automation-controller"]
    }
    reflector = {
      provider_arn               = var.eks_oidc_provider
      namespace_service_accounts = ["flux-system:image-reflector-controller"]
    }
  }

  policies = {
    ecr_ro = aws_iam_policy.flux_ecr_ro.arn
  }
}

💡 One IRSA role for both controllers simplifies management; separation is optional.

3️⃣ Annotate Flux ServiceAccounts

# ------------------------------------------------------
# Controller annotations for SA
# ------------------------------------------------------
resource "kubernetes_annotations" "flux_irsa_sa" {
  for_each    = toset(["reflector", "automation"])
  api_version = "v1"
  kind        = "ServiceAccount"

  metadata {
    name      = "image-${each.value}-controller"
    namespace = "flux-system"
  }

  annotations = {
    "eks.amazonaws.com/role-arn" = module.flux_irsa.arn
  }
}

4️⃣ Use of Bitbucket credential

# ------------------------------------------------------
# FluxCD namespace
# ------------------------------------------------------
resource "kubernetes_namespace_v1" "flux_system" {
  metadata {
    name = var.flux_namespace
    annotations = {
      name                              = var.flux_namespace
      "kustomize.toolkit.fluxcd.io/ssa" = "Ignore"
    }
  }
  depends_on = [null_resource.kube_config]

  lifecycle {
    ignore_changes = [
      metadata[0].labels,
      metadata.0.annotations,
    ]
    create_before_destroy = false
  }
}

# ------------------------------------------------------
# Inject BB credential
# ------------------------------------------------------
resource "kubernetes_secret_v1" "bb_passwd" {
  metadata {
    name      = "bb-${var.bb_user}-passwd"
    namespace = kubernetes_namespace_v1.flux_system.id
  }
  data = {
    username = var.bb_user
    password = var.bb_user_secrets.apwd
  }
  type = "Opaque"
}

5️⃣ Flux Operator & Flux Instance Helm releases

# ------------------------------------------------------
# Flux Operators and Instance helm-chart
# ------------------------------------------------------
resource "helm_release" "flux_operator" {
  repository       = "oci://ghcr.io/controlplaneio-fluxcd/charts"
  chart            = "flux-operator"
  version          = "0.32.0"
  name             = "flux-operator"
  namespace        = kubernetes_namespace_v1.flux_system.id
  create_namespace = false
  wait             = true
}

resource "helm_release" "flux_instance" {
  depends_on = [helm_release.flux_operator]
  repository = "oci://ghcr.io/controlplaneio-fluxcd/charts"
  chart      = "flux-instance"
  name       = "flux"
  namespace  = kubernetes_namespace_v1.flux_system.id

  # Flux components and kustomize patches
  values = [
    file("${path.module}/values/components.yaml")
  ]

  # Flux distribution
  set {
    name  = "installCRDs"
    value = true
  }
  set {
    name  = "instance.distribution.version"
    value = var.flux_version
  }
  set {
    name  = "instance.distribution.registry"
    value = var.flux_registry
  }

  # Configure Flux Git sync
  set {
    name  = "instance.sync.eks"
    value = "GitRepository"
  }
  set {
    name  = "instance.sync.url"
    value = "${var.bb_base_url}/${var.bb_flex_repo}.git"
  }
  set {
    name  = "instance.sync.path"
    value = var.git_path          #e.g. "/"
  }
  set {
    name  = "instance.sync.ref"
    value = var.git_ref           #e.g. "refs/heads/main"
  }
  set {
    name  = "instance.sync.pullSecret"
    value = kubernetes_secret_v1.bb_passwd.metadata[0].name
  }
}

📝 Note: installCRDs=true ensures proper CRD registration before Flux Instance is applied.

📝 Note: instance.distribution.version can be safely upgraded once IRSA/ECR is stable.

✅ Verification Checklist

1️⃣ IRSA projected token exists

kubectl exec -n flux-system deploy/image-reflector-controller -- \        ls -l    /var/run/secrets/eks.amazonaws.com/serviceaccount
total 0
lrwxrwxrwx    1 root     1337            12 Oct 27 19:49 token -> ..data/token

2️⃣ Token audience is STS

kubectl exec -n flux-system deploy/image-reflector-controller -- \
  cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token \
  | jq -R 'split(".") | .[1] | @base64d | fromjson' | jq '.aud,.sub'
[
  "sts.amazonaws.com"
]
"system:serviceaccount:flux-system:image-reflector-controller"

3️⃣ ImageRepository status

kubectl get imagerepository -n flux-system mainapp -o yaml | yq '.status.conditions'
[
  {
    "lastTransitionTime": "2025-10-27T18:48:23Z",
    "message": "successful scan: found 2 tags",
    "observedGeneration": 1,
    "reason": "Succeeded",
    "status": "True",
    "type": "Ready"
  }
]

🧰 Troubleshooting

Symptom	Probable Cause	Fix
`no EC2 IMDS role found`	IRSA not active or the Pod still not using it	Annotate SA + restart pod
Only `token` file exists	Normal (modern IRSA token projection)	Decode token for `aud: sts.amazonaws.com`
`AccessDenied: ecr:GetAuthorizationToken`	Role missing token permission	Add `ecr:GetAuthorizationToken` on `*`
`AccessDenied` on repo read	Wrong ARN pattern	Use exact ARN (e.g.`repository/<prefix>/<app_name>`) or allow `repository/*`
SA annotation missing	FluxInstance CRD type is list of strings	Patch SAs with Terraform annotations

🔒 Security Notes

Scope ECR access to specific repositories whenever possible (e.g. repository/<repo>)
Keep ecr:GetAuthorizationToken unscoped
IRSA isolates pod access — no node IAM exposure
Rotate IAM policies periodically and audit for unused repositories

🏁 Conclusion

By combining FluxCD with IRSA, we’ve achieved:

Pod-scoped ECR authentication (no IMDS dependency)
Least-privilege IAM access
Fully automated GitOps workflow via Terraform

This setup ensures that your CI/CD pipeline runs natively within EKS — secure, scalable, and elegant.

🕑 Change Log

Operator: v0.32.0
Flux: v2.4.0
IAM module: v6.2.1
Shared IRSA for image controllers
Correct repo ARN (no nested wildcard)

EFS Data Across Multiple EKS Applications Using Terraform

Santanu Das — Fri, 24 Oct 2025 20:17:07 +0000

The Problem
The Solution
Achievements
Verification
The Conclusion

In the first part of this guide, we walked through setting up Amazon EFS with Amazon EKS using Terraform — covering file system provisioning, IAM integration, and CSI driver configuration. And just after I thought we are back in business, one of the developers very proudly announce: This won't work for all of our apps; at least two of the apps need to share the same data directory. 🤪

Went back to the drawing board with one subtle challenge:

How do we safely share the same EFS directory between multiple EKS applications while keeping our Terraform code clean, not over-engineered, declarative, and idempotent?

Eventually came up with some ideas: This article shows how did we solve that precisely — using distinct EFS access points per unique path, rather than per app, here at Zenler.

⚙️ The Problem

When multiple applications each create their own EFS access point, even if they use the same absolute path (e.g. /data/zenler-app), AWS treats those as separate mount namespaces.

That means:

Actual EFS path	Access Point View
`/data/zenler-app/file1.txt`	`/file1.txt` (inside both access points)
`/data/zenler-app`	`/` (inside both access points)

However, EFS does not create a shared view between two access points mounted at the same root-path unless they were both created pointing to the same inode (directory object).

Example:

App	Configured Path	Actual Mount Namespace
tenant-app	`/data/zenler-app`	Isolated Access Point A
zenler-app	`/data/zenler-app`	Isolated Access Point B

That last bit is subtle: When we created the first access point, AWS created (if not already existing) the path /data/zenler-app and initialized it with UID/GID 1000:1000.
When we created the second access point, even though we wrote the same path (/data/zenler-app), AWS treats that as its own isolated root handle (like a different "view" into the same filesystem), not a symbolic reference to the existing directory.

In effect, each access point got its own chroot view that does not automatically unify — the same logical path string doesn’t mean same underlying inode reference. As a result, pods using these different access points can’t see each other’s files, because each access point’s root_directory.path becomes its own root scope.

✅ The Solution — Access Points per Distinct Path

The solution is simple but powerful: Deduplicate access points by path, so each unique EFS directory has exactly one access point.

1️⃣ First - Normalize the EFS volume paths

Created a local variable:

locals {
  efs_access_points = {
    for K2, V2 in var.apps_config : K2 => {
      path = (V2.efs_volume.path == null ? "/data/${K2}" : V2.efs_volume.path)
    } if V2.efs_volume.attached
  }
}

In our setup, it outputs something like this:

{
  "tenant-app" = { "path" = "/data/zenler-app" }
  "zenler-app" = { "path" = "/data/zenler-app" }
}

2️⃣ Create one EFS Access Point per unique path

resource "aws_efs_access_point" "this" {
  for_each = var.service_enabled ? toset(
    distinct(concat([
      for KV in values(local.efs_access_points) : values(KV)
    ]...))
  ) : []

  file_system_id = aws_efs_file_system.this[var.service_name].id

  posix_user {
    uid = 1000
    gid = 1000
  }

  root_directory {
    path = each.key
    creation_info {
      owner_uid   = 1000
      owner_gid   = 1000
      permissions = "0775"
    }
  }

  tags = {
    Name     = "${local.efs_template_name}ap-${replace(each.key, "/", "-")}"
    Module   = var.tf_module_name
    Resource = "aws_efs_access_point.this"
  }
}

Now Terraform will only create one access point per distinct directory.
If two apps share /data/zenler-app, they’ll reuse the same EFS access point automatically.

3️⃣ Re-associate apps with the correct Access Point

Then, in efs_pvs module, only change the efs_access_points parameter to this:

# ------------------------------------------------------
# EFS-PVS local-module reference
# ------------------------------------------------------
module "efs_pvs" {
  source = "./modules//efs_pvs"
  efs_access_points = {
    for K1, V1 in local.efs_access_points : K1 => {
      apid = aws_efs_access_point.this[V1.path].id
      nmsp = var.apps_config[K1].name_space
    }
  }
  efs_dns_name       = aws_efs_file_system.this.dns_name
  efs_file_system_id = aws_efs_file_system.this.id
  efs_name_prefix    = local.efs_template_name
}

which will return the set, exactly the formatted like before (with the app-name as the Key) but the with the same apid for the apps need to friendly to each other:

{
  "tenant-app" = {
    "apid" = "fsap-08be57c06a0b1c2d4"
    "nmsp" = "zenler-apps"
  }
  "zenler-app" = {
    "apid" = "fsap-08be57c06a0b1c2d4"
    "nmsp" = "zenler-apps"
  }
}

and the rest of the things in the module/efs_pvs stay as it is, without any change required in either PersistentVolume or PersistentVolumeClaim resource.

And that's what actually it!! 🙌
(until the next call from the app teams 😬)

🧠 Achievements

Benefit	Description
1️⃣ Shared access	Multiple pods can see the same files in real time
2️⃣ Fewer resources	Only one AP per path instead of one per app
3️⃣ Simpler IAM & POSIX model	Each directory has a single enforcement layer
4️⃣ Idempotent Terraform	Prevents churn and duplicate APs
5️⃣ Flexible design	Seamless unique and shared paths

🔍 Quick Verification

aws efs describe-access-points --file-system-id <fs-id>
kubectl exec -it pod/tenant-app -- ls /mnt/efs
kubectl exec -it pod/zenler-app -- ls /mnt/efs

✅ The Conclusion

By deduplicating EFS access points by unique path, you can safely allow multiple EKS applications to share the same directory on Amazon EFS — without namespace conflicts or code complexity.

This pattern keeps your Terraform configuration declarative, scalable, and ready for production.

Automating Transit Gateway Attachment Tagging Using EventBridge and Lambda

Santanu Das — Thu, 23 Oct 2025 17:52:03 +0000

The Problem
Underlying Issues
Solution Overview
Architecture Diagram
Implementation Details
Function Code
IAM Requirements
Testing Approach
Terraform Implementation
Conclusion
End Result
Troubleshooting
Key Takeaways
Future Enhancements

In a multi-account AWS Organization, managing shared Transit Gateways can quickly become messy. Every account owns its own attachments — but when those attachments are created cross-account (for example, via Terraform), the TGW owner account receives untagged attachments.

This post explains how we automated cross-account TGW attachment tagging at Zenler using CloudTrail, EventBridge, and Lambda — ensuring TGW owner account automatically inherits the requester’s tags within seconds, when a requester VPC attaches to a shared TGW.

TL;DR
This setup automatically copies VPC tags from the requester account to the TGW attachment in the owner account. It is fully event-driven, serverless, and scales with TGW usage — no polling or manual intervention required.

The Problem We Have Had

In our multi-account AWS environment, we rely on Transit Gateway (TGW) to connect shared networking components with application VPCs. However, when a TGW attachment is created across accounts (e.g., by Terraform in a requester account), the attachment in the TGW owner account is created without tags.

This caused several issues:

Attachments appeared as untagged resources in the TGW owner account.
Reporting and cost-allocation automation relying on tags failed.
Manual tagging was error-prone and inconsistent.

We needed an automated, cross-account, event-driven mechanism to apply consistent tags to new TGW attachments as soon as they are created or accepted.

Underlying Issues

By default:

The requester’s VPC tags do not propagate to the TGW attachment.
TGW create/accept API calls are management events, recorded only in CloudTrail, not directly visible to EventBridge unless CloudTrail is integrated.
The TGW attachment resource resides in the TGW owner account, so tagging must happen there, not in the requester’s account.

Solution Overview

We implemented an EventBridge + Lambda automation that listens for CloudTrail management events related to TGW attachments and automatically applies tags from the requester’s VPC.

Event Flow Diagram:
CloudTrail → EventBridge → Lambda → AssumeRole → Tag TGW attachment

Workflow Summary:

CloudTrail logs CreateTransitGatewayVpcAttachment and AcceptTransitGatewayVpcAttachment events.
CloudTrail is configured to send events to EventBridge.
An EventBridge rule filters TGW attachment events.
The rule triggers a Lambda function in the TGW owner account.
The Lambda:
- Parses the CloudTrail event (supports both new and old formats).
- Identifies the requester account and VPC ID.
- Assumes a cross-account role into the requester account.
- Retrieves VPC tags and applies them to the TGW attachment, adding a suffix to the Name tag.

Architecture Diagram

Implementation Details

When a requester account creates a TGW VPC attachment, the TGW owner account receives the attachment resource without any inherited tags. This function then listens to CloudTrail → EventBridge events, to identify the new TGW attachment(s), and automatically applies the VPC tags from the requester’s account to the corresponding TGW attachment in the owner’s account.

1️⃣ CloudTrail Configuration:

Multi-region trail enabled (optional, if needed).
Management events (IncludeManagementEvents = true).
Integrated with EventBridge (event_bus_name = "default").

2️⃣ EventBridge Rule Filter

The EventBridge rule filter defines which AWS CloudTrail events should trigger the Lambda function, making sure ensures it's only invoked when Transit Gateway (TGW) attachments are created or accepted — not for every EC2 API call.

{
  "source": ["aws.ec2"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["ec2.amazonaws.com"],
    "eventName": [
      "CreateTransitGatewayVpcAttachment",
      "AcceptTransitGatewayVpcAttachment"
    ]
  }
}

3️⃣ Event Sample

The filter is evaluated each time CloudTrail logs a new management event and inspects each incoming event, like the one below:

{
  "version": "0",
  "id": "test-id",
  "detail-type": "AWS API Call via CloudTrail",
  "source": "aws.ec2",
  "account": "123456789012",
  "time": "2025-09-29T10:05:01Z",
  "region": "eu-west-2",
  "resources": [],
  "detail": {
    "eventVersion": "1.10",
    "userIdentity": {
      "type": "AWSAccount",
      "principalId": "AROA6Q5A2P2LQJEJCKFPD:aws-go-sdk-1759140300350852736",
      "accountId": "123456789012"
    },
    "eventTime": "2025-09-29T10:05:01Z",
    "eventSource": "ec2.amazonaws.com",
    "eventName": "CreateTransitGatewayVpcAttachment",
    "awsRegion": "eu-west-2",
    "sourceIPAddress": "25.xxx.xxx.xxx",
    "userAgent": "APN/1.0 HashiCorp/1.0 Terraform/1.11.0 (+https://www.terraform.io) terraform-provider-aws/6.2.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.36.5 ua/2.1 os/linux lang/go#1.24.4 md/GOOS#linux md/GOARCH#arm64 api/ec2#1.229.0 m/i",
    "requestParameters": {
      "CreateTransitGatewayVpcAttachmentRequest": {
        "Options": {
          "Ipv6Support": "disable",
          "ApplianceModeSupport": "enable",
          "DnsSupport": "enable"
        },
        "TransitGatewayId": "tgw-87654321b6418c2bc",
        "VpcId": "vpc-34239b0b1106a01f0",
        "TagSpecifications": {
          "ResourceType": "transit-gateway-attachment",
          "tag": 1,
          "Tag": [
            {
              "Value": "Santanu Das",
              "tag": 1,
              "Key": "Author"
            },
            {
              "Value": "zenler",
              "tag": 2,
              "Key": "CostCentre"
            },
            { .... }
          ]
        },
        "SubnetIds": [
          {
            "tag": 1,
            "content": "subnet-04b7a3c2512345678"
          },
          {
            "tag": 2,
            "content": "subnet-12345678017c583d0"
          }
        ]
      }
    },
    "responseElements": {
      "CreateTransitGatewayVpcAttachmentResponse": {
        "xmlns": "http://ec2.amazonaws.com/doc/2016-11-15/",
        "transitGatewayVpcAttachment": {
          "tagSet": "HIDDEN_DUE_TO_SECURITY_REASONS",
          "creationTime": "2025-09-29T10:05:01.000Z",
          "transitGatewayAttachmentId": "tgw-attach-0b575dfd1fcbdbf6b",
          "transitGatewayId": "tgw-87654321b6418c2bc",
          "vpcId": "vpc-34239b0b1106a01f0",
          "options": {
            "applianceModeSupport": "enable",
            "securityGroupReferencingSupport": "enable",
            "dnsSupport": "enable",
            "ipv6Support": "disable"
          },
          "state": "pending",
          "vpcOwnerId": "123456789012",
          "subnetIds": {
            "item": [
              "subnet-04b7a3c2512345678",
              "subnet-12345678017c583d0"
            ]
          }
        },
        "requestId": "11a9ce7e-1f5d-4c69-86cf-d4043309f8ad"
      }
    },
    "requestID": "11a9ce7e-1f5d-4c69-86cf-d4043309f8ad",
    "eventID": "498e93ad-edc8-415d-be8e-7420d71c4b10",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "123456789013",
    "sharedEventID": "facd716b-20c2-4496-9109-d10c4965bbf3",
    "eventCategory": "Management",
    "tlsDetails": {
      "tlsVersion": "TLSv1.3",
      "cipherSuite": "TLS_AES_128_GCM_SHA256",
      "clientProvidedHostHeader": "ec2.eu-west-2.amazonaws.com"
    }
  }
}

4️⃣ Lambda Function

Receives only those events related to TGW attachment operations.
Handles both CreateTransitGatewayVpcAttachmentResponse and AcceptTransitGatewayVpcAttachmentResponse.
Fallback for legacy format transitGatewayAttachment.
Detects requester account dynamically via userIdentity.accountId.
Cross-account role assumption for tag retrieval.
Appends suffix to Name tag and applies in TGW owner account.

The Function Code

ℹ️ The code below is written according to the event as sampled above. It may need to readjusting the part under # Handle new CloudTrail Create or Accept responses

import os

class Config:
    def __init__(self):
        self.CROSS_ROLE_NAME = os.environ.get('XACC_ROLE_NAME', 'xaccount-tgw-autotag-Role')
        self.TAG_NAME_PREFIX = os.environ.get('TAG_NAME_PREFIX', 'TGW-Attachment')
        self.TAG_NAME_SUFFIX = os.environ.get('TAG_NAME_SUFFIX', 'xacc-att')

# Usage
config = Config()

=================================================================

import boto3
import os, json, logging
from config import config

logger = logging.getLogger()
logger.setLevel(logging.INFO)

DEFAULT_NAME_PREFIX = config.TAG_NAME_PREFIX
NAME_TAG_SUFFIX     = config.TAG_NAME_SUFFIX
CROSS_ROLE_NAME     = config.CROSS_ROLE_NAME

def assume_role(account_id, role_name):
    sts = boto3.client("sts")
    role_arn = f"arn:aws:iam::{account_id}:role/{role_name}"

    logger.info(f"Attempting to assume role {role_arn}")
    try:
        response = sts.assume_role(
            RoleArn=role_arn,
            RoleSessionName="TGWAutotagSession"
        )
        creds = response["Credentials"]
        assumed_session = boto3.Session(
            aws_access_key_id=creds["AccessKeyId"],
            aws_secret_access_key=creds["SecretAccessKey"],
            aws_session_token=creds["SessionToken"],
        )
        # Debug identity
        identity = assumed_session.client("sts").get_caller_identity()
        logger.info(f"Assumed role identity: {identity}")
        return assumed_session
    except Exception as e:
        logger.error(f"Failed to assume role {role_arn}: {str(e)}")
        raise

def handler(event, context):
    logger.info("Received event: %s", json.dumps(event))

    try:
        detail = event["detail"]["responseElements"]

        # Handle new CloudTrail Create or Accept responses
        if "CreateTransitGatewayVpcAttachmentResponse" in detail:
            tgw_attach = detail["CreateTransitGatewayVpcAttachmentResponse"]["transitGatewayVpcAttachment"]
        elif "AcceptTransitGatewayVpcAttachmentResponse" in detail:
            tgw_attach = detail["AcceptTransitGatewayVpcAttachmentResponse"]["transitGatewayVpcAttachment"]

        # Otherwise, Fallback: older CloudTrail format
        elif "transitGatewayAttachment" in detail:
            tgw_attach = detail["transitGatewayAttachment"]

        else:
            raise KeyError("No recognizable TGW attachment block in event")

        attachment_id = tgw_attach["transitGatewayAttachmentId"]
        vpc_id = tgw_attach.get("vpcId") or tgw_attach.get("resourceId")

        # requester_account from New or fallback to Old format
        if "userIdentity" in event["detail"] and "accountId" in event["detail"]["userIdentity"]:
            requester_account = event["detail"]["userIdentity"]["accountId"]  # new format
        else:
            requester_account = event["account"]  # old format

    except Exception as e:
        logger.error(f"Could not parse event for attachment: {str(e)}")
        return

    logger.info(f"Processing TGW Attachment: {attachment_id} from account {requester_account}, VPC {vpc_id}")

    # Step 1: Assume requester role
    try:
        requester_session = assume_role(requester_account, CROSS_ROLE_NAME)
        ec2_requester = requester_session.client("ec2")
        vpc = ec2_requester.describe_vpcs(VpcIds=[vpc_id])["Vpcs"][0]
        vpc_tags = vpc.get("Tags", [])
        logger.info(f"Requester VPC tags: {vpc_tags}")
    except Exception as e:
        logger.error(f"Failed to fetch VPC tags from requester account {requester_account}: {str(e)}")
        return

    # Step 2: Build tags
    if vpc_tags:
        tags_to_copy = []
        for tag in vpc_tags:
            key = tag["Key"]
            value = tag["Value"]

            # If it's the Name tag, add suffix
            if key == "Name":
                value = f"{value}-{NAME_TAG_SUFFIX}"

            tags_to_copy.append({"Key": key, "Value": value})
    else:
        tags_to_copy = [{"Key": "Name", "Value": f"{DEFAULT_NAME_PREFIX}-{vpc_id}"}]

    # Print out final tags before applying
    logger.info(f"Final tags_to_copy for TGW attachment {attachment_id}: {tags_to_copy}")

    # Step 3: Apply tags in TGW owner account
    try:
        ec2_owner = boto3.client("ec2")
        ec2_owner.create_tags(Resources=[attachment_id], Tags=tags_to_copy)
        logger.info(f"✅ Successfully tagged TGW attachment {attachment_id}")
    except Exception as e:
        logger.error(f"❌ Failed to tag TGW attachment {attachment_id}: {str(e)}")

Configuration Parameters

Variable	Description	Example
`TAG_NAME_PREFIX`	Default prefix if no VPC name exists	`auto-tgw`
`TAG_NAME_SUFFIX`	Suffix appended to “Name” tag	`-xacc-att`
`CROSS_ROLE_NAME`	IAM role assumed in requester account	`TGWAutoTagCrossAccountRole`

Core Functions

1️⃣ assume_role(account_id, role_name)
Assumes the specified IAM role in the target account and returns a boto3 session object.

Responsibilities:

Builds role ARN dynamically: arn:aws:iam:::role/
Uses sts.assume_role for temporary credentials.
Returns a boto3.Session with assumed credentials.

Logging:

Logs identity confirmation from sts.get_caller_identity() for verification.

2️⃣ handler(event, context)
Main entry point for the Lambda function.

Primary Responsibilities:

Parse CloudTrail event for:
- TGW Attachment ID
- VPC ID
- Requester Account ID
Assume cross-account IAM role into requester account.
Retrieve tags from requester VPC.
Construct tags_to_copy list.
Apply tags to TGW attachment in the TGW owner account.

Code Flow Explanation

Event Parsing

1️⃣ Handles multiple CloudTrail formats for backward compatibility:

if "CreateTransitGatewayVpcAttachmentResponse" in detail:
    tgw_attach = detail["CreateTransitGatewayVpcAttachmentResponse"]["transitGatewayVpcAttachment"]
elif "AcceptTransitGatewayVpcAttachmentResponse" in detail:
    tgw_attach = detail["AcceptTransitGatewayVpcAttachmentResponse"]["transitGatewayVpcAttachment"]
elif "transitGatewayAttachment" in detail:  # legacy format
    tgw_attach = detail["transitGatewayAttachment"]

2️⃣ Determines requester account:

if "userIdentity" in event["detail"] and "accountId" in event["detail"]["userIdentity"]:
    requester_account = event["detail"]["userIdentity"]["accountId"]
else:
    requester_account = event["account"]

Cross-Account Tag Retrieval

requester_session = assume_role(requester_account, CROSS_ROLE_NAME)
ec2_requester = requester_session.client("ec2")
vpc = ec2_requester.describe_vpcs(VpcIds=[vpc_id])["Vpcs"][0]
vpc_tags = vpc.get("Tags", [])

Tag Construction

1️⃣ Adds suffix to Name tag:

for tag in vpc_tags:
    if tag["Key"] == "Name":
        tag["Value"] = f"{tag['Value']}-{NAME_TAG_SUFFIX}"

2️⃣ Fallback when no tags exist:

tags_to_copy = [{"Key": "Name", "Value": f"{DEFAULT_NAME_PREFIX}-{vpc_id}"}]

IAM Requirements

TGW Owner Account (Lambda Execution Role)
1️⃣ MUST allow:

{
  "Action": [
    "ec2:CreateTags",
    "sts:AssumeRole"
  ],
  "Resource": "*"
}

Requester Account (Cross-Account Role)
1️⃣ MUST trust TGW Owner account’s Lambda role:

{
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::<tgw-owner-account>:role/<lambda-exec-role>" },
  "Action": "sts:AssumeRole"
}

2️⃣ and MUST allow:

{
  "Action": "ec2:DescribeVpcs",
  "Resource": "*"
}

🧪 Testing Approach

Use Lambda test console with real CloudTrail event payloads.
Verify:
1. Event parsing logs (attachment ID, requester account, VPC ID)
2. Successful assume role identity.
3. Correct tag propagation in TGW console.

Terraform Implementation

The code-snippet below are purely for reference purpose. Our TF projects are bootstraped from Terragrunt and it's a modified version of copy/paste from there, which may not exactly fit in every other development environment.

ℹ️ This work-flow is part of our in-house VPC module, where it does the TGW attachemnt during the VPC creation. The condition var.run_acc != var.tgw_owner_acc determins whether to execute the associated TF resource in the currently running_account or not. provider = aws.tgw switches to the TGW owner's account for the deployment

1️⃣ Lambda Excution Role/Policy

# ------------------------------------------------------
# Policy to attach to Lambda excution Role
# ------------------------------------------------------
data "aws_iam_policy_document" "tgw_owner_acc" {
  count = var.run_acc != var.tgw_owner_acc ? 1 : 0

  statement {
    actions = [
      "ec2:CreateTags",
      "ec2:DescribeTransitGatewayAttachments"
    ]
    effect    = "Allow"
    resources = ["*"]
    sid       = "AllowLambdaCreateTgwAttachmentTags"
  }

  statement {
    actions   = ["sts:AssumeRole"]
    effect    = "Allow"
    resources = [aws_iam_role.tgw_this_acc[local.vpc_name_pfx].arn]
    sid       = "AllowLambdaToAssumeRole"
  }
}

// Generate the policy from above document
resource "aws_iam_policy" "tgw_owner_acc" {
  for_each = var.run_acc != var.tgw_owner_acc ? toset(
    values(var.env_tgw_info)[*].owner_name
  ) : []
  name        = "${lower(each.value)}-${upper(var.run_acc)}-tgw-autotag-Policy"
  path        = "/"
  description = "IAM policy for logging from a lambda"
  policy      = data.aws_iam_policy_document.tgw_owner_acc[0].json

  tags = merge(
    var.extra_tags,
    {
      Name     = "${each.value}-tgw-autotag-Policy"
      Module   = var.tf_module_name
      Resource = "aws_iam_policy.tgw_xacc_autotag"
    }
  )
  provider = aws.tgw
}

// Attach policy to lambda excution Role
resource "aws_iam_role_policy_attachment" "tgw_owner_acc" {
  for_each = var.run_acc != var.tgw_owner_acc ? toset(
    values(var.env_tgw_info)[*].owner_name
  ) : []
  role       = "${lower(each.value)}-${var.lambda_role_suffix}"
  policy_arn = aws_iam_policy.tgw_owner_acc[each.value].arn
  provider   = aws.tgw
}

2️⃣ IAM Role in Requester's Account

# ------------------------------------------------------
# Local IAM role for Lambda to assume
# ------------------------------------------------------
resource "aws_iam_role" "tgw_this_acc" {
  for_each = var.run_acc != var.tgw_owner_acc ? toset(
    [local.vpc_name_pfx]
  ) : []
  #name = "${each.value}-tgw-autotag-Role"
  name = var.xacc_tgw_role_name

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Principal = {
          AWS = [
            for ID in values(var.env_tgw_info)[*].owner_id :
            "arn:aws:iam::${ID}:root"
          ]
        },
        Action = "sts:AssumeRole"
      }
    ]
  })

  tags = merge(
    var.extra_tags,
    {
      Name     = "${each.value}-tgw-autotag-Role"
      Module   = var.tf_module_name
      Resource = "aws_iam_role.tgw_xacc_autotag"
    }
  )
}

//
data "aws_iam_policy_document" "tgw_this_acc" {
  statement {
    actions   = ["ec2:DescribeVpcs"]
    effect    = "Allow"
    resources = ["*"]
    sid       = "AllowLambdaReadOnlyVpcAccess"
  }
}

resource "aws_iam_policy" "tgw_this_acc" {
  for_each = var.run_acc != var.tgw_owner_acc ? toset(
    [local.vpc_name_pfx]
  ) : []
  name        = "${each.value}-tgw-autotag-Policy"
  path        = "/"
  description = "IAM policy for logging from a lambda"
  policy      = data.aws_iam_policy_document.tgw_this_acc.json

  tags = merge(
    var.extra_tags,
    {
      Name     = "${each.value}-tgw-autotag-Policy"
      Module   = var.tf_module_name
      Resource = "aws_iam_policy.tgw_xacc_autotag"
    }
  )
}

// Attach policy to lambda Role
resource "aws_iam_role_policy_attachment" "tgw_this_acc" {
  for_each = var.run_acc != var.tgw_owner_acc ? toset(
    [local.vpc_name_pfx]
  ) : []
  role       = aws_iam_role.tgw_this_acc[each.value].name
  policy_arn = aws_iam_policy.tgw_this_acc[each.value].arn
}

3️⃣ Core Lambda Resources

# ------------------------------------------------------
# Lambda function resource
# ------------------------------------------------------
data "archive_file" "tgw_autotag" {
  count       = var.run_acc == var.tgw_owner_acc ? 1 : 0
  type        = "zip"
  output_path = "${path.module}/autotag_lambda.zip"

  source {
    content  = file("${path.module}/lambda/function.py")
    filename = "function.py"
  }
  source {
    content  = file("${path.module}/lambda/config.py")
    filename = "config.py"
  }
}

// CloudWatch for Lambda function
resource "aws_cloudwatch_log_group" "tgw_autotag" {
  count = var.run_acc == var.tgw_owner_acc ? 1 : 0
  name  = "/aws/lambda/${local.vpc_name_pfx}-tgw-autotag"
  #
  retention_in_days = 14
}

// Lambda-function
resource "aws_lambda_function" "tgw_autotag" {
  count            = var.run_acc == var.tgw_owner_acc ? 1 : 0
  filename         = data.archive_file.tgw_autotag[0].output_path
  function_name    = "${local.vpc_name_pfx}-tgw-autotag"
  handler          = "function.handler"
  role             = var.acc_lambda_role_arn
  runtime          = "python3.12"
  timeout          = 12
  source_code_hash = data.archive_file.tgw_autotag[0].output_base64sha256

  # Advanced logging controls (optional)
  logging_config {
    log_format            = "JSON"
    application_log_level = "INFO"
    system_log_level      = "WARN"
  }

  # Environment variables for config.py
  environment {
    variables = {
      TAG_NAME_PREFIX = local.vpc_name_pfx
      TAG_NAME_SUFFIX = "att-xacc"
      XACC_ROLE_NAME  = var.xacc_tgw_role_name
    }
  }

  # Direct dependency
  depends_on = [
    aws_cloudwatch_log_group.tgw_autotag,
  ]
}

4️⃣ CW Logs Group for CloudTrail

resource "aws_cloudwatch_log_group" "ct_tgw_autotag" {
  count = var.run_acc == var.tgw_owner_acc ? 1 : 0
  name  = "/aws/cloudtrail/${local.vpc_name_pfx}-tgw-autotag"
  #
  retention_in_days = 14
}

// IAM role for CloudTrail to write to CloudWatch Logs
resource "aws_iam_role" "ct_tgw_autotag" {
  count = var.run_acc == var.tgw_owner_acc ? 1 : 0
  name  = "${lower(var.aws_acc_name)}-cloudtrail-assume-Role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Service = "cloudtrail.amazonaws.com"
      }
      Action = "sts:AssumeRole"
    }]
  })
}

//
resource "aws_iam_role_policy" "ct_tgw_autotag" {
  count = var.run_acc == var.tgw_owner_acc ? 1 : 0
  role  = aws_iam_role.ct_tgw_autotag[0].id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = "${aws_cloudwatch_log_group.ct_tgw_autotag[0].arn}:*"
      }
    ]
  })
}

5️⃣ CloudTrail + EventBridge Integration

resource "aws_cloudtrail" "tgw_autotag" {
  count                 = var.run_acc == var.tgw_owner_acc ? 1 : 0
  name                  = "${lower(var.aws_acc_name)}-tgw-attachment-trail"
  enable_logging        = true
  s3_bucket_name        = var.svc_logging_bucket.name
  is_multi_region_trail = false

  cloud_watch_logs_group_arn    = "${aws_cloudwatch_log_group.ct_tgw_autotag[0].arn}:*"
  cloud_watch_logs_role_arn     = aws_iam_role.ct_tgw_autotag[0].arn
  include_global_service_events = true

  # This is the key part: enable EventBridge integration
  event_selector {
    read_write_type           = "All"
    include_management_events = true

    data_resource {
      type   = "AWS::Lambda::Function"
      values = [aws_lambda_function.tgw_autotag[0].arn]
    }
  }

  tags = {
    Module   = var.tf_module_name
    Resource = "aws_cloudtrail.tgw_autotag"
  }
}

6️⃣ EventBridge Rule & Target

resource "aws_cloudwatch_event_rule" "acer_autotag" {
  count       = var.run_acc == var.tgw_owner_acc ? 1 : 0
  name        = "${lower(var.aws_acc_name)}-tgw-attachment-created"
  description = "Triggers when TGW attachments are created"
  event_pattern = jsonencode({
    source      = ["aws.ec2"],
    detail-type = ["AWS API Call via CloudTrail"],
    detail = {
      eventSource = ["ec2.amazonaws.com"],
      eventName = [
        "AcceptTransitGatewayVpcAttachment",
        "CreateTransitGatewayVpcAttachment",
      ]
    }
  })
}

// Lambda target
resource "aws_cloudwatch_event_target" "acet_autotag" {
  count     = var.run_acc == var.tgw_owner_acc ? 1 : 0
  rule      = aws_cloudwatch_event_rule.acer_autotag[0].name
  target_id = "TGWAutotagLambda"
  arn       = aws_lambda_function.tgw_autotag[0].arn
}

// allow EventBridge
resource "aws_lambda_permission" "alp_autotag" {
  count         = var.run_acc == var.tgw_owner_acc ? 1 : 0
  statement_id  = "AllowExecutionFromEventBridge"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.tgw_autotag[0].function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.acer_autotag[0].arn
}

Conclusion: The Benefits

Benefit	Description
Operational Consistency	Ensures every TGW attachment inherits tags from its requester VPC.
Cross-Account Governance	Uses IAM roles and least-privilege policies to enforce separation of duties.
Accurate Cost Allocation	Tags align with Cost Center, Environment, and Ownership standards.
Event-Driven Scalability	Fully serverless; no polling or scheduled jobs.
Future-Proof	Handles both legacy and new CloudTrail event formats.

End Result

The ones with -att are the native to the TGW owner's account and the ones with -att-xacc are cross-account from requesters' account.

Troubleshooting

Error Events	Possible Actions
EventBridge not triggering	Ensure CloudTrail includes management events
Lambda `AccessDenied`	Verify cross-account IAM trust and attached permissions
Attachment not tagged	Event might arrive while TGW attachment is still in pending; consider adding a short delay or retry

Key Takeaways

Keep CloudTrail broad; let EventBridge do the filtering.
Enable include_management_events = true to catch TGW API calls.
Always use ec2.amazonaws.com as the event source for TGW-related APIs.
Avoid excluding EC2 management events unless absolutely necessary.

Future Enhancements

As it goes, there are always some room for improvements. Those are a few things in the bucket-list to implement:

Add SNS notifications for tagging failures.
Extend to tag DeleteTransitGatewayVpcAttachment cleanup events.
Integrate with AWS Service Catalog TagOptions for standardization.
Publish metrics via CloudWatch Insights for audit reporting.

Forem: Santanu Das

Automating Externally-Managed Stateful Rule Groups in AWS Network Firewall Using Terraform (Part-2)

Table Of Contents

Architectural Overview

Engineering Walkthrough

Source Side Config:

Generating the Source of Truth

Configuring the S3 Event Bridge

Establishing the Cross-Account Trust

Capturing and Forwarding the Event(s)

Target End Config:

Custom Event Bus and Permissions

Filtering for Actionable Changes

Lambda IAM Role & Permissions

Runtime Environment:

AWS Lambda Function

Explicit Invocation

Wiring the Pipeline

Filewall-Lambda Function

Function Code

Helper Functions

At the Day's End

Automating Externally-Managed Stateful Rule Groups in AWS Network Firewall Using Terraform (Part 1)

Table Of Contents

Background Overview

Architectural Requirements

The Core Challenges: Dynamic Ingress at Scale

The War: Declarative Tooling in Ephemeral Environments

And Peace: A Hybrid Ownership Model

Engineering Overview

Key Architectural Concepts

Policy Structure & Rule Families

The Conflict Resolution

The Rules of Engagement

The Parent Module: Orchestrating Rule Groups

1️⃣ Defining the Evaluation Hierarchy:

2️⃣ Static Rule Generator

3️⃣ Defining Suricata Rule Groups:

4️⃣ Implementing Domain List Rule Groups:

5️⃣ Integrating AWS-Managed Rule Groups:

The Child Module: Orchestrating the Rule Groups

1️⃣ Static Suricata — Terraform-owned:

2️⃣ Dynamic Suricata — Externally-owned:

3️⃣ Domainlist Rule Groups:

Building the Final Stateful Rule Group Order

The Transformation Logic:

The Final Policy:

The Resulting Impact:

Lessons Learned (Part 1)

What’s Coming in Part-2

BootStrapping Aurora RDS Databases using Lambda and Terraform (Part 3)

Table of Contents

Orchestrating with AWS Step Functions

The Architecture: The State Machine

The Working-flow

Implementation: Infrastructure as Code

Terraform Resources - Root module

Terraform Resources - Child module

Validation: The Orchestration in Action

Conclusion: An automated Database Platform

In This Series

BootStrapping Aurora RDS Databases using Lambda and Terraform (Part 2)

Table Of Contents

The Solution: Decoupling via Purpose-Built Lambda

Strategic Benefits

Core Design Concepts & Requirements

The Secret Sauce: Aurora Data API

The Modular Code Structure

File Structure

Component Responsibilities

The Blueprint: Implementation & Code

IaC: The Terraform Layer

The Logic: Python Lambda Code

The Outcome: Day-Zero Readiness

Deployment Verification

Validating the Execution Flow

In This Series

BootStrapping Aurora RDS Databases using Lambda and Terraform (Part 1)

Table Of Contents

The Problems

1️⃣ Easy installation using `cURL`:

2️⃣ Or, if you prefer `git clone`: