Forem: Darryl Ruggles

EKS and The Cross-AZ Tax: How to Stop Paying AWS $0.02/GB for Traffic That Should Never Leave Your Availability Zone

Darryl Ruggles — Fri, 03 Apr 2026 14:02:49 +0000

I run EKS clusters across three Availability Zones because that's what production resilience typically demands. You probably do too. But here's something that many teams overlook: in a standard 3-AZ cluster, roughly two-thirds of east-west traffic crosses AZ boundaries for no reason. AWS charges $0.01/GB in each direction for that, $0.02/GB round trip, and it adds up fast.

This isn't a theoretical problem. A cluster pushing 10 TB/month of inter-service traffic with 50-67% crossing AZ boundaries racks up $50-67/month in cross-AZ charges ($0.01/GB per direction). Scale that to 50 TB/month and you're looking at $250-335/month. For a cost that's entirely avoidable with a one-line YAML change on your Services.

I built a test environment to measure the actual impact of the major cross-AZ optimization strategies that AWS and Kubernetes provide. This article walks through what I found, what actually works, and what the documentation doesn't tell you. All the code is Terraform and Kubernetes 1.35, using EKS Auto Mode.

If you have not read my earlier article, A Complete Terraform Setup for EKS Auto Mode, that one covers the compute side of EKS cost optimization (Auto Mode, Graviton NodePools, Spot fallback chains, and the Terraform module structure I use). This article picks up where that one left off: you have a well-provisioned cluster, but your networking costs are quietly eating into the savings you've worked hard to achieve on the compute side.

Where the Money Goes

Before optimizing anything, I wanted to understand exactly where cross-AZ traffic happens. There are three places, and the first one is by far the biggest.

East-West Traffic: The Silent Majority

When Pod A calls Pod B through a Service, kube-proxy picks a backend pod using iptables rules. By default, it distributes randomly across every endpoint regardless of zone. The math is straightforward: in a 3-AZ cluster with evenly distributed pods, any request has a (N-1)/N = 67% chance of landing in a different AZ.

That 67% is the number that matters. It means two-thirds of your entire service mesh is paying cross-AZ charges by default, and nothing in the Kubernetes or EKS setup warns you about it.

North-South Traffic: The Extra Hop

The AWS Load Balancer Controller supports "instance" mode and "ip" mode for targeting. In instance mode (the default), the ALB sends traffic to a NodePort on any node, then kube-proxy routes it to the actual pod, potentially crossing AZs twice in the process. In ip mode, the ALB targets pod IPs directly. Switching to IP target mode eliminates the problem entirely.

NAT Gateway Traffic: Death by a Thousand Pulls

Pods in private subnets reach the internet through NAT Gateways. If you have a single NAT in one AZ (which is what most "getting started" guides recommend), pods in other AZs cross boundaries just to reach the NAT at $0.045/GB processing plus the cross-AZ charge. Container image pulls from ECR are the worst offender here - a cluster pulling even modest images several times a day adds up.

Here is what a typical unoptimized cluster looks like. The red lines are cross-AZ traffic you are paying for:

The Test Setup

I built this with Terraform using the same modular approach from my EKS Auto Mode article: community modules for VPC and EKS, the alekc/kubectl provider for Kubernetes resources. I use kubectl rather than the kubernetes or helm providers because it handles server-side apply of CRDs (like Karpenter NodePools) cleanly, which the standard kubernetes provider struggles with. If you followed that article, the structure here will look familiar. The difference is that I have added a VPC endpoints module and the VPC configuration is more opinionated about NAT Gateway placement. Three modules: VPC, EKS, and VPC endpoints.

The VPC: One NAT Per AZ

The first infrastructure decision that matters is NAT Gateway placement. Here is what I used:

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.cluster_name}-vpc"
  cidr = var.vpc_cidr
  azs  = var.azs

  private_subnets = [
    cidrsubnet(var.vpc_cidr, 4, 0),
    cidrsubnet(var.vpc_cidr, 4, 1),
    cidrsubnet(var.vpc_cidr, 4, 2),
  ]

  public_subnets = [
    cidrsubnet(var.vpc_cidr, 4, 3),
    cidrsubnet(var.vpc_cidr, 4, 4),
    cidrsubnet(var.vpc_cidr, 4, 5),
  ]

  # This is the setting that matters
  enable_nat_gateway     = true
  single_nat_gateway     = false
  one_nat_gateway_per_az = true

  enable_dns_hostnames = true
  enable_dns_support   = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = 1
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = 1
  }

  tags = var.tags
}

Each additional NAT Gateway costs about $32/month. You save $0.01/GB in each direction ($0.02/GB round trip) on traffic that no longer crosses AZ boundaries to reach the NAT. That means the extra NAT pays for itself at just 1.6 TB/month of internet-bound traffic per AZ. The NAT processing charge ($0.045/GB) applies regardless of which AZ the NAT is in, so that cost doesn't change with this optimization. For most production clusters, 1.6 TB/month of internet traffic per AZ is a low bar to clear.

I see many Terraform examples online that set single_nat_gateway = true to save money in dev environments. That's fine for dev, but if you copy that pattern into production you are silently adding cross-AZ charges to every internet-bound packet from two of your three AZs.

VPC Endpoints: The Optimization Most Teams Skip

This is the optimization that surprises most teams. Without VPC endpoints, every ECR image pull and CloudWatch log shipment routes through your NAT Gateway at $0.045/GB. I always add endpoints for the services that generate the most traffic:

# S3 Gateway - FREE. No hourly cost, no data cost. No reason not to have this.
resource "aws_vpc_endpoint" "s3" {
  vpc_id            = var.vpc_id
  service_name      = "com.amazonaws.${data.aws_region.current.name}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = var.route_table_ids
}

# ECR API - for docker login and manifest operations
resource "aws_vpc_endpoint" "ecr_api" {
  vpc_id              = var.vpc_id
  service_name        = "com.amazonaws.${data.aws_region.current.name}.ecr.api"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = var.private_subnet_ids
  security_group_ids  = [aws_security_group.vpc_endpoints.id]
  private_dns_enabled = true
}

# ECR Docker - for image layer pulls (this is the high-volume one)
resource "aws_vpc_endpoint" "ecr_dkr" {
  vpc_id              = var.vpc_id
  service_name        = "com.amazonaws.${data.aws_region.current.name}.ecr.dkr"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = var.private_subnet_ids
  security_group_ids  = [aws_security_group.vpc_endpoints.id]
  private_dns_enabled = true
}

# STS - for IRSA and Pod Identity. Low volume but breaks auth if missing.
resource "aws_vpc_endpoint" "sts" {
  vpc_id              = var.vpc_id
  service_name        = "com.amazonaws.${data.aws_region.current.name}.sts"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = var.private_subnet_ids
  security_group_ids  = [aws_security_group.vpc_endpoints.id]
  private_dns_enabled = true
}

The interface endpoints cost about $7.20/month per AZ ($21.60 total across 3 AZs, per endpoint). The S3 and DynamoDB gateway endpoints are free. At $0.045/GB NAT processing, the interface endpoints pay for themselves once you push more than about 480 GB/month through each service. ECR image pulls alone blow past that in most clusters.

A nuance worth understanding: ECR uses two separate service endpoints. The ecr.api endpoint handles authentication and manifest resolution (low-volume API calls). The ecr.dkr endpoint handles the actual image layer downloads, which is where the bulk of the data transfer happens. If you still have a NAT Gateway (and you likely do for other internet-bound traffic), you don't strictly need both. The ecr.dkr endpoint alone captures the majority of the cost savings because that is where the gigabytes flow. The ecr.api calls are small enough that routing them through NAT is negligible. I include both in the repo for completeness, but if you're watching your interface endpoint spend, ecr.dkr plus the free S3 gateway endpoint are the ones that move the needle. ECR stores image layers in S3, which is why the S3 gateway endpoint is critical for ECR cost savings. The STS endpoint is shown in the code above. I also add a CloudWatch Logs endpoint in the full module for container log shipping.

Another gotcha: the security group on your interface endpoints must allow inbound HTTPS (443) from your VPC CIDR. Skip that and you get timeouts that look like DNS failures. Here is what the security group looks like:

resource "aws_security_group" "vpc_endpoints" {
  name_prefix = "${var.cluster_name}-vpce-"
  description = "Security group for VPC interface endpoints"
  vpc_id      = var.vpc_id

  ingress {
    description = "HTTPS from VPC"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = [var.vpc_cidr]
  }
}

This allows the entire VPC CIDR, which is fine for most setups. In a tighter security posture, scope the source to your private subnet CIDRs or reference the cluster's security group ID directly.

EKS Auto Mode on Kubernetes 1.35

I used Kubernetes 1.35, the latest version available on EKS as of March 2026. Traffic Distribution was introduced as alpha in 1.30 with PreferClose, which graduated to GA in 1.33. In 1.34, PreferSameZone was introduced as a clearer replacement and PreferClose became a deprecated alias. Both PreferSameZone and PreferSameNode graduated to stable in 1.35. More on both below.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"

  name               = var.cluster_name
  kubernetes_version = "1.35"

  vpc_id     = var.vpc_id
  subnet_ids = var.private_subnet_ids

  # When node_pools are specified, the module automatically creates
  # the required Auto Mode IAM resources.
  compute_config = {
    enabled    = true
    node_pools = ["general-purpose", "system"]
  }

  # Private access for in-VPC communication, public for kubectl access.
  # In production, restrict public access to specific CIDRs or disable it.
  endpoint_private_access = true
  endpoint_public_access  = true

  # Control plane logging for observability
  enabled_log_types = ["api", "audit", "authenticator"]

  # Cross-AZ traffic visibility in the EKS console
  addons = {
    amazon-cloudwatch-observability = {
      most_recent = true
    }
  }

  # Fine for demos. For production, use scoped access entries with narrower policies.
  enable_cluster_creator_admin_permissions = true

  tags = var.tags
}

I also added a Graviton NodePool at weight 10 for better price-performance, using the same weighted NodePool pattern I described in the Auto Mode article. That isn't strictly related to cross-AZ optimization, but when you're already optimizing costs, the extra 20% from Graviton is worth grabbing. The full NodePool manifest is in terraform/modules/eks/main.tf.

Important Auto Mode detail: both the NodePool and NodeClass API groups are eks.amazonaws.com in Auto Mode, not karpenter.k8s.aws as in self-managed Karpenter installations. I got bitten by this when I first tried to reuse manifests from a self-managed cluster. The resource silently fails to apply with the wrong API group.

Measuring Cross-AZ Traffic: Build a Tool, Don't Guess

The AWS docs describe these routing strategies well enough, but they don't show you how to verify that they are actually working. I built two things for this:

A Zone-Aware Echo Server

Instead of using a generic HTTP echo server, I wrote a small FastAPI app that returns its own AZ, node, and pod name in every response:

# app/zone-echo/server.py
@app.get("/")
async def echo():
    return {
        "zone": _get_zone(),
        "node": os.environ.get("NODE_NAME", "unknown"),
        "pod": os.environ.get("POD_NAME", "unknown"),
        "pod_ip": os.environ.get("POD_IP", "unknown"),
    }

The zone is resolved at startup by querying the Kubernetes API for the node's topology.kubernetes.io/zone label (IMDS is blocked on Auto Mode's Bottlerocket nodes). If you adapt this for non-Auto-Mode clusters, ensure IMDSv2 is enforced (http_tokens = "required") and set the hop limit appropriately. Each response tells you exactly where it came from with zero ambiguity.

A Traffic Measurement Tool

The second tool sends hundreds of requests to each service variant and counts how many stayed in-zone versus crossed AZ boundaries:

# app/traffic-test/measure.py (abbreviated)
def run_test(service: str, num_requests: int, my_zone: str, my_node: str) -> RoutingStats:
    stats = RoutingStats(service=service)

    for i in range(num_requests):
        body, latency = send_request(service)
        stats.total += 1

        if body is None:
            stats.errors += 1
            continue

        stats.latencies_ms.append(latency)
        resp_zone = body.get("zone", "unknown")

        if resp_zone == my_zone:
            stats.same_zone += 1
        else:
            stats.cross_zone += 1

    return stats

Run it as a Kubernetes Job and check the logs:

kubectl apply -f k8s/monitoring/traffic-test-job.yaml
kubectl logs -n cross-az-demo job/traffic-test

The output looks like this:

==============================================================================
CROSS-AZ TRAFFIC MEASUREMENT REPORT
==============================================================================
Source zone: us-east-1a
Source node: i-07812b1c2b6ed6405

Service                        Reqs  Same-Zone  Cross-AZ  Errors  Avg(ms)  P99(ms)
--------------------------------------------------------------------------------
backend-baseline               5000     49.9%     50.1%       0     2.4      4.2
backend-prefer-same-zone       5000    100.0%      0.0%       0     1.8      3.1
backend-prefer-same-node       5000    100.0%      0.0%       0     1.6      2.5
backend-topology-aware         5000    100.0%      0.0%       0     1.5      2.5

--- Zone Distribution per Service ---
  backend-baseline: us-east-1a: 2497, us-east-1b: 827, us-east-1c: 1676
  backend-prefer-same-zone: us-east-1a: 5000
  backend-prefer-same-node: us-east-1a: 5000
  backend-topology-aware: us-east-1a: 5000

--- Cost Estimate (at 10 TB/month east-west traffic) ---
  backend-baseline: $100.12/month (5,006 GB cross-AZ)
  backend-prefer-same-zone: $0.00/month (0 GB cross-AZ)
  backend-prefer-same-node: $0.00/month (0 GB cross-AZ)
  backend-topology-aware: $0.00/month (0 GB cross-AZ)

  Best improvement: backend-prefer-same-zone
  Cross-AZ reduction: 100.0%
  Estimated monthly savings: $100.12
==============================================================================

The baseline came in at 50% cross-AZ, not the theoretical 67%. That is because I had 3 of my 6 backend pods in us-east-1a (where the test ran), so random distribution gives 3/6 = 50% same-zone. The exact number depends on how many pods are in the caller's zone relative to the total. In a real cluster with dozens of services and uneven pod counts, the cross-AZ percentage will vary per service.

The important result: all three optimized strategies hit 100% same-zone across 5,000 requests. Zero cross-AZ leakage. With stable pods and no churn during the test, there was no endpoint propagation delay to cause even a single cross-AZ request.

PreferSameNode edged out PreferSameZone on latency (1.6ms average versus 1.8ms) because it routes to pods on the same node, not just the same zone. The baseline's 2.4ms average includes the cross-AZ network hop penalty.

This is why I say "build a tool, don't guess." The AWS documentation suggests 60-80% reduction from topology-aware routing. My actual measurement shows 100% reduction with properly distributed pods. That gap is significant enough to change how you think about the ROI.

What Happens During Pod Churn?

Stable pods are one thing. I wanted to know what happens when pods are being created and destroyed, which is the reality of any production cluster running rolling deployments. I ran the same 5,000-request test while simultaneously scaling the backend from 6 to 12 to 3 to 9 and back, repeatedly.

=== RESULTS DURING POD CHURN ===

Service                        Reqs  Same-Zone  Cross-AZ  Errors  Avg(ms)  P99(ms)
--------------------------------------------------------------------------------
backend-baseline               5000     54.7%     42.3%     151     2.6      6.3
backend-prefer-same-zone       5000    100.0%      0.0%       0     1.7      3.5
backend-prefer-same-node       5000    100.0%      0.0%       0     1.7      5.4
backend-topology-aware         5000    100.0%      0.0%       0     1.8      5.3

All three optimized strategies held at 100% same-zone during churn. Not a single cross-AZ request leaked through while pods were being created and destroyed.

The baseline told a different story: 151 errors (3%) from requests hitting endpoints that were mid-termination. The optimized strategies had zero errors because zone-local endpoints remained available throughout the churn even as remote ones cycled. This is an availability benefit I didn't expect on top of the cost savings.

P99 latency did increase under churn. PreferSameNode went from 2.5ms to 5.4ms as the endpoint list updated. PreferSameZone stayed more stable at 3.5ms. That's expected and acceptable.

One thing I got wrong in my initial prediction: I expected Topology Aware Routing (TAR) to drop hints during the churn and leak cross-AZ traffic. It did not. With on-demand instances and even pod distribution, TAR's proportional allocation held steady. The hint-dropping issue likely requires more extreme capacity changes, like Spot interruptions removing entire nodes from a zone, rather than the gradual scaling I tested here.

The Routing Strategies: What Actually Works

I tested five approaches. Here is my honest ranking.

1. Traffic Distribution: PreferSameZone (Use This One)

This is the winner. One field on your Service spec:

apiVersion: v1
kind: Service
metadata:
  name: backend-prefer-same-zone
  namespace: cross-az-demo
spec:
  type: ClusterIP
  trafficDistribution: PreferSameZone  # stable in K8s 1.35, replaces deprecated PreferClose
  selector:
    app: backend
  ports:
    - port: 8080
      targetPort: 8080

That's it. One line. kube-proxy routes to same-zone endpoints first. If no local endpoints exist, it falls back to any available endpoint. PreferSameZone was introduced in 1.34 and graduated to stable in 1.35. No feature gates, no annotation. It is a first-class spec field.

I recommend this as the default for every ClusterIP Service in your cluster unless you have a specific reason not to use it. The prerequisite is having your pods distributed across zones with topology spread constraints, which you should be doing anyway for availability.

What about AZ failures? During an AZ outage, pods in the failed zone lose their local endpoints. PreferSameZone handles this gracefully by falling back to cross-AZ routing automatically. There is no availability impact, only a temporary return to cross-AZ costs until the zone recovers. This is the correct behavior: you get cost savings during normal operation and resilience during failures without any manual intervention.

2. Topology Aware Routing (It Works, But PreferSameZone is Better)

The older approach, using an annotation:

metadata:
  annotations:
    service.kubernetes.io/topology-mode: Auto

TAR uses a more complex algorithm: the EndpointSlice controller allocates endpoints proportionally across zones and sets "hints" for each one. The idea is to balance both locality and load distribution.

In my testing, it works. The numbers are close to PreferSameZone. But I found two problems that make me recommend PreferSameZone instead:

Problem 1: Spot instances break it. TAR recalculates proportional allocation based on zone capacity. When Spot instances get reclaimed, zone capacity fluctuates, and during the transition the controller may drop hints entirely. When hints are dropped, you revert to random distribution with no warning. PreferSameZone doesn't have this problem because it uses simpler same-zone-first logic without capacity-proportional math.

Problem 2: The annotation is deprecated. The service.kubernetes.io/topology-mode: Auto annotation won't graduate to GA. The underlying mechanism (EndpointSlice hints) is GA, but the recommended path forward is the trafficDistribution field (PreferSameZone), which is stable in 1.35. There's no reason to use the deprecated annotation when the replacement is available and simpler.

3. PreferSameNode: The Best of Both Worlds (New in 1.35)

Kubernetes 1.35 added a new Traffic Distribution option that solves a real gap:

spec:
  trafficDistribution: PreferSameNode

This routes to endpoints on the same node first, then falls back to same-zone, then cluster-wide. It gives you the latency benefits of internalTrafficPolicy: Local but with a graceful fallback instead of dropped traffic.

This is the option I wish had existed from the start. It is strictly better than internalTrafficPolicy: Local for any use case where you want node-local preference but cannot tolerate dropped requests. Stable since 1.35.

I recommend this for latency-sensitive service pairs that are often (but not always) co-located. For pure cost optimization, PreferSameZone is sufficient. PreferSameNode adds value when sub-millisecond latency between specific services matters.

4. Internal Traffic Policy: Local (The Nuclear Option)

spec:
  internalTrafficPolicy: Local

This restricts traffic to endpoints on the same node, not just the same zone. Zero cross-AZ traffic is possible. Zero cross-node traffic either.

I call this the nuclear option because if no endpoint exists on the calling pod's node, the traffic is dropped. Not rerouted or retried - just dropped. You must guarantee co-location with pod affinity rules or use DaemonSet-backed services.

I proved this during testing. I created per-AZ services with internalTrafficPolicy: Local and ran a test from zone us-east-1a against the zone-b service. The test hung indefinitely. Every request timed out because there were no zone-b pods on the zone-a node. There were no errors and no fallbacks - just a silent failure.

When it does work (when a backend pod exists on the same node), the latency is the best of any strategy: 1.6ms average in my testing, versus 1.8ms for PreferSameZone. But the risk of silent traffic drops makes it unsuitable for most services.

I use this for exactly two situations:

DaemonSet services (every node has a local endpoint by definition)
Tightly coupled pod pairs with strict requiredDuringSchedulingIgnoredDuringExecution affinity

For everything else, PreferSameZone is safer and almost as effective. And now with PreferSameNode in 1.35, there is even less reason to use Local. You get the same node-local preference with a graceful fallback.

5. Per-AZ Deployments (When PreferSameZone Is Not Enough)

There is a subtle scaling problem that PreferSameZone doesn't solve. Say traffic to your backend spikes in us-east-1a. The HPA scales the Deployment from 6 to 12 replicas. Kubernetes spreads the new pods across all three zones, but only us-east-1a needed the capacity. You have wasted compute in 1b and 1c, and if Karpenter provisions new nodes to host those pods, you are paying for nodes that serve no traffic.

The fix is per-AZ Deployments with independent HPAs:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend-az-a
spec:
  replicas: 2
  selector:
    matchLabels:
      app: backend-zonal
      zone: az-a
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: topology.kubernetes.io/zone
                    operator: In
                    values: ["us-east-1a"]
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: backend-az-a
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: backend-az-a
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Multiply by three AZs. You get triple the manifests and triple the HPA objects to monitor. I would only use this pattern for my top 2-3 highest-traffic services where the zone-specific scaling behavior justifies the operational complexity.

The full 3-AZ manifest is in k8s/cross-az-optimization/05-per-az-deployment.yaml.

Load Balancer: Switch to IP Mode

For services fronted by an ALB, switch the target type from "instance" to "ip":

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/scheme: internet-facing

In instance mode, traffic flows: ALB - NodePort (any node) - kube-proxy - Pod (maybe different AZ). That is up to two cross-AZ hops. In IP mode: ALB - Pod IP directly. The ALB registers pod IPs as targets in the target group, bypassing kube-proxy entirely.

Combine this with trafficDistribution: PreferSameZone on the backing Service for the best result.

What the AWS Documentation Doesn't Tell You

I ran into several things during testing that aren't obvious from the official docs:

Topology spread constraints need minDomains with Auto Mode. This one bit me during testing. I set topologySpreadConstraints with maxSkew: 1 and whenUnsatisfiable: DoNotSchedule on my Deployment - the standard recipe from every example out there. But with EKS Auto Mode, you start from zero nodes. Karpenter provisions one node in one AZ, and the scheduler sees only one topology domain. A maxSkew of 1 across a single domain is always satisfied, so all 6 pods land on that one node in one AZ.

The fix is minDomains: 3. This tells the scheduler that at least 3 zones must exist before scheduling is satisfied, which forces Karpenter to provision nodes in other AZs:

topologySpreadConstraints:
  - maxSkew: 1
    minDomains: 3
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: backend

Most topology spread examples assume you already have nodes in multiple AZs. With Auto Mode and Karpenter, you do not. Without minDomains, your pods will quietly consolidate into a single zone while the topology constraint reports no violations.

The EKS console has built-in cross-AZ visibility now. Enable the amazon-cloudwatch-observability addon (included in the Terraform code above) and you get a service map in the EKS console showing cross-AZ flows with data volume. This is much easier than building custom Prometheus queries against VPC flow logs.

VPC endpoint interface charges don't include cross-AZ data transfer. This is buried in a "What's New" announcement from 2022. Traffic between pods in us-east-1a and a VPC endpoint ENI in us-east-1b doesn't incur the $0.01/GB cross-AZ charge. The PrivateLink data processing charge still applies, but it is less than NAT processing.

PreferSameZone occasionally sends cross-AZ during endpoint updates. When pods are added or removed, there's a brief window where EndpointSlice hints haven't propagated. During this window, kube-proxy may route to any endpoint. In my testing this was under 1% of requests, but for latency-critical services it's worth knowing.

CoreDNS is a cross-AZ blind spot. DNS lookups via the kube-dns Service are themselves subject to cross-AZ routing, and as of this writing you can't set trafficDistribution on the CoreDNS addon service directly. There's an open EKS feature request for this. If DNS latency matters to you, this is a gap to watch.

Finding cross-AZ charges in Cost Explorer. Beyond the EKS console's network observability, you can identify cross-AZ charges in AWS Cost Explorer by filtering for the DataTransfer-Regional-Bytes and NatGateway-Bytes usage types. These are the line items that this article's optimizations target.

Production Considerations

This article focuses on FinOps, but if you are taking these patterns into production, a few security points are worth noting:

Cluster access: The demo uses enable_cluster_creator_admin_permissions = true for simplicity. For production, define scoped access entries with narrower policies (e.g., AmazonEKSViewPolicy for read-only users, AmazonEKSClusterAdminPolicy only for platform teams).
Workload identity: EKS Auto Mode with module v21+ uses Pod Identity by default. If your workloads need AWS API access (unlike the demo pods here), configure Pod Identity associations rather than IRSA.
VPC endpoint security groups: The demo allows all VPC CIDR traffic on port 443. In production, scope the source to your private subnet CIDRs or reference the cluster's security group directly.
Network ACLs: The default NACL allows all traffic. Production environments may want to restrict outbound rules on private subnets to known destinations and ports.
IMDSv2: Auto Mode uses Bottlerocket, which enforces IMDSv2. If you adapt this code for self-managed node groups with Amazon Linux, ensure IMDSv2 is enforced (http_tokens = "required" in your launch template) and set http_put_response_hop_limit = 2 for containerized workloads.
Encryption at rest: The demo doesn't configure a custom KMS key for EKS secrets encryption. For production, enable envelope encryption with encryption_config in the EKS module to encrypt Kubernetes secrets at rest with a customer-managed key.
Encryption in transit: Traffic between pods is unencrypted by default. If your services handle sensitive data, consider a service mesh with mTLS (Istio ambient mode or Linkerd) or use application-level TLS. Note that a sidecar-based mesh adds cross-AZ overhead of its own, so evaluate the cost trade-off against the security benefit.

What This Costs and What You Save

Here is my cost analysis for a cluster with 10 TB/month east-west traffic and 2 TB/month AWS service traffic (ECR, S3, CloudWatch):

Category	Before	After	Delta
East-west cross-AZ	$100/month	$0/month	-$100
NAT processing (ECR/S3/logs)	$90/month	$0/month	-$90
NAT cross-AZ (single NAT)	$13/month	$0/month	-$13
VPC interface endpoints (4x3 AZs)	$0	$86/month	+$86
Additional NAT Gateways (2 extra)	$0	$65/month	+$65
Net change	$236/month	$152/month	-$84/month

At 10 TB/month, the savings are modest at about $84/month. But cross-AZ costs scale linearly with traffic. At 50 TB/month east-west, the east-west savings alone hit $650/month. At 100 TB/month, over $1,300/month. The infrastructure costs (VPC endpoints, extra NATs) stay flat.

The real point is that the traffic routing changes (PreferSameZone on Services and IP mode on Ingress) are free. There is zero infrastructure cost and only one line of YAML per Service. Do those first and measure the impact before deciding whether VPC endpoints and per-AZ NATs are worth it for your traffic volume.

My Recommended Order of Operations

Add VPC endpoints for S3 (free), ECR, STS, and CloudWatch Logs. This is the biggest win for most teams. The S3 and DynamoDB gateway endpoints are free, and the ECR interface endpoint eliminates NAT processing charges ($0.045/GB) on every container image pull. If your cluster pulls images regularly, this alone can save more than all the other optimizations combined.
Add trafficDistribution: PreferSameZone to all ClusterIP Services. The highest-impact change for east-west traffic. It's a single field (stable in K8s 1.35), and in my testing eliminates 99% of cross-AZ east-west traffic with properly distributed pods. For latency-sensitive pairs, consider PreferSameNode (stable in 1.35).
Add topology spread constraints to all Deployments. PreferSameZone only works if pods are actually distributed across zones. If they aren't, this is your real first step.
Switch ALB Ingress resources to IP target mode. Another free change. One annotation eliminates the NodePort-to-pod cross-AZ hop.
Deploy one NAT Gateway per AZ. Worth it when your cross-AZ NAT traffic exceeds ~1.6 TB/month per AZ.
Consider per-AZ Deployments for your top 2-3 highest-traffic services. Only if you have zone-specific scaling problems that PreferSameZone doesn't solve.

Getting Started

The complete code is in the companion repository. Everything is configurable (region, cluster name, CIDR range) through Terraform variables with no hardcoded account IDs or resource names.

Prerequisites: AWS CLI configured, Terraform 1.10+, Docker, kubectl, Python 3.13+.

The fastest path is the one-command setup:

git clone https://github.com/darryl-ruggles/eks-cross-az-exploration
cd eks-cross-az-exploration

# Deploy everything and run the measurement (takes ~15 minutes)
make setup

That runs Terraform, builds and pushes container images to ECR, deploys the demo app with all service variants, and executes the traffic measurement job. At the end you get the comparison report.

If you prefer to step through it:

# 1. Deploy infrastructure (VPC, EKS, VPC endpoints, ECR repos)
make infra

# 2. Configure kubectl
make kubeconfig

# 3. Build and push container images
make images

# 4. Deploy the demo app and all routing strategies
make deploy

# 5. Run the cross-AZ measurement
make test

# 6. See pod distribution across AZs
make show-distribution

# 7. Run again with PreferSameNode included
make test-all

# 8. Clean up when done
make teardown

All configuration is overridable: make test NUM_REQUESTS=1000, make infra (uses variables.tf defaults; edit region, cluster_name, vpc_cidr as needed).

IMPORTANT: This deploys real AWS resources that cost money. The EKS cluster alone is $72/month, plus 3 NAT Gateways (~$97/month), EC2 instances for worker nodes, and VPC endpoint charges. Run make teardown when you're done testing. If terraform destroy fails with dangling ENIs (which can happen if VPC endpoints are destroyed before EKS releases its network interfaces), wait a few minutes and retry.

The measurement report gives you actual numbers for your cluster, which matters more than any estimate in a blog post.

Cross-AZ data transfer is one of those costs that seems small on any individual request but compounds into a meaningful line item at scale. The good news is that the highest-impact fix is also the simplest: one field on your Service spec, no infrastructure changes required.

Resources

Connect with me on X, Bluesky, LinkedIn, GitHub, Medium, Dev.to, or the AWS Community. Check out more of my projects at darryl-ruggles.cloud and join the Believe In Serverless community.

AWS Lambda Now Knows Where It Lives: Exploring the New AZ Metadata

Darryl Ruggles — Fri, 27 Mar 2026 01:30:21 +0000

Lambda was the last holdout. EC2 has had its Instance Metadata Service since the beginning. ECS and Fargate have their task metadata endpoints. But Lambda? If your function needed to know which Availability Zone it was running in, you were out of luck - unless you enjoyed building elaborate workarounds with private API Gateway endpoints and DescribeNetworkInterfaces calls.

On March 19, 2026, AWS quietly fixed this with a new metadata endpoint that exposes the AZ ID of the execution environment. No VPC is required, no IAM permissions are needed, and there are no API calls to external services. It's just a localhost HTTP endpoint that returns your AZ in under a millisecond.

I built a small demo app to explore three practical uses for this new metadata, using Powertools for AWS Lambda for observability throughout (I wrote about Powertools best practices previously). The code is on GitHub if you want to deploy it yourself.

What's in the Box

Lambda now sets two new reserved environment variables in every execution environment:

Variable	Description
`AWS_LAMBDA_METADATA_API`	Address of the metadata server (e.g., `169.254.100.1:9001`)
`AWS_LAMBDA_METADATA_TOKEN`	Auth token unique to this execution environment

You make a GET request to the endpoint with the token as a Bearer header, and you get back your AZ ID:

GET http://${AWS_LAMBDA_METADATA_API}/2026-01-15/metadata/execution-environment
Authorization: Bearer ${AWS_LAMBDA_METADATA_TOKEN}

{
  "AvailabilityZoneID": "use1-az1"
}

That's it. The response is cached for 12 hours and marked immutable - it never changes within an execution environment.

Powertools for AWS Lambda makes it even simpler:

from aws_lambda_powertools.utilities.metadata import get_lambda_metadata

metadata = get_lambda_metadata()
az_id = metadata.availability_zone_id  # "use1-az1"

Powertools handles the caching and SnapStart invalidation automatically.

AZ IDs vs AZ Names

One important detail: the metadata returns AZ IDs (use1-az1), not AZ names (us-east-1a). This matters because AZ IDs are consistent across all AWS accounts - use1-az1 is always the same physical data center regardless of which account you're in. AZ names are mapped differently per account to distribute load, so my us-east-1a might be a completely different physical zone than yours.

If you need the AZ name for your account, you can resolve it with a single EC2 API call:

ec2 = boto3.client("ec2")
resp = ec2.describe_availability_zones(
    Filters=[{"Name": "zone-id", "Values": ["use1-az1"]}]
)
az_name = resp["AvailabilityZones"][0]["ZoneName"]  # "us-east-1a"

Cache this - it does not change.

The Demo App

I built a single Lambda function with three API routes using Powertools Event Handler, deployed with SAM. Each route demonstrates a different use case for the new AZ metadata.

Service	Purpose
API Gateway	REST API with X-Ray tracing
Lambda (Python 3.13, ARM64)	Single function, 3 routes via Powertools Event Handler
Powertools Layer	Logger, Tracer, Metrics, Event Handler, Lambda Metadata
CloudWatch Dashboard	Invocations, cold starts, routing, and chaos metrics by AZ

The Foundation: AZ-Enriched Observability

Before getting to the three routes, the most important pattern is in the module initialization and handler. I resolve the AZ ID once during cold start and set it as a default metric dimension:

from aws_lambda_powertools import Logger, Metrics, Tracer
from aws_lambda_powertools.event_handler import APIGatewayRestResolver

logger = Logger()
tracer = Tracer()
metrics = Metrics()
app = APIGatewayRestResolver()

# Resolve AZ ID at module level - metadata endpoint is localhost, sub-millisecond
AZ_ID = _init_az_id()

# Every custom metric gets the AZ dimension automatically
if AZ_ID != "unknown":
    metrics.set_default_dimensions(AvailabilityZoneID=AZ_ID)

Then in the handler, following the Powertools three-decorator pattern, I inject the AZ ID into every log line and X-Ray trace:

@logger.inject_lambda_context(log_event=True)
@tracer.capture_lambda_handler
@metrics.log_metrics(capture_cold_start_metric=True)
def lambda_handler(event: dict, context: LambdaContext) -> dict:
    logger.append_keys(availability_zone_id=AZ_ID)
    tracer.put_annotation("availability_zone_id", AZ_ID)
    return app.resolve(event, context)

Now every log line from this function includes the AZ:

{
    "level": "INFO",
    "message": "Routing decision",
    "availability_zone_id": "use1-az4",
    "endpoint": "cache-az4.internal:6379",
    "same_az": true,
    "service": "lambda-metadata-explorer"
}

During an AZ incident, you can filter to the affected zone instantly in CloudWatch Logs Insights:

fields @timestamp, availability_zone_id, message, level
| filter availability_zone_id = "use1-az4"
| filter level = "ERROR"
| sort @timestamp desc

The tracer.put_annotation call adds the AZ ID as an X-Ray annotation on every trace. You can filter traces by AZ in the X-Ray console using:

annotation.availability_zone_id = "use1-az4"

This lets you isolate all traces from a specific AZ during an incident. Click any trace segment and check the Annotations tab to see the AZ ID.

This alone is worth the feature. Before March 19, you simply could not do this with Lambda.

Use Case 1: Exploring the Metadata (GET /metadata)

The first route is straightforward - it returns the AZ metadata from both the Powertools utility and the raw HTTP endpoint, plus the resolved AZ name. This is mostly educational, showing developers both access methods side by side:

@app.get("/metadata")
@tracer.capture_method
def explore_metadata():
    az_name = _resolve_az_name()  # EC2 API call, cached after first use
    metrics.add_metric(name="MetadataRequests", unit=MetricUnit.Count, value=1)

    return {
        "availability_zone": {
            "az_id": AZ_ID,
            "az_name": az_name,
        },
        "execution_context": {
            "function_name": os.environ.get("AWS_LAMBDA_FUNCTION_NAME", "local"),
            "region": os.environ.get("AWS_REGION", "unknown"),
            "memory_mb": os.environ.get("AWS_LAMBDA_FUNCTION_MEMORY_SIZE", "unknown"),
        },
    }

Seeing the Multi-AZ Distribution

There's a subtlety here. If you call the endpoint 20 times sequentially, you'll likely see the same AZ every time. That's because Lambda reuses warm execution environments, and a warm instance stays in the same AZ for its lifetime.

To actually see Lambda's multi-AZ distribution, you need to force it to spin up multiple execution environments simultaneously. The included test script fires 60 concurrent requests using xargs -P, which overwhelms a single warm instance and forces Lambda to create new environments across different AZs:

# 60 total requests, 20 concurrent at a time
./scripts/test_az_distribution.sh $API_URL 20 60

--- AZ Distribution ---

Count  AZ ID          AZ Name
-----  -------------- -------
22     use1-az4       us-east-1d
20     use1-az1       us-east-1a
18     use1-az6       us-east-1f

Unique AZs observed: 3
Total responses:     60

Lambda spreads execution environments across multiple AZs for resilience. Now you can actually see it happening, and with the AZ-enriched CloudWatch dashboard, you can monitor this distribution in real time.

This is also why the chaos engineering use case (Use Case 3) needs concurrent traffic. If you only have one warm instance, targeting a specific AZ is hit-or-miss. With concurrent load, you'll have instances spread across AZs, and the ones in the target AZ will return 503 while the others continue serving normally - exactly how a real AZ outage would behave.

Use Case 2: Same-AZ Routing (POST /route)

This is the use case with the most immediate dollar impact. Cross-AZ data transfer costs $0.01/GB in each direction. If you have a Lambda function making thousands of calls per second to an ElastiCache cluster or RDS read replica, and it's routing to whatever endpoint it gets, you're paying for cross-AZ traffic on roughly two-thirds of those calls (assuming 3 AZs).

With the AZ metadata, you can route to the same-AZ endpoint:

@app.post("/route")
@tracer.capture_method
def route_request():
    body = app.current_event.json_body or {}

    # In production, load from SSM Parameter Store or service discovery
    endpoint_map = json.loads(os.environ.get("AZ_ENDPOINT_MAP", "{}")) or {
        "use1-az1": "cache-az1.internal:6379",
        "use1-az2": "cache-az2.internal:6379",
        "use1-az4": "cache-az4.internal:6379",
        "use1-az6": "cache-az6.internal:6379",
    }
    fallback = os.environ.get("FALLBACK_ENDPOINT", "cache-primary.internal:6379")

    selected = endpoint_map.get(AZ_ID, fallback)
    same_az = AZ_ID in endpoint_map

    metrics.add_metric(name="RoutingDecisions", unit=MetricUnit.Count, value=1)
    tracer.put_annotation("routing_type", "same-az" if same_az else "cross-az")

    return {
        "az_id": AZ_ID,
        "routing": {"selected_endpoint": selected, "same_az": same_az},
        "cost_insight": {
            "cross_az_cost_per_gb": "$0.01 per direction ($0.02 round-trip)",
            "same_az_cost_per_gb": "$0.00",
        },
    }

The endpoint map is a simple AZ-ID-to-endpoint mapping. In production, you'd load this from SSM Parameter Store using the Powertools Parameters utility with caching:

from aws_lambda_powertools.utilities import parameters

endpoint_map = parameters.get_parameter("/myapp/az-endpoint-map", transform="json")

For an ElastiCache cluster, you'd use the node endpoints for each AZ. For RDS with read replicas, you'd map to the replica in each AZ. The key insight is that you now know which AZ you're in and can make an informed routing decision.

Cost math: At 1 TB/month of data transfer with 3 AZs, roughly 67% crosses AZ boundaries. AWS charges $0.01/GB in each direction, so cross-AZ round-trips cost $0.02/GB effectively. That's ~$13.40/month in cross-AZ charges for this example. With same-AZ routing, it drops to near zero. The savings scale linearly with traffic volume.

Use Case 3: AZ-Aware Chaos Engineering (POST /chaos)

The third route is about resilience testing. You specify a target AZ, and if the function happens to be running there, it injects a failure. This lets you test how your upstream systems - load balancers, retry logic, circuit breakers - handle an AZ going unhealthy.

@app.post("/chaos")
@tracer.capture_method
def chaos_test():
    body = app.current_event.json_body or {}
    target_az = body.get("target_az", "")
    failure_mode = body.get("failure_mode", "error")

    if target_az and AZ_ID == target_az:
        metrics.add_metric(name="ChaosFailuresInjected", unit=MetricUnit.Count, value=1)
        logger.warning("Chaos: injecting failure", extra={
            "target_az": target_az,
            "failure_mode": failure_mode,
        })

        if failure_mode == "latency":
            time.sleep(min(body.get("delay_seconds", 3), 10))
            return {"az_id": AZ_ID, "chaos": {"triggered": True, "mode": "latency"}}

        raise ServiceError(503, f"Simulated AZ failure in {AZ_ID}")

    return {"az_id": AZ_ID, "chaos": {"triggered": False, "target_az": target_az}}

Two failure modes:

error - Returns HTTP 503. Tests whether callers retry and eventually hit a healthy AZ.
latency - Injects a delay (capped at 10 seconds). Tests whether callers have appropriate timeouts and circuit breakers.

In a real project, you'd combine this with an ALB or API Gateway with health checks. Target a specific AZ and observe whether traffic shifts to healthy AZs. The CloudWatch dashboard shows failures concentrated in the target AZ while other AZs continue serving normally - exactly what you'd see during a real AZ impairment.

One important detail for testing: make test-chaos auto-detects your current warm instance's AZ and targets it, guaranteeing a triggered failure. But for a more realistic simulation, fire concurrent traffic while targeting a specific AZ - some requests will fail (those in the target AZ) while others succeed (those in healthy AZs).

Before this feature, you'd need to deploy different code per AZ or use external feature flags combined with some mechanism to map invocations to AZs. Now it's a simple string comparison.

The SAM Template

The deployment is a straightforward SAM template - one function, one API, one CloudWatch dashboard:

Globals:
  Function:
    Runtime: python3.13
    Architectures:
      - arm64
    Timeout: 30
    MemorySize: 256
    Tracing: Active
    LoggingConfig:
      LogFormat: JSON

Resources:
  AZMetadataFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Sub lambda-metadata-explorer-${Environment}
      CodeUri: src/
      Handler: app.lambda_handler
      Layers:
        - !Sub arn:aws:lambda:${AWS::Region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python313-arm64:${PowertoolsLayerVersion}
      Environment:
        Variables:
          POWERTOOLS_SERVICE_NAME: lambda-metadata-explorer
          POWERTOOLS_METRICS_NAMESPACE: AZMetadataExplorer
          POWERTOOLS_LOG_LEVEL: !Ref LogLevel
          POWERTOOLS_TRACER_CAPTURE_RESPONSE: 'true'
          POWERTOOLS_TRACER_CAPTURE_ERROR: 'true'
      Policies:
        - Statement:
            - Effect: Allow
              Action:
                - ec2:DescribeAvailabilityZones
              Resource: '*'
      Events:
        GetMetadata:
          Type: Api
          Properties:
            Path: /metadata
            Method: get
        PostRoute:
          Type: Api
          Properties:
            Path: /route
            Method: post
        PostChaos:
          Type: Api
          Properties:
            Path: /chaos
            Method: post

ARM64 (Graviton) for better price-performance. Powertools via the AWS-managed Lambda Layer (plus pip bundling from requirements.txt to ensure v3.26.0+ with the metadata utility is available even if the Layer hasn't been updated yet). X-Ray tracing active. JSON log format. The only extra IAM permission is ec2:DescribeAvailabilityZones for the AZ name resolution - the metadata endpoint itself requires no permissions.

The template also creates a CloudWatch dashboard with AZ distribution widgets. The AZ dimension comes from metrics.set_default_dimensions(AvailabilityZoneID=AZ_ID) which tags every custom metric with the availability zone. The dashboard references explicit metric definitions for each AZ:

["AZMetadataExplorer", "MetadataRequests", "AvailabilityZoneID", "use1-az1", "service", "lambda-metadata-explorer"]

One gotcha: the ColdStart metric uses Powertools' built-in dimensions (function_name + service) rather than our custom AZ dimension, because Powertools captures it before the handler runs. So the Cold Starts widget shows a total count rather than a per-AZ breakdown.

Deploy and Try It

Prerequisites: AWS CLI, SAM CLI, Python 3.13+, an AWS account.

git clone https://github.com/RDarrylR/lambda-az-metadata.git
cd lambda-az-metadata

# Build and deploy
sam build --profile your-profile --region us-east-1
sam deploy --guided --profile your-profile --region us-east-1

# Or use the Makefile
make deploy PROFILE=your-profile

After deploying, grab the API URL from the stack outputs and explore:

# See your AZ metadata
curl -s $API_URL/metadata | python3 -m json.tool

# Test same-AZ routing
curl -s -X POST $API_URL/route \
  -H "Content-Type: application/json" \
  -d '{"message": "hello"}' | python3 -m json.tool

# Test chaos engineering (replace use1-az1 with an AZ from your /metadata response)
curl -s -X POST $API_URL/chaos \
  -H "Content-Type: application/json" \
  -d '{"target_az": "use1-az1", "failure_mode": "error"}' | python3 -m json.tool

# See the AZ distribution across 60 concurrent invocations
make test-distribution

Cleanup:

sam delete --stack-name lambda-az-metadata --profile your-profile --region us-east-1

What It Costs

This demo runs well within the AWS free tier. Lambda gives you 1M requests/month and 400K GB-seconds. API Gateway gives you 1M calls for the first 12 months. The metadata endpoint itself is free - no additional charges.

Gotchas

A few things to watch for:

Region-specific AZ IDs. The demo code and dashboard use us-east-1 AZ IDs (use1-az1 through use1-az6). If you deploy to a different region, update the default endpoint map in app.py and the dashboard metrics in template.yaml with your region's AZ IDs (e.g., usw2-az1 for us-west-2).
Caching is your friend. The metadata response is immutable within an execution environment. Resolve it once at module level and forget about it. There is no reason to call the endpoint on every invocation.
VPC doesn't matter. The metadata endpoint is localhost inside the execution environment - it works identically with or without a VPC. No VPC endpoints, NAT gateways, or security group rules needed.
SnapStart changes things. If you use SnapStart, the execution environment might move to a different AZ after restore. The metadata endpoint cooperates with SnapStart by reducing the cache TTL during initialization, so Powertools can detect the need to re-fetch after restore and handles this automatically. But this demo resolves AZ_ID at module level during init - that code runs during the snapshot phase. After restore, the cached module-level variable could be wrong. For SnapStart, move the AZ resolution into the handler or use a @register_checkpoint_restore hook to re-resolve after restore.
The response may grow. The documentation says additional fields may be added in future updates. Don't fail on unknown fields.
log_event=True and CAPTURE_RESPONSE in production. I use both here for debugging visibility, but in production, disable them if your payloads contain sensitive data. log_event=True logs full request bodies to CloudWatch, and POWERTOOLS_TRACER_CAPTURE_RESPONSE stores full response bodies as X-Ray metadata. Control both via environment variables without code changes.
Layer version. The get_lambda_metadata() utility requires Powertools v3.26.0+ (Lambda Layer version 30+, released March 20, 2026). The SAM template defaults to version 30. Check the layer version history for the latest.
Import path. As of v3.26.0, the working import path is from aws_lambda_powertools.utilities.metadata import get_lambda_metadata. The Lambda documentation currently shows utilities.lambda_metadata, which may be corrected in a future docs update. Verify against your installed version if in doubt.

What's Next

The metadata endpoint currently returns only AvailabilityZoneID, but the versioned API path (/2026-01-15/) and the documentation's note about future fields suggest more is coming. Execution environment ID? Host-level metadata? We'll see.

For now, the AZ ID alone unlocks meaningful improvements in observability, cost optimization, and resilience testing. It's one of those features that's so obviously useful, you wonder why it took this long.

The code is on GitHub. Clone it, deploy it, and check your CloudWatch dashboard to see which AZs your Lambda functions are actually running in.

Resources

Connect with me on X, Bluesky, LinkedIn, GitHub, Medium, Dev.to, or the AWS Community. Check out more of my projects at darryl-ruggles.cloud and join the Believe In Serverless community.

A Complete Terraform Setup for EKS Auto Mode: Is it right for you?

Darryl Ruggles — Tue, 24 Mar 2026 16:41:12 +0000

One of the things I appreciate about AWS is the range of container options available. I recently wrote about why ECS is my default choice for containers on AWS - it's simple, powerful, and handles most workloads well. But when you need Kubernetes specifically - for portability, ecosystem tooling, or team expertise - EKS Auto Mode is an interesting approach worth exploring.

Instead of managing node groups, installing Karpenter, configuring the VPC CNI plugin, deploying the AWS Load Balancer Controller, setting up the EBS CSI driver, and keeping all of those components updated and compatible with each other - you enable a single flag and AWS handles all of it. The data plane becomes a managed service. The limitations that Auto Mode has are not insignificant but depending on the use case it may make sense for many teams to consider using Auto Mode.

I built a complete production-ready EKS Auto Mode cluster with Terraform to see how it works in practice. All the code is on GitHub: github.com/RDarrylR/eks-auto-mode-terraform.

This post walks through everything: what Auto Mode actually manages, how the Terraform setup works, deploying a real application, autoscaling from pods to nodes, observability with Container Insights, the security posture you get out of the box, custom NodePools for cost optimization, and the honest trade-offs you should know about before adopting it.

Why EKS Auto Mode?

If you've ever managed an EKS cluster, you know the operational burden. The control plane is managed by AWS, but everything else is on you:

Node management - Choosing instance types, managing AMIs, patching OS, handling node drains and replacements
Networking - Installing and configuring the VPC CNI plugin, managing pod IP allocation, setting up network policies
Storage - Deploying the EBS CSI driver, managing StorageClasses, handling volume lifecycle
Load balancing - Installing the AWS Load Balancer Controller, configuring IngressClasses, managing target group bindings
Scaling - Deploying Karpenter or Cluster Autoscaler, configuring NodePools, tuning consolidation policies
Add-on compatibility - Making sure all these components work together after every Kubernetes version upgrade

Multiply this across multiple clusters and environments, and you are spending more time on infrastructure plumbing than on the applications running on top of it.

EKS Auto Mode eliminates all of this. AWS takes over the entire data plane - compute, networking, storage, load balancing, and health monitoring. The only pods running in your cluster are the Kubernetes Metrics Server. Everything else runs off-cluster as managed services.

What AWS Manages For You

When you enable Auto Mode, AWS takes responsibility for six major areas:

Compute

Auto Mode uses Karpenter under the hood, running it as a managed service outside your cluster. You never see Karpenter pods, never upgrade them, never troubleshoot their OOM kills. Karpenter is a significant upgrade over the standard Cluster Autoscaler in my experience. With Auto Mode, AWS handles:

Node provisioning, scaling, and termination
AMI selection - custom Bottlerocket variants with SELinux enforcing and read-only root filesystem
OS patching via a continuous deployment pipeline with CVE scanning and staged rollout
14-day default node expiration (configurable up to 21 days) with automatic replacement
Spot Instance interruption handling
GPU support with drivers for NVIDIA, Neuron (Trainium/Inferentia), and EFA

The nodes are EC2 managed instances - a new EC2 primitive where operational control is delegated to AWS. They appear in your account, use your Reserved Instances and Savings Plans, but you cannot SSH into them or modify their configuration directly.

Networking

The VPC CNI runs as a managed component:

Pod networking with IPv4 and IPv6 support
Network Policy enforcement
Node-local DNS - CoreDNS runs as a system service directly on each node rather than as a traditional cluster deployment. This eliminates cross-node DNS latency and removes CoreDNS pods from your workload scheduling
Pod Identity Agent (built-in, no add-on installation)

Storage

The EBS CSI driver is managed with provisioner ebs.csi.eks.amazonaws.com:

Persistent volume provisioning with configurable IOPS, throughput, and encryption
Ephemeral storage configuration
Automatic NVMe setup for supported instance types

Load Balancing

The AWS Load Balancer Controller runs as a managed service:

ALB and NLB provisioning from Kubernetes Ingress/Service resources
Uses IngressClass alb (controller: eks.amazonaws.com/alb) and loadBalancerClass eks.amazonaws.com/nlb
Supports Ingress API and Service annotations only
Gateway API is not supported. The managed ALB controller only supports the Ingress API and Service annotations. This could be a deal breaker for teams adopting the Gateway API.

Health Monitoring

Automatic detection and repair of unhealthy nodes
Replaces unresponsive kubelets and PID-exhausted nodes
Reports through Kubernetes events and conditions

The Terraform Setup

I used the terraform-aws-modules/eks/aws community module (v21.x) with the AWS provider ~> 6.0. There is no separate official AWS Terraform module for EKS Auto Mode - the community module maintained by Anton Babenko is the de facto standard for EKS on Terraform, and it has first-class Auto Mode support. When you set compute_config.enabled = true, the module automatically handles the storage and load balancing configuration, hardcodes the right bootstrap settings, creates least-privilege IAM roles, and attaches the five managed policies the cluster role needs. You could build this with raw aws_eks_cluster resources, but the module eliminates several hundred lines of boilerplate and is referenced in AWS's own documentation.

The complete setup creates a production-ready cluster with VPC, Auto Mode enabled, custom NodePools, Container Insights observability, and least-privilege IAM configuration. Here is what the full infrastructure looks like:

Provider Configuration

terraform {
  required_version = ">= 1.10"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.38"
    }
    kubectl = {
      source  = "alekc/kubectl"
      version = "~> 2.1"
    }
  }
}

provider "aws" {
  region = var.aws_region

  default_tags {
    tags = {
      Project     = var.project_name
      Environment = var.environment
      ManagedBy   = "terraform"
    }
  }
}

A note on the kubectl provider: I use the alekc/kubectl fork rather than gavinbunney/kubectl. The alekc fork is actively maintained and handles CRD resources (like Karpenter NodePools and NodeClasses) reliably.

VPC

The VPC uses the community VPC module (v6.x) with both private and public subnets. The key detail is the subnet tags - EKS Auto Mode uses these to discover where to place nodes and load balancers:

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 6.0"

  name = "${var.project_name}-vpc"
  cidr = "10.0.0.0/16"

  azs             = slice(data.aws_availability_zones.available.names, 0, 3)
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = 1
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = 1
  }
}

A few things worth noting:

Single NAT gateway for cost optimization in non-production. Use one per AZ for production HA.
No VPC endpoints in this demo for simplicity. Production setups should add VPC endpoints for ECR (both ecr.api and ecr.dkr), S3, STS, and CloudWatch to reduce NAT gateway costs and keep traffic private. This is especially impactful at scale - NAT gateway data processing charges add up quickly.
Public subnet tags with kubernetes.io/role/elb tell the managed load balancer controller where to create internet-facing ALBs/NLBs.
Private subnet tags with kubernetes.io/role/internal-elb are for internal load balancers.
Nodes always run in private subnets. The load balancers sit in public subnets.

EKS Cluster with Auto Mode

This is where Auto Mode gets enabled:

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"

  name               = var.project_name
  kubernetes_version = "1.35"

  endpoint_public_access = true
  # UNCOMMENT for production: restrict to your IP ranges or use private-only access
  # cluster_endpoint_public_access_cidrs = ["YOUR_CIDR/32"]

  # Enable Auto Mode - this single block enables:
  #   - Managed Karpenter (compute)
  #   - Managed EBS CSI driver (storage_config derived from compute_config.enabled)
  #   - Managed ALB/NLB Controller (elastic_load_balancing derived from compute_config.enabled)
  #   - Managed VPC CNI, kube-proxy, CoreDNS (bootstrap_self_managed_addons hardcoded false)
  compute_config = {
    enabled    = true
    node_pools = ["general-purpose", "system"]
  }

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  # Auto Mode IAM resources - creates the node IAM role with required policies
  create_node_iam_role           = true
  create_auto_mode_iam_resources = true

  # Cluster access - grants the deploying identity cluster admin
  enable_cluster_creator_admin_permissions = true

  # Addons - metrics-server for HPA + Container Insights for observability
  addons = {
    metrics-server = {
      most_recent = true
    }
    amazon-cloudwatch-observability = {
      most_recent = true
      pod_identity_association = [{
        role_arn        = aws_iam_role.cloudwatch.arn
        service_account = "cloudwatch-agent"
      }]
    }
  }
}

Production notes: This demo uses a public API endpoint without CIDR restrictions and local Terraform state. For production, restrict cluster_endpoint_public_access_cidrs to your corporate IP ranges (or use private-only access with a VPN/bastion), and configure a remote backend (S3 + DynamoDB) for state locking and team collaboration.

The critical parameters:

compute_config.enabled = true - This is the flag that enables Auto Mode. The module automatically derives storage_config (block storage) and kubernetes_network_config (elastic load balancing) from this single flag - you do not need to set them separately.
compute_config.node_pools - The built-in pools to activate. general-purpose handles your application workloads (On-Demand, C/M/R families, gen 4+, AMD and ARM). system handles EKS internal add-ons with a CriticalAddonsOnly taint.
bootstrap_self_managed_addons - The module hardcodes this to false when Auto Mode is enabled, so you do not need to set it. Auto Mode manages VPC CNI, kube-proxy, and CoreDNS off-cluster.
create_auto_mode_iam_resources = true - Creates the node IAM role with AmazonEKSWorkerNodeMinimalPolicy and AmazonEC2ContainerRegistryPullOnly - the least-privilege policies AWS recommends for Auto Mode (not the broader AmazonEKSWorkerNodePolicy or AmazonEC2ContainerRegistryReadOnly).
create_node_iam_role = true - Must be true (default) so the module creates the Auto Mode node role with nodeRoleArn. Setting this to false without providing a custom node_role_arn causes the EKS API to reject the cluster.
addons - Deploys the metrics-server (required for HPA to read CPU/memory metrics) and CloudWatch Container Insights with Pod Identity for observability (more on this below).

The module also automatically attaches five managed policies to the cluster IAM role: AmazonEKSClusterPolicy, AmazonEKSComputePolicy, AmazonEKSBlockStoragePolicy, AmazonEKSLoadBalancingPolicy, and AmazonEKSNetworkingPolicy. Plus a custom tags policy for Auto Mode resource tagging.

Important Terraform caveat: When disabling Auto Mode later, you must first apply with compute_config.enabled = false before removing the block entirely. Removing it in one step causes issues.

IAM Architecture

The setup follows AWS recommended least-privilege principles with three distinct roles:

Role	Policies	Purpose
Cluster Role	AmazonEKSClusterPolicy, AmazonEKSComputePolicy, AmazonEKSBlockStoragePolicy, AmazonEKSLoadBalancingPolicy, AmazonEKSNetworkingPolicy	Cluster control plane + Auto Mode capabilities
Node Role	AmazonEKSWorkerNodeMinimalPolicy, AmazonEC2ContainerRegistryPullOnly	Minimal node permissions (not the broader WorkerNodePolicy)
CloudWatch Role	CloudWatchAgentServerPolicy, AWSXrayWriteOnlyAccess	Container Insights via Pod Identity

The node role uses AmazonEKSWorkerNodeMinimalPolicy instead of the broader AmazonEKSWorkerNodePolicy, and AmazonEC2ContainerRegistryPullOnly instead of AmazonEC2ContainerRegistryReadOnly. These are the least-privilege policies AWS specifically recommends for Auto Mode.

Custom NodePool for Spot Instances

The built-in general-purpose pool uses On-Demand instances only, with C, M, and R instance families (generation 4+, both AMD and ARM/Graviton). For cost optimization, you can create a custom NodePool that uses Spot:

resource "kubectl_manifest" "spot_nodeclass" {
  yaml_body = yamlencode({
    apiVersion = "eks.amazonaws.com/v1"
    kind       = "NodeClass"
    metadata = {
      name = "spot-class"
    }
    spec = {
      role = module.eks.node_iam_role_name
      subnetSelectorTerms = [
        {
          tags = {
            "kubernetes.io/role/internal-elb" = "1"
          }
        }
      ]
      securityGroupSelectorTerms = [
        {
          tags = {
            "aws:eks:cluster-name" = var.project_name
          }
        }
      ]
    }
  })

  depends_on = [module.eks]
}

resource "kubectl_manifest" "spot_nodepool" {
  yaml_body = yamlencode({
    apiVersion = "karpenter.sh/v1"
    kind       = "NodePool"
    metadata = {
      name = "spot-compute"
    }
    spec = {
      template = {
        metadata = {
          labels = {
            "workload-type" = "spot-eligible"
          }
        }
        spec = {
          nodeClassRef = {
            group = "eks.amazonaws.com"
            kind  = "NodeClass"
            name  = "spot-class"
          }
          requirements = [
            {
              key      = "karpenter.sh/capacity-type"
              operator = "In"
              values   = ["spot"]
            },
            {
              key      = "eks.amazonaws.com/instance-category"
              operator = "In"
              values   = ["c", "m", "r"]
            },
            {
              key      = "eks.amazonaws.com/instance-generation"
              operator = "Gte"
              values   = ["5"]
            }
          ]
          expireAfter = "336h"
        }
      }
      disruption = {
        consolidationPolicy = "WhenEmptyOrUnderutilized"
        consolidateAfter    = "1m"
        budgets = [
          { nodes = "20%" }
        ]
      }
      weight = 80
    }
  })

  depends_on = [
    module.eks,
    kubectl_manifest.spot_nodeclass,
  ]
}

Key differences from self-managed Karpenter:

NodeClass uses eks.amazonaws.com/v1 instead of karpenter.k8s.aws/v1
NodeClass requires role - You must specify role (IAM role name) or instanceProfile. The EKS API rejects NodeClasses without one of these. Use module.eks.node_iam_role_name from the Terraform module
Labels use eks.amazonaws.com/* instead of karpenter.k8s.aws/*
NodePool API is the same karpenter.sh/v1 in both cases
consolidateAfter is required in the disruption block - omitting it causes a validation error
The weight field (80 for Spot vs the default pool) determines scheduling preference
Instance families (C/M/R) and generation (5+) are specified via requirements - Karpenter selects the specific instance size based on pod resource requests at scheduling time

To direct workloads to Spot nodes, add a nodeSelector:

spec:
  nodeSelector:
    workload-type: spot-eligible

Deploying a Real Application

To demonstrate the full workflow, I deployed a Python FastAPI application with an ALB, autoscaling, and a load generator for scaling demos.

Application Code

A demo API with four endpoints designed to showcase Auto Mode's capabilities:

/health - Health check for probes and ALB health checks
/info - Returns pod and node details including the instance type Karpenter selected, whether the node is Spot or On-Demand, which NodePool it belongs to, and the AZ. Uses the Kubernetes API with a read-only RBAC binding to query node labels
/compute/{iterations} - CPU-intensive endpoint for triggering HPA scaling
/stress/{mb}?seconds=30 - Allocates memory to demonstrate memory pressure. Holds the allocation then auto-releases

The most interesting part is the /info endpoint, which queries the Kubernetes API to read node labels set by Karpenter:

def _get_node_info():
    """Fetch node labels via Kubernetes API to show Auto Mode scheduling decisions."""
    from kubernetes import client, config
    config.load_incluster_config()
    v1 = client.CoreV1Api()

    node = v1.read_node(os.getenv("NODE_NAME"))
    labels = node.metadata.labels or {}

    return {
        "node_name": os.getenv("NODE_NAME"),
        "instance_type": labels.get("node.kubernetes.io/instance-type", "unknown"),
        "capacity_type": labels.get("karpenter.sh/capacity-type", "unknown"),
        "nodepool": labels.get("karpenter.sh/nodepool", "unknown"),
        "zone": labels.get("topology.kubernetes.io/zone", "unknown"),
        "arch": labels.get("kubernetes.io/arch", "unknown"),
        "compute_type": labels.get("eks.amazonaws.com/compute-type", "unknown"),
    }

The other endpoints: /compute/{iterations} does CPU-intensive work for HPA scaling demos, and /stress/{mb} allocates memory (auto-releases after a configurable hold period) for memory pressure demos. Full app code on GitHub.

Here is a real response from the running demo:

{
  "pod": {
    "name": "demo-api-847f8d6ff5-9gvsq",
    "namespace": "demo",
    "ip": "10.0.3.184",
    "service_account": "demo-api"
  },
  "node": {
    "node_name": "i-01e881b330651bbc6",
    "instance_type": "c7g.large",
    "capacity_type": "spot",
    "nodepool": "spot-compute",
    "zone": "us-east-1c",
    "arch": "arm64",
    "compute_type": "auto"
  }
}

Notice Karpenter selected a c7g.large - a Graviton (ARM) Spot instance. This is a key insight: Karpenter optimizes for cost, and Graviton Spot instances are often the cheapest available option. The built-in general-purpose pool includes both AMD and ARM architectures, and the custom Spot pool does too. Hit the endpoint multiple times and you may see responses from different pods on different nodes, potentially in different AZs.

Building the Container Image

The Dockerfile is a standard Python 3.13-slim image running uvicorn.

Important: Build for the right architecture. Auto Mode's NodePools include ARM (Graviton) instances by default - Karpenter often selects them because they are the cheapest option. You need to target the right platform:

# Build for ARM (Graviton nodes - most common with Auto Mode Spot)
docker buildx build --platform linux/arm64 -t my-app:latest --load .

# Or if your nodes are AMD64
docker buildx build --platform linux/amd64 -t my-app:latest --load .

Using :latest tags with Auto Mode requires imagePullPolicy: Always in the deployment to ensure nodes pull the newest image on every pod creation.

Kubernetes Manifests

The key manifest design decisions (full manifests on GitHub):

RBAC for /info - The pod needs a ServiceAccount with a ClusterRole to read node labels. This is a minimal read-only get on nodes:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get"]

Downward API env vars - The deployment injects NODE_NAME, POD_NAME, POD_NAMESPACE, POD_IP, and POD_SERVICE_ACCOUNT via the Kubernetes downward API. The /info endpoint uses NODE_NAME to query the Kubernetes API for that node's labels:

env:
  - name: NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName

Other important details:

imagePullPolicy: Always is required when using :latest tags, since nodes cache images
IngressClass is alb (not eks.amazonaws.com/alb, since / is invalid in RFC 1123 subdomain names). You must explicitly create the IngressClassParams and IngressClass resources - Auto Mode does not auto-register them
Resource requests are set on every container. This is critical with Auto Mode - Karpenter provisions nodes based on requested resources. If you don't set requests, Auto Mode has no signal for right-sizing
Health probes are defined for proper rolling updates and load balancer health checks

Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: demo-api
  namespace: demo
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: demo-api
  minReplicas: 2
  maxReplicas: 10
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 120
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 70

The HPA scales on both CPU and memory - whichever metric exceeds its threshold first triggers a scale-up. The /compute endpoint drives CPU scaling, while the /stress endpoint drives memory scaling. Both trigger the same pod-to-node scaling chain through Auto Mode.

The stabilizationWindowSeconds: 120 shortens the default 5-minute stabilization window for scale-down, allowing the HPA to reduce replicas faster after load drops - useful for demos and workloads where you want quicker consolidation.

Important: The HPA requires the Kubernetes Metrics Server to read CPU/memory utilization. The metrics-server is deployed as an EKS addon in the Terraform configuration (see the EKS cluster configuration above). Without it, HPA reports cpu: <unknown> and never scales.

PodDisruptionBudget

Since Auto Mode consolidation is enabled by default, a PDB protects your application during node replacement:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: demo-api
  namespace: demo
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: demo-api

This ensures at least one pod is always available when Auto Mode consolidates or rotates nodes. Without a PDB, Karpenter could drain all pods from a node simultaneously during consolidation.

When the HPA scales pods up and existing nodes can't fit them, Auto Mode's managed Karpenter automatically provisions new nodes. When pods scale down and nodes become underutilized, consolidation kicks in and nodes are terminated. You never interact with the scaling infrastructure directly.

Autoscaling in Action

One of the most compelling features of Auto Mode is how seamlessly pod-level and node-level scaling work together. The repo includes everything needed to demonstrate this end-to-end.

Manual Scaling Demo

The quickest way to see Auto Mode's node provisioning is to scale the deployment beyond what existing nodes can handle:

# Scale to 15 replicas - watch Auto Mode provision new nodes
kubectl scale deployment demo-api -n demo --replicas=15

# Watch the scaling in real time
kubectl get pods -n demo -o wide -w    # Pods scheduling across nodes
kubectl get nodes -w                    # New nodes appearing
kubectl get events -A --sort-by='.lastTimestamp' | grep -E 'Nominated|Launched'

# Scale back down - watch consolidation
kubectl scale deployment demo-api -n demo --replicas=2

When you scale to 15 replicas, each requesting 256m CPU and 512Mi memory, the existing nodes cannot fit all the pods. Auto Mode's managed Karpenter evaluates the pending pods, selects optimal instance types from the C/M/R families (generation 4+, including Graviton), and provisions new nodes. The instance type is determined by the aggregate resource requests and cost optimization - Karpenter might select a c7g.large (Graviton) over an m7i.large if it is cheaper and fits the workload. You can verify this by hitting the /info endpoint to see exactly what Karpenter chose.

When you scale back down, pods are removed and nodes become underutilized. Auto Mode's consolidation policy (WhenEmptyOrUnderutilized) kicks in, draining and terminating nodes that are no longer needed.

Load-Driven Scaling Demo

For a more realistic demonstration, the repo includes a load generator - a busybox pod that continuously hits /compute/500000 to drive CPU utilization above the HPA threshold.

The full scaling chain works like this:

Load generator drives traffic to the API
CPU utilization exceeds the HPA's 60% target
HPA increases the replica count
New pods enter Pending state (no room on existing nodes)
Auto Mode's Karpenter detects pending pods and provisions new nodes
Pods schedule onto the new nodes and begin serving traffic

# Start the load generator
kubectl apply -f k8s/load-generator.yaml

# Watch the full chain: HPA -> pods -> nodes
kubectl get hpa -n demo -w
kubectl get pods -n demo -o wide -w
kubectl get nodes -w

# Stop the load
kubectl delete -f k8s/load-generator.yaml
# HPA scales pods down, Auto Mode consolidates nodes

The Makefile includes convenience commands for all of this: make demo-scale-up, make demo-scale-down, make demo-load-start, make demo-load-stop, and make demo-watch.

Observability with Container Insights

Since February 2026, Auto Mode supports Managed Capability Logging - you can deliver logs from compute autoscaling (Karpenter), block storage, load balancing, and pod networking to CloudWatch Logs, S3, or Kinesis Data Firehose using CloudWatch Vended Logs. This closed what was previously the biggest observability gap.

CloudWatch Container Insights complements this by providing node, pod, and container-level metrics with pre-built dashboards.

Terraform Setup

The CloudWatch observability addon is deployed as an EKS addon with Pod Identity - the recommended approach for Auto Mode:

# In the EKS module
addons = {
  metrics-server = {
    most_recent = true
  }
  amazon-cloudwatch-observability = {
    most_recent = true
    pod_identity_association = [{
      role_arn        = aws_iam_role.cloudwatch.arn
      service_account = "cloudwatch-agent"
    }]
  }
}

The metrics-server addon is required for HPA to read CPU and memory utilization. Without it, HPA cannot function.

The IAM role uses Pod Identity (not IRSA) with least-privilege policies:

data "aws_iam_policy_document" "cloudwatch_assume" {
  statement {
    actions = ["sts:AssumeRole", "sts:TagSession"]
    principals {
      type        = "Service"
      identifiers = ["pods.eks.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "cloudwatch" {
  name               = "${var.project_name}-cloudwatch-observability"
  assume_role_policy = data.aws_iam_policy_document.cloudwatch_assume.json
}

resource "aws_iam_role_policy_attachment" "cloudwatch_agent" {
  role       = aws_iam_role.cloudwatch.name
  policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
}

resource "aws_iam_role_policy_attachment" "xray_write" {
  role       = aws_iam_role.cloudwatch.name
  policy_arn = "arn:aws:iam::aws:policy/AWSXrayWriteOnlyAccess"
}

Key decisions:

Pod Identity over IRSA - Pod Identity is the recommended approach for EKS addons. It is built into Auto Mode (the Pod Identity Agent runs on every node automatically).
Trust principal is pods.eks.amazonaws.com - This is the Pod Identity service, not ec2.amazonaws.com.
AWSXrayWriteOnlyAccess - Required alongside CloudWatchAgentServerPolicy for full observability including trace collection.

What You Get

Once deployed, Container Insights provides:

Node metrics - CPU, memory, network, and disk utilization per node
Pod metrics - CPU and memory usage per pod, with requests vs actual
Container-level performance - Individual container metrics within pods
Cluster dashboards - Pre-built CloudWatch dashboards for cluster health
Log collection - Application and system logs forwarded to CloudWatch Logs

This complements Managed Capability Logging by providing resource-level metrics that controller logs don't surface. Container Insights gives you the visibility to understand scaling behavior, identify resource bottlenecks, and verify that consolidation is working as expected.

Security Out of the Box

Auto Mode provides a strong security posture with minimal configuration. Here is what you get by default:

Node Security

Bottlerocket OS - Minimal, purpose-built container OS with SELinux enforcing and a read-only root filesystem. No package manager, no shell access.
IMDSv2 enforced - Instance metadata hop limit hardcoded to 1, preventing containers from accessing node credentials via IMDS.
No SSH/SSM access - Nodes are locked down. You cannot open a shell on an Auto Mode node.
14-day node rotation - Nodes expire after 14 days by default (configurable up to 21 days) and are automatically replaced with the latest patched Bottlerocket AMI, ensuring CVE fixes are applied continuously.
Drift detection - When AWS releases a new AMI (approximately weekly), Auto Mode detects the drift and replaces nodes with the updated image.

Data Security

Encrypted EBS volumes - Auto Mode's managed EBS CSI driver supports encrypted volumes via StorageClass parameters (encrypted: "true"), ensuring persistent volumes are encrypted at rest.
allowedTopologies - StorageClasses can restrict volume provisioning to Auto Mode compute nodes via eks.amazonaws.com/compute-type: auto.
Secrets encryption - EKS supports envelope encryption for Kubernetes secrets using KMS. Add encryption_config to the EKS module to encrypt etcd secrets at rest with your own KMS key. This is not enabled by default but is recommended for production clusters handling sensitive data.

IAM Security

Least-privilege node role - Uses AmazonEKSWorkerNodeMinimalPolicy (not the broader AmazonEKSWorkerNodePolicy) and AmazonEC2ContainerRegistryPullOnly (not AmazonEC2ContainerRegistryReadOnly).
Pod Identity - Built-in, used for the CloudWatch addon. No need for IRSA or node-level IAM policies for application workloads.
Separate IAM roles - Cluster role, node role, and addon roles are distinct with non-overlapping permissions.

Network Security

Private subnets - Nodes run exclusively in private subnets with no direct internet access.
Managed VPC CNI - Network policies are enforced at the CNI level without additional tooling.
NAT gateway - Outbound traffic routes through NAT, providing a single egress point.

Resource Requests Matter More Than Ever

This deserves its own section because it is the single most important operational consideration with Auto Mode.

In a traditional EKS setup with Managed Node Groups, you might have fixed-size nodes (e.g., m5.xlarge) and the Kubernetes scheduler bins pods onto available capacity. Wasted capacity is absorbed by the fixed node pool.

With Auto Mode (and Karpenter in general), nodes are provisioned to fit the requested resources. If your pod requests 256m CPU and 512Mi memory, Karpenter might provision a small instance just for that pod. If your pod actually needs 2 vCPU but only requests 256m, it will get throttled.

Best practices for resource requests:

Always set both CPU and memory requests on every container
Use tools like Goldilocks or VPA in recommendation mode to determine actual usage before setting requests
Set requests close to actual usage - not the theoretical maximum
Set limits higher than requests for burst capacity, but not orders of magnitude higher
Monitor actual vs requested using Container Insights or Prometheus

The Karpenter Connection

Understanding that Auto Mode is Karpenter helps explain its behavior:

Consolidation is enabled by default. If a node is empty or underutilized, Auto Mode replaces it with a better-fitting instance
Node expiration is 14 days by default (max 21 days). Nodes are automatically replaced, picking up the latest Bottlerocket AMI
Drift detection triggers replacement when AWS releases a new AMI (approximately weekly)
Disruption budget of 10% prevents too many nodes from being replaced simultaneously
Instance selection is automatic - Karpenter evaluates the C/M/R families (generation 4+ for built-in, 5+ for our Spot pool) and picks the optimal size based on pending pod requests

Since February 2026, you can access Karpenter's logs through Managed Capability Logging. Configure CloudWatch Vended Logs to deliver compute autoscaling logs to CloudWatch Logs, S3, or Kinesis Data Firehose. For quick troubleshooting, Kubernetes events (from eks-auto-mode/compute) remain the fastest signal.

# View Auto Mode compute events
kubectl get events -A --sort-by='.lastTimestamp' | tail -20

# Check node provisioning
kubectl get nodes -o wide --show-labels

# See NodePool status
kubectl get nodepools
kubectl describe nodepool general-purpose

Cost Analysis

EKS Auto Mode has three cost components:

Component	Cost
EKS cluster fee	$0.10/hour ($72/month)
EC2 instances	Standard pricing (On-Demand, RI, Savings Plans, Spot)
Auto Mode management fee	~12% on top of EC2 instance costs

Example Monthly Costs

For a cluster running 3x m6i.xlarge (4 vCPU, 16 GiB each) on On-Demand:

Item	Monthly Cost
EKS cluster	$72
3x m6i.xlarge On-Demand	~$432
Auto Mode fee (~12%)	~$52
NAT Gateway (single)	~$32 + data
ALB (light traffic)	~$22
Total	~$610/month

Compare to self-managed EKS with Karpenter:

Item	Monthly Cost
EKS cluster	$72
3x m6i.xlarge On-Demand	~$432
Karpenter pod overhead	~$0 (runs on existing nodes)
NAT Gateway	~$32 + data
ALB	~$22
Total	~$558/month

The ~$52/month premium (9-12% of compute) buys you elimination of operational overhead for Karpenter, VPC CNI, EBS CSI driver, ALB controller, Pod Identity Agent, AMI lifecycle, OS patching, and node health monitoring. For most teams, that operational burden costs far more than $52/month in engineer time.

Cost Optimization Strategies

Custom Spot NodePools - Create a Spot pool (shown above) for fault-tolerant workloads. Spot savings of 60-70% significantly offset the management fee
Right-size resource requests - Auto Mode provisions exactly what you request. Accurate requests mean less wasted capacity
Consolidation - Enabled by default. Auto Mode automatically bin-packs and replaces underutilized nodes
Compute Savings Plans - Apply to Auto Mode EC2 instances just like regular instances

Recent Feature Additions

Auto Mode has been gaining features steadily since GA. Notable additions through early 2026:

Managed Capability Logging (February 2026) - Deliver logs from compute autoscaling, block storage, load balancing, and pod networking to CloudWatch Logs, S3, or Kinesis Data Firehose via CloudWatch Vended Logs. This closed the biggest observability gap.
Secondary IP mode (December 2025) - IP conservation for large clusters
SOCI parallel pull (November 2025) - Up to 60% faster container image pulls for GPU instances
Static-capacity NodePools (November 2025) - Fixed replica count for predictable workloads that need a guaranteed number of nodes
GovCloud availability for Auto Mode (October 2025) - US-East and US-West GovCloud regions
FIPS-compatible AMIs (October 2025) - For US compliance requirements
Local Zones support (October 2025) - Deploy Auto Mode nodes to edge locations
associatePublicIPAddress (August 2025) - Control public IP assignment in NodeClass
ODCR/Capacity Reservation support (June 2025) - Use On-Demand Capacity Reservations with Auto Mode
Pod subnet and security group selectors (June 2025) - podSubnetSelectorTerms and podSecurityGroupSelectorTerms in NodeClass for network isolation

Static-capacity NodePools and ODCR support are particularly relevant for enterprise readers who need guaranteed capacity for critical workloads.

Current Limitations

Auto Mode is not the right choice for every workload. Here are the constraints:

No custom AMIs - Only AWS-provided Bottlerocket variants. If you need Amazon Linux, Ubuntu, or custom-built images with compliance tooling baked in, Auto Mode won't work. Workaround: deploy host-level tooling as DaemonSets
No SSH/SSM access to nodes - Nodes are locked down. You can get node logs through Kubernetes APIs and EKS troubleshooting tools, but cannot open a shell on a node
Controller logs require extra setup - Karpenter, ALB Controller, and other managed components run off-cluster. Since February 2026, you can access their logs via Managed Capability Logging (CloudWatch Vended Logs), but this requires explicit configuration - it is not enabled by default
No self-managed Karpenter - Cannot coexist. Must uninstall before enabling Auto Mode
No third-party CNI - Only the managed VPC CNI. Calico and Cilium (as CNI) are not supported. Cilium for network policy only may work
Linux only - No Windows node support. This matters for .NET shops running Windows containers
IMDSv2 hop limit of 1 - Hardcoded, cannot be changed. This breaks some tools that need IMDS access from within pods (Datadog agent, some AWS SDK configurations). Workaround: pods that require IMDS access must run with hostNetwork: true
No DRA - Dynamic Resource Allocation is not yet supported
No Gateway API - The managed ALB controller only supports the Ingress API and Service annotations (for NLB). If you need Gateway API (HTTPRoute, GatewayClass), you would need to disable Auto Mode's load balancing capability and self-manage the AWS Load Balancer Controller, which defeats much of the purpose of Auto Mode

The most common blocker I see discussed is the lack of custom AMIs and SSH access. If your security team requires specific hardening beyond what Bottlerocket provides, or your compliance framework mandates SSH-based auditing, Auto Mode is not ready for you yet.

Migration Considerations

If you have an existing EKS cluster, migration to Auto Mode is possible but has significant caveats.

Storage Migration

PVCs created with the self-managed EBS CSI driver (ebs.csi.aws.com) cannot be directly mounted by Auto Mode's driver (ebs.csi.eks.amazonaws.com). You need to recreate PV/PVC resources. AWS provides the eks-auto-mode-ebs-migration-tool to help.

Load Balancer Migration

Existing load balancers from the self-managed AWS Load Balancer Controller cannot be transferred to the managed controller. You need a blue-green approach - create new LBs under the managed controller and shift traffic via DNS.

From Self-Managed Karpenter

Uninstall self-managed Karpenter
Uninstall AWS Load Balancer Controller
Enable Auto Mode
Recreate NodePool/NodeClass resources with Auto Mode API versions

Real-World Advice

The Playtomic engineering team documented their migration experience and concluded: "If you're already running an EKS cluster with Karpenter and AWS Load Balancer Controller, we don't recommend migrating to EKS Auto Mode." Their recommendation is to adopt Auto Mode for new clusters.

I agree with this assessment. The migration path has enough friction that the ROI for existing, well-running clusters is questionable. For greenfield deployments, Auto Mode is the obvious choice (if the limitations don't block you and you want an easier approach to running full Kubernetes on AWS)

When to Use Auto Mode vs Standard EKS vs Fargate

Criteria	EKS Auto Mode	Standard EKS	EKS on Fargate
Node management	AWS managed	You manage	Serverless (no visible nodes)
Custom AMIs	No	Yes	N/A
SSH to nodes	No	Yes	N/A
DaemonSets	Yes	Yes	No
Persistent storage	Yes (managed EBS)	Yes	EFS only
GPU support	Yes (managed drivers)	Yes	No
Instance type control	Via NodePool constraints	Full control	N/A
Third-party CNI	No	Yes	No
Windows	No	Yes	No
Cost premium	~12% on compute	None	Fargate pricing
Operational burden	Low	High	Lowest

Choose Auto Mode when you want Kubernetes with minimal operational overhead, your workloads run on Linux, you don't need custom AMIs or SSH access, and you value AWS managing the full data plane.

Choose Standard EKS when you need custom AMIs, third-party CNIs, want to use Gateway API, need SSH access, or your team already has mature Karpenter automation.

Choose Fargate when you have lightweight stateless workloads, need true serverless (no node concept), or want the absolute simplest deployment model.

Best Practices

After building and testing this setup, here are the practices I recommend:

Always set resource requests on every container. Auto Mode provisions based on requests. Missing requests means unpredictable bin-packing and potentially oversized nodes.
Use custom Spot NodePools for non-critical workloads. The built-in general-purpose pool is On-Demand only. Creating a Spot pool can cut compute costs by 60-70% for batch jobs, dev environments, and other interruptible work.
Tag your subnets correctly. Auto Mode discovers subnets via kubernetes.io/role/elb and kubernetes.io/role/internal-elb tags. Missing tags means load balancers and nodes won't land where you expect.
Use the right resource class names. StorageClass provisioner is ebs.csi.eks.amazonaws.com, IngressClass is alb (not eks.amazonaws.com/alb), NLB loadBalancerClass is eks.amazonaws.com/nlb. You must create IngressClassParams and IngressClass resources explicitly - Auto Mode does not auto-register them.
Configure PodDisruptionBudgets. Auto Mode consolidates underutilized nodes by default. PDBs ensure your application maintains availability during node replacement and consolidation. The demo includes a PDB with minAvailable: 1.
Enable observability. Configure Managed Capability Logging for controller logs (compute, storage, LB, networking) and deploy Container Insights with Pod Identity for node and pod metrics.
Start with Auto Mode for new clusters. Don't migrate existing clusters unless you have a compelling reason. The storage and load balancer migration paths add significant risk.
Test the IMDSv2 hop limit. If your pods use the AWS SDK, IMDS, or tools like the Datadog agent that rely on instance metadata, verify they work with the hardcoded hop limit of 1. Pods that need IMDS access must use hostNetwork: true.
Use least-privilege IAM. Stick with AmazonEKSWorkerNodeMinimalPolicy and AmazonEC2ContainerRegistryPullOnly for the node role. Use Pod Identity for application IAM needs.

The Complete Terraform

The full Terraform code for this article is available on GitHub: github.com/RDarrylR/eks-auto-mode-terraform

The repository includes:

VPC with public and private subnets across 3 AZs
EKS cluster with Auto Mode enabled (Kubernetes 1.35)
Custom Spot NodePool and NodeClass (Karpenter CRDs)
Least-privilege IAM roles (cluster, node, CloudWatch)
CloudWatch Container Insights with Pod Identity
ECR repository for the demo application
Demo FastAPI app with /info (node scheduling details), /compute (CPU stress), and /stress (memory pressure) endpoints
Kubernetes manifests with ALB, HPA, and IngressClass configuration
Load generator for autoscaling demos
Makefile with deployment, scaling demo, and inspection commands

# Deploy the infrastructure
make init
make plan
make apply

# Build and push the demo app
make docker-build
make docker-push

# Deploy the application
make deploy-app

# Run the scaling demo
make demo-scale-up    # Scale to 15 replicas, watch node provisioning
make demo-watch       # Watch HPA, pods, nodes, and events
make demo-scale-down  # Scale back, watch consolidation

# Or trigger HPA-driven scaling
make demo-load-start  # Deploy load generator
make demo-watch       # Watch the full scaling chain
make demo-load-stop   # Clean up

# Clean up
make delete-app
make destroy

CLEANUP (IMPORTANT!!)

If you deploy this infrastructure, it will cost you real money (~$610/month for the default configuration). Please do not forget about it.

Make sure to delete all resources when you are done:

make delete-app   # Remove K8s resources first (avoids orphaned LBs/ENIs)
make destroy      # Destroy all Terraform resources

The EKS cluster, EC2 instances, NAT gateway, and ALB all incur hourly charges. Even if you are not running any application workloads, the cluster and VPC infrastructure will continue to cost you money until it is destroyed.

Wrapping Up

EKS Auto Mode represents a significant shift in how AWS approaches Kubernetes. Instead of handing you a control plane and saying "good luck with the rest," AWS now manages the entire stack - compute, networking, storage, and add-ons - while you focus on your applications. It's similar in philosophy to Fargate, but with full EC2 visibility and Kubernetes compatibility.

The ~12% cost premium is real, but the operational simplification is substantial. No more Karpenter upgrades, no more VPC CNI compatibility issues, no more EBS CSI driver version mismatches, no more ALB Controller deployments. For teams that want Kubernetes without the infrastructure burden, Auto Mode delivers.

The Terraform setup with the community module is straightforward - enable a flag, configure your NodePools, and deploy. The biggest adjustment is remembering the Auto Mode resource class names (eks.amazonaws.com/*) and accepting that you are trading visibility and control for operational simplicity.

The security posture is strong out of the box - Bottlerocket with SELinux, IMDSv2 enforced, encrypted storage, least-privilege IAM, and automatic node rotation. Observability is now comprehensive - Managed Capability Logging delivers controller logs to CloudWatch, and Container Insights provides pod and node metrics. And the autoscaling story is seamless - HPA drives pod scaling, Karpenter drives node scaling, consolidation cleans up the rest.

If you are starting a new Kubernetes project on AWS in 2026, Auto Mode should be considered for your default choice unless you have a problem with one of the things in the limitations list or have a specific reason to manage the data plane yourself.

Resources

EKS Auto Mode Overview - AWS Docs
EKS Auto Mode Best Practices
Under the Hood: Amazon EKS Auto Mode
Getting Started with EKS Auto Mode
EKS Auto Mode Release Notes
terraform-aws-modules/eks Auto Mode Example
Migrate from Karpenter to Auto Mode
EKS Auto Mode Security Whitepaper
Playtomic Migration Experience
Amazon EKS Pricing
Full Terraform Code - GitHub
AWS London Ontario User Group - Meetups, talks, and community for AWS builders in the London, Ontario area
AWS London Ontario User Group YouTube - Recorded talks and presentations

Connect with me on X, Bluesky, LinkedIn, GitHub, Medium, Dev.to, or the AWS Community. Check out more of my projects at darryl-ruggles.cloud and join the Believe In Serverless community.

Elastic Container Service (ECS): My default choice for containers on AWS

Darryl Ruggles — Wed, 18 Mar 2026 14:08:11 +0000

Amazon Elastic Container Service is the default AWS service I reach for whenever I need to run containers. Whether it's a batch processing pipeline that fans out across hundreds of Fargate tasks or a FastAPI backend sitting behind an Application Load Balancer, ECS handles the orchestration without the operational complexity of Kubernetes. The control plane is free, the AWS integration is deep, and as of early 2026, the deployment capabilities rival anything in the container ecosystem.

I recently presented on ECS and decided to write down the things I have learned from building real projects into one place. This blog post is the companion to that presentation - a deep dive into what ECS offers, how I use it, and how you can start building with it today.

Why Containers?

Before we talk about ECS specifically, let's talk about why containers matter. Four core principles make containers compelling:

Consistency - The same image runs identically on your laptop, in CI, and in production. No more "works on my machine."
Isolation - Each container gets its own filesystem, networking, and process space. Multiple services on the same host without conflicts.
Efficiency - Containers share the host OS kernel. Startup in seconds, not minutes. Far less overhead than virtual machines.
Portability - A Docker image runs on ECS, EKS, Lambda, or your own servers. Your business logic stays runtime-agnostic.

In my Aurora DSQL Kabob Store project, I made this a deliberate design decision - keeping business logic runtime-agnostic so the same FastAPI application could deploy on Fargate, EC2, EKS, or Lambda with minimal adapter code.

What is Amazon ECS?

ECS is a fully managed container orchestration service. You define what to run and how, and ECS handles placement, scaling, availability, and integration with the rest of AWS.

Four things make ECS stand out:

No control plane cost - Unlike EKS (~$75/month per cluster), the ECS orchestration layer is completely free. You only pay for the compute your containers use.
Deep AWS integration - IAM roles per task, CloudWatch Container Insights, native ALB target groups, Secrets Manager injection, and tight integration with every major AWS service.
Flexible compute - Choose between Fargate (serverless), EC2 (self-managed), or the new Managed Instances (AWS-managed EC2).
Deployment sophistication - Rolling updates, native blue/green, canary, and linear deployments all built in.

Core Concepts

Five building blocks make up ECS:

Cluster - A logical grouping of tasks and services. Think of it as your namespace. A cluster can span Fargate, EC2, and Managed Instances simultaneously.

Task Definition - The blueprint. A JSON document that specifies container images, CPU, memory, networking mode, volumes, IAM roles, and logging configuration. Versioned - each registration creates a new revision (e.g., my-app:3).

Task - A running instance of a task definition. One or more containers working together. On Fargate, each task gets its own elastic network interface and private IP.

Service - Maintains a desired count of tasks. Handles replacement of failed tasks, load balancer registration, auto scaling, and deployments.

Container Instance - An EC2 instance running the ECS agent, registered to a cluster. Only relevant if you're using the EC2 launch type.

Compute Options

ECS gives you four ways to provide compute for your containers. Choosing the right one depends on your workload characteristics, cost sensitivity, and operational preferences.

Fargate (Serverless) - The Default Choice

Fargate is what I recommend for most workloads. With it, there are no EC2 instances to manage. You specify CPU and memory at the task level, and AWS handles everything underneath.

CPU (vCPU)	Memory Options	Notes
0.25	512 MiB, 1-2 GB	Linux only
0.5	1-4 GB	Linux only
1	2-8 GB	Linux and Windows
2	4-16 GB	Linux and Windows
4	8-30 GB	Linux and Windows
8	16-60 GB	Linux only
16	32-120 GB	Linux only

Fargate supports both x86_64 and ARM64 (Graviton) architectures. Graviton gives you roughly 20% better price-performance for most workloads. Pricing is per-second based on vCPU and memory consumed.

Fargate Spot offers up to 70% savings for fault-tolerant workloads. When AWS reclaims capacity, tasks receive a SIGTERM with a 2-minute warning. I use this for batch processing jobs where interruption just means retrying one file.

EC2 Launch Type

This offers full control over the underlying instances. You choose the AMI, instance type, and manage patching and scaling yourself. Choose EC2 when you need GPUs, custom AMIs, specific instance families, or when sustained high utilization makes reserved instances cheaper than Fargate.

The trade-off is clear: more control, more operational burden.

ECS Managed Instances

Launched in September 2025, Managed Instances bridge the gap between Fargate simplicity and EC2 flexibility. AWS handles provisioning, auto-scaling, Bottlerocket OS patching (14-day cycles), and host replacement. You control instance type selection via attribute-based selection - say "I need 4 GPUs" and ECS picks the right instance.

The "start before stop" principle for host replacement is particularly nice - new capacity comes up before old goes down, maintaining availability throughout.

This is the answer for GPU workloads and ML inference where Fargate isn't an option but you don't want to manage EC2 fleets.

Capacity Providers

Capacity providers are the recommended way to configure compute. The strategy uses two parameters:

Base - Minimum tasks guaranteed on a specific provider (only one provider can have a base)
Weight - Relative proportion of tasks after the base is filled

Example: base 2 on FARGATE, weight 4 on FARGATE_SPOT, weight 1 on FARGATE. Your first 2 tasks are guaranteed to use on-demand Fargate. After that, 4 out of every 5 new tasks go to Spot. Cost optimization with a reliability floor.

Task Definitions - The Blueprint

The task definition is where you define everything about your containers. Here are the critical parameters:

Container Definitions

Each task definition contains one or more container definitions. Key parameters include:

image - Docker image from ECR (Elastic Container Registry), Docker Hub, or any private registry
essential - If an essential container stops, the entire task stops. Your main app is essential; your log router sidecar might not be
portMappings - Container ports, with named ports for Service Connect
healthCheck - CMD-SHELL command with configurable interval, timeout, retries, and start period
dependsOn - Container startup ordering with conditions: START, COMPLETE, SUCCESS, HEALTHY
restartPolicy - Container-level restarts without killing the entire task. Configurable attempt period (60-1800 seconds) and ignored exit codes

Task Role vs Execution Role

This distinction trips people up:

	Task Role	Execution Role
Purpose	Permissions for your application code	Permissions for the ECS agent
Used by	Your containers calling AWS APIs	ECS pulling images, pushing logs, fetching secrets
Example	S3 read/write, DynamoDB access	`ecr:GetAuthorizationToken`, `logs:CreateLogStream`, `secretsmanager:GetSecretValue`

Two different roles, two different purposes. The task role follows least privilege for your application. The execution role is about infrastructure plumbing.

Secrets Injection

ECS natively injects secrets as environment variables from Secrets Manager or SSM Parameter Store:

{
  "secrets": [
    {
      "name": "DB_PASSWORD",
      "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789:secret:my-secret"
    },
    {
      "name": "API_KEY",
      "valueFrom": "arn:aws:ssm:us-east-1:123456789:parameter/my-param"
    }
  ]
}

Secrets Manager supports specific JSON keys (arn:...secret:my-secret:username::) and version staging. Never bake secrets into container images or task definitions.

Networking

Networking Modes

Mode	Description	Use With
awsvpc	Each task gets its own ENI and private IP. Per-task security groups. Required for Fargate.	Fargate, EC2, Managed Instances
bridge	Docker's virtual network. Dynamic port mapping with ALB.	EC2 only
host	Containers use host's network directly. No port isolation.	EC2 only
none	No external networking.	EC2 only

Recommendation: Use awsvpc unless you have a specific reason not to. It's the only mode that works everywhere and gives you per-task security groups.

VPC Architecture

For production workloads, run ECS tasks in private subnets. Use VPC endpoints for ECR, S3, and CloudWatch to avoid NAT gateway data transfer costs. This is the biggest hidden cost in ECS architectures - NAT gateways charging for every image pull and log push.

Service Connect

Service Connect is the recommended way to handle service-to-service communication. It automatically injects an Envoy proxy as a sidecar, providing:

Service discovery via Cloud Map namespaces
Client-side load balancing with retries and outlier detection
Free application-level traffic metrics in CloudWatch (request count, latency, error rates)
Support for HTTP, HTTP2, gRPC, and TCP
Per-request Envoy access logs (October 2025)

Service Connect replaces AWS App Mesh, which will be discontinued in September 2026.

Deployment Strategies

ECS has the most sophisticated deployment options of any container orchestrator on AWS. As of March 2026, four strategies are available natively.

Rolling Update (Default)

Gradually replaces old tasks with new ones. Controlled by:

minimumHealthyPercent (default 100%) - Minimum tasks that must remain running
maximumPercent (default 200%) - Maximum tasks allowed during deployment

For zero-downtime with a desired count of 4: min 100%, max 200% means ECS starts 4 new tasks, waits for them to be healthy, then stops the 4 old tasks.

Blue/Green

Built into ECS without CodeDeploy dependency. Provisions 100% new capacity ("green"), validates, then shifts all production traffic at once. Key features:

Six Lambda lifecycle hooks: pre-scale-up, post-scale-up, test traffic shift, production traffic shift, post-test, post-production
Configurable bake time for instant rollback window
Works with ALB, NLB, and Service Connect

Canary (October 2025)

Two-stage deployment: shift a small percentage of traffic first (configurable from 0.1% to 99.9%), validate with real production traffic, then shift the rest. Ideal for critical user-facing services where you want to limit blast radius.

Linear (October 2025)

Gradual traffic shift in equal increments as small as 3%, with configurable bake time between each step. The most conservative approach - allows monitoring at each increment.

Deployment Circuit Breaker

The safety net across all strategies. If tasks keep failing to start or pass health checks, ECS automatically stops the deployment and optionally rolls back to the last successful version. You can wire CloudWatch Alarms into the circuit breaker to detect application-level failures, not just infrastructure failures.

Service Auto Scaling

ECS uses Application Auto Scaling with four policy types:

Target Tracking - Set a target metric value (e.g., CPU at 50%). Simplest to configure - works like a thermostat.
Step Scaling - Define explicit threshold/action pairs. React differently at different severity levels.
Scheduled Scaling - Time-based. Scale up for business hours, down at night. Supports scaling to zero (set minimum capacity to 0).
Predictive Scaling - ML-based. Analyzes historical patterns and proactively scales before demand hits. Doesn't trigger scale-ins on its own - pair with target tracking.

Important behavior: scale-in is automatically paused during deployments to protect availability.

Storage Options

Storage	Persistence	Shared	Use Case
Ephemeral	Task lifetime	Within task	Temp files, caches. Default 20 GiB, up to 200 GiB on Fargate
EFS	Persistent	Across tasks	Shared config, models, content. Multi-AZ, IAM auth
EBS	Configurable	Single task	High-IOPS data processing. One volume per task
Bind Mounts	Task lifetime	Within task	Container-to-container data sharing

EFS is the most versatile - persistent, shared across tasks, supports IAM authorization and transit encryption. EBS is for high-performance block storage when EFS throughput is insufficient. One gotcha with EFS though is that it can take minutes to create a new EFS but this is typically a one time thing for a given application.

Security

Security in ECS follows the principle of least privilege at the task level:

Task roles - Each task definition gets its own IAM role. Your batch processor gets S3 and SQS access. Your API gets DynamoDB access. Not a shared instance profile.
Secrets injection - Secrets Manager and SSM Parameter Store values injected as environment variables at startup.
Network isolation - awsvpc mode gives each task its own security group. Run tasks in private subnets.
Read-only root filesystem - Run containers with readonlyRootFilesystem: true for hardening.
Image scanning - ECR enhanced scanning with Amazon Inspector continuously scans for OS and language package vulnerabilities. As of 2026, it supports minimal base images like scratch and distroless, and shows which images are running in your clusters.

Observability

CloudWatch Container Insights (Enhanced)

Container Insights provides granular metrics at the cluster, service, task, and container level. The honeycomb visualization gives you cluster health at a glance - alarm state and utilization side by side. Deployment tracking alongside infrastructure anomalies. Cross-account monitoring for unified views. Can be enabled per-cluster or account-wide.

Logging

The awslogs driver sends container logs directly to CloudWatch Logs. As of June 2025, the default mode switched from blocking to non-blocking - if the log buffer fills up, excess logs are dropped rather than blocking your application. This prioritizes availability over logging completeness.

For advanced log routing, FireLens with Fluent Bit as a sidecar routes logs to any destination - CloudWatch, S3, Elasticsearch, Datadog, Splunk. Different containers can route to different destinations.

Tracing

Deploy the AWS Distro for OpenTelemetry (ADOT) collector as a sidecar. It receives OTLP traces on port 4317 (gRPC) or 4318 (HTTP) and exports to X-Ray automatically. This replaces the legacy X-Ray daemon approach.

ECS Exec

ECS Exec lets you shell into a running container directly - the equivalent of docker exec but for tasks running on Fargate or EC2. It uses AWS Systems Manager (SSM) under the hood, so there's no need to open inbound ports or SSH. I use this all the time - it's one of the most useful ECS features IMO.

To enable it, set enableExecuteCommand: true on your service or run task call. Then:

aws ecs execute-command \
  --cluster my-cluster \
  --task abc123 \
  --container my-app \
  --interactive \
  --command "/bin/sh"

This is invaluable for debugging - inspecting environment variables, checking network connectivity, verifying mounted volumes, or tailing logs inside the container. A few things to keep in mind:

It must be enabled before the task launches - you can't retroactively enable it on already-running tasks. For services, enabling it requires a new deployment
The task role needs SSM permissions (ssmmessages:CreateControlChannel, ssmmessages:CreateDataChannel, ssmmessages:OpenControlChannel, ssmmessages:OpenDataChannel)
The container image needs a shell (/bin/sh or /bin/bash) - scratch and distroless images won't work
All sessions are logged to CloudWatch or S3 for audit
Works with both Fargate and EC2 launch types

For quick diagnostics, the amazon-ecs-exec-checker script validates that your task, role, and agent are configured correctly.

Real-World Architecture: Batch Processing with Fargate

My Serverless Data Processor project demonstrates the batch processing pattern.

The architecture: S3 upload triggers a Lambda that extracts files. Step Functions distributed map fans out processing across Fargate tasks - each file gets its own container. The containers use the waitForTaskToken pattern - Step Functions passes a callback token as an environment variable, the Rust container processes the data, then calls send_task_success to signal completion.

Key details:

Fargate at minimum specs: 0.25 vCPU, 512 MB RAM
OpenTelemetry sidecar for CloudWatch metrics
Container images in ECR with multi-stage Docker builds
Written in Rust for the worker containers
Infrastructure managed with Terraform

Here is the Step Functions integration that launches Fargate tasks:

{
  "Type": "Task",
  "Resource": "arn:aws:states:::ecs:runTask.waitForTaskToken",
  "Parameters": {
    "LaunchType": "FARGATE",
    "Cluster": "${ecs_cluster}",
    "TaskDefinition": "${task_def_name}",
    "NetworkConfiguration": {
      "AwsvpcConfiguration": {
        "Subnets": ["${fargate_subnet}"],
        "SecurityGroups": ["${vpc_default_sg}"]
      }
    },
    "Overrides": {
      "ContainerOverrides": [{
        "Name": "store_data_processor_daily",
        "Environment": [
          {"Name": "TASK_TOKEN", "Value.$": "$$.Task.Token"},
          {"Name": "S3_BUCKET", "Value.$": "$.BatchInput.source_bucket_name"},
          {"Name": "S3_KEY", "Value.$": "$.Items[0].Key"}
        ]
      }]
    }
  }
}

This pattern works for any fan-out workload - ETL, media processing, report generation, ML batch inference. Each task is independent, starts in seconds, processes its data, and terminates. You pay only for the compute time used.

I also used this same Fargate + callback token pattern in my Serverless Pizza Ordering project, where the Fargate container simulated pizza preparation and delivery - chosen over Lambda because the "AI" insisted some pizzas could take more than 15 minutes.

Real-World Architecture: API Backend on Fargate

My Aurora DSQL Kabob Store project uses ECS Fargate as an always-on API backend.

React frontend behind CloudFront, Application Load Balancer routing to FastAPI containers on Fargate, connecting to Aurora DSQL for multi-region active-active writes.

The key design decision: keep business logic runtime-agnostic. The same FastAPI application uses direct psycopg2 queries (not ORM) so it can deploy across Fargate, ECS on EC2, Lambda, or EKS with minimal adapter code.

Fargate costs about 20-30% more than equivalent EC2 on-demand for sustained workloads, but the operational simplicity during development is worth it. In practice, real migrations from Fargate to EC2 often yield smaller savings than expected - Tines reported only ~5% compute cost savings after migrating, though they saw 30% faster job processing and 10% lower P95 latency from having dedicated hardware. The biggest cost was actually the VPC infrastructure - NAT gateways at ~$2-3/day - not ECS itself. I terraform destroy when not actively developing.

Terraform Examples

Here is the core Terraform for an ECS Fargate setup, taken from my Serverless Data Processor project:

Cluster

resource "aws_ecs_cluster" "ecs_cluster" {
  name = "${var.project_name}-cluster"

  setting {
    name  = "containerInsights"
    value = "enabled"
  }
}

Task Definition

resource "aws_ecs_task_definition" "fargate_processor_task" {
  family                   = var.task_definition_name
  execution_role_arn       = aws_iam_role.ecs_task_execution_role.arn
  task_role_arn            = aws_iam_role.ecs_task_role.arn
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = var.fargate_cpu
  memory                   = var.fargate_memory

  container_definitions = templatefile(
    "${path.module}/container-definitions.json.tpl",
    {
      app_image           = var.app_image
      fargate_cpu         = var.fargate_cpu
      fargate_memory      = var.fargate_memory
      aws_region          = var.aws_region
      project_name        = var.project_name
      task_container_name = var.task_container_name
    }
  )
}

Container Definition Template

[
  {
    "cpu": ${fargate_cpu},
    "essential": true,
    "image": "${app_image}",
    "memory": ${fargate_memory},
    "name": "${task_container_name}",
    "networkMode": "awsvpc",
    "environment": [
      {"name": "S3_BUCKET", "value": "my-bucket"},
      {"name": "S3_KEY", "value": "data/input.json"}
    ],
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/ecs/${project_name}",
        "awslogs-region": "${aws_region}",
        "awslogs-stream-prefix": "${project_name}-log-stream"
      }
    }
  },
  {
    "image": "public.ecr.aws/aws-observability/aws-otel-collector:latest",
    "name": "aws-otel-collector",
    "essential": false,
    "command": ["--config=/etc/ecs/ecs-cloudwatch.yaml"],
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/ecs/${project_name}-otel",
        "awslogs-region": "${aws_region}",
        "awslogs-stream-prefix": "otel"
      }
    }
  }
]

ECS Service with ALB (for always-on workloads)

resource "aws_ecs_service" "api_service" {
  name            = "${var.project_name}-service"
  cluster         = aws_ecs_cluster.ecs_cluster.id
  task_definition = aws_ecs_task_definition.api_task.arn
  desired_count   = var.desired_count
  launch_type     = "FARGATE"

  network_configuration {
    subnets          = var.private_subnet_ids
    security_groups  = [aws_security_group.ecs_tasks.id]
    assign_public_ip = false
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.api.arn
    container_name   = var.container_name
    container_port   = var.container_port
  }

  deployment_circuit_breaker {
    enable   = true
    rollback = true
  }

  depends_on = [aws_lb_listener.api]
}

Three resources for the core ECS setup: cluster, task definition, and service. The container definition template handles the application specifics. The full Terraform for both projects is in the GitHub repos linked at the end.

Recent Features (2025-2026)

ECS has had a remarkable year of feature launches:

AI Developer Tools (December 2025) - ECS MCP Server for AI-assisted development and operations. Natural language commands for cluster management.
ECS Express Mode (November 2025) - Deploy a production-ready containerized web app with just three inputs: a container image, a task execution role, and an infrastructure role. Provisions Fargate, ALB with SSL, auto scaling, monitoring, and a unique URL. Up to 25 services can share one ALB. No additional charge beyond the underlying resources.
Canary and Linear Deployments (October 2025) - Fine-grained traffic shifting. Canary from 0.1% to 99.9%, linear in increments as small as 3%.
Service Connect Envoy Access Logs (October 2025) - Per-request telemetry for HTTP, HTTP2, gRPC, and TCP.
ECS Managed Instances (September 2025) - AWS-managed EC2 with Bottlerocket OS. Attribute-based instance selection for GPUs and specialized hardware.
Native Blue/Green Deployments (July 2025) - Built into ECS without CodeDeploy. Six Lambda lifecycle hooks for testing and approval at each phase. Configurable bake time for instant rollback.
Non-Blocking Log Driver Default (June 2025) - Prioritizes task availability over logging completeness.

ECS vs EKS - When to Use What

This is the most common question I get. Both solve the same fundamental problem - running containers reliably at scale.

Criteria	ECS	EKS
Control plane cost	Free	~$75/month per cluster
Learning curve	AWS-native concepts	Kubernetes concepts
AWS integration	Deep, native	Good, via add-ons
Multi-cloud	AWS only	Portable K8s manifests
Ecosystem	AWS tooling	Helm, ArgoCD, Istio, operators
Managed compute	Fargate, Managed Instances	Fargate, managed node groups

Choose ECS when your team is AWS-focused, you want operational simplicity, you value the free control plane, and your workloads are straightforward services and batch jobs.

Choose EKS when your team knows Kubernetes, you need multi-cloud portability, you want the Kubernetes ecosystem (Helm, ArgoCD, custom operators), or you're running complex stateful workloads.

Most organizations pick based on team expertise and existing tooling, not technical limitations.

ECS vs Lambda - Containers vs Functions

Another comparison that comes up frequently:

Criteria	ECS/Fargate	Lambda
Max duration	Unlimited	15 minutes
Max memory	120 GB	10 GB
Startup	Seconds (image pull)	Milliseconds (warm) to seconds (cold)
Pricing	Per-second (vCPU + memory)	Per-invocation + duration
Scaling	Service auto scaling	Automatic per-request
Best for	Long-running, resource-heavy, always-on	Event-driven, short-lived, bursty

In my projects I use both. Lambda for event handling - S3 triggers, API endpoints, file extraction. Fargate for heavy processing - data transformation, ML inference, container workloads that need full runtime control.

Best Practices

After building several production systems with ECS, here are the practices I've found most valuable:

Use Fargate unless you need GPUs or specific instance types. The operational simplicity is worth the cost premium for most workloads.
Use awsvpc networking mode everywhere. It's the only mode that works on all compute types and gives you per-task security groups.
Enable the deployment circuit breaker with rollback. This catches failed deployments before they impact all traffic.
Use capacity provider strategies to mix Spot and on-demand. A base of on-demand with weighted Spot gives you cost savings with a reliability floor.
Inject secrets via Secrets Manager. Never bake them into images or pass them as plain environment variables.
Enable Container Insights. The per-task metrics and honeycomb visualization are invaluable for debugging.
Use Service Connect for service-to-service communication. Free traffic metrics and managed Envoy proxies with no code changes.
Use VPC endpoints for ECR, S3, and CloudWatch. NAT gateway data transfer costs are the biggest hidden expense in ECS architectures.
Use multi-stage Docker builds. Keep images small. A Rust binary in a scratch image is a few megabytes. A Python app in a slim image with only production dependencies.
Define health checks in the task definition. Don't rely solely on ALB health checks - container-level health checks catch issues faster.

Pricing - You Pay for Compute, Not Orchestration

The most important thing to know: ECS orchestration is free. You only pay for the compute resources your containers consume.

Fargate - Per-second billing for vCPU ($0.04048/hour) and memory ($0.004445/GB/hour). Spot is up to 70% less. Compute Savings Plans can reduce costs by up to 49% (3-year all-upfront) or ~20% (1-year no-upfront).
EC2 - Standard instance pricing. Use Savings Plans or Reserved Instances for sustained workloads.
Managed Instances - EC2 instance pricing plus a management fee for automated provisioning, patching, and host replacement.

Hidden costs to watch:

NAT gateways - $0.045/GB for data processed. Use VPC endpoints.
ALB - Fixed hourly cost plus per-LCU. Up to 25 ECS Express Mode services can share one ALB.
ECR storage - $0.10/GB/month. Use lifecycle policies to clean up old images.
Ephemeral storage - Fargate charges $0.000111/GB/hour above the default 20 GiB.

Cost optimization strategies:

Right-sizing - The single biggest lever. Reducing from 1 vCPU/2GB to 0.5 vCPU/1GB can yield ~45-50% lower Fargate task cost.
Scheduled shutdowns - Running dev/staging environments only during business hours (8 hours/day, 5 days/week) can reduce costs by over 75%.
Savings Plans - Commit to consistent usage for 1-3 years. Even no-upfront 1-year plans save ~20% on Fargate.

For my projects, the ECS cost has been minimal. The batch processor runs tasks for seconds at minimum specs. The Kabob Store's main cost was VPC infrastructure ($2-3/day), not ECS.

Things to Know

A few operational details worth keeping in mind:

Task placement - Fargate handles placement automatically. For EC2, use the binpack placement strategy to consolidate workloads on fewer instances and reduce waste.
Task recycling - Fargate tasks on platform version 1.4.0+ are recycled after 14 days of continuous running. Your service will gradually replace old tasks.
ENI limits - In awsvpc mode on EC2, each task needs an ENI. Enable ENI trunking to increase density (requires CloudFormation custom resources).
Image pull time - Large images slow task startup. Keep images lean. Set ECS_IMAGE_PULL_BEHAVIOR=prefer-cached on EC2 instances to use cached images when available.
Spot instance draining - For EC2 Spot instances, set ECS_ENABLE_SPOT_INSTANCE_DRAINING=true on the ECS agent for graceful task termination.
Service quotas - Default Fargate vCPU quota is 6 on new accounts (up to 4,000 in production). Request increases proactively.
Force new deployment - If you update a secret or parameter store value, the running tasks won't pick it up automatically. Force a new deployment to refresh.

Wrapping Up

ECS is the container orchestration service I use most on AWS. The free control plane, deep AWS integration, and flexible compute options make it the right choice for most container workloads that don't require Kubernetes-specific tooling.

The recent feature launches have been particularly impressive - native blue/green without CodeDeploy, canary and linear deployments, Managed Instances for GPU workloads, and Express Mode for rapid prototyping. Combined with Fargate's serverless simplicity and Service Connect's built-in service mesh, ECS has matured into a comprehensive platform for running containers at any scale.

I've used it for batch data processing with Step Functions fan-out, pizza ordering with long-running container workflows, and multi-region API backends with Aurora DSQL. In every case, ECS handled the orchestration cleanly while I focused on the application logic.

If you're running containers on AWS and haven't looked at ECS recently, the current feature set is worth a fresh evaluation. Start with a Fargate service behind an ALB, enable Container Insights, and go from there.

Resources

Amazon ECS Documentation
Amazon ECS Best Practices Guide
Serverless Data Processor - Step Functions + Fargate - Batch processing with ECS
Serverless Pizza Ordering - Long-running Fargate container workflows
Aurora DSQL Kabob Store - FastAPI on Fargate with multi-region database
Step Functions + Fargate GitHub Repo - Full Terraform and Rust container code
Serverless Pizza GitHub Repo - Full Terraform and container code
AWS London Ontario User Group - Meetups, talks, and community for AWS builders in the London, Ontario area
AWS London Ontario User Group YouTube - Recorded talks and presentations

Connect with me on X, Bluesky, LinkedIn, GitHub, Medium, Dev.to, or the AWS Community. Check out more of my projects at darryl-ruggles.cloud and join the Believe In Serverless community.

Amazon EventBridge: The Event-Driven Backbone of AWS (And My Favourite Service)

Darryl Ruggles — Fri, 13 Mar 2026 01:29:07 +0000

I have been building serverless applications on AWS for years now, and if you asked me to pick one service that keeps showing up in almost every project I build, it would be Amazon EventBridge without hesitation. EventBridge is my favourite AWS service. It offers rules for reacting to events, pipes for bridging data across AWS services, a scheduler for time-based triggers, and a schema registry for discoverability - all fully serverless and practically free.

I recently had the opportunity to present on EventBridge to the AWS Cloud Club at Amity University Chhattisgarh, and I decided to consolidate key details about this service into one place. This blog post is the companion to that presentation - a dive into why EventBridge matters, how I use it, and how you can start building with it today.

Why Event-Driven Architecture?

Before we talk about EventBridge specifically, let's talk about why event-driven architecture (EDA) is worth adopting. Four core principles make EDA compelling:

Loose coupling - Producers emit events without knowing who consumes them. You can add new consumers without touching existing code.
Real-time reactivity - Respond to changes as they happen, instead of polling for updates.
Scalability by default - Event buses handle millions of events. Consumers scale independently.
Extensibility - New capabilities just subscribe to existing events. No rewiring required.

EventBridge is purpose-built to make this simple, scalable, and serverless on AWS.

The pattern is straightforward. On the left, producers - S3, IAM, CloudWatch, your applications - emit events onto EventBridge. In the middle, EventBridge routes them via rules and content-based patterns. On the right, consumers - Lambda, SNS, SQS, Step Functions - each subscribe independently. Producers don't know about consumers. Consumers don't know about producers. EventBridge is the decoupling layer.

What is Amazon EventBridge?

EventBridge is a serverless event bus that helps you receive, filter, transform, route, and deliver events. It has four major capabilities:

Event Buses

Default Event Bus - Every AWS account gets one automatically. It receives virtually all mutating API calls via CloudTrail - anything that creates, modifies, or deletes something across 250+ AWS services. Events flow whether you are listening or not. And it is completely free.
Custom Event Buses - Create your own buses and publish custom events. Cross-account sharing via resource policies. As of January 2025, EventBridge also supports direct cross-account delivery to targets like SQS, Lambda, SNS, Kinesis, and API Gateway - no intermediary event bus required in the target account.
SaaS Partner Buses - Partners like Datadog and PagerDuty can send events directly into your bus.

Rules & Event Patterns

Rules match events and route them to targets. You can have up to 5 targets per rule. The pattern matching is remarkably flexible:

Prefix/suffix matching - Match on the beginning or end of string values
Anything-but - Exclude specific values
Numeric ranges - Filter by greater-than, less-than, or between
Exists - Check for the presence or absence of a field
OR logic - Arrays match any value in the list

Matching happens at the EventBridge level - you only pay for events that actually match your rules.

Scheduler

Three types of schedules: rate-based, cron expressions, and one-time schedules. Supports millions of schedules with integrations across 270+ AWS services and over 6,000 API operations. Includes flexible time windows, automatic retries, and dead-letter queues. EventBridge Scheduler replaces the older scheduled rules and is significantly more capable.

Pipes

Pipes provide point-to-point integration with a pipeline architecture: Source → Filter → Enrich → Transform → Target. They replace the "glue Lambda" functions that many of us have written to connect services. Filtered events are free.

For example, a DynamoDB stream can flow into Pipes, filter for only INSERT events, enrich the data with a Lambda function, and deliver to an SQS queue - all without writing a custom Lambda to glue the pieces together.

Real-World Pattern: Security Monitoring with the Account Watcher

The best way to understand EventBridge is to see it in action. My Serverless AWS Account Watcher project is a perfect example. The challenge was simple: keeping track of all the actions being done in your AWS account can be a real challenge, especially from a security perspective. I needed to know immediately when someone deleted an S3 bucket, created an IAM access key, or logged into the console.

The architecture relies entirely on the default event bus. CloudTrail captures API activity - S3 deletions, IAM changes, console logins - and delivers those events to EventBridge automatically. Rules match specific event patterns and trigger a Lambda function that formats notifications and fans them out to SNS (for email) and a Slack webhook.

Here is what the SAM template looks like for the EventBridge rule:

Events:
  S3EventsRule:
    Type: EventBridgeRule
    Properties:
      Pattern:
        source:
          - aws.s3
        detail-type:
          - "AWS API Call via CloudTrail"
        detail:
          eventName:
            - DeleteBucket
            - DeleteBucketPolicy
            - PutBucketPolicy

That is it. A pattern and a source. SAM handles the rule creation, permissions, and target wiring. The Lambda handler uses Python's match/case to route different event types:

match event_name:
    case "DeleteBucket":
        message = f"S3 bucket '{bucket_name}' was deleted by {user}"
    case "CreateAccessKey":
        message = f"New access key created for {target_user} by {user}"
    case "ConsoleLogin":
        message = f"Console login by {user} from {source_ip}"

The last two lines of the handler are the entire fan-out - send_slack_message() and publish_to_sns(). Two notification channels, completely decoupled from the event detection logic.

The extensibility story is powerful here. Want to monitor a new API action? Add one line to the event pattern. Want to add a Microsoft Teams notification? Add one more function call. The event detection, routing, and notification logic are all independent.

I wrote about this project in detail in my Serverless AWS Account Watcher blog post - the full SAM template, Lambda code, and deployment instructions are all there.

Real-World Pattern: Health Check Alerts

My Serverless Site Health Check Notification System takes the same EventBridge pattern in a different direction. Route53 health checks monitor endpoints globally via HTTP, HTTPS, or TCP. When a health check fails, CloudWatch transitions an alarm state, and that state change flows through the default event bus.

I took advantage of the suffix matching. The EventBridge rule matches alarm names ending with -HealthCheckAlarm:

{
  "source": ["aws.cloudwatch"],
  "detail-type": ["CloudWatch Alarm State Change"],
  "detail": {
    "alarmName": [{ "suffix": "-HealthCheckAlarm" }]
  }
}

This means you can add new Route53 health checks without ever updating the EventBridge rule or redeploying your Lambda. Name the CloudWatch alarm with the -HealthCheckAlarm suffix, and it automatically triggers the existing notification pipeline. This is the extensibility promise of EDA in practice - new health checks, no code changes.

Real-World Pattern: Event-Driven Sales Analytics

In my Serverless Sales Analytics Platform, EventBridge plays a dual role. First, S3 object creation events flow through the default bus - when a store uploads daily sales data, an EventBridge rule matches the bucket and prefix, triggering a Step Functions workflow to transform and analyze the data. Second, an EventBridge Scheduler rule runs at 11 PM daily as a fallback, ensuring the analysis pipeline executes even if stores have not reported.

This combination of event-driven triggers and scheduled fallbacks is a pattern I use frequently. EventBridge handles both seamlessly - reactive processing when data arrives, and scheduled processing as a safety net.

EventBridge vs SNS vs SQS - When to Use What

This is one of the most common questions I get. All three are messaging services, but they serve different purposes:

Feature	EventBridge	SNS	SQS
What it does	Routes events based on content	Broadcasts to subscribers	Queues messages for consumers to pull
Filtering	Prefix, suffix, numeric, anything-but, exists	Basic attribute filters	None
AWS service events	250+ services on default bus, free	You wire it up yourself	You wire it up yourself
Schema discovery	Built-in Schema Registry	No	No
Targets	20+ supported target types	Lambda, SQS, HTTP, email subscriptions	Consumer pulls at their own pace
Ordering	Best-effort	Best-effort	FIFO available

Use EventBridge for intelligent content-based routing and reacting to AWS service events. Use SNS for simple pub/sub fan-out when you already know the event structure. Use SQS for guaranteed message processing with consumer-controlled throughput.

In practice, they work together. My Account Watcher uses EventBridge for routing and SNS for email delivery - each service doing what it does best.

EventBridge vs Kafka

Another comparison that comes up: Kafka is a distributed log for high-throughput streaming with replay capabilities. EventBridge is a managed event router for AWS-native architectures. They are fundamentally different tools. Kafka excels at sustained high-throughput data streaming (millions of messages per second). EventBridge excels at reactive event routing with content-based filtering and zero infrastructure management.

They can also work together - EventBridge Pipes can consume directly from Kafka topics.

Best Practices

After building several production systems with EventBridge, here are the practices I have found most valuable:

Use specific event patterns - The more precise your pattern, the fewer unnecessary Lambda invocations you pay for.
Implement dead-letter queues on rule targets - When EventBridge cannot deliver to a target, the DLQ captures the event for investigation. Use Lambda Destinations OnFailure for richer error context.
Use input transformers - Shape the event payload before it reaches your target. Reduce parsing logic in your Lambda functions.
Separate buses for different domains - Custom buses keep application events isolated from the AWS service events on the default bus.
Prefer Scheduler over scheduled rules - EventBridge Scheduler is the newer, more capable replacement. Use it for all new time-based triggers.
Archive and Replay for debugging - Enable archiving so you can replay events after fixing a bug. This is invaluable in production.
Implement idempotency - EventBridge provides at-least-once delivery for most targets. Your consumers should handle duplicate events gracefully.
Use SAM, Terraform, or CDK for infrastructure - Define your rules, patterns, and targets as code. The SAM EventBridgeRule event type is incredibly concise.

Pricing - Practically Free

This might be the most surprising part for people new to EventBridge:

Default event bus: Completely free. Every AWS service event that flows through it costs you nothing.
Custom events: $1.00 per million events. Each 64 KB chunk of a payload counts as one event.
Scheduler: $1.00 per million invocations, with a generous free tier of 14 million invocations per month.
Pipes: $0.40 per million requests after filtering. Filtered-out events are not charged.

For my projects - Account Watcher, Health Check Alerts, Sales Analytics - the EventBridge cost has been essentially $0. The Lambda invocations and SNS messages that EventBridge triggers are also well within free tier for low-to-moderate volume alerting.

Things to Know

A few operational details worth keeping in mind:

At-least-once delivery - EventBridge delivers events at least once for most targets, and AWS service events are delivered with either "best effort" or "durable" guarantees depending on the service. Design your consumers to be idempotent.
5 targets per rule - Need more? Create additional rules with the same pattern.
1 MB event size limit - As of January 2026, EventBridge increased the payload limit from 256 KB to 1 MB. For larger payloads, put the data in S3 and pass the reference in the event. Note that CloudTrail events are still capped at 256 KB.
Sub-second latency - EventBridge routes events quickly, but CloudTrail-sourced events have a 1–5 minute delay from the original API call.
No ordering guarantee - The default bus does not guarantee event order. Handle this in your consumer logic.

The EDA Trade-offs - Being Honest

I believe in being transparent about trade-offs rather than only presenting the benefits. Event-driven architecture is not universally better than request/response - it is a different set of trade-offs.

Debugging is harder. In request/response, you get a stack trace from top to bottom. In EDA, an event flows through multiple decoupled services - when something fails, you are piecing together a distributed trace across Lambda logs, EventBridge metrics, and DLQ messages.

Eventual consistency, not immediate. If a user creates a resource and your event handler has not processed yet, a subsequent read might show stale data. You need to design your UX to tolerate this lag.

Testing is more complex. You cannot just call an endpoint and check the response. You need to verify that events were emitted, rules matched, targets invoked, and downstream side effects occurred.

"Invisible" coupling. EDA is loosely coupled at the code level, but you still have coupling on the event schema. If a producer changes their event format, consumers break silently. Schema Registry helps, but it requires discipline.

The good framing: use request/response when you need synchronous answers and simple debugging. Use EDA when you need loose coupling, scalability, extensibility, and real-time reactivity. Most real systems use both.

Recent Updates Worth Knowing

EventBridge continues to evolve. Here are the most notable changes from the past year:

Scheduler quota monitoring (Feb 2026) - EventBridge Scheduler now emits resource count metrics to CloudWatch, so you can monitor how close you are to quota limits and request increases proactively.
1 MB event payload (Jan 2026) - The event size limit increased from 256 KB to 1 MB, eliminating the need for data chunking or S3 offloading in most cases. Particularly useful for LLM prompts, telemetry signals, and complex ML output payloads.
Enhanced visual rule builder (Nov 2025) - The console now includes an intuitive visual rule builder with a comprehensive event catalog and schema-aware drag-and-drop canvas. Makes it much easier to discover available events from 250+ AWS services and build patterns without referencing individual service docs.
SQS fair queue targets (Nov 2025) - EventBridge can now target SQS fair queues, enabling fairer message distribution across consumer groups in multi-tenant systems.

Wrapping Up

EventBridge sits at the center of event-driven architecture on AWS. The default event bus gives you free, automatic visibility into virtually every mutating API call across your account. Rules and patterns give you precise, content-based routing. Scheduler replaces cron jobs with a fully managed, scalable solution. Pipes eliminate glue code for point-to-point integrations.

I have used it for security monitoring, health check alerting, and data pipeline orchestration - and every time, EventBridge has been the simplest, most cost-effective piece of the architecture. It is the service I reach for first when I need to connect systems, react to changes, or automate workflows. If you are building on AWS and have not explored EventBridge yet, start with the default event bus. Write one rule. Match one event. You will be hooked.

If you are interested in going deeper, I recently presented this material to the AWS Cloud Club at Amity University Chhattisgarh.

Resources

Amazon EventBridge Documentation
Serverless AWS Account Watcher - My blog post on security monitoring with EventBridge
Serverless Site Health Check Notification System - Health check alerting with EventBridge suffix matching
Building a Serverless Sales Analytics Platform - EventBridge event routing + scheduled triggers
Account Watcher GitHub Repo
Health Check GitHub Repo
AWS Cloud Club at Amity University Chhattisgarh - EventBridge Presentation
AWS London Ontario User Group - Meetups, talks, and community for AWS builders in the London, Ontario area
AWS London Ontario User Group YouTube - Recorded talks and presentations

Connect with me on X, Bluesky, LinkedIn, Medium, Dev.to, or the AWS Community. Check out more of my projects at darryl-ruggles.cloud and join the Believe In Serverless community.

Building a Serverless Blogging Platform with AWS and Claude Code

Darryl Ruggles — Wed, 11 Mar 2026 01:10:10 +0000

How I replaced Hashnode with a custom-built, full-featured blogging platform in under 10 hours - with an AI pair programmer doing the heavy lifting.

You can see the new site here -> Darryl's World of Cloud.

Why I Left My Old Platform

For the past couple of years, I hosted my blog on Hashnode with a Vercel-backed custom domain. It worked. Articles rendered fine, the editor was decent, and I didn't have to think about infrastructure. But over time, the limitations started adding up.

Newsletters were the first frustration. Setting up a mailing list, customizing email templates, and scheduling sends all required workarounds or third-party integrations. Email handling felt like an afterthought bolted onto a developer blogging platform. I wanted to send article roundups, curated link collections, and freeform newsletters - all from the same place I write articles. There wasn't an easy way to make all that work there.

Then there was the visibility problem. I am a cloud solutions architect. I work with AWS infrastructure every day. Yet my own blog was a black box. I couldn't see the CDN configuration, couldn't tune caching behavior, couldn't add custom security headers or WAF rules. The hosting details were hidden behind an abstraction I didn't control.

I also wanted to learn. I had been working with several AWS services professionally but hadn't built a complete, production-facing application for my own use that stitched them all together - CloudFront distributions, Lambda function URLs, DynamoDB single-table design, SES email delivery, Cognito authentication. Building my own platform and managing it over time is the best way to learn these services deeply and have something concrete to write about.

So I decided to build it myself. Not just migrate the content - build the entire platform from scratch. The editor, the publishing pipeline, the newsletter system, the analytics, the infrastructure. All of it. The question was whether I could do it fast enough to make the effort worthwhile.

The final push to actually work on this project came when I read a recent article from Ran Isenberg titled Claude Built My Wix Website in 3 Hours - Is SaaS Dead?, where he went through a similar process and I knew it was time to start myself. I have learned so much from all of his work and am grateful for the motivation here.

What I Built - A Modern, Full-Featured Platform

The result is a modern, visually clean blogging platform that loads fast. Every public page is pre-rendered static HTML served from CloudFront edge locations worldwide. No server-side rendering, no client-side hydration, no JavaScript frameworks on the public site - just HTML and CSS delivered from the nearest edge location. Page loads are sub-second globally.

Compared to what I had before, the design is more polished and the performance is noticeably snappier. I went with a clean, card-based homepage layout with a hero section, tag-based navigation, and a reading experience that stays out of your way. The typography, spacing, and code syntax highlighting are all tuned exactly how I want them - because I own every pixel. And if I ever want to change anything it's only a few minutes away from being live.

But the real wins are in the features I could never get working the way I wanted on a hosted platform.

Newsletter Infrastructure Done Right

I haven't sent out any newsletters publicly as of yet but I have done lots of testing and iterating on the format and the infrastructure is now in place. The newsletter system supports three distinct types. Freeform newsletters use the same markdown editor as articles - write whatever you want and send it. Article roundups let you select published articles and automatically generate card-based layouts with excerpts and links. Curated link collections pull in external URLs, auto-fetch their metadata (title, description, image) via OpenGraph tags with Bedrock AI as a fallback, and render them as rich link cards.

All three types support scheduling, email preview before send, and are archived as browsable pages on the public site. Everything runs through Amazon SES - no Mailchimp, no SendGrid, no third-party email service.

Article Preview Before Publishing

Time-limited private URLs let you see exactly how a draft will look on the live site before making it public with the same template, same styles, and same layout as the real articles have. Each preview link expires in a short time, so you can share drafts with reviewers without worrying about stale links lingering.

Comments with Reader Registration

Readers sign up to be able to add article comments. They can verify their email and once approved by me they can comment on posts. The admin side has a full moderation queue - approve, reject, or ban users. No third-party comment widgets are used or injecting tracking scripts into your site.

Email Subscriptions and Notifications

Readers subscribe with double opt-in email verification, manage their preferences, and get notified automatically when new articles are published. One-click unsubscribe in every email. All of it is built on SES and DynamoDB.

Custom Analytics

Built-in page view tracking with zero third-party scripts. I can see referral sources, geographic distribution, device types, and more about who browses articles on the site - with hourly, daily, and monthly granularity. I would have had to pay a lot more per month to see this with Vercel but here it's almost free.

Developer Tooling and Infrastructure

A WYSIWYG markdown editor with source toggle, AI-powered SEO title and description generation via Amazon Bedrock, drag-and-drop image upload to S3 with pre-signed URLs, automated backups with cross-account disaster recovery, and CI/CD with GitHub Actions deploying on every push to main.

The platform runs on 16+ AWS services including CloudFront, Lambda, DynamoDB, S3, SES, Cognito, and more with automated backups and lots of security controls. My favorite Infrastructure as Code (IaC) tool Terraform is used. My monthly cost is in the single dollars due to using serverless/managed services with pay-per-use pricing.

The AI-Assisted Development Experience

I have worked in the AI/ML space for a number of years now. I have seen lots of promising ideas but haven't been completely sold on most of the tools from this space being super useful or becoming real difference makers in my day-to-day life. I have been experimenting with AI-based coding tools for over a year now with lots of hope. I was skeptical and not super impressed with what they could do a year or more ago. I had tried GitHub Copilot back then and found the autocomplete suggestions useful but hardly earth shattering. I experimented with Cursor, Kiro, OpenCode, and a few others. The early outputs were rough - code that looked plausible but missed edge cases, made incorrect API calls, or ignored the architectural context of the project.

But the improvement over the last number of months has been dramatic. I settled on Claude Code with Opus 4.5/4.6 as my primary development tool, and the workflow it enables is genuinely different from anything I have experienced in 25+ years of software development.

The Workflow

The pattern is simple: spend a few minutes describing what you want in English, and within minutes the feature is working. Not hours of scaffolding and boilerplate - minutes. Need a newsletter scheduling system with EventBridge? Describe the requirements, review the generated Terraform and Lambda code, test it, push it. Need a comment moderation queue with Cognito integration? Same thing. Of course there are caveats and things to seriously keep an eye on - more on those below.

The key is that you are not just generating code snippets. Claude Code understands the full project context - the file structure, the existing patterns, the infrastructure configuration, the naming conventions. When I asked it to add newsletter support, it knew where the API routes lived, how the DynamoDB tables were structured, what the Terraform modules looked like, and how the admin SPA was organized. It generated coordinated changes across a dozen files that all fit together coherently.

This is what sets it apart from autocomplete-style tools. The context window spans the entire project. It reads your Terraform state, your Lambda handlers, your React components, and your CLAUDE.md instructions. The result is code that feels like it was written by someone who has been on the project for months, not a tool that just saw a single file.

MCP Servers Extended the Capabilities

One of the features that made Claude Code especially effective for most projects is Model Context Protocol (MCP) servers. These are plugins that give the AI access to specialized tools and documentation right in the development flow. Some of the ones I use a lot are described here.

The Terraform MCP server provides instant access to AWS provider documentation, Checkov security scanning, and module search. Instead of tab-switching to the Terraform docs, Claude Code looks up resource attributes, checks for security misconfigurations, and finds community modules - all inline during development.

The AWS Knowledge MCP server gives inline access to AWS documentation. When I needed to understand CloudFront OAC signing behavior or SES configuration set options, the documentation was available without leaving the editor.

The AWS Serverless MCP server provides Lambda patterns, SAM guidance, and event source mapping configuration. Useful for getting the Lambda function URL streaming configuration right and understanding best practices.

The AWS Diagram MCP server generates architecture diagrams directly from code, producing decent diagrams with little effort. I have to say the diagram server still has room for improvement as there are too many overlapping lines and it tends to generate very vertical diagrams.

The Timeline

Looking at the git history, the build of my new site happened by working on it for a few hours a day over five calendar days:

Day 1: Core platform from scratch plus Hashnode migration - a couple of hours to get articles rendering, the admin editor working, and all existing content imported
Day 2: Comments, email subscriptions, CloudWatch monitoring, Slack alerts, CI/CD pipeline, and automated testing
Day 3: Analytics refinements, backup setup, custom domain setup
Day 4: Full newsletter system - three newsletter types, scheduling, preview, archive pages
Day 5: Article preview with random URLs, firewall tuning, polish and bug fixes

The final count: 70 commits, 11+ major features in five days. Total hands-on development time: under 10 hours for a full-featured, production-quality blogging platform. Monthly cost: well under $10 - serverless pay-per-use means you only pay for what you use, and a personal blog's traffic keeps costs negligible.

Honest Assessment - Not Magic, But Powerful

I want to be clear about something: Claude Code makes mistakes. Often. It's not a magic box that produces perfect code on the first try. It's not a tool to hand to your New Grad developer and expect them to produce quality code that's secure and meets the design specs. You still need experienced developers to guide the process and push back when things are going down the wrong path.

It will occasionally hallucinate API parameters that do not exist. It will sometimes choose an architectural pattern that's technically correct but wrong for the specific context. It will miss edge cases that an experienced developer would catch immediately. It will sometimes try to solve a problem by adding complexity when the right answer is to simplify. It will use the wrong versions of tools and many other bad things.

But here is the thing - an experienced developer who knows the services, languages, and patterns can catch these mistakes and push back. The AI responds well to correction. You say "that parameter does not exist on this resource," and it fixes it. You say "this should be async, not synchronous," and it restructures the code. The collaboration works because the human brings judgment and the AI brings speed.

Where Human Judgment Was Critical

Several times during the build, my experience was the difference between a working system and a broken one.

Security architecture decisions required human judgment at every turn. IAM least privilege policies, backup tool hardening, encryption key rotation, SES sending authorization - these are areas where "works on my machine" is not good enough. Each security decision needed careful review against best practices.

Synchronous vs. asynchronous patterns were a recurring design choice. When should code work synchronously vs. asynchronously? When should newsletter sends be immediate vs. scheduled through EventBridge? These are architectural decisions that depend on user experience requirements and operational constraints, not just technical feasibility.

The Review Layer

Code generated by AI must be scrutinized with the same rigor as code written by a junior developer - maybe more. For this project, every pull request went through GitHub Copilot and Amazon Q Developer code reviews in addition to my own review. This layered approach caught issues that any single reviewer might miss.

For anything production-facing, especially with security implications, there's no substitute for thorough review, penetration testing, and proper validation. AI-assisted development is fast, but speed without scrutiny is a liability.

Key Takeaways

AI-assisted development is real and practical today. Not for every task, and not without supervision, but the productivity gains are substantial for developers who know their domain.
MCP servers are a force multiplier. Having documentation, security scanning, and infrastructure tools available inline eliminates context switching and keeps you in flow.
Serverless is ideal for personal projects. Pay-per-use pricing means your blog costs pennies when nobody is reading it and scales automatically when a post hits the front page.
The human in the loop matters more than the AI. Domain expertise, security awareness, and architectural judgment are what turn AI-generated code into production-quality software.
Build things you will actually use. The best way to learn anything is to build something real that you will maintain and improve over time.

Closing Thoughts

I was lucky enough to be at AWS re:Invent 2025 in Las Vegas and spoke with many people on the topic of AI technology and the impact on our industry. There is a lot to be concerned about but there is so much promise. I was in the room for the final keynote from Werner Vogels and was truly inspired by his talk about the The Dawn of the Renaissance Developer.

I haven't had near this much fun coding in years. There is something deeply satisfying about describing a feature in plain English, watching it materialize in code, reviewing and refining it, and then seeing it live on your own infrastructure minutes later. The feedback loop is extraordinarily tight.

My backlog of features to build grows faster than I can ship them. Dark mode, reading time estimates, related articles, full-text search, RSS improvements, an about page redesign - the list keeps growing. Every time I use the platform to write a post, I notice something I want to improve. That's a good sign. It means the platform is useful enough to invest in, and I care enough about it to keep iterating.

If you have been putting off a side project because you thought it would take too long, the barrier has never been lower. A working prototype of almost anything is now a weekend away. The tools are that good - if you bring the expertise to guide them.

This platform is live and serving the very post you are reading right now. If something looks off, that's on me - and probably on my backlog already.

IMPORTANT: Remember to always validate security practices for anything public-facing - especially anything for production use. AI-assisted development makes it easy to move fast, and moving fast without security review is how breaches happen. Take the time to review IAM policies, test authentication flows, scan for vulnerabilities, and audit your infrastructure. The speed gains from AI should buy you more time for security - not less.

Connect with me on X, Bluesky, LinkedIn, Medium, Dev.to, GitHub, or the AWS Community. Check out more of my projects at darryl-ruggles.cloud and join the Believe In Serverless community.

Bi-directional Voice-Controlled Recipe Assistant with Nova Sonic 2

Darryl Ruggles — Sun, 08 Mar 2026 01:19:36 +0000

What if your recipe assistant could talk back?

I have been building a serverless Family Recipe Assistant that searches my family's recipe collection, calculates nutrition from USDA data, and handles multi-turn conversations through a web UI. It works well. But every time I am in the kitchen with flour on my hands, reaching for my phone to type "how long do I bake the banana bread?" feels wrong.

I wanted to just ask.

The text-based assistant already had a "cooking mode" that read recipes aloud using Amazon Polly. But listening to a long recipe read start-to-finish by a TTS voice is surprisingly tedious - you cannot ask it to slow down, skip ahead, or clarify a step without going back to the screen and typing. What I really wanted was a conversation: "What is the next step?" or "How much butter was that again?" while my hands are covered in dough.

Amazon Nova Sonic v2 launched recently with sub-700ms speech-to-speech latency and a 1M token context window. The Strands Agents SDK added experimental support for bidirectional streaming through a BidiAgent class that wraps the WebSocket complexity into something remarkably simple. I decided to wire the two together and add my existing recipe tools - search, timers, nutrition lookup, and unit conversion - to build a voice-controlled kitchen assistant.

This post walks through how I built it, what worked, and what surprised me. It works great on my laptop and even on my Android phone and IPad. The complete code is on GitHub: github.com/RDarrylR/serverless-family-recipes-bidirectional-nova-sonic.

What is Bidirectional Streaming?

Traditional voice assistants follow a rigid pattern: listen, transcribe, think, generate text, synthesize speech, play audio. Each step waits for the previous one to finish. The result is a noticeable pause between your question and the response.

Bidirectional streaming changes this. The agent maintains a persistent WebSocket connection where audio flows in both directions simultaneously. Nova Sonic v2 handles speech understanding and generation in a single model - there is no separate transcription or TTS step. The practical effect is that the agent starts responding while you are still finishing your sentence, and you can interrupt it mid-answer to course-correct.

If you have used a native WebSocket implementation before, you know this means managing connection lifecycle, audio encoding, event multiplexing, and error recovery. That is roughly 150 lines of plumbing code before you write any business logic.

Strands BidiAgent reduces this to about 20 lines.

Architecture

The system has two layers. A React frontend in the browser captures microphone audio using the Web Audio API (with built-in echo cancellation - no headset needed) and sends it over a WebSocket to a FastAPI server. The server runs a Strands BidiAgent that forwards audio to Nova Sonic v2, dispatches tool calls, and streams response audio back to the browser for playback.

The components:

BidiAgent - The Strands orchestrator. Manages the streaming session, routes audio to the model, and dispatches tool calls.
BidiNovaSonicModel - The model adapter. Establishes a bidirectional WebSocket to Bedrock and handles the Nova Sonic protocol.
Kitchen tools - Four @tool decorated Python functions that the agent can call during conversation without interrupting the audio stream.
Bedrock Knowledge Base - My existing recipe collection, indexed with Titan Embed V2 and stored in S3 Vectors.
USDA FoodData Central - Public API for nutrition data.

The key insight is that tool execution happens concurrently with audio streaming. When you ask "find me a pasta recipe," the agent calls the search_recipes tool while continuing to listen for follow-up input. No blocking, no silence gap.

Prerequisites

Before you start, you will need:

Python 3.13+ (3.12 minimum for Nova Sonic)
Node.js 18+ (for the Vite frontend dev server)
An AWS account with Bedrock model access enabled for Nova Sonic v2
PortAudio system library (brew install portaudio on macOS) - required as a transitive dependency of the Strands SDK
uv for Python dependency management

git clone https://github.com/RDarrylR/serverless-family-recipes-bidirectional-nova-sonic.git
cd serverless-family-recipes-bidirectional-nova-sonic
uv sync
make install-frontend

Building the Agent

The server is a FastAPI WebSocket endpoint that bridges browser audio to BidiAgent. The key insight is that BidiAgent accepts plain callables for I/O. At its simplest, you can pass ws.receive_json and ws.send_json directly. In practice, I wrap them with thin functions for message size limits and cost tracking, but the pattern stays the same - no custom I/O classes needed.

from fastapi import FastAPI, WebSocket, WebSocketDisconnect
from strands.experimental.bidi import BidiAgent
from strands.experimental.bidi.models import BidiNovaSonicModel
from strands.experimental.bidi.tools import stop_conversation

from config import AWS_REGION, NOVA_SONIC_VOICE, SYSTEM_PROMPT
from tools import search_recipes, set_timer, nutrition_lookup, convert_units

app = FastAPI()

sonic_model = BidiNovaSonicModel(
    region_name=AWS_REGION,
    provider_config={
        "audio": {
            "input_rate": 16000,
            "output_rate": 24000,
            "voice": NOVA_SONIC_VOICE,
        }
    },
)

@app.websocket("/ws")
async def websocket_endpoint(ws: WebSocket):
    agent = BidiAgent(
        model=sonic_model,
        tools=[search_recipes, set_timer, nutrition_lookup, convert_units, stop_conversation],
        system_prompt=SYSTEM_PROMPT,
    )

    try:
        await ws.accept()
        await agent.run(
            inputs=[ws.receive_json],
            outputs=[ws.send_json],
        )
    except WebSocketDisconnect:
        logger.info("Client disconnected")
    finally:
        await agent.stop()

That is the core of the agent. A new BidiAgent is created per connection - no shared state between sessions. The BidiNovaSonicModel handles the WebSocket connection to Bedrock. The agent.run() method ties them together and runs until the user says "stop" (triggering the stop_conversation tool) or the browser disconnects.

The provider_config sets the audio sample rates (16kHz input from the browser, 24kHz output from Nova Sonic) and the voice. Nova Sonic v2 supports several voices - I went with "tiffany" for a natural conversational tone, but you can also use "amy" or "puck".

Important: Voice IDs must be lowercase. Passing "Tiffany" instead of "tiffany" results in a ValidationException: Received invalid id error that silently kills the session. This is not documented anywhere obvious - I discovered it by enabling debug logging on the Strands Nova Sonic model class.

Why `stop_conversation`?

The stop_conversation tool is a built-in Strands tool that gracefully shuts down the streaming session. Without it, the only way to stop the agent is Ctrl+C, which does not cleanly close the WebSocket. The system prompt tells the agent to use this tool when users say "goodbye" or "stop."

Adding Kitchen Tools

Tools work identically to standard Strands agents. You decorate a function with @tool, add a docstring that describes when to use it, and pass it to the agent. The model decides when to call tools based on the conversation.

Recipe Search

This tool connects to my existing Bedrock Knowledge Base, which contains my family's recipe collection indexed with Titan Embed V2.

import boto3
from strands import tool
from config import AWS_REGION, BEDROCK_KB_ID

bedrock_agent_runtime = boto3.client("bedrock-agent-runtime", region_name=AWS_REGION)

@tool
def search_recipes(query: str) -> str:
    """Search the recipe knowledge base for recipes matching the query.

    Use this tool whenever a user asks about recipes, ingredients, or cooking methods.

    Args:
        query: Natural language search query about recipes
    """
    response = bedrock_agent_runtime.retrieve(
        knowledgeBaseId=BEDROCK_KB_ID,
        retrievalQuery={"text": query},
        retrievalConfiguration={
            "vectorSearchConfiguration": {"numberOfResults": 3}
        },
    )

    results = response.get("retrievalResults", [])
    if not results:
        return "No recipes found matching that query."

    # Group chunks by source file so multi-chunk recipes are reassembled
    source_chunks = {}
    for r in results:
        score = r.get("score", 0)
        if score < 0.3:
            continue
        text = r.get("content", {}).get("text", "").strip()
        source = r.get("location", {}).get("s3Location", {}).get("uri", "")
        source_name = source.split("/")[-1].replace(".md", "") if source else "unknown"

        if source not in source_chunks:
            source_chunks[source] = {"name": source_name, "score": score, "texts": []}
        source_chunks[source]["texts"].append(text)

    chunks = []
    for entry in source_chunks.values():
        merged = "\n\n".join(entry["texts"])
        chunks.append(f"Recipe: {entry['name']} (score: {entry['score']:.2f})\n{merged}")

    return f"Found {len(source_chunks)} recipe(s):\n\n" + "\n\n".join(chunks)

The chunk-merging logic is important. Bedrock KB returns chunks, not complete documents. A long recipe might be split across multiple chunks. By grouping chunks that share the same S3 source URI, the tool reassembles the full recipe text before handing it to the model.

The score threshold of 0.3 filters out weak matches. Without it, the agent would sometimes confidently describe a recipe that was only tangentially related to the query.

Cooking Timer

import asyncio
from strands import tool

@tool
def set_timer(minutes: int, label: str = "cooking") -> str:
    """Set a cooking timer for the specified number of minutes.

    Args:
        minutes: Number of minutes for the timer
        label: A short description of what the timer is for
    """
    if minutes <= 0:
        return "Timer must be at least 1 minute."

    loop = asyncio.get_running_loop()
    loop.create_task(_timer_callback(minutes, label))
    return f"Timer set: {label} for {minutes} minutes. I'll let you know when it's done."

async def _timer_callback(minutes: int, label: str):
    await asyncio.sleep(minutes * 60)
    print(f"\n** TIMER DONE: {label} ({minutes} minutes) **\n")

The timer runs as an asyncio background task. When it expires, it prints to the console. In a production version, this would trigger a spoken notification through the audio output - but since Nova Sonic sessions have an 8-minute limit, long timers outlive the session anyway. I will address this in the trade-offs section.

Nutrition Lookup

import requests
from strands import tool

@tool
def nutrition_lookup(food_item: str) -> str:
    """Look up nutrition information for a food item using USDA FoodData Central.

    Args:
        food_item: The food item to look up, e.g. "chicken breast" or "brown rice"
    """
    response = requests.get(
        "https://api.nal.usda.gov/fdc/v1/foods/search",
        params={"query": food_item, "pageSize": 1, "api_key": USDA_API_KEY},
        timeout=10,
    )

    foods = response.json().get("foods", [])
    if not foods:
        return f"No nutrition data found for '{food_item}'."

    food = foods[0]
    # Extract key nutrients from the USDA response
    nutrients = {}
    for n in food.get("foodNutrients", []):
        name = n.get("nutrientName", "")
        if name in TARGET_NUTRIENTS:
            nutrients[TARGET_NUTRIENTS[name]] = f"{n['value']} {n.get('unitName', '')}"

    return f"Nutrition for {food['description']} (per 100g):\n{json.dumps(nutrients, indent=2)}"

The USDA FoodData Central API is free and does not require registration for the DEMO_KEY. For production use, you should get a proper API key at https://fdc.nal.usda.gov/api-key-signup.

Unit Converter

from strands import tool

VOLUME_TO_ML = {
    "cup": 236.588, "tbsp": 14.787, "tsp": 4.929,
    "ml": 1.0, "liter": 1000.0, "fl oz": 29.5735,
}

WEIGHT_TO_G = {
    "g": 1.0, "kg": 1000.0, "oz": 28.3495, "lb": 453.592,
}

@tool
def convert_units(amount: float, from_unit: str, to_unit: str) -> str:
    """Convert between cooking measurement units.

    Args:
        amount: The numeric amount to convert
        from_unit: The unit to convert from
        to_unit: The unit to convert to
    """
    f, t = from_unit.lower(), to_unit.lower()

    # Temperature
    if f in ("f", "fahrenheit") and t in ("c", "celsius"):
        return f"{amount} F = {(amount - 32) * 5 / 9:.1f} C"

    # Volume
    if f in VOLUME_TO_ML and t in VOLUME_TO_ML:
        result = amount * VOLUME_TO_ML[f] / VOLUME_TO_ML[t]
        return f"{amount} {from_unit} = {result:.2f} {to_unit}"

    # Weight
    if f in WEIGHT_TO_G and t in WEIGHT_TO_G:
        result = amount * WEIGHT_TO_G[f] / WEIGHT_TO_G[t]
        return f"{amount} {from_unit} = {result:.2f} {to_unit}"

    return f"Cannot convert {from_unit} to {to_unit}"

This tool is intentionally simple. It handles volume, weight, and temperature - the three conversion types that come up most often while cooking. When someone asks to convert volume to weight (e.g., "how many grams is a cup of flour?"), the tool explains that the answer depends on the ingredient's density rather than guessing.

The Magic: Interruptions

The feature that makes bidirectional streaming feel genuinely different from a traditional voice assistant is interruption handling. You can talk over the agent mid-sentence, and it will stop, process your new input, and respond to that instead.

For example:

You: "Set a timer for 10 minutes"
Agent: "Timer set for 10 min-"
You: "Actually, make that 15"
Agent: "Got it, I have updated the timer to 15 minutes."

This works because the agent is always listening, even while generating output audio. When it detects that the user has started speaking, it cancels the current output and processes the interruption as new input. Strands handles the mechanics - clearing the output buffer, signaling the model to stop generating, and routing the new audio through the same pipeline.

From a developer perspective, you do not need to write any interruption logic. It just works.

Infrastructure

For local development, you only need IAM permissions for Bedrock access. The Terraform configuration creates an IAM role with least-privilege permissions:

resource "aws_iam_role_policy" "bedrock_access" {
  name = "bedrock-access"
  role = aws_iam_role.voice_agent.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "NovaSonicAccess"
        Effect = "Allow"
        Action = [
          "bedrock:InvokeModel",
          "bedrock:InvokeModelWithResponseStream",
          "bedrock:InvokeModelWithBidirectionalStream",
        ]
        Resource = [
          "arn:aws:bedrock:us-east-1::foundation-model/amazon.nova-2-sonic-v1*",
        ]
      },
      {
        Sid    = "KnowledgeBaseAccess"
        Effect = "Allow"
        Action = ["bedrock:Retrieve"]
        Resource = [
          "arn:aws:bedrock:us-east-1:ACCOUNT:knowledge-base/YOUR_KB_ID",
        ]
      }
    ]
  })
}

Important: Model ID mismatch. The Bedrock foundation model ARN for Nova Sonic v2 is amazon.nova-2-sonic-v1:0 - not amazon.nova-sonic-v2 as you might guess from the marketing name. If your IAM policy uses the wrong model ID pattern, the BidiAgent will establish a connection to Nova Sonic and appear to be working, but the model silently fails to process audio. No error is returned - the agent simply produces no output. This cost me several hours of debugging. Check your IAM policy resource ARNs if the agent connects but never responds.

If you are just experimenting, your default AWS CLI credentials with Bedrock access will work without the Terraform setup. The IAM role becomes useful when you want to follow least-privilege principles or deploy to AgentCore.

Trade-offs and Gotchas

The 8-Minute Session Limit

Nova Sonic v2 sessions time out after 8 minutes. For a kitchen assistant, this is a real constraint - you might be cooking for an hour and want to check on timers, look up steps, or ask follow-up questions throughout.

The workaround is session rotation: detect when a session is about to expire, save context, and start a new session. Strands does not handle this automatically yet. For the demo, 8 minutes is enough to show the concept, but a production kitchen assistant would need this solved.

For comparison, OpenAI Realtime supports 60-minute sessions, though at higher cost and without the tight AWS integration.

Echo Cancellation

Nova Sonic does not handle echo cancellation server-side - it expects clean audio at the input. If the agent's output audio gets picked up by the microphone, you get a feedback loop where the agent starts responding to itself.

The browser handles this transparently. The Web Audio API provides built-in acoustic echo cancellation via getUserMedia({ echoCancellation: true }) - the same mechanism that makes WebRTC video calls work without headsets. The browser subtracts the known output signal from the microphone input, so you can use laptop speakers and mic directly. No headset required.

If you port to other platforms, the same principle applies: iOS (AVAudioSession) and Android audio frameworks include system-level AEC, and smart speakers use hardware beamforming. The key point is that echo cancellation is an I/O concern, not a model concern - solve it at the client layer.

Observability Gap: No Invocation Logging

Bedrock invocation logging does not cover the InvokeModelWithBidirectionalStream API that Nova Sonic uses. If you have invocation logging enabled and expect to see Nova Sonic calls in CloudWatch, you will not find them. The API calls appear in CloudTrail as events, but the actual conversation content (audio, transcripts, tool calls) is not captured.

For cost tracking, Nova Sonic is priced by audio duration (input and output seconds) rather than text tokens. AWS Cost Explorer filtered by Bedrock service is the most reliable way to monitor spend.

Timer Notifications Outlive Sessions

Because of the 8-minute session limit, a timer set for 20 minutes will expire after the Nova Sonic session has ended. The timer still fires (it is a local asyncio task), but it can only print to the console - it cannot speak through the agent. A production version would need a separate notification channel (push notification, separate audio alert, etc.).

Sample Conversation

Here is a typical session showing the back-and-forth between the user and the assistant, including tool calls. The transcripts below are captured from the browser UI - Nova Sonic generates both the audio and the text transcript simultaneously.

You: "What pasta recipes do you have?"

[search_recipes tool called]

Assistant: "I found two pasta recipes in the collection. The first is a classic spaghetti carbonara - it uses eggs, pecorino romano, guanciale, and black pepper. No cream, as the traditional version gets its
richness from the egg and cheese sauce. The second is a baked ziti with Italian sausage that serves about eight people. Would you like the full details on either one?"

You: "How many calories in a cup of pasta?"

[nutrition_lookup tool called]

Assistant: "One cup of cooked spaghetti has about 220 calories, with 43 grams of carbohydrates and 8 grams of protein. That is for plain pasta without sauce."

You: "OK let's make the carbonara. Set a timer for 12 minutes for the pasta."

[set_timer tool called]

Assistant: "Timer set for 12 minutes for the pasta. While the pasta cooks, let me walk you through the sauce. You will want to whisk together three egg yolks and one whole egg with about a cup of grated
pecorino-"

You: "Wait, how much pecorino was that?"

Assistant: "One cup of grated pecorino romano. You can mix in some parmesan too - about half and half works well."

You: "What is that in grams?"

[convert_units tool called]

Assistant: "One cup of grated pecorino is roughly 100 grams, and one cup of parmesan is about 80 grams. So for a half-and-half mix, that would be 50 grams pecorino and 40 grams parmesan."

Notice how the assistant handles interruptions naturally (cutting off mid-sentence when asked about pecorino), chains tool calls in context (nutrition lookup followed by unit conversion), and maintains conversational flow throughout.

Try It Yourself

The complete code is on GitHub: github.com/RDarrylR/serverless-family-recipes-bidirectional-nova-sonic

git clone https://github.com/RDarrylR/serverless-family-recipes-bidirectional-nova-sonic.git
cd serverless-family-recipes-bidirectional-nova-sonic
uv sync
make install-frontend

# Terminal 1: WebSocket server
make serve

# Terminal 2: Vite dev server
make serve-frontend

Open http://localhost:5173, click the microphone, and start talking. No headset needed - the browser handles echo cancellation.

You will need Bedrock model access enabled for Nova Sonic v2 in your AWS account. If you are using the recipe search tool, you will also need a Bedrock Knowledge Base with your recipes indexed.

The Audio Pacing Problem

This was the most surprising issue I hit. After wiring everything up, the assistant's voice worked - but after tool calls (like searching recipes or looking up nutrition), all the speech played back bunched together with no natural pauses between sentences. Short responses sounded fine. Long responses after tool calls sounded like someone hit 2x speed.

The root cause: Nova Sonic generates audio faster than real-time after receiving a tool result. When the model has the full tool output text available at once, it produces all the speech audio in a burst rather than at natural speech pace. With WebSocket forwarding, there is no backpressure - the server receives hundreds of audio chunks in seconds and immediately forwards them all to the browser.

The fix is a sufficiently large client-side ring buffer. The browser's AudioWorklet maintains a ring buffer sized for 60 seconds of audio at 24kHz. Audio arrives in bursts from the server, gets queued in the ring buffer, and plays back at the correct hardware rate. The AudioWorklet's process() callback naturally paces playback - it pulls exactly 128 samples per callback at the AudioContext sample rate, regardless of how fast data arrives.

class AudioPlayerProcessor extends AudioWorkletProcessor {
  constructor() {
    super();
    // 60 seconds at 24kHz - handles faster-than-realtime bursts
    this._bufferSize = 24000 * 60;
    this._buffer = new Float32Array(this._bufferSize);
    this._writePos = 0;
    this._readPos = 0;

    this.port.onmessage = (event) => {
      if (event.data.type === 'audio') {
        this._enqueue(event.data.samples);
      } else if (event.data.type === 'barge-in') {
        this._readPos = this._writePos; // Clear instantly
      }
    };
  }

  process(inputs, outputs) {
    const output = outputs[0][0];
    for (let i = 0; i < output.length; i++) {
      if (this._readPos < this._writePos) {
        output[i] = this._buffer[this._readPos % this._bufferSize];
        this._readPos++;
      } else {
        output[i] = 0; // Silence when buffer is empty
      }
    }
    return true;
  }
}

The initial version used a 5-second buffer, which overflowed on longer responses - the write pointer wrapped around and overwrote unplayed samples, causing garbled audio. 60 seconds handles even the longest recipe descriptions comfortably.

For interruptions (barge-in), the server sends a bidi_interruption event and the browser sets readPos = writePos, which instantly silences playback. New audio from the updated response then fills the buffer from that point.

Two AudioContexts

The browser uses two separate AudioContext instances: one at 16kHz for capture and one at 24kHz for playback. These cannot share a context because Web Audio requires a single sample rate per context, and Nova Sonic's input and output rates differ.

Capture uses ScriptProcessorNode (deprecated but universal) to grab Float32 audio frames, convert them to PCM16, base64-encode, and send over WebSocket. Playback uses an AudioWorkletNode with a ring buffer for gapless streaming - the audio thread pulls samples continuously, and barge-in clears the buffer instantly when the server signals an interruption.

Deploying to AgentCore

For local development, this runs as two processes: a FastAPI server and a Vite dev server with a WebSocket proxy. For production, I deployed the voice agent to AWS Bedrock AgentCore Runtime as a Docker container.

The deployed architecture:

Frontend - React SPA on S3 + CloudFront, same pattern as the text-based recipe assistant
Auth - Cognito User Pool (email/password) + Identity Pool. The browser exchanges a Cognito JWT for temporary AWS credentials, then uses those to SigV4-sign a WebSocket URL directly to AgentCore. No API Gateway or Lambda in the WebSocket path.
Agent Runtime - The FastAPI server runs as an ARM64 container in AgentCore. The same server.py that runs locally deploys without changes - AgentCore handles health checks (/ping), scaling, and WebSocket proxying.
Dual mode - The frontend auto-detects the deployment mode. If VITE_AGENT_RUNTIME_ARN is set, it uses SigV4-signed WebSocket to AgentCore. If not, it connects to the local Vite proxy. No code changes needed to switch between local and deployed.

The Dockerfile

The container packages the same FastAPI server that runs locally. ARM64 is required by AgentCore Runtime.

FROM --platform=linux/arm64 public.ecr.aws/docker/library/python:3.13-slim

# PyAudio is a transitive dependency of strands-agents[bidi-all]
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    libasound-dev libportaudio2 libportaudiocpp0 \
    portaudio19-dev gcc python3-dev && \
    apt-get clean && rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY config.py server.py ./
COPY tools/ ./tools/

RUN useradd -m -u 1000 appuser && chown -R appuser:appuser /app
USER appuser

ENV CONTAINER_ENV=true
ENV PYTHONUNBUFFERED=1
EXPOSE 8080

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
    CMD ["python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/ping').read()"]

CMD ["opentelemetry-instrument", "uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8080"]

The CONTAINER_ENV flag tells server.py to bind to 0.0.0.0 instead of 127.0.0.1. The opentelemetry-instrument wrapper is required for AgentCore to capture container logs in CloudWatch - without it, stdout and stderr are silently dropped. The aws-opentelemetry-distro package in requirements.txt provides this wrapper. The health check hits the /ping endpoint that AgentCore polls every few seconds.

Terraform: AgentCore IAM Role

The container needs an IAM role that AgentCore assumes on its behalf. This is where the model ID gotcha matters most - get the ARN wrong and Nova Sonic silently ignores audio.

resource "aws_iam_role" "agentcore" {
  name = "${var.project_name}-${var.environment}-agentcore"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "bedrock-agentcore.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy" "agentcore" {
  name = "agentcore-permissions"
  role = aws_iam_role.agentcore.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "NovaSonicAccess"
        Effect = "Allow"
        Action = [
          "bedrock:InvokeModel",
          "bedrock:InvokeModelWithResponseStream",
          "bedrock:InvokeModelWithBidirectionalStream",
        ]
        # NOTE: The model ID is amazon.nova-2-sonic-v1, NOT amazon.nova-sonic-v2
        Resource = [
          "arn:aws:bedrock:${var.aws_region}::foundation-model/amazon.nova-2-sonic-v1*"
        ]
      },
      {
        Sid      = "KnowledgeBaseAccess"
        Effect   = "Allow"
        Action   = ["bedrock:Retrieve"]
        Resource = [
          "arn:aws:bedrock:${var.aws_region}:${data.aws_caller_identity.current.account_id}:knowledge-base/${var.knowledge_base_id}"
        ]
      },
      {
        Sid    = "ECRImageAccess"
        Effect = "Allow"
        Action = ["ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer"]
        Resource = [
          "arn:aws:ecr:${var.aws_region}:${data.aws_caller_identity.current.account_id}:repository/*"
        ]
      },
      {
        Sid      = "ECRTokenAccess"
        Effect   = "Allow"
        Action   = ["ecr:GetAuthorizationToken"]
        Resource = "*"
      },
      # CloudWatch, X-Ray, and WorkloadIdentity statements omitted for brevity
    ]
  })
}

The bedrock-agentcore.amazonaws.com service principal in the trust policy allows AgentCore to assume this role when running the container. The policy grants exactly what the agent needs: invoke Nova Sonic, retrieve from the Knowledge Base, and pull the container image from ECR.

Terraform: Cognito Authentication

The browser needs AWS credentials to sign WebSocket requests. Cognito provides a two-step flow: User Pool for authentication, Identity Pool for credential exchange.

# User Pool - email/password authentication
resource "aws_cognito_user_pool" "main" {
  name                     = "${var.project_name}-${var.environment}-users"
  username_attributes      = ["email"]
  auto_verified_attributes = ["email"]

  admin_create_user_config {
    allow_admin_create_user_only = true  # No self-signup for the demo
  }

  password_policy {
    minimum_length    = 8
    require_lowercase = true
    require_uppercase = true
    require_numbers   = true
  }
}

# Public client for the SPA - no client secret
resource "aws_cognito_user_pool_client" "web" {
  name            = "${var.project_name}-${var.environment}-web-client"
  user_pool_id    = aws_cognito_user_pool.main.id
  generate_secret = false

  explicit_auth_flows = [
    "ALLOW_USER_SRP_AUTH",
    "ALLOW_REFRESH_TOKEN_AUTH",
  ]
}

# Identity Pool - exchange JWT for temporary AWS credentials
resource "aws_cognito_identity_pool" "main" {
  identity_pool_name               = "${var.project_name}-${var.environment}-identity"
  allow_unauthenticated_identities = false

  cognito_identity_providers {
    client_id               = aws_cognito_user_pool_client.web.id
    provider_name           = aws_cognito_user_pool.main.endpoint
    server_side_token_check = false
  }
}

# IAM role that authenticated users assume
resource "aws_iam_role" "cognito_authenticated" {
  name = "${var.project_name}-${var.environment}-cognito-auth-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Federated = "cognito-identity.amazonaws.com" }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "cognito-identity.amazonaws.com:aud" = aws_cognito_identity_pool.main.id
        }
        "ForAnyValue:StringLike" = {
          "cognito-identity.amazonaws.com:amr" = "authenticated"
        }
      }
    }]
  })
}

# Grant authenticated users permission to invoke the agent via WebSocket
resource "aws_iam_role_policy" "agentcore_invoke" {
  count = var.agent_runtime_arn != "" ? 1 : 0
  name  = "agentcore-invoke"
  role  = aws_iam_role.cognito_authenticated.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Action = [
        "bedrock-agentcore:InvokeAgentRuntime",
        "bedrock-agentcore:InvokeAgentRuntimeWithWebSocketStream",
        "bedrock-agentcore:InvokeRuntime",
        "bedrock-agentcore:InvokeRuntimeWithResponseStream",
      ]
      Resource = "*"
    }]
  })
}

The count = var.agent_runtime_arn != "" ? 1 : 0 handles a chicken-and-egg problem: Terraform creates the Cognito resources first, then you create the AgentCore runtime via CLI (there is no Terraform provider for AgentCore yet), then re-run Terraform with the runtime ARN to attach the invoke policy.

IAM resource scoping gotcha: You might expect to scope Resource to your specific runtime ARN instead of "*". I tried this and it breaks WebSocket connections with a silent failure - the browser gets a connection refused with no useful error. The InvokeAgentRuntimeWithWebSocketStream action evaluates against a resource ARN that includes session and qualifier components, not just the runtime ARN. The official AWS sample uses Resource: "*" for the same reason. Until AWS documents the exact resource ARN format for WebSocket invocations, "*" is the only option that works. The blast radius is limited since these are Cognito-scoped temporary credentials that can only call bedrock-agentcore:Invoke* actions.

SigV4 WebSocket Presigning

This is the most unusual piece of the deployment. The browser cannot use a simple WebSocket URL - it needs to SigV4-sign the request using temporary AWS credentials from Cognito. The signing process constructs the URL, signs it with the AWS Signature Version 4 algorithm, and appends the signature as query parameters.

import { getAWSCredentials } from './aws-credentials.js';
import { Sha256 } from '@aws-crypto/sha256-js';
import { SignatureV4 } from '@aws-sdk/signature-v4';
import { HttpRequest } from '@smithy/protocol-http';

export async function getPresignedWebSocketUrl(agentRuntimeArn, sessionId) {
  const region = import.meta.env.VITE_REGION || 'us-east-1';
  const credentials = await getAWSCredentials();

  // Build the AgentCore WebSocket URL
  const encodedArn = encodeURIComponent(agentRuntimeArn);
  const url = new URL(
    `https://bedrock-agentcore.${region}.amazonaws.com/runtimes/${encodedArn}/ws`
  );
  url.searchParams.set('qualifier', 'DEFAULT');
  url.searchParams.set('X-Amzn-Bedrock-AgentCore-Runtime-Session-Id', sessionId);

  // Sign with SigV4
  const signer = new SignatureV4({
    service: 'bedrock-agentcore',
    region,
    credentials: {
      accessKeyId: credentials.accessKeyId,
      secretAccessKey: credentials.secretAccessKey,
      sessionToken: credentials.sessionToken,
    },
    sha256: Sha256,
  });

  const request = new HttpRequest({
    method: 'GET',
    protocol: 'https:',
    hostname: url.hostname,
    path: url.pathname,
    query: Object.fromEntries(url.searchParams),
    headers: { host: url.hostname },
  });

  const signed = await signer.presign(request, { expiresIn: 3600 });

  // Convert to wss:// URL
  const queryString = Object.entries(signed.query || {})
    .map(([k, v]) => `${encodeURIComponent(k)}=${encodeURIComponent(String(v))}`)
    .join('&');

  return `wss://${signed.hostname}${signed.path}?${queryString}`;
}

The credential exchange happens one step earlier. The browser takes the Cognito ID token and exchanges it for temporary AWS credentials via the Identity Pool:

import { CognitoIdentityClient, GetIdCommand,
         GetCredentialsForIdentityCommand } from '@aws-sdk/client-cognito-identity';

export async function getAWSCredentials() {
  const idToken = await getIdToken();
  const client = new CognitoIdentityClient({ region });
  const providerName = `cognito-idp.${region}.amazonaws.com/${userPoolId}`;

  // Step 1: Get identity ID from the token
  const { IdentityId } = await client.send(
    new GetIdCommand({
      IdentityPoolId: identityPoolId,
      Logins: { [providerName]: idToken },
    })
  );

  // Step 2: Exchange for temporary AWS credentials
  const { Credentials } = await client.send(
    new GetCredentialsForIdentityCommand({
      IdentityId,
      Logins: { [providerName]: idToken },
    })
  );

  return {
    accessKeyId: Credentials.AccessKeyId,
    secretAccessKey: Credentials.SecretKey,
    sessionToken: Credentials.SessionToken,
  };
}

These temporary credentials are scoped by the Cognito authenticated role - they can only call bedrock-agentcore:Invoke* actions, not other AWS services.

Deployment Commands

The full deployment workflow:

# 1. Provision infrastructure (S3, CloudFront, Cognito, ECR)
make apply

# 2. Build and push the ARM64 container to ECR
make docker-build
make docker-push

# 3. Create the AgentCore runtime (first time only)
make create-agent

# 4. Add the runtime ARN to terraform.tfvars, re-apply for Cognito IAM policy
make apply

# 5. Generate frontend .env from Terraform outputs
make setup-env

# 6. Build and deploy frontend to S3 + invalidate CloudFront cache
make deploy-frontend

For subsequent code changes, make deploy-agent rebuilds the container and updates the runtime in one step.

Key deployment gotcha: AgentCore container deployments require unique ECR image tags to force replacement. Pushing a new image with the latest tag does not trigger a container update - you must use a unique tag (I use v$(date +%Y%m%d%H%M%S)) and pass it to update-agent-runtime.

What Does It Cost?

Nova Sonic pricing is per-token, not per-minute. Audio converts at roughly 25 tokens per second.

Token type	Price per 1,000 tokens
Speech input	$0.0034
Speech output	$0.0136
Text input	$0.00006
Text output	$0.00024

Speech tokens dominate the bill. Text tokens only apply to tool call inputs/outputs and any conversation history injected as text - these are negligible for a voice-only session.

Typical session costs:

A 5-minute cooking session where the user asks 4-5 questions and the assistant responds with recipe steps and tool results:

Speech input (~2 min of user speaking): 2 x 60 x 25 = 3,000 tokens = $0.010
Speech output (~3 min of assistant speaking): 3 x 60 x 25 = 4,500 tokens = $0.061
Text tokens (tool calls/results): ~2,000 tokens = $0.0005
Total: ~$0.07 per session

For comparison, an 8-minute session (the Nova Sonic maximum) with continuous conversation:

Speech input (~3.5 min): 5,250 tokens = $0.018
Speech output (~4.5 min): 6,750 tokens = $0.092
Total: ~$0.11 per session

That is roughly 80% cheaper than OpenAI's GPT-4o Realtime API for equivalent conversation length.

Tracking costs in code:

AWS does not currently publish CloudWatch metrics for InvokeModelWithBidirectionalStream - the standard Bedrock metrics only cover InvokeModel, Converse, and their streaming variants. Model invocation logging and Application Inference Profiles (for cost tagging) also do not support bidirectional streaming yet.

The practical workaround is application-level cost estimation. The server logs session duration, audio chunk counts, and an estimated cost for each session:

# Nova Sonic pricing (per 1,000 tokens)
SPEECH_INPUT_PRICE_PER_1K = 0.0034
SPEECH_OUTPUT_PRICE_PER_1K = 0.0136
TOKENS_PER_SECOND_AUDIO = 25

# In the WebSocket handler:
session_start = time.monotonic()
input_audio_chunks = 0
output_audio_chunks = 0

# ... wrap receive/send to count audio chunks ...

# In the finally block:
session_duration = time.monotonic() - session_start
input_tokens = session_duration * TOKENS_PER_SECOND_AUDIO
output_tokens = session_duration * TOKENS_PER_SECOND_AUDIO
estimated_cost = (input_tokens / 1000) * SPEECH_INPUT_PRICE_PER_1K \
               + (output_tokens / 1000) * SPEECH_OUTPUT_PRICE_PER_1K

logger.info(
    "Session ended: duration=%.1fs, est_cost=$%.4f",
    session_duration, estimated_cost,
)

This uses wall-clock time as an upper bound (both speakers are not active simultaneously for the full duration). For aggregate cost monitoring, AWS Cost Explorer does show Bedrock charges broken down by model, so you can track monthly Nova Sonic spend at the account level - you just cannot get per-invocation breakdowns without application-level logging.

Converging Voice and Text: Future Integration

I now have two separate recipe assistants - a text-based version that uses CloudFront + Lambda + AgentCore for HTTP/SSE streaming, and this voice version that uses CloudFront + Cognito + AgentCore for WebSocket streaming. They share the same Bedrock Knowledge Base and similar tool implementations, but have separate infrastructure and deployment pipelines.

The natural next step is to combine them into a single application with both input modes. The architecture would look like:

Unified frontend - A single React SPA that offers both a chat interface and a voice interface. The text chat already uses SSE streaming through Lambda; the voice mode connects directly to AgentCore via WebSocket. Both modes share the same auth (Cognito) and the same CloudFront distribution.
Shared agent - Both modes could use the same AgentCore runtime with the same tools. The text path would go through Lambda (for SSE streaming), while the voice path would connect directly via WebSocket. The agent code is already nearly identical - same tools, same Knowledge Base, same system prompt. The main difference is the I/O layer.
Shared infrastructure - The Terraform modules for auth, CDN, and storage are similar between the two projects. Combining them eliminates duplication and simplifies deployment.
Mode switching - In the kitchen, you might start with text ("what should I make for dinner?"), switch to voice once you start cooking ("what is the next step?"), and go back to text when things get noisy. A unified app would make this seamless.

The biggest challenge is the streaming protocol difference. The text assistant uses HTTP POST + SSE (request-response), while the voice assistant uses a persistent bidirectional WebSocket. Lambda handles the text path well but maybe isn't the best approach for the WebSocket connections needed here. AgentCore handles Websockets for this case well. The solution may be to have both paths - Lambda for text, AgentCore container for voice - behind the same CloudFront distribution.

What's Next

Session rotation - Automatically reconnect when the 8-minute Nova Sonic limit is reached, preserving conversation context
Unified app - Merge the text and voice assistants into a single application with both input modes, shared tools, and shared infrastructure
Alexa integration - Strands BidiAgent with a custom I/O adapter could bridge to the Alexa Skills Kit for hands-free kitchen use
Multi-modal input - Nova Sonic v2 supports text input alongside audio. Adding a text fallback for when voice is not practical (quiet environments, accessibility needs)

The gap between a text-based agent and a voice-based agent is smaller than I expected. Strands abstracts the hard parts - WebSocket management, audio encoding, interruption handling, concurrent tool execution - and lets you focus on the tools and the system prompt. The same @tool decorator, the same docstring-based tool selection, the same Bedrock integration. Just with a microphone instead of a text box.

If you have an existing Strands agent, adding voice is closer to a weekend project than a rewrite.

Connect with me on X, Bluesky, LinkedIn, Medium, Dev.to, GitHub, or the AWS Community. Check out more of my projects at darryl-ruggles.cloud and join the Believe In Serverless community.

A Serverless Recipe Assistant with Bedrock AgentCore, Knowledge Bases, and S3 Vectors

Darryl Ruggles — Thu, 05 Mar 2026 16:01:24 +0000

During the last couple of years I have changed my life by adopting a low-carb/keto diet and lots of regular cardio and weight training. As part of this I have accumulated many recipes using alternative ingredients like nut flours, sugar substitutes, and more. Also our family has accumulated more than 600 recipes over the years - everything from bread experiments I've been perfecting, to my grandmother's handwritten brown bread recipe on a stained index card, to my wife's collection of holiday baking favourites bookmarked from blogs that no longer exist. They were scattered across PDFs, photos of handwritten cards, screenshots, and dead URLs. Finding anything was painful - I'd scroll through folders, skim titles, and still end up Googling a recipe I already had saved somewhere.

I wanted a way to just ask for what I needed: "what keto bread recipes do we have?", "what's in grandma's homemade brown bread?", or "what are the macros in that almond flour waffle recipe?" - and get answers from our own collection, with the option to look up accurate nutrition from the USDA database for any recipe or ingredient list.

This project started as a way to learn Amazon Bedrock Knowledge Bases and Bedrock AgentCore by building something I'd actually use every day. Along the way it turned into a full-stack serverless application with recipe search, real-time USDA nutrition lookup, text-to-speech cooking mode, and streaming chat - all for about $0.08 per half month in AWS costs.

You can check out the complete source code in my GitHub repo here → Serverless Family Recipe Assistant Repo

The Problem

Our family recipe collection is a mix of everything - food blog PDFs, Instagram screenshots, handwritten cards from my grandmother and mother-in-law, scribbled notes on the backs of envelopes, and bookmarked URLs that had since gone dead. Some of the oldest recipes were on stained, faded cards where the handwriting was barely legible. Others were photos taken at awkward angles with half the ingredients cut off.

Beyond just finding recipes, I wanted to track nutrition. I've been following a keto diet for a few years, and knowing the macros - especially net carbs - matters. But most family recipes don't come with a nutrition label. I wanted to be able to ask "what are the macros in grandma's homemade brown bread?" and get a real answer calculated from USDA data, not a guess.

Traditional approaches (a database with manual data entry, or simple full-text search) wouldn't cut it. Manually transcribing 600+ recipes - many handwritten - would take forever. And I didn't want to fill out search forms. I wanted to type "show me chicken recipes under 10g net carbs" and get actual answers from our own collection.

The Solution - Architecture Overview

The system uses a layered serverless architecture: CloudFront + Lambda Function URL + WAF as a secure proxy layer in front of Bedrock AgentCore. CloudFront serves both the React frontend (from S3) and the API (/chat path) from a single distribution, giving you CDN caching, HTTPS, and WAF integration with one resource. WAF provides rate limiting (100 requests per 5 minutes per IP) and bad input protection. The Lambda function, configured with RESPONSE_STREAM mode, enables true SSE streaming with no timeout constraint, bridges JWT authentication to IAM, and signs requests to AgentCore with SigV4.

Why This Layered Architecture?

Each layer in the stack serves a specific purpose:

CloudFront - single HTTPS domain for both frontend and API, CDN edge caching for static assets, WAF integration for rate limiting and input filtering, and origin access control (OAC) for secure S3 and Lambda origins
Lambda Function URL - auth bridging (JWT to IAM), SigV4 request signing to AgentCore, RESPONSE_STREAM invoke mode for true SSE and request/response transformation
WAF - rate limiting per IP to prevent abuse, and managed rule groups for bad input protection
AgentCore - managed agent runtime with auto-scaling to zero, built-in telemetry, container management, and memory

Why CloudFront + Lambda Function URL (Not API Gateway or ALB)

The browser needs to talk to AgentCore, but AgentCore uses IAM auth and the browser has JWT tokens. Something has to sit in the middle to bridge that gap, provide HTTPS, and add WAF protection. Three AWS options can do this:

API Gateway REST API (v1) is the traditional choice. It has a built-in Cognito authorizer, request validation, usage plans, and API keys. But it buffers responses with a 29-second timeout (I know you can open a support ticket and get this increased but i didn't want to go down that path) - there's no streaming support. An agent response that involves a KB search, a USDA API call, and a multi-paragraph answer easily takes 10-30 seconds, and complex queries can run longer.

Application Load Balancer supports WebSockets and long connections with no timeout ceiling, and gives you weighted target groups for canary deployments. But ALBs have a fixed hourly cost (~$16/month minimum) regardless of traffic. For a personal project handling a few requests per day, that's more than the rest of the infrastructure combined.

CloudFront + Lambda Function URL is what we went with. Lambda Function URL with RESPONSE_STREAM invoke mode supports true SSE with no timeout constraint -- the Lambda streams chunks as they arrive from AgentCore, and CloudFront forwards them to the browser. CloudFront adds CDN edge caching for the frontend, HTTPS termination, and WAF integration, all from a single distribution. The Lambda handles JWT-to-IAM auth bridging and SigV4 request signing. At low traffic, the entire layer costs $0 (CloudFront and Lambda both have generous free tiers).

The tradeoff: you lose API Gateway's built-in JWT authorizer and request validation, so you handle those in Lambda code. You also lose ALB's weighted routing for canary deployments. For a personal/family app, those aren't needed. For a multi-tenant SaaS product, you'd likely want API Gateway's usage plans and API keys, and accept the streaming limitation (or use WebSockets instead of SSE).

One subtlety worth noting: CloudFront's origin read timeout defaults to 60 seconds. Once streaming starts, each SSE chunk resets that clock, so long responses work fine as long as data keeps flowing. But if the agent goes quiet during a long tool execution (say, a slow external API call), CloudFront could drop the connection. The Lambda sends SSE keepalive comments (: keepalive) every 30 seconds to prevent this.

Why AgentCore?

Deploying an AI agent to production involves a lot more than writing the agent logic. You need an HTTP endpoint that can handle long-running streaming connections, authentication middleware, auto-scaling that can handle bursty traffic, container orchestration, health checks, and observability. In a traditional setup, you'd wire together API Gateway (or an ALB), ECS or EKS, ECR for container images, CloudWatch for logs, and X-Ray for tracing. That's a lot of infrastructure to build and maintain for what is, at its core, a Python function that takes a prompt and yields text.

Bedrock AgentCore collapses all of that into a managed runtime. You write a Python function decorated with @app.entrypoint, define your auth and memory config in a YAML file, and deploy with one CLI command. AgentCore handles:

HTTP ingress with SSE support - long-lived streaming connections work out of the box, no ALB timeout tuning or WebSocket configuration needed
IAM authentication - AgentCore uses IAM auth, and the Lambda proxy signs each request with SigV4 using its execution role credentials
Auto-scaling to zero - consumption-based pricing means you pay only for vCPU-seconds and memory-seconds while handling requests. No minimum instance count, no idle compute. My half-month bill for AgentCore was $0.06
Container management - AgentCore builds and runs your container. You don't write Dockerfiles, manage ECS task definitions, or configure load balancers
Built-in telemetry - OTLP-compatible tracing and metrics with no setup. AgentCore provides the telemetry endpoint automatically

The practical result: the entire agent deployment is a make deploy-agent command that takes about 2 minutes. Compare that to the hours you'd spend configuring ECS + ALB + API Gateway + CloudWatch + X-Ray for the same functionality.

The Lambda function in front of AgentCore is intentionally thin - it handles auth bridging (decoding the JWT and signing the request with SigV4) and streams the response back through CloudFront. All the heavy lifting - LLM inference, tool execution, memory management, and observability - stays in AgentCore where it belongs.

Services Used

Service	Role
CloudFront	CDN, HTTPS, WAF integration, single distribution for frontend + API
Lambda	Streaming proxy - bridges JWT auth to IAM, SigV4 signing
WAF	Rate limiting (100 req/5min per IP), bad input protection
Bedrock AgentCore	Managed runtime for the Strands SDK agent
Bedrock Knowledge Base	Vector search over recipe collection
S3 Vectors	Zero-idle-cost vector storage for embeddings
Amazon S3	Recipe markdown storage + frontend static hosting
Amazon Cognito	User authentication (User Pool + Identity Pool)
Amazon Polly	Text-to-speech for cooking mode
Bedrock LLM	Nova Pro (default) or Claude Sonnet 4
Titan Embed V2	Embedding model for Knowledge Base
Terraform	Infrastructure as code

Recipe Ingestion Pipeline

Before the agent can search recipes, they need to be converted from their original formats into something a Knowledge Base can index. The ingestion pipeline handles this in three stages.

Stage 1: Preprocessing - Where AI Really Shines

The preprocess_recipes.py script takes raw source files - PDFs, images, DOCX files - and converts them into structured markdown using Amazon Nova Pro as a multimodal LLM.

This is where AI made the biggest practical difference. Many of our family recipes were photos of handwritten cards - some decades old, stained with cooking splatters, faded ink, and the kind of cursive that only the person who wrote it could normally read. Nova Pro's multimodal capabilities handled these remarkably well. It could read through coffee stains, interpret faded handwriting, and even make reasonable guesses at abbreviated ingredient names ("chx" → "chicken", "tbls" → "tablespoon"). A few needed manual corrections, but the vast majority were parsed correctly on the first pass - work that would have taken weeks to do by hand.

Each recipe gets a consistent format:

Title, description, source URL
Ingredients with quantities
Numbered instructions
Nutrition per serving (from the source, or LLM-estimated and marked [estimated])
Tags for dietary categories

Stage 2: USDA Nutrition Enrichment

About 57% of recipes had LLM-estimated nutrition that wasn't very accurate. The enrich_nutrition.py script re-calculates nutrition for each recipe using the USDA FoodData Central API:

Parse each ingredient line (quantity, unit, name) using regex
Look up each ingredient in the USDA database (Foundation + SR Legacy data types)
Convert from per-100g USDA data to actual recipe quantities using a unit conversion table
Replace [estimated] markers with [USDA] markers

This runs as a batch process locally and caches API responses to avoid redundant lookups.

Stage 3: Upload & Index

The sync_recipes.py script uploads the processed markdown files to S3 and triggers a Knowledge Base ingestion job. The KB uses Titan Embed V2 to generate embeddings and stores them in S3 Vectors with semantic chunking (300 max tokens, 95% breakpoint threshold).

make preprocess    # PDFs/images → structured markdown via Nova Pro
make sync          # Upload to S3 + trigger KB ingestion

The Beauty of Plain Markdown

Real-world data ingestion is messy. No matter how good the AI is, you end up with typos, misread quantities, garbled ingredient names, and the occasional recipe that just won't parse at all. One of my grandmother's recipes came through with "1/2 cup fleur" instead of "flour". Another had the instructions jumbled because the photo was taken at an angle. A few recipes from scanned PDFs had entire sections missing where the scan quality dropped off.

This is where the decision to use plain markdown pays off. The processed recipes are just .md files sitting in data/processed/. If the AI misread an ingredient, got a quantity wrong, or produced something unparseable, you open the file in any text editor, fix it, and run make sync. The Knowledge Base re-ingests in a couple of minutes and the agent immediately returns the corrected version. No database migrations, no redeployment, no rebuild. It's the simplest possible feedback loop: edit a text file, push, done.

I ended up doing a manual pass through about 10% of the recipes after the initial AI processing. That's still massively better than transcribing all 600+ by hand, and the corrections took minutes each rather than the 5-10 minutes it would take to type out a full recipe from scratch.

Adding New Recipes

The collection keeps growing. When I find a new recipe I want to keep, the process is simple: save the source file (PDF, screenshot, photo, or even just copy-paste the text into a .md file) into your recipe source directory (configured via RECIPE_SOURCE_DIR in .env), run make preprocess to let Nova Pro convert it to structured markdown, review and fix any issues in the output, then make sync to push it to S3 and re-index the Knowledge Base. The new recipe is searchable within a couple of minutes. For recipes that are already in a clean text format, I sometimes skip the AI preprocessing entirely and just write the markdown file directly.

The Agent - Strands SDK on AgentCore

The agent is built with the Strands Agents SDK and deployed to Bedrock AgentCore. It's a surprisingly small amount of code - the core entrypoint is about 40 lines of Python.

Why Strands SDK

There are several Python frameworks for building LLM-powered agents - LangChain, LlamaIndex, AutoGen, CrewAI, and others. I chose Strands for a few reasons.

Minimal abstraction. Strands doesn't try to wrap everything in its own object model. Tools are plain Python functions decorated with @tool. The agent is constructed with a model, a system prompt, and a list of tools. There's no chain-of-thought pipeline to configure, no "memory module" to plug in, no retriever-adapter pattern. You write Python, not framework DSL. When I needed to debug why the agent was calling search_recipes twice for the same query, I could read through the Strands source in an afternoon - it's a thin layer over the Bedrock Converse API, not a 50-module abstraction.

Native streaming. The agent exposes an async generator (agent.stream_async()) that yields events as they happen - text deltas, tool invocations, and metadata. This maps directly to the SSE pattern the frontend needs. There's no callback system to wire up and no post-processing step to convert the response into a streamable format. Each yield from the agent becomes an SSE data: line.

First-class AgentCore integration. Strands was built alongside AgentCore, so the deployment model is native. The @app.entrypoint decorator, the BedrockAgentCoreApp class, and the AgentCoreMemorySessionManager all come from the SDK. There's no glue code needed to bridge the framework to the runtime. This also means the SDK's streaming protocol matches what AgentCore expects - you yield dicts and they become SSE events.

Predictable behavior. The agent loop is straightforward: send prompt to LLM, if the LLM requests a tool call then execute it and feed the result back, repeat until the LLM produces a final response. The max_iterations parameter caps cycles to prevent runaway loops. There's no autonomous planning step, no chain selection logic, and no implicit retries that change behavior in surprising ways. For a recipe assistant with two tools, this simplicity is exactly right.

Open source. Strands is Apache 2.0 licensed, so there's no vendor lock-in concern beyond the Bedrock model provider integration (and even that is pluggable - Strands supports other providers).

The trade-off is that Strands is newer and less battle-tested than tools like LangChain. The documentation is still catching up, and some features (like long-term memory and multi-agent coordination) are less mature. For a focused single-agent application like this, those gaps didn't matter. For a complex multi-agent orchestration system, you might want to evaluate more carefully.

Entrypoint

@app.entrypoint
async def handle(event, context):
    prompt = event.get("prompt", "")

    # Frontend sends {cognito_sub}_{conversation_uuid} as session_id.
    # AgentCore validates the JWT but strips the Authorization header,
    # so we derive user_id from the session_id prefix.
    session_id = event.get("session_id") or context.session_id or "anonymous"
    user_id = session_id.rsplit("_", 1)[0] if "_" in session_id else session_id

    config = AgentCoreMemoryConfig(
        memory_id=MEMORY_ID,
        session_id=session_id,
        actor_id=user_id,
    )

    for attempt in range(MAX_RETRIES + 1):
        try:
            with AgentCoreMemorySessionManager(config, region_name=REGION) as session_manager:
                agent = Agent(
                    model=model,
                    system_prompt=SYSTEM_PROMPT,
                    tools=[search_recipes, calculate_nutrition],
                    session_manager=session_manager,
                )
                last_tool = None
                async for stream_event in agent.stream_async(prompt):
                    if "data" in stream_event:
                        yield {"chunk": stream_event["data"]}
                    elif stream_event.get("current_tool_use"):
                        tool_name = stream_event["current_tool_use"].get("name", "")
                        if tool_name and tool_name != last_tool:
                            last_tool = tool_name
                            yield {"tool_use": tool_name}
                return
        except Exception as e:
            if attempt == MAX_RETRIES:
                raise
            await asyncio.sleep(1)

Each yield becomes an SSE event that the frontend consumes in real time. The agent forwards both text chunks and tool-use notifications so the UI can show what's happening.

LLM Configuration

The agent supports multiple models, selected via the ACTIVE_LLM environment variable at deploy time:

Model	ID	Use Case
`nova` (default)	`us.amazon.nova-pro-v1:0`	General use, free tier
`nova-lite`	`us.amazon.nova-lite-v1:0`	Faster, cheaper
`claude`	`us.anthropic.claude-sonnet-4-20250514-v1:0`	Higher quality
`claude-haiku`	`us.anthropic.claude-3-5-haiku-20241022-v1:0`	Fast and cheap

Memory - Multi-Turn Conversations

One of the things that makes this feel like a real assistant rather than a search box is that the agent remembers the conversation. Without memory, every prompt is independent - the agent has no idea what you asked 10 seconds ago. You'd have to repeat context every time: "In the almond flour waffle recipe you just showed me, what are the macros?" becomes meaningless if the agent doesn't know which recipe it just showed you.

Building this yourself means managing a conversation store (typically DynamoDB or Redis), writing serialization logic to pack and unpack conversation turns, handling TTLs and cleanup, scoping sessions by user and conversation, and injecting the right context window into each LLM call. It's not difficult code, but it's infrastructure and logic that has nothing to do with your agent's actual purpose.

AgentCore Memory handles all of this as a managed service. You configure a memory ID and session scope, wrap your request in an AgentCoreMemorySessionManager context manager, and the SDK takes care of the rest - loading prior turns on entry, saving the new turn on exit. The result is natural multi-turn conversations:

You: "What keto bread recipes do we have?" Agent: (searches KB, returns 3 recipes) You: "What are the macros in the second one?" Agent: (knows which recipe you mean, calculates nutrition) You: "How does that compare to the first one?"

Each conversation gets a unique session ID ({cognito_sub}_{conversation_uuid}) scoped by user. Memory events expire after 30 days. Clicking "New Chat" generates a fresh UUID, giving the agent a clean slate with no prior context. The total code to integrate memory is about 6 lines - the config object and the with block.

Custom Tools - Recipe Search & Nutrition

The agent has two tools it can call during a conversation.

search_recipes

Queries the Bedrock Knowledge Base using vector similarity search. Key implementation details:

Top 3 results to avoid overwhelming the LLM context
Score threshold of 0.3 to discard irrelevant matches
Chunk merging - recipes split across multiple embedding chunks get reassembled into a single result. Without this, you'd get fragments of the same recipe appearing as separate results.

@tool
def search_recipes(query: str) -> str:
    """Search the recipe knowledge base. Always use this tool first
    when users ask about recipes, ingredients, or cooking."""

calculate_nutrition - Real-Time USDA Lookup

This is one of the most useful features for anyone tracking macros. You can ask the agent to calculate nutrition for anything - a recipe from the collection, a custom ingredient list, or even "what are the macros in 2 cups of almond flour and 3 eggs?" The agent calls the USDA FoodData Central API in real time and returns a detailed breakdown.

@tool
def calculate_nutrition(ingredients: str) -> str:
    """Calculate nutrition for a list of ingredients using USDA data.
    Input: newline-separated ingredients with quantities."""

The tool parses quantities and units, looks up each ingredient in the USDA database (Foundation + SR Legacy data types), converts from per-100g USDA values to actual recipe quantities, and returns a formatted markdown table with per-ingredient breakdowns and totals - calories, net carbs, total carbs, fat, protein, and fiber. The UI shows "Calculating nutrition..." while this runs, and the results come back with a [USDA] marker so you know the data source.

This was especially valuable for our older family recipes that never had nutrition info. I can now ask "what are the macros in grandma's homemade brown bread?" and get real USDA-backed numbers instead of guesses.

The Frontend - React 19 with SSE Streaming

The frontend is a React 19 single-page application built with Vite. No TypeScript - plain JSX throughout. It POSTs to the CloudFront /chat endpoint, which proxies through the Lambda Function URL to AgentCore. The request includes the JWT in the Authorization header and an x-amz-content-sha256 header (SHA-256 hash of the request body, required by CloudFront OAC for POST requests). CloudFront proxies to the Lambda Function URL, which streams SSE back through CloudFront. The frontend uses fetch with ReadableStream to consume these SSE events, which gives more control than the EventSource API (custom headers for auth, POST method, abort support). The VITE_API_URL environment variable points to the CloudFront /chat path.

SSE Streaming

The chat hook processes two types of SSE payloads:

const parsed = JSON.parse(json);

if (parsed.tool_use) {
    // Show "Searching recipes..." or "Calculating nutrition..."
    onMessage({ type: 'tool_use', tool_name: parsed.tool_use });
} else if (parsed.chunk) {
    // Append text to the current message, render markdown
    onMessage({ type: 'chunk', content: parsed.chunk });
}

The frontend strips \ blocks from Claude's extended thinking output before displaying text to the user.

Abort Handling - Harder Than It Sounds

Streaming responses from an LLM can take a while, especially when the agent is searching the KB and then generating a long recipe with full instructions and nutrition. Users need to be able to cancel mid-stream. The send button swaps to a red stop button during streaming, and clicking it triggers AbortController.abort() to close the HTTP connection.

The tricky part is what happens to memory. When you abort, AgentCore may have already saved a partial conversation turn - half a recipe, a dangling tool call, or garbled markdown. If the next request loads that corrupted context, the agent gets confused and produces garbage.

The fix: on abort, the client generates a new conversation UUID and stores it in sessionStorage. The next request uses this fresh session ID, so AgentCore Memory loads a clean slate instead of the corrupted partial turn. The trade-off is that you lose conversation history from before the abort, but that's better than garbled responses.

This same session-reset mechanism powers the "New Chat" button - it's the same operation, just intentional rather than error recovery.

S3 Vectors - The $0.00 Vector Store

This was the most surprising part of the project. Traditional RAG setups use OpenSearch Serverless as the vector store, which costs a minimum of ~$350/month for 2 OCUs (OpenSearch Compute Units) - even with zero traffic. For a personal recipe project, that's a non-starter.

S3 Vectors provides the same semantic search capability with truly pay-per-query pricing:

Vector Store	Monthly Cost (dev)	Monthly Cost (prod, ~10K queries)
OpenSearch Serverless	~$350	~$350+
S3 Vectors	~$0.00	~$0.40

S3 Vectors pricing: $0.00004/query + $0.0006/GB-month storage. At my development usage of ~200 KB queries per half month, the cost rounds to zero.

The vector index uses 1024 dimensions (matching Titan Embed V2), cosine distance similarity, and float32 precision. It's configured via Terraform:

resource "aws_s3vectors_index" "recipes" {
  vector_bucket_name = aws_s3vectors_vector_bucket.recipes.name
  index_name         = "${var.project_name}-${var.environment}-recipes"

  metadata {
    dimension     = 1024
    distance_type = "cosine"
    data_type     = "float32"
  }
}

Authentication - Cognito JWT

Authentication uses Amazon Cognito with a User Pool for signup/signin and an Identity Pool for temporary AWS credentials (used by the browser for Polly TTS). The auth flow bridges JWT-based frontend authentication to IAM-based AgentCore authentication through the Lambda proxy layer.

The flow:

User signs in via the React auth screen (email + password, SRP auth)
Cognito returns access + ID + refresh tokens
Frontend sends POST to CloudFront /chat with Bearer JWT in the Authorization header and an x-amz-content-sha256 header (SHA-256 hash of the request body)
A CloudFront Function copies Authorization to X-Forwarded-Authorization before OAC replaces the Authorization header with SigV4 credentials for the Lambda Function URL origin
Lambda decodes the JWT from X-Forwarded-Authorization to extract the user identity (sub claim) for session scoping
Lambda signs the request to AgentCore with SigV4 (IAM auth) using its execution role credentials
For text-to-speech, the ID token is exchanged via the Identity Pool for temporary IAM credentials scoped to polly:SynthesizeSpeech

No Amplify - just amazon-cognito-identity-js for the Cognito SDK, keeping the bundle small.

Real-Time Tool Visibility

When the agent calls a tool, there's a noticeable delay while the Knowledge Base search or USDA API call completes. Instead of showing generic typing dots, the UI shows what the agent is actually doing.

The Strands SDK emits current_tool_use events during streaming. The agent forwards these as {"tool_use": "search_recipes"} SSE payloads. The frontend maps tool names to human-readable labels:

Tool Name	UI Label
`search_recipes`	"Searching recipes..."
`calculate_nutrition`	"Calculating nutrition..."

The label appears next to the animated typing dots and disappears as soon as text starts streaming. It's a small touch but makes a real difference in perceived responsiveness.

Text-to-Speech Cooking Mode

When you're actually cooking, you don't want to keep looking at your phone with flour-covered hands. Cooking mode uses Amazon Polly to automatically read each bot response aloud as it completes.

Toggle on/off with a speaker button in the header
Per-message play/pause/restart controls on every bot response
Uses Polly's generative voice via temporary Cognito Identity Pool credentials
Streams audio directly from the browser - no Lambda or backend involvement

Observability - Traces, Metrics, and Logs

Setting observability.enabled: true in the AgentCore YAML config gives you structured logs and service-level metrics in CloudWatch without any code changes. But the real payoff comes from distributed tracing - seeing exactly what happens inside the agent during every request.

Beyond AgentCore, the CloudFront and Lambda layers also contribute to the observability story. CloudFront access logs capture every request to both the frontend and the /chat API, including client IP, response time, and cache status. Lambda CloudWatch logs provide request-level tracing through the proxy layer, including JWT validation results, AgentCore response status, and streaming duration. Together these give you end-to-end visibility from the browser through CloudFront, Lambda, and into AgentCore.

What AgentCore Provides by Default

With observability enabled, AgentCore automatically emits:

Service-level metrics - invocation count, error rate, latency (p50/p95/p99), token usage
Structured JSON logs - every log statement includes traceId, spanId, requestId, and sessionId for correlation
Memory spans - trace segments for AgentCore Memory read/write operations

These show up in the CloudWatch GenAI Observability dashboard and X-Ray without any code changes. But there's a catch - by default, these are the only spans you get. The Bedrock LLM invocations, Knowledge Base queries, tool executions, and external HTTP calls are invisible in the trace.

Unlocking Full Distributed Tracing

The Strands Agents SDK has comprehensive built-in OpenTelemetry instrumentation that creates a rich hierarchy of trace spans:

Span Type	What It Captures
Agent	Top-level span for the entire invocation, aggregate token usage
Cycle	Each event loop iteration (reasoning cycle)
LLM	Individual Bedrock model invocations with prompts, completions, and token counts
Tool	Tool execution with name, inputs, outputs, and timing

But these spans only get exported if a tracer provider is configured. The required piece is the AWS Distro for OpenTelemetry (aws-opentelemetry-distro). Adding this single package to requirements.txt does three things at once:

Configures a global tracer provider so the Strands SDK actually exports its spans
Auto-instruments botocore/boto3 capturing Bedrock InvokeModel and KB Retrieve calls as child spans
Auto-instruments the requests library capturing external HTTP calls (like the USDA API) as child spans

AgentCore Runtime automatically runs the opentelemetry-instrument wrapper when this package is present - no code changes needed beyond adding the dependency.

# agent/requirements.txt
strands-agents[otel]>=0.1.0
aws-opentelemetry-distro>=0.10.0

What the Traces Look Like

After adding the ADOT distro, a single recipe search request produces a trace like this:

AgentCore Runtime Request
  └── Agent Span (Strands)
      ├── Cycle 1
      │   ├── LLM Invoke (bedrock converse)
      │   ├── Tool: search_recipes
      │   │   └── bedrock-agent-runtime.Retrieve
      │   └── LLM Invoke (bedrock converse)
      └── Memory Write (AgentCore)

A more complex request involving nutrition calculation shows the external API call:

AgentCore Runtime Request
  └── Agent Span (Strands)
      ├── Cycle 1
      │   ├── LLM Invoke (bedrock converse)
      │   ├── Tool: search_recipes
      │   │   └── bedrock-agent-runtime.Retrieve
      │   └── LLM Invoke (bedrock converse)
      ├── Cycle 2
      │   ├── LLM Invoke (bedrock converse)
      │   ├── Tool: calculate_nutrition
      │   │   └── HTTP GET api.nal.usda.gov
      │   └── LLM Invoke (bedrock converse)
      └── Memory Write (AgentCore)

Each span includes timing data, so you can immediately see where latency lives. In practice, the KB retrieval and LLM inference dominate - the USDA API calls are fast by comparison.

Custom Trace Attributes

The Strands SDK supports custom trace attributes on the Agent constructor, which get attached to every span in the request. This is useful for filtering traces by user or session in CloudWatch:

agent = Agent(
    model=model,
    system_prompt=SYSTEM_PROMPT,
    tools=[search_recipes, calculate_nutrition],
    trace_attributes={
        "session.id": session_id,
        "user.id": user_id,
    },
)

Cost Breakdown

This is the part that surprised me the most. Here are the actual AWS costs for the first half of February 2026:

Service	Cost	Notes
Bedrock (LLM)	$0.00	Free tier - Nova Pro, Titan Embed V2
Bedrock AgentCore	$0.06	~6 vCPU-hours, consumption-based
S3 (storage + vectors)	$0.02	Recipes, frontend, embeddings
CloudFront	$0.00	Free tier (1 TB transfer/month)
Lambda	$0.00	Free tier (1M requests/month)
WAF	$0.00	Free tier covers dev usage
Cognito	$0.00	Free tier (< 50K MAU)
CloudWatch	$0.00	Free tier covers logs + metrics
Project total	~$0.08	Half month

The entire project costs $0.08 for a half month of development usage. The Terraform provisions zero compute resources - no EC2 instances, no ECS tasks, no ECR repositories. Every service is either covered by free tier or has pay-per-use pricing with no idle cost.

Why So Cheap?

Four architectural decisions keep costs near zero:

S3 Vectors over OpenSearch Serverless - saves ~$350/month at any scale
AgentCore over self-managed compute - no idle costs, no infrastructure to manage. An equivalent ECS Fargate setup would cost $30-50/month minimum even with zero traffic
CloudFront + Lambda + WAF - all three are covered by free tier at development usage levels. CloudFront gives you CDN, HTTPS, and WAF integration without adding meaningful cost
Bedrock free tier - Nova Pro's free tier (50K input token requests, 300K output tokens/month) covers development usage comfortably

Projected Costs at Scale

Scenario	Monthly Est.
Light (1 user, ~50 queries/day, Nova Pro)	~$6.50
Medium (10 users, ~500 queries/day, Nova Pro)	~$44
Heavy (100 users, ~5K queries/day, Claude Sonnet 4)	~$995

Model choice is the dominant cost factor. Switching from Claude Sonnet 4 ($3.00/M input tokens) to Nova Pro ($0.80/M) at the same volume drops the LLM cost dramatically. The CloudFront + WAF layer adds roughly $5-10/month at medium to heavy scale (WAF web ACL base fee + per-request charges), which is negligible compared to LLM costs.

Key Learnings

S3 Vectors changes the RAG cost equation

Before S3 Vectors, the minimum cost of entry for a RAG application was ~$350/month for OpenSearch Serverless. That made it impractical for personal projects, prototypes, or low-traffic production workloads. S3 Vectors eliminates the idle cost entirely. This is a big deal for anyone building RAG applications that don't need millisecond latency.

AgentCore is the simplest way to deploy an agent

No Dockerfiles to maintain, no ECS task definitions, no load balancers. You write a Python function decorated with @app.entrypoint, run make deploy-agent, and you have a managed HTTP endpoint with JWT auth, auto-scaling, and telemetry. The trade-off is less control over the runtime environment, but for most agent workloads that's a fine trade.

Chunk merging is essential for Knowledge Base search

Bedrock Knowledge Base splits documents into chunks for embedding. When a recipe spans multiple chunks, the default behavior returns each chunk as a separate search result - so you might get the ingredients from one chunk and the instructions from another, but never the complete recipe. Grouping chunks by source document and merging them back together was critical for usable results.

Stream everything

The difference between waiting 8 seconds for a complete response and seeing the first words appear after a few seconds is enormous for user experience. The CloudFront to Lambda to AgentCore streaming chain delivers text chunks to the browser as they're generated, with incremental markdown rendering that makes the agent feel responsive even when it's doing multi-step tool calls behind the scenes. Adding tool-use indicators ("Searching recipes...") takes this further - the user knows the agent is working, not stalled.

Multimodal AI is a game-changer for digitizing family recipes

I was genuinely surprised at how well Nova Pro handled our oldest, most damaged recipes. Handwritten cards with coffee stains, faded ink, and cramped cursive that I could barely read myself - the model parsed them into structured ingredients and instructions with maybe a 90% accuracy rate. The remaining 10% needed minor corrections (a misread quantity here, a garbled ingredient name there), but the alternative was typing 600+ recipes by hand. This alone saved weeks of work and made the entire project feasible.

Understanding Strands SDK event loop and guardrails

The Strands Agents SDK event loop runs until the model emits end_turn (done generating) or max_tokens (response truncated). There's no built-in iteration cap - the model decides when to stop. For a recipe assistant with two tools this works fine, but for safety I added a BeforeToolCallEvent hook that cancels tool execution after 10 calls per request, preventing runaway loops:

class MaxToolCallsHook:
    def __init__(self, max_calls=10):
        self.max_calls = max_calls
        self._count = 0

    def register_hooks(self, registry, **kwargs):
        registry.add_callback(BeforeToolCallEvent, self._on_before_tool_call)

    def _on_before_tool_call(self, event):
        self._count += 1
        if self._count > self.max_calls:
            event.cancel_tool = "Maximum tool calls reached."

The hooks system is one of the cleanest parts of the Strands SDK - you implement HookProvider with a register_hooks method, and the registry gives you typed events for every stage of the agent lifecycle: BeforeModelCallEvent, AfterModelCallEvent, BeforeToolCallEvent, AfterToolCallEvent, and more. The cancel_tool field on BeforeToolCallEvent is particularly useful - it short-circuits the tool call and feeds the cancel message back to the model as a tool result, prompting it to wrap up with what it has.

On the token side, max_tokens=8192 on the BedrockModel prevents runaway generation. If the model hits this limit, the SDK raises a MaxTokensReachedException rather than silently truncating. Understanding these SDK internals - hooks, token limits, and stop reasons - is important for building agents that behave predictably.

What's Next

The long-term vision for this project includes:

Meal planning - generate weekly meal plans from the recipe collection
Shopping lists - aggregate ingredients, normalize quantities, deduplicate
Store availability - check prices at Loblaws, Walmart, Farm Boy, Metro, Amazon.ca, Costco.ca
Custom domain with ACM certificate - use a branded domain name with an AWS Certificate Manager TLS certificate instead of the default CloudFront distribution domain
Voice input - Web Speech API for hands-free queries while cooking
Multi-platform - PWA improvements for iPad/Android, and eventually an Alexa skill
CI/CD - GitHub Actions for automated testing and deployment

Try It Yourself

You can clone the GitHub repo and deploy this in your own AWS account. The infrastructure is fully defined in Terraform.

Prerequisites

AWS account
AWS CLI, AgentCore CLI
Terraform >= 1.14
Python 3.13+ with uv
Node.js 20+ with npm
A USDA FoodData Central API key (free -sign up here)

Quick Start

Bootstrap requires two passes because Terraform needs the AgentCore runtime ARN, but the agent can't be deployed until the infrastructure (KB, Cognito, S3) exists. See the README for full configuration details.

git clone https://github.com/RDarrylR/serverless-family-recipe-assistant.git
cd serverless-family-recipe-assistant

# 1. Install dependencies
make init               # Install Python deps
make install-frontend   # Install frontend deps

# 2. First Terraform pass (infrastructure without agent)
cd infrastructure
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars: set alert_email and aws_profile
terraform init && terraform apply
cd ..

# 3. Configure and deploy agent
aws xray update-trace-segment-destination --destination CloudWatchLogs  # one-time
make agent-config       # Generate AgentCore YAML from Terraform outputs
make setup-env          # Generate .env and frontend/.env from Terraform outputs
# Edit .env: set USDA_API_KEY, RECIPE_SOURCE_DIR
make deploy-agent       # Deploy agent to AgentCore
# Edit .env: set AGENT_RUNTIME_ID (from "make agent-status") and
#   MEMORY_ID (from "grep memory_id agent/.bedrock_agentcore.yaml")

# 4. Second Terraform pass (wire up Lambda to AgentCore)
# Edit terraform.tfvars: set agent_runtime_arn, add CloudFront URL to cognito_callback_urls
make apply              # Re-apply Terraform (wires Lambda to AgentCore)
make setup-env          # Regenerate .env files (preserves manual values)

# 5. Process recipes and deploy
make preprocess         # Convert raw recipes to markdown
make sync               # Upload to S3 + index in Knowledge Base
make deploy-frontend    # Build React app + deploy to S3 + invalidate CloudFront

CLEANUP (IMPORTANT!!)

If you deploy this yourself please understand some of the included resources will cost you a small amount of real money. Please don't forget about it.

Please MAKE SURE TO DELETE the infrastructure if you are no longer using it. Running terraform destroy from the infrastructure/ directory will handle this, or you can delete resources manually in the AWS console.

For more articles from me please visit my blog at Darryl's World of Cloud or find me on Bluesky, X, LinkedIn, Medium, Dev.to, or the AWS Community.

For tons of great serverless content and discussions please join the Believe In Serverless community we have put together at this link: Believe In Serverless Community

AWS Lambda Durable Functions - Build a Loan Approval Workflow with Checkpoints, Callbacks, and a React Frontend

Darryl Ruggles — Sun, 01 Mar 2026 16:10:53 +0000

I have been using AWS Step Functions for workflow orchestration for a long time now. I used it in my Serverless Data Processor project to coordinate batch data processing and in the Serverless Cloud Pizzeria to manage pizza order progress with task tokens. Step Functions is a solid service that does a lot of things well, especially when you want visual observability and native AWS service integrations.

When AWS announced Lambda Durable Functions at re:Invent 2025, I was immediately curious about how it compared and whether it could handle the kind of workflows I typically build. After spending some time reading the docs, community articles, and building a demo project, I wanted to share what I learned and walk through a working example.

The full source for this project can be found here on GitHub: https://github.com/RDarrylR/lambda-durable-demo

The Problem with Long-Running Lambda Workflows

If you have spent any time building serverless applications on AWS you have likely run into the Lambda 15-minute timeout limit. For many use cases this is fine. But as soon as you need to do things like wait for a human to approve something, call an external service that takes a while, or string together a pipeline with multiple steps that could take varying amounts of time you start running into challenges.

Before durable functions the typical options were:

Build your own state machine using DynamoDB and SQS to track progress and resume work across multiple Lambda invocations. This works but involves a lot of glue code that has nothing to do with your actual business logic.
Use Step Functions to define your workflow as a state machine in Amazon States Language (ASL). Step Functions handles the orchestration, retries, and state management for you. This is a great option and one I have used many times. The tradeoff is you are writing Amazon States Language (ASL) state machine definitions rather than code, and local testing can be more involved. I really do not like fighting with ASL.

Lambda Durable Functions give you a third option, keep writing normal Python or Node.js and let Lambda handle the checkpointing and replay mechanics for you.

What Are Lambda Durable Functions?

The core idea behind durable functions is based on four concepts:

Checkpoint: Every time you complete a step the result is persisted automatically
Replay: If the function gets interrupted it re-runs your handler from the top but skips any step that already has a cached result
Suspend: Wait operations pause the execution entirely, with zero compute charges while the clock ticks
Resume: A new Lambda invocation picks up where the last one left off with all context restored

The mental model is this: your code runs many times, but each step only executes once. Once you internalize that, everything else clicks.

The service launched in December 2025 with support for Python 3.13 and 3.14, and Node.js 22 and 24. The SDKs are open source on GitHub. Total execution time can be up to one year spread across many invocations even though each individual invocation is still subject to the standard 15-minute Lambda timeout.

The Demo Project - Loan Approval Workflow

To really understand how this works I built a full-stack loan approval workflow with a React frontend. This demonstrates checkpoints, parallel execution, two different callback patterns, DynamoDB progress tracking, and real-time frontend updates.

Architecture

The project has four AWS resources defined in a SAM template:

LoanProgressTable - DynamoDB table for tracking workflow progress that the frontend polls
LoanWorkflowFunction - The durable function that runs the entire loan approval workflow
FraudCheckFunction - A separate Lambda that simulates an external fraud check service and demonstrates the callback pattern
LoanApi + LoanApiFunction - API Gateway HttpApi with a Lambda handler that exposes POST /apply, GET /status/{id}, and POST /approve/{id}

The Workflow

There are three hardcoded demo profiles that produce predictable outcomes so you can demonstrate all the paths:

Profile	SIN (last 4)	Default Amount	Outcome
Alice Johnson	1111	$150,000	Always approved (triggers manager approval for amounts >= $100K)
Bob Martinez	2222	$50,000	Always denied (credit score too low)
Charlie Wilson	3333	$25,000	Approved if <= $25,000, denied if above

The Code - Walking Through the Durable Workflow

Let's look at the key parts of the code. I will focus on the durable function itself since that is where all the interesting stuff happens.

Making a Function Durable

The first thing you need to know is that making a Lambda function durable requires two things. In your code you use the @durable_execution decorator on the handler and the @durable_step decorator on each step function:

from aws_durable_execution_sdk_python import (
    DurableContext,
    StepContext,
    durable_execution,
    durable_step,
)

@durable_step
def validate_application(step_context: StepContext, application: dict) -> dict:
    """Validate the loan application fields."""
    required = [
        "application_id", "applicant_name", "ssn_last4",
        "annual_income", "loan_amount", "loan_purpose",
    ]
    missing = [f for f in required if f not in application]
    if missing:
        raise ValueError(f"Missing required fields: {missing}")

    # ... validation logic ...

    return {
        "application_id": application["application_id"],
        "status": "validated",
        "estimated_dti": round(dti_estimate, 2),
        # ... more fields ...
    }

This is just a normal Python function with a decorator. It receives a StepContext as the first argument and returns a result. That result gets checkpointed automatically. If the Lambda replays later this step will return the cached result instead of running the validation logic again.

In the SAM template you add DurableConfig to the function resource:

LoanWorkflowFunction:
  Type: AWS::Serverless::Function
  Properties:
    FunctionName: LoanWorkflowFunction
    CodeUri: src/
    Handler: loan_demo.lambda_handler
    Timeout: 300
    AutoPublishAlias: live
    DurableConfig:
      ExecutionTimeout: 3600           # 1 hour total workflow time
      RetentionPeriodInDays: 3        # Keep checkpoint history 3 days

Two important things to notice here. DurableConfig is what enables the feature and it cannot be added to existing functions, you have to create the function with it from the start. And AutoPublishAlias: live gives us a qualified ARN. You should always invoke durable functions via a version or alias, never $LATEST, because suspended executions need to replay against the same code that started them.

The Main Handler - Orchestrating the Workflow

Here is the core of the handler that ties everything together:

@durable_execution
def lambda_handler(event: dict, context: DurableContext) -> dict:
    table = get_progress_table()
    application_id = event["application_id"]

    try:
        # Step 1: Validate Application
        validated = context.step(validate_application(event))

        # Step 2: Parallel Credit Bureau Checks
        bureaus = ["equifax", "transunion", "experian"]
        credit_results = context.parallel([
            lambda ctx, b=b: ctx.step(
                pull_credit_report(b, validated["ssn_last4"])
            )
            for b in bureaus
        ])
        credit_reports = credit_results.get_results()

        # Step 3: Risk Assessment
        risk = context.step(calculate_risk_score(
            credit_reports, validated["ssn_last4"], validated["loan_amount"]
        ))

        # Denied path
        if risk["decision"] == "denied":
            return {"status": "denied", "reason": f"Risk tier: {risk['risk_tier']}"}

        # Step 4: Manager Approval (if >= $100,000)
        if validated["loan_amount"] >= 100000:
            approval_result = context.wait_for_callback(
                submit_manager_approval,
                name="manager-approval",
                config=WaitForCallbackConfig(timeout=Duration.from_minutes(30)),
            )
            if not approval_result.get("approved"):
                return {"status": "denied", "reason": "Manager denied"}

        # Step 5: External Fraud Check (Callback)
        fraud_result = context.wait_for_callback(
            submit_fraud_check,
            name="fraud-check",
            config=WaitForCallbackConfig(timeout=Duration.from_minutes(5)),
        )

        # Step 6: Generate Offer and Disburse
        offer = context.step(generate_loan_offer(validated, risk))
        disbursement = context.step(disburse_funds(offer))

        return {"status": "approved", "disbursement_ref": disbursement["disbursement_ref"]}

    except Exception as error:
        raise

I want to call out a few things that I found really interesting while building this.

Parallel Execution

The context.parallel() call runs three credit bureau checks concurrently. Each one is checkpointed independently. If two of the three bureaus succeed and the third fails on the first invocation, when Lambda replays only the failed one re-executes. The other two return their cached results instantly.

credit_results = context.parallel([
    lambda ctx, b=b: ctx.step(
        pull_credit_report(b, validated["ssn_last4"])
    )
    for b in bureaus
])
credit_reports = credit_results.get_results()

I have done parallel execution in Step Functions before using the Parallel state and it works well. But here it just feels more natural, it is a list comprehension that produces concurrent work. No separate branch definitions needed.

Two Callback Patterns

The demo uses context.wait_for_callback() in two places and they demonstrate two different real-world patterns.

Manager Approval - Human-in-the-Loop

For loans of $100K or more the workflow suspends and waits for a manager to approve. The setup function stores the callback ID in DynamoDB so the React frontend can read it and show an approval modal:

def submit_manager_approval(callback_id, _ctx):
    """Store callback_id in DynamoDB so the frontend can send the approval."""
    tbl = get_progress_table()
    tbl.update_item(
        Key={"application_id": validated["application_id"]},
        UpdateExpression="SET callback_id = :cid",
        ExpressionAttributeValues={":cid": callback_id},
    )

approval_result = context.wait_for_callback(
    submit_manager_approval,
    name="manager-approval",
    config=WaitForCallbackConfig(timeout=Duration.from_minutes(30)),
)

When the manager clicks approve in the React UI the API Lambda reads the callback ID from DynamoDB and calls send_durable_execution_callback_success to resume the workflow:

lambda_client.send_durable_execution_callback_success(
    CallbackId=callback_id,
    Result=json.dumps({"approved": approved}),
)

The Lambda that runs the durable workflow is completely idle during this time with zero compute cost. The manager could take 5 minutes or 5 hours but you only pay when the workflow actually resumes.

I have done something similar with Step Functions task tokens in my pizza ordering project where the token was passed to a Fargate container to call back when processing was complete. The concept is the same but with durable functions the callback is part of the same Python file rather than requiring coordination between separate state machine definitions and external compute.

Fraud Check - External Service Callback

The second callback demonstrates integrating with an external service. The workflow invokes a separate Lambda asynchronously and passes it the callback ID:

def submit_fraud_check(callback_id, _ctx):
    """Invoke the external fraud check Lambda, passing the callback_id."""
    fraud_lambda = boto3.client("lambda")
    fraud_lambda.invoke(
        FunctionName=os.environ["FRAUD_CHECK_FUNCTION"],
        InvocationType="Event",
        Payload=json.dumps({
            "callback_id": callback_id,
            "application_id": validated["application_id"],
            "applicant_name": validated["applicant_name"],
        }),
    )

The fraud check Lambda does its processing (simulated with a 5-second delay in the demo) and then sends the callback:

# fraud_check.py
def lambda_handler(event, context):
    callback_id = event["callback_id"]

    # Simulate processing
    time.sleep(5)

    result = {
        "fraud_check": "passed",
        "risk_indicators": 0,
        "checked_by": "FraudCheckService-v2",
    }

    # Resume the suspended durable execution
    lambda_client.send_durable_execution_callback_success(
        CallbackId=callback_id,
        Result=json.dumps(result),
    )

This pattern works for any external system - a microservice, a webhook handler, a third-party API that calls you back. The important thing is that the calling system has the callback ID and the IAM permission to call SendDurableExecutionCallbackSuccess on the durable function.

Real-Time Progress Tracking

One thing I wanted in this demo was the ability to see the workflow progressing in real time from the frontend. The durable workflow writes progress entries to DynamoDB at each step:

def log_progress(table, application_id, step, message, status, level="info", result=None):
    """Append a log entry and update status in DynamoDB."""
    timestamp = datetime.now(timezone.utc).isoformat()
    log_entry = {
        "timestamp": timestamp,
        "step": step,
        "message": message,
        "level": level,
    }
    table.update_item(
        Key={"application_id": application_id},
        UpdateExpression=(
            "SET #logs = list_append(if_not_exists(#logs, :empty_list), :new_log), "
            "current_step = :step, #status = :status, updated_at = :ts"
        ),
        # ...
    )

The React frontend polls GET /status/{applicationId} every couple of seconds and updates the UI as each step completes. There is nothing durable-function-specific about this, it is just polling a DynamoDB table. But it makes the demo much more compelling because you can watch the workflow progress through each checkpoint.

Handling Replays

One subtle issue I had to deal with was replay detection for the progress logging. Remember that when the durable function resumes after a callback the handler re-executes from the top and completed steps return their cached results. But the log_progress calls between steps are not inside steps, they are regular Python code that runs again on every replay.

I solved this with a counter-based approach:

from collections import Counter
existing = table.get_item(Key={"application_id": application_id}).get("Item", {})
prior_counts = Counter(entry["step"] for entry in existing.get("logs", []))
call_counts = Counter()

def log(step, message, status, level="info", result=None):
    call_counts[step] += 1
    if call_counts[step] <= prior_counts.get(step, 0):
        message = f"[REPLAY] {message}"
        level = "replay"
    log_progress(table, application_id, step, message, status, level, result)

This compares how many times each step has been logged before versus how many times we have called log in the current invocation. If we have already logged that many entries for a step it must be a replay and we tag it accordingly. This is a good example of the kind of thing you need to think about with the replay model that you don't have to worry about with Step Functions.

The SAM Template

The full SAM template sets up everything: a DynamoDB table, three Lambda functions, API Gateway with CORS, and all the IAM permissions. Here is the key section for the durable workflow function:

LoanWorkflowFunction:
  Type: AWS::Serverless::Function
  Properties:
    FunctionName: LoanWorkflowFunction
    CodeUri: src/
    Handler: loan_demo.lambda_handler
    Timeout: 300
    AutoPublishAlias: live
    DurableConfig:
      ExecutionTimeout: 300
      RetentionPeriodInDays: 3
    Environment:
      Variables:
        PROGRESS_TABLE: !Ref LoanProgressTable
        FRAUD_CHECK_FUNCTION: !Ref FraudCheckFunction
    Policies:
      - AWSLambdaBasicExecutionRole
      - Statement:
          - Effect: Allow
            Action:
              - lambda:ManageDurableState
              - lambda:GetDurableExecution
              - lambda:ListDurableExecutions
            Resource: !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:LoanWorkflowFunction*"
      - DynamoDBCrudPolicy:
          TableName: !Ref LoanProgressTable
      - Statement:
          - Effect: Allow
            Action:
              - lambda:InvokeFunction
            Resource: !GetAtt FraudCheckFunction.Arn

Notice the lambda:ManageDurableState permission - the function needs this to manage its own checkpoint data. The fraud check function needs a separate permission, lambda:SendDurableExecutionCallbackSuccess, to send the callback that resumes the workflow.

Durable Functions vs Step Functions - When Each Wins

After building this project and spending time with both services I have a much clearer picture of when each one makes sense. Here is how I think about it.

Choose Durable Functions When:

Your workflow is code-centric business logic. If the interesting part is the Python code between steps: validation rules, scoring algorithms, data transformations, durable functions keep everything in one place.
You need third-party libraries. In this demo I used AWS Lambda Powertools for logging, tracing, and metrics. In a real-world scenario you might need pandas for data processing, the Stripe SDK for payments, or an AI SDK for model calls. With durable functions you just add them to requirements.txt. In Step Functions each dependency would need its own Lambda function.
You have complex branching and looping. An AI agent that chains LLM calls, branches on model output, and retries on rate limits is just a while loop in Python. Expressing that as an ASL state machine means fighting the declarative model.
You want local testing. The durable execution testing SDK simulates the checkpoint engine in-process with pytest. No Docker containers, no local Lambda emulator, no AWS credentials needed.

Choose Step Functions When:

You are orchestrating across native AWS services. If your workflow is Glue crawler → Athena query → S3 export → SNS notification with zero custom logic between steps, Step Functions calls those services directly without Lambda. You can't beat zero-compute orchestration.
Visual observability matters. When the ETL pipeline fails at 2 AM, the on-call engineer opens the Step Functions console and sees one red box. They click it and see the error. No log searching.
You have compliance requirements. SOC 2 auditors can review a visual state machine. They can see every step, every permission, every error path without reading Python code.
The workflow is mostly waiting. A notification workflow that waits 30 days, checks something, and sends an email, five state transitions over a month, costs essentially nothing in Step Functions. No Lambda overhead during the waits.

The Hybrid Pattern

Worth mentioning that the best architecture often uses both. Step Functions for the macro-orchestration (routing, cross-service coordination, audit trail) and durable functions for the micro-orchestration (complex business logic inside individual steps). Step Functions calls a Lambda, that Lambda happens to be a durable function with its own internal workflow. Best of both worlds.

Things to Watch Out For

There are a few key things to watch out for that I ran into or learned about while building this that are worth being aware of.

Replay demands determinism. Since your code runs multiple times you cannot branch on things like random numbers, timestamps, or any mutable external state outside of steps. Everything non-deterministic must go inside a @durable_step. In my credit bureau check I generate scores using a seeded random number generator based on the SSN, the seed ensures the same score every time for the same input which keeps the replay deterministic.

The 15-minute invocation limit still applies. Each individual replay invocation has the standard Lambda timeout. Durable functions do not change that. What changes is the total execution can span many invocations over up to a year. Use waits between long steps to reset the window.

Version pinning is essential. If your code changes while an execution is suspended for three days waiting for a manager approval, the replay will run different code than the original. The AutoPublishAlias: live in the SAM template handles this by always creating a new version on deploy.

Observability is still maturing. Logging across replays can be confusing, you see the same log lines from both the original execution and the replay. The Durable Executions tab in the Lambda console is helpful but there is no jump-to-logs feature yet. I recommend investing in structured logging from day one with something like AWS Lambda Powertools.

This is a new service. Launched December 2025. You cannot convert existing functions. The SDKs are evolving. Pin your SDK versions and bundle them with your deployment package.

Try It Out

If you want to try this out for yourself the full source code is available on GitHub: https://github.com/RDarrylR/lambda-durable-demo

# Clone the repo
git clone https://github.com/RDarrylR/lambda-durable-demo
cd lambda-durable-demo

# Build and deploy the backend
sam build
sam deploy --guided   # first time

# Note the LoanApiUrl output, then setup the frontend
cd frontend
npm install
echo "VITE_API_URL=https://YOUR-API-ID.execute-api.us-east-1.amazonaws.com" > .env
npm run dev

The frontend runs at http://localhost:5173. Click one of the demo profiles (Alice, Bob, or Charlie), submit the application, and watch the workflow progress in real time. For Alice's $150K loan you will see the manager approval modal pop up, approve it and watch the rest of the workflow complete through the fraud check and disbursement.

I have lots of plans to expand on this in the future including adding automated tests with the durable execution testing SDK and exploring the hybrid pattern with Step Functions.

For more articles from me please visit my blog at Darryl's World of Cloud or find me on Bluesky, X, LinkedIn, Medium, Dev.to, or the AWS Community.

For tons of great serverless content and discussions please join the Believe In Serverless community we have put together at this link: Believe In Serverless Community

Building a Serverless Sales Analytics Platform with AI Insights for Under $10/Month

Darryl Ruggles — Sat, 29 Nov 2025 15:34:26 +0000

I have a number of projects I’ve been working on that are not finished yet but this is one I wanted to complete before I finished packing and got on the plane to attend AWS Re:Invent in Las Vegas this year. I'm hoping to pick up new techniques and meet many other people who build event-driven architectures every day to learn from them.

I see so many great examples of using the managed and serverless services that cloud providers like AWS offer. Being able to build such a complete solution that costs less than $10 a month to run is a common occurrence with these types of builds. You can examine your requirements and budget for any given project and choose from the many tools that are available to use with just an API call and only get charged based on how much you use them.

You can try this project out for yourself by checking out the code found in my Github repo here → Github Repo

The Challenge

Smurf Memorabilia Inc., is a fictional retail chain with multiple store locations and it needs a way to:

Collect daily sales data from each store location
Transform and store that data efficiently
Generate AI-powered business insights
Visualize results in dashboards

The key requirements include: low cost, minimal operational overhead, and pay only for what you use.

Stores will upload their sales data each day in an agreed format. The data will be processed and analysis will be done. Analytics data will be updated and AI-based recommendations will be made. Key people will receive daily emails or SMS messages of what is happening.

The Solution: 100% Serverless Architecture

My solution involves an event-driven ETL platform using managed AWS services. There are no servers to patch, no capacity to plan, and no minimum fees. You pay only when data flows through the system.

Services Used

Service	Role	Pricing Model
AWS Lambda	All compute (17 functions)	Per invocation + duration
S3	Object storage	Per GB stored + requests
DynamoDB	Metrics database	Per read/write unit (on-demand)
Step Functions	Workflow orchestration	Per state transition
EventBridge	Event routing	Free tier covers most use cases
Bedrock	AI analysis (Nova Lite)	Per token processed
API Gateway	REST API	Per request
SNS	Notifications	Per message

These are a few of the managed/serverless offerings from AWS. You can piece together as many of these as you need to build your architecture. These scale automatically from zero to whatever capacity you need.

Smart Data Storage with Apache Parquet

One of the key architectural decisions was converting the raw uploaded JSON sales data into Apache Parquet format. This columnar storage format delivers significant benefits:

Huge Compression

Our 30-day dataset comparison:

Raw JSON uploads: 53.1 MB
Parquet files: 4.7 MB

The examples I have get an 11x reduction in size using the default parquet compression algorithm but it can be changed to use even higher compression if needed. This results in great savings for storage and faster query performance.

Why Parquet?

Columnar Storage: Only reads the columns you need, not entire rows
Built-in Compression: Uses efficient encoding (dictionary, run-length, delta)
Schema Enforcement: Explicit types prevent data quality issues
Ecosystem Support: Works with Athena, Spark, Pandas, and most analytics tools

Type-Safe Schema

We define an explicit PyArrow schema to ensure data quality. We want to make sure we keep track of which Smurf loot is popular every day and follow the trends.

PARQUET_SCHEMA = pa.schema([
    ("transaction_id", pa.string()),
    ("transaction_timestamp", pa.timestamp("ms")),
    ("item_sku", pa.string()),
    ("item_name", pa.string()),
    ("quantity", pa.int32()),
    ("unit_price", pa.decimal128(10, 2)),
    ("line_total", pa.decimal128(10, 2)),
    ("discount_amount", pa.decimal128(10, 2)),
    ("payment_method", pa.string()),
    ("customer_id", pa.string()),
])

This schema ensures that decimal precision is maintained (critical for financial data) and timestamps are properly typed for time-series analysis.

Hive-Style Partitioning for Efficient Queries

Raw uploads arrive with flat filenames like store_0001_2025-11-27.json. We transform these into a Hive-style partition structure:

s3://bucket/processed/
├── year=2025/
│   └── month=11/
│       ├── day=27/
│       │   ├── store_id=0001/data.parquet
│       │   ├── store_id=0002/data.parquet
│       │   └── ...
│       └── day=28/
│           └── ...

Why This Structure Matters

Partition Pruning: When you query "all sales for November 2025", tools like Amazon Athena only scan files in year=2025/month=11/ - not the entire dataset. This means:

Faster queries
Lower costs (Athena charges per TB scanned)
Better organization

The Transformation Code:

# Parse: store_0001_2025-11-27.json
store_id, year, month, day = parse_filename(filename)

# Output: year=2025/month=11/day=27/store_id=0001/data.parquet
output_key = f"processed/year={year}/month={month}/day={day}/store_id={store_id}/data.parquet"

This simple transformation enables sophisticated analytics without complex ETL pipelines.

Two Analytics Options (Web-based and a more standard Business Intelligence approach)

I wanted to show how you could use multiple approaches to analyze the sales data. We need to use the best approaches to keep track of those 3-apple tall blue creatures and all the ways their fans want to remember them. One in a more simple web version that is built in ReactJS and runs in your browser. I also built a prototype version of Amazon Quick Suite dashboards. Depending on the audience one of these approaches will likely work (or you could build something else.

Option 1: React Dashboard (Developer-Friendly)

The project includes a custom ReactJS application that queries the API directly:

The Web-based analytics approach is likely best for:

Custom visualizations
Embedding in existing applications
Provides Full control over the user experience
No additional licensing costs

The React dashboard provides:

Real-time metrics display
File upload interface with drag-and-drop
Historical trend charts
AI-generated insights and recommendations

Option 2: Amazon Quick Suite (Business-Friendly)

This approach offers a managed Business Intelligence (BI) service that imports data from S3:

The Quick Suite approach is likely best for:

Business users who need self-service analytics
Ad-hoc exploration without writing code
Sharing dashboards with stakeholders
Built-in visualizations (no frontend development)

The current project exports five datasets to S3 in newline-delimited JSON format:

Store summaries (daily metrics per store)
Top products (best sellers)
Anomalies (AI-detected unusual patterns)
Trends (week-over-week analysis)
Recommendations (AI-generated action items)

Quick Suite's SPICE engine imports this data for fast, interactive dashboards.

Choosing between the analytics approach to use:

Factor	React Dashboard	Quick Suite
Cost	Included (API calls only)	$24/month per author, $3/month per reader
Setup	Requires development	Point-and-click
Customization	Unlimited	Template-based
User Type	Developers	Business analysts
Embedding	Full control	Quick Suite embedding

Many organizations could use both: ReactJS for customer-facing features, Quick Suite for internal analytics.

Event-Driven Processing

The platform uses an event-driven architecture where each component reacts to events rather than polling for work. I always try to use this type of architecture unless the use-case really doesn’t fit it. AWS Step Functions are used to drive the data upload processing as well as the recommendation and analytics flow handling.

Upload Processing Flow

Store uploads JSON file to S3 (via presigned URL)
S3 emits Object Created event
EventBridge routes event to Step Functions
Step Functions orchestrates the processing pipeline:

* Validate schema

* Convert to Parquet

* Calculate metrics

* Store in DynamoDB

* Check if all stores reported

Daily Analysis Trigger

When the last store uploads for a day, the system automatically triggers a smurfy comprehensive analysis:

The analysis runs exactly when the data is ready. But what if a store fails to report? A scheduled EventBridge rule runs at 11 PM local time as a fallback, ensuring you always get a daily report - even with partial data. The scheduler checks if analysis already ran for that day and skips if so.

If invalid data is uploaded, the key stakeholders will receive email or SNS notifications to follow up with users. If the processing flow fails on the first attempt it has built-in retry and backoff mechanisms.

Daily Email Reports

Once analysis completes, the platform automatically sends a daily summary email via SNS containing:

Total revenue across all stores
Top performing store of the day
AI-detected anomalies and unusual patterns
Business recommendations from Bedrock

Stakeholders receive insights in their inbox without logging into any dashboard.

AI-Powered Insights with Amazon Bedrock

The solution uses Amazon Bedrock with the Nova Lite model (configurable to whatever model you want) to generate business intelligence:

Anomaly Detection: Identifies stores with unusual revenue patterns
Trend Analysis: Compares current performance to historical baselines
Recommendations: Generates actionable business advice

Bedrock is pay-per-token with no minimum commitment - so it’s perfect for batch processing workloads.

The Cost Breakdown

Here's what this platform actually costs for a typical month (e.g., 330 file uploads = multiple stores × 30 days):

Service	Monthly Cost	Notes
Lambda	~$2.00	17 functions, ~1000 invocations each
Step Functions	~$0.50	360 workflow executions
DynamoDB	~$1.00	On-demand mode, ~1000 ops
S3	~$0.01	~60 MB stored
Bedrock	~$5.00	Nova Lite, 30 daily analyses
EventBridge	~$0.00	Free tier
SNS	~$0.10	Email notifications
CloudWatch Alarms	~$0.00	7 alarms (first 10 free)
Total	~$8.61

Add Quick Suite (if needed) for $24/month per author to build dashboards, or just $3/month per reader for view-only access.

Why is this all so cheap?

ARM64 Architecture: Lambda on Graviton2 is ~20% cheaper than x86
Parquet Compression: ~ 11x less storage than JSON
On-Demand DynamoDB: Pay only for actual read/write operations
Event-Driven: No idle compute costs

Infrastructure as Code (IaC)

I’m a big advocate of using IaC for everything. My favourite tools for this are Terraform, the Serverless Application Model (SAM), and the Cloud Development Kit (CDK). In this case there is VPC provisioning and a lot of resources so I chose my go-to tool Terraform. One command deploys everything with Terraform:

terraform apply

Here are some key snippets from the infrastructure code:

Lambda Functions (ARM64 for Cost Savings)

Lambda is best place to host your business logic when code execution times are short. All 17 Lambda functions use ARM64 architecture (Graviton2) for ~ 20% cost savings:

resource "aws_lambda_function" "process_upload" {
  filename      = data.archive_file.process_upload_zip.output_path
  function_name = "process_upload"
  role          = aws_iam_role.lambda_role.arn
  handler       = "process_upload.lambda_handler"
  runtime       = "python3.13"
  architectures = ["arm64"]
  timeout       = 30
  memory_size   = 1024

  layers = [local.powertools_layer_arn, local.pandas_layer_arn]

  tracing_config {
    mode = "Active"
  }

  environment {
    variables = merge(local.powertools_env_vars, {
      S3_BUCKET        = aws_s3_bucket.upload_bucket.id
      PROCESSED_PREFIX = var.processed_prefix
    })
  }
}

DynamoDB (Pay-Per-Request)

DynamoDB is my favourite database to use with AWS. It is truly serverless and tables are ready to use in seconds. It offers on-demand billing which means zero compute cost when idle:

resource "aws_dynamodb_table" "sales_data" {
  name         = "SalesData"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "PK"
  range_key    = "SK"

  attribute {
    name = "PK"
    type = "S"
  }

  attribute {
    name = "SK"
    type = "S"
  }

  # GSI for querying by date across all stores
  global_secondary_index {
    name            = "GSI1"
    hash_key        = "GSI1PK"
    range_key       = "GSI1SK"
    projection_type = "ALL"
  }
}

EventBridge (S3 to Step Functions)

Eventbridge is my favourite AWS service. It offers rules for reacting to events, pipes for bridging data across AWS services, and a nice scheduler. Here i’m using a simple rule that routes S3 uploads to the processing workflow:

resource "aws_cloudwatch_event_rule" "s3_upload" {
  name        = "capture-s3-uploads"
  description = "Capture all S3 object uploads"

  event_pattern = jsonencode({
    source      = ["aws.s3"]
    detail-type = ["Object Created"]
    detail = {
      bucket = {
        name = [aws_s3_bucket.upload_bucket.id]
      }
      object = {
        key = [{ prefix = var.upload_prefix }]
      }
    }
  })
}

resource "aws_cloudwatch_event_target" "step_function" {
  rule      = aws_cloudwatch_event_rule.s3_upload.name
  target_id = "UploadProcessorStepFunction"
  arn       = aws_sfn_state_machine.upload_processor.arn
  role_arn  = aws_iam_role.eventbridge_step_function_role.arn
}

Step Functions (Workflow Orchestration)

In many cases you want to tightly control and track the flow of processing in your app. AWS Step Function state machines are defined as JSON templates with Lambda ARNs injected:

resource "aws_sfn_state_machine" "upload_processor" {
  name     = "upload-processor"
  role_arn = aws_iam_role.step_function_role.arn

  definition = templatefile("${path.module}/../backend/state-machines/upload-processor.json", {
    process_upload_lambda_arn        = aws_lambda_function.process_upload.arn
    calculate_metrics_lambda_arn     = aws_lambda_function.calculate_metrics.arn
    write_metrics_lambda_arn         = aws_lambda_function.write_metrics.arn
    check_all_stores_lambda_arn      = aws_lambda_function.check_all_stores.arn
    sns_alerts_topic_arn             = aws_sns_topic.sales_alerts.arn
    daily_analysis_state_machine_arn = aws_sfn_state_machine.daily_analysis.arn
  })
}

S3 Bucket (Secure by Default)

S3 is at the core of storing data for so many apps today. My setup has Public access blocked, encryption enabled, and EventBridge notifications on:

resource "aws_s3_bucket_public_access_block" "upload_bucket_public_access_block" {
  bucket = aws_s3_bucket.upload_bucket.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_notification" "bucket_notification" {
  bucket      = aws_s3_bucket.upload_bucket.id
  eventbridge = true
}

The complete infrastructure includes:

17 Lambda functions
2 Step Functions state machines
API Gateway with 5 endpoints
DynamoDB table with GSI
S3 bucket with security policies
EventBridge rules
SNS topics
IAM roles with least-privilege policies

To set all this up there is no clicking through console pages and no manual configuration drift.

Key Takeaways

Serverless doesn't mean simple - it means you focus on business logic instead of infrastructure.
Parquet is worth the conversion - the great compression pays for itself in storage and query costs.
Hive partitioning enables scale - organize data for how it will be queried, not how it arrives.
Event-driven beats polling - let AWS route events instead of writing schedulers.
Pay-as-you-go works - for variable workloads, managed services beat reserved capacity.
Offer analytics options - different users have different needs; support both custom dashboards and BI tools.

Try It Yourself

The complete source code for my solution is available on GitHub, including:

Terraform infrastructure definitions
17 Lambda functions (Python 3.13)
React frontend application
Sample data generator
Quick Suite setup scripts

Deploy your own instance and start processing data in under 30 minutes.

Built with AWS Lambda, Step Functions, S3, DynamoDB, EventBridge, Bedrock, API Gateway, SNS, and optionally Quick Suite.

CLEANUP (IMPORTANT!!)

If you do end up deploying this yourself please understand some of the included resources will cost you a small amount of real money. Please don’t forget about it.

Please MAKE SURE TO DELETE the stack if you are no longer using it. Running terraform destroy can take care of this or you can delete the server in the AWS console.

Try the setup in your AWS account

You can clone the Github Repo and try this out in your own AWS account. The README.md file mentions any changes you need to make for it to work in your AWS account.

Please let me know if you have any suggestions or problems trying out this example project.

For more articles from me please visit my blog at Darryl's World of Cloud or find me on Bluesky, X, LinkedIn, Medium, Dev.to, or the AWS Community.

For tons of great serverless content and discussions please join the Believe In Serverless community we have put together at this link: Believe In Serverless Community

Aurora DSQL - Build A Serverless Multi-Region E-Commerce Platform

Darryl Ruggles — Tue, 18 Nov 2025 05:11:45 +0000

Introduction

I’ve always been a big fan of managed and truly serverless services offered by public cloud providers like AWS. I want to be able to prototype and build applications with as little infrastructure handling and management as possible. My time should be spent focusing on the business logic of the problem at hand.

I really like using services on AWS like Lambda, API Gateway, the Simple Queue Service (SQS), Simple Notification Service (SNS), and many others. For a database platform I have almost always used DynamoDB. DynamoDB tables can be provisioned and ready to use in seconds. The service is highly performant at any scale and you just pay for what you use. I don’t have to pay $100’s per month for something I may only use once a week.

In recent times we’ve had a resurgence of interest in SQL-based databases. Of course we all learned about databases with these but I’ve typically avoided them whenever possible due to all the setup and management required and how long it takes to start using these after you create them. I know the interface to and API for DynamoDB can be rather cryptic and difficult to get used to but it’s a very serverless offering - so right up my alley.

When AWS announced Aurora DSQL at re:Invent 2024, it really struck a chord with me and seemed to give me another big option to take advantage of. I was honestly quite disappointed when I started reading details about much of the functionality I was used to with SQL databases not being supported in DSQL. As time has passed I have understood why the DSQL team made many of the choices they did to get the performance and consistency they want. I have read a lot of articles and watched videos from Marc Brooker (Marc’s Blog) and others (AWS DSQL Blogs) and appreciate the work that went into DSQL and it’s innovative design.

Aurora DSQL is a multi-region distributed SQL database that provisions in under 60 seconds and bills only for actual usage. No instances to size, no standby replicas to pay for when you're not using them, and multi-region replication is built-in when you need it. Currently multi-region support only allow pairs of AWS regions in the same general parts of the world but the team is working on supporting more cases like pairs of regions much farther apart (think one in the US and one in Europe or something like that) and possible support for things like CDC (Change Data Capture) where changes are streamed via an interface. Aurora DSQL ensures that all reads and writes to any Regional endpoint are strongly consistent and durable. This is very tough to accomplish in a very fast and scalable fashion like they have accomplished.

The Kabob Store

I wanted to build a demo project (Github repo here) that I could expand on in later blogs and code repos. I have chosen to build “The Kabob Store” to start working with Aurora DSQL along with other AWS services. Who doesn’t like Kabobs and tasty Baclava anyway? This e-commerce platform is the start of my future Kabob empire but for now it’s a practical test: a fully functional e-commerce platform with menu browsing, cart management, order placement, and order history. It uses Aurora DSQL for data storage, Elastic Container Service (ECS) with Fargate for compute, and demonstrates whether DSQL can replace DynamoDB as the default choice for serverless applications that need relational data.

In the past I typically focused on using serverless compute via AWS Lambda for most projects. I think most people have come to the realization that there are many ways to solves problems and sticking to the same one for everything is not the best approach. I have spent a lot of time working with containers over the years - be it in local Kubernetes installs setup via kubeadm, cloud provider Kubernetes cluster like the Elastic Kubernetes Service (EKS) on AWS, and the Elastic Container Service (ECS) on AWS. I have seen that these work really well for many use cases.

I see the job of a solution architect as taking the requirements given for any problem and the boundaries set to go and sort through the vast set of available tools and platforms and build a solution that best meets the goals and budget. This doesn’t always mean using your favourite approaches and tools. For me this was almost always to use AWS serverless tools and event-driven architectures in the past. In the last few years I have been spending a lot more time mixing in things like container-based solutions, simple VM setups, and almost any approach that gets the job done.

I think one of the keys to allowing this flexibility is structuring business logic code so that it doesn’t know (or care) much about where it’s running and typically should not directly interact with most of the surrounding infrastructure. If you can setup your projects in this way it should be quite easy to move from running in AWS Lambda to running in Fargate on ECS to running straight is some VM.

The Kabob Store I present here is a full stack solution that includes a ReactJS front end. It’s not using all the latest front end tech - more a plain React Single Page App (SPA). I am more of a backend developer but did teach myself ReactJS a number of years ago and have built a few front end apps when needed.

Kabob Store Architecture

The Kabob Store uses containers on ECS Fargate rather than Lambda functions. This deserves explanation since I typically default to Lambda for serverless compute.

┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ React │────▶│ ALB │────▶│ FastAPI │
│ Frontend │ │ (Route) │ │ Backend │
└─────────────┘ └─────────────┘ └─────────────┘
│ │
└──────── ECS Fargate ──────────────────┤
│
┌──────▼──────┐
│Aurora DSQL │
│(Multi-Region)│
└─────────────┘

Why Containers Instead of Lambda?

For this project, containers provide flexibility. The FastAPI application runs in a container that could deploy to multiple runtimes:

The business logic doesn't care about the runtime. With minimal adapter code, the same application can deploy across all these platforms. This matters because project requirements change:

Development/staging: Fargate's simplicity wins (no servers to manage)

Production at scale: ECS on EC2 becomes more cost-effective (Fargate pricing is roughly 20-30% higher than equivalent EC2)

Lambda: Works well for this workload, but has 15-minute timeout limits and specific deployment constraints

For the Kabob Store, I chose Fargate for operational simplicity during development. If traffic scales significantly, migrating to ECS on EC2 workers requires no code changes. Just Terraform adjustments to swap Fargate launch type for EC2 launch type and add an Auto Scaling Group.

The principle: write business logic that's portable across runtimes. Choose the runtime based on current requirements, not because the code is locked into it.

Backend: FastAPI Without the ORM

The backend uses FastAPI with direct psycopg2 queries instead of an ORM. This keeps the business logic focused and portable. In the future I will move to using an ORM but for now I just wanted to keep it simple.

# Direct psycopg2 with parameterized queries
cursor.execute("""
    INSERT INTO orders (id, customer_name, customer_email, items, total_amount)
    VALUES (%s::UUID, %s, %s, %s::JSONB, %s)
    RETURNING *
""", (order_id, name, email, items_json, total))
result = cursor.fetchone()
conn.commit()

This code is runtime-agnostic. It works in:

A Fargate container (current deployment)
A Lambda function using container images
An EC2-based ECS service
A Kubernetes pod in EKS

The application doesn't use Fargate-specific features or Lambda-specific event handlers. The container listens on a port and handles HTTP requests. Where it runs is an infrastructure decision, not a code decision.

The Security Layer Cake

With great power comes great responsibility. An e-commerce platform needs good security, so I implemented some basic best practices but it’s just a start with much more to be done. As I evolve this project I will add more. For now we’re using Pydantic input validation.

Layer 1: Pydantic Validation with Custom Validators

class OrderCreate(BaseModel):
    customer_name: str = Field(..., min_length=2, max_length=100)
    customer_email: EmailStr  # Pydantic's built-in email validation
    items: List[OrderItemCreate]

    @validator('customer_email')
    def validate_email_not_disposable(cls, v):
        disposable_domains = ['tempmail.com', 'throwaway.email', '10minutemail.com']
        domain = v.split('@')[1].lower()
        if domain in disposable_domains:
            raise ValueError('Disposable email addresses are not allowed')
        return v

    @validator('customer_name')
    def validate_name(cls, v):
        if not re.match(r"^[a-zA-Z\s\-']+$", v):
            raise ValueError('Name contains invalid characters')
        return v

Layer 2: Client-Side Validation

The frontend validates inputs before submission, providing immediate user feedback:

// Name validation - letters, spaces, hyphens, apostrophes only
if (!/^[a-zA-Z\s\-']+$/.test(customerData.name)) {
  errors.name = 'Name can only contain letters, spaces, hyphens, and apostrophes';
}

// Email validation with TLD requirement
const emailRegex = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
if (!emailRegex.test(customerData.email)) {
  errors.email = 'Please enter a valid email address';
}

Layer 3: Request Middleware

@app.middleware("http")
async def validate_request(request: Request, call_next):
    suspicious_patterns = [
        '../',           # Path traversal
        '<script',       # XSS attempts
        'DROP TABLE',    # SQL injection
        '\x00',          # Null byte injection
    ]

    path = str(request.url)
    for pattern in suspicious_patterns:
        if pattern.lower() in path.lower():
            return JSONResponse(status_code=400,
                               content={"detail": "Invalid request"})

Layer 4: Parameterized Queries

All SQL queries use parameterization to prevent SQL injection:

# Never do this (even with validation)
query = f"INSERT INTO orders VALUES ('{order_id}', '{name}'...)"

# Always do this
cursor.execute(
    "INSERT INTO orders VALUES (%s::UUID, %s, %s, %s, %s)",
    (order_id, name, email, items_json, total)
)

Infrastructure as Code (With Terraform)

I am a very big proponent of using Infrastructure as Code (IaC) and my go-to tool for this is Terraform. Setting up all your resources in Terraform stacks makes it super easy to setup (and tear down) everything wherever you need. AWS services have very good Terraform support and even things like setting up multi-region DSQL clusters can be done via Terraform. Here I’m using the official terraform-aws-modules/rds-aurora DSQL module. The entire infrastructure is defined in Terraform, making it reproducible and version able.

module "dsql_primary" {
  source  = "terraform-aws-modules/rds-aurora/aws//modules/dsql"
  version = "~> 9.0"

  deletion_protection_enabled = false
  witness_region              = "us-west-2"
  create_cluster_peering      = true
  clusters                    = [module.dsql_secondary.arn]

  tags = {
    Name        = "${var.project_name}-dsql-primary"
    Environment = var.environment
  }
}

module "dsql_secondary" {
  source  = "terraform-aws-modules/rds-aurora/aws//modules/dsql"
  version = "~> 9.0"

  providers = {
    aws = aws.secondary  # us-east-2
  }

  deletion_protection_enabled = false
  witness_region              = "us-west-2"
  create_cluster_peering      = true
  clusters                    = [module.dsql_primary.arn]

  tags = {
    Name        = "${var.project_name}-dsql-secondary"
    Environment = var.environment
  }
}

The dsql module handles cluster peering automatically, creating a multi-region DSQL setup with strong consistency across regions. One terraform apply creates multi-region DSQL clusters (primary in us-east-1, secondary in us-east-2, witness in us-west-2).

My current store implementation involves setting up a VPC, Subnets, and all the other infrastructure needed to run the Elastic Container Service. The app stack is not running in multiple AWS regions though so it doesn’t really take advantage of the DSQL database being multi-region. In future versions I will implement true multi-region support for everything with duplicate application stacks in each region and Route53 failover routing.

Multi-Region DSQL Configuration (Not Really Utilized Yet)

The infrastructure creates DSQL clusters in multiple US regions (us-east-1 primary, us-east-2 secondary) with us-west-2 configured as the witness region using the official Terraform module. This provides data replication and disaster recovery capabilities within the US. Note that the witness region is just a configuration setting for maintaining quorum - there's no actual DSQL cluster in us-west-2, only in us-east-1 and us-east-2. However, the current application always connects to the primary cluster in us-east-1, regardless of where the user is located.

Aurora DSQL's multi-region setup is conceptually similar to DynamoDB Global Tables - both replicate data across multiple AWS regions with strong consistency and automatic failover. The key difference: DSQL gives you SQL with PostgreSQL compatibility, while Global Tables use DynamoDB's NoSQL model.

Important limitation: DSQL multi-region clusters are currently restricted to geographic groupings. You can link clusters within the US (us-east-1, us-east-2, us-west-2), within Europe (eu-west-1, eu-west-2, eu-west-3), or within Asia Pacific (ap-northeast-1, ap-northeast-2, ap-northeast-3), but not across continents. For true global data synchronization across continents, DynamoDB Global Tables remains the better choice.

Aurora DSQL's multi-region feature shines when you have a multi-region application within the same geographic area that can route users to their nearest cluster. In that scenario, East Coast US users could connect to us-east-1 while West Coast users connect to us-west-2, both accessing the same strongly consistent data with lower latency. The witness region maintains quorum for strong consistency.

For this initial demo application with a single-region deployment (all ECS tasks in us-east-1), the multi-region clusters provide excellent data protection and fast disaster recovery within the US, but we're not leveraging the performance benefits of local reads. A future version could deploy the application stack in multiple US regions with Route53 routing users to their nearest endpoint, fully utilizing DSQL's regional multi-region capabilities.

What's Next?

The Kabob Store is just the beginning. Here's what's on the roadmap:

Authentication: Adding AWS Cognito for user accounts and login
Observability: I plan to implement full OpenTelemetry observability for the platform.
Store Dashboard: Real-time order management interface for store staff with Server-Sent Events or Websockets for instant order notifications
Payments: Integrating Stripe for actual transactions
AI Ordering Agent: Conversational ordering interface using Amazon Bedrock AgentCore and Strands framework
Analytics: Building a QuickSight dashboard for business metrics

Prerequisites

If you’re going to setup the Kabob store demo code for yourself you will need the following:

AWS Account with admin permissions
Terraform >= 1.5.0
Docker for container builds
AWS CLI configured
~$2-3/day budget for testing

Try It Yourself

The entire project is open source. You can deploy your own Kabob Store:

# Clone the repo
git clone https://github.com/RDarrylR/kabob-store

# Deploy infrastructure
cd infrastructure
terraform init
terraform apply

# Build and push container images to ECR
# Then update ECS services to deploy
# See README.md for detailed deployment steps

# Visit your ALB URL and start ordering kabobs!

Conclusion

Aurora DSQL offers most of what I've been looking for: SQL with DynamoDB-like operational characteristics. It provisions in seconds, scales automatically, and bills only for usage. For the first time, I can choose SQL for a new project without accepting always-on infrastructure costs or extended provisioning times.

The container-based approach provides similar flexibility. The same application code can run on Lambda, Fargate, ECS on EC2, or EKS. I can choose whichever runtime fits the current requirements and cost profile. During development, Fargate eliminates server management. At scale, ECS on EC2 reduces costs. If requirements change, the code doesn't need to.

The Kabob Store demonstrates a straightforward architecture: runtime-portable business logic, parameterized SQL queries, explicit transaction boundaries, multi-layer validation, and scoped IAM permissions. The entire stack deploys with terraform apply and produces a multi-region e-commerce platform with data redundancy across US regions. When requirements change (more traffic, different cost targets, specific compliance needs), the code can move to different infrastructure without rewriting the business logic.

For my projects, the decision tree has expanded. As a solution architect I always want to have as many tools to choose from. DynamoDB remains the right choice when its data model fits naturally. Lambda remains the default for event-driven workloads. But when I need SQL with serverless economics, or containers that can move between runtimes, these are now viable options. The Kabob Store proves they work in practice.

CLEANUP (IMPORTANT!!)

If you do end up deploying the Kabob Store yourself please understand some of the included resources will cost you real money. For a short period of time it won’t be much but running the VPC and NAT Gateway will incur you daily charges. Please don’t forget about it.

Please MAKE SURE TO DELETE the stack if you are no longer using it. Running terraform destroy can take care of this or you can delete the server in the AWS console.

Try the setup in your AWS account

You can clone the Github Repo and try this out in your own AWS account. The README.md file mentions any changes you need to make for it to work in your AWS account.

Please let me know if you have any suggestions or problems trying out this example project.

For more articles from me please visit my blog at Darryl's World of Cloud or find me on Bluesky, X, LinkedIn, Medium, Dev.to, or the AWS Community.

For tons of great serverless content and discussions please join the Believe In Serverless community we have put together at this link: Believe In Serverless Community

My AWS Summit Toronto experience and GenAI

Darryl Ruggles — Sat, 06 Sep 2025 13:31:32 +0000

I have been experimenting and working with AWS for many years now. Way back when, I started out by purchasing a training course put out by Yan Cui (featuring his cat) and have learned so much since then. I was fascinated with the idea of the scale and flexibility possible using cloud services and the things you could do with serverless services like AWS Lambda and S3.

As my journey has progressed over the years I have interacted with so many amazing people and learned so much from them. I have joined a number of communities and regularly participate in online sessions, webinars, and other events to learn whatever I can. I have spent the most time in the last couple of years hanging out at the Believe In Serverless Community. It is a great place to discuss and share ideas on serverless approaches in the cloud and there are tons of great people hanging out there.

One thing I haven’t been able to do though is meet most of the people in the AWS (and related) communities in person. This past week I was able to attend the AWS Summit in Toronto, Canada and had a great experience meeting so many people and learning new things. The experience has really energized me to do more in the future. I have attended similar conferences in the past like Java One but am very glad to have had this experience.

The AWS Summit Toronto

Toronto is about 4 hours drive from Ottawa where I live but it’s the closest AWS Summit location to me. I will be attending AWS Re:Invent for the first time this year but when I saw the opportunity to take part in something earlier I decided it was a no brainer. I’m glad I did as I met many local people from the AWS Community and many of them may not be able to be in Las Vegas in December as well.

I have been super busy with a project at work the last couple of weeks and it was a great time to be able to head out and spend some time not thinking about my work projects. I was able to stay in Toronto the day before and after the summit and took these days to relax, do some workouts, and start to feel normal again.

I ended up picking up my badge for the summit the day before to save time on summit day as I had a full plan of talks to attend. I have never been in the Metro Toronto Convention Centre before and kind of got lost finding my way to the badge pickup location. I did eventually find it and it all felt a lot more real seeing the huge setup that was there for the AWS Partners Summit. I didn’t attend that this year but maybe next time as our company is part of the program.

I didn’t sleep super well on the night before the summit as I didn’t want to be late and miss anything. I had a much better journey on summit day and used the pathways above the rail tracks in the downtown area to go to directly to the Convention Centre. I was inside and waiting about 45 minutes before the first session I wanted to attend.

As I was researching the session catalog in the days before the summit it was clear there would be a real focus on GenAI topics. This is really no surprise as we’ve seen this from AWS (and the tech world in general) for a long time now. Many people don’t like this but it’s the reality of where we are at today and the sheer amount of interest (and promise) in this space. I was able to pick out 7 sessions I wanted to attend that didn’t overlap. Three of these were GenAI related, one was the keynote, one was related to my favourite subject of serverless computing on AWS and there were two others. I think I had a good mix of what interests me today.

It’s all about community and personal interactions

The real highlights for me from the AWS Summit Toronto were meeting and seeing presentations from AWS people who I have learned so much from over the years and meeting so many AWS community members.

Tracy Wang from AWS organized a mixer at the end of the summit day at a nearby hotel where I got to finally meet so many people from the local community. Many of these people live rather close to me but I have not met them in person. It’s just not the same meeting someone on Zoom or other online video platforms as seeing them in person. Meeting AWS Heroes like Cyril Bandolo and Andrew Brown was a real honor. I have spoken with them online before but in person is so much better. I also met so many other really interesting people that seem very passionate about AWS. I will probably miss mentioning some of them but meeting people like Cyril, Andrew, Rola Dali, Bansi Delwadia, Hubert Chan, Farah Abdirahman, Tracy Wang, Aaron Brighton, Varun Kumar Manik and more was a real honour and inspires me.

For my work with Ciena, we have many colleagues in locations around the world including in many cities in the US as well as our offices in Gurugram (Delhi), India. I have had the pleasure to travel to India a number of times to see our offices there and meet almost all the people I have worked with. These experiences have greatly enhanced my interactions with all these people on all the other times when we’re on opposite sides of the world working together on projects. Once you see people in person and see their mannerisms and how they respond in person it is so much easier.

Even in cases with people who I work with based at our offices in Ottawa, Canada so much has changed in the last number of years. I have been at this for many years and in the earlier part of my career I learned so much from the more experienced people in my company by just hanging out at the office and in the coffee rooms. I would not be the architect, developer and overall person I am now without these experiences. It has to be difficult for recent people entering our field. Real human interactions like meeting people in person is really needed to build good relationships IMO. People from the community coming together at events like the Toronto Summit is very important and we really need to try and do more of this.

Highlights from the sessions I attended

AWS is really doing an all-out effort to gain traction in the industry in the Generative AI space. Just a few years ago I was working on a more traditional AI/ML team in an MLOps role using AWS and Kubernetes. Back then Natural Language Processing (NLPs) were just one subfield we were working on that didn’t seem to have much more promise than others. Of course that all changed thanks to the development of transformers and the ability to train huge language models. The rest is history and the world has been transformed with GenAI.

DAT302 (Building cost-effective RAG-based apps with Amazon Aurora)

Many companies are starting out with GenAI by using Retrieval Augmented Generation (RAG) approaches where their specific data. Using RAG with generic Large Language Models makes it possible for companies to get real wins with GenAI approaches for minimal costs and effort. Using RAG along with Embedding Models, Vector databases and more can still really add up in costs if not optimized. The session DAT302 (Building cost-effective RAG-based apps with Amazon Aurora) presented by Shayon Sanyal and Aditya Samant included a number of good tips on optimizing spend with these approaches.

Some ideas included using multiple levels of models to reduce costs like first passing raw data to a smaller and cheaper model to summarize and then only passing this smaller set to the more expensive and powerful models. There were also specific settings with Aurora databases that work best with RAG.

SVS201-R (Building Production-ready Agentic AI Architectures with AWS Serverless)

This is honestly one of the two sessions I was most looking forward to. Anton Aleksandrov and Heeki Park are excellent presenters and I have watched and read so much of their content over the years. We had a great discussion about the approaches you can use for building Agents using tools like the Amazon Bedrock, the Strands SDK, AWS Lambda, the Elastic Container Service (ECS), Step Functions, and of course the new Bedrock AgentCore suite of tools.

Anton and Heeki discussed how agents communicate with each other, require memory to work together in an Agentic fashion and how Agents are not really all that different than micro-services. The audience got lots of high level ideas on how best to approach agents and all the tooling AWS offers to help. Seeing these guys in person and chatting with Anton were highlights for me.

The Keynote with Andrew Kent Warfield

I know some people find keynotes at conferences like this boring but I was glad to able to be in the room for this one. It was great to hear how someone who grew up in Canada has become a key part of AWS. As with most of the rest of the conference there was a heavy focus in the Keynote on the various GenAI products AWS has been working on but there were also some nice stories about the role AWS has played with many Canadian companies and driving innovation here.

SEC302 (Building a Secure EKS Framework For Financial Services)

I have been setting up and administrating Kubernetes clusters and using the Elastic Kubernetes Service (EKS) on AWS for a number of years now and we run multiple clusters in production. I wanted to go to this session to understand what the recommended approaches are to make sure everything is secure. Raj Bagwe and Preetam Rebello did a great job walking through many best practices to ensure security in your installs.

Items like using private connectivity, enabling control plane logs, using encryption for everything possible, setting up clusters using the least privileges possible, using EKS Pod Identities, Network Policies, OPA, and other tools, looking at using Service Meshes like Istio, and so many other great tips were presented. In my case it was great to see that we are already in really good shape as we pretty much used all the approaches discussed. Once the slides are available I will go over everything in detail to make sure and I did provide lots of comments and feedback during this session.

ANT401 (Architectural Patterns for Near Real-time analytics on AWS)

In the GenAI and telecom equipment space where my work is focused we have a number of use cases for ingesting large amounts of network data and the using it to build applications. I have worked on multiple project with this kind of approach and have used Kinesis streams as well as custom solutions.

Mazrim Mehrtens and Vinodh Sadayamuthu discussed some best practices they recommend to AWS customers including the use of Kinesis or Amazon Managed Streaming for Apache Kafka (MSK) for ingestion and then tools like the Managed Service for Apache Flink for super performant data processing. We have looked at using Flink in the past but based on this presentation I plan to take another look. There were some great tips and I am really looking forward to looking at the slide and the example github repos that were discussed.

In the GenAI space one real use case I see here is building models that can predict or detect anomalies by doing inferences with real time data on GenAI models and also the constant retraining and updating of these models. Being able to use some of the techniques discussed by Mazrim and Vinodh could really help.

AIM301 (Deploy, secure, and observe Production-grade AI Agents with AgentCore)

This is the other of the two sessions I was most looking forward to. The announcement of the preview for AgentCore at the AWS Summit New York was a super interesting development for me. There are so many tools and libraries out there for working with GenAI but there really are a smaller number that seem ready to be able to help deploy real Agentic AI apps at scale with proper security and reliability.

Fei Yuan and Satveer Khurpa did a great job describing some key components of AgentCore in detail. Fei especially did a great job white-boarding some of the key components and flow of the AgentCore Identity piece. In order for Agentic AI apps to be production ready I believe security has to be front and centre. I did find the details of how the Identity module works to be rather complicated but got a really good idea of how it works from this session. I also found out they have an AgentCore Discord server so I joined that after the session and am really looking forward to interacting with Fei and other people working on AgentCore.

On the way back from Toronto on the train I was watching more videos on AgentCore and am planning on my next project migrating some existing code to run with AgentCore. I really think this set of tools (along with the Strands SDK) will play a big part for the future of companies deploying Agentic Apps in production.

SVS203 (Unlock Productivity: Tools and Techniques for Serverless Developers)

At heart I am a serverless-first person and you can't think about serverless on AWS without people like Julian Wood. I was not able to attend the full session as it overlapped with the AgentCore one but I was lucky enough to hear Julian Wood go over some great improvements in the Serverless space on AWS. Chris McPeek and Janak Agarwal also were part of this session and demonstrated many of the other super useful features AWS has been putting out in the serverless space.

At the end of the session, Julian showed a great demo of using Q to build a serverless app for him including SAM templates, running locally via SAM and also building a front-end for an app in seconds. It is truly amazing (and somewhat scary) what GenAI tools like Q can do and they are only going to get better.

A highlight for me was getting to chat with Julian for a few minutes after the session. I have attended so many streams where he presents all the latest in serverless and has all the key people from AWS with him.

Tips for others on attending summits and conferences

Some tips for people attending this summit (which likely applies to others as well)

If you’re travelling, book your (refundable) hotel well in advance. I’m kicking myself as I had a number of decent priced options back in June when I started planning but didn’t book and got stuck with a rather expensive (and non-refundable) booking maybe due to the Toronto International Film Festival being at the same time.
Scout out the summit location ahead of time if possible. I’m glad I did this the day before on badge pickup day and knew much better where to go on summit day.
Bring your own refreshments if you can. There were some provided as part of the summit but others were available at a rather high cost.
Plan out which sessions you want to see and make sure to show up at the room 30 minutes before as all the ones I attended were full by the time the door opened to let us in.
Ask questions during the sessions, share your insights, and introduce yourself to the presenters after if there is time and opportunity. I really got a lot out of personally saying hi to so many people that I have attended online sessions before and read content from.
Attend the keynote! It may sound boring but keynotes at conferences like this are always great to see and you get a much better sense of the overall purpose and direction of the company and domain.

Conclusion

When I first thought about attending the AWS Summit Toronto I wasn't really sure if it was would be worth it. Since the summit itself was free I wasn't expecting too much compared to something like Re:Invent but I was wrong on that front. I gained so much in knowledge and interactions on this trip and hope to do this again in the future. Now I'm looking even more forward to Re:Invent where it will be all of what I experienced but times 3 or 4. I'm also invigorated to do more and learn more.

For more articles from me please visit my blog at Darryl's World of Cloud or find me on X, LinkedIn, Medium, Dev.to, or the AWS Community.

For tons of great serverless content and discussions please join the Believe In Serverless community we have put together at this link: Believe In Serverless Community