Forem: Clariza Look

Building a Cross-Account S3 to Azure Backup System with ECS Fargate

Clariza Look — Mon, 06 Oct 2025 07:27:10 +0000

The Challenge

When you need to transfer large database backups (85GB+) from AWS S3 to Azure Blob Storage as part of the Disaster Recovery measure, AWS Lambda's 15-minute timeout becomes a significant constraint. The backup files take 60-90 minutes to transfer, making Lambda unsuitable for this use case.

REPORT RequestId: b000000b6a-000-0000-0000-0a0f0bf0c000 Duration: 900000.00 ms Billed Duration: 901344 ms Memory Size: 1024 MB Max Memory Used: 156 MB Init Duration: 1343.19 ms Status: Task timed out

Additionally, in enterprise environments with multi-account AWS setups, you often need to:

Build and store Docker images in a central CICD account
Deploy infrastructure to separate environment accounts (dev, staging, prod)
Maintain proper security boundaries between accounts

This article walks through building a production-ready backup offloading system that handles these challenges using ECS Fargate and cross-account deployment patterns.

Architecture Overview

[Account A: CICD Account]
├── ECR Repository (Docker Images)
├── CodeBuild (CI/CD)
└── CodePipeline

[Account B: Target Account for Deployment]
├── S3 Bucket (Backup Files)
├── SQS Queue
├── Lambda (ECS Trigger)
├── ECS Fargate (Transfer Task)
└── Secrets Manager (Azure Credentials)

Flow:
S3 → Event Notification → SQS → Lambda → ECS Fargate → Azure Blob Storage

Why This Architecture?

ECS Fargate over Lambda:

No timeout limits (Lambda: 15 min max)
Handles files of any size
Streaming transfer (no local storage limits)
Better for long-running tasks

Multi-Account Pattern:

Centralized Docker image management
Clear separation of concerns
Reusable images across environments
Enhanced security boundaries

Implementation Guide

Part 1: Docker Container for Backup Processing

The heart of the system is a Python container that streams files from S3 to Azure:

docker/backup_processor/Dockerfile:

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

CMD ["python", "app.py"]

docker/backup_processor/requirements.txt:

azure-storage-blob==12.19.0
boto3

Key features of the transfer application:

Retrieves Azure credentials from AWS Secrets Manager
Streams data from S3 (no local storage needed)
Parallel uploads to Azure (8x concurrency)
Emits CloudWatch metrics for monitoring
Handles errors gracefully

Part 2: Multi-Account Docker Image Management

In Account A (CICD account) - Build and Push:

The CI/CD pipeline builds the Docker image and pushes it to ECR in Account A:

# buildspec.yml
pre_build:
  commands:
    # Login to Account A ECR
    - aws ecr get-login-password --region $REGION | docker login --username AWS --password-stdin $ACCOUNT_A.dkr.ecr.$REGION.amazonaws.com

    # Build and push image
    - cd docker/backup_processor
    - docker build -t backup-processor:$IMAGE_TAG .
    - docker tag backup-processor:$IMAGE_TAG $ACCOUNT_A.dkr.ecr.$REGION.amazonaws.com/backup-processor:$IMAGE_TAG
    - docker push $ACCOUNT_A.dkr.ecr.$REGION.amazonaws.com/backup-processor:$IMAGE_TAG

    # Set image URI for deployment
    - export DOCKER_IMAGE_URI="$ACCOUNT_A.dkr.ecr.$REGION.amazonaws.com/backup-processor:$IMAGE_TAG"

ECR Repository Policy in Account A:

Allow Account B to pull images:

{
  "Version": "2008-10-17",
  "Statement": [{
    "Sid": "AllowAccountBPull",
    "Effect": "Allow",
    "Principal": {
      "AWS": "arn:aws:iam::ACCOUNT_B:root"
    },
    "Action": [
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchGetImage",
      "ecr:BatchCheckLayerAvailability"
    ]
  }]
}

Part 3: ECS Infrastructure in Account B

CDK Stack Structure:

class BackupOffloaderStack(Stack):
    def __init__(self, scope, construct_id, **kwargs):
        super().__init__(scope, construct_id, **kwargs)

        # Get Docker image URI from environment (set by buildspec)
        docker_image_uri = os.environ.get('DOCKER_IMAGE_URI')

        # Create ECS cluster
        cluster = ecs.Cluster(self, "BackupCluster",
            cluster_name="backup-processor-cluster",
            vpc=self.vpc
        )

        # Create task definition
        task_definition = ecs.FargateTaskDefinition(
            self, "BackupTaskDef",
            memory_limit_mib=2048,
            cpu=1024,
            task_role=task_role,
            execution_role=execution_role
        )

        # Add container referencing cross-account image
        container = task_definition.add_container(
            "BackupContainer",
            image=ecs.ContainerImage.from_registry(docker_image_uri),
            logging=ecs.LogDrivers.aws_logs(
                stream_prefix="backup-processor",
                log_retention=logs.RetentionDays.ONE_MONTH
            ),
            environment={
                "AZURE_SECRET_NAME": "azure-credentials",
                "LOG_LEVEL": "INFO"
            }
        )

Critical: Cross-Account ECR Permissions

The ECS Task Execution Role in Account B needs permission to pull from Account A's ECR:

execution_role = iam.Role(
    self, "ExecutionRole",
    assumed_by=iam.ServicePrincipal("ecs-tasks.amazonaws.com"),
    managed_policies=[
        iam.ManagedPolicy.from_aws_managed_policy_name(
            "service-role/AmazonECSTaskExecutionRolePolicy"
        )
    ],
    inline_policies={
        "CrossAccountECRAccess": iam.PolicyDocument(
            statements=[
                iam.PolicyStatement(
                    effect=iam.Effect.ALLOW,
                    actions=["ecr:GetAuthorizationToken"],
                    resources=["*"]
                ),
                iam.PolicyStatement(
                    effect=iam.Effect.ALLOW,
                    actions=[
                        "ecr:BatchCheckLayerAvailability",
                        "ecr:GetDownloadUrlForLayer",
                        "ecr:BatchGetImage"
                    ],
                    resources=[
                        f"arn:aws:ecr:region:ACCOUNT_A:repository/backup-processor"
                    ]
                )
            ]
        )
    }
)

Part 4: Event-Driven Trigger System

S3 → SQS → Lambda → ECS Flow:

# Lambda function to trigger ECS tasks
trigger_code = """
import boto3
import json

ecs_client = boto3.client('ecs')

def lambda_handler(event, context):
    for record in event.get('Records', []):
        message = json.loads(record['body'])

        # Extract S3 details
        s3_event = message['Records'][0]
        bucket = s3_event['s3']['bucket']['name']
        key = s3_event['s3']['object']['key']

        # Trigger ECS task
        ecs_client.run_task(
            cluster='backup-processor-cluster',
            taskDefinition='BackupTaskDef',
            launchType='FARGATE',
            networkConfiguration={
                'awsvpcConfiguration': {
                    'subnets': SUBNET_IDS,
                    'assignPublicIp': 'ENABLED'
                }
            },
            overrides={
                'containerOverrides': [{
                    'name': 'BackupContainer',
                    'environment': [
                        {'name': 'S3_BUCKET', 'value': bucket},
                        {'name': 'S3_KEY', 'value': key}
                    ]
                }]
            }
        )
"""

Part 5: Deployment Pipeline

Complete CI/CD Flow:

Pre-build Phase (Account A):
- Build Docker image
- Push to Account A ECR
- Set DOCKER_IMAGE_URI environment variable
Assume Role to Account B:
- Assume deployment role in Account B
- Switch AWS credentials
Build Phase (Account B):
- CDK synth (reads DOCKER_IMAGE_URI)
- CDK deploy infrastructure
- ECS tasks pull image from Account A ECR

Key Configuration:

# buildspec.yml
phases:
  pre_build:
    commands:
      # Build in Account A
      - docker build -t $IMAGE_NAME .
      - docker push $ACCOUNT_A_ECR/$IMAGE_NAME:$TAG
      - export DOCKER_IMAGE_URI="$ACCOUNT_A_ECR/$IMAGE_NAME:$TAG"

      # Assume role to Account B
      - aws sts assume-role --role-arn $ACCOUNT_B_ROLE ...
      - export AWS_ACCESS_KEY_ID=...

  build:
    commands:
      # Deploy to Account B (uses DOCKER_IMAGE_URI)
      - cdk deploy

Monitoring and Operations

CloudWatch Metrics

Track transfer success and performance:

cloudwatch.put_metric_data(
    Namespace='BackupOffloader',
    MetricData=[
        {'MetricName': 'BackupsCopied', 'Value': 1, 'Unit': 'Count'},
        {'MetricName': 'TransferDuration', 'Value': duration, 'Unit': 'Seconds'},
        {'MetricName': 'BytesTransferred', 'Value': bytes, 'Unit': 'Bytes'}
    ]
)

Log Groups

ECS Task Logs:
Lambda Trigger Logs:

Troubleshooting Cross-Account Issues

CannotPullContainerError (403 Forbidden):

Verify ECR repository policy in Account A allows Account B
Check execution role has cross-account ECR permissions
Confirm image exists and tag is correct

Security Considerations

Credentials Management:
- Azure credentials in Secrets Manager (encrypted)
- IAM roles for AWS resource access (no long-term keys)
Network Security:
- ECS tasks in private subnets with NAT gateway
- Security groups restrict traffic
Cross-Account Access:
- Least-privilege IAM policies
- Specific ECR repository access only
- Regular credential rotation

Key Takeaways

ECS Fargate removes Lambda timeout constraints for long-running data transfers
Multi-account patterns enhance security and enable centralized image management
Cross-account ECR access requires proper permissions on both accounts
Event-driven architecture scales automatically with backup frequency
Infrastructure as Code makes the system reproducible and maintainable

Conclusion

This architecture demonstrates how to build production-ready, cross-account data transfer systems on AWS. By combining ECS Fargate's flexibility with proper multi-account IAM configuration, you can handle large-scale data movements that exceed Lambda's limitations while maintaining security boundaries between environments.

The pattern shown here is applicable beyond backup systems - any long-running data processing task that needs to span AWS accounts can benefit from this approach.

Understanding `curl`: When to Choose Silent Automation Over Verbose Debugging

Clariza Look — Mon, 11 Aug 2025 07:16:09 +0000

When working with APIs and web services, curl is often your best friend. But did you know that how you use curl can dramatically change both what you see and what you can do with the results? Today, let's explore two completely different approaches to the same HTTP request and understand when to use each.

What is API Gateway?

Before diving into curl commands, let's quickly understand what we're testing. Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. Think of it as the front door to your applications - it handles all the HTTP requests and routes them to your backend services.

Key Benefits:

Traffic Management - Handle millions of concurrent API calls
Security - Built-in authorization, authentication, and API keys
Monitoring - CloudWatch integration for metrics and logging
Caching - Reduce latency and backend load
Throttling - Protect your backend from traffic spikes

Creating API Gateway with CDK Python

Here's how you can create an API Gateway using AWS CDK with Python:

from aws_cdk import (
    Stack,
    aws_apigateway as apigateway,
    aws_lambda as _lambda,
    CfnOutput
)
from constructs import Construct

class ApiGatewayStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # Create a Lambda function (your backend)
        api_lambda = _lambda.Function(
            self, "ApiLambda",
            runtime=_lambda.Runtime.PYTHON_3_9,
            handler="lambda_function.lambda_handler",
            code=_lambda.Code.from_asset("lambda"),
        )

        # Create the API Gateway
        api = apigateway.RestApi(
            self, "MyApi",
            rest_api_name="My Service API",
            description="This service serves my application.",
            default_cors_preflight_options=apigateway.CorsOptions(
                allow_origins=apigateway.Cors.ALL_ORIGINS,
                allow_methods=apigateway.Cors.ALL_METHODS,
            )
        )

        # Create API Gateway integration with Lambda
        lambda_integration = apigateway.LambdaIntegration(
            api_lambda,
            request_templates={"application/json": '{ "statusCode": "200" }'}
        )

        # Add a resource and method
        api.root.add_method("GET", lambda_integration)

        # Add a resource path
        items = api.root.add_resource("items")
        items.add_method("GET", lambda_integration)
        items.add_method("POST", lambda_integration)

        # Output the API Gateway URL - This is what we'll test!
        CfnOutput(
            self, "ApiGatewayUrl",
            value=api.url,
            description="API Gateway endpoint URL",
            export_name=f"{self.stack_name}-ApiGatewayUrl"
        )

This CDK code creates:

A Lambda function as your backend service
An API Gateway that routes requests to the Lambda
Most importantly: An output called ApiGatewayUrl that exports the endpoint

This exported URL is exactly what our curl commands will be testing!

The Setup: Testing an API Gateway

Now that you understand what we're working with, imagine you're building a CI/CD pipeline that needs to verify your API Gateway is working after deployment. You have two options for testing the endpoint:

Option 1: The Silent Operator

response_code=$(curl -s -o /dev/null -w "%{http_code}" "$API_URL" --connect-timeout 10 --max-time 30 2>/dev/null || echo "000")

Option 2: The Verbose Detective

curl -v $API_URL

Same endpoint, completely different philosophies. Let's dive into what makes each special.

The Silent Operator: Automation's Best Friend

What It Does

The silent approach is like a ninja—it gets in, gets the job done, and gets out without making a fuss. Here's what each part does:

curl -s                    # Silent mode (no progress bars or chatter)
     -o /dev/null         # Throw away the response body
     -w "%{http_code}"    # Only output the HTTP status code
     --connect-timeout 10 # Give up connecting after 10 seconds
     --max-time 30        # Total operation timeout of 30 seconds
     2>/dev/null          # Hide any error messages
     || echo "000"        # If everything fails, return "000"

What You Get

That's it. Just a clean, three-digit HTTP status code that tells you exactly what happened:

200 = Success
404 = Not found (but API Gateway is working)
403 = Forbidden (API Gateway working, needs auth)
000 = Complete failure (connection issues)

Perfect For:

✅ CI/CD Pipelines - Clean, parseable output
✅ Health Check Scripts - Just need pass/fail
✅ Monitoring Systems - Programmatic status checking
✅ Automation - When you need to make decisions based on response codes

The Verbose Detective: When You Need Answers

What It Does

The verbose approach is like a chatty detective who tells you everything they're thinking. It shows you the entire conversation between your computer and the server.

What You Get

* Trying 203.0.113.42:443...
* Connected to example.execute-api.us-east-1.amazonaws.com (203.0.113.42) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=*.execute-api.us-east-1.amazonaws.com
*  start date: Dec  5 00:00:00 2023 GMT
*  expire date: Jan  3 23:59:59 2025 GMT
*  subjectAltName: host "example.execute-api.us-east-1.amazonaws.com" matched cert's "*.execute-api.us-east-1.amazonaws.com"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M02
*  SSL certificate verify ok.
> GET /prod/ HTTP/1.1
> Host: example.execute-api.us-east-1.amazonaws.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Date: Sun, 11 Aug 2025 04:30:00 GMT
< Content-Type: application/json
< Content-Length: 36
< Connection: keep-alive
< x-amzn-RequestId: example-request-id
< x-amzn-ErrorType: UnknownOperationException
< x-amz-apigw-id: example-apigw-id
< 
{"message":"Missing Authentication Token"}
* Connection #0 to host example.execute-api.us-east-1.amazonaws.com left intact

Perfect For:

✅ Debugging SSL/TLS Issues - See certificate details and handshake
✅ Network Troubleshooting - See DNS resolution and connection details
✅ API Development - Understand exactly what headers are being sent/received
✅ Security Analysis - Verify certificates and encryption protocols
✅ Learning - Understand how HTTP really works under the hood

Real-World Example: CI/CD Pipeline

Let's see how this plays out in a real AWS CodeBuild pipeline that tests an API Gateway after deployment:

The Pipeline Context

# In your CI/CD pipeline, after deploying your API Gateway:
# 1. Deploy CloudFormation stack with API Gateway
# 2. Deploy application to ECS
# 3. Test if API Gateway is reachable ← We are here
# 4. Continue to production (or fail if test fails)

Using the Silent Approach

# Get just the status code for automated decision making
response_code=$(curl -s -o /dev/null -w "%{http_code}" "$API_URL" --connect-timeout 10 --max-time 30 2>/dev/null || echo "000")

case $response_code in
    200)
        echo "✅ SUCCESS: API Gateway working perfectly"
        ;;
    404)
        echo "✅ SUCCESS: API Gateway deployed (just no root handler)"
        ;;
    403|401)
        echo "✅ SUCCESS: API Gateway deployed (needs authentication)"
        ;;
    000)
        echo "❌ FAILURE: Cannot reach API Gateway"
        exit 1  # Fail the pipeline
        ;;
    *)
        echo "✅ SUCCESS: API Gateway responding (HTTP $response_code)"
        ;;
esac

This gives you clean, actionable results:

✅ SUCCESS: API Gateway deployed (just no root handler)

When You'd Use Verbose Instead

If your pipeline fails and you need to understand why:

# Run this manually to debug
curl -v https://your-api-gateway-url.amazonaws.com/prod/

You might discover:

DNS resolution issues
SSL certificate problems
Firewall blocking connections
Incorrect URL construction
Authentication header requirements

The Key Takeaway

Silent curl is your automation workhorse—fast, reliable, and perfect for scripts that need to make decisions based on HTTP status codes.

Verbose curl is your debugging superhero—it tells you everything that's happening so you can diagnose and fix issues.

Quick Decision Matrix

Use Case	Command	Why
CI/CD health checks	Silent	Clean output, fast execution, easy to parse
Monitoring scripts	Silent	Programmatic status checking
API debugging	Verbose	See full request/response details
SSL troubleshooting	Verbose	Certificate and handshake details
Network diagnostics	Verbose	DNS, connection, and timing info
Learning HTTP	Verbose	See what's really happening

Pro Tips

Combine them: Use silent in automation, verbose for debugging when things go wrong
Add timeouts: Always use --connect-timeout and --max-time in production scripts
Handle failures: Use || echo "000" to get a predictable response on complete failures
Parse carefully: Clean up response codes with tr -d '\n' if you get extra characters

The next time you reach for curl, ask yourself: "Do I need to make a decision, or do I need to understand what's happening?" The answer will guide you to the right approach.

Embracing CDK: Use Case to Move from CloudFormation YAML to CDK?

Clariza Look — Wed, 06 Aug 2025 02:33:26 +0000

The Dilemma Every Engineering Team Faces

Picture this scenario: Your team has been successfully running AWS infrastructure for years using CloudFormation YAML templates. They work, they're familiar, and everyone knows how to maintain them. Meanwhile, your newer projects are built with AWS CDK, complete with comprehensive unit testing, type safety, and modern development practices.

Sound familiar? You're not alone. Many engineering teams find themselves at this crossroads, managing a hybrid infrastructure landscape where legacy CloudFormation templates coexist with modern CDK applications. The question isn't whether both approaches work—they do. The question is whether this fragmentation is serving your team's long-term interests.

Understanding the Players: CloudFormation vs CDK

Before diving into the migration story, let's clarify what we're talking about.

What is AWS CloudFormation?

AWS CloudFormation is Amazon's native Infrastructure as Code (IaC) service that allows you to define your AWS resources using JSON or YAML templates. Think of it as a blueprint that tells AWS exactly what infrastructure to create.

Here's a simple CloudFormation YAML example creating an S3 bucket:

AWSTemplateFormatVersion: '2010-09-09'
Description: 'Simple S3 bucket example'

Resources:
  MyS3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: my-example-bucket
      VersioningConfiguration:
        Status: Enabled
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true

Outputs:
  BucketName:
    Description: 'Name of the S3 bucket'
    Value: !Ref MyS3Bucket
    Export:
      Name: MyS3BucketName

CloudFormation Characteristics:

Declarative: You describe what you want, not how to create it
YAML/JSON syntax: Human-readable but can become verbose
AWS native: Built specifically for AWS resources
Template-based: Static files that AWS reads and executes

What is AWS CDK?

AWS CDK (Cloud Development Kit) is a newer approach that lets you define cloud infrastructure using familiar programming languages like TypeScript, Python, Java, or C#. Instead of writing YAML or JSON, you write actual code.

Here's the same S3 bucket example using CDK (TypeScript):

import { Stack, StackProps } from 'aws-cdk-lib';
import { Bucket, BlockPublicAccess, BucketAccessControl } from 'aws-cdk-lib/aws-s3';
import { CfnOutput } from 'aws-cdk-lib';
import { Construct } from 'constructs';

export class MyInfrastructureStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    // Create S3 bucket with sensible defaults
    const myBucket = new Bucket(this, 'MyS3Bucket', {
      bucketName: 'my-example-bucket',
      versioned: true,
      blockPublicAccess: BlockPublicAccess.BLOCK_ALL
    });

    // Export bucket name for other stacks
    new CfnOutput(this, 'BucketName', {
      value: myBucket.bucketName,
      exportName: 'MyS3BucketName',
      description: 'Name of the S3 bucket'
    });
  }
}

CDK Characteristics:

Programmatic: Uses actual programming languages with IDE support
Higher-level abstractions: Smart defaults and best practices built-in
Type safety: Compile-time error checking
Testable: Full unit testing capabilities like any other code
Generates CloudFormation: CDK compiles down to CloudFormation templates

The Key Differences at a Glance

Aspect	CloudFormation YAML	AWS CDK
Language	YAML/JSON markup	TypeScript, Python, Java, C#
IDE Support	Basic syntax highlighting	Full IntelliSense, autocomplete, refactoring
Testing	Limited to template validation	Full unit testing with mocking
Linting	cfn-lint for template validation	ESLint, Prettier, language-specific linters
Reusability	Copy/paste templates	Object-oriented inheritance and composition
Learning Curve	Learn YAML syntax + AWS resources	Use existing programming skills
Error Detection	Runtime (during deployment)	Compile-time + runtime

A More Complex Example: The Difference Becomes Clear

Let's look at a more realistic example—creating an API Gateway that triggers a Lambda function and stores data in an S3 bucket. This shows how the complexity gap widens with real infrastructure.

CloudFormation YAML approach:

AWSTemplateFormatVersion: '2010-09-09'
Description: 'API Gateway with Lambda and S3'

Resources:
  # S3 Bucket for data storage
  DataBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: my-api-data-bucket
      VersioningConfiguration:
        Status: Enabled
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true

  # IAM Role for Lambda execution
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: S3Access
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - s3:GetObject
                  - s3:PutObject
                  - s3:DeleteObject
                Resource: !Sub '${DataBucket}/*'

  # Lambda Function
  ApiLambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: api-handler
      Runtime: nodejs18.x
      Handler: index.handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Code:
        ZipFile: |
          exports.handler = async (event) => {
            return {
              statusCode: 200,
              body: JSON.stringify({ message: 'Hello from Lambda!' })
            };
          };
      Environment:
        Variables:
          BUCKET_NAME: !Ref DataBucket

  # Permission for API Gateway to invoke Lambda
  LambdaInvokePermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref ApiLambdaFunction
      Action: lambda:InvokeFunction
      Principal: apigateway.amazonaws.com
      SourceArn: !Sub '${ApiGateway}/*/*'

  # API Gateway REST API
  ApiGateway:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: MyAPI
      Description: API for Lambda integration
      EndpointConfiguration:
        Types:
          - REGIONAL

  # API Gateway Resource
  ApiResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      RestApiId: !Ref ApiGateway
      ParentId: !GetAtt ApiGateway.RootResourceId
      PathPart: data

  # API Gateway Method
  ApiMethod:
    Type: AWS::ApiGateway::Method
    Properties:
      RestApiId: !Ref ApiGateway
      ResourceId: !Ref ApiResource
      HttpMethod: POST
      AuthorizationType: NONE
      Integration:
        Type: AWS_PROXY
        IntegrationHttpMethod: POST
        Uri: !Sub 'arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ApiLambdaFunction.Arn}/invocations'

  # API Gateway Deployment
  ApiDeployment:
    Type: AWS::ApiGateway::Deployment
    DependsOn:
      - ApiMethod
    Properties:
      RestApiId: !Ref ApiGateway
      StageName: prod

Outputs:
  ApiGatewayUrl:
    Description: API Gateway endpoint URL
    Value: !Sub 'https://${ApiGateway}.execute-api.${AWS::Region}.amazonaws.com/prod'
    Export:
      Name: ApiGatewayUrl

  BucketName:
    Description: S3 bucket name
    Value: !Ref DataBucket
    Export:
      Name: DataBucketName

  LambdaFunctionArn:
    Description: Lambda function ARN
    Value: !GetAtt ApiLambdaFunction.Arn
    Export:
      Name: LambdaFunctionArn

CDK approach (TypeScript):

import { Stack, StackProps, CfnOutput } from 'aws-cdk-lib';
import { RestApi, LambdaIntegration } from 'aws-cdk-lib/aws-apigateway';
import { Function, Runtime, Code } from 'aws-cdk-lib/aws-lambda';
import { Bucket, BlockPublicAccess } from 'aws-cdk-lib/aws-s3';
import { Construct } from 'constructs';

export class ApiStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    // S3 Bucket for data storage
    const dataBucket = new Bucket(this, 'DataBucket', {
      bucketName: 'my-api-data-bucket',
      versioned: true,
      blockPublicAccess: BlockPublicAccess.BLOCK_ALL
    });

    // Lambda Function
    const apiFunction = new Function(this, 'ApiLambdaFunction', {
      functionName: 'api-handler',
      runtime: Runtime.NODEJS_18_X,
      handler: 'index.handler',
      code: Code.fromInline(`
        exports.handler = async (event) => {
          return {
            statusCode: 200,
            body: JSON.stringify({ message: 'Hello from Lambda!' })
          };
        };
      `),
      environment: {
        BUCKET_NAME: dataBucket.bucketName
      }
    });

    // Grant Lambda permissions to access S3 bucket
    dataBucket.grantReadWrite(apiFunction);

    // API Gateway
    const api = new RestApi(this, 'ApiGateway', {
      restApiName: 'MyAPI',
      description: 'API for Lambda integration'
    });

    // Create API resource and method
    const dataResource = api.root.addResource('data');
    dataResource.addMethod('POST', new LambdaIntegration(apiFunction));

    // Outputs
    new CfnOutput(this, 'ApiGatewayUrl', {
      value: api.url,
      exportName: 'ApiGatewayUrl',
      description: 'API Gateway endpoint URL'
    });

    new CfnOutput(this, 'BucketName', {
      value: dataBucket.bucketName,
      exportName: 'DataBucketName',
      description: 'S3 bucket name'
    });

    new CfnOutput(this, 'LambdaFunctionArn', {
      value: apiFunction.functionArn,
      exportName: 'LambdaFunctionArn',
      description: 'Lambda function ARN'
    });
  }
}

The difference is striking:

CloudFormation: ~105 lines of YAML with manual IAM role creation, permissions, and resource linking
CDK: ~50 lines of TypeScript with automatic IAM role generation and permission management

The CDK version automatically creates the Lambda execution role, sets up the proper IAM policies for S3 access, configures API Gateway permissions, and handles the deployment stage. The dataBucket.grantReadWrite(apiFunction) line alone replaces 20+ lines of IAM policy configuration in CloudFormation.

Deployment Process

The Hidden Costs of Infrastructure Fragmentation

Two Worlds, Double the Overhead

When your infrastructure spans both CloudFormation YAML and CDK, you're essentially maintaining two different technology stacks:

Dual toolchains: Different testing frameworks, deployment pipelines, and development workflows
Split expertise: Team members specializing in either YAML or CDK, creating knowledge silos
Inconsistent practices: Different approaches to testing, monitoring, and troubleshooting
Onboarding complexity: New team members need to learn both approaches

The Testing Gap

Here's where the pain really shows. Your CDK projects likely have:

Comprehensive unit tests validating resource configurations
Integration tests ensuring components work together
Automated validation in CI/CD pipelines
Rapid feedback loops for developers

Meanwhile, your CloudFormation YAML templates probably have:

Manual deployment processes
Limited or no automated testing
Post-deployment validation (if any)
Slower feedback cycles when issues arise

This isn't just a technical debt issue—it's a reliability and velocity problem.

The Strategic Case for CDK Migration

1. Standardization as a Force Multiplier

When every project follows the same patterns, magic happens:

Knowledge Transfer Accelerates: A developer who masters testing on one CDK project can immediately apply that knowledge to any other project. No context switching between YAML syntax and TypeScript/Python.

Best Practices Propagate: That clever testing pattern you developed for your API Gateway? It can be instantly applied across all projects, not just the CDK ones.

Tooling Investment Pays Off: Every improvement to your CDK testing framework, every CI/CD optimization, benefits your entire infrastructure portfolio.

2. Operational Excellence Through Consistency

Consider these operational benefits:

Unified Monitoring: All projects can leverage the same observability patterns, making it easier to understand system behavior across your entire infrastructure.

Consistent Debugging: When something goes wrong at 2 AM, you want your on-call engineer using familiar tools and approaches, regardless of which project is failing.

Predictable Maintenance: Security updates, dependency management, and refactoring become routine when you're working with a single technology stack.

3. The Compound Effect of Testing and Code Quality

Here's what comprehensive testing and linting really delivers:

// CDK with TypeScript linting and testing
test('API Gateway integrates correctly with Load Balancer', () => {
  const template = Template.fromStack(stack);

  template.hasResourceProperties('AWS::ApiGateway::VpcLink', {
    TargetArns: [{ Ref: 'NetworkLoadBalancer' }]
  });
});

// ESLint catches issues before deployment
const api = new RestApi(this, 'MyApi', {
  restApiName: 'my-api',  // ESLint enforces naming conventions
  description: 'API for Lambda integration'  // Required by custom linting rules
});

CloudFormation Linting:

# Basic template validation
cfn-lint template.yml
# Checks for syntax errors and basic AWS resource validation

CDK Linting & Quality Tools:

# TypeScript compilation catches type errors
npm run build

# ESLint for code quality and style
npm run lint

# Prettier for consistent formatting
npm run format

# Unit tests with coverage
npm run test

# CDK-specific validation
cdk synth  # Validates and generates CloudFormation

When you can validate your infrastructure changes before deployment with comprehensive linting, type checking, and testing, you move faster. When you move faster with confidence, you innovate more. When you innovate more, you deliver better outcomes for your users.

Making the Migration Case: A Real-World Example

Let's walk through a practical scenario. Imagine you have a legacy CloudFormation template managing your API infrastructure. It's deployed manually, tested manually, and updated carefully.

Now consider the CDK equivalent:

Before: CloudFormation YAML

# 200+ lines of YAML
# Manual testing
# No type safety
# Documentation in separate files
# Manual parameter management

After: CDK with Testing

export class ApiInfrastructureStack extends Stack {
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);

    // Self-documenting, type-safe infrastructure
    const cluster = new ecs.Cluster(this, 'ApiCluster', {
      vpc: props.vpc,
      containerInsights: true
    });

    const loadBalancer = new elbv2.NetworkLoadBalancer(this, 'ApiNLB', {
      vpc: props.vpc,
      internetFacing: false
    });

    // Outputs that dependent stacks can safely import
    new CfnOutput(this, 'LoadBalancerDNS', {
      value: loadBalancer.loadBalancerDnsName,
      exportName: `${this.stackName}-LoadBalancerDNS`
    });
  }
}

The CDK version isn't just more testable—it's more readable, more maintainable, and less error-prone.

The Migration Strategy That Actually Works

Phase 1: Parallel Implementation (Weeks 1-2)

Create the CDK equivalent while keeping your existing CloudFormation template running. This isn't about big-bang replacement—it's about building confidence.

Phase 2: Test Everything (Week 3)

Implement the comprehensive testing that makes CDK valuable:

Unit tests for resource configuration
Integration tests for component interaction
End-to-end tests for user-facing functionality

Phase 3: Validate in Non-Production (Weeks 4-5)

Deploy the CDK version alongside your existing infrastructure in non-production environments. Compare outputs, validate behavior, and build team confidence.

Phase 4: Production Cutover (Week 6)

With comprehensive testing and non-production validation complete, the production migration becomes a low-risk operation.

ROI: When Does Migration Pay Off?

The mathematics are compelling:

Initial Investment: 4-6 weeks of development time
Ongoing Benefits:

50% faster infrastructure changes (thanks to testing confidence)
75% reduction in deployment-related incidents
100% improvement in developer onboarding speed for infrastructure work
Infinite improvement in sleep quality for your on-call engineers

But the real ROI isn't just in time saved—it's in opportunities created. When your infrastructure is reliable and rapidly changeable, your team can focus on building features instead of fighting deployment issues.

The Counterarguments (And Why They Don't Hold Up)

"Our CloudFormation templates work fine": Yes, they work. But working and optimal are different things. Your flip phone worked fine too, but you probably upgraded for good reasons.

"Migration is risky": True, but so is maintaining fragmented infrastructure. The gradual migration approach outlined above minimizes risk while maximizing learning.

"We don't have time": You don't have time not to do this. Every day you spend maintaining two different infrastructure approaches is time you're not spending on features that matter to your users.

The Bottom Line

Infrastructure standardization isn't just about choosing between CloudFormation and CDK—it's about choosing between fragmentation and focus. When your team can channel all their infrastructure expertise into a single, well-tested approach, everyone wins.

Your legacy CloudFormation templates served you well, but they don't have to define your future. The question isn't whether you can afford to migrate to CDK—it's whether you can afford not to.

The path forward is clear: embrace standardization, invest in testing, and build infrastructure that accelerates your team rather than slowing it down. Your future self (and your on-call schedule) will thank you.

Ready to start your CDK migration journey? Begin with a single, non-critical CloudFormation template and apply the gradual migration strategy outlined above. Small steps, big wins.

Troubleshooting AWS API Gateway VPC Link with Network Load Balancer

Clariza Look — Fri, 11 Jul 2025 11:30:35 +0000

Building secure, private API architectures on AWS can be deceptively complex. What appears to be a straightforward integration between API Gateway, VPC Links, and Network Load Balancers often reveals multiple layers of configuration challenges that aren't immediately obvious from the documentation.

This post chronicles a real-world troubleshooting journey that started with what seemed like a simple infrastructure deployment and evolved into a deep investigation across Lambda runtimes, API Gateway authentication, and AWS networking internals.

Whether setting up our first private API integration or debugging existing infrastructure, this troubleshooting story will help us understand common pitfalls and their solutions in AWS private networking architectures.

Goal

The goal was to create a secure, private API architecture with the following requirements:

Private API Gateway accessible only within the VPC A
Secure authentication using API keys and custom authorization logic
High availability through load balancing across multiple containers
Private networking with no internet exposure
Scalable backend using Fargate containerized application

Target Architecture

┌─────────────────────────────────────────────────────────────────┐
│                           VPC                                   │
│                                                                 │
│  ┌─────────────────┐    ┌──────────────────┐                    │
│  │   EC2 Client    │    │  API Gateway     │                    │
│  │   (Testing)     │───▶│  using VPC link  │                    │
│  └─────────────────┘    └──────────────────┘                    │
│                                   │                             │
│                          ┌────────▼────────┐                    │
│                          │  Private API    │                    │
│                          │    Gateway      │                    │
│                          │                 │                    │
│                          │ ┌─────────────┐ │                    │
│                          │ │   Lambda    │ │                    │
│                          │ │ Authorizer  │ │                    │
│                          │ └─────────────┘ │                    │
│                          └────────┬────────┘                    │
│                                   │                             │
│                              ┌────▼────┐                        │
│                              │VPC Link │                        │
│                              └────┬────┘                        │
│                                   │                             │
│                          ┌────────▼────────┐                    │
│                          │Internal Network │                    │
│                          │ Load Balancer   │                    │
│                          │   (Port 80)     │                    │
│                          └────────┬────────┘                    │
│                                   │                             │
│              ┌────────────────────┼────────────────────┐        │
│              │                    │                    │        │
│    ┌─────────▼─────────┐ ┌────────▼────────┐ ┌────────▼────────┐│
│    │   Fargate Task    │ │   Fargate Task  │ │   Fargate Task  ││
│    │   (Container 1)   │ │   (Container 2) │ │   (Container 3) ││
│    │                   │ │                 │ │                 ││
│    │ ┌───────────────┐ │ │ ┌─────────────┐ │ │ ┌─────────────┐ ││
│    │ │ Application   │ │ │ │Application  │ │ │ │Application  │ ││
│    │ │   Server      │ │ │ │   Server    │ │ │ │   Server    │ ││
│    │ └───────────────┘ │ │ └─────────────┘ │ │ └─────────────┘ ││
│    └───────────────────┘ └─────────────────┘ └─────────────────┘│
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

External Internet ❌ (No direct access - Private API only)
Public subnet access ❌ (No direct access - except who are inside the private subnets of the vpc)

The target architecture involved:

Private API Gateway (accessible only within VPC)
VPC Link for private integration
Internal Network Load Balancer
Fargate containers as targets
Custom Lambda authorizer for API key validation used by the API Gw

The Challenge

What seemed like a straightforward setup quickly revealed multiple layers of complexity. Despite following AWS documentation and best practices, I still encountered a series of issues that required deep troubleshooting across different AWS services.

Initially, our infrastructure appeared to work perfectly during development. Direct access to the NLB returned responses immediately, target groups were healthy, and everything looked correct.

> curl -v NLB-1xxxxx234567897.elb.ap-southeast-2.amazonaws.com

* Host NLB-1xxxxx234567897.elb.ap-southeast-2.amazonaws.com:80 was resolved.

* IPv6: (none)
* IPv4: xx.xxx.xxx.00, xx.xxx.x.111
*   Trying xx.xxx.x.111:80...
* Connected to NLB-1xxxxx234567897.elb.ap-southeast-2.amazonaws.com (xx.xxx.xx.00) port 80
* using HTTP/1.x
> GET / HTTP/1.1
> Host: NLB-1xxxxx234567897.elb.ap-southeast-2.amazonaws.com 
> User-Agent: curl/x.x.xx
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 200 OK
< x-correlation-id: aas19175-1b77-3333-0000-123456787
< X-Powered-By: Express
< Content-Type: text/html; charset=utf-8
< Content-Length: 5
< ETag: W/"5-fvYUSdz1234556789"
< Date: Fri, 11 Jul 2025 08:48:13 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
< 
* Connection #0 to host NLB-1xxxxx234567897.elb.ap-southeast-2.amazonaws.com  left intact
hello

However, when tying to access the API GW through the VPC Link integration (I created an EC2 instance that can access the private subnets of the VPC and curl from there), I encountered various issues that weren't immediately obvious.

The First Challenge: Lambda Runtime Issues

Our Lambda authorizer had been working reliably for over three years, originally built with Node.js 12.x when the infrastructure was first deployed. However, when I needed to redeploy the stack, I encountered an immediate issue:

Error: Cannot find module 'aws-sdk'

The Root Cause: AWS had deprecated Node.js 12.x runtime, and our deployment pipeline automatically upgraded to Node.js 18.x. However, Node.js 18.x runtime doesn't include AWS SDK v2 by default - a breaking change that affected our existing Lambda code.

Legacy Code (Node.js 12.x - worked for 3 years):

var AWS = require('aws-sdk');
var ssm = new AWS.SSM({region: 'ap-southeast-2'});
const paramValue = await ssm.getParameter({...}).promise();

Updated Code (Node.js 18.x - required migration):

const { SSMClient, GetParameterCommand } = require('@aws-sdk/client-ssm');
const ssmClient = new SSMClient({ region: 'ap-southeast-2' });
const paramValue = await ssmClient.send(new GetParameterCommand({...}));

Lesson Learned: AWS runtime deprecations can affect long-running infrastructure. When Lambda runtimes are deprecated, existing functions continue to work, but new deployments require code updates to use supported runtimes and SDK versions.

The Second Challenge: API Key Configuration

Even after fixing the Lambda runtime issue, I continued getting 403 Forbidden responses.

> curl -H "x-api-key: xxxxxx" -H "accept: application/json" -X GET "https://12345apigateway.execute-api.ap-southeast-2.amazonaws.com/" -i


HTTP/1.1 403 Forbidden
Server: Server
Date: Thu, 10 Jul 2025 06:05:15 GMT
Content-Type: application/json
Content-Length: 24
Connection: keep-alive
x-amzn-RequestId: 123ff3234-0000-xxx
x-amzn-ErrorType: ForbiddenException
x-amz-apigw-id: 1er12344556=

{"message":"Forbidden"}

The Lambda authorizer logs showed it was finding the API key and returning an "Allow" policy, but API Gateway was still rejecting requests.

The Missing Piece: API Usage Plans require explicit API key associations!

While I had created both the Usage Plan and API Key using the Cloudformation template, I was missing the crucial link between them:

ApiUsagePlanKey:
  Type: AWS::ApiGateway::UsagePlanKey
  Properties:
    KeyId: !Ref ApiKey
    KeyType: API_KEY
    UsagePlanId: !Ref UsagePlan

This is a common gotcha that many developers encounter when setting up API Gateway authentication.

The Third Challenge: VPC Link Connectivity

With authentication working, I faced a new issue: VPC Link was timing out after 11 seconds with "Internal server error" messages.

> curl -H "x-api-key: xxxxxx" -H "accept: application/json" -X GET "https://12345apigateway.execute-api.ap-southeast-2.amazonaws.com/" -i


HTTP/1.1 500 Internal Server Error
Server: Server
Date: Fri, 11 Jul 2025 08:03:15 GMT
Content-Type: application/json
Content-Length: 36
Connection: keep-alive
x-amzn-RequestId: 34d5467890-15db-4a40-9acb-dr123456789
x-amzn-ErrorType: InternalServerErrorException
x-amz-apigw-id: EEgTmDq5556tYRwd=

{"message": "Internal server error"}

This was particularly puzzling because:

✅ Direct NLB access worked perfectly
✅ VPC Link showed as "AVAILABLE"
✅ Target groups were healthy
✅ Security groups seemed correct

The Mysterious Timeout Pattern

The pattern was clear: Direct connectivity worked flawlessly, but VPC Link couldn't reach the same NLB endpoints.

Deep Dive into Network Security

I spent considerable time investigating network-level issues:

Security Group Configuration

Initially, I configured the NLB security group to allow traffic from specific subnet CIDR blocks where the NLB was deployed. This seemed logical but didn't work.

# This approach failed
SecurityGroupIngress:
  - IpProtocol: tcp
    FromPort: 80
    ToPort: 80
    CidrIp: xx.x.x.0/22  # NLB subnet 1
  - IpProtocol: tcp
    FromPort: 80
    ToPort: 80
    CidrIp: xx.x.x.0/22  # NLB subnet 2

The Debugging Process

I systematically verified:

Network ACLs: Properly configured with allow rules
DNS Resolution: NLB resolved to correct private IPs
Target Health: All Fargate containers responding
VPC Endpoints: API Gateway VPC endpoint working correctly
Direct Connectivity: Manual curl tests from EC2 instances succeeded

The Breakthrough Discovery

After extensive troubleshooting, I discovered that only allowing 0.0.0.0/0 in the NLB security group made VPC Link work. This was concerning from a security perspective but provided a crucial clue about the traffic flow.

The Real Solution: AWS Documentation

While preparing to contact AWS Support about this networking puzzle, I discovered the official solution in AWS documentation:

For NLBs used with VPC Link, you should disable security group evaluation.

This is documented in the AWS API Gateway Developer Guide under point 4b.

So I updated my CDK for the creating the NLB:

#PrivateLink traffic enforcement
NetworkLoadBalancer:
  Type: AWS::ElasticLoadBalancingV2::LoadBalancer
  Properties:
    Name: !Join ['', [!Ref ServiceName, NetworkLoadBalancer]]
    Type: network
    Scheme: internal
    Subnets: !Split
      - ","
      - Fn::ImportValue: !Sub ${VpcStack}-PrivateSubnetIDs
    SecurityGroups:
      - !Ref NetworkLoadBalancerSecurityGroup
    # This property controls PrivateLink traffic enforcement
    EnforceSecurityGroupInboundRulesOnPrivateLinkTraffic: "off"
    LoadBalancerAttributes:
      - Key: load_balancing.cross_zone.enabled
        Value: "true"
      - Key: access_logs.s3.enabled
        Value: "false"
      - Key: deletion_protection.enabled
        Value: "false"

How it looks like in the console.

Why This Works

VPC Link uses AWS-managed infrastructure that operates outside your VPC's IP ranges. Rather than trying to guess or discover these IP ranges, AWS recommends disabling security group evaluation for NLBs used in VPC Link integrations.

Security is still maintained through:

Private subnet placement (no internet gateway routes)
API Gateway authentication layers
Network ACLs (if configured restrictively)
Application-level security controls

Key Takeaways

1. AWS SDK Version Compatibility

When upgrading Lambda runtimes, be aware of SDK version changes. Node.js 18.x requires AWS SDK v3.

2. API Gateway Usage Plan Associations

Creating an API key and usage plan isn't enough - you must explicitly associate them with AWS::ApiGateway::UsagePlanKey.

3. VPC Link Security Groups

For NLBs used with VPC Link, follow AWS guidance and disable security group evaluation rather than trying to configure specific IP ranges.

4. Systematic Troubleshooting

When facing complex networking issues:

Verify each component independently
Test direct connectivity vs. service-mediated connectivity
Check AWS documentation for service-specific configuration patterns
Consider contacting AWS Support for service-level investigations

5. Documentation First

Before implementing complex workarounds, always check the official AWS documentation. The solution to our VPC Link issue was documented but easy to miss.

Final Architecture

Our final working configuration:

API Gateway with proper Usage Plan and API key associations
Lambda authorizer using AWS SDK v3
VPC Link connecting to NLB with security group evaluation disabled
Internal NLB in private subnets
Fargate containers handling application logic

This setup provides a secure, scalable private API architecture that follows AWS best practices.

Conclusion

Complex AWS networking scenarios often require understanding the interaction between multiple services. What seemed like a simple VPC Link + NLB integration revealed several layers of configuration requirements, from Lambda runtime compatibility to service-specific security patterns.

The key to successful troubleshooting is methodical verification of each component, careful reading of AWS documentation, and understanding that AWS services sometimes have specific configuration patterns that differ from general networking principles.

When in doubt, the AWS documentation and support team are invaluable resources for understanding the intended architecture patterns for complex service integrations.

Building a Scalable WAF Log Pipeline: From AWS WAF to Cortex XDR with CDK

Clariza Look — Tue, 24 Jun 2025 02:22:44 +0000

Introduction

As organizations increasingly rely on AWS Web Application Firewall (WAF) to protect their applications, the need for effective log management and security analytics becomes critical. In this post, we'll walk through building a complete, production-ready pipeline that automatically processes WAF logs from CloudWatch to S3 and integrates with Palo Alto Cortex XDR for advanced security analytics.

What is Cortex XDR?

Cortex XDR (Extended Detection and Response) is Palo Alto Networks' cloud-native security platform that provides comprehensive threat detection, investigation, and response capabilities. It's designed to:

Unified Security Operations:

Correlate data across endpoints, networks, and cloud environments
Detect advanced threats using machine learning and behavioral analytics
Automate incident response with playbooks and orchestration
Provide centralized visibility into your entire security ecosystem

Why Cortex XDR for WAF Logs:

Web attack detection: Identifies application-layer attacks, bot traffic, and malicious patterns
Threat intelligence integration: Correlates WAF events with global threat data
Automated blocking: Can trigger automated firewall rules based on WAF patterns
Compliance support: Helps meet security logging and monitoring requirements

For organizations serious about application security, integrating WAF logs with Cortex XDR transforms raw log data into actionable security intelligence.

The Challenge

WAF logs provide invaluable security insights, but managing them effectively presents several challenges:

Volume: WAF can generate massive amounts of log data
Cost: Storing logs in CloudWatch can become expensive at scale
Analysis: Raw logs need to be processed and analyzed for security insights
Integration: Security teams need logs in their SIEM/SOAR platforms

Our Solution Architecture

We'll build an automated pipeline using AWS CDK that:

Captures WAF logs from CloudWatch Logs
Processes logs via Lambda function
Stores compressed logs in S3 with lifecycle management
Notifies Cortex XDR via SQS for real-time analysis
Provides secure access through IAM roles

Architecture Diagram

Data Flow:

WAF generates security logs from web traffic
CloudWatch Logs captures and stores log events
Lambda processes and compresses logs for efficient storage
S3 stores compressed logs with automatic lifecycle transitions
SQS notifies Cortex XDR of new log files for real-time analysis

Implementation with AWS CDK

Let's build this infrastructure using AWS CDK in Python:

Key Architectural Decisions

Before diving into the implementation, it's worth explaining a crucial design choice we made regarding log processing timing.

Real-time vs Batch Processing

When configuring CloudWatch Logs subscription filters, you have two main options:

Real-time processing: Logs are sent to Lambda immediately as they arrive
Batch processing: Logs are buffered and sent in 5-minute intervals

We chose real-time processing for several critical reasons:

The Hidden Cost Most Analyses Miss

Many cost comparisons focus only on Lambda execution costs, but there's a crucial piece missing from traditional analysis - CloudWatch Logs Insights charges! This oversight completely changes the cost equation.

What Traditional Analysis Misses:

The 5-minute batch approach requires CloudWatch Logs Insights queries to retrieve and batch the logs before sending to Lambda. Most analyses only compare:

✅ Lambda invocation costs
✅ Lambda compute costs
✅ S3 PUT request costs
❌ CloudWatch Logs Insights scanning charges (the biggest expense!)

5-Minute Batch Approach (Hidden Costs):

Lambda invocations: 8,640/month × $0.0000002 = $0.001
Lambda compute: Lower cost ✅
S3 PUTs: Fewer PUTs ✅
CloudWatch Logs Insights: 8,640 queries × data scanned ❌

Real-time Subscription Approach:

Lambda invocations: 720,000/month × $0.0000002 = $0.14
Lambda compute: Higher cost ❌
S3 PUTs: More PUTs ❌
CloudWatch Logs Insights: $0 ✅

Security Response Time: WAF logs contain potential security threats and attack patterns. In cybersecurity, every minute counts. Real-time processing ensures that:

Suspicious activity is detected immediately
Security teams can respond to active attacks quickly
Cortex XDR can trigger automated responses without delay

Operational Visibility: Real-time logs provide immediate insight into:

Application performance issues
Traffic anomalies
DDoS attack patterns
Bot detection events

Cost vs. Value Trade-off:

When you factor in CloudWatch Logs Insights charges, real-time processing becomes significantly MORE cost-effective than batch processing. Here's the actual cost breakdown:

Estimated Daily Savings
For moderate WAF traffic (1GB logs/day):

Approach	Lambda Invocations	Lambda Duration	CW API	Total/Day
Real-time	$0.005	$0.015	$0	$0.02
Hourly	$0.0001	$0.008	$0.50	$0.51

Key Insights:

CloudWatch API Penalty: Hourly batching requires CloudWatch Logs API calls to query and retrieve logs, costing $0.50/day - 25x more than the entire real-time processing cost.
Real-time processing eliminates API costs by streaming logs directly via subscription filters, avoiding expensive CloudWatch queries entirely.

The analysis shows real-time processing is 96% cheaper ($0.02 vs $0.51/day) while providing:

Immediate threat detection
Real-time security response capabilities
Better error isolation and recovery
No CloudWatch API scanning charges

Monthly savings: $14.70 ($0.60 vs $15.30) - real-time pays for itself and saves money!

Cortex XDR Integration:

Security platforms like Cortex XDR are designed for real-time threat detection. Delayed log ingestion can:

Miss time-sensitive attack patterns
Reduce the effectiveness of machine learning models
Impact incident response capabilities

This architectural choice reflects our security-first approach, prioritizing threat detection speed over minor cost optimizations.

Project Structure

waf-log-pipeline/
├── app.py
├── requirements.txt
├── waf_log_pipeline/
│   ├── __init__.py
│   └── waf_log_pipeline_stack.py
└── tests/
    └── unit/
        └── test_waf_log_pipeline_stack.py

Core CDK Stack

import aws_cdk as cdk
from aws_cdk import (
    Stack,
    aws_s3 as s3,
    aws_lambda as _lambda,
    aws_iam as iam,
    aws_logs as logs,
    aws_sqs as sqs,
    aws_s3_notifications as s3n,
    CfnOutput,
    Duration,
    RemovalPolicy
)

class WafLogPipelineStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # S3 Bucket for log storage with lifecycle management
        self.waf_logs_bucket = s3.Bucket(
            self, "WafLogsBucket",
            bucket_name="company-waf-logs-for-cortex",
            encryption=s3.BucketEncryption.S3_MANAGED,
            versioning=False,
            block_public_access=s3.BlockPublicAccess.BLOCK_ALL,
            lifecycle_rules=[
                s3.LifecycleRule(
                    id="waf-logs-lifecycle",
                    enabled=True,
                    transitions=[
                        s3.Transition(
                            storage_class=s3.StorageClass.INFREQUENT_ACCESS,
                            transition_after=Duration.days(30)
                        ),
                        s3.Transition(
                            storage_class=s3.StorageClass.GLACIER,
                            transition_after=Duration.days(90)
                        ),
                        s3.Transition(
                            storage_class=s3.StorageClass.DEEP_ARCHIVE,
                            transition_after=Duration.days(365)
                        )
                    ],
                    expiration=Duration.days(2555)  # 7 years
                )
            ],
            removal_policy=RemovalPolicy.DESTROY
        )

        # Dead Letter Queue for failed messages
        cortex_dlq = sqs.Queue(
            self, "CortexXDRNotificationDLQ",
            queue_name="cortex-xdr-waf-logs-notifications-dlq",
            retention_period=Duration.days(14)
        )

        # SQS Queue for Cortex XDR notifications
        self.cortex_notification_queue = sqs.Queue(
            self, "CortexXDRNotificationQueue",
            queue_name="cortex-xdr-waf-logs-notifications",
            visibility_timeout=Duration.minutes(15),
            dead_letter_queue=sqs.DeadLetterQueue(
                max_receive_count=3,
                queue=cortex_dlq
            )
        )

        # Lambda function for log processing
        self.waf_log_processor = _lambda.Function(
            self, "WafLogProcessor",
            runtime=_lambda.Runtime.PYTHON_3_12,
            handler="index.handler",
            code=_lambda.Code.from_inline(self._get_lambda_code()),
            timeout=Duration.minutes(5),
            memory_size=512,
            environment={
                "S3_BUCKET": self.waf_logs_bucket.bucket_name,
                "S3_PREFIX": "raw"
            }
        )

        # IAM permissions for Lambda
        self.waf_logs_bucket.grant_write(self.waf_log_processor)

        # CloudWatch Logs subscription filter
        self.subscription_filter = logs.SubscriptionFilter(
            self, "WafLogSubscriptionFilter",
            log_group=logs.LogGroup.from_log_group_name(
                self, "WafLogGroup", "aws-waf-logs-company"
            ),
            destination=logs_destinations.LambdaDestination(self.waf_log_processor),
            filter_pattern=logs.FilterPattern.all_events()
        )

        # S3 bucket notifications to SQS
        self.waf_logs_bucket.add_event_notification(
            s3.EventType.OBJECT_CREATED,
            s3n.SqsDestination(self.cortex_notification_queue),
            s3.NotificationKeyFilter(prefix="raw/", suffix=".jsonl.gz")
        )

        # IAM role for Cortex XDR access
        self.cortex_xdr_role = iam.Role(
            self, "CortexXDRRole",
            role_name="CortexXDR-WAFLogs-AssumedRole",
            assumed_by=iam.ServicePrincipal("cortex.paloaltonetworks.com"),
            external_ids=["your-external-id-here"],
            inline_policies={
                "CortexXDRS3Access": iam.PolicyDocument(
                    statements=[
                        iam.PolicyStatement(
                            effect=iam.Effect.ALLOW,
                            actions=["s3:GetObject"],
                            resources=[f"{self.waf_logs_bucket.bucket_arn}/*"]
                        ),
                        iam.PolicyStatement(
                            effect=iam.Effect.ALLOW,
                            actions=["s3:ListBucket"],
                            resources=[self.waf_logs_bucket.bucket_arn]
                        )
                    ]
                ),
                "CortexXDRSQSAccess": iam.PolicyDocument(
                    statements=[
                        iam.PolicyStatement(
                            effect=iam.Effect.ALLOW,
                            actions=[
                                "sqs:ReceiveMessage",
                                "sqs:DeleteMessage",
                                "sqs:ChangeMessageVisibility"
                            ],
                            resources=[self.cortex_notification_queue.queue_arn]
                        )
                    ]
                )
            }
        )

    def _get_lambda_code(self) -> str:
        return '''
import json
import boto3
import gzip
import base64
from datetime import datetime
import os

s3_client = boto3.client('s3')

def handler(event, context):
    """Process CloudWatch Logs and store in S3."""

    s3_bucket = os.environ['S3_BUCKET']
    s3_prefix = os.environ['S3_PREFIX']

    # Parse CloudWatch Logs event
    cw_data = event['awslogs']['data']
    compressed_payload = base64.b64decode(cw_data)
    uncompressed_payload = gzip.decompress(compressed_payload)
    log_data = json.loads(uncompressed_payload)

    # Process log events
    processed_logs = []
    for log_event in log_data['logEvents']:
        try:
            # Parse WAF log (already in JSON format)
            waf_log = json.loads(log_event['message'])
            processed_logs.append(json.dumps(waf_log))
        except json.JSONDecodeError:
            # Handle non-JSON log entries
            processed_logs.append(json.dumps({
                "timestamp": log_event['timestamp'],
                "message": log_event['message']
            }))

    # Create filename with timestamp
    timestamp = datetime.utcnow().strftime('%Y%m%d-%H%M%S-%f')[:-3]
    filename = f"{s3_prefix}/waf-logs-{timestamp}.jsonl.gz"

    # Compress and upload to S3
    if processed_logs:
        content = '\\n'.join(processed_logs)
        compressed_content = gzip.compress(content.encode('utf-8'))

        s3_client.put_object(
            Bucket=s3_bucket,
            Key=filename,
            Body=compressed_content,
            ContentType='application/gzip',
            ContentEncoding='gzip'
        )

        print(f"Uploaded {len(processed_logs)} log entries to s3://{s3_bucket}/{filename}")

    return {
        'statusCode': 200,
        'body': json.dumps(f'Processed {len(processed_logs)} log entries')
    }
'''

Testing Infrastructure

Create comprehensive but simple tests to validate your infrastructure:

import aws_cdk as core
from aws_cdk import assertions
from waf_log_pipeline.waf_log_pipeline_stack import WafLogPipelineStack

def test_s3_bucket_exists():
    """Test that S3 bucket is created."""
    app = core.App()
    stack = WafLogPipelineStack(app, "test-stack")
    template = assertions.Template.from_stack(stack)

    template.resource_count_is("AWS::S3::Bucket", 1)

def test_lambda_function_exists():
    """Test that Lambda function is created."""
    app = core.App()
    stack = WafLogPipelineStack(app, "test-stack")
    template = assertions.Template.from_stack(stack)

    template.resource_count_is("AWS::Lambda::Function", 2)  # Main + CDK helper

def test_sqs_queues_exist():
    """Test that SQS queues are created."""
    app = core.App()
    stack = WafLogPipelineStack(app, "test-stack")
    template = assertions.Template.from_stack(stack)

    template.resource_count_is("AWS::SQS::Queue", 2)  # Main + DLQ

def test_cortex_role_exists():
    """Test that Cortex XDR role is created."""
    app = core.App()
    stack = WafLogPipelineStack(app, "test-stack")
    template = assertions.Template.from_stack(stack)

    template.has_resource_properties("AWS::IAM::Role", {
        "RoleName": "CortexXDR-WAFLogs-AssumedRole"
    })

Deployment

Deploy your infrastructure with these simple commands:

# Install dependencies
pip install -r requirements.txt

# Bootstrap CDK (first time only)
cdk bootstrap

# Deploy the stack
cdk deploy

# Run tests
python -m pytest tests/ -v

Cortex XDR Integration

Once deployed, configure Cortex XDR to consume the logs:

1. Get Integration Details

# Retrieve configuration values
aws cloudformation describe-stacks \
  --stack-name waf-log-pipeline \
  --query 'Stacks[0].Outputs'

2. Configure in Cortex XDR

Navigate to: Settings → Data Sources → Add Data Source → Amazon S3

Configuration:

SQS URL: Use the queue URL from stack outputs
Role ARN: Use the Cortex role ARN from outputs
External ID: Your configured external ID
Log Type: Generic
Log Format: JSON
Compression: gzip
Vendor: AWS
Product: WAF

Key Features & Benefits

Cost Optimization

S3 Lifecycle Management: Automatically transitions logs to cheaper storage classes
Compressed Storage: Gzip compression reduces storage costs by ~70%
CloudWatch Log Offloading: Reduces expensive CloudWatch Logs storage

Scalability

Lambda Auto-scaling: Handles varying log volumes automatically
SQS Buffering: Manages traffic spikes and processing delays
S3 Infinite Scale: No storage capacity concerns

Reliability

Dead Letter Queue: Captures failed message processing
Error Handling: Robust Lambda error handling and retries
Infrastructure as Code: Consistent, repeatable deployments

Security

IAM Least Privilege: Minimal required permissions for each component
Encryption: S3 server-side encryption enabled
VPC Integration: Can be deployed in VPC for additional isolation

Monitoring and Troubleshooting

CloudWatch Metrics to Monitor

Lambda function errors and duration
SQS queue depth and message age
S3 PUT/GET request metrics

Common Issues and Solutions

Lambda timeouts: Increase memory allocation or timeout duration
SQS message accumulation: Check Lambda error logs and DLQ
Missing logs in Cortex: Verify IAM permissions and SQS configuration

Cost Analysis

For a typical deployment processing 1GB of WAF logs daily:

S3 Storage: ~$0.50/month (with lifecycle transitions)
Lambda Execution: ~$2.00/month
SQS Messages: ~$0.10/month
Data Transfer: Minimal within same region

Total: ~$2.60/month vs ~$15/month keeping logs in CloudWatch

Conclusion

This pipeline provides a robust, cost-effective solution for WAF log management and security analytics. By leveraging Infrastructure as Code with AWS CDK, we've created a maintainable, scalable system that integrates seamlessly with modern security platforms like Cortex XDR.

The combination of automated processing, cost optimization, and real-time security analytics makes this architecture ideal for organizations serious about application security monitoring.

References:

Cortext XDR
Cloudwatch Pricing
Ingest generic logs from Amazon S3 Ready to implement this in your environment?

Troubleshooting SSL Certificate Chain Issues in Enterprise Integration Platforms: A Real-World Case Study

Clariza Look — Tue, 17 Jun 2025 06:39:16 +0000

How to diagnose and fix PKIX certificate path building errors when integrating with external APIs.

The Problem: A Mysterious SSL Error

Picture this: You're working on an integration between your enterprise middleware platform and a third-party API endpoint (api.partner-services.com), and suddenly you're hit with this cryptic error:

The request operation failed. Exception: com.ibm.jsse2.util.h: PKIX path building failed: 

java.security.cert.CertPathBuilderException: 

PKIXCertPathBuilderImpl could not build a valid CertPath.; 
internal cause is: java.security.cert.CertPathValidatorException: 

The certificate issued by 
CN=SSL.com EV Root Certification Authority RSA R2, O=SSL Corporation, L=Houston, ST=Texas, C=US is not trusted

If you've ever encountered this error, you know how frustrating it can be. The integration was working fine, but now it's failing with SSL certificate validation issues. Let's walk through how to diagnose and fix this step-by-step.

Understanding the Error

The error message tells us several key things:

PKIX path building failed: The system can't build a complete certificate chain
Certificate not trusted: A specific SSL.com certificate isn't being trusted
Root cause: Missing or incomplete certificate chain in the trust store

Step 1: Identify the Failing Endpoint

First, we need to identify exactly which endpoint is causing the SSL error. This information can usually be found in:

Platform logs or activity monitor
Integration flow configuration
HTTP connector settings
Error details in the runtime logs

In our case, the failing endpoint was: api.partner-services.com:443/oauth/token

Step 2: Analyze the Certificate Chain

This is the crucial step that most troubleshooting guides skip. We need to see the actual certificate chain that the server is presenting. Use OpenSSL to inspect the complete chain:

openssl s_client -showcerts -connect api.partner-services.com:443

What the Output Reveals

The OpenSSL output showed us a 4-level certificate chain:

Certificate chain
 0 s:C=US, ST=California, O=Partner Services Inc, CN=api.partner-services.com
   i:C=US, O=SSL Corporation, CN=Entrust OV TLS Issuing RSA CA 1

 1 s:C=US, O=SSL Corporation, CN=Entrust OV TLS Issuing RSA CA 1
   i:C=US, O=SSL Corporation, CN=SSL.com TLS RSA Root CA 2022

 2 s:C=US, O=SSL Corporation, CN=SSL.com TLS RSA Root CA 2022
   i:C=US, ST=Texas, L=Houston, O=SSL Corporation, CN=SSL.com EV Root Certification Authority RSA R2

 3 s:C=US, ST=Texas, L=Houston, O=SSL Corporation, CN=SSL.com EV Root Certification Authority RSA R2
   i:C=US, ST=Texas, L=Houston, O=SSL Corporation, CN=SSL.com EV Root Certification Authority RSA R2

This revealed the complete trust chain:

End Certificate: api.partner-services.com
Intermediate CA: Entrust OV TLS Issuing RSA CA 1
Intermediate CA: SSL.com TLS RSA Root CA 2022
Root CA: SSL.com EV Root Certification Authority RSA R2

Step 3: Identify Missing Certificates

Comparing what our integration platform had versus what was needed:

✅ SSL.com EV Root Certification Authority RSA R2 (already imported)
❌ SSL.com TLS RSA Root CA 2022 (missing)
❌ Entrust OV TLS Issuing RSA CA 1 (missing)

The issue was clear - we had the root certificate but were missing the two intermediate certificates needed to complete the chain.

Step 4: Locate and Download Missing Certificates

Finding SSL.com Certificates
Navigate to SSL.com's Certificate Repository and locate:

1. SSL.com TLS RSA Root CA 2022

File: SSLcom-TLS-Root-2022-RSA.pem
SHA1: 72:B2:76:44:BD:09:46:BF:3C:51:0C:00:0F:44:30:16:8F:E2:89:8A

2. Entrust OV TLS Issuing RSA CA 1

Serial: 6FE63EDE5FC1C03AFB6D7A85BD3A156D
SHA1: CA:AA:C3:E1:6B:D2:42:66:88:7C:C8:56:73:C0:18:F4:6A:13:9B:0F

Download both certificates in PEM format.

Step 5: Import Certificates in Your Integration Platform

Import Process

Navigate to your platform's SSL certificate management section
Select "Import from Clipboard" or similar option
Paste the complete PEM certificate (including BEGIN/END lines)
Set certificate type as "CA Certificate" or "Intermediate Certificate"
Provide descriptive aliases:
- SSL.com-TLS-Root-2022-RSA
- Entrust-OV-TLS-Issuing-RSA-CA-1

Troubleshooting Import Issues

If you encounter "Couldn't establish a chain of trust" during import:

✅ Check "Trusted" or "Mark as trusted" during import
✅ Import certificates in order (root to intermediate)
✅ Use "CA Certificate" import option instead of "TrustStore Entry"
✅ Ensure complete PEM format with proper BEGIN/END lines

Step 6: Deploy and Test

After importing both certificates:

Deploy the configuration in your integration platform
Wait 1-5 minutes for changes to take effect
Test the integration that was previously failing
Monitor logs for any remaining certificate errors

Understanding Certificate Chains: Why Not Just One Certificate?

You might wonder why we need multiple certificates instead of just one. Here's why certificate chains exist:

Security Benefits

Root CA Protection: Root certificates are kept offline and ultra-secure
Risk Limitation: If an intermediate is compromised, only that level needs replacement
Compartmentalization: Different intermediates serve different purposes

Operational Benefits

Scalability: One root can't physically sign millions of certificates
Flexibility: Easier to revoke or replace intermediate certificates
Specialization: Different validation levels and certificate types

Trust Model

Browser Trust: Browsers only trust ~150 pre-installed root CAs
Delegation: Root CAs delegate trust to intermediate CAs
Chain Validation: Each level vouches for the level below it

Who Is Responsible for Implementing Certificate Chains?

Understanding certificate chain responsibilities helps prevent issues and clarifies troubleshooting ownership:

Certificate Authorities (CAs) Design the Chain

SSL.com, DigiCert, Let's Encrypt, etc. design and operate the certificate hierarchy
They decide how many intermediate levels to use for security and scalability
They create and manage the intermediate CAs
They determine the chain structure for operational efficiency

Service Providers Configure the Chain

API providers (like api.partner-services.com) configure their web servers to present the complete chain
They install the end certificate + all intermediate certificates on their servers
They're responsible for ensuring clients can validate the full chain
They handle certificate renewals and chain updates

Integration Teams Handle Missing Components

Integration developers import any missing intermediate/root certificates into client systems
Your middleware platform needs the complete chain to validate connections
You troubleshoot and fix incomplete chains in your trust store
You monitor for certificate-related integration failures

Infrastructure Teams Maintain Trust Stores

IT/Security teams maintain the root certificate trust stores across the organization
They ensure enterprise systems have necessary CA certificates pre-installed
They handle certificate lifecycle management, renewals, and security policies
They establish standards for certificate validation and trust

Shared Responsibility Model
The ideal approach follows a shared responsibility model:

Service providers should present complete certificate chains
Clients should be prepared to import missing intermediates when needed
Both parties should monitor certificate health and expiration dates
Organizations should have processes for handling certificate issues

In this Case Study:

SSL.com: Designed the 4-level certificate chain architecture
Partner Services: Configured their server to present the complete chain
Integration Team: Imported the missing intermediate certificates to fix the validation
IT Infrastructure: Maintains the overall certificate trust infrastructure

Remember: While service providers should ideally present complete chains, integration teams must be prepared to handle missing intermediate certificates as part of robust API integration practices.

Key Takeaways

Certificate chains require ALL certificates from root to intermediate
OpenSSL is your best friend for diagnosing certificate chain issues
Import intermediate certificates even if you have the root
Check certificate repositories of the issuing CA for missing certificates
Mark certificates as trusted during import to avoid validation errors

Prevention Tips

Regularly audit your certificate trust store
Monitor certificate expiration dates and renewal schedules
Test integrations in staging environments before production
Document certificate dependencies for your integrations
Set up monitoring for SSL certificate validation errors

Conclusion

SSL certificate chain issues can be frustrating, but with the right diagnostic approach, they're entirely solvable. The key is understanding that certificate validation requires the complete chain of trust, not just the root certificate.

By using tools like OpenSSL to inspect certificate chains and systematically importing missing intermediate certificates, you can resolve these issues quickly and prevent them from recurring.

Remember: when in doubt, check the complete certificate chain first. It's often the missing piece that solves the puzzle!

Have you encountered similar SSL certificate issues in your integrations? Share your experiences and solutions in the comments below.

Additional Resources

Building Serverless Application with AWS CDK: ECS Fargate & DynamoDB

Clariza Look — Tue, 10 Jun 2025 07:18:54 +0000

AWS offers a powerful combination of services that enable developers to build serverless architectures without managing underlying infrastructure. In this guide, we'll explore how to build a robust serverless application using AWS CDK with ECS Fargate and DynamoDB.

Why This Architecture Matters

The serverless paradigm has transformed how we think about application deployment. By combining ECS Fargate for containerized compute with DynamoDB for managed NoSQL storage, we create an architecture that scales automatically, reduces operational overhead, and optimizes costs.

The AWS CDK (Cloud Development Kit) makes this entire setup reproducible and maintainable through code.

Github Repository Example to Build Serverless Application with AWS CDK: ECS Fargate & DynamoDB

Architecture Deep Dive

Our serverless architecture follows a clean, layered approach that separates concerns while maintaining high availability:

             ┌───────────────────────────────────┐
             │      Internet Users               │
             └───────────────────────────────────┘
                             │
                             ▼
             ┌───────────────────────────────────┐
             │  Application Load Balancer (ALB)  │
             │       - Listens on Port 80        │
             │       - Routes to ECS Service     │
             └───────────────────────────────────┘
                             │
                             ▼
     ┌─────────────────────────────────────────────┐
     │          ECS Fargate Service (2 Tasks)      │
     │ ┌─────────────────────────────────────────┐ │
     │ │  Task Definition                        │ │
     │ │  - Nginx Container (Port 3000)          │ │
     │ │  - Logs to AWS CloudWatch               │ │
     │ │  - Reads/Writes to DynamoDB             │ │
     │ └─────────────────────────────────────────┘ │
     └─────────────────────────────────────────────┘
                             │
                             ▼
           ┌───────────────────────────────────┐
           │        DynamoDB Table             │
           │  - Stores messages data           │
           │  - PAY_PER_REQUEST mode           │
           └───────────────────────────────────┘

Traffic Flow Breakdown

Request Journey: Internet users send requests to the Application Load Balancer, which intelligently distributes traffic across multiple ECS Fargate tasks running in different availability zones. Each task runs an Nginx container that processes requests and interacts with DynamoDB for data persistence.

Scaling Mechanism: The ECS service automatically adjusts the number of running tasks based on CPU and memory utilization, while DynamoDB scales read/write capacity on-demand, ensuring consistent performance regardless of load.

Technology Stack Explained

AWS CDK (TypeScript)

The AWS CDK serves as our infrastructure-as-code foundation. Unlike traditional CloudFormation templates, CDK allows us to define cloud resources using familiar programming languages, making infrastructure more maintainable and testable.

Key Benefits:

Type safety and IDE support
Reusable constructs and patterns
Automatic CloudFormation generation
Built-in best practices and security

Amazon ECS Fargate

Fargate eliminates the need to provision and manage EC2 instances for containerized applications. It provides a serverless compute engine that automatically handles infrastructure scaling, patching, and security.

Why Fargate Over Lambda?

Longer execution times (no 15-minute limit)
Full control over runtime environment
Better suited for HTTP services and APIs
Container-based deployment flexibility

Amazon DynamoDB

As a fully managed NoSQL database, DynamoDB provides single-digit millisecond latency at any scale. The pay-per-request billing model aligns perfectly with serverless principles.

DynamoDB Advantages:

Automatic scaling without downtime
Built-in security with encryption at rest
Global secondary indexes for flexible querying
Point-in-time recovery and backups

Project Structure and Organization

We are using AWS Cloud Development Kit (CDK) to deploy our infrastructure to AWS. The structure looks like below:

├── bin/                    # CDK entry point
├── lib/                    # CDK Stack Definition
│   ├── ecs-construct-example-stack.ts  # Main CDK Stack
├── cdk.json               # CDK Configuration
├── package.json           # Dependencies
├── tsconfig.json          # TypeScript Configuration
└── README.md             # Project Documentation

The structure follows CDK best practices, separating the application entry point (bin/) from the actual infrastructure definitions (lib/). This organization makes the codebase maintainable and allows for easy extension with additional stacks or constructs.

Deployment Process

Prerequisites Setup

Before deploying, ensure the development environment is properly configured (in the terminal):

# Install Node.js 18.x or higher
node --version

# Configure AWS CLI with appropriate permissions
aws configure

# Install AWS CDK globally
npm install -g aws-cdk

# Bootstrap CDK (first-time setup)
cdk bootstrap

# Verify CDK installation
cdk --version

To setup the first CDK template, follow the steps in CDK Setup template to get the CDK file structure format similar to the above.

Deployment Steps

1. Updating the `bin` folder

Update the file from the bin folder with ecs-construct-example.ts. It contains the entry point files for your CDK application. These are the files that get run when you execute CDK commands like cdk deploy or cdk synth.

As you can see from the script below, it is importing the files from the lib folder called EcsFargateStack.

// bin/ecs-construct-example.ts

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { EcsFargateStack } from '../lib/ecs-construct-example-stack';

const app = new cdk.App();

new EcsFargateStack(app, 'EcsFargateStack', {
  projectName: 'SimpleEcs',
  environment: 'Dev',
  env: {
    account: process.env.CDK_DEFAULT_ACCOUNT,
    region: process.env.CDK_DEFAULT_REGION
    }
});

2. Creating the stack definition in the `lib` folder

Let's go to the lib folder and update one of the files with lib/ecs-construct-example-stack.ts.

Github Reference for the lib folder to setup the ecs fargate stack

// in lib/ecs-construct-example-stack.ts

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as dynamodb from 'aws-cdk-lib/aws-dynamodb';
import { RemovalPolicy, CfnOutput } from 'aws-cdk-lib';
import * as elasticloadbalancing from 'aws-cdk-lib/aws-elasticloadbalancingv2';

interface ECSStackProps extends cdk.StackProps {
  projectName: string;
  environment: string;
}

export class EcsFargateStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: ECSStackProps) {
    super(scope, id, props);

    const projectName = props.projectName;
    const projectPrefix = `${projectName}-${props.environment}-server`;

    // Create a Dynamo db table
    const dynamoTable = new dynamodb.Table(this, 'DynamoDbMessages', {
      partitionKey: {
        name: 'id',
        type: dynamodb.AttributeType.STRING
      }, 
      sortKey: {
        name: 'created_at',
        type: dynamodb.AttributeType.NUMBER
      },
      billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
      removalPolicy: RemovalPolicy.DESTROY, 
    });

    new CfnOutput(this, 'TableName', { value: dynamoTable.tableName });

    /******** 
     * 1. Create a vpc for the ecs or Can also reference a vpc if 
     * it has been created prior this deployment
    *********/
    const vpc = new ec2.Vpc(this, "Vpc", {
      vpcName:'primary-vpc',
      natGateways: 1,
      subnetConfiguration: [{  
          cidrMask: 24, subnetType: ec2.SubnetType.PUBLIC, name: "Public" },
        {  
          cidrMask: 24, subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS, name: "Private"}],
      maxAzs: 2 
    });

    // Create a security group that allows HTTP traffic on port 80 for the ECS container
    const ecsSecGroup = new ec2.SecurityGroup(this, 'EcsSecurityGroup', {
      securityGroupName: `${projectName}-${props.environment}-SecGroup`,
      vpc,
      allowAllOutbound: false,
    });

    ecsSecGroup.addIngressRule(ec2.Peer.ipv4('0.0.0.0/0'), ec2.Port.tcp(3000));

    // Create cluster
    const cluster = new ecs.Cluster(this, `ECS-Cluster`, {
      clusterName: `${projectPrefix}-cluster`,
      vpc,
    });

    const executionRolePolicy =  new iam.PolicyStatement({
      effect: iam.Effect.ALLOW,
      resources: ['*'],
      actions: [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ]
    });

    // Create a fargate task definition 
    const fargateTaskDefinition = new ecs.FargateTaskDefinition(
      this, 'FargateTaskDefinition', {
      memoryLimitMiB: 512,
      cpu: 256,
    });

fargateTaskDefinition.addToExecutionRolePolicy(executionRolePolicy);

   fargateTaskDefinition.addToTaskRolePolicy(new iam.PolicyStatement({
      effect: iam.Effect.ALLOW,
      resources: [dynamoTable.tableArn],
      actions: ['dynamodb:*']
    }));

    const ecsContainer = fargateTaskDefinition.addContainer('NginxServerContainer', {
      image: ecs.ContainerImage.fromRegistry('nginx:latest'),
      cpu: 100,
      memoryLimitMiB: 256,
      essential: true,
      logging: ecs.LogDrivers.awsLogs({streamPrefix: 'nginx-ecs-server'}),
      environment: { 
        'DYNAMODB_MESSAGES_TABLE': dynamoTable.tableName,
        'APP_ID' : 'nginx-ecs-server'
      }
    });

    ecsContainer.addPortMappings({
      containerPort: 3000,
      protocol: ecs.Protocol.TCP,
    });

    // Create the service
    const ecsFargateService = new ecs.FargateService(this, 'Ecs-Service', {
      cluster,
      taskDefinition: fargateTaskDefinition,
      desiredCount: 2,
      assignPublicIp: false,
      securityGroups: [ecsSecGroup],
    });

    const lb = new elasticloadbalancing.ApplicationLoadBalancer(this, 'ALB', {
      vpc,
      internetFacing: true
    });

    const albListener = lb.addListener('AlbListener', {
      port: 80,
    });

    albListener.addTargets('Target', {
      port: 80,
      targets: [ecsFargateService],
      healthCheck: { path: '/api/' }
    });

    albListener.connections.allowDefaultPortFromAnyIpv4('Open to all');
  }
}

This ECS Fargate stack sets up a load-balanced, containerized web application running on ECS Fargate with DynamoDB backend storage.

DynamoDB Table - Creates a table for messages with partition key 'id' and sort key 'created_at'
VPC Setup - Creates a new VPC with public and private subnets across 2 availability zones
Security Group - Allows inbound traffic on port 3000 for the ECS containers
ECS Cluster - Sets up a Fargate cluster to run containerized applications
Task Definition - Configures a Fargate task with 512 MiB memory and 256 CPU units
Container - Runs an nginx:latest container with DynamoDB table name as environment variable
ECS Service - Deploys 2 instances of the task in private subnets without public IPs
Load Balancer - Creates an internet-facing ALB that routes traffic to the ECS service on port 80
IAM Permissions - Grants the task execution role ECR/CloudWatch permissions and task role DynamoDB access

3. Deploy

We first install the node dependencies and then test the cdk

# Install project dependencies
npm install

# Preview changes before deployment
cdk diff

# Deploy the complete infrastructure
cdk deploy

The deployment process creates a comprehensive serverless infrastructure including VPC networking, security groups, IAM roles, and all necessary AWS resources. The entire process typically completes within 10-15 minutes.

Real-World Use Cases

1. Chat API Backend

This architecture excels as a microservices backend for real-time messaging applications. The ALB efficiently routes WebSocket connections and HTTP requests to Fargate tasks, while DynamoDB stores messages, user sessions, and conversation metadata.

Implementation Flow:

Users send messages through the frontend application
ALB routes requests to available ECS tasks
Nginx processes the requests and validates user authentication
Message data gets stored in DynamoDB with timestamps and metadata
Real-time updates are pushed back to connected clients

2. Serverless Web Application Backend

Perfect for supporting modern frontend frameworks like React, Angular, or Vue.js, this architecture provides a scalable API layer that handles user authentication, data processing, and business logic.

Typical API Endpoints:

/api/users - User management and profiles
/api/messages - Message CRUD operations
/api/analytics - Dashboard metrics and reporting
/api/health - System health monitoring

3. Analytics Dashboard System

The architecture supports complex analytical workloads by leveraging DynamoDB's querying capabilities and ECS Fargate's processing power for data aggregation and report generation.

Security and Best Practices

This structure follows best practices for:

Network Security

VPC isolation with public and private subnets
Security groups restricting access to necessary ports only
ALB SSL termination for encrypted communication

IAM and Access Control

Least privilege access principles
Task-specific IAM roles for ECS services
DynamoDB access limited to required operations only

Monitoring and Observability

CloudWatch integration for comprehensive logging
Application-level metrics and health checks
Automated alerting for performance anomalies

Cost Optimization

This serverless architecture optimizes costs through several mechanisms:

Pay-per-use pricing: DynamoDB's on-demand billing and Fargate's per-second billing ensure you only pay for actual resource consumption.

Automatic scaling: Resources scale down during low-traffic periods, significantly reducing costs compared to always-on infrastructure.

No idle capacity: Unlike traditional server-based approaches, there's no wasted capacity during off-peak hours.

Conclusion

The beauty of this architecture lies in its simplicity to deploy yet powerful enough to handle production workloads. Whether you're building a startup MVP or scaling an enterprise application, this serverless foundation provides the flexibility and reliability needed for modern cloud applications.

Start by cloning the repository, following the deployment steps, and customizing the Nginx container to serve your specific application logic. The modular CDK structure makes it easy to extend with additional AWS services like ElastiCache for caching, RDS for relational data, or SNS for notifications.

The future of cloud applications is serverless, and this architecture provides a proven foundation for building scalable, cost-effective solutions that grow with your business needs.

Building and Deploying a Nginx Web Application with Docker and AWS EC2: A Step-by-Step Guide

Clariza Look — Thu, 05 Jun 2025 09:34:41 +0000

Introduction

In this blog, I am gonna walk you through creating a simple yet robust web application deployment pipeline using Nginx, Docker, Amazon ECR, and AWS EC2. By the end of this tutorial, we'll have a fully automated deployment system that can serve as a foundation for more complex applications.

Github repository for Building and Deploying a Nginx Web Application with Docker and AWS EC2

What We're Building

This project demonstrates a complete deployment workflow featuring:

A lightweight Nginx web server serving static content
Containerized application using Docker
Cloud-based image storage with Amazon ECR
Automated deployment on AWS EC2
CI/CD pipeline for seamless updates

Prerequisites
Before diving in, ensure we have:

AWS CLI configured with appropriate permissions
Docker installed on the local machine
Basic understanding of Docker, AWS services, and command line operations
An AWS account with ECR and EC2 access

Project Architecture Overview

The application follows a modern containerized deployment pattern:

This project structure is organized for clarity and maintainability:

nginx-app-project/
├── .github/workflows/
│   └── deploy.yml                   # CI/CD Pipeline
├── nginx/
│   └── nginx.conf                   # Web server configuration
├── infra/
│   └── infrastructure_stack.py      # AWS infrastructure setup
├── test/unit/
│   └── test_infra_stack.py         # Infrastructure tests
├── src/                            # Static web content
├── Dockerfile                      # Container definition
└── README.md

Step 1: Setting Up The Local Development Environment

Start by creating the project directory and basic file structure:

> mkdir nginx-app-deployment
> cd nginx-app-deployment
> mkdir -p nginx src infra test/unit .github/workflows

Create the main HTML content in the src/ directory. This is where the static website files will live. For a simple example:

<!-- src/index.html -->
<!DOCTYPE html>
<html>
<head>
    <title>My Nginx App</title>
</head>
<body>
    <h1>Welcome to My Containerized Nginx Application</h1>
    <p>Successfully deployed with Docker and AWS!</p>
</body>
</html>

Step 2: Configuring Nginx

Create a custom Nginx configuration file to optimize the web server:

events {
    worker_connections 1024;
}

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    server {
        listen 80;
        server_name localhost;

        location / {
            root /usr/share/nginx/html;
            index index.html;
            try_files $uri $uri/ /index.html;
        }

        # Health check endpoint
        location /health {
            access_log off;
            return 200 "healthy\n";
            add_header Content-Type text/plain;
        }
    }
}

This configuration provides a basic web server setup with a health check endpoint that's useful for monitoring.

Step 3: Creating the Dockerfile

The Dockerfile defines how the application will be containerized:

FROM nginx:alpine

# Copy custom nginx configuration
COPY nginx/nginx.conf /etc/nginx/nginx.conf

# Copy static content
COPY src/ /usr/share/nginx/html/

# Expose port 80
EXPOSE 80

# Nginx runs in foreground by default in the base image
CMD ["nginx", "-g", "daemon off;"]

This lightweight Alpine-based image keeps the container size minimal while providing all necessary functionality.

Step 4: Building and Testing Locally

Before deploying to the cloud, test the application locally. From the laptop's terminal:

# Build the Docker image
> docker build -t nginx-app-local .

# Run the container
> docker run -d -p 8080:80 --name nginx-test nginx-app-local

# Test the application
> curl http://localhost:8080
> curl http://localhost:8080/health

Visit http://localhost:8080 in the browser to verify everything works correctly.

Step 5: Setting Up Amazon ECR

Amazon Elastic Container Registry will store the Docker images:

# Create ECR repository
> aws ecr create-repository --repository-name nginx-app --region the-region

# Get login token and authenticate Docker
> aws ecr get-login-password --region the-region | docker login --username AWS --password-stdin the-account-id.dkr.ecr.the-region.amazonaws.com

Step 6: Building and Pushing to ECR

Tag and push the image to ECR. In the laptop's terminal:

# Tag
> docker tag nginx-app-local:latest the-account-id.dkr.ecr.the-region.amazonaws.com/nginx-app:latest

# Push to ECR
> docker push the-account-id.dkr.ecr.the-region.amazonaws.com/nginx-app:latest

Step 7: Infrastructure Setup with Python CDK

Create the infrastructure setup script. It will deploy a few AWS resources including ec2, security group, iam roles, etc. It will also setup the user data that will be the command to be executed when the ec2 instance is deployed.

# infra/infrastructure_stack.py
from aws_cdk import (
    Stack,
    aws_ec2 as ec2,
    aws_iam as iam,
    Tags,
    CfnOutput
)
from constructs import Construct

class NginxCicdStack(Stack):

    def __init__(self, scope: Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        # Use the default VPC
        vpc = ec2.Vpc.from_lookup(self, "DefaultVpc", is_default=True)

        # Define the security group
        security_group = ec2.SecurityGroup(self, "SecurityGroup",
            vpc=vpc,
            description="Allow SSH and HTTP access",
            allow_all_outbound=True
        )
        security_group.add_ingress_rule(ec2.Peer.ipv4("0.0.0.0/0"), ec2.Port.tcp(22), "Allow SSH access")
        security_group.add_ingress_rule(ec2.Peer.ipv4("0.0.0.0/0"), ec2.Port.tcp(80), "Allow HTTP access")

        # Create a role for the EC2 instance with ECR access
        instance_role = iam.Role(self, "InstanceRole",
            assumed_by=iam.ServicePrincipal("ec2.amazonaws.com")
        )

        # Add ECR policy to the role
        instance_role.add_managed_policy(
            iam.ManagedPolicy.from_aws_managed_policy_name("AmazonEC2ContainerRegistryFullAccess")
        )

        # Define the EC2 instance with the custom role
        ec2_instance = ec2.Instance(self, "Instance",
            instance_type=ec2.InstanceType("t2.micro"),
            machine_image=ec2.MachineImage.latest_amazon_linux2(),
            vpc=vpc,
            security_group=security_group,
            key_name="rsakey",
            role=instance_role,  # Assign the role with ECR permissions
            user_data=ec2.UserData.custom(self._get_user_data())
        )

        # Add a Name Tag to EC2 instance
        Tags.of(ec2_instance).add("Name", "NginxInstance")

        # Output the instance ID and public IP
        CfnOutput(self, "InstanceId", value=ec2_instance.instance_id)
        CfnOutput(self, "InstancePublicIp", value=ec2_instance.instance_public_ip)

    def _get_user_data(self):
        return """#!/bin/bash
# Update system packages
yum update -y

# Install Docker
amazon-linux-extras install docker -y
systemctl start docker
systemctl enable docker
usermod -a -G docker ec2-user

# Install AWS CLI v2 if needed
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip -q awscliv2.zip
./aws/install --update
rm -rf aws awscliv2.zip

# Create a status file to signal instance is ready
touch /tmp/instance_ready

# Log completion
echo "Instance setup complete"
"""

Step 8: Deploying Infrastructure

Deploy the infrastructure using CDK:

# Install CDK if not already installed
npm install -g aws-cdk

# Bootstrap CDK (first time only)
cdk bootstrap

# Deploy the stack
cdk deploy

Step 9: Automated Deployment with GitHub Actions

Why Create a CI/CD Pipeline?

Before diving into the implementation, it's important to understand why we're creating a Continuous Integration/Continuous Deployment (CI/CD) pipeline and how it transforms our development workflow.

The Problem Without CI/CD:

Without automation, deploying updates to the application involves numerous manual steps below (that are prone to human error):

Manually building Docker images on your local machine
Remembering to tag images with the correct version
Manually pushing images to ECR
SSH-ing into EC2 instances to pull new images
Stopping and starting containers manually
Risk of deploying different versions across environments
No rollback strategy if something goes wrong
Time-consuming process that discourages frequent deployments

What CI/CD Solves:

A well-designed CI/CD pipeline addresses these challenges by automating the entire deployment process:

Consistency: Every deployment follows the exact same process, eliminating "it works on my machine" problems
Speed: Automated deployments happen in minutes rather than hours
Reliability: Reduces human error through automation
Traceability: Every deployment is tied to a specific code commit
Rollback capability: Easy to revert to previous versions if issues arise
Testing integration: Automatically runs tests before deployment
Multi-environment support: Can deploy to development, staging, and production with the same process

Our Pipeline Strategy:

The GitHub Actions pipeline implements a deployment workflow that triggers automatically when code changes are pushed to the main branch. Here's what happens behind the scenes:

Trigger: Developer pushes code to the main branch
Build: Pipeline automatically builds a new Docker image
Test: (Can be extended to run automated tests)
Tag: Image is tagged with the Git commit SHA for traceability
Push: Image is pushed to Amazon ECR
Deploy: EC2 instance is updated with the new image
Verify: Health checks ensure the deployment was successful

This approach ensures that every code change goes through the same rigorous, automated process, making deployments predictable and reliable.

Create a CI/CD pipeline:

First iteration

In this first iteration of the deploy.yml file, we can see the steps it does to Build, tag, and push image to Amazon ECR and deploy the ec2 instance.

# .github/workflows/deploy.yml
name: Deploy Nginx App

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

env:
  AWS_REGION: us-east-1
  ECR_REPOSITORY: nginx-app

jobs:
  deploy:
    name: Deploy
    runs-on: ubuntu-latest

    steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v2
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: ${{ env.AWS_REGION }}

    - name: Login to Amazon ECR
      id: login-ecr
      uses: aws-actions/amazon-ecr-login@v1

    - name: Build, tag, and push image to Amazon ECR
      id: build-image
      env:
        ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
        IMAGE_TAG: ${{ github.sha }}
      run: |
        docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
        docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
        echo "image=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" >> $GITHUB_OUTPUT

    - name: Deploy to EC2
      env:
        ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
        IMAGE_TAG: ${{ github.sha }}
      run: |
        # Add deployment script here
        echo "Deployment completed with image: $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG"

Final Iteration:

Meanwhile in the final iteration of the workflow, this is how the entire process will look like for the cicd pipeline.

name: Deploy Nginx App

on:
  push:
    branches:
      - main

jobs:
  build-docker-setup-infra:
    runs-on: ubuntu-latest
    outputs:
      instance_ip: ${{ steps.wait-for-instance.outputs.instance_ip }}

    env:
      REPOSITORY_URI: ${{ secrets.ECR_REPOSITORY_URI }}
      CDK_DEFAULT_ACCOUNT: ${{ secrets.AWS_ACCOUNT_ID }}
      CDK_DEFAULT_REGION: ap-southeast-2

    steps:
      - name: Checkout Code
        uses: actions/checkout@v3

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ap-southeast-2

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v1

      - name: Build, Tag, and Push Docker Image
        run: |
          set -x
          IMAGE_TAG=latest
          docker build -t $REPOSITORY_URI:$IMAGE_TAG .
          docker push $REPOSITORY_URI:$IMAGE_TAG

      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: '3.8'

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r infra/requirements.txt

      - name: Install AWS CDK
        run: npm install -g aws-cdk

      - name: Deploy CDK Stack
        working-directory: ./infra
        run: cdk deploy --require-approval never

      - name: Wait for EC2 Instance to be Ready
        id: wait-for-instance
        run: |
          # Get the instance ID from CDK outputs if possible
          STACK_OUTPUTS=$(aws cloudformation describe-stacks --stack-name nginx-cicd-stack --query "Stacks[0].Outputs" --output json)
          INSTANCE_ID=$(echo $STACK_OUTPUTS | jq -r '.[] | select(.OutputKey=="InstanceId") | .OutputValue')
          INSTANCE_IP=$(echo $STACK_OUTPUTS | jq -r '.[] | select(.OutputKey=="InstancePublicIp") | .OutputValue')

          # Fallback to searching by tag if outputs aren't available
          if [ -z "$INSTANCE_ID" ] || [ "$INSTANCE_ID" == "null" ]; then
            echo "Looking up instance by tag..."
            INSTANCE_ID=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=NginxInstance" "Name=instance-state-name,Values=pending,running" --query "Reservations[0].Instances[0].InstanceId" --output text)

            if [ -z "$INSTANCE_ID" ] || [ "$INSTANCE_ID" == "None" ]; then
              echo "Error: No running EC2 instance with tag 'NginxInstance' found."
              exit 1
            fi

            INSTANCE_IP=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID --query "Reservations[0].Instances[0].PublicIpAddress" --output text)
          fi

          echo "Found instance $INSTANCE_ID with IP $INSTANCE_IP"

          # Wait for the instance to be running
          echo "Waiting for instance to be in running state..."
          aws ec2 wait instance-running --instance-ids $INSTANCE_ID

          # Wait for status checks to pass
          echo "Waiting for instance status checks to pass..."
          aws ec2 wait instance-status-ok --instance-ids $INSTANCE_ID

          # Extra verification of SSH availability
          echo "Verifying SSH connectivity..."
          counter=0
          max_attempts=10
          while [ $counter -lt $max_attempts ]; do
            if nc -z -w5 $INSTANCE_IP 22; then
              echo "SSH port is open!"
              break
            fi
            echo "Waiting for SSH port to open... (attempt $((counter+1))/$max_attempts)"
            sleep 10
            counter=$((counter+1))
          done

          if [ $counter -eq $max_attempts ]; then
            echo "Warning: Could not verify SSH connectivity after $max_attempts attempts"
          fi

          echo "Instance ID: $INSTANCE_ID"
          echo "Instance IP: $INSTANCE_IP"
          echo "instance_ip=$INSTANCE_IP" >> $GITHUB_OUTPUT

  deploy:
    needs: build-docker-setup-infra
    runs-on: ubuntu-latest

    env:
      REPOSITORY_URI: ${{ secrets.ECR_REPOSITORY_URI }}

    steps:
      - name: Debug Output IP
        run: 'echo "Using IP address: ${{ needs.build-docker-setup-infra.outputs.instance_ip }}"'

      - name: Install SSH Client
        run: sudo apt-get install -y openssh-client

      - name: Create .ssh Directory
        run: mkdir -p ~/.ssh

      - name: Add SSH Key
        run: |
          echo "${{ secrets.SSH_PRIVATE_KEY }}" > ~/.ssh/id_rsa
          chmod 600 ~/.ssh/id_rsa
          ssh-keygen -lf ~/.ssh/id_rsa || echo "Key verification failed"

      - name: SSH into EC2 and Deploy Docker Image
        env:
          INSTANCE_IP: ${{ needs.build-docker-setup-infra.outputs.instance_ip }}
          REPO_URI: ${{ secrets.ECR_REPOSITORY_URI }}
        run: |
          ssh -i ~/.ssh/id_rsa -o StrictHostKeyChecking=no -o ConnectTimeout=30 ec2-user@${INSTANCE_IP} "
            # Check what's using port 80
            echo 'Checking what process is using port 80...'
            sudo lsof -i :80 || echo 'No process found by lsof'

            # Stop Nginx if it's running
            sudo systemctl stop nginx || echo 'Nginx not running or not installed'

            # Kill any process using port 80
            sudo fuser -k 80/tcp || echo 'No process killed'

            # Stop any running Docker containers using port 80
            sudo docker ps -q --filter publish=80 | xargs -r sudo docker stop
            sudo docker ps -q --filter publish=80 | xargs -r sudo docker rm

            # Start Docker and proceed with deployment
            sudo systemctl start docker &&
            sudo systemctl enable docker &&
            aws ecr get-login-password --region ap-southeast-2 | sudo docker login --username AWS --password-stdin ${REPO_URI} &&
            sudo docker pull ${REPO_URI}:latest &&
            sudo docker run -d -p 80:80 ${REPO_URI}:latest"

Step 10: SSH Deployment and Container Management

Once the EC2 instance is running, connect and deploy the container:

# SSH into the EC2 instance
ssh -i the-key.pem ec2-user@the-ec2-public-ip

# Login to ECR
aws ecr get-login-password --region the-region | docker login --username AWS --password-stdin the-account-id.dkr.ecr.t-heregion.amazonaws.com

# Pull and run the image
docker pull the-account-id.dkr.ecr.the-region.amazonaws.com/nginx-app:latest
docker run -d -p 80:80 --name nginx-app the-account-id.dkr.ecr.the-region.amazonaws.com/nginx-app:latest

# Verify deployment
curl http://localhost/health

Step 11: Testing and Validation

Create unit tests for the infrastructure:

# test/unit/test_infra_stack.py
import aws_cdk as core
import aws_cdk.assertions as assertions
from infra.infrastructure_stack import InfrastructureStack

def test_ec2_instance_created():
    app = core.App()
    stack = InfrastructureStack(app, "test-stack")
    template = assertions.Template.from_stack(stack)

    template.has_resource_properties("AWS::EC2::Instance", {
        "InstanceType": "t3.micro"
    })

def test_security_group_rules():
    app = core.App()
    stack = InfrastructureStack(app, "test-stack")
    template = assertions.Template.from_stack(stack)

    template.has_resource_properties("AWS::EC2::SecurityGroup", {
        "SecurityGroupIngress": [
            {
                "IpProtocol": "tcp",
                "FromPort": 80,
                "ToPort": 80
            }
        ]
    })

Run tests with:

python -m pytest test/

Monitoring and Maintenance

Monitor the deployment with these commands:

# Check container status
docker ps

# View logs
docker logs nginx-app

# Update deployment
docker pull the-account-id.dkr.ecr.the-region.amazonaws.com/nginx-app:latest
docker stop nginx-app
docker rm nginx-app
docker run -d -p 80:80 --name nginx-app the-account-id.dkr.ecr.the-region.amazonaws.com/nginx-app:latest

Next Steps and Enhancements

This foundation can be extended with:

Load balancing with Application Load Balancer
Auto Scaling Groups for high availability
CloudWatch monitoring and alerting
SSL/TLS certificate management
Database integration
Container orchestration with ECS or EKS

Conclusion

You've successfully created a complete deployment pipeline for a containerized Nginx application using modern DevOps practices. This setup provides a solid foundation for more complex applications while demonstrating key concepts in containerization, cloud infrastructure, and automated deployment.

The combination of Docker, AWS ECR, and EC2 creates a scalable, maintainable deployment solution that can grow with the application needs. The automated CI/CD pipeline ensures consistent deployments while the infrastructure-as-code approach makes the setup reproducible and version-controlled.

For full reference, here is the github I have created:

Github repository for Building and Deploying a Nginx Web Application with Docker and AWS EC2

Setting up APIcast in Red Hat OpenShift Service on AWS (ROSA) via Operator

Clariza Look — Tue, 27 May 2025 08:46:58 +0000

Last week, we took the first steps into the world of API management by setting up a self-hosted 3scale environment. Now, we're taking it to the next level by deploying 3scale APIcast on Red Hat OpenShift Service on AWS (ROSA) using the ROSA operator.

Here's what we learned along the way and why this migration makes sense for the infrastructure.

The Starting Point: Setting up Self-Hosted 3scale

The journey began with a traditional self-hosted 3scale deployment. This gave us hands-on experience with:

API Management Fundamentals: Understanding how 3scale handles API keys, rate limiting, and developer portals
Configuration Management: Learning the ins and outs of API policies, plans, and applications
Integration Patterns: Connecting the backend services to the 3scale gateway

While the self-hosted approach worked well for getting started, we quickly realized we wanted the scalability and managed services that come with a cloud-native approach.

What is ROSA?

Red Hat OpenShift Service on AWS (ROSA) is a fully-managed OpenShift service that runs natively on Amazon Web Services (AWS). It combines Red Hat OpenShift with the scalability and reliability of AWS infrastructure.

Key ROSA Benefits:

Fully Managed: Red Hat handles cluster operations, updates, and maintenance
AWS Native: Deep integration with AWS services like EBS, ELB, and VPC
Enterprise Ready: Built-in security, compliance, and support from Red Hat
Pay-as-you-go: Flexible pricing model based on actual usage

APIcast Deployment Options in ROSA

ROSA provides multiple ways to deploy 3scale APIcast, each suited for different use cases:

1. Operator-Based Deployment (Recommended)

Declarative Configuration: Uses Custom Resource Definitions (CRDs)
Lifecycle Management: Automatic updates and scaling
GitOps Ready: Perfect for CI/CD pipelines
Multi-Environment Support: Easy promotion between dev/staging/prod

2. Helm Charts

Template-Based: Traditional Kubernetes deployment method
Customizable: Fine-grained control over deployment parameters
Version Control: Easy rollback and version management

3. Manual YAML Deployment

Direct Control: Complete control over all Kubernetes resources
Learning Purpose: Great for understanding underlying components
Custom Scenarios: When you need specific configurations not covered by operators

For this setup, we chose the operator-based approach because it provides the best balance of simplicity and control, especially when integrating with the existing self-hosted 3scale management layer.

Why Replicate APIcast on ROSA?

The decision to deploy APIcast on ROSA wasn't about replacing the self-hosted 3scale management layer — it was about creating a safe, isolated testing environment that replicates our current setup.

The Goal: Keep the existing self-hosted 3scale Admin Portal and System while deploying APIcast gateways on ROSA for isolating the current working system in ROSA with a safe way of testing APIs reaching the backend services.

High Availability: Eliminate single points of failure in your API gateway layer
Blue-Green Deployments: Enable zero-downtime updates by switching traffic between replicas
Testing Environment: Create a production-like environment for testing API changes
Disaster Recovery: Maintain backup gateways in different availability zones
Performance Testing: Isolate load testing from production traffic

Understanding the Architecture

Before diving into the implementation, it's crucial to understand where APIcast fits in the request flow:

ROSA (Red Hat OpenShift Service on AWS) contains:

OpenShift Route: Acts as the ingress controller, handling external traffic routing
Kubernetes Service: Provides load balancing and service discovery for the APIcast pods
3scale APIcast Pods: Multiple instances of the API gateway running as containers

3scale Components:

APIcast Gateway Pods (running in ROSA): Handle the actual API traffic, rate limiting, authentication, and proxying to backend services
3scale Management (self-hosted): Can be deployed separately to manage API policies, analytics, and configuration

Traffic Flow:

Internet traffic comes through the OpenShift Route
Gets load balanced by the Kubernetes Service
Distributed across multiple APIcast gateway pods
APIcast pods proxy requests to backend APIs
3scale Management provides configuration and policies to APIcast pods

The key distinction is that APIcast runs as containerized pods within ROSA, while the 3scale Management component can be self-hosted either within the same ROSA cluster or on separate infrastructure, depending on the deployment preferences.

The APIcast Gateway Role:

Authentication: Validates API keys and handles OAuth flows
Rate Limiting: Enforces quotas and throttling policies
Policy Enforcement: Applies transformation, caching, and security policies
Analytics Collection: Gathers metrics for monitoring and billing
Request Routing: Directs traffic to appropriate backend services

This positioning makes APIcast the critical control point for all API traffic—it's not just a proxy, but an intelligent gateway that enforces the API strategy.

Deploying APIcast on ROSA with the Operator

With the ROSA cluster already in place and the 3scale operator installed, we were ready to deploy APIcast instances that would integrate with the existing self-hosted 3scale management layer.

This guide walks through the complete process of replicating an existing APIcast deployment connected to a self-hosted 3scale management platform.

Configuring APIcast Custom Resources

The operator uses Custom Resource Definitions (CRDs) to manage APIcast deployments:

apiVersion: apps.3scale.net/v1alpha1
kind: APIcast
metadata:
  name: apicast-development
  namespace: apicast
spec:
  adminPortalURL: https://self-hosted-3scale-admin.example.com
  exposedHost:
    host: apps.[domain of our openshift].openshiftapps.com
  replicas: 3
  resources:
    limits:
      cpu: 1000m
      memory: 128Mi
    requests:
      cpu: 500m
      memory: 64Mi

Prerequisites

OpenShift cluster access with appropriate permissions
Working self-hosted 3scale deployment
Access to 3scale admin portal
oc CLI tool configured
curl for testing

Step 1: Analyze Existing APIcast Configuration

First, we need to understand how the current APIcast is configured. This ensures the replica maintains the same behavior.

# Get the deployment configuration
oc get deployment [source-apicast-name] -n [source-namespace] -o yaml > apicast-config.yaml

# Check environment variables
oc get deployment [source-apicast-name] -n [source-namespace] \
  -o jsonpath='{.spec.template.spec.containers[0].env}' | jq .

Key Environment Variables to Note

[
  {
    "name": "THREESCALE_PORTAL_ENDPOINT",
    "value": "https://TOKEN@3scale-admin.apps.cluster.domain.com"
  },
  {
    "name": "THREESCALE_DEPLOYMENT_ENV",
    "value": "development"
  }
]

Step 2: Create the Replica Namespace

Organize the apicast replica in a dedicated namespace for better isolation and management.

# Create replica namespace
oc create namespace apicast-replica

# Label for organization
oc label namespace apicast-replica purpose=apicast-replica

Step 3: Deploy APIcast Replica

Create Deployment Configuration (based from the apicast-config.yaml with updated metadata names)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apicast-replica
  namespace: apicast-replica
  labels:
    app: apicast-replica
spec:
  replicas: 1
  selector:
    matchLabels:
      app: apicast-replica
  template:
    metadata:
      labels:
        app: apicast-replica
    spec:
      containers:
      - name: apicast
        image: quay.io/3scale/apicast:latest
        env:
        - name: THREESCALE_PORTAL_ENDPOINT
          value: "https://TOKEN@3scale-admin.apps.[domain of our openshift].openshiftapps.com"
        - name: THREESCALE_DEPLOYMENT_ENV
          value: "development"
        - name: APICAST_CONFIGURATION_LOADER
          value: "boot"
        - name: APICAST_LOG_LEVEL
          value: "notice"
        - name: APICAST_PATH_ROUTING
          value: "true"
        - name: APICAST_RESPONSE_CODES
          value: "true"
        livenessProbe:
          httpGet:
            path: /status/live
            port: management
          initialDelaySeconds: 10
          timeoutSeconds: 1
        readinessProbe:
          httpGet:
            path: /status/ready
            port: management
          initialDelaySeconds: 15
          timeoutSeconds: 1

Deploy the Replica

# Apply the deployment
oc apply -f apicast-replica-deployment.yaml

# Watch the deployment
oc get pods -n apicast-replica -w

Step 4: Create Service and Route for the Apicast

Service Configuration

apiVersion: v1
kind: Service
metadata:
  name: apicast-replica-service
  namespace: apicast-replica
spec:
  selector:
    app: apicast-replica
  ports:
  - name: proxy
    port: 8080
    targetPort: 8080
  - name: management
    port: 8090
    targetPort: 8090

Route Configuration

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: api-replica
  namespace: apicast-replica
spec:
  host: api-replica.apps.[domain of our openshift].openshiftapps.com
  to:
    kind: Service
    name: apicast-replica-service
  port:
    targetPort: proxy
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect

Then apply configs

# Apply service and route
oc apply -f apicast-replica-service.yaml
oc apply -f apicast-replica-route.yaml

If we go to Rosa and check, it will look like this

Step 5: Configure 3scale for the Replica

APIcast loads services based on the gateway URL configured in 3scale, not just the connection to 3scale.

The Configuration Challenge

When APIcast starts, it:

Connects to 3scale using THREESCALE_PORTAL_ENDPOINT
Downloads service configurations for THREESCALE_DEPLOYMENT_ENV (staging/production)
Only loads services where the proxy endpoint matches the incoming request host

Update Product Configuration

In the self-hosted 3scale Admin Portal:

Go to Products > Select the product (e.g., "Test API")
Go to Integration > Settings
Update the Staging Public Base URL:

https://apicast-replica.apps.[domain].openshiftapps.com

Critical Step: Go to Integration → Configuration

Click "Promote v[X] to Staging"

Important: The "Promote to Staging" step is what actually deploys your configuration changes to APIcast. Without this, APIcast continues using the old configuration.

Step 6: Restart and Verify

Restart APIcast to Load New Configuration

#Force APIcast to reload configuration
> oc rollout restart deployment/apicast-replica -n apicast-replica

#Result
> deployment.apps/apicast-replica restarted

# Wait for pods to be ready
> oc get pods -n apicast-replica -w

Test Authentication

curl -v "https://api-replica.../status/ready" -H "user-key: API_APPLICATION_KEY"

Test Actual API Endpoints

curl -X POST 'https://api-replica.../your/api/endpoint' \
     -H 'user-key: APPLICATION_API_KEY' \
     -H 'Content-Type: application/json' \
     -d '{"test": "data"}'

Conclusion

Replicating APIcast with self-hosted 3scale requires careful attention to service configuration alignment. The key insight is that APIcast service discovery depends on matching the configured gateway URLs in 3scale with the actual domains your replica serves.

By following this guide, we've created a fully functional APIcast replica that can serve as a backup, testing environment, or part of a blue-green deployment strategy. Remember that the "Promote to Staging" step in 3scale is critical - configuration changes don't take effect until promoted and APIcast is restarted.

The replica approach provides excellent flexibility for API gateway management while maintaining high availability and enabling safe deployment practices. With proper monitoring and procedures in place, you can confidently manage multiple APIcast instances serving the critical API infrastructure.

Additional Resources

This guide was created based on real-world APIcast replica deployment experience. The configuration examples are anonymized but reflect actual working deployments.

Migrating to Self-Hosted 3scale API Management on ROSA Kubernetes

Clariza Look — Fri, 16 May 2025 12:50:53 +0000

I was tasked to migrate from a Red Hat-hosted 3scale portal to a self-hosted version in ROSA (Red Hat OpenShift Service on AWS). This presented quite a challenge as my knowledge in Kubernetes was mostly theoretical, based on studying for the Kubernetes and Cloud Native Associate (KCNA) certification exam.

The goal was to recreate a self-hosted version of 3scale using an operator in ROSA, but what I thought would be a straightforward deployment turned into a valuable learning experience.

What is Red Hat-Managed/hosted 3scale?

When using Red Hat-hosted 3scale (also known as "SaaS" or managed 3scale), all infrastructure complexities are abstracted away. Red Hat handles the deployment, maintenance, updates, and scaling of the platform.

As a user, you simply access a provided portal URL and focus on managing your APIs rather than worrying about the underlying infrastructure. Your daily tasks revolve around the actual API management activities like adding backends, configuring products, creating applications, setting up authentication, and managing rate limits.

It's a convenient option that requires minimal operational overhead, allowing your team to focus on API strategy rather than platform management.

What is self-hosted 3scale?

In contrast, self-hosted 3scale brings both flexibility and responsibility. You gain complete control over your deployment configuration, integration with internal systems, customization options, and data locality.

Since the infrastructure runs on Kubernetes (in my case, ROSA - Red Hat OpenShift Service on AWS), you have access to all the native Kubernetes capabilities for scaling, monitoring, and management.

However, this freedom comes with the need to manage the entire application lifecycle within the Kubernetes ecosystem: installation via operators or templates, configuration through custom resources, scaling via horizontal pod autoscalers, implementing backup strategies, and handling upgrades.

You're responsible for ensuring high availability with proper pod distribution, performance tuning through resource allocation, and troubleshooting any issues that arise in both the 3scale application components and the underlying Kubernetes resources.

The Migration

Migrating from managed to self-hosted represented a significant shift in responsibilities, and I was about to discover just how much Red Hat had been handling behind the scenes.

This blog post documents a real-world troubleshooting journey that encountered and overcame significant challenges:

Missing Routes for Admin Access
DNS resolution issues preventing access to Red Hat's container registry
Architecture mismatch between my ARM-based MacBook for and the x86_64 docker container images required for deployment
PVC Access Mode Issues
Resource Constraints
Missing Service for App Components

By sharing this experience, I hope to help others who might encounter similar issues during their deployment process, especially those who are transitioning from theoretical Kubernetes knowledge to practical application.

The Initial Deployment Attempt

We started by creating a dedicated namespace for our 3scale deployment:

oc create namespace 3scale-backup

After switching to this namespace (oc project 3scale-backup), we downloaded the 3scale API Management Platform template:

curl -o amp.yml https://raw.githubusercontent.com/3scale/3scale-amp-openshift-templates/master/amp/amp.yml

Then we tried to deploy 3scale using this template:

oc new-app --file=amp.yml \
  --param WILDCARD_DOMAIN=apps.[domain of your openshift].openshiftapps.com \
  --param ADMIN_PASSWORD=password123

The template processing appeared successful, creating numerous resources:

Imagestreams
Deployment configs
Services
Routes
Persistent volume claims
Secrets

oc get all -n [your namespace]

However, when checking the status of the pods, we noticed that many deployments were either not starting, with errors, crashLoopBackOff or stuck in initialization phases:

oc get pods

While some components like Redis and database pods were running fine, critical components like backend-listener, backend-worker, and backend-cron were not deploying at all.

The system components were also failing during initialization.

Challenge 1: Missing Routes for Admin Access

Our first challenge was that the URLs for accessing the admin portal https://3scale-admin.apps.[YOUR-ROSA-DOMAIN].openshiftapps.com were showing "Application is not available".

The reason was simple - the template had not created the necessary routes for our self-hosted 3scale services. We manually created them:

oc create route edge system-admin --service=system-provider --hostname=3scale-admin.apps.[YOUR-ROSA-DOMAIN].openshiftapps.com
oc create route edge system-developer --service=system-developer --hostname=3scale.apps.[YOUR-ROSA-DOMAIN].openshiftapps.com
oc create route edge system-master --service=system-master --hostname=master.apps.[YOUR-ROSA-DOMAIN].openshiftapps.com

However, after creating the routes, the admin portal still wasn't accessible. Digging into the logs with oc logs system-app-1-hook-pre, we discovered a more fundamental issue.

Challenge 2: DNS Resolution Issues

The pre-deployment hook was failing with a specific error:

ThreeScale::Core::APIClient::ConnectionError: connection refused: backend-listener:80

Further investigation revealed that the backend components weren't deployed at all. When checking the deployment configs:

oc get dc/backend-listener
NAME               REVISION   DESIRED   CURRENT   TRIGGERED BY
backend-listener   0          1         0         config,image(amp-backend:2.12)

We saw that backend-listener, backend-worker, and backend-cron had REVISION 0 and CURRENT 0, indicating they hadn't been deployed.

The root cause was found in the imagestream:

oc describe imagestream amp-backend

This showed an error:

error: Import failed (InternalError): Internal error occurred: registry.redhat.com/3scale-amp2/backend-rhel8:3scale2.12: Get "https://registry.redhat.com/v2/": dial tcp: lookup registry.redhat.com on 100.10.0.11:23: no such host

Our OpenShift cluster couldn't resolve the hostname registry.redhat.com due to DNS issues. This was confirmed by attempting to run:

nslookup registry.redhat.com

Which returned "No answer" from the DNS server.

Or technically it does not pull the image required from the RedHat registry to the pods.

So our workaround was to manually pull the image to the docker (locally), then push that image to the namespace's private RedHat registry.

Challenge 3: Architecture Mismatch

While working to address the DNS issues , we discovered another challenge - we were trying to pull Red Hat's container images on an ARM64-based machine (likely an Apple Silicon Mac), but the images were only available for x86_64 architecture.

When attempting to pull the images directly:

docker login [RedHat Credentials]
docker pull registry.redhat.io/3scale-amp2/backend-rhel8:3scale2.12

We received:

no matching manifest for linux/arm64/v8 in the manifest list entries

The Solution Process

We implemented a multi-step solution to overcome these challenges:

Step 1: Authentication with Red Hat Registry

First, we logged in to the Red Hat Container Registry:

docker login registry.redhat.io

Step 2: Architecture-aware Image Pulling

Because I was using macOS and having the docker desktop installed in it, the pulled image does not match with the operating system's arch.

To overcome the architecture mismatch, we explicitly specified the platform when pulling:

docker pull --platform linux/amd64 registry.redhat.io/3scale-amp2/backend-rhel8:3scale2.12

This successfully pulled the image by using Rosetta 2 emulation on macOS.

Step 3: Exposing the OpenShift Registry

To make our OpenShift registry accessible:

oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge

Step 4: Pushing Images to Internal Registry

We pushed the pulled images to our OpenShift internal registry:

# Get credentials
TOKEN=$(oc whoami -t)
REGISTRY=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')

# Login to registry
docker login -u kubeadmin -p $TOKEN $REGISTRY

# Tag and push
docker tag registry.redhat.io/3scale-amp2/backend-rhel8:3scale2.12 $REGISTRY/[namespace]/amp-backend:2.12
docker push $REGISTRY/[namespace]/amp-backend:2.12

Step 5: Updating ImageStreams

We updated the imagestream to point to our locally pushed image:

oc tag $REGISTRY/[namespace]/amp-backend:2.12 amp-backend:2.12 --source=docker

This automatically triggered the deployment due to the ImageChange trigger on the deployment config.

Results

After implementing these steps for the backend-listener component, the deployment began successfully (at least for this resource!).

Challenge 4: PVC Access Mode Issues

So we went to the 3Scale self-hosted Admin Portal and checked that still it's not working.

We checked the pods and found out that some of them are having issues.

The error logs show that several deployments are failing because their pods are taking too long to become available (timeout errors):

apicast-production-1-deploy: "pods took longer than 1800 seconds to become available"
system-sidekiq-1-deploy: "pods took longer than 1200 seconds to become available"
system-sphinx-1-deploy: "pods took longer than 1200 seconds to become available"

This typically happens when pods are stuck in a pending or initializing state for too long.

So we checked the logs of the problematic pods and checked the PVC as well.

# Check logs for apicast-production deployment
oc logs apicast-production-1-deploy

# Check logs for system-sidekiq deployment
oc logs system-sidekiq-1-deploy

# Check logs for system-sphinx deployment
oc logs system-sphinx-1-deploy

# Check events for the pending pod
oc describe pod system-app-1-hook-pre

We discovered a storage issue where the system-storage PVC was failing to provision:

oc get pvc
NAME                    STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
backend-redis-storage   Bound     pvc-ss987s5b-026a-4srg-au97-549d8958933a   1Gi        RWO            gp3            53m
mysql-storage           Bound     pvc-72s43210-s033-4c8w-ar53-043bf3kk1496   1Gi        RWO            gp3            53m
system-redis-storage    Bound     pvc-s3r1111d2-4d57-41dd-9066-c488eda666d4   1Gi        RWO            gp3            53m
system-storage          Pending

The error was related to access modes:

failed to provision volume with StorageClass "gp3": rpc error: code = InvalidArgument desc = Volume capabilities MULTI_NODE_MULTI_WRITER not supported. Only AccessModes[ReadWriteOnce] supported.

We fixed it by creating a new PVC with the correct access mode:

# First, delete pods using this PVC
oc delete pod system-app-1-hook-pre

# Back up the current PVC definition
oc get pvc system-storage -o yaml > system-storage-pvc.yaml

# Delete the stuck PVC
oc delete pvc system-storage

# Create a new PVC with the correct settings
oc create -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: system-storage
  namespace: [Namespace]
  labels:
    app: 3scale-api-management
    threescale_component: system
    threescale_component_element: app
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: gp3
EOF

After fixing the PVC issue, restart the deployments:

oc rollout retry dc/system-app
oc rollout retry dc/apicast-production
oc rollout retry dc/backend-listener
oc rollout retry dc/system-sidekiq
oc rollout retry dc/system-sphinx

The PVC issue got fixed, and the system-storage PVC is now correctly bound to a volume.

oc get pvc

NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
backend-redis-storage   Bound     pvc-ss987s5b-026a-4srg-au97-549d8958933a   1Gi        RWO            gp3            53m
mysql-storage           Bound     pvc-72s43210-s033-4c8w-ar53-043bf3kk1496   1Gi        RWO            gp3            53m
system-redis-storage    Bound     pvc-s3r1111d2-4d57-41dd-9066-c488eda666d4   1Gi        RWO            gp3            53m
system-storage          Bound    pvc-2286196a-8885-490s-11c1-654320bd8a5a6   1Gi        RWO            gp3            116s

Challenge 5: Resource Constraints

Even after resolving the PVC issue, pods were still stuck in Pending state due to insufficient resources:

oc describe pod system-app-2-mr25z

...
Warning  FailedScheduling  2m34s  default-scheduler  0/9 nodes are available: 2 Insufficient cpu, 3 node(s) had untolerated taint {node-role.kubernetes.io/infra: }, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 6 node(s) had volume node affinity conflict.

We reduced the resource requirements to make the pods fit on the available nodes:

oc patch dc/system-app -p '{"spec":{"template":{"spec":{"containers":[{"name":"system-master","resources":{"requests":{"cpu":"25m","memory":"400Mi"}}},{"name":"system-provider","resources":{"requests":{"cpu":"25m","memory":"400Mi"}}},{"name":"system-developer","resources":{"requests":{"cpu":"25m","memory":"400Mi"}}}]}}}}'
oc patch dc/apicast-production -p '{"spec":{"template":{"spec":{"containers":[{"name":"apicast-production","resources":{"requests":{"cpu":"25m","memory":"128Mi"}}}]}}}}'
oc patch dc/system-sidekiq -p '{"spec":{"template":{"spec":{"containers":[{"name":"system-sidekiq","resources":{"requests":{"cpu":"25m","memory":"250Mi"}}}]}}}}'
oc patch dc/system-sphinx -p '{"spec":{"template":{"spec":{"containers":[{"name":"system-sphinx","resources":{"requests":{"cpu":"25m","memory":"250Mi"}}}]}}}}'

After applying these patches, we restarted the failed components:

# Retry system-sidekiq and system-sphinx deployments
oc rollout retry dc/system-sidekiq
oc rollout retry dc/system-sphinx

That got fixed too!!


➜  oc get services

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
apicast-production   ClusterIP   xxx.xx.xxx.xxx   <none>        8080/TCP,8090/TCP   83m
apicast-staging      ClusterIP   xxx.xx.xxx.xxx    <none>        8080/TCP,8090/TCP   83m
backend-listener     ClusterIP   xxx.xx.xxx.xxx   <none>        3000/TCP            83m
backend-redis        ClusterIP   xxx.xx.xxx.xxx     <none>        6379/TCP            83m
system-developer     ClusterIP   xxx.xx.xxx.xxx    <none>        3000/TCP            83m
system-master        ClusterIP   xxx.xx.xxx.xxx     <none>        3000/TCP            83m
system-memcache      ClusterIP   xxx.xx.xxx.xxx    <none>        11211/TCP           83m
system-mysql         ClusterIP   xxx.xx.xx.xxx     <none>        3306/TCP            83m
system-provider      ClusterIP   xxx.xx.x.xxx      <none>        3000/TCP            83m
system-redis         ClusterIP   xxx.xx.xxx.xxx   <none>        6379/TCP            83m
system-sphinx        ClusterIP   xxx.xx.xxx.xxx   <none>        9306/TCP            83m
zync                 ClusterIP   xxx.xx.xxx.xx     <none>        8080/TCP            83m
zync-database        ClusterIP   xxx.xx.xx.xxx    <none>        5432/TCP            83m

➜  oc get routes

NAME                         HOST/PORT                                                             PATH         SERVICES             PORT      TERMINATION     WILDCARD
backend                      backend-3scale.apps.[YOUR-DOMAIN].openshiftapps.com                               backend-listener     http      edge/Allow      None
system-admin                 3scale-admin.apps.[YOUR-DOMAIN].openshiftapps.com                                 system-app           3000      edge/Allow      None
system-developer             3scale.apps.[YOUR-DOMAIN].openshiftapps.com                          /developer   system-app           3001      edge/Allow      None
system-master                master.apps.[YOUR-DOMAIN].openshiftapps.com                                       system-app           3002      edge/Allow      None
system-provider              3scale.apps.[YOUR-DOMAIN].openshiftapps.com                                       system-app           3000      edge/Allow      None
zync-3scale-api-hhhjs        api-3scale-apicast-production.apps.[YOUR-DOMAIN].openshiftapps.com                apicast-production   gateway   edge/Redirect   None
zync-3scale-api-phh9n        api-3scale-apicast-staging.apps.[YOUR-DOMAIN].p1.openshiftapps.com                   apicast-staging      gateway   edge/Redirect   None
zync-3scale-master-nhhht     HostAlreadyClaimed                                                                 system-master        http      edge/Redirect   None
zync-3scale-provider-q9hh9   HostAlreadyClaimed                                                                 system-developer     http      edge/Redirect   None
zync-3scale-provider-shh6z   HostAlreadyClaimed                                                                 system-provider      http      edge/Redirect   None

Since the containers are starting up, it should be a matter of minutes before we can access the admin portal.

We were just pod status, and when system-app shows 3/3 ready, tried accessing the admin portal at:

https://3scale-admin.apps.[YOUR-DOMAIN].openshiftapps.com

But then, it is still UNAVAILABLE.

Challenge 6: Missing Service for App Components

Even after all pods were running, the admin portal was not accessible. The issue was that we created routes pointing to a service named "system-app" which didn't exist:

oc get routes

NAME                         HOST/PORT                                                  PATH        SERVICES      PORT    TERMINATION  WILDCARD
system-admin                 3scale-admin.apps.[YOUR-DOMAIN].openshiftapps.com                     system-app    3000    edge/Allow   None
system-developer             3scale.apps.[YOUR-DOMAIN].openshiftapps.com              /developer   system-app    3001    edge/Allow   None
system-master                master.apps.[YOUR-DOMAIN].openshiftapps.com                           system-app    3002    edge/Allow   None
system-provider              3scale.apps.[YOUR-DOMAIN].openshiftapps.com                           system-app    3000    edge/Allow   None

oc describe service system-app

Error from server (NotFound): services "system-app" not found

We fixed this by creating the missing service:

bash
oc create -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: system-app
  namespace: [NAMESPACE]
  labels:
    app: 3scale-api-management
spec:
  ports:
  - name: provider
    port: 3000
    protocol: TCP
    targetPort: 3000
  - name: developer
    port: 3001
    protocol: TCP
    targetPort: 3001
  - name: master
    port: 3002
    protocol: TCP
    targetPort: 3002
  selector:
    deploymentConfig: system-app
  type: ClusterIP
EOF

Final Result
After working through all these challenges, we finally had a fully operational 3scale deployment:

bash
oc get pods

NAME                        READY   STATUS      RESTARTS      AGE
apicast-production-4-7hh00  1/1     Running     0             2m12s
apicast-staging-1-6hh00     1/1     Running     0             83m
backend-cron-2-6955a        1/1     Running     0             23m
backend-listener-1-5hh00    1/1     Running     0             26m
backend-redis-1-mhh005      1/1     Running     0             57m
backend-worker-2-lr8gb      1/1     Running     0             23m
system-app-3-7ln8g          3/3     Running     0             85s
system-memcache-1-xddig     1/1     Running     0             80m
system-mysql-1-ee4wt        1/1     Running     0             80m
system-redis-1-45hh0        1/1     Running     0             80m
zync-1-l7ghy                1/1     Running     0             80m
zync-database-1-dt3l9       1/1     Running     0             80m
zync-que-1-wwri9            1/1     Running     2 (80m ago)   80m

With all components running, finally, we were able to access the 3scale admin portal and begin configuring our APIs.

Verify Deployment

# Check all pods are running
oc get pods

# Expected output should show all pods in Running or Completed state:
# - system-app-X-XXXXX (3/3 Running)
# - apicast-production-X-XXXXX (1/1 Running)
# - apicast-staging-X-XXXXX (1/1 Running)
# - system-sidekiq-X-XXXXX (1/1 Running)
# - system-sphinx-X-XXXXX (1/1 Running)
# - backend-* pods (1/1 Running)
# - system-mysql-X-XXXXX (1/1 Running)
# - system-redis-X-XXXXX (1/1 Running)
# - zync-* pods (1/1 Running)

Key Lessons Learned

DNS Resolution is Critical: Ensure your OpenShift cluster can resolve external registry hostnames before attempting deployments that rely on them.
Architecture Awareness: When working with enterprise container images on ARM-based development machines, be explicit about architecture requirements using the --platform flag.
Manual Image Mirroring: In restricted environments, manually pulling and pushing images to an internal registry is a viable workaround.
ImageStream Mechanics: Understanding how OpenShift's ImageStreams work is essential for troubleshooting deployment issues.
Network Policies: In enterprise environments, network policies may restrict access to external registries, requiring coordination with network administrators.

Common Issues and Solutions

Pod stuck in Pending: Usually resource constraints - reduce resource requests
PVC mounting issues: Check storage class and access modes
Routes not working: Ensure services exist and selectors match pods
Application not available: Create missing system-app service
Database connection issues: Check that system-mysql pod is running and accessible

Final Verification Checklist

All pods are in Running state (except completed deployment/hook pods)
Routes are accessible and don't show "Application not available"
Can log into Admin Portal successfully
Can access Developer Portal
Master Portal is accessible (if needed)
Default passwords have been changed

Once we successfully access the 3scale portals, we can begin migrating the existing 3scale components from another environment or start adding the new components in 3scale.

A. Backend APIs

API definitions and configurations
Authentication settings
Rate limiting rules

B. Products (API Products)

Product configurations
Application plans
Pricing rules
Methods and metrics

C. Applications

Application keys and secrets
Application plans assignments
Usage statistics (if needed)

D. Accounts and Users

Developer accounts
Admin users
Access permissions

E. Policies

Custom policies
Policy chains
Configuration settings

F. Developer Portal

Custom pages and templates
Documentation
CMS content

Conclusion

Deploying complex solutions like 3scale API Management in restricted network environments or across architecture boundaries presents unique challenges. By understanding the underlying issues and implementing a systematic approach to manually mirror images, we were able to overcome these obstacles.

While this process requires more manual effort than a standard deployment, it demonstrates the flexibility of OpenShift's container management capabilities and provides a path forward for deployments in environments with similar restrictions.

From Infra to Platform: A Day in the Life of a DevOps Team -Building Developer Tools

Clariza Look — Wed, 07 May 2025 05:47:13 +0000

Today marked a shift in our DevOps journey — from managing infrastructure to building a platform that empowers developers to move faster and smarter.

The Meeting That Sparked a Platform Vision

Our team, traditionally focused on infrastructure management for one of our core applications, had a productive meeting with our Solutions Architect — who also leads the development team behind another application.

The goal? To brainstorm how we can optimized dev operational processes that will make life easier for our internal developers through automation.

The Problem: Repetitive Tasks Slowing Down Developers

Developers often perform repetitive infrastructure-related tasks like database refreshes. These actions, while simple, are time-consuming and error-prone when done manually. The idea was simple but powerful: a one-click web interface (an api) that allows developers to trigger these tasks securely and reliably.

The Vision: A Self-Service Platform Tool

We envisioned a Single Page Application (SPA) where users authenticate via Azure AD, then access buttons linked to backend APIs that trigger specific tasks. Think of it as a developer control panel — a self-service tool that abstracts the complexity of infrastructure operations.

Architecture Talk: From Frontend to Infra

The conversation quickly shifted toward how we’d architect and build this platform. Some of the key topics we discussed included:

Authentication: Azure Active Directory (OAuth2) for SSO and secure access.
Frontend: Possibly React, due to its community support and simplicity with SPAs.
Backend: Python or TypeScript for the API layer — both have pros and cons, and we’re weighing based on team familiarity and ecosystem.
Infrastructure as Code (IaC): CDK vs. SST — should we go full AWS-native with CDK, or use SST for a more streamlined developer experience?
CI/CD Pipeline: How to build and deploy the platform efficiently — integrating testing, linting, and deployment workflows.
Repository Strategy: Monorepo vs. multi-repo — placing frontend, backend, and infra code all in one place for cohesion or splitting them for separation of concerns.
Testing: Not just the API, but also integration with cloud components and edge cases that could arise in automation.

DevOps Wearing a Dev Hat

As infrastructure folks, it felt a bit like we were crossing into developer territory — but in reality, we’re just evolving. This isn’t traditional app development; it's platform engineering — building tools and experiences for developers to accelerate their workflow, all while maintaining security and reliability.

What’s Next

We’re still in the planning phase, but the momentum is real. This platform could become a cornerstone of how our organization builds and delivers software. It's DevOps at its finest: merging development practices with operational expertise to create something truly impactful.

Systems Programming and Overstimulation: Finding Balance in a Complex Technical Industry

Clariza Look — Fri, 11 Apr 2025 03:18:08 +0000

In the demanding world of systems programming and DevOps, where we work with infrastructure, deployment pipelines, and complex system configurations, cognitive overload isn't just an occasional annoyance—it's an occupational hazard.

As a mid-level DevOps engineer who transitioned from a marketing career, I've experienced a significant shift in how my brain handles daily work and mental stimulation.

From Marketing Multitasking to Technical Deep Focus

My career transition from marketing specialist to DevOps engineer brought an unexpected challenge: a complete rewiring of how I manage cognitive load and attention.

In marketing, my days were filled with varied, shorter tasks—crafting content for ad messaging, targeting, and bidding strategies for Google ads, analyzing campaign metrics, coordinating with design and dev teams, and jumping between multiple client projects. This diversity of work actually suited a certain type of attention management where context-switching was frequent but the depth of focus required for each task was relatively manageable.

DevOps engineering creates an entirely different cognitive environment with a perfect storm of factors that lead to overstimulation:

1. Extreme Technical Depth

When debugging why a newly deployed application returns a 404 error or troubleshooting a Lambda function that can't connect to a database, we're not just dealing with a single layer of abstraction. We're simultaneously juggling:

Network configuration and security groups
Container settings or serverless execution contexts
IAM permissions and access policies
Application configurations across environments
CI/CD pipeline variables and deployment logs

Each of these domains demands deep technical understanding, and we're constantly switching between them as we solve problems. This mental context-switching exacts a cognitive toll that accumulates throughout the workday.

2. High-Stakes Decision Making

In systems programming, the consequences of errors can be severe. A subtle if else condition in the infrastructure setup might cause rare data path not being found. An inefficient security patterns could create security vulnerabilities and risk affecting credentials gets exposed to the web. This creates an environment where:

Every decision feels consequential
The margin for error is slim
The pressure to "get it right" is constant

This pressure amplifies the mental load and can lead to decision fatigue—a state where our ability to make good technical choices diminishes as the day progresses.

3. Attention Fragmentation

Modern development environments don't help matters. A typical systems programmer might simultaneously manage:

Multiple terminal sessions
Several IDE windows
Documentation browsers
Communication tools like Slack or Discord
Monitoring dashboards
CI/CD pipelines

Each of these information sources competes for our attention, creating a constant state of partial focus that leaves us mentally drained.

The Debugging Intensity: A Different Kind of Focus

The most jarring transition has been debugging complex systems issues. In marketing, I could often multitask effectively—perhaps answering emails while sitting in on a conference call, or editing copy while monitoring social media responses.

With DevOps and systems work, debugging demands a completely different cognitive mode. When troubleshooting a failed deployment pipeline or tracking down an intermittent network issue, there's no multitasking. There's only the problem and a deep, sustained focus that can last for hours. This intense, single-track concentration is mentally taxing in ways my previous career never prepared me for.

Where marketing work often provided frequent small dopamine hits from completing numerous smaller tasks throughout the day, debugging provides no satisfaction until the entire problem is solved—which might take hours or even days of concentrated effort.

Recognizing the Signs of Overstimulation

Since transitioning to technical field, I've learned to recognize when my brain is approaching its processing limits:

Debugging Plateau: When I've been staring at a complex certificate not found issue for hours and my analytical ability starts to degrade
Implementation Paralysis: When faced with multiple valid approaches to a systems design problem, I suddenly can't evaluate the tradeoffs clearly
Documentation Blindness: Reading the same paragraph of technical documentation repeatedly and not understanding it
Environmental Sensitivity: Becoming unusually irritated by background conversations, office noise, or even the sound of keyboard clicks
Technical Tunnel Vision: Fixating on one approach to a problem while missing obvious alternatives

These are all signs that my cognitive systems are overloaded and need a reset.

Strategies That Actually Work

Through trial and error in my journey from marketing specialist to mid-level DevOps engineer, I've developed several techniques to manage this new type of overstimulation without sacrificing productivity:

1. Intentional Attention Management

Rather than attempting to multitask constantly (which research consistently shows is ineffective), I block my day into distinct modes:

Deep Work Blocks: 90-120 minute sessions of uninterrupted focus on a single complex problem, with all notifications disabled
Shallow Work Periods: Time allocated for code reviews, emails, and other less cognitively demanding tasks
Collaboration Windows: Scheduled times for meetings and pair programming, concentrated into specific parts of the day

This approach allows my brain to settle into appropriate levels of focus for each type of work rather than constantly shifting gears.

2. Environmental Design

I've carefully crafted my work environment to minimize unnecessary stimulation:

Noise-canceling headphones playing ambient sound or instrumental music
A (slightly) clean desk policy with minimal visual distractions
A dedicated secondary monitor solely for documentation
Strategic use of natural lighting

3. Cognitive Offloading

Instead of keeping complex system details in working memory, I aggressively externalize information:

Detailed diagrams of system components and interactions
Personal documentation of debugging processes
Checklists for recurring complex procedures
Digital note-taking systems that capture contextual details about problems

This practice preserves mental bandwidth for the creative problem-solving aspects of systems programming.

4. Physiological Reset Techniques

When I notice signs of overstimulation, I use specific physical interventions:

Brief outdoor walks without devices
Even 5–10 minutes outside, unplugged from screens, helps ground me and clear mental clutter.
Modified 4-7-8 breathing exercises
Deep, paced breathing reduces cortisol levels and helps shift me into a calmer state.
Quick bursts of physical activity
A few pushups, dumbbell exercises, jumping jacks, or even spontaneous mid-day karaoke can rewire my focus by engaging different neural pathways.
Hydration + protein-based snacks
Staying hydrated and having a quick protein snack keeps my energy stable and wards off the fatigue that contributes to overwhelm.
Low-carb, high-fiber lunches
This dietary choice helps minimize sugar spikes and the post-lunch energy crash that can derail my productivity.
Talking to people
Sometimes just a short conversation—especially with someone positive or empathetic—helps reset my nervous system and reminds me I'm not alone.

These simple biological interventions have surprising effects on cognitive resilience.

5. Technical Context Management

To reduce the mental overhead of switching between complex systems:

I maintain detailed "context documents" for each major project
I use consistent development environments across projects
I create automation scripts for repetitive development tasks
I implement code organization patterns

These practices reduce the "reorientation tax" paid when moving between different technical domains.

Embracing the Career Transition

The transition from marketing's varied, multitasking-friendly environment to the deep technical focus required in DevOps represents more than just learning new skills—it's required rewiring how I approach work at a fundamental level.

Some days I still find myself instinctively reaching to check email or Slack during a debugging session, a habit that served me well in marketing but now only fragments my focus and extends the time needed to solve complex technical problems.

I've had to accept that my previous ability to juggle multiple streams of attention—once a point of pride in my marketing career—is actually counterproductive in my new role. The mental muscles I'm developing now are different: sustained concentration, methodical problem decomposition, and the patience to trace complex system interactions without taking shortcuts.

The Bigger Picture: Sustainable Technical Expertise

As technical professionals, we're knowledge workers whose primary tool is our cognitive capacity. Protecting this resource isn't just about short-term productivity—it's about career longevity.

The reality is that burnout and chronic overstimulation can permanently degrade our ability to solve complex problems. I've watched colleagues leave the field entirely after years of ignoring their cognitive limits.

Learning to work with—rather than against—our brain's attentional systems is perhaps the most important meta-skill we can develop in technical roles. The technologies we work with will change, but the fundamental challenge of managing cognitive load will remain constant.

By developing awareness of our mental states and implementing deliberate practices to manage overstimulation, we can sustainably tackle complex systems challenges for decades rather than burning bright and fading fast—even when coming from non-traditional backgrounds like marketing.

What strategies do you use to manage cognitive load in your technical work? Share your experiences in the comments below.