Forem: Maureen Chebet

Optimizing CI/CD Pipelines on AWS: From 45 Minutes to Under 15

Maureen Chebet — Wed, 26 Nov 2025 18:38:54 +0000

The Performance Crisis

Your Jenkins pipeline is a bottleneck:

45 minutes to complete a single pipeline run
2 deployments per day maximum (developers waiting hours for feedback)
High failure rate due to flaky tests
Resource contention on build servers (queues backing up)

The business impact? Slow feature delivery, frustrated developers, and deployment bottlenecks.

Requirements:

Reduce pipeline time to under 15 minutes (70% reduction!)
Enable multiple deployments per day
Improve reliability (reduce flaky test failures)
Maintain security and quality gates

In this article, I'll walk through optimizing a CI/CD pipeline on AWS, covering parallelization, test optimization, caching strategies, infrastructure improvements, and monitoring.

Current State Analysis

Identifying Bottlenecks

Typical 45-Minute Pipeline Breakdown:

Source Checkout:        2 min
Dependency Install:     8 min  ← Bottleneck
Unit Tests:            12 min  ← Can parallelize
Integration Tests:      15 min  ← Can parallelize
E2E Tests:             20 min  ← Major bottleneck
Security Scan:          5 min  ← Can parallelize
Build Artifact:         3 min
Deploy:                 2 min
─────────────────────────────
Total:                 45 min

Key Issues:

Sequential execution (everything runs one after another)
No caching (dependencies downloaded every time)
Slow test execution (not optimized)
Resource constraints (single build server)

Solution Architecture

Optimized Pipeline Design

┌─────────────────────────────────────────────────────────┐
│                    Source Stage                          │
│              (CodeCommit/GitHub)                         │
└────────────────────┬────────────────────────────────────┘
                     │
        ┌────────────┴────────────┐
        │                         │
┌───────▼────────┐      ┌─────────▼──────────┐
│  Build Stage   │      │  Security Scan      │  ← Parallel
│  (Parallel)    │      │  (Parallel)         │
└───────┬────────┘      └─────────┬──────────┘
        │                          │
        └────────────┬─────────────┘
                     │
        ┌────────────┴────────────┐
        │                         │
┌───────▼────────┐      ┌─────────▼──────────┐
│  Unit Tests    │      │  Integration Tests │  ← Parallel
│  (Parallel)    │      │  (Parallel)        │
└───────┬────────┘      └─────────┬──────────┘
        │                          │
        └────────────┬─────────────┘
                     │
        ┌────────────┴────────────┐
        │                         │
┌───────▼────────┐      ┌─────────▼──────────┐
│  E2E Tests     │      │  Build Artifact    │  ← Parallel
│  (Optimized)   │      │  (Cached)          │
└───────┬────────┘      └─────────┬──────────┘
        │                          │
        └────────────┬─────────────┘
                     │
              ┌──────▼──────┐
              │   Deploy    │
              └─────────────┘

Target: Under 15 Minutes

Phase 1: Parallelization Strategy

AWS CodePipeline with Parallel Actions

Pipeline Definition:

{
  "pipeline": {
    "name": "payment-app-optimized-pipeline",
    "stages": [
      {
        "name": "Source",
        "actions": [{
          "name": "SourceAction",
          "actionTypeId": {
            "category": "Source",
            "owner": "AWS",
            "provider": "CodeCommit",
            "version": "1"
          },
          "outputArtifacts": [{"name": "SourceOutput"}]
        }]
      },
      {
        "name": "BuildAndTest",
        "actions": [
          {
            "name": "Build",
            "actionTypeId": {
              "category": "Build",
              "owner": "AWS",
              "provider": "CodeBuild",
              "version": "1"
            },
            "inputArtifacts": [{"name": "SourceOutput"}],
            "outputArtifacts": [{"name": "BuildOutput"}],
            "configuration": {
              "ProjectName": "payment-app-build"
            }
          },
          {
            "name": "SecurityScan",
            "runOrder": 1,
            "actionTypeId": {
              "category": "Build",
              "owner": "AWS",
              "provider": "CodeBuild",
              "version": "1"
            },
            "inputArtifacts": [{"name": "SourceOutput"}],
            "outputArtifacts": [{"name": "SecurityScanOutput"}],
            "configuration": {
              "ProjectName": "payment-app-security-scan"
            }
          }
        ]
      },
      {
        "name": "Test",
        "actions": [
          {
            "name": "UnitTests",
            "runOrder": 1,
            "actionTypeId": {
              "category": "Test",
              "owner": "AWS",
              "provider": "CodeBuild",
              "version": "1"
            },
            "inputArtifacts": [{"name": "BuildOutput"}],
            "outputArtifacts": [{"name": "UnitTestOutput"}],
            "configuration": {
              "ProjectName": "payment-app-unit-tests"
            }
          },
          {
            "name": "IntegrationTests",
            "runOrder": 1,
            "actionTypeId": {
              "category": "Test",
              "owner": "AWS",
              "provider": "CodeBuild",
              "version": "1"
            },
            "inputArtifacts": [{"name": "BuildOutput"}],
            "outputArtifacts": [{"name": "IntegrationTestOutput"}],
            "configuration": {
              "ProjectName": "payment-app-integration-tests"
            }
          }
        ]
      },
      {
        "name": "E2ETest",
        "actions": [{
          "name": "E2ETests",
          "actionTypeId": {
            "category": "Test",
            "owner": "AWS",
            "provider": "CodeBuild",
            "version": "1"
          },
          "inputArtifacts": [{"name": "BuildOutput"}],
          "configuration": {
            "ProjectName": "payment-app-e2e-tests"
          }
        }]
      },
      {
        "name": "Deploy",
        "actions": [{
          "name": "DeployToStaging",
          "actionTypeId": {
            "category": "Deploy",
            "owner": "AWS",
            "provider": "ECS",
            "version": "1"
          },
          "inputArtifacts": [{"name": "BuildOutput"}],
          "configuration": {
            "ClusterName": "payment-staging",
            "ServiceName": "payment-service"
          }
        }]
      }
    ]
  }
}

Parallel Test Execution

Split Tests by Category:

# test-runner.py
import subprocess
import sys
import os

def run_tests_in_parallel():
    """Run tests in parallel based on category"""

    test_categories = {
        'unit': 'tests/unit',
        'integration': 'tests/integration',
        'e2e': 'tests/e2e'
    }

    processes = []

    for category, test_path in test_categories.items():
        # Run each category in parallel
        cmd = [
            'pytest',
            test_path,
            f'--junitxml=test-results-{category}.xml',
            '--maxfail=5',  # Fail fast
            '-n', 'auto'  # Parallel execution within category
        ]

        process = subprocess.Popen(
            cmd,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE
        )
        processes.append((category, process))

    # Wait for all to complete
    results = {}
    for category, process in processes:
        stdout, stderr = process.communicate()
        results[category] = {
            'returncode': process.returncode,
            'stdout': stdout.decode(),
            'stderr': stderr.decode()
        }

    # Check results
    failed = [cat for cat, res in results.items() if res['returncode'] != 0]

    if failed:
        print(f"❌ Tests failed in: {', '.join(failed)}")
        sys.exit(1)

    print("✅ All tests passed")
    return 0

if __name__ == '__main__':
    sys.exit(run_tests_in_parallel())

CodeBuild Buildspec for Parallel Tests:

# buildspec-tests.yml
version: 0.2
phases:
  install:
    runtime-versions:
      python: 3.9
      nodejs: 18
    commands:
      - echo Installing dependencies...
      - pip install -r requirements.txt
      - npm ci
  pre_build:
    commands:
      - echo Setting up test environment...
      - |
        # Start test dependencies in background
        docker-compose up -d postgres redis
        sleep 10  # Wait for services to be ready
  build:
    commands:
      - echo Running tests in parallel...
      - |
        # Run unit and integration tests in parallel
        python test-runner.py &
        INTEGRATION_PID=$!

        # Run unit tests separately for faster feedback
        pytest tests/unit --junitxml=unit-results.xml -n auto &
        UNIT_PID=$!

        # Wait for both
        wait $UNIT_PID
        UNIT_EXIT=$?

        wait $INTEGRATION_PID
        INTEGRATION_EXIT=$?

        if [ $UNIT_EXIT -ne 0 ] || [ $INTEGRATION_EXIT -ne 0 ]; then
          echo "Tests failed"
          exit 1
        fi
  post_build:
    commands:
      - echo Uploading test results...
      - |
        aws s3 cp unit-results.xml s3://test-results/payment-app/unit-${CODEBUILD_BUILD_ID}.xml
        aws s3 cp integration-results.xml s3://test-results/payment-app/integration-${CODEBUILD_BUILD_ID}.xml
artifacts:
  files:
    - '**/*'
  reports:
    unit-tests:
      files:
        - 'unit-results.xml'
    integration-tests:
      files:
        - 'integration-results.xml'

Phase 2: Test Optimization

Test Categorization Strategy

Fast Tests (Run First):

# tests/unit/test_fast.py
import pytest

@pytest.mark.fast
def test_calculate_total():
    """Fast unit test - runs in < 100ms"""
    assert calculate_total([1, 2, 3]) == 6

@pytest.mark.fast
def test_validate_email():
    """Fast validation test"""
    assert validate_email("test@example.com") == True

Slow Tests (Run After Fast Pass):

# tests/integration/test_slow.py
import pytest

@pytest.mark.slow
@pytest.mark.integration
def test_database_transaction():
    """Slow integration test - requires database"""
    # Test implementation
    pass

@pytest.mark.slow
@pytest.mark.e2e
def test_full_payment_flow():
    """Slow E2E test - requires full stack"""
    # Test implementation
    pass

Pytest Configuration:

# pytest.ini
[pytest]
markers =
    fast: Fast tests (< 100ms)
    slow: Slow tests (> 1s)
    unit: Unit tests
    integration: Integration tests
    e2e: End-to-end tests

# Run fast tests first
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*

# Parallel execution
addopts = 
    -n auto
    -m "not slow"  # Skip slow tests in initial run
    --maxfail=5
    --tb=short

Test Execution Strategy:

#!/bin/bash
# run-tests-optimized.sh

echo "Phase 1: Running fast tests..."
pytest -m "fast" --maxfail=5 -n auto

if [ $? -ne 0 ]; then
    echo "❌ Fast tests failed, stopping pipeline"
    exit 1
fi

echo "Phase 2: Running slow tests..."
pytest -m "slow" --maxfail=3 -n 2  # Fewer parallel for slow tests

if [ $? -ne 0 ]; then
    echo "❌ Slow tests failed"
    exit 1
fi

echo "✅ All tests passed"

Test Flakiness Reduction

Retry Strategy for Flaky Tests:

# conftest.py
import pytest
from pytest_retry import retry

@pytest.fixture(autouse=True)
def setup_test_environment():
    """Setup test environment with retries"""
    yield
    # Cleanup

# Retry decorator for known flaky tests
@pytest.mark.flaky(reruns=3, reruns_delay=2)
def test_external_api_call():
    """Test that sometimes fails due to network issues"""
    # Test implementation
    pass

Test Isolation:

# tests/conftest.py
import pytest
import asyncio

@pytest.fixture(scope="function")
def isolated_database():
    """Create isolated database for each test"""
    # Create temporary database
    db = create_test_database()
    yield db
    # Cleanup
    db.drop()

@pytest.fixture(scope="function")
def clean_cache():
    """Clear cache before each test"""
    cache.clear()
    yield
    cache.clear()

Test Data Management:

# tests/fixtures.py
import pytest
from faker import Faker

fake = Faker()

@pytest.fixture
def sample_customer():
    """Generate sample customer data"""
    return {
        'id': fake.uuid4(),
        'name': fake.name(),
        'email': fake.email(),
        'created_at': fake.date_time()
    }

@pytest.fixture
def sample_transaction(sample_customer):
    """Generate sample transaction"""
    return {
        'id': fake.uuid4(),
        'customer_id': sample_customer['id'],
        'amount': fake.pydecimal(left_digits=3, right_digits=2, positive=True),
        'status': 'pending'
    }

E2E Test Optimization

Selective E2E Testing:

# Only run E2E tests for critical paths
E2E_TEST_PATHS = [
    'tests/e2e/test_payment_flow.py::test_successful_payment',
    'tests/e2e/test_payment_flow.py::test_payment_failure',
    'tests/e2e/test_user_registration.py::test_new_user_signup'
]

# Run subset of E2E tests
pytest ${E2E_TEST_PATHS[@]} --maxfail=1

E2E Test Parallelization with Test Containers:

# docker-compose.test.yml
version: '3.8'
services:
  test-runner-1:
    build: .
    command: pytest tests/e2e/test_payment_flow.py
    environment:
      - TEST_ENV=staging-1

  test-runner-2:
    build: .
    command: pytest tests/e2e/test_user_flow.py
    environment:
      - TEST_ENV=staging-2

  test-runner-3:
    build: .
    command: pytest tests/e2e/test_admin_flow.py
    environment:
      - TEST_ENV=staging-3

Phase 3: Caching and Dependency Management

CodeBuild Caching

Enable Local Caching:

{
  "cache": {
    "type": "LOCAL",
    "modes": [
      "LOCAL_DOCKER_LAYER_CACHE",
      "LOCAL_SOURCE_CACHE",
      "LOCAL_CUSTOM_CACHE"
    ]
  }
}

Dependency Caching:

# buildspec-with-cache.yml
version: 0.2
cache:
  paths:
    - 'node_modules/**/*'
    - '.venv/**/*'
    - '.m2/**/*'
phases:
  install:
    commands:
      - echo Restoring cache...
      - |
        # Python dependencies
        if [ -d ".venv" ]; then
          echo "Using cached Python virtual environment"
        else
          python -m venv .venv
          source .venv/bin/activate
          pip install -r requirements.txt
        fi
      - |
        # Node.js dependencies
        if [ -d "node_modules" ]; then
          echo "Using cached node_modules"
        else
          npm ci --prefer-offline --no-audit
        fi
      - |
        # Maven dependencies (Java)
        if [ -d ".m2" ]; then
          echo "Using cached Maven repository"
        else
          mvn dependency:go-offline
        fi
  build:
    commands:
      - echo Building application...
      - npm run build
      - mvn package -DskipTests

Docker Layer Caching

Optimize Dockerfile for Caching:

# Stage 1: Dependencies (cached if package files don't change)
FROM node:18-slim AS deps
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production && \
    npm cache clean --force

# Stage 2: Build (cached if source doesn't change)
FROM node:18-slim AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build

# Stage 3: Runtime (minimal)
FROM node:18-slim
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=deps /app/node_modules ./node_modules
CMD ["node", "dist/index.js"]

CodeBuild Docker Caching:

# buildspec-docker-cache.yml
version: 0.2
phases:
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
  build:
    commands:
      - echo Building Docker image with cache...
      - |
        # Pull previous image for layer caching
        docker pull $IMAGE_REPO_NAME:latest || true
      - |
        # Build with cache
        docker build \
          --cache-from $IMAGE_REPO_NAME:latest \
          -t $IMAGE_REPO_NAME:$IMAGE_TAG \
          -t $IMAGE_REPO_NAME:latest \
          .
  post_build:
    commands:
      - echo Pushing image...
      - docker push $IMAGE_REPO_NAME:$IMAGE_TAG
      - docker push $IMAGE_REPO_NAME:latest

S3 Caching for Large Artifacts

import boto3
import hashlib
import os

s3 = boto3.client('s3')
CACHE_BUCKET = 'payment-app-build-cache'

def get_cache_key(file_path):
    """Generate cache key from file hash"""
    with open(file_path, 'rb') as f:
        file_hash = hashlib.md5(f.read()).hexdigest()
    return f"cache/{file_path}/{file_hash}"

def restore_from_cache(file_path, cache_key):
    """Restore file from S3 cache"""
    try:
        s3.download_file(CACHE_BUCKET, cache_key, file_path)
        print(f"✅ Restored {file_path} from cache")
        return True
    except s3.exceptions.NoSuchKey:
        print(f"Cache miss for {file_path}")
        return False

def save_to_cache(file_path, cache_key):
    """Save file to S3 cache"""
    try:
        s3.upload_file(file_path, CACHE_BUCKET, cache_key)
        print(f"✅ Saved {file_path} to cache")
    except Exception as e:
        print(f"Failed to save cache: {e}")

# Usage in build script
if restore_from_cache('node_modules.tar.gz', get_cache_key('package.json')):
    tar -xzf node_modules.tar.gz
else:
    npm ci
    tar -czf node_modules.tar.gz node_modules
    save_to_cache('node_modules.tar.gz', get_cache_key('package.json'))

Phase 4: Infrastructure Improvements

CodeBuild Compute Optimization

Use Larger Instances for Faster Builds:

# Update CodeBuild project to use larger instances
aws codebuild update-project \
  --name payment-app-build \
  --compute-type BUILD_GENERAL1_LARGE  # 8 vCPU, 15 GB RAM

# For very large builds
aws codebuild update-project \
  --name payment-app-build \
  --compute-type BUILD_GENERAL1_2XLARGE  # 16 vCPU, 32 GB RAM

Auto-Scaling Build Fleet:

import boto3

codebuild = boto3.client('codebuild')

def create_fleet_for_parallel_builds():
    """Create build fleet for parallel execution"""

    # Create fleet with multiple instances
    codebuild.create_fleet(
        name='payment-app-fleet',
        baseCapacity=2,
        environmentType='LINUX_CONTAINER',
        computeType='BUILD_GENERAL1_LARGE',
        image='aws/codebuild/standard:5.0',
        fleetServiceRole='arn:aws:iam::account:role/CodeBuildFleetRole'
    )

# Use fleet in project
codebuild.update_project(
    name='payment-app-build',
    fleet={
        'fleetArn': 'arn:aws:codebuild:region:account:fleet/payment-app-fleet'
    }
)

ECS Build Agents (Alternative to CodeBuild)

ECS Task for Builds:

{
  "family": "build-agent",
  "containerDefinitions": [
    {
      "name": "build-container",
      "image": "aws/codebuild/standard:5.0",
      "memory": 8192,
      "cpu": 4096,
      "environment": [
        {"name": "AWS_REGION", "value": "us-east-1"}
      ]
    }
  ],
  "requiresCompatibilities": ["FARGATE"],
  "networkMode": "awsvpc"
}

Auto-Scale Build Agents:

import boto3

ecs = boto3.client('ecs')
application_autoscaling = boto3.client('application-autoscaling')

def setup_build_agent_autoscaling():
    """Setup auto-scaling for ECS build agents"""

    # Register scalable target
    application_autoscaling.register_scalable_target(
        ServiceNamespace='ecs',
        ResourceId='service/build-cluster/build-agent-service',
        ScalableDimension='ecs:service:DesiredCount',
        MinCapacity=1,
        MaxCapacity=10
    )

    # Create scaling policy based on queue depth
    application_autoscaling.put_scaling_policy(
        ServiceNamespace='ecs',
        ResourceId='service/build-cluster/build-agent-service',
        ScalableDimension='ecs:service:DesiredCount',
        PolicyName='scale-on-queue-depth',
        PolicyType='TargetTrackingScaling',
        TargetTrackingScalingPolicyConfiguration={
            'CustomizedMetricSpecification': {
                'MetricName': 'ApproximateNumberOfMessagesVisible',
                Namespace='AWS/SQS',
                Statistic='Average',
                Unit='Count'
            },
            'TargetValue': 5.0,  # Scale when queue has 5+ messages
            'ScaleInCooldown': 300,
            'ScaleOutCooldown': 60
        }
    )

Network Optimization

VPC Endpoints for Faster S3 Access:

# Create VPC endpoint for S3 (faster than internet)
aws ec2 create-vpc-endpoint \
  --vpc-id vpc-12345678 \
  --service-name com.amazonaws.us-east-1.s3 \
  --route-table-ids rtb-12345678

# Create VPC endpoint for ECR
aws ec2 create-vpc-endpoint \
  --vpc-id vpc-12345678 \
  --service-name com.amazonaws.us-east-1.ecr.dkr \
  --vpc-endpoint-type Interface \
  --subnet-ids subnet-123 subnet-456 \
  --security-group-ids sg-12345678

Phase 5: Monitoring and Metrics

Pipeline Metrics Dashboard

import boto3
from datetime import datetime, timedelta

codebuild = boto3.client('codebuild')
cloudwatch = boto3.client('cloudwatch')

def get_pipeline_metrics():
    """Collect pipeline performance metrics"""

    # Get recent builds
    builds = codebuild.batch_get_builds(
        ids=get_recent_build_ids()
    )

    metrics = {
        'total_builds': len(builds['builds']),
        'successful_builds': len([b for b in builds['builds'] if b['buildStatus'] == 'SUCCEEDED']),
        'failed_builds': len([b for b in builds['builds'] if b['buildStatus'] == 'FAILED']),
        'avg_duration': calculate_avg_duration(builds['builds']),
        'p95_duration': calculate_p95_duration(builds['builds']),
        'p99_duration': calculate_p99_duration(builds['builds'])
    }

    return metrics

def calculate_avg_duration(builds):
    """Calculate average build duration"""
    durations = []
    for build in builds:
        if build.get('endTime') and build.get('startTime'):
            duration = (build['endTime'] - build['startTime']).total_seconds()
            durations.append(duration)

    return sum(durations) / len(durations) if durations else 0

def publish_metrics_to_cloudwatch(metrics):
    """Publish metrics to CloudWatch"""

    cloudwatch.put_metric_data(
        Namespace='PaymentApp/CI-CD',
        MetricData=[
            {
                'MetricName': 'PipelineDuration',
                'Value': metrics['avg_duration'],
                'Unit': 'Seconds',
                'Timestamp': datetime.utcnow()
            },
            {
                'MetricName': 'PipelineSuccessRate',
                'Value': (metrics['successful_builds'] / metrics['total_builds']) * 100,
                'Unit': 'Percent',
                'Timestamp': datetime.utcnow()
            }
        ]
    )

CloudWatch Dashboard

def create_pipeline_dashboard():
    """Create CloudWatch dashboard for pipeline metrics"""

    dashboard = {
        "widgets": [
            {
                "type": "metric",
                "properties": {
                    "metrics": [
                        ["PaymentApp/CI-CD", "PipelineDuration", {"stat": "Average"}],
                        [".", "PipelineDuration", {"stat": "p95"}],
                        [".", "PipelineDuration", {"stat": "p99"}]
                    ],
                    "period": 300,
                    "stat": "Average",
                    "region": "us-east-1",
                    "title": "Pipeline Duration"
                }
            },
            {
                "type": "metric",
                "properties": {
                    "metrics": [
                        ["PaymentApp/CI-CD", "PipelineSuccessRate", {"stat": "Average"}],
                        ["AWS/CodeBuild", "Builds", {"stat": "Sum", "dimensions": [{"Name": "ProjectName", "Value": "payment-app-build"}]}]
                    ],
                    "period": 300,
                    "stat": "Average",
                    "region": "us-east-1",
                    "title": "Pipeline Success Rate"
                }
            },
            {
                "type": "log",
                "properties": {
                    "query": "SOURCE '/aws/codebuild/payment-app-build' | fields @timestamp, @message\n| filter @message like /ERROR/\n| sort @timestamp desc\n| limit 100",
                    "region": "us-east-1",
                    "title": "Build Errors"
                }
            }
        ]
    }

    cloudwatch.put_dashboard(
        DashboardName='CI-CD-Pipeline',
        DashboardBody=json.dumps(dashboard)
    )

Build Time Breakdown Analysis

def analyze_build_time_breakdown(build_id):
    """Analyze time spent in each phase"""

    # Get build logs
    logs = codebuild.batch_get_builds(ids=[build_id])
    build = logs['builds'][0]

    # Parse CloudWatch Logs for timing
    log_group = f"/aws/codebuild/{build['projectName']}"

    # Extract phase timings from logs
    phases = {
        'install': 0,
        'pre_build': 0,
        'build': 0,
        'post_build': 0
    }

    # Query CloudWatch Logs Insights
    logs_client = boto3.client('logs')

    response = logs_client.start_query(
        logGroupName=log_group,
        startTime=int((build['startTime'] - timedelta(minutes=1)).timestamp()),
        endTime=int((build['endTime'] + timedelta(minutes=1)).timestamp()),
        queryString="""
        fields @timestamp, @message
        | filter @message like /PHASE/
        | stats count() by @message
        """
    )

    # Process results to calculate phase durations
    # (Implementation depends on log format)

    return phases

Alerts for Pipeline Issues

def create_pipeline_alerts():
    """Create CloudWatch alarms for pipeline issues"""

    alarms = [
        {
            'AlarmName': 'pipeline-duration-too-long',
            'MetricName': 'PipelineDuration',
            'Namespace': 'PaymentApp/CI-CD',
            'Statistic': 'Average',
            'Period': 300,
            'EvaluationPeriods': 2,
            'Threshold': 900,  # 15 minutes
            'ComparisonOperator': 'GreaterThanThreshold'
        },
        {
            'AlarmName': 'pipeline-failure-rate-high',
            'MetricName': 'PipelineSuccessRate',
            'Namespace': 'PaymentApp/CI-CD',
            'Statistic': 'Average',
            'Period': 3600,
            'EvaluationPeriods': 1,
            'Threshold': 90,  # Less than 90% success rate
            'ComparisonOperator': 'LessThanThreshold'
        }
    ]

    for alarm in alarms:
        cloudwatch.put_metric_alarm(**alarm)
        print(f"Created alarm: {alarm['AlarmName']}")

Optimized Pipeline Timeline

Before Optimization

Source:        2 min
Dependencies:  8 min
Build:          5 min
Unit Tests:    12 min
Integration:   15 min
E2E:           20 min
Security:       5 min
Deploy:         2 min
─────────────────────
Total:         45 min

After Optimization

Source:        2 min
Build + Security (parallel):  5 min  ← Was 13 min
Unit + Integration (parallel): 8 min  ← Was 27 min
E2E (optimized):              6 min  ← Was 20 min
Deploy:                       2 min
─────────────────────────────────────
Total:                        13 min  ← Was 45 min (71% reduction!)

Implementation Checklist

Week 1: Setup and Configuration

Create optimized CodePipeline
Configure CodeBuild projects with caching
Set up parallel test execution
Enable CloudWatch monitoring

Week 2: Test Optimization

Categorize tests (fast/slow)
Implement test retry strategy
Optimize E2E tests
Reduce flaky tests

Week 3: Caching Implementation

Enable CodeBuild local caching
Implement Docker layer caching
Set up S3 caching for large artifacts
Verify cache hit rates

Week 4: Infrastructure Improvements

Upgrade to larger CodeBuild instances
Set up VPC endpoints
Configure auto-scaling build agents
Optimize network configuration

Week 5: Monitoring and Validation

Create CloudWatch dashboards
Set up alerts
Validate pipeline performance
Document improvements

Success Metrics

Target Metrics

target_metrics = {
    'pipeline_duration_minutes': 15,  # Target: < 15 minutes
    'success_rate_percent': 95,  # Target: > 95%
    'cache_hit_rate_percent': 80,  # Target: > 80%
    'parallel_execution_percent': 60,  # Target: 60% of pipeline parallel
    'deployments_per_day': 5,  # Target: 5+ deployments per day
    'developer_feedback_time_minutes': 15  # Target: < 15 minutes
}

Measuring Success

def measure_pipeline_improvement():
    """Compare before and after metrics"""

    before = {
        'avg_duration': 45,  # minutes
        'success_rate': 85,  # percent
        'deployments_per_day': 2
    }

    after = {
        'avg_duration': 13,  # minutes
        'success_rate': 96,  # percent
        'deployments_per_day': 5
    }

    improvements = {
        'duration_reduction': ((before['avg_duration'] - after['avg_duration']) / before['avg_duration']) * 100,
        'success_rate_improvement': after['success_rate'] - before['success_rate'],
        'deployment_frequency_increase': ((after['deployments_per_day'] - before['deployments_per_day']) / before['deployments_per_day']) * 100
    }

    print(f"Duration reduction: {improvements['duration_reduction']:.1f}%")
    print(f"Success rate improvement: {improvements['success_rate_improvement']:.1f}%")
    print(f"Deployment frequency increase: {improvements['deployment_frequency_increase']:.1f}%")

    return improvements

Best Practices Summary

Do's

Parallelize everything possible (tests, builds, scans)
Cache aggressively (dependencies, Docker layers, artifacts)
Run fast tests first (fail fast on critical issues)
Optimize test execution (parallel, selective, isolated)
Use larger build instances for faster execution
Monitor pipeline metrics continuously
Reduce flaky tests (retry, isolate, fix root causes)
Optimize Docker builds (layer caching, multi-stage)

Don'ts

Don't run tests sequentially when they can run in parallel
Don't skip caching (huge time savings)
Don't run all E2E tests on every commit
Don't ignore flaky tests (fix or remove them)
Don't use small build instances for large builds
Don't deploy without metrics (measure everything)
Don't ignore build time breakdown (identify bottlenecks)

Conclusion

Optimizing a CI/CD pipeline from 45 minutes to under 15 minutes requires a systematic approach:

Parallelization reduces total time by running tasks concurrently
Test optimization focuses on fast feedback and reliability
Caching eliminates redundant work (dependencies, Docker layers)
Infrastructure improvements provide more compute power
Monitoring ensures continuous optimization

The result? A fast, reliable pipeline that enables multiple deployments per day, improves developer experience, and maintains security and quality gates.

Additional Resources

Securing Container Lifecycle on AWS: From Build to Runtime

Maureen Chebet — Wed, 26 Nov 2025 18:17:16 +0000

The Security Wake-Up Call

Your containerized applications are under scrutiny. Security scans have revealed critical issues:

Base images with known vulnerabilities (CVE-2023-XXXX, CVE-2024-XXXX)
Runtime secrets exposed in image layers (API keys, passwords visible in image history)
No image signing or verification (anyone can push malicious images)
Compliance gaps for financial services (PCI-DSS, SOC 2 requirements not met)

This isn't just about fixing a few Dockerfiles. You need to secure the entire container lifecycle: build, registry, deployment, and runtime.

In this article, I'll walk through a comprehensive container security strategy on AWS that addresses vulnerabilities, implements secrets management, enables image signing, enforces runtime security, and automates compliance.

Container Security Lifecycle

The Four Pillars

┌─────────────┐    ┌──────────────┐    ┌─────────────┐    ┌──────────────┐
│   BUILD     │ →  │   REGISTRY   │ →  │  DEPLOYMENT │ →  │   RUNTIME    │
│             │    │              │    │             │    │              │
│ Secure base │    │ Image scan   │    │ Policy      │    │ Least        │
│ Multi-stage │    │ Signing      │    │ validation  │    │ privilege    │
│ No secrets  │    │ Access       │    │ RBAC        │    │ Isolation    │
└─────────────┘    └──────────────┘    └─────────────┘    └──────────────┘

Phase 1: Secure Base Images and Minimal Attack Surface

The Problem: Vulnerable Base Images

Many Dockerfiles start with:

FROM ubuntu:latest
# or
FROM node:latest

These base images often contain:

Unnecessary packages and tools
Known vulnerabilities
Large attack surface
Outdated packages

Solution: Minimal, Trusted Base Images

Option 1: Use Distroless Images

# Before: Vulnerable base image
FROM node:18
COPY . .
RUN npm install
CMD ["node", "app.js"]

# After: Distroless base image
FROM node:18-slim AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

FROM gcr.io/distroless/nodejs18-debian11
WORKDIR /app
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app .
USER nonroot:nonroot
CMD ["app.js"]

Benefits:

No shell, no package manager (reduces attack surface)
Minimal OS footprint
Only runtime dependencies

Option 2: Alpine Linux

FROM alpine:3.18 AS builder
RUN apk add --no-cache nodejs npm
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

FROM alpine:3.18
RUN apk add --no-cache nodejs
WORKDIR /app
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app .
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001
USER nodejs:nodejs
CMD ["node", "app.js"]

Option 3: AWS-Optimized Base Images

# Use AWS-provided base images
FROM public.ecr.aws/lambda/nodejs:18
# or
FROM public.ecr.aws/docker/library/node:18-slim

Multi-Stage Build Best Practices

Secure Multi-Stage Build:

# Stage 1: Build dependencies
FROM node:18-slim AS deps
WORKDIR /app

# Copy only package files first (better layer caching)
COPY package*.json ./

# Install dependencies with security flags
RUN npm ci --only=production --ignore-scripts && \
    npm audit --audit-level=moderate && \
    rm -rf /tmp/* /var/tmp/*

# Stage 2: Build application
FROM node:18-slim AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .

# Build with security hardening
RUN npm run build && \
    npm prune --production && \
    rm -rf /tmp/* /var/tmp/*

# Stage 3: Runtime (minimal)
FROM gcr.io/distroless/nodejs18-debian11:nonroot
WORKDIR /app

# Copy only production artifacts
COPY --from=builder --chown=nonroot:nonroot /app/dist ./dist
COPY --from=builder --chown=nonroot:nonroot /app/node_modules ./node_modules
COPY --from=builder --chown=nonroot:nonroot /app/package.json ./

# No secrets, no build tools, no shell
EXPOSE 8080
CMD ["dist/index.js"]

Base Image Vulnerability Scanning

Pre-Build Base Image Check:

#!/bin/bash
# check-base-image.sh

BASE_IMAGE=$1
if [ -z "$BASE_IMAGE" ]; then
    echo "Usage: $0 <base-image>"
    exit 1
fi

echo "Scanning base image: $BASE_IMAGE"

# Use Trivy to scan base image
trivy image --severity HIGH,CRITICAL --exit-code 1 "$BASE_IMAGE"

if [ $? -eq 0 ]; then
    echo "Base image passed security scan"
    exit 0
else
    echo "Base image has critical vulnerabilities"
    exit 1
fi

Integrate into CI/CD:

# .github/workflows/docker-build.yml
name: Build and Scan

on:
  push:
    branches: [main]

jobs:
  check-base-image:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Check base image
        run: |
          BASE_IMAGE=$(grep '^FROM' Dockerfile | head -1 | awk '{print $2}')
          ./check-base-image.sh "$BASE_IMAGE"

  build:
    needs: check-base-image
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Build Docker image
        run: docker build -t payment-app:latest .

Automated Base Image Updates

Dependabot for Docker:

# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: "docker"
    directory: "/"
    schedule:
      interval: "weekly"
    open-pull-requests-limit: 10

Phase 2: Secrets Management (Not in Images)

The Problem: Secrets in Image Layers

Common Anti-Pattern:

# NEVER DO THIS
FROM node:18
ENV DB_PASSWORD=mysecretpassword123
ENV API_KEY=sk_live_1234567890
COPY . .
RUN npm install
CMD ["node", "app.js"]

Why This is Dangerous:

Secrets are visible in image history: docker history <image>
Secrets are in image layers (even if removed in later layers)
Anyone with image access can extract secrets
Secrets are in version control (if Dockerfile is committed)

Solution: AWS Secrets Manager Integration

Secure Dockerfile:

FROM node:18-slim AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

FROM gcr.io/distroless/nodejs18-debian11:nonroot
WORKDIR /app
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app .

# No secrets in image!
# Secrets will be injected at runtime via environment variables
# or mounted volumes from AWS Secrets Manager

EXPOSE 8080
CMD ["app.js"]

Application Code - Fetching Secrets:

// app.js
const AWS = require('aws-sdk');
const secretsManager = new AWS.SecretsManager({ region: process.env.AWS_REGION });

async function getSecret(secretName) {
    try {
        const data = await secretsManager.getSecretValue({ SecretId: secretName }).promise();
        return JSON.parse(data.SecretString);
    } catch (error) {
        console.error(`Error retrieving secret ${secretName}:`, error);
        throw error;
    }
}

// Fetch secrets at startup
let dbCredentials;
async function initializeSecrets() {
    dbCredentials = await getSecret('payment-app/database/credentials');
    process.env.DB_HOST = dbCredentials.host;
    process.env.DB_USER = dbCredentials.username;
    process.env.DB_PASSWORD = dbCredentials.password;
}

initializeSecrets().then(() => {
    // Start application
    app.listen(8080);
});

ECS Task Definition with Secrets:

{
  "family": "payment-app",
  "containerDefinitions": [
    {
      "name": "payment-app",
      "image": "123456789.dkr.ecr.us-east-1.amazonaws.com/payment-app:latest",
      "secrets": [
        {
          "name": "DB_PASSWORD",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789:secret:payment-app/database/credentials:password::"
        },
        {
          "name": "API_KEY",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789:secret:payment-app/api-key::"
        }
      ],
      "environment": [
        {
          "name": "AWS_REGION",
          "value": "us-east-1"
        }
      ]
    }
  ],
  "taskRoleArn": "arn:aws:iam::123456789:role/ecs-task-role",
  "executionRoleArn": "arn:aws:iam::123456789:role/ecs-execution-role"
}

IAM Role for Secrets Access:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": [
        "arn:aws:secretsmanager:us-east-1:123456789:secret:payment-app/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt"
      ],
      "Resource": [
        "arn:aws:kms:us-east-1:123456789:key/secrets-key-id"
      ],
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "secretsmanager.us-east-1.amazonaws.com"
        }
      }
    }
  ]
}

Kubernetes Secrets (If Using EKS)

External Secrets Operator:

# external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payment-app-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: payment-app-secrets
    creationPolicy: Owner
  data:
    - secretKey: db-password
      remoteRef:
        key: payment-app/database/credentials
        property: password
    - secretKey: api-key
      remoteRef:
        key: payment-app/api-key

Pre-Commit Hook to Detect Secrets

#!/usr/bin/env python3
# .git/hooks/pre-commit

import subprocess
import re
import sys

def detect_secrets_in_dockerfile():
    """Detect secrets in Dockerfile before commit"""

    patterns = [
        (r'ENV\s+\w*PASSWORD\s*=\s*["\']([^"\']+)["\']', 'Password in ENV'),
        (r'ENV\s+\w*SECRET\s*=\s*["\']([^"\']+)["\']', 'Secret in ENV'),
        (r'ENV\s+\w*KEY\s*=\s*["\']([^"\']+)["\']', 'Key in ENV'),
        (r'--password\s+["\']([^"\']+)["\']', 'Password in command'),
        (r'apikey["\']?\s*[:=]\s*["\']([^"\']+)["\']', 'API key detected'),
    ]

    try:
        with open('Dockerfile', 'r') as f:
            content = f.read()

        violations = []
        for pattern, message in patterns:
            matches = re.findall(pattern, content, re.IGNORECASE)
            if matches:
                violations.append(f"{message}: {len(matches)} found")

        if violations:
            print("❌ SECURITY VIOLATION: Secrets detected in Dockerfile!")
            for violation in violations:
                print(f"  - {violation}")
            print("\nUse AWS Secrets Manager instead.")
            print("See: https://docs.aws.amazon.com/secretsmanager/")
            sys.exit(1)

        print("✅ No secrets detected in Dockerfile")
        return 0
    except FileNotFoundError:
        return 0

if __name__ == '__main__':
    sys.exit(detect_secrets_in_dockerfile())

Phase 3: Image Scanning and Signing

Amazon ECR Image Scanning

Enable Automatic Scanning:

# Enable automatic scanning on push
aws ecr put-image-scanning-configuration \
  --repository-name payment-app \
  --image-scanning-configuration scanOnPush=true

# Scan existing images
aws ecr start-image-scan \
  --repository-name payment-app \
  --image-id imageTag=latest

Get Scan Results:

# Get scan findings
aws ecr describe-image-scan-findings \
  --repository-name payment-app \
  --image-id imageTag=latest \
  --query 'imageScanFindings' \
  --output json

Fail Build on Critical Vulnerabilities:

# check-ecr-scan-results.py
import boto3
import sys
import json

ecr = boto3.client('ecr')

def check_scan_results(repo_name, image_tag):
    """Check ECR scan results and fail if critical issues found"""

    response = ecr.describe-image-scan-findings(
        repositoryName=repo_name,
        imageId={'imageTag': image_tag}
    )

    findings = response.get('imageScanFindings', {})
    finding_counts = findings.get('findingCounts', {})

    critical_count = finding_counts.get('CRITICAL', 0)
    high_count = finding_counts.get('HIGH', 0)

    print(f"Scan Results:")
    print(f"  Critical: {critical_count}")
    print(f"  High: {high_count}")
    print(f"  Medium: {finding_counts.get('MEDIUM', 0)}")
    print(f"  Low: {finding_counts.get('LOW', 0)}")

    # Fail if critical or too many high severity
    if critical_count > 0:
        print(f"❌ Build failed: {critical_count} CRITICAL vulnerabilities found")
        sys.exit(1)

    if high_count > 5:
        print(f"❌ Build failed: {high_count} HIGH vulnerabilities found (max 5 allowed)")
        sys.exit(1)

    print("✅ Image scan passed")
    return 0

if __name__ == '__main__':
    if len(sys.argv) < 3:
        print("Usage: python check-ecr-scan-results.py <repo-name> <image-tag>")
        sys.exit(1)

    sys.exit(check_scan_results(sys.argv[1], sys.argv[2]))

Integrate into CI/CD:

# buildspec.yml
version: 0.2
phases:
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
      - REPOSITORY_URI=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME
      - COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
      - IMAGE_TAG=${COMMIT_HASH:=latest}
  build:
    commands:
      - echo Building Docker image...
      - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $REPOSITORY_URI:$IMAGE_TAG
  post_build:
    commands:
      - echo Pushing Docker image...
      - docker push $REPOSITORY_URI:$IMAGE_TAG
      - echo Waiting for image scan to complete...
      - |
        aws ecr wait image-scan-complete \
          --repository-name $IMAGE_REPO_NAME \
          --image-id imageTag=$IMAGE_TAG \
          --max-attempts 30 \
          --delay 10
      - echo Checking scan results...
      - |
        python check-ecr-scan-results.py $IMAGE_REPO_NAME $IMAGE_TAG
      - echo Image scan passed, proceeding with deployment

Image Signing with AWS Signer

Create Signing Profile:

# Create signing profile
aws signer put-signing-profile \
  --profile-name payment-app-signing-profile \
  --platform-id Notation-OCI-SHA384-ECDSA \
  --signing-material certificateArn=arn:aws:acm:region:account:certificate/cert-id

# Get signing profile
aws signer get-signing-profile \
  --profile-name payment-app-signing-profile

Sign Image:

# Sign image after push
aws signer start-signing-job \
  --source '{
    "s3": {
      "bucketName": "payment-app-artifacts",
      "key": "payment-app-latest.tar"
    }
  }' \
  --destination '{
    "s3": {
      "bucketName": "payment-app-artifacts",
      "prefix": "signed/"
    }
  }' \
  --profile-name payment-app-signing-profile

Verify Image Signature:

# Install notation CLI
# https://notaryproject.dev/docs/installation/

# Verify signature
notation verify \
  --certificate-file certificate.pem \
  123456789.dkr.ecr.us-east-1.amazonaws.com/payment-app:latest

Enforce Signature Verification in ECS:

import boto3

ecs = boto3.client('ecs')

def verify_image_signature(image_uri):
    """Verify image signature before deployment"""

    # Extract image details
    # Check signature using notation or AWS Signer API

    # If signature invalid, reject deployment
    pass

def create_service_with_signature_check(task_definition):
    """Create ECS service only if image is signed"""

    # Verify signature first
    image_uri = task_definition['containerDefinitions'][0]['image']
    if not verify_image_signature(image_uri):
        raise ValueError(f"Image {image_uri} is not signed or signature invalid")

    # Create service
    ecs.create_service(
        cluster='payment-cluster',
        serviceName='payment-service',
        taskDefinition=task_definition['family'],
        desiredCount=2
    )

Third-Party Scanning Tools

Trivy Integration:

# Install Trivy
wget https://github.com/aquasecurity/trivy/releases/download/v0.45.0/trivy_0.45.0_Linux-64bit.tar.gz
tar -xzf trivy_0.45.0_Linux-64bit.tar.gz

# Scan image
trivy image --severity HIGH,CRITICAL \
  --format json \
  --output trivy-results.json \
  123456789.dkr.ecr.us-east-1.amazonaws.com/payment-app:latest

# Fail build on critical findings
trivy image --exit-code 1 --severity CRITICAL \
  123456789.dkr.ecr.us-east-1.amazonaws.com/payment-app:latest

Snyk Integration:

# Install Snyk
npm install -g snyk

# Authenticate
snyk auth $SNYK_TOKEN

# Scan Docker image
snyk container test 123456789.dkr.ecr.us-east-1.amazonaws.com/payment-app:latest \
  --severity-threshold=high \
  --json > snyk-results.json

Phase 4: Runtime Security (Least Privilege)

ECS Task Security Configuration

Non-Root User:

{
  "containerDefinitions": [
    {
      "name": "payment-app",
      "image": "123456789.dkr.ecr.us-east-1.amazonaws.com/payment-app:latest",
      "user": "1001:1001",
      "readonlyRootFilesystem": true,
      "privileged": false,
      "linuxParameters": {
        "capabilities": {
          "drop": ["ALL"],
          "add": ["NET_BIND_SERVICE"]
        }
      }
    }
  ]
}

Resource Limits:

{
  "containerDefinitions": [
    {
      "name": "payment-app",
      "memory": 512,
      "memoryReservation": 256,
      "cpu": 256,
      "ulimits": [
        {
          "name": "nofile",
          "softLimit": 1024,
          "hardLimit": 2048
        }
      ]
    }
  ]
}

EKS Pod Security Standards

Pod Security Policy:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: payment-app-psp
spec:
  privileged: false
  allowPrivilegeEscalation: false
  requiredDropCapabilities:
    - ALL
  volumes:
    - 'configMap'
    - 'secret'
    - 'emptyDir'
  runAsUser:
    rule: 'MustRunAsNonRoot'
  seLinux:
    rule: 'RunAsAny'
  fsGroup:
    rule: 'RunAsAny'
  readOnlyRootFilesystem: true

Security Context:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-app
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
        fsGroup: 1001
      containers:
      - name: payment-app
        image: 123456789.dkr.ecr.us-east-1.amazonaws.com/payment-app:latest
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop:
              - ALL
            add:
              - NET_BIND_SERVICE
        resources:
          limits:
            memory: "512Mi"
            cpu: "500m"
          requests:
            memory: "256Mi"
            cpu: "250m"

Network Policies (EKS)

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payment-app-netpol
spec:
  podSelector:
    matchLabels:
      app: payment-app
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: nginx-ingress
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: database
    ports:
    - protocol: TCP
      port: 5432
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    ports:
    - protocol: UDP
      port: 53

AWS App Mesh for Service Isolation

apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: payment-app
spec:
  podSelector:
    matchLabels:
      app: payment-app
  listeners:
  - portMapping:
      port: 8080
      protocol: http
  serviceDiscovery:
    dns:
      hostname: payment-app.payment.svc.cluster.local
  backends:
  - virtualService:
      virtualServiceName: database.payment.svc.cluster.local

Runtime Security Monitoring

Amazon Inspector for Runtime Scanning:

# Enable Inspector for ECS
aws inspector2 enable \
  --resource-types EC2 \
  --account-ids 123456789

# Create assessment target
aws inspector2 create-assessment-target \
  --name payment-app-ecs \
  --resource-group-arn arn:aws:resource-groups:us-east-1:123456789:group/payment-app

CloudWatch Container Insights:

# Enable Container Insights for ECS
aws ecs update-cluster \
  --cluster payment-cluster \
  --settings name=containerInsights,value=enabled

Phase 5: Compliance Automation

AWS Config Rules for Container Compliance

ECR Compliance Rule:

import boto3
import json

def evaluate_ecr_compliance(configuration_item):
    """Evaluate ECR repository compliance"""

    compliance_status = 'COMPLIANT'
    annotation = ''

    # Check if image scanning is enabled
    if not configuration_item.get('configuration', {}).get('imageScanningConfiguration', {}).get('scanOnPush', False):
        compliance_status = 'NON_COMPLIANT'
        annotation = 'ECR image scanning must be enabled for PCI-DSS compliance'

    # Check if encryption is enabled
    if not configuration_item.get('configuration', {}).get('encryptionConfiguration', {}).get('encryptionType') == 'AES256':
        compliance_status = 'NON_COMPLIANT'
        annotation = 'ECR encryption must be enabled'

    return {
        'compliance_type': compliance_status,
        'annotation': annotation
    }

def lambda_handler(event, context):
    """Lambda handler for Config custom rule"""

    config = boto3.client('config')
    configuration_item = json.loads(event['invokingEvent'])['configurationItem']
    evaluation = evaluate_ecr_compliance(configuration_item)

    config.put_evaluations(
        Evaluations=[{
            'ComplianceResourceType': configuration_item['resourceType'],
            'ComplianceResourceId': configuration_item['resourceId'],
            'ComplianceType': evaluation['compliance_type'],
            'Annotation': evaluation['annotation'],
            'OrderingTimestamp': configuration_item['configurationItemCaptureTime']
        }]
    )

    return evaluation

ECS Task Definition Compliance:

def evaluate_ecs_task_compliance(configuration_item):
    """Evaluate ECS task definition compliance"""

    compliance_status = 'COMPLIANT'
    violations = []

    config = configuration_item.get('configuration', {})
    containers = config.get('containerDefinitions', [])

    for container in containers:
        # Check if running as root
        if container.get('user') is None or container.get('user') == 'root':
            violations.append('Container must not run as root user')

        # Check if readonly root filesystem
        if not container.get('readonlyRootFilesystem', False):
            violations.append('Container must have readonly root filesystem')

        # Check if privileged
        if container.get('privileged', False):
            violations.append('Container must not run in privileged mode')

        # Check capabilities
        linux_params = container.get('linuxParameters', {})
        capabilities = linux_params.get('capabilities', {})
        if 'ALL' not in capabilities.get('drop', []):
            violations.append('Container must drop ALL capabilities')

    if violations:
        compliance_status = 'NON_COMPLIANT'
        annotation = '; '.join(violations)
    else:
        annotation = 'Task definition meets security requirements'

    return {
        'compliance_type': compliance_status,
        'annotation': annotation
    }

Automated Compliance Reporting

import boto3
from datetime import datetime

config = boto3.client('config')
s3 = boto3.client('s3')

def generate_compliance_report():
    """Generate container compliance report"""

    # Get compliance summary
    response = config.get_compliance_summary_by_config_rule(
        ConfigRuleNames=[
            'ecr-image-scanning-enabled',
            'ecs-task-non-root-user',
            'ecs-task-readonly-filesystem'
        ]
    )

    report = {
        'timestamp': datetime.utcnow().isoformat(),
        'compliance_summary': response.get('ComplianceSummariesByConfigRule', []),
        'overall_compliance': calculate_compliance_percentage(response)
    }

    # Save to S3 for audit trail
    s3.put_object(
        Bucket='compliance-reports',
        Key=f"container-compliance-{datetime.utcnow().strftime('%Y-%m-%d')}.json",
        Body=json.dumps(report, indent=2),
        ServerSideEncryption='AES256'
    )

    return report

def calculate_compliance_percentage(response):
    """Calculate overall compliance percentage"""

    total_resources = 0
    compliant_resources = 0

    for rule_summary in response.get('ComplianceSummariesByConfigRule', []):
        summary = rule_summary.get('ComplianceSummary', {})
        total_resources += summary.get('ComplianceResourceCount', {}).get('CappedCount', 0)
        compliant_resources += summary.get('CompliantResourceCount', {}).get('CappedCount', 0)

    if total_resources == 0:
        return 100.0

    return (compliant_resources / total_resources) * 100

Policy Enforcement with OPA (Open Policy Agent)

OPA Policy for Container Security:

# container-security.rego
package container.security

# Deny if container runs as root
deny[msg] {
    input.container.user == "root"
    msg := "Container must not run as root user"
}

# Deny if privileged mode enabled
deny[msg] {
    input.container.privileged == true
    msg := "Container must not run in privileged mode"
}

# Deny if readonly root filesystem not enabled
deny[msg] {
    input.container.readonlyRootFilesystem != true
    msg := "Container must have readonly root filesystem"
}

# Deny if ALL capabilities not dropped
deny[msg] {
    not "ALL" in input.container.linuxParameters.capabilities.drop
    msg := "Container must drop ALL capabilities"
}

Integrate OPA with ECS:

import requests
import json

def validate_task_definition_with_opa(task_definition):
    """Validate task definition against OPA policies"""

    opa_url = "http://opa-service:8181/v1/data/container/security"

    # Prepare input for OPA
    input_data = {
        "container": task_definition['containerDefinitions'][0]
    }

    response = requests.post(
        f"{opa_url}/deny",
        json={"input": input_data}
    )

    if response.status_code == 200:
        result = response.json()
        if result.get('result'):
            # Policy violations found
            violations = result['result']
            raise ValueError(f"Policy violations: {violations}")

    return True

Complete CI/CD Pipeline with Security

Secure Build Pipeline

# pipeline.yaml
version: 0.2
phases:
  pre_build:
    commands:
      - echo Checking base image...
      - BASE_IMAGE=$(grep '^FROM' Dockerfile | head -1 | awk '{print $2}')
      - trivy image --severity HIGH,CRITICAL --exit-code 1 "$BASE_IMAGE"
      - echo Checking for secrets in Dockerfile...
      - python check-dockerfile-secrets.py
  build:
    commands:
      - echo Building Docker image...
      - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $REPOSITORY_URI:$IMAGE_TAG
  post_build:
    commands:
      - echo Scanning image with Trivy...
      - trivy image --severity HIGH,CRITICAL --exit-code 1 $REPOSITORY_URI:$IMAGE_TAG
      - echo Pushing to ECR...
      - docker push $REPOSITORY_URI:$IMAGE_TAG
      - echo Waiting for ECR scan...
      - |
        aws ecr wait image-scan-complete \
          --repository-name $IMAGE_REPO_NAME \
          --image-id imageTag=$IMAGE_TAG
      - echo Checking ECR scan results...
      - python check-ecr-scan-results.py $IMAGE_REPO_NAME $IMAGE_TAG
      - echo Signing image...
      - aws signer start-signing-job --profile-name payment-app-signing-profile --source ...
      - echo Validating task definition...
      - python validate-task-definition.py task-definition.json
      - echo All security checks passed!

Best Practices Summary

Do's ✅

Use minimal base images (distroless, alpine)
Multi-stage builds to reduce image size
Never include secrets in images
Scan images before deployment
Sign images for integrity verification
Run as non-root user
Drop all capabilities by default
Use readonly root filesystem
Set resource limits
Enable network policies

Don'ts ❌

Don't use latest tags in production
Don't run as root user
Don't include secrets in images or Dockerfiles
Don't skip scanning before deployment
Don't use privileged mode unless absolutely necessary
Don't ignore security findings
Don't disable read-only filesystem without justification
Don't forget to sign images

Conclusion

Securing containers requires a comprehensive approach across the entire lifecycle. Key takeaways:

Minimal base images reduce attack surface significantly
AWS Secrets Manager eliminates secrets from images
ECR scanning catches vulnerabilities before deployment
Image signing ensures integrity and authenticity
Runtime security (least privilege, isolation) protects running containers
Compliance automation ensures continuous adherence to standards

The result? A secure, compliant containerized application that meets financial services requirements while maintaining operational efficiency.

Additional Resources

Scaling a Payment Platform for Black Friday: A 10x Traffic Spike Strategy on AWS

Maureen Chebet — Wed, 26 Nov 2025 18:03:13 +0000

The Challenge

Black Friday is coming in 8 weeks, and your payment platform needs to handle a massive traffic spike:

Normal load: 500 transactions/second
Expected peak: 5,000 transactions/second (10x increase!)
Current capacity: Only 1,500 transactions/second
Budget constraint: Minimize costs and auto-scale down after the event
Requirement: Zero impact on current service levels during preparation

This isn't just about adding more servers. It requires a comprehensive scaling strategy across the entire stack: application layer, database, caching, networking, and monitoring.

In this article, I'll walk through a complete AWS-based scaling strategy that handles the 10x traffic spike while optimizing costs and maintaining security and compliance.

Scaling Strategy Overview

Multi-Layer Approach

Scaling for a 10x traffic spike requires addressing every layer:

┌─────────────────────────────────────┐
│   Application Load Balancer (ALB)   │  ← Distribute traffic
├─────────────────────────────────────┤
│   Application Tier (Auto-Scaling)    │  ← Horizontal scaling
├─────────────────────────────────────┤
│   Caching Layer (ElastiCache)        │  ← Reduce DB load
├─────────────────────────────────────┤
│   Database Tier (RDS + Replicas)    │  ← Read replicas + pooling
└─────────────────────────────────────┘

Key Principles

Horizontal Scaling: Add more instances, not bigger ones
Cost Optimization: Use Reserved Instances for baseline, On-Demand for peak
Predictive Scaling: Pre-scale before traffic arrives
Comprehensive Testing: Validate before the event
Auto-Scale Down: Aggressively reduce capacity after peak

Phase 1: Application Layer Auto-Scaling

Current State Analysis

# Analyze current capacity
# Current: 1,500 TPS with 10 instances = 150 TPS per instance
# Target: 5,000 TPS with 20% headroom = 6,000 TPS capacity
# Required: ~40 instances (4x scaling)

# Calculate instance requirements
CURRENT_INSTANCES=10
CURRENT_TPS=1500
TARGET_TPS=5000
HEADROOM_PERCENT=20
TARGET_CAPACITY=$(echo "$TARGET_TPS * (1 + $HEADROOM_PERCENT / 100)" | bc)
INSTANCES_NEEDED=$(echo "($TARGET_CAPACITY / ($CURRENT_TPS / $CURRENT_INSTANCES))" | bc)

echo "Target capacity: $TARGET_CAPACITY TPS"
echo "Instances needed: $INSTANCES_NEEDED"

Auto Scaling Group Configuration

CloudFormation Template:

Resources:
  ApplicationAutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      MinSize: 10
      MaxSize: 50
      DesiredCapacity: 10
      VPCZoneIdentifier:
        - !Ref AppSubnet1
        - !Ref AppSubnet2
        - !Ref AppSubnet3
      LaunchTemplate:
        LaunchTemplateId: !Ref ApplicationLaunchTemplate
        Version: !GetAtt ApplicationLaunchTemplate.LatestVersionNumber
      TargetGroupARNs:
        - !Ref ApplicationTargetGroup
      HealthCheckType: ELB
      HealthCheckGracePeriod: 300
      Cooldown: 300
      Tags:
        - Key: Name
          Value: PaymentApp
          PropagateAtLaunch: true
        - Key: Environment
          Value: Production
          PropagateAtLaunch: true

  ScaleUpPolicy:
    Type: AWS::AutoScaling::ScalingPolicy
    Properties:
      AutoScalingGroupName: !Ref ApplicationAutoScalingGroup
      PolicyType: TargetTrackingScaling
      TargetTrackingConfiguration:
        PredefinedMetricSpecification:
          PredefinedMetricType: ASGAverageCPUUtilization
        TargetValue: 70.0
        ScaleInCooldown: 300
        ScaleOutCooldown: 60

  ScheduledScaleUp:
    Type: AWS::AutoScaling::ScheduledAction
    Properties:
      AutoScalingGroupName: !Ref ApplicationAutoScalingGroup
      ScheduledActionName: BlackFridayScaleUp
      DesiredCapacity: 40
      Recurrence: "0 2 24 11 *"  # Nov 24 at 2 AM UTC (2 hours before peak)
      MinSize: 40
      MaxSize: 50

  ScheduledScaleDown:
    Type: AWS::AutoScaling::ScheduledAction
    Properties:
      AutoScalingGroupName: !Ref ApplicationAutoScalingGroup
      ScheduledActionName: BlackFridayScaleDown
      DesiredCapacity: 10
      Recurrence: "0 2 26 11 *"  # Nov 26 at 2 AM UTC (after Black Friday)
      MinSize: 10
      MaxSize: 50

Advanced Scaling Policies

Multi-Metric Scaling:

{
  "TargetTrackingScalingPolicies": [
    {
      "TargetValue": 70.0,
      "PredefinedMetricSpecification": {
        "PredefinedMetricType": "ASGAverageCPUUtilization"
      },
      "ScaleInCooldown": 300,
      "ScaleOutCooldown": 60
    },
    {
      "TargetValue": 500.0,
      "PredefinedMetricSpecification": {
        "PredefinedMetricType": "ALBRequestCountPerTarget",
        "ResourceLabel": "app/payment-alb/123456789/targetgroup/payment-tg/123456789"
      },
      "ScaleInCooldown": 300,
      "ScaleOutCooldown": 60
    }
  ]
}

Custom Metric Scaling (Response Time):

import boto3

cloudwatch = boto3.client('cloudwatch')
autoscaling = boto3.client('autoscaling')

def create_response_time_scaling_policy():
    """Create scaling policy based on response time"""

    # Put custom metric for response time
    cloudwatch.put_metric_data(
        Namespace='PaymentApp/Performance',
        MetricData=[{
            'MetricName': 'P95ResponseTime',
            'Value': 0.5,  # 500ms threshold
            'Unit': 'Seconds'
        }]
    )

    # Create scaling policy
    autoscaling.put_scaling_policy(
        AutoScalingGroupName='payment-app-asg',
        PolicyName='scale-on-response-time',
        PolicyType='TargetTrackingScaling',
        TargetTrackingConfiguration={
            'CustomizedMetricSpecification': {
                'MetricName': 'P95ResponseTime',
                'Namespace': 'PaymentApp/Performance',
                'Statistic': 'Average',
                'Unit': 'Seconds'
            },
            'TargetValue': 0.5,  # Scale if p95 > 500ms
            'ScaleInCooldown': 300,
            'ScaleOutCooldown': 60
        }
    )

ECS Auto-Scaling (If Using Containers)

# Register scalable target
aws application-autoscaling register-scalable-target \
  --service-namespace ecs \
  --scalable-dimension ecs:service:DesiredCount \
  --resource-id service/payment-cluster/payment-service \
  --min-capacity 10 \
  --max-capacity 50 \
  --role-arn arn:aws:iam::account:role/ecs-autoscaling-role

# Create scaling policy
aws application-autoscaling put-scaling-policy \
  --service-namespace ecs \
  --scalable-dimension ecs:service:DesiredCount \
  --resource-id service/payment-cluster/payment-service \
  --policy-name payment-service-scaling \
  --policy-type TargetTrackingScaling \
  --target-tracking-scaling-policy-configuration '{
    "TargetValue": 70.0,
    "PredefinedMetricSpecification": {
      "PredefinedMetricType": "ECSServiceAverageCPUUtilization"
    },
    "ScaleInCooldown": 300,
    "ScaleOutCooldown": 60
  }'

Warm Pools for Faster Scaling

# Create warm pool for faster instance startup
aws autoscaling put-warm-pool \
  --auto-scaling-group-name payment-app-asg \
  --min-size 5 \
  --max-size 10 \
  --instance-reuse-policy ReuseOnScaleIn=true

Phase 2: Database Scaling Strategy

Read Replicas Setup

Create RDS Read Replicas:

# Create read replica for Black Friday
aws rds create-db-instance-read-replica \
  --db-instance-identifier payment-db-read-replica-1 \
  --source-db-instance-identifier payment-db-primary \
  --db-instance-class db.r5.4xlarge \
  --publicly-accessible false \
  --availability-zone us-east-1a

# Create additional replicas
aws rds create-db-instance-read-replica \
  --db-instance-identifier payment-db-read-replica-2 \
  --source-db-instance-identifier payment-db-primary \
  --db-instance-class db.r5.4xlarge \
  --publicly-accessible false \
  --availability-zone us-east-1b

aws rds create-db-instance-read-replica \
  --db-instance-identifier payment-db-read-replica-3 \
  --source-db-instance-identifier payment-db-primary \
  --db-instance-class db.r5.4xlarge \
  --publicly-accessible false \
  --availability-zone us-east-1c

Read Replica Auto-Scaling:

import boto3
import json

rds = boto3.client('rds')
cloudwatch = boto3.client('cloudwatch')

def scale_read_replicas_based_on_load():
    """Dynamically add/remove read replicas based on load"""

    # Get current read replica count
    replicas = rds.describe_db_instances(
        Filters=[{
            'Name': 'db-instance-id',
            'Values': ['payment-db-read-replica-*']
        }]
    )

    current_replica_count = len(replicas.get('DBInstances', []))

    # Get database load metrics
    response = cloudwatch.get_metric_statistics(
        Namespace='AWS/RDS',
        MetricName='CPUUtilization',
        Dimensions=[
            {'Name': 'DBInstanceIdentifier', 'Value': 'payment-db-primary'}
        ],
        StartTime=datetime.utcnow() - timedelta(minutes=5),
        EndTime=datetime.utcnow(),
        Period=300,
        Statistics=['Average']
    )

    avg_cpu = response['Datapoints'][0]['Average'] if response['Datapoints'] else 0

    # Scale logic
    if avg_cpu > 80 and current_replica_count < 5:
        # Create new replica
        replica_id = f"payment-db-read-replica-{current_replica_count + 1}"
        rds.create_db_instance_read_replica(
            DBInstanceIdentifier=replica_id,
            SourceDBInstanceIdentifier='payment-db-primary',
            DBInstanceClass='db.r5.4xlarge'
        )
        print(f"Created read replica: {replica_id}")
    elif avg_cpu < 40 and current_replica_count > 2:
        # Remove replica (keep at least 2)
        replica_to_delete = f"payment-db-read-replica-{current_replica_count}"
        rds.delete_db_instance(DBInstanceIdentifier=replica_to_delete)
        print(f"Deleted read replica: {replica_to_delete}")

RDS Proxy for Connection Pooling

Create RDS Proxy:

# Create RDS Proxy for connection pooling
aws rds create-db-proxy \
  --db-proxy-name payment-db-proxy \
  --engine-family POSTGRESQL \
  --auth '{
    "AuthScheme": "SECRETS",
    "SecretArn": "arn:aws:secretsmanager:region:account:secret:payment-db-credentials",
    "IAMAuth": "DISABLED"
  }' \
  --role-arn arn:aws:iam::account:role/rds-proxy-role \
  --vpc-subnet-ids subnet-123 subnet-456 subnet-789 \
  --vpc-security-group-ids sg-12345678 \
  --require-tls \
  --idle-client-timeout 1800 \
  --max-connections-percent 100 \
  --max-idle-connections-percent 50

# Register database targets
aws rds register-db-proxy-targets \
  --db-proxy-name payment-db-proxy \
  --targets '[
    {
      "RdsResourceId": "payment-db-primary",
      "TargetRole": "WRITER"
    },
    {
      "RdsResourceId": "payment-db-read-replica-1",
      "TargetRole": "READER"
    },
    {
      "RdsResourceId": "payment-db-read-replica-2",
      "TargetRole": "READER"
    }
  ]'

Application Connection Using RDS Proxy:

import psycopg2
import boto3

def get_db_connection():
    """Get database connection through RDS Proxy"""

    # RDS Proxy endpoint
    proxy_endpoint = "payment-db-proxy.proxy-xxxxx.us-east-1.rds.amazonaws.com"

    # Get credentials from Secrets Manager
    secrets_client = boto3.client('secretsmanager')
    secret = secrets_client.get_secret_value(SecretId='payment-db-credentials')
    credentials = json.loads(secret['SecretString'])

    # Connect through proxy (automatic read/write splitting)
    connection = psycopg2.connect(
        host=proxy_endpoint,
        port=5432,
        database=credentials['database'],
        user=credentials['username'],
        password=credentials['password']
    )

    return connection

Database Query Optimization

Identify Slow Queries:

-- Enable query logging
ALTER DATABASE paymentdb SET log_min_duration_statement = 1000;  -- Log queries > 1s

-- Find slow queries
SELECT 
    pid,
    now() - pg_stat_activity.query_start AS duration,
    query,
    state
FROM pg_stat_activity
WHERE state = 'active'
  AND now() - pg_stat_activity.query_start > interval '1 second'
ORDER BY duration DESC;

-- Analyze query execution plan
EXPLAIN ANALYZE
SELECT * FROM transactions 
WHERE customer_id = $1 
  AND created_at >= NOW() - INTERVAL '30 days';

Optimize with Indexes:

-- Create indexes for common queries
CREATE INDEX CONCURRENTLY idx_transactions_customer_date 
ON transactions(customer_id, created_at DESC);

CREATE INDEX CONCURRENTLY idx_transactions_status 
ON transactions(status) 
WHERE status IN ('pending', 'processing');

-- Update statistics
ANALYZE transactions;

Phase 3: Caching Strategy

Amazon ElastiCache for Redis

Create Redis Cluster:

# Create Redis cluster for caching
aws elasticache create-cache-cluster \
  --cache-cluster-id payment-cache \
  --cache-node-type cache.r6g.xlarge \
  --engine redis \
  --num-cache-nodes 3 \
  --cache-parameter-group-name default.redis7 \
  --security-group-ids sg-cache-12345678 \
  --subnet-group-name payment-cache-subnet-group \
  --engine-version 7.0 \
  --port 6379 \
  --snapshot-retention-limit 7 \
  --automatic-failover-enabled \
  --multi-az-enabled

Application-Level Caching:

import redis
import json
import hashlib

redis_client = redis.Redis(
    host='payment-cache.xxxxx.cache.amazonaws.com',
    port=6379,
    decode_responses=True
)

def get_cached_data(cache_key, fetch_function, ttl=300):
    """Generic caching function"""

    # Try cache first
    cached = redis_client.get(cache_key)
    if cached:
        return json.loads(cached)

    # Cache miss - fetch from source
    data = fetch_function()

    # Store in cache
    redis_client.setex(
        cache_key,
        ttl,
        json.dumps(data)
    )

    return data

def get_customer_profile(customer_id):
    """Get customer profile with caching"""

    cache_key = f"customer:profile:{customer_id}"

    def fetch_from_db():
        # Database query
        conn = get_db_connection()
        cursor = conn.cursor()
        cursor.execute(
            "SELECT * FROM customers WHERE id = %s",
            (customer_id,)
        )
        return cursor.fetchone()

    return get_cached_data(cache_key, fetch_from_db, ttl=600)

def get_transaction_history(customer_id, days=30):
    """Get transaction history with caching"""

    cache_key = f"transactions:{customer_id}:{days}"

    def fetch_from_db():
        conn = get_db_connection()
        cursor = conn.cursor()
        cursor.execute("""
            SELECT * FROM transactions 
            WHERE customer_id = %s 
              AND created_at >= NOW() - INTERVAL '%s days'
            ORDER BY created_at DESC
        """, (customer_id, days))
        return cursor.fetchall()

    # Shorter TTL for transaction data (more dynamic)
    return get_cached_data(cache_key, fetch_from_db, ttl=60)

Cache Warming Before Black Friday:

def warm_cache_before_black_friday():
    """Pre-populate cache with frequently accessed data"""

    # Get top 1000 customers by transaction volume
    conn = get_db_connection()
    cursor = conn.cursor()
    cursor.execute("""
        SELECT customer_id, COUNT(*) as tx_count
        FROM transactions
        WHERE created_at >= NOW() - INTERVAL '90 days'
        GROUP BY customer_id
        ORDER BY tx_count DESC
        LIMIT 1000
    """)

    top_customers = cursor.fetchall()

    # Warm cache for each customer
    for customer_id, _ in top_customers:
        get_customer_profile(customer_id)
        get_transaction_history(customer_id, days=30)
        print(f"Warmed cache for customer {customer_id}")

    print(f"Cache warming complete for {len(top_customers)} customers")

Application Load Balancer Caching

ALB with CloudFront for Static Content:

# Create CloudFront distribution for static assets
aws cloudfront create-distribution \
  --distribution-config '{
    "CallerReference": "payment-app-static",
    "Aliases": {
      "Quantity": 1,
      "Items": ["static.payment.example.com"]
    },
    "DefaultRootObject": "index.html",
    "Origins": {
      "Quantity": 1,
      "Items": [{
        "Id": "payment-alb",
        "DomainName": "payment-alb-123456789.us-east-1.elb.amazonaws.com",
        "CustomOriginConfig": {
          "HTTPPort": 80,
          "HTTPSPort": 443,
          "OriginProtocolPolicy": "https-only"
        }
      }]
    },
    "DefaultCacheBehavior": {
      "TargetOriginId": "payment-alb",
      "ViewerProtocolPolicy": "redirect-to-https",
      "AllowedMethods": {
        "Quantity": 2,
        "Items": ["GET", "HEAD"],
        "CachedMethods": {
          "Quantity": 2,
          "Items": ["GET", "HEAD"]
        }
      },
      "Compress": true,
      "MinTTL": 0,
      "DefaultTTL": 86400,
      "MaxTTL": 31536000
    },
    "Enabled": true
  }'

Phase 4: Load Testing and Validation

AWS Distributed Load Testing

Create Load Test Configuration:

# load-test-config.yaml
testName: BlackFridayLoadTest
testDescription: Validate 10x traffic spike handling
testDuration: 3600  # 1 hour
rampUp: 600  # 10 minutes ramp-up
targetTPS: 5000
scenarios:
  - name: payment_processing
    weight: 70
    script: |
      POST /api/payments
      Headers:
        Authorization: Bearer ${token}
      Body:
        amount: ${random(10, 1000)}
        currency: USD
        customer_id: ${random_customer_id}

  - name: transaction_history
    weight: 20
    script: |
      GET /api/transactions?customer_id=${random_customer_id}
      Headers:
        Authorization: Bearer ${token}

  - name: account_balance
    weight: 10
    script: |
      GET /api/accounts/${random_customer_id}/balance
      Headers:
        Authorization: Bearer ${token}

Run Load Test:

# Create load test
aws iotdeviceadvisor create-suite-definition \
  --suite-definition-configuration '{
    "suiteDefinitionName": "BlackFridayLoadTest",
    "devices": [{
      "thingArn": "arn:aws:iot:region:account:thing/load-test-device"
    }]
  }'

# Or use AWS Distributed Load Testing service
aws load-testing create-test-run \
  --test-name BlackFridayLoadTest \
  --test-config file://load-test-config.yaml \
  --target-url https://payment.example.com

Custom Load Testing Script:

import asyncio
import aiohttp
import time
from statistics import mean, median

async def send_request(session, url, data):
    """Send single request"""
    start = time.time()
    try:
        async with session.post(url, json=data) as response:
            await response.read()
            duration = time.time() - start
            return {
                'status': response.status,
                'duration': duration,
                'success': response.status == 200
            }
    except Exception as e:
        return {
            'status': 0,
            'duration': time.time() - start,
            'success': False,
            'error': str(e)
        }

async def load_test(target_tps, duration_seconds):
    """Run load test"""

    url = "https://payment.example.com/api/payments"
    concurrent_requests = target_tps  # 1 request per second per connection

    results = []
    start_time = time.time()

    async with aiohttp.ClientSession() as session:
        while time.time() - start_time < duration_seconds:
            tasks = []
            for _ in range(concurrent_requests):
                data = {
                    'amount': 100.0,
                    'currency': 'USD',
                    'customer_id': 'test-customer-123'
                }
                tasks.append(send_request(session, url, data))

            batch_results = await asyncio.gather(*tasks)
            results.extend(batch_results)

            # Wait 1 second before next batch
            await asyncio.sleep(1)

    # Analyze results
    durations = [r['duration'] for r in results]
    successes = [r['success'] for r in results]

    print(f"Total requests: {len(results)}")
    print(f"Success rate: {sum(successes) / len(successes) * 100:.2f}%")
    print(f"Average response time: {mean(durations):.3f}s")
    print(f"Median response time: {median(durations):.3f}s")
    print(f"P95 response time: {sorted(durations)[int(len(durations) * 0.95)]:.3f}s")
    print(f"P99 response time: {sorted(durations)[int(len(durations) * 0.99)]:.3f}s")

    return results

# Run load test
asyncio.run(load_test(target_tps=5000, duration_seconds=3600))

Load Testing Phases

Week 1-2: Baseline Testing

# Establish baseline
python load_test.py --target-tps 500 --duration 300

Week 3-5: Incremental Testing

# Test at 1x, 2x, 3x, 5x, 10x
for multiplier in 1 2 3 5 10; do
  target_tps=$((500 * multiplier))
  echo "Testing at ${target_tps} TPS"
  python load_test.py --target-tps $target_tps --duration 600
done

Week 6: Stress Testing

# Push beyond expected load
python load_test.py --target-tps 6000 --duration 1800  # 30 minutes

Week 7: Soak Testing

# Sustained load at peak
python load_test.py --target-tps 5000 --duration 14400  # 4 hours

Phase 5: Cost Optimization

Reserved Instances Strategy

# Purchase Reserved Instances for baseline capacity
aws ec2 purchase-reserved-instances-offering \
  --reserved-instances-offering-id 12345678-1234-1234-1234-123456789012 \
  --instance-count 10 \
  --instance-type t3.large

# RDS Reserved Instances
aws rds purchase-reserved-db-instances-offering \
  --reserved-db-instances-offering-id 12345678-1234-1234-1234-123456789012 \
  --db-instance-count 1 \
  --db-instance-class db.r5.4xlarge

Spot Instances for Non-Critical Workloads

import boto3

ec2 = boto3.client('ec2')
autoscaling = boto3.client('autoscaling')

def configure_spot_instances():
    """Configure spot instances for cost savings"""

    # Create launch template with spot instances
    ec2.create_launch_template(
        LaunchTemplateName='payment-app-spot',
        LaunchTemplateData={
            'ImageId': 'ami-12345678',
            'InstanceType': 't3.large',
            'InstanceMarketOptions': {
                'MarketType': 'spot',
                'SpotOptions': {
                    'MaxPrice': '0.10',  # Maximum bid price
                    'SpotInstanceType': 'one-time',
                    'InstanceInterruptionBehavior': 'terminate'
                }
            }
        }
    )

    # Create separate ASG for spot instances (for non-critical workloads)
    autoscaling.create_auto_scaling_group(
        AutoScalingGroupName='payment-app-spot-asg',
        LaunchTemplate={
            'LaunchTemplateName': 'payment-app-spot',
            'Version': '$Latest'
        },
        MinSize=0,
        MaxSize=20,
        DesiredCapacity=0,  # Start with 0, scale up only when needed
        VPCZoneIdentifier='subnet-123,subnet-456',
        Tags=[
            {
                'Key': 'InstanceType',
                'Value': 'Spot',
                'PropagateAtLaunch': True
            }
        ]
    )

Auto-Scaling Down After Peak

import boto3
from datetime import datetime, timedelta

autoscaling = boto3.client('autoscaling')
cloudwatch = boto3.client('cloudwatch')

def aggressive_scale_down():
    """Aggressively scale down after Black Friday"""

    # Check if traffic has dropped
    response = cloudwatch.get_metric_statistics(
        Namespace='AWS/ApplicationELB',
        MetricName='RequestCount',
        Dimensions=[
            {'Name': 'LoadBalancer', 'Value': 'app/payment-alb/123456789'}
        ],
        StartTime=datetime.utcnow() - timedelta(hours=2),
        EndTime=datetime.utcnow(),
        Period=300,
        Statistics=['Sum']
    )

    # Calculate average requests per second
    total_requests = sum([d['Sum'] for d in response['Datapoints']])
    avg_rps = total_requests / (2 * 3600)  # 2 hours

    # If below normal (500 TPS), scale down aggressively
    if avg_rps < 600:  # 20% above normal
        autoscaling.set_desired_capacity(
            AutoScalingGroupName='payment-app-asg',
            DesiredCapacity=10,  # Back to baseline
            HonorCooldown=False  # Ignore cooldown for faster scale-down
        )
        print(f"Scaled down to baseline (current RPS: {avg_rps:.2f})")

Cost Monitoring Dashboard

import boto3
from datetime import datetime, timedelta

ce = boto3.client('ce')  # Cost Explorer

def get_daily_costs():
    """Get daily AWS costs"""

    end_date = datetime.now().strftime('%Y-%m-%d')
    start_date = (datetime.now() - timedelta(days=7)).strftime('%Y-%m-%d')

    response = ce.get_cost_and_usage(
        TimePeriod={
            'Start': start_date,
            'End': end_date
        },
        Granularity='DAILY',
        Metrics=['UnblendedCost'],
        GroupBy=[
            {'Type': 'DIMENSION', 'Key': 'SERVICE'},
            {'Type': 'DIMENSION', 'Key': 'USAGE_TYPE'}
        ]
    )

    daily_costs = {}
    for result in response['ResultsByTime']:
        date = result['TimePeriod']['Start']
        daily_costs[date] = {}

        for group in result['Groups']:
            service = group['Keys'][0]
            cost = float(group['Metrics']['UnblendedCost']['Amount'])
            daily_costs[date][service] = cost

    return daily_costs

# Create CloudWatch dashboard for costs
cloudwatch = boto3.client('cloudwatch')

def create_cost_dashboard():
    """Create CloudWatch dashboard for cost monitoring"""

    dashboard_body = {
        "widgets": [
            {
                "type": "metric",
                "properties": {
                    "metrics": [
                        ["AWS/Billing", "EstimatedCharges", {"stat": "Maximum"}]
                    ],
                    "period": 86400,
                    "stat": "Maximum",
                    "region": "us-east-1",
                    "title": "Daily AWS Costs"
                }
            }
        ]
    }

    cloudwatch.put_dashboard(
        DashboardName='BlackFridayCosts',
        DashboardBody=json.dumps(dashboard_body)
    )

Phase 6: Monitoring and Observability

CloudWatch Dashboards

Real-Time Operations Dashboard:

import boto3
import json

cloudwatch = boto3.client('cloudwatch')

def create_black_friday_dashboard():
    """Create comprehensive monitoring dashboard"""

    dashboard = {
        "widgets": [
            {
                "type": "metric",
                "properties": {
                    "metrics": [
                        ["AWS/ApplicationELB", "RequestCount", {"stat": "Sum", "period": 60}],
                        [".", "TargetResponseTime", {"stat": "Average", "period": 60}],
                        [".", "HTTPCode_Target_2XX_Count", {"stat": "Sum", "period": 60}],
                        [".", "HTTPCode_Target_5XX_Count", {"stat": "Sum", "period": 60}]
                    ],
                    "period": 60,
                    "stat": "Average",
                    "region": "us-east-1",
                    "title": "Application Load Balancer Metrics"
                }
            },
            {
                "type": "metric",
                "properties": {
                    "metrics": [
                        ["AWS/AutoScaling", "GroupDesiredCapacity", {"stat": "Average", "dimensions": [{"Name": "AutoScalingGroupName", "Value": "payment-app-asg"}]}],
                        [".", "GroupInServiceInstances", {"stat": "Average", "dimensions": [{"Name": "AutoScalingGroupName", "Value": "payment-app-asg"}]}],
                        [".", "GroupTotalInstances", {"stat": "Average", "dimensions": [{"Name": "AutoScalingGroupName", "Value": "payment-app-asg"}]}]
                    ],
                    "period": 60,
                    "stat": "Average",
                    "region": "us-east-1",
                    "title": "Auto Scaling Group"
                }
            },
            {
                "type": "metric",
                "properties": {
                    "metrics": [
                        ["AWS/RDS", "CPUUtilization", {"stat": "Average", "dimensions": [{"Name": "DBInstanceIdentifier", "Value": "payment-db-primary"}]}],
                        [".", "DatabaseConnections", {"stat": "Average", "dimensions": [{"Name": "DBInstanceIdentifier", "Value": "payment-db-primary"}]}],
                        [".", "ReadLatency", {"stat": "Average", "dimensions": [{"Name": "DBInstanceIdentifier", "Value": "payment-db-primary"}]}],
                        [".", "WriteLatency", {"stat": "Average", "dimensions": [{"Name": "DBInstanceIdentifier", "Value": "payment-db-primary"}]}]
                    ],
                    "period": 60,
                    "stat": "Average",
                    "region": "us-east-1",
                    "title": "RDS Database Metrics"
                }
            },
            {
                "type": "metric",
                "properties": {
                    "metrics": [
                        ["AWS/ElastiCache", "CPUUtilization", {"stat": "Average", "dimensions": [{"Name": "CacheClusterId", "Value": "payment-cache"}]}],
                        [".", "NetworkBytesIn", {"stat": "Sum", "dimensions": [{"Name": "CacheClusterId", "Value": "payment-cache"}]}],
                        [".", "NetworkBytesOut", {"stat": "Sum", "dimensions": [{"Name": "CacheClusterId", "Value": "payment-cache"}]}]
                    ],
                    "period": 60,
                    "stat": "Average",
                    "region": "us-east-1",
                    "title": "ElastiCache Metrics"
                }
            },
            {
                "type": "log",
                "properties": {
                    "query": "SOURCE '/aws/ecs/payment-service' | fields @timestamp, @message\n| filter @message like /ERROR/\n| sort @timestamp desc\n| limit 100",
                    "region": "us-east-1",
                    "title": "Application Errors",
                    "view": "table"
                }
            }
        ]
    }

    cloudwatch.put_dashboard(
        DashboardName='BlackFridayMonitoring',
        DashboardBody=json.dumps(dashboard)
    )

Critical Alerts

def create_critical_alerts():
    """Create CloudWatch alarms for critical metrics"""

    alarms = [
        {
            'AlarmName': 'high-error-rate',
            'MetricName': 'HTTPCode_Target_5XX_Count',
            'Namespace': 'AWS/ApplicationELB',
            'Statistic': 'Sum',
            'Period': 60,
            'EvaluationPeriods': 2,
            'Threshold': 10,
            'ComparisonOperator': 'GreaterThanThreshold'
        },
        {
            'AlarmName': 'high-response-time',
            'MetricName': 'TargetResponseTime',
            'Namespace': 'AWS/ApplicationELB',
            'Statistic': 'Average',
            'Period': 60,
            'EvaluationPeriods': 3,
            'Threshold': 1.0,  # 1 second
            'ComparisonOperator': 'GreaterThanThreshold'
        },
        {
            'AlarmName': 'database-connection-pool-exhausted',
            'MetricName': 'DatabaseConnections',
            'Namespace': 'AWS/RDS',
            'Statistic': 'Average',
            'Period': 300,
            'EvaluationPeriods': 2,
            'Threshold': 80,  # 80% of max connections
            'ComparisonOperator': 'GreaterThanThreshold'
        }
    ]

    for alarm in alarms:
        cloudwatch.put_metric_alarm(**alarm)
        print(f"Created alarm: {alarm['AlarmName']}")

Phase 7: Security and Compliance

Maintain Security During Scaling

# Ensure security groups allow traffic
aws ec2 authorize-security-group-ingress \
  --group-id sg-12345678 \
  --protocol tcp \
  --port 8080 \
  --source-group sg-alb-12345678

# Enable WAF on ALB
aws wafv2 create-web-acl \
  --name payment-app-waf \
  --scope REGIONAL \
  --default-action Allow={} \
  --rules file://waf-rules.json

Compliance Monitoring

def check_compliance_during_peak():
    """Verify compliance during peak load"""

    config = boto3.client('config')

    # Check encryption compliance
    response = config.get_compliance_summary_by_config_rule(
        ConfigRuleNames=['encrypted-volumes']
    )

    compliance = response['ComplianceSummariesByConfigRule'][0]

    if compliance['ComplianceSummary']['NonCompliantResourceCount']['CappedCount'] > 0:
        print("⚠️ Compliance issue detected during peak load")
        return False

    print("✅ All compliance checks passed")
    return True

Implementation Timeline

8-Week Preparation Plan

Weeks 1-2: Planning and Design

Finalize architecture
Create CloudFormation templates
Set up monitoring

Weeks 3-4: Infrastructure Implementation

Deploy auto-scaling groups
Set up read replicas
Configure ElastiCache
Implement RDS Proxy

Weeks 5-6: Load Testing Phase 1

Baseline testing
Incremental load testing (1x, 2x, 3x, 5x)
Identify and fix bottlenecks

Week 7: Load Testing Phase 2

Stress testing (6,000 TPS)
Soak testing (4 hours at 5,000 TPS)
Failover testing

Week 8: Final Preparations

Review all configurations
Team training
Dry run procedures
Final checklist

Black Friday (Event Day)

Pre-scale 2 hours before peak
Continuous monitoring
Incident response team on standby

Post-Black Friday

Scale down within 24 hours
Post-event review
Cost analysis
Lessons learned

Success Metrics

Performance Targets

target_metrics = {
    'response_time_p95_ms': 500,  # < 500ms
    'response_time_p99_ms': 1000,  # < 1s
    'error_rate_percent': 0.1,  # < 0.1%
    'transaction_success_rate_percent': 99.9,  # > 99.9%
    'availability_percent': 99.99,  # > 99.99%
    'auto_scaling_response_time_minutes': 5,  # < 5 minutes
    'cost_increase_percent': 400  # 4x for peak period only
}

Conclusion

Scaling for a 10x traffic spike requires a comprehensive, multi-layered approach. Key takeaways:

Horizontal auto-scaling with predictive scaling prevents capacity issues
Read replicas and RDS Proxy handle database load efficiently
Multi-level caching reduces database pressure significantly
Comprehensive load testing validates the strategy before the event
Cost optimization through Reserved Instances and aggressive scale-down
Real-time monitoring ensures visibility during peak load

The result? A payment platform that handles Black Friday traffic seamlessly while maintaining security, compliance, and cost efficiency.

Additional Resources

Building a DevSecOps Pipeline on AWS: From Security Audit to Daily Deployments

Maureen Chebet — Wed, 26 Nov 2025 17:39:25 +0000

The Wake-Up Call

After a security audit, your organization received some sobering findings:

Vulnerable dependencies in container images
Hardcoded secrets in the codebase (API keys, passwords, tokens)
Missing security scanning in CI/CD pipelines
No automated compliance checks
Slow security review process (3-5 days per release)

The challenge? Implement a comprehensive DevSecOps pipeline that:

Maintains daily deployment frequency
Doesn't add significant delays to the pipeline
Ensures regulatory compliance (PCI-DSS, GDPR, SOC 2)
Builds security into the culture, not just the tools

In this article, I'll walk through building a complete DevSecOps pipeline on AWS that addresses all these concerns while maintaining developer velocity.

DevSecOps Pipeline Architecture

The Shift-Left Philosophy

Shift-Left Security means catching security issues as early as possible in the development lifecycle:

Developer → Pre-Commit → CI Pipeline → Pre-Deployment → Production
   ↓            ↓            ↓              ↓              ↓
Local Scan   Git Hooks   SAST/DAST    Container Scan  Runtime Protection

The earlier we catch issues, the cheaper and faster they are to fix.

AWS DevSecOps Stack

Core AWS Services:

AWS CodePipeline - CI/CD orchestration
AWS CodeBuild - Build and security scanning
AWS CodeCommit/GitHub - Source code management
Amazon ECR - Container registry with scanning
AWS Secrets Manager - Secret management
AWS Security Hub - Centralized security findings
Amazon Inspector - Vulnerability assessment
AWS Config - Compliance monitoring
AWS IAM - Access control and least privilege

Phase 1: Secret Management Solution

The Problem: Hardcoded Secrets

Hardcoded secrets are a critical security risk. They can be:

Exposed in version control
Leaked in logs or error messages
Accessible to anyone with repository access

Solution: AWS Secrets Manager

Step 1: Migrate Existing Secrets

import boto3
import json
import re

secrets_client = boto3.client('secretsmanager')

def migrate_secrets_from_codebase():
    """Scan codebase and migrate hardcoded secrets to Secrets Manager"""

    # Patterns to detect secrets
    secret_patterns = {
        'api_key': r'api[_-]?key["\']?\s*[:=]\s*["\']([^"\']+)["\']',
        'password': r'password["\']?\s*[:=]\s*["\']([^"\']+)["\']',
        'token': r'token["\']?\s*[:=]\s*["\']([^"\']+)["\']',
        'secret': r'secret["\']?\s*[:=]\s*["\']([^"\']+)["\']'
    }

    # Scan files (example)
    files_to_scan = ['config.py', 'settings.py', '.env.example']

    for file_path in files_to_scan:
        with open(file_path, 'r') as f:
            content = f.read()

            for secret_type, pattern in secret_patterns.items():
                matches = re.findall(pattern, content, re.IGNORECASE)
                for match in matches:
                    # Create secret in Secrets Manager
                    secret_name = f"payment-app/{secret_type}/{file_path}"
                    secrets_client.create_secret(
                        Name=secret_name,
                        SecretString=match,
                        Description=f"Migrated from {file_path}"
                    )
                    print(f"Created secret: {secret_name}")

# Run migration
migrate_secrets_from_codebase()

Step 2: Update Application Code

import boto3
import json

secrets_client = boto3.client('secretsmanager')

def get_secret(secret_name):
    """Retrieve secret from AWS Secrets Manager"""
    try:
        response = secrets_client.get_secret_value(SecretId=secret_name)
        return json.loads(response['SecretString'])
    except Exception as e:
        print(f"Error retrieving secret: {e}")
        raise

# Usage in application
def connect_to_database():
    """Connect to database using secrets from Secrets Manager"""
    db_secrets = get_secret('payment-app/database/credentials')

    connection = psycopg2.connect(
        host=db_secrets['host'],
        database=db_secrets['database'],
        user=db_secrets['username'],
        password=db_secrets['password']
    )
    return connection

Step 3: Automatic Secret Rotation

import boto3

lambda_client = boto3.client('lambda')

def create_rotation_lambda():
    """Create Lambda function for automatic secret rotation"""

    rotation_code = '''
import boto3
import json
import psycopg2

def lambda_handler(event, context):
    """Rotate database password"""
    secrets_client = boto3.client('secretsmanager')

    # Get current secret
    secret_arn = event['SecretId']
    current_secret = json.loads(
        secrets_client.get_secret_value(SecretId=secret_arn)['SecretString']
    )

    # Generate new password
    new_password = generate_secure_password()

    # Update database
    conn = psycopg2.connect(
        host=current_secret['host'],
        database=current_secret['database'],
        user=current_secret['username'],
        password=current_secret['password']
    )
    # Update password in database
    # ... (implementation)

    # Update secret
    secrets_client.update_secret(
        SecretId=secret_arn,
        SecretString=json.dumps({
            **current_secret,
            'password': new_password
        })
    )

    return {'statusCode': 200}
'''

    # Create Lambda function
    lambda_client.create_function(
        FunctionName='rotate-db-secret',
        Runtime='python3.9',
        Role='arn:aws:iam::account:role/secret-rotation-role',
        Handler='index.lambda_handler',
        Code={'ZipFile': rotation_code.encode()}
    )

# Enable automatic rotation
secrets_client = boto3.client('secretsmanager')
secrets_client.rotate_secret(
    SecretId='payment-app/database/credentials',
    RotationLambdaARN='arn:aws:lambda:region:account:function:rotate-db-secret',
    RotationRules={
        'AutomaticallyAfterDays': 30
    }
)

Step 4: Pre-Commit Hook to Detect Secrets

#!/usr/bin/env python3
# .git/hooks/pre-commit

import subprocess
import re
import sys

def detect_secrets():
    """Detect potential secrets before commit"""

    # Get staged files
    result = subprocess.run(
        ['git', 'diff', '--cached', '--name-only'],
        capture_output=True,
        text=True
    )
    staged_files = result.stdout.strip().split('\n')

    # Secret patterns
    patterns = [
        (r'password\s*[:=]\s*["\']([^"\']+)["\']', 'Password detected'),
        (r'api[_-]?key\s*[:=]\s*["\']([^"\']+)["\']', 'API key detected'),
        (r'secret\s*[:=]\s*["\']([^"\']+)["\']', 'Secret detected'),
        (r'-----BEGIN (RSA |OPENSSH )?PRIVATE KEY-----', 'Private key detected'),
    ]

    violations = []

    for file_path in staged_files:
        if not file_path:
            continue

        try:
            with open(file_path, 'r') as f:
                content = f.read()
                for pattern, message in patterns:
                    if re.search(pattern, content, re.IGNORECASE):
                        violations.append(f"{file_path}: {message}")
        except Exception:
            continue

    if violations:
        print("❌ SECURITY VIOLATION: Potential secrets detected!")
        for violation in violations:
            print(f"  - {violation}")
        print("\nPlease use AWS Secrets Manager instead.")
        print("See: https://docs.aws.amazon.com/secretsmanager/")
        sys.exit(1)

    print("✅ No secrets detected in staged files")
    return 0

if __name__ == '__main__':
    sys.exit(detect_secrets())

Make it executable:

chmod +x .git/hooks/pre-commit

Phase 2: Security Scanning Strategy

SAST (Static Application Security Testing)

AWS CodeBuild with SAST Tools

# buildspec-sast.yml
version: 0.2
phases:
  pre_build:
    commands:
      - echo Installing SAST tools...
      - |
        # Install Semgrep for SAST
        pip install semgrep
      - |
        # Install Bandit for Python security scanning
        pip install bandit
  build:
    commands:
      - echo Running SAST scans...
      - |
        # Semgrep scan
        semgrep --config=auto \
          --json \
          --output=semgrep-results.json \
          .
      - |
        # Bandit scan (for Python)
        bandit -r . \
          -f json \
          -o bandit-results.json \
          || true
      - |
        # SonarQube scan (if using)
        sonar-scanner \
          -Dsonar.projectKey=payment-app \
          -Dsonar.sources=. \
          -Dsonar.host.url=$SONARQUBE_URL \
          -Dsonar.login=$SONARQUBE_TOKEN
  post_build:
    commands:
      - echo Uploading SAST results...
      - |
        # Upload to AWS Security Hub
        aws securityhub batch-import-findings \
          --findings file://convert-to-security-hub-format.json
      - |
        # Fail build if critical issues found
        python check-sast-results.py

SAST Results Checker:

# check-sast-results.py
import json
import sys

def check_sast_results():
    """Check SAST results and fail build if critical issues found"""

    with open('semgrep-results.json', 'r') as f:
        semgrep_results = json.load(f)

    critical_issues = []

    for result in semgrep_results.get('results', []):
        severity = result.get('extra', {}).get('severity', '')
        if severity in ['ERROR', 'WARNING']:
            critical_issues.append({
                'file': result.get('path', ''),
                'message': result.get('message', ''),
                'severity': severity
            })

    if critical_issues:
        print("Critical security issues found:")
        for issue in critical_issues:
            print(f"  - {issue['file']}: {issue['message']} ({issue['severity']})")
        sys.exit(1)

    print("SAST scan passed")
    return 0

if __name__ == '__main__':
    sys.exit(check_sast_results())

DAST (Dynamic Application Security Testing)

AWS CodeBuild with OWASP ZAP:

# buildspec-dast.yml
version: 0.2
phases:
  pre_build:
    commands:
      - echo Installing DAST tools...
      - |
        # Install OWASP ZAP
        wget https://github.com/zaproxy/zaproxy/releases/download/v2.12.0/ZAP_2.12.0_Linux.tar.gz
        tar -xzf ZAP_2.12.0_Linux.tar.gz
  build:
    commands:
      - echo Starting application for DAST...
      - |
        # Start application (example)
        docker-compose up -d payment-app
        sleep 30  # Wait for app to be ready
      - |
        # Run OWASP ZAP baseline scan
        ./ZAP_2.12.0/zap-baseline.py \
          -t http://localhost:8080 \
          -J zap-report.json
      - |
        # Run OWASP ZAP full scan (more thorough)
        ./ZAP_2.12.0/zap-full-scan.py \
          -t http://localhost:8080 \
          -J zap-full-report.json \
          -I  # Ignore warnings
  post_build:
    commands:
      - echo Processing DAST results...
      - |
        # Convert and upload to Security Hub
        python process-dast-results.py zap-report.json
      - |
        # Fail if critical vulnerabilities found
        python check-dast-results.py zap-report.json

DAST Results Processor:

# process-dast-results.py
import json
import sys
import boto3

securityhub = boto3.client('securityhub')

def process_dast_results(report_file):
    """Process DAST results and send to Security Hub"""

    with open(report_file, 'r') as f:
        zap_results = json.load(f)

    findings = []

    for site in zap_results.get('site', []):
        for alert in site.get('alerts', []):
            severity_map = {
                'High': 'HIGH',
                'Medium': 'MEDIUM',
                'Low': 'LOW',
                'Informational': 'INFORMATIONAL'
            }

            severity = severity_map.get(alert.get('risk', ''), 'INFORMATIONAL')

            if severity in ['HIGH', 'MEDIUM']:
                finding = {
                    'SchemaVersion': '2018-10-08',
                    'Id': f"zap-{alert.get('pluginid', 'unknown')}",
                    'ProductArn': 'arn:aws:securityhub:region:account:product/account/default',
                    'GeneratorId': 'owasp-zap',
                    'AwsAccountId': 'account-id',
                    'Types': ['Security Findings'],
                    'CreatedAt': '2024-01-01T00:00:00Z',
                    'UpdatedAt': '2024-01-01T00:00:00Z',
                    'Severity': {
                        'Label': severity
                    },
                    'Title': alert.get('name', 'Unknown vulnerability'),
                    'Description': alert.get('desc', ''),
                    'Remediation': {
                        'Recommendation': {
                            'Text': alert.get('solution', '')
                        }
                    }
                }
                findings.append(finding)

    if findings:
        # Batch import to Security Hub
        securityhub.batch_import_findings(Findings=findings)
        print(f"✅ Imported {len(findings)} findings to Security Hub")

    return findings

if __name__ == '__main__':
    if len(sys.argv) < 2:
        print("Usage: python process-dast-results.py <zap-report.json>")
        sys.exit(1)

    findings = process_dast_results(sys.argv[1])
    sys.exit(0 if not findings else 1)

Container Image Scanning

Amazon ECR Image Scanning:

# Enable automatic image scanning on push
aws ecr put-image-scanning-configuration \
  --repository-name payment-app \
  --image-scanning-configuration scanOnPush=true

# Scan existing images
aws ecr start-image-scan \
  --repository-name payment-app \
  --image-id imageTag=latest

# Get scan results
aws ecr describe-image-scan-findings \
  --repository-name payment-app \
  --image-id imageTag=latest

CodeBuild Integration:

# buildspec-container-scan.yml
version: 0.2
phases:
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
      - REPOSITORY_URI=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME
      - COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
      - IMAGE_TAG=${COMMIT_HASH:=latest}
  build:
    commands:
      - echo Building Docker image...
      - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $REPOSITORY_URI:$IMAGE_TAG
  post_build:
    commands:
      - echo Pushing Docker image...
      - docker push $REPOSITORY_URI:$IMAGE_TAG
      - echo Waiting for image scan to complete...
      - |
        # Wait for scan to complete (ECR scans automatically on push)
        aws ecr wait image-scan-complete \
          --repository-name $IMAGE_REPO_NAME \
          --image-id imageTag=$IMAGE_TAG \
          --max-attempts 30 \
          --delay 10
      - echo Retrieving scan results...
      - |
        # Get scan findings
        SCAN_RESULTS=$(aws ecr describe-image-scan-findings \
          --repository-name $IMAGE_REPO_NAME \
          --image-id imageTag=$IMAGE_TAG \
          --query 'imageScanFindings' \
          --output json)
      - |
        # Check for critical vulnerabilities
        CRITICAL_COUNT=$(echo $SCAN_RESULTS | jq '.findingCounts.CRITICAL // 0')
        HIGH_COUNT=$(echo $SCAN_RESULTS | jq '.findingCounts.HIGH // 0')

        if [ "$CRITICAL_COUNT" -gt 0 ] || [ "$HIGH_COUNT" -gt 5 ]; then
          echo "❌ Critical vulnerabilities found in image!"
          echo "Critical: $CRITICAL_COUNT, High: $HIGH_COUNT"
          exit 1
        fi

        echo "Image scan passed (Critical: $CRITICAL_COUNT, High: $HIGH_COUNT)"
      - |
        # Export scan results to Security Hub
        python export-ecr-findings.py $IMAGE_REPO_NAME $IMAGE_TAG

ECR Findings Exporter:

# export-ecr-findings.py
import boto3
import json
import sys

def export_ecr_findings(repo_name, image_tag):
    """Export ECR scan findings to Security Hub"""

    ecr = boto3.client('ecr')
    securityhub = boto3.client('securityhub')

    # Get scan findings
    response = ecr.describe_image_scan_findings(
        repositoryName=repo_name,
        imageId={'imageTag': image_tag}
    )

    findings = response.get('imageScanFindings', {})
    findings_list = findings.get('findings', [])

    security_hub_findings = []

    for finding in findings_list:
        severity_map = {
            'CRITICAL': 'CRITICAL',
            'HIGH': 'HIGH',
            'MEDIUM': 'MEDIUM',
            'LOW': 'LOW',
            'INFORMATIONAL': 'INFORMATIONAL'
        }

        severity = severity_map.get(finding.get('severity', ''), 'INFORMATIONAL')

        # Only export HIGH and CRITICAL
        if severity in ['CRITICAL', 'HIGH']:
            security_hub_finding = {
                'SchemaVersion': '2018-10-08',
                'Id': f"ecr-{repo_name}-{image_tag}-{finding.get('name', 'unknown')}",
                'ProductArn': f'arn:aws:securityhub:region:account:product/account/default',
                'GeneratorId': 'amazon-ecr-image-scan',
                'AwsAccountId': 'account-id',
                'Types': ['Vulnerabilities'],
                'CreatedAt': finding.get('firstObservedAt', ''),
                'UpdatedAt': finding.get('lastObservedAt', ''),
                'Severity': {
                    'Label': severity
                },
                'Title': finding.get('name', 'Unknown vulnerability'),
                'Description': finding.get('description', ''),
                'Remediation': {
                    'Recommendation': {
                        'Text': finding.get('remediation', {}).get('recommendation', {}).get('text', '')
                    }
                },
                'Resources': [{
                    'Type': 'AwsEcrContainerImage',
                    'Id': f"{repo_name}:{image_tag}",
                    'Region': 'us-east-1'
                }]
            }
            security_hub_findings.append(security_hub_finding)

    if security_hub_findings:
        securityhub.batch_import_findings(Findings=security_hub_findings)
        print(f"✅ Exported {len(security_hub_findings)} findings to Security Hub")

    return security_hub_findings

if __name__ == '__main__':
    if len(sys.argv) < 3:
        print("Usage: python export-ecr-findings.py <repo-name> <image-tag>")
        sys.exit(1)

    findings = export_ecr_findings(sys.argv[1], sys.argv[2])
    sys.exit(0)

Dependency Scanning

Scanning Dependencies for Vulnerabilities:

# buildspec-dependency-scan.yml
version: 0.2
phases:
  pre_build:
    commands:
      - echo Installing dependency scanning tools...
      - |
        # For Python
        pip install safety
      - |
        # For Node.js
        npm install -g npm-audit-resolver
      - |
        # For Java (Maven)
        # Use OWASP Dependency-Check
        wget https://github.com/jeremylong/DependencyCheck/releases/download/v8.4.0/dependency-check-8.4.0-release.zip
        unzip dependency-check-8.4.0-release.zip
  build:
    commands:
      - echo Scanning dependencies...
      - |
        # Python safety check
        safety check --json --output safety-report.json || true
      - |
        # Node.js npm audit
        npm audit --json > npm-audit-report.json || true
      - |
        # Java/Maven dependency check
        ./dependency-check/bin/dependency-check.sh \
          --project payment-app \
          --scan . \
          --format JSON \
          --out dependency-check-report.json
  post_build:
    commands:
      - echo Processing dependency scan results...
      - |
        # Process and upload to Security Hub
        python process-dependency-results.py
      - |
        # Fail build if critical vulnerabilities found
        python check-dependency-results.py

Phase 3: Complete DevSecOps Pipeline

AWS CodePipeline with Security Gates

{
  "pipeline": {
    "name": "payment-app-devsecops-pipeline",
    "roleArn": "arn:aws:iam::account:role/CodePipelineServiceRole",
    "artifactStore": {
      "type": "S3",
      "location": "payment-app-artifacts"
    },
    "stages": [
      {
        "name": "Source",
        "actions": [{
          "name": "SourceAction",
          "actionTypeId": {
            "category": "Source",
            "owner": "AWS",
            "provider": "CodeCommit",
            "version": "1"
          },
          "outputArtifacts": [{"name": "SourceOutput"}],
          "configuration": {
            "RepositoryName": "payment-app",
            "BranchName": "main"
          }
        }]
      },
      {
        "name": "PreBuildSecurity",
        "actions": [{
          "name": "SecretScan",
          "actionTypeId": {
            "category": "Build",
            "owner": "AWS",
            "provider": "CodeBuild",
            "version": "1"
          },
          "inputArtifacts": [{"name": "SourceOutput"}],
          "outputArtifacts": [{"name": "SecretScanOutput"}],
          "configuration": {
            "ProjectName": "payment-app-secret-scan"
          }
        }]
      },
      {
        "name": "Build",
        "actions": [{
          "name": "BuildAndSAST",
          "actionTypeId": {
            "category": "Build",
            "owner": "AWS",
            "provider": "CodeBuild",
            "version": "1"
          },
          "inputArtifacts": [{"name": "SourceOutput"}],
          "outputArtifacts": [{"name": "BuildOutput"}],
          "configuration": {
            "ProjectName": "payment-app-build-sast"
          }
        }]
      },
      {
        "name": "SecurityGates",
        "actions": [
          {
            "name": "DependencyScan",
            "actionTypeId": {
              "category": "Build",
              "owner": "AWS",
              "provider": "CodeBuild",
              "version": "1"
            },
            "inputArtifacts": [{"name": "BuildOutput"}],
            "outputArtifacts": [{"name": "DependencyScanOutput"}],
            "configuration": {
              "ProjectName": "payment-app-dependency-scan"
            }
          },
          {
            "name": "ContainerBuildAndScan",
            "actionTypeId": {
              "category": "Build",
              "owner": "AWS",
              "provider": "CodeBuild",
              "version": "1"
            },
            "inputArtifacts": [{"name": "BuildOutput"}],
            "outputArtifacts": [{"name": "ContainerOutput"}],
            "configuration": {
              "ProjectName": "payment-app-container-build-scan"
            }
          }
        ]
      },
      {
        "name": "DAST",
        "actions": [{
          "name": "DynamicScan",
          "actionTypeId": {
            "category": "Test",
            "owner": "AWS",
            "provider": "CodeBuild",
            "version": "1"
          },
          "inputArtifacts": [{"name": "ContainerOutput"}],
          "outputArtifacts": [{"name": "DASTOutput"}],
          "configuration": {
            "ProjectName": "payment-app-dast"
          }
        }]
      },
      {
        "name": "DeployToStaging",
        "actions": [{
          "name": "DeployStaging",
          "actionTypeId": {
            "category": "Deploy",
            "owner": "AWS",
            "provider": "ECS",
            "version": "1"
          },
          "inputArtifacts": [{"name": "ContainerOutput"}],
          "configuration": {
            "ClusterName": "payment-staging",
            "ServiceName": "payment-service"
          }
        }]
      },
      {
        "name": "ComplianceCheck",
        "actions": [{
          "name": "ComplianceValidation",
          "actionTypeId": {
            "category": "Test",
            "owner": "AWS",
            "provider": "CodeBuild",
            "version": "1"
          },
          "inputArtifacts": [{"name": "DASTOutput"}],
          "configuration": {
            "ProjectName": "payment-app-compliance-check"
          }
        }]
      },
      {
        "name": "DeployToProduction",
        "actions": [{
          "name": "DeployProduction",
          "actionTypeId": {
            "category": "Deploy",
            "owner": "AWS",
            "provider": "ECS",
            "version": "1"
          },
          "inputArtifacts": [{"name": "ContainerOutput"}],
          "configuration": {
            "ClusterName": "payment-production",
            "ServiceName": "payment-service"
          }
        }]
      }
    ]
  }
}

Security Gates Implementation

Lambda Function for Security Gate Evaluation:

# security-gate-evaluator.py
import boto3
import json

securityhub = boto3.client('securityhub')
codebuild = boto3.client('codebuild')

def evaluate_security_gates(build_id):
    """Evaluate security gates before allowing deployment"""

    # Get Security Hub findings for this build
    findings = securityhub.get_findings(
        Filters={
            'ResourceId': [{'Value': build_id, 'Comparison': 'EQUALS'}],
            'SeverityLabel': [
                {'Value': 'CRITICAL', 'Comparison': 'EQUALS'},
                {'Value': 'HIGH', 'Comparison': 'EQUALS'}
            ]
        }
    )

    critical_findings = [
        f for f in findings.get('Findings', [])
        if f.get('Severity', {}).get('Label') == 'CRITICAL'
    ]

    high_findings = [
        f for f in findings.get('Findings', [])
        if f.get('Severity', {}).get('Label') == 'HIGH'
    ]

    # Gate rules
    gate_passed = True
    reasons = []

    # Rule 1: No CRITICAL findings allowed
    if critical_findings:
        gate_passed = False
        reasons.append(f"{len(critical_findings)} CRITICAL findings found")

    # Rule 2: Maximum 3 HIGH findings allowed
    if len(high_findings) > 3:
        gate_passed = False
        reasons.append(f"{len(high_findings)} HIGH findings found (max 3 allowed)")

    # Rule 3: No hardcoded secrets
    secret_findings = [
        f for f in findings.get('Findings', [])
        if 'secret' in f.get('Title', '').lower() or 'password' in f.get('Title', '').lower()
    ]
    if secret_findings:
        gate_passed = False
        reasons.append(f"{len(secret_findings)} secret-related findings found")

    result = {
        'gate_passed': gate_passed,
        'reasons': reasons,
        'critical_count': len(critical_findings),
        'high_count': len(high_findings)
    }

    return result

def lambda_handler(event, context):
    """Lambda handler for security gate evaluation"""

    build_id = event.get('build_id')
    result = evaluate_security_gates(build_id)

    if not result['gate_passed']:
        # Stop pipeline
        raise Exception(f"Security gate failed: {', '.join(result['reasons'])}")

    return {
        'statusCode': 200,
        'body': json.dumps(result)
    }

Phase 4: Compliance Automation

AWS Config for Compliance Monitoring

Enable AWS Config:

# Enable Config
aws configservice put-configuration-recorder \
  --configuration-recorder name=default,roleArn=arn:aws:iam::account:role/ConfigRole

# Start recording
aws configservice start-configuration-recorder \
  --configuration-recorder-name default

PCI-DSS Compliance Rules:

{
  "ConfigRuleName": "pci-dss-encryption-check",
  "Description": "Check that EBS volumes are encrypted (PCI-DSS requirement)",
  "Scope": {
    "ComplianceResourceTypes": ["AWS::EC2::Volume"]
  },
  "Source": {
    "Owner": "AWS",
    "SourceIdentifier": "ENCRYPTED_VOLUMES"
  }
}

Custom Compliance Rule:

# custom-compliance-rule.py
import boto3
import json

config = boto3.client('config')

def evaluate_compliance(configuration_item):
    """Evaluate if resource is compliant"""

    compliance_status = 'COMPLIANT'
    annotation = ''

    # Example: Check if RDS instance has encryption enabled
    if configuration_item['resourceType'] == 'AWS::RDS::DBInstance':
        if not configuration_item.get('configuration', {}).get('storageEncrypted', False):
            compliance_status = 'NON_COMPLIANT'
            annotation = 'RDS instance must have encryption enabled for PCI-DSS compliance'

    # Example: Check if security groups allow unrestricted access
    if configuration_item['resourceType'] == 'AWS::EC2::SecurityGroup':
        ip_permissions = configuration_item.get('configuration', {}).get('ipPermissions', [])
        for perm in ip_permissions:
            for ip_range in perm.get('ipRanges', []):
                if ip_range.get('cidrIp') == '0.0.0.0/0':
                    compliance_status = 'NON_COMPLIANT'
                    annotation = 'Security group allows unrestricted access (0.0.0.0/0)'

    return {
        'compliance_type': compliance_status,
        'annotation': annotation
    }

def lambda_handler(event, context):
    """Lambda handler for Config custom rule"""

    configuration_item = json.loads(event['invokingEvent'])['configurationItem']
    evaluation = evaluate_compliance(configuration_item)

    config.put_evaluations(
        Evaluations=[{
            'ComplianceResourceType': configuration_item['resourceType'],
            'ComplianceResourceId': configuration_item['resourceId'],
            'ComplianceType': evaluation['compliance_type'],
            'Annotation': evaluation['annotation'],
            'OrderingTimestamp': configuration_item['configurationItemCaptureTime']
        }]
    )

    return evaluation

Automated Compliance Reporting

# compliance-reporter.py
import boto3
from datetime import datetime

config = boto3.client('config')
s3 = boto3.client('s3')

def generate_compliance_report():
    """Generate compliance report for audit"""

    # Get compliance summary
    response = config.get_compliance_summary_by_config_rule()

    report = {
        'timestamp': datetime.utcnow().isoformat(),
        'compliance_summary': response.get('ComplianceSummariesByConfigRule', []),
        'overall_compliance': calculate_overall_compliance(response)
    }

    # Generate detailed findings
    findings = []
    for rule_summary in response.get('ComplianceSummariesByConfigRule', []):
        if rule_summary.get('ComplianceSummary', {}).get('NonCompliantResourceCount', {}).get('CappedCount', 0) > 0:
            findings.append({
                'rule_name': rule_summary.get('ConfigRuleName', ''),
                'non_compliant_count': rule_summary.get('ComplianceSummary', {}).get('NonCompliantResourceCount', {}).get('CappedCount', 0)
            })

    report['findings'] = findings

    # Save to S3 for audit trail
    s3.put_object(
        Bucket='compliance-reports',
        Key=f"compliance-report-{datetime.utcnow().strftime('%Y-%m-%d')}.json",
        Body=json.dumps(report, indent=2)
    )

    return report

def calculate_overall_compliance(response):
    """Calculate overall compliance percentage"""

    total_resources = 0
    compliant_resources = 0

    for rule_summary in response.get('ComplianceSummariesByConfigRule', []):
        summary = rule_summary.get('ComplianceSummary', {})
        total_resources += summary.get('ComplianceResourceCount', {}).get('CappedCount', 0)
        compliant_resources += summary.get('CompliantResourceCount', {}).get('CappedCount', 0)

    if total_resources == 0:
        return 100.0

    return (compliant_resources / total_resources) * 100

# Schedule with EventBridge
import boto3

events = boto3.client('events')

events.put_rule(
    Name='daily-compliance-report',
    ScheduleExpression='cron(0 2 * * ? *)',  # Daily at 2 AM
    State='ENABLED'
)

events.put_targets(
    Rule='daily-compliance-report',
    Targets=[{
        'Id': '1',
        'Arn': 'arn:aws:lambda:region:account:function:compliance-reporter',
        'Input': json.dumps({})
    }]
)

Phase 5: Security Hub Integration

Centralized Security Findings

Enable AWS Security Hub:

# Enable Security Hub
aws securityhub enable-security-hub

# Enable security standards
aws securityhub batch-enable-standards \
  --standards-subscription-requests \
    StandardsArn=arn:aws:securityhub:region::standards/aws-foundational-security-best-practices/v/1.0.0 \
    StandardsArn=arn:aws:securityhub:region::standards/pci-dss/v/3.2.1

Aggregate Findings from All Sources:

# security-findings-aggregator.py
import boto3
from datetime import datetime, timedelta

securityhub = boto3.client('securityhub')

def aggregate_security_findings():
    """Aggregate security findings from all sources"""

    # Get findings from last 24 hours
    end_time = datetime.utcnow()
    start_time = end_time - timedelta(days=1)

    findings = securityhub.get_findings(
        Filters={
            'CreatedAt': [{
                'Start': start_time.isoformat(),
                'End': end_time.isoformat()
            }],
            'SeverityLabel': [
                {'Value': 'CRITICAL', 'Comparison': 'EQUALS'},
                {'Value': 'HIGH', 'Comparison': 'EQUALS'}
            ]
        },
        MaxResults=100
    )

    # Group by source
    findings_by_source = {}
    for finding in findings.get('Findings', []):
        source = finding.get('ProductFields', {}).get('aws/securityhub/SourceIdentifier', 'unknown')
        if source not in findings_by_source:
            findings_by_source[source] = []
        findings_by_source[source].append(finding)

    # Generate summary
    summary = {
        'timestamp': datetime.utcnow().isoformat(),
        'total_findings': len(findings.get('Findings', [])),
        'findings_by_source': {
            source: len(finds) for source, finds in findings_by_source.items()
        },
        'critical_count': len([f for f in findings.get('Findings', []) if f.get('Severity', {}).get('Label') == 'CRITICAL']),
        'high_count': len([f for f in findings.get('Findings', []) if f.get('Severity', {}).get('Label') == 'HIGH'])
    }

    return summary

Phase 6: Pipeline Performance Optimization

Parallel Execution

Run Security Scans in Parallel:

# buildspec-parallel-security.yml
version: 0.2
phases:
  build:
    commands:
      - echo Running security scans in parallel...
      - |
        # Run SAST, dependency scan, and secret scan in parallel
        semgrep --config=auto . &
        safety check &
        git-secrets --scan &
        wait  # Wait for all to complete
      - echo All security scans completed

Caching for Faster Builds

CodeBuild Cache Configuration:

{
  "cache": {
    "type": "LOCAL",
    "modes": [
      "LOCAL_DOCKER_LAYER_CACHE",
      "LOCAL_SOURCE_CACHE"
    ]
  }
}

Docker Layer Caching:

# Dockerfile with layer caching
FROM maven:3.8-openjdk-11 AS dependencies
COPY pom.xml .
RUN mvn dependency:go-offline

FROM dependencies AS build
COPY src ./src
RUN mvn package -DskipTests

FROM openjdk:11-jre-slim
COPY --from=build /app/target/*.jar app.jar
ENTRYPOINT ["java", "-jar", "app.jar"]

Fail-Fast Strategy

Early Failure Detection:

# fail-fast-security-checks.py
import sys
import subprocess

def run_fast_security_checks():
    """Run quick security checks that fail fast"""

    checks = [
        ('Secret detection', 'git-secrets --scan'),
        ('Basic SAST', 'semgrep --config=auto --error'),
        ('Dependency vulnerabilities', 'safety check --short-report')
    ]

    failed_checks = []

    for check_name, command in checks:
        print(f"Running {check_name}...")
        result = subprocess.run(
            command.split(),
            capture_output=True,
            timeout=60  # Fail fast with timeout
        )

        if result.returncode != 0:
            failed_checks.append(check_name)
            print(f"❌ {check_name} failed")
            print(result.stdout.decode())
        else:
            print(f"✅ {check_name} passed")

    if failed_checks:
        print(f"\n❌ Failed checks: {', '.join(failed_checks)}")
        sys.exit(1)

    print("\n✅ All fast security checks passed")
    return 0

if __name__ == '__main__':
    sys.exit(run_fast_security_checks())

Phase 7: Team Training and Adoption

Developer Onboarding

Security Training Materials:

# Security Best Practices Guide

## Secret Management
- Never commit secrets to version control
- Use AWS Secrets Manager for all secrets
- Rotate secrets regularly

## Dependency Management
- Keep dependencies up to date
- Review security advisories
- Use dependency scanning tools

## Code Security
- Follow OWASP Top 10 guidelines
- Use SAST tools before committing
- Review security findings promptly

Automated Security Reminders

GitHub/GitLab Integration:

# security-reminder-bot.py
import boto3
import requests

def send_security_reminder(pr_number, findings_count):
    """Send reminder about security findings in PR"""

    message = f"""
 Security Review Required

PR #{pr_number} has {findings_count} security findings that need attention.

Please review and address:
- Critical findings must be fixed before merge
- High findings should be addressed or justified
- Medium/Low findings can be tracked as technical debt

View findings: https://security-hub.aws.amazon.com/findings
"""

    # Send to Slack/Teams
    webhook_url = os.environ.get('SLACK_WEBHOOK_URL')
    requests.post(webhook_url, json={'text': message})

Security Champions Program

Identify and Train Security Champions:

# security-champion-tracker.py
import boto3

def identify_security_champions():
    """Identify developers who actively address security issues"""

    codecommit = boto3.client('codecommit')

    # Analyze commit history for security-related commits
    # Track who fixes security issues
    # Recognize top contributors

    champions = [
        {
            'name': 'Developer Name',
            'security_fixes': 15,
            'areas': ['SAST fixes', 'Dependency updates', 'Secret management']
        }
    ]

    return champions

Metrics and Success Criteria

Key Performance Indicators

# devsecops-metrics.py
import boto3
from datetime import datetime, timedelta

def calculate_devsecops_metrics():
    """Calculate DevSecOps pipeline metrics"""

    codebuild = boto3.client('codebuild')
    securityhub = boto3.client('securityhub')

    # Get pipeline metrics
    builds = codebuild.batch_get_builds(
        ids=get_recent_build_ids()
    )

    metrics = {
        'pipeline_duration': calculate_avg_pipeline_duration(builds),
        'security_scan_time': calculate_avg_scan_time(builds),
        'findings_detected': get_findings_count_last_30_days(),
        'findings_fixed': get_findings_fixed_count(),
        'deployment_frequency': get_deployment_frequency(),
        'mean_time_to_remediate': calculate_mttr()
    }

    return metrics

def calculate_avg_pipeline_duration(builds):
    """Calculate average pipeline duration"""
    durations = []
    for build in builds.get('builds', []):
        start = build.get('startTime')
        end = build.get('endTime')
        if start and end:
            duration = (end - start).total_seconds()
            durations.append(duration)

    return sum(durations) / len(durations) if durations else 0

# Target metrics
target_metrics = {
    'pipeline_duration_minutes': 15,  # Target: < 15 minutes
    'security_scan_time_minutes': 5,  # Target: < 5 minutes
    'findings_detected_per_week': 0,  # Target: Reduce over time
    'mean_time_to_remediate_hours': 24,  # Target: < 24 hours
    'deployment_frequency_per_day': 1  # Target: Maintain daily deployments
}

Best Practices Summary

Do's ✅

Shift-Left: Run security checks as early as possible
Automate Everything: Don't rely on manual security reviews
Fail Fast: Catch issues early, fail builds quickly
Centralize Findings: Use Security Hub for unified view
Educate Team: Security is everyone's responsibility
Measure Everything: Track metrics to improve

Don'ts ❌

Don't Block Developers: Balance security with velocity
Don't Ignore False Positives: Tune rules to reduce noise
Don't Skip Compliance: Automate compliance checks
Don't Forget Runtime: Security doesn't end at deployment
Don't Set It and Forget It: Continuously improve the pipeline

Conclusion

Building a DevSecOps pipeline on AWS requires integrating security at every stage of the development lifecycle. Key takeaways:

AWS Secrets Manager eliminates hardcoded secrets
Multi-layered scanning (SAST, DAST, container, dependency) catches issues early
Security Hub provides centralized visibility
Automated compliance ensures regulatory requirements are met
Security gates prevent vulnerable code from reaching production
Performance optimization maintains daily deployment frequency

The result? A secure, compliant, and fast development pipeline that builds security into your culture, not just your tools.

Additional Resources

Migrating a mobile banking app Platform to AWS with Zero Downtime: A Complete Guide

Maureen Chebet — Wed, 26 Nov 2025 17:24:12 +0000

The Challenge

Your fintech company needs to migrate from on-premise infrastructure to AWS within 6 months. The requirements are strict:

Zero downtime during migration
8TB PostgreSQL database to migrate
Legacy Spring Boot monolithic application
On-premise Jenkins CI/CD to transform
Compliance: PCI-DSS and GDPR requirements
Cost reduction: 40% savings target
Uptime improvement: From 99.9% to 99.99%

This isn't just a lift-and-shift. It's a transformation that must maintain business continuity while modernizing infrastructure, improving security, and reducing costs.

In this article, I'll walk through a comprehensive AWS migration strategy that ensures zero downtime, maintains compliance, and optimizes costs.

Migration Strategy Overview

Why AWS Database Migration Service (DMS) is Critical

For a zero-downtime migration of an 8TB database, AWS Database Migration Service (DMS) is the cornerstone. It supports continuous replication, allowing you to migrate with minimal downtime.

Migration Approach: Blue-Green Deployment

Blue Environment: Current on-premise infrastructure (production)
Green Environment: New AWS infrastructure (parallel)
Gradual Cutover: Route traffic incrementally from Blue to Green
Rollback Capability: Switch back to Blue if issues arise

High-Level Migration Phases

Phase 1: Assessment & Planning (Weeks 1-4)
Phase 2: AWS Foundation Setup (Weeks 5-8)
Phase 3: Application Migration (Weeks 9-16)
Phase 4: Database Migration (Weeks 17-22)
Phase 5: CI/CD Migration (Weeks 19-22)
Phase 6: Cutover & Validation (Weeks 23-24)
Phase 7: Optimization & Decommissioning (Weeks 25-26)

Phase 1: Assessment & Planning

Application Discovery

AWS Application Discovery Service helps inventory your on-premise infrastructure:

# Install discovery agent on on-premise servers
# Automatically discovers:
# - Server configurations
# - Network dependencies
# - Application dependencies
# - Performance metrics

Key Metrics to Collect:

CPU, memory, disk utilization patterns
Network traffic patterns
Database query patterns
Application dependencies
Security requirements

Database Assessment

AWS Schema Conversion Tool (SCT) for PostgreSQL analysis:

# Analyze PostgreSQL database
aws dms describe-schemas \
  --endpoint-arn arn:aws:dms:region:account:endpoint:source-endpoint

# Key assessments:
# - Database size: 8TB
# - Table count and sizes
# - Index usage
# - Foreign key relationships
# - Stored procedures and functions
# - Custom data types

Cost Estimation

AWS Pricing Calculator and AWS Cost Explorer:

# Estimate monthly costs
# Components to estimate:
# - EC2 instances (application servers)
# - RDS PostgreSQL (database)
# - EBS volumes (storage)
# - Data transfer costs
# - Backup storage
# - Monitoring and logging

Target: 40% cost reduction

Reserved Instances: 30-40% savings
Right-sizing: 20-30% savings
Auto-scaling: 15-25% savings
Managed services: Reduced operational overhead

Phase 2: AWS Foundation Setup

Network Architecture

VPC Design for Compliance:

# VPC Structure
VPC: 10.0.0.0/16
  ├── Public Subnets (2 AZs)
  │   ├── NAT Gateways
  │   └── Load Balancers
  ├── Application Subnets (2 AZs)
  │   └── EC2 Instances (Private)
  ├── Database Subnets (2 AZs)
  │   └── RDS Instances (Private)
  └── Management Subnets
      └── Bastion Hosts

Security Groups Configuration:

# Application Security Group
aws ec2 create-security-group \
  --group-name payment-app-sg \
  --description "Security group for payment application" \
  --vpc-id vpc-12345678

# Allow HTTPS from ALB
aws ec2 authorize-security-group-ingress \
  --group-id sg-12345678 \
  --protocol tcp \
  --port 8080 \
  --source-group sg-alb-12345678

# Database Security Group
aws ec2 create-security-group \
  --group-name payment-db-sg \
  --description "Security group for payment database" \
  --vpc-id vpc-12345678

# Allow PostgreSQL from application only
aws ec2 authorize-security-group-ingress \
  --group-id sg-db-12345678 \
  --protocol tcp \
  --port 5432 \
  --source-group sg-app-12345678

Identity and Access Management

IAM Roles for Least Privilege:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "rds-db:connect"
      ],
      "Resource": "arn:aws:rds-db:region:account:dbuser:db-instance-id/payment-app-user",
      "Condition": {
        "StringEquals": {
          "rds-db:database-name": "paymentdb"
        }
      }
    }
  ]
}

AWS Secrets Manager for credential management:

import boto3
import json

secrets_client = boto3.client('secretsmanager')

def get_database_credentials():
    """Retrieve database credentials from Secrets Manager"""
    response = secrets_client.get_secret_value(
        SecretId='payment-db-credentials'
    )
    return json.loads(response['SecretString'])

# Credentials automatically rotated
# No hardcoded secrets in code

Compliance Setup

AWS Config for compliance monitoring:

# Enable AWS Config
aws configservice put-configuration-recorder \
  --configuration-recorder name=default,roleArn=arn:aws:iam::account:role/config-role

# Configure rules for PCI-DSS compliance
aws configservice put-config-rule \
  --config-rule file://pci-dss-config-rule.json

CloudTrail for audit logging:

# Enable CloudTrail
aws cloudtrail create-trail \
  --name payment-audit-trail \
  --s3-bucket-name payment-audit-logs \
  --is-multi-region-trail \
  --enable-log-file-validation

Phase 3: Application Migration

Infrastructure as Code

AWS CloudFormation for reproducible infrastructure:

# application-stack.yaml
Resources:
  ApplicationLoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Type: application
      Scheme: internet-facing
      Subnets:
        - !Ref PublicSubnet1
        - !Ref PublicSubnet2
      SecurityGroups:
        - !Ref ALBSecurityGroup
      LoadBalancerAttributes:
        - Key: idle_timeout.timeout_seconds
          Value: '60'

  ApplicationTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Port: 8080
      Protocol: HTTP
      VpcId: !Ref VPC
      HealthCheckPath: /health
      HealthCheckIntervalSeconds: 30
      HealthCheckTimeoutSeconds: 5
      HealthyThresholdCount: 2
      UnhealthyThresholdCount: 3

  ApplicationAutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      MinSize: 2
      MaxSize: 10
      DesiredCapacity: 4
      VPCZoneIdentifier:
        - !Ref AppSubnet1
        - !Ref AppSubnet2
      LaunchTemplate:
        LaunchTemplateId: !Ref ApplicationLaunchTemplate
        Version: !GetAtt ApplicationLaunchTemplate.LatestVersionNumber
      TargetGroupARNs:
        - !Ref ApplicationTargetGroup
      HealthCheckType: ELB
      HealthCheckGracePeriod: 300

Application Deployment

AWS Elastic Beanstalk (for simpler deployment) or ECS/EKS (for containerized):

# Option 1: Elastic Beanstalk
eb init -p java-11 payment-app --region us-east-1
eb create payment-app-prod \
  --instance-type t3.large \
  --min-instances 2 \
  --max-instances 10 \
  --envvars SPRING_PROFILES_ACTIVE=production

# Option 2: ECS (if containerizing)
aws ecs create-service \
  --cluster payment-cluster \
  --service-name payment-service \
  --task-definition payment-app:1 \
  --desired-count 4 \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=[subnet-123,subnet-456],securityGroups=[sg-123],assignPublicIp=DISABLED}"

Blue-Green Deployment Setup

AWS CodeDeploy for zero-downtime deployments:

# appspec.yml
version: 0.0
Resources:
  - TargetService:
      Type: AWS::ECS::Service
      Properties:
        TaskDefinition: <TASK_DEFINITION>
        LoadBalancerInfo:
          ContainerName: "payment-app"
          ContainerPort: 8080
Hooks:
  - BeforeInstall: "validate.sh"
  - AfterInstall: "health-check.sh"
  - AfterAllowTestTraffic: "smoke-tests.sh"

Traffic Shifting Strategy:

# Gradually shift traffic from on-premise to AWS
# Week 1: 5% traffic to AWS
# Week 2: 10% traffic to AWS
# Week 3: 25% traffic to AWS
# Week 4: 50% traffic to AWS
# Week 5: 75% traffic to AWS
# Week 6: 100% traffic to AWS (cutover)

# Using Route 53 weighted routing
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123456789 \
  --change-batch file://traffic-shift-5-percent.json

Route 53 Weighted Routing Configuration:

{
  "Changes": [{
    "Action": "UPSERT",
    "ResourceRecordSet": {
      "Name": "payment.example.com",
      "Type": "A",
      "SetIdentifier": "on-premise",
      "Weight": 95,
      "AliasTarget": {
        "DNSName": "on-premise-lb.example.com",
        "EvaluateTargetHealth": true,
        "HostedZoneId": "Z123456789"
      }
    }
  }, {
    "Action": "UPSERT",
    "ResourceRecordSet": {
      "Name": "payment.example.com",
      "Type": "A",
      "SetIdentifier": "aws",
      "Weight": 5,
      "AliasTarget": {
        "DNSName": "aws-alb-123456789.us-east-1.elb.amazonaws.com",
        "EvaluateTargetHealth": true,
        "HostedZoneId": "Z123456789"
      }
    }
  }]
}

Phase 4: Database Migration (The Critical Phase)

Pre-Migration Preparation

Database Optimization:

-- Analyze database before migration
ANALYZE;

-- Check for unused indexes
SELECT schemaname, tablename, indexname, idx_scan, idx_tup_read, idx_tup_fetch
FROM pg_stat_user_indexes
WHERE idx_scan = 0
ORDER BY pg_relation_size(indexrelid) DESC;

-- Vacuum and reindex
VACUUM FULL ANALYZE;
REINDEX DATABASE paymentdb;

Estimate Migration Time:

# Test migration speed with sample data
# Typical DMS throughput: 100-500 MB/s
# 8TB = 8,192 GB = 8,388,608 MB
# At 200 MB/s: ~11.6 hours for initial load
# Plus ongoing replication for cutover

AWS DMS Setup

Step 1: Create Source Endpoint (On-Premise)

# Create source endpoint pointing to on-premise PostgreSQL
aws dms create-endpoint \
  --endpoint-identifier on-premise-postgres \
  --endpoint-type source \
  --engine-name postgres \
  --server-name on-premise-db.example.com \
  --port 5432 \
  --username postgres \
  --password 'secure-password' \
  --database-name paymentdb \
  --extra-connection-attributes "heartbeatFrequency=5;"

Step 2: Create Target Endpoint (AWS RDS)

# Create RDS PostgreSQL instance
aws rds create-db-instance \
  --db-instance-identifier payment-db-aws \
  --db-instance-class db.r5.4xlarge \
  --engine postgres \
  --engine-version 13.7 \
  --master-username postgres \
  --master-user-password 'secure-password' \
  --allocated-storage 10000 \
  --storage-type gp3 \
  --storage-encrypted \
  --vpc-security-group-ids sg-db-12345678 \
  --db-subnet-group-name payment-db-subnet-group \
  --backup-retention-period 35 \
  --multi-az \
  --publicly-accessible false

# Create target endpoint
aws dms create-endpoint \
  --endpoint-identifier aws-rds-postgres \
  --endpoint-type target \
  --engine-name postgres \
  --server-name payment-db-aws.xxxxx.us-east-1.rds.amazonaws.com \
  --port 5432 \
  --username postgres \
  --password 'secure-password' \
  --database-name paymentdb

Step 3: Create DMS Replication Instance

# Create replication instance (sized for 8TB migration)
aws dms create-replication-instance \
  --replication-instance-identifier payment-dms-instance \
  --replication-instance-class dms.r5.4xlarge \
  --allocated-storage 1000 \
  --vpc-security-group-ids sg-dms-12345678 \
  --replication-subnet-group-identifier payment-dms-subnet-group \
  --publicly-accessible false \
  --multi-az

Step 4: Create Migration Task

# Full load + CDC (Change Data Capture) for zero downtime
aws dms create-replication-task \
  --replication-task-identifier payment-db-migration \
  --source-endpoint-arn arn:aws:dms:region:account:endpoint:on-premise-postgres \
  --target-endpoint-arn arn:aws:dms:region:account:endpoint:aws-rds-postgres \
  --replication-instance-arn arn:aws:dms:region:account:rep:payment-dms-instance \
  --migration-type full-load-and-cdc \
  --table-mappings file://table-mappings.json \
  --replication-task-settings file://task-settings.json

Table Mappings Configuration:

{
  "rules": [
    {
      "rule-type": "selection",
      "rule-id": "1",
      "rule-name": "1",
      "object-locator": {
        "schema-name": "public",
        "table-name": "%"
      },
      "rule-action": "include"
    },
    {
      "rule-type": "transformation",
      "rule-id": "2",
      "rule-name": "2",
      "rule-action": "remove-column",
      "object-locator": {
        "schema-name": "public",
        "table-name": "audit_log",
        "column-name": "sensitive_data"
      }
    }
  ]
}

Task Settings for Performance:

{
  "TargetMetadata": {
    "TargetSchema": "public",
    "SupportLobs": true,
    "FullLobMode": false,
    "LobChunkSize": 64,
    "LimitedSizeLobMode": true,
    "LobMaxSize": 32,
    "InlineLobMaxSize": 0,
    "LoadMaxFileSize": 0,
    "ParallelLoadThreads": 0,
    "ParallelLoadBufferSize": 0,
    "BatchApplyEnabled": false,
    "TaskRecoveryTableEnabled": false
  },
  "FullLoadSettings": {
    "TargetTablePrepMode": "DO_NOTHING",
    "CreatePkAfterFullLoad": false,
    "StopTaskCachedChangesApplied": false,
    "StopTaskCachedChangesNotApplied": false,
    "MaxFullLoadSubTasks": 8,
    "TransactionConsistencyTimeout": 600,
    "CommitRate": 10000
  },
  "Logging": {
    "EnableLogging": true,
    "LogComponents": [
      {
        "Id": "SOURCE_UNLOAD",
        "Severity": "LOGGER_SEVERITY_DEFAULT"
      },
      {
        "Id": "TARGET_LOAD",
        "Severity": "LOGGER_SEVERITY_DEFAULT"
      }
    ]
  }
}

Migration Execution

Phase 1: Full Load (Initial Data Migration)

# Start replication task
aws dms start-replication-task \
  --replication-task-arn arn:aws:dms:region:account:task:payment-db-migration \
  --start-replication-task-type start-replication

# Monitor progress
aws dms describe-replication-tasks \
  --replication-task-arn arn:aws:dms:region:account:task:payment-db-migration

Monitor Migration Progress:

import boto3
import time

dms = boto3.client('dms')

def monitor_migration(task_arn):
    """Monitor DMS task progress"""
    while True:
        response = dms.describe_replication_tasks(
            ReplicationTaskArn=task_arn
        )
        task = response['ReplicationTasks'][0]

        status = task['Status']
        progress = task.get('ReplicationTaskStats', {})

        print(f"Status: {status}")
        print(f"Tables Loaded: {progress.get('TablesLoaded', 0)}")
        print(f"Tables Loading: {progress.get('TablesLoading', 0)}")
        print(f"Tables Queued: {progress.get('TablesQueued', 0)}")

        if status == 'stopped':
            break

        time.sleep(60)

# Run monitoring
monitor_migration('arn:aws:dms:region:account:task:payment-db-migration')

Phase 2: CDC (Change Data Capture) - Zero Downtime Window

Once full load completes, DMS automatically switches to CDC mode, capturing all changes from the source database in real-time.

Phase 3: Cutover

# 1. Stop application writes to on-premise database (brief window)
# 2. Wait for CDC to catch up (verify lag is near zero)
aws dms describe-replication-tasks \
  --replication-task-arn arn:aws:dms:region:account:task:payment-db-migration \
  --query 'ReplicationTasks[0].ReplicationTaskStats.CdcLatency'

# 3. Update application configuration to point to AWS RDS
# 4. Resume application writes to AWS RDS
# 5. Monitor for issues
# 6. If issues, rollback by pointing back to on-premise

Verification Script:

import psycopg2
import boto3

def verify_data_consistency():
    """Compare record counts between source and target"""

    # Connect to on-premise (source)
    source_conn = psycopg2.connect(
        host="on-premise-db.example.com",
        database="paymentdb",
        user="postgres",
        password="password"
    )

    # Connect to AWS RDS (target)
    target_conn = psycopg2.connect(
        host="payment-db-aws.xxxxx.us-east-1.rds.amazonaws.com",
        database="paymentdb",
        user="postgres",
        password="password"
    )

    source_cur = source_conn.cursor()
    target_cur = target_conn.cursor()

    # Compare key tables
    tables = ['transactions', 'customers', 'accounts']

    for table in tables:
        source_cur.execute(f"SELECT COUNT(*) FROM {table}")
        target_cur.execute(f"SELECT COUNT(*) FROM {table}")

        source_count = source_cur.fetchone()[0]
        target_count = target_cur.fetchone()[0]

        if source_count != target_count:
            print(f"WARNING: {table} count mismatch: {source_count} vs {target_count}")
        else:
            print(f"✓ {table}: {source_count} records match")

    source_conn.close()
    target_conn.close()

Phase 5: CI/CD Migration

Migrating from Jenkins to AWS

Option 1: AWS CodePipeline + CodeBuild

# buildspec.yml
version: 0.2
phases:
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
      - COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
      - IMAGE_TAG=${COMMIT_HASH:=latest}
  build:
    commands:
      - echo Build started on `date`
      - echo Building the Docker image...
      - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
  post_build:
    commands:
      - echo Build completed on `date`
      - echo Pushing the Docker image...
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
      - echo Writing image definitions file...
      - printf '[{"name":"payment-app","imageUri":"%s"}]' $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG > imagedefinitions.json
artifacts:
  files:
    - imagedefinitions.json

CodePipeline Pipeline Definition:

{
  "pipeline": {
    "name": "payment-app-pipeline",
    "roleArn": "arn:aws:iam::account:role/CodePipelineServiceRole",
    "artifactStore": {
      "type": "S3",
      "location": "payment-app-artifacts"
    },
    "stages": [
      {
        "name": "Source",
        "actions": [{
          "name": "SourceAction",
          "actionTypeId": {
            "category": "Source",
            "owner": "AWS",
            "provider": "CodeCommit",
            "version": "1"
          },
          "outputArtifacts": [{"name": "SourceOutput"}],
          "configuration": {
            "RepositoryName": "payment-app",
            "BranchName": "main"
          }
        }]
      },
      {
        "name": "Build",
        "actions": [{
          "name": "BuildAction",
          "actionTypeId": {
            "category": "Build",
            "owner": "AWS",
            "provider": "CodeBuild",
            "version": "1"
          },
          "inputArtifacts": [{"name": "SourceOutput"}],
          "outputArtifacts": [{"name": "BuildOutput"}],
          "configuration": {
            "ProjectName": "payment-app-build"
          }
        }]
      },
      {
        "name": "Deploy",
        "actions": [{
          "name": "DeployAction",
          "actionTypeId": {
            "category": "Deploy",
            "owner": "AWS",
            "provider": "ECS",
            "version": "1"
          },
          "inputArtifacts": [{"name": "BuildOutput"}],
          "configuration": {
            "ClusterName": "payment-cluster",
            "ServiceName": "payment-service"
          }
        }]
      }
    ]
  }
}

Option 2: AWS CodePipeline with GitHub Actions

# .github/workflows/deploy.yml
name: Deploy to AWS

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1

      - name: Build and push Docker image
        run: |
          docker build -t payment-app:${{ github.sha }} .
          aws ecr get-login-password | docker login --username AWS --password-stdin ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com
          docker push ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com/payment-app:${{ github.sha }}

      - name: Deploy to ECS
        run: |
          aws ecs update-service \
            --cluster payment-cluster \
            --service payment-service \
            --force-new-deployment

Security Scanning in CI/CD

AWS CodeBuild with Security Scanning:

# buildspec-with-security.yml
version: 0.2
phases:
  pre_build:
    commands:
      - echo Running security scans...
      - |
        # SAST scanning
        docker run --rm -v "$PWD:/src" returntocorp/semgrep semgrep --config=auto /src
      - |
        # Dependency scanning
        docker run --rm -v "$PWD:/src" owasp/dependency-check --scan /src --format JSON
  build:
    commands:
      - echo Building application...
      - mvn clean package
  post_build:
    commands:
      - echo Scanning container image...
      - |
        # Container image scanning
        aws ecr start-image-scan \
          --repository-name payment-app \
          --image-id imageTag=latest

Phase 6: Monitoring & Validation

CloudWatch Dashboards

Application Health Dashboard:

# Create CloudWatch dashboard
aws cloudwatch put-dashboard \
  --dashboard-name payment-app-health \
  --dashboard-body file://dashboard.json

Dashboard Configuration:

{
  "widgets": [
    {
      "type": "metric",
      "properties": {
        "metrics": [
          ["AWS/ApplicationELB", "RequestCount", {"stat": "Sum"}],
          [".", "TargetResponseTime", {"stat": "Average"}],
          [".", "HTTPCode_Target_5XX_Count", {"stat": "Sum"}]
        ],
        "period": 300,
        "stat": "Average",
        "region": "us-east-1",
        "title": "Application Load Balancer Metrics"
      }
    },
    {
      "type": "metric",
      "properties": {
        "metrics": [
          ["AWS/RDS", "DatabaseConnections", {"stat": "Average"}],
          [".", "CPUUtilization", {"stat": "Average"}],
          [".", "FreeableMemory", {"stat": "Average"}]
        ],
        "period": 300,
        "stat": "Average",
        "region": "us-east-1",
        "title": "RDS Database Metrics"
      }
    }
  ]
}

Validation Tests

Automated Validation Script:

import boto3
import requests
import time

def validate_migration():
    """Comprehensive validation of migrated system"""

    results = {
        "application_health": False,
        "database_connectivity": False,
        "transaction_processing": False,
        "performance_metrics": False
    }

    # 1. Application Health Check
    try:
        response = requests.get("https://payment.example.com/health", timeout=5)
        if response.status_code == 200:
            results["application_health"] = True
    except Exception as e:
        print(f"Application health check failed: {e}")

    # 2. Database Connectivity
    rds = boto3.client('rds')
    try:
        response = rds.describe_db_instances(
            DBInstanceIdentifier='payment-db-aws'
        )
        if response['DBInstances'][0]['DBInstanceStatus'] == 'available':
            results["database_connectivity"] = True
    except Exception as e:
        print(f"Database connectivity check failed: {e}")

    # 3. Transaction Processing Test
    # Send test transaction and verify
    try:
        test_response = requests.post(
            "https://payment.example.com/api/transactions",
            json={"amount": 1.00, "currency": "USD"},
            timeout=10
        )
        if test_response.status_code == 200:
            results["transaction_processing"] = True
    except Exception as e:
        print(f"Transaction processing test failed: {e}")

    # 4. Performance Metrics
    cloudwatch = boto3.client('cloudwatch')
    end_time = time.time()
    start_time = end_time - 3600  # Last hour

    try:
        response = cloudwatch.get_metric_statistics(
            Namespace='AWS/ApplicationELB',
            MetricName='TargetResponseTime',
            Dimensions=[
                {'Name': 'LoadBalancer', 'Value': 'app/payment-alb/123456789'}
            ],
            StartTime=start_time,
            EndTime=end_time,
            Period=300,
            Statistics=['Average']
        )

        if response['Datapoints']:
            avg_response_time = sum([d['Average'] for d in response['Datapoints']]) / len(response['Datapoints'])
            if avg_response_time < 0.5:  # Less than 500ms
                results["performance_metrics"] = True
    except Exception as e:
        print(f"Performance metrics check failed: {e}")

    return results

# Run validation
validation_results = validate_migration()
print("Migration Validation Results:")
for check, passed in validation_results.items():
    status = "✓ PASS" if passed else "✗ FAIL"
    print(f"{check}: {status}")

Phase 7: Cost Optimization

Reserved Instances Strategy

# Purchase Reserved Instances for baseline capacity
aws ec2 purchase-reserved-instances-offering \
  --reserved-instances-offering-id 12345678-1234-1234-1234-123456789012 \
  --instance-count 4

# RDS Reserved Instances
aws rds purchase-reserved-db-instances-offering \
  --reserved-db-instances-offering-id 12345678-1234-1234-1234-123456789012 \
  --db-instance-count 1

Auto-Scaling for Cost Efficiency

{
  "TargetTrackingScalingPolicies": [
    {
      "TargetValue": 70.0,
      "PredefinedMetricSpecification": {
        "PredefinedMetricType": "ECSServiceAverageCPUUtilization"
      },
      "ScaleInCooldown": 300,
      "ScaleOutCooldown": 60
    }
  ]
}

Cost Monitoring

AWS Cost Explorer API:

import boto3
from datetime import datetime, timedelta

ce = boto3.client('ce')

def get_daily_costs():
    """Get daily AWS costs"""
    end_date = datetime.now().strftime('%Y-%m-%d')
    start_date = (datetime.now() - timedelta(days=30)).strftime('%Y-%m-%d')

    response = ce.get_cost_and_usage(
        TimePeriod={
            'Start': start_date,
            'End': end_date
        },
        Granularity='DAILY',
        Metrics=['UnblendedCost'],
        GroupBy=[
            {'Type': 'DIMENSION', 'Key': 'SERVICE'}
        ]
    )

    return response

# Monitor costs
costs = get_daily_costs()
for result in costs['ResultsByTime']:
    print(f"Date: {result['TimePeriod']['Start']}")
    for group in result['Groups']:
        service = group['Keys'][0]
        amount = group['Metrics']['UnblendedCost']['Amount']
        print(f"  {service}: ${amount}")

Security & Compliance

Encryption

Encryption at Rest:

# RDS encryption
aws rds create-db-instance \
  --db-instance-identifier payment-db-aws \
  --storage-encrypted \
  --kms-key-id arn:aws:kms:region:account:key/key-id

# EBS encryption
aws ec2 create-volume \
  --size 100 \
  --volume-type gp3 \
  --encrypted \
  --kms-key-id arn:aws:kms:region:account:key/key-id

Encryption in Transit:

# Application Load Balancer with SSL/TLS
Resources:
  ApplicationLoadBalancer:
    Properties:
      Listeners:
        - Protocol: HTTPS
          Port: 443
          Certificates:
            - CertificateArn: arn:aws:acm:region:account:certificate/cert-id
          DefaultActions:
            - Type: forward
              TargetGroupArn: !Ref ApplicationTargetGroup

PCI-DSS Compliance

AWS PCI-DSS Compliance Checklist:

Network Segmentation: VPC with private subnets
Encryption: At rest and in transit
Access Control: IAM roles, security groups
Monitoring: CloudWatch, CloudTrail, GuardDuty
Vulnerability Management: Security scanning in CI/CD
Audit Logging: CloudTrail, VPC Flow Logs

AWS GuardDuty for threat detection:

# Enable GuardDuty
aws guardduty create-detector \
  --enable \
  --finding-publishing-frequency FIFTEEN_MINUTES

GDPR Compliance

Data Residency:

# Ensure RDS is in correct region
aws rds describe-db-instances \
  --db-instance-identifier payment-db-aws \
  --query 'DBInstances[0].AvailabilityZone'

Data Retention Policies:

import boto3

def configure_s3_lifecycle():
    """Configure S3 lifecycle for GDPR data retention"""
    s3 = boto3.client('s3')

    s3.put_bucket_lifecycle_configuration(
        Bucket='payment-audit-logs',
        LifecycleConfiguration={
            'Rules': [{
                'Id': 'DeleteAfter7Years',
                'Status': 'Enabled',
                'Expiration': {
                    'Days': 2555  # 7 years
                }
            }]
        }
    )

Rollback Strategy

Database Rollback

# If issues occur, point application back to on-premise database
# DMS can reverse replication if needed
aws dms create-replication-task \
  --replication-task-identifier payment-db-rollback \
  --source-endpoint-arn arn:aws:dms:region:account:endpoint:aws-rds-postgres \
  --target-endpoint-arn arn:aws:dms:region:account:endpoint:on-premise-postgres \
  --replication-instance-arn arn:aws:dms:region:account:rep:payment-dms-instance \
  --migration-type cdc-only \
  --table-mappings file://table-mappings.json

Application Rollback

# Route traffic back to on-premise using Route 53
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123456789 \
  --change-batch file://rollback-traffic.json

Rollback Decision Matrix:

IF error_rate > 1% OR response_time > 2s:
    → Rollback to on-premise
ELSE IF data_integrity_issues:
    → Rollback to on-premise
ELSE IF security_incident:
    → Rollback to on-premise
ELSE:
    → Continue monitoring

Success Metrics

Migration Success Criteria

Zero Downtime: No customer-facing downtime during migration
Data Integrity: 100% data consistency between source and target
Performance: Response times ≤ on-premise baseline
Cost Reduction: 40% cost savings achieved
Uptime: 99.99% availability (vs. 99.9% before)
Compliance: PCI-DSS and GDPR requirements met

Key Performance Indicators

# Migration KPIs
migration_kpis = {
    "downtime_minutes": 0,  # Target: 0
    "data_consistency_percent": 100,  # Target: 100%
    "cost_reduction_percent": 40,  # Target: 40%
    "uptime_percent": 99.99,  # Target: 99.99%
    "migration_duration_days": 180,  # Target: < 180 days
    "rollback_incidents": 0  # Target: 0
}

Lessons Learned & Best Practices

Critical Success Factors

Thorough Planning: Spend adequate time in assessment phase
Incremental Migration: Don't try to migrate everything at once
Comprehensive Testing: Test every component before cutover
Monitoring: Real-time visibility is essential
Rollback Readiness: Always have a rollback plan
Team Training: Ensure team is trained on AWS services

Common Pitfalls to Avoid

Underestimating database migration time
Not testing rollback procedures
Ignoring network latency between on-premise and AWS
Forgetting to update DNS TTL before cutover
Not monitoring costs during migration
Skipping security and compliance validation

Conclusion

Migrating a fintech platform to AWS with zero downtime is complex but achievable with the right strategy. Key takeaways:

AWS DMS enables zero-downtime database migration for large databases
Blue-Green deployment with gradual traffic shifting minimizes risk
Infrastructure as Code ensures reproducibility and rollback capability
Comprehensive monitoring provides visibility throughout migration
Cost optimization through Reserved Instances and auto-scaling achieves savings targets

The migration journey transforms not just infrastructure, but also processes, security posture, and operational efficiency. With careful planning and execution, you can achieve zero downtime, maintain compliance, and realize significant cost savings.

Additional Resources

Responding to a Critical Production Incident: A Fintech Case Study with AWS

Maureen Chebet — Wed, 26 Nov 2025 13:41:32 +0000

The 2:30 AM Wake-Up Call

Picture this: It's 2:30 AM, and you receive an alert that your payment processing system is experiencing:

45% of payment transactions failing
Database connection pool exhausted
Response times increased from 200ms to 15 seconds (75x degradation!)
Customer complaints flooding social media

The system processes over 1 million transactions daily and handles sensitive financial data. The last deployment was 6 hours ago. This is a critical production incident that requires immediate, systematic response.

In this article, I'll walk through how to handle this scenario using AWS services, covering immediate triage, diagnosis, resolution, and post-incident improvements.

Understanding the Incident

Before diving into solutions, let's understand what we're dealing with:

Scale: 1M+ transactions daily
Impact: Financial losses, customer trust, regulatory compliance
Urgency: Every minute counts
Complexity: Multiple potential root causes (deployment, capacity, database, code)

In fintech, we can't afford ad-hoc fixes. Everything must be systematic, documented, and compliant.

Phase 1: Immediate Response and Triage (First 15 Minutes)

Step 1: Establish Incident War Room

AWS Services Used:

Amazon Chime or Slack (integrated with AWS) for real-time communication
AWS Systems Manager Session Manager for secure access without SSH keys

# Quick access to EC2 instances for diagnosis
aws ssm start-session --target i-1234567890abcdef0

Step 2: Gather Critical Metrics

Amazon CloudWatch is your first line of defense. Immediately check:

# Check database connection pool metrics
aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name DatabaseConnections \
  --dimensions Name=DBInstanceIdentifier,Value=payment-db \
  --start-time 2024-01-15T02:00:00Z \
  --end-time 2024-01-15T02:30:00Z \
  --period 300 \
  --statistics Average,Maximum

Key CloudWatch Dashboards to Check:

RDS Performance Insights - Database connection pool, query performance
Application Load Balancer - Request count, response times, error rates
ECS/EC2 Metrics - CPU, memory, network utilization
Custom Application Metrics - Transaction success rate, error rates

Step 3: Check Recent Changes

AWS Config and CloudTrail help identify what changed:

# Check recent API calls that might have caused the issue
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=ModifyDBInstance \
  --start-time 2024-01-15T20:00:00Z

Step 4: Immediate Stabilization Actions

If database connection pool is exhausted:

Check RDS Performance Insights (Real-time database monitoring):
- Navigate to RDS Console → Performance Insights
- Identify top SQL statements consuming connections
- Look for long-running queries or lock contention
Temporary Connection Pool Increase (with caution):

   -- Check current connection limit
   SHOW max_connections;

   -- In RDS, you can modify parameter group (but this requires restart)
   -- Better: Check for connection leaks in application

Kill Long-Running Queries (only if safe):

   -- Identify problematic queries
   SELECT pid, now() - pg_stat_activity.query_start AS duration, query 
   FROM pg_stat_activity 
   WHERE state = 'active' AND now() - pg_stat_activity.query_start > interval '5 minutes';

   -- Kill specific query (ONLY if not critical transaction)
   SELECT pg_terminate_backend(pid);

⚠️ Critical Warning: In payment systems, killing connections can cause data integrity issues. Always verify the query is not processing a critical transaction.

Phase 2: Systematic Diagnosis (15-60 Minutes)

Infrastructure Layer Diagnosis

Amazon CloudWatch Metrics:

# Check EC2/ECS resource utilization
aws cloudwatch get-metric-statistics \
  --namespace AWS/ECS \
  --metric-name CPUUtilization \
  --dimensions Name=ServiceName,Value=payment-service \
  --start-time 2024-01-15T02:00:00Z \
  --end-time 2024-01-15T02:30:00Z \
  --period 300 \
  --statistics Average,Maximum

Key Metrics to Check:

CPU utilization > 80%?
Memory utilization > 90%?
Network bandwidth saturated?
Disk I/O wait times high?

Database Layer Deep Dive

Amazon RDS Performance Insights provides real-time analysis:

Top SQL Statements - Which queries are consuming most time?
Wait Events - What is the database waiting on? (I/O, locks, CPU)
Database Load - Average Active Sessions (AAS) vs. capacity

RDS Enhanced Monitoring (if enabled):

Provides OS-level metrics every 60 seconds
Check for I/O bottlenecks, memory pressure

Query Performance Analysis:

-- Check for slow queries
SELECT 
  pid,
  now() - pg_stat_activity.query_start AS duration,
  query,
  state,
  wait_event_type,
  wait_event
FROM pg_stat_activity 
WHERE state != 'idle'
ORDER BY duration DESC;

Application Layer Analysis

AWS X-Ray for distributed tracing:

# If X-Ray is integrated, check traces for slow requests
from aws_xray_sdk.core import xray_recorder

@xray_recorder.capture('process_payment')
def process_payment(transaction):
    # Your payment processing logic
    pass

CloudWatch Logs Insights for log analysis:

-- Query application logs for errors
fields @timestamp, @message
| filter @message like /ERROR/
| filter @message like /connection|pool|timeout/
| sort @timestamp desc
| limit 100

Common Issues to Look For:

Connection leaks (connections not being closed)
Memory leaks (gradual memory increase)
Thread pool exhaustion
Cascading failures (circuit breakers tripped)

Network Layer

VPC Flow Logs (if enabled):

Check for network latency between application and database
Look for packet loss or connection resets

CloudWatch Network Metrics:

Check ALB target response times
Verify no network partitions

Phase 3: Resolution Strategy

The resolution depends on the root cause. Here are common scenarios:

Scenario A: Code Issue from Recent Deployment

If the deployment 6 hours ago introduced a bug:

Immediate Rollback using AWS CodeDeploy:

# Rollback to previous deployment
aws deploy create-deployment \
  --application-name payment-app \
  --deployment-group-name production \
  --revision revisionType=AppSpecContent,appSpecContent='{
    "version": 0.0,
    "Resources": [{
      "TargetService": {
        "Type": "AWS::ECS::Service",
        "Properties": {
          "TaskDefinition": "arn:aws:ecs:region:account:task-definition/previous-version"
        }
      }
    }]
  }'

Database Migration Rollback (if applicable):
- If deployment included schema changes, check if they're reversible
- Use AWS Database Migration Service (DMS) for complex rollbacks
- Or manually reverse migrations if safe
Validation:
- Monitor CloudWatch metrics for improvement
- Verify transaction success rate returns to normal
- Check database connection pool utilization

Scenario B: Database Performance Issue

If queries are slow due to missing indexes or stale statistics:

Add Missing Indexes (carefully - can lock tables):

-- Analyze query execution plan first
EXPLAIN ANALYZE SELECT * FROM transactions WHERE customer_id = '123';

-- If missing index, add it (during low traffic if possible)
CREATE INDEX CONCURRENTLY idx_transactions_customer_id 
ON transactions(customer_id);

Update Query Planner Statistics:

ANALYZE transactions;

Enable RDS Query Insights for automatic query optimization recommendations

Scenario C: Capacity Issue

If system simply ran out of capacity:

Auto-Scaling Application Tier:

# Manually trigger scaling if auto-scaling is slow
aws application-autoscaling register-scalable-target \
  --service-namespace ecs \
  --scalable-dimension ecs:service:DesiredCount \
  --resource-id service/payment-cluster/payment-service \
  --min-capacity 10 \
  --max-capacity 50

Scale RDS Instance (if using Provisioned IOPS):

# Modify RDS instance (requires downtime for some changes)
aws rds modify-db-instance \
  --db-instance-identifier payment-db \
  --db-instance-class db.r5.2xlarge \
  --apply-immediately

Enable Read Replicas to offload read queries:

# Create read replica (if not already exists)
aws rds create-db-instance-read-replica \
  --db-instance-identifier payment-db-read-replica \
  --source-db-instance-identifier payment-db

Implement Caching with Amazon ElastiCache:

import redis
import json

# Cache frequently accessed data
cache = redis.Redis(
    host='payment-cache.xxxxx.cache.amazonaws.com',
    port=6379,
    decode_responses=True
)

def get_customer_data(customer_id):
    # Check cache first
    cached = cache.get(f"customer:{customer_id}")
    if cached:
        return json.loads(cached)

    # If not in cache, query database
    data = query_database(customer_id)
    cache.setex(f"customer:{customer_id}", 300, json.dumps(data))
    return data

General Resolution Principles

Fix Root Cause, Not Symptoms
- Don't just increase connection pool size if connections aren't being released
- Identify why connections are stuck
Incremental Changes with Validation
- Make one change at a time
- Validate each change before proceeding
Rollback Capability
- Every change must be reversible
- Use infrastructure as code (CloudFormation/Terraform) for easy rollback
Data Integrity First
- Never compromise transaction atomicity
- Use database transactions for multi-step fixes

Phase 4: Communication Plan

Internal Communication

AWS Services:

Amazon Chime or Slack (with AWS Chatbot integration)
Amazon SNS for automated alerts

Communication Template:

INCIDENT UPDATE - Payment System

Status: ACTIVE
Time: 02:45 AM
Impact: 45% transaction failure rate
Root Cause: [Under investigation / Identified: connection pool exhaustion]
Actions: [Current actions being taken]
ETA: [Conservative estimate]
Next Update: 03:00 AM

External Communication

AWS Services:

Amazon Route 53 Health Checks for status page
CloudWatch Alarms → SNS → Status page API

Status Page Message:

"We're aware of an issue affecting some payment transactions and are working to resolve it. We'll provide updates every 30 minutes. Thank you for your patience."

Automated Status Updates

import boto3

sns = boto3.client('sns')

def send_status_update(message):
    sns.publish(
        TopicArn='arn:aws:sns:region:account:incident-updates',
        Message=message,
        Subject='Payment System Incident Update'
    )

Phase 5: Post-Incident Improvements

1. Enhanced Monitoring and Alerting

CloudWatch Alarms for Proactive Detection:

# Create alarm for database connection pool
aws cloudwatch put-metric-alarm \
  --alarm-name payment-db-connection-pool-warning \
  --alarm-description "Alert when connection pool exceeds 80%" \
  --metric-name DatabaseConnections \
  --namespace AWS/RDS \
  --statistic Average \
  --period 300 \
  --evaluation-periods 2 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=DBInstanceIdentifier,Value=payment-db

CloudWatch Composite Alarms for complex conditions:

{
  "AlarmName": "payment-system-health",
  "AlarmRule": "ALARM(payment-db-connections) OR ALARM(payment-error-rate) OR ALARM(payment-response-time)"
}

AWS X-Ray Service Map for end-to-end visibility:

Automatically maps service dependencies
Identifies bottlenecks in distributed systems

2. Database Resilience Improvements

RDS Automated Backups:

Ensure point-in-time recovery is enabled
Test restore procedures regularly

RDS Multi-AZ Deployment:

# Enable Multi-AZ for high availability
aws rds modify-db-instance \
  --db-instance-identifier payment-db \
  --multi-az \
  --apply-immediately

Connection Pool Monitoring:

import boto3
from datetime import datetime

cloudwatch = boto3.client('cloudwatch')

def monitor_connection_pool():
    # Custom metric for application-level connection pool
    cloudwatch.put_metric_data(
        Namespace='PaymentApp/ConnectionPool',
        MetricData=[{
            'MetricName': 'ActiveConnections',
            'Value': get_active_connections(),
            'Timestamp': datetime.utcnow(),
            'Unit': 'Count'
        }]
    )

3. Deployment Safety

AWS CodeDeploy with Blue/Green Deployment:

# appspec.yml
version: 0.0
Resources:
  - TargetService:
      Type: AWS::ECS::Service
      Properties:
        TaskDefinition: <TASK_DEFINITION>
        LoadBalancerInfo:
          ContainerName: "payment-app"
          ContainerPort: 8080
        PlatformVersion: "LATEST"
Hooks:
  - BeforeInstall: "validate.sh"
  - AfterInstall: "health-check.sh"

Canary Deployments with AWS App Mesh:

Gradually route traffic to new version
Automatic rollback on error threshold

Pre-Deployment Database Impact Analysis:

Use AWS DMS to test migrations on copy of production data
Run EXPLAIN ANALYZE on all new queries

4. Auto-Scaling Configuration

Application Auto-Scaling:

{
  "TargetTrackingScalingPolicies": [{
    "TargetValue": 70.0,
    "PredefinedMetricSpecification": {
      "PredefinedMetricType": "ECSServiceAverageCPUUtilization"
    },
    "ScaleInCooldown": 300,
    "ScaleOutCooldown": 60
  }]
}

Database Auto-Scaling (Aurora Serverless v2):

# Migrate to Aurora Serverless v2 for automatic scaling
aws rds create-db-cluster \
  --db-cluster-identifier payment-aurora \
  --engine aurora-postgresql \
  --serverless-v2-scaling-configuration MinCapacity=2,MaxCapacity=16

5. Incident Response Automation

AWS Systems Manager Automation Documents:

# incident-response-automation.yaml
schemaVersion: '0.3'
description: Automated incident response for connection pool exhaustion
assumeRole: 'arn:aws:iam::account:role/IncidentResponseRole'
mainSteps:
  - name: gatherMetrics
    action: 'aws:executeScript'
    inputs:
      Runtime: python3.8
      Script: |
        import boto3
        cloudwatch = boto3.client('cloudwatch')
        # Gather metrics and log to S3
  - name: scaleApplication
    action: 'aws:executeAwsApi'
    inputs:
      Service: ecs
      Api: UpdateService
      ServiceName: payment-service
      DesiredCount: 20

AWS Lambda for Automated Responses:

import boto3
import json

def lambda_handler(event, context):
    """
    Automated response to CloudWatch alarm
    """
    alarm_name = event['AlarmName']

    if 'connection-pool' in alarm_name.lower():
        # Scale application
        ecs = boto3.client('ecs')
        ecs.update_service(
            cluster='payment-cluster',
            service='payment-service',
            desiredCount=20
        )

        # Send notification
        sns = boto3.client('sns')
        sns.publish(
            TopicArn='arn:aws:sns:region:account:incident-response',
            Message=f'Automated scaling triggered for {alarm_name}'
        )

    return {'statusCode': 200}

Key Takeaways

Monitoring is Critical: CloudWatch, X-Ray, and RDS Performance Insights provide the visibility needed for rapid diagnosis
Automation Saves Time: Automated scaling, rollbacks, and incident response reduce MTTR
Systematic Approach: Follow a structured diagnosis methodology rather than random fixes
Document Everything: CloudTrail and CloudWatch Logs provide audit trails for compliance
Learn and Improve: Use post-incident analysis to prevent recurrence

AWS Services Checklist for Incident Response

✅ CloudWatch - Metrics, logs, alarms, dashboards
✅ RDS Performance Insights - Real-time database analysis
✅ X-Ray - Distributed tracing
✅ CloudTrail - Audit log of API calls
✅ Systems Manager - Secure access and automation
✅ CodeDeploy - Safe rollbacks
✅ ElastiCache - Caching to reduce database load
✅ SNS - Incident notifications
✅ Lambda - Automated responses
✅ ECS Auto Scaling - Dynamic capacity adjustment

Conclusion

Handling a critical production incident in a fintech environment requires a balance of speed, thoroughness, and compliance. AWS provides a comprehensive toolkit for monitoring, diagnosis, resolution, and prevention.

The key is to:

Detect early with proactive monitoring
Diagnose systematically using AWS observability tools
Resolve safely with proper rollback capabilities
Communicate transparently with stakeholders
Improve continuously based on learnings

Remember: In fintech, every incident is an opportunity to make the system more resilient. Use AWS services to turn reactive firefighting into proactive system reliability.

Additional Resources

Tags: #aws #devops #incident-response #fintech #cloudwatch #rds #production #reliability

Port Forwarding to Amazon MQ

Maureen Chebet — Mon, 26 Feb 2024 09:31:48 +0000

Overview
RabbitMQ is a powerful message broker widely used in modern applications for handling message queues. However, ensuring the security of your RabbitMQ instance is crucial, especially in a production environment. One way to enhance security is by deploying RabbitMQ in a private subnet. In this article, we'll explore how to set up a private RabbitMQ instance on AWS and enable access to it securely using port forwarding through AWS Systems Manager (SSM) Agent.

Prerequisites

An AWS account
Amazon MQ.

Install and Configure SSM Agent

Depending on your OS, download ssm agent here

Configure awscli using either of the following:
a) aws configure

b) export sso credentials

Start port forwarding session:
Linux & MacOS

aws ssm start-session \
    --target instance-id \
    --document-name AWS-StartPortForwardingSessionToRemoteHost \
    --parameters '{"host":["my.example.mq.eu-west-1.amazonaws.com"],"portNumber":["443"], "localPortNumber":["4000"]}'

Windows

aws ssm start-session ^
    --target instance-id ^
    --document-name AWS-StartPortForwardingSessionToRemoteHost ^
    --parameters host="my.example.us-east-2.rds.amazonaws.com",portNumber="443",localPortNumber="4000"

You can use a different port other than 4000

In a separate tab open /etc/hosts with vim or nano and the following line.

127.0.0.1          my.example.mq.eu-west-1.amazonaws.com

Note: Please change my.example.mq.eu-west-1.amazonaws.com. You can get it as shown:

Navigate to your browser:

Conclusion
You are now able to access your broker outside the VPC from your local browser.

Configure Jenkins using Ansible on Amazon Linux 2023

Maureen Chebet — Sat, 24 Feb 2024 12:29:06 +0000

Overview

Jenkins is a powerful automation server that can be used for continuous integration and continuous delivery (CI/CD) pipelines. Ansible is a popular configuration management tool that allows you to automate the provisioning and configuration of infrastructure. In this article, we'll explore how to use Ansible to configure Jenkins on Amazon Linux in the year 2023.

Prerequisites
Before we begin, make sure you have basic knowledge of Ansible and Jenkins and the following prerequisites:

An AWS account
An Amazon Linux instance up and running. (Amazon Linux 2023)
SSH access to your Amazon Linux instance.

Note: Make sure to open the necessary ports (e.g., port 8080 for Jenkins) in the security group settings.

Step 2: Install Ansible

Connect into the ec2 instance either using ssh or session manager the run:
sudo yum update && yum install ansible

Check ansible version

ansible --version

We will install Jenkins using Ansible playbook.

cat > setup_jenkins.yml<< EOF 
---
- hosts: localhost
  connection: local
  vars:
    java_packages:
      - java-17-amazon-corretto-devel
    jenkins_packages:
      - jenkins  
  tasks:
    - name: Download Jenkins repository file
      get_url:
        url: https://pkg.jenkins.io/redhat-stable/jenkins.repo
        dest: /etc/yum.repos.d/jenkins.repo
      become: true

    - name: Import Jenkins-CI key
      shell: rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key
      become: true

    - name: Install Java
      yum:
        name: "{{ java_packages }}"
        state: present
      become: true

    - name: Install Jenkins
      yum:
        name: "{{ jenkins_packages }}"
        state: present
      become: true

    - name: Start Jenkins service
      service:
        name: jenkins
        state: started
      become: true
EOF

Then run:
ansible-playbook setup_jenkins.yml
On success you should be able to see below output

Now go to your browser at http://:8080/
You should find your password as shown:

Get the password:
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
Unlock Jenkins: Follow the on-screen instructions to unlock Jenkins and complete the initial setup.
Configure Jenkins: Customize Jenkins settings and install plugins as needed to suit your requirements.

Conclusion
By following these steps, you have successfully configured Jenkins using Ansible on an Amazon Linux 2023 instance. You can now use Jenkins to automate your CI/CD workflows and streamline your development processes.
If you are experiencing an issue contact me on LinkedIn

Transfer Existing Domain To Another AWS Account

Maureen Chebet — Wed, 14 Jun 2023 09:00:56 +0000

Prerequisites

Correct IAM Permission:

The user is assigned the AdministratorAccess managed policy.
The user is assigned the AmazonRoute53DomainsFullAccess managed policy.
The user is assigned the AmazonRoute53FullAccess managed policy.
The user is assigned the PowerUserAccess managed policy.
The user has permission to perform all the following actions: TransferDomains, DisableDomainTransferLock, and RetrieveDomainAuthCode.
Registered domain in AWS

Step 1: To transfer a domain to a different AWS account

Sign in to AWS by using the AWS account that the domain is currently registered to.
Open the Route 53 console.
In the navigation pane, choose Registered domains.
Choose the name of the domain that you want to transfer to another AWS account.
Above the Details section, in the Transfer out dropdown, choose Transfer to another AWS account.
On the Transfer to another AWS account dialog, enter the destination account ID. You can get this ID from the destination AWS account owner.
Choose Confirm.
On the Generate password dialog, copy the password, and forward it to the receiving AWS account owner. On the Requests page, the Status for the domain will display In progress, and the Type will display Internal transfer out.

Step 2: To accept a domain transfer from a different AWS account

Sign into AWS by using the AWS account that is receiving the domain.
Open the Route 53 console.
In the navigation pane, choose Requests.
On the Requests page, select the radio button next to the domain name you are transferring from another AWS account. If the domain is ready to be transferred the Status is Action required and the Type is Internal transfer of domain in.
You have three days to accept the request. If the transfer isn’t accepted in three days, the transfer request is canceled.
In the Action dropdown, choose Accept.
You can also choose Reject to cancel the transfer process.
If you accepted, on the Transfer domain to your account ** page, in the Password section, enter the password you received from the originating account owner.
Accept the terms and conditions, and choose **Next.
Navigate to the Requests page to monitor the transfer status and other steps to complete.
After the transfer completes, you can update the contact information. For more information, see Updating contact information and ownership for a domain.

To migrate the existing hosted zone to the new account, see next article.

Configure Autoscaler on EKS

Maureen Chebet — Mon, 19 Dec 2022 09:44:51 +0000

Overview
Amazon EKS is a fully managed service, kubernetes allows the scalability to adjust to sudden traffic changes seamlessly. Cluster Autoscaler automatically adjusts the number of nodes in your cluster when pods fail or are rescheduled onto other nodes. The Cluster Autoscaler is typically installed as a Deployment in your cluster. It uses leader election to ensure high availability, but scaling is done by only one replica at a time.

Prerequisites

An existing Amazon EKS cluster
An existing IAM OIDC provider for your cluster. Determine whether you have one or need to create one.
Node groups with Auto Scaling groups tags

Create IAM policy and role
Create an IAM policy. Paste the following content into a file named cluster-autoscaler-policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:ResourceTag/k8s.io/cluster-autoscaler/my-cluster": "owned"
                }
            }
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeAutoScalingGroups",
                "ec2:DescribeLaunchTemplateVersions",
                "autoscaling:DescribeTags",
                "autoscaling:DescribeLaunchConfigurations"
            ],
            "Resource": "*"
        }
    ]
}

then create the policy with the following command. You can change the value for policy-name.

aws iam create-policy \
--policy-name AmazonEKSClusterAutoscalerPolicy \
--policy-document file://cluster-autoscaler-policy.json

then attach the above policy as shown:

eksctl create iamserviceaccount \
  --cluster=<YOUR CLUSTER NAME> \
  --namespace=kube-system \
  --name=cluster-autoscaler \
--attach-policy-arn=arn:aws:iam::ACCOUNT_ID:policy/ClusterAutoscaler-autoDiscovery \
  --override-existing-serviceaccounts \
  --approve

Deploy cluster autoscaler
To deploy the Cluster Autoscaler:

Download the Cluster Autoscaler YAML file.

wget curl -o cluster-autoscaler-autodiscover.yaml https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml

Modify the YAML file and replace with your cluster name. Also consider replacing the cpu and memory values as determined by your environment.

Apply the YAML file to your cluster.

kubectl apply -f cluster-autoscaler-autodiscover.yaml

Annotate the cluster-autoscaler service account with the ARN of the IAM role that you created previously. Replace the example values with your own values.

kubectl annotate serviceaccount cluster-autoscaler \
 -n kube-system \  eks.amazonaws.com/role-arn=arn:aws:iam::ACCOUNT_ID:role/AmazonEKSClusterAutoscalerRole

Patch the deployment to add the cluster-autoscaler.kubernetes.io/safe-to-evict annotation to the Cluster

kubectl annotate serviceaccount cluster-autoscaler \
  -n kube-system \
  eks.amazonaws.com/role-arn=arn:aws:iam::ACCOUNT_ID:role/AmazonEKSClusterAutoscalerRole

Edit the Cluster Autoscaler deployment with the following command.

kubectl -n kube-system edit deployment.apps/cluster-autoscaler

Add the following lines as shown

--balance-similar-node-groups --skip-nodes-with-system-pods=false

Find the latest Cluster Autoscaler version here that matches the Kubernetes major and minor version of your cluster.

Set the Cluster Autoscaler image tag to the version that you recorded in the previous step with the following command.

kubectl set image deployment cluster-autoscaler \
  -n kube-system \
  cluster-autoscaler=k8s.gcr.io/autoscaling/cluster-autoscaler:v1.23.0

View your Cluster Autoscaler logs

kubectl -n kube-system logs -f deployment.apps/cluster-autoscaler

New to AWS? Let’s get you started.

Maureen Chebet — Tue, 05 Jul 2022 17:43:42 +0000

If you are an enthusiast of technology or into business, then you have definitely heard of Amazon Web Services(AWS). In terms of business, AWS business model is very revolutionary when it comes to lowering other businesses overhead for server costs by hosting almost all of the infrastructure on Amazon Data Centers.

AWS is the top cloud provider and it offers a wide range of remote cloud services for application development like analytics, blockchain, AI etc. and can assist individuals and organizations in the development and sustainable growth of any application. Following are the key building blocks of the AWS environment as shown below:

AWS VPC

A VPC is a virtual network specific to you within AWS for you to hold all your AWS services. It is a logical data center in AWS and will have gateways, route tables, network access control lists (ACL), subnets and security groups. When you open up a service within a public cloud, it is effectively open to the world and can be at risk to attacks from the internet. In order to lock your instances down and secure them against attacks from the outside, you lock them within a VPC. The VPC restricts what sort of traffic, IP addresses and also the users that can access your instances.

AWS Elastic Compute Cloud(EC2)

This is a virtual server that helps you run numerous applications on the AWS infrastructure. It allows subscribers to run applications in a computing environment that is practically capable of serving an unlimited set of virtual machines. Let’s say that you have a database or software you need hosted on a machine. Back in the day, you would probably host this on an actual machine of some sort. Depending on your business, this setup could be pretty pricey or pretty risky. So software like Amazon EC2 allows you to rent a portion of a larger machine so that you can eliminate the risk of a disaster and lower the cost of hardware.
That’s the basics of it. It also comes with other great features that would eliminate the need for an infrastructure team to manage.
Let’s say I needed a dedicated server to put my WordPress installation on. I would easily be able to do this using AWS using an easy one-click install and then any other changes needed I could just SSH into my machine and have full access. All for the price of shared hosting like GoDaddy. Awesome.

AWS S3

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This service is more like a virtual drive to store and retrieve data. According to the AWS definition, “You can use Amazon S3 to store and retrieve any amount of data at any time, from anywhere on the web.”

Files in S3 are stored in buckets ( Buckets are like folders )
S3 is object-based storage (Like Images, videos, documents etc)
Bucket names should be unique globally because when we create a bucket in S3 it will have a unique URL.
It's a good practice to create buckets in the nearest region to you or for your customers.

AWS DynamoDB

Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale. DynamoDB is a no-sql database that scales with the amount of data coming in. What makes a no-sql database different from a SQL database is that it doesn’t use SQL and doesn’t have relational features.
The benefits for this is how lightweight the database can be as a result of not having these features. So AWS DynamoDB is an awesome free data storage option, however NoSQL Databases has its own use cases as does Relational Databases. So it is important to figure out what kind of storage option is best for your specific project before going forward.

AWS docs provideon the basics of DynamoDB, below is a list of some of the basic terminology I have gotten to know about DynamoDB:

Primary Key

This is a simple one, Primary key uniquely identifies the 'item' in the table, it can be a single attribute or a combination of attributes.

Image credit AWS docs

Tables

Strange name, given DynamoDB is NoSQL, However, it serves as a good mental model, and helps understand how data is grouped together. These 'tables' consist of a collection of 'items'.

Items

Items can be thought of as 'rows' in the tables, But unlike traditional tables, it does not have or need a specific schema. Aside from the primary key, it can contain any number of 'key-value' relations, and can also store nested objects upto 32 levels deep.

AWS Lambda

AWS Lambda is an event-driven serverless compute platform, spinning up the service in response to an event. AWS Lambda lets you run code without provisioning or managing servers.
The function itself has the following important aspects associated with it:

Trigger: A set of activities which invokes the function (runs the code you provide). The activity could be anything like a new object coming to your S3 bucket, a website or a service going down, an API call, etc.
The actual function: This is the run-time code that constitutes the function. AWS supports Python, Node.js, C#, Go and Java8 as runtime environments.
Resources: Each function can be assigned certain Roles, which grants the function certain privileges such as reading S3 bucket contents, writing results to a database and so on.

Lambda allows you to do this easily without having to deploy anything. Just create an instance and insert your Node, Python code etc and you will have the functionality of a server to return the results of your code for you. What makes this great is that it charges you on a usage basis. Only pay what you have to.
This means you can create your applications using your frontend frameworks and not have to worry about setting up your server.

AWS LIGHTSAIL

Lightsail is an easy-to-use cloud platform that offers you everything needed to build an application or website, plus a cost-effective, monthly plan. Whether you’re new to the cloud or looking to get on the cloud quickly with AWS infrastructure you trust, we’ve got you covered.

This one is more straightforward than the other ones. Essentially, it takes all different types of application you would need (Database, server, etc) and puts it in a container which is easy for you to use. I chose Lightsail over EC2 since it is a managed system focused on Wordpress. It also has a pre-set image for Wordpress so it’s pretty quick to launch.

AWS Lightsail offers a variety of options such as supporting a Windows server or a choice of Linux distros, thus helping core professionals to take advantage of this service package.
One of the best advantages of using AWS Lightsail is that the size of your website doesn’t matter. This service is able to host your website on the AWS environment seamlessly.

AWS ANALYTICS

AWS offers real-time web analytics with Kinesis Data Analytics solution that allows you to:

Easily track your website activities.
See who visits your website
From where are they visiting
What are they doing on your website
It provides a framework to analyze and visualize the metrics so that you can add more metrics instead of managing underlying infrastructure.

Conclusion

At a glance AWS can seem tough and overwhelming, however, its cloud services will deliver your business-critical enterprise applications reliably and cost-effectively, while giving you opportunities to cut delivery times for new products and services and replicate infrastructure environments quickly and easily. Moreover, it allows you to choose what kind of software makes sense for your scenario.
To get started in the ecosystem, take advantage of their free-tier. Once you understand what some of the services are, you can then start to understand most of them and how they can apply to what you want to use for your world domination.
Learn the basic shell commands in Linux like ssh, cd , mv , mkdir, rm, touch and so on. Make sure you know atleast vim, VS code or a similar editor.

Going serverless - Export CloudWatch Logs to S3

Maureen Chebet — Tue, 05 Jul 2022 16:57:32 +0000

Amazon CloudWatch logs lets you monitor, collect, act, analyze, stream metrics and access your log files from Amazon EC2 instances, AWS CloudTrail, Lambda functions, VPC flow logs, or other resources. CloudWatch can be costly and in order keep the costs under control it is ideal to move older logs to S3 for a long term retention period.

To accomplish this, we will be consuming the following AWS services:

S3 (Bucket & Bucket Policy)
IAM (Lambda Role)
CloudWatch (Events Rules)
Lambda (Functions)
Step Functions

1. Create S3 bucket
Create an S3 Bucket to export the logs to. To allow CloudWatch to PUT (WRITE access) objects to the bucket attach the following policy:

{
    "Version": "2008-10-17",
    "Id": "Policy1335892530063",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.YOUR-REGION.amazonaws.com"
            },
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.YOUR-REGION.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/*",
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }
            }
        }
    ]
}

2. Create IAM role
Here we are creating IAM role i.e ‘Export-CloudWatch-Logs-To-S3-Lambda-Role’ for Lambda to log events and write to the S3 bucket we created. Add the following policy to the created role.

3. Create and configure Lambda function
With Lambda you don’t have to provision or manage servers. You can simply paste the code below to your function and it will run when triggered. In this section of the article, we will set up the Lambda Function.

Navigate to Lambda >> Functions >> Create Function from there you will need to perform the following:

Choose ‘Author from Scratch’.

Function Name: Export-CloudWatch-Logs-To-S3

Runtime: Node.js 16.x

Under Permissions, choose existing role. Select ‘Use an existing role’, and choose the IAM we created earlier. Now go ahead and click on Create Function.

You will need to paste the code above under the Function Code section as shown in the screenshot below:

NB: Change bucket-name to the one created above and the region you are deploying to

4. Create Step Function State Machine

We will need a step machine to export multiple log groups data as one export task is accomplish at a time.

Go to services >> Step Functions >> Get started >> Create state machine >> Write your workflow in code >> put the following JSON in

{
    "StartAt": "CreateExportTask",
    "States": {
        "CreateExportTask": {
            "Type": "Task",
            "Resource": "LAMBDA_FUNCTION_ARN",
            "Next": "IsAllLogsExported"
        },
        "IsAllLogsExported": {
            "Type": "Choice",
            "Choices": [
                {
                    "Variable": "$.continue",
                    "BooleanEquals": true,
                    "Next": "CreateExportTask"
                }
            ],
            "Default": "SuccessState"
        },
        "SuccessState": {
            "Type": "Succeed"
        }
    }
}

(Put previously created lambda function ARN in JSON in State machine definition) → click on Create state machine.

5. Create CloudWatch event rule

In order to schedule the Lambda Function to run or trigger at a specific time, you will need to create and configure a CloudWatch Events Rule.

To create a new rule, you will need to navigate to CloudWatch >> Rules >> Go to Amazon EventBridge >> Create Rule. In this example, trigger the Lambda Function at 11:00 PM EAT. (Adjust the time to your needs) Targets >> AWS service >> select Step Functions state machine ( select previously created state machine) >> Execution role (create a new role) >> Additional settings >> from the dropdown select Constant (JSON text) and put the following JSON. Review and Create rule.

{
    "region":"REGION",
    "logGroupFilter":"/aws",
    "s3BucketName":"BUCKET_NAME",
    "logFolderName":"cloudwatchlogs"
}

NB: use ["/aws", "/prod"] for a list of logGroupFilter

Here region is the region where you set up this solution. Only log groups contain logGroupFilter value will be exported. s3BucketName is the destination bucket and in that what value we are giving in logFolderName, that named folder will be created and in that logs will be stored.

Choose Create a new role for this specific resource (this role is needed for CloudWatch Events needs permission to send events to your Step Functions state machine >> click on Configure details

Conclusion

This solution export the logs everyday to S3 bucket. With this, we are able to manage costs usage on cloudwatch. Go ahead and add a lifecycle to S3 bucket such that the logs become stale after say, 90 days then they get deleted.