Forem: JM Rifkhan

Deploying ML Models to Production: AWS Lambda vs ECS vs EKS - A Data-Driven Comparison

JM Rifkhan — Tue, 11 Nov 2025 10:51:19 +0000

A comprehensive, hands-on guide to choosing the right AWS platform for your ML inference workloads

Introduction

"Which AWS platform should I use to deploy my machine learning model?"

As an AWS Community Builder working with ML teams, I hear this question constantly. The answer is always the same: "It depends." But on what, exactly? Cost? Performance? Team expertise? Scale?

Instead of giving theoretical advice, I decided to build something concrete: a production-ready sentiment analysis model deployed across all three major AWS container orchestration platforms Lambda, ECS Fargate, and EKS then benchmark them rigorously with real load tests and actual cost calculations.

The results surprised me. Lambda isn't always cheaper. EKS isn't always faster. And for most teams, the "obvious" choice might be wrong.

In this post, I'll share everything I learned building this end-to-end MLOps project, including:

Complete implementation details with code
Real performance benchmarks (10,000+ requests tested)
Actual AWS cost breakdowns at different scales
A decision framework for choosing the right platform
Production deployment best practices

By the end, you'll have a clear, data-driven understanding of when to use each platform and a complete reference implementation you can deploy yourself.

Full project repository: Available at the end of this post with all code, Infrastructure as Code, and documentation.

The Challenge: Deploying ML Models at Scale

Machine learning models are fundamentally different from traditional web applications. They:

Require significant memory (often 1-10GB for transformer models)
Have cold start penalties (model loading takes seconds)
Need GPU acceleration for some workloads
Consume varying CPU based on input size
Must scale quickly under traffic spikes

These unique requirements mean traditional deployment advice doesn't always apply. A platform perfect for a REST API might be terrible for ML inference and vice versa.

What We're Building

I built a complete MLOps pipeline for sentiment analysis using:

Model: DistilBERT (a lightweight BERT variant)

Size: ~250MB
Task: Classify text as POSITIVE or NEGATIVE
Accuracy: 92.3% on IMDb dataset
Inference time: 100-300ms per request

API: FastAPI application with:

Single prediction endpoint (/predict)
Batch prediction endpoint (/predict/batch)
Health checks (/health)
Prometheus metrics (/metrics)

Infrastructure: Everything deployed using:

Terraform for all AWS resources
Docker for containerization
Kubernetes manifests for EKS
Locust for load testing

The beauty of this approach: identical code running on three different platforms, making our comparison truly apples-to-apples.

Architecture Overview

Here's the high-level architecture that's consistent across all platforms:

Each platform has its own infrastructure setup, but the application code is identical. As shown in the diagram, all three deployments share common infrastructure (ECR for container images, IAM for permissions, CloudWatch for monitoring) while using different compute platforms.

Implementation Deep Dive

Building the ML Model

I chose DistilBERT because it's 40% smaller and 60% faster than BERT while retaining 97% of its language understanding. Perfect for production inference.

from transformers import AutoModelForSequenceClassification, AutoTokenizer
from datasets import load_dataset

# Load pre-trained model
model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased",
    num_labels=2  # Binary: positive/negative
)

# Train on IMDb dataset
dataset = load_dataset("imdb")
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_train,
    eval_dataset=tokenized_eval
)

trainer.train()

Training Results:

Accuracy: 92.3%
F1 Score: 0.92
Training time: ~15 minutes on CPU
Model size: 268MB

The model loads in 2-3 seconds and processes individual predictions in 100-150ms on CPU.

Creating the FastAPI Application

FastAPI gives us async support, automatic validation, and built-in API documentation:

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

app = FastAPI(title="Sentiment Analysis API")

class PredictionRequest(BaseModel):
    text: str

class PredictionResponse(BaseModel):
    text: str
    sentiment: str
    confidence: float
    processing_time_ms: float

# Load model on startup
@app.on_event("startup")
async def load_model():
    global tokenizer, model
    tokenizer = AutoTokenizer.from_pretrained("/opt/ml/model")
    model = AutoModelForSequenceClassification.from_pretrained("/opt/ml/model")
    model.eval()

@app.post("/predict", response_model=PredictionResponse)
async def predict(request: PredictionRequest):
    start_time = time.time()

    # Tokenize input
    inputs = tokenizer(
        request.text,
        return_tensors="pt",
        truncation=True,
        max_length=512
    )

    # Run inference
    with torch.no_grad():
        outputs = model(**inputs)
        predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)

    sentiment = "POSITIVE" if predictions[0][1] > predictions[0][0] else "NEGATIVE"
    confidence = max(predictions[0]).item()

    return PredictionResponse(
        text=request.text,
        sentiment=sentiment,
        confidence=confidence,
        processing_time_ms=(time.time() - start_time) * 1000
    )

This gives us:

Automatic request/response validation
Type safety
OpenAPI documentation at /docs
Async request handling for better throughput

Containerization Strategy

I built two Docker images using multi-stage builds:

Standard Image (for ECS/EKS):

# Stage 1: Build dependencies
FROM python:3.11-slim as builder
WORKDIR /build
COPY requirements.txt .
RUN pip install --user -r requirements.txt

# Stage 2: Runtime
FROM python:3.11-slim
COPY --from=builder /root/.local /root/.local
ENV PATH=/root/.local/bin:$PATH

WORKDIR /app
COPY app/ /app/
COPY model/exported_model/ /opt/ml/model/

EXPOSE 8080
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

Lambda Image:

FROM public.ecr.aws/lambda/python:3.11

COPY requirements.txt ${LAMBDA_TASK_ROOT}/
RUN pip install --no-cache-dir -r ${LAMBDA_TASK_ROOT}/requirements.txt

COPY app/ ${LAMBDA_TASK_ROOT}/
COPY model/exported_model/ /opt/ml/model/

CMD ["main.handler"]

The Lambda image uses AWS's base image and includes the Mangum adapter to translate API Gateway events to ASGI.

Final image sizes:

Standard: 1.2GB
Lambda: 1.3GB

Platform-Specific Deployments

AWS Lambda: Serverless Inference

Lambda deployment uses container images (not ZIP files) to accommodate our large model:

resource "aws_lambda_function" "ml_inference" {
  function_name = "ml-inference"
  package_type  = "Image"
  image_uri     = "${ecr_repository_url}:lambda-latest"

  timeout     = 300  # 5 minutes
  memory_size = 3008 # ~3GB (affects CPU too)

  environment {
    variables = {
      DEPLOYMENT_TYPE = "lambda"
      MODEL_PATH      = "/opt/ml/model"
    }
  }
}

I added API Gateway HTTP API for RESTful access:

resource "aws_apigatewayv2_api" "lambda_api" {
  name          = "ml-inference-api"
  protocol_type = "HTTP"
}

Key Features:

Auto-scaling (up to 1000 concurrent executions)
Pay-per-use pricing
No infrastructure management
15-minute timeout limit

Challenges:

Cold starts (3-5 seconds for model loading)
10GB memory limit
250MB unzipped deployment package limit (container images can be 10GB)

Amazon ECS Fargate: Managed Containers

ECS provides a middle ground between serverless and Kubernetes:

resource "aws_ecs_cluster" "main" {
  name = "ml-inference-cluster"

  setting {
    name  = "containerInsights"
    value = "enabled"
  }
}

resource "aws_ecs_task_definition" "main" {
  family                   = "ml-inference"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = "2048"  # 2 vCPU
  memory                   = "4096"  # 4GB

  container_definitions = jsonencode([{
    name  = "ml-inference"
    image = "${ecr_repository_url}:latest"

    portMappings = [{
      containerPort = 8080
      protocol      = "tcp"
    }]

    healthCheck = {
      command     = ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"]
      interval    = 30
      timeout     = 5
      retries     = 3
    }
  }])
}

resource "aws_ecs_service" "main" {
  name            = "ml-inference-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.main.arn
  desired_count   = 2
  launch_type     = "FARGATE"

  load_balancer {
    target_group_arn = aws_lb_target_group.main.arn
    container_name   = "ml-inference"
    container_port   = 8080
  }
}

I configured auto-scaling based on CPU utilization:

resource "aws_appautoscaling_policy" "ecs_cpu" {
  policy_type = "TargetTrackingScaling"

  target_tracking_scaling_policy_configuration {
    target_value = 70.0

    predefined_metric_specification {
      predefined_metric_type = "ECSServiceAverageCPUUtilization"
    }
  }
}

Key Features:

No cold starts (always-on containers)
Application Load Balancer for traffic distribution
Auto-scaling (1-10 tasks)
VPC networking with private subnets

Challenges:

Always running (costs money even with zero traffic)
Less flexible than Kubernetes
Requires ALB ($16/month base cost)

Amazon EKS: Full Kubernetes

EKS gives us the full power of Kubernetes:

resource "aws_eks_cluster" "main" {
  name     = "ml-inference-cluster"
  version  = "1.28"
  role_arn = aws_iam_role.eks_cluster.arn

  vpc_config {
    subnet_ids = concat(
      aws_subnet.private[*].id,
      aws_subnet.public[*].id
    )
  }
}

resource "aws_eks_node_group" "main" {
  cluster_name    = aws_eks_cluster.main.name
  node_role_arn   = aws_iam_role.eks_node_group.arn
  subnet_ids      = aws_subnet.private[*].id
  instance_types  = ["t3.large"]

  scaling_config {
    desired_size = 2
    max_size     = 5
    min_size     = 1
  }
}

Kubernetes deployment with Horizontal Pod Autoscaler:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-inference
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: ml-inference
        image: <ECR_URL>:latest
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"
          limits:
            memory: "4Gi"
            cpu: "2000m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 60
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ml-inference-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ml-inference
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        averageUtilization: 70

Key Features:

Full Kubernetes API and ecosystem
Advanced deployment strategies (blue/green, canary)
Multi-region/multi-cloud portability
Rich monitoring with Prometheus

Challenges:

Most expensive ($73/month for control plane alone)
Requires Kubernetes expertise
Complex setup and maintenance
Overkill for simple workloads

The Benchmarks

I ran comprehensive load tests using Locust with realistic traffic patterns:

Test Configuration:

Duration: 5 minutes per platform
Concurrent users: 10
Request mix: 70% single predictions, 30% batch (5-10 items)
Sample texts: 20 different movie reviews
Total requests: 10,000+ per platform

Performance Results

Metric	Lambda	ECS Fargate	EKS	Winner
Mean Latency	245ms	156ms	128ms	🏆 EKS
Median Latency	198ms	142ms	115ms	🏆 EKS
P95 Latency	890ms	312ms	267ms	🏆 EKS
P99 Latency	1,450ms	498ms	389ms	🏆 EKS
Cold Start	3-5s	N/A	N/A	🏆 ECS/EKS
Throughput (RPS)	42	98	156	🏆 EKS
Success Rate	99.8%	99.9%	99.9%	-

Key Observations:

EKS dominates on raw performance:
- 48% lower latency than Lambda
- 18% lower latency than ECS
- 3.7x better throughput than Lambda
Lambda's variability is concerning:
- P99 latency is 7.3x higher than median
- Cold starts add 3-5 seconds unpredictably
- Even with provisioned concurrency, variance is high
ECS provides predictable performance:
- Consistent latency (small gap between median and P99)
- No cold starts
- Good balance for most workloads

Cost Analysis

Here's where things get interesting. I calculated actual AWS costs at different scales:

Monthly Costs for 10,000 Requests/Day

Platform	Infrastructure	Request Costs	Total	Per 1K Requests
Lambda	$0	$28.50	$28.50	$0.095
ECS	$75.08	$0	$75.08	$0.250
EKS	$194.96	$0	$194.96	$0.650

Lambda Cost Breakdown:

Requests: 300,000/month × $0.20 per 1M = $0.06
Compute: 300,000 × 0.25s × 3GB × $0.0000166667 = $28.44
Total: $28.50/month

ECS Cost Breakdown:

vCPU: 2 × $0.04048/hour × 730 hours = $59.10
Memory: 4GB × $0.004445/hour × 730 hours = $12.98
ALB: $16.00/month (base) + $0.008/LCU
Total: ~$75.08/month

EKS Cost Breakdown:

Control Plane: $0.10/hour × 730 hours = $73.00
Worker Nodes: 2 × t3.large × $0.0832/hour × 730 hours = $121.47
LoadBalancer: ~$0.50/month
Total: ~$194.96/month

Cost at Different Scales

This is where the "it depends" becomes clear:

Daily Requests	Lambda	ECS	EKS	Winner	Reason
1,000	$2.85	$75.08	$194.96	🏆 Lambda	96% cheaper
5,000	$14.25	$75.08	$194.96	🏆 Lambda	81% cheaper
10,000	$28.50	$75.08	$194.96	🏆 Lambda	62% cheaper
25,000	$71.25	$75.08	$194.96	🏆 Lambda	5% cheaper
35,000	$99.75	$75.08	$194.96	🏆 ECS	Crossover
50,000	$142.50	$75.08	$194.96	🏆 ECS	47% cheaper
75,000	$213.75	$88.08	$194.96	🏆 EKS	9% cheaper
85,000	$242.25	$88.08	$194.96	🏆 EKS	Crossover
100,000	$285.00	$105.00	$194.96	🏆 EKS	32% cheaper
250,000	$712.50	$140.00	$210.00	🏆 EKS	70% cheaper
500,000	$1,425.00	$180.00	$225.00	🏆 EKS	84% cheaper

Critical Insights:

Lambda is cheapest until ~35K requests/day - Perfect for startups and low-traffic apps
ECS is the sweet spot for medium traffic (35K-85K requests/day) - Best balance for growing companies
EKS wins at high scale (>85K requests/day) - Enterprise-grade performance and cost
The crossover points matter - Most teams will hit the ECS sweet spot

Real-World Cost Optimization

These are base costs. Here's how to optimize each:

Lambda Optimization:

Use Compute Savings Plans: -17% ($23.66 vs $28.50)
Switch to ARM64 (Graviton2): -20% ($22.80 vs $28.50)
Right-size memory allocation
Combined savings: ~30%

ECS Optimization:

Use Fargate Spot: -70% on compute ($34.92 vs $75.08)
Use Savings Plans: -17%
Right-size task definitions
Combined savings: ~65%

EKS Optimization:

Use Spot instances for nodes: -70% ($109.46 vs $194.96)
Use Reserved Instances for baseline: -40%
Implement cluster autoscaler
Combined savings: ~60%

Decision Framework

Based on my testing, here's when to choose each platform:

Choose AWS Lambda When:

✅ Traffic Characteristics:

Sporadic/unpredictable patterns
Daily traffic spikes with long idle periods
<35K requests/day consistently
Event-driven workloads

✅ Business Requirements:

Zero infrastructure management
Fast time-to-market (deploy in minutes)
Pay only for actual usage
Prototyping or MVP

✅ Technical Constraints:

Model <10GB
Inference time <15 minutes
Can tolerate 3-5s cold starts
Team has limited ops experience

❌ Avoid Lambda When:

Need sub-100ms latency consistently
Model >10GB
High, steady traffic (>50K req/day)
Can't tolerate cold start delays

Real-World Example:
A startup analyzing customer feedback emails receives 500 emails/day in bursts after business hours. Lambda cost: $1.43/day vs ECS $75.08/month. Lambda saves $2,100/year.

Choose Amazon ECS Fargate When:

✅ Traffic Characteristics:

Steady, predictable traffic
35K-85K requests/day
Business hours traffic with some variability
Need consistent performance

✅ Business Requirements:

Balance of control and simplicity
Team familiar with Docker
Moderate budget
Want managed infrastructure

✅ Technical Constraints:

No cold starts acceptable
Need 100-200ms latency
Standard container patterns
No Kubernetes expertise

❌ Avoid ECS When:

Very low traffic (<10K req/day) - waste money
Need advanced orchestration (blue/green, canary)
Multi-region/multi-cloud required
Team already proficient in Kubernetes

Real-World Example:
A B2B SaaS analyzing 50K customer support tickets/day during business hours (9am-6pm). ECS with auto-scaling provides predictable costs (~$75/month) and performance. Perfect fit.

Choose Amazon EKS When:

✅ Traffic Characteristics:

High, sustained traffic (>85K req/day)
Need to scale to 1M+ requests/day
Global traffic across regions
Complex microservices architecture

✅ Business Requirements:

Enterprise-grade performance
Multi-region/multi-cloud strategy
Advanced deployment patterns
Strong DevOps/SRE team

✅ Technical Constraints:

Need <100ms latency
Complex service mesh
Advanced monitoring (Prometheus, Grafana)
Team has Kubernetes expertise

❌ Avoid EKS When:

Small team (<5 engineers)
No Kubernetes experience
Simple, single-service deployment
Budget constraints
Rapid prototyping needed

Real-World Example:
An enterprise processing 1M+ social media posts/day across US, EU, and APAC regions. EKS provides the performance, scalability, and multi-region capabilities needed. Cost: ~$220/month (optimized with Spot) vs $4,275/month on Lambda.

Deployment Pipeline and CI/CD

One of the key advantages of using containerized deployments is that we can use the same CI/CD pipeline for all three platforms. Here's our deployment flow:

Pipeline Architecture

Our MLOps pipeline is built using GitHub Actions and consists of five automated stages:

1. Source Control & Triggers

Trigger: Every push to main branch
Code checkout with full history for versioning
Environment setup: Python 3.11, Docker BuildKit, AWS credentials

2. Build Stage (2-3 minutes)

- name: Build Docker Images
  run: |
    # Build standard image for ECS/EKS
    docker build -t ml-inference:latest -f Dockerfile .

    # Build Lambda-optimized image
    docker build -t ml-inference:lambda-latest -f Dockerfile.lambda .

    # Tag with version and commit SHA
    docker tag ml-inference:latest $ECR_URI:v${VERSION}
    docker tag ml-inference:latest $ECR_URI:${GITHUB_SHA}

Multi-stage builds reduce image size by 60%:

Stage 1: Build dependencies (includes compilers)
Stage 2: Runtime-only (production-ready)

3. Test & Security Stage (2-4 minutes)

- name: Run Tests
  run: |
    # Unit tests
    pytest tests/ --cov=app --cov-report=xml

    # Integration tests
    pytest tests/integration/ --maxfail=1

    # Model validation
    python scripts/validate_model.py

Security Scanning:

Trivy: Container vulnerability scanning
Bandit: Python code security analysis
SAST: Static application security testing
Dependency check: Known CVE detection

- name: Security Scan
  run: |
    trivy image --severity HIGH,CRITICAL ml-inference:latest
    bandit -r app/ -f json -o security-report.json

4. Push to ECR (1-2 minutes)

- name: Push to Amazon ECR
  run: |
    aws ecr get-login-password | docker login --username AWS --password-stdin $ECR_URI

    # Push all tags
    docker push $ECR_URI:latest
    docker push $ECR_URI:lambda-latest
    docker push $ECR_URI:v${VERSION}
    docker push $ECR_URI:${GITHUB_SHA}

    # Enable image scanning
    aws ecr start-image-scan --repository-name ml-inference --image-id imageTag=latest

Image Management:

Keep last 10 images (lifecycle policy)
Automatic vulnerability scanning on push
Cross-region replication for DR

5. Parallel Deployment (3-5 minutes)

The pipeline deploys to all three platforms simultaneously:

deploy:
  strategy:
    matrix:
      platform: [lambda, ecs, eks]
  steps:
    - name: Deploy to ${{ matrix.platform }}
      run: ./scripts/deploy-${{ matrix.platform }}.sh

Lambda Deployment:

aws lambda update-function-code \
  --function-name ml-inference \
  --image-uri $ECR_URI:lambda-latest

aws lambda wait function-updated \
  --function-name ml-inference

# Update alias to point to new version
aws lambda update-alias \
  --function-name ml-inference \
  --name production \
  --function-version $NEW_VERSION

ECS Deployment (Rolling Update):

# Create new task definition revision
aws ecs register-task-definition \
  --cli-input-json file://task-definition.json

# Update service (rolling deployment)
aws ecs update-service \
  --cluster ml-inference-cluster \
  --service ml-inference-service \
  --task-definition ml-inference:${NEW_REVISION} \
  --force-new-deployment

# Wait for stable state
aws ecs wait services-stable \
  --cluster ml-inference-cluster \
  --services ml-inference-service

EKS Deployment (Kubernetes):

# Update image tag in deployment
kubectl set image deployment/ml-inference \
  ml-inference=$ECR_URI:latest \
  --record

# Rolling update with zero downtime
kubectl rollout status deployment/ml-inference

# Rollback if health checks fail
if ! kubectl rollout status deployment/ml-inference; then
  kubectl rollout undo deployment/ml-inference
  exit 1
fi

6. Post-Deployment Verification

Smoke Tests:

def run_smoke_tests(endpoint):
    # Health check
    response = requests.get(f"{endpoint}/health")
    assert response.status_code == 200

    # Prediction test
    test_payload = {"text": "This movie was fantastic!"}
    response = requests.post(f"{endpoint}/predict", json=test_payload)
    assert response.status_code == 200
    assert response.json()["sentiment"] in ["POSITIVE", "NEGATIVE"]

    # Latency check
    assert response.elapsed.total_seconds() < 1.0

Monitoring Integration:

- name: Update Deployment Metrics
  run: |
    aws cloudwatch put-metric-data \
      --namespace MLOps/Deployment \
      --metric-name DeploymentSuccess \
      --value 1 \
      --dimensions Platform=${{ matrix.platform }},Version=$VERSION

CI/CD Benefits

This unified approach provides:

✅ Consistency: Same code, same tests, same images across all platforms
✅ Speed: 8-12 minutes from commit to production (all platforms)
✅ Safety: Automated testing, security scanning, gradual rollouts
✅ Visibility: Full deployment history, metrics, and audit logs
✅ Reliability: Automatic rollbacks on failure, zero-downtime deployments

Deployment Metrics

Over the last 90 days:

Deployment frequency: 3-5 per week
Success rate: 98.7%
Mean time to deploy: 9.5 minutes
Failed deployments: Auto-rollback within 2 minutes
Zero production incidents from failed deployments

Shared Infrastructure

All three platforms leverage common AWS infrastructure to ensure consistency, security, and cost efficiency:

1. Network Architecture (VPC)

VPC Configuration:

resource "aws_vpc" "main" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "ml-inference-vpc"
  }
}

Multi-AZ Design for High Availability:

us-east-1a (AZ1):
├─ Public Subnet: 10.0.0.0/24
│  ├─ ALB/NLB
│  └─ NAT Gateway 1
└─ Private Subnet: 10.0.10.0/24
   ├─ ECS Tasks
   ├─ EKS Worker Nodes
   └─ Lambda ENIs

us-east-1b (AZ2):
├─ Public Subnet: 10.0.1.0/24
│  ├─ ALB/NLB (standby)
│  └─ NAT Gateway 2
└─ Private Subnet: 10.0.11.0/24
   ├─ ECS Tasks
   ├─ EKS Worker Nodes
   └─ Lambda ENIs

Security Groups:

# ALB Security Group (public-facing)
resource "aws_security_group" "alb" {
  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]  # HTTPS from internet
  }

  egress {
    from_port       = 8080
    to_port         = 8080
    protocol        = "tcp"
    security_groups = [aws_security_group.ecs_tasks.id]
  }
}

# ECS/EKS Security Group (private)
resource "aws_security_group" "compute" {
  ingress {
    from_port       = 8080
    to_port         = 8080
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id]  # Only from ALB
  }

  egress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]  # HTTPS to AWS services
  }
}

Network Costs:

NAT Gateway: $0.045/hour × 2 × 730 hours = $65.70/month
Data Transfer: ~$0.09/GB (first 10TB)
VPC endpoints for S3/ECR: Free (saves data transfer costs)

2. Container Registry (ECR)

Repository Configuration:

resource "aws_ecr_repository" "ml_inference" {
  name                 = "ml-inference-comparison"
  image_tag_mutability = "MUTABLE"

  image_scanning_configuration {
    scan_on_push = true
  }

  encryption_configuration {
    encryption_type = "AES256"
  }
}

Lifecycle Policy (Cost Optimization):

{
  "rules": [
    {
      "rulePriority": 1,
      "description": "Keep last 10 images",
      "selection": {
        "tagStatus": "any",
        "countType": "imageCountMoreThan",
        "countNumber": 10
      },
      "action": {
        "type": "expire"
      }
    }
  ]
}

Multi-Platform Image Support:

# Build multi-architecture images
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t $ECR_URI:latest \
  --push .

Repository Features:

Automatic scanning: Trivy integration for CVE detection
Immutable tags: Production images are immutable
Cross-region replication: DR to us-west-2
Lifecycle management: Auto-delete old images

ECR Costs:

Storage: $0.10/GB/month
Average usage: 3GB (10 images × 300MB)
Monthly cost: ~$0.30

3. IAM Roles & Policies

Principle of Least Privilege:

Each platform gets its own IAM role with minimal permissions:

Lambda Execution Role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:/aws/lambda/ml-inference*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:PutMetricData"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "cloudwatch:namespace": "MLInference"
        }
      }
    }
  ]
}

ECS Task Role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:/ecs/ml-inference*"
    }
  ]
}

EKS Node Role (IRSA - IAM Roles for Service Accounts):

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ml-inference-sa
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/ml-inference-eks-pod-role

Secrets Management:

resource "aws_secretsmanager_secret" "model_config" {
  name = "ml-inference/model-config"

  rotation_rules {
    automatically_after_days = 30
  }
}

4. Monitoring & Logging (CloudWatch)

Centralized Logging:

CloudWatch Log Groups:
├─ /aws/lambda/ml-inference          (Lambda logs)
├─ /ecs/ml-inference                 (ECS logs)
├─ /aws/eks/ml-inference-cluster     (EKS control plane)
└─ /aws/containerinsights/ml-inference (EKS pod logs)

Log Retention & Costs:

resource "aws_cloudwatch_log_group" "lambda" {
  name              = "/aws/lambda/ml-inference"
  retention_in_days = 30  # Balance cost vs. debugging needs

  tags = {
    Platform = "Lambda"
  }
}

Custom Metrics:

# Application sends custom metrics
cloudwatch.put_metric_data(
    Namespace='MLInference',
    MetricData=[
        {
            'MetricName': 'ModelInferenceTime',
            'Value': inference_duration_ms,
            'Unit': 'Milliseconds',
            'Dimensions': [
                {'Name': 'Platform', 'Value': 'lambda'},
                {'Name': 'ModelVersion', 'Value': 'v1.2.0'}
            ],
            'Timestamp': datetime.utcnow()
        }
    ]
)

CloudWatch Dashboard:

{
  "widgets": [
    {
      "type": "metric",
      "properties": {
        "metrics": [
          [ "MLInference", "InferenceLatency", { "stat": "Average" } ],
          [ ".", ".", { "stat": "p99" } ]
        ],
        "period": 300,
        "stat": "Average",
        "region": "us-east-1",
        "title": "Inference Latency"
      }
    },
    {
      "type": "metric",
      "properties": {
        "metrics": [
          [ "AWS/Lambda", "Invocations", { "label": "Lambda" } ],
          [ "AWS/ECS", "TaskCount", { "label": "ECS" } ],
          [ "ContainerInsights", "pod_number_of_running_pods", { "label": "EKS" } ]
        ],
        "title": "Active Compute Units"
      }
    }
  ]
}

Alarms:

resource "aws_cloudwatch_metric_alarm" "high_error_rate" {
  alarm_name          = "ml-inference-high-error-rate"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "5XXError"
  namespace           = "AWS/ApiGateway"
  period              = 60
  statistic           = "Sum"
  threshold           = 10
  alarm_description   = "This metric monitors API error rate"
  alarm_actions       = [aws_sns_topic.alerts.arn]
}

CloudWatch Costs:

Logs ingestion: $0.50/GB
Logs storage: $0.03/GB/month
Custom metrics: $0.30/metric/month
Average monthly: ~$15-20

5. Cost Summary - Shared Infrastructure

Component	Monthly Cost	Notes
VPC	$0	No charge for VPC itself
NAT Gateways (2)	$65.70	$0.045/hour × 2 × 730 hours
Data Transfer	~$5-10	Variable based on traffic
ECR	$0.30	3GB storage
CloudWatch Logs	$10-15	30-day retention
CloudWatch Metrics	$5	~15 custom metrics
Secrets Manager	$0.40	$0.40/secret/month
Total	~$86-96	Shared across all platforms

Cost Optimization Tips:

Use VPC Endpoints for S3/ECR (saves data transfer costs)
Compress CloudWatch Logs (reduces storage by 70%)
Archive old logs to S3 (10× cheaper: $0.023/GB vs $0.03/GB)
Use CloudWatch Logs Insights instead of exporting to S3
Enable ECR lifecycle policies (auto-delete unused images)

Infrastructure as Code

All shared infrastructure is defined in Terraform:

terraform/
├── modules/
│   ├── vpc/                 # VPC, subnets, NAT gateways
│   ├── ecr/                 # Container registry
│   ├── iam/                 # Roles and policies
│   └── monitoring/          # CloudWatch, alarms
├── environments/
│   ├── dev/
│   ├── staging/
│   └── production/
└── main.tf

Benefits of Shared Infrastructure:

✅ Cost Reduction: Single VPC, NAT gateways, and ECR for all platforms
✅ Consistency: Same networking, security, and monitoring everywhere
✅ Simplified Management: One IaC codebase, centralized changes
✅ Security: Centralized IAM policies, unified audit logs
✅ Reliability: Multi-AZ design, automatic failover

Production Best Practices

Security Hardening

Lambda:

# Use IAM for authorization
resource "aws_lambda_function_url" "ml_inference" {
  authorization_type = "AWS_IAM"  # Not NONE!
}

# Scan container images
resource "aws_ecr_repository" "ml_inference" {
  image_scanning_configuration {
    scan_on_push = true
  }
}

# Use Secrets Manager
environment {
  variables = {
    DB_SECRET_ARN = aws_secretsmanager_secret.db.arn
  }
}

ECS/EKS:

Use private subnets for compute
Security groups with minimal inbound rules
IAM roles for task/pod authentication
Enable encryption at rest and in transit

Monitoring and Observability

All three platforms integrate with CloudWatch, but I added custom metrics:

import boto3

cloudwatch = boto3.client('cloudwatch')

def log_prediction(latency_ms, sentiment):
    cloudwatch.put_metric_data(
        Namespace='MLInference',
        MetricData=[
            {
                'MetricName': 'InferenceLatency',
                'Value': latency_ms,
                'Unit': 'Milliseconds',
                'Dimensions': [
                    {'Name': 'Sentiment', 'Value': sentiment},
                    {'Name': 'Platform', 'Value': os.getenv('DEPLOYMENT_TYPE')}
                ]
            }
        ]
    )

For EKS, I added Prometheus metrics:

from prometheus_client import Counter, Histogram

PREDICTION_LATENCY = Histogram(
    'prediction_latency_seconds',
    'Prediction latency in seconds'
)

PREDICTION_COUNT = Counter(
    'predictions_total',
    'Total predictions',
    ['sentiment']
)

@app.post("/predict")
async def predict(request: PredictionRequest):
    with PREDICTION_LATENCY.time():
        result = model_analyzer.predict(request.text)
        PREDICTION_COUNT.labels(sentiment=result['sentiment']).inc()
        return result

Auto-Scaling Configuration

Lambda: Works out of the box, but configure reserved concurrency for production:

resource "aws_lambda_function" "ml_inference" {
  reserved_concurrent_executions = 100
}

ECS: Target tracking auto-scaling:

resource "aws_appautoscaling_policy" "ecs_cpu" {
  target_tracking_scaling_policy_configuration {
    target_value = 70.0
    scale_in_cooldown  = 300  # 5 min
    scale_out_cooldown = 60   # 1 min
  }
}

EKS: Horizontal Pod Autoscaler with custom metrics:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        averageUtilization: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 30

Lessons Learned

1. Lambda Cold Starts Are Real (And Painful)

Even with 3GB memory and optimized container images, Lambda cold starts took 3-5 seconds. That's unacceptable for user-facing applications. Provisioned concurrency helps but adds significant cost.

Mitigation:

Keep functions warm with scheduled pings
Use provisioned concurrency for critical paths
Accept cold starts for async/batch workloads

2. ECS is Underrated

Most teams jump from Lambda to EKS, skipping ECS entirely. That's a mistake. ECS Fargate provides 80% of Kubernetes benefits with 20% of the complexity. For most teams, it's the Goldilocks solution.

3. Right-Sizing Matters More Than Platform

A poorly configured EKS cluster can cost more than a well-optimized Lambda setup. I saved:

30% on Lambda by right-sizing memory
65% on ECS using Fargate Spot
60% on EKS using Spot instances

Focus on optimization before switching platforms.

4. Benchmark With Real Traffic

My synthetic tests showed Lambda performing better than production. Real traffic has:

Variable input lengths (affecting inference time)
Burst patterns (causing cold starts)
Geographic distribution (network latency)

Always test with production-like data.

5. Total Cost of Ownership Includes People

EKS might be cheaper at scale, but requires Kubernetes expertise. If you need to hire a DevOps engineer ($120K/year), Lambda suddenly looks very affordable.

Factor in:

Team expertise and training
Operational overhead
On-call burden
Time to market

Conclusion

So, which platform should you choose?

For most teams starting out: ECS Fargate

It provides:

No cold starts (predictable performance)
Reasonable cost at medium scale
Familiar Docker patterns
Managed infrastructure
Path to EKS if you outgrow it

Start with Lambda if you have:

Truly unpredictable traffic
<35K requests/day
Strong cost constraints
No ops team

Upgrade to EKS when you hit:

>85K requests/day sustained
Need for advanced orchestration
Multi-region requirements
Team with K8s expertise

The beauty of this architecture: you can migrate between platforms with minimal code changes. Start simple, scale as needed.

Get the Complete Project

I've open-sourced everything:

Repository includes:

✅ Complete model training code
✅ FastAPI application
✅ Dockerfiles for all platforms
✅ Terraform modules (Lambda, ECS, EKS)
✅ Kubernetes manifests
✅ Load testing suite
✅ Deployment scripts
✅ Comprehensive documentation

Quick start:

git clone https://github.com/rifkhan107/mlops-inference-comparison.git
cd mlops-inference-comparison
make quickstart  # Deploys everything in 60 minutes

Learn more:

What's Next?

In future posts, I'll explore:

GPU-accelerated inference on EKS with NVIDIA GPUs
Multi-model serving with SageMaker Multi-Model Endpoints
A/B testing ML models in production
Feature stores for real-time inference

Let's Connect

I'm passionate about MLOps and cloud-native architectures. Let's discuss your ML deployment challenges:

Twitter: rifkhan107
LinkedIn: rifkhan107
GitHub: rifkhan107
AWS Community Builders: rifkhan

Questions about the project? Open an issue on GitHub or reach out on Twitter.

Found this helpful? ⭐ Star the repository and share with your team!

Event-Driven Batch Processing on AWS: From Scheduled Tasks to Auto-Scaling Workloads

JM Rifkhan — Sat, 08 Nov 2025 17:08:45 +0000

As DevOps engineers, we've all been there: running EC2 instances 24/7 to process batch jobs that only run a few hours per day. The meter keeps ticking whether you're processing millions of images or sitting idle at 3 AM. Your AWS bill arrives, and you wonder: "Why am I paying for servers that do nothing most of the time?"

I've seen this pattern repeatedly across startups, enterprises, and everything in between. The traditional approach to batch processing always-on infrastructure made sense in the data center era. But in the cloud? It's like leaving your car running in the parking lot all day because you might need it later.

In this comprehensive guide, I'll walk you through three production-ready patterns for implementing batch processing on AWS using containers. You'll learn when to use each pattern, how to implement auto-scaling based on queue depth, and how to reduce your batch processing costs by up to 74%. More importantly, you'll understand the architectural decisions behind each pattern and how to adapt them to your specific use cases.

All code examples are production-ready and available on GitHub, complete with Terraform infrastructure, monitoring configurations, and deployment guides. Let's dive in.

The Problem with Traditional Batch Processing

Let's start with a real scenario that mirrors what I've seen dozens of times. Imagine you're running a SaaS platform that processes user-uploaded documents PDFs that need to be converted, analyzed, and indexed. Your current setup uses two t3.medium EC2 instances running constantly, costing about $73 per month.

But here's the reality of your workload:

6 AM - 10 AM: Heavy processing (users uploading files before work)
10 AM - 5 PM: Moderate, sporadic activity
5 PM - 7 PM: Another spike (end-of-day uploads)
7 PM - 6 AM: Nearly idle (maybe 5% utilization)

You're paying for 100% capacity but using it fully only about 25% of the time. The math is brutal: you're paying $55/month for idle servers doing nothing.

Traditional batch processing faces four major challenges:

1. Always-On Infrastructure Costs

In the on-premises world, this made sense servers were already purchased. But in the cloud, you're paying by the hour. Those idle servers are burning money while you sleep.

2. Manual Scaling is Reactive, Not Proactive

You get a sudden spike in uploads. Your servers max out. Users wait. You get paged. You manually spin up more instances. By the time they're ready, the spike has passed. Then you forget to turn them off and waste more money.

3. Poor Resource Utilization

Your jobs vary in size. Some take 30 seconds, others take 10 minutes. With fixed-size instances, you either over-provision (waste money) or under-provision (slow performance). There's no middle ground.

4. Operational Complexity

Managing servers means patching, monitoring, log rotation, security updates, and capacity planning. Every hour spent on infrastructure is an hour not spent on features that differentiate your product.

Enter event-driven batch processing with containers. Instead of always-on servers, you run tasks only when there's work to do. Instead of manual scaling, your infrastructure responds automatically to workload changes. Instead of managing servers, you define tasks and let AWS handle the rest.

Sounds too good to be true? Let me show you how it works.

Understanding Event-Driven Architecture

Before we dive into specific patterns, let's understand what "event-driven" means in this context.

Traditional batch processing is time-driven: "Run this job every day at 2 AM."

Event-driven batch processing is trigger-driven: "Run this job when something happens."

That "something" could be:

A message arriving in a queue
A file uploaded to S3
An API request
A schedule (yes, scheduled tasks can be event-driven too!)
A state change in your application

The key difference is responsiveness. Time-driven systems process on a fixed schedule regardless of actual need. Event-driven systems process in response to actual demand.

This shift changes everything:

Costs: Pay only when processing
Performance: Respond immediately to demand
Scalability: Automatically match capacity to workload
Simplicity: No capacity planning needed

Now let's explore three production-ready patterns that implement this approach.

Pattern 1: Scheduled Tasks with ECS and EventBridge

Let's start simple. Not every batch job needs to be event-driven in the reactive sense. Sometimes you genuinely need to run something on a schedule. But that doesn't mean you need servers running 24/7.

Architecture Overview

EventBridge (formerly CloudWatch Events) acts as your scheduler. At the specified time, it triggers an ECS task. The task runs your containerized application, completes its work, and terminates. You pay only for the compute time used typically minutes, not hours.

Real-World Use Case: Daily Sales Reports

Let me walk you through a concrete example I implemented for an e-commerce platform.

Requirement: Generate daily sales reports aggregating data from RDS, calculate metrics, create PDF reports, and upload to S3 for the executive team.

Previous Solution: A t3.small instance ($15/month) running cron, active 24/7.

New Solution:

Container with Python script + ReportLab for PDF generation
EventBridge rule: cron(0 6 * * ? *) (6 AM UTC daily)
Task spec: 0.5 vCPU, 1 GB memory
Average runtime: 4 minutes

Cost Breakdown:

Fargate vCPU: $0.04048 per vCPU per hour
Fargate Memory: $0.004445 per GB per hour

Daily cost:
(0.5 vCPU × $0.04048) + (1 GB × $0.004445) = $0.024685/hour
4 minutes = 0.067 hours
Cost per run = $0.0017

Monthly cost (30 days): $0.051
Add CloudWatch, S3, SNS: ~$1.50/month
Total: ~$1.55/month

Compare this to the $15/month for an always-on t3.small. That's a 90% cost reduction. Over a year, that's $162 saved enough to fund several other AWS services.

Implementation Deep Dive

Here's what the task definition looks like in Terraform:

resource "aws_ecs_task_definition" "daily_report" {
  family                   = "daily-sales-report"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = "512"    # 0.5 vCPU
  memory                   = "1024"   # 1 GB

  container_definitions = jsonencode([{
    name  = "report-generator"
    image = "${aws_ecr_repository.reports.repository_url}:latest"

    environment = [
      {
        name  = "REPORT_TYPE"
        value = "daily-sales"
      },
      {
        name  = "S3_BUCKET"
        value = aws_s3_bucket.reports.id
      }
    ]

    logConfiguration = {
      logDriver = "awslogs"
      options = {
        "awslogs-group"         = "/ecs/daily-reports"
        "awslogs-region"        = "us-east-1"
        "awslogs-stream-prefix" = "report"
      }
    }
  }])
}

The EventBridge rule that triggers it:

resource "aws_cloudwatch_event_rule" "daily_report" {
  name                = "daily-sales-report-trigger"
  description         = "Trigger daily sales report at 6 AM UTC"
  schedule_expression = "cron(0 6 * * ? *)"
}

resource "aws_cloudwatch_event_target" "ecs_task" {
  rule     = aws_cloudwatch_event_rule.daily_report.name
  arn      = aws_ecs_cluster.reports.arn
  role_arn = aws_iam_role.eventbridge_ecs.arn

  ecs_target {
    task_count          = 1
    task_definition_arn = aws_ecs_task_definition.daily_report.arn
    launch_type         = "FARGATE"

    network_configuration {
      subnets          = aws_subnet.private[*].id
      security_groups  = [aws_security_group.ecs_tasks.id]
      assign_public_ip = false
    }
  }
}

When to Use This Pattern

Perfect for:

Daily/weekly/monthly reports
Scheduled backups
Database maintenance tasks
Periodic data cleanup
Regular data synchronization
Certificate renewals
Compliance checks

Not suitable for:

Variable workloads with unpredictable timing
Jobs triggered by external events
Processing queues of variable size
Workloads requiring high parallelism

Pro Tips:

Use SNS to get notified of failures silent failures are the worst
Set appropriate task timeouts to prevent runaway jobs
Use CloudWatch Logs Insights to track execution time trends
Consider multiple tasks for different report types rather than one monolithic task

Pattern 2: Event-Driven Auto-Scaling with ECS and SQS

Now we're getting to the heart of event-driven batch processing. This is the pattern I recommend for most production workloads, and it's where the real magic happens.

Architecture Overview

This architecture responds dynamically to workload. Messages arrive in SQS, Application Auto Scaling watches the queue depth, and ECS automatically spins up or down tasks to maintain your target backlog per task.

The Auto-Scaling Magic Explained

Let me break down exactly how auto-scaling works, because understanding this is crucial to optimizing your workload.

Application Auto Scaling uses a target tracking scaling policy. You define a target value for a metric, and AWS automatically adjusts capacity to maintain that target.

The metric we care about is "backlog per instance":

Backlog per Instance = Queue Messages / Running Tasks

Let's say you configure a target of 5 messages per task. Here's what happens in different scenarios:

Scenario 1: Steady State

Queue has 5 messages
1 task running
Backlog = 5 / 1 = 5 (matches target)
No scaling action

Scenario 2: Sudden Spike

Queue jumps to 100 messages (someone uploaded a batch)
Still 1 task running
Backlog = 100 / 1 = 100 (way above target of 5)
Scale-out action triggered
Target tasks = 100 / 5 = 20
But you've set max_tasks = 10
Service scales to 10 tasks
New backlog = 100 / 10 = 10 (still above target, but best we can do)

Scenario 3: Processing Complete

Queue is empty
10 tasks running
Backlog = 0 / 10 = 0 (below target)
Scale-in action triggered after cooldown
Service scales to min_tasks = 1

Scenario 4: Gradual Increase

Queue grows from 5 to 25 messages over 10 minutes
Service gradually scales from 1 to 5 tasks
Maintains target backlog throughout

Cooldown Periods: The Unsung Heroes

Cooldown periods prevent "flapping" rapid scaling up and down that wastes money and causes instability.

target_tracking_scaling_policy_configuration {
  target_value       = 5
  scale_in_cooldown  = 300  # 5 minutes
  scale_out_cooldown = 60   # 1 minute
}

Why different cooldowns?

Scale-out is fast (60 seconds): When messages are backing up, you want to respond quickly
Scale-in is slow (300 seconds): Tasks are cheap when idle; better to keep them around briefly in case more work arrives

I learned this the hard way. Initially, I set both to 60 seconds. The result? Constant scaling churn. A batch of 50 messages would arrive, scale up to 10 tasks, finish in 2 minutes, immediately scale down, then another batch would arrive 30 seconds later and scale back up. Each scaling action takes time, and you're actually processing slower due to all the churn.

With a 5-minute scale-in cooldown, the service stays at capacity for a bit after processing, ready for the next batch. This is especially important for "bursty" workloads with periodic spikes.

Real-World Implementation: Image Processing Pipeline

Let me share a real implementation I built for a photo sharing platform.

Requirements:

Process user-uploaded images (resize, optimize, generate thumbnails)
Variable workload: 100-1000 uploads per hour
Process each image in ~2 minutes
Store results in S3

Architecture:

S3 bucket for uploads (with event notifications)
Lambda function to create SQS messages from S3 events
SQS queue for processing jobs
ECS service with auto-scaling (1-10 tasks)
S3 bucket for processed images

The Application Code:

import boto3
import json
from PIL import Image
import io

sqs = boto3.client('sqs')
s3 = boto3.client('s3')

QUEUE_URL = os.environ['SQS_QUEUE_URL']
OUTPUT_BUCKET = os.environ['OUTPUT_BUCKET']

def process_image(bucket, key):
    """Download, process, and upload image"""
    # Download original
    response = s3.get_object(Bucket=bucket, Key=key)
    img = Image.open(io.BytesIO(response['Body'].read()))

    # Generate sizes
    sizes = {
        'thumbnail': (150, 150),
        'medium': (800, 800),
        'large': (1920, 1920)
    }

    results = []
    for size_name, dimensions in sizes.items():
        # Resize maintaining aspect ratio
        img_resized = img.copy()
        img_resized.thumbnail(dimensions, Image.LANCZOS)

        # Convert to bytes
        buffer = io.BytesIO()
        img_resized.save(buffer, format='JPEG', quality=85, optimize=True)
        buffer.seek(0)

        # Upload to S3
        output_key = f"processed/{size_name}/{key}"
        s3.put_object(
            Bucket=OUTPUT_BUCKET,
            Key=output_key,
            Body=buffer,
            ContentType='image/jpeg'
        )

        results.append({
            'size': size_name,
            'key': output_key,
            'dimensions': img_resized.size
        })

    return results

def process_messages():
    """Main processing loop"""
    messages_processed = 0
    max_messages = int(os.environ.get('MAX_MESSAGES', '10'))

    while messages_processed < max_messages:
        # Long polling for efficiency
        response = sqs.receive_message(
            QueueUrl=QUEUE_URL,
            MaxNumberOfMessages=1,
            WaitTimeSeconds=20,
            VisibilityTimeout=300  # 5 minutes
        )

        messages = response.get('Messages', [])
        if not messages:
            print("No messages available, exiting")
            break

        for message in messages:
            try:
                # Parse message
                body = json.loads(message['Body'])
                bucket = body['bucket']
                key = body['key']

                print(f"Processing {key}...")

                # Process image
                results = process_image(bucket, key)

                # Delete message on success
                sqs.delete_message(
                    QueueUrl=QUEUE_URL,
                    ReceiptHandle=message['ReceiptHandle']
                )

                messages_processed += 1
                print(f"Processed {key}: {results}")

            except Exception as e:
                print(f"Error processing message: {e}")
                # Message will return to queue after visibility timeout
                # After 3 failures, it goes to DLQ

    print(f"Processed {messages_processed} messages")

Key Design Decisions:

Long Polling (WaitTimeSeconds=20): Reduces empty receives, lowers costs, and increases efficiency
Visibility Timeout (300s): Set to 2.5× average processing time (2 minutes × 2.5 = 5 minutes)
Max Messages Limit: Task processes up to 10 messages then exits, allowing fresh tasks to start
Delete Only on Success: Messages remain in queue if processing fails, triggering retries
Graceful Error Handling: Exceptions are logged but don't crash the task

The Infrastructure: Terraform Configuration

Here's the complete auto-scaling configuration:

# SQS Queue
resource "aws_sqs_queue" "image_processing" {
  name                       = "image-processing-queue"
  visibility_timeout_seconds = 300
  message_retention_seconds  = 86400  # 24 hours
  receive_wait_time_seconds  = 20     # Long polling

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.dlq.arn
    maxReceiveCount     = 3
  })
}

# Dead Letter Queue
resource "aws_sqs_queue" "dlq" {
  name                      = "image-processing-dlq"
  message_retention_seconds = 1209600  # 14 days
}

# ECS Service
resource "aws_ecs_service" "image_processor" {
  name            = "image-processor"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.processor.arn
  launch_type     = "FARGATE"
  desired_count   = 1  # Start with 1, let auto-scaling adjust

  network_configuration {
    subnets          = aws_subnet.private[*].id
    security_groups  = [aws_security_group.ecs_tasks.id]
    assign_public_ip = false
  }
}

# Auto Scaling Target
resource "aws_appautoscaling_target" "ecs_target" {
  max_capacity       = 10
  min_capacity       = 1
  resource_id        = "service/${aws_ecs_cluster.main.name}/${aws_ecs_service.image_processor.name}"
  scalable_dimension = "ecs:service:DesiredCount"
  service_namespace  = "ecs"
}

# Auto Scaling Policy
resource "aws_appautoscaling_policy" "scale_on_sqs" {
  name               = "scale-on-sqs-depth"
  policy_type        = "TargetTrackingScaling"
  resource_id        = aws_appautoscaling_target.ecs_target.resource_id
  scalable_dimension = aws_appautoscaling_target.ecs_target.scalable_dimension
  service_namespace  = aws_appautoscaling_target.ecs_target.service_namespace

  target_tracking_scaling_policy_configuration {
    target_value       = 5.0
    scale_in_cooldown  = 300
    scale_out_cooldown = 60

    customized_metric_specification {
      metrics {
        label = "Queue depth"
        id    = "m1"
        metric_stat {
          metric {
            namespace   = "AWS/SQS"
            metric_name = "ApproximateNumberOfMessagesVisible"
            dimensions {
              name  = "QueueName"
              value = aws_sqs_queue.image_processing.name
            }
          }
          stat = "Average"
        }
        return_data = false
      }

      metrics {
        label = "Running task count"
        id    = "m2"
        metric_stat {
          metric {
            namespace   = "ECS/ContainerInsights"
            metric_name = "RunningTaskCount"
            dimensions {
              name  = "ServiceName"
              value = aws_ecs_service.image_processor.name
            }
            dimensions {
              name  = "ClusterName"
              value = aws_ecs_cluster.main.name
            }
          }
          stat = "Average"
        }
        return_data = false
      }

      metrics {
        label       = "Backlog per instance"
        id          = "e1"
        expression  = "m1 / m2"
        return_data = true
      }
    }
  }
}

Testing and Validation

I tested this setup with realistic workloads. Here's what I observed:

Test 1: Burst Processing

Sent 100 messages to queue
Started with 1 task
Within 90 seconds, scaled to 10 tasks
Processed all 100 images in 3.5 minutes
Scaled back to 1 task after 5-minute cooldown
Total time from first message to completion: ~9 minutes

Test 2: Steady Flow

Sent messages at 5/minute rate
Service maintained 1 task (perfect for this rate)
No unnecessary scaling
Average processing latency: 2 minutes

Test 3: Variable Workload

Alternated between 2/minute and 20/minute
Service scaled between 1-4 tasks
Adapted smoothly to changes
No message backlog

Cost Analysis: Real Numbers

Let's calculate the actual costs for processing 10,000 images per day:

Assumptions:

Average processing time: 2 minutes per image
Task size: 0.5 vCPU, 1 GB memory
Images spread throughout the day (not all at once)
Average of 3-4 tasks running at peak, 1 at off-peak

Fargate Costs:

Per hour rates:
vCPU: $0.04048 per vCPU-hour
Memory: $0.004445 per GB-hour

Per task per hour:
(0.5 × $0.04048) + (1 × $0.004445) = $0.024685

Total processing time per day:
10,000 images × 2 minutes = 20,000 minutes = 333 hours

Daily cost: 333 hours × $0.024685 = $8.22
Monthly cost: $8.22 × 30 = $246.60

Wait, that seems high! But remember, this is 333 task-hours. With auto-scaling averaging 3-4 tasks, that's about 100 actual hours of runtime per day. Let's recalculate:

Average concurrent tasks: 3.5
Hours per day needed: 333 task-hours / 3.5 tasks = 95 hours
But we want parallelism, so actual calendar time: 24 hours
Average load: 333 task-hours / 24 hours = 13.9 task-hours on average

More realistic calculation:
Peak hours (8 hours/day): 6 tasks average
Off-peak (16 hours/day): 1-2 tasks average

Peak: 8 hours × 6 tasks × $0.024685 = $1.18
Off-peak: 16 hours × 1.5 tasks × $0.024685 = $0.59
Daily total: $1.77
Monthly: $1.77 × 30 = $53.10

Additional AWS costs:

SQS requests: 10,000 messages/day × 30 = 300,000/month = $0.12
S3 storage (processed images): ~$5/month
CloudWatch Logs: ~$2/month
Data transfer: ~$1/month

Total monthly cost: ~$61

Compare this to running EC2 instances 24/7:

To handle peak load (6 images/minute), you'd need 3× t3.medium instances
Cost: 3 × $30 = $90/month (with reserved instances)
With on-demand: 3 × $30 × 1.4 = $126/month

Savings: $65/month (52% reduction with reserved instances, or 107% vs on-demand)

But more importantly: automatic scaling means you never drop requests during spikes, and you never pay for idle capacity during slow periods.

Dead Letter Queue: Your Safety Net

The DLQ deserves special attention because it's your early warning system for problems.

When messages go to DLQ:

Image file is corrupted
S3 permissions issue
Application bug causing exception
Timeout (processing takes longer than visibility timeout)

DLQ Configuration:

resource "aws_sqs_queue" "dlq" {
  name                      = "image-processing-dlq"
  message_retention_seconds = 1209600  # 14 days - longer than main queue
}

# Alarm when ANY message appears in DLQ
resource "aws_cloudwatch_metric_alarm" "dlq_alarm" {
  alarm_name          = "image-processing-dlq-messages"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "ApproximateNumberOfMessagesVisible"
  namespace           = "AWS/SQS"
  period              = 60
  statistic           = "Average"
  threshold           = 0
  alarm_description   = "Alert when messages land in DLQ"
  alarm_actions       = [aws_sns_topic.alerts.arn]

  dimensions = {
    QueueName = aws_sqs_queue.dlq.name
  }
}

Investigating DLQ Messages:

# View messages in DLQ
aws sqs receive-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/image-processing-dlq \
  --max-number-of-messages 10

# Common reasons and solutions:
# 1. Corrupted images → Add validation before processing
# 2. Permissions → Check IAM role has S3 access
# 3. Timeout → Increase visibility timeout or optimize processing
# 4. Bug → Fix code and redrive messages

Redriving Messages:

Once you've fixed the issue, you can redrive messages from DLQ back to the main queue:

import boto3

sqs = boto3.client('sqs')

dlq_url = 'https://sqs.us-east-1.amazonaws.com/123456789/image-processing-dlq'
main_queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789/image-processing-queue'

# Move messages back
while True:
    response = sqs.receive_message(
        QueueUrl=dlq_url,
        MaxNumberOfMessages=10
    )

    messages = response.get('Messages', [])
    if not messages:
        break

    for message in messages:
        # Send to main queue
        sqs.send_message(
            QueueUrl=main_queue_url,
            MessageBody=message['Body']
        )

        # Delete from DLQ
        sqs.delete_message(
            QueueUrl=dlq_url,
            ReceiptHandle=message['ReceiptHandle']
        )

    print(f"Redrove {len(messages)} messages")

When to Use This Pattern

Perfect for:

Image/video processing
Document conversion and analysis
ETL pipelines with variable data volumes
Batch notifications/emails
Data transformation jobs
File processing triggered by S3 uploads
Any workload with unpredictable timing or volume

Not suitable for:

Real-time processing (sub-second latency requirements)
Jobs requiring strict ordering (use FIFO queues, but note scaling limitations)
Extremely high-throughput workloads (>100k messages/minute may need different approach)

Pattern 3: Advanced Scheduling with EKS Jobs and CronJobs

Now let's explore the Kubernetes approach. This pattern is more complex but offers capabilities that the previous two don't.

Why Choose EKS for Batch Processing?

First, let's be clear: EKS adds complexity. You're managing Kubernetes, which has a steep learning curve. The control plane costs $73/month before you even run a single workload.

So why would you choose this?

You're already running EKS for other workloads
You need advanced scheduling (dependencies, DAGs, complex workflows)
You want built-in parallel processing with workqueue pattern
You need multi-tenancy with namespace isolation
You want portability (run same workloads on-prem, other clouds)

If none of these apply, stick with Pattern 2. But if you're already in the Kubernetes ecosystem, let's make it work for batch processing.

Architecture Overview

Kubernetes Jobs Explained

A Kubernetes Job creates one or more Pods and ensures they run to completion. Unlike Deployments (which restart failed Pods indefinitely), Jobs track successful completions and stop when done.

Simple Job Example:

apiVersion: batch/v1
kind: Job
metadata:
  name: data-processing-job
spec:
  completions: 1      # Run once successfully
  backoffLimit: 3     # Retry up to 3 times on failure

  template:
    spec:
      restartPolicy: Never  # Don't restart on failure (Job handles retries)

      containers:
      - name: processor
        image: myregistry/data-processor:latest
        env:
        - name: JOB_TYPE
          value: "daily-aggregation"

        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
          limits:
            cpu: "1000m"
            memory: "2Gi"

Parallel Processing:

Here's where Jobs shine. You can process multiple items in parallel:

apiVersion: batch/v1
kind: Job
metadata:
  name: parallel-data-job
spec:
  completions: 100      # Process 100 batches
  parallelism: 10       # Run 10 Pods at once
  backoffLimit: 5

  template:
    spec:
      restartPolicy: Never

      containers:
      - name: processor
        image: myregistry/data-processor:latest
        env:
        - name: BATCH_SIZE
          value: "1000"
        # Each Pod processes one batch of 1000 items
        # 10 Pods process 10 batches simultaneously
        # Total: 100 batches = 100,000 items

This creates 10 Pods initially. As each Pod completes, a new one starts until all 100 completions are reached. You process 100,000 items with 10 workers running in parallel automatically.

CronJobs: Scheduled Kubernetes Jobs

CronJobs are Jobs that run on a schedule:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: daily-report
spec:
  schedule: "0 2 * * *"  # 2 AM daily
  timeZone: "America/New_York"  # Kubernetes 1.25+

  # Job history
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3

  # Concurrency control
  concurrencyPolicy: Forbid  # Don't run if previous job still running

  # Miss tolerance
  startingDeadlineSeconds: 300  # If we miss the schedule by >5 minutes, skip

  jobTemplate:
    spec:
      backoffLimit: 2

      template:
        spec:
          restartPolicy: OnFailure

          containers:
          - name: report-generator
            image: myregistry/report-generator:latest
            env:
            - name: REPORT_TYPE
              value: "daily-summary"

Real-World Example: ETL Pipeline with Parallel Processing

Let me share an ETL implementation I built for a data analytics platform.

Requirements:

Extract data from 50 different data sources nightly
Transform and load into data warehouse
Each source takes 5-15 minutes to process
Total data: ~500 GB per night

ECS Approach Issues:

Would need 50 sequential runs (6-12 hours total) OR
Complex orchestration with Step Functions
Difficult to track which sources succeeded/failed

EKS Solution:

apiVersion: batch/v1
kind: Job
metadata:
  name: nightly-etl-{{ .Values.date }}
spec:
  completions: 50      # One per data source
  parallelism: 10      # Process 10 sources simultaneously
  backoffLimit: 2

  template:
    metadata:
      labels:
        app: etl-processor
        date: "{{ .Values.date }}"

    spec:
      serviceAccountName: etl-processor-sa  # IRSA for AWS access

      containers:
      - name: etl
        image: myregistry/etl-processor:v2.1.0

        env:
        - name: DATA_SOURCE
          valueFrom:
            configMapKeyRef:
              name: etl-sources
              key: source-$(JOB_COMPLETION_INDEX)

        - name: TARGET_DATABASE
          valueFrom:
            secretKeyRef:
              name: warehouse-credentials
              key: connection-string

        - name: JOB_COMPLETION_INDEX
          valueFrom:
            fieldRef:
              fieldPath: metadata.annotations['batch.kubernetes.io/job-completion-index']

        resources:
          requests:
            cpu: "2000m"
            memory: "4Gi"
          limits:
            cpu: "4000m"
            memory: "8Gi"

        volumeMounts:
        - name: temp-data
          mountPath: /tmp/etl

      volumes:
      - name: temp-data
        emptyDir:
          sizeLimit: "20Gi"

      nodeSelector:
        workload-type: batch  # Use specific node pool

ConfigMap with Data Sources:

apiVersion: v1
kind: ConfigMap
metadata:
  name: etl-sources
data:
  source-0: "s3://data/source-crm/"
  source-1: "s3://data/source-analytics/"
  source-2: "s3://data/source-warehouse/"
  # ... 47 more sources

This setup processes 10 sources in parallel. As each completes, a new Pod starts for the next source. Total time: ~90 minutes instead of 6-12 hours.

Benefits:

Built-in parallelism (no custom orchestration)
Easy to track completions (kubectl get pods)
Automatic retries on failure
Simple to add/remove sources (edit ConfigMap)

IRSA: Secure AWS Access from Kubernetes

One challenge with EKS is accessing AWS services securely. IRSA (IAM Roles for Service Accounts) solves this elegantly.

Without IRSA (bad):

Store AWS credentials in Secrets
Credentials could leak
Difficult rotation
Shared credentials across workloads

With IRSA (good):

Each Service Account gets a unique IAM role
Temporary credentials injected automatically
No long-lived credentials
Fine-grained permissions per workload

Setup:

# IAM Role
resource "aws_iam_role" "etl_processor" {
  name = "eks-etl-processor-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${local.oidc_provider}"
      }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${local.oidc_provider}:sub" = "system:serviceaccount:default:etl-processor-sa"
        }
      }
    }]
  })
}

# IAM Policy
resource "aws_iam_role_policy" "etl_processor" {
  role = aws_iam_role.etl_processor.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:GetObject",
          "s3:ListBucket"
        ]
        Resource = [
          "arn:aws:s3:::data/*",
          "arn:aws:s3:::data"
        ]
      },
      {
        Effect = "Allow"
        Action = [
          "rds:DescribeDBInstances"
        ]
        Resource = "*"
      }
    ]
  })
}

Kubernetes Service Account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: etl-processor-sa
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/eks-etl-processor-role

Your Pods automatically get temporary AWS credentials with only the permissions they need.

Cost Comparison

EKS Costs (for ETL example):

EKS Control Plane: $73/month

Worker Nodes (for batch processing):
- Use managed node group with auto-scaling
- t3.xlarge instances (4 vCPU, 16 GB) for batch workloads
- On-demand: $0.1664/hour × 24 × 30 = $120/month per node
- Reserved (1-year): ~$75/month per node
- Spot: ~$50/month per node

For ETL running nightly (90 minutes/day):
- 2 spot instances: $100/month
- Actually running: 90 min/day = 45 hours/month
- Effective cost: 45/720 × $100 = $6.25 for compute
- Control plane: $73
- Total: ~$79/month

BUT: Control plane is shared across all EKS workloads
If you're already running EKS, incremental cost is just the nodes: ~$6-50/month depending on spot vs reserved

When EKS Makes Sense:

You're already paying for EKS control plane
You need to run multiple batch workloads (share the $73 cost)
Complex workflows benefit from Kubernetes orchestration

When ECS is Better:

Standalone batch workloads
Simpler requirements
Team isn't familiar with Kubernetes

Monitoring and Observability

Regardless of which pattern you choose, monitoring is crucial. Here's what I've learned about effective monitoring for batch workloads.

Essential Metrics

For ECS + SQS Pattern:

Queue Depth (ApproximateNumberOfMessagesVisible)
- Alert if > 1000 for more than 10 minutes (backlog building)
- Graph to see daily patterns
Queue Age (ApproximateAgeOfOldestMessage)
- Alert if > 30 minutes (processing too slow)
- Indicates scaling issues or stuck messages
Task Count (RunningTaskCount vs DesiredCount)
- Verify auto-scaling is working
- Alert if desired != running for > 5 minutes (scaling issues)
DLQ Depth (ApproximateNumberOfMessagesVisible on DLQ)
- Alert immediately on any messages (critical failures)
Error Rate (custom metric from application logs)
- Track success vs failure rate
- Alert if >5% failure rate

CloudWatch Dashboard

Here's a CloudWatch Dashboard configuration I use:

{
  "widgets": [
    {
      "type": "metric",
      "properties": {
        "metrics": [
          [ "AWS/SQS", "ApproximateNumberOfMessagesVisible", { "stat": "Average" } ],
          [ ".", "ApproximateNumberOfMessagesNotVisible", { "stat": "Average" } ]
        ],
        "view": "timeSeries",
        "region": "us-east-1",
        "title": "Queue Depth",
        "period": 300,
        "yAxis": { "left": { "min": 0 } }
      }
    },
    {
      "type": "metric",
      "properties": {
        "metrics": [
          [ "ECS/ContainerInsights", "RunningTaskCount", { "stat": "Average" } ],
          [ ".", "DesiredTaskCount", { "stat": "Average" } ]
        ],
        "view": "timeSeries",
        "title": "ECS Tasks",
        "period": 60
      }
    }
  ]
}

CloudWatch Logs Insights Queries

Find All Errors:

fields @timestamp, @message
| filter @message like /ERROR/
| stats count(*) as error_count by bin(5m)
| sort @timestamp desc

Average Processing Time:

fields @timestamp, @message
| filter @message like /Processing completed/
| parse @message "duration: * seconds" as duration
| stats avg(duration) as avg_time, max(duration) as max_time, min(duration) as min_time

Top Error Messages:

fields @message
| filter @message like /ERROR/
| parse @message "*ERROR*: *" as prefix, error_msg
| stats count(*) as occurrences by error_msg
| sort occurrences desc
| limit 10

Throughput Analysis:

fields @timestamp
| filter @message like /Messages Processed/
| parse @message "Messages Processed: *" as count
| stats sum(count) as total_processed by bin(1h)
| sort @timestamp

CloudWatch Alarms

Critical Alarms (page immediately):

# DLQ has messages
resource "aws_cloudwatch_metric_alarm" "dlq_messages" {
  alarm_name          = "batch-dlq-critical"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "ApproximateNumberOfMessagesVisible"
  namespace           = "AWS/SQS"
  period              = 60
  statistic           = "Average"
  threshold           = 0
  alarm_description   = "Messages in DLQ - immediate investigation needed"
  alarm_actions       = [aws_sns_topic.critical_alerts.arn]
}

# All tasks failing
resource "aws_cloudwatch_metric_alarm" "no_running_tasks" {
  alarm_name          = "batch-no-tasks-running"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = 3
  metric_name         = "RunningTaskCount"
  namespace           = "ECS/ContainerInsights"
  period              = 300
  statistic           = "Average"
  threshold           = 1
  alarm_description   = "No tasks running but queue has messages"
}

Warning Alarms (investigate soon):

# Old messages in queue
resource "aws_cloudwatch_metric_alarm" "old_messages" {
  alarm_name          = "batch-old-messages"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "ApproximateAgeOfOldestMessage"
  namespace           = "AWS/SQS"
  period              = 300
  statistic           = "Maximum"
  threshold           = 1800  # 30 minutes
  alarm_description   = "Messages stuck in queue"
  alarm_actions       = [aws_sns_topic.warnings.arn]
}

# High error rate
resource "aws_cloudwatch_metric_alarm" "high_error_rate" {
  alarm_name          = "batch-high-errors"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "ApplicationErrors"
  namespace           = "CustomApp/Batch"
  period              = 300
  statistic           = "Sum"
  threshold           = 10
  alarm_description   = "Too many application errors"
}

Production Best Practices

After running these patterns in production for several clients, here are my battle-tested recommendations.

1. Design for Idempotency

This is the most important practice. Your jobs will be retried network issues, task failures, scaling events and you need to handle it gracefully.

Bad (non-idempotent):

def process_order(order_id):
    # Get order
    order = db.get_order(order_id)

    # Update inventory
    db.execute("UPDATE inventory SET quantity = quantity - 1")

    # Charge customer
    stripe.charge(order.customer_id, order.amount)

    # Mark as processed
    db.execute("UPDATE orders SET processed = true")

If this fails after charging but before marking as processed, a retry will charge twice!

Good (idempotent):

def process_order(order_id):
    # Check if already processed
    order = db.get_order(order_id)
    if order.processed:
        logger.info(f"Order {order_id} already processed, skipping")
        return

    # Use transaction with optimistic locking
    with db.transaction():
        # Lock row
        order = db.get_order_for_update(order_id)

        if order.processed:
            return  # Another task already processed it

        # Idempotent inventory update
        result = db.execute(
            "UPDATE inventory SET quantity = quantity - 1 WHERE quantity > 0"
        )
        if result.rows_affected == 0:
            raise OutOfStockError()

        # Idempotent payment (use idempotency key)
        stripe.charge(
            order.customer_id,
            order.amount,
            idempotency_key=f"order-{order_id}"
        )

        # Mark processed
        db.execute(
            "UPDATE orders SET processed = true WHERE id = ?",
            order_id
        )

2. Set Appropriate Timeouts

SQS Visibility Timeout should be:

At least 6× your average processing time
Accounts for retries and variability
Long enough to prevent duplicate processing

Example:

Average processing: 2 minutes
99th percentile: 5 minutes
Visibility timeout: 5 × 6 = 30 minutes? No, too long.
Better: 5 × 2 = 10 minutes (allows retries within visibility window)

ECS Task Timeout:

# In task definition
"stopTimeout": 120  # Seconds to wait for graceful shutdown

Application Timeout:

# In your code
PROCESSING_TIMEOUT = int(os.environ.get('TIMEOUT', '300'))  # 5 minutes

@timeout(PROCESSING_TIMEOUT)
def process_message(message):
    # Your processing logic
    pass

3. Implement Structured Logging

Logs are your best friend when debugging. Make them useful.

Bad:

print("Processing started")
print("Error occurred")

Good:

import logging
import json

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def log_structured(level, message, **kwargs):
    log_entry = {
        'timestamp': datetime.utcnow().isoformat(),
        'level': level,
        'message': message,
        **kwargs
    }
    logger.log(getattr(logging, level), json.dumps(log_entry))

# Usage
log_structured('INFO', 'Processing started',
               message_id=message_id,
               job_type='image_processing',
               input_size=file_size)

log_structured('ERROR', 'Processing failed',
               message_id=message_id,
               error=str(e),
               traceback=traceback.format_exc())

Now your CloudWatch Logs Insights queries can easily filter and aggregate:

fields @timestamp, message, job_type, error
| filter level = "ERROR"
| stats count(*) by job_type, error

4. Right-Size Your Tasks

Don't over-provision. Monitor actual usage and adjust.

Monitor:

# Get actual resource usage
aws ecs describe-tasks \
  --cluster my-cluster \
  --tasks task-id \
  --include TAGS \
  --query 'tasks[0].containers[0].{CPU:cpu,Memory:memory}'

CloudWatch Container Insights shows detailed metrics:

CPU utilization
Memory utilization
Network I/O

If you're using 100 MB but allocated 1024 MB, you're wasting ~90% of memory costs.

Start conservatively:

256 CPU, 512 MB memory
Monitor for a week
Adjust based on actual usage
Leave 20-30% headroom for spikes

5. Use Spot for Non-Critical Workloads

Fargate Spot can save up to 70% on compute costs.

When to use Spot:

Processing can tolerate 2-minute interruption notice
You have retry logic (SQS handles this automatically)
Not time-critical (interruptions add latency)

Configuration:

resource "aws_ecs_service" "batch_processor" {
  capacity_provider_strategy {
    capacity_provider = "FARGATE_SPOT"
    weight            = 100  # 100% Spot
    base              = 0
  }

  # Or mix Spot and On-Demand
  capacity_provider_strategy {
    capacity_provider = "FARGATE_SPOT"
    weight            = 70
  }

  capacity_provider_strategy {
    capacity_provider = "FARGATE"
    weight            = 30
    base              = 1  # At least 1 On-Demand task
  }
}

My recommendation: Use 100% Spot for batch processing. The interruption rate is low (<5% in my experience), and retries handle it gracefully.

6. Implement Circuit Breakers

If your downstream service is down, don't keep hammering it.

class CircuitBreaker:
    def __init__(self, failure_threshold=5, timeout=60):
        self.failure_count = 0
        self.failure_threshold = failure_threshold
        self.timeout = timeout
        self.last_failure_time = None
        self.state = 'CLOSED'  # CLOSED, OPEN, HALF_OPEN

    def call(self, func, *args, **kwargs):
        if self.state == 'OPEN':
            if time.time() - self.last_failure_time > self.timeout:
                self.state = 'HALF_OPEN'
            else:
                raise CircuitBreakerOpen("Service unavailable")

        try:
            result = func(*args, **kwargs)
            if self.state == 'HALF_OPEN':
                self.state = 'CLOSED'
                self.failure_count = 0
            return result
        except Exception as e:
            self.failure_count += 1
            self.last_failure_time = time.time()

            if self.failure_count >= self.failure_threshold:
                self.state = 'OPEN'

            raise

# Usage
db_breaker = CircuitBreaker(failure_threshold=5, timeout=60)

def process_message(message):
    try:
        data = db_breaker.call(database.query, message['query'])
        # Process data
    except CircuitBreakerOpen:
        # Don't delete message, let it retry later
        logger.warning("Circuit breaker open, skipping message")
        return False

7. Monitor Your Costs

Set up AWS Cost Anomaly Detection:

resource "aws_ce_anomaly_monitor" "batch_processing" {
  name              = "batch-processing-monitor"
  monitor_type      = "DIMENSIONAL"
  monitor_dimension = "SERVICE"
}

resource "aws_ce_anomaly_subscription" "batch_processing" {
  name      = "batch-cost-anomalies"
  frequency = "DAILY"

  monitor_arn_list = [
    aws_ce_anomaly_monitor.batch_processing.arn
  ]

  subscriber {
    type    = "EMAIL"
    address = "devops@example.com"
  }

  threshold_expression {
    dimension {
      key           = "ANOMALY_TOTAL_IMPACT_ABSOLUTE"
      values        = ["100"]  # Alert if anomaly > $100
      match_options = ["GREATER_THAN_OR_EQUAL"]
    }
  }
}

This alerts you if costs suddenly spike (e.g., infinite loop creating tasks).

Real-World Results and Lessons Learned

Let me share some actual results from production deployments.

Case Study 1: E-Commerce Image Processing

Client: Mid-size e-commerce platform
Workload: Process product images (1000-5000/day)
Previous Setup: 2× t3.medium EC2 instances running 24/7

Implementation: ECS + SQS auto-scaling pattern

Results:

Cost: $73/month → $22/month (70% reduction)
Performance: Same average latency, better p99 (auto-scaling handles spikes)
Operational: Zero manual scaling interventions in 6 months
Incidents: 1 DLQ alert (corrupted image upload, not our bug)

Lesson Learned: Set visibility timeout properly. Initially set to 2 minutes (our average processing time), led to duplicate processing during retries. Changed to 6 minutes, problem solved.

Case Study 2: Data Analytics ETL

Client: B2B SaaS analytics platform
Workload: Nightly ETL from 50 data sources
Previous Setup: Custom scheduling on EC2, sequential processing (8-10 hours)

Implementation: EKS Jobs with parallel processing

Results:

Processing Time: 8-10 hours → 90 minutes (80% reduction)
Cost: $200/month (always-on) → $85/month including EKS control plane
Reliability: Built-in retries eliminated manual interventions
Developer Experience: Much easier to add new data sources

Lesson Learned: Kubernetes has a learning curve, but parallel Job processing is powerful. For teams already using EKS, this pattern is excellent. For teams new to Kubernetes, the complexity might not be worth it ECS with Step Functions could work.

Case Study 3: Document Generation SaaS

Client: Document automation startup
Workload: Generate PDFs from templates (variable: 10-1000/hour)
Previous Setup: Always-on containers, manual scaling

Implementation: ECS + SQS auto-scaling

Results:

Cost: $90/month → $35/month (61% reduction)
Scaling: Handles traffic spikes automatically
Time to Market: Deployed in 1 day vs 2 weeks for custom solution

Lesson Learned: Start simple. We initially designed a complex workflow with Step Functions. Realized SQS + auto-scaling solved 95% of use cases. Deployed much faster with simpler architecture.

Getting Started: Your Action Plan

Ready to implement one of these patterns? Here's your step-by-step action plan.

Step 1: Choose Your Pattern

Use Pattern 1 (ECS Scheduled) if:

You need time-based execution only
Workload is predictable and consistent
You want the simplest possible setup

Use Pattern 2 (ECS + SQS) if:

Workload varies throughout the day/week
You need event-driven processing
You want automatic scaling
This is my recommendation for 80% of use cases

Use Pattern 3 (EKS Jobs) if:

You're already running EKS
You need complex parallel processing
You have Kubernetes expertise
You need advanced orchestration

Step 2: Set Up Your Development Environment

# Install required tools

# Install AWS CLI (v2)
sudo apt update
sudo apt install -y unzip curl

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws --version


# Configure AWS CLI
aws configure

# Install Terraform (latest)
sudo apt update && sudo apt install -y gnupg software-properties-common curl

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
sudo tee /etc/apt/sources.list.d/hashicorp.list

sudo apt update && sudo apt install -y terraform
terraform -version

# Install kubectl (latest stable)
sudo apt update && sudo apt install -y curl apt-transport-https gnupg

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | \
sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] \
https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /" | \
sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt update && sudo apt install -y kubectl
kubectl version --client


# Clone the repository
git https://github.com/rifkhan107/aws-batch-processing-containers.git
cd aws-batch-processing-containers

Step 3: Deploy Pattern 2 (Recommended Starting Point)

# Navigate to pattern directory
cd ecs-sqs-autoscaling

# Deploy infrastructure
cd terraform
terraform init
terraform apply

# Note the outputs
ECR_URL=$(terraform output -raw ecr_repository_url)
QUEUE_URL=$(terraform output -raw sqs_queue_url)

# Build and push Docker image
cd ../docker
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin $ECR_URL

docker build -t batch-processor .
docker tag batch-processor:latest $ECR_URL:latest
docker push $ECR_URL:latest

# Update ECS service to use new image
cd ../terraform
aws ecs update-service \
  --cluster $(terraform output -raw ecs_cluster_name) \
  --service $(terraform output -raw ecs_service_name) \
  --force-new-deployment

# Test with sample messages
LAMBDA_FUNCTION=$(terraform output -raw lambda_function_name)
aws lambda invoke \
  --function-name $LAMBDA_FUNCTION \
  --payload '{"num_messages": 20}' \
  response.json

# Watch it scale
watch -n 5 "aws ecs describe-services \
  --cluster $(terraform output -raw ecs_cluster_name) \
  --services $(terraform output -raw ecs_service_name) \
  --query 'services[0].[desiredCount,runningCount]'"

Step 4: Customize for Your Use Case

Modify the application (docker/app.py):
- Replace processing logic with your actual work
- Adjust environment variables
- Add any dependencies to requirements.txt
Tune auto-scaling (terraform/variables.tf):
- Adjust target_messages_per_task (default: 5)
- Set min_tasks and max_tasks based on your needs
- Modify task CPU and memory
Set up monitoring (monitoring/):
- Deploy CloudWatch Dashboard
- Configure alarms
- Set up SNS notifications

Step 5: Test Thoroughly

# Send test load
for i in {1..5}; do
  aws lambda invoke \
    --function-name $LAMBDA_FUNCTION \
    --payload '{"num_messages": 50}' \
    response-$i.json
done

# Monitor CloudWatch Logs
aws logs tail /ecs/ecs-sqs-batch-sqs-processor --follow

# Check queue depth
aws sqs get-queue-attributes \
  --queue-url $QUEUE_URL \
  --attribute-names ApproximateNumberOfMessagesVisible

# Verify results in S3
aws s3 ls s3://$(terraform output -raw s3_bucket_name)/results/ --recursive

Step 6: Deploy to Production

Pre-production checklist:

[ ] All alarms configured and tested
[ ] DLQ monitoring in place
[ ] Task timeout appropriate for your workload
[ ] IAM roles follow least-privilege
[ ] Costs estimated and approved
[ ] Team trained on troubleshooting
[ ] Runbook documented

Deployment:

Deploy to staging first
Run production-like load tests
Monitor for 1 week
Gradually route production traffic
Monitor costs and performance

Conclusion

Event-driven batch processing with containers represents a fundamental shift from traditional always-on infrastructure. Instead of paying for capacity you might need, you pay for capacity you actually use. Instead of manual scaling, you get automatic responsiveness to demand.

The three patterns I've shared scheduled tasks, SQS auto-scaling, and EKS Jobs cover the vast majority of batch processing use cases. For most teams, I recommend starting with Pattern 2 (ECS + SQS auto-scaling). It offers the best balance of cost savings, automatic scaling, and operational simplicity.

Key takeaways:

Event-driven saves money (50-90% in real deployments)
Auto-scaling eliminates operational burden (no more manual capacity management)
Start simple (Pattern 2 handles 80% of use cases)
Monitor everything (especially your DLQ)
Design for idempotency (retries will happen)

The complete code, infrastructure templates, and deployment guides are available on GitHub. Everything is production-ready and battle-tested.

What will you build with this? I'd love to hear about your batch processing challenges and how these patterns help solve them. Find me on Twitter, LinkedIn, or open an issue on GitHub.

Happy building, and may your batch jobs scale automatically! 🚀

Additional Resources

GitHub Repository: https://github.com/rifkhan107/aws-batch-processing-containers - Complete code and deployment guides
AWS ECS Best Practices: https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/
AWS SQS Developer Guide: https://docs.aws.amazon.com/sqs/
EKS Best Practices: https://aws.github.io/aws-eks-best-practices/

From AI to Arcade: Building a Pygame Adventure with Amazon Q

JM Rifkhan — Wed, 21 May 2025 11:55:06 +0000

Falling Stars Game: Powered by Amazon Q CLI

The Falling Stars Game is an addictive arcade game I built entirely using Amazon Q CLI, a generative AI coding assistant that made turning my ideas into reality a breeze. Using Python and Pygame, this game puts you in control of an airship, catching falling objects, dodging deadly ones, and snagging power-ups to rack up points. In this post, I’ll dive into how Amazon Q CLI brought this game to life, highlight the game’s core features, and share a glimpse of the development process with a screenshot of Amazon Q in action. Whether you’re a coder or a gamer, this project shows what AI can do!

Installing Amazon Q CLI on Windows via WSL

Amazon Q CLI doesn't have a native Windows version, but it can be easily installed and used through the Windows Subsystem for Linux (WSL). This method gives you access to a Linux environment directly from your Windows machine, making it straightforward to work with the CLI tools designed for Linux.

Here's a step-by-step guide to get Amazon Q CLI running on Windows using WSL with Ubuntu:

Installing Amazon Q CLI on Ubuntu (WSL)

Check glibc Version Before downloading, determine which installer version you need by checking your glibc version:

ldd --version

If glibc is 2.34 or newer, use the standard version.

If glibc is older, use the musl version.

Download the Appropriate Installer Choose the correct command based on your architecture and glibc version.

Standard version (glibc 2.34+)

Linux x86-64:

curl --proto '=https' --tlsv1.2 -sSf "https://desktop-release.q.us-east-1.amazonaws.com/latest/q-x86_64-linux.zip" -o "q.zip"

Linux ARM (aarch64):

curl --proto '=https' --tlsv1.2 -sSf "https://desktop-release.q.us-east-1.amazonaws.com/latest/q-aarch64-linux.zip" -o "q.zip"
Musl version (glibc < 2.34)

Linux x86-64 (musl):

curl --proto '=https' --tlsv1.2 -sSf "https://desktop-release.q.us-east-1.amazonaws.com/latest/q-x86_64-linux-musl.zip" -o "q.zip"

Linux ARM (aarch64, musl):

curl --proto '=https' --tlsv1.2 -sSf "https://desktop-release.q.us-east-1.amazonaws.com/latest/q-aarch64-linux-musl.zip" -o "q.zip"

Install Amazon Q CLI Unzip the package and run the installer:

unzip q.zip ./q/install.sh

By default, Amazon Q CLI will be installed to: ~/.local/bin.

Configure AWS Credentials (Optional) If you haven't already, configure your AWS credentials within the WSL environment: aws configure

This step is necessary for Amazon Q CLI to interact with your AWS account.

Configure your AWS credentials within the WSL environment using the following command:

q login

For additional options and detailed instructions for other platforms, refer to the official Amazon Q CLI Installation Documentation.

Amazon Q CLI: My Coding Sidekick

Amazon Q CLI is an AI tool that generates code, suggests features, and debugs issues through simple command-line prompts. I used it to build Falling Stars Game from the ground up, starting with basic concepts and iterating to a polished product. With prompts like “create a Pygame game where a player catches falling objects,” Amazon Q churned out working code, suggested additions like power-ups and levels, and helped fix bugs when things went off track. Its ability to understand my intent and provide Pygame-specific solutions saved hours of work.

The CLI’s conversational style let me refine the game step-by-step, from spawning objects to adding sound effects. For example, when collision detection was glitchy, a prompt like “fix object collision in Pygame” got me a corrected algorithm. The screenshot below (to be added) captures Amazon Q in action, showing how it shaped the game’s core mechanics.

Amazon Q CLI in Action

Initially, I gave a simple prompt to Amazon Q, like:

'Can you guide me to create a game using Pygame?'

It gave me the Python code for the game in a single file, so I asked to structure it like a proper project. As a result, Amazon Q provided me with the following file structure for my game project:

Amazon Q suggested this clean setup for the game:

falling_stars_game/
├── README.md
├── assets
│   ├── images
│   │   ├── background.jpg
│   │   ├── dangerous_object.png
│   │   ├── normal_object.png
│   │   ├── player.png
│   │   ├── powerup_life.png
│   │   ├── powerup_slower.png
│   │   ├── powerup_wider.png
│   │   └── special_object.png
│   └── sounds
│       ├── background.mp3
│       ├── catch.wav
│       ├── gameover.wav
│       └── powerup.wav
├── run_game.py
└── src
    └── game.py

assets/: Holds images and sounds.
src/game.py: Core game logic.
run_game.py: Launches the game.
README.md: Guides setup and gameplay.

Game Overview

Falling Stars Game is a fast-paced arcade challenge set on an 800x600 screen. You control a movable brick wall to catch falling stars for points while avoiding red ones that end the game. Power-ups occasionally drop to give you an edge. With custom graphics, sound effects, and toggleable music, the game feels polished and immersive. A five-level difficulty system keeps the gameplay engaging. Every component — from the game loop to the user interface — was built using the Amazon Q CLI.

Key Features

Objects in the game:

⭐ Purple-background star: +1 point, falls at a steady speed.
⭐ White-background star (Special): +5 points, faster and harder to catch.
🔥 Red fire object: Game over if caught — avoid at all costs.
❤️ Heart: Grants an extra life.
⏱️ Green timer clock: Temporarily slows down all falling objects, making them easier to catch.
🔷 Blue wider object: Temporarily increases the width of your brick wall, making catching objects easier.

Levels:

Five levels, advancing every 50 points. Higher levels mean faster objects, quicker spawns, and more green objects.

Visuals & Audio:

Custom .png images for the brick wal, objects, and power-ups.
Sounds for catches (catch.wav), power-ups (powerup.wav), and game over (gameover.wav).
Background music (background.mp3), toggleable with the M key.
UI shows score, level, lives (as icons), and power-up timers.

Controls:

Left/Right Arrows: Move the airship.
P: Pause/unpause.
M: Toggle music.
R: Restart after game over.

Code Highlights

Here's the complete game.py implementation:

import pygame
import random
import sys
import os
from pathlib import Path

# Get the base directory
BASE_DIR = Path(__file__).resolve().parent.parent
ASSETS_DIR = BASE_DIR / "assets"
IMAGES_DIR = ASSETS_DIR / "images"
SOUNDS_DIR = ASSETS_DIR / "sounds"

# Initialize Pygame
pygame.init()
pygame.mixer.init()  # Initialize sound mixer

# Screen dimensions
SCREEN_WIDTH = 800
SCREEN_HEIGHT = 600

# Colors
WHITE = (255, 255, 255)
BLACK = (0, 0, 0)
RED = (255, 0, 0)
BLUE = (0, 0, 255)
GREEN = (0, 255, 0)
YELLOW = (255, 255, 0)
PURPLE = (128, 0, 128)

# Create the screen
screen = pygame.display.set_mode((SCREEN_WIDTH, SCREEN_HEIGHT))
pygame.display.set_caption("Falling Stars Game")

# Load images
try:
    background_img = pygame.image.load(IMAGES_DIR / "background.jpg").convert()
    background_img = pygame.transform.scale(background_img, (SCREEN_WIDTH, SCREEN_HEIGHT))

    # Load normal player image
    player_img = pygame.image.load(IMAGES_DIR / "player.png").convert_alpha()
    # Scale the player image to the correct size for the game
    player_img = pygame.transform.scale(player_img, (100, 80))  # Increased height to show more of the image

    normal_obj_img = pygame.image.load(IMAGES_DIR / "normal_object.png").convert_alpha()
    normal_obj_img = pygame.transform.scale(normal_obj_img, (50, 50))

    special_obj_img = pygame.image.load(IMAGES_DIR / "special_object.png").convert_alpha()
    special_obj_img = pygame.transform.scale(special_obj_img, (50, 50))

    dangerous_obj_img = pygame.image.load(IMAGES_DIR / "dangerous_object.png").convert_alpha()
    dangerous_obj_img = pygame.transform.scale(dangerous_obj_img, (50, 50))

    powerup_wider_img = pygame.image.load(IMAGES_DIR / "powerup_wider.png").convert_alpha()
    powerup_wider_img = pygame.transform.scale(powerup_wider_img, (50, 50))

    powerup_slower_img = pygame.image.load(IMAGES_DIR / "powerup_slower.png").convert_alpha()
    # Make the slower powerup larger (60x60 pixels)
    powerup_slower_img = pygame.transform.scale(powerup_slower_img, (60, 60))

    powerup_life_img = pygame.image.load(IMAGES_DIR / "powerup_life.png").convert_alpha()
    powerup_life_img = pygame.transform.scale(powerup_life_img, (50, 50))

    # Dictionary to store images for easy access
    images = {
        'player': player_img,
        'normal': normal_obj_img,
        'special': special_obj_img,
        'dangerous': dangerous_obj_img,
        'wider': powerup_wider_img,
        'slower': powerup_slower_img,
        'extra_life': powerup_life_img
    }

except Exception as e:
    print(f"Error loading images: {e}")
    # We'll use colored rectangles if images fail to load
    images = {}

# Load sounds
try:
    catch_sound = pygame.mixer.Sound(SOUNDS_DIR / "catch.wav")
    powerup_sound = pygame.mixer.Sound(SOUNDS_DIR / "powerup.wav")
    game_over_sound = pygame.mixer.Sound(SOUNDS_DIR / "gameover.wav")

    # Set volume
    catch_sound.set_volume(0.5)
    powerup_sound.set_volume(0.7)
    game_over_sound.set_volume(0.8)

    # Try to load background music
    try:
        pygame.mixer.music.load(SOUNDS_DIR / "background.mp3")
        pygame.mixer.music.set_volume(0.3)
        pygame.mixer.music.play(-1)  # -1 means loop indefinitely
        music_playing = True
    except:
        print("Background music not found")
        music_playing = False

except Exception as e:
    print(f"Error loading sounds: {e}")
    # Create dummy sound objects
    catch_sound = None
    powerup_sound = None
    game_over_sound = None
    music_playing = False

# Player settings
player_width = 100
player_height = 80  # Increased height to show more of the image
player_x = SCREEN_WIDTH // 2 - player_width // 2
player_y = SCREEN_HEIGHT - 90  # Moved player up a bit to accommodate taller image
player_speed = 10
player_lives = 3

# Object settings
object_size = 50
objects = []
spawn_timer = 0

# Power-up settings
powerups = []
powerup_timer = 0
powerup_delay = 300  # Frames between powerup spawns
active_powerups = {}
wider_level = 1  # Track the current width level (1=normal, 2=double, 3=triple)

# Game settings
score = 0
level = 1
font = pygame.font.Font(None, 36)
small_font = pygame.font.Font(None, 24)
clock = pygame.time.Clock()
game_over = False
paused = False
music_playing = music_playing if 'music_playing' in locals() else False

# Level settings
level_settings = {
    1: {"spawn_delay": 60, "object_speed": 3, "special_chance": 0.1},
    2: {"spawn_delay": 50, "object_speed": 4, "special_chance": 0.15},
    3: {"spawn_delay": 40, "object_speed": 5, "special_chance": 0.2},
    4: {"spawn_delay": 30, "object_speed": 6, "special_chance": 0.25},
    5: {"spawn_delay": 20, "object_speed": 7, "special_chance": 0.3}
}

# Current level settings
current_settings = level_settings[1]
spawn_delay = current_settings["spawn_delay"]
object_speed = current_settings["object_speed"]
special_chance = current_settings["special_chance"]

def draw_player(x, y):
    if 'player' in images:
        # If we have a player image, use it
        if 'wider' in active_powerups and active_powerups['wider'] > 0:
            # Draw multiple player images side by side based on wider_level
            img_width = player_img.get_width()

            # Draw the images side by side with slight overlap
            for i in range(wider_level):
                pos_x = x + (i * (img_width - 20))  # Overlap by 20 pixels
                screen.blit(images['player'], (pos_x, y))
        else:
            # Use normal airship image
            screen.blit(images['player'], (x, y))
    else:
        # Fallback to a rectangle
        if 'wider' in active_powerups and active_powerups['wider'] > 0:
            pygame.draw.rect(screen, BLUE, [x, y, player_width, player_height])
        else:
            pygame.draw.rect(screen, BLUE, [x, y, 100, player_height])

    # Draw player lives
    for i in range(player_lives):
        if 'extra_life' in images:
            screen.blit(pygame.transform.scale(images['extra_life'], (20, 20)), (10 + i * 30, 40))
        else:
            pygame.draw.rect(screen, RED, [10 + i * 30, 40, 20, 20])

def create_object():
    x = random.randint(0, SCREEN_WIDTH - object_size)
    obj_type = random.choices(
        ['normal', 'special', 'dangerous'], 
        weights=[0.7, special_chance, 0.3-special_chance], 
        k=1
    )[0]

    speed_variation = random.uniform(0.8, 1.2)

    if obj_type == 'normal':
        color = RED
        points = 1
        speed = object_speed * speed_variation
    elif obj_type == 'special':
        color = GREEN
        points = 5
        speed = object_speed * 1.2 * speed_variation
    else:  # dangerous
        color = PURPLE
        points = 0  # Points don't matter as game will end if caught
        speed = object_speed * 0.9 * speed_variation

    return {'x': x, 'y': 0, 'type': obj_type, 'color': color, 'points': points, 'speed': speed}

def create_powerup():
    x = random.randint(0, SCREEN_WIDTH - object_size)
    powerup_type = random.choice(['wider', 'slower', 'extra_life'])

    if powerup_type == 'wider':
        color = YELLOW
        size = object_size
    elif powerup_type == 'slower':
        color = (0, 255, 255)  # Cyan
        size = 60  # Larger size for slower powerup
    else:  # extra_life
        color = (255, 105, 180)  # Pink
        size = object_size

    return {'x': x, 'y': 0, 'type': powerup_type, 'color': color, 'speed': object_speed * 0.7, 'size': size}

def draw_objects():
    for obj in objects:
        if obj['type'] in images:
            screen.blit(images[obj['type']], (obj['x'], obj['y']))
        else:
            pygame.draw.rect(screen, obj['color'], [obj['x'], obj['y'], object_size, object_size])

def draw_powerups():
    for pu in powerups:
        if pu['type'] in images:
            # Special handling for the slower powerup to make it larger
            if pu['type'] == 'slower':
                screen.blit(images[pu['type']], (pu['x'] - 5, pu['y'] - 5))  # Adjust position to center it
            else:
                screen.blit(images[pu['type']], (pu['x'], pu['y']))
        else:
            # Fallback to colored rectangles
            if pu['type'] == 'slower':
                # Draw a larger rectangle for slower powerup
                pygame.draw.rect(screen, pu['color'], [pu['x'] - 5, pu['y'] - 5, 60, 60])
                # Draw a "S" on the powerup
                text = small_font.render("S", True, BLACK)
                screen.blit(text, (pu['x'] + 25, pu['y'] + 20))
            else:
                pygame.draw.rect(screen, pu['color'], [pu['x'], pu['y'], object_size, object_size])
                # Draw a "P" on the powerup
                text = small_font.render("P", True, BLACK)
                screen.blit(text, (pu['x'] + 20, pu['y'] + 15))

def show_score():
    score_text = font.render(f"Score: {score}", True, WHITE)
    level_text = font.render(f"Level: {level}", True, WHITE)
    lives_text = font.render(f"Lives: {player_lives}", True, WHITE)
    screen.blit(score_text, (10, 10))
    screen.blit(level_text, (SCREEN_WIDTH - 120, 10))
    screen.blit(lives_text, (10, 50))

    # Show music status
    music_text = small_font.render("Music: " + ("ON" if music_playing else "OFF") + " (M to toggle)", True, WHITE)
    screen.blit(music_text, (SCREEN_WIDTH - 220, 50))

    # Show active powerups with icons
    y_offset = 90
    for powerup_type, time_left in active_powerups.items():
        if time_left > 0:
            if powerup_type in images:
                # Draw a small version of the powerup icon
                small_icon = pygame.transform.scale(images[powerup_type], (20, 20))
                screen.blit(small_icon, (10, y_offset))

                # For wider powerup, show the current level
                if powerup_type == 'wider':
                    powerup_text = small_font.render(f"{powerup_type.capitalize()} (x{wider_level}): {time_left//60}s", True, YELLOW)
                else:
                    powerup_text = small_font.render(f"{powerup_type.capitalize()}: {time_left//60}s", True, YELLOW)

                screen.blit(powerup_text, (35, y_offset + 2))  # Align text with icon
            else:
                # Fallback if image not available
                if powerup_type == 'wider':
                    powerup_text = small_font.render(f"{powerup_type.capitalize()} (x{wider_level}): {time_left//60}s", True, YELLOW)
                else:
                    powerup_text = small_font.render(f"{powerup_type.capitalize()}: {time_left//60}s", True, YELLOW)
                screen.blit(powerup_text, (10, y_offset))
            y_offset += 25

def check_level_up():
    global level, current_settings, spawn_delay, object_speed, special_chance

    # Level up every 50 points
    new_level = min(5, 1 + score // 50)

    if new_level > level:
        level = new_level
        current_settings = level_settings[level]
        spawn_delay = current_settings["spawn_delay"]
        object_speed = current_settings["object_speed"]
        special_chance = current_settings["special_chance"]

        # Show level up message
        return True
    return False

def apply_powerup(powerup_type):
    global player_width, object_speed, player_lives, wider_level

    if powerup_type == 'wider':
        # Increase the wider level (max 3)
        wider_level = min(3, wider_level + 1)

        # Set width based on wider_level
        if wider_level == 2:
            player_width = 160  # Enough for 2 images with overlap
        elif wider_level == 3:
            player_width = 220  # Enough for 3 images with overlap

        active_powerups['wider'] = 600  # 10 seconds (60 fps * 10)
    elif powerup_type == 'slower':
        object_speed *= 0.7
        active_powerups['slower'] = 300  # 5 seconds
    elif powerup_type == 'extra_life':
        player_lives = min(5, player_lives + 1)
    # Play powerup sound
    if powerup_sound:
        powerup_sound.play()

def update_powerups():
    global player_width, object_speed, wider_level

    for powerup_type in list(active_powerups.keys()):
        active_powerups[powerup_type] -= 1

        if active_powerups[powerup_type] <= 0:
            # Powerup expired
            if powerup_type == 'wider':
                player_width = 100  # Reset to default airship width
                wider_level = 1     # Reset wider level
            elif powerup_type == 'slower':
                object_speed = current_settings["object_speed"]  # Reset to level default

            del active_powerups[powerup_type]

def show_message(message, y_offset=0):
    text = font.render(message, True, WHITE)
    text_rect = text.get_rect(center=(SCREEN_WIDTH // 2, SCREEN_HEIGHT // 2 + y_offset))
    # Draw a semi-transparent background for the text
    s = pygame.Surface((text_rect.width + 20, text_rect.height + 20))
    s.set_alpha(200)
    s.fill(BLACK)
    screen.blit(s, (text_rect.x - 10, text_rect.y - 10))
    screen.blit(text, text_rect)

# Main game loop
running = True
level_up_message_timer = 0
show_level_up = False

while running:
    # Handle events
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            running = False
        elif event.type == pygame.KEYDOWN:
            if event.key == pygame.K_p:
                paused = not paused
            elif event.key == pygame.K_m:
                # Toggle music on/off
                if music_playing:
                    pygame.mixer.music.pause()
                    music_playing = False
                else:
                    try:
                        pygame.mixer.music.unpause()
                        music_playing = True
                    except:
                        # If music wasn't loaded or was stopped (not paused)
                        try:
                            pygame.mixer.music.play(-1)
                            music_playing = True
                        except:
                            pass
            elif event.key == pygame.K_r and game_over:
                # Reset game
                game_over = False
                score = 0
                level = 1
                player_lives = 3
                player_width = 100
                wider_level = 1  # Reset wider level
                objects = []
                powerups = []
                active_powerups = {}
                player_x = SCREEN_WIDTH // 2 - player_width // 2
                current_settings = level_settings[1]
                spawn_delay = current_settings["spawn_delay"]
                object_speed = current_settings["object_speed"]
                special_chance = current_settings["special_chance"]

                # Restart background music
                try:
                    pygame.mixer.music.play(-1)
                except:
                    pass

    # Draw background
    if 'background' in locals() and background_img:
        screen.blit(background_img, (0, 0))
    else:
        # Fallback to a gradient background
        for y in range(0, SCREEN_HEIGHT, 2):
            color_value = 50 + (y / SCREEN_HEIGHT * 50)
            pygame.draw.rect(screen, (50, 50, color_value), [0, y, SCREEN_WIDTH, 2])

    if not game_over and not paused:
        # Move player
        keys = pygame.key.get_pressed()
        if keys[pygame.K_LEFT] and player_x > 0:
            player_x -= player_speed
        if keys[pygame.K_RIGHT] and player_x < SCREEN_WIDTH - player_width:
            player_x += player_speed

        # Update powerups
        update_powerups()

        # Spawn new objects
        spawn_timer += 1
        if spawn_timer >= spawn_delay:
            objects.append(create_object())
            spawn_timer = 0

        # Spawn powerups
        powerup_timer += 1
        if powerup_timer >= powerup_delay:
            powerups.append(create_powerup())
            powerup_timer = 0

        # Move objects
        for obj in objects[:]:
            obj['y'] += obj['speed']

            # Check if object is caught
            if (obj['y'] + object_size > player_y and 
                obj['y'] < player_y + player_height and
                obj['x'] + object_size > player_x and
                obj['x'] < player_x + player_width):
                objects.remove(obj)

                # Check if dangerous object was caught
                if obj['type'] == 'dangerous':
                    # Game over immediately if dangerous object is caught
                    game_over = True
                    if game_over_sound:
                        game_over_sound.play()
                    # Stop background music
                    pygame.mixer.music.stop()
                else:
                    # Add points for non-dangerous objects
                    score += obj['points']

                    # Play catch sound
                    if catch_sound:
                        catch_sound.play()

                    # Check for level up
                    if check_level_up():
                        show_level_up = True
                        level_up_message_timer = 180  # Show for 3 seconds

            # Check if object is missed
            elif obj['y'] > SCREEN_HEIGHT:
                objects.remove(obj)
                if obj['type'] != 'dangerous':  # Only lose lives for non-dangerous objects
                    player_lives -= 1
                    if player_lives <= 0:
                        game_over = True
                        if game_over_sound:
                            game_over_sound.play()
                        # Stop background music
                        pygame.mixer.music.stop()

        # Move powerups
        for pu in powerups[:]:
            pu['y'] += pu['speed']

            # Check if powerup is caught
            if (pu['y'] + (pu['size'] if 'size' in pu else object_size) > player_y and 
                pu['y'] < player_y + player_height and
                pu['x'] + (pu['size'] if 'size' in pu else object_size) > player_x and
                pu['x'] < player_x + player_width):
                powerups.remove(pu)
                apply_powerup(pu['type'])

            # Check if powerup is missed
            elif pu['y'] > SCREEN_HEIGHT:
                powerups.remove(pu)

    # Draw everything
    draw_player(player_x, player_y)
    draw_objects()
    draw_powerups()
    show_score()

    # Show level up message
    if show_level_up:
        level_up_message_timer -= 1
        show_message(f"Level Up! Level {level}", -50)
        if level_up_message_timer <= 0:
            show_level_up = False

    if game_over:
        show_message("Game Over! Press R to restart")
        show_message("Final Score: " + str(score), 50)

    if paused:
        show_message("PAUSED - Press P to continue")

    # Update display
    pygame.display.flip()
    clock.tick(60)

# Quit the game
pygame.quit()
sys.exit()

Progressive Widening Power-up

The game implements a unique widening system where collecting multiple power-ups increases your paddle width in stages:

def apply_powerup(powerup_type):
    global player_width, object_speed, player_lives, wider_level
    if powerup_type == 'wider':
        wider_level = min(3, wider_level + 1)
        if wider_level == 2:
            player_width = 160
        elif wider_level == 3:
            player_width = 220
        active_powerups['wider'] = 600

Multiple Object Types

The game features three types of falling objects with different behaviors:

def create_object():
    obj_type = random.choices(
        ['normal', 'special', 'dangerous'],
        weights=[0.7, special_chance, 0.3-special_chance],
        k=1
    )[0]

    if obj_type == 'normal':
        color = RED
        points = 1
    elif obj_type == 'special':
        color = GREEN
        points = 5
    else:  # dangerous
        color = PURPLE
        points = 0  # Game over if caught

Level System
The game automatically progresses through 5 difficulty levels:

level_settings = {
    1: {"spawn_delay": 60, "object_speed": 3, "special_chance": 0.1},
    2: {"spawn_delay": 50, "object_speed": 4, "special_chance": 0.15},
    3: {"spawn_delay": 40, "object_speed": 5, "special_chance": 0.2},
    4: {"spawn_delay": 30, "object_speed": 6, "special_chance": 0.25},
    5: {"spawn_delay": 20, "object_speed": 7, "special_chance": 0.3}
}

What Stands Out:

Assets: Loads images and sounds with fallbacks (gradient background, colored rectangles).
Spawning: create_object balances object types with random.choices, adjusting for level difficulty.
Power-Ups: apply_powerup dynamically adjusts paddle width and object speed.
UI: Shows lives as icons, power-up timers, and a semi-transparent message box for pause/game over.
Loop: Runs at 60 FPS, handling input, updates, and rendering.
Amazon Q structured the code cleanly, using pathlib for paths and suggesting a level_settings dictionary for easy tweaks.

Building with Amazon Q CLI

Developing the game was a smooth ride with Amazon Q CLI:

Starting Out: A prompt like “set up a Pygame game with a moving paddle” gave me the initial loop and player controls.
Adding Objects: “Add falling objects with different types” led to the create_object function, with Amazon Q suggesting weighted probabilities for balance.
Power-Ups: “Implement power-ups for wider paddle and extra lives” introduced the wider_level system and timers.
Polish: Prompts like “add background music and UI” brought in audio and a clean interface with lives displayed as icons.
Debugging: When power-ups didn’t reset properly, “fix power-up expiration” got me the update_powerups function. Amazon Q also caught asset loading errors early.
The CLI’s speed and smarts made iterating fun, though I had to be specific with prompts to get exactly what I wanted, like centering the “S” on the slower power-up.

How to Play game:

clone the project repository

git clone https://github.com/rifkhan107/falling_stars_game-amazaon-q

Install pygame

pip install pygame

cd falling_stars_game

cd falling_stars_game
python3 run_game.py

Play:

Use Left/Right arrows to move.
Catch red/green objects, dodge purple.
Grab power-ups for advantages.
P (pause), M (music toggle), R (restart).
Takeaways

Amazon Q CLI was a powerhouse:

Wins: Fast code generation, creative ideas (like the wider paddle progression), and quick fixes.
Hurdles: Needed clear prompts for complex features; minor manual tweaks for polish.

What’s Next?

With Amazon Q, I could:

Save high scores to a file.
Add a “shield” power-up to block purple objects.
Build a start menu with options.

Final Thoughts

Falling Stars Game is proof Amazon Q CLI can turn a game idea into reality fast. It’s fun to play, looks sharp, and was built with AI smarts. Add the screenshot to see Amazon Q in action, then try the game or use Q to code your own. Happy gaming and coding!

Setting Up Monitoring on Amazon EKS: Deploying Prometheus and Grafana

JM Rifkhan — Sun, 01 Dec 2024 08:40:28 +0000

Monitoring is a critical aspect of managing Kubernetes workloads. This blog guides you through setting up Prometheus and Grafana on an Amazon EKS cluster to monitor cluster resources and application performance effectively. We will also address potential challenges like storage provisioning and showcase the seamless integration between Prometheus and Grafana.

Step 1: Create an Amazon EKS Cluster

We assume you are starting with an Amazon EKS cluster. Here's how you can create one using eksctl.

EKS Cluster Configuration File

Create a file named cluster.yml with the following content:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: basic-cluster
  region: us-west-2

nodeGroups:
  - name: ng-1
    instanceType: m5a.large
    desiredCapacity: 2
    volumeSize: 30
    ssh:
      allow: false
    iam:
      withAddonPolicies:
        ebs: true
        cloudWatch: true
  - name: ng-2
    instanceType: m5a.large
    desiredCapacity: 2
    volumeSize: 30

iam:
  withOIDC: true`

Run the following command to create the cluster:

eksctl create cluster -f cluster.yml

Step 2: Install the AWS EBS CSI Driver Add-On

Prometheus requires persistent storage for metrics data. To dynamically provision Amazon EBS volumes, install the AWS EBS CSI Driver add-on.

eksctl create addon \
  --name aws-ebs-csi-driver \
  --cluster basic-cluster \
  --region us-west-2

Why the AWS EBS CSI Driver is Required?
The EBS CSI Driver enables Kubernetes to manage EBS volumes dynamically. Prometheus uses PersistentVolumeClaims (PVCs) to request storage, and without this driver, the PVCs cannot bind to an EBS volume, leaving the Prometheus pods in a Pending state.

Step 3: Deploy Prometheus

First, create a dedicated namespace for Prometheus:

kubectl create namespace prometheus

Add the Prometheus Community Helm repository and update Helm:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update prometheus-community

Install Prometheus with persistent storage using Helm:

 helm upgrade -i prometheus prometheus-community/prometheus \
    --namespace prometheus \
    --set alertmanager.persistence.storageClass="gp2" \
    --set server.persistentVolume.storageClass="gp2" \
    --set server.service.type=LoadBalancer

Ensure all Prometheus pods are running:

kubectl get pods -n prometheus

Prometheus is exposed via a LoadBalancer service. To access the Prometheus server UI:

kubectl get svc -n prometheus

Open your web browser and navigate to EXTERNAL-IP

Step 4: Deploy Grafana

Grafana provides advanced visualization capabilities and integrates seamlessly with Prometheus.

Add the Grafana Helm repository and update Helm:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Deploy Grafana with persistent storage:

helm upgrade -i grafana grafana/grafana \
  --namespace prometheus \
  --set persistence.storageClassName="gp2" \
  --set persistence.enabled=true \
  --set adminPassword="admin123" \
  --set service.type=LoadBalancer

Retrieve the LoadBalancer IP for Grafana:

kubectl get svc -n prometheus | grep grafana

Step 5: Import Dashboard in Grafana

Ensure you have imported the 15661 dashboard into Grafana:
Navigate to Dashboards > Import.
Enter 15661 in the import dialog and select the data source (Prometheus).
Save the imported dashboard.

Deploying a NestJS Application to Amazon ECS using GitHub Actions and Terraform

JM Rifkhan — Wed, 04 Sep 2024 10:15:55 +0000

Introduction

In this blog, we will explore how to deploy a NestJS application to Amazon ECS (Elastic Container Service) using GitHub Actions for a seamless CI/CD pipeline. We’ll leverage several AWS services, including Application Load Balancer (ALB), Elastic Container Registry (ECR), ECS, ElastiCache for Redis, IAM, EC2, S3, and VPC, to create a microservice architecture.

Please Note: In the blog, I have only considered the deployment to ECS and the CICD security aspects. For managing environmental secrets securely, please ensure you use AWS Secrets Manager or SSM Parameter Store to manage and retrieve sensitive information, such as database credentials and API keys, within your ECS tasks and pipeline.

Prerequisites

Before getting started, ensure you have the following:

An AWS account
Basic knowledge of Docker, GitHub Actions, and AWS services
A pre-configured NestJS application with the structure detailed below
Terraform installed locally

GitHub URL of the NestJS application can be found here, and the Terraform scripts can be found here.

Containerizing the NestJS Application

The first step is to containerize the NestJS application using Docker. Below is a sample Dockerfile that you can use:

FROM node:lts-alpine3.19

# Create and set the working directory inside the container
WORKDIR /app

# Copy package.json and package-lock.json files
COPY package*.json ./

# Install dependencies
RUN npm install

# Copy the rest of the application code to the working directory
COPY . .

# Build the application
RUN npm run build

# Expose port and start application
EXPOSE 3000

CMD [ "node", "dist/main.js" ]

This Dockerfile sets up the environment, installs dependencies, builds the NestJS application, and exposes port 3000, which is the default port for the application.

Create an ECS Cluster

Navigate to the Amazon ECS Console

Go to the Amazon ECS console.

Create a New Cluster:

Click on "Create Cluster".

Configure Cluster Settings:

Cluster Name:
Enter a name for your cluster. For example, nestjs-ecs-cluster.
The cluster name must be 1 to 255 characters long and can include a-z, A-Z, 0-9, hyphens (-), and underscores (_).

Infrastructure:

Select "Customized" to manually configure the cluster's infrastructure.

Configure Network Settings

VPC:

Select the VPC in which your ECS cluster will be deployed. If you don't have a specific VPC, you can use the default VPC or create a new one.

Subnets:

Select the subnets where your EC2 instances will be launched and where your tasks will run. It's recommended to use multiple subnets across different Availability Zones for redundancy.

Security Group:

Choose an Existing Security Group:
Select an existing security group that has the necessary inbound and outbound rules for your application.

Create a New Security Group:

Alternatively, create a new security group and configure the rules to allow traffic on the required ports, such as port 3000 for your NestJS application.

Auto-assign Public IP:

Set the Auto-assign public IP option to "Use subnet setting" or choose to auto-assign a public IP if your instances need to be accessible over the internet.

Review and Create the Cluster

Review Configuration:

Review all the settings you have configured for the cluster, EC2 instances, networking, and security.

Create Cluster:

Click "Create" to start the creation of your ECS cluster with the configured settings.
Wait for Cluster Creation

The creation process may take a few minutes. Once complete, you will have an ECS cluster set up with EC2 instances ready to run your tasks.

Register Task Definitions

Once your cluster is created, you can proceed to register task definitions that will run on the EC2 instances in this cluster.

Create an ECR Repository

Create a New Repository:

Click on "Create Repository".
Enter a name for your repository, such as nestjs-app-repo.
Leave the defaults for other settings unless you have specific requirements.
Click "Create Repository".

Push Your Docker Image to ECR:

After creating the repository, follow the steps provided by ECR to push your Docker image. This typically includes:
Authenticating Docker to the ECR registry.
Building your Docker image.
Tagging the image with your ECR repository URI.
Pushing the image to ECR.

Example commands:

Retrieve an authentication token and authenticate your Docker client to your registry. Use the AWS CLI:

aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/d3k6a8t8

Build your Docker image using the following command.

docker build -t nestjs-ecs-cluster/nestjs-app .

After the build completes, tag your image so you can push the image to this repository:

docker tag nestjs-ecs-cluster/nestjs-app:latest public.ecr.aws/d3k6a8t8/nestjs-ecs-cluster/nestjs-app:latest

Run the following command to push this image to your newly created AWS repository:

docker push public.ecr.aws/d3k6a8t8/nestjs-ecs-cluster/nestjs-app:latest

Creating the Task Definition

After your Docker image is available in ECR, proceed to create the task definition in the ECS console:

Create a Task Definition:

Select "Task Definitions" from the side menu and click on "Create new Task Definition".

Choose Launch Type Compatibility:

Select the launch type (EC2 or Fargate) depending on your ECS cluster configuration.

Configure Container Definitions:

Add a container definition that points to the Docker image in your ECR repository.

Set up port mappings, environment variables, resource limits, and any other required settings.

For this application, we need the following environment variables:

PORT=3000
MONGODB_URI=mongodb://localhost:27017/test
REDIS_HOST=localhost
REDIS_PORT=6379"

Save and Use the Task Definition:

Save the task definition and use it in your ECS service to deploy your NestJS application.

Creating an ECS Service

Create a New Service:

Click on "Create" under the Services tab in your cluster.

Compute Configuration:

Compute options:

Select "Capacity provider strategy" as the Compute options if you're using EC2 instances for the service.

Task Definition:

Select the task definition you previously created that references your Docker image in ECR.

Service Name:

Enter a name for your service, such as nestjs-service.
Number of Tasks:
Specify the desired number of tasks to run simultaneously. For example, you can set this to 2.

Configure the Load Balancer (Optional but Recommended):

If you want to distribute traffic across your tasks, it's recommended to use an Application Load Balancer (ALB).

Load Balancing Type:

Select "Application Load Balancer".

Load Balancer Name:

Choose your existing ALB or create a new one if needed.
Container to Load Balance:
Specify the container that will receive traffic from the load balancer. Choose the container port that you exposed in the task definition (e.g., port 3000 for your NestJS application).
Target Group:
Create a new target group or select an existing one for your ECS tasks. This target group will be used to route traffic to the running tasks.

Health Check Path:

Specify the health check path that the ALB will use to monitor the health of your tasks. For example, if you have a /health endpoint in your NestJS app, set the health check path to /health.

Review and Create:

Review the configuration of your ECS service, including task definition, load balancing, and auto-scaling settings.
Click "Create Service" to deploy your service.

Transitioning to Automated Deployment

Up until now, we have been deploying the application using the AWS Management Console manually. Now, let's focus on automation. The Terraform script will create all the necessary AWS resources for this application, and we will use GitHub Actions to automate the deployment process.

Terraform directory structure:

.
├── backend.tf
├── locals.tf
├── main.tf
├── modules
│   ├── alb
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   ├── ecr
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   ├── ecs
│   │   ├── asg.tf
│   │   ├── ecs.sh
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   ├── elasticache
│   │   ├── main.tf
│   │   ├── output.tf
│   │   └── variables.tf
│   ├── iam
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   ├── mongodb_ec2
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   ├── nsg
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   ├── s3
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   └── vpc
│       ├── main.tf
│       ├── outputs.tf
│       └── variables.tf
├── outputs.tf
├── providers.tf
├── terraform.tfvars
└── variables.tf

Steps to Create AWS Resources Using Terraform

Clone the Terraform script template:

Clone the repository containing the Terraform script to your local machine.

Navigate to the Terraform template directory:

Move into the directory where the Terraform files are located.

Run the following command to initialize the working directory that contains the Terraform configuration files:

terraform init
terraform plan
terraform apply --auto-approve

Consideration for the GitHub Actions Workflow

We need to ensure our ECS task definition is updated during the deployment process. I have created the following:

Please don't forget to update your AWS credentials under Settings > Secrets and Variables > Actions with the required AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

ECS Task Definition File:

Located at .aws/tps-microservice-td.json This contains the configuration for our ECS task, including environment variables, container configuration, and resource limits.

{
  "containerDefinitions": [
    {
      "name": "tps-container",
      "image": "861569119044.dkr.ecr.us-west-2.amazonaws.com/nestify-ecr:33a0d65a43f0b7fb2daa9d15c6ca2a9af3fdbfe3",
      "cpu": 0,
      "portMappings": [
        {
          "name": "tps-container-3000-tcp",
          "containerPort": 3000,
          "hostPort": 3000,
          "protocol": "tcp",
          "appProtocol": "http"
        }
      ],
      "essential": true,
      "environment": [
        {
          "name": "REDIS_PORT",
          "value": "6379"
        },
        {
          "name": "PORT",
          "value": "3000"
        },
        {
          "name": "MONGODB_URI",
          "value": "mongodb://54.68.197.219:27017/test"
        },
        {
          "name": "REDIS_HOST",
          "value": "nestify-redis.qtsuer.0001.usw2.cache.amazonaws.com"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/tps-microservice-td",
          "awslogs-region": "us-west-2",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ],
  "family": "tps-microservice-td",
  "taskRoleArn": "arn:aws:iam::861569119044:role/ecsTaskExecutionRole",
  "executionRoleArn": "arn:aws:iam::861569119044:role/ecsTaskExecutionRole",
  "networkMode": "bridge",
  "cpu": "128",
  "memory": "128",
  "requiresCompatibilities": [
    "EC2"
  ],
  "runtimePlatform": {
    "cpuArchitecture": "X86_64",
    "operatingSystemFamily": "LINUX"
  }
}

GitHub Actions Workflow:

The workflow file is located at .github/workflows/main.yml. This file handles the automation of our deployment process, including updating the ECS task definition when changes are made.

name: Deploy to Amazon ECS

on:
  push:
    branches:
      - master

env:
  AWS_REGION: us-west-2
  ECR_REPOSITORY: nestify-ecr
  ECS_SERVICE: tps-microservice-svc
  ECS_CLUSTER: nestify-ecs-cluster
  ECS_TASK_DEFINITION: .aws/tps-microservice-td.json
  CONTAINER_NAME: tps-container

jobs:
  build_and_deploy:
    name: Build and Deploy to ECS
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v1

      - name: Build Docker Image
        run: |
          docker build -t ${{ env.ECR_REPOSITORY }}:${{ github.sha }} .

      - name: Push Docker Image to Amazon ECR
        run: |
          docker tag ${{ env.ECR_REPOSITORY }}:${{ github.sha }} ${{ steps.login-ecr.outputs.registry }}/${{ env.ECR_REPOSITORY }}:${{ github.sha }}
          docker push ${{ steps.login-ecr.outputs.registry }}/${{ env.ECR_REPOSITORY }}:${{ github.sha }}

      - name: Fill in the new image ID in the Amazon ECS task definition
        id: task-def
        uses: aws-actions/amazon-ecs-render-task-definition@v1
        with:
          task-definition: ${{ env.ECS_TASK_DEFINITION }}
          container-name: ${{ env.CONTAINER_NAME }}
          image: ${{ steps.login-ecr.outputs.registry }}/${{ env.ECR_REPOSITORY }}:${{ github.sha }}

      - name: Deploy Amazon ECS task definition
        uses: aws-actions/amazon-ecs-deploy-task-definition@v1
        with:
          task-definition: ${{ steps.task-def.outputs.task-definition }}
          service: ${{ env.ECS_SERVICE }}
          cluster: ${{ env.ECS_CLUSTER }}
          wait-for-service-stability: true

Once your pipeline completes, check with your ALB (Application Load Balancer) endpoint to verify the deployment.

That's it! You’ve successfully deployed your application on ECS using GitHub Actions and Terraform. 🚀😊

Securing Amazon EKS: A Comprehensive Guide to Implementing HTTPS with AWS Application Load Balancer, Route 53, and ACM

JM Rifkhan — Sat, 13 Jan 2024 09:19:14 +0000

Introduction

Begin by explaining the importance of using SSL (HTTPS) for enhancing the security and credibility of web services. Emphasize the role of SSL in encrypting data and protecting sensitive information. Introduce the use of Amazon EKS for container orchestration, AWS Application Load Balancer for distributing incoming application traffic, Route 53 for domain name system (DNS) web services, and ACM for managing secure sockets layer/transport layer security (SSL/TLS) certificates.

Prerequisites

List the prerequisites for the reader:

An active AWS account with necessary permissions.
AWS cli
eksctl cli
helm

Architecture Overview

Step-by-Step Guide

1. Creating the EKS Cluster

Cluster creation process with the following command:

eksctl create cluster -f cluster-config.yaml

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: basic-cluster
  region: us-west-2
  version: "1.28"

nodeGroups:
  - name: ng-1
    instanceType: m5.large
    desiredCapacity: 2
    ssh:
      allow: true # if you want to SSH into your nodes, set this to true
      publicKeyPath: ~/.ssh/id_rsa.pub # path to your SSH public key file

2. Deploy the AWS Load Balancer Controller

AWS Elastic Load Balancing Application Load Balancer (ALB) is a popular AWS service that load balances incoming traffic at the application layer (layer 7) across multiple targets, such as Amazon EC2 instances, in multiple Availability Zones.

ALB supports multiple features including:

host or path based routing
TLS (Transport Layer Security) termination, WebSockets
HTTP/2
AWS WAF (Web Application Firewall) integration
integrated access logs, and health checks

Create IAM OIDC provider

The primary purpose of this command is to enable IAM roles for service accounts in your EKS cluster. This feature allows Kubernetes service accounts to assume IAM roles, giving you a way to manage permissions for the pods that run in your cluster. By using IAM roles for service accounts, you can follow the best practices of least privilege and not have to assign broad IAM permissions to the nodes in your cluster.

eksctl utils associate-iam-oidc-provider \
    --region us-west-2 \
    --cluster basic-cluster \
    --approve

This setup is especially useful when you need to give specific AWS permissions to certain pods running in your EKS cluster without granting those permissions to all pods on the node. It's a security best practice in managing Kubernetes clusters on AWS.

Download an IAM policy for the LBC using one of the following commands:

curl -o iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.6.1/docs/install/iam_policy.json

reate an IAM policy named AWSLoadBalancerControllerIAMPolicy. If you downloaded a different policy, replace iam-policy with the name of the policy that you downloaded.

aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam-policy.json

Take note of the policy ARN that's returned.

Create an IAM role and Kubernetes ServiceAccount for the LBC. Use the ARN from the previous step.

eksctl create iamserviceaccount \
--cluster=<cluster-name> \
--namespace=kube-system \
--name=aws-load-balancer-controller \
--attach-policy-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:policy/AWSLoadBalancerControllerIAMPolicy \
--override-existing-serviceaccounts \
--region <region-code> \
--approve

Helm install command for clusters with IRSA:

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=basic-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

After installing the AWS ALB Ingress Controller in your Amazon EKS cluster, it's important to ensure that it has been successfully deployed and is running as expected. You can verify this by checking the pods in the kube-system namespace, where the ALB Ingress Controller should be running.

Run the following command in your terminal:

kubectl get pod -n kube-system

Deploy Sample Application

Now let’s deploy a sample 2048 game into our Kubernetes cluster and use the Ingress resource to expose it to traffic:

Deploy 2048 game resources:

---
apiVersion: v1
kind: Namespace
metadata:
  name: game-2048
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: game-2048
  name: deployment-2048
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: app-2048
  replicas: 5
  template:
    metadata:
      labels:
        app.kubernetes.io/name: app-2048
    spec:
      containers:
      - image: alexwhen/docker-2048
        imagePullPolicy: Always
        name: app-2048
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  namespace: game-2048
  name: service-2048
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  type: NodePort
  selector:
    app.kubernetes.io/name: app-2048
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  namespace: game-2048
  name: ingress-2048
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - path: /*
            backend:
              serviceName: service-2048
              servicePort: 80

After few seconds, verify that the Ingress resource is enabled:

kubectl get ingress/ingress-2048 -n game-2048

You should be able to see the following output

NAME           HOSTS   ADDRESS                                                                  PORTS   AGE
ingress-2048   *       k8s-game2048-ingress2-8ae3738fd5-251279030.us-east-2.elb.amazonaws.com   80      6m20s

Configuring a Custom Domain and Enabling HTTPS

Up to this point, we have successfully deployed our application in the Amazon EKS cluster and accessed it via the default DNS name provided by the AWS Application Load Balancer (ALB). Now, let's take it a step further by configuring a separate, more user-friendly domain name for our application and enabling HTTPS to ensure secure communication.

Acquiring and Configuring a Domain Name

Domain Registration: If you don’t already have a domain, you can register one through services like Amazon Route 53 or any other domain registrar.
Route 53 Hosted Zone: Once you have your domain, set up a hosted zone in AWS Route 53. This will allow you to manage the DNS records for your domain within AWS.

Integrating the Domain with ALB

Updating DNS Records: Point your domain to the ALB by updating the DNS records in the Route 53 hosted zone. You’ll typically add an A record (or CNAME if necessary) that maps your domain name to the ALB’s DNS name.

Creating a Hosted Zone

If you haven’t already, create a new hosted zone for your domain. A hosted zone is essentially a container for the DNS settings for your domain.
Once the hosted zone is set up, you'll see a set of default record sets. These are the NS (Name Server) and SOA (Start of Authority) records.

Adding DNS Records

Click on the “Create Record” or “Create Record Set” button in the hosted zone for your domain

Configuring Record Details

Record Name: Enter the desired subdomain or leave it blank for the root domain.
Record Type: Choose 'A – IPv4 address' if you are mapping the domain to an IP address or 'CNAME' if you are mapping it to a domain name. For ALB, you’ll typically use an A record.
Value: For an A record, input the IP address of your ALB. For a CNAME, enter the ALB’s DNS name.
Routing Policy: Select “Simple routing”. This policy is used when you want to route traffic to a single resource, like your ALB.

Saving the Record Set

Review the settings and click “Create” to add the record to your hosted zone.
It might take some time for the changes to propagate through the DNS system.

Verifying the Configuration

After the records have propagated, verify that your domain correctly points to your ALB. You can do this by accessing your domain in a web browser or using a DNS lookup tool.

Issuing an SSL/TLS Certificate via AWS Certificate Manager (ACM)

With your domain now correctly pointing to your AWS infrastructure, the next crucial step is to secure it with an SSL/TLS certificate. AWS Certificate Manager simplifies this process. Here’s how to issue a certificate:

Requesting a New Certificate

Click on “Request a certificate”.
Choose “Request a public certificate” and click “Next”.

Adding Domain Names

Enter your domain name. You can request a certificate for a specific subdomain (like www.yourdomain.com) or for all subdomains under a domain using a wildcard (like *.yourdomain.com).
Click “Next” after entering your domain details.

Selecting Validation Method

Choose a validation method. ACM offers two methods: DNS validation and email validation.

DNS Validation: Recommended for its simplicity and integration with Route 53. ACM will provide DNS records that you must add to your Route 53 hosted zone. This proves that you own or control the domain.
Email Validation: ACM sends emails to the contact addresses listed for the domain in WHOIS and to a set of five common administrative addresses. You must follow the instructions in the email to validate domain ownership.

Review and Request

Review your request details and click “Confirm and request”.

Validating the Certificate

Follow the instructions provided by ACM to validate your domain, depending on the method you chose.
Once validated, the status of your certificate will change to “Issued”.

Updating the Ingress Resource for HTTPS

With the SSL/TLS certificate issued and associated with your ALB, the next step is to update the Ingress resource in your Kubernetes deployment. This will direct traffic through the ALB using HTTPS. Here's how to update your Ingress resource with the necessary annotations for the ACM certificate:

Update with Certificate ARN: Modify the alb.ingress.kubernetes.io/certificate-arn annotation to include the ARN of your newly issued ACM certificate. Replace the actual ARN value with {certificate-arn} placeholder.

Ater all the configurations and updates, your Kubernetes manifest file for the Ingress resource should look like this.

---
apiVersion: v1
kind: Namespace
metadata:
  name: game-2048
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: game-2048
  name: deployment-2048
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: app-2048
  replicas: 5
  template:
    metadata:
      labels:
        app.kubernetes.io/name: app-2048
    spec:
      containers:
      - image: public.ecr.aws/l6m2t8p7/docker-2048:latest
        imagePullPolicy: Always
        name: app-2048
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  namespace: game-2048
  name: service-2048
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  type: NodePort
  selector:
    app.kubernetes.io/name: app-2048
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: game-2048
  name: ingress-2048
  annotations:
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
    alb.ingress.kubernetes.io/certificate-arn: {certificate-arn}
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/ssl-redirect: '443'
spec:
  ingressClassName: alb
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: service-2048
            port:
              number: 80

Apply the updated configuration to your Kubernetes cluster using:

kubectl apply -f your-ingress-resource-file.yaml

Verifying the Setup

After applying the changes, it's important to verify that the Ingress is correctly configured:

Check the Ingress status using kubectl get ingress -n game-2048.
Ensure that your domain now serves traffic over HTTPS and that the browser shows a secure connection.

Conclusion

By updating the Ingress resource with the ACM certificate ARN and ensuring the proper routing and SSL redirection, your application is now securely accessible over HTTPS. This not only secures the data transmission but also improves trust with your users.

Mastering AWS ECS with CloudFormation: A Comprehensive Guide

JM Rifkhan — Tue, 19 Dec 2023 22:54:29 +0000

Introduction

In the dynamic landscape of cloud computing, efficient and secure infrastructure deployment is a cornerstone of success. AWS CloudFormation represents a powerful tool in this realm, offering automation and precision. This comprehensive guide dives into a CloudFormation template designed for establishing an Elastic Container Service (ECS) cluster across public and private subnets—a configuration that balances accessibility, security, and optimal performance.

Understanding CloudFormation and ECS

Before we delve into the template, let's understand the key concepts:

AWS CloudFormation

CloudFormation is an AWS service that helps you model and set up your Amazon Web Services resources. It allows you to use a simple text file to automate and manage the deployment of resources, avoiding the time-consuming and error-prone manual process.

AWS Elastic Container Service (ECS)

ECS is a highly scalable, high-performance container orchestration service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon EC2 instances or AWS Fargate.

The Architecture Overview

The proposed architecture uses AWS CloudFormation to automate the deployment of an ECS cluster. This architecture is designed to provide:

Scalability to handle varying loads.
Enhanced security through isolated network environments.
High availability across multiple availability zones.

Detailed Breakdown of the CloudFormation Template

This section will cover each component of the CloudFormation template in detail.

VPC and Subnet Configuration

VPC Setup
CIDR Block: The VPC is configured with a 10.0.0.0/16 CIDR block, offering a large range of IP addresses.
DNS Support: DNS support and hostnames are enabled for better network management and resolution.

Subnet Planning
Public Subnets: Two public subnets are designed for resources that need to be connected to the internet.
Private Subnets: Two private subnets are used for backend systems that don't require direct internet access.

Internet Gateway and Routing

Internet Gateway
The Internet Gateway serves as a bridge between the VPC and the internet, facilitating communication for resources in the public subnet.

Route Tables
Public Route Table: Routes traffic from the public subnet to the Internet Gateway.
Private Route Table: Manages internal traffic within the VPC.

NAT Gateways for Private Subnet Internet Access

The NAT Gateways are crucial for allowing resources in the private subnets to access the internet for updates and patches while keeping them secure from inbound internet traffic.

Elastic Load Balancing

Application Load Balancer (ALB)
Placement: Located within the public subnets to balance incoming internet traffic.
Functionality: Distributes traffic to different targets within the ECS cluster based on predefined rules, enhancing performance and fault tolerance.

ECS Cluster Setup

Fargate Integration
Serverless Approach: Utilizing AWS Fargate removes the need to provision and manage servers for containerized applications.
Cluster Configuration: The template includes the setup for an ECS cluster, ready to host containerized applications.

Security Groups and Network Access

Security Group for ALB
Regulates the traffic to and from the load balancer, ensuring that only legitimate requests reach the application services.
Container Security Group
Controls the communication between the load balancer and the ECS services, crucial for maintaining the integrity and security of the applications.

Setting Up an ECS Service and Task Definition

After discussing the theoretical aspects of the architecture and CloudFormation, it's time to show a practical implementation. Once the infrastructure is in place, the next critical steps involve creating the ECS service and task definitions.

ECS Service and Task Definition

An ECS service is a configuration that enables you to run and maintain a specified number of instances of a task definition simultaneously. A task definition is a blueprint for your application that describes one or more containers.

Creating an ECS Service

In our setup, we've created an ECS service that utilizes a task definition to deploy a react-app. Here's a breakdown of the components involved:

Launch Type: Utilizes Fargate for serverless deployment.
Load Balancer: An Application Load Balancer is configured to distribute traffic to the containers.

Network and Security Configuration

Network Configuration: The service is associated with the VPC and subnets created by our CloudFormation template.
Security Groups: Security groups are assigned to the service to control the traffic according to the predefined rules.

Testing the Load Balancer

With the ECS service deployed, the final step is to verify that the Application Load Balancer correctly distributes incoming traffic to our react-app service.

DNS Record Testing: By accessing the DNS name provided by the load balancer, you can confirm that the service is reachable and that the load balancing is functioning as expected.

CloudFormation Template and Deployment

Here, I provide the entire CloudFormation template code. This template is the blueprint for the entire infrastructure setup described above.



AWSTemplateFormatVersion: '2010-09-09'

Description: AWS core resources to create an ECS cluster spanning public and private subnets. Supports

             public facing load balancers.

Mappings:

  SubnetConfig:

    VPC:

      CIDR: '10.0.0.0/16'

    PublicOne:

      CIDR: '10.0.0.0/24'

    PublicTwo:

      CIDR: '10.0.1.0/24'

    PrivateOne:

      CIDR: '10.0.2.0/24'

    PrivateTwo:

      CIDR: '10.0.3.0/24'

Resources:

  VPC:

    Type: AWS::EC2::VPC

    Properties:

      EnableDnsSupport: true

      EnableDnsHostnames: true

      CidrBlock: !FindInMap ['SubnetConfig', 'VPC', 'CIDR']

PublicSubnetOne:

    Type: AWS::EC2::Subnet

    Properties:

      AvailabilityZone: !Select

        - 0

        - Fn::GetAZs: !Ref 'AWS::Region'

      VpcId: !Ref 'VPC'

      CidrBlock: !FindInMap ['SubnetConfig', 'PublicOne', 'CIDR']

      MapPublicIpOnLaunch: true

  PublicSubnetTwo:

    Type: AWS::EC2::Subnet

    Properties:

      AvailabilityZone: !Select

        - 1

        - Fn::GetAZs: !Ref 'AWS::Region'

      VpcId: !Ref 'VPC'

      CidrBlock: !FindInMap ['SubnetConfig', 'PublicTwo', 'CIDR']

      MapPublicIpOnLaunch: true

PrivateSubnetOne:

    Type: AWS::EC2::Subnet

    Properties:

      AvailabilityZone: !Select

        - 0

        - Fn::GetAZs: !Ref 'AWS::Region'

      VpcId: !Ref 'VPC'

      CidrBlock: !FindInMap ['SubnetConfig', 'PrivateOne', 'CIDR']

  PrivateSubnetTwo:

    Type: AWS::EC2::Subnet

    Properties:

      AvailabilityZone: !Select

        - 1

        - Fn::GetAZs: !Ref 'AWS::Region'

      VpcId: !Ref 'VPC'

      CidrBlock: !FindInMap ['SubnetConfig', 'PrivateTwo', 'CIDR']

InternetGateway:

    Type: AWS::EC2::InternetGateway

  GatewayAttachement:

    Type: AWS::EC2::VPCGatewayAttachment

    Properties:

      VpcId: !Ref 'VPC'

      InternetGatewayId: !Ref 'InternetGateway'

  PublicRouteTable:

    Type: AWS::EC2::RouteTable

    Properties:

      VpcId: !Ref 'VPC'

  PublicRoute:

    Type: AWS::EC2::Route

    DependsOn: GatewayAttachement

    Properties:

      RouteTableId: !Ref 'PublicRouteTable'

      DestinationCidrBlock: '0.0.0.0/0'

      GatewayId: !Ref 'InternetGateway'

  PublicSubnetOneRouteTableAssociation:

    Type: AWS::EC2::SubnetRouteTableAssociation

    Properties:

      SubnetId: !Ref PublicSubnetOne

      RouteTableId: !Ref PublicRouteTable

  PublicSubnetTwoRouteTableAssociation:

    Type: AWS::EC2::SubnetRouteTableAssociation

    Properties:

      SubnetId: !Ref PublicSubnetTwo

      RouteTableId: !Ref PublicRouteTable

NatGatewayOneAttachment:

    Type: AWS::EC2::EIP

    DependsOn: GatewayAttachement

    Properties:

        Domain: vpc

  NatGatewayTwoAttachment:

    Type: AWS::EC2::EIP

    DependsOn: GatewayAttachement

    Properties:

        Domain: vpc

  NatGatewayOne:

    Type: AWS::EC2::NatGateway

    Properties:

      AllocationId: !GetAtt NatGatewayOneAttachment.AllocationId

      SubnetId: !Ref PublicSubnetOne

  NatGatewayTwo:

    Type: AWS::EC2::NatGateway

    Properties:

      AllocationId: !GetAtt NatGatewayTwoAttachment.AllocationId

      SubnetId: !Ref PublicSubnetTwo

  PrivateRouteTableOne:

    Type: AWS::EC2::RouteTable

    Properties:

      VpcId: !Ref 'VPC'

  PrivateRouteOne:

    Type: AWS::EC2::Route

    Properties:

      RouteTableId: !Ref PrivateRouteTableOne

      DestinationCidrBlock: 0.0.0.0/0

      NatGatewayId: !Ref NatGatewayOne

  PrivateRouteTableOneAssociation:

    Type: AWS::EC2::SubnetRouteTableAssociation

    Properties:

      RouteTableId: !Ref PrivateRouteTableOne

      SubnetId: !Ref PrivateSubnetOne

  PrivateRouteTableTwo:

    Type: AWS::EC2::RouteTable

    Properties:

      VpcId: !Ref 'VPC'

  PrivateRouteTwo:

    Type: AWS::EC2::Route

    Properties:

      RouteTableId: !Ref PrivateRouteTableTwo

      DestinationCidrBlock: 0.0.0.0/0

      NatGatewayId: !Ref NatGatewayTwo

  PrivateRouteTableTwoAssociation:

    Type: AWS::EC2::SubnetRouteTableAssociation

    Properties:

      RouteTableId: !Ref PrivateRouteTableTwo

      SubnetId: !Ref PrivateSubnetTwo

####

  # ALB related resources

  ####

  PublicLoadBalancerSG:

    Type: AWS::EC2::SecurityGroup

    Properties:

      GroupDescription: Access to the public facing load balancer

      VpcId: !Ref 'VPC'

      SecurityGroupIngress:

          - CidrIp: 0.0.0.0/0

            IpProtocol: -1

  PublicLoadBalancer:

    Type: AWS::ElasticLoadBalancingV2::LoadBalancer

    Properties:

      Scheme: internet-facing

      LoadBalancerAttributes:

      - Key: idle_timeout.timeout_seconds

        Value: '30'

      Subnets:

        - !Ref 'PublicSubnetOne'

        - !Ref 'PublicSubnetTwo'

      SecurityGroups: [!Ref 'PublicLoadBalancerSG']

  # A dummy target group is used to setup the ALB to just drop traffic

  # initially, before any real service target groups have been added.

  DummyTargetGroupPublic:

    Type: AWS::ElasticLoadBalancingV2::TargetGroup

    Properties:

      HealthCheckIntervalSeconds: 6

      HealthCheckPath: /

      HealthCheckProtocol: HTTP

      HealthCheckTimeoutSeconds: 5

      HealthyThresholdCount: 2

      Port: 80

      Protocol: HTTP

      UnhealthyThresholdCount: 2

      VpcId: !Ref 'VPC'

  PublicLoadBalancerListener:

    Type: AWS::ElasticLoadBalancingV2::Listener

    DependsOn:

      - PublicLoadBalancer

    Properties:

      DefaultActions:

        - TargetGroupArn: !Ref 'DummyTargetGroupPublic'

          Type: 'forward'

      LoadBalancerArn: !Ref 'PublicLoadBalancer'

      Port: 80

      Protocol: HTTP

  ####

  # ECS related resources

  ####

  ECSCluster:

    Type: AWS::ECS::Cluster

    Properties:

      ClusterName: 'enhanced-architecture-fargate'

  ContainerSecurityGroup:

    Type: AWS::EC2::SecurityGroup

    Properties:

      GroupDescription: Access to the containers

      VpcId: !Ref 'VPC'

      SecurityGroupIngress:

        - IpProtocol: tcp

          FromPort: 0

          ToPort: 65535

          SourceSecurityGroupId: !Ref 'PublicLoadBalancerSG'

######################################

Outputs:

  VpcId:

    Description: The ID of the VPC that this stack is deployed in

    Value: !Ref 'VPC'

  PublicSubnetOne:

    Description: Public subnet one

    Value: !Ref 'PublicSubnetOne'

  PublicSubnetTwo:

    Description: Public subnet two

    Value: !Ref 'PublicSubnetTwo'

  PrivateSubnetOne:

    Description: Private subnet one

    Value: !Ref 'PrivateSubnetOne'

  PrivateSubnetTwo:

    Description: Private subnet two

    Value: !Ref 'PrivateSubnetTwo'

  ExternalUrl:

    Description: The url of the external load balancer

    Value: !Sub http://${PublicLoadBalancer.DNSName}

Deploying the Stack with AWS CLI

To deploy this CloudFormation stack, you can use the following AWS CLI command. This command initializes the creation of the resources as defined in the template.



aws cloudformation create-stack --capabilities CAPABILITY_IAM --stack-name enhanced-architecture --template-body file://./path_to_template_file.yaml

Benefits of This Architecture

Scalability

The setup is inherently scalable, easily adapting to increased traffic and workloads without significant manual intervention.

Security

The segregation of public and private subnets ensures a secure environment for backend services, protecting sensitive data and operations.

High Availability

The multi-AZ deployment ensures that the system remains operational and robust, even in the event of an individual zone's failure.

Cost-Efficiency

With AWS Fargate, you only pay for the resources your containers use, leading to optimized costs especially beneficial for varying workloads.

Conclusion

This guide presents a detailed view of deploying a scalable, secure, and highly available ECS cluster using AWS CloudFormation. The provided template is a robust foundation for any organization looking to leverage the power of AWS for their containerized applications. By automating infrastructure deployment, this approach not only saves time but also significantly reduces the potential for error, ensuring a reliable and efficient cloud environment.

References and Further Reading

The journey through AWS ECS and CloudFormation doesn't stop here. For those eager to delve deeper and expand their knowledge, the following resources offer a wealth of information:

AWS CloudFormation User Guide

A comprehensive guide to AWS CloudFormation, detailing concepts, template references, and best practices.

AWS CloudFormation User Guide
AWS Elastic Container Service Documentation

Everything you need to know about ECS, from basics to advanced topics, is covered in the official AWS documentation.

AWS ECS Documentation
AWS Fargate Documentation

Learn more about the serverless compute engine for containers and how to use it with ECS.

AWS Fargate Documentation
Best Practices for Security in Amazon ECS

Understand how to secure your containerized applications and infrastructure in ECS.

Security Best Practices in Amazon ECS

Peace!✌️

Using Azure DevOps Pipeline to Build, Test, Scan, and Deploy a React Application to AKS

JM Rifkhan — Fri, 14 Apr 2023 20:41:00 +0000

In this blog post, we will walk through the process of setting up an Azure DevOps pipeline to build, test, scan, and deploy a React application to Azure Kubernetes Service (AKS). We will also integrate Trivy, an open-source vulnerability scanner, to scan the Docker image for security vulnerabilities.

Prerequisites:

An Azure DevOps account
An Azure Kubernetes Service (AKS) cluster
An Azure Container Registry
Trivy installed on your Azure DevOps

Project Architecture

1. Set up the Azure DevOps Pipeline

First, create a new pipeline in your Azure DevOps project. Then, add the following YAML code to your azure-pipelines.yml file. This pipeline has four stages: Test, Build, Scan and Push Image, and Deploy.



trigger:
  branches:
    include:
    - master

variables:
  acrName: 'acr234346'
  imageName: 'crypt-react-app'
  k8sNamespace: 'default'
  k8sDeploymentName: 'deployment'


stages:
- stage: Test
  displayName: 'Test stage'
  jobs:
  - job: Test
    displayName: 'Test Job'
    pool:
      vmImage: 'ubuntu-latest'
    steps:
    - checkout: self
    - task: NodeTool@0
      inputs:
        versionSpec: '16.x'
      displayName: 'Install Node.js'
    - script: |
        npm ci
        npm run test
      displayName: 'Run tests'


- stage: Build
  displayName: 'Build stage'
  dependsOn: Test
  condition: succeeded()
  jobs:
  - job: Build
    displayName: 'Build Job'
    pool:
      vmImage: 'ubuntu-latest'
    steps:
    - checkout: self

    - task: Docker@2
      displayName: 'Build and push Docker image'
      inputs:
        containerRegistry: 'docker-reg'
        repository: '$(imageName)'
        command: 'buildAndPush'
        Dockerfile: '**/Dockerfile'


- stage: Scan
  displayName: 'Scan Image'
  dependsOn: Build
  condition: succeeded()
  jobs:
  - job: ScanImage
    displayName: 'Scan Image Job'
    pool:
      vmImage: 'ubuntu-latest'
    steps:
    - task: Docker@2
      displayName: 'Login to ACR'
      inputs:
        containerRegistry: 'docker-reg'
        repository: '$(imageName)'
        command: 'login'
    - task: trivy@1
      inputs:
        version: 'latest'
        docker: false
        loginDockerConfig: true
        image: $(acrName).azurecr.io/$(imageName):$(Build.BuildId)
        ignoreUnfixed: true
      displayName: Scan Docker image with Trivy


- stage: Deploy
  displayName: 'Deploy stage'
  dependsOn: Scan
  condition: succeeded()
  jobs:
  - deployment: Deploy
    displayName: 'Deploy Job'
    environment: 'production'
    pool:
      vmImage: 'ubuntu-latest'
    strategy:
      runOnce:
        deploy:
          steps:
          - checkout: self
          - task: Kubernetes@1
            displayName: 'Deploy Deployment to AKS'
            inputs:
              connectionType: 'Azure Resource Manager'
              azureSubscriptionEndpoint: 'my-connection'
              azureResourceGroup: 'dev-rg'
              kubernetesCluster: 'demo-cluster'
              useClusterAdmin: true
              command: 'apply'
              useConfigurationFile: true
              configurationType: 'inline'
              inline: |
                apiVersion: apps/v1
                kind: Deployment
                metadata:
                  name: deployment
                  namespace: default
                spec:
                  replicas: 1
                  selector:
                    matchLabels:
                      app: crypto-app
                  template:
                    metadata:
                      labels:
                        app: crypto-app
                    spec:
                      containers:
                      - name: $(imageName)
                        image: $(acrName).azurecr.io/$(imageName):$(Build.BuildId)
                        ports:
                        - containerPort: 5000
              secretType: 'dockerRegistry'
              containerRegistryType: 'Azure Container Registry'

          - task: Kubernetes@1
            displayName: 'Deploy Service to AKS'
            inputs:
              connectionType: 'Azure Resource Manager'
              azureSubscriptionEndpoint: 'my-connection'
              azureResourceGroup: 'dev-rg'
              kubernetesCluster: 'demo-cluster'
              useClusterAdmin: true
              command: 'apply'
              useConfigurationFile: true
              configurationType: 'inline'
              inline: |
                apiVersion: v1
                kind: Service
                metadata:
                  name: crypto-app-service
                  namespace: default
                spec:
                  selector:
                    app: crypto-app
                  ports:
                    - protocol: TCP
                      port: 80
                      targetPort: 5000
                  type: LoadBalancer
              secretType: 'dockerRegistry'
              containerRegistryType: 'Azure Container Registry'

2. Configure the Kubernetes Deployment and Service

In the Deploy stage of the pipeline, we will deploy our application to AKS using Kubernetes Deployment and Service resources. The pipeline is configured to use inline YAML for these resources, making it easy to modify and maintain the configuration.

In the Deployment resource, we specify the Docker image to be used, pulled from the Azure Container Registry (ACR) created in the previous stage. The Service resource is set up as a LoadBalancer, allowing external access to the application.

Below is the code snippet for the Deploy stage, which includes the Deployment and Service resources:



- stage: Deploy
  displayName: 'Deploy stage'
  dependsOn: Scan
  condition: succeeded()
  jobs:
  - deployment: Deploy
    displayName: 'Deploy Job'
    environment: 'production'
    pool:
      vmImage: 'ubuntu-latest'
    strategy:
      runOnce:
        deploy:
          steps:
          - checkout: self
          - task: Kubernetes@1
            displayName: 'Deploy Deployment to AKS'
            inputs:
              connectionType: 'Azure Resource Manager'
              azureSubscriptionEndpoint: 'my-connection'
              azureResourceGroup: 'dev-rg'
              kubernetesCluster: 'demo-cluster'
              useClusterAdmin: true
              command: 'apply'
              useConfigurationFile: true
              configurationType: 'inline'
              inline: |
                apiVersion: apps/v1
                kind: Deployment
                metadata:
                  name: deployment
                  namespace: default
                spec:
                  replicas: 1
                  selector:
                    matchLabels:
                      app: crypto-app
                  template:
                    metadata:
                      labels:
                        app: crypto-app
                    spec:
                      containers:
                      - name: $(imageName)
                        image: $(acrName).azurecr.io/$(imageName):$(Build.BuildId)
                        ports:
                        - containerPort: 5000
              secretType: 'dockerRegistry'
              containerRegistryType: 'Azure Container Registry'

          - task: Kubernetes@1
            displayName: 'Deploy Service to AKS'
            inputs:
              connectionType: 'Azure Resource Manager'
              azureSubscriptionEndpoint: 'my-connection'
              azureResourceGroup: 'dev-rg'
              kubernetesCluster: 'demo-cluster'
              useClusterAdmin: true
              command: 'apply'
              useConfigurationFile: true
              configurationType: 'inline'
              inline: |
                apiVersion: v1
                kind: Service
                metadata:
                  name: crypto-app-service
                  namespace: default
                spec:
                  selector:
                    app: crypto-app
                  ports:
                    - protocol: TCP
                      port: 80
                      targetPort: 5000
                  type: LoadBalancer
              secretType: 'dockerRegistry'
              containerRegistryType: 'Azure Container Registry'

3. Scan Docker Image with Trivy

Trivy is an open-source scanner for vulnerabilities in container images, it detects vulnerabilities of OS packages (Alpine, Red Hat Universal Base Image, etc.) and application dependencies (Bundler, Composer, npm, yarn, etc.). It is highly configurable and easy to use. In this step, we will be integrating Trivy with our pipeline to scan our Docker image before pushing it to the ACR.

To use Trivy, we need to add a new job in our pipeline under the ScanAndPushImage stage. This job will run Trivy to scan the Docker image and report any vulnerabilities found. We will use the Trivy task provided by the Azure DevOps Marketplace.

The Trivy task needs the image name and tag to scan, and the scan results are saved in a JSON file. We can configure the task to fail if any critical vulnerabilities are found.

Here's the updated pipeline code with the Trivy job:



- stage: Scan
  displayName: 'Scan Image'
  dependsOn: Build
  condition: succeeded()
  jobs:
  - job: ScanImage
    displayName: 'Scan Image Job'
    pool:
      vmImage: 'ubuntu-latest'
    steps:
    - task: Docker@2
      displayName: 'Login to ACR'
      inputs:
        containerRegistry: 'docker-reg'
        repository: '$(imageName)'
        command: 'login'
    - task: trivy@1
      inputs:
        version: 'latest'
        docker: false
        loginDockerConfig: true
        image: $(acrName).azurecr.io/$(imageName):$(Build.BuildId)
        ignoreUnfixed: true
      displayName: Scan Docker image with Trivy

Conclusion

In conclusion, we have successfully set up a CI/CD pipeline for a React app that includes testing, building, scanning, and deploying. We started by creating a basic React app and setting up a GitHub repository to store the code. Then, we configured an Azure Pipeline that automatically builds, tests, and deploys the app to a Kubernetes cluster running on Azure Kubernetes Service (AKS). We used Docker to containerize the app and stored the container image in an Azure Container Registry (ACR). We also integrated the Trivy vulnerability scanner to scan the Docker image for security issues before deploying it to the cluster.

The pipeline consists of several stages, each with its own set of jobs and tasks. In the first stage, the app is tested using Jest and Enzyme, ensuring that it functions correctly before moving on to the next stages. In the second stage, the app is built and a Docker image is created. In the third stage, the Docker image is scanned for vulnerabilities using Trivy, and then pushed to the ACR. In the final stage, the app is deployed to the AKS cluster using Kubernetes.

Throughout this project, we have demonstrated how a CI/CD pipeline can help automate the software development process, reducing the chance of errors and increasing the speed of delivery. With the tools and techniques used here, we have also shown how security can be integrated into the pipeline, ensuring that the app is safe to use and deploy in a production environment.

In summary, this project provides a solid foundation for building and deploying modern web applications using continuous integration and deployment techniques.

Source code: [(https://github.com/rifkhan107/crypto-currency-sample-website.git)]