Forem: cypher682

Designing a Production-Grade Blue-Green ECS Platform on AWS with Terraform

cypher682 — Tue, 24 Feb 2026 15:02:49 +0000

Most AWS tutorials stop at "it works."

I wanted to build something closer to what a real engineering team would operate: network isolation, IAM least privilege, blue-green deployments, secrets management, and clean teardown—all defined as code.

This article walks through the architecture, design decisions, tradeoffs, and the 8 real issues I encountered along the way.

GitHub Repository: ecs-production-platform

Total Cost: $0.12 for complete validation (4-hour session)

What Was Built
Architecture Overview
Security Design
Blue-Green Deployment Mechanics
What Broke (8 Issues)
Production Tradeoffs
Cost Analysis

What Was Built

A production-aligned ECS Fargate platform running a Flask API backed by PostgreSQL:

Networking

Custom VPC (10.0.0.0/16) across 2 Availability Zones
Public subnets for ALB and ECS tasks
Private subnets for RDS (no internet route)

Compute

ECS Fargate services (no EC2 instance management)
Application Load Balancer with HTTPS (ACM certificate, TLS 1.3)
Blue-green target groups for zero-downtime deployments

Data

RDS PostgreSQL 15.12 (single-AZ for free tier)
Private subnet only, no public endpoint

Security

IAM role separation (execution vs task)
SSM Parameter Store for secrets (KMS encrypted)
Security group layering (internet → ALB → ECS → RDS)

Infrastructure as Code

100% Terraform (modular design)
Remote state in S3 with DynamoDB locking
Reusable modules (networking, IAM, ALB, ECS, RDS)

36 AWS resources deployed, tested, and cleanly destroyed.

Architecture Overview

┌─────────────────────────────────────────────┐
│              Internet                        │
└──────────────────┬──────────────────────────┘
                   │
         ┌─────────▼─────────┐
         │    Route 53 DNS    │
         └─────────┬──────────┘
                   │
         ┌─────────▼──────────┐
         │  ACM Certificate    │
         │    (TLS 1.3)        │
         └─────────┬───────────┘
                   │
    ┌──────────────▼──────────────┐
    │  Application Load Balancer  │
    │   (Public Subnets)          │
    └──────┬──────────────┬───────┘
           │              │
    ┌──────▼─────┐ ┌─────▼──────┐
    │ Blue TG    │ │ Green TG   │
    │ (Weight:   │ │ (Weight:   │
    │  100%)     │ │   0%)      │
    └──────┬─────┘ └─────┬──────┘
           │              │
    ┌──────▼──────────────▼──────┐
    │   ECS Fargate Services      │
    │   (Public Subnets)          │
    │   • 2 tasks (blue)          │
    │   • 0 tasks (green standby) │
    └──────────────┬──────────────┘
                   │
         ┌─────────▼──────────┐
         │  RDS PostgreSQL     │
         │  (Private Subnet)   │
         │  • No public IP     │
         │  • Port 5432 only   │
         └─────────────────────┘

Key Design Choices:

ECS in Public Subnets: Cost optimization—saves $33/month on NAT Gateway
Single-AZ RDS: Free tier constraint—production would use Multi-AZ
Security Groups: Each layer enforces isolation for the next

Terraform Module Structure

Instead of one monolithic configuration, concerns are separated into focused modules:

terraform/
├── modules/
│   ├── networking/     # VPC, subnets, security groups, DB subnet group
│   ├── iam/            # Task execution role, task role, policies
│   ├── alb/            # Load balancer, target groups, listeners
│   ├── ecs/            # Cluster, services, task definitions
│   ├── rds/            # PostgreSQL instance, parameter group
│   └── cicd/           # GitHub Actions IAM role (design only)
└── environments/
    └── prod/
        ├── main.tf         # Module composition
        ├── variables.tf    # Input variables
        ├── outputs.tf      # Stack outputs
        ├── backend.tf      # S3 + DynamoDB state
        └── versions.tf     # Provider versions

Module Communication:

Explicit inputs and outputs only
No cross-module resource references
No hidden dependencies

Example Module Call:

module "ecs" {
  source = "../../modules/ecs"

  project_name                = var.project_name
  container_image             = var.container_image
  ecs_task_execution_role_arn = module.iam.ecs_task_execution_role_arn
  ecs_task_role_arn           = module.iam.ecs_task_role_arn
  public_subnet_ids           = module.networking.public_subnet_ids
  ecs_security_group_id       = module.networking.ecs_tasks_security_group_id
  target_group_arn            = module.alb.blue_target_group_arn

  # Database connection
  db_host                     = module.rds.db_address
  db_password_ssm_param       = "/ecs-prod/db/password"
}

This keeps the architecture composable and prevents circular dependency hell.

Security Design

1. Network Isolation (Security Groups)

Traffic flows in one direction only:

Internet (0.0.0.0/0)
    ↓ Port 443/80
┌───────────────────┐
│  ALB SG           │
│  sg-0373513dd...  │
└────────┬──────────┘
         ↓ Port 8000 (from ALB SG only)
┌────────────────────┐
│  ECS Tasks SG      │
│  sg-09a3082e31...  │
└────────┬───────────┘
         ↓ Port 5432 (from ECS SG only)
┌────────────────────┐
│  RDS SG            │
│  sg-07a5aae1f9...  │
└────────────────────┘

Security Group Rules:

ALB Security Group:

ingress {
  description = "HTTPS from internet"
  from_port   = 443
  to_port     = 443
  protocol    = "tcp"
  cidr_blocks = ["0.0.0.0/0"]
}

egress {
  description     = "To ECS tasks only"
  from_port       = 8000
  to_port         = 8000
  protocol        = "tcp"
  security_groups = [aws_security_group.ecs_tasks.id]
}

ECS Tasks Security Group:

ingress {
  description     = "From ALB only"
  from_port       = 8000
  to_port         = 8000
  protocol        = "tcp"
  security_groups = [aws_security_group.alb.id]
}

egress {
  description     = "To RDS only"
  from_port       = 5432
  to_port         = 5432
  protocol        = "tcp"
  security_groups = [aws_security_group.rds.id]
}

RDS Security Group:

ingress {
  description     = "PostgreSQL from ECS only"
  from_port       = 5432
  to_port         = 5432
  protocol        = "tcp"
  security_groups = [aws_security_group.ecs_tasks.id]
}

# No egress rules - database doesn't need outbound

Result:

No direct internet access to ECS tasks
No public database endpoint
No bypass paths

2. IAM Role Separation

Two distinct roles prevent privilege escalation:

Task Execution Role (infrastructure operations):

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "ecr:GetAuthorizationToken",
      "ecr:BatchCheckLayerAvailability",
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchGetImage",
      "logs:CreateLogStream",
      "logs:PutLogEvents",
      "ssm:GetParameters"
    ],
    "Resource": "*"
  }]
}

Task Role (application runtime):

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "ssm:GetParameter",
    "Resource": "arn:aws:ssm:us-east-1:*:parameter/ecs-prod/*"
  }]
}

Why This Matters:

If a container is compromised, the attacker inherits only the task role—not the execution role. They can't:

Pull arbitrary images from ECR
Write to CloudWatch logs outside their stream
Access SSM parameters outside /ecs-prod/*

3. Secrets Management Flow

# 1. Generate password
PASSWORD=$(openssl rand -base64 32 | tr -d "=+/" | cut -c1-25)

# 2. Store in SSM (KMS encrypted)
aws ssm put-parameter \
  --name "/ecs-prod/db/password" \
  --value "$PASSWORD" \
  --type "SecureString"

# 3. Reference in task definition
{
  "name": "DB_PASSWORD_SSM_PARAM",
  "value": "/ecs-prod/db/password"
}

# 4. Application fetches at runtime
import boto3
ssm = boto3.client('ssm')
password = ssm.get_parameter(
    Name=os.environ['DB_PASSWORD_SSM_PARAM'],
    WithDecryption=True
)['Parameter']['Value']

Never in:

Source control
Docker image
Environment variables (plaintext)
Terraform state (marked sensitive)

Blue-Green Deployment Mechanics

The ALB HTTPS listener can route traffic to two separate target groups with configurable weights.

Initial State

Blue Target Group:  ████████████████████ 100% (2 healthy tasks)
Green Target Group: -------------------- 0%  (0 tasks)

Deployment Process

Step 1: Build New Version

# Update app version in app.py
# Build Docker image
docker build -t 758620460011.dkr.ecr.us-east-1.amazonaws.com/ecs-prod/flask-app:v2 .

# Push to ECR
docker push 758620460011.dkr.ecr.us-east-1.amazonaws.com/ecs-prod/flask-app:v2

Step 2: Deploy to Green

# Scale up green service
aws ecs update-service \
  --cluster ecs-prod-cluster \
  --service ecs-prod-service-green \
  --desired-count 2

# Wait for health checks (90 seconds)
aws ecs wait services-stable \
  --cluster ecs-prod-cluster \
  --services ecs-prod-service-green

Step 3: Switch Traffic (< 1 Second)

# Get listener and target group ARNs
LISTENER_ARN=$(terraform output -raw alb_listener_arn)
GREEN_TG=$(terraform output -raw green_target_group_arn)

# Instant traffic switch
aws elbv2 modify-listener \
  --listener-arn $LISTENER_ARN \
  --default-actions Type=forward,TargetGroupArn=$GREEN_TG

Step 4: Validate and Cleanup

# Monitor green deployment
curl https://app.cipherpol.xyz/health
# {"version":"2.0.0","deployment":"green","status":"healthy"}

# After 15 minutes of monitoring, scale down blue
aws ecs update-service \
  --cluster ecs-prod-cluster \
  --service ecs-prod-service \
  --desired-count 0

Why This Works

No DNS propagation delays — Traffic switches at ALB layer

No container restarts — Only listener weight changes

Instant rollback — Reverse the listener modification

No downtime — ALB handles connection draining

Rollback Command (Same as Deploy):

# Get blue target group
BLUE_TG=$(terraform output -raw blue_target_group_arn)

# Instant rollback
aws elbv2 modify-listener \
  --listener-arn $LISTENER_ARN \
  --default-actions Type=forward,TargetGroupArn=$BLUE_TG

Data Persistence Validation

Both blue and green services connect to the same RDS instance. I validated this explicitly:

# 1. Create items while blue is active
curl -X POST https://app.cipherpol.xyz/items \
  -H "Content-Type: application/json" \
  -d '{"name":"Test Item 1"}'

curl -X POST https://app.cipherpol.xyz/items \
  -H "Content-Type: application/json" \
  -d '{"name":"Test Item 2"}'

# 2. Switch traffic to green
aws elbv2 modify-listener --listener-arn $LISTENER_ARN --default-actions Type=forward,TargetGroupArn=$GREEN_TG

# 3. Verify data persists
curl https://app.cipherpol.xyz/items | jq
{
  "count": 2,
  "items": [
    {"id": 2, "name": "Test Item 2", "created_at": "2026-02-19T10:54:03"},
    {"id": 1, "name": "Test Item 1", "created_at": "2026-02-19T10:53:02"}
  ]
}

# 4. Create new item on green
curl -X POST https://app.cipherpol.xyz/items \
  -H "Content-Type: application/json" \
  -d '{"name":"Created on Green v2.0"}'

# 5. Rollback to blue
aws elbv2 modify-listener --listener-arn $LISTENER_ARN --default-actions Type=forward,TargetGroupArn=$BLUE_TG

# 6. Confirm all items still present
curl https://app.cipherpol.xyz/items | jq '.count'
# 3

Result:

5 items persisted across 3 deployment cycles

The deployment layer is stateless

The database is the single source of truth

What Broke (And What I Learned)

Issue 1: RDS Connection Delay After "Available" Status

Problem: ECS tasks failed health checks immediately after terraform apply completed RDS creation.

Symptoms:

ecs-prod-service: unhealthy targets: 2/2
Task stopped reason: Task failed container health checks

Root Cause:

RDS reports available status when the instance is running, but doesn't accept connections for another 60-90 seconds while:

Background processes initialize
Performance schema loads
Cache warms up

Fix:

Health check grace period (60s) absorbed the delay:

resource "aws_ecs_service" "app" {
  health_check_grace_period_seconds = 60
  # Tasks retry connection until RDS is ready
}

Lesson: AWS resource statuses don't always mean "ready for traffic." Plan for initialization time.

Issue 2: Docker HEALTHCHECK Missing Dependency

Problem: 100+ ECS task restarts. ALB target health showed healthy, but ECS kept replacing tasks.

Symptoms:

# ALB target group
Target health: healthy (2/2)

# ECS service events
Unhealthy container: flask-app
Task stopped, starting replacement

Root Cause:

# Original (broken)
HEALTHCHECK CMD curl -f http://localhost:8000/health || exit 1

But curl wasn't in python:3.11-slim base image.

Why ALB Passed but Container Failed:

ALB health check: HTTP request from outside the container (port 8000)
Container health check: Command executed inside the container

Fix:

# Use Python instead of curl
HEALTHCHECK CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=2)"

Lesson:

ALB health checks and container health checks are independent control loops:

ALB health: Determines if task receives traffic
Container health: Determines if ECS replaces the task

ECS uses container health, not ALB health.

Issue 3: Git Bash Path Conversion on Windows

Problem: SSM parameter /ecs-prod/db/password became C:\Program Files\Git\ecs-prod\db\password

Error:

ParameterNotFound: /C:/Program Files/Git/ecs-prod/db/password

Root Cause: Git Bash on Windows auto-converts Unix-style paths to Windows paths.

Fix:

# Disable path conversion
export MSYS_NO_PATHCONV=1

# Then run AWS CLI
aws ssm get-parameter --name /ecs-prod/db/password --with-decryption

Alternative: Use Windows Command Prompt or PowerShell for AWS CLI commands.

Lesson: Git Bash is great for Unix tools, but AWS CLI needs special handling on Windows.

Issue 4: ALB Listener Syntax Constraint

Problem: After configuring blue-green with weighted routing, subsequent updates using simple syntax failed.

Error:

An error occurred (ValidationError): Cannot use both TargetGroupArn and ForwardConfig in the same action

Root Cause: Once you use ForwardConfig (weighted routing), the ALB API remembers this and requires full JSON syntax for all future updates.

Simple syntax (stopped working):

--default-actions Type=forward,TargetGroupArn=arn:aws:...

Required syntax:

--default-actions '[{
  "Type": "forward",
  "ForwardConfig": {
    "TargetGroups": [{
      "TargetGroupArn": "arn:aws:...",
      "Weight": 100
    }]
  }
}]'

Lesson: ALB API is stateful. Once you use advanced features, you can't revert to simple syntax. Document this in runbooks.

Issue 5: PostgreSQL Minor Version Retirement

Problem: Terraform apply failed with:

InvalidParameterValue: Cannot find version 15.4 for postgres

Root Cause: AWS retired PostgreSQL 15.4 in favor of 15.12 (latest patch version).

Fix:

# Before
engine_version = "15.4"

# After
engine_version = "15.12"

Better Fix (Production):

# Pin major version, allow minor updates
engine_version = "15"

Lesson: AWS manages minor version lifecycle. Pin major versions intentionally, but expect patch version changes.

Issue 6: Terraform State Lock Timeout

Problem: terraform plan hung for 5 minutes, then failed:

Error acquiring state lock: timeout waiting for lock

Root Cause: DynamoDB lock table had wrong key schema:

# Wrong
AttributeName=id,KeyType=HASH

# Correct
AttributeName=LockID,KeyType=HASH

Fix:

# Delete wrong table
aws dynamodb delete-table --table-name terraform-state-lock

# Recreate with correct schema
aws dynamodb create-table \
  --table-name terraform-state-lock \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST

Lesson: Terraform's DynamoDB lock table requires exactly LockID as the partition key. Case-sensitive.

Issue 7: ECR Image Pull Failures (Intermittent)

Problem: Some task launches failed with:

CannotPullContainerError: API error: manifest unknown

Root Cause: Task execution role was missing ecr:BatchGetImage permission.

Fix: Attached AWS managed policy:

resource "aws_iam_role_policy_attachment" "ecs_task_execution" {
  role       = aws_iam_role.ecs_task_execution.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

Lesson: Custom IAM policies are error-prone. Use AWS managed policies where possible, then restrict with conditions if needed.

Issue 8: Terraform vs Manual Scaling Conflict

Problem: Terraform tried to update blue service's desired_count while I was manually scaling for testing:

Error: concurrent modification detected
Service is being modified by another operation

Fix: Added lifecycle rule to ignore runtime changes:

resource "aws_ecs_service" "app" {
  # ... other config

  lifecycle {
    ignore_changes = [desired_count]
  }
}

Lesson: When testing blue-green manually, let Terraform manage infrastructure but ignore runtime scaling changes. Use ignore_changes selectively.

What I'd Change in Production

This Project	Production Standard	Reason	Cost Impact
ECS in public subnets	Private subnets + NAT Gateway	Defense-in-depth, reduced attack surface	+$33/month
Single-AZ RDS	Multi-AZ RDS	99.95% SLA vs 99.5%, automatic failover	+$15/month
Manual ALB switch	AWS CodeDeploy blue-green	Automated rollback based on CloudWatch alarms	$0 (free)
No autoscaling	ECS Service Auto Scaling	Handle traffic spikes, reduce idle costs	Variable
SSM Parameter Store	AWS Secrets Manager	Automatic rotation, better audit	+$0.40/secret
No read replicas	RDS read replica	Offload read traffic from primary	+$15/month
7-day log retention	30-90 day retention	Compliance, longer incident investigation	+$2/month

Total Production Cost: ~$80-100/month

This Project Cost: $0.12 for validation

Cost Savings Breakdown:

NAT Gateway: $33 saved
Multi-AZ RDS: $15 saved
Secrets Manager: $0.40 saved
Read replica: $15 saved
Total: $63.40/month saved

Cost Breakdown (Actual)

4-Hour Validation Session

Service	Hourly Rate	Hours	Cost
ALB	$0.0225	3	$0.0675
ECS Fargate (2 tasks × 0.25 vCPU)	Free tier	3	$0
RDS db.t3.micro (single-AZ)	Free tier	7	$0
Route 53 Hosted Zone	$0.50/month prorated	3 days	$0.05
S3 State Bucket	<$0.01	—	$0
CloudWatch Logs	Free tier (5GB)	—	$0
Data Transfer	Free tier (100GB)	<1GB	$0
Total			$0.12

Monthly Cost If Kept Running

Service	Monthly Cost	Notes
ALB	$16.20	720 hours × $0.0225
ECS Fargate	$0	Within 400 vCPU-hour free tier
RDS db.t3.micro	$0	Within 750-hour free tier
Route 53	$0.50	Hosted zone
Storage	<$1	S3, CloudWatch
Total	~$17/month

Comparison to EKS:

EKS control plane: $72/month
ECS Fargate control plane: $0
Savings: $72/month for equivalent compute

Key Takeaways

What Worked Well

Modular Terraform — Each module had single responsibility, debugging was easier

Blue-green switching — True zero downtime, instant rollback capability

Security group layering — Network isolation without complexity

SSM secrets — No credentials in code, images, or state files

Free tier optimization — Validated production patterns for $0.12

Documentation — 8 real issues documented with root causes and fixes

What I'd Improve

Terragrunt for DRY — Multi-environment deployments without duplication

Automated testing — Pre-deployment health checks in CI/CD

CodeDeploy integration — Production should automate blue-green

Observability — CloudWatch dashboards for latency, errors, saturation

Database migrations — Flyway or Liquibase for schema versioning

Chaos engineering — Terminate random tasks to test resilience

Repository & Evidence

Full source code with detailed documentation:

GitHub: cypher682/ecs-production-platform

What's included:

Complete Terraform modules (networking, IAM, ALB, ECS, RDS)
Flask application with Dockerfile and health checks
GitHub Actions workflow (OIDC design, not tested live)
Operational runbooks (deployment failure, database connection, rollback)
Lessons learned documentation (8 issues with root cause analysis)
Cost analysis (actual vs production projection)
Evidence files (Terraform outputs, CloudWatch logs, test results)

Documentation structure:

docs/
├── ARCHITECTURE.md            # Design decisions and network diagrams
├── 01_IMPLEMENTATION.md       # Phase-by-phase build log
├── 02_LESSONS_LEARNED.md      # 8 issues with fixes
├── 03_COST_ANALYSIS.md        # Detailed cost breakdown
├── 04_SECURITY.md             # IAM policies, secrets flow
├── 05_CICD_DESIGN.md          # GitHub Actions workflow design
└── runbooks/
    ├── deployment-failure.md  # What to do when deploy fails
    ├── database-connection.md # Troubleshooting RDS connectivity
    └── rollback-procedure.md  # Step-by-step rollback guide

Questions or Feedback?

If you're building something similar or have questions about:

Blue-green deployment patterns
IAM least privilege design
AWS cost optimization strategies
Terraform module architecture
Debugging ECS task failures

Drop a comment below! I'll respond with specific examples from this build.

This project was built as a portfolio sprint to demonstrate production-ready AWS skills. The platform was deployed, validated with 5 CRUD operations, and destroyed within 4 hours—total cost $0.12. All code and documentation available on GitHub.

#AWS #Terraform #DevOps #ECS #InfrastructureAsCode #BlueGreenDeployment #CloudEngineering #Docker #PostgreSQL

Building a Production-Grade AWS Cost & Security Auditor

cypher682 — Thu, 12 Feb 2026 08:36:28 +0000

Cloud environments naturally drift. Costs creep up. Security posture degrades. Manual audits do not scale, and periodic reviews miss issues that emerge between checks.

I needed an auditing tool that could:

Identify cost waste — idle EC2 instances, unattached EBS volumes, orphaned snapshots
Detect security misconfigurations — public S3 buckets, overly permissive security groups, weak IAM hygiene
Map findings to a known framework — CIS AWS Foundations Benchmark
Operate safely — strictly read-only, no automated deletion or remediation

This article walks through the key design decisions, trade-offs, and lessons learned from building it.

Core Design Principles

1. Read-Only by Design

Decision: No auto-remediation. No destructive permissions.

Automatically deleting or modifying cloud resources is risky, especially in production. An instance that appears idle may be a disaster-recovery standby, a scheduled batch worker, or part of a failover strategy.

The tool’s role is to surface risk and waste, not to make irreversible decisions.

IAM policy scope:

{
  "Effect": "Allow",
  "Action": [
    "ec2:Describe*",
    "s3:Get*",
    "s3:List*",
    "iam:List*",
    "iam:Get*"
  ],
  "Resource": "*"
}

No Delete, Terminate, or Modify permissions. The blast radius is limited to discovery only.

This constraint shaped the entire architecture and made the tool safe to run against live accounts.

2. Defining “Idle” Using CloudWatch Metrics

Problem: “Idle” is ambiguous in cloud systems.

CPU utilization is an imperfect signal, but it is widely available and easy to reason about. I defined idle EC2 instances as those with:

Average CPU utilization < 5%
Observed over a 7-day window

Implementation:

def get_cpu_utilization(self, instance_id: str, days: int = 7) -> float:
    end_time = datetime.utcnow()
    start_time = end_time - timedelta(days=days)

    response = self.cloudwatch.get_metric_statistics(
        Namespace='AWS/EC2',
        MetricName='CPUUtilization',
        Dimensions=[{'Name': 'InstanceId', 'Value': instance_id}],
        StartTime=start_time,
        EndTime=end_time,
        Period=86400,  # 1 day
        Statistics=['Average']
    )

    if response['Datapoints']:
        return sum(dp['Average'] for dp in response['Datapoints']) / len(response['Datapoints'])
    return -1.0

Trade-offs:

Misses bursty or scheduled workloads (batch jobs, ML training)
Flags instances that are intentionally dormant

Rather than hiding these limitations, they are explicitly documented. Transparency builds trust in tooling.

3. Aligning Findings with CIS Benchmarks

Raw findings are less useful without context. Mapping issues to the CIS AWS Foundations Benchmark adds structure and credibility.

Example:

findings.append({
    'bucket_name': bucket_name,
    'issue': 'Public access enabled',
    'severity': 'HIGH',
    'cis_control': '2.1.5',
    'remediation': 'Enable S3 Block Public Access'
})

This approach:

Makes findings actionable
Aligns with how security teams think
Signals familiarity with compliance-driven environments

Results from a Real AWS Account

Running the auditor against my own AWS account produced the following:

SECURITY FINDINGS: 32
  CRITICAL: 11  (SSH/RDP open to 0.0.0.0/0)
  HIGH:      9  (IAM users without MFA, public S3 buckets)
  MEDIUM:   12  (stale access keys, permissive policies)

Even a relatively small account accumulated meaningful security drift. Continuous auditing is not optional at scale.

Handling False Positives

One flagged issue was my own AuditorToolReadOnly IAM policy using:

"Resource": "*"

At first glance, this looks overly permissive. In practice, it is required. Read-only IAM and EC2 discovery APIs (List*, Describe*) cannot be scoped to specific ARNs that are not yet known.

Key point:

Not all flagged issues are actionable
False positives should be documented, not ignored

This distinction is critical in real operational environments.

What I Would Improve for a Production Deployment

If this were moving beyond a portfolio project:

Multi-region scanning instead of single-region execution
Historical persistence using DynamoDB for trend analysis
AWS Cost Explorer integration for real billing data
Alerting via Slack or SNS for critical findings
IAM Access Analyzer integration for deeper policy analysis

The current scope balances realism with complexity without overengineering.

Key Takeaways

Read-only audits reduce risk and build trust when running against live environments
Cost and security signals are more useful when tied to metrics and known frameworks
Not every finding should be auto-remediated; judgment still matters

These principles guided the design choices throughout this project.

Try It Yourself

Repository:
https://github.com/cypher682/aws-cost-security-auditor

Run locally:

git clone https://github.com/cypher682/aws-cost-security-auditor
cd aws-cost-security-auditor
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt
python src/full_audit.py --profile auditor-role

See the remediation guidance in docs/REMEDIATION_PLAYBOOK.md.

What’s Next

Planned extensions:

Multi-account support (AWS Organizations)
RDS idle detection
Lambda cost analysis

Links:

X: https://twitter.com/cypher682
LinkedIn: https://linkedin.com/in/suleiman-abdulrahman-dev

This project is part of my portfolio focused on production-grade cloud platform engineering.

Building a Production-Ready Microservices Platform with CI/CD on AWS Free Tier

cypher682 — Thu, 11 Dec 2025 14:37:20 +0000

Introduction

Building a complete microservices architecture with professional DevOps practices can be intimidating. This guide walks you through creating a production-grade system using AWS free tier, demonstrating real-world patterns that mid-level engineers can apply immediately.

We'll build a polyglot microservices platform with:

Three microservices in different languages (Node.js, Python, Go)
Complete CI/CD pipeline with automated testing and security scanning
Infrastructure as Code with Terraform
Monitoring with Prometheus and Grafana
All running on AWS free tier for under $2/month

Architecture Overview

Our platform consists of three microservices behind an API Gateway, all running on a single EC2 instance with Docker Compose. While this isn't the scalability of Kubernetes, it demonstrates core concepts without the complexity or cost.

The API Gateway handles routing and authentication, forwarding requests to the User Service (Python/FastAPI) for user operations and Product Service (Go/Gin) for product management. Both services interact with separate DynamoDB tables.

Why This Stack?

Polyglot Architecture: Using multiple languages demonstrates how microservices enable technology diversity. Each service uses the best tool for its job.

Docker Compose on EC2: Kubernetes is powerful but complex. Docker Compose provides orchestration sufficient for small-medium deployments while remaining in free tier.

DynamoDB: Pay-per-request pricing with generous free tier (25GB, 25 RCU/WCU) makes it ideal for demos that might sit idle.

GitHub Actions: Native GitHub integration means no external CI/CD service needed.

Infrastructure Setup

We use Terraform to create all AWS resources. The infrastructure includes:

VPC with public subnet
EC2 t2.micro instance
DynamoDB tables for users and products
Route53 hosted zone
IAM roles with least-privilege policies
Security groups allowing only necessary ports

The EC2 instance runs Docker and is configured via user data script to install Docker Engine during launch.

resource "aws_instance" "app" {
  ami                    = data.aws_ami.ubuntu.id
  instance_type          = "t2.micro"
  vpc_security_group_ids = [aws_security_group.ec2.id]
  iam_instance_profile   = aws_iam_instance_profile.ec2.name

  user_data = <<-EOF
              #!/bin/bash
              apt-get update
              curl -fsSL https://get.docker.com | sh
              usermod -aG docker ubuntu
              EOF
}

DynamoDB tables use on-demand billing to stay within free tier:

resource "aws_dynamodb_table" "users" {
  name         = "aws-microservices-cicd-users"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "userId"

  attribute {
    name = "userId"
    type = "S"
  }
}

Microservices Implementation

API Gateway (Node.js)

The API Gateway handles authentication via API keys stored in AWS Systems Manager Parameter Store and routes requests to appropriate services.

const authMiddleware = (req, res, next) => {
  const apiKey = req.headers['x-api-key'];
  if (!apiKey || apiKey !== API_KEY) {
    return res.status(401).json({ error: 'Unauthorized' });
  }
  next();
};

app.use('/users', authMiddleware, async (req, res) => {
  const response = await axios({
    method: req.method,
    url: `${USER_SERVICE_URL}${req.path}`,
    data: req.body
  });
  res.status(response.status).json(response.data);
});

User Service (Python)

FastAPI provides automatic API documentation and Pydantic validation:

@app.post("/", response_model=UserResponse, status_code=201)
async def create_user(user: User):
    user_id = str(uuid.uuid4())
    timestamp = datetime.utcnow().isoformat()

    table.put_item(
        Item={
            'userId': user_id,
            'email': user.email,
            'name': user.name,
            'createdAt': timestamp
        }
    )
    return item

Product Service (Go)

Go's performance makes it ideal for high-throughput services:

func createProduct(c *gin.Context) {
    var product Product
    c.ShouldBindJSON(&product)

    product.ProductID = uuid.New().String()
    product.CreatedAt = time.Now().UTC().Format(time.RFC3339)

    av, _ := dynamodbattribute.MarshalMap(product)
    db.PutItem(&dynamodb.PutItemInput{
        TableName: aws.String(tableName),
        Item: av,
    })

    c.JSON(201, product)
}

CI/CD Pipeline

The GitHub Actions pipeline has two workflows:

CI Pipeline (on pull requests):

Lint code for all three services
Run unit tests
Build Docker images
Scan containers with Trivy for vulnerabilities
Upload security findings to GitHub Security

CD Pipeline (on merge to main):

Build Docker images for all services
Push to Docker Hub with latest and SHA tags
SSH to EC2 instance
Pull new images
Restart containers with docker-compose
Run health checks
Rollback on failure

- name: Deploy to EC2
  run: |
    ssh ubuntu@${{ steps.get-ip.outputs.instance_ip }} << 'EOF'
      cd /home/ubuntu/app
      docker-compose pull
      docker-compose up -d
    EOF

Security Implementation

Container Scanning: Trivy scans images for known vulnerabilities before deployment. Critical and high severity findings block the pipeline.

Secrets Management: API keys and sensitive data live in AWS Systems Manager Parameter Store, not in environment variables or code.

SSL/TLS: Let's Encrypt provides free SSL certificates, configured via Ansible during initial setup.

IAM Policies: EC2 instance has minimal permissions, only accessing specific DynamoDB tables and Parameter Store paths.

resource "aws_iam_role_policy" "dynamodb" {
  policy = jsonencode({
    Statement = [{
      Effect = "Allow"
      Action = [
        "dynamodb:PutItem",
        "dynamodb:GetItem",
        "dynamodb:Query"
      ]
      Resource = [
        aws_dynamodb_table.users.arn,
        aws_dynamodb_table.products.arn
      ]
    }]
  })
}

Monitoring Stack

Prometheus scrapes metrics from all three services every 15 seconds. Each service exposes a /metrics endpoint with custom business metrics plus standard HTTP metrics.

Grafana visualizes the data with dashboards showing:

Request rates and latencies per service
Error rates (4xx, 5xx responses)
Container resource usage (CPU, memory)
DynamoDB operation metrics
Custom business metrics (user registrations, product creates)

The monitoring stack runs alongside the application services in Docker Compose:

prometheus:
  image: prom/prometheus:latest
  volumes:
    - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml
  ports:
    - "9090:9090"

grafana:
  image: grafana/grafana:latest
  environment:
    - GF_SECURITY_ADMIN_PASSWORD=admin
  ports:
    - "3001:3000"

Configuration Management with Ansible

Ansible handles EC2 setup tasks that are complex or stateful:

Installing Docker and Docker Compose
Configuring UFW firewall
Setting up Nginx with SSL
Obtaining Let's Encrypt certificates
Creating application directories
Installing Prometheus Node Exporter

The setup playbook runs once after Terraform creates the infrastructure:

- name: Install Docker
  apt:
    name:
      - docker-ce
      - docker-ce-cli
      - containerd.io
    state: present

- name: Obtain SSL certificate
  command: >
    certbot --nginx -d api.cipherpol.xyz
    --non-interactive --agree-tos

Cost Optimization

This architecture costs $0.50-2.00/month:

Route53: $0.50/month for hosted zone (only unavoidable cost)

EC2 t2.micro: Free tier provides 750 hours/month, enough for one instance running 24/7

DynamoDB: Free tier includes 25GB storage and 25 RCU/WCU, sufficient for development and small production workloads

Data Transfer: First 1GB/month free, typically sufficient for API traffic

S3: Used for Terraform state, negligible cost

The key is using on-demand pricing for DynamoDB (no provisioned capacity charges) and staying within EC2 free tier limits.

Lessons Learned

Keep it Simple: Docker Compose on one instance is simpler than ECS/EKS and adequate for many workloads.

Security First: Even demo projects should implement proper security. Trivy scanning and proper IAM policies cost nothing but prevent vulnerabilities.

Monitoring Matters: Prometheus and Grafana add minimal overhead but provide invaluable visibility into system behavior.

Automate Everything: Manual deployments are error-prone. GitHub Actions makes automation straightforward.

Cost-Aware Design: Understanding free tier limits enables building production-quality systems for minimal cost.

Real Challenges Faced:

Docker build failures due to deprecated npm flags and missing dependency files
IAM permission mismatches between parameter paths and policies
Port conflicts between services requiring careful docker-compose configuration
Nginx SSL configuration needing manual proxy_pass setup after certbot
Multi-stage Docker builds requiring proper user permission handling

Trade-offs and Limitations

Single Point of Failure: One EC2 instance means no high availability. For production, consider auto-scaling groups.

Limited Scalability: Docker Compose doesn't provide automatic scaling. When traffic grows, consider ECS or EKS.

Shared Resources: All services share EC2 resources. A memory leak in one service affects others.

Manual SSL Renewal: While automated via cron, this isn't as robust as AWS Certificate Manager.

Real-World Applications

This architecture pattern works well for:

Internal tools and admin dashboards
API backends for mobile apps
Prototypes and MVPs
Side projects and personal applications
Learning and portfolio projects

It's not suitable for:

High-traffic consumer applications
Systems requiring 99.99% uptime
Workloads with unpredictable traffic spikes
Applications requiring compliance certifications

Next Steps

To evolve this architecture:

Add Authentication Service: Implement OAuth2/JWT for user authentication
Implement Caching: Add Redis for frequently accessed data
Queue System: Use SQS for asynchronous processing
API Gateway Replacement: Consider AWS API Gateway for rate limiting and caching
Multi-Region: Deploy in multiple regions for disaster recovery
Kubernetes Migration: When scaling requirements justify complexity

Conclusion

Building a complete microservices platform doesn't require expensive infrastructure or complex orchestration. This project demonstrates that professional DevOps practices are accessible and affordable.

The skills developed here—Infrastructure as Code, CI/CD pipelines, container orchestration, and monitoring—translate directly to enterprise environments. The architecture patterns scale from side projects to production systems.

Most importantly, this hands-on experience with real tools and services is more valuable than theoretical knowledge. You now have a working platform to experiment with, break, fix, and improve.

The complete source code is available on GitHub, along with detailed setup instructions and troubleshooting guides. Fork it, modify it, and make it your own.

Resources

GitHub Repository: aws-microservices-cicd
AWS Free Tier: https://aws.amazon.com/free
Docker Compose Documentation: https://docs.docker.com/compose
Terraform AWS Provider: https://registry.terraform.io/providers/hashicorp/aws
Prometheus Monitoring: https://prometheus.io/docs
GitHub Actions: https://docs.github.com/actions

Complete Beginner's Guide to Blue-Green Deployment with Nginx and Real-Time Alerting

cypher682 — Tue, 09 Dec 2025 14:49:42 +0000

Introduction

Welcome to this comprehensive guide on Blue-Green Deployment - a powerful deployment strategy used by companies like Netflix, Amazon, and Facebook to achieve zero-downtime deployments. This project demonstrates how to implement a production-ready blue-green deployment system with automatic failover and real-time Slack alerting.

Repository: HNG DevOps on GitHub

What You'll Learn:

What blue-green deployment is and why it matters
How to implement automatic failover with Nginx
How to build a real-time monitoring and alerting system
How to achieve zero-downtime deployments
How to integrate Slack notifications for DevOps alerts

Prerequisites:

Basic understanding of Docker
Familiarity with command line
Basic knowledge of web servers (helpful but not required)

What is Blue-Green Deployment?

The Problem: Traditional Deployments

Imagine you're running a website. When you deploy a new version:

You stop the old version
Deploy the new version
Start the new version

Problem: During steps 1-3, your website is DOWN. Users see errors. You lose money.

The Solution: Blue-Green Deployment

Instead of having one environment, you have TWO identical environments:

BLUE (Production) - Currently serving users
GREEN (Staging) - New version waiting to go live

When you're ready to deploy:

Deploy new version to GREEN
Test GREEN thoroughly
Switch traffic from BLUE to GREEN instantly
If something goes wrong, switch back to BLUE instantly

Result: ZERO DOWNTIME

Real-World Analogy

Think of it like having two stages at a concert:

Stage 1 (Blue): Band is performing, audience is watching
Stage 2 (Green): Next band is setting up and sound-checking

When it's time to switch:

Rotate the stage 180°
Audience now sees Stage 2 (Green)
Stage 1 (Blue) becomes the setup area for the next act

If the new band has technical issues, rotate back to Stage 1 instantly!

Project Overview

This project implements a production-ready blue-green deployment system with:

Core Features

Automatic Failover
- Nginx detects when Blue instance fails
- Automatically routes all traffic to Green
- Zero failed requests to users
Real-Time Alerting
- Python watcher monitors Nginx logs
- Detects failover events instantly
- Sends alerts to Slack
- Monitors error rates
Zero-Downtime Deployment
- Deploy new version to inactive instance
- Switch traffic instantly
- Rollback in seconds if needed
Structured Logging
- Every request logged with metadata
- Pool information (blue/green)
- Release version
- Response times

Understanding the Core Concepts

1. Blue-Green vs. Load Balancing

Load Balancing:

Request 1 → Blue
Request 2 → Green
Request 3 → Blue
Request 4 → Green

Traffic is distributed between instances.

Blue-Green (This Project):

All Requests → Blue (Primary)
              Green (Backup, standby)

If Blue fails:
All Requests → Green (Backup becomes active)

Traffic goes to ONE instance at a time. The other is a hot standby.

2. Nginx Upstream Configuration

Nginx can route traffic to multiple backend servers (upstreams). This project uses:

upstream app {
    server app_blue:3000 max_fails=1 fail_timeout=5s;
    server app_green:3000 backup max_fails=1 fail_timeout=5s;
}

Key Directives:

server app_blue:3000 - Primary server
backup - Only use if primary fails
max_fails=1 - Mark as failed after 1 error
fail_timeout=5s - Try again after 5 seconds

3. Failover Mechanism

When a request fails:

Nginx tries Blue instance
Blue returns 5xx error or times out
Nginx marks Blue as failed
Nginx retries request to Green (backup)
User receives successful response from Green
All subsequent requests go to Green

Result: User never sees an error!

Architecture Overview

System Architecture Diagram

                Internet
                   |
                   v
            +--------------+
            |    Nginx     |
            |  (Port 8080) |
            +--------------+
                   |
    +--------------+--------------+
    |                             |
    v                             v
+----------+                +----------+
| App Blue | (Primary)      |App Green | (Backup)
|Port 3000 |                |Port 3000 |
+----------+                +----------+
    |                             |
    +-------------+---------------+
                  |
                  v
          +--------------+
          | Nginx Logs   |
          +--------------+
                  |
                  v
        +------------------+
        | Alert Watcher    |
        | (Python)         |
        +------------------+
                  |
                  v
          +--------------+
          |    Slack     |
          +--------------+

Component Breakdown

1. Nginx Proxy

Role: Traffic router and load balancer

Responsibilities:

Route all incoming requests
Detect backend failures
Perform automatic failover
Log all requests with metadata

2. App Blue (Primary Instance)

Role: Primary application server

Environment Variables:

APP_POOL=blue
RELEASE_ID=blue-release-1.0.0
PORT=3000

3. App Green (Backup Instance)

Role: Backup application server (hot standby)

Environment Variables:

APP_POOL=green
RELEASE_ID=green-release-1.0.0
PORT=3000

4. Alert Watcher (Python)

Role: Real-time log monitoring and alerting

Responsibilities:

Tail Nginx access logs
Parse structured log entries
Detect failover events
Monitor error rates
Send Slack alerts

Technology Stack

Component	Technology	Purpose
Reverse Proxy	Nginx (Alpine)	Traffic routing & failover
Application	Python/Flask	Demo web application
Monitoring	Python 3.11	Log watcher & alerting
Alerting	Slack Webhooks	Real-time notifications
Containerization	Docker	Package all services
Orchestration	Docker Compose	Manage multi-container setup

Setting Up the Project

Step 1: Clone the Repository

git clone https://github.com/cypher682/hng13-stage-3-devops.git
cd hng13-stage-3-devops

Step 2: Understand the Project Structure

hng13-stage-3-devops/
├── docker-compose.yml       # Container orchestration
├── nginx.conf.template      # Nginx configuration template
├── entrypoint.sh           # Nginx startup script
├── watcher.py              # Python log monitoring script
├── requirements.txt        # Python dependencies
├── .env.example            # Environment variables template
├── test-failover.sh        # Failover testing script
└── public/                 # Static HTML files

Step 3: Configure Environment Variables

Copy the example environment file:

cp .env.example .env

Edit .env with your configuration:

# Application Configuration
PORT=3000
ACTIVE_POOL=blue

# Docker Images
BLUE_IMAGE=yimikaade/wonderful:latest
GREEN_IMAGE=yimikaade/wonderful:latest

# Release Identifiers
RELEASE_ID_BLUE=blue-release-1.0.0
RELEASE_ID_GREEN=green-release-1.0.0

# Slack Integration
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL

# Alert Configuration
ERROR_RATE_THRESHOLD=2          # Percentage
WINDOW_SIZE=200                 # Number of requests
ALERT_COOLDOWN_SEC=300          # 5 minutes

Step 4: Set Up Slack Webhook

Go to Slack API
Create a new Slack App
Enable Incoming Webhooks
Create a webhook for your channel
Copy the webhook URL to .env

Step 5: Build and Start Services

# Build all containers
docker compose build

# Start all services in detached mode
docker compose up -d

# Verify all services are running
docker compose ps

Expected Output:

NAME              STATUS    PORTS
nginx_proxy       Up        0.0.0.0:8080->80/tcp
app_blue          Up        3000/tcp
app_green         Up        3000/tcp
alert_watcher     Up

Step 6: Verify the Deployment

Open your browser and navigate to:

http://localhost:8080/version

You should see:

{
  "pool": "blue",
  "release": "blue-release-1.0.0",
  "status": "healthy"
}

Understanding the Configuration

Nginx Configuration Template

The nginx.conf.template uses environment variable substitution:

upstream app {
    server app_blue:${PORT} max_fails=1 fail_timeout=5s;
    server app_green:${PORT} backup max_fails=1 fail_timeout=5s;
}

server {
    listen 80;

    location / {
        proxy_pass http://app;
        proxy_next_upstream error timeout http_502 http_503 http_504;

        # Timeouts for fast failure detection
        proxy_connect_timeout 2s;
        proxy_send_timeout 3s;
        proxy_read_timeout 3s;
    }
}

Key Configuration Explained

1. Proxy Next Upstream

proxy_next_upstream error timeout http_502 http_503 http_504;

What it does:
Automatically retry the request to the next upstream (Green) if:

Connection error occurs
Request times out
Upstream returns 502, 503, or 504

Result: User never sees these errors!

2. Aggressive Timeouts

proxy_connect_timeout 2s;
proxy_send_timeout 3s;
proxy_read_timeout 3s;

Why so short?

Detect failures fast
Failover happens in seconds, not minutes
Better user experience

3. Structured Logging

log_format detailed 
    'pool=$upstream_http_x_app_pool '
    'release=$upstream_http_x_release_id '
    'upstream_status=$upstream_status '
    'latency=$request_time ';

access_log /var/log/nginx/access.log detailed;

Log Example:

pool=blue release=blue-release-1.0.0 upstream_status=200 latency=0.045

How Failover Works

Normal Operation (Blue Active)

User Request → Nginx → App Blue → Success (200 OK)
                       ↓
                   Nginx Logs: pool=blue

Failure Scenario

Let's trace what happens when Blue crashes:

Step 1: User Makes Request

User → GET http://localhost:8080/

Step 2: Nginx Tries Blue

Nginx → app_blue:3000
        ↓
    Connection Refused (Blue is down)

Step 3: Nginx Detects Failure

Nginx marks app_blue as FAILED
(max_fails=1 threshold reached)

Step 4: Nginx Retries to Green

Nginx → app_green:3000
        ↓
    Success! (200 OK)

Step 5: User Receives Response

User ← 200 OK from Green
(User never knew Blue failed!)

Step 6: Alert Watcher Detects Failover

# watcher.py detects pool change
if pool != self.last_alerted_pool:
    send_slack_alert("Failover detected: blue → green")

Step 7: Slack Alert Sent

Failover detected: blue → green (to backup)
Release: green-release-1.0.0
Upstream: 172.18.0.4:3000
Request time: 0.05s

Real-Time Alerting System

Alert Watcher Architecture

The watcher.py script monitors Nginx logs in real-time:

class AlertWatcher:
    def __init__(self):
        # Track current state
        self.current_pool = ACTIVE_POOL
        self.last_alerted_pool = ACTIVE_POOL

        # Rolling window for error rate
        self.request_window = deque(maxlen=WINDOW_SIZE)

        # Cooldown timers
        self.last_failover_alert_time = 0
        self.last_error_alert_time = 0

How It Works

1. Log Tailing

def tail_log_file(self):
    with open(LOG_FILE, 'r') as log:
        log.seek(0, 2)  # Go to end of file
        while True:
            line = log.readline()
            if not line:
                time.sleep(0.1)
                continue
            self.process_log_line(line)

Explanation:

Opens log file in read mode
Seeks to end (like tail -f)
Continuously reads new lines
Processes each line in real-time

2. Failover Detection

def check_failover(self, pool):
    if pool == self.last_alerted_pool:
        return  # No change

    # Determine alert type
    if pool == ACTIVE_POOL:
        title = f"Recovery detected: back to {pool}"
    else:
        title = f"Failover detected: {self.last_alerted_pool} → {pool}"

    # Send alert
    self.send_slack_alert(title)

3. Error Rate Monitoring

def check_error_rate(self, all_statuses):
    # Check if any status is 5xx
    is_5xx = any(500 <= s < 600 for s in all_statuses)

    # Add to sliding window
    self.request_window.append(is_5xx)

    # Calculate error rate
    error_count = sum(self.request_window)
    total_requests = len(self.request_window)
    error_rate = (error_count / total_requests) * 100

    # Check threshold
    if error_rate > ERROR_RATE_THRESHOLD:
        self.send_slack_alert(
            f"High upstream error rate: {error_rate:.2f}%"
        )

Alert Types

1. Failover Alert

Failover detected: blue → green (to backup)
Release: green-release-1.0.0
Upstream: 172.18.0.4:3000
Request time: 0.05s

When: Primary instance fails, traffic switches to backup

2. Recovery Alert

Recovery detected: back to blue (primary)
Release: blue-release-1.0.0
Upstream: 172.18.0.3:3000
Request time: 0.03s

When: Primary instance recovers, traffic returns to primary

3. High Error Rate Alert

High upstream error rate
5xx in upstream attempts: 5.50% over last 200 requests (threshold 2%).
Current pool: green

When: Error rate exceeds threshold in sliding window

Testing the Deployment

Test 1: Verify Normal Operation

# Check which pool is active
curl http://localhost:8080/version

# Expected output:
{
  "pool": "blue",
  "release": "blue-release-1.0.0"
}

Test 2: Trigger Failover

Manual Failover Test:

# Stop Blue instance
docker compose stop app_blue

# Make requests
for i in {1..10}; do
  curl http://localhost:8080/version
  sleep 1
done

# You should see responses from Green
# Check Slack for failover alert

Test 3: Verify Zero Downtime

# In one terminal, continuously make requests
while true; do
  curl -s http://localhost:8080/version | jq -r '.pool'
  sleep 0.5
done

# In another terminal, stop Blue
docker compose stop app_blue

# Observe: No failed requests!
# Output switches from "blue" to "green" seamlessly

Test 4: Test Recovery

# Restart Blue instance
docker compose start app_blue

# Wait for health check to pass (10-15 seconds)
sleep 15

# Make requests
curl http://localhost:8080/version

# Should show Blue is active again
# Check Slack for recovery alert

Test 5: View Nginx Logs

# View real-time logs
docker compose exec nginx_proxy tail -f /var/log/nginx/access.log

# Example output:
pool=blue release=blue-release-1.0.0 upstream_status=200 latency=0.023
pool=green release=green-release-1.0.0 upstream_status=502, 200 latency=0.045

Common Issues and Troubleshooting

Issue 1: Failover Not Happening

Symptoms: Blue fails but traffic doesn't switch to Green

Solutions:

# Check Nginx config
docker compose exec nginx_proxy cat /etc/nginx/nginx.conf.processed

# Restart Nginx
docker compose restart nginx_proxy

# Verify Green is healthy
docker compose exec app_green wget -qO- http://localhost:3000/healthz

Issue 2: Alerts Not Sending to Slack

Symptoms: Failover happens but no Slack notification

Solutions:

Verify Webhook URL:

# Check .env file
cat .env | grep SLACK_WEBHOOK_URL

# Test webhook manually
curl -X POST YOUR_WEBHOOK_URL \
  -H 'Content-Type: application/json' \
  -d '{"text":"Test alert"}'

Restart Watcher:

docker compose restart alert_watcher

Issue 3: Both Instances Receiving Traffic

Symptoms: Requests alternate between Blue and Green

Solution:

# Ensure Green has "backup" directive
# Check nginx.conf.template
# Rebuild and restart
docker compose down
docker compose up -d --build

Debugging Commands Cheat Sheet

# View all container status
docker compose ps

# View all logs
docker compose logs

# View specific service logs
docker compose logs -f nginx_proxy

# Execute command in container
docker compose exec nginx_proxy sh

# Restart specific service
docker compose restart app_blue

# Clean up everything
docker compose down -v

Key Takeaways

What You've Learned

Blue-Green Deployment
- How to implement zero-downtime deployments
- Difference between blue-green and load balancing
- When to use blue-green vs. other strategies
Nginx as a Reverse Proxy
- Upstream configuration
- Failover mechanisms
- Health checks and timeouts
- Structured logging
Real-Time Monitoring
- Log tailing and parsing
- Event detection
- Sliding window calculations
- Alert deduplication
Slack Integration
- Webhook setup
- Alert formatting
- Error handling
Docker Orchestration
- Multi-container applications
- Service dependencies
- Volume management
- Health checks

Real-World Applications

This pattern is used by:

Netflix - Canary deployments with instant rollback
Amazon - Blue-green for critical services
Heroku - Platform-level blue-green deployments
GitHub - Zero-downtime deployments

Next Steps

Enhance the System
- Add database with replication
- Implement canary deployments
- Add A/B testing capabilities
Improve Monitoring
- Add Prometheus metrics
- Create Grafana dashboards
- Implement distributed tracing
Scale the Architecture
- Deploy to Kubernetes
- Use managed load balancers
- Implement auto-scaling

Additional Resources

Documentation

Tools

Nginx - High-performance web server
Docker - Containerization platform
Slack - Team communication

Conclusion

Congratulations! You've just learned how to implement a production-ready blue-green deployment system with automatic failover and real-time alerting. This is a critical skill for modern DevOps engineers.

Remember:

Test thoroughly before deploying to production
Monitor continuously - you can't fix what you can't see
Automate everything - manual processes lead to errors
Document your decisions - future you will thank you

Blue-green deployment is just one piece of the DevOps puzzle. The principles you've learned here - automation, monitoring, resilience, and rapid recovery - apply to all aspects of modern infrastructure.

Keep experimenting, keep learning, and most importantly, keep building!

Full Project Repository: HNG DevOps on GitHub

Happy deploying!

Complete Beginner's Guide to Building a Microservices TODO Application with Docker and Traefik

cypher682 — Tue, 09 Dec 2025 14:35:15 +0000

Introduction

Welcome! This guide will walk you through a real-world microservices application - a TODO app that demonstrates how different services written in different programming languages can work together seamlessly. If you're new to DevOps or microservices, don't worry - I'll explain everything step by step!

Repository: HNG DevOps on GitHub

What You'll Learn:

How microservices architecture works in practice
How to use Docker to containerize multiple services
How to set up Traefik as a reverse proxy
How different programming languages work together
How services communicate with each other
How to implement authentication across microservices

What is a Microservices Architecture?

The Traditional Way (Monolithic)

Imagine building a house where everything - kitchen, bedroom, bathroom - is one giant room. If you want to renovate the kitchen, you might affect the bedroom too. That's a monolithic application.

The Microservices Way

Now imagine a house where each room is a separate, self-contained unit. You can renovate the kitchen without touching the bedroom. Each room has its own entrance and can be accessed independently. That's microservices architecture.

Why Microservices?

Independent Development: Different teams can work on different services

Technology Flexibility: Use the best language/framework for each service

Scalability: Scale only the services that need it

Fault Isolation: If one service fails, others keep running

Easier Maintenance: Smaller codebases are easier to understand

Project Overview

This project is a TODO application that allows users to:

Register and log in (Authentication)
Create, read, update, and delete TODO items
View user profiles
Track operations through logging

The Magic Behind It

The application is split into 5 independent microservices:

Frontend (Vue.js) - The user interface
Auth API (Go) - Handles user authentication
TODOs API (Node.js) - Manages TODO items
Users API (Java/Spring Boot) - Manages user profiles
Log Message Processor (Python) - Processes and logs operations

Understanding the Components

Let's break down each component in simple terms:

1. Frontend (Vue.js - JavaScript)

What it does: This is what users see and interact with - the web interface.

Think of it as: The cashier at a restaurant who takes your order and brings you food.

Why Vue.js? It's a modern, lightweight JavaScript framework perfect for building interactive user interfaces.

2. Auth API (Go)

What it does: Handles user authentication and generates JWT (JSON Web Tokens) for secure access.

Think of it as: The security guard who checks your ID and gives you a badge to enter the building.

How JWT Works:

1. User logs in with username/password
2. Auth API verifies credentials
3. Auth API generates a JWT token (like a temporary pass)
4. User includes this token in future requests
5. Other services verify the token to confirm identity

Why Go? Go is fast, efficient, and excellent for building high-performance APIs.

3. TODOs API (Node.js)

What it does: Manages all TODO operations - create, read, update, delete.

Think of it as: The kitchen that prepares your food orders.

Redis Integration:
When you create or delete a TODO, the API sends a message to Redis (a message queue), which the Log Message Processor picks up and logs.

Why Node.js? Node.js is great for I/O-heavy operations and has a rich ecosystem of packages.

4. Users API (Java/Spring Boot)

What it does: Manages user profile information.

Think of it as: The HR department that maintains employee records.

Why Java/Spring Boot? Spring Boot is enterprise-grade, robust, and widely used in production environments.

5. Log Message Processor (Python)

What it does: Listens to Redis queue and logs TODO operations.

Think of it as: The security camera system that records all activities.

Why Python? Python is simple, readable, and perfect for quick scripting tasks.

6. Traefik (Reverse Proxy)

What it does: Routes incoming requests to the correct service and handles SSL/TLS certificates.

Think of it as: The receptionist who directs visitors to the right department.

Key Features:

Automatic service discovery
SSL/TLS certificate management (Let's Encrypt)
HTTP to HTTPS redirection
Path-based routing

Technology Stack

Here's a summary of all technologies used:

Component	Technology	Purpose
Frontend	Vue.js	User interface
Auth API	Go	Authentication service
TODOs API	Node.js	TODO management
Users API	Java/Spring Boot	User profile management
Log Processor	Python	Logging service
Message Queue	Redis	Inter-service communication
Reverse Proxy	Traefik	Request routing & SSL
Containerization	Docker	Package services
Orchestration	Docker Compose	Manage multiple containers

Architecture Overview

Visual Architecture

                    Internet
                       |
                       v
              +----------------+
              |    Traefik     |
              | (Port 80/443)  |
              +----------------+
                       |
    +------------------+------------------+
    |                  |                  |
    v                  v                  v
+----------+    +----------+    +----------+
| Frontend |    | Auth API |    |TODOs API |
| (Vue.js) |    |   (Go)   |    | (Node.js)|
+----------+    +----------+    +----------+
    |                  |                  |
    |                  v                  |
    |          +-------------+            |
    +--------->|  Users API  |<-----------+
               | (Java)      |
               +-------------+


+-------------+              +-------------+
| Log Message |<--Redis------|  TODOs API  |
| Processor   |              +-------------+
| (Python)    |
+-------------+

Request Flow Example

Let's trace what happens when a user creates a TODO:

User Action: User fills out TODO form and clicks "Create"
Frontend: Sends POST request to https://example.com/api/todos
Traefik: Receives request, checks routing rules, forwards to TODOs API
TODOs API:
- Validates JWT token
- Creates TODO in database
- Publishes "TODO created" message to Redis
- Returns success response
Redis: Stores message in queue
Log Message Processor:
- Reads message from Redis
- Logs the operation
Frontend: Receives success response, updates UI

Setting Up the Project

Prerequisites

Before you start, make sure you have:

Docker installed (Download here)
Docker Compose installed (usually comes with Docker Desktop)
Basic command line knowledge
A text editor (VS Code, Sublime, etc.)

Step 1: Clone the Repository

git clone https://github.com/cypher682/hng13-stage-3-devops.git
cd DevOps-Stage-6

Step 2: Understand the Project Structure

DevOps-Stage-6/
├── frontend/              # Vue.js application
│   ├── Dockerfile        # Instructions to build frontend container
│   └── src/              # Source code
├── auth-api/             # Go authentication service
│   ├── Dockerfile
│   └── main.go
├── todos-api/            # Node.js TODO service
│   ├── Dockerfile
│   └── server.js
├── users-api/            # Java Spring Boot service
│   ├── Dockerfile
│   └── src/
├── log-message-processor/ # Python logging service
│   ├── Dockerfile
│   └── processor.py
├── docker-compose.yml    # Orchestration configuration
└── .env                  # Environment variables

Step 3: Configure Environment Variables

The .env file contains configuration for all services:

# User credentials (for testing)
USER1_USERNAME=user1
USER1_PASSWORD=password1

# JWT Secret (for token generation)
JWT_SECRET=myfancysecret

# Redis Configuration
REDIS_HOST=redis-queue
REDIS_PORT=6379
REDIS_CHANNEL=log_channel

Important: In production, never commit .env files with real credentials!

Step 4: Build and Run

# Build all services
docker compose build

# Start all services
docker compose up -d

# Check if all services are running
docker compose ps

What -d means: Runs containers in "detached" mode (in the background).

Step 5: Verify Services

Check that all containers are running:

docker compose ps

You should see all services with status "Up" or "Running":

traefik
frontend
auth-api
todos-api
users-api
log-message-processor
redis-queue

Understanding Docker Compose

What is Docker Compose?

Docker Compose is a tool for defining and running multi-container Docker applications. Instead of running each container manually, you define everything in a docker-compose.yml file.

Key Sections Explained

1. Services Definition

services:
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped

Explanation:

services: - Defines all containers
traefik: - Service name
image: - Docker image to use
container_name: - Name for the container
restart: - Restart policy

2. Port Mapping

ports:
  - "80:80"
  - "443:443"

Format: HOST_PORT:CONTAINER_PORT

Explanation:

Maps port 80 on your computer to port 80 in the container
Allows you to access services from your browser

3. Networks

networks:
  - app-network

Explanation:

All services on the same network can communicate with each other
Services can reference each other by service name (e.g., http://auth-api:8081)

4. Traefik Labels (Routing)

labels:
  - "traefik.enable=true"
  - "traefik.http.routers.frontend.rule=Host(`example.com`)"
  - "traefik.http.routers.frontend.entrypoints=websecure"

Explanation:

Tells Traefik to route requests for example.com to this service
Enables HTTPS with automatic certificate generation

How Services Communicate

Internal Communication (Service-to-Service)

Services communicate using service names as hostnames:

// In Frontend (Vue.js)
const AUTH_API_ADDRESS = 'http://auth-api:8081';
const TODOS_API_ADDRESS = 'http://todos-api:8082';

Why this works:

Docker Compose creates a network where services can find each other by name
No need for IP addresses

External Communication (User-to-Service)

Users access services through Traefik:

User → https://example.com → Traefik → Frontend
User → https://example.com/api/auth → Traefik → Auth API
User → https://example.com/api/todos → Traefik → TODOs API

Authentication Flow

1. User submits login form
   ↓
2. Frontend sends credentials to Auth API
   ↓
3. Auth API verifies with Users API
   ↓
4. Auth API generates JWT token
   ↓
5. Frontend stores token
   ↓
6. Frontend includes token in all future requests
   ↓
7. Each API validates token before processing

Testing the Application

1. Access the Frontend

Open your browser and navigate to:

http://localhost:8080

2. Test Authentication

Username: user1
Password: password1

3. Create a TODO

Click "Add TODO"
Enter a title and description
Click "Save"
Check that the TODO appears in the list

4. Check Logs

View logs from the Log Message Processor:

docker compose logs -f log-message-processor

You should see log entries for TODO creation.

5. Test API Endpoints Directly

Get all users:

curl http://localhost:8080/api/users

Login (get JWT token):

curl -X POST http://localhost:8080/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"user1","password":"password1"}'

Common Issues and Troubleshooting

Issue 1: Containers Won't Start

Symptoms: Containers show as "Exited" or "Restarting"

Solutions:

# Check logs for specific service
docker compose logs auth-api

# Rebuild containers
docker compose down
docker compose build --no-cache
docker compose up -d

Issue 2: Port Already in Use

Error: bind: address already in use

Solutions:

# Find what's using the port (Windows)
netstat -ano | findstr :80

# Change port in docker-compose.yml
ports:
  - "8000:80"  # Use port 8000 instead

Issue 3: Services Can't Communicate

Symptoms: Frontend can't reach APIs

Solutions:

# Verify all services are on the same network
docker network inspect devops-stage-6_app-network

# Restart networking
docker compose down
docker compose up -d

Debugging Commands Cheat Sheet

# View all running containers
docker compose ps

# View logs for all services
docker compose logs

# View logs for specific service
docker compose logs -f frontend

# Restart specific service
docker compose restart todos-api

# Stop all services
docker compose down

# View resource usage
docker stats

Key Takeaways

What You've Learned

Microservices Architecture
- How to split an application into independent services
- Benefits of using different languages for different services
- How services communicate with each other
Docker & Containerization
- How to containerize applications
- How to use Docker Compose for multi-container applications
- How to manage container networking
Reverse Proxy with Traefik
- How to route requests to different services
- How to automatically manage SSL certificates
- How to use labels for service discovery
Inter-Service Communication
- Synchronous communication (HTTP/REST)
- Asynchronous communication (Message Queues)
- Authentication across services (JWT)

Real-World Applications

This architecture pattern is used by companies like:

Netflix - Hundreds of microservices
Amazon - Service-oriented architecture
Uber - Microservices for different features
Spotify - Independent teams, independent services

Next Steps

To deepen your understanding:

Modify the Application
- Add a new microservice (e.g., Comments API)
- Implement a database for persistent storage
- Add user registration functionality
Improve the Infrastructure
- Add monitoring with Prometheus and Grafana
- Implement centralized logging
- Add CI/CD pipeline with GitHub Actions
Security Enhancements
- Implement rate limiting
- Add API gateway
- Use secrets management

Additional Resources

Documentation

Tools

Docker Desktop
Postman - API testing
Portainer - Docker management UI

Conclusion

Congratulations! You've just learned how to build, deploy, and manage a complete microservices application. This project demonstrates real-world DevOps practices used by major tech companies.

Remember:

Start small - Don't try to build everything at once
Understand each component - Know what each service does
Experiment - Break things and fix them (that's how you learn!)
Read logs - They're your best friend when debugging
Keep learning - DevOps is constantly evolving

The skills you've gained here - Docker, microservices, reverse proxies, and service orchestration - are highly valuable in the industry. Keep practicing, keep building, and most importantly, keep learning!

Full Project Repository: HNG DevOps on GitHub

Building Your Own Virtual Private Cloud on Linux: A Deep Dive into Network Namespaces

cypher682 — Sun, 09 Nov 2025 17:07:03 +0000

Introduction: Why Build a VPC from Scratch?

Amazon Web Services revolutionized cloud computing with Virtual Private Clouds (VPCs), allowing users to create isolated network environments in the cloud. But have you ever wondered how VPCs actually work under the hood?

In this project, I recreated AWS VPC functionality on a single Linux machine using native networking primitives. No Docker, no Kubernetes—just pure Linux networking: network namespaces, bridges, veth pairs, and iptables.

What You'll Learn

How network isolation works at the kernel level
Linux network namespaces as lightweight containers
Bridging and routing fundamentals
NAT implementation with iptables
Building infrastructure automation tools

Real-World Applications

Understanding cloud provider networking internals - See how AWS/Azure/GCP implement VPCs
Building custom network isolation for multi-tenant systems
DevOps and infrastructure automation skills - Create your own networking tools
Debugging complex network issues - Deep knowledge of Linux networking stack

Architecture Overview

A VPC in AWS provides:

Isolated network space with your own IP range (CIDR)
Subnets that partition your VPC into smaller networks
Internet Gateway for public subnet internet access
NAT Gateway for private subnet outbound access
Security Groups for firewall rules
VPC Peering for cross-VPC communication

My Implementation Stack

┌─────────────────────────────────────────────┐
│           Linux Host System                  │
│                                              │
│  ┌────────────────────────────────────────┐ │
│  │    VPC1 (10.0.0.0/16)                  │ │
│  │                                        │ │
│  │  br-vpc1 (Linux Bridge = VPC Router)  │ │
│  │        │                    │          │ │
│  │    veth-pair           veth-pair      │ │
│  │        │                    │          │ │
│  │  ┌─────▼────┐        ┌─────▼────┐    │ │
│  │  │ Public   │        │ Private  │    │ │
│  │  │ Subnet   │        │ Subnet   │    │ │
│  │  │ Namespace│        │ Namespace│    │ │
│  │  │10.0.1.2  │        │10.0.2.2  │    │ │
│  │  └─────┬────┘        └─────┬────┘    │ │
│  │        │                    X          │ │
│  │    NAT (iptables)      No Internet    │ │
│  └────────┼────────────────────────────────┘│
│           │                                  │
│      [eth0] ──► Internet                   │
└─────────────────────────────────────────────┘

Component Mapping

AWS VPC Concept	Linux Implementation
VPC	Linux Bridge (br-vpc1)
Subnet	Network Namespace
Subnet Connection	veth pair
Internet Gateway	iptables NAT MASQUERADE
Route Table	ip route commands
Security Group	iptables INPUT rules
VPC Peering	veth pair between bridges

Part 1: Understanding Network Namespaces

Network namespaces are Linux's way of creating isolated network stacks. Each namespace has its own:

Network interfaces
IP addresses
Routing tables
iptables rules

Why This Matters: This is the same technology Docker uses for container networking. Understanding this gives you deep insights into containerization.

Creating Isolation

# In vpcctl.py - SubnetManager.add()
run_command(f"ip netns add {ns_name}")  # Create isolated namespace

When you create a namespace, the Linux kernel creates a completely separate network stack. Processes inside can't see or access the host's network—perfect isolation!

Test It Yourself

# Create namespace
sudo ip netns add test-ns

# List interfaces in host
ip link

# List interfaces in namespace (only loopback!)
sudo ip netns exec test-ns ip link

You'll notice the namespace starts with only a loopback interface. It's completely isolated!

Part 2: Connecting Namespaces - The veth Pair Magic

Network namespaces are isolated, so we need a way to connect them. Enter veth pairs—virtual ethernet cables.

Conceptual Model

Think of a veth pair as a virtual ethernet cable with two ends:

One end plugs into the namespace
Other end plugs into the host or bridge

# Creating the connection
run_command(f"ip link add {veth_host} type veth peer name {veth_ns}")
run_command(f"ip link set {veth_host} master {bridge}")  # Connect to bridge
run_command(f"ip link set {veth_ns} netns {ns_name}")    # Move to namespace

Why This Works: Packets entering one end of the veth pair automatically come out the other end—like a wormhole for network traffic.

IP Address Assignment

net_info = NetworkUtils.get_network_info(cidr)
# Assign IP inside namespace
run_command(
    f"ip netns exec {ns_name} ip addr add "
    f"{net_info['first_host']}/{net_info['prefix']} dev {veth_ns}"
)

Key Insight: We use the first usable IP (.2) for the subnet, and the bridge gets the gateway IP (.1). This mirrors how AWS assigns IPs in subnets.

Part 3: The Bridge - Your VPC Router

A Linux bridge is like a virtual network switch. It forwards packets between connected interfaces at Layer 2.

Bridge Creation

bridge_name = f"br-{name}"
run_command(f"ip link add {bridge_name} type bridge")
run_command(f"ip addr add {net_info['gateway']}/{net_info['prefix']} dev {bridge_name}")
run_command(f"ip link set {bridge_name} up")

Critical Configuration

# Disable bridge netfilter - allows direct L2 forwarding
run_command("sysctl -w net.bridge.bridge-nf-call-iptables=0")

Why This Matters: By default, Linux bridges pass traffic through iptables. For intra-VPC communication, we want direct Layer 2 switching for performance—just like a real network switch.

Routing Setup

# Inside namespace: route everything through bridge gateway
run_command(f"ip netns exec {ns_name} ip route add default via {gateway_ip}")

This makes the bridge act as the default gateway—all traffic from namespaces flows through it.

Part 4: NAT Gateway - Internet Access

Private networks use RFC 1918 addresses (10.x.x.x, 172.16.x.x, 192.168.x.x) that aren't routable on the internet. NAT (Network Address Translation) solves this.

NAT Implementation

# Enable IP forwarding (routing between interfaces)
run_command("sysctl -w net.ipv4.ip_forward=1")

# MASQUERADE: Replace source IP with host's public IP
run_command(
    f"iptables -t nat -A POSTROUTING -s {cidr} -o {interface} -j MASQUERADE"
)

# Allow forwarding from VPC to internet
run_command(
    f"iptables -A FORWARD -s {cidr} -i {bridge} -o {interface} -j ACCEPT"
)

# Allow return traffic
run_command(
    f"iptables -A FORWARD -d {cidr} -i {interface} -o {bridge} "
    f"-m state --state RELATED,ESTABLISHED -j ACCEPT"
)

How MASQUERADE Works

Packet leaves namespace with source IP 10.0.1.2
Reaches host via bridge
iptables MASQUERADE rewrites source to host's public IP
Internet sees request from host, not internal IP
Response comes back, iptables rewrites destination back to 10.0.1.2
Packet forwarded to namespace

Key Insight: We only NAT public subnets. Private subnets remain isolated—they can reach other subnets within the VPC but not the internet.

Part 5: VPC Isolation & Peering

Default Isolation

Without configuration, VPCs can't communicate. Why?

Each VPC has its own bridge
No routes exist between bridges
iptables FORWARD policy is DROP by default

Testing Isolation

# From vpc1 namespace, try to reach vpc2
sudo ip netns exec vpc1-public ping 172.16.1.2
# Result: Network unreachable (no route to 172.16.0.0/16)

VPC Peering Implementation

The tricky part: packets from a namespace need to reach another VPC's bridge.

Solution:

# Create veth pair between bridges
run_command(f"ip link add {veth1} type veth peer name {veth2}")
run_command(f"ip link set {veth1} master {vpc1['bridge']}")
run_command(f"ip link set {veth2} master {vpc2['bridge']}")

# CRITICAL: Add routes in EACH namespace
for subnet in vpc1["subnets"]:
    ns_name = subnet["namespace"]
    run_command(
        f"ip netns exec {ns_name} ip route add {vpc2['cidr']} "
        f"via {vpc1['gateway']}"
    )

Why This Works

Namespace sends packet to VPC2 CIDR
Route points to its own gateway (bridge)
Bridge forwards to peering veth pair
Packet arrives at VPC2's bridge
VPC2 bridge forwards to destination namespace

Common Mistake I Made: Initially, I added routes on the host routing table. This doesn't work because packets originate from inside namespaces, which have their own routing tables!

Part 6: Firewall Rules (Security Groups)

AWS Security Groups are stateful firewalls. We simulate this with iptables inside namespaces.

JSON Policy Design

{
  "subnet": "10.0.1.0/24",
  "ingress": [
    {"port": 80, "protocol": "tcp", "action": "allow"},
    {"port": 22, "protocol": "tcp", "action": "deny"}
  ]
}

Implementation

# Essential: Allow established connections (stateful behavior)
run_command(
    f"ip netns exec {ns_name} iptables -A INPUT "
    f"-m state --state ESTABLISHED,RELATED -j ACCEPT"
)

# Apply custom rules
for rule in policy.get("ingress", []):
    port = rule["port"]
    protocol = rule["protocol"]
    action = "ACCEPT" if rule["action"] == "allow" else "DROP"
    run_command(
        f"ip netns exec {ns_name} iptables -A INPUT "
        f"-p {protocol} --dport {port} -j {action}"
    )

Why ESTABLISHED,RELATED Matters: Without this, responses to outbound connections would be blocked. The stateful rule tracks connections and allows return traffic automatically.

The CLI Tool: vpcctl

I built a Python CLI tool to automate all VPC operations.

Installation

git clone https://github.com/cypher682/hng13-stage-4-devops.git
cd hng13-stage-4-devops
chmod +x vpcctl.py

Usage Examples

Create VPC:

sudo ./vpcctl.py vpc create --name vpc1 --cidr 10.0.0.0/16

Add Subnets:

# Public subnet
sudo ./vpcctl.py subnet add \
  --vpc vpc1 \
  --name public \
  --cidr 10.0.1.0/24 \
  --type public

# Private subnet
sudo ./vpcctl.py subnet add \
  --vpc vpc1 \
  --name private \
  --cidr 10.0.2.0/24 \
  --type private

Enable NAT Gateway:

sudo ./vpcctl.py nat enable --vpc vpc1 --interface eth0

Deploy Web Server:

sudo ./vpcctl.py deploy webserver \
  --vpc vpc1 \
  --subnet public \
  --port 8080

Create VPC Peering:

sudo ./vpcctl.py peer create --vpc1 vpc1 --vpc2 vpc2

Apply Firewall Rules:

sudo ./vpcctl.py firewall apply \
  --vpc vpc1 \
  --subnet public \
  --policy policy.json

List All VPCs:

sudo ./vpcctl.py vpc list

Delete VPC:

sudo ./vpcctl.py vpc delete --name vpc1

Testing & Validation

Test 1: Intra-VPC Communication

# Create VPC with two subnets
sudo ./vpcctl.py vpc create --name vpc1 --cidr 10.0.0.0/16
sudo ./vpcctl.py subnet add --vpc vpc1 --name public \
  --cidr 10.0.1.0/24 --type public
sudo ./vpcctl.py subnet add --vpc vpc1 --name private \
  --cidr 10.0.2.0/24 --type private

# Test connectivity
sudo ip netns exec vpc1-public ping -c 3 10.0.2.2

✅ Expected: Success (same VPC, bridge routes traffic)

What's Happening:

Packet leaves public namespace (10.0.1.2)
Goes through veth to bridge
Bridge forwards to private subnet's veth
Arrives at private namespace (10.0.2.2)

Test 2: NAT Gateway

# Enable NAT
sudo ./vpcctl.py nat enable --vpc vpc1 --interface eth0

# Test public subnet internet access
sudo ip netns exec vpc1-public ping -c 3 8.8.8.8

✅ Expected: Success

# Test private subnet (should fail)
sudo ip netns exec vpc1-private ping -c 2 8.8.8.8

✅ Expected: Timeout (no NAT rule for private subnet)

Verification:

# Check NAT rules
sudo iptables -t nat -L POSTROUTING -n -v
# Should show MASQUERADE rule for 10.0.1.0/24 only

Test 3: VPC Isolation

# Create second VPC
sudo ./vpcctl.py vpc create --name vpc2 --cidr 172.16.0.0/16
sudo ./vpcctl.py subnet add --vpc vpc2 --name public \
  --cidr 172.16.1.0/24 --type public

# Try to communicate (should fail)
sudo timeout 3 ip netns exec vpc1-public ping 172.16.1.2

✅ Expected: Network unreachable

Test 4: VPC Peering

# Create peering
sudo ./vpcctl.py peer create --vpc1 vpc1 --vpc2 vpc2

# Now communication should work
sudo ip netns exec vpc1-public ping -c 3 172.16.1.2

✅ Expected: Success

# Verify routes were added
sudo ip netns exec vpc1-public ip route
# Should show: 172.16.0.0/16 via 10.0.0.1

Test 5: Firewall Rules

# Deploy web server
sudo ./vpcctl.py deploy webserver --vpc vpc1 --subnet public --port 8080

# Test before firewall
curl http://10.0.1.2:8080

✅ Works

# Apply restrictive policy
sudo ./vpcctl.py firewall apply \
  --vpc vpc1 \
  --subnet public \
  --policy test-policy.json

# Test allowed port
curl http://10.0.1.2:8080

✅ Still works (port 8080 is allowed in policy)

Complete Demo Script

Here's the full automated demo:

#!/bin/bash

# 1. Create VPC
sudo ./vpcctl.py vpc create --name vpc1 --cidr 10.0.0.0/16

# 2. Add subnets
sudo ./vpcctl.py subnet add --vpc vpc1 --name public \
  --cidr 10.0.1.0/24 --type public
sudo ./vpcctl.py subnet add --vpc vpc1 --name private \
  --cidr 10.0.2.0/24 --type private

# 3. Enable NAT
IFACE=$(ip route | grep default | awk '{print $5}' | head -1)
sudo ./vpcctl.py nat enable --vpc vpc1 --interface $IFACE

# 4. Deploy web servers
sudo ./vpcctl.py deploy webserver --vpc vpc1 --subnet public --port 8080
sudo ./vpcctl.py deploy webserver --vpc vpc1 --subnet private --port 8081

# 5. Test intra-VPC communication
sudo ip netns exec vpc1-public ping -c 3 10.0.2.2

# 6. Test NAT gateway
sudo ip netns exec vpc1-public ping -c 3 8.8.8.8
sudo timeout 3 ip netns exec vpc1-private ping -c 2 8.8.8.8

# 7. Create second VPC
sudo ./vpcctl.py vpc create --name vpc2 --cidr 172.16.0.0/16
sudo ./vpcctl.py subnet add --vpc vpc2 --name public \
  --cidr 172.16.1.0/24 --type public
sudo ./vpcctl.py nat enable --vpc vpc2 --interface $IFACE

# 8. Test VPC isolation
sudo timeout 3 ip netns exec vpc1-public ping -c 2 172.16.1.2

# 9. Create VPC peering
sudo ./vpcctl.py peer create --vpc1 vpc1 --vpc2 vpc2

# 10. Test after peering
sudo ip netns exec vpc1-public ping -c 3 172.16.1.2

# 11. Apply firewall rules
sudo ./vpcctl.py firewall apply --vpc vpc1 --subnet public \
  --policy test-policy.json

# 12. View logs
tail -30 /var/lib/vpcctl/vpcctl.log

# 13. List resources
sudo ./vpcctl.py vpc list

# 14. Cleanup
sudo ./vpcctl.py vpc delete --name vpc1
sudo ./vpcctl.py vpc delete --name vpc2

Cleanup Process

Proper cleanup is critical. Orphaned namespaces and iptables rules can cause issues.

# Delete VPC (automated cleanup)
sudo ./vpcctl.py vpc delete --name vpc1

What happens internally:

Kill all processes in namespaces
Delete namespaces
Remove veth pairs
Delete iptables rules
Remove bridge
Clean up state file

Emergency Cleanup

sudo ./cleanup.sh
# Removes ALL network resources created by vpcctl

Verification

# Should be empty
ip netns list
ip link show type bridge | grep br-

# Check iptables
sudo iptables -L FORWARD -n
sudo iptables -t nat -L POSTROUTING -n

Challenges & Solutions

Challenge 1: VPC Peering Not Working

Problem: Routes added to host routing table, but packets originate from namespaces.

Solution: Add routes inside each namespace pointing to their respective gateways.

# Wrong approach
run_command(f"ip route add {vpc2['cidr']} via {vpc2_peer_ip}")

# Correct approach
for subnet in vpc1["subnets"]:
    ns_name = subnet["namespace"]
    run_command(
        f"ip netns exec {ns_name} ip route add {vpc2['cidr']} "
        f"via {vpc1['gateway']}"
    )

Challenge 2: Firewall Blocking Everything

Problem: Applied rules but forgot to allow established connections.

Solution: Always add stateful rules first:

run_command(
    f"ip netns exec {ns_name} iptables -A INPUT "
    f"-m state --state ESTABLISHED,RELATED -j ACCEPT"
)

Challenge 3: Bridge Netfilter Interference

Problem: iptables was processing bridge traffic, causing performance issues.

Solution: Disable bridge netfilter:

sysctl -w net.bridge.bridge-nf-call-iptables=0

Key Takeaways

Network Namespaces provide true isolation—the foundation of containers
veth Pairs are the glue connecting isolated environments
Bridges act as virtual switches for Layer 2 forwarding
iptables is incredibly powerful for NAT, routing, and firewalling
Proper cleanup is essential for infrastructure automation

Real-World Applications

Container Networking: Docker/Kubernetes use the same primitives
Multi-tenant Systems: Isolate customer workloads
Cloud Provider Internals: Understanding how AWS VPC really works
Security: Network segmentation and isolation strategies

Architecture Deep Dive

Let me explain how packets flow through the system:

Scenario 1: Public Subnet → Internet

[Namespace] 10.0.1.2
     ↓ (veth pair)
[Bridge] br-vpc1 (10.0.0.1)
     ↓ (routing decision)
[iptables NAT] MASQUERADE (rewrites source IP)
     ↓
[eth0] → Internet

Scenario 2: Namespace → Namespace (Same VPC)

[Namespace A] 10.0.1.2
     ↓ (veth pair)
[Bridge] br-vpc1 (L2 switching)
     ↓ (veth pair)
[Namespace B] 10.0.2.2

Scenario 3: VPC Peering

[VPC1 Namespace] 10.0.1.2
     ↓ (veth pair)
[VPC1 Bridge] br-vpc1
     ↓ (peering veth pair)
[VPC2 Bridge] br-vpc2
     ↓ (veth pair)
[VPC2 Namespace] 172.16.1.2

Performance Considerations

Bridge vs Router

Using bridges instead of routing gives us:

Lower latency - Layer 2 switching is faster than Layer 3 routing
Higher throughput - No routing table lookups for intra-VPC traffic
Simpler configuration - Bridges handle MAC learning automatically

Namespace Overhead

Network namespaces are lightweight:

~1KB memory per namespace
Negligible CPU overhead for namespace switching
Near-native performance for network operations

Scalability Limits

On a typical Linux system:

~100,000+ namespaces possible
Limited by file descriptors and memory
iptables rules become the bottleneck at scale (~10,000+ rules)

Security Considerations

Isolation Guarantees

Network namespaces provide:

Complete network stack isolation
Separate iptables rules
Independent routing tables
Process isolation (can't see other namespace processes)

Attack Surface

Potential security concerns:

Host compromise affects all VPCs
Bridge vulnerabilities (MAC flooding, ARP spoofing)
iptables misconfigurations can leak traffic

Best Practices

Least privilege - Only enable NAT for public subnets
Default deny - Block all traffic, then allow specific flows
Audit logging - Log all iptables rules and changes
Regular cleanup - Remove unused resources

Future Enhancements

What's Missing?

Compared to AWS VPC, this implementation lacks:

IPv6 Support
- Current: IPv4 only
- Enhancement: Dual-stack networking
DNS Server
- Current: No internal DNS
- Enhancement: dnsmasq in each VPC
DHCP
- Current: Static IP assignment
- Enhancement: Dynamic IP allocation
Network ACLs
- Current: Security groups only
- Enhancement: Subnet-level firewalls
VPC Flow Logs
- Current: Basic logging
- Enhancement: Detailed traffic logging
Elastic IPs
- Current: No persistent public IPs
- Enhancement: Static IP mapping

Implementation Ideas

DNS Server:

def add_dns_server(vpc_name):
    # Start dnsmasq in namespace
    run_command(
        f"ip netns exec {vpc_name}-dns "
        f"dnsmasq --interface=lo --bind-interfaces "
        f"--listen-address=10.0.0.2"
    )

DHCP Server:

def add_dhcp_server(vpc_name, cidr):
    # Configure dnsmasq for DHCP
    run_command(
        f"ip netns exec {vpc_name}-dhcp "
        f"dnsmasq --dhcp-range={start_ip},{end_ip},12h"
    )

Comparison with Real Cloud VPCs

Feature	My Implementation	AWS VPC
Isolation	✅ Network namespaces	✅ Hypervisor-level
Subnets	✅ Multiple per VPC	✅ Multiple per VPC
NAT Gateway	✅ iptables MASQUERADE	✅ Managed NAT service
VPC Peering	✅ veth pairs	✅ Software-defined networking
Security Groups	✅ iptables rules	✅ Stateful firewall
Network ACLs	❌ Not implemented	✅ Subnet-level firewall
DNS	❌ Not implemented	✅ Route 53 integration
DHCP	❌ Static IPs only	✅ DHCP options
Flow Logs	⚠️ Basic logging	✅ Detailed flow logs
IPv6	❌ IPv4 only	✅ Dual-stack
Multi-region	❌ Single host	✅ Global infrastructure
HA/Redundancy	❌ Single point of failure	✅ Multi-AZ redundancy

Learning Resources

Official Documentation

Books

Linux Networking Cookbook by Carla Schroder
TCP/IP Illustrated by W. Richard Stevens
Linux Kernel Networking by Rami Rosen

Online Courses

Linux Foundation: Linux Networking and Administration
Pluralsight: Linux Networking Fundamentals
Udemy: Linux Networking Masterclass

Project Repository

GitHub: [https://github.com/cypher682/vpcctl-linux-networking]

Video Demo: [https://youtu.be/7LYUl3hc3xE]

Repository Structure

vpcctl-project/
├── README.md              # Complete documentation
├── vpcctl             # Main CLI tool
├── demo.sh                # Automated demo
├── cleanup.sh             # Emergency cleanup
├── docs/
│   ├── architecture-diagram.png

Conclusion

Building a VPC from scratch taught me more about networking than reading documentation ever could. Understanding these primitives—namespaces, bridges, veth pairs, and iptables—gives you superpowers when debugging container networking, understanding cloud provider internals, or designing multi-tenant systems.

Key Lessons:

Linux networking is powerful - You don't need specialized tools for complex networking
Cloud abstractions are implementations - AWS VPC is just well-packaged Linux networking
Isolation is achievable - Network namespaces provide true isolation
Automation is essential - Infrastructure as code makes everything reproducible

Got questions? Drop them in the comments below! 👇

Found this useful? Star the repo and share with fellow DevOps engineers! ⭐

GitHub:https://github.com/cypher682)

Forem: cypher682

Designing a Production-Grade Blue-Green ECS Platform on AWS with Terraform

Table of Contents

What Was Built

Architecture Overview

Terraform Module Structure

Security Design

1. Network Isolation (Security Groups)

2. IAM Role Separation

3. Secrets Management Flow

Blue-Green Deployment Mechanics

Initial State

Deployment Process

Why This Works

Data Persistence Validation

What Broke (And What I Learned)

Issue 1: RDS Connection Delay After "Available" Status

Issue 2: Docker HEALTHCHECK Missing Dependency

Issue 3: Git Bash Path Conversion on Windows

Issue 4: ALB Listener Syntax Constraint

Issue 5: PostgreSQL Minor Version Retirement

Issue 6: Terraform State Lock Timeout

Issue 7: ECR Image Pull Failures (Intermittent)

Issue 8: Terraform vs Manual Scaling Conflict

What I'd Change in Production

Cost Breakdown (Actual)

4-Hour Validation Session

Monthly Cost If Kept Running

Key Takeaways

What Worked Well

What I'd Improve

Repository & Evidence

Questions or Feedback?

Building a Production-Grade AWS Cost & Security Auditor

Core Design Principles

1. Read-Only by Design

2. Defining “Idle” Using CloudWatch Metrics

3. Aligning Findings with CIS Benchmarks

Results from a Real AWS Account

Handling False Positives

What I Would Improve for a Production Deployment

Key Takeaways

Try It Yourself

What’s Next

Building a Production-Ready Microservices Platform with CI/CD on AWS Free Tier

Introduction

Architecture Overview

Why This Stack?

Infrastructure Setup

Microservices Implementation

API Gateway (Node.js)

User Service (Python)

Product Service (Go)

CI/CD Pipeline

Security Implementation

Monitoring Stack

Configuration Management with Ansible

Cost Optimization

Lessons Learned

Trade-offs and Limitations

Real-World Applications

Next Steps

Conclusion

Resources

Complete Beginner's Guide to Blue-Green Deployment with Nginx and Real-Time Alerting

Introduction

What is Blue-Green Deployment?

The Problem: Traditional Deployments

The Solution: Blue-Green Deployment

Real-World Analogy

Project Overview

Core Features

Understanding the Core Concepts

1. Blue-Green vs. Load Balancing

2. Nginx Upstream Configuration

3. Failover Mechanism

Architecture Overview

System Architecture Diagram

Component Breakdown

1. Nginx Proxy