Forem: Ukeme David Eseme

How I Built a Fully Automated Ethereum PoW Faucet on AWS Using Terraform + Ansible

Ukeme David Eseme — Wed, 21 Jan 2026 04:32:16 +0000

Introduction

As I step into 2026, I made a deliberate decision to narrow my focus: Node Operations and DevOps for blockchain infrastructure.

To build real depth in this niche, I started working on advanced, production-style node operations projects. One of my first targets was the Optimism ecosystem.

While setting up an end-to-end OP Stack Rollup deployment on Sepolia testnet, I quickly ran into a practical problem — I needed between 2–3 SepoliaETH, but I only had 0.5.

Public faucets were slow, rate-limited, and unreliable for serious testing. Instead of waiting, I turned this bottleneck into a side quest: building my own self-hosted Ethereum PoW faucet using proper automation and infrastructure-as-code practices.

In this article, I’ll walk through how I built a fully automated faucet stack on AWS using Terraform, Ansible, and a single orchestration script, complete with monitoring and observability.

Overview

This article documents the principles and processes for automating the complete deployment of a PoWFaucet node on AWS infrastructure, including a comprehensive monitoring stack. Using Infrastructure as Code (IaC) principles with Terraform and Ansible, you can deploy a production-ready Ethereum testnet faucet in minutes.

What is PoWFaucet?

A faucet in blockchain terminology is a service that distributes small amounts of cryptocurrency for free, typically on test networks (testnets). Developers and testers use faucets to obtain testnet tokens needed for deploying and testing smart contracts, dApps, and other blockchain applications without spending real money.

PoWFaucet is a proof-of-work based faucet for Ethereum testnets. Instead of traditional captcha-based rate limiting, users must solve computational puzzles to receive testnet ETH. This approach effectively prevents abuse while maintaining accessibility.

Use Cases

Testnet Operators: Deploy faucets for Ethereum testnets (Sepolia, Holesky, etc.)
Development Teams: Provide testnet ETH to developers
Educational Institutions: Support blockchain education programs
Protocol Testing: Facilitate testing of smart contracts and dApps

Prerequisites

AWS CLI configured with appropriate credentials
Terraform >= 1.0
Ansible >= 2.9
SSH key pair generated locally
Alchemy or infura ETH RPC Url for testnet, its free
Crypto Wallet (metamask, trustwallet etc...)
Have some testnet ETH in your wallet

To follow up with this article, access the project repo here: pow-faucet-node-repo

Architecture: Two Layers Working in Harmony

This setup follows the same automation patterns I use when managing blockchain infrastructure in production environments: infrastructure as code, configuration automation, and observability-first design. The goal isn’t just to run a faucet — it’s to build something reproducible, scalable, and maintainable.

The beauty of this project lies in how it separates concerns into two distinct layers that work together seamlessly.

Infrastructure Layer: Terraform Does the Heavy Lifting

Terraform handles all the AWS infrastructure provisioning.

We're talking about:

Compute: t3.medium EC2 instance running Ubuntu LTS
Storage: 20GB encrypted gp3 EBS volume
Network: Dedicated VPC with public subnet and Elastic IP
Security: Security group with controlled port access

Terraform Modules

The infrastructure is organized into reusable modules:

VPC Module: Creates isolated network with public subnet, internet gateway, and route tables
EC2 Module: Provisions instance with security group, manages SSH keys and Elastic IP association

This modular approach means you can easily spin up multiple faucets or adjust configurations without rewriting everything from scratch.

Application Layer: Ansible Orchestrates the Software

Once Terraform has built our infrastructure playground, Ansible steps in to configure everything. The playbook—over 400 lines of carefully orchestrated tasks—handles the installation and configuration of not just PoWFaucet itself, but an entire monitoring stack.

The Ansible playbook automates installation of:

System Setup:
- Package installation
- User creation
PoWFaucet Application
- Node.js 24 runtime
- PoWFaucet from official repository
- Systemd service for automatic startup
Monitoring Stack
- Prometheus: Metrics collection and time-series database
- Node Exporter: System-level metrics (CPU, memory, disk, network)
- Loki: Log aggregation system
- Promtail: Log shipping agent
- Grafana: Unified visualization dashboard

Configuration Management

One of the key design decisions was keeping secrets out of version control while still maintaining reproducibility. The solution? Jinja2 templates combined with environment variables.

.env file for secrets (RPC URL, private keys, Network)
Jinja2 templates for dynamic configuration
Version-controlled infrastructure definitions

ethRpcHost: "{{ eth_rpc_url }}"
ethWalletKey: "{{ eth_wallet_key }}"
faucetTitle: "{{ network }} Testnet Faucet"

Variables are injected from environment at deployment time, keeping secrets out of version control.

Security Considerations

SSH key-based authentication only
Security group restricts access to necessary ports
EBS volume encryption at rest
Secrets managed via environment variables (never committed)
Services run as non-root users
Grafana password change enforced on first login

Automated Deployment

Now, this is where everything crystallizes into something beautiful:

deploy.sh

This single script is the orchestrator that brings Terraform and Ansible together into one seamless experience; Hence the title of this post: The One Script to rule them all 😄

It's the difference between spending an afternoon manually running commands and grabbing a coffee while your infrastructure deploys itself.

This script:

Loads environment variables
Provisions AWS infrastructure
Configures the instance
Deploys all services
Imports Grafana dashboards

Script Workflow

Prerequisites Check: Validates AWS CLI, Terraform, Ansible installation
Environment Setup: Loads secrets from .env file
Infrastructure Provisioning: Terraform creates AWS resources
SSH Wait: Ensures instance is ready for configuration
Configuration Management: Ansible installs and configures all services
Verification: Displays access URLs for all services

Quick Four (4) Step Deployment

Step 1:

git clone https://github.com/UkemeSkywalker/pow-faucet-node.git

Step 2:

cd pow-faucet-node

Step 3:

Copy cp .env.example .env
Edit .env and add:

NETWORK: Your desired Ethereum testnet network (Sepoli / Hoodi etc..)
ETH_RPC_URL: Your Ethereum RPC endpoint (Alchemy/Infura)
ETH_WALLET_PRIVATE_KEY: Faucet wallet private key (without 0x prefix)

Step 4: The One Script to Rule Them All

./deploy.sh [path_to_ssh_private_key]

Example:

./deploy.sh ~/.ssh/id_rsa

For more details on manual deployment steps, check the repo README file

Once deployment is complete you should be able to access the faucet through the browser using

<your-machine-public-IP>:8080

Example

18.211.242.139:8080

Next step is to open your Metamask or which ever crypto wallet you use, get your ETH wallet address and insert it in the UI as shown below, then click Start Mining

You should see this window, to confirm mining process has started

Monitoring

Running infrastructure without monitoring is like driving with your eyes closed—you might be fine for a while, but eventually something will go wrong and you won't see it coming.

So, once mining is in progress, you can access prometheus through port 9090, to make queries from there, but its standard practice to do all of that in Grafana

<your-machine-public-IP>:9090

Example

18.211.242.139:9090

Metrics Collection

Prometheus scrapes metrics from:

Itself (Prometheus internals)
Node Exporter (system metrics)
Loki (log ingestion metrics)
Promtail (log shipping metrics)

Visualization

The automation steps in the ansible playbook already connected the datasources and imported dashboard to Grafana.

Grafana provides:

Dashboard 1860: Node Exporter metrics (system performance)
Dashboard 14055: Loki stack monitoring (log pipeline health)
Explore View: Ad-hoc log queries using LogQL

Navigate to dashboards and access the already integrated dashboards there for metrics and logs

Claim Rewards

Claiming mined tokens is very easy, just click on the Stop Mining & Claim Rewards button, then wait for the transaction to complete, then check your wallet for claimed reward, or view transaction history in the block explorer.

Lessons from the Trenches

This project was eye opening, as i learnt some subtle downsides, that aren't obvious at first.

Its worthy to note that most PoW Faucet that utilizes the browser for its User Interface, actually uses the browser for mining and not the machine its hosted on.
All threads and workers would use the compute resources in your local machine, which might spin up your PC fan.
Also note that your wallet address needs to be verified on Gitcoin Passport, to prevent any form of authentication issues, I didn't face this, cause i have already verified my account before then.
The amount of already existing Testnet ETH tokens in your wallet would determine the speed of mining and the amount of tokens rewarded

Final Thoughts

Automating this faucet saved me hours of manual setup and removed my dependency on unreliable public faucets. More importantly, it reinforced something I strongly believe in as an infrastructure engineer: if you need something repeatedly, automate it properly once.

If you’re running blockchain nodes, building rollups, or managing Web3 infrastructure at scale, this pattern applies far beyond faucets.

👉 The full deployment code is available on GitHub: pow-faucet-node-repo

If this helped you, consider starring 🌠 the repo and connecting with me — I regularly share practical infrastructure automation workflows for Web3 and cloud environments.

Resources

For those who want to dive deeper, check out the PoWFaucet GitHub repository for the application itself. The Terraform AWS Provider documentation is invaluable for understanding infrastructure options. Ansible's documentation covers configuration management in depth. And for monitoring, the Prometheus and Grafana Loki docs are your best friends.

Building an AI-Powered IoT Analytics Pipeline on AWS: From Shoe Sensors to Predictive Insights

Ukeme David Eseme — Sat, 23 Aug 2025 13:57:55 +0000

In the world of connected devices, turning raw sensor data into real-time insights requires a robust, scalable, and secure architecture.

In this post, we’ll walk through an end-to-end AWS IoT analytics pipeline—inspired by a use case of a smart shoe with an embedded IoT chip.

This architecture highlights how raw telemetry flows from devices to dashboards, with machine learning inference and notifications along the way.

🚀 Ingestion Layer: Connecting the Shoe to the Cloud

The pipeline begins at the Shoe IoT Chip, which transmits sensor readings (e.g., motion, pressure, gait) via Bluetooth to a Mobile App.

From here, two main ingestion paths exist:

HTTPS → API Gateway for structured data and commands.
MQTT → AWS IoT Core for lightweight, event-driven telemetry.

This dual-ingestion strategy ensures flexibility—supporting both synchronous API calls and asynchronous device messaging.

🔄 Messaging Layer: Handling High-Volume Data

Once inside the AWS ecosystem, the data may fan out into the Messaging Layer:

Amazon Kinesis Data Streams handles real-time streaming ingestion, enabling downstream processing with low latency.
Amazon SQS (Simple Queue Service) provides durable event buffering, ideal for decoupled microservices and event-triggered processing.

This separation ensures the system can scale with bursts of IoT data while maintaining reliability.

🧠 Compute Layer: Processing and Machine Learning

At the core of the pipeline is the Compute Layer, powered by AWS Lambda and Amazon SageMaker:

Lambda (Ingest): Acts as the real-time bridge, consuming from Kinesis or IoT Core and normalizing payloads.
SageMaker Endpoint: Provides inference and prediction, such as anomaly detection, step classification, or injury risk modeling.
Lambda (Process): Handles post-inference workflows, including analytics aggregation and pushing processed results to storage or alerts.

This serverless approach eliminates infrastructure management while ensuring event-driven scaling.

💾 Storage Layer: Persisting Raw and Processed Data

Processed insights and raw logs are stored in a multi-tier storage strategy:

Amazon S3 for raw and pre-processed datasets (useful for model retraining).
Amazon DynamoDB for fast lookups of processed results (e.g., “latest gait analysis score for a user”).
Amazon Aurora (with time-series extensions) for advanced analytical queries and reporting.

This combination balances cost efficiency, query performance, and long-term data retention.

🔐 Security and Monitoring

A production-grade IoT solution requires strong governance:

AWS Cognito enables secure user authentication, while IAM enforces role-based access control across services.
CloudWatch and AWS X-Ray provide observability, enabling teams to trace events, track anomalies, and gather performance metrics.
Optional SNS Integration ensures real-time push notifications (e.g., “Abnormal gait detected, please rest your foot”).

🧩 Putting It All Together

The architecture represents a scalable, secure, and intelligent IoT-to-ML pipeline:

Data Capture: IoT chip → Mobile App → API Gateway / IoT Core.
Data Transport: Kinesis / SQS ensures reliable flow.
Data Intelligence: Lambda + SageMaker perform real-time inference.
Data Persistence: S3, DynamoDB, and Aurora store and organize results.
Security & Monitoring: Cognito, IAM, CloudWatch, and X-Ray ensure governance and visibility.
User Interaction: Notifications and dashboards close the loop with end-users.

This setup doesn’t just collect sensor data—it transforms it into real-time, actionable insights for athletes, healthcare providers, or everyday users.

🌍 Applications Beyond Shoes

While our example revolves around a smart shoe, this architecture generalizes to multiple IoT domains:

Healthcare wearables (vital signs monitoring)
Industrial IoT (machine health prediction)
Smart homes (energy optimization and anomaly detection)
Agriculture IoT (soil/moisture telemetry with predictive yield analytics)

By leveraging AWS serverless and managed AI services, organizations can build scalable IoT solutions without reinventing the wheel.

Final Thoughts

Building intelligent IoT systems isn’t just about connecting devices—it’s about designing an ecosystem where data flows securely, insights are generated in real-time, and end-users receive value instantly.

This AWS-powered pipeline demonstrates how to integrate IoT, serverless compute, machine learning, and storage into a production-ready smart system.

How I Survived the Great Kubernetes Exodus: Migrating EKS Cluster from v1.26 to v1.33 on AWS

Ukeme David Eseme — Sat, 05 Jul 2025 00:25:04 +0000

A comprehensive tale of migrating a production AWS Kubernetes cluster with 6000+ resources, 46 CRDs, 7 SSL certificates, 12 Namespaces and zero downtime

Introduction: The Challenge Ahead

Upgrading a production-grade Kubernetes cluster is never a walk in the park—especially when it spans multiple environments, critical workloads, and tight deadlines.

So when it was time to migrate a clients 3-4 years old Amazon EKS cluster from v1.26 to v1.33, I knew it wouldn’t just be a version bump—it would be a battlefield.

This cluster wasn't just any cluster—it was a complex ecosystem running critical healthcare applications with:

46 Custom Resource Definitions (CRDs) across multiple systems
7 production domains with SSL certificates
Critical data in PostgreSQL databases
Zero downtime tolerance for production services
Complex networking with Istio service mesh
Monitoring stack with Prometheus and Grafana

This is the story of how we successfully migrated this beast using a hybrid approach, the challenges we faced, and the lessons we learned along the way.

Chapter 1: The Reconnaissance Phase

Mapping the Battlefield

Before diving into the migration, we needed to understand exactly what we were dealing with. There was no gitops, no manifest files, what we got was AWS access, Lens and an outdated cluster that needs to be upgraded.

Kubernetes enforces a strict version skew policy, especially when you’re using managed services like Elastic Kubernetes Service (EKS).

The control plane must always be:

one minor version ahead of the kubelets (worker nodes).
All supporting tools—kubeadm, kubelet, kubectl, and add-ons—must also respect this version skew policy.

So what does this mean?

If your control plane is running v1.33, your worker nodes can only be on v1.32 or v1.33. Nothing lower.
And no, you can’t jump straight from v1.26 to v1.33. You must upgrade sequentially: v1.26 → v1.27 → v1.28 → ... → v1.33

Each upgrade step? A potential minefield of broken dependencies, deprecated APIs, and mysterious behavior.

💀 The Aging Cluster

The cluster I inherited was running Kubernetes v1.26—with some workloads and CRDs that hadn’t been touched in about 4 years.
It was ancient. It was fragile. And it was about to get a rude awakening.

🧪 First Attempt: The “By-the-Book” Upgrade

I tried to play nice.
The goal: Upgrade the cluster manually, step-by-step from v1.26 **all the way to **v1.33.

But the moment I moved from v1.26 → v1.27, the floodgates opened:

Pods crashing from all directions,
Incompatible controllers acting out,
Deprecation warnings lighting up the logs like Christmas trees.

Let’s just say—manual upgrades were off the table.

🛠️ Second Attempt: The Manifest Extraction Strategy

Time to pivot.

The new plan?

Spin up a fresh EKS cluster running v1.33, then lift-and-shift resources from the old cluster.

Step 1: Extract All Resources
From the old cluster I ran:

kubectl get all --all-namespaces -o yaml > all-resources.yaml

Then I backed up other critical components:

ConfigMaps
Secrets
PVCs
Ingresses
CRDs
RBAC
ServiceAccounts

kubectl get configmaps,secrets,persistentvolumeclaims,ingresses,customresourcedefinitions,roles,rolebindings,clusterroles,clusterrolebindings,serviceaccounts --all-namespaces -o yaml > extras.yaml

Step 2: Apply to the New Cluster
Switched context:

kubectl config use-context <cluster-arn>

And then:

kubectl apply -f all-resources.yaml extras.yaml

Boom—in one swoop, everything started deploying into the new cluster.

For a moment, I thought:

“Wow… that was easy. Too easy.”

🚨 Reality Check: The Spaghetti Hit the Fan

After 8 hours of hopeful waiting, the nightmare unfolded:

CrashLoopBackOff
ImagePullBackOff
Pending Pods
Service Not Reachable
VolumeMount and PVC errors everywhere

It was YAML spaghetti, tangled and broken.

The old cluster’s legacy configurations simply did not translate cleanly to the modern version.
And now, I had to dig in deep—resource by resource, namespace by namespace, to rebuild sanity, which i didn't have the time and luxury for.

⚙️ Third Attempt: Enter Velero

The next strategy? Use Velero.
Install it in the old cluster, run a full backup, switch contexts, and restore everything into the shiny new v1.33 cluster.

Simple, right?

Not quite.

Velero pods immediately got stuck in Pending.
Why?

Insufficient resources in the old cluster
CNI-related issues that blocked network provisioning

So instead of backup and restore magic, I found myself deep in another rabbit hole.

🧠 Fourth Attempt: Organized Manifest Extraction — The Breakthrough

Out of frustration, I raised the issue during a session in the AWS DevOps Study Group.

That’s when Theo and Jaypee stepped in with game-changing advice:

“Forget giant YAML dumps. Instead, extract manifests systematically, grouped by namespace and resource type. Organize them in folders. Leverage Amazon Q in VS Code to make sense of the structure.”

It was a lightbulb moment💡.

I restructured the entire migration approach based on their idea—breaking down the cluster into modular, categorized directories.
It brought clarity, control, and confidence back to the process.

📦 The CRD Explosion

Once things were neatly organized, the real scale of the system came into focus.

Major CRDs We Had to Handle:

Istio Service Mesh: 12 CRDs managing traffic routing and security
Prometheus/Monitoring: 8 CRDs for metrics and alerting
Cert-Manager: 7 CRDs handling SSL certificate automation
Velero Backup: 8 CRDs for disaster recovery
AWS Controllers: 11 CRDs for cloud integration

🧮 Total: 46 CRDs — each one a potential migration minefield

🔍 Custom Resources Inventory

Beyond the CRDs themselves, the custom resources were no less intimidating:

11+ TLS Certificates across multiple namespaces
6+ ServiceMonitors for Prometheus scraping
Multiple PrometheusRules for alerting
VirtualServices and DestinationRules for Istio routing

The message was clear:

This wasn’t a “one-file kubectl apply” kind of migration.

✅ API Compatibility Victory

With the structure in place, we ran API compatibility checks using Pluto and a custom script generated via Amazon Q in VS Code:

./scripts/api-compatibility-check.sh

Result:

✅ No deprecated or incompatible API versions found.

A small win—but a huge morale boost in a complex migration journey.

📦 Chapter 2: The Data Dilemma

💡 Choosing Our Weapon: Manual EBS Snapshots

When it came to migrating persistent data, we faced a critical decision. Several options were on the table:

Velero backups – our usual go-to, but ruled out due to earlier issues with pod scheduling and CNI errors.
Database dumps – possible, but slow, error-prone, and fragile under pressure.
Manual EBS snapshots – low-level, reliable, and simple

After weighing the risks, we went old-school with manual EBS snapshots.
They offered direct access to data volumes with minimal tooling—and in a high-stakes migration, simplicity is a virtue.

Sometimes, the old ways are still the best ways.

🛠️ Automation to the Rescue

To streamline the snapshot process, I wrote a simple backup script:

./scripts/manual-ebs-backup.sh

It handled the tagging and snapshot creation for each critical volume, ensuring traceability and rollback capability.

🔐 Critical Volumes Backed Up

Here are some of the most important data volumes we preserved:

tools/pgadmin-pgadmin4 → snap-06257a13c49e125b1
sonarqube/data-sonarqube-postgresql-0 → snap-0e590f608a631fcc3 Each snapshot became a lifeline, preserving vital stateful components of our workloads as we prepped the new cluster.

🏗️ Chapter 3: Building the New Kingdom

Once the old cluster was archived and dissected, it was time to construct the new realm—clean, modern, and battle-hardened.

⚙️ The Foundation: CRD Installation Order Matters

One of the most overlooked but mission-critical lessons we learned during this journey:

The order in which you install your CRDs can make or break your cluster.
Install them in the wrong sequence, and you’ll find yourself swimming in cryptic errors, broken controllers, and cascading failures that seem to come from nowhere.

After a lot of trial, and error (especially with istio, gave me a lot of trouble), I landed on a battle-tested CRD deployment sequence:

# 1. Cert-Manager (many other components rely on it for TLS provisioning)
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true

# 2. Monitoring Stack (metrics, alerting, dashboards)
helm install prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace

# 3. AWS Integration (Load balancer controller, IAM roles, etc.)
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system

# 4. Service Mesh (Istio control plane)
istioctl install --set values.defaultRevision=default

🧘 **Pro Tip: **After each installation, wait until the operator and all dependent pods are fully healthy before continuing.
Kubernetes is fast… but rushing this step will cost you hours down the line.

🧬 Data Resurrection: Bringing Back Our State

With the new infrastructure laid out, it was time to resurrect the lifeblood of the platform—its data.

Using our EBS snapshots from earlier, we restored the volumes and re-attached them to their rightful claimants:

bash scripts/restore-ebs-volumes.sh

Restored Volumes:

tools/pgadmin → vol-0166bbae7bd2eb793
sonarqube/postgresql → vol-0262e16e1bd5df028

Held my breath… and then—

✅ PersistentVolumes bound successfully
✅ StatefulSets recovered
✅ Pods restarted with their original data

It was official: our new kingdom had data, structure, and a beating heart.

🎭 Chapter 4: The Application Deployment Dance

The Dependency Choreography

Deploying applications in Kubernetes isn’t just about applying YAML files—it’s a delicate choreography of interdependent resources, where the order of execution can make or break your deployment.

Get the sequence wrong, and you’re looking at a cascade of errors:
missing secrets, broken RBAC, unbound PVCs, and pods stuck in limbo.
We approached it like conducting an orchestra—each instrument with its cue.

🪜 Step-by-Step Deployment Strategy

1. Foundation First: ServiceAccounts, ConfigMaps, and Secrets
These are the building blocks of your cluster environment.
No app should be launched before its supporting config and identity infrastructure are in place.

kubectl apply -f manifests/*/serviceaccounts/
kubectl apply -f manifests/*/configmaps/
kubectl apply -f manifests/*/secrets/

2. RBAC Granting the Right Access
Once identities are in place, we assign the right permissions using Roles and RoleBindings—especially for monitoring and system tools.

kubectl apply -f manifests/monitoring/roles/
kubectl apply -f manifests/monitoring/rolebindings/

⚠️ Lesson: Don’t skip this step or your logging agents and monitoring stack will sit silently—failing without errors.

3. Persistent Storage: Claim Before You Launch
Storage is like the stage on which your stateful applications perform.
We provisioned all PersistentVolumeClaims (PVCs) before deploying workloads to avoid CrashLoopBackOff errors related to missing mounts.

kubectl apply -f manifests/tools/persistentvolumeclaims/
kubectl apply -f manifests/sonarqube/persistentvolumeclaims/

4. Workloads: Let the Apps Take the Stage
With the foundation solid and access configured, it was time to deploy the actual workloads—both stateless and stateful.

kubectl apply -f manifests/tools/deployments/
kubectl apply -f manifests/sonarqube/statefulsets/
# ... and the rest

Status: Applications Deployed and Running ✅

At first glance, everything seemed perfect—pods were green, services were responsive, and dashboards were lighting up.

I exhaled.
But the celebration didn’t last long.

Behind those green pods were networking glitches, DNS surprises, and service discovery issues lurking in the shadows—ready to pounce.

🔐 Chapter 5: The Great SSL Certificate Saga

Just when I thought the migration was complete and everything was running smoothly, the ghost of SSL past returned to haunt.

The Mystery of the Expired Certificates

Just when we thought we were done, we discovered a critical issue:

NAMESPACE     NAME                 CLASS    HOSTS                                         PORTS   AGE
qaclinicaly    bida-fe-clinicaly    <none>   bida-fe-qaclinicaly.example.net         80,443   59s

At first glance, it looked fine. But a quick curl and browser visit revealed a nasty surprise:

“Your connection is not private”
“This site’s security certificate expired 95 days ago”

Another issue, that should cause panic and confusion, but I was calm. We can fix this!
Upon further inspection, every certificate in the cluster was showing:

READY: False

Cert-manager was deployed. The pods were healthy. But nothing was being issued.

🔎 The Missing Link: ClusterIssuer

Digging deeper into the logs, I found the root cause:

The ClusterIssuer for Let’s Encrypt was missing entirely.

Without it, Cert-Manager had no idea how to obtain or renew certificates.
Somehow, it had slipped through the cracks during our migration process.

🛠️ The Quick Fix

Recreated the missing ClusterIssuer using the standard ACME configuration:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: good-devops@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx

Applied it to the cluster

kubectl apply -f cluster-issuer.yaml

Despite the ClusterIssuer being present and healthy, the certificates still wouldn’t renew. The plot thickened...

⚠️ Chapter 6: The AWS Load Balancer Controller Nightmare

Just when I thought the worst was behind me, the AWS Load Balancer Controller decided to stir up fresh chaos.

🧩 The IAM Permission Maze

The first clue came from the controller logs—littered with authorization errors like this:

"error":"operation error EC2: DescribeAvailabilityZones, https response error StatusCode: 403, RequestID: 3ba25abe-7bb2-4b05-bb33-26fde9696931, api error UnauthorizedOperation: You are not authorized to perform this operation"

That 403 told me everything I needed to know:

The controller lacked the necessary IAM permissions to interact with AWS APIs.

What followed was a deep dive into the AWS IAM Policy abyss—where small misconfigurations can lead to hours of head-scratching and trial-and-error debugging.

🔐 The Fix: A Proper IAM Role and Trust Policy

To get the controller working, I created a dedicated IAM role with the required permissions using Amazon Q, and then annotated the Kubernetes service account to assume it.

# Create the IAM role
aws iam create-role \
  --role-name AmazonEKS_AWS_Load_Balancer_Controller \
  --assume-role-policy-document file://aws-lb-controller-trust-policy.json

# Attach the managed policy
aws iam attach-role-policy \
  --role-name AmazonEKS_AWS_Load_Balancer_Controller \
  --policy-arn arn:aws:iam::830714671200:policy/AWSLoadBalancerControllerIAMPolicy

# Annotate the controller's service account in Kubernetes
kubectl annotate serviceaccount aws-load-balancer-controller \
  -n kube-system \
  eks.amazonaws.com/role-arn=arn:aws:iam::830714671200:role/AmazonEKS_AWS_Load_Balancer_Controller \
  --overwrite

With the IAM role in place and attached, I expected smooth sailing—but Kubernetes had other plans.

🌐 The Internal vs Internet-Facing Revelation

Even with the right permissions, certificates still weren’t issuing.
Let’s Encrypt couldn’t validate the ACME HTTP-01 challenge—and I soon discovered why.

Running this command:

aws elbv2 describe-load-balancers \
  --names k8s-ingressn-ingressn-9a8b080581 \
  --region eu-central-1 \
  --query 'LoadBalancers[0].Scheme'

Returned:

json
"internal"

The NGINX ingress LoadBalancer was internal, which made it unreachable from the internet—completely blocking Let’s Encrypt from reaching the verification endpoint.

🛠️ The Fix: Force Internet-Facing Scheme

I updated the annotation on the NGINX controller service:

kubectl annotate svc ingress-nginx-controller \
  -n ingress-nginx \
  service.beta.kubernetes.io/aws-load-balancer-scheme=internet-facing \
  --overwrite

This change recreated the LoadBalancer, this time with internet-facing access.

🌐 Chapter 7: The DNS Migration Challenge

The Automated Solution

Once the internet-facing LoadBalancer was live and SSL certs were flowing, there was still one critical piece left: DNS.

The new LoadBalancer came with a new DNS name, and I had seven production domains that needed to point to it.

Doing this manually in the Route 53 console?
Slow. Risky. Error-prone.

⚙️ The Automated Solution

To avoid mistakes and speed things up, I wrote a script to automate the DNS updates using the AWS CLI.

#!/bin/bash
HOSTED_ZONE_ID="Z037069025V45CB576XJD"
NEW_LB="k8s-ingressn-ingressn-testing12345-9287c75b76ge25zc.elb.eu-central-1.amazonaws.com"
NEW_LB_ZONE_ID="Z3F0SRJ5LGBH90"

update_dns_record() {
    local domain=$1
    aws route53 change-resource-record-sets \
        --hosted-zone-id "$HOSTED_ZONE_ID" \
        --change-batch "{
            \"Changes\": [{
                \"Action\": \"UPSERT\",
                \"ResourceRecordSet\": {
                    \"Name\": \"$domain\",
                    \"Type\": \"A\",
                    \"AliasTarget\": {
                        \"DNSName\": \"$NEW_LB\",
                        \"EvaluateTargetHealth\": true,
                        \"HostedZoneId\": \"$NEW_LB_ZONE_ID\"
                    }
                }
            }]
        }"
}

By calling update_dns_record with each domain, I was able to quickly and safely redirect traffic to the new cluster.

✅ Domains migrated:
Here are the domains I successfully updated:

kafka-dev.example.net
pgadmin-dev.example.net
sonarqube.example.net
bida-fe-qaclinicaly.example.net
bida-gateway-qaclinicaly.example.net
bida-fe-qaprod.example.net
eduaid-admin-qaprod.example.net

Each one now points to the new LoadBalancer, resolving to the right service in the new EKS cluster.

🏁 Chapter 8: The Final Victory

⚔️ The Moment of Truth

After battling through IAM issues, LoadBalancer headaches, DNS rewiring, and countless YAML files, it all came down to one final moment: Would the certificates issue successfully?

I decided to start fresh and purge any leftover Cert-Manager resources to ensure there were no stale or broken states hanging around:

# Clean slate approach
kubectl delete challenges --all --all-namespaces
kubectl delete orders --all --all-namespaces
kubectl delete certificates --all --all-namespaces

Then I waited.....
Refreshed.....
Checked logs.....
Waited some more....

✅ And Then—Success

NAMESPACE    NAME                                            READY   SECRET                                          AGE
qaclinicaly   bida-fe-qaclinicaly.example.net-crt        True    bida-fe-qaclinicaly.example.net-crt        3m
qaclinicaly   bida-gateway-qaclinicaly.example.net-crt   True    bida-gateway-qaclinicaly.example.net-crt   3m
qaprod       bida-fe-qaprod.example.net-crt            True    aida-fe-qaprod.example.net-crt            2m59s
qaprod       eduaid-admin-qaprod.example.net-crt          True    eduaid-admin-qaprod.example.net-crt          2m59s
sonarqube    sonarqube.example.net-crt                 True    sonarqube.example.net-crt                 2m59s
tools        kafka-dev.example.net-tls                 True    kafka-dev.example.net-tls                 2m59s
tools        pgadmin-dev.example.net-tls               True    pgadmin-dev.example.net-tls               2m59s

ALL 7 CERTIFICATES flipped to : READY = TRUE 🎉

📘 Chapter 9: Lessons Learned

🔧 Technical Insights

CRD Installation Order is Critical: Install core dependencies first. Cert-manager before anything else.
IAM Permissions are Tricky: Minimal IAM policies might pass linting, but they’ll fail at runtime. Use comprehensive, purpose-built roles.
LoadBalancer Schemes Matter: The difference between internal and internet-facing can break certificate validation entirely.
DNS Automation Saves Time and Sanity: Manual Route 53 updates are error-prone. Automate with scripts and avoid the guesswork.
EBS Snapshots are Underrated:
Sometimes the simplest tools are the most reliable. EBS snapshots gave me peace of mind and fast recovery.

🧠 Operational Insights
Plan for the Unexpected:

SSL certificate issues took more time than the core migration itself.
Automate Early, Automate Often:

The scripts I wrote saved hours and helped enforce repeatable processes.
Document Everything:

Every command, every fix, every gotcha—write it down. It pays off when something goes wrong (and it will).
Be Patient:

DNS propagation and cert validation can be slow. Don’t panic—just wait.
Always Have a Rollback Plan:

Keeping the old cluster alive gave me confidence to move fast with less fear of failure.

🛠️ Custom Tools That Saved Us

scripts/update-dns-records.sh - Automated DNS cutover
scripts/manual-ebs-backup.sh - Fast and reliable data backup
letsencrypt-clusterissuer.yaml - Enabled SSL cert automation
Comprehensive IAM policies - Smooth AWS integration with the load balancer controller

📊 Chapter 10: The Final Status

✅ Migration Scorecard

Area	Status
Infrastructure	46 CRDs and all operators deployed ✅
Data Migration	EBS volumes restored successfully ✅
DNS Migration	All 7 domains updated ✅
SSL Certificates	All validated and active ✅
LoadBalancer	Internet-facing and functional ✅
Applications	Fully deployed and operational ✅

Performance Metrics

Total Migration Time: ~18 hours (including troubleshooting)
Downtime: 0 minutes (DNS cutover was seamless)
Data Loss: 0 bytes
Certificate Validation Time: 3 minutes (after fixes)
DNS Propagation Time: 2-5 minutes

Conclusion: The Journey's End

What started as a routine Kubernetes version upgrade turned into an epic journey through the depths of AWS IAM policies, LoadBalancer configurations, and SSL certificate validation. We faced challenges we never expected and learned lessons that will serve us well in future migrations.

The key takeaway? Kubernetes migrations are never just about Kubernetes. They're about the entire ecosystem—DNS, SSL certificates, cloud provider integrations, and all the moving parts that make modern applications work.

Our hybrid approach using manual EBS snapshots proved to be the right choice for our use case. While it required more manual work upfront, it gave us confidence in our data integrity and a clear rollback path.

What's Next?

With our new v1.33 cluster running smoothly, we're already planning for the future:

Implementing GitOps for better deployment automation
Enhancing our monitoring and alerting
Preparing for the next major version upgrade (with better automation!)

Final Words

To anyone embarking on a similar journey: expect the unexpected, automate everything you can, and always have a rollback plan. The path may be challenging, but the destination—a modern, secure, and scalable Kubernetes cluster—is worth every debugging session.

Migration Status: ✅ COMPLETE

The cluster is dead, long live the cluster!

Django’s Game of Life Meets AWS ECS – The Ultimate Deployment Hack!

Ukeme David Eseme — Wed, 29 Jan 2025 23:54:23 +0000

Introduction

Prerequisites

Project Setup

Project Structure

AWS Infrastructure

ECR Repository Setup
Export Environmental Variables
IAM Roles & Permissions
ECS Cluster Creation

Manually build and push image to ECR

Build Docker Image
Login to ECR
Tag and push image

Task Definition Configuration

Dynamically update task definition file
Register task definition

Deploy Game Service

Input Service Details
Load balancing

View Deployed Game

Access Load Balancer End-point

Introduction

The Game of Life, created by mathematician John Conway in 1970, is a fascinating example of cellular automation where simple rules create complex patterns.

Our project takes this classic simulation and implements it as a web application using Django.

By deploying it on Amazon Elastic Container Service (ECS), we're making this mathematical marvel accessible through the cloud, demonstrating how modern container orchestration can bring traditional concepts to life in a scalable, reliable way.

Prerequisites

AWS Account
AWS CLI configured
Docker installed
Git repository with the Game of Life code

Project Setup

View project repo Here

git clone https://github.com/UkemeSkywalker/game_of_life

Navigate into the project.

cd game_of_life

Project Structure

game-of-life/
├── Dockerfile
├── buildspec.yml
├── requirements.txt
├── manage.py
├── game_of_life/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   ├── asgi.py
│   └── wsgi.py
├── life/
│   ├── templates/
│   │   └── life/
│   │       ├── landing.html
│   │       ├── select_pattern.html
│   │       └── game.html
│   └── [other app files]
└── ecs/
    └── task-definition.json

AWS Infrastructure

Step 1: ECR Repository Setup

Create ECR Repo

aws ecr create-repository \
  --repository-name game-of-life \
  --image-scanning-configuration scanOnPush=true

Next steps, would be to export the needed environmental variables, run the below command

Step 2: Export Environmental Variables

export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export AWS_REGION=us-east-1
export ECR_REPOSITORY_URI=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/game-of-life

Test Login Command

aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com

expected output should be Login Succeeded

Step 3: IAM Roles & Permissions

IAM > Roles > Create Role > Use Case: Elastic Container Service > 
ECS Task > Select Policy: AmazonECSTaskExecutionRolePolicy

Trusted entity type: AWS service
Service: Elastic Container Service
Use Case: Elastic Container Service Task

Next, add permissions policies

Select: AmazonECSTaskExecutionRolePolicy

Role Name: ecsTaskExecutionRole
Then create the role, it should look similar to the below.

Step 4: ECS Cluster Creation
Run the below command to create the cluster

aws ecs create-cluster \
  --cluster-name game-of-life \
  --capacity-providers FARGATE

Once cluster is created, it should output data similar to the below

Confirm created cluster Login to your AWS console and navigate to ECS, you should see the created cluster.

Manually build and push image to ECR

Next step is to build and push the project docker image to ECS

Step 5: Build Docker Image

The Dockerfile, already exist in the root directory of the project.
Run the below command to build the image.

docker build --platform linux/x86_64 -t game-of-life .

To view the created image, run the docker command:

docker images

Step 6: Login to ECR

aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com

Step 7: Tag and push image

docker tag game-of-life:latest $ECR_REPOSITORY_URI
docker push $ECR_REPOSITORY_URI

Check ECR for the pushed docker image

Task Definition Configuration

In the project directory, you would find the task definition file in ecs/task-definition.json

Navigate to the ecs directory in the project folder

cd ecs

Then run the below command to replace the placeholders with previously exported environmental variable values in the task-definition.json file.

Step 8: Dynamically update task definition file

sed -i '' "s/\${AWS_ACCOUNT_ID}/$AWS_ACCOUNT_ID/g" task-definition.json
sed -i '' "s/\${AWS_REGION}/$AWS_REGION/g" task-definition.json
escaped_uri=$(echo $ECR_REPOSITORY_URI | sed 's/\//\\\//g')
sed -i '' "s/\${ECR_REPOSITORY_URI}/$escaped_uri/g" task-definition.json

Below is the content of the task-definition.json file

{
    "family": "game-of-life",
    "networkMode": "awsvpc",
    "requiresCompatibilities": ["FARGATE"],
    "cpu": "256",
    "memory": "512",
    "executionRoleArn": "arn:aws:iam::${AWS_ACCOUNT_ID}:role/ecsTaskExecutionRole",
    "runtimePlatform": {
        "operatingSystemFamily": "LINUX",
        "cpuArchitecture": "X86_64"
    },
    "containerDefinitions": [
        {
            "name": "game-of-life",
            "image": "${ECR_REPOSITORY_URI}:latest",
            "portMappings": [
                {
                    "containerPort": 8000,
                    "protocol": "tcp"
                }
            ],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "/ecs/game-of-life",
                    "awslogs-region": "${AWS_REGION}",
                    "awslogs-stream-prefix": "ecs"
                }
            }
        }
    ]
}

Step 9: Register task definition

To create the task definition, run the below command while in the ecs directory.

aws ecs register-task-definition --cli-input-json file://task-definition.json

Amazon Elastic Container Service > Task > definitions >game-of-life

Deploy Game Service

Amazon Elastic Container Service > Clusters > game-of-life

In the service tab click on create

Step 10: Input Service Details

Environment section: leave everything as default
Deployment configuration: Only change the Family dropdown to the created task definition game-of-life and the Revision as latest
Service name: game-of-life-svc

Step 11: Load balancing

Scroll down and select the Use load balancing check box.
Load balancer type: select Application Load Balancer
Add the Load balancer name:game-of-life

Leave every other thing as default, and scroll to the end and click on create

View Deployed Game

Step 12: Access Load Balancer End-point

Once the service has been deployed, click on the service name game-of-life-svc

In the service section, select the Configuration and networking tab

Scroll down to Network configuration, copy the loadbalancer endpoint DNS names, and input it in your browser URL.

Congratulations! 🥳🥳🥳

Conclusion

Successfully deploying the Game of Life on AWS ECS showcases how traditional applications can be modernized using container technology and cloud infrastructure.

Through this deployment, we've leveraged AWS's managed container orchestration to ensure our application runs reliably and can scale as needed.

The combination of Django's robust web framework capabilities with AWS's infrastructure provides a stable platform for users to explore Conway's mathematical masterpiece, demonstrating the perfect blend of classical computing concepts with modern cloud architecture.

Automated Nginx and Fluentd Deployment on AWS EC2 using Ansible: A DevOps Guide

Ukeme David Eseme — Sun, 19 Jan 2025 17:27:43 +0000

Table of Content

Introduction
- Brief overview of the problem: Managing web servers and logs at scale
- Why AWS EC2 + Ansible + Nginx + Fluentd is a powerful combination
- Why This Stack Works Together
Prerequisites
Infrastructure Setup
- Create security group
- Security Group configuration for web traffic
- Verify the security group rules
- Create your key pair
- Launch EC2 instance using created security group
- Copy keypair to ssh directory
Ansible Configuration
- Folder Structure
- Prepare workspace environment
- Create the host.yaml file
- Test ansible connection
Set Up Roles
Common Task
Nginx Playbook
Security Playbook
Log Management FluentD Playbook
Deployment and Testing

Introduction

Brief overview of the problem: Managing web servers and logs at scale

In today's cloud environments, managing web servers and logs at scale presents significant challenges.

DevOps teams struggle with manual server configurations, inconsistent deployments, and the overwhelming task of processing massive log data for insights and security analysis.

Our automated solution combines AWS, Ansible, Nginx, and Fluentd to streamline these operations efficiently.

Why AWS EC2 + Ansible + Nginx + Fluentd is a powerful combination

AWS EC2 + Ansible + Nginx + Fluentd creates a robust web infrastructure by combining cloud scalability with automation and efficient logging.

AWS EC2 provides the flexible compute resources, Ansible automates deployment and configuration tasks, Nginx serves as a high-performance web server, and Fluentd handles comprehensive log collection and processing.

Why This Stack Works Together

AWS EC2 (Infrastructure)

Provides on-demand, scalable computing resources
Offers multiple instance types to match workload needs
Integrates seamlessly with other AWS services
Enables global deployment with multiple regions

Ansible (Automation)

Automates server configuration and application deployment
Uses simple YAML syntax for easy maintenance
Requires no agents on managed servers (agentless)
Ensures consistent configurations across all servers

Nginx (Web Server)

Delivers high-performance web serving capabilities
Handles concurrent connections efficiently
Provides reverse proxy and load balancing features
Offers robust security features and SSL/TLS support

Fluentd (Log Management)

Collects and processes logs from multiple sources
Offers flexible routing of log data
Integrates well with various data outputs (S3, CloudWatch)
Provides reliable log buffering and failover*

Prerequisites

Basic Knowledge of AWS & Ansible
AWS CLI installation on local machine
Configure AWS CLI
Install Ansible

Infrastructure Setup

Before we proceed, with setting up infrastructure. It is best we confirm Ansible is installed on our machine and AWS CLI is well configured.

ansible --version

aws sts get-caller-identity

Now we can go ahead with setting up infrastructure.

Create security group

Next step is to create the security group, copy and paste the below code in your terminal

aws ec2 create-security-group \
    --group-name nginx-web-server-sg \
    --description "Security group for Nginx web server and SSH access"

This command would create a security group and output the

GroupId
and SecurityGroupArn;

{
    "GroupId": "sg-0ee2b6c700c11e902",
    "SecurityGroupArn": "arn:aws:ec2:us-east-1:910883278292:security-group/sg-0ee2b6c700c11e902"
}

You can login to your AWS console to confirm the creation of the Security Group

Security Group configuration for web traffic

Lets Add the necessary inbound rules for HTTP (80), HTTPS (443), and SSH (22):

Store the Security Group ID in a variable (from the previous command output; Replace with your actual Security Group ID)

export SG_ID="sg-0ee2b6c700c11e902"

confirm you have exported your SG_ID
echo $SG_ID

Allow HTTP (port 80)

aws ec2 authorize-security-group-ingress \
    --group-id $SG_ID \
    --protocol tcp \
    --port 80 \
    --cidr 0.0.0.0/0

The above command should output similar results (Remember your security ID is different from mine)

{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-081264c3c8fe3e58c",
            "GroupId": "sg-0ee2b6c700c11e902",
            "GroupOwnerId": "910883278292",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 80,
            "ToPort": 80,
            "CidrIpv4": "0.0.0.0/0",
            "SecurityGroupRuleArn": "arn:aws:ec2:us-east-1:910883278292:security-group-rule/sgr-081264c3c8fe3e58c"
        }
    ]
}

Allow HTTPS (port 443)

aws ec2 authorize-security-group-ingress \
    --group-id $SG_ID \
    --protocol tcp \
    --port 443 \
    --cidr 0.0.0.0/0

Command outcrt should be similar

{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-06591dee28547a3eb",
            "GroupId": "sg-0ee2b6c700c11e902",
            "GroupOwnerId": "910883278292",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 443,
            "ToPort": 443,
            "CidrIpv4": "0.0.0.0/0",
            "SecurityGroupRuleArn": "arn:aws:ec2:us-east-1:910883278292:security-group-rule/sgr-06591dee28547a3eb"
        }
    ]
}

Allow SSH (port 22) - Best practice is to limit this to your IP

aws ec2 authorize-security-group-ingress \
    --group-id $SG_ID \
    --protocol tcp \
    --port 22 \
    --cidr 0.0.0.0/0

Note: for a more secure environment, its advisable to only allow your specific IPs instead of using 0.0.0.0/0 as the cidr block

Command output should be similar

{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-06c1f6c3205247aea",
            "GroupId": "sg-0ee2b6c700c11e902",
            "GroupOwnerId": "910883278292",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 22,
            "ToPort": 22,
            "CidrIpv4": "105.113.64.38/32",
            "SecurityGroupRuleArn": "arn:aws:ec2:us-east-1:910883278292:security-group-rule/sgr-06c1f6c3205247aea"
        }
    ]
}

Verify the security group rules

aws ec2 describe-security-groups --group-ids $SG_ID

Due to the how lengthy the command outputs are, I wont be posting it here.

Create your key pair

This command would create your key pair, and save it on your local machine

aws ec2 create-key-pair --key-name nginx-server-key --query 'KeyMaterial' --output text > nginx-server-key.pem

Confirm the created key
cat nginx-server-key.pem

Launch EC2 instance using created security group

aws ec2 run-instances \
    --image-id ami-0e1bed4f06a3b463d \
    --instance-type t2.micro \
    --key-name nginx-server-key \
    --security-group-ids $SG_ID \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=nginx-fluentd-server}]'

*Image AMI: * ami-0e1bed4f06a3b463d : installs Ubuntu 22
*Instance Type: * vCPUs: 1, Memory: 1 GiB
Key Pair: Key=Name,Value=nginx-fluentd-server : identifies as the name of the server

Copy keypair to ssh directory

cp nginx-server-key.pem ~/.ssh

SSH into EC2 instance

You need this step to confirm you can actually log into the machine.

Change key pair permissions chmod 400 ~/.ssh/nginx-server-key.pem
Connect using ssh ssh -i ~/.ssh/nginx-server-key.pem ubuntu@<instance-ip-address>

Once you can log into the instance via ssh, we can move to the next section.

Ansible Configuration

Folder Structure

├── hosts.yaml
├── site.yaml
└── roles/
    ├── common/
    ├── └── defaults/
    │   |   └── main.yaml
    │   └── tasks/
    │   |    └── main.yaml
    │   └── handlers/
    │       └── main.yaml
    ├── nginx/
    │   └── tasks/
    │   |   └── main.yaml
    │   └── templates/
    │       └── nginx-logrotate.j2
    ├── security/
    |   └── defaults/
    │   |   └── main.yaml
    │   └── tasks/
    │       └── main.yaml
    └── fluentd/
        └── defaults/
        |   └── main.yaml
        └── tasks/
        |   └── main.yml
        └── files/
        |   └── denylist.txt
        └── templates/
            └── td-agent.conf.j2

Prepare workspace environment

Open your preferred terminal or use vscode terminal, create a new folder called AWS-Nginx-Ansible-FluentD and navigate into that folder

mkdir AWS-Nginx-Ansible-FluentD
cd AWS-Nginx-Ansible-FluentD

Create the host.yaml file

The hosts.yaml file is an Ansible inventory file that defines the target servers (hosts) Ansible will manage.
This file tells Ansible where and how to connect to the managed nodes for executing tasks or running playbooks.

touch host.yaml

add the below code

---
nginx_server:
  hosts:
    nginx-server-1:
      ansible_host: <your-instance-public-ip>
      ansible_user: ubuntu
      ansible_ssh_private_key_file: "{{ lookup('env', 'HOME') }}/.ssh/nginx-server-key.pem"
      ansible_ssh_common_args: "-o StrictHostKeyChecking=no"

Note: change the value of ansible_host to your created instance IP

Test ansible connection

Run the below command in the project folder terminal

ansible nginx-server -i hosts.yaml -m ping

If connection is successfull, you should see the below result

nginx-server-1 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3.10"
    },
    "changed": false,
    "ping": "pong"
}

Set Up Roles

In Ansible, roles are a way to organize and reuse configurations by breaking them into modular, reusable components.
Each role contains specific;

Tasks,
Variables,
Templates,
Files,
and handlers required to perform a particular function, such as installing a web server or configuring a database.

By using roles, you can structure your playbooks more efficiently, making them cleaner, more scalable, and easier to maintain

Create site.yaml: This file is the root directory for all roles.

touch site.yaml

Add the below snippet to the playbook

---
- name: Configure Web Server with Nginx and Fluentd
  hosts: nginx_server
  become: yes

  roles:
    - common
    - nginx
    - security
    - fluentd

Common Task

Create Required directories and files

In the root directory, create a folder called common

mkdir common

Create defaults, handler & task folders

mkdir common/defaults common/handler common/tasks

Create the main files in defaults, handler & tasks

touch common/defaults/main.yaml common/handler/main.yaml common/tasks/main.yaml

Common Playbook

variable definition: Add the below code to common/defaults/main.yaml The content of the file defines settings for your Ansible tasks. In simple terms, it’s configuring paths and settings for Nginx and Fluentd, as well as setting a rule for how long logs are retained.

---
nginx_html_root: /var/www/html
fluentd_config_dir: /etc/td-agent
log_retention_days: 5

Handler Task List: Add the below code to common/handler/main.yaml Restart Nginx: Restarts the nginx service to apply any changes or ensure it is running. Restart Fluentd: Restarts the fluentd service to reload its configuration or ensure it is operational.

---
- name: restart nginx
  service:
    name: nginx
    state: restarted

- name: restart fluentd
  service:
    name: fluentd
    state: restarted

Main Common Task: Add the below code to common/tasks/main.yaml The content of the file updates the package cache on systems using apt (like Debian or Ubuntu).

---
- name: Update apt cache
  apt:
    update_cache: yes
    cache_valid_time: 3600

update_cache: yes: Refreshes the list of available packages from repositories.
cache_valid_time: 3600: Ensures the cache is valid for 3600 seconds (1 hour), skipping updates if the cache is still recent.

Nginx Playbook

In the root directory, create a folder called nginx

mkdir nginx

Create nginx task folder

mkdir nginx/tasks nginx/templates

Create the nginx tasks & files

touch nginx/tasks/main.yaml nginx/templates/nginx-logrotate.j2

Nginx Task playbook: Add the below code to nginx/tasks/main.yaml

---
- name: Install Nginx
  apt:
    name: nginx
    state: present

- name: Create simple HTML page
  copy:
    content: |
      <!DOCTYPE html>
      <html>
      <head><title>Hello, World!</title></head>
      <body><h1>Hello, World!</h1></body>
      </html>
    dest: "{{ nginx_html_root }}/index.html"
    mode: '0644'
  notify: restart nginx

- name: Enable and start Nginx
  service:
    name: nginx
    state: started
    enabled: yes

- name: Configure logrotate for Nginx
  template:
    src: nginx-logrotate.j2
    dest: /etc/logrotate.d/nginx
    mode: '0644'

This is an Ansible Playbook Task List that sets up and configures Nginx with the following steps:

Install Nginx: Ensures the Nginx package is installed on the target machine.
Create a Simple HTML Page: Copies a basic “Hello, World!” HTML file to the Nginx web root (nginx_html_root), with proper permissions (0644). It also triggers a restart nginx handler when changes occur.
Enable and Start Nginx: Ensures the Nginx service is running (started) and configured to start automatically on system boot (enabled).
Configure Logrotate for Nginx: Deploys a log rotation configuration file for Nginx using a Jinja2 template (nginx-logrotate.j2), setting appropriate permissions (0644).

Nginx Jinja Template: Add the below code to nginx/templates/nginx-logrotate.j2

/var/log/nginx/*.log {
    daily
    rotate {{ log_retention_days }}
    missingok
    notifempty
    compress
    delaycompress
    postrotate
        invoke-rc.d nginx rotate >/dev/null 2>&1
    endscript
}

This template configures log rotation for Nginx logs, ensuring logs don’t grow indefinitely.

It rotates logs daily, keeps them for a specified number of days, compresses old logs, and notifies Nginx after rotation to ensure smooth operation.
The log_retention_days variable makes the retention period dynamic.

Security Playbook

In the root directory, create a folder called security

mkdir security

Create security defaults & task folder

mkdir security/defaults security/tasks

Create the security defaults & tasks files

touch security/defaults/main.yaml security/tasks/main.yaml

Security variable definition: Add the below code to security/defaults/main.yaml

This file sets firewall rules to allow essential traffic and disables unnecessary services to enhance security.

---
ufw_rules:
  - { rule: 'allow', port: '80', proto: 'tcp' }
  - { rule: 'allow', port: '443', proto: 'tcp' }
  - { rule: 'allow', port: '22', proto: 'tcp' }

disabled_services:
  - rpcbind
  - cups
  - avahi-daemon

Security Task playbook: Add the below code to security/tasks/main.yaml

---
- name: Install UFW
  apt:
    name: ufw
    state: present

- name: Configure UFW rules
  ufw:
    rule: "{{ item.rule }}"
    port: "{{ item.port }}"
    proto: "{{ item.proto }}"
    state: enabled
  with_items: "{{ ufw_rules }}"

- name: Disable unnecessary services
  service:
    name: "{{ item }}"
    state: stopped
    enabled: no
  with_items: "{{ disabled_services }}"
  ignore_errors: yes

These tasks install and configure a firewall for secure access while stopping and disabling unneeded services to improve security and performance.

FluentD Playbook

Fluentd is used to centralize and manage log data efficiently, enabling better monitoring, troubleshooting, and analytics for applications and systems.

In the root directory, create a folder called fluentd

mkdir fluentd

Create fluentd defaults, files, task & templates folder

mkdir fluentd/defaults fluentd/files fluentd/tasks  fluentd/templates

Create the fluentd defaults, files, task & templates files

touch fluentd/defaults/main.yaml fluentd/files/denylist.txt fluentd/tasks/main.yaml fluentd/templates/td-agent.conf.j2

fluentd variable definition: Add the below code to fluentd/defaults/main.yaml

fluentd_config_dir: "/etc/fluentd"

This variable tells Ansible (or any script using it) where to find or manage Fluentd’s configuration files, which typically include settings for log inputs, filters, and outputs.

Fluentd files: To deny a list of IP addresses, you can add them to the fluentd/files/denylist.txtfile.

192.168.1.100
10.0.0.50

Fluentd Task Playbook: Add the below code to fluentd/tasks/main.yaml

---
- name: Download and run Fluentd installation script
  shell: |
    curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-jammy-fluent-package5-lts.sh | sh
  args:
    executable: /bin/bash

- name: Verify Fluentd installation
  command: fluentd --version
  register: fluentd_version

- debug:
    var: fluentd_version.stdout

- name: Create Fluentd config directory
  file:
    path: "{{ fluentd_config_dir }}"
    state: directory
    mode: '0755'
  become: yes

- name: Create Fluentd config
  template:
    src: td-agent.conf.j2
    dest: "{{ fluentd_config_dir }}/td-agent.conf"
    mode: '0644'
  notify: restart fluentd

- name: Enable and start Fluentd
  service:
    name: fluentd
    state: started
    enabled: yes

This playbook automates installing Fluentd, setting up its configuration, and ensuring the service is up and running.

Fluentd Configuration Jinja Templates: Add the below code to fluentd/templates/td-agent.conf.j2

# Input for Nginx access logs
<source>
  @type tail
  path /var/log/nginx/access.log
  pos_file /var/log/td-agent/nginx.access.pos
  tag nginx.access
  <parse>
    @type nginx
  </parse>
</source>

# Input for Nginx error logs
<source>
  @type tail
  path /var/log/nginx/error.log
  pos_file /var/log/td-agent/nginx.error.pos
  tag nginx.error
  <parse>
    @type regexp
    expression /^(?<time>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?<log_level>\w+)\] (?<pid>\d+).*?: (?<message>.*)$/
    time_format %Y/%m/%d %H:%M:%S
  </parse>
</source>

# Filter to check IPs against denylist
<filter nginx.access>
  @type grep
  <regexp>
    key remote_addr
    pattern /^(?!#{File.readlines("#{ENV['FLUENT_CONFIG_DIR'] || '/etc/td-agent'}/denied_ips/denylist.txt").map(&:strip).join('|')}).*$/
  </regexp>
</filter>

# Route normal logs (non-denied IPs)
<match nginx.access>
  @type file
  path /var/log/td-agent/nginx_access
  append true
  <buffer>
    timekey 1d
    timekey_use_utc true
    timekey_wait 10m
  </buffer>
  <format>
    @type json
  </format>
</match>

# Route denied IP logs to audit file
<match nginx.access>
  @type copy
  <store>
    @type file
    path /var/log/td-agent/audit/denylist_audit
    append true
    <buffer>
      timekey 1d
      timekey_use_utc true
      timekey_wait 10m
    </buffer>
    <format>
      @type json
      include_time_key true
      time_key timestamp
    </format>
  </store>
</match>

# Handle error logs
<match nginx.error>
  @type file
  path /var/log/td-agent/nginx_error
  append true
  <buffer>
    timekey 1d
    timekey_use_utc true
    timekey_wait 10m
  </buffer>
  <format>
    @type json
  </format>
</match>

This configuration collects and processes Nginx access and error logs. It checks access logs against a denylist, routes normal logs and denied IP logs to separate files, and stores error logs in a structured JSON format for easy analysis.

This setup is ideal for monitoring, auditing, and managing log data efficiently.

If you’ve made it this far, well done! Congratulations!

Deployment and Testing

Running the Playbook

Check syntax

ansible-playbook -i hosts.yaml site.yaml --syntax-check

Run playbook

ansible-playbook -i hosts.yaml site.yaml

Ansible would run through all the task listed in the site.yaml file, and run each of the playbooks sequentially.

If you face an error with TASK [security : Disable unnecessary services] theres no need to panic, it just means, you don't have the unnecessary services to disable and you playbook, would continue to the other task

Verification Steps

Check Nginx status

ansible webservers -i hosts.yaml -m shell -a "systemctl status nginx"

or input the instance public ip on a browser, to get the Hello, World! page

Check Fluentd status

ansible webservers -i hosts.yaml -m shell -a "systemctl status td-agent"

Test log processing
First make a curl request to the server

ansible nginx_server -i hosts.yaml -m shell -a "curl http://localhost"

Then check the tail of the fluentd logs

ansible nginx_server -i hosts.yaml -m shell -a "tail  /var/log/fluent/fluentd.log"

Conclusion

This solution provides a robust, automated approach to deploying and managing web servers on AWS.
By combining Ansible's automation capabilities with AWS services, we create a scalable and maintainable infrastructure that follows DevOps best practices.

🎉 Congratulations! If you've made it this far, you've just learned how to:

Launch and configure an EC2 instance like a pro 🚀
Set up a secure web server that would make security experts nod in approval 🔒
Create a logging system that catches every digital whisper 🔍
Automate everything so smoothly that your future self will thank you ⚡

Fun fact: The automation you just built will save you approximately 127 cups of coffee worth of manual configuration time per year! ☕

🚀 Unleashing the Power of Cloud Magic: Transforming a Lone AWS EC2 Instance into a K8s Powerhouse! 🌐🔥

Ukeme David Eseme — Wed, 07 Feb 2024 09:27:55 +0000

Table of Content:

Prequisite
Introduction
SSH into EC2
Install Docker
Install Kubectl
Install KIND
Setup Kubernetes Cluster
Setup Visualizer (KubeOps View)

Perquisite

EC2 instance running Amazon Linux 2023 AMI - How to Video, Doc
Available private key pair for the instance

Introduction

Welcome to the realm of cloud enchantment! In this captivating journey, we will delve into the art of transforming a solitary AWS EC2 instance into a formidable Kubernetes (K8s) powerhouse. Brace yourself as we unravel the secrets of cloud magic, unlocking the potential of your EC2 instance to orchestrate a dynamic Kubernetes cluster. With a touch of innovation and a dash of determination, you'll soon wield the power of the cloud like never before.

SSH into EC2

SSH stands for "Secure Shell." It is a cryptographic network protocol used for securely connecting to a remote server or device over an unsecured network.

To connect via SSH to the virtual Machine (EC2), you would need a secure shell client like Putty or MobaXterm or just your plain terminal.

I would be using Tabby Terminal.

Open the terminal and navigate to the directory were you downloaded the *EC2 Key Pair *.

In my case its

cd ~/Downloads/

Run this command, if necessary, to ensure your key is not publicly viewable.

chmod 400 "your-Key-pair-file.pem"

Connect to your instance using its Public DNS For example

ssh -i "your-Key-pair-file.pem" ec2-user@ec2-your-ip.compute-1.amazonaws.com

Once you are connected to the instance, you should see a welcome screen like the below.

Install Docker

Amazon Linux 2023 uses dnf as its package manager.

1. Update AL2023 Packages

Since its a new linux VM, run the below command to perform an update.

sudo dnf update

This command is used to update the installed packages and package cache on a Fedora system.

2. Installing Docker on Amazon Linux 2023

sudo dnf install docker

The above installs the Docker Engine, the Docker command-line interface, and the containerd runtime.

3. Start and Enable Docker Service

After installation docker services, don't start up by default, we have to manually start the process.

sudo systemctl start docker

Also, we want to set docker to automatically start with system boot

sudo systemctl enable docker

To be sure docker is currently running as expected, we need to check its status.

sudo systemctl status docker

You should have a similar result, like the image below.

4. Enable Docker to run without requiring sudo

Once the installation is finished, it's cumbersome to use sudo every time you want to execute Docker commands. To alleviate this inconvenience, we need to include our current user in the Docker group. Utilize the provided command to accomplish this."

sudo usermod -aG docker $USER

apply the changes to the docker group

newgrp docker

To verify and check docker version

docker version

You should have data similar to the below image

Install Kubectl

kubectl is a command-line interface (CLI) tool used to interact with Kubernetes clusters. It allows users to perform various operations on Kubernetes resources, such as deploying applications, managing pods, services, and deployments, inspecting cluster resources, and debugging cluster issues.

Note: You should ensure that the version of kubectl you use is within one minor version of your Kubernetes cluster. For instance, a client with version v1.29 can communicate effectively with control planes of versions v1.28, v1.29, and v1.30. Utilizing the most recent compatible kubectl version is essential to prevent unexpected complications.

1. Install the kubectl binary on Linux using curl:

Download the latest release with the command:

     curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

2. Validate the binary (optional)

Download the kubectl checksum file:

   curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"

Validate the kubectl binary against the checksum file:

echo "$(cat kubectl.sha256)  kubectl" | sha256sum --check

If valid, the output is:
kubectl: OK

3. Install kubectl

sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Test to ensure the version you installed is up-to-date:

kubectl version --client

Or use this for detailed view of version:

kubectl version --client --output=yaml

Install KIND

We would be using KIND to create our kubernetes cluster.
What is KIND ?

In Kubernetes, "Kind" refers to Kubernetes in Docker. It is a tool for running local Kubernetes clusters using Docker container "nodes".
Its is a lightweight and easy-to-use Kubernetes environment for testing and development purposes.

For AMD64 / x86_64

[ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.21.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

Confirm KIND is installed.

 kind --version

You should see the current version of KIND installed.

Setup Kubernetes Cluster

in your terminal create a new file with the below command:

nano three-node-cluster.yml

Paste this code in the editor

# three node (two workers) cluster config
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 32000
    hostPort: 32000
    listenAddress: "0.0.0.0"
    protocol: tcp
  - containerPort: 32100
    hostPort: 32100
    listenAddress: "0.0.0.0"
    protocol: tcp
  - containerPort: 30000
    hostPort: 30000
    listenAddress: "0.0.0.0"
    protocol: tcp
  - containerPort: 30100
    hostPort: 30100
    listenAddress: "0.0.0.0"
    protocol: tcp
  - containerPort: 30200
    hostPort: 30200
    listenAddress: "0.0.0.0"
    protocol: tcp
  - containerPort: 30300
    hostPort: 30300
    listenAddress: "0.0.0.0"
    protocol: tcp
  - containerPort: 30400
    hostPort: 30400
    listenAddress: "0.0.0.0"
    protocol: tcp
- role: worker
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    listenAddress: "0.0.0.0"
    protocol: tcp
  - containerPort: 8000
    hostPort: 8000
    listenAddress: "0.0.0.0"
    protocol: tcp
  - containerPort: 8080
    hostPort: 8001
    listenAddress: "0.0.0.0"
    protocol: tcp

- role: worker

Use Ctrl+x To save the changes and exit editing.

Now let's break down the configuration:

kind: Specifies the kind of resource being defined, which is a Cluster in this case.
apiVersion: Specifies the version of the Kubernetes API being used.
nodes: Specifies the configuration for the nodes in the cluster.

The first node:

Defined is a control-plane node (role: control-plane). This node has extraPortMappings configured, which maps container ports to host ports. This is useful for accessing services running inside Kubernetes from outside the cluster. The listed container ports are mapped to the same host ports (32000, 32100, 30000, 30100, 30200, 30300, 30400) and listen on all available network interfaces (0.0.0.0) using the TCP protocol.

The second node (role: worker)

Also has extraPortMappings configured. It maps container ports 80, 8000, and 8080 to host ports 80, 8000, and 8001 respectively.

The last node

is simply specified with role: worker, but it doesn't have any extraPortMappings configured.

Create the KIND Cluster

kind create cluster --config three-node-cluster.yml

Once its done, to get the cluster info

kubectl cluster-info --context kind-kind

Get a list of the running nodes

 kubectl get nodes

View all running pods across all namespaces

 kubectl get pods -A

Setup Visualizer

KubeOps View is a read-only system dashboard for multiple Kubernetes clusters, providing a common operational picture for understanding cluster setups in a visual way. It allows users to render nodes, indicate their overall status, show node capacity, and more.

1. Install Git

sudo dnf install git

2. Clone Git Repo

git clone https://github.com/UkemeSkywalker/kube-ops-view

3. Apply kubeOps deployment

Navigate to the clone repository

cd kube-ops-view/

Apply deployment

kubectl apply -f deploy/

4. Check Deployment

kubectl get pods

5. Update EC2 security group inbound rules

In your Ec2 instance details page, scroll down and navigate to the security section.
Click on the default security group. It should take you to the dashboard.

Click on edit inbound rules, and add a new rule

Type:	Custom TCP
Port:	32000
Source:	select My IP

Click on save rule.

6. Finally, Access the visualizer on your browser

http://your-ec2-pubic-ip:32000/#scale=2.0

Conclusion

As our adventure draws to a close, you now possess the knowledge and prowess to harness the full potential of your AWS EC2 instance. From the exhilarating setup of Docker and KIND to the orchestration of your very own Kubernetes cluster, you've embarked on a journey filled with discovery and empowerment.

With KubeOps View offering a visual glimpse into your cloud domain, the possibilities are endless. Embrace the magic of the cloud, and may your Kubernetes endeavors continue to flourish in the ever-expanding landscape of technology.

Until next time, may your clouds be clear and your clusters be mighty! ✨🌐🚀

Building a Node.js Express Application with AWS CodeBuild and Amazon ECR🚀

Ukeme David Eseme — Fri, 19 Jan 2024 07:08:09 +0000

Table of Content:

Introduction
What is AWS CodeBuild
What is Amazon ECR
Prerequisites
Step 1: Buildspec.yml
Step 2: IAM Roles and Permissions
Step 3: Create a CodeBuild Project
Step 4: Run CodeBuild project

Introduction

In this article, we'll explore how to use AWS CodeBuild to build a Node.js Express application, create a Docker image, and push it to Amazon ECR.

What is AWS CodeBuild:

AWS CodeBuild is a fully managed continuous integration service by Amazon Web Services that automates the build and testing phases of software development.

Developers define build projects using buildspec.yml files, specifying the steps for building Docker images and running tests.

With support for various programming languages, build tools, and seamless integrations with AWS services and version control systems, CodeBuild facilitates the efficient creation, testing, and deployment of applications within a scalable and customizable environment.

What is Amazon ECR:

Amazon Elastic Container Registry (Amazon ECR) is a fully managed Docker container registry service provided by Amazon Web Services (AWS). It allows developers to store, manage, and deploy Docker container images.

ECR integrates seamlessly with other AWS services, making it easy to build, store, and deploy containerized applications using tools like Amazon ECS (Elastic Container Service) and AWS Fargate. With features such as image scanning, encryption, and fine-grained access control, Amazon ECR provides a secure and scalable solution for container image management within the AWS ecosystem.

Without delay, let's delve into the practical aspects of the topic at hand.

Prerequisites

Before you begin, make sure you have the following prerequisites:

An AWS account
A Node.js Express application hosted on a version control system like CodeCommit or GitHub
Already setup ECR repository

Step 1: Buildspec.yml

Create a buildspec.yml file in the root of your Node.js Express project. This file defines the build steps for CodeBuild.

version: 0.2

phases:
  install: 
    runtime-versions:
      nodejs: latest

  pre_build:
    commands:
      - echo Logging in to Amazon ECR...    
      - aws --version
      - aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
      - COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)


  build:
    commands:
      - echo Build started on `date`
      - echo Building the Docker image...
      - docker build -t $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG .
      - docker tag $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest

  post_build:
    commands:
      - echo Build completed on `date` 
      - echo Pushing the Docker image...
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG

Step 2: IAM Roles and Permissions

To enable CodeBuild to push Docker images to Amazon ECR, it is essential to establish IAM roles and permissions.

Begin by crafting an IAM role specifically for CodeBuild, equipped with the necessary authorizations

ECR permissions to facilitate the pushing of Docker images to your repository. Opt for the aws-managed AmazonEC2ContainerRegistryPowerUser role for a streamlined approach.
CloudWatch Logs permissions to enable the writing of build logs to CloudWatch Logs. Activating CloudWatch during the setup will automatically add the requisite policies.

Step 3: Create a CodeBuild Project

a) To create a CodeBuild project, you need to follow these steps.

Open the AWS Management Console.
Navigate to the CodeBuild service.
Click on "Create build project.

b) Give your project a name and description, and select the source code location as the newly created AWS CodeCommit repository, and the associated branch.

c) For the Environment section, use the below settings:

Provisioning mode: On-demand
Environment image: Managed image
Compute: EC2
Operating system: Amazon Linux
Runtime(s): Standard
Image: use the latest standard image
Image version: Always use the latest image for this runtime version

d) For the Service role, select the Existing IAM Role, you created earlier with AmazonEC2ContainerRegistryPowerUser

e) Additional configuration:
Scroll down to Environment Variables and update your environmental variables

f) For Build specifications, select Use a buildspec file and for Logs, add CloudWatch Group name, Stream name and click Create build project

Step 4: Run CodeBuild project

Select the codebuild project and click on Start build
Wait for build to Succeed!
Then check ECR for build Image.

And that's it! You have successfully used AWS CodeBuild to build a Node.js Express application, create a Docker image, and push it to Amazon ECR.

Decoding EC2 Placement Groups: Unveiling the Pros and Cons of Cluster, Spread, Partition Strategies and its implementation

Ukeme David Eseme — Wed, 15 Nov 2023 16:53:43 +0000

Amazon EC2 Placement Groups are a feature of Amazon Elastic Compute Cloud (EC2) that enable you to influence the placement of instances within the AWS infrastructure.

on a specific Availability Zone
A specific rack
or Zone based placement etc

The placement group you choose for your instances can significantly impact the performance, availability, and fault tolerance of your applications.

There are three (3) types of EC2 Placement Groups:

Cluster Placement Group.
Spread Placement Group.
Partition Placement Group

Cluster Placement Group:

Instances are placed on the same hardware, meaning instances in a cluster placement group are placed in close proximity to each other, providing low-latency communication between them.

Use Cases: Ideal for applications that require low-latency, high-throughput communication between instances. Commonly used for high-performance computing (HPC) workloads.

Pros:

1. Low Latency: Instances in a cluster placement group share the same hardware, leading to low-latency communication.

2. High Throughput: Ideal for high-performance computing (HPC) workloads that require high throughput and minimal communication overhead.

3. Performance Optimization: Well-suited for applications that benefit from instances being close to each other, minimizing network latency.

Cons:

1. Limited Fault Tolerance: All instances are on the same hardware, making them susceptible to correlated failures.

2. Instance Type Limitations: Not all instance types are supported in cluster placement groups.

3. No Cross-Region Support: Cluster placement groups cannot span multiple AWS regions.

Spread Placement Group:

Instances in a spread placement group are placed on distinct underlying hardware to reduce the risk of simultaneous failures.

Use Cases: Useful for applications that require a small number of critical instances to be kept separate for fault tolerance.

1. Fault Tolerance: Offers high fault tolerance, as instances are placed on distinct underlying hardware.
2. Isolation: Provides high isolation between instances, reducing the risk of correlated failures.
3. Availability Zone Support: Can span multiple Availability Zones, enhancing availability.

Cons:

1. Communication Overhead: Instances in a spread placement group may experience higher communication overhead due to being on separate hardware.

2. Limited Instance Type Support: Not all instance types are supported in spread placement groups.

3. Limited Number of Instances: There is a limit on the number of instances per Availability Zone in a spread placement group.
(Maximum of 7 instances per group per Availability Zone.)

Partition Placement Group:

Instances are spread across logical partitions, each with its own set of racks.

Use Cases: Suitable for large distributed and replicated workloads, such as big data and database systems.
Pros:

1. Fault Tolerance: Offers better fault tolerance than cluster placement groups, as instances are spread across different logical partitions.

2. Scalability: Can handle a larger number of instances compared to cluster placement groups.

3. Isolation: Provides moderate isolation between instances due to partitioning.

Cons:

1. Communication Overhead: Instances in different partitions may have higher communication overhead compared to those in a cluster placement group.

2. Instance Type Limitations: Instances in a partition placement group must be of the same type.

3. Partition Limitations: There is a limit on the number of partitions per Availability Zone.

How to create placement group

Sign in to your AWS console and navigate to the EC2 Dashboard, on the left navigation section, select Placement Groups, under Network & Security.
On the top right hand corner, select the "create placement group" button
Input the Name, select your placement group strategy, then click the "create group button", on the low right corner. To implement the created placement group in an instance.
Upon creation of an Instance, scroll down to the advanced details section, scroll down to placement group and select the created placement group.

Conclusion:

EC2 Placement Groups provide a way to optimize the placement of your instances based on your specific workload requirements, whether it's maximizing performance, achieving fault tolerance, or enhancing isolation between instances.

Turbocharging EC2: A SysOps Guide to ENA and EFA

Ukeme David Eseme — Wed, 15 Nov 2023 13:59:40 +0000

Introduction

Brief overview of Amazon EC2's role in cloud computing.

Amazon Elastic Compute Cloud (EC2) is a foundational service offered by Amazon Web Services (AWS).

It provides scalable compute capacity on the cloud, allowing users to run virtual servers known as instances. These instances serve as the building blocks for a wide range of applications, from simple web servers to complex, distributed systems.

Understanding ENA and EFA Basics

What is ENA?

ENA is an acronym for Elastic Network Adapter, provides high-performance networking capabilities, including support for up to 100 Gbps of network bandwidth. This is crucial for applications and workloads that require low-latency and high-throughput network connectivity.

Elastic Network Adapter (ENA) Features :

1. Reduced Network Latency:
ENA is designed to minimize network latency, making it suitable for applications that require fast and responsive network communication.

2. Performance Scaling: ENA is designed to scale with the performance characteristics of the underlying EC2 instance. It can adapt to the varying requirements of different workloads.

3. Improved Security Groups Performance: ENA is designed to enhance the performance of security groups, allowing for efficient filtering of inbound and outbound traffic at the network interface level.

What is EFA?

Elastic Fabric Adapter (EFA) is a specialized networking interface designed for high-performance computing (HPC) workloads within the Amazon EC2 environment.

Specifically designed to meet the low-latency and high-throughput demands of HPC applications, EFA plays a crucial role in enabling seamless communication between instances in a cluster.

Elastic Fabric Adapter (EFA) Use Case

1. MPI (Message Passing Interface) Workloads: EFA significantly enhances MPI-based applications, which heavily rely on inter-node communication. It ensures efficient message passing between nodes, crucial for parallel processing in scientific simulations and computational research.

2. Large-Scale Simulations: EFA's advantages shine in HPC simulations, where multiple instances collaborate on intricate computations. The low-latency communication facilitated by EFA accelerates the exchange of data, improving overall simulation performance.

3. Data-Parallel Applications: Workloads that involve parallel data processing, such as distributed data analytics and machine learning, benefit from EFA's ability to facilitate quick and reliable communication between nodes.

4. Precision Medicine and Genomics: In genomics research, where analyzing massive datasets across multiple nodes is common, EFA aids in speeding up inter-node communication, contributing to faster genomic sequencing and analysis.

Financial Modeling with MPI: EFA is instrumental in financial applications that leverage MPI standards for parallel computation. It reduces communication bottlenecks, enabling quicker analysis and decision-making in complex financial modeling scenarios

The similarities and differences between Elastic Network Adapter (ENA) and Elastic Fabric Adapter (EFA):

Feature	Elastic Network Adapter (ENA)	Elastic Fabric Adapter (EFA)
Purpose	Enhances general network performance for EC2 instances	Optimizes inter-instance communication in HPC and ML
Use Cases	- High-performance computing (HPC)	- High-performance computing (HPC)
	- Machine learning and data analytics	- Machine learning and deep learning
	- Big data processing	- Clustered applications, parallel processing
	- Content delivery	- High-performance storage systems
	- Network-intensive applications
Throughput	High throughput for general networking	High throughput for inter-instance communication
Latency	Low latency for general networking	Low latency for inter-instance communication
Packet Size Support	Supports jumbo frames for efficient data transfer	Efficient handling of large packets and message sizes
Packet Offloading	Offloads checksum calculations	Provides offloading for collective operations and more
Traffic Mirroring	Supports traffic mirroring for network analysis	-
Instance Support	Supported by a wide range of EC2 instance types	Limited instances support; primarily HPC-optimized types
MPI Workloads	Generally used for various workloads	Optimized for Message Passing Interface (MPI) workloads
Security Group Support	Enhances the performance of security groups	-
Custom Networking	Provides advanced networking features	Focused on optimized communication within clusters
AWS Compatibility	Compatible with a broader range of EC2 instance types	Primarily designed for specific HPC-optimized instances

Bottom line is, If you just want enhanced networking for lower latency, look for an Elastic Network Adapter (ENA).
If you have a High Performance Cluster (HPC), use Elastic Fabric Adapter (EFA).

Lets do a quick ENA Demo

Note: ENA is only available for newer generation instances.

Spin-up a new EC2 instance running, a new generation instance type e.g t3.micro, running an Amazon Linux AMI
SSH or Instance Connect into the running instance and run
modinfo ena this would display details about the Elastic Network Adapter (ENA) kernel module, showing that enhanced networking can be leveraged.
Then run ethtool -i eth0 to display information about the specified network interface

Conclusion

In summary, when selecting EC2 instances, it's important to check whether the instance type supports ENA, as not all instance types include this network adapter.

ENA and EFA are valuable enhancements for SysOps professionals, enabling them to optimize network performance, enhance efficiency, and support specialized workloads such as HPC and machine learning. Keeping abreast of these networking enhancements allows SysOps professionals to make informed decisions when architecting and managing AWS environments.

AWS CodeCommit vs. Git: Choosing the Right Version Control System for Your Project

Ukeme David Eseme — Sat, 01 Jul 2023 21:55:41 +0000

Version control systems play a crucial role in software development, enabling teams to efficiently manage code, collaborate, and track changes. When it comes to choosing a version control system for your project, AWS CodeCommit and Git are two popular options.

In this blog post, we will explore the similarities, differences, and considerations for selecting between AWS CodeCommit and Git.

By understanding their features, workflows, and integration capabilities, you can make an informed decision that aligns with your project's requirements.

Understanding Git:

Git is an open-source distributed version control system widely adopted in the software development community. It provides a decentralized approach, allowing developers to have a local copy of the entire code repository. Git offers powerful branching and merging capabilities, facilitating parallel development, collaboration, and code versioning. Git repositories can be hosted on various platforms, including GitHub, GitLab, and Bitbucket.

Introducing AWS CodeCommit:

AWS CodeCommit is a fully managed source code control service provided by Amazon Web Services (AWS). It offers a secure and scalable environment for hosting private Git repositories. CodeCommit is integrated with other AWS services, such as AWS CodePipeline and AWS CodeBuild, enabling end-to-end CI/CD workflows. It provides features like access control, code reviews, and pull requests, ensuring a robust and collaborative development process within the AWS ecosystem.

Comparing AWS CodeCommit and Git:

To make an informed decision about the version control system for your project, let's compare AWS CodeCommit and Git based on several factors:

1. Hosting and Scalability:

Git repositories can be hosted on a variety of platforms, offering flexibility and the ability to choose a hosting provider that aligns with your needs. On the other hand, CodeCommit provides a fully managed and scalable hosting environment within the AWS ecosystem, ensuring high availability, security, and seamless integration with other AWS services.

2. Integration with AWS Services:

CodeCommit offers tight integration with other AWS DevOps tools, such as AWS CodePipeline, facilitating streamlined CI/CD workflows. Git repositories, on the other hand, can be integrated with various CI/CD platforms and services, allowing for flexibility in tooling choices.

3. Security and Access Control:

Both Git and CodeCommit provide mechanisms for securing code repositories and managing access control. CodeCommit leverages AWS Identity and Access Management (IAM) for fine-grained access control and integrates with AWS Key Management Service (KMS) for encryption at rest. Git repositories can implement access controls and authentication mechanisms based on the hosting platform's features.

4. Community and Ecosystem:

Git has a vast and vibrant community, with a wealth of resources, plugins, and integrations available. It offers extensive documentation, tutorials, and community support. While CodeCommit is relatively newer and has a smaller community, it benefits from the broader AWS ecosystem and integrates seamlessly with other AWS services.

5. Cost Considerations:

Git repositories hosted on third-party platforms may have associated costs based on usage, storage, and additional features. AWS CodeCommit pricing is based on active users, repository size, and data transfer, with a free tier available for small-scale projects.

Choosing between AWS CodeCommit and Git depends on your project's requirements, development workflow, integration needs, and familiarity with the platforms.

Conclusion

Git provides flexibility, a mature ecosystem, and multiple hosting options, while AWS CodeCommit offers a managed and integrated environment within the AWS ecosystem.
Consider factors such as scalability, security, integration capabilities, community support, and cost when making your decision. By evaluating these aspects, you can select the version control system that best suits your project's needs and empowers your development team.

Infrastructure as Code: Managing Docker Containers using AWS DevOps Tools

Ukeme David Eseme — Fri, 30 Jun 2023 18:17:56 +0000

Introduction:

In the world of modern software development, managing infrastructure has become a critical aspect of the DevOps lifecycle. Infrastructure as Code (IaC) has emerged as a best practice that allows developers to define and manage their infrastructure using code.
This blog post will explore how AWS DevOps tools can be leveraged to manage Docker containers using Infrastructure as Code principles. We'll dive into the key concepts and demonstrate practical examples using code snippets.

Understanding Infrastructure as Code:

Infrastructure as Code involves treating infrastructure components, such as servers, networks, and services, as programmable resources. This approach allows for version control, reproducibility, and automation, which are crucial for efficient infrastructure management. By using IaC, developers can define and provision their infrastructure using declarative code, enabling consistent deployments and eliminating manual configuration drift.

Managing Docker Containers with AWS DevOps Tools:

AWS provides a set of powerful DevOps tools that seamlessly integrate with Docker containers, enabling effective management and deployment. Let's explore some key AWS services and how they can be utilized for managing Docker containers as code.

1. AWS CloudFormation:

AWS CloudFormation is a powerful service that allows you to define and provision your AWS infrastructure using declarative templates. With CloudFormation, you can define a stack that includes various AWS resources, such as EC2 instances, VPCs, and security groups.
To manage Docker containers, you can use CloudFormation to create and configure the necessary resources, such as Amazon Elastic Container Service (ECS) clusters, task definitions, and services.

Example CloudFormation template snippet for defining an ECS service:

Resources:
  MyEcsService:
    Type: AWS::ECS::Service
    Properties:
      Cluster: !Ref MyEcsCluster
      TaskDefinition: !Ref MyEcsTaskDefinition
      DesiredCount: 2
      LaunchType: FARGATE
      NetworkConfiguration:
        AwsvpcConfiguration:
          Subnets:
            - !Ref MySubnet1
            - !Ref MySubnet2
          SecurityGroups:
            - !Ref MySecurityGroup

2. AWS CodePipeline:

AWS CodePipeline is a fully managed continuous integration and continuous delivery (CI/CD) service. It enables you to automate your software release workflows, including the deployment of Docker containers. CodePipeline integrates with various AWS services, including AWS CodeCommit, AWS CodeBuild, and AWS CodeDeploy. You can configure a pipeline that automatically builds and deploys your Docker images to Amazon Elastic Container Registry (ECR) or ECS.

Example CodePipeline configuration for building and deploying Docker containers:

Stages:
  - Name: Source
    Actions:
      - Name: SourceAction
        ActionTypeId:
          Category: Source
          Owner: AWS
          Provider: CodeCommit
          Version: "1"
        Configuration:
          RepositoryName: MyCodeRepo
          BranchName: main
        OutputArtifacts:
          - Name: source

  - Name: Build
    Actions:
      - Name: BuildAction
        ActionTypeId:
          Category: Build
          Owner: AWS
          Provider: CodeBuild
          Version: "1"
        Configuration:
          ProjectName: MyCodeBuildProject
        InputArtifacts:
          - Name: source
        OutputArtifacts:
          - Name: build

  - Name: Deploy
    Actions:
      - Name: DeployAction
        ActionTypeId:
          Category: Deploy
          Owner: AWS
          Provider: ECS
          Version: "1"
        Configuration:
          ClusterName: MyEcsCluster
          ServiceName: MyEcsService
          FileName: imagedefinitions.json
          Image1ArtifactName: build

3. AWS Elastic Beanstalk:

AWS Elastic Beanstalk is a fully managed platform that simplifies deploying and scaling applications. With Elastic Beanstalk, you can easily deploy your Docker containers without worrying about the underlying infrastructure. Elastic Beanstalk abstracts away the complexities of infrastructure management and provides a simple deployment model.

Example Elastic Beanstalk configuration for Docker container deployment:

Resources:
  MyElasticBeanstalkEnvironment:
    Type: AWS::ElasticBeanstalk::Environment
    Properties:
      ApplicationName: MyApplication
      EnvironmentName: MyEnvironment
      SolutionStackName: "64bit Amazon Linux 2 v3.4.3 running Docker"
      OptionSettings:
        - Namespace: aws:elasticbeanstalk:environment
          OptionName: EnvironmentType
          Value: SingleInstance
        - Namespace: aws:elasticbeanstalk:application:environment
          OptionName: MyEnvironmentVariable
          Value: MyValue

By leveraging AWS DevOps tools such as CloudFormation, CodePipeline, and Elastic Beanstalk, you can effectively manage Docker containers using Infrastructure as Code principles.

This approach provides numerous benefits, including version control, repeatability, and automation. By treating infrastructure as code, you can achieve consistency, scalability, and efficiency in managing your Dockerized applications on the AWS platform.

Creating An IAM User on AWS

Ukeme David Eseme — Fri, 03 Feb 2023 09:03:23 +0000

AWS Identity and Access Management (IAM) is a web service that helps you securely control access to AWS resources.

IAM is used to control authentication (who is signed in) and authorization (permissions) of AWS resources.

Identity and Access Management is responsible for:

Fine-grained access control to AWS resources
Analysis features to validate and fine-tune policies
Integration with external identity management solutions
AWS multi-factor authentication

IAM cloud identity tools are more secure and flexible than traditional username and password solutions

What is an IAM user?

An IAM user is a long-term credentialed identity used to interact with AWS in an account.

AWS has redesigned the Users List experience to make it easier to use.

Let's create an IAM User in 6 Simple Steps:

Step 1 — Login and Navigation.

Login to your AWS console with an administrative user or profile.
On the Top left corner, click on services, scroll down to Security, Identity & Compliance, click on IAM

Step 2 — Identity and Access Management(IAM) Dashboard.

On the left-hand corner of the IAM dashboard, under Access Management, click** Users*. Click on "Add Users*" > "Specify User Details."

Step 3 — Specify User Details.

a) Enter a User name, and tick Enable console access checkbox.
b) Generate Console password
c) Uncheck Users must create a new password at the next sign-in —
The reason why we unchecked this is because, we don’t want to keep changing our password on every sign-in…

Well… it all depends on your security preference 🤷
d) Click Next.

Step 4 — Set Permissions
Select Attach policies directly from the Permissions menu.

NB. The best practice is to attach the policies to a group, then add the created user to that group, but for this session, we would attach the policies directly to the user.

Under Permissions policies, for this test scenario, we want this user to have full access to Amazon S3.

In the search bar, search for "S3" and click on AmazonS3FullAccess*. You can also click the *Plus Icon, to view the selected policy in json format.

Click Next

Step 5 — Review
In this step, you are given the chance to review your choices and also have the option to create tags.

Click Create User

Step 6 — Retrieve the password
You can view and download the user’s password below, or email the user's instructions for signing in to the AWS Management Console. This is the only time you can view and download this password.

Under Console Password, click show to view your newly created password.
Click Return to users list.

User-created successfully 😊

Forem: Ukeme David Eseme

How I Built a Fully Automated Ethereum PoW Faucet on AWS Using Terraform + Ansible

Introduction

Overview

What is PoWFaucet?

Use Cases

Prerequisites

Architecture: Two Layers Working in Harmony

Infrastructure Layer: Terraform Does the Heavy Lifting

Terraform Modules

Application Layer: Ansible Orchestrates the Software

Configuration Management

Security Considerations

Automated Deployment

Script Workflow

Quick Four (4) Step Deployment

Monitoring

Metrics Collection

Visualization

Claim Rewards

Lessons from the Trenches

Final Thoughts

Resources

Building an AI-Powered IoT Analytics Pipeline on AWS: From Shoe Sensors to Predictive Insights

🚀 Ingestion Layer: Connecting the Shoe to the Cloud

🔄 Messaging Layer: Handling High-Volume Data

🧠 Compute Layer: Processing and Machine Learning

💾 Storage Layer: Persisting Raw and Processed Data

🔐 Security and Monitoring

🧩 Putting It All Together

🌍 Applications Beyond Shoes

Final Thoughts

How I Survived the Great Kubernetes Exodus: Migrating EKS Cluster from v1.26 to v1.33 on AWS

Introduction: The Challenge Ahead

Chapter 1: The Reconnaissance Phase

Mapping the Battlefield

💀 The Aging Cluster

🧪 First Attempt: The “By-the-Book” Upgrade

🛠️ Second Attempt: The Manifest Extraction Strategy

🚨 Reality Check: The Spaghetti Hit the Fan

⚙️ Third Attempt: Enter Velero

🧠 Fourth Attempt: Organized Manifest Extraction — The Breakthrough

📦 The CRD Explosion

🔍 Custom Resources Inventory

✅ API Compatibility Victory

📦 Chapter 2: The Data Dilemma

💡 Choosing Our Weapon: Manual EBS Snapshots

🛠️ Automation to the Rescue

🔐 Critical Volumes Backed Up

🏗️ Chapter 3: Building the New Kingdom

⚙️ The Foundation: CRD Installation Order Matters

🧬 Data Resurrection: Bringing Back Our State

🎭 Chapter 4: The Application Deployment Dance

The Dependency Choreography

🪜 Step-by-Step Deployment Strategy

🔐 Chapter 5: The Great SSL Certificate Saga

The Mystery of the Expired Certificates

🔎 The Missing Link: ClusterIssuer

🛠️ The Quick Fix

⚠️ Chapter 6: The AWS Load Balancer Controller Nightmare

🧩 The IAM Permission Maze

🔐 The Fix: A Proper IAM Role and Trust Policy

🌐 The Internal vs Internet-Facing Revelation

🛠️ The Fix: Force Internet-Facing Scheme

🌐 Chapter 7: The DNS Migration Challenge

The Automated Solution

⚙️ The Automated Solution

🏁 Chapter 8: The Final Victory

⚔️ The Moment of Truth

📘 Chapter 9: Lessons Learned

🔧 Technical Insights

🧠 Operational Insights

🛠️ Custom Tools That Saved Us

📊 Chapter 10: The Final Status

Performance Metrics

Conclusion: The Journey's End

What's Next?

Final Words

Django’s Game of Life Meets AWS ECS – The Ultimate Deployment Hack!

Table of Contents