Forem: Filbert Nana Blessing

How I Built a Production-Grade DevOps Project From Scratch

Filbert Nana Blessing — Sat, 14 Mar 2026 12:49:18 +0000

A walkthrough of building a real CI/CD pipeline, AWS infrastructure, and containerised app — the way it's actually done in production.

Why I Built This

Most DevOps tutorials show you how to deploy a "Hello World" app to a single EC2 instance with hardcoded AWS keys. That's fine for learning the basics, but it doesn't reflect what production engineering actually looks like.

I wanted to build something I could point to and say — this is how I would do it at a real company. No shortcuts, no tutorial hand-holding.

The result: a fully automated pipeline that takes code from a git push to a live HTTPS endpoint on AWS, with security scanning, infrastructure as code, observability, and zero static credentials anywhere.

Live URL: https://tasks.therealblessing.com
GitHub: github.com/nanafilbert/cicd-aws-terraform-deploy

What I Built

A Node.js task manager API with a Kanban dashboard UI, deployed to AWS behind an Application Load Balancer with:

8-stage GitHub Actions CI/CD pipeline
OIDC keyless AWS authentication — no static credentials
Modular Terraform — VPC, ALB, ASG, EC2, all in reusable modules
Multi-stage Docker build — tests run inside the build, broken images can't be pushed
Trivy CVE scanning — pipeline fails on HIGH/CRITICAL vulnerabilities
ACM SSL certificate with custom domain
Prometheus + Grafana observability stack locally
ASG instance refresh for zero-downtime deployments

The Architecture

GitHub Actions (OIDC) → Docker Hub → AWS
                                      │
                              ALB (HTTPS:443)
                              ACM Certificate
                              HTTP → HTTPS redirect
                                      │
                          Auto Scaling Group (EC2 t3.small)
                                      │
                              Docker Container
                              Node.js API :3000

The pipeline authenticates to AWS using OIDC short-lived tokens — no AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY anywhere. Every deploy triggers an ASG instance refresh that replaces EC2 instances with fresh ones pulling the latest image.

The CI/CD Pipeline

Eight stages, every one intentional:

1. Lint

ESLint checks code quality. Fails fast before any expensive steps.

2. Test

Jest runs 19 integration tests with coverage enforced at 80%. If tests fail, nothing gets built or deployed.

3. Security Scan

Trivy scans the filesystem for known CVEs in dependencies. Fails the pipeline on any HIGH or CRITICAL unfixed vulnerability. This caught real issues during development — Alpine CVEs and npm transitive dependency vulnerabilities that needed pinning.

4. Build & Push

Multi-stage Docker build. Three stages:

deps — installs only production dependencies
test — runs Jest inside the build process
production — minimal Alpine 3.21, non-root user, only what's needed to run

The test stage is critical. If your tests fail, the image doesn't get built. You physically cannot push a broken image.

5. Terraform Plan

terraform plan runs and saves the plan as an artifact. This is what gets applied in the next stage — no variables re-injected, no drift between plan and apply.

6. Deploy

terraform apply using the saved plan. Followed immediately by an explicit ASG instance refresh:

aws autoscaling start-instance-refresh \
  --auto-scaling-group-name $ASG_NAME \
  --preferences '{"MinHealthyPercentage": 50, "InstanceWarmup": 60}'

This is what actually gets new code onto the instances. Without triggering the refresh, the ASG would keep running the old image indefinitely.

7. Smoke Test

Polls /health/ready for up to 6 minutes after deploy. If the app never becomes healthy, the pipeline fails and you know immediately.

8. Summary

A pass/fail table written to the GitHub Actions job summary. Clean, visible at a glance.

The Terraform Setup

Three independent modules:

networking — VPC, public subnets across two AZs, internet gateway, route tables.

security — Security groups. The ALB accepts traffic from anywhere on 80 and 443. The app security group only accepts traffic from the ALB security group — EC2 instances are never directly reachable from the internet.

compute — ALB with HTTP redirect to HTTPS and an HTTPS listener with ACM certificate, launch template with IMDSv2 required, ASG with rolling instance refresh, IAM role for EC2 with SSM access.

A bootstrap/ folder handles the one-time setup that must exist before the pipeline can run — the OIDC provider, IAM role, S3 state bucket, and DynamoDB lock table.

Remote state in S3 with DynamoDB locking means the pipeline and local Terraform commands never conflict.

OIDC — The Right Way to Authenticate

This was the most important decision in the project.

The traditional approach is to create an IAM user, generate access keys, and store them as GitHub secrets. This works but creates long-lived credentials that can be leaked, rotated incorrectly, or forgotten.

OIDC works differently. GitHub Actions requests a short-lived token from GitHub's OIDC provider. AWS verifies that token against a trust policy and issues temporary credentials. The whole exchange happens in seconds and the credentials expire when the job ends.

permissions:
  id-token: write
  contents: read

- uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
    aws-region: ${{ secrets.AWS_REGION }}

The trust policy on the IAM role restricts assumption to this specific GitHub repo only:

"Condition": {
  "StringLike": {
    "token.actions.githubusercontent.com:sub": 
      "repo:nanafilbert/cicd-aws-terraform-deploy:*"
  }
}

No static credentials. Nothing to rotate. Nothing to leak.

Bugs I Hit (And What They Taught Me)

The Docker permission bug — The production container was crashing with Cannot find module '/app/src/app.js' even though the file clearly existed in the image. Took a while to figure out: I had set chmod -R 550 on the app directory. Read and execute, but no execute on directories means Node.js can't traverse the path. Changed to 755 and it worked immediately. The lesson: file permission bugs are silent and confusing — always verify what your non-root user can actually access.

The HSTS loop — After adding HTTPS, all API calls from the browser were being upgraded to HTTPS even when I explicitly typed http://. Helmet's default configuration sets a Strict-Transport-Security header, which tells browsers to remember to always use HTTPS for this origin. Even clearing the cache wasn't enough — had to explicitly clear the HSTS policy in Chrome's chrome://net-internals/#hsts and disable the header in Helmet for the HTTP-only ALB endpoint.

The instance refresh gap — After every deploy the new Docker image was pushed to Docker Hub, but the EC2 instance kept running the old one. Terraform saw no infrastructure changes so it didn't replace anything. The fix was to explicitly trigger an ASG instance refresh in the pipeline after every apply. Without that step, automation is an illusion — you're just pushing images that never get deployed.

The Terraform state lock — A failed pipeline run left a lock on the state file. Subsequent runs couldn't acquire the lock and failed immediately. Learned that terraform force-unlock -force <lock-id> from the correct working directory resolves this, and added auto-unlock logic to the plan job for future failures.

Observability

The app exposes Prometheus metrics via prom-client:

const promClient = require("prom-client");
promClient.collectDefaultMetrics({ register });

app.get("/health/metrics", async (req, res) => {
  res.set("Content-Type", register.contentType);
  res.send(await register.metrics());
});

Locally, docker-compose up starts the full stack — app, nginx, Prometheus, and Grafana. Prometheus scrapes /health/metrics every 15 seconds. Grafana visualizes CPU usage, heap memory, event loop lag, and active handles in real time.

Running a load test against the local API makes the graphs spike visibly — useful for demonstrating the observability story to anyone reviewing the project.

What I Would Add Next

RDS PostgreSQL — tasks currently live in memory and reset on deploy. A real database would make this production-ready in a deeper sense.
CloudWatch alarms — alert on unhealthy host count and high CPU before users notice.
WAF — Web Application Firewall in front of the ALB for rate limiting and bot protection at the infrastructure level.

Final Thoughts

The most valuable part of this project wasn't the technology — it was the debugging. Every bug I hit taught me something real: how file permissions work in containers, how browsers cache security policies, how Terraform state locking works, how ASG instance refresh interacts with deploy automation.

That's the difference between following a tutorial and building something yourself. The tutorial gives you the happy path. Building it yourself gives you everything else.

If you're building a DevOps portfolio, don't copy a tutorial. Pick a problem, build something real, and let it break. That's where the learning actually happens.

The full source code is at github.com/nanafilbert/cicd-aws-terraform-deploy and the live app is running at https://tasks.therealblessing.com.

How I Got Charged for an Elastic IP — And What I Learned About Cleaning Up AWS Resources

Filbert Nana Blessing — Tue, 29 Apr 2025 19:22:44 +0000

canonical_url: [https://follow-my-journey-in-devops.hashnode.dev/how-i-got-charged-for-an-elastic-ip-and-what-i-learned-about-cleaning-up-aws-resources]

Introduction

When I started experimenting with AWS as part of my journey into cloud and DevOps, I was focused on learning how to launch EC2 instances, SSH into them, and set up web servers. What I didn't expect was to get charged for something I wasn't even actively using: an Elastic IP. In this post, I want to share what happened, what I learned, and how you can avoid this common beginner mistake.

If you're new to AWS, check out my first post where I share how I got started experimenting with EC2 and the AWS Free Tier.

What Is an Elastic IP?

An Elastic IP (EIP) is a static public IPv4 address provided by AWS. Unlike regular public IPs (which change when you stop and start an instance), an Elastic IP remains the same until you explicitly release it. It's particularly useful if you want to ensure a consistent IP address for your application or server.

However, there's a catch: while Elastic IPs are free when attached to a running EC2 instance, they incur charges when they're not in use.

My Mistake

I tried using the AWS CLI to create an EC2 instance from my terminal. I wrote a script and ran it, but it didn’t seem to work. I tried again, and the result was the same. So I left it and instead used the AWS Management Console, which worked immediately.

What I didn’t realize was that even though my CLI attempts failed, they somehow triggered the allocation of two Elastic IPs to my account. The next day, a little over 24 hours later, I received a billing alert. I was surprised and curious because I was still using the Free Tier and hadn’t expected any charges. That’s when I discovered the Elastic IPs were the cause.

Why AWS Charges for Idle Elastic IPs

AWS offers one Elastic IP per running EC2 instance for free. But once that IP is no longer attached to a running instance, it’s considered idle. The reason behind the charge is simple: AWS wants to prevent users from hoarding public IP addresses, which are a limited resource.

Here are the key points:

Free: When the Elastic IP is associated with a running instance.

Charged: If the instance is stopped or terminated but the Elastic IP remains allocated.

What I Learned About Resource Cleanup

This experience taught me a lot about how AWS billing works and why it's so important to clean up unused resources:

Releasing Elastic IPs: Terminating an instance does not automatically release the EIP. You have to do it manually.

Monitoring Costs: Even if your instance is off, your billing might still increase due to leftover resources.

Check the VPC section of your billing dashboard — that’s where Elastic IP charges show up.

Billing Alarms: Set up CloudWatch billing alarms to alert you if your usage goes above $0.01 or any threshold you’re comfortable with.

Tips to Avoid Elastic IP Charges

If you're a beginner using AWS Free Tier, here are a few tips to help avoid charges like this:

Use Elastic IPs only when necessary.

Prefer auto-assigned public IPs for short-term or test projects.

Always release unused Elastic IPs manually.

Set billing alerts early.

Regularly review the Cost Explorer or Billing Dashboard to track your usage.

Conclusion

Getting charged for an idle Elastic IP was frustrating at first, but it became a valuable lesson in AWS resource management. Now I know to double-check every service I'm using and to always clean up after myself. If you're just starting out with AWS, I hope this post saves you from a similar surprise.

Have you had a similar experience with AWS costs? Share your story or tips in the comments below

canonical_url: [https://follow-my-journey-in-devops.hashnode.dev/how-i-got-charged-for-an-elastic-ip-and-what-i-learned-about-cleaning-up-aws-resources]

Launching Your First EC2 Instance: A Beginner's Guide 🚀

Filbert Nana Blessing — Sat, 26 Apr 2025 10:15:01 +0000

Launching Your First EC2 Instance: A Beginner's Guide 🚀

If you're just getting started with AWS, one of the first services you'll use is Amazon EC2 (Elastic Compute Cloud).

EC2 lets you create virtual machines (called instances) that you can access and run just like real computers.

In this guide, I'll walk you through the simple steps to create your very first EC2 instance — no complicated jargon!

Step 1: Open the EC2 Dashboard

Log in to your AWS Management Console.
At the top of the page, type EC2 into the search bar and click on the EC2 service.
On the left-hand menu, click Instances, then click Launch Instance to start the creation process.

Step 2: Name Your Instance

Give your instance a name.
It can be anything you want — something that makes it easy to recognize, especially when you have multiple instances later on.

Step 3: Choose an Amazon Machine Image (AMI)

Select an Amazon Machine Image (AMI).
The AMI includes:
- An Operating System (like Ubuntu, Amazon Linux, or Windows)
- Optional application software
- Custom configurations

Think of the AMI as the "template" for your virtual machine.

Step 4: Select the Instance Type

Choose your instance type.
For beginners and those using the AWS Free Tier, the best choice is t2.micro — it gives you enough resources without any cost.

Step 5: Create a Key Pair

You need a key pair to securely connect (SSH) to your instance.

Create a new key pair and give it a name.
AWS will generate:
- A public key (AWS stores this)
- A private key (you download as a .pem file)

⚠️ Important:

Download the .pem file immediately! AWS will not allow you to download it again.

Keep the file safe — you’ll need it when connecting to your instance.

Step 6: Configure Network Settings

Under Network Settings, click Edit.
AWS automatically assigns you a VPC and Subnet — you can leave the default options.

Then, configure your Security Group:

Allow SSH traffic (port 22) — to connect to your instance.
Allow HTTP (port 80) and HTTPS (port 443) traffic — if you plan to host a website.

Step 7: Configure Storage

AWS offers 8 GiB of storage by default with the Free Tier.
This is usually enough to get started, but you can adjust it based on your needs.

Step 8: Review and Launch

Take a moment to review all your settings.
If everything looks good, click Launch!

Your instance will move to the Running state.

Wait until you see Status Checks: 2/2 passed before connecting.

Step 9: Connect to Your Instance

Once your instance is running:

Click Connect in the AWS Console and follow the provided instructions.

Or manually SSH from your terminal or PowerShell:


bash
ssh -i /path/to/your-keypair.pem ubuntu@your-ec2-public-ip




Final Thoughts 🎉
Congratulations!
You've just launched and connected to your very first EC2 instance!

This skill lays the foundation for everything else you’ll build in the cloud.
Keep exploring, try different instance types, set up a basic web server — and most importantly — enjoy the learning journey! 🚀

💬 Got questions or stuck on a step? Drop a comment below — I’d love to help!


Originally published on my blog at Hashnode
(https://follow-my-journey-in-devops.hashnode.dev/launching-your-first-ec2-instance-a-beginners-guide)