Forem: Shireen Bano A

Production-Grade Container Hardening

Shireen Bano A — Thu, 19 Feb 2026 21:21:11 +0000

In modern DevOps, running containers as root isn't just sloppy — it's an open invitation. If your application is compromised while running as root, the attacker isn't just inside your app. They own the entire container. Every secret, every mounted volume, every network socket.

The good news? You can architect containers where a successful exploit lands an attacker in a box with nothing — no shell, no tools, no write access, no privileges. That's what this article is about.

We're building a hardened, production-grade container designed to run on AWS ECS Fargate, using defense-in-depth at every layer: the image, the process manager, the filesystem, and the task definition itself.

Layer 1: The Multi-Stage Build — Asset Stripping, Not Just Space Saving

Most developers know multi-stage builds shrink image size. Fewer realize they're also your first line of defense.
The strategy is simple: build dirty, run clean. Your first stage installs compilers, pulls npm packages, runs tests — all the messy work. Your final stage inherits none of it.

# Stage 1: The dirty build environment
FROM node:20-alpine AS builder
WORKDIR /app
COPY . .
RUN npm ci && npm run build

# Stage 2: The clean runtime — no npm, no git, no source code
FROM nginx:1.25-alpine
COPY --from=builder --chown=appuser:appgroup /app/client/dist /usr/share/nginx/html

Notice the --chown flag on the COPY instruction. Files land with the correct ownership immediately — no root middleman, no chmod dance afterward.

Layer 2: The Ghost Account — Least Privilege as Architecture

Alpine Linux defaults to running everything as root. We fix that immediately.

RUN addgroup -S appgroup && adduser -S appuser -G appgroup

The -S flag creates a system user — no password, no login shell, no home directory with a .bashrc to backdoor. It's a ghost account: it exists only so the kernel has a non-root identity to assign to your process.

USER appuser

This single line changes everything. From this point forward, every RUN, CMD, and ENTRYPOINT executes as appuser. The ceiling is enforced by the OS itself.

Layer 3: Taming Nginx — The Privileged Citizen Problem

Here's where it gets interesting. Standard Nginx assumes it's running as root. It wants to write its PID file to /var/run/nginx.pid and its logs to /var/log/nginx/. Our appuser is forbidden from touching either of those paths.
Rather than granting extra permissions, we patch Nginx to work within our constraints:

#Redirect Nginx internals to paths appuser actually owns
RUN sed -i 's|pid /var/run/nginx.pid;|pid /tmp/nginx.pid;|g' /etc/nginx/nginx.conf

# Pre-create the temp paths and hand them to appuser
RUN mkdir -p /tmp/client_body /tmp/proxy_temp /var/cache/nginx \
    && chown -R appuser:appgroup /tmp /var/cache/nginx

We're not lowering the security bar to accommodate Nginx — we're forcing Nginx to operate within our security model. The PID file and all scratch storage move to /tmp, which we then mount as ephemeral, hardened tmpfs volumes in the Fargate task definition.

linuxParameters = {
  tmpfs = [
    { containerPath = "/tmp",      size = 128, mountOptions = ["noexec", "nosuid", "nodev"] },
    { containerPath = "/app/logs", size = 64,  mountOptions = ["noexec", "nosuid", "nodev"] },
  ]
  readonlyRootFilesystem = true
}

Those three mount options are doing serious work:

noexec — Nothing in /tmp can be executed. Even if an attacker writes a binary there, it won't run.
nosuid — Blocks privilege escalation via setuid binaries dropped into the volume.
nodev — Prevents creation of device files that could be used to bypass hardware-level security.

And readonlyRootFilesystem = true is the crown jewel: the entire container filesystem is immutable at runtime. The only writable paths are the explicitly mounted tmpfs volumes — and those can't execute anything.

Layer 4: Supervisord Without the Crown

In a traditional setup, a process manager like systemd runs as root. We use Supervisord, and we strip its crown before it starts

[supervisord]
user=appuser
logfile=/tmp/supervisord.log
pidfile=/tmp/supervisord.pid

[program:nginx]
command=nginx -g 'daemon off;'
stdout_logfile=/dev/stdout
stderr_logfile=/dev/stderr

user=appuser means even the manager of processes has no administrative power. It coordinates but cannot escalate.
The stdout_logfile=/dev/stdout line solves another problem quietly: logs are streamed directly to Docker's logging driver and never written to disk inside the container. No sensitive log data sitting in a writable layer. No persistence for an attacker to mine.

Layer 5: Dropping Linux Capabilities — Cutting the Kernel's Leash

Even a non-root user can hold Linux capabilities — granular kernel permissions like the ability to bind low-numbered ports (NET_BIND_SERVICE), manipulate network interfaces (NET_ADMIN), or bypass file permission checks (DAC_OVERRIDE).

Every capability your container holds is an attack surface. The principle is simple: if you don't need it, drop it.

In your Fargate task definition:
linuxParameters = {
  capabilities = {
    drop = ["ALL"]
  }
}

After dropping all capabilities, your container's /proc/self/status should show:

CapEff: 0000000000000000
CapBnd: 0000000000000000

CapEff at zero means the process has no active kernel privileges. CapBnd at zero means it can never acquire any — capabilities removed from the bounding set cannot be added back. The kernel's leash is cut.

Layer 6: The Task Definition as a Second Lock

Your Dockerfile hardens the image. Your Fargate Task Definition hardens the runtime. These are two independent locks on the same door.

json"containerDefinitions": [
  {
    "privileged": false,
    "user": "appuser",
    "readonlyRootFilesystem": true,
    "linuxParameters": {
      "capabilities": { "drop": ["ALL"] }
    }
  }
]

Why does this matter if the Dockerfile already sets USER appuser? Because the Task Definition is enforced by the AWS Fargate agent at runtime, independently of what the image contains. Even if someone pushes a misconfigured image that forgot the USER directive, Fargate will still enforce appuser. Defense-in-depth means each layer protects against the failure of the layer before it.

privileged: false is the explicit rejection of the Docker --privileged flag, which would otherwise give the container near-full host access. On Fargate, the "host" is AWS's infrastructure — you definitely don't want that.

Layer 7: Trust But Verify — Container Image Signing with Notation

Hardening your runtime is only half the story. How do you know the image you're deploying is the one you built? Supply chain attacks — where a malicious image is substituted somewhere between your registry and your cluster — are a growing threat.

Notation (a CNCF project) lets you cryptographically sign container images and verify those signatures before deployment.

bash# Sign after pushing to ECR
notation sign <your-ecr-registry>/your-app:latest

Verify before deploying

notation verify <your-ecr-registry>/your-app:latest

Integrate this into your CI/CD pipeline: sign on push, verify on deploy. If the signature doesn't match, the deployment doesn't happen. You get cryptographic proof that what's running in Fargate is exactly what your pipeline built — no substitutions, no tampering.

🛡️ Security Hardening Matrix

Layer	Focus	Risk	Threat	Mitigation
L1	Multi-Stage Build	Build-time residue (compilers, `.git`, secrets) left in image.	Lateral Movement: Attackers use leftover tools to compile malware or pivot deeper into the network.	Build Dirty, Run Clean: Separate build and runtime stages; only production assets move to the final image.
L2	Non-Root Identity	Containers running as `root` (UID 0) by default.	Host Escape: Exploits (e.g., CVE-2024-21626) allow a root process to break out and control the host.	Ghost Accounts: Create a system `appuser` with no shell or home directory to enforce a permission "ceiling."
L3	Hardened Nginx	App requires root access to write to system paths like `/var/run`.	Runtime Tampering: Attackers overwrite configs or web files to serve malware or redirect traffic.	User-Owned Paths: Patch Nginx to use `/tmp` for PIDs/cache and `chown` those paths to the `appuser`.
L4	Unprivileged Manager	Process managers (Supervisord) traditionally running with root "crowns."	Privilege Escalation: A hijacked manager grants the attacker "God Mode" over all managed sub-processes.	Powerless Manager: Run `supervisord` as `appuser` and stream logs to `stdout` to prevent local data mining.
L5	Immutable Filesystem	Writable runtime layers allow attackers to modify the OS environment.	Malware Persistence: Attackers download web shells or scripts that survive as long as the container runs.	The "DVD" Model: Enable `readonlyRootFilesystem` and use `tmpfs` mounts with `noexec` to kill execution.
L6	Kernel Capabilities	Granular kernel permissions (Capabilities) active by default.	Privilege Jumping: Attackers use `NET_RAW` or `DAC_OVERRIDE` to sniff traffic or bypass file security.	The Blackout: Use `drop = ["ALL"]` to zero out `CapEff` and `CapBnd`, stripping all kernel-level privileges.
L7	Image Signing	Unverified images pulled from an untrusted or compromised registry.	Supply Chain Attack: An attacker swaps a legitimate image with a "poisoned" version containing a backdoor.	Notation (CNCF): Cryptographically sign images in CI/CD and verify signatures before every deployment.

Understanding AWS Autoscaling with Grafana

Shireen Bano A — Tue, 10 Feb 2026 18:25:00 +0000

Architecture Overview

My application is deployed on AWS as a containerized system:
React frontend served by Nginx
Node.js backend deployed as a Docker container

The backend relies heavily on:

RDS (reads/writes)
S3 (uploads/downloads)
Gemini API (LLM inference)

This architecture is intentionally realistic — it represents the type of stack many modern apps use today.
The Goal: High-Stress Scaling Through Load Testing

I wanted to validate autoscaling behavior under pressure. Specifically:

Can ECS scale out when traffic spikes?

How fast does it scale?

Does latency stay stable?

Does error rate increase under stress?

Which dependency becomes the bottleneck first?

Load Testing Strategy (k6)
To make the test realistic, I didn’t just hit a single endpoint repeatedly. Instead, I created a k6 test with two parallel scenarios:

1) Backend Load Scenario (Triggers Scaling)
This scenario generates the high traffic volume needed to push the backend and observe ECS behavior.

Warm-up at 20 users
Spike instantly to 500 users
Hold for 9 minutes
Drop back down

Observe scale-in behavior

2) UI Monitoring Scenario (Real User Flow)
This scenario runs a small number of browser-based users to monitor actual UI behavior while the system is under stress.

This includes:

login
navigation to medical history
viewing a PDF report
adding a condition
requesting recipes
uploading a report

This helped validate whether the UI stayed usable during the stress event.

The First Surprise: 500 VUs Did Not Spike CPU
At 500 virtual users, I expected ECS CPU utilization to become the main bottleneck. Instead, the CPU stayed surprisingly low — barely crossing 18%, even while the load test pushed close to 25,000 requests through the system. At first, this felt wrong, and I genuinely questioned whether my load test was working. But the test was fine — my assumption was not. After digging deeper, I realized the application workload simply wasn’t CPU-intensive. Most of the request time was spent waiting on external dependencies like RDS reads/writes, S3 uploads/downloads, and Gemini API responses. This made the system primarily I/O-bound, which explains why CPU-based autoscaling did not react strongly, even under heavy traffic.

This was one of the biggest lessons from the experiment:

High traffic does not always mean high CPU.

As you can see in the above graph, I almost hit more than 20000 request.

At the same time, here is my CPU and memory utilization graph, hitting no more than 20%.

According to my autoscaling policy, CPU utilization must cross 70% before CloudWatch triggers the alarm. Since my application isn’t naturally CPU-intensive, I wasn’t sure how else to push CPU high enough to test scaling properly.

So I manually generated CPU stress inside the running ECS container by executing an infinite loop using the following command:

aws ecs execute-command --cluster recipe-finder-prod-cluster \
    --task a55518997ca84f24bc2fd614cbc18f20 \
    --container recipe-finder-api \
    --interactive \
    --command "/bin/sh -c 'while true; do :; done & while true; do :; done'"

Within a few minutes, this forced the container CPU to spike aggressively, reaching a consistent ~99% utilization.

Now the real question becomes: how long does scale-out actually take?

Based on the autoscaling configuration, CloudWatch requires 60 seconds of sustained CPU breach before it enters the ALARM state. Once the alarm is triggered, ECS detects it and begins launching new tasks.

Event	Timestamp
CPU crossed 70%	12:09:00
Alarm triggered	12:13:25
Desired tasks increased	12:14:00
New task running	12:15:00

CPU crossed the 70% threshold at 12:09, but the CloudWatch alarm didn’t trigger until 12:13. ECS then increased the desired task count at 12:14, and the new task became fully running by 12:15 — meaning the full scale-out process took roughly 6 minutes from threshold breach to a healthy new task.

So, Autoscaling doesn’t react the instant CPU crossed the 70% threshold. CloudWatch evaluates CPU in 1-minute datapoints, and my alarm required 3 breaching datapoints within 3 minutes. Only after the alarm entered the ALARM state did ECS trigger scale-out and launch new Fargate tasks.

Now let's look at the scale in process:

Event	Timestamp	Notes
CPU fell below scale-in threshold (<63%)	12:34	Based on Grafana
Low alarm triggered (OK → ALARM)	12:49	15-min evaluation period complete
ECS desired tasks decreased	12:50	ECS starts stopping tasks
Extra task stopped (scale-in complete)	12:52	Task fully terminated

Notice how the alarm now triggers 15 minutes after the CPU fell below threshold, matching the Low alarm rule of 15 datapoints in 15 minutes.

Closing Note:

Autoscaling ensures your application can handle spikes, but it comes with temporary performance trade-offs:

During scale-out: When CPU spikes and new Fargate tasks are being launched, your application may briefly return 5xx errors or slower responses. In our experiment, we did see 5% errors for a few minutes during the initial warm-up period before the new tasks fully came online. This “warm-up latency” is an inherent part of reactive autoscaling.

During scale-in: ECS gradually terminates idle tasks once the Low alarm confirms sustained low CPU. This process is intentionally slow to avoid task flapping, ensuring that users aren’t suddenly impacted if traffic spikes again.

Observing CPU, alarm state, and task events together helps understand exactly how long users may experience degraded performance during scaling, and informs decisions about pre-warming, thresholds, and evaluation periods to minimize those user-facing impacts.

Github link:

Shireenbanu / AI-recipe-finder

Application Overview

This application helps users manage their health by securely storing medical history, lab reports, and personal profile information. Based on a patient’s conditions, it generates personalized healthy recipes using a recommendation engine integrated with the Gemini API. The goal is to provide actionable nutrition guidance while maintaining HIPAA compliance, data privacy, and secure storage. It also caches generated recipes for quick retrieval and seamless user experience.

How to install:

    terraform apply
    terrform destroy  #to destroy the infra

Recipe Finder Application:

1) Profile (CRUD + Database Reads/Writes)

Users can view and update profile information.

This workflow represents the most typical web-app traffic pattern: read and write operations to the database

2) Medical History (Uploads + Processing + AI Pipeline)

Users can upload lab reports and add medical conditions.

Once submitted, the backend processes the medical data and sends it to a recommendation engine, which then forwards structured…

View on GitHub

Learn What it Takes to Be a Solid Rails Developer!

Shireen Bano A — Tue, 24 Jan 2023 14:41:50 +0000

Unlock the full potential of Ruby on Rails and build web applications that will leave your peers in awe. But before you can do that, you need to master these essential concepts that are the building blocks of any successful Rails project. Let's dive in and explore them one by one

1. Familiarize yourself with Active record association and its best practices

bigbinary.com

I regret not finding this article when I was struggling to learn active record associations🥲(It's so good). This article will not only explain the various types of associations, but also provide a clear overview of how the associations interact with the database through the use of foreign key and primary key constructs.

2. Design patterns are the MACHINE GUNS🚀 of your codebase

The Best Design Patterns in Ruby On Rails

Learn how to implement 10 basic design patterns in Ruby on Rails and know how it helps refactor MVC components in Rails.

bacancytechnology.com

Clean code should never be underestimated, as it can save significant time for both yourself and other developers when debugging. Design patterns are the best practices followed by industry leaders. They not only make your code more readable (eg: Query objects) but also reduces code complexity(eg: Form objects) by separating the concerns (eg: Service object). Therefore, easy to maintain and review.

3. Only Unit testing can save you from Production crashes🔥

Back to Basics: Writing Unit Tests First

Step-by-step instructions for learning Test-Driven Development (TDD) in Ruby. There’s nothing to fear! It’s fun.

thoughtbot.com

Testing your code with every single input under the sun after every tiny tweak is like a recurring nightmare, amirite? 😴. That's where unit testing comes into rescue. It's a smart way writing code, where you write tests first before you even write the code. Basically, you make changes to your code, run the tests and boom👊🏻!, all the failed tests show up on your screen. Check out the above article for more details.

Code Coverage Tutorial: Branch, Statement, Decision, FSM

Code coverage is a measure which describes the degree of which the source code of the program has been tested. Following are major code coverage methods Statement Coverage, Condition Coverage, Branch Coverage, Toggle Coverage, FSM Coverage

guru99.com

Just writing test for the concerned parts of the code doesn't make your code production crash proof. Writing quality test that can provide maximum code coverage is what makes your code crash proof (99.9% times😅). The article above breaks down all the ways to make sure you're testing every nook and cranny of your code before it goes live. .

4. Code optimization is our Ultimate Goal💪🏻

There's a lot of things that can slow down our code execution, like using too many resources at the same time, bad design choices, and unoptimized code, but the one thing we can control is our code. So, let's make sure it's running as smooth as butter.
The following are the most common causes of poor application performance.

Slow view rendering :

Run any view of your rails application and check the logs in the terminal.

You will be able to find a line that will let you know how long it took for your view to load, how long it took for ActiveRecord to fetch data from the database, and how much time was spent creating new objects. Slow view rendering can be caused by factors such as an excessive amount of logic in the view, a large number of partials, and unoptimized images and other assets.

teamhq.app

If your view is taking forever to load and you can't figure out why, it might be a good idea to give the above article a quick read.

N+1 queries :

A common performance issue in Rails is the N+1 query problem, where a separate database query is made for each item in a collection. This can cause a lot of unnecessary database queries and slow down the view rendering.
To understand what n+1 queries are in rails, you can refer the below article.

Rails N+1 queries and eager loading

Junko T. ・ Dec 26 '20 ・ 3 min read

#rails #sql #beginners #performance

Large number of partials :

When you've got too many little chunks of views (partials) hanging around, it can slow down how fast your page loads. It's best to keep that number down, But If your existing application has too many partials which are not optimizable then its time to move to view_components. View components are a way to group common view logic together, which can simplify your code and make your application perform better.

If you're interested in experimenting with it, the article below can serve as a good starting point.

honeybadger.io

Lack of caching :

Rails, by default, does not include caching functionality. However, it does provide several options for caching through the use of third-party gems, such as the dalli gem for memcached caching and the redis-rails gem for Redis caching. Additionally, Rails also provides caching options through built-in methods such as cache and expires_in.

There are several gems available that can help identify and solve caching issues in Rails code. These gems can analyze your application and provide recommendations for optimizing caching strategies, such as identifying areas where caching can be added or improved. Some examples of such gems include rails_best_practices, bullet, and rack-mini-profiler.

If you're feeling like a mad scientist ready to conduct some experiments with these gems, the below post can be your lab coat and goggles😜.

Fix it until you make it, a simple Ruby on Rails Performance Guide | by Jorge Najera | The Startup | Medium

Jorge Najera ・ Jun 5, 2020 ・
Medium

All in all, Clean code practices for Ruby on Rails developer are a combination of using OOP principles, adhering to Rails conventions, automated testing, and keeping the codebase organized and well-commented.