Forem: Abraham Acha

SwiftDeploy: Building a Self-Writing Infrastructure Manager with Policy Enforcement — A Complete Technical Walkthrough

Abraham Acha — Wed, 06 May 2026 04:01:09 +0000

How I built a CLI tool that generates its own infrastructure configs, manages a full containerised stack, enforces deployment policies through OPA, exposes Prometheus metrics, and produces a live audit trail — all from a single YAML file.

The Problem We're Solving
Architecture Overview
Stage 4A — The Engine
- The Manifest
- The Python HTTP Service
- The Dockerfile
- The Templates
- The CLI
- The Two Deployment Modes
Stage 4B — The Eyes and Brain
- Prometheus Metrics
- OPA Policy Engine
- Gated Lifecycle
- The Status Dashboard
- The Audit Trail
The Debugging Sagas
Full Deployment Walkthrough
Key Lessons Learned

The Problem We're Solving {#the-problem}

Every time you spin up a new service in a real DevOps environment, you repeat the same manual work:

Write an Nginx config
Write a Docker Compose file
Run Docker commands
Check if things are healthy
Hope nobody deploys when the disk is full
Hope nobody promotes a canary that's throwing 60% errors

SwiftDeploy solves all of this. One YAML manifest describes your entire deployment. A CLI tool generates every config file from it, manages the container lifecycle, enforces safety policies before allowing deployments, exposes real-time metrics, and produces a full audit trail.

The golden rule: the manifest is the single source of truth. Everything else is generated.

Architecture Overview {#architecture}

┌─────────────────────────────────────────────────────────────────────────────┐
│                    SwiftDeploy — Full System Architecture                   │
├──────────────────┬──────────────────────────────────────┬───────────────────┤
│   ZONE 1         │          ZONE 2                      │   ZONE 3          │
│   Operator       │     Host Machine / Docker Engine     │   Generated Files │
│                  │                                      │                   │
│  [Operator]      │  ┌─── swiftdeploy-net (bridge) ───┐  │  nginx.conf       │
│      │           │  │                                │  │  docker-compose   │
│      ▼           │  │  [nginx:8080]──────►[app:3000] │  │  history.jsonl    │
│  manifest.yaml   │  │   PUBLIC           INTERNAL    │  │  audit_report.md  │
│  (source of      │  │      │                 │       │  │                   │
│   truth)         │  │      └──[logs vol]─────┘       │  │                   │
│      │           │  │                                │  │                   │
│      ▼           │  │  [opa:8181]                    │  │                   │
│  swiftdeploy     │  │  localhost only                │  │                   │
│  CLI             │  │  NOT via nginx ✗               │  │                   │
│  ├─ init         │  └────────────────────────────────┘  │                   │
│  ├─ validate     │                                      │                   │
│  ├─ deploy ──────┼──► pre-deploy: OPA infra check       │                   │
│  ├─ promote ─────┼──► pre-promote: OPA canary check     │                   │
│  ├─ status ──────┼──► scrapes /metrics every 5s ───────►│ history.jsonl     │
│  ├─ audit ───────┼────────────────────────────────────►│ audit_report.md   │
│  └─ teardown     │                                      │                   │
└──────────────────┴──────────────────────────────────────┴───────────────────┘

Stage 4A — The Engine {#stage-4a}

Stage 4A is the foundation. It answers one question: how do you build a tool that writes its own infrastructure configs?

The Project Structure

swiftdeploy/
├── manifest.yaml                    ← the ONLY file you edit
├── swiftdeploy                      ← CLI executable
├── Dockerfile                       ← app image definition
├── app/
│   └── main.py                      ← Python HTTP service
├── templates/
│   ├── nginx.conf.tmpl              ← nginx template
│   └── docker-compose.yml.tmpl     ← compose template
├── policies/                        ← Stage 4B addition
│   ├── infrastructure.rego
│   ├── canary.rego
│   └── data.json
├── nginx.conf                       ← generated (gitignored)
└── docker-compose.yml               ← generated (gitignored)

The Manifest {#the-manifest}

manifest.yaml is the brain of the entire system. Every component reads from it directly or via generated files.

services:
  image: swift-deploy-1-node:latest
  port: 3000
  mode: stable                    # stable or canary
  version: "1.0.0"
  restart_policy: unless-stopped
  log_volume: swiftdeploy-logs

nginx:
  image: nginx:latest
  port: 8080
  proxy_timeout: 30

opa:
  image: openpolicyagent/opa:latest-static
  port: 8181

network:
  name: swiftdeploy-net
  driver_type: bridge

contact: "ops@swiftdeploy.local"

Every field propagates through the system. Change nginx.proxy_timeout here and it updates in nginx.conf on the next init. Change services.mode here and the entire deployment mode switches on the next promote.

The Python HTTP Service {#the-python-service}

The app is a from-scratch HTTP server using only Python's stdlib — no Flask, no FastAPI. Three endpoints in Stage 4A, four in Stage 4B.

Configuration from environment

MODE        = os.environ.get("MODE", "stable")
APP_VERSION = os.environ.get("APP_VERSION", "1.0.0")
APP_PORT    = int(os.environ.get("APP_PORT", "3000"))
START_TIME  = time.time()

Configuration comes entirely from environment variables injected by Docker Compose at runtime. START_TIME is captured at module load — this is how /healthz calculates uptime without a database.

Thread-safe chaos state

chaos_lock = threading.Lock()
chaos_state = {"mode": None, "duration": None, "rate": None}

def get_chaos():
    with chaos_lock:
        return dict(chaos_state)  # returns a copy — callers can't mutate internal state

def set_chaos(state):
    with chaos_lock:
        chaos_state.update(state)

The Lock prevents race conditions when multiple requests read and write chaos state simultaneously. dict(chaos_state) returns a copy so the caller never holds a reference to the mutable internal dict.

The three Stage 4A endpoints

GET / — welcome

self.send_json(200, {
    "message": "Welcome to SwiftDeploy API",
    "mode": MODE,
    "version": APP_VERSION,
    "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
})

GET /healthz — liveness check

uptime = round(time.time() - START_TIME, 2)
self.send_json(200, {
    "status": "ok",
    "mode": MODE,
    "version": APP_VERSION,
    "uptime_seconds": uptime,
})

The /healthz endpoint does three jobs: proves the server is alive (Docker healthcheck), reports current mode (so promote can confirm the switch happened), and reports uptime (useful for debugging restart loops).

POST /chaos — chaos injection (canary only)

if MODE != "canary":
    self.send_json(403, {"error": "chaos endpoint only available in canary mode"})
    return

length = int(self.headers.get("Content-Length", 0))
data   = json.loads(self.rfile.read(length))
mode   = data.get("mode")

if mode == "slow":
    set_chaos({"mode": "slow", "duration": data.get("duration", 2), "rate": None})
elif mode == "error":
    set_chaos({"mode": "error", "duration": None, "rate": data.get("rate", 0.5)})
elif mode == "recover":
    set_chaos({"mode": None, "duration": None, "rate": None})

Reading Content-Length before rfile.read() is mandatory HTTP protocol — otherwise read() blocks forever waiting for data that never arrives.

The chaos modes:

slow — injects time.sleep(N) before responding, simulating a slow upstream
error — uses random.random() < rate to return 500 on a configurable percentage of requests
recover — clears all chaos state, returning to normal behaviour

The Dockerfile {#the-dockerfile}

FROM python:3.12-alpine

RUN addgroup -S appgroup && adduser -S appuser -G appgroup

WORKDIR /app
COPY app/main.py .
RUN chown -R appuser:appgroup /app

USER appuser

ENV MODE=stable
ENV APP_VERSION=1.0.0
ENV APP_PORT=3000

EXPOSE 3000

HEALTHCHECK --interval=10s --timeout=5s --start-period=15s --retries=5 \
  CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:3000/healthz', timeout=4)" || exit 1

CMD ["python", "main.py"]

Why Alpine? python:3.12-alpine is ~60MB. python:3.12 (Debian) is ~1GB. We need under 300MB.

Why non-root? If someone exploits the app, they get a powerless user, not root access to the server.

Why Python urllib for the healthcheck? This was a hard-won lesson. wget couldn't resolve localhost inside Alpine on WSL2 + Docker Desktop — a known network namespace quirk. Python's urllib bypasses the system resolver entirely and connects directly via socket. More reliable, no external tool needed.

The Templates {#the-templates}

Templates are blueprints with {{ placeholder }} values that the CLI replaces with real values from the manifest.

nginx.conf.tmpl key sections:

upstream app_backend {
    server app:{{ service_port }};
    keepalive 32;
}

log_format swiftdeploy '$time_iso8601 | $status | ${request_time}s | $upstream_addr | $request';

server {
    listen {{ nginx_port }};

    proxy_connect_timeout {{ proxy_timeout }}s;
    proxy_send_timeout    {{ proxy_timeout }}s;
    proxy_read_timeout    {{ proxy_timeout }}s;

    add_header X-Deployed-By swiftdeploy always;
    proxy_pass_header X-Mode;

    location @error502 {
        default_type application/json;
        return 502 '{"error":"Bad Gateway","code":502,"service":"app","contact":"{{ contact }}"}';
    }
}

Design decisions:

Custom log format — ISO timestamp, status code, response time, upstream IP, full request on one line
JSON error bodies — APIs need machine-readable errors, not nginx HTML pages
proxy_pass_header X-Mode — nginx strips custom headers by default; this forwards the canary header through to clients
keepalive 32 — maintains 32 persistent connections to the upstream, reducing connection overhead

docker-compose.yml.tmpl key sections:

app:
  expose:
    - "{{ service_port }}"    # container-to-container only, NEVER published to host

nginx:
  ports:
    - "{{ nginx_port }}:{{ nginx_port }}"    # only nginx faces the world

  depends_on:
    app:
      condition: service_healthy    # nginx waits for app healthcheck to pass

The expose vs ports distinction is a security boundary. expose = container-to-container only. ports = host-facing. The app is never reachable from outside Docker.

The CLI — Five Stage 4A Subcommands {#the-cli}

The template engine

def render_template(tmpl_path, context):
    with open(tmpl_path) as f:
        content = f.read()
    for key, val in context.items():
        content = content.replace("{{ " + key + " }}", str(val))
    return content

Five lines. No Jinja2. Simple string replacement. The templates are straightforward enough that a minimal custom engine is cleaner than pulling in a dependency.

`init` — generates everything from the manifest

def cmd_init():
    manifest = load_manifest()
    ctx      = build_context(manifest)

    nginx_conf   = render_template(NGINX_TMPL, ctx)
    compose_conf = render_template(COMPOSE_TMPL, ctx)

    with open(NGINX_OUT, "w") as f:   f.write(nginx_conf)
    with open(COMPOSE_OUT, "w") as f: f.write(compose_conf)

The grader deletes generated files and re-runs init to verify regeneration. Because we built it correctly, this is a non-issue.

`validate` — five pre-flight checks

# Check 1: manifest.yaml exists and is valid YAML
# Check 2: all required fields present and non-empty
# Check 3: docker image inspect — exits 0 if exists
# Check 4: ss -tlnp | grep :8080 — non-empty means port in use
# Check 5: nginx -t via isolated Docker container

Check 5 is the most interesting. We run nginx -t in a temporary container — validating syntax without needing nginx installed on the host. But we hit a complication: app:3000 (the upstream hostname) can't be resolved in an isolated container. Fix: swap server app: with server 127.0.0.1: in a temp copy before testing. Actual nginx.conf on disk is untouched.

test_content = data.replace("server app:", "server 127.0.0.1:")

`deploy` — brings up the stack and blocks until healthy

deadline = time.time() + 60
while time.time() < deadline:
    try:
        with urllib.request.urlopen(f"http://localhost:{nginx_port}/healthz", timeout=3) as resp:
            if json.loads(resp.read()).get("status") == "ok":
                healthy = True
                break
    except Exception:
        pass    # container still starting — connection refused is normal
    time.sleep(2)

docker compose up -d returns as soon as containers are created, not when they're healthy. The polling loop tries /healthz every 2 seconds for 60 seconds. Only {"status": "ok"} breaks the loop.

`promote` — rolling restart with zero nginx downtime

# 1. Update manifest in-place
content = re.sub(r"(mode:\s*)(\S+)", f"\\g<1>{target_mode}", content, count=1)

# 2. Regenerate docker-compose.yml with new MODE env var

# 3. Restart ONLY the app container — nginx stays up
run(compose_cmd("up -d --no-deps app"))

# 4. Confirm mode via /healthz

--no-deps is the key — it tells Compose to restart only the app service without touching nginx. Zero proxy downtime during the switch.

The Two Deployment Modes {#deployment-modes}

Stable mode — normal production behaviour. Clean responses. No special headers. /chaos returns 403.

Canary mode — test mode before full rollout:

if MODE == "canary":
    self.send_header("X-Mode", "canary")    # on EVERY response

Canary mode:

Adds X-Mode: canary to every response — callers can identify which mode they're hitting
Unlocks /chaos — lets you simulate slow responses, random errors, then recover
You promote with ./swiftdeploy promote canary, stress test, then ./swiftdeploy promote stable to roll back

Stage 4B — The Eyes and The Brain {#stage-4b}

Stage 4A built the engine. Stage 4B adds observability and policy enforcement. The stack now has eyes (metrics), a brain (OPA policies), and memory (audit trail).

Prometheus Metrics — The Eyes {#prometheus-metrics}

The app gains a /metrics endpoint exposing five metric types in Prometheus text format.

Tracking infrastructure

# Counter: {(method, path, status_code): count}
request_counts = {}

# Histogram state per path
HISTOGRAM_BUCKETS = [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
request_durations = {}

def record_request(method, path, status_code, duration_seconds):
    with metrics_lock:
        key = (method, path, str(status_code))
        request_counts[key] = request_counts.get(key, 0) + 1

        if path not in request_durations:
            request_durations[path] = {
                "buckets": {str(le): 0 for le in HISTOGRAM_BUCKETS},
                "sum": 0.0,
                "count": 0,
            }
        hist = request_durations[path]
        hist["sum"] += duration_seconds
        hist["count"] += 1
        for le in HISTOGRAM_BUCKETS:
            if duration_seconds <= le:
                hist["buckets"][str(le)] += 1

record_request() is called after every request regardless of path — timing wraps the entire handler:

def do_GET(self):
    start  = time.time()
    path   = self.path.split("?")[0]
    status = self._handle_get()
    record_request("GET", path, status, time.time() - start)

The five metrics

# 1. http_requests_total — counter, labels: method, path, status_code
# 2. http_request_duration_seconds — histogram with standard buckets
# 3. app_uptime_seconds — gauge
# 4. app_mode — gauge: 0=stable, 1=canary
# 5. chaos_active — gauge: 0=none, 1=slow, 2=error

The /metrics output

# HELP http_requests_total Total number of HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",path="/",status_code="200"} 42
http_requests_total{method="GET",path="/healthz",status_code="200"} 60
http_requests_total{method="GET",path="/",status_code="500"} 38

# HELP http_request_duration_seconds HTTP request latency in seconds
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{path="/",le="0.005"} 40
http_request_duration_seconds_bucket{path="/",le="+Inf"} 80
http_request_duration_seconds_sum{path="/"} 0.042381
http_request_duration_seconds_count{path="/"} 80

# HELP app_mode Current deployment mode (0=stable, 1=canary)
# TYPE app_mode gauge
app_mode 1

# HELP chaos_active Current chaos state (0=none, 1=slow, 2=error)
# TYPE chaos_active gauge
chaos_active 2

OPA Policy Engine — The Brain {#opa-policy-engine}

Open Policy Agent is a dedicated container whose only job is making allow/deny decisions based on rules written in Rego. The core principle: the CLI never makes allow/deny decisions itself. All decision logic lives exclusively in OPA.

Why OPA instead of if/else in the CLI?

If the policy logic lives in the CLI, changing a threshold means editing Python code, rebuilding, redeploying. With OPA, you edit data.json, restart the OPA container, and the new threshold is live. Policy as code, not policy as application logic.

data.json — thresholds live here, never hardcoded in Rego

{
  "thresholds": {
    "min_disk_free_gb": 10.0,
    "max_cpu_load": 2.0,
    "min_mem_free_percent": 10.0,
    "max_error_rate_percent": 1.0,
    "max_p99_latency_ms": 500.0
  }
}

infrastructure.rego — pre-deploy policy

package swiftdeploy.infrastructure

import future.keywords.if
import future.keywords.contains

default allow := false

allow if {
    disk_ok
    cpu_ok
    mem_ok
}

disk_ok if { input.disk_free_gb >= data.thresholds.min_disk_free_gb }
cpu_ok  if { input.cpu_load_1m  <= data.thresholds.max_cpu_load }
mem_ok  if { input.mem_free_percent >= data.thresholds.min_mem_free_percent }

reasons contains msg if {
    not disk_ok
    msg := sprintf(
        "disk_free_gb is %.1f, minimum required is %.1f",
        [input.disk_free_gb, data.thresholds.min_disk_free_gb]
    )
}

# decision is what the CLI reads — never a bare boolean
decision := {
    "allow":   allow,
    "reasons": reasons,
    "domain":  "infrastructure",
    "checked": {
        "disk_free_gb":     input.disk_free_gb,
        "cpu_load_1m":      input.cpu_load_1m,
        "mem_free_percent": input.mem_free_percent,
    },
}

Why import future.keywords? The openpolicyagent/opa:latest-static image uses Rego v1 which requires explicit if and contains keywords. Without these imports, OPA crashes on startup. This was discovered the hard way during testing.

canary.rego — pre-promote policy

package swiftdeploy.canary

import future.keywords.if
import future.keywords.contains

default allow := false

allow if {
    error_rate_ok
    latency_ok
}

error_rate_ok if { input.error_rate_percent <= data.thresholds.max_error_rate_percent }
latency_ok    if { input.p99_latency_ms     <= data.thresholds.max_p99_latency_ms }

reasons contains msg if {
    not error_rate_ok
    msg := sprintf(
        "error_rate is %.2f%%, maximum allowed is %.2f%%",
        [input.error_rate_percent, data.thresholds.max_error_rate_percent]
    )
}

decision := {
    "allow":   allow,
    "reasons": reasons,
    "domain":  "canary",
    "checked": {
        "error_rate_percent": input.error_rate_percent,
        "p99_latency_ms":     input.p99_latency_ms,
        "window_seconds":     input.window_seconds,
    },
}

OPA isolation — no leakage via nginx

In docker-compose.yml.tmpl:

opa:
  ports:
    - "127.0.0.1:{{ opa_port }}:8181"    # localhost only — never 0.0.0.0

127.0.0.1:8181 means only the host machine can reach OPA. The nginx container on port 8080 has no route to OPA. This is enforced at the Docker network binding level, not just convention.

The policy query function

def query_opa(manifest, package, input_data):
    url     = f"{opa_url(manifest)}/v1/data/{package.replace('.', '/')}/decision"
    payload = json.dumps({"input": input_data}).encode()
    try:
        req = urllib.request.Request(url, data=payload,
              headers={"Content-Type": "application/json"}, method="POST")
        with urllib.request.urlopen(req, timeout=5) as resp:
            body   = json.loads(resp.read())
            result = body.get("result")
            if result is None:
                return None, "OPA returned empty result — check policy package name"
            return result, None
    except urllib.error.URLError as e:
        return None, f"OPA unreachable: {e.reason}"
    except Exception as e:
        return None, f"OPA query failed: {e}"

Every distinct failure mode returns a different error string. The CLI never crashes or hangs when OPA is unavailable — it warns and fails open. This is intentional: you don't want OPA unavailability to block emergency deployments.

Gated Lifecycle — The CLI Brain {#gated-lifecycle}

Pre-deploy check

def cmd_deploy():
    manifest   = load_manifest()
    host_stats = get_host_stats()

    # Collect host stats
    disk_free_gb     = shutil.disk_usage("/").free / (1024 ** 3)
    cpu_load_1m      = float(open("/proc/loadavg").read().split()[0])
    mem_free_percent = (meminfo["MemAvailable"] / meminfo["MemTotal"]) * 100

    # Send to OPA
    allowed = enforce_policy(manifest, "swiftdeploy.infrastructure", host_stats, "infrastructure")

    if not allowed:
        append_history({"event": "deploy_blocked", "reason": "infrastructure_policy"})
        sys.exit(1)

    # Only reach here if OPA allows
    run(compose_cmd("up -d --build"))

If the disk is full, OPA returns:

  ✘ Policy [infrastructure] DENIED
    ! disk_free_gb is 8.2, minimum required is 10.0

  Deployment blocked by policy: infrastructure

Pre-promote check

def cmd_promote(target_mode):
    if target_mode == "canary":
        raw        = scrape_metrics(nginx_port)
        metrics    = parse_prometheus(raw)
        error_rate = calculate_error_rate(metrics)
        p99_ms     = calculate_p99_latency_ms(metrics)

        allowed = enforce_policy(manifest, "swiftdeploy.canary",
            {"error_rate_percent": error_rate, "p99_latency_ms": p99_ms, "window_seconds": 30},
            "canary safety"
        )
        if not allowed:
            sys.exit(1)

P99 latency calculation from histogram

def calculate_p99_latency_ms(metrics, path_filter=None):
    buckets     = {}
    total_count = 0

    for entry in metrics.get("http_request_duration_seconds_bucket", []):
        le = entry["labels"].get("le", "")
        if le == "+Inf":
            total_count = max(total_count, entry["value"])
            continue
        buckets[float(le)] = buckets.get(float(le), 0) + entry["value"]

    if total_count == 0:
        return 0.0

    p99_threshold = total_count * 0.99
    for le in sorted(buckets.keys()):
        if buckets[le] >= p99_threshold:
            return round(le * 1000, 2)    # seconds → milliseconds
    return 10000.0

P99 means: the smallest histogram bucket where 99% of requests have completed. If 99 out of 100 requests finished within 250ms, P99 = 250ms.

The Status Dashboard — The Eyes {#status-dashboard}

def cmd_status():
    while True:
        raw        = scrape_metrics(nginx_port)
        metrics    = parse_prometheus(raw)
        error_rate = calculate_error_rate(metrics)
        p99_ms     = calculate_p99_latency_ms(metrics)

        # Query OPA for live compliance
        infra_dec,  _ = query_opa(manifest, "swiftdeploy.infrastructure", get_host_stats())
        canary_dec, _ = query_opa(manifest, "swiftdeploy.canary",
                            {"error_rate_percent": error_rate, "p99_latency_ms": p99_ms, "window_seconds": 30})

        os.system("clear")
        # ... render dashboard ...

        append_history({
            "event":              "status_scrape",
            "error_rate_percent": error_rate,
            "p99_latency_ms":     p99_ms,
            "mode":               mode_str,
            "chaos":              chaos_str,
            "policy_infra_pass":  infra_dec.get("allow") if infra_dec else None,
            "policy_canary_pass": canary_dec.get("allow") if canary_dec else None,
        })

        time.sleep(5)

What the dashboard looks like with chaos active:

  SwiftDeploy Status Dashboard  2026-05-05T21:00:37Z
  ────────────────────────────────────────────────

  ── Throughput ──────────────────────────────────
  req/s       : 2.4
  error rate  : 56.45%      ← red
  P99 latency : 5.0ms

  ── App State ───────────────────────────────────
  mode        : canary
  chaos       : error       ← red
  uptime      : 316s

  ── Policy Compliance ───────────────────────────
  ✔  infrastructure       PASS
  ✘  canary               FAIL
       ! error_rate is 56.45%, maximum allowed is 1.00%

  Refreshing every 5s — Ctrl+C to exit

This is exactly what real SRE dashboards do — they show you the current state AND whether it violates policy in real time.

The Audit Trail — The Memory {#audit-trail}

Every event appends a JSON line to history.jsonl:

{"timestamp":"2026-05-05T20:34:51Z","event":"deploy","mode":"stable"}
{"timestamp":"2026-05-05T20:55:22Z","event":"promote","target_mode":"canary"}
{"timestamp":"2026-05-05T20:55:23Z","event":"status_scrape","error_rate_percent":62.5,"chaos":"error","policy_canary_pass":false}
{"timestamp":"2026-05-05T21:01:01Z","event":"promote","target_mode":"stable"}

swiftdeploy audit reads this file and generates audit_report.md:

## Mode Changes

| Timestamp | From | To |
|-----------|------|----|
| 2026-05-05T20:34:51Z | unknown | stable |
| 2026-05-05T20:55:22Z | stable  | canary |
| 2026-05-05T21:01:01Z | canary  | stable |

## Policy Violations

| Timestamp | Infrastructure | Canary | Error Rate | P99 |
|-----------|---------------|--------|------------|-----|
| 2026-05-05T20:55:23Z | ✔ PASS | ✘ FAIL | 62.5% | 5.0ms |
| 2026-05-05T20:55:28Z | ✔ PASS | ✘ FAIL | 63.6% | 5.0ms |

This report renders perfectly as GitHub Flavored Markdown — every table, every checkmark, every timestamp.

The Debugging Sagas {#debugging}

No real DevOps project ships without war stories. Here are the ones that taught the most.

Saga 1 — Six layers of healthcheck failure

The app container kept showing unhealthy despite the server running fine. The debugging sequence:

Failure 1: ${APP_PORT} doesn't expand in Dockerfile HEALTHCHECK CMD. Env vars evaluate at build time, not runtime. Fixed by hardcoding 3000.

Failure 2: localhost doesn't resolve inside Alpine's healthcheck context on WSL2 + Docker Desktop. Fixed by using 127.0.0.1.

Failure 3: wget with 127.0.0.1 still failed. Confirmed the server WAS listening:

docker exec swiftdeploy-app ss -tlnp
# tcp LISTEN 0.0.0.0:3000
docker exec swiftdeploy-app python -c "import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:3000/healthz').read())"
# b'{"status": "ok", ...}'   ← works via exec, not via healthcheck

This is a known WSL2 + Docker Desktop network namespace issue. Fixed by using Python's urllib instead of wget.

Failure 4: Docker cache was serving the old image despite the Dockerfile fix. Fixed with --no-cache.

Failure 5: The docker-compose.yml template had its own healthcheck block overriding the Dockerfile. Docker Compose healthcheck always wins. Fixed the template too.

Failure 6: The healthcheck YAML block had 3 spaces indent instead of 4. A single space difference caused a YAML parse error. Fixed by carefully rewriting the block.

Saga 2 — OPA Rego v1 syntax

The openpolicyagent/opa:latest-static image enforces strict Rego v1 syntax. Our policies used the older syntax:

# OLD — crashes on latest OPA
allow {
    disk_ok
}
reasons[msg] {
    not disk_ok
    msg := "..."
}

# NEW — Rego v1 required syntax
allow if {
    disk_ok
}
reasons contains msg if {
    not disk_ok
    msg := "..."
}

Without import future.keywords.if and import future.keywords.contains at the top of each file, OPA refuses to start.

Saga 3 — WSL2 path spaces breaking docker run

The project lived at /mnt/c/Users/RAZER BLADE/Desktop/HNG/hng-swiftdeploy. The space in RAZER BLADE caused docker run -v {path}:... to split the path at the space, making Docker interpret the second half as an image name.

docker: invalid reference format: repository name
(Desktop/HNG/hng-swiftdeploy/nginx.conf) must be lowercase

Fixed by quoting all paths containing the project directory and using subprocess.run with a list instead of shell=True to avoid shell word-splitting entirely.

Full Deployment Walkthrough {#deployment}

# 1. Build the image
docker build -t swift-deploy-1-node:latest .

# 2. Validate pre-flight checks
./swiftdeploy validate

# 3. Deploy (OPA policy check runs first)
./swiftdeploy deploy

# 4. Verify metrics
curl http://localhost:8080/metrics

# 5. Verify OPA isolation
curl http://127.0.0.1:8181/health    # works — internal
curl http://localhost:8080/v1/data   # 404 — nginx blocks it

# 6. Launch status dashboard
./swiftdeploy status

# 7. Promote to canary (OPA canary policy check runs first)
./swiftdeploy promote canary

# 8. Inject chaos
curl -X POST http://localhost:8080/chaos \
  -H "Content-Type: application/json" \
  -d '{"mode": "error", "rate": 0.5}'

# 9. Watch status dashboard catch it — canary policy FAIL visible in real time

# 10. Recover
curl -X POST http://localhost:8080/chaos \
  -H "Content-Type: application/json" \
  -d '{"mode": "recover"}'

# 11. Promote back to stable
./swiftdeploy promote stable

# 12. Generate audit report
./swiftdeploy audit
cat audit_report.md

# 13. Teardown
./swiftdeploy teardown --clean

Key Lessons Learned {#lessons}

1. Docker Compose healthcheck overrides Dockerfile HEALTHCHECK. Always check both places when healthchecks misbehave. Compose wins every time.

2. WSL2 has a different network namespace for healthchecks than for docker exec. If something works via exec but not via healthcheck, it's almost certainly a tool or network namespace issue. Python's stdlib is more portable than wget in this environment.

3. OPA Rego v1 requires explicit keywords. latest-static means the latest OPA — which enforces Rego v1 syntax. Always import future.keywords.if and import future.keywords.contains.

4. expose vs ports is a security boundary, not documentation. expose = container-to-container only. ports = host-facing. Binding OPA to 127.0.0.1 enforces isolation at the network level.

5. The CLI should never make policy decisions. Every time you add an if/else for a deployment condition in the CLI, you're doing OPA's job badly. Push all allow/deny logic into Rego. The CLI's job is to collect data and surface decisions.

6. P99 latency is more useful than average. An average latency of 10ms can hide the fact that 1 in 100 requests takes 5 seconds. P99 exposes that tail. Always instrument histograms, not just averages.

7. Declarative infrastructure pays off immediately. The grader deletes generated files and re-runs init. Because the manifest is always there and regeneration is instantaneous, this is a non-issue. Manual configs would have been a problem.

8. An audit trail is not optional. history.jsonl made it trivial to answer "when did chaos start?", "which policy was failing?", "how long was the canary running before we promoted?" These questions matter in production incidents.

Conclusion

SwiftDeploy started as a task requirement and became a complete mental model for how modern deployment tooling works. Every major concept is here:

Declarative infrastructure — describe what you want, generate everything else
Immutable configs — generated files are outputs, never inputs
Policy as code — OPA enforces safety standards that can't be bypassed
Observability — Prometheus metrics feed the dashboard and the policy engine
Audit trail — every event recorded, every violation surfaced

The combination of Stage 4A and 4B forms a complete deployment lifecycle: generate → validate → deploy (gated) → promote (gated) → observe → audit → tear down.

The full source code is available at: https://github.com/AirFluke/hng-swiftdeploy

Tags: #devops #docker #nginx #python #opa #prometheus #infrastructure #hng

SwiftDeploy: Building a Declarative Infrastructure Manager from Scratch; A Complete Technical Walkthrough

Abraham Acha — Sun, 03 May 2026 17:27:03 +0000

How I built a CLI tool that generates, manages, and monitors a full containerised stack from a single YAML manifest — and everything that broke along the way.

Introduction

Most DevOps tasks ask you to configure infrastructure manually. You write an Nginx config. You write a Docker Compose file. You run commands. You check if things are healthy. You repeat this every time you spin up a new service.

SwiftDeploy flips that script entirely.

The premise is simple but powerful: one YAML file describes your entire deployment, and a CLI tool derives everything else from it. No handwritten Nginx configs. No manually crafted Docker Compose files. No guessing at container states. You edit the manifest, you run the CLI, and your stack is live.

This article is a complete technical walkthrough of how I built SwiftDeploy for the HNG DevOps Internship Stage 4A task — covering the architecture, every component, every design decision, the bugs I hit (including a particularly nasty WSL2 healthcheck issue), and the debugging process that resolved them.

By the end, you'll understand:

How to build a declarative infrastructure tool from scratch
How template-based config generation works
How to write a multi-subcommand Python CLI
How Docker healthchecks work and why they fail in unexpected ways
How nginx reverse proxying, canary deployments, and chaos engineering fit together

Let's get into it.

The Architecture

Before writing a single line of code, I mapped out how the pieces would connect.

You (the human)
     │
     │  edit only this
     ▼
manifest.yaml  ─────────────────────────────────────┐
                                                     │
                                                     ▼
                                           swiftdeploy (CLI)
                                                     │
                         ┌───────────────────────────┤
                         │                           │
                         ▼                           ▼
                   nginx.conf              docker-compose.yml
               (generated file)            (generated file)
                         │                           │
                         └──────────────┬────────────┘
                                        │
                                        ▼
                               Docker starts 2 containers
                                        │
                         ┌──────────────┴──────────────┐
                         │                             │
                         ▼                             ▼
                [nginx container]            [app container]
                 port 8080 (public)           port 3000 (private)
                         │                             │
                         └──── nginx proxies ──────────┘

Internet → port 8080 → nginx → port 3000 → Python app

The key architectural decisions:

The manifest is the single source of truth. Everything — nginx timeouts, container ports, network names, deployment mode — lives in manifest.yaml. The CLI reads it and generates everything else.
The app is never exposed directly. All traffic flows through nginx on port 8080. The app container only uses expose (internal Docker network), never ports (host-facing).
Generated files are gitignored. nginx.conf and docker-compose.yml are outputs, not inputs. They're always regeneratable from the manifest. The grader tests this explicitly — deleting generated files and re-running init.
The CLI is self-contained. One executable Python script, no framework, handles five subcommands.

The Project Structure

swiftdeploy/
├── manifest.yaml               ← the ONLY file you edit
├── swiftdeploy                 ← CLI executable
├── Dockerfile                  ← app image definition
├── app/
│   └── main.py                 ← Python HTTP service
├── templates/
│   ├── nginx.conf.tmpl         ← nginx template
│   └── docker-compose.yml.tmpl ← compose template
├── nginx.conf                  ← generated by init (gitignored)
├── docker-compose.yml          ← generated by init (gitignored)
├── .gitignore
└── README.md

Component 1: The Manifest

manifest.yaml is the brain of the entire system. Every other component reads from it either directly or via the generated files.

services:
  image: swift-deploy-1-node:latest
  port: 3000
  mode: stable
  version: "1.0.0"
  restart_policy: unless-stopped
  log_volume: swiftdeploy-logs

nginx:
  image: nginx:latest
  port: 8080
  proxy_timeout: 30

network:
  name: swiftdeploy-net
  driver_type: bridge

contact: "ops@swiftdeploy.local"

The design intention is that this reads like infrastructure-as-documentation. You can look at this file and understand the entire deployment without reading a single generated config.

What each field controls

services.mode — controls whether the app runs in stable or canary mode. The CLI's promote subcommand updates this in-place.
nginx.proxy_timeout — propagates into the nginx config as proxy_connect_timeout, proxy_send_timeout, and proxy_read_timeout.
contact — injected into nginx's custom JSON error bodies for 502/503/504 responses.
log_volume — a named Docker volume shared between the app container and nginx, so both write logs to the same persistent location.

Component 2: The Python HTTP Service

The app is a from-scratch HTTP server built on Python's http.server stdlib — no Flask, no FastAPI, no external dependencies. This keeps the Docker image small and the container startup fast.

The server setup

import os
import time
import random
import threading
import json
from http.server import HTTPServer, BaseHTTPRequestHandler

MODE = os.environ.get("MODE", "stable")
APP_VERSION = os.environ.get("APP_VERSION", "1.0.0")
APP_PORT = int(os.environ.get("APP_PORT", "3000"))

START_TIME = time.time()

Configuration comes entirely from environment variables injected by Docker Compose at runtime. The defaults exist only as fallbacks for local development.

START_TIME is captured at module load — this is how /healthz calculates uptime without a database or external state store.

Thread-safe chaos state

chaos_lock = threading.Lock()
chaos_state = {"mode": None, "duration": None, "rate": None}

def get_chaos():
    with chaos_lock:
        return dict(chaos_state)

def set_chaos(state):
    with chaos_lock:
        chaos_state.update(state)

Python's http.server handles each request in the same thread by default (it's not async), but I added explicit thread safety here anyway — the Lock ensures that if you ever extend this to a threaded server, chaos state reads and writes remain atomic. dict(chaos_state) returns a copy, preventing the caller from holding a reference to the mutable internal state.

The request handler

class Handler(BaseHTTPRequestHandler):
    def log_message(self, format, *args):
        pass  # suppress default logging — nginx handles access logs

    def send_json(self, code, body, extra_headers=None):
        payload = json.dumps(body).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("X-Deployed-By", "swiftdeploy")
        if MODE == "canary":
            self.send_header("X-Mode", "canary")
        self.end_headers()
        self.wfile.write(payload)

send_json is a helper that consolidates all the boilerplate of setting response codes, content type, and custom headers in one place. Every route calls it — this is the DRY principle applied to HTTP handlers.

Suppressing log_message is intentional. The default Python HTTP server writes its own access log to stdout, which would duplicate what nginx already logs in the structured format we defined.

The three routes

GET / — welcome endpoint

if self.path == "/":
    self.send_json(200, {
        "message": "Welcome to SwiftDeploy API",
        "mode": MODE,
        "version": APP_VERSION,
        "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    })

GET /healthz — liveness check

elif self.path == "/healthz":
    uptime = round(time.time() - START_TIME, 2)
    self.send_json(200, {
        "status": "ok",
        "mode": MODE,
        "version": APP_VERSION,
        "uptime_seconds": uptime,
    })

The /healthz endpoint does three jobs simultaneously: it proves the server is alive (Docker healthcheck), it reports the current mode (so promote can confirm the switch happened), and it reports uptime (useful for debugging restart loops).

POST /chaos — chaos injection (canary only)

def do_POST(self):
    if self.path == "/chaos":
        if MODE != "canary":
            self.send_json(403, {"error": "chaos endpoint only available in canary mode"})
            return

        length = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(length)
        data = json.loads(body)
        mode = data.get("mode")

        if mode == "slow":
            set_chaos({"mode": "slow", "duration": data.get("duration", 2), "rate": None})
        elif mode == "error":
            set_chaos({"mode": "error", "duration": None, "rate": data.get("rate", 0.5)})
        elif mode == "recover":
            set_chaos({"mode": None, "duration": None, "rate": None})

Reading Content-Length before calling rfile.read() is standard HTTP protocol — you must know exactly how many bytes to read, otherwise the read blocks waiting for more data that never comes.

The chaos modes:

slow — injects time.sleep(N) before responding, simulating a slow upstream
error — uses random.random() < rate to return 500 on a configurable percentage of requests
recover — clears all chaos state, returning to normal behaviour

This is real chaos engineering in miniature — the same concept used by tools like Chaos Monkey, just scoped to a single service.

Component 3: The Dockerfile

FROM python:3.12-alpine

RUN addgroup -S appgroup && adduser -S appuser -G appgroup

WORKDIR /app
COPY app/main.py .
RUN chown -R appuser:appgroup /app

USER appuser

ENV MODE=stable
ENV APP_VERSION=1.0.0
ENV APP_PORT=3000

EXPOSE 3000

HEALTHCHECK --interval=10s --timeout=5s --start-period=15s --retries=5 \
  CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:3000/healthz', timeout=4)" || exit 1

CMD ["python", "main.py"]

Why Alpine?

python:3.12-alpine is approximately 60MB. python:3.12 (Debian-based) is approximately 1GB. The task requires images under 300MB. Alpine gets us there with room to spare.

Why non-root?

The addgroup / adduser pattern is a security baseline. If someone exploits a vulnerability in the app, they get a user with zero privileges — no ability to write to system directories, install packages, or escalate. Running as root inside a container means a container escape gives the attacker root on the host.

The healthcheck evolution

The Dockerfile healthcheck went through several iterations during development. The original:

# BROKEN — env vars don't expand in CMD array form
HEALTHCHECK CMD wget -qO- http://localhost:${APP_PORT}/healthz || exit 1

${APP_PORT} doesn't expand inside the Dockerfile HEALTHCHECK CMD array — it's evaluated at build time, not runtime, so it literally tries to connect to http://localhost:${APP_PORT}. Fixed by hardcoding:

# BROKEN on WSL2 — localhost doesn't resolve inside Alpine healthcheck context
HEALTHCHECK CMD wget -qO- http://localhost:3000/healthz || exit 1

This also failed on WSL2 + Docker Desktop. The wget inside Alpine couldn't resolve localhost to 127.0.0.1 in the healthcheck execution context (a known WSL2 networking quirk). Switching to 127.0.0.1 still failed because of how Docker Desktop on WSL2 handles the network namespace for healthcheck processes.

The final working solution:

# WORKS — uses Python's urllib, no external tool, no DNS resolution
HEALTHCHECK --interval=10s --timeout=5s --start-period=15s --retries=5 \
  CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:3000/healthz', timeout=4)" || exit 1

Using Python's own urllib sidesteps the wget/DNS issue entirely. Python's socket layer handles 127.0.0.1 directly without going through the system resolver.

Component 4: The Templates

Templates are the bridge between the manifest and the generated configs. They contain placeholders in {{ key }} format that the CLI replaces with real values.

nginx.conf.tmpl

upstream app_backend {
    server app:{{ service_port }};
    keepalive 32;
}

log_format swiftdeploy '$time_iso8601 | $status | ${request_time}s | $upstream_addr | $request';

server {
    listen {{ nginx_port }};
    server_name _;

    access_log /var/log/nginx/access.log swiftdeploy;

    proxy_connect_timeout {{ proxy_timeout }}s;
    proxy_send_timeout {{ proxy_timeout }}s;
    proxy_read_timeout {{ proxy_timeout }}s;

    add_header X-Deployed-By swiftdeploy always;
    proxy_pass_header X-Mode;

    location / {
        proxy_pass http://app_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    error_page 502 = @error502;
    error_page 503 = @error503;
    error_page 504 = @error504;

    location @error502 {
        default_type application/json;
        add_header X-Deployed-By swiftdeploy always;
        return 502 '{"error":"Bad Gateway","code":502,"service":"app","contact":"{{ contact }}"}';
    }
    # ... 503, 504 same pattern
}

Design decisions worth noting:

Custom log format: $time_iso8601 | $status | ${request_time}s | $upstream_addr | $request gives you timestamp, HTTP status, response time in seconds, the upstream container IP, and the full request line — everything you need for debugging in one line.
JSON error bodies: Rather than nginx's default HTML error pages, we return structured JSON. This is essential for APIs — clients expect JSON and need machine-readable error codes.
proxy_pass_header X-Mode: nginx strips most custom headers by default. This directive explicitly forwards the X-Mode: canary header from the upstream app through to the client, so callers can identify which mode they're talking to.
keepalive 32: keeps 32 persistent connections to the upstream, reducing connection overhead under load.

docker-compose.yml.tmpl

services:
  app:
    image: {{ service_image }}
    container_name: swiftdeploy-app
    environment:
      MODE: "{{ mode }}"
      APP_VERSION: "{{ version }}"
      APP_PORT: "{{ service_port }}"
    networks:
      - {{ network_name }}
    volumes:
      - {{ log_volume }}:/app/logs
    restart: {{ restart_policy }}
    user: "appuser"
    cap_drop:
      - NET_ADMIN
      - SYS_ADMIN
    healthcheck:
      test: ["CMD-SHELL", "python -c 'import urllib.request; urllib.request.urlopen(\"http://127.0.0.1:3000/healthz\", timeout=4)'"]
      interval: 10s
      timeout: 5s
      start_period: 15s
      retries: 5
    expose:
      - "{{ service_port }}"

  nginx:
    image: {{ nginx_image }}
    container_name: swiftdeploy-nginx
    ports:
      - "{{ nginx_port }}:{{ nginx_port }}"
    volumes:
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
      - {{ log_volume }}:/var/log/nginx
    networks:
      - {{ network_name }}
    restart: {{ restart_policy }}
    depends_on:
      app:
        condition: service_healthy

The expose vs ports distinction is critical:

expose: ["3000"] — makes port 3000 reachable between containers on the same Docker network. Not published to the host.
ports: ["8080:8080"] — publishes port 8080 to the host machine and the outside world.

The app uses expose only. There is no way to reach port 3000 from outside Docker. All traffic must enter through nginx on 8080.

depends_on: condition: service_healthy means nginx won't start until the app container's healthcheck passes. This prevents nginx from starting and immediately returning 502s because the upstream isn't ready yet.

Component 5: The CLI

The swiftdeploy CLI is a single Python script with five subcommands. Here's how each works internally.

The template engine

def render_template(tmpl_path, context):
    with open(tmpl_path) as f:
        content = f.read()
    for key, val in context.items():
        content = content.replace("{{ " + key + " }}", str(val))
    return content

Five lines. No Jinja2. No external library. Simple string replacement. This is intentional — the templates are straightforward enough that a minimal custom engine is cleaner than pulling in a dependency.

The context builder

def build_context(manifest):
    svc = manifest["services"]
    ngx = manifest["nginx"]
    net = manifest["network"]
    return {
        "service_image":  svc["image"],
        "service_port":   svc["port"],
        "mode":           svc.get("mode", "stable"),
        "version":        svc.get("version", "1.0.0"),
        "restart_policy": svc.get("restart_policy", "unless-stopped"),
        "log_volume":     svc.get("log_volume", "swiftdeploy-logs"),
        "nginx_image":    ngx["image"],
        "nginx_port":     ngx["port"],
        "proxy_timeout":  ngx.get("proxy_timeout", 30),
        "network_name":   net["name"],
        "network_driver": net["driver_type"],
        "contact":        manifest.get("contact", "ops@swiftdeploy.local"),
    }

This translates the nested YAML structure into the flat dictionary that matches the {{ placeholders }} in the templates. It's the glue layer between manifest and generated configs.

`init` subcommand

def cmd_init():
    manifest = load_manifest()
    ctx = build_context(manifest)

    nginx_conf = render_template(NGINX_TMPL, ctx)
    with open(NGINX_OUT, "w") as f:
        f.write(nginx_conf)

    compose_conf = render_template(COMPOSE_TMPL, ctx)
    with open(COMPOSE_OUT, "w") as f:
        f.write(compose_conf)

Straightforward: load manifest → build context → render both templates → write files. The grader deletes the generated files and re-runs this to verify regeneration.

`validate` subcommand — 5 pre-flight checks

# Check 1: manifest.yaml exists and is valid YAML
try:
    manifest = load_manifest()
    ok("manifest.yaml found and parsed successfully")
except yaml.YAMLError as e:
    fail(f"Invalid YAML: {e}")

# Check 2: required fields present and non-empty
required = {
    "services.image": manifest.get("services", {}).get("image"),
    "services.port":  manifest.get("services", {}).get("port"),
    "nginx.image":    manifest.get("nginx", {}).get("image"),
    "nginx.port":     manifest.get("nginx", {}).get("port"),
    "network.name":   manifest.get("network", {}).get("name"),
    "network.driver_type": manifest.get("network", {}).get("driver_type"),
}

# Check 3: Docker image exists locally
result = run(f"docker image inspect {image}", capture=True, check=False)
# exit code 0 = found, non-zero = not found

# Check 4: Nginx port not already bound
result = run(f"ss -tlnp | grep ':{nginx_port} '", capture=True, check=False)
# stdout non-empty = port in use

# Check 5: nginx.conf syntactically valid
result = subprocess.run(
    ["docker", "run", "--rm",
     "-v", f"{test_conf_path}:/etc/nginx/conf.d/default.conf:ro",
     "nginx:latest", "nginx", "-t"],
    capture_output=True, text=True
)
combined = result.stdout + result.stderr
if "successful" in combined:
    ok("nginx.conf syntax is valid")

Check 5 is the most interesting. Running nginx -t in an isolated container is elegant — it validates syntax without needing nginx installed on the host. However, we hit a complication: in an isolated container, app:3000 (the upstream hostname) can't be resolved because there's no Docker network. nginx refuses to start if it can't resolve upstream hostnames, even for a syntax check.

The fix: before handing the config to the test container, swap server app: with server 127.0.0.1: in a temporary copy. 127.0.0.1 always resolves, so nginx validates the rest of the syntax (listen ports, timeouts, location blocks, error pages) correctly. The actual nginx.conf on disk is untouched.

`deploy` subcommand

def cmd_deploy():
    cmd_init()
    run(compose_cmd("up -d --build"))

    deadline = time.time() + 60
    healthy = False
    while time.time() < deadline:
        try:
            url = f"http://localhost:{nginx_port}/healthz"
            with urllib.request.urlopen(url, timeout=3) as resp:
                body = json.loads(resp.read())
                if body.get("status") == "ok":
                    healthy = True
                    break
        except Exception:
            pass
        time.sleep(2)

    if not healthy:
        fail("Health checks did not pass within 60 seconds")
        sys.exit(1)

The polling loop is the key part. Containers don't start instantly. docker compose up -d returns as soon as the containers are created, not when they're healthy. The loop hits /healthz through nginx every 2 seconds for up to 60 seconds. Connection refused, timeout, bad JSON — all exceptions are caught and ignored. Only {"status": "ok"} breaks the loop successfully.

`promote` subcommand

def cmd_promote(target_mode):
    # 1. Update manifest in-place using regex
    with open(MANIFEST_PATH) as f:
        content = f.read()
    content = re.sub(r"(mode:\s*)(\S+)", f"\\g<1>{target_mode}", content, count=1)
    with open(MANIFEST_PATH, "w") as f:
        f.write(content)

    # 2. Regenerate docker-compose.yml only
    manifest = load_manifest()
    ctx = build_context(manifest)
    compose_conf = render_template(COMPOSE_TMPL, ctx)
    with open(COMPOSE_OUT, "w") as f:
        f.write(compose_conf)

    # 3. Restart app container only — nginx stays up
    run(compose_cmd("up -d --no-deps app"))

    # 4. Confirm mode via /healthz
    deadline = time.time() + 30
    while time.time() < deadline:
        try:
            with urllib.request.urlopen(url, timeout=3) as resp:
                body = json.loads(resp.read())
                if body.get("mode") == target_mode and body.get("status") == "ok":
                    confirmed = True
                    break
        except Exception:
            pass
        time.sleep(2)

--no-deps in docker compose up -d --no-deps app is the rolling restart mechanism. It tells Compose to restart only the app service without touching nginx. Since nginx is already running and healthy, there's zero downtime at the proxy level — nginx continues serving requests while the app container restarts with the new mode.

The regex re.sub(r"(mode:\s*)(\S+)", f"\\g<1>{target_mode}", content, count=1) uses a backreference \\g<1> to preserve the mode: prefix and only replace the value. count=1 ensures only the first occurrence is replaced.

The Debugging Saga: WSL2 + Docker Desktop Healthchecks

This section documents the most painful part of the build — a cascade of healthcheck failures that took multiple debugging rounds to resolve.

Failure 1: `${APP_PORT}` not expanding

wget: can't connect to remote host: Connection refused

Root cause: environment variables don't expand in Dockerfile HEALTHCHECK CMD at runtime. ${APP_PORT} was being passed literally to wget. Fix: hardcode 3000.

Failure 2: `localhost` not resolving in Alpine

wget: can't connect to remote host: Connection refused

Same error, different cause. Inside the Alpine container's healthcheck execution context on WSL2 + Docker Desktop, localhost wasn't resolving to 127.0.0.1. Fix: use 127.0.0.1 explicitly.

Failure 3: `wget` still failing with `127.0.0.1`

wget: can't connect to remote host: Connection refused

Confirmed the server was listening:

docker exec swiftdeploy-app ss -tlnp
# tcp LISTEN 0.0.0.0:3000

docker exec swiftdeploy-app python -c "import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:3000/healthz').read())"
# b'{"status": "ok", ...}'

The server was reachable via docker exec but not from the healthcheck process. This is a known WSL2 + Docker Desktop network namespace issue — the healthcheck runs in a slightly different network context than docker exec. Fix: replace wget with Python's urllib:

CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:3000/healthz', timeout=4)" || exit 1

Failure 4: Docker cache serving old image

After fixing the Dockerfile, the healthcheck was still running wget. The container was using a cached image layer. Fix:

docker rmi swift-deploy-1-node:latest
docker build --no-cache -t swift-deploy-1-node:latest .

Failure 5: docker-compose.yml overriding the Dockerfile healthcheck

Even after fixing the image, the container was still running wget. Discovery:

grep -A5 "healthcheck" docker-compose.yml
# test: ["CMD", "wget", "-qO-", "http://localhost:3000/healthz"]

The docker-compose.yml template had its own healthcheck definition that was overriding the Dockerfile's. Docker Compose healthcheck always wins over Dockerfile HEALTHCHECK. Fixed the template to use the Python urllib approach and correct YAML quoting:

healthcheck:
  test: ["CMD-SHELL", "python -c 'import urllib.request; urllib.request.urlopen(\"http://127.0.0.1:3000/healthz\", timeout=4)'"]
  interval: 10s
  timeout: 5s
  start_period: 15s
  retries: 5

Failure 6: YAML indentation error

The healthcheck block had 3 spaces of indentation instead of 4 — a single space difference that broke YAML parsing entirely:

   healthcheck:  # ← 3 spaces — YAML parse error

    healthcheck:  # ← 4 spaces — correct

After all six of these were resolved, the deploy succeeded:

✔ Container swiftdeploy-app    Healthy   6.8s
✔ Container swiftdeploy-nginx  Started   7.2s
✔ Stack is healthy! Listening on port 8080

Deployment Walkthrough

1. Build the image

docker build -t swift-deploy-1-node:latest .

2. Validate

./swiftdeploy validate

[1/5] manifest.yaml exists and is valid YAML
  ✔ manifest.yaml found and parsed successfully

[2/5] Required fields are present and non-empty
  ✔ services.image = swift-deploy-1-node:latest
  ✔ services.port = 3000
  ✔ nginx.image = nginx:latest
  ✔ nginx.port = 8080
  ✔ network.name = swiftdeploy-net
  ✔ network.driver_type = bridge

[3/5] Docker image exists locally
  ✔ Image found: swift-deploy-1-node:latest

[4/5] Nginx port not already bound on host
  ✔ Port 8080 is free

[5/5] Generated nginx.conf is syntactically valid
  ✔ nginx.conf syntax is valid

ALL CHECKS PASSED ✔

3. Deploy

./swiftdeploy deploy

✔ Container swiftdeploy-app    Healthy
✔ Container swiftdeploy-nginx  Started
✔ Stack is healthy! Listening on port 8080

deploy complete.

4. Test endpoints

curl http://localhost:8080/
# {"message": "Welcome to SwiftDeploy API", "mode": "stable", "version": "1.0.0", "timestamp": "..."}

curl http://localhost:8080/healthz
# {"status": "ok", "mode": "stable", "version": "1.0.0", "uptime_seconds": 12.4}

5. Promote to canary

./swiftdeploy promote canary
curl http://localhost:8080/healthz
# {"status": "ok", "mode": "canary", ...}

curl -I http://localhost:8080/
# X-Mode: canary
# X-Deployed-By: swiftdeploy

6. Test chaos (canary mode only)

# Slow mode — 3 second delay
curl -X POST http://localhost:8080/chaos \
  -H "Content-Type: application/json" \
  -d '{"mode": "slow", "duration": 3}'

time curl http://localhost:8080/
# real    0m3.012s

# Error mode — 50% 500 errors
curl -X POST http://localhost:8080/chaos \
  -H "Content-Type: application/json" \
  -d '{"mode": "error", "rate": 0.5}'

# Recover
curl -X POST http://localhost:8080/chaos \
  -H "Content-Type: application/json" \
  -d '{"mode": "recover"}'

7. View nginx access logs

docker logs swiftdeploy-nginx

2026-05-03T14:23:01+00:00 | 200 | 0.002s | 172.18.0.3:3000 | GET / HTTP/1.1
2026-05-03T14:23:05+00:00 | 200 | 0.001s | 172.18.0.3:3000 | GET /healthz HTTP/1.1
2026-05-03T14:23:12+00:00 | 500 | 0.001s | 172.18.0.3:3000 | GET / HTTP/1.1

8. Teardown

./swiftdeploy teardown --clean

Key Learnings

1. Docker Compose healthcheck overrides Dockerfile HEALTHCHECK. This is documented but easy to miss. If both exist, Compose wins. Always check your generated compose file when healthchecks misbehave.

2. WSL2 + Docker Desktop has a quirky network namespace for healthchecks. docker exec and HEALTHCHECK run in slightly different contexts. If something works via exec but not via healthcheck, it's almost always a network namespace or tool availability issue. Python's stdlib is more portable than wget in this environment.

3. --no-cache is essential when debugging Dockerfile changes. Docker's layer caching is aggressive. If you change a HEALTHCHECK line but the layers above it are cached, Docker will use the old healthcheck. Always docker rmi and --no-cache when debugging image-level issues.

4. expose vs ports in Docker Compose is a security boundary, not just documentation. expose is container-to-container only. ports is host-facing. Using expose for the app and ports only for nginx enforces the proxy pattern at the infrastructure level.

5. YAML indentation is unforgiving. A single space difference between 3 and 4 spaces of indentation produces a cryptic parse error. Always use a YAML linter or at minimum python3 -c "import yaml; yaml.safe_load(open('file.yml'))" to validate before deploying.

6. Declarative infrastructure pays off immediately. The grader deletes generated files and re-runs init — because we built it right, this is a non-issue. The manifest is always there, the templates are always there, and regeneration is instantaneous.

Conclusion

SwiftDeploy started as a task requirement and ended up being a genuinely useful mental model for how declarative infrastructure tools work. Tools like Terraform, Helm, and Pulumi are all variations of the same core idea: describe what you want, let the tool figure out how to get there.

Building this from scratch — the template engine, the CLI subcommands, the healthcheck polling loop, the rolling restart — gives you an appreciation for what those tools are doing under the hood at scale.

The debugging journey through six layers of healthcheck failures was frustrating in the moment but valuable in retrospect. Every failure taught something concrete about how Docker, WSL2, Alpine Linux, nginx, and Python's network stack interact.

The full source code is available at: https://github.com/AirFluke/hng-swiftdeploy

Built for HNG DevOps Internship — Stage 4A: SwiftDeploy

Tags: #devops #docker #nginx #python #infrastructure #hng

How I Built a Real-Time DDoS Detection Engine from Scratch (No Fail2Ban)

Abraham Acha — Sun, 26 Apr 2026 15:31:10 +0000

Assuming my boss walked in and said "we're getting hit with suspicious traffic and I need you to build something that detects and responds to it automatically," I had two choices: reach for Fail2Ban like everyone else, or build something I actually understood from the ground up.

I chose to build it from scratch. This post explains exactly how I did it — in plain English, with real code, and without assuming you've ever touched security tooling before.

By the end of this post you will understand:

What the system does and why it matters
How a sliding window works (and why it's not just "count requests per minute")
How the system learns what normal traffic looks like
How it decides something is an attack
How it uses iptables to cut off an attacker at the firewall level

Let's go.

What This Project Does and Why It Matters

We run a cloud storage platform built on Nextcloud. It's public-facing, which means anyone on the internet can send requests to it — including bots, scanners, and attackers.

Two types of attacks concern us most:

A single aggressive IP — one machine sending hundreds of requests per second, trying to brute-force logins, scrape data, or simply overwhelm the server. This is a classic DDoS from a single source.

A global traffic spike — thousands of different IPs each sending a small number of requests. No single IP looks suspicious, but the total volume is crushing the server. This is a distributed DDoS.

The tool I built watches every HTTP request in real time, learns what "normal" traffic looks like, and fires an automatic response the moment something deviates — whether it's a single aggressive IP or a global spike.

For a single aggressive IP: it adds an iptables firewall rule to drop every packet from that IP, sends a Slack alert, and automatically releases the ban on a backoff schedule (10 minutes, then 30, then 2 hours, then permanent for repeat offenders).

For a global spike: it sends a Slack alert (no single IP to block).

The whole thing runs as a daemon — a continuously running background process — alongside the Nextcloud stack in Docker. It never sleeps, never needs a cron job, and never relies on hardcoded thresholds.

Here's the architecture at a glance:

Internet → Nginx (reverse proxy) → Nextcloud
                ↓ writes JSON logs
         [HNG-nginx-logs volume]
                ↓ reads logs
         Python Daemon
           ├── monitor.py    (tail the log file)
           ├── baseline.py   (learn normal traffic)
           ├── detector.py   (spot anomalies)
           ├── blocker.py    (iptables + audit log)
           ├── unbanner.py   (auto-release bans)
           ├── notifier.py   (Slack alerts)
           └── dashboard.py  (live web UI)

Everything is wired together by main.py and configured through a single config.yaml file — no hardcoded numbers anywhere in the code.

Part 1: Reading the Traffic — The Log Monitor

Before we can detect anything, we need to see the traffic. Nginx is our reverse proxy — every HTTP request passes through it. We configure Nginx to write each request as a JSON object to a log file.

Here's the Nginx log format we defined:

log_format hng_json escape=json
    '{'
        '"source_ip":"$remote_addr",'
        '"timestamp":"$time_iso8601",'
        '"method":"$request_method",'
        '"path":"$uri",'
        '"status":$status,'
        '"response_size":$body_bytes_sent,'
        '"request_time":$request_time,'
        '"http_user_agent":"$http_user_agent"'
    '}';

access_log /var/log/nginx/hng-access.log hng_json;

Every request now produces a line like this in the log file:

{
  "source_ip": "1.2.3.4",
  "timestamp": "2026-04-26T14:08:21+00:00",
  "method": "GET",
  "path": "/index.php/login",
  "status": 401,
  "response_size": 1842,
  "request_time": 0.043,
  "http_user_agent": "Mozilla/5.0..."
}

Now our Python daemon needs to read these lines as they appear — in real time, without missing any, and without crashing if the log file gets rotated. This is called "tailing" a file, like tail -f in Linux.

Here's how monitor.py does it:

import os
import time
import json
import queue

def parse_line(raw: str) -> dict | None:
    """Parse one JSON log line. Returns None if malformed."""
    raw = raw.strip()
    if not raw:
        return None
    try:
        entry = json.loads(raw)
    except json.JSONDecodeError:
        return None  # drop malformed lines silently

    # Make sure all required fields are present
    required = {"source_ip", "timestamp", "method", "path", "status", "response_size"}
    if required - entry.keys():
        return None

    # Coerce types and add a wall-clock timestamp for window math
    entry["status"] = int(entry["status"])
    entry["response_size"] = int(entry["response_size"])
    entry["_parsed_at"] = time.time()
    return entry


class LogMonitor:
    def run(self):
        fh = open("/var/log/nginx/hng-access.log", "r")
        fh.seek(0, 2)  # jump to END of file — don't replay old traffic

        current_inode = os.stat(fh.name).st_ino

        while True:
            line = fh.readline()
            if line:
                entry = parse_line(line)
                if entry:
                    self.out_queue.put(entry)  # send downstream
            else:
                # Check if the file was rotated (inode changed)
                new_inode = os.stat(fh.name).st_ino
                if new_inode != current_inode:
                    fh.close()
                    fh = open(fh.name, "r")
                    current_inode = new_inode
                else:
                    time.sleep(0.1)  # no new data, yield CPU briefly

Three things worth understanding here:

fh.seek(0, 2) — When we start the daemon, the log file already has entries from before we launched. We don't want to process all of those as if they're live traffic. Seeking to position 2 (the end) means we only see new lines written after the daemon starts.

Inode check — Log files get rotated (renamed and replaced with a fresh empty file) periodically. If we don't detect this, we'd keep reading the old file and miss all new traffic. The inode (a unique file identifier in Linux) changes when the file is replaced, so we check it on every "no new data" cycle.

The queue — We put parsed entries into a queue.Queue. This decouples reading from processing. The monitor thread just reads and queues; the main thread drains the queue and feeds entries to the baseline and detector. If the detector is temporarily slow, entries buffer in the queue rather than being dropped.

Part 2: How the Sliding Window Works

This is the core data structure of the entire detection system. Get this right and everything else falls into place.

The wrong way: per-minute counters

The naive approach is to count requests per minute:

# WRONG — this is not a sliding window
requests_this_minute = 0
last_minute = int(time.time() / 60)

def on_request():
    current_minute = int(time.time() / 60)
    if current_minute != last_minute:
        requests_this_minute = 0  # reset
        last_minute = current_minute
    requests_this_minute += 1

The problem: imagine an attacker sends 1000 requests at 14:00:59 and another 1000 at 14:01:01. The per-minute counter resets at 14:01:00, so each minute only shows 1000 requests. But in reality, 2000 requests arrived in a 2-second window — a massive spike that the counter completely misses.

The right way: a deque of timestamps

A sliding window stores the exact timestamp of every request in a collections.deque. When you want the current rate, you evict old entries and count what remains.

from collections import deque
import time

# One deque per IP, one global
ip_window = deque()    # stores timestamps of requests from this IP
WINDOW = 60            # seconds

def on_request(ip: str, now: float):
    # Step 1: Add this request's timestamp
    ip_window.append(now)

    # Step 2: Evict entries older than 60 seconds from the LEFT
    cutoff = now - WINDOW
    while ip_window and ip_window[0] < cutoff:
        ip_window.popleft()   # O(1) — this is why we use deque, not list

    # Step 3: Rate = requests still in window / window size
    rate = len(ip_window) / WINDOW
    return rate  # requests per second over the last 60 seconds

Let's trace through a real example:

Time 14:00:00  → request arrives → deque: [14:00:00]             rate = 1/60 = 0.017 req/s
Time 14:00:01  → request arrives → deque: [14:00:00, 14:00:01]   rate = 2/60 = 0.033 req/s
...normal traffic...
Time 14:01:30  → attacker starts sending 100 req/s
Time 14:01:31  → deque has 100 new entries + a few old ones
               → evict everything before 14:00:31
               → 100 entries remain → rate = 100/60 = 1.67 req/s  ← anomaly detected

The window "slides" forward with real time. Old entries fall off the left; new entries pile up on the right. The rate you get is always "how many requests from this IP in the last 60 seconds."

Why deque and not list?

list.pop(0) is O(n) — it has to shift every element one position to the left every time you evict. With thousands of entries under a DDoS, this becomes very slow.

deque.popleft() is O(1) — it just moves a pointer. No matter how many entries are in the deque, eviction is instant.

Here's the actual implementation from our detector.py:

from collections import deque, defaultdict
from dataclasses import dataclass, field

@dataclass
class IPState:
    """All per-IP sliding window state."""
    timestamps: deque = field(default_factory=deque)  # request timestamps
    error_ts: deque = field(default_factory=deque)     # 4xx/5xx timestamps

# One IPState per IP address, created on first request
ip_states = defaultdict(IPState)

# One global window for all traffic combined
global_ts: deque = deque()

def _evict(dq: deque, window: float, now: float):
    """Remove timestamps older than `window` seconds from the left."""
    cutoff = now - window
    while dq and dq[0] < cutoff:
        dq.popleft()

def _rate(dq: deque, window: float) -> float:
    """Requests per second over the window."""
    return len(dq) / window

def process(entry: dict):
    ip = entry["source_ip"]
    now = entry["_parsed_at"]

    state = ip_states[ip]

    # Update per-IP window
    state.timestamps.append(now)
    _evict(state.timestamps, 60, now)
    ip_rate = _rate(state.timestamps, 60)

    # Update global window
    global_ts.append(now)
    _evict(global_ts, 60, now)
    global_rate = _rate(global_ts, 60)

We maintain two separate windows: one per IP (so we can detect individual attackers) and one global (so we can detect distributed attacks from many IPs). Both are updated on every single request.

Part 3: How the Baseline Learns from Traffic

Detecting anomalies requires knowing what normal looks like. If normal traffic is 50 req/s, then 100 req/s is suspicious. But if normal traffic is 5 req/s, then even 15 req/s might be an attack.

The key insight: normal changes over time. A file-sharing platform sees more traffic during business hours than at 3am. A global service sees more traffic from Asia during Asian business hours. You cannot hardcode a threshold — you have to learn it.

Per-second bucketing

Our baseline engine counts requests per second and stores those counts in a rolling 30-minute window:

from collections import deque
import math
import time
from datetime import datetime

# Stores one count per second, up to 30 minutes worth (1800 buckets)
second_buckets = deque(maxlen=1800)

current_second = int(time.time())
current_count = 0.0

def record(entry: dict):
    global current_second, current_count

    now = entry["_parsed_at"]
    bucket_sec = int(now)

    # When the clock second changes, flush the completed bucket
    if bucket_sec > current_second:
        # Fill any empty seconds (gaps where no traffic arrived)
        for s in range(current_second, bucket_sec):
            second_buckets.append(current_count)
            current_count = 0.0
        current_second = bucket_sec

    current_count += 1.0

So if 47 requests arrived in the second ending at 14:05:00, the deque gets 47.0 appended. If the next second was quiet (zero requests), it gets 0.0. The deque holds up to 1800 of these values — 30 minutes of history.

Computing mean and standard deviation

Every 60 seconds, we recompute the baseline from whatever is in the bucket deque:

def _recalculate():
    samples = list(second_buckets)
    n = len(samples)

    if n < 30:
        return  # not enough data yet, keep the floor values

    # Mean: average requests per second over the last 30 minutes
    mean = sum(samples) / n

    # Standard deviation: how much does traffic vary?
    variance = sum((x - mean) ** 2 for x in samples) / (n - 1)
    stddev = math.sqrt(variance)

    # Apply floor values to prevent false positives during quiet periods
    mean = max(mean, 1.0)      # never let mean drop below 1 req/s
    stddev = max(stddev, 0.5)  # never let stddev drop below 0.5

    effective_mean = mean
    effective_stddev = stddev

Why do we need floor values?

Imagine the server is quiet at 3am with 0.1 req/s average. Without floor values:

mean = 0.1, stddev = 0.05
A burst to just 0.25 req/s gives z-score = (0.25 - 0.1) / 0.05 = 3.0 → false alarm!

With floor_mean = 1.0:

The denominator stays reasonable
You need a genuine spike to trigger detection

Per-hour preference

Traffic patterns repeat by hour of day. We also maintain 24 separate per-hour slot deques:

hour_slots = {h: deque(maxlen=1800) for h in range(24)}

# When flushing a bucket, also file it into the current hour's slot
hour = datetime.fromtimestamp(current_second).hour
hour_slots[hour].append(current_count)

At recalculation time, if the current hour's slot has at least 30 samples, we use it instead of the 30-minute window. This means 9am traffic is compared to yesterday's 9am baseline, not to the 8:30am baseline — much more accurate.

Part 4: How the Detection Logic Makes a Decision

We have a rate (from the sliding window) and a baseline (mean and stddev). How do we decide if the rate is an attack?

Z-score: how many standard deviations above normal?

The z-score is a way of asking "how unusual is this number compared to what we normally see?"

z-score = (current_rate - baseline_mean) / baseline_stddev

If the baseline is mean=5 req/s, stddev=2:

Current rate 7 req/s → z = (7-5)/2 = 1.0 → normal variation
Current rate 11 req/s → z = (11-5)/2 = 3.0 → suspicious
Current rate 25 req/s → z = (25-5)/2 = 10.0 → attack

We flag anything with z-score > 3.0. In a normal distribution, only 0.13% of values naturally exceed z=3. So a false positive rate of 0.13% is acceptable.

Multiplier: absolute rate check

Z-score alone has a weakness: if traffic is very consistent (low stddev), even a modest real attack might not produce a high z-score. So we add a second condition:

if rate > 5 × mean → flag it

If normal is 5 req/s and someone hits 26 req/s, that's more than 5× — flag it regardless of stddev.

Either condition firing triggers the response. Here's the actual detection code:

def _zscore(rate: float, mean: float, stddev: float) -> float:
    if stddev == 0:
        return 0.0
    return (rate - mean) / stddev

def check_anomaly(ip_rate, mean, stddev):
    z_thresh = 3.0   # loaded from config.yaml
    m_thresh = 5.0   # loaded from config.yaml

    ip_z = _zscore(ip_rate, mean, stddev)

    # Either condition triggers the alert
    is_anomaly = (ip_z > z_thresh) or (ip_rate > m_thresh * mean and mean > 0)
    return is_anomaly, ip_z

Error surge tightening

There's a third detection mode for slow attacks. An attacker doing credential stuffing (trying username/password combinations) won't necessarily send huge traffic volumes — but they'll generate lots of 401 Unauthorized responses.

We track error rates separately:

# If this IP's 4xx/5xx rate is 3× the baseline error rate,
# tighten the thresholds for this IP
if err_rate >= 3.0 * baseline_error_mean:
    z_thresh = 2.0   # lower bar → easier to trigger
    m_thresh = 3.0   # lower bar → easier to trigger
    state.tightened = True

This catches slow, stealthy attacks that would otherwise fly under the radar.

Putting it all together

Here's the full process() function that runs on every log line:

def process(self, entry: dict):
    ip = entry["source_ip"]
    now = entry["_parsed_at"]
    status = entry.get("status", 200)

    mean, stddev = self._baseline.get_baseline()

    with self._lock:
        state = self._ip_states[ip]

        # 1. Update sliding windows
        state.timestamps.append(now)
        self._evict(state.timestamps, 60, now)

        self._global_ts.append(now)
        self._evict(self._global_ts, 60, now)

        ip_rate = len(state.timestamps) / 60
        global_rate = len(self._global_ts) / 60

        # 2. Check error surge → tighten thresholds if needed
        if status >= 400:
            state.error_ts.append(now)
        self._evict(state.error_ts, 60, now)
        err_rate = len(state.error_ts) / 60

        z_thresh = 2.0 if state.tightened else 3.0
        m_thresh = 3.0 if state.tightened else 5.0

        # 3. Score and decide
        ip_z = (ip_rate - mean) / stddev if stddev > 0 else 0
        ip_anomaly = (ip_z > z_thresh) or (ip_rate > m_thresh * mean)

        # 4. Fire callback if anomalous (with 10-second cooldown)
        if ip_anomaly:
            last = self._last_fired.get(ip, 0)
            if now - last > 10:
                self._last_fired[ip] = now
                self.on_ip_anomaly(ip, {"ip_rate": ip_rate, "zscore": ip_z, ...})

        # 5. Check global anomaly separately
        g_z = (global_rate - mean) / stddev if stddev > 0 else 0
        if g_z > 3.0 or global_rate > 5.0 * mean:
            if now - self._global_last_fired > 30:
                self._global_last_fired = now
                self.on_global_anomaly({"global_rate": global_rate, ...})

When on_ip_anomaly fires, it calls the blocker. When on_global_anomaly fires, it calls the notifier directly (no ban — you can't ban the whole internet).

Part 5: How iptables Blocks an IP

iptables is the Linux kernel's firewall. It processes every network packet that enters or leaves your machine and applies rules to decide what to do with each one.

Think of it like a bouncer at a club. The bouncer has a list of rules:

Allow anyone with a VIP pass (ACCEPT)
Drop anyone on the blacklist (DROP)
Default: let everyone in (ACCEPT)

When we detect an attack from IP 1.2.3.4, we add a rule to the top of the INPUT chain (the chain that handles incoming packets):

iptables -I INPUT -s 1.2.3.4 -j DROP

Breaking this down:

-I INPUT — Insert at the top of the INPUT chain (before other rules)
-s 1.2.3.4 — Match packets with source IP 1.2.3.4
-j DROP — Silently drop matching packets (no response sent to attacker)

The -I (insert) rather than -A (append) is important. Rules are checked top-to-bottom. By inserting at the top, our DROP rule is checked before any ACCEPT rules — the attacker is blocked immediately.

You can verify the rule was added:

$ iptables -L INPUT -n -v
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
DROP       all  --  1.2.3.4             0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0           0.0.0.0/0  state RELATED,ESTABLISHED
...

Here's the Python code that issues this command:

import subprocess

def _iptables_add(ip: str):
    """Add a DROP rule for this IP. Check first to avoid duplicates."""
    # Check if rule already exists
    check = subprocess.run(
        ["iptables", "-C", "INPUT", "-s", ip, "-j", "DROP"],
        capture_output=True
    )
    if check.returncode == 0:
        return  # already banned, nothing to do

    # Add the rule
    subprocess.run(
        ["iptables", "-I", "INPUT", "-s", ip, "-j", "DROP"],
        check=True,
        capture_output=True
    )

def _iptables_remove(ip: str):
    """Remove the DROP rule. Loop in case it was added more than once."""
    while True:
        result = subprocess.run(
            ["iptables", "-D", "INPUT", "-s", ip, "-j", "DROP"],
            capture_output=True
        )
        if result.returncode != 0:
            break  # no more rules for this IP

The backoff schedule

We don't ban forever on the first offence. The ban duration follows a backoff schedule:

Offence	Duration
1st ban	10 minutes
2nd ban	30 minutes
3rd ban	2 hours
4th+ ban	Permanent

A background thread (the unbanner) wakes up every 30 seconds and checks whether any bans have expired:

import threading
import time

class UnbannerThread(threading.Thread):
    def run(self):
        while not self._stop.wait(30):  # check every 30 seconds
            expired = self._blocker.get_expired_bans()
            for ip in expired:
                record = self._blocker.unban(ip)
                if record:
                    self._notifier.send_unban(record)  # Slack alert

def get_expired_bans(self) -> list[str]:
    now = datetime.utcnow()
    return [
        ip for ip, rec in self._bans.items()
        if rec.expires_at is not None and now >= rec.expires_at
    ]

When an IP is unbanned, iptables -D INPUT -s <ip> -j DROP removes the rule, and a Slack notification goes out so the team knows the ban was lifted.

Part 6: Slack Alerts

Every ban, unban, and global anomaly sends a Slack message. We use Slack's Incoming Webhooks — a URL you POST JSON to, and it appears as a message in your channel.

import requests
import json

def send_ban(ip: str, info: dict, record):
    text = (
        f":rotating_light: *IP BANNED* — `{ip}`\n"
        f"*Rate:* `{info['ip_rate']:.2f} req/s`  "
        f"*Baseline:* `{info['mean']:.2f}`  "
        f"*Z-score:* `{info['zscore']:.2f}`\n"
        f"*Duration:* `{record.duration_label}`  (ban #{record.ban_count})\n"
        f"*Time:* {timestamp}"
    )
    requests.post(
        webhook_url,
        data=json.dumps({"text": text}),
        headers={"Content-Type": "application/json"},
        timeout=5
    )

Critically, Slack calls happen in a thread pool — they never block the main detection loop. If Slack is slow or down, we don't miss detecting the next attack:

from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=4)

def send_ban_async(ip, info, record):
    executor.submit(_post_to_slack, ip, info, record)

Part 7: The Live Dashboard

The dashboard is a Flask web app that serves a single HTML page which polls /api/metrics every 3 seconds via JavaScript fetch().

from flask import Flask, jsonify
import psutil

app = Flask(__name__)

@app.route("/api/metrics")
def metrics():
    snap = detector.get_snapshot()
    mean, stddev = baseline.get_baseline()
    banned = blocker.get_banned_ips()

    return jsonify({
        "uptime": get_uptime_string(),
        "global_rate": snap["global_rate"],
        "top_ips": snap["top_ips"],
        "mean": round(mean, 3),
        "stddev": round(stddev, 3),
        "banned": [{"ip": r.ip, "duration": r.duration_label, ...} for r in banned],
        "cpu_pct": psutil.cpu_percent(),
        "mem_pct": psutil.virtual_memory().percent,
        "audit_tail": read_last_20_audit_lines(),
    })

The frontend JavaScript refreshes this every 3 seconds and updates the DOM without a full page reload:

async function refresh() {
    const r = await fetch('/api/metrics');
    const d = await r.json();

    document.getElementById('global-rate').textContent = d.global_rate.toFixed(2) + ' req/s';
    document.getElementById('mean').textContent = d.mean;

    // Color the rate red if it's approaching the anomaly threshold
    const rateEl = document.getElementById('global-rate');
    rateEl.style.color = d.global_rate > d.mean * 3 ? '#f87171' : '#4ade80';

    // Render banned IPs table
    const bannedRows = d.banned.map(b =>
        `<tr><td>${b.ip}</td><td>${b.duration}</td><td>${b.banned_at}</td></tr>`
    ).join('');
    document.getElementById('banned-table').innerHTML = bannedRows || '<tr><td colspan="3">No active bans ✓</td></tr>';
}

setInterval(refresh, 3000);
refresh(); // immediate first load

Part 8: The Audit Log

Every significant event is written to a structured audit log in a format designed to be both human-readable and grep-able:

[2026-04-26T14:08:21Z] BAN ip=1.2.3.4 | condition=zscore=4.21 rate=87.3req/s mean=5.2 | rate=87.300 | baseline=5.200 | duration=10m
[2026-04-26T14:18:22Z] UNBAN ip=1.2.3.4 | condition=zscore=4.21 rate=87.3req/s mean=5.2 | rate=87.300 | baseline=5.200 | duration=10m
[2026-04-26T14:07:06Z] BASELINE_RECALC ip=N/A | condition=recalculation | rate=5.200 | baseline=1.100 | duration=N/A

You can search these logs:

# All bans today
grep "BAN " /var/log/detector/audit.log | grep "2026-04-26"

# All recalculations (to see baseline evolution)
grep "BASELINE_RECALC" /var/log/detector/audit.log

# How often was 1.2.3.4 banned?
grep "ip=1.2.3.4" /var/log/detector/audit.log | grep "^.*BAN "

Putting It All Together

Here's main.py — the entry point that wires every component together:

import queue
import signal
import threading

# 1. Build all components
log_queue = queue.Queue(maxsize=100_000)
monitor = LogMonitor(log_path, log_queue)
baseline = BaselineEngine(cfg)
blocker = BlockerEngine(cfg, audit_path)
notifier = NotifierEngine(cfg)
detector = DetectorEngine(cfg, baseline)

# 2. Wire the anomaly callbacks
def on_ip_anomaly(ip, info):
    if blocker.is_banned(ip):
        return  # already banned, skip
    record = blocker.ban(ip, condition=..., rate=info["ip_rate"], baseline=info["mean"])
    notifier.send_ban(ip, info, record)

def on_global_anomaly(info):
    notifier.send_global_anomaly(info)

detector.on_ip_anomaly = on_ip_anomaly
detector.on_global_anomaly = on_global_anomaly

# 3. Start background threads
monitor.start()
UnbannerThread(blocker, notifier).start()
run_dashboard(host="0.0.0.0", port=8080)

# 4. Main loop — drain queue, feed baseline and detector
while True:
    try:
        entry = log_queue.get(timeout=0.05)
        baseline.record(entry)   # update statistics
        detector.process(entry)  # check for anomalies
    except queue.Empty:
        pass  # no new log lines, loop again

The design is intentionally simple: one hot path (queue drain → baseline → detector) runs in the main thread. All I/O — Slack posts, iptables calls, dashboard requests — happens in separate threads and never blocks detection.

What I Learned

Building this from scratch taught me things that using Fail2Ban never would have:

Deques are the right tool for sliding windows. The O(1) popleft makes a real difference when you're processing thousands of events per second. A list would have worked fine until the DDoS actually started — which is exactly when you need it to be fast.

Baselines must have floors. My first version had no floor values. It triggered constantly at night because the stddev got so low that normal morning traffic looked like an anomaly. One line of code (mean = max(mean, 1.0)) fixed it.

Per-hour slots matter more than you'd think. Without them, the 30-minute window picks up the tail of the previous hour's traffic and skews the baseline for the current hour. The hour slots give you a cleaner "what does this hour normally look like" answer.

iptables rules apply to the host kernel, not the container. The detector container has to run with network_mode: host and CAP_NET_ADMIN. If you run it in a bridge network and add iptables rules inside the container, those rules only affect traffic inside the container's network namespace — not the actual packets arriving at the server.

Thread safety is non-negotiable. The log monitor, the main loop, the Flask dashboard, the unbanner, and the Slack notifier all touch shared state. A threading.Lock() around every mutation took 20 minutes to add and saved hours of debugging mysterious state corruption.

Running It Yourself

The full source code is on GitHub: github.com/Airfluke/hng-detector

To run it on a fresh Ubuntu VPS:

# Install Docker
curl -fsSL https://get.docker.com | sh

# Clone and configure
git clone https://github.com/AirFluke/hng-detector.git
cd hng-detector
cp .env.example .env
nano .env  # add your Slack webhook URL and server IP

# Launch everything
docker compose up -d --build

# Watch the detector work
docker logs -f hng-detector

# Test it with a traffic burst
apt install apache2-utils
ab -n 2000 -c 150 http://localhost/

# Check iptables for the ban
iptables -L INPUT -n -v

The dashboard will be live at http://your-server-ip:8080.

Conclusion

Security tooling doesn't have to be a black box. The core ideas here — a deque-based sliding window, a rolling statistical baseline, z-score anomaly detection, and iptables for enforcement — are each individually simple. The power comes from wiring them together into a continuously running daemon that never sleeps.

If you're new to security tooling, I hope this post demystified what's happening under the hood of the tools you've always treated as magic. Go build something.

Found this useful? Share it with someone learning DevSecOps. Questions? Drop them in the comments.

Forem: Abraham Acha

SwiftDeploy: Building a Self-Writing Infrastructure Manager with Policy Enforcement — A Complete Technical Walkthrough

Table of Contents

The Problem We're Solving {#the-problem}

Architecture Overview {#architecture}

Stage 4A — The Engine {#stage-4a}

The Project Structure

The Manifest {#the-manifest}

The Python HTTP Service {#the-python-service}

Configuration from environment

Thread-safe chaos state

The three Stage 4A endpoints

The Dockerfile {#the-dockerfile}

The Templates {#the-templates}

The CLI — Five Stage 4A Subcommands {#the-cli}

The template engine

init — generates everything from the manifest

validate — five pre-flight checks

deploy — brings up the stack and blocks until healthy

promote — rolling restart with zero nginx downtime

The Two Deployment Modes {#deployment-modes}

Stage 4B — The Eyes and The Brain {#stage-4b}

Prometheus Metrics — The Eyes {#prometheus-metrics}

Tracking infrastructure

The five metrics

The /metrics output

OPA Policy Engine — The Brain {#opa-policy-engine}

Why OPA instead of if/else in the CLI?

data.json — thresholds live here, never hardcoded in Rego

infrastructure.rego — pre-deploy policy

canary.rego — pre-promote policy

OPA isolation — no leakage via nginx

The policy query function

Gated Lifecycle — The CLI Brain {#gated-lifecycle}

Pre-deploy check

Pre-promote check

P99 latency calculation from histogram

The Status Dashboard — The Eyes {#status-dashboard}

The Audit Trail — The Memory {#audit-trail}

The Debugging Sagas {#debugging}

Saga 1 — Six layers of healthcheck failure

Saga 2 — OPA Rego v1 syntax

Saga 3 — WSL2 path spaces breaking docker run

Full Deployment Walkthrough {#deployment}

Key Lessons Learned {#lessons}

Conclusion

SwiftDeploy: Building a Declarative Infrastructure Manager from Scratch; A Complete Technical Walkthrough

Introduction

The Architecture

The Project Structure

Component 1: The Manifest

What each field controls

Component 2: The Python HTTP Service

The server setup

Thread-safe chaos state

The request handler

The three routes

Component 3: The Dockerfile

Why Alpine?

Why non-root?

The healthcheck evolution

Component 4: The Templates

nginx.conf.tmpl

docker-compose.yml.tmpl

Component 5: The CLI

The template engine

The context builder

init subcommand

validate subcommand — 5 pre-flight checks

deploy subcommand

promote subcommand

The Debugging Saga: WSL2 + Docker Desktop Healthchecks

Failure 1: ${APP_PORT} not expanding

Failure 2: localhost not resolving in Alpine

Failure 3: wget still failing with 127.0.0.1

Failure 4: Docker cache serving old image

Failure 5: docker-compose.yml overriding the Dockerfile healthcheck

Failure 6: YAML indentation error

Deployment Walkthrough

1. Build the image

`init` — generates everything from the manifest

`validate` — five pre-flight checks

`deploy` — brings up the stack and blocks until healthy

`promote` — rolling restart with zero nginx downtime

`init` subcommand

`validate` subcommand — 5 pre-flight checks

`deploy` subcommand

`promote` subcommand

Failure 1: `${APP_PORT}` not expanding

Failure 2: `localhost` not resolving in Alpine

Failure 3: `wget` still failing with `127.0.0.1`