Forem: iapilgrim

Master the ECR Lifecycle: Automating Image Cleanup

iapilgrim — Mon, 23 Mar 2026 11:29:23 +0000

Why Lifecycle Policies?

Every time your CI/CD pipeline runs docker push, you're adding roughly 200MB–1GB of data to your AWS bill. Without a policy, that data sits there forever. Lifecycle policies allow you to define rules like:

"Keep only the last 10 images."
"Delete anything older than 14 days."
"Expire untagged images immediately."

The "Perfect" Policy for Dev/Staging

For most teams, the best balance between "safety" and "savings" is a two-rule policy.

1. The "Untagged" Rule (Priority 1)

When you push a new image with the same tag (like :latest), the old image becomes "untagged." These are orphaned layers that serve no purpose. Delete them after 24 hours.

2. The "Age" Rule (Priority 2)

Delete any image that hasn't been pushed in the last 30 days. This ensures that even if you stop a project, its storage costs don't haunt you for years.

The JSON Configuration

{
  "rules": [
    {
      "rulePriority": 1,
      "description": "Cleanup orphaned/untagged images",
      "selection": {
        "tagStatus": "untagged",
        "countType": "sinceImagePushed",
        "countUnit": "days",
        "countNumber": 1
      },
      "action": { "type": "expire" }
    },
    {
      "rulePriority": 2,
      "description": "Delete images older than 30 days",
      "selection": {
        "tagStatus": "any",
        "countType": "sinceImagePushed",
        "countUnit": "days",
        "countNumber": 30
      },
      "action": { "type": "expire" }
    }
  ]
}

Critical Concepts: `sinceImagePushed` vs. `imageCountMoreThan`

sinceImagePushed: Best for time-based compliance. (e.g., "We only keep 30 days of history").
imageCountMoreThan: Best for storage predictability. (e.g., "I only ever want to pay for 10 images per repo").

How to Test Without Breaking Production

The biggest fear is deleting an image that a production server might need during an auto-scaling event.

The "Lifecycle Policy Preview" is your best friend.

Navigate to your ECR Repo -> Lifecycle Policy.
Click Actions -> Create Preview.
AWS will generate a list of exactly which images would be deleted if the policy ran right now. Check this list against your current production version before hitting save.

The 24-Hour Rule

Remember: ECR policies are not instantaneous. After you click "Save," AWS schedules a background task. It usually takes 24 hours for the images to actually disappear from your console and for the storage metrics to drop in CloudWatch.

Pro-Tip: Infrastructure as Code (Terraform)

Don't click around the console for 50 repos. Add this to your Terraform module:

resource "aws_ecr_lifecycle_policy" "default" {
  repository = aws_ecr_repository.app.name

  policy = <<EOF
{
    "rules": [
        {
            "rulePriority": 1,
            "description": "Keep last 30 images",
            "selection": {
                "tagStatus": "any",
                "countType": "imageCountMoreThan",
                "countNumber": 30
            },
            "action": {
                "type": "expire"
            }
        }
    ]
}
EOF
}

Deploying a Base Sepolia Node with Docker

iapilgrim — Sat, 21 Mar 2026 07:22:07 +0000

We're using QuickNode for Base Sepolia. This post is how to setup our own node.

1. Prerequisites

Docker & Docker Compose installed.
L1 RPC Endpoint: A synced Ethereum Sepolia node (e.g., Geth + Lighthouse).
L1 Beacon Endpoint: Required for post-Canyon/Ecotone consensus.
Hardware: Minimum 16GB RAM and 1.5TB+ NVMe SSD.

2. The Automated Setup Script

We use a wrapper script to handle directory creation, JWT generation, and repository patching. This ensures your local paths are correctly mapped into the Docker containers.

Save as setup-base-sepolia.sh:

#!/usr/bin/env bash
set -e

echo "🚀 Initializing Base Sepolia Setup..."

BASE_DIR="/node-data/testnet/base-sepolia"
REPO_DIR="/opt/base-node"

# 1. Create directory layout
echo "📁 Creating data directories..."
mkdir -p ${BASE_DIR}/{op-geth,op-node,shared}

# 2. Generate the JWT Secret (The "Handshake" Key)
if [ ! -f "${BASE_DIR}/shared/jwt.txt" ]; then
  echo "🔑 Generating 32-byte hex JWT..."
  # CRITICAL: No 0x prefix, no newlines.
  openssl rand -hex 32 | tr -d "\n" > ${BASE_DIR}/shared/jwt.txt
  chmod 644 ${BASE_DIR}/shared/jwt.txt
fi

# 3. Clone the official Base Node Repo
echo "📦 Cloning base-org/node..."
if [ ! -d "${REPO_DIR}" ]; then
  git clone https://github.com/base-org/node.git ${REPO_DIR}
fi

cd ${REPO_DIR}

# 4. Create .env.sepolia
echo "⚙️ Writing environment configuration..."
cat > .env.sepolia <<EOF
# L1 Endpoints (Replace with your actual L1 IPs)
OP_NODE_L1_ETH_RPC=http://testnet:8585
OP_NODE_L1_BEACON=http://testnet-lighthouse:5052

# L2 Execution & Auth
OP_NODE_L2_ENGINE_RPC=http://execution:8551
OP_NODE_L2_ENGINE_AUTH=jwt
OP_NODE_L2_ENGINE_JWT_SECRET=/shared/jwt.txt

# Network & Sync
OP_NODE_NETWORK=base-sepolia
OP_NODE_SYNC_MODE=snap
EOF

# 5. Patch docker-compose.yml for custom volumes
echo "🛠 Patching docker-compose volumes..."
# Map op-geth and op-node to host paths
sed -i 's|${HOST_DATA_DIR}:/data|/node-data/testnet/base-sepolia/op-geth:/data|g' docker-compose.yml
# Inject the shared JWT volume into both services
sed -i '/volumes:/a \      - /node-data/testnet/base-sepolia/shared:/shared' docker-compose.yml

# 6. Start the stack
echo "🚀 Starting containers..."
docker compose --env-file .env.sepolia up -d

3. Critical Configuration Check

For the node to function, your entrypoint scripts inside the repo must use the correct environment variables.

execution-entrypoint: Ensure --authrpc.jwtsecret="$OP_NODE_L2_ENGINE_JWT_SECRET" is set.
op-node-entrypoint: Ensure --l2.jwt-secret="$OP_NODE_L2_ENGINE_JWT_SECRET" is set.
Networking: Ensure --rpc.addr=0.0.0.0 is set in both to allow external monitoring.

4. Troubleshooting & Mastery 🔍

If your node isn't syncing, check these three layers in order:

Layer 1: The JWT Handshake

If op-node cannot talk to op-geth, you will see 401 Unauthorized.

Check: docker exec execution-1 cat /shared/jwt.txt
Fix: Ensure the file is exactly 64 characters long. If it contains "true" or a file path instead of hex, your script logic is overwriting the secret.

Layer 2: L1 Connectivity

The op-node derives L2 blocks from L1 data. If L1 is down, L2 stops.

Check: docker exec node-1 curl -s http://testnet:8585
Fix: Ensure your L1 node is fully synced and reachable on the Docker network.

Layer 3: P2P Peering

In snap sync mode, you need peers to download the state.

Check: curl -s -d '{"id":1,"jsonrpc":"2.0","method":"opp2p_peerStats"}' http://localhost:9645 | jq
Fix: Ensure ports 30303 (Geth) and 9222 (Node) are open on your firewall.

5. Monitoring Your Progress

The most important command for a node operator is the Sync Status. It tells you exactly where you are in history.

The "Scoreboard" Command:

watch -n 5 'curl -s -X POST -H "Content-Type: application/json" --data "{\"jsonrpc\":\"2.0\",\"method\":\"optimism_syncStatus\",\"params\":[],\"id\":1}" http://localhost:9645 | jq ".result.local_safe_l2.number, .result.local_safe_l2.timestamp"'

First Number: Current L2 Block Height.
Second Number: Unix Timestamp of that block (compare this to the current time to see how many days/months you are behind).

Summary

Generate a clean JWT.
Align your volumes so both containers see the same /shared/jwt.txt.
Point to a healthy L1 RPC and Beacon node.
Monitor via optimism_syncStatus.

Automating Container Image Updates with FluxCD (Hands-On Tutorial)

iapilgrim — Sat, 21 Mar 2026 05:06:50 +0000

Modern GitOps workflows aim to keep your Kubernetes cluster fully synchronized with what is defined in Git. One powerful feature of Flux is Image Automation, which automatically updates container image tags in your Git repository whenever a new image becomes available.

In this tutorial, we walk through how image automation works and how to troubleshoot a common issue involving Git authentication.

1. What is Flux Image Automation?

FluxCD provides several components that work together to automate image updates.

The workflow looks like this:

Container Registry
↓
ImageRepository (scans tags)
↓
ImagePolicy (selects allowed tags)
↓
ImageUpdateAutomation (commits updates to Git)
↓
Flux Kustomization deploys updated workload

Instead of manually updating image tags in Git, Flux automatically commits the new tag when a matching image appears in your container registry.

2. Repository Structure Used in This Lab

Example GitOps layout:

flux-minikube-lab
├── apps
│   └── web-server
│       ├── web-server.yaml
│       ├── sealed-db-pass.yaml
│       └── kustomization.yaml
│
└── clusters
    └── my-cluster
        ├── image-reflect.yaml
        ├── image-policy.yaml
        ├── web-server-sync.yaml
        └── kustomization.yaml

The cluster Kustomization references infrastructure and application sync resources.

3. Application Deployment

The web server manifests are defined in:

apps/web-server/

Example kustomization.yaml:

resources:
  - web-server.yaml
  - sealed-db-pass.yaml

A Flux Kustomization sync object deploys the app:

kind: Kustomization
metadata:
  name: web-server-sync
  namespace: flux-system
spec:
  interval: 1m
  path: ./apps/web-server
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  targetNamespace: engineering

This instructs Flux to deploy the web server into the engineering namespace.

4. Image Automation Setup

Three resources enable automated updates:

Image Repository

This scans container registry tags.

kind: ImageRepository

Image Policy

Defines which tags are acceptable (e.g., semver or latest).

kind: ImagePolicy

Image Update Automation

Updates the Git repo when a new tag matches the policy.

kind: ImageUpdateAutomation

5. Triggering Automation

To manually trigger automation:

flux reconcile image update flux-system

Flux will:

Check registry tags
Apply the image policy
Update manifests in Git
Commit and push the change

6. Troubleshooting a Common Error

While testing automation, the following error appeared:

failed to update source:
failed to push to remote:
ERROR: The key you are authenticating with has been marked as read only

Checking the automation status:

flux get image update

Output:

READY: False
MESSAGE: failed to push to remote

7. Root Cause

ImageUpdateAutomation needs write access to the Git repository to commit updated image tags.

However, the repository authentication key was configured as read-only.

This allowed Flux to pull manifests but prevented it from pushing updates.

8. Fix: Recreate the Flux Git Secret with Write Access

Step 1 — Generate a New SSH Key

ssh-keygen

Step 2 — Add the Public Key to GitHub

Navigate to the repository settings:

Settings → Deploy Keys

Add the new public key and enable:

Allow write access

Step 3 — Recreate the Flux Git Secret

flux create secret git flux-system \
  --url=ssh://git@github.com/pilgrim2go/flux-minikube-lab \
  --private-key-file=$PWD/fluxcd-test \
  -n flux-system

This updates the authentication credentials used by Flux.

Step 4 — Re-run Automation

flux reconcile image update flux-system

Verify:

flux get image update

Expected result:

READY: True
MESSAGE: committed and pushed update

A new commit should appear in the repository with the updated image tag.

9. Helpful Debug Commands

Check Flux resources:

flux get all

Inspect automation status:

flux get image update

View controller logs:

kubectl logs -n flux-system deploy/image-automation-controller

Trigger Git sync:

flux reconcile source git flux-system

10. Best Practices

When using Flux image automation:

• Use a dedicated deploy key or bot account
• Ensure Git credentials allow read and write access
• Keep Git as the single source of truth
• Avoid manual cluster changes outside GitOps

Conclusion

Flux image automation eliminates manual image updates and ensures that your Kubernetes workloads always run the latest approved container images.

With the proper Git authentication setup, Flux can automatically:

• Detect new container images
• Update manifests in Git
• Trigger GitOps deployments

This enables a fully automated and auditable Kubernetes delivery pipeline.

FluxCD Image Automation Error Troubleshooting

iapilgrim — Sat, 21 Mar 2026 05:05:02 +0000

Problem

Running:

flux reconcile image update flux-system

Result:

failed to update source: failed to push to remote
ERROR: The key you are authenticating with has been marked as read only

And:

flux get image update

shows:

READY: False
MESSAGE: failed to push to remote

Root Cause

FluxCD ImageUpdateAutomation needs to commit and push updates to the Git repository when it updates container image tags.

Pipeline:

Container Registry
        ↓
ImageRepository
        ↓
ImagePolicy
        ↓
ImageUpdateAutomation
        ↓
Git Commit + Push
        ↓
Flux Kustomization deploys update

If the Git credential is read-only, the push fails.

Diagnosis Steps

1. Check Image Automation Status

flux get image update

Look for:

READY: False
failed to push to remote

2. Inspect the Automation Object

kubectl get imageupdateautomation -A

Example:

STATUS: failed to update source

3. Check the Git Source

flux get sources git

This confirms Flux can read the repo.

But pushing still fails if the key is read-only.

4. Confirm Git Authentication

Check the Git secret:

kubectl get secret flux-system -n flux-system

This secret contains the SSH key Flux uses.

Fix Implemented

You recreated the Git authentication secret with a write-enabled SSH key.

1️⃣ Generate SSH Key

ssh-keygen

2️⃣ Add Public Key to GitHub

Go to repo:

Settings → Deploy Keys

Add:

fluxcd-test.pub

Enable:

Allow write access

3️⃣ Recreate Flux Git Secret

flux create secret git flux-system \
  --url=ssh://git@github.com/pilgrim2go/flux-minikube-lab \
  --private-key-file=$PWD/fluxcd-test \
  -n flux-system

This updates the Git credential used by Flux.

4️⃣ Trigger Automation

flux reconcile image update flux-system

Expected Result

flux get image update

READY: True
MESSAGE: committed and pushed update

You should also see a commit in Git like:

flux: update image tag

Useful Debug Commands

Check full Flux status:

flux get all

Check automation logs:

kubectl logs -n flux-system deploy/image-automation-controller

Test Git sync:

flux reconcile source git flux-system

Best Practice

Use a dedicated Flux deploy key with:

read + write

instead of personal access tokens when using SSH Git repositories.

FluxCD journey with Minikube

iapilgrim — Fri, 20 Mar 2026 14:33:52 +0000

🚀 Phase 1: The Manual Foundation

Goal: Set up the cluster and deploy a "Hello World" app the old-fashioned way to understand what we are automating.

🛠️ Step 1: Install Tools

# Install the Big Three (macOS example)
brew install minikube kubectl fluxcd/tap/flux

🏗️ Step 2: Start Minikube

minikube start --cpus 2 --memory 4096 --driver=docker
minikube addons enable ingress

📂 Step 3: Directory Layout

Create this structure on your local machine:

flux-lab/
└── base/
    ├── kustomization.yaml
    └── web-server.yaml

📄 Step 4: The Manifests

flux-lab/base/web-server.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-server
  namespace: engineering
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.25
        ports:
        - containerPort: 80

flux-lab/base/kustomization.yaml

resources:
  - web-server.yaml

🚀 Step 5: Deploy Manually

kubectl create namespace engineering
kubectl apply -k flux-lab/base/
kubectl get pods -n engineering

🤖 Phase 2: The Great Automation (FluxCD)

Goal: Connect GitHub to Minikube. From this point on, we never use kubectl apply again.

🛠️ Step 1: Environment Setup

export GITHUB_TOKEN=your_personal_access_token
export GITHUB_USER=your_github_username

🏗️ Step 2: Bootstrap Flux

flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=flux-minikube-lab \
  --branch=main \
  --path=clusters/my-cluster \
  --personal

📂 Step 3: Final Git Directory Layout

Clone your new repo and organize it exactly like this:

flux-minikube-lab/
├── apps/
│   └── web-server/
│       ├── kustomization.yaml
│       └── web-server.yaml
└── clusters/
    └── my-cluster/
        ├── flux-system/         # (Auto-generated)
        └── web-server-sync.yaml

📄 Step 4: Create the "Sync" Instruction

clusters/my-cluster/web-server-sync.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: web-server-sync
  namespace: flux-system
spec:
  interval: 1m
  path: ./apps/web-server
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  targetNamespace: engineering

🚀 Step 5: Push and Pray (The GitOps Way)

git add .
git commit -m "Onboard web-server to GitOps"
git push origin main

# Force immediate sync
flux reconcile kustomization flux-system --with-source

🔐 Phase 3: The Secret Sauce (Sealed Secrets)

Goal: Store passwords in GitHub securely using encryption.

🏗️ Step 1: Install Infrastructure

Place these files in infrastructure/sources/ and infrastructure/controllers/.

clusters/my-cluster/infra-sync.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infra-sync
  namespace: flux-system
spec:
  interval: 1h
  path: ./infrastructure
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

📂 Step 2: Final Phase 3 Directory Layout

flux-minikube-lab/
├── apps/
│   └── web-server/
│       ├── kustomization.yaml   # (Update to include sealed-db-pass.yaml)
│       ├── web-server.yaml
│       └── sealed-db-pass.yaml  # (Generated)
├── clusters/
│   └── my-cluster/
│       ├── infra-sync.yaml
│       └── web-server-sync.yaml
└── infrastructure/
    ├── controllers/
    │   └── sealed-secrets.yaml
    └── sources/
        └── sealed-secrets.yaml

🔐 Step 3: Create an Encrypted Secret

# 1. Create a raw secret (DO NOT PUSH TO GIT)
kubectl create secret generic mwd-db-pass \
  --from-literal=password=SuperSecret123 \
  --namespace engineering \
  --dry-run=client -o yaml > temp.yaml

# 2. Encrypt it using the cluster's key
kubeseal \
  --controller-name sealed-secrets \
  --controller-namespace flux-system \
  --format yaml < temp.yaml > apps/web-server/sealed-db-pass.yaml

# 3. Clean up
rm temp.yaml

🚀 Step 4: Deploy

git add .
git commit -m "Add sealed secret"
git push origin main
flux reconcile kustomization infra-sync --with-source
flux reconcile kustomization web-server-sync --with-source

🧠 Summary of Progress

Phase 1: Learned Kubernetes resources.
Phase 2: Learned FluxCD automation and the "Pull Model."
Phase 3: Learned Security and encryption in Git.

🧠 Migrating BSC Testnet Data to a New Disk (KVM + Docker)

iapilgrim — Thu, 19 Mar 2026 03:44:50 +0000

Running blockchain nodes at scale quickly turns into a storage management problem.
In this guide, I’ll walk through a real-world migration of a BSC testnet dataset (~243GB) to a new disk — without breaking the node — and share key lessons learned.

📊 Initial Situation

Inside the VM:

/dev/vdb1 → /node-data (6.3T disk, almost full)

Breakdown:

Mainnet: ~5.8TB
Testnet: ~243GB

Problem:

Disk usage: 100%
Free space: ~41GB ⚠️

👉 Risk: node crash, DB corruption, sync failure

🎯 Goal

Move testnet data to a new disk
Keep mainnet untouched
Avoid complex Docker changes
Minimize downtime

🏗️ Architecture Before

vdb (6.3T)
└── /node-data
    ├── mainnet (~5.8T)
    └── testnet (~243G)

🚀 Architecture After

vdb → mainnet
vdc → testnet

/bsc-testnet → real mount
/node-data/testnet → bind mount → /bsc-testnet

Docker still uses:

/node-data/testnet:/bsc/node

👉 No container config change needed.

⚡ Step-by-Step Migration

1️⃣ Attach new disk (on host)

qemu-img create -f qcow2 /path/bsc-testnet.qcow2 400G

virsh attach-disk bsc \
  /path/bsc-testnet.qcow2 \
  vdc \
  --targetbus virtio \
  --subdriver qcow2 \
  --persistent

2️⃣ Prepare disk (inside VM)

Skip partitioning (simpler):

mkfs.ext4 /dev/vdc
mkdir /bsc-testnet
mount /dev/vdc /bsc-testnet

3️⃣ Copy data

rsync -avh /node-data/testnet/ /bsc-testnet/

4️⃣ Switch using bind mount (no Docker change)

docker stop testnet

mv /node-data/testnet /node-data/testnet-old
mkdir /node-data/testnet

mount --bind /bsc-testnet /node-data/testnet

docker start testnet

5️⃣ Cleanup (after verification)

rm -rf /node-data/testnet-old

6️⃣ Make persistent

Edit /etc/fstab:

UUID=<vdc-uuid> /bsc-testnet ext4 defaults 0 0
/bsc-testnet /node-data/testnet none bind 0 0

Test:

mount -a

🔥 Key Lessons Learned

1️⃣ You don’t need to touch Docker

Instead of:

editing volumes
recreating containers

👉 Just move the filesystem underneath

This is safer and faster.

2️⃣ Bind mounts are extremely powerful

/node-data/testnet → /bsc-testnet

Acts like:

transparent redirect
zero config change
instant rollback

3️⃣ “target is busy” = you are inside the directory

Classic mistake:

umount /node-data/testnet
→ target is busy

Cause:

cwd = /node-data/testnet

Fix:

cd /

4️⃣ Duplicate mounts can happen easily

Running mount --bind multiple times creates stacked mounts.

Check with:

mount | grep testnet
findmnt /node-data/testnet

Fix:

umount /node-data/testnet (repeat until gone)

5️⃣ Never unmount while container is running

Even if it “works”:

DB writes can fail
corruption risk
node may resync from scratch

👉 Always:

docker stop → umount → remount → start

6️⃣ Partitioning is optional

For dedicated disks:

mkfs.ext4 /dev/vdc

is simpler than:

fdisk → /dev/vdc1

7️⃣ Blockchain nodes will ALWAYS outgrow your disk

After migration:

mainnet still ~92% full

👉 This is temporary relief.

You must plan:

disk expansion (qemu-img resize)
or data rebalancing

📈 Results

After migration:

Freed ~243GB on main disk
Isolated IO between mainnet/testnet
No Docker reconfiguration
Minimal downtime (~1–2 minutes)

🧠 Final Takeaway

The key mindset shift:

Don’t move applications — move the filesystem underneath them.

This approach:

reduces risk
simplifies operations
scales better for large datasets (TBs)

🚀 What’s Next

For production-grade setups:

Separate disks per chain
Use virsh blockcopy for live migration
Implement storage balancing strategy across NVMe pools

vLLM Request Lifecycle (Where TTFT is measured)

iapilgrim — Wed, 11 Mar 2026 13:12:54 +0000

Observing LLM Latency: Monitoring Time-To-First-Token in vLLM

Large language model APIs often feel fast even when generating long responses.
The key reason is streaming tokens — the response starts quickly while the rest of the answer is still being generated.

The most important metric for this experience is Time-To-First-Token (TTFT).

This tutorial explains:

what TTFT is
how vLLM exposes TTFT metrics
how Prometheus calculates percentiles
how to build a Grafana dashboard

What is Time-To-First-Token (TTFT)?

TTFT measures how long it takes from request arrival to the first generated token.

LLM inference pipeline:

Request arrives
      ↓
Prompt tokenization
      ↓
Prefill (model processes prompt)
      ↓
FIRST TOKEN GENERATED  ← TTFT measured here
      ↓
Token streaming
      ↓
Response complete

Users perceive the system as fast if TTFT is small.

Typical targets:

System	Good TTFT
Chat UI	<2s
GPU inference	<4s
Heavy batching	<8s

vLLM TTFT Metric

vLLM exports a Prometheus histogram metric:

vllm:time_to_first_token_seconds_bucket

Example metrics:

vllm:time_to_first_token_seconds_bucket{le="1"} 1
vllm:time_to_first_token_seconds_bucket{le="2.5"} 73
vllm:time_to_first_token_seconds_bucket{le="10"} 103
vllm:time_to_first_token_seconds_bucket{le="20"} 104

These represent histogram buckets.

Example interpretation:

TTFT	Requests
≤1s	1
≤2.5s	73
≤10s	103
≤20s	104

Buckets are cumulative counters.

Common Causes of High TTFT

If TTFT increases, check:

GPU saturation

Too many requests in queue.

Large prompts

Prefill phase takes longer.

Batch scheduling delay

Large batch sizes increase wait time.

KV cache limits

Cache misses slow inference.

Recommended vLLM Observability Metrics

Monitor these together:

Metric	Meaning
TTFT	time until response starts
tokens/sec	generation speed
request latency	full response time
active requests	queue pressure
GPU utilization	hardware bottleneck

Summary

Monitoring TTFT helps maintain a responsive LLM system.

Pipeline:

vLLM metrics
     ↓
Prometheus histogram
     ↓
PromQL percentile query
     ↓
Grafana visualization

With proper monitoring, you can detect:

GPU saturation
batching issues
prompt bottlenecks

before users notice degraded performance.

Troubleshooting: Fixing k6 GPG Key Errors in Ubuntu

iapilgrim — Wed, 11 Mar 2026 03:41:07 +0000

Setting up k6 on Ubuntu should be a straightforward process, but GPG key errors can occasionally turn a "five-minute task" into a troubleshooting marathon. If you’ve encountered the dreaded NO_PUBKEY or unsupported filetype errors, this guide will walk you through the fix and the proper installation.

Modern Linux distributions have moved away from apt-key add (which is now deprecated) toward specific keyrings located in /usr/share/keyrings. This "sandboxes" the security keys so that the k6 key can only be used to verify k6 packages, making your overall system much more secure.

🛠️ Troubleshooting: Fixing k6 GPG Key Errors

When you see errors like "The following signatures couldn't be verified" or "unsupported filetype," it means your system doesn't trust the k6 repository or the key file you downloaded is corrupted (usually saved as plain text instead of binary).

Step 1: Clean Up Old Keys

If you have a broken keyring file, apt will continue to complain until it's gone. Start by removing any existing k6 keyring:

sudo rm /usr/share/keyrings/k6-archive-keyring.gpg

Step 2: Download and "Dearmor" the Key

The key must be in a binary format for modern Ubuntu versions. We use gpg --dearmor to convert the text-based key from k6 into the required binary format.

curl -fsSL https://dl.k6.io/key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/k6-archive-keyring.gpg

Step 3: Set Correct Permissions

System services need to be able to read this file. Set the permissions to 644:

sudo chmod 644 /usr/share/keyrings/k6-archive-keyring.gpg

📦 Setting Up the Repository

Now that the security "handshake" is ready, you need to tell Ubuntu exactly where to find k6 and which key to use for verification.

Step 4: Add the k6 Source List

Run this command to create (or overwrite) the source file. Note the signed-by tag—this is the secret sauce that links the repo to the key we just created.

echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list

🚀 Final Installation

With the repository correctly configured and the key verified, you can now update your package list and install k6.

Step 5: Update and Install

sudo apt-get update
sudo apt-get install k6

Step 6: Verify Success

Confirm k6 is installed and ready to go by checking the version:

k6 version

Overlay vs Underlay Networking Across Regions with Azure Kubernetes Service

iapilgrim — Mon, 09 Mar 2026 03:17:28 +0000

In this tutorial we build a complete AKS networking lab to demonstrate:

AKS Overlay networking
AKS Underlay (Azure CNI) networking
Cross-region VNet peering
Pod-to-pod communication across clusters
Access to Azure SQL via Private Endpoint
Service exposure via NodePort

Final architecture:

Region 1 (Southeast Asia)
VNet: 10.0.0.0/16
--------------------------------

AKS Overlay Cluster
Nodes: 10.0.x.x
Pods : 192.168.x.x

Private Endpoint
Azure SQL
10.0.4.4

Private DNS Zone
privatelink.database.windows.net

Region 2 (East Asia)
VNet: 10.1.0.0/16
--------------------------------

AKS Underlay Cluster
Nodes: 10.1.x.x
Pods : 10.1.x.x

VNet Peering
10.0.0.0/16  <---->  10.1.0.0/16

1. Create the Overlay AKS Cluster

Overlay networking allows pods to use non-VNet IP ranges, which prevents subnet exhaustion.

Create resource group

RG_NETWORK=rg-aks-network
LOCATION=southeastasia

az group create \
  --name $RG_NETWORK \
  --location $LOCATION

Create VNet

VNET_NAME=vnet-aks-overlay

az network vnet create \
  --resource-group $RG_NETWORK \
  --name $VNET_NAME \
  --address-prefix 10.0.0.0/16 \
  --subnet-name aks-subnet \
  --subnet-prefix 10.0.1.0/24

Create AKS overlay cluster

AKS_OVERLAY=aks-overlay

SUBNET_ID=$(az network vnet subnet show \
  --resource-group $RG_NETWORK \
  --vnet-name $VNET_NAME \
  --name aks-subnet \
  --query id -o tsv)

az aks create \
  --resource-group $RG_NETWORK \
  --name $AKS_OVERLAY \
  --location $LOCATION \
  --node-count 2 \
  --network-plugin azure \
  --network-plugin-mode overlay \
  --pod-cidr 192.168.0.0/16 \
  --vnet-subnet-id $SUBNET_ID \
  --generate-ssh-keys

Connect kubectl

az aks get-credentials \
  --resource-group $RG_NETWORK \
  --name $AKS_OVERLAY \
  --context aks-overlay

Verify overlay pod networking

Deploy a test pod.

kubectl run overlay-test \
  --image=busybox \
  --restart=Never \
  -- sleep 3600

Check pod IP:

kubectl get pods -o wide

Example:

overlay-test   192.168.0.4

Overlay pods use 192.168.x.x, not VNet IPs.

2. Create Azure SQL with Private Endpoint

Create SQL server:

SQL_SERVER=aks-lab-sql-31445

az sql server create \
  --name $SQL_SERVER \
  --resource-group $RG_NETWORK \
  --location $LOCATION \
  --admin-user sqladmin \
  --admin-password <password>

Create private endpoint

az network private-endpoint create \
  --resource-group $RG_NETWORK \
  --name sql-pe \
  --vnet-name $VNET_NAME \
  --subnet aks-subnet \
  --private-connection-resource-id $(az sql server show --name $SQL_SERVER --resource-group $RG_NETWORK --query id -o tsv) \
  --group-id sqlServer \
  --connection-name sql-pe-connection

Create Private DNS zone

az network private-dns zone create \
  --resource-group $RG_NETWORK \
  --name privatelink.database.windows.net

Link overlay VNet

az network private-dns link vnet create \
  --resource-group $RG_NETWORK \
  --zone-name privatelink.database.windows.net \
  --name link-overlay-vnet \
  --virtual-network $VNET_NAME \
  --registration-enabled false

Test SQL access from overlay pod

kubectl exec -it overlay-test -- sh

nslookup <sql-server>.database.windows.net

Expected:

10.0.x.x

3. Create Underlay AKS Cluster in Another Region

Underlay clusters assign real VNet IPs to pods.

Create resource group

RG_UNDERLAY=rg-aks-underlay
UNDERLAY_LOCATION=eastasia

az group create \
  --name $RG_UNDERLAY \
  --location $UNDERLAY_LOCATION

Create VNet

VNET_UNDERLAY=vnet-aks-underlay

az network vnet create \
  --resource-group $RG_UNDERLAY \
  --name $VNET_UNDERLAY \
  --location $UNDERLAY_LOCATION \
  --address-prefix 10.1.0.0/16 \
  --subnet-name aks-subnet \
  --subnet-prefix 10.1.1.0/24

Create underlay AKS cluster

SUBNET_UNDERLAY=$(az network vnet subnet show \
  --resource-group $RG_UNDERLAY \
  --vnet-name $VNET_UNDERLAY \
  --name aks-subnet \
  --query id -o tsv)

az aks create \
  --resource-group $RG_UNDERLAY \
  --name aks-underlay \
  --location $UNDERLAY_LOCATION \
  --node-count 2 \
  --network-plugin azure \
  --vnet-subnet-id $SUBNET_UNDERLAY \
  --service-cidr 172.17.0.0/16 \
  --dns-service-ip 172.17.0.10 \
  --generate-ssh-keys

Connect kubectl

az aks get-credentials \
  --resource-group $RG_UNDERLAY \
  --name aks-underlay \
  --context aks-underlay

Verify pod IP

kubectl --context aks-underlay run underlay-test \
  --image=busybox \
  --restart=Never \
  -- sleep 3600

Check IP:

10.1.x.x

Pods now consume VNet IPs.

4. Peer the VNets

Overlay and underlay clusters must communicate.

Overlay → Underlay

az network vnet peering create \
  --resource-group $RG_NETWORK \
  --name overlay-to-underlay \
  --vnet-name $VNET_NAME \
  --remote-vnet /subscriptions/<sub>/resourceGroups/$RG_UNDERLAY/providers/Microsoft.Network/virtualNetworks/$VNET_UNDERLAY \
  --allow-vnet-access

Underlay → Overlay

az network vnet peering create \
  --resource-group $RG_UNDERLAY \
  --name underlay-to-overlay \
  --vnet-name $VNET_UNDERLAY \
  --remote-vnet /subscriptions/<sub>/resourceGroups/$RG_NETWORK/providers/Microsoft.Network/virtualNetworks/$VNET_NAME \
  --allow-vnet-access

5. Link Private DNS to Underlay VNet

Without this step, the underlay cluster resolves public SQL endpoints.

az network private-dns link vnet create \
  --resource-group $RG_NETWORK \
  --zone-name privatelink.database.windows.net \
  --name link-underlay-vnet \
  --virtual-network /subscriptions/<sub>/resourceGroups/$RG_UNDERLAY/providers/Microsoft.Network/virtualNetworks/$VNET_UNDERLAY \
  --registration-enabled false

6. Communication Testing Scenarios

Scenario 1 — Overlay Pod → Underlay Pod

Get underlay pod IP:

10.1.1.18

From overlay pod:

kubectl --context aks-overlay exec -it overlay-test -- sh

ping 10.1.1.18

Success proves cross-region VNet routing works.

Scenario 2 — Underlay Pod → Azure SQL Private Endpoint

Enter underlay pod:

kubectl exec -it underlay-test -- sh

nslookup <sql-server>.database.windows.net

Result:

10.0.4.4

Test SQL port:

nc -zv <sql-server>.database.windows.net 1433

Expected:

Connection succeeded

Scenario 3 — Overlay Pod → Underlay Service

Deploy nginx in underlay cluster:

kubectl create deployment nginx --image=nginx

Expose via NodePort:

kubectl expose deployment nginx \
  --type NodePort \
  --port 80 \
  --name nginx-nodeport

Check NodePort:

80:31904/TCP

Get node IP:

10.1.1.33

Test from overlay pod:

kubectl --context aks-overlay exec -it overlay-test -- sh

wget -qO- http://10.1.1.33:31904

Result:

Welcome to nginx!

Key Networking Differences

Feature	Overlay	Underlay
Pod IP	Overlay CIDR	VNet subnet
VNet IP usage	No	Yes
Subnet exhaustion risk	Low	High
Direct VNet reachability	No	Yes

Final Result

This lab demonstrates:

Overlay networking
Underlay networking
Cross-region VNet peering
AKS pod communication
Azure SQL Private Endpoint connectivity
Kubernetes service exposure

Deploy Azure SQL with Private Endpoint and Test Connectivity from AKS

iapilgrim — Sun, 08 Mar 2026 07:14:08 +0000

This tutorial demonstrates how to:

Create networking for AKS
Deploy an AKS cluster using Azure CNI Overlay
Deploy Azure SQL Database
Secure it using a Private Endpoint
Fix DNS resolution using Private DNS Zone
Verify connectivity from Kubernetes using debugging pods

The goal is to allow AKS pods to securely connect to Azure SQL using private networking only.

1. Define Environment Variables

First define reusable variables.

LOCATION=southeastasia

RG_NETWORK=rg-aks-network
RG_OVERLAY=rg-aks-overlay
RG_UNDERLAY=rg-aks-underlay

VNET_NAME=vnet-aks-lab

SUBNET_OVERLAY=subnet-overlay
SUBNET_UNDERLAY=subnet-underlay

AKS_OVERLAY=aks-overlay
AKS_UNDERLAY=aks-underlay

2. Create Resource Groups

Create resource groups to organize infrastructure.

az group create --name $RG_NETWORK --location $LOCATION
az group create --name $RG_OVERLAY --location $LOCATION
az group create --name $RG_UNDERLAY --location $LOCATION

3. Create Virtual Network

Create a VNet that will host:

AKS nodes
Private endpoints
Services

az network vnet create \
  --resource-group $RG_NETWORK \
  --name $VNET_NAME \
  --address-prefix 10.0.0.0/16

4. Create Subnets

Overlay subnet (AKS nodes)

az network vnet subnet create \
  --resource-group $RG_NETWORK \
  --vnet-name $VNET_NAME \
  --name $SUBNET_OVERLAY \
  --address-prefix 10.0.1.0/24

Underlay subnet

az network vnet subnet create \
  --resource-group $RG_NETWORK \
  --vnet-name $VNET_NAME \
  --name $SUBNET_UNDERLAY \
  --address-prefix 10.0.2.0/24

Retrieve subnet IDs:

OVERLAY_SUBNET_ID=$(az network vnet subnet show \
  --resource-group $RG_NETWORK \
  --vnet-name $VNET_NAME \
  --name $SUBNET_OVERLAY \
  --query id -o tsv)

UNDERLAY_SUBNET_ID=$(az network vnet subnet show \
  --resource-group $RG_NETWORK \
  --vnet-name $VNET_NAME \
  --name $SUBNET_UNDERLAY \
  --query id -o tsv)

echo $OVERLAY_SUBNET_ID
echo $UNDERLAY_SUBNET_ID

5. Deploy AKS Cluster (Azure CNI Overlay)

Create the AKS cluster.

az aks create \
  --resource-group $RG_OVERLAY \
  --name $AKS_OVERLAY \
  --location $LOCATION \
  --node-count 2 \
  --network-plugin azure \
  --network-plugin-mode overlay \
  --pod-cidr 192.168.0.0/16 \
  --service-cidr 172.16.0.0/16 \
  --dns-service-ip 172.16.0.10 \
  --vnet-subnet-id $OVERLAY_SUBNET_ID \
  --generate-ssh-keys

6. Connect kubectl to the Cluster

Retrieve credentials.

az aks get-credentials \
  --resource-group $RG_OVERLAY \
  --name $AKS_OVERLAY

Verify cluster status.

kubectl get nodes -o wide
kubectl get pods -o wide

7. Create Subnets for Services

Service subnet

az network vnet subnet create \
  --resource-group $RG_NETWORK \
  --vnet-name $VNET_NAME \
  --name subnet-service \
  --address-prefix 10.0.3.0/24

Private Endpoint subnet

az network vnet subnet create \
  --resource-group $RG_NETWORK \
  --vnet-name $VNET_NAME \
  --name subnet-private-endpoint \
  --address-prefix 10.0.4.0/24

8. Create Azure SQL Server

Generate a unique name.

SQL_SERVER=aks-lab-sql-$RANDOM

Create SQL server.

az sql server create \
  --name $SQL_SERVER \
  --resource-group $RG_NETWORK \
  --location $LOCATION \
  --admin-user sqladmin \
  --admin-password 'StrongPass123!'

Ensure provider is registered.

az provider register --namespace Microsoft.Sql
az provider show --namespace Microsoft.Sql --query "registrationState"

9. Create SQL Database

az sql db create \
  --resource-group $RG_NETWORK \
  --server $SQL_SERVER \
  --name demo-db \
  --service-objective Basic

Retrieve the SQL resource ID.

SQL_ID=$(az sql server show \
  --name $SQL_SERVER \
  --resource-group $RG_NETWORK \
  --query id -o tsv)

echo $SQL_ID

10. Create Private Endpoint for Azure SQL

az network private-endpoint create \
  --name sql-private-endpoint \
  --resource-group $RG_NETWORK \
  --vnet-name $VNET_NAME \
  --subnet subnet-private-endpoint \
  --private-connection-resource-id $SQL_ID \
  --group-id sqlServer \
  --connection-name sqlConnection

11. Verify Private Endpoint IP

Get network interface ID.

NIC_ID=$(az network private-endpoint show \
  --name sql-private-endpoint \
  --resource-group $RG_NETWORK \
  --query "networkInterfaces[0].id" \
  -o tsv)

echo $NIC_ID

Retrieve private IP.

az network nic show \
  --ids $NIC_ID \
  --query "ipConfigurations[0].privateIPAddress" \
  -o tsv

Example output:

10.0.4.4

12. Test Network from AKS Pod

Launch a debugging pod.

Image:
nicolaka/netshoot

kubectl run net-test \
  --image=nicolaka/netshoot \
  -it --rm -- bash

Inside the pod you can test DNS and networking.

13. Test SQL Client Pod

Run a SQL client container.

kubectl run sql-client \
  --image=mcr.microsoft.com/mssql-tools \
  -it --rm -- bash

14. Create Private DNS Zone

Azure SQL private endpoints require DNS mapping.

Create the private DNS zone:

az network private-dns zone create \
  --resource-group $RG_NETWORK \
  --name privatelink.database.windows.net

Link the DNS zone to the VNet.

az network private-dns link vnet create \
  --resource-group $RG_NETWORK \
  --zone-name privatelink.database.windows.net \
  --name sql-dns-link \
  --virtual-network $VNET_NAME \
  --registration-enabled false

Attach the private endpoint to the DNS zone.

az network private-endpoint dns-zone-group create \
  --resource-group $RG_NETWORK \
  --endpoint-name sql-private-endpoint \
  --name sql-zone-group \
  --private-dns-zone privatelink.database.windows.net \
  --zone-name sql

15. Verify DNS Resolution from AKS

Run another debug pod.

kubectl run net-test \
  --image=nicolaka/netshoot \
  -it --rm -- bash

Test DNS:

nslookup <your-sql-server>.database.windows.net

Expected result:

10.0.4.4

16. Connect to Azure SQL

Run SQL client.

kubectl run sql-client \
  --image=mcr.microsoft.com/mssql-tools \
  -it --rm -- bash

Connect using sqlcmd.

sqlcmd \
  -S aks-lab-sql-31445.database.windows.net \
  -U sqladmin \
  -P 'StrongPass123!' \
  -d demo-db

If successful you will see:

1>

This means you are connected to the SQL server.

Example query:

SELECT @@VERSION;
GO

Final Architecture

AKS Pod
   │
   │ DNS Query
   ▼
Private DNS Zone
(privatelink.database.windows.net)
   │
   ▼
Private Endpoint (10.0.4.4)
   │
   ▼
Azure SQL Database

✅ You now have Azure SQL accessible privately from AKS pods.

How to Permanently Disable a systemd Service on Ubuntu

iapilgrim — Thu, 05 Mar 2026 13:45:24 +0000

Sometimes you install software that registers itself as a systemd service and starts automatically at boot. If you don’t want it running anymore, you can disable or even mask it so it never starts again. Here’s a step-by-step guide using vllm.service as an example.

Step 1: Stop the Service Immediately

First, stop the service if it’s currently running:

sudo systemctl stop vllm.service

Step 2: Disable the Service at Boot

Prevent the service from starting automatically on reboot:

sudo systemctl disable vllm.service

This removes the symlink from the systemd startup sequence.

Step 3: Mask the Service (Optional but Stronger)

Masking a service ensures it cannot be started manually or by another dependency.

If the unit file already exists in /etc/systemd/system/, you’ll need to rename or remove it before masking:

sudo mv /etc/systemd/system/vllm.service /etc/systemd/system/vllm.service.backup
sudo systemctl daemon-reload
sudo systemctl mask vllm.service

Step 4: Verify the Status

Check whether the service is disabled or masked:

systemctl is-enabled vllm.service
systemctl status vllm.service

disabled → won’t start at boot
masked → cannot be started at all

Step 5: (Optional) Re-enable Later

If you change your mind, simply unmask and re-enable:

sudo systemctl unmask vllm.service
sudo systemctl enable vllm.service

Final Thoughts

Disable if you just don’t want it auto-starting.
Mask if you want to block it completely.
Remove the unit file if you’re sure you’ll never use it again.

This approach ensures the service stays off permanently, even after rebooting.

Troubleshooting KVM Issues

iapilgrim — Thu, 05 Mar 2026 13:44:18 +0000

🧱 1️⃣ Initial Symptom

❌ “Booting from hard disk…” (hangs)

What This Means Technically

Firmware attempted to boot from the virtual disk but:

Could not find a valid bootloader
Or could not understand the partition scheme
Or firmware mode didn’t match OS install mode

This message comes from firmware (not Linux).

In KVM environments, firmware is provided by:

Legacy BIOS (SeaBIOS)
UEFI (OVMF from TianoCore)

🔎 2️⃣ First Investigation: Is the Disk Corrupted?

We checked:

qemu-img info
virt-filesystems

What We Found

qcow2 format valid
Not corrupted
GPT layout
/dev/sda1 → vfat (511M)
/dev/sda2 → ext4
/dev/sda3 → swap

Critical Clue

vfat EFI partition = UEFI installation

That immediately ruled out:

Disk corruption
Missing root filesystem
Missing GRUB files

💡 Lesson: Always inspect disk structure before guessing.

⚡ 3️⃣ Root Cause #1 — Firmware Mismatch

Your VM XML showed:

<os>
  <type arch='x86_64' machine='pc-q35-9.2'>hvm</type>
  <boot dev='hd'/>
</os>

No <loader> entry → That means legacy BIOS mode

But your disk layout proved:

Installed using UEFI
GPT partition table
EFI System Partition

Why BIOS Could Not Boot

Legacy BIOS:

Looks for MBR
Expects GRUB in MBR stage1
Does not understand EFI partitions

UEFI:

Reads FAT EFI partition
Loads .efi binaries

So BIOS firmware had no idea how to boot your disk.

💡 Lesson:
UEFI install + BIOS firmware = guaranteed boot failure.

This is extremely common in:

KVM
Proxmox VE
OpenStack

🔧 4️⃣ Fix #1 — Install OVMF (UEFI Firmware)

Your system didn’t even have OVMF installed:

/usr/share/OVMF/OVMF_CODE.fd → missing

We installed firmware package and found:

OVMF_CODE_4M.fd
OVMF_VARS_4M.fd

Then updated VM XML:

<loader readonly='yes' type='pflash'>
  /usr/share/OVMF/OVMF_CODE_4M.fd
</loader>
<nvram>...</nvram>

Now firmware matched disk layout.

💡 Lesson:
Hypervisor provides firmware. OS depends on firmware type.

🧨 5️⃣ New Symptom — UEFI Shell Appeared

After switching to UEFI, VM booted into:

Shell>

This meant:

UEFI firmware is working
EFI partition exists
But no boot entry in NVRAM

🧠 Why That Happened

When you switched firmware modes:

New NVRAM file was created
It had no stored boot entries
So firmware didn't know about /EFI/debian/grubx64.efi

This is expected behavior.

💡 Lesson:
UEFI boot depends on:

EFI partition
Bootloader file
NVRAM boot entry

All three must exist.

🛠 6️⃣ Manual Boot from EFI Shell

Inside shell:

fs0:
cd EFI\debian
grubx64.efi

GRUB launched → Debian booted.

This proved:

Disk is healthy
GRUB installed correctly
Root filesystem intact

💡 Lesson:
UEFI shell is a powerful debugging tool.

🔐 7️⃣ Root Password Problem

There is no default root password in Debian.

In modern Debian:

If you leave root password blank → root login disabled
You must use sudo user

You chose to reset password offline using:

guestmount + chroot

That demonstrated:

Mount qcow2 offline
Enter filesystem
Modify system safely

💡 Lesson:
Virtual disks are just files.
You can perform offline surgery safely.

🏁 8️⃣ Permanent Fix — Register GRUB Properly

After booting successfully:

grub-install
update-grub
efibootmgr

You got:

Boot0003* debian
BootOrder: 0003,...

That means:

GRUB installed to EFI partition
UEFI boot entry created
Boot order corrected

Now VM boots automatically.

💡 Lesson:
UEFI requires proper NVRAM registration.

📚 Full Timeline of Issues

Phase	Problem	Root Cause	Fix
1	Hang at boot	Firmware mismatch	Enable UEFI
2	OVMF missing	Firmware not installed	Install ovmf
3	XML validation errors	Wrong schema structure	Correct `<os>` block
4	UEFI shell	No NVRAM boot entry	Manual grub launch
5	Root login fail	No default password	Reset via chroot
6	Manual boot required	Boot entry not registered	`grub-install`

🧠 Core Concepts You Mastered

🔹 Firmware Layers

BIOS vs UEFI
SeaBIOS vs OVMF

🔹 Disk Layout

GPT vs MBR
EFI partition
Root partition
Swap

🔹 libvirt XML

<loader>
<nvram>
Machine type (q35)

🔹 UEFI Mechanics

EFI shell
.efi bootloaders
NVRAM boot entries
efibootmgr

🔹 Virtual Disk Forensics

qemu-img
virt-filesystems
guestmount
chroot repairs

🚀 Biggest Takeaway

The problem was never the OS.

It was the relationship between:

Firmware
↕
Bootloader
↕
Partition scheme
↕
Kernel

In virtualization, firmware mismatches are one of the most common boot failures.

Now you understand that chain completely.

🏆 What Level You’re At Now

You moved from:

“VM won’t boot”

To:

Debugging firmware, UEFI NVRAM, partition schemes, and GRUB inside KVM.

That’s real infrastructure-level troubleshooting.

Forem: iapilgrim

Master the ECR Lifecycle: Automating Image Cleanup

Why Lifecycle Policies?

The "Perfect" Policy for Dev/Staging

1. The "Untagged" Rule (Priority 1)

2. The "Age" Rule (Priority 2)

The JSON Configuration

Critical Concepts: sinceImagePushed vs. imageCountMoreThan

How to Test Without Breaking Production

The 24-Hour Rule

Pro-Tip: Infrastructure as Code (Terraform)

Deploying a Base Sepolia Node with Docker

1. Prerequisites

2. The Automated Setup Script

3. Critical Configuration Check

4. Troubleshooting & Mastery 🔍

Layer 1: The JWT Handshake

Layer 2: L1 Connectivity

Layer 3: P2P Peering

5. Monitoring Your Progress

Summary

Automating Container Image Updates with FluxCD (Hands-On Tutorial)

1. What is Flux Image Automation?

2. Repository Structure Used in This Lab

3. Application Deployment

4. Image Automation Setup

Image Repository

Image Policy

Image Update Automation

5. Triggering Automation

6. Troubleshooting a Common Error

7. Root Cause

8. Fix: Recreate the Flux Git Secret with Write Access

Step 1 — Generate a New SSH Key

Step 2 — Add the Public Key to GitHub

Step 3 — Recreate the Flux Git Secret

Step 4 — Re-run Automation

9. Helpful Debug Commands

10. Best Practices

Conclusion

FluxCD Image Automation Error Troubleshooting

Problem

Root Cause

Diagnosis Steps

1. Check Image Automation Status

2. Inspect the Automation Object

3. Check the Git Source

4. Confirm Git Authentication

Fix Implemented

1️⃣ Generate SSH Key

2️⃣ Add Public Key to GitHub

3️⃣ Recreate Flux Git Secret

4️⃣ Trigger Automation

Expected Result

Useful Debug Commands

Best Practice

FluxCD journey with Minikube

🚀 Phase 1: The Manual Foundation

🛠️ Step 1: Install Tools

🏗️ Step 2: Start Minikube

📂 Step 3: Directory Layout

📄 Step 4: The Manifests

🚀 Step 5: Deploy Manually

🤖 Phase 2: The Great Automation (FluxCD)

🛠️ Step 1: Environment Setup

🏗️ Step 2: Bootstrap Flux

📂 Step 3: Final Git Directory Layout

📄 Step 4: Create the "Sync" Instruction

🚀 Step 5: Push and Pray (The GitOps Way)

🔐 Phase 3: The Secret Sauce (Sealed Secrets)

🏗️ Step 1: Install Infrastructure

📂 Step 2: Final Phase 3 Directory Layout

🔐 Step 3: Create an Encrypted Secret

🚀 Step 4: Deploy

🧠 Summary of Progress

🧠 Migrating BSC Testnet Data to a New Disk (KVM + Docker)

📊 Initial Situation

🎯 Goal

🏗️ Architecture Before

🚀 Architecture After

Critical Concepts: `sinceImagePushed` vs. `imageCountMoreThan`