Forem: Hamdi (KHELIL) LION

🧠 Demystifying Metrics in Kubernetes

Hamdi (KHELIL) LION — Mon, 09 Feb 2026 07:48:25 +0000

Kubernetes does not magically know when to scale your workloads 🤖
It relies on metrics exposed through dedicated APIs to make scaling decisions.

There are three types of metrics you need to understand:

⚙️ Resource Metrics
📊 Custom Metrics
🌍 External Metrics

Each one represents a different kind of pressure on your system.

🤔 Why Metrics Matter for Autoscaling

Autoscaling in Kubernetes is mainly handled by the Horizontal Pod Autoscaler (HPA).

The HPA keeps asking:

“Hey… are my Pods struggling?” 😅

The answer comes from metrics. Without them, Kubernetes is basically guessing.

⚙️ 1 Resource Metrics = Pod Health Signals

These are the native Kubernetes metrics.

They come from the Metrics Server and only cover:

🧮 CPU usage
🧠 Memory usage

They are exposed via:

metrics.k8s.io

🧩 What they represent

They describe resource consumption, not business traffic.

Your app might be slow because of a database… but CPU could still be chill 🧊

📄 Example HPA based on CPU

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

If average CPU across Pods goes above 70 percent → more replicas 🔥

✅ Pros

Super simple
Works out of the box

❌ Limits

CPU is not always equal to user traffic
Memory often reacts too late

📊 2 Custom Metrics = Your App Talking to Kubernetes

Custom metrics come from applications inside your cluster.

They describe business or application load 💼

They are exposed through:

custom.metrics.k8s.io

🔁 Typical data flow

App exposes /metrics
Prometheus scrapes
Prometheus Adapter maps metrics → Kubernetes API

Now Kubernetes can ask:

“How busy is this Deployment really?” 👀

🌐 Example 1 HTTP requests per second

Metric in Prometheus:

http_requests_per_second

🧠 What it means
Real traffic handled by each Pod

📦 In Kubernetes
A metric attached to Pods

metrics:
- type: Pods
  pods:
    metric:
      name: http_requests_per_second
    target:
      type: AverageValue
      averageValue: 200

If each Pod handles more than 200 requests per second → scale out 🚀

⏱ Example 2 Request duration

Metric:

request_duration_seconds

🧠 What it means
Application performance and saturation

Used as an Object metric:

- type: Object
  object:
    metric:
      name: avg_request_duration
    describedObject:
      apiVersion: apps/v1
      kind: Deployment
      name: api
    target:
      type: Value
      value: 0.5

If average latency goes above 500ms → time to add Pods 🏃‍♂️

🧵 Example 3 Active background jobs

Metric:

active_background_jobs

🧠 What it means
Internal workload of a worker

Each Pod reports its own load, and HPA scales when workers are overloaded 📈

✅ Pros

Scaling reflects real app behavior
Way smarter than CPU only

❌ Limits

Requires Prometheus + Prometheus Adapter
More components to maintain 😬

🌍 3 External Metrics = Work Waiting Outside the Cluster

External metrics come from systems outside Kubernetes.

They are exposed via:

external.metrics.k8s.io

These metrics describe work your Pods must process, even if it lives elsewhere 🌎

📬 Example 1 SQS queue length

Metric:

ApproximateNumberOfMessagesVisible

🧠 What it means
Number of messages waiting in the queue

In Kubernetes
A global metric not tied to specific Pods

If the queue grows → spawn more workers 💪

🐘 Example 2 Kafka consumer lag

Metric:

kafka_consumer_lag

🧠 What it means
Delay between producers and consumers

More lag = your consumers are falling behind 😱
Scale them up!

📦 Example 3 Redis job queue size

Metric:

redis_list_length

🧠 What it means
Number of jobs waiting in Redis queues

Perfect for worker autoscaling 🔄

⏰ Example 4 Time based scaling

Scale more Pods during office hours
Scale down at night 🌙

This is also treated as an external signal, because it’s not tied to Pod resource usage.

🧮 How HPA Uses These Metrics

HPA periodically queries:

Metric Type	API	What it measures
Resource	metrics.k8s.io	CPU and memory
Custom	custom.metrics.k8s.io	App level load
External	external.metrics.k8s.io	Event or system load

Then it calculates how many replicas you need 📐

⚡ KEDA vs Prometheus Adapter

Here comes the game changer 🎮

KEDA (Kubernetes Event Driven Autoscaling) focuses on event driven autoscaling and makes custom and external metrics way easier to use.

🧩 With Prometheus Adapter

You must:

Run Prometheus
Install and configure the Adapter
Write mapping rules
Manage RBAC and certs

It works, but it’s heavy 🏋️

⚡ With KEDA

You define a ScaledObject and KEDA does the magic ✨

Example SQS scaler:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker-scaler
spec:
  scaleTargetRef:
    name: worker
  minReplicaCount: 0
  maxReplicaCount: 20
  triggers:
  - type: aws-sqs-queue
    metadata:
      queueURL: https://sqs.eu-west-1.amazonaws.com/123/my-queue
      queueLength: "10"
      awsRegion: eu-west-1

KEDA fetches the metric
Exposes it to Kubernetes
Creates and manages the HPA
Handles provider auth 🔐

All without Prometheus Adapter 🤯

🧭 When to Use What

If you want to scale on…	Use…
CPU or memory	Resource metrics
HTTP traffic or app load	Custom metrics
Queues, streams, SaaS	External metrics + KEDA
Events or scale to zero	KEDA

🎯 Final Takeaway

Kubernetes becomes truly powerful when scaling is driven by real workload signals, not just CPU.

⚙️ Resource metrics are the starting point
📊 Custom metrics bring application awareness
🌍 External metrics unlock event driven architectures
⚡ KEDA makes advanced autoscaling simple and production friendly

Once you understand these three metric types, autoscaling stops being magic and becomes a design tool you control 💡🚀

Happy clustering :)

Building a Kubernetes homelab the hard but right way 🧱☸️

Hamdi (KHELIL) LION — Tue, 20 Jan 2026 10:32:46 +0000

If you have been following my previous articles, you already know the drill 😄
I like setups that are:

boring
repeatable
close to real production

This article is not about spinning up a quick cluster.
It is about building a platform.

A platform you can:

rebuild from scratch
scale without fear
explain to a client or a teammate
reuse in real consulting missions

And yes, it runs at home 👀

The mindset first 🧠

Before touching any tool, I decided on one rule:

👉 no manual action that I cannot reproduce with code

That single rule drives everything else.

So the stack becomes very natural:

Proxmox for virtualization
Terraform to create machines
Cloud Init to bootstrap the OS
Kubespray to install Kubernetes
Helmfile for everything after day 1

Each layer has one job only.
No overlap. No shortcuts.

The big picture 🗺️

Here is what actually happens when I type terraform apply 👇

Proxmox clones VMs from a cloud init template
Each VM boots with the right user, SSH keys and network
Nodes are reachable immediately
Kubespray turns them into a real HA Kubernetes cluster
Helmfile deploys platform components on top

Once you understand that flow, debugging becomes easy and scaling becomes boring.
And boring is good 😌

Step 1 Proxmox as a real compute layer ⚙️

I treat Proxmox like a private cloud, not like a lab UI.

No clicking. No guessing.

Terraform talks directly to the Proxmox API and creates Kubernetes nodes exactly the same way every time.

What I am doing here

cloning from a golden cloud init template
injecting user data via snippets
assigning static IPs
tagging nodes for clarity
keeping compute concerns separate from Kubernetes

At this stage, Kubernetes does not exist yet.
And that is exactly what I want 👍

Step 2 The cluster is just data 📐

This is one of my favorite parts.

The cluster topology lives in a tfvars file.
No logic. No magic. Just data.

This is extremely important because it means:

the same code can create one cluster or ten
topology changes do not require refactoring
environments stay consistent over time

Adding a node or a whole new cluster is just editing data 🔁

Step 3 Reusable Terraform modules 🧩

Everything goes through a compute module.

Terraform is responsible for:

machines
networking
bootstrapping access

And nothing else.

Once the VMs exist, Terraform is basically done.

This clear boundary avoids a lot of confusion later.

Step 4 Kubespray does the heavy lifting ☸️

Kubespray is where Kubernetes actually comes to life.

It handles:

HA control plane
stacked etcd
container runtime
CNI and kubelet configuration
sane defaults and hardening

This is not a quick kubeadm script.
This is a production grade installer that I fully trust.

Step 5 Bootstrap only the essentials 🚦

At cluster creation time, I only enable what is strictly required:

metrics server
ingress nginx
metallb
gateway api

Nothing fancy. Nothing opinionated.

The goal here is to have a usable Kubernetes cluster, not a fully loaded platform.

Everything else can wait.

Step 6 Day 2 is simple and explicit with Helmfile 📦

This is important.

There is no fancy GitOps setup here.

No Argo CD bootstrap.
No Flux managing Flux.
No recursive GitOps inception 😄

Instead:

Helmfile is run explicitly
changes are intentional
failures are visible
debugging stays simple

For a single cluster or a homelab, this is often the best tradeoff.

When GitOps really makes sense 🚀

Now, things change when you have multiple clusters to manage.

At that point, this setup becomes extremely powerful.

Because:

Terraform already creates the machines
Kubespray already installs Kubernetes
Helmfile already describes the platform state

You can easily plug a GitOps layer on top to:

create a new cluster from scratch
apply a standard baseline
end up with a ready to use Kubernetes platform
with almost zero manual action

Think of it as:
👉 one command to pop a full Kubernetes cluster, ready for workloads

This is where Argo CD or Flux start to shine, but only when the scale justifies it.

Until then, keeping things simple is often the most mature choice 🧘

Why I like this approach so much ❤️

Because it grows with you.

simple when you have one cluster
scalable when you have many
understandable at every step
close to how real platforms are built

No magic.
No hidden automation.
Just solid engineering.

If you can run this at home, you can run it anywhere 🌍

What is next 🔮

In upcoming articles I will dive into:

storage choices and csi
dns and external-dns
security with kyverno
observability and metrics
multi cluster patterns

Stay tuned 👋 and happy clustering :)

Building a Kubernetes HomeLab - The Hard Way - DNS first, or everything else lies to you 🔥🌐

Hamdi (KHELIL) LION — Mon, 22 Dec 2025 08:08:54 +0000

Let’s be very clear from the start 👇
If DNS is shaky, everything above it becomes unreliable, misleading, and borderline hostile.

You will blame Kubernetes.
You will blame cert-manager.
You will blame yourself.

And it will still be DNS 😅

This article builds a real DNS layer, outside the cluster, the way it is done in production environments.
Not fancy. Not clever. Just solid.

DNS is not a feature, it is the foundation 🧱

Most homelabs treat DNS like a checkbox:

router default
maybe 1.1.1.1
maybe 8.8.8.8
call it a day

This works until you introduce:

multiple Proxmox nodes
Kubernetes bootstrap
internal TLS
ExternalDNS
service to service communication At that point, DNS stops being optional. It becomes infrastructure.

If this layer lies, everything else lies with it.

what I assume before we start breaking things 🔧

This article assumes:

you are comfortable with Linux and SSH
you control your LAN
you can assign static IPs
you are fine rebuilding things if needed

Network baseline used in examples:

LAN CIDR: 192.168.1.0/24
router: 192.168.1.1
DNS node: 192.168.1.169
internal domain: home.arpa

Why home.arpa?

defined by RFC 8375
designed for private networks
no collision with public DNS
predictable behavior across operating systems

.local is convenient.
.local is also a trap ☠️

how DNS flows in this homelab, no magic involved 🧭

The DNS request path is intentionally boring:

client
→ Pi-hole
→ Unbound
→ root servers
→ TLD servers
→ authoritative servers

Why this design?

Pi-hole sits in front:

single DNS endpoint for the entire LAN
full visibility into queries
local DNS records for infrastructure
API compatible with ExternalDNS later

Unbound sits behind:

full recursive resolver
DNSSEC validation
zero dependency on Google or Cloudflare
behaves like enterprise DNS stacks

One DNS server for the LAN:

deterministic behavior
easier troubleshooting
required for Kubernetes stability

building the DNS node, properly and calmly 🧱 A Raspberry Pi that does exactly one thing 🥧

Reserve a static IP from your router:

hostname: dns-01
ip: 192.168.1.169

Update the system:

sudo apt update
sudo apt upgrade -y
sudo apt install curl gnupg lsb-release -y

This node is sacred:

DNS only
no Docker
no Kubernetes
no experiments

If DNS breaks, you want zero doubt about the cause.

installing Pi-hole, the DNS entry point 🚦

Pi-hole will be the only DNS server exposed to the LAN.

Install it:

curl -sSL https://install.pi-hole.net | bash

During the installer:

interface: eth0
upstream DNS: custom
do not select any public DNS provider
enable the web admin interface

Once installed:

http://192.168.1.169/admin

Immediately change the admin password:

pihole -a -p

At this stage:

Pi-hole is answering DNS queries
but it should not resolve anything yet

installing Unbound, the actual DNS resolver 🔁

Unbound will do the real work: recursive resolution and DNSSEC validation.

Install Unbound:

sudo apt install unbound -y

Create a dedicated configuration for Pi-hole:

sudo tee /etc/unbound/unbound.conf.d/pi-hole.conf > /dev/null <<EOF
server:
  interface: 127.0.0.1
  port: 5335
  do-ip4: yes
  do-ip6: no
  do-udp: yes
  do-tcp: yes
  root-hints: "/var/lib/unbound/root.hints"
  harden-glue: yes
  harden-dnssec-stripped: yes
  use-caps-for-id: yes
  edns-buffer-size: 1232
  prefetch: yes
  verbosity: 1

forward-zone:
  name: "."
  forward-addr: 0.0.0.0
EOF

Fetch root hints:

sudo curl -o /var/lib/unbound/root.hints https://www.internic.net/domain/named.root

Enable and start Unbound:

sudo systemctl restart unbound
sudo systemctl enable unbound

At this point:

Unbound listens on 127.0.0.1:5335
performs full recursive resolution
validates DNSSEC

connecting Pi-hole to Unbound, the clean way 🔗

In the Pi-hole admin UI:

settings
dns
upstream DNS servers
custom server: 127.0.0.1#5335
disable all other upstream resolvers

Now the chain is complete:

clients talk to Pi-hole
Pi-hole forwards to Unbound
Unbound talks to the internet

No leaks.
No shortcuts.

naming things like an adult 🏷️

internal DNS records

Define local DNS records in Pi-hole:

proxmox-01.home.arpa   192.168.1.41
proxmox-02.home.arpa   192.168.1.42
proxmox-03.home.arpa   192.168.1.43
nas-01.home.arpa       192.168.1.50

ip plan

component   | ip
router          | 192.168.1.1
dns-01          | 192.168.1.169
proxmox-01  | 192.168.1.41
proxmox-02  | 192.168.1.42
proxmox-03  | 192.168.1.43
synology    | 192.168.1.50

Predictable naming beats clever naming.
Every single time.

proof before trust, always 🔍

Test recursive resolution:

dig google.com @192.168.1.169

Test DNSSEC:

dig dnssec-failed.org @192.168.1.169

Expected result:

SERVFAIL means DNSSEC is working ✅ Test internal names:

dig proxmox-01.home.arpa

Watch live DNS traffic:

pihole -t

If something breaks later, this is where you start.

how people accidentally sabotage their own DNS ☠️

Adding a public DNS as secondary:

internal names stop resolving
ExternalDNS becomes unreliable
cert-manager fails silently

Using .local:

mDNS conflicts
inconsistent resolution
debugging hell

Running DNS inside Kubernetes first:

bootstrap deadlock
cluster depends on itself
guaranteed pain

why this DNS layer unlocks everything else 🚀

With this setup in place:

Proxmox nodes resolve consistently
Kubernetes nodes bootstrap cleanly
ExternalDNS can manage records safely
cert-manager can issue internal certificates
ingress becomes predictable

This is the point where the homelab stops feeling fragile.

Next article:
Teaching your router who’s boss 🧠📡

lessons learned the hard way, so you do not have to 😅

DNS should be boring.
Silent.
Predictable.

If DNS is exciting, something is wrong.

This setup is:

reproducible
observable
production-inspired
intentionally boring

Hard now.
Peace later.

Happy Clustering :)

🏗️ Setting Up a Proxmox Cluster for My Homelab Kubernetes Environment

Hamdi (KHELIL) LION — Wed, 05 Nov 2025 06:27:25 +0000

Hey everyone 👋

Now that you know the hardware powering my homelab
it is time to show how I turned those machines into a Proxmox cluster ready to host a production style Kubernetes environment 🧑‍💻⚙️

This article covers the entire setup
including fixes for a few annoying real world problems 😅

Let’s get into it 🚀

✅ Step 1. Update BIOS on Lenovo Tiny Nodes

Before installing Proxmox
I strongly recommend updating the BIOS of each node

Why it matters
✅ fixes stability bugs
✅ improves CPU performance and thermal handling
✅ avoids random shutdowns or throttling
✅ unlocks better hardware support

Old BIOS = unpredictable issues later
Flash now and enjoy peace later ✅

✅ Step 2. Install Proxmox VE

I installed Proxmox VE on each of my three Lenovo M920q nodes
using this simple naming and addressing plan:

Node Hostname IP

Node 1 pve1 192.168.1.21
Node 2 pve2 192.168.1.22
Node 3 pve3 192.168.1.23

DNS points to my Raspberry Pi ✅

Installation takes a few minutes
and Proxmox is ready through a browser 🎉

✅ Step 3. Fix NIC Offloading Stability Issues

This one hit me hard 😬
I kept encountering:

❌ random node connectivity loss
❌ nodes freezing during network traffic
❌ SSH dropping without reason

Cause
👉 NIC offloading on Intel network adapters

Fix using this excellent community script
https://community-scripts.github.io/ProxmoxVE/scripts?id=nic-offloading-fix

After applying this
the cluster became rock solid ✅

If you run Lenovo Tiny
this fix is almost mandatory 👀

✅ Step 4. Create the Proxmox Cluster

On node pve1:

pvecm create homelab-cluster

On other nodes:

pvecm add 192.168.1.21

Now
I can manage all nodes from a single interface ✅
VMs can migrate easily between hosts ✅
Monitoring is unified ✅

Production vibes at home 😎

✅ Step 5. Flat Network Setup First

Managed Switch Comes Later

For now
the network is simple:

one Gigabit flat network

vmbr0 bridging the physical NIC

DHCP/static addressing for VMs

Benefits
✅ simple
✅ stable
✅ easy to troubleshoot

Future upgrade:
🔹 Managed PoE switch
🔹 VLAN segmentation for Kubernetes traffic
🔹 Insights and better monitoring

Great now
but room to grow ✅

✅ Step 6. Main Storage: Synology NAS (NFS + iSCSI)

Persistent workloads require persistent storage
so my Synology NAS handles all Kubernetes storage

✅ NFS for most apps
✅ iSCSI for database style block volumes
✅ RAID and snapshots for safety
✅ Backup target for Proxmox VMs

The NAS is the data backbone of the homelab
exactly how storage is handled in production 🚀

✅ Step 7. Automated Ubuntu 25.x Templates With Packer

To avoid manual VM setup
I automated template building using Packer

Operating system
🧩 Ubuntu Server 25.x
Modern kernel and great container support

My Packer template creates VMs that are:
✔ cloud init ready
✔ qemu guest agent installed
✔ ssh enabled
✔ updated and hardened
✔ swap disabled for Kubernetes

This allows me to deploy
a brand new Kubernetes node VM in under 30 seconds 🤯

Fast
repeatable
version controlled

Infrastructure as code for real 😁

⚠️ Important Real World Gotchas

Here are issues I personally hit
and how I fixed them ✅

Problem Cause Solution

Node freezes or network drops NIC offloading Apply NIC fix ✅
Unexpected reboots or throttling Old BIOS Flash BIOS ✅
High CPU temps under load Small chassis Tune cooling profile ✅
VMs or pods OOM Overcommit Reserve RAM for hypervisor ✅

Homelabs teach you technology
by breaking and fixing it
the best learning path 🔥

✅ What We Have Achieved So Far

Feature Status

Three node Proxmox cluster ✅
Network stability ✅
NAS integrated for storage ✅
Custom VM templates ✅
Automation friendly ✅
Kubernetes ready foundation ✅

A real mini datacenter is now running in my home
quietly
efficiently
and fully under my control 🏡✨

✅ Coming Next

Deploying Kubernetes with Kubespray

In the next article you will see:

🔥 3 high availability control plane nodes
🔥 6 workers ready for workloads
🔥 DNS fully internal
🔥 main storage already integrated
🔥 GitOps and monitoring soon after

This is when the real fun begins 😎🔥

Stay tuned

🧰 The Hardware That Powers My Homelab Kubernetes Cluster

Hamdi (KHELIL) LION — Tue, 28 Oct 2025 09:44:55 +0000

Hey everyone 👋

Before diving deeper into the software stack of my homelab
I want to fully introduce the physical components that make this whole project come alive 🧑‍🔧

This article focuses entirely on hardware choices and why each element matters

Let’s explore the building blocks of my mini data center 🏡⚡

🖥️ The Compute Layer

Three Lenovo ThinkCentre M920q Tiny

These three machines are the heart of the homelab
They provide the CPU and memory resources needed to run Kubernetes workloads inside virtual machines

Why I chose them
✅ compact and silent
✅ excellent performance per watt
✅ reliable and low power for 24x7
✅ scalable memory support

Specs per node

Intel CPU with multiple cores
32 GB RAM
NVMe SSD for fast VM storage

Together they give me:
🧠 96 GB RAM total
⚡ 36 vCPUs total

This allows me to:

simulate real production workloads
run 3 HA control plane nodes
run 6 worker nodes
add utility VMs when needed

Small machines
big power 😁

🌐 The Networking Layer

Gigabit Switch Today

Managed PoE Switch Tomorrow

Right now my network is built on a simple Gigabit Ethernet switch
Flat network
no VLANs
no advanced routing

This keeps things:
✅ simple to maintain
✅ easy to debug
✅ perfectly functional for a first iteration

But I am already thinking ahead 👀

A managed switch with PoE is on my radar because it would allow:
🧩 VLAN segmentation for test environments
📡 powering future access points or cameras directly from the switch
🛡️ network isolation for storage, monitoring or nodes
📈 better performance analysis through advanced metrics

For now
simplicity wins
but I want room to grow

🍓 The Brain of the LAN

Raspberry Pi as DNS Server

DNS is a critical piece in any infrastructure
especially Kubernetes

The Raspberry Pi handles:
✅ local DNS resolution
✅ service discovery across the home network
✅ potential Pi hole extension later

It ensures zero reliance on public DNS for internal services
Fast
lightweight
and rock solid 🧩

📦 The Data Layer

Synology NAS for Storage and Backups

Applications need persistent data
databases
monitoring stack
GitOps metadata

The Synology NAS provides:
✅ NFS shares for general storage
✅ iSCSI volumes for testing block level workloads
✅ snapshots and RAID protection
✅ backup space for Proxmox VMs

It is the data backbone of the homelab
keeping stateful workloads safe 🧡

🔌 Hardware Architecture Diagram

Here is a more complete view of the hardware layout

                Internet
                    │
                 Router
               with VPN
                    │
             ┌──────┴──────┐
             │             │
       Gigabit Switch      │
    (Flat network today    │
     but future Managed PoE)
             │
 ┌───────────┼───────────────┬───────────┐
 │           │               │           │
[Node 1]  [Node 2]        [Node 3]   Synology NAS
Proxmox   Proxmox         Proxmox         
 │         │               │              
 └─────────┴───────────┬───┘              
                       │                  
                Raspberry Pi (DNS)

Everything works together as a single coherent platform
Compute + Networking + DNS + Storage
self hosted and fully controlled ✅

🎯 Why this hardware matters

Every piece of this hardware plays a key role

Hardware	Role
Lenovo Tiny Nodes	Compute layer for Kubernetes
Gigabit Switch	Fast and reliable local communications
Managed PoE Switch future	Network control and segmentation
Raspberry Pi	DNS authority for all internal traffic
Synology NAS	Persistent data and VM backup storage

This creates a production style environment
ready for real workloads and real failures
without cloud overhead or noise pollution

A smart and scalable foundation 🤝

✅ Up Next

Next article in the series
➡️ Turning this hardware into a Proxmox cluster
➡️ VM management
➡️ Template strategy
➡️ Solid virtualization layer for Kubernetes

Stay tuned
The fun is only beginning 😎

🏡 Why I Built My Own Homelab to Run Kubernetes

Hamdi (KHELIL) LION — Sun, 26 Oct 2025 20:10:21 +0000

Hey everyone 👋

If you have been following my posts, you might have noticed that I have been pretty quiet lately 😅
There is a good reason for that

I have been spending a lot of time building my own homelab a place where I can safely test, break, and rebuild complex infrastructures without worrying about cloud costs or production risks 🧑‍💻

My goal is simple
Create a production grade Kubernetes environment at home using Proxmox for virtualization and Kubespray for cluster provisioning 💡
This environment allows me to test real world scenarios and also experiment with features for my Kubermates project 🚀

Let me walk you through the why and the how

⚙️ The Motivation

As a DevOps and SRE engineer, Kubernetes is part of my daily life.
But testing new tools, scaling strategies, GitOps workflows, or security rules often requires:

✅ complete freedom
✅ zero fear of downtime
✅ repeatability
✅ controlled costs

A homelab gives me full control of the stack including: network, storage, virtualization, routing, DNS, automation and resilience strategies 💪

It rapidly became much more than a hobby. It is now my mini data center

🧩 Proxmox as the Foundation

To host everything, I chose Proxmox VE

Reasons behind this choice:
🖥️ clean and powerful web UI and CLI
⚙️ excellent virtualization and storage integration
🌐 flexible networking and clustering
🔁 snapshots and backup system made for tests and disasters

Each hypervisor node runs multiple virtual machines that will become control planes, workers or utility nodes like DNS and monitoring services

Proxmox gives me the freedom to rebuild everything as many times as needed

☁️ Kubernetes Deployment With Kubespray

Once the virtual machines were ready, I needed a repeatable approach to deploy Kubernetes.
I selected Kubespray because:

🔥 it is production proven
🧩 uses Ansible which makes automation easy
🔧 allows deep customization
🔄 supports highly available clusters

This lets me rebuild the entire cluster in minutes while keeping the configuration under version control

Perfect for experimentation

💪 Homelab Specs

Right now, my homelab is running with:

🧠 96 GB of RAM
⚡ 36 vCPUs

This gives enough headroom to simulate:

Autoscaling and performance constraints

Monitoring with Prometheus, Loki, Thanos etc

GitOps flows with Argo CD and Helmfile

Network isolation and multi tenancy

Production security patterns

Computing constraints actually make the optimization challenges even more interesting

🔬 A Playground for Kubermates

If you know me, you know I am working on Kubermates
This homelab is now my real life testing ground for the platform

Here I can:
✅ test onboarding flows and automation patterns
✅ simulate multi tenant setups
✅ validate best practices for deploy and operate
✅ try both cluster and application level policies

Everything is reproducible and controlled
Exactly what I need to test responsibly

🌐 Secure Remote Access With WireGuard VPN

Although everything runs at home, I still want to access the environment securely when I travel or work away from my desk

For this, I use WireGuard VPN running on a small gateway VM

Why WireGuard

🔒 strong encryption

⚡ ultra fast performance

🧩 minimal configuration

📱 clients for Linux, Mac, Windows, iOS and Android

My access architecture:

Only VPN traffic can reach the Proxmox cluster and Kubernetes nodes

No ports are exposed directly to the internet

Internal DNS and Ingress work exactly as inside the local network

So I get full access to Grafana, Argo CD, dashboards, kubectl and SSH
as if I was physically at home 🏠

This keeps everything private and secure
Perfect for a personal lab

🧠 Key Lessons Learned So Far

Building Kubernetes at home gives you a deeper understanding of what is behind the cloud curtain

Main takeaways

Networking is always trickier than expected 😅

Local DNS matters more than anyone admits

Good VM templates save hours of setup time

Resource constraints teach real FinOps awareness

Breaking things makes you learn faster

And the best part
It is incredibly fun

🚀 What Comes Next

This is just the beginning

Upcoming articles in the series
1️⃣ Proxmox hardware and virtualization best practices
2️⃣ Kubespray automation and cluster bootstrapping
3️⃣ DNS and networking architecture for homelabs
4️⃣ GitOps, certificates and observability stack deployment
5️⃣ How to simulate production workloads at home

The goal
Help anyone build a professional grade Kubernetes experience locally
without cloud billing surprises

Thanks for reading and stay tuned for more 🔥

🧩 GitHub Actions Composite vs Reusable Workflows

Hamdi (KHELIL) LION — Fri, 18 Jul 2025 08:41:53 +0000

How to standardize and supercharge your CI/CD pipelines across projects

When your teams manage multiple projects with similar deployment patterns, repeating the same GitHub Actions steps over and over can become tedious, error-prone, and hard to maintain

Thankfully, GitHub Actions offers two powerful solutions to help standardize, reuse, and scale your CI/CD pipelines: Composite Actions and Reusable Workflows. When used together, they form a clean, modular, and DRY (don’t repeat yourself) CI/CD strategy

🔄 Composite Actions vs Reusable Workflows

🧱 Composite Actions

Composite Actions allow you to group steps (like docker build, terraform plan, or trivy scan) into a reusable component. Think of it like a function.

Best for:

Reusing logic (e.g., build and push images)
Hiding complex logic from the main workflow
Keeping workflows minimal and maintainable

Use it with:

uses: ./.github/actions/your-action

🔁 Reusable Workflows

Reusable Workflows are full workflows that can be invoked with inputs and secrets.

Best for:

High-level CI/CD orchestration (build, test, scan, deploy)
Enforcing common practices across repositories
Abstracting full pipelines

Use it with:

uses: org/repo/.github/workflows/your-workflow.yml@ref

🐳 Example: Docker Build and Scan with Trivy

Here is an example of a composite action to build and push a Docker image to Azure Container Registry (ACR) and scan it using Trivy.

`.github/actions/build-and-scan/action.yml`

name: Docker Build and Push

inputs:
  dockerfile:
    default: Dockerfile
  context:
    default: .
  image_name:
    required: true
  tag:
    required: true
  build_args:
    default: ""
  report_name:
    default: trivy-report

runs:
  using: "composite"
  steps:
    - uses: azure/login@v2
      with:
        creds: ${{ env.AZURE_CREDENTIALS }}

    - uses: azure/docker-login@v2
      with:
        login-server: ${{ env.CONTAINER_REGISTRY }}
        username: ${{ env.ACR_USERNAME }}
        password: ${{ env.ACR_PASSWORD }}

    - name: Build and Push Image
      shell: bash
      run: |
        docker build ${{ inputs.build_args }} \
          -t ${{ env.CONTAINER_REGISTRY }}/${{ inputs.image_name }}:${{ inputs.tag }} \
          -f "${{ inputs.dockerfile }}" "${{ inputs.context }}"
        docker push ${{ env.CONTAINER_REGISTRY }}/${{ inputs.image_name }}:${{ inputs.tag }}

    - name: Scan with Trivy
      uses: aquasecurity/trivy-action@0.32.0
      with:
        image-ref: '${{ env.CONTAINER_REGISTRY }}/${{ inputs.image_name }}:${{ inputs.tag }}'
        format: 'sarif'
        output: 'trivy-results.sarif'
        exit-code: 0

    - name: Upload SARIF to GitHub Security tab
      uses: github/codeql-action/upload-sarif@v3
      with:
        sarif_file: 'trivy-results.sarif'
        category: trivy

Usage in Workflow

- name: Build and Scan
  uses: ./.github/actions/build-and-scan
  with:
    image_name: my-app
    tag: ${{ github.sha }}

🌍 Example: Terraform/Terragrunt Validation with Checkov

Now let’s see a reusable workflow to validate Terraform code and scan for misconfigurations using Checkov.

`.github/workflows/infra-check.yml`

name: Infra Validation Workflow

on:
  workflow_call:
    inputs:
      working_dir:
        required: true
        type: string

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Terragrunt and OpenTofu
        run: |
          curl -s https://raw.githubusercontent.com/opentofu/opentofu/main/scripts/install.sh | bash
          curl -L https://github.com/gruntwork-io/terragrunt/releases/latest/download/terragrunt_linux_amd64 -o terragrunt
          chmod +x terragrunt
          sudo mv terragrunt /usr/local/bin/

      - name: Validate HCL Syntax
        run: |
          terragrunt hclfmt --terragrunt-check --terragrunt-diff --recursive
        working-directory: ${{ inputs.working_dir }}

      - name: Check Misconfigurations with Checkov
        uses: bridgecrewio/checkov-action@v12
        with:
          directory: ${{ inputs.working_dir }}
          quiet: true

Usage in Project Workflow

jobs:
  call-validation:
    uses: your-org/your-repo/.github/workflows/infra-check.yml@main
    with:
      working_dir: infrastructure/staging/eu-west-1

⚙️ Best Practices

✅ Keep composite actions in .github/actions
✅ Keep reusable workflows in .github/workflows
✅ Abstract frequently repeated patterns
✅ Use inputs, outputs, and env vars to increase flexibility
✅ Use workflow_call and workflow_dispatch to structure triggers
✅ Defer secrets to the calling workflow

💡 Conclusion

Using composite actions and reusable workflows together can transform your CI/CD processes into clean, repeatable, and secure pipelines. Whether you're building containers, provisioning infrastructure, or scanning for vulnerabilities, this modular approach will scale with your teams and your systems.

Enjoy building better pipelines !

🚀🌐 Elevating Infrastructure: From Terraform/Terragrunt Foundations to Platform Engineering 😊

Hamdi (KHELIL) LION — Fri, 18 Apr 2025 10:24:24 +0000

Hey there, cloud adventurers! 🚀 Let’s chat about why keeping Terraform (or OpenTofu) and Terragrunt in their own lanes is absolutely essential—and how using Terraform JSON tfvars makes life easier when you’re building nifty tools on top. Ready? Let’s dive in! 😄

Why Separation Is a Must, Not an Option 🙅‍♂️🙅‍♀️

It might be tempting to mix Terraform and Terragrunt into one big file—after all, they work together, right? But trust me, keeping them decoupled is a game‑changer:

🔒 Clear boundaries: Terraform focuses purely on resource definitions, while Terragrunt handles orchestration, remote state, and DRY patterns.
🔄 Independent versioning: You can upgrade your Terraform (or OpenTofu) modules (semantic versioning FTW!) without touching your Terragrunt configs—and vice versa!
📦 Reusability everywhere: Modules stay generic and shareable across projects when they don’t carry Terragrunt baggage.

By giving each tool its own space, you simplify CI/CD, make upgrades safer, and keep responsibilities crystal‑clear. Win‑win! 🏆

Module Versioning with Terraform 🎯

Hosting Terraform (or OpenTofu) modules in a versioned repo (GitHub, GitLab, private registry—you choose!) lets you tag releases like v1.2.3. Then, Terragrunt can lock to that exact version:

// modules/storage-account/main.tf
resource "azurerm_storage_account" "sa" {
  name                     = var.account_name
  resource_group_name      = var.resource_group_name
  location                 = var.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

variable "account_name" {
  type = string
}

variable "resource_group_name" {
  type = string
}

variable "location" {
  type = string
}

// modules/storage-account/versions.tf
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 3.0"
    }
  }
  required_version = ">= 1.5"
}

Publishing this as, say,

git::ssh://git@github.com/myorg/terraform-modules-repo.git//storage-account?ref=v1.0.0

means everyone downstream knows exactly what they’re getting—no surprises! 🎉

Decoupling Terraform (or OpenTofu) & Terragrunt 🛠️

For true decoupling, host your Terraform/OpenTofu modules in a separate Git repository, each with its own lifecycle, version tags, and release process. Then reference those modules from Terragrunt using Git sources. This keeps module development and environment orchestration fully independent.

Example Terragrunt config:

# live/dev/terragrunt.hcl
terraform {
  source = "git::ssh://git@github.com/myorg/terraform-modules-repo.git//storage-account?ref=v1.0.0"
}

inputs = jsondecode(file("${get_terragrunt_dir()}/variables.tfvars.json"))

The modules repository evolves on its own schedule.
Terragrunt configs in live/ reference specific module versions via Git source.

This pattern prevents accidental cross‑pollination of concerns, making upgrades, reviews, and rollbacks a breeze. ✨

Why Terraform JSON `tfvars` Rocks 📑✨

Sure, HCL is human‑friendly, but JSON tfvars shine when you need to build tools:

🤖 Machine‑readable: Every language can parse JSON out of the box—no extra libraries needed!
✔️ Schema validation: Hook up a JSON Schema to catch mistakes before you even hit “apply.”
💡 Dynamic generation: Your self‑service portal or CLI can spin out a JSON file in milliseconds.

Example variables.tfvars.json:

{
  "account_name": "myappstorage-${env}",
  "resource_group_name": "rg-${env}",
  "location": "eastus",
  "tags": {
    "environment": "dev",
    "owner": "platform-team"
  }
}

And Terragrunt gobbles it up with:

# live/dev/terragrunt.hcl
terraform {
  source = "git::ssh://git@github.com/myorg/terraform-modules-repo.git//storage-account?ref=v1.0.0"
}

inputs = jsondecode(file("${get_terragrunt_dir()}/variables.tfvars.json"))

See how clean that is? 🥳

Building Your Own Abstraction Tools 🧰

Looking to give your team a self‑service experience? Here are two solid approaches:

1. CLI with Go & Cobra ⚙️

Fast scaffolding: Cobra generates a project structure with commands, subcommands, and flag parsing out of the box.
Type safety & performance: Go’s static typing ensures reliable flag handling, and compiled binaries run blazingly fast.
Plugin architecture: Easily hook in custom modules—for example, a command like infra gen that generates Terraform JSON tfvars files for your Terragrunt inputs.
Example snippet:

  package main

  import (
    "encoding/json"
    "io/ioutil"
    "github.com/spf13/cobra"
  )

  var (
    accountName       string
    resourceGroupName string
    location          string
    env               string
    owner             string
    varsFile          string
  )

  var rootCmd = &cobra.Command{
    Use:   "infra",
    Short: "Self-service infra CLI",
  }

  var genCmd = &cobra.Command{
    Use:   "gen",
    Short: "Generate Terraform tfvars JSON file",
    RunE: func(cmd *cobra.Command, args []string) error {
      config := map[string]interface{}{
        "account_name":        accountName,
        "resource_group_name": resourceGroupName,
        "location":            location,
        "tags": map[string]string{
          "environment": env,
          "owner":       owner,
        },
      }
      data, err := json.MarshalIndent(config, "", "  ")
      if err != nil {
        return err
      }
      return ioutil.WriteFile(varsFile, data, 0644)
    },
  }

  func init() {
    genCmd.Flags().StringVar(&accountName, "account-name", "", "Azure storage account name")
    genCmd.Flags().StringVar(&resourceGroupName, "resource-group", "", "Resource group name")
    genCmd.Flags().StringVar(&location, "location", "eastus", "Azure location")
    genCmd.Flags().StringVar(&env, "env", "dev", "Deployment environment")
    genCmd.Flags().StringVar(&owner, "owner", "platform-team", "Resource owner")
    genCmd.Flags().StringVar(&varsFile, "out", "variables.tfvars.json", "Output JSON tfvars file")
    rootCmd.AddCommand(genCmd)
  }

2. Self‑service portal with Backstage 🎭

Unified developer portal: Backstage by Spotify lets you surface docs, tooling, and pipelines in one UI.
Infrastructure as Code plugin: Create a custom Backstage plugin to list available Terraform/OpenTofu modules and trigger Terragrunt runs with JSON inputs.
Policy & approval workflows: Integrate with existing Identity/Access tools and GitOps pipelines for reviews before provisioning.
Rich catalog: Use Backstage’s software catalog to track module versions, show metadata (owner, tags, docs), and link to Git repos.

Combine these approaches with strict Terraform/Terragrunt separation and JSON tfvars, and you’ll empower your team with both CLI efficiency and a polished web portal.

Wrapping Up 🎁

Separating Terraform (or OpenTofu) modules from Terragrunt configs isn’t just a “nice to have”—it’s essential for maintainable, scalable infrastructure. Pair that with JSON tfvars, and you’re set to build amazing tooling that your whole team will love. Happy provisioning! 🌟

🔐 Secure Secret Management with SOPS in Helm 🚀

Hamdi (KHELIL) LION — Thu, 27 Feb 2025 08:15:10 +0000

When managing applications deployed on Kubernetes, keeping secrets safe while still making them accessible to Helm charts is a challenge. Storing secrets in plaintext is a security risk 🚨 — and that’s where SOPS (Secrets OPerationS) and the Helm Secrets plugin come in!

In this guide, we’ll cover:

✅ How to use SOPS with age and GPG
✅ How to configure SOPS with sops.yaml for better management
✅ How to use Helm Secrets Plugin to manage encrypted secrets directly in your Helm charts
✅ A GitHub Actions workflow to securely deploy Helm charts using encrypted secrets

📌 Why Use SOPS with Helm?

SOPS is an open-source tool from Mozilla that lets you encrypt and decrypt secrets with ease. When combined with the Helm Secrets plugin, you can safely store your sensitive data in Git repositories and automatically decrypt them during Helm deployments. Here’s why it’s awesome:

✅ Keeps secrets encrypted in your repos
✅ Works with YAML, JSON, and ENV files
✅ Integrates seamlessly with Helm via the Helm Secrets plugin
✅ Fits perfectly into CI/CD pipelines like GitHub Actions for secure deployments

🔑 Using SOPS with `age`

Age is a modern, simple, and secure encryption tool. If you’re new to encryption, age is a great alternative to GPG.

✨ Step 1: Install `age` and `sops`

Install age and sops:

sudo apt install age    # Ubuntu/Debian

✨ Step 2: Generate an `age` Key

Run:

age-keygen -o ~/.config/sops/age/keys.txt

This will generate a key similar to:

# public key: age1xxxxxxx
AGE-SECRET-KEY-1XXXXXXYYYYYYYYZZZZZZ

Copy the public key (age1xxxxxxx)—this will be used for encryption.

✨ Step 3: Encrypt a YAML File with SOPS

Create a file called secrets.yaml:

db_user: "admin"
db_password: "supersecret"

Now, encrypt it using SOPS:

sops --encrypt --age age1xxxxxxx -i secrets.yaml

When you open secrets.yaml, you’ll see it’s fully encrypted! 🛡️

To decrypt:

sops --decrypt secrets.yaml

🔧 Configuring `sops.yaml` for Better Management

Instead of specifying the encryption method manually every time, SOPS supports a configuration file (.sops.yaml). This makes it easier to manage secrets across your team.

Create .sops.yaml in your repository:

creation_rules:
  - path_regex: secrets/.*\.yaml$
    age:
      - age1xxxxxxx  # Replace with your public key
  - path_regex: secrets/.*\.json$
    pgp:
      - ABC12345  # Replace with your GPG key ID

Now, when encrypting secrets inside the secrets/ folder, SOPS will automatically use the right encryption method! 🎉

Encrypt a new secret:

sops --encrypt -i secrets/app.yaml

🛠️ Using Helm with the Helm Secrets Plugin

The Helm Secrets plugin allows you to work with encrypted secrets directly in your Helm charts—no need to expose sensitive data!

✨ Step 1: Install the Helm Secrets Plugin

Install the plugin with:

helm plugin install https://github.com/jkroepke/helm-secrets

This plugin leverages SOPS to decrypt your secret files during Helm chart deployments.

✨ Step 2: Encrypt Your Secrets File

Create a file named secrets.yaml (if you haven’t already):

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  username: YWRtaW4=          # base64 encoded
  password: c3VwZXJzZWNyZXQ=   # base64 encoded

Encrypt it using SOPS:

sops --encrypt -i secrets.yaml

✨ Step 3: Deploy with Helm Using Encrypted Secrets

Deploy your Helm chart using the encrypted secrets file:

helm secrets upgrade --install my-release ./my-chart

The Helm Secrets plugin will automatically decrypt secrets.yaml during the deployment process. 🚀

🤖 Using SOPS and Helm in GitHub Actions

Integrate your secure secrets management into your CI/CD pipeline with GitHub Actions. Here’s an example workflow that deploys your Helm chart with encrypted secrets:

✨ Step 1: Store the `age` Private Key in GitHub Secrets

In your GitHub repository, navigate to Settings → Secrets and variables → Actions, and add:

SOPS_AGE_KEY: The private key from ~/.config/sops/age/keys.txt

✨ Step 2: Create the GitHub Actions Workflow

Create .github/workflows/deploy.yml:

name: Deploy with Helm & SOPS

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Install dependencies
        run: |
          sudo apt-get update
          sudo apt-get install -y sops age
          curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
          helm plugin install https://github.com/jkroepke/helm-secrets

      - name: Set up SOPS
        run: |
          mkdir -p ~/.config/sops/age/
          echo "${{ secrets.SOPS_AGE_KEY }}" > ~/.config/sops/age/keys.txt
          chmod 600 ~/.config/sops/age/keys.txt

      - name: Deploy with Helm
        run: |
          helm secrets upgrade --install my-release ./my-chart

🔥 What Happens in This Workflow?

Checks out the code ✅
Installs SOPS, age, Helm, and the Helm Secrets plugin ✅
Loads the age private key from GitHub Secrets ✅
Deploys the Helm chart with decrypted secrets on the fly ✅

Security Tip:

Make sure that any decrypted files are never committed to your repository! Always keep them out of version control. 🔒

🎯 Wrapping Up

SOPS and the Helm Secrets plugin offer a powerful and secure way to manage secrets in your Kubernetes deployments. With age encryption, a handy .sops.yaml configuration, and seamless integration via Helm, managing secrets has never been easier! 💪

By integrating these tools into your workflow, you get:

✅ Encrypted secrets safely stored in Git repositories
✅ Automatic decryption during Helm deployments
✅ Secure usage of secrets in CI/CD pipelines

Want to take it a step further? Try exploring AWS KMS, GCP KMS, or Azure Key Vault for even tighter security! 🔐🚀

Have questions or suggestions? Drop them in the comments! 💬

Happy clustering and stay safe! 🔐😊

🔐 Secure Secret Management with SOPS in Terraform & Terragrunt

Hamdi (KHELIL) LION — Wed, 26 Feb 2025 14:44:59 +0000

When managing infrastructure as code (IaC), keeping secrets safe while still making them accessible to Terraform/Terragrunt is a challenge. Storing secrets in plaintext is a security risk 🚨—and that’s where SOPS (Secrets OPerationS) comes in!

In this guide, we’ll cover:

✅ How to use SOPS with age and GPG
✅ How to configure SOPS with sops.yaml for better management
✅ How to use Terragrunt’s built-in SOPS decryption (without run_cmd)
✅ A GitHub Actions workflow to securely use secrets in CI/CD

📌 Why Use SOPS?

SOPS is an open-source tool from Mozilla that lets you encrypt and decrypt secrets easily. It supports multiple encryption methods, including GPG, AWS KMS, Azure Key Vault, Google Cloud KMS, and age.

Here’s why it’s awesome:

✅ Keeps secrets encrypted in Git repositories
✅ Works with YAML, JSON, ENV files
✅ Has built-in support in Terragrunt (no extra scripting needed!)
✅ Integrates with GitHub Actions, Kubernetes, and CI/CD pipelines

Now, let’s see how to use SOPS with age and GPG, then configure it properly for Terragrunt and GitHub Actions.

🔑 Using SOPS with `age`

Age is a modern, simple, and secure encryption tool. If you’re new to encryption, age is a great alternative to GPG.

✨ Step 1: Install `age` and `sops`

First, install age and sops:

sudo apt install age    # Ubuntu/Debian

✨ Step 2: Generate an `age` Key

Run:

age-keygen -o ~/.config/sops/age/keys.txt

This will generate a key similar to:

# public key: age1xxxxxxx
AGE-SECRET-KEY-1XXXXXXYYYYYYYYZZZZZZ

Copy the public key (age1xxxxxxx)—this will be used for encryption.

✨ Step 3: Encrypt a YAML File with SOPS

Create a file called secrets.yaml:

db_user: "admin"
db_password: "supersecret"

Now, encrypt it using SOPS:

sops --encrypt --age age1xxxxxxx -i secrets.yaml

If you open secrets.yaml, you'll see it's fully encrypted! 🛡️

To decrypt:

sops --decrypt secrets.yaml

🔧 Configuring `sops.yaml` for Better Management

Instead of specifying the encryption method manually every time, SOPS supports a configuration file (.sops.yaml). This makes it easier to manage secrets across teams.

Create .sops.yaml in your repository:

creation_rules:
  - path_regex: secrets/.*\.yaml$
    age:
      - age1xxxxxxx  # Replace with your public key
  - path_regex: secrets/.*\.json$
    pgp:
      - ABC12345  # Replace with your GPG key ID

Now, when encrypting secrets inside the secrets/ folder, SOPS will automatically use the right encryption method! 🎉

Encrypt a new secret:

sops --encrypt -i secrets/app.yaml

⚙️ Using SOPS with Terragrunt’s Built-in Decryption

Terragrunt has native support for SOPS, meaning you don’t need to use run_cmd(). Instead, you can directly reference encrypted files in your terragrunt.hcl.

✨ Step 1: Encrypt the Secrets

Create secrets.yaml:

aws_access_key: "AKIAxxxxxxxxxxxx"
aws_secret_key: "abcdefghijklmno1234567890"

Encrypt it:

sops --encrypt -i secrets.yaml

✨ Step 2: Use Terragrunt's Built-in SOPS Decryption

Modify terragrunt.hcl:

locals {
  secrets = yamldecode(sops_decrypt_file("secrets.yaml"))
}

inputs = {
  aws_access_key = local.secrets.aws_access_key
  aws_secret_key = local.secrets.aws_secret_key
}

Now, when you run:

terragrunt apply

Terragrunt automatically decrypts secrets.yaml without requiring external scripts! 🚀

🤖 Using SOPS in GitHub Actions

When using GitHub Actions, we need to decrypt secrets safely without exposing them.

✨ Step 1: Store the `age` Private Key in GitHub Secrets

Go to GitHub → Your Repo → Settings → Secrets and variables → Actions, and add:

SOPS_AGE_KEY: The private key from ~/.config/sops/age/keys.txt

✨ Step 2: Use SOPS in a GitHub Workflow

Create .github/workflows/deploy.yml:

name: Deploy with Terraform & SOPS

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Install dependencies
        run: |
          sudo apt-get update
          sudo apt-get install -y sops age

      - name: Set up SOPS
        run: |
          mkdir -p ~/.config/sops/age/
          echo "${{ secrets.SOPS_AGE_KEY }}" > ~/.config/sops/age/keys.txt
          chmod 600 ~/.config/sops/age/keys.txt

      - name: Decrypt Secrets
        run: sops --decrypt secrets.yaml > secrets.decrypted.yaml

      - name: Deploy with Terraform
        run: |
          terragrunt run-all apply --auto-approve

🔥 What Happens in This Workflow?

Checks out the code ✅
Installs SOPS & age ✅
Loads the age private key from GitHub Secrets ✅
Decrypts secrets into a temporary file ✅
Runs Terraform/Terragrunt with the decrypted secrets ✅

Security Tip:
Make sure secrets.decrypted.yaml is ignored in .gitignore and is never committed to Git!

🎯 Wrapping Up

SOPS is a powerful and secure way to manage secrets for Terraform, Terragrunt, and GitHub Actions. With age encryption, .sops.yaml for better configuration, and Terragrunt's built-in decryption, managing secrets has never been easier! 💪

By integrating SOPS into your workflow, you get:

✅ Encrypted secrets in Git repositories
✅ Easy decryption in Terraform/Terragrunt
✅ Safe usage of secrets in CI/CD

Want to take it a step further? Try using AWS KMS, GCP KMS, or Azure Key Vault instead of age/GPG for even tighter security! 🔐🚀

Have questions or suggestions? Drop them in the comments! 💬

Happy clustering and stay safe ! 🔐

Helm Chart Essentials & Writing Effective Charts 🚀

Hamdi (KHELIL) LION — Thu, 23 Jan 2025 10:50:53 +0000

Helm charts are a powerful way to define, install, and upgrade Kubernetes applications. By packaging all the Kubernetes manifests and parameters in a neat, reproducible format, Helm simplifies the deployment process for engineers and DevOps teams. In this article, we’ll explore some best practices for writing effective Helm charts, introduce the Helm Schema plugin for validation, show how to include tests to ensure reliability, discuss helm-docs for automated documentation generation, and share an additional resource for testing and linting. Let’s get started! 🎉

1. Getting Started with Helm Charts 🏁

What Is Helm?

Helm is a package manager for Kubernetes, similar to how apt/yum/brew work for operating systems. It helps you:

Package your Kubernetes resources into self-contained units called charts.
Install and upgrade your applications with version tracking.
Simplify complex deployments by parameterizing configurations in YAML.

Anatomy of a Helm Chart

A typical Helm chart includes the following structure:

my-awesome-chart/
├── Chart.yaml
├── values.yaml
├── templates/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── tests/
│       └── test-connection.yaml
└── charts/

Chart.yaml: Contains metadata about your chart, like name and version.
values.yaml: Holds the default values your chart will use unless overridden by the user.
templates/: Contains Kubernetes manifests that get rendered using the values specified in values.yaml.
charts/: A directory to hold other charts your chart depends on (sub-charts).
tests/: A folder inside templates/ where you can define chart tests.

2. Best Practices for Writing Effective Helm Charts 🏆

Follow a Consistent Structure

Keeping a clean chart directory structure helps others (and your future self) navigate easily.
Version Management
- Update version in Chart.yaml according to SemVer principles.
- Keep your appVersion (in Chart.yaml) aligned with the application’s version it deploys.
Leverage values.yaml
- Store user-configurable defaults in values.yaml.
- Use clear naming to describe each parameter.
Use Helpers and Templates
- Create _helpers.tpl for small, reusable snippets (like labels or names).
- Standardize your resource naming (e.g., {{ include "my-awesome-chart.fullname" . }}).
Document Everything
- Add a README.md with usage instructions, examples, and default values.
- Provide context for each parameter in values.yaml.

Following these tips ensures your charts stay consistent, maintainable, and user-friendly. 🙌

3. Introducing the Helm Schema Plugin 📐

When you have multiple parameters and complex configurations, validating your values.yaml files becomes crucial. The Helm Schema plugin uses JSON Schema to validate your Helm chart values. This helps ensure that the values provided by end users conform to the expected data types, structures, and constraints.

Installation & Usage

Install the plugin:

   helm plugin install https://github.com/dadav/helm-schema

Add a values.schema.json: In your chart’s root, create a file named values.schema.json. This file defines the schema of your values. For example:

   {
     "$schema": "http://json-schema.org/draft-07/schema#",
     "type": "object",
     "properties": {
       "replicaCount": {
         "type": "integer",
         "minimum": 1
       },
       "image": {
         "type": "object",
         "properties": {
           "repository": {
             "type": "string"
           },
           "tag": {
             "type": "string"
           }
         },
         "required": ["repository"]
       }
     },
     "required": ["replicaCount", "image"]
   }

Validate your values:

   helm schema my-awesome-chart/values.schema.json -f values.yaml

This will check if values.yaml meets the constraints defined in values.schema.json. If something is off (e.g., missing required keys, using a string instead of an integer), it will throw a validation error.

Benefits of Using Helm Schema

Early Catch 🐞: Validate charts before deploying to production, catching misconfiguration early.
Clear Documentation 💡: The JSON schema itself acts as documentation, helping end users understand required and optional fields.
Better Collaboration 🤝: With the schema enforced, teams can rest assured that values used in different environments adhere to the same rules.

4. Including Tests in Your Helm Charts ✅

Testing is critical to verify that your Kubernetes resources deploy and function as expected. Helm offers a straightforward way to define and run tests within your chart. These tests are essentially Kubernetes jobs/pods that run checks to ensure your application is responding as intended.

How Helm Tests Work

Create a Test File: In your chart’s templates/tests/ directory, create a file (e.g., test-connection.yaml).
Add Test Annotations: Include special annotations recognized by Helm:

   apiVersion: v1
   kind: Pod
   metadata:
     name: "{{ include "my-awesome-chart.fullname" . }}-test-connection"
     labels:
       {{- include "my-awesome-chart.labels" . | nindent 4 }}
     annotations:
       "helm.sh/hook": test
       "helm.sh/hook-delete-policy": before-hook-creation
   spec:
     containers:
       - name: test-connection
         image: busybox
         command: ['sh', '-c', 'echo "Running connection test..."; sleep 1; exit 0']
     restartPolicy: Never

helm.sh/hook: test tells Helm that this Pod is used for testing.
helm.sh/hook-delete-policy: before-hook-creation cleans up old test pods before creating new ones.

Run the Tests: After installing or upgrading your chart, simply run:

   helm test my-awesome-chart-release

Helm will execute the test pods, and if they all pass (i.e., containers exit with code 0), your chart tests succeed! 🎉

Writing Meaningful Tests

Functionality Tests: Verify that your application can handle basic requests (e.g., curl the service endpoint).
Readiness Tests: Check your application’s endpoints for readiness or health endpoints.
Configuration Tests: If your chart has multiple configuration options, test them in ephemeral environments to ensure no regressions.

5. Auto-Generating Documentation with helm-docs 📝

Maintaining up-to-date documentation can be tricky as your chart evolves. That’s where helm-docs comes in. It automatically generates documentation for your Helm chart by reading metadata from your chart files (like Chart.yaml and values.yaml).

Installing helm-docs

brew install norwoodj/tap/helm-docs

(See the helm-docs repo for alternative installation methods.)

Using helm-docs

Add a README.md.gotmpl Create a template for your documentation—usually named something like README.md.gotmpl. You can define sections for chart metadata, usage, and parameters. For example:

   # {{ .Name }}

   {{ .Description }}

   ## Parameters
   | Name | Description | Value |
   ||-|-|
   {{- range .Values }}
   | {{ .Path }} | {{ .Description }} | {{ .Default | quote }} |
   {{- end }}

Run helm-docs Navigate to your chart directory and run:

   helm-docs

This command will parse your values.yaml, Chart.yaml, and any comments or metadata to generate a README.md file based on your template.

Commit & Share Once README.md is generated, commit it to your repository. Now, anyone viewing your chart can see the current parameters, default values, and usage instructions at a glance.

Benefits of helm-docs

Automatic Updates ⏰: Whenever you update values.yaml or Chart.yaml, re-running helm-docs keeps your README in sync.
Consistent Format 📄: Enforces a uniform structure across multiple charts in your organization.
Time Savings ⏲️: Spend less time manually editing documentation and more time improving your charts!

6. Wrapping It Up 🎁

By following these guidelines:

Structure your charts consistently.
Leverage the Helm Schema plugin to validate configurations using JSON Schema.
Include tests to confirm your chart works as intended.
Auto-generate documentation with helm-docs to keep docs current and accurate.

…you’ll have robust, maintainable, and user-friendly Helm charts. Whether you’re deploying a tiny microservice or a huge enterprise application, these best practices will help you ship fast and confidently. ⚡

Additional Resources

Helm Official Docs
Helm Schema Plugin
helm-docs
Further Reading: Ensuring Effective Helm Charts with Linting, Testing, and Diff Checks

If you have any questions or tips, feel free to share them in the comments below. Together, we can make Helm charts smoother to write, more reliable to deploy, and easier to maintain! 🥳

Happy Helming and clustering ! 🐳

Unlocking Secrets with External Secrets Operator 🔐✨

Hamdi (KHELIL) LION — Fri, 03 Jan 2025 20:00:01 +0000

In modern cloud-native applications, securely managing sensitive data like API keys, database credentials, and certificates is a top priority. Two powerful tools stand out when integrating secrets into Kubernetes: External Secrets Operator and SecretStoreProviders plugin (for Azure and AWS). Let’s dive into how to use them, their differences, and when to pick one over the other. 🚀

What Is External Secrets Operator? 🤔

The External Secrets Operator (ESO) simplifies secret management in Kubernetes by integrating external secret stores directly into your cluster. Instead of manually creating Kubernetes Secrets, ESO syncs secrets from providers like AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault.

Why Use External Secrets Operator? 🌟

Centralized Secret Management: Manage secrets in your provider of choice while syncing them to Kubernetes.
Automation: No need to manually update Kubernetes Secrets when values change.
Security: Leverages providers' robust access control and encryption mechanisms.

How to Use External Secrets Operator 🛠️

Here’s a quick guide to setting up ESO:

Step 1: Install External Secrets Operator

Install ESO via Helm:

helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm install external-secrets external-secrets/external-secrets

Step 2: Configure a SecretStore

Create a SecretStore resource to connect to your external provider. For example, if you're using AWS Secrets Manager:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secret-store
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-west-2

For Azure Key Vault:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: azure-secret-store
spec:
  provider:
    azurekv:
      tenantId: <your-tenant-id>
      vaultUrl: https://<your-vault-name>.vault.azure.net/

Step 3: Define an ExternalSecret

Bind your external secret to a Kubernetes Secret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-app-secret
spec:
  secretStoreRef:
    name: aws-secret-store
  target:
    name: my-k8s-secret
  data:
    - secretKey: apiKey
      remoteRef:
        key: /my-app/api-key

This example pulls the /my-app/api-key from AWS Secrets Manager and creates a Kubernetes Secret named my-k8s-secret.

For templating values:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-template-secret
spec:
  secretStoreRef:
    name: azure-secret-store
  target:
    name: templated-secret
  data:
    - secretKey: connectionString
      remoteRef:
        key: /database/connection-string
  dataFrom:
    - extract: key-values

How Does It Compare to SecretStoreProviders Plugin? ⚖️

Both ESO and the SecretStoreProviders plugin enable secure secret management in Kubernetes. Here’s how they stack up:

1. Scope and Features 🌍

External Secrets Operator:
- Supports multiple providers (AWS, Azure, GCP, HashiCorp Vault, etc.).
- Extensive customization and features like templating.
SecretStoreProviders Plugin:
- Focused primarily on Azure Key Vault and AWS Secrets Manager.
- Limited to CSI driver-based integration.

2. Installation and Complexity 🤹

ESO: Requires deploying the operator and defining SecretStore and ExternalSecret resources. It’s slightly more complex but offers more flexibility.
SecretStoreProviders: Simpler setup with a CSI driver. However, its capabilities are narrower.

3. Performance and Scalability 🚀

ESO: Handles more complex scenarios but may introduce slight overhead due to the operator’s reconciliation process.
SecretStoreProviders: Lightweight but less feature-rich, making it better for straightforward use cases.

4. Provider Support 🌐

ESO: Supports a broad range of providers beyond AWS and Azure.
SecretStoreProviders: Limited to Azure Key Vault and AWS Secrets Manager.

5. Example Use Cases

ESO:
- Sync multiple secrets from AWS Secrets Manager and Azure Key Vault into a single Kubernetes namespace.
- Templating and merging secrets from different sources into one Secret.
- Using annotations to dynamically manage secret lifecycles.
SecretStoreProviders:
- Directly mount Azure Key Vault secrets as files in a pod.
- Lightweight secret access for apps needing minimal Kubernetes API interaction.

How to Use SecretStoreProviders Plugin for Azure and AWS 🔧

Azure Key Vault Example

Install the Azure Key Vault CSI driver:

kubectl apply -f https://raw.githubusercontent.com/Azure/secrets-store-csi-driver-provider-azure/main/deployment/secretproviderclass.yaml

Create a SecretProviderClass to define the secrets you want to access:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: azure-keyvault
spec:
provider: azure
parameters:
    usePodIdentity: "false"
    clientID: "<your-client-id>"
    tenantID: "<your-tenant-id>"
    keyvaultName: "<your-keyvault-name>"
    objects: |
    array:
        - |
        objectName: secret1
        objectType: secret
        - |
        objectName: certificate1
        objectType: cert

Mount the secrets in a pod:

apiVersion: v1
kind: Pod
metadata:
name: nginx-secrets-store
spec:
containers:
    - name: nginx
    image: nginx
    volumeMounts:
        - name: secrets-store-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
volumes:
    - name: secrets-store-inline
    csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
        secretProviderClass: "azure-keyvault"

AWS Secrets Manager Example

Install the AWS Secrets Manager CSI driver:

kubectl apply -k "github.com/aws/secrets-store-csi-driver-provider-aws/deployment?ref=main"

Create a SecretProviderClass to define the secrets:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: aws-secrets
spec:
provider: aws
parameters:
    objects: |
    - objectName: "mySecret"
        objectType: "secretsmanager"
        region: "us-west-2"

Mount the secrets in a pod:

apiVersion: v1
kind: Pod
metadata:
name: nginx-secrets-store
spec:
containers:
    - name: nginx
    image: nginx
    volumeMounts:
        - name: secrets-store-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
volumes:
    - name: secrets-store-inline
    csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
        secretProviderClass: "aws-secrets"

Which One Should You Choose? 🤷‍♀️

Use External Secrets Operator if:

You need to work with multiple providers.
Your use case involves advanced templating or secret transformation.
Flexibility and extensibility are priorities.

Use SecretStoreProviders Plugin if:

You primarily work with Azure or AWS secrets.
Simplicity and minimal resource usage are key.
You prefer to use the Kubernetes CSI driver.

Conclusion 🎉

Both External Secrets Operator and SecretStoreProviders plugin are excellent tools for managing secrets in Kubernetes. Your choice depends on your use case, complexity, and cloud provider preferences. With either option, you’ll be well-equipped to handle secrets securely in your Kubernetes clusters. 🔒

Happy Clustering! 🚀

Forem: Hamdi (KHELIL) LION

🧠 Demystifying Metrics in Kubernetes

🤔 Why Metrics Matter for Autoscaling

⚙️ 1 Resource Metrics = Pod Health Signals

🧩 What they represent

📄 Example HPA based on CPU

✅ Pros

❌ Limits

📊 2 Custom Metrics = Your App Talking to Kubernetes

🔁 Typical data flow

🌐 Example 1 HTTP requests per second

⏱ Example 2 Request duration

🧵 Example 3 Active background jobs

✅ Pros

❌ Limits

🌍 3 External Metrics = Work Waiting Outside the Cluster

📬 Example 1 SQS queue length

🐘 Example 2 Kafka consumer lag

📦 Example 3 Redis job queue size

⏰ Example 4 Time based scaling

🧮 How HPA Uses These Metrics

⚡ KEDA vs Prometheus Adapter

🧩 With Prometheus Adapter

⚡ With KEDA

🧭 When to Use What

🎯 Final Takeaway

Building a Kubernetes homelab the hard but right way 🧱☸️

The mindset first 🧠

The big picture 🗺️

Step 1 Proxmox as a real compute layer ⚙️

What I am doing here

Step 2 The cluster is just data 📐

Step 3 Reusable Terraform modules 🧩

Step 4 Kubespray does the heavy lifting ☸️

Step 5 Bootstrap only the essentials 🚦

Step 6 Day 2 is simple and explicit with Helmfile 📦

When GitOps really makes sense 🚀

Why I like this approach so much ❤️

What is next 🔮

Building a Kubernetes HomeLab - The Hard Way - DNS first, or everything else lies to you 🔥🌐

🏗️ Setting Up a Proxmox Cluster for My Homelab Kubernetes Environment

🧰 The Hardware That Powers My Homelab Kubernetes Cluster

🖥️ The Compute Layer

Three Lenovo ThinkCentre M920q Tiny

🌐 The Networking Layer

Gigabit Switch Today

🍓 The Brain of the LAN

Raspberry Pi as DNS Server

📦 The Data Layer

Synology NAS for Storage and Backups

🔌 Hardware Architecture Diagram

🎯 Why this hardware matters

✅ Up Next

🏡 Why I Built My Own Homelab to Run Kubernetes

🧩 GitHub Actions Composite vs Reusable Workflows

How to standardize and supercharge your CI/CD pipelines across projects

🔄 Composite Actions vs Reusable Workflows

🧱 Composite Actions

🔁 Reusable Workflows

🐳 Example: Docker Build and Scan with Trivy

.github/actions/build-and-scan/action.yml

Usage in Workflow

🌍 Example: Terraform/Terragrunt Validation with Checkov

.github/workflows/infra-check.yml

Usage in Project Workflow

⚙️ Best Practices

💡 Conclusion

🚀🌐 Elevating Infrastructure: From Terraform/Terragrunt Foundations to Platform Engineering 😊

Why Separation Is a Must, Not an Option 🙅‍♂️🙅‍♀️

Module Versioning with Terraform 🎯

Decoupling Terraform (or OpenTofu) & Terragrunt 🛠️

Why Terraform JSON tfvars Rocks 📑✨

Building Your Own Abstraction Tools 🧰

Wrapping Up 🎁

🔐 Secure Secret Management with SOPS in Helm 🚀

📌 Why Use SOPS with Helm?

🔑 Using SOPS with age

✨ Step 1: Install age and sops

✨ Step 2: Generate an age Key

✨ Step 3: Encrypt a YAML File with SOPS

`.github/actions/build-and-scan/action.yml`

`.github/workflows/infra-check.yml`

Why Terraform JSON `tfvars` Rocks 📑✨

🔑 Using SOPS with `age`

✨ Step 1: Install `age` and `sops`

✨ Step 2: Generate an `age` Key

🔧 Configuring `sops.yaml` for Better Management

✨ Step 1: Store the `age` Private Key in GitHub Secrets

🔑 Using SOPS with `age`

✨ Step 1: Install `age` and `sops`

✨ Step 2: Generate an `age` Key

🔧 Configuring `sops.yaml` for Better Management

✨ Step 1: Store the `age` Private Key in GitHub Secrets