Forem: Andre Nogueira

Home Lab: Chapter 8 — Kubernetes Storage with Rook-Ceph

Andre Nogueira — Wed, 05 Nov 2025 19:26:08 +0000

Howdy!

We've come a long way! We've set up our Kubernetes cluster, configured GitOps with ArgoCD, managed secrets securely, exposed applications through Ingress, and set up DNS with SSL certificates. But there's one critical piece we haven't addressed yet: storage.

In this chapter, we'll tackle one of the most challenging aspects of running Kubernetes in a homelab environment - persistent storage. Specifically, we'll explore how I implemented a distributed storage solution using Rook-Ceph to provide reliable, scalable block storage across my cluster.

The Storage Challenge

When you run applications on Kubernetes, especially stateful ones like databases, message queues, or monitoring systems, you need persistent storage that survives pod restarts and node failures. Without it, losing a pod means losing all your data.

In a cloud environment, this is straightforward - you just request a volume from your cloud provider. But in a homelab, you need to build this yourself.

Why Not Just Use Local Storage?

You might think, "Can't I just mount a local directory on each node?" Technically yes, but there are serious drawbacks:

No redundancy - If a node fails, your data is gone
Poor availability - Pods can't migrate between nodes
Limited capacity - Bound by individual node storage
Manual management - You have to handle backups yourself

For a homelab that aims to mimic production environments, this isn't acceptable.

Introducing Rook-Ceph

Rook is a cloud-native storage orchestrator that automates the deployment and management of storage systems in Kubernetes. Ceph is a distributed storage platform that provides block storage, object storage, and file system storage.

Together, Rook-Ceph gives you:

Distributed storage - Data replicated across multiple nodes
Self-healing - Automatic recovery from node failures
High availability - Pods can migrate freely
Scalability - Add nodes to expand storage
Production-ready - Used by enterprises worldwide

The Perfect Fit for Homelabs

Rook-Ceph is particularly well-suited for homelab environments because:

It uses node local disks, so no external storage appliances needed
It's open-source and free
It's battle-tested in production
It manages itself using Kubernetes native resources
It provides excellent observability and dashboards

Implementation

Now that we understand why Rook-Ceph is a great fit, let's dive into how I implemented it in my homelab. I'll walk through the deployment strategy, cluster configuration, storage classes, and some key design decisions that make this setup reliable and scalable.

Deployment Strategy

In my setup, I deployed Rook-Ceph using Kustomize through ArgoCD (as we configured in Chapter 4). This ensures:

Infrastructure-as-code approach
Automated deployments
Easy reproducibility
Version control of all configurations

Cluster Configuration

Here's the core Rook-Ceph cluster setup:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v18.2.2
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3              # 3 monitors for quorum
    allowMultiplePerNode: false
  dashboard:
    enabled: true        # Web dashboard for monitoring
  storage:
    useAllNodes: true    # Use all nodes in cluster
    useAllDevices: true  # Use all available disks

Storage Classes

I configured a StorageClass to define how storage should be provisioned:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-ceph-block
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  clusterID: rook-ceph
  pool: replicapool
  imageFormat: "2"
  imageFeatures: layering,fast-diff,object-map,deep-flatten,exclusive-lock
  csi.storage.k8s.io/fstype: xfs
reclaimPolicy: Retain      # Keep volumes after deletion
allowVolumeExpansion: true # Scale volumes on demand

Key Design Decisions

Monitor Count (3): Ceph requires a quorum. With 3 monitors, the cluster tolerates 1 failure. Given my 3-node setup, one monitor per node is ideal.

Use All Nodes: This ensures distributed storage across the entire cluster, maximizing redundancy.

Use All Devices: Any available disk on any node becomes part of the Ceph cluster.

Retain Reclaim Policy: When a PVC is deleted, the underlying volume is retained (not deleted), providing data safety.

XFS Filesystem: More performant and reliable than ext4 for this use case.

Object Storage

Beyond block storage, Rook-Ceph also provides Object Storage (S3-compatible) through its Radosgw component:

apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: ceph-objectstore
  namespace: rook-ceph
spec:
  metadataPool:
    failureDomain: host
    replicated:
      size: 3
  dataPool:
    failureDomain: host
    erasureCoded:
      dataChunks: 2
      codingChunks: 1
  gateway:
    port: 80
    instances: 2

This enables me to:

Back up applications to S3-compatible storage
Host private registries
Create self-hosted object storage alternatives to AWS S3

Monitoring and Operations

The Rook-Ceph dashboard provides visibility into:

Cluster health and status
Capacity and usage metrics
OSD (storage) performance
Pool configurations
Real-time alerts

Accessing it is straightforward through port-forwarding:

kubectl port-forward -n rook-ceph svc/rook-ceph-mgr-dashboard 7000:7000

Real-World Usage

With Rook-Ceph in place, provisioning storage for applications is trivial:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-database-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: rook-ceph-block
  resources:
    requests:
      storage: 10Gi

Applications request storage, Rook automatically provisions it across the cluster, and data is protected through replication. The complexity is hidden, the benefits are clear.

Conclusion

With Rook-Ceph in place, my homelab now has a production-grade distributed storage system. Applications no longer need to worry about node failures - storage is replicated, self-healing, and highly available.

The Complete Foundation

This chapter marks the completion of the foundational Kubernetes setup. Over these 8 chapters, we've built all the bare bones infrastructure needed to run applications reliably:

Hardware & Network (Ch. 1) - The physical foundation
Base Infrastructure (Ch. 2) - OS, networking, security
Kubernetes Cluster (Ch. 3) - Orchestration platform
GitOps (ArgoCD) (Ch. 4) - Automated deployments
Secrets Management (Ch. 5) - Secure configurations ✅ Ingress & Load Balancing (Ch. 6) - External access ✅ DNS & SSL (Ch. 7) - Domain names and encryption ✅ Distributed Storage (Ch. 8) - Persistent data

What This Enables

With this foundation in place, you can now:

Deploy stateful applications with confidence
Know they'll survive node failures
Scale storage by adding nodes
Update applications safely with zero downtime
Manage secrets securely
Access services via stable domain names with valid certificates
Automate everything through version control

This is genuinely production-grade infrastructure - the kind you'd see in enterprise environments, but tailored for a homelab.

What's Next?

From here, the real fun begins. In future quests, we'll explore:

Running actual applications (databases, message queues, cache layers)
Monitoring and observability (metrics, logs, alerts)
CI/CD pipelines (automated testing and deployments)
Backup strategies and disaster recovery
Advanced networking and service mesh concepts

But for now, we have a solid, resilient, production-ready platform. Every component we've built is battle-tested, scalable, and self-healing. That's something to be proud of. 🎉

Originally published at https://techquests.dev on November 5, 2025.

Home Lab: Chapter 7 — Kubernetes DNS and SSL

Andre Nogueira — Fri, 03 Oct 2025 18:25:38 +0000

Howdy,

Our environment is starting to take shape. We have a Kubernetes cluster up and running, an Ingress Controller managing external access to our services, and a way to handle secrets. The next step is making sure our services are accessible from the outside world. To do this, we need to configure DNS and SSL.

Getting a Domain

Before configuring DNS, you need a domain name to access your services. If you don't have one yet, you can register a domain with any registrar you prefer. Popular options include Namecheap, GoDaddy, or Google Domains - previously known as Google Domains. What's important is that you have full access to manage the DNS records for that domain.

Choose a domain that's easy to remember and type. Trust me - you'll thank yourself later when testing and sharing URLs.

While it's not strictly required, I recommend using a domain you own rather than an internal-only domain. Why? Because issuing valid SSL certificates for your own domain is straightforward, allowing you to access your services over HTTPS without headaches.

If you decide to stick with an internal domain, you'll need to use a self-signed certificate or one issued by a private certificate authority (CA). This works fine for internal use, but accessing the services externally can be tricky. Browsers won't trust your private CA by default, so you'll see warnings unless you install your root CA certificate on your system or browser. Using a domain you own avoids this hassle entirely.

DNS and SSL

DNS (Domain Name System) is what translates human-readable domain names into IP addresses, allowing us to access websites and services without memorizing numbers. We've touched on DNS before, but it's worth revisiting since it plays a crucial role in exposing our services. In a previous chapter, we set up a DNS server to resolve some of our internal services - mostly infrastructure-related. Now, we want to extend DNS to resolve the domain names for services that will be accessible both externally and internally.

Because we have two different scenarios, we'll need two DNS setups:

Internal-facing applications - accessible only within our network.
Public-facing applications - accessible from the internet.

To keep things simple, I'll use two separate DNS servers for these scenarios. One server will manage public records, and the other will manage internal records. This isn't strictly required - we could use a single DNS server for both - but separating them helps avoid conflicts and keeps things organized.

Some configuration details will vary depending on the DNS solution you choose. In this guide, I'll be using Bind9 for internal-facing applications and Cloudflare for public-facing applications. You can pick whichever DNS servers you prefer, as long as you can manage both internal and external records without conflicts.

Below is a high-level overview of the DNS and SSL setup for our Homelab:

Internal facing DNS records

For internal-facing applications, we'll be using Bind9, an open-source authoritative DNS server. This setup allows us to:

Host internal DNS records for services accessible only within our network (e.g., nginx.<INTERNAL_DOMAIN>).
Resolve public domains by forwarding requests to external resolvers such as 1.1.1.1 (Cloudflare) or 8.8.8.8 (Google).

By combining Bind9 with Unbound as a forwarder, Bind9 becomes our primary internal DNS server capable of resolving both internal and external domains.

Bind9 Setup

We can install Bind9 using the official Helm chart and manage it via GitOps with ArgoCD. Here's an example bind9.yaml:

# bind9.yaml
# ...
    repoURL: https://github.com/johanneskastl/helm-charts.git
    targetRevision: bind9-0.5.1
    path: charts/bind9
    helm:
      valuesObject:
        image:
          repository: internetsystemsconsortium/bind9
          tag: "9.21" # 9.19 is not available
        service:
          dns-udp:
            type: NodePort
            ports:
              dns-udp:
                nodePort: 30053
        chartMode: authoritative
        persistence:
          config:
            enabled: true
          bind9namedconf:
            enabled: true
            name: bind9-named-config
          bind9userconfiguration:
            enabled: true
            name: bind9-config
# ...

This configuration:

Uses the official internetsystemsconsortium/bind9 image.
Exposes DNS on port 30053 via a NodePort service - this allows external access to the DNS server.
Persists configuration files so that data is not lost when the pod restarts.
Sets up Bind9 in authoritative mode, meaning it will manage DNS records for our internal domain.

Bind9 Configuration

We define the zones and DNS records using a named configuration. A named configuration specifies the zones for which the Bind9 server is authoritative and the associated records.

# bind9-config.yaml
named.conf.local: |
    key "tsig-key" {
        algorithm hmac-sha512;
        secret "<SECRET>";
    };
    zone "<INTERNAL_DOMAIN>" in {
        type master;
        file "/named_config/<INTERNAL_DOMAIN>.zone";
        journal "/config/<INTERNAL_DOMAIN>.zone.jnl";
        notify no;
        allow-transfer {
            key "tsig-key";
        };
        update-policy {
            grant tsig-key zonesub ANY;
        };
    };
  <INTERNAL_DOMAIN>.zone: |
    $TTL 3600 ; 1 hour
    @   IN SOA  <INTERNAL_DOMAIN>. <EMAIL>. (
                  2025040601 ; serial
                  43200      ; refresh (12 hours)
                  3600       ; retry (1 hour)
                  604800     ; expire (1 week)
                  3600       ; minimum (1 hour)
                )
        IN NS     ns.<INTERNAL_DOMAIN>.
    ns  IN A      x.x.x.105

Explanation:

TSIG key: Stands for Transaction Signature - it is used to authenticate and secure dynamic updates to the zone without exposing the server publicly.
SOA record: Defines the authoritative server and key timing parameters for DNS propagation.
NS record: Defines the name server for the zone.
A record: Points the name server to the Bind9 server's IP (x.x.x.105 - cluster VIP address).
Replace <INTERNAL_DOMAIN> with your internal domain and <EMAIL> with the administrator email. Increment the serial number (2025040601) on every update.

We define global options for Bind9 in a separate configuration:

# bind9-named-config.yaml
named.conf: |
    options {
      directory "/var/cache/bind";

      dnssec-validation auto;
      listen-on port 5053 { any; };
      listen-on-v6 port 5053 { any; };
      recursion no;
      allow-query { any; };

      querylog no;

    };
    include "/named_config/named.conf.local";

    // No default zones configured.
    // This server is authoritative-only.

Explanation:

directory: Location for cache files.
dnssec-validation auto: Verifies authenticity of external DNS records.
recursion no: Server does not perform recursive lookups - it only serves authoritative zones.
allow-query - any: Accept queries from any IP.
Includes the named.conf.local for zone definitions.

After creating bind9.yaml, bind9-named-config.yaml, and <INTERNAL_DOMAIN>.zone:

Push the files to your Git repository.
ArgoCD will detect changes and deploy Bind9 with the defined configuration.

This setup ensures your internal DNS is authoritative, secure, and persistent, and supports dynamic updates for internal-facing applications.

Record Creation

With the DNS server up and running, we can now start adding records using ExternalDNS.

ExternalDNS is a Kubernetes controller that automatically manages DNS records for cluster resources such as Services, Ingresses, and more. By adding the external-dns.alpha.kubernetes.io/hostname annotation to a Kubernetes resource, ExternalDNS can dynamically create or update the corresponding DNS record. It supports multiple DNS providers, including Cloudflare, AWS Route 53, Google Cloud DNS, and - most relevant to us - rfc2136.

RFC2136 is a DNS update protocol supported by Bind9, which allows us to update DNS records dynamically. With this, ExternalDNS can manage Bind9 records automatically.

To install ExternalDNS, we can use the official Helm chart. For GitOps-based installation via ArgoCD, create an Application object in an external-dns.yaml file:

# external-dns.yaml
# ...
    chart: external-dns
    repoURL: https://charts.bitnami.com/bitnami
    targetRevision: 6.7.2
    helm:
      valuesObject:
        provider: rfc2136
        regexDomainFilter:
          - <INTERNAL_DOMAIN>
        rfc2136:
          host: dns-bind9-dns-tcp.dns.svc.cluster.local
          port: 53
          zone: <INTERNAL_DOMAIN>
          secretName: external-dns-tsig-key
          tsigKeyname: tsig-key
          tsigSecretAlg: hmac-sha512
          tsigAxfr: true
# ...

Here, the provider is set to rfc2136, pointing to our Bind9 service. The zone is the domain we want to manage, and TSIG keys are used for secure updates.

The keys can be generated using the tsig-keygen:

tsig-keygen -a hmac-sha512 tsig-key

The output will be a key in the format:

key "tsig-key" {
    algorithm hmac-sha512;
    secret "C4cYZr0v8IL2l58k0QZtyHd1hMqAbbUOTrZ9I/4WwjIJhkFX3x06BPiRZPXx/Iu76FEy/GzOnMYzPi40CfZ+PQ==";
};

We can then grab the secret and store it in a Kubernetes secret, external-dns-tsig-key.

# external-dns-tsig-key.yaml
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: external-dns-tsig-key
  namespace: dns
stringData:
  rfc2136_tsig_secret: C4cYZr0v8IL2l58k0QZtyHd1hMqAbbUOTrZ9I/4WwjIJhkFX3x06BPiRZPXx/Iu76FEy/GzOnMYzPi40CfZ+PQ==

TLS Certificates

With DNS in place, we also need secure HTTPS access for our applications. Enter Cert Manager - a Kubernetes controller that automates TLS certificate issuance and renewal. Cert Manager supports multiple issuers, including Let's Encrypt, which we'll use.

Install Cert Manager via Helm and GitOps with an Application object in cert-manager.yaml:

# cert-manager.yaml
# ...
    chart: cert-manager
    repoURL: https://charts.jetstack.io
    targetRevision: 1.15.1
    helm:
      valuesObject:
        installCRDs: true
        extraArgs:
          - --dns01-recursive-nameservers-only
          - --dns01-recursive-nameservers=1.1.1.1:53
# ...

This installs the necessary CRDs for certificate management and configures DNS01 challenges to work with recursive nameservers.

Next, create a ClusterIssuer for Let's Encrypt:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: "<EMAIL>"
    privateKeySecretRef:
      name: letsencrypt-production
    solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cert-manager-cf-api-token
              key: token

This config allows Cert Manager to issue certificates for our internal apps using DNS01 challenges via Cloudflare.

Certificates can then be requested by annotating an Ingress resource:

cert-manager.io/cluster-issuer: letsencrypt

Testing the Internal Setup

To test, deploy a simple nginx application with an Ingress:

# nginx-internal-test.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-internal
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-internal
  template:
    metadata:
      labels:
        app: nginx-internal
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-internal
  namespace: default
spec:
  selector:
    app: nginx-internal
  ports:
    - name: http
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-internal
  namespace: default
  annotations:
    external-dns.alpha.kubernetes.io/hostname: nginx.
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  rules:
    - host: nginx.
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nginx-internal
                port:
                  name: http
    tls:
      - hosts:
          - nginx.
        secretName: nginx-tls

Apply the configuration:

# Apply the configuration
kubectl apply -f nginx-internal-test.yaml

Once applied, the following should happen:

The nginx-internal Deployment will be created and the pod will start running.
The nginx-internal Service will be created, exposing the pod on port 80.
The nginx-internal Ingress resource will be created, and the hostname nginx.<INTERNAL_DOMAIN> will be managed by ExternalDNS.
The letsencrypt ClusterIssuer will be used to issue a TLS certificate for the hostname nginx.<INTERNAL_DOMAIN>.

We can check the status of the Ingress resource with:
```
## Check the status of the Ingress resource
kubectl get ingress nginx -n default
```
This should show the hostname and the TLS certificate that was issued. If everything is working correctly, we should be able to access the nginx application using https://nginx.<INTERNAL_DOMAIN>.
The ExternalDNS controller will automatically create a DNS record for nginx.<INTERNAL_DOMAIN> in Bind9, pointing to the IP of the Ingress Controller.
The Bind9 server will be able to resolve the hostname nginx.<INTERNAL_DOMAIN> to the IP of the Ingress Controller, allowing access from the internal network.
The TLS certificate will be issued by Let's Encrypt and will be valid for nginx.<INTERNAL_DOMAIN>. This allows HTTPS access without browser warnings.

Check the status of the TLS certificate:
```
## Check the status of the TLS certificate
kubectl get certificate nginx-tls -n default
```
This should display the certificate's status and expiration date. If everything is working correctly, the certificate should be valid for the next few months.
After issuance, a new secret named nginx-tls will be created, containing the TLS certificate and private key. The Ingress Controller will use this secret to terminate TLS connections.
The Ingress resource will automatically use the nginx-tls secret for TLS termination.

After completing these steps, the nginx application should be accessible over HTTPS at <https://nginx>.<INTERNAL_DOMAIN> without warnings.

Test the application:

# Test the application
curl "https://nginx.<INTERNAL_DOMAIN>" @x.x.x.101:30053

Since the Bind9 service is exposed on port 30053 across three nodes, you can use any node for testing.

Unbound Forwarder

As a final step, we can configure Unbound to forward DNS queries to our Bind9 server. This allows Unbound to act as a DNS resolver for the internal network while still resolving public domain names normally.

To configure this:

Open the OpnSense Interface.
Navigate to Services -> Unbound DNS -> Query Forwarding.
Add a new forwarding entry with the following settings:
- Domain: <INTERNAL_DOMAIN>
- Forward IP: x.x.x.101 (select the node you want to forward queries to)
- Port: 30053
Save and apply the changes.

Once applied:

Any DNS query for <SERVICE>.<INTERNAL_DOMAIN> will be forwarded by Unbound to the Bind9 server.
Bind9 will respond with the internal record from its authoritative zone if it exists.
Public domains will continue to be resolved via Unbound's configured upstream resolvers (e.g., 1.1.1.1, 8.8.8.8).
This setup ensures that internal-facing applications are accessible from anywhere inside the network using their internal hostnames.

You can verify this by running:

# Resolve an internal application using Unbound
dig nginx.<INTERNAL_DOMAIN> @<FIREWALL_IP>

; &lt;&lt;&gt;&gt; DiG 9.10.6 &lt;&lt;&gt;&gt; nginx. @
;; global options: +cmd
;; Got answer:
;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 23690
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;nginx.. IN   A

;; ANSWER SECTION:
nginx.. 0 IN  A       x.x.x.105

;; Query time: 15 msec
;; SERVER: #53()

The result should return the internal IP address of the Ingress Controller as provided by Bind9.

We now should be able to access our internal services using their hostname without needing to specify the DNS server.

# Test the internal service
curl "http://nginx-internal.<INTERNAL_DOMAIN>"

Public facing DNS records

For public-facing applications, we'll be using Cloudflare as our DNS provider. Cloudflare is a content delivery network (CDN) that offers a fast, secure, and reliable network for websites and applications. On top of that, it provides DNS services, allowing us to manage domain names and resolve them to IP addresses easily.

I chose Cloudflare because their free tier lets us manage DNS records and resolve them to IP addresses without cost.

Both DNS records and TLS certificates will be managed through Cloudflare. We'll also take advantage of other features offered in their free plan, like Cloudflare Tunnels, which will simplify securely exposing our services to the public internet.

Cloudflare Tunnels

Cloudflare Tunnels (CF Tunnels) let us expose internal services to the public internet without revealing our own IP address. As the name suggests, they act as a tunnel between the server running your service and Cloudflare itself. When a client accesses your public domain, the IP it sees will be one of Cloudflare's public IP addresses. CF then routes the traffic through the tunnel, letting it reach your infrastructure safely.

This approach minimizes our attack surface. If we exposed our own IP, we'd be more vulnerable to attacks like DoS or DDoS. By letting Cloudflare handle the initial traffic, we automatically gain features like IP allowlists, attack protection, and traffic control - features we'd otherwise have to implement ourselves. Most importantly for us, it hides our IP, manages SSL certificates, and handles DNS records automatically.

Luckily, there's a Kubernetes-friendly project called cloudflare-operator that simplifies setting up CF Tunnels. It provides custom Kubernetes resources to manage tunnels directly from your cluster.

Installing the Cloudflare Operator

We can install the operator in our Kubernetes cluster via ArgoCD, just like we've done in previous chapters:

# cf-operator.yaml
# ...
    project: default
    source:
      repoURL: https://github.com/adyanth/cloudflare-operator.git
      targetRevision: main
      path: config/default
    destination:
      name: in-cluster
      namespace: cf-operator
# ...

Next, we need a Cloudflare API token with the following permissions:

Cloudflare Tunnel: Edit
Account Settings: Read
<PUBLIC_DOMAIN> DNS: Edit

We store this token as a Kubernetes secret:

# cf-api-token.yaml
apiVersion: v1
kind: Secret
metadata:
  name: cf-api-token
  namespace: cf-operator
type: Opaque
data:
  CLOUDFLARE_API_TOKEN: "<BASE64_ENCODED_TOKEN>""

Creating a Tunnel

We then define a ClusterTunnel resource to manage the Cloudflare tunnel:

# cf-tunnel.yaml
apiVersion: networking.cfargotunnel.com/v1alpha1
kind: ClusterTunnel
metadata:
  name: cf-tunnel
spec:
  newTunnel:
    name: cf-tunnel
  size: 2
  cloudflare:
    email: "<EMAIL>"
    domain: "<PUBLIC_DOMAIN>"
    secret: cf-api-token
    accountName: "<ACCOUNT_NAME>"

Expose application

With the tunnel in place, we expose our apps using a TunnelBinding resource:

# cf-expose-nginx.yaml
apiVersion: networking.cfargotunnel.com/v1alpha1
kind: TunnelBinding
metadata:
  name: expose-nginx
  namespace: default
subjects:
  - name: nginx-default
    spec:
      fqdn: nginx.<PUBLIC_DOMAIN>
      target: http://nginx.default.svc.cluster.local:8080
      noTlsVerify: false
tunnelRef:
  kind: ClusterTunnel
  name: cf-tunnel

Cloudflare automatically creates the DNS records and generates TLS certificates for the application.

Testing the Public Setup

To demonstrate how Cloudflare Tunnels works, let's deploy a simple nginx application and expose it via the tunnel previously created.

# nginx-external-test.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-external
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-external
  template:
    metadata:
      labels:
        app: nginx-external
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-external
  namespace: default
spec:
  selector:
    app: nginx-external
  ports:
    - name: http
      port: 80
      targetPort: 80

Apply the configuration:

# Apply the configuration
kubectl apply -f nginx-external-test.yaml

Once applied, the following should happen:

The nginx Deployment will be created and the pod will start running.
The nginx Service will be created, exposing the pod on port 80.
The cf-nginx TunnelBinding will be created, linking the Cloudflare Tunnel to the nginx service.
The Cloudflare Tunnel will be established, allowing external traffic to reach the nginx service.
The Cloudflare DNS records will be created, pointing to the tunnel.
The Cloudflare SSL certificates will be issued for the nginx service.
The nginx service will be accessible via the public domain.

We can check the status of the tunnel using the following command:

kubectl describe tunnelbinding cf-nginx -n default

Where we'll see all of this happening directly from the resource events. It should look like this:

  Type    Reason          Age   From                 Message
  ----    ------          ----  ----                 -------
  Normal  Configuring     15m   cloudflare-operator  Configuring ConfigMap
  Normal  ApplyingConfig  15m   cloudflare-operator  Applying ConfigMap to Deployment
  Normal  AppliedConfig   15m   cloudflare-operator  ConfigMap applied to Deployment
  Normal  Configured      15m   cloudflare-operator  Configured Cloudflare Tunnel
  Normal  MetaSet         15m   cloudflare-operator  TunnelBinding Finalizer and Labels added
  Normal  CreatedDns      15m   cloudflare-operator  Inserted/Updated DNS/TXT entry

With these resources in place, the Cloudflare Tunnel can forward external traffic to the nginx service using the TunnelBinding we created earlier. Users can now access the application via:

# Test the application
curl "https://nginx.<PUBLIC_DOMAIN>"

This setup demonstrates the full flow: Cloudflare handles DNS & TLS, tunnels the traffic to our cluster, and the Service routes it to the Deployment pod.

Conclusion

In this chapter, we tackled one of the most important steps in making our cluster truly usable from anywhere: DNS and SSL. We mapped out the architecture, set up Bind9 for rock-solid internal DNS, and leaned on Cloudflare for public-facing names - all with automation in mind. Thanks to ExternalDNS and Cert-Manager, record creation and TLS issuance now happen without manual intervention, keeping everything secure and up to date.

With this in place, our homelab services have:

A clean separation between internal and public DNS management.
Automated DNS updates directly from Kubernetes resources.
Seamless HTTPS access - internally and externally - without scary browser warnings.

The end result? Any service we spin up can be securely exposed, tested, and shared with almost no extra work. We're no longer manually juggling DNS zones or dealing with certificate renewal headaches - it's all declarative, reproducible, and in sync with our GitOps flow.

From here, we can focus on deploying more useful applications, knowing that they'll just work whether we're inside the lab or halfway across the world. In the next chapter, we'll start putting this setup to use by deploying real workloads and integrating them into our automated homelab stack.

Originally published at https://techquests.dev on August 15, 2025.

Home Lab: Chapter 6 — Kubernetes Ingress Controller

Andre Nogueira — Fri, 03 Oct 2025 18:22:02 +0000

Howdy,

In this chapter, we're going to look at how to expose services running inside our Kubernetes cluster to the outside world using an Ingress Controller. We'll be using the NGINX Ingress Controller and taking full advantage of our GitOps setup with ArgoCD.

Let's get into it.

What Is an Ingress Controller?

An Ingress Controller is a Kubernetes resource that handles external access to services running inside your cluster - usually over HTTP or HTTPS. You can think of it as a smart traffic router sitting at the edge of your cluster.

It watches for Ingress resources and knows how to route traffic accordingly. It also supports other nice things like:

Load balancing
SSL termination
Name-based virtual hosting

In other words, it gives you centralized control over how incoming traffic is handled.

Preparing the Cluster

Before installing the Ingress Controller, we need to make sure it can actually receive external traffic. For that, we'll expose it using a Kubernetes Service of type LoadBalancer.

The LoadBalancer service will act as the public entry point, and from there, the Ingress Controller will decide how to route the traffic internally.

Now, since we're using Cilium as our CNI (as mentioned in Chapter 3), we've got a cool feature available to us: Cilium LB IPAM (Load Balancer IP Address Management). This lets us assign specific IPs to LoadBalancer services - perfect for when we want to reserve a static IP for our Ingress Controller.

This is especially useful if we plan to point a DNS record to the controller later, which is exactly what we'll do in future chapters.

Assigning a Static IP with Cilium

To assign a static IP using Cilium, we need to define a CiliumLoadBalancerIPPool object. This object tells Cilium which IPs it can use for LoadBalancer services, and under what conditions.

Here's the configuration:

# ippool.yaml
---
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: default-pool
spec:
  blocks:
    - cidr: "x.x.x.105/32"
  serviceSelector:
    matchLabels:
      "io.kubernetes.service.namespace": "ingress-nginx"

This will assign the IP x.x.x.105 to any LoadBalancer service in the ingress-nginx namespace.

To apply it, we can run the following command:

# Apply the configuration
kubectl apply -f ippool.yaml

With this in place, the Ingress Controller will get that static IP when we install it.

Installing NGINX Ingress Controller

Now that the IP pool is in place, we can move on to installing the NGINX Ingress Controller. Installing the NGINX Ingress Controller is straightforward. Like ArgoCD and Cilium, we'll use its official Helm chart. But since we already have a fully working GitOps setup (thanks to our ArgoCD configuration in the previous chapter), we can add the NGINX Ingress Controller to the Git repository and have it installed by ArgoCD. We can do this by creating an Application object that will define the NGINX Ingress Controller. We'll define it in a ingress-nginx.yaml file with the following content:

# ingress-nginx.yaml
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ingress-nginx
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    chart: ingress-nginx
    repoURL: https://kubernetes.github.io/ingress-nginx
    targetRevision: 4.12.1
    helm:
      valuesObject:
        controller:
          service:
            type: LoadBalancer
            externalTrafficPolicy: Cluster
            annotations:
              io.cilium/lb-ipam-ips: 'x.x.x.105'
          metrics:
            enabled: true
  destination:
    name: in-cluster
    namespace: ingress-nginx
  syncPolicy:
    automated:
      prune: false
      selfHeal: true
      allowEmpty: false
    syncOptions:
      - CreateNamespace=true
    managedNamespaceMetadata:
      labels:
        pod-security.kubernetes.io/enforce: privileged
        pod-security.kubernetes.io/enforce-version: latest
  info:
    - name: 'Description:'
      value: Ingress controller

This file defines an ArgoCD Application that will install the NGINX Ingress Controller using the Helm chart from the official repository. Here's a quick breakdown of what each part does:

service.type: LoadBalancer: creates a service of type LoadBalancer that will expose the Ingress Controller to the outside world.
service.externalTrafficPolicy: Cluster: allows the traffic to be routed to all the nodes in the cluster instead of just the node where the pod is running.
io.cilium/lb-ipam-ips: assigns the previous allocated IP address to the Load Balancer service.
metrics.enabled: true: enables the metrics server for the Ingress Controller, exposing metrics like the number of requests, response time, etc.
pod-security.kubernetes.io/enforce: sets the Pod Security Standards to privileged, allowing the Ingress Controller to run with elevated privileges. This is necessary for the Ingress Controller to function correctly.

Once this file is added to your Git repo, ArgoCD will pick it up and deploy everything.

We can follow the progress of the installation by checking the ArgoCD UI by running kubectl port-forward svc/argocd-server -n argocd 8080:80 and navigating to http://localhost:8080. You can log in with the default credentials (admin/admin) or the credentials you set up in the previous chapter.

Testing the Ingress Controller

Once ArgoCD has deployed the controller, we can check if it's working by making a simple curl request to the IP we assigned:

# Make a request to the Ingress Controller
curl http://x.x.x.105

We now should see something like this:

## Response from the Ingress Controller
<html>
  <head><title>404 Not Found</title></head>
  <body>
    <center><h1>404 Not Found</h1></center>
    <hr><center>nginx</center>
  </body>
</html>

That 404 is actually a good sign - it means the Ingress Controller is up, responding to requests, and just doesn't have any routes defined yet (because we haven't created any Ingress resources).

Visual Recap

Here's what we just set up, in diagram form:

Conclusion

In this chapter, we set up the foundation for managing external traffic in our Kubernetes cluster. Here's what we accomplished:

We used Cilium's LB IPAM feature to assign a static IP to a LoadBalancer service.
We installed the NGINX Ingress Controller using ArgoCD and Helm.
We validated that the controller is working by sending it a direct request.

Now that we've got an Ingress Controller up and reachable from the outside, we can move on to the next step: making it actually useful by adding routing rules, setting up DNS, and handling SSL/TLS termination.

In the next chapter, we'll look at how to:

Configure DNS records pointing to your static IP
Automatically issue and renew SSL certificates using cert-manager
Route traffic to real services using Ingress resources

We're just getting started with Ingress - but the foundation is solid. Catch you in the next one.

Originally published at https://techquests.dev on July 29, 2025.

Home Lab: Chapter 5 — Kubernetes Managing Secrets

Andre Nogueira — Fri, 03 Oct 2025 18:05:53 +0000

Howdy,

Secrets are a fundamental part of any application - it's how we securely store sensitive information. In Kubernetes, there are several approaches to handling secrets. In this chapter, we'll explore different ways to manage secrets in Kubernetes.

What is a Secret?

First off, what exactly is a Secret? A Secret is a Kubernetes object that holds a small amount of sensitive data - like a password, token, or key. Without Secrets, you might have to hard-code these values into Pod specs or container images. Users can create Secrets manually, and Kubernetes also generates some automatically.

Our Scenario

In the previous chapter, we already needed to work with some sensitive data:

We generated Talos Secrets, which include a bundle of crucial credentials.
We created ArgoCD Secrets, which hold a GitHub private key for repository access.

So the question is: How can we securely manage these secrets while still getting the automation benefits from Kubernetes and ArgoCD?

Secrets Management Options

You've got a few choices for managing secrets in Kubernetes:

Built-in Kubernetes Secrets
Third-party secret managers (Vault, AWS/Azure/GCP Key Management, etc.)
Kubernetes operators (e.g. Sealed Secrets by Bitnami)

For simplicity in this chapter, we'll stick with built-in Kubernetes Secrets - though we may revisit other options later.

Kubernetes Secrets

Kubernetes Secrets let you store things like passwords, OAuth tokens, and SSH keys securely. It's much safer and more flexible than embedding secrets directly in Pod specs or container images. However, remember: Kubernetes Secrets are only Base64 encoded by default, not encrypted. So if you require encryption at rest, layer in a third-party tool.

Managing Secrets

While Kubernetes makes it easy to use Secrets, we still need an automated and secure way to manage them. That's where ArgoCD comes in. With ArgoCD, we can store our secrets in a Git repository and use a Kustomize overlay to apply them to the cluster.

Kustomize lets you customize raw, template-free YAML files without altering the originals. This means we can keep clean, reusable manifests while layering in environment-specific or sensitive configurations, like secrets, on top.

So far, so good. But you might be wondering:

"How can we safely store secrets in Git?"

Great question! The answer is encryption - and this is where SOPS comes in.

SOPS (Secrets OPerationS) is a flexible tool that encrypts files in a way that lets you decrypt them later when needed. It supports various encryption backends: PGP, GnuPG, AWS KMS, Azure Key Vault, Google Cloud KMS, Vault, and more.

For this setup, I'll be using age - a modern, simple, and secure encryption tool that serves as a lightweight alternative to GPG.

By combining:

Kustomize
SOPS + age
And the KSOPS plugin (a Kustomize plugin for SOPS)

... we get a powerful GitOps-friendly workflow:

This setup allows us to store encrypted secrets in Git, and ArgoCD will automatically decrypt them using KSOPS when applying them to the cluster. This way, we can manage our secrets securely while still benefiting from GitOps automation.

Once the secrets are decrypted, they are applied to the cluster as standard Kubernetes Secrets, which can then be consumed by Pods and other resources.

How to use KSOPS

Generate an age key pair:

## Generate the age keys
age-keygen -o ~/.config/sops/age/keys.txt

We should get as the output of this command:

age-keygen -o key.txt
Public key: age1efe0s548vkwgvjkdtgu4exf9v4mtltjv6rn5yww33yd75ad7r5xsjq7f8l

Create a Kubernetes secret to store the age public key:

## Create the sops-age secret
kubectl create secret generic sops-age \
  --namespace argocd \
  --from-file=keys.txt=~/.config/sops/age/keys.txt

Create a .sops.yaml to define which files/fields should be encrypted:
```
# .sops.yaml
---
stores:
yaml:
    indent: 2
creation_rules:
- path_regex: secrets.yaml
    encrypted_regex: '^(id|secret|bootstraptoken|secretboxencryptionsecret|token|ca|crt|key)$'
    age: age1efe0s548vkwgvjkdtgu4exf9v4mtltjv6rn5yww33yd75ad7r5xsjq7f8l
```
This configuration instructs SOPS to encrypt the secrets.yaml file using the age1efe0s548vkwgvjkdtgu4exf9v4mtltjv6rn5yww33yd75ad7r5xsjq7f8l public key. It targets any field that matches the following regular expression:

The age1efe0s548vkwgvjkdtgu4exf9v4mtltjv6rn5yww33yd75ad7r5xsjq7f8l key is the public key generated during the creation of the sops-age Kubernetes secret
Encrypt your file:
```
## Encrypt the file content
ksops -e -i secrets.yaml
```
If we now inspect the secrets.yaml file, we'll see that the content is now
encrypted. To decrypt the file content back to its original state, we can run
the following command:
```
## Decrypt the file content
ksops -d -i secrets.yaml
```
Decrypting the file content requires the private key that we generated when we created the sops-age secret.

Config ArgoCD to manage Secrets

To configure ArgoCD to manage secrets, we need to tweak the values.yaml file that we created in the previous Chapter 4 and add the following configuration:

# values.yaml
---
repoServer:
  env:
    - name: XDG_CONFIG_HOME
      value: /.config
    - name: SOPS_AGE_KEY_FILE
      value: /.config/sops/age/keys.txt
  volumes:
    - name: custom-tools
      emptyDir: {}
    - name: sops-age
      secret:
        secretName: sops-age
  initContainers:
    - name: install-ksops
      image: viaductoss/ksops:v4.3.3
      command: ['/bin/sh', '-c']
      args:
        - echo "Installing KSOPS...";
          mv ksops /custom-tools/;
          mv kustomize /custom-tools/;
          echo "Done.";
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
  volumeMounts:
    - mountPath: /usr/local/bin/kustomize
      name: custom-tools
      subPath: kustomize
    - mountPath: /.config/kustomize/plugin/viaduct.ai/v1/ksops/ksops
      name: custom-tools
      subPath: ksops
    - mountPath: /.config/sops/age/keys.txt
      name: sops-age
      subPath: keys.txt

This configuration enables ArgoCD to manage secrets securely using KSOPS and age. It sets the XDG_CONFIG_HOME environment variable to /.config, directing SOPS to look for its configuration files there. The SOPS_AGE_KEY_FILE is set to /.config/sops/age/keys.txt, so SOPS can locate the age private key used for decryption.

Two volumes are defined: custom-tools, an emptyDir volume for storing the KSOPS and Kustomize binaries, and sops-age, a secret volume that holds the age key file. An initContainer named install-ksops installs the necessary binaries into the custom-tools volume before the main ArgoCD container starts.

The volumes are mounted inside the pod: custom-tools is mounted at /usr/local/bin/kustomize and /custom-tools to make the binaries accessible, while sops-age is mounted at /.config/sops/age/keys.txt so that the decryption key is available for SOPS during runtime.

We also need to create the sops-age secret that contains the age keys file:

# Generate the age keys
age-keygen -o ~/.config/sops/age/keys.txt

## Create the secret
cat ~/.config/sops/age/keys.txt | kubectl create secret generic sops-age --namespace argocd --from-file=keys.txt=/dev/stdin

With the secrets now created, the only thing left to do is apply the
new configuration to the cluster:

# Upgrade the ArgoCD installation
helm upgrade --install argocd argo/argo-cd \
    --namespace argocd \
    --values values.yaml

Voilà! ArgoCD is now KSOPS-enabled with age key support. We now have a secure and automated way to manage our secrets in a Git repository and still have them applied to the cluster in a secure way.

Adding Secrets to the Git

Create your secret manifest, e.g. example-secret.yaml:

# example-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: example-secret
stringData:
  foo: bar

Add an entry in .sops.yaml to match and encrypt foo:

# .sops.yaml
# ...
- path_regex: example-secret.yaml
  encrypted_regex: '^foo'
  age: age1efe0s548vkwgvjkdtgu4exf9v4mtltjv6rn5yww33yd75ad7r5xsjq7f8l

And then we can encrypt it by running:

## Encrypt the file content
ksops -e -i example-secret.yaml

If we now inspect the example-secret.yaml file, we'll see that the content is
now encrypted:

# example-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: example-secret
stringData:
  foo: ENC[AES256_GCM,data:s7FsAPs=,iv:ywvzww/Jq342vkENSEXLxopD8aAf3jCE0TPfwILJz1Q=,tag:DYFKhAVr7pgf1cW5+cevbw==,type:str]
sops:
  kms: []
  gcp_kms: []
  azure_kv: []
  hc_vault: []
  age:
    - recipient: age1efe0s548vkwgvjkdtgu4exf9v4mtltjv6rn5yww33yd75ad7r5xsjq7f8l
      enc: |
        -----BEGIN AGE ENCRYPTED FILE-----
        YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0cXo4YTNaYTdGT1Y3U29N
        WFR4WlJISTRUaU1jTXVzTUFqZCsxQVBoaEZvCmRQZVAzMS9ZM3RqbTlrSEdyUmJj
        ejFZNDVlMEpZY2s3Z1VSdTdQYWk3MmMKLS0tIFQxVUhibEVHSVJtb09XNkRxcVN5
        TUVIcmdaRloyUVZzckNMbkpVVXo5WjQK70C/ZvuailOheaSXMM5Rx+CGXZ9K98tw
        ++Q6PZPafdZxkwSIRjZU6ihAk0L6TXs3MJ93yvn/n3CA9zQp9tDuXg==
        -----END AGE ENCRYPTED FILE-----
  lastmodified: '2025-04-02T21:12:33Z'
  mac: ENC[AES256_GCM,data:t3MIPtm19pt+Ov27VkQvrDM/4IN48KXiOpQQlP1czWn12sv68pMt/fALxnrSM3jgv2q0reG5j9vJlA9zFPVw8sdudZ7mmY+HoFIfp8ryZOqX1Ro2hBPR4aj9eXBZT5Gjwf8eOYgYKRdOev6pRmtTA5wJ2qRAZkhvBm3mHHp7d+E=,iv:57/nNwc25K0J632kgo7MX7J0FyUN13ED7wwww9qOAMQ=,tag:qV1/qTDyAUrIFah2AeXrmQ==,type:str]
  pgp: []
  encrypted_regex: ^foo
  version: 3.9.0

As we can see, the foo field is now encrypted, and a new sops field has been added, which contains the encryption metadata.

The kms, gcp_kms, azure_kv, hc_vault, and pgp fields are lists of encryption keys for their respective backends. Since none of these were used in this case, they are empty.

The age field lists the age keys used to encrypt the file. Here, we used the age1efe0s548vkwgvjkdtgu4exf9v4mtltjv6rn5yww33yd75ad7r5xsjq7f8l key, so the field includes both the recipient and the corresponding encrypted payload.

The lastmodified field records when the file was last updated.

The mac field is a message authentication code used to verify the file's integrity and ensure it hasn't been tampered with.

The encrypted_regex field specifies a regular expression used by SOPS to determine which fields in the document should be encrypted.

The version field indicates the SOPS file format version used for encryption.

Add a Kustomize overlay by creating a kustomization.yaml file along with a KSOPS generator:

# kustomization.yaml
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
name: example-secret
generators:
- example-secret-generator.yaml

And the associated generator file:

# example-secret-generator.yaml
---
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
name: example-secret-generator
files:
- example-secret.yaml

This overlay tells Kustomize to use the KSOPS plugin to decrypt example-secret.yaml before applying it to the cluster, enabling secure GitOps-driven secrets management.

Now you can push the changes to your Git repository and let ArgoCD handle the deployment. When ArgoCD applies the configuration, it will automatically decrypt the secret using the provided Age keys and apply it to the cluster as a standard Kubernetes Secret.

Ensure that ArgoCD is already configured to track the correct Git repository and that the example-secret.yaml file is located in the expected directory.

With that, the setup is complete. You now have a secure and automated workflow for managing secrets through Git and ArgoCD.

Conclusion

In this chapter I've demonstrated how to securely manage Kubernetes secrets using GitOps principles. By integrating KSOPS with ArgoCD and Kustomize, we can encrypt secrets, store them in Git, and have them decrypted and applied to the cluster automatically.

While this setup offers a solid foundation, it's not the most advanced solution in terms of security. For production environments requiring features like access controls, audit logging, or automatic key rotation, consider tools such as HashiCorp Vault or a Kubernetes-native solution like Sealed Secrets.

That said, this approach strikes a good balance between simplicity, security, and GitOps compatibility - making it an excellent starting point for secret management in Kubernetes.

Originally published at https://techquests.dev on July 2, 2025.

Home Lab: Chapter 4 — Kubernetes GitOps with ArgoCD

Andre Nogueira — Tue, 10 Jun 2025 15:08:44 +0000

Howdy!

Ever since I discovered GitOps, I've been in love with the concept. The idea of managing all your infrastructure configuration from a centralized Git repository - and having a tool automatically apply those changes - is incredibly powerful.

GitOps brings together Infrastructure as Code (IaC) and Continuous Integration/Continuous Deployment (CI/CD) in a seamless, declarative workflow. It's the ideal way to manage a Kubernetes cluster.

In the past, I've used Flux to implement GitOps, but I've always been curious about ArgoCD. After hearing so many good things about it, I decided it was finally time to give it a try - and this project was the perfect opportunity.

What is ArgoCD?

ArgoCD is a declarative GitOps continuous delivery tool for Kubernetes. It follows the GitOps pattern of using Git repositories as the source of truth for defining the desired application state.

It is implemented as a Kubernetes controller that continuously monitors running applications and compares their current live state against the desired target state (as defined in Git). A deployment is considered in sync when the live state matches the target state. If they differ, ArgoCD performs a kubectl apply to reconcile the live state with the target state.

Installation

Installing ArgoCD is straightforward. We can use the official Helm chart. First, add the ArgoCD Helm repository:

# Create the namespace
kubectl create namespace argocd

# Add the repository
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

Since we're using Cilium as the CNI, we need to exclude Cilium resources from ArgoCD's control. Create a values.yaml file with the following:

# values.yaml
configs:
  cm:
    resource.exclusions: |
      - apiGroups:
          - cilium.io
        kinds:
          - CiliumIdentity
        clusters:
          - "*"

This prevents ArgoCD from managing Cilium resources, which could interfere with Cilium's operation. For more details, see Troubleshooting Cilium deployed with Argo CD.

We can also provide a custom admin password for the ArgoCD UI. The password must be hashed. You can generate a hash using htpasswd:

# Generate the password hash
htpasswd -nbBC 10 "" <PASSWORD> | tr -d ':\n'

Then add it to the values.yaml file:

# values.yaml
configs:
  secret:
    argocdServerAdminPassword: <PASSWORD_HASH>

Now install the chart:

helm upgrade --install argocd argo/argo-cd \
    --namespace argocd \
    --values values.yaml

This installs ArgoCD into the argocd namespace. You can access the UI via port forwarding:

kubectl port-forward svc/argocd-server -n argocd 8080:80

Add Repository

With ArgoCD installed, the next step is to connect a Git repository containing your Kubernetes manifests. ArgoCD includes an operator and several CRDs (Custom Resource Definitions). We use the Application CRD to define which repository and path to sync.

Here's an example init.yaml:

# init.yaml
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: init
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: <REPO_URL>
    targetRevision: HEAD
    path: <PATH>
  destination:
    name: in-cluster
    namespace: argocd
  syncPolicy:
    automated:
      selfHeal: true
  info:
    - name: 'Description:'
      value: 'Entrypoint to all homelab apps'

For more details, see the ArgoCD Application Spec.

If your repository is private, create a secret with credentials and label it for ArgoCD:

# github.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: github
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: <REPO_URL>
  sshPrivateKey: <SSH_PRIVATE_KEY>

Apply the secret and application:

kubectl apply -f github.yaml
kubectl apply -f init.yaml

Once applied, you should see the application appear in the ArgoCD UI.

You can access the UI at by port forwarding the service:

kubectl port-forward svc/argocd-server -n argocd 8080:443

Then navigate to http://localhost:8080 in your browser. The default username is admin, and the password is the one you set earlier.

Adding Applications

With the repository connected, you can start adding applications. Here's an example app.yaml to deploy a Helm chart:

# app.yaml
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: <APP_NAME>
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    chart: <CHART_NAME>
    repoURL: <CHART_REPO>
    targetRevision: <CHART_VERSION>
    helm: {}
  destination:
    name: in-cluster
    namespace: <NAMESPACE>
  syncPolicy:
    automated:
      prune: false
      selfHeal: true
      allowEmpty: false
    syncOptions:
      - CreateNamespace=true
  info:
    - name: 'Description:'
      value: My first Application with ArgoCD

Ensure this file is in the <PATH> specified in your init.yaml.

Push the file to your repository, and ArgoCD will detect it. You can then sync it through the UI and watch your application get deployed - this is the magic of GitOps.

Conclusion

ArgoCD is a powerful tool that enables managing Kubernetes clusters declaratively through Git. By treating your Git repository as the source of truth, you gain version control, automation, and a clear audit trail of infrastructure changes.

It's an excellent way to combine IaC and CI/CD - and I'm excited to explore what more I can do with it!

Originally published at https://techquests.dev on May 31, 2025.

Home Lab: Chapter 3 — Kubernetes Setup

Andre Nogueira — Thu, 15 May 2025 23:28:48 +0000

Howdy!

In this chapter, I'll walk through the setup of the Kubernetes cluster. For the Operating System (OS) of the nodes, I'll be using Talos. As mentioned earlier, the cluster will consist of three physical machines. Since Kubernetes uses a control-plane/worker model and we only have three nodes, each one will serve as both a control-plane and a worker. This setup allows workloads to be scheduled on all nodes while maintaining control-plane functionality.

What is Talos?

Talos is a modern, minimalistic operating system designed specifically to run Kubernetes-and nothing else. It is immutable, meaning the OS is read-only and cannot be modified. This immutability improves security, making it more difficult for attackers to alter the system.

Talos is also built to be managed entirely via Kubernetes, simplifying cluster operations. On a personal note, it's a project I’ve been following for some time, and I’m excited to finally try it out.

Setting up Talos

Talos provides a command-line tool called talosctl, which is used to manage and interact with the cluster. Similar to how kubectl is used for managing Kubernetes resources, talosctl is used to create, configure, and operate the Talos-based infrastructure itself.

To setup the cluster, you first need to download the talosctl binary. You can download it from the Talos releases page or you can use the following command to download it:

# Linux
curl -sL https://talos.dev/install | sh

# MacOS
brew install siderolabs/tap/talosctl

I'll go through each step I took to bootstrap my cluster, but in short, the process to install Talos OS involves the following steps:

Download the Talos image
Flash the image to a USB drive
Boot the node from the USB drive
Install Talos on the nodes
Reboot the nodes

Preparing Nodes

Before we begin the configuration, there’s some initial setup we need to complete-specifically, assigning IP addresses to the nodes. Each node will be given a dedicated IP address to make identification and management easier. All nodes will be connected to the DMZ network, and for simplicity, we’ll assign their IPs using DHCP.

While DHCP typically assigns IP addresses dynamically, we can configure static leases to ensure each node always receives the same IP. This is done by mapping each node’s MAC address to a specific IP address in the DHCP settings.

To do this, we’ll go into the DHCP configuration on the OPNSense interface and set up static mappings for each node.

Node 1: x.x.x.101
Node 2: x.x.x.102
Node 3: x.x.x.103

In addition, we can restrict the range of IPs in the DHCP pool - for example, from x.x.x.101 to x.x.x.104 - since we also want to reserve an IP for the NAS. This limited range ensures that only a small, predefined set of IP addresses is available for assignment. It adds an extra layer of control by preventing the DHCP server from assigning addresses to unexpected devices that might join the network.

Setting up the Nodes

To set up the nodes, we'll first need to download the Talos image and flash it to a USB drive. The image can be obtained from the Talos releases page.

Once downloaded, you can flash the image to a USB drive using a tool like dd. Here’s an example command:

sudo dd if=talos.iso of=/dev/sdX bs=4M status=progress && sync

Replace /dev/sdX with the path to the USB drive. Be careful with this
command as it will overwrite the data on the USB drive.

Once the image has been successfully flashed to the USB drive, you can proceed to boot the node from it. To do this, you may need to enter the BIOS or UEFI settings and configure the boot order to prioritize the USB drive.

After the node boots into the Talos installer, you can install Talos onto the system using the following command:

talosctl install --node x.x.x.x

This command installs Talos on the node. Once the installation is complete, simply reboot the machine-it will now boot directly into Talos. Repeat this process for each node in the cluster to complete the installation.

Prepare Nodes Config

Once We've downloaded the talosctl binary, we can use it to generate the initial cluster configuration with the following command:

talosctl gen config

This command generates a default configuration, which won't fully meet our needs. In the next section, we'll customize it accordingly.

Running this command produces three files:

talosconfig: Used by talosctl to connect to and manage the cluster.
controlplane.yaml: Configuration used to bootstrap control plane nodes.
worker.yaml: Configuration used to bootstrap worker nodes. We won’t be using this file, since all of our nodes will act as control plane nodes (while still being able to run workloads).

Patching Nodes Config

To modify the configuration of the control plane nodes (or workers), we could manually edit the generated files. However, a more structured and maintainable approach is to use patches. talosctl supports adding patches to the configuration, which allows us to organize our changes cleanly and consistently-especially useful when managing multiple nodes or environments.

We'll be applying the following modifications using patches:

Allow Control Plane Workloads: This will enable workloads to be scheduled on the control plane nodes. Since I want the control plane nodes to also act as worker nodes, this configuration is essential to allow scheduling of workloads on them.
```
# patches/allow-controlplane-workloads.yaml
cluster:
  allowSchedulingOnControlPlanes: true
```
Control Plane Node 1: This configuration is specific to the Node 1. It will essentially add a specific hostname, which should make it easily identifiable.
```
# patches/control-plane-node-1.yaml
- op: replace
  path: /machine/network/hostname
  value: clustarino-k8s-1
```

Control Plane Node 2: Same configuration, but with a different hostname.

# patches/control-plane-node-2.yaml
- op: replace
  path: /machine/network/hostname
  value: clustarino-k8s-2

Control Plane Node 3: Same configuration, but with a different hostname.

# patches/control-plane-node-3.yaml
- op: replace
  path: /machine/network/hostname
  value: clustarino-k8s-3

Interface Names: Allow the interface IDs to be more easily identifiable:

# patches/interface-names.yaml
machine:
  install:
    extraKernelArgs:
      - net.ifnames=0

DHCP: This will enable DHCP on the Ethernet interface - because of the previous config, it will allow the interface to be identified by this name.
```
# patches/dhcp.yaml
machine:
  network:
    interfaces:
      - interface: eth0
        dhcp: true
```

Disable Kubeproxy and CNI: This will disable the Kubeproxy and the default CNI that comes with Talos. It will allow us to install our own.

# patches/disable-kube-proxy-and-cni.yaml
cluster:
  network:
    cni:
      name: none
  proxy:
    disabled: true

DNS: In the previous chapter, I enabled DNS on the Custom Router, which is the DNS server we’ll be using here.

Although the DNS address should be automatically assigned via DHCP, I’ll hardcode it in the configuration to allow for easy changes in the future.
```
# patches/dns.yaml
machine:
  network:
    nameservers:
      - x.x.x.x
```
NTP: The NTP we'll also be hardcoding it. For this one. I'll be using also the one in our Custom Router.

The NTP server should also be automatically assigned via DHCP, but I’ll also hardcode it in the configuration to allow for easy changes in the future.

  # patches/ntp.yaml
  machine:
    time:
      disabled: false
      servers:
        - x.x.x.x

Disk: I’ll be adding an additional disk - a USB stick - which will be used as the main OS disk for Talos.

If you have questions about this decision, please refer to Chapter 1 of this series.
```
# patches/install-disk.yaml
machine:
  install:
    disk: /dev/nvme0n1
```
Metrics Server: The Metrics Server is a cluster-wide aggregator of resource usage data, such as CPU and memory consumption. It collects metrics from each node and pod, enabling features like autoscaling and resource monitoring through tools like kubectl top. To enable it, we need to add the following configuration:
```
# patches/metrics-server.yaml
machine:
  kubelet:
    extraArgs:
      rotate-server-certificates: true

  files:
    - content: |
        [metrics]
          address = "0.0.0.0:11234"
      path: /var/cri/conf.d/metrics.toml
      op: create

cluster:
  extraManifests:
    - https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/main/deploy/standalone-install.yaml
    - https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
```
By default, the certificates used by the Kubelet aren’t recognized by the Metrics Server. To fix this, we need to enable certificate rotation and ensure the Kubelet uses certificates trusted by the Metrics Server. This is important because the Kubelet is the agent running on each node that manages pods and reports resource usage to the control plane. Secure communication between the Kubelet and the Metrics Server relies on trusted certificates, so enabling certificate rotation helps keep these credentials up-to-date and accepted, ensuring reliable metrics collection and cluster security.
VIP (Virtual IP): Since I’ll be using multiple nodes, I need a simple way to connect to both the services we’re hosting and the cluster itself. To achieve this, we can configure a VIP (Virtual IP) - an IP address shared across the nodes. This allows clients to reach the cluster through a single, stable address regardless of which node is handling the request.
```
# patches/vip.yaml
machine:
  network:
    interfaces:
      - interface: eth0
        vip:
          ip: x.x.x.105
```

Generating secrets

With all these patches defined, the only remaining step is to generate the secrets needed for the control planes to communicate securely with each other, for new nodes to join the cluster, and for clients - like kubectl - to connect.

talosctl includes a utility to generate these secrets. You can create them by running:

talosctl gen secrets --output-file outputs/secrets.yaml

Generating final Nodes Config

Now that we have all the patches and secrets defined, we can generate the final configuration for the nodes. This is done by running the following command:

talosctl gen config clustarino https://x.x.x.105:6443 \
    --with-secrets outputs/secrets.yaml \
    --config-patch @patches/allow-controlplane-workloads.yaml \
    --config-patch @patches/dhcp.yaml \
    --config-patch @patches/disable-kube-proxy-and-cni.yaml \
    --config-patch @patches/install-disk.yaml \
    --config-patch @patches/interface-names.yaml \
    --config-patch @patches/metrics-server.yaml \
    --config-patch @patches/ntp.yaml \
    --config-patch-control-plane @patches/vip.yaml \
    --output rendered/

This command will output the same three files mentioned earlier, but now they will include all of our additional configurations.

Since we need to provide some node-specific configuration as well, we also have to run:

talosctl machineconfig patch \
    --patch @patches/control-plane-node-1.yaml \
    rendered/controlplane.yaml | yq - > nodes/control-plane-node-1.yaml

talosctl machineconfig patch \
    --patch @patches/control-plane-node-2.yaml \
    rendered/controlplane.yaml | yq - > nodes/control-plane-node-2.yaml

talosctl machineconfig patch \
    --patch @patches/control-plane-node-3.yaml \
    rendered/controlplane.yaml | yq - > nodes/control-plane-node-3.yaml

Applying config to the Nodes

With all the configuration generated, we can now apply it to each node in the cluster. But first, we need to identify the IP address of each node. Although we’re using DHCP, we can still assign static IP addresses by configuring DHCP leases.

To do this, navigate to the leases configuration section in OPNSense and set up the following static mappings:

This will make it easy to identify each machine. So now we can apply the
configuration by simply typing:

talosctl apply -f nodes/control-plane-node-1.yaml --node x.x.x.101 --insecure
talosctl apply -f nodes/control-plane-node-2.yaml --node x.x.x.102 --insecure
talosctl apply -f nodes/control-plane-node-3.yaml --node x.x.x.103 --insecure

Bootstrapping the Cluster

With all the configuration in place, it’s finally time to bootstrap the cluster. First, we need to specify the cluster endpoints by running:

talosctl config endpoint x.x.x.101 x.x.x.102 x.x.x.103

And now, the moment we’ve been waiting for - to start the bootstrapping process, run the following command for each node:

talosctl bootstrap --node x.x.x.x

This command needs to be run against one of the control plane nodes. Since all our nodes serve as control plane nodes, you can run it on any of them.

We can monitor the bootstrap progress by running:

talosctl dashboard --node x.x.x.x

This will open a dashboard where you can view real-time logs and track the status of the cluster.

Connecting to the Cluster

Now that our configuration is ready, we can finally apply it. As mentioned earlier, when generating the config, we obtained a file named talosconfig, which provides talosctl with the necessary context to interact with our newly created cluster.

You can place this file in the default Talos config location (~/.talos/config), or alternatively, set the TALOSCONFIG environment variable to point to its path. For example:

# In our case, it will be in the `rendered` folder
export TALOSCONFIG=./rendered/talosconfig

Now, to generate a kubeconfig file for accessing the Kubernetes cluster, run the following command:

talosctl kubeconfig --node x.x.x.x

To validate the connection, we can run:

kubectl get nodes

We should see the following output:

NAME               STATUS      ROLES           AGE     VERSION
clustarino-k8s-1   Not Ready   control-plane   2m14s   v1.30.1
clustarino-k8s-2   Not Ready   control-plane   2m16s   v1.30.1
clustarino-k8s-3   Not Ready   control-plane   2m2s    v1.30.1

The nodes will be in the Not Ready state until we install a CNI (Control Network Interface).

Adding CNI

Since we disabled the default CNI that comes with Talos, we need to install our own. For this, we'll be using Cilium. Cilium is an open-source software that provides transparent, secure networking between application services deployed on container platforms like Docker and Kubernetes.

The main reason I chose Cilium is simply that I wanted to try it out. I’ve been using GKE in my day-to-day work and wanted to explore Cilium’s capabilities in a more controlled environment.

To install Cilium, we can run the following command:

# Add the repository
helm repo add cilium https://helm.cilium.io/
helm repo update

# Install the chart
helm upgrade --install cilium cilium/cilium \
    --namespace kube-system \
    --set ipam.mode=kubernetes \
    --set hostFirewall.enabled=true \
    --set hubble.relay.enabled=true \
    --set hubble.ui.enabled=true \
    --set kubeProxyReplacement=true \
    --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
    --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
    --set cgroup.autoMount.enabled=false \
    --set cgroup.hostRoot=/sys/fs/cgroup \
    --set k8sServiceHost=localhost \
    --set k8sServicePort=7445

I chose to use Helm to install Cilium because, in my opinion, it’s the easiest method and is officially maintained by the Cilium team. Helm also allows us to deploy Hubble, a powerful network observability tool built into Cilium.

The command above installs Cilium with the following configuration options:

ipam.mode=kubernetes: Enables Cilium to use Kubernetes’ IP Address Management (IPAM) for assigning pod IPs.
hostFirewall.enabled=true: Activates the host firewall within Cilium for enhanced security.
hubble.relay.enabled=true: Enables the Hubble relay component.
hubble.ui.enabled=true: Enables the Hubble UI for network observability.
kubeProxyReplacement=true: Replaces the default kube-proxy with Cilium’s implementation.
securityContext.capabilities.ciliumAgent: Sets specific capabilities for the Cilium agent.
securityContext.capabilities.cleanCiliumState: Sets capabilities to clean up Cilium state when needed.
cgroup.autoMount.enabled=false: Disables automatic mounting of cgroups.
cgroup.hostRoot=/sys/fs/cgroup: Specifies the host root directory for cgroups.
k8sServiceHost=localhost: Sets the Kubernetes API server host to localhost.
k8sServicePort=7445: Sets the Kubernetes API server port.

Once the installation completes, you can verify the status of the Cilium pods by running:

kubectl get pods -n kube-system

Network Policies

Now that Cilium is installed, we can add some network policies to control and allow traffic between the nodes. To do this, create a network-policies.yaml file with the following content:

---
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: host-fw-control-plane
spec:
  description: 'control-plane specific access rules.'
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/control-plane: ''
  ingress:
    # Allow access to kube api from anywhere.
    - fromEntities:
        - world
        - cluster
      toPorts:
        - ports:
            - port: '6443'
              protocol: 'TCP'

    # Allow access to talos from anywhere.
    # https://www.talos.dev/v1.10/learn-more/talos-network-connectivity/
    - fromEntities:
        - world
        - cluster
      toPorts:
        - ports:
            - port: '50000'
              protocol: 'TCP'
            - port: '50001'
              protocol: 'TCP'

    # Allow kube-proxy-replacement from kube-apiserver.
    - fromEntities:
        - kube-apiserver
      toPorts:
        - ports:
            - port: '10250'
              protocol: 'TCP'
            - port: '4244'
              protocol: 'TCP'

    # Allow access from hubble-relay to hubble-peer (running on the node).
    - fromEndpoints:
        - matchLabels:
            k8s-app: hubble-relay
      toPorts:
        - ports:
            - port: '4244'
              protocol: 'TCP'

      # Allow metrics-server to scrape.
    - fromEndpoints:
        - matchLabels:
            k8s-app: metrics-server
      toPorts:
        - ports:
            - port: '10250'
              protocol: 'TCP'

    # Allow ICMP Ping from/to anywhere.
    - icmps:
        - fields:
            - type: 8
              family: IPv4
            - type: 128
              family: IPv6

    # Allow cilium tunnel/health checks from other nodes.
    - fromEntities:
        - remote-node
      toPorts:
        - ports:
            - port: '8472'
              protocol: 'UDP'
            - port: '4240'
              protocol: 'TCP'

    # Allow access to etcd and api from other nodes.
    - fromEntities:
        - remote-node
      toPorts:
        - ports:
            - port: '2379'
              protocol: 'TCP'
            - port: '2380'
              protocol: 'TCP'
            - port: '51871'
              protocol: 'UDP'

    # Allow access to etcd and api from unconfigured nodes.
    - fromCIDR:
        - x.x.x.101/32
        - x.x.x.102/32
        - x.x.x.103/32
      toPorts:
        - ports:
            - port: '2379'
              protocol: 'TCP'
            - port: '2380'
              protocol: 'TCP'
            - port: '51871'
              protocol: 'UDP'

    # Allow HTTP and HTTPS access from anywhere.
    - fromEntities:
        - world
        - cluster
      toPorts:
        - ports:
            - port: '80'
              protocol: 'TCP'
            - port: '443'
              protocol: 'TCP'

    # Allow access from inside the cluster to the admission controller.
    - fromEntities:
        - cluster
      toPorts:
        - ports:
            - port: '8443'
              protocol: 'TCP'

This configuration will:

Allow access to the Kubernetes API server from anywhere.
Allow access to Talos OS management ports from anywhere.
Allow kube-apiserver to communicate with kubelet and Cilium agent (kube-proxy replacement).
Allow Hubble relay pods to communicate with Hubble peers running on the nodes.
Allow metrics-server to scrape kubelet metrics for monitoring.
Allow ICMP Echo Request (ping) from/to anywhere for network diagnostics.
Allow Cilium overlay networking (VXLAN/UDP tunnels) and health checks between cluster nodes.
Allow etcd communication and API access between cluster nodes.
Allow etcd and API access from specific unconfigured node IP addresses.
Allow public HTTP (port 80) and HTTPS (port 443) access to services on the nodes.
Allow intra-cluster traffic to access the Kubernetes admission controller on port 8443.

We can apply this configuration by running the following command:

kubectl apply -f network-policies.yaml

This will apply the network policies to the cluster. After applying them, we can check the status of the nodes with:

kubectl get nodes

If everything is set up correctly, you should see the nodes in the Ready state, indicating they are healthy and fully functional within the cluster.

Conclusion

This concludes the setup of the Kubernetes cluster. We have successfully bootstrapped the cluster and installed Cilium as the CNI. With this, the base setup of the Kubernetes cluster is complete.

While the cluster is now up and running, there are still a few components missing that will allow us to expose services outside the cluster. In the next chapter, we will walk through the setup of the Ingress Controller, which will enable external access to the services hosted within the cluster.

Originally published at https://techquests.dev on May 15, 2025.

Home Lab: Chapter 2 — Base Foundations

Andre Nogueira — Fri, 25 Apr 2025 15:52:28 +0000

Howdy!

Everything needs a base to be built on top of. Nothing can be done out of the blue. Take a house for example: it needs a blueprint, then we need to start building the foundation to then build on top of. This is what I'll be tackling in this part: our baseline Infrastructure, our network layer.

Network

This chapter focuses on the actual network setup of my homelab, specifically excluding Kubernetes' internal networks or any network layers created later. This is the network that will enable me to connect to my homelab from my desktop and host the services I desire. My network consists of two main components:

Wifi Network: This is the primary network of my home, provided by my ISP's router, connecting all of my domestic devices. Due to the location of my homelab, I need to extend this network to reach my devices' location.
Custom Router: I’m configuring this router to have four network zones: WAN, LAN, DMZ, and VPN. This setup gives me full control over my network and allows me to manage the traffic between them.

Wifi Network

As I mentioned in the previous chapter, I need to extend the network from my router to the location of my devices, while also augmenting the Wi-Fi coverage throughout the rest of the house.

This was a straightforward process: plug and play. I simply connected the main unit to a power outlet, inserted an Ethernet cable from my home router into it, and placed one of the receivers near my custom router. Then, I downloaded the Devolo application for additional configuration:

Enabled 5G Wi-Fi only, as all my devices can connect to this network, eliminating the need for a 2.4G network.
Set the same SSID (Service Set Identifier, the name of my network) as my home network to extend coverage throughout the house.
Renamed the devices for easier identification:
- Powerline 1 -> PL-Router
- Powerline 2 -> PL-Office

Custom Router

OPNsense was the elected OS (Operating System) for my router. With four ports on my router, I’ll utilize each physical interface for a dedicated network:

Lan port 1: WAN - facilitating my router's internet connection.
Lan port 2: LAN - enabling my desktop's connection to the router.
Lan port 3: DMZ - hosting all my primary home services.
Lan port 4: VPN - housing the VPN server.

This setup achieves physical network separation, each with its own configuration.

The installation is straightforward:

Download ISO of OPNsense from OPNSense Website
Create a bootable image on a thumb drive
Boot machine from the thumb drive
Follow the installation wizard
- Assigned the initial WAN network (PL-Office -> WAN Interface)
- Assigned the initial LAN network (Desktop -> LAN Interface) - this will also serve as the management interface, exposing the OPNsense web dashboard.

Once installed, I accessed the OPNsense web dashboard from my desktop using the machine's IP address - later, a DNS (Domain Name System) record will be created to avoid memorizing all IP addresses. I then created additional interfaces by assigning LAN ports 3 and 4 to DMZ and VPN, respectively.

WAN

WAN (Wide Area Network) typically refers to the interface used for internet access. In the case of my custom router, it serves exactly that role - providing connectivity between the router (and all connected devices) and the internet. The network setup involves a direct connection between just two devices: the router and a powerline adapter. However, since the adapter operates in bridge mode and uses the same IP range as the router, it effectively allows seamless communication between the router and any device on my Wi-Fi network.

LAN

A LAN (Local Area Network) typically describes a network within a home or organization. It is generally private, contrasting the public WAN. Here, the LAN network will connect to a single device initially. Though considered naming this interface Management, reflecting its current purpose (connecting my desktop to the router, later accessing the DMZ), I chose LAN as it is a broader name to accommodate potential future device additions.

DMZ

A DMZ (Demilitarized Zone) network sits between the LAN and WAN, used to host services requiring internet access without exposing the LAN. While my homelab’s DMZ aims to host internet-accessible services, it will also allow access from the LAN, maintaining separation. Hence, I opted to create a DMZ network.

Although I'll maintain LAN access from the DMZ, devices on the Wi-Fi network won’t reach the DMZ

This network will host my primary home services and core homelab infrastructure.

VPN

A VPN (Virtual Private Network) allows secure access to private networks through public internet connections. In my setup, I'm using a Raspberry Pi as the VPN server to enable global access to my homelab, as long as I have internet connectivity - obviously.

Since I'll be using only one device on this network, I only require two IP addresses - one for the Wi-Fi interface and another for the Ethernet interface, which connects to the VPN network. All other devices will access the VPN remotely, beginning with the Wi-Fi interface and then transitioning to the Ethernet interface for VPN network connections.

DNS

To simplify service access, I’ll create a DNS server for name resolution, allowing me to access my services by using the defined names instead of their IP addresses.

I've chosen Unbound as my DNS server because it’s easy to use and allows me to create the DNS records I need. It also works well with OPNsense, making DNS record management simpler. I'll use this DNS as the resolver for all my devices, letting me access my services by their names and resolving external domains too.

The installation is straight forward:

Access the OPNsense web dashboard.
Navigate to System -> Settings -> General.
Find the DNS section and enable the DNS Resolver.
Save changes and add DNS records.

This is what the final configuration looks like:

Connecting Networks

With all networks established, firewall configuration requires adjustment to
allow communication between the ones that I need to cross-connect:

LAN -> DMZ
LAN -> WAN
VPN -> DMZ
VPN -> WAN
DMZ -> LAN
DMZ -> WAN
DMZ -> VPN

Essentially, the goal is to allow LAN connections to access the internet and DMZ for local infrastructure access. VPN will require similar configurations. Bi direcional communication must be enabled for these connections.

This configuration might be too broad but it will be enough for an initial setup. I may revisit it in configuration.

Conclusion

This concludes the foundational infrastructure setup - My Homelab's Network. I covered the various networks established, outlined requirements, and detailed configurations. While broad in scope, it serves as an excellent starting point for future enhancements.

Originally published at https://techquests.dev on April 25, 2025.

Home Lab: Chapter 1 — Requirements, Hardware, Software and Architecture

Andre Nogueira — Sun, 13 Apr 2025 23:27:04 +0000

Howdy!

Welcome to the first quest I'll be tackling! This one might take a while to complete, as it will serve as the foundation for everything that follows - my Home Lab.

I've had the idea of creating a Home Lab for quite some time now - a place to test things, learn new technologies, and essentially just have some fun.

While exploring similar projects, I noticed a common theme: most people build a Home Lab to self-host services they use - NAS servers, media servers, VPNs, etc. While I do plan to host some of those too, my main focus is on building my own applications and services, not just hosting them. I want to recreate what you'd typically find with on a cloud provider or in a software development company - a platform that supports development.

My goal is to create a developer-focused platform where building, testing, deploying, and monitoring applications is the core.

This will be extremely opinionated, based on my experience. I’ll be using tools I’m familiar with, as well as exploring others I want to learn.

Requirements

Let's start with some requirements I've gathered for my setup.

Orchestration

For this project, I want to use Kubernetes as the main orchestrator. I've been using Kubernetes for quite a while now and, while Docker might be a better approach for home use, I want my setup to be fairly close to what you would see in a real production environment. After all, I expect to host some of my own projects, one of them being this blog.

So, if you're reading this, it means that I've already set up my Home Lab, and some of the requirements that we'll be mentioning have already been met.

Many have given up on self-hosting Kubernetes for something simpler. I might follow that same path, as warned by some of my colleagues. But until then, let's continue.

As I plan to use Kubernetes, I'll need at least 3 nodes. After some research, I decided to use 3 mini computers since I wanted to run the nodes on actual hardware instead of virtual machines. Initially, I considered using Raspberry Pis, but many of the tools I plan on using do not support the arm64 architecture. Additionally, most enterprise applications run on amd64, so it made sense to stick with that to keep my setup as close to real-world scenarios as possible.

Network

For the internal network, I wanted to separate it from my home network. That meant I needed a router. Not just to create separation, but also to explore networking configurations and have a physical distinction between home and development environments.

To achieve this setup, I chose a mini computer equipped with multiple
Ethernet ports to function as a router. This device will handle the connection between my home router and all devices within the DMZ (Demilitarized Zone) network. Implementing a DMZ provides an additional layer of security by isolating internal resources from direct external access.

To make future expansion easier, I added a switch to the internal network. This will allow me to connect more nodes or devices to the DMZ network as needed.

Storage

Storage is a crucial requirement, so I added a NAS server. It stores backups for stateful applications (e.g., databases), my personal data, and serves as a self-hosted alternative to services like iCloud, Google Drive, and Dropbox.

Given the above, here is the list of hardware I’ll be using:

1 Mini computer with additional LAN ports (router)
1 Switch (for the DMZ network)
3 Mini computers (K8s nodes)
1 Computer (NAS)

Hardware

With the plan in place, it was time to start doing some research.

Router

As mentioned, I wanted a mini computer with extra ports for the router. I've worked with Cisco routers and pfSense in the past, but for this setup, I wanted to try something new. After some research, I went with OPNsense - an open-source firewall/router software that's a fork of pfSense.

I found a great mini PC on AliExpress:

4 LAN ports
Intel i3 N305 (8 cores / 8 threads)
32GB RAM
1TB NVMe SSD
Passive cooling
Solid build quality
~450€

Powerline

My office is far from the router, and I wanted a wired connection. Drilling through walls and running long cables weren’t appealing, and I didn’t want to use Wi-Fi extenders. That left me with Powerline adapters.

Powerline devices uses the electrical wiring to transmit network signals. I chose the Devolo Magic 2, which extends Wi-Fi coverage and also provides
Ethernet ports. While the speed droped quite a bit, from ~500 Mbps to ~150 Mbps, this is still acceptable for my needs.

Switch

For the switch, I reused a 5-port model I had lying around (TP-Link TL-SF1005D) It’s quiet,compact, and fits my current requirements.

Nodes

I wanted the nodes to be quiet, compact, and performant. I found 3 mini PCs with the following specs:

Ryzen 7 5700U (8 cores / 16 threads)
16GB RAM (upgradable to 32GB)
512GB SSD
Passive cooling
Good build quality
~250€

They came with two M.2 slots, and I wanted to separate the OS from the data. So, I added three Kingston DataTraveler Exodia thumb drives (one per node) to boot the OS while keeping the M.2 SSDs for local node storage.

NAS

Long-term storage has always been something I needed. If you just want reliability, I’d recommend a cloud provider like Google Drive or a prebuilt NAS from Synology or QNAP. But I’m here to learn, so I went DIY.

I found a second-hand custom-built NAS:

5 bays (no drives included)
Intel N5000 (4 cores / 4 threads)
64GB ECC RAM
Passive cooling
Good build quality
400€

Software

With the hardware ready, it was time to look at the software stack.

Wireguard

I wanted to access the DMZ network remotely, just like I would with a cloud provider. I chose WireGuard as the VPN solution. It’s lightweight, fast, and secure.

I'll be hosting it on a Raspberry Pi 4B.

OPNSense

OPNsense is not just a router - it’s a flexible firewall platform with plugin support. I’ll use it alongside Unbound DNS to:

Assign hostnames to devices
Set up a local DNS server
Define static IPs

This setup offers much more flexibility than the router provided by my ISP.

Kubernetes

Kubernetes is the orchestrator of choice. To run it, I'm using Talos - a minimal OS built specifically for Kubernetes. Talos aligns perfectly with my goals: it's secure, immutable, and easy to manage.

Kubernetes alone isn’t enough, so I’ll be adding these tools:

Networking: Cilium for pod-to-pod communication and network policies.
Ingress: NGINX Ingress Controller for external access.
Storage: Ceph for persistent volumes.
Certificates: Cert Manager to automate certificate management.
Metrics & Logs: Talos has built-in metrics support, which I’ll use for monitoring.
CI/CD: ArgoCD for GitOps-based deployments.

This will be the base stack. More tools will be added as the platform evolves.

Homelab Overview

To better understand the components that we'll be using, here is a diagram of the components that we'll be using:

Architecture

Here’s a rough overview of the network setup (read right to left):

ISP connects to the Home Router (standard consumer-grade router).
Powerline Adapter connects the Home Router to the Custom Router in my office.
Custom Router manages the DMZ network, where all critical infrastructure lives.
Raspberry Pi runs the VPN server to allow remote access to the DMZ.

Network Breakdown

Wi-Fi Network: Main home network. Used by everyday devices not part of the DMZ.
WAN Network: Connects the Home Router to the Custom Router. Provides internet access to all custom networks.
DMZ Network: Hosts the Kubernetes nodes, NAS, and future services. Managed by the Custom Router.
VPN Network: Contains the Raspberry Pi and allows external access into the DMZ.
LAN Network: Contains the desktop PC. It didn’t make sense to have it on Wi-Fi, so it gets its own segment.

This is the initial setup. It’s designed to be modular and expandable over time.

Conclusion

This wraps up the first chapter of my Home Lab series. We covered the motivations and requirements behind the project, explored the hardware and software choices, and reviewed the overall architecture of the platform I’m aiming to build.

I'm really excited to start building this, as I feel like this will be a great learning experience - not just in terms of configuration, but also in exploring, architecture, planning, implementation, and of course, documenting and sharing.

I hope you’ll enjoy this journey as much as I will. Stay tuned for the next chapter! And if you have any suggestions, feel free to reach out.

Originally published at https://techquests.dev on April 11, 2025.