Forem: Noah Makau

The Day 2 Reality of Running a Kubernetes Lab on Your Mac: Stop/Start, CKS Scenarios, and What I Learned Building It.

Noah Makau — Fri, 22 May 2026 14:42:11 +0000

Part 7 of 7 — The Mac Kubernetes Lab: A Production-Mirror Setup from Scratch.

Previously in Part 6: We wired up Vault Kubernetes auth, installed Crossplane with the AWS provider, and applied LimitRanges mirroring the production configuration that resulted from a real disk-pressure incident. The EKS mirror is complete. Now we focus on Day 2 — actually using the thing sustainably.

There’s a question I had when I first put this lab together that nobody had answered for me cleanly: Will my cluster survive a stop?

You’re going to want to stop it. The whole point of a local lab is that it gets out of your way when you’re not using it. A four-VM cluster running 24/7 is fine on paper, but you’ve also got Slack, six browser tabs, and an IDE eating memory. Knowing exactly what survives a stop, and what doesn't, and what manual steps you need on the way back up is the difference between "lab I use every day" and "lab I built once and abandoned."

This final article is the runbook I wish I had at the start. Stop/start without losing state, CKS exam scenarios this cluster is purposely built for, and the shell setup that makes the whole thing pleasant to live with.

The full setup, recapped:

Seven articles in, here’s what’s running:

Cluster 1 — `kubectx orbstack`

OrbStack-native K8s, single-node
Istio with *.k8s.orb.local wildcard DNS
Vault in dev mode
Crossplane
Always-on, idles at around 512 MB

Cluster 2 — `kubectx lab-cluster`

kubeadm Kubernetes 1.34, 3 nodes
Vault PKI (3-tier hierarchy, exported intermediate CA)
Istio 1.26 revision-based, MetalLB for LoadBalancer
Crossplane with the AWS provider
Vault Kubernetes auth
Run on-demand

Stop/start without losing state

The biggest question I had when I first set this up was the one I led with — will the cluster survive a stop? The answer is yes, with one exception.

Stopping

# 💻 Mac
orb stop -a      # stop all VMs
orb stop k8s     # stop the native cluster (optional — fine to leave running)

What persists:

All Kubernetes objects — deployments, services, configmaps, secrets, PVCs
etcd data on cp01 (stored on the VM’s disk)
Vault data at /opt/vault/data (file backend, persists on disk)
Calico/Cilium, Istio, Crossplane, MetalLB configurations
The Mac kubeconfig

What is released:

All CPU — drops to zero immediately
All RAM — fully returned to macOS
~8 GB of disk remains used.

Starting back up

# 💻 Mac
orb start -a

Then follow these steps in order:

Step 1 — Unseal Vault. This is the one manual step that can’t be automated away:

# 🖥️ VM: vault
export VAULT_ADDR='http://127.0.0.1:8200'
vault operator unseal $(grep 'Unseal Key 1' ~/vault-init.txt | awk '{print $NF}')
vault status
# Sealed: false ← what you want

Vault seals itself on every shutdown by design. That’s a security feature; an unsealed Vault that survives reboots is, by definition, less secure. In production, you’d use auto-unseal with AWS KMS or Azure Key Vault. For the lab, one manual unseal command is fine.

Step 2 — Verify the cluster:

# 💻 Mac
kubectx lab-cluster
kubectl get nodes   # Ready within ~30 seconds
kubectl get pods -A # everything restarts automatically

Step 3 — Re-export session variables: This is the friction I underestimated when I started. Environment variables don’t persist across SSH sessions anything you exported in a previous session is gone:

# 🖥️ VM: vault (when doing PKI or auth work)
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_ROOT_TOKEN=$(grep 'Initial Root Token' ~/vault-init.txt | awk '{print $NF}')

# 🖥️ VM: cp01 (when doing kubeadm or cert work)
export CP_IP=$(hostname -I | awk '{print $1}')

# 💻 Mac (when doing Istio or MetalLB work)
export VAULT_IP=$(orb run -m vault hostname -I | awk '{print $1}')
export INGRESS_IP=$(kubectl get svc istio-ingress -n istio-system \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Step 4 — Regenerate /tmp files if needed. /tmp on OrbStack VMs clears on reboot:

# 🖥️ VM: vault
vault read -field=certificate pki_k8s/issuer/default > /tmp/lab-ca.crt

Persistence quick reference

Shell setup that makes it pleasant

Put these in your ~/.zshrc:

# 💻 Mac — add to ~/.zshrc
alias klab="kubectx lab-cluster"
alias korb="kubectx orbstack"
alias kns="kubens"
alias k="kubectl"
alias kgp="kubectl get pods -A"
alias kgn="kubectl get nodes -o wide"
alias orbup="orb start -a"
alias orbdown="orb stop -a"

Daily flow becomes short:

orbup               # start everything
ssh vault@orb       # unseal Vault
vault operator unseal $(grep 'Unseal Key 1' ~/vault-init.txt | awk '{print $NF}')
# exit vault VM
klab                # switch to lab cluster
kgn                 # verify nodes
kgp                 # verify pods

CKS exam preparation:

The VM lab cluster is purpose-built for CKS. Real kubeadm cluster, real etcd, real kubelet config files. The exam gives you a similar environment, and having practiced the same scenarios on a cluster where you control every layer makes a meaningful difference.

The scenarios I practice most:

Pod Security Admission:

PSA replaced PodSecurityPolicy in Kubernetes 1.25. The CKS exam tests your ability to enforce pod security standards at the namespace level.

# 💻 Mac
kubectl create namespace restricted-ns

# Enforce the restricted profile - blocks privilege escalation, host networking, etc.
kubectl label namespace restricted-ns \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn=restricted

# Test - this pod should be blocked
kubectl run test --image=nginx -n restricted-ns
# Error from server (Forbidden): pods "test" is forbidden: violates PodSecurity

RBAC

# 💻 Mac
 # Create a role that can only read pods
kubectl create role pod-reader \
  --verb=get,list,watch \
  --resource=pods \
  -n restricted-ns

# Bind it to a service account
kubectl create rolebinding pod-reader-binding \
  --role=pod-reader \
  --serviceaccount=restricted-ns:default \
  -n restricted-ns

# Test with impersonation
kubectl auth can-i list pods \
  --as=system:serviceaccount:restricted-ns:default \
  -n restricted-ns
# yes

kubectl auth can-i delete pods \
  --as=system:serviceaccount:restricted-ns:default \
  -n restricted-ns
# no

Audit policy:

The exam often asks you to configure an audit policy on the control plane. This requires editing the kube-apiserver static pod manifest directly:

# 🖥️ VM: cp01
sudo tee /etc/kubernetes/audit-policy.yaml <<EOF
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
  resources:
  - group: ""
    resources: ["secrets"]
- level: Metadata
  resources:
  - group: ""
    resources: ["pods"]
- level: None
  users: ["system:kube-proxy"]
EOF

Add these flags to /etc/kubernetes/manifests/kube-apiserver.yaml:

- --audit-policy-file=/etc/kubernetes/audit-policy.yaml
- --audit-log-path=/var/log/kubernetes/audit.log
- --audit-log-maxage=30
- --audit-log-maxbackup=10

The kubelet restarts kube-apiserver automatically when the manifest changes.

NetworkPolicy: default-deny

# 💻 Mac
# Deny all ingress and egress by default
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
EOF
# Then selectively allow what's needed
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
EOF

Short-lived admin certificates from Vault PKI

A CKS best practice — use short-lived credentials instead of long-lived kubeconfig files:

# 🖥️ VM: vault
# Issue a 1-hour admin certificate
vault write pki_k8s/issue/admin \
  common_name="kubernetes-admin" ttl=1h

The output gives you a certificate and private key. Build a kubeconfig from them that expires in one hour. Instead of a kubeconfig with a long-lived client cert, you issue a fresh cert each session. When it expires, access is revoked automatically. In production, this is enforced at the Vault role level (max_ttl=2h) you physically cannot issue a cert with a longer TTL.

Trivy image scanning

CKS includes container image security. Trivy is the tool used on the exam:

# 🖥️ VM: cp01 — install trivy
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sudo sh -s -- -b /usr/local/bin

# Scan an image
trivy image nginx:latest

# Scan for HIGH and CRITICAL only
trivy image --severity HIGH,CRITICAL nginx:latest

Five things I learned building this

OrbStack is genuinely better than Multipass, not just faster. The native DNS, instant boot, and real LoadBalancer IPs remove an entire category of friction I had normalised. I didn’t realise how much time I was spending on /etc/hosts edits until I stopped having to do them.
T*he M1 vs M4 CNI difference is a kernel capability issue, not an OrbStack bug.* Once I understood that iptables NAT is restricted in unprivileged LXC containers on M1, Cilium was the obvious fix. Knowing this also makes it easier to debug similar issues in other restricted container environments; CI systems, Docker-in-Docker, anywhere Kubernetes is running inside something it doesn’t quite own.
**Vault PKI is worth the setup cost. **You could let kubeadm generate self-signed certs and skip a whole chapter. But having a lab that uses the same certificate hierarchy as production means the mental model transfers directly. Short-lived admin certs stop being a theoretical best practice and start being how you actually work.
Session variables are the biggest day-to-day friction. Anything that doesn’t persist to .bashrc gets lost between sessions. I've been burned by an empty $CP_IP, causing a "not a valid IP address" error in the kubeadm config more times than I'd like to admit. Persist in what you can.
Document as you go. This whole series came out of runbook notes I was writing for myself while I built the lab. Writing each step down caught several places where the process was more manual than it needed to be, and meant I could replicate the setup on another machine in a day instead of a weekend. If you’ve built something complicated, writing it up is one of the higher-leverage things you can do, even if you never publish it.

The full series

Part 1: Why I Replaced Multipass with OrbStack and what an M1 vs M4 Mac taught me about local Kubernetes.
Part 2: One Command, One Working Kubernetes Cluster! Building My Daily-Driver Lab on OrbStack.
Part 3: Building a Production-Grade Vault PKI for a Local kubeadm Cluster Without the Shortcuts.
Part 4: Same Cluster, Different Mac: A Debugging Story About Unprivileged LXC Containers, iptables, and Why Cilium Replaces kube-proxy.
Part 5: How I Practise Istio Upgrades Locally Before Touching Production EKS.
Part 6: The Disk-Pressure Incident That Taught Me to Always Set LimitRanges and Other Lessons from Mirroring EKS Locally.
Part 7 (this article): The Day 2 Reality of Running a Kubernetes Lab on Your Mac: Stop/Start, CKS Scenarios, and What I Learned Building It.

Resources

OrbStack documentation
OrbStack pricing — free for personal use, paid for commercial/team use
Cilium installation guide
Vault Kubernetes auth documentation
Crossplane documentation
Istio revision-based upgrades
CKS curriculum

← Part 6: The Disk-Pressure Incident That Taught Me to Always Set LimitRanges and Other Lessons from Mirroring EKS Locally.

I’m Noah Makau, a DevSecOps engineer based in Nairobi. I run a small DevOps consultancy and hold CKA, CKAD, and the AWS Solutions Architect Professional certifications, currently preparing for CKS. I write about Kubernetes, Vault, Crossplane, and the day-to-day of running platforms that actually have to stay up.

Originally published at blog.arkilasystems.com

The Disk-Pressure Incident That Taught Me to Always Set LimitRanges and Other Lessons from Mirroring EKS Locally.

Noah Makau — Fri, 22 May 2026 13:26:56 +0000

Part 6 of 7 — The Mac Kubernetes Lab: A Production-Mirror Setup from Scratch.

Previously in Part 5: We installed Istio with revision-based upgrades, MetalLB for LoadBalancer IPs, and practised traffic management with Gateways, VirtualServices, and fault injection. The cluster behaves. Now we wire up the last three pieces that turn it from “a working local cluster” into “a real mirror of our production EKS.”

The cluster works. Istio is running. MetalLB is handing out IPs. But it’s still missing three layers that make the production parity actually meaningful:

Vault Kubernetes auth — pods authenticate to Vault with their service account tokens, the same way they do in production. No hardcoded secrets, no static credentials.
Crossplane with the AWS provider — infrastructure compositions you can develop and test locally before they touch real AWS resources, or any other thought of OpenStack?
LimitRanges — default resource requests on every namespace. This one comes from a real incident I want to talk about.

The LimitRange story is the most important of the three, so I’ll tell it properly when we get there. First, the auth layer.

Vault Kubernetes auth.

Vault’s Kubernetes auth method lets pods authenticate by presenting their service account JWT. Vault validates the token against the Kubernetes API server and exchanges it for a Vault token with the appropriate policies attached.

On the production EKS clusters at work, this is how microservices retrieve database credentials, API keys, and TLS certificates: no hard-coded secrets, no secret sprawl, every issuance audit-logged in Vault.

Setting it up locally means I can test the full injection workflow without a VPN, and debug failures on a cluster where the stakes are zero.

Installing the Vault agent injector.

We deploy just the Vault agent injector in the lab cluster. It points to the external Vault VM rather than running its own Vault server:

# 💻 Mac
kubectx lab-cluster

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

# Get the vault VM IP - does not persist across sessions
export VAULT_IP=$(orb run -m vault hostname -I | awk '{print $1}')
echo "VAULT_IP=$VAULT_IP"
helm install vault hashicorp/vault \
  --namespace vault --create-namespace \
  --set "injector.externalVaultAddr=http://$VAULT_IP:8200"

kubectl get pods -n vault
# vault-agent-injector-xxx   1/1   Running   0   30s

Configuring K8s auth on Vault.

Run this on the vault VM, pointing Vault at the lab cluster’s API server:

# 🖥️ VM: vault

# Re-export - always required, doesn't persist across sessions
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_ROOT_TOKEN=$(grep 'Initial Root Token' ~/vault-init.txt | awk '{print $NF}')

# If Vault is sealed after a reboot:
# vault operator unseal $(grep 'Unseal Key 1' ~/vault-init.txt | awk '{print $NF}')
vault login $VAULT_ROOT_TOKEN

# Get CP_IP from the Mac terminal: orb run -m cp01 hostname -I | awk '{print $1}'
export CP_IP=<cp01-ip>

# Regenerate the CA cert if /tmp was cleared after reboot
vault read -field=certificate pki_k8s/issuer/default > /tmp/lab-ca.crt

# Enable Kubernetes auth (safe to re-run - ignores "already enabled")
vault auth enable -path=lab-k8s kubernetes 2>/dev/null || echo "already enabled"

# Configure - point Vault at the lab cluster API server
vault write auth/lab-k8s/config \
  kubernetes_host="https://$CP_IP:6443" \
  kubernetes_ca_cert=@/tmp/lab-ca.crt

vault read auth/lab-k8s/config

Testing K8s auth.

Create a simple role and test it from a pod:

# 🖥️ VM: vault

# Create a policy
vault policy write read-secrets - <<EOF
path "secret/data/myapp/*" {
  capabilities = ["read"]
}
EOF
# Create a K8s auth role
vault write auth/lab-k8s/role/myapp \
  bound_service_account_names=myapp \
  bound_service_account_namespaces=default \
  policies=read-secrets \
  ttl=1h

# Write a test secret
vault secrets enable -path=secret kv-v2 2>/dev/null || true
vault kv put secret/myapp/config db_password="supersecret"

# 💻 Mac — deploy a pod with Vault annotations
kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp
  namespace: default
---
apiVersion: v1
kind: Pod
metadata:
  name: vault-test
  namespace: default
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "myapp"
    vault.hashicorp.com/agent-inject-secret-config: "secret/data/myapp/config"
spec:
  serviceAccountName: myapp
  containers:
  - name: app
    image: busybox
    command: ["sleep", "3600"]
EOF

# Check the secret was injected
kubectl exec vault-test -c app -- cat /vault/secrets/config
# db_password: supersecret

If that last line returns the password, the whole chain works: service account JWT → Vault validation → Vault token → secret retrieval → file injection. Every link of the chain is what a real production app does.

Crossplane.

Crossplane turns a Kubernetes cluster into a universal control plane for cloud infrastructure. Instead of Terraform modules or CloudFormation stacks, you define infrastructure as Kubernetes custom resources, and Crossplane reconciles them continuously.
I use it at work to provision AWS resources (EKS node groups, RDS, S3 buckets, IAM roles) and VMware Cloud Director resources through a custom provider. The lab version mirrors the AWS side of that.

Installation:

# 💻 Mac
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update

# Composition Functions are enabled by default in recent versions.
# The --enable-composition-functions flag was removed.
helm install crossplane crossplane-stable/crossplane \
  --namespace crossplane-system --create-namespace

kubectl get pods -n crossplane-system -w
# NAME                                       READY   STATUS    AGE
# crossplane-xxx                             1/1     Running   60s
# crossplane-rbac-manager-xxx               1/1     Running   60s

Installing the AWS provider

💻 Mac

kubectl apply -f - <<EOF
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: provider-aws-ec2
spec:
package: xpkg.upbound.io/upbound/provider-aws-ec2:latest
EOF

kubectl get pkg

NAME INSTALLED HEALTHY PACKAGE AGE

provider-aws-ec2 True True xpkg.upbound.io/upbound/provider-... 60s

A minimal composition:

A bare-minimum ProviderConfig enough to verify the install is working:

# 💻 Mac
kubectl apply -f - <<EOF
apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: aws-creds
      key: creds
EOF

In a real setup, you create a IRSA ( IAM Role for Service Account) to authenticate and give the provider permission to create and monitor resources. For local validation, the provider installs, and the compositions can be validated structurally without ever calling AWS.

The LimitRange story.

This is the one that came from a real incident at work.

We had repeated disk-pressure events in our production EKS cluster. Pods with no resource requests had crept into a few namespaces — someone deployed a YAML that omitted resources: entirely, and nobody caught it in review. The Kubernetes scheduler had no signal about their consumption, so nodes ended up overcommitted. Then ephemeral storage filled up, eviction kicked in, and a couple of unrelated pods went down with it. Total downtime measured in tens of minutes. Cause-and-effect chain that took a while to untangle.

The fix is one of the most boring features in Kubernetes: LimitRanges. They set default resource requests and limits at the namespace level. Any container that doesn’t specify its own requests gets the defaults applied automatically by the admission controller. The scheduler always has a signal. Overcommit becomes a deliberate choice, not an accident.

# 💻 Mac
kubectl apply -f - <<EOF
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: default
spec:
  limits:
  - default:
      memory: 512Mi
      cpu: 500m
    defaultRequest:
      memory: 128Mi
      cpu: 100m
    max:
      ephemeral-storage: 2Gi
    type: Container
EOF

Apply this to every namespace that hosts workloads. In production, I now apply it as a post-provisioning step on every new namespace:

# 💻 Mac — apply to multiple namespaces
for ns in default vault crossplane-system istio-system; do
  kubectl apply -f limitrange.yaml -n $ns
done

The `ephemeral-storage` max is the part that specifically addresses the disk-pressure failure mode — it bounds how much scratch space a container can consume, which is what spirals when ephemeral storage runs unbounded.

Verifying the complete EKS mirror.

Let’s confirm the whole stack is up:

# 💻 Mac
kubectl get nodes -o wide
# NAME       STATUS   ROLES           VERSION
# cp01       Ready    control-plane   v1.34.x
# worker01   Ready    <none>          v1.34.x
# worker02   Ready    <none>          v1.34.x

kubectl get pods -A
# Cilium/Calico, CoreDNS, istiod-1-26, MetalLB, Crossplane, Vault injector - all Running

Local vs production — what’s the same and what differs:

The only meaningful differences are the CNI (because of OrbStack’s VM capabilities, as we covered in Part 4) and the LoadBalancer implementation. Everything else is identical in configuration. The mental model from this lab transfers directly to the production cluster, and vice versa.

In the final article: How to stop and start the lab without losing state, the CKS exam scenarios this cluster was purpose-built for, and the shell aliases that make the whole thing pleasant to live with.

← Part 5: How I Practise Istio Upgrades Locally Before Touching Production EKS | Part 7: The Day 2 Reality of Running a Kubernetes Lab on Your Mac: Stop/Start, CKS Scenarios, and What I Learned Building It →

I’m Noah Makau, a DevSecOps engineer based in Nairobi. I run a small DevOps consultancy and hold CKA, CKAD, and the AWS Solutions Architect Professional certifications , currently preparing for CKS. I write about Kubernetes, Vault, Crossplane, and the day-to-day of running platforms that actually have to stay up.
originally published at blog.arkilasystems.com

How I Practise Istio Upgrades Locally Before Touching Production EKS.

Noah Makau — Fri, 22 May 2026 12:44:07 +0000

Part 5 of 7 — The Mac Kubernetes Lab: A Production-Mirror Setup from Scratch.

Previously in Part 4: kubeadm 1.34 is bootstrapped, the M1 vs M4 CNI problem is solved (Calico on M4, Cilium on M1), and the three-node cluster is running with certificates signed by our Vault PKI. Now we install Istio and give the cluster real LoadBalancer IPs.

I’ve done an in-place Istio upgrade in production exactly once. A misconfigured istiod update broke sidecar injection across every namespace simultaneously. That was the last time.

The revision-based approach is the alternative. Run multiple versions of the Istio control plane in parallel. Install the new version alongside the old. Migrate namespaces one at a time. Verify the injection is working at each step. Then remove the old version when you’re confident. If anything misbehaves, you relabel the namespace back to the old revision. Full rollback at any point, no scrambling.
This is how I handle Istio upgrades on the production EKS clusters at work. Building it locally first means the production upgrade procedure is something I’ve already practised on a cluster where the stakes are zero.

Installing Istio via Helm (revision-based)

# 💻 Mac
kubectx lab-cluster

helm repo add istio https://istio-release.storage.googleapis.com/charts
helm repo update

# Step 1 - Base CRDs with default revision set to 1-26
helm install istio-base istio/base \
  --namespace istio-system --create-namespace \
  --set defaultRevision=1-26

# Step 2 - istiod with revision tag
# The revision label (1-26) is what namespaces reference for sidecar injection.
helm install istiod-1-26 istio/istiod \
  --namespace istio-system \
  --set revision=1-26 \
  --set global.proxy.resources.requests.cpu=50m \
  --set global.proxy.resources.requests.memory=128Mi \
  --set pilot.resources.requests.cpu=100m \
  --set pilot.resources.requests.memory=256Mi \
  --wait

# Step 3 - Ingress gateway tied to revision 1-26
helm install istio-ingress istio/gateway \
  --namespace istio-system \
  --set revision=1-26 \
  --set service.type=LoadBalancer

# Label the namespace for injection using the revision label.
# Note: use istio.io/rev=1-26, NOT istio-injection=enabled.
# The revision label tells the webhook which control plane to use.
kubectl label namespace default istio.io/rev=1-26

Verify istiod is running:

# 💻 Mac
kubectl get pods -n istio-system
# NAME                           READY   STATUS    AGE
# istiod-1-26-xxx                1/1     Running   60s
# istio-ingress-xxx              1/1     Running   45s

The MetalLB problem.

On the OrbStack-native cluster from Part 2, LoadBalancer services get real IPs automatically. On the VM cluster, there’s no cloud provider doing that work, services stay in pending state indefinitely.

# 💻 Mac
kubectl get svc istio-ingress -n istio-system
# NAME            TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)
# istio-ingress   LoadBalancer   10.x.x.x     <pending>     80:xxx

MetalLB solves this. It implements the Kubernetes LoadBalancer spec for bare-metal and VM environments. It watches for LoadBalancer services and assigns IPs from a pool you define.

Installing MetalLB

# 💻 Mac
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.5/config/manifests/metallb-native.yaml
kubectl get pods -n metallb-system -w
# Wait for the controller and speaker pods to be Running

Configuring the IP pool.

The IP pool must be in the same subnet as your OrbStack VMs. Get the subnet first:

# 💻 Mac
orb run -m cp01 hostname -I
# M4: 192.168.139.200
# M1 with Cilium: 192.168.139.159 10.0.0.105 fd07:...
#                 ^^^^^ use this — the first 192.168.x.x IP

M1 quirk: With Cilium running, hostname -I returns multiple IPs: the OrbStack internal IP, a Cilium pod-network IP, and an IPv6 address. Always use the first 192.168.x.x IP. The others are internal to Cilium's networking and don't help you here.
Configure the pool using a range that doesn’t overlap with the VM IPs:

# 💻 Mac
kubectl apply -f - <<EOF
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: lab-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.139.200-192.168.139.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: lab-l2
  namespace: metallb-system
EOF

Watch the ingress gateway get its IP:

# 💻 Mac
kubectl get svc istio-ingress -n istio-system -w
# NAME            TYPE           EXTERNAL-IP       PORT(S)
# istio-ingress   LoadBalancer   192.168.139.200   80:xxx,443:xxx

Setting up Mac /etc/hosts.

Unlike the native cluster, the VM cluster doesn’t have wildcard DNS. We add entries to /etc/hosts manually:

# 💻 Mac
export INGRESS_IP=$(kubectl get svc istio-ingress -n istio-system \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $INGRESS_IP

# Remove any stale entry from a previous attempt
sudo sed -i '' '/lab.local/d' /etc/hosts
echo "$INGRESS_IP  httpbin.lab.local bookinfo.lab.local api.lab.local" \
  | sudo tee -a /etc/hosts

Gateway + VirtualService on the VM cluster.

With MetalLB handing out real IPs, we can create Gateways and VirtualServices using *.lab.local hostnames.

Bookinfo the Istio reference app.

Bookinfo is Istio’s reference demo. It’s perfect for testing traffic management because it has multiple services with multiple versions of one of them.

# 💻 Mac
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.26/samples/bookinfo/platform/kube/bookinfo.yaml

kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: bookinfo-gateway
spec:
  selector:
    istio: ingress
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "bookinfo.lab.local"
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: bookinfo
spec:
  hosts:
  - "bookinfo.lab.local"
  gateways:
  - bookinfo-gateway
  http:
  - match:
    - uri:
        prefix: /productpage
    route:
    - destination:
        host: productpage
        port:
          number: 9080
EOF

Open http://bookinfo.lab.local/productpage in your Mac browser.

Traffic splitting.

Now we can practise the same canary deployment patterns used in production:

# 💻 Mac
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: reviews-dr
spec:
  host: reviews
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  - name: v3
    labels:
      version: v3
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews-canary
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 80
    - destination:
        host: reviews
        subset: v2
      weight: 20
EOF

Refresh the bookinfo page a few times. 80% of requests go to reviews-v1 (no stars), 20% to reviews-v2 (black stars).

Header-based routing.

Route specific users to a different version based on a request header:

# 💻 Mac
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews-header
spec:
  hosts:
  - reviews
  http:
  - match:
    - headers:
        end-user:
          exact: user
    route:
    - destination:
        host: reviews
        subset: v3
  - route:
    - destination:
        host: reviews
        subset: v1
EOF

Fault injection.

Introduce artificial latency and errors to test resilience:

# 💻 Mac
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: ratings-fault
spec:
  hosts:
  - ratings
  http:
  - fault:
      delay:
        percentage:
          value: 50.0
        fixedDelay: 3s
      abort:
        percentage:
          value: 10.0
        httpStatus: 500
    route:
    - destination:
        host: ratings
        port:
          number: 9080
EOF

50% of requests to ratings will get a 3-second delay. 10% will return 500. Useful for testing timeout and retry configurations on calling services.

Strict mTLS.

Enforce mutual TLS across the mesh:

# 💻 Mac
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: default
spec:
  mtls:
    mode: STRICT
EOF

Practising an Istio upgrade.

This is where the revision-based setup actually pays off. Let’s simulate upgrading from 1.26 to 1.28:

# 💻 Mac
# Step 1 - Install the new control plane alongside the old
helm install istiod-1-28 istio/istiod \
  --namespace istio-system \
  --set revision=1-28 \
  --wait

# Step 2 - Migrate a namespace to the new revision
kubectl label namespace default istio.io/rev=1-28 --overwrite

# Step 3 - Restart pods to pick up the new sidecar version
kubectl rollout restart deployment -n default

# Step 4 - Verify new sidecars
kubectl get pods -n default -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.sidecar\.istio\.io/status}{"\n"}{end}'

# Step 5 - Once you're satisfied, remove the old control plane
helm uninstall istiod-1-26 -n istio-system

During Steps 1–3, both `istiod-1-26` and `istiod-1-28` are running simultaneously. If anything goes wrong after migrating a namespace, you can roll back by re-labelling the namespace back to `1-26`. This is exactly the procedure I use for production EKS Istio upgrades, and having practised it locally means there's no improvising on the day.

Where we are
The VM cluster now has:
✅ Istio 1.26 installed with revision label (istiod-1-26)
✅ Ingress gateway with a real LoadBalancer IP from MetalLB
✅ /etc/hosts on the Mac routing *.lab.local to the ingress IP
✅ Bookinfo deployed with traffic splitting, header routing, and fault injection
✅ Strict mTLS enforced across the default namespace
✅ Experience practising a revision-based Istio upgrade

In Part 6, we complete the EKS mirror: Vault Kubernetes auth, Crossplane with the AWS provider, and a LimitRange configuration that mirrors a production gap we hit at work, the one that caused a real disk-pressure incident.

← Part 4: Same Cluster, Different Mac: A Debugging Story About Unprivileged LXC Containers, iptables, and Why Cilium Replaces kube-proxy | Part 6: The Disk-Pressure Incident That Taught Me to Always Set LimitRanges — and Other Lessons from Mirroring EKS Locally →

originally published at blog.arkilasystems.com

Same Cluster, Different Mac: A Debugging Story About Unprivileged LXC Containers, iptables, and Why Cilium Replaces kube-proxy.

Noah Makau — Fri, 22 May 2026 12:02:26 +0000

Part 4 of 7 — The Mac Kubernetes Lab: A Production-Mirror Setup from Scratch.

I built a four-VM Kubernetes lab on my M4 Mac. Everything worked the first time. kubeadm init finished cleanly, Calico installed without complaint, and every node went Ready within a minute. The cluster behaved exactly like a small, slightly quieter version of the EKS clusters I run at work.

Months later, on a whim, I tried to put the same setup on my old M1 Pro. The VMs came up. kubeadm init succeeded. I applied the Calico manifests. Nothing worked!

Nodes stayed NotReady indefinitely. kube-proxy crashed in a loop. The tigera-operator pod kept logging the same line:

dial tcp 10.96.0.1:443: connect: connection refused

That’s the Kubernetes API server’s ClusterIP, the address every system component uses to talk to the cluster from inside the cluster. It should always be reachable. It wasn’t.

I spent the evening trying every CNI option I knew, then reading kernel source code I’d never had a reason to look at. What I found was a kernel capability difference between Apple Silicon generations that probably affects more setups than just mine. This is the walkthrough I wish I had when I started debugging.

The setup before things broke:

Quick recap of where we are in the series. We’re standing up Kubernetes 1.34 via kubeadm on four OrbStack VMs:

vault — runs HashiCorp Vault as the cluster's certificate authority
cp01 — the control plane node
worker01 and worker02 — workers Parts 2 and 3 built the supporting pieces: the native OrbStack cluster, the Vault PKI with a Root CA and a Kubernetes Intermediate CA, and the four VMs themselves. By the time we get to kubeadm init, the CA cert and key are already at /etc/kubernetes/pki/ca.crt and /etc/kubernetes/pki/ca.key on cp01, ready for kubeadm to pick up.

What I’m about to walk through happened on both my M4 and my M1. The pre-reqs and the kubeadm init are identical on both. The CNI choice is what diverges.

Pre-reqs on the three Kubernetes nodes.

Run this on cp01, worker01, and worker02. The minimal OrbStack Ubuntu image is missing a few packages, and the order matters,install gpg before adding the Kubernetes apt repo, or you'll end up with a broken, unsigned repo entry that blocks every future apt update.

# 🖥️ VM: all nodes (cp01, worker01, worker02)

# Disable swap
sudo swapoff -a && sudo sed -i '/swap/d' /etc/fstab

# Kernel modules
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay br_netfilter

# Sysctl for bridge networking
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sudo sysctl --system

# containerd
sudo apt update && sudo apt install -y containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd && sudo systemctl enable containerd

# Clean any stale K8s repo entries before adding the new one
sudo rm -f /etc/apt/sources.list.d/kubernetes.list
sudo rm -f /etc/apt/keyrings/kubernetes-apt-keyring.gpg
sudo apt update && sudo apt install -y gpg curl apt-transport-https ca-certificates
sudo mkdir -p /etc/apt/keyrings

# K8s 1.34 repo
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] \
  https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /" \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt update && sudo apt install -y kubelet kubeadm kubectl

sudo apt-mark hold kubelet kubeadm kubectl

kubectl version --client && kubeadm version

Two things worth flagging from experience. First, the Kubernetes apt repo URL pattern is `core:/stable:/v1.34/deb/` - note the colons. Get one wrong, and you'll spend ten minutes wondering why `apt` is returning a 404. Second, if you skip the cleanup step and your previous attempt left a broken repo entry, the `apt update` after adding the new repo will fail silently, and apt install kubelet will install nothing useful.

`kubeadm init` on cp01.

Two things to know about the config:

The API version is v1beta4 in kubeadm 1.34. The old v1beta3 is deprecated and will warn. The KubeletConfiguration group also changed from kubelet.kubeadm.k8s.io to kubelet.config.k8s.io. If you're copying a kubeadm config from an older tutorial, both of those need updating.
The pod CIDR depends on the CNI. Use 10.244.0.0/16 for Cilium (M1) or 192.168.0.0/16 for Calico (M4). These are each CNI's default; getting it wrong here means re-running kubeadm reset later, which is no fun.

# 🖥️ VM: cp01

# Set CP_IP explicitly - VM environment vars don't persist across sessions
export CP_IP=$(hostname -I | awk '{print $1}')
echo "CP_IP=$CP_IP"   # verify - an empty value here causes "not a valid IP address" later

sudo tee /tmp/kubeadm-config.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: "$CP_IP"
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: "1.34.0"
controlPlaneEndpoint: "$CP_IP:6443"
networking:
  podSubnet: "10.244.0.0/16"    # M1 + Cilium: 10.244.0.0/16 | M4 + Calico: 192.168.0.0/16
  serviceSubnet: "10.96.0.0/12"
apiServer:
  certSANs:
  - "$CP_IP"
  - "cp01"
  - "cp01.lab.local"
  - "kubernetes"
  - "127.0.0.1"
  - "10.96.0.1"
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
EOF
grep advertiseAddress /tmp/kubeadm-config.yaml   # sanity check before running init
sudo kubeadm init --config /tmp/kubeadm-config.yaml
mkdir -p $HOME/.kube
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Because the CA cert and key are already at /etc/kubernetes/pki/ from Part 3, kubeadm detects them and uses the Vault-managed CA instead of generating self-signed certs. You'll see:

[certs] Using existing ca certificate authority

This is the moment I always pause and verify. If kubeadm says “Generating ca certificate”, something went wrong with the CA distribution step in Part 3, most likely the file permissions on ca.key, which has to be 0600.

The CNI fork in the road!

This is where the M1 and M4 paths split.

On M4 Calico, the obvious choice.

Calico is the straightforward, well-trodden answer on M4. Full iptables support, BGP routing, NetworkPolicy enforcement. It mirrors what most production EKS clusters use (modulo the AWS VPC CNI specifics).

# 🖥️ VM: cp01 (M4)
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/custom-resources.yaml

# Verify:
kubectl get pods -n calico-system -w
kubectl get nodes -w

Nodes flip to Ready within 60–90 seconds of the Calico pods running. This is the path I expected to work everywhere.

On M1, Calico fails, and figuring out why takes a while!

Here’s what I saw on the M1 when I ran the exact Calico installation:

tigera-operator pod logs:
  dial tcp 10.96.0.1:443: connect: connection refused

kube-proxy pod events:
  open /proc/sys/net/netfilter/nf_conntrack_max: permission denied
  iptables: No chain/target/match by that name.

node status:
  cp01       NotReady    control-plane
  worker01   NotReady    <none>
  worker02   NotReady    <none>

The kube-proxy line is the one that turned out to matter. Permission denied writing to /proc/sys/net/netfilter/nf_conntrack_max. That's a kernel-level write. Something about the container environment was preventing kube-proxy from touching netfilter state.

After enough digging through OrbStack’s docs and the LXC documentation, the pieces fell into place. OrbStack VMs are unprivileged LXC containers.

On M4, the LXC container’s capability set allows the netfilter and iptables NAT modifications that kube-proxy needs.

On M1, M2, and M3, that capability set is more restricted — kube-proxy can’t write the iptables KUBE-SERVICES chain.

The downstream failure chain:

kubeadm init completes. The API server is running and reachable via the node IP.
kube-proxy starts. It tries to write iptables NAT rules so that 10.96.0.1:443 (the Kubernetes API ClusterIP) is DNAT'd to the API server's real address.
The write fails. The KUBE-SERVICES chain never gets populated.
10.96.0.1 is unreachable from inside the cluster.
Every CNI plugin that needs to call the Kubernetes API via the ClusterIP fails to initialise. Calico’s tigera-operator is one of them.
Nodes never go Ready.

So Calico isn’t the problem. kube-proxy is the problem. And kube-proxy is the problem because the container runtime isn’t giving it the capabilities it needs.

This is where Cilium comes in.

Why Cilium fixes this!

Cilium is a CNI that uses eBPF programs loaded directly into the kernel instead of iptables. The eBPF programs implement service routing. What kube-proxy normally does with iptables, Cilium does with eBPF maps, and they don’t need the iptables NAT capability to function.

Cilium also has a kubeProxyReplacement mode that completely replaces kube-proxy. No kube-proxy, no failing iptables writes, no broken ClusterIP routing.

On a Mac where iptables NAT works fine, this is just a different way to do the same thing. On an M1, it’s the difference between a working cluster and a broken one.

Installing Cilium on M1.

# 🖥️ VM: cp01 (M1)

# Step 1 - Remove kube-proxy entirely. Cilium will replace it.
kubectl -n kube-system delete daemonset kube-proxy 2>/dev/null || true
kubectl -n kube-system delete configmap kube-proxy 2>/dev/null || true

# Step 2 - Install Cilium CLI
curl -L --fail --remote-name-all \
  https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-arm64.tar.gz
sudo tar xzvfC cilium-linux-arm64.tar.gz /usr/local/bin
rm cilium-linux-arm64.tar.gz

# Step 3 - Install Cilium with kubeProxyReplacement enabled
export CP_IP=$(hostname -I | awk '{print $1}')
cilium install \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=$CP_IP \
  --set k8sServicePort=6443

# Step 4 - Wait for Cilium to be fully up
cilium status --wait

Two flags are non-negotiable here. kubeProxyReplacement=true is what tells Cilium to handle service routing itself via eBPF. k8sServiceHost=$CP_IP and k8sServicePort=6443 give Cilium the real address of the API server so it doesn't try to reach it via the broken ClusterIP. Without those, Cilium itself can't bootstrap.

The verification step that matters:

# 🖥️ VM: cp01
curl -k https://10.96.0.1:443/healthz --connect-timeout 5
# Expected: ok

If that curl returns ok, the eBPF service routing is working and the cluster is functional. CoreDNS will come up. System pods will go Ready. The cluster is real.

# 🖥️ VM: cp01
kubectl get pods -A -w

Joining the workers

# 🖥️ VM: cp01 — generate the join command
kubeadm token create --print-join-command

Run on worker01 then worker02:

# 🖥️ VM: worker01 and worker02 (run on each)
sudo kubeadm reset -f
sudo rm -rf /etc/cni/net.d /var/lib/cni

# Paste the join command from cp01
sudo kubeadm join <cp01-ip>:6443 --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --cri-socket unix:///run/containerd/containerd.sock

# 🖥️ VM: cp01 — watch nodes come up
kubectl get nodes -w
# NAME       STATUS   ROLES           VERSION
# cp01       Ready    control-plane   v1.34.x
# worker01   Ready    <none>          v1.34.x
# worker02   Ready    <none>          v1.34.x

Getting the kubeconfig onto your Mac.

The detail that catches everyone out: copy from ~/.kube/config on cp01, not /etc/kubernetes/admin.conf. The admin.conf requires sudo, and orb run injects extra output that corrupts the YAML.

Also, every fresh kubeadm init rotates the certificates. You have to re-copy the kubeconfig after every init - if you skip this step, you'll get "the server has asked for the client to provide credentials" and spend twenty minutes confused about why your perfectly good cluster is rejecting you.

# 💻 Mac
# Copy from the user kubeconfig - NOT /etc/kubernetes/admin.conf
ssh cp01@orb "cat ~/.kube/config" > /tmp/lab-kubeconfig.yaml
head -5 /tmp/lab-kubeconfig.yaml   # must start with: apiVersion: v1

# Remove ALL stale entries - including the user object
kubectl config delete-context lab-cluster 2>/dev/null || true
kubectl config delete-cluster kubernetes 2>/dev/null || true
kubectl config delete-user kubernetes-admin 2>/dev/null || true

# Rename and merge
kubectl --kubeconfig /tmp/lab-kubeconfig.yaml \
  config rename-context kubernetes-admin@kubernetes lab-cluster
KUBECONFIG=~/.kube/config:/tmp/lab-kubeconfig.yaml \
  kubectl config view --flatten > /tmp/merged-kube.yaml
cp /tmp/merged-kube.yaml ~/.kube/config

# Update the server IP to the cp01 VM IP
export CP_IP=$(orb run -m cp01 hostname -I | awk '{print $1}')
kubectl config set-cluster kubernetes --server=https://$CP_IP:6443
kubectx lab-cluster
kubectl get nodes

Verifying the Vault CA is actually in use.
This is the part that confirms the whole certificate hierarchy we built in Part 3 is working end-to-end. The cluster certs should be signed by the Vault Intermediate CA, not kubeadm’s default self-signed CA:

# 🖥️ VM: cp01
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -issuer
# issuer=CN=Lab K8s Intermediate CA

openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -subject
# subject=CN=Lab K8s Intermediate CA

If those don’t match what’s in Vault, something went wrong in the CA distribution, and you’re probably running with a self-signed kubeadm CA, which works, but it’s not what we set out to build.

The summary I wish I had:

The general lesson, which is worth more than the specific OrbStack/Apple Silicon story: anywhere you’re running Kubernetes inside an unprivileged Linux container — OrbStack, LXC on a server, certain Docker-in-Docker setups, some CI environments — iptables NAT may not work, and a kube-proxy-replacing CNI like Cilium is the move. I went into this thinking I had an Apple Silicon problem. What I had was a Linux container capability problem that happens to show up most loudly on Apple Silicon today.

In Part 5, we install Istio with revision-based upgrades and MetalLB to give this cluster real LoadBalancer IPs, the same upgrade pattern I use on the production EKS clusters this lab is mirroring.

← Part 3: Building a Production-Grade Vault PKI for a Local kubeadm Cluster Without the Shortcuts | Part 5: How I Practise Istio Upgrades Locally Before Touching Production EKS →

originally published at blog.arkilasystems.com

Building a Production-Grade Vault PKI for a Local kubeadm Cluster Without the Shortcuts.

Noah Makau — Fri, 22 May 2026 10:29:24 +0000

Part 3 of 7 — The Mac Kubernetes Lab: A Production-Mirror Setup from Scratch.

Previously in Part 2: Cluster 1- the OrbStack-native daily driver is up with Istio, Vault dev mode, and Crossplane. Now we start the harder half: a four-VM kubeadm cluster that mirrors a real production EKS setup.

Most kubeadm tutorials skip the part that matters most for production parity. They let kubeadm generate self-signed certificates and call it done. The cluster works, but the certificate hierarchy looks nothing like what you actually run in production, where leaf certs come from an intermediate CA that’s signed by a root CA, all managed by HashiCorp Vault.

This article walks through the boring-but-important version. Four OrbStack VMs, real networking, and a 3-tier Vault PKI that will sign every certificate in the Kubernetes cluster we boot up in Part 4. By the end of it, the mental model in your lab matches the mental model in production — short-lived admin certs, role-based issuance, full audit trail.

Why a separate VM cluster at all?

The OrbStack-native cluster from Part 2 is excellent for daily iteration, but it has a hard limitation: it’s single-node. You can’t test multi-node behaviours; node affinity, pod disruption budgets, rolling upgrades across nodes, or the kind of disk-pressure incidents that happen when one node misbehaves.

Cluster 2 is built for that. Four VMs, a genuine kubeadm-bootstrapped cluster, and a certificate authority managed by Vault, the same PKI approach used in many enterprise Kubernetes deployments. It’s also the cluster I use for CKS exam preparation, because the exam gives you something very similar.

Creating the VMs.

OrbStack VMs are Ubuntu Noble (24.04) on ARM64. They boot in under three seconds and share memory with the host - they only consume what they actually need.

# 💻 Mac
orb create ubuntu:noble vault
orb create ubuntu:noble cp01
orb create ubuntu:noble worker01
orb create ubuntu:noble worker02

# Verify
orb list

Getting into any VM is short:

# 💻 Mac
ssh vault@orb
ssh cp01@orb
ssh worker01@orb
ssh worker02@orb

No key management. No IP lookup. OrbStack handles SSH automatically — which alone is worth the switch for me, since I’ve spent too much of my life copy-pasting SSH commands around.

Record the IPs:

# 💻 Mac
for vm in vault cp01 worker01 worker02; do
  echo "$vm: $(orb run -m $vm hostname -I | awk '{print $1}')"
done

Output looks like:

vault: 192.168.139.100
cp01: 192.168.139.101
worker01: 192.168.139.102
worker02: 192.168.139.103

Set these on your Mac and persist them on each VM:

# 💻 Mac
export VAULT_IP=192.168.139.100
export CP_IP=192.168.139.101
export W1_IP=192.168.139.102
export W2_IP=192.168.139.103

# 🖥️ VM: vault / cp01 / worker01 / worker02 (run on each)
cat >> ~/.bashrc <<EOF
export VAULT_IP=<vault-ip>
export CP_IP=<cp01-ip>
export W1_IP=<worker01-ip>
export W2_IP=<worker02-ip>
EOF
source ~/.bashrc

Why persist to .bashrc? OrbStack VM shell sessions don't share environment variables. Every time you SSH into a VM, you start with a clean environment. Persisting the IPs to .bashrc means they're always available without re-exporting. This is the single biggest day-to-day friction in the entire lab — I'll come back to it in Part 7.

/etc/hosts on all VMs:

Each VM needs to resolve the others by hostname:

# 🖥️ VM: vault / cp01 / worker01 / worker02 (run on each)
sudo tee -a /etc/hosts <<EOF
$VAULT_IP    vault vault.lab.local
$CP_IP       cp01 cp01.lab.local kubernetes
$W1_IP       worker01 worker01.lab.local
$W2_IP       worker02 worker02.lab.local
EOF

Why Vault PKI, not kubeadm’s defaults?

This is the part most local-cluster tutorials skip. They use whatever self-signed certificates kubeadm generates automatically. That works, but it has nothing in common with how certificates are managed in a real enterprise cluster.

In production, I use Vault PKI because it gives me:

A proper certificate hierarchy — Root CA → Intermediate CA → leaf certificates.
Short-lived credentials — admin certificates with one- or two-hour TTLs instead of long-lived kubeconfig credentials.
Auditability — every certificate issuance is logged in Vault.
Role-based issuance — separate roles for kube-apiserver, kubelet, etcd, and admin, each with its own constraints. Setting it up locally means the mental model from the lab matches production. It also means when something goes wrong with cert rotation in production, you’ve already debugged it once on your laptop, where the stakes are zero.

Installing Vault on the vault VM.

The minimal OrbStack Ubuntu image doesn’t include gpg by default. This trips people up. Adding the HashiCorp repo before installing gpg leaves a broken, unsigned repo entry that blocks every future apt update call. Always install gpg first and clean up any stale entries from a previous attempt.

# 🖥️ VM: vault

# Remove any stale repo entry from a previous attempt
sudo rm -f /etc/apt/sources.list.d/hashicorp.list

# Install prerequisites first
sudo apt update && sudo apt install -y gpg curl apt-transport-https ca-certificates jq

# Import HashiCorp GPG key
curl -fsSL https://apt.releases.hashicorp.com/gpg | \
  sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

# Add repo with signed-by reference
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
  https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
  | sudo tee /etc/apt/sources.list.d/hashicorp.list

sudo apt update && sudo apt install -y vault

Configure and start Vault:

# 🖥️ VM: vault

sudo mkdir -p /opt/vault/data

sudo chown -R vault:vault /opt/vault/data

sudo tee /etc/vault.d/vault.hcl <<EOF
storage "file" {
  path = "/opt/vault/data"
}
listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = true
}
ui       = true
api_addr = "http://vault.lab.local:8200"
EOF

sudo chown vault:vault /etc/vault.d/vault.hcl

sudo systemctl enable vault && sudo systemctl start vault

export VAULT_ADDR='http://127.0.0.1:8200'

vault status

Initialize and unseal:

# 🖥️ VM: vault

# Save to home dir - writing to /root gives "permission denied"
vault operator init -key-shares=1 -key-threshold=1 > ~/vault-init.txt

export VAULT_UNSEAL_KEY=$(grep 'Unseal Key 1' ~/vault-init.txt | awk '{print $NF}')
export VAULT_ROOT_TOKEN=$(grep 'Initial Root Token' ~/vault-init.txt | awk '{print $NF}')

vault operator unseal $VAULT_UNSEAL_KEY
vault login $VAULT_ROOT_TOKEN

# Persist so these survive session restarts
echo "export VAULT_ADDR='http://127.0.0.1:8200'" >> ~/.bashrc
echo "export VAULT_TOKEN=$(grep 'Initial Root Token' ~/vault-init.txt | awk '{print $NF}')" >> ~/.bashrc

source ~/.bashrc

Save ~/vault-init.txt carefully. With a single key share, this file contains everything needed to unseal and access Vault. In production, use multiple key shares and distribute them. For a lab, one share is fine.

Building the 3-tier PKI:

Root CA:

The Root CA signs nothing directly except the Intermediate CA. In a real production setup, the root key would be kept offline. Here we keep it in the Vault, but never use it for anything else after the initial intermediate signing.

# 🖥️ VM: vault

vault secrets enable -path=pki pki
vault secrets tune -max-lease-ttl=87600h pki   # 10 years

vault write -field=certificate pki/root/generate/internal \
  common_name="Lab Root CA" ttl=87600h > /tmp/root-ca.crt

vault write pki/config/urls \
  issuing_certificates="http://vault.lab.local:8200/v1/pki/ca" \
  crl_distribution_points="http://vault.lab.local:8200/v1/pki/crl"

K8s Intermediate CA: the part most guides get wrong:

Here is where most guides go wrong. The default intermediate CA type in Vault is internal; the private key is generated inside Vault and never leaves. That's great for security, but kubeadm needs the CA private key on disk at /etc/kubernetes/pki/ca.key to sign cluster certificates.
You must use the exported type. The private key is returned only once, at generation time, so save the full JSON response before extracting anything from it. If you lose this output, you can't recover the key from Vault later. I learned this the hard way.

# 🖥️ VM: vault

vault secrets enable -path=pki_k8s pki
vault secrets tune -max-lease-ttl=43800h pki_k8s   # 5 years

# CRITICAL: use 'exported' not 'internal'
# Save the FULL JSON - private key is only returned at generation time
vault write -format=json pki_k8s/intermediate/generate/exported \
  common_name="Lab K8s Intermediate CA" \
  | tee /tmp/intermediate-full.json \
  | jq -r '.data.csr' > /tmp/k8s-intermediate.csr

# Extract and save the private key immediately
jq -r '.data.private_key' /tmp/intermediate-full.json > /tmp/ca.key

# Verify - must show -----BEGIN RSA PRIVATE KEY-----
cat /tmp/ca.key

Sign the intermediate with the Root CA and import it back:

# 🖥️ VM: vault

vault write -format=json pki/root/sign-intermediate \
  csr=@/tmp/k8s-intermediate.csr format=pem_bundle ttl=43800h \
  | jq -r '.data.certificate' > /tmp/k8s-intermediate-signed.pem

vault write pki_k8s/intermediate/set-signed \
  certificate=@/tmp/k8s-intermediate-signed.pem

# Configure AIA URLs - resolves the "authority information access" warning
vault write pki_k8s/config/urls \
  issuing_certificates="http://vault.lab.local:8200/v1/pki_k8s/ca" \
  crl_distribution_points="http://vault.lab.local:8200/v1/pki_k8s/crl"

Roles:

Roles define what certificates each component of the cluster can request:

# 🖥️ VM: vault

# API server - SANs for all DNS names and IPs the apiserver answers on
vault write pki_k8s/roles/kube-apiserver \
  allowed_domains="kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster.local,cp01,cp01.lab.local" \
  allow_bare_domains=true allow_subdomains=true allow_ip_sans=true max_ttl=8760h

# Kubelet - nodes in the system:nodes group
vault write pki_k8s/roles/kubelet \
  allowed_domains="system:node" allow_bare_domains=true \
  organization="system:nodes" max_ttl=8760h

# Admin - short-lived, 2 hour maximum (CKS best practice).
# This forces regular credential rotation and prevents long-lived kubeconfig files.
vault write pki_k8s/roles/admin \
  allowed_domains="kubernetes-admin" allow_bare_domains=true \
  organization="system:masters" server_flag=false client_flag=true max_ttl=2h

# etcd
vault write pki_k8s/roles/etcd \
  allow_any_name=true max_ttl=8760h

# Policy for cluster bootstrap
vault policy write k8s-pki - <<EOF
path "pki_k8s/issue/*" { capabilities = ["create","update"] }
path "pki_k8s/sign/*"  { capabilities = ["create","update"] }
path "pki_k8s/ca"      { capabilities = ["read"] }
path "pki_k8s/ca/pem"  { capabilities = ["read"] }
path "pki_k8s/crl"     { capabilities = ["read"] }
EOF

Distributing the CA cert and key to cp01

kubeadm looks for /etc/kubernetes/pki/ca.crt and /etc/kubernetes/pki/ca.key before init. If both files are present, it uses them as the cluster CA instead of generating new self-signed certificates, which is exactly what we want.

# 🖥️ VM: vault — issue an apiserver cert to extract the CA cert
export CP_IP=<cp01-ip>   # set explicitly — does not persist across sessions
echo "CP_IP=$CP_IP"

vault write -format=json pki_k8s/issue/kube-apiserver \
  common_name="kubernetes" \
  alt_names="kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster.local,cp01" \
  ip_sans="$CP_IP,127.0.0.1,10.96.0.1" ttl=8760h > /tmp/apiserver-cert.json
jq -r '.data.issuing_ca' /tmp/apiserver-cert.json > /tmp/ca.crt

# 💻 Mac — pull both files and push them to cp01
orb run -m vault cat /tmp/ca.crt > /tmp/lab-ca.crt
orb run -m vault cat /tmp/ca.key > /tmp/lab-ca.key

orb run -m cp01 sudo mkdir -p /etc/kubernetes/pki
cat /tmp/lab-ca.crt | orb run -m cp01 sudo tee /etc/kubernetes/pki/ca.crt
cat /tmp/lab-ca.key | orb run -m cp01 sudo tee /etc/kubernetes/pki/ca.key

# kubeadm requires strict permissions on ca.key
orb run -m cp01 sudo chmod 600 /etc/kubernetes/pki/ca.key
orb run -m cp01 sudo ls -la /etc/kubernetes/pki/
# -rw-r--r-- ca.crt
# -rw------- ca.key

Why orb run -m vault cat instead of scp or orb shell? When you need file content on the Mac, orb run pipes stdout directly. orb shell opens an interactive session and SCP requires setting up keys. For simple file reads, orb run -m cat > local-file is the cleanest approach.

A note on /tmp

/tmp on OrbStack VMs does not persist across reboots. The CA cert and key you just extracted will be gone the next time you restart the vault VM. That's fine — you can always regenerate them from Vault:

# 🖥️ VM: vault — regenerate CA cert after reboot
vault read -field=certificate pki_k8s/issuer/default > /tmp/lab-ca.crt

The private key (/tmp/ca.key) was saved from the exported generation step. If you need it again, re-export from ~/intermediate-full.json, which is in your home directory and does persist:

# 🖥️ VM: vault
jq -r '.data.private_key' ~/intermediate-full.json > /tmp/ca.key

Where we are

At this point:

✅ Four OrbStack VMs created and networked
✅ /etc/hosts configured on all VMs for hostname resolution
✅ Vault installed, initialised, and unsealed on the vault VM
✅ 3-tier PKI: Root CA → K8s Intermediate CA (exported type)
✅ Roles for kube-apiserver, kubelet, etcd, and short-lived admin
✅ ca.crt and ca.key placed on cp01 ready for kubeadm In Part 4, we run kubeadm init and hit the most interesting problem in this entire setup — why Calico works on M4 but quietly fails on M1, and how Cilium's eBPF dataplane solves it.

← Part 2: One Command, One Working Kubernetes Cluster! Building My Daily-Driver Lab on OrbStack | Part 4: Same Cluster, Different Mac: A Debugging Story About Unprivileged LXC Containers, iptables, and Why Cilium Replaces kube-proxy →

originally published at blog.arkilasystems.com

One Command, One Working Kubernetes Cluster! Building My Daily-Driver Lab on OrbStack.

Noah Makau — Fri, 22 May 2026 09:41:26 +0000

Part 2 of 7 — The Mac Kubernetes Lab: A Production-Mirror Setup from Scratch.

Previously in Part 1: I walked through why I replaced Multipass with OrbStack, the dual-cluster architecture I settled on, and a preview of the M1 vs M4 CNI problem that’s coming in Part 4.

The cluster I am going to set up in this article is the one I spend most of my working day inside. It’s a single-node Kubernetes cluster, always on, idles at around 512 MB of memory, has real LoadBalancer IPs and wildcard DNS out of the box. No MetalLB, /etc/hosts editing,kubectl port-forward muscle memory. By the time this article is done, it will also have Istio, Vault, and Crossplane running. Total elapsed time, the first time you do it: about ten minutes.

If you’ve ever built a local Kubernetes cluster and then spent the next twenty minutes wiring up MetalLB and editing /etc/hosts so you can actually reach a service from a browser, this is going to feel almost suspicious.

# 💻 Mac
orb start k8s

# Confirm cluster is udr
kubectl get nodes
# NAME       STATUS   ROLES                  AGE   VERSION
# orbstack   Ready    control-plane,master   30s   v1.33.x

kubectl config current-context
# orbstack

That’s all! No kubeadm, no CNI configuration, no certificate management. The cluster is up and reachable in under thirty seconds the first time, and instantly on every subsequent start.

What makes OrbStack’s networking special?

This is where OrbStack genuinely earns its keep. On a typical local cluster; kind, minikube, kubeadm, LoadBalancer services stay in pending state until you install MetalLB on OrbStack:

LoadBalancer services get a real, reachable IP automatically.
*.k8s.orb.local wildcard DNS resolves from your Mac browser, with no /etc/hosts entry.
cluster.local DNS also resolves from your Mac.
Every service type is reachable without kubectl port-forward.

⚠️ The wildcard DNS only resolves on your Mac. Other devices on your network won’t see *.k8s.orb.local. If you need a service reachable from another machine, that's what Cluster 2 (Parts 3–6) is for.

Installing Istio via Helm.

I use Helm rather than istioctl for two reasons.
First, it's how I manage Istio on the production EKS clusters at work, so the muscle memory transfers.
Second, Helm gives fine-grained control over resource requests, which matters on a laptop.

# 💻 Mac
kubectx orbstack

# Add helm charts
helm repo add istio https://istio-release.storage.googleapis.com/charts
helm repo update

# Step 1 - Base CRDs
helm install istio-base istio/base \
  --namespace istio-system --create-namespace \
  --set defaultRevision=default

# Step 2 - Control plane
# The PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION flag is required on OrbStack
# to prevent DNS resolution conflicts with the host network.
helm install istiod istio/istiod \
  --namespace istio-system \
  --set pilot.env.PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION=true \
  --set global.proxy.resources.requests.cpu=10m \
  --set global.proxy.resources.requests.memory=64Mi \
  --wait

# Step 3 - Ingress gateway
# OrbStack assigns a real LoadBalancer IP automatically - no MetalLB needed.
helm install istio-ingress istio/gateway \
  --namespace istio-ingress --create-namespace \
  --set service.type=LoadBalancer

# Verify the gateway got an EXTERNAL-IP
kubectl get svc -n istio-ingress
# NAME            TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)
# istio-ingress   LoadBalancer   10.x.x.x     198.19.x.x    80:xxx/TCP

# Enable sidecar injection on the default namespace
kubectl label namespace default istio-injection=enabled

Gateways and Virtual Services on the native cluster.

This is the moment OrbStack feels like cheating. You create a Gateway pointing at *.k8s.orb.local, and it just works from your Mac browser. No IP lookups. No /etc/hosts, no 127.0.0.1:8080 proxying.

How the traffic actually flows.

The Gateway resource binds to the istio-ingress LoadBalancer service, OrbStack intercepts traffic to the *.k8s.orb.localwildcard domain at the Mac level and routes it to that LoadBalancer IP. The Virtual Service then routes inside the cluster to the right service.

Example 1: Basic routing with httpbin:

# 💻 Mac
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      containers:
      - name: httpbin
        image: kennethreitz/httpbin:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
spec:
  selector:
    app: httpbin
  ports:
  - port: 80
    targetPort: 80
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: httpbin-gateway
spec:
  selector:
    istio: ingress
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "httpbin.k8s.orb.local"
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
  - "httpbin.k8s.orb.local"
  gateways:
  - httpbin-gateway
  http:
  - route:
    - destination:
        host: httpbin
        port:
          number: 80
EOF

Open http://httpbin.k8s.orb.local in your Mac browser. It works immediately.

Example 2: Traffic splitting (canary):

The same pattern used in production canary deployments:

# 💻 Mac
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp-split
spec:
  hosts:
  - "myapp.k8s.orb.local"
  gateways:
  - myapp-gateway
  http:
  - route:
    - destination:
        host: myapp-v1
        port:
          number: 80
      weight: 80
    - destination:
        host: myapp-v2
        port:
          number: 80
      weight: 20
EOF

Example 3: Path-based routing:

Route /v1 and /v2 to different backend services:

# 💻 Mac
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: path-routing
spec:
  hosts:
  - "api.k8s.orb.local"
  gateways:
  - api-gateway
  http:
  - match:
    - uri:
        prefix: /v1
    route:
    - destination:
        host: api-v1-svc
        port:
          number: 80
  - match:
    - uri:
        prefix: /v2
    route:
    - destination:
        host: api-v2-svc
        port:
          number: 80
EOF

One OrbStack gotcha: don’t enable httpsRedirect!

Do not use httpsRedirect: true in a Gateway on the native cluster. OrbStack intercepts LoadBalancer traffic in a way that causes an infinite 301 redirect loop when TLS redirect is enabled.

# ❌ This breaks on OrbStack native K8s
servers:
- tls:
    httpsRedirect: true

# ✅ Use plain HTTP on the native cluster
servers:
- port:
    number: 80
    protocol: HTTP

For TLS testing, use Cluster 2 (the VM cluster, coming in Part 3), where you have full control over the network stack.

Installing the rest of the daily stack:

Vault in dev mode.

Dev mode trades durability for speed. No unsealing, no persistence concerns, instant startup. It’s the right call for the daily-driver cluster where I’m testing AppRole workflows, PKI policies, or Kubernetes auth configurations, and the value is in iteration speed, not data preservation.

# 💻 Mac
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

helm install vault hashicorp/vault \
  --namespace vault --create-namespace \
  --set "server.dev.enabled=true"

kubectl get pods -n vault
# vault-0   1/1   Running   0   30s

For production-grade Vault with HA Raft storage, use Cluster 2 (Part 3). Dev mode here is intentional — it trades durability for speed.

Crossplane:

Crossplane turns the Kubernetes cluster into a universal control plane for cloud infrastructure. I use it heavily with the AWS provider at work.

# 💻 Mac
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update

# Note: --enable-composition-functions was removed in newer versions.
# Composition Functions are enabled by default now.
helm install crossplane crossplane-stable/crossplane \
  --namespace crossplane-system --create-namespace

kubectl get pods -n crossplane-system

Stop/start without losing your work.

# 💻 Mac

# Stop - releases all CPU and RAM
orb stop k8s

# Start - full state persists (deployments, services, secrets, configmaps)
orb start k8s

# Verify
kubectx orbstack
kubectl get nodes
kubectl get pods -A

The native cluster state persists across stop/start. Vault dev mode data is lost on restart by design, but everything else, Crossplane installations, Istio configuration, and your workload deployments come back exactly as provisioned.

Quick reference for this cluster:

What’s next.

The daily-driver cluster is the easy half of this lab. The hard half is the one that mirrors production, a real multi-node kubeadm cluster running on four VMs, with HashiCorp Vault as its certificate authority.

That’s where Part 3 picks up: creating the VMs, wiring their networking, and standing up a 3-tier Vault PKI that will sign every certificate in the cluster.

← Part 1: Why I Replaced Multipass with OrbStack.. | Part 3: Building a Production-Grade Vault PKI for a Local kubeadm Cluster Without the Shortcuts →

I’m Noah Makau, a DevSecOps engineer based in Nairobi. I run a small DevOps consultancy and hold CKA, CKAD, and AWS Solutions Architect Professional certifications, currently preparing for CKS. I write about Kubernetes, Vault, Crossplane, and the day-to-day of running platforms that actually have to stay up.

Originally published at blog.arkilasystems.com.

Why I Replaced Multipass with OrbStack and what an M1 vs M4 Mac taught me about local Kubernetes.

Noah Makau — Fri, 15 May 2026 21:00:00 +0000

Part 1 of 7 — "The Mac Kubernetes Lab: A Production-Mirror Setup from Scratch"

Here’s something nobody warned me about running Kubernetes on Apple Silicon: the same lab setup will work on one chip and fail silently on another. I learned this twice.

The first time was when my Multipass-based cluster broke after I moved from an M1 Pro to an M4. Everything I’d put together over months — four VMs, a kubeadm cluster, the lot — stopped working in a way the error messages couldn’t quite explain. The QEMU drivers Multipass relies on weren’t compatible with the M4’s hardware, and there was no quick fix. The setup was gone.

The second time was months later. I’d long since replaced Multipass with OrbStack on the M4, and it had been working beautifully. Out of curiosity, I tried to put the same OrbStack cluster on my old M1. The VMs came up fine. kubeadm finished. Calico installed. And then everything sat in NotReady for an entire evening while I read kernel docs to figure out why iptables NAT was silently misbehaving inside an unprivileged LXC container on one chip but not the other.

What I ended up with is a Kubernetes lab that runs on either chip, mirrors a real EKS production cluster, and taught me more about Apple’s Virtualization Framework and Linux container capabilities than I ever wanted to know.

This is part one of a seven-part series walking through that setup from scratch. By the end, you’ll have a dual-cluster Kubernetes environment on your Mac — one fast daily-driver cluster for iteration, one full VM-based cluster that mirrors production EKS, complete with Vault PKI, Istio, and Crossplane. We’ll start by replacing Multipass.

The Multipass setup that worked, then didn’t.

For most of 202,4 my local Kubernetes setup was on Multipass. Four Ubuntu VMs on my M1 Pro, a kubeadm cluster, and a working developer loop. It wasn’t fast: Multipass VMs took 30 to 60 seconds to boot, and pre-allocated memory, whether you used it or not. It worked, and it was familiar. Then I upgraded to an M4, and it stopped.

Multipass on macOS uses QEMU under the hood. The QEMU driver bundled with the version of Multipass I had didn’t play nicely with the M4’s silicon. VMs failed to launch, with errors that didn’t map cleanly to actionable issues. I spent a weekend on the Multipass GitHub issues, looking for someone with the same problem and a fix. That weekend is when I tried OrbStack instead, and I never went back.

What OrbStack is, and why the switch wasn’t just about boot time.

OrbStack is a macOS-native tool for running Linux VMs and Docker containers. The thing that matters for our purposes is that it’s built specifically for Apple Silicon, using Apple’s Virtualization Framework, written in Swift, Go, Rust, and C — not a port of something originally designed for x86.

The numbers that made me commit:

The boot time alone is a different category of experience. Spinning up a four-VM cluster used to be a “go make coffee” event. With OrbStack, the whole cluster is up in under 15 seconds.

But the part I underestimated was the networking. OrbStack shiwith a built-in Kubernetes cluster: one command and you have a working cluster with real LoadBalancer IPs and wildcard DNS out of the box. No MetalLB, and/etc/hosts editing. You apply a Service of type LoadBalancer, and you can hit it from your Mac browser at something.k8s.orb.local. The first time it just worked, I genuinely thought I had misconfigured it.

💡 Pricing note: OrbStack is free for personal use, which covers everything in this series. If you’re rolling it out across a team or using it commercially at work, check orbstack.dev/pricing. The grey area worth knowing: a home lab on your own Mac is free. A work laptop doing your day job is commercial use.

The Architecture: Two Clusters, One Tool

After a few weeks with OrbStack, I settled on a dual-cluster setup. Two clusters, two purposes.

Cluster 1 — OrbStack Native K8s (Daily Driver)

The built-in cluster handles fast iteration work:

Crossplane compositions and provider development
HashiCorp Vault AppRole workflows
Helm chart testing
Istio Gateway and VirtualService experimentation — though I break this constantly, which is fine

Switch to it with kubectx orbstack. Services are reachable at *.k8s.orb.local from your browser immediately.

Cluster 2 — VM kubeadm Cluster (EKS Mirror + CKS Lab)

Four OrbStack Linux VMs running a real kubeadm-bootstrapped cluster:

K8s 1.34 — matching our upcoming EKS upgrade target at Arkila Systems
Vault PKI as the cluster Certificate Authority
Istio with revision-based upgrades, identical to our EKS approach
Crossplane with AWS provider
Multi-node topology (control plane + 2 workers) mirroring production

This is also my CKS exam preparation environment — Pod Security Admission, audit policies, NetworkPolicy, short-lived admin certificates via Vault.

Apple Silicon Compatibility — M1 vs M4

This is the part I didn’t see coming until I tried it.

OrbStack works flawlessly on M4. I built the entire dual-cluster setup, including the kubeadm VM cluster, and every component came up on the first try. Months later, on a whim, I tried to replicate the same setup on my M1 Pro. The VMs booted, kubeadm initialised, and I installed Calico.

Nothing worked!!

Nodes stayed NotReady. kube-proxy crashed in a loop. The tigera-operator pod logged dial tcp 10.96.0.1:443: connect: connection refused. The Calico pods looked healthy from the outside, but were completely non-functional from the inside.

The root cause turned out to be a kernel capability difference:

	M4 Mac	M1 / M2 / M3 Mac
OrbStack VM type	Unprivileged LXC	Unprivileged LXC
iptables NAT	✅ Works	❌ Restricted
Recommended CNI	Calico	Cilium (eBPF)
kube-proxy	Standard	Replaced by Cilium

OrbStack VMs run as unprivileged LXC containers on both chips. On M4, the iptables NAT table is fully writable from inside the container. On M1, M2, and M3, it isn’t! Kube-proxy can’t write the KUBE-SERVICES chain, which means ClusterIP services aren't reachable, which means any CNI plugin that tries to call the Kubernetes API server through the ClusterIP 10.96.0.1 fails to start. Calico is one of those plugins. So is half the ecosystem!

The fix is Cilium, which uses eBPF-based service routing instead of iptables and completely replaces kube-proxy. I’ll cover the full debugging walk-through and the Cilium install in Part 4, because it’s the most technically interesting thing in this series — and probably useful to anyone running unprivileged Linux containers in restricted environments, not just OrbStack on Apple Silicon.

What You'll Need

Apple Silicon Mac (M1, M2, M3, or M4)
Homebrew installed
At least 16 GB RAM — 8 GB will technically work but you'll feel it when all four VMs are running
About 20 GB free disk space

# 💻 Mac — check your chip
system_profiler SPHardwareDataType | grep Chip

# Check available disk
df -h ~

If you’re on an Intel Mac, this series isn’t for you — none of it depends on x86, but the whole point is the Apple Silicon experience, and I haven’t tested any of it on Intel.

The Multipass → OrbStack migration, if you’re coming from Multipass.

The command mapping in short:

Multipass	OrbStack
`multipass launch ubuntu`	`orb create ubuntu`
`multipass shell <name>`	`ssh <name>@orb`
`multipass exec <name> -- cmd`	`orb run -m <name> cmd`
`multipass list`	`orb list`
`multipass delete <name>`	`orb delete <name>`
`multipass stop <name>`	`orb stop <name>`
`multipass stop --all`	`orb stop -a`

Install OrbStack and clean out Multipass:

# 💻 Mac
brew install orbstack
open -a OrbStack   # required once for first-time GUI setup

# Remove Multipass
brew uninstall multipass
sudo rm -rf /var/root/Library/Application\ Support/multipassd
sudo rm -rf ~/Library/Application\ Support/multipass

OrbStack auto-installs orb, docker, and kubectl on your PATH. If you already have any of those installed via Homebrew, you may want to check which version comes first in your shell.

What's Coming in This Series

Part 1 (this article): Why I Replaced Multipass with OrbStack and what an M1 vs M4 Mac taught me about local Kubernetes.
Part 2: Cluster 1 — Native K8s daily driver with Istio, Vault, Crossplane
Part 3: Cluster 2 — VM creation, networking, and Vault PKI bootstrap
Part 4: kubeadm 1.34 — M1 vs M4 CNI deep dive (Calico vs Cilium)
Part 5: Istio revision-based upgrades and MetalLB on the VM cluster
Part 6: Vault K8s auth and Crossplane — mirroring your EKS stack
Part 7: Day 2 operations, CKS lab scenarios, and making it all stick

Part 2: One Command, One Working Kubernetes Cluster! Building My Daily-Driver Lab on OrbStack →

Originally publisher at blog.arkilasystems.com

Forem: Noah Makau

The Day 2 Reality of Running a Kubernetes Lab on Your Mac: Stop/Start, CKS Scenarios, and What I Learned Building It.

This final article is the runbook I wish I had at the start. Stop/start without losing state, CKS exam scenarios this cluster is purposely built for, and the shell setup that makes the whole thing pleasant to live with.

The full setup, recapped:

Cluster 1 — kubectx orbstack

Cluster 2 — kubectx lab-cluster

Stop/start without losing state

Stopping

What persists:

What is released:

Starting back up

Persistence quick reference

Shell setup that makes it pleasant

CKS exam preparation:

The scenarios I practice most:

Pod Security Admission:

RBAC

Audit policy:

NetworkPolicy: default-deny

Short-lived admin certificates from Vault PKI

Trivy image scanning

Five things I learned building this

The full series

Resources

← Part 6: The Disk-Pressure Incident That Taught Me to Always Set LimitRanges and Other Lessons from Mirroring EKS Locally.

The Disk-Pressure Incident That Taught Me to Always Set LimitRanges and Other Lessons from Mirroring EKS Locally.

Vault Kubernetes auth.

Installing the Vault agent injector.

Configuring K8s auth on Vault.

Testing K8s auth.

Crossplane.

Installation:

Installing the AWS provider

💻 Mac

NAME INSTALLED HEALTHY PACKAGE AGE

provider-aws-ec2 True True xpkg.upbound.io/upbound/provider-... 60s

A minimal composition:

In a real setup, you create a IRSA ( IAM Role for Service Account) to authenticate and give the provider permission to create and monitor resources. For local validation, the provider installs, and the compositions can be validated structurally without ever calling AWS.

The LimitRange story.

The ephemeral-storage max is the part that specifically addresses the disk-pressure failure mode — it bounds how much scratch space a container can consume, which is what spirals when ephemeral storage runs unbounded.

Verifying the complete EKS mirror.

Local vs production — what’s the same and what differs:

How I Practise Istio Upgrades Locally Before Touching Production EKS.

Installing Istio via Helm (revision-based)

The MetalLB problem.

Installing MetalLB

Configuring the IP pool.

Setting up Mac /etc/hosts.

Gateway + VirtualService on the VM cluster.

Bookinfo the Istio reference app.

Traffic splitting.

Header-based routing.

Fault injection.

Strict mTLS.

Practising an Istio upgrade.

← Part 4: Same Cluster, Different Mac: A Debugging Story About Unprivileged LXC Containers, iptables, and Why Cilium Replaces kube-proxy | Part 6: The Disk-Pressure Incident That Taught Me to Always Set LimitRanges — and Other Lessons from Mirroring EKS Locally →

Same Cluster, Different Mac: A Debugging Story About Unprivileged LXC Containers, iptables, and Why Cilium Replaces kube-proxy.

The setup before things broke:

Pre-reqs on the three Kubernetes nodes.

kubeadm init on cp01.

The CNI fork in the road!

On M4 Calico, the obvious choice.

On M1, Calico fails, and figuring out why takes a while!

Why Cilium fixes this!

Installing Cilium on M1.

Joining the workers

Getting the kubeconfig onto your Mac.

The summary I wish I had:

← Part 3: Building a Production-Grade Vault PKI for a Local kubeadm Cluster Without the Shortcuts | Part 5: How I Practise Istio Upgrades Locally Before Touching Production EKS →

Building a Production-Grade Vault PKI for a Local kubeadm Cluster Without the Shortcuts.

Why a separate VM cluster at all?

Creating the VMs.

Record the IPs:

/etc/hosts on all VMs:

Why Vault PKI, not kubeadm’s defaults?

Installing Vault on the vault VM.

Configure and start Vault:

Initialize and unseal:

Building the 3-tier PKI:

Root CA:

Cluster 1 — `kubectx orbstack`

Cluster 2 — `kubectx lab-cluster`

The `ephemeral-storage` max is the part that specifically addresses the disk-pressure failure mode — it bounds how much scratch space a container can consume, which is what spirals when ephemeral storage runs unbounded.

`kubeadm init` on cp01.