Forem: Pendela BhargavaSai

The Definitive Guide to Lightweight Kubernetes: KIND, Minikube, MicroK8s, K3s, Vcluster, k0s, and RKE2 Compared

Pendela BhargavaSai — Thu, 23 Apr 2026 03:18:00 +0000

TL;DR — There is no single "best" lightweight Kubernetes. KIND wins CI/CD, Minikube wins local dev UX, MicroK8s wins on Ubuntu, K3s wins edge and production, Vcluster wins multi-tenancy, k0s wins zero-dependency ops, and RKE2 wins enterprise compliance. This post explains why — with architecture diagrams, feature tables, and real-world guidance.

Why Lightweight Kubernetes Matters
The Contenders at a Glance
KIND — Kubernetes IN Docker
Minikube — The Developer's Workhorse
MicroK8s — Zero-Ops by Canonical
K3s — Production-Grade at the Edge
Vcluster — Kubernetes Inside Kubernetes
k0s — Zero Dependencies, Zero Friction
RKE2 — Security-First Enterprise K8s
Scoring Across 8 Dimensions
Use Case Decision Guide
Final Verdict

Why Lightweight Kubernetes Matters

Full-fat Kubernetes — the kind you run on a 3-master, 6-worker production cluster — is extraordinary infrastructure. It is also deeply impractical when you need to:

Spin up a throwaway cluster in a GitHub Actions runner in under 30 seconds
Run Kubernetes on a Raspberry Pi with 1 GB of RAM
Give every developer on your team their own isolated cluster without buying new hardware
Deploy to a factory floor where the "server" is an ARM SBC with no internet access

The Kubernetes ecosystem responded by producing a rich family of lightweight distributions, each making different trade-offs. By 2025, the major players are KIND, Minikube, MicroK8s, K3s, Vcluster, k0s, and RKE2 — and choosing between them is genuinely consequential.

This guide gives you the full picture: architecture, components, features, limitations, scoring, and concrete use-case guidance, all in one place.

The Contenders at a Glance

Tool	Creator	Year	Primary Use Case	Min RAM	Binary Size
KIND	Kubernetes SIG Testing	2019	CI/CD testing	2 GB	N/A (uses Docker)
Minikube	Kubernetes Community	2016	Local development	2 GB	~100 MB
MicroK8s	Canonical (Ubuntu)	2018	Ubuntu / Edge	540 MB	Snap package
K3s	Rancher Labs (SUSE)	2019	Edge / Production	512 MB	< 100 MB
Vcluster	Loft Labs	2021	Multi-tenancy	Host-dependent	Helm chart
k0s	Mirantis	2020	Zero-dependency ops	1 GB	~230 MB
RKE2	Rancher (SUSE)	2021	Enterprise / Compliance	4 GB	~300 MB

Each of these is CNCF-compatible and capable of running real Kubernetes workloads. The differences are in where, how, and at what cost they do it.

1. KIND — Kubernetes IN Docker

What It Is

KIND (Kubernetes IN Docker) was built by the Kubernetes SIG Testing team for one purpose: to test Kubernetes itself. Every node in a KIND cluster is a Docker container. The control plane runs in one container, worker nodes in others, and they communicate over a Docker bridge network called kindnet.

KIND runs every Kubernetes node as a Docker container. There is no VM, no hypervisor, no separate OS. The kindnet CNI is a purpose-built bridge that understands this container-as-node topology. The practical effect is that KIND clusters are disposable, fast, and completely ephemeral — perfect for testing but incapable of persistence.

Because there is no VM involved, KIND clusters start in about 30 seconds and use only Docker's existing networking and storage. You can run a dozen isolated clusters on a single laptop.

Architecture


┌─────────────────────────────────────────────────────---┐
│                   Docker Host                          │
│                                                        │
│  ┌─────────────────────┐   ┌──────────────────────┐    │
│  │   Control Plane     │   │     Worker 1         │    │
│  │   (container)       │──▶│     (container)      │   │
│  │                     │   │                      │    │
│  │  • API Server       │   │  • kubelet           │    │
│  │  • etcd             │   │  • kube-proxy        │    │
│  │  • Scheduler        │──▶│  • Pod A  • Pod B    │    │
│  │  • Controller Mgr   │   └──────────────────────┘    │
│  │  • kindnet CNI      │     ┌──────────────────────┐  │
│  └─────────────────────┘     │     Worker 2         │  │
│                              │     (container)      │  │
│  ┌──────────────────┐        │  • kubelet + pods    │  │
│  │  Port-forwarding │        └──────────────────────┘  │
│  │  localhost:6443  │                                  │
│  └──────────────────┘                                  │
└─────────────────────────────────────────────────────---┘

Core Components

kindnet — Custom CNI using a kernel bridge, purpose-built for KIND's container-as-node model
etcd — Full etcd running inside the control-plane container
containerd — Container runtime inside each node-container (Docker-in-Docker)
kubeadm — KIND uses kubeadm internally to bootstrap the cluster

Key Features

True multi-node clusters (control plane + N workers) on a single host
Custom node images — test against any Kubernetes version
Rootless mode via rootless Docker/Podman
IPv6 and dual-stack support
Create multiple isolated clusters simultaneously
Parallel cluster creation
KUBECONFIG auto-export
Optimised for GitHub Actions, GitLab CI, and Jenkins

Quick Start


# Install

curl  -Lo  ./kind  <https://kind.sigs.k8s.io/dl/v0.22.0/kind-linux-amd64>

chmod  +x  ./kind && sudo  mv  ./kind  /usr/local/bin/kind

# Create a single-node cluster

kind  create  cluster

# Create a multi-node cluster

cat <<EOF | kind  create  cluster  --config=-

kind: Cluster

apiVersion: kind.x-k8s.io/v1alpha4

nodes:

- role: control-plane

- role: worker

- role: worker

EOF

# Delete cluster

kind  delete  cluster

Pros

Blazing fast — 30-second cluster creation, no hypervisor boot time
Zero VM overhead — runs entirely inside Docker containers
True multi-node topology on one host
Exact Kubernetes version control via node images
Perfect for ephemeral CI environments
No LoadBalancer hacks needed for testing (use NodePort)
Widely supported in CI platforms

Cons

Requires Docker or Podman to be running
Not production-ready under any circumstances
No GPU passthrough
LoadBalancer type services need MetalLB or similar
Volumes are lost when the cluster is deleted
No addon ecosystem

Best For

CI/CD pipelines — specifically integration testing that needs a real multi-node Kubernetes topology without the boot time of a VM-based solution.

2. Minikube - The Developer's Workhorse

What It Is

Minikube is the original local Kubernetes project, released in 2016 and still the most feature-rich local development option. It runs a Kubernetes cluster inside a VM, a container, or directly on the host, and brings an unmatched addon ecosystem of 30+ pre-packaged integrations.

Minikube is the only distribution that abstracts over drivers — it runs identically whether the underlying host is a VM (VirtualBox, HyperKit, KVM), a container (Docker, Podman), or bare metal. This flexibility comes at the cost of startup time and memory, but it means Minikube works for every developer on every operating system.

If you've ever run kubectl apply -f on your laptop, you've probably used Minikube.

Architecture

┌──────────────────────────────────────────────────────────-┐
│             VM / Docker / Podman Driver                   │
│                                                           │
│  ┌─────────────────────────────────────────────────────┐  │
│  │            Single Node (All-in-One)                 │  │
│  │                                                     │  │
│  │  Control Plane                  Data Plane          │  │
│  │  ┌──────────┐  ┌────────────┐   ┌─────────────────┐ │  │
│  │  │API Server│  │etcd        │   │kubelet          │ │  │
│  │  └──────────┘  └────────────┘   │kube-proxy       │ │  │
│  │  ┌──────────┐  ┌────────────┐   │Pod A • Pod B    │ │  │
│  │  │Scheduler │  │Ctrl Manager│   └─────────────────┘ │  │
│  │  └──────────┘  └────────────┘                       │  │
│  └─────────────────────────────────────────────────────┘  │
│                                                           │
│  ┌─────────────────────────────────────────────────────┐  │
│  │                   Addons Layer                      │  │
│  │  Dashboard │ Ingress │ Metrics │ Registry │ Istio   │  │
│  └─────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────-┘

Core Components

Multiple drivers — HyperKit, VirtualBox, KVM2, Docker, Podman, SSH
etcd — Full etcd as the backing store
Calico or Flannel — CNI (configurable per driver)
Addon controller — Manages the 30+ available addon services

Key Features

30+ addons including Istio, Knative, Linkerd, GPU operator, registry, and more
Built-in Kubernetes dashboard (minikube dashboard)
GPU passthrough in VM mode
LoadBalancer via minikube tunnel
Multiple profile management (run several clusters simultaneously)
Image caching to speed up repeated pulls
minikube service command for easy port access
Built-in image loading (minikube image load)

Quick Start


# Install (Linux)

curl  -LO  <https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64>

sudo  install  minikube-linux-amd64  /usr/local/bin/minikube

# Start with Docker driver

minikube  start  --driver=docker

# Enable addons

minikube  addons  enable  ingress

minikube  addons  enable  metrics-server

minikube  addons  enable  dashboard

# Open dashboard

minikube  dashboard

# LoadBalancer support

minikube  tunnel  # Run in separate terminal

# Delete

minikube  delete

Pros

Easiest getting-started experience of any K8s tool
Unmatched addon ecosystem (30+ addons)
GPU passthrough support (VirtualBox/KVM drivers)
Built-in dashboard requires zero configuration
Works on macOS, Linux, and Windows
Multiple profiles = multiple clusters
Best documentation and community support

Cons

Slow startup in VM mode (~2 minutes)
High memory consumption, especially with VM driver
Primarily a single-node environment
Not production-ready
LoadBalancer requires keeping minikube tunnel running separately
Battery-intensive on laptops
Multi-node support exists but is limited and buggy

Best For

Local development — especially developers who want a full Kubernetes experience with addons, dashboards, and GPU support without deep infrastructure expertise.

3. MicroK8s - Zero-Ops by Canonical

What It Is

MicroK8s is Canonical's packaging of Kubernetes as a snap. It installs as a single command, self-heals via systemd, updates automatically through snap channels, and has the lowest memory footprint of any full-featured Kubernetes distribution at just 540 MB.

MicroK8s is unique in using dqlite — a distributed SQLite engine developed by Canonical — as an alternative to etcd for HA mode. This dramatically simplifies the operational burden of running a multi-master cluster: no external etcd cluster needed, just microk8s add-node on each machine.

Unlike KIND and Minikube, MicroK8s is designed for both development and light production workloads. Its HA mode using dqlite (a distributed version of SQLite) supports clustering without requiring a full etcd setup.

Architecture


┌──────────────────────────────────────────────────────────-┐
│                  Snap Package (systemd)                   │
│                                                           │
│  ┌──────────────────────┐   ┌────────────────────────┐    │
│  │    Node 1 (Master)    │   │      Node 2           │    │
│  │                       │   │                       │    │
│  │  • API Server         │──▶│  • kubelet            │    │
│  │  • dqlite (HA store)  │   │  • kube-proxy         │    │
│  │  • Scheduler          │   │  • Calico CNI          │   │
│  │  • Controller Manager │   │  • Pods                │   │
│  │  • Calico CNI         │   └────────────────────────┘   │
│  │  • Auto-updater       │                                │
│  └──────────────────────┘                                 │
│                                                           │
│  ┌──────────────────────────────────────────────────────┐ │
│  │         Addon Engine (microk8s enable <addon>)       │ │
│  │  Istio │ Knative │ GPU │ Registry │ Dashboard │ More │ │
│  └──────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────-┘

Core Components

dqlite — Distributed SQLite for HA without the operational burden of etcd
Calico CNI — Production-grade networking with network policy support
Snap daemon — Manages the entire lifecycle including automatic updates
Addon engine — microk8s enable <name> installs curated addons

Key Features

Lowest memory footprint: 540 MB minimum
HA clustering via microk8s add-node
Automatic channel-based updates with rollback
GPU operator addon for ML/AI workloads
Strict snap confinement for security
ARM64 and x86 native support
Observability stack addon (Prometheus, Grafana)
Built-in image registry

Quick Start


# Install via snap

sudo  snap  install  microk8s  --classic

# Add your user to the microk8s group

sudo  usermod  -aG  microk8s  $USER

newgrp  microk8s

# Check status

microk8s  status  --wait-ready

# Enable core addons

microk8s  enable  dns  ingress  metrics-server  dashboard

# Use kubectl

microk8s  kubectl  get  nodes

# Add worker node (run on master, then copy join command to worker)

microk8s  add-node

# Uninstall

sudo  snap  remove  microk8s

Pros

Lowest RAM usage of all full-featured distributions (540 MB)
Best Ubuntu and Linux integration through the snap ecosystem
Self-healing via systemd — restarts automatically on failure
HA multi-node with a simple add-node workflow
Automatic updates through snap channels (stable, candidate, beta)
Production-capable for light workloads
ARM64 support for Raspberry Pi and ARM servers

Cons

Snap packaging limits portability to non-Ubuntu systems
Ubuntu-centric design — snap is not available everywhere
Addon conflicts can occur (Istio + other service meshes, for example)
Strict snap confinement can block some host filesystem operations
dqlite is still maturing compared to battle-tested etcd
Automatic updates can cause unplanned restarts without configuration

Best For

Ubuntu workstations and edge servers — if you're on Ubuntu, MicroK8s is the most native Kubernetes experience available.

4. K3s - Production-Grade at the Edge

What It Is

K3s is the single most consequential lightweight Kubernetes project of the past five years. Released by Rancher Labs (now SUSE) in 2019, it packs a complete, CNCF-certified Kubernetes distribution into a single binary under 100 MB. It runs on 512 MB of RAM, boots in 30 seconds, and runs identically on a Raspberry Pi, a factory floor ARM controller, and a cloud VM.

K3s achieves its sub-100 MB size by bundling everything into a single Go binary with no external dependencies, using SQLite as a default backing store (which requires no cluster management), and removing upstream K8s features that aren't needed in its target environments (Windows nodes, cloud-provider integrations, certain alpha features).

K3s is not a toy. It is used in production by thousands of organisations worldwide.

Architecture


┌────────────────────────────────────────────────────────────────┐
│                      k3s binary (< 100 MB)                     │
│                                                                │
│  ┌─────────────────────────────────┐                           │
│  │          k3s Server             │                           │
│  │  (Control Plane + Optional DP)  │──────────┐                │
│  │                                 │          │                │
│  │  • API Server                   │          ▼                │
│  │  • SQLite (default) / etcd / PG │   ┌─────────────────┐     │
│  │  • Scheduler                    │   │   k3s Agent 1   │     │
│  │  • Controller Manager           │   │   (Worker Node) │     │
│  │  • Flannel CNI (built-in)       │   │  • kubelet      │     │
│  │  • Traefik Ingress              │   │  • kube-proxy   │     │
│  │  • CoreDNS                      │──▶│  • Flannel     │     │
│  │  • local-path-provisioner       │   │  • Pods         │     │
│  │  • Helm controller              │   └─────────────────┘     │
│  └─────────────────────────────────┘          │                │
│                                               ▼                │
│                                        ┌─────────────────┐     │
│                                        │ k3s Agent 2     │     │
│                                        │ (ARM / IoT)     │     │
│                                        └─────────────────┘     │
└────────────────────────────────────────────────────────────────┘

Core Components

Single binary — Packages containerd, CNI plugins, CoreDNS, Traefik, and more
SQLite — Default data store, ideal for single-server or small clusters
Embedded etcd — Available for HA clusters (3+ servers)
External DB — PostgreSQL, MySQL, or etcd for larger deployments
Flannel CNI — Built-in overlay networking, zero extra configuration
Traefik — Ingress controller included out of the box
Helm controller — Manage Helm charts via CRDs
local-path-provisioner — Dynamic PVC provisioning on local disk

Key Features

CNCF-certified — passes full Kubernetes conformance tests
Single binary < 100 MB with everything bundled
Multiple storage backends: SQLite, etcd, PostgreSQL, MySQL
ARM64 and ARMv7 first-class support
Air-gap / offline install support (critical for edge deployments)
Auto TLS with Let's Encrypt for Traefik
Server + Agent role split for control/data plane separation
Automatic certificate rotation

Quick Start


# Install server (master) — one command

curl  -sfL  <https://get.k3s.io> | sh  -

# Check status

sudo  systemctl  status  k3s

sudo  kubectl  get  nodes

# Get the node join token

sudo  cat  /var/lib/rancher/k3s/server/node-token

# Join a worker node (run on the worker)

curl  -sfL  <https://get.k3s.io> | \\

K3S_URL=https://<SERVER_IP>:6443  \\

K3S_TOKEN=<NODE_TOKEN> \\

sh  -

# Use kubectl without sudo

mkdir  -p  ~/.kube

sudo  cp  /etc/rancher/k3s/k3s.yaml  ~/.kube/config

sudo  chown $(id  -u):$(id  -g) ~/.kube/config

# Uninstall

/usr/local/bin/k3s-uninstall.sh  # server

/usr/local/bin/k3s-agent-uninstall.sh  # agent

HA Setup (Embedded etcd)


# First server node

curl  -sfL  <https://get.k3s.io> | sh  -s  -  server  \\

--cluster-init

# Additional server nodes

curl  -sfL  <https://get.k3s.io> | sh  -s  -  server  \\

--server https://<FIRST_SERVER_IP>:6443 \\

--token <NODE_TOKEN>

Pros

CNCF-certified — genuine, conformant Kubernetes, not a cut-down imitation
Single binary under 100 MB — deploy to anything
512 MB RAM minimum — runs on Raspberry Pi 3
30-second cold start
SQLite for small clusters, etcd for HA — right tool for every scale
Traefik ingress out of the box — production workloads with zero extra config
ARM64 and ARMv7 native — best IoT Kubernetes support in the market
Air-gap install — works in completely offline environments

Cons

SQLite backend not suitable for clusters exceeding ~50 nodes
Some upstream Kubernetes features are stripped (Alpha features, some cloud integrations)
Default CNI is Flannel only (using Calico requires additional configuration)
No built-in dashboard
Less rich addon ecosystem than Minikube or MicroK8s
Limited Windows node support

Best For

Edge computing, IoT, production on resource-constrained hardware, and any environment where the binary size and startup time of a traditional Kubernetes distribution is prohibitive.

5. Vcluster — Kubernetes Inside Kubernetes

What It Is

Vcluster takes a completely different approach to "lightweight Kubernetes." Rather than running alongside a host operating system, it runs inside an existing Kubernetes cluster. Each virtual cluster is a set of pods in a namespace, but from the user's perspective it is a completely isolated Kubernetes cluster with its own API server, etcd, and full Kubernetes API.

This makes Vcluster the definitive answer to the multi-tenancy problem: instead of giving teams namespace isolation (which shares the API server and exposes blast radius), you give each team their own cluster for the cost of a few pods.

Vcluster is architecturally unique in the field. Its virtual control plane (API server + etcd + scheduler + controller manager) runs as pods inside a host cluster namespace. A component called the Syncer watches the virtual cluster's API and translates virtual resources into real host resources — a virtual Pod becomes a real Pod in the host namespace with a remapped name.

Architecture

┌──────────────────────────────────────────────────────────────┐
│               Host Kubernetes Cluster (any provider)         │
│                                                              │
│  ┌────────────────────┐  ┌────────────────────┐              │
│  │    vcluster 1      │  │    vcluster 2       │             │
│  │  (Team A ns)       │  │  (Team B ns)        │             │
│  │                    │  │                     │             │
│  │  Virtual API Srv   │  │  Virtual API Srv    │             │
│  │  In-process etcd   │  │  In-process etcd    │             │
│  │  Syncer pod        │  │  Syncer pod         │             │
│  │  ┌────┐  ┌────┐    │  │  ┌────┐  ┌────┐    │              │
│  │  │PodA│  │PodB│    │  │  │PodC│  │PodD│    │              │
│  │  └─┬──┘  └─┬──┘    │  │  └─┬──┘  └─┬──┘    │              │
│  │    │sync   │sync   │  │    │sync   │sync    │             │
│  └────┼───────┼───────┘  └────┼───────┼────────┘             │
│       ▼       ▼               ▼       ▼                      │
│  ┌──────────────────────────────────────────────────────┐    │
│  │  Shared Worker Nodes — Host CNI, Storage, Hardware   │    │
│  └──────────────────────────────────────────────────────┘    │
└──────────────────────────────────────────────────────────────┘

The Syncer is the key innovation: it translates virtual cluster resources into real host cluster resources. A Pod created in vcluster 1 becomes a real Pod in the host cluster's namespace, but with a remapped name that prevents conflicts.

Core Components

Virtual API Server — Full Kubernetes API, runs as a pod in the host cluster
In-process etcd — Embedded etcd for the virtual cluster's state
Syncer — Reconciles virtual resources to host cluster resources
vcluster CLI — Manages lifecycle: create, connect, delete, list

Key Features

Full Kubernetes API isolation per virtual cluster
Works on top of any Kubernetes (EKS, GKE, AKS, K3s, RKE2, etc.)
~10 second spin-up time — fastest of all solutions
No extra hardware — uses existing cluster nodes
CRD isolation — each vcluster has its own CRDs
RBAC isolation — separate RBAC per vcluster
Helm chart deployment — deploy via standard Helm
On-demand creation and deletion

Quick Start


# Install vcluster CLI

curl  -L  -o  vcluster  "<https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64>"

chmod  +x  vcluster && sudo  mv  vcluster  /usr/local/bin

# Create a virtual cluster

vcluster  create  my-vcluster  --namespace  team-a

# Connect to it (sets KUBECONFIG automatically)

vcluster  connect  my-vcluster  --namespace  team-a

# Now kubectl talks to the vcluster

kubectl  get  nodes

kubectl  create  deployment  nginx  --image=nginx

# Disconnect

vcluster  disconnect

# Delete

vcluster  delete  my-vcluster  --namespace  team-a

Pros

Full Kubernetes API isolation per tenant — no shared API server blast radius
10-second spin-up — fastest cluster creation of all solutions reviewed
No extra hardware — reuses host cluster's nodes entirely
Works on any cloud or on-premises Kubernetes
Cost-efficient multi-tenancy at scale
Each team gets the full kubectl experience
Easy to create and delete on demand for short-lived environments

Cons

Not standalone — requires a host Kubernetes cluster to exist first
Cannot create real nodes — virtual only
Advanced networking between vclusters is complex
Some cluster-scoped resources (like ClusterRoles and CRDs) are not fully isolated
Requires privileged pod access on the host cluster
Newer project — less battle-tested than K3s or Minikube
Node-level debugging is limited

Best For

Multi-tenant development environments, per-team isolated clusters, and CI/CD environments where many short-lived clusters need to be spun up and torn down rapidly on existing infrastructure.

6. k0s — Zero Dependencies, Zero Friction

What It Is

k0s (pronounced "kay-zeros") from Mirantis lives up to its name: zero host OS dependencies. It is a single binary that includes everything needed to run Kubernetes without requiring any specific kernel modules, swap configuration, or package manager. It works on any Linux distribution out of the box.

k0s uses an eBPF-based CNI called kube-router, includes Autopilot for automated upgrades, and offers FIPS 140-2 compliance — a feature set that appeals strongly to regulated industries.

k0s prioritises deployment universality. By bundling containerd and all CNI plugins into the binary itself and requiring no kernel module configuration from the host OS, it can be dropped onto virtually any Linux system and run. The eBPF-based kube-router CNI offers modern packet processing without iptables overhead.

Architecture


┌──────────────────────────────────────────────────────┐
│             k0s binary (systemd / OpenRC)            │
│                                                      │
│  ┌──────────────────────────┐                        │
│  │     k0s controller       │                        │
│  │   (Control Plane)        │───────────────┐        │
│  │                          │               │        │
│  │  • API Server            │               ▼        │
│  │  • etcd (embedded)       │   ┌─────────────────┐  │
│  │  • Scheduler             │   │  k0s worker 1   │  │
│  │  • Controller Manager    │   │                 │  │
│  │  • containerd            │   │  • kubelet      │  │
│  │  • kube-router (eBPF)    │──▶│  • kube-router  │  │
│  │  • Autopilot updater     │   │  • containerd   │  │
│  └──────────────────────────┘   │  • Pods         │  │
│                                  └─────────────────┘ │
│  k0sctl tool → manages cluster lifecycle             │
└──────────────────────────────────────────────────────┘

Key Features

Truly zero host OS dependencies — no kernel module requirements
FIPS 140-2 compliance mode available
eBPF-based networking via kube-router
Autopilot automated upgrades
k0sctl for full cluster lifecycle management
ARM64 native support
Air-gap install support
Works on any Linux OS (Debian, RHEL, Alpine, CoreOS, etc.)

Quick Start


# Download k0s

curl  -sSLf  <https://get.k0s.sh> | sudo  sh

# Install and start as a service

sudo  k0s  install  controller  --single

sudo  k0s  start

# Get kubeconfig

sudo  k0s  kubeconfig  admin > ~/.kube/config

# Check cluster

kubectl  get  nodes

# Add a worker node — generate join token on controller

sudo  k0s  token  create  --role=worker

# On the worker node

sudo  k0s  install  worker  --token-file  /path/to/token

sudo  k0s  start

Pros

Truly zero host OS dependencies — works on any Linux, no special kernel configuration
FIPS 140-2 compliance for regulated industries
eBPF-based networking with kube-router is modern and efficient
Autopilot handles automated upgrades safely
k0sctl provides a proper cluster lifecycle management tool
No swap or kernel module pre-requirements
Air-gap support

Cons

Smaller community than K3s or MicroK8s
Less rich addon ecosystem
k0sctl adds an additional tool to the workflow
Some CNI plugins need manual configuration beyond kube-router
Enterprise support is a paid product from Mirantis
Fewer third-party integrations and tutorials

Best For

Environments where host OS diversity is a challenge — mixed Linux distributions, heavily locked-down servers, or compliance-driven deployments needing FIPS 140-2.

7. RKE2 — Security-First Enterprise K8s

What It Is

RKE2 (Rancher Kubernetes Engine 2) is the enterprise evolution of K3s. Where K3s optimises for minimal resource usage and edge deployability, RKE2 optimises for security hardening and compliance. It ships hardened by default with CIS Kubernetes Benchmark compliance, FIPS 140-2 support, automatic etcd snapshots, and deep Rancher integration.

RKE2 starts from K3s's architecture and adds a hardening layer: Pod Security Admission enforced by default, etcd encryption at rest, CIS-compliant API server flags, audit logging enabled, and Canal CNI with network policy enforcement. It is Kubernetes made appropriate for government and financial sector requirements.

If K3s is the lightweight sports car, RKE2 is the armoured vehicle. More resource-intensive, harder to damage.

Architecture

┌───────────────────────────────────────────────────────────┐
│              RKE2 Server (Hardened Control Plane)         │
│                                                           │
│  ┌──────────────────────────────────────────────────────┐ │
│  │  CIS-Hardened Kubernetes                             │ │
│  │                                                      │ │
│  │  • Hardened API Server (PSP enforced)                │ │
│  │  • etcd with automated snapshots                     │ │
│  │  • Hardened Scheduler & Controller Manager           │ │
│  │  • Canal / Calico / Cilium CNI (configurable)        │ │
│  │  • containerd runtime                                │ │
│  │  • Cert-manager + auto rotation                      │ │
│  └──────────────────────────────────────────────────────┘ │
│                    │                                      │
│          ┌─────────┴──────────┐                           │
│          ▼                    ▼                           │
│  ┌────────────────┐  ┌────────────────┐                   │
│  │  RKE2 Agent 1  │  │  RKE2 Agent 2  │                   │
│  │  (Worker)      │  │  (Worker)      │                   │
│  └────────────────┘  └────────────────┘                   │
│                                                           │
│  ┌──────────────────────────────────────────────────────┐ │
│  │  Rancher Management Plane (optional)                 │ │
│  └──────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────┘

Key Features

CIS Kubernetes Benchmark v1.6 compliant by default
FIPS 140-2 cryptographic compliance
etcd with automated periodic snapshots and restoration
Multiple CNI options: Canal (default), Calico, Cilium
Automated certificate rotation
Helm chart integration
Air-gap install support
Deep Rancher management platform integration
Role-based node configuration

Quick Start


# Install RKE2 server

curl  -sfL  <https://get.rke2.io> | sh  -

systemctl  enable  rke2-server.service

systemctl  start  rke2-server.service

# Get kubeconfig

export  KUBECONFIG=/etc/rancher/rke2/rke2.yaml

# Get join token for workers

cat  /var/lib/rancher/rke2/server/node-token

# On worker nodes

curl  -sfL  <https://get.rke2.io> | INSTALL_RKE2_TYPE="agent"  sh  -

mkdir  -p  /etc/rancher/rke2/

cat > /etc/rancher/rke2/config.yaml <<EOF

server: https://<SERVER_IP>:9345

token: <NODE_TOKEN>

EOF

systemctl  enable  rke2-agent.service

systemctl  start  rke2-agent.service

Pros

CIS Kubernetes Benchmark compliance out of the box — no manual hardening
FIPS 140-2 for regulated environments (finance, government, healthcare)
Automated etcd snapshots — point-in-time restore capability
Multiple CNI choices (Canal, Calico, Cilium) for varied network requirements
Excellent Rancher multi-cluster management integration
Automated certificate rotation
Strong air-gap support for isolated environments

Cons

4 GB RAM minimum makes it unsuitable for edge/IoT
Longer startup time (~2 minutes)
More operationally complex than K3s
Overkill for non-compliance use cases
Tightly coupled to the Rancher ecosystem
Larger binary and resource footprint
etcd only — no SQLite lightweight option

Best For

Enterprise, compliance-driven, and government workloads where security hardening and audit-readiness are non-negotiable.

Scoring Across 8 Dimensions

Scores are relative (1–10, higher is better for most dimensions):

Dimension	KIND	Minikube	MicroK8s	K3s	Vcluster	k0s	RKE2
Ease of use	7	9	8	7	5	6	4
Production readiness	2	2	7	9	8	8	10
Resource efficiency	7	5	9	10	8	8	3
Multi-node support	9	4	8	9	8	8	9
Addon ecosystem	5	9	8	6	5	4	7
Edge / IoT fit	1	1	6	10	1	7	3
Multi-tenancy	1	1	2	3	10	2	4
CI/CD suitability	10	7	7	7	9	6	5

Use Case Decision Guide

Your Situation	Best Choice	Runner-Up
GitHub Actions / GitLab CI pipelines	KIND	Vcluster
Local development on macOS/Windows/Linux	Minikube	MicroK8s
Developer on Ubuntu workstation	MicroK8s	K3s
Raspberry Pi cluster at home	K3s	MicroK8s
Industrial IoT / factory floor	K3s	k0s
ARM-based edge server	K3s	MicroK8s
Production workload on lightweight infra	K3s	MicroK8s
Government / regulated enterprise	RKE2	k0s
FIPS 140-2 compliance required	RKE2 or k0s	—
Multi-tenant dev environments	Vcluster	Namespace isolation
Per-team isolated clusters	Vcluster	KIND
Mixed Linux OS fleet	k0s	K3s
Air-gap / offline environment	K3s	k0s or RKE2
Testing Kubernetes itself	KIND	—
HA on bare metal with minimal ops	MicroK8s	K3s embedded etcd
Kubernetes with Rancher management	RKE2	K3s

The Decision Tree

Do you need production-grade?
├── No → Is it for CI/CD testing?
│         ├── Yes → KIND
│         └── No  → Are you on Ubuntu?
│                   ├── Yes → MicroK8s
│                   └── No  → Minikube
└── Yes → Do you need compliance (FIPS/CIS)?
          ├── Yes → RKE2 (CIS+FIPS) or k0s (FIPS)
          └── No  → Is it edge/IoT/ARM?
                    ├── Yes → K3s
                    └── No  → Need multi-tenancy?
                              ├── Yes → Vcluster
                              └── No  → K3s or MicroK8s

Final Verdict

After a thorough review, the landscape shakes out clearly:

K3s is the most remarkable project in the lightweight Kubernetes space. It delivers a complete, CNCF-certified Kubernetes distribution in under 100 MB, runs on 512 MB of RAM, and works in air-gapped ARM environments. For the vast majority of production lightweight Kubernetes use cases, K3s is the correct answer.

Vcluster solves a problem no other distribution addresses: genuine Kubernetes API-level multi-tenancy without dedicated hardware. If you need to give 10 teams their own isolated clusters, Vcluster is the only sensible approach.

KIND is indispensable for CI/CD. If you run Kubernetes integration tests in any CI system, KIND's 30-second, Docker-native, multi-node clusters are the right tool with no close competitor.

Minikube remains the best onboarding experience for developers who are new to Kubernetes. The addon ecosystem and built-in dashboard lower the barrier to entry substantially.

MicroK8s is the best Kubernetes for Ubuntu. If your team lives on Ubuntu workstations and servers, snap-based installation, self-healing, and dqlite HA make it the most frictionless operational experience on that platform.

k0s fills an important niche: mixed Linux fleets and environments where zero host OS dependencies matter more than community size or addon richness.

RKE2 is the right answer when your compliance officer needs CIS Kubernetes Benchmark and FIPS 140-2. The resource overhead is the price of admission to heavily regulated sectors.

Resources

This post was written in April 2025. Kubernetes moves fast — always check the official documentation for the latest version information.

Tags: kubernetes k8s k3s kind minikube microk8s vcluster k0s rke2 devops infrastructure edge-computing cloud-native containers cncf

Running k3s on Proxmox: A Multi-Node Cluster with a VM and LXC Worker — The Hard Way and Back

Pendela BhargavaSai — Tue, 21 Apr 2026 03:30:00 +0000

A practical guide covering installation, troubleshooting, and the real story of getting k3s to run inside an LXC container

Introduction

Kubernetes is powerful but notorious for being heavy. k3s, the lightweight Kubernetes distribution from Rancher, fixes that. It strips out legacy APIs, bundles containerd, and ships as a single binary under 100MB. It is perfect for homelabs, edge deployments, and resource-constrained environments.
(more about k3s: https://traefik.io/glossary/k3s-explained/)

This is the first of a series of posts describing how to bootstrap a Kubernetes cluster on Proxmox using ubuntu VM and LXC containers. By the end of the series, the aim is to have a fully working Kubernetes (K3S) install including MetalLB load balancer, Gateway API controller and an Istio service mesh. I’ll also have some sample applications installed for good measure.

Basically why do I need a Kubernetes cluster ?

At work, I’ve used large K8S clusters in production environments (AWS), clusters are abstracted away behind platform teams, which is efficient for delivery but leaves gaps in understanding how scheduling, networking, storage, and controllers really behave under the hood. Setting up your own cluster gives you that missing layer of operational intuition: you get to break things, debug them, and understand why they broke. For someone already running a fairly complex home setup, using Kubernetes as a unifying platform to experiment, whether or not you fully migrate all your Docker Compose stacks—is less about necessity and more about building practical, transferable expertise.

In this post I document how I built a three-node k3s cluster on Proxmox VE with:

1 master node — a Proxmox VM running Ubuntu
1 VM worker node — a standard Proxmox VM (worker1)
1 LXC worker node — a Proxmox LXC container (worker2)

The VM setup was straightforward. The LXC setup was not. This post focuses heavily on the LXC journey — the errors, the fixes, the Linux internals involved, and what it finally took to make it work.

Part 1: Setting Up the Master Node

Installing k3s Server

On the master VM, installing k3s is a single command:


curl  -sfL  https://get.k3s.io | sh  -

k3s sets up a systemd service, installs containerd, and bootstraps a single-node Kubernetes cluster automatically.

Fixing kubectl Access

After installation, running kubectl get nodes immediately fails:

The connection to the server localhost:8080 was refused

This happens because kubectl defaults to localhost:8080 when no kubeconfig is set. k3s stores its kubeconfig at /etc/rancher/k3s/k3s.yaml. The fix:


mkdir  -p  ~/.kube

sudo  cp  /etc/rancher/k3s/k3s.yaml  ~/.kube/config

sudo  chown  $USER:$USER  ~/.kube/config

Or export it permanently:


echo  'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml' >> ~/.bashrc

source  ~/.bashrc

Retrieve the Node Token

Worker nodes need a token to join the cluster. Grab it from the master:


sudo  cat  /var/lib/rancher/k3s/server/node-token

Keep this value — it is used in every worker join command.

Part 2: Adding the VM Worker (worker1)

Joining the Cluster

On the worker VM, run:


curl  -sfL  https://get.k3s.io | \

K3S_URL=https://192.168.1.44:6443  \

K3S_TOKEN=<node-token> \

sh  -

Problem: Node Password Rejected

The agent started but immediately logged:


Node password rejected, duplicate hostname or contents of

'/etc/rancher/node/password' may not match server node-passwd entry

This happened because the worker VM had previously joined the cluster. k3s stores a node password on both the node (/etc/rancher/node/password) and the master (as a Kubernetes secret). When they don't match, the server rejects the node.

Fix — on the worker:


sudo  systemctl  stop  k3s-agent

sudo  rm  -f  /etc/rancher/node/password

sudo  rm  -rf  /var/lib/rancher/k3s/agent/

sudo  systemctl  start  k3s-agent

Fix — on the master, delete the stale secret:


kubectl  get  secrets  -n  kube-system | grep  node-password

kubectl  delete  secret  worker1.node-password.k3s  -n  kube-system

Problem: Duplicate Hostname

Both the master and worker had the hostname k3s. k3s uses the hostname as the node name, so the server rejected the second node as a duplicate.

Fix — rename the worker:


sudo  hostnamectl  set-hostname  worker1

After renaming and cleaning up the stale secret, the worker joined successfully.

Part 3: The LXC Worker — The Real Story

What is an LXC Container?

LXC (Linux Containers) is a lightweight virtualisation technology. Unlike VMs which emulate full hardware, LXC containers share the host kernel directly. They use Linux namespaces for isolation and cgroups for resource control. They are faster and more efficient than VMs but have less isolation.

Proxmox LXC containers can be privileged (root inside = root on host) or unprivileged (root inside maps to a regular user on host via UID namespacing). Unprivileged is the default and more secure option.

Creating the LXC Container

In Proxmox, I created a Debian Trixie LXC container with:

Joining the Cluster


curl  -sfL  https://get.k3s.io | \

K3S_URL=https://192.168.1.44:6443  \

K3S_TOKEN=<node-token> \

sh  -

The install script ran and printed [INFO] systemd: Starting k3s-agent — and then nothing. It just hung.

Checking the journal:


journalctl  -u  k3s-agent  -f

Error 1: `/dev/kmsg: no such file or directory`


Error: failed to run Kubelet: failed to create kubelet: open /dev/kmsg: no such file or directory

What is /dev/kmsg?

/dev/kmsg is the kernel message buffer device. The Linux kernel uses it to log messages (this is what dmesg reads). kubelet uses it to watch for OOM (Out of Memory) kill events via the oomWatcher. Without it, kubelet refuses to start.

In an unprivileged LXC container, /dev/kmsg does not exist because the container does not have access to kernel devices.

Fix — bind mount from host:

In /etc/pve/lxc/209.conf on the Proxmox host:


lxc.mount.entry: /dev/kmsg dev/kmsg none bind,create=file

This bind mounts the host's /dev/kmsg into the container. Stop and start (not restart) the LXC:


pct  stop  209

pct  start  209

Error 2: `/dev/kmsg: operation not permitted`

After adding the bind mount, the error changed slightly:


open /dev/kmsg: operation not permitted

The file now existed in the container but the process was not allowed to open it. The container was still running in user namespace mode (unprivileged), and AppArmor was blocking the access.

Fix — disable AppArmor restriction:


lxc.apparmor.profile: unconfined

AppArmor is a Linux Security Module that applies mandatory access control policies. The default Proxmox LXC AppArmor profile blocks access to kernel devices like /dev/kmsg. Setting it to unconfined removes all AppArmor restrictions for this container.

Error 3: `/proc/sys/kernel/panic: read-only file system`


Failed to start ContainerManager:

open /proc/sys/kernel/panic: read-only file system

open /proc/sys/kernel/panic_on_oops: read-only file system

open /proc/sys/vm/overcommit_memory: read-only file system

What is /proc/sys?

/proc is a virtual filesystem the kernel exposes so userspace can read and write kernel parameters. /proc/sys/ specifically contains sysctl values — tuneable kernel settings.

kubelet needs to write to these on startup:

kernel/panic — configure kernel panic timeout
kernel/panic_on_oops — whether a kernel oops causes a panic
vm/overcommit_memory — memory overcommit policy

In an unprivileged LXC container, /proc is mounted read-only for safety. Any process inside the container (even root inside) cannot modify these values.

Fix — mount proc and sys as read-write:


lxc.mount.auto: "proc:rw sys:rw"

This tells LXC to mount /proc and /sys with read-write access instead of the default read-only.

Error 4: Various Permission Denied Errors


write /proc/self/oom_score_adj: permission denied

Failed to set sysctl: open /proc/sys/net/netfilter/nf_conntrack_max: permission denied

These were caused by the container still running as unprivileged — the process was root inside the container but mapped to a normal user on the host, so many privileged operations were blocked.

Fix — switch to privileged container:


unprivileged: 0

This is the most significant change. A privileged container maps root inside to actual root on the host. This removes the UID namespace remapping that caused most of the permission errors.

Also needed:


lxc.cgroup2.devices.allow: a

lxc.cap.drop:

cgroup2.devices.allow: a — allows the container access to all devices via the cgroup device controller
cap.drop: (empty) — prevents Proxmox from dropping any Linux capabilities. By default, Proxmox drops capabilities like CAP_SYS_ADMIN, CAP_NET_ADMIN, and CAP_SYS_PTRACE from LXC containers. k3s needs these.

Also needed: `features: keyctl=1,nesting=1`

keyctl=1 — enables the Linux kernel keyring inside the container. containerd uses this to securely store credentials and keys for image pulls.
nesting=1 — enables nested containerisation. k3s runs containerd inside the LXC container, and containerd runs pods (more containers) inside itself. Without nesting enabled, Proxmox blocks the inner container creation.

Final Working LXC Config

After applying all these changes and doing a full pct stop / pct start:


journalctl  -u  k3s-agent  -f

# ... containerd is now running

# ... Server ACTIVE

# ... Started kubelet

Summary: What Each Modification Does

Part 4: LXC as a k3s Worker — Features and Limitations

Features / Advantages

Resource efficiency — LXC containers consume significantly less memory and CPU than VMs. A VM needs a full OS kernel in memory. An LXC container shares the host kernel, so the overhead is minimal. worker2 running k3s uses around 250–300MB RAM idle versus a VM which would use 500MB+ for the OS alone.

Fast startup — LXC containers start in 1–3 seconds versus 15–30 seconds for a VM. For ephemeral worker nodes or autoscaling scenarios this matters.

Storage efficiency — LXC uses the host filesystem directly (with a root filesystem overlay). No separate virtual disk emulation layer. I/O is closer to bare metal performance.

Simple networking — LXC containers participate in the same Proxmox bridge (vmbr0) as VMs. No extra networking configuration is needed for k3s to communicate between the master VM and the LXC worker.

Density — you can run more LXC containers on the same Proxmox host than VMs, making it ideal for testing multi-node cluster topologies on limited hardware.

Limitations

Shared kernel — no kernel version isolation — all LXC containers on a host run the same kernel version as the host. You cannot run a different kernel inside an LXC container. This matters if you need a specific kernel feature or version for your workloads.

Privileged mode is a security trade-off — to get k3s working we had to switch to a privileged container and disable AppArmor. In a privileged container, a root escape inside the container gives root on the host. For a homelab or trusted environment this is acceptable; for production or multi-tenant setups it is a significant risk.

No hardware virtualisation — LXC containers cannot run nested VMs. If your workloads need hardware-level isolation or GPU passthrough in the container, a VM is required.

Kernel module limitations — the LXC container cannot load kernel modules that aren't already loaded on the host. During setup we saw:


modprobe: FATAL: Module br_netfilter not found

These modules need to be loaded on the Proxmox host, not inside the container.

Some syscalls are blocked — even in privileged mode, certain syscalls that could affect the host are restricted. This can cause subtle compatibility issues with some container workloads.

Not suitable for untrusted workloads — because the kernel is shared, a kernel exploit inside an LXC container could theoretically affect the host and all other containers. Never run untrusted code in a privileged LXC container.

Conclusion

Getting k3s running on a Proxmox LXC container is absolutely possible, but it requires understanding why each restriction exists and selectively removing the ones that conflict with k3s's requirements. The journey from a blank LXC to a working cluster node touched on AppArmor, Linux capabilities, cgroups, kernel device access, namespace nesting, and virtual filesystem permissions.

The key takeaway: LXC containers are not VMs. They share the host kernel, and every security restriction that makes them safe is also a potential blocker for complex software like k3s that expects a full OS environment. The solution is not to blindly disable everything — it is to understand each error, trace it to the underlying Linux feature, and make the minimal change required to unblock it.

The final cluster — one control plane VM and two workers (one VM, one LXC) — runs stably with k3s managing scheduling, networking, and DNS across all three nodes via CoreDNS.

I now have a vanilla multi-node Kubernetes cluster running in a Ubuntu VM and an LXC container and accessible from my machine. It’s got nothing deployed inside it yet, but that’s easily fixed.... see u in part 2.

*Built on Proxmox VE with k3s v1.34.6+k3s1 — Debian Trixie LXC — Ubuntu VM nodes

"Why can’t I just mount S3 like a drive?” AWS finally answering that question in 2026

Pendela BhargavaSai — Sun, 12 Apr 2026 13:35:35 +0000

From "why can't I just mount S3 like a drive?" to AWS finally answering that question in 2026.

I've had that conversation more times than I can count.

A developer joins a new AWS project, looks at the architecture, and asks: "We're already storing everything in S3 — why do we also need EFS? Can't we just mount S3 directly?"

And every time, the answer was the same patient explanation about object storage vs file systems, why they're fundamentally different, and why you need separate services for separate workloads. It was the right answer. It just wasn't a satisfying one.

That changed in April 2026 when AWS launched S3 Files — and suddenly that conversation got a lot shorter.

But before we get there, let's start from the beginning. Because understanding why S3 Files matters requires understanding the problem it's solving. And that means understanding the full AWS storage landscape.

The AWS Storage Trinity (Before S3 Files)

AWS has three primary storage services, each built for a completely different purpose. Engineers often get confused because on the surface they all seem to do the same thing: store data. But the way they store it — and who can access it and how — is completely different.

Here's the simplest way I know to think about it:

S3 is like a giant library. You can store billions of books (objects), and anyone with the right access can retrieve any book. But to fix a typo on page 47, you have to reprint the entire book.
EBS is like a hard drive physically attached to your computer. Super fast, but only your computer can use it.
EFS is like a shared office filing cabinet on a network. Anyone in the office can open a drawer, pull out a folder, and edit a document — at the same time.

Let's go deeper on each one.

Amazon S3 — Object Storage Built for Scale

S3 (Simple Storage Service) launched in 2006 and fundamentally changed how the world thinks about storing data. The core idea is simple: you have buckets, and inside buckets you store objects. Each object is just a file plus its metadata, stored at a unique key (think of it like a URL).

What makes S3 special

Virtually unlimited scale. S3 stores more than 500 trillion objects across hundreds of exabytes today.
11 nines of durability (99.999999999%). AWS automatically replicates your data across at least three Availability Zones.
Pay only for what you use. No minimum capacity, no infrastructure to manage.
Multiple storage classes. From S3 Standard (~$0.023/GB) down to Glacier Deep Archive (~$0.00099/GB) for data you almost never touch.

The one thing S3 cannot do

Here's the catch that trips everyone up: S3 is not a file system.

When you store something in S3, it becomes an immutable object. If you want to change even a single character in a file, you have to download the entire object, make your change, and re-upload the whole thing as a new object. There's no such thing as "open this file and edit line 47." That's just not how object storage works.

This isn't a bug — it's by design. The immutability of objects is part of what makes S3 so durable and scalable. But it creates real friction for any workload that needs to work with data the way normal applications do: open a file, read some bytes, write some bytes, save.

# What you can do with S3
aws s3 cp myfile.txt s3://my-bucket/myfile.txt    # upload
aws s3 cp s3://my-bucket/myfile.txt ./myfile.txt  # download
aws s3 rm s3://my-bucket/myfile.txt               # delete

# What you CANNOT do
# Open myfile.txt and append a line — impossible without full re-upload

Amazon EBS — The Fast Attached Drive

EBS (Elastic Block Store) is block storage — the AWS equivalent of an SSD attached directly to your server. When you launch an EC2 instance, the root volume (where the operating system lives) is an EBS volume.

What EBS is good at

Speed. EBS delivers single-digit millisecond latency because it behaves like a local disk.
POSIX semantics. You can open files, write individual bytes, seek to specific positions — everything a normal file system supports.
Consistency. What you write is immediately readable. No eventual consistency concerns.

The hard limit of EBS

EBS volumes can only be attached to one EC2 instance at a time (with some multi-attach exceptions for specific use cases).

This means if you have a cluster of 10 EC2 instances all running your application, each one needs its own EBS volume. They can't share data through EBS. If instance A writes a file, instance B can't see it without some kind of sync mechanism.

EC2 Instance A  →  EBS Volume A  (can't share)
EC2 Instance B  →  EBS Volume B  (separate, isolated)
EC2 Instance C  →  EBS Volume C  (separate, isolated)

For single-instance workloads — databases, operating system volumes, single-server applications — EBS is excellent. The moment you need shared storage across multiple servers, you hit a wall.

Amazon EFS — The Shared Network Drive

EFS (Elastic File System) is AWS's managed Network File System (NFS). Think of it as a shared drive that any number of EC2 instances, containers, or Lambda functions can mount simultaneously and use like a local file system.

What EFS solves

Concurrent access. Thousands of compute resources can mount and use the same EFS volume at the same time.
Full POSIX semantics. Open files, edit bytes in-place, file locking, directory operations — everything works.
Scales automatically. The file system grows and shrinks as you add or remove files. No capacity planning required.
Sub-millisecond latency on Standard tier.

EC2 Instance A  ──┐
EC2 Instance B  ──┤──→  EFS Volume  (all share the same files)
EC2 Instance C  ──┘
Lambda Function ──┘

Where EFS falls short

The pricing model. EFS charges you for every gigabyte stored, whether you touched it this month or not. Standard tier is $0.30/GB-month — roughly 13x more expensive than S3 Standard per gigabyte.

This is fine when your data is "hot" (actively accessed). It's painful when you have petabytes of data where only a fraction is actively used at any time. You end up paying full file system prices for data that's sitting idle.

And the other problem: EFS has zero native integration with S3. They're completely separate systems. Your data lake is in S3. Your compute needs EFS. So you write sync scripts to copy data back and forth — and now you have two copies of everything, two storage bills, and a manual process that breaks at the worst possible times.

The Old Workflow Pain (The Problem All of This Creates)

Before S3 Files, a typical ML or data engineering team's workflow looked like this:

S3 Data Lake
    ↓  (manual copy — takes time, costs money)
EFS Volume
    ↓  (mount on EC2)
EC2 Training Job
    ↓  (output back to EFS)
    ↓  (another manual copy)
S3 Data Lake  ← results stored here for analytics

Every arrow in that diagram is a point of failure. Every copy step is a delay, a cost, and a potential for the two copies to drift out of sync. Engineers were spending real engineering hours maintaining these sync pipelines — hours that weren't building anything valuable.

This is the problem that s3fs tried to solve, years before AWS had an official answer.

s3fs-fuse — The Community's Workaround

If you've been working with AWS for a few years, you've probably encountered s3fs-fuse. It's an open-source FUSE (Filesystem in Userspace) tool that lets you mount an S3 bucket as a local directory on Linux, macOS, or FreeBSD.

# Install
sudo apt-get install s3fs

# Configure credentials
echo "ACCESS_KEY_ID:SECRET_ACCESS_KEY" > ~/.passwd-s3fs
chmod 600 ~/.passwd-s3fs

# Mount your bucket
s3fs my-bucket /mnt/s3-data -o passwd_file=~/.passwd-s3fs

After that, you can run ls, cp, cat — your S3 bucket looks like a local folder. For a quick demo or a simple use case, it feels magical.

What's actually happening under the hood

Here's the thing nobody tells you upfront: s3fs isn't really giving you file system access to S3. It's translating file commands into S3 API calls — and the translation has serious limitations.

When you "edit" a file through s3fs, this is what actually happens:

You: nano myfile.txt  (make a small change, save)
     ↓
s3fs: GET entire object from S3 → download to local temp cache
s3fs: You edit the local temp copy
s3fs: On file close → PUT entire object back to S3 (full re-upload)

Change one character in a 10GB file? s3fs downloads all 10GB, makes the change, and uploads all 10GB again. Every time.

The real limitations you need to know

No file locking. If two processes try to write to the same file through s3fs at the same time, you get data corruption. Not an error message — silent data corruption.

No atomic renames. Renaming a file in s3fs copies it to a new key and deletes the old one. Any application that relies on atomic renames (which includes most databases and many log processors) will break.

Slow directory listings. Every ls is a ListObjects API call to S3. On a bucket with millions of objects, this is painfully slow.

No hard links or symbolic links. S3 simply doesn't support them.

Operation          | What s3fs does              | Problem
-------------------|-----------------------------|-----------------------
Read file          | GET entire object           | Slow for large files
Edit file          | Download → edit → full PUT  | Expensive re-upload
Append to file     | Rewrite entire object       | Very expensive
Rename file        | Copy + Delete               | Not atomic
File lock          | Not supported               | Data corruption risk
List directory     | ListObjects API call        | Slow on large buckets

s3fs works well for lightweight, read-heavy, single-process use cases. But the moment you need multi-process access, in-place edits, or production reliability — it starts breaking down. The community built it because AWS didn't have a better answer. Eventually, AWS tried building their own version.

Mountpoint for S3 — AWS's Open-Source Attempt (2023)

In 2023, AWS released Mountpoint for S3, their own open-source FUSE client. It was faster than s3fs-fuse and better optimised for cloud-native read-heavy workloads.

But it still couldn't do in-place edits, directory renames, or file locking. It was better than s3fs-fuse, but it still hit the same fundamental ceiling: you can't make S3's API behave like a real file system by pretending.

AWS knew this. Internally, they'd been trying to solve it properly for years.

Amazon S3 Files — The Real Solution (April 2026)

On April 7, 2026, AWS launched S3 Files — and it's the most significant S3 update since the service launched.

The internal project was even called "EFS3" at one point. One engineer on the team described the design process as "a battle of unpalatable compromises." Getting object storage and file system semantics to truly coexist is genuinely hard engineering. Every design decision forced a tradeoff where either the file presentation or the object presentation had to give something up.

What they landed on is clever: instead of trying to make the S3 API behave like a file system (which is what s3fs does), they did the opposite — they took a real, production-grade file system (EFS) and connected it directly to S3 storage.

How S3 Files actually works

S3 Files uses a two-tier architecture:

Tier 1 — EFS Cache Layer (hot data)

Stores your active working set: recently written files, recently read files, metadata
Delivers ~1ms latency
Serves small files (under 128KB by default) entirely from cache
Handles all NFS file operations — open, read, write, rename, lock

Tier 2 — S3 Bucket (your full dataset)

Holds your complete data at normal S3 prices (~$0.023/GB)
Large reads (1MB+) bypass the cache entirely and stream directly from S3 for free
Changes made through the file system sync back to S3 automatically within minutes

Your Application
      ↓  (NFS mount — standard Linux file operations)
EFS Cache Layer  ←→  Smart Router
      ↓                    ↓
   Hot data            Cold/large data
   (~1ms)              (streams from S3, free)
      ↓                    ↓
      └────────────────────┘
                  ↓
            S3 Bucket
       (your data, always here)

The key insight: your data never leaves S3. The EFS cache is just a smart caching layer on top. You're not maintaining two copies — you have one copy in S3, accessible via both the S3 API and the file system mount simultaneously.

OLD way to New way

Getting started in 3 steps

Step 1: Create an S3 file system

In the AWS Console → S3 → File Systems → Create file system. Enter your bucket name, done.

Or via CLI:

aws s3api create-file-system --bucket my-bucket
aws s3api create-mount-target --file-system-id fs-xxxx --subnet-id subnet-xxxx

Step 2: Mount it on your EC2 instance

Make sure the amazon-efs-utils package is installed (preinstalled on AWS AMIs), then:

sudo mkdir /mnt/s3files
sudo mount -t s3files fs-0aa860d05df9afdfe:/ /mnt/s3files

Step 3: Use it like any local directory

# Create a file
echo "Hello S3 Files" > /mnt/s3files/hello.txt

# Edit it in place
echo "New line added" >> /mnt/s3files/hello.txt

# List files
ls -la /mnt/s3files/

# The same data is accessible via S3 API too
aws s3 ls s3://my-bucket/

Changes you make through the file system mount appear in S3 within minutes. Changes made directly to the S3 bucket appear in the file system within seconds.

Security — what you need to know

IAM integration for access control at both file system and object level
Data encrypted in transit using TLS 1.3
Data encrypted at rest using SSE-S3 (or KMS if you prefer customer-managed keys)
POSIX permissions (UID/GID) stored as S3 object metadata
Monitor via CloudWatch metrics and CloudTrail logs

Pricing — the part that actually makes sense

S3 Files charges EFS-level rates, but only on the fraction of data you're actively working with:

What you pay for	Rate
High-performance storage (hot data)	$0.30/GB-month
Reads (small files served from cache)	$0.03/GB
Writes	$0.06/GB
Everything else in your S3 bucket	Standard S3 rates (~$0.023/GB)

If you have a 100TB dataset but only 1TB is actively used at any time — you pay EFS rates on 1TB and S3 rates on the other 99TB. AWS claims up to 90% cost savings compared to the old pattern of cycling data between S3 and a dedicated EFS volume.

Putting It All Together — Which Service Should You Use?

Here's the honest answer:

Use this	When you need
S3	Bulk storage, backups, data lakes, analytics, static assets, anything accessed via API
EBS	OS volumes, databases, single-instance high-performance storage
EFS	Shared file system for legacy NAS migration, on-premises workloads moving to cloud, apps that need pure NFS without S3
S3 Files	ML pipelines, agentic AI workflows, data engineering, any workload where both S3 API and file system access are needed
s3fs-fuse	Quick prototypes, read-heavy single-process scripts, legacy apps where you can't change the architecture

The quick comparison

Why This Matters for ML and AI Workloads

If you're building machine learning pipelines or agentic AI systems, S3 Files is worth paying close attention to.

The old workflow was: data lives in S3 → copy to EFS before training → run training job → copy results back to S3. For large datasets, that copy step alone could take hours. You were also paying double storage costs during the transition.

With S3 Files, your training job mounts the S3 bucket directly. The EFS cache warms up as your training reads data. No copy step. No sync script. No duplicate storage.

For agentic AI systems specifically — where multiple agents need to coordinate through shared files, read from each other's outputs, maintain shared state — S3 Files provides exactly the concurrent NFS access with close-to-open consistency that these workloads need. Standard Python file operations, standard shell tools, all working against data that lives in S3.

The Short Version

For a decade, AWS storage was a choice: pay S3 prices and lose file system semantics, or pay EFS prices and lose S3 integration. Teams wrote sync scripts, maintained duplicate data, and spent engineering time on storage plumbing instead of actual product work.

s3fs-fuse was the community's best attempt at a workaround — and it worked, up to a point. But it was always emulating file system behavior on top of an API that wasn't designed for it.

S3 Files is the first time AWS has genuinely solved this at the right layer. Real NFS semantics, real S3 storage, real production reliability. One bucket, two protocols, no compromises.

If you've ever maintained a sync script between your data lake and your compute layer — you know exactly what problem this solves. And you know exactly how good it feels to delete that script.

Resources

Published April 2026. All pricing figures reflect us-east-1 as of the time of writing.

If this helped you, drop a reaction or leave a comment — curious what storage patterns others are running into in the wild.

Forem: Pendela BhargavaSai

The Definitive Guide to Lightweight Kubernetes: KIND, Minikube, MicroK8s, K3s, Vcluster, k0s, and RKE2 Compared

Table of Contents

Why Lightweight Kubernetes Matters

The Contenders at a Glance

1. KIND — Kubernetes IN Docker

What It Is

Architecture

Core Components

Key Features

Quick Start

Pros

Cons

Best For

2. Minikube - The Developer's Workhorse

What It Is

Architecture

Core Components

Key Features

Quick Start

Pros

Cons

Best For

3. MicroK8s - Zero-Ops by Canonical

What It Is

Architecture

Core Components

Key Features

Quick Start

Pros

Cons

Best For

4. K3s - Production-Grade at the Edge

What It Is

Architecture

Core Components

Key Features

Quick Start

HA Setup (Embedded etcd)

Pros

Cons

Best For

5. Vcluster — Kubernetes Inside Kubernetes

What It Is

Architecture

Core Components

Key Features

Quick Start

Pros

Cons

Best For

6. k0s — Zero Dependencies, Zero Friction

What It Is

Architecture

Key Features

Quick Start

Pros

Cons

Best For

7. RKE2 — Security-First Enterprise K8s

What It Is

Architecture

Key Features

Quick Start

Pros

Cons

Best For

Scoring Across 8 Dimensions

Use Case Decision Guide

The Decision Tree

Final Verdict

Resources

Running k3s on Proxmox: A Multi-Node Cluster with a VM and LXC Worker — The Hard Way and Back

Introduction

Basically why do I need a Kubernetes cluster ?

Part 1: Setting Up the Master Node

Installing k3s Server

Fixing kubectl Access

Retrieve the Node Token

Part 2: Adding the VM Worker (worker1)

Error 1: `/dev/kmsg: no such file or directory`

Error 2: `/dev/kmsg: operation not permitted`

Error 3: `/proc/sys/kernel/panic: read-only file system`

Also needed: `features: keyctl=1,nesting=1`