Forem: Parimal

Why Cilium Outperforms AWS VPC CNI: A Deep Dive into Kubernetes Networking

Parimal — Thu, 14 Aug 2025 09:18:57 +0000

Running Kubernetes at scale in AWS presents unique networking challenges that can significantly impact your application performance and operational efficiency. While AWS VPC CNI serves as the default networking solution for EKS clusters, it often becomes the bottleneck when dealing with high-scale or dynamic workloads. Enter Cilium - an eBPF-powered CNI that's revolutionizing how we think about Kubernetes networking.

The Foundation: Understanding CNI in Kubernetes

Container Network Interface (CNI) plugins serve as the backbone of Kubernetes networking, handling three critical responsibilities:

IP Address Management: Allocating unique IP addresses to pods
Network Configuration: Setting up routes and network interfaces for inter-pod communication
Network Integration: Bridging container networking with the underlying infrastructure

When a pod initializes, Kubernetes delegates to the CNI plugin with a simple request: "Assign an IP to this pod and ensure it can communicate with the cluster." Without a functional CNI, your pods remain isolated islands with no networking capabilities.

AWS VPC CNI: The Default Choice and Its Limitations

Amazon EKS ships with AWS VPC CNI as the default networking solution, designed to integrate seamlessly with AWS networking primitives.

Architecture Deep Dive

AWS VPC CNI operates on a straightforward principle:

Each worker node receives a primary Elastic Network Interface (ENI)
Additional secondary ENIs can be attached based on instance capacity
Each ENI supports multiple secondary IP addresses
Pods receive VPC-native IP addresses directly from the subnet pool

The Benefits

Native VPC Integration: Pods become first-class citizens in your VPC, enabling direct communication with other AWS services without additional network hops.

Zero Encapsulation Overhead: Network packets flow through native AWS routing without additional headers or processing overhead.

Security Group Integration: Pods can leverage existing VPC security group policies for network access control.

The Performance Bottlenecks

Despite its integration advantages, AWS VPC CNI introduces several scalability constraints:

ENI Management Latency: Each ENI attachment requires AWS API calls, introducing latency measured in seconds rather than milliseconds. During rapid scaling events, this becomes a significant bottleneck.

Subnet IP Address Exhaustion: Every pod consumes a routable VPC IP address, leading to subnet exhaustion in large clusters.

Instance-Specific Scaling Limits: The maximum number of pods per node is constrained by the ENI and IP limits of your EC2 instance type. For example, an m5.large instance supports only 3 ENIs with 10 IPs each, limiting you to approximately 29 pods per node.

Limited Observability: Network flow visibility requires additional tooling and configuration, complicating troubleshooting and security auditing.

Cilium: The eBPF-Powered Alternative

Cilium leverages Extended Berkeley Packet Filter (eBPF) technology to provide high-performance networking with advanced observability and security features baked in.

Core Advantages

Hubble Integration: Real-time network flow observability without additional agents or performance overhead.

Advanced Network Policies: Support for Layer 3, 4, and 7 filtering with HTTP-aware rules.

Service Mesh Without Sidecars: Built-in load balancing, encryption, and traffic management without the resource overhead of traditional service mesh proxies.

Flexible IPAM Options: Multiple IP address management modes to suit different architectural requirements.

IPAM Mode Comparison

Cilium supports multiple IPAM strategies, each optimized for different use cases:

ENI Mode: Functions similarly to AWS VPC CNI, using secondary ENI IPs while adding Cilium's observability and policy features.

Cluster Pool (Overlay) Mode: Manages IP addresses from Cilium-controlled pools, using VXLAN or Geneve encapsulation for pod-to-pod communication.

Kubernetes Mode: Delegates IP management to Kubernetes' native IPAM, providing flexibility for custom implementations.

Performance Analysis: Where Cilium Excels

The performance differential between AWS VPC CNI and Cilium becomes most apparent during pod lifecycle operations and scaling events.

AWS VPC CNI Pod Startup Sequence

1. Pod creation request → Kubelet
2. CNI invocation → IP address requirement
3. ENI capacity check → Available secondary IPs
4. ENI attachment (if needed) → AWS API call (2-5 seconds)
5. Secondary IP allocation → AWS API call
6. Network interface configuration → Pod ready

This sequence introduces significant latency, particularly when ENI limits are reached and new interfaces must be provisioned.

Cilium Overlay Mode Startup Sequence

1. Pod creation request → Kubelet  
2. CNI invocation → IP address requirement
3. Instant IP allocation → From pre-allocated pool
4. eBPF program configuration → Millisecond-level operation
5. Network interface ready → Pod ready

The elimination of AWS API dependencies results in pod networking readiness in milliseconds rather than seconds.

Trade-off Considerations

Encapsulation Overhead: Overlay networking introduces minimal packet processing overhead due to VXLAN/Geneve headers.

VPC Integration: Pods in overlay mode aren't directly addressable from the VPC, requiring ingress controllers or NodePort services for external access.

Network Policies: eBPF-based policy enforcement often outperforms iptables-based alternatives, especially at scale.

Real-World Migration Impact

Organizations migrating from AWS VPC CNI to Cilium overlay mode typically report:

Before Migration Challenges

Pod scaling operations taking multiple minutes due to ENI provisioning delays
Frequent subnet IP address exhaustion requiring subnet expansion or cluster restructuring
Complex toolchain requirements for network observability and security policy enforcement
Difficulty troubleshooting inter-service communication issues

Post-Migration Improvements

Pod networking readiness reduced to sub-second timeframes
Elimination of subnet IP address constraints enabling higher pod density
Unified platform for networking, security, and observability through Cilium and Hubble
Enhanced debugging capabilities with flow-level visibility

Decision Framework: Choosing the Right CNI

Your CNI choice should align with your specific requirements and constraints:

Choose AWS VPC CNI When:

Regulatory compliance mandates VPC-native pod IP addresses
Direct pod-to-AWS-service communication is required without additional network hops
Your workloads are relatively static with predictable scaling patterns
You have sufficient subnet IP address space allocated

Choose Cilium ENI Mode When:

You need VPC-native IPs but want enhanced observability and security features
Compliance requirements are flexible regarding network encapsulation
You're planning to implement advanced network policies

Choose Cilium Overlay Mode When:

Rapid scaling and high pod density are critical requirements
Subnet IP address management is becoming operationally complex
You need comprehensive network observability and security policy enforcement
Your applications can work with ingress-based external connectivity

Implementation Considerations

Migration Strategy

Migrating from AWS VPC CNI to Cilium requires careful planning:

Cluster Preparation: Ensure your EKS cluster version supports alternative CNIs
Network Policy Audit: Review existing security groups and translate to Cilium network policies
Service Discovery: Verify that your service discovery mechanisms work with overlay networking
Monitoring Integration: Plan for migrating network monitoring from AWS-native tools to Hubble

Performance Optimization

eBPF Program Efficiency: Cilium's eBPF programs are compiled for your specific kernel version, ensuring optimal performance.

CPU and Memory Usage: Cilium typically uses fewer resources than traditional iptables-based CNIs, especially as the number of network policies grows.

Network Throughput: While overlay networking introduces minimal overhead, direct benchmarking in your environment is recommended.

The Future of Kubernetes Networking

eBPF technology continues evolving rapidly, with new capabilities being added regularly. Cilium's position at the forefront of this evolution means choosing it today provides access to emerging features like:

Advanced load balancing algorithms without external load balancers
Multi-cluster networking with transparent service discovery across clusters
Enhanced security features including runtime threat detection
Performance optimizations that leverage new eBPF capabilities

Conclusion

While AWS VPC CNI remains a solid choice for straightforward, compliance-driven Kubernetes deployments, Cilium offers compelling advantages for organizations prioritizing performance, scalability, and operational simplicity. The combination of eBPF-powered networking, comprehensive observability through Hubble, and flexible IPAM options makes Cilium particularly attractive for dynamic, high-scale workloads.

The decision ultimately depends on your specific requirements, but as Kubernetes environments grow in complexity and scale, the advanced capabilities provided by Cilium's eBPF foundation position it as the networking solution for the future of container orchestration.

Have you experienced ENI limits or scaling challenges with AWS VPC CNI? Share your experiences and questions about Cilium migration in the comments below.

Demystifying mTLS in Kubernetes: Certs, Components, and Cluster Security.

Parimal — Mon, 04 Aug 2025 08:09:18 +0000

🧩 Introduction

When we talk about Kubernetes security, most developers think of Role-Based Access Control (RBAC) or Network Policies. But before these higher-level controls come into play, there's a foundational layer that silently ensures every component in the cluster is speaking to a trusted source — mutual TLS (mTLS).

In a self-managed Kubernetes cluster, unlike cloud-managed solutions like EKS or GKE, you're responsible for configuring, maintaining, and rotating these certificates. Understanding how mTLS works—and knowing where each certificate lives—gives you deeper visibility into the cluster's internal trust system and can help you troubleshoot or harden your deployment.

This post will walk you through:

What mTLS means in the Kubernetes control plane
Which components use certificates and why
Where those certificates live in your filesystem
How to inspect and manage them in your self-hosted cluster

🔐 What is mTLS in Kubernetes?

mTLS (Mutual TLS) is a mechanism where both client and server authenticate each other using certificates. In Kubernetes, this isn't just used to encrypt traffic; it's also a crucial part of authenticating various components within the control plane.

Let's look at a few examples:

When the kubelet talks to the API server, it presents a certificate proving it is a legitimate node
When the controller-manager communicates with etcd, both use TLS certificates to verify identity
Even users interacting with the cluster via kubectl often use certificates behind the scenes (depending on your setup)

This secure, verified communication is enforced using X.509 certificates, which are generated during cluster setup (commonly via kubeadm) and stored in predictable locations in the filesystem.

🧩 Kubernetes Components Using Certificates (mTLS)

Here's a breakdown of the key components and where their TLS certificates typically live in a self-managed (kubeadm-based) cluster:

Component	Certificate Path	Purpose / Notes
API Server	`/etc/kubernetes/pki/apiserver.crt` & `.key`	Server cert for API requests
API Server CA	`/etc/kubernetes/pki/ca.crt` & `.key`	Root CA that signs other certs
Controller Manager	`/etc/kubernetes/pki/controller-manager.crt` & `.key`	Used to talk to API server
Scheduler	`/etc/kubernetes/pki/scheduler.crt` & `.key`	Talks to API server securely
Etcd (server)	`/etc/kubernetes/pki/etcd/server.crt` & `.key`	API server & etcd authenticate each other
Etcd (peer)	`/etc/kubernetes/pki/etcd/peer.crt` & `.key`	For peer-to-peer etcd clustering
Etcd CA	`/etc/kubernetes/pki/etcd/ca.crt` & `.key`	Separate CA from main API server
Kubelet	`/var/lib/kubelet/pki/kubelet-client.crt` & `.key`	Authenticates with API server
Kube-Proxy	`/etc/kubernetes/pki/kube-proxy.crt` & `.key`	Talks to API server for service updates
Front Proxy	`/etc/kubernetes/pki/front-proxy-client.crt`	For API aggregation layer
Admin User (kubectl)	`/etc/kubernetes/admin.conf`	Includes user certs for API access

🛠️ Who Issues These Certificates?

In self-managed Kubernetes clusters, especially those initialized with kubeadm, the certificate lifecycle is handled during setup — and it's your responsibility to maintain them over time.

🔐 CA (Certificate Authority)

When you run kubeadm init, it generates a root Certificate Authority (CA) stored at:

/etc/kubernetes/pki/ca.crt
/etc/kubernetes/pki/ca.key

This CA is then used to sign the server and client certificates for all core components (API server, controller manager, scheduler, etc.).

There's also a separate CA for etcd:

/etc/kubernetes/pki/etcd/ca.crt
/etc/kubernetes/pki/etcd/ca.key

⚙️ Inspecting the Certificates

You can list all certs created by kubeadm using:

sudo kubeadm certs list

And you can check expiration dates with:

sudo kubeadm certs check-expiration

You'll typically see that most certs expire in 1 year by default.

🔁 Renewing Certificates

To renew all expiring certs (except kubelet), you can run:

sudo kubeadm certs renew all

Note: The kubelet uses automatic rotation, but only if the RotateKubeletClientCertificate setting is enabled in /var/lib/kubelet/config.yaml.

🔁 mTLS in Action: Example Flow

Let's take a concrete example: how the kubelet authenticates with the API server.

The kubelet starts up and wants to register the node with the cluster
It presents its client certificate (/var/lib/kubelet/pki/kubelet-client.crt) to the API server
The API server checks:

Is the cert signed by the CA (ca.crt) it trusts?
Is it expired?
Does it belong to a valid node?

If valid, the kubelet is authenticated and proceeds to perform actions like:
- Registering the node
- Posting status updates
- Watching for Pod objects

This exact mTLS exchange also happens between:

API server ↔ controller-manager
API server ↔ scheduler
API server ↔ etcd
etcd ↔ etcd peers

Each time, both sides present certificates and verify identities before any communication happens — enforcing strict trust boundaries within the control plane.

🔎 How to Inspect Kubernetes Certificates Manually

Sometimes you'll want to manually inspect a certificate — whether for troubleshooting, verifying SANs, or checking expiration dates. You can use openssl for this.

📂 Example: Inspect the API Server certificate

sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout

This will output details like:

Issuer – usually kubernetes-ca
Validity – Not Before / Not After (expiration)
Subject – e.g., CN=kube-apiserver
X509v3 Subject Alternative Name – e.g., IPs and DNS names like kubernetes, kubernetes.default, etc.

📂 Example: Check kubelet certificate

sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client.crt -text -noout

This helps confirm:

Which node the cert is tied to
If rotation has occurred
Whether the cert has expired or is about to expire

🛡️ Security & Hardening Tips

When you manage your own Kubernetes control plane, here are a few things to keep in mind:

🔄 Rotate Certificates Regularly

Certs expire after 1 year by default
Use kubeadm certs renew or set up automation
Monitor expiration with:

  kubeadm certs check-expiration

🔐 Protect Your CA Keys

Keep /etc/kubernetes/pki/ca.key and /etc/kubernetes/pki/etcd/ca.key strictly restricted
Never expose CA keys to unauthorized users or automation tools

🚫 Never Expose etcd Publicly

etcd should never be exposed to the public internet
Ensure it only listens on localhost or secure internal networks
Always enforce client certs (--cert-file, --key-file, --client-cert-auth)

🔍 Audit TLS Flags in Static Pod Manifests

Check /etc/kubernetes/manifests/ for your API server pod config:

- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
- --client-ca-file=/etc/kubernetes/pki/ca.crt

Ensure all these values are set and point to correct, non-expired certs.

✅ Conclusion

Mutual TLS is at the heart of Kubernetes' internal security model — silently ensuring that every component talks only to what it can trust. In cloud-managed clusters, this is handled for you, but in self-managed setups, you're the certificate authority, quite literally.

In this post, you learned:

What mTLS is and why it matters in Kubernetes
Which components use it and where the certs are stored
How certificates are issued and rotated
How to inspect and secure them

Understanding these mechanics gives you a strong edge as a DevOps engineer, especially when operating production clusters or preparing for certifications like the CKA.

What's your experience with managing certificates in Kubernetes? Have you encountered any certificate-related issues in your clusters? Share your thoughts in the comments below!

Deploy Your First Web App with AWS App Runner: Fully Managed & Container-Ready

Parimal — Wed, 30 Jul 2025 08:06:58 +0000

What is AWS App Runner?
Why Use App Runner?
When to Use App Runner
Pre-requisites
Deployment Guide
Observability & Scaling
Pricing Overview
Pros & Limitations
Final Thoughts
Further Reading

What is AWS App Runner?

AWS App Runner is a fully managed service that makes it easy to deploy containerized applications from your source code or container image. You don't need to manage servers, orchestration, or scaling logic—App Runner handles it all.

Whether your app lives in a GitHub repo or a container registry (like Amazon ECR), App Runner can automatically build and deploy it, scaling up and down as needed.

Why Use App Runner?

Compared to traditional services like EC2, ECS, or even Lambda, App Runner offers:

🛠️ Zero infrastructure management – No need to manage VPCs, load balancers, or clusters
🚀 Quick deployments – Deploy from GitHub or ECR in a few steps
📈 Auto-scaling – Automatically handles traffic spikes
🔁 CI/CD support – Automatic redeploys on every code change (when using GitHub)

It's especially useful when you need to move fast with a minimal setup.

When to Use App Runner

✅ Perfect for:

You want to deploy containerized apps but don't want to manage ECS or Kubernetes
You need rapid prototyping or dev environments
You're building web APIs or frontend services
You prefer GitHub-to-production workflows with minimal infrastructure fuss

Pre-requisites

Before we begin, make sure you have:

[ ] An AWS account
[ ] A public GitHub repository with a working web app and Dockerfile OR A container image in Amazon ECR (Elastic Container Registry)
[ ] Basic knowledge of Docker and containers

Deployment Guide

🚀 Deploying a Sample App with App Runner (Using GitHub)

Let's deploy a simple Node.js or Python Flask app (you can use any stack really) via GitHub:

Sample Application: GitHub Repository

Step 1: Open AWS App Runner Console

Go to the App Runner console

Step 2: Create Service

Click "Create service"
Choose "Source code repository"
Select GitHub and connect your account
Pick your repository and branch

Step 3: Configure Build & Deploy

Select "Runtime" based on your application
Select Build command and Start command based on your application
Set port (e.g., 5000 for Flask, 3000 for Node.js)
(Optional) Add environment variables if needed

Step 4: Service Settings

Set service name
Choose CPU and memory settings
Auto scaling
Observability

Step 5: Deploy!

Review and click "Create & deploy"
Wait for App Runner to build and deploy your app
Once deployed, you'll get a public HTTPS URL 🎉

Observability & Scaling

App Runner provides:

Integrated CloudWatch Logs
Health checks
Auto-scaling based on requests per second
Metrics like CPU, memory usage, and request counts

You can monitor your app without additional tooling.

Pricing Overview

💸 App Runner charges based on:

Compute time (vCPU + memory usage)
Requests served

Example: A small app with light traffic will cost only a few dollars a month. There's even a free tier!

For details: App Runner pricing

Pros & Limitations

👍 Pros

No infrastructure setup
Auto SSL and HTTPS
Integrated CI/CD
Fast and developer-friendly

👎 Limitations

Limited to HTTP apps (not suitable for background jobs)
No fine-grained networking control (compared to ECS/VPC)
Limited AWS region support

Final Thoughts

AWS App Runner is a great fit for teams that want to focus on writing code, not managing infrastructure. Whether you're spinning up a side project, internal tool, or even small production service, it gets you from GitHub to live URL in minutes.

It's not a one-size-fits-all solution (you'll want ECS or Kubernetes for complex workflows), but for many use cases, it's just right.

Setting Up a Multi-Node Kubernetes Cluster with Kind on Windows

Parimal — Tue, 29 Jul 2025 08:49:33 +0000

Want to test Kubernetes workloads locally without the complexity of cloud setup? Kind (Kubernetes in Docker) is your answer. In this guide, I'll show you how to install Kind on Windows and create a multi-node cluster that's perfect for local development and testing.

What You'll Achieve

By the end of this tutorial, you'll have:

✅ Kind properly installed on Windows with WSL2
✅ A working multi-node Kubernetes cluster (1 control-plane + 2 workers)
✅ kubectl configured to manage your cluster
✅ A solid foundation for Kubernetes experimentation

Why Choose Kind?

Kind (Kubernetes IN Docker) stands out for local development because it:

Starts fast - Clusters spin up in under 2 minutes
Runs lightweight - Uses Docker containers instead of heavy VMs
Supports multi-node - Test realistic cluster scenarios locally
Matches production - Runs actual Kubernetes, not a simulation
Integrates easily - Works seamlessly with your existing Docker workflow

Prerequisites

Before we start, ensure you have:

Windows 10/11 with WSL2 enabled
Docker Desktop installed and running
Chocolatey package manager
Basic familiarity with command line

Step 1: Install Chocolatey (If Not Already Installed)

Chocolatey makes installing tools on Windows much easier. Run this in PowerShell as Administrator:

Set-ExecutionPolicy Bypass -Scope Process -Force
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072
iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

Verify installation:

choco --version

Step 2: Configure WSL2

Ensure WSL2 is your default version for optimal Docker performance:

# Set WSL2 as default
wsl --set-default-version 2

# Verify WSL2 is working
wsl --status

Step 3: Install Docker Desktop

Install Docker Desktop with Chocolatey:

choco install docker-desktop -y

Important: During setup, make sure to:

Enable WSL2 integration in Docker Desktop settings
Restart Docker Desktop after configuration

Verify Docker is working:

docker --version
docker run hello-world

Step 4: Install kubectl and Kind

Install both tools using Chocolatey:

# Install kubectl for cluster management
choco install kubernetes-cli -y

# Install Kind
choco install kind -y

Verify installations:

kubectl version --client
kind version

Step 5: Create Your First Multi-Node Cluster

Create a Cluster Configuration

Create a file named kind-config.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker

This simple configuration creates:

1 control-plane node (manages the cluster)
2 worker nodes (run your applications)

Launch the Cluster

kind create cluster --name my-cluster --config kind-config.yaml

This command will:

Pull Kubernetes node images
Create Docker containers for each node
Set up networking between nodes
Configure kubectl to connect to your cluster

Wait time: Usually 1-2 minutes depending on your internet speed.

Step 6: Verify Your Cluster

Check that everything is working:

# View cluster information
kubectl cluster-info

# List all nodes
kubectl get nodes

# Check system pods are running
kubectl get pods -n kube-system

Expected output for kubectl get nodes:

NAME                       STATUS   ROLES           AGE   VERSION
my-cluster-control-plane   Ready    control-plane   2m    v1.28.0
my-cluster-worker          Ready    <none>          1m    v1.28.0
my-cluster-worker2         Ready    <none>          1m    v1.28.0

Quick Test: Deploy a Simple Pod

Let's confirm your cluster works by running a simple nginx pod:

# Create a test pod
kubectl run test-nginx --image=nginx --port=80

# Check the pod is running
kubectl get pods

# Clean up
kubectl delete pod test-nginx

Essential Kind Commands

Here are the key commands you'll use regularly:

# List all Kind clusters
kind get clusters

# Delete a cluster
kind delete cluster --name my-cluster

# Create cluster with different name
kind create cluster --name dev-cluster

# Switch between clusters (if you have multiple)
kubectl config get-contexts
kubectl config use-context kind-my-cluster

Troubleshooting Common Issues

"Cannot connect to Docker daemon"

Ensure Docker Desktop is running
Check WSL2 integration is enabled in Docker settings

Cluster creation hangs

# Clean up and retry
kind delete cluster --name my-cluster
docker system prune -f
kind create cluster --name my-cluster --config kind-config.yaml

kubectl context issues

# Check current context
kubectl config current-context

# Should show: kind-my-cluster

What's Next?

Now that you have Kind installed and a working cluster, you're ready to:

Deploy applications and test them locally
Experiment with Kubernetes features safely
Set up CI/CD pipelines
Learn about networking, storage, and security

In upcoming posts, I'll cover deploying applications, setting up ingress controllers, and advanced Kind configurations.

Cleanup

When you're done experimenting, clean up your resources:

# Delete the cluster
kind delete cluster --name my-cluster

# Verify cleanup
kind get clusters

Got Kind up and running? You now have a powerful local Kubernetes environment at your fingertips! What's the first thing you're planning to deploy on your new cluster? Let me know in the comments below.