Forem: Manish Pillai

Why we need Ingress in Kubernetes?

Manish Pillai — Sun, 05 Apr 2026 16:20:25 +0000

When you want to expose your service/pod outside your cluster, you can use NodePort and LoadBalancer services. They work for a "Hello World," but has some drawbacks.

The Problem with the "Standard" way

Using a NodePort means your users have to type NodeIP:31054. It’s ugly, insecure, and if that specific Node goes down, your app is "dead" to the world.

Using a LoadBalancer service is better, but it has three big headaches:

Cost: One Cloud LB per service = a massive bill.
L4 vs L7: Standard L4 LoadBalancers don’t understand URL paths like /api and don’t provide advanced routing or centralized SSL handling.
Scaling: Managing 50 different entry points for 50 microservices is a management nightmare.

The Solution: Ingress

Ingress is a single entry point that handles all the "traffic flow" work.

While the Ingress Resource is just a set of rules (YAML), the Ingress Controller (like Nginx or Traefik) is the actual engine that moves the data.

How do we access Ingress from the Outside?

The Ingress Controller itself is just another Pod in your cluster. To get traffic to the Controller from the internet, we usually do one of two things:

Option A: Create one single Type: LoadBalancer service that points only to the Ingress Controller. Now, one IP handles all your domains.
Option B: Use a NodePort on the Ingress Controller and point your external DNS (like GoDaddy/Cloudflare) to your Node IPs.

Practical Example: Routing & SSL

Here is how you configure an Ingress to handle different services and SSL termination in one place:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: main-gateway
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
    - myapp.com
    secretName: myapp-tls-secret
  rules:
  - host: myapp.com
    http:
      paths:
      - path: /billing
        pathType: Prefix
        backend:
          service:
            name: billing-service
            port:
              number: 80
      - path: /
        pathType: Prefix
        backend:
          service:
            name: main-web-service
            port:
              number: 80

Pros

Path-Based Routing: You can send /billing to the Billing Pod and / to the Web Pod using the same IP.
Centralized SSL: You handle HTTPS certificates at the Ingress level instead of inside every individual app.
Cost Efficiency: You only pay for one LoadBalancer, no matter how many services you have.

The Entry: You point one external LoadBalancer to your Ingress Controller to get traffic into the cluster.
The Logic: The Ingress Rules then decide which internal Service should actually handle the request.

In short: LoadBalancers get traffic TO the cluster; Ingress tells traffic WHERE to go inside.

How Kubernetes Resolves Service DNS

Manish Pillai — Sun, 05 Apr 2026 10:44:34 +0000

When you create a Service in Kubernetes, you get a stable ClusterIP. But let’s be honest—nobody wants to hardcode 10.96.0.10 into their application code. We want to use names, like auth-service or payment-api.

How does Kubernetes know that payment-api actually means 10.96.0.10? That’s where CoreDNS comes in.

What is CoreDNS?

CoreDNS is a flexible, extensible DNS server that sits inside your cluster as a standard Deployment. It usually has two or more replicas for high availability.

Its job is simple: Watch and Cache.

It watches the Kubernetes API for any new Services or EndpointSlices.
Whenever you create/update a Service, CoreDNS updates its internal memory cache almost instantly.

How your Pods "Know" to ask CoreDNS

When a Pod is created, the Kubelet injects a specific configuration into the Pod's /etc/resolv.conf file.

If you exec into any Pod and run cat /etc/resolv.conf, you’ll see something like:

nameserver 10.96.0.10  # This is the IP of the CoreDNS Service
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Because of this file, every time your app tries to connect to a URL, it doesn't go to the public internet first—it asks CoreDNS inside the cluster.

Scenarios

Scenario A: Internal Query
You call http://my-service. CoreDNS looks at its cache, finds the ClusterIP, and hands it back. Done.

Scenario B: External Query (e.g., www.google.com)
CoreDNS checks its cache and says, "I don't know who that is." Instead of giving up, it forwards the request to the upstream DNS (the default DNS configured on the worker node).

The Corefile (Configuration)

The behavior of CoreDNS is defined in a ConfigMap called the Corefile. It’s mounted directly into the CoreDNS pods. It looks something like this:

.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

Key parts of this config:

kubernetes: This plugin tells CoreDNS to answer DNS queries based on IP addresses of Kubernetes pods and services.
forward . /etc/resolv.conf: This is the "What if?" logic. If it's not a K8s service, send it to the node's DNS.
cache: Keeps things fast so we don't hit the API server for every single request.

Summary

Kubelet : It writes the CoreDNS IP into every Pod's /etc/resolv.conf so they know where to ask.

CoreDNS : It watches the K8s API to map Service names to IPs and forwards everything else to the Node's DNS.

How Kubernetes Hands Out IPs

Manish Pillai — Sat, 04 Apr 2026 05:59:48 +0000

If you've ever looked at a Kubernetes cluster and wondered why Pods and Services get the IPs they do, you’ve probably bumped into the term CIDR.

It sounds intimidating, but it’s actually just a clever way to keep the "address book" of your cluster organized. Let’s break it down like humans.

First off: What is CIDR?

Think of CIDR as a way to define a territory of IP addresses without having to list every single one.

Example: 10.244.0.0/16
Instead of saying, "I have 65,536 addresses from 10.244.0.0 to 10.244.255.255," we just use that one little string.

The number after the slash tells you how much of the address is "locked" (the network) and how much is "free real estate" (the hosts).

/16 = A massive range (Lots of room for nodes).
/24 = A smaller chunk (Perfect for a single node).

Why this matters in Kubernetes

When you spin up a cluster, it gets a big "bucket" of IPs (a /16). But a cluster has multiple nodes. To keep things from getting messy, Kubernetes splits that big bucket into smaller bowls (/24) for each node.

The Hierarchy:

Cluster Level: 10.244.0.0/16
- Node 1: 10.244.1.0/24
- Node 2: 10.244.2.0/24

The Two Types of CIDR

A healthy cluster usually manages two separate "buckets":

Pod CIDR: For your actual containers.
Service CIDR: For the stable entry points (Services).

How Pod IPs are assigned

When a node joins your cluster, the kube-controller-manager says, "Welcome! Here is your personal slice of the IP pie."

The Split: The controller assigns a small range (like a /24) to that specific node.
The Creation: You deploy a Pod.
The CNI (Calico/Flannel): The CNI plugin looks at the node's assigned range and grabs a free IP.
- Result: 10.244.1.5

Because every node has its own unique range, Pod IPs never overlap. No collisions.

How Service IPs are assigned

Service IPs are a bit different. They don't care about nodes because Services are virtual.

The Request: You create a Service.
The API Server: The kube-apiserver looks at the global Service CIDR bucket.
The Assignment: It picks an IP (e.g., 10.96.0.10) and stamps it on the Service.

You won't find this IP on any actual hardware interface. It lives purely in the cluster's iptables/IPVS.

The "Why": Why different components?

This used to confuse me. Why does the Controller Manager handle Pods, but the API Server handles Services?

Pod CIDRs are dynamic: Nodes come and go. We need a "Controller" to constantly watch the cluster and hand out ranges to new nodes.
Service CIDRs are static: It’s just a flat pool of IPs. The API Server can just grab the next available one from the list while it's processing your YAML.

Summary

Pod IP: Assigned by CNI from the Node’s specific slice.
Service IP: Assigned by the API Server from the Cluster’s global pool.

How Kubernetes Maps Service IP to Pods

Manish Pillai — Fri, 03 Apr 2026 07:05:44 +0000

When you first learn Kubernetes, you hear:

“Pods talk to each other using Services.”

It sounds simple, but the Service IP doesn't actually exist on any physical interface.

The Problem: Pods are Ephemeral

Pods are temporary. If you delete a Pod and it is recreated, it gets a new IP address.

Frontend calls Backend via Pod IP: 10.244.1.5
Backend restarts: The old IP no longer exists.
Result: Connection failure. ❌

The Solution: Services

Kubernetes provides Services to act as a stable entry point. They offer:

Stable IP (ClusterIP)
DNS name
Load balancing

Instead of calling a specific Pod, the frontend calls the backend-service.

The Catch: The "Ghost" IP

A Service IP (e.g., 10.96.0.10) is:

Not assigned to any Pod.
Not owned by any Node.
Not visible in ifconfig or ip addr.

How it Works: Data vs. Routing

1. Finding the Pods

When you define a Service with a selector, Kubernetes creates an EndpointSlice. This is essentially a list of the actual Pod IPs behind that Service.

Note: This is just data stored in the API Server. It doesn't route anything yet.

2. `kube-proxy` Steps In

kube-proxy runs on every node as a DaemonSet. Its job is to watch the API Server for new Services and Endpoints and translate them into local networking rules.

3. Creating the Rules (iptables)

kube-proxy uses iptables (a Linux kernel feature) to intercept packets. It sets up a chain of rules:

Match: "Is this packet going to Service IP 10.96.0.10?"
Select: "Pick one of the available Pod IPs from the list."
Rewrite (DNAT): Change the destination IP from the Service IP to the chosen Pod IP.

The Request Flow

When a request is sent:

Client/Pod sends a packet to the Service IP.
Linux Kernel checks the iptables rules (installed by kube-proxy).
DNAT happens: The destination is rewritten to a Pod IP.
The packet is routed to the actual Pod.

`kube-proxy` Modes

While the concept remains the same, the efficiency of how rules are matched varies:

Mode	Technology	Performance	Scaling
iptables	Linux rule list	Standard	Slower as you add thousands of services
IPVS	Linux Load Balancer	High	Very fast; uses hash tables for lookups
Userspace	`kube-proxy` process	Low	Slow (Old, no longer used)

Key Takeaway

The name kube-proxy is confusing because, in modern Kubernetes, it is not a proxy. It doesn't sit in the middle of your traffic.

kube-proxy runs on each node and manages the networking rules on each node to enable Service discovery and load balancing.
The Linux Kernel is the Data Plane that does the actual heavy lifting of routing packets (using iptables or IPVS).

Remember: kube-proxy sets things up; the kernel does the real work.

Understanding HOTP and TOTP in Two-Factor Authentication

Manish Pillai — Sun, 28 Sep 2025 17:57:37 +0000

In most websites today, security doesn’t stop at just a username and password. To add an extra layer of protection against phishing and unauthorized access, Two-Factor Authentication (2FA) is commonly used.

There are different ways to implement 2FA:

SMS OTPs sent to your mobile
Push notifications (e.g., GitHub’s mobile approval)
Authenticator apps like Google Authenticator, Microsoft Authenticator, or Authy

Among these, authenticator apps are the most widely used. They generate one-time passwords (OTPs) that refresh either:

On demand (HOTP – Hash-based OTP), or
After a fixed period (TOTP – Time-based OTP).

Example: QR Code + Authenticator App

When you enable 2FA on a website, you’re usually presented with a QR code to scan in your authenticator app:

The QR code encodes a special URI in the following format:

otpauth://totp/Issuer:Account?secret=BASE32SECRET&issuer=IssuerName&algorithm=SHA1&digits=6&period=30

Key parts:

otpauth://totp → scheme (could also be hotp)
Issuer:Account → helps you identify which service/account the OTP belongs to
secret → Base32 encoded secret key shared between server and your app
algorithm → hashing algorithm (usually SHA-1, sometimes SHA-256/512)
digits → OTP length (6 or 8)
period → validity window in seconds (default: 30s for TOTP)

Your authenticator app extracts these details and starts generating OTPs automatically.

HOTP – Hash-based One-Time Password

HOTP is counter-based. Both the server and your authenticator app maintain:

A secret key (shared during setup)
A counter (incremented each time an OTP is generated)

Flow diagram:

Steps:

Combine the secret key and counter.
Hash them using the chosen algorithm (usually SHA-1), producing a 20-byte HMAC code.
Apply dynamic truncation to extract a 6–8 digit OTP.
The generated OTP is sent to the server, which independently performs the same calculation to verify it.

Note: Each byte is 8 bits, so a 20-byte HMAC gives a 160-bit code.

TOTP – Time-based One-Time Password

TOTP is time-based. Instead of a counter, it uses the current timestamp:
0 The server and client compute a time step (e.g., floor(currentUnixTime / period))

The time step is treated as the counter in the HOTP algorithm

Flow diagram:

Steps:

Compute the time step C = floor(currentUnixTime / period).
Combine the secret key and time step, then hash with HMAC.
Apply dynamic truncation to get the OTP.
OTP is valid for the selected time period (e.g., 30 seconds).

This allows both server and client to generate the same OTP independently, as long as their clocks are synchronized.

Dynamic Truncation

Dynamic truncation is a common step in both HOTP and TOTP:

Steps:

Take the last byte of the HMAC result and perform a bitwise AND with 0x0F (decimal 15) to calculate an offset.
Select 4 bytes starting from the offset.
Convert these 4 bytes to a 31-bit integer (ignore the sign bit).
Apply modulo 10^d (where d is the number of digits, usually 6 or 8) to get the final OTP.

This ensures the OTP is a fixed-length numeric code, even though the HMAC is 20 bytes

Differences: HOTP vs TOTP

Feature	HOTP	TOTP
Counter/Time	Incremented counter	Time step (current time / period)
Validity	Valid until used	Valid only for the time period
Sync Requirement	Counter sync required	Clock sync required
Use Case	Hardware tokens, banking apps	Mobile authenticator apps, web apps

Additional Notes

Always use SHA-1 or stronger algorithms (from server POV).
For TOTP, if the client and server clocks drift, the OTP may fail. - Many servers implement a window of 1–2 time steps to allow minor clock differences.
HOTP is mostly used for hardware tokens, while TOTP is widely used in mobile apps.
Avoid SMS-based OTPs where possible due to SIM swap attacks.

Check out slide

Check out video

SSE - Server Sent Event

Manish Pillai — Wed, 24 Sep 2025 03:36:03 +0000

If you’ve ever wanted your server to push live updates to a browser without the client constantly polling, Server-Sent Events (SSE) are a simple and efficient solution. In this post, we’ll explore what SSE is, how to implement it in Golang, and why the client side requires EventSource.

I’ve also made a short video demo to show it in action, which you can check out below.

What is SSE?

SSE is a standard that allows a server to send continuous updates over a single HTTP connection to the client. Unlike WebSockets, SSE is unidirectional—the server sends data, but the client cannot push messages back on the same connection.

Common use cases:

Live dashboards and monitoring systems.
Chat notifications or social feed updates.
Real-time logs or stock price tickers.

Creating an SSE Server in Go

Here’s how to implement a basic SSE server using Go:

package main

import (
    "fmt"
    "net/http"
    "time"

    "github.com/gorilla/mux"
)

func main() {
    router := mux.NewRouter()
    server := http.Server{
        Addr:    ":8080",
        Handler: router,
    }

    router.HandleFunc("/sse", handleEvents).Methods("GET")

    fmt.Println("SSE server running on :8080")
    err := server.ListenAndServe()
    if err != nil {
        panic(err)
    }
}

func handleEvents(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Access-Control-Allow-Origin", "*")
    w.Header().Set("Content-Type", "text/event-stream")
    w.Header().Set("Cache-Control", "no-cache")
    w.Header().Set("Connection", "keep-alive")

    ticker := time.NewTicker(2 * time.Second)
    defer ticker.Stop()

    for {
        select {
        case <-ticker.C:
            fmt.Fprintf(w, "data: %s\n\n", time.Now().String())
            if f, ok := w.(http.Flusher); ok { f.Flush() }
        case <-r.Context().Done():
            return
        }
    }
}

Key points:

Content-Type: text/event-stream is mandatory.
Each message must start with data: and end with a double newline \n\n.
Use http.Flusher to push data immediately without buffering.

The Client Side – EventSource

SSE is designed for browsers, which support a built-in EventSource API. This makes it very simple to receive server events:

<script>
const es = new EventSource("http://localhost:8080/sse");

es.onopen = () => console.log("SSE connected");
es.onmessage = e => console.log("message:", e.data);
es.onerror = e => console.error("SSE error", e);
</script>

EventSource automatically reconnects if the connection drops.
It parses data: messages sent by the server.
This is why the client usually needs a browser or JS runtime that supports EventSource.

SSE vs Sockets

While SSE is perfect for server → client streaming, for bidirectional communication where the client also sends messages to the server in real time, WebSockets are more suitable.

SSE:

Unidirectional (server → client)
Works over HTTP/HTTPS
Simpler to implement

WebSockets:

Bidirectional
More complex but flexible
Great for chat apps, multiplayer games, and live collaboration tools

Quick Note on Streaming HTTP

SSE is a type of streaming HTTP, where the server sends data in chunks without closing the connection. Streaming HTTP itself is not limited to browsers and can be used in server-to-server communication as well.

Check Out the Video Demo

I’ve created a short video demonstrating how to implement SSE in Go, including the server code and how EventSource handles the data in a browser.

Conclusion:
SSE is a lightweight and efficient way to push live updates from servers to browsers. It’s simpler than WebSockets when you only need server-to-client communication, making it ideal for dashboards, logs, and notifications.

No Code Admin Panel Platform

Manish Pillai — Mon, 22 Sep 2025 18:34:52 +0000

𝐍𝐨-𝐂𝐨𝐝𝐞 𝐀𝐝𝐦𝐢𝐧 𝐏𝐚𝐧𝐞𝐥 𝐏𝐥𝐚𝐭𝐟𝐨𝐫𝐦

𝐖𝐡𝐚𝐭 𝐢𝐭 𝐝𝐨𝐞𝐬: Automates the creation of admin panels using a simple drag-and-drop interface, eliminating the need to build them from scratch for each client.

𝐁𝐚𝐜𝐤𝐞𝐧𝐝: Built entirely by me in GoLang.

𝐅𝐫𝐨𝐧𝐭𝐞𝐧𝐝: The UI was generated with the help of AI.

𝐊𝐞𝐲 𝐅𝐞𝐚𝐭𝐮𝐫𝐞𝐬:
⇨ Creates tables with primary and foreign key relations.
⇨ Manages users and their access requests.
⇨ Supports CRUD (Create, Read, Update, Delete) permissions on data.

𝐅𝐮𝐭𝐮𝐫𝐞 𝐏𝐥𝐚𝐧𝐬: I'm working on adding customizable role-based permissions and improving the UI. I would be happy, if anyone wants to collab on frontend.

Key Management Service in Kubernetes — Part 2

Manish Pillai — Sun, 01 Jun 2025 17:55:26 +0000

Welcome back to our series on Key Management Service (KMS) in Kubernetes! In Part 1, we laid the groundwork; now, in Part 2, we're diving into the critical concept of encryption at rest.

What Exactly is Encryption at Rest?

Simply put, encryption at rest in Kubernetes refers to how the API server encrypts data before storing it in etcd. Think of etcd as the brain of your Kubernetes cluster - it's where all your cluster's configuration data, state, and secrets live.

By default, the Kubernetes API server stores resources in etcd as plain text. This means if someone gains unauthorized access to your etcd, they can read all your sensitive data, including secrets, without any effort. This is a significant security risk.

While encryption at rest applies to any Kubernetes resource, in this series, we'll continue to focus on Secrets due to their inherently sensitive nature.

The good news is Kubernetes provides a way to encrypt this data before it hits etcd. This is primarily done through the --encryption-provider-config argument passed to the kube-apiserver process, which points to a configuration file.

Introducing EncryptionConfiguration

In Kubernetes, the encryption at rest behavior is configured using an EncryptionConfiguration resource. This powerful configuration allows you to specify which resources should be encrypted and using which encryption providers.

Let's look at an example configuration to understand its structure:

# CAUTION: This is an example configuration and should NOT be used for production clusters without careful consideration.
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
      - pandas.awesome.bears.example # An example custom resource API
    providers:
      # The 'identity' provider stores resources as plain text (no encryption).
      # If listed first, it means data is NOT encrypted.
      - identity: {}
      - aesgcm:
          keys:
            - name: key1
              secret: c2VjcmV0IGlzIHNlY3VyZQ== # Base64 encoded key
            - name: key2
              secret: dGhpcyBpcyBwYXNzd29yZA== # Base64 encoded key
      - aescbc:
          keys:
            - name: key1
              secret: c2VjcmV0IGlzIHNlY3VyZQ== # Base64 encoded key
            - name: key2
              secret: dGhpcyBpcyBwYXNzd29yZA== # Base64 encoded key
      - secretbox:
          keys:
            - name: key1
              secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY= # Base64 encoded key
  - resources:
      - events
    providers:
      - identity: {} # Do not encrypt Events
  - resources:
      - '*.apps' # Wildcard match (Kubernetes 1.27+)
    providers:
      - aescbc:
          keys:
          - name: key2
            secret: c2VjcmV0IGlzIHNlY3VyZSwgb3IgaXMgYXQ/Cg==
  - resources:
      - '*.*' # Wildcard match (Kubernetes 1.27+)
    providers:
      - aescbc:
          keys:
          - name: key3
            secret: c2VjcmV0IGlzIHNlY3VyZSwgSSB0aGluaw==

Key Takeaways from the EncryptionConfiguration:

resources: This array specifies which Kubernetes resources you want to apply encryption to. You can target specific resources (e.g., secrets), or use wildcard matches (e.g., '*.apps', '*.*') available in Kubernetes 1.27 and later.
providers: This is an ordered list of encryption providers. Kubernetes attempts to use the first provider in the list to encrypt new data. When decrypting, it tries providers in order until one succeeds.
identity: {}: This provider means no encryption; data is stored in plain text. If this is the first provider for a resource, it means new data for that resource will not be encrypted.
aesgcm, aescbc, secretbox: These are different encryption algorithms. You define named keys (base64 encoded) for each.

Demo: Encrypting Secrets in Minikube

Let's walk through a practical example using a Minikube cluster to see encryption at rest in action.

1. Start Your Minikube Cluster:
Make sure your Minikube cluster is up and running.

2. Create the EncryptionConfiguration File:
Create a file named encryption-conf.yaml with the following content:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets # We are only encrypting secrets for this demo
    providers:
      - aescbc: # Using AES-CBC encryption
          keys:
            - name: key1
              # IMPORTANT: This is a randomly generated key.
              # Use a strong, unique, and base64-encoded key in production.
              secret: lvAp17Ae2o/yTdxz2qyC6zjVzuS+sBdhkwCccgsSsUg=
      - identity: {} # Fallback to identity (plaintext) if AES-CBC fails, or for older data

Understanding the Demo Configuration:

We're specifically targeting secrets for encryption.
We're using the aescbc provider with a single key named key1. Remember to use a truly random and strong key for production environments! The secret value needs to be base64 encoded.
The identity: {} provider is included as a fallback. This is crucial for smooth rotation and decryption of older data, or if the primary encryption provider encounters an issue.

3. Configure the API Server to Use the Encryption Configuration:
Now, we need to tell the kube-apiserver to use this configuration file. In Minikube, you can achieve this by modifying the API server's Pod definition.

First, locate the kube-apiserver pod definition (it's usually in /etc/kubernetes/manifests/ on the control plane node).

Next, you need to modify the kube-apiserver static pod manifest to include the --encryption-provider-config argument and mount the directory containing your encryption-conf.yaml file.

Here's an example of how the kube-apiserver pod manifest snippet would look after modification:

apiVersion: v1
kind: Pod
metadata:
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    # ... other kube-apiserver arguments
    - --encryption-provider-config=/etc/kubernetes/enc/encryption-conf.yaml # <--- Add this line
    volumeMounts:
    - mountPath: /etc/kubernetes/enc # <--- Mount path for your config file
      name: enc-vol
      readOnly: true
    # ... other volume mounts
  volumes:
  - hostPath:
      path: /etc/kubernetes/encryption # <--- Host path where your config file lives
      type: DirectoryOrCreate
    name: enc-vol
  # ... other volumes

Important: You will need to place your encryption-conf.yaml file in the /etc/kubernetes/encryption directory on your Minikube VM (or the control plane node for a full cluster) for the hostPath volume mount to work correctly.

After saving the changes, the kube-apiserver pod will automatically restart, applying the new encryption configuration.

4. Create a Secret:
Let's create a new secret in a test namespace:

kubectl create namespace test
kubectl -n=test create secret generic new-secret --from-literal=key1=supersecret

5. Retrieve the Secret (as perceived by kubectl):
Now, let's retrieve the secret using kubectl:

kubectl get secret new-secret -o yaml -n test

You'll get output similar to this:

apiVersion: v1
data:
  key1: c3VwZXJzZWNyZXQ= # Still base64 encoded!
kind: Secret
metadata:
  creationTimestamp: "2025-05-31T10:50:26Z"
  name: new-secret
  namespace: test
  resourceVersion: "6614"
  uid: a8655bde-be5f-4624-b905-61524de56ebe
type: Opaque

Notice that the data field still shows a base64-encoded value. This is crucial: kubectl always displays secrets in their base64-encoded form. This output doesn't tell us whether the secret is encrypted at rest in etcd.

6. Verify Encryption in Etcd:
This is where the magic happens! We'll directly inspect how the secret is stored in etcd.
First, exec into the etcd-minikube pod:

kubectl -n=kube-system exec -it etcd-minikube -- sh

Then, use etcdctl to retrieve the raw value of the secret from etcd:

ETCDCTL_API=3 etcdctl --cacert /var/lib/minikube/certs/etcd/ca.crt --cert /var/lib/minikube/certs/etcd/server.crt --key /var/lib/minikube/certs/etcd/server.key get /registry/secrets/test/new-secret

You'll see output that looks something like this:

/registry/secrets/test/new-secret
k8s:enc:aescbc:v1:key1:??|??<z@0-ki_<Q?
                                       jғ?[3Ox??t%??q??d??e??%?gy??ER8{s????
                                                                            ?r?UO%{?+C??h
                                                                                         ???w?II?; ??v??r????q?????t?YA?"??j?" ??f9?$FD?9T?.F?\?<???kc/Q?W0?
                                                                                                                                                                                   ?;\U??l?n???nW?^HlA?C?Ռ]=?U{j??|??pe?
?Z?Y?XY9??
          ?uuO?2??xK[޹??~7v

This is the key difference!

Without EncryptionConfiguration, the data in etcd would be easily readable.
With EncryptionConfiguration applied, you see an unreadable, garbled string prefixed with k8s:enc:aescbc:v1:key1:.

This prefix tells you:

k8s:enc: This data is encrypted by Kubernetes.
aescbc: The encryption algorithm used is AES-CBC.
v1: The version of the encryption scheme.
key1: The specific key (key1 from our EncryptionConfiguration) that was used for encryption.

This confirms that your secret is now encrypted at rest in etcd!

Key Management Service in Kubernetes — Part 1

Manish Pillai — Sat, 17 May 2025 16:28:05 +0000

Key Management Service (KMS) is a way to manage your secrets in a more secure manner. But before diving into KMS, let’s do a quick primer on Kubernetes Secrets.

What Are Kubernetes Secrets?

A Secret is any sensitive information—such as a database password, an API token, or cloud credentials. In most applications, you separate such configuration data from the actual application logic.

That’s where Kubernetes Secrets come in. You can store confidential configuration as a separate Kubernetes resource called a Secret.

Where Are Secrets Stored?

Kubernetes stores secrets in etcd, which is the key-value store used by the Kubernetes control plane. Unless encryption at rest is enabled, the secrets are stored in plaintext in etcd—more on this later.

How to Create Kubernetes Secrets?

Let’s walk through three ways to create secrets in Kubernetes. For reference, I’m using an OpenShift cluster with the oc CLI, but everything here applies to kubectl as well.

Create a Namespace

% oc new-project kms

a. Creating a Secret from a File

% echo -n 'somepassword' > password.txt
% oc create secret generic kms-file-secret \
  --from-file=password=password.txt

In the first line, we create a file password.txt with the contents somepassword. The second command creates a secret named kms-file-secret by reading the contents of that file.

To inspect the created secret:

% oc get secret kms-file-secret -o yaml

You'll see:

% oc get secret kms-file-secret  -o yaml
apiVersion: v1
data:
  password: c29tZXBhc3N3b3Jk
kind: Secret
metadata:
  creationTimestamp: "2025-05-16T13:35:30Z"
  name: kms-file-secret
  namespace: kms
  resourceVersion: "807159"
  uid: 66691c51-4a6c-4a15-a6b1-1d2de6d8fff7
type: Opaque

That string c29tZXBhc3N3b3Jk is just a base64 encoded version of somepassword.

b. Creating a Secret from a Literal Value

% oc create secret generic kms-literal-secret \
  --from-literal=password=somepassword

This creates a secret by passing the value directly as a literal string. The result is the same base64 encoded password in the data field:

% oc get secret kms-literal-secret -o yaml 
apiVersion: v1
data:
  password: c29tZXBhc3N3b3Jk
kind: Secret
metadata:
  creationTimestamp: "2025-05-16T14:05:34Z"
  name: kms-literal-secret
  namespace: kms
  resourceVersion: "819400"
  uid: d4ecd109-cd95-4afe-b2e8-77b2da3bcfca
type: Opaque

c. Creating a Secret Using a Manifest File

You can also define secrets in YAML files using either the data or stringData fields.

stringData: human-readable strings (Kubernetes will encode them to base64)
data: requires values to already be base64 encoded

Let’s go with the data field here.

% echo -n 'somepassword' | base64
c29tZXBhc3N3b3Jk

Now use that in a manifest:

% oc apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: kms-manifest-secret
  namespace: kms
type: Opaque
data:
  password: c29tZXBhc3N3b3Jk
EOF

Verifying:

% oc get secret kms-manifest-secret -o yaml    
apiVersion: v1
data:
  password: c29tZXBhc3N3b3Jk
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"password":"c29tZXBhc3N3b3Jk"},"kind":"Secret","metadata":{"annotations":{},"name":"kms-manifest-secret","namespace":"kms"},"type":"Opaque"}
  creationTimestamp: "2025-05-16T14:24:58Z"
  name: kms-manifest-secret
  namespace: kms
  resourceVersion: "827156"
  uid: ebb157dd-c493-47af-94c4-ba5d96ff9a29
type: Opaque

Why Use Kubernetes Secrets?

Security: Better than storing sensitive info in ConfigMaps (which are plaintext).
Isolation: Keeps secrets separate from your application code.
Portability: Easily consumed by multiple Pods using environment variables or mounted volumes.

But… What’s the Problem?

At first glance, the secret looks like it’s encrypted due to its unreadable format:

data:
  password: c29tZXBhc3N3b3Jk

But that’s just base64 encoding—it’s easily reversible:

% oc get secret kms-file-secret -o jsonpath='{.data.password}' | base64 --decode
somepassword

So, if RBAC policies are misconfigured or someone has unauthorized read access to secrets, they can easily extract sensitive data. Not good.

So, What’s the Solution?

To make secrets more secure, Kubernetes offers multiple enhancements:

Encryption at rest: Secrets are encrypted before being stored in etcd.
KMS providers: Use cloud-based or external Key Management Service to encrypt secrets (e.g., AWS KMS, Azure Key Vault, HashiCorp Vault).
Secrets Store CSI Driver: Mount secrets from external providers directly into Pods.

We’ll dive into these solutions in Part 2 of the series.

From VMs to Unikernels: The Evolution of Application Deployment

Manish Pillai — Sun, 13 Apr 2025 18:52:38 +0000

While many millennials and Gen-Z engineers haven't witnessed the full journey of computing evolution, it's fascinating to explore how hardware and software abstraction has transformed over the years — especially with the rise of the cloud.

One of the most impactful areas of this evolution is in how we deploy and run services: through virtualization, containerization, and more recently, unikernels

Virtualization

Virtualization is the practice of abstracting physical hardware to run multiple isolated environments (virtual machines) on a single physical host. This is typically achieved through a hypervisor — such as VMware ESXi, KVM, Xen, or Hyper-V — which manages and allocates resources to each VM.

Let’s say you have two services: Service A and Service B.

To deploy them using virtualization:

You create two separate VMs.
Each VM runs its own guest OS and kernel.
You deploy one service per VM.

Great! Your services are isolated and running.

But here's the catch...

Each VM carries a full OS stack, leading to duplicate resource usage, longer boot times, and heavier system overhead.

Containerization

Containerization — popularized by Docker — addressed these inefficiencies. Instead of virtualizing the entire hardware stack, containers share the host OS kernel, isolating applications only at the process level using namespaces and cgroups.

With containers:

Service A and B run as isolated containers.
They share the same kernel but maintain isolated user spaces.
Containers are lighter, faster to start, and use less memory and disk.

Boot times are significantly reduced, and resource utilization improves dramatically.

However, containers still rely on the underlying host OS and a container runtime (e.g., Docker Engine or containerd), which introduces some attack surface and runtime dependencies.

Unikernels

Unikernels take minimalism to the next level.

A unikernel is a single-purpose, single-address-space image that includes:

Only the required parts of the OS.
The application logic.
No shells, no package managers, no unused ports.

This makes them ultra-secure (smaller attack surface), blazing fast to boot, and extremely lightweight.

Inside Unikernels

Traditional Operating System Architecture

In traditional operating systems (like Linux or Windows), the system is divided into two separate address spaces:

User Space:
- This is where user applications (like your web browser, database, or backend services) run.
- It includes user-level libraries that the app uses to interact with the system.
- However, it doesn't have direct access to hardware or low-level resources for safety and security.
Kernel Space:
- This is the core of the OS and has full control over the system's hardware.
- It includes essential components like the process scheduler, memory manager, networking stack, and device drivers.
- All user applications must make system calls to request services from the kernel (like file I/O, network access, etc.).

So essentially, applications (user space) rely on the kernel(kernel space) to function. This separation enforces security, stability, and resource control — but it also adds layers of abstraction and overhead.

Unikernel Architecture

In the unikernel model, things are drastically simplified.

There is no separation between user space and kernel space — instead, both the application and only the essential parts of the OS are compiled into a single binary that runs directly on hardware or a hypervisor.

Here's what makes it unique:

Single-purpose: A unikernel is built to run just one application — nothing more.
App + OS as one: The application is bundled with exactly the OS functionalities it needs (e.g., TCP/IP stack, filesystem, scheduler).
No shell, no package manager, no general-purpose OS features — which also means:
- Reduced attack surface
- Fast boot time
- Smaller memory and disk footprint

Because everything exists in a single address space, function calls (even system-level ones) are just regular function calls — no costly system calls or context switches

Unikernel Providers

Here are a few notable unikernel implementations:

MirageOS – Functional, OCaml-based unikernel.
IncludeOS – C++ based unikernel.
OSv – Designed to run single-application workloads (Java, etc.).
NanoVMs – Commercial unikernel platform for production deployment.

Reference:

My Learnings About Etcd

Manish Pillai — Fri, 11 Apr 2025 03:35:41 +0000

This is my first ever technical blog, so do correct if I am wrong, not so technically strong right now, just sharing my learnings.

Etcd is a distributed key-value store, somewhat like Redis, but it operates quite differently under the hood (more on this later).
It's implemented in Golang and is fully open-source.

While etcd can be paired with many systems, its most prominent use case is in Kubernetes, where it's a critical component of the control plane.

How is Etcd used in Kubernetes?

If you're familiar with Kubernetes architecture, you know it consists of control plane components and worker nodes. The control plane is responsible for managing the overall cluster state - including scheduling, maintaining desired state, responding to cluster events, and more.

But where does Kubernetes store all of its metadata? Things like:

Pod definitions
Deployment states
Configuration data
Secrets and ConfigMaps
Cluster state

That's where etcd comes into the picture.

All cluster data is stored in etcd in a key-value format. Whenever the kube-api-server needs to fetch or persist cluster state, it communicates directly with etcd.

Etcd can either be:

Embedded as part of the control plane (commonly deployed alongside kube-apiserver)
Or hosted as a separate, external cluster, often in high-availability production environments.

Is Etcd Distributed?

Yes, etcd is a distributed system designed for fault-tolerance and high availability. You can run multiple instances (etcd nodes or members) in a cluster.

To maintain consistency across nodes, etcd uses the RAFT consensus algorithm.

A Quick Look at the RAFT Algorithm

The RAFT consensus algorithm ensures that the etcd cluster agrees on the current state, even in the presence of failures.

Here's how it works:

Among all nodes, one is elected as the leader.
The leader handles all client requests (writes) and replicates changes to follower nodes.
If the leader goes down, a new leader is automatically elected from the followers.

This ensures strong consistency, meaning all clients see the same data regardless of which node they connect to.

ScyllaDB, a distributed NoSQL database also uses the RAFT algorithm for leader elections. Refer link.

Storage Engine and Data Model - How Etcd Stores Data

Just like many traditional databases use a storage engine to handle how data is written to disk, etcd does the same.
For example:

MySQL uses InnoDB
SQLite has its own built-in storage engine
MongoDB uses WiredTiger

Each storage engine follows a different data structure design - like B+ Trees (great for read-heavy operations) or LSM Trees (optimized for write-heavy workloads).

So what does etcd use?
Etcd uses a storage engine called BoltDB (specifically, a fork called bbolt).

BoltDB is a B+ Tree-based key-value store that persists data to disk and provides excellent support for consistent and predictable reads, which aligns perfectly with etcd's goal of being a strongly-consistent store for configuration data.

You can read more details here.

How Etcd Stores Data (and How It's Different from Redis)

Etcd stores data using a B+ Tree–based storage engine called bbolt (a fork of BoltDB), which writes data persistently to disk. Unlike Redis, which primarily keeps data in memory for lightning-fast access (and optionally persists it), etcd is designed for strong consistency and durability, even across restarts or crashes.

It uses a multi-version concurrency control (MVCC) model, where every update creates a new revision instead of modifying data in-place.
This allows etcd to support features like watching changes, accessing historical versions, and time-travel queries - all while keeping disk usage optimized using compaction.

This makes etcd an ideal choice for systems like Kubernetes where data integrity and change tracking are more critical than raw speed.

You can read more about the differences here.

So, if it stores all the history, won't the disk/memory gets full?

Etcd periodically performs compaction, which cleans up old revisions and reduces disk usage while keeping recent history intact.

That's all I've learned about Etcd so far. It might not be perfect or super in-depth, but I feel like I now have a decent understanding of how things work under the hood. I'm still exploring and learning about it - and this is just the beginning.

Forem: Manish Pillai

Why we need Ingress in Kubernetes?

The Problem with the "Standard" way

The Solution: Ingress

How do we access Ingress from the Outside?

Practical Example: Routing & SSL

Pros

How Kubernetes Resolves Service DNS

What is CoreDNS?

How your Pods "Know" to ask CoreDNS

Scenarios

The Corefile (Configuration)

Summary

How Kubernetes Hands Out IPs

First off: What is CIDR?

Why this matters in Kubernetes

The Two Types of CIDR

How Pod IPs are assigned

How Service IPs are assigned

The "Why": Why different components?

Summary

How Kubernetes Maps Service IP to Pods

The Problem: Pods are Ephemeral

The Solution: Services

The Catch: The "Ghost" IP

How it Works: Data vs. Routing

1. Finding the Pods

2. kube-proxy Steps In

3. Creating the Rules (iptables)

The Request Flow

kube-proxy Modes

Key Takeaway

Understanding HOTP and TOTP in Two-Factor Authentication

Example: QR Code + Authenticator App

HOTP – Hash-based One-Time Password

TOTP – Time-based One-Time Password

Dynamic Truncation

Differences: HOTP vs TOTP

Additional Notes

SSE - Server Sent Event

No Code Admin Panel Platform

Key Management Service in Kubernetes — Part 2

What Exactly is Encryption at Rest?

Introducing EncryptionConfiguration

Demo: Encrypting Secrets in Minikube

Key Management Service in Kubernetes — Part 1

What Are Kubernetes Secrets?

Where Are Secrets Stored?

How to Create Kubernetes Secrets?

Create a Namespace

a. Creating a Secret from a File

b. Creating a Secret from a Literal Value

c. Creating a Secret Using a Manifest File

Why Use Kubernetes Secrets?

But… What’s the Problem?

So, What’s the Solution?

From VMs to Unikernels: The Evolution of Application Deployment

Virtualization

Containerization

Unikernels

Inside Unikernels

Traditional Operating System Architecture

Unikernel Architecture

Unikernel Providers

My Learnings About Etcd

How is Etcd used in Kubernetes?

Is Etcd Distributed?

A Quick Look at the RAFT Algorithm

Storage Engine and Data Model - How Etcd Stores Data

How Etcd Stores Data (and How It's Different from Redis)

So, if it stores all the history, won't the disk/memory gets full?

2. `kube-proxy` Steps In

`kube-proxy` Modes