Forem: Mikhail [azalio] Petrov

Kubernetes Agent Blind to New Mounts? Demystifying Mount Propagation

Mikhail [azalio] Petrov — Wed, 23 Apr 2025 20:43:54 +0000

How `mountPropagation: HostToContainer` leverages Linux namespaces to solve a common agent problem

Ever run into this frustrating scenario? You deploy a Kubernetes agent (like an I/O limiter, monitoring tool, or operator) that needs to interact with PersistentVolumes mounted by Kubelet for other pods. It works fine initially, seeing all existing mounts. But then, a new pod gets scheduled, its PV gets mounted by Kubelet... and your agent is completely blind to it! 😠 Restarting the agent fixes it, but that's just a workaround, not a solution.

I recently wrestled with this exact issue while building an I/O limiter that needed device MAJOR:MINOR numbers from pod mounts. Let's dive into why this happens and how to fix it properly using Kubernetes mountPropagation.

The Root Cause: Linux Mount Namespaces

The core of the issue lies in Linux mount namespaces. By default, each container in Kubernetes gets its own isolated mount namespace. Think of it as a private view of the system's mount points. When a container starts, it gets a copy of the host's mount table at that moment.

Crucially, subsequent mount events happening on the host (like Kubelet mounting a new PV under /var/lib/kubelet/pods/) are not automatically reflected inside the container's isolated namespace. This default behavior corresponds to private propagation in Linux terminology.

In Kubernetes, this default isolation is represented by:

mountPropagation: None # (Default if not specified)

This isolation is great for security and preventing containers from interfering with each other or the host, but it causes the "blind agent" problem.

The Solution: Kubernetes `mountPropagation`

Kubernetes provides the mountPropagation field within a volumeMount definition to control how mount events are shared between the host and the container's namespace. It bridges the isolation gap when needed. There are three modes:

None (Default): Complete isolation. No mount events propagate in either direction. (Linux: private)
HostToContainer: One-way street! Mount events from the host propagate into the container. Mounts created inside the container do not propagate back to the host. (Linux: rslave - recursive slave). This is exactly what we need for our agent! 👍
Bidirectional: Two-way street. Mounts propagate from host to container AND from container back to the host (and other containers in the pod using Bidirectional). (Linux: rshared - recursive shared). Use with extreme caution! This can affect the host system and requires the container to be privileged or have CAP_SYS_ADMIN.

The Recipe: Fixing the Blind Agent

To allow our agent container to see new PVs mounted by Kubelet under /var/lib/kubelet/pods after the agent has started, we need to mount that host directory into the agent and set mountPropagation: HostToContainer:

# Example Pod spec for the agent
spec:
  containers:
  - name: my-observing-agent
    image: my-agent-image:latest
    # Optional: If your agent needs to *do* something with mounts,
    # it might need privileges beyond just seeing them.
    # Bidirectional *requires* privileged or CAP_SYS_ADMIN.
    # HostToContainer itself doesn't require extra privileges just to see mounts.
    # securityContext:
    #   privileged: false
    #   capabilities:
    #     add: ["SYS_ADMIN"] # Example if needed for other operations
    volumeMounts:
    - name: kubelet-pods-dir
      # The path inside the container where the host dir will be mounted
      mountPath: /mnt/kubelet-pods
      # The magic setting!
      mountPropagation: HostToContainer
  volumes:
  - name: kubelet-pods-dir
    hostPath:
      # The directory on the host we want to observe
      path: /var/lib/kubelet/pods
      # Ensures the path exists on the host
      type: DirectoryOrCreate

With this configuration, any new directory or mount point created by Kubelet under /var/lib/kubelet/pods on the host will automatically appear under /mnt/kubelet-pods inside the my-observing-agent container, without requiring a restart. Problem solved! ✅

Under the Hood: How It Works

What happens when you set HostToContainer?

Kubelet: Sees mountPropagation: HostToContainer in the Pod spec.
CRI Request: Kubelet translates this to the corresponding enum (PROPAGATION_HOST_TO_CONTAINER) in its request to the Container Runtime Interface (CRI).
Container Runtime (containerd, CRI-O): Receives the request and uses Linux system calls (specifically the mount syscall with flags like MS_SLAVE | MS_REC) to configure the container's mount point (/mnt/kubelet-pods in our example) as a recursive slave (rslave) of the corresponding host mount point (/var/lib/kubelet/pods).
Propagation: Because the container's mount is now a slave of the host's mount, any mount events occurring under the host path are automatically propagated by the Linux kernel into the container's mount namespace at the slave mount point.

Important Considerations

Security (Bidirectional): Seriously, be careful with Bidirectional. A container mount propagating back to the host can have significant security implications, especially in multi-tenant clusters. It generally requires the container to run as privileged or have CAP_SYS_ADMIN.
Privileges (HostToContainer): Simply observing mounts with HostToContainer doesn't inherently require extra privileges. However, if your agent needs to perform actions within those propagated mounts (like modifying files owned by root, mounting things itself), it might need appropriate capabilities (CAP_SYS_ADMIN) or run as privileged, independent of the mountPropagation setting itself.
Debugging: If things aren't working, check the mount table inside the container (findmnt or cat /proc/self/mountinfo) and compare it to the host's. Look for the master: entries or propagation flags (shared, slave) associated with your mount point.
Historical Note: For a brief period around Kubernetes 1.10, the default was accidentally changed to HostToContainer before being reverted. If you're managing a very old cluster, be aware of this possibility (#62462).

Conclusion

Kubernetes mountPropagation is a powerful mechanism rooted in Linux mount namespaces that allows you to selectively break container isolation for specific use cases. Understanding the HostToContainer mode (rslave) is key to solving the common problem of agents needing visibility into dynamically created host mounts, like those managed by Kubelet. By applying it correctly, you can build more robust and reliable agents and operators without resorting to restarting them.

Connect with me on LinkedIn!

Securing Kubernetes API Server Health Checks Without Anonymous Access

Mikhail [azalio] Petrov — Sun, 13 Apr 2025 19:11:00 +0000

How to disable anonymous auth globally but keep /livez, /readyz, and /healthz accessible

Recently, a security-savvy colleague posed an interesting question: "Is it possible to disable anonymous access to the Kubernetes API server entirely, but still allow the /livez, /readyz, and /healthz endpoints to work?" 🤔

Honestly, I didn't know the answer offhand, so I dove into the Kubernetes source code and relevant KEPs (Kubernetes Enhancement Proposals). Here's what I found and how you can implement it.

The Default Scenario: Anonymous Access Enabled

As you probably know, by default, kube-apiserver allows anonymous authentication:

--anonymous-auth     Default: true

Enables anonymous requests to the secure port of the API server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of system:anonymous, and a group name of system:unauthenticated.

When you set up a cluster using kubeadm with default settings, anonymous requests to health endpoints (and potentially others like /version, depending on default bindings) succeed:

# curl -k https://<YOUR_API_SERVER_IP>:6443/livez ; echo
ok

Looking at the audit logs, we can confirm the request is handled by system:anonymous:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "2caca66d-ccec-4b1f-8116-86741193cdce", // Example ID
  "stage": "ResponseComplete",
  "requestURI": "/livez",
  "verb": "get",
  "user": {
    "username": "system:anonymous", // <--- Anonymous user
    "groups": [
      "system:unauthenticated"
    ]
  },
  "sourceIPs": [ "<SOURCE_IP>" ],
  "userAgent": "curl/7.88.1",
  "responseStatus": { "code": 200 },
  "requestReceivedTimestamp": "2025-04-13T14:41:33.630618Z", // Example timestamp
  "stageTimestamp": "2025-04-13T14:41:33.630908Z", // Example timestamp
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:public-info-viewer\"..."
  }
}

However, anonymous users can't access other API paths like /apis:

{
  // ... audit metadata ...
  "requestURI": "/apis",
  "verb": "get",
  "user": {
    "username": "system:anonymous",
    "groups": [ "system:unauthenticated" ]
  },
  "responseStatus": {
    "status": "Failure",
    "message": "forbidden: User \"system:anonymous\" cannot get path \"/apis\"",
    "reason": "Forbidden",
    "code": 403 // <--- Forbidden
  },
  // ... timestamps and annotations ...
  "annotations": {
    "authorization.k8s.io/decision": "forbid",
    "authorization.k8s.io/reason": ""
  }
}

While this default setup restricts access, leaving anonymous authentication enabled at all, even for specific paths, can be a security concern. Misconfigured RBAC or potential vulnerabilities could expose more than intended. 😟

The Solution: KEP-4633 and AuthenticationConfiguration

Digging into the Kubernetes source code led me to KEP-4633: Make anonymous authentication configuration endpoints configurable. This KEP addresses the exact concern of wanting to disable anonymous access globally while still allowing essential health checks (which don't necessarily need full TCP checks to be useful).

The solution, documented here, involves using an AuthenticationConfiguration object.

How to Implement It:

Disable Global Anonymous Auth:
Modify your kube-apiserver manifest (usually a static pod in /etc/kubernetes/manifests/) to include these flags:

spec:
  containers:
  - command:
    - kube-apiserver
    # ... other flags ...
    - --anonymous-auth=false # <--- Disable global anonymous auth
    - --authentication-config=/etc/kubernetes/auth-config.yaml # <--- Point to the config file
    volumeMounts:
    # ... other volume mounts ...
    - mountPath: /etc/kubernetes/auth-config.yaml
      name: auth-config-volume
      readOnly: true
  volumes:
  # ... other volumes ...
  - name: auth-config-volume
    hostPath:
      path: /etc/kubernetes/auth-config.yaml
      type: File

Create the Authentication Configuration File:
Create the following file on your control plane node(s) at /etc/kubernetes/auth-config.yaml:

# /etc/kubernetes/auth-config.yaml
apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
anonymous:
  # Enable anonymous access ONLY for the paths listed below
  enabled: true
  conditions:
  - path: /livez
  - path: /readyz
  - path: /healthz

Ensure the kube-apiserver pod restarts to pick up the changes.

The Result: Granular Anonymous Access

With this configuration:

✅ Requests to /livez, /readyz, and /healthz are still allowed for system:anonymous. The audit log looks similar to the default case for these paths.

{
  // ... audit metadata ...
  "requestURI": "/livez",
  "verb": "get",
  "user": {
    "username": "system:anonymous", // <--- Still anonymous
    "groups": [ "system:unauthenticated" ]
  },
  "responseStatus": { "code": 200 }, // <--- Still OK
  // ... timestamps and annotations ...
}

❌ Requests to any other path (like /apis) without valid credentials will now receive a 401 Unauthorized instead of a 403 Forbidden, because the request isn't even treated as anonymous anymore.

# curl -k https://<YOUR_API_SERVER_IP>:6443/apis ; echo
Unauthorized

The audit log confirms this:

{
  // ... audit metadata ...
  "requestURI": "/apis",
  "verb": "get",
  "user": {}, // <--- No user identified (not even anonymous)
  "responseStatus": {
    "status": "Failure",
    "message": "Unauthorized",
    "reason": "Unauthorized",
    "code": 401 // <--- Unauthorized
  },
  // ... timestamps ...
}

Feature Availability

According to the KEP issue tracker, this feature graduated to Alpha in Kubernetes v1.31 and Beta in v1.32.

This approach provides a more secure posture by default, explicitly allowing anonymous access only where strictly necessary for health checks, without relying solely on RBAC for protection against anonymous requests to other endpoints. Now you can tighten security without breaking essential cluster monitoring! 👍

Connect with me on LinkedIn!

Complete Guide: Cilium L2 Announcements for LoadBalancer Services in Bare-Metal Kubernetes

Mikhail [azalio] Petrov — Wed, 22 Jan 2025 13:06:36 +0000

Introduction
Environment Setup
Nginx Deployment
Ingress Configuration
LoadBalancer IP Pool Configuration
ARP Issue
Enabling L2 Announcements
How It Works
- L2 Policy Configuration
- Lease Acquisition
- BPF Map for ARP
Packet Path
Additional Resources
- Cilium L2 Announcements vs Proxy ARP
- Cached vs Non-Cached Maps in Cilium

Introduction: Why L2 Announcements in Kubernetes?

Target Audience

This material is intended for:

Kubernetes cluster administrators
Network engineers working with Cilium technologies (eBPF, CNI)
Infrastructure specialists familiar with L2/L3 networking basics
Professionals interested in Kubernetes networking

📚 Terminology (recommended)

Gratuitous ARP - Special ARP packet broadcast by a node when changing its MAC or IP address to update neighbor ARP caches.

Kubernetes Lease - Resource leasing mechanism through API, ensuring exclusive access to resources (IP addresses in this case) for cluster nodes.

Bare-Metal Kubernetes - Cluster infrastructure deployed on physical servers without cloud providers.

Problem Statement

When working with Kubernetes in bare-metal environments, accessing LoadBalancer services from external networks often becomes challenging. Traditional solutions like MetalLB require additional components and can be overkill for simple scenarios.

Cilium provides native L2 announcement capabilities with:

ARP response handling for LoadBalancer IPs
High availability through lease mechanism
Integration with existing network infrastructure

This guide covers:

Configuring L2 announcements in Cilium
ARP response mechanics through BPF
Practical tips for working with Cilium

The primary goal is to demonstrate Kubernetes service accessibility in bare-metal environments using Cilium's native capabilities.

Environment Setup

Environment setup is described in the README.md

Network diagram:

Pod Subnets:
- 10.200.0.0/24 for server
- 10.200.1.0/24 for node-0
- 10.200.2.0/24 for node-1

Kubernetes and Cilium Configuration

VMs created on ARM Mac using Vagrant.

jumpbox - 192.168.56.10 - client outside Kubernetes cluster.

server - 192.168.56.20 - control plane

node-0 - 192.168.56.50 - k8s node

node-1 - 192.168.56.60 - k8s node

Cilium with native routing (no tunnels). Version v1.16.5.

helm upgrade --install cilium cilium/cilium --version 1.16.5 --namespace kube-system \
  --set l2announcements.enabled=true \
  --set externalIPs.enabled=true \
  --set kubeProxyReplacement=true \
  --set ipam.mode=kubernetes \
  --set k8sServiceHost=192.168.56.20 \
  --set k8sServicePort=6443 \
  --set operator.replicas=1 \
  --set routingMode=native \
  --set ipv4NativeRoutingCIDR=10.200.0.0/22 \
  --set endpointRoutes.enabled=true \
  --set ingressController.enabled=true \
  --set ingressController.loadbalancerMode=dedicated

Kernel info:

# uname -a
Linux node-1 6.1.0-20-arm64 #1 SMP Debian 6.1.85-1 (2024-04-11) aarch64 aarch64 aarch64 GNU/Linux

Deploy Nginx

In this section we'll deploy a simple Nginx web server.

Step 1. Create Deployment

# vagrant ssh server
# sudo bash

cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
EOF

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
EOF

Step 2. Verify Deployment

Check service status:

# kubectl get pod -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
nginx-96b9d695-25swg   1/1     Running   0          55s   10.200.2.189   node-1   <none>           <none>

# kubectl get svc nginx
NAME    TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
nginx   ClusterIP   10.96.79.80   <none>        80/TCP    89s

# curl 10.96.79.80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

# curl 10.200.2.189
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

↑ Table of Contents | ← Back

Configure Ingress

Cilium documentation

Apply ingress manifest:

cat << EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: basic-ingress
  namespace: default
spec:
  ingressClassName: cilium
  rules:
  - http:
      paths:
      - backend:
          service:
            name: nginx
            port:
              number: 80
        path: /
        pathType: Prefix
EOF

# kubectl get ingress
NAME            CLASS    HOSTS   ADDRESS   PORTS   AGE
basic-ingress   cilium   *                 80      40s

This creates a corresponding service:

# kubectl get svc cilium-ingress-basic-ingress
NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
cilium-ingress-basic-ingress   LoadBalancer   10.96.156.194   <pending>     80:31017/TCP,443:32600/TCP   115s

The EXTERNAL-IP remains in pending state due to missing IP pool configuration.

↑ Table of Contents | ← Back

Configure LoadBalancer IP Pool

Cilium supports IP assignment for LoadBalancer services.

CiliumLoadBalancerIPPool

Official documentation

Create CiliumLoadBalancerIPPool:

cat << EOF | kubectl apply -f -
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "blue-pool"
spec:
  blocks:
  - cidr: "10.0.10.0/24"
EOF

Now the service gets an IP:

# kubectl get svc cilium-ingress-basic-ingress
NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
cilium-ingress-basic-ingress   LoadBalancer   10.96.156.194   10.0.10.0     80:31017/TCP,443:32600/TCP   4m9s

Service is now accessible:

# curl 10.0.10.0
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

↑ Table of Contents | ← Back

ARP Issue

Testing from jumpbox client:

# vagrant ssh jumpbox
root@jumpbox:/home/vagrant# curl 10.0.10.0
curl: (7) Failed to connect to 10.0.10.0 port 80 after 3074 ms: Couldn't connect to server

Why? No ARP responses received.

root@server:/home/vagrant# tcpdump -n -i any arp host 10.0.10.0
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes

21:15:05.927064 eth1  B   ARP, Request who-has 10.0.10.0 tell 192.168.56.10, length 46
21:15:06.948513 eth1  B   ARP, Request who-has 10.0.10.0 tell 192.168.56.10, length 46
21:15:07.973210 eth1  B   ARP, Request who-has 10.0.10.0 tell 192.168.56.10, length 46
21:15:08.998950 eth1  B   ARP, Request who-has 10.0.10.0 tell 192.168.56.10, length 46
21:15:10.024080 eth1  B   ARP, Request who-has 10.0.10.0 tell 192.168.56.10, length 46
21:15:11.050053 eth1  B   ARP, Request who-has 10.0.10.0 tell 192.168.56.10, length 46

root@jumpbox:/home/vagrant# arp -n 10.0.10.0
Address                  HWtype  HWaddress           Flags Mask            Iface
10.0.10.0                        (incomplete)                              eth1

↑ Table of Contents | ← Back

Enable L2 Announcements

Enable ARP announcements.

root@server:/home/vagrant# tcpdump -n -i any arp host 10.0.10.0 & # background tcpdump
[1] 17207

root@server:/home/vagrant# cat << EOF | kubectl apply -f -
apiVersion: "cilium.io/v2alpha1"
kind: CiliumL2AnnouncementPolicy
metadata:
  name: policy1
spec:
  nodeSelector:
    matchExpressions:
      - key: node-role.kubernetes.io/control-plane
        operator: DoesNotExist
  interfaces:
  - ^eth[0-9]+
  externalIPs: true
  loadBalancerIPs: true
EOF
ciliuml2announcementpolicy.cilium.io/policy1 created

21:18:52.093372 eth1  B   ARP, Reply 10.0.10.0 is-at 00:0c:29:0d:b7:76, length 46
21:18:52.102795 eth0  B   ARP, Reply 10.0.10.0 is-at 00:0c:29:0d:b7:6c, length 46

root@jumpbox:/home/vagrant# tcpdump -n -i any arp
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes

21:18:52.113122 eth1  B   ARP, Reply 10.0.10.0 is-at 00:0c:29:0d:b7:76, length 46
21:18:52.113211 eth1  B   ARP, Reply 10.0.10.1 is-at 00:0c:29:e3:b1:b2, length 46
21:18:52.122245 eth0  B   ARP, Reply 10.0.10.1 is-at 00:0c:29:e3:b1:a8, length 46
21:18:52.122495 eth0  B   ARP, Reply 10.0.10.0 is-at 00:0c:29:0d:b7:6c, length 46

root@jumpbox:/home/vagrant# arp -n 10.0.10.0
Address                  HWtype  HWaddress           Flags Mask            Iface
10.0.10.0                ether   00:0c:29:0d:b7:76   C                     eth1

The MAC 00:0c:29:0d:b7:76 belongs to node-1's eth1 interface:

root@node-1:/home/vagrant# ip addr sh | grep -1 00:0c:29:0d:b7:76
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:0d:b7:76 brd ff:ff:ff:ff:ff:ff
    altname enp18s0

Verify lease acquisition:

# kubectl get lease -n kube-system cilium-l2announce-default-cilium-ingress-basic-ingress
NAME                                                     HOLDER   AGE
cilium-l2announce-default-cilium-ingress-basic-ingress   node-1   8m49s

↑ Table of Contents | ← Back

How It Works

L2 Policy Configuration
Lease Acquisition
BPF Map for ARP

For detailed documentation see Cilium L2 Announcements.

L2 Policy Configuration

Configure L2 announcement policy

apiVersion: "cilium.io/v2alpha1"
kind: CiliumL2AnnouncementPolicy
metadata:
  name: policy1
spec:
  nodeSelector:
    matchExpressions:
      - key: node-role.kubernetes.io/control-plane
        operator: DoesNotExist
  interfaces:
  - ^eth[0-9]+
  externalIPs: true
  loadBalancerIPs: true

Lease Acquisition

Cilium acquires lease

root@server:/home/vagrant# kubectl get lease -n kube-system | grep l2announce
cilium-l2announce-default-cilium-ingress-basic-ingress   node-1                                                                      16m
cilium-l2announce-kube-system-cilium-ingress             node-0                                                                      16m

BPF Map for ARP

BPF map created for ARP responses

# kubectl get svc cilium-ingress-basic-ingress
NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
cilium-ingress-basic-ingress   LoadBalancer   10.96.156.194   10.0.10.0     80:31017/TCP,443:32600/TCP   24h

root@node-1:/home/cilium# bpftool map show pinned /sys/fs/bpf/tc/globals/cilium_l2_responder_v4
72: hash  name cilium_l2_respo  flags 0x1
    key 8B  value 8B  max_entries 4096  memlock 65536B
    btf_id 125
root@node-1:/home/cilium# bpftool map dump pinned /sys/fs/bpf/tc/globals/cilium_l2_responder_v4
[{
        "key": {
            "ip4": 655370, # IP
            "ifindex": 2   # Interface index
        },
        "value": {
            "responses_sent": 0
        }
    },{
        "key": {
            "ip4": 655370,
            "ifindex": 3
        },
        "value": {
            "responses_sent": 3 # ARP responses count
        }
    }
]

The number 655370 represents IP 10.0.10.0 in little-endian format.

Packet flow through ARP response system:

↑ Table of Contents | ← Back

Packet Path

ARP request processing on node-1:

 +-------------------+
 |   External Host   |
 |    (jumpbox)      |
 +--------+----------+
          | ARP Request "Who has 10.0.10.0?"
          v
 +-------------------+
 |   Interface eth1  |
 |     node-1        |
 +--------+----------+
          | TC ingress
          v
 +-------------------+
 |  BPF Program      |
 | cil_from_netdev   |
 +--------+----------+
          | handle_netdev
          v
 +-------------------+
 |  do_netdev()      |
 +--------+----------+
          | ARP check
          v
 +-------------------+
 | handle_l2_announcement() |
 +--------+----------+
          | Checks:
          | 1. Agent liveness
          | 2. Valid ARP
          | 3. L2_RESPONDER_MAP4 entry
          v
 +-------------------+
 | arp_respond()     |
 +--------+----------+
          | Prepare ARP reply
          v
 +-------------------+
 | ctx_redirect()    |
 +--------+----------+
          | Egress redirect
          v
 +-------------------+
 |   Interface eth1  |
 |     node-1        |
 +--------+----------+
          | ARP Reply
          v
 +-------------------+
 |   External Host   |
 |    (jumpbox)      |
 +-------------------+

↑ Table of Contents | ← Back

Additional Resources

Cilium L2 Announcements vs Proxy ARP

Q: How do Cilium L2 announcements differ from enabling proxy_arp?

A: Proxy ARP handles ARP at kernel level while Cilium uses Kubernetes API for distributed control and BPF for performance.

Disable L2 announcements:

root@server:/home/vagrant# kubectl delete -f workshop/l2.yaml
ciliuml2announcementpolicy.cilium.io "policy1" deleted

Enable proxy_arp on node-1:

root@node-1:/home/vagrant# sysctl -w net.ipv4.conf.eth1.proxy_arp=1
net.ipv4.conf.eth1.proxy_arp = 1

Proxy ARP responses come from kernel:

root@jumpbox:/home/vagrant# arping -I eth1 10.0.10.0
ARPING 10.0.10.0
Timeout
Timeout
Timeout
Timeout
60 bytes from 00:0c:29:0d:b7:76 (10.0.10.0): index=0 time=449.655 msec

Cached vs Non-Cached BPF Maps

Diagnostic tools behavior differs for cached maps:

root@node-1:/home/cilium# cilium-dbg map list
Name                       Num entries   Num errors   Cache enabled
cilium_policy_00215        3             0            true
# ...
cilium_l2_responder_v4     0             0            false

Actual map contents via bpftool:

root@node-1:/home/cilium# bpftool map dump pinned /sys/fs/bpf/tc/globals/cilium_l2_responder_v4
[{
        "key": {
            "ip4": 655370,
            "ifindex": 2
        },
        "value": {
            "responses_sent": 0
        }
    },{
        "key": {
            "ip4": 655370,
            "ifindex": 3
        },
        "value": {
            "responses_sent": 40
        }
    }
]

↑ Table of Contents | ← Back

Conclusion: Benefits of Cilium L2 Announcements

This guide demonstrated Cilium's L2 announcement implementation for bare-metal Kubernetes LoadBalancer services. Key aspects covered:

ARP request handling via BPF programs
High availability through lease system
Diagnostic techniques

Key advantages:

Native Kubernetes integration
eBPF-based packet processing performance
Automatic load distribution
No external dependencies
Standard L2 protocol support

Particularly valuable for hybrid and on-premise environments requiring cloud-like service accessibility while maintaining bare-metal flexibility.

↑ Table of Contents

Forem: Mikhail [azalio] Petrov

Kubernetes Agent Blind to New Mounts? Demystifying Mount Propagation

How mountPropagation: HostToContainer leverages Linux namespaces to solve a common agent problem

The Root Cause: Linux Mount Namespaces

The Solution: Kubernetes mountPropagation

The Recipe: Fixing the Blind Agent

Under the Hood: How It Works

Important Considerations

Conclusion

Securing Kubernetes API Server Health Checks Without Anonymous Access

How to disable anonymous auth globally but keep /livez, /readyz, and /healthz accessible

The Default Scenario: Anonymous Access Enabled

The Solution: KEP-4633 and AuthenticationConfiguration

The Result: Granular Anonymous Access

Feature Availability

Complete Guide: Cilium L2 Announcements for LoadBalancer Services in Bare-Metal Kubernetes

Table of Contents

Introduction: Why L2 Announcements in Kubernetes?

Target Audience

Problem Statement

Environment Setup

Deploy Nginx

Step 1. Create Deployment

Step 2. Verify Deployment

Configure Ingress

Configure LoadBalancer IP Pool

CiliumLoadBalancerIPPool

ARP Issue

Enable L2 Announcements

How It Works

L2 Policy Configuration

Lease Acquisition

BPF Map for ARP

Packet Path

Additional Resources

Cilium L2 Announcements vs Proxy ARP

Cached vs Non-Cached BPF Maps

Conclusion: Benefits of Cilium L2 Announcements

How `mountPropagation: HostToContainer` leverages Linux namespaces to solve a common agent problem

The Solution: Kubernetes `mountPropagation`