Forem: Yuva

Kubernetes in a Hurry: From kube-proxy to ServiceMesh(Q&A Format)

Yuva — Sun, 04 Jan 2026 03:01:57 +0000

Test Your Knowledge
Ready to test your understanding? Take the Networking Quiz — 34 questions covering everything from ARP to service mesh.

Part 1: Kubernetes Networking

Q: What are the three fundamental requirements of Kubernetes networking?

Kubernetes has a simple networking model with three fundamental requirements:

Every pod gets its own IP address
Pods can communicate with all other pods: Without NAT
Node-to-Pod Communication: All cluster Nodes must be able to communicate with all Pods without NAT

This model is implemented by CNI (Container Network Interface) plugins.

Q: What are the basic connectivity requirements to run k8s?

1.Static IP Addresses: All nodes must be assigned static IP addresses or DHCP reservations. Using dynamic IPs that change can break cluster communication and etcd quorum.

Full L2/L3 Connectivity: Every node must have full network connectivity to every other node in the cluster. This can be over a private or public network, provided there is no NAT between nodes.
Unique Identifiers: Each node must have a unique hostname, MAC address, and product_uuid (found in /sys/class/dmi/id/product_uuid)

Q: What are the OS requirements requirements to run k8s?

Disable Swap: Swap must be disabled on all nodes for the kubelet to function correctly.
Kernel Modules: Ensure br_netfilter and overlay modules are loaded to allow bridged traffic to be processed by iptables.
Time Synchronization: Highly accurate time sync (e.g., via Chrony or NTP) is required across all nodes to prevent certificate validation failures and etcd instability.

Q: What is CNI?

CNI (Container Network Interface) is the standard for Kubernetes networking plugins. When a pod starts, it's like a new apartment being built. The CNI plugin is like the city planning department that assigns the new apartment an address (IP address), connects it to the street (creates veth pair), and gives the resident a mailbox (network namespace). The pod can now send and receive letters (packets) just like any other apartment in the city!

Q: What are the CNI plugin responsibilities?

Create network namespace for the pod
Create veth pair (one end in pod, one in host)
Assign IP address from IPAM (IP Address Management) — IPAM allocates IPs from a configured CIDR range
Configure routes so pod can reach other pods and services
Set up overlay network (if needed) for cross-node communication

Q: What are popular CNI plugins?

CNI Plugin	Approach	Overlay Protocol	Key Features
Cilium	eBPF-based routing	Geneve, VXLAN, or native	Network policies, observability, service mesh integration
Calico	BGP routing	VXLAN or native	Network policies, BGP integration
Flannel	Simple overlay	VXLAN	Simple, minimal configuration
Weave	Mesh overlay	Custom (sleeve/fastdp)	Automatic mesh networking

For enterprise and multi-tenant Kubernetes deployments, Cilium and Calico are often preferred due to their robust network policy enforcement, performance benefits (eBPF for Cilium), and integration with BGP for native routing, which are critical for security and scalability.

Q: What makes cilium special?

Cilium uses eBPF (extended Berkeley Packet Filter) for high-performance networking:

Key features:

eBPF-based routing: Faster than iptables, no kernel bypass needed
Network policies: Enforced at the kernel level using eBPF programs
Observability: Built-in metrics and tracing
Service mesh integration: Can replace kube-proxy

Routing modes:

Geneve overlay (default) — Uses Geneve encapsulation for cross-node pod communication. Default choice when overlay is needed. Provides rich metadata in TLV options for advanced network policy enforcement.
VXLAN overlay — Alternative to Geneve for compatibility with environments that don't support Geneve. Similar functionality but without TLV extensibility.
Native routing — No overlay encapsulation. Routes pod IPs directly through the underlying network. Requires routable pod IPs and BGP or static routes.
BGP routing — Uses BGP to advertise pod CIDR routes to network infrastructure. Enables native routing with dynamic route distribution. Works with routers, cloud provider route tables, and other BGP-speaking devices.

Q: What is IPAM (IP Address Management)?

IPAM is the component of CNI plugins that manages IP address allocation. Each CNI plugin includes an IPAM plugin (or uses a standalone one like host-local) that:

Allocates IP addresses from a configured CIDR range (e.g., 10.244.0.0/16)
Tracks which IPs are assigned to which pods
Releases IPs when pods are deleted
Prevents IP conflicts by ensuring each pod gets a unique IP

Q: How do pods on the same node communicate?

When two pods are on the same node, they communicate through the node's bridge. When two pods are on the same node, it's like two apartments in the same building. You write a letter to your neighbor, drop it in the building's mailroom (bridge), and the mailroom immediately delivers it to your neighbor's apartment. No need for the postal service—it's all handled within the building!

Q: How do pods on different nodes communicate?

When pods are on different nodes, the CNI plugin uses an overlay network (VXLAN or Geneve). It's like sending a letter from one building to another across town. You write your letter (original packet) and put it in an inner envelope addressed to your friend's apartment (destination pod IP). The building's mailroom (CNI plugin) puts that inner envelope inside an outer envelope addressed to the destination building (node IP). The postal service (underlay network) delivers the outer envelope to the destination building, where the mailroom there opens it and delivers the inner envelope to your friend's apartment. Your friend never sees the outer envelope—they just receive your letter!

Q: Why do we need Kubernetes Services?

Pods are ephemeral — they can be created, destroyed, and moved. Services provide a stable endpoint for pods. Instead of tracking individual pod IPs (which change constantly), applications use a Service IP that remains constant. kube-proxy maintains the mapping between Service IPs and pod IPs, automatically updating it as pods are created or destroyed.

Q: What are the different Service types?

Service Type	Use Case	How It Works
ClusterIP	Internal communication	Virtual IP (10.96.0.0/12) that kube-proxy routes to pod IPs
NodePort	External access via node IP	Opens a port (30000-32767) on all nodes, routes to ClusterIP
LoadBalancer	Cloud provider integration	Creates external load balancer, routes to NodePort
ExternalName	External service alias	DNS CNAME to external service
Headless	Direct pod access	No ClusterIP, DNS returns pod IPs directly

Q: How does ClusterIP work?

When a Service is created, Kubernetes assigns it a ClusterIP (a virtual IP, e.g., 10.96.0.100) from the cluster's service CIDR (typically 10.96.0.0/12)
kube-proxy on each node creates iptables rules that map the ClusterIP → Pod IPs (based on Endpoints/EndpointSlices)
When traffic arrives at a node destined for the ClusterIP, kube-proxy's iptables rules perform DNAT (Destination NAT), rewriting the destination IP from the ClusterIP to a selected pod IP (load balanced across available pods)
If the selected pod is on a different node, the overlay network (VXLAN/Geneve) handles routing the packet to the destination pod's node

Q: What is kube-proxy?

Kube-proxy manages Service-to-Pod connectivity. It is a network proxy that runs on each node and maintains network rules (usually via iptables or IPVS) to map Kubernetes Service virtual IPs to the actual Pod IPs assigned by the CNI. It has three modes:

iptables mode (default): Creates iptables rules for Service IP → Pod IP mapping. Fast and efficient, but rules can become large with many services.
ipvs mode: Uses Linux IPVS (IP Virtual Server) for load balancing. Better performance and scalability than iptables for large clusters.
userspace mode (deprecated): Proxy runs in userspace. Slower and rarely used.

Q: Can you show an example of how kube-proxy redirects Service traffic?

# Service 10.96.0.100:80 → Pods 10.244.1.5:8080, 10.244.2.7:8080
$ iptables -t nat -L KUBE-SERVICES
Chain KUBE-SERVICES
KUBE-SVC-XXX  tcp  --  anywhere  10.96.0.100  tcp dpt:80

$ iptables -t nat -L KUBE-SVC-XXX
Chain KUBE-SVC-XXX
KUBE-SEP-AAA  all  --  anywhere  anywhere  statistic mode random probability 0.5
KUBE-SEP-BBB  all  --  anywhere  anywhere  # remaining 50%

Q: Why do we need Endpoints?

Endpoints solve a fundamental problem in Kubernetes: Services have stable IPs, but Pods have ephemeral IPs that change constantly.

The problem:

A Service provides a stable virtual IP (e.g., 10.96.0.100) that applications can use
But pods are ephemeral—they get new IPs every time they start, restart, or move to a different node
When a pod is created, destroyed, or scaled, its IP changes
kube-proxy needs to know which actual pod IPs to route traffic to

Without Endpoints:

kube-proxy would have to constantly query the Kubernetes API to find pod IPs
This would be inefficient and slow
There would be no single source of truth for "which pods belong to this Service?"

With Endpoints:

Kubernetes automatically creates and maintains an Endpoints resource for each Service
The Endpoints resource lists all current pod IPs that match the Service selector
kube-proxy watches the Endpoints resource (not individual pods)
When pods change, the Endpoints resource is updated automatically
kube-proxy gets notified of changes and updates its routing rules (iptables/IPVS)

Q: What are EndpointSlices?

A newer, more scalable alternative to Endpoints (introduced in Kubernetes 1.16, GA in 1.21)
Splits endpoints across multiple slice resources (up to 100 endpoints per slice)
Reduces the size of individual resources, improving performance in large clusters
Provides better scalability: a Service with 1000 pods creates ~10 EndpointSlices instead of 1 large Endpoints resource
Includes additional metadata like topology hints (which zone/node pods are in)

Example EndpointSlice:

apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: backend-abc123
  labels:
    kubernetes.io/service-name: backend
addressType: IPv4
ports:
- name: http
  port: 8080
  protocol: TCP
endpoints:
- addresses:
  - "10.244.1.5"
  conditions:
    ready: true
  nodeName: node-1
  zone: us-west-1a
- addresses:
  - "10.244.2.10"
  conditions:
    ready: true
  nodeName: node-2
  zone: us-west-1b

How kube-proxy uses them:

kube-proxy watches Endpoints or EndpointSlices (EndpointSlices preferred in modern clusters)
When endpoints change (pod created/destroyed), kube-proxy updates iptables or IPVS rules
Rules map Service IP → Pod IPs, enabling load balancing across pods
The watch mechanism ensures rules stay synchronized with actual pod state

Why EndpointSlices matter:

Performance: Smaller resources mean faster API server processing and less network traffic
Scalability: Can handle services with thousands of pods without creating massive single resources
Topology awareness: Includes zone/node information for better routing decisions
Future-proof: Foundation for advanced features like topology-aware routing

Q: What is CoreDNS?

CoreDNS is the default DNS server in Kubernetes. It:

Runs as a Deployment in the kube-system namespace
Watches Kubernetes Services and Endpoints
Automatically creates DNS records for all Services
Resolves service names to Service IPs (ClusterIP)
Supports custom DNS entries via ConfigMaps

When a pod queries backend.default.svc.cluster.local, CoreDNS returns the Service IP (e.g., 10.96.0.100), which kube-proxy then routes to an actual pod IP.

Q: How does service discovery work with DNS?

Kubernetes provides DNS for services via CoreDNS. Applications can use service names (e.g., backend.default.svc.cluster.local) instead of IP addresses. CoreDNS resolves service names to Service IPs, which kube-proxy then routes to pod IPs.

Q: What is the DNS naming format?

Format: <service>.<namespace>.svc.cluster.local
Short form: <service>.<namespace> or just <service> (same namespace)
Example: backend.default.svc.cluster.local → 10.96.0.100

Q: What is Ingress and how does it relate to Services?

Ingress provides HTTP/HTTPS routing from outside the cluster to Services. Unlike Services (which provide internal cluster networking), Ingress handles external access.

Ingress Controller: A reverse proxy (e.g., NGINX, Traefik, Envoy) that runs in the cluster and implements Ingress rules
Ingress Resource: Defines routing rules (host, path → Service)
Flow: External request → Ingress Controller → Service → Pod

Ingress works on top of Services—it routes external traffic to the appropriate Service, which then routes to pods. For advanced routing (canary, A/B testing, mTLS), service mesh is often used instead of or alongside Ingress.

Q: What is gateway api and how is it different from ingress controller?

The API replaces the "one-size-fits-all" Ingress object with three distinct resources, each designed for a specific organizational role:
GatewayClass (Infrastructure Provider): A cluster-scoped resource that defines a specific type of load balancer or proxy implementation (e.g., an AWS NLB, NGINX, or Istio).
Gateway (Cluster Operator): An instantiation of a GatewayClass. It defines the actual entry point where traffic is received, including configuration for specific listeners (ports and protocols like HTTP, HTTPS, TCP, or UDP) and TLS termination.
HTTPRoute / GRPCRoute (App Developer): Resource-specific rules that define how traffic should be routed from a Gateway to backend Services based on hostnames, paths, or headers

Key Benefits over Ingress:

Built-in Advanced Routing: Natively supports features like header-based matching, traffic splitting (canary rollouts), and request mirroring without requiring custom, non-portable annotations.
Broader Protocol Support: Beyond HTTP/HTTPS, it officially supports gRPC, TCP, UDP, and WebSockets.
Separation of Concerns: Teams can manage their own routing rules (via HTTPRoute) independently from the shared infrastructure (via Gateway), reducing the risk of accidental misconfigurations across a cluster.
Portability: As a standardized specification, configurations are portable between different vendors (e.g., from Envoy Gateway to Traefik) without needing to rewrite complex vendor-specific rules.
Cross-Namespace Routing: Allows a single Gateway to route traffic to Services in different namespaces securely through the use of ReferenceGrant objects.

Q: What are Network Policies?

Network Policies allow you to control traffic between pods using label selectors. They act as pod-level firewalls, allowing or denying traffic based on source pod labels, destination pod labels, and ports. Network Policies are enforced by CNI plugins (Cilium, Calico) at the kernel level, before packets reach the pod.

Q: Can you show an example NetworkPolicy?

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

This policy says: "Only pods labeled app: frontend can talk to pods labeled app: backend on port 8080."

Q: How are Network Policies implemented?

Cilium: Uses eBPF and Geneve TLV options for policy enforcement
Calico: Uses iptables rules and BGP for policy distribution
Flannel: Does not support Network Policies (needs Calico or Cilium)

NetworkPolicy vs service mesh authorization (multi-tenant isolation)

Aspect	NetworkPolicy (CNI)	Service mesh auth (sidecar)
Scope	L3/L4 (IP, port)	L7 (HTTP/gRPC) + identity
Enforcer	CNI dataplane (node)	Sidecar proxy (pod)
Best for	Blast-radius limits between namespaces/tenants; default-deny	App-level allow/deny, mTLS, per-route rules
Debug with	`kubectl get networkpolicies -A`, `cilium monitor`/`calicoctl`	Sidecar logs, `AuthorizationPolicy`/`ServerAuthorization`

Q: What is the Kubernetes networking stack?

The Kubernetes networking stack is built in layers, with each layer providing specific functionality:

┌─────────────────────────────────────┐
│  Service Mesh (L7)                  │  ← Identity, observability, traffic mgmt
│  (Istio, Linkerd)                   │
├─────────────────────────────────────┤
│  Services (kube-proxy)              │  ← Service discovery, load balancing
│  ClusterIP, NodePort, LoadBalancer  │
├─────────────────────────────────────┤
│  CNI Plugin (L2/L3)                 │  ← Pod networking, overlay networks
│  (Cilium, Calico, Flannel)          │
├─────────────────────────────────────┤
│  Linux Networking Primitives        │  ← Network namespaces, veth, bridges
└─────────────────────────────────────┘

Key takeaways:

CNI plugins provide pod networking using native or overlay networks (VXLAN/Geneve)
Services abstract pod IPs using virtual IPs and kube-proxy
Network Policies control traffic between pods
Service mesh adds Layer 7 capabilities on top of everything

Part 2: Cilium Special

Q: What are the requirements for Cilium native routing mode?

Native routing mode eliminates overlay encapsulation, routing pod IPs directly through the underlying network. This requires:

Routable pod IPs: Pod IPs (from the pod CIDR, e.g., 10.244.0.0/16) must be routable in your network infrastructure. The underlying network (routers, switches, cloud provider networking) must know how to route traffic to pod IPs.
BGP or static routes: You need either:
- BGP: Cilium can use BGP to advertise pod CIDR routes to your network infrastructure (routers, cloud provider route tables)
- Static routes: Manually configure routes in your network infrastructure pointing pod CIDRs to Kubernetes nodes
No IP conflicts: Pod IPs must not conflict with existing IPs in your network.

When to use native routing:

You have control over network infrastructure (on-premises, custom cloud networking)
You want maximum performance (no encapsulation overhead)
Your network supports BGP or you can configure static routes
Pod IPs can be made routable in your network

When to use overlay (Geneve/VXLAN):

Cloud provider environments where pod IPs aren't routable
You want simplicity (no BGP/static route configuration)
Network infrastructure doesn't support routing pod CIDRs
You need the rich metadata capabilities of Geneve TLV options

Q: How does Cilium use BGP?

Cilium supports BGP for native routing mode, allowing it to advertise pod CIDR routes to network infrastructure without using overlay encapsulation. Each Cilium node runs a BGP daemon that advertises its pod CIDR to BGP peers (routers, cloud provider route tables), enabling the network infrastructure to learn pod IP routes and route traffic directly to pods without encapsulation overhead.

Use cases for Cilium BGP:

On-premises deployments: Advertise pod routes to physical routers/switches
Cloud environments: Integrate with cloud provider route tables (AWS Route Tables, Azure Route Tables, GCP Routes)
Hybrid cloud: Connect on-premises and cloud networks via BGP
Large-scale clusters: Native routing performs better than overlay at scale
Integration with existing BGP infrastructure: Works with existing network equipment

BGP vs overlay in Cilium:

BGP (native routing): Better performance, no encapsulation overhead, requires BGP-capable infrastructure
Overlay (Geneve/VXLAN): Works everywhere, simpler setup, adds encapsulation overhead

Configuration example:
Cilium can be configured to use BGP by enabling the BGP control plane and specifying BGP peers (routers or route reflectors). The BGP daemon then advertises pod CIDRs to peers, enabling native routing.

Q: How does Cilium use Geneve for network policy?

Cilium is a popular CNI plugin that uses Geneve overlay and eBPF for advanced networking. With Cilium and Geneve, when you send a letter, the building's security system (Cilium agent) checks your ID, looks up the security policy, and attaches security stickers to the outer envelope: "From: frontend-workload", "Policy: allow-frontend-to-backend", "Security Clearance: Level 3". When the letter arrives at the destination building, the security guard there reads the stickers, verifies the policy allows this communication, and only then delivers the letter. If the stickers don't match the policy, the letter is rejected!

Part 3: Service Mesh and Workload Identity

Q: What problem does service mesh solve?

As microservices architectures became the norm in the 2010s, developers faced new challenges that infrastructure-layer networking (VLAN, VXLAN, Geneve) couldn't solve. While overlay networks could route packets between hosts using IP addresses, they operated at Layer 2/3 and couldn't help with application-layer concerns.

The fundamental problem: In a microservices world, services are ephemeral—they start, stop, scale, and move constantly. IP addresses change. Network boundaries are fluid. Traditional networking assumptions broke down.

Why this was painful:

Service discovery: You couldn't hardcode IPs because containers/pods get new IPs every time they restart or scale. Every service needed to implement its own service discovery (Consul, etcd, custom solutions), leading to inconsistency across teams. When "backend" had 10 instances, which one should you call? How do you load balance? Infrastructure networking could route packets, but couldn't answer "where is the backend service?"
Security between services: Every team was implementing their own TLS, authentication, and authorization logic. Some services used certificates, others used API keys, some had no security at all. This created security gaps, inconsistent implementations, and maintenance nightmares. When you had 50 microservices, you had 50 different security implementations to maintain.
Observability: When a user request failed, which service was the problem? Was it the frontend? The API gateway? The auth service? The database? There was no way to trace a request as it flowed through multiple services. Each service logged independently, but correlating logs across services was nearly impossible. You couldn't see the "big picture" of how services communicated.
Traffic management: Every service needed to implement retry logic, timeouts, circuit breakers, and load balancing. When backend was slow, frontend would retry—but how many times? With what backoff? What if backend was completely down—should you fail fast or keep retrying? Each team made different decisions, leading to cascading failures and inconsistent behavior.
Zero-trust security: Traditional network security relied on firewalls and network boundaries: "trust everything inside the network, block everything outside." But in microservices, there is no "inside"—services move, IPs change, and the network boundary is meaningless. An attacker who compromised one service could access all services on the same network. You needed identity-based security: "trust based on who you are, not where you are."

The breaking point: Developers were writing the same networking code (retry logic, TLS, metrics, tracing, service discovery) in every microservice. This was expensive, error-prone, and inconsistent. Service mesh emerged around 2016-2018 (with projects like Linkerd and Istio) to solve these problems by moving networking concerns out of application code and into a dedicated infrastructure layer that worked transparently for all services.

Q: What is a service mesh?

A service mesh is a dedicated infrastructure layer that handles service-to-service communication, security, observability, and traffic management for microservices. Unlike VLAN/VXLAN/Geneve which route packets at Layer 2/3 (infrastructure layer) using IP addresses, service mesh routes requests at Layer 7 (application layer) using service names and DNS.

Architecture: A service mesh consists of a control plane (e.g., Istio's istiod) that distributes configuration to sidecar proxies (e.g., Envoy) running alongside each application container. The sidecars handle mTLS, routing, and observability transparently. For detailed architecture diagrams, see the Istio Architecture documentation.

Q: How does service mesh integrate with Kubernetes networking?

Service mesh works on top of Kubernetes networking, adding Layer 7 capabilities. The integration flow:

App uses service name (backend.default.svc.cluster.local or just backend)
Sidecar resolves via Kubernetes DNS → Service IP (10.96.0.100)
Service discovery → Pod IP (10.244.2.10)
Overlay network (VXLAN/Geneve) routes packet to destination pod
Service mesh adds mTLS, observability, traffic management

Service mesh leverages Kubernetes service discovery and DNS, then adds identity-based security, request-level observability, and traffic management on top of the existing pod-to-pod networking.

Q: How does service mesh relate to overlay networks?

💡 Key Insight

Service mesh works on top of overlay networks. You still need VXLAN/Geneve to route packets between hosts/containers at the infrastructure layer. Service mesh then adds Layer 7 routing and capabilities transparently to applications.

While VLAN/VXLAN/Geneve are like the postal service that routes envelopes based on addresses (infrastructure layer), service mesh is like a smart assistant who reads your letter and routes it based on what you wrote (application layer). You write "Send this to the accounting department" (service name), and the assistant looks up which building and apartment that is, puts your letter in the right envelope (with security and tracking), and sends it. The assistant also adds a return envelope with your identity certificate, so the recipient knows it's really from you. The postal service (overlay network) still delivers the physical envelope, but the assistant (service mesh) handles the "who talks to whom" and "is this allowed" logic.

Q: What is GAMMA?

The GAMMA (Gateway API for Mesh Management and Administration) routes internal (East-West) traffic by repurposing standard Gateway API Route objects (like HTTPRoute or GRPCRoute) but changing their Parent Reference.

Graduation: GAMMA's work for supporting service mesh use cases (East-West traffic) graduated to the Standard Channel (GA) starting with Gateway API v1.1.0 in early 2024.
Core Feature: The primary GAMMA feature—binding a Route (like HTTPRoute) directly to a Service as a parent—is fully stable and supported by major service meshes like Cilium, Istio, and Linkerd.

Q: What is workload identity?

Workload Identity is a way to identify and authenticate workloads (containers, VMs, processes) using cryptographically verifiable certificates rather than IP addresses. Instead of saying "Allow mail from 10.0.0.5" (IP-based), we now say "Allow mail from the Accounting Department" (identity-based). Each workload gets a certificate (like a company ID badge) that proves who they are. When you send a letter, you include a copy of your ID badge in the envelope. The recipient checks: "Is this person from Accounting? Yes, they're allowed to send me mail." It's like moving from checking return addresses (which can be faked) to checking photo IDs (which can't be easily forged).

Q: What is the evolution of workload identity and what problem was it solving?

Workload identity evolved to solve the fundamental problem that IP addresses are not a reliable way to identify workloads in modern, dynamic environments.

The old way: IP-based security (pre-2010s)

Firewall rules: "Allow 10.0.0.5 to access database on port 5432"
Network segmentation: "Everything in subnet 10.0.1.0/24 is trusted"
This worked when servers had static IPs and rarely moved

Why IP-based security broke down:

Containers and VMs are ephemeral: A container gets a new IP every time it starts. Your firewall rule for 10.0.0.5 is useless when the container restarts and gets 10.0.0.47.
Scaling breaks rules: When you scale from 1 backend instance to 10, you can't maintain firewall rules for each IP. You'd need to update rules constantly.
Workloads move: A pod moves from Node A to Node B? Its IP changes. Your security rules break.
IPs can be spoofed: An attacker who compromises one workload can spoof IPs to appear as another workload.
No context: An IP address tells you nothing about what the workload is or who it belongs to. Is 10.0.0.5 the frontend? The backend? A test service? You can't tell from the IP.

The evolution:

Service accounts (early 2010s): Platforms like Kubernetes introduced service accounts—a step toward identity, but platform-specific. Kubernetes service accounts only work in Kubernetes. AWS IAM roles only work in AWS. No portability.
Workload identity (2018-present): Standards like SPIFFE emerged to provide portable, verifiable workload identity. Each workload gets a certificate (SVID - SPIFFE Verifiable Identity Document) that proves:
- Who it is: spiffe://cluster.local/ns/prod/sa/frontend
- Where it came from: The certificate is cryptographically signed, so it can't be forged
- What it can do: Policies can be written in terms of identity, not IPs

The breakthrough: Instead of "Allow IP 10.0.0.5", you now say "Allow workloads with identity spiffe://cluster.local/ns/prod/sa/frontend". The identity stays the same even when the IP changes. It works across platforms (Kubernetes, VMs, bare metal). It's cryptographically verifiable, so it can't be spoofed.

Q: When did service mesh and workload identity get integrated?

Service mesh and workload identity evolved separately at first, then became tightly integrated:

2016-2017: Early service meshes (Linkerd 1.0 in 2016, Istio 0.1 in 2017) initially used platform-specific identities:
- Kubernetes service accounts in Kubernetes environments
- No standard identity format
- Identity was tied to the platform
2017-2018: SPIFFE emerges: The SPIFFE project started in 2017 to create a standard for workload identity that works across platforms. SPIFFE provided the foundation, but service meshes weren't using it yet.
2019-2020: The integration begins: Service meshes started adopting SPIFFE:
- Istio 1.4 (2020): Added SPIFFE integration, allowing Istio to issue SPIFFE SVIDs
- Linkerd 2.7+ (2020): Integrated SPIFFE for workload identity
- This was the "marriage" - service meshes could now use standardized, portable workload identity
2020-present: Deep integration: Modern service meshes are built around workload identity:
- Identity is no longer optional—it's core to how service mesh works
- mTLS uses workload identity certificates (SVIDs)
- Authorization policies are written in terms of workload identity
- Works across Kubernetes, VMs, and bare metal

Why the integration matters: Before SPIFFE integration, service meshes were platform-locked. A Kubernetes service account identity couldn't be verified by a VM-based service. With SPIFFE, the same workload identity works everywhere, enabling true multi-platform service mesh deployments and zero-trust security across heterogeneous environments.

Q: What are the identity models used by service mesh?

Service meshes support three main identity models, each with different trade-offs:

SPIFFE/SVID: SPIFFE (Secure Production Identity Framework for Everyone) provides a standard, portable workload identity via SVIDs (SPIFFE Verifiable Identity Documents). SPIFFE identities are cryptographically verifiable certificates that work across platforms (Kubernetes, VMs, bare metal). This is the most portable and future-proof approach. Used by Istio (with SPIFFE integration) and Linkerd. Learn more: SPIFFE Documentation, SPIFFE Specification
Platform service accounts: Service meshes can use platform-specific service accounts (e.g., Kubernetes service accounts, AWS IAM roles, Azure managed identities) as workload identity. This is simpler to set up but ties you to a specific platform—a Kubernetes service account identity can't be verified by a VM-based service. Good for single-platform deployments.

Kubernetes service accounts are the most common example. Each pod can be assigned a service account, and the service mesh uses this to identify the workload. The service account name (e.g., frontend-sa) becomes part of the workload identity. However, this only works within Kubernetes—you can't use a Kubernetes service account identity to authenticate to a VM-based service. Learn more: Kubernetes Service Accounts, Using Service Accounts with Istio, Linkerd Service Account Identity
Custom identity providers: Some meshes integrate with cloud provider IAM (AWS IAM, Azure AD, GCP IAM) or custom identity systems. This allows leveraging existing identity infrastructure but requires custom integration work and may not be portable across platforms. Learn more: AWS App Mesh IAM Integration, Azure Service Mesh Identity

Q: What are the constraints of service mesh?

CPU overhead: Every request goes through a proxy, adding 1-5ms latency and consuming CPU
Complexity: Debugging distributed systems with sidecars is harder; requires understanding both application and mesh behavior
Latency: Additional hops add milliseconds (though often acceptable for the benefits)
Resource consumption: Each workload requires a sidecar proxy, increasing memory and CPU usage

Q: Do I need service mesh?

Short answer: It depends. Service mesh solves real problems, but it's not always the right solution.

The pragmatic approach:

Start simple: Use API gateways, load balancers, and basic monitoring first
Add service mesh when you feel the pain: When you find yourself writing the same networking code in every service, or when observability becomes impossible
Consider alternatives: API gateways (Kong, Ambassador) can provide some service mesh features without the complexity
Evaluate the trade-offs: Service mesh adds complexity and overhead. Make sure the benefits (security, observability, traffic management) justify the cost

Remember: Service mesh is infrastructure. Like any infrastructure, it should solve problems you actually have, not problems you might have someday. If you're not experiencing the pain points service mesh solves, you probably don't need it yet.

Q: Can you walk through an example of frontend calling backend using identity?

This section details the complete packet journey in a Kubernetes cluster augmented with a service mesh, illustrating the interplay of all the components discussed so far. The diagram below shows the flow using SPIFFE identities for workload authentication:

Q: What's the difference between IP-based and identity-based security in Kubernetes?

The key principle: identity-based security replaces IP-based security in modern Kubernetes deployments.

Old (IP-based):

Allow 10.0.0.5
Deny 10.0.0.6
Network Policies based on pod IPs (which change constantly)

New (Identity-based):

Allow spiffe://cluster.local/ns/prod/sa/frontend
Deny spiffe://cluster.local/ns/dev/*
Authorization policies based on workload identity (which stays constant)

Q: Where do I create identity-based security rules and who enforces them in Kubernetes?

Identity-based security rules are created as Kubernetes Custom Resources (CRDs) and enforced by the service mesh data plane (sidecar proxies).

Where rules are created:

Istio: Create AuthorizationPolicy resources in Kubernetes:

  apiVersion: security.istio.io/v1beta1
  kind: AuthorizationPolicy
  metadata:
    name: allow-frontend-to-backend
    namespace: prod
  spec:
    selector:
      matchLabels:
        app: backend
    action: ALLOW
    rules:
    - from:
      - source:
          principals: ["cluster.local/ns/prod/sa/frontend"]

Linkerd: Create Server and ServerAuthorization resources:

  apiVersion: policy.linkerd.io/v1beta2
  kind: ServerAuthorization
  metadata:
    name: backend-authz
    namespace: prod
  spec:
    server:
      name: backend
    client:
      meshTLS:
        identities:
        - "frontend.prod.serviceaccount.identity.linkerd.cluster.local"

General pattern: Rules are defined as YAML manifests and applied via kubectl apply, just like any Kubernetes resource.

Who enforces them:

Service mesh control plane (istiod, Linkerd control plane): Distributes the rules to all sidecar proxies
Sidecar proxies (Envoy, Linkerd proxy): Enforce the rules at runtime—they intercept traffic, verify workload identity, and allow/deny requests based on the policies
No application code changes: The application doesn't know about the rules—the sidecar proxy handles enforcement transparently

Example flow:

You create an AuthorizationPolicy with kubectl apply
Istio control plane (istiod) reads the policy and distributes it to all Envoy sidecars
When frontend tries to call backend, Envoy sidecar checks: "Does this request come from spiffe://cluster.local/ns/prod/sa/frontend?"
Envoy looks up the policy: "Yes, frontend is allowed to talk to backend" → Request proceeds
If the identity doesn't match, Envoy rejects the request with HTTP 403

Key insight: The rules are Kubernetes resources (like Deployments or Services), but enforcement happens in the service mesh data plane (sidecar proxies), not in Kubernetes itself. This gives you identity-based security without modifying application code.

Q: What is the complete packet flow from app to app via overlay network in Kubernetes with service mesh?

Here's the complete journey of a packet from one pod to another:

References

Service Mesh and Identity

SPIFFE: Secure Production Identity Framework for Everyone
Istio: Connect, Secure, Control, and Observe Services
Linkerd: Ultra Lightweight Service Mesh for Kubernetes

Container Networking

CNI Specification: Container Network Interface
Cilium: eBPF-based Networking, Security, and Observability

This article is part of the "Learning in a Hurry" series, designed to help engineers quickly understand complex technical concepts through analogies and practical examples.

Networking in a Hurry: From ARP to Geneve(Q&A Format)

Yuva — Thu, 01 Jan 2026 21:31:26 +0000

Understanding modern cloud networking through the lens of envelopes, mailrooms, and postal services

I have spent quite few days debugging kubernetes networking issues. I realized that I had gaps in my understanding of networking components/terms. So I went on a mission with my AI friend to ask all the questions that I can possibly ask to better my mental model of what goes on beneath the orchestrator and below is the documentation of all that learning.

Skip to the quiz if needed.

Part 1: The Fundamentals

The OSI Model

Q: What is the OSI model and why do I need to know it?

The OSI (Open Systems Interconnection) model divides networking into seven layers. Each layer only communicates with the layers directly above and below it. It's your mental map for understanding how networking works.

Q: What are the seven layers of the OSI model?

Layer 7: Application — HTTP, DNS, SSH
Layer 6: Presentation — TLS/SSL, Compression
Layer 5: Session — Connection management
Layer 4: Transport — TCP, UDP (Ports)
Layer 3: Network — IP (Routing)
Layer 2: Data Link — Ethernet, MAC (Switching)
Layer 1: Physical — Cables, Radio, Fiber

Q: What's the TCP/IP model?

For practical purposes, we often use the simpler TCP/IP model which combines some layers:

Application Layer — HTTP, DNS, SSH
Transport Layer — TCP, UDP
Internet Layer — IP, ICMP
Network Access Layer — Ethernet, Wi-Fi

Q: What does "L2 over L3 tunneling" mean?

Normally, Layer 3 packets (IP packets) are encapsulated inside Layer 2 frames (Ethernet frames). This is the standard way networking works: an IP packet gets wrapped in an Ethernet frame with MAC addresses, and the frame is delivered to the next hop.

"L2 over L3 tunneling" reverses this: it wraps an entire Ethernet frame (Layer 2) inside an IP packet (Layer 3). This is what technologies like VXLAN and Geneve do.

Why is this useful? It allows you to create virtual Layer 2 networks that span across Layer 3 infrastructure. For example, you can have two VMs in different data centers that appear to be on the same Layer 2 network, even though they're separated by routers and IP networks. The original Ethernet frame (with its MAC addresses) is preserved inside the IP packet, allowing Layer 2 protocols and features to work across the tunnel.

Layer 2: Getting to Your Neighbor

Q: What is Layer 2 about?

Layer 2 is about communication within a local network segment—devices that can reach each other without going through a router.

Q: What is a MAC address?

Every network interface card (NIC) has a unique 48-bit MAC address (Media Access Control), written as six pairs of hex digits like 00:1A:2B:3C:4D:5E. The first 3 bytes identify the manufacturer (OUI), and the last 3 bytes are unique to the device.

Examples:

00:50:56:xx:xx:xx → VMware
02:42:xx:xx:xx:xx → Docker
52:54:00:xx:xx:xx → QEMU/KVM

Q: What is ARP and why do I need it?

When Host A wants to send a packet to Host B (same subnet), it knows B's IP address but not its MAC address. ARP (Address Resolution Protocol) solves this.

Q: How does ARP work?

Think of IP addresses as street addresses and MAC addresses as the actual mailbox. Before you can deliver a letter, you need to know which mailbox (MAC) belongs to that address (IP). ARP is like shouting down the street: "Who lives at 192.168.1.20?" and waiting for the owner to respond with their mailbox number.

Q: How do I view my ARP cache?

$ ip neigh
192.168.1.1 dev eth0 lladdr 00:11:22:33:44:55 REACHABLE
192.168.1.20 dev eth0 lladdr bb:bb:bb:bb:bb:bb STALE

Q: What is a switch and how does it work?

A switch is a Layer 2 device that learns which MAC addresses are on which ports by observing traffic. A switch is like a smart mail carrier who learns the neighborhood. When you send a letter, the carrier looks at the return address (source MAC) and remembers "this person lives on Elm Street." When mail arrives for that person, the carrier knows exactly which street to go to, instead of delivering to every house.

Q: How does MAC learning work on a switch?

Layer 3: Getting Across Town

Q: What is Layer 3 about?

Layer 3 is about communication between networks—when you need to go beyond your local segment.

Q: What is an IP address?

An IP address is a unique identifier assigned to each device on a network. There are two versions in use today: IPv4 and IPv6.

IPv4 (Internet Protocol version 4):

A 32-bit number, written as four octets (8 bits each) separated by dots
Example: 192.168.1.100
Each octet ranges from 0-255
Total address space: 2^32 = 4,294,967,296 addresses (~4.3 billion)
Format: xxx.xxx.xxx.xxx where each xxx is 0-255

IPv6 (Internet Protocol version 6):

A 128-bit number, written as eight groups of four hexadecimal digits separated by colons
Example: 2001:0db8:85a3:0000:0000:8a2e:0370:7334
Can be shortened by removing leading zeros: 2001:db8:85a3::8a2e:370:7334
Total address space: 2^128 = 340,282,366,920,938,463,463,374,607,431,768,211,456 addresses (~340 undecillion)
Format: xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx where each xxxx is 0-FFFF

Why do we need IPv6?

IPv4 address exhaustion is the primary driver. With only ~4.3 billion addresses and billions of devices (computers, phones, IoT devices, servers), we've run out of public IPv4 addresses. This has led to:

NAT (Network Address Translation) overuse: Multiple devices sharing one public IP, which breaks the end-to-end principle of the internet
Address scarcity: Organizations paying premium prices for IPv4 address blocks
Complexity: Multiple layers of NAT making networking harder to troubleshoot

IPv6 solves this by providing:

Vast address space: Enough addresses for every device on Earth (and trillions more)
Simplified networking: No NAT needed—every device can have a globally routable address
Better performance: Simpler packet headers, more efficient routing
Built-in security: IPsec support is mandatory in IPv6
Auto-configuration: Devices can automatically configure their addresses (SLAAC)
Better mobile support: Improved handling of devices moving between networks

The transition: While IPv6 is the future, we're in a transition period. Most networks support both (dual-stack), allowing devices to use either protocol. IPv4 will likely remain in use for decades due to legacy systems, but new deployments increasingly prioritize IPv6.

Q: What is a subnet and how does CIDR notation work?

A subnet defines which part of the address is the "network" and which is the "host". In CIDR notation 192.168.1.100/24:

Network portion: first 24 bits (192.168.1)
Host portion: last 8 bits (.100)
Subnet mask: 255.255.255.0
Network: 192.168.1.0
Broadcast: 192.168.1.255
Total hosts: 254

Q: What are common subnet sizes?

CIDR	Subnet Mask	Use Case	Hosts
/8	255.0.0.0	Large enterprise (10.x.x.x)	16 million
/16	255.255.0.0	Medium network (172.16.x.x)	65,534
/24	255.255.255.0	Typical LAN	254
/32	255.255.255.255	Single host	1

Q: How do you calculate the number of hosts in a subnet?

The formula is: 2^(host bits) - 2

Host bits = 32 - CIDR prefix (for IPv4)
Subtract 2 because the network address (all zeros) and broadcast address (all ones) cannot be assigned to hosts

Examples:

/24 subnet:

Host bits: 32 - 24 = 8 bits
Total addresses: 2^8 = 256
Usable hosts: 256 - 2 = 254

/16 subnet:

Host bits: 32 - 16 = 16 bits
Total addresses: 2^16 = 65,536
Usable hosts: 65,536 - 2 = 65,534

Why not 254 × 254?

A common misconception is that /16 = 254 × 254 = 64,516. This is incorrect because:

In a /16 subnet, the host portion is the last 16 bits (the last two octets combined)
This gives us 2^16 = 65,536 total addresses, not 254 × 254
The 254 × 254 calculation would only apply if we were thinking of it as two separate /24 subnets, which is not how /16 works
In a /16, all 16 host bits are used together as one address space

/8 subnet:

Host bits: 32 - 8 = 24 bits
Total addresses: 2^24 = 16,777,216
Usable hosts: 16,777,216 - 2 = 16,777,214 (often rounded to "16 million")

Q: How does a host decide if a destination is local or remote?

When a host wants to send a packet, it first asks: "Is the destination on my local network?" Before sending a letter, you check: "Is this address on my street?" If yes, you just walk over and deliver it yourself (ARP and direct delivery). If no, you drop it in the mailbox for the postal service to handle (send to default gateway/router). You don't need to know the entire postal system—just whether it's local or needs to go through the post office!

Q: What is a router and how does routing work?

A router connects multiple networks. It uses a routing table to decide where to send each packet. A router is like a post office sorting facility. When a letter arrives, the postal worker looks at the destination address (IP) and checks the routing table—a big directory that says "letters for 10.0.5.0 go to the downtown post office, letters for 10.0.1.0 go to the local branch." The router doesn't change the address on your envelope (IP stays the same), but it knows which "next post office" (next hop) to send it to.

Example routing table:

10.0.1.0/24 → eth0 (directly connected)
10.0.2.0/24 → eth1 (directly connected)
10.0.5.0/24 → via 10.0.2.254 (next hop)
0.0.0.0/0 → via 203.0.113.1 (default route)

Q: Do IP addresses change as packets traverse the network?

💡 Key Insight

IP addresses never change as packets traverse the network. Only MAC addresses change at each hop.

Why IP addresses stay the same:

IP addresses are logical addresses that represent the final destination (and source) of the packet. Think of them as the address written on your envelope—the destination address (10.0.5.100) is where you want the letter to ultimately arrive, and the return address (192.168.1.50) is where it came from. These never change because they represent the actual source and destination hosts.

Why MAC addresses change:

MAC addresses are physical addresses that represent the immediate next hop. At each router or switch, the packet needs to be delivered to the next device in the path. The MAC address is rewritten to point to the next hop's physical interface.

Think of it like sending a letter from New York to Los Angeles: your envelope has the final destination address written on it (the IP address: "To: 456 Oak Ave, Los Angeles"), which never changes. But at each post office, postal workers add a new routing label (the MAC address) that says "deliver to the next post office's mailroom." These routing labels change at each sorting facility: "Route to Chicago sorting center" → "Route to Denver sorting center" → "Route to LA local post office" → "Deliver to 456 Oak Ave". Each label is specific to the next hop and gets replaced at each facility, while the original addresses on the envelope remain unchanged throughout the journey.

Example: Packet traveling from Host A to Host B through two routers

The diagram below shows how a packet travels from Host A (192.168.1.50) to Host B (10.0.5.100) through two routers, demonstrating how MAC addresses change at each hop while IP addresses remain unchanged:

What happens at each router:

Router receives the Ethernet frame with destination MAC = router's interface MAC
Router strips off the Ethernet header (Layer 2)
Router examines the IP header (Layer 3) to see the destination IP
Router looks up the destination IP in its routing table
Router determines the next hop (another router or the final destination)
Router uses ARP (if needed) to find the MAC address of the next hop
Router creates a new Ethernet frame with:

- Source MAC = router's outgoing interface MAC
- Destination MAC = next hop's MAC address
- The original IP packet (unchanged) as the payload

Why this design matters:

IP addresses provide end-to-end addressing: the packet knows where it's going and where it came from, regardless of the path taken
MAC addresses provide hop-by-hop delivery: each device only needs to know how to reach the next device, not the entire path
This separation allows routing to be flexible: if a router goes down, packets can take a different path, but the IP addresses remain the same
It enables NAT and other middlebox functions: devices in the middle can see and modify the IP packet if needed, but the fundamental source/destination remain

Exception: The one case where IP addresses do change is when NAT is involved. NAT devices (like home routers) rewrite the source IP address (and sometimes destination IP) as packets pass through. However, this is a special case of address translation, not normal routing. In normal routing without NAT, IP addresses remain unchanged.

Q: What is TTL and why is it important?

Every IP packet has a TTL (Time To Live) field that decrements at each router. If it reaches 0, the packet is dropped. TTL is like a "maximum number of post offices" stamp on your envelope. Every time your letter goes through a post office (router), they stamp it with one less number. If your letter has been through 64 post offices and still hasn't arrived, it's probably lost in a loop somewhere, so the post office throws it away. This prevents letters from bouncing between post offices forever if someone made a routing mistake!

Example: Host A (TTL=64) → Router 1 (TTL=63) → Router 2 (TTL=62) → Router 3 (TTL=61) → Host B

Q: What is NAT and how does it work?

NAT (Network Address Translation) allows multiple devices with private IPs to share a single public IP. NAT is like an apartment building's mailroom. You write a letter with your apartment number (private IP like 192.168.1.10) as the return address, but when it goes out to the world, the mailroom clerk changes the return address on the envelope to the building's public address (203.0.113.50) and keeps a note: "Apartment 10's letter is actually from port 40001." When a reply comes back addressed to the building, the clerk looks up their notes and forwards it to your apartment. The outside world never sees your private address!

Q: What are the private IP ranges?

Private IP ranges (RFC 1918):

10.0.0.0/8 — Large enterprises
172.16.0.0/12 — Medium networks
192.168.0.0/16 — Home/small office

Q: What is BGP (Border Gateway Protocol)?

BGP (Border Gateway Protocol) is the routing protocol used to exchange routing information between autonomous systems (ASes) on the internet. It's the protocol that makes the internet work by allowing different networks to learn how to reach each other.

What BGP does:

Exchanges routes: Routers running BGP tell each other which IP address ranges (prefixes) they can reach
Path selection: BGP uses attributes (AS path, local preference, etc.) to choose the best path among multiple options
Loop prevention: BGP prevents routing loops by tracking which autonomous systems a route has passed through
Policy enforcement: Network administrators can set policies to prefer certain paths or block certain routes

BGP in different contexts:

Internet BGP (eBGP): Used between different organizations/ISPs on the public internet. This is what connects the entire internet together. Each organization has an Autonomous System Number (ASN) and advertises their IP ranges to peers.
Internal BGP (iBGP): Used within a single organization to distribute routes between routers in the same autonomous system.
Data center/Cloud BGP: Used in modern data centers and cloud environments to:
- Advertise pod/service IP ranges to network infrastructure
- Enable native routing without overlay encapsulation
- Integrate with cloud provider route tables
- Support large-scale Kubernetes deployments

How BGP works (simplified):

Router A advertises: "I can reach 10.244.0.0/16"
Router B receives this advertisement and stores it in its routing table
Router B can now forward packets destined for 10.244.0.0/16 to Router A
Router B may also advertise this route to other routers (depending on policy)
If Router A goes down or withdraws the route, Router B removes it and finds an alternative path

BGP vs static routes:

Static routes: Manually configured, don't adapt to changes, don't scale
BGP: Dynamic, automatically adapts to network changes, scales to internet size, supports policies

Further reading:

RFC 4271: A Border Gateway Protocol 4 (BGP-4) — The BGP specification
RFC 7938: Use of BGP for Routing in Large-Scale Data Centers — BGP in data centers
BGP Best Practices — Operational best practices

Linux networking primitives

Q: What are the Linux building blocks for container networking?

Before Kubernetes can run pods, Linux provides the building blocks for container isolation: network namespaces, veth pairs, bridges, and iptables.

Q: What is a network namespace?

A network namespace gives a process its own isolated network stack—its own interfaces, routes, and iptables rules. Each container gets its own namespace, completely isolated from the host and other containers.

Q: How do I list network namespaces?

# List network namespaces
$ ip netns list
cni-12345678-abcd-1234-abcd-1234567890ab

# Run a command in a namespace
$ ip netns exec cni-12345678 ip addr

Q: Can I create a network namespce manually?

Yes, you can create a network namespace (netns) manually on a Linux system. This is a common way to experiment with the low-level building blocks that container runtimes use to provide network isolation.

# Create a ns
$  sudo ip netns add test-netns

# Execute commands inside a network namespace
$  sudo ip netns exec test-netns ip link list  
    lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

# Bring up the lo interface
$ sudo ip netns exec my-netns ip link set lo up

Q: How to create isolated network namespace witout sudo?

To create a network namespace without root privileges, you must combine it with a User Namespace. This allows you to map your current unprivileged user to the root user within the isolated environment, granting necessary CAP_NET_ADMIN capabilities to configure that specific network stack.

unshare -r -n /bin/bash

# Create an annonymous network namespace
# -r (--map-root-user): Maps your current UID to 0 (root) inside the namespace.
# -n (--net): Creates a new, empty network namespace.
# /bin/bash: Starts a new shell session inside these namespaces.

Note: The ip netns list command does not list annonymous ns. Instead use lsns -t net to see the created ns.

Q: What is a veth pair?

A veth pair is like a virtual Ethernet cable with two ends. One end stays in the host namespace, the other goes into the container. A veth pair is like a mail slot connecting two rooms. When you drop a letter (packet) into the slot in your room (container), it immediately appears in the other room (host namespace). It's a direct, private connection—like having your own dedicated mail chute that no one else can use.

The operational status of a veth pair is linked. If one end is set to DOWN, the entire virtual link is considered down, mirroring the behavior of a physical cable being unplugged

Q: How to connect two netns with veth pair?

Create a veth pair: One end will stay on the host, and the other will go into the namespace. sudo ip link add veth-host type veth peer name veth-ns
Move one end into the namespace: sudo ip link set veth-ns netns <namespace_name>
Assign IP addresses: Give both ends an IP in the same private subnet (e.g., 10.1.1.0/24).

  #Host: 
  sudo ip addr add 10.1.1.1/24 dev veth-host
  #Namespace: 
  sudo ip netns exec <name> ip addr add 10.1.1.2/24 dev veth-ns``

Bring interfaces up:

  sudo ip link set veth-host up
  sudo ip netns exec <name> ip link set veth-ns up

Access: You can now reach your application by hitting the namespace's IP (10.1.1.2) from the host.

Q: What is a bridge?

A bridge is a virtual Layer 2 switch inside the kernel. It connects all the veth pairs together. A bridge is like a shared mailroom in an apartment building. Each apartment (container) has its own mail slot (veth pair) connecting to the mailroom (bridge). When you send a letter to your neighbor, you drop it in your slot, it arrives in the mailroom, and the mailroom knows which slot belongs to your neighbor and delivers it there. The mailroom (bridge) learns which apartment (container) is connected to which slot (MAC address) by watching the return addresses on letters.

Q: What is TAP? When do I need TAP vs vEth?

While veth pairs are the standard for connecting two kernel-level entities (like two network namespaces), TAP interfaces are essential for scenarios where the network traffic must be handled by user-space software. vEth is a linux primitive while a file-based interface for raw frames is universal.

If veth is a direct pneumatic tube (Kernel-to-Kernel), TAP is a digital scanner (Kernel-to-Software). The usecases below shows different types of software.

Usecases:

Virtual Machines (LimaVM, QEMU): Provides the "hardware" network card for a VM. The hypervisor reads frames from the TAP device and injects them into the guest OS.
Rootless Networking: Tools like slirp4netns (used by Podman and Lima's user-v2) use TAP to provide internet access to containers without requiring sudo.
Network Monitoring: Used to capture and analyze raw traffic for security (IDS/Firewalls) without disrupting the actual flow.

Easiest way to remember this is by using the access method.

Method	Primitives Used	Why it's used
Rootful (containers)	veth pair + Bridge	Used by standard Docker/Kubernetes; requires root to create virtual links.
Rootful (VMs)	TAP interface + Bridge	Used by KVM/Proxmox to link the Kernel to Hypervisor software.
Rootless (Unprivileged)	TAP interface + slirp4netns	Used for rootless containers; bypasses root requirements by using user-space networking.

Note: In Rootless mode: Ip address assigned to the TAP interface is not visible to host and the host cannot ping that ip.

Q: Does TAP connect to bridge?

A TAP interface (like vnet0) can be created and manually attached to a Linux Bridge (br0) in the host kernel using administrative privileges.

A Rootless example:

Imagine your host machine is the building, but you aren't the landlord—you're just a tenant. You aren't allowed to install new pneumatic tubes (veth) or modify the main sorting table (Linux Bridge). Lets take an example of Lima VM Network

The "Virtual Office" (The Lima VM): You are running a VM. It needs a network.
The TAP Slot (Inside the Namespace): Lima creates a TAP interface inside a private namespace. This is your "Digital Mail Slot."
The Software Clerk (The User-v2 Daemon): Instead of a physical sorting table, there is a Software Clerk (a process running on your host). This clerk has "hands" on the TAP slot.
The "Virtual Bridge": If you have two Lima VMs, they both have TAP slots. The Software Clerk holds both TAP slots in its hands. When VM1 sends a letter, the clerk reads it from the first TAP slot and manually "tosses" it into the second TAP slot.

In this scenario, the "Bridge" is just a piece of logic inside the clerk’s brain (the software code).

Q: What is iptables?

iptables (and its successor nftables) is how Linux manipulates network traffic. iptables is like a postal inspector with a rulebook. As letters (packets) flow through the post office (Linux kernel), the inspector checks each one against the rules: "Letters to 10.96.0.100? Change the address to 10.244.1.5. Letters from 10.0.0.5? Block them. Letters to port 80? Route them to port 8080 instead." The inspector can rewrite addresses, block letters, or redirect them—all without the sender or receiver knowing.

Q: What are the iptables chains?

In Linux, the processing of packets follows a strict sequence of tables within each hook. The tables are listed below in their actual order of execution for each hook.

PREROUTING: Applied to all incoming packets before a routing decision is made.
INPUT: Applied to packets destined for a local process/socket.
FORWARD: Applied to packets routed through the host (Pod-to-Pod on different nodes).
OUTPUT: Applied to packets generated by a local process.
POSTROUTING: Applied to all outgoing packets after routing is complete.

Table Execution Order by Hook

Hook (Chain)	1st Table	2nd Table	3rd Table	4th Table	5th Table
PREROUTING	raw	mangle	nat (DNAT)	-	-
INPUT	mangle	filter	security	nat (SNAT*)	-
FORWARD	mangle	filter	security	-	-
OUTPUT	raw	mangle	nat (DNAT)	filter	security
POSTROUTING	mangle	nat (SNAT)	-	-	-

Note: The nat table in the INPUT chain was introduced in later kernel versions to allow SNAT for traffic destined for the local host.

Table Function Definitions

Table	Purpose	Common Targets
raw	De-prioritizes connection tracking.	NOTRACK, DROP
mangle	Modifies IP header fields (TTL, TOS) or marks packets.	MARK, TOS, TTL
nat	Changes Source or Destination IP/Ports.	SNAT, DNAT, MASQUERADE, REDIRECT
filter	The "Firewall." Decisions on packet delivery.	ACCEPT, DROP, REJECT
security	Implements SELinux security context marks.	SECMARK, CONNSECMARK

Q: How does l2 bridge and iptables work together?

In Linux, Layer 2 (L2) bridges and iptables (which typically operates at Layer 3) work together through a kernel bridge netfilter framework. This interaction allows the system to apply advanced IP-level filtering and NAT to traffic that would otherwise stay purely at the Ethernet frame level. Normally, an L2 bridge forwards traffic based on MAC addresses, bypassing the L3 IP stack where iptables resides. To bridge this gap, Linux uses the br_netfilter kernel module.

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sudo sysctl --system

Q: What is Linux IPVS (IP Virtual Server)?

IPVS (IP Virtual Server) is a Linux kernel feature that provides Layer 4 load balancing. It's built into the Linux kernel and operates at the network layer, making it faster and more efficient than iptables for load balancing scenarios.

How IPVS works:

IPVS creates a virtual IP (VIP) that represents a service
Traffic to the VIP is distributed across multiple real servers (backend pods) using load balancing algorithms
IPVS maintains a connection table in kernel memory, tracking active connections
Load balancing happens in the kernel, avoiding the overhead of userspace processing

IPVS vs iptables for load balancing:

iptables: Uses NAT rules (DNAT) to rewrite destination IPs. With many services, the iptables rule chain becomes long, and every packet must traverse the chain until it matches. This is O(n) complexity—the more rules, the longer it takes.
IPVS: Uses a hash table for O(1) lookup of backend servers. More efficient for large numbers of services (thousands). Also supports more load balancing algorithms (round-robin, least connections, source hashing, etc.).

When to use IPVS:

Large clusters with many services (1000+)
Need better performance and lower latency
Want more load balancing algorithm options
Can enable IPVS kernel modules (ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh, etc.)

When to use iptables:

Smaller clusters (< 1000 services)
Simpler setup (no kernel modules needed)
Default and well-tested option

Layer 4: Which Application?

Q: What does Layer 4 add to networking?

Layer 4 adds ports to identify which application should receive the data.

Q: What are ports and why do we need them?

A single IP can run many services. Ports (0-65535) identify each one. An IP address is like a building address, and ports are like apartment numbers or department mailboxes. When you send mail to "123 Main St, Apartment 80" (IP:port), the mailroom knows to deliver it to the web server department (port 80), not the database department (port 5432). One building (one IP) can have many departments (many ports), each handling different types of mail!

Q: What are some well-known ports?

22 → SSH
80 → HTTP
443 → HTTPS
53 → DNS

Q: What is a 5-tuple?

A connection is uniquely identified by the 5-tuple:

Protocol: TCP or UDP
Source IP
Source Port
Destination IP
Destination Port

Q: What's the difference between TCP and UDP?

Aspect	TCP	UDP
Connection	Connection-oriented (handshake first)	Connectionless (fire and forget)
Reliability	Guaranteed delivery, ordering	No guarantees
Use case	HTTP, SSH, Database queries	DNS, Video streaming, VXLAN tunnels
Overhead	Higher (acknowledgments, retries)	Lower (just send it)
Header size	20+ bytes (with options)	8 bytes (fixed)

TCP is like registered mail with delivery confirmation. You send a letter, the recipient signs for it and sends back a confirmation card. If you don't get the confirmation, you send another letter. The postal service guarantees your letter arrives in order. UDP is like regular mail—you drop it in the mailbox and hope it gets there. It's faster and cheaper, but there's no guarantee. For important documents (web pages, database queries), you use registered mail (TCP). For quick notes where losing one doesn't matter (video streaming, DNS lookups), you use regular mail (UDP).

Q: What is the TCP three-way handshake?

Before TCP can send data, it establishes a connection through a three-way handshake. This ensures both sides are ready to communicate and agree on initial sequence numbers. Think of it like a phone call: you dial (SYN), the other person picks up and says "hello" (SYN-ACK), and you confirm "yes, I can hear you" (ACK). Only then do you start talking.

Q: What is the TCP connection termination process?

TCP uses a four-way handshake to gracefully terminate a connection:

FIN: The sender sends a FIN (finish) to indicate it has no more data to send.
ACK: The receiver acknowledges the FIN.
FIN: The receiver sends its own FIN to indicate it has finished sending data.
ACK: The sender acknowledges the receiver's FIN.

Q: What is the TCP sliding window and why we need it?

Why We Use It
Without a sliding window, TCP would be "Stop-and-Wait": the sender would send one packet and wait for an acknowledgment (ACK) before sending the next. This would be incredibly slow, especially on high-latency links.
The sliding window solves two problems:

Throughput Efficiency: It allows the sender to have multiple packets "in flight" at once, filling the network "pipe."
Buffer Protection: It prevents the receiver's memory buffer from overflowing. If the receiver's application is slow (e.g., a slow disk write), the window shrinks to tell the sender to slow down.

How It Works
The control of the sliding window is a dynamic "handshake" between the receiver and the sender.

The Receiver's Role (Flow Control)
The receiver controls the window size through a field in the TCP header called the Receive Window (rwnd).
Advertising: In every ACK sent back to the sender, the receiver includes the current size of its available buffer.
Zero Window: If the receiver's buffer is completely full, it sends an ACK with a window size of 0. The sender then stops transmitting and periodically sends "Zero Window Probes" to see if space has opened up.

The Sender's Role (Congestion Control)
The sender does not just blindly follow the receiver's advertised window. It maintains its own internal limit called the Congestion Window (cwnd), based on how much the network (routers/switches) can handle.
The Formula: The actual amount of data sent is always min(rwnd, cwnd).

Scaling (Window Scaling)
The original TCP specification limited the window size to 65,535 bytes (64 KB). On modern high-speed networks (10Gbps+), this is too small.

TCP Window Scale Option: This allows the window to be scaled up to 1 GB.
Configuration: On Linux, this is controlled by the sysctl parameter: net.ipv4.tcp_window_scaling = 1

Tuning the Buffers
While the window slides automatically, you control the maximum potential size of that window by adjusting the Linux network buffer limits:

  Read Buffer: net.ipv4.tcp_rmem (min, default, max)
  Write Buffer: net.ipv4.tcp_wmem (min, default, max)

By increasing these values in /etc/sysctl.conf, you allow the sliding window to grow larger, which is essential for high-latency, high-bandwidth connections (like communicating between data centers across continents)

Part 2: Network Virtualization Technologies

VLAN: Network Segmentation

Q: What is VLAN?

A VLAN (Virtual Local Area Network) is a logical network segment created within a physical network. It allows you to group devices together logically, even if they're not physically connected to the same switch. VLANs are identified by a VLAN ID (a number from 1-4094) that is added to Ethernet frames as a tag. Think of VLANs as creating separate "virtual neighborhoods" within the same physical building—devices in VLAN 10 can't directly communicate with devices in VLAN 20, even though they might be connected to the same physical switch, just like people in different apartment buildings on the same street.

Q: What problem did VLANs solve?

In the early days, Ethernet was a "flat" network where every device heard everyone else's broadcasts. W. David Sincoskie invented the VLAN at Bellcore in the 1980s to break these large, noisy broadcast domains into smaller, manageable logical groups. The technology was later standardized as IEEE 802.1Q.

Q: How do VLANs work?

A VLAN adds a 4-byte 802.1Q tag to the Ethernet frame. The switch reads this tag and only forwards the frame to ports in the same VLAN. Think of VLANs as colored envelopes. When you send a letter in a blue envelope (VLAN 10), the mail carrier (switch) only delivers it to mailboxes that accept blue envelopes. Letters in green envelopes (VLAN 20) go to different mailboxes. Even though all the mailboxes are on the same street (same physical switch), the colored envelopes keep the mail separated—blue letters never mix with green letters.

Q: Can you walk through a VLAN example?

When Host A (192.168.10.5) sends to Host B (192.168.10.6) on VLAN 10, the switch reads the 802.1Q tag and forwards the frame only to ports in VLAN 10, ensuring Host C on VLAN 20 never sees the traffic:

Q: What were the constraints of VLANs?

Physical port binding: VLANs were tied to the physical switch port. If you moved your desk, a network engineer had to manually reconfigure the switch.
The 4,094 ceiling: With only a 12-bit ID, you could only have 4,094 usable networks—plenty for an office, but a disaster for the upcoming cloud era.

VLANs were like having only 4,094 different envelope colors available. Once you used all the colors, you couldn't create new networks. Also, if you moved to a different building (different switch port), you had to tell the mailroom "I'm now using blue envelopes instead of green," and they had to manually update their records. This didn't work well when people (VMs) were moving constantly!

VXLAN: Network Virtualization

Q: What is VXLAN?

VXLAN (Virtual eXtensible Local Area Network) is a network virtualization technology that encapsulates Layer 2 Ethernet frames inside Layer 3 UDP packets. This creates an "overlay network" that allows VMs and containers to communicate as if they're on the same local network, even when they're on different physical servers or data centers. VXLAN uses a 24-bit Virtual Network Identifier (VNI) to create up to 16.7 million logical networks, far exceeding VLAN's 4,094 limit. The key innovation is that VXLAN decouples the logical network from the physical network infrastructure—VMs can move between physical servers without changing their network identity, and the physical network only sees IP traffic between servers, not the virtual network details.

Think of VXLAN like putting an envelope inside another envelope. You write your letter (original L2 frame with VM's MAC addresses) and put it in an inner envelope addressed to the destination VM. Then you put that inner envelope inside an outer envelope addressed to the destination server (VTEP IP address). The postal service (physical network) only looks at the outer envelope and delivers it to the server. The server then opens the outer envelope, takes out the inner envelope, and delivers it to the VM. The postal service never sees what's inside—they just see mail between servers!

In a multi-tenant environment, VXLAN is a cornerstone. It allows different tenants to have their own logically isolated networks (using unique VNIs) that share the same underlying physical infrastructure, preventing tenants from seeing each other's traffic.

Q: What is the VXLAN architecture?

Overlay Network (Virtual): VMs think they're on the same L2 segment
Underlay Network (Physical): Physical network routes between VTEPs (VXLAN Tunnel End Points)

Q: How many networks does VXLAN support?

The physical switches just saw traffic between servers, while the VMs felt like they're on one giant, 16.7-million-segment logical switch (thanks to the 24-bit VNI: 2^24 = 16,777,216 possible networks).

Q: Can you walk through a VXLAN example flow?

The diagram below shows the complete VXLAN encapsulation and decapsulation process:

Q: Why are they called tunnels?

They are called tunnels because they create a private, direct-path "shortcut" for your data through an existing network, similar to how a physical tunnel allows a car to pass through a mountain instead of driving over every peak. In networking, a "tunnel" isn't a physical wire; it is a logical path created by encapsulation.

Here's how it works: You write a letter to your friend (original L2 frame with VM MAC addresses). You put it in an inner envelope addressed to your friend's apartment (destination VM). Then you put that inner envelope inside an outer envelope addressed to your friend's building (VTEP IP address). The outer envelope has a special label (VNI) that says "Building 5000" so the receiving building knows which floor to deliver it to. The postal service (physical network) only looks at the address on the outer envelope. They see "Deliver to Building 10.0.0.2" and route it there. They have no idea there's another envelope inside, or that it's really meant for someone in apartment 192.168.10.6. When the letter arrives at Building 10.0.0.2, the mailroom (VTEP) opens the outer envelope, reads the VNI label ("Building 5000"), and delivers the inner envelope to the correct apartment (VM). Your friend receives the letter as if you sent it directly—they never see the outer envelope! VXLAN is the "outer envelope," and the VTEP (VXLAN Tunnel End Point) is the "building's mailroom" that handles the envelope wrapping and unwrapping.

Q: How do VTEPs discover each other?

VXLAN requires a control plane to map VM MAC addresses to VTEP IP addresses. Common approaches:

Method	Description
Multicast	VTEPs join multicast groups per VNI. Broadcast ARP requests are sent via multicast. Simple but requires multicast support in underlay.
BGP-EVPN	BGP extensions for Ethernet VPN (RFC 7432). VTEPs exchange MAC/IP routes via BGP. Used in large-scale deployments (Cisco ACI, Juniper).
Centralized Controller	SDN controller (e.g., VMware NSX, OpenStack Neutron) maintains MAC-to-VTEP mappings. VTEPs query controller for unknown destinations.
Distributed Database	etcd or similar stores MAC-to-VTEP mappings. Used by container networking plugins.

Q: Why does VXLAN use UDP?

VXLAN uses UDP (User Datagram Protocol) as its transport protocol for several important reasons:

Reliability is handled at a higher layer: The inner Ethernet frame already contains TCP/IP traffic, which provides its own reliability mechanisms. If a TCP packet inside the VXLAN tunnel is lost, TCP will retransmit it. Adding TCP reliability at the tunnel level would create redundant retransmissions and actually hurt performance.
Lower overhead: UDP has a fixed 8-byte header compared to TCP's 20+ byte header (which can grow with options). For tunnel traffic that may carry thousands of packets per second, this overhead reduction matters significantly.
Hardware offloading: Modern network interface cards (NICs) can offload UDP encapsulation/decapsulation to hardware, improving performance. TCP's stateful nature makes hardware offloading more complex and less efficient.
No connection state: UDP is connectionless, meaning there's no connection establishment (three-way handshake) or teardown overhead. This is crucial for tunnel traffic where you want to forward packets as quickly as possible without maintaining connection state.
Avoids TCP-in-TCP problems: If VXLAN used TCP, you'd have TCP inside TCP. This creates problems like:
- Head-of-line blocking: If one TCP segment is lost, all subsequent segments wait
- Congestion control conflicts: Inner and outer TCP connections compete
- Retransmission storms: Both layers trying to retransmit the same data

Q: What are the MTU and fragmentation considerations for VXLAN?

VXLAN encapsulation adds approximately 50 bytes to each packet:

Outer Ethernet: 14 bytes
Outer IP: 20 bytes
Outer UDP: 8 bytes
VXLAN header: 8 bytes
Total overhead: ~50 bytes

If the underlay MTU is 1500 bytes (standard Ethernet), the effective overlay MTU becomes 1450 bytes. Packets larger than this will be fragmented, causing performance degradation.

Q: How do I avoid VXLAN fragmentation?

💡 Avoiding VXLAN Fragmentation

Configure underlay MTU to 1550+ bytes (jumbo frames) to avoid fragmentation, or reduce overlay MTU to 1450 bytes.

⚠️ Operator smell for MTU issues

Cross-node traffic works for small payloads but gRPC/HTTPS calls with larger bodies RST or time out. Quick test: ping -M do -s 1472 <remote-node-ip>; if it fails, drop pod MTU to 1450 or raise underlay MTU.

Q: What are the constraints of VXLAN?

Fixed 8-byte header: No room for custom metadata beyond the VNI
Limited extensibility: Can't carry security policies or telemetry inline
Control plane dependency: Requires additional infrastructure for MAC-to-VTEP discovery

Geneve: Extensible Network Virtualization

Q: What problem did Geneve solve?

As we moved into containers and cloud-native platforms, even VXLAN started to show its age. Modern platforms needed to carry more than just a "Network ID"—they needed to carry security policies, telemetry, and "who is talking to whom".

Q: What is Geneve?

Geneve (Generic Network Virtualization Encapsulation) arrived to solve the "fixed header" problem of VXLAN. Its extensible design allows developers to add custom data (Type-Length-Value options) to every packet, which is critical for the complex routing and security required by modern SDN platforms like VMware NSX and cloud-native networking solutions.

Geneve is like VXLAN's envelope-inside-envelope, but with sticky notes attached to the outer envelope. You still put your letter (original packet) in an inner envelope, then put that in an outer envelope. But now you can attach metadata stickers to the outer envelope: "Security Policy: Allow-123", "Source: frontend-workload", "Telemetry: latency-tracked". The receiving building (VTEP) reads these stickers before opening the envelope, so it knows how to handle the letter—check security permissions, log metrics, route based on identity. VXLAN's outer envelope was blank except for the address; Geneve's outer envelope is covered in useful information!

For enterprise Kubernetes multi-tenancy, Geneve's extensible TLV options become crucial for enforcing fine-grained network policies and carrying tenant-specific metadata, allowing a single underlying network to enforce diverse security rules for multiple isolated tenants.

Key difference from VXLAN: VXLAN has a fixed 8-byte header, while Geneve has a variable-length header (8+ bytes) that can include TLV options for extensibility. This allows Geneve to carry metadata like security policies and telemetry inline with each packet.

Q: What are examples of Geneve TLV options?

Geneve TLV options can carry various types of metadata. Common examples include: Security Policy ID (Class 0x0102, Type 1) containing policy identifiers like "policy-xyz-123"; Telemetry Data (Class 0x0103, Type 2) with metrics such as "latency=5ms, hop=3"; and Source Identity (Class 0x0104, Type 3) identifying workloads like "workload=frontend-abc". These options allow the network to enforce security policies and collect observability data at the packet level without requiring separate control plane messages.

Q: Can you show an example of Geneve with security metadata?

In cloud-native environments, Geneve TLV options carry security policies and source identity. When the frontend workload sends a packet, it's like writing a letter and putting it in an inner envelope. The SDN controller (like a security guard) checks the sender's ID, looks up the security policy, and attaches stickers to the outer envelope: "From: frontend-workload", "Policy: allow-frontend-to-backend", "Security Level: High". When the letter arrives at the destination building, the security guard there reads the stickers, verifies "Yes, frontend is allowed to talk to backend," and only then opens the envelope and delivers it. If the stickers said "Deny," the letter would be rejected without even opening it!

Q: What are the MTU considerations for Geneve?

Geneve overhead is variable due to TLV options:

Base overhead: ~38 bytes (Ethernet + IP + UDP + Geneve base header)
TLV options: 0-252 bytes (variable)
Total overhead: 38-290 bytes

Configure underlay MTU accordingly. For example, with 100 bytes of TLV options, you need at least 1600 bytes MTU to avoid fragmentation.

Q: What are the constraints of Geneve?

Processing overhead: Variable-length options require more parsing than fixed headers
Hardware support: Older NICs may not offload Geneve efficiently, especially with TLV options
Complexity: TLV parsing adds CPU overhead compared to VXLAN's simple header

References

Standards and RFCs

RFC 826: Ethernet Address Resolution Protocol (ARP)
RFC 791: Internet Protocol (IP)
RFC 793: Transmission Control Protocol (TCP)
RFC 768: User Datagram Protocol (UDP)
RFC 1918: Address Allocation for Private Internets
RFC 7348: Virtual eXtensible Local Area Network (VXLAN)
RFC 8926: Geneve: Generic Network Virtualization Encapsulation
RFC 7432: BGP MPLS-Based Ethernet VPN (BGP-EVPN)
IEEE 802.1Q: Virtual Bridged Local Area Networks

This article is part of the "Learning in a Hurry" series, designed to help engineers quickly understand complex technical concepts through analogies and practical examples.

Improve your creativity by deliberate practice

Yuva — Wed, 05 Jun 2019 23:57:39 +0000

My name is Yuva and I am a Software Engineer. I have been engineering and product-ing for 14 years. I have never thought of me as a creative person.

I looked at people playing music that are in their element, eyes closed and their face full of joy, completely immersed in what they are doing. They are in creative flow!

I yearned for that feeling of creative flow. In reality, I have experienced that feeling in my software engineering career few times. My very first time was in 2007, I solved a problem in a very creative way. I was brimming with joy. I put my hand in the air and punched it few times. I high-fived every person that passed me. I was a very shy developer back then so it was a big deal. Those experiences were few and far between. I want to experience the creative bursts very often.

I know one thing for sure. I work hard and I was determined to learn any skill necessary to get to experience that emotional state again and often.

Do you think you are creative?

If you don't, I am going to try to make you believe you are. If you do think you are creative, I am going to give you some tools that worked for me.

Talent and Creativity

First thing I learned is that I confused talent for creativity. Talent is something we are innately capable of doing. Some talents are useful like that of an NBA legend Wilt Chamberlin. When you are towering 7ft in high-school, you may as well play basketball.

Some talents are not so useful. I can do a wall sit for 10 mins.

Like the old proverb goes
Talent is like an asshole, everyone has one

Wait!? that does not sound right. oh! Well, I am going to run with it.

Talent and creativity are not the same thing. Creativity is not an elusive thing that only a select few people posses. Creativity belongs to all of us and we make many creative decisions everyday, often not consciously.

What is creativity?

Scientists explain creativity as making connections between seemingly odd or different ideas from the pool of knowledge we store in our brains to solve a new problem or create something that is novel, good and useful.

Its our brains doing what they do says Michael Grybko, Research Scientist, University of Washington

I think I got this one. I started seeing The Eureka Moment everywhere. Entertainment industry has given us so many examples of creative genius at play. Like the famous middle-out compression algorithm idea that Richard Hendricks(Silicon Valley) gets from seemingly unrelated conversation.

I cannot unsee it. It has almost become predictable. There is a team that struggles to solve a complicated problem and the protagonist would watch someone do something very normal like pouring a coffee, bouncing a ball, fighting with a vending machine and the protagonist's eyes will light up. Cue the eureka moment.

If creativity is a natural brain activity? Why am I not generating a lot of creative ideas on a daily basis?

Before I try to figure out the answer, let's bust some more myths.

Myth busting time

I have to be right brained
This is one of the physcology fads that was disproportionately exaggerated like the Myers briggs personality types.Creativity required both sides of the brain.
I need drugs
- I wanted this to be true so bad. I had convinced myself, when I take ayahuasca or LSD, all truths will be revealed. Unfortunately science does not backup this claim. I have tried CBD and it has only put me to sleep so far.
- LSD and Cannabis is mostly associated the creatives. This is one for the books where correlation is mistaken for causation.
I have to wait for the apple to hit my head
- Even without trying very hard, I have experienced bouts of creative moments few times in my life. Advances in neuroscience had made me believe that I do not have to wait around for the next bright idea to fall in my lap.

I feel so educated already. I understood creativity as science and tv describes it. I have busted some myths I had for long time. I feel ready for the next step. How to deliberately generate creative ideas?

I know my brain is constantly making connections. I dreamt about a problem I had at work and my cat, lullaby, was in my dream solving it. She is a very smart cat but I don't think I can use that idea in real life.

Creativity de-mystified

Let's break down the definition of creativity.

Memory

More things I have in memory, more likely that I can make a connection. I used to remember phone numbers of all my friends and relatives in memory. Our brains are constantly collecting information and encoding it to create short term memory. When things are repetitive, the brain skips remembering the details.
We are more likely to convert the short term memory into long term memory when there is an association or impact. Most of us remember exactly where we were, even what we were wearing or whom we were with when 9/11 happened.
The long term declarative memory is where we keep our pool of information and this is very important to fuel creativity.
The knowledge pool
Next piece of the creativity puzzle is the knowledge. In-order make connections between loosely coupled things, we need to know about those things. If I do not have any knowledge about the domain that I am working in, it is highly unlikely that I will come up with creative solutions.
Recall
This is the part where the brain makes connections between loosely related topics to come up with creative ideas. This is the most important and difficult part of becoming deliberately creative.

Five practical ways to start improving our ability to make creative neural connections.

Improve declarative memory
- Chunking techniques
  
  Even though chunking techniques are associated with improving the working memory, chunking involves searching for patterns to chunk, noticing patterns and remembering them are valuable exercises that help in improving declarative memory and recall.
  
  Consciousness and chunking allow us to turn the dull sludge of independent episodes in our lives into a shimmering, dense web, interlinked by all the myriad patterns we spot. It becomes a positive feedback loop, making the detection of new connections even easier, and creates a domain ripe for understanding how things actually work, of reaching that supremely powerful realm of discerning the mechanism of things. At the same time, our memory system becomes far more efficient, effective — and intelligent — than it could ever be without such refined methods to extract useful structure from raw data.
  
  Deep reading on pattern recognition
- Memory palace technique
  A Memory Palace is an imaginary location in the mind to store mnemonic images. The most common type of memory palace involves making a journey through a place well known to the person, like a building or town.
- Using the human connection
  Teach someone else or engage in a debate about any random topic. This also helps in providing another datapoint for getting that information into the declarative memory.
Improving the knowledge pool
- Be curious about the world around. Be present. I started to notice things when I go on walks and ask a simple why questions. This would lead me into long wikipedia trail from which I retain some of it. I end up sharing what I learned with my wife. I recently learned our nervous system is only 500mil years old while life existed for 4 billion years. Fascinating! right?
- Read books / listen to podcasts. Select some books and podcasts that are unrelated or distantly related from the main field of interest. Psychology and economics related podcasts and books are my interests outside of software engineering.
Take a walk before switching contexts or learning new things
- Before going into any important meetings, especially brainstorming sessions, take a small walk to clear the mind, breathe deep and think about the topic of the meeting. This seems very simple but there has been experiments done to show the benefits of increasing the blood flow and just changing the scenery in taking on challenging tasks. I have the calendar set to always schedule my meetings to end in 10 mins before an hour. Even though we like to group the meetings, make sure to give at-least 10 minute break between them and go on a walk.
Multitask at a slower phase
- I have been doing this for a while. In the beginning, I picked two projects with deadlines. This is the wrong way to do it. Have one active project that has deadlines, namely an engineering project at work and pick another on that I can do in leisure. Writing and making lunch n learn presentations have been my side project for a bit and I am having a blast. I have found out that I really enjoy writing and every time I needed a break from work, I start researching my writing topic or start writing/editing. Sometimes wrong answer is stuck in in head. Changing context helps to flush the stuck solution.
- Easy to think outside of the box if you can move from one box to the other
- Many successful creative people have serious hobbies. Richard Feynman was obsessed with cracking safes or Charles Darwin was obsessed with earthworms.
Marinate the ideas
- The early mornings and right before bed times are prime time for making neural connections. I love waking up at 5:30AM but stay in that lucid state while thinking about my presentations or ways to represent a complex idea or think of the things to do for the day.
- Think about the problem you are working on right before you go to sleep or just before you completely wake up. The brain is yet to be bombarded with tons of sensory information that it can process the problem at hand easier.

Creativity in teams

Creativity can come from groups not just individuals. If we want to be creative as a team, we should also practice the creativity boosting strategies together.

Brainstorming is one of the common tools used by companies to explore the group creativity.

Tips to improve brainstorming:

There are no bad ideas - is a bad idea There are bad ideas and we need to call them out. Not to humiliate the person providing the idea but to set a standard for the ideas that are being generated. This can only be possible if there is a deep understanding between the members and everyone feels safe to throw out half baked ideas while also be willing to be criticized. Structure and safety are two most important elements to conduct a productive brainstorming meeting.
Go for quantity of ideas rather than quality
Build on top each other's ideas. The group owns all the ideas that they generated and keeping the generation and validation separate helps remove the biases. The person that generated the idea may not be the same person that is championing for the idea. This way, an idea gets its attention whether the idea is generated by a very shy introvert in the room or someone that commands attention by their communication skills.

Thank you for reading! I would love to receive any comments or feedback to improve the content or my writing style. Please leave a comment if you want me to write another article expanding on creativity in groups.

A Beginners’s Guide to Serverless, Faas, and serverless web architecture

Yuva — Tue, 21 May 2019 16:18:21 +0000

Lately we hear about serverless everywhere, as a natural progression from monolith to microservices to serverless.

My favorite definition of Serverless is from urban dictionary

The name of a fad everyone loves to hate. Where the architectural model is shifted from running processes to running functions with *no control* or need to integrate with the operating system.

Why is no control so contested? As with any trade off, the answer is: it depends. Less control also means less responsibility, so is less control always a bad thing? If you are deploying simple, stateless, non-CPU intensive applications then a serverless architecture has a huge advantage: agility. Templated and reusable pieces let us spin up experimental apps very quickly without the responsibility of managing and patching any infrastructure at all.`

Microservices vs Serverless

A big difference between microservices and serverless is their execution lifetime. A microservice is expected to be always available while serverless functions come alive when needed, execute, and terminate.

IaaC — Infra as as service is like renting the land and parking an RV on it to live in it. You own the RV and everything else in it to make it habitable.

PaaC — Platform as a service is like leasing a house and living in it. The owner takes care of the house and its improvements while the renter makes it livable.

FaaC — Function as a service is like staying in a hotel room whenever you need a place to stay.

Reasons to use Serverless

Only pay for the time it is executing.
Our service doesn’t have to be up and running all the time.
Increases development speed. Easy to build and deploy.
Automatic scaling.
Platform agnostic. Just build the business logic and move it to any platform provider.

Reasons not to use Serverless

Decomposing the application into small functions might not always be the best solution.
If there is a state that needs to be exchanged between functions, inter-function communication and some expensive plumbing is necessary.
Complex error handling can be difficult. More operational overhead when maintaining 100s of functions.
Using native libraries is difficult and not recommended.
Memory, CPU and network are limited to what is provided.

When to use Serverless?

If we have a simple, stateless, non CPU intensive applications are perfect for serveless. AWS recommends using lambdas for the following use cases

Data processing solutions

File processing
Stream processing

Backends

IOT
Mobile
Web

Serverless web app architecture

My search for reference architectures on deploying a simple web application using Serverless turned up the architecture below

The front end:

A storage service to keep all the web assets (html, css, js, images).
A distribution system to manage the caching and availability of the site.
A DNS service to map custom domain to the cloud distribution system.

Serverless back end:

API Gateway to expose API.
Serverless functions to execute the business logic.
Database to persist state. — Database can be SQL or document/key-value database. The selection depends on the way you interact with the data in the store. If the way we retrieve the information is always going to be the same, a document database like dynamodb works great. Watch Martin Fowler’s talk on noSQL to get an understanding.

This three tier architecture is provider agnostic and can be used in Google Cloud Platform or Azure Cloud Services or IBM Open Whisk.

Serverless architecture in Google Cloud Platform

Which service provider to chose?

Most serverless providers have similar solutions and functionality. The solution that works best will depend on what you, your team, and organization know and requirements of your application. If you have no constraints, I’d recommended getting started in AWS.

	AWS Lambda	Google Cloud Functions	Azure Functions
Pricing	1M/month free	2M/month free	1M/month free
Languages	Node.js, Python, Java, .NET	JavaScript (many in beta)	Node.js, Python, PHP, C#, F#
Triggers	API, S3, DynamoDB	HTTP, Any GCF services (firebase, analytics, Pub/Sub, Storage)	API, Cron, Azure Events, Azure
Max execution time	15 mins	1 min - 9 mins(upgrades)	5 mins - 10 mins(upgrades)
Concurrency	1000 executions	Unlimited for HTTP. 1000 executions	Unlimited

What if we need more of an open source solution

You are covered. There are many open source Serverless solutions out there!

Rohit Akiwatkar wrote a comprehensive article about the Serverless open source solutions.

References

Next up: Understanding AWS Lambda

Understanding AWS Lambda

Yuva — Tue, 21 May 2019 16:17:39 +0000

At Peaksware, we use AWS solutions for our serverless needs. Understanding Lambda functions involves

How AWS scales up/down and picks up code changes?
Understanding AWS Lambda companions like Layers and Step functions.
Coding best practices when writing lambda functions.

Benchmarking information in this section is summarized from a detailed paper written by Wang Liang and colleagues — Peeking Behind the Curtains of Serverless Platforms

Scaling up Lambda

When a request comes in, AWS creates a container on top of a VM in one of AWS’s hosts. The container has the runtime and the function code. The function is called and lambda waits for the execution to finish. The execution stops on

Successful completion of the function code
Exception in the function code
Execution timeout

As more requests comes in, AWS scheduler could use the same container, spin up new containers in the same VM or add more hosts+VMs. AWS handles concurrent requests by initializing up to 1000 concurrent containers. The containers are packed tightly within a VM to optimize for the VM’s memory utilization. Even two different functions from the same AWS account can potentially share the same VM.

Cold start

A cold start may involve launching a new container, setting up the runtime environment, and deploying a function, which will take more time to handle a request than reusing an existing container.
Cold start latency decreases as the allocated function memory increases. One possible explanation is that AWS allocates CPU power proportionally to the memory size; with more CPU power, environment set up becomes faster.

Scaling down Lambda

AWS will keep the container idle for arbitrary amount of time to respond to the requests faster. AWS will shut down half of the idle instances of a function approximately every 300 seconds until there are two or three instances left, and eventually shut down the remaining instances after 27 minutes. A virtual machine can stay up for 9hrs.

Updating Lambda code

There is a small chance that incoming requests could be handled by an old version of the function. The scheduler is not atomic, if you update a function while there are 50 or more concurrent requests hitting the function, 3.8% of instances might run an inconsistent version. 6 seconds seem to be the ideal time between function update and requests to avoid running older version of the function.

You can also prevent this by using blue-green deployment methods using code deploy instead of updating the function directly.

Lambda Layers

Lambda functions are executed in containers and lambda layers is a logical extension of existing lambda functionality.

The code size (including all the libs used by the function code) should be under 50MB.
Layers are a nice way to package the dependencies separately so the deployment package size is reduced.
Layers are useful to share common code across multiple functions.
A function can have up to 5 layers and have a max code size of 250MB.

AWS Step functions

AWS step functions allows us to create a state machine by chaining together Lambda functions.

In distributed microservice based architecture world, coordination between microservices is done using a pub-sub or event stream model. AWS has taken a stab at abstracting the communication between loosely dependent functions using step functions. Step functions can be seen as orchestrator of functions that helps create and visualize the entire workflow.

Use cases for step functions

ETL data processing
Move few tasks from a monolith to serverless.
Orchestrate Lambda functions into a Service.

AWS documentation provides more use cases and examples.

Writing code for Lambda Functions

Even though the blast radius is limited with a Serverless Function, all the engineering best practices still apply with some minor implementation changes.

Logging — Use the provided logging mechanism (CloudWatch) and take the log stream to a central location using ELK stack for further processing.
Versioning — AWS provides the option to version the function code. Versions are immutable and provides the opportunity to rollback the lambda code if needed.
Monitoring — Use the tools like DeadLetterQueue, SNS topics and CloudWatch alarms to get notifications on failure. Monitor 5XX, 4XX on gateway. Monitor Throttling, Errors and ConcurrentExecutions on Lambda.
Security — Create a separate role for lambda execution and provide only the needed privileges. Keep lambda behind a VPC if the lambda has access to any private resources.

Are we are ready to put all this knowledge to use? Follow along to the next chapter.

Next: Deploying a web app using Lambda, API Gateway, DynamoDB and S3 with Serverless Framework

Deploying a web app using Lambda, API Gateway, DynamoDB and S3 with Serverless Framework

Yuva — Tue, 21 May 2019 16:17:06 +0000

Step0: Understand the architecture

We are going to use the simplest architecture today to create and deploy a web app.

Step1: Write the app

The app we are creating is a simple idea-board app where anyone can create an idea, add comments, upvote or delete an idea. Find the code for this app in git.

React frontend — Jennifer Peavler created a simple react app.
Backend — A serverless python crud handler.

Serverless backend is written differently compared to traditional server backend.

Lambda needs an entry point and the entry point is a function that takes event, context as arguments.

Step2: Select a deployment method

Principle: Deploy and manage both resources and application code using a version control.

AWS — provides Code Deploy and Code Pipeline to deploy a serverless application using Cloud Formation templates. The templates are used to create stacks that in-turn will manage all the resources specified.
Serverless Framework — provides abstraction over AWS cloud-formation templates. Create a deployment yml file that will generate cloud-formation templates to create stacks that in-turn will manage all the resources specified.

I picked Serverless Framework to deploy our app because

Minimum configuration — very easy to manage multiple lambda functions, layers and other related resources.
Cloud agnostic — Can be used to deploy to multiple cloud service providers.
Widely adopted — Has plenty of plugin support most use-cases
Support for local development, stages and rollback

Step3: Write the deployment yml

Start by creating a serverless.yml file and specifying provider requirements.

service: idea-app-api
provider:
  name: aws
  runtime: python3.7
  memorySize: 512
 timeout: 30

Specify the Lambda function and its triggers. For our app, we have one handler for five types of HTTP triggers.

functions:
idea-crud:
handler:  ideas.handler
events:
  - http:
      path: ideas
      method: post
      cors: true
  - http:
      path: ideas/{id}
      method: patch
      cors: true
  - http:
      path: ideas/{id}
      method: get
      cors: true
  - http:
      path: ideas/{id}
      method: delete
      cors: true
  - http:
      path: ideas
      method: get
      cors: true

Serverless Framwork allows us to specify resources that we can write directly in AWS cloud formation template. For our example, we will create all the needed resources for the database and S3 bucket using serverless.

s3-bucket.yml

  Resources:
  TheBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: <WEBSITE BUCKET NAME>
      AccessControl: PublicRead
      WebsiteConfiguration:
        IndexDocument: index.html
        ErrorDocument: error.html
  TheBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref TheBucket
      PolicyDocument:
        Id: MyPolicy
        Version: '2012-10-17'
        Statement:
          - Sid: PublicReadForGetBucketObjects
            Effect: Allow
            Principal: '*'
            Action: 's3:GetObject'
            Resource: !Join
            - ''
            - - 'arn:aws:s3:::'
              - !Ref TheBucket
              - /*

Complete serverless.yml with S3, dynamo and gateway error resources.

Step4: Deploy!

We have created the resources and the application code. Now it is time to deploy them in AWS. Issue Serverless Framework command to deploy resources. Use --aws-profile option to specify a profile if you have multiple AWS accounts. It will take few mins when you deploy for the first time.

$ >> sls deploy --aws-profile playground                                                                                                                                                      
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Creating Stack...
Serverless: Checking Stack create progress...
.....
Serverless: Stack create finished...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service idea-app-api.zip file to S3 (2.5 KB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
..................................................................
Serverless: Stack update finished...
Service Information
service: idea-app-api
stage: dev
region: us-east-1
stack: idea-app-api-dev
resources: 22
api keys:
  None
endpoints:
  POST - https://tm0ndmiyt9.execute-api.us-east-1.amazonaws.com/dev/ideas
  PATCH - https://tm0ndmiyt9.execute-api.us-east-1.amazonaws.com/dev/ideas/{id}
  GET - https://tm0ndmiyt9.execute-api.us-east-1.amazonaws.com/dev/ideas/{id}
  DELETE - https://tm0ndmiyt9.execute-api.us-east-1.amazonaws.com/dev/ideas/{id}
  GET - https://tm0ndmiyt9.execute-api.us-east-1.amazonaws.com/dev/ideas
functions:
  idea-crud: idea-app-api-dev-idea-crud
layers:
  None

Serverless successfully created all the resources , deployed our code to Lambda, and created API-Gateway endpoints.

Update the config file (idea-board/app-client/src/config.js) in the frontend to the correct endpoint.

export default {
    apiGateway: {
        REGION: "us-east-1",
        URL: "https://tm0ndmiyt9.execute-api.us-east-1.amazonaws.com/dev"
    }
};

We are almost there! Don’t quit now

Lets push the react app to S3

$ >> npm run build
$ >> aws s3 sync ./build s3://<bucket-name>

Hurray! We did it.

The site is now public and accessible from http://bucket-name.s3-website-us-east-1.amazonaws.com/

Thank you for reading! I would love to receive feedback on my work. Please feel free to comment and let me know how to improve my writing.

CircleCI deployment with AWS role assumption

Yuva — Tue, 21 May 2019 00:49:00 +0000

Problem:

We had three separate AWS accounts for dev, staging and production. We have a master account and we used role assumption to access the rest of the accounts for the development stages. We needed CircleCI to deploy to different stages using AWS Role Assumption.

Note: This article assumes you already have the working knowledge of CircleCI and how to setup CircleCI for your project.

Solution:

We solved the problem using aws config, CircleCI contexts and some yml magic.

Step1:

Create a config file aws_config and add it to the .circleci folder.

   [default]
   region = us-east-1
   output = json

   [profile dev]
   role_arn = arn:aws:iam::<account_id>:role/CircleCi-role
   source_profile = default

   [profile staging]
   role_arn = arn:aws:iam::<account_id>:role/CircleCi-role
   source_profile = default

   [profile production]
   role_arn = arn:aws:iam::<account_id>:role/CircleCi-role
   source_profile = default

Step2:

CircleCI supports environment variables by context.

Login to CircleCI and go to Settings
Create a org-global context and add AWS_ACCESS_KEY and AWS_SECRET_KEY.
Create three contexts for each stage and add AWS_PROFILEenv variable. Set the variable to 'dev', 'staging' and 'production' in respective contexts.

Step3:

Add a step in CircleCI config.yml to setup the credentials and config file under the org-default context. You only need to do this once. The .aws folder could be persisted and shared across jobs.

Note: I am using python executor and pipenv has already installed awscli as a dependency.

     - run:
          name: AWS configure
          command: |
            pipenv run  aws configure set aws_access_key_id $AWS_ACCESS_KEY
            pipenv run  aws configure set aws_secret_access_key $AWS_SECRET_KEY
            cp .circleci/aws_config ~/.aws/config

This step creates a credentials file with the key_id and access_key for the default profile.
The config file in the previous step uses the profile from the credentials file.

Step4:

Pass profile name in parameter to aws cli commands in your CircleCI deployment.

aws s3 sync ./local s3://bucket --profile dev

AWSCLI commands can also read from the env variable directly

aws s3 sync ./local s3://bucket

In case, you have a custom script or a Serverless Framework, you can pass the profile.

sls deploy --aws-profile $AWS_PROFILE

Complete example:

Recommended reading: Set up AWSCLI using RoleAssumption and MFA.

Visual Studio Code set up to improve developer productivity

Yuva — Mon, 20 May 2019 20:49:24 +0000

VS code has become the go to code editor lately. I configured VSCode to write code, write articles, access database, do code reviews and communicate with the team. The goal is to keep the movement from one app to the other to a minimum.

I am sharing my configuration here and I hope to learn more from others.

Extensions

Git lens - Pick a file in source control and view every version and changes in each version directly in the editor.
Docker - Start, stop containers that run your code directly from editor.
Git PR - Ever wanted to click through the code that you are reviewing? May be wanted to see all the references to a method in the code? Now, you can browse the code being reviewed in the editor with git PR.
Team Chat - Bring slack messages to vscode so you don't have to see another screen or app.
Markdown All in One - Write your documents or articles in markdown directly in the code editor.
Paste Image -- Paste images directly to markdown files.
PostgresSQL -- Create connections to dev/uat/prod databases and execute any SQL command.
Visual Studio IntelliCode - -Autocomplete for Python/JS/TS/Java code.
SonarLint -- Best multi language linter.
Code Spell Checker - Tells you to use meaningful words in code and also useful in spellchecking the documents you write.

settings.json

{
    "workbench.colorTheme": "Solarized Dark",
    "workbench.sideBar.location": "left",
    "workbench.settings.enableNaturalLanguageSearch": false,
    "workbench.statusBar.feedback.visible": false,
    "window.zoomLevel": 1,
    "explorer.confirmDragAndDrop": false,
    "explorer.sortOrder": "filesFirst",
    "explorer.openEditors.visible": 0,
    "breadcrumbs.enabled": true,
    "files.autoSave": "afterDelay",
    "files.insertFinalNewline": true,
    "files.trimTrailingWhitespace": true,
    "editor.suggestSelection": "first",
    "editor.minimap.enabled": false,
    "editor.cursorSmoothCaretAnimation": true,
    "editor.formatOnPaste": true,
    "editor.formatOnSave": true,
    "editor.cursorBlinking": "phase",
    "editor.smoothScrolling": true,
    "editor.renderWhitespace": "all",
    "git.autofetch": true,
    "git.postCommitCommand": "push",
    "git.alwaysShowStagedChangesResourceGroup": true,
    "gitlens.codeLens.authors.enabled": false,
        "gitlens.advanced.messages": {
        "suppressFileNotUnderSourceControlWarning": true
    },
    "gitlens.views.repositories.files.layout": "list",
    "cSpell.allowCompoundWords": true,
    "cSpell.userWords": [
        "oauth",
        "postgres",
        "repo",
        "venmo"
    ],
    "cSpell.enabledLanguageIds": [
        "markdown",
        "plaintext",
        "text"
    ]
    "telemetry.enableCrashReporter": false,
    "telemetry.enableTelemetry": false,
    "githubPullRequests.telemetry.enabled": false,
    "eslint.autoFixOnSave": true,
    "javascript.updateImportsOnFileMove.enabled": "always",
}

Do not abuse the assert

Yuva — Mon, 13 May 2019 20:37:25 +0000

Asserts are for programmers to find bugs or situations that should never happen. The program should execute normally if all the asserts are removed. Assertive programming section in Pragmatic Programmer states

Whenever you find yourself thinking “but of course that could never happen,” add code to check it.

It is very easy to take that advise and add asserts to handle every negative case. Most languages provide a convenient way to do this check in debug code, which is typically turned off in production deployments after testing. Asserts are not meant to be used for normal checking (i.e. error handling)

Don’t use asserts for normal error handling

Null checks — Consider a python example where we are going to access a dict.

def create_auth_header(oauth):
    assert oauth
    return {"Authorization": "Bearer " + oauth['access_token']}

Assert here does not provide anything additional that the language errors do not provide. TypeError or KeyError (in python) provides more information than AssertionError. Very similar to try and catch , asserts littered all over the code distract the programmer that is reading the code from the business logic.

Validating user input — A good program always validates user input, but this should never be done with assertions. Exceptions are there for this reason. In Python, all asserts can be ignored when running the program in optimized mode. All the user input validations could be bypassed accidentally by another developer.

When do I use asserts then?

Use asserts as a tool to alert you when you think the cause and effect are separated by a lot of code. Ever been in situation where a bug in one piece of code shows up as odd behavior in a completely different module making debugging a nightmare? We have all been there. We try to avoid being in that situation by asking a lot of ‘what if’ questions when we write code. What if the input is null? What if the key that I am looking for is not available? What if the world ended? If the distance between the cause i.e the input is null and the effect i.e system exits with TypeError is not within the function or method or class itself, there is a need for an assert.

Let's take a look at an example where assert might save us a lot of time. This example is taken from Python wiki . Please check the other examples there as well. They are very helpful in understanding asserts.

class PrintQueueList:
   ...
     def add(self, new_queue):
       assert new_queue not in self._list, \
          "%r is already in %r" % (self, new_queue)
       assert isinstance(new_queue, PrintQueue), \
          "%r is not a print queue" % new_queue

Here assert is used to check duplicates in a list. It is good to fail fast in this case and sleep well at night knowing that there cannot be any duplicates in this list especially the effect of a duplicate might only be noticeable further down the execution path and the symptoms might not directly point to the root cause.

In summary,

Don’t use asserts to check inputs if the language can fail fast for you
Don’t use asserts for user input validation
Don’t turn off asserts in production
Do use asserts sparingly when the distance between cause and effect is not immediate and hence reducing the side effects
Do use asserts to prevent your system from going into an inconsistent or non-performant state due to programmer errors. Assertions are great for runtime (i.e external cause errors). “We didn’t test this case because we never thought we could get here” i.e. things that “should never happen”

How do you evaluate a software framework?

Yuva — Mon, 13 May 2019 19:38:02 +0000

We do a lot of reading and testing to compare multiple frameworks available in the market when you are starting a project. I wanted to share my methods and hope to learn what other software devs do.

My methodology

Time box general research and list two to three frameworks/libraries that I want to try.
Try to solve the most complex problem in the requirement with the items selected in step 1. Timebox this step as well. How much you can solve with a same amount of time is also a good indicator of how easy it is to use a selected software.
Fill out the rubric for each and compare them after giving myself a day break after the research.

Framework Evaluation Rubric

What is the fundamental problem it is solving?
How intrusive/opinionated is it?
Can I abstract it away so that my code can be protected from framework specific leakage?
What is the cost of changing my mind later?
What is the philosophy of extensions? How easy or difficult it is to write extensions?
Maturity, community, activeness and open bugs

Do you have a rubric to compare the characteristics that are important to you?

Thanks Karthikkannan for giving me the outline.

Create a static site with Python, MkDocs, and S3

Yuva — Sun, 12 May 2019 18:32:03 +0000

We needed a lightweight solution to provide our beta customers with documentation, upcoming features, and FAQs. We wanted to use markdown for developer happiness and wanted something simple and easy to use. We settled on using Python MkDocs to generate a static site that would be hosted in AWS S3.

We decided to add customer documentation as part of the same git repo as the code and started looking for solutions to create a static site from markdown. I picked MkDocs since it was very light weight and easy to get started on.

Building a static site from markdowns

Install MkDocs. pip install mkdocs

Create the folder structure needed by MkDocs. Below is our sample site:

 documents
     -- mkdocs.yml
     -- docs
       -- index.md
       -- faq.md
       -- images/
       -- stylesheets/

Install material theme for mkdocs pip install mkdocs-material

Setup mkdocs by updating mkdocs.yml

site_name: TrainingPeaks Beta
site_description: How to documents for Beta users.
theme:
    name: 'material'
    favicon: 'images/favicon.ico'
    extra_css:
       - 'stylesheets/extra.css'

We added a few minor modifications to the material theme:

Set up the favicon.
Provide an additional stylesheet that has some minor overrides.

Serve site locally mkdocs serve

Deploy MkDocs site to S3

Setup S3

I recommend using cloudformation template to create a public bucket to host the documents. AWS documentation on S3 site templates.

Create a yaml file

AWSTemplateFormatVersion: 2010-09-09
Description: Resources to host documentation.

Parameters:
SiteName:
    Description: Site name
    Type: String
    Default: tp-beta-learning

Resources:
S3Bucket:
    Type: AWS::S3::Bucket
    Properties:
        BucketName: !Ref SiteName
        AccessControl: PublicRead
        WebsiteConfiguration:
            IndexDocument: index.html
            ErrorDocument: error.html

BucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
    PolicyDocument:
        Id: MyPolicy
        Version: 2012-10-17
        Statement:
          - Sid: PublicReadForGetBucketObjects
            Effect: Allow
            Principal: '*'
            Action: 's3:GetObject'
            Resource: !Join
            - ''
            - - 'arn:aws:s3:::'
                - !Ref S3Bucket
                - /*
    Bucket: !Ref S3Bucket

Outputs:
WebsiteURL:
    Value: !GetAtt
        - S3Bucket
        - WebsiteURL
    Description: URL for website hosted on S3

Deploy the cloud formation using aws cli.

aws cloudformation create-stack --template-body file://help-site.yaml --stack-name Help-Site
Verify that the stack creation is successful. We only need to do this step once.

Deploy Site to S3

In the local directory, run mkdocs build. This command will create a site folder with html files.
Deploy site to s3 aws s3 sync ./site s3://tp-beta-learning --recursive

The site is up and running in the url http://<bucket-name>.s3-website-<region>.amazonaws.com.

Next step is to use a custom domain for the site.

Docker and Docker Compose

Yuva — Sun, 12 May 2019 15:23:35 +0000

In order to understand docker, we have to go back in time and study the evolution of containers and how we got to where we are!

What is a container?

From the docker site

“Containers are a way to package software in a format that can run isolated on a shared operating system. Unlike VMs, containers do not bundle a full operating system - only libraries and settings required to make the software work are needed. This makes for efficient, lightweight, self-contained systems and guarantees that software will always run the same, regardless of where it’s deployed.”

Lets unpack it a bit

Back in the late nineties, VMWare introdcued the concept of running multiple OS in the same hardware.
In the late 2000s, kernel level namespacing was introduced that allows shared global resources like network and disk to be isolated by namespaces.
In early 2010s, Containerization was born and it took virtualization at the OS level and added shared libs/bin as well. This also means we cannot run two containers that are dependent on different operating systems in the same host unless we are using a VM.
Namespaces are the true magic behind containers. Principles are from linux containers and docker implemented its own OCI runtime called runc Virtual Machines are virtualization at the hardware level Containers are virtualization at the OS/Software level

Advantages of using containers

Speed
- Execution speed - Because containers use underlying host os,we get speeds as close a process natively running on the host os.
- Startup speed - Containers can be started in less than a second. They are very modular and can share the underlying libs/bins when needed along with host os.
- Operational speed - Containers enable faster iterations of application. There is less overhead in creating a container with new code changes and move it through the pipeline to production.
Consistency
- Build an image once and use it any where. The same image that is used to run the tests is used in production. This avoids the works in my machine problems.
- Not just in production. Containers helps in running tests consistently. Ever had a scenario where all tests passed in your machine but the CI failed your tests?.
Scalability
- We can specify exactly how much resources a single container can consume(CPU and memory). By understanding the available resources, containers can be packed densely to minimize wastage of CPU and memory. Scale containers within one host before scaling the instance.
Flexibility
- Containers are portable. Well to an extent (as long as the host is running some form of linux or linux vm).
- You can move a container from one machine to another very quickly. Imagine something went wrong while patching a security hole in the host OS, we simply move the container to a different host and resume service very quickly.

Enter Docker

Docker as a company

In 2013, Docker created the first container platform.
In 2015, Docker created the Open Container Initiative - - governance structure around container image and runtime specification. They also donated the first runtime to OCI. The current runtime used by docker and many other platforms is runC - written in golang.

Docker runtime/daemon/engine

Docker Engine is built for linux.
Docker for Mac uses HyperKit to run a lightweight Alpine Linux virtual machine.
Docker teamed up microsoft to create Windows OCI runtime available in Windows 10 or Windows server 2016.

Docker Cli

Docker cli commands look very similar to git commands. Many of them share the context as well.
1. git pull will get source from origin to local
2. docker pull <image> will get the docker image from remote registry to local
Docker follows a client server model so the cli can connect to local docker server or the remote server

Docker Images

An image is a read-only template with instructions for creating a Docker container. Often, an image is based on another image, with some additional customization. For example, we may build an image which is based on the ubuntu image, but installs the Apache web server and your application, as well as the configuration details needed to make your application run.

We need a Dockerfile to create an image. Let's look at an example. This is an image to run a python flask application using gunicorn

FROM python:3.7.3-stretch

ADD . /code
WORKDIR /code

COPY Pipfile Pipfile.lock /code/

RUN apt-get update

RUN apt-get install postgresql postgresql-client --yes && \
    apt-get -qy install netcat && \
    pip install --upgrade pip setuptools wheel &&\
    pip install --upgrade pipenv && \
    pipenv install --dev --system --ignore-pipfile

CMD ["/usr/local/bin/gunicorn", "--config", "wsgi.config", "coach_desk:create_app('development')"]

Images are a collection of immutable layers.

Each instruction in a Dockerfile above creates a layer in the image. When we change the Dockerfile and rebuild the image, only those layers which have changed are rebuilt. This is part of what makes images so lightweight, small, and fast, when compared to other virtualization technologies.

Images can also be built on top of other images.

The first line in the dockerfile is FROM which specifies the image that the current image is being built from. Lets look the Dockerfile that is used to create the python image. This image is built from buildpack-deps:stretch which provides all the basic tools to support any language.

FROM buildpack-deps:stretch

# ensure local python is preferred over distribution python
ENV PATH /usr/local/bin:$PATH

# http://bugs.python.org/issue19846
# > At the moment, setting "LANG=C" on a Linux system *fundamentally breaks Python 3*, and that's not OK.
ENV LANG C.UTF-8

# extra dependencies (over what buildpack-deps already includes)
RUN apt-get update && apt-get install -y --no-install-recommends \❗
        tk-dev \
        uuid-dev \
    && rm -rf /var/lib/apt/lists/*

ENV GPG_KEY 0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D

ENV PYTHON_VERSION 3.7.3
...

buildpack-deps:stretch is built from buildpack-deps:stretch-scm which is built from buildpack-deps:stretch-curl which is built from debian:stretch which is built from scratch.

FROM scratch
ADD rootfs.tar.xz /
CMD ["bash"]

If I had 1000 dockerfiles that are all built from python:3.7.3-stretch, the related layers are not downloaded 1000 times but only once. Same goes with containers, when we run a python container 1000 times, python is installed only once and reused.

Docker registry

Registry is a place to store all the images. When we install docker, we have a local registry where all the images we create a stored.
Try docker images to list all the images currently in your local registry
docker hub is a public repository that has over 100,000 images. That would be the first place to look for pre-built images that we could use directly or use as a base image to build on top.

We can move the images from local to remote using docker push and docker pull commands. The default remote registry is docker hub unless we specify explicitly. At Peaksware, we use Amazon ECR to store our production docker images.

Docker Compose

Compose was introduced so we do not have to build and start every container manually. Compose is a tool for defining and running multi-container Docker applications. Compose was initially created for development and testing purpose. Docker with recent releases have made compose yml to be used to create a docker swarm.

Using docker compose for a real application

A microservice that our team works on needs the right postgres database with all the migrations and aws cli setup in-order to run the service locally in a developer machine. The service was fairly new and the backend kept evolving and we need a quick way to spin up everything that is needed to get the service up for front end developers that are dependent on this service. Docker compose came in handy, the compose file below would spin up two containers

Python docker and install all dependencies and start the webserver
Postgres database

version: '3.7'

services:
  web:
    build:
      context: ..
      dockerfile: df.dev.Dockerfile

    environment:
      - DB_URI=postgres://postgres:postgres@db/idea_box

    command: bash -c  "flask migrate && flask run -p 5000 -h 0.0.0.0"
    ports:
      - 5000:5000
    links:
      - db
    volumes:
      - ../:/code

  db:
    image: postgres:10.1
    environment:
      POSTGRES_DB: idea_box

We can specify the dependencies using links. The web service container will wait until the db container is up before executing the entry command bash -c "flask migrate && flask run -p 5000 -h 0.0.0.0" which would run the migration and start the flask server.

If any one wants to run this service, they don't have to install python or flask or postgres instead the developer runs docker-compose -f docker-compose.yml up and wait for the api to be available at localhost:5000

Adding real complexity

This works great when your service is that simple. In reality, we had to add a queue and lambda function to process the queue and send messages to a different service. Unfortunately, We found localstack which emulates AWS services.

We can spin up SQS instance locally using localstack and create a queue using an init shell script that is called via entrypoint in localstack container.

This still does not represent the complete service. We still need a local lambda function that would read from the queue and push the message to another service. This is where I found the effort it takes to set up the entire service within docker compose out weighed the benefits.

version: '3.7'

services:
  web:
    build:
      context: ..
      dockerfile: .df.dev.Dockerfile

    environment:
      - DB_URI=postgres://postgres:postgres@db/coach_desk
      - AWS_ACCESS_KEY_ID=foo
      - AWS_SECRET_ACCESS_KEY=bar
      - AWS_ENDPOINT=http://aws:4576

    command: bash -c  "flask migrate && flask run -p 5000 -h 0.0.0.0"
    ports:
      - 5000:5000
    links:
      - db
      - aws
    volumes:
      - ../:/code

  db:
    image: postgres:10.1
    environment:
      POSTGRES_DB: coach_desk


  aws:
    image: localstack/localstack
    ports:
      - 4576:4576
      - 8080:8080
    environment:
      - SERVICES=sqs
      - DEBUG=True
    volumes:
      - ./localstack:/docker-entrypoint-initaws.d

Even with some of the complexities involved in using docker compose for a real service, I would recommend experimenting with it to see if it works for your team. I would love to hear how you/your team use docker for development and testing.

Developing with Docker

No painful developer machine setup. With compose, anyone can spin up a service pretty quick without having to install all dependencies that they will never use.
Consistent outcome - from dev to prod. Builds are reproducible, reliable, and tested to function exactly as expected on production.
Speed up testing - We have tests that need test database and we have the tear-down after each test/group of tests to clean up the database. I am working on ways to run parallel tests with database running in multiple containers for our project.
Code reviews can be painless - each dev can attach an image with their code review that the reviewer can quickly spin up without having to interrupt what they are doing and test a different version of the code.
Quick Fixes can be quick - When developers find bugs, they can fix them in the development environment and redeploy them to the test environment for testing and validation.When testing is complete, getting the fix to the customer is as simple as pushing the updated image to the production environment.