Forem: Sadeek M

Debugging Networking in a Kubernetes Cluster

Sadeek M — Fri, 06 Dec 2024 07:52:31 +0000

Networking issues in Kubernetes can be tricky due to its distributed nature. This guide outlines common tools and steps to troubleshoot networking problems in a Kubernetes cluster.

Basic Networking Components in Kubernetes Understanding the key networking components helps in debugging:

Pods: Each Pod gets its own IP.

Services: Abstracts access to Pods, often via ClusterIP, NodePort, or LoadBalancer.

Network Policies: Control traffic flow between Pods and other endpoints.

CNI Plugin: Responsible for pod networking (e.g., Calico, Flannel, Cilium).

Verify Pod Networking Check Pod IPs and Status Start by checking the Pod's IP addresses and their status:

kubectl get pods -o wide

Ensure the Pods have IP addresses and are in the Running state.

Exec into a Pod and Ping Another Pod
Test network connectivity between Pods.

kubectl exec -it <pod_name> -- ping <target_pod_ip>

If the ping fails:

Verify that the Pods are in the same namespace.
Check if any NetworkPolicy is restricting traffic.
Check DNS Resolution in Pods
Pods use the cluster DNS (usually kube-dns or CoreDNS):

kubectl exec -it <pod_name> -- nslookup <service_name>

If DNS resolution fails, check the DNS service:

kubectl get svc -n kube-system
kubectl logs -n kube-system <coredns_pod>

Check Kubernetes Services Inspect Services and Endpoints Ensure the Service is correctly configured and has valid endpoints:

kubectl get svc
kubectl describe svc <service_name>
kubectl get endpoints <service_name>

Missing Endpoints: Check if the Pods backing the Service are running.
Misconfigured Service: Verify port configurations and selectors.

Diagnose Node Network Issues Check Node Connectivity Ensure nodes can communicate with each other:

kubectl get nodes -o wide
ping <node_ip>

Verify Network Interfaces
Check the network interfaces on nodes:

ip addr

Check Node Routing Table
Ensure correct routes are in place for Pod networking:

ip route

Debugging CNI Plugin Issues Check CNI Configuration Inspect CNI configuration files on the node (usually in /etc/cni/net.d/):

cat /etc/cni/net.d/*

Check CNI Logs
Look for CNI plugin logs for errors:

journalctl -u kubelet | grep -i cni

Reinstall or Restart CNI Plugin
If issues persist, restart the CNI plugin or reinstall it:

systemctl restart kubelet

Network Policies Debugging List Network Policies Ensure that NetworkPolicies are not blocking traffic:

kubectl get networkpolicy
kubectl describe networkpolicy <policy_name>

Test Policy Impact
Run tests to see if traffic is blocked:

kubectl exec -it <pod_name> -- curl <target_pod_or_service>

If blocked, review NetworkPolicy ingress/egress rules.

Inspect Kube Proxy Check Kube Proxy Status Ensure kube-proxy is running on all nodes:

kubectl get pods -n kube-system -o wide | grep kube-proxy

Check Kube Proxy Logs
Inspect logs for any errors:

kubectl logs -n kube-system <kube-proxy_pod>

Diagnose Load Balancers and Ingress Check Ingress Configuration Inspect the Ingress resource for misconfigurations:

kubectl get ingress
kubectl describe ingress <ingress_name>

Verify Load Balancer Status
Check if the external load balancer has been provisioned:

kubectl get svc -o wide

Use Debugging Tools Install netshoot Pod A useful container for networking debugging:

kubectl run netshoot --image nicolaka/netshoot -- sleep 3600

Exec into it and use tools like curl, ping, and dig:

kubectl exec -it netshoot -- sh

Use tcpdump
Capture network traffic on a Pod:

kubectl exec -it <pod_name> -- tcpdump -i eth0

Common Issues and Solutions Issue Possible Cause Solution

Pods can't communicate CNI plugin issues Restart or check CNI configuration(check the logs also)
DNS resolution fails CoreDNS issues Restart CoreDNS and check logs
Service has no endpoints Pod labels mismatch Verify labels match service selectors
NetworkPolicy blocking traffic Restrictive policy Adjust NetworkPolicy rules

Conclusion
Debugging networking in Kubernetes requires a methodical approach, starting from Pods, Services, Nodes, and moving to CNI plugins and network policies. Using the tools and techniques outlined above, you can identify and resolve common networking issues effectively.

Debugging Kubernetes cluster part 2

Sadeek M — Thu, 05 Dec 2024 18:48:31 +0000

Debugging a Kubernetes cluster requires a deep understanding of its components and inter dependencies. Here’s a comprehensive Part 2 guide focusing on advanced debugging techniques for common cluster issues:

Node Issues

A. Node Not Ready

Check Node Status:bash

kubectl get nodes
kubectl describe node <node-name>

Inspect Kubelet Logs:
SSH into the node and review logs for errors:bash

journalctl -u kubelet -l

Possible Causes:
Resource exhaustion (e.g., CPU, memory, disk).
Misconfigured networking (e.g., unable to reach the API server).
Issues with container runtime (Docker, containerd).
B. Node Disk Pressure or Memory Pressure

Check Allocations:bash

kubectl describe node <node-name> | grep Allocated

Clean Up Disk Space:
Remove unused images and logs:bash

docker system prune

Reconfigure Resource Limits:
Adjust resource requests and limits for pods.

Pod Issues

A. Pod Stuck in Pending

Inspect Events:bash

kubectl describe pod <pod-name>

Possible Causes:
Insufficient resources: Check node capacity and pod requests.
Scheduling constraints: Inspect nodeSelector, taints, and tolerations.
Networking issues: Ensure the CNI plugin is functioning correctly.
B. CrashLoopBackOff

View Logs:bash

kubectl logs <pod-name> --previous

Check Events:bash

kubectl describe pod <pod-name>

Debugging Steps:
Ensure the container's entrypoint is correct.
Verify environment variables and mounted volumes.
Test locally using the same image.
C. Container Image Pull Issues

Inspect Events:bash

kubectl describe pod <pod-name>

Common Errors:
Unauthorized: Verify image pull secrets.
Image not found: Confirm the image exists in the registry.

Networking Issues

A. Pods Can't Communicate

Ping Other Pods:bash

kubectl exec -it <pod-name> -- ping <pod-ip>

Check Network Policies:bash

kubectl get networkpolicy -n <namespace>

Debugging CNI Plugins:
Inspect CNI logs:bash

cat /var/log/containers/<cni-plugin-name>*.log

B. Service Not Accessible

Check Service Description:bash

kubectl describe svc <service-name>

Inspect Endpoints:bash

kubectl get endpoints <service-name>

Test Connectivity:
From within a pod:bash

curl http://<service-name>.<namespace>:<port>

API Server Issues

Inspect Logs:bash

journalctl -u kube-apiserver

Test API Server Availability:bash

kubectl get --raw /healthz

Common Causes:
SSL/TLS issues: Check certificates and CA bundle.
Resource bottlenecks: Monitor CPU/memory usage.

Persistent Volume Issues

A. PVC Pending

Inspect Events:bash

kubectl describe pvc <pvc-name>

Common Causes:
No matching StorageClass.
Insufficient storage on nodes.
B. PV Bound But Pod Can't Mount

Inspect Logs:bash

kubectl logs <pod-name>

Debugging Steps:
Verify volume permissions.
Test mounting the volume manually on a node.

Cluster DNS Issues

Test DNS Resolution:bash

kubectl exec -it -- nslookup

Inspect CoreDNS Logs:bash

kubectl logs -n kube-system <coredns-pod-name>

Common Fixes:
Restart CoreDNS pods if unresponsive.
Validate ConfigMap for CoreDNS (kubectl get cm -n kube-system coredns).

Troubleshooting Tools

A. kubectl Debugging Tools

Debug running pods:bash

kubectl exec -it <pod-name> -- /bin/sh

Debug containers with ephemeral containers (Kubernetes v1.18+):bash

kubectl debug -it <pod-name> --image=busybox

B. Third-Party Tools

Lens: GUI for Kubernetes cluster monitoring.
K9s: Terminal-based cluster management.
kubectl-trace: System-level tracing for Kubernetes.
C. Logs Aggregation

Use tools like Fluentd, ELK Stack, or Loki for centralized logging.

Proactive Cluster Monitoring

Implement monitoring systems like Prometheus, Grafana, or Datadog.
Set up alerting for critical metrics (e.g., node health, pod restarts).

Example: Debugging Workflow for a Non-Responsive Service

Check Pod Status:bash

kubectl get pods -n <namespace>

Describe the Service:bash

kubectl describe svc <service-name> -n <namespace>

Inspect Logs:bash

kubectl logs <pod-name> -n <namespace>

Test Connectivity:
From within a cluster:bash

curl http://<service-name>.<namespace>:<port>

From outside:bash

curl http://<external-ip>:<port>

This deeper dive equips you to troubleshoot and resolve complex Kubernetes issues effectively. Let me know if you'd like specific scenarios or additional examples!

Debugging a Kubernetes Cluster Part 1

Sadeek M — Thu, 05 Dec 2024 08:58:52 +0000

Debugging a Kubernetes cluster can be challenging, but by using systematic approaches and the right tools, you can efficiently diagnose and resolve issues. This guide provides an overview of common debugging methods and tools to help troubleshoot problems in a Kubernetes environment.

Understand the Problem Scope

Questions to Consider:

Is the issue affecting all nodes or a specific pod?
Are services unreachable?
Is the control plane responding correctly?
Are logs indicating specific errors?
Identifying the scope helps narrow down the troubleshooting process.

Check Cluster Components

a. Verify Node Status

Check if all nodes are healthy and ready:

kubectl get nodes

If a node is NotReady, inspect it further:

kubectl describe node <node-name>

Common issues:

Insufficient resources.
Network connectivity problems.
Crashed kubelet service.
Restart kubelet if needed:

sudo systemctl restart kubelet

b. Inspect Control Plane Components

Verify the health of control plane components on the master node(s):

Check etcd:

ETCDCTL_API=3 etcdctl endpoint health

Check Kubernetes API Server:

kubectl get --raw='/healthz'

Check Scheduler and Controller Manager logs:

sudo journalctl -u kube-scheduler
sudo journalctl -u kube-controller-manager

Investigate Pods

a. List All Pods

kubectl get pods -A

b. Describe the Problematic Pod

kubectl describe pod <pod-name> -n <namespace>

Look for:

Events section for errors (e.g., image pull errors, resource limits).
Status and readiness probes.
c. View Pod Logs

kubectl logs <pod-name> -n <namespace>

For multi-container pods:

kubectl logs <pod-name> -n <namespace> -c <container-name>

Debugging Nodes and Networking

a. Check Node Resources

kubectl top node

b. Debug Networking Issues

Test pod-to-pod connectivity using kubectl exec:

kubectl exec -it <pod-name> -- curl <service-ip>

Inspect service endpoints:

kubectl get endpoints

Verify DNS resolution:

kubectl exec -it <pod-name> -- nslookup <service-name>

Inspect network policies:

kubectl describe networkpolicy -n <namespace>

Inspect Persistent Volume Issues

Check PersistentVolume (PV) and PersistentVolumeClaim (PVC) status:

kubectl get pv
kubectl get pvc -n <namespace>

Describe the PVC for detailed information:

kubectl describe pvc <pvc-name> -n <namespace>

Advanced Debugging Tools

a. Use kubectl debug

Spin up a debug container in the same namespace:

kubectl debug <pod-name> -n <namespace> --image=busybox --target=<container-name>

b. Use strace and tcpdump

For deeper system-level debugging:

Install strace or tcpdump in the container.
Attach a terminal and analyze system calls or network packets.
c. Leverage Monitoring Tools

Prometheus/Grafana: Monitor cluster metrics.
ELK Stack: Analyze cluster and application logs.
K9s: A terminal-based UI for managing Kubernetes clusters.

Common Troubleshooting Commands

a. Restart Pod

Force a pod to restart:

kubectl delete pod <pod-name> -n <namespace>

b. Drain a Node

Safely remove workloads from a node:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

c. Restart Deployment

kubectl rollout restart deployment/<deployment-name> -n <namespace>

Consult Logs and Events

Check cluster-wide events:

kubectl get events -A

Inspect cluster-level logs on the master node:

sudo journalctl -u kubelet

Conclusion

Debugging a Kubernetes cluster involves a combination of high-level checks, log inspection, and targeted analysis. By following the steps outlined in this guide, you can systematically identify and resolve issues, ensuring a stable and reliable Kubernetes environment.

Basic Linux Debugging Guide

Sadeek M — Wed, 04 Dec 2024 13:39:50 +0000

Debugging in Linux is essential for troubleshooting system issues, application bugs, or network problems. Here’s a simple guide to get you started with common Linux debugging tools and techniques.

1. Checking System Logs
Linux logs are a great starting point for debugging. Logs capture events related to the kernel, applications, and services.

View Logs with journalctl

# List all processes
ps aux

# View processes in real-time
top

Find a Process by Name with pgrep

pgrep <process_name>

Debug a Process with strace
strace traces system calls made by a process.

strace -p <pid>`

Debug a Process with gdb
gdb is used for debugging compiled applications.

gdb <program_name> <pid>

Networking Debugging Check Active Connections with netstat or ss

Using netstat

netstat -tuln

Using ss

ss -tuln

Test Connectivity with ping or curl

ping <hostname_or_ip>

curl -v http://
Check Firewall Rules with iptables or ufw

Using iptables

sudo iptables -L -v

Using ufw

sudo ufw status verbose

Disk and Filesystem Debugging Check Disk Usage with df and du

Disk space usage

df -h

Directory size usage

du -sh <directory_name>

Check Disk Health with fsck and smartctl

Check and repair filesystem (run in unmounted mode)

sudo fsck /dev/sdX

Check disk health using smartctl

sudo smartctl -a /dev/sdX

Memory Debugging Check Memory Usage with free and vmstat

Memory usage

free -h

System performance and memory statistics

vmstat 1 5

Analyze Memory Leaks with valgrind

valgrind --leak-check=yes ./your_program

Kernel Debugging Check Kernel Messages with dmesg

dmesg | tail

Debug Kernel Panics with kdump
Ensure kdump is installed and configured for capturing kernel crash dumps.

sudo systemctl start kdump

Conclusion
Effective debugging in Linux requires a good understanding of system logs, processes, network, disk, and memory. Start with logs, monitor processes, trace system calls, and analyze system resources. Mastering these tools will help you quickly diagnose and resolve issues.