Forem: Robert Scott

Part 2 Networking the proper way!

Robert Scott — Mon, 15 Sep 2025 03:10:38 +0000

From Networking Nightmare to Instant Success: Setting up Kubernetes with Tailscale

Part 2 of my Kubernetes learning journey - sometimes the solution is simpler than you think

Yesterday, I shared my troubleshooting marathon trying to set up a two-node Kubernetes cluster between my home lab and an Azure VM. After hours of debugging network connectivity, container runtimes, and firewall rules, I made a decision that changed everything: start over with proper networking from day one.

The Problem Recap

My initial setup looked like this:

Control Plane: Home tower at 192.168.1.244
Worker Node: Azure VM at 172.16.0.4
Result: Complete network isolation - they couldn't even ping each other

The fundamental issue wasn't Kubernetes configuration - it was trying to bridge two completely different network environments:

My home network (192.168.1.x subnet)
Azure's virtual network (172.16.0.x subnet)

Even after opening firewall ports, configuring security groups, and debugging routing tables, the basic connectivity just wasn't there.

The Lightbulb Moment: Tailscale

Instead of continuing to fight network routing, I decided to embrace an overlay network solution. Tailscale had been running on my home tower already, but I'd been trying to work around it rather than leverage it.

The plan became simple:

Install Tailscale on the Azure VM
Use Tailscale IPs for all Kubernetes communication
Let Tailscale handle the networking complexity

Setting up Tailscale on Azure VM

Step 1: Configure Network Security Group for Tailscale

This is crucial for Azure VMs - Tailscale needs specific UDP ports open to establish connections. Before creating the VM, I added a Network Security Group rule:

Rule: Allow Tailscale UDP
Port: 41641 (Tailscale's default UDP port)
Protocol: UDP
Source: Any
Destination: Any
Action: Allow

Without this security group rule, Tailscale can install but won't be able to establish connections through Azure's firewall.

Step 2: Create the VM with Proper Storage

Next, I set up the Azure VM with:

Ubuntu 24.04 LTS
Additional SSD for container storage (temporary storage is fine for learning)
Default virtual network settings (since Tailscale overlays on top)
Applied the Tailscale security group from Step 1

# Format the additional storage
sudo fdisk /dev/sdb  # Create partition
sudo mkfs.ext4 /dev/sdb1  # Format
sudo mkdir /mnt/data  # Mount point
sudo mount /dev/sdb1 /mnt/data
echo '/dev/sdb1 /mnt/data ext4 defaults 0 0' | sudo tee -a /etc/fstab

Step 3: Install Tailscale with Pre-Auth

Instead of the generic Tailscale installation, I used the pre-configured script from my Tailscale admin dashboard:

# SSH into the Azure VM
ssh username@vm-public-ip

# Run the account-specific install script (from Tailscale dashboard)
curl -fsSL https://tailscale.com/install.sh | sh -s -- --authkey=tskey-auth-xxxxx

The beauty of this approach:

No manual authentication - the VM automatically joins my network
Immediate connectivity - shows up in the dashboard instantly
Pre-configured - includes any device tags or policies I've set

The Magic Moment

After the installation completed, I ran the test that had been failing for hours the day before:

# From Azure VM, ping my home tower's Tailscale IP
ping 100.64.x.x  # My tower's Tailscale IP

INSTANTLY - ping responses started flooding in:

64 bytes from 100.64.x.x: icmp_seq=1 ttl=64 time=12.3 ms
64 bytes from 100.64.x.x: icmp_seq=2 ttl=64 time=11.8 ms
64 bytes from 100.64.x.x: icmp_seq=3 ttl=64 time=12.1 ms

After a day of network timeouts and connection failures, seeing those ping responses was absolutely magical.

Why This Approach Works So Well

Tailscale eliminates network complexity:

No subnet routing - both machines appear on the same virtual network
No firewall configuration - Tailscale handles NAT traversal automatically
No port forwarding - direct machine-to-machine communication
Encrypted by default - secure communication across the internet
Works anywhere - home networks, cloud providers, mobile devices

For Kubernetes specifically:

Control plane and worker nodes can communicate directly
No need to expose API server ports to the internet
Pod-to-pod networking works seamlessly across locations
Can easily add more nodes from anywhere

The Difference is Night and Day

Before Tailscale (Local networking):

# From Azure VM
ping 192.168.1.244
# Result: Network unreachable

telnet 192.168.1.244 6443
# Result: Connection timed out

After Tailscale:

# From Azure VM  
ping 100.64.x.x
# Result: Instant responses

telnet 100.64.x.x 6443  
# Result: Connected immediately

Key Takeaways for Hybrid Kubernetes Clusters

Don't fight the network - use overlay solutions for cross-environment setups
Tailscale is perfect for hybrid clouds - seamlessly connects on-premises and cloud resources
Start with networking first - get connectivity working before diving into Kubernetes configuration
Pre-authenticated installation - use account-specific scripts for automated setups
Sometimes starting over is fastest - don't be afraid to rebuild with lessons learned

What's Next

Now that I have rock-solid networking between my home lab and Azure VM, I can focus on what I actually wanted to learn: Kubernetes cluster management. Tomorrow I'll:

Run kubeadm init on my home tower using Tailscale IPs
Join the Azure VM as a worker node (which should actually work this time!)
Deploy some test applications across both nodes
Explore pod scheduling, persistent volumes, and service discovery

The best part? All of this will happen over secure, encrypted Tailscale connections without any additional network configuration.

The Bottom Line

Sometimes the solution isn't debugging the existing approach - it's stepping back and choosing a better tool for the job. Tailscale transformed my networking nightmare into a 5-minute setup with instant connectivity.

If you're building hybrid Kubernetes clusters or just need to connect resources across different networks, don't fight with subnets and firewalls. Use Tailscale and focus on the problems you actually want to solve.

Next up: Actually setting up that Kubernetes cluster now that the machines can talk to each other! Stay tuned for Part 3 where we finally get to run those kubectl commands.

kubernetes #networking #tailscale #azure #hybridcloud #devops #infrastructure

Kubernetes cluster marathon!

Robert Scott — Sun, 14 Sep 2025 04:52:55 +0000

Setting up a Kubernetes Cluster on Ubuntu 24.04: A Troubleshooting Journey

Or: How I learned that sometimes starting over is the best solution

Setting up Kubernetes should be straightforward, right? Well, as I discovered today, reality has other plans. Here's my troubleshooting journey setting up a two-node Kubernetes cluster on Ubuntu 24.04, complete with all the roadblocks I hit and how to fix them.

The Initial Problem: Package Repository Issues

My first hurdle came immediately when trying to install kubectl and kubelet:

sudo apt install kubectl kubelet kubeadm
# Error: couldn't find the programs kubectl and kubelet

The Fix: Updated Repository URLs

The issue was that Google changed their package repository URLs in 2024, but many tutorials still reference the old packages.cloud.google.com URLs. Here's the correct way for Ubuntu 24.04:

# Remove any old repository entries
sudo rm -f /etc/apt/sources.list.d/kubernetes.list
sudo rm -f /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# Add the current official Kubernetes repository
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg

# Add the official GPG key (note the updated URL)
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# Add the repository
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

# Update and install
sudo apt-get update
sudo apt-get install -y kubectl kubelet kubeadm
sudo apt-mark hold kubelet kubeadm kubectl

Problem #2: CRI Socket Confusion

When running kubeadm init, I got:

error: define which one you wish to use by setting crisocket field for kubeadm

This happens when you have multiple container runtimes installed. Ubuntu 24.04 can have both Docker and containerd available.

The Fix: Choose Your Container Runtime

I chose containerd (recommended for modern Kubernetes):

# Install and configure containerd
sudo apt-get install -y containerd

# Generate proper config with CRI enabled
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

# Make sure CRI plugin isn't disabled
sudo sed -i 's/disabled_plugins = \["cri"\]/disabled_plugins = []/g' /etc/containerd/config.toml

# Enable and start
sudo systemctl enable --now containerd

# Use explicit CRI socket
sudo kubeadm init --cri-socket unix:///var/run/containerd/containerd.sock

Problem #3: The CRI v1 Runtime API Error

Even after installing containerd, I got:

failed to create new CRI runtime service: validate service connection: validate CRI v1 runtime API for endpoint "unix:///var/run/containerd/containerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService

The Fix: Proper containerd Configuration

The default containerd config sometimes has the CRI plugin disabled. The key is generating a proper config:

# Stop containerd
sudo systemctl stop containerd

# Remove bad config
sudo rm /etc/containerd/config.toml

# Generate proper config
sudo containerd config default | sudo tee /etc/containerd/config.toml

# Enable SystemdCgroup (required for Ubuntu 24.04)
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

# Ensure CRI plugin is enabled
grep -A5 -B5 disabled_plugins /etc/containerd/config.toml

# Restart
sudo systemctl start containerd

Test with:

sudo crictl version

Problem #4: Hostname Resolution Issues

During kubeadm init, I got errors about kubelet not being able to reach the hostname. Ubuntu 24.04 sets up hostname resolution with:

127.0.0.1 for localhost
127.0.1.1 for your hostname

But 127.0.1.1 isn't reachable from other machines!

The Fix: Use Your Real Network IP

# Find your actual network IP
ip route get 8.8.8.8 | grep -oP 'src \K\S+'

# Update /etc/hosts to use real IP instead of 127.0.1.1
sudo sed -i 's/127.0.1.1/192.168.1.244/' /etc/hosts

# Initialize with your real IP
sudo kubeadm init --cri-socket unix:///var/run/containerd/containerd.sock --apiserver-advertise-address=192.168.1.244

Problem #5: Port Already in Use

Even after kubeadm reset, I kept getting "port 6443 is in use" errors.

The Fix: Thorough Cleanup

# Reset with CRI socket specified
sudo kubeadm reset --force --cri-socket unix:///var/run/containerd/containerd.sock

# Clean up everything
sudo rm -rf /etc/kubernetes/
sudo rm -rf /var/lib/etcd/
sudo rm -rf /var/lib/kubelet/
sudo rm -rf ~/.kube/
sudo rm -rf /etc/cni/net.d/

# Reset iptables
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X

# Kill any hanging processes
sudo pkill -f kube-apiserver
sudo pkill -f etcd
sudo pkill -f kubelet

# Restart services
sudo systemctl restart containerd
sudo systemctl restart kubelet

Problem #6: Worker Node Network Connectivity

When trying to join my worker node, I got:

error execution phase preflight: couldn't validate the identity of the API server - failed to request cluster info configmap: the client timed out waiting for headers

The worker node simply couldn't reach the control plane, even though I was using the correct IP address.

The Root Cause: Network Complexity

This is where things got complicated. I had:

Tailscale running on the control plane but not the worker
Potential firewall issues
VM networking complications

The Final Solution: Sometimes Starting Over Is Best

After hours of debugging network connectivity, container runtime conflicts, and configuration issues, I realized something important: it's okay to start over.

Instead of continuing to debug a complex setup with multiple moving parts, the better approach was:

Start with a clean VM
Set up Tailscale first (before any Kubernetes components)
Use a single container runtime from the beginning
Use Tailscale IPs for all cluster communication

This eliminates:

Network routing issues
Firewall complications
IP address confusion
Container runtime conflicts

Key Takeaways

Google changed Kubernetes repository URLs in 2024 - use the new pkgs.k8s.io URLs
Ubuntu 24.04 needs SystemdCgroup enabled for containerd
Always specify the CRI socket when you have multiple container runtimes
Use your real network IP, not 127.0.1.1 for multi-node clusters
Thorough cleanup is essential when resetting kubeadm
Network connectivity issues are the hardest to debug - consider using overlay networks like Tailscale from the start
Starting over with a plan beats fixing a messy setup

The Most Important Lesson

Don't feel bad about starting over! Kubernetes has a steep learning curve, and networking issues can be genuinely tricky even for experienced developers. Sometimes the fastest path to success is a clean slate with lessons learned.

Getting the control plane running (which I did!) is actually the hardest part. The worker node join should be straightforward once the networking is sorted out properly.

Have you faced similar Kubernetes setup challenges? What was your biggest hurdle? Share your experiences in the comments!

kubernetes #ubuntu #devops #troubleshooting #containerization #networking

The journey to complete the docker+kubernetes pair

Robert Scott — Tue, 09 Sep 2025 00:13:07 +0000

I've been using docker for a while. Tonight I downloaded kubernetes and launched mini kubernetes to mess around with it. I deployed a simple nginx node but sadly failed to get the port to work.
It's interesting because docker is my default go to for running any application now. Kubernetes on the other hand is definitely a new monster! Just like I had no idea what I was doing with docker, I'll get the hang of kubernetes!
I did successfully follow most of the documentation for kubernetes until I couldn't get anything to work. Luckily my buddy copilot saved me! The ability to learn with these different AI tools now is pretty remarkable.
At least tonight ive learned some kubectl commands and tried a deployment. I'll probably just go straight to hard mode and deploy my own container running on docker right now. Don't worry I'll let you know how it goes!
Visit my LinkedIn!
https://www.linkedin.com/in/robert-scott-74a093382?utm_source=share&utm_campaign=share_via&utm_content=profile&utm_medium=android_app

DevOps from the Driver's seat part 1

Robert Scott — Fri, 05 Sep 2025 00:27:00 +0000

The hum of the diesel engine has been apart of my soundtrack for a decade. From a lumber yard to trash truck to chemicals, I've hauled it all. Somewhere between the long stretches of highway and the loading racks, I realized I was always chasing something else.
These days, "DevOps engineer" feels less like a job title and more like a philosophy, which fits me perfectly. I've spent ten years solving problems on the road, but I've always been drawn to building, optimizing, and figuring out how systems of all kinds work together.
Driving gives me hours to think and listen to tutorials on whatever catches my curiosity. One day, I stumbled across a video about home labs. Within minutes, I was hooked! My wife would say when i get hooked into something i go all in!
Soon I was comparing Ubuntu vs windows, bookmarking Docker guides, and sketching network configurations online while loading chemicals. I had no idea how far down this rabbit hole I'd go...or how much it would change my life. I'm sure most of you know how deep that hole can go! Within the first week after getting home i bought a USB drive and duel-booted my old gaming computer with Ubuntu 24.04. Now I'm more confident with CLI then running windows. I'm constantly trying to learn everything I can while leaving time to keep learning python.
The more I learned, the more i realized this wasn't just a hobby. It was the same problem-solving rush I'd felt on the road, but now i was building services that could grow, scale, and maybe even become a business. I started to see how this could lead to something more and pay me back all the years I've built up my vast array of skills. It's interesting to see how all of my non-tech related life experience's have prepped me for what's to come.
More coming as my journey continues!