Forem: Diana

Kubernetes: How Your Apps Find Each Other 🌐

Diana — Wed, 25 Mar 2026 04:13:59 +0000

Introduction: From Addresses to Orchestration

In my previous posts, we covered the high-level architecture of Kubernetes and the networking fundamentals that help explain how devices find each other:

IP addresses (street names + house numbers)
Ports (doors)
Private vs. Public IPs
Localhost
DNS

Now it's time to connect the dots. Inside a Kubernetes cluster:

Pods get their own IP addresses
Services act as stable phone book entries so nothing gets lost when Pods come and go
Ingress controllers manage traffic from the outside world

Sound familiar? It's every concept from Networking 101, applied to containers.

Kubernetes Networking: The Building's Internal Address System 🏢

Let's bring our networking knowledge into the Kubernetes building. Remember, we have:

A cluster (the building)
Nodes (the floors)
Pods (the rooms)

In Kubernetes, every Pod gets its own IP address. That's like every room in our building getting its own unique address.

The Three Networking Challenges Kubernetes Solves

Challenge	Question	Building Analogy
1. Pod-to-Pod Communication	How does Room 101 talk to Room 205?	Rooms calling each other directly
2. Service Discovery	How do I find the coffee shop if it moved?	A building directory that stays up to date
3. External Access	How do customers outside find us?	Getting visitors into the building

Let's tackle each one.

Challenge 1: Pod-to-Pod Communication 🗣️

In Kubernetes, every Pod can talk to every other Pod directly using their IP addresses, no special configuration needed.

It's like every room in the building having its own phone number, and every phone being able to call any other phone.

Administrators can restrict this later with NetworkPolicies, but out of the box, everything is open.

Every room can call every other room directly, even across floors!

Key Point: Pods can communicate across Nodes. Kubernetes handles the routing behind the scenes.

Room 102 (Floor 1) → Room 202 (Floor 2) ✅
Room 102 (Floor 1) → Room 101 (Floor 1) ✅
Any room → Any other room ✅

But here's the problem...

Challenge 2: Service Discovery: The Problem with Pod IP Addresses 😰

Remember how we said Pods are ephemeral? They come and go. They crash and get replaced. They scale up and down. Every time a Pod is replaced, it gets a new IP address.

Imagine this scenario in our building:

Monday:
  Coffee Shop is in Room 101 (IP: 10.244.1.5)
  Everyone knows to go to Room 101 ✅

Tuesday:
  Coffee Shop crashed! Kubernetes restarts it.
  New Coffee Shop is in Room 205 (IP: 10.244.2.10)
  Nobody knows the new address ❌

Wednesday:
  We scaled to 3 Coffee Shops.
  They're in Room 101, Room 205, and Room 302.
  Which one do customers go to?! 🤯

This is like a coffee shop that changes its street address every day. Your regular customers would never find you! This is where having something stable, something permanent that people can always rely on, makes all the difference.

That's exactly what Services solve.

Kubernetes Services: The Building Directory 📞

A Service in Kubernetes is a stable address that always points to the right Pods, no matter where they are or how many there are.

Remember the DNS phone book from Networking 101? Services work the same way. They're stable phone book entries for your Pods. Think of it as the building's front desk directory.

Instead of telling customers "Go to Room 101" (which might change tomorrow), you tell them "Call the Coffee Shop extension." The front desk always connects them to the right room.

Without a Service:
  Customer → "Where's the coffee shop?" → ??? 😵

With a Service:
  Customer → "Coffee Shop, please" → Front Desk → Room 101 ✅
                                                  → Room 205 ✅
                                                  → Room 302 ✅

What a Service gives you:

Stable IP Address

Every Service gets a ClusterIP that never changes. Pods come and go, but the Service address stays the same.

Load Balancing

Distributes incoming traffic across multiple Pods so no single one gets overloaded.

3 coffee shops open in the building? The front desk doesn't send every customer to the same one. It spreads them out so no single shop gets overwhelmed. The Service does this automatically as you scale up your Pods.

DNS Name

Just like google.com maps to an IP address, Kubernetes gives every Service a name like:

coffee-shop.default.svc.cluster.local

So Pods can find each other by name instead of memorizing IPs.

Part	Meaning
`coffee-shop`	Service name
`default`	Namespace
`svc`	Service
`cluster.local`	"Inside this cluster"

What's a Namespace? Going back to our building analogy, Namespaces are like the different departments on each floor. Marketing, Engineering, HR, all in the same building but keeping their stuff separate. How teams separate things is case by case depending on their setup.

These DNS names only work inside the cluster's private network. Remember how devices on your home network find each other using private IPs like 192.168.1.x? Same idea here.

How Services Find Pods: Labels and Selectors 🏷️

Services don't track Pods by their IP addresses (because those change). Instead, they use labels. Every Pod gets a label, and the Service is configured to find all Pods with a matching label.

Service: "Find all Pods labeled app=coffee-shop"

  Pod 1 (Room 101) - label: app=coffee-shop  ✅ Match!
  Pod 2 (Room 205) - label: app=coffee-shop  ✅ Match!
  Pod 3 (Room 302) - label: app=coffee-shop  ✅ Match!
  Pod 4 (Room 110) - label: app=bakery       ❌ Not a match

It's like the front desk has a rule: "Anyone wearing a Coffee Shop uniform works at the coffee shop."

New employees (Pods) automatically get found, and ones who leave are automatically removed. No manual updates needed.

Service Types + Ingress 🔧

Kubernetes gives you different types of Services, plus a separate resource called Ingress. Let's look at the ones that matter most when you're getting started.

1. ClusterIP: The Internal Extension 📞

Default type, internal communication only

This is like an internal phone extension. Only people inside the building can use it. Customers outside can't dial this number.

When to use it: When your services only need to talk to each other inside the cluster. For example, your frontend app calling your backend API.

2. NodePort: The Side Entrance 🚪

Exposes the Service on each Node's IP at a static port

This is like adding a side entrance to the building. People outside can now get in, but they need to know which building (Node) to go to and which door (port) to use.

When to use it: For development and testing, or when you want quick external access without a cloud load balancer.

3. Ingress: The Traffic Cop 🚦

A separate resource (not a Service type) that routes external traffic to multiple Services based on rules

Ingress is like a traffic cop standing in the lobby of your building. Visitors walk in, and the traffic cop checks where they're trying to go and directs them to the right department.

Heading to coffee.com/menu? Go left. coffee.com/order? Go right. tea.com? Take the elevator to floor 3.

Instead of having a separate entrance for every service, you have one entrance with a traffic cop handling all the routing.

When to use it: When you have multiple services and want one entry point with smart routing. It gives you features like SSL termination and path-based routing.

Challenge 3: External Access 🌍

How do real users reach your app from the internet? We introduced Ingress above, but there's more to the picture:

LoadBalancers, service meshes (like Istio), and how domains even get routed to your cluster
The setup looks very different on cloud (a lot is handled for you) vs. on-prem (your team manages it)

This deserves its own deep dive. Coming in a future post.

Putting It All Together: The Full Picture 🗺️

Here's how they compare:

Type	Access	Use Case	Analogy
ClusterIP	Internal only	Service-to-service calls	Internal phone ext.
NodePort	External via Node IP + port	Dev/testing, quick access	Side entrance
Ingress	External via smart routing	Production, multi-service	Traffic cop

What's Next? 🚀

Now that we understand how networking works inside a Kubernetes cluster, I'm deciding between two directions for the next post:

A hands-on walkthrough where we deploy a simple app and see Services and Ingress working in a real cluster
A deeper dive into traffic management covering LoadBalancers, service meshes like Istio, and how production teams actually handle external traffic

Let me know in the comments which one you'd find more useful! 👇

Additional Resources 📚

Prerequisite Reading:

Networking 101: The Concepts Nobody Explained Simply: My companion post covering IP addresses, ports, DNS, localhost, and private vs. public networks. Start here if networking is new to you

Kubernetes Networking:

Kubernetes Networking Concepts (Official Docs): Deep dive into the networking model

Services & Ingress:

Kubernetes Services (Official Docs): Comprehensive reference for all Service types
Kubernetes Ingress (Official Docs): Everything about Ingress resources and controllers
Gateway API: The next generation of Kubernetes traffic routing, evolving beyond Ingress

Hands-On Practice:

Minikube Networking Guide: How to access services running in Minikube
Kubernetes Networking Tutorial (Official): Interactive tutorial on Services

Networking 101: The Concepts Explained Simply 🌐

Diana — Mon, 02 Feb 2026 15:22:30 +0000

Why Networking Feels Intimidating (But Shouldn't)

I'll be honest, networking used to make my eyes glaze over. IP addresses, ports, subnets... it felt like learning another language.

But once I found the right mental model, it clicked. So if you don't know what an IP address is or how it works, that's completely fine. By the end of this post, you'll understand how devices find each other on a network, and you'll have the foundation you need for topics like cloud computing, Kubernetes, or just understanding how the internet works.

Let's start from scratch.

An IP Address Is Just a Street Address 🏘️

Keith Barker has an easy breakdown in his video "Take 5 Minutes now to learn about IP addresses" that inspired the analogy I'm about to use. If you want to go even deeper, his "Subnet Saturday #1: IPv4 Basics" is a more detailed walkthrough of how IP addresses actually work.

Here's the core idea:

Think about your home address. It has two parts:

The street name: tells you which neighborhood you're in
The house number: tells you which specific house on that street

An IP address works the same way:

The network portion: this is the "street name," identifying which network you're on
The host portion: this is the "house number," identifying the specific device on that network

For example, take the IP address 192.168.1.25:

  192.168.1  .  25
  ─────────     ──
  Street Name   House Number
  (Network)     (Host)

Every device on a network needs its own unique address, just like every house on a street needs its own number. Two houses can't have the same number on the same street, and two devices can't have the same IP address on the same network.

Ports: The Doors on Each House 🚪

Now, each house (device) can have multiple doors. These are ports. Different doors lead to different services:

Door 80: the front door (web traffic / HTTP)
Door 443: the secure front door (HTTPS)
Door 22: the back door for maintenance (SSH)

So a full address to reach a specific service looks like this:

  192.168.1.25:80
  ─────────────  ──
  House Address   Door Number
  (IP Address)    (Port)

How a Request Finds the Right Device 🗺️

So when you type something like 192.168.1.25:80 into a browser, how does the request actually get to the right place? The network reads the address just like a mail carrier delivers a letter:

Think of it like a mail carrier delivering a package:

Read the street name: "Which neighborhood is this going to?" That's the network portion
Find the right house: "Which house on this street?" That's the host portion
Knock on the right door: "Which service do they want?" That's the port

Local vs. External IP Addresses: Private Streets vs. Public Highways 🏡🌍

Not all IP addresses are created equal. There are two kinds:

Local (Private) IP Addresses: These are like street addresses inside a gated community. They only work within that community. Devices on your home WiFi network talk to each other using private IPs like 192.168.1.x or 10.0.0.x. If you tried to give someone outside your network that address, they wouldn't be able to find you, it's a private street that doesn't exist on any public map.

External (Public) IP Addresses: This is like your community's main gate address, the one that shows up on a public map. It's the address the outside world uses to reach your network. Your internet provider gives you one, and it's how websites know where to send data back to you.

Your Home Network (Gated Community):
  📱 Phone     → 192.168.1.10  (Private — only works inside)
  💻 Laptop    → 192.168.1.25  (Private — only works inside)
  🖥️ Desktop   → 192.168.1.30  (Private — only works inside)

Your Router (The Main Gate):
  🌐 Public IP → 73.45.123.89  (Public — the outside world uses this)

Your router translates between the two. When your laptop requests a website, the router swaps the private address for the public one on the way out, and swaps it back on the way in. The outside world only ever sees the public address.

How Does the Response Find Its Way Back? 📬

You might be wondering, when a website sends data back, how does it know where to send it? Every request includes a return address, just like a letter. The website reads the return address and sends the response right back.

But here's the thing, your laptop might be talking to many websites at the same time. You've got a browser, Netflix, Slack, all running at once. How does your router know which response goes to which app?

Remember ports? Your laptop uses a different port number for each connection. Think of it as each app getting its own temporary mailbox:

Your Laptop right now:
  🌐 Browser tab 1  → mailbox 52431 → talking to dev.to
  🌐 Browser tab 2  → mailbox 52432 → talking to google.com
  🎬 Netflix         → mailbox 52433 → talking to netflix.com
  💬 Slack           → mailbox 52434 → talking to slack.com

When a response comes back, your router checks which mailbox (port) it's addressed to and forwards it to the right app. That's how one device can have dozens of connections running at the same time without anything getting mixed up.

localhost: Talking to Yourself 🪞

You might have seen localhost or 127.0.0.1 before. This is a special address that means "this machine, right here." It's not even a street address, it's like knocking on your own front door.

When developers say "run it on localhost," they mean "run it on your own computer, and access it from your own computer." Nothing goes across the network. It's completely internal to your machine.

localhost = 127.0.0.1 = "myself"

You → localhost:3000 → Your own machine, door 3000

You'll run into this all the time when developing locally, running a web app on localhost:3000, a database on localhost:5432, etc. It just means those services are running on your own machine and you're accessing them from your own machine.

DNS: Names Instead of Numbers 📖

Remembering IP addresses like 192.168.1.25 is annoying. That's why DNS (Domain Name System) exists, it's like a phone book that translates human-friendly names into IP addresses.

You already use DNS every day:

You type google.com → DNS translates it to 142.250.80.46
You type dev.to → DNS translates it to an IP address behind the scenes

You never have to memorize the actual numbers. DNS handles that for you.

This concept shows up everywhere in tech. Cloud platforms, container orchestration tools like Kubernetes, and internal company networks all use their own private DNS so that services can find each other by name instead of by IP address. The idea is always the same: names are easier to remember than numbers, so let a phone book handle the translation.

Putting It All Together 🗺️

Let's trace a real request from start to finish. You type dev.to in your browser, here's what happens:

Every concept we covered plays a role:

DNS translates the name to an IP address
Your router translates your private IP to a public one
The network portion of the IP routes the request to the right network
The host portion finds the right device
The port delivers the request to the right service

What's Next? 🎯

Now that you understand how devices find each other on a network, you have the foundation for some really powerful topics:

Cloud computing: Understanding public vs. private networks is key to how cloud infrastructure works
Kubernetes networking: Containers need to find each other too, and they use all the concepts we just covered (IP addresses, DNS, ports, private networks)
Web development: Every API call, every database connection, every deployment involves networking

In my next post, I'll take these exact concepts and show you how they apply inside a Kubernetes cluster, where Pods get their own IP addresses, Services act as stable phone book entries, and Ingress controllers manage traffic from the outside world. If you're new to Kubernetes, check out my intro post first: Kubernetes: Making Sense of the Madness. Stay tuned! 👇

Additional Resources 📚

Further Reading:

How DNS Works (Cloudflare): Visual explanation of the DNS lookup process
Public vs. Private IP Addresses (Avast): More detail on how private and public addresses work together

Kubernetes: Making Sense of the Madness 🤯

Diana — Fri, 10 Jan 2025 15:22:12 +0000

Kubernetes: Who Knew it Could Be This Hard? 🙃

Let’s be honest, new tools can feel intimidating, right? I avoided Kubernetes for ages because it felt like being handed a 500-piece LEGO set.... but with no picture on the box. Sure, all the pieces are there, and someone tells you, “It’s easy, just follow the instructions.” Read the documentation??? pshhh that's absurd !

At first learning it was overwhelming, and let’s face it, it still is!! What helped me was stepping back and looking at the high-level architecture. Understanding how the pieces fit together made everything feel a little less chaotic. That’s where I want to start with you: the big picture. Let’s break it down and make Kubernetes a little more approachable.

VMs vs. Containers: There’s No Place Like Shared Space 🏢

Before we dive into container orchestration, let’s talk about what a container is and how it works with Virtual Machines (VMs).

A Virtual Machine is like a fully furnished apartment, it has its own utilities (operating system) and runs independently in a shared building (physical server). It’s great for isolation but can be heavy since each VM carries its own setup.

A Container is like a room within that apartment. Each room has everything it needs (your app and its dependencies), but it shares the apartment’s utilities (VM’s operating system) with the other rooms. This makes containers faster and lighter than spinning up separate apartments (VMs) for every workload.

Container Orchestration: The Smart Manager 🤓

Kubernetes is an open-source container orchestration system that helps you manage containers efficiently, especially when dealing with large-scale applications. It ensures your containers are running, scaling, and self-healing, without you having to intervene manually. Think of container orchestration as a smart manager for your rooms (containers).

Kubernetes focuses on managing the rooms (containers), ensuring they run smoothly by:

Deploying containers where they’re needed.
Distributing resources like CPU and memory to the containers that need them most.
Restarting containers automatically if something breaks (self-healing).
Scaling containers up or down based on demand (adding or removing a container).
Balancing traffic across containers to prevent any single one from being overwhelmed.

Without tools like Kubernetes, handling hundreds or thousands of containers would be chaotic. While it doesn’t directly manage your physical servers or VMs, Kubernetes ensures your containers are orchestrated efficiently on top of the infrastructure you provide.

Kubernetes Architecture: Building Blocks of Orchestration 🧩

Cluster: The Big Picture 🖼️

A cluster is like a team working on a project. Each team member has a specific role (some do research, some write, some present), but they all work together toward a shared goal.

In Kubernetes, a cluster is the entire system where containers are deployed and managed. It consists of:

Nodes: The team members doing the actual work.
Control Plane: The project manager that coordinates the team and keeps everything running smoothly.

Think of the cluster as a collection of machines working together to deploy and manage your containers.

Nodes: The Workers 👷‍♀️

A Node is a single machine (physical or virtual) in your cluster. It’s where the real work happens, containers are deployed and run here.

There are two types of nodes in Kubernetes:

Worker Nodes: This is where containers are deployed and run inside Pods.These handle workloads by running containers inside Pods.

Each worker node includes:

kubelet: This is the primary agent running on the worker node. It communicates with the Control Plane and ensures that the containers in Pods are running as expected.
kube-proxy: This manages network rules on the node, enabling communication between Pods and Services across the cluster.

Master Node: This node hosts the Control Plane, which doesn’t handle workloads directly but instead manages the cluster by assigning tasks to worker nodes and keeping everything on track.

Note: The master node is the machine where the Control Plane components run. While they’re often used interchangeably in conversation, the Control Plane refers to the software managing the cluster, whereas the master node refers to the physical or virtual machine hosting that software.

Control Plane: The Brain of Kubernetes 🧠

The Control Plane acts as the brain of the cluster, coordinating all the nodes to ensure your containers run efficiently. It consists of several key components:

API Server: The entry point to the cluster. This is how you interact with Kubernetes, whether through kubectl or other tools.
kubectl: is a command-line tool that lets you communicate with the API Server to manage your cluster. You can use it to deploy applications, inspect resources, and even troubleshoot issues, all from your terminal.
Scheduler: Decides which node should run each container based on resource availability and requirements.
Controller Manager: Monitors the cluster’s state and ensures the desired configuration is maintained (e.g., restarting containers if they fail).
etcd: A key-value store that acts as Kubernetes’ memory, storing all configuration and the current state of the cluster.

The Control Plane is what gives Kubernetes its intelligence, managing workloads, and ensuring everything works as expected.

Pods: The Smallest Building Block 🫛

Kubernetes doesn’t directly run containers, it wraps them in Pods. A Pod is the smallest deployable unit in Kubernetes. Typically, a Pod contains a single container, but it can hold multiple containers if they need to work together (e.g., sharing storage or networking). Think of a Pod as a "toolbox" that holds everything a container needs to function properly, including storage, networking, and configuration.

Services: Connecting the Dots 🌉

While Pods are temporary (they can crash, restart, or be replaced), Services act as stable endpoints for communication.

For example, if a Pod running your backend app crashes and Kubernetes replaces it, the Service ensures your frontend can still communicate with the new Pod seamlessly. Services are like bridges that connect different parts of your application, ensuring reliable communication even in a dynamic environment. This ensures that even as Pods come and go, your app stays connected and functional.

Diagram from the official Kubernetes site

I also recommend checking out this Udemy course, it helped me understand Kubernetes and break it down this way.

Connecting the Dots: OpenTelemetry for Beginners

Diana — Tue, 19 Nov 2024 20:25:03 +0000

If you’ve ever felt overwhelmed by words like “collector,” “instrumentation,” or “tracing,” you’re not alone. I’m right there with you 😵‍💫, still learning and piecing everything together. But as I keep connecting the dots, tools like OpenTelemetry are slowly starting to make more sense. Attending Honeycomb's workshop for OpenTelemetry really helped paint a picture of the different components and how they work together. Let me share what I’ve figured out so far in the simplest way I can, so maybe the puzzle will click for you, too.

Understanding OpenTelemetry

Think of OpenTelemetry (OTel) as a toolset for understanding what’s happening inside your software. Imagine you are organizing a large mailing process where you need to send packages to people. You want to know:

When each package 📦 is picked up.
Where it goes next ❓.
When it gets delivered .

Let's break down some of the components of OpenTelemetry using this analogy:

Instrumentation: Attach a tracking label to each package. This is how you collect data about each package's journey. In software, this means adding sensors to your code to collect data about what’s happening.

Logs: Logs are like notes that track every event happening during the package's journey, like when it enters a facility or leaves for delivery. You can export logs to different backends, such as a logging service, to keep track of everything.
Metrics: Metrics are like summaries that tell you how many packages were delivered in a day, how long deliveries took, or how many were delayed. These metrics can be exported to specialized monitoring services to analyze overall performance.
Traces: Traces are like a detailed route map showing the complete journey of an individual package, from pickup to delivery. They can be exported to a different observability platform that helps you analyze each step of the journey in detail.

Collector: Imagine a central hub where all tracking information is gathered. The Collector collects data from all the tracking labels, processes it, and sends it for analysis. The Collector can gather data from multiple sources and send it to different places. However, it isn't always necessary, you can export data directly from your app to an observability platform.

Receiver: The Collector uses receivers to gather data from different sources. A receiver is like the entry point for data into the Collector, allowing it to receive telemetry data from various components like your application, services, or other systems.
Processor: Once the data is received, the processor takes over. Think of the processor as the person in the central hub who organizes, filters, and enhances the tracking information before it is sent out. Processors allow you to modify the data, such as adding additional attributes or dropping unnecessary information, ensuring that only the most relevant data moves forward.

Exporter: The Exporter sends the collected data to its final destination, like an observability platform or database. You can configure your app to export directly to a backend or use a Collector to handle the exporting.

Observability Platform: This is like a dashboard that shows the progress of all packages. It turns the collected data into insights, graphs, alerts, and dashboards, so you can understand what's happening.

Using OpenTelemetry

Currently, I am working with:

Azure Kubernetes Service (AKS)
Database: For data storage.
Redis: For caching.
Azure App Configuration: To manage application settings centrally.
Java with SpringBoot

We need a way to track the interactions and flow between all these components. This will help us understand how they work together and gain valuable insights, not just identify potential issues.

Starting with Azure Monitor

My journey began with the Azure Monitor instrumentation library in my Spring Boot app. This approach allowed me to auto-instrument our services and send telemetry data directly to Azure Application Insights without any code changes. For setup details, you can refer to the official Azure Monitor setup documentation.

Limitations

I wanted to add custom properties to the telemetry data and be more intentional with the data we are sending, ensuring we collect only what is necessary and meaningful.

Code Changes:
Adding custom properties was not as straightforward as I hoped. (Note: There might be a way to do this without modifying code, but I couldn’t get it to work.). For more intentional insights tailored to our business needs, manual instrumentation is required.

Vendor Lock-In: I also wanted to avoid being locked into a specific vendor.

Experimentation: I had been experimenting with OpenTelemetry on the side, specifically using auto-instrumentation for Python on one of my FastAPI services.

Switching to OpenTelemetry

Initial Goal: I wanted to avoid using a Collector and instead export telemetry data directly to Azure. The Collector adds an extra layer of setup and configuration, which feels overwhelming while I'm still trying to grasp all of this.

Limitation: The Spring OpenTelemetry auto-instrumentation library defaults to using OTLP as the exporter, and I couldn't find a straightforward way to configure it to export directly to Azure.

Azure Monitor Exporter: I thought Azure Monitor Exporter could be used to send data directly to Azure and avoid the Collector while using the auto-instrumentation Spring starter for OpenTelemetry.

Introducing the Collector: Although my initial goal was to avoid adding a Collector, I eventually introduced one to solve the challenge of exporting directly to Azure Monitor. It turned out to be more manageable than I had anticipated 😮‍💨.

Note: If anyone knows how to do this without a Collector, I'd love to know!

Deploying the Collector

OTel Helm Chart: I used the OpenTelemetry Helm chart to deploy the Collector in our AKS cluster.

Collector Contrib Image: I referenced the Collector contrib image, which includes Azure Monitor as an available exporter.

Outcome: Since we have a lot of services, the Collector gives us the ability to transform and enrich data before exporting it. Additionally, having a central place for processing data makes managing and scaling telemetry across multiple services much easier.

Setup without Collector

+----------------------------------------+
|          Instrumented Applications     |
| (e.g., FastAPI, Spring Boot with OTel) |
|----------------------------------------|
| - Generates telemetry (traces, metrics,|
|   logs) using OTel SDK                 |
| - Has its own Exporter 🚀              |
+-----------------|----------------------+
                  |
                  v
+----------------------------------------+
|         🛰️ Observability Backend       |
|----------------------------------------|
| - Data Storage                         |
| - Query & Visualization (e.g.,Azure    |
| Honeycomb, Grafana, Prometheus         |
| 📊 Insights and Monitoring             |
+----------------------------------------+

Setup with Collector

+----------------------------------------+
|          Instrumented Applications     |
| (e.g., FastAPI, Spring Boot with OTel) |
|----------------------------------------|
| - Generates telemetry (traces, metrics,|
|   logs) using OTel SDK                 |
| - Has simple Exporter to Collector 🚀  |
+-----------------|----------------------+
                  |
                  v
+----------------------------------------+
|       🚀 OpenTelemetry Collector       |
|----------------------------------------|
| 📥 Receivers                           |
|   - Collect telemetry data from apps   |
|                                        |
| 🔄 Processors                          |
|   - Transform and enrich data          |
|                                        |
| 📤 Exporters                           |
|   - Send data to observability backend |
+-----------------|----------------------+
                  |
                  v
+----------------------------------------+
|          🛰️ Observability Backend      |
|----------------------------------------|
| - Data Storage                         |
| - Query & Visualization (e.g., Azure   |
|   Honeycomb, Prometheus, Grafana)      |  
| 📊 Insights and Monitoring             |
+----------------------------------------+

Sample values.yaml for deploying collector to AKS using Helm


image:
  repository: "otel/opentelemetry-collector-contrib"
  tag: "latest"
config:
  receivers:
    otlp:
      protocols:
        http:
          endpoint: "0.0.0.0:4318"
        grpc:
          endpoint: "0.0.0.0:4317"

  processors:
    batch:
      send_batch_size: 1024

  exporters:
    azuremonitor:
      connection_string: "<your-connection-string>"

  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [batch]
        exporters: [azuremonitor]
      metrics:
        receivers: [otlp]
        processors: [batch]
        exporters: [azuremonitor]
      logs:
        receivers: [otlp]
        processors: [batch]
        exporters: [azuremonitor]
  extensions:
    health_check:
        endpoint: "0.0.0.0:13133"

Preparing for an OpenTelemetry Workshop

Diana — Wed, 06 Nov 2024 06:44:14 +0000

I recently learned that Honeycomb is hosting a virtual workshop on November 12th called Honeycomb's Advanced Instrumentation with OpenTelemetry. I enjoyed learning about Honeycomb's mission at their Observability Day, so I’m looking forward to this one 🙊.

If you're curious about instrumentation with OpenTelemetry, this is a great opportunity to learn tips and get insights. The event will cover advanced instrumentation, distributed tracing, debugging, and observability strategies.

Getting Started with OpenTelemetry

I've come to see OpenTelemetry as a bridge between applications and observability platforms like Honeycomb. It helps streamline how we collect, process, and analyze data, making it easier to understand our application's behavior. OpenTelemetry’s auto-instrumentation option gives us a quick way to start collecting data with no code changes. If you haven’t used OpenTelemetry before, this is a great first step to get familiar before attending the workshop.

OpenTelemetry at High Level

Imagine OpenTelemetry as adding sensors to your application so it can capture data like metrics, logs, and traces. Here’s my understanding:

Instrument: Adding markers in your code is like installing sensors in a car to track important information, such as fuel levels, speed, or engine temperature. OpenTelemetry’s auto-instrumentation provides a quick setup to gather essential data, like request latency and error rates, with no code changes.

Collect: Once data is captured, it needs to be collected and processed, similar to how a car’s central computer gathers information from various sensors. OpenTelemetry’s Collector serves as this central hub, pulling in data from one or multiple sources. You can customize how the data is transformed using a configuration file to fit your specific needs. The Collector can run as a standalone service for centralized data collection, or as a lightweight agent on the same host as your application.

Transfer to an Observability Platform: After processing, the data is sent to an observability platform, like a car’s dashboard, where you can view, analyze, and interpret it. Platforms like Honeycomb let you track trends, find issues, and make decisions to improve your application’s health and performance.

My Journey to Understanding Observability

Diana — Thu, 31 Oct 2024 22:54:22 +0000

First Impressions 🤔

When I first heard about observability, my reaction was, “Observability? Isn’t that just monitoring?” But as I dove deeper, I discovered it was much more. It’s an approach that could change how we understand and solve system issues, fostering a more collaborative and connected work culture. Observability isn't just about tools or troubleshooting, it hints at a cultural shift toward a more fulfilling work experience 💡.

So, What Is Observability? 😵‍💫

At its heart, observability is about understanding system performance through the data it produces. Originating from control theory and adapted for software engineering, observability is built on three main pillars:

Logs 📖: Logs serve as the system’s diary, recording events and errors to help teams troubleshoot.

Metrics 📈: Like a report card, metrics reveal performance over time, capturing things like response times and error rates.

Traces 👣: Traces track the journey of requests through the system, exposing bottlenecks and data flow.

To me, observability isn’t just about these pillars; it’s about diving into the inner workings of our system with curiosity. This perspective helps us look beyond the familiar to uncover insights that lead to effective, long-term solutions.

Navigating the Tooling Landscape 🛠️

As I explored tools to achieve observability, I found there was a tool for every step, some focused on data collection, others on visualization for each pillar. But with so many options, it quickly became overwhelming to decide where to start and how to integrate them without adding unnecessary complexity. I realized that many organizations face a similar challenge: balancing the adoption of new tools with the stability of existing workflows. My goal was to understand how others strike this balance to build resilient systems that go beyond quick fixes and late-night troubleshooting.

Observability: More Than Meets the Metrics 🕵️‍♀️

Recently, I attended Honeycomb’s Observability Day and learned about a new way of thinking about observability. Honeycomb introduced Observability 2.0, a natural evolution designed to meet the needs of today’s complex systems.

In traditional observability, or Observability 1.0, logs, metrics, and traces are treated as separate pieces, each needing its own analysis. While this can work, it often adds complexity by requiring teams to stitch together information from different tools.

Observability 2.0, on the other hand, brings all of these data streams together into a single “source of truth.”
This integrated approach makes it easier to spot patterns, identify root causes, and gain valuable insights without jumping between tools or datasets. For me, Observability 2.0 represents a simpler, more flexible way forward, offering a clearer path for anyone looking to improve their systems and stay ready for whatever challenges come next.

Connecting the Puzzle Pieces 🧩

At the conference, we explored tools Honeycomb integrates with its platform: OpenTelemetry, Gremlin, and Honeycomb itself. Each tool simplified the observability process with its own specific purpose, and during our hands-on labs, the pieces started coming together in my mind.

OpenTelemetry: Acts as the 'messenger' for observability data, gathering logs, metrics, and traces from various sources and delivering them to Honeycomb. Its role as a 'data collector and transformer' finally clicked for me, motivating me to start exploring OpenTelemetry in my own projects.

Honeycomb: Functions as the “control room,” displaying all data from OpenTelemetry in an accessible format. It provides a comprehensive view of system activity, enabling teams to detect patterns, troubleshoot, and improve performance.

Gremlin: A chaos engineering tool, Gremlin simulates real-world issues like server outages, helping teams identify weaknesses before they escalate, like a “practice drill” for your system.

Looking Ahead 🤩

As I continue this journey, I’m genuinely excited to explore OpenTelemetry more deeply. Seeing its potential through Honeycomb’s perspective has given me new insight into how it can reshape our approach to observability. In my next post, I’ll share my hands-on experiences with OpenTelemetry, its benefits, challenges, and the lessons I’m picking up along the way.
I’d love for you to join me as we dig into these tools together, bringing curiosity and a collaborative spirit to our development practices.