Forem: Peter McConnell

Building an XDP eBPF Program with C and Golang: A Step-by-Step Guide

Peter McConnell — Thu, 18 May 2023 14:09:52 +0000

Introduction

In today's highly connected and data-driven world, network performance is crucial for ensuring efficient communication and optimal user experience. XDP (eXpress Data Path) and eBPF (extended Berkeley Packet Filter) have emerged as powerful technologies that enable high-performance packet processing and network optimization. In this step-by-step guide, we will explore the process of building an XDP eBPF program using C and Golang. XDP allows for early packet interception at the network interface driver level, while eBPF provides a flexible and efficient execution environment for custom packet processing logic. Together, they offer an unprecedented level of control and performance in networking applications. Our project, named "dilih" (drop it like it's hot), demonstrates how to build a simple chaos engineering tool that arbitrarily drops packets on a given network interface, which can be a useful tool to allow application developers to understand how their products behave when the network isn't. Through this guide, you will gain a basic understanding of XDP, eBPF, and their practical applications in network manipulation.

note: you can find the entire codebase for this article here: https://github.com/peter-mcconnell/dilih

Project Overview

The project aims to build an XDP (eXpress Data Path) eBPF (extended Berkeley Packet Filter) program using C and Golang. Named "dilih" (drop it like it's hot, as it drops packets like ... they're hot?), the program serves as a simple chaos engineering tool that randomly drops around 50% of packets on a given network interface. This project demonstrates the power and flexibility of XDP and eBPF in controlling packet processing at high speed, making it an ideal starting point for understanding these technologies.

The XDP eBPF program, implemented in C, hooks into the Linux kernel's networking stack at an early stage to intercept packets and decide their fate. Using a simple randomization mechanism the program selectively drops packets allowing for controlled chaos in network traffic. Additionally, the program utilizes eBPF's perf event mechanism to gather statistics and measure the processing time for dropped and passed packets.

The accompanying Golang application interacts with the XDP eBPF program, providing a user-friendly interface to monitor the packet drop behavior and visualize performance statistics. It leverages eBPF maps to extract and aggregate the collected data from kernel space, allowing users to gain insights into the impact of dropped packets and the efficiency of packet processing.

Setting Up the Development Environment

To get started with building the XDP eBPF program with C and Golang, you need to set up your development environment. Follow these steps to ensure that you have all the necessary tools and dependencies in place:

Install Development Tools

First, ensure that you have the required development tools installed on your system. This includes packages like clang, llvm, and bpftool. You can install these tools using the package manager available on your Linux distribution however I would recommend investing a little time to build these tools from source as it will give you greater control over the flags and features built into these tools.

If you are curious about my exact LLVM / Clang setup, I use the following ansible tasks for my configuration:

https://github.com/peter-mcconnell/.dotfiles/blob/master/tasks/llvm.yaml
https://github.com/peter-mcconnell/.dotfiles/blob/master/tasks/debugtools.yaml
https://github.com/peter-mcconnell/.dotfiles/blob/master/tasks/docker.yaml
Install Golang

Next, you need to install Golang, which is the programming language used for the accompanying Golang application. Visit the official Golang website at https://golang.org and follow the installation instructions specific to your operating system. Once installed, make sure the go command is accessible from the command line by adding the appropriate binary directory to your system's PATH.

If you are curious about my exact Golang setup, I use the following ansible task for my configuration:

https://github.com/peter-mcconnell/.dotfiles/blob/master/tasks/golang.yaml
Install Project Dependencies

go dependencies

Navigate to the project's root directory and install the required Golang dependencies by running the following command:
```
go mod download
```
This command will fetch and install the necessary Golang packages defined in the project's go.mod file.

libbpf

We are going to use libbpf in our C code. In the dilih repo we added this as a git submodule but you can choose to manage it elsewhere if you like. When we build our C program later we'll include libbpf with -I../libbpf/src
(Optional) IDE configuration

Whatever your editor of choice should be, invest some time in making sure it is set up for C and Golang. Particularly for autocomplete, linting, symbol detection etc. This will make your life much easier.

If you are curious about my exact setup, I use the following repo to install neovim, configure my LSP and setup everything else I need for development:

https://github.com/peter-mcconnell/.dotfiles/

Writing the XDP eBPF Program in C

The XDP (eXpress Data Path) program is implemented using the eBPF (extended Berkeley Packet Filter) framework in C, with some help from libbpf. It allows us to intercept packets at an early stage in the Linux kernel's networking stack and perform custom packet processing logic. In this section, we will walk through the steps to write the XDP eBPF program in C.

Understanding the Program Logic

Before diving into the code, let's understand the logic of our XDP program. The goal is to randomly drop ~50% of packets on a given network interface. We will use a randomization mechanism to decide whether to drop or pass each packet ("is random number even?"). The program will also collect statistics and measure the processing time for dropped and passed packets using eBPF's perf event mechanism. Our BPF program runs in kernel space but we'll want to expose data to userspace, so we'll use BPF maps to expose data to our Go program.
Creating the Program Source File

Start by creating a new file called dilih_kern.c in a ./bpf/ directory in your project. This file will contain our XDP eBPF program logic. Open the file in your favorite text editor.
Defining the required headers and structures

To begin, include the necessary headers and define the required structures for our XDP program. We need bpf.h and bpf_helpers.h which provide useful structures and helper functions.
```
#include <linux/bpf.h>
#include <bpf_helpers.h>
```
Defining Data Structures and Maps

Next, define the necessary data structures and maps that our XDP program will utilize. We will use a struct to represent the perf event data, and a BPF_MAP_TYPE_PERF_EVENT_ARRAY map to store the perf events. Define the following structures and maps:
```
struct perf_trace_event {
    __u64 timestamp;
    __u32 processing_time_ns;
    __u8 type;
};

#define TYPE_ENTER 1
#define TYPE_DROP 2
#define TYPE_PASS 3

struct {
    __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
    __uint(key_size, sizeof(int));
    __uint(value_size, sizeof(struct perf_trace_event));
    __uint(max_entries, 1024);
} output_map SEC(".maps");
```
The output_map map will be used to store the perf events generated by our XDP program. The TYPE_* definitions will make our code more readable later.
Implementing the XDP Program Function

Now, it's time to implement the XDP program function itself. Begin by declaring the XDP function with the appropriate signature:
```
SEC("xdp")
int xdp_dilih(struct xdp_md *ctx)
{
    // Add program logic here ... detailed in the next step
}
```
The xdp_dilih function will serve as our XDP eBPF program entry point. It will be called for every incoming packet.
Handling Perf Events and Collecting Data

Inside the xdp_dilih function, we can handle perf events to collect data and measure processing time. We have already defined the output_map to store these events. Use the bpf_perf_event_output helper function to emit perf events to the map.
```
struct perf_trace_event e = {};

// Perf event for entering xdp program
e.timestamp = bpf_ktime_get_ns();
e.type = TYPE_ENTER;
e.processing_time_ns = 0;
bpf_perf_event_output(ctx, &output_map, BPF_F_CURRENT_CPU, &e, sizeof(e));

// Packet dropping logic
if (bpf_get_prandom_u32() % 2 == 0) {
    // Perf event for dropping packet
    e.type = TYPE_DROP;
    __u64 ts = bpf_ktime_get_ns();
    e.processing_time_ns = ts - e.timestamp;
    e.timestamp = ts;
    bpf_perf_event_output(ctx, &output_map, BPF_F_CURRENT_CPU, &e, sizeof(e));
    return XDP_DROP;
}

// Perf event for passing packet
e.type = TYPE_PASS;
__u64 ts = bpf_ktime_get_ns();
e.processing_time_ns = ts - e.timestamp;
e.timestamp = ts;
bpf_perf_event_output(ctx, &output_map, BPF_F_CURRENT_CPU, &e, sizeof(e));

return XDP_PASS;
```
In this section of the code, we handle the perf events to collect data and measure the processing time of dropped and passed packets. We first emit a perf event when entering the XDP program (type 1). Then, we use a randomization mechanism to decide whether to drop or pass the packet. If the packet is dropped, we emit a perf event with type 2 and return XDP_DROP. If the packet is passed, we emit a perf event with type 3 and return XDP_PASS.

The bpf_ktime_get_ns() function is used to measure the timestamp (nanoseconds since system boot, excluding suspend time) and processing time of the packet. The bpf_get_prandom_u32() function generates a random value that helps in deciding whether to drop or pass the packet - the "is this random number even?" part.

Additionally, we use bpf_printk() to print debug messages that can be accessed through the kernel's trace buffer.

That concludes the implementation of the XDP eBPF program in C. This program will selectively drop packets based on a randomization mechanism and emit perf events for collecting data and measuring processing time.

Compiling and Loading the XDP eBPF Program

Once we have written the XDP eBPF program in C, the next step is to compile it and load it into the kernel. In this section, we will walk through the steps to compile and load the XDP eBPF program.

Compiling the XDP Program

To compile the XDP program, we will use the LLVM Clang compiler with the appropriate flags. Open a terminal and navigate to the bpf directory where the dilih_kern.c file is located. Then, run the following command:

clang -S \
    -g \
    -target bpf \
    -I../libbpf/src\
    -Wall \
    -Werror \
    -O2 -emit-llvm -c -o dilih_kern.ll dilih_kern.c

Let me explain the flags:

-S Emit an intermediate representation (IR) assembly code file instead of generating object code. This step is used to generate LLVM IR code.
-g Include BTF information
-target bpf Specify the target architecture as "bpf" (Berkeley Packet Filter), indicating that the code being compiled is intended to run on eBPF.
-I../libbpf/src Add the path ../libbpf/src to the include search paths. This allows the compiler to find the necessary header files (bpf helper files) from the libbpf library.`Wall Enable all compiler warning messages.
-Werror Treat all warnings as errors, causing the compilation process to fail if any warnings are encountered.
-O2 Apply optimization level 2 to the generated code. This level of optimization focuses on improving performance without sacrificing code size. This is actually a requirement for some BPF usecases, though I'm struggling to recall right now what they are. TODO
-emit-llvm Instruct the compiler to emit LLVM IR code as the output.
-c Compile the input source file without linking, producing an object file.
-o dilih_kern.ll: Specify the output file name for the generated LLVM IR code as dilih_kern.ll.

Now we use the llc command is used to further process the LLVM IR code and generate the final object file:

llc -march=bpf -filetype=obj -O2 -o dilih_kern.o dilih_kern.ll

Let me explain the flags:

-march=bpf Specify the target architecture as "bpf" for the code generation stage.
-filetype=obj Specify the desired output file type as an object file.
-O2 Apply optimization level 2 to the generated code during the code generation stage.
-o Specify the output file name for the generated object code as dilih_kern.o.

This command compiles the dilih_kern.c file into a BPF object file named dilih_kern.o. The -target bpf flag specifies the target architecture as BPF, and the -O2 flag enables optimization.

Loading the XDP Program

To load the XDP program into the kernel, we will use the bpftool command-line utility. Ensure that you have the bpftool utility installed on your system. If it's not already installed, you can typically install it using your distribution's package manager.

In the terminal, run the following command to load the XDP program:

sudo bpftool prog load dilih\_kern.o /sys/fs/bpf/dilih

This command loads the dilih_kern.o object file and pins it into the /sys/fs/bpf/dilih location. Adjust the path as necessary based on your system configuration. The bpftool utility will handle the loading process and verify the program's validity.

Attaching the XDP Program

After loading the XDP program, we need to attach it to a network interface to start intercepting packets. To attach the XDP program, run the following command:

sudo bpftool net attach xdp pinned /sys/fs/bpf/dilih dev <interface>

you can get by running `ip link'

Replace with the name of the network interface you want to attach the XDP program to. For example, eth0. This command attaches the XDP program to the specified interface, enabling it to intercept incoming packets.

Makefile

For convenience, lets throw some of what we learned above into a Makefile at ./bpf/Makefile. I'll not go into depths about how Makefiles work in this article but will summarise the functionality after the code snippet:
``

TARGET = dilih
BPF_TARGET = ${TARGET:=_kern}
BPF_C = ${BPF_TARGET:=.c}
BPF_OBJ = ${BPF_C:.c=.o}

BPF_PINNED_PATH := /sys/fs/bpf/$(TARGET)
XDP_NAME := dilih
DEV := ens160

xdp: $(BPF_OBJ)
        -bpftool net detach xdpgeneric dev $(DEV)
        rm -f $(BPF_PINNED_PATH)
        bpftool prog load $(BPF_OBJ) $(BPF_PINNED_PATH)
        bpftool net attach xdpgeneric pinned $(BPF_PINNED_PATH) dev $(DEV)

$(BPF_OBJ): %.o: %.c
        clang -S \
                -g \
                -target bpf \
          -I../libbpf/src\
                -Wall \
                -Werror \
                -O2 -emit-llvm -c -o ${@:.o=.ll} $<
        llc -march=bpf -filetype=obj -O2 -o $@ ${@:.o=.ll}

clean:
        -bpftool net detach xdpgeneric dev $(DEV)
        sudo rm -f $(BPF_PINNED_PATH)
        rm -f $(BPF_OBJ)
        rm -f ${BPF_OBJ:.o=.ll}

With this file in place and make installed you can run something like DEV=eth0 make to compile and load the eBPF program and DEV=eth0 make clean to remove the files and unload the eBPF program.

Congratulations! You have successfully compiled and loaded the XDP eBPF program into the kernel and attached it to a network interface. The program is now ready to intercept and process packets based on your defined logic.

Please note that the compilation and loading process may vary slightly depending on your system configuration and specific requirements. Make sure to adjust the commands accordingly and refer to the documentation of the tools and utilities used.

Writing the Golang Application

In this section, we will write a Golang application that interacts with the XDP eBPF program and collects metrics. The Golang application will communicate with the loaded XDP program, read the perf events, and display statistics based on the collected data.

Writing the Golang Application Code

Let's create a new file named main.go and open it in a text editor. This file will contain the code for our Golang application. Copy and paste the following code into main.go:

package main

import (
        "encoding/binary"
        "fmt"
        "net"
        "os"
        "os/signal"
        "syscall"

        "github.com/cilium/ebpf"
        "github.com/cilium/ebpf/link"
        "github.com/cilium/ebpf/perf"
)

const (
        TYPE_ENTER = 1
        TYPE_DROP  = 2
        TYPE_PASS  = 3
)

type event struct {
        TimeSinceBoot  uint64
        ProcessingTime uint32
        Type           uint8
}

const ringBufferSize = 128 // size of ring buffer used to calculate average processing times
type ringBuffer struct {
        data   [ringBufferSize]uint32
        start  int
        pointer int
        filled bool
}

func (rb *ringBuffer) add(val uint32) {
        if rb.pointer < ringBufferSize {
                rb.pointer++
        } else {
                rb.filled = true
                rb.pointer= 1
        }
        rb.data[rb.pointer-1] = val
}

func (rb *ringBuffer) avg() float32 {
        if rb.pointer == 0 {
                return 0
        }
        sum := uint32(0)
        for _, val := range rb.data {
                sum += uint32(val)
        }
        if rb.filled {
                return float32(sum) / float32(ringBufferSize)
        }
        return float32(sum) / float32(rb.pointer)
}

func main() {
        spec, err := ebpf.LoadCollectionSpec("bpf/dilih_kern.o")
        if err != nil {
                panic(err)
        }

        coll, err := ebpf.NewCollection(spec)
        if err != nil {
                panic(fmt.Sprintf("Failed to create new collection: %v\n", err))
        }
        defer coll.Close()

        prog := coll.Programs["xdp_dilih"]
        if prog == nil {
                panic("No program named 'xdp_dilih' found in collection")
        }

        iface := os.Getenv("INTERFACE")
        if iface == "" {
                panic("No interface specified. Please set the INTERFACE environment variable to the name of the interface to be use")
        }
        iface_idx, err := net.InterfaceByName(iface)
        if err != nil {
                panic(fmt.Sprintf("Failed to get interface %s: %v\n", iface, err))
        }
        opts := link.XDPOptions{
                Program:   prog,
                Interface: iface_idx.Index,
                // Flags is one of XDPAttachFlags (optional).
        }
        lnk, err := link.AttachXDP(opts)
        if err != nil {
                panic(err)
        }
        defer lnk.Close()

        fmt.Println("Successfully loaded and attached BPF program.")

        // handle perf events
        outputMap, ok := coll.Maps["output_map"]
        if !ok {
                panic("No map named 'output_map' found in collection")
        }
        perfEvent, err := perf.NewReader(outputMap, 4096)
        if err != nil {
                panic(fmt.Sprintf("Failed to create perf event reader: %v\n", err))
        }
        defer perfEvent.Close()
        buckets := map[uint8]uint32{
                TYPE_ENTER: 0, // bpf program entered
                TYPE_DROP: 0, // bpf program dropped
                TYPE_PASS: 0, // bpf program passed
        }

        processingTimePassed := &ringBuffer{}
        processingTimeDropped := &ringBuffer{}

        go func() {
                // var event event
                for {
                        record, err := perfEvent.Read()
                        if err != nil {
                                fmt.Println(err)
                                continue
                        }

                        var e event
                        if len(record.RawSample) < 12 {
                                fmt.Println("Invalid sample size")
                                continue
                        }
                        // time since boot in the first 8 bytes
                        e.TimeSinceBoot = binary.LittleEndian.Uint64(record.RawSample[:8])
                        // processing time in the next 4 bytes
                        e.ProcessingTime = binary.LittleEndian.Uint32(record.RawSample[8:12])
                        // type in the last byte
                        e.Type = uint8(record.RawSample[12])
                        buckets[e.Type]++

                        if e.Type == TYPE_ENTER {
                                continue
                        }
                        if e.Type == TYPE_DROP {
                                processingTimeDropped.add(e.ProcessingTime)
                        } else if e.Type == TYPE_PASS {
                                processingTimePassed.add(e.ProcessingTime)
                        }

                        fmt.Print("\033[H\033[2J")
                        fmt.Printf("total: %d. passed: %d. dropped: %d. passed processing time avg (ns): %f. dropped processing time avg (ns): %f\n", buckets[TYPE_ENTER], buckets[TYPE_PASS], buckets[TYPE_DROP], processingTimePassed.avg(), processingTimeDropped.avg())
                }
        }()

        c := make(chan os.Signal, 1)
        signal.Notify(c, os.Interrupt, syscall.SIGTERM)
        <-c
}

You may need to run go mod init && go mod tidy if you haven't already done so.

The code sets up the necessary components for the Golang application. It loads the BPF program, attaches it to the specified network interface, and initializes the perf event reader. However, the code to read and process the perf events is yet to be implemented.

That concludes the content for the "Writing the Golang Application" section. Feel free to modify and customize the content according to your needs.

Building and Running the Project

Now that we have implemented the XDP eBPF program in C and the Golang application, let's build and run the project.

Building the XDP eBPF Program

You can skip this step if you have already compiled the dilih_kern.o from the steps above.

Before building the XDP eBPF program, ensure that you have the necessary build tools and dependencies installed on your system. You can refer to the project's README or documentation for the specific requirements.

To build the XDP eBPF program, navigate to the bpf directory and run the following command:

make

This command will compile the C code and generate the dilih_kern.o object file.

Building the Golang Application

To build the Golang application, make sure you are in the root directory of the project. Run the following command:

CGO_ENABLED=0 go build

This command will compile the Golang code and generate an executable binary file. Note: we do not need CGO for our application. We could leave it enabled if we wish, but I like to use CGO_ENABLED=0 when I can as it results in a statically compiled binary that I can easily load into containers.

Running the Go Project

sudo ./dilih

You should start to see a summary of the packets processed on the given interface:

Make sure to run the application with elevated privileges (sudo) to access the necessary resources.

The Golang application will start collecting data from the XDP program and display statistics based on the received perf events.

Cleaning Up
To clean up the project and remove the XDP program from the network interface, run the following command:

sudo make clean

This command will detach the XDP program from the network interface and remove any associated artifacts.

That's it! You have successfully built and run the project. Experiment with different network interfaces and observe the packet drop statistics displayed by the Golang application.

Feel free to explore additional features and modifications to enhance the project further.

Testing and Verifying the XDP eBPF Program

Testing and verifying the functionality of the XDP eBPF program is an essential step to ensure its correctness and effectiveness. In this section, we'll cover some testing techniques and verification methods for the XDP program.

Test Environment Setup

To create a suitable test environment, we'll utilize virtual network interfaces (veth devices) to simulate network traffic and observe the behavior of the XDP program.

Install the iproute2 package if it's not already installed on your system. This package provides the necessary tools to manage network interfaces.

Create a pair of veth devices using the following commands:

sudo ip link add veth0 type veth peer name veth1

This command will create two virtual network interfaces (veth0 and veth1) that are connected to each other.

Set the interfaces up and assign IP addresses to them:

sudo ip link set veth0 up
sudo ip link set veth1 up
sudo ip addr add 10.0.0.1/24 dev veth0
sudo ip addr add 10.0.0.2/24 dev veth1

This will bring up the interfaces and assign IP addresses (10.0.0.1 and 10.0.0.2) to them.

With the veth devices set up, we can proceed to test and verify the XDP eBPF program's functionality.

Packet Drop Verification

One of the primary functionalities of the XDP program is to drop a certain percentage of packets. We can verify this behavior by sending packets between the veth devices and observing the packet drop rate.

Open two terminal windows and navigate to the project directory in both of them.

In the first terminal, run the following command to listen for ICMP echo requests (ping):

sudo tcpdump -i veth1 icmp

In the second terminal, send ICMP echo requests (ping) from veth0 to veth1 using the following command:

sudo ip netns exec veth0 ping 10.0.0.2

Observe the output in the first terminal. You should see the ICMP echo requests being captured.

Analyze the packet capture to verify the packet drop rate. If the XDP program is working correctly, approximately 50% of the ICMP echo requests should be dropped, resulting in a reduced number of captured packets.

By performing packet drop verification tests, you can ensure that the XDP program is functioning as expected and dropping packets according to the specified percentage.

Performance Analysis

In addition to functional verification, it's crucial to analyze the performance impact of the XDP eBPF program. This analysis helps evaluate the efficiency and overhead introduced by the program.

Use the provided Golang application to collect performance metrics and statistics from the XDP program. Refer to the "Building and Running the Project" section for instructions on how to run the Golang application.
Monitor and observe the average processing time for both passed and dropped packets. The Golang application displays the average processing time in nanoseconds (ns) for each packet type.

If the average processing time is consistently low, it indicates that the XDP program is performing efficiently and causing minimal processing overhead.

If the average processing time is significantly high, it may indicate that the XDP program is introducing a considerable processing overhead, which may require optimization or further investigation.
Collect data and analyze the performance metrics over an extended period of network traffic to identify any patterns or trends. Look for anomalies or deviations in the processing time that could indicate potential bottlenecks or inefficiencies.
Experiment with different packet drop percentages and observe their impact on the average processing time. By varying the drop rate, you can assess the trade-off between packet loss and processing efficiency.
Performing performance analysis allows you to gain insights into the impact of the XDP eBPF program on network performance and make informed decisions about its optimization and tuning.

Integration and System Testing
To ensure the proper integration of the XDP eBPF program into the overall system, it's essential to perform integration and system testing. This involves testing the interaction between the XDP program, the network stack, and other components of the system.

Construct a test scenario that closely resembles the production environment in which the XDP program will operate. Consider factors such as network traffic patterns, system load, and the presence of other networking components.

Generate realistic network traffic using tools such as packet generators, traffic simulators, or actual production traffic if available.

Monitor the system's behavior, including packet processing, performance metrics, and system resource utilization. Ensure that the XDP program functions as expected and does not introduce any adverse effects on the system.

Test corner cases and edge conditions to validate the robustness and resilience of the XDP program. This includes scenarios such as high network traffic volumes, unusual packet structures, or unexpected network events.

By conducting integration and system testing, you can ensure that the XDP eBPF program seamlessly integrates into the broader system and operates reliably under various conditions.

Conclusion

In this article, we explored the process of building an XDP eBPF program with C and Golang. We started by understanding the basics of XDP and eBPF, followed by setting up the development environment and writing the XDP eBPF program in C. We then integrated the program with a Golang application to collect and analyze performance metrics.

Throughout the journey, we learned how to compile and load the XDP program, build the Golang application, and leverage the power of eBPF and XDP to manipulate network packets and introduce controlled chaos. We also discussed testing methodologies to ensure the correctness, efficiency, and integration of the XDP program within the system.

The ability to leverage eBPF and XDP opens up a world of possibilities for network programmability, performance optimization, and security enhancements. By harnessing the flexibility and programmability of eBPF, developers can create powerful and efficient networking applications.

We encourage you to explore further possibilities with XDP and eBPF, experiment with different scenarios, and dive deeper into the rich ecosystem of eBPF tools and libraries available. Embrace the power of XDP and eBPF to unlock new horizons in network programming and performance optimization.

Happy coding!

Additional Resources

To further expand your knowledge and explore the world of XDP, eBPF, and network programming, here are some valuable resources:

The Cilium Project: Cilium is an open-source project that provides networking and security capabilities powered by eBPF. Their documentation and codebase offer in-depth insights into eBPF and its applications. Visit their website at cilium.io for more information.
The iovisor Project: iovisor is an open-source project that focuses on building tools, libraries, and infrastructure for eBPF-based tracing, monitoring, and networking. Their website at iovisor.org hosts a wealth of resources, including tutorials, documentation, and sample code.
The BCC (BPF Compiler Collection): BCC is a collection of powerful command-line tools and libraries that leverage eBPF for various tracing and performance analysis tasks. The official GitHub repository at github.com/iovisor/bcc provides extensive documentation and examples to help you dive deep into eBPF.
eBPF.io: eBPF.io is a community-driven website dedicated to providing resources, tutorials, and news about eBPF. It features articles, case studies, and a curated list of tools and libraries related to eBPF. Explore the website at ebpf.io to stay up-to-date with the latest developments in the eBPF ecosystem.
Linux Kernel Documentation: The Linux kernel documentation includes a comprehensive section on eBPF and XDP, covering various aspects, including API references, usage examples, and implementation details. Access the documentation at www.kernel.org/doc/html/latest/bpf to gain a deep understanding of the underlying mechanisms.

These resources serve as valuable references and provide opportunities for further learning and exploration. Delve into the world of XDP, eBPF, and network programming, and unlock the full potential of these technologies in your networking projects.

You can also find the entire codebase for this article here: https://github.com/peter-mcconnell/dilih

Docker Overlayfs: How filesystems work in Docker

Peter McConnell — Thu, 19 Jan 2023 12:53:51 +0000

This is a brief follow up to my article on Docker networking: Network Namespaces, Docker Bridge and DNS

Docker uses the OverlayFS file system to manage the file system of its containers. When a container is run, Docker creates a new layer for the container's file system on top of the base image. This allows the container to have its own file system that is isolated from the host system and other containers.

Running the ubuntu:22.04 image we can see the root file system differs from the host where I'm running it. Below you can see there is a file in root called /.dockerenv:

$ docker run --rm -ti ubuntu:22.04 bash
root@541cc3b62543:/# ls -al /
total 56
drwxr-xr-x   1 root root 4096 Jan 19 11:51 .
drwxr-xr-x   1 root root 4096 Jan 19 11:51 ..
-rwxr-xr-x   1 root root    0 Jan 19 11:51 .dockerenv
lrwxrwxrwx   1 root root    7 Nov 30 02:04 bin -> usr/bin
drwxr-xr-x   2 root root 4096 Apr 18  2022 boot
drwxr-xr-x   5 root root  360 Jan 19 11:51 dev
drwxr-xr-x   1 root root 4096 Jan 19 11:51 etc
drwxr-xr-x   2 root root 4096 Apr 18  2022 home
lrwxrwxrwx   1 root root    7 Nov 30 02:04 lib -> usr/lib
lrwxrwxrwx   1 root root    9 Nov 30 02:04 lib32 -> usr/lib32
lrwxrwxrwx   1 root root    9 Nov 30 02:04 lib64 -> usr/lib64
lrwxrwxrwx   1 root root   10 Nov 30 02:04 libx32 -> usr/libx32
drwxr-xr-x   2 root root 4096 Nov 30 02:04 media
drwxr-xr-x   2 root root 4096 Nov 30 02:04 mnt
drwxr-xr-x   2 root root 4096 Nov 30 02:04 opt
dr-xr-xr-x 491 root root    0 Jan 19 11:51 proc
drwx------   2 root root 4096 Nov 30 02:07 root
drwxr-xr-x   5 root root 4096 Nov 30 02:07 run
lrwxrwxrwx   1 root root    8 Nov 30 02:04 sbin -> usr/sbin
drwxr-xr-x   2 root root 4096 Nov 30 02:04 srv
dr-xr-xr-x  13 root root    0 Jan 19 11:51 sys
drwxrwxrwt   2 root root 4096 Nov 30 02:07 tmp
drwxr-xr-x  14 root root 4096 Nov 30 02:04 usr
drwxr-xr-x  11 root root 4096 Nov 30 02:07 var

Which does not exist at root on the host running the container:

root@541cc3b62543:/# 
root@541cc3b62543:/# exit
exit
$ stat /.dockerenv
stat: cannot statx '/.dockerenv': No such file or directory

So ... where does it exist? To inspect the layers of a running container, you can use the "docker inspect" command followed by the container ID or name. This will return a JSON object containing information about the container, including its layers. To view this we'll re-run our ubuntu:22.04 container, grab the ID and inspect it:

$ docker run --rm -d -ti ubuntu:22.04 bash
6a9014d7ebfddb3a107b29aca3764f24e51f64fda1e8b8cec135c18923daefeb
# lower directory
$ docker inspect 6a9014d7ebfddb3a107b29aca3764f24e51f64fda1e8b8cec135c18923daefeb -f '{{.GraphDriver.Data.LowerDir}}'
/dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a-init/diff:/dockerstore/overlay2/bb9057b4f1980fe004301f181c3313c15c2a75b7c7b7c5a6fe80159d2275f0d3/diff

# upper directory
$ docker inspect 6a9014d7ebfddb3a107b29aca3764f24e51f64fda1e8b8cec135c18923daefeb -f '{{.GraphDriver.Data.UpperDir}}'
/dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a/diff

# merged directory
$ docker inspect 6a9014d7ebfddb3a107b29aca3764f24e51f64fda1e8b8cec135c18923daefeb -f '{{.GraphDriver.Data.MergedDir}}'
/dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a/merged

I'll keep this container running and we'll dig into these contents shortly.

When a container is run, its layers are stored in the host system's file system, typically in the /var/lib/docker/overlay2 directory. You can see mine is in /dockerstore/ as I have manually set data-root in /etc/docker/daemon.json for the host that I'm testing this on. Each layer is represented by a directory that contains the files and directories that make up that layer. The topmost layer is the one that the container is currently using, and the lower layers are the ones that are inherited from the base image.

The advantages of using layers in Docker include:

Smaller image size, since multiple containers can share a common base image
Faster container startup time, since only the changes made to the container are stored in new layers
Easier to manage and update containers, since changes can be made to a container's layer without affecting the base image
Greater security, since each container's file system is isolated from other containers and the host system.

Please keep in mind that the information is general and may vary depending on specific scenarios.

Now lets take a deeper look at the filesystem for our running container.

LowerDir

This value is unique in the outputs above in that it's actually two paths, separated by a colon:

$ docker inspect 6a9014d7ebfddb3a107b29aca3764f24e51f64fda1e8b8cec135c18923daefeb -f '{{.GraphDriver.Data.LowerDir}}'
/dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a-init/diff:/dockerstore/overlay2/bb9057b4f1980fe004301f181c3313c15c2a75b7c7b7c5a6fe80159d2275f0d3/diff

The first part (left side of :) is the path to the init layer of the container. this is the layer that contains the initial filesystem of the container, which is based on the base image. We can take a look at the contents of that layer with ls:

sudo ls /dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a-init/diff
dev  etc

The second part (right side of :) is the path to the layer of the container that includes changes from the rest of the Dockerfile. Again we can take a look:

sudo ls /dockerstore/overlay2/bb9057b4f1980fe004301f181c3313c15c2a75b7c7b7c5a6fe80159d2275f0d3/diff
bin  boot  dev  etc  home  lib  lib32  lib64  libx32  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

To better visualise this, lets create our own Dockerfile:

FROM ubuntu:22.04

RUN mkdir -p /testinglowerdir/ && echo -n "hellothere" > /testinglowerdir/foo

Now, given what we learned above, when we run this container the first part of LowerDir should contain all the contents for ubuntu:22.04 and the second part of LowerDir should contain only /testinglowerdir/:

$ docker build -t=test .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM ubuntu:22.04
 ---> 6b7dfa7e8fdb
Step 2/2 : RUN mkdir -p /testinglowerdir/ && echo -n "hellothere" > /testinglowerdir/foo
 ---> Running in e71a7cd5541c
Removing intermediate container e71a7cd5541c
 ---> df924945a2b0
Successfully built df924945a2b0
Successfully tagged test:latest
$ docker run --rm -d -ti test bash
9c9fe0bcd283bc0c9649b77246115e3a09e8885efd53f0e9de09de537bea9188
$ docker inspect 9c9fe0bcd283bc0c9649b77246115e3a09e8885efd53f0e9de09de537bea9188 -f '{{.GraphDriver.Data.LowerDir}}'
/dockerstore/overlay2/5501fd185b14a60317f3e0db485bb8f8c5cf41b7cb1ed0688526ba918938b7bf-init/diff:/dockerstore/overlay2/4d49e9a62bad55c3761ab08ded87f56010b28a40f264896c01e5c1c653b826a8/diff:/dockerstore/overlay2/bb9057b4f1980fe004301f181c3313c15c2a75b7c7b7c5a6fe80159d2275f0d3/diff
$ # show directory contents for second part of LowerDir
$ sudo ls /dockerstore/overlay2/4d49e9a62bad55c3761ab08ded87f56010b28a40f264896c01e5c1c653b826a8/diff
testinglowerdir

UpperDir

$ docker inspect 6a9014d7ebfddb3a107b29aca3764f24e51f64fda1e8b8cec135c18923daefeb -f '{{.GraphDriver.Data.UpperDir}}'
/dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a/diff

The UpperDir contains changes that we've made at runtime. To see this in action we can exec into our container and create a simple directory with a file in the root directory:

docker exec -ti 6a9 bash
root@6a9014d7ebfd:/# mkdir /tutorial
root@6a9014d7ebfd:/# echo 'iseeyou' > /tutorial/ohai

We can now see this in our UpperDir directory:

$ sudo ls /dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a/diff/
root  tutorial
$ sudo cat /dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a/diff/tutorial/ohai
iseeyou

Want to quickly see what files are being created by a running container? This is something the UpperDir can tell you.

MergedDir

$ docker inspect 6a9014d7ebfddb3a107b29aca3764f24e51f64fda1e8b8cec135c18923daefeb -f '{{.GraphDriver.Data.MergedDir}}'
/dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a/merged

I'm sure you've guessed what this one is... This is the merged structure:

$ sudo ls /dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a/merged
bin  boot  dev  etc  home  lib  lib32  lib64  libx32  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  tutorial  usr  var

Here you can see all of the directories from the LowerDir and UpperDir together. We can chroot into this directory to "see what docker sees":

sudo chroot /dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a/merged /bin/bash
root@pete:/# ls -al
total 72
drwxr-xr-x  1 root root 4096 Jan 19 12:21 .
drwxr-xr-x  1 root root 4096 Jan 19 12:21 ..
-rwxr-xr-x  1 root root    0 Jan 19 11:56 .dockerenv
lrwxrwxrwx  1 root root    7 Nov 30 02:04 bin -> usr/bin
drwxr-xr-x  2 root root 4096 Apr 18  2022 boot
drwxr-xr-x  1 root root 4096 Jan 19 11:56 dev
drwxr-xr-x  1 root root 4096 Jan 19 11:56 etc
drwxr-xr-x  2 root root 4096 Apr 18  2022 home
lrwxrwxrwx  1 root root    7 Nov 30 02:04 lib -> usr/lib
lrwxrwxrwx  1 root root    9 Nov 30 02:04 lib32 -> usr/lib32
lrwxrwxrwx  1 root root    9 Nov 30 02:04 lib64 -> usr/lib64
lrwxrwxrwx  1 root root   10 Nov 30 02:04 libx32 -> usr/libx32
drwxr-xr-x  2 root root 4096 Nov 30 02:04 media
drwxr-xr-x  2 root root 4096 Nov 30 02:04 mnt
drwxr-xr-x  2 root root 4096 Nov 30 02:04 opt
drwxr-xr-x  2 root root 4096 Apr 18  2022 proc
drwx------  1 root root 4096 Jan 19 12:16 root
drwxr-xr-x  5 root root 4096 Nov 30 02:07 run
lrwxrwxrwx  1 root root    8 Nov 30 02:04 sbin -> usr/sbin
drwxr-xr-x  2 root root 4096 Nov 30 02:04 srv
drwxr-xr-x  2 root root 4096 Apr 18  2022 sys
drwxrwxrwt  2 root root 4096 Nov 30 02:07 tmp
drwxr-xr-x  2 root root 4096 Jan 19 12:20 tutorial
drwxr-xr-x 14 root root 4096 Nov 30 02:04 usr
drwxr-xr-x 11 root root 4096 Nov 30 02:07 var
root@pete:/# cat /tutorial/ohai 
iseeyou
root@pete:/#

Pretty sweet! Another way / a "better" way that we can get this view is with nsenter:

$ sudo nsenter --target $(docker inspect --format {{.State.Pid}} 6a9) --mount --uts --ipc --net --pid
root@6a9014d7ebfd:/# cat /tutorial/ohai 
iseeyou
root@6a9014d7ebfd:/#

Do it yourself

This has been a quick look into how Docker avails of OverlayFS, but you can of course do this yourself. The basic syntax is:

mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=workdir target

lowerdir is the lower filesystem
upperdir is the upper filesystem
workdir is a directory where the OverlayFS stores metadata about the overlay
target is the mount point where the overlay will be mounted

For example, if you have two directories, /mnt/lower and /mnt/upper, you can create an OverlayFS file system that combines them at /mnt/overlay with the following command:

mount -t overlay overlay -o lowerdir=/mnt/lower,upperdir=/mnt/upper,workdir=/mnt/workdir /mnt/overlay

To view the contents of the overlay, you can simply navigate to the mount point (in this example, /mnt/overlay) and use standard Linux commands to view the files and directories.

You can also use lsblk command to view the mounted overlays in your system and also you can unmount the overlays using umount command.

Please keep in mind that this is a basic example and there are many other options and settings that can be used when creating an OverlayFS file system.

Docker networking: Network Namespaces, Docker and DNS

Peter McConnell — Thu, 19 Jan 2023 08:37:22 +0000

Ever wondered how docker compose lets you communicate between services? This article takes a high level look at network namespaces, Dockers internal DNS and Docker bridge.

Network namespaces are a powerful feature in Linux that allows for the isolation of network stacks, creating multiple virtual networks on a single host. This concept is particularly useful for scenarios such as containerization, where each container needs its own independent network stack. In this article we'll take a look at how docker / docker compose utilize this technology to grant containers network isolation and also take a look at how docker handles cross-container networking.

simple docker run

To see network namespaces in action, we'll run a simple container (without the --network=host flag):

# run nginx in detached mode on host port 8080
docker run --name dummynginx -p 8080:80 -d nginx

If we ran this with --network=host it would share the network namespace of the host and therefore would not have it's own network namespace.

Now we can check to see if we any network namespaces were created for this container:

$ sudo lsns -t net | grep nginx
[sudo] password for pete: 
        NS TYPE NPROCS   PID USER     NETNSID NSFS                           COMMAND
...
4026533009 net      17 52981 root           0 /run/docker/netns/3161e4b47f76 nginx: master process nginx -g daemon off;

In the output above we can see a network namespace was created for a docker nsfs with the command nginx. This seems like a likely culprit. We could confirm this by checking for the nsfs in docker inspect <container id>.

Each container runs in its own network namespace, which means it has its own set of network interfaces, IP addresses, and routing tables. This isolation allows each container to have its own network configuration, without interfering with the host's network or other containers. You can view these resources from the network namespace by using the ip command with nsenter.

In addition, the isolation also allows you to use the same port on the host for different services running in different containers. This is useful when you want to run multiple instances of the same service on a single host, each with its own IP address and port.

Also, this isolation helps in providing security boundaries between different containers, as they can't see or interact with each other's network stack unless explicitly configured to do so.

docker compose

Let's remove that container and see how things look with docker compose which can allow multiple containers to communicate with eachother:

# remove the old nginx container
docker rm -f dummynginx

Now we'll create a simple docker-compose.yaml:

---
version: '3.1'
services:
  nginx_a:
    image: 'nginx:latest'
    ports:
      - '8080:80'
  nginx_b:
    image: 'nginx:latest'

With this file stored as docker-compose.yaml we'll spin it up then check lsns:

$ docker compose up -d
[+] Running 3/3
 ⠿  Network linuxnetworks_default      Created  0.0s
 ⠿ Container linuxnetworks-nginx_b-1   Started  0.3s
 ⠿ Container linuxnetworks-nginx_a-1   Started  0.6s
$ sudo lsns -t net | grep nginx
[sudo] password for pete: 
        NS TYPE NPROCS   PID USER     NETNSID NSFS                           COMMAND
...
4026533008 net      17 57327 root           0 /run/docker/netns/20790e4f73be nginx: master process nginx -g daemon off;
4026533108 net      17 57501 root           1 /run/docker/netns/11b21af68bd5 nginx: master process nginx -g daemon off;

Two network namespaces; one for each container. It's worth noting if we had replicas in each service, each container within that service would get their own network namespace. So how does docker compose allow nginx_a to speak to nginx_b given each container has it's own network namespace?

Firstly lets clarify the mystery we're trying to solve:

docker compose exec nginx_a curl nginx_b
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

We connected to the nginx_a container, then curl'ed nginx_b. How does it work? We'll dig in over the next few sections to understand how Docker makes this possible.

a quick note on the 'links:' property

docker compose supports a links: property. This is not required to allow services to speak to each other. It's main purpose is to allow you to create aliases or to use hostname resolution, linking between containers in different docker compose files.

more info: https://docs.docker.com/compose/compose-file/#links

dockers internal DNS server

Docker runs its own internal DNS server at 127.0.0.11. How do we know this? We can see it:

docker compose exec -ti nginx_a bash
root@b315617ce0dd:/# cat /etc/resolv.conf 
nameserver 127.0.0.11
options ndots:0
root@b315617ce0dd:/#

We can query this from nginx_a by installing dnsutils and using nslookup or dig:

docker compose exec -ti nginx_a bash
root@b315617ce0dd:/# apt update && apt install dnsutils
root@b315617ce0dd:/# nslookup nginx_b 127.0.0.11
Server:         127.0.0.11
Address:        127.0.0.11#53

Non-authoritative answer:
Name:   nginx_b
Address: 172.21.0.2

root@b315617ce0dd:/#

Just to be sure that IP is correct we can validate the IP for nginx_b with docker inspect:

docker inspect $(docker ps -f 'Name=nginx_b' -q) -f '{{.NetworkSettings.Networks.linuxnetworks_default.IPAddress}}'
172.21.0.2

Looks good.

When a container is created and connected to a network, Docker automatically creates a DNS record for that container, using the container name as the hostname and the IP address of the container as the record's value. This enables other containers on the same network to access each other by name, rather than needing to know the IP address of the target container.

virtual network interfaces

When we ran docker compose it created a virtual network interface inside each containers network namespace. To look at the resources in a network namespace we first need to know where the nsfs is. Looking at the example output above from lsns above we seen this line:

4026533008 net      17 57327 root           0 /run/docker/netns/20790e4f73be nginx: master process nginx -g daemon off;

You'll see in the nsfs column we have: /run/docker/netns/20790e4f73be. With this we can enter the namespace and run some commands from that namespace.

You can also get this from docker itself. Running docker ps you can get the CONTAINER ID:

docker ps
CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS          PORTS                                   NAMES
b315617ce0dd   nginx:latest   "/docker-entrypoint.…"   16 minutes ago   Up 16 minutes   0.0.0.0:8080->80/tcp, :::8080->80/tcp   linuxnetworks-nginx_a-1
da9caf14c90c   nginx:latest   "/docker-entrypoint.…"   16 minutes ago   Up 16 minutes   80/tcp                                  linuxnetworks-nginx_b-1

Then you can run inspect to see the network settings:

docker inspect b315617ce0dd -f '{{.NetworkSettings.SandboxKey}}'
/var/run/docker/netns/11b21af68bd5

Or the terribly ugly:

docker inspect $(docker ps -q -f 'Name=nginx_a') -f '{{.NetworkSettings.SandboxKey}}'

Armed with the nsfs path we can enter the namespace and run commands. Lets take a quick look at what interfaces are available in this namespace:

sudo nsenter --net=/run/docker/netns/11b21af68bd5 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
47: eth0@if48: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ac:15:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0

Cool! We're finally peaking behind the curtain.

exploring the namespace

Now that we have the ability to run commands from the namespace we can request some more information about this interface with ip addr show:

sudo nsenter --net=/run/docker/netns/11b21af68bd5 ip addr show eth0
47: eth0@if48: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:15:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.21.0.3/16 brd 172.21.255.255 scope global eth0
       valid_lft forever preferred_lft forever

We can also get the route table information with ip route show:

sudo nsenter --net=/run/docker/netns/11b21af68bd5 ip route show
default via 172.21.0.1 dev eth0 
172.21.0.0/16 dev eth0 proto kernel scope link src 172.21.0.3

And we can even run good old faithful netstat to check out which ports we're listening on:

sudo nsenter --net=/run/docker/netns/11b21af68bd5 netstat -tulnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.11:32889        0.0.0.0:*               LISTEN      16568/dockerd       
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      57501/nginx: master 
tcp6       0      0 :::80                   :::*                    LISTEN      57501/nginx: master 
udp        0      0 127.0.0.11:51543        0.0.0.0:*                           16568/dockerd

docker bridge

Using the ip route command above we seen:

sudo nsenter --net=/run/docker/netns/11b21af68bd5 ip route show
default via 172.21.0.1 dev eth0 
172.21.0.0/16 dev eth0 proto kernel scope link src 172.21.0.3

So what is 172.21.0.1? That is the default gateway, the docker bridge. How can we know? Well, firstly we can list docker networks with:

$ docker network ls
NETWORK ID     NAME                    DRIVER    SCOPE
a52a287eb9f2   bridge                  bridge    local
8f5afced6d3f   host                    host      local
19508f629e30   none                    null      local

And then we can inspect the networks, like so:

$ docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "a52a287eb9f2af463e14ac4a97583b96a9dc66e28b5ac67967321386834e4e24",
        "Created": "2023-01-18T20:00:03.09775174Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

We can see the Gateway for bridge is set to 172.17.0.1.

The Docker bridge default gateway is responsible for connecting containers running in separate network namespaces, using the IP addresses and MAC addresses of the containers connected to it to forward packets to the correct container. When a packet is received by the bridge, it checks the destination IP address of the packet and compares it to the IP addresses of the connected containers. If a match is found, the packet is then forwarded to the corresponding container using the container's MAC address. Additionally, the bridge also uses network address translation (NAT) to rewrite the source IP address of the packet to that of the bridge, allowing the containers to communicate with external networks.

so ... how does nginx_a speak to nginx_b? (recap)

The request starts from nginx_a with curl nginx_a. nginx_a is configured to resolve DNS with Dockers internal DNS, as defined in its /etc/resolv.conf file. Dockers Internal DNS resolves nginx_b to 172.21.0.2 which is within the range of the default gateway in, the docker bridge, as defined in this containers route table. The Docker Bridge acts as a gateway and rewrites the source IP of the packet to the NAT to allow for return traffic using NAT, then forwards the request onto the desired destination at 172.21.0.2. The request then works its way back to the real source, via the default gateway once more.

network_mode

docker compose supports another interesting property that we'll take a quick look at: network_mode:. With this setting we can tell services to share the same network resources. As we're using nginx in both services currently they will both try to bind on port 80 which will cause problems if they are sharing the same resources so for nginx_b we'll create a unique nginx config file netsvc.conf:

server {
    listen       9080;
    server_name  localhost;
    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }
}

This is as simple an nginx config as you can get. Notably we're telling it to listen on 9080 instead of the default 80 so that they can run in the same network namespace without colliding.

Now we'll create an updated docker-compose-netsvc.yaml file:

---
version: '3.1'
services:
  nginx_a:
    image: 'nginx:latest'
    ports:
      - '8080:80'
      - '9080:9080'
  nginx_b:
    image: 'nginx:latest'
    volumes:
      - ./netsvc.conf:/etc/nginx/conf.d/default.conf:ro
    network_mode: "service:nginx_a"

See how we put our port-mapping for nginx_b (to :9080) in nginx_a? With this file in place we'll run our new compose:

docker compose -f docker-compose-netsvc.yaml up -d
[+] Running 2/2
 ⠿ Container linuxnetworks-nginx_a-1  Started  0.4s
 ⠿ Container linuxnetworks-nginx_b-1  Started  0.6s

Running lsns this time we can see that we only have one network namespace (compared to two that we seen with the original docker-compose.yaml file):

sudo lsns -t net | grep nginx
4026533007 net      34 148278 root           0 /run/docker/netns/1e55105e0804 nginx: master process nginx -g daemon off;

We do still have two containers of course:

docker compose ps
NAME                      IMAGE               COMMAND                  SERVICE             CREATED             STATUS              PORTS
linuxnetworks-nginx_a-1   nginx:latest        "/docker-entrypoint.…"   nginx_a             5 minutes ago       Up 5 minutes        0.0.0.0:8080->80/tcp, :::8080->80/tcp, 0.0.0.0:9080->9080/tcp, :::9080->9080/tcp
linuxnetworks-nginx_b-1   nginx:latest        "/docker-entrypoint.…"   nginx_b             5 minutes ago       Up 5 minutes

We can see from docker inspect the nginx_b container has no visible network settings:

$ docker inspect $(docker ps -f 'Name=nginx_b' -q) -f '{{.NetworkSettings.Networks.linuxnetworks_default.IPAddress}}'
<no value>

$ docker inspect $(docker ps -f 'Name=nginx_b' -q) -f '{{.NetworkSettings.SandboxKey}}'

We can check the nginx_a network settings and see that all looks good there:

$ docker inspect $(docker ps -f 'Name=nginx_a' -q) -f '{{.NetworkSettings.Networks.linuxnetworks_default.IPAddress}}'
172.22.0.2
$ docker inspect $(docker ps -f 'Name=nginx_a' -q) -f '{{.NetworkSettings.SandboxKey}}'
/var/run/docker/netns/1e55105e0804

sudo nsenter --net=/var/run/docker/netns/1e55105e0804 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
80: eth0@if81: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ac:16:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0

You can imagine, say, running a VPN as a service and then forcing all other services to use that network namespace and resources. Super cool.

I hope this article has taught you a little about network namespaces and Dockers networking. Happy Hacking o/

Advice for engineers wanting to 'make it'

Peter McConnell — Tue, 10 Jan 2023 11:28:43 +0000

The advice I wish I had been given

As someone with two decades of experience in the tech industry, I have been fortunate to have enjoyed success, but my path has been far from deliberate. In hindsight, there are some steps I could have taken to have increased my ability to be a better engineer and ultimately to have reached my goals sooner. If I could time travel, this is the advice I would give to my younger self:

Understand what you want and how to get it: Working in tech because it's your hobby is great, but you should spend some time thinking about how you want to optimise your time. Consider what you want from your career as you (usually) only get one and time is short. Whether it's an early retirement, mass influence on tech, or reaching millions of users, understanding what you want and using it as the guiding light for decision making is a much more efficient strategy than just going with the flow. And remember, "money" is a valid answer.
Set goals: Now that you know what you want, determine the incremental steps you need to take to get there. Consider who can help you achieve those goals and how you can position yourself to achieve them. Think about what you need to provide to ensure it's a good deal for all parties involved.
It's a craft: So, build on it. That means deliberately doing the things you don't enjoy too. If you don't understand a part of the stack that affects your job, don't ignore it. Force yourself to learn the hard things. Getting better and learning things you don't want to learn is possibly the hardest, yet most important skill I can recommend developing. Being able to deeply understand the work you're doing gives you the confidence to do so much more.
Be someone that's easy to work with: You spend 40+ hours a week with your colleagues, so be someone they want to have around. Being kind, easy to work with, fun, dependable, diligent, a great communicator and having their backs are all extremely important for your career. Sadly, this is an area where some people lack. I've seen technically brilliant individuals held back due to issues with soft skills. I found Simon Sinek's talk on Performance vs. Trust a good example of this: https://www.youtube.com/watch?v=kJdXjtSnZTI&ab_channel=MikeKnight.
You're a service: To state the obvious, an employment is a contract an employer creates in exchange for a service (that's the work that you do). If you want to raise your price, it had better be a good deal for the employer. So, what are they getting? Consider this:

Bob was hired at $100k per year. In a year, Bob developed four features. None of these features brought the business any value. Was that a good deal for the business?

Jane was also hired at $100k per year. In a year, Jane developed a single feature that increased revenue by $300k. Was that a good deal for the business?

It's obvious when in black and white but I know I've had times early in my career when I was accidentally coasting, not really thinking about how good of a deal I was for my employer. Try to ensure that you are working on a) increasing revenue, b) decreasing the cost of business, or c) acting as a catalyst for many others. Talk to your manager about wanting to optimize your effectiveness in these areas. Ensure that you can demonstrate this value on demand if required. Always try to remember that you're being paid for a return of value and it is your duty to ensure you can maximise that value.

Overtime: If your goal is work/life balance, this may not be relevant. However, the reality is that you often get out what you put in. You can have an entire career clocking in 40-hour weeks, but from what I've observed, everyone who gets ahead does so by digging in and putting in the extra work. Don't burn yourself out, of course! Manage your time and take it easy when you can, but when the pressure is on for you or your team, be prepared to knuckle down and put in some overtime. This both solidifies your position as someone who can be depended on and also gives you more time in the saddle.

In summary, the steps outlined above can be distilled into a single plan:

Understand your goals and what you want to achieve
Set specific and measurable goals to get there
Continuously improve your craft and skills, even if it's uncomfortable
Be a valuable team member and easy to work with
Clearly communicate and demonstrate your value to your employer
Be prepared to put in the extra work when necessary.

By following these steps, you can increase your chances of success and reach your desired career outcomes, whether that's financial stability, influence in the industry or a balance lifestyle. Remember that career paths are rarely straight and it's not uncommon to change direction or re-evaluate goals, so be flexible and always keep learning.

Golang debugging tips

Peter McConnell — Sun, 08 Jan 2023 11:54:27 +0000

the 'code' for this article can be found here: https://github.com/peter-mcconnell/petermcconnell.com/blob/main/assets/dummy/godebug/main.go

debugging Golang - the context

This is the tool I reach for when a program isn't behaving how I expect it to and I want to dig into the internals / browse the state of the program at particular points so as to realise why my understanding of the program is wrong.

what are the requirements?

The debugger of choice (for me) is dlv / delve. The reasons for this are at the end of the article.

Install delve: https://github.com/go-delve/delve

git clone https://github.com/go-delve/delve
cd delve
go install github.com/go-delve/delve/cmd/dlv

Have the following pieces of information:

a list of 1 or more points in my code where I'd like to set breakpoints
- filename and line numbers

example program

For the sake of this article we'll create a simple application to debug:

// main.go
package main

import "fmt"

func doubleit(val int) int {
        return val * 3  // should be * 2
}

func main() {
        fmt.Printf("doubleit 2: %d\n", doubleit(2))
        fmt.Printf("doubleit 4: %d\n", doubleit(4))
        fmt.Printf("doubleit 8: %d\n", doubleit(8))
}

When we run this with go run main.go we want it to double the numbers we pass but for some reason we're getting different results.

We will also need a go.mod for Delve to run so also perform a go mod init from this directory.

debug it

To get started from the project directory

dlv debug
Type 'help' for list of commands.
(dlv)

Now set some breakpoints. For this example we'll say we want to debug our doubleit method - the first line of which is at :7:

dlv debug
Type 'help' for list of commands.
(dlv) b main.go:7
Breakpoint 1 set at 0x49c8bb for main.doubleit() ./main.go:7

When we've added all of the breakpoints that we need we can instruct the program to run with c:

dlv debug
Type 'help' for list of commands.
(dlv) b main.go:7
Breakpoint 1 set at 0x49c8bb for main.doubleit() ./main.go:7
(dlv) c
> main.doubleit() ./main.go:7 (hits goroutine(1):1 total:1) (PC: 0x49c8bb)
     2:package main
     3:
     4:import "fmt"
     5:
     6:func doubleit(val int) int {
=>   7: return val * 3 // should be * 2
     8:}
     9:
    10:func main() {
    11: fmt.Printf("doubleit 2: %d\n", doubleit(2))
    12: fmt.Printf("doubleit 4: %d\n", doubleit(4))
(dlv)

Now we've ran our program with an attached debugger and it has paused execution at the breakpoint we set. We can run args to see which arguments where passed to the method:

(dlv) c
> main.doubleit() ./main.go:7 (hits goroutine(1):1 total:1) (PC: 0x49c8bb)
     2:package main
     3:
     4:import "fmt"
     5:
     6:func doubleit(val int) int {
=>   7: return val * 3 // should be * 2
     8:}
     9:
    10:func main() {
    11: fmt.Printf("doubleit 2: %d\n", doubleit(2))
    12: fmt.Printf("doubleit 4: %d\n", doubleit(4))
(dlv) args
val = 2
~r0 = 0

So in this point in the program we're in the doubleit method when it was invoked with a val value of 2. We can print this and other variables using p:

(dlv) p val
2

We can even call methods from this point using call:

(dlv) call doubleit(6)
> main.doubleit() ./main.go:7 (hits goroutine(6):1 total:2) (PC: 0x49c8bb)
     2:package main
     3:
     4:import "fmt"
     5:
     6:func doubleit(val int) int {
=>   7: return val * 3 // should be * 2
     8:}
     9:
    10:func main() {
    11: fmt.Printf("doubleit 2: %d\n", doubleit(2))
    12: fmt.Printf("doubleit 4: %d\n", doubleit(4))
(dlv) args
val = 6

In the example above we hit our own breakpoint set earlier allowing us to print the args for the call.

To walk over the execution we can press n to go to the next point of execution:

(dlv) n
> main.doubleit() ./main.go:6 (PC: 0x49c8a0)
     1:// main.go
     2:package main
     3:
     4:import "fmt"
     5:
=>   6:func doubleit(val int) int {
     7: return val * 3 // should be * 2
     8:}
     9:
    10:func main() {
    11: fmt.Printf("doubleit 2: %d\n", doubleit(2))
(dlv) n
> main.doubleit() ./main.go:7 (hits goroutine(6):2 total:3) (PC: 0x49c8bb)
     2:package main
     3:
     4:import "fmt"
     5:
     6:func doubleit(val int) int {
=>   7: return val * 3 // should be * 2
     8:}
     9:
    10:func main() {
    11: fmt.Printf("doubleit 2: %d\n", doubleit(2))
    12: fmt.Printf("doubleit 4: %d\n", doubleit(4))
(dlv) n

and view the backtrace with bt:

(dlv) bt
0  0x000000000049c8bb in main.doubleit
   at ./main.go:7
1  0x000000000046251f in debugCall256
   at :0
2  0x0000000000407484 in runtime.debugCallWrap2
   at /usr/local/go/src/runtime/debugcall.go:251
3  0x00000000004073b3 in runtime.debugCallWrap1
   at /usr/local/go/src/runtime/debugcall.go:203
4  0x0000000000464ca1 in runtime.goexit
   at /usr/local/go/src/runtime/asm_amd64.s:1594

To view the code around the current point of execution just press l:

(dlv) l
> main.doubleit() ./main.go:7 (hits goroutine(6):3 total:4) (PC: 0x49c8bb)
     2:package main
     3:
     4:import "fmt"
     5:
     6:func doubleit(val int) int {
=>   7: return val * 3 // should be * 2
     8:}
     9:
    10:func main() {
    11: fmt.Printf("doubleit 2: %d\n", doubleit(2))
    12: fmt.Printf("doubleit 4: %d\n", doubleit(4))

Which of course shows our very hard to find logic error, * 3 instead of * 2.

Note: you can also set breakpoints in the stdlib functions (paths will vary depending on your setup):

(dlv) b src/net/http/request.go:899
Breakpoint 1 set at 0x794599 for net/http.NewRequestWithContext() /usr/local/go./net/http/request.go:899

summary

The example above is extremely trivial - where dlv and it's ilk shine are on complex usecases where you may not even know what methods are between the input and output, such as debugging the stdlib. Just this week I used dlv to identify why a POST wasn't honouring a 307 temporary redirect - on inspection, using dlv, I learned that the body is disregarded if it is an unrecognised type https://github.com/golang/go/blob/master/src/net/http/request.go#L899. Having to do this without a debugger would have taken quite a bit of code hopping - the debugger took care of that for me and allowed me to validate argument values as I did it.

why not gdb?

I know some folk feel strongly that gdb is the tool to use for debugging go code, but given the Golang docs itself encourage you to use Delve over GDB I personally stay away from it:

Note that Delve is a better alternative to GDB when debugging Go programs built with the standard toolchain. It understands the Go runtime, data structures, and expressions better than GDB. Delve currently supports Linux, OSX, and Windows on amd64. For the most up-to-date list of supported platforms, please see the Delve documentation.

Exceptions here may be usage of cgo but I'll leave that out for now.

source: https://tip.golang.org/doc/gdb

Python debugging tips

Peter McConnell — Sun, 08 Jan 2023 11:53:10 +0000

the 'code' for this article can be found here: https://github.com/peter-mcconnell/petermcconnell.com/blob/main/assets/dummy/pydebug/main.py

debugging Python - the context

This is the flow I take when faced with a new Python codebase. I often find myself having to debug codebases I've never seen before which has forced me to become very comfortable being lost in code and to develop some patterns that help me find my way. This is what I'm sharing with you today.

I should note that I live in terminals - constantly connecting to servers, containers, colleagues machines, my own homelab etc. To compound this fact my editor of choice also lives in the terminal (Neovim). For that reason this guide is TERMINAL based and as such does not include IDE-based debugging flows (which are solid from what I've seen).

what are the requirements?

The debugger of choice (for me) is ipdb. The reasons for this are at the end of the article.

Install ipdb:

pip3 install --user ipdb

We'll also need to gather information from the refining scope section below.

refining scope

Often (my own usecase) my Python debugging story typically starts with: "This app is broken. It's doing X" which tells me very little about what's wrong and where to look. My first objective is to make the size of the problem statement as small / tight as possible. To do so, before I've looked at any code I try to do the following:

validate that it appears to be an issue with the code and categorise it
- perf issue
- logic issue
- flakiness
- dependency issue
- etc
identify which version of that app I need to debug & where I can get it
identify which part of the codebase (file location, method, line)
identify required inputs (method arguments, environment variables, third party sources etc)
understand what has been tried already to fix the problem
identify stakeholders, urgency etc ...

This serves a few purposes:

ensure I can reproduce the bug
reduce the scope of things that I need to look at
help me understand the business logic / expected results

At this point I should have the confidence to know that the problem requires debugging.

example application

To get started create the following file. This is the simplest possible example I could create so as to keep signal/noise ratio in favour of the actual debugging steps:

#!/usr/bin/env python
# main.py
def doubleit(val):
    return val * 3

if __name__ == "__main__":
    print("doubleit 2: %d", doubleit(2))
    print("doubleit 4: %d", doubleit(4))
    print("doubleit 8: %d", doubleit(8))

We'll use this simple example for our debugging.

using ipdb

From the information gathered earlier lets imagine the outputs were that the program above is spitting out the wrong values. We expect the doubleit lines to show their values being doubled but instead they seem to be trebled (yes, it's obvious why, but imagine this is a very large program and you don't know why the output is what it is).

With that information to hand we can look for the doubleit method and add set some breakpoints so that we can explore the program as it's running to understand the state:

#!/usr/bin/env python
# main.py
def doubleit(val):
    import ipdb       # < added this line
    ipdb.set_trace()  # < added this line
    return val * 3

if __name__ == "__main__":
    print("doubleit 2: %d", doubleit(2))
    print("doubleit 4: %d", doubleit(4))
    print("doubleit 8: %d", doubleit(8))

We can continue to add ipdb.set_trace() points throughout our code. Generally speaking when I am running this for the first time I'll tend to just drop one or two points in the codebase that I know are going to be in the path, with the expectation that I'll manually step through the execution to learn how it flows. When we've added all of the breakpoints that we need we can instruct the program to run with python main.py:

$ python main.py
> /home/pete/go/src/github.com/peter-mcconnell/petermcconnell.com/assets/dummy/pydebug/main.py(6)doubleit()
      5     ipdb.set_trace()
----> 6     return val * 3
      7

ipdb>

Now we've ran our program with an attached debugger and it has paused execution at the breakpoint we set. We can run args to see which arguments where passed to the method:

> /home/pete/go/src/github.com/peter-mcconnell/petermcconnell.com/assets/dummy/pydebug/main.py(6)doubleit()
      5     ipdb.set_trace()
----> 6     return val * 3
      7

ipdb> args
val = 2

So in this point in the program we're in the doubleit method when it was invoked with a val value of 2. We can print this and other variables using p:

ipdb> p val
2

or just the variable name on its own:

ipdb> val
2

We can even call methods from this point:

ipdb> doubleit(6)
18

To walk over the execution we can press n to go to the next point of execution:

ipdb> doubleit(6)
18
ipdb> n
--Return--
6
> /home/pete/go/src/github.com/peter-mcconnell/petermcconnell.com/assets/dummy/pydebug/main.py(6)doubleit()
      5     ipdb.set_trace()
----> 6     return val * 3
      7

and view the backtrace with bt:

ipdb> bt
  /home/pete/go/src/github.com/peter-mcconnell/petermcconnell.com/assets/dummy/pydebug/main.py(9)<module>()
      8 if __name__ == "__main__":
----> 9     print("doubleit 2: %d", doubleit(2))
     10     print("doubleit 4: %d", doubleit(4))

6
> /home/pete/go/src/github.com/peter-mcconnell/petermcconnell.com/assets/dummy/pydebug/main.py(6)doubleit()
      5     ipdb.set_trace()
----> 6     return val * 3
      7

To view the code around the current point of execution just press l:

ipdb> l
      1 #!/usr/bin/env python
      2 # main.py
      3 def doubleit(val):
      4     import ipdb
      5     ipdb.set_trace()
----> 6     return val * 3
      7
      8 if __name__ == "__main__":
      9     print("doubleit 2: %d", doubleit(2))
     10     print("doubleit 4: %d", doubleit(4))
     11     print("doubleit 8: %d", doubleit(8))

Which of course shows our very hard to find logic error, * 3 instead of * 2.

Note: you can also set breakpoints in the stdlib functions (paths will vary depending on your setup):

ipdb> b /home/pete/.local/lib/python3.10/site-packages/requests/api.py:14
Breakpoint 1 at /home/pete/.local/lib/python3.10/site-packages/requests/api.py:14

debug flow

Using the commands above I can begin my cyclic process of narrowing in on the fix:

repro -> explore -> understand -> tweak -> repeat

More often than not this means I only need to understand a very small part of the application and can ignore code that isn't relevant to the immediate issue.

At a more detailed level this process looks like:

(repro) write a test that triggers the bug in as simple terms as I can express
(explore) set breakpoints
(explore) run pytest with the -s flag so that I can interact with ipdb
(explore) use args to check the arguments for the method that I'm in
(explore) print surrounding variable values
(explore) ensure the state of the program makes sense for my current breakpoint. If not, I need an earlier breakpoint. If so, continue with n
(explore) repeat these steps until I've reached the point that the program is in a seemingly erroneous state
- (understand) it's at this stage I'll take time to properly read the surrounding code and experiment with variable values to see if I can get the program to act in the expected manner
- (understand) depending on the category of bug I'll look for algorithmic complexity issues, stack overflow issues, parameter edgecases, logging quality, randomness factors etc. This is when the editor setup shines. see neovim section
- (tweak) I'll make minor adjustments to the code which I believe will nudge the program into the right place

Once I'm happy that my small tweaks are having the desired effect I'll perform some tidy ups and look for opportunities to harden the code with type checking / improved logging / more tests.

neovim

This section describes my neovim configuration for Python debugging at a high level. In short my debugging / code exploration flow boils down to:

telescope https://github.com/nvim-telescope/telescope.nvim
- allows me to ctrl + f scan directories for files
- allows me to set up keybindings for scanning any common directories
coc https://github.com/neoclide/coc.nvim
- code complete in all of the languages I need
- function descriptions
gd - default vim keybinding for go-to-definition. Jumps me into a function that I'm wanting to understand
ctrl + o / ctrl + i - default vim keybindings for go to last / next jump point. Really useful as I'm scanning code - I can keep jumping through definitions with gd then ctrl + o my way back / ctrl + i my way back down as I'm trying to build an understanding

You can see my full Neovim config here: https://github.com/peter-mcconnell/.dotfiles/blob/master/config/nvim/init.vim

summary

The example above is extremely trivial - where ipdb and it's ilk shine are on complex usecases where you may not even know what methods are between the input and output, such as debugging the stdlib. Just this week I used ipdb to identify why a codebase long forgotten was throwing an obscure error for a given dataset. By using ipdb I reproduced the scenario and just before the point at which I knew it would error created a break point that allowed me to inspect program state and better understand the conditions leading to the error, resulting in a quick patch.

why not pdb?

Bells and whistles; I like that ipdb has better color support and tab completion. You could absolutely get the same results with pdb.

Perf engineering with Python 3.12

Peter McConnell — Sat, 31 Dec 2022 10:11:50 +0000

overview

3.12 brings perf profiling! In this article we take a look at how the new perf profiling support helps reduce our dummy Python script from 36 seconds to 0.8 seconds. We'll introduce the Linux tool perf and also FlameGraph.pl, look at some disassembly and go bug hunting. You can view the code for this article here: https://github.com/peter-mcconnell/petermcconnell.com/tree/main/assets/dummy/perf_py_proj

Take a second to go check out https://docs.python.org/3.12/howto/perf_profiling.html and indeed the changelog at https://www.python.org/downloads/release/python-3120a3/. The important part (for this post) from these links is:

"""
The Linux perf profiler is a very powerful tool that allows you to profile and obtain information about the performance of your application. perf also has a very vibrant ecosystem of tools that aid with the analysis of the data that it produces.

The main problem with using the perf profiler with Python applications is that perf only allows to get information about native symbols, this is, the names of the functions and procedures written in C. This means that the names and file names of the Python functions in your code will not appear in the output of the perf.

Since Python 3.12, the interpreter can run in a special mode that allows Python functions to appear in the output of the perf profiler. When this mode is enabled, the interpreter will interpose a small piece of code compiled on the fly before the execution of every Python function and it will teach perf the relationship between this piece of code and the associated Python function using perf map files.
"""

writing a "bad" program

I'm excited to try this, so lets get going. Firstly lets create a python script for us to profile. I'm doing this before installing Python 3.12 as I want to create a FlameGraph of how this process looks in 3.10 verses 3.12. Here we have a script that attempts to perform lookups against a large list:

import time


def run_dummy(numbers):
    for findme in range(100000):
        if findme in numbers:
            print("found", findme)
        else:
            print("missed", findme)


if __name__ == "__main__":
    # create a large sized input to show off inefficiency
    numbers = [i for i in range(20000000)]

    start_time = time.time()  # get the current time [start]
    run_dummy(numbers)  # run our inefficient method
    end_time = time.time()  # get the current time [end]

    duration = end_time - start_time  # Calculate the duration
    print(f"Duration: {duration} seconds")  # Print the duration

Running this I get the following result:

python3.10 assets/dummy/perf_py_proj/before.py
...
found 99992
found 99993
found 99994
found 99995
found 99996
found 99997
found 99998
found 99999
Duration: 36.06884431838989 seconds

36 seconds is bad enough for us to pick up a reasonable amount of samples.

flamegraphs!

Now we can create our FlameGraph:

# record profile to "perf.data" file (default output)
perf record -F 99 -g -- python3.10 assets/dummy/perf_py_proj/before.py
# read perf.data (created above) and display trace output
perf script > out.perf
# fold stack samples into single lines
# here I reference ~/FlameGraph/ - you can get this from https://github.com/brendangregg/FlameGraph
~/FlameGraph/stackcollapse-perf.pl out.perf > out.folded
# generate flamegraph
~/FlameGraph/flamegraph.pl out.folded > ./assets/perf_example_python3.10.svg

This gives us a nice SVG that visualises the traces:

This isn't useful ... I can see most of the time was spent in "new_keys_object.lto_priv.0" but that is meaningless in the context of the code.

time for Python 3.12...

First I need to install it - the steps for this vary depending on OS - follow the build instructions here for your environment: https://github.com/python/cpython/tree/v3.12.0a3#build-instructions

# for me on ubuntu:22.04
# ensure I have python3-dbg installed
sudo apt-get install python3-dbg

# build python
export CFLAGS="-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer"
./configure --enable-optimizations
make
make test
sudo make install
unset CFLAGS

# after this I reset my systems python3 symlink to 3.10 as 3.12 isn't yet stable
# for testing python3.12 I'll call "python3.12" instead of "python3"
ln -sf /usr/local/bin/python3.10 /usr/local/bin/python3

With that installed I first need to enable perf support. This is detailed in https://docs.python.org/3.12/howto/perf_profiling.html and there are three options: 1) an environment variable, 2) an -X option or 3) dynamically using sys. I'll go for the environment variable approach as I don't mind everything being profiled for a small script:

export PYTHONPERFSUPPORT=1

Now we simply repeat the process above using the python3.12 binary instead:

# record profile to "perf.data" file (default output)
perf record -F 99 -g -- python3.12 assets/dummy/perf_py_proj/before.py
# read perf.data (created above) and display trace output
perf script > out.perf
# fold stack samples into single lines
# here I reference ~/FlameGraph/ - you can get this from https://github.com/brendangregg/FlameGraph
~/FlameGraph/stackcollapse-perf.pl out.perf > out.folded
# generate flamegraph
~/FlameGraph/flamegraph.pl out.folded > ./assets/perf_example_python3.12.before.svg

First we'll take a peek at the report with perf report -g -i perf.data

Awesome! We can see our Python function names and script names!

Now we can take a look at the updated SVG that visualises the traces with Python 3.12:

This is already looking much more useful. We see the majority of the time is being spent doing comparisons and in the list_contains method. We can also see the specific file before.py and method run_dummy that is calling it.

investigation time / the fix

Now that we know where in our code the problem is, we can take a look at the source code in CPython to see why the list_contains method would be so slow: https://github.com/python/cpython/blob/199507b81a302ea19f93593965b1e5088195a6c5/Objects/listobject.c#L440

note: you may not always have access to the source code - in circumstances such as this you can view the disassembly in perf report directly to get some idea of what's going on. I'll add a quick section at the end showing how this looks

// I found this by going to https://github.com/python/cpython/ and searching for "list_contains"

static int
list_contains(PyListObject *a, PyObject *el)
{
    PyObject *item;
    Py_ssize_t i;
    int cmp;

    for (i = 0, cmp = 0 ; cmp == 0 && i < Py_SIZE(a); ++i) {
        item = PyList_GET_ITEM(a, i);
        Py_INCREF(item);
        cmp = PyObject_RichCompareBool(item, el, Py_EQ);
        Py_DECREF(item);
    }
    return cmp;
}

Nasty... looking at this code I can see that each time it is invoked it iterates over the array and performs a comparison against each item. That's far from ideal for our usecase, so lets go back to the Python code we wrote. Our Flamegraph shows us that the problem is in our run_dummy method:

def run_dummy(numbers):
    for findme in range(100000):
        if findme in numbers:  #  <- this is what triggers list_contains
            print("found", findme)
        else:
            print("missed", findme)

We can't really change that line as it is doing what we want it to do - identifying if an integer is in numbers. Perhaps we can change the numbers data type to one better suited to lookups. In our existing code we have:

    numbers = [i for i in range(20000000)]

    start_time = time.time()  # get the current time [start]
    run_dummy(numbers)  # run our inefficient method

Here we used a LIST data type for our "numbers", which under the hood (in CPython) is implemented as dynamically-sized arrays and as such are nowhere near as efficient (O(N)) as the likes of a Hashtable for looking up an item (which is O(1)). A SET on the other hand (another Python data type) is implemented as a Hashtable and would give us the fast lookup we're looking for. Lets change the data type in our Python code and see what the impact is:

    # we'll just change this line, casting numbers to a set before running run_dummy
    run_dummy(set(numbers))  # passing a set() for fast lookups

Now we can repeat the steps as above to generate our new flamegraph:

# record profile to "perf.data" file (default output)
perf record -F 99 -g -- python3.12 assets/dummy/perf_py_proj/after.py
...
found 99998
found 99999
Duration: 0.8350753784179688 seconds
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.039 MB perf.data (134 samples) ]

Already we can see that things have massively improved. Where previously this was taking 36 seconds to run it is now taking 0.8 seconds! Lets continue creating our flamegraph for the new improved code:

# read perf.data (created above) and display trace output
perf script > out.perf
# fold stack samples into single lines
# here I reference ~/FlameGraph/ - you can get this from https://github.com/brendangregg/FlameGraph
~/FlameGraph/stackcollapse-perf.pl out.perf > out.folded
# generate flamegraph
~/FlameGraph/flamegraph.pl out.folded > ./assets/perf_example_python3.12.after.svg

This is a much healthier looking Flamegraph and our application is now much faster as a result. The perf profiling support in Python 3.12 brings a tremendously useful tool to software engineers that want to deliver fast programs and I'm excited to see the impact this will have on the language.

bonus round: what to do when you can't access the source code?

Sometimes you don't have access to the underlying code which can make trying to understand what's going on much more difficult. Thankfully perf report allows us to view the dissassembled code which can help paint a picture of what the machine is actually doing. This is a reasonable first place to look - I tend to prefer the source code if I can get hold of it as it allows me to git blame / view the associated commits and PRs. To view this you can do the following:

Open the perf report and select the line we're interested in:

# this assumes we have already ran 'perf record' to generate perf.data ...
perf report -g -i perf.data

Press enter and choose the annotate option:

Behold! Here we can see both the C code and the machine instructions. Super useful! You can compare the screenshot below against the code snippet we're interested in: https://github.com/python/cpython/blob/199507b81a302ea19f93593965b1e5088195a6c5/Objects/listobject.c#L440

chatGPT - building an automated database testing tool

Peter McConnell — Sat, 31 Dec 2022 10:08:37 +0000

Creating an automated database testing tool with ChatGPT

Last night I thought I'd try to get ChatGPT to make an automated database testing tool and the results were quite promising.

In conclusion, with guidance, it was able to build a project from scratch that ran a python script and postgres database. It generated some random schema and values for the randomly generated tables. It provided a Python script which would introspect the database and execute queries against it.

Did it all work out of the box? No. There are some bugs to fix in the python script it generated. However the effort to go in and fix those is not high and certainly the whole end-to-end process is cheaper, time-wise, compared to starting from scratch.

I found that the bugs it encountered were largely due to my lack of clarity or ordering of questions posed to it. It was quite capable of fixing its own mistakes / updating the existing code to match the new requirements when requested to do so.

The only real issue I encountered were general API errors that one would expect of something so popular in an early preview state.

I came away from this experiment viewing ChatGPT and whatever follows it as a really useful development aide for those who already know how to program. It helped me build a tool faster than I could have had I sat down to do it from scratch. I don't view it as a replacement for software engineers yet for two main reasons - firstly: for non-trivial applications I suspect the person feeding requirements into the system (or "prompt engineer") needs to have a reasonable idea of how to build software in the first place, so as to know how to form requests and to correct mistakes / close gaps. secondly: the code being generated isn't always sound - without an experienced engineer reviewing and taking ownership of whatever code is produced (ownership being important for maintainence reasons) then there's little guarantee that you will get what you are hoping for.

However; this is still very early days. Can the problems outlined be closed further? Absolutely. Will this sort of tooling be "bad" for software engineering as a whole, long-term? Perhaps. Personally I'm very excited to have this tool in my arsenal - already it has allowed me to scaffold prototype applications quickly. Would I use it for production code in a workplace? No more or less than I would snippets from stackoverflow or it's ilk. For now.

Github repository: https://github.com/peter-mcconnell/gpt_sql_test_generator

Screenshots:

A quick introduction to basic debugging tools for Linux systems

Peter McConnell — Sat, 31 Dec 2022 09:57:13 +0000

Firstly: let me qualify "basic". I'm using this term because the breadth of debugging tools on Linux is so large. "basic" does not mean that these tools are super simple to understand deeply or if you aren't already an expert in them that you are somehow "noob"; that's not the case at all.

Secondly: there is no inventing being done here. I'm just surfacing information in this article which you can already find in the man pages. I would strongly recommend you check the relevant man page for the tool you find of interest in this article.

Thirdly: for the curious, these tools generally get their information from existing counters and statistics in /proc/ and /sys/. In the man pages for these tools you can see a FILES section which details the data source for the given tool.

"crisis tools"

This section is taken from Brendan Gregg's book 'System Performance' (4.1.2). It covers not only where you can install some of the binaries mentioned in this article but also other useful tools for debugging. All credit to him for this list.

package	provides
procps	ps(1), vmstat(8), uptime(1), top(1)
util-linux	dmesg(1), lsblk(1), lscpu(1)
sysstat	iostat(1), mpstat(1), pidstat(1), sar(1)
iproute2	ip(8), ss(8), nstat(8), tc(8)
numactl	numastat(8)
linux-tools-common linux-tools-$(uname -r)	perf(1), turbostat(8)
bcc-tools (aka bpfcc-tools)	opensnoop(8), execsnoop(8), runqlat(8), runqlen(8), softirqs(8), hardirqs(8), ext4slower(8), ext4dist(8), biotop(8), biosnoop(8), biolatency(8), tcptop(8), tcplife(8), trace(8), argdist(8), funccount(8), stackcount(8), profile(8) and more ...
bpftrace	bpftrace, basic versions of opensnoop(8), iolatency(8), iosnoop(8), bitesize(8), funccount(8), kprobe(8)
perf-tools-unstable	Ftrace versions of opensnoop(8), execsnoop(8), iolatency(8), iosnoop(8), bitesize(8), funccount(8), kprobe(8)
trace-cmd	trace-cmd(1)
nicstat	nicstat(1)
ethtool	ethtool(8)
tiptop	tiptop(1)
msr-tools	rdmsr(8), wrmsr(8)
github.com/brendangregg/msr-cloud-tools	showboost(8), cpuhot(8), cputemp(8)
github.com/brendangregg/pmc-cloud-tools	pmcarch(8), cpucache(8), lcache(8), tlbstat(8), resstalls(8)

top

When: A reasonable first place to look.

This is a command-line utility in Linux that allows users to view the processes running on their system and monitor their resource usage in real-time. It can be used to identify performance bottlenecks, track the usage of system resources, and troubleshoot issues on a Linux system.

The output of top will include a list of processes running on the system, along with the following information for each process:

The process ID (PID)
The user owning the process
The CPU and memory usage of the process
The command that started the process

From the output you can identify some performance issues with your system. For example, if a single process is consuming a large amount of CPU or memory resources, it could be causing performance issues for your system. Similarly, if multiple processes are consuming high amounts of resources, it could indicate that your system is struggling to keep up with demand and may be a bottleneck.

One common problem that top can help identify is resource contention which occurs when multiple processes or applications are competing for the same system resources.

I suspect this tool is familiar to many of you already:

However what you may not realise is that you can pull much more data from it. Press 'f' when in top to access the fields management view and select other columns to display:

htop is a visually prettier alternative to top and that can make understanding the system at a glance easier.

mpstat

When: you think there's an issue with the CPU utilization.

mpstat is a command-line tool that is used to monitor the performance of a Linux system by displaying the CPU usage of each processor and the overall system.

Imagine that you are the system administrator of a web server that is experiencing slow performance and perhaps even downtime. You check the web server logs but nothing obvious is standing out so you begin to suspect there's something else wrong with the system. One general check we can do is on the CPU cores - this is where mpstat comes in.

mpstat -P ALL

This will display the CPU usage statistics for all CPU cores on the system. The output will look something like this:

Linux 4.9.0-8-amd64 (server.petermcconnell.com)  01/01/2023  _x86_64_    (2 CPU)

02:34:56 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:34:56 AM  all    20.00    0.00    5.00    0.00    0.00    0.00    0.00    0.00    0.00   75.00
02:34:56 AM    0    22.50    0.00    6.25    0.00    0.00    0.00    0.00    0.00    0.00   71.25
02:34:56 AM    1     7.50    0.00    3.75    0.00    0.00    0.00    0.00    0.00    0.00   88.75

The first line shows the Linux version, hostname, and CPU architecture. The second line displays the column headers, which show the various metrics that are being measured. The third line, labeled "all," shows the overall CPU usage for all CPU cores combined. The fourth and fifth lines show the CPU usage for each individual CPU core.

In this example, we can see that CPU 0 is running at a higher utilization than CPU 1. This could indicate that there is an issue with CPU 0 that is causing it to work harder than it should. To further investigate the issue we can opt to print these statistics every second by adding a "1" to the end of the command:

mpstat -P ALL 1

This will print statistics from each core every second, showing the current CPU usage. By watching the output over time, we can see if there are any patterns or spikes in the usage of a particular CPU core.

If we notice that one of the CPU cores is consistently running at a higher utilization than the others, we can try to identify the cause of the problem. One way to do this is by using the top command to see which processes are using the most CPU resources. It could be that the program itself is misconfigured and not properly distributing load over the available cores.

iostat

When: you think there's an issue with IO.

The iostat tool is a command-line utility in Linux that allows users to monitor input/output (I/O) statistics for devices, partitions, and network filesystems. It can be used to identify performance bottlenecks and track the usage of system resources by I/O operations.

To use iostat, you need to specify the interval at which you want to collect data, as well as the devices or partitions you want to monitor. For example, the following command will display I/O statistics for all devices every 2 seconds, 5 times:

$ iostat -p ALL 2 5
Linux 5.4.0-72-generic (myputer)    30/12/22    _x86_64_    (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.40   15.48    4.48    0.99    0.00   77.66

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
loop0             0.00         0.00         0.00         0.00          0          0          0
loop1             0.00         0.00         0.00         0.00          0          0          0
loop2             0.00         0.00         0.00         0.00          0          0          0
loop3             0.00         0.00         0.00         0.00          0          0          0
loop4             0.00         0.00         0.00         0.00          0          0          0
loop5             0.00         0.00         0.00         0.00          0          0          0
loop6             0.00         0.00         0.00         0.00          0          0          0
loop7             0.00         0.00         0.00         0.00          0          0          0
sda               4.83        79.27       106.76         0.00     858983    1156820          0

... repeat

You can also get more simple summaries - for example, CPU utilization report:

$ iostat -c
Linux 5.4.0-72-generic (urputer)    30/12/22    _x86_64_    (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.36   15.09    4.37    0.95    0.00   78.22

Or a device utilization report:

iostat -d -h -p ALL
Linux 5.4.0-72-generic (xerxes)     30/12/22    _x86_64_    (4 CPU)

      tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd Device
     0.00         0.0k         0.0k         0.0k       0.0k       0.0k       0.0k loop0
     0.00         0.0k         0.0k         0.0k       0.0k       0.0k       0.0k loop1
     0.00         0.0k         0.0k         0.0k       0.0k       0.0k       0.0k loop2
     0.00         0.0k         0.0k         0.0k       0.0k       0.0k       0.0k loop3
     0.00         0.0k         0.0k         0.0k       0.0k       0.0k       0.0k loop4
     0.00         0.0k         0.0k         0.0k       0.0k       0.0k       0.0k loop5
     0.00         0.0k         0.0k         0.0k       0.0k       0.0k       0.0k loop6
     0.00         0.0k         0.0k         0.0k       0.0k       0.0k       0.0k loop7
     4.82        73.5k       103.6k         0.0k     841.2M       1.2G       0.0k sda
     0.01         0.3k         0.0k         0.0k       3.1M       0.5k       0.0k sda1
     0.00         0.0k         0.0k         0.0k       2.0k       0.0k       0.0k sda2
     4.71        73.1k       103.6k         0.0k     836.0M       1.2G       0.0k sda5

We're seeing a lot of devices here which are showing to be inactive. We can easily hide those from view - rarely when debugging an issue do you want to see statistics for a device that isn't being used. To hide these, pass -z:

iostat -d -h -p ALL -z
Linux 5.4.0-72-generic (xerxes)     30/12/22    _x86_64_    (4 CPU)

      tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd Device
     4.84        72.1k       103.1k         0.0k     841.2M       1.2G       0.0k sda
     0.01         0.3k         0.0k         0.0k       3.1M       0.5k       0.0k sda1
     0.00         0.0k         0.0k         0.0k       2.0k       0.0k       0.0k sda2
     4.73        71.7k       103.1k         0.0k     836.0M       1.2G       0.0k sda5

By analyzing this data, you can identify performance issues with your devices or partitions. For example, if the number of read or write operations is consistently high, it could indicate that the device or partition is heavily used and may be a bottleneck. Similarly, if the average time taken for each I/O operation is consistently high, it could indicate that the device or partition is struggling to keep up with demand.

A common problem that iostat can help identify is disk saturation, which occurs when a disk is being used to its maximum capacity and can't keep up with the demand for I/O operations. This can lead to slow performance and potential data loss. In iostat output this problem may manifest as a consistently high percentage of time the device is busy with I/O operations and a high average time taken for each I/O operation.

sar

When: general purpose. you think there's an issue with CPU, Memory, Network or Block devices. Can record findings to files on disk for comparisson later (which is the main selling point of sar over, say vmstat)

The sar (System Activity Report) tool is a command-line utility in Linux that allows users to monitor various system performance metrics over time. It can be used to identify performance bottlenecks, track the usage of system resources, and troubleshoot issues on a Linux system.

To use sar, you need to specify the interval at which you want to collect data and the system performance metrics you want to monitor. For example, the following command will display CPU utilization statistics every 2 seconds:

sar 2

The output of sar will include various performance metrics, depending on the options you specify. Some of the metrics that sar can monitor include:

CPU utilization
Memory usage
Disk I/O activity
Network activity
Load average (a measure of the system's CPU and I/O utilization)

By analyzing this data, you can identify any performance issues with your system. For example, if the CPU utilization is consistently high, it could indicate that your system is struggling to keep up with demand and may be a bottleneck. Similarly, if the memory usage is consistently high, it could indicate that your system is running out of available memory.

sar stores its reports in /var/log/sysstat/ by default and will look for files in this directory when you ask it for some reports. The file it looks for changes depending on the INTERVAL value passed to it. This can result in an error if a report isn't available from yesterday:

# -B tells sar to create a report on paging statistics
sar -B
Cannot open /var/log/sysstat/sa30: No such file or directory
Please check if data collecting is enabled

You can specify 0 as the interval time which tells sar to use statistics for the time since the system was started:

sar 0 -B
Linux 5.4.0-72-generic (herputer)   30/12/22    _x86_64_    (4 CPU)

17:51:10     pgpgin/s pgpgout/s   fault/s  majflt/s  pgfree/s pgscank/s pgscand/s pgsteal/s    %vmeff
17:51:10        66.01     99.20   3074.99      0.38   1853.06      0.00      0.00      0.00      0.00

sar can also be used to report on I/O and transfer rate statistics:

sar 0 -b
Linux 5.4.0-72-generic (hisputer)   30/12/22    _x86_64_    (4 CPU)

17:53:49          tps      rtps      wtps      dtps   bread/s   bwrtn/s   bdscd/s
17:53:49         4.64      1.32      3.32      0.00    130.43    196.40      0.00

... and block devices:

sar 0 -d
Linux 5.4.0-72-generic (ourputer)   30/12/22    _x86_64_    (4 CPU)

17:55:12          DEV       tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util
17:55:12       dev8-0      4.63     64.81     97.66      0.00     35.13      0.05     11.69      2.11

... and more. sar is a very powerful tool - please check out the man page for more info.

vmstat

When: you think there's an issue with memory usage or I/O activity.

The vmstat (Virtual Memory Statistics) tool is a command-line utility in Linux that allows users to monitor various system performance metrics, including memory usage and I/O activity. It can be used to identify performance bottlenecks, track the usage of system resources, and troubleshoot issues on a Linux system.

To use vmstat you need to specify the interval at which you want to collect data. For example, the following command will display system performance statistics every 2 seconds:

vmstat 2

The output of vmstat will include the following information:

The number of processes in various states (e.g. running, waiting, blocked)
The amount of memory being used, including the amount of available memory
The number of page faults (when a process requests a page of memory that is not in physical memory and must be retrieved from disk)
The amount of I/O activity, including the number of read and write operations per second

free

When: you want to assess the state of memory on a machine.

free -w -h
              total        used        free      shared     buffers       cache   available
Mem:           15Gi       1.1Gi        12Gi       224Mi        95Mi       1.4Gi        13Gi
Swap:         2.0Gi          0B       2.0Gi

Here we see:

The total amount of available and used memory
The amount of used and available swap space (virtual memory on disk used to store data when the system's physical memory is full)
The number of used and available buffers (a type of cache used to store data temporarily) and cache (data stored in memory to speed up access to frequently used data)

lsblk

When: you assume there's an issue with the block devices and want to first check everything is in place.

The lsblk (List Block Devices) command is a command-line utility in Linux that allows users to view a list of block devices (e.g. hard drives, SSDs, USB drives) attached to their system. It can be used to view the available block devices, their mount points, and their partition layouts. You may want to use this tool to validate if block devices are available and mounted.

To use lsblk, you simply need to enter the command followed by any desired options. The following command will display a list of all block devices on the system:

$ lsblk -f
NAME   FSTYPE LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
├─sda1 vfat         29C1-1F85                               511M     0% /boot/efi
├─sda2
└─sda5 ext4         e3a6ec94-05ab-4e15-a310-e5797a2c55e9    419G     3% /

The output of lsblk will include the following information for each block device:

The device name (e.g. sda, sdb)
The device type (e.g. disk, partition)
The size of the device
The mount point (if the device is mounted)
The filesystem type (if the device is formatted with a filesystem)

df

When: you're aware of disk space issues and want to check the current availability.

The df (Disk Free) command is a command-line utility in Linux that allows users to view the amount of available and used disk space on their system. It can be used to identify if a disk is approaching or at capacity.

To use df, you simply need to enter the command followed by any desired options. The following command will display the total amount of available and used disk space on the system:

df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5       457G   15G  420G   4% /

The output of df will include the following information for each file system:

The file system name (e.g. /dev/sda1)
The total size of the file system
The amount of used and available space on the file system
The percentage of used space on the file system
The mount point (the location where the file system is mounted in the directory structure)

perf

When: you are aware of a processing issue and want to understand which process it's coming from, or you want to drill into the process to understand why it's not performing to the expected standard.

This tool allows users to monitor various performance metrics and events on their system. It can be used to identify performance bottlenecks, track the usage of system resources, and troubleshoot issues on a Linux system. Unlike the other tools in this article perf allows us to actually peek into the running program to see which parts of it are consuming cpu cycles. This is an incredibly valuable tool to have for those wanting to performance engineer on Linux and I have a practical example of this detailed in https://www.petermcconnell.com/posts/perf_eng_with_py12/.

To use perf, you need to specify the type of performance data you want to collect, as well as the specific events or metrics you want to monitor. To get general cpu profiling you can run:

perf top -a

Perf also allows you to gather performance statistics. For example, the following command will collect CPU performance data for the cpu-clock event:

perf stat -e cpu-clock
^C
 Performance counter stats for 'system wide':

         11,633.97 msec cpu-clock                 #    3.998 CPUs utilized

       2.909972618 seconds time elapsed

The scope of what you can do with perf is huge - too large for this article, but I couldn't leave it off the list. Perhaps a perf deep dive article will come here soon ...

uptime

When: you want to get a quick understanding of system load over the past 1, 5, 15 minutes or want to see how long the machine has been up for.

uptime
19:26:35 up  5:12,  1 user,  load average: 0.33, 0.37, 0.37

What is DevOps?

Peter McConnell — Sat, 31 Dec 2022 09:47:03 +0000

The term “devops” has been floating around since the late 2000s and frankly has always annoyed me. Not that I think the intent was bad but rather the adoption was so varied and confusing that simply saying the word out loud seemed to make things worse and lead to fear and confusion. I’d like to make “DevOps” the “Voldermort” of tech buzzwords - “He-Who-Must-Not-Be-Named”. This happened with Agile also which is even more confusing given it has a “manifesto” that’s all but a single paragraph, but I’ll leave that for another day.

The Wikipedia (https://en.wikipedia.org/wiki/DevOps) page for DevOps is fairly hand-wavey which somewhat highlights the issue. Like most documents talking to “organisational change” it’s a word-salad with no real actionable takeaways.

I’ve had several “DevOps” jobs and each of them were functionally and organisationally different. All projects were delivered but done so more often than not via silos. Wasn’t this the problem “DevOps” was meant to solve?

Speaking of … What is the problem to solve? Shipping code to production quickly and reliably. In ye olden times there were issues of siloed development teams and operations teams meaning the devs wouldn’t optimise for production workloads and the operators wouldn’t have a clue what they were shipping onto their servers. There’s a lot of domain expertise in both camps - asking people to become experts of both was unreasonable.

But that was also a different time … we didn’t have the abundance of tooling and services that we do today. Kubernetes didn’t exist. Docker didn’t exist. Cloud offerings were pretty light. With the tools available today, asking SWEs to take on more responsibility to account for their applications in production is a reasonable ask.

"so what should we do?"

Frankly I think we’re now in a world that looks more like the pre-devops days. SWE teams are no longer handing over a package of code and saying “hey, take my source code, install these dependencies and run this” but are now in control of their applications deployment manifest and can containerise their applications. The original problem to solve feels like much smaller a problem now.

Operators can run a platform such as Kubernetes which for the most part the other engineering teams can treat like a PAAS. “How will devs know how to configure their apps to run on the platform” - they’ll have to learn. Somehow. Guardrails should be put in place (e.g. policies) to stop people “doing bad things” but it shouldn’t go as far as “that devops engineer will write your cicd config” or “that devops engineer will write your kubernetes config”. Teams MUST own their config, and to own it they need to understand it.

So it shouldn’t be a team or a job title. There shouldn’t be “devops tools” or a “devops environment”. A problem can never be “a devops issue”. You might have a kubernetes team, an OPA/IAM/network policy team, a streaming services team etc. Just don’t have a “devops” team.

We shouldn’t give it a new name either. Stop giving everything a name just so you can sell books and tshirts. This just creates distance from the problem to solve - we all just want to ship code reliably so that our business meets its goals and so that we don’t spend overtime fixing bugs or doing manual work that could have been done by a computer.

Forem: Peter McConnell

Building an XDP eBPF Program with C and Golang: A Step-by-Step Guide

Introduction

Project Overview

Setting Up the Development Environment

Writing the XDP eBPF Program in C

Compiling and Loading the XDP eBPF Program

Compiling the XDP Program

Loading the XDP Program

Attaching the XDP Program

you can get by running `ip link'

Makefile

Writing the Golang Application

Writing the Golang Application Code

Building and Running the Project

Building the XDP eBPF Program

Building the Golang Application

Running the Go Project

Testing and Verifying the XDP eBPF Program

Test Environment Setup

Packet Drop Verification

Performance Analysis

Conclusion

Additional Resources

Docker Overlayfs: How filesystems work in Docker

LowerDir

UpperDir

MergedDir

Do it yourself

Docker networking: Network Namespaces, Docker and DNS

simple docker run

docker compose

a quick note on the 'links:' property

dockers internal DNS server

virtual network interfaces

exploring the namespace

docker bridge

so ... how does nginx_a speak to nginx_b? (recap)

network_mode

Advice for engineers wanting to 'make it'

The advice I wish I had been given

Golang debugging tips

debugging Golang - the context

what are the requirements?

example program

debug it

summary

why not gdb?

Python debugging tips

debugging Python - the context

what are the requirements?

refining scope

example application

using ipdb

debug flow

neovim

summary

why not pdb?

Perf engineering with Python 3.12

overview

writing a "bad" program

flamegraphs!

time for Python 3.12...

investigation time / the fix

bonus round: what to do when you can't access the source code?

recommended reading

chatGPT - building an automated database testing tool

Creating an automated database testing tool with ChatGPT

A quick introduction to basic debugging tools for Linux systems

"crisis tools"

top

mpstat

iostat

sar

vmstat

free

lsblk

df

perf

uptime