Forem: Mahesh Rayas

04 – eBPF Uprobes: Tracing gRPC Headers by Unpacking Go Function Internals

Mahesh Rayas — Sun, 03 Aug 2025 08:43:21 +0000

Introduction to Uprobes

In the previous post, we explored kprobes and demonstrated a simple eBPF example that instruments kernel-space HTTP servers to extract HTTP headers—without modifying the source code. In today's blog, we shift our focus to uprobes, which enable similar instrumentation for user-space applications, addressing use cases that kprobes cannot cover.

Much like how kprobes allow us to attach eBPF programs to kernel functions, uprobes let us hook into functions in user-space binaries. This provides powerful observability capabilities for modern applications written in languages like Go. In particular, we'll explore how to instrument Go binaries by attaching uprobes to standard library functions (e.g., net/http, google.golang.org/grpc) and user-defined logic.

To hook into a Go function using uprobes, we must understand how Go lays out its function arguments in memory. This is especially important for Go 1.17+, which introduced a register-based calling convention that changed how arguments are passed between functions. We'll learn how to inspect process memory and extract function arguments by understanding Go's calling conventions, memory addressing, and pointer dereferencing techniques.
Go binaries are increasingly common in cloud-native environments. Instrumenting them at runtime using eBPF uprobes enables deep observability without modifying or recompiling source code—making it ideal for production debugging and monitoring.

In this post, we will:

Understand how Go passes function arguments (e.g., int, string, struct) in Go 1.17 and beyond
Use tools like nm and objdump to locate function symbols in Go binaries.
Attach uprobes to Go functions and extract runtime arguments.
Use delve to debug Go programs and inspect memory layout for function hooking.
Apply these techniques to parse gRPC headers from Go-based gRPC services using eBPF uprobes.

By the end of this post, you'll have a solid understanding of how to leverage uprobes for deep introspection of Go applications, opening up new possibilities for observability and debugging in production environments.

Architecture Overview

Prerequisites and Setup

Before diving into the technical details, let's establish our testing environment and the tools we'll need throughout this tutorial.

System Requirements

Linux x86_64 (kernel 4.1+ for uprobe support)
Root privileges or appropriate capabilities for eBPF operations

Development Tools

Rust: Our userspace framework for eBPF program development and attachment.

Go 1.17+: Target language for our instrumentation examples
delve: Go debugger for inspecting memory layouts and validating our approach

binutils: For nm, objdump, and other binary analysis tools.

Installation and Setup

Install delve if you haven't already:

go install github.com/go-delve/delve/cmd/dlv@latest

Enable ptrace for delve to attach to running processes:

echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

Sample Go Application

We'll work with two Go applications throughout this tutorial:

sample_go: A simple Go program designed to help us analyze Go's memory layout and calling conventions for different argument types (int, string, struct).

sample_grpc: A Go gRPC server that we'll instrument to extract gRPC headers using our Rust+eBPF tracer.

Rust + eBPF

trace-grpc-headers. Go through the readme to get it up and running

Go Calling Convention: The Register-Based Revolution (Go 1.17+)

Note: This blog focuses specifically on the amd64 architecture.

Starting from Go 1.17, the Go compiler adopted a register-based calling convention, moving away from the traditional stack-only approach used in earlier versions. This change significantly impacts how we extract function arguments in eBPF uprobes.

The Register Allocation Strategy

Go 1.17+ uses a predefined set of 9 registers for passing function arguments on x86_64:

RAX, RBX, RCX, RDI, RSI, R8, R9, R10, R11

Our Instrumentation Strategy

To successfully extract function arguments in our eBPF uprobe, we need to follow a systematic approach:

Locate the target function: Get the offset address of the Go function we want to instrument
Analyze argument layout: Determine whether arguments are passed in registers, on the stack, or a combination of both
Map arguments to locations: Identify which specific registers or stack offsets contain our target data

Once we have this information, we can accurately probe the respective Go function when it's invoked and extract the arguments we're interested in.

Essential Reading

This approach is based on Go's official ABI specification. I highly recommend reading the Go ABI Internal Documentation to understand the detailed rules governing how Go passes arguments to functions.
With this foundation in place, let's move on to the practical steps of locating functions and determining memory offsets.

Locating Functions in Go Binaries

Before we can attach uprobes, we need to identify the specific functions we want to instrument within the Go binary. Go provides several approaches for function discovery.

Symbol Tables and Debugging Information

Go binaries contain symbol tables that map function names to their memory addresses. However, the availability and format of these symbols depend on how the binary was compiled.

In our case,lets compile it with

# Preserves all symbols and disables optimizations
go build -gcflags="all=-N -l" -o sample

Finding Target Functions

Use these tools to explore available functions in your Go binary:

# List all functions with addresses:
nm -n myapp | grep " T "

# Search for specific function:
nm -n myapp | grep "main.hello"

you can also use objdump or go tool to extract the info

Determining Memory Offsets

Once we've located our target function in the symbol table, we need to calculate the correct offset address for uprobe attachment. This offset represents the function's location within the process's virtual address space.

Calculating Function Offsets

The offset address is part of the virtual address space memory layout. We attach our uprobe to this calculated offset address.

Step 1: Get the function's virtual address

nm -n server | grep MyHandler

Output:

0000000000498f60 T main.MyHandler

Step 2: Find the base load address

readelf -l server | grep "LOAD"

Output:

LOAD 0x0000000000000000 0x0000000000400000 0x0000000000400000

Step 3: Calculate the offset

offset = function_address - base_load_address
offset = 0x498f60 - 0x400000 = 0x98f60

Programmatic Offset Calculation

In our Rust userspace program, this offset calculation is handled by helper functions:

// Calculate offset for the target function
let offset = calculate_function_offset(&binary_path, "main.MyHandler")?;

// Attach uprobe to the calculated offset
go_sk
    .progs
    .handle_hello_int
    .attach_uprobe(false, pid, &binary_path, offset as usize)?;

Deep Dive: Go Argument Passing Patterns

Let's examine how Go passes different data types as function arguments, following the official ABI specification.

Integers

Simple types occupy one register each:

func hello_int(x int) int {
    fmt.Printf("hello_int %d\n", x)
    return x
}

eBPF extraction:

 int x = (int)ctx->ax;

Important: If the function has a receiver, the receiver takes RAX and arguments shift to subsequent registers.

Strings: Pointer + Length Structure

Go strings are internally represented as:

type string struct {
    ptr *byte //pointer to data
    len int // Length
}

For a function with multiple arguments:

func hello_int_string(x int, y string) int {
    fmt.Printf("hello_int_string %d %s\n", x, y)
    return x
}

eBPF extraction:

    int x = (int)ctx->ax;            // First argument (int) -> RAX
    void *str_ptr = (void *)ctx->bx; // String data pointer -> RBX
    long str_len_raw = ctx->cx;      // String length (as long) -> RCX

Structs: Value vs Pointer Passing

Struct by value: Individual fields are assigned to registers sequentially

"If T is a struct type, recursively register-assign each field of V."

Struct by pointer: Only the pointer occupies one register

"If T is a pointer type, assign V to register I and increment I."

The choice dramatically affects extraction strategy. See our sample code for both scenarios.

Slices: Three-Field Structure

Go slices contain three components:

type slice struct {
    ptr *elementType  // Data pointer
    len int          // Element count  
    cap int          // Capacity
}

Each field occupies consecutive registers when passed by value.

Floating Point Numbers: The XMM Challenge

Floats use separate XMM registers (X0-X14), not general-purpose registers. This creates limitations:

Extractable: Floats in structs passed by pointer (dereference from memory)
Not directly extractable: Individual float arguments or floats in value-passed structs

eBPF currently cannot access XMM registers directly. See this StackOverflow discussionfor details.

Workaround: Use pointer-based struct passing for float extraction.

Practical Application: gRPC Header Extraction

Now comes the exciting part—applying everything we've learned to solve a real-world problem: extracting gRPC headers from a running Go service without touching the source code.

The Challenge: gRPC vs HTTP/1.1

Unlike our previous blog where we parsed HTTP/1.1 headers (which are plain text), gRPC presents a tougher challenge. gRPC headers are compressed using HPACK, making direct parsing from network packets nearly impossible.
But here's where our uprobe knowledge becomes powerful—we can intercept the headers after they've been decompressed by the Go runtime.

Finding the Right Function to Hook

The key insight came from studying how observability tools like Pixie solve this problem. However, their examples target older Go versions and don't work with Go 1.17+'s register-based calling convention.
After digging through the gRPC-Go codebase, I identified the perfect target:

func (t *http2Server) operateHeaders(
    ctx context.Context, 
    frame *http2.MetaHeadersFrame, 
    handle func(*ServerStream)
) error

This function in the grpc-go package processes decompressed header fields just before they're sent to the client—exactly what we need!

Detective Work with Delve

But where does the frame argument live—register or stack? Time for some debugging:

dlv attach 366200
(dlv) break google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders
(dlv) continue

# When breakpoint hits...
(dlv) args
frame = ("*golang.org/x/net/http2.MetaHeadersFrame")(0xc00020e0c0)

(dlv) regs
Rdi = 0x000000c00020e0c0  # Bingo! Frame pointer is in RDI

Perfect match! The frame pointer lives in the RDI register, exactly as our calling convention knowledge predicted.

The Structs We Need to Parse

Armed with the register location, we now need to parse two key structures:

MetaHeadersFrame : Contains the frame metadata
Fields : Individual header name-value pairs

The eBPF Implementation

Using our memory layout knowledge from the previous sections, we can now traverse these structures in eBPF:

void *frame_ptr = (void *)ctx->di;

Try it yourself

Full Working code: https://github.com/maheshrayas/blogs/tree/main/ebpf/04-tracing-grpc-headers/code

The repo includes:

Sample gRPC server with various header types
Complete eBPF program with detailed comments
Rust userspace code for uprobe attachment
Step-by-step instructions to reproduce the results : readme.md

What You'll See
Currently, the basic implementation outputs header data to the kernel trace pipe:

Currently it prints all the headers key, value in /sys/kernel/tracing/trace_pipe, we can also pass these headers to userspace using Maps(RingBuffer or PerfEventArray) just like we did in our pervious example.

Debugging, Troubleshooting and Learnings

Working with Go uprobes can be challenging, especially when dealing with memory layout parsing and argument location detection. Here are the key issues you'll likely encounter and approaches to solve them.

Memory Layout Parsing: The Silent Killer

The most frustrating bugs come from incorrect memory address parsing. A single byte offset error can lead to:

Garbage data extraction
Kernel panics in extreme cases
Silent failures with no obvious symptoms

Best practices:

Always validate extracted pointers before dereferencing
Use boundary checks in your eBPF programs
Test with simple data types first, then move to complex structs. Thats how I got started doing a simple types and I have attached the sample for you to familar.

Register vs Stack: The Ongoing Mystery

Determining whether arguments are passed in registers or on the stack remains tricky. While I used delve for this analysis and my understanding on ABI specs, I'm not entirely confident this is the most reliable approach.

Potential alternative approaches (still researching):

DWARF debug information: Parse function signatures and calling conventions from debug symbols
Static analysis using go tool compile -S
Assembly inspection with objdump -d

Call for input: If you have experience with static analysis techniques for determining Go argument passing, I'd love to hear your insights! Feel free to reach out or contribute to the repository.

Conclusion

Throughout this post, we've journeyed from understanding Go's register-based calling convention to successfully extracting gRPC headers from running applications using eBPF uprobes. We've learned how to locate functions in Go binaries, calculate memory offsets, parse different argument types, and apply these techniques to solve real observability challenges.
The techniques we've covered form the foundation for powerful runtime introspection capabilities. By combining uprobe instrumentation with efficient userspace communication through ring buffers, we can build sophisticated observability tools that operate without any application code changes.

What's Coming Next

I'm currently building a comprehensive observability tool that combines both gRPC and HTTP/1.1 header parsing capabilities, designed specifically for Kubernetes environments. This tool will provide complete HTTP path visibility and distributed tracing across your entire service mesh—all through eBPF instrumentation.
But that's just the beginning. My next blog post will be even more exciting, where I'll share a complete open-source project ready for production use in Kubernetes environments. We'll explore how to deploy eBPF-based observability at scale, handle multi-node clusters, and use the tool to increase the kubernetes security.

Stay tuned for the next installment where theory meets production reality. The best is yet to come!

# 03- Lets understand Kprobes & Kretprobes

Mahesh Rayas — Sun, 20 Jul 2025 09:16:07 +0000

When it comes to tracing the Linux kernel, kprobes and kretprobes are some of the oldest and most flexible tools in the eBPF ecosystem. In fact, kprobes were introduced long before eBPF, first appearing in Linux kernel 2.6.9 (2004), enabling dynamic tracing of almost any kernel function without requiring kernel recompilation.

eBPF came later (around kernel 3.15+) and builds upon kprobes by allowing users to attach safe, sandboxed, and highly programmable code to kernel functions, including kprobes and kretprobes. This combination provides a powerful foundation for modern observability and performance tooling.

While kprobes and kretprobes typically have higher overhead compared to other eBPF program types like XDP or tracepoints, their versatility makes them invaluable. They can attach to virtually any kernel function, enabling deep visibility into system behavior — which is why they are widely adopted in many popular eBPF-based observability tools.

Note: Kprobe cannot probe the blacklist function

Inner workings

With kprobes, you can attach custom eBPF programs to nearly any kernel function. When that function is called, the attached eBPF program is triggered at its entry point, then the original function continues as usual.
If you want to know the details on how kprobe works, I would recommend reading https://www.kernel.org/doc/html/latest/trace/kprobes.html#how-does-a-kprobe-work

In contrast, kretprobes are triggered when the function returns. This allows you to capture return values or output arguments — making them ideal for debugging, performance monitoring, or return-value-based logic.

In this post, we’ll demonstrate how to trace HTTP/1.1 headers using kprobes and extract them for observability. Since SSL/TLS encrypts the payload, this technique only applies to unencrypted HTTP traffic.

In a future post, we'll explore how to trace HTTP/2 (gRPC) headers using uprobes, which allow visibility into user-space libraries like golang.org/x/net/http2.

More details on Probing eBPF:

Before we dive into examples

Kprobes provide unmatched flexibility to dynamically instrument almost any kernel function. However, attaching probes indiscriminately can introduce performance overhead or stability risks. It’s best practice to:

Target well-known, stable kernel functions related to syscalls, networking, or drivers.
Confirm symbol presence and type in /proc/kallsyms (look for uppercase T symbols).
Test probes carefully on a non-production system.

Keep in mind kernel version differences and module load/unload events may affect probe reliability.

Typical kprobe targets include kernel functions related to syscalls, networking stacks, drivers, and filesystem operations, enabling tracing of specific kernel events.

Performance impact: Kprobes add instrumentation overhead because they inject code on function entry or return. While usually small, probing high-frequency functions can cause noticeable CPU overhead. Hence needs to be performance tested before its production ready.

/proc/kallsyms is a special file in Linux that lists all currently loaded kernel symbols, including function names, memory addresses, and variable symbols. This file is particularly useful when working with kprobes, as it helps identify kernel functions that can be dynamically instrumented.

You can find all the symbols using

sudo cat /proc/kallsyms

However, it’s important to note:

Function availability can differ based on kernel version, architecture (e.g., x86 vs ARM), and build configurations.
Some symbols may be inlined, optimized away, or renamed in newer kernels.

So, when writing portable eBPF programs using kprobes, always validate your target symbol on the specific kernel using /proc/kallsyms.

for example


> sudo cat /proc/kallsyms | grep "sys_read"


ffffffffb52edf30 T __pfx___x64_sys_read
ffffffffb52edf40 T __x64_sys_read
ffffffffb52edf70 T __pfx___ia32_sys_read
ffffffffb52edf80 T __ia32_sys_read

Symbols marked with T indicate global (exported) kernel text symbols, i.e., functions you can reliably attach kprobes to.

Let's Dive into the Example

Working example: https://github.com/maheshrayas/blogs/tree/main/ebpf/03-tracing-http-headers/code/trace-http1-headers

This example traces HTTP/1.1 headers of a server using eBPF. Here's the setup:

The HTTP server exposes a basic REST API and is not SSL encrypted. This is important because we cannot trace encrypted data in the kernel—encryption and decryption happen in user-space libraries like OpenSSL.
A Rust user-space application is run, and the PID of the HTTP server is supplied as input.
The user-space program loads and attaches kprobes to relevant kernel functions (explained in detail below).
From user-space, we update an eBPF map to track the PID of http server.

⚠️ Note: In containerized environments, the PID inside a container differs from what the host kernel sees. So, we must trace using the host PID and map it the container/pod process id. We shall see in future post how exactly we can map the PID on host to container/pod process in kubernetes echo system.

let key = pid.to_ne_bytes();
let value = 1u8.to_ne_bytes();
if let Err(_) = http_sk.maps.tracked_pids.update(&key, &value, MapFlags::ANY){
    warn!("failed to update maps for pid {}\n",pid);
}

eBPF Program Logic to extract http headers

We attach three probes to two different kernel functions.

1.kretprobe/__sys_accept4 -> handle_accept4_ret

The first kretprobe is attached to the kernel function __sys_accept4, which is invoked during the accept4() syscall.
This is where a server accepts a new incoming connection.

From this kretprobe, we extract the file descriptor (fd) returned by accept4.
We check if the calling process's PID exists in the tracked_pids eBPF map.
If it matches, we store the new fd in another eBPF map called active_app_sockets.

SEC("kretprobe/__sys_accept4")
int handle_accept4_ret(struct pt_regs *ctx)
{
    int new_fd = PT_REGS_RC(ctx);
    // Logic to filter and store fd
}

2.kprobe/ksys_read -> kprobe_ksys_read_entry

This kprobe is attached to the ksys_read() kernel function, which is called when a process reads from an fd.

We check if the fd is already in the active_app_sockets map.
If yes, we store the buf pointer in the read_args_map.

SEC("kprobe/ksys_read")
int kprobe_ksys_read_entry(struct pt_regs *ctx)
{
    // Check if fd is tracked
    // Save the 'buf' argument into a map for use in the return probe
}

What is buf?

The buf argument is a pointer to a user-space memory buffer.
The kernel will copy data into this buffer as a result of the read syscall.
Since eBPF programs run in kernel space, accessing this pointer directly is not safe or valid unless we carefully copy from user memory using helpers like bpf_probe_read_user.

3.kretprobe/ksys_read -> kretprobe_ksys_read_exit

Once the syscall ksys_read() is invoked, the actual data is not yet read into user space — it happens later, by the time the function returns. Therefore, to access the actual read data, we attach a kretprobe to ksys_read. This allows us to retrieve the number of bytes that were read and use that length to copy the data from user space buffer.

SEC("kretprobe/ksys_read")
int kretprobe_ksys_read_exit(struct pt_regs *ctx)
{
    // Get the Pid 
    // Read the actual data using bpf_probe_read_user
    // Check if its HTTP data
    // and send it the data to user space if the pid is that of http server.
}

4.Parsing Perf Event Data in Userspace

The user-space process continuously polls the perf event array for incoming data. Once data is received, it parses the raw binary payload emitted from the eBPF program.

One critical aspect to consider during this parsing step is struct alignment and memory layout compatibility between kernel space (C) and user space (Rust). When decoding the raw binary data from the perf buffer into a Rust struct, the layout of the Rust struct must exactly match the layout of the C struct used in the eBPF program.

To ensure compatibility:

Use the same field order and types in your Rust struct.
Add the #[repr(C)] attribute to enforce C-like memory layout in Rust.

This ensures safe and correct parsing of the data, avoiding undefined behavior due to padding or misalignment between the two languages.

#[repr(C)]
#[derive(Clone, Copy)]
pub struct HttpEventData {
    pid: u32,
    fd: i32,
    data_len: u32,
    data: [u8; 256],
}

fn handle_event(_cpu: i32, data: &[u8]) {
    let event = unsafe { std::ptr::read_unaligned(data.as_ptr() as *const HttpEventData) };

    // Convert the raw data to a string
   //...
}

Wrapping Up

In this post, we explored:

How to use kprobes and kretprobes to trace syscalls like ksys_read & accept, paving the way to trace HTTP headers.
How to capture user-space data (like HTTP headers) inside the kernel using eBPF.
Safe parsing of raw perf event data in Rust, using #[repr(C)] and field alignment.

This lays the groundwork for powerful observability tools that require no code changes in the application layer — everything is captured transparently from the kernel.

What's Next?

In the next part of this series, we'll explore how to parse gRPC headers from a Go HTTP/2 server, diving deeper into protocol parsing.
Later, we’ll bring everything together to build a Kubernetes-native observability tool that can trace HTTP headers — all without any modification or instrumentation to the applications themselves.

Stay tuned — it gets even more exciting from here!

# 02 - Understanding eBPF Core Building Blocks

Mahesh Rayas — Sun, 13 Jul 2025 08:32:19 +0000

In our previous post, we built a syscall tracer with Rust + eBPF. Today, we’ll unpack the core components that made it work — and the deeper mechanics powering real-world eBPF tools. Understanding these components will help you build more sophisticated observability, security and networking tools.

1. BPF Program Types & Hook Points

Program types define what the eBPF program can do, while hook points define where it runs in the kernel.

Key Program Types:

XDP (eXpress Data Path)

Runs at the earliest point in network packet processing
Perfect for DDoS protection, load balancing
Can DROP, PASS, REDIRECT, or TX packets

TC (Traffic Control)

Attaches to network ingress/egress points
More context than XDP, can modify packets
Used for advanced packet filtering and shaping

Kprobes/Kretprobes

Dynamic tracing of any kernel function
Kprobe = function entry, Kretprobe = function return

Uprobes/Uretprobes

Trace userspace applications (like Go, Python apps)
Can inspect function arguments, return values
Great for application performance monitoring

Tracepoints

Static kernel instrumentation points
More stable than kprobes across kernel versions
Predefined events like sys_enter_openat
You can view all available tracepoints in: /sys/kernel/tracing/events
our syscall tracer used sys_enter !

CGROUP

Control resource access per container/process group
Implement custom policies (network, file access)

Socket Programs

Filter/redirect packets at socket level
Implement custom load balancers, firewalls

Note: While I’ve covered the most commonly used program types here—those I’m most familiar with—there are several more (and evolving) types supported by the Linux kernel. For a complete and up-to-date list, refer to the official documentation: Supported eBPF Program Types

2. BPF Verifier: The Safety Guardian

The BPF verifier is what makes eBPF safe. It performs static analysis on the program before loading it into the kernel.

What the Verifier Checks:

Memory Safety

No out-of-bounds array access
No null pointer dereferences
Proper initialization of variables

Control Flow

No infinite loops (bounded loops only since Linux 5.3)
All code paths must exit
No unreachable code

Helper Function Usage

Only approved helper functions can be called
Correct argument types and counts

Stack Usage

Limited to 512 bytes of stack space
No stack overflow protection needed

// ❌ This will be REJECTED by verifier
for (int i = 0; ; i++) {  // Infinite loop
    // code
}

// ✅ This will be ACCEPTED
for (int i = 0; i < 100; i++) {  // Bounded loop
    // code  
}

Why This Matters: The verifier ensures the eBPF program cannot crash the kernel, making it safe to run in production.

3. BPF Maps: Data Storage & Communication

BPF Maps are the primary way to:

Store state between eBPF program invocations
Communicate between kernel and userspace
Share data between different eBPF programs

Essential Map Types:

Hash Maps - Key-value storage

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 1024);
    __type(key, u32);           // PID
    __type(value, u64);         // Syscall count
} syscall_counts SEC(".maps");

Arrays - Index-based access

struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __uint(max_entries, 256);   // One per syscall number
    __type(key, u32);
    __type(value, u64);
} syscall_stats SEC(".maps");

Perf Event Arrays - High-speed data streaming

struct {
    __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
    __uint(key_size, sizeof(u32));
    __uint(value_size, sizeof(u32));
} syscall_events SEC(".maps");

You can find the usage of this in the pervious blog post code

// Send data to userspace
bpf_perf_event_output(ctx, &events, BPF_F_CURRENT_CPU, &data, sizeof(data));

Ring Buffers - Modern alternative to perf events (Linux 5.8+)

Lower overhead, better memory efficiency
Built-in backpressure handling

Map Usage Patterns:

Counters: Track metrics (packets, syscalls, errors)
Caching: Store lookup results to avoid repeated work
State machines: Track connection states, user sessions
Configuration: Runtime configuration from userspace

4. Helper Functions: Kernel API

eBPF programs can't directly call kernel functions. Instead, they use helper functions - a curated set of safe kernel APIs.

Common Helper Categories:

Debugging & Logging


bpf_printk("PID %d called syscall %d\n", pid, syscall_nr);

Map Operations

// reading from the map
void *value = bpf_map_lookup_elem(&my_map, &key);
// updating/inserting into map
bpf_map_update_elem(&my_map, &key, &value, BPF_ANY);
// delete the Key/Value from map
bpf_map_delete_elem(&my_map, &key);

Context Information

// This is to get the Pid and Tid of a process
// Usage: We did use this helper function to read the pid from kernel in the previous blog
u64 pid_tgid = bpf_get_current_pid_tgid();
u32 pid = pid_tgid >> 32;

Network Operations

// Redirect packet to another interface
bpf_redirect(ifindex, 0);

// Modify packet data
bpf_skb_store_bytes(skb, offset, data, len, 0);

Time & Random

u64 timestamp = bpf_ktime_get_ns();
u32 random = bpf_get_prandom_u32();

Usage of helper function depends on the eBPF program types.

5. BTF & CO-RE: Write Once, Run Everywhere

BTF (BPF Type Format) and CO-RE (Compile Once, Run Everywhere) solve the kernel compatibility problem.

The Problem:

Kernel structures change between versions:

// Kernel 5.4
struct task_struct {
    int pid;
    char comm[16];
    // ... other fields
};

// Kernel 5.10 - field moved!
struct task_struct {
    char comm[16];  
    int pid;        // Different offset!
    // ... other fields  
};

The Solution - CO-RE:

We briefly introduced this in the previous post. Now, let’s explore how CO-RE works and why it's crucial for building portable eBPF programs.

CO-RE leverages BTF (BPF Type Format) to make eBPF programs kernel-version aware at runtime, without recompiling for each kernel.

#include "vmlinux.h"  // Generated BTF types

struct task_struct *task = (struct task_struct *)bpf_get_current_task();
// CO-RE automatically adjusts field offsets at load time!
int pid = BPF_CORE_READ(task, pid);

Here’s what’s happening:

vmlinux.h contains type definitions extracted from the kernel using BTF.
BPF_CORE_READ() safely reads the pid field from task_struct, regardless of its offset.
When you load the eBPF program, libbpf compares compiled offsets with the actual kernel’s layout and relocates field access as needed.

Benefits:

Compile once, run on any kernel version
Automatic field offset relocation
Runtime kernel adaptation

This is why build.rs includes vmlinux - it provides BTF types for the target kernel!

6. BPF Tail Calls: Program Chaining

Tail calls allow one eBPF program to call another, enabling:

Complex logic across multiple programs
Runtime program updates
Modular architectures

// Program array map
struct {
    __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
    __uint(max_entries, 10);
    __type(key, u32);
    __type(value, u32);
} programs SEC(".maps");

// Tail call to another program
bpf_tail_call(ctx, &programs, program_index);
// This program ends here - control transfers completely

Use Cases:

Packet processing pipelines
Feature toggles (enable/disable functionality)

7. BPF Token: Delegated Privileges

BPF Token enables secure eBPF in containers by allowing privilege delegation.

The Problem:

eBPF traditionally requires CAP_BPF or CAP_SYS_ADMIN, which gives too many privileges for containers.

The Solution:

# Privileged process (container runtime) creates token
bpftool token create /sys/fs/bpf/token delegate_cmds prog_load,map_create

# Unprivileged container uses token
bpf_prog_load_token(prog_fd, token_fd, ...)

Benefits:

Containers can run eBPF without root
Fine-grained permission control
Better security isolation

Note: BPF Token is available starting Linux 6.7. See kernel docs for full usage.

8. eBPF Tools Ecosystem

bpftool - The Swiss Army Knife

bpftool

# List all programs
bpftool prog list

# Show program details
bpftool prog show id 123

# Dump program bytecode
bpftool prog dump xlated id 123

# Pin programs to filesystem
bpftool prog pin id 123 /sys/fs/bpf/my_prog

bpftrace - Dynamic Tracing Scripts

bpftrace is a quick way to debug your eBPF program.

# Trace file opens
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s opened %s\n", comm, str(args->filename)); }'

# Network packet counting
bpftrace -e 'kprobe:dev_queue_xmit { @packets[comm] = count(); }'

BCC - Python eBPF Framework

bcc is a powerful toolkit for writing and running eBPF programs.

Higher-level Python interface
Great for prototyping and learning
Compiles C code at runtime

Rust/libbpf-rs approach which we discussed in the previous blog, is more modern and performant than BCC!

9. Putting It All Together

Here's how these components work in syscall tracer which is demostrated in the previous blog:

Program Type: Tracepoint (sys_enter_*)
Hook Point: Syscall entry points
Verifier: Validated the program safety
Maps: Perf event array for data streaming
Helper Functions: bpf_get_current_pid_tgid(), bpf_perf_event_output()
BTF/CO-RE: Automatic kernel compatibility
Tools: bpftool for debugging, Rust for userspace

Next Steps

With a solid understanding of eBPF’s foundational building blocks, you’re now ready to go beyond theory and start building powerful, production-grade tools.

In upcoming posts, we’ll use these primitives to implement practical, real-world applications—tapping into different eBPF program types and kernel hook points. While we won’t spoil what’s next, expect deep dives into applied observability, security, and networking powered by eBPF and Rust.

To explore more in the meantime:

Introduction to eBPF – Learn the concepts behind the technology
eBPF Documentation Hub – Official docs, tutorials, and developer guides

# Intro into eBPF and Rust

Mahesh Rayas — Mon, 07 Jul 2025 08:20:02 +0000

In this post, we’ll explore how to build high-performance Linux tracing tools using eBPF and Rust. Whether you're just getting started with eBPF or curious about integrating it with modern Rust tooling, this guide will walk you through the fundamentals, tooling choices, and a working syscall tracing example.

What is eBPF

eBPF is a technology that allows developers to run safe, sandboxed programs in the Linux kernel. These programs can be used for networking, observability, and security, without modifying kernel source code or loading kernel modules.

eBPF runs inside a restricted virtual machine in the kernel — somewhat like how WebAssembly (WASM) or the JVM operates in userspace — ensuring safety and control over what the program can do.
One of the biggest advantages of eBPF is that it allows extending kernel behavior dynamically, without waiting for upstream patches or rebooting systems. Since the programs run in kernel context, they can observe and act on events with minimal overhead, making them ideal for high-performance tracing, filtering, and enforcement tasks.

You can find a more in-depth explanation at ebpf.io.

Why Rust

Rust is a powerful, modern systems programming language focused on performance, memory safety, and concurrency — all without a garbage collector.

It is an excellent fit for working with eBPF for several reasons:

Memory Safety: Unlike C, Rust prevents common bugs like null pointer dereferences, buffer overflows, and use-after-free — which are critical when interacting with low-level kernel interfaces.

Strong Tooling: Tools like Cargo, rust-analyzer, clippy, and libbpf-cargo make building and testing eBPF applications in Rust a seamless experience.

What are the ready-to-use rust libraries

libbpf-rs

A safe and idiomatic Rust wrapper around the widely-used C libbpf library.
Designed and maintained by some of the same people behind libbpf.
Integrates well with tools like bpftool and libbpf-cargo.
You still write your eBPF programs in C, but use Rust for the userspace loader.
Best for: Production-grade systems, high compatibility, and CO-RE (Compile Once Run Everywhere).

Link: https://github.com/libbpf/libbpf-rs

Aya

A pure Rust eBPF toolchain — you write both the userspace and eBPF program in Rust.
No need for C or clang toolchain.
Includes support for XDP, tracepoints, uprobes, perf buffers, maps, etc.
Still maturing, but advancing fast with great community support.
Best for: All-Rust toolchains, embedded use cases, and easier portability.

Link: https://aya-rs.dev/

Why libbpf-rs

Before diving deeper into libbpf-rs, let’s understand the role and significance of libbpf itself.

What is libbpf?

When you write and compile an eBPF program, it often relies on the internal structure of kernel data types. However, these kernel structures can change between Linux versions — for example, a struct in Linux 5.6 may have an additional field or a different layout in Linux 6.1. This mismatch can cause your eBPF program to behave incorrectly or even fail verification when loaded into a newer kernel.

To address this versioning problem, the libbpf library — often referred to as the eBPF loader — introduced a feature called CO-RE (Compile Once, Run Everywhere). This powerful capability allows you to compile your eBPF program once and run it across different kernel versions without recompilation.

How CO-RE Works

Under the hood, CO-RE leverages vmlinux.h, a header file that contains all the type definitions used by your running kernel.

To generate it, you can run:

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

This vmlinux.h file is created from the kernel’s BTF (BPF Type Format) data and can be used in your .bpf.c programs to access kernel structures safely and portably.

When the eBPF program is loaded, libbpf inspects the fields your program accesses and ensures they align correctly with the actual in-kernel data layout — even if the structure has shifted. CO-RE handles this through relocation logic at load time, making your program kernel-version aware without being hardcoded to one layout.

If you're interested in a deep technical dive, I highly recommend Andrii Nakryiko’s blog post on CO-RE.

Why I Chose libbpf-rs

I initially explored aya-rs, a pure-Rust eBPF toolchain that allows writing both userspace and kernel-space components in Rust. While Aya is ergonomic and modern, I encountered issues when using certain eBPF helper functions — the verifier would reject the program.

After some investigation and community input (gh issue) I learned that these problems were caused by CO-RE incompatibilities with the Rust compiler (rustc). Since CO-RE support in aya is still maturing and relies on experimental LLVM features for Rust, I decided to switch.

That’s when I discovered libbpf-rs, a safe and idiomatic Rust wrapper around the stable and battle-tested libbpf C library.

Why libbpf-rs Works Well for My Use Case

It provides first-class CO-RE support, meaning portability is built-in.

It allows me to write the eBPF program in C, which aligns with most of the production-grade open source projects like Cilium, Tracee, and others.

Because much of the ecosystem and documentation around eBPF is written in or assumes C, using C for the eBPF part makes it easier to understand, reuse, and extend existing examples.

The userspace logic remains in Rust, giving me the safety, ergonomics, and modern tooling of the Rust ecosystem while relying on a stable foundation for kernel interaction.

eBPF + Rust: Architecture Diagram

Here’s a high-level look at the lifecycle of an eBPF program integrated with a Rust userspace application:

Write an eBPF program alongside the userspace Rust code.
Compile the source to generate eBPF bytecode and the Rust binary.
Run the userspace binary.
The userspace binary attempts to load the eBPF bytecode into the kernel.
The eBPF verifier (more on this in future posts) checks that the eBPF program is safe to run in the kernel.
If verification fails, the program is rejected. Otherwise, the eBPF bytecode is JIT-compiled into machine instructions and successfully loaded into the kernel.
Once loaded, the userspace binary attaches the eBPF program to relevant kernel hook points.
Communication between the Rust userspace and kernel space is handled using eBPF Maps.

We'll explore these primitives in depth with hands-on examples in upcoming posts.

Example: Tracing Syscalls with eBPF and Rust

Let’s walk through a simple example that traces all syscalls invoked on a Linux system.

Below is a snapshot of the project structure. You can also explore and run the full code from the GitHub repo.
Make sure your system meets the eBPF prerequisites before running.

└── syscall-tracer
    ├── build.rs   # Builds the C code and generates eBPF bytecode & Rust bindings
    ├── Cargo.lock
    ├── Cargo.toml
    ├── Cross.toml # Cross-compilation configuration
    ├── README.md
    └── src
        ├── bpf
        │   ├── syscall.bpf.c  # eBPF program
        │   ├── syscall.skel.rs # Auto-generated bindings by libbpf-cargo
        │   └── vmlinux.h # CO-RE header for kernel type definitions
        ├── lib.rs
        ├── log.rs
        └── main.rs   # Rust userspace: loads, attaches eBPF and handles kernel events

This is how the output might look when tracing syscalls:

What’s Next?

This post is just the beginning. There’s a lot more to explore with eBPF and Rust — from powerful capabilities to real-world applications.

I’ll be sharing more advanced and interesting examples in future posts. Stay tuned — you won’t want to miss what’s coming next!

Conclusion

Whether you're building observability tools or experimenting with kernel internals, eBPF + Rust offers a powerful, safe foundation. The possibilities ahead are exciting — and we're just getting started.