Forem: Arthur Picerna

Troubleshoot Container OOM Kills with eBPF

Arthur Picerna — Mon, 16 Jun 2025 05:23:26 +0000

So I had a little time on my hands a couple weeks ago and decided to explore how to use eBPF to monitor container OOM kills.

Why?

Out Of Memory (OOM) kills are an early sign of saturation in Linux systems. It happens when a machine runs out of available memory space and starts killing processes until it can breathe again.

In the context of containers, with things such as Kubernetes or Docker, OOMs happen when a container's memory usage exceeds its memory limits. The offending container processes get selected and OOM killed. In essence, they receive a SIGKILL signal and unexpectedly terminate any ongoing task.

OOM kills might lead to unexpected service degradation and prompt a mild "How did that happen?" (or "wtf just happened" depending on the situation) from whoever is investigating the crime scene. Trick being that all evidence is mostly lost to the void as the offending process is gone. So yea, back to the roots...

How do OOM kills happen in containers?

Container resource limits are enforced through cgroups. Neither Kubernetes nor Docker have any control whatsoever over which process gets killed. It is the responsibility of the underlying Linux host kernel to enforce its resource boundaries.

Journal kernel logs displaying a memory cgroup 'out of memory' message:

Excessive container memory usage, hence OOM kills, can be attributed to either:

Under-provisioned container memory
Faulty container application behavior
A bit of both 1. and 2.

Picture how an unoptimized process that needs to use a lot of memory for inbound requests might be prone to OOM kills. Now imagine what happens in peak traffic conditions if one container goes down and the same load gets redistributed to the remaining containers while it restarts? Highly likely to cause cascading failures.

If the process chosen by the kernel is the container's main process, then the main process will exit which will stop the container and that would typically trigger a container restart. It takes some time to restart a container, time which is not spent running its workload.
But the OOM killer could instead select a child process within the container's main process, then the container itself will outlive the OOM kill and new child processes might eventually respawn. One could argue that is even worse because the symptom is much less obvious.

Unless you already know why OOMs happen and are fine with it, e.g. voluntarily relying on OOMs to restart containers; in most cases though you'd really want to be able to assert what is happening in your system at the time of the OOM kills.

ℹ️ cAdvisor metrics can be used to track containers memory usage and OOM kills to some extent. In this article we merely focus on exposing Linux kernel's internal memory management stats.

That's where eBPF comes in.

eBPF enters the room

The extended Berkeley Packet Filter (eBPF) allows you to run sandboxed probe programs without changing the Linux kernel source code or loading kernel modules. Think of it as a way to hook to kernel function and execute custom code. This opens a plethora of possibilities, amongst which troubleshooting OOM kills.

There are 2 main eBPF toolkits you can use: BCC and bpftrace. Both of them provide an 'oomkill' probe example, respectively tools/oomkill.py for BCC and tools/oomkill.bt for bpftrace. As a side note both were written by Brendan Gregg.

I found bpftrace syntax more concise and easier to understand so I tried running that locally, here is an example output:

❯ sudo /usr/sbin/oomkill.bt
Attaching 2 probes...
Tracing oom_kill_process()... Hit Ctrl-C to end.
19:44:13 Triggered by PID 1516777 ("python3"), OOM kill of PID 1516777 ("python3"), 32768 pages, loadavg: 0.72 0.64 0.71 3/2230 1517322

The above shows the process PID, total pages count and CPU loadavg, from the host perspective. It doesn't give much container context, let's see if we can change that.

Here is the original bpftrace oomkill probe code:

#ifndef BPFTRACE_HAVE_BTF
#include <linux/oom.h>
#endif

BEGIN
{
    printf("Tracing oom_kill_process()... Hit Ctrl-C to end.\n");
}

kprobe:oom_kill_process
{
    $oc = (struct oom_control *)arg0;
    time("%H:%M:%S ");
    printf("Triggered by PID %d (\"%s\"), ", pid, comm);
    printf("OOM kill of PID %d (\"%s\"), %d pages, loadavg: ",
        $oc->chosen->pid, $oc->chosen->comm, $oc->totalpages);
    cat("/proc/loadavg");
}

Let's make sense of it, starting with this line:

kprobe:oom_kill_process

It tells that we define a kernel probe (kprobe) that will hook onto the oom_kill_process() function. Makes sense.

Now let's look at the following line:

$oc = (struct oom_control *)arg0;

It tells that we receive the function's first argument arg0 and type cast it into an oom_control struct. But wait, how do we know that the oom_kill_process() function receives an oom_control struct as first argument? Well, let's take a look at the Linux kernel source code.

Short dive into the Linux kernel

Go to the torvalds/linux git repo and search for the oom_kill_process() function declaration, like this.

From mm/oom_kill.c you can see:

static void oom_kill_process(struct oom_control *oc, const char *message)

Not only have we confirmed where the oom_control argument comes from but now we know that oom_kill_process() receives a second string argument message.

Next, what about what's inside an oom_control struct? Might there be anything else useful in there? Let's have a look.

You know the drill, search for oom_control in the source code, like this.

From include/linux/oom.h you can see:

/*
 * Details of the page allocation that triggered the oom killer that are used to
 * determine what should be killed.
 */
struct oom_control {
    /* Used to determine cpuset */
    struct zonelist *zonelist;

    /* Used to determine mempolicy */
    nodemask_t *nodemask;

    /* Memory cgroup in which oom is invoked, or NULL for global oom */
    struct mem_cgroup *memcg;

    /* Used to determine cpuset and node locality requirement */
    const gfp_t gfp_mask;

    /*
     * order == -1 means the oom kill is required by sysrq, otherwise only
     * for display purposes.
     */
    const int order;

    /* Used by oom implementation, do not set */
    unsigned long totalpages;
    struct task_struct *chosen;
    long chosen_points;

    /* Used to print the constraint info. */
    enum oom_constraint constraint;
};

There's some interesting stuff in there. First of all, we now understand where the $oc->totalpages in the original oomkill probe's code comes from. And we can see a few other promising properties such as memcg, chosen and chosen_points. Let's unpack what these are in more detail.

memcg is a mem_cgroup object that holds the container's cgroup memory usage and limits.

chosen is a task_struct object that holds the chosen process properties amongst which you can find mm and nsproxy that respectively hold memory management and container's namespaces information.

chosen_points comes from the oom_badness() score calculation that is used by the OOM killer to select which process to kill.

The formula for the OOM badness score calculation ends up looking something like this:

chosen_points = 
    mm_filepages + 
    mm_anonpages + 
    mm_shmempages + 
    mm_swapents + 
    mm_pgtables_bytes / PAGE_SIZE + 
    oom_score_adj * totalpages / 1000

ℹ️ PAGE_SIZE is a constant which may vary by system but is generally equivalent to 4096 bytes.

With the process resident set size (RSS) = mm_filepages + mm_anonpages + mm_shmempages + mm_swapents.

In essence, it's summing all of the process RSS memory footprint, counted in pages, plus some page tables space reserved by the kernel and an adjustable factor that can be set by the user per process. The main take away is that the process with highest RSS, hence a higher number of oom badness points, is the most likely to get OOM killed. And from the $oc->chosen->active_mm->rss_stat object you can access the various components of that process RSS.

As the above are the main variables which influence the OOM killer's process selection let's see if we can include them in the output of our probe.

A revised oomkill probe, for containers

Let's mkdir container-oomkill-probe && cd container-oomkill-probe.

Here is the revised bpftrace script, let's name it container_oomkill.bt:

#!/usr/bin/env bpftrace
/*
 * container_oomkill    Trace OOM killer in containers.
 *        For Linux, uses bpftrace and eBPF.
 *
 * This traces the kernel out-of-memory killer by using kernel dynamic tracing of oom_kill_process().
 * Prints the process host pid, container id, cgroup path, command and a few other stats.
 * Note: There's no guarantee that the OOM killed process is within a "container", this script just assumes it is.
 *
 * Example of usage:
 *
 * # ./container_oomkill.bt
 * Tracing oom_kill_process()... Ctrl-C to end.
 *
 * Adapted from the original bpftrace's tools/oomkill.bt by Brendan Gregg:
 * -> https://github.com/bpftrace/bpftrace/blob/master/tools/oomkill.bt
 */

#ifndef BPFTRACE_HAVE_BTF
#include <linux/oom.h>
#endif

BEGIN
{
    printf("Tracing oom_kill_process()... Hit Ctrl-C to end.\n");
}

// fn: static void oom_kill_process(struct oom_control *oc, const char *message)
// https://github.com/torvalds/linux/blob/master/mm/oom_kill.c#L1017
kprobe:oom_kill_process
{
    $oc = (struct oom_control *)arg0;
    $message = str(arg1);

    // print datetime with milliseconds precision
    printf("%s", strftime("%Y-%m-%d %H:%M:%S", nsecs));
    printf(",%03d", (nsecs % 1000000000) / 1000000);

    // print labels
    printf(" probe=\"%s\"\n",
        probe);
    printf("  message=\"%s\"\n",
        $message);
    printf("  host_pid=\"%d\" container_id=\"%s\" command=\"%s\"\n",
        $oc->chosen->pid,
        $oc->chosen->nsproxy->uts_ns->name.nodename,
        $oc->chosen->comm);

    // oom_control stats
    printf("  oc_totalpages=\"%d\" oc_chosen_points=\"%d\"\n",
        $oc->totalpages,        // = mem + swap
        $oc->chosen_points);    // = filepages + anonpages + swapents + shmempages + pgtables_bytes / PAGE_SIZE + oom_score_adj * totalpages / 1000

    // cgroup stats
    printf("  memcg_memory_usage_pages=\"%d\" memcg_memory_max_pages=\"%d\" memcg_memory_low_pages=\"%d\"\n",
        $oc->memcg->memory.usage.counter,   // memory usage in pages
        $oc->memcg->memory.max,             // memory hard limit
        $oc->memcg->memory.low);            // memory request
    printf("  memcg_swap_current_pages=\"%d\" memcg_swap_max_pages=\"%d\" memcg_swappiness=\"%d\"\n",
        $oc->memcg->swap.usage.counter,     // swap usage in pages
        $oc->memcg->swap.max,               // swap hard limit
        $oc->memcg->swappiness);

    // stats used in OOM badness calculation
    printf("  mm_rss_filepages=\"%d\" mm_rss_anonpages=\"%d\" mm_rss_swapents=\"%d\" mm_rss_shmempages=\"%d\"\n",
        $oc->chosen->mm->rss_stat[0].count,
        $oc->chosen->mm->rss_stat[1].count,
        $oc->chosen->mm->rss_stat[2].count,
        $oc->chosen->mm->rss_stat[3].count);
    // in case you get hit by
    // "ERROR: The array index operator [] can only be used on arrays and pointers, found record..."
    // "ERROR: Can not access field 'count' on expression of type 'none'..."
    // prior to linux 6.2 $oc->chosen->mm->rss_stat is a mm_rss_stat struct
    // https://github.com/torvalds/linux/commit/f1a7941243c102a44e8847e3b94ff4ff3ec56f25#diff-dc57f7b72015cf5f95444ec4f8a60f85d773f40b96ac59bf55b281cd63c06142
    // you can use the version below instead
    //printf("  mm_rss_filepages=\"%d\" mm_rss_anonpages=\"%d\" mm_rss_swapents=\"%d\" mm_rss_shmempages=\"%d\"\n",
    //    $oc->chosen->mm->rss_stat.count[0].counter,
    //    $oc->chosen->mm->rss_stat.count[1].counter,
    //    $oc->chosen->mm->rss_stat.count[2].counter,
    //    $oc->chosen->mm->rss_stat.count[3].counter);
    printf("  mm_pgtables_bytes=\"%d\"\n",
        $oc->chosen->mm->pgtables_bytes.counter);
    printf("  proc_oom_score_adj=\"%d\"\n",
        $oc->chosen->signal->oom_score_adj);    // score adj used in oom_badness calculation

    // minor and major page faults
    printf("  proc_min_flt=\"%d\" proc_maj_flt=\"%d\"\n",
        $oc->chosen->min_flt,                   // minor page faults
        $oc->chosen->maj_flt);                  // major page faults

    // calculated stats
    printf("  uptime_ms=\"%lld\"\n",
        (nsecs - $oc->chosen->start_time) / 1000000);
}

A few noteworthy bits:

$oc->chosen->nsproxy->uts_ns->name.nodename gets the container id
$oc->memcg->memory contains the cgroup memory usage and limits, $oc->memcg->memory.max is what you would call in Kubernetes the "resource limit" while $oc->memcg->memory.low would be the "resource request"
$oc->memcg->swap contains the cgroup swap usage and limits, which might be relevant as we saw that swapents is one of the factors that influences OOM killer's selection
$oc->chosen->mm->rss_stat contains the process RSS stats list, ordered as defined in the enum NR_MM_COUNTERS, respectively MM_FILEPAGES, MM_ANONPAGES, MM_SWAPENTS, MM_SHMEMPAGES
$oc->chosen->signal->oom_score_adj and $oc->chosen->mm->pgtables_bytes.counter are part of the variables we saw being used in the OOM badness calculation so we can display that as well

Now let's write a Dockerfile to build an image with bpftrace installed and the updated container_oomkill.bt so that the probe runs from within a container:

FROM debian:stable-slim

RUN apt-get update && \
    apt-get install -y \
    linux-headers-generic \
    bpftrace \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY *.bt /app/

CMD ["bpftrace", "container_oomkill.bt"]

Build command:

docker build -t container-oomkill-probe .

Run command:

docker run --privileged --pid=host -v /sys:/sys:ro container-oomkill-probe

You should see somethings like this:

❯ docker run --privileged --pid=host -v /sys:/sys:ro container-oomkill-probe
Attaching 2 probes...
Tracing oom_kill_process()... Hit Ctrl-C to end.

Looks fine and all, but how do we trigger the probe?

Test or it didn't happen

Here is a simple python script that hogs memory by 4MiB increments:

#!/usr/bin/env python3

import logging
import time

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s: %(message)s")

i = 0
memory = []
while True:
    i += 1
    logging.info(f"Allocating {i * 4} MiB...")
    memory.append(bytearray(4 * 1024 * 1024))
    time.sleep(0.1)

Save it in a new stress-mem/ folder as stress-mem/main.py.

For testing purposes you can run the script in a 128MiB memory constrained Docker container with the container oomkill probe alongside it.

docker-compose.yml file:

services:

  probe:
    build:
      context: .
      dockerfile: Dockerfile
    volumes:
      - /sys:/sys:ro  # required to access /sys/fs/cgroup/docker/CONTAINER_ID
    privileged: true  # required to run bpftrace, as an alternative could try setting CAP_BPF, CAP_PERFMON
    pid: host         # required to trace processes from host

  stress-mem:
    image: python:3-slim
    volumes:
      - ./stress-mem:/app
    working_dir: /app
    command: python3 main.py
    restart: on-failure
    deploy:
      replicas: 1
      resources:
        limits:
          memory: 128MiB
        reservations:
          memory: 64MiB

Run command:

docker compose up --build

Sample output:

But wait, how come if we defined a 128MiB limit our stress-mem container is able to allocate up to 252MiB?

Surely our probe is completely wrong and we've wasted our valuable time on something useless. Unless...

Notice the mm_rss_swapents="32549"? Well, that is swap. And swap isn't disabled by default in Docker. Memory gets "swapped to disk" when memory pressure forces the kernel to do so. You can confirm that the system is struggling seeing the proc_maj_flt count. Major page faults happen when the system reads from disk instead of RAM.

Do we want swap? No we don't. Let's disable that. In our docker-compose.yml we can disable swap by setting memswap_limit equivalent to the memory limit, which is pretty confusing indeed.

  stress-mem:
    # ...
    memswap_limit: 128MiB   # disable swap by setting memswap_limit = mem_limit

Rerun the docker compose up --build command and you should see:

Well that's much better. No major page faults, no swap, the process got OOM killed at 124MiB which is the last increment before it could reach 128MiB. Perfect. Our probe works yay!

Wrapping up

Taking a step back, here is what we did with our new container oomkill probe:

Show container contextualized information from the process memory cgroup and namespace
Expose kernel's internal memory management stats providing additional insights on the reasons a process might have been OOM killed

It's not much, but you are now fully equipped with the tools and knowledge to find out more, build your own eBPF probes, understand how the Linux kernel works.

References and further reading

TL;DR:

Here's the code.

glhf 😉

Publish your Markdown docs on GitHub Pages

Arthur Picerna — Tue, 10 Aug 2021 19:12:18 +0000

As a developer you've most likely switched machines, even OS several times. Maybe you've performed a specific set of operations that you'll have to repeat in the future? want to remember some command lines? share code snippets? capture online course notes?

For all of the reasons above, and many more, having a personal documentation website is ideal. It lets you organize and synthesize all the information that is relevant to YOU in your own personal way, with the added benefit of being easily shareable.

What we'll do

A project containing all your Markdown files, hosted on Github Pages.

Requirements

Python 3 (tested with version 3.9.5+)
Git (2.12+)
Make

How we do it

Install `mkdocs`

We'll use mkdocs as static site generator, so let's install it:

pip install mkdocs

Create your docs source project

Now create the basic project structure, in your projects folder:

mkdocs new GITHUB_USERNAME
cd GITHUB_USERNAME

Replace GITHUB_USERNAME with your GitHub username or whatever you prefer to name your project. We'll be referencing it in the next steps and it will become clearer why you might want to choose that project name.

Check the created files in that folder, you'll see something like the following:

./
-- docs/
-- mkdocs.yml

Go ahead and try running mkdocs serve then open http://127.0.0.1:8000 in your browser. And voilà! You have a personal documentation local website. Any .md file added in docs/ will appear as a new web page. Try to edit docs/index.md for example.

mkdocs.yml contains your project configuration. You can customize your site name, navigation layout and so on. A simple example would be:

site_name: your site name
site_url: https://GITHUB_USERNAME.github.io
site_author: you
site_description: a short description of your website

nav:
  - About: index.md

You can also use the readthedocs provided theme by adding the following:

theme:
  name: readthedocs
  highlightjs: true
  hljs_languages:
    - yaml

For reference see https://www.mkdocs.org/user-guide/configuration/

At this point you should have something like this locally:

For a local documentation project that would be enough. But we'll go ahead and make this deployable to your GitHub page.

Create the GitHub repositories

GitHub allows you to host a website in a specific repository matching GITHUB_USERNAME.github.io, with your actual GitHub username in place of GITHUB_USERNAME.

Luckily mkdocs makes it easy to push your project build files to your GitHub page through the mkdocs gh-deploy command.

The downside of mkdocs gh-deploy is that it wipes out all the files in GITHUB_USERNAME.github.io/ to generate the website files (html, css, js).

To overcome this constraint and simplify the deploy process you'll track your GitHub page repo, GITHUB_USERNAME.github.io, independently as a git submodule of your docs source project. Which will allow to run mkdocs gh-deploy from within the docs source project itself.

Ultimately you'll need two distinct repositories: one for your GitHub page, another one for your source files.

Create the GitHub page repo

Go to https://github.com/new and create your GitHub page repository
Name it GITHUB_USERNAME.github.io (replace GITHUB_USERNAME with your actual GitHub username)

📝 Edit 16 Apr 2023
If you did not choose your GitHub username in lieu of GITHUB_USERNAME then you might have to do a couple of extra steps:

From the github.io repo, choose Settings

Under Code and automation, click on Pages

In Source > Deploy from a branch

In Branch > choose main and save

Type in the URL of your website "my-new-docs-website.github.io" and save

Thanks for the feedback @konsoul!

Create the source repo

Go to https://github.com/new and create your source repository
Name it GITHUB_USERNAME (replace GITHUB_USERNAME with your actual GitHub username or anything else you chose to name the new mkdocs project previously created)

📝 A repository named as your username is a special repository on GitHub. Its README.md will appear on your public profile. So you can customize that as well.

At this point you should have two GitHub repositories:

Git repo	Description
GITHUB_USERNAME.git (or anything else you chose to name it)	Source repo with markdown files
GITHUB_USERNAME.github.io.git	GitHub page repo with static web files

Initialize the git repositories

Back in the folder created previously:

git init

echo site/ > .gitignore

git add .
git commit -m "first commit"

git branch -M main
git remote add origin SOURCE_REPO # replace SOURCE_REPO with your source repo HTTPS or SSH adddress
git push -u origin main

Then initialize the GitHub page repo:

# still in your source folder

mkdir GITHUB_USERNAME.github.io
cd GITHUB_USERNAME.github.io

git init
git commit --allow-empty -m "first commit"

git branch -M main
git remote add origin GH_PAGE_REPO # replace GH_PAGE_REPO with your source repo HTTPS or SSH adddress
git push -u origin main

cd ..

You should now have two folders tracked in their respective GitHub repositories:

GITHUB_USERNAME/                  Source repo
-- GITHUB_USERNAME.github.io/     GitHub page repo
-- docs/
-- .gitignore
-- mkdocs.yml

Now delete GITHUB_USERNAME.github.io/ since there's no need for it to exist independently:

rm -rvf ./GITHUB_USERNAME.github.io

Add your GitHub page repo as a submodule of your source repo instead:

git submodule add -b main GH_PAGE_REPO # replace GH_PAGE_REPO with your source repo HTTPS or SSH adddress
git add GITHUB_USERNAME.github.io
git commit -m "add github page submodule"
git push

Deploy your docs to your GitHub page

Run:

cd GITHUB_USERNAME.github.io
mkdocs gh-deploy --config-file ../mkdocs.yml --remote-branch main
cd ..

Go ahead and open https://GITHUB_USERNAME.github.io in your browser. And voilà! Your very own personal documentation on your GitHub page.

📝 You might have to wait a few seconds for your changes to be deployed.

Add deploy command in Makefile

The mkdocs gh-deploy command in itself isn't very user friendly nor easy to remember so we'll add a simple Makefile to make (word choice on purpose) it easier to deploy changes.

Create a Makefile in your source folder with the following:

SHELL := /bin/bash
GH_PAGE := GITHUB_USERNAME.github.io

.PHONY: deploy
deploy:
    cd $(GH_PAGE) && mkdocs gh-deploy --config-file ../mkdocs.yml --remote-branch main

📝 Ensure to paste Makefile indentation as Tabs or else you might get an error in the lines of [...] *** missing separator. Stop.

Now you can simply run make deploy whenever you want to publish something.

Keep both repositories in sync

You might also want to update latest commit version of your GitHub page submodule whenever a deploy is performed. Not technically required, but good for tracking purposes. Add the following in your Makefile:

.PHONY: update-build-version
update-build-version:
    git submodule update --remote --merge
    git add $(GH_PAGE)
    git commit -m "ci: update build version"

.PHONY: publish
publish: deploy update-build-version
    git push

make publish will handle the deploy and update your submodule version.

A typical workflow would be:

make some changes to your markdown files or mkdocs.yml
git commit -am "commit message"
git push to your source repo
make publish to your GitHub page

Why so complicated ?

Can't I just run mkdocs build and use site/ as my GitHub page?
You can! See the GitHub docs. You will need to commit the build site/ artifacts, or you could even use a different branch for the GitHub page altogether. IMHO I'd much rather track the actual source code and deploy the minimal amount of files in the public project, and having two completely unrelated branches in the same repository is kind of an overly obscure solution. Hence the submodule approach. But that is my biased opinion, use whichever solution works best for you!

Just show me the code

Here is a sample project containing the output of everything described in this post:

https://github.com/ar2pi/sample-mkdocs-gh-project

Summary

You just created a brand new GitHub page hosting all your desired markdown generated documentation. Which can also contain your cv, portfolio, todo list, random thoughts, cat pictures, etc. you name it. Go nuts, the web is limitless.

Happy hacking!