Forem: Patrick Smyth

Deep Dive 🤿: Where Does Grype Data Come From?

Patrick Smyth — Tue, 12 Nov 2024 16:16:18 +0000

Grype is a vulnerability scanner for container images and filesystems. It's developed by Anchore and written in Golang. When you point Grype at a container image, it will scan the files and folders on that image, compare what it finds to a database of CVEs (known vulnerabilities), and spit out a report telling you what CVEs have been detected.

We like Grype at Chainguard because it's open source, customizable, and reliable enough to integrate into our CI and CVE remediation workflows. (You can read more about why we like Grype in this post.)

I know, my photoshop skills scare even me sometimes.

In this article, we'll answer a question that comes up frequently: where does Grype's vulnerability data come from? In the process, we'll take a look at Grype's open data pipe line and do some light analysis of the vulnerability data that Grype uses to scan containers.

How Grype Works

If you haven't used Grype before, here's a brief overview of how it works.

You point Grype at a container image (or filesystem).
Grype downloads a fresh instance of its vulnerability.db database, then scans the image for specific packages, files, configurations, and so on, building a manifest in the form of a Software Bill of Materials (SBOM) itemizing the software contained in the image. (Under the hood, Grype uses a sister tool, Syft, for this step.)
Grype then compares the specific versions of each package against the vulnerability data in its database.
Finally, a list of CVEs detected in the image is returned to the user.

For a comprehensive overview of Grype's functionality, check out Using Grype to Scan Container Images for Vulnerabilities on Chainguard Academy.

Grype's Data Sources

Grype relies on a set of upstream providers for its vulnerability data. As of November 2024, the list of providers includes:

Note that the above links are to endpoints where data is provided. Chainguard is one of the upstream providers, and updating scanners like Grype on the fixed status of packages in our upstream OS, Wolfi, is a key element in maintaining the low-to-no CVE status of Chainguard Images).

Grype's vulnerability.db gets rebuilt daily from data sourced from these upstream providers. To build this database, Grype uses two open source tools, vunnel and grype-db. The vunnel tool downloads, standardizes, and stores vulnerability data from the above upstream providers. Basically, it accesses the various provider endpoints and stores a local vulnerability database and metadata for each provider locally. The grype-db utility collates this vulnerability data, building a much smaller vulnerability.db usable by Grype.

Building the Grype Database with `vunnel` and `grype-db`

In this section, we'll try out the vunnel and grype-db utilities, building a local vulnerability cache and database.

Since a built-daily vulnerability.db file gets downloaded every time you run a Grype scan, why would you want to build Grype's vulnerability.db manually? Building manually is useful if:

You want to use a subset of upstream sources
You'd like to integrate other sources to create a custom vulnerability.db
You require older Grype schemas
You'd like to contribute to Grype
You want to understand more about Grype's upstream providers and data structure

`vunnel`

Thegrype-db utility uses vunnel under the hood, but let's first try out vunnel explicitly to see how it works. You'll need Python 3 installed for this section, and I'll assume it's accessible on your system using the python command. (vunnel is written in Python.)

First, let's create a project folder:

mkdir -p ~/vulnerability-data && cd $_

Next, create a virtual environment and activate it:

python -m venv venv && source venv/bin/activate

Now install vunnel to the activated virtual environment:

pip install vunnel

Once vunnel is installed, we can use the vunnel list command to show the current list of providers:

vunnel list

alpine
amazon
chainguard
...
sles
ubuntu
wolfi

You can download a local cache of provider data for any of these providers with the following (using chainguard as an example provider):

vunnel run chainguard

This creates a data folder where all provider data used as input and the standardized output database are contained:

data
└── chainguard
    ├── checksums
    ├── input
    │   └── secdb
    │       └── security.json
    ├── metadata.json
    └── results
        └── results.db

`grype-db`

The grype-db utility can pull provider data with vunnel under the hood, and can also collect and package up this data into the vulnerability.db file used by Grype.

In the following, we'll use grype-db to download all provider data using vunnel under the hood, then build the vulnerability.db file. The process of downloading all provider data can take some time and uses about 8 GB of disk space.

First, download the grype-db script to the project folder we created previously:

curl -sSfL https://raw.githubusercontent.com/anchore/grype-db/main/install.sh | sh -s -- -b .

If you'd like to build from all available data, you'll need a GitHub token capable of authenticating as a user. This is because GitHub rate limits API access for non-authenticated users. You can follow these instructions provided by GitHub, but in short head to this token settings page on GitHub. Remember to safeguard your token as you would a password, and I recommend creating a scoped and short-lived (i.e. 7 days) token.

Once you have your token, create a configuration file for grype-db in our ~/vulnerability-data project folder. First, set your generated GitHub token to an environment variable:

GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Next, generate the config in our ~/vulnerability-data project folder:

cat << EOF > ~/vulnerability-data/.grype-db.yaml
provider:
  vunnel:
    executor: local
    generate-configs: true
    env:
      GITHUB_TOKEN: $GITHUB_TOKEN
EOF

Now we have the grype-db script and the .grype-db.yaml configuration file in our project folder. Let's run a command that will pull all provider data, create a database file, and package it up for inclusion in a CI or other workflow. (For this step, you'll need vunnel available, so the virtual environment created in the previous section on vunnel will need to still be activated, and you should run this command from the project folder where we downloaded the grype-db script.)

./grype-db -g

Downloading and processing all provider data can take a long time, possibly hours, so go watch Master of the Flying Guillotine or get some work done, I guess. ☕

Once the process completes, a build folder will have been created in our ~/vulnerability-data/ project folder:

build
├── listing.json
├── metadata.json
├── provider-metadata.json
├── vulnerability.db
└── vulnerability-db_v5_2024-11-05T19:09:08Z_1730848219.tar.gz

The vulnerability.db database should be the same as the database built daily for use by Grype.

Grype's `vulnerability.db`

I plan on following up this post with an analysis of the data in Grype's vulnerability.db database, but here are some quick notes on the structure of this SQLite3 file:

When Grype runs, it checks against the last time the database was updated. If it's been longer than a day (Grype rebuilds the database daily) a new vulnerability.db is downloaded to a cache. - On Linux, it's stored in ~/.cache/grype/db/5/vulnerability.db, where the numbered folder (5) corresponds to the current Grype schema version number.) On Mac OS, it's stored in ~/Library/Caches/grype/db by default.

The vulnerability.db database has five tables, but only two have significant data. The vulnerability_metadata table stores information on CVEs as they apply on a per-platform basis. The entities in the vulnerability table represent vulnerabilities as they apply to specific package versions.

The platforms with the most vulnerability metadata entries are Ubuntu, NVD (NIST's National Vulnerability Database,, and Susa.

While these results are somewhat interesting when considering where Grype data comes from, the number here reflects many factors, mainly the date the provider started recording vulnerabilities and, for platform-specific providers, the attack surface of the platform. Other details such as duplication of distros lower the signal here and would require more analysis to parse out.

We can also check the number of vulnerability metadata entries by year:

This chart mainly shows a movement toward maturity in the ecosystem, leading to some stability after 2016-2017. (The early twenty-teens were a time of rapid development in cloud technologies in particular.)

Digging into the data in Grype's vulnerability.db can also help to answer much more specific questions about how CVE affect different platforms. For example, imagine we host a mailserver and we wish to know the fixed status of CVE-2024-37383, a known-exploited vulnerability which allows cross-site scripting in RoundCube, a webmail client. We can narrow the data in the vulnerability table to answer this question:

fix_state	namespace
fixed	nvd:cpe
fixed	debian:distro:debian:11
fixed	debian:distro:debian:12
fixed	debian:distro:debian:13
fixed	debian:distro:debian:unstable
not-fixed	ubuntu:distro:ubuntu:20.04
not-fixed	ubuntu:distro:ubuntu:22.04
fixed	ubuntu:distro:ubuntu:23.10

In a follow-up post, I'll show you how to load Grype's vulnerability.db database into Pandas to get a better sense of Grype's data schema and how it can be used to answer specific questions in platform security and broader CVE trends.

Conclusion

One of the most remarkable aspects of the Grype image scanner is the openness of its data pipeline. This is great for transparency and makes Grype's vulnerability.db into a more flexible and useful tool. If you've found this post useful, let us know and maybe you'll see more security deep dives like this. 🛡️🤿 You can follow Chainguard on dev.to or LinkedIn or sign up for our newsletter to keep in touch.

Resources

Securely Containerize a Python Application with Chainguard Images

Patrick Smyth — Fri, 12 Apr 2024 19:09:57 +0000

Overview

Containerization technologies such as Docker have revolutionized the development to production pipeline, making it easier than ever to reproduce environments and set up and maintain infrastructure. Unfortunately, containers come with their own risks and overhead. While standard images can be convenient for development, putting large images into production increases the attack surface and can demand additional and unnecessary resources in deployment.

In this tutorial, we'll build on a Chainguard Image to containerize a Python application in a multi-stage build process. By using a Chainguard Image, our containerized application benefits from a minimalist design that reduces attack surface and from a set of features focused on security and ease of development. In addition, the use of a multistage build process will allow us to develop our application with access to tools such as package managers while removing these potential vulnerabilities later in the build process.

Creating the Application

The Python application we will containerize in this tutorial, Chainguard Timeteller, will provide the user with the local time in ten randomly chosen international timezones. It also displays the time in UTC and in the user's local timezone. The application depends on pytz, allowing us to draw from the Olson tz database in selecting our random timezones.

Let's start by creating our main application script.
First, open your terminal and run the below line to create a new folder called timeteller in your home directory:

mkdir ~/timeteller && cd ~/timeteller

Open a new file called main.py in your preferred text editor. We'll use the widely available Nano editor in this tutorial.

nano main.py

Copy the following Python code into main.py:

from datetime import datetime
from time import tzname
from random import sample
from pytz import timezone, common_timezones


def tz_aware_now(tz=timezone("utc")):
    """Create a timezone-aware datetime for the current time and a provided timezone. Defaults to UTC."""
    now = datetime.now(tz)

    return now


def pretty_print_time(t):
    """Given a datetime object, return a human-readable and nicely-formatted time string.."""

    # Generate an ordinal suffix, i.e. "th" or "st" based on the day
    day = t.day
    day_ordinal_suffix = str(day) + (
        "th"
        if 4 <= day % 100 <= 20
        else {1: "st", 2: "dayd", 3: "rd"}.get(day % 10, "th")
    )

    date_pretty = " ".join([t.strftime("%-I:%M %p on %B"), day_ordinal_suffix])

    return date_pretty


def generate_timezone_message():
    """Create the message to display to a user. The message includes a greeting, the current time in UTC, the user's local timezone and time, and ten local times from randomly selected timezones."""

    local_time = tz_aware_now(tz_aware_now().astimezone().tzinfo)

    random_timezones = sample(common_timezones, 10)

    timezones_map = {
        zone: pretty_print_time(tz_aware_now(timezone(zone)))
        for zone in random_timezones
    }

    printable_timezones = [
        " ".join(["    ", "➤", zone, timezones_map[zone]]) for zone in timezones_map
    ]

    message = "\n".join(
        [
            "\nWelcome to Chainguard Timeteller! \n",
            f"🌎 The current time in UTC is {pretty_print_time(tz_aware_now())}.",
            f"❓ Your current timezone is {tzname[0]}.",
            f"⏰ Your local time is {pretty_print_time(local_time)}.\n",
            "Local times from ten randomly chosen timezones around the world:\n",
            *printable_timezones,
        ]
    )

    return message


if __name__ == "__main__":
    print(generate_timezone_message())

If you're using Nano, you can save the file by pressing Control-x, y, and Enter in sequence.

In this script, we define functions to return the current time in a specific timezone, generate nicely-formatted lines for each region, and pull together a message to the user. When run directly, the script prints the generated message, including the time in ten randomly selected timezones, to the console. The code depends on a library, pytz, not in the standard library, and we'll have to install it during our build process.

Because our code depends on pytz, a package not in Python's standard library, we'll also need to specify our dependencies. Open a requirements.txt file using your text editor:

nano requirements.txt

Copy the below into the file.

pytz==2024.1

Here, we specify the version of pytz we want to use. Once you're done, save the file.

Before we build our container, let's test that our script works. First, install our dependency using the pip package manager:

pip install -r requirements.txt

Depending on your system, you may need to use the pip3 command instead of the pip command.

Once pytz has installed, run the script with the below command:

python main.py

Depending on your system, you may need to use the python3 command instead of the python command. You should receive output similar to the following:


Welcome to Chainguard Timeteller! 

🌎 The current time in UTC is 6:07 PM on February 18th.
❓ Your current timezone is EST.
⏰ Your local time is 1:07 PM on February 18th.

Local times from ten randomly chosen timezones around the world:

     ➤ Europe/Dublin 6:07 PM on February 18th
     ➤ Africa/Lagos 7:07 PM on February 18th
     ➤ America/Tortola 2:07 PM on February 18th
     [...]

When you see the above output with local times in ten randomly-selected timezones, you'll know the application is working. You're ready to containerize Timeteller using a base image from Chainguard Images.

Multi-Stage Build Using Chainguard Images

Now that we have our application in place, we're ready to containerize it using a multi-stage build process. This build process works as follows:

We begin the build by pulling a development version of the python-latest Chainguard Image as our base image.
We copy the requirements.txt file to the image, activate our virtual environment, and install our dependency using pip.
We now pull the minimal runtime version of the python-latest image. This image does not contain pip or an interactive shell.
We copy our virtual environment (now with access to our dependency) from the dev image to the minimal runtime image.
We activate our virtual environment on the minimal runtime image and run the application.

Create a file named Dockerfile in your timeteller folder:

nano Dockerfile

Copy the following build instructions to the Dockerfile:

FROM cgr.dev/chainguard/python:latest-dev as builder

ENV LANG=C.UTF-8
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
ENV PATH="/timeteller/venv/bin:$PATH"

WORKDIR /timeteller

RUN python -m venv /timeteller/venv
COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

FROM cgr.dev/chainguard/python:latest

ENV TZ="America/Chicago"

WORKDIR /timeteller

ENV PYTHONUNBUFFERED=1
ENV PATH="/venv/bin:$PATH"


COPY main.py ./
COPY --from=builder /timeteller/venv /venv

ENTRYPOINT [ "python", "/timeteller/main.py" ]

Change the timezone in the line ENV TZ="America/Chicago"to your own local timezone. (Without providing this information, the detected timezone in the container would be UTC.)

Save the file. We should now be ready to perform the build. Run the following:

docker build . -t timeteller

This will build the image from the instructions in our Dockerfile and tag it with the name timeteller.

If the image build is successful, you're ready to run it with the following:

docker run --rm timeteller

You should see the output from our application as above, including the ten randomly selected local times. Congratulations! You've successfully containerized and run an application using Chainguard Images in a multi-stage build process.

The final image based on python:latest does not contain either pip or interactive shells such as sh or bash. Since package managers and shells are common vectors for attackers, having neither as part of our runtime image decreases the attack surface of our application in production. The multi-stage build process allows us to use tools such as shells and package managers during development while allowing us to keep our production image securely minimal.

Advantages of Chainguard Images

Chainguard Images provide a happy medium between superminimal images such as scratch and more complex distribution-based images such as Alpine or Debian. Chainguard Images aim specifically to reduce complexity, intentionally including only those software components necessary for runtime. Further, Chainguard Images are based on a distroless philosophy, meaning that they strip out additional software components traditionally associated with a distribution. Typically, a Chainguard Image contains only an application runtime, root certificates, a minimal file structure, and a small number of core system libraries.

Each Chainguard Image comes with a comprehensive SBOM (Software Bill of Materials). This allows users of Chainguard Images to check against known vulnerabilities, adhere to the legal terms of software licenses, and ensure software integrity.

Finally, the focus on minimal builds results in significantly fewer CVEs on your runtime images. Before we end this tutorial, let's scan for CVEs in our Timeteller image using an industry-standard tool, Docker Scout.

To use Docker Scout, you'll first have to have a Docker Hub account. Follow the installation instructions for Docker Scout on GitHub. Once Docker Scout is installed, you can sign in to Docker Hub on the command line with the docker login command.

Once we have Docker Scout installed, we can use it to scan for vulnerabilities with the following command:

docker scout cves timeteller

Running Docker Scout on our Timeteller image (built from a Chainguard Image) on February 18th, 2024 produced the following report:

    ✓ Image stored for indexing
    ✓ Indexed 32 packages
    ✓ No vulnerable package detected


## Overview

                    │       Analyzed Image         
────────────────────┼──────────────────────────────
  Target            │  timeteller:latest           
    digest          │  0cca410be7e4                
    platform        │ linux/amd64                  
    vulnerabilities │    0C     0H     0M     0L   
    size            │ 28 MB                        
    packages        │ 32                           


## Packages and Vulnerabilities

  No vulnerable packages detected

As you can see, no CVEs were detected in the Timeteller image at time of writing, a relatively rare outcome in the fast-paced world of container vulnerabilities and exposures. While your results with Chainguard Images won't always be this free of vulnerabilities, using Chainguard Images as your base will reduce your CVE incidence rate by 80% compared to comparable industry alternatives.

In this tutorial, you containerized a Python application with Chainguard Images in a multi-stage build process. This resulted in a runtime image with only the software components required to run our application. This focus on reducing software complexity resulted in a runtime image with a demonstrably low number of CVEs—zero in this case. Now that you understand the advantages of building your production infrastructure on Chainguard Images, you're ready to go forth and secure your own production environment.