Forem: Kyle Galbraith

Depot Changelog: June 2025

Kyle Galbraith — Thu, 10 Jul 2025 17:37:56 +0000

We shipped some awesome new features and improvements in June. Things like our latest egress filtering capabilities, audit logging, and Windows runners. Here is everything we shipped

Egress filtering for GitHub Actions Runners

We've shipped an awesome security feature to Depot GitHub Actions Runners. You can enable egress filtering to control exactly which IP addresses, hostnames, and CIDR ranges your GitHub Actions can talk to.

Get all the details in our launch post

Now available: Audit logging

We've rolled out support for audit logging across Depot. This allows you to get fine grained information about what actions are taken in your Depot organization.

Read the announcement post

Windows GitHub Actions runners are now GA

We've completed all the work to make our Windows runners generally available to all organizations across Depot. You can see all of the nitty gritty details and runner labels for our Windows runners in our docs.

You can also read our full launch post on our blog

depot cargo for faster Rust builds

We released a new CLI command called depot cargo that wraps your cargo command with Depot Cache automatically for exponentially faster Rust builds.

Check out the changelog entry for how to use it

Dependabot now runs on Depot GitHub Actions runners

You can now run all of your Dependabot jobs on Depot GitHub Actions runners to take advantage of our Ultra Runners, faster caching, unlimited concurrency, and more.

Check out our changelog entry for more details on how to enable it

And more good stuff...

depot CLI v2.88.0 includes several bug fixes and new features
Add support to depot push to push without Docker config credentials -- more details in our changelog entry
Fix for loading cache only targets in depot bake
Improved documentation for building depot CLI from source

Now available: Build autoscaling for everyone

Kyle Galbraith — Wed, 02 Jul 2025 20:08:18 +0000

When we first launched Depot, our goal was to make Docker image builds exponentially faster. Why? Because we experienced the absolute drudgery of waiting for container builds locally and in CI. The modern day equivalent of watching paint dry because saving and loading layer cache over networks negated all performance benefits of caching, and building multi-platform images required emulation, bringing builds to a crawl.

So, we built the solution we had always wanted: a fast, shareable, and reliable container build service that could be used from any existing CI workflow or anywhere you were using docker build. Today, Depot's container build service is used by thousands of developers to build Docker images faster than ever before. Saving engineering teams tens of thousands of hours in build time every week.

We don't bullshit about benchmarks, here is our benchmark for building PostHog's container images.

What is the container build architecture?

Before we get into the details of how autoscaling works, it's worth understanding how Depot's container build service works without it. When you run a container build, Depot runs an optimized version of BuildKit to process the build and cache layers to a persistent NVMe drive.

Behind the scenes, the flow looks like this:

You run depot build, which informs our control plane that you'd like to run a container build. We infer the architecture of the build based on the client architecture, or we look at the --platform flag if it is specified.
The control plane informs our provisioning system that a request for a container builder has been made for the specified platforms.
The provisioning system spins up one or more BuildKit builders to process the build. The number of builders is determined by the number of platforms in the build request (i.e., a multi-platform build will spin up a builder for each platform, such as linux/amd64, linux/arm64, etc.).
The authentication details and IP of the builders are returned to the client, and the depot build command connects directly to the builder to run the build.

In this flow, each BuildKit builder is responsible for processing the build and caching layers to a persistent NVMe drive for the architecture it supports.

By default, these BuildKit builders can process multiple jobs concurrently on the same host. This is a feature of BuildKit that enables deduplication of work across builds that share similar steps and layers.

So in this model, multiple depot build commands can run concurrently on the same BuildKit builder, and the builder will process them in parallel. This is great for most use cases, but it does have a limitation: the number of concurrent builds is limited by the resources of the single BuildKit builder.

But one limitation has remained

But one limitation has existed since the beginning: container builds could only run on a single BuildKit builder. Today, we are excited to announce the general availability of container build autoscaling, which removes that limitation, and also explain why you may not always want to use it.

How does container build autoscaling work?

With container build autoscaling, we can now automatically scale out your container builds to multiple BuildKit builders based on the number of concurrent builds you want to process on a single builder. This means that if you have a large number of concurrent builds, Depot will automatically spin up additional BuildKit builders to process them in parallel.

We've added a step 5 to our previous flow:

The control plane and provisioning system automatically scale out additional BuildKit builders based on the number of concurrent builds you want processed on a single builder. The authentication details and IPs of the additional builders are returned to the client, and the depot build command connects directly to all of the builders to run the build.

When should I use container build autoscaling?

This is by far the most important question to ask because of the tradeoffs involved. Container build autoscaling is a powerful feature that can significantly speed up your container builds. Still, it does come with some tradeoffs, like cache cloning and losing the ability to deduplicate work across builds that share similar steps and layers.

That said, container build autoscaling is particularly useful when a single build can consume all of the resources of a single BuildKit builder, or you have a large number of concurrent builds that chew through all of the resources as well.

In these cases, we recommend starting with sizing up your container builder, which you can see the full sizes available on our pricing page. This allows you to run larger builds on a single builder without needing to scale out to multiple builders.

However, for instances where you have a large number of concurrent builds or a single build that consumes all the resources of a single builder, container build autoscaling is a great way to speed up your builds.

How do I enable container build autoscaling?

To turn on container build autoscaling, you will need to navigate to your Depot project settings for the container build project you want to enable it for. From there, you can enable the autoscaling feature in the settings tab and specify the number of concurrent builds you want to process on a single builder.

Once enabled, all new builds will automatically scale out to multiple BuildKit builders based on the number of concurrent builds you specified. You can also adjust the number of concurrent builds at any time in the project settings.

What about the layer cache?

Autoscaling does come with tradeoffs, and one of those is that the layer cache the additional builders operate on is a clone of the main builder's layer cache. This means that the additional builders operate on a copy of the layer cache, and that copy is not written back to the main builder's layer cache. This means that the additional builders will not be able to share layers with the main builder, and you will not be able to take advantage of the deduplication of work across builds that share similar steps and layers.

Billing details

Container build autoscaling is available on all plans. Any cache clones created by the autoscaling feature are not persisted beyond the lifetime of the builder. This means that when the autoscaled builders are terminated, their layer cache clones are also deleted. Thus, cache clones do not count towards storage billing.

What's next?

We're excited to make this feature generally available to everyone using Depot for container builds. We believe that this is going to help folks get even faster Docker image builds, and we can't wait to see the new use cases for build performance that this enables.

We're working on some additional features here to make this feature available on our Build API as well as add some additional logic around how the cache clones are managed.

If you have any questions or feedback about this feature, please reach out to us on Discord and let us know.

Now available: Claude Code sessions in Depot

Kyle Galbraith — Tue, 01 Jul 2025 21:01:30 +0000

We've been using Claude Code at Depot since pretty much the moment it dropped. It's been a game changer for everything from our day to day development to debugging production issues. However, after using it for a while, we realized that we were all getting really annoyed by how challenging it was to share sessions with each other or resume them in ephemeral environments (like CI jobs!).

So, in traditional Depot fashion, we went and fixed our own problem, and today we're releasing Claude Code sessions in Depot. Now available to all Depot users on any of our pricing plans.

So, what are Claude Code sessions in Depot?

We built a new command for the Depot CLI, depot claude, that allows you to create, resume, and manage Claude Code sessions across your entire organization. This means you can now save your AI coding sessions in Depot, share them with your team, and pick up where you left off—no matter which machine or environment you're working in.

With Claude Code sessions in Depot, you can now get all of the following right out of the box:

Developers and AI agents - Hand off work between human developers and automated Claude agents
Team members - Share complex problem-solving sessions across time zones and teams
Local and CI environments - Start debugging locally and continue in CI, or vice versa
Different stages of development - Maintain context from design through implementation to review

How do you use Claude Code sessions in Depot?

To make use of Claude Code sessions in Depot, you first need to make sure you have the Depot CLI installed.

With the CLI installed, run depot login to authenticate your local CLI with your Depot organization. This is required to ensure your sessions are securely stored and accessible across your team.

With that, you're ready to start using Claude Code sessions in Depot. To create and manage these sessions, you
simply use the depot claude command instead of the regular claude command. This command automatically handles session persistence, allowing you to create and resume sessions with ease.

# Start a new session with a custom ID
depot claude --session-id feature-auth-redesign

This opens Claude Code, but it is wrapped in Depot's session management. You can interact with Claude as you normally would, but your session state is now saved to your Depot organization. Every interaction you have with Claude will be stored securely in your organization, allowing you to resume it later from any machine or environment.

If you don't specify a session ID, Depot will generate one automatically for you. You can also resume existing sessions by using the --resume flag with the session ID.

# Resume an existing session by ID
depot claude --resume feature-auth-redesign

You can pass any prompts or parameters you want to Claude, just like you would with the regular claude command. For example, you can specify a prompt to continue working on a specific task:

# Resume a session and provide a prompt
depot claude --resume feature-auth-redesign --model opus -p "continue implementing the authentication flow"

You can even use this from CI as well! Install Claude Code in your CI environment, authenticate with Depot, and then run the depot claude command to resume a session and continue working on it.

# In your GitHub Actions workflow
- uses: depot/setup-action@v1
- run: npm install @anthropic-ai/claude-code
- run: |
    depot claude --resume pr-${{ github.event.pull_request.number }} \
 -p "review this PR for security issues and best practices"

Uhhh, I don't remember my session?

Yeah, we had the same problem. So we added a way to list all of your sessions so you can see what you have available to resume. You can list all available sessions in your organization using the list-sessions command:

depot claude list-sessions

From there, you can choose to resume any sessions by selecting the session from the list.

When you resume a session, Depot will automatically load the last state of that session, including any code, prompts, and context you had previously set up. This means you can seamlessly continue your work without losing any progress. Once you exit again, your session state is saved back to Depot, ready for you or your teammates to pick up later.

Where could this be better?

There are a few areas worth being aware of as you begin using Depot's Claude Code sessions. Here is a quick list of things we know about that we are working on improving:

Work produced in sessions are not automatically committed to your git repository. So, if you start a session, we don't automatically create a branch for your session that Claude will work on. This is something we are looking to improve in the future, but for now, we recommend creating a branch for your session and naming your session after that branch. This way, you can easily find the work done in that session.
Claude Code must still be installed in your environment. Depot's Claude Code sessions are a wrapper around the existing Claude Code functionality, so you still need to have the Claude Code CLI installed and configured in your environment. Depot's sessions make it easier to manage and resume those sessions across different machines and environments.

How does it actually work?

The claude CLI already saves each of its session in a line-delimited JSON file in $HOME/.claude. To enable cross-machine session sharing then, the depot claude command executes the claude binary already on your machine and watches for changes to that session file on disk, saving any changes in it via the Depot API.

To resume a session, the depot CLI checks the name of the session being resumed, fetches its contents from the Depot API and writes them to disk, then executes the claude CLI.

In this way sessions can be named, shared, and resumed from anywhere!

Why we built this

Honestly? Because it was fun and we needed it ourselves. But also because we believe that AI coding agents are going to fundamentally change how we build software. They are already helping us write code faster, debug issues more effectively, and even design new features.

This makes our own use of Claude Code significantly more powerful. By allowing AI agents to maintain context across sessions, we can leverage their capabilities in a way that feels natural and integrated into our existing workflows.

We can have a Claude Code agent start working on a complex feature, hand it off to a human developer for review, and then have the agent pick up where it left off—all while maintaining full context of the conversation and code changes. That feature can then be submitted as a pull request, where other agents review it, and we can pick up the session of that review agent to address any feedback.

It's powerful stuff, and we think it will be a helpful tool to accelerate teams even more.

Conclusion

Depot is focused on building tools and services that accelerate delivery pipelines. We started by accelerating Docker image builds, then added our technology and expertise to GitHub Actions, and followed up with our own remote cache service, Depot Cache.

The combo of all these tools is helping teams build and ship software faster than ever before.

With the addition of Claude Code sessions, we're taking another step towards accelerating a new facet of the software delivery pipeline by making it faster & simpler for engineers and coding agents to interact on the same codebase.

We hope you enjoy this as much as we do. If you have any questions, feedback, or ideas for how we can make this even better, please hop into our Community Discord and let us know!

Self-hosted GitHub Actions runners aren't free

Kyle Galbraith — Tue, 03 Jun 2025 20:05:45 +0000

We released Depot GitHub Actions Runners a year ago. Our runners are anywhere between 3-10x faster with 10x faster caching as well. They come pre-configured with a lot of slick automatic add-ons, like RAM disks for faster disk access in jobs that need it, and automatic integration with our remote cache service for tools like Bazel, Gradle, Turborepo, and others.

Since launching, we've seen a lot of teams come to us from self-hosted GitHub Actions runners. Why? Because they're burned out from all of the operational overhead and complexity of it.

In this post, we highlight the problems and hidden costs with self-hosted GitHub Actions runners.

Do you like operational overhead?

Self-hosting GitHub Actions runners isn't the "set it and forget it" option that folks may think it is.

The truth is that self-hosted runners increase your operational overhead. The entire system will have to be monitored and maintained. This includes things like:

Maintaining AMIs: You'll need to regularly update your AMIs with the latest security patches and updates. GitHub publishes all of the runner image definitions for Packer. But it's a time-consuming and error-prone process.
Babysitting infrastructure: You can choose between running ephemeral runners or persistent runners. Ephemeral runners are easier to maintain but can be slower to start up and may not be as reliable. Persistent runners are faster but require more maintenance and can be more prone to failure. Either way, you'll need to monitor the underlying infrastructure and rapidly fix any issues that arise.
Debugging GitHub Runner API issues: The GitHub Runner API has bugs and inconsistencies that can cause problems with your self-hosted runners. This can be frustrating to debug and a total time drain.

The overhead of self-hosting GitHub Actions is significantly more cumbersome and time-consuming than people expect when getting started.

Free isn't actually free with GitHub Actions

The biggest misconception about self-hosting GitHub Actions runners is that it's free. This is simply not true. There are a lot of hidden costs associated with self-hosting GitHub Actions runners that can add up quickly.

While the cost per minute is $0, compared to the $0.008 per minute for GitHub-hosted runners, the reality is that self-hosting GitHub Actions runners can be significantly more expensive than you might think.

The most obvious cost is infrastructure. You'll need to pay for the instances, storage, network transfer, and any other resources you use to support the runners.

The infrastructure costs only become more problematic the larger your team and CI workloads grow over time.

Here are some common ways folks try to combat the rising costs:

Leverage spot instances or savings plans to reduce instance costs

Spot instances are the first thing folks grab to reduce the cost footprint of self-hosted runners. These can certainly save you money, but you have to accept the risk of losing runners midway through a job, and live with the developer experience around that. The spot market can also be hit and miss, depending on the instance types you're requesting.

When Spot fails, folks often turn to AWS Savings Plans. This is a great way to save money, but now you're financially locked into self-hosted, and you end up paying for unused capacity on nights and weekends.

Egress traffic

Egress traffic is another hidden cost that can add up quickly. If you're not careful with what you're sending out from the runner, you can spend truckloads of money on egress traffic.

The human cost

This is hard to calculate out of the gate, but it's there from the beginning and throughout the entire time you choose to self-host GitHub Actions. Humans always have to maintain and babysit the infrastructure and glue components for anything self-hosted.

The reality is that self-hosting GitHub Actions runners can be a time sink. You'll need to maintain the infrastructure, monitor the runners, and troubleshoot any issues. You'll have to become an expert in GitHub Actions.

This takes time away from the critical stuff like building features and shipping products for your customers and users.

This also, ultimately, impacts the developer experience. Spending time dealing with CI issues can be frustrating and downright demoralizing for developers who want to get their work done. The constant context switching between working on the code that matters and dealing with CI issues or quirks like queue time can be a huge drain on morale. This is especially true if you're using self-hosted runners that are unreliable or slow.

There are tools to help

There are solutions and workarounds. Before making the case for Depot, it's worth mentioning solutions that we have seen folks trying. While they have their own set of problems as well, they do provide advantages over going it entirely alone.

ARC

Also known as actions-runner-controller, ARC is a project from GitHub that allows you to run GitHub Actions runners inside of your own Kubernetes cluster. It is designed to orchestrate and scale based on the number of workflows running across your GitHub repository, organization, or enterprise connections.

Effectively, the runners scale up and down as containers inside of your K8s cluster. This comes with tradeoffs like needing to manage docker-in-docker setups, choose autoscaling policies, and deal with the complexity of Kubernetes (which some teams may have no problem with and others find more complicated).

It doesn't solve the human cost of maintaining the infrastructure, the cost of the infrastructure itself, and it's still prone to the instability and quirks from GitHub's API.

Self-hosted runners with the Terraform module: `github-aws-runners/terraform-aws-github-runner`

Formerly known as the philips-labs module, this strategy allows you to deploy self-hosted GitHub Actions runners on AWS using Terraform. Much like ARC, it provides the glue components in a self-contained module that you can deploy to your AWS account and get connected to your GitHub organization.

It runs on EC2 instances and defaults to creating ephemeral runners on spot instances. You can bring your own AMIs, and it has similar scaling policies to ARC that you can choose from. However, the infrastructure footprint is even larger than ARC in that there are a lot of glue pieces running in Lambda, EC2, and S3, and additional one-off things you have to configure, like a GitHub App to authenticate with GitHub. To date, it also only supports Linux and Windows runners.

It comes with all of the infrastructure and human costs of self-hosting GitHub Actions runners, but it provides a lot of flexibility and control over the infrastructure. The complexity and challenges of dealing with GitHub API outages, quirks, and optimizing for queue times still rests with you.

What about Depot?

Depot is a build acceleration platform for engineering teams that want to build faster without the operational overhead, complexity, and time drain of doing it all themselves. We focus on accelerating GitHub Actions by providing our own runners optimized for performance, reliability, and interfacing with GitHub Actions.

If you use Depot GitHub Actions runners on our hosted platform, you get all of the following benefits:

Up to 10x faster runners. We've optimized everything about the GitHub Actions runner to be faster. This includes things like fast I/O via ramdisks, faster caching, and automatic integration with our suite of remote cache supported tools like Bazel, Gradle, Turborepo, and others.
Single tenant runners. We launch a runner within 3-5 seconds, and it is dedicated to your job. When your job is done, we nuke the runner from orbit so it will never be used again.
No concurrency limits. Launch as many jobs as you'd like, we don't limit the number of jobs that you can run at a time.
Per-second billing. We don't do the bizarre GitHub thing where they round up your 30 second build to 1 minute to charge you for it. We charge by the minute but track by the second. So you get two 30 second builds before we charge you 1 minute.
No egress charges. We don't charge you for egress traffic. We don't charge you for ingress traffic either, but we don't have any limits on that.
Infrastructure is our concern, not yours. You don't have to think about your infrastructure costs, wasted capacity on nights and weekends, network or storage costs, or any other hidden costs that come with self-hosting GitHub Actions runners. We take care of all of that for you.
No human costs. You don't have to worry about maintaining the infrastructure, monitoring the runners, or troubleshooting any GitHub API issues. We take care of all of that for you.
No operational overhead. You don't have to worry about maintaining AMIs, debugging GitHub Runner API issues, or dealing with the complexities of self-hosting GitHub Actions runners. We take care of all of that for you.

You can self-host the data plane if you want

We also offer the ability to deploy the Depot data plane into your own environment. This means you get all the security & compliance benefits of self-hosted, leverage any compute commitments you have, and don't have any of the operations overhead of self-hosting GitHub Actions yourself. The pain and quirks of running GitHub Actions runners rest with us, and you get all of Depot's performance.

Conclusion

Every team and organization is different. Some teams are happy to self-host GitHub Actions runners and deal with the overhead and complexity of it. Others are not.

Either way, having all of the information and thinking through the complexities and challenges, allows you to at least go into the process with eyes wide open and avoid any surprises. We personally believe that all builds, no matter whether they're GitHub Actions, Docker, Bazel, Gradle, or whatever, should be fast, reliable, and as easy to get started with as possible. You should get exponentially faster builds and never have to think about it again.

That's why we built Depot. We think Depot is the best way to accelerate your builds and get the most out of your GitHub Actions runners with whatever flexibility you prefer, whether it's self-hosted or in our cloud. If you're interested in learning more, sign up for a 7-day free trial and give things a spin and if you have any questions or just want to bounce around ideas, feel free to hop in our Community Discord.

Faster Claude Code agents in GitHub Actions

Kyle Galbraith — Mon, 02 Jun 2025 17:09:58 +0000

It's been an exciting week as the news starts rolling in about bringing agentic coding out of the IDE and your own terminal and into other parts of our development workflows. Anthropic has brought my favorite agentic coding interaction, Claude Code, to GitHub Actions.

This means that we can now use Claude Code to write and run code by simply asking Claude in pull requests and issues. Behind the scenes, Claude Code runs inside of GitHub Actions workflows that we control, making it easier to automate tasks and improve our development processes.

In this post, we're going to explore how to use Claude Code in GitHub Actions and how to make it even faster and cheaper with the power of Depot GitHub Actions runners.

What is Claude Code?

Claude Code is an agentic coding tool that lives in your terminal and has a deep understanding of your codebase. It can help you code faster by giving it specific prompts to go work on in the background. Until now, it's been limited to running on your local machine as the only supported path. However, others have been working on bringing it to more places, like GitHub Actions.

But now, an official Claude Code GitHub App and an official GitHub Action make it even easier to use Claude Code in your GitHub Actions workflows. This is a game changer, as it allows you to use GitHub Actions as a background automation platform. Multiple Claude Code agents work on your codebase in parallel, processing different tasks simultaneously, significantly speeding up your development process.

Configuring Claude Code in GitHub Actions

To get started, it's recommended that you install the official GitHub app for Claude. This app allows you to tag @claude in your issues and PRs with a specific prompt to go work on.

Installing the app can actually be done directly from claude in your terminal:

claude /install-github-app

This installs the GitHub App to your repository, adds your Anthropic API key as a secret, and configures a workflow in .github/workflows/claude.yml that looks like this:

name: Claude Code

on:
  issue_comment:
    types: [created]
  pull_request_review_comment:
    types: [created]
  issues:
    types: [opened, assigned]
  pull_request_review:
    types: [submitted]

jobs:
  claude:
    if: |
      (github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
      (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
      (github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
      (github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')))
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 1

      - name: Run Claude Code
        id: claude
        uses: anthropics/claude-code-action@beta
        with:
          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}

Once you've installed this, you must merge in the pull request Claude opens before you can use it.

Current gotchas

Because this is a beta action, things are very much in flux and under development. There are a few things to note:

You will need to make sure you have the GitHub CLI installed (i.e., gh) before you run the install command.
If you get Couldn't install GitHub App: gh: Not Found (HTTP 404) when running /install-github-app, it means that your local gh auth token doesn't have workflow permissions (i.e., it can't create the new claude.yml workflow). You can fix this by running gh auth refresh -h github.com -s workflow and then re-running the /install-github-app command.
The default tools in the action are pretty tightly scoped. They are listed below, and anything you add to allowed_tools will be appended to the end of this list:

Edit
Glob
Grep
LS
Read
Write
mcp__github_file_ops__commit_files
mcp__github_file_ops__delete_files
mcp__github__update_issue_comment

Having Claude Code open pull requests after completing tasks

The initial example workflows are heavily focused on running Claude Code to tag issues, add comments to issues, and add pull request reviews. This is great, but it doesn't really take advantage of the full power of Claude Code.

One of the things I'm most excited about is the ability to have Claude Code open pull requests after completing tasks. This is a great way to automate getting code changes reviewed and merged into your codebase.

To do this, you must update the permissions section of the workflow to include pull-request: write, so Claude can open pull requests. You must also include mcp__github__create_pull_request in the allowed_tools input.

Here is what the Claude Code workflow looks like with these changes:

name: Claude Code

on:
  issue_comment:
    types: [created]
  pull_request_review_comment:
    types: [created]
  issues:
    types: [opened, assigned]
  pull_request_review:
    types: [submitted]

jobs:
  claude:
    if: |
      (github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
      (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
      (github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
      (github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')))
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
+      pull-requests: write
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 1

      - name: Run Claude Code
        id: claude
        uses: anthropics/claude-code-action@beta
        with:
          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
+          allowed_tools: 'mcp__github__create_pull_request'

With that change, you can now tag @claude in your issues to have it open a pull request for the changes it made. Here is an example from our open-source docs repository, depot/docs:

Speeding up Claude Code with Depot GitHub Actions runners

What's particularly cool about Anthropic making agentic coding backed by GitHub Actions is that it allows you to control where you want the Claude Code agent to run. This means that you can use Depot GitHub Actions runners to run Claude Code in a much faster environment than the default GitHub-hosted runners.

We've built out our GitHub Actions runners to have faster CPUs, ramdisks with faster IO, and network speeds that make caching exponentially faster. They are also half the cost of GitHub-hosted runners, meaning you can run your Claude Code agents for a fraction of the cost.

To move a Claude Code agent to run on Depot GitHub Actions runners, just update the runs-on section of the workflow to use depot-ubuntu-latest instead of ubuntu-latest. Here is what the updated workflow looks like:

name: Claude Code

on:
  issue_comment:
    types: [created]
  pull_request_review_comment:
    types: [created]
  issues:
    types: [opened, assigned]
  pull_request_review:
    types: [submitted]

jobs:
  claude:
    if: |
      (github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
      (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
      (github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
      (github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')))
-    runs-on: ubuntu-latest
+    runs-on: depot-ubuntu-latest
    permissions:
      contents: read
      id-token: write
      pull-requests: write
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 1

      - name: Run Claude Code
        id: claude
        uses: anthropics/claude-code-action@beta
        with:
          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
          allowed_tools: 'mcp__github__create_pull_request'

Making this switch will make your Claude workflow significantly cheaper and faster. If you have a Claude Code agent that, on average, takes 5 minutes, it will cost you about $0.04/session on GitHub-hosted runners. That same session on Depot GitHub Actions runners will cost you $0.02/session on our faster default runners.

But you can also tune this further with any of our runner types. For example, we have a small runner with only 2 cores, 2 GB of RAM, and 100 GB of disk at just $0.002/minute. This is an excellent option for running Claude Code agents doing small tasks that don't require a lot of resources. This runner type costs $0.01/session on Depot GitHub Actions runners.

Conclusion

So there you have it - Claude Code on Depot runners is a powerful combo. Let's break down why this matters:

Save money: Running Claude on Depot runners costs half what you'd pay on GitHub-hosted runners. If you're pinching pennies, our small runners drop the price to just $0.01 per session for simple tasks.
Speed things up: Depot's faster CPUs, RAM disks, and network make everything run smoother. Your AI agents will thank you.
Get more done: With Claude agents handling routine stuff in parallel, your team can focus on the interesting problems instead of the boring ones.
Super simple setup: You're good to go with a few tweaks to your YAML file. No complex configuration needed.

This whole "AI agents in CI" thing is still pretty new, but is clearly where some portion of development is heading. The coolest part is how easy it is to set up - install the Claude GitHub App, add those permissions for PR creation, and switch to depot-ubuntu-latest. That's literally it.

I'm really excited to see what people build with this setup. Having multiple Claude agents working on different parts of your codebase while running on faster and cheaper infrastructure feels like a glimpse into the future of development.

Want to try it yourself? Sign up for a 7-day free trial of Depot and get access to all our build performance for GitHub Actions, Docker, Bazel, Turborepo, Gradle, and much more.

Kyle Galbraith

CEO & Co-founder of Depot

Platform Engineer who despises slow builds turned founder. Expat living in 🇫🇷

BuildKit in depth: Docker's build engine explained

Kyle Galbraith — Thu, 01 Feb 2024 12:28:45 +0000

This article explains how BuildKit works in depth, why it's faster than Docker's previous build engine, and what it looks like under the hood.

BuildKit is Docker's new default build engine as of Docker Engine v23.0.

Despite BuildKit being used by millions of developers, the documentation out there is relatively sparse. This has led to it being seen as a bit of a black box. At Depot, we've been working with (and reverse engineering) BuildKit for years and have developed a deep understanding of it throughout this process. Now we understand the inner workings, we have a better appreciation for it and want to share that knowledge with you.

In this article, we explain how BuildKit works under the hood, covering everything from frontends and backends to LLB (low-level build) and DAGs (directed acyclic graphs). We help to demystify BuildKit and explain why it's such an improvement over Docker's original build engine.

What is BuildKit?

BuildKit is a build engine that takes a configuration file (such as a Dockerfile) and converts it into a built artifact (such as a Docker image). It's faster than Docker's original build engine due to its ability to optimize your build by parallelizing build steps whenever possible and through more advanced layer caching capabilities.

BuildKit speeds up Docker builds with parallelization

A Dockerfile can consist of build stages, each of which can contain one or more steps. BuildKit can determine the dependencies between each stage in the build process. If two stages can be run in parallel, they will be. Stages are a great way to break your Docker image build up into parallelizable steps — for example, you could install your dependencies, build your application at the same time, and then combine the two to form your final image.

To take advantage of parallelization, you must rewrite your Dockerfile to use multi-stage builds. A _stage _is a section of your Dockerfile that starts with a FROM statement and continues until you reach another FROM statement. Stages can be run in parallel, so by this mechanism, the steps in one stage can run in parallel with the steps in another.

It's worth noting that the steps within a stage run in a linear order, but the order in which stages run may not be linear. To determine the order in which stages will be run, BuildKit detects the name of each stage — which, for a Dockerfile, is the word after the as keyword in a FROM statement.

To determine the stage that another stage depends on, we look at the word _after _the FROM keyword. In the example below, FROM docker-image as stage1 means that the stage1 stage depends on the Docker image from Docker Hub, and FROM stage1 as stage2 means that the stage2 stage depends on the stage1 stage. It's possible to chain many stages together in this way.

FROM docker-image as stage1
RUN command1

FROM stage1 as stage2
RUN command2

FROM stage2 as stage3
RUN command3

It's also possible to have multiple stages depend on one stage:

FROM docker-image as parent
…
FROM parent as child1
…
FROM parent as child2

BuildKit is able to evaluate the structure of these FROMstatements and work out the dependency tree between the steps in each stage.

Optimize your Dockerfile to take advantage of parallelization

Rewriting your Dockerfile to use multi-stage builds will allow you to take advantage of the speed improvements that BuildKit brings to Docker.

In the example below you can see an unoptimized file with a single stage and an optimized file with two stages: one named build, and another which is unnamed (this is the naming convention for the final stage in a Dockerfile).

Optimizing this Dockerfile allows steps such as enabling Corepack to run in parallel with copying the package.json
file and the pnpm-lock.yaml file into the /app directory.

Once your Dockerfile has been optimized to run in multiple stages, BuildKit can run them in parallel. Below you can see the difference between using BuildKit to run multiple stages in parallel (for building and deploying a Node app) and running all steps sequentially (without multi-stage builds). This Node app deployment example will be used throughout this article, and the Dockerfiles — both optimized for BuildKit (multi-stage) and unoptimized (basic) — are available on our GitHub.

If you use BuildKit to parallelize your stages, your build will complete much faster.

BuildKit speeds up Docker builds with layer caching

BuildKit is also able to improve build performance through clever use of layer caching. With layer caching, each step of your Dockerfile (such as RUN, COPY, and ADD) is cached individually, as a separate reusable layer.

Often, individual layers can be reused, as the results of a build step can be retrieved from the cache rather than rebuilt every time. This eliminates many steps from the build process and often dramatically increases overall build performance.

The hierarchy of layers in BuildKit's layer cache is a tree structure, so if one build step has changed between builds, that build step plus all its child steps in the hierarchy must be rebuilt. With traditional single-stage builds, every single step depends on the previous step, so it can be immensely frustrating if you have a RUN statement that invalidates the cache in an early part of your Dockerfile — because all subsequent statements must be recomputed any time that statement runs. The order of your statements in a Dockerfile has a major impact on optimizing your build to leverage caching.

However, if you've optimized your Dockerfile for BuildKit, used multi-stage builds, and ordered your statements to maximize cache hits, you can reuse previous build results much more frequently.

BuildKit under the hood

"BuildKit builds are based on a binary intermediate format called LLB that is used for defining the dependency graph for processes running part of your build. tl;dr: LLB is to Dockerfile what LLVM IR is to C."

— BuildKit's README file.

To truly understand how BuildKit works, let's unpack this statement. BuildKit has taken inspiration from compiler designers by creating an intermediate representation between the input and the output to its system. In compiler design, an intermediate representation is a data structure or some human-readable code (such as assembly language) that sits between the source code input and the machine code output. This intermediate representation is later converted into different types of machine code for each different machine the code needs to run on.

BuildKit uses this same principle by inserting an intermediate representation between the Dockerfile input and the final Docker image. BuildKit's intermediate representation is known as a low-level build (LLB), which is a directed acyclic graph (DAG) data structure that sits at the heart of BuildKit's information flow.

The flow of information through BuildKit: frontends, backends and LLB

Continuing with the compiler comparison, BuildKit also uses the concept of frontends and backends.

The frontend is the part of BuildKit that takes the input (usually a Dockerfile) and converts it to LLB. BuildKit has frontends for a variety of different inputs including Nix, HLB, and Bass, all of which take different inputs but build Docker images, and CargoWharf, which is used to build something else entirely (a Rust project). This shows the versatility BuildKit has to build many different types of artifacts, even though the most common use currently is building Docker images from Dockerfiles.

The backend takes the LLB as an input and converts it into a build artifact (such as a Docker image) for the machine architecture that you've specified. It builds the artifact by using a container runtime — either runc or containerd (which uses runc under the hood anyway).

BuildKit's frontend acts as an interface between the input (Dockerfile) and the LLB. The backend is the interface
between the LLB and the output (Docker image).

BuildKit's LLB

We've referred to the LLB a few times so far — but what exactly is it?

It's a directed acyclic graph (DAG) data structure, which is a special type of graph data structure. In a DAG, each event is represented as a node with arrows that flow in a particular direction, hence the word “directed.” Arrows start at a parent node and end on a child node. Child nodes are only allowed to execute after _all _parent nodes have finished executing.

There can be no loops in a DAG, hence the word “acyclic.” This is necessary for modeling build steps, as if a build process allowed loops, the process would never complete because two steps would require each other to finish before each one starts!

BuildKit's LLB DAG is used to represent which build steps depend on each other and the order in which everything needs to happen. This ensures that certain steps don't occur before other steps are completed (like installing a package before downloading it).

In the case of Docker builds, BuildKit uses its Docker frontend to create the LLB from the Dockerfile. For example, this Dockerfile for building and deploying a Node app would create the following LLB DAG:

In this LLB DAG, each node represents an operation that can happen. Each LLB operation can take one or more
filesystems as its input and output one or more filesystems.

To help you understand more about the LLB operations that your Dockerfile would translate to, we built a free tool that converts any given Dockerfile into LLB through a real-time editor. Our Dockerfile Explorer is easy to use — simply paste your Dockerfile into the box on the left and then view the LLB operations on the right.

Our Node Dockerfile creates a number of LLB operations, the first three of which can be viewed below. Each operation has a type such as SourceOp or ExecOp, a unique identifier in the form of a hash value, and some extra data like the environment and the commands to be run. The hash values indicate the dependencies between the operations. For example, the first ExecOp operation has a hash value of 0534a47f, and the second ExecOp operation takes as its input an operation with a hash of the same value (0534a47f). This shows that these two operations are directly linked on the LLB DAG.

The different BuildKit LLB operations explained

SourceOp

This loads source files or images from a source location, such as DockerHub, a Git repository, or your local build context.

All SourceOp operations that originated from a Dockerfile have been generated from Dockerfile FROM statements.

ExecOp

ExecOp always executes a command. It's equivalent to Dockerfile RUN statements.

FileOp

This is for operations that relate to files or directories, including Dockerfile statements such as ADD (add a file or directory), COPY (copy a file or directory), or WORKDIR (set the working directory of your Docker container).

It's possible to use this operation to copy the output of other steps in different stages into a single step. Using our example Dockerfile, the COPY --from statement copies some of the resources from the output of the previous build stage into the final stage.

FROM node:20
…
COPY --from=build /appbuild /app/build

We can use the Dockerfile Explorer to see how BuildKit deals with this — it takes the output of the final step in each stage and adds them together.

MergeOp

MergeOp allows you to merge multiple inputs into a single flat layer (and is the underlying mechanism behind Docker's COPY --link).

DiffOp

This is a way of calculating the difference between two inputs and producing a single output with the difference represented as a new layer, which you might then want to merge into another layer using MergeOp.

However, this operation is currently not available for the Dockerfile frontend.

BuildOp

This is an experimental operation that implements nested LLB builds (for example, running one LLB build that produces another dynamic LLB).

This operation is also unavailable for the Docker frontend.

BuildKit speeds up your Docker builds using its LLB DAG

Although BuildKit can take multiple frontends, the Dockerfile frontend is by far the most popular. BuildKit uses its Dockerfile frontend to convert statements from your Dockerfile into a DAG of LLB operations — including SourceOp, ExecOp and FileOp — and then it uses that LLB format to build an artifact, like a Docker image, for the specified architectures that were requested.

At Depot, we've taken what was already great about BuildKit and further optimized it to build Docker images up to 20x faster on cloud builders with persistent caching. We've developed our own drop-in replacement CLI, depot build, that can be used to replace your existing docker build wherever you're building images today. Sign up today for our 7-day free trial and try it out for yourself.

Build Docker images faster using build cache

Kyle Galbraith — Sun, 07 Jan 2024 11:07:04 +0000

When working with Docker, the faster we can build an image, the quicker our development workflows and deployment pipelines can be. Docker's build cache, also known as the layer cache, is a powerful tool that can significantly speed up an image build when it can be tapped into across builds. In this post, we'll explore how Docker's build cache works and share strategies for using it effectively to optimize your Dockerfiles & image builds.

Understanding Docker Build Cache

Before we dive into optimizations, let's understand how Docker's build cache works. Each instruction in a Dockerfile creates a layer in the final image. Think of these layers as building blocks, each adding new content on top of the previous layers.

When a layer changes, Docker invalidates the cache for that layer and all subsequent layers, requiring them to be rebuilt. For instance, if you modify a source file in your project, the COPY command will have to run again to reflect those changes in the image, leading to cache invalidation.

Tips for efficiently using the Docker build cache

The more we can avoid cache invalidation, or the later we can have our cache invalidate, the faster our Docker image builds can be. Let's explore some strategies for doing just that.

Order your layers wisely

Ordering our commands in a Dockerfile can play a huge role in leveraging the Docker layer cache and how often we invalidate it. Let's take a look at an example:

FROM node:20

WORKDIR /app
COPY . .
RUN npm install
RUN npm build

This is an inefficient Dockerfile. The COPY command will invalidate the cache for all subsequent layers whenever a file changes in our project, forcing our npm install and npm build commands to execute even if none of our dependencies changed. We can improve this by being more thoughtful about when we copy in our source code and install our dependencies:

FROM node:20

WORKDIR /app
COPY package.json package-lock.json /app/
RUN npm install
COPY . .
RUN npm build

We've moved our source code copy to after our npm install command. We copy in our package.json and package-lock.json to install our dependencies. We then copy in our source code and execute our npm build.

This is a small change that can have a significant impact on build time. Now, instead of every source code change forcing us to reinstall our dependencies, we only have to do so when our package.json or package-lock.json files change.

Keep your layers small and focused

The less stuff in our build, the faster our Docker image build can be. By keeping our layers small and focused, we can keep our image smaller, cache smaller, and reduce the number of things that can invalidate the cache.

We've written other posts about keeping Docker images small that are worth reading in conjunction with this post:

Here are a few tips and tricks that are relevant to efficiently using the Docker build cache.

Avoid copying files that are not needed

A common mistake we see is copying in files not needed in the final image. For instance, if we are building a Node.js application, we may inadvertently copy in our node_modules directory when, in fact, we are running npm install again in our Dockerfile.

This is a waste of time and space. A good guiding principle is to only copy in the files and directories we know are needed in our final image. So, if we take our earlier example:

FROM node:20

WORKDIR /app
COPY package.json package-lock.json /app/
RUN npm install
COPY . .
RUN npm build

In addition, our docker build is invoked with the full context of our

.git/
node_modules/
app/
  index.js
  package.json
  package-lock.json
README.md
Dockerfile

Our COPY command is copying in our entire build context; we can easily visualize our build context with our debug build context feature. In this example, we are copying in many unnecessary files and directories like .git, node_modules, and our README.

It is far better to be more specific with our COPY command:

COPY ./app /app

Now, we are only copying the app folder into our build and final image.

Use `.dockerignore` to exclude files and directories

Sometimes, knowing exactly what files and directories are needed in our final image can be tricky. So we can use a .dockerignore file to explicitly define the files we know should be excluded. For our example above, we could create a .dockerignore file with the following contents:

.git
node_modules
README.md

Avoid unnecessary dependencies from package managers

We commonly install dependencies into our images from package managers like npm, pip, apt, apk, etc. for Node, Python, Debian, and Alpine images. It's important to be mindful of what dependencies we are installing and if they are needed in our final image. There are tricks we can sometimes use like --no-install-recommends to avoid package managers installing additional dependencies that are not needed.

Sometimes dependencies are only needed for building our application, but not for running it; in those cases, we can leverage multi-stage builds to avoid having them in our final image.

Leverage the `RUN` cache for finer-grained control

Also known as BuildKit cache mounts. This specialized cache allows us to do more fine-grained caching across builds. Here is an example of the RUN cache in action with a Ubuntu image:

FROM ubuntu
RUN rm -f /etc/apt/apt.conf.d/docker-clean
RUN \
    --mount=type=cache,target=/var/cache/apt \
    apt update && apt-get --no-install-recommends install -y gcc

Reduce the total number of layers

The more layers we have in our image, the more layers we have to rebuild when the cache is invalidated, and the more opportunities for the cache to be invalidated. Below are some handy tips for reducing the number of layers in our image.

Combine multiple `RUN` commands where possible

The number one Dockerfile lint issue we've detected in Depot is multiple consecutive run instructions. The more we combine RUN commands, the fewer layers we will have in our image. For example, if we had a Dockerfile like this:

RUN some-command-that-creates-big-file
RUN some-command-that-removes-big-file

This creates an unnecessary layer in our image, the layer that initially downloaded the big file. We can combine these two commands into a single RUN command:

RUN some-command-that-creates-big-file && \
    some-command-that-removes-big-file

This downloads the big file and removes it all within a single layer, saving us an intermediate layer with the large file present.

Be thoughtful about base images

The base image we use can significantly impact the number of layers in our image. Choosing an image closely related to the application or service we are containerizing can avoid recreating unnecessary layers. It can also help us stay updated with security patches and other updates that a particular framework or tool may have.

It's also worth considering using smaller base images to improve build performance and reduce final image size. For instance, if we are building a Node.js application, we may be able to use the node:alpine image instead of the node image. This can reduce the number of layers and final image size in our image.

Take advantage of multi-stage builds

A multi-stage build allows us to have multiple FROM instructions in our Dockerfile. This can be useful for reducing the number of layers in our final image. For instance, if we are building a Node.js application, we may have a Dockerfile like this:

FROM node:20-alpine AS build

WORKDIR /app
COPY package.json yarn.lock tsconfig.json ./
RUN yarn install --immutable
COPY src/ ./src/
RUN yarn build

FROM node:20-alpine
WORKDIR /app
COPY --from=build /app/node_modules /app/node_modules
COPY --from=build /app/dist /app/dist
CMD ["node", "./dist/index.js"]

Conclusion

The Docker build cache, when leveraged correctly, can significantly speed up our Docker image builds. By being mindful of how we order our layers, what we copy into our image, and how we structure our Dockerfiles, we can make our builds faster and more efficient.

Using the Docker build cache efficiently can speed up your internal development, CI/CD pipelines, and deployments. With Depot, we had another speed boost to this problem by persisting your cache automatically across a distributed storage cluster that your entire team and CI workflows can share. With even faster caching and native Intel & Arm CPUs for zero-emulation builds, we've seen Depot folks getting 30x faster Docker image builds.

If you want to learn more about how Depot can help you optimize your Docker image builds, sign up for our free trial.

How to build multi-platform Docker images in GitHub Actions

Kyle Galbraith — Mon, 30 Oct 2023 10:02:00 +0000

In this post, we will focus on building multi-platform Docker images, as well as Arm images, in GitHub Actions.

Multi-platform images and Arm support

By default, Docker images are built for the architecture of the machine running the build. If you build an image on an Intel machine, the image will be built for Intel. If you build an image on an Arm machine, the image will be built for Arm.

If you want to build an image for a different architecture than the machine you are building on, you can specify the --platform flag during a Docker build. For example, if you are building on an Intel machine and want to build an Arm image, you can specify --platform linux/arm64.

docker build --platform linux/arm64 .

A Docker image can also be built for multiple architectures simultaneously. This produces what is often referred to as a multi-platform or multi-architecture image. If you want to build a multi-platform Docker image for both Intel and Arm, you can specify multiple platforms in the --platform flag.

docker build --platform linux/arm64,linux/amd64 .

A multi-platform Docker image build triggers two builds, one for each architecture, and produces a single image that supports both platforms. But, to build that image, one of the architectures must be emulated using qemu emulation. If this is a multi-platform image built in CI, like GitHub Actions, the Arm portion (i.e., linux/arm64) will be emulated.

Alternatively, there is the option to configure docker buildx build to use multiple builders, one for each platform. This method removes the need to emulate the non-host machine architecture. But, in exchange, you have to run your own native builders. For more details, we have a blog post on running your own builder instances.

Building multi-platform Docker images in GitHub Actions

This section assumes you know the basics of building Docker images in GitHub Actions. If you are new to building Docker images in GitHub Actions, we have a blog post that covers the basics of building Docker images in GitHub Actions.

To build a multi-platform Docker image in GitHub Actions, we must configure QEMU emulation and buildx in our workflow.

name: Build multi-platform Docker image

on:
  push: {}

jobs:
  build-with-docker:
    name: Build multi-platform Docker image
    runs-on: ubuntu-20.04
    steps:
      - uses: actions/checkout@v3
      - uses: docker/setup-qemu-action@v3
      - uses: docker/setup-buildx-action@v3
      - uses: docker/build-push-action@v5
        with:
          context: .
          platforms: linux/amd64,linux/arm64
          cache-from: type=gha
          cache-to: type=gha,mode=max

Three steps in our workflow are worth noting:

docker/setup-qemu-action configures QEMU emulation for the Arm portion of our multi-platform Docker image build. This is required for multi-platform Docker image builds in GitHub Actions, as the hosted runners are Intel machines.
docker/setup-buildx-action configures buildx for our workflow. It's required for multi-platform Docker image builds in GitHub Actions that use docker/build-push-action.
docker/build-push-action builds and pushes our Docker image. We specify the platforms argument to build our image for both Intel and Arm architectures. If we wanted to push our image to a registry, we could add an additional step above docker/build-push-action to login to our registry and then specify the push argument. We specify the cache-from and cache-to parameters to store the Docker layer cache via the GitHub Cache API.

- uses: docker/login-action@v3
  with:
    username: ${{ secrets.DOCKERHUB_USERNAME }}
    password: ${{ secrets.DOCKERHUB_TOKEN }}
- uses: docker/build-push-action@v5
  with:
    context: .
    platforms: linux/amd64,linux/arm64
    push: true
    tags: <your-repo>:<your-tag>
    cache-from: type=gha
    cache-to: type=gha,mode=max

If you wanted to build a Docker image for only Arm, you would change the platforms argument to just linux/arm64 but keep the rest of the workflow to use QEMU emulation.

This is a basic example of building multi-platform Docker images in GitHub Actions with QEMU emulation. It's functional, but it has limitations that are worth noting:

The Arm portion of the build is emulated using QEMU. We've seen up to 30x speedups when building on native Arm machines vs emulating Arm in CI via our benchmarks like Temporal's multi-platform build.
The GitHub Cache API only supports a maximum size of 10 GB for the entire repository, each architecture will have to share this 10 GB limit.
Loading and saving cache over networks is slow, meaning the loading and saving could negate any performance benefits of using the cached layers for simple image builds
The cache is locked to GitHub Actions and can't be used in other systems or on local machines

An alternative to emulation is to run your own GitHub Action runners on Arm instances. This comes with a significant increase in infrastructure and complexity you must manage.

A managed solution

We built Depot to eliminate the pain of emulation and other limitations above, not only in GitHub Actions but in all CI providers.

Depot is a remote container build service that runs an optimized version of BuildKit on native cloud instances for Intel and Arm. We manage your Docker layer cache on fast NVMe SSDs that make your cache instantly available across builds. The layer cache can be used from your CI build in GitHub Actions and your local machine when you use depot build.

Each build runs on a dedicated single-tenant BuildKit machine, using native Intel or Arm CPUs. Each instance includes 16 CPUs, 32GB memory, and the persistent layer cache can be expanded up to 500 GB.

Depot effectively provides on-demand access to one or more remote BuildKit builders and automatically routes each build platform to the machine best suited to build for that platform.

If you are interested in trying out Depot in your GitHub Actions workflow, check out our GitHub Actions integration guide and get started with Depot.

Faster Docker builds for Arm without emulation

Kyle Galbraith — Tue, 17 Oct 2023 08:40:16 +0000

Building a Docker image for the Arm architecture is loaded with inefficiencies. With the adoption of Arm-based devices like M1 / M2 MacBooks, and the growing popularity of Arm-based servers like AWS Graviton, it is becoming more important to build Arm and multi-platform containers. It can be a challenge to build these containers efficiently.

Emulation is painfully slow

Today, most people build Arm images using emulation. Why? Because emulation is built into Docker and buildx out of the box. By passing the --platform linux/arm64 flag to docker buildx build, Docker will use emulation to build the image for Arm if the host architecture is Intel.

docker buildx build --platform linux/arm64 -t org/repo:tag .

Or, to build an image for multiple architectures, also know as a multi-platform image, you can pass multiple platforms. Here we tell it to build an image for both Intel & Arm in parallel:

docker buildx build --platform linux/arm64,linux/amd64 -t org/repo:tag .

Side note on multi-platform images

Building multi-architecture Docker images like the example above results in one half of the build happening on the native host platform and the other half happening in an emulated platform. But multiple container images aren't produced. It's one image that contains a image manifest that states which platforms this Docker container image can run on. You can use tools like docker buildx imagetools or docker manifest to actually inspect these manifest (note: docker manifest is still an experimental feature).

So if you were to docker run --rm org/repo:tag and you were on an Arm server, the daemon will ask the Docker registry for the image manifest and select the image with a matching platform to use for the launched container.

Emulation is a logical place to start as that is what Docker Desktop supports out of the box when installing Docker. But it's slow, really slow, and it gets exponentially worse for more complex applications:

Mastodon's emulated builds take around 55 minutes to complete.
Temporal's emulated builds take as many as 80 minutes to complete!

The benchmarks shown above are happening in GitHub Actions with Intel runners and asking for multi-platform images. So, when we need to build the Arm image (linux/arm64), we have to use emulation during the Docker build of that architecture.

Building Docker images for Arm natively

You can use other tricks like cross-compilation in your Dockerfile to try and work around the slowness of emulation. But it's not a great experience. You have to get crafty with multi-stage builds and maintain cross-compilation toolchains.

The far better option is to build Docker images for Arm natively by running the builds on real Arm hardware.

Unfortunately, this isn't a great experience if you're trying to do it yourself. You must run your own builder instances, maintain them, keep them up to date, and ensure they are always available. It's a lot of DevOps work.

The fastest way to build Docker images for Arm

With Depot, you get native Intel & Arm builders right out of the box—no emulation, no complicated cross-compilation, and no running your own builders. Just fast builds on native hardware.

It's as simple as installing our depot CLI and running our configure-docker command:

depot configure-docker
docker buildx build --platform linux/amd64,linux/arm64 -t org/repo:tag .
[+] Building 0.9s (32/32) FINISHED                                                                                                                                                     docker-container:depot_456
 => [depot] build: https://depot.dev/orgs/123/projects/456/builds/dw0n0x4b4g                                                                                                          0.0s
 => [depot] build: https://depot.dev/orgs/123/projects/456/builds/ttcb3q4ss5                                                                                                          0.0s
 => [depot] launching arm64 machine                                                                                                                                                                                                 0.4s
 => [depot] launching amd64 machine                                                                                                                                                                                                 0.3s
 => [depot] connecting to arm64 machine                                                                                                                                                                                             0.1s
 => [depot] connecting to amd64 machine
 => [internal] load .dockerignore                                                                                                                                                                                                   0.1s
 => => transferring context: 116B                                                                                                                                                                                                   0.1s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                0.1s
 => => transferring dockerfile: 435B                                                                                                                                                                                                0.1s
 => [internal] load .dockerignore                                                                                                                                                                                                   0.1s
 => => transferring context: 116B                                                                                                                                                                                                   0.1s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                0.1s
 => => transferring dockerfile: 435B                                                                                                                                                                                                0.1s
 => [linux/amd64 internal] load metadata for docker.io/library/node:16-alpine                                                                                                                                                       0.5s
 => [linux/arm64 internal] load metadata for docker.io/library/node:16-alpine

With a Depot project configured, you can now build native multi-platform Docker images for Arm without the pain of emulation, cross-compilation, or running your own builders.

The results of building Docker images for Arm with Depot speak for themselves:

The Mastodon benchmark went from 55 minutes with emulation, down to 3 minutes with native CPUs.
The Temporal benchmark went from 80 minutes with emulation, down to 2 minutes with native CPUs.

Try it out

Depot launches on-demand builders for both Intel & Arm with 16 CPUs, 32GB of memory, and up to 500GB of persistent cache storage that is shared across all your builds and teammates.

If you're looking for the fastest way to build Docker images for Arm, sign up for Depot and try it yourself.

The complete guide to getting started with building Docker images

Kyle Galbraith — Wed, 27 Sep 2023 08:09:42 +0000

Packaging applications and services into containers has been around for a while. Docker was a technology that came out of another idea called DotCloud in 2013. So, even the Docker containers we know and love today are a decade old. But it's important to remember that the underlying technology of a Docker container is even older.

What is Docker?

The underlying technologies backing Docker containers are low-level Linux kernel components like cgroups, namespaces, and a union-capable file system like OverlayFS. These technologies are what allow Docker containers to be so lightweight and portable. Combined, they allow a single Linux VM to run multiple containers.

Installing Docker

To get started with Docker, you need to install it first. Depending on what you're running containers on, there are multiple ways to do that. Here are three Docker installation guides:

Each Docker installation guide ultimately installs Docker Desktop and configures the Docker engine on the given operating system.

What is a Docker image vs. a Docker container?

When getting started with Docker, a common question is, what is the difference between a Docker image and a Docker container? A Docker image is a series of layers stacked on each other that form the dependencies and source code needed to run your application. During a Docker image build, all those layers get packaged together to produce a final Docker image.

A Docker container is a runnable instance of a Docker image. You can run multiple containers with the same image to run multiple copies of your application or service.

A Docker image is the source code and dependencies packaged together, and the Docker container is the running instance of that image.

So, what is a Dockerfile?

A Dockerfile is a file that contains instructions for how to build a Docker image. It's a text file that includes a series of instructions that are executed in order to build a Docker image. The Dockerfile is the recipe that produces our Docker image.

As we will see in a minute, a Dockerfile is executed from top to bottom during a given docker image build. Instructions are invoked in order, and each instruction generally maps to an image layer. Those layers, stacked on top of each other one after the other, form our final Docker image.

Dockerfile instructions and what they do

Several different instructions can be used in a Dockerfile. Each instruction is a command that is executed during the build process. The most common instructions are:

Instruction	What it does
`FROM`	Defines a new build stage and sets the base image for that stage
`RUN`	Executes any commands it is given in a new layer on top of the current image that has been built up to that point
`COPY`	Copies the contents from a source directory to the filesystem at the path passed in to a new layer in the image
`ADD`	A more advanced version of `COPY` that supports things like local tar extraction and remote URLs
`CMD`	Defines the defaults for when this image is launched as a container, usually includes an executable to invoke, but that's not required
`ENTRYPOINT`	Configures the executables or commands that will run once the container is initialized
`USER`	Sets the user that the container is run under, often used to run containers as non-root
`LABEL`	Adds key-value labels to the image being built; note that labels are passed down from base images
`ARG`	Define build-time only variables that can be used during the Docker image build
`ENV`	Sets environment variables from within the Docker image that can be used during the build process or when the container is run
`EXPOSE`	Defines a port that the container will listen on when the image is run as a container
`WORKDIR`	Sets the working directory for the commands that follow it
`VOLUME`	Creates a mount point with a specific name that is bound to a mounted volume from the underlying host or another container

Building a Docker image

Now that we have a solid foundation, we can build a Docker image and see how a Docker image build works with a sample application. For this example, we will use an example Fastify API that uses TypeScript and pnpm for package management. The example project can be git clone on our GitHub.

After cloning the project, we can run pnpm install and pnpm build from the root of the example to install our dependencies and build our TypeScript source code.

pnpm install && pnpm build

After building the code, we should see a dist directory with our compiled code.

ls dist/
  index.js
  index.js.map

We can now run the example API outside a Docker container to see if it works as expected. We use curl to hit a /health endpoint on that API that returns a simple JSON response.

pnpm start
curl localhost:3000/health
{"alive":true}

Keeping our Docker image size down

Before jumping straight into writing a Dockerfile for our example project, we need to start with a .dockerignore first.

A .dockerignore file tells the build what files and directories to ignore and exclude from the Docker build context when we run docker build. Our project git repositories often contain many files and folders that we don't need in our final image or the build context itself.

node_modules
Dockerfile
.git
.gitignore
dist/**
README.md

This .dockerignore file tells the Docker build to ignore all of these files and directories during the build. These files will be excluded from the Docker build context and thus won't be copied via any COPY or ADD instructions.

Writing a Dockerfile

Now that we have a .dockerignore file, we can write our Dockerfile. For a more advanced Dockerfile that is highly optimized, we can use our best-practice Dockerfile for Node.js & pnpm. The optimized Dockerfile uses multi-stage builds, optimized Docker layer caching, and BuildKit cache mounts to optimize the docker build image process.

Simple example Dockerfile

For this post, we will use a more straightforward example Dockerfile to walk through core concepts.

First, we need a base image for our Docker image to be built from. Since our example project is in Node, an official Node base image like node:20 is an excellent place to start.

FROM node:20

Once we have a base image, we can install our dependencies and build our application. We first enable corepack, an experimental tool for managing versions of package managers. We then copy in our package.json and pnpm-lock.yaml files and install our dependencies. Finally, we copy in our source code and build our application.

RUN corepack enable

COPY package.json pnpm-lock.yaml ./
RUN pnpm install --frozen-lockfile
COPY . .
RUN pnpm build

The final thing to do is to set our CMD instruction, which tells the container what to run when it's launched. In our case, we want to run our compiled index.js file, our API.

ENV NODE_ENV production
CMD ["node", "./dist/index.js"]

Note: This Dockerfile is not optimized for size or build performance. It's meant to be an example to follow along with. For an optimized version that uses multi-stage builds and Docker layer caching, see our best-practice Dockerfile for Node.js & pnpm.

Building our Docker image

Now that we have a Dockerfile, we can start building our Docker image with docker build. We can run this command from the root of our example project. We tag our resulting image with the name fastify-example via the --tag flag.

docker build --tag fastify-example .

If we run the docker images command, we should see our new image in our list of container images.

docker images
REPOSITORY                                TAG       IMAGE ID       CREATED          SIZE
fastify-example                           latest    7e3f51733ddd   8 seconds ago    1.18GB

Running our Docker image

Now that we have built our Docker image with the fastify-example tag, we can try running it locally. We can run a Docker container of our image via the docker run command.

We run our Docker container with the -p (i.e., --port) flag to specify that we want to forward traffic from port 8080 on our host machine to port 3000 in the container because our API is listening on this. We also run our container with the -d flag, which tells the Docker Daemon to run the container detached in the background.

docker run -p 8080:3000 -d fastify-example

We can verify our container is running via the docker ps command.

docker ps
CONTAINER ID   IMAGE             COMMAND                  CREATED         STATUS         PORTS                    NAMES
5595944ea42b   fastify-example   "docker-entrypoint.s…"   3 seconds ago   Up 2 seconds   0.0.0.0:8080->3000/tcp   peaceful_brahmagupta

We can also verify our Docker container is up and working by hitting the /health endpoint with curl.

curl localhost:8080/health
{"alive":true}

We can also use other Docker CLI commands like docker logs to see the logs from our container. Note that the logs command expects the container ID or name, not the image name. From our example above, the name of our container is peaceful_brahmagupta.

docker logs peaceful_brahmagupta
{"level":30,"time":1695637221083,"pid":1,"hostname":"6e8107cd9149","msg":"Server listening at http://0.0.0.0:3000"}

We can use the docker inspect command to get a low-level description of our container. The JSON output can be helpful for debugging and troubleshooting.

docker inspect peaceful_brahmagupta

Finally, we can call docker stop to stop our container with a graceful shutdown, or we can call docker kill to kill our container, which will terminate it immediately.

Pushing our Docker image to a registry

When we build a Docker image locally or via our remote builders in Depot, the container image, by default, is kept on the machine that ran the Docker image build. When we want to run the Docker image locally, as we saw in the earlier step, the image staying on our machine is excellent.

But, most of the time, we want to push our image to a Docker container registry so we can share it with other developers, deploy it to our production environments, etc.

There are numerous container registries like Docker Hub, Amazon Elastic Container Registry (ECR), GCP Artifact Registry, and GitHub Container Registry. For this example, we will assume we are using GitHub Container Registry.

To push to a Docker container registry, we generally need to call docker login to authenticate to our registry. For GitHub Container Registry, we can use the ghcr.io hostname, our GitHub username, and a personal access token (PAT) to authenticate.

docker login ghcr.io -u GITHUB_USERNAME --password GITHUB_PAT
> Login succeeded

After logging into our container registry, we can build our image with a tag that includes the registry hostname, our GitHub username, and the image name. We also specify the --push flag, which will push our image to the registry we've tagged it with.

docker build -t ghcr.io/GITHUB_USERNAME/fastify-example:latest --push .

Alternatively, we can use docker tag and docker push to push an image we've built locally to a registry.

docker tag fastify-example ghcr.io/GITHUB_USERNAME/fastify-example:latest

This tags our fastify-example Docker image with ghcr.io/GITHUB_USERNAME/fastify-example:latest, and then we can push it to our registry.

docker push ghcr.io/GITHUB_USERNAME/fastify-example:latest

Conclusion

We've covered in this post how to get started with Docker and build a Docker image from a Dockerfile. We've also covered how to run a Docker container from that image and how to push that image to a container registry. All of these are handy to know when it comes to working with containers locally and in production.

With Depot, we build the Docker image up to 20x faster and provide critical insights about how to rewrite your Dockerfile to build faster, leverage caching, and more. We remove the need to think about the artifacts Docker produces, allowing you to focus on writing your own code and getting it into production faster.

You can sign up for an account and get your first 60 minutes of build time free. If you have questions comments or want to chat more about containers, check out our Community Discord.

Top 10 common Dockerfile linting issues

Kyle Galbraith — Fri, 15 Sep 2023 12:46:23 +0000

We recently announced the ability to lint Dockerfiles on build in our recent lint & build blog post.

Running a Dockerfile linter on a Docker image we want to build can allow us to follow some of the best practices around writing efficient Docker images. Efficient could mean faster builds or smaller image sizes.

This post covers the ten most common Dockerfile linting issues we've seen flowing through Depot to date. We expect these to change over time, but hopefully they can give everyone a good starting point for improving their Dockerfiles. We'll cover each issue, why it's a problem, and how to fix it.

How to lint a Dockerfile

With Depot, we make use of two Dockerfile linters, hadolint and a set of Dockerfile linter rules that Semgrep has written to make a bit of a smarter Dockerfile linter.

To lint a Dockerfile on-demand with Depot, we can pass the --lint flag during a build, which will run before the build.

Of course, we can also run hadolint ourselves locally without Depot with our own specific rules and config file. Or even use the hadolint Dockerfile linter UI. To run hadolint locally you can either install it via brew or use the Docker image and pipe your Dockerfile into it:

hadolint Dockerfile
# or use the Docker image
docker run --rm -i ghcr.io/hadolint/hadolint < Dockerfile

1. Multiple consecutive `RUN` instructions

Also known as lint error DL3059 from hadolint.

This is the most common issue we see with Dockerfiles flowing through Depot. It's present in nearly 30% of all Dockerfiles we've seen. The problem is that multiple RUN instructions are in a row that could be condensed. For example:

RUN download_a_really_big_file
RUN remove_the_really_big_file

It's helpful to know how Docker layer caching works to understand why this might be problematic. In short, each new RUN statement in a Dockerfile results in a new layer in the final image.

In this example, we create a new layer when we download the big file and another layer when we remove it. Both layers will be present in the final image. So, the final image will contain the big file in the first layer, making the final image larger than it needs to be.

However, DL3059 can also be problematic if we use two different RUN statements to install packages. For example:

RUN fetch_package_registry_list
RUN install_some_package

The first RUN statement will fetch the package registry list in this example. The second RUN statement will install the package. But if the package registry list changes between the first and second RUN statements, then the package registry list will be out of date when we install the package.

Solution to `DL3059`

When working with large files that we add and remove during a docker build, combining those operations into one atomic RUN statement is helpful.

RUN download_a_really_big_file && \
    remove_the_really_big_file

This reduces the final image size by removing the intermediate layer that contains the big file as we download and remove it in the same RUN statement. Note that this can have cache implications if you combine RUN statements with things that can be cached with things that frequently invalidate the cache. In those situations, you likely want to keep the portion that can be cached in its own RUN statement.

For the package registry example, we want to combine the fetch registry list with the install package into one RUN statement.

RUN fetch_package_registry_list && \
    install_some_package

This ensures that the package registry list is updated when we install the package instead of potentially being outdated.

2. Pin versions during `apt-get install`

A more controversial Dockfile linting issue is DL3008 from hadolint. This issue is also present in 30% of all Dockerfiles. The problem arises when not pinning versions during apt-get install. For example:

FROM ubuntu:22.04
RUN apt-get update && \
    apt-get install -y some-package

When you don't version pin, you're not forcing the docker build to verify it has a specific version and thus the required packages you may need. This can lead to unexpected behavior when you build your Dockerfile or run the resulting image if you inadvertently installed a newer version of a package than you expected.

Solution to `DL3008`

FROM ubuntu:22.04
RUN apt-get update && \
    apt-get install -y some-package=1.2.*

By pinning the version of some-package, the build is forced to retrieve the particular version. This allows you to build up guarantees about the packages you're installing in your Dockerfile and the dependencies of those packages.

The reason it's controversial is because version pinning runs the risk of needing to catch up on security updates. For example, suppose you pin a package version with a security vulnerability. In that case, you risk not getting your security update when you build your Dockerfile until you change the version to a new one. This is why it's essential to understand the packages you're installing and the security implications of pinning versions.

3. Use `--no-install-recommends` to avoid installing unnecessary packages

Another widespread linter error is DL3015, installing unnecessary packages with apt-get. This is present in 22% of all Dockerfiles. The issue arises when we're not using the --no-install-recommends flag during apt-get install. For example:

FROM ubuntu:22.04
RUN apt-get update && \
    apt-get install -y some-package

When you don't use the --no-install-recommends flag, you install all the recommended packages for the package and the package itself. Potentially increasing the final size of your Docker image by installing packages you don't need.

Soltuion to `DL3015`

FROM ubuntu:22.04
RUN apt-get update && \
    apt-get install -y some-package --no-install-recommends

The solution is to pass the flag --no-install-recommends to apt-get install. This will prevent the installation of recommended packages and reduce the final size of your container image. It's essential to understand the recommended packages for the packages you're installing to ensure you're getting all the dependencies.

4. Avoid using the cache directory when using `pip install`

Docker layer caching comes in again when we're talking about pip install during a Docker build. Hadolint error DL3042 is present in 18% of all Dockerfiles. The issue arises when we're not telling pip install not to use a cache directory in our Dockerfile. For example:

FROM python:3.11
RUN pip3 install mysql-connector-python

When you don't tell pip install not to use a cache directory, it will install the package and keep a cache directory for that package, which creates an unnecessary cache entry for every package you've installed via pip in that layer. When you have lots of packages, this can increase your final Docker image size.

Solution to `DL3042`

FROM python:3.11
RUN pip3 install --no-cache-dir mysql-connector-python

We don't need a cache directory for our pip packages because we don't need to reinstall packages when building a Docker image. The Docker layer cache can be used instead. Turning off the cache directory makes our final image smaller.

5. Remove the `apt-get` lists after installing packages

As we explored in our post around reducing Docker image sizes, keeping container image sizes down often returns to the actual docker build process. Hadolint error DL3009 is present in 16% of all Dockerfiles. The issue arises when we're not removing the apt-get lists after installing packages. For example:

FROM ubuntu:22.04
RUN apt-get update && \
    apt-get install -y some-package --no-install-recommends

Our earlier example for DL3015, shown here, can be optimized further to keep the final image size down. By not cleaning up the apt-get cache, it's written into the layer for that RUN statement. We are taking up valuable space in our final image.

Solution to `DL3009`

FROM ubuntu:22.04
RUN apt-get update && \
    apt-get install -y some-package --no-install-recommends && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

Here, we are combining the installation of some-package with the clean-up of the apt-get cache so that installing and clean-up happen in one atomic RUN statement. This keeps the final image size down by removing the apt-get cache from the final image and doesn't introduce another layer into the final image.

6. Make use of `WORKDIR` instead of `RUN cd some-path`

Another common Dockerfile linter issue is DL3003, using RUN cd instead of the WORKDIR statement. This is present in 14% of all Dockerfiles. Here is a typical example:

FROM ubuntu:22.04
RUN cd /usr/src/app && git clone git@github.com:depot/some-repo.git

Each RUN statement executes inside its own shell, and most commands can work with absolute paths.

Solution to `DL3003`

FROM ubuntu:22.04
WORKDIR /usr/src/app
RUN git clone git@github.com:depot/some-repo.git

When changing directories, you can use the WORKDIR statement, which spawns the shell in your specified directory. The only exception is when you need to do something inside the subshell; in that scenario, you still need to use cd.

7. Pin versions when installing packages via `pip`

Like DL3008, the Dockerfile linter issue DL3013 is the same idea but applied to pip install instead of apt-get install. This is present in 13% of all Dockerfiles. Here is a typical example:

FROM python:3.11
RUN pip3 install --no-cache-dir mysql-connector-python

When you don't version pin, you're not forcing the docker build to verify it has a specific version and thus the required packages you may need. As we saw in DL3008, this can have unexpected behavior if we install a different version than what we originally installed when we created the Dockerfile.

Solution to `DL3013`

FROM python:3.11
RUN pip3 install --no-cache-dir mysql-connector-python==8.1.0

By pinning the version of mysql-connector-python, the docker build is forced to retrieve the particular version regardless of what may be in the Docker layer cache.

8. Use JSON notation for `CMD` and `ENTRYPOINT` arguments

This Dockerfile lint error, DL3025, comes down to correctness when running the image. It's present in 12% of all Dockerfiles. Here are typical examples for both statements where this comes up:

FROM ubuntu:22.04
ENTRYPOINT foo run-server

FROM ubuntu:22.04
CMD foo run-server

When we don't use JSON notation for CMD and ENTRYPOINT arguments, the executables referenced won't receive signals from the OS correctly. This is particularly relevant when talking about how to signal to a running container that it is being shut down (i.e., a SIGTERM).

Solution to `DL3025`

FROM ubuntu:22.04
ENTRYPOINT ["foo", "run-server"]

FROM ubuntu:22.04
CMD ["foo", "run-server"]

By using JSON notation, the executable will be the containers PID 1 and, therefore, receive signals from the OS. Two additional things to note about this notation:

CMD doesn't process environment variables in shell form (i.e., $FOO_BAR) because of the side effect of how sh -c is used as the default entry point. So, we must handle environment variables ourselves outside the CMD statement.
The CMD statement is parsed as a JSON array, so we must use double quotes ("") instead of single quotes('') to correctly pass our arguments.

9. Use `apt-get` or `apt-cache` instead of the user facing `apt`

The command, apt, is meant to be an end-user tool and not to be used in Dockerfile RUN statements. So, DL3027 flags this Dockerfile lint error when we use apt instead of apt-get or apt-cache. This is present in 9% of all Dockerfiles. Here is a typical example:

FROM ubuntu:22.04
RUN apt install -y some-package=1.2.*

Solution to `DL3027`

FROM ubuntu:22.04
RUN apt-get install -y some-package=1.2.*

The interface of apt is not guaranteed across versions by Linux distributions. So it's better to use apt-get or apt-cache, which are more stable.

10. Pin versions when installing packages via `apk add`

As we've seen in DL3008 and DL3013, pinning versions is also important for apk add in Alpine-based Dockerfiles. This is present in 8% of all Dockerfiles. Here is a typical example:

FROM alpine:3.7
RUN apk --no-cache add some-package

Solution to `DL3018`

FROM alpine:3.7
RUN apk --no-cache add some-package=~1.2.3

The rationale is the same: version pinning forces the docker build to fetch the pinned version regardless of what may be in the Docker layer cache. An important thing to note for Alpine-based images is that we are using partial pinning here via the ~ syntax. We can pin to a specific version via some-package=1.2.3, but this will fail the build if this package is removed.

Conclusion

In this post, we looked at the top 10 most common Dockerfile linting issues we're seeing as builds are flowing through Depot. As we saw, they can vary in severity and impact. But they all have the potential to improve your Dockerfiles and your builds. Each issue comes with its own set of pros and cons.

For example, pinning versions can guarantee a specific state when building Docker images but have the downside of potentially missing security updates. Or using --no-install-recommends can avoid making your image bigger for dependencies you don't need or use. But it can also mean you miss a dependency that you need.

This post has given you some ideas on improving your Dockerfiles and your builds via linting. If you want to learn more about how Depot can help you improve your Dockerfiles on-demand, check out our recent post on linting and building Dockerfiles.

If you're looking to make your Docker image build process faster either for native Intel or Arm images, sign up for an account and give things a try. We make it easy to run your first build with either docker build or depot build via our quickstart guide.

How to clear Docker cache and free up space on your system

Kyle Galbraith — Tue, 08 Aug 2023 14:33:59 +0000

Docker persists build cache, containers, images, and volumes to disk. Over time, these things can take up a lot of space on your system, either locally or in CI. In this post, we'll look at the different Docker artifacts that can take up space on your system, how to clear them individually, and how to use docker system prune to clear Docker cache.

A short refresher on Docker caching

Docker uses layer caching to reuse previously computed build results. Each instruction in a Dockerfile is associated with a layer that contains the changes caused by executing that instruction. If previous layers, as well as any inputs to an instruction, haven't changed, and the instruction has already been run and cached previously, Docker will use the cached layer for it. Otherwise, Docker will rebuild that layer and all layers that follow it.

The Docker layers for which the hash of the inputs (such as source code files on disk or the parent layer) haven't changed get loaded from the cache and reused. For layers where the hash of inputs has changed, the layers get recomputed.

Using a cached layer is much faster than recomputing an instruction from scratch. So, generally, you want as much of your Docker build as possible to come from the cache and to only rebuild layers that have changed since the last build.

One of the main factors that affects how many of the layers in your image need to be rebuilt is the ordering of operations in your Dockerfile.

How much disk space is Docker using?

The first step is knowing the disk usage of Docker. We can use the docker system df command to get a breakdown of how much disk space is being taken up by various artifacts.

docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          138       34        36.18GB   34.15GB (94%)
Containers      74        18        834.8kB   834.6kB (99%)
Local Volumes   118       6         15.31GB   15.14GB (98%)
Build Cache     245       0         1.13GB    1.13GB

Docker uses 36.18 GB for images, 834.8 kB for containers, 15.31 GB for local volumes, and 1.13 GB for the Docker build cache. This comes to about 50 GB of space in total, and a large chunk of it is reclaimable.

What space can we claim back without affecting Docker build performance?

It's generally quite safe to remove unused Docker images and layers — unless you are building in CI. For CI, clearing the layers might affect performance, so it's better not to do it. Instead, jump to our CI-focused section.

Removing containers from the Docker cache

We can use the docker container prune command to clear the disk space used by containers. This command will remove all stopped containers from the system.

We can omit the -f flag here and in subsequent examples to get a confirmation prompt before artifacts are removed.

docker container prune -f
Deleted Containers:
399d7e3679bf9b14a1c7045cc89c056f2efe31d0a32f186c5e9cb6ebbbf42c8e

Total reclaimed space: 834.6kB

Which containers are unused?

We can see the IDs of unused containers by running the docker ps command with filters on the status of the container. A container is unused if it has a status of exited or dead.

docker ps --filter status=exited --filter status=dead -q
11bc2aa92622
355901f38ecb
263e9bde1f24

Note: If we want to know the size of the unused container, we can replace the -q flag with -s to get the size and other metadata about the container.

Removing all containers

If we want to remove all containers from the system, we can stop any running containers and then use the same prune command. Do this by feeding the output of docker ps -q into the docker stop or docker kill command if you want to kill the container forcibly.

docker stop $(docker ps -q)
docker container prune

Another option is the docker rm command, which can be used with docker ps -a -q to remove all containers.

docker rm $(docker ps -a -q)

Note: The docker rm command forces the removal of a running container via a SIGKILL signal. This is the same as the docker kill command. The docker ps -a -q command will list all containers on the system, including running containers, and feed that into the docker rm command.

Removing images

Docker images can take up a significant amount of disk space. We accumulate new images when base images change or build new ones via docker build, for example. We can use the docker image prune command to remove unused images from the system.

By default, it only removes dangling images, which are not associated with any container and don't have tags.

docker image prune -f
Deleted Images:
deleted: sha256:6f096c9fa1568f7566d4acaf57d20383851bcc433853df793f404375c8d975d6
...

Total reclaimed space: 2.751GB

We reclaimed over 2.7 GB of space by removing dangling images. But, if we recall from the output of our docker system df command, we have 34.15 GB of reclaimable images.

Where is the rest of that space coming from? These are images on our system that are tagged or associated with a container. We can run the docker image prune- a command to force the removal of these images as well, assuming they're unused images.

docker image prune -a -f
Deleted Images:
untagged: k8s.gcr.io/etcd:3.4.13-0
untagged: k8s.gcr.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2
deleted: sha256:0369cf4303ffdb467dc219990960a9baa8512a54b0ad9283eaf55bd6c0adb934
deleted: sha256:f3cecccfe2bea1cbd18db5eae847c3a9c8253663bf30a41288f541dc1470b41e

Total reclaimed space: 22.66GB

In this way, we remove all unused images not associated with a container, not just the dangling ones.

Removing volumes

Volumes are never cleaned up automatically in Docker because they could contain valuable data. But, if we know that we no longer need the data in a volume, we can remove it with the docker volume prune command. This removes all anonymous volumes not used by any containers.

docker volume prune -f
Total reclaimed space: 0B

Interestingly, we see that we didn't reclaim any space. This is because we have volumes that are associated with containers. We can see these volumes by running the docker volume ls command.

DRIVER    VOLUME NAME
local     0a44f085adc881ac9bb9cdcd659c28910b11fdf4c07aa4c38d0cca21c76d4ac4
local     0d3ee99b36edfada7834044f2caa063ac8eaf82b0dda8935ae9d8be2bffe404c
...

We get an output that shows the driver and the volume name. The command docker volume prune only removes anonymous volumes. These volumes are not named and don't have a specific source from outside the container. We can use the docker volume rm -a command to remove all volumes.

docker volume prune -a -f
Deleted Volumes:
c0c240b680d70fffef420b8699eeee3f0a49eec4cc55706036f38135ae121be0
2ce324adb91e2e6286d655b7cdaaaba4b6b363770d01ec88672e26c3e2704d9e

Total reclaimed space: 15.31GB

Removing build cache

To remove the Docker build cache, we can run the docker buildx prune command to clear the build cache of the default builder.

docker buildx prune -f
ID                                        RECLAIMABLE   SIZE        LAST ACCESSED
pw11qgl0xs4zwy533i2x61pef*                true          54B         12 days ago
y37tt0kfwn1px9fnjqwxk7dnk                 true          0B          12 days ago
sq3f8r0qrqh4rniemd396s5gq*                true          154.1kB     12 days ago

Total:  5.806GB

If we want to remove the build cache for a specific builder, we can use the --builder flag to specify the builder name.

docker buildx prune --builder builder-name -f

Removing networks

While Docker networks don't take up disk space on our machine, they do create network bridges, iptables, and routing table entries. So, similarly to the other artifacts, we can remove unused networks with the docker network prune command to clean these up.

docker network prune -f
Deleted Networks:
test-network-1

Removing everything with `docker system prune`

We can remove all unused artifacts Docker has produced by running docker system prune. This will remove all unused containers, images, networks, and build cache.

docker system prune -f
Deleted Images:
deleted: sha256:93477d5bde9ef0d3d7d6d2054dc58cbce1c1ca159a7b33a7b9d23cd1fe7436a3

Deleted build cache objects:
6mm1daa19k1gdijlde3l2bidb
vq294gub98yx8mjgwila989k1
xd2x5q3s6c6dh5y9ruazo4dlm

Total reclaimed space: 419.6MB

By default, this command will not remove volumes and only removes dangling Docker images. We can use the --volumes flag to remove volumes as well. We can also add the -a flag again to remove all images not associated with a container.

docker system prune --volumes -a -f

Managing Docker build cache in CI

If you are building Docker images in a CI environment, you can, of course, use the above commands as well. However, your builds might not be fully taking advantage of the Docker build cache for the following structural reasons:

In a CI environment with ephemeral runners, such as GitHub Actions or GitLab CI, build cache isn't persisted across builds without saving/loading it over the network to somewhere off of the ephemeral runner. Saving and loading the cache is therefore slow because the network transfer speed is slow.
By default, all CI runners are ephemeral unless you run your own.
If there is disk space, it's usually capped at 10 to 15 GB. Thus, if you're building large images with large layers, you will likely exhaust it.

Even if you are building very small images and only keep essential layers in the Docker cache on each CI runner, your builds will likely not use the cache and thus be quite slow, as computing each layer on every build can take a while.

Loading and saving cache from ephemeral CI runners via the network can take a considerable amount of time, negating the benefits of caching compared with always rebuilding all layers from scratch.

Then how can you optimize the use of the Docker cache in CI systems?

Consider using Depot. Depot automatically persists the cache across builds on a real SSD disk. This makes Docker builds that use Depot up to twenty times faster than building Docker images on CI without it. With Depot, you build with a full Docker cache without the network overhead.

Compared with network-based caching systems, Depot relies on fast SSDs to make the cache instantly available for Docker builds.

Adding Depot to your build only takes a few minutes — use the depot CLI as a drop-in replacement for the docker CLI, or use an environment variable without changing the rest of the configuration.

Learn more about using Depot in a CI environment →

Forem: Kyle Galbraith

Depot Changelog: June 2025

Egress filtering for GitHub Actions Runners

Now available: Audit logging

Windows GitHub Actions runners are now GA

depot cargo for faster Rust builds

Dependabot now runs on Depot GitHub Actions runners

And more good stuff...

Now available: Build autoscaling for everyone

What is the container build architecture?

But one limitation has remained

How does container build autoscaling work?

When should I use container build autoscaling?

How do I enable container build autoscaling?

What about the layer cache?

Billing details

What's next?

Now available: Claude Code sessions in Depot

So, what are Claude Code sessions in Depot?

How do you use Claude Code sessions in Depot?

Uhhh, I don't remember my session?

Where could this be better?

How does it actually work?

Why we built this

Conclusion

Self-hosted GitHub Actions runners aren't free

Do you like operational overhead?

Free isn't actually free with GitHub Actions

Leverage spot instances or savings plans to reduce instance costs

Egress traffic

The human cost

There are tools to help

ARC

Self-hosted runners with the Terraform module: github-aws-runners/terraform-aws-github-runner

What about Depot?

You can self-host the data plane if you want

Conclusion

Faster Claude Code agents in GitHub Actions

What is Claude Code?

Configuring Claude Code in GitHub Actions

Current gotchas

Having Claude Code open pull requests after completing tasks

Speeding up Claude Code with Depot GitHub Actions runners

Conclusion

BuildKit in depth: Docker's build engine explained

What is BuildKit?

BuildKit speeds up Docker builds with parallelization

Optimize your Dockerfile to take advantage of parallelization

BuildKit speeds up Docker builds with layer caching

BuildKit under the hood

The flow of information through BuildKit: frontends, backends and LLB

BuildKit's LLB

The different BuildKit LLB operations explained

SourceOp

ExecOp

FileOp

MergeOp

DiffOp

BuildOp

BuildKit speeds up your Docker builds using its LLB DAG

Build Docker images faster using build cache

Understanding Docker Build Cache

Tips for efficiently using the Docker build cache

Order your layers wisely

Keep your layers small and focused

Avoid copying files that are not needed

Use .dockerignore to exclude files and directories

Avoid unnecessary dependencies from package managers

Leverage the RUN cache for finer-grained control

Reduce the total number of layers

Combine multiple RUN commands where possible

Be thoughtful about base images

Take advantage of multi-stage builds

Conclusion

How to build multi-platform Docker images in GitHub Actions

Multi-platform images and Arm support

Building multi-platform Docker images in GitHub Actions

A managed solution

Faster Docker builds for Arm without emulation

Emulation is painfully slow

Self-hosted runners with the Terraform module: `github-aws-runners/terraform-aws-github-runner`

Use `.dockerignore` to exclude files and directories

Leverage the `RUN` cache for finer-grained control

Combine multiple `RUN` commands where possible

1. Multiple consecutive `RUN` instructions

Solution to `DL3059`

2. Pin versions during `apt-get install`

Solution to `DL3008`

3. Use `--no-install-recommends` to avoid installing unnecessary packages

Soltuion to `DL3015`

4. Avoid using the cache directory when using `pip install`

Solution to `DL3042`

5. Remove the `apt-get` lists after installing packages

Solution to `DL3009`

6. Make use of `WORKDIR` instead of `RUN cd some-path`

Solution to `DL3003`

7. Pin versions when installing packages via `pip`

Solution to `DL3013`

8. Use JSON notation for `CMD` and `ENTRYPOINT` arguments

Solution to `DL3025`

9. Use `apt-get` or `apt-cache` instead of the user facing `apt`

Solution to `DL3027`

10. Pin versions when installing packages via `apk add`

Solution to `DL3018`

Removing everything with `docker system prune`