Forem: Depot

Optimize your Dockerfile for 5x faster builds

The Charmizard — Fri, 23 Jan 2026 22:07:48 +0000

Slow Docker builds are annoying. You make a small code change, run docker build, and wait. This post shows you how to optimize your Dockerfile for faster builds.

These Dockerfile optimization techniques will work whether you're building locally or in CI, and for any language. The examples in this post use a Node.js application with about 40 dependencies (including common packages like ORMs, UI libraries, and AWS SDKs) to represent a production React/Node application.

Here's a preview of the biggest wins after optimization:

Metric	Before	After	Improvement
Rebuild time (after code change)	37.4s	7.1s	5.3x faster
Final image size	2.49 GB	1.11 GB	55% smaller

Note: Build times can vary by 10-15% between runs depending on system load, network conditions, and npm registry response times. The values in this post are averages across three builds.

The problem: A slow Dockerfile

The example Dockerfile takes about 37 seconds to rebuild after code changes. The optimizations in the following sections bring that down to about 7 seconds. 37 seconds isn't the worst, but saving 30 seconds per build adds up. More importantly, the benefits of these optimizations scale with your app.

Here's an unoptimized Dockerfile that rebuilds everything on every code change:

FROM node:22
WORKDIR /app
COPY . .
RUN npm ci
RUN npm run build
CMD ["npm", "start"]

When you change a single line of application code, Docker invalidates the cache at the COPY . . instruction and rebuilds everything after it. So even though package.json didn't change, Docker reinstalls all the npm packages.

Build times:

First build: 37.1s
After code change: 37.4s (still rebuilds everything)

The rebuild is about as slow as the first build because the cache invalidates when Docker reaches the first changed instruction (COPY . .) and forces a complete reinstall of dependencies. The main problem? The order of the instructions. The Dockerfile copies all the files before installing dependencies, so any file change forces a full dependency reinstall.

Order instructions to maximize cache reuse

The Docker build cache invalidates at the first changed instruction and rebuilds everything after it. To minimize rebuilds, order your Dockerfile from least to most frequently changed.

For most apps, dependencies change less often than your source code, so you should install dependencies before you copy source files. For example:

FROM node:22
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build
CMD ["npm", "start"]

Build time improvement:

Before: 37.4s (rebuilds everything)
After: 7.1s (reuses dependency cache)
Improvement: 5.3x faster

Reordering the Dockerfile instructions cut rebuild time by 81% (from 37.4 seconds to 7.1 seconds). The builder reuses the cached npm ci layer when only the application code changes and the package manifest files remain unchanged.

How the Docker build cache reuses layers

The builder calculates a cache checksum from package.json and package-lock.json. As long as those files don't change, the npm ci layer stays cached—even when you modify other source files.

For details on how the builder determines cache validity, see Cache invalidation for ADD and COPY instructions in our Ultimate guide to Docker build cache post.

Optimize instruction order for any language

You can apply the principle of ordering instructions from least to most frequently changed for any language. Copy your dependency manifest files first, install dependencies, then copy your source code.

Python example:

COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .

Go example:

COPY go.mod go.sum ./
RUN go mod download
COPY . .

Ruby example:

COPY Gemfile Gemfile.lock ./
RUN bundle install
COPY . .

Exclude unnecessary files from the build

Every file in your build context can invalidate the cache. If you copy a directory that includes files Docker doesn't need, changing those files forces a rebuild even though the files don't affect your application.

Use a .dockerignore file

A .dockerignore file works like .gitignore but for Docker builds. To exclude files and directories from the build context, create a .dockerignore file in the same directory as your Dockerfile.

An example .dockerignore file:

node_modules
npm-debug.log
.git
.gitignore
README.md
.env
.vscode
.idea
dist
build
*.log

Why exclude these files from your Docker build?

Some files are rebuilt during the build.
- node_modules, dist, build: Regenerated by npm ci and npm run build.
Development files aren't needed at runtime.
- .env: May contain secrets, changes frequently.
- .vscode, .idea: Editor configs.
- README.md: Documentation.
Large files slow down context transfer.
- .git: Version history (can be hundreds of MB).
- *.log: Accumulated logs.

Build context size reduced

Excluding unnecessary files reduces the build context size, which speeds up the initial context transfer to the builder and prevents cache invalidation from irrelevant file changes.

Build context size:

Without .dockerignore: 325 MB
With .dockerignore: 1.2 kB
Reduction: 99.6% (from 325 MB to 1.2 kB)

Without .dockerignore, Docker copies your entire local node_modules directory (which gets rebuilt anyway) and Git history (which isn't needed in the image).

Separate build and runtime dependencies

Most apps need build tools that aren't required at runtime. Multi-stage builds let you use a full build environment to compile your application, then copy only the runtime artifacts to a minimal final image.

First, let's see what a single-stage build includes, then we'll split it into stages.

Single-stage build with all dependencies

A single-stage build includes everything:

FROM node:22
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build
CMD ["npm", "start"]

This image includes:

Build tools from the base image
All npm dependencies (dev and production)
Source code and build artifacts

Final image size: 2.49 GB

Multi-stage build splits build and runtime

Multi-stage builds split the process into distinct stages:

# Stage 1: deps - install all dependencies once
FROM node:22 AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci

# Stage 2: build - inherits from deps, compiles the application
FROM deps AS build
COPY . .
RUN npm run build

# Stage 3: prod - minimal runtime image with production dependencies only
FROM node:22-slim AS prod
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY --from=build /app/dist ./dist
COPY --from=build /app/public ./public
CMD ["npm", "start"]

The deps stage installs all dependencies once. The build stage inherits from deps (reusing the installed dependencies without reinstalling), copies the source code, and builds the application. The production stage starts fresh with node:22-slim and runs npm ci --omit=dev to install only production dependencies (no dev tools or testing frameworks).

The key here is that the build stage uses FROM deps, so it doesn't need to reinstall dependencies. This pattern is cleaner and more efficient than reinstalling dependencies in each stage.

Image size improvement:

Single-stage build: 2.49 GB
Multi-stage build: 1.11 GB
Improvement: 55% smaller

The multi-stage build maintains fast rebuild times for code changes (between 7 and 8 seconds) because the deps stage remains cached when only source code changes. The size reduction is an additional benefit that doesn't compromise rebuild speed.

Cache benefits:

Changes to the source code get rebuilt from the build stage forward, but don't affect earlier stages.
Changes to the production stage don't invalidate the build stage cache.
Each stage maintains its own cache layers independently.

Note: BuildKit executes independent stages in parallel. If your application has multiple components that can be built independently, you can build them simultaneously. For more information, see Stage parallelism in the How BuildKit parallelizes your builds post.

Use cache mounts for package managers

Cache mounts provide persistent storage for package manager caches across builds. Unlike the regular build cache that invalidates when dependencies change, cache mounts let the package manager maintain its own cache of downloaded packages.

While code changes are more common, optimizing dependency updates matters when you're actively developing and frequently adding packages.

Cache mounts are most useful when:

Your project has hundreds of dependencies.
You frequently add or update individual dependencies.
Package downloads are slow (large packages or slow network).
You're building on the same machine or builder repeatedly.

Without cache mounts

The baseline for this test uses the multi-stage Dockerfile from the previous section and adds a new dependency. Unlike code changes, which take about 7 seconds because the deps stage remains cached, dependency changes force the deps stage to rebuild along with all downstream stages.

Standard dependency installation downloads packages every time dependencies change:

COPY package.json package-lock.json ./
RUN npm ci

When you add a new dependency, the npm ci layer rebuilds with an empty npm cache, forcing npm to re-download all packages.

Build time without cache mounts: 32.3s

With cache mounts

Cache mounts persist the npm cache directory across builds:

COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci

The --mount=type=cache,target=/root/.npm option tells the builder to mount persistent storage at npm's cache directory. When dependencies change and the npm ci layer needs to rebuild, npm can reuse packages from this persisted cache instead of downloading them again.

Build time improvement (adding one new package):

Without cache mount: 32.3s
With cache mount: 28.5s
Improvement: 3.8s faster (12% improvement)

The improvement here is pretty minimal because npm ci is already fast with modern npm and good network connections.

Cache mounts for other package managers

Python (pip) example:

RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

Go example:

RUN --mount=type=cache,target=/go/pkg/mod \
    go mod download

Ruby example:

RUN --mount=type=cache,target=/usr/local/bundle/cache \
    bundle install

Each package manager has its own default cache location. The cache mount makes that location persistent across builds.

Cache mount persistence

Cache mounts persist on a single Docker daemon or builder instance. When you build locally, the cache persists between your local builds. In CI environments, cache mount persistence depends on your setup.

Ephemeral CI runners (GitHub Actions default runners, fresh containers): Cache mounts don't persist between builds. You'll need to configure external cache backends.
Persistent builders (self-hosted runners, Depot, BuildKit with remote cache): Cache mounts persist and provide speed benefits.

For details on using cache mounts in CI, see How to use BuildKit cache mounts in CI providers.

Choose the right base image

Your base image choice affects both image size and cache behavior. Smaller base images mean less data to download and store in cache layers.

Base image options for Node.js

The following table lists Node 22 base images. The image sizes vary between architectures and versions, but the use cases remain the same.

Image	Use case
`node:22`	Full development environment with build tools and system libraries.
`node:22-slim`	Roughly 75% smaller than the full base image. Production runtime without build tools and documentation.
`node:22-alpine`	Roughly 85% smaller than the full base image. Minimal runtime based on Alpine Linux.

Note: Image sizes vary by Node version.

When to use each base image

Use full base images (for example, node:22) for:

Build stages in multi-stage builds where you need compilation tools
Development environments
Applications with native dependencies that need build toolchains

Use slim images (for example, node:22-slim) for:

Production runtime stages
Applications without native dependencies or with pre-compiled binaries
When you need glibc compatibility (most Linux binaries expect glibc)

Use Alpine images (for example, node:22-alpine) for:

Minimal production deployments where size is critical
Applications without native dependencies
When you've verified your native dependencies work with musl libc

Example build with optimized base images

Here's an example of a production multi-stage build with appropriate base images:

# Stage 1: deps - install all dependencies
FROM node:22 AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci

# Stage 2: build - inherits from deps for compilation
FROM deps AS build
COPY . .
RUN npm run build

# Stage 3: prod - uses slim image for smaller final image
FROM node:22-slim AS prod
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY --from=build /app/dist ./dist
COPY --from=build /app/public ./public
CMD ["npm", "start"]

This example gives you full access to build tools during compilation but keeps the final image small.

How to find optimization opportunities

To optimize effectively, measure where time is spent during your builds.

View detailed build output

Use the plain progress output to show container output and see timing for each build step:

docker build --progress=plain .

The --progress flag output shows:

Time spent on each instruction
Whether layers came from cache (CACHED markers)
Context transfer times
Layer push/pull times

To find optimization opportunities, look for steps that take more than 10 seconds and aren't marked as CACHED.

Identify optimization opportunities

Look for the following in your builds:

Long-running RUN instructions that execute frequently (good candidates for better caching or cache mounts)
Large COPY operations that invalidate often (add to .dockerignore)
Sequential operations that could be parallelized (multi-stage builds)
Repeated package downloads that could use cache mounts

Complete optimized example

Optimization checklist:

Reorder instructions (least to most frequently changed)
Add .dockerignore file
Use multi-stage builds
Add cache mounts for package managers
Choose appropriate base images

Here's the full optimized Dockerfile incorporating all techniques:

# Stage 1: deps - install all dependencies once
FROM node:22 AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci

# Stage 2: build - compile using the full dependency tree
FROM deps AS build
COPY . .
RUN npm run build

# Stage 3: prod - minimal runtime image with only production dependencies
FROM node:22-slim AS prod
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci --omit=dev
COPY --from=build /app/dist ./dist
COPY --from=build /app/public ./public
CMD ["npm", "start"]

With the following .dockerignore file:

node_modules
npm-debug.log
.git
.gitignore
README.md
.env
.vscode
.idea
dist
build
*.log

Final results

Optimization	Primary benefit	Impact
Instruction reordering	Code change rebuilds	5.3x faster (biggest win)
Multi-stage builds	Image size	55% smaller
Cache mounts	Dependency updates	12% faster
.dockerignore	Build context	99.6% smaller context

Conclusion

These optimizations made rebuild time over 5x faster and reduced image size by 55%. If you need a place to start, go with instruction reordering. It delivers the biggest speed improvement in 30 seconds of work.

Building your Docker image using Depot can make your builds even faster. See Depot container builds for how it all works.

Written by: Andrea Anderson - Technical Writer at Depot

How BuildKit Parallelizes Your Builds

The Charmizard — Fri, 21 Nov 2025 02:02:46 +0000

When you run docker build, you might assume your Dockerfile instructions execute one after another, like a traditional script. But behind the scenes, BuildKit is doing something far more sophisticated. At the heart of BuildKit lies a DAG (Directed Acyclic Graph) solver that transforms your Dockerfile into an optimized execution plan, identifying all parallelizable operations while maintaining dependency order.

We've previously covered how BuildKit works here, but today we're discussing how BuildKit parallelizes your builds to make them faster and more efficient.

Understanding BuildKit's DAG graph solver

BuildKit parses build instructions into something called LLB (Low-Level Build) format, creating a dependency graph of all the operations needed to produce your final image:

The DAG solver examines each instruction in your build and determines what it depends on. If instruction B needs the output of instruction A, they run sequentially. But if instructions B and C both only depend on A, they can run at the same time once A completes. This dependency analysis happens before any actual building starts, allowing BuildKit to create the most efficient execution plan possible.

This graph-based approach is what enables BuildKit to be fully concurrent. Every node in the graph represents a build operation, and BuildKit can execute any nodes that don't have unmet dependencies. It's constantly looking for work it can parallelize, which is why modern container builds can be surprisingly fast when structured properly.

Three levels of parallelism in action

Once BuildKit has built its execution graph, parallelism happens at three distinct levels, each contributing to faster builds in different ways.

Stage parallelism

The most visible form of parallelism happens when you have multiple stages in a multi-stage Dockerfile that don't depend on each other. Consider a typical web application with both frontend and backend components:

FROM node:20 AS frontend
WORKDIR /app
COPY frontend/ .
RUN npm install && npm build

FROM golang:1.21 AS backend
WORKDIR /app
COPY backend/ .
RUN go build -o server

FROM alpine AS final
COPY --from=frontend /app/dist /static
COPY --from=backend /app/server /usr/bin/

BuildKit recognizes that the frontend and backend stages have no dependencies on each other. While your Node.js dependencies are installing and your React app is building, BuildKit is simultaneously compiling your Go backend on a separate thread or CPU core. The final stage waits for both to complete, but you've effectively cut your build time by running these independent workloads in parallel.

Instruction parallelism

Even within a single stage, BuildKit finds opportunities for parallel execution. When you have multiple COPY instructions that don't depend on each other, or when different branches of your build graph can be resolved independently, BuildKit executes them concurrently.

This is particularly noticeable when you're copying multiple directories or files that will be processed separately later. BuildKit can fetch all these resources in parallel rather than sequentially, shaving precious seconds off your build time. The solver is smart enough to understand which operations can safely overlap and which must wait for their dependencies.

Deduplication across concurrent builds

While not being very obvious, perhaps the most clever optimization is BuildKit's automatic deduplication. This isn't just about caching, but about recognizing when multiple parts of your build are trying to do the exact same thing and ensuring that work happens only once.

Imagine you're building multiple services that all start from the same base image and run npm ci with identical package.json files. Without deduplication, each service would run its own npm ci, even if they're building simultaneously. But BuildKit is smarter than that. It uses content-addressable storage and checksums to identify when different build contexts would produce identical layers.

When BuildKit detects this situation, something interesting happens. The first build starts computing the layer while the others wait. Once the first build completes that npm ci, all the waiting builds immediately use that result and move on. The same operation that would have run three times runs only once.

This deduplication happens automatically across concurrent builds on the same runner, whether they're triggered by different developers pushing to the same repository or a docker bake command building multiple targets at once.

Scaling beyond a single builder with Depot's auto-scaling

While BuildKit's parallelism features are powerful, they're ultimately limited by the resources of the machine they're running on. A single builder can only do so much, even with perfect parallelization. This is where Depot's autoscaling feature changes the game.

By default, all builds for a Depot project run on a single BuildKit instance per architecture. This maximizes cache utilization and deduplication benefits, as all builds share the same cache and can deduplicate work across each other. But when you have resource-intensive builds or high build volume, that single instance can become a bottleneck.

With auto-scaling enabled, Depot automatically provisions additional BuildKit builders when your concurrent build limit is reached. Each new builder starts with a clone of the main builder's cache, so they're not starting from scratch. This means you can handle sudden spikes in build demand, like when multiple developers push changes simultaneously or when your CI/CD pipeline triggers numerous builds at once.

The relationship between BuildKit's parallelism and Depot's auto-scaling is complementary. BuildKit optimizes execution within each builder, finding every opportunity to parallelize and deduplicate work. Depot's auto-scaling ensures you have enough builders to handle your workload, preventing any single builder from becoming overwhelmed.

What this means for your builds

The combination of BuildKit's intelligent DAG solver, multi-level parallelism, automatic deduplication, and Depot's autoscaling creates a build system that's both fast and efficient. It's not just about throwing more compute at the problem, but rather being smart about how that compute is used, ensuring that work is never duplicated unnecessarily and that every available CPU cycle is put to good use.

Every build you run benefits from BuildKit's intelligent parallelism, with the DAG solver constantly finding ways to save time without sacrificing correctness. Depot's auto-scaling takes it further by ensuring you always have enough builders to handle the load. Together, they make "waiting for builds" a thing of the past.

For more details on configuring build parallelism and auto-scaling in your Depot projects, see our build parallelism documentation.

Author:
Pedro Guerra
Support Engineer @ Depot

A marketer's hour of hell: What I finally get about slow builds

The Charmizard — Thu, 23 Oct 2025 18:07:43 +0000

As a marketer and self-taught developer, I didn't fully grasp the pain of slow builds and broken AF CI/CD pipelines until this week when AWS US-East-1 went down. In my previous positions, I would ask for a change and the engineers would deal with it. From the outside looking in, if it took forever to deploy or they refused to deploy on Friday, or (insert other stereotype about engineers here), it was obviously just an excuse. Right?

Wrong. And this week, my foot was rightfully shoved in my mouth.

Here's the thing about those "stereotypes", seasoned developers aren't being dramatic when they take forever to deploy or refuse Friday deploys. They're being smart. And I finally get it now.

Life with fast builds

As the head of developer marketing here at Depot, I'm spoiled. If a change needs to be made to the website:

I create a branch
make my changes and test locally
create the PR
merge and deploy to preview

I see my changes in preview within 2 minutes, and then deploy to live within another 2 minutes. There's no "don't deploy" engineering culture here at Depot because, well... if something goes wrong, you can fix it within minutes instead of hours.

That was my exact workflow until this week.

When everything slowed down

When AWS US-East-1 went down, like much of the internet, we felt it too. I went to make a simple adjustment for tracking a link in PostHog. Made my changes, created my PR, and bam... lint failure. This happens all the time because I forget to run 'fmt', so I quickly ran it and pushed without stopping to check locally, because the last 99 times I've run 'fmt' it hasn't made a damn bit of difference. (Lesson learned, btw).

Waited for checks, merged, and deployed.

That push broke the docs pages. An essential piece to ANY devtool website.

I panicked. Broke out in a cold sweat. Fixed the very small error and pushed the fix, but this time the build took almost 6x longer than usual to fail. Made one more small adjustment, pushed again, another 6x longer build that finally succeeded. Then a long slog to get the fix out to production.

Overall, the docs page was down for close to an hour. With Depot, in my everyday workflow, with AWS-US-East-1 running on all cylinders, this fix would have taken me 10 minutes tops.

That's 50 minutes our docs weren't serving our customers. 50 minutes of wasted time.

Now imagine if that were your entire product going down for that long.

The realization

Depot makes our builds so fast that even a temporary slowdown felt painful. Our usual 2-minute builds stretched to 12+ minutes, and suddenly I understood the frustration engineering teams face daily when their builds crawl. If this brief taste of slow builds was this excruciating, I can't imagine dealing with it as your baseline.

Those engineers who seem overly cautious about deploys? They're not making excuses. They're managing risk in an environment where a simple fix can take an hour instead of minutes. This week, I finally felt that pain myself and it gave me a whole new appreciation for why speed matters.

We analyzed 66,821 GitHub Actions runs: 9 hidden gems you're missing

The Charmizard — Thu, 02 Oct 2025 14:41:33 +0000

Do you stick to official GitHub Actions in your workflows? Makes sense. Why risk your CI pipeline on some random third-party action?

Turns out, the biggest CI wins could be hiding in the actions marketplace, where other developers have solved the exact problems you're facing.

We analyzed 66,821 workflow runs across all organizations using Depot's runners to uncover the third-party GitHub Actions ecosystem. The results revealed some hidden gems that could transform your workflows.

Scope of third-party adoption

189 third-party actions with multi-organization adoption (we filtered out single-organization actions to focus on broadly useful tools)
63 authors/organizations

As you’d expect, the actions run the gamut from specialized tools for specific languages to general productivity enhancers.

9 third-party actions that are worth your time

Depot customers are concerned about CI speed and naturally have adopted actions to speed up performance. Here are 9 actions with relatively low adoption that deliver significant value.

1. Build step optimization

dorny/paths-filter - Used by 11% of organizations

I think this is super cool. This action detects which files changed in a PR and sets outputs you can use to conditionally run jobs. It’s great when you want to control running individual jobs or steps only when certain file changes happen.

- uses: dorny/paths-filter@v3
  id: changes
  with:
    filters: |
      backend:
        - 'src/api/**'
      frontend:
        - 'src/web/**'

- name: Run backend tests
  if: steps.changes.outputs.backend == 'true'
  run: pnpm run test:api

2. Fast Python package management

astral-sh/setup-uv - Used by 7% of organizations

Installing Python dependencies can take a really long time. Astral’s uv comes to the rescue.

For my most recent Python project, I’ve switched over to uv and am getting about 6x faster installs. It’s great. By default the astral-sh/setup-uv action caches and that makes things even better.

- uses: astral-sh/setup-uv@v6
- name: Install dependencies
  run: uv pip install -r requirements.txt

3. Compilation caching

mozilla-actions/sccache-action - Used by 6% of organizations

We're always trying to speed up compiles. This action speeds up compilation for Rust, C++, and other compiled languages by caching compilation results across CI runs. Depot runners are already set up to use sccache to speed up builds.

For rocker, my Rust rewrite of docker, using sccache cut the build and test time by half.

- uses: mozilla-actions/sccache-action@v0.0.9
- name: Build project
  run: RUSTC_WRAPPER=sccache cargo build --release

4. System package caching

awalsh128/cache-apt-pkgs-action - Used by 2% of organizations

In CI, installing packages from apt can take a long time. This action can cache packages eliminating repeated package downloads and installations. There are caveats if your package has pre- or post-scripts. Well worth a try.

- uses: awalsh128/cache-apt-pkgs-action@v1
  with:
    packages: libssl-dev

5. Robust CI pipelines

nick-fields/retry - Used by 4% of organizations

As much as we engineers don’t want it to be true, it’s not uncommon for tests to be flaky. nick-fields/retry can automatically retry failed steps with configurable backoff. I’m not exactly sure how to categorize this one; feels like it removes some of the need to restart CI runs, so, performance? In any case, it might be controversial, but it’s definitely pragmatic.

- uses: nick-fields/retry@v3
  with:
    timeout_minutes: 10
    max_attempts: 3
    command: pnpm run integration-tests

6. Better PR feedback

marocchino/sticky-pull-request-comment - Used by 4% of organizations

Updates a single comment on PRs. This is nice as it can help reviewers get context from a PR instead of digging through CI logs. This example shows how to put the contents of a file as a comment in the PR.

- uses: marocchino/sticky-pull-request-comment@v2
  with:
    path: coverage-results.md

7. Beautiful test result reports

dorny/test-reporter - Used by 3% of organizations

Another nice action by dorny. With this test failures are immediately visible in PR checks with detailed context. For me, it is pretty painful to search through go test logs for the word fail. Too many tests or logs have that as their name! Here is a way to get a nice simplified view. I should note that this supports more than go tests!

- name: Run tests
  run: go test -json ./... > testresults.json
- name: Test Report
  uses: dorny/test-reporter@v2
  with:
    name: Go Tests
    path: testresults.json
    reporter: golang-json

8. Universal binary installer

taiki-e/install-action - Used by 3% of organizations

This simplifies and speeds up getting the right tools into the CI environment. It installs precompiled binaries from GitHub releases with automatic caching and platform detection. The GitHub repo includes a list of all the tools it supports ready to go. Really nice and simple.

- uses: taiki-e/install-action@v2
  with:
    tool: cargo-nextest,just,cargo-hack

9. Enforce PR standards

amannn/action-semantic-pull-request - Used by 3% of organizations

I mean, I like and use conventional commits, but I figure your commit message is your own. If you want to enforce semantic PR titles for automated changelogs and better commit history, you can use this.

- uses: amannn/action-semantic-pull-request@v5
  with:
    types: |
      fix
      feat
      docs
      ci
      chore

Honorable mentions

The downside of having a really interesting dataset is that you have to pick only a few. I figured I’d share a few more metrics as bellwethers for our industry.

AI and automation:

anthropics/claude-code-action - AI-powered code review (4% adoption)

Testing and quality:

chromaui/action - Visual regression testing (3% adoption)
cypress-io/github-action - E2E testing with zero config (3% adoption)
codecov/codecov-action - Coverage reporting (7% adoption)

Infrastructure:

pulumi/actions - Infrastructure as code deployments (3% adoption)
hashicorp/setup-terraform - Terraform workflows (4% adoption)

Security and secrets:

dopplerhq/cli-action - Centralized secrets management (3% adoption)
1password/load-secrets-action - 1Password integration (<1% adoption)

Language-specific:

pnpm/action-setup - Fast Node.js package management (17% adoption)
ruby/setup-ruby - Ruby environment setup (5% adoption)
dtolnay/rust-toolchain - Rust toolchain management (6% adoption)
oven-sh/setup-bun - Bun JavaScript runtime (6% adoption)

The bottom line

The GitHub Actions third-party ecosystem has matured into a sophisticated toolkit where specialized solutions often outperform general-purpose alternatives. Maybe it's time to see what you're missing?

Git is really cool, actually

The Charmizard — Sat, 27 Sep 2025 00:17:00 +0000

I have a favorite Git protocol operation. It's fetch.

Before you judge me for having strong opinions about wire protocols, hear me out: fetch reveals the best things about Git's design. The conversation between your client and the server is simple on the surface: the client says "I have these commits" and the server responds with exactly what's missing. Nothing wasted or redundant. Just a stateless request-response exchange that manages to sync distributed repositories without either side maintaining complex session state. Let's break down how it works.

git's smart protocol is really cool, actually

Modern Git clients and servers use a "smart protocol". Essentially, it lets the Git client negotiate with the server on what objects it needs. On the surface, the Git smart protocol is pretty simple:

Client -> Server [what can you do?]
Server -> Client [can list references, grab commits, and push up commits]
Client -> Server [please list refs matching x filters, or send me some objects I'm missing, or take these new commits, or …]
Server -> Client [here's what you need]

There are a couple things to note here:

Git's smart protocol is request -> response. The client sends a single request to the server, and the server statelessly gives a response. This design makes it much easier to scale servers and reduces the amount of complexity (and bugs :P) of Git server implementations. And new Git features (like partial clones) can be added without worrying as much about backwards compatibility.
The protocol works over multiple different transports: HTTPS, SSH, or even Git's own (legacy) protocol.

That's the high level picture. But things get interesting when we dive into an actual operation. Let's take a closer look at my favorite Git protocol operation (if I dare to have a favorite): fetch.

example: fetch

Fetch, of course, follows the same request/response pattern:

Client -> Server: "hey, I'd like to fetch [commit/branch]. I have these commits, so I don't need them or any commits before them. I'm done specifying what I want."

Here's an example of a real request. You can see something similar by running GIT_TRACE_PACKET=1 git fetch origin main on a repo with some data to fetch:

# 1. The server advertises capabilities
# 2. The client asks what commits are available

# 3. The server responds with commits and their corresponding branches
fetch< 8f2037f2b3df3d34baec05df27f41f9c4fe601cd refs/heads/main
fetch< 3a54ae1da97c7c4d404829232bc0a2b824c78d13 refs/tags/v0.0.1
...
fetch< 0000 # we're done, server

# 4. Client requests objects (wants/haves + pack optimizations)
fetch> command=fetch
fetch> agent=git/2.51.0-Darwin
fetch> object-format=sha1
fetch> 0001
fetch> thin-pack
fetch> ofs-delta
# Ask for main's tip; tell the server what we already have
fetch> want 8f2037f2b3df3d34baec05df27f41f9c4fe601cd
fetch> have 56cf9cd609ce7f39bc9bfeacea14cee05a137164
fetch> have 733ea9d857c90e23369942515c341ef56f16c680
fetch> have ed4b55cfbcd73e853c782d48346d929898dcf54a
.. many many more refs
fetch> 0000 # we're done, please respond

# 5. Server acks the common base(s), signals readiness
fetch< acknowledgments
fetch< ACK 56cf9cd609ce7f39bc9bfeacea14cee05a137164
fetch< ACK 733ea9d857c90e23369942515c341ef56f16c680
...
fetch< ready
fetch< 0001

# Packfile over sideband (progress + data multiplexed)
fetch< packfile
sideband< \2Enumerating objects: 16, done.
sideband< \2Compressing objects: 100% (9/9), done.
sideband< PACK ...                  # (binary packfile)
sideband< \2Total 16 (delta 7), reused 14 (delta 7), pack-reused 0

This is pretty sick. The server only sends exactly the data needed, and nothing more. Sync is a hard problem, and this solution is elegant from the client's side.

The main downside is that crawling the commit graph to figure out what needs to be sent is expensive for the server. Lots of very smart folks have done a lot of work to optimize this (GitLab has a cool doc on some of the work they did), but the first graph crawl of large repos will almost always be painful.

If you were looking closely, you might have noticed this specific line:

sideband< PACK ...                  # (binary packfile)

Once the server figures out what objects you're missing, how does it actually send them? That's where packfiles come in.

packfiles are really cool, actually

Git has four kinds of objects:

Blobs (raw file contents)
Trees (directories pointing to blobs and other trees)
Commits (metadata + pointers to a root tree and parent commits)
Tags (human-readable labels pointing to any object)

Put those together, and you get the full history of a project.

Think of these objects like this:

Tag (v1.0.0)
 └── Commit (abc123)
     ├── Author:   Billy <billy@example.com>
     ├── Committer: Billy <billy@example.com>
     ├── Message:  "Add src/util.c"
     ├── Parent:   (789abc)
     └── Tree (def456)
         ├── Blob README.md   (sha1:111111)
         ├── Blob main.c      (sha1:222222)
         └── Tree src/ (ghi789)
             └── Blob util.c  (sha1:333333)

Object files are normally just loose files in .git/objects/. Sending all those objects over a network individually would be expensive and inefficient. Packfiles are Git's solution: they're like compressed tarballs for Git objects, but smarter. They compress, deduplicate, and send diffs instead of whole files.

Here's a toy example. Suppose you had a blob with the contents:

Hello world

And then you commit a change to make it:

Hello universe

Git can send a delta that says something like:

reuse characters 0 through 6 ("Hello ") of the old blob
insert "universe"

Deltas also stack on top of each other, by default up to a maximum of 50 before Git just creates a new object. When you have hundreds or thousands of commits, this can save a ton on network bandwidth.

Packfiles also come with pack index files that map object IDs to their byte offsets inside the (sometimes very large) binary pack. Instead of scanning through a multi GB packfile, Git uses the index to find exactly what it needs. For example, when reversing deltas to create the final file, the index points Git to each delta.

Put it all together, and you can clone the Linux kernel, with millions of commits and gigabytes of history, in a single highly compressed packfile.

Git's protocol sorta sucks

Before I give any critiques, I can't stress enough that I think Git's protocol is genuinely beautiful. Git isn't trying to offer a complete development workflow; it's trying to offer a distributed version control system (VCS). With that goal, it's more than succeeded. It's the most popular VCS by far, and for good reason.

However, we can also be honest and say that software development has changed since 2005. Think of the rest of this section as a wish list of what I'd want from a future VCS.

Git & modern development

At its core, Git only has two concepts: objects and refs. That simplicity is excellent for a content addressable storage system, but insufficient for modern software development. Reviews, pull requests, issues, and CI are all a major part of modern development. None of these exist in the protocol itself. We end up naming everything by branch names, then layering meaning. While the client speaks "blobs/trees/commits," the rest of the world speaks "did CI pass and do two reviewers with CODEOWNERS approval sign off?"

Authorization has a smaller but still notable gap. The protocol can say "you can fetch" and "you can push to these refs," and that's about it. Real policy-required reviews, linear history, protected paths, merge queues, and status checks all live outside the protocol. When a push is rejected, the wire just says "no." Was it a missing approval? A failing check? A merge queue gate? Wrong trailer? You have to click around a UI to learn the actual reason. We've built sophisticated policy engines around Git, but the client gets a shrug.

Out-of-band CI workflow

CI exposes a different hole. There's no first class handshake like "prove this commit is green before I accept it" or "block until my queued build finishes." Instead, hosts stitch it together with webhooks and REST calls: you push, runners start, statuses get attached, dashboards turn red or green. It works, but it's out of band. The transport doesn't know your push is gated on a pipeline, so the workflow devolves into: push -> alt-tab -> browser -> refresh -> repeat.

The protocol stays simple and stable; everything that makes software development feel modern happens off to the side.

FIN

At the end of the day, Git’s protocol is still really cool 😎. The whole design is built on a handful of simple ideas: content addressable objects, stateless request/response exchanges, and packfiles to make distribution efficient. Those ingredients helped to create the most popular VCS on the planet. The protocol is simple, extensible, and elegant.

But it’s also clear that Git, by itself, isn’t the whole story anymore. Almost everything that makes modern development “modern” happens outside the protocol. Git’s wire format just shrugs and says: “objects and refs.” Hosting providers, plugins, and extensions do the rest. That separation is both a blessing (Git stays stable and compatible) and a curse (every workflow feels bolted on).

So if you care about sync or VCS protocols, Git is both an inspiration and a challenge. Maybe the future is Git plus layers. Maybe it’s a brand new VCS that takes CI, reviews, and policy as first-class citizens. Either way, Git is here to stay for a while, and it's for a good reason.

Written by Billy @ Depot

Collaborating with Claude on docs

The Charmizard — Fri, 26 Sep 2025 13:20:38 +0000

You've built a new feature, tested it, and are ready to finally ship it. You check in with the team to make sure everything is good to go before launching it for customers and then someone asks “Do we have docs for this?”

Documentation often becomes an afterthought in software development, so I want to explore using a Claude agent to help me identify documentation gaps and generate initial drafts for the missing documentation.

I’m using the Depot Claude remote sandboxes, because I think being able to return to my session and share it with the team will be useful. For more information on these, check out this quickstart guide.

The workflow

Understanding the change

I want to start by making sure that Claude understands the change being made. For the simplicity of this example, I’ll choose a small change represented by a single commit to try out the workflow. I’m going to start a Depot Claude session and provide my own session id so it’s easy to reference again when I next resume it.

depot claude \
  --org ORG_ID \
  --repository https://github.com/depot/cli.git \
  --branch main \
  --session-id doc-hiding-summaries \
  "Can you explain the changes from this commit: 4febf18142a6bfafaaad7258b9c34f2cb251cb07"


✓ Claude sandbox started!
  Session ID: doc-hiding-summaries
  Link: https://depot.dev/orgs/ORG_ID/claude/doc-hiding-summaries

Going to the provided link, I can see that Claude has a good understanding of this change. For larger changes, you might need more back-and-forth to ensure alignment, but this single commit was straightforward enough.

Check for documentation gaps

From here, I basically want to know whether or not this is already documented. If so, I may want to expand the documentation given this new change. If it’s not present, I want to add it in general. So we can resume the session and ask it to pull down another repo that hosts our docs and check for us.

depot claude \
  --org ORG_ID \
  --resume doc-hiding-summaries \
  "I want to check if there is already documentation for this feature in our docs repo. \
  Can you pull down the docs repo's main branch and check? \
  The repository is here: https://github.com/depot/docs.git"

It looks like we need to make sure the agent has permission to clone the repository. I’ll ask it to do this again but this time provide the --allowedTools flag to specify the tools it’ll be allowed to use so that we don’t run into access issues.

depot claude \
  --org ORG_ID \
  --resume doc-hiding-summaries \
  --allowedTools "Bash(git:*)" \
  "I want to check if there is already documentation for this feature in our docs repo. \
  Can you pull down the docs repo's main branch and check? \
  The repository is here: https://github.com/depot/docs.git"

The --allowedTools "Bash(git:*)" flag grants Claude permission to use git commands, which is safer than bypassing all permission checks.

Based on the output, I can see that it successfully cloned the docs repo and checked to see if this feature was already documented (it was not). Great, so we should probably add some new documentation about this environment variable; not just this particular change, but for its usage in general.

Generate a draft

At this point, I think we know enough to start generating some documentation. I’m going to ask Claude to look at holistically how that environment variable is used to get a full understanding of it. I then want it to pick a place in the docs to add some of this documentation and pitch the change it would make. I’m still pretty hands on when it comes to working with AI as it’s still a newer workflow for me, so I like to see the plan before we make changes.

depot claude \
  --org ORG_ID \
  --resume doc-hiding-summaries \
  "Since there is no documentation for this environment variable at all, can you look at more generally how it's used and what it can do? \
  Then I'd like you to find a good place in the docs repo where you think it would make the most sense to add this documentation. \
  And finally pitch the documentation change you think should be made based on your findings."

I can see that Claude was able to successfully get a more holistic picture of how to use this environment variable, and it was able to find a good place for it. I disagree a bit with where Claude placed it on the page, but overall that seems mostly good to me. I’m going to provide a bit of feedback before doing anything else.

depot claude \
  --org ORG_ID \
  --resume doc-hiding-summaries \
  "I like what you've done. \
  Instead of an Environment Variables section after authentication, can you create an FAQ section at the bottom of the page where you generally answer how to minimize the build output and rewrite this in that format"

This seems reasonable to me as a starting point and now I want to get feedback from others.

Back into the codebase

At this point, I will ask Claude to checkout a branch, commit this change, and then create a draft pull request to get this back into the codebase.

depot claude \
  --org ORG_ID \
  --resume doc-hiding-summaries \
  --allowedTools "Edit" "Bash(git:*)" "Bash(gh pr:*)" \
  "Can you create a branch, commit this change, and then create a pull request in the docs repo with this. \
  I'd like you to make sure the link to this sandbox is in the description: https://depot.dev/orgs/ORG_ID/claude/doc-hiding-summaries in case anyone needs further context."

I have a pull request now which is great and I can see the changes. This is also the point where CI checks are running and I can see that there are some formatting and lint errors. I’ll ask the agent to use our package json scripts to format these changes.

depot claude \
  --org ORG_ID \
  --resume doc-hiding-summaries \
  --allowedTools "Edit" "Bash(git:*)" \
  "Can you check the package.json and run the formatter"

I officially have a PR with these changes and they're passing all the checks, ready for iterating on and formal review.

Now it's time to work with the team to make sure the docs are right and follow our normal standards.

Team collaboration and context sharing

This is where the collaborative aspect of Depot sandboxes becomes really valuable. When a teammate needs context about the code I'm documenting, they can access the same Claude session via the shared URL. All the context, the original change analysis, repository exploration, and reasoning behind documentation decisions are preserved and easily viewable by others.

For example, if a reviewer questions why we placed the documentation in the FAQ section rather than a dedicated environment variables page, they can ask Claude directly in the same session. Claude has full context of the decision-making process and can explain the reasoning or help iterate on alternatives.

This collaborative aspect eliminates the typical back-and-forth of "Why did you document it this way?" or "Can you explain what this feature actually does?"

A more complex experiment

The first workflow I tried used a very tiny feature that was entirely undocumented. I want to try this out on a larger feature that was built with a lot more changes and iterations and has some partial documentation. My workflows tend to be more hands-on as I dip my toes into incorporating agents into my workflow, but if you wanted to experiment with more free reign and complex changes you could try a prompt like this:

depot claude \
  --org ORG_ID \
  --repository https://github.com/depot/cli.git \
  --branch main \
  --session-id docs-claude-sandboxes \
  --allowedTools "Edit" "Bash(git:*)" "Bash(gh pr:*)" \
  "We have recently added changes to the depot claude command. \
  It now defaults to starting claude in a remote sandbox rather than using a local claude on your local machine. \
  Can you validate that we have docs for this in the docs repo: https://github.com/depot/docs.git and then add or update them and generally make sure they are up to date. \
  I want to show examples both with how to start the depot claude locally and remotely. \
  Then please create a branch and commit this change and then create a draft pull request so that i can review it"

I tried it, and ultimately this ended up being faster than manually looking through and writing these docs myself. I have tweaks I want to make and this provides a great starting point that I can iterate from. The hardest part of most of these workflows was making sure I had the correct --allowedTools configured.

When this workflow shines

Based on this experiment, I've found this approach works particularly well for:

Complex features spanning multiple files: Claude can analyze the entire codebase to understand feature scope and usage patterns.
Cross-repository documentation: When your docs live in a separate repo from your code, Claude can work across both seamlessly.
Post-development documentation: Great for that common scenario where features are built first and docs come later.

Limitations

Currently this interaction is limited to using the CLI so there’s no way to kick off these remote sessions via the UI; there’s a lot of back and forth between the terminal and the UI. This may be improved upon in the future, but it’s the current state of the world.

Additionally, the workflow requires some familiarity with the --allowedTools syntax to grant appropriate permissions safely. You can use the --dangerously-skip-permissions as an alternative here but, as the flag name implies, this may be risky. You'll need to experiment with permissions based on what actions you want Claude to take.

It’s a great starting off point but at this stage, you likely still need human intervention to make sure the docs created are in the right place and capture the right context for what you want documented. These models do a great job but it’s still a good idea to double check everything and push back if something seems incorrect.

Final thoughts

This is my first time experimenting with this workflow and I wanted to see how I could use these sandboxes to improve my process of adding docs after I’ve finished with a feature. I could see this being especially useful with not-so-tiny changes or when you have different people working on the features than writing the documentation.

There are a lot more iterations of this workflow that can be explored. Some interesting possibilities:

make this a part of PR process that gets kicked off by a code change
add context to a particular session over time during development
point the agent to an existing style guide to ensure consistent documentation

Want to try this yourself? Pick a small feature that needs docs, set up a Claude session with depot claude, and walk through the workflow.

Written by Iris @ Depot

Using AI as my engineering copilot (not autopilot)

The Charmizard — Tue, 09 Sep 2025 13:29:09 +0000

AI has suddenly become accessible in ways that actually matter for my day-to-day engineering work. I'm not just telling AI to build something and hoping for the best. I still want control over how things are implemented, especially when I'm the one who has to maintain the code.

Let me share how I've integrated AI (primarily Claude) into my workflow while keeping my hands firmly on the wheel.

The problem with "just make it work"

I recently asked Claude to help with a feature involving switching between table and filtered views on a GitHub Actions log detail page. Technically, it worked. But the implementation? Lots of useEffect hooks that would trigger each other to achieve the final result. While it worked, it was a lot more complex to read and reason through which can make it easier to introduce unexpected bugs and behaviors in future changes. If I was reviewing this code in a PR, I would leave comments to see if there was a more straightforward way of achieving this result. There was and with the guidance of "let's do this in a more declarative way that doesn't rely so heavily on useEffects ...", I was able to get a much cleaner solution.

Making something work isn't always enough. As engineers most of our time is spent reading code, not just writing it. If the logic is convoluted and hard to follow, we've solved one problem while creating another.

How I actually use AI

Navigating unfamiliar territory

At Depot, we have multiple codebases, and I'm definitely more familiar with some than others. When I need to dive into unfamiliar code, I don't ask Claude to "explain this entire codebase" since that's too broad to be useful. Instead, I focus my questions:

"I'm looking for where validation happens for this specific type of request. What are all the different contexts?"
"I found something in this file. Is this relevant to what I'm looking for? What does it actually do?"

While Claude is super powerful, it's not 100% always right, and I often find myself pushing back: "You said this, but I found this other thing that seems to contradict it. Am I misunderstanding?" About half the time, Claude responds with some version of "You know what, you're right." It can be a great starting point while you find your bearings in an unfamiliar codebase allowing you both to learn as you go, but it's always useful to double check.

Pairing with a very fast typist

I come from a background of formal pair programming where someone is the driver who primarily handles typing and making the changes, and the other is the navigator who guides the direction of what changes are made. With Claude's interactive mode, I've adapted this approach. I'm the navigator; Claude is the driver.

Here's how it works: I'll start with "Here's a page I want as a boilerplate. Create a new one based on this but just render hello world." Then: "Now fetch this specific data from our database" (with details about the where clause and which table to use).

Every step, I'm reviewing the code, making sure it makes sense. The difference from human pairing is the 30-second wait while Claude types, but the principle is the same. I'm guiding the implementation based on how I would build it, just faster.

Sometimes Claude tries to be "helpful" and re-adds things I deliberately removed. So I've learned to say: "Hey, I've made some changes. Start from here and move forward."

In general, I've found using interactive mode to review each change along the way to be useful in making sure Claude is on the right track. This gives me opportunities to tweak direction and also try different implementations out along the way.

The tedious stuff

AI really shines here:

Comparing log files to spot timing differences or anomalies (so much faster than my human eyes)
Making a specific component generic so I can reuse it
Finding every place in the codebase where we update certain records so I can add a new field consistently

For verification, I'll implement something myself, then ask: "I intended to do X. Did I miss anywhere?" It's like having a second pair of eyes that never gets tired.

Data visualization and queries

We've set up MCP to give Claude read-only access to some databases. Now I can ask: "Have we seen an increase in build failures in the last hour?" Claude figures out which tables to query, writes the SQL, executes it, and gives me a graph with the most recent failing builds.

There are still times when I'm writing one off queries, but Claude does a great job especially across multiple databases pulling relevant data to help me identify and visualize any trends that may be relevant.

Workflow intelligence

These agents help with:

Providing insight into the key changes made since the last time I looked at some code.
Acting as an automated code reviewer that catches when my PR description says one thing but the implementation does another (great at finding typos, occasionally catches real issues)
Generating test cases based on existing code (though you still need human judgment about what's worth testing)

All of these workflows work great, but they have one major limitation.

Remote agents for holding context

One of the limitations with using Claude sessions is losing context from session to session. Switching machines and switching context can result in starting from scratch since it doesn't remember the previous conversations or decisions made along the way.

This is where Depot's remote agent sandboxes can help. Instead of running Claude locally, it runs in a persistent cloud environment with full access to your project files, git history, and terminal.

This can be useful when I want to tell it to periodically do something without having to keep providing the same context, and I don't need the interactive mode. You can get started with our Claude Code quickstart guide.

The bottom line

AI has been a very powerful tool to accelerate my work, not replace my thinking. It's been helpful with understanding unfamiliar concepts and codebases while also still enabling me to have control over the architecture and implementation of changes I make. It can do a lot of the tedious work for you and implement a surprising amount of changes on its own, but it can work even better to create maintainable solutions when you provide insight and guidance along the way.

The key is maintaining that balance: letting AI handle the tedious parts while keeping control over the architecture, patterns, and maintainability of the code. Because at the end of the day, I'm not just trying to make something work. I'm building something that my team and I will need to understand, modify, and maintain for months or years to come.

FAQ

What's the difference between AI as a copilot vs autopilot?

Using AI as an autopilot means letting it make architectural decisions and write code without oversight. As a copilot, you guide the implementation while AI handles the mechanical work—like having a very fast typist who follows your directions.

How can I share Claude Code sessions with my team?

Depot's Claude Code sessions allow you to share and resume AI coding sessions across your entire team. Anyone in your organization can pick up where you left off, making collaboration with AI agents seamless.

Can I run Claude Code in CI/CD pipelines?

Yes! With Depot's remote agent sandboxes and Claude Code in GitHub Actions, you can run AI coding agents in your CI workflows. This enables automated code reviews, updates, and maintenance tasks directly in your development pipeline.

What are remote agent sandboxes?

Remote agent sandboxes provide persistent cloud environments where AI agents like Claude Code can work with full filesystem context, Git integration, and project history. No more starting from scratch—your AI agent maintains context across sessions.

Written by Iris @ Depot

From Go code to container image with Depot API

The Charmizard — Thu, 28 Aug 2025 16:33:28 +0000

In this post, we're going to create a more specialized container building system using the lower-level Depot API Go SDK. This is a more advanced use case that builds on our previous blog post. Here, we are going to create a tool that gets tar content and builds a container.

How Depot's build flow works

With the Depot API, there are three steps to build a container.

Register a build with Depot.
Acquire a Depot build machine.
Configure, build, and push a container.

Before you start

Before you start, ensure you have a Depot project setup and an organization API token generated.

Install the Go package

To start, let's add these imports and run go mod tidy:

import (
    "context"
    "encoding/json"
    "log"
    "os"
    "time"

    "github.com/depot/depot-go/build"
    "github.com/depot/depot-go/machine"
    cliv1 "github.com/depot/depot-go/proto/depot/cli/v1"
    "github.com/moby/buildkit/client"
    "github.com/moby/buildkit/session"
    "github.com/moby/buildkit/session/upload/uploadprovider"
)

Register a build

First, we request a new build with the Depot API. This request associates a build with a specific project's isolated cache. When registered, the Depot API will respond with the new build's ID and a one-time build token. We defer reporting the build result, buildErr, to the API. Any non-nil error is a build failure.

token := os.Getenv("DEPOT_TOKEN")
project := os.Getenv("DEPOT_PROJECT_ID")
req := &cliv1.CreateBuildRequest{
    ProjectId: project,
}
build, err := build.NewBuild(ctx, req, token)
if err != nil {
    log.Fatal(err)
}

var buildErr error
defer build.Finish(buildErr)

Acquire a builder machine

Next, we use the build ID and build token to request a new ephemeral builder machine.

var buildkit *machine.Machine
buildkit, buildErr = machine.Acquire(ctx, build.ID, build.Token, "arm64")
if buildErr != nil {
    return
}

Since you're interacting directly with the underlying BuildKit in Depot, you must remember to release the machine when the build completes. By default, machines stay on for two minutes after a build to handle additional builds.

defer buildkit.Release()

Connect to BuildKit

Once we have the BuildKit connection from the Acquire call, we connect to the machine’s BuildKit using mutual TLS. Once connected we can use all the power of BuildKit in Depot.

var buildkitClient *client.Client
buildkitClient, buildErr = buildkit.Connect(ctx)
if buildErr != nil {
    return
}

Configure the build

We create a BuildKit solver configuration. BuildKit conceptualizes builds as a graph of operations that are solved. In this case, we are going to configure the solver to receive a tar that contains the Dockerfile and all the needed content to run the build.

/*
 * howdy.tar.gz is a compressed tar archive that contains the Dockerfile and
 * any other files needed to build the image.
 */
r, _ := os.Open("howdy.tar.gz")

uploader := uploadprovider.New()
// Special buildkit URL for HTTP over gRPC over gRPC.
contextURL := uploader.Add(r)

solverOptions := client.SolveOpt{
    Frontend: "dockerfile.v0",
    FrontendAttrs: map[string]string{
        "platform": "linux/arm64",
        "context":  contextURL, // The tar file
    },
    Session: []session.Attachable{
        uploader,
    },
}

We provide BuildKit with a session attachable uploader. We need to explain BuildKit's rather unique solutions to client/server communication. The reasons for this design choice are unclear. The "what" is a method for a server to request content from a client.

An attachable is a BuildKit concept for a gRPC service hosted on the client on top of a bi-directional gRPC stream. This client-hosted service can be called by the server. In other words, the server tunnels gRPC requests back to the client.

BuildKit's uploadprovider takes this tunneling approach further by layering HTTP/1.1 requests on top of the gRPC-over-gRPC connection. As you can imagine, this layering of networking can be brittle to network drops and thus build retries are not uncommon.

┌────────────────┐                                                             ┌────────────────┐
│Buildkit Client │                                                             │ Buildkit Server│
└────────────────┘                                                             └────────────────┘
         │                                                                              │
         │                                                                              │
        ┌┴─┐   Layer 1: Client calls Server Session Service                          ┌──┤
        │  │───(Bidirectional gRPC)────────────────────────────────────────────────▶ │  │
        │  │                                                                         │  │
        │ ┌┴─┐                   Layer 2: Server calls Client hosted Upload Service  │  │
        │ │  │ ◀────────────────────────────────────(Bi-Di gRPC tunneled over gRPC)──│  │
        │ │  │                                                                       │  │
        │ │  │                                                                       │  │
        │ │  │                                      Layer 3: Server asks for upload  │  │
        │ │ ┌┴─┐ ◀───────────────────────────────────(HTTP/1.1 over gRPC over gRPC)──│  │
        │ │ │  │                                                                     │  │
        │ │ │  │                                                                     │  │
        │ │ │  │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ▶ │  │
        │ │ └┬─┘                                                                     │  │
        │ │  │                                                                       │  │
        │ │  │─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─▶ │  │
        │ └┬─┘                                                                       │  │
        │  │                                                                         │  │
        │  │ ◀ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ └─┬┘
        └┬─┘                                                                           │
         │                                                                             │
         │                                                                             │
         ▼                                                                             ▼

Nevertheless, we serve the tar file from the client to the BuildKit server over this protocol.

Stream build output

Next, we can run a goroutine that prints the output of each step of the build.

buildStatusCh := make(chan *client.SolveStatus, 10)
go func() {
    enc := json.NewEncoder(os.Stdout)
    enc.SetIndent("", "  ")
    for status := range buildStatusCh {
        _ = enc.Encode(status)
    }
}()

BuildKit will return status messages for each step of the build. Here, we're printing them as JSON to stdout.

Build the container

Lastly, we ask BuildKit to solve our request to build the container. The build will reuse cached steps stored in the project cache.

_, buildErr = buildkitClient.Solve(ctx, nil, solverOptions, buildStatusCh)
if buildErr != nil {
    return
}

With this example, you could build a service that receives a tar and creates an image.

Conclusion

With Depot's Go SDK, you can build sophisticated container building services without managing BuildKit infrastructure yourself. This approach gives you the performance benefits of Depot's optimized build machines—including native ARM64 support, persistent caches, and sub-second machine provisioning—while maintaining full control over the build process.

The pattern we've explored here powers many production services that need programmatic container builds: CI/CD platforms, development environments, and SaaS applications that containerize user code. By leveraging Depot's API, you get enterprise-grade build performance without the operational overhead.

Ready to get started? Generate an organization API token and explore our API documentation to begin building your own container services.

FAQ

What is Depot's Go SDK?
Depot's Go SDK provides low-level access to BuildKit infrastructure for programmatic container builds. It allows you to build Docker images from your applications without managing BuildKit servers yourself.

When should I use the API instead of the CLI?
Use Depot's API when you need to build container images programmatically, such as in SaaS platforms that package user code or CI/CD systems that need custom build logic. The CLI is better for interactive development workflows.

Do I need to manage BuildKit infrastructure?
No, Depot handles all BuildKit infrastructure including machine provisioning, scaling, and maintenance. You get ephemeral build machines that spin up in seconds and automatically clean up after builds.

How does caching work with the API?
Each Depot project gets isolated cache storage that persists across builds. The BuildKit solver automatically reuses cached layers, dramatically speeding up subsequent builds with similar content.

Written by: Chris Goller @ Depot

Guide to faster Rust builds in CI

The Charmizard — Fri, 08 Aug 2025 20:51:12 +0000

We’ll be building the Zed project, a code editor written in Rust. Zed is a large codebase with many dependencies, making it a good candidate for exploring build optimizations. We'll also be starting off using Depot's GitHub Actions runners, which have a ton of optimizations already implemented. I would know, I worked on some of them!

Just for fun, I built Zed with GitHub’s equivalent runner. We ran the same baseline workflow on ubuntu-latest-8. The GitHub runner workflow took 43m 16s - a staggering 61.1% slower than Depot’s 26m 51s baseline. The easiest optimization you can make to your CI workflows is to use Depot!

Toolchain optimizations

Mold linker

First, let’s work on optimizing the Rust toolchain itself. The fewer changes we need to make to this codebase, the better. We can start by replacing the default linker (lld on Linux) with mold. This linker is designed to be really fast, especially for projects with large build outputs.

Results

The mold linker showed negligible results for the Zed codebase. When applied only to release builds (not tests), the mold workflow completed in 28m 49s - just 0.7% slower than baseline.

Test execution took 15m 27s and release builds took 11m 35s. The minimal difference doesn't justify the added complexity of configuring a different linker for the Zed codebase.

Nightly features

The nightly Rust compiler has several features that can help to speed up builds, if you’re willing to switch Rust compiler versions. The two that we’ll use in this example are:

-Z share-generics: This feature allows the compiler to share generic code across different compilation units, which can significantly reduce build times for large projects.
-Z threads=8: This feature allows the compiler to parse files and expand macros in parallel

Results

Switching to Rust nightly provided significant benefits. The nightly workflow completed in 26m 30s - a 7.3% improvement over baseline.

Total time: 26m 30s (7.3% faster than baseline)
Test execution: 15m 43s (similar to baseline)
Build time: 9m 1s (22.7% faster than baseline)

The nightly features (-Z threads=8 and -Z share-generics=y) dramatically improve build times by allowing the compiler to do more work concurrently and avoid redundant compilation of generic code.

Caching strategies

Caching dependency downloads

Let’s get into the real meat of speeding up Rust builds: caching. Cargo, the Rust package manager, can download hundreds of dependencies before compiling them. This can be slow, especially for large projects with many dependencies. By caching these downloads, we can speed up build times significantly.

Results

Caching Cargo dependencies showed good improvements when combined with nightly features. The cargo caching workflow with nightly completed in 26m 18s - 8.0% faster than baseline.

By caching the ~/.cargo directories between runs, we avoid re-downloading and re-indexing hundreds of crates.

Sccache

sccache is a compiler cache for Rust that can significantly speed up builds by caching compiled artifacts. It works by storing the output of compilation in a local or remote cache, so that subsequent builds can reuse these artifacts instead of recompiling everything from scratch.

Depot Cache has built-in support for sccache storage, allowing you to share cached artifacts across different CI runs and local development environments. Using the depot CLI, you can easily set up sccache to use Depot as its cache backend by running the depot cargo command.

Results

Using depot cargo with sccache showed promising results with a warm cache:

Cold cache: 33m 23s (16.8% slower than baseline, 26.8% slower than previous optimization)

Test execution: 13m 18s (12.3% faster)
Build time: 14m 54s (27.7% slower)

Warm cache: 25m 18s (11.5% faster than baseline, 4.0% faster than previous optimization)

Test execution: 13m 12s (12.9% faster)
Build time: 10m 48s (7.4% faster)

With a warm cache, sccache provides meaningful speedups for both test and build times.

Compiler backend alternatives

Compiler backends?

For those of you who aren't compiler nerds, rustc uses LLVM as its backend for machine code generation. You can think of LLVM as a library that the Rust compiler uses to generate machine code for different architectures. However, LLVM can be slow to compile large projects, since it’s designed for optimized machine code first, and compile speed second. This is where alternative backends come in.

Cranelift for debug builds and tests

Cranelift is an alternative backend for the Rust compiler, designed to compile fast enough to be used in development iterations. It trades off some runtime performance for faster compile times, making it a great choice for debugging and testing builds.

Results

Cranelift fails to compile the Zed codebase on both stable and nightly Rust. The cranelift workflow shows:

error: asm! and global_asm! sym operands are not yet supported
error: could not compile `wasmtime-fiber` (lib) due to 1 previous error

This is a known limitation of Cranelift - it doesn't support all LLVM features, particularly inline assembly. Many Rust projects that depend on low-level crates (like wasmtime in Zed's case) cannot use Cranelift as a drop-in replacement. This limitation exists regardless of whether you use stable or nightly Rust.

For now, we'll revert this change. While Cranelift can provide dramatic speedups for projects it supports, it won't work for our example codebase.

CI-specific optimizations

Cargo nextest with matrix builds for massive parallelization

cargo-nextest is a next-generation test runner for Rust that can bring significant performance improvements over the default cargo test. The main advantage is that it runs each test in its own separate process, allowing us to run many tests in parallel.

Results

Cargo nextest showed excellent performance with a warm sccache. The nextest workflow completed in 18m 34s - a 35.0% improvement over baseline. Test execution took 10m 46s, which is 28.9% faster than baseline.

Nextest delivers substantial performance improvements when combined with warm sccache, making it an excellent choice for CI pipelines.

Architecture optimization

The single biggest impact on build times is often the hardware you’re running on. More CPU cores generally mean faster builds, especially for highly parallelizable tasks like Rust compilation. Doubling your CI costs isn’t always an option, of course.

Results

To test the impact of more CPU cores, we doubled the runner size from 8 to 16 cores. The nextest-16-cores workflow results were interesting:

Total time: 21m 25s (25.1% faster than baseline)
Test execution: 7m 31s (50.3% faster test execution than baseline)
Build time: 11m 54s (slower than 8-core due to cache warmth differences)

The 16-core configuration showed excellent test parallelization (7m 31s vs 10m 46s on 8 cores), but the overall time was impacted by sccache variability. This demonstrates that raw CPU scaling isn't always the answer - cache effectiveness can be more important than core count.

Platform-specific optimizations

Linux I/O optimizations with RAM disks

RAM disks provide a filesystem backed by memory instead of persistent storage, offering dramatically faster I/O operations. For Rust builds, this can significantly reduce the time spent reading and writing intermediate compilation artifacts.

Our runners reserve some memory for a RAM disk. When a job starts, VMs will automatically create and mount this RAM disk to the runner, which helps to make every I/O operation, including reading and writing files for compilation, much faster.

For self-hosted runners or other CI providers:

sudo mkdir -p /mnt/ramdisk
sudo mount -t tmpfs -o size=16G tmpfs /mnt/ramdisk

# Alternative: ramfs (pure RAM, no size limits - use with caution!)
# sudo mount -t ramfs ramfs /mnt/ramdisk

# Configure Rust to use it
export CARGO_TARGET_DIR=/mnt/ramdisk/target
export TMPDIR=/mnt/ramdisk/tmp

Summary

After testing various Rust build optimizations on the Zed codebase, here are the key findings:

Optimization	Total	Tests	Build	vs Baseline
GitHub Runner	43m 16s	~30m	~13m	+51.2%
Baseline (Depot)	28m 36s	15m 10s	11m 40s	-
Mold linker	28m 49s	15m 27s	11m 35s	+0.7%
Nightly features	26m 30s	15m 43s	9m 1s	-7.3%
Cargo caching	26m 18s	~14m	~10m	-8.0%
Depot cargo (warm)	25m 18s	13m 12s	10m 48s	-11.5%
Cranelift	Failed	-	-	N/A
Nextest	18m 34s	10m 46s	5m 46s	-35.0%
Nextest (16 cores)	21m 25s	7m 31s	11m 54s	-25.1%

Fin

The biggest wins came from using nextest with warm sccache (35.0% improvement on 8 cores) and combining multiple optimizations. Depot cargo with warm sccache provided an 11.5% improvement, while properly configured nightly features added 7.3%.

Ready to speed up your own Rust builds? Start with Depot's optimized runners and work your way through these techniques - your future self (and your CI bill) will thank you.

FAQ

What's the single most impactful optimization for Rust CI builds? Cargo nextest with warm sccache provides exceptional performance. We achieved a 35.0% speedup on standard 8-core runners.

Should I use sccache for all my Rust builds? It depends on your use case. sccache speeds up test compilation (11-14% faster) but can slow down release builds by up to 50%. Consider a hybrid approach: use sccache for tests and regular cargo for release builds.

Is switching to Rust nightly worth it for build performance? Yes, if configured properly. Nightly features like -Z share-generics=y and -Z threads=8 provided a 7.3% overall speedup with 22.7% faster build times in our testing. Make sure to actually pass these flags via RUSTFLAGS, not just install the nightly toolchain.

Why did Cranelift fail to compile the Zed codebase? Cranelift doesn't support all LLVM features, particularly inline assembly. Projects with low-level dependencies (like wasmtime) often can't use Cranelift as a drop-in replacement, despite its potential for dramatic speedups.

What's the easiest optimization to implement right now? Using cargo nextest. With warm sccache, this single change delivered a 35.0% speedup. It's a drop-in replacement for cargo test that runs tests in parallel more efficiently.

Author: Billy Batista @ Depot.dev

A practical guide to debugging GitHub Actions

The Charmizard — Fri, 25 Jul 2025 00:59:19 +0000

GitHub jobs fail for reasons that make no sense all the time. You haven't changed anything in your code, yet the workflow breaks anyway. It's frustrating because you just want your changes to build and deploy. Debugging action runners can be a headache, but here's how to start untangling what's going wrong.

GitHub Actions failures

GitHub and GitHub actions are a huge part of so many engineers' workflow. It’s something that people rely upon heavily in their day to day work life. We want to not have to think about how a job is getting run for the most part and just see everything go green so changes can be merged and deployed.

So when there are issues, it’s very disruptive to teams.

The team impact

Not only can issues with actions and jobs just generally be frustrating, but they can be pretty costly to your team. If it’s an error that doesn’t seem relevant to the change, a lot of people’s first reaction can be to re-run the failed jobs. For flaky issues, this may fix this particular instance of a problem, but it can also mean waiting a long time just for another failure to occur.

These failures, especially if a test suite is running on the job, may cause people to stop trusting that particular job / check. This may be a problem down the line if this failure is at some point identifying a real issue. Additionally, these failures might be occurring when a change needs to get in quickly. If you need to put out a quick fix for something and a job failure is causing you to be unable to deploy the change, this is going to be a big problem especially if this is during an incident or some other high stakes circumstance.

Debugging GitHub Actions

In many cases these jobs are running a test suite, so sometimes the problem is actually the change being made. Other jobs may end up with build failures which can also possibly be the fault of the change being made. However, what about the case where the problem isn’t actually the code change? The issue may be that the runner your job is executing on is running out of memory or utilizing all of its CPU. There is also the case where your jobs aren’t failing but they are taking longer to run but it’s not clear as to why.

Utilizing GitHub Logs

In the GitHub UI, you have the ability to look through the job logs to see what is going on with your job. In the case that there is a really explicit error, this UI does a good job at scrolling you to and highlighting what the error that caused the job to fail. This is often useful for identifying whether the issue is related to the code change.

If it’s not clearly related to the code change, it can be used as a starting off point to investigate what the actual issue is. Sometimes the error that causes the job to fail is not as obvious and requires more investigation to determine the problem.

Log point in time resource usage

When GitHub Actions jobs fail unexpectedly, memory exhaustion is often the culprit, but the symptoms aren't always obvious in the logs. Rather than guessing, you can add simple monitoring steps to your workflows that capture resource usage before and after critical operations. For memory diagnostics, adding free -h and df -h commands provides immediate visibility into RAM and disk utilization at key points in your build process. If CPU bottlenecks are suspected, logging system information with nproc, lscpu, and uptime reveals processor constraints and load averages. Wrapping expensive operations with the time command shows exactly where delays occur.

- name: Check memory usage
  run: |
    echo "Memory usage before:"
    free -h
    df -h

    # Your actual build/test steps here

    echo "Memory usage after:"
    free -h

- name: Monitor CPU usage
  run: |
    echo "CPU info:"
    nproc
    lscpu
    echo "Load average:"
    uptime
    # Time your critical steps
    time make build

This approach requires no external dependencies or API keys, just a few extra workflow steps that output directly to your job logs, making it perfect for quick debugging sessions or permanent monitoring of resource-intensive builds. The downside of this approach is that you now have to spend time searching through your logs to try and find where these values were outputted and also keep track of them if there are multiple of these outputs.

Utilizing an observability tool

Looking at logs is simple enough if you have very few locations you’re logging this data. You may want to collect this data more than just a single time which will require searching through GitHub logs to find all the instances of these values. While identifying the failure-inducing error in GitHub's logs works rather well, viewing other more mundane logs can be tedious and slow. The logs for each non-errored step begin in a collapsed view and don't return in search results until you uncollapse the step and then alter the filter. Typically to speed up this search, I utilize viewing the raw logs so I can more quickly search through them to find the information I seek, but this does forgo a lot of nice formatting that makes the logs easier to parse. A more practical alternative to trudging through these logs would be to send this usage data as metrics to an external observability platform.

Report point in time resource usage to an external observability tool

If you want to get these same point in time metrics but have them viewable beyond just searching through your GitHub logs, you could opt to collect and send these metrics to an external observability tool like DataDog for managing and viewing usage metrics you collect. For GitHub-hosted runners, you can implement point-in-time monitoring by sending custom metrics to observability platforms like DataDog during workflow execution. This involves capturing system metrics (memory usage, CPU load, disk utilization) at key moments such as before builds, after tests, or continuously during long-running processes and posting them via HTTP API.

Point-in-time metrics can be effective enough for identifying resource bottlenecks, tracking performance trends over time, and correlating system state with job failures. The implementation is straightforward: collect metrics using standard Unix commands, format them as JSON, and send via curl to DataDog's series endpoint with appropriate tags for filtering and analysis.

Here's a simple example that captures and sends memory utilization during a workflow:

- name: Send metrics to DataDog
  run: |
    # Simple curl commands
    MEM_USED=$(free | grep Mem | awk '{print ($3/$2) * 100.0}')
    curl -X POST "https://api.datadoghq.com/api/v1/series" \
      -H "DD-API-KEY: ${{ secrets.DD_API_KEY }}" \
      -H "Content-Type: application/json" \
      -d '{
        "series": [{
          "metric": "github.actions.memory.percent",
          "points": [[$(date +%s), $MEM_USED]],
          "tags": ["repo:${{ github.repository }}"]
        }]
      }'

Agent with a self hosted runner for continuous usage metrics

If you're using GitHub-hosted runners, you're limited to point-in-time metrics collection during job execution. However, self-hosted runners offer the opportunity for more continuous monitoring by installing observability agents directly on the runner infrastructure. For example with a DataDog agent running consistently on your self-hosted runners, you get baseline system metrics including CPU, memory, disk, network collected every 30-60 seconds. This provides much richer insights into runner performance and its resource utilization.

Install the agent on your runner:

DD_API_KEY=your_api_key bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"

Add a step to your workflow to tag the metrics emitted from the agent:

- name: Tag DataDog metrics for this job
  run: |
    # Add GitHub-specific tags to the runner's metrics
    curl -X POST "https://api.datadoghq.com/api/v1/tags/hosts/$HOSTNAME" \
      -H "Content-Type: application/json" \
      -H "DD-API-KEY: ${{ secrets.DD_API_KEY }}" \
      -d '{
        "tags": [
          "github_job:${{ github.job }}",
          "github_repo:${{ github.repository }}",
          "github_run_id:${{ github.run_id }}"
        ]
      }'

This is going to provide a lot of data that can be used to help identify your issues but is likely a more heavy handed approach than many would take. This requires you both to manage your own self-hosted runners as well as utilize an external observability tool. If you’re not already managing self hosted runners or utilizing an observability tool, this is likely a far too heavy and expensive approach for debugging this.

Our approach: built-in observability

Similarly to the last option, we use an agent for collecting job data. Our setup involves deploying an OpenTelemetry (OTEL) collector directly on Depot action runners that continuously captures observability data while the job executes. We store the following to be graphed and viewed in the Depot app:

Memory utilization metrics with sub-second granularity throughout the job lifecycle
CPU utilization metrics with detailed timing information
Complete job logs with step associations
Process-level resource utilization for granular visibility
Out of memory (OOM) events with detailed timing information

The goal of this approach is to collect the information that commonly provides insights around what the problem was without requiring users to make changes to their workflows or to adopt an observability or data visualization tool to access this data which is sometimes prohibitively expensive. Out of memory indicators provide very clear feedback that the problem is memory constraints. The peak resource utilization for CPU and memory values provide insights into whether or not one of these is a constraint on the system, ultimately being the culprit for the job failure or slowdown. Process level metrics for a particular step allow you to more deeply identify which process may be hogging resources and causing problems. Your job logs and their timings can be used to further identify where other slow downs occurred while also providing an easier interface for searching through and identifying relevant events.

Taking action

Now that you have more data to be able to identify what is actually causing the failures or slow downs, you can take targeted actions to alleviate the issue.

High memory usage

Immediate solutions

Upgrade runner types: Use a larger runner with more memory. This has the tradeoff of being more expensive.
Implement build caching: Cache dependencies and build artifacts to reduce memory overhead during subsequent runs.
Set memory limits: Prevent memory runaway by only allowing a limited amount of memory for a particular process.

Optimize resource-heavy operations

Docker builds: Use multi-stage builds and layer caching to reduce memory requirements.
Large jobs: Break monolithic jobs into smaller sequential jobs that consume less memory.

Identify memory leaks

If you see sudden memory spikes between "before" and "after" measurements, investigate:

Test suites that don't properly clean up resources
Build processes that accumulate temporary files
Applications that don't release memory between operations

CPU bottlenecks

Immediate solutions

Upgrade runner types: Use a larger runner with more CPU cores. This has the tradeoff of being more expensive.
Limit parallel operations: Avoid overwhelming the runner's CPU cores.

Split and distribute work

Job matrices: Use matrix strategies to distribute CPU-intensive work across multiple runners simultaneously.
Sequential jobs: Chain dependent jobs instead of running everything in parallel to reduce peak CPU load.

Conclusion

When job failures happen, they can be frustrating and time-consuming. Debugging them means understanding the context so you can separate the real cause from the noise. The key to effective GitHub Actions debugging is picking the right strategy for your team’s scale and needs.

Start simple with logging for one-off issues, and as your CI/CD usage grows, consider adding an external observability tool.

If you're already using Depot as your job runner, you get built-in metrics in the job details to kickstart this investigation. The next time a job fails unexpectedly, you'll be equipped to pinpoint whether it's a code issue or a resource constraint and fix it faster.

FAQ

Why do GitHub Actions jobs fail even when my code hasn't changed? Because many failures are caused by the environment, not your code. Common causes include resource limits (memory, CPU, disk), flaky network dependencies, outdated cache layers, or GitHub-hosted runner instability.

How can I tell if the failure is a real bug or just a flaky runner? Start by checking the logs for consistent errors. If rerunning the job produces different results or passes without changes, it's likely a flaky issue caused by resource constraints, timing, or network hiccups.

What's the simplest way to debug a flaky job? Add lightweight resource logging (free -h, df -h, uptime) around critical steps and compare runs. This shows if memory or CPU is maxing out. For deeper analysis, use an observability tool or a runner with built-in metrics (like Depot).

What's the fastest fix for 'No space left on device'? Use docker system prune to clean up unused layers, remove unnecessary artifacts, and ensure .dockerignore excludes large files. If space is still tight, upgrade the runner size.

Authored by Iris @ Depot

How we automated GitHub Actions Runner updates with Claude

The Charmizard — Thu, 17 Jul 2025 00:41:26 +0000

We recently launched Claude Code Sessions in Depot, a feature that allows you to share Claude code sessions with both developers and your CI workflows. In our previous blog post, we noted that "we've been using Claude Code at Depot since pretty much the moment it dropped," but we didn't elaborate on how. This article will demonstrate one of our most valuable CI uses: consistently keeping our forks updated.

Background

Problem: We waste hours updating the GHA upstream image

All of our GHA runners run on their own isolated EC2 instances. As such, we need to build an image that these runners can load, called an AMI. This sounds like it'd be pretty easy! Github keeps the definitions for their runner images open source, so it'd just be a matter of modifying their source to work with our runners, and then building the AMI. Unfortunately, it's not quite that simple. We make a large number of modifications to the image to help improve performance, and alongside that, Github's runners run on Azure and ours run on AWS. This means that, instead of just using their source as is, we need to modify it to work with our AMI build software. The way we go about this is by keeping upstream in a git submodule, and then running many of their scripts with a patch file that modifies them to work with our runners. This is a lot of work, and it takes a lot of time to keep up with their changes.

Keeping our fork in sync with upstream requires a pretty significant amount of developer effort that compounds over time. Each pull from upstream involves pulling in dozens of commits, reviewing every change for compatibility issues, and ensuring that everything introduced by upstream will be compatible with our existing runner software.

It isn't just the raw time investment that's a problem either, it's the cognitive load of context switching from other deep and intensive work to this task. Kyle, our co-founder and CEO here at Depot, talks about it frequently, but context switching is one of the biggest hidden killers of developer productivity. It shatters whatever flow you had going, and quite frankly, it just doesn't feel great.

For many months now, we've been wanting to automate this process. Generative AI is the hot new thing, and consuming large amounts of data to output smaller amounts is one area where LLMs can really shine.

What we needed Claude to do

There's two main issues we want to tackle here:

Keeping a patch file for arm64 up to date
Knowing what changed between each update

The first is keeping a file called arm64.patch up to date. Essentially, Github's runner images are built for x86_64, and we need to modify them to work on arm64. This patch file contains all the modifications we need to make to the upstream source to get it building for our arm machine images. We also need to generally know 'what actually changed since our last update?'. While there are many commits made to the upstream repository, not all of them are relevant to us. We need to be able to quickly identify which changes are important or potentially breaking, and which ones we can ignore. Using depot claude, we can make this process pretty smooth, by having a CI workflow that consistently keeps us up to date!

Solution: Using Depot Claude to summarize and analyze changes

Generating the arm64 patch

Let's get the more tedious work out of the way first: regenerating our arm64 patches. Thinking about what we need Claude to do, we need it to:

Attempt to apply the existing arm64 patch file
If it fails, modify the patch file to apply cleanly, keeping in mind the goal of ensuring this new machine image will build and run correctly on arm64
Repeat the first two tasks until the patch applies cleanly

Here's the actual prompt we're using as of writing, with a little bash to set up the context for Claude:

cat <<'EOF'
# ARM64 Patch Conflict Resolution

## Context
We maintain ARM64 patches that modify the upstream GitHub Actions runner image to work on ARM64 architecture. These patches are ESSENTIAL and must be preserved. The patches failed to apply cleanly after an upstream update, and I need you to fix the conflicts while keeping all the ARM64 modifications.

## CRITICAL: YOU MUST MODIFY THE PATCH TO WORK FOR ARM64
The ARM64 patches are necessary for the runner to work on ARM64. You must:
1. Keep ALL existing ARM64 modifications
2. Only adjust line numbers or context to match the new upstream code
3. NEVER suggest removing a patch file or patch content
4. ALL files that are patched MUST remain patched
5. **CRITICAL**: The new patch MUST make it so that, post-patch, things will run for ARM64

## Your Task
You need to modify the arm64/arm64.patch file to resolve conflicts AND ensure ARM64 functionality. The patch application is failing, which means the line numbers or context in the patch no longer match the upstream files.

**IMPORTANT**: You are NOT just copying the patch file. You must MODIFY it to ensure ARM64 compatibility.

To fix this:
1. Look at the current arm64/arm64.patch file
2. Check the upstream files that are being patched to understand how they've changed
3. Update the patch file with the correct line numbers and context
4. Ensure all ARM64-specific changes are preserved and functional:
   - Change amd64 → arm64, x86_64 → aarch64
   - Update download URLs to ARM64 versions
   - Modify tool configurations for ARM64 (e.g., JAVA_HOME paths)
   - Remove/disable features that don't work on ARM64
   - Ensure binary installations use ARM64 binaries
5. **VERIFY**: The final patch must result in a working ARM64 runner image

## Failed Files
The following issues were detected:
EOF
echo "$failed_files"

cat <<'EOF'

Please update the arm64/arm64.patch file to fix these conflicts. Remember:
- Keep ALL ARM64 modifications
- Only update line numbers and context
- Test that the patch applies cleanly
- DO NOT remove any patches or suggest workarounds that would lose ARM64 functionality
- **CRITICAL**: The new patch MUST make it so that, post-patch, things will run for ARM64
- You are MODIFYING the patch to ensure ARM64 compatibility, not just copying it
- You may not consider the task complete until `make` runs without issue
EOF

We have a shell script around running depot claude that essentially sets up the context needed, checks if the patch file needs to be modified at all, and then runs the prompt. Once Claude modifies the patch file, we can run it through our build process to verify that it applies correctly:

resolve_conflicts() {
    print_message "$YELLOW" "Attempting automatic conflict resolution with depot claude..."
    # grabs the error output from attempting to apply patches
    make_output=$(make 2>&1 || true)

    failed_files="${make_output}"

    prompt=$(create_prompt "$failed_files")

    print_message "$YELLOW" "Running depot claude to resolve conflicts..."

    session_id="arm64-patch-$(date +%Y%m%d-%H%M%S)"

    if ! depot claude --model opus --org cl0wyyk6k39487ebgraxasinja --session-id "$session_id" --allowedTools "Read Edit Write Grep Glob Bash" -p "$prompt"; then
        print_message "$RED" "depot claude failed to run"
        return 1
    fi

    print_message "$YELLOW" "Testing if conflicts were resolved..."
    if apply_arm64_patches; then
        print_message "$GREEN" "Conflicts resolved successfully!"
        if generate_patch_file; then
            print_message "$GREEN" "Patch file updated successfully!"
            return 0
        else
            print_message "$RED" "Failed to generate patch file after conflict resolution"
            return 1
        fi
    else
        print_message "$RED" "Conflicts still exist after automatic resolution attempt."
        return 1
    fi
}

One of the really nice things about using depot claude for this kind of task is that, after a workflow run, we can take a look at why it made certain decisions. That lets us tweak the prompt as needed to make further improvements.

⏺ Now let me check the upstream install-kubernetes-tools.sh file to see what has changed:

⏺ Read(upstream/images/ubuntu/scripts/build/install-kubernetes-tools.sh)
  ⎿  Read 50 lines (ctrl+r to expand)

⏺ Update Todos
  ⎿  ☒ Examine the current arm64/arm64.patch file
     ☒ Check the failing file install-kubernetes-tools.sh in upstream
     ☐ Analyze the reject file to understand the conflict
     ☐ Update the arm64.patch file with correct context and line numbers
     ☐ Test that the patch applies cleanly
     ☐ Run make to verify everything works

⏺ Let me check the reject file to understand exactly what failed:

⏺ Read(arm64/generated/scripts/build/install-kubernetes-tools.sh.rej)
  ⎿  Read 20 lines (ctrl+r to expand)

⏺ I can see the issue. The patch is trying to apply changes for minikube v1.34.0 with specific
  pinned version, but the upstream file now uses "latest" version with a different download pattern.
   Let me update the patch file to match the new upstream structure:

⏺ Update Todos
  ⎿  ☒ Examine the current arm64/arm64.patch file
     ☒ Check the failing file install-kubernetes-tools.sh in upstream
     ☒ Analyze the reject file to understand the conflict
     ☐ Update the arm64.patch file with correct context and line numbers
     ☐ Test that the patch applies cleanly
     ☐ Run make to verify everything works

⏺ Now I'll update the arm64.patch file to fix the kubernetes-tools section:

Finally, we run this bash script in a larger GHA workflow. It's set to run daily to ensure that we always have the latest patch file ready to go:

# this is a simplified version of the actual workflow file
name: Update Upstream

on:
  schedule:
    - cron: '0 17 * * *'

jobs:
  update-upstream:
    runs-on: depot-ubuntu-latest-small
    permissions:
      contents: write
      pull-requests: write

    steps:
      # ... a few setup steps
      - name: Update upstream submodule
        run: |
          cd "$WORKTREE_DIR"
          git submodule update --init --force

          CURRENT_COMMIT=$(git submodule status upstream | awk '{print $1}' | sed 's/^[+-]//')
          echo "Current submodule commit: $CURRENT_COMMIT"

          cd upstream
          git fetch origin main
          LATEST_COMMIT=$(git rev-parse origin/main)
          echo "Latest upstream commit: $LATEST_COMMIT"

          if [ "$CURRENT_COMMIT" != "$LATEST_COMMIT" ]; then
            echo "Update needed: $CURRENT_COMMIT -> $LATEST_COMMIT"
            git checkout main
            git pull origin main
            cd ..
            git add upstream
            echo "CHANGES_EXIST=true" >> $GITHUB_ENV
          else
            echo "Submodule already up to date"
            cd ..
            echo "CHANGES_EXIST=false" >> $GITHUB_ENV
          fi

      - name: Generate ARM64 patches
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          DEPOT_TOKEN: ${{ secrets.DEPOT_TOKEN }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          cd "$WORKTREE_DIR"
          bash .github/scripts/generate-arm64-patches.sh

          if [ "$(git status --porcelain arm64/ x86/ | wc -l)" -gt "0" ]; then
            echo "ARM64_PATCHES_UPDATED=true" >> $GITHUB_ENV
          else
            echo "ARM64_PATCHES_UPDATED=false" >> $GITHUB_ENV
          fi

      - name: Commit ARM64 patches
        if: env.ARM64_PATCHES_UPDATED == 'true'
        run: |
          cd "$WORKTREE_DIR"
          git add arm64/ x86/

          COMMIT_MSG="Update ARM64 patches for upstream changes"$'\n\n'"- Applied patches using make"$'\n'"- Regenerated patches using make generate-patch"$'\n'"- Resolved any patch conflicts automatically"

          git commit -m "$COMMIT_MSG"

      - name: Push changes
        if: env.CHANGES_EXIST == 'true'
        run: |
          cd "$WORKTREE_DIR"
          git push -u origin $BRANCH_NAME

This is already a major improvement to the productivity of Depot developers. Testing, iterating on, and applying the patch file was not only time consuming, but also required a lot of context switching. Now, we can just run this script and let Depot Claude handle the heavy lifting, with human developers just verifying Claude's output and making any necessary adjustments.

Analyzing breaking changes

Now let's go even further. The real heavy work with updating upstream involves reviewing all the changes made. Most of these are just version upgrades, or other minor details. The more serious changes can include:

Upgrades to the Linux kernel or other system libraries
Buildkit or Buildx upgrades
Major version upgrades of commonly used packages

Given this context, being able to ask Depot Claude to summarize the changes and analyze them for potential breaking changes could be a huge time saver. This would still, of course, require a human touch to verify that the changes are acceptable, but it would allow us to focus on the most important changes rather than having to sift through every single line of the diff.

One interesting problem is that there's been dozens of changes in the past that have caused issues. Having to update the prompt in git to constantly account for these changes would be a pain. Instead, we can create an original session file that all other breaking changes analyses work off of.

We already have a nice list of every issue that's been introduced in our Git history, so let's go ahead and feed that to a new depot claude session:

depot claude --session-id=update-upstream

╭───────────────────────────────────────────────────╮
│ ✻ Welcome to Claude Code!                         │
│                                                   │
│   /help for help, /status for your current setup  │
│                                                   │
│   cwd: /Users/billy/Work/github-actions-image     │
╰───────────────────────────────────────────────────╯

 ※ Tip: Run claude --continue or claude --resume to resume a conversation

> Here is a list of all previous upstream changes that have caused issues:
...
**Upstream Change**: Updated Docker Compose from 2.35.1 to 2.36.0/2.36.2
**Issue**: Version mismatch between different architectures and platforms
**Fixes**:
- `a341ec95` - fix: update ubuntu 22.04 docker compose to 2.36.0
- `711e1cf5` - fix: upgrade x86 docker compose to 2.36.0
- `9a0ad0d7` - fix: apply docker compose 2.36.0 to x86
- `4317f9c9` - fix: update arm64 and x86 patches (updated to 2.36.2)

**Issue**: PowerShell execution permissions broken since 7.4.2
**Fix**: `1c87a95f` - fix: pwsh issue since 7.4.2 by changing execution permissions
 ...
 Do you understand so far?

Next, we'll ask Depot Claude to summarize the changes in the upstream repository, and analyze them for potential breaking changes based on previous issues. We'll also ask it to provide its response in JSON, so that we can easily parse it later:

cat > "$PROMPT_FILE" << 'EOF'
Analyze the upstream changes and identify any breaking changes that might affect users of this GitHub Actions runner image.

Please respond with ONLY a JSON object (no markdown, no explanations before or after) in this format:
{
  "breaking_changes": [
    {
      "file": "filename",
      "description": "Brief description of the breaking change",
      "impact": "How this affects users",
      "mitigation": "How users can adapt"
    }
  ],
  "summary": "One-line summary of the analysis"
}

Changed files:
EOF
echo "$CHANGED_FILES" >> "$PROMPT_FILE"

echo -e "\n\nCommit messages:" >> "$PROMPT_FILE"
echo "$COMMIT_MESSAGES" >> "$PROMPT_FILE"

echo -e "\n\nDiff content:" >> "$PROMPT_FILE"
echo "$DIFF_CONTENT" >> "$PROMPT_FILE"
EOF

We want depot claude to start with the previous session we created, but save its output (and thus its analysis we can review later) to a new session. We can do this by taking advantage of the --session-id and --resume flags:

depot claude --resume=update-upstream --session-id="breaking-changes-$(date +%Y%m%d)" --model opus -p "$(cat $PROMPT_FILE)"

Notice, we're resuming from the update-upstream session we made earlier, but saving to a new breaking-changes session.

Next, we'll want to take that JSON output, and turn it into a nice to read text summary:

JSON_CONTENT=$(cat breaking-changes-analysis.json)
JQ_OUTPUT=$(mktemp)
JQ_ERROR=$(mktemp)

if echo "$JSON_CONTENT" | jq -r '
    if .breaking_changes | length > 0 then
        "## Breaking Changes Detected\n" +
        (.breaking_changes[] |
            "\n### " + .title +
            "\n**Risk Level:** " + .risk_level +
            "\n\n" + .description +
            "\n\n**Mitigation:** " + .mitigation + "\n"
        ) +
        "\n---\n**Summary:** " + .summary
    else
        .summary
    end
' > "$JQ_OUTPUT" 2> "$JQ_ERROR"; then
    cat "$JQ_OUTPUT" > breaking-changes-summary.txt
    echo "Breaking changes summary written to breaking-changes-summary.txt"
    cat breaking-changes-summary.txt
else
    echo "jq processing failed:"
    cat "$JQ_ERROR"
    echo "Failed to process JSON with jq" > breaking-changes-summary.txt
fi

With this new breaking-changes-summary.txt, we can go ahead and commit those changes to a branch.

- name: Check for breaking changes
id: breaking-changes
env:
    GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    DEPOT_TOKEN: ${{ secrets.DEPOT_TOKEN }}
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: |
    cd "$WORKTREE_DIR"
    bash .github/scripts/detect-breaking-changes.sh

    echo "BREAKING_CHANGES<<EOF" >> $GITHUB_OUTPUT
    cat breaking-changes-summary.txt >> $GITHUB_OUTPUT
    echo "EOF" >> $GITHUB_OUTPUT

    echo "BREAKING_CHANGES_SUMMARY<<EOF" >> $GITHUB_ENV
    cat breaking-changes-summary.txt >> $GITHUB_ENV
    echo "EOF" >> $GITHUB_ENV

- name: Commit upstream changes
run: |
    cd "$WORKTREE_DIR"
    git add upstream

    CURRENT_SHA=$(git submodule status upstream | awk '{print $1}' | sed 's/^[+-]//')
    NEW_SHA=$(cd upstream && git rev-parse HEAD)

    COMMIT_MSG="Update upstream repository"$'\n\n'"Updates upstream from $CURRENT_SHA to $NEW_SHA"$'\n\n'"Commits:"

    # List all commits between old and new
    cd upstream
    COMMITS=$(git log --oneline --no-decorate $CURRENT_SHA..$NEW_SHA)
    cd ..

    COMMIT_MSG="${COMMIT_MSG}"$'\n'"${COMMITS}"

    # Add information about breaking changes if any were found
    if [ -n "$BREAKING_CHANGES_SUMMARY" ] && [ "$BREAKING_CHANGES_SUMMARY" != "No significant breaking changes detected." ]; then
    COMMIT_MSG="${COMMIT_MSG}"$'\n\n'"Breaking changes detected - see PR description for details"
    fi

    COMMIT_MSG="${COMMIT_MSG}"

    git commit -m "$COMMIT_MSG"

We have some code to either create a new PR, or to update an existing one, depending on whether it already exists or not. The PR is kept in a draft state until a human engineer has time to review and merge the changes.

Fin

What used to consume hours of developer time every week now runs automatically in the background. Our arm64 patches stay fresh, breaking changes get flagged before they break anything, and we can focus on building features instead of maintaining forks.

We're already looking at expanding this approach to other forks we're maintaining, and all of us at Depot are pretty excited for future improvements we can make. If you have a similar need to maintain forks, run and monitor many agents, or just to analyze your codebase, get started with depot claude today!

Author: Billy Batista @ Depot

Depot Changelog: June 2025

Kyle Galbraith — Thu, 10 Jul 2025 17:37:56 +0000

We shipped some awesome new features and improvements in June. Things like our latest egress filtering capabilities, audit logging, and Windows runners. Here is everything we shipped

Egress filtering for GitHub Actions Runners

We've shipped an awesome security feature to Depot GitHub Actions Runners. You can enable egress filtering to control exactly which IP addresses, hostnames, and CIDR ranges your GitHub Actions can talk to.

Get all the details in our launch post

Now available: Audit logging

We've rolled out support for audit logging across Depot. This allows you to get fine grained information about what actions are taken in your Depot organization.

Read the announcement post

Windows GitHub Actions runners are now GA

We've completed all the work to make our Windows runners generally available to all organizations across Depot. You can see all of the nitty gritty details and runner labels for our Windows runners in our docs.

You can also read our full launch post on our blog

depot cargo for faster Rust builds

We released a new CLI command called depot cargo that wraps your cargo command with Depot Cache automatically for exponentially faster Rust builds.

Check out the changelog entry for how to use it

Dependabot now runs on Depot GitHub Actions runners

You can now run all of your Dependabot jobs on Depot GitHub Actions runners to take advantage of our Ultra Runners, faster caching, unlimited concurrency, and more.

Check out our changelog entry for more details on how to enable it

And more good stuff...

depot CLI v2.88.0 includes several bug fixes and new features
Add support to depot push to push without Docker config credentials -- more details in our changelog entry
Fix for loading cache only targets in depot bake
Improved documentation for building depot CLI from source

Forem: Depot

Optimize your Dockerfile for 5x faster builds

The problem: A slow Dockerfile

Order instructions to maximize cache reuse

How the Docker build cache reuses layers

Optimize instruction order for any language

Exclude unnecessary files from the build

Use a .dockerignore file

Build context size reduced

Separate build and runtime dependencies

Single-stage build with all dependencies

Multi-stage build splits build and runtime

Use cache mounts for package managers

Without cache mounts

With cache mounts

Cache mounts for other package managers

Cache mount persistence

Choose the right base image

Base image options for Node.js

When to use each base image

Example build with optimized base images

How to find optimization opportunities

View detailed build output

Identify optimization opportunities

Complete optimized example

Final results

Conclusion

How BuildKit Parallelizes Your Builds

Understanding BuildKit's DAG graph solver

Three levels of parallelism in action

Stage parallelism

Instruction parallelism

Deduplication across concurrent builds

Scaling beyond a single builder with Depot's auto-scaling

What this means for your builds

A marketer's hour of hell: What I finally get about slow builds

Life with fast builds

When everything slowed down

The realization

Related posts

We analyzed 66,821 GitHub Actions runs: 9 hidden gems you're missing

Scope of third-party adoption

9 third-party actions that are worth your time

1. Build step optimization

2. Fast Python package management

3. Compilation caching

4. System package caching

5. Robust CI pipelines

6. Better PR feedback

7. Beautiful test result reports

8. Universal binary installer

9. Enforce PR standards

Honorable mentions

The bottom line

Git is really cool, actually

git's smart protocol is really cool, actually

example: fetch

packfiles are really cool, actually

Git's protocol sorta sucks

Git & modern development

Out-of-band CI workflow

FIN

Related posts

Collaborating with Claude on docs

The workflow

Understanding the change

Check for documentation gaps

Generate a draft

Back into the codebase

Team collaboration and context sharing

A more complex experiment

When this workflow shines

Limitations

Final thoughts

Using AI as my engineering copilot (not autopilot)

The problem with "just make it work"

How I actually use AI

Navigating unfamiliar territory

Pairing with a very fast typist

The tedious stuff