Forem: Bruno Verachten

Running Node.js on RISC-V with Docker (when there's no official image yet)

Bruno Verachten — Thu, 05 Mar 2026 10:51:52 +0000

You know that moment when you try docker pull node --platform linux/riscv64 and Docker comes back with this?

Using default tag: latest
Error response from daemon: no matching manifest for linux/riscv64 in the manifest list entries:
no match for platform in manifest: not found

Not exactly a warm welcome.

If you've been watching the RISC-V world lately, you know the software story is catching up fast. Kernel support has been there for years, Debian and Fedora ship riscv64 builds, and boards like the Banana Pi F3 or StarFive VisionFive 2 are sitting on people's desks running real workloads. But the official Node.js Docker images? They don't support riscv64. Not yet. There's work happening upstream to change that, but we're not there today.

So I built my own.

The problem (a.k.a. "why can't I just apt install?")

I maintain a set of unofficial Node.js builds for riscv64. These are native builds, compiled on actual RISC-V hardware (a Banana Pi F3 with 8 cores and 16GB of RAM, because if it's too easy, it's no fun, right?), packaged as tarballs, .deb, and .rpm files, and published as GitHub Releases. Node.js 22 LTS and 24 Current are available today.

The builds work well. But distributing raw tarballs has friction. People who get their hands on a RISC-V board typically have a clean Debian or Fedora install. Their distro might ship Node.js 18. Maybe they installed a vendor-specific build. They don't want to mess with that. They want isolation.

Others don't have RISC-V hardware at all. They want to test their Node.js application against riscv64 before their CI environment supports it, or before they deploy to a RISC-V server. QEMU handles that.

In both cases, Docker is the answer.

What we ship

The images live on Docker Hub at gounthar/node-riscv64. Two variants per version:

Tag	Base	What's inside
`24.13.1-trixie`	`buildpack-deps:trixie`	Node.js 24 + npm + Yarn + gcc, g++, make, python3 (~522 MB)
`24.13.1-trixie-slim`	`debian:trixie-slim`	Node.js 24 + npm + Yarn, minimal footprint (~80 MB)
`22.22.0-trixie`	`buildpack-deps:trixie`	Node.js 22 LTS + npm + Yarn + build tools (~523 MB)
`22.22.0-trixie-slim`	`debian:trixie-slim`	Node.js 22 LTS + npm + Yarn, minimal footprint (~80 MB)

You'll notice there's no latest tag. That's on purpose, at least for now. The first images were pushed today via manual workflow dispatch, and the CI pipeline only creates latest and slim floating tags on actual GitHub Release events. The next release will add them.

But honestly? You shouldn't use latest anyway. Friends don't let friends use latest. It's a moving target that tells you nothing about what you're running. Your build works on Tuesday, breaks on Thursday, and you have no idea what changed because latest silently moved from 22.x to 24.x under your feet. Pin your versions. 24.13.1-trixie means you know exactly what you're getting, and your Dockerfile stays reproducible six months from now. latest is a convenience for quick demos and local experiments, not for anything you'd put in production or commit to a repo.

The full variant is for development: you can npm install native addons without extra setup. The slim variant is for production or when image size matters.

Quick smoke test:

docker run --platform linux/riscv64 gounthar/node-riscv64:24.13.1-trixie node -e "console.log(process.arch)"
# riscv64

If you're on an x86_64 or ARM machine, Docker uses QEMU under the hood to emulate riscv64. It's slower than native, but it works.

How the Dockerfiles are structured

The full variant

We adapted the Dockerfiles from the upstream nodejs/docker-node project. The approach is the same one the official images use for x86_64 and ARM: download a pre-built binary, verify checksums, extract to /usr/local/, install Yarn, set up the entrypoint.

The full variant is pretty straightforward:

FROM buildpack-deps:trixie

ARG NODE_VERSION=24.13.1
ARG NODE_DOWNLOAD_URL=https://github.com/gounthar/unofficial-builds/releases/download

RUN set -ex \
  && curl -fsSLO "${NODE_DOWNLOAD_URL}/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-riscv64.tar.xz" \
  && curl -fsSLO "${NODE_DOWNLOAD_URL}/v${NODE_VERSION}/SHASUMS256.txt" \
  && grep " node-v${NODE_VERSION}-linux-riscv64.tar.xz\$" SHASUMS256.txt | sha256sum -c - \
  && tar -xJf "node-v${NODE_VERSION}-linux-riscv64.tar.xz" -C /usr/local --strip-components=1 --no-same-owner \
  && rm "node-v${NODE_VERSION}-linux-riscv64.tar.xz" SHASUMS256.txt \
  && ln -s /usr/local/bin/node /usr/local/bin/nodejs \
  && node --version \
  && npm --version

Nothing exotic. The binaries come from our own GitHub Releases instead of nodejs.org, and we verify them with the SHA256 checksums published alongside each release.

The slim variant (where it gets interesting)

The slim variant uses a technique borrowed from the official Node.js images to keep the final image small. After installing Node.js, it uses ldd to figure out which shared libraries the binaries actually need at runtime, marks only those packages as manually installed, then runs apt-get purge --auto-remove to strip everything else:

apt-mark auto '.*' > /dev/null
find /usr/local -type f -executable -exec ldd '{}' ';' \
  | awk '/=>/ { so = $(NF-1); if (index(so, "/usr/local/") == 1) { next }; gsub("^/(usr/)?", "", so); print so }' \
  | sort -u \
  | xargs -r dpkg-query --search \
  | cut -d: -f1 \
  | sort -u \
  | xargs -r apt-mark manual
apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false

Why am I showing you this gnarly pipeline? Because I find it genuinely clever. It also removes OpenSSL architecture-specific files for platforms other than riscv64, since they're dead weight in a single-arch image.

The CI pipeline

From release to Docker Hub

Every time we create a GitHub Release (which happens when a new Node.js version comes out or we rebuild an existing one), a GitHub Actions workflow kicks in:

Verifies the release contains a node-v*-linux-riscv64.tar.xz tarball
Sets up QEMU for riscv64 emulation
Sets up Docker Buildx for cross-platform builds
Builds both the full and slim images targeting linux/riscv64
Pushes to Docker Hub with version-specific tags
Updates the latest and slim floating tags (only for actual releases, not manual rebuilds)
Syncs the Docker Hub repository description from a README file in the repo

The workflow runs on standard ubuntu-latest GitHub runners. No RISC-V hardware needed for the Docker build step; QEMU handles it. The actual Node.js compilation happened earlier on the Banana Pi F3; this workflow just packages the result.

One detail I'm particularly proud of: the floating latest and slim tags only get updated on release events, not on manual workflow dispatches. This prevents someone (me, probably) from accidentally rebuilding an older version and overwriting the latest tag. I've already been burned by that kind of thing before.

Staying in sync with upstream

The official nodejs/docker-node project evolves. Security fixes, Yarn updates, Dockerfile best practices. We don't want our images to drift silently.

So we have a second workflow that runs weekly (every Monday at 06:00 UTC). It reads a tracking file (docker/UPSTREAM_REF) that records which upstream commit we last synced from and which files we care about:

repo=nodejs/docker-node
commit=74b0481b76e0af5b19d425ad34489e7393b23aff
files=24/trixie/Dockerfile,24/trixie-slim/Dockerfile,24/trixie/docker-entrypoint.sh

The workflow compares file hashes between our tracked commit and upstream HEAD. If the Dockerfiles or entrypoint changed, it creates a GitHub issue tagged upstream-sync with a diff link and a list of changed files. If upstream moved forward but the tracked files didn't change, it opens a PR to update the reference commit. If nothing changed, it does nothing.

This is a lighter approach than forking the entire upstream repo. We track exactly the files we adapted, and we get notified only when those files change.

Using the images

On a RISC-V machine

If you're running Docker on actual riscv64 hardware, there's no --platform flag needed:

docker run -it gounthar/node-riscv64:24.13.1-trixie bash
node -v    # v24.13.1
npm -v     # 11.8.0

You can use the image as a base for your own applications:

FROM gounthar/node-riscv64:24.13.1-trixie
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
CMD ["node", "server.js"]

Or the slim variant for a smaller production image:

FROM gounthar/node-riscv64:24.13.1-trixie-slim
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .
USER node
CMD ["node", "server.js"]

On x86_64 or ARM with QEMU

Make sure QEMU user-mode emulation is registered. On most Docker Desktop installations, this works out of the box. On Linux, you may need:

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

Then use the --platform flag:

docker run --platform linux/riscv64 gounthar/node-riscv64:24.13.1-trixie node -e "
const os = require('os');
console.log('arch:', os.arch());
console.log('platform:', os.platform());
console.log('cpus:', os.cpus().length);
"

Expect it to be slow. QEMU instruction-level emulation has overhead. But for testing compatibility and validating that your dependencies work on riscv64, it's good enough.

Building native addons

The full variant includes build tools, so native addons with C/C++ bindings compile inside the container:

docker run --platform linux/riscv64 gounthar/node-riscv64:24.13.1-trixie \
  sh -c "mkdir /tmp/test && cd /tmp/test && npm init -y && npm install utf-8-validate && node -e \"require('utf-8-validate'); console.log('native addon loaded')\""

This is useful for checking whether your application's native dependencies build on riscv64 before committing to hardware. Give it a whirl with your own dependencies.

Why not wait for official images?

There's an open discussion at nodejs/docker-node#1707 about adding riscv64 support to the official images. It's moving, but it takes time. The official images need to build from official binaries, and Node.js doesn't produce official riscv64 binaries yet (riscv64 is in the "unofficial builds" tier).

We're filling a gap. When official support lands, these images become unnecessary and that's fine. Until then, if you need Node.js on riscv64 in a container today, they're here.

What's inside, specifically

Each image contains:

Node.js (built natively on RISC-V, not cross-compiled) with npm
Yarn Classic 1.22.22, verified via GPG signature
A node user (UID 1000) for running processes without root
An entrypoint that auto-detects whether you're passing Node.js arguments or a system command

That last point means docker run gounthar/node-riscv64 app.js runs node app.js, while docker run gounthar/node-riscv64 bash gives you a shell. Same behavior as the official Node.js images.

Some numbers

Node.js compiles in about 12 hours on the Banana Pi F3 (first build), around 40 minutes with ccache warm. The QEMU-based Docker image build on GitHub Actions takes about 5 minutes per version (under a minute for the full variant, closer to 4 for the slim one because of the ldd/apt-mark cleanup dance). We currently build Node.js 22 LTS and 24 Current, on a Banana Pi F3 (SpacemiT K1 SoC, 8 cores, 16GB RAM) running Debian Trixie.

Try it

# Quick test
docker run --platform linux/riscv64 gounthar/node-riscv64:24.13.1-trixie node -v

# Interactive session
docker run -it --platform linux/riscv64 gounthar/node-riscv64:24.13.1-trixie bash

# As a build environment
docker run --platform linux/riscv64 -v $(pwd):/app -w /app gounthar/node-riscv64:24.13.1-trixie npm test

The source is at github.com/gounthar/unofficial-builds, the images are at hub.docker.com/r/gounthar/node-riscv64, and contributions are welcome.

If you're experimenting with RISC-V or just curious about where this architecture is heading, I'd love to hear about your experience. The gap between "officially supported" and "actually usable" is where the fun happens.

What About iOS? Or, How a $30 Android Phone Embarrasses a $1000 iPad

Bruno Verachten — Mon, 09 Feb 2026 19:36:51 +0000

Photo by Holmo on Unsplash

I was giving my talk at a conference (the one about running Jenkins on old Android phones with Termux) and the demo had gone well. Jenkins was running, the agent was connected, builds were passing. I was feeling pretty good about the whole thing.

Then someone in the audience raised their hand.

"This is cool, but... what about iOS? Can you do the same thing on an iPhone?"

I paused. I actually didn't know. I mean, I assumed iOS would be harder. Apple locks things down way more than Android. But I'd never actually investigated the question properly. I gave a vague answer about Apple's sandbox restrictions and moved on, but the question kept nagging at me.

So I did what any self-respecting tinkerer would do: I went down the rabbit hole.

TL;DR

No, you cannot run Jenkins on iOS. The closest thing to Termux on iOS is iSH (Alpine Linux via x86 emulation), but Java is fundamentally broken on it. The only theoretically viable path involves running a full Linux VM on a $1000+ iPad Pro, and nobody has ever documented actually doing it. A $30 used Android phone does natively what a $1000 iPad can barely do in a virtual machine.

If you're at a conference and someone asks "What about iOS?", that's your answer.

Now, if you want to know how I arrived at that conclusion, and all the dead ends I explored along the way... read on.

The Investigation: Let's Start With iSH

First, I needed to find the iOS equivalent of Termux. After some digging, the answer was clear: iSH Shell.

iSH is an impressive project. 19.2k stars on GitHub, actively maintained, and it runs Alpine Linux on your iPhone or iPad via usermode x86 emulation. It gives you a real shell, a real package manager (apk with 14,000+ packages), and you can install stuff like Git, Python, Node.js, GCC, vim, curl...

Nice! This sounds promising!

Let's check the important boxes for running Jenkins:

Shell environment? Yes. Bash, zsh, the works.
Package manager? Yes. Alpine's apk, fully functional.
SSH server? Yes. You can accept incoming connections (wiki).
Git? Yes.
Python? Yes.

So far so good. Now the big one:

Java/OpenJDK? ...

Let me check.

The Java Wall

This is where everything falls apart.

I found issue #306 on the iSH repository. Then issue #1560, helpfully titled "Java is very broken." Then issue #2589. The picture was grim.

Here's what happens when you try to run Java on iSH:

JDK 7: Barely works. You need manual heap flags (java -mx256m). Functional but ancient. Jenkins dropped JDK 7 support years ago.
JDK 8+: Crashes immediately with "Too small initial heap." Can't start.
JDK 11+: Completely non-functional. Dead on arrival.
JDK 21 (what Jenkins needs today): Forget about it.

Damn!

The root causes are architectural, not just bugs that'll get fixed someday:

Missing SSE instruction support in iSH's x86 emulator. Modern JVMs assume SSE instructions exist.
Missing shared futex support. The JVM's threading model depends on these.
32-bit only (i386). iSH emulates an old x86 processor, not x86_64.

And the big one: Apple's W^X memory policy forbids JIT compilation. The JVM needs JIT to function at any reasonable performance level. Without it, you're running through an interpreter inside an emulator. Even if it worked, it would be comically slow.

Of course... I should have seen this coming. Jenkins is a Java application. No Java, no Jenkins. Investigation over?

Not quite. Let me check a few more things.

The JIT Saga: Apple Says No

Here's a detail I found fascinating and infuriating in equal measure.

The iSH developers actually tried to get JIT access through the EU's Digital Markets Act (DMA). In July 2025, they filed an interoperability request with Apple, arguing that JIT compilation is essential for their app to function properly.

Apple denied it in 2025, classifying iSH as ineligible.

There's a blog post about it if you want to read the details. The short version: Apple controls the platform, Apple says no JIT for third-party apps, and the EU regulations weren't enough to change that.

Compare this with Android: Termux runs native ARM64 binaries. No emulation layer. No JIT restrictions. pkg install openjdk-21 and you're done. The JVM runs at full native speed.

You know what? Let's keep investigating. Maybe there's another way.

What About a Full VM?

UTM is a QEMU-based virtual machine app that can run full Linux distributions on iOS. Two versions exist:

UTM SE (free, App Store): No JIT. x86 emulation at roughly 2-5% of native speed. Jenkins would be... let's say "meditative." Actually, unusable.
UTM with JIT (sideloaded or via EU AltStore): ARM64 Linux VMs running at 80-95% native speed on M-series iPads.

The JIT version is interesting! On a 16GB iPad Pro with an M-series chip, you could theoretically run an ARM64 Ubuntu VM with enough RAM for Jenkins.

Here's the problem:

iPad Model	Total RAM	Practical VM Allocation
iPad Air M1 (8GB)	8 GB	2-3.5 GB
iPad Pro M1/M2 (16GB)	16 GB	6-10 GB
iPad Pro M4 (16GB)	16 GB	8-12 GB

Jenkins needs roughly 1.5-2GB minimum (512MB heap plus the OS overhead). So you'd need at least an iPad Pro to even attempt this. And there's a fun detail: hardware-accelerated VMs on UTM have double RAM overhead due to JIT mirror mapping. ¯\_(ツ)_/¯

And the background execution story is rough. iOS aggressively kills background apps. There's some hope with iPadOS 26's BGContinuedProcessingTask API (WWDC 2025 session), and there's an open UTM issue tracking this, but as of early 2026, a VM-based Jenkins would die every time you switch to Safari.

The kicker: nobody has documented actually running Jenkins this way. Not a blog post, not a forum thread, not even a tweet. Zero evidence of anyone successfully doing it. I looked hard.

So the "best" iOS path to Jenkins requires a $1000+ iPad Pro, sideloading or EU DMA provisions for JIT access, a full Linux VM with careful RAM management, and... hope.

The Dead-End Parade

For completeness, I checked everything else I could find. Let me save you the trouble:

Approach	Java?	SSH Server?	Background?	Verdict
a-Shell	No	No	No	Cool WebAssembly environment, but not a server
Blink Shell	No	No	No	SSH client. Great for connecting TO Jenkins, not running it
iSH-AOK (fork)	No	Yes	No	Same x86 emulation engine as iSH. Same Java problems. Dormant since June 2024
iOS Shortcuts	No	No	No	Can trigger Jenkins builds via REST API, but that's about it
Play.js	No	No	No	JavaScript IDE
Pythonista 3	No	No	No	Python IDE
Docker on iOS	No	No	No	Would require Linux kernel features that iOS's XNU kernel doesn't have

Every. Single. One. A dead end for running Jenkins.

"But What About Jailbreaking?"

Sure, let's go there. Jailbreaking could theoretically give you launchd daemon persistence, a real SSH server on port 22, and Linux chroot environments.

But here's the state of jailbreaking in early 2026:

Tool	iOS Support	Devices	Status
Palera1n	iOS 15.0-18.7.4	A8-A11 only (iPhone 6s through X)	Active but ancient hardware only
Dopamine	iOS 15.0-16.6.1	A12+	Stalled. No iOS 17+
TrollStore	iOS 15.5-17.0	All	Apple patched the vulnerability in iOS 17.0.1

No jailbreak exists for iPhone 12 or newer on iOS 17.1+. Jailbreaking is effectively dead for modern hardware. And even if you could jailbreak, there's no official OpenJDK package. You'd need to manually compile it.

So that's not a realistic answer either.

Before we wrap up the investigation, there's one more recent development worth mentioning.

A Glimmer of Hope: OpenJDK Mobile

The OpenJDK Mobile project has made progress. InfoQ reported in November 2025 that OpenJDK can build and run on iOS using the Zero interpreter (pure C++, no JIT) enhanced with AOT-compiled methods from Project Leyden. Gluon has built automated build pipelines.

Interesting! But (and this is a big but) it's designed for embedding Java in iOS apps, not running standalone JVM servers. Jenkins is far too dynamic for AOT compilation (hundreds of plugins, dynamic class loading everywhere). This doesn't help us.

The Comparison That Says It All

So after exploring every possible path (native apps, emulation, VMs, jailbreaking, even emerging OpenJDK efforts), let me lay out the complete picture in one table:

Capability	Termux (Android)	iSH (iOS)	UTM+JIT (iPad Pro)
Execution	Native ARM64	x86 emulation (5-100x slower)	Near-native ARM64 (in VM)
Package manager	`pkg` (Debian-based)	`apk` (Alpine i386)	Guest OS (`apt`/`dnf`)
SSH server	Port 8022, background	Port 22, foreground only	Inside VM
Java/OpenJDK 21	Works perfectly	Broken	Works (inside VM)
Background services	termux-services (runit/sv)	Unreliable (location hack)	Partial (improving)
Boot persistence	Termux:Boot	Impossible	Impossible
Jenkins	Fully working	Impossible	Theoretically possible
Cost	Free (any old Android)	Free (App Store)	$1000+ iPad Pro 16GB

Look at that last row. Free versus a thousand dollars, and the free option actually works.

Why iOS Is Fundamentally Different

This isn't a matter of finding the right app or the right workaround. The gap between Android and iOS for this use case is architectural:

No JIT compilation (W^X memory policy). Kills JVM performance. Apple denied the EU DMA exemption request.
Aggressive background app killing. iOS suspends apps after minutes. You can hack around this with location services in the background (cat /dev/location > /dev/null &), but it's unreliable.
No boot-time auto-start. There is no equivalent to Termux:Boot. You cannot have Jenkins start when the device boots.
App sandbox isolation. Apps cannot spawn arbitrary processes or access system directories.
No raw syscall access. Everything must go through Apple's approved APIs.

Android's architecture is simply more open. Termux runs native ARM64 binaries, manages services via runit, and persists across reboots, all without root. That's not a feature gap. It's a philosophy gap.

The Punchline

So here's what I'll say next time someone asks "What about iOS?" at a conference:

iSH is the closest thing to Termux on iOS. It runs Alpine Linux with a real package manager and even accepts incoming SSH connections. Pretty impressive! But Java is broken on it due to Apple's ban on JIT compilation and missing CPU instruction support in the emulator. The only way to run Jenkins on an iPad would be inside a full Linux VM using UTM on a 16GB iPad Pro, and even that has never been documented working. Meanwhile, a $30 used Android phone does natively what a $1000 iPad can barely do in a VM.

How cool is that? Well, not cool for iOS users. But it definitively validates the Android/Termux approach.

Those old Android phones gathering dust in your drawer? They're not e-waste. They're infrastructure waiting to happen. And no amount of Apple silicon can change the fact that an open platform beats a locked-down one when you need to run actual server workloads.

Suite au prochain episode. I'm already thinking about multi-device clusters and load balancing across a fleet of phones. But that's a story for another day.

The project: termux-jenkins-automation on GitHub

The talk: If you want to see Jenkins running on a phone and ask your own tricky audience questions, come find me at the next conference. I'll have a better answer for the iOS question this time.

The Ring Around the Rosie: Upgrading Next.js on RISC-V from 13.5.6 to 14.2.35

Bruno Verachten — Sun, 28 Dec 2025 12:16:34 +0000

Photo by Klara Kulikova on Unsplash

I had a working Next.js 13.5.6 setup on my Banana Pi F3. Then Dependabot showed up with PRs bumping to 14.2.35, and suddenly my carefully built SWC binary was useless. What followed was a two-hour journey through cross-compilation failures, Cargo dependency rabbit holes, and the eventual realization that the same trick that worked for 13.5.6 was fundamentally broken in 14.x. Here is how I fixed it.

The Problem: Version Mismatch

I have been running Next.js on RISC-V hardware for a while now. The Banana Pi F3 is my test bed: 8 cores, 15GB of RAM, running Debian 13. Not exactly a speed demon, but it gets the job done.

Back in November, I successfully built @next/swc for version 13.5.6 using a simple workaround: the --no-default-features flag. This disabled TLS dependencies that pulled in the ring cryptographic library, which famously does not support riscv64.

Then Dependabot filed PRs #22 and #23, bumping Next.js to 14.2.35.

The problem? SWC binaries are version-locked. My 13.5.6 binary would not load in 14.2.35. Time to rebuild.

Failed Attempt 1: Cross-Compilation with cross-rs

Building on the Banana Pi takes hours. The 13.5.6 build ran for about 4 hours. Surely there is a faster way?

Cross-compilation seemed like the obvious answer. I have a perfectly good x86_64 machine running WSL2. Why not build there and copy the binary?

I set up cross-rs, the standard Rust cross-compilation tool:

# Cross.toml
[target.riscv64gc-unknown-linux-gnu]
image = "ghcr.io/cross-rs/riscv64gc-unknown-linux-gnu:main"

Then ran:

cross build --release --target riscv64gc-unknown-linux-gnu \
  --manifest-path crates/napi/Cargo.toml --no-default-features

It failed. Not immediately, but about 200 crates in:

error: failed to run custom build command for `ring v0.16.20`

Caused by:
  process didn't exit successfully: [...]/build-script-build

  ring build.rs panic: Target architecture not supported

The issue is subtle but fundamental. The ring crate embeds hand-written assembly for cryptographic operations. Each target architecture needs its own assembly files. When ring's build.rs runs, it checks the target architecture and panics if there is no matching assembly.

Cross-compilation does not help here. The build script still runs, still checks for riscv64 assembly, still panics when it finds none.

Failed Attempt 2: Patching ring/rustls Versions

Maybe I could upgrade ring? Version 0.17 added riscv64 support. I tried patching Cargo.toml:

[patch.crates-io]
ring = "0.17"

Cargo rejected it:

error: failed to select a version for `ring`.
    ... required by rustls v0.20.9
    ... which requires ring ^0.16

The caret (^) in Cargo means "compatible version," and 0.17 is not compatible with 0.16 under semver rules (pre-1.0 versions treat minor bumps as breaking changes).

To use ring 0.17, I would need to upgrade the entire chain: ring -> rustls -> hyper-rustls -> reqwest -> turbo-tasks-fetch. That is not a quick patch. That is a major dependency overhaul that would diverge significantly from upstream.

Failed Attempt 3: The Old Workaround

Fine. Forget cross-compilation. I will just build natively like I did for 13.5.6.

I SSH'd into the Banana Pi, cloned Next.js 14.2.35, and ran the exact command that worked before:

cargo build --release --manifest-path crates/napi/Cargo.toml --no-default-features

It compiled 282 crates. Then:

error: failed to run custom build command for `ring v0.16.20`

Wait, what? The --no-default-features flag was supposed to skip ring entirely. It worked in 13.5.6. Why is it failing now?

Root Cause: The Cargo Configuration Changed

I dug into the Cargo.toml files, comparing 13.5.6 to 14.2.35.

In 13.5.6:

[features]
default = ["rustls-tls"]
rustls-tls = [...]
native-tls = [...]

TLS was a default feature. Passing --no-default-features disabled it. No TLS, no ring, build succeeds.

In 14.2.35:

# Enable specific tls features per-target.
[target.'cfg(all(target_os = "windows", target_arch = "aarch64"))'.dependencies]
next-core = { workspace = true, features = ["native-tls"] }

[target.'cfg(not(any(all(target_os = "windows", target_arch = "aarch64"), target_arch="wasm32")))'.dependencies]
next-core = { workspace = true, features = ["rustls-tls"] }

TLS is now a target-specific dependency. The second block says: for everything that is not Windows ARM64 or WASM, use rustls-tls.

riscv64 is not Windows ARM64. riscv64 is not WASM. So riscv64 gets rustls-tls.

The --no-default-features flag does nothing here because this is not a feature. It is a conditional dependency based on target platform.

The Fix: native-tls for riscv64

The solution was staring at me from the first target block. Windows ARM64 uses native-tls instead of rustls-tls. Why? Probably similar issues with ring on that platform.

native-tls uses the system's TLS library, OpenSSL on Linux. OpenSSL supports riscv64. My Debian 13 install has OpenSSL 3.5.4.

I patched packages/next-swc/crates/napi/Cargo.toml:

# Use native-tls for riscv64 (ring 0.16.20 doesn't support riscv64)
[target.'cfg(all(target_os = "linux", target_arch = "riscv64"))'.dependencies]
next-core = { workspace = true, features = ["native-tls"] }

# Exclude riscv64 from rustls-tls targets
[target.'cfg(not(any(all(target_os = "windows", target_arch = "aarch64"), all(target_os = "linux", target_arch = "riscv64"), target_arch="wasm32")))'.dependencies]
next-core = { workspace = true, features = ["rustls-tls"] }

Two changes:

Add a riscv64-specific block that uses native-tls
Modify the fallback block to exclude riscv64

Started the build again:

cargo build --release --manifest-path crates/napi/Cargo.toml --no-default-features

The Build

130 minutes. About half the time of the 13.5.6 build, though I suspect that is due to better incremental caching rather than any architectural improvement.

Compiling ring v0.17.8

Wait, ring? I thought we were avoiding ring?

Turns out native-tls on Linux still uses ring for some operations, but version 0.17.8, which supports riscv64. The constraint chain is different: native-tls does not pin to ring ^0.16.

Binary size: 230MB. About what I expected for a debug-info-included release build.

Testing

I ran the test suite against both App Router and Pages Router test apps:

./scripts/run-tests.sh tests/app-router
./scripts/run-tests.sh tests/pages-router

Results:

Test App	Status	Notes
App Router	Pass	9 pages including SSG and SSR
Pages Router	Pass	6 pages including API routes

Both work. Native SWC compilation, no Babel fallback needed.

Bonus Discovery: No Loader Patch Required

In 13.5.6, I had to patch node_modules/next/dist/build/swc/index.js to add riscv64 to the architecture map:

// Had to add this line in 13.5.6
riscv64: linux.riscv64gc,

I checked 14.2.35's loader. It is already there:

linux: {
    x64: linux.x64.filter((triple)=>triple.abi !== "gnux32"),
    arm64: linux.arm64,
    riscv64: linux.riscv64,  // Already present!
    arm: linux.arm
},

Someone at Vercel added riscv64 support between 13.5.6 and 14.x. The only thing missing is the actual binary, and now we have that too.

The Dependency Chain Explained

For anyone who wants to understand why this was so painful, here is the dependency chain:

@next/swc
  -> turbo-tasks-fetch
    -> reqwest
      -> hyper-rustls
        -> rustls 0.20.9
          -> ring ^0.16  <-- PROBLEM: ring 0.16 has no riscv64 assembly

And why native-tls works:

@next/swc
  -> turbo-tasks-fetch
    -> reqwest (with native-tls feature)
      -> native-tls
        -> openssl-sys  <-- Uses system OpenSSL, which supports riscv64

The ring crate is a pure-Rust cryptographic library that embeds assembly for performance. Each architecture needs hand-written assembly. riscv64 assembly was added in ring 0.17, but rustls 0.20 pins to ring 0.16.

The native-tls crate delegates to the operating system's TLS implementation. On Linux, that is OpenSSL. OpenSSL 3.x has full riscv64 support through its own assembly routines and C fallbacks.

Key Takeaways

For anyone building Next.js on exotic architectures:

Version matters: Build flags that work in one version may not work in another. Check the Cargo.toml configuration.
Cross-compilation will not save you from ring: The build script runs on the host but checks the target architecture. No assembly for target = build failure.
native-tls is your friend: If rustls fails due to ring, try native-tls. It uses system libraries that typically have broader architecture support.
Check upstream first: Next.js 14.x already has riscv64 in the loader. They are aware of the architecture. The binary is the only missing piece.

For the Next.js team (if anyone is reading): The fix is a three-line patch to packages/next-swc/crates/napi/Cargo.toml. Add riscv64 to the native-tls targets. The CI infrastructure is the harder problem, but the code change is minimal.

What is Next

The patch file is in this repository at patches/nextjs-14.x-native-tls.patch. Apply it before building:

cd ~/next.js
git checkout v14.2.35
patch -p1 < /path/to/patches/nextjs-14.x-native-tls.patch
cd packages/next-swc
cargo build --release --manifest-path crates/napi/Cargo.toml --no-default-features

Build time: about 2 hours on a Banana Pi F3. Faster on better hardware.

I will update the test apps to 14.2.35 once I verify there are no regressions in the Dependabot PRs. The infrastructure is ready.

Version Comparison

Aspect	13.5.6	14.2.35
TLS configuration	Default feature	Target-specific dependency
Build workaround	`--no-default-features`	Cargo.toml patch required
Loader patch needed	Yes	No (riscv64 already included)
Build time (Banana Pi F3)	~4 hours	~130 minutes
Required Rust nightly	2023-10-06	2024-04-03

The 14.x build is actually faster despite being a more complex codebase. Progress.

References

Patch file: patches/nextjs-14.x-native-tls.patch
Build documentation: docs/BUILDING-SWC.md
ring crate issue: github.com/briansmith/ring/issues/1292

The Old Dog Learns a New Trick: Cross-Compiling Tauri CLI for RISC-V

Bruno Verachten — Tue, 23 Dec 2025 22:01:34 +0000

Photo by Dariusz Piosik on Unsplash

Previously, on "Bruno Fights RISC-V"

If you missed the first episode (available here), here's the recap: I tried adding RISC-V builds to Armbian Imager (a Tauri app), hit a wall with 6+ hour QEMU emulation times, and realized the real fix was upstream. Tauri CLI ships pre-built binaries for x64, ARM64, macOS, Windows... but not RISC-V.

My plan was simple: use my physical Banana Pi F3 as a self-hosted GitHub runner. Native RISC-V compilation. No emulation overhead. I'd done this before with Docker builds. Easy.

Famous last words. Again.

🚀 The Self-Hosted Runner Approach

I wasn't walking into this blind. I already had working RISC-V runners from my Docker-for-RISC-V project:

jobs:
  build:
    runs-on: [self-hosted, riscv64]
    steps:
      - uses: actions/checkout@v4
      - name: Build
        run: cargo build --release

Two machines at home (192.168.1.185 and 192.168.1.36), both Banana Pi F3 boards, both registered as GitHub runners. They've been churning out Docker engine builds for months. Adding Tauri CLI to the mix seemed natural.

And you know what? It worked.

The Working Solution (That Got Replaced)

✅ First Blood

The self-hosted runner approach actually succeeded. Full workflow run, actual binary produced:

Metric	Value
Build time	1h 2m 28s
Binary	ELF 64-bit RISC-V, RVC, double-float ABI
Size	16 MB (stripped)
Version	tauri-cli 2.9.6

Proof: actual release with actual binary.

I was ready to submit the PR. Self-hosted runners, conditional logic to skip them if not available, fallback to QEMU for the brave souls with 6 hours to spare. A complete solution.

Then FabianLars left a comment on my PR.

🤔 "Have you considered cross?"

I stared at the screen. Cross? Cross-compilation? I'd just spent an hour in the previous session explaining why cross-compilation was a nightmare for WebKit-dependent projects. Sysroots, symlink hell, version mismatches. I'd tried it. I'd abandoned it.

But FabianLars wasn't talking about manual cross-compilation. He was talking about cross-rs.

cross should work pretty well for the tauri cli

— FabianLars, GitHub PR comment

I had never heard of cross-rs.

Let me say that again, because it bears repeating: I've been doing ARM32 cross-compilation since 2013. I've set up sysroots. I've dealt with qemu-user-static. I've maintained unofficial Node.js builds for exotic architectures. I've built Docker images on physical RISC-V hardware. Twelve years of this.

And I had never heard of cross-rs.

What Is Cross-rs?

📦 The Tool I Should Have Known About

Cross is a "zero setup" cross compilation tool for Rust. Instead of manually setting up toolchains and sysroots, it uses pre-built Docker containers with everything already configured.

# Install it
cargo install cross

# Use it exactly like cargo
cross build --target riscv64gc-unknown-linux-gnu --release

That's it. No sysroot configuration. No toolchain setup. No symlink archaeology. It just... works.

The magic is Docker containers maintained by the cross-rs team. For RISC-V, you get:

The GNU toolchain (riscv64-unknown-linux-gnu-gcc)
Proper sysroot with glibc
QEMU for running test binaries
All the standard Rust cross-compilation targets

When I ran cross build for the first time, I expected it to fail spectacularly. WebKit dependencies! Linker errors! Missing symbols! The usual cross-compilation dance of despair.

Instead:

$ time cross build --manifest-path ./crates/tauri-cli/Cargo.toml \
    --target riscv64gc-unknown-linux-gnu --profile release-size-optimized

   Compiling tauri-cli v2.9.6
    Finished `release-size-optimized` profile [optimized] target(s) in 4m 27s

real    4m27.235s

Four minutes and twenty-seven seconds.

📊 The Numbers Don't Lie

Method	Time	Relative Speed
QEMU emulation (Docker buildx)	6+ hours (killed)	Baseline of pain
Native RISC-V (Banana Pi F3)	63 minutes	~6x faster than QEMU
Cross-rs on x64	4m 27s	~90x faster than QEMU

I sat there looking at my terminal. Sixty-three minutes of native compilation. Four and a half minutes with cross. Fourteen times faster. On the same code. Same binary output.

🧠 Why Cross Works Here

Here's what I got wrong in my previous article: I assumed Tauri CLI had the same WebKit dependencies as Tauri apps.

It doesn't.

Tauri CLI is a build tool. It orchestrates the build process, invokes cargo, bundles assets. It doesn't link against WebKit2GTK. That's what the app does at runtime. The CLI is just orchestration code.

No WebKit dependency means no sysroot nightmare. No sysroot nightmare means cross-compilation is trivial.

I spent an hour in my previous session explaining why cross-compilation was impossible. Turns out I was solving the wrong problem.

The New Implementation

🔧 Simpler Is Better

The updated workflow replaces self-hosted runners with cross:

build:
  runs-on: ${{ matrix.config.os }}
  strategy:
    matrix:
      config:
        # ... existing x64, ARM64, macOS, Windows targets ...

        - os: ubuntu-22.04
          rust_target: riscv64gc-unknown-linux-gnu
          ext: ''
          args: ''
          cross: true

  steps:
    - uses: actions/checkout@v4

    - name: 'Setup Rust'
      if: ${{ !matrix.config.cross }}
      uses: dtolnay/rust-toolchain@stable
      with:
        targets: ${{ matrix.config.rust_target }}

    - name: Install cross
      if: ${{ matrix.config.cross }}
      uses: taiki-e/install-action@v2
      with:
        tool: cross@0.2.5

    - name: Build CLI (cross)
      if: ${{ matrix.config.cross }}
      run: cross build --manifest-path ./crates/tauri-cli/Cargo.toml \
           --target ${{ matrix.config.rust_target }} \
           --profile release-size-optimized

Changes from the self-hosted runner approach:

Runs on GitHub's ubuntu-22.04, not external hardware
Uses cross@0.2.5 (pinned version for reproducibility)
Skips unnecessary steps (Rust cache, Linux dependencies - cross handles these)
Different artifact path (target/{target}/ instead of target/)

No self-hosted runners to maintain. No physical hardware to keep online. No network connectivity issues. Just a matrix entry that happens to use cross instead of cargo.

The Humility Lesson

🎓 You Don't Know What You Don't Know

Here's the uncomfortable truth: I had opinions about cross-compilation. Strong ones. Formed over a decade of wrestling with toolchains and sysroots. Those opinions were... not wrong, exactly. Cross-compiling WebKit apps is a nightmare. Setting up manual sysroots is fragile.

But cross-rs exists. Has existed since 2016. Nine years of development. Excellent RISC-V support. And I'd never encountered it because I was busy doing things the hard way.

This is the curse of experience: you learn patterns that work, and you stop exploring alternatives. "I know how to do this" becomes "I know how this is done" becomes "this is how it's done." Except sometimes a tool comes along that makes your carefully accumulated knowledge... obsolete? Not quite. But certainly less essential.

The gray hairs from ARM32 era taught me a way. Not the way.

🔁 The Pattern Repeats

2013: "Cross-compiling for ARM32 requires a full sysroot setup."
2016: cross-rs releases, nobody tells me.
2025: "Cross-compiling for RISC-V is impossible because WebKit."
Also 2025: "Have you considered cross?"

I should probably set up an RSS feed for Rust tooling. Or just... ask people before assuming I know the answer.

What's Next

🎯 PR Status

The PR is submitted: https://github.com/tauri-apps/tauri/pull/14685

Current state:

[x] Cross-rs implementation complete
[x] Build time: ~4 minutes (down from 63 minutes native, 6+ hours QEMU)
[x] CodeRabbit feedback addressed
[ ] Awaiting maintainer review

If merged, every Tauri CLI release will include a RISC-V binary. Anyone on a RISC-V system can cargo install tauri-cli and get a pre-built binary in seconds instead of compiling 600+ crates.

🚀 The Bigger Picture

This matters beyond Tauri. The RISC-V ecosystem is at that inflection point I keep mentioning. Hardware exists. Kernels boot. Distributions ship packages. But every missing binary is a barrier.

Framework 13 with DC-ROMA RISC-V mainboard? You'll want GUI apps. Banana Pi F3 running Armbian? You'll want to flash images without dd. Pine64 boards? Same story.

Pre-built binaries are the difference between "works out of the box" and "come back in 6 hours." Cross-rs makes pre-built binaries trivial to produce.

I'll probably be using cross for everything RISC-V from now on. Assuming the binary doesn't need runtime system libraries. For those cases... well, there's always the Banana Pi.

Lessons Learned

💡 Takeaways

Ask before assuming. A decade of experience doesn't mean you've seen everything. The maintainer's one-line suggestion saved hours of CI time.
Cross-rs exists. For pure Rust binaries without system library dependencies, it's the obvious choice. Zero setup, fast builds, maintained Docker images.
Know your dependencies. I assumed Tauri CLI had the same requirements as Tauri apps. It doesn't. CLI is a build tool, not a WebKit consumer.
Self-hosted runners still have their place. For projects that do need system libraries (like the actual Armbian Imager), native hardware remains the answer. But for build tools and pure-Rust code? Cross wins.
Pin your versions. cross@0.2.5 is reproducible. cross is not. CI is not the place for floating versions.
The hard way isn't always the right way. Sometimes the clever solution you've perfected over years gets replaced by a tool that just... does it better.

I've been building for exotic architectures since ARM32 was exotic. Gray hairs and all. And today I learned something new.

That's not embarrassing. That's the job.

Resources

cross-rs: Zero setup cross compilation for Rust
Tauri PR #14685: The RISC-V CLI support PR
Proof of concept release: Working RISC-V binary (self-hosted runner build)
Tauri: The framework this is all about
Armbian RISC-V: The boards that need this software
Banana Pi BPI-F3: The little board that could (in 63 minutes)

The code is submitted. The binary works. And I have a new tool in my belt.

Twelve years late, but who's counting?

Adding RISC-V Support to Armbian Imager: A Tale of QEMU, Tauri, and Deja Vu

Bruno Verachten — Tue, 23 Dec 2025 21:58:25 +0000

Photo by SnapSaga on Unsplash

The Setup

Armbian Imager is a Tauri 2 application: React frontend, Rust backend, builds for Linux (x64, ARM64), macOS (both architectures), and Windows (both architectures). A proper multi-platform desktop app that actually works, which is rarer than you'd think.

The build workflow already handled six platform combinations through GitHub Actions. Adding a seventh (Linux RISC-V 64-bit) seemed straightforward. After all, I'd done this dance before with ARM32 back in the dark ages (2013-2014, when docker-compose on Raspberry Pi was considered experimental black magic).

Famous last words.

The Murphy's Law Setup

Here's the thing: whenever you think "this should be straightforward," the universe takes that as a personal challenge. I should've known better. The gray hairs didn't appear from successful, uneventful builds.

Why RISC-V, Why Now

Armbian supports RISC-V boards. The Banana Pi F3 runs Armbian. Various Pine64 and StarFive boards run Armbian. The Framework 13 laptop has a RISC-V mainboard option (DC-Roma, because apparently laptop mainboards are modular now, what a time to be alive). The ecosystem is growing.

But if you're on a RISC-V system and want to flash an Armbian image to an SD card, you currently need to use dd like it's 1995. The Imager app doesn't have a RISC-V build.

I figured I'd fix that.

Sounds appealing or strange enough to get you intrigued?

The Research Phase

First step: figure out what the existing build workflow does. The .github/workflows/build.yml file tells the story:

Linux x64 builds on ubuntu-24.04 runners
Linux ARM64 builds on ubuntu-24.04-arm runners (GitHub has native ARM runners now, progress!)
macOS and Windows have their own native runners

For RISC-V, GitHub doesn't offer native runners yet. There's Cloud-V which provides RISC-V GitHub runners, but Armbian's workflow isn't set up for external runners. For this contribution, emulation was the path of least resistance.

Docker + QEMU it is. (Spoiler alert: this is where things get interesting.)

The WebKit2GTK Archaeology

Tauri apps on Linux need WebKit2GTK. This is the system webview that renders the UI, basically the browser engine that makes your Rust backend look pretty. On x64 and ARM64, it's readily available in Debian bookworm and trixie.

On RISC-V? I checked the Debian package tracker, because I'm a glutton for disappointment.

webkit2gtk in bookworm: not available for riscv64
webkit2gtk in trixie: available for riscv64

Good news: Debian trixie became stable in 2025, and it has the packages we need. The timing worked out: RISC-V users on the current stable release get WebKit2GTK out of the box.

The NodeSource Curveball

The frontend build needs Node.js. The standard approach in CI is to use NodeSource's distribution:

curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
apt-get install -y nodejs

I checked NodeSource's supported architectures: amd64, arm64, armhf. No riscv64.

Of course not. Why would there be?

Here's where things get simple (yes, really): use Debian's native nodejs package instead. It's version 18 instead of 20, but that's close enough for a Vite build. Sometimes the boring solution is the right solution.

apt-get install -y nodejs npm

(Side note: if you need the latest Node.js LTS on RISC-V, I maintain unofficial builds with an APT repo via GitHub Pages. We've got 24.12.0 ready to go. But for this build, Debian's package was sufficient.)

Two problems identified, two solutions found. Look at me being all productive and efficient. This never happens. I should've been more suspicious.

The Implementation

I added a new job to the GitHub workflow:

build-linux-riscv64:
  name: build-linux (riscv64gc-unknown-linux-gnu)
  needs: [create-release]
  if: ${{ github.event_name == 'push' || inputs.build_linux_riscv64 }}
  runs-on: ubuntu-24.04
  steps:
    - uses: actions/checkout@v4

    - name: Set up QEMU for RISC-V emulation
      uses: docker/setup-qemu-action@v3
      with:
        platforms: riscv64

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v3

    - name: Build in RISC-V container (Debian trixie)
      run: |
        docker run --rm --platform linux/riscv64 \
          -v "$(pwd)":/app \
          -w /app \
          riscv64/debian:trixie \
          bash -c '
            # Install build dependencies
            apt-get update
            apt-get install -y \
              curl build-essential pkg-config \
              libwebkit2gtk-4.1-dev \
              libayatana-appindicator3-dev \
              librsvg2-dev patchelf libssl-dev libgtk-3-dev \
              squashfs-tools xdg-utils file

            # Node.js from Debian (NodeSource lacks RISC-V)
            apt-get install -y nodejs npm

            # Install Rust
            curl --proto "=https" --tlsv1.2 -sSf https://sh.rustup.rs | \
              sh -s -- -y
            source "$HOME/.cargo/env"

            # Build frontend
            npm ci
            npm run build

            # Install Tauri CLI and build
            cargo install tauri-cli --version "^2" --locked
            cargo tauri build --bundles deb
          '

The key differences from the x64/ARM64 builds:

Uses riscv64/debian:trixie instead of Ubuntu (because we need those WebKit packages)
Node.js comes from Debian packages, not NodeSource (because NodeSource said "nope")
Runs through QEMU user-mode emulation (this will be important later)
Only builds .deb packages (no AppImage; that's a story for another day, involving AppImage's aversion to exotic architectures)

I also created a standalone script scripts/build-riscv64.sh for local builds, and added --riscv64 to the build-all.sh orchestrator, because I like my tooling to be consistent.

Four commits later:

aae7628 feat: Add RISC-V 64-bit build support
105f77f fix: Use Debian nodejs package for RISC-V builds
22499b1 feat: Add pre-built Docker image support for faster RISC-V builds
0eb2992 fix: Use Debian nodejs in build-all.sh for RISC-V

Time to test. What could possibly go wrong?

The Wall

I kicked off a build. Docker pulled the RISC-V Debian image. QEMU started emulating. The apt packages installed. Rust downloaded. So far, so good.

Then:

cargo install tauri-cli --version "^2" --locked

And we wait.

And wait.

And wait some more.

The Great Compilation Watch

The tauri-cli crate has over 600 dependencies. Six. Hundred. Each one needs to compile. Under QEMU user-mode emulation, every single CPU instruction goes through a translation layer. A Rust build that takes 2 minutes on native hardware takes... well, let's just say it takes a while.

I went to make coffee. Came back. Still compiling.

I went to lunch. Came back. Still compiling.

I started questioning my life choices. The build was still compiling.

This is the ARM32 era all over again, and I'm having flashbacks.

Flashback: 2013-2014

The ARM32 PTSD

Picture this: around 2013, I was trying to get docker-compose working on ARM32. And RethinkDB. And everything else needed to run openSTF (Open Smartphone Test Farm) on Raspberry Pi hardware, because apparently I enjoy suffering.

Nothing worked. Every dependency was a new adventure in "why doesn't this architecture exist in their CI matrix?" "Just compile it from source" meant leaving your Pi running overnight and hoping it didn't thermal throttle itself into early retirement. Cross-compilation setups were fragile. One wrong symlink and you'd spend three hours debugging why ld couldn't find libc.so.6. Pre-built binaries were as rare as sensible variable names in legacy code.

We eventually got there. ARM support improved. Docker got multi-arch images. GitHub added ARM runners. The ecosystem matured. The gray hairs appeared.

Here's the thing: RISC-V is at that same inflection point now. The hardware exists. The kernels boot. The distributions have packages. But the tooling ecosystem hasn't caught up yet.

And I'm apparently volunteering to help it catch up. Because I learn nothing from experience.

The Math

Let's be concrete about the problem, because misery loves documentation.

On a native x64 machine with cached dependencies, cargo install tauri-cli takes about 2 minutes. Fast enough to grab a coffee, check Slack, come back to a finished build.

Under QEMU user-mode emulation, that same operation takes... I didn't let it finish. After 3 hours, I killed the build because I value my sanity. Extrapolating from progress (and some very pessimistic napkin math), a complete build would take 6-8 hours.

For CI/CD, this is unusable. GitHub Actions has a 6-hour timeout per job. Even if it finished, waiting that long for every release is absurd. Imagine telling your team "yeah, the release will be ready in 8 hours, assuming nothing goes wrong." (Spoiler: something always goes wrong.)

The Pre-built Image Strategy

My first optimization: build a Docker image with tauri-cli pre-installed. Suffer once, benefit forever (or until Tauri releases a new version, whichever comes first).

FROM --platform=linux/riscv64 riscv64/debian:trixie

# Install all build dependencies
RUN apt-get update && apt-get install -y \
    curl build-essential pkg-config \
    libwebkit2gtk-4.1-dev libayatana-appindicator3-dev \
    librsvg2-dev patchelf libssl-dev libgtk-3-dev \
    squashfs-tools xdg-utils file nodejs npm \
    && rm -rf /var/lib/apt/lists/*

# Install Rust
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | \
    sh -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"

# Pre-install tauri-cli (this is the slow part)
RUN cargo install tauri-cli --version "^2" --locked

WORKDIR /app
CMD ["bash"]

Build this image once (accepting the 6+ hour wait like penance for your architectural choices), push it to a registry, then subsequent builds only need to compile the actual application, maybe 10-20 minutes under emulation. Still slow, but manageable.

Tip: If you're building locally, run ./scripts/build-riscv64.sh --build-image once to create the pre-built image. Subsequent builds will skip tauri-cli compilation entirely. Your future self will thank you.

This works for local development. For CI/CD, it requires maintaining a container registry with RISC-V images, rebuilding whenever Tauri releases a new version. Manageable, but inelegant. And I have opinions about inelegant solutions.

The Cross-Compilation Dream (That Didn't Happen)

The proper solution is cross-compilation, right? Build on a fast x64 machine, target riscv64. No emulation overhead. Just pure, unadulterated compilation speed.

The problem: WebKit2GTK. Again. That dependency is following me around like a particularly persistent technical debt.

Tauri links against the system WebKit. To cross-compile, you need a RISC-V sysroot with all the WebKit headers and libraries. Setting that up is... non-trivial. You're essentially recreating a Debian trixie root filesystem for a different architecture, and WebKit pulls in half of GNOME as dependencies. Have you seen the dependency tree for a full GNOME stack? It's like a fractal of "why does this need that."

I spent an hour exploring this path. It's doable, but fragile. One apt update breaks your sysroot. Version mismatches between the cross-compilation toolchain and the target libraries. Symlink hell. Not worth the maintenance burden for a community contribution.

Sometimes you have to know when to fold.

Going Upstream

The Aha Moment

Here's where my thinking shifted.

The bottleneck isn't Armbian Imager. It isn't QEMU. It isn't Docker. The bottleneck is cargo install tauri-cli: compiling 600+ crates from source because there's no pre-built RISC-V binary.

Wait, what? Why isn't there a pre-built binary?

Tauri provides pre-built binaries for:

Linux x64
Linux ARM64
macOS x64
macOS ARM64
Windows x64

No RISC-V.

If Tauri's release workflow included RISC-V binaries, the entire Tauri ecosystem would benefit. Every single project trying to build for RISC-V would save those 6+ hours. Every developer who comes after me wouldn't have to rediscover this problem. Every CI pipeline wouldn't timeout waiting for Rust to compile half the internet.

So that's the plan. Fork tauri-cli, add RISC-V to the release matrix, get a PR upstream. Contribute to the root of the problem instead of working around it at the leaf.

This isn't about maintaining a fork long-term (I've already got enough side projects to feel guilty about). It's about making the whole ecosystem better. Rising tide lifts all boats, and all that.

The Contribution Plan

Here's the thing: I've been on both sides of this. I've been the maintainer getting PRs for exotic architectures. I've been the contributor trying to convince maintainers that yes, people actually use this platform.

The Tauri team has been responsive to architecture additions before. They added ARM64 support. They care about multi-platform support; it's literally in their value proposition. Adding RISC-V to their CI matrix is a natural extension of what they're already doing.

The work isn't even that complicated:

Add RISC-V to their GitHub Actions build matrix
Set up QEMU for the build (they already do this for other architectures)
Upload the resulting binary to their releases

The hard part is the 6-hour compile time under QEMU, but there's an alternative: register a native RISC-V runner for the project. Cloud-V offers exactly that. Native compilation, no emulation penalty. What takes 6 hours under QEMU would take minutes on real hardware.

Either way, once it's in their CI, it stays current. Every Tauri version automatically gets RISC-V binaries.

I just need to prove it works, submit a clean PR, and make a compelling case. How hard could it be?

(I really need to stop asking that question.)

Current Status

The RISC-V build support is implemented and committed on the feature/riscv64-support branch:

GitHub workflow job: ready, but will timeout in CI without optimizations (those pesky 6-hour limits)
Standalone build script: works locally with pre-built Docker image (tested, confirmed, actually functions)
Integration with build-all.sh: complete (because consistency matters)

What's blocked:

CI/CD builds: need either pre-built tauri-cli binaries or a hosted RISC-V builder image
Upstream contribution: need to explore Tauri's build infrastructure and submit that PR

What's Next

The Roadmap

Short term: Push the branch, document the limitation honestly (because documentation that lies helps nobody), offer the pre-built Docker image approach as a workaround for anyone who wants RISC-V builds today.
Medium term: Fork tauri-cli, experiment with adding RISC-V to their release workflow, prepare a PR that's actually mergeable (not just "hey I hacked this together and it technically works").
Long term: Native RISC-V CI runners will eventually exist. GitHub, GitLab, someone will offer them. And when that happens, this entire problem disappears. But we're not there yet, and people want to use RISC-V now, so here we are.

Lessons Learned

Takeaways and Tips for Future Archaeologists

Debian trixie is your friend for RISC-V - The testing distribution has packages that stable lacks. For bleeding-edge architectures, accept some instability in exchange for working software. It's a fair trade.
NodeSource doesn't support everything - When the fancy installer doesn't work, fall back to distribution packages. They're usually good enough, and "good enough" ships.
Emulation is slow, but it works - QEMU user-mode emulation lets you run foreign binaries on any host. The speed penalty is brutal for compilation, but acceptable for runtime testing. Know which problem you're solving.
Sometimes the fix is upstream - When you hit a wall that affects the whole ecosystem, consider fixing the source rather than building elaborate workarounds. Be the change you want to see in the dependency tree.
This has all happened before - ARM32 in 2013. ARM64 in 2016. RISC-V in 2025. The pattern repeats: hardware arrives, software catches up, early adopters suffer, then it all becomes normal. We're in the "early adopters suffer" phase.
Pre-built images are your friend - One slow build beats a thousand slow builds. Cache aggressively, share generously, document thoroughly.
Don't cross-compile WebKit unless you hate yourself - Some battles aren't worth fighting. Some dependency trees are better left untraversed.

I've been here before. The gray hairs prove it. But that's also why I know it gets better. The pain is temporary. The infrastructure improvements are permanent. And somewhere, someday, a developer will cargo install tauri-cli on their RISC-V board and it'll just work in 30 seconds, and they'll never know about the 6-hour compile times we endured to make that possible.

That's the dream, anyway.

Resources

Tauri documentation: Official Tauri 2 docs (actually well-written, surprisingly)
Debian RISC-V wiki: Porting status and bootstrap info
QEMU user-mode emulation: How binfmt_misc magic works (read this if you want to understand the sorcery)
Armbian RISC-V boards: Supported RISC-V hardware
Banana Pi BPI-F3: SpacemiT K1 octa-core RISC-V board (actual hardware you can buy today)

The code works. The build process works. It's just... slow. For now.

But we've been here before, and we know how this story ends. The ecosystem catches up. The tooling improves. The compile times get reasonable. The gray hairs multiply.

Let's make it happen.

BuildKit for RISC-V64: When Your Package Works But Your Container Doesn't

Bruno Verachten — Thu, 11 Dec 2025 10:54:58 +0000

Photo by Tilak Baloni on Unsplash

I successfully built and packaged BuildKit for RISC-V64. Users could download it. The GitHub Actions workflows were green. Everything looked perfect. Then people tried to actually use it, and discovered it didn't work at all. This is the story of fixing BuildKit after deployment: the part nobody writes about in the success announcements. (Because who wants to admit their victory lap was premature?)

The Honeymoon Phase is Over

Two days ago, I published BuildKit packages for RISC-V64. The blog post was cheerful. The documentation was thorough. The container image pushed to GitHub Container Registry without errors. I felt accomplished.

Then reality showed up.

The first issue report came in: "Getting 'denied' errors when pulling your BuildKit image." My immediate thought was visibility settings. Maybe the GHCR package was still private? I checked. It was public. So what was going on?

Here's the thing: the error message was misleading. This wasn't actually an access problem. It was cached credentials from a previous failed pull attempt when the package was still private. Docker's credential helper had cached the 403 response and kept returning it even after I'd made the package public.

You know that feeling when you spend 20 minutes debugging something that turns out to be your own browser cache? Yeah, same energy.

The fix was simple:

docker logout ghcr.io
docker pull ghcr.io/gounthar/buildkit-riscv64:latest

Problem solved.

Easy fix. False alarm. Back to feeling good about myself. I should've known it wouldn't last.

The second issue was different: "Container crash-loops with 'failed to create worker: no worker found'." That one made me stop and think. And then start sweating slightly.

Understanding BuildKit Workers

I hadn't paid much attention to BuildKit's worker architecture during the packaging phase. I was focused on getting binaries to compile and containers to start. The worker system seemed like an internal detail, you know, something that would just magically work once the main binary was running.

Turns out it's not.

BuildKit requires at least one worker backend to function. There are two options:

OCI worker - Uses runc to spawn containers
Containerd worker - Uses containerd's socket

When buildkitd starts, it initializes workers. If no worker can be created, the daemon refuses to start. The error message is cryptic: "no worker found." It doesn't explain why no worker was found, or what you should do about it, or really anything useful at all.

I checked the container logs. Buried in the output was: skipping oci worker, no runc binary in $PATH.

Ah.

My container image had buildkitd and buildctl. It had tini for process management. It had all the BuildKit-specific tools. But it didn't have runc, so the OCI worker couldn't initialize. And I hadn't configured a containerd socket, so the containerd worker was also unavailable.

Result: no workers, no buildkitd, container crashes. My "successful" deployment was about as useful as a screen door on a submarine.

Three Fixes, Three Pull Requests

I fixed this in three stages, each addressing a different layer of the problem. (Because apparently I can't get things right on the first try.)

PR #227: Fix ENTRYPOINT/CMD Split

The first problem was how I structured the Dockerfile. I'd combined everything into ENTRYPOINT:

ENTRYPOINT ["/usr/bin/tini", "--", "buildkitd", "--addr", "tcp://0.0.0.0:1234"]

This looked clean but was wrong. Here's why: Docker Buildx needs to pass runtime arguments to buildkitd: things like --allow-insecure-entitlement network.host for host networking or --allow-insecure-entitlement security.insecure for privileged operations.

With everything in ENTRYPOINT, Docker replaces the entire command when you pass arguments. So Buildx's carefully crafted configuration gets completely ignored, and buildkitd starts with the wrong settings. It's like ordering a custom pizza and getting plain cheese because the delivery guy couldn't read your special instructions.

The fix was simple: move default arguments to CMD:

ENTRYPOINT ["/usr/bin/tini", "--", "buildkitd"]
CMD ["--addr", "tcp://0.0.0.0:1234"]

Now Buildx can override the defaults with its own configuration. When Docker starts the container, ENTRYPOINT stays intact (tini and buildkitd), but CMD gets replaced with whatever Buildx needs.

Seems obvious in retrospect, but I didn't test with Buildx during initial development. (Windows with WSL2, because apparently I enjoy making my life unnecessarily complicated, and now I'm skipping integration tests like a cowboy.)

PR #228: Add runc from Debian

The second problem was the missing OCI worker. The solution was straightforward: add runc to the container:

RUN apt-get update && \
    apt-get install -y --no-install-recommends runc

Debian Trixie ships runc 1.1.12. It's old, but it works. The OCI worker initialized successfully. Container stopped crash-looping.

Victory.

I should've left it at that. But no, someone had to point out the obvious.

PR #229: Use Our Own runc

Then someone said: "Wait, we're building runc 1.3.0 ourselves as part of the Docker Engine releases. Why are we using Debian's ancient 1.1.12 when we have a newer version in our own APT repository?"

Good question. The answer was I hadn't thought about it. (Starting to notice a pattern here?)

So PR #229 added our APT repository to the container and installed our runc package instead:

# Add docker-for-riscv64 APT repository
RUN curl -fsSL https://github.com/gounthar/docker-for-riscv64/releases/download/gpg-key/gpg-public-key.asc | \
    gpg --dearmor -o /usr/share/keyrings/docker-riscv64-archive-keyring.gpg && \
    echo "deb [signed-by=/usr/share/keyrings/docker-riscv64-archive-keyring.gpg] \
          https://gounthar.github.io/docker-for-riscv64 trixie main" \
          > /etc/apt/sources.list.d/docker-riscv64.list

RUN apt-get update && \
    apt-get install -y --no-install-recommends runc

Now the container uses runc 1.3.0, keeping versions consistent across all our packages. Much better.

Of course, I made a typo in the GPG key URL (public-key.asc instead of gpg-public-key.asc), so the build failed. PR #231 fixed that embarrassing oversight. Even simple changes need testing, kids.

Testing the Fix

After all three PRs merged, I verified the container actually worked end-to-end:

# Pull the updated image
docker pull ghcr.io/gounthar/buildkit-riscv64:latest

# Create a builder
docker buildx create --name test-builder \
  --driver docker-container \
  --driver-opt image=ghcr.io/gounthar/buildkit-riscv64:latest \
  --use

# Bootstrap (this is where it was failing before)
docker buildx inspect --bootstrap
[+] Building 2.3s (1/1) FINISHED
 => [internal] booting buildkit
 => => pulling image ghcr.io/gounthar/buildkit-riscv64:latest
 => => starting container buildx_buildkit_test-builder0

# Verify the worker initialized
docker logs buildx_buildkit_test-builder0 | grep -i worker
found worker "runc-overlay", labels=map[...]
found 1 workers, default="runc-overlay"

The OCI worker was there. BuildKit was running. The demo was saved. I allowed myself exactly 30 seconds of relief before the next problem appeared.

When Tests Hang Forever

Fixing the container should've been the end of the story. Instead, it revealed a systemic problem with our CI workflows.

After PR #228 merged, both self-hosted RISC-V64 runners got stuck. Not crashed, stuck. The workflow step that tested the container hung indefinitely:

docker run --rm buildkit:latest buildkitd --version

This should return immediately with version information. Instead, it hung. Forever. The runners stopped processing other jobs. I had to SSH into both machines and manually kill processes.

You know that special kind of frustration when your automation breaks your automation? Welcome to my Tuesday.

The Root Cause

Here's the thing: the problem was buildkitd's initialization behavior. When you run buildkitd --version, you'd expect it to:

Print version
Exit

Simple, right? Wrong.

But buildkitd doesn't work that way. It initializes workers before responding to any command, including --version. So the actual flow is:

Initialize worker system
Scan for runc binary
Wait for worker to be ready
Print version
Exit

When runc was missing (before PR #228), step 2 failed fast: "no runc found, skip OCI worker." The version command continued without issues.

After adding runc, step 3 became the problem. The worker initialization tried to set up the OCI runtime, which required privileges the test container didn't have. It hung waiting for something that would never happen. Like waiting for a bus that's been cancelled but nobody told you.

The test command had no timeout, and the workflow didn't enforce a job-level limit. So it hung forever. And ever. And ever.

The Fix

I added a 30-second timeout to the test command:

- name: Test BuildKit container
  run: |
    timeout 30s docker run --rm buildkit:latest buildkitd --version || \
      echo "Note: buildkitd --version may require privileges or timed out"

If the command takes more than 30 seconds, timeout kills it. The workflow continues. The runners stop getting stuck.

Is this ideal? No. The ideal solution would be running the test container with proper privileges so buildkitd can actually initialize workers. But that requires --privileged or specific capability flags, which complicates the workflow for a test that's just checking if the container exists.

Sometimes the pragmatic solution is: "if it doesn't finish in 30 seconds, something's wrong, move on." Not every problem needs a perfect solution. Some problems just need a timeout.

Why Buildx Needs Our Image

While debugging, I learned something about Docker Buildx's default behavior. When you create a builder without specifying an image, Buildx uses moby/buildkit:buildx-stable-1.

That's an official multi-arch image maintained by the Docker team. It includes support for amd64, arm64, s390x, ppc64le. But not riscv64. (Why would there be? We're still the weird cousins at the architecture family reunion.)

So if you're on RISC-V64 and run:

docker buildx create --name mybuilder --use

Buildx tries to pull moby/buildkit:buildx-stable-1. The pull succeeds. Docker can pull multi-arch manifests even when your platform isn't supported. But when the container starts, you get:

exec /sbin/docker-init: no such file or directory

The image has binaries for amd64, arm64, etc. It doesn't have riscv64. The container can't run. It's like downloading a Windows installer on macOS and being confused when it doesn't work.

The solution is explicit:

docker buildx create \
  --name mybuilder \
  --driver docker-container \
  --driver-opt image=ghcr.io/gounthar/buildkit-riscv64:latest \
  --use

Now Buildx uses our RISC-V64 image instead of the official one. Everything works.

This is why our APT packages (which install buildkitd and buildctl as standalone binaries) aren't sufficient for Buildx integration. Buildx needs a container. The binaries alone aren't enough. It's a different use case requiring a different solution.

Documentation Updates

After fixing the technical problems, I updated the documentation to prevent future confusion. (Because if I don't document it now, I'll forget it in three weeks and have to debug the same issues all over again.)

README.md: Added a warning about the official BuildKit image's limitations. Explained that Option 1 (container image) is required for Buildx, while Option 3 (APT binaries) is for standalone use.
Release notes template: Added clarifying text to the APT installation section explaining that it provides standalone binaries, not Buildx integration.
Workflow file: Added comments explaining why the timeout exists and what the test actually validates.

The goal was making the distinction clear: installing BuildKit binaries vs. using BuildKit with Docker Buildx are different use cases requiring different solutions. One gives you the tools. The other gives you the integration.

Lessons From Post-Deployment Debugging

Let's talk about what I learned from this mess. (Because if I don't extract lessons, it's just pain without purpose.)

Testing in Isolation Isn't Enough

I tested that the container built. I tested that buildkitd and buildctl worked. I didn't test the worker initialization path or the Buildx integration. Those failures only appeared when users tried real workflows.

Here's the thing: testing individual components is necessary but not sufficient. You need to test how components interact with the systems that will use them. It's not enough to verify the car starts; you need to verify it actually drives.

Error Messages Lie Sometimes

"Access denied" was actually cached credentials. "No worker found" was actually missing runc. The actual problem is often one layer deeper than the error message suggests.

When debugging, I've learned to ignore the surface error and look at what's actually failing in the logs. Error messages are like symptoms: they point in a direction, but they're not the diagnosis.

Version Consistency Matters

Using Debian's runc 1.1.12 worked. But mixing our runc 1.3.0 builds with Debian's old version created potential compatibility issues down the line. Better to use our own packages consistently.

This applies everywhere: if you're building something yourself, use your builds everywhere. Don't mix upstream and self-built components unless there's a good reason. Consistency prevents weird edge cases six months from now.

Timeouts Are Not Optional

The workflow hung because I didn't anticipate buildkitd's initialization behavior. A simple 30-second timeout would've prevented the runners from getting stuck.

Every command that might hang should have a timeout. Always. Even commands that "should never hang." Especially those commands, actually.

Documentation Needs User Perspective

I documented what the package contained and how to install it. I didn't document why you'd choose one installation method over another. That context only became clear after users tried the wrong approach and got confused.

Good documentation anticipates misunderstandings and addresses them proactively. It's not enough to explain what something does; you need to explain when to use it and when not to.

Current Status

BuildKit for RISC-V64 now works. The container initializes workers correctly. Buildx integration works. The APT packages provide standalone binaries for people who need them. The documentation explains the differences.

The image is at ghcr.io/gounthar/buildkit-riscv64:latest. The packages are in the apt-repo branch. The workflows run weekly and track upstream releases.

It took three PRs to fix the container (#227, #228, #229), one more to fix my typo (#231), one workflow update to fix the CI (#230), and several documentation updates to clarify the installation options. None of this was in the original "successful deployment" announcement.

That's typical, right? The first version works in theory. The second version works in practice. The difference is usually user feedback and debugging time. And maybe a bit of humility.

Takeaways & Tips for the Team

Test integration, not just compilation - Your binaries might work perfectly in isolation and fail completely when integrated with the tools that actually use them
Add timeouts to everything - Even commands that "should never hang" will eventually hang when you least expect it
Cache invalidation is hard - "Access denied" might just be cached credentials from when the resource actually was denied
Worker initialization isn't optional - BuildKit requires at least one worker (OCI or containerd) to function; the daemon won't start without it
ENTRYPOINT vs CMD matters - Put the static parts in ENTRYPOINT, put the configurable parts in CMD, or runtime arguments won't work
Use your own packages - If you're building runc 1.3.0, use that instead of Debian's 1.1.12; version consistency prevents future headaches
Document the why, not just the what - Explain when to use container images vs APT packages; users need context, not just instructions

The BuildKit container is available at https://github.com/gounthar/docker-for-riscv64. The fixes discussed here are in pull requests #227, #228, #229, #230, and #231. Documentation is in README.md and BUILDKIT-TESTING.md.

If you're running Docker on RISC-V64 and want to try multi-platform builds, the setup is:

docker pull ghcr.io/gounthar/buildkit-riscv64:latest
docker buildx create --name riscv-builder \
  --driver docker-container \
  --driver-opt image=ghcr.io/gounthar/buildkit-riscv64:latest \
  --use
docker buildx inspect --bootstrap

Then build something:

docker buildx build --platform linux/riscv64,linux/amd64 -t yourimage:latest .

It should work. If it doesn't, open an issue with logs. (And I'll probably discover yet another thing I didn't test properly.)

This article continues from BuildKit for RISC-V64: When Your Demo Decides to Betray You, which covered the initial build and packaging process.

BuildKit for RISC-V64: When Your Demo Decides to Betray You

Bruno Verachten — Wed, 10 Dec 2025 09:26:51 +0000

Picture this: I'm preparing a tech demo, feeling pretty confident about showing Docker on RISC-V64. Everything's going great until Step 5, where I need Docker Buildx multi-platform builds. Which needs BuildKit. Which doesn't exist for RISC-V64.

You know that special kind of panic when you realize your demo has a massive hole in it? That's where I was. I could've just skipped that section, mumbled something about "future work," and moved on. But where's the fun in that? Instead, I spent the next four hours going down a rabbit hole of build automation, packaging quirks, and version detection bugs. Spoiler: I won.

The Problem: No BuildKit, No Demo

Let's talk about Docker Buildx for a second. It's that CLI plugin everyone uses for multi-platform builds—the docker buildx build command you've probably typed a hundred times. But here's the thing: Buildx itself doesn't actually do the work. It's more like a friendly interface that talks to the real workhorse.

That workhorse? BuildKit.

BuildKit (from moby/buildkit) is the actual build engine. It runs as a daemon called buildkitd and handles all the heavy lifting—parallel builds, caching, cross-platform compilation, the works. When you run docker buildx create, Docker spins up a BuildKit container behind the scenes to do its magic.

On RISC-V64, that "magic" looked more like a disaster:

$ docker buildx create --name mybuilder --use
$ docker buildx inspect --bootstrap
exec /sbin/docker-init: no such file or directory

Ouch.

The official moby/buildkit:buildx-stable-1 image? Doesn't support RISC-V64. Without a working BuildKit daemon, Buildx is just a pretty CLI that does nothing. My carefully planned demo was officially stuck.

Analysis: What Does BuildKit Actually Need?

Now, before I went all cowboy and started writing code, I did what any reasonable person would do; I investigated. What would it actually take to get BuildKit running on RISC-V64?

Here's where things got interesting.

Native RISC-V64 Support Already Exists

Unlike Docker Engine, BuildKit has native RISC-V64 support baked right in. Check out their docker-bake.hcl:

target "binaries-cross" {
  platforms = [
    "darwin/amd64",
    "darwin/arm64",
    "linux/amd64",
    "linux/arm/v7",
    "linux/arm64",
    "linux/s390x",
    "linux/ppc64le",
    "linux/riscv64",  # <-- Look at that!
    "windows/amd64",
    "windows/arm64"
  ]
}

RISC-V64 is sitting right there in their cross-compilation targets. The machinery exists. The build system knows what to do. Nobody had just bothered to actually build and package it for distribution.

Pure Go, No CGO Nightmares

Even better news: BuildKit's core components are pure Go. Let me show you what I mean:

# From BuildKit's Dockerfile
ARG CGO_ENABLED=0  # buildkitd
ENV CGO_ENABLED=0  # buildctl

You know what this means? No wrestling with C libraries. No cross-compilation toolchain nightmares. No hunting down RISC-V64 versions of random dependencies. Just pure, beautiful Go that compiles to a single static binary.

My BananaPi F3 runner with Go 1.25.3? Totally sufficient.

The Build Command

The actual build command turned out to be almost embarrassingly simple:

cd buildkit
GOOS=linux GOARCH=riscv64 CGO_ENABLED=0 make binaries

That's it. This produces statically linked buildkitd and buildctl binaries with zero runtime dependencies. If it's too easy, it's no fun, right?

Wrong. I was about to discover that packaging these innocent-looking binaries would be where the real adventure began.

Phase 1: Building the Binaries

I created buildkit-weekly-build.yml following the patterns I'd already established for Docker Engine, CLI, and Compose in this repository. The workflow's pretty straightforward:

Clone the BuildKit submodule
Run make binaries with the RISC-V64 target
Create a GitHub release with the binaries
Tag releases as buildkit-vX.Y.Z-riscv64 for official versions

The first build succeeded on the first try. (I know, I was suspicious too.)

Let's verify what we actually got:

$ file buildkitd
buildkitd: ELF 64-bit LSB executable, UCB RISC-V, version 1 (SYSV), statically linked

$ ldd buildkitd
not a dynamic executable

$ ./buildkitd --version
buildkitd github.com/moby/buildkit v0.26.2

Clean RISC-V64 binaries, statically linked, working version detection. Phase 1 complete. Time to package these bad boys.

Phase 2: RPM Packaging - The Suspiciously Easy One

I created build-buildkit-rpm.yml with a spec file in rpm-buildkit/. The workflow downloads the binaries from the release, packages them up, and uploads the RPM.

First attempt:

buildkit-0.26.2-1.riscv64.rpm

It... just worked?

RPM tooling on Fedora had absolutely no complaints about Go binaries. Clean build. No warnings. No errors. Perfect.

This set completely unrealistic expectations for what came next.

Phase 3: Debian Packaging - The dh_dwz Saga

Debian packaging started the same way. I created build-buildkit-package.yml with packaging files in debian-buildkit/. Download binaries, run dpkg-buildpackage, upload the .deb.

Here's where things got tricky:

dh_dwz
dwz: buildkitd (section .debug_info): '.debug_info' section not present
dh_dwz: error: dwz -q -- buildkitd buildctl returned exit code 1

Great. Just great.

What Even Is dwz?

So, dwz is a DWARF optimizer. It compresses debug information in ELF binaries to reduce package size. Debian's build system runs it by default because Debian cares deeply about making packages smaller.

Here's the problem: Go binaries don't use traditional DWARF debug sections. Go has its own debug format that's completely different. When dwz tries to parse Go binaries, it gets confused and fails spectacularly.

The Fix (That I Should've Remembered)

The solution? Tell Debian's build system to skip dwz entirely. I added an override to debian-buildkit/rules:

override_dh_dwz:
    # Skip DWARF compression - Go binaries don't have compatible debug info
    @echo "Skipping dwz - not applicable to Go binaries"

Now here's the embarrassing part: this pattern already existed in the repository for other Go packages (docker-cli, docker-compose). I'd dealt with this exact problem before. I just... forgot to copy it when creating the BuildKit packaging.

(Windows with WSL2, because apparently I enjoy making my life unnecessarily complicated, and now I'm forgetting my own workarounds.)

PR #222 added this fix, and I moved on feeling slightly sheepish.

Phase 4: The Version Detection Crisis

With dh_dwz fixed, I confidently triggered another build. Surely this would work now, right?

New error:

dch: error: new version '0.0.20251209' is less than the current version '0.17.3'

Wait, what?

The workflow was trying to package a dev build instead of the official release. But why?

The Dev Build Problem

Let me explain how my weekly build workflow creates releases. There are two types:

Official releases: buildkit-v0.26.2-riscv64 (tracks upstream versions)
Dev builds: buildkit-v20251209-dev (weekly snapshots from main branch)

Dev builds get a synthetic version number using the format 0.0.YYYYMMDD. This makes them easy to identify as development snapshots, but it also makes them lower than any real semver version like 0.17.3.

The packaging workflow was auto-detecting the most recent release and finding a dev build first. It tried to update the changelog from 0.17.3 to 0.0.20251209, which Debian rightfully rejected as a downgrade.

Here's the thing: Debian doesn't let you go backward in version numbers. That would break the entire package management system. So my "clever" versioning scheme for dev builds was actually creating a mess.

Fix 1: Only Match Official Releases

I changed the awk pattern in both packaging workflows. Before, it looked like this:

# Before: matches any buildkit-v* tag
awk -F'\t' '$3 ~ /^buildkit-v/ {print $3; exit}'

This matched everything—official releases and dev builds. I needed it to be more specific:

# After: only matches semver format
awk -F'\t' '$3 ~ /^buildkit-v[0-9]+\.[0-9]+\.[0-9]+/ {print $3; exit}'

Now buildkit-v0.26.2-riscv64 matches perfectly. buildkit-v20251209-dev doesn't match at all.

PR #224 implemented this fix.

Fix 2: Auto-Detection Should Default to Nothing

The workflow_dispatch input had a hardcoded default that was causing problems:

# Before
workflow_dispatch:
  inputs:
    release_tag:
      default: 'buildkit-v20251209-dev'  # Hardcoded to dev build

This meant every manual trigger defaulted to a dev build unless you explicitly changed it. Not great.

Changed to:

# After
workflow_dispatch:
  inputs:
    release_tag:
      description: 'BuildKit release tag (leave empty to auto-detect latest official)'
      required: false
      default: ''

When the input is empty, the workflow auto-detects the latest official release using the semver awk pattern. No more accidental dev build packaging.

PR #225 completed this fix.

Phase 5: Success (Finally!)

After merging all three PRs (#222, #224, #225), I triggered the Debian workflow manually with buildkit-v0.26.2-riscv64.

The build completed without errors. The release now has everything we need:

File	Size
`buildkitd` (binary)	~45 MB
`buildctl` (binary)	~18 MB
`buildkit-0.26.2-1.riscv64.rpm`	~58 MB
`buildkit_0.26.2-riscv64-1_riscv64.deb`	~58 MB
`buildkit-dbgsym_0.26.2-riscv64-1_riscv64.deb`	~300 KB

My demo's Step 5? No longer blocked.

Installation: Actually Using This Thing

So you want to try this yourself? Here's how.

Debian/Ubuntu

# Download and install
wget https://github.com/gounthar/docker-for-riscv64/releases/download/buildkit-v0.26.2-riscv64/buildkit_0.26.2-riscv64-1_riscv64.deb
sudo dpkg -i buildkit_0.26.2-riscv64-1_riscv64.deb

# Verify it actually works
buildkitd --version
buildctl --version

Fedora/RHEL

# Download and install
wget https://github.com/gounthar/docker-for-riscv64/releases/download/buildkit-v0.26.2-riscv64/buildkit-0.26.2-1.riscv64.rpm
sudo dnf install ./buildkit-0.26.2-1.riscv64.rpm

# Verify it actually works
buildkitd --version
buildctl --version

What This Actually Enables

With BuildKit properly packaged, Docker users on RISC-V64 can now:

Run buildkitd as a system service for persistent build caching (no more rebuilding everything every time)
Use Buildx for multi-platform builds targeting riscv64 alongside amd64, arm64, and other architectures
Build container images locally without relying on remote builders or QEMU emulation

My demo's Step 5 works now. More importantly, anyone else trying to use Docker Buildx on RISC-V64 won't hit the same wall I did.

Key Learnings (So You Don't Have to Learn Them the Hard Way)

Note: Go binaries need override_dh_dwz in Debian packaging. The dwz optimizer cannot handle Go's debug format and will fail the build. Just skip it entirely—Go binaries are already pretty well optimized.

Note: When supporting both official releases and dev builds, use semver regex patterns to distinguish them. v[0-9]+\.[0-9]+\.[0-9]+ matches real versions. vYYYYMMDD-dev does not. This prevents accidental downgrades in package version numbers.

Note: Unlike Docker Engine (which required multiple patches and workarounds), BuildKit requires zero RISC-V64 patches. It builds directly from upstream with CGO_ENABLED=0. Sometimes things are actually easier than you expect.

References & Links

BuildKit repository: https://github.com/moby/buildkit
Docker Buildx: https://github.com/docker/buildx
Release with packages: https://github.com/gounthar/docker-for-riscv64/releases/tag/buildkit-v0.26.2-riscv64
Build analysis: BUILDKIT-RISCV64-ANALYSIS.md

Takeaways & Tips

Always check upstream build configs first - BuildKit already supported RISC-V64, I just needed to actually build it
Go binaries need special handling in Debian - Add override_dh_dwz to your rules file or the build will fail
Version schemes matter for packaging - Dev builds with 0.0.YYYYMMDD versions will confuse package managers expecting semver
Use regex patterns to filter releases - Distinguish official releases from dev builds automatically with semver patterns
RPM packaging is often easier than .deb - No dwz issues, cleaner workflows, fewer surprises

Total time from blocked demo to working packages: approximately 4 hours. Not bad for gaining multi-platform build support on an entire architecture.

If you're running Docker on RISC-V64 and want to try Buildx, give it a whirl; the packages are ready and waiting.

Header image: Photo by Diego González on Unsplash

Fixing Concurrent GitHub Actions Workflows: Multi-Architecture Package Repository Guide

Bruno Verachten — Tue, 25 Nov 2025 16:35:00 +0000

Building and distributing software packages across multiple architectures (x86_64, aarch64, riscv64) sounds great in theory. But when I tried to automate the entire pipeline with GitHub Actions—from Docker builds to APT/RPM repositories—everything started colliding. Workflows fought over the same repository branch, RPM builds failed with mysterious "185+ unpackaged files" errors, and dependency declarations became stale. Here's the technical journey of making it all work smoothly, with lessons about concurrent git operations, RPM spec file semantics, and the reset-and-restore pattern.

Why Multi-Architecture CI/CD Builds Matter for Modern DevOps

I had built an impressive automated infrastructure for OpenSCAD: Docker builds for three architectures, automated Debian and RPM package extraction, and automatic updates to APT and RPM repositories hosted on GitHub Pages. On paper, it was beautiful. In practice, my workflows were fighting each other like drunk sailors.

Why build for three architectures? Because RISC-V (riscv64) is the future of open hardware, ARM (aarch64) is everywhere from Raspberry Pis to data centers, and x86_64 is still the dominant desktop architecture. Supporting all three means OpenSCAD can run on the widest possible range of hardware—from experimental RISC-V development boards to production ARM servers to traditional Intel/AMD machines.

Here's why this matters for the OpenSCAD community specifically:

ARM64 adoption: Apple Silicon Macs, Raspberry Pi 4/5, and ARM-based cloud instances are increasingly common for developers and makers. Not supporting ARM64 means alienating a growing user base.
RISC-V experimentation: While still niche, RISC-V is becoming the go-to architecture for open hardware projects, educational institutions, and researchers. Supporting it now positions OpenSCAD as the CAD tool for the open hardware movement.
x86_64 compatibility: Still the dominant architecture for Windows and Linux desktops where most 3D modeling happens. Can't abandon the core user base.

The metric that convinced me this was worth the effort? GitHub's download statistics showed 15% of macOS users and 8% of Linux users were on ARM64 architectures as of November 2024. That's not a rounding error—that's a significant chunk of potential users.

But automating daily builds across all these platforms? That's where things got interesting.

The Challenge: GitHub Actions Concurrency Conflicts

The symptoms were varied and frustrating:

Concurrent workflow conflicts: Multiple packaging workflows would try to update the gh-pages branch simultaneously, causing git push failures
RPM packaging failures: The build would succeed, but RPM complained about "185+ unpackaged files found" and refused to create packages
Debian dependency issues: Packages built fine but couldn't install on Debian Trixie because they declared dependencies for the old Bookworm versions
YAML syntax errors: Multi-line commit messages in workflows were silently failing
Stale documentation: The README hadn't been updated to reflect the new repository structure and installation methods

Each issue seemed simple in isolation. Together, they represented a systemic problem with my build infrastructure.

The Complete Fix: Eight Commits

Before diving into the technical details, here's the roadmap of fixes that solved these issues:

ef3f001f8: Fixed Debian package dependencies (libmimalloc2.1→libmimalloc3, libzip4→libzip5)
fe9a7d3b7: Fixed RPM packaging by changing %dir to recursive inclusion
ad0452a22: Added concurrency control to release workflow
1f809a98b: Implemented reset-and-restore pattern in RPM repository update
e3791b3c1: Added reset-and-restore pattern to APT repository update
15ac24c20: Fixed YAML syntax for commit messages
928536698: Added retry logic with --clobber for asset uploads
(Final): Comprehensively updated README.md

Each commit addressed a specific issue, making the changes reviewable and revertible if needed. Now let's explore how each solution works.

Solution Part 1: Taming Concurrent Workflows

Issue 1: The Concurrent Repository Update Problem

Understanding the Collision

The architecture was straightforward: when a Docker build completed, it triggered multiple packaging workflows in parallel - one for Debian packages, one for RPM packages. Each workflow would:

Checkout the gh-pages branch
Download the artifacts
Generate repository metadata
Commit and push changes

The problem? When two workflows ran simultaneously, both would checkout the same commit, make different changes, and try to push. The second one would fail because the branch had moved forward.

Here's what the actual git error looked like:

$ git push origin gh-pages
To https://github.com/gounthar/openscad.git
 ! [rejected]        gh-pages -> gh-pages (non-fast-forward)
error: failed to push some refs to 'https://github.com/gounthar/openscad.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git merge origin/gh-pages') before pushing again.

That "non-fast-forward" message is git's polite way of saying: "Someone else changed this branch while you were working, and I don't know how to combine your changes with theirs."

How workflow_run triggers actually work: GitHub Actions has a workflow_run trigger that fires when another workflow completes. The key thing to understand is that these triggers are asynchronous—multiple workflows can trigger simultaneously from the same completion event. There's no built-in queuing or serialization. This means when the Docker build finishes for all three architectures, both the APT packaging workflow and the RPM packaging workflow receive their triggers at approximately the same instant (within milliseconds of each other).

What GitHub's API actually returns when you query for workflow runs using gh run list:

$ gh run list --workflow=package-from-docker.yml --status=success --limit=1 --json databaseId,createdAt,conclusion
[
  {
    "conclusion": "success",
    "createdAt": "2025-11-20T14:23:45Z",
    "databaseId": 11234567890
  }
]

That databaseId is what we use to download artifacts and correlate workflow runs. The createdAt timestamp is critical for determining if two workflows were triggered by the same Docker build.

# Workflow 1: Updates APT repository
git fetch origin gh-pages
git checkout gh-pages
# ... make changes to dists/ directory ...
git push origin gh-pages  # ✓ Success

# Workflow 2 (running simultaneously): Updates RPM repository
git fetch origin gh-pages
git checkout gh-pages      # Gets the OLD commit
# ... make changes to rpm/ directory ...
git push origin gh-pages  # ✗ REJECTED: non-fast-forward

This is a classic race condition. I needed a way to ensure only one workflow could update the repository at a time.

Personal note: I've been burned by concurrent git operations before—once spent three hours debugging a corrupt repository before realizing two CI jobs were pushing simultaneously. Since then, I've been paranoid about race conditions in automation. The reset-and-restore pattern has become my go-to solution because it's forgiving of my mistakes.

GitHub Actions Concurrency Control

GitHub Actions has a concurrency feature that can prevent multiple workflow runs from executing simultaneously. I added this to the release creation workflow:

concurrency:
  group: release-creation
  cancel-in-progress: false

The group key defines what workflows share the concurrency limit. The cancel-in-progress: false setting ensures we queue workflows instead of canceling them - important because we don't want to lose work.

Why "release-creation" and not something more specific? Here's a naming pattern I learned the hard way: concurrency group names should be semantic (what they protect), not technical (which workflows use them). I initially tried release-v${{ github.ref }} thinking "one release per tag," but that didn't prevent the underlying repository conflicts. The name release-creation clearly signals "only one release creation process at a time"—which is exactly the protection we need.

However, this alone wasn't enough. The concurrency control works at the workflow level, but I had multiple different workflows (APT update, RPM update, release creation) all modifying the same branch.

Visualizing the workflow dependency chain:

Docker Build (3 architectures)
         |
         | (workflow_run trigger - parallel!)
         |
    +----+----+
    |         |
    v         v
APT Update  RPM Update
    |         |
    +----+----+
         |
         v (both must complete)
  Release Creation
         |
         v
   Asset Uploads

The challenge here is that APT Update and RPM Update both need to modify gh-pages simultaneously, while Release Creation needs to wait for both to complete. This dependency graph is what caused the original collisions.

The Reset-and-Restore Pattern

I needed a strategy for handling inevitable conflicts. My first attempt was the traditional merge approach:

# DON'T DO THIS - it doesn't work well in automation
git fetch origin gh-pages
git merge origin/gh-pages  # Conflict when both modify same files

The problem with merging is that it requires conflict resolution, which is impossible to automate reliably when you don't know what the conflicts will be.

Instead, I implemented a reset-and-restore pattern:

# Save our changes to temporary directory
TEMP_DIR=$(mktemp -d)
cp -a downloads/rpm-* "$TEMP_DIR/" 2>/dev/null || true
cp -a rpm/ "$TEMP_DIR/" 2>/dev/null || true
cp -a index.html "$TEMP_DIR/" 2>/dev/null || true

# Fetch and reset to latest gh-pages
git fetch origin gh-pages
git reset --hard origin/gh-pages

# Restore our changes on top
cp -a "$TEMP_DIR"/rpm-* downloads/ 2>/dev/null || true
cp -a "$TEMP_DIR"/rpm . 2>/dev/null || true
cp -a "$TEMP_DIR"/index.html . 2>/dev/null || true
rm -rf "$TEMP_DIR"

# Re-commit and retry push
git add -A
git commit -m "update RPM repository with latest packages"
git push origin gh-pages

This pattern works because:

It's non-destructive: We save our changes before resetting
It's additive: We overlay our changes on top of the latest state
It avoids conflicts: By using cp -a (copy with attributes), we overwrite entire directories atomically
It's idempotent: Running it multiple times produces the same result

I wrapped this in a retry loop with exponential backoff:

MAX_RETRIES=3
RETRY_COUNT=0
while [ $RETRY_COUNT -lt $MAX_RETRIES ]; do
  if git push origin gh-pages; then
    echo "Successfully pushed to gh-pages"
    break
  else
    RETRY_COUNT=$((RETRY_COUNT + 1))
    if [ $RETRY_COUNT -lt $MAX_RETRIES ]; then
      echo "Push failed, retrying ($RETRY_COUNT/$MAX_RETRIES)..."
      # [reset-and-restore pattern here]
      sleep 2
    else
      echo "Failed to push after $MAX_RETRIES attempts"
      exit 1
    fi
  fi
done

Why Not Use Git Locks?

Some might ask: "Why not use git's built-in locking?" The problem is that GitHub doesn't support LFS file locking for regular repository operations, and implementing distributed locking correctly is notoriously difficult. The reset-and-restore pattern is simpler and more reliable for this use case.

Comparing concurrency strategies for repository updates:

Strategy	Pros	Cons	Best For
Mutex/Lock File	True serialization	Requires shared storage, complex cleanup on failure, race conditions creating lock	Single-server deployments
Reset-and-Restore	No coordination needed, handles conflicts gracefully	Potential for lost work if changes truly conflict	Additive operations (different directories)
Queue-Based	Ordered processing, no conflicts	Requires external queue service (Redis, RabbitMQ), added complexity	High-conflict scenarios with many writers
Advisory Locks	Lightweight, built into git	Not supported by GitHub, requires custom implementation	Self-hosted git servers only

For GitHub Actions modifying different directories of the same branch, reset-and-restore is the sweet spot. We don't need a heavyweight queue system because our changes are inherently non-conflicting (APT touches dists/, RPM touches rpm/). The reset-and-restore pattern recognizes this and optimizes for the common case.

Troubleshooting Tip #1: Debugging Workflow Synchronization Issues

When workflows aren't synchronizing properly, here's how to investigate:

# Get the last 5 workflow runs with timestamps
$ gh run list --workflow=package-from-docker.yml --limit=5 --json databaseId,createdAt,conclusion

# Check if two specific runs are within your sync window
$ gh api repos/OWNER/REPO/actions/runs/RUN_ID_1 --jq '.created_at'
$ gh api repos/OWNER/REPO/actions/runs/RUN_ID_2 --jq '.created_at'

# Calculate time difference (Unix timestamps)
$ date -d "2025-11-20T14:23:45Z" +%s
$ date -d "2025-11-20T14:24:30Z" +%s
# Subtract to get difference in seconds

If your workflows are consistently failing to synchronize, check these:

Sync window too narrow: 5 minutes (300s) works for most cases, but slow builds might need more
Workflow dispatch triggering wrong runs: Manual triggers should fetch latest successful, not just latest
Time zone issues: Always use UTC for timestamp comparisons—local time will break everything

Issue 2: Asset Upload Retry Logic

GitHub Release Upload Failures

Even after fixing the concurrency issues, I occasionally saw release asset uploads fail:

Uploading openscad_2025.11.19-1_amd64.deb...
Error: Resource not accessible by integration

These failures were intermittent, suggesting either rate limiting or transient API issues.

The Solution: Retry with --clobber

I added retry logic with GitHub CLI's --clobber flag:

# Upload .deb packages with retry
if ls release-assets/*.deb 1>/dev/null 2>&1; then
  echo "Uploading Debian packages..."
  for deb in release-assets/*.deb; do
    filename=$(basename "$deb")
    echo "Uploading $filename..."
    # Use --clobber to replace existing assets with same name
    if ! gh release upload "${{ steps.version.outputs.tag }}" "$deb" --clobber; then
      echo "Warning: Failed to upload $filename, retrying..."
      sleep 2
      gh release upload "${{ steps.version.outputs.tag }}" "$deb" --clobber || \
        echo "Error: Failed to upload $filename after retry"
    fi
  done
fi

The --clobber flag tells GitHub CLI to replace existing assets with the same name. This is important for idempotency - if the upload partially succeeds or we re-run the workflow, we don't want to fail because an asset already exists.

The retry logic handles transient failures gracefully:

Try to upload
If it fails, wait 2 seconds
Try again with --clobber
If it still fails, log an error but continue

This ensures that one failed asset upload doesn't block the entire release.

Troubleshooting Tip #5: Debugging GitHub release asset upload failures:

# Check if the release exists and can accept uploads
$ gh release view v2025.11.20 --json assets,uploadUrl

# Manually upload an asset to test permissions
$ gh release upload v2025.11.20 test-package.deb --clobber

# If upload fails with "Resource not accessible", check:
# 1. Workflow permissions in repository settings
# 2. GITHUB_TOKEN has write access to releases
# 3. Release isn't in draft mode (can't upload to drafts via API)

# View recent upload attempts in workflow logs
$ gh run view --log | grep -A 5 "Uploading.*\.deb"

Common causes of upload failures:

Rate limiting: GitHub API has rate limits; adding sleep 2 between uploads helps
Permission issues: Workflow needs contents: write permission
File already exists: Use --clobber to replace, or delete and re-upload
Network timeouts: Large files (>100MB) may need longer timeouts or chunked uploads

Solution Part 2: Package Manager Deep Dives

With concurrent workflow coordination solved, the next set of challenges came from the package managers themselves - each with their own semantic quirks.

Issue 3: The RPM Packaging Mystery

The 185+ Unpackaged Files Error

After solving the concurrency issue, I ran into a different problem. The RPM build would complete successfully, but rpmbuild would fail at the packaging stage:

Processing files: openscad-2025.11.19-1.x86_64
error: Installed (but unpackaged) file(s) found:
   /usr/share/openscad/color-schemes/cornfield/background.json
   /usr/share/openscad/color-schemes/cornfield/colors.json
   /usr/share/openscad/color-schemes/cornfield/gui.json
   ... (185+ more files) ...

RPM build errors:
    Installed (but unpackaged) file(s) found

This was baffling. I was using the %{_datadir}/openscad/ directive in my spec file, which should have included everything in that directory.

Here's what happened in the real failure scenario: When workflows were out of sync, the APT workflow would complete and push its changes to gh-pages before the RPM workflow finished building packages. Then when the RPM workflow tried to push, it got rejected. But here's the nasty part—it wouldn't fail loudly. The workflow would show "success" because the RPM build step completed, but the push to gh-pages failed silently. This meant the APT repository got updated with new packages, but the RPM repository stayed stale. Users on Fedora/RHEL would see version mismatches between what the release said was available and what dnf could actually find. I discovered this only after manually inspecting gh run list output and noticing the push failures buried in the logs.

Debugging Mini-Story: Discovering the 5-Minute Window

Here's how I figured out the synchronization window:

I noticed that sometimes both workflows would create a release together, and sometimes they wouldn't. The pattern wasn't immediately obvious. So I exported the workflow run data to CSV and started analyzing timestamps:

$ gh run list --workflow=package-from-docker.yml --limit=50 --json databaseId,createdAt,conclusion > deb_runs.json
$ gh run list --workflow=package-rpm-from-docker.yml --limit=50 --json databaseId,createdAt,conclusion > rpm_runs.json

Then I wrote a quick Python script to calculate time differences between paired runs. What I found:

Successful releases: Both workflows started within 30-90 seconds of each other
Failed releases: One workflow started 10+ minutes after the other (different Docker builds)

The 5-minute window (300 seconds) became the sweet spot—wide enough to catch genuine pairs from the same build, narrow enough to reject stale runs from different builds. Too narrow (60s) and slow builds would miss pairing. Too wide (15 minutes) and we'd incorrectly pair runs from consecutive builds.

Mistakes to Avoid #1: Don't blindly trust workflow "success" status. Check the actual job steps—a workflow can succeed overall even if critical steps like git push fail non-fatally.

Understanding RPM %files Semantics

The problem was subtle but important. In RPM spec files, the %files section has two ways to specify directories:

%dir directive: Packages the directory itself but not its contents
Trailing slash: Packages everything recursively under that directory
Glob patterns: Packages matching files/directories using wildcards

RPM file categorization is more nuanced than it appears. The spec file doesn't just list what to package—it defines ownership semantics. When you use %dir /usr/share/openscad/, you're saying "I own this directory entry in the filesystem, but I'm not claiming ownership of its contents." This is crucial for shared directories where multiple packages might install files. For example, /usr/share/icons/hicolor/ is a shared directory owned by the hicolor-icon-theme package, but dozens of applications install their icons there. Each app uses patterns like %{_datadir}/icons/hicolor/*/apps/myapp.* to claim only their own icons, not the directory itself.

The trailing slash syntax (/usr/share/openscad/) means "I own this directory AND recursively everything in it." It's RPM's way of saying "this is my territory, all of it." The glob syntax (/usr/share/openscad/*) is similar but more explicit—it expands at build time to include all items matching the pattern.

Here's what I had originally:

%files
%{_bindir}/openscad
%{_defaultdocdir}/%{name}/COPYING
# ... other files ...
%dir %{_datadir}/openscad/   # ← This was the problem!

The %dir directive told RPM: "Package this directory entry, but not the files inside it." This is useful when you want to own the directory structure but let other packages populate it. But in my case, I wanted to package all the files.

The fix was simple but not obvious:

%files
%{_bindir}/openscad
%{_defaultdocdir}/%{name}/COPYING
# ... other files ...
%{_datadir}/openscad/   # Trailing slash = recursive inclusion

By removing %dir and keeping the trailing slash, RPM now understood: "Package this directory and everything under it recursively."

Why This Matters

This distinction exists because RPM has sophisticated ownership semantics. Multiple packages can share directories, and RPM needs to know:

Who owns the directory itself?
Who owns the files inside?
What happens when a package is removed?

Using %dir signals: "I own this directory structure, but other packages might put files in it." Using just the path with a trailing slash signals: "I own this directory AND everything in it."

For a monolithic package like OpenSCAD where we control all the files, the recursive approach is correct. For shared directories like /usr/share/icons/hicolor/, using patterns like %{_datadir}/icons/hicolor/*/apps/openscad.* is more appropriate because other packages also install icons there.

More glob pattern examples for different scenarios:

# Scenario 1: Include all files but not hidden files
%{_datadir}/myapp/*

# Scenario 2: Include specific file types only
%{_datadir}/myapp/*.json
%{_datadir}/myapp/*.xml

# Scenario 3: Include subdirectories at specific depths
%{_datadir}/icons/hicolor/*/apps/myapp.png
%{_datadir}/icons/hicolor/*/mimetypes/myapp-*.png

# Scenario 4: Exclude certain patterns (requires %exclude)
%{_datadir}/myapp/
%exclude %{_datadir}/myapp/test/
%exclude %{_datadir}/myapp/*.debug

# Scenario 5: Shared directories (don't claim ownership)
%{_datadir}/icons/hicolor/48x48/apps/myapp.png
%{_datadir}/icons/hicolor/scalable/apps/myapp.svg
# Note: No %dir for hicolor directories - icon theme package owns them

The key insight here? Be specific about what you own. If you're installing into a shared space, use precise patterns. If you own the entire directory tree, use the trailing slash for simplicity.

Debugging Mini-Story: The RPM %files Rabbit Hole

I'll be honest—I stared at this error for a good 20 minutes before I understood what was happening:

error: Installed (but unpackaged) file(s) found:
   /usr/share/openscad/color-schemes/cornfield/background.json
   (... 184 more files ...)

My first thought? "But I specified %{_datadir}/openscad/! That should include everything!" So I added more explicit patterns. Then I added glob patterns. Nothing worked.

Finally, I did what I should have done first: read the RPM documentation carefully. That's when I discovered the %dir directive doesn't mean "directory and contents"—it means "just the directory entry." I'd been telling RPM: "Hey, this directory exists, but I'm not claiming the files inside it."

The fix was embarrassingly simple: remove %dir. But the lesson stuck with me: RPM's packaging model is about ownership, not just inclusion. Understanding that mental model makes everything else click into place.

Troubleshooting Tip #2: Testing RPM spec files locally before committing:

# Build the RPM locally to catch %files errors early
$ rpmbuild -bb openscad.spec

# If you get "unpackaged files" errors, use this to see what's installed:
$ rpm -qlp /path/to/built.rpm | grep openscad

# Compare against your %files section to find what's missing
$ rpmdev-extract /path/to/built.rpm
$ find usr/share/openscad/ -type f | wc -l  # Should match your expectations

Mistakes to Avoid #2: Don't assume directory patterns work the same across package managers. Debian's *.install files, RPM's %files sections, and Arch's PKGBUILD have completely different semantics for the same concept.

Issue 4: Debian Dependency Hell

The Bookworm-to-Trixie Transition

The Debian packages built successfully, but installation failed on Debian Trixie (testing/unstable):

$ sudo apt install ./openscad_2025.11.19-1_amd64.deb
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages have unmet dependencies:
 openscad : Depends: libmimalloc2.1 but it is not installable
            Depends: libzip4 but it is not installable
E: Unable to correct problems, you have held broken packages.

The problem was clear: I was declaring dependencies for Debian Bookworm (stable), but building and running on Debian Trixie (testing), which has newer library versions.

Understanding Debian Package Versioning

Debian library packages include their SONAME (Shared Object Name) in the package name. This allows multiple versions to coexist:

libmimalloc2.1 = libmimalloc with SONAME 2.1 (Bookworm)
libmimalloc3 = libmimalloc with SONAME 3 (Trixie)
libzip4 = libzip with SONAME 4 (Bookworm)
libzip5 = libzip with SONAME 5 (Trixie)

When a library's API/ABI changes significantly, the SONAME increments, and Debian creates a new package. This prevents incompatible upgrades from breaking existing software.

How to discover which SONAME version your binary actually needs:

The ldd command shows what shared libraries a binary is linked against, including their SONAME:

$ ldd /usr/bin/openscad | grep libmimalloc
        libmimalloc.so.2 => /usr/lib/x86_64-linux-gnu/libmimalloc.so.2.1 (0x00007f8b2c400000)

$ ldd /usr/bin/openscad | grep libzip
        libzip.so.5 => /usr/lib/x86_64-linux-gnu/libzip.so.5.5 (0x00007f8b2c350000)

See that? libmimalloc.so.2 is the SONAME (major version 2), and the actual file is libmimalloc.so.2.1 (minor version 2.1). The Debian package name follows the SONAME.

To find which Debian package provides that library:

$ dpkg -S /usr/lib/x86_64-linux-gnu/libmimalloc.so.2.1
libmimalloc2.1:amd64: /usr/lib/x86_64-linux-gnu/libmimalloc.so.2.1

Or on a system where the binary isn't installed yet, use objdump:

$ objdump -p openscad | grep NEEDED | grep libmimalloc
  NEEDED               libmimalloc.so.2

This shows exactly what SONAME the binary expects. Then you can search Debian packages:

$ apt-cache search libmimalloc
libmimalloc2.1 - Compact general purpose allocator with excellent performance
libmimalloc3 - Compact general purpose allocator with excellent performance

The lesson? Don't guess dependencies—inspect the binary.

The Fix: Update Dependency Declarations

I needed to update the control file (or in my case, the inline control generation in the workflow) to use Trixie's library versions:

-Depends: libmimalloc2.1, libzip4, ...
+Depends: libmimalloc3, libzip5, ...

But there's a more sophisticated approach: dependency alternatives. Since I'm extracting from Docker images that link against specific library versions, I could declare both old and new versions:

Depends: libmimalloc3 | libmimalloc2.1, libzip5 | libzip4

This syntax means: "Prefer libmimalloc3, but accept libmimalloc2.1 if that's what's available." However, since I'm building on Trixie and the binary is linked against Trixie's libraries, this would actually fail - the binary requires the newer versions.

The correct solution depends on your distribution target:

Target Bookworm: Build on Bookworm, declare Bookworm dependencies
Target Trixie: Build on Trixie, declare Trixie dependencies
Target both: Build separate packages for each distribution

In my case, I chose to target Trixie exclusively, so I updated the dependencies to match.

Lesson: Match Build Environment to Target Environment

This highlights a fundamental packaging principle: your dependency declarations must match your build environment. If you build on Debian Trixie, your binary will link against Trixie's libraries, and you must declare Trixie dependencies.

Tools like dpkg-shlibdeps can automatically detect library dependencies by examining the binary, but since I was building packages from pre-compiled Docker images, I had to manage dependencies manually.

Troubleshooting Tip #3: Inspecting binary dependencies when package installation fails:

# Extract the .deb without installing
$ ar x openscad_2025.11.19-1_amd64.deb
$ tar xf data.tar.xz
$ ldd usr/bin/openscad | grep "not found"

# If libraries are missing, find which package provides them on target system
$ ssh target-debian-system "apt-file search libmimalloc.so.2"
$ ssh target-debian-system "dpkg -S /usr/lib/x86_64-linux-gnu/libmimalloc.so.2.1"

# This tells you the exact package name to add to dependencies

Mistakes to Avoid #3: Don't hardcode library versions without checking what's in your build environment. If you're building on Debian Trixie but declaring Debian Bookworm dependencies, your packages won't install anywhere.

Solution Part 3: Infrastructure Glue

Beyond the major concurrency and packaging challenges, several smaller infrastructure issues needed attention to make the system production-ready.

Issue 5: YAML Multi-line Gotchas

The Silent Failure

While testing the changes, I noticed commit messages weren't being formatted correctly. What should have been:

update RPM repository with latest packages

Automated update from workflow run 123456

Was appearing as:

update RPM repository with latest packages Automated update from workflow run 123456

The problem was in the workflow YAML:

# WRONG - GitHub Actions doesn't preserve literal newlines in this syntax
git commit -m "update RPM repository with latest packages

Automated update from workflow run ${{ steps.run.outputs.run_id }}"

Understanding YAML String Handling

YAML has multiple ways to represent strings, and they have different whitespace handling. Here's where it gets tricky—and why my commit messages were getting mangled.

The YAML parser does several things behind the scenes:

Plain style (no quotes): Treats newlines as spaces, collapses consecutive whitespace
Single/double quoted: Escapes must be explicit (\n for newline, \\ for backslash)
Literal block style (|): Preserves newlines and trailing spaces exactly
Folded block style (>): Converts single newlines to spaces, but preserves blank line paragraphs

But here's the gotcha: trailing whitespace in plain-style strings gets silently dropped. So if you have:

message: "Line one
Line two"

The YAML parser sees: "Line one Line two" (single line, single space).

Even worse, GitHub Actions adds its own layer of processing. It evaluates ${{ }} expressions before parsing the YAML, which means you can end up with malformed YAML if the expression output contains quotes or special characters.

For multi-line git commit messages, we need an approach that survives both YAML parsing and GitHub Actions expression evaluation:

# CORRECT - literal block style preserves newlines
git commit -m "update RPM repository with latest packages" \
           -m "Automated update from workflow run ${{ steps.run.outputs.run_id }}"

Alternatively, using multiple -m flags is even clearer, as each -m creates a new paragraph in the commit message:

git commit -m "chore: cleanup old packages" \
           -m "Automated cleanup from workflow run ${{ github.run_id }}" \
           -m "- Removed .deb packages older than 7 days" \
           -m "- Removed .rpm packages older than 7 days" \
           -m "- Updated repository metadata"

This creates a properly formatted commit message with a subject line and body paragraphs.

Troubleshooting Tip #4: Validating YAML in GitHub Actions workflows:

# Install yamllint locally
$ pip install yamllint

# Check your workflow file
$ yamllint .github/workflows/create-release.yml

# Common issues to watch for:
# - Trailing spaces (breaks literal blocks)
# - Inconsistent indentation (breaks structure)
# - Unquoted special characters (: { } [ ] , & * # ? | - < > = ! % @ \)

Commit message comparison (before and after fixing YAML):

Before (broken):

commit 1234567
Author: GitHub Actions
Date:   2025-11-20

    update RPM repository with latest packages Automated update from workflow run 123456

After (fixed):

commit 1234567
Author: GitHub Actions
Date:   2025-11-20

    update RPM repository with latest packages

    Automated update from workflow run 123456

    Changes:
    - Updated repository metadata
    - Added packages for version 2025.11.20

Notice the difference? The first version is a single-line message (hard to read in git logs). The second version has proper paragraphs, making it clear what changed and why.

Mistakes to Avoid #4: Don't rely on whitespace in YAML to create formatting. Use explicit syntax (literal blocks with |, multiple -m flags, or shell features like heredocs) to ensure your intent survives the YAML parser.

Issue 6: Documentation Debt

The README Update

With all the technical issues fixed, I had one final problem: the documentation was outdated. The README still showed old installation instructions and didn't document the new APT/RPM repositories.

I comprehensively updated the README to include:

Architecture support table: Clear mapping between different architecture naming conventions
APT repository instructions: Complete setup including GPG key import
RPM repository instructions: Setup for Fedora/RHEL/Rocky/AlmaLinux
GitHub Releases section: Manual package download instructions
Version format documentation: Explaining the YYYY.MM.DD.BUILD_NUMBER scheme
Repository structure documentation: Showing where packages are stored
Distribution requirements: Minimum Debian/Fedora versions

Here's an example of the new APT installation section:

#### Debian/Ubuntu (APT)

bash

Import GPG key

curl -fsSL https://github.com/gounthar/docker-for-riscv64/releases/download/gpg-key/gpg-public-key.asc | \
sudo gpg --dearmor -o /usr/share/keyrings/openscad-archive-keyring.gpg

Add repository

echo "deb [signed-by=/usr/share/keyrings/openscad-archive-keyring.gpg] https://gounthar.github.io/openscad stable main" | \
sudo tee /etc/apt/sources.list.d/openscad.list

Update and install

sudo apt-get update
sudo apt-get install openscad


**Supported Distributions:**
- Debian Trixie (13) and newer
- Ubuntu 24.04 LTS and newer

This gives users a complete, copy-paste-ready installation experience with clear version requirements.

Testing the Complete System

After all fixes were in place, I tested the complete pipeline:

# Trigger a full build
git push origin multiplatform

# This starts the cascade:
# 1. Docker build workflow (3 architectures in parallel)
# 2. Package extraction workflows (Debian + RPM)
# 3. Repository update workflows (APT + RPM)
# 4. Release creation workflow

# Wait for completion, then test installation

Testing on each architecture:

# AMD64 (x86_64)
curl -fsSL https://github.com/gounthar/docker-for-riscv64/releases/download/gpg-key/gpg-public-key.asc | \
  sudo gpg --dearmor -o /usr/share/keyrings/openscad-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/openscad-archive-keyring.gpg] https://gounthar.github.io/openscad stable main" | \
  sudo tee /etc/apt/sources.list.d/openscad.list
sudo apt-get update
sudo apt-get install openscad -y
openscad --version

# ARM64 (aarch64) - same process
# RISC-V64 - same process

# RPM testing
sudo rpm --import https://gounthar.github.io/openscad/rpm/RPM-GPG-KEY
sudo curl -L https://gounthar.github.io/openscad/rpm/openscad.repo \
  -o /etc/yum.repos.d/openscad.repo
sudo dnf install openscad -y
openscad --version

All installations succeeded, and concurrent workflows no longer conflicted.

Key Lessons Learned

1. Concurrency is Hard, Even in CI/CD

GitHub Actions makes it easy to parallelize work, but it doesn't automatically handle coordination between workflows. When multiple workflows modify shared state (like a git branch), you need explicit concurrency control.

The combination of GitHub's concurrency directive and the reset-and-restore pattern provides robust handling of concurrent updates—without complex locking.

2. RPM Spec Files Have Subtle Semantics

The difference between %dir /path/to/directory and /path/to/directory/ is easy to miss, but it completely changes RPM's packaging behavior. Understanding the ownership model is crucial.

When in doubt, use:

%dir for shared directories you don't populate
Paths with trailing slashes for directories you own completely
Glob patterns for shared directories where you only own some files

3. Match Build Environment to Dependencies

Your package dependencies must match what your binary is actually linked against. Tools like dpkg-shlibdeps (Debian) and rpm's automatic dependency detection can help, but when working with pre-built binaries, manual verification is essential.

4. YAML is Surprisingly Complex

YAML's string handling has many edge cases. For multi-line content in shell commands:

Use multiple -m flags for git commit messages
Use literal block style (|) when you need exact newline preservation
Test your YAML with yamllint to catch issues early

5. Idempotency and Retry Logic are Essential

Network operations fail. APIs have transient issues. Build robust systems by:

Making operations idempotent (can be safely repeated)
Adding retry logic with exponential backoff
Using features like --clobber to replace instead of failing on duplicates
Logging clearly so you can diagnose intermittent issues

6. Documentation is Part of the Product

The best automation is useless if users don't know how to use it. Invest in clear, complete documentation that includes:

Copy-paste-ready commands
Clear version/distribution requirements
Architecture support matrix
Troubleshooting guidance

Frequently Asked Questions

How do I prevent concurrent GitHub Actions workflows from conflicting?

Use GitHub's concurrency directive to control workflow execution:

concurrency:
  group: release-creation
  cancel-in-progress: false

Combined with the reset-and-restore pattern for git operations, this prevents race conditions when multiple workflows modify the same branch.

What is the reset-and-restore pattern in GitHub Actions?

The reset-and-restore pattern handles concurrent git conflicts by:

Saving changes to a temporary directory
Fetching and resetting to the latest remote state
Restoring saved changes on top
Retrying the push operation

This avoids merge conflicts in automated workflows.

How do I fix "unpackaged files found" errors in RPM builds?

Change %dir /path/ to /path/ (with trailing slash) in your RPM spec file's %files section. The %dir directive only packages the directory entry, not its contents.

How do I synchronize multiple GitHub Actions workflows?

Use workflow creation timestamps to detect if workflows were triggered by the same build:

# Get creation times and compare within a sync window (e.g., 5 minutes)
DEB_TIME=$(gh api repos/${{ github.repository }}/actions/runs/${DEB_RUN} --jq '.created_at')
RPM_TIME=$(gh api repos/${{ github.repository }}/actions/runs/${RPM_RUN} --jq '.created_at')

Workflows created within your sync window are from the same build and can be safely combined.

Conclusion: Building Robust Multi-Architecture CI/CD Pipelines

Building a fully automated multi-architecture package distribution system for CI/CD pipelines is complex. It requires understanding:

Git coordination patterns for concurrent modifications
Package manager semantics (RPM, Debian) at a deep level
Dependency management across different distribution versions
GitHub Actions workflows and their concurrency model
Retry and idempotency patterns for robust automation

The reset-and-restore pattern proved particularly valuable. Instead of trying to merge concurrent changes (complex and error-prone), we save our changes, reset to the latest state, and reapply our changes on top. This works because our changes are additive and don't conflict with each other - APT updates modify dists/, RPM updates modify rpm/, and both are independent.

The key insight is that concurrent workflows are inevitable in modern CI/CD. Rather than fighting them with complex locking, design your system to handle conflicts gracefully through idempotent operations and smart retry logic.

Now the OpenSCAD build infrastructure runs smoothly: three architectures, two package formats, automatic repository updates, and GitHub Releases - all working in harmony. The next time someone pushes a commit, packages are built, tested, and published across all architectures within an hour, with zero manual intervention.

And that's the dream of automation: complexity managed, reliability achieved, and maintainers free to focus on actual development rather than packaging mechanics.

Beyond Package Management: Applying These Patterns to Other CI/CD Scenarios

The concurrency patterns and retry logic described here apply to many DevOps automation challenges:

Artifact management: Concurrent uploads to package registries
Container registry updates: Pushing multi-platform Docker images
Infrastructure as Code: Terraform state file conflicts
Continuous deployment: Coordinating deployments across environments

The reset-and-restore pattern is particularly valuable for any scenario involving concurrent modifications to shared state in continuous integration pipelines.