Howto: WASM runtimes in Docker / Colima

Bart Guijt — Fri, 12 Jan 2024 18:37:15 +0000

TL;DR

I could not find any guide how to add WASM container capability to Docker running on Colima. This guide provides a few Colima templates for exactly this, which adds WasmEdge, Wasmtime and Wasmer runtime types.

Context

Instead of installing Docker Desktop and enabling the WASM features I decided to see how I could add WASM runtime capabilities to a Colima setup. Colima is a great alternative to Docker Desktop if the licensing terms prevent you from using it.

Forgive me for not explaining everything from scratch, I just finished the last (and best) template and I needed to write it down so I could share it.

Three alternative Colima templates

I created these three Colima templates:

colima-crun-wasmedge: Use the 'crun' runtime to launch WASM containers in WasmEdge;
colima-runwasi-git: Use the Runwasi WASM shims in containerd to launch WASM containers in WasmEdge, Wasmtime or Wasmer, where the Runwasi shims are compiled from git master;
colima-runwasi: Same as colima-runwasi-git, but uses the latest releases of the Runwasi shims instead of building them.

`colima-crun-wasmedge`

This was my first trial of getting WASM containers to work. To enable running WASM loads, crun needs to be compiled with WasmEdge (or Wasmtime) support. I could only get WasmEdge working, but the basic idea is the same.

Here is the template (edit the Colima template with $ colima template). I added comments to the parts relevant for the WASM config, other comments are removed for brevity:

Template

cpu: 4
disk: 60
memory: 12
arch: host
hostname: colima
autoActivate: true
forwardAgent: false

# I only tested this with 'docker', not 'containerd':
runtime: docker

kubernetes:
  enabled: false
  version: v1.24.3+k3s1
  k3sArgs: []

network:
  address: true
  dns: []
  dnsHosts:
    host.docker.internal: host.lima.internal

# Added:
# - containerd-snapshotter: true (meaning containerd will be used for pulling images)
# - default-runtime / runtimes: crun (instead of the default 'runc')
docker:
  default-runtime: crun
  features:
    buildkit: true
    containerd-snapshotter: true
  runtimes:
    crun:
      path: /usr/local/bin/crun

vmType: vz
rosetta: true
mountType: virtiofs
mountInotify: false
cpuType: host

# This provisioning script installs WasmEdge and builds crun with wasmedge support:
provision:
  - mode: system
    script: |
      [ -f /etc/docker/daemon.json ] && echo "Already provisioned!" && exit 0
      echo "Install system updates:"
      apt-get update -y
      apt-get upgrade -y
      echo "Install WasmEdge and crun dependencies:"
      # NOTE: packages curl git python3 already installed:
      apt-get install -y make gcc build-essential pkgconf libtool libsystemd-dev libprotobuf-c-dev libcap-dev libseccomp-dev libyajl-dev libgcrypt20-dev go-md2man autoconf automake criu xz-utils
      apt-get clean -y
  - mode: user
    script: |
      [ -f /etc/docker/daemon.json ] && echo "Already provisioned!" && exit 0
      echo "Installing WasmEdge:"
      curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | sudo bash -s -- -p /usr/local
      echo
      echo "`wasmedge -v` installed!"
      # NOTE: I failed to configure Wasmtime properly - turned off for now:
      #echo "Installing Wasmtime:"
      #curl -sSf https://wasmtime.dev/install.sh | bash
      #sudo cp .wasmtime/bin/* /usr/local/bin/
      #rm -rf .wasmtime
      #echo "`wasmtime -V` installed!"
      echo "Install crun:"
      git clone https://github.com/containers/crun
      cd crun
      ./autogen.sh
      #./configure --with-wasmedge --with-wasmtime
      ./configure --with-wasmedge
      make
      sudo make install
      crun -v
      echo "crun installed! Replacing runc with crun:"
      # NOTE: replacing runc with crun is to simplify containerd config
      TRC=`which runc`
      sudo rm -rf $TRC
      sudo cp `which crun` $TRC
      echo "Configuring containerd:"
      sudo mkdir -p /etc/containerd/
      containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
      echo "Restarting/reloading docker/containerd services:"
      sudo systemctl daemon-reload
      sudo systemctl restart containerd
      # As soon as Colima writes its /etc/docker/daemon.json file (right after this provisioning script),
      # it will also start the Docker daemon. If we stop Docker here, the changes will actually take effect:
      sudo systemctl stop docker

sshConfig: true
mounts: []
env: {}

Caveats

First colima start time is long

It takes 1.5 minutes on my M1 max mac to start the container.

Extra docker run parameters

After starting Colima with this template, be aware that you need to supply some specific parameters (--platform and --annotation) to the docker run command, e.g.:

$ docker run --rm -dp 8080:8080 \
         --platform wasi/wasm32 \
         --annotation "run.oci.handler=wasm" \
         michaelirwin244/wasm-example:latest

or:

$ docker run --rm -dp 8080:8080 \
         --platform wasi/wasm32 \
         --annotation "module.wasm.image/variant=compat-smart" \
         michaelirwin244/wasm-example:latest

Stopping container waits for 10 seconds

For some reason the SIGTERM signal is not honoured in the WASM runtime. Stop the container with a specified timeout:

$ docker stop -t 0 <container hash/name>

`colima-runwasi-git`

My second attempt was to get the Runwasi shims working with Docker, since I did not like the necessary --annotation parameter which are not needed for the 'official' Docker Desktop configuration. This template remedies that drawback.

Here is the template (edit the Colima template with $ colima template). I added comments to the parts relevant for the WASM config, other comments are removed for brevity:

Template

cpu: 4
disk: 60
memory: 12
arch: host
hostname: colima
autoActivate: true
forwardAgent: false

# I only tested this with 'docker', not 'containerd':
runtime: docker

kubernetes:
  enabled: false
  version: v1.24.3+k3s1
  k3sArgs: []

network:
  address: true
  dns: []
  dnsHosts:
    host.docker.internal: host.lima.internal

# Added:
# - containerd-snapshotter: true (meaning containerd will be used for pulling images)
docker:
  features:
    buildkit: true
    containerd-snapshotter: true

vmType: vz
rosetta: true
mountType: virtiofs
mountInotify: false
cpuType: host

# This provisioning script installs build dependencies, WasmEdge and builds the WASM runtime shims for containerd.
# NOTE: this takes a LOOONG time!
provision:
  - mode: system
    script: |
      [ -f /etc/docker/daemon.json ] && echo "Already provisioned!" && exit 0
      echo "Installing system updates:"
      apt-get update -y
      apt-get upgrade -y
      echo "Installing WasmEdge and runwasi build dependencies:"
      # NOTE: packages curl, git and python3 already installed:
      apt-get install -y make gcc build-essential pkgconf libtool libsystemd-dev libprotobuf-c-dev libcap-dev libseccomp-dev libyajl-dev libgcrypt20-dev go-md2man autoconf automake criu pkg-config libdbus-glib-1-dev libelf-dev libclang-dev libzstd-dev protobuf-compiler xz-utils
      apt-get clean -y
  - mode: user
    script: |
      [ -f /etc/docker/daemon.json ] && echo "Already provisioned!" && exit 0
      #
      # Setting vars for this script:
      #
      # Which WASM runtimes to install (wasmedge, wasmtime and wasmer are supported):
      WASM_RUNTIMES="wasmedge wasmtime wasmer"
      #
      # Location of the containerd config file:
      CONTAINERD_CONFIG="/etc/containerd/config.toml"
      #
      # Target location for the WASM runtimes and containerd shims ($TARGET/bin and $TARGET/lib):
      TARGET="/usr/local"
      #
      # Install rustup:
      #
      echo "Installing rustup for building runwasi:"
      curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain none -y
      source "$HOME/.cargo/env"
      #
      # Install selected WASM runtimes and containerd shims:
      #
      [[ -z "${WASM_RUNTIMES// /}" ]] && echo "No WASM runtimes selected - exiting!" && exit 0
      git clone https://github.com/containerd/runwasi
      echo "Installing WASM runtimes and building containerd shims: ${WASM_RUNTIMES}:"
      sudo mkdir -p /etc/containerd/
      containerd config default | sudo tee $CONTAINERD_CONFIG >/dev/null
      for runtimeName in $WASM_RUNTIMES; do
        case $runtimeName in
          wasmedge)
            echo "Installing WasmEdge:"
            curl -sSfL https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | sudo bash -s -- -p $TARGET
            echo
            echo "`wasmedge -v` installed!"
            ;;
          wasmtime)
            echo "Installing wasmtime:"
            curl -sSfL https://wasmtime.dev/install.sh | bash
            sudo cp .wasmtime/bin/* ${TARGET}/bin/
            rm -rf .wasmtime
            echo "`wasmtime -V` installed!"
            ;;
          wasmer)
            echo "Installing wasmer:"
            curl -sSfL https://get.wasmer.io | sh
            sudo cp .wasmer/bin/* ${TARGET}/bin/
            sudo cp .wasmer/lib/* ${TARGET}/lib/
            rm -rf .wasmer
            echo "`wasmer -V` installed!"
            ;;
          *)
            echo "ERROR: WASM runtime $runtimeName is not supported!"
            exit 1
            ;;
        esac
        cd runwasi
        echo "Building containerd-shim-${runtimeName}:"
        cargo build -p containerd-shim-${runtimeName} --release
        echo "Installing containerd-shim-${runtimeName}-v1:"
        sudo install ./target/release/containerd-shim-${runtimeName}-v1 ${TARGET}/bin
        sudo ln -sf ${TARGET}/bin/containerd-shim-${runtimeName}-v1 ${TARGET}/bin/containerd-shim-${runtimeName}d-v1
        sudo ln -sf ${TARGET}/bin/containerd-shim-${runtimeName}-v1 ${TARGET}/bin/containerd-${runtimeName}d
        echo "containerd-shim-${runtimeName} installed."
        cd ..
        echo "[plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.${runtimeName}]" | sudo tee -a $CONTAINERD_CONFIG >/dev/null
        echo "  runtime_type = \"io.containerd.${runtimeName}.v1\"" | sudo tee -a $CONTAINERD_CONFIG >/dev/null
      done
      echo "containerd WASM runtimes and shims installed."
      #
      # Restart the systemctl services to pick up the installed shims.
      # NOTE: We need to 'stop' docker because at this point the actual daemon.json config is not yet provisioned:
      #
      echo "Restarting/reloading docker/containerd services:"
      sudo systemctl daemon-reload
      sudo systemctl restart containerd
      sudo systemctl stop docker

sshConfig: true
mounts: []
env: {}

Caveats

First Colima start time is super long

It takes a couple of minutes on my M1 max mac to start the container. Have some patience!

Stopping container waits for 10 seconds

For some reason the SIGTERM signal is not honoured in the WASM runtime. Stop the container with a specified timeout:

$ docker stop -t 0 <container hash/name>

`colima-runwasi`

This template is the same as the previous template, except it doesn't build the containerd shims, it downloads the latest releases. Starting Colima with this template for the first time takes less than 50 seconds on my M1 max machine.

Here is the template (edit the Colima template with $ colima template). I added comments to the parts relevant for the WASM config, other comments are removed for brevity:

Template

cpu: 4
disk: 60
memory: 12
arch: host
hostname: colima
autoActivate: true
forwardAgent: false

# I only tested this with 'docker', not 'containerd':
runtime: docker

kubernetes:
  enabled: false
  version: v1.24.3+k3s1
  k3sArgs: []

network:
  address: true
  dns: []
  dnsHosts:
    host.docker.internal: host.lima.internal

# Added:
# - containerd-snapshotter: true (meaning containerd will be used for pulling images)
docker:
  features:
    buildkit: true
    containerd-snapshotter: true

vmType: vz
rosetta: true
mountType: virtiofs
mountInotify: false
cpuType: host

# Custom provision scripts for the virtual machine.
provision:
  - mode: system
    script: |
      [ -f /etc/docker/daemon.json ] && echo "Already provisioned!" && exit 0
      echo "Installing system updates:"
      apt-get update -y
      apt-get upgrade -y
      echo "Installing dependency for wasmtime installer:"
      apt-get install -y xz-utils
      apt-get clean -y
  - mode: user
    script: |
      [ -f /etc/docker/daemon.json ] && echo "Already provisioned!" && exit 0
      #
      # Setting vars for this script:
      #
      # Which WASM runtimes to install (wasmedge, wasmtime and wasmer are supported):
      WASM_RUNTIMES="wasmedge wasmtime wasmer"
      #
      # Location of the containerd config file:
      CONTAINERD_CONFIG="/etc/containerd/config.toml"
      #
      # Target location for the WASM runtimes and containerd shims ($TARGET/bin and $TARGET/lib):
      TARGET="/usr/local"
      #
      # Install selected WASM runtimes and containerd shims:
      #
      [[ -z "${WASM_RUNTIMES// /}" ]] && echo "No WASM runtimes selected - exiting!" && exit 0
      echo "Installing WASM runtimes and containerd shims: ${WASM_RUNTIMES}:"
      sudo mkdir -p /etc/containerd/
      containerd config default | sudo tee $CONTAINERD_CONFIG >/dev/null
      for runtimeName in $WASM_RUNTIMES; do
        case $runtimeName in
          wasmedge)
            echo "Installing WasmEdge:"
            curl -sSfL https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | sudo bash -s -- -p $TARGET
            echo
            echo "`wasmedge -v` installed!"
            ;;
          wasmtime)
            echo "Installing wasmtime:"
            curl -sSfL https://wasmtime.dev/install.sh | bash
            sudo cp .wasmtime/bin/* ${TARGET}/bin/
            rm -rf .wasmtime
            echo "`wasmtime -V` installed!"
            ;;
          wasmer)
            echo "Installing wasmer:"
            curl -sSfL https://get.wasmer.io | sh
            sudo cp .wasmer/bin/* ${TARGET}/bin/
            sudo cp .wasmer/lib/* ${TARGET}/lib/
            rm -rf .wasmer
            echo "`wasmer -V` installed!"
            ;;
          *)
            echo "ERROR: WASM runtime $runtimeName is not supported!"
            exit 1
            ;;
        esac
        shimVersion=$(curl -s https://api.github.com/repos/containerd/runwasi/tags | grep tarball | grep "shim-${runtimeName}" | grep -Eo 'https://[^\"]*' | head -1 | tr "/" "\n" | tail -n 1)
        shimUrl="https://github.com/containerd/runwasi/releases/download/containerd-shim-${runtimeName}/${shimVersion}/containerd-shim-${runtimeName}-`uname -m`.tar.gz"
        echo "Installing runwasi shim version $shimVersion for $runtimeName runtime from ${shimUrl}:"
        curl -sSfL $shimUrl | sudo tar xvz -C ${TARGET}/bin/
        echo "[plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.${runtimeName}]" | sudo tee -a $CONTAINERD_CONFIG >/dev/null
        echo "  runtime_type = \"io.containerd.${runtimeName}.v1\"" | sudo tee -a $CONTAINERD_CONFIG >/dev/null
      done
      echo "containerd WASM runtimes and shims installed."
      #
      # Restart the systemctl services to pick up the installed shims.
      # NOTE: We need to 'stop' docker because at this point the actual daemon.json config is not yet provisioned:
      #
      echo "Restarting/reloading docker/containerd services:"
      sudo systemctl daemon-reload
      sudo systemctl restart containerd
      sudo systemctl stop docker

sshConfig: true
mounts: []
env: {}

Caveats

Stopping container waits for 10 seconds

For some reason the SIGTERM signal is not honoured in the WASM runtime. Stop the container with a specified timeout:

$ docker stop -t 0 <container hash/name>

Putting it all together

Let's go from Colima installation to running a WASM program in Docker.

Prepare Colima

First, if you haven't done this already, install Colima:

$ brew install colima

Next, change the default Colima template:

$ colima template

This command opens (by default) a vim editor with the default colima yaml template. In this editor, replace all contents with the template from colima-runwasi above.

In vim: Type the following key sequence to empty the template:
:%d<Enter>
The screen must be empty now.

Next, put vim in -- INSERT -- mode by typing the i key.
Then, paste the contents of the colima-runwasi template into the editor.

Finally, save the template by typing the following key sequence:
<ESC>:wq!<Enter>

The Colima template is now configured to install WASM runtimes. Start Colima:

$ colima start -v

Depending on the speed of your machine, this takes approximately between 50 and 90 seconds.

Install additional Docker plugins

$ brew install docker-buildx docker-compose

Make sure to follow instructions for the installed Docker plugins:

$ mkdir -p ~/.docker/cli-plugins
$ ln -sfn $(brew --prefix)/opt/docker-buildx/bin/docker-buildx ~/.docker/cli-plugins/docker-buildx
$ ln -sfn $(brew --prefix)/opt/docker-compose/bin/docker-compose ~/.docker/cli-plugins/docker-compose

Build a WASM image

Check out a project with rust examples:

$ git clone https://github.com/second-state/rust-examples.git

Cloning into 'rust-examples'...
remote: Enumerating objects: 338, done.
remote: Counting objects: 100% (73/73), done.
remote: Compressing objects: 100% (51/51), done.
remote: Total 338 (delta 42), reused 40 (delta 20), pack-reused 265
Receiving objects: 100% (338/338), 54.27 KiB | 9.04 MiB/s, done.
Resolving deltas: 100% (214/214), done.

Next, enter the rust-examples/hello directory and build the wasm image:

$ cd rust-examples/hello
$ docker buildx build --provenance=false \
         --platform wasi/wasm32 \
         -t secondstate/rust-example-hello .

[+] Building 3.6s (15/15) FINISHED
 => [internal] load .dockerignore
 => => transferring context: 2B
 => [internal] load build definition from Dockerfile
 => => transferring dockerfile: 563B
 => resolve image config for docker.io/docker/dockerfile:1
 => [auth] docker/dockerfile:pull token for registry-1.docker.io
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021
 => => resolve docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021
 => [internal] load metadata for docker.io/library/rust:1.64
 => [auth] library/rust:pull token for registry-1.docker.io
 => [buildbase 1/3] FROM docker.io/library/rust:1.64@sha256:53ded1c919ea0dc23be959e28037238d8c321cd88fbac04d818f210031fccc3b
 => => resolve docker.io/library/rust:1.64@sha256:53ded1c919ea0dc23be959e28037238d8c321cd88fbac04d818f210031fccc3b
 => [internal] load build context
 => => transferring context: 426B
 => CACHED [buildbase 2/3] WORKDIR /src
 => CACHED [buildbase 3/3] RUN <<EOT bash
 => [build 1/3] COPY Cargo.toml .
 => [build 2/3] COPY src ./src
 => [build 3/3] RUN cargo build --target wasm32-wasi --release
 => exporting to image
 => => exporting layers
 => => exporting manifest sha256:506f9c13793e6fd5ce556e36130a7e6977b68720379a393b1a0d2acc23aef501
 => => exporting config sha256:1274e89bbbc5c385dae15ac1b768cf2acaae290172018a03efd2dae1d2f0e872
 => => naming to docker.io/secondstate/rust-example-hello:latest
 => => unpacking to docker.io/secondstate/rust-example-hello:latest

Check the image's existence by running the following command:

$ docker images

REPOSITORY                       TAG       IMAGE ID       CREATED         SIZE
secondstate/rust-example-hello   latest    506f9c13793e   3 minutes ago   2.53MB

Or, check its architecture to verify it is a WASM image:

$ docker inspect \
         --format='{{.Os}}/{{.Architecture}}' \
         secondstate/rust-example-hello

wasi/wasm32

Run the WASM image

Run the WASM image with the following command:

$ docker run --rm \
         --platform wasi/wasm32 \
         --runtime io.containerd.wasmedge.v1 \
         secondstate/rust-example-hello:latest

Hello WasmEdge!

NOTE: The specified runtime above is io.containerd.wasmedge.v1. You have three options here:

WasmEdge: io.containerd.wasmedge.v1

Wasmtime: io.containerd.wasmtime.v1

Wasmer: io.containerd.wasmer.v1

Conclusion

If you are serious about using WASM containers in Docker / Colima, I suggest to use the last template colima-runwasi. The other templates are merely evolutionary exercises in getting to know the Docker parts better.

Let me know how this works for you!

Resources

The information in this Howto is assembled from these links:

EDITS

14 FEB 2024 - added xz-utils to list of install dependencies for wasmtime

Forem: Bart Guijt

Howto: WASM runtimes in Docker / Colima

TL;DR

Context

Three alternative Colima templates

colima-crun-wasmedge

Template

Caveats

First colima start time is long

Extra docker run parameters

Stopping container waits for 10 seconds

colima-runwasi-git

Template

Caveats

First Colima start time is super long

Stopping container waits for 10 seconds

colima-runwasi

Template

Caveats

Stopping container waits for 10 seconds

Putting it all together

Prepare Colima

Install additional Docker plugins

Build a WASM image

Run the WASM image

Conclusion

Resources

EDITS

`colima-crun-wasmedge`

`colima-runwasi-git`

`colima-runwasi`