Deploy Elixir apps utilizing Rust NIFs on Render.com

Frank Kumro — Wed, 19 Feb 2020 23:11:04 +0000

After a long week of dealing with servers at my day job I had no urge to do the same with my side projects. I looked at Render.com and decided to try them out. All was well until I decided to add a mix package that built a Rust NIF. The default build container used by Render for Elixir projects didn't include Rust but after chatting with their team on Slack I came up with a solution, Docker.

The first step is to determine what containers would have the functionality I needed. Then they would be combined using a multi-stage Docker build resulting in a container that could build Elixir projects with Rust NIFs. This was as easy as copying and pasting from the official Elixir and Rust Dockerfiles with very minor modifications.

TLDR; linking to the complete Dockerfile at bottom of article.

Erlang

The starting point for all the containers would be Erlang, 22-slim to be precise. It's important to determine what base container the Erlang 22-slim container is using as it will need to be the same for each additional container.

Erlang uses debian:buster per the Erlang Dockerfile. Thus all our other containers must also be derived from debian:buster.

The first line of our Dockerfile will be specifying the container to use for the first stage, building Rust. Using the AS instruction allows future steps to utilize the container built in this step by name.

FROM erlang:22-slim AS rust_builder

Rust

The contents of the official Rust Dockerfile can be copied into the rust_builder step as long as the Rust container uses buster.

Our Dockerfile for Rust is shown below:

FROM erlang:22-slim as rust_builder

ENV RUSTUP_HOME=/usr/local/rustup \
    CARGO_HOME=/usr/local/cargo \
    PATH=/usr/local/cargo/bin:$PATH \
    RUST_VERSION=1.41.0

RUN set -eux; \
    apt-get update; \
    apt-get install -y --no-install-recommends \
        ca-certificates \
        gcc \
        libc6-dev \
        wget \
        ; \
    dpkgArch="$(dpkg --print-architecture)"; \
    case "${dpkgArch##*-}" in \
        amd64) rustArch='x86_64-unknown-linux-gnu'; rustupSha256='ad1f8b5199b3b9e231472ed7aa08d2e5d1d539198a15c5b1e53c746aad81d27b' ;; \
        armhf) rustArch='armv7-unknown-linux-gnueabihf'; rustupSha256='6c6c3789dabf12171c7f500e06d21d8004b5318a5083df8b0b02c0e5ef1d017b' ;; \
        arm64) rustArch='aarch64-unknown-linux-gnu'; rustupSha256='26942c80234bac34b3c1352abbd9187d3e23b43dae3cf56a9f9c1ea8ee53076d' ;; \
        i386) rustArch='i686-unknown-linux-gnu'; rustupSha256='27ae12bc294a34e566579deba3e066245d09b8871dc021ef45fc715dced05297' ;; \
        *) echo >&2 "unsupported architecture: ${dpkgArch}"; exit 1 ;; \
    esac; \
    url="https://static.rust-lang.org/rustup/archive/1.21.1/${rustArch}/rustup-init"; \
    wget "$url"; \
    echo "${rustupSha256} *rustup-init" | sha256sum -c -; \
    chmod +x rustup-init; \
    ./rustup-init -y --no-modify-path --profile minimal --default-toolchain $RUST_VERSION; \
    rm rustup-init; \
    chmod -R a+w $RUSTUP_HOME $CARGO_HOME; \
    rustup --version; \
    cargo --version; \
    rustc --version; \
    update-ca-certificates;

The lines at the bottom of the default Rust Dockerfile that removes the apt cache and uninstalls packages were removed. This is necessary as future build steps will utilize the cache and packages and if we removed them it would increase the build time when the containers are not cached.

Updating the CA certificates is also performed during this stage otherwise HTTPS calls may fail. I've experienced such failures when updating cargo and not having the CA certs up to date.

Elixir

Following the same method as the Rust container, Elixir can build upon the rust_builder stage using the official Elixir Dockerfile.

To utilize a previously built stage it is referenced using the FROM instruction. The example below will use the rust_builder stage and the resulting changes will be named elixir_builder.

FROM rust_builder AS elixir_builder

The contents below are copied from the Official Elixir Dockerfile again with slight modifications. Similar to the Rust stage, the commands to remove the apt cache and packages are removed. It is also in this stage that NodeJS 10 is installed from nodesource.

FROM rust_builder as elixir_builder

# elixir expects utf8.
ENV ELIXIR_VERSION="v1.10.1" \
    LANG=C.UTF-8

RUN set -xe \
    && ELIXIR_DOWNLOAD_URL="https://github.com/elixir-lang/elixir/archive/${ELIXIR_VERSION}.tar.gz" \
    && ELIXIR_DOWNLOAD_SHA256="bf10dc5cb084382384d69cc26b4f670a3eb0a97a6491182f4dcf540457f06c07" \
    && buildDeps=' \
        curl \
        make \
    ' \
    && apt-get update \
    && apt-get install -y --no-install-recommends $buildDeps \
    && curl -fSL -o elixir-src.tar.gz $ELIXIR_DOWNLOAD_URL \
    && echo "$ELIXIR_DOWNLOAD_SHA256 elixir-src.tar.gz" | sha256sum -c - \
    && mkdir -p /usr/local/src/elixir \
    && tar -xzC /usr/local/src/elixir --strip-components=1 -f elixir-src.tar.gz \
    && rm elixir-src.tar.gz \
    && cd /usr/local/src/elixir \
    && make install clean \
  && apt-get install -y git apt-transport-https ca-certificates \
  && update-ca-certificates \
  && curl -sL https://deb.nodesource.com/setup_10.x | bash - \
  && apt-get install -y nodejs;

CMD ["iex"]

Application configuration

Multiple environment variables are used for the application container setup. Using the combination of ARG and ENV allows the access of the variables during build and execution. These are set via the Render dashboard.

Build variables

Environment variables required for building the container are:

APP_NAME - The name of your application, used when starting. Utilized in the final step of the Dockerfile.

PORT - Port that will serve the application

Run time variables

The application must be configured to utilize the variables available during run time. These variables are:

DATABASE_URL - The URL which allows the application to connect to the database.

POOL_SIZE - The number of connections to the database the application will establish.

SECRET_KEY_BASE - Base string used for encryption/decryption of secrets

PORT - Port that will serve the application

These can bet set in the config/releases.exs file using System.get_env/1.

database_url =
  System.get_env("DATABASE_URL") ||
    raise """
    environment variable DATABASE_URL is missing.
    For example: ecto://USER:PASS@HOST/DATABASE
    """

config :my_app, MyApp.Repo,
  url: database_url,
  pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10"),
  ssl: true

secret_key_base =
  System.get_env("SECRET_KEY_BASE") ||
    raise """
    environment variable SECRET_KEY_BASE is missing.
    You can generate one by calling: mix phx.gen.secret
    """

config :my_app, MyAppWeb.Endpoint,
  http: [:inet6, port: String.to_integer(System.get_env("PORT") || "4000")],
  secret_key_base: secret_key_base,
  server: true

The host must also be set in config/prod.exs.

config :my_app, MyAppWeb.Endpoint,
  url: [host: System.get_env("RENDER_EXTERNAL_HOSTNAME") || "example.com", port: 80],
  cache_static_manifest: "priv/static/cache_manifest.json"

Building a release

The final build step is using mix release to build and package the Elixir application. The current working directory is used to copy the source code into the containers /app directory and then the release is built. This step utilizes the results of the elixir_builder step and named it app_builder.

FROM elixir_builder as app_builder

RUN mkdir /app
WORKDIR /app

COPY . .

ENV MIX_ENV=prod

RUN set -xe \
  && mix local.rebar --force \
  && mix local.hex --force \
  && mix do deps.get, compile \
  && npm install --prefix ./assets \
  && npm run deploy --prefix ./assets \
  && mix do phx.digest, release --overwrite

Building the app run time container

Now that the application release is built we can throw away our previous steps to ensure we have the smallest container possible. Locales and environment variables are setup along with the apt cache being removed to keep the final container size down.

Privileged access

The application is ran as the nobody user which does not have permissions to bind to privileged ports. Although Render does allow this type of binding I avoided it as it is unnecessary and I prefer to run applications with as few permissions as possible.

FROM debian:buster-slim

ARG APP_NAME
ARG PORT
ARG SECRET_KEY_BASE
ARG DATABASE_URL
ARG POOL_SIZE

RUN mkdir /app
WORKDIR /app

ENV LANG=en_US.UTF-8 \
  LANGUAGE=en_US.UTF-8 \
  LC_ALL=en_US.UTF-8 \
  MIX_ENV=prod \
  SHELL=/bin/bash \
  APP_NAME=$APP_NAME \
  PORT=$PORT \
  HOME=/app \
  SECRET_KEY_BASE=$SECRET_KEY_BASE \
  DATABASE_URL=$DATABASE_URL \
  POOL_SIZE=$POOL_SIZE

# Setup locales to prevent VM from starting with latin1
# Install application runtime deps
RUN set -xe \
  && apt-get update \
  && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends openssl ca-certificates locales \
  && sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen \
  && dpkg-reconfigure --frontend=noninteractive locales \
  && update-locale LANG=en_US.UTF-8 \
  && rm -rf /var/lib/apt/lists/*

COPY --from=app_builder /app/_build/prod/rel/${APP_NAME} .

EXPOSE ${PORT}

RUN chown -R nobody: /app
USER nobody

CMD bin/${APP_NAME} start

Testing locally

Before setting up your Render application it's best to ensure the container is built properly. The two build variables detailed earlier will need to be passed to docker using the --build-env argument. For this example my project is named hades.

docker build --build-arg PORT=10000 --build-arg APP_NAME=hades . -t hades

To run the container I found specifying the environment variables in a env.list file to work best. The contents of this file are in the format of KEY=value.

SECRET_KEY=jBZE5O2W0EsMQ7dCQBO7ZbOchN9ORFG82k1LVlRF/9qjs9iqQZGg9LE59n4y5tTV
POOL_SIZE=10
DATABASE_URL=postgres://test_user:test_password@postgres.render.com/test_database

The container is then started with docker run specifying the env.list file:

docker run --env-file=env.list hades

A full build and run locally was captured using asciinema.

Configuring Render

When a new web service is created on Render the Environment must be set to Docker. If your Dockerfile is in the root of your repository you can continue on to setting the environment variables.

Render supports storing your Dockerfile anywhere in your repository. This is set using the Dockerfile Path under the Advanced section of the setup. The Docker Build Context Directory must also be set if the root of your repository shouldn't be used when building, this article assumes the root directory is used for both settings.

Setting environment variables

The environment variables described within this article must be manually set per application that is hosted on Render. You are free to pick any values for SECRET_KEY_BASE / PORT however the remaining variables should be set depending on your database / application.

POOL_SIZE should be set according to the connection limit governed by the database tier.

DATABASE_URL is set using the Internal Connection String from the database page.

APP_NAME should match your application name.

RENDER_EXTERNAL_HOSTNAME is only used if you are using the Render subdomain. If you are not then I would hard code the value in the config file.

Those are the steps necessary to build and deploy an Elixir project with Rust NIFs on Render.com. Anytime you push changes to your configured branch a new container will be built and deployed.

TLDR

The complete Dockerfile is available on Gitlab: https://gitlab.com/fkumro/elixir-rust-render

Notes:

The build time on Render will most likely be much slower than your local machine. The builds on the $7 plan took around 17 minutes and docker containers don't seem to be cached. Hopefully Render improves their build machines in the future.
Migrations are not covered in this article.
The application would benefit from a health endpoint that Render can use to determine if the container should be killed and a new one started.

Forem: Frank Kumro