Forem: Wojtek Siudzinski

Quick and dirty auto-deployment to Raspberry Pi

Wojtek Siudzinski — Sat, 17 Feb 2024 16:07:57 +0000

Some projects require a quick way to deploy code to a Raspberry Pi (or other small device running Linux behing NAT) and it's not critical enough to need things like Kubernetes, Ansible or Docker. It was exactly the case for a small RGB LED matrix we set up at our office to show some KPIs. It was running an OG Raspberry Pi A and we just want to be able to update the code remotly.

The setup

To make the whole thing a bit easier we decided on a Git based setup. The Pi will keep checking a remote repo for changes and if there are any, it will pull them and restart the systemd service driving the display.

Git setup

The Pi needs to be able to access the repository therefore we needed to add the public SSH key of the Pi as a Deploy key:

Generate a new key on the Pi (only if it doesn't have one yet) with:

$ ssh-keygen -t ed25519
$ cat ~/.ssh/id_ed25519.pub

Copy the resulting key and paste it in Settings -> Deploy keys -> Add deploy key in your GitHub repo
Clone the repo on the Pi via ssh

Main application service

First you need to define a systemd service for your main application:

[Unit]
Description=rgb-matrix

[Service]
Type=simple
Restart=always
RestartSec=5
ExecStart=/home/pi/rgb-matrix/run.sh
WorkingDirectory=/home/pi/rgb-matrix
StandardOutput=append:/home/pi/rgb-matrix/stdout.log
StandardError=append:/home/pi/rgb-matrix/stderr.log
User=root
Group=root

[Install]
WantedBy=multi-user.target

In our case the application has to run as a root but in your case there might be a better user for that. Once created, you need to enable it:

$ sudo ln -s ${PWD}/rgb-matrix.service /etc/systemd/system/rgb-matrix.service
$ sudo systemctl daemon-reload
$ sudo systemctl enable rgb-matrix.service
$ sudo systemctl start rgb-matrix.service

Auto-deployment service

Now we need a small script that will be periodically checking the remote repo:

#!/bin/bash

# Configuration
SERVICE_NAME="rgb-matrix" # The name of the systemd service you want to restart

# Function to check for updates and pull them
check_and_pull() {
    git fetch origin

    # Check if there are updates available
    if ! git diff --quiet origin/main; then
        echo "Changes detected, pulling updates..."
        git pull
        echo "Updates pulled successfully."

        # Restart the systemd service
        echo "Restarting the service: $SERVICE_NAME..."
        systemctl restart "$SERVICE_NAME"
        echo "Service restarted successfully."
    fi
}

# Main loop to periodically check for updates
while true; do
    check_and_pull
    # Wait for a specified interval before checking again
    sleep 60 # Checks every 60 seconds, adjust as needed
done

Similarily to the main application, we need to create a systemd service for the watcher:

[Unit]
Description=watcher

[Service]
Type=simple
Restart=always
RestartSec=5
ExecStart=/home/pi/rgb-matrix/watcher.sh
WorkingDirectory=/home/pi/rgb-matrix
StandardOutput=append:/home/pi/rgb-matrix/watcher-stdout.log
StandardError=append:/home/pi/rgb-matrix/watcher-stderr.log
User=root
Group=root

[Install]
WantedBy=multi-user.target

and enable it as well:

$ sudo ln -s ${PWD}/watcher.service /etc/systemd/system/watcher.service
$ sudo systemctl daemon-reload
$ sudo systemctl enable watcher.service
$ sudo systemctl start watcher.service

Done

Now any changes pushed to the remote repo will get automatically fetched and deployed. This of course does not take into account any migrations, installation of dependencies, rollbacks etc. but that's ok :)

Spinning up a free IPFS webrtc-star discovery server with Heroku

Wojtek Siudzinski — Mon, 25 May 2020 18:03:52 +0000

While building the Web Microphone app with IPFS I sumbled upon an error when trying to use older examples but using the latest js-ipfs:

Error: no valid addresses were provided for transports [WebSockets,WebRTCStar,Circuit]

It got me puzzled but Vasco cleared it out in this thread:

Vasco Santos

@vascosantos10

@dietrich @ThePatToner @jacobheun @ThePatToner I see that you are trying to listen on a websocket-star address. Unfortunately, we did a poor job on announcing it, but we are no supporting websocket-star addresses since js-ipfs@0.41

15:38 PM - 19 May 2020

0 0

It turns out that it's necessary to use a new discovery mechanism, webrtc-star. This seemed like an easy way to fix it, just change the server to the hosted randezvous server mentioned in readme and we're done! But nothing is ever that easy. It was failing for me with a 500 error and I shouldn't be surprised as the documentation clearly states:

it should not be used for apps in production

The recommended route is to deploy it on your own. So I tried to get it done in shortes amount of time and preferably without a need to pay for the whole server. Fortunately, Heroku supports Docker containers now and they can be deployed literally with couple of lines! Here's how you do it:

Instructions

First install the Heroku CLI and then run:

# Login to Heroku
$ heroku login
# Login to the Container Registry
$ heroku container:login
# Clone the webrtc-star repo
$ git clone https://github.com/libp2p/js-libp2p-webrtc-star.git
$ cd js-libp2p-webrtc-star
# Create a Heroku app
$ heroku create
# Build and push the image
$ heroku container:push web
# Release the image to your app
$ heroku container:release web
# Scale to one free worker
$ heroku ps:scale web=1
# Open the app in the browser
$ heroku open

Now the server should welcome you with its address:

Use this address in your IPFS app and you're done!

Go for Particle Argon, Boron or Xenon

Wojtek Siudzinski — Thu, 16 Apr 2020 13:28:57 +0000

With the release of TinyGo 0.13, you can now write firmware for Particle Argon, Boron and Xenon in Go:

TinyGo

@tinygolang

We just released version 0.13! Go 1.14 + LLVM 10 + support for 2^5 boards like the @particle #argon + #WebAssembly updates + so much more. Massive thank you to our amazing contributors!

github.com/tinygo-org/tin…

#golang #llvm #tinygo #wasm #microcontrollers #embedded

18:14 PM - 14 Apr 2020

32 70

There are some caveats for now, namely:

TinyGo can't coexist with the Particle Device OS. Meaning flashing your Go app to the device will erase Particle bootloader and system. It is still possible to restore the device to "stock" but it requires manual flashing of all necessary parts.
TinyGo doesn't support Argon's WiFi coprocessor nor Boron's cellular modems yet, meaning for now you can only use the devices offline. Ayke is making progress on support for the nRF Soft Device that allows BLE support (and Mesh in the future) but it's still experimental.
You need to use the Particle Debugger to do all the flashing. It might be possible to use Adafruit nRF52 Bootloader (same as used for CircuitPython) in the future to allow flashing over the USB.

That being said, running TinyGo might be a nice alternative to CircuitPython if you need more speed. How fast is it? Well, it can handle a real-time flight controller for a drone on Arduino Nano33 IoT with SAMD21 Cortex-M0 at 48 MHz:

Or drive six 32x32 RGB LED matrices on Cortex-M4:

conejo 🐇🐰⚡

@_conejo

Do you want to make your own? The code is open source! >> github.com/aykevl/things/… @TinyGolang

18:08 PM - 14 Mar 2020

4 20

Additionally, a strong advantage of TinyGo is that TinyGo is written in Go itself! Meaning, you don't need to know C/C++ to understand/contribute or write low-level code.

Got you interested? Let's proceed to the...

Instructions

Prerequisites

Particle Argon/Boron/Xenon $11-$55
Particle Debugger $12

Necessary software

Install OpenOCD. If you have Particle Workbench installed, it should already download it for you to ~/.particle/toolchains/openocd directory. Make sure the openocd binary is in your PATH.
Install TinyGo

Blinky

Well, that's it really. Now you're ready to go write Go:

package main

import (
    // machine package exposes all hardware available on the microcontroller
    "machine"
    "time"
)

func main() {
    // All the pins are listed here: https://tinygo.org/microcontrollers/machine/particle-xenon/
    led := machine.LED
    // Make it an output
    led.Configure(machine.PinConfig{Mode: machine.PinOutput})
    // Repeat forever
    for {
        // Turn it on (it's active low)
        led.Low()
        time.Sleep(time.Millisecond * 500)

        // And off
        led.High()
        time.Sleep(time.Millisecond * 500)
    }
}

Now you can flash your app with:

$ tinygo flash -target=particle-xenon blink

(assuming you save the Go code above in blink directory). Also, you should set the -target flag to the module you're using.

Summary

Even although Argon and Boron can't connect to the internet yet, peripherals like UART, GPIO, SPI, I2C, ADC, and PWM are working, and there are already plenty of drivers for devices like displays, sensors, etc.

I hope you'll have fun with TinyGo and if you like to chat about it or contribute, feel free to join the Slack channel and check out the GitHub repos.

Single Dockerfile for testing and production using multi-stage builds

Wojtek Siudzinski — Fri, 27 Mar 2020 11:11:07 +0000

In different contexts like development, running tests or serving production, our apps have different needs when it comes to their Docker images. For development, you might want a live-reload server with all necessary dependencies. For testing you probably need some more packages that you definitely don't want on production.

There are ways to go about it, the first one that pops in mind is using multiple Dockerfiles and just telling Docker which one to use. But this approach has its limits, especially along the road, where each file can very easily develop its own life, diverging from each other beyond recognition.

Fortunately since Docker 17.05 we have the concept of multi-stage builds. An ability to declare separate images in one Dockerfile and freely inherit or copy data between each other. This is the exact approach I used when building images for Lox:

#############################################
# Base container with all necessary deps
FROM tiangolo/uvicorn-gunicorn:python3.7-alpine3.8 AS base

ENV HOME=/app 
    BUILD_DEPS="build-base linux-headers postgresql postgresql-dev libffi-dev"
WORKDIR ${HOME}

# Copy the pipenv files
COPY Pipfile ${WORKDIR}/
COPY Pipfile.lock ${WORKDIR}/

# 1. Install system dependencies
# 2. Install pipenv
# 3. Use pipenv to install app deps
# 4. Remove system deps to save space
RUN apk add --no-cache ${BUILD_DEPS} \
    && pip install --no-cache-dir pipenv \
    && pipenv install --system \
    && apk del ${BUILD_DEPS}

#############################################
# Test container from a common base
FROM base AS test
# Same as in the base image but this time we also install the --dev packages
RUN apk add --no-cache ${BUILD_DEPS} \
    && pip install --no-cache-dir pipenv \
    && pipenv install --system --dev \
    && apk del ${BUILD_DEPS}

#############################################
# Live container with Webpack watcher
FROM base AS live
# Install Node.js dependencies
RUN apk add --no-cache nodejs npm
# Install all Webpack dependencies
COPY package.json package-lock.json ./
RUN npm install
# Finally copy the current sources and build the bundle
COPY . .
RUN npm run bundle:build

#############################################
# Final container with the app
FROM base AS production
# Only copy the ready app dir from the live step
COPY --from=live ${WORKDIR} ${WORKDIR}

Note: for the sake of brevity, this Dockerfile doesn't contain all size improvements and will produce slightly bigger images.

Now you're able to build separate images for each of your needs:

Testing

$ docker build -t suda/lox/test --target test .

Development w/ live-reload

$ docker build -t suda/lox/live --target live .

Production

$ docker build -t suda/lox --target production .

P.S.: You might've noticed I used the great Python base image from Sebastián Ramírez and if you want to deploy a Django project using ASGI, I have some instructions that can help you.

Production-ready Django 3 ASGI Docker image

Wojtek Siudzinski — Thu, 02 Jan 2020 16:49:49 +0000

With Django 3 released, it's a great moment to jump in on all the async goodies it provides. Unfortunately for me, it means dropping my uWSGI Docker config and figuring out a new approach.

Fortunately, Sebastián Ramírez of fastapi fame, created a very nice base image using uvicorn: uvicorn-gunicorn-docker. One thing missing was the ability to serve static files, but here's where WhiteNoise comes in.

Let's jump in to the setup.

Instructions

Compared to my uWSGI setup, here we just need to create a Dockerfile:

ARG PYTHON_VERSION=3.7

# Build dependencies in separate container
FROM tiangolo/uvicorn-gunicorn:python${PYTHON_VERSION}-alpine3.8 AS builder
ENV WORKDIR /app
COPY Pipfile ${WORKDIR}/
COPY Pipfile.lock ${WORKDIR}/

RUN cd ${WORKDIR} \
    && pip install pipenv \
    && pipenv install --system

# Create the final container with the app
FROM tiangolo/uvicorn-gunicorn:python${PYTHON_VERSION}-alpine3.8

ENV USER=docker \
    GROUP=docker \
    UID=12345 \
    GID=23456 \
    HOME=/app \
    PYTHONUNBUFFERED=1
WORKDIR ${HOME}

# Create user/group
RUN addgroup --gid "${GID}" "${GROUP}" \
    && adduser \
    --disabled-password \
    --gecos "" \
    --home "$(pwd)" \
    --ingroup "${GROUP}" \
    --no-create-home \
    --uid "${UID}" \
    "${USER}"

# Run as docker user
USER ${USER}
# Copy installed packages
COPY --from=builder /usr/local/lib/python3.7/site-packages /usr/local/lib/python3.7/site-packages
# Copy the application
COPY --chown=docker:docker . .
# Collect the static files
RUN python manage.py collectstatic --noinput

Serving the static files

For this purpouse, I'm using WhiteNoise which needs to be added to dependencies:

$ pipenv install whitenoise

and then added whitenoise.middleware.WhiteNoiseMiddleware to the top of the MIDDLEWARE array in settings.py, right below the SecurityMiddleware:

MIDDLEWARE = [
  'django.middleware.security.SecurityMiddleware',
  'whitenoise.middleware.WhiteNoiseMiddleware',
  # ...
]

Side note: I know many people are wondering why I'm so set on serving static files, instead of just using S3 buit I think David explained it well:

Shouldn’t I be pushing my static files to S3 using something like Django-Storages?

No. (...) problem with a push-based approach to handling static files is that it adds complexity and fragility to your deployment process.

Running the image

uvicorn needs to know what is the module is the main application, therefore we need to set the APP_MODULE environment variable to myapp.asgi:application replacing myapp with the name of your app. This assumes you have the asgi.py file (ASGI equivalent of the wsgi.py) which will be generated automatically when creating new Django 3.0 project but if you're migrating from an older version, you can use this one:

"""
ASGI config for myapp project.

It exposes the ASGI callable as a module-level variable named ``application``.

For more information on this file, see
https://docs.djangoproject.com/en/3.0/howto/deployment/asgi/
"""

import os

from django.core.asgi import get_asgi_application

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myapp.settings')

application = get_asgi_application()

Notes

Running in async mode, means you should not use possibly blocking, synchronous methods (like the ORM) in the main thread. Quick tip here is, that if you get SynchronousOnlyOperation exception you might want to wrap it with sync_to_async:

from asgiref.sync import sync_to_async

def my_db_function():
    <do orm stuff>

await sync_to_async(my_db_function)()

Deploying Django to production with uWSGI

Wojtek Siudzinski — Sun, 01 Sep 2019 12:28:00 +0000

There are many posts about dockerizing Django apps but I feel like there's some room for improvement.

Many of the approaches only focused on development flow, making the resulting image great for iterating but not ideal for production usage. Other ones, produced very big images, were running the application as root or ignored handling static files and deferred it to S3. With these images I had several goals:

Production-ready image (following general security tips)
Using alpine base for a small image
Multi-stage build for even smaller and cleaner image
Handling static files inside the same container

I decided on using uWSGI which actually also can serve static files, making my life much easier.

Note: I'm using pipenv to manage virtualenv and all dependencies (and I recommend you use it too), therefore some instructions might need tweaking if you're using bare pip.

Note 2: If you prefer running in ASGI, I wrote instructions for uvicorn based Docker image as well.

Instructions

Setup `uWSGI`

Start by adding it to your dependencies:

$ pipenv install uwsgi

Then create uwsgi.ini file in your project root directory:

[uwsgi]
chdir = /app
uid = $(UID)
gid = $(GID)
module = $(UWSGI_MODULE)
processes = $(UWSGI_PROCESSES)
threads = $(UWSGI_THREADS)
procname-prefix-spaced = uwsgi:$(UWSGI_MODULE)

http-socket = :8080
http-enable-proxy-protocol = 1
http-auto-chunked = true
http-keepalive = 75
http-timeout = 75
stats = :1717
stats-http = 1
offload-threads = $(UWSGI_OFFLOAD_THREADS)

# Better startup/shutdown in docker:
die-on-term = 1
lazy-apps = 0

vacuum = 1
master = 1
enable-threads = true
thunder-lock = 1
buffer-size = 65535

# Logging
log-x-forwarded-for = true

# Avoid errors on aborted client connections
ignore-sigpipe = true
ignore-write-errors = true
disable-write-exception = true

no-defer-accept = 1

# Limits, Kill requests after 120 seconds
harakiri = 120
harakiri-verbose = true
post-buffering = 4096

# Custom headers
add-header = X-Content-Type-Options: nosniff
add-header = X-XSS-Protection: 1; mode=block
add-header = Strict-Transport-Security: max-age=16070400
add-header = Connection: Keep-Alive

# Static file serving with caching headers and gzip
static-map = /static=/app/staticfiles
static-map = /media=/app/media
static-safe = /usr/local/lib/python3.7/site-packages/
static-gzip-dir = /app/staticfiles/
static-expires = /app/staticfiles/CACHE/* $(UWSGI_STATIC_EXPIRES)
static-expires = /app/media/cache/* $(UWSGI_STATIC_EXPIRES)
static-expires = /app/staticfiles/frontend/img/* $(UWSGI_STATIC_EXPIRES)
static-expires = /app/staticfiles/frontend/fonts/* $(UWSGI_STATIC_EXPIRES)
static-expires = /app/* 3600
route-uri = ^/static/ addheader:Vary: Accept-Encoding
error-route-uri = ^/static/ addheader:Cache-Control: no-cache

# Cache stat() calls
cache2 = name=statcalls,items=30
static-cache-paths = 86400

# Redirect http -> https
route-if = equal:${HTTP_X_FORWARDED_PROTO};http redirect-permanent:https://${HTTP_HOST}${REQUEST_URI}

Note: The author of the config above is Diederik van der Boor 🙇

Create the `Dockerfile`

# Build argument allowing to change Python version
ARG PYTHON_VERSION=3.7

# Build dependencies in a separate container
FROM python:${PYTHON_VERSION}-alpine AS builder
ENV WORKDIR /app
COPY Pipfile ${WORKDIR}/
COPY Pipfile.lock ${WORKDIR}/

RUN cd ${WORKDIR} \
    && pip install pipenv \
    && pipenv install --system

# Create the final container with the app
FROM python:${PYTHON_VERSION}-alpine

ENV USER=docker \
    GROUP=docker \
    UID=12345 \
    GID=23456 \
    HOME=/app \
    PYTHONUNBUFFERED=1
WORKDIR ${HOME}

# Create user/group
RUN addgroup --gid "${GID}" "${GROUP}" \
    && adduser \
    --disabled-password \
    --gecos "" \
    --home "$(pwd)" \
    --ingroup "${GROUP}" \
    --no-create-home \
    --uid "${UID}" \
    "${USER}"

# Run as docker user
USER ${USER}
# Copy installed packages
COPY --from=builder /usr/local/lib/python3.7/site-packages /usr/local/lib/python3.7/site-packages
# Copy uWSGI binary
COPY --from=builder /usr/local/bin/uwsgi /usr/local/bin/uwsgi
# Copy the application
COPY --chown=docker:docker . .
# Collect static files
RUN python manage.py collectstatic --noinput

ENTRYPOINT ["uwsgi", "--ini", "uwsgi.ini"]
EXPOSE 8080

Start container

To work correctly, you need to set some environment variables. If you're using Docker Compose, you could use similar docker-compose.yml file:

version: "3"
services:
  app:
    build: .
    image: foo/bar
    ports:
      - "8080:8080"
    environment:
      - UWSGI_MODULE=myapp.wsgi:application
      - UWSGI_PROCESSES=10
      - UWSGI_THREADS=2
      - UWSGI_OFFLOAD_THREADS=10
      - UWSGI_STATIC_EXPIRES=86400

then you can build the image with the following command:

$ docker-compose build app

Conclusion

I'm quite happy with this setup and it has been working in production for some time. Please let me know if you have any comments and if I could improve this more!

Forem: Wojtek Siudzinski

Quick and dirty auto-deployment to Raspberry Pi

The setup

Git setup

Main application service

Auto-deployment service

Done

Spinning up a free IPFS webrtc-star discovery server with Heroku

Instructions

Go for Particle Argon, Boron or Xenon

Instructions

Prerequisites

Necessary software

Blinky

Summary

Single Dockerfile for testing and production using multi-stage builds

Testing

Development w/ live-reload

Production

Production-ready Django 3 ASGI Docker image

Instructions

Serving the static files

Running the image

Notes

Deploying Django to production with uWSGI

Instructions

Setup uWSGI

Create the Dockerfile

Start container

Conclusion

Setup `uWSGI`

Create the `Dockerfile`