Forem: Patrice Ferlet

Git worktree like a boss

Patrice Ferlet — Wed, 18 Mar 2026 21:14:01 +0000

If there’s one Git tool that few people know about, it’s “worktree.” Once you’ve mastered this tool, you’ll find it incredibly hard to do without it. But you have to use it the right way. Here’s a method I’ve been using for a very long time

TL;DR

In an empty directory:

# 1. Clone the repo into a hidden .bare folder
git clone --bare git@github.com:user/repo.git .bare

# 2. Tell the root folder where the Git history is hidden
echo "gitdir: ./.bare" > .git

# 3. Fix the fetch configuration to see all remote branches
git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"

# 4. Create your first worktree (the main branch)
git worktree add main
git worktree add <name of the branch>

# Each folder correspond to a branch, everything is centralized
# Read the doc... or follow the article

I provide you a script you can put in your PATH to ease the process. Take a look at the end of the article.

First of all, what is a “worktree”?

In the fast-paced world of software development, context switching is often an unavoidable reality. You’ve likely been there: you’re deep into a complex feature branch when a critical production bug suddenly demands your immediate attention.

Or, sometimes, you have to manage 2 branches at the same time because you are preparing a big migration, refactoring or something complex that will be merged. So you need direct to several "working trees".

Traditionally, developers have relied on git stash to clear their workspace or simply cloned the entire repository into a separate folder. However, these methods can be clunky, error-prone, or heavy on disk space. This is where Git Worktree comes in.

Introduced in version 2.5, git worktree allows you to have multiple working directories attached to a single repository. Instead of having just one "checked out" branch at a time, you can have several branches checked out simultaneously in different folders. All these folders share the same underlying .git history, meaning you don't need to re-download data or deal with multiple independent clones.

You will understand why it's so powerful as soon as you will follow my method 😄

The bad way

I often see articles that explains you can do a worktree using "../name-of-worktree". That works and I can understand why it's the "common way to do". The developper has already cloned the repository and it does this to quickly have a second workspace for a branch.

But... it's not what I advice.

How to do better

What I do for a long time now is to use a "bare" clone. This will not clone the repository but an history of the répository. So, after the below command, you are not ready to work. Let's take an example with my project:

mkdir -p ~/Projects/Katenary
cd ~/Projects/Katenary
git clone --bare git@repo.katenary.io:Katenary/katenary.git .bare

In the .bare directory, you'll see something like:

.bare
├── config
├── description
├── HEAD
├── hooks
├── info
├── objects
├── packed-refs
└── refs

When you use the Bare Clone method, you want your root project folder to stay clean. The goal is to have a hidden administrative folder and then your various worktrees (branches) as sibling folders.

Here is the "magic trick" to make the root folder behave like a Git repository without actually cluttering it:

echo "gitdir: ./.bare" > .git

By default, if you are inside your project folder but outside a worktree, Git won't know where the history is. By adding this one-line .git file, you are telling your terminal: "Treat this entire directory as part of the Git project located in ./bare."

This allows you to run git worktree commands from the root of your project instead of having to cd into the .bare folder every time. It’s the glue that holds the "Pro" structure together.

And now, ladies and gentlemen !

# I will explain this line in a few moments
git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"

# let's start to work on different branches
git worktree add main
git worktree add feature/better-depends-on

This makes:

.
├── feature
│   └── better-depends-on
└── main

I now see 2 folders, main and better-dependencies. Each directory contains the content of the desired branch. I can work in both of them, push, pull, and so on.

The "Blind Clone" Trap: Why Your Fetch is Empty

If you've followed the steps so far, you might run into a confusing situation. You try to add a worktree for an existing remote branch, but Git instead creates a new, empty branch starting from main.

Look at your terminal. If git fetch only shows this:

git fetch
* branch            HEAD       -> FETCH_HEAD

You are officially "Blind."

Your local repository knows the code exists, but it has no idea that other branches exist on the server.

When you use git clone --bare, Git assumes you want a backup or a server-side mirror, not a workspace for active development. To save resources, it doesn't set up "Remote Tracking." It only fetches the default branch (main or master).

The Fix: Mapping the Remote
To turn your bare repository into a fully-aware development hub, you need to tell Git: "I want to see everything that happens on the origin server."

Run this command:

git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"

Now, run git fetch --all again. You should see a list of all your teammates' branches appearing in your terminal.

Wait... it's only like having cloned twice, right ?

At first glance, you might think: "Why go through all this trouble? If I need to work on two branches at once, I’ll just clone the repo into two different folders and be done with it."

Technically, you could. But comparing a Git Worktree setup to multiple clones is like comparing a synchronized multi-room audio system to buying two separate stereos and trying to press "Play" at the exact same millisecond.

Here is why the "Multiple Clones" method is actually a trap:

1. The Storage Tax

If your repository is 500MB, two clones take up 1GB. With Worktrees, you have one .git folder (the "source of truth"). Each additional worktree only adds the weight of the actual text files you are editing. Your disk will thank you.

2. The "Single Source of Truth" Problem

This is the real game-changer. In a multi-clone setup:

If you git fetch in Clone A, Clone B knows nothing about it.
If you create a local branch in Clone A, you have to push it to the server (or a local remote) just to see it in Clone B.

With Worktrees, they all share the same internal database. If you fetch in your main worktree, the new branches are instantly available in your feature-xyz worktree. It’s one brain, many bodies.

3. Hooks and Configs

Ever spent 20 minutes setting up a specific Git hook or a local config, only to realize you have to do it all over again because you're in a different clone?

Since Worktrees share the same .git directory, your hooks, aliases, and local configurations are universal across all your working folders.

4. The "Safety Valve"

Git is smart: it won't let you check out the same branch in two different worktrees at the same time. This prevents you from accidentally overwriting your own work or creating massive HEAD conflicts—a protection you don't have if you're just using multiple clones.

In short: Multiple clones are isolated silos. Worktrees are a unified workspace.

The "pro" setup

# 1. Clone the repo into a hidden .bare folder
git clone --bare git@github.com:user/repo.git .bare

# 2. Tell the root folder where the Git history is hidden
echo "gitdir: ./.bare" > .git

# 3. Fix the fetch configuration to see all remote branches
git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"

# 4. Create your first worktree (the main branch)
git worktree add main

And now, a if you're in the "main" folder, an urgent hotfix must be done.

# inside the "main" directory
git worktree add ../hotfix
cd ../hotfix

This create a branch and the folder where you can work without having to stop working on "main" branch.

To be explicit:

git worktree add <where to create the tree> [-b name of the branch to create] [branch name reference]

For example, I need to create "feature/fix-db" from "feature/improve-db". I'm in the root workspace:

git worktree add improve-db -b feeature/fix-db feature/improve-db

The directory is created, I can now go to "./improve-db" and do my job.

Of course, you can use subdirectories (place all features in a features directory) and git worktree create them by default if you use slashes in your branchnames. I mean that git worktree add features/A creates features/A directory. And the branch is named this way, and ready to be pushed.

Some traps

Removing a worktree with "rm -rf" doesn't really remove the worktree (no panic, it doesn't remove remote branches). Then if you want to get it back, you will think that "git worktree add" one more time will respawn the directory. But, an error will appear saying that the worktree already exists.

Is this is the case, you can do:

git worktree list

You will see "prunable" worktrees. You can then prune them:

git worktree prune

Then, you can use the add subcommand.

You can protect worktrees to not being pruned with the "lock" subcommand, and of course "unlock" it at any time.

Conclusion: Give it a Week

I’ll be honest: if you’ve been using git stash and multiple clones for years, the Git Worktree workflow might feel like "over-engineering" a problem you don't think you have. It’s hard to see the immediate value when you're used to the friction of traditional context switching.

But here is my challenge to you: Try it for a week.

Force yourself to use a Bare Clone setup on your next big project. The moment you realize you can run a heavy test suite in one folder while simultaneously hotfixing a bug in another—without ever losing your place or your "flow"—it will click.

You’ll stop thinking about "switching branches" and start thinking about "parallel workspaces." Your terminal will be cleaner, your disk space will be optimized, and your brain will thank you for not having to remember what you stashed three hours ago.

Once you go Worktree, you rarely go back.

A simple script

I placed this in my ~/.local/bin folder named wtree (don't forget to chmod +x wtree):

#!/bin/bash

# Usage: wtree <git url> (in an empty directory)

REPO_URL=$1
SCRIPT_NAME=$(basename "$0")

if [ -z "$REPO_URL" ]; then
    echo "❌ Error: Missing repository URL."
    echo "Usage: $0 <repo-url>"
    exit 1
fi

# 1. Check if the current directory is empty
# We allow the script itself to be present
FILES_IN_DIR=$(ls -A | grep -v "$SCRIPT_NAME")

if [ ! -z "$FILES_IN_DIR" ]; then
    echo "❌ Error: Current directory is not empty!"
    echo "To keep your worktree setup clean, please run this in a fresh folder."
    exit 1
fi

echo "🚀 Initializing Pro Git Worktree setup..."

# 2. Clone as bare into a hidden directory
git clone --bare "$REPO_URL" .bare

# 3. Link the root to the bare repository
echo "gitdir: ./.bare" > .git

# 4. Enable remote tracking (The "Remote Awareness" fix)
# This ensures 'git fetch' sees all branches from the server
git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"

# 5. Fetch everything
git fetch --all

echo "✅ Setup complete!"
echo "Next step: git worktree add main"

[Boost]

Patrice Ferlet — Thu, 23 Jan 2025 21:03:51 +0000

Simple backup service with rclone, restic and systemd on WebDAV (or another storage)

Patrice Ferlet ・ Jan 23

#backup #systemd #rclone #restic

Simple backup service with rclone, restic and systemd on WebDAV (or another storage)

Patrice Ferlet — Thu, 23 Jan 2025 21:03:31 +0000

Let me show you how I back up my computers with restic and rclone using simple systemd configuration.

On my previous article, I presented how I was able to have a nice NAS (storage) with Nextcloud. It has got the good idea to propose a WebDAV endpoint that can be mounted on our Linux computers, or on Windows with the client.

By chance, Gnome provides an 'Online Account' option to automatically mount my remote hard drive on Nautilus, the file manager, and to synchronize contacts, calendars, etc.

But, I wanted to have a “real” backup service too.

Because a "real" backup system is not only to send files on a remote drive. It is a complete snapshot management, with incremental checks, compression, security...

This signifies that I intend to create backups for certain directories, excluding others, to retrieve files from a period of 7 days, 1 week, 1 month, or even 1 year ago.

This is what we call an "incremental" backup system.

☝️ This article proposes a backup configuration for a "user" on Linux but you can adapt the configuration to backup a server or your system as an administrator.

💡 What we will see in this article

What are Restic and Rclone commands
How to back up some directories and restore them
How to automate the backup with systemd for a Linux user (or for an admin account)

TL;DR

# Create your configuration to connect a storage
# if restic doesn't provide the connector (e.g. for WebDav).
# Name your connection, for example "nas"
rclone config

# initialize the backup storage in a directory,
# e.g. Backups
restic -r rclone:nas:Backups init

# Create a simple backup one shot
restic -r rclone:nas:Backups backup $HOME/Images --verbose

# Forget some backups:
restic -r rclone:nas:Backups \
    forget --prune \
    --keep-daily 7 \
    --keep-weekly 4 \
    --keep-monthly 6 \
    --keep-yearly 1

# This is to make management easiers:
# Create files to list directories to backup and what to exclude.
# It support "patterns"
echo "$HOME/Documents" >> ~/.config/backup.list
echo "*.tmp" >> ~/.config/backup.exclude


# example of backup command:
restic -r rclone:nas:Backups backup \
    --files-from ~/.config/backup.list \
    --exclude-file ~/.config/backup.exclude

# Scroll to the bottom of the article to find systemd
# service and timers

# Done !

Please read the article to see how to restore a snapshot, to simplify the management, …

Incremental backup and GFS Backups?

An incremental backup is a method of backing up solely the modifications from the most recent backup to conserve space.

Moreover, we can do a GFS backup.

GFS? What is this?

GFS means " Grandfather-Father-Son Backup", this is a method to make daily backup every day but:

All backups older than a certain date, such as 7 days old, are removed.
Keep a backup each week for four weeks.
Keep a backup every month for six months.
Keep one backup per year, per year.

This facilitates the storage of substantial data in a timely manner while utilizing minimal space.

It's a very common method, I use it at work to backup dozen of servers..

🎉 ❤️ And yes, Restic can do all of this!

Restic

Restic is a backup command line. It can use many local or remote endpoints like SFTP, Samba, or S3 protocol.

Restic can make backups, and make snapshot restoration, cleanup, mounting snapshots to navigate into, secure data by encryption, and many more! 👏

Unfortunatly, Restic doesn't support WebDav, but there is a solution: using rclone adapter.

Restic will create a “repository”. It's a remote or local path where Restic will keep snapshots, data, and status.

We will initialize it in a while.

Rclone

Rclone is a command line to send / synchronize and manage local or remote directories. It is compatible with a huge number of providers like Google Drive, Amazon S3, S3 compatible, Samba, Proton Drive, Dropbox… and WebDAV!

Create a Rclone configuration

OK, in my case, I need to contact my local WebDAV (nas.home is a domain locally assigned by my router, an Orange Livebox)

On my Nextcloud account, browsing my files, I can see the WebDAV URL in the settings:

In a terminal, I only do:

rclone config

I needed to find the WebDAV option, then Nextcloud, then give my URL, user and password.

Another way is to use one command line to set it up all:

# change the url, user and password of course
rclone config create nas \
webdav \
url https://nas.home/remote.php/dav/files/Patrice \
user Patrice \
vendor nextcloud \
pass $(rclone obscure 'TYPE YOUR NextCloud Password here')

⚠️ The obscure command is not a encrption method. It "obscures" the password. It is only made to avoid "direct reading". Rclone can read the original string.

To check if it works:

# liste directories only
rclone lsd nas:

If it works, you may continue

Restic, initialization

We have to initialize a Restic repository. This will create the data and configuration on the remote storage.

So, we created a nas: Rclone configuration to access our WebDAV endpoint; let's use it:

restic -r rclone:nas:Backups init

The command will ask you to provide a password. This password is used to encrypt your backup! This is not your WebDAV password, this is a secret to avoid others to read your backups.

⚠️ If you lose this secret, you will never be able to restore snapshots later!

Now, Restic is ready.

Test Restic backup

First, create a simple directory where we will store a simple file.

mkdir /tmp/data
echo "hello" > /tmp/hello.txt
date > /tmp/data/file.txt

Let backup this directory:

restic -r rclone:nas:Backups backup /tmp/data
# it will ask you the repository password

# output
repository 52592651 opened (version 2, compression level auto)
no parent snapshot found, will read all files
[0:01] 100.00%  5 / 5 index files loaded

Files:           2 new,     0 changed,     0 unmodified
Dirs:            2 new,     0 changed,     0 unmodified
Added to the repository: 1.902 KiB (1.361 KiB stored)

processed 2 files, 38 B in 0:03
snapshot cb53a1b9 saved

The directory is now snapshot in the Backups repository.

Let's take a look:

restic -r rclone:nas:Backups snapshots
# output
repository 52592651 opened (version 2, compression level auto)
ID        Time                 Host            Tags        Paths                           Size
--------------------------------------------------------------------
cb53a1b9  2025-01-22 22:13:21  patrice-laptop              /tmp/data                       38 B
--------------------------------------------------------------------
1 snapshots

(Actually there are many other snapshots in my output as I have already made several backups)

OK, so the snapshot is here. Let's break our local directory:

# oops, accidentally removed :)
rm /tmp/data/hello.txt

# remove the date, write "broken"
echo "broken" > /tmp/data/file.txt

How to recover?

The simplest is to use the snapshot ID and specify where to restore:

restic -r rclone:nas:Backups restore cb53a1b9 --target /
# output
repository 52592651 opened (version 2, compression level auto)
[0:03] 100.00%  7 / 7 index files loaded
restoring snapshot cb53a1b9 of [/tmp/data] at 2025-01-22 22:13:21.433737756 +0100 CET by metal3d@patrice-laptop to /tmp/data
Summary: Restored 4 files/dirs (38 B) in 0:04

Why "/" and not "/tmp/data"? Because the full path is stored in restic.

ℹ️ You can use latest instead of the cb53a1b9 ID (which is different for you). latest means "the latest snapshot". Using the ID is interesting to get an older snapshot, for example if the latest snapshot contains a broken file.

Let it restoring, and…

$ tree /tmp/data
/tmp/data/
├── file.txt
└── hello.txt

# back!
$ echo /tmp/hello.txt
hello

# not broken :)
$ echo /tmp/file.txt
mer. 22 janv. 2025 22:12:47 CET

Then:

you can create other snapshots, it “incrementally” adds files.
if nothing changed in your local directory, so the snapshot will be “empty”, it's normal
only the newest files and changed content will be stored in the following snapshots — it saves space!
you can recover snapshots at a certain date

But... We need to cleanup very old backups

To clean backups, the forget command is simple to use.

I want to keep 7 days of backups, but also 1 backup per week for 4 weeks, and 1 backup per month for 6 months, and finally one backup per year for one year…

restic -r rclone:nas:Backups forget \
--keep-daily 7 \
--keep-weekly 4 \
--keep-monthly 6 \
--keep-yearly 1 \
--prune

OK, there are a lot of information. And there are plenty of options and commands in restic. Read the doc 😉

⚠️ Crucial:

If you never heard about incremental backup, you may wonder if a large directory will consume all your CPU and bandwidth every day.

No

This is why a “real” backup system is very intriguing. The first backup is a bit long as there is no track of the files you're backing up.

The next backup will only take into account the unknown and changed files.

So then:

you can revert files as if they were versioned
and the next backups will be faster, really faster

Some nice features

Something very interesting when you want to restore a file, is that you can use --include to only get one file or directory. Moreover, there is a find subcommand to find a file.

But, another way is to “mount” the snapshots on your host.

Yes, you can navigate the snapshots

Mounting snapshots

It only works on Linux... OK for me 😉

If you use your WebDAV drive, or the web interface of Nextcloud, the snapshots are encrypted and not usable:

But, using restic mount:

mkdir /tmp/restic
restic -r rclone:nas:Backup mount /tmp/restic

You can now browse the snapshots:

This is very comfortable to be able to navigate snapshots, at a certain date, check the content of a file and to be able to copy and paste it.

Now, automate with Systemd

There are several new things that are interesting to know:

you can set the Restic password as RESTIC_PASSWORD environment variable
you can set the Restic repository name in the RESTIC_REPOSITORY environment variable
you can provide several directories to snapshot at once
the list can be set in the command line, or inside a text file

So, with this information, we can create a user service and associated timer — this to automate backup.

💡 If your want to use Restic for a server or to backup a root system, so you only need to change the paths to /etc/systemd/system to save files, and do not use --user option.

The service in “~/.config/systemd/user/backup.service”:

[Unit]
Description=Backup service to NextCloud

[Service]
Type=oneshot
Nice=19
# environment variables to simplify commands
Environment=RESTIC_PASSWORD=YOUR PASSWORD HERE
Environment=RESTIC_REPOSITORY=rclone:nas:Backups

# Call sequentially
ExecStart=/usr/bin/restic backup --files-from %h/.config/backup.list --exclude-file %h/.config/backup.exclude --verbose
ExecStart=/usr/bin/restic forget --prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --keep-yearly 1

[Install]
WantedBy=multi-user.target

The Type=oneshot is important:

you can start several commands sequentially
it will not restart as other services that are expected to be continuously “alive”

As you can see, I'm using files to list directories and exclusion.

Let's create them:

# for example, backup Document folder
echo $HOME/Documents >> ~/.config/backup.list
# and exclude all .tmp files
echo '*.tmp' >> ~/.config/backup.exclude

You can edit the file with a text editor, of course.

Then, create the timer in ~/.config/systemd/user/backup.timer:

[Unit]
Description=Planify the Restic backup service

[Timer]
# you can set specific date / time
# OnCalendar=*-*-* 02:00:00
# or simply leave the system default which
# is 3:00 am
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

It's important to have the same basename for the timer and the service.

OK, let's go:

systemctl --user daemon-reload
systemctl --user enable --now backup.timer

And voilà! Every day, a snapshot will be made for all directories you've listed in the list file. And it avoids all directory or patterns that you've listed in the exclude file.

To check if the backup works, you can, of course, launch the service right now:

# Warning, if the backup is huge, this command will
# stay alive for a long time. Test with a tiny directory to backup.
systemctl --user start backup

Conclusion

Restic and Rclone are both powerful. Typing a few command lines to manage backup is much more comfortable on Unix/Like and Linux systems (it works on FreeBSD, OpenBSD, macOS…)

It can work on Windows computers. But I never tried (I don't have any Windows machine at home)

Anyway, one more time systemd is IMHO one of the best things that happened to Linux. Whatever others think, I love it.

And, again, using a terminal command that does one thing, but does it well, is a power in our hands.

[Boost]

Patrice Ferlet — Sun, 19 Jan 2025 20:51:18 +0000

How to resolve the dlopen problem with Nvidia and PyTorch or Tensorflow inside a virtual env

Patrice Ferlet ・ Jan 19

#nvidia #pytorch #tensorflow #gpu

How to resolve the dlopen problem with Nvidia and PyTorch or Tensorflow inside a virtual env

Patrice Ferlet — Sun, 19 Jan 2025 20:50:58 +0000

If you install PyTorch or Tensorflow with cuda dependencies, you probably have the same problem as I had: the GPU is not detected and an error about dlopen appears. This article explains why and how to fix it.

TL;DR

# before launching your python, poetry run, pipenv run commands
export LD_LIBRARY_PATH=$(find .venv -name "*.so*" | grep nvidia | xargs dirname | sort -u | paste -d ":" -s -)

Using Nvidia pip packages

I really don't like having CUDA inside my root system on Linux. It's not open source, it's huge, and I often need to fix symlinks on several versions. This is not comfortable, and it needs to add a repository.

By chance, Tensorflow or PyTorch can work with pip packages from Nvidia.

You can, for example, use nvidia-cudnn-cu11 to install cuDnn for CUDA v11 inside your virtual environment. This even using poetry, pipenv or manually.

Tensorflow offers a nice subpackage that installs everything needed:

# or poetry add, or pipenv install ...
pip install "tensorflow[and-cuda]"

OK, that's fun. But here is the problem...

The problem

Trying to use my GPU gives me an error. Actually, the libraries are not found and so Nvidia isn't able to use my GPU.

Let me show you a simple case:

# you could remve this after your tests
mkdir -p Projects/ML/testgpu
cd Projects/ML/testgpu

# install with poetry
mkdir .venv
poetry init --no-interaction --python "^3.12"
poetry env use 3.12

# could be long
poetry add "tensorflow[and-cuda]"

poetry run python -c "import tensorflow;tensorflow.keras.Sequential().compile()"

# in the output:
#...
W0000 00:00:1737318822.063024   15978 gpu_device.cc:2344] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...

OK, what's the problem?

You installed "CUDA" in ".venv" but Nvidia uses subdirectories and paths that are not activated by default.

We need to force the LD_LIBRARY_PATH environment to include every directories where Nvidia installed their shared libraries...

But there are a lot of libraries, a lot of ".so" files with suffix.

The solution

By chance, we are using Linux!

On our terminal, we have got some very powerful tools:

find can "find" (surprise...) files in a directory with a pattern
grep can filter the output to find what we need
xargs can be used to apply a command on each output line, and we can use dirname to get the directory name
sort can sort the output and also remove doubles with "-u" (unique) option
paste is a very nice tool to concatenate strings

So, let's try:

find .venv -name "*.so*" | grep nvidia | xargs dirname | sort -u
# output:
.venv/lib/python3.12/site-packages/nvidia/cublas/lib
.venv/lib/python3.12/site-packages/nvidia/cuda_cupti/lib
.venv/lib/python3.12/site-packages/nvidia/cuda_nvcc/nvvm/lib64
.venv/lib/python3.12/site-packages/nvidia/cuda_nvrtc/lib
.venv/lib/python3.12/site-packages/nvidia/cuda_runtime/lib
.venv/lib/python3.12/site-packages/nvidia/cudnn/lib
.venv/lib/python3.12/site-packages/nvidia/cufft/lib
.venv/lib/python3.12/site-packages/nvidia/curand/lib
.venv/lib/python3.12/site-packages/nvidia/cusolver/lib
.venv/lib/python3.12/site-packages/nvidia/cusparse/lib
.venv/lib/python3.12/site-packages/nvidia/nccl/lib
.venv/lib/python3.12/site-packages/nvidia/nvjitlink/lib

# let's concatenate
find .venv -name "*.so*" | grep nvidia | xargs dirname | sort -u | paste -d ":" -s -
# output:
.venv/lib/python3.12/site-packages/nvidia/cublas/lib:.venv/lib/python3.12/site-packages/nvidia/cuda_cupti/lib:.venv/lib/python3.12/site-packages/nvidia/cuda_nvcc/nvvm/lib64:.venv/lib/python3.12/site-packages/nvidia/cuda_nvrtc/lib:.venv/lib/python3.12/site-packages/nvidia/cuda_runtime/lib:.venv/lib/python3.12/site-packages/nvidia/cudnn/lib:.venv/lib/python3.12/site-packages/nvidia/cufft/lib:.venv/lib/python3.12/site-packages/nvidia/curand/lib:.venv/lib/python3.12/site-packages/nvidia/cusolver/lib:.venv/lib/python3.12/site-packages/nvidia/cusparse/lib:.venv/lib/python3.12/site-packages/nvidia/nccl/lib:.venv/lib/python3.12/site-packages/nvidia/nvjitlink/lib

# OK, so make the LD_LIBRARY_PATH variable:
export LD_LIBRARY_PATH=$(find .venv -name "*.so*" | grep nvidia | xargs dirname | sort -u | paste -d ":" -s -)

And so, in the same terminal:

poetry run python -c "import tensorflow;tensorflow.keras.Sequential().compile()"
# output
#...
I0000 00:00:1737319126.393498   16936 gpu_device.cc:2022] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 4169 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3060 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6

Hooray, my GPU is now detected!

Nextcloud on Raspberry Pi - Fedora + Podman Quadlets

Patrice Ferlet — Wed, 15 Jan 2025 13:45:20 +0000

Let me show you how to install Nextcloud on a Raspberry Pi using Fedora, Podman and Quadlet (containers). It's simple !

What is Quadlet

A Quadlet is a way to create a service (with systemd) from a container. It uses Podman, a container engine. As it is OCI complient, the Docker images are compatibles with Podman.

Don't be impressed by the number of command and configuration lines in this article. The "Quadlet" part is very simple. Only the configurations required for Nginx, Nextcloud and the creation of a mount point for the hard disk are a little more tricky. But that's not Podman's fault.

Quadlet management is now integrated inside Podman. If you've got Podman, you can create Quadlets 😄

Podman is installed by default on Fedora 41. It works with amd64 and aarm64 CPU. So it works on Raspberry Pi too.

Quadlets are easy to write, easy to maintain, and it helps a lot to manage services as if they are real services.

What is Nextcloud

Nextcloud is a complete cloud service providing file shares, online office tools, messaging with visio-conference, and many others tools. It could be used as NAS as it provides a WebDav sharing service per user.

It works great on Raspberry Pi v4.

The prerequisites

You need to install Fedora 41 aarm64 version on your Raspberry (actually, it should work on others distributions, but I love Fedora).

Because I'm using Nextcloud to store files, I mounted a 2To USB SSD disk on the blue USB port (USB 3). I will show you how to make it available at boot time. If you want to use an internal directory, it's up to you. But, keep in mind that your SD card will not support read/write for a long time. It's recommended to use an external hard drive.

Then you'll need an SSH connection and root access.

Mounting the drive.

It's optional, you'll need this part only if you use an external hard drive.

First, please, format the partition to EXT4 or BTRFS. It's important in order not having problems with labeling and file authorizations. On my side, I only used the “Format drive” option in the file explorer on my personal computer.

Then, plug the drive on your Raspberry Pi (use the blue port if possible, as it is USB 3.0 and is faster).

You should now be able to access the drive. Use dmesg command line to check if the hard drive is detected and the device name (something like /dev/sda or /dev/sdb)

We need to mount it somewhere. I choose /mnt/nas.

Create a directory where to mount the drive :

sudo mkdir /mnt/nas

Then, find the UUID of the partition on your hard drive. (Actually, we can mount the drive with its device path, but it may change later. It's safer to use the partition UUID)

My hard drive is detected as /dev/sda:

sudo lsblk --fs
NAME        FSTYPE FSVER LABEL         UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda
└─sda1      ext4   1.0   NAS           2ce801a3-4911-4d55-ad17-44b1c91d6391    1,7T     0% /mnt/nas
mmcblk1
├─mmcblk1p1 vfat   FAT32               22A3-9756                             562,3M     6% /boot/efi
├─mmcblk1p2 ext4   1.0                 c8e29478-8fe9-49bc-90cd-e816cbe753b8    579M    34% /boot
└─mmcblk1p3 btrfs        fedora_fedora 66ed970b-b91a-483e-9b82-b10fdfc12ea7   51,9G    10% /var/lib/containers/storage/overlay
                                                                                           /home
                                                                                           /
zram0                                                                                      [SWAP]

I keep the UUID and I added this line in /etc/fstab :

UUID=2ce801a3-4911-4d55-ad17-44b1c91d6391 /mnt/nas      ext4    defaults        0 0

Now, typing mount -a I can see that my hard drive is mounted in /mnt/nas:

mount | grep sda1
/dev/sda1 on /mnt/nas type ext4 (rw,relatime,seclabel,stripe=256)

It's OK. Now, each time the Raspberry restart, the external disk is mounted at the right place.

Prepare TLS/SSL certificate

Because I want to use SSL (HTTPS), I will create the certificate. Everything is saved in the SSD hard drive to be able to reinstall everything without losing my mind.

You need to know the hostname you use to contact your Raspberry. In my case, my router applies .home on the network devices. As I set up my Rasbperry to be named "nas", the hostname is "nas.home". Be sure to be able to contact it (ping it)

# change these information
HOSTNAME=nas
DOMAIN=.home
O=Home
COUNTRY=FR
STATE=France
LOC=Laval

# to name your machine, DO NOT INCLUDE DOMAIN
hostnamectl hostname $HOSTNAME

# generate certificates
mkdir -p /mnt/nas/nginx/ssl

# 3650 days = 10 years
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
  -keyout /mnt/nas/ssl/${HOSTNAME}${DOMAIN}.key \
  -out /mnt/nas/ssl/${HOSTNAME}${DOMAIN}.crt \
  -subj "/C=${COUNTRY}/ST=${STATE}/L=${LOCATION}/O=${O}/CN=${HOSTNAME}${DOMAIN}"

The key and the certificate are now created.

You should backup the private key.

Nginx configuration

As we will use Nginx as a container, we need to make a reverse proxy to the container name. As we will name the Nextcloud container "Nextcloud", it's pretty simple. Replace the line I indicate if you have another name to use as server name.

Edit and save it in /mnt/nas/nginx/conf.d/nextcloud.conf:

server {
    listen [::]:443 ssl;
    listen 443 ssl;
    server_name nas.home; # replace this

    ssl_certificate /etc/nginx/ssl/nas.home.crt; # replace this
    ssl_certificate_key /etc/nginx/ssl/nas.home.key; # replace this

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    client_max_body_size 8G;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

    location / {
        # to the "nextcloud" container
        proxy_pass http://nexcloud:80;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_redirect off;
        proxy_buffering off;
    }
}

server {
    listen [::]:80;
    listen 80;
    server_name nas.home; # replace this

    # Redirect HTTP to HTTPS
    return 301 https://$host$request_uri;
}

Create the Quadlets

It's now time to create our services. Instead of installing everything by service, we will use containers.

First, I need a "network". It's only to make all my containers communicating in the same namespace. This way, I don't have to publish unnecessary ports to the host. I only require the nginx ports.

touch /etc/containers/systemd/nextcloud.network

The file can be empty, the default values are OK.

I then create 3 containers, but you can avoid the "valkey" container if you don't want to use it (it's a recommendation, it works without)

The "valkey" Quadlet is defined like this:

# /etc/containers/systemd/valkey.container
[Container]
Image=docker.io/valkey/valkey
ContainerName=valkey
Network=nexcloud.network


[Service]
Restart=always

[Install]
WantedBy=default.target

The Nextcloud Quadlet is defined this way:

[Container]
Image=docker.io/nextcloud:30
ContainerName=nextcloud
Network=nexcloud.network

# where to store data
Volume=/mnt/nas/nextcloud:/var/www/html:z

# wait for valkey
Requires=valkey.container
After=valkey.container

[Service]
Restart=always

[Install]
WantedBy=default.target

Note that the Nextcloud container should start only if Valkey is started.

Then, the nginx container:

[Container]
Image=docker.io/nginx:latest
ContainerName=nginx
Network=nexcloud.network
# export 80 and 443 ports to the network
Port=443:443
Port=80:80
# mount certificates and configuration
Volume=/mnt/nas/nginx/ssl:/etc/nginx/ssl:z
Volume=/mnt/nas/nginx/conf.d:/etc/nginx/conf.d:z

After having created these 3 files, you can reload systemd and start containers :

systemctl daemon-reload
systemctl start valkey
systemctl start nextcloud
systemctl start nginx

The ":z" suffix is required on Red Hat based distribution to set SeLinux labels to allow the write access. Debian based distribution ignores the prefix. So, keep it, there is no problem.

You'll probably need to open firewall ports:

firewall-cmd --add-service=http --permanent
firewall-cmd --add-service=https --permanent
firewall-cmd --reload

Wait! There is something more to do! Do not navigate to Nextcloud yet!

Last configuration

Now that Nextcloud has started, you should have /mnt/nas/nextcloud/config/config.php file. You need to alter some points.

First, force HTTPS and hostname:

  'overwrite.cli.url' => 'https://nas.home', // your FQDN
  'overwritehost' => 'nas.home', // same, your FQDN
  'overwriteprotocol' => 'https',

And, if you want to use Vakey, as I do, add this:

  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array(
    'host' => 'valkey', // this is the container name
    'port' => 6379,
  ),

And now you can navigate to https://nas.home (use your own domain)

Precision

It is possible to not use Nginx container and use the Nginx package. But, on Red Hat based distribution, SELinux will refuse to leave Nginx to proxy pass to a container. If you decided to use nginx package:

change the Nextcloud container to bind 8080:80, or whatever you want
change nginx configuration to point on localhost:8080 instead of nextcloud:80
allow the HTTP connection from nginx to container:

setsebool -P httpd_can_network_connect on

Restart nginx, nginx -s reload and it's OK.

Conclusion

You probably think it's a bit complex, but actually it's not.

Check one more time, the Quadlet section is very simple. The only "hard" topics are the hard drive mount point automation and the Nginx and Nextcloud configuration. Everything that is related to Podman and Quadlet was simple.

Also, I propose Valkey (a Redis fork by Linux Foundation), but you can avoid it.

Quadlet are very easy to use. As soon as you understand the syntax, you'll never use "compose" file anymore for long-term services.

You forget RethinkDB, it's a shame

Patrice Ferlet — Mon, 20 Nov 2023 12:53:32 +0000

There is no only MongoDB... and while you may think of ArangoDB, or CouchDB, you haven't thought of RethinkDB. This project must be saved, urgently!

In the world of database servers, there are many different solutions. PostgreSQL is potentially one of the most reliable solutions, MariaDB has beaten MySQL to the punch, and there are countless projects based on MongoDB.

Document databases have become commonplace.

It's practical, it's quick to set up, and it's easy to manage the schema within the application. As a result, the data is in JSON/BSON form, and is structured as a result.

All's well in the best of worlds.

OK, have you ever tried to quickly:

scale your database, I mean "QUICKLY"?
listen for changes in a table, to use like a bus, providing events to your clients/users?
having a UI where you can see what happens on tables, where are replicas, shards,... ?

Yes, of cours, you can scale, shard, monitor MongoDB, CouchDB. But please go ahead and follow my article.

I've been poking around, testing and breaking database servers for a long time (more than 20 years today). But a few years ago I came across a jewel, the grail, one of the best solutions available. Under the radar, shunned for whatever reason, RethinkDB is nonetheless one of the finest database server projects I've ever tested.

And the pleasure of using this server can be summed up in 3 points:

it's as simple as can be, with a web-based interface for querying, administration, and monitoring
With the "changes API", I can listen to a table and wait for changes. So it can be used as a bus, and I can use server side event with ease in my web applications.
efficient scalability through sharding and replication. Can be configured in no time at all, with no need for in-depth skills.

Wanna test?

Below, you can replace "podman" commands to "docker" of course. I prefer podman while I'm using Fedora, and because it works rootless.

# Create a network. It's important for what happens next.
podman network create rdb

# Start our fist node, bind 8080 to see the 
# web interface
# It's important to name the container, and
# to connect the container to a network. 
# You'll see why...
podman run --rm -it \
   --network rdb \
   --name rdb1 -p 8080:8080 \
   docker.io/rethinkdb

OK, now visit http://localhost:8080 and see the web interface.

The 8080 port is the web admin page port. RethinkDB uses the default 28015 port for query (from your application), and 29015 for clustering.

You can now visit the "Tables" tab to create databases and tables. No scheme to define, of course. At this time, we cannot "replicate" and "shard" the data. But, it will be OK in a few seconds.

Because, we will add a new node in the cluster. Ready?

# Create a new node in the network, but now
# we tell rethinkdb to "join" one node
# (whatever the node!), here "rdb1".
podman run --rm -it \
    --network rdb \
    --name rdb2 \
    docker.io/rethinkdb \
    rethinkdb -j rdb1

That's important, with podman and docker, to be in the same network to be able to communicate by "name" (via name resolution).

And see:

Yes, 2 servers now!

You can now configure your tables to be replicated and/or shard.

Add more and more nodes, you can tell the "rethinkdb" to join any node in the cluster, RethinkDB will do the job for you.

In our example, with Docker or Podman, you only need to ensure that you connect the container to the same network. This to use name resolution. In Kubernetes, you will need to create Statefulset and use headless services. In bare metal and VM, use whatever you want to contact one node.

Let's create a table

Let's create an "app" database, and add a "movies" table:

It's time to create some entries. Go to "data explorer" tab, and type this:

r.db("app").table("movies").insert([
  {name: "Titanic", director: "Cameron"},
  {name: "Citizen Kane", director: "Welles"}
])

I will not give details. But yes you can:

add indexes

add structures

create DB and tables with RQL langage (Javascript like)

and many other things

This inserts 2 new entries. As we gave to insert() an array. Of course you can add one entry.

If you now visit the table view, you can see this:

Nice, isn't it? A monitor of the table. OK, let's replicate the data to be sure that, if one node fails, we can get it back from a second server. Click "Reconfigure" button and do this:

By saying that we now need 2 replicas, RethinkDB will do what it must to do to replicate the data.

And, as you can see now, the documents are distributed in several servers.

Of course, all we did here can be made programatically!

The queries are easy to understand:

The "Changes" API

One of the best features of RethinkDB is the "changes" api. This is a very interesting thing that I use a lot.

You can listen for "changes" in a table, and react to this changes. Then you can create a serverside event handler in you web server, or whatever you want, to send messages to clients. It's very easy to understand:

r.db("app").table("movies").changes()

Each changes provide the "old" and "new" element that changed in the table. And of course, you can filter the changes!

Using with Python, Go, JS, Ruby, ...

RethinkDB officially support Python, JS, Java and Ruby drivers. But there are plenty of drivers for Go, C#, Dart, Perl, PHP, Lua... See the documentation page here

But it's dying by inches (or not)

The developers of this service have gone to work for Stripe. Kudos to them, without any irony, because they richly deserve their success.

The sad thing is that RethinkDB is now only benefiting from a few advances and maintenance operations. It works, the project is maintained at arm's length, certain pull-requests continue to be integrated. But the fear is growing for its users. Because you don't want to lose a tool like this.

Really, we've got to do something to save it.

But what? Of course, sponsorship is the first thing to do. If companies are listening, the message is clear:

you benefit from free software, and so much the better! but help us to offer you continuity of development. Give to the projects you use.

But there's another thing you can do at your own level: use it!

Because a tool can only feel useful if it's used.

I use RethinkDB against all fears of seeing it disappear. I refuse to succumb to the fear of obsolescence that I constantly hear.

RethinkDB is developed in C++ and is completely opensource. There are no major bugs and as a result:

we can therefore say that it doesn't logically need frequent updating
we can be sure that it can be recompiled for future OSes

Clearly, I'm not afraid to say that RethinkDB is production-ready, reliable, usable and future-proof.

So I've got three messages for you

Users: Go for it, use it, thank the developers, go and click on the little star in GitHub to show your encouragement not to let it die in a corner.
Companies, CIOs, CTOs: use it. Test it. Stop falling into the trap of commercial arguments for certain solutions, even opensource ones. RethinkDB works and offers you a service. A service you won't find as complete with other solutions. And for God's sake, donate to the opensource tools you use!
Developers, RethinkDB authors: please don't give up. To date, there's no document database solution that offers such simple scalability, with such a well-designed "changes" API system, and that combines simplicity and efficiency.

Understand how to use C libraries in Go, with CGO

Patrice Ferlet — Mon, 09 Oct 2023 08:50:42 +0000

You probably ever heard about CGO, and that Go can use shared libraries from your system to use the power of C. But how? I will explain the process, and you'll see that it's not that complicated – in fact, it's quite simple.

This tutorial is made for very beginners, but you need to have a bit of knowledge in C programming.

I wrote this tutorial for the simple reason that I find the official documentation, as well as the tutorials I've read, a little austere. For my part, I needed to start from a very simple situation, a real "hello world", and slowly move towards the use of a shared library. So this is my vision, my way of looking at things, which I offer you here.

What are shared libraries?

Shared libraries, also known as dynamic link libraries (DLL) on Windows or shared objects (SO) on Unix-based systems like Linux, are files containing compiled code that multiple programs can use simultaneously. Instead of having the code for a particular task duplicated in every program that needs to perform that task, the code is stored in a shared library. This way, programs can use the functions and procedures from these libraries without having to include the code directly in their files.

And Go can take advantage of this. Like Python or Rust, of course.

What does CGO?

It is a tool that allows calling C functions and using C libraries from Go code. It serves as the bridge between Go and C languages, enabling Go programs to incorporate existing C libraries and leverage existing C codebases.

Actually, it can do this:

Passing Go functionalities to C but this is not what we will see today
Calling C functions directly from a "comment" above the import "C" statement. Or from a .c file inside the project.
Using C libraries to use already compiled functions and types in a shared library.

CGO is provided with Go, but you'll need a C compiler like GCC on Ming (for Windows).

CGO will use the "C" package where all C functions and variables are accessibles. It will then use the C compiler to make the link to your Go project.

Let's try to use C in Go!

First, just to understand what does CGO, we will call a C function that we will create.

Create a project, for example in ~/Projects/testCGO and inside this directory:

go mod init testcgo

Then create a main.go file and write this:

package main

// #include <stdio.h>
// void hello() {
//    printf("Hello from C")
//}
import "C"

func main(){
    // let's call it
    C.hello()
}

In your terminal, type go run . and see the result. Yes, it says "Hello from C".

But, what does CGO here?

Actually, CGO has used the comment above the import "C" statement and it shared the function in the "C" package.

That means that the hello() function, developped in C, is accessible as C.hello() in Go. The "C" package is like a namespace where C variables and function are accessed.

But, we can do this a bit prettier. Coding in comments can be useful if we don't have too many lines of C to do, but when the project becomes substantial, it can quickly become a bit annoying. So, let's use real C source files.

In the same directory, create the hello.c file:

#include <stdio.h>

void hello() {
    printf("Hello from C in another file")
}

And, a hello.h file to declare the function:

void hello();

Then, in your main.go file, replace the content to get this:

package main

// #include "hello.h"
import "C"

func main(){
    // let's call it
    C.hello()
}

This way, we're using the include statement, which is a C inclusion syntax. CGO has no problem:

go run main.go
Hello from C in another file

One more time, CGO takes the comments above the import "C" and because the #include statement is a valid C call, it compiles the C file without any problem.

We've just seen the base. Having C code on one side and a Go project on the other, we now know how to connect the two. But of course, there will be more restrictive things to manage.

How to work with C types?

Let's change the hello.c file to accept an argument and say hello to someone.

#include <stdio.h>

// say hello to the name
void hello(char* name) {
    printf("Hello %s\n", name);
}

Of course, change the header file to decralre the function:

void hello(char*);

Then, change the main.go file to now try to say hello to John:

package main

// #include "hello.h"
import "C"

func main() {
    C.hello("John")
}

This will fail...

./main.go:7:10: cannot use "John" (untyped string constant) as *_Ctype_char value in argument to (_Cfunc_hello)

That's something very important to keep in mind, we need to cast vars from and to Go and C.

Go ease the work with many types, like array, pointers, and strings. When you need to send or receive a variable from or to C, the types are not exactly the same. So we need to "cast" types. But, no panic, after a while you will do it naturally.

So, how to fix this?

We need to modify the Go string to char* type. We can use C.char type but that needs to manually allocate memory. Instead, there is a C.CString type which is a bit easier to use.

func main() {
    name := C.CString("Gopher")
    C.hello(name)
}

And now, it works!

This is the "complex" part of using CGO, you will need to convert, cast, manipulate the variables type to ensure that it will work.
And because it's C, there is no garbage collector for C variables, so you need to free memory when needed (using C.free())

So let's use a C library!

Now that we have seen how CGO can compile C code, let's try to tell it to link shared libraries.

For the example, I will use the very simple "libuuid".

You need to install the devel package of the library to get the header files. On Fedora, that was a simple sudo dnf install libuuid-devel command line.

To be able to generate a UUID, you need to read the documentation of the library (yes... RTFM...) - of course, I already did it and I can explain how to generate a UUID.

// In C:
// we need a uuid_t variable to initalize
uuid uuid_t;
// then we generate the random string. I use the random form
// but you can use other generate methods.
uuid_generate_random(uuid);
// To get a uuid string, we need to "unparse"
char uuid_str[37];
uuid_unparse_lower(uuid, uuid_str);
// In the uuid_str char*, there is a uuid

OK, so let's try.

We will include the uuid/uuid.h file, commonly in /usr/include on Linux, that CGO will find. And let's use the types and functions from this header file.

package main

// #include <uuid/uuid.h>
import "C"
import "fmt"

func main() {
    var uuid C.uuid_t
    var uuid_str *C.char
    uuid_str = (*C.char)(C.malloc(37))
    C.uuid_generate_random(uuid)
    C.uuid_unparse(uuid, uuid_str)
    fmt.Println(C.GoString(uuid_str))
}

This will fail...

The first problem, here, is that the typdedef doesn't work. We need to read the error to understand that, actually, uuid_t is a uchar*. But, not exactly... Actually, reading the header file, you'll see that it's a char[16].

Remember, in C, a char* is like a char[] (I'm grossly oversimplifying here).
But, libuuid declares the uuid_t with 16 chars of size, that means that, using pointer form, we need to allocate the memory with malloc.

So let's change the line to:

var uuid *C.uchar
uuid = (*C.uchar)(C.malloc(16))

You need to know a bit of C to work. Here, what I do is a simple C uuid = (uchar*)malloc(16) transposed with "C" package in Go.

Let's run one more time and...

/usr/lib/golang/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: /tmp/go-link-645695464/000001.o: in function `_cgo_b1eb6bfc450b_Cfunc_uuid_generate_random':
/tmp/go-build/cgo-gcc-prolog:49: undefined reference to `uuid_generate_random'
/usr/bin/ld: /tmp/go-link-645695464/000001.o: in function `_cgo_b1eb6bfc450b_Cfunc_uuid_unparse':
/tmp/go-build/cgo-gcc-prolog:62: undefined reference to `uuid_unparse'
collect2: error: ld returned 1 exit status

That's now the time to use the "linker". Of course, we need to tel the compiler to use the libuuid.so shared library.

Understand the problem. Earlier, we used our own C source files that are compiled with CGO. But, now, we want to use "already" compiled sources to a ".so" library (or .dll for Windows). The header files are there to only provide function declaration (name, arguments and return types).

This is a common thing in C/C++ - that makes compilation very smart and fast, because we don't need to compile the libraries. We only "link" them.

To inform CGO to link the shared library, we can use a specific flags to the compiler. For libuuid it's a simple -luuid (understand -l uuid) to append. This will link libuuid.so to our binary. And Go proposes to specify these arguments inside the comments, as a #cgo: statement.

Above the inclusion of the header file, only append a special instruction:

// #cgo LDFLAGS: -luuid
// #include <uuid/uuid.h>
import "C"

And now it's OK

 go run  .
78137255-35a3-4f61-af7c-e04bf9eb513a

That's it, you have a UUID generated by a C shared library.

The entire source code is:

package main

// #cgo LDFLAGS: -luuid
// #include <uuid/uuid.h>
import "C"
import "fmt"

func main() {
    var uuid *C.uchar
    var uuid_str *C.char
    uuid = (*C.uchar)(C.malloc(16))
    uuid_str = (*C.char)(C.malloc(37))
    C.uuid_generate_random(uuid)
    C.uuid_unparse(uuid, uuid_str)
    fmt.Println(C.GoString(uuid_str))
}

It works very well, but is it practical? No...

Make it better

Calling C functions, with on-the-fly type casting, is impractical, unattractive and not at all easy to maintain.

Using C, the functions are way simpler to use:

uuid_t uuid;
uuid_generate_random(uuid);
char *str = malloc(37); // because 36 chars + \0
uuid_unparse_lower(uuid, str);
// and I can return "str" variable
// that contains a UUID

Then...

What do we really want? A function that gives us a UUID. So we're going to do something very practical:

code a function in C that will make our work easier, and just make sure we have access to it in our Go program.

OK, try:

package main

// #cgo LDFLAGS: -luuid
//
// #include <uuid/uuid.h>
// #include <stdlib.h>
//
// // create a uuid function in C to return a uuid char*
// char* _go_uuid() {
//   uuid_t uuid;
//   uuid_generate_random(uuid);
//   char *str = malloc(37);
//   uuid_unparse_lower(uuid, str);
//   return str;
// }
import "C"
import "fmt"

// uuid generates a UUID using the C shared library.
// It returns a Go string.
func uuid() string {
    return C.GoString(C._go_uuid())
}

func main() {
    // and now it's simple to use
    myuuid := uuid() // this is a go string now
    fmt.Println(myuuid)
}

Of course, we could create the _go_uuid() function in a C source file and create a .h file to declare our function. Then, include go_uuid.h.

What we did here is very common when we want to bind shared libraries to Go. We create some helper functions to cast types and to call C function without asking the user to use the C package by itself.

And this is how https://github.com/go-gst/go-gst, https://github.com/go-gl/glfw, and even https://fyne.io/ are using system libraries to propose a lot of functionalities.

Reminder

So, what you need to keep in mind when you want to use shared libraries:

the "C" package gives access to C functions, types and variables
you can include header files using comments above the import of "C" pacakge"
you can provide LDFLAGS and CFLAGS to the compiler using the #cgo statement in comments
you often need to cast types to Go types, or Go types to C
you can create helpers in comments, in C, to ease the use of the librairies

Conclusion

C isn't the only language that lets you generate shared libraries. You can, of course, generate them in Go, Rust, Python, etc. But to date, it's C that's most widely used to generate libraries.

Having the possibility to use C, or calling C functions from a shared library opens Go to a wide range of powerful functionalities.

Obviously, we prefer libraries developed entirely in Go. This avoids dependence on a library that the user will have to intall on his system, or by sharing this library with him (in the form of .so or .dll). As Go is a language that normally uses static compilation, it's a "bit of a pity" to force the passage. But it's very useful in practice. For example, the very powerful Gstreamer library would be very complicated to recreate entirely in Go. It was created in C and works very well on many platforms. So here, having a dependency on this library is an excellent solution to open streaming sound and video to Go.

You'll need some knowledge of C to be able to link a library in Go. But you don't have to be a specialist. You just need to find the right variable cast, and create a helper function from time to time.

In any case, I hope that my article has opened the way for you, cleared up a few misunderstandings, and that you'll be able to use shared libraries with Go!

How to develop a UI and an API at the same time, with no headaches

Patrice Ferlet — Wed, 08 Mar 2023 10:12:12 +0000

You develop a WEB interface with ParcelJS, Angular, Vue, React... And of course, this interface will have to contact the API that you code in parallel. So you'll spend a little time fighting with CORS headers, having to configure the access point to take into account the production environment...

It's a shame, though. I am a strong supporter of the separation of view and control, and I think that a reverse-proxy is of course important in production. But, you are certainly coding your backend with a language and/or framework that knows how to serve WEB pages and static content.

This article shows the method with ParcelJS, but it will be easy enough to adapt it for any other bundler that allows to do automatic build. The main idea, as you will have understood, is not to use the built-in web server, but to make your backend application do it.

The "bad" workflow

Before seeing the method I prefer to use, let me show you what I often see in many projects. The bad method...

Using ParcelJS, it makes it very easy to start developing an interface. Because parcel builds the application and use a "live reload" system. So, using parcel or parcel serve is simple.

This will make parcel start a WEB service on port 1234, we just have to go to the page https://localhost:1234 and that's it.

But what we are interested in is that our WEB application connects to an API that we develop in Go or Python, for example.

Obviously, the API will listen to another port, and this gets complicated.

Above all, the application should be configured to connect to the port of the API, in the development phase this is different from the production phase
but even if we work locally, it is likely that your navigator prevents access to the API due to the management of "Cross Origin" headers (CORS).

And then, usually, I see project managers to install a reverse proxy, manipulating headers, making the Web UI to be able to connect the API to another port, and of course using a "settings" file that is changed from developmment environment to production.

That's a bit complex, isn't it?

So what can we do? How to ease the developmment phase?

We make our API the web server!

And you'll see that this will resolve both development and production builds. Yeah.

Let's go for a little project

We start by creating a working directory, and we will do this, for example, in Go (we will see later for the Python with Flask)

mkdir -p ~/Project/demo-web1
cd !$
# we create the "ui" here
mkdir -p ui
cd !$
# then create the project
yarn init
yarn add --dev parcel rimraf

Of course, you can add TypeScript, ESLint, and whatever the tools you need.

Then, change the package.json file to set up scripts and the "source" entrypoint + add scripts:

{
  //...
  "source": "src/index.html",
  //...
  "scripts": {
    "start": "rimraf dist .parcel-cache; parcel watch",
    "build": "rimraf dist .parcel-cache; parcel build"
  },
  //...
}

It's important to change the "main" entry point to "source" and to point it on the "index.html" file.

OK, now create src/index.html page:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width" />
    <title>Exemple</title>
    <script src="./main.js" type="module"></script>
  </head>
  <body>
    <h1>Exemple</h1>
    <div id="message"></div>
    <button id="button">Click me</button>
  </body>
</html>

Also, create this script (src/main.js):

// call the /api/demo
function callApi() {
  fetch("/api/demo")
    .then((response) => response.json())
    .then((data) => {
      document.querySelector("#message").innerHTML = data.date;
    });
}

document.addEventListener("DOMContentLoaded", () => {
  document.querySelector("#button").addEventListener("click", callApi);
});

Note that the call with fetch does not change the port, does not provide a particular url, it will be a call made on the same host. Since we will be serving the application via our API

So, at this time, you should have this structure:

└── ui
    ├── node_modules
    ├── package.json
    ├── src
    └── yarn.lock

Did you realize that we don't use parcel or parcel serve but parcel watch ?

We now don't ask parcel to "serve" the interface, we only need to build tha application to dist folder. And we will serve this with our API/Server! That's all!

It's time to create the API and server.

Let's go up a level to demo-web1 – we'll create the API here, but feel free to create a subdirectory if you like. You'll just have to change the path of the static folder.

In short!

# if you're still in the ui directory, go up
cd ..

# create the api
go mod init example

And here, create a main.go file that contains this:

package main

import (
    "fmt"
    "net/http"
    "time"
)

func main() {

    // respond to /api/demo, send the date in json format
    http.HandleFunc("/api/demo", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        t := time.Now()
        w.WriteHeader(http.StatusOK)
        fmt.Fprintf(w, `{"date": "%s"}`, t.Format(time.RFC3339))
    })

    // serve the ui/dist directory as static
    http.Handle("/", http.FileServer(http.Dir("ui/dist")))

    // start the server
    log.Println("Starting server on port 8080")
    http.ListenAndServe(":8080", nil)
}

You should now have this project structure:

├── go.mod
├── main.go
└── ui
    ├── node_modules
    ├── package.json
    ├── src
    │   ├── index.html
    │   └── main.js
    └── yarn.lock

We can now access two endpoints:

"/api/demo" returns the date (yes, it's fast-coded, we could use the encoding/json package, but that's just to show you)
any other path will try to go to ui/dist

ui/dist is where parcel will transpile our web interface - it doesn't exists yet, but it will be there.

OK, so now we can make sure that our UI is completed at each modification, on one side, and we launch the API on the other. To restart the Go application at each change, I often use entr which is very easy to use.

The "entr" command which is a standard package in Linux (install this with dnf or apt) and could be found for Mac.

We will now run two commands:

one that will compile the web interface code continuously, at each change
another that will simply recompile the API

They have to run in parallel, so we will need two terminals for the moment.

# in the first terminal
cd ui && yarn run start

# in the second terminal
find . -name "*.go" | entr -r go run main.go

For those who have tmux on their machine, a nice little trick is to run the following command. It can work perfectly well in a Makefile by the way
tmux new-session \
    "cd ui && yarn run start;" \
    split-window -h \
    "find . -name '*.go' | entr -r go run main.go"

Then go to http://localhost:8080, click on the button, and the date is displayed.

Same with Flask

With Python and Flask (or Quart), you can do the same.

You can, if you want, remove the go.mod and main.go file if you created them earlier.

OK, let's create a virtualenv, and install Flask.

# ensure you're not in "ui" directory then

python -mvenv venv
source venv/bin/activate
pip install flask

That's now OK – time to create our application.

Create the main.py file and type:

from datetime import datetime

from flask import Flask, jsonify

app = Flask(__name__)

# use ui/dist as static
app.static_folder = "ui/dist"


@app.route("/api/demo")
def demo():
    """return the date and hour in json format"""
    date = datetime.now().strftime("%d/%m/%Y %H:%M:%S")
    return jsonify(date)


@app.route("/")
@app.route("/<path:path>")
def index(path="index.html"):
    """Serve static files"""
    return app.send_static_file(path)


if __name__ == "__main__":
    app.run(debug=True)

One more time, start the UI generation, and now the Python service:

# in on terminal
cd ui && yarn run start

# in anther one (ensure you activated the venv before)
python main.py

And go to http://localhost:5000

In debug mode, Flask reloads the server when Python sources changes. No need to use entr.

One more time, any changes in the UI is automatically compiled.

And so, for deployment ?

Because the parcel watch process compiles everythin in dist directory, you will only need to build sources. The application, whatever if you use Go, Python, or any other language, will find the static files.

It's very cool when we want to use Docker (or Podman) in multistage build:

For the Go version, we can use a "scratch" image to only get UI and server binary. No need to get a Linux distribution.

FROM node:alpine as js-build
COPY ./ui /app
RUN set -xe \
    cd /app; \
    yarn install; \
    yarn run build;

FROM golang:alpine as go-build
COPY ./ /app
RUN set -xe; \
    cd /app; \
    go mod tidy; \
    CGO_ENABLED=0 go build -o app.bin *.go

FROM scratch
COPY --from=js-build /app /app/ui
COPY --from=go-build /app/app.bin /app/app.bin
EXPOSE 8080
WORKDIR /app
CMD ["/app/app.bin"]

For Python, we can make approximately the same or use gunicorn to serve the application.

FROM node:alpine as js-build
COPY ./ui /app
RUN set -xe \
    cd /app; \
    yarn install; \
    yarn run build;

FROM python:alpine
COPY ./ /app
COPY --from=js-build /app /app/ui
RUN set -xe; \
    pip install flask gunicorn

EXPOSE 8000
WORKDIR /app
CMD ["gunicorn", "main:app"]

Conclusion

This way of doing things only works if you use a language or framework that is capable of serving static content, and if the WEB interface is coded with a tool that knows how to compile continuously without necessarily having a Web service.

Thus, we eliminate PHP... Because with it, it is necessary, in most cases, to use a Web server (nginx + FPM or Apache).

On the other hand, with Go, Python, Rust, or even Ruby, everything will be fine because these languages are stand-alone.

I have seen, too often, many engineers, both beginners and experienced, launch a bundler in web server mode and struggle with configuration management + CORS in development mode.

The fear of using the API service to provide the pages is still big. Indeed, we can think that it would be good to use a "real" HTTP server to provide the WEB pages later (with a reverse proxy for example)

However, to go fast, for a small project, or simply to propose a first iteration of the application, this method that I propose is more than enough. And it has the merit to be adapted to a reverse-proxy.

Note that it is not forbidden to serve, as I propose in this article, the static pages via the API. In fact, the rule that states that it is imperative to go through a reverse-proxy is a bit outdated, because modern languages and current fameworks are quite good in this exercise.

Don't make me say what I didn't say, an upstream reverse-proxy is of course a good security guarantee. It will allow you to manage the TLS/SSL, the cache, some access rules or the load balancing!

But this reverse-proxy can perfectly call the API to have the static pages, to cache them, instead of reading them directly from the disk. This will also allow you to manage templating or content patching from your API. So, don't forbid yourself to do it.

In short, there is nothing wrong (really, nothing wrong at all) with letting the libraries provide the static pages, if only for a while.

Vue + TS without class component ? No way!

Patrice Ferlet — Thu, 15 Sep 2022 12:47:39 +0000

It is with a huge disappointment that I almost stopped using Vue, because TypeScript support is oriented towards the Composition API, abandoning Class Components. Fortunately, all is not lost!

TL;DR

# create a Vite based project (official way)
npm init vue@3

# install vue-facing-decorator
# cd [to the project], then
npm -i --save vue-facing-decorator

<template>
    <h2>{{ title }}</h2>
    <span>{{ message }}</span>
    <button @click="onStuff">Click me</button>
</template>

<script>
// use the modul to import Component and Vue
import { Component, Prop, Vue } from "vue-facing-decorator";

// and you can now use class component in Vue3 / Vite projects
@Component
export default class MyComponent extends Vue {
    // Property from the component tag
    @Prop message!: String

    // public methods and properties can be 
    // acccessed from the template too
    title = "Test"
    onStuff() {
        // example of handler...
    }
}
</script>

No need of "hooks", it's pure TS.

I love TS, and I love Vue

I develop, among other things, Web applications with mainly 3 frameworks. One of which leaves me speechless:

Vue
Angular
React (I don't like...)

You notice that I did not suffix the names with "JS", because I use mostly TypeScript.

I use TypeScript for a reason, I like a component to be defined as a class, to have properties and methods, and to use the principles of encapsulation and variable scope. So, in essence, I like OOP, and I'm absolutely a fan of having type checking within TypeScript. It makes the code much more secure.

Previously, using vue-cli to create my projects, I was able to activate TypeScript and "class component" syntax. I was so happy.

So, Vue with TypeScript was closed to what I can do in Angular:

<script>
// with vue-class-component

@Options
export default class MyComponent extends Vue {
    myData = "foo"
    myMehtod() {
         // code here...
    }
}
</script>

Everything was fine until this thunderclap...

Today, vue-cli is deprecated to create a new project. The new way is to use Vite and it does'nt provides vue-class-component module and there is no option to force the "old way".

And adding the module fails if the project is Vite based.

Vue(TS) is abandoning vue-class-component, preferring to ask developers to use the Composition API.
See https://github.com/vuejs/vue-class-component/issues/569

And, if you're not sure that Vue doesn't like class (exactly like React, and it's a shame in my opinion), read this:
https://vuejs.org/guide/extras/composition-api-faq.html#will-options-api-be-deprecated

So... Here is the "new appreciated way to do" so far:

<script setup lang="ts">
// now...
import { reactive, ref } from "vue";

const myData = ref("foo") // or raeactive()

function myMethod() {
    // code here
}
</script>

What's the problem dude?

You probably think it's cool. It's easier to write the component logic for those who used to work in JS.

And, be honnest, it's now closed to what we can see in React.

And I don't like React. I've got many reasons that I will expose in a future article. But the substance is this:

We lose the notion that a component is an "object". We also need to call functions to declare that a variable is reactive, and we need a "JS" syntax to create computed functions for example.

That's not pretty.

To illustrate what I mean:

// This is somehting I prefer

@Component
class Example extends Vue {
    private msg = "Hello";
    title = "The title";

    aMethod(){
        // this method can be called in my template
    }

    // make a computed "message" variable
    set message(val: String) {
        this.msg = val.trim();
    }
    get message():String {
        return this.msg;
    }
}

// This is what to do in a "setup" tag
import { computed, ref } from "vue";

const msg = "Hello";
const title = ref("The Title");

// make a computed "message" variable
const message = computed({ 
    set(val: String) => {msg = val.trim()},
    get():String => this.msg
});

Of course, there is less of code lines. But the readability is bad (my opinion, of course).

The good paradigm I have in mind is that a component is an object that I can instantiate. So, it must be a class.

If I want a poor support of TypeScript, using hooks or functions to declare typed objects, reactive (or ref for primitives) or computed variables, so I can use React... And, as said earlier, I don't like React.

The solution!

By chance, I'm not the only one who loves to work with TypeScript using the strongness of the langage.

There is a fantastic module named vue-facing-decorator here: https://facing-dev.github.io/vue-facing-decorator/

facing-dev / vue-facing-decorator

Vue typescript class component decorators

Read me

Designed for vue 3, do the same work like vue-class-component and vue-property-decorator.

Community desired vue class component with typescript decorators.
Compatible with both stage3 and stage2 decorators.
Compatible with both TypeScript and JavaScript projects.
Safety. Transform ES class to vue option api according to specifications.
Performance. Once transform on project loading, ready for everywhere.
Support ES class inheritance, vue extends and vue mixins.
Official recommended.

Welcome to suggest and contribute.

Donate

opencollective

Document

To document

Document languages: English, 简体中文, Portuguese

Wellknown issues

https://facing-dev.github.io/vue-facing-decorator/#/en/wellknown-issues/wellknown-issues

Discussion

To discord https://discord.gg/4exxtFgkcz

QQ群号 766350254

Awesome List

vuex-facing-decorator

Binding helpers for Vuex and vue-facing-decorator.

View on GitHub

And it's very easy to use.

Create a project with TypeScript support:

npm init vue@3
# ==> and select "TypeScript" support in the wizzard

Then, inside the project, install the module:

npm i --save vue-facing-decorator

In the tsconfig.json file, there is a section named compilerOptions, only add this:

{
    "compilerOptions": {
        "experimentalDecorators": true
    }
}

And now, you can import Component decorator, and Vue base class from vue-facing-decorator. Easy 😄

How to use?

It's pretty simple. Create a component like this:

// example, TodoItem.vue
<template>
   <li> <input type="checkbox" v-model="done" /> {{ title }}</li>
</template>

<script lang="ts">
import { Component, Vue } from "vue-facing-decorator";

@Component
export default class TodoItem extends Vue {
    done = false
    title = ""
}
</script>

Of course, you can use properties, for example to send a "Todo" object in the <TodoItem :todo="todo" /> component:

// example, TodoItem.vue
<template>
    <li> 
        <input 
            type="checkbox"
            v-model="todo.done" />
        {{ todo.title }}
    </li>
</template>

<script lang="ts">
import Todo from "@/classes/todo";
import { Component, Vue } from "vue-facing-decorator";

@Component
export default class TodoItem extends Vue {
    @Prop todo!: Todo;
}
</script>

Computed are simple getters and setters (with get and set keywords), methods are simple methods, ...

You can see the documentation to see how to use the others decorators (like @Watch for example) and the options that you can pass to them.

Conclusion

I have absolutely nothing against the fact that you may like the Vue Composition API. It's just a way of coding that doesn't suit me. I have a clear preference for Angular precisely because this Framework uses TypeScript in the most beautiful way.

Vue, with vue-class-component brought me the best of both worlds. Beautiful TypeScript, with the ease inherent to Vue. The abandonment of this module almost drove me away.

But the vue-facing-decorator module makes up for it in a big way. And I thank the authors of this module.

Note: There is also another project that provides class component at setup stage. I'm not satisfied, but maybe you can appreciate: https://github.com/fmfe/vue-class-setup

Create Nginx extensions in JavaScript

Patrice Ferlet — Sat, 23 Jul 2022 06:29:37 +0000

If you need to adapt content, authorize, filter, change the behavior of Nginx, so njs can be the solution. Nginx proposes a JavaScript backend to manipulate requests and responses and even streams. This is njs and it is very powerful.

Nginx is a very good HTTP server and reverse proxy. It is simple to configure, easy to start, and it "does the job". We often use it as a proxy to backends. But, sometimes, you can feel some limitations. For example if you need to authorize users to several backends with a specific identity provider with a weird API. Or, sometimes, this is the backend which is problematic.

For example, I serve a Rancher backend, and Rancher provides its kubeconfig YAML file (dynamically generated for each user) with a certificate inside that I need to remove.

In short, I need to "filter" the content, because I cannot change the behavior of the backend.

And this is exactly when njs can be used!

What I could do is simply to detect that the user claims the KubeConfig file, remove the certificate entry in the YAML, and serve the response file. This is one of the vast variety of manipulation that you can do with njs.

What it njs?

njs is a JavaScript engine integrated to Nginx. njs provides a different approach that starts a JS VM on each needed process. It actually can use common JavaScript modules (like fs for example), and it's ES6 compliant. That means that you will code the scripts without the need of changing your habits.

What you can do

There are plenty of things that you can do with njs:

Make authorization (with or without backend)
Manipulate the output content, headers, status...
Interact with streams
Use cryptography
And so on...

You must keep in mind that njs is not intend to create an application. Nginx is not an application server, it's a web server and a reverse proxy. So, it will not substitute a "real" web framework. But, it will help to fix some things that are hard to do.

Read this before!

Go to the documentation page here to check the global variables which are already defined by Nginx/njs, you'll see that there are many methods to trace, crypt, decode, or manipulate the requests.

Do not spend too much of time to read the page, but just take a look to be aware of the possibilities.

It's not activated by default

njs is not activated by default. It's a module that you need to load at startup.

The legacy method to activate it is to add load_module modules/ngx_http_js_module.so; on top of nginx.conf file.

For docker, you can change the "command" to force the module to be loaded:

docker run --rm -it nginx:alpine \
nginx -g "daemon off; load_module modules/ngx_http_js_module.so;"

Then in your http services, you can now load a script and call functions for different situation.

Prepare the tests

To follow this article, and be able to test, we will start a Nginx service and a "Ghost" blog engine.

This will help to startup:

mkdir -p ~/Projects/nginx-tests
cd ~/Projects/nginx-tests
mkdir -p nginx/conf.d
touch nginx/conf.d/default.conf

cat << EOF | tee docker-compose.yaml
version: "3"
services:
  # our nginx reverse proxy
  # with njs activated
  http:
    image: nginx:alpine
    command: nginx -g "daemon off; load_module modules/ngx_http_js_module.so;"
    volumes:
      - ./nginx/conf.d:/etc/nginx/conf.d:z
    ports:
      - 80:80

  # a backend
  ghost:
    image: ghost
    depends_on:
      - http
    environment:
      url: http://example.localhost
EOF

Debug

Sometimes, your script fails to return somthing, you've got an error 500 and nothing is clearly displayd in the nginx logs.

What you can do is to add this in the "location":

error_log /var/log/nginx/error.log debug;

For docker/podman users, use:

# see the logs in real time
docker-compose logs -f http

And you can use in your script :

r.log("something here")
r.log(njs.dump(variable))

When you finally found the problems, you can then remove the "debug" suffix of the "error_log" directive.

There are many ways, many options, many situations

Nginx manages 2 types of configuration: http and stream. By chance, njs can be used for both.

In this article, we will only speak about the "http" mode.

When we work with "http" mode, we can tell to Nginx to use JavaScript to manipulate:

The content (that means that we will generate the entire request before the call)
The headers only (after request)
The body only (after request)
Set variables
...

The pipeline is not hard: import a javascript file, then use a Nginx directive to use a function for a situation. Nothing more.

First example, create a content

A first, let's create a file named example.js. To make it easy to test, put this in nginx/conf.d/example.js – in production environment, it's better to change the location of your scripts.

OK, so, this is the content:

function information(r) {
    r.return(200, JSON.stringify({
        "hello": "world",
    }));
}

// make the function(s) available
export default {information}

The r variable (argument of the function) is a "request" object provided by Nginx. There are many methods and properties that we can use, here we only use r.return().

It's now time to make the JavaScript function to be called as the content maker. In nginx/conf.d/default.conf, append this:

js_import /etc/nginx/conf.d/example.js;

server {
    listen 80;
    server_name example.localhost;
    location / {
        js_content example.information;
    }
}

Yes, that's all. We import the script, and we use the function as js_content. That means that Nginx will release the request to the script, and the script can yield the content.

Start (or restart) the container and hit the "example.localhost" domain:

$ docker-compose up -d
$ curl example.localhost
{"hello": "world"}

That's the first example. Here, we only generate a JSON output.

We can now do some nice things, like replacing the content.

Replacing the content

njs proposes several fetch APIs to get, sub-request or internally redirect the request.

In this example, we will replace the "/about/" page content by inserting a message inside.

The following configuration is not perfect (actually we could match location /about and call our JavaScript, but I want to show you the internal redirection) — but it will show you some cool stuffs that njs allows.

Change the nginx/conf.d/default.conf file with this:

js_import /etc/nginx/conf.d/example.js;

upstream ghost_backend {
    server ghost:2368;
}

server {
    listen 80;
    server_name example.localhost;
    gunzip on;

    # call our js for /exemple url
    location = /exemple {
        js_content example.information;
    }

    # match everything
    location / {
        js_content example.replaceAboutPage;
    }

    # call the "ghost blog" backend
    # note the "arobase" that creates a named location
    location @ghost_backend {
        # important !
        subrequest_output_buffer_size 16k;

        proxy_pass http://ghost_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off;
    }
}

In this file, you have to pay attention on this:

We force "gunzip" to uncompress proxied responses. This is important if the backend forces gzip responses, and you want to make replacements.
We also make the "subrequest output buffer" size to 16k, you can set it to 128k if needed (for large CSS files if you subrequest them for example).

This is important because we will make subrequests later, and the buffer will be filled too fast

Then, in the example.js file, add the replaceAboutPage function and export it :

function information(r) {
  r.return(
    200,
    JSON.stringify({
      hello: "world",
    })
  );
}

async function replaceAboutPage(r) {
  if (r.uri == "/about/") {
    r.headersOut["content-type"] = "text/html";
    r.return(200, "Changed");
  } else {
    r.internalRedirect("@ghost_backend");
  }
}

export default { information, replaceAboutPage };

Take a minute to read the replaceAboutPage function. In this example, we only:

return a page with "Changed" inside if the URI is "/about"
either we use @ghost_backend location to proxy the blog

Restart the nginx container:

docker-compose restart http

And visit http://example.locahost. Then go to the "/about" page using the "About" link on top.

Nice, so now, we can do a better replacement.

We will need to "subrequest" the page. But a subrequest needs a "real URI path". So, let's add a location in default.conf first.

# a reversed uri.
# We remove the prefix and use the @ghost_backend
location /__reversed {
    internal;
    rewrite ^/__reversed/(.*)$ /$1 break;
    try_files $uri @ghost_backend;
}

Then, let's go to example.js and replace the replaceAboutPage function to this one:

//...
async function replaceAboutPage(r) {
  if (r.uri == "/about/") {
    r.subrequest(`/__reversed${r.uri}`) // call the reversed url
      .then((res) => {
        // copy the response headersOut
        Object.keys(res.headersOut).forEach((key) => {
          r.headersOut[key] = res.headersOut[key];
        });
        // replace the end of "header" tag to append a title
        const responseBuffer = res.responseBuffer
          .toString()
          .replace("</header>", "</header><h1>Reversed</h1>");
        r.return(res.status, responseBuffer);
      })
      .catch((err) => {
        r.log(err);
        r.return(500, "Internal Server Error");
      });
  } else {
    // in any other case
    r.internalRedirect("@ghost_backend");
  }
}
//...

One more time, restart http container:

docker-compose restart http

Then visit http://example.locahost/about/ – you should see:

Second method, use `js_body_filter`

A probably better solution is to use js_body_filter instead of js_content. This leaves Nginx makes the proxy pass, then we can manipulate the body.

So, let's change the default.conf file to this:

js_import /etc/nginx/conf.d/example.js;

upstream ghost_backend {
    server ghost:2368;
}

server {
    listen 80;
    server_name example.localhost;
    gunzip on;

    # call our js for /exemple url
    location = /exemple {
        js_content example.information;
    }

    # call the "ghost blog" backend
    location / {
        js_body_filter example.replaceAboutPage;
        subrequest_output_buffer_size 16k;
        proxy_pass http://ghost_backend;
        proxy_set_header Host $host;
        proxy_set_header Accept-Encoding "";
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off;
    }
}

Very important, here we force the Accept-Encoding to be empty, because Ghost will return a gzipped content that is impossible (at this time) to decompress from javascript.

Then change the JavaScript replaceAboutPage function to this:

async function replaceAboutPage(r, data, flags) {
  if (r.uri == "/about/") {
    r.sendBuffer(data.toString().replace("</header>", "</header>Reversed"), flags);
  } else {
    r.sendBuffer(data, flags);
  }
}

r is the request
data is the data to send, here it's the content taken from the Ghost backend
flags is an object with "last" flags set to true or false

The function needs to use sendBuffer() to send the data to the client.

Of course there are many others things to do, like changing the "Content-Length" header, but this works.

It's very efficient and that's a good method to make some replacement, content checks or fixes to a response without the need to make a subrequest.

Make a `fetch` to outside

njs provides a global ngx variable. This object proposes the fetch API which is more or less the same as you can find in modern browsers.

Let's add a method to call the Dev.to API.

Please, do not abuse the API, it's a simple example, and you are pleased to not overload the servers

The following function will get the list of my articles.

async function getMetal3D(r) {
  ngx
    .fetch("https://dev.to/api/articles?username=metal3d", {
      headers: {
        "User-Agent": "Nginx",
      },
    })
    .then((res) => res.json())
    .then((response) => {
      const titles = response.map((article) => article.title);
      r.headersOut["Content-Type"] = "application/json";
      r.return(200, JSON.stringify(titles));
    })
    .catch((error) => {
      r.log(error);
    });
}

// don't forget to export functions
export default { information, replaceAboutPage, getMetal3D };

OK, now you think that you'll only need to add a js_content call inside the default.conf file. But... there will be some problems.

ngx object need a "resolver" to know how to resolve the domain name
also, you need to define where it can find certificate authorities

But, that's not so hard to do:

location /metal3d {
    resolver 9.9.9.9;
    js_fetch_trusted_certificate /etc/ssl/certs/ca-certificates.crt;
    js_content example.getMetal3D;
}

So, one more time, restart the http container and call the /metal3d URI:

docker-compose restart http

curl http://example.localhost/metal3d

# response:
[
    "Python, the usefulness of \"dataclass\"",
    "Fixing a Promise return object to be typed in Typescript",
    "Flask / Quart - manage module loading and splitting",
    "Change local container http(s) reverse proxy to Pathwae"
]

Conclusion

Nginx proposes a very useful JavaScript extension system. It's easy to realize the large possibilities that can be made with it.

It's possible to create a custom cache system, to manipulate headers, reading token, validate authorizations, change the content, create a proxy on API, and many others things.

We only checked how to manipulate content, but it's also possible to manipulate "streams" with events.

Go to the documentation pages:

NJS doc: https://nginx.org/en/docs/njs/
JS object that you can use: https://nginx.org/en/docs/njs/reference.html
Module directive:
- http https://nginx.org/en/docs/http/ngx_http_js_module.html
- stream https://nginx.org/en/docs/stream/ngx_stream_js_module.html
Examples on GitHub: https://github.com/nginx/njs-examples/

Python, the usefulness of "dataclass"

Patrice Ferlet — Mon, 04 Jul 2022 11:29:26 +0000

When you need to describe a "data", you commonly use classes (instead of dict) – and Python offers a very elegant way to manage this.

A Data Objects represents data which can be saved inside a database. This concept is in the heart of SQLAlchemy, but as the name should be obvious: it's for SQL Database (in general). Today, there are now document databases too (like MongoDB, ArangoDB, RethinkDB that I love so much, or even PostgreSQL). So, a "data" is like a "structured and typed document" that you save "as is". That's not the same paradigm, not the same controls. There are advantages and disadvantages, but we won't debate that here.

The topic, today, is that Python can help a lot to define our "data classes" in a "generic" way with controls and initialization.

At this time, you generally use a "pure plain object"… this will change.

For example, to describe a "user":

class User:
    username = ""
    email = ""
    password = ""
    level = 0

Of course, you need a bit more controls. You require a constructor to initialize the properties, maybe a setter to make the password and to avoid it to be represented in JSON...

And there comes "dataclass", a very cool built-in package in Python.

Dataclasses?

If you read the corresponding documentation page, you'll discover a treasure. This package offers easy to use, but powerful, decorator and functions to manage a "data object" by defining a class.

A dataclass defines constructor, operator, and managed fields if they have got annotations. Either the property is only for internal use. This helps a lot to manage what is part of the data, and what is not.

Let me show you the very basic way to use it, using annotations instead of values:

from dataclasses import dataclass

@dataclass
class User:
    username: str
    email: str
    password: str
    level: int
    example = "foo" # This is not a field, no annotation

Now, the class has got __init__ function to create the object with keywords argument, a __repr__() method, and a __eq__ method that overrides the equal operator.

There are more things to see later, but let's check the usage :

user1 = User(
    username="John",
    email="me@test.com",
    password="foobar",
    level=0,
)
user2 = User(
    username="John",
    email="me2@test.com",
    password="foobar",
    level=0,
)

# show a nice object representation
print(user1)

# try comparison
print(user1 == user2)  # False, email differs

OK, that's nice, but we can do more… a lot more!

Dataclasses fields

Let's imagine we want to create a user without setting its level, because we define that the level should be "0" by default.

The issue is that the __init__ constructor defines it. So at this time we must provide a value when we build the object.

The dataclasses packages provides a function named field that will help a lot to ease the development.

from dataclasses import dataclass, field

@dataclass
class User:
    username: str
    email: str
    password: str
    level: int = field(default=0) # set field as optional

# test
user1 = User(
    username="joe",
    email="me@test.com",
    password="123456") # no level provided

# but the level is set to 0
print(user1.level)
# >> 0

And that's not the end. We can do a lot of things.

Not always need for constructor, use "post init"

Sometimes you want to make something when an object is instantiated. So, the first reaction is to create a constructor. But, of course, here, the dataclass decorator provides one and it's well made to manage default values.

That's why you can create a __post_init__ method. It is called right after the constructor.

For example, let's make a check on the password length.

""" User management """
from dataclasses import dataclass, field


@dataclass
class User:
    """A user object"""

    username: str
    email: str
    password: str
    level: int = field(default=0)  # set field as optional

    def __post_init__(self):
        if len(self.password.strip()) < 8:
            raise ValueError("Password must be at least 8 characters long")

That's enough to make some validation.

Of course, you can manage this with setter or getter, but I only show you an example.

When dataclass becomes central

You may think that it is only a gadget, a "too simple management" of class that represents data.

Now, it's time to see when simplicity provides controls.

I will present how this may help you to create an API with Quart and a bit of Quart Schema. If you already use Flask, that will not be a problem as Quart is a "fork" which only make it asynchronous.

Before the use of this, you probably do something like this:


@api.route("/user")
async def user():
    user = User(
        username="John",
        email="me@test.com",
        password="foo"
    )
    return jsonify({
        "username": user.username,
        "email": user.email,
        "level": user.level,
    })

Of course, you probably created methods to transform the data to JSON, or to dict. But now, with dataclass + quart-schema, it's way more explicit.

First, you must declare that the application is encapsulated to a Schema:

from quart import Quart
from quart_schema import QuartSchema, validate_response

api = Quart(__name__)
QuartSchema(api)

Then, you are able to return an object, no need to jsonify or to manage transformation!

@api.route("/user")
async def user() -> User:
    return User(
        username="John",
        email="me@test.com",
        password="foo"
    )

That works:

http -b :5000/user
{
    "email": "me@test.com",
    "level": 0,
    "password": "12345678",
    "username": "John"
}

OK, but… The password…

Hide fields

The problem here is of course that the password is sent to the response.

Of course, you will hash the password in database and you must never send back the password

There are plenty of possibilities, but I will propose you one that I prefer.

In "my" design view, there are several kind of data to manage:

a user, that is the representation of what I can show to everybody
a data user, that is the representation of what I manage in database
some specific user representation for "login" or "registration" process

So, here is my example:

@dataclass
class User:
    """User data"""

    username: str
    email: str
    level: int = field(default=0, kw_only=True)


@dataclass
class DataUser(User):
    """Data class for User in database"""

    password: str = field(kw_only=True)

    def hash_password(self):
        """Hash password with SHA1"""
        self.password = sha1(self.password.encode()).hexdigest()

The DataUser class inherits fields from User. We must force kw_only to ensure that field with default values doesn't interfere with derived class fields.

@app.route("/user/<email>")
async def get_user(email: str) -> User:
    """Get a user"""
    user = db("users").table("users").get(email)
    del user["password"]
    return User(**user)

I want to insist: I'm SURE that the password will never be sent back to the response, because the User() construction will raise an exception if I provide the password in argument.

And to save a user:

@app.route("/user", methods=["POST"])
async def create_user():
    """Create a new user"""
    sent = await request.json
    user = DataUser(**sent)
    user.hash_password()

    res = db("app").table("users").insert(asdict(user))
    # check errors... then

    del res["password"]
    return User(**res)

asdict() is taken from the dataclasses package, it builds a complete dictionary from your dataclass. So it's easy to use with a document database like Mongo or RethinkDB.

Note that using validate_request and validate_response from Quart Schema simplies a lot the method. For example:

@app.route("/user", methods=["POST"])
@validate_request(DataUser)
async def create_user(data: DataUser) -> User:
    """Create a new user"""

    data.hash_password()
    res = db("app").table("users").insert(asdict(data))

    # check errors... then
    del res["password"]
    return User(**res)

Last words

Anyway, what I hope you to understand is the interest of using dataclass and the fields, field, asdict and other functions to make your data structures easy to use and to manage.