Forem: vladkens

From iTerm To WezTerm

vladkens — Fri, 10 Jan 2025 04:53:27 +0000

For many years, I used iTerm2 as my main terminal emulator and probably spent hundreds of hours in it. Overall, I was satisfied with it, despite some strange recent updates like adding AI features, KeyChain integration, and security vulnerabilities.

In recent years, new terminal emulators have appeared. I tried using them mainly for testing macmon. A couple of years ago, I tried switching to kitty, which was faster due to GPU acceleration. However, it required too much customization and still looked very non-native for macOS. GPU acceleration was added to iTerm as well, so I stayed with it.

I also tried Alacritty, but it is very basic and lacks the features I need. Recently, I tested Ghostty, which has gained huge attention – it has nice defaults, but its RAM usage is concerning (around 250MB per empty tab). Currently, it also lacks buffer search, which makes the terminal useless for me.

I heard a lot of positive feedback about WezTerm from bobuk. So I decided to try it out and use it for some time – and I am very satisfied with it.

The next part of the article will be about how to make WezTerm understandable for iTerm users to try it or switch completely.

Initial

WezTerm can be installed from Homebrew:

brew install --cask wezterm

When you first open WezTerm, it looks plain, non-native, and not very functional. But actually, it has everything you need. This is because WezTerm is cross-platform, unlike iTerm2, so the author left minimal default settings, leaving the rest of the configuration to the user. My goal is to achieve a similar experience in WezTerm as I had in iTerm.

WezTerm does not have a graphical interface for configuration, and the setup is done through a Lua file (this is unusual for me, but Vim users are familiar with it). The configuration file can be located at ~/.wezterm.lua or ~/.config/wezterm/wezterm.lua. I prefer the second path because it is where Fish and starship store configs too.

Create a configuration file:

mkdir -p ~/.config/wezterm/ && touch ~/.config/wezterm/wezterm.lua

And then add the boilerplate code for configuration:

local wezterm = require "wezterm"
local config = wezterm.config_builder()
local action = wezterm.action

-- (here will be added actual configuration)

return config

Appearance

First, I want to change the default font, its size, and disable ligatures (because it's important to see the actual characters in case of some logs).

config.font = wezterm.font {
  family = 'JetBrains Mono',
  weight = 'Medium',
  harfbuzz_features = { 'calt=0', 'clig=0', 'liga=0' }, -- disable ligatures
}
config.font_size = 14.0
config.line_height = 1.0

Save the file, and the changes should be applied immediately (yes, no reload required!).

Next, I want to change the theme and set it to follow the current system theme automatically. For the past year, I have been using Catppuccin themes, and I am very happy with them. They rarely have issues where some text colors blend with the background. To make the theme switch automatically based on the current system theme, we need to write a small Lua function (I copied it from the documentation):

local function scheme_for_appearance(appearance)
  if appearance:find "Dark" then
    return "Catppuccin Macchiato"
  else
    return "Catppuccin Latte"
  end
end

config.color_scheme = scheme_for_appearance(wezterm.gui.get_appearance())

I forgot to mention that the Catppuccin theme, like many other popular themes, is pre-installed in WezTerm, so you don't need to install it separately like in iTerm, which is very cool. Of course, you can also set up your own colors.

The default padding also seems a bit large, so let's make it smaller. I also had a Blinking Line Cursor (I don't remember if this is the default or not in iTerm). Let's set both configurations:

config.window_padding = { left = '0.5cell', right = '0.5cell', top = '0.5cell', bottom = '0.5cell' }
config.default_cursor_style = 'BlinkingBar'

Window style

In iTerm, I used the Minimal theme to save vertical space for tabs. We can do the same in WezTerm (the default tabs don't look very good). Also possible to add some transparency and blur to the window. I didn't use this in iTerm, but let's try it now:

config.window_decorations = 'RESIZE|INTEGRATED_BUTTONS'
config.window_background_opacity = 0.96
config.macos_window_background_blur = 20

Maximize on start-up

In iTerm, I usually set large values for Cols/Rows to make the start window bigger. This approach is a bit outdated, so let's make the window open in full screen when the terminal starts.

-- https://github.com/wez/wezterm/issues/3299#issuecomment-2145712082
wezterm.on("gui-startup", function(cmd)
  local active = wezterm.gui.screens().active
    local tab, pane, window = wezterm.mux.spawn_window(cmd or {})
  window:gui_window():set_position(active.x, active.y)
  window:gui_window():set_inner_size(active.width, active.height)
end)

Key bindings

Finally, the most important part is setting up hotkeys – it's hard to give up what you've been used to for years. An important thing for me is Split Panels with Cmd + D / Cmd + Shift + D. Cmd + W should close the current active panel, not the whole tab. Cmd + K – clear the current screen. Cmd + ← / Cmd + → for start/end of the line. I think these are all defaults from iTerm, so let's make them the same in WezTerm:

config.keys = {
  { key = 'd', mods = 'CMD|SHIFT', action = action.SplitVertical { domain = 'CurrentPaneDomain' } },
  { key = 'd', mods = 'CMD', action = action.SplitHorizontal { domain = 'CurrentPaneDomain' } },
  { key = 'k', mods = 'CMD', action = action.ClearScrollback 'ScrollbackAndViewport' },
  { key = 'w', mods = 'CMD', action = action.CloseCurrentPane { confirm = false } },
  { key = 'w', mods = 'CMD|SHIFT', action = action.CloseCurrentTab { confirm = false } },
  { key = 'LeftArrow', mods = 'CMD', action = action.SendKey { key = 'Home' } },
  { key = 'RightArrow', mods = 'CMD', action = action.SendKey { key = 'End' } },
  { key = 'p', mods = 'CMD|SHIFT', action = action.ActivateCommandPalette },
}

WezTerm also has a Command Execution menu. I haven't used it much yet, but just in case, I bind it to the same keys as in VSCode / Zed – Cmd + Shift + P.

Note: In the configuration, the key value is case-sensitive, so { key = 'd', ... } and { key = 'D', ... } are different things (basically key='D' is the same as key='d', mods='SHIFT'). I initially got confused with this.

Fin

Overall, I am very satisfied with WezTerm and have fully switched to using it daily. One cool feature I want to mention is the quick connect to servers in Known hosts via SSH (Command Palette → type server name → Enter OR Nav Bar → Shell → Click on server name). WezTerm also has Workspaces – an alternative to tmux, but I haven't used it actively yet.

Here a gist with the final config from this article.

2024 In Review

vladkens — Thu, 02 Jan 2025 01:08:37 +0000

I have never written an annual report on my work before, but since I also spend time on open source projects, I decided to write one (maybe it will become a tradition).

Programming Languages and Tools

Rust

The main change in my work was the active use of Rust in the second half of the past year. I mainly write in Rust where I used to write in Go. Overall, the language is more complex than Go and sometimes confusing with the borrow checker, but in general, it is quite convenient to write – I can't say that the speed of writing code significantly decreases compared to other languages – in general, the approaches that need to be used are roughly the same for all languages.

What I like about Rust:

crates ecosystem – many libraries available and can be installed in one command
single binary file and low memory consumption
ability to write applications of various directions (web-app, tui, gui, parsers, etc.)

What I don't like about Rust:

slow Docker image builds (my solution)
a large number of sub-dependencies for web projects (any project on axum will contain 200+ dependencies)
use of C-binding in "core" libraries like rust-tls (ring) – complicates cross-compilation because you have to maintain a C toolchain as well (fixed through Zig build, but it looks monstrous)
lack of finished solutions for serde (default values, partial initialization)

It's nice to write various parsers in Rust, and then they can be coolly parallelized (with rayon or tokio), in Go I had a problems with this due to immutable strings and lack of GC control.

Main languages

The main languages remained the same: Python, NodeJS/TypeScript. I also actively used NextJS on several projects, overall I was satisfied – it's faster and easier to write frontends, but it's quite difficult to find problem's solutions sometimes (many Github issues, often they are open, and the comments contain answers that no longer work).

The main fail of the year in the NodeJS ecosystem is the update of ESLint from 8 -> 9 – completely incompatible API with all existing plugins, the configuration file became larger and more complex – it's not clear why the authors made this diversion. I also tried Binome as an alternative to ESLint / Prettier, but it's completely different and you need to redo a lot, and not everything you need is available at the moment, I continue to monitor this project.

I also tried Zig, but didn't write anything serious. Personally, I didn't understand why this language is needed. At the time I tried it, there was no even package manager yet. Obviously Zig is a competitor to C (not Rust), but C is now much richer in its toolkit and support in other languages. The only thing I can point out is zig build which can build C projects too and does it with less problems.

About LLM

I want to separately note about use of GPTs in work. I have an opportunity to do business tasks in the format of parsing and analyzing texts. What a couple of years ago was done with a bunch of regex, etc., can now be done through GPT and it works great! I tried using local Llama2, but it would have taken weeks to process my data on M2 chip. I redo the solution to use ChatGPT API and processed the data in a couple of days. It's good that all LLMs support openai-client API in Python, so the code almost didn't need to be rewritten. Among the disadvantages of LLama-like aka local running, I also want to note the lack of support for Function calling (to guarantee to bring the LLM response to the desired structure, in my case). The last time I did something serious was more than six months ago, maybe a lot has changed. I also tested gemma2, the inference speed was impressive, and the quality of the answers was not much worse.

As local runners, I used llama.cpp and https://github.com/ml-explore/mlx – the second one uses Mac resources better (checked through macmon), but new models come out a bit slower on it. Some people use ollama, but I didn't understand why, considering that it has an incompatible client with openai-client API and slower performance.

Blog

Initially, I ran my blog only on Medium, but it's not comfortable to write text there – it's much easier to write and edit in the same editor where I write code (plus my editor has Copilot). Therefore, I decided to move my blog to a static page generator. As an engine, I chose Zola (Rust-based, simple, and fast); I also tried Astro, but it's too big and complex for such a simple task. Overall, Zola is good, but it lacks some filters in templating, not enough flexibility in file structure, and not all popular languages are supported in code highlighting.

I left Medium as a fallback for old readers. I also made a cross-post on dev.to, but there are very few views. Medium is disappointed with the lack of the ability to automate cross-posting (they have a deprecated API, but it does not support updating articles and code highlights). Most likely, it's possible to reverse engineer how they save articles through their editor (changes are sent as diffs). Does anyone else need this besides me?

New projects

Last year I launched several new projects (in chronological order):

timewiz.cc (react, webapp) – a tool to find common working hours for distributed teams (an alternative to worldtimebuddy.com, I don't remember exactly why I made my own version). Score: fail.
ecloop (c, cli) – secp256k1 elliptic curve implementation for Apple chips. Mainly an educational project to understand cryptography, SIMD optimizations, etc. Added several standard use cases for finding Bitcoin keys by brute force, but mathematically this is pointless, so as I said earlier, this is more of an interesting educational project (and I wanted to publish a series of articles on cryptography, but didn't have time last year, I hope to do it this year). Some people use this project in hopes of finding something. I also had ideas to add Metal API support – but that takes a lot of time, so I switched to other projects. Score: okay.
macmon (rust, tui) – TUI program for monitoring MacOS performance (CPU/GPU/ANE usage, RAM, temps). My first public Rust project, a lot of reverse engineering of how powermetrics in MacOS works, C-bindings. The project is used both by users in TUI mode and by other projects in headless mode (JSON metrics). Score: success.
ghstats (rust, axum, htmx, webapp) – a small self-hosted service that collects and saves GitHub repo statistics, has several built-in views and an API to retrieve all statistics. The project started well, but then kind of fizzled out in terms of issues and feature requests, most likely users are satisfied with the current functionality. I would like to add more visualizations, but I haven't had time for that yet. The main thing for now is that the statistics are collected and saved, so something can be done with this in the future. Technically, I like that the service consumes only 10 MB of RAM. Score: okay.
ogp (rust, axum, api-service) – a small project for generating OGP articles preview as a service. Used in this blog (you can check by sending this article to your friends in a messenger or social network 😀). Worked with the SVG standard (I thought it was more adequate) and generating PNG from SVG. Score: more of fail.
compose-updater (python, api-service) – a project for updating the version of a docker image from CI in a closed infrastructure. I didn't find anything ready to solve this task. There is watchtower, but it's more about keeping up-to-date docker images, rather than updating a specific one, and also I couldn't run it with ECR repos – the author made a super minimalist image in which I couldn't enter through sh to debug. The second alternative is umputun/updater, but it's more about running different shell commands through a webhook. I chose Python because I didn't find a Rust crate for Docker API (most likely it exists, I just didn't understand which one is really good). Score: – (I didn't promote the project anywhere, I use it for my own purposes for now).

Existing projects

twscrape (python, lib) – a library for parsing data from X/Twitter. Mainly the project is in maintenance mode, I check its functionality every few months, and there have been no requests for new features.
apigen-ts (typescript, lib) – a library for generating TypeScript client OpenAPI spec. I continue to use it in my projects; I also see that there are a small number of other users. My relevance has slightly decreased due to the NextJS/TRPC stack, but in other configurations (e.g. FastApi + React), it is still relevant, so use it. Periodically there are requests to fix something or add a feature.

Abandoned projects

I also have several projects that I didn't finish and abandoned. I'll write about them here, maybe someone needs something from this, then I can return to on it.

anypost – cross-posting between static blog, Medium, Dev.to, Substack, and collecting joined article statistics. I abandoned the project due to the lack of an API at Medium – the only solution I found was through protocol reverse and session cookie.
bcloop – a program to walk Bitcoin-like blockchain and compile the final UTXO list. I abandoned it because I couldn't figure out how to effectively create a HashMap-like structure with external memory support to run on computers with small RAM (less than 32GB, LOL).
tg-watcher – a program that listens to all incoming messages in Telegram (chats & channels) and saves their history in a separate database with the ability to search through these messages later. I have the first version that just saves messages in json files, but I didn't develop the idea further.

Devices

I'm not a big tech geek, but I want to say about a few devices which made my life easier:

MeLE Fanless Quieter 4C – a small fanless PC for homelab, I use it as a native Linux via SSH for testing various programs and keep some cron jobs on it for data collection. It's cool that it's silent because it has no fan, but it heats up a lot under load. The performance of the N100 chip surprised me – only 2x slower than M2 chip (tested with ecloop bench, single core).
Promate HDMI-PD100 – nice cable format to connect a monitor and my Mac. One cable is needed to charge and transmit the video stream. Before that, I used hubs, but it's a big block. I'm glad I see fewer wires.

At Glance

Over the year, 8 articles were written, totaling 5203 views. Top-3 articles:

Table sorting and pagination with HTMX – 1714
macmon – Mac Usage Monitor in Terminal – 953
Setting up WireGuard VPN at AWS with Terraform – 642

Data aggregated from this site (with umami), Medium, and Dev.to. Script.

Public GitHub projects – totaling 312 commits, 49 issues closed, 43 releases created, and 1394 stars gained. Top-3 repos:

twscrape – +802 stars / +52515 views
macmon – +420 stars / +6899 views
ghstats – +128 stars / +2757 views

Data aggregated with ghstats.

I hope I will see a different top with new projects this year 😀

Donations

In twscrape repository I was asked to make it possible to send coffee as donations. Over the year I received 38 cups of coffee from 10 people. I spend these coffees on an annual Parallels subscription, which allows me to test some apps on Windows. Also, I partially paid from this fund for MeLE and VPS, which I used to test apps as well. Many thanks for supporting my work!

Deploy FastAPI application with SQLite on Fly.io

vladkens — Tue, 22 Oct 2024 01:30:21 +0000

Cloud solutions are good for medium and large projects, but too heavy for small personal projects. If you want to launch something small (a few api endpoins and a small repository), there are three options:

Use the same approaches as for "big" projects (AWS ECS/EKS, RDS), but they are redundant, and infrastructure code can be larger than code of the actual project. Also it's expensive (~$100).
Use serverless solutions (Lambda, Vercel). Most cloud providers have such solutions, but these services have difficulties with simple databases – they provide cheap vendor solutions (AWS) or require a managed database, which again is expensive (mostly nothing for serverless, ~$20 for DB)
Use VPS with Docker. It is cheap (~$5 for small machine) and almost no need to manage infrastructure, but deployments sucks (needs private or self-hosted regestry, SSH access from CI).

I usually write my small applications using SQLite, it's a handy little single file database that works in any programming language and can be copied to local machine to analyze data for example. So I was looking for some middleware solution that combines the serveless approach, ease of deployment and ability to use SQLite and found Fly.io.

Setup

If you don't have an account in Fly.io – you need to create it. Also CLI tool called flyctl required to manage projects. Fly.io can be deployed both locally and from CI.

flyctl makes deploy from project's root folder from Dockerfile, which is cool, because same Dockerfile can be used in other systems. For play with Fly.io, I prepared a simple FastAPI project that stores state in database – generic url shortener with click counting.

Dockerfile:

FROM python:3.13-alpine
WORKDIR /app

COPY ./requirements.txt .
RUN pip install --no-cache-dir --upgrade -r requirements.txt

COPY . /app

ENV HOST=0.0.0.0 PORT=8080
EXPOSE ${PORT}
CMD uvicorn main:app --host ${HOST} --port ${PORT}

main.py:

import asyncio
import random
import string
from urllib.parse import urlparse

import aiosqlite
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import RedirectResponse

DB_PATH = "/data/app.db"

app = FastAPI()

async def get_db() -> aiosqlite.Connection:
    if db := getattr(get_db, "_db", None):
        if db.is_alive:
            return db

    db = await aiosqlite.connect(DB_PATH, loop=asyncio.get_event_loop())
    db.row_factory = aiosqlite.Row

    qs = """
    CREATE TABLE IF NOT EXISTS links (
        created_at INTEGER DEFAULT (strftime('%s', 'now')),
        short_code TEXT PRIMARY KEY,
        full_url TEXT NOT NULL,
        clicks INTEGER DEFAULT 0
    )
    """

    await db.execute(qs)
    await db.commit()

    setattr(get_db, "_db", db)
    return db

def random_code(length=8) -> str:
    alphabet = string.ascii_letters + string.digits
    return "".join(random.choice(alphabet) for x in range(length))

def is_valid_url(url: str) -> bool:
    try:
        parts = urlparse(url)
        return all([parts.scheme, parts.netloc])
    except ValueError:
        return False

@app.post("/")
async def shorten(url: str, req: Request):
    if not is_valid_url(url):
        raise HTTPException(status_code=400, detail="Invalid URL")

    host = req.headers.get("host")
    if host is None:
        raise HTTPException(status_code=500, detail="Missing host header")

    short_code = random_code()
    db = await get_db()
    qs = "INSERT INTO links (short_code, full_url) VALUES (?, ?)"
    await db.execute(qs, (short_code, url))
    await db.commit()

    return f"https://{host}/{short_code}"

@app.get("/")
async def list_links():
    db = await get_db()
    qs = "SELECT short_code, full_url, clicks FROM links ORDER BY created_at DESC"
    async with db.execute(qs) as cursor:
        return await cursor.fetchall()

@app.get("/{short_code}")
async def redirect(short_code: str):
    db = await get_db()
    qs = """
    UPDATE links SET clicks = clicks + 1 WHERE short_code = ?
    RETURNING full_url
    """

    async with db.execute(qs, (short_code,)) as cursor:
        if row := await cursor.fetchone():
            return RedirectResponse(row["full_url"])

    raise HTTPException(status_code=404)

requirements.txt:

aiosqlite
fastapi
uvicorn

Deploy

To deploy our code, first we need to create a Fly.io project. This can be done either in the web interface or with flyctl. To create proejct with CLU tool in root folder (where code located) flyctl launch should be runned. This command will offer to select desired hardware and will create fly.toml file:

fly launch --build-only

You can modify project in future by changing parameters in this file or via web ui. The basic fly.toml looks fine, but SQLite requires Storage, which can be created with:

fly volumes create sqlite_data -s 1 -r ams

where -s 1 sets volume size to 1 GB (default is 3 GB), and -r is region in which volume will be created (use same region in which Fly.io project is created). You can always change storage size later.

The last thing to do is to add a mounts section to fly.toml, which attaches the volume to the application:

[mounts]
source = "sqlite_data"
destination = "/data"

sqlite_data is the name of the storage, /data is the path where volume will be connected. This is essentially same as docker run --mount source=sqlite_data,target=/data or corresponding Docker Compose section.

SQLite cannot be writable from more than one app, and Fly.io by default creates 2 instances for an app, so we can specify the number of replicas as one just in case:

fly scale count 1

All configurations are done now and we can deploy our app with command:

fly deploy

The app should boot successfully and the public DNS name will be printed to console. Now we can check it out by posting some url to shortener:

❯ curl -X POST 'https://fly-fastapi-sqlite.fly.dev/?url=https://example.com'
https://fly-fastapi-sqlite.fly.dev/8ydmfAcK

Then we can visit this link, it should redirect to https://example.com. Finally, we can check that clicks are updated:

❯ curl -s 'https://fly-fastapi-sqlite.fly.dev/' | jq
[
  {
    "short_code": "8ydmfAcK",
    "full_url": "https://example.com",
    "clicks": 1
  }
]

To check that database state saved between deployments, we can perform new deployment with fly deploy and check that links list remained same as above (1 link, 1 click).

Migrations

If you are using an external solution for migrations, rather than running them from code when app starts, then only way to run migration is to put it in Dockerfile as part of the RUN command.

Backup

We can connect to machine with fly ssh console and then in /data folder interact with database file. Also we can copy database file to local machine with:

fly ssh sftp get /data/app.db ./app-backup.db

Conclusion

Fly.io is a simple and convenient service for deploying applications. Deploy works from Docker Containers, additional services include PSQL, Redis, S3 like storage (unlike Vercel). It's cheap, the cheapest service costs 3 dollars (1 shared CPU / 256 MB) – maybe even less, if you have little traffic – container shuts down after a few minutes without activity and automatically turns on when traffic appears.

On downside, there is no built-in solution for scheduled tasks – instead, the official solution is to set up a separate server with crontab and run tasks from it – it's kind of creepy.

Setting up WireGuard VPN at AWS with Terraform

vladkens — Sat, 12 Oct 2024 22:32:34 +0000

All resources in AWS work inside private VPC. Sometimes you may need to access these resources from local computer (e.g. to interact with database). Some resources, like RDS, have the option to enable public access to them – but this is unsecure. Of course you can configure Security Group to allow access to public resource only from allowed IPs to make this setup a bit better, but still in this case all your colleagues must to have static IPs, which is not always true.

Objective

We need to provide a quick and easy solution to give access to internal AWS VPC resources for our team. In general, there are two ways to do this:

Bastion / Jump Host – EC2 instance with SSH access for each member
Private VPN – EC2 instance with VPN config for each member

In both cases we need to to setup a custom EC2 instance, so second variant seems more convenient to me.

Preparation

In addition to the traditional Terraform client, we will need WireGuard CLI tools to generate a key pair. You can install WireGuard tools on macOS via brew:

brew install wireguard-tools

Initial Terraform

First, as usual, we need to configure Terraform Provider and get VPC and Public Subnet data where we want to host our VPN server. On this step you can export VPC / Public Subnet from another layer or pass them via variables. I will access my VPC and Public Subnet by name.

terraform {
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.70" }
  }
}

provider "aws" {
  profile = "default"   # Change this to your AWS profile
  region  = "us-east-1" # Change this to your AWS region
}

# MARK: Getting VPC and Subnets

locals {
  vpc_name = "my_vpc" # Change this to yor VPC name
}

data "aws_vpc" "vpc" {
  filter {
    name   = "tag:Name"
    values = [local.vpc_name]
  }
}

data "aws_subnets" "public" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.vpc.id]
  }

  tags = { Tier = "Public" } # This is custom tag
}

WireGuard Server

To setup WireGuard server we need to know AMI ID of OS will be running on EC2 instance, prepare key pairs for the server and for client (peers), create a security group to control network access, create and assing Elastic IP to be sure that the server IP will be same.

Generate key pairs

First we need to create key pairs which can be generated with this command:

bash -c 'priv=$(wg genkey); pub=$(echo $priv | wg pubkey); printf "$priv\n$pub\n"'

In output first line is privkey, second is pubkey.

I will generate it twice, one for server, second for first client (peer). Then for each new peer you will need to generate new key pairs. I will same generated keys to secrets.yaml file, but you can use different ways to store secrets.

Generate secrets for server and first peer (do not use this keys in your setup):

❯ bash -c 'priv=$(wg genkey); pub=$(echo $priv | wg pubkey); printf "$priv\n$pub\n"'
0McsAe5Y/oupwJeju+94ZFl1bmp2KIX2bBQGk0cSWXQ=
KeV+BlZMfCgRhdfaHqpdPp23Nu/otir+6VR02DER7lU=

❯ bash -c 'priv=$(wg genkey); pub=$(echo $priv | wg pubkey); printf "$priv\n$pub\n"'
SK71fHXpTo2cGKvlLLTlu+0AqpwaNWG30rzGXFIryk0=
wf3j1JLhXw/u6iDZf9Qhq0lU34exF8mrHdGzz/u+xFU=

Then save generated keys to secrets.yaml

wg_server:
  privkey: 0McsAe5Y/oupwJeju+94ZFl1bmp2KIX2bBQGk0cSWXQ=
  pubkey: KeV+BlZMfCgRhdfaHqpdPp23Nu/otir+6VR02DER7lU=

wg_peers:
  - name: user1
    addr: 172.16.16.1
    privkey: SK71fHXpTo2cGKvlLLTlu+0AqpwaNWG30rzGXFIryk0=
    pubkey: wf3j1JLhXw/u6iDZf9Qhq0lU34exF8mrHdGzz/u+xFU=

Setup script

Then we need create WireGuard setup script which will be runned during EC2 creation. User data will be generated from this script and populated from secrets.yaml file. So create templates directory with two files in it: wg-init.tpl (general setup script) and wg-peer.tpl (peer info).

#!/bin/bash
sudo apt update && apt -y install net-tools wireguard

# https://github.com/vainkop/terraform-aws-wireguard/blob/master/templates/user-data.txt
sudo mkdir -p /etc/wireguard
sudo cat > /etc/wireguard/wg0.conf <<- EOF
[Interface]
PrivateKey = ${wg_privkey}
ListenPort = ${wg_port}
Address = ${wg_cidr}
PostUp = sysctl -w -q net.ipv4.ip_forward=1
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o ENI -j MASQUERADE
PostDown = sysctl -w -q net.ipv4.ip_forward=0
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o ENI -j MASQUERADE

${wg_peers}
EOF

# Make sure we replace the "ENI" placeholder with the actual network interface name
export ENI=$(ip route get 8.8.8.8 | grep 8.8.8.8 | awk '{print $5}')
sudo sed -i "s/ENI/$ENI/g" /etc/wireguard/wg0.conf

sudo wg-quick up wg0
sudo systemctl enable wg-quick@wg0

[Peer]
# ${peer_name}
PublicKey = ${peer_pubkey}
AllowedIPs = ${peer_addr}

Next we need to prepare user data script in terraform file from this templates.

# MARK: Config

locals {
  wg_cidr = "172.16.16.0/20" # CIDR of WireGuard Server
  wg_port = 51820            # Port of WireGuard Server
  secrets = yamldecode(file("secrets.yaml"))

  userdata = templatefile("templates/wg-init.tpl", {
    wg_cidr    = local.wg_cidr
    wg_port    = local.wg_port
    wg_privkey = local.secrets.wg_server.privkey
    wg_peers = join("\n", [
      for p in local.secrets.wg_peers :
      templatefile("templates/wg-peer.tpl", {
        peer_name   = p.name
        peer_pubkey = p.pubkey
        peer_addr   = p.addr
      })
    ])
  })
}

Server setup

Then we need to select Ubuntu AMI, create Security Group and Elastic IP.

# MARK: Server dependencies

data "aws_ami" "ubuntu" {
  owners      = ["099720109477"] # Canonical
  most_recent = true
  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-arm64-server-*"]
  }
}

resource "aws_eip" "wireguard" {
  tags = { Name = "wireguard" }
}

resource "aws_security_group" "wireguard" {
  name        = "${local.vpc_name}-wireguard"
  description = "SG for WireGuard VPN Server"
  vpc_id      = data.aws_vpc.vpc.id
  ingress {
    from_port   = local.wg_port
    to_port     = local.wg_port
    protocol    = "udp"
    cidr_blocks = ["0.0.0.0/0"]
    description = "Allow WireGuard Traffic"
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Next we can setup a server and assing Elastic IP to it:

resource "aws_instance" "wireguard" {
  count                       = 1
  ami                         = data.aws_ami.ubuntu.id
  instance_type               = "t4g.nano"
  subnet_id                   = data.aws_subnets.public.ids[0]
  vpc_security_group_ids      = [aws_security_group.wireguard.id]
  user_data                   = local.userdata
  user_data_replace_on_change = true
  tags                        = { Name = "wireguard" }
}

resource "aws_eip_association" "wireguard" {
  instance_id   = aws_instance.wireguard[0].id
  allocation_id = aws_eip.wireguard.id
}

Client configuration

To access to WireGuard server we need to have WireGuard client app and valid client config with .conf extension.

We can generated this configs with Terraform Templates as well. So first, lets add new template file called client-conf.tpl:

[Interface]
Address = ${client_addr}/32
PrivateKey = ${client_privkey}
DNS = ${client_dns}

[Peer]
Endpoint = ${server_addr}
PublicKey = ${server_pubkey}
AllowedIPs = ${client_routes}
PersistentKeepalive = 25

And add this code to generate client config to out terraform file:

# MARK: Export client configuration

locals {
  # https://docs.aws.amazon.com/vpc/latest/userguide/AmazonDNS-concepts.html#AmazonDNS
  client_dns = ["169.254.169.253", "1.1.1.1", "1.0.0.1"]
  client_routes = [
    data.aws_vpc.vpc.cidr_block, # All IPs in the VPC
    "169.254.169.253/32",        # AWS DNS
    # "0.0.0.0/32",                # Route all traffic through VPN (uncomment if you want this)
  ]
}

resource "local_file" "peer_conf" {
  for_each = { for index, p in local.secrets.wg_peers : p.name => p }
  filename = "generated/${each.value.name}.conf"
  content = templatefile("templates/client-conf.tpl", {
    server_addr    = "${aws_eip.wireguard.public_ip}:${local.wg_port}",
    server_pubkey  = local.secrets.wg_server.pubkey,
    client_addr    = each.value.addr,
    client_privkey = each.value.privkey,
    client_dns     = join(",", local.client_dns),
    client_routes  = join(",", local.client_routes),
  })
}

Execute

Finally we can run terraform init & terraform apply. In less then a minute VPN server will be created and user client will be exported to generated/user1.conf. Then we can share this file with our teammates.

To check VPN connection working fine you can export .conf into WireGuard client. Then for example I check my RDS instance before VPN enabled and after.

❯ dig +short my-rds.4oexsiqydp8q.us-east-1.rds.amazonaws.com
ec2-34-226-xx-xx.compute-1.amazonaws.com.
34.226.xx.xx

❯ dig +short my-rds.4oexsiqydp8q.us-east-1.rds.amazonaws.com
ec2-34-226-xx-xx.compute-1.amazonaws.com.
10.10.20.80

Adding clients

Adding new client configuration is quite simple. We need to generate new key pair:

❯ bash -c 'priv=$(wg genkey); pub=$(echo $priv | wg pubkey); printf "$priv\n$pub\n"'
ALqgaUiWd0rtcLza2K143x5RwS0Y5zwh3YwHx/L7nV0=
u1yQmh/G7n+S8aCp0PRuRuAuacmqYNzHLCPOcl9aS28=

Then update secrets.yaml file:

wg_peers:
  # ...
  - name: user2
    addr: 172.16.16.2
    privkey: ALqgaUiWd0rtcLza2K143x5RwS0Y5zwh3YwHx/L7nV0=
    pubkey: u1yQmh/G7n+S8aCp0PRuRuAuacmqYNzHLCPOcl9aS28=

And run terraform apply again. Server will be re-created (!) with new configuration. Note that existing clients will be disconnected during deploy.

Summary

WireGuard provides easy solution for creating VPN server. We created cheap and fast setup to access to internal AWS VPC with ability to create different clients configurations, which can be shared with out team members.

PS. Source files of this article can be found here.

Fast multi-arch Docker build for Rust projects

vladkens — Sat, 05 Oct 2024 01:01:58 +0000

Rust allows programs to run very fast and memory efficient, but it has a cost – compile time. In web development, it is standard practice to deliver programs as Docker images and then run them in Kubernetes / Amazon ECS / Docker Compose / etc. After the popularity of ARM processors in recent years, programmers have been faced with the additional step of preparing multi-arch Docker images (meaning that the same image should be able to run natively on both x86-64/amd64 and aarch64/arm64 processors).

Preparing Docker images for interpreted languages is usually not a problem – Docker Buildx is used, which in turn runs inside the emulator and builds the image for each architucture natively. NodeJS / Python just install dependencies via npm / pip, copy the project code, does a little polishing and that's pretty much it. Even in compiled Go this approach works fine because Go has an extensive standard library. In the case of Rust, compiling even a simple web application is a rebuilding of the “universe” – almost any web application requires: HTTP-server, HTTP-client (which in turn requires a library to work with cryptography for https), asynchronous runtime (tokie, etc), serialization/desirialization (serde, serde_json); which in Rust should be installed as external libraries (crates) and should be compiled each time when program builded.

Although Rust compiler has a lot of work to do, it can do it quickly. Even not strongest CI, will build an average project in a couple of minutes. However, this is true only in case of building on native architecture (for example building amd64 binary on an amd64 processor). As soon as we need to build a multi-arch image, we have to do emulation and the compilation speed drops dramatically. For example, on my simple public project – ghstats, building multi-arch image from scratch took about 50 minutes, when same build for native architecture takes 2-3 minutes.

Building time can be optimized by proper usage of Docker layers, so that the step with rebuilding dependencies occurs only when they are actually changed. So only the first build will be long and following builds will be fast. Unfortunately, Rust infrastructure have a problem in this point –
any change in Cargo.toml file (for example version number) will invalidate Docker layer and triggers full rebuild of all dependencies.

Problem definition

So there are two problems with building multi-arch Docker images for Rust projects:

Layer invalidation and full rebuild on any change in Cargo.toml
Very slow multi-arch build due to emulation

Better dependencies building

Easiest way to solve the first problem is to use cargo-chef which was created exacly for it. cargo-chef converts Cargo.toml & Cargo.lock into a special recipe.json file that will remain unchanged until project dependencies are unchanged. Then we can use this json file to cache expensive Docker layer that compiles dependencies.

Dockerfile with cargo-chef will use multi-stage build splited to 5 parts:

Installing cargo-chef and build dependencies (OpenSSL, linux-headers, etc)
Preparing recipe.json file with project dependencies description
Installing & builiding project dependencies from recipe.json
Building whole project
Copy binaries to runtime image and final polishing

Example of Dockerfile with cargo-chef and multi-stage build (I use alpine in my projects, but you can use other base-images).

# (1) installing cargo-chef & build deps
FROM rust:alpine AS chef
WORKDIR /app
RUN cargo install --locked cargo-chef
RUN apk add --no-cache musl-dev openssl-dev

# (2) prepating recipe file
FROM chef AS planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

# (3) building project deps, cache magic happen on COPY command
FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --recipe-path recipe.json --release

# (4) actual project build
COPY . .
RUN cargo build -r

# (5) runtime image, you can use any base image you want
FROM alpine:latest AS runtime
WORKDIR /app
COPY --from=builder /app/target/release/prog /app/prog
CMD "/app/prog"

This approach will speed up Docker image build many times while project dependencies are not changed. Because Docker builds stages separately, planner stage will be executed each time (but cargo-chef is fast!) and builder stage will be partially cached until recipe.json file are same.

If you need single architecture build this will work fine. But if you need multi-arch images, this approach still be slow.

Multi-arch image with cross-compilation

Multi-arch builds using Docker Buildx run very slow due to QEMU emulation. If we get rid of emulation, compilation will be at full speed. Rust has built-in cross-compilation for other architectures, so we adapt it in Docker build.

Cross-compilation in Rust itself works fine, but some crates are based on C libraries (OpenSSL, SQLite, etc). Compiling and linking C code is quite complicated and not always clear (usually you have to look for error codes somewhere on Stack Overflow or Github Issues until you get the right set of compilers/header files added). There is another tool that surprisingly solves the problem of cross-compiling C code very well – Zig (actually this is programming language, but they have build toolchain as well).

To connect Zig build toolchain with Rust I will use cargo-zigbuild crate. I other Docker file looks pretty same to our cargo-chef variant, expect I added second target in Cargo to build and cargo build replaced with cargo zigbuild.

# (1) this stage will be run always on current arch
# zigbuild & Cargo targets added
FROM --platform=$BUILDPLATFORM rust:alpine AS chef
WORKDIR /app
ENV PKG_CONFIG_SYSROOT_DIR=/
RUN apk add --no-cache musl-dev openssl-dev zig
RUN cargo install --locked cargo-zigbuild cargo-chef
RUN rustup target add x86_64-unknown-linux-musl aarch64-unknown-linux-musl

# (2) nothing changed
FROM chef AS planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

# (3) building project deps: need to specify all targets; zigbuild used
FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --recipe-path recipe.json --release --zigbuild \
  --target x86_64-unknown-linux-musl --target aarch64-unknown-linux-musl

# (4) actuall project build for all targets
# binary renamed to easier copy in runtime stage
COPY . .
RUN cargo zigbuild -r \
    --target x86_64-unknown-linux-musl --target aarch64-unknown-linux-musl && \
  mkdir /app/linux && \
  cp target/aarch64-unknown-linux-musl/release/prog /app/linux/arm64 && \
  cp target/x86_64-unknown-linux-musl/release/prog /app/linux/amd64

# (5) this staged will be emulated as was before
# TARGETPLATFORM usage to copy right binary from builder stage
# ARG populated by docker itself
FROM alpine:latest AS runtime
WORKDIR /app
ARG TARGETPLATFORM
COPY --from=builder /app/${TARGETPLATFORM} /app/prog
CMD "/app/prog"

In general that all. Such build approach will work faster. For my project:

initial build 50 min -> 13 min (3.8x)
code updated 7 min -> 3 min (2.3x)

Additional performance can be achieved by moving first step into separate base image, then our main Docker image will not need to build cargo-zibguild and cargo-chef (-2 min from initial build).

apigen-ts – Simple TypeScript API Client Generator

vladkens — Wed, 04 Sep 2024 00:51:48 +0000

OpenAPI / Swagger is a good way to describe API contracts between backend / frontend or different services. Nowadays frontend is almost always TypeScript + some framework (React, Vue, Angular, etc). Backend in turn can be in NodeJS (and then in principle you can share contracts with code reuse), or in another language (e.g. Python / Go / Scala / Rust) and then code reuse and API synchronization becomes a problem.

OpenAPI can solve this problem through codegen – backend should qualitatively describe the available API methods and data types, after that you can use the client generation utility and get ready typed TypeScript code.

OpenAPI has a standard tool for such generation, but it has a number of problems:

Tool is written in Java, which requires special software to be installed on dev computer / CI
Generate client is extensive and consists of many .ts files, which makes it not user friendly (in my opinion)

In JavaScript ecosystem it is common to use npm (or yarn / pnpm – whatever you like more) as a package manager. But the point is the same: packages are written in JavaScript, installed from one place, versioned.

In npm you can find several alternative apigen utilities written in TypeScript which provides a better development experience. I used one of those too (swagger-typescript-api), until they changed interfaces a few times and I had to fix my several projects. I tried to find an alternative in npm, but all available generators (as of late 2023) also have their own problems: generate a lot of files, are difficult to configure, break compatibility between versions, require to install additional packages.

Meet apigen-ts

I have a couple of projects in NestJS / FastAPI in support mode, for which I just need a stable API client generation utility that is simple and won't change often. So I wrote my own TypeScript HTTP API generation package from Swagger specification – apigen-ts.

Originally I wrote apigen-ts for personal projects, but then I share it to my friends and they also started to use apigen-ts in their projects. Now I want to share it with the community, maybe it will be useful for someone too. So library features:

supports Swagger / OpenAPI v2 / v3 / v3.1 (parsed with redocly)
generates a typed TypeScript client
single output file, single api class
exports data models for use in your code
automatic code formatting with prettier (peer dependency)
ability to automatic dates parsing

Usage

Usage is as simple as possible and is divided into two steps: generating a client from specification (local file or remote url; yaml or json), using the generated typed client in the app.

Generation of an arbitrary client can be performed by command:

npx -y apigen-ts https://petstore3.swagger.io/api/v3/openapi.json ./api-client.ts

Or you can install apigen-ts as a dev dependency (npm i -D apigen-ts) and create a task to generate the client in package.json:

{
  "name": "my-frontend",
  "scripts": {
    "apigen": "apigen-ts http://127.0.0.1:3000/docs-json ./api-client.ts --parse-dates && tsc --noEmit"
  }
}

Then use: npm run apigen or yarn apigen.

Generated file contains a client class named ApiClient and TypeScript types used in API endpoints. Next you can use them in app code in the following way:

import { ApiClient, Pet } from "./api-client";

const api = new ApiClient({
  baseUrl: "https://example.com",
  headers: { Authorization: "secret-token" },
});

// GET https://example.com/pet/findByStatus?status=sold
await api.pet.findPetsByStatus({ status: "sold" }); // -> Pet[]

Generator tries to create method names in format like {domain}.{operation}. The types used in Swagger spec can be imported directly into code (in example above Pet is generated type). This approach makes it easy to track changes in contract and fix the application code where types are updated or methods are changed (of course if you describe your Swagger well). Just run new api generation and check frontend code for errors afterwards with TypeScript.

Also apigen-ts works well in CI pipelines, when on new commits to the backend part you run frontend api client generation and check code compatibility.

The generated client uses native fetch API (does not require additional packages) and can be used both in browser and NodeJS environment. If you use prettier in your project, the generated client will be automatically formatted according to project settings.

More configuration options and customization abilities can be in readme.

Conclusion

Backend ↔ frontend (or backend ↔ backend) communication is monotonous and repetitive work from project to project. Do not waste time and effort on manual writing of api clients. Instead use simple tools to automate process and keep code quality.

You can support the project with a star ⭐️ or leave an issue / feature request on Github page.

Table sorting and pagination with HTMX

vladkens — Sat, 24 Aug 2024 16:57:22 +0000

I recently tried HTMX for my new project - ghstats - a dashboard of Github repository traffic in a single interface for longer than 14 days. This project was planned as a self-hosted service, so I was thinking about a really simple and memory-less tech stack. Last time I played with Rust, so I decided to use it instead of NodeJS / Python. Of course, if I'm generating static HTML on the server side, I have two options for implementing table sorting and pagination: use query parameters and do everything on backend, or use some JavaScript to call data from the backend and render the table on the client side. But HTMX offers a new, third way: write all the logic on the backend and replace the necessary parts of HTML with just a few tag attributes. Let's see how it works.

Initial setup

To demonstrate table sorting and pagination, I'll be using the same stack I used for my service: axum as a backend framework and maud for HTML templating (it's a DSL over Rust templates).

Let's create new project with:

cargo init htmx-example && cd-html-example

Then install dependencies:

cargo add tokio axum maud --features tokio/full,maud/axum

and do initial setup in src/main.rs file:

use axum::{response::IntoResponse, routing::get, Router};
use maud::html;

async fn index_page() -> impl IntoResponse {
  html! {
    h1 { "Hello, World!" }
  }
}

#[tokio::main]
async fn main() {
  let service = Router::new().route("/", get(index_page)).into_make_service();
  let listener = tokio::net::TcpListener::bind("127.0.0.1:8080").await.unwrap();
  axum::serve(listener, service).await.unwrap();
}

Then we cant run this with cargo watch -x run and open http://127.0.0.1:8080 in the browser. "Hello, World!" should be displayed.

Creating a static table

Let's add crates to generate random data:

cargo add fake rand rand_chacha --features fake/derive

and define simple Contact struct (like our application is simple CMS).

use fake::faker::company::en::*;
use fake::faker::internet::en::*;
use fake::faker::name::en::*;
use fake::faker::phone_number::en::*;
use fake::{Dummy, Fake, Faker};
use rand::prelude::*;
use rand_chacha::ChaCha8Rng;

#[derive(Debug, Clone, Dummy)]
struct Contact {
  id: u64,

  #[dummy(faker = "Name()")]
  name: String,

  #[dummy(faker = "CompanyName()")]
  company: String,

  #[dummy(faker = "FreeEmail()")]
  email: String,

  #[dummy(faker = "PhoneNumber()")]
  phone: String,
}

fn get_contacts(n: usize) -> Vec<Contact> {
  let mut items = Vec::with_capacity(n);
  let mut rng = ChaCha8Rng::seed_from_u64(42);
  for _ in 0..n {
    items.push(Faker.fake_with_rng(&mut rng));
  }
  items
}

async fn index_page() -> impl IntoResponse {
    let contacts = get_contacts(100);
    println!(">> {:?}", contacts[0]);
    println!(">> {:?}", contacts[1]);

  html! {
    h1 { "Hello, World!" }
  }
}

Now on each request same random contacts will be generated, so we can use this data in our table.
get_contacts function emulates database query in real environment.

I will also add base layout for page with PicoCSS framework to have nice look:

fn base(html: maud::Markup) -> impl IntoResponse {
  html! {
    html {
      head {
        meta charset="utf-8" {}
        title { "Contacts" }

        link rel="stylesheet" href="https://unpkg.com/@picocss/pico@2.0" {}
        script src="https://unpkg.com/htmx.org@2.0" {}
      }
      body class="container" {
        { (html) }
      }
    }
  }
}

And finally update index_page to render contact table:

async fn index_page() -> impl IntoResponse {
  let contacts = get_contacts(100);

  base(html! {
    table {
      thead {
        tr {
          th { "ID" }
          th { "Name" }
          th { "Company" }
          th { "Email" }
          th { "Phone" }
        }
      }
      tbody {
        @for contact in contacts {
          tr {
            td { (contact.id) }
            td { (contact.name) }
            td { (contact.company) }
            td { (contact.email) }
            td { (contact.phone) }
          }
        }
      }
    }
  })
}

We can request again http://127.0.0.1:8080 and see our table with random contacts (which on each request at same).

Sorting table

Now is the time to add sorting to our table. We will use HTMX to send requests to the server and get sorted HTML back. First, we need to add sorting logic to our backend. We will use query parameters to pass sorting column and direction.

To make it easier, I will add struct TableFilter which will represent sorting and pagination parameters, for code simplicity I will parse Request manually rather than using Query extractor and serde crate:

use std::collections::HashMap;
use axum::extract::{Query, Request};

#[derive(Debug)]
struct TableFilter {
  sort: String,
  order: String,
  page: u32,
  per_page: u32,
}

fn parse_table_filter(req: &Request) -> TableFilter {
  let qs: Query<HashMap<String, String>> = Query::try_from_uri(req.uri()).unwrap();
  let sort = qs.get("sort").unwrap_or(&"name".to_string()).to_string();
  let order = qs.get("order").unwrap_or(&"desc".to_string()).to_string();
  let page = qs.get("page").unwrap_or(&"1".to_string()).parse().unwrap();
  let per_page = qs.get("per_page").unwrap_or(&"10".to_string()).parse().unwrap();
  TableFilter { sort, order, page, per_page }
}

Next, we need to create a function to render th with HTMX attributes to handle sorting:


fn th(title: &str, qs: &TableFilter) -> maud::Markup {
  let id = title.to_lowercase();
  let order = if qs.sort == id && qs.order == "desc" { "asc" } else { "desc" };
  let url = format!("/?sort={}&order={}&page=1&per_page={}", id, order, qs.per_page);

  html! {
    th scope="col" style="cursor: pointer;"
      hx-trigger="click"
      hx-get=(url)
      hx-target="#contacts_table"
      hx-swap="outerHTML"
    {
      (title)
      @if qs.sort == id {
        span style="margin-left: 0.5em;" {
          @if qs.order == "asc" { "↑" } @else { "↓" }
        }
      }
    }
  }
}

Update get_contacts to accept TableFilter and sort items / do pagination:


fn get_contacts(n: usize, qs: &TableFilter) -> (Vec<Contact>, u32) {
  let mut items: Vec<Contact> = Vec::with_capacity(n);
  let mut rng = ChaCha8Rng::seed_from_u64(42);
  for _ in 0..n {
    items.push(Faker.fake_with_rng(&mut rng));
  }

  items.sort_by(|a, b| {
    let cmp = match qs.sort.as_str() {
      "id" => a.id.cmp(&b.id),
      "name" => a.name.cmp(&b.name),
      "company" => a.company.cmp(&b.company),
      "email" => a.email.cmp(&b.email),
      "phone" => a.phone.cmp(&b.phone),
      _ => a.id.cmp(&b.id),
    };

    if qs.order == "asc" {
      cmp
    } else {
      cmp.reverse()
    }
  });

  let pages = (items.len() as f64 / qs.per_page as f64).ceil() as u32;
  let range_s = (qs.page as usize - 1) * qs.per_page as usize;
  let range_e = qs.page as usize * qs.per_page as usize;
  let items = items[range_s..range_e].to_vec();

  (items, pages)
}

And finally, we can update our index_page to use TableFilter and th:

async fn index_page(req: Request) -> impl IntoResponse {
  let qs = parse_table_filter(&req);
  let (contacts, pages) = get_contacts(100, &qs);

  base(html! {
    // id added to tell HTMX where to put new data
    table id="contacts_table" {
      thead {
        tr {
          (th("ID", &qs))
          (th("Name", &qs))
          (th("Company", &qs))
          (th("Email", &qs))
          (th("Phone", &qs))
        }
      }
      // same as before
    }
  })
}

That's it! Now we can sort our table by clicking on the column header. HTMX will send a request to the server with sorting parameters and replace the table with the new sorted data. Let's me explain HTMX attributes:

hx-trigger - event which will trigger the request, in our case it's click
hx-get - URL to send the request and HTTP method (can be GET or POST)
hx-target - CSS selector to replace with new data
hx-swap - how to replace the target, in our case it's outerHTML which will replace the whole table (by default it's inserting new data into the target)

All this attributes handled automatically by HTMX, so we just need to include HTMX script in <head>.

Pagination

In previous step we already parse pagination filters to struct, so lets add code to render pagination links. I will use code from my another react project, which originally from StackOverflow answer:

// https://stackoverflow.com/a/70263913
fn calc_pagination(page: usize, total: usize, len: usize) -> Vec<Option<usize>> {
  use std::cmp::{max, min};

  let len = if len == 0 { 5 } else { len };
  let total = max(total, page);
  let start =
    max(1, min(page as isize - ((len - 3) as isize / 2), total as isize - len as isize + 2))
      as usize;
  let end = min(total, max(page + (len - 2) / 2, len - 1));

  let mut result = Vec::new();

  if start > 2 {
    result.push(Some(1));
    result.push(None);
  } else if start > 1 {
    result.push(Some(1));
  }

  for i in start..=end {
    result.push(Some(i));
  }

  if end < total - 1 {
    result.push(None);
    result.push(Some(total));
  } else if end < total {
    result.push(Some(total));
  }

  result
}

Let's define two more helper function which will render pagination link and delimiter (like ...):

fn pagination_link(qs: &TableFilter, i: u32) -> maud::Markup {
  let url = format!("/?sort={}&order={}&page={}&per_page={}", qs.sort, qs.order, i, qs.per_page);
  html! {
    li {
      a href=(url)
        hx-trigger="click"
        hx-get=(url)
        hx-target="#contacts_table"
        hx-swap="outerHTML"
      { (i) }
    }
  }
}

fn pagination_delim() -> maud::Markup {
  html! {
    li {
      span { "..." }
    }
  }
}

And finally we can write pagination function, which will return list of links and delimiters:

fn pagination(pages: u32, qs: &TableFilter) -> maud::Markup {
  let items = calc_pagination(qs.page as usize, pages as usize, 5);

  html!(
    nav {
      ul {
        @for item in items {
          @if let Some(i) = item {
            (pagination_link(qs, i as u32))
          } @else {
            (pagination_delim())
          }
        }
      }
    }
  )
}

async fn index_page(req: Request) -> impl IntoResponse {
    // ...
    base(html! {
        // ..
        // add pagination call after table
        (pagination(pages, &qs))
    })
}

Partial HTML updates

Before this step, we always replaced the whole page with new HTML content. This can be done without HTMX at all, so what the point of using it? It's right, we have very simple layout before. Let's make it more complex to show nice feature of HTMX - partial updates.

First of all let's move table generation into separate function:

fn get_contacts_table(contacts: Vec<Contact>, qs: &TableFilter, pages: u32) -> maud::Markup {
  html! {
    // id moved from table to div
    div id="contacts_table" {
        table {
          thead {
            tr {
              (th("ID", &qs))
              (th("Name", &qs))
              (th("Company", &qs))
              (th("Email", &qs))
              (th("Phone", &qs))
            }
          }
          tbody {
            @for contact in contacts {
              tr {
                td { (contact.id) }
                td { (contact.name) }
                td { (contact.company) }
                td { (contact.email) }
                td { (contact.phone) }
              }
            }
          }
        }

        (pagination(pages, &qs))
    }
  }
}

Now we can call this code in any other place and have contacts table. Now let's add some navigation and other content to our main page:

async fn index_page(req: Request) -> impl IntoResponse {
  let qs = parse_table_filter(req);
  let (contacts, pages) = get_contacts(1000, &qs);

  base(html! {
    nav {
      ul {
        li { a href="/dashboard" { "Dashboard" } }
        li { a href="/" { "Contacts" } }
        li { a href="/settings" { "Settings" } }
      }
    }

    h1 { "Contacts" }
    (get_contacts_table(contacts, &qs, pages))
  })
}

Right now in the application we have a simple navigation. But when we interact with the table, a full page will be returned, and the result will look like a nested page in a nested page... This is wrong. Let's fix this with HTMX (and this is actually where HTMX comes power).

When browser loads the page it load as regular page with default headers. When request send by HTMX it will set extra headers like HX-Request and HX-Target. We can use this headers to detect which part of HTML should be returned. Let's add this code to our index_page:

fn get_hx_target(req: &Request) -> Option<&str> {
  match req.headers().get("hx-target") {
    Some(x) => Some(x.to_str().unwrap_or_default()),
    None => None,
  }
}

async fn index_page(req: Request) -> impl IntoResponse {
  let qs = parse_table_filter(&req);
  let (contacts, pages) = get_contacts(1000, &qs);

  match get_hx_target(&req) {
    Some("contacts_table") => return get_contacts_table(contacts, &qs, pages).into_response(),
    _ => {}
  }

  base(html! {
    // ... this part is the same
  })
  .into_response() // added into_response() to relax rust type inference
}

So in this code we check if hx-target header is present. If not - we return full page. If it's present and equal to contacts_table - we return only table. This is how partial updates works in HTMX. We can split page to smaller parts and update it independently.

Conclusion

HTMX is good utility for simple application which allows to add some interactivity without writing a lot of JavaScript code. It easy to use and can be integrated with any backend language, because not require any special server-side support. It's also very lightweight and can be used in any project without any additional dependencies.

PS. Source files of this article can be found here.