Forem: Odilon HUGONNOT

Linux network tuning: TCP BBR, NIC ring buffers, and SFTP throughput

Odilon HUGONNOT — Fri, 01 May 2026 09:00:02 +0000

The server is on a Gbit link. ethtool confirms 1000 Mbps on the interface. And yet every SFTP transfer caps out somewhere around 800 KB/s. Not 80 MB/s — 800 kilobytes per second. Less than 1% of the theoretical capacity.

The problem isn't bandwidth. It's a stack of bad defaults: a congestion control algorithm designed for year-2000 networks, microscopically small NIC ring buffers, and application socket buffers sized for DSL lines. Here are the five tweaks that fixed it.

TCP BBR: replacing CUBIC

CUBIC has been Linux's default congestion control algorithm since 2006. It works by growing the congestion window (cwnd) until it detects a packet loss, then cutting it in half. On modern low-latency, high-bandwidth networks, this loss-triggered reaction is a problem: a single dropped packet (thermal noise, NIC buffer overflow) causes a 50% throughput drop followed by a slow recovery.

BBR (Bottleneck Bandwidth and Round-trip propagation time), developed by Google in 2016, doesn't react to loss — it models the actual link throughput by continuously measuring bandwidth and minimum RTT. It holds the window at the optimal level without waiting for drops. On a Gbit link under real traffic, the difference is immediately noticeable.

# /etc/sysctl.d/99-bbr.conf
net.ipv4.tcp_congestion_control = bbr        # BBR replaces CUBIC
net.ipv4.tcp_slow_start_after_idle = 0       # keep cwnd between download pauses
net.ipv4.tcp_no_metrics_save = 1             # ignore TCP metrics from previous sessions
net.ipv4.tcp_fastopen = 3                    # TCP Fast Open client+server (-1 RTT on reconnects)
net.ipv4.tcp_mtu_probing = 1                 # MTU probing if ICMP is blocked upstream
net.ipv4.tcp_wmem = 4096 262144 16777216     # TCP write buffer: default 256 KB, max 16 MB
net.core.netdev_max_backlog = 5000           # packet queue before kernel processing at 1 Gbit/s
net.core.netdev_budget = 600                 # packets processed per NAPI cycle

# Apply immediately (persists across reboots via the file)
sysctl -p /etc/sysctl.d/99-bbr.conf

# Verify
sysctl net.ipv4.tcp_congestion_control
# → net.ipv4.tcp_congestion_control = bbr

tcp_slow_start_after_idle deserves special attention for SFTP. By default, Linux resets cwnd to 10 after a period of inactivity on a TCP connection. An interactive SFTP session alternates activity and pauses — every time you start a new transfer after browsing a directory, TCP restarts in slow start. Disabling this avoids that unnecessary reset.

lsmod | grep bbr
# → tcp_bbr               20480  1
# If absent: modprobe tcp_bbr

NIC ring buffers: the main cause of drops

This is probably the highest-impact change, and the least documented one. A NIC ring buffer is a memory region between the network card and the kernel. The NIC deposits received packets there. If the buffer fills before the kernel processes them, new packets are silently dropped.

The default on most Ethernet cards is 256 or 512 entries RX. At 1 Gbit/s with 1500-byte frames, that's ~83,000 packets per second to absorb. A 256-entry buffer can saturate in under a millisecond if the kernel is temporarily busy elsewhere. The result: invisible drops, a collapsing CUBIC cwnd, and stalled throughput.

# Check current state
ethtool -g enp2s0f0
# Ring parameters for enp2s0f0:
# Pre-set maximums:
# RX:     4096
# TX:     4096
# Current hardware settings:
# RX:     256    ← too small
# TX:     256    ← too small

# Apply (takes effect immediately, lost on reboot)
ethtool -G enp2s0f0 rx 4096 tx 4096

# Verify
ethtool -g enp2s0f0
# Current hardware settings:
# RX:     4096
# TX:     4096

To make the setting persistent, add it to /etc/network/interfaces after the interface configuration:

# In /etc/network/interfaces, after dns-search (or at the end of the iface block)
    post-up ethtool -G enp2s0f0 rx 4096 tx 4096

Replace enp2s0f0 with your actual interface name (ip link show to list). Not all NICs support 4096 — ethtool -g shows the maximums under "Pre-set maximums".

# Monitor NIC drops in real time
watch -n1 ethtool -S enp2s0f0 | grep -i drop
# rx_missed_errors and rx_fifo_errors indicate ring buffer drops

ProFTPD and Apache: application-level buffers

Once the kernel and NIC layers are tuned, application buffers become the next bottleneck. ProFTPD and Apache defaults are sized for modest connections — they haven't been meaningfully updated since a time when Gbit/s was reserved for carrier backbones.

ProFTPD — socket buffers

ProFTPD exposes SocketOptions to control socket buffer sizes at the TCP level. Default values (rcvbuf and sndbuf) are typically 87 KB or less. Bumping to 1 MB lets the kernel hold more data in flight without waiting on client ACKs:

# In /etc/proftpd/proftpd.conf, after UseSendFile on
SocketOptions rcvbuf 1048576 sndbuf 1048576

systemctl reload proftpd

Apache — send buffer

Apache has its own SendBufferSize directive that controls the TCP send buffer size for HTTP connections. Without it, Apache uses the system default, often 87 KB. On a Gbit link serving large static files or substantial API responses, this is a real constraint.

# In /etc/apache2/apache2.conf (global level)
SendBufferSize 1048576

apache2ctl configtest && systemctl reload apache2

TLS: session cache and OCSP Stapling

A full TLS handshake costs 1 to 2 extra RTTs. On HTTPS, a client that reconnects frequently — browser reopening connections, transfer tool with reconnects, monitoring — multiplies these RTTs unnecessarily. TLS session cache allows resuming an existing session without a full handshake.

OCSP Stapling solves a different problem: without stapling, the browser must contact the CA's OCSP server to verify certificate validity. That's an additional external round-trip, potentially slow, on every new connection. With stapling, Apache prefetches the OCSP response and includes it in the TLS handshake — the client doesn't need to contact the CA.

# In /etc/apache2/mods-enabled/ssl.conf

# Shared memory session cache: 10 MB (~40,000 simultaneous sessions)
SSLSessionCache         shmcb:${APACHE_RUN_DIR}/ssl_scache(10485760)
SSLSessionCacheTimeout  3600

# OCSP Stapling: Apache prefetches and caches the OCSP response
SSLUseStapling          on
SSLStaplingCache        shmcb:/var/run/apache2/ssl_stapling(2097152)
SSLStaplingReturnResponderErrors off

apache2ctl configtest && systemctl reload apache2

SSLStaplingReturnResponderErrors off matters: without it, if the upstream OCSP server is temporarily unreachable, Apache returns the error to the client instead of falling back gracefully. Not what you want in production.

Results and verification

After applying all these changes, SFTP throughput went from ~800 KB/s to several tens of MB/s on the same Gbit connection. The bottleneck was never the network — it was the accumulation of bad defaults at each layer of the stack.

# BBR active
sysctl net.ipv4.tcp_congestion_control
# → bbr

# NIC ring buffers
ethtool -g enp2s0f0 | grep -A5 "Current hardware"
# → RX: 4096 / TX: 4096

# TCP stats (retransmissions, drops)
ss -s
# → retrans: 0/0 ideally

# SFTP throughput test (1 GB file)
sftp user@server <<< $'put /tmp/testfile /tmp/testfile'

# Verify OCSP Stapling
openssl s_client -connect your-domain.com:443 -status < /dev/null 2>&1 | grep -i "OCSP"
# → OCSP Response Status: successful (0x0)

# Verify TLS session resumption
openssl s_client -connect your-domain.com:443 -reconnect 2>&1 | grep "Reused"
# → Reused, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384

Conclusion

These settings should have been the defaults for the kernel and daemons long ago. BBR has been stable since 2016. A 256-entry ring buffer on a Gbit card makes no sense. An 87 KB socket buffer on a modern connection is folklore.

The technical difficulty is low — each change is a single line. The problem is that these defaults produce no visible error: throughput is just "disappointing", logs show nothing, and you spend hours looking at the application before checking the network stack.

One caveat: adapt the values to your actual topology. tcp_wmem at 16 MB on a machine with 512 MB RAM and 1000 concurrent connections may create other problems. These parameters are calibrated for a dedicated server with few simultaneous connections on a local or datacenter network.

📄 Associated CLAUDE.md

View • Download • Catalogue

Securing a dedicated Linux Debian 12 server — Complete post-incident guide

Odilon HUGONNOT — Thu, 30 Apr 2026 09:00:03 +0000

An ordinary Tuesday morning. I glance at Apache logs before starting work — a conditioned reflex, rarely useful. Except that morning. In error.log, hundreds of identical lines, all from the same IP: 13.37.248.113. HTTP Digest authentication attempts in a loop, combining common usernames with generic passwords.

The server held. HTTP Digest auth with a correct password is a solid barrier against brute force if the password is strong. But the incident still triggered a full audit I'd been putting off for too long.

This server hosts a private seedbox shared with about twenty users, a PHP website behind Apache with HTTP Digest authentication, SFTP access via ProFTPD, a wiki in Docker behind an Apache reverse proxy, and Jellyfin for streaming. A classic setup for a semi-professional personal server. Here is everything that was reviewed, fixed, and automated.

1. Detecting and responding to a brute force attack

Reading Apache logs to identify attempts

Apache's HTTP Digest authentication logs its failures in error.log, not access.log. The codes to look for are AH01790 and AH01794.

# Count failed attempts by IP over the last 24h
grep "AH0179" /var/log/apache2/error.log | grep -oP '\[client \K[0-9.]+' | sort | uniq -c | sort -rn | head -20

# See full lines for a specific IP
grep "13.37.248.113" /var/log/apache2/error.log | tail -30

Log lines look like this:

[Tue Feb 18 07:23:41.412893 2026] [auth_digest:error] [pid 1234] [client 13.37.248.113:58432] AH01790: user admin: password mismatch: /protected/
[Tue Feb 18 07:23:41.718204 2026] [auth_digest:error] [pid 1234] [client 13.37.248.113:58433] AH01794: user root in realm "Private Area" not found: /protected/
[Tue Feb 18 07:23:42.091337 2026] [auth_digest:error] [pid 1234] [client 13.37.248.113:58434] AH01790: user administrator: password mismatch: /protected/

Three distinct patterns in these logs: password mismatch (known user, wrong password), not found (non-existent user), and sometimes nonce mismatch (replay of expired challenge). The attacker was clearly testing a list of generic login/password pairs.

Identifying the attacker

# Quick whois
whois 13.37.248.113

# Geolocation without installing anything
curl -s https://ipinfo.io/13.37.248.113/json

Result: IP belonging to Amazon AWS eu-west-3 (Paris). Classic. Cheap AWS VPS instances are used en masse for this kind of operation because they're easy to create, difficult to trace back to a real person, and often poorly monitored. The IP has since been reported on AbuseIPDB with around sixty reports.

Verifying no intrusion occurred

Before doing anything else, verify that nothing actually got in. The brute force may have found something before it was noticed.

# Successful SSH connections (look for unexpected logins)
grep "Accepted" /var/log/auth.log | tail -50

# Failed SSH attempts from the same IP
grep "13.37.248.113" /var/log/auth.log

# SFTP activity (ProFTPD log)
grep "13.37.248.113" /var/log/proftpd/proftpd.log 2>/dev/null

# Check timestamps on sensitive files
stat /etc/passwd /etc/shadow /etc/sudoers
ls -la /root/.ssh/
ls -la /home/

# Look for files modified recently in /etc (last 24h)
find /etc -newer /tmp/ref_file -ls 2>/dev/null
# Create the reference file first: touch -d "24 hours ago" /tmp/ref_file

In this specific case, nothing. The attack was limited to HTTP Digest, without touching SSH. But that doesn't change the need to close the weak points that could have been exploited.

2. fail2ban — Protection against brute force

fail2ban analyzes logs and bans IPs that exceed an attempt threshold. The default config is a good starting point, but it has significant gaps for this specific setup.

apt install fail2ban
systemctl enable fail2ban

All customizations go in /etc/fail2ban/jail.local (never modify jail.conf directly — it will be overwritten during updates).

SSH jail

# /etc/fail2ban/jail.local
[DEFAULT]
bantime  = 86400    ; 24h
findtime = 600      ; detection window: 10 minutes
maxretry = 5
banaction = iptables-multiport

[sshd]
enabled  = true
port     = ssh
logpath  = %(sshd_log)s
backend  = %(sshd_backend)s
maxretry = 5

apache-auth jail — the default filter trap

fail2ban includes a default apache-auth filter. It does not detect HTTP Digest authentication errors from Apache 2.4. The default filter looks for patterns like Authorization Required or Basic Auth errors — not the AH01790 / AH01794 from the auth_digest module.

A custom filter is required:

# /etc/fail2ban/filter.d/apache-auth.local
[Definition]
failregex = \[client <HOST>:.*\] AH01790: user .+: password mismatch
            \[client <HOST>:.*\] AH01794: user .+ in realm .+ not found
            \[client <HOST>:.*\] AH01788: .+nonce from .+ received on .+ - not found

Then the corresponding jail in jail.local:

[apache-auth]
enabled  = true
filter   = apache-auth
port     = http,https
logpath  = /var/log/apache2/error.log
maxretry = 8
bantime  = 86400
findtime = 300

To test the filter before enabling it in production:

# Test the filter against a real log excerpt
fail2ban-regex /var/log/apache2/error.log /etc/fail2ban/filter.d/apache-auth.local --print-all-matched

ProFTPD jail

[proftpd]
enabled  = true
port     = ftp,ftp-data,ftps,ftps-data
logpath  = /var/log/proftpd/proftpd.log
maxretry = 6
bantime  = 86400

Checking jail status

# Overview
fail2ban-client status

# Details of a specific jail
fail2ban-client status apache-auth
fail2ban-client status sshd

# Manually unban an IP (if you ban yourself)
fail2ban-client set sshd unbanip 1.2.3.4

# fail2ban logs
tail -f /var/log/fail2ban.log

3. SSH/SFTP — Restricting access

About twenty users have SFTP access to upload and retrieve files. None of them need a full SSH shell. That's unnecessary attack surface.

sftponly group and ChrootDirectory

# Create the group
groupadd sftponly

# Add existing users
usermod -aG sftponly alice
usermod -aG sftponly bob
# etc.

In /etc/ssh/sshd_config, add this block at the end (it must come after any existing Match directive):

Match Group sftponly
    ForceCommand internal-sftp -l INFO -f AUTH
    ChrootDirectory %h
    AllowTcpForwarding no
    X11Forwarding no
    PermitTunnel no
    AllowAgentForwarding no

The chroot trap. The ChrootDirectory directive imposes a severe and counter-intuitive constraint: the chroot root directory must belong to root:root with 755 permissions. If it's the user's home directory and they own it, SSH refuses the connection silently — the user just sees a connection error with no explanation on the client side.

# Correct structure for an SFTP chroot
# The home must be root:root 755
ls -la /home/alice/
# drwxr-xr-x  3 root  root  4096 ...

# Subdirectories belong to the user
ls -la /home/alice/downloads/
# drwxr-xr-x  2 alice alice 4096 ...

# Fix permissions if needed
chown root:root /home/alice
chmod 755 /home/alice

# The user must still be able to write somewhere
mkdir -p /home/alice/uploads
chown alice:alice /home/alice/uploads
chmod 755 /home/alice/uploads

Detailed SFTP logging

By default, SFTP transfers are not logged in a useful way. Enable logging in the Subsystem directive:

# In /etc/ssh/sshd_config
# Replace the existing Subsystem line with:
Subsystem sftp internal-sftp -l INFO -f AUTH

Transfers then appear in /var/log/auth.log with the format sftp-server[PID]: open "/path/file.txt" flags READ mode 0666.

Disable root SSH login

# In /etc/ssh/sshd_config
PermitRootLogin prohibit-password

# Verify no unknown key exists
cat /root/.ssh/authorized_keys
# If the file contains keys you don't recognize: security incident

prohibit-password (formerly without-password) forbids password login but allows SSH keys. It's safer than no if you need emergency access via key.

# Reload SSH after modification
sshd -t && systemctl reload sshd

4. Permissions and sensitive files

After verifying nothing was modified, this is the opportunity to put permissions into a correct state once and for all.

Credential files

# htdigest files — readable by root and www-data only
chown root:www-data /etc/apache2/.htdigest
chmod 640 /etc/apache2/.htdigest

# Configuration files with passwords
chown root:www-data /var/www/html/config.php
chmod 640 /var/www/html/config.php

# Normal user home directories
chmod 700 /home/alice
chown alice:alice /home/alice

# Exception: home dirs of chrooted SFTP users → root:root 755 (see section 3)

Block web access to sensitive directories

The natural temptation is to use <Location> with Require all denied. Problem: in Apache 2.4 with HTTP Digest authentication configured at the parent level, Require directives can interact unexpectedly with inherited auth. In some configurations, access is simply challenged with a Digest prompt instead of being denied.

RewriteRule is more reliable because it runs in Apache's processing pipeline before authentication:

# In .htaccess or the vhost
RewriteEngine On

# Block direct access to internal data
RewriteRule ^/data/internal - [F,L]
RewriteRule ^/uploads/private - [F,L]

# Block config files accidentally exposed
RewriteRule \.(env|log|sql|bak)$ - [F,L]

The [F] flag returns a 403 immediately. [L] stops processing further rules.

5. Sudoers — Principle of least privilege

By default on Debian, the installation often creates a user in the sudo group. In a context where the server is shared and exposes services, keeping accounts with full sudo is unnecessary risk.

# List who has sudo
getent group sudo

# Remove sudo from a non-root account
deluser adminuser sudo

# Verify
sudo -l -U adminuser
# "User adminuser is not allowed to run sudo"

If the admin needs elevation, su is sufficient — they know the root password. No need for sudo on a personal server.

www-data and scripts with privileges

If PHP scripts need to execute system commands with elevated privileges (reload Apache, run a maintenance script), never put www-data in sudo globally. Create a file in /etc/sudoers.d/ with only what is strictly necessary:

# /etc/sudoers.d/www-data
# Always use absolute paths
www-data ALL=(ALL) NOPASSWD: /usr/sbin/apachectl graceful
www-data ALL=(ALL) NOPASSWD: /usr/local/bin/maintenance-script.sh

# Correct permissions on this file
chmod 440 /etc/sudoers.d/www-data

# Check syntax before saving
visudo -c -f /etc/sudoers.d/www-data

Scripts called by www-data must validate their inputs. If a parameter is passed from PHP, use escapeshellarg() and validate the format before calling shell_exec():

<?php
// Validate the parameter before using it
$user = $_POST['username'] ?? '';
if (!preg_match('/^[a-z][a-z0-9_]{2,31}$/', $user)) {
    die('Invalid username format');
}

$escaped = escapeshellarg($user);
$output = shell_exec("sudo /usr/local/bin/create-user-dir.sh $escaped");
?>

6. Audit and intrusion detection (auditd)

fail2ban reacts. auditd observes and records. The two are complementary.

apt install auditd audispd-plugins
systemctl enable auditd

Create the rules file /etc/audit/rules.d/security.rules:

# /etc/audit/rules.d/security.rules

# sudoers modifications
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers

# SSH configuration
-w /etc/ssh/sshd_config -p wa -k sshd_config

# Root SSH keys
-w /root/.ssh/authorized_keys -p wa -k root_keys

# System accounts
-w /etc/passwd -p wa -k accounts
-w /etc/shadow -p wa -k accounts
-w /etc/group -p wa -k accounts

# su and sudo execution
-w /usr/bin/su -p x -k su_exec
-w /usr/bin/sudo -p x -k sudo_exec

# Crontab
-w /etc/crontab -p wa -k crontab
-w /etc/cron.d/ -p wa -k crontab
-w /var/spool/cron/ -p wa -k crontab

# fail2ban configuration
-w /etc/fail2ban/ -p wa -k fail2ban

# Sensitive application configuration files
-w /var/www/html/config.php -p rwa -k app_config
-w /etc/apache2/ -p wa -k apache_config

# Load rules without restarting
augenrules --load

# Verify active rules
auditctl -l

# Consult events (last 24h)
ausearch -ts yesterday -te now | aureport -f -i | head -50

Process accounting with acct

acct records every command executed by every user, with timestamp and duration. Less verbose than auditd, but excellent for a post-incident audit: "what did user X do yesterday between 2pm and 3pm?"

apt install acct
accton on  # Enable recording

# See recent commands by user
lastcomm --user alice | head -30

# Recent root commands
lastcomm --user root | head -50

# Filter by specific command
lastcomm --command bash

rkhunter — Rootkit scanner

apt install rkhunter

# Initialize the baseline (do this immediately after a clean installation)
rkhunter --update
rkhunter --propupd

# Full scan
rkhunter --check --skip-keypress

# After a system update, regenerate the baseline
apt upgrade && rkhunter --propupd

rkhunter will generate false positives at first — legitimate files it doesn't recognize. Go through the warnings once to qualify them. Real rootkits don't hide in plain sight (if the system is already compromised, rkhunter will likely be bypassed), but the tool is useful for detecting unexpected modifications to system binaries.

7. Automated security audit with AI analysis

The problem with monitoring logs on an exposed server: the background noise is enormous. Hundreds of SSH attempts per day from Chinese and Russian IPs is the norm. Alerting on all of them by email would mean never reading your email.

The solution put in place: collect the raw report every night, pass this report to Claude via CLI to distinguish background noise from real incidents, and only send an email if something warrants attention.

Collection script

#!/bin/bash
# /root/scripts/security-audit.sh
# Runs every night via cron: 3 0 * * * /root/scripts/security-audit.sh

set -euo pipefail

REPORT_DIR="/var/log/security-audit"
TODAY=$(date +%Y-%m-%d)
REPORT="$REPORT_DIR/$TODAY.txt"

mkdir -p "$REPORT_DIR"
chmod 700 "$REPORT_DIR"

{
    echo "=== SECURITY REPORT - $TODAY ==="
    echo "Generated on: $(date)"
    echo ""

    echo "=== AUDITD EVENTS (24h) ==="
    ausearch -ts yesterday -te now 2>/dev/null | aureport -f -i 2>/dev/null | tail -100 || echo "auditd: no events"
    echo ""

    echo "=== FAIL2BAN - ACTIVE BANS ==="
    for jail in sshd apache-auth proftpd; do
        echo "--- $jail ---"
        fail2ban-client status "$jail" 2>/dev/null || echo "jail $jail not active"
    done
    echo ""

    echo "=== SSH - FAILED ATTEMPTS (24h) ==="
    grep "Failed password" /var/log/auth.log | grep "$(date +%b)" | tail -50 || echo "none"
    echo ""

    echo "=== SSH - SUCCESSFUL CONNECTIONS (24h) ==="
    grep "Accepted" /var/log/auth.log | grep "$(date +%b)" || echo "none"
    echo ""

    echo "=== LISTENING PORTS (check) ==="
    ss -tlnp
    echo ""

    echo "=== ESTABLISHED OUTBOUND CONNECTIONS ==="
    ss -tnp state established | grep -v "127.0.0.1" | grep -v "::1" | head -30
    echo ""

    # rkhunter scan only on Sundays (day 7)
    if [ "$(date +%u)" -eq 7 ]; then
        echo "=== RKHUNTER SCAN (weekly) ==="
        rkhunter --check --skip-keypress --quiet 2>&1 | tail -30 || echo "rkhunter: error"
        echo ""
    fi

    echo "=== END OF REPORT ==="
} > "$REPORT" 2>&1

chmod 600 "$REPORT"

# Run AI analysis separately
/root/scripts/security-analyze.sh "$REPORT"

AI analysis script

#!/bin/bash
# /root/scripts/security-analyze.sh
# Receives the report path as argument

set -euo pipefail

REPORT="${1:-}"
ADMIN_EMAIL="admin@example.com"
TODAY=$(date +%Y-%m-%d)
ANALYSIS_TIMEOUT=60

if [ -z "$REPORT" ] || [ ! -f "$REPORT" ]; then
    echo "Usage: $0 /path/to/report.txt" >&2
    exit 1
fi

# Prompt for Claude — ask for JSON only to make parsing easy
PROMPT="Analyze this Linux server security report. Distinguish real incidents from normal events (SSH attempts from random IPs are typical background noise). A real incident would be: a successful SSH connection from an unknown IP, a sensitive file change in auditd, an unexpected open port, or an abnormally high volume of attempts on a specific service. Reply ONLY with valid JSON, no markdown, no explanation: {\"alert\": true/false, \"summary\": \"2-3 sentence summary\", \"details\": [\"point1\", \"point2\"]}"

# Claude call with timeout
AI_RESPONSE=""
AI_ERROR=0

if command -v claude >/dev/null 2>&1; then
    AI_RESPONSE=$(timeout "$ANALYSIS_TIMEOUT" bash -c "cat '$REPORT' | claude --print --model claude-haiku-4-5 '$PROMPT'" 2>/dev/null) || AI_ERROR=1
else
    AI_ERROR=1
fi

# Safe fallback: if AI is unavailable, send raw report
# We don't miss an incident because the API was down
if [ "$AI_ERROR" -eq 1 ] || [ -z "$AI_RESPONSE" ]; then
    mail -s "[SECURITY] Report $TODAY - AI analysis unavailable" "$ADMIN_EMAIL" < "$REPORT"
    exit 0
fi

# Parse the response JSON
ALERT=$(echo "$AI_RESPONSE" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('alert', False))" 2>/dev/null || echo "parse_error")

if [ "$ALERT" = "parse_error" ]; then
    # Invalid JSON → fallback raw report
    mail -s "[SECURITY] Report $TODAY - invalid AI response" "$ADMIN_EMAIL" < "$REPORT"
    exit 0
fi

if [ "$ALERT" = "True" ]; then
    SUMMARY=$(echo "$AI_RESPONSE" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('summary', 'N/A'))" 2>/dev/null || echo "N/A")

    {
        echo "AI Analysis: $SUMMARY"
        echo ""
        echo "--- Full report ---"
        cat "$REPORT"
    } | mail -s "[SECURITY ALERT] $TODAY - Incident detected" "$ADMIN_EMAIL"
fi

# If alert=False: nothing. The report is archived in $REPORT_DIR for manual review.

# Make executable and schedule
chmod 700 /root/scripts/security-audit.sh
chmod 700 /root/scripts/security-analyze.sh

# Root crontab
crontab -e
# Add:
# 0 3 * * * /root/scripts/security-audit.sh

The advantage of the systematic fallback: if Claude API is down, times out, or returns invalid JSON, the raw report is sent anyway. You don't risk missing a real incident because the third-party analysis service was unavailable that night.

8. PHP security

Secure sessions

By default, PHP sessions are not configured to resist cookie theft. Add to php.ini or at the start of every script that uses sessions:

<?php
// Configure before session_start()
ini_set('session.cookie_httponly', 1);   // Inaccessible from JavaScript
ini_set('session.cookie_secure', 1);     // HTTPS only
ini_set('session.cookie_samesite', 'Lax');  // Partial CSRF protection
ini_set('session.use_strict_mode', 1);   // Reject session IDs not generated by the server

session_start();
?>

Or in /etc/php/8.x/apache2/php.ini to apply globally:

session.cookie_httponly = 1
session.cookie_secure = 1
session.cookie_samesite = Lax
session.use_strict_mode = 1

CSRF protection

Every form that performs an action (modification, deletion, submission) must be protected with a CSRF token. The minimal but correct implementation:

<?php
session_start();

// Token generation (once per session or per form)
if (empty($_SESSION['csrf_token'])) {
    $_SESSION['csrf_token'] = bin2hex(random_bytes(32));
}

// In the HTML form
// <input type="hidden" name="csrf_token" value="<?= htmlspecialchars($_SESSION['csrf_token']) ?>">

// Verification on POST requests
if ($_SERVER['REQUEST_METHOD'] === 'POST') {
    $submitted_token = $_POST['csrf_token'] ?? '';
    if (!hash_equals($_SESSION['csrf_token'], $submitted_token)) {
        http_response_code(403);
        die('Invalid CSRF token');
    }
    // Regenerate after use for single-use forms
    $_SESSION['csrf_token'] = bin2hex(random_bytes(32));
}
?>

Rate limiting on sensitive actions

Without a database or Redis, basic session-based rate limiting is sufficient for small sites:

<?php
function check_rate_limit(string $action, int $max_attempts, int $window_seconds): bool
{
    $key = 'rate_limit_' . $action;
    $now = time();

    if (!isset($_SESSION[$key])) {
        $_SESSION[$key] = ['count' => 0, 'reset_at' => $now + $window_seconds];
    }

    if ($now > $_SESSION[$key]['reset_at']) {
        $_SESSION[$key] = ['count' => 0, 'reset_at' => $now + $window_seconds];
    }

    $_SESSION[$key]['count']++;

    return $_SESSION[$key]['count'] <= $max_attempts;
}

// Usage
if (!check_rate_limit('login', 5, 300)) {  // 5 attempts per 5 minutes
    http_response_code(429);
    die('Too many attempts. Please try again in a few minutes.');
}
?>

9. Apache HTTP headers

These headers strengthen security on the browser side. They don't prevent a server intrusion, but they reduce the attack surface for XSS and clickjacking vulnerabilities.

# In .htaccess or the vhost config
# Requires mod_headers: a2enmod headers

<IfModule mod_headers.c>
    # Prevents the browser from guessing the MIME type
    Header always set X-Content-Type-Options "nosniff"

    # Prevents inclusion in iframes (clickjacking protection)
    Header always set X-Frame-Options "SAMEORIGIN"

    # Controls information sent in the Referer header
    Header always set Referrer-Policy "strict-origin-when-cross-origin"

    # Disables sensitive unused features
    Header always set Permissions-Policy "camera=(), microphone=(), geolocation=()"

    # Remove Apache version from responses
    Header always unset X-Powered-By
    ServerTokens Prod
    ServerSignature Off
</IfModule>

HSTS (Strict-Transport-Security) is handled automatically by Certbot/Let's Encrypt if the site is on HTTPS. Don't configure it manually unless you know exactly what you're doing — a too-long duration with an HTTPS config error can make the site inaccessible for months from browsers that cached the header.

10. Docker — Isolating containers

When Apache acts as a reverse proxy to Docker containers, a common shortcut is to expose container ports on all network interfaces. The result: the service is directly accessible from the Internet, completely bypassing the reverse proxy and all the authentication that comes with it.

# /srv/wiki/docker-compose.yml

services:
  wiki:
    image: requarks/wiki:2
    # BAD — accessible from any IP on port 3000
    ports:
      - "3000:3000"

    # GOOD — only from localhost, the reverse proxy can reach it, Internet cannot
    ports:
      - "127.0.0.1:3000:3000"

The Apache reverse proxy config:

<VirtualHost *:443>
    ServerName wiki.example.com

    ProxyPreserveHost On
    ProxyPass / http://127.0.0.1:3000/
    ProxyPassReverse / http://127.0.0.1:3000/

    # Authentication is handled here, not in the container
    <Location /admin>
        AuthType Digest
        AuthName "Admin Area"
        AuthDigestProvider file
        AuthUserFile /etc/apache2/.htdigest
        Require valid-user
    </Location>
</VirtualHost>

Check existing containers that might have this problem:

# List exposed Docker ports
docker ps --format "table {{.Names}}\t{{.Ports}}"

# Look for bindings on 0.0.0.0 (problematic)
docker ps --format "{{.Ports}}" | grep "0.0.0.0"

11. Automatic updates

apt install unattended-upgrades apt-listchanges

# Enable
dpkg-reconfigure -plow unattended-upgrades

Verify the generated configuration in /etc/apt/apt.conf.d/20auto-upgrades:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";

On Debian 12, security updates are included by default in the unattended-upgrades configuration. Check /etc/apt/apt.conf.d/50unattended-upgrades to ensure the Debian-Security origins are uncommented.

# Simulate what would be updated
unattended-upgrade --dry-run -d

# Force an update now
unattended-upgrade -d

# View logs of past updates
cat /var/log/unattended-upgrades/unattended-upgrades.log | tail -50

12. Log retention

Debian keeps system logs for 4 weeks by default. If you detect an incident and want to know what happened 6 weeks ago, you have nothing. Extend retention now, before you need it.

# /etc/logrotate.d/rsyslog — change rotate 4 to rotate 13 for ~3 months
# (check the existing content first)
cat /etc/logrotate.d/rsyslog

# Modified version
/var/log/syslog
/var/log/auth.log
/var/log/kern.log
/var/log/mail.log
/var/log/daemon.log
{
    rotate 13
    weekly
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

# /etc/logrotate.d/apache2 — extend to 365 days
# Look for "rotate" in the existing file and adapt
# Recommended format for Apache: daily rotation, 365 files
/var/log/apache2/*.log {
    daily
    rotate 365
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        if invoke-rc.d apache2 status > /dev/null 2>&1; then
            invoke-rc.d apache2 reload > /dev/null 2>&1
        fi
    endscript
}

# fail2ban: 54 weeks (~1 year)
# /etc/logrotate.d/fail2ban
/var/log/fail2ban.log {
    weekly
    rotate 54
    compress
    delaycompress
    missingok
    postrotate
        fail2ban-client flushlogs 1>/dev/null || true
    endscript
}

# Test logrotate config
logrotate --debug /etc/logrotate.conf

13. Password changes post-incident

The least glamorous but most critical part. After any suspicion of intrusion, even if the analysis concludes it was a failed attempt, change all passwords that could have been exposed. When in doubt, change them.

htdigest password

# Change a user's password in an htdigest file
htdigest /etc/apache2/.htdigest "Private Area" alice

# Verify the resulting file (format: user:realm:md5_hash)
cat /etc/apache2/.htdigest

System passwords

# Change a user's password
passwd alice

# Change root password
passwd root

# Force change on next login
chage -d 0 alice

Checking consistency

The classic trap: a password is referenced in multiple places. Before validating a change, search for all occurrences:

# Search for references to a username in configs
grep -r "alice" /etc/apache2/ 2>/dev/null
grep -r "alice" /etc/proftpd/ 2>/dev/null
grep -r "alice" /var/www/html/ 2>/dev/null

# Find htpasswd and htdigest files
find /etc/apache2 /var/www -name ".htpasswd" -o -name ".htdigest" 2>/dev/null

A password can live in the htdigest file, in an application's configuration (wiki, seedbox), in maintenance scripts, and in internal documentation. Updating just one of the four and re-explaining everything to all users three weeks later is not a recommended experience.

Conclusion — Security hardening checklist

What this audit produced concretely, summarized for a quick review:

fail2ban installed and configured with custom filter for Apache auth_digest
Active jails: sshd, apache-auth, proftpd
sftponly group with chroot and ForceCommand internal-sftp
Chrooted users' home dirs at root:root 755 (unavoidable constraint)
Detailed SFTP logging enabled in sshd_config
Root SSH login set to prohibit-password, authorized_keys verified
Credential file permissions reviewed (640, root:www-data)
Sensitive directories blocked by RewriteRule (not Location)
Accounts without sudo need removed from the sudo group
www-data sudoers limited to strictly necessary commands
auditd installed with rules on critical files
acct installed for per-user command history
rkhunter installed, baseline initialized
Daily audit script with AI analysis and raw report fallback
Secure PHP sessions (httponly, secure, samesite)
CSRF protection on forms
Security HTTP headers configured in Apache
Docker ports bound to 127.0.0.1 only
unattended-upgrades active for security updates
Extended log retention (3 months for auth.log, 1 year for Apache)
All potentially exposed passwords changed

A server exposed on the Internet will always be attacked. That comes with the territory. The question isn't to prevent attempts — that's impossible — but to ensure that attempts fail, that any eventual successes are detected quickly, and that you have the logs needed to understand what happened.

This audit took about two days of work spread over a week. Most points should have been done at initial installation. That's rarely the case. What matters is doing it before something serious happens — and automating monitoring so you don't have to revisit it manually every week.

Testing a production Bash script: 38 tests without a framework

Odilon HUGONNOT — Wed, 29 Apr 2026 09:00:02 +0000

The context: a Bash script to display Claude Code rate limits in a terminal status bar. At first it seems trivial — a few functions, reading a JSON file, some formatting. But the script quickly ends up handling: JSON parsing with jq, time formatting with timezone, percentage calculations, Unicode progress bar generation, countdown logic. And above all: several edge cases (missing data, corrupted JSON, quotas at 0, unknown timezone).

Without tests, every change becomes a lottery. You change the countdown formatting, you don't know if you broke the percentage calculation. You add active model handling, you don't know if the corrupted JSON fixtures still pass. With 38 unit tests on a Bash script, you modify with confidence — and bash tests/test_statusline.sh in 2 seconds confirms nothing is broken.

Why no Bash testing framework

Bash testing frameworks exist: bats-core, shunit2, shellspec. They're good. For this project, the choice was not to add one — zero external dependencies in the repo, one-liner install. A test framework is a dependency. The constraint was simple: tests run with native bash, nothing to install.

Result: ~100 lines of homemade test helpers. No magic, no DSL, no plugin to update — just functions and conventions. For a 300-line script, that's the right level of engineering. A testing framework to test a Bash script is often more complexity than the script itself.

The basic structure: assertions and counters

The test file starts with two global counters and two assertion functions:

#!/usr/bin/env bash
# tests/test_statusline.sh

PASS=0
FAIL=0

assert_eq() {
    local description="$1"
    local expected="$2"
    local actual="$3"

    if [[ "$expected" == "$actual" ]]; then
        echo "  ✓ $description"
        ((PASS++))
    else
        echo "  ✗ $description"
        echo "    expected: $(echo "$expected" | cat -A)"
        echo "    actual:   $(echo "$actual" | cat -A)"
        ((FAIL++))
    fi
}

assert_contains() {
    local description="$1"
    local needle="$2"
    local haystack="$3"

    if [[ "$haystack" == *"$needle"* ]]; then
        echo "  ✓ $description"
        ((PASS++))
    else
        echo "  ✗ $description"
        echo "    expected to contain: $needle"
        echo "    actual: $haystack"
        ((FAIL++))
    fi
}

# At the end of the test file:
echo ""
echo "Results: $PASS passed, $FAIL failed"
[[ $FAIL -eq 0 ]] || exit 1

The cat -A in the error message displays invisible characters (spaces, tabs, \n, $ at end of line) — essential for debugging tests that fail on an extra space in a formatted string. Without it, you spend 20 minutes comparing two strings that look identical visually but aren't equal.

Testing Bash functions: the isolation problem

The main script (statusline.sh) isn't designed to be sourced in tests — it executes top-level code as soon as it's launched. The pattern to isolate functions without modifying production behavior:

# In statusline.sh, wrap the executable code:
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    # Executed only when the script is run directly
    main "$@"
fi

# Functions defined in the file remain available when sourced

In the test file, source the functions without triggering main:

# Source functions without executing main
source "$(dirname "$0")/../statusline.sh"

# Now we can test functions directly
result=$(format_percentage 67)
assert_eq "67% gives yellow" "🟡" "$result"

BASH_SOURCE[0] holds the path of the currently executing script. $0 holds the path of the script launched by the user. When you source a file, these two values differ — the if doesn't execute.

Mocking external dependencies

The script calls jq, reads JSON files, uses date with timezone. For unit tests on Bash scripts, you mock by redefining commands in the test scope — Bash resolves functions before binaries in PATH:

# Mock jq to return controlled data
jq() {
    echo "42"
}
export -f jq

result=$(get_context_percentage)
assert_eq "context at 42%" "42" "$result"

# Clean up the mock after the test
unset -f jq

For more complex JSON data, create fixtures in tests/fixtures/ and override the path variable via the environment:

USAGE_FILE="tests/fixtures/usage_normal.json" \
    result=$(render_statusline)
assert_contains "displays the model" "Sonnet 4.6" "$result"

USAGE_FILE="tests/fixtures/usage_corrupted.json" \
    result=$(render_statusline)
assert_contains "corrupted JSON → fallback" "N/A" "$result"

The advantage: fixtures are real readable JSON files, versionable, easy to update when the format changes. No need to mock jq for each case — you provide the input data directly.

A concrete example: testing the Unicode progress bar

The progress bar (▓▓▓░░░░░) is one of the most tested functions — it must correctly handle edge cases of Bash integer division:

test_progress_bar() {
    echo "--- Progress bar ---"
    assert_eq "0%  → 8 empty blocks"     "░░░░░░░░" "$(make_bar 0)"
    assert_eq "50% → 4 full 4 empty"     "▓▓▓▓░░░░" "$(make_bar 50)"
    assert_eq "100%→ 8 full blocks"      "▓▓▓▓▓▓▓▓" "$(make_bar 100)"
    assert_eq "34% → rounded down"       "▓▓░░░░░░" "$(make_bar 34)"
    assert_eq "99% → 7 full"             "▓▓▓▓▓▓▓░" "$(make_bar 99)"
}

5 tests, 5 edge cases. The 34% case is the most important: 34 * 8 / 100 = 2.72 → rounded to 2 by Bash integer division (no bc, no awk). Without an explicit test, this behavior is discovered in production when the bar displays one too many blocks. The 99% case verifies that the last block stays empty — 7 full, not 8.

The result: 38 tests, 0 external framework

Final organization of tests by category:

Progress bar (5 tests) — boundary values, integer division rounding
Color coding (6 tests) — green/yellow/red thresholds by percentage
Percentage formatting (4 tests) — display with %, zero, 100
Time formatting (8 tests) — countdown, timezone, reset times, midnight
JSON parsing (7 tests) — missing data, null values, corrupted JSON
Full statusline render (8 tests) — complete output with real data fixtures

Run: bash tests/test_statusline.sh. Output:

--- Progress bar ---
  ✓ 0% → 8 empty blocks
  ✓ 50% → 4 full 4 empty
  ✓ 100% → 8 full blocks
  ✓ 34% → rounded down
  ✓ 99% → 7 full

--- Color coding ---
  ✓ 0% → green
  ✓ 74% → green
  ✓ 75% → yellow
  ✓ 89% → yellow
  ✓ 90% → red
  ✓ 100% → red

[...]

Results: 38 passed, 0 failed

What it changes in practice

The statusline evolved 12+ times in a single day of development: adding the active model, overhauling the countdown, adding git dirty status, handling exhausted quotas. With each change, bash tests/test_statusline.sh in 2 seconds. Regressions are caught immediately — not after restarting Claude Code to see that the bar displays N/A instead of the percentage.

Bash has no typing, no strict linter, no IDE that highlights logic errors. Tests compensate. This isn't a substitute for a real language — it's the minimal safety net that makes the script modifiable without anxiety.

Conclusion

100 lines of helpers. 38 Bash unit tests. Zero external dependencies. For a production Bash script that needs to be modifiable safely, that's the right investment. The pattern is portable to any Bash project: assert_eq, assert_contains, mock by function redefinition, fixtures as JSON files. No need for a testing framework to test a 300-line script.

The full project is on GitHub: github.com/ohugonnot/claude-code-statusline

Migrating Postman to Bruno in a monorepo: the practical guide

Odilon HUGONNOT — Tue, 28 Apr 2026 09:00:02 +0000

The ticket lands on a Monday morning: "Migrate Postman collections to Bruno, integrate them into the monorepo." You've used Postman for years without thinking too hard about it. Bruno you've heard of but never opened. The official docs list features. Medium articles explain "the 10 reasons to switch." Neither tells you concretely how to start clean on a monorepo with 6 services — patient API, lab, doctor portal, backoffice, billing, drug database — 3 environments, and secrets you can't commit.

This is that guide. No preamble, no progressive intro — straight to what you need to know so the result is clean from the first commit.

Why Bruno

The real reason isn't "Bruno is better than Postman." It's that Postman has drifted toward a cloud-first model that creates concrete problems in a team:

Postman collections live in Postman's cloud, not in your repo. They're not reviewable in a PR, they don't follow branches, and if someone modifies a request without telling anyone, it shows up nowhere in your version history.
Workspace sharing requires an account and a paid Postman organization plan once you go beyond 4 or 5 active developers.
The Postman client weighs 300+ MB (Electron). Anecdotal on its own, but representative of the model: grow the tool to justify the cloud value.
Postman environment variables are stored in Postman's cloud, which creates friction for secrets: either you put them there (and lose control) or you manage them outside (and you're maintaining two separate configurations).

Bruno solves all of this with one architectural decision: collections are text files (the .bru format, a simple DSL) stored in a directory. No account, no cloud, no magic sync. A git clone and everything is there. Request changes show up in diffs, PRs, and git blame. That's the only real reason to migrate.

The concepts in 5 minutes

Bruno has three concepts to understand before opening the interface. Two are similar to Postman, one is meaningfully different.

Collection — a directory of .bru files. When you open Bruno and create a collection, you pick a folder on your disk. That's it. Each request is a .bru file in that folder or a subfolder.

Environment — a named set of variables, defined in the Bruno UI and stored in the collection directory under an environments/ subfolder, one .bru file per environment. These files are version-controlled. They hold non-sensitive variables: base URLs, ports, domain names, feature flags. You can put base_url = http://localhost:8080 in there; you should not put a JWT token or a client certificate.

.env — a standard .env file at the root of the collection, gitignored. It holds secrets: tokens, passwords, paths to certificates. Bruno loads it automatically. In your .bru files, you access environment variables with {{TOKEN}} (which looks first in the active Environment, then in the .env) or with process.env.TOKEN in scripts.

Concept

Version-controlled?

Use for

environments/*.bru

Yes

base_url, ports, domain names

.env

No (gitignored)

Tokens, certificates, passwords

.bru (requests)

Yes

API call definitions

Structure in the monorepo

First decision: where to put the collections in the monorepo. Options include postman/ (legacy name, avoid it), api-collection/ (verbose), api/ (ambiguous if you already have Go packages called api), bruno/ (the tool name — explicit, short, unambiguous).

The bruno/ choice has an added benefit: it's visible in the tree that this is a Bruno collection, not a Go code directory. When someone clones the repo for the first time, they don't have to guess what's in that folder.

Recommended structure for 6 services:

bruno/
├── environments/
│   ├── local.bru
│   ├── dev.bru
│   └── prod.bru
├── .env              # gitignored — tokens, certs
├── .gitignore
├── patient-api/      # Patient API — REST
│   ├── health.bru
│   ├── patients/
│   │   ├── create-patient.bru
│   │   └── get-patient.bru
│   └── ...
├── lab-service/      # Lab Service — gRPC
│   ├── lab-results.bru
│   └── ...
├── doctor-portal/    # REST
│   └── ...
├── backoffice/       # REST
│   └── ...
├── billing-service/  # REST
│   └── ...
└── drug-db-api/      # REST — external API (e.g. OpenFDA)
    └── ...

The .gitignore inside bruno/ contains at minimum:

.env

One service = one subfolder. Inside, you can create as many subfolders as needed to organize requests by resource or feature. Bruno renders the directory tree as-is in its explorer.

Managing environments

Environment files are .bru files with a slightly different syntax from request files. Here's what a realistic local.bru looks like:

vars {
  patient_url: http://localhost:8080
  lab_url: localhost:9090
  doctor_url: http://localhost:8081
  backoffice_url: http://localhost:8082
  billing_url: http://localhost:8083
  drug_db_url: https://api.openfda.gov
  grpc_tls: false
}

vars:secret [
  AUTH_TOKEN
  DRUG_DB_API_KEY
]

The vars:secret block lists the names of secret variables — it declares their existence without their value. The actual values come from the .env. This is what allows Bruno to handle display (masked in the UI) without storing secrets in the versioned file.

The dev.bru file looks like this:

vars {
  patient_url: https://patient-dev.acme-internal.com
  lab_url: lab-dev.acme-internal.com:443
  doctor_url: https://doctor-dev.acme-internal.com
  backoffice_url: https://backoffice-dev.acme-internal.com
  billing_url: https://billing-dev.acme-internal.com
  drug_db_url: https://api.openfda.gov
  grpc_tls: true
}

vars:secret [
  AUTH_TOKEN
  DRUG_DB_API_KEY
]

To switch environments in Bruno: dropdown menu in the top right of the interface, select local, dev, or prod. All requests immediately use the new environment's variables.

Secrets in .env

The .env at the root of bruno/ holds the actual values of secret variables declared in the environments:

AUTH_TOKEN=
DRUG_DB_API_KEY=
CLIENT_CERT_PATH=/home/user/.certs/client.crt
CLIENT_KEY_PATH=/home/user/.certs/client.key

In a .bru request file, you reference these variables with double-brace syntax:

headers {
  Authorization: Bearer {{AUTH_TOKEN}}
  X-DrugDB-Key: {{DRUG_DB_API_KEY}}
}

Bruno resolves {{AUTH_TOKEN}} by looking in order: active Environment variables, then .env. If you've defined AUTH_TOKEN in both, the Environment wins. That's the expected behavior: you can override a secret per environment without touching the .env file.

For client certificates (mutual TLS), Bruno supports configuration in the collection settings. You can reference paths from the .env: {{CLIENT_CERT_PATH}} in the certificate path field.

gRPC with server reflection

Acme's lab service exposes a gRPC service — analysis results stream in as each machine finishes a measurement, without waiting for the full batch to complete. The good news: Bruno supports gRPC natively, and it supports server reflection mode — you don't need the .proto file to discover and call methods. The server responds to a reflection request and Bruno builds the list of available methods.

For reflection to work, your Go service needs to have registered the reflection service. If it hasn't yet, two lines to add in main.go:

import "google.golang.org/grpc/reflection"

// In the function that configures the gRPC server:
reflection.Register(grpcServer)

Here's what a .bru file for a unary gRPC request looks like:

meta {
  name: Get Lab Results
  type: grpc
  seq: 1
}

url: {{lab_url}}

body {
  {
    "patient_id": "{{patient_id}}",
    "test_type": "blood_panel",
    "date_from": "2026-01-01"
  }
}

grpc {
  method: LabResultService/GetResults
  reflect: true
  tls: {{grpc_tls}}
}

The reflect: true field tells Bruno to use server reflection to fetch the schema. tls: {{grpc_tls}} uses the environment variable — false locally, true on dev/prod. The URL is just host:port, no scheme. The patient_id in the body can be injected from a pre-request script or copied from a previous REST response (patient record creation).

For gRPC streaming (server-stream, client-stream, bidirectional), Bruno supports that too but the configuration differs slightly — the official docs cover that case better than any summary can.

What a real .bru file looks like

This is where people coming from Postman are most surprised: requests are plain text files with a simple syntax, not 200-line nested JSON. Here's a complete REST request with bearer auth, JSON body, and custom headers:

meta {
  name: Create Patient Record
  type: http
  seq: 1
}

post {
  url: {{patient_url}}/v1/patients
  body: json
  auth: bearer
}

auth:bearer {
  token: {{AUTH_TOKEN}}
}

headers {
  X-Request-ID: {{$randomUUID}}
  X-Client-Version: 2.1.0
}

body:json {
  {
    "last_name": "Dupont",
    "first_name": "Marie",
    "dob": "1985-03-12",
    "nss": "{{$randomUUID}}"
  }
}

tests {
  test("status is 201", function() {
    expect(res.status).to.equal(201);
  });

  test("has patient id", function() {
    expect(res.body.patient_id).to.be.a("string");
  });
}

A few things worth noting about this format:

seq controls the display order in Bruno's explorer.
{{$randomUUID}} is a built-in dynamic variable — Bruno provides several ($timestamp, $randomInt, etc.).
The tests block uses Chai for assertions — same syntax as Postman if you were using it before v10.
There's no pre-request script in some weird JSON/DSL format — it's plain JavaScript in a script:pre-request { } block.

This file is readable in a diff. If someone changes the body or adds a header, you see it in the PR. That's the fundamental difference from Postman.

Migrating from Postman

Bruno supports import from Postman Collection v2.1.

What migrates cleanly: simple REST requests, headers, JSON/form-data bodies, non-secret environment variables. Bruno generates one .bru file per request, folder structure is preserved.

What doesn't migrate correctly:

Complex pre-request scripts — Postman scripts using pm.sendRequest() or advanced environment manipulation need to be rewritten manually. Bruno's syntax is similar but not identical.
Postman Flows — completely Postman-specific, no equivalent in Bruno.
Monitors and mock servers — Postman cloud features, out of scope for Bruno.
Secret variables — you normally don't export these, so they won't appear in the exported JSON anyway.

Pragmatic recommendation: import one collection, check the result, fix what's broken. Don't bulk-import all 6 services without verifying. Critical requests (auth, resource creation) are worth recreating by hand in 5 minutes rather than importing and leaving with unpredictable behavior. The .bru format is simple enough for that.

Step by step

Download and install Bruno — available at usebruno.com/downloads for macOS, Linux, and Windows. No account required, no cloud, standard installation.
Export from Postman — in Postman, right-click on the collection → Export → Collection v2.1 → save the JSON file locally.
Create the Bruno collection in the repo — in Bruno: File → Open Collection → select the bruno/ folder in the monorepo (or create a new folder if it doesn't exist yet). Bruno opens that folder as the collection root.
Import the Postman collection — in Bruno: File → Import Collection → Postman Collection v2.1 → select the JSON exported in step 2. Bruno creates one .bru file per request in the current folder, preserving the folder structure.
Create the environments — in Bruno: Environments menu (gear icon, top right) → Create Environment → name it local. Add your variables: base_url, ports, etc. Repeat for dev and prod. Bruno creates the corresponding files in bruno/environments/.
Create the .env and .gitignore — create bruno/.env with your secrets (tokens, API keys, certificate paths). Create bruno/.gitignore with at minimum .env in it. Commit the .gitignore, never commit the .env.
Verify and fix — test each imported request one by one. Fix variables if needed ({{variable}} syntax from Postman is compatible with Bruno, but complex pre-request scripts need to be rewritten manually in a script:pre-request { } block using plain JavaScript).
Commit — git add bruno/ → git commit -m "feat: migrate API collections from Postman to Bruno". The .bru files, environments/*.bru, and .gitignore are version-controlled. The .env never is.

Conclusion

The Postman → Bruno migration takes half a day for a monorepo of this size. Half of that time is initial setup (structure, environments, .env). The other half is verifying that imported requests actually work.

What you gain isn't "a better API tool." It's that collections become code: reviewable in PRs, diffable commit by commit, consistent across all developers without manual sync. When someone adds an endpoint or changes a parameter, it's in the same commit as the Go code.

That's enough to justify the migration.

Testing a Go exchange API client: mocks, httptest and testcontainers

Odilon HUGONNOT — Mon, 27 Apr 2026 09:00:03 +0000

The Go service runs locally, tests pass. Then you remember the tests are hitting the real Binance API. The next day, CI fails: rate limit exceeded, unstable network, or the exchange quietly renamed a field in its response. Tests that depend on external APIs are flaky at best, unusable in CI at worst.

The problem is structural: exchanges don't have usable sandbox environments (Binance testnet exists, but with constraints that make automated integration testing difficult), rate limits are aggressive, and some operations like PlaceOrder have real side effects. A different approach is needed.

The problem with external APIs in tests

Three concrete reasons you can't test against real exchange APIs in CI:

Rate limits: Binance caps at 1200 requests/minute on the order book endpoint. Ten developers running tests in parallel, and the first ones get banned.
No reliable sandbox: OKEx and Coinbase Pro have test environments, but their availability isn't guaranteed, the data isn't representative, and authentication is yet another CI problem to solve.
Side effects: a test that places a real order during a demo is the kind of incident you don't forget.

Three approaches to work around this, in order of complexity:

Approach

Use case

Speed

HTTP mock (httptest.NewServer)

Test API response parsing

Very fast

Interface mock (mockgen)

Test business logic in isolation

Very fast

Testcontainers

Integration tests with a real DB

Slow (10-30s)

Defining an Exchange interface

The key insight: define a Go interface that all exchange clients implement. This is the foundation of testability. Without it, the service depends on a concrete type and becomes impossible to mock properly.

// exchange/client.go

type OrderBook struct {
    Bids []Level `json:"bids"`
    Asks []Level `json:"asks"`
}

type Level struct {
    Price decimal.Decimal
    Qty   decimal.Decimal
}

type Order struct {
    Pair     string
    Side     string // "buy" | "sell"
    Quantity decimal.Decimal
    Price    decimal.Decimal
}

type OrderResult struct {
    OrderID   string
    Status    string
    FilledQty decimal.Decimal
}

// ExchangeClient is the interface all exchange clients implement.
// The service depends on this interface, never on concrete types.
type ExchangeClient interface {
    GetOrderBook(ctx context.Context, pair string) (*OrderBook, error)
    PlaceOrder(ctx context.Context, order Order) (*OrderResult, error)
    GetBalance(ctx context.Context) (map[string]decimal.Decimal, error)
}

Each exchange implements the interface. The service receives ExchangeClient as a parameter — not *BinanceClient or *CoinbaseClient:

// exchange/binance.go
type BinanceClient struct {
    baseURL    string
    apiKey     string
    secretKey  string
    httpClient *http.Client
}

func NewBinanceClient(apiKey, secretKey string) *BinanceClient {
    return &BinanceClient{
        baseURL:    "https://api.binance.com",
        apiKey:     apiKey,
        secretKey:  secretKey,
        httpClient: &http.Client{Timeout: 10 * time.Second},
    }
}

// BinanceClient implements ExchangeClient.
func (c *BinanceClient) GetOrderBook(ctx context.Context, pair string) (*OrderBook, error) {
    // real HTTP call to Binance
}

// exchange/coinbase.go
type CoinbaseClient struct { /* ... */ }

func (c *CoinbaseClient) GetOrderBook(ctx context.Context, pair string) (*OrderBook, error) {
    // real HTTP call to Coinbase
}

// aggregator/service.go
type AggregatorService struct {
    exchanges []ExchangeClient
}

func NewAggregatorService(exchanges ...ExchangeClient) *AggregatorService {
    return &AggregatorService{exchanges: exchanges}
}

// In tests, pass mocks. In production, pass real clients.
func main() {
    binance := exchange.NewBinanceClient(os.Getenv("BINANCE_KEY"), os.Getenv("BINANCE_SECRET"))
    coinbase := exchange.NewCoinbaseClient(os.Getenv("COINBASE_KEY"), os.Getenv("COINBASE_SECRET"))

    svc := aggregator.NewAggregatorService(binance, coinbase)
}

Mocks with mockgen

Once the interface is defined, mockgen generates mocks automatically. Don't write them by hand — maintenance becomes a nightmare as soon as the interface evolves.

// exchange/client.go
// Add this directive to generate mocks

//go:generate mockgen -destination=../mocks/mock_exchange.go -package=mocks . ExchangeClient

go generate ./exchange/...

The generated file at mocks/mock_exchange.go provides a ready-to-use MockExchangeClient. Table-driven test for a service that aggregates order books from multiple exchanges:

// aggregator/service_test.go

func TestAggregateOrderBooks(t *testing.T) {
    tests := []struct {
        name        string
        binanceOB   *exchange.OrderBook
        coinbaseOB  *exchange.OrderBook
        binanceErr  error
        coinbaseErr error
        wantBestBid decimal.Decimal
        wantErr     bool
    }{
        {
            name: "normal case — best bid on Coinbase",
            binanceOB: &exchange.OrderBook{
                Bids: []exchange.Level{
                    {Price: decimal.NewFromFloat(65000), Qty: decimal.NewFromFloat(0.5)},
                },
            },
            coinbaseOB: &exchange.OrderBook{
                Bids: []exchange.Level{
                    {Price: decimal.NewFromFloat(65100), Qty: decimal.NewFromFloat(0.3)},
                },
            },
            wantBestBid: decimal.NewFromFloat(65100),
        },
        {
            name:       "exchange down — return an error",
            binanceErr: errors.New("connection refused"),
            wantErr:    true,
        },
        {
            name: "empty order book — skip that exchange",
            binanceOB: &exchange.OrderBook{
                Bids: []exchange.Level{
                    {Price: decimal.NewFromFloat(64900), Qty: decimal.NewFromFloat(1.0)},
                },
            },
            coinbaseOB: &exchange.OrderBook{Bids: []exchange.Level{}},
            wantBestBid: decimal.NewFromFloat(64900),
        },
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            ctrl := gomock.NewController(t)
            defer ctrl.Finish()

            binanceMock := mocks.NewMockExchangeClient(ctrl)
            coinbaseMock := mocks.NewMockExchangeClient(ctrl)

            binanceMock.EXPECT().
                GetOrderBook(gomock.Any(), "BTCUSDT").
                Return(tt.binanceOB, tt.binanceErr)

            if tt.binanceErr == nil {
                coinbaseMock.EXPECT().
                    GetOrderBook(gomock.Any(), "BTCUSDT").
                    Return(tt.coinbaseOB, tt.coinbaseErr)
            }

            svc := aggregator.NewAggregatorService(binanceMock, coinbaseMock)
            result, err := svc.GetBestBid(context.Background(), "BTCUSDT")

            if tt.wantErr {
                require.Error(t, err)
                return
            }
            require.NoError(t, err)
            assert.True(t, tt.wantBestBid.Equal(result),
                "expected %s, got %s", tt.wantBestBid, result)
        })
    }
}

The test verifies the service's behavior, not its implementation. If OKEx is added tomorrow, you add a mock in the tests without touching any existing logic.

HTTP mock server for testing parsing

Interface mocks test business logic, but not the HTTP layer. If Binance changes its response format, the mocks won't catch it — they return Go structs, not JSON.

To test that BinanceClient.GetOrderBook() correctly parses the real API format, use httptest.NewServer:

// exchange/binance_test.go

func TestBinanceOrderBookParsing(t *testing.T) {
    // Real format from Binance API GET /api/v3/depth
    server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        assert.Equal(t, "/api/v3/depth", r.URL.Path)
        assert.Equal(t, "BTCUSDT", r.URL.Query().Get("symbol"))

        w.Header().Set("Content-Type", "application/json")
        json.NewEncoder(w).Encode(map[string]interface{}{
            "lastUpdateId": 1027024,
            "bids": [][]string{
                {"65000.00000000", "0.50000000"},
                {"64950.00000000", "1.20000000"},
            },
            "asks": [][]string{
                {"65001.00000000", "0.30000000"},
                {"65010.00000000", "0.80000000"},
            },
        })
    }))
    defer server.Close()

    // Point the client at the test server
    client := exchange.NewBinanceClientWithBaseURL(server.URL, "test-key", "test-secret")

    ob, err := client.GetOrderBook(context.Background(), "BTCUSDT")

    require.NoError(t, err)
    require.Len(t, ob.Bids, 2)
    require.Len(t, ob.Asks, 2)

    assert.True(t, decimal.NewFromFloat(65000).Equal(ob.Bids[0].Price))
    assert.True(t, decimal.NewFromFloat(0.5).Equal(ob.Bids[0].Qty))
    assert.True(t, decimal.NewFromFloat(65001).Equal(ob.Asks[0].Price))
}

This test catches real bugs: renamed field, unexpected decimal format, unexpected element order in the array. It also documents exactly what the client expects from the API — useful when Binance ships a breaking change.

For this to work, the constructor needs a configurable baseURL. Good practice anyway — it lets you point to Binance testnet or a proxy in production:

func NewBinanceClientWithBaseURL(baseURL, apiKey, secretKey string) *BinanceClient {
    return &BinanceClient{
        baseURL:    baseURL,
        apiKey:     apiKey,
        secretKey:  secretKey,
        httpClient: &http.Client{Timeout: 10 * time.Second},
    }
}

func NewBinanceClient(apiKey, secretKey string) *BinanceClient {
    return NewBinanceClientWithBaseURL("https://api.binance.com", apiKey, secretKey)
}

Testcontainers for integration tests

Mocks cover business logic and HTTP parsing. One case remains: verifying that trades are correctly persisted to the database. A real PostgreSQL is needed here — DB mocks don't test real SQL queries, constraints, or transactions.

testcontainers-go starts a Docker container from within the test, then tears it down when done. No manual setup, no shared database where developers step on each other:

// repository/trade_test.go
// +build integration

func TestSaveTradeIntegration(t *testing.T) {
    ctx := context.Background()

    pg, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
        ContainerRequest: testcontainers.ContainerRequest{
            Image:        "postgres:16",
            ExposedPorts: []string{"5432/tcp"},
            Env: map[string]string{
                "POSTGRES_PASSWORD": "test",
                "POSTGRES_DB":       "trades_test",
                "POSTGRES_USER":     "test",
            },
            WaitingFor: wait.ForListeningPort("5432/tcp"),
        },
        Started: true,
    })
    require.NoError(t, err)
    defer pg.Terminate(ctx)

    host, _ := pg.Host(ctx)
    port, _ := pg.MappedPort(ctx, "5432/tcp")

    dsn := fmt.Sprintf("postgres://test:test@%s:%s/trades_test?sslmode=disable", host, port.Port())
    db, err := sqlx.ConnectContext(ctx, "pgx", dsn)
    require.NoError(t, err)
    defer db.Close()

    // Apply migrations
    require.NoError(t, runMigrations(db))

    repo := repository.NewTradeRepository(db)

    trade := &model.Trade{
        Exchange:   "binance",
        Pair:       "BTCUSDT",
        Side:       "buy",
        Price:      decimal.NewFromFloat(65000),
        Quantity:   decimal.NewFromFloat(0.1),
        ExecutedAt: time.Now().UTC(),
    }

    err = repo.Save(ctx, trade)
    require.NoError(t, err)
    require.NotEmpty(t, trade.ID)

    // Verify persistence
    saved, err := repo.GetByID(ctx, trade.ID)
    require.NoError(t, err)
    assert.Equal(t, "binance", saved.Exchange)
    assert.True(t, trade.Price.Equal(saved.Price))
}

The // +build integration build tag isolates these tests from unit tests. Run them separately:

# Unit tests only (fast, run on every commit)
go test ./...

# Integration tests (slow, run on main branch or pre-deploy)
go test -tags=integration ./...

Testcontainers takes between 10 and 30 seconds to start the container (depending on whether the image is cached). Acceptable for an integration suite, not for unit tests run 50 times a day.

What not to test

With mocks and httptest, it's tempting to test everything. Three things to avoid:

Testing that the Binance API works. That's not your code. If the Binance API is down, your service should detect the error and propagate it correctly — that's what you test, not Binance's uptime.

Over-mocking. If every call returns exactly what you told it to return, you end up testing the mock, not the service. A test that passes because the mocks are perfectly configured to pass is a false positive. The httptest.NewServer with real JSON is more honest than a mock that returns a Go struct directly.

Testing implementation rather than behavior. A test that verifies GetOrderBook was called exactly twice in a specific order is brittle. What matters: the service returns the best bid. How it gets there is its own business.

Conclusion

The layered structure — Exchange interface → business service → repository — isn't an abstraction for its own sake. It's what makes testing possible without hitting real APIs. The business service doesn't know whether it's talking to Binance or a mock. The test mock doesn't know whether the service actually calls the exchange.

In practice, all three test levels coexist in the project: unit tests with mockgen for fast logic, httptest.NewServer to validate API response parsing, and testcontainers for persistence integration tests. Each has its role. None replaces the others.

The most valuable test is often the HTTP parsing one. Business logic changes slowly. Exchange API formats change without warning.

DDD in Go applied to crypto exchange APIs

Odilon HUGONNOT — Sun, 26 Apr 2026 09:00:04 +0000

You write CQRS. You talk about aggregates. You emit domain events. But where do these concepts come from? Domain-Driven Design. Without understanding DDD, CQRS is just a pattern you copy-paste and hope it holds. With DDD, it becomes a tool you use consciously, for the right reasons.

This article lays the conceptual foundations. The concrete terrain: a Go service consuming APIs from multiple crypto exchanges (Binance, OKEx, Coinbase). Because an Order in trading is something any trader understands — that's exactly the kind of domain where DDD pays off.

Domain first, technology second

DDD's core principle is uncomfortable when you come from a technical background: the business model takes precedence over technology. You don't start with "what database", or "what API does Binance expose". You start with "what is an Order in this system? What is a Position? What happens when an order is partially filled?"

For a crypto trading service, the domain is: orders you place, positions that evolve, market data arriving continuously, a portfolio whose value you monitor. Not "a Binance wrapper". Not "a websocket consumer". Those are implementation details.

The practical difference: if your model is domain-centric, you can replace Binance with OKEx without touching your business logic. If your model is API-centric, you rewrite everything every time an exchange changes an endpoint.

DDD doesn't say "ignore technology". It says "don't let technology contaminate your business model".

Bounded contexts: splitting without cutting to the bone

This is the most practical DDD concept, and the most frequently misunderstood. A bounded context is an explicit boundary inside which a model has a precise, coherent meaning. Outside that boundary, the same word can mean something different — and that's acceptable.

For a crypto trading service, three natural bounded contexts emerge:

Market Data

Prices, order books, tickers, recent trades. This context is read-heavy, near real-time. Binance can push 10 updates per second on BTC/USDT. The model here is simple: immutable snapshots. A Ticker in Market Data has no mutable state — it's just a value at a point in time.

Trading

Orders, executions, positions. This context is command-heavy. An Order here has a full lifecycle: created, submitted, partially filled, cancelled. Business logic is concentrated here: validation rules, exchange rejection handling, fee calculation. This is where your aggregates live.

Portfolio

Balances, P&L, trade history. This context is a read model oriented toward reporting. It consumes events emitted by Trading to maintain a coherent view of the portfolio. Updates can be slightly delayed — a few seconds of lag is acceptable for reporting.

Why separate them? Because they have radically different constraints. Market Data at 10 updates/sec doesn't need the same model as Portfolio reporting aggregating data over 3 months. Putting everything in the same model means making compromises that make everything more complex without any benefit.

Bounded Context

Load

Consistency

Business complexity

Market Data

Very high (websockets)

Eventual OK

Low

Trading

Moderate (commands)

Strong (transactional)

High

Portfolio

Low (read model)

Eventual OK

Low

Aggregates and Value Objects in Go

Two core DDD building blocks that Go implements naturally, without any framework.

A Value Object is immutable. Its identity is defined by its value, not by a reference. Price{Amount: 42000, Currency: "USD"} equals another Price with the same fields. In Go, this translates to a struct without a pointer, passed by value.


// Value Object — immutable, equality by value
type Price struct {
    Amount   decimal.Decimal
    Currency string
}

func (p Price) IsZero() bool {
    return p.Amount.IsZero()
}

func (p Price) Add(other Price) (Price, error) {
    if p.Currency != other.Currency {
        return Price{}, fmt.Errorf("currency mismatch: %s vs %s", p.Currency, other.Currency)
    }
    return Price{Amount: p.Amount.Add(other.Amount), Currency: p.Currency}, nil
}

type TradingPair struct {
    Base  string // "BTC"
    Quote string // "USDT"
}

func (tp TradingPair) String() string {
    return tp.Base + "/" + tp.Quote
}

An Aggregate is a unit of consistency. All modifications go through its methods — never through direct field access. It guarantees that its business invariants are always upheld. And in event sourcing, it emits events to describe what happened.


type OrderStatus int

const (
    StatusPending   OrderStatus = iota
    StatusPartial
    StatusFilled
    StatusCancelled
)

// Aggregate — consistency guaranteed, modification through methods only
type Order struct {
    id           OrderID
    pair         TradingPair
    side         Side // Buy / Sell
    quantity     decimal.Decimal
    filledQty    decimal.Decimal
    status       OrderStatus
    events       []DomainEvent
}

func NewOrder(pair TradingPair, side Side, qty decimal.Decimal) (*Order, error) {
    if qty.IsZero() || qty.IsNegative() {
        return nil, fmt.Errorf("invalid quantity: %s", qty)
    }
    o := &Order{
        id:       NewOrderID(),
        pair:     pair,
        side:     side,
        quantity: qty,
        status:   StatusPending,
    }
    o.events = append(o.events, OrderCreated{
        OrderID: o.id,
        Pair:    pair,
        Side:    side,
        Qty:     qty,
    })
    return o, nil
}

func (o *Order) Fill(qty decimal.Decimal, price Price) error {
    if o.status != StatusPending && o.status != StatusPartial {
        return ErrOrderNotFillable
    }
    if qty.GreaterThan(o.quantity.Sub(o.filledQty)) {
        return fmt.Errorf("fill qty %s exceeds remaining %s", qty, o.quantity.Sub(o.filledQty))
    }

    o.filledQty = o.filledQty.Add(qty)
    if o.filledQty.Equal(o.quantity) {
        o.status = StatusFilled
    } else {
        o.status = StatusPartial
    }

    o.events = append(o.events, OrderFilled{
        OrderID: o.id,
        Qty:     qty,
        Price:   price,
    })
    return nil
}

func (o *Order) Cancel() error {
    if o.status == StatusFilled || o.status == StatusCancelled {
        return ErrOrderAlreadyTerminal
    }
    o.status = StatusCancelled
    o.events = append(o.events, OrderCancelled{OrderID: o.id})
    return nil
}

// Events are retrieved then purged after persistence
func (o *Order) PopEvents() []DomainEvent {
    events := o.events
    o.events = nil
    return events
}

Notice what's absent: no database access, no HTTP calls, no external dependencies. The aggregate is pure Go. It expresses business rules and generates events — that's it. Persistence is someone else's problem.

Anti-Corruption Layer: isolating external APIs

Binance returns prices as strings: "4.00000000". OKEx uses a different format. Coinbase yet another. If you let these external formats leak into your domain, you have a problem: your model becomes dependent on the whims of three different exchanges.

The Anti-Corruption Layer (ACL) is the translation boundary. Outside: the exchange's model. Inside: your domain. The ACL translates, and your domain never hears about Binance.


// External model — what Binance actually sends
type binanceOrderBook struct {
    LastUpdateID int64      `json:"lastUpdateId"`
    Bids         [][]string `json:"bids"` // [["price", "qty"], ...]
    Asks         [][]string `json:"asks"`
}

// ACL: translates Binance model to domain
func (c *BinanceClient) GetOrderBook(ctx context.Context, pair TradingPair) (*domain.OrderBook, error) {
    raw, err := c.fetchRaw(ctx, pair.String())
    if err != nil {
        return nil, fmt.Errorf("binance order book %s: %w", pair, err)
    }
    return translateBinanceOrderBook(raw)
}

func translateBinanceOrderBook(raw *binanceOrderBook) (*domain.OrderBook, error) {
    bids := make([]domain.PriceLevel, 0, len(raw.Bids))
    for _, b := range raw.Bids {
        if len(b) != 2 {
            return nil, fmt.Errorf("unexpected bid format: %v", b)
        }
        price, err := decimal.NewFromString(b[0])
        if err != nil {
            return nil, fmt.Errorf("invalid bid price %q: %w", b[0], err)
        }
        qty, err := decimal.NewFromString(b[1])
        if err != nil {
            return nil, fmt.Errorf("invalid bid qty %q: %w", b[1], err)
        }
        bids = append(bids, domain.PriceLevel{
            Price:    domain.Price{Amount: price, Currency: "USDT"},
            Quantity: qty,
        })
    }
    // same for asks...
    return &domain.OrderBook{Bids: bids}, nil
}

And for OKEx, the same interface, a different translation:


// External OKEx model — completely different format
type okexOrderBook struct {
    Data []struct {
        Asks      [][]string `json:"asks"` // [["price", "qty", "liquidated", "count"], ...]
        Bids      [][]string `json:"bids"`
        Timestamp string     `json:"ts"`
    } `json:"data"`
}

func (c *OKExClient) GetOrderBook(ctx context.Context, pair TradingPair) (*domain.OrderBook, error) {
    raw, err := c.fetchRaw(ctx, pair)
    if err != nil {
        return nil, fmt.Errorf("okex order book %s: %w", pair, err)
    }
    return translateOKExOrderBook(raw)
}

Your trading code has no idea who's providing the order book. It calls an OrderBookProvider interface and receives a domain.OrderBook. If Binance changes its API from v3 to v4 with a new response format, only translateBinanceOrderBook changes. Zero impact on business logic.


// Interface defined by the domain — not by the exchanges
type OrderBookProvider interface {
    GetOrderBook(ctx context.Context, pair TradingPair) (*OrderBook, error)
}

// The trading service has no dependency on Binance, OKEx or Coinbase
type TradingService struct {
    orderBook  OrderBookProvider
    orderRepo  OrderRepository
}

DDD and CQRS: how they fit together

If you've read the articles on CQRS aggregates and command handlers, you've already seen DDD in action without it being explicitly named.

DDD gives you the model: aggregates, Value Objects, domain events, bounded contexts. CQRS is the architectural pattern that exploits that model. The relationship is direct:

A Command expresses a business intent (PlaceOrder, CancelOrder)
The command handler loads the aggregate, calls a business method
The aggregate validates the invariant and emits a DomainEvent
The event is persisted, then propagated to read models (Portfolio, history)


// Command — business intent
type PlaceOrderCommand struct {
    Pair     TradingPair
    Side     Side
    Quantity decimal.Decimal
}

// Handler — orchestrates without business logic
func (h *PlaceOrderHandler) Handle(ctx context.Context, cmd PlaceOrderCommand) error {
    order, err := domain.NewOrder(cmd.Pair, cmd.Side, cmd.Quantity)
    if err != nil {
        return fmt.Errorf("invalid order: %w", err)
    }

    // Submit to exchange via ACL
    exchangeID, err := h.exchange.SubmitOrder(ctx, order)
    if err != nil {
        return fmt.Errorf("exchange submission: %w", err)
    }
    order.SetExchangeID(exchangeID)

    // Persist the aggregate and its events
    return h.orderRepo.Save(ctx, order)
}

Without DDD, the command handler accumulates business logic. With DDD, it orchestrates: load, call, persist. The logic belongs to the aggregate.

What DDD doesn't solve

DDD adds complexity. Abstraction layers, interfaces everywhere, translations between models. On a simple CRUD service — user list, a few REST endpoints, no complex business rules — it's clear over-engineering.

For a crypto trading engine with business rules spanning multiple exchanges, invariants to maintain, divergent external formats, and logic that evolves with the market — it pays off. The cost of abstraction is offset by the ability to change one piece without bringing down the others.

The most honest test: if a domain expert — a trader — can read your code and recognize their business concepts, DDD is working. If your code talks about BinanceWebsocketMessageParser and RestAPIResponseDTO, you've let technology invade the domain.

Conclusion

DDD is not an architecture, it's a modeling philosophy. It says: code should speak the language of the people who understand the problem, not the language of the people who understand the technology. In a crypto trading service, that means Order, Position, TradingPair — not BinanceResponseWrapper.

Bounded contexts provide the structure. Aggregates provide consistency. Anti-Corruption Layers provide isolation. And when you add CQRS on top, you have a system where each part has a clear responsibility and explicit boundaries.

Next time you write a command handler, ask yourself: does this logic belong in the handler or in the aggregate? If the answer is "in the aggregate", you're doing DDD.

Advanced PostgreSQL: JSONB, partial indexes and partitioning

Odilon HUGONNOT — Sat, 25 Apr 2026 09:00:03 +0000

A trades table with 50 million rows. The query "which active trades for this user this month" takes 4 seconds. After applying the four techniques in this article: 12 ms.

This isn't magic — it's a solid understanding of what PostgreSQL actually does under the hood. I already covered slow query diagnosis with EXPLAIN ANALYZE basics. Here we go further: the four levers that make a real difference on high-volume production tables.

JSONB: storing flexible schemas without sacrificing performance

Every crypto exchange exposes a different API. Binance returns a clientOrderId, Kraken a userref, Coinbase a client_order_id nested inside a sub-object. If you create a dedicated column for each field from each exchange, your table ends up with 40 columns where 35 are NULL for every single row.

JSONB solves this problem — provided you understand when to use it.

`json` vs `jsonb` — always use `jsonb`

json stores the raw text as-is. jsonb decomposes it into a binary representation at insert time. The consequences:

jsonb is indexable (GIN, expression indexes)
jsonb supports containment operators (@>, ?)
jsonb is slightly slower to write, but significantly faster to read

Simple rule: always use jsonb.

Essential operators

-- Access a key (returns jsonb)
SELECT exchange_meta -> 'exchange' FROM trades;

-- Access a key (returns text)
SELECT exchange_meta ->> 'exchange' FROM trades;

-- Containment: does the JSON contain this sub-object?
SELECT * FROM trades WHERE exchange_meta @> '{"exchange": "binance"}';

-- Key existence
SELECT * FROM trades WHERE exchange_meta ? 'client_order_id';

Adding the column and GIN index

-- Store exchange-specific metadata without extra columns
ALTER TABLE trades ADD COLUMN exchange_meta JSONB;

-- GIN index for containment queries (@> operator)
CREATE INDEX idx_trades_exchange_meta ON trades USING GIN (exchange_meta);

-- Query: all trades with a specific exchange order ID
SELECT * FROM trades WHERE exchange_meta @> '{"order_id": "12345"}';

-- Query: all Binance trades with status filled
SELECT * FROM trades WHERE exchange_meta @> '{"exchange": "binance", "status": "filled"}';

The GIN index on exchange_meta automatically covers all keys in your documents. No need to anticipate upfront which keys you'll query against.

When not to use JSONB

JSONB is not an excuse to skip data modeling. If you regularly filter on a field — user_id, status, created_at — that field must be a real column with a proper B-tree index. Putting those fields inside JSON forfeits all planner optimizations.

Rule: JSONB for variable data that is rarely filtered directly. Typed columns for everything that is frequently filtered, sorted, or joined.

Partial indexes: only index what matters

This is the most underused PostgreSQL feature. A partial index only covers rows that satisfy a WHERE clause. Its size can be 100x smaller than a full index — and it fits entirely in RAM.

The problem with full indexes

-- Full index on status: indexes ALL 50 MILLION rows,
-- including the 49.9 million closed trades nobody queries
CREATE INDEX idx_trades_status ON trades(status);

-- Approximate size
SELECT pg_size_pretty(pg_relation_size('idx_trades_status'));
-- → 1 GB

The solution: partial index on active rows only

-- Partial index: only the ~100,000 active trades
CREATE INDEX idx_trades_active ON trades(user_id, created_at)
WHERE status = 'active';

-- Actual size
SELECT pg_size_pretty(pg_relation_size('idx_trades_active'));
-- → 3 MB  ← fits entirely in shared memory

-- This query uses the partial index directly
SELECT * FROM trades
WHERE user_id = 123
  AND status = 'active'
ORDER BY created_at DESC;

PostgreSQL knows that a query with WHERE status = 'active' can use this index — the index predicate is a subset of the query condition. The planner selects the partial index automatically.

Other common use cases

-- Unverified emails (minority of users)
CREATE INDEX idx_users_unverified ON users(created_at)
WHERE verified = false;

-- Pending jobs in a queue
CREATE INDEX idx_jobs_pending ON jobs(priority, created_at)
WHERE status = 'pending';

-- Open orders
CREATE INDEX idx_orders_open ON orders(user_id, amount)
WHERE closed_at IS NULL;

The principle is always the same: identify the subset of rows your hot queries target, and index only that subset.

EXPLAIN ANALYZE: reading between the lines

"Seq Scan bad, Index Scan good" — that's the naive reading. The reality is more nuanced. Here's how to read an execution plan correctly.

The full command

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT * FROM trades
WHERE user_id = 123
  AND status = 'active'
ORDER BY created_at DESC
LIMIT 20;

ANALYZE actually executes the query and measures real timings. BUFFERS reports cache and disk activity. Never run this on a DELETE or UPDATE without wrapping it in a transaction.

Decoding the plan

Limit  (cost=0.56..18.42 rows=20 width=112) (actual time=0.124..0.198 rows=20 loops=1)
  ->  Index Only Scan Backward using idx_trades_active on trades
        (cost=0.56..2891.33 rows=3201 width=112)
        (actual time=0.121..0.183 rows=20 loops=1)
      Index Cond: (user_id = 123)
      Heap Fetches: 0
      Buffers: shared hit=5 read=0
Planning Time: 0.312 ms
Execution Time: 0.221 ms

The key nodes:

Index Only Scan — the best case. PostgreSQL reads the index without touching the heap (the physical table). Heap Fetches: 0 confirms zero table access. This is possible because all needed columns are covered by the index.
rows=3201 estimated vs rows=20 actual — a moderate discrepancy, acceptable. A 100x divergence means stale statistics; run ANALYZE trades;.
Buffers: shared hit=5 read=0 — everything served from memory cache, zero disk I/O. If read is high, your working set doesn't fit in RAM.

Pattern to watch for: the expensive Sort node

Sort  (cost=15234.12..15489.23 rows=102044 width=112)
      (actual time=1823.44..2104.12 rows=102044 loops=1)
  Sort Key: created_at DESC
  Sort Method: external merge  Disk: 8432kB

external merge Disk means the sort spilled to disk — work_mem is insufficient, or the index should include the sort column. Fix: include created_at in the index so rows come out pre-sorted.

-- Before: index without sort order
CREATE INDEX idx_trades_user ON trades(user_id) WHERE status = 'active';

-- After: composite index with native sort direction
CREATE INDEX idx_trades_user_date ON trades(user_id, created_at DESC)
WHERE status = 'active';
-- → The Sort node disappears from the plan

Date partitioning: the solution for massive tables

Beyond 10 million rows on an append-only table (trades, logs, events), date-range partitioning becomes necessary. The idea: physically split the table into sub-tables by time period. A query for "this month" only scans one partition instead of the whole thing.

Creating the partitioned table

-- Parent table — defines the structure and partition key
CREATE TABLE trades (
    id          BIGSERIAL,
    user_id     BIGINT       NOT NULL,
    pair        TEXT         NOT NULL,
    amount      NUMERIC(20, 8) NOT NULL,
    status      TEXT         NOT NULL DEFAULT 'active',
    exchange_meta JSONB,
    created_at  TIMESTAMPTZ  NOT NULL DEFAULT NOW()
) PARTITION BY RANGE (created_at);

-- Monthly partitions
CREATE TABLE trades_2026_01 PARTITION OF trades
    FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');

CREATE TABLE trades_2026_02 PARTITION OF trades
    FOR VALUES FROM ('2026-02-01') TO ('2026-03-01');

CREATE TABLE trades_2026_03 PARTITION OF trades
    FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');

Partition pruning in action

EXPLAIN SELECT * FROM trades
WHERE created_at >= '2026-03-01'
  AND created_at < '2026-04-01'
  AND user_id = 123;

Append  (cost=0.29..45.23 rows=18 width=156)
  Partitions: trades_2026_03
  ->  Index Scan using trades_2026_03_user_id_idx on trades_2026_03
        (cost=0.29..45.23 rows=18 width=156)
      Index Cond: (user_id = 123)

Partitions: trades_2026_03 — PostgreSQL eliminated all other partitions at planning time. It physically accesses only March 2026.

Operational benefits

Instant archiving: DROP TABLE trades_2024_01 removes a full month of data without a massive DELETE or table lock.
Targeted VACUUM: autovacuum runs partition by partition. Frozen old partitions are never reprocessed.
Local indexes: indexes created on the parent table are automatically created on each partition — each one smaller, faster to build, and more likely to fit in memory.

-- Creating an index applies to all existing and future partitions
CREATE INDEX ON trades(user_id, created_at DESC) WHERE status = 'active';

-- Archive January 2024 without locking the table
DROP TABLE trades_2024_01;

-- Or detach first, then archive
ALTER TABLE trades DETACH PARTITION trades_2024_01;

Combining all four

Here is the final state of the table after applying all four techniques:

-- 1. Table partitioned by month
CREATE TABLE trades (
    id            BIGSERIAL,
    user_id       BIGINT         NOT NULL,
    pair          TEXT           NOT NULL,
    amount        NUMERIC(20, 8) NOT NULL,
    status        TEXT           NOT NULL DEFAULT 'active',
    exchange_meta JSONB,                          -- 2. JSONB for variable metadata
    created_at    TIMESTAMPTZ    NOT NULL DEFAULT NOW()
) PARTITION BY RANGE (created_at);

-- Monthly partitions (automate with pg_partman in production)
CREATE TABLE trades_2026_03 PARTITION OF trades
    FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');

-- 3. Partial index: only active trades, composite with sort direction
CREATE INDEX idx_trades_active ON trades(user_id, created_at DESC)
WHERE status = 'active';

-- 4. GIN index for containment queries on exchange_meta
CREATE INDEX idx_trades_exchange_meta ON trades USING GIN (exchange_meta);

The query that used to take 4 seconds:

-- Before: Seq Scan over 50M rows, 4 seconds
SELECT * FROM trades
WHERE user_id = 123
  AND status = 'active'
  AND created_at >= date_trunc('month', NOW())
ORDER BY created_at DESC
LIMIT 20;

-- After: partition pruning + composite partial index
-- Execution Time: 12 ms

Index Only Scan Backward using idx_trades_active on trades_2026_03
  Index Cond: (user_id = 123)
  Filter: (created_at >= date_trunc('month', now()))
  Heap Fetches: 0
  Buffers: shared hit=4 read=0

The gain comes from cumulative effect: partitioning narrows the scope to a single monthly partition, the partial index only covers active trades within that partition, and the index column order eliminates the sort. PostgreSQL touches only the 4 memory pages it strictly needs.

Conclusion

The logical optimization order for a high-volume table:

Partial indexes first: immediate gain, zero schema change, spectacular impact on hot queries targeting a data subset.
JSONB for variable data: when external API responses have unpredictable fields, JSONB with a GIN index avoids a proliferation of NULL columns. It does not replace typed columns for frequently filtered fields.
Partitioning when the table is truly large: from 10M rows on append-only data. The upfront investment is higher (migration, partition automation, pg_partman), but the long-term operational benefits are substantial.

And at every step, EXPLAIN (ANALYZE, BUFFERS) to verify the planner is doing what you expect. Row estimates and the hit/read ratio don't lie.

PostgreSQL: debugging a slow query and optimizing it

Odilon HUGONNOT — Fri, 24 Apr 2026 09:00:03 +0000

It's Friday at 6:30 PM. Your query takes 4 seconds to respond in prod. The client sent you a screenshot of their loading screen. Your phone is ringing. There's a cold beer waiting for you. That's the context in which you're going to debug a slow PostgreSQL query.

Good news: 80% of PostgreSQL performance problems have an identifiable cause in under 10 minutes with the right tools. This guide follows the logical order of a real investigation — not an exhaustive PG feature catalog, but a method that works.

EXPLAIN vs EXPLAIN ANALYZE: one predicts, the other executes

The classic first mistake: using EXPLAIN alone and wondering why the numbers don't match what you observe in prod.

EXPLAIN displays the execution plan that PostgreSQL thinks it will use, based on the statistics it has in memory. It doesn't execute the query. Costs are estimates, rows are predictions.

EXPLAIN ANALYZE actually executes the query and enriches the plan with measured values: real time, actual rows processed, number of loops. That's what you want. Warning: it actually runs the query — don't run it on a DELETE or UPDATE without a BEGIN/ROLLBACK.

-- Safe for write queries
BEGIN;
EXPLAIN ANALYZE
    UPDATE orders SET status = 'processed' WHERE created_at < NOW() - INTERVAL '30 days';
ROLLBACK;

-- For a SELECT, no special precautions needed
EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
    SELECT u.email, COUNT(o.id) AS order_count
    FROM users u
    JOIN orders o ON o.user_id = u.id
    WHERE u.created_at >= '2025-01-01'
    GROUP BY u.email
    ORDER BY order_count DESC
    LIMIT 20;

The BUFFERS option is valuable: it shows how many blocks were read from cache (shared hit) vs from disk (read). A query with lots of disk reads on a table that should be cached is a strong signal.

What to look for in the output

Three things to spot immediately:

Seq Scan on a large table: PostgreSQL scans every row one by one. On a 50,000-row table, this is often acceptable. On 10 million, it's not.
Large gap between rows estimated and rows actual: if PostgreSQL expected to filter 50 rows and processed 80,000, its statistics are stale or the query is poorly written.
Node with disproportionate time: the plan is a tree. The slowest node is the culprit. The displayed time is cumulative — subtract the time of child nodes to isolate a node's own cost.

-- Example of problematic EXPLAIN ANALYZE output
Seq Scan on orders  (cost=0.00..48520.00 rows=2000 width=120)
                    (actual time=0.042..3841.223 rows=1847291 loops=1)
  Filter: (status = 'pending' AND created_at > '2024-01-01')
  Rows Removed by Filter: 152709
Planning Time: 1.2 ms
Execution Time: 4102.8 ms

-- What we see:
-- 1. Seq Scan on "orders" → no index used
-- 2. rows estimated = 2,000, rows actual = 1,847,291 → catastrophic statistics
-- 3. 4 seconds → matches exactly what the client is seeing

The usual suspects

1. Missing index on a filtered column

The most common case. An orders table with 2 million rows, a query filtering on user_id, no index. PostgreSQL does a full Seq Scan. The fix takes 30 seconds.

-- The slow query
SELECT id, total, status
FROM orders
WHERE user_id = 42
ORDER BY created_at DESC;

-- EXPLAIN ANALYZE shows:
-- Seq Scan on orders  (actual time=0.031..2341.5 rows=47 loops=1)
-- Filter: (user_id = 42)
-- Rows Removed by Filter: 1999953

-- The fix
CREATE INDEX CONCURRENTLY idx_orders_user_id ON orders(user_id);
-- CONCURRENTLY: creates the index without locking the table for writes

-- After creation, the same query:
-- Index Scan using idx_orders_user_id on orders
-- (actual time=0.041..0.189 rows=47 loops=1)
-- 2.3 seconds → 0.2 ms

2. Index exists but isn't used

A trickier situation: the index exists, but PostgreSQL doesn't use it. Three main reasons.

Function in the WHERE clause: the moment you apply a function to an indexed column, the index is unusable — PostgreSQL can't scan the index on the transformed value.

-- Index on email, but unused
CREATE INDEX idx_users_email ON users(email);

-- The index does nothing here
SELECT * FROM users WHERE LOWER(email) = 'alice@example.com';

-- Expression index — solves the problem
CREATE INDEX idx_users_email_lower ON users(LOWER(email));

-- Or rewrite the query if data is already lowercase
SELECT * FROM users WHERE email = 'alice@example.com';

LIKE with leading wildcard: LIKE '%dupont' can't use a B-tree index, because it requires scanning all values to find those that end with "dupont". LIKE 'dupont%', on the other hand, can.

-- Guaranteed Seq Scan, even with an index on last_name
SELECT * FROM customers WHERE last_name LIKE '%martin';

-- Uses the B-tree index
SELECT * FROM customers WHERE last_name LIKE 'martin%';

-- For "contains" searches, use pg_trgm + GIN
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX idx_customers_last_name_trgm
    ON customers USING GIN(last_name gin_trgm_ops);

-- Now this LIKE uses the index
SELECT * FROM customers WHERE last_name LIKE '%martin%';

Poor cardinality: if a column has very few distinct values (e.g. status with 3 possible values on 2 million rows), PostgreSQL may estimate that a Seq Scan is faster than an Index Scan followed by 600,000 random lookups. It's often right. The solution: a partial index.

-- 2 million orders, 95% have status = 'completed', 5% status = 'pending'
-- A simple index on status is not very useful for 'completed'
-- But very useful for 'pending' (100,000 rows out of 2 million)

-- Partial index: only indexes pending rows
CREATE INDEX idx_orders_pending
    ON orders(created_at)
    WHERE status = 'pending';

-- This query now uses the partial index
SELECT id, user_id, total
FROM orders
WHERE status = 'pending'
  AND created_at > NOW() - INTERVAL '7 days'
ORDER BY created_at DESC;

3. The N+1 problem via ORM

Not specific to PostgreSQL, but PostgreSQL suffers from it as much as any other database. The ORM loads 100 articles, then for each article makes a separate query to load the author. Result: 101 queries instead of one.

-- What the ORM generates (N+1)
SELECT * FROM articles LIMIT 100;
SELECT * FROM users WHERE id = 1;
SELECT * FROM users WHERE id = 2;
-- ... 98 more times

-- What it should do
SELECT a.id, a.title, a.body, a.published_at,
       u.id AS author_id, u.name AS author_name, u.avatar_url
FROM articles a
JOIN users u ON u.id = a.author_id
ORDER BY a.published_at DESC
LIMIT 100;

Detection happens in the logs or via pg_stat_statements (see below): dozens of identical queries with just one parameter changing, executed in rapid succession. In Go: plain SQL with a single JOIN. In Laravel/Django: eager loading.

4. Stale statistics

PostgreSQL maintains statistics on data distribution in each table (via autovacuum). If these statistics are outdated — after a large import, a bulk DELETE — the planner makes bad decisions. Symptom: large gap between rows estimated and rows actual in EXPLAIN ANALYZE.

-- Update statistics without blocking writes
ANALYZE orders;

-- After a bulk load or data purge
VACUUM ANALYZE orders;

-- Check statistics freshness
SELECT schemaname, tablename, last_analyze, last_autoanalyze, n_live_tup, n_dead_tup
FROM pg_stat_user_tables
WHERE tablename = 'orders';

5. Too many rows returned

Sometimes the query is correctly indexed but returns 50,000 rows when you're displaying 20. Add LIMIT and paginate properly. For sorted lists on large volumes, offset-based pagination itself becomes a problem beyond a few thousand pages: OFFSET 100000 LIMIT 20 forces PG to scan 100,020 rows to return 20.

-- Offset pagination — slow on large pages
SELECT id, title, published_at
FROM articles
ORDER BY published_at DESC
LIMIT 20 OFFSET 10000;

-- Cursor pagination (keyset pagination) — fast regardless of page number
SELECT id, title, published_at
FROM articles
WHERE published_at < '2024-06-15 10:23:00'  -- last value from the previous page
ORDER BY published_at DESC
LIMIT 20;

Choosing the right index type

PostgreSQL offers several index types. In practice, you use three of them.

B-tree (default)

Covers 95% of cases: equality, inequality, sorting, range queries. It's the default when you write CREATE INDEX without specifying a type. Effective on high-cardinality columns (identifiers, emails, dates, amounts).

CREATE INDEX idx_orders_created_at ON orders(created_at);
CREATE INDEX idx_users_email ON users(email);

-- Composite index: column order matters.
-- Useful for queries that filter on user_id AND sort on created_at.
CREATE INDEX idx_orders_user_created ON orders(user_id, created_at DESC);

GIN — Full-text search and JSONB

For searches in text (tsvector), arrays, or JSONB columns. A B-tree index on a JSONB field doesn't help for queries like WHERE metadata @> '{"role": "admin"}'.

-- GIN index for JSONB queries
CREATE INDEX idx_users_metadata ON users USING GIN(metadata);

-- Now efficient
SELECT * FROM users WHERE metadata @> '{"role": "admin", "active": true}';

-- GIN index for full-text search
ALTER TABLE articles ADD COLUMN search_vector tsvector;
UPDATE articles SET search_vector =
    to_tsvector('english', COALESCE(title, '') || ' ' || COALESCE(body, ''));

CREATE INDEX idx_articles_search ON articles USING GIN(search_vector);

SELECT title FROM articles
WHERE search_vector @@ plainto_tsquery('english', 'postgresql optimization index');

BRIN — Sequential timestamps

BRIN (Block Range INdex) is tiny in size and highly efficient for columns whose values grow naturally with insertion: timestamps, sequential IDs. The principle: instead of indexing each value, it stores min/max values per data block. On a log table of several hundred GB, it can drastically reduce query time with an index of just a few MB.

-- Log table with 500 million rows.
-- A B-tree index on created_at would weigh ~15 GB.
-- A BRIN index weighs a few MB.
CREATE INDEX idx_logs_created_at_brin ON access_logs USING BRIN(created_at);

-- Efficient for time range queries
SELECT COUNT(*), path
FROM access_logs
WHERE created_at >= '2026-02-01' AND created_at < '2026-03-01'
GROUP BY path
ORDER BY count DESC;

pg_stat_statements: finding expensive queries in production

Up to now we knew which query to analyze. In production, you often don't. pg_stat_statements accumulates statistics on all executed queries: total time, call count, mean time. This is where you find queries that run 10,000 times per hour and take 200ms each — less visible than a 4-second query, but far more impactful in aggregate.

-- Enable the extension (requires a PostgreSQL restart)
-- In postgresql.conf: shared_preload_libraries = 'pg_stat_statements'
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

-- The 10 most expensive queries by total time
SELECT
    LEFT(query, 100) AS query_short,
    calls,
    ROUND(total_exec_time::numeric, 2) AS total_ms,
    ROUND(mean_exec_time::numeric, 2) AS mean_ms,
    ROUND(stddev_exec_time::numeric, 2) AS stddev_ms,
    rows
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;

-- Queries with the highest standard deviation (time instability)
-- Possible symptom: different plan depending on parameters, or random cache miss
SELECT
    LEFT(query, 100) AS query_short,
    calls,
    ROUND(mean_exec_time::numeric, 2) AS mean_ms,
    ROUND(stddev_exec_time::numeric, 2) AS stddev_ms
FROM pg_stat_statements
WHERE calls > 100
ORDER BY stddev_exec_time DESC
LIMIT 10;

-- Reset stats (useful after a deployment or optimization)
SELECT pg_stat_statements_reset();

pg_stat_statements doesn't interrupt service and has no measurable performance impact. There's no reason not to enable it in production on all your PostgreSQL clusters.

Quick tips

Force PostgreSQL to show the plan with an index

Sometimes PostgreSQL chooses a Seq Scan when an index exists, and you want to see what the plan would look like with the index. SET enable_seqscan = off forces the planner to prefer index scans for the current session. Useful for diagnosis, not as a permanent fix.

-- In your session only — no global change
SET enable_seqscan = off;

EXPLAIN ANALYZE
    SELECT * FROM orders WHERE user_id = 42;

-- Restore to normal
SET enable_seqscan = on;

-- Interpretation:
-- If the plan with index is faster → create the index or run ANALYZE (stale stats).
-- If the plan with index is slower → PostgreSQL was right to do the Seq Scan.
--   Perhaps the query returns too many rows for an index to help.

work_mem for sorts and hash joins

When you see Sort Method: external merge Disk in EXPLAIN ANALYZE, PostgreSQL ran out of memory and had to write to disk for sorting. Increasing work_mem for the session can fix the problem immediately.

-- Default work_mem = 4MB — insufficient for large aggregations
SET work_mem = '64MB';

EXPLAIN ANALYZE
    SELECT user_id, SUM(total) AS revenue
    FROM orders
    WHERE created_at >= '2025-01-01'
    GROUP BY user_id
    ORDER BY revenue DESC;

-- With sufficient work_mem:
-- Sort Method: quicksort  Memory: 12kB  ← all in RAM, fast

-- Without sufficient work_mem:
-- Sort Method: external merge  Disk: 48MB  ← writes to disk, slow

-- Warning: don't increase work_mem globally without calculation.
-- It applies PER sort operation AND per concurrent connection.
-- 100 connections × 5 sorts × 64MB = potentially 32 GB consumed.
-- Reserve it for sessions that need it, or adjust max_connections.

VACUUM ANALYZE after a bulk load

-- After an import or bulk DELETE
VACUUM ANALYZE orders;

-- VACUUM reclaims space from dead tuples (after UPDATE/DELETE)
-- ANALYZE updates planner statistics
-- Both together: essential after any large data modification

-- Check table bloat
SELECT n_dead_tup, n_live_tup,
       ROUND(100.0 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 1) AS dead_pct
FROM pg_stat_user_tables
WHERE tablename = 'orders';
-- If dead_pct > 10-20%, a VACUUM is needed

The method in summary

When a PostgreSQL query is slow, here's the investigation order that resolves the vast majority of cases:

EXPLAIN (ANALYZE, BUFFERS) on the offending query. Identify the slowest node. Look for Seq Scan on a large table, gap between rows estimated / actual.
Missing index? Create it with CONCURRENTLY in production. Verify the query is written to benefit from it — no function on the indexed column, no LIKE '%...'.
Stale statistics? VACUUM ANALYZE on the affected table.
In production with no specific query to target? pg_stat_statements to identify offenders by total time or call count.

80% of PostgreSQL performance problems are solved with EXPLAIN ANALYZE and a well-placed index. The remaining 20% involve configuration tuning (shared_buffers, work_mem, max_connections), rewriting complex queries, or table partitioning — but that's a topic for another article.

The beer can wait another 10 minutes. The client can't.

📄 Associated CLAUDE.md

View • Download • Catalog

CQRS in Go — Part 4: PostgreSQL as an event store

Odilon HUGONNOT — Thu, 23 Apr 2026 09:00:02 +0000

CQRS in Go series:

Part 1: the aggregate, Transition() and Clone()
Part 2: command handlers without side effects
Part 3: sagas and event choreography
Part 4: PostgreSQL as an event store

The first three parts laid the groundwork: immutable aggregates, pure command handlers, sagas through choreography. One central question remains — where do we persist the events? This part answers that question using PostgreSQL as an append-only event store, with no additional dependencies.

Why PostgreSQL and not EventStoreDB or Kafka

EventStoreDB is an excellent tool, designed specifically for event sourcing. But it's an additional component to deploy, monitor, back up, and maintain. For a team already running PostgreSQL in production, adding EventStoreDB means doubling the operational footprint for a database.

Kafka is often mentioned in this context. It's a message bus, not an event store. Reading by aggregate ID is not its strong suit — Kafka is optimized for sequential partition consumption, not for "give me all events for order CMD-4521 in order". Default retention is time-limited. Reconstituting an aggregate from Kafka requires non-trivial work.

PostgreSQL, you already have it. Native ACID, JSONB with indexing, backups integrated into your existing infrastructure, monitoring your team knows, pg_dump, replication, point-in-time recovery. For 90% of projects, an append-only table in PostgreSQL is more than enough. Moving to a dedicated tool makes sense when you're processing millions of events per second — a threshold most projects never reach.

The es_events table — the core

Everything rests on a single table:

CREATE TABLE es_events (
    id            BIGSERIAL PRIMARY KEY,
    aggregate_id  UUID NOT NULL,
    aggregate_type VARCHAR(64) NOT NULL,
    event_type    VARCHAR(128) NOT NULL,
    event_data    JSONB NOT NULL,
    metadata      JSONB DEFAULT '{}',
    version       INT NOT NULL,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    UNIQUE (aggregate_id, version)
);

CREATE INDEX idx_es_events_aggregate ON es_events (aggregate_id, version);

Each column has a specific role:

id (BIGSERIAL): global position in the stream. This is the cursor used by projectors to track where they are. Monotonically increasing, never reused.
aggregate_id: the identifier of the entity involved — the UUID of the order, payment, or shipment. This is the primary read key for reconstituting an aggregate.
aggregate_type: the type of the aggregate — "order", "payment", "shipping". Allows filtering events by domain and avoids UUID collisions between different types.
event_type: the name of the event — "OrderPlaced", "PaymentConfirmed", "ShipmentDispatched". Used to deserialize event_data into the correct Go type.
event_data (JSONB): the serialized event payload. JSONB enables indexing and field-level queries when needed, without sacrificing flexibility.
metadata (JSONB): traceability information — who triggered the action (user ID), correlation ID, causation ID, client timestamp. Not domain data, but indispensable in production.
version: sequence number per aggregate. The first event of an aggregate is at version 1, the second at version 2, and so on.

The UNIQUE (aggregate_id, version) constraint is the centerpiece of concurrent safety. It makes it impossible to insert two events with the same version on the same aggregate. This is the optimistic locking mechanism — no read lock, a constraint violation on write in case of conflict.

Append — writing an event

type EventStore struct {
    db *sqlx.DB
}

func (s *EventStore) Append(ctx context.Context, aggregateID uuid.UUID, aggregateType string, expectedVersion int, events []StorableEvent) error {
    tx, err := s.db.BeginTxx(ctx, nil)
    if err != nil {
        return fmt.Errorf("begin tx: %w", err)
    }
    defer tx.Rollback()

    for i, event := range events {
        version := expectedVersion + i + 1

        data, err := json.Marshal(event.Data)
        if err != nil {
            return fmt.Errorf("marshal event: %w", err)
        }

        _, err = tx.ExecContext(ctx, `
            INSERT INTO es_events (aggregate_id, aggregate_type, event_type, event_data, metadata, version)
            VALUES ($1, $2, $3, $4, $5, $6)
        `, aggregateID, aggregateType, event.Type, data, event.Metadata, version)

        if err != nil {
            // UNIQUE violation = version conflict = optimistic locking
            if isUniqueViolation(err) {
                return ErrConcurrencyConflict
            }
            return fmt.Errorf("insert event: %w", err)
        }
    }

    return tx.Commit()
}

The expectedVersion parameter is the number of the last known event at the time the aggregate was loaded. If the handler produces two events, they will be inserted at versions expectedVersion + 1 and expectedVersion + 2.

If another goroutine wrote an event between loading and writing, the UNIQUE constraint kicks in — PostgreSQL rejects the insert with a constraint violation, translated into ErrConcurrencyConflict. The client can retry: it will reload the aggregate with the new event, recalculate the state, and re-execute the handler. This is optimistic locking — no read lock, just a check at write time.

Load — reloading an aggregate

func (s *EventStore) Load(ctx context.Context, aggregateID uuid.UUID) ([]StoredEvent, error) {
    var events []StoredEvent

    err := s.db.SelectContext(ctx, &events, `
        SELECT id, aggregate_id, aggregate_type, event_type, event_data, metadata, version, created_at
        FROM es_events
        WHERE aggregate_id = $1
        ORDER BY version ASC
    `, aggregateID)

    if err != nil {
        return nil, fmt.Errorf("loading events: %w", err)
    }

    return events, nil
}

The aggregate's state is never stored directly. It's always recalculated by replaying events in order, calling Transition() on each one (see Part 1). This is the replay. The result is deterministic — the same events always produce the same state. This is what makes the system testable and auditable.

Subscriptions — feeding projections

Projectors — the components that maintain read views — must be notified of new events. The simplest solution: a positions table.

CREATE TABLE es_subscriptions (
    subscriber_id  VARCHAR(128) PRIMARY KEY,
    last_event_id  BIGINT NOT NULL DEFAULT 0,
    updated_at     TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Each projector records the identifier of the last event it processed. On startup or after a crash, it resumes from this position. The polling pattern:

type Subscriber struct {
    db           *sqlx.DB
    subscriberID string
    handler      EventHandler
}

func (s *Subscriber) Poll(ctx context.Context) error {
    // 1. Read the last processed position
    var lastID int64
    s.db.GetContext(ctx, &lastID,
        "SELECT last_event_id FROM es_subscriptions WHERE subscriber_id = $1",
        s.subscriberID,
    )

    // 2. Load new events
    var events []StoredEvent
    s.db.SelectContext(ctx, &events,
        "SELECT * FROM es_events WHERE id > $1 ORDER BY id ASC LIMIT 100",
        lastID,
    )

    // 3. Process within a transaction
    for _, event := range events {
        tx, _ := s.db.BeginTxx(ctx, nil)

        if err := s.handler.Handle(ctx, tx, event); err != nil {
            tx.Rollback()
            return fmt.Errorf("handling event %d: %w", event.ID, err)
        }

        // 4. Update the position in the same transaction
        tx.ExecContext(ctx,
            "UPDATE es_subscriptions SET last_event_id = $1, updated_at = NOW() WHERE subscriber_id = $2",
            event.ID, s.subscriberID,
        )

        tx.Commit()
    }

    return nil
}

The critical detail is in point 4: the position update and event processing are in the same transaction. If the process crashes between processing and updating the position, the event will be reprocessed on the next poll. This is at-least-once delivery — handlers must be idempotent to absorb duplicates without side effects.

Idempotency in practice: a handler that inserts a row into a read view can use INSERT ... ON CONFLICT DO NOTHING, or check whether the row already exists. The deduplication key is generally the event_id or a combination of unique business fields.

Real-time notifications with LISTEN/NOTIFY

Polling introduces latency — at best a few hundred milliseconds if the poll runs frequently, at worst several seconds. For read views that need to be reactive, PostgreSQL offers a native mechanism: LISTEN/NOTIFY.

// Event store side: notify after each INSERT
func (s *EventStore) Append(ctx context.Context, aggregateID uuid.UUID, aggregateType string, expectedVersion int, events []StorableEvent) error {
    tx, err := s.db.BeginTxx(ctx, nil)
    if err != nil {
        return fmt.Errorf("begin tx: %w", err)
    }
    defer tx.Rollback()

    // ... insert events ...

    // Notify subscribers
    if _, err := tx.ExecContext(ctx, "NOTIFY es_events_channel"); err != nil {
        return fmt.Errorf("notify: %w", err)
    }

    return tx.Commit()
}

// Subscriber side: listen instead of polling
func (s *Subscriber) Listen(ctx context.Context) error {
    conn, err := s.db.Conn(ctx)
    if err != nil {
        return fmt.Errorf("acquire conn: %w", err)
    }
    defer conn.Close()

    if _, err := conn.ExecContext(ctx, "LISTEN es_events_channel"); err != nil {
        return fmt.Errorf("listen: %w", err)
    }

    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        default:
        }

        // Wait for notification (blocking with timeout)
        _, err := conn.WaitForNotification(ctx)
        if err != nil {
            return fmt.Errorf("wait notification: %w", err)
        }

        // Poll immediately after the notification
        if err := s.Poll(ctx); err != nil {
            return err
        }
    }
}

The notification is sent inside the Append transaction — it's only delivered to listeners once the transaction commits. No false alarms on rollback.

In production, combine both approaches: LISTEN for real-time reactivity under normal conditions, periodic polling (every 5 to 30 seconds) as a safety net in case of reconnection or missed notification. The dedicated LISTEN connection must not be taken from the standard connection pool — it occupies a connection permanently.

The command gateway — putting it all together

With the event store in place, the complete flow for a command becomes explicit:

type CommandGateway struct {
    store *EventStore
}

func (gw *CommandGateway) Execute(ctx context.Context, aggregateID uuid.UUID, cmd Command, handler CommandHandler) error {
    // 1. Load events
    storedEvents, err := gw.store.Load(ctx, aggregateID)
    if err != nil {
        return fmt.Errorf("load aggregate: %w", err)
    }

    // 2. Reconstruct state
    state := Replay(storedEvents)
    currentVersion := len(storedEvents)

    // 3. Execute the handler (pure, no side effects)
    newEvents, err := handler.Handle(ctx, state, cmd)
    if err != nil {
        return err
    }

    // 4. Persist the new events
    return gw.store.Append(ctx, aggregateID, cmd.AggregateType(), currentVersion, newEvents)
}

If two commands arrive simultaneously on the same aggregate, the second fails with ErrConcurrencyConflict. The client can retry — it will reload the aggregate including the events from the first command, recalculate the state, and re-execute the handler on the updated state. In practice, real conflicts are rare on well-bounded aggregates; most concurrent operations touch different aggregates.

This is also the pattern that makes command handlers testable without infrastructure (Part 2): the gateway injects the state, the handler produces events, the gateway persists. Each responsibility is isolated.

Performance — when events accumulate

An aggregate with a few dozen events reloads in microseconds. An aggregate with 10,000 events — a long-lived order, a very active user account — starts to weigh on the replay. The solution is snapshots.

CREATE TABLE es_snapshots (
    aggregate_id   UUID PRIMARY KEY,
    aggregate_type VARCHAR(64) NOT NULL,
    version        INT NOT NULL,
    state_data     JSONB NOT NULL,
    created_at     TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

A snapshot is a captured, serialized state at a given point in time, associated with its version. Loading with a snapshot proceeds in two steps:

Load the most recent snapshot for the aggregate (if it exists)
Load only the events after the snapshot's version
Replay the partial events on the snapshot's state

The simplest practical rule: create a snapshot every 100 events. Most aggregates never reach this limit — a typical e-commerce aggregate generates 5 to 20 events over its lifetime. Snapshots are only necessary for high-volume aggregates: accounts, wallets, inventories updated hundreds of times per day.

Important: snapshots are an optimization, not a change to the data model. Events remain the source of truth. A corrupted or deleted snapshot is not data loss — the aggregate can be reconstituted from the beginning of its events.

Series summary

Across four parts, we built a complete CQRS implementation in Go:

Part 1 — The aggregate: immutable state, Transition() for applying events, Clone() for dry-runs. The foundation that makes everything else testable.
Part 2 — Command handlers: pure functions that take a state and a command, return events or an error. No database, no HTTP, testable in complete isolation.
Part 3 — Sagas and choreography: coordination between aggregates through published events. Each service reacts to what interests it, without direct coupling. Sagas handle compensation in case of distributed failure.
Part 4 — The event store: PostgreSQL as the persistence infrastructure. Append-only table, optimistic locking through UNIQUE constraint, subscriptions for projectors, LISTEN/NOTIFY for reactivity, snapshots for large-scale performance.

These four patterns combine: the command gateway loads via the event store, calls the pure handler, persists the produced events. Subscribers read the global stream and maintain read views. Sagas listen for certain event types and trigger commands on other aggregates.

The result is a system that is auditable by construction — the complete history is in the events —, testable at every layer, and operable with the PostgreSQL infrastructure you already manage.

📄 Associated CLAUDE.md

View • Download • Catalogue

CQRS in Go — Part 3: sagas and event choreography

Odilon HUGONNOT — Wed, 22 Apr 2026 09:00:03 +0000

CQRS in Go series:

Part 1: the aggregate, Transition() and Clone()
Part 2: command handlers without side effects
Part 3: sagas and event choreography
Part 4: PostgreSQL as an event store

The problem — two aggregates need to cooperate

Concrete scenario: a payment is validated. The Payment aggregate emits a PaymentConfirmed event. In response, a shipment needs to be created in the Shipping aggregate.

The problem: the two aggregates don't know each other, and that's intentional. An aggregate that directly calls another aggregate is a fundamental violation of the pattern. The aggregate is an isolated unit of consistency. If Payment knows about Shipping, you're recreating tight coupling — exactly what you were trying to avoid by leaving the classic transactional model.

The question is: how do you make Shipping react to something that happened in Payment, without one knowing the other? There are two families of answers: orchestration and choreography.

Orchestration vs choreography

Orchestration sets up a central conductor — a SagaManager — that knows all the steps, all transitions, and all possible rollbacks. When PaymentConfirmed arrives, the SagaManager explicitly calls the Shipping service, then the Email service, then the Stock service. The flow is readable in one place. In case of error, the manager knows exactly what to compensate.

The downside: the SagaManager knows everyone. Adding a new step to the flow forces you to modify this central component. It becomes a tight coupling point between domains that should ignore each other.

Choreography works differently. Each aggregate emits events. Independent handlers listen to these events and trigger commands on other aggregates. Nobody centralizes knowledge of the flow — it emerges from the sum of reactions.

Choreography wins in most cases because it scales better, adding new behavior on an existing event doesn't touch existing code, and each handler can evolve, be redeployed, or even fail without blocking the others. The price to pay: the flow is not readable in one place — we'll come back to that in section 6.

The event handler — the link between aggregates

The central tool of choreography is the event handler. It listens to events from one aggregate and triggers commands on another. Its responsibility is narrow: translate an event into a command. Nothing more.

// PaymentEventHandler reacts to Payment events to trigger Shipping.
type PaymentEventHandler struct {
    shippingCmd ShippingCommandService
}

func (h PaymentEventHandler) Handle(ctx context.Context, event DomainEvent) error {
    switch e := event.(type) {
    case PaymentConfirmed:
        return h.shippingCmd.CreateShipment(ctx, CreateShipmentCmd{
            OrderID:    e.OrderID,
            CustomerID: e.CustomerID,
            Items:      e.Items,
            Address:    e.ShippingAddress,
        })
    }
    return nil
}

The event handler contains no business logic. It doesn't decide whether the shipment should be created — that decision was already made by the fact that PaymentConfirmed was emitted. It doesn't validate data — that's the Shipping command handler's job. It translates, nothing more.

If the logic in an event handler starts to grow — conditions, branches, multiple calls — it's a signal that an aggregate is missing somewhere, or that a responsibility hasn't been properly carved out.

Saga idempotency

An event can be redelivered. This happens whenever a handler successfully processes an event but crashes before acknowledging the message. The broker (Kafka, RabbitMQ, or even a simple queue table in PostgreSQL) doesn't know the processing succeeded — it redelivers. If the saga isn't idempotent, you create two shipments for a single order.

The simplest solution: check before acting. If a shipment already exists for this OrderID, treat it as a duplicate and ignore it.

func (h PaymentEventHandler) Handle(ctx context.Context, event DomainEvent) error {
    switch e := event.(type) {
    case PaymentConfirmed:
        err := h.shippingCmd.CreateShipment(ctx, CreateShipmentCmd{
            OrderID:    e.OrderID,
            CustomerID: e.CustomerID,
            Items:      e.Items,
            Address:    e.ShippingAddress,
        })
        // If the shipment already exists, it's a duplicate — ignore it.
        if errors.Is(err, ErrShipmentAlreadyExists) {
            return nil
        }
        return err
    }
    return nil
}

This ErrShipmentAlreadyExists error must be a sentinel error defined in the Shipping package, and the CreateShipment command handler must check uniqueness on OrderID before inserting — ideally with a unique constraint in the database to make the check atomic.

The other approach is an idempotency key: pass the event's ID as an idempotency key to the command handler, which stores it and silently rejects any duplicate with the same key. This is more generic but requires a dedicated tracking table. Both approaches are valid; the choice often depends on whether the command handler is already instrumented for idempotency keys (see Part 2 of the series).

Partial failures

An event can trigger multiple independent reactions: create the shipment, send the confirmation email, decrement stock. If all these reactions are in a single handler and the email fails, the shipment is created but the stock is never updated. The handler failed midway and the final state is inconsistent.

// Bad: one big handler doing everything sequentially.
func (h BigHandler) Handle(ctx context.Context, event DomainEvent) error {
    switch e := event.(type) {
    case PaymentConfirmed:
        if err := h.shipping.Create(ctx, toShipmentCmd(e)); err != nil {
            return err // email and stock will never be processed
        }
        if err := h.email.Send(ctx, toConfirmationEmail(e)); err != nil {
            return err // stock will never be processed
        }
        if err := h.stock.Update(ctx, toStockCmd(e)); err != nil {
            return err
        }
    }
    return nil
}

The solution is to never put multiple responsibilities in the same handler. One handler, one action.

// One handler per responsibility. Each can fail independently.

type ShippingOnPayment struct {
    shippingCmd ShippingCommandService
}

func (h ShippingOnPayment) Handle(ctx context.Context, event DomainEvent) error {
    if e, ok := event.(PaymentConfirmed); ok {
        return h.shippingCmd.CreateShipment(ctx, CreateShipmentCmd{
            OrderID: e.OrderID,
            Items:   e.Items,
            Address: e.ShippingAddress,
        })
    }
    return nil
}

type EmailOnPayment struct {
    mailer EmailService
}

func (h EmailOnPayment) Handle(ctx context.Context, event DomainEvent) error {
    if e, ok := event.(PaymentConfirmed); ok {
        return h.mailer.SendConfirmation(ctx, e.CustomerEmail, e.OrderID)
    }
    return nil
}

type StockOnPayment struct {
    stockCmd StockCommandService
}

func (h StockOnPayment) Handle(ctx context.Context, event DomainEvent) error {
    if e, ok := event.(PaymentConfirmed); ok {
        return h.stockCmd.DecrementStock(ctx, DecrementStockCmd{
            Items: e.Items,
        })
    }
    return nil
}

These three handlers are registered separately on the PaymentConfirmed event. If the email fails, it will be retried independently — without blocking the shipment or the stock update. A failure in one branch doesn't poison the others.

Retry with exponential backoff is handled by the infrastructure (broker, worker), not the handler. The handler has one responsibility: try to perform its action and return an error if it fails. The system decides how many times to retry.

Eventual consistency is an accepted consequence. After a crash, the stock will be updated on the next retry — in a few seconds or minutes depending on the retry policy. For most e-commerce use cases, this is acceptable. If it's not acceptable, you need a distributed transaction, and you're leaving the scope of choreography.

Tracing the flow — the event/handler table

In choreography, the flow is not readable in any single file. It's distributed across dozens of handlers. Without documentation, it's impossible to know what gets triggered when PaymentConfirmed is emitted. That's the cost of choreography.

The practical answer is a mapping table, maintained manually or generated from handler registrations. For each event, list the handlers and the action they perform. This is the system's map.

Event

Handler

Action

PaymentConfirmed

ShippingOnPayment

Creates the shipment

PaymentConfirmed

EmailOnPayment

Order confirmation email

PaymentConfirmed

StockOnPayment

Decrements stock

ShipmentCreated

EmailOnShipment

"Your package has shipped" email

ShipmentCreated

TrackingOnShipment

Creates the delivery tracking

This table should live in the project documentation, not in a comment buried in some file. It answers the question "what happens when this event is emitted?" — a question you ask frequently in production when something goes wrong.

In Go, you can also centralize handler registrations in a wire.go or handlers.go file per domain, which makes the mapping readable in code:

func registerPaymentHandlers(bus EventBus, deps Dependencies) {
    bus.Register(PaymentConfirmed{}, ShippingOnPayment{shippingCmd: deps.ShippingCmd})
    bus.Register(PaymentConfirmed{}, EmailOnPayment{mailer: deps.Mailer})
    bus.Register(PaymentConfirmed{}, StockOnPayment{stockCmd: deps.StockCmd})
}

All handlers tied to PaymentConfirmed are visible in one function. No need to grep the entire codebase to find what reacts to this event.

When choreography isn't enough

Choreography has one limit: flows with compensation. Compensation = undoing an already-completed step because a subsequent step failed. The classic example: booking a hotel, a flight, and a rental car for a trip. If the flight is unavailable, you need to cancel the hotel reservation that already succeeded.

In pure choreography, this scenario forces you to emit a FlightBookingFailed event and have a handler that cancels the hotel. This works, but the handler needs to know whether the hotel was booked for this trip — which implies shared state. You're rebuilding an orchestrator, but implicitly and scattered across the codebase.

In this case, a process manager is justified. A process manager is an event handler with its own state. It listens to events from different steps, maintains progress state, and decides on compensations if needed.

type BookingProcessManager struct {
    bookingID uuid.UUID
    hotelOK   bool
    flightOK  bool
    carOK     bool
    hotelCmd  HotelCommandService
}

func (pm *BookingProcessManager) Handle(ctx context.Context, event DomainEvent) error {
    switch e := event.(type) {
    case HotelReserved:
        pm.hotelOK = true
        // Trigger the flight booking
        return pm.flightCmd.BookFlight(ctx, BookFlightCmd{
            BookingID: pm.bookingID,
            FlightRef: e.FlightRef,
        })

    case FlightBooked:
        pm.flightOK = true
        // Trigger the car reservation
        return pm.carCmd.ReserveCar(ctx, ReserveCarCmd{
            BookingID: pm.bookingID,
        })

    case FlightBookingFailed:
        // Compensation: cancel the hotel if already booked
        if pm.hotelOK {
            return pm.hotelCmd.CancelReservation(ctx, CancelHotelCmd{
                BookingID: pm.bookingID,
            })
        }
    }
    return nil
}

The difference from a global SagaManager: this process manager only knows the travel booking flow. It's not a centralization point for all system flows. One process manager per saga, not one manager for everything.

Its state must be persisted — in a database or the event store — to survive restarts. This is the only place in a choreographed architecture where you explicitly accept having a component with its own state and centralized flow logic. The rest of the system continues to operate through choreography.

Summary

An aggregate cannot call another aggregate directly — choreography through events is the standard solution.
The event handler is a translator: it transforms an event into a command on another aggregate. No business logic inside.
Sagas must be idempotent: an event can be redelivered, the handler must detect and ignore duplicates.
One handler per responsibility: never put multiple independent actions in a single handler. A partial failure must not block other branches.
Eventual consistency is an accepted consequence: a branch that fails will be retried independently, without blocking the others.
The event/handler table is the essential documentation for a choreographed system. Without it, the flow is unreadable.
For flows with compensation, a process manager with its own state is justified — but one per saga, not a global manager.

Part 4 covers the PostgreSQL event store: how to persist events, manage aggregate versioning, and implement replay from the database.

📄 Associated CLAUDE.md

View • Download • Catalogue

CQRS in Go — Part 2: side-effect-free command handlers

Odilon HUGONNOT — Tue, 21 Apr 2026 09:00:02 +0000

CQRS in Go series:

Part 1: the aggregate, Transition() and Clone()
Part 2: command handlers without side effects
Part 3: sagas and event choreography
Part 4: PostgreSQL as an event store

If you followed Part 1, you know what an aggregate is and how Transition() reconstructs state from events. Now, a more concrete question: who decides to emit those events? Who validates that a command is legitimate? That's the role of the command handler.

And that's where many CQRS projects drift. The handler starts clean, then accumulates dependencies. A repo here, an email service there. Three months later, you have a test that spins up a database to verify that a required field is actually required.

There's a better way.

The problem with classic handlers

Here's what a "standard" handler looks like — the one you write when you apply CQRS without pushing the reasoning all the way through:

// ❌ Classic handler — depends on everything
func (h *OrderHandler) PlaceOrder(ctx context.Context, cmd PlaceOrderCmd) error {
    customer, err := h.customerRepo.Get(ctx, cmd.CustomerID)
    if err != nil {
        return err
    }

    order := Order{
        ID:         uuid.New(),
        CustomerID: customer.ID,
        Items:      cmd.Items,
    }

    if err := h.orderRepo.Save(ctx, order); err != nil {
        return err
    }
    if err := h.emailService.Send(customer.Email, "Order placed"); err != nil {
        return err
    }

    return nil
}

To test this function, you need: a mock customerRepo, a mock orderRepo, a mock emailService. Three mocks for a single test. And each test becomes a wiring exercise — you spend more time configuring the environment than testing the business logic.

The worst part: the actual business logic fits in five lines. The rest is I/O. And that's precisely the problem: I/O and business logic are mixed together.

The magic signature

The solution fits in one signature:

Handle(ctx context.Context, state Order, cmd PlaceOrderCmd) (Events, error)

Three important things in this signature:

The handler receives the current state of the aggregate — not a repository, not a DB connection. The state is already reconstructed when the handler is called.
It returns events — not a modified state, not a boolean. Facts: "here's what happened".
Zero I/O, zero side effects. The handler doesn't even know PostgreSQL exists.

Here's the complete implementation:

type PlaceOrderCmd struct {
    OrderID    uuid.UUID
    CustomerID uuid.UUID
    Items      []OrderItem
}

func (cmd PlaceOrderCmd) Validate() error {
    if cmd.OrderID == uuid.Nil {
        return fmt.Errorf("order ID required")
    }
    if len(cmd.Items) == 0 {
        return fmt.Errorf("at least one item required")
    }
    return nil
}

type PlaceOrderHandler struct{}

func (h PlaceOrderHandler) Handle(ctx context.Context, state Order, cmd PlaceOrderCmd) (Events, error) {
    if err := cmd.Validate(); err != nil {
        return nil, err
    }

    // Business invariant: can't place an order that's already placed
    if state.Status == OrderStatusPlaced {
        return nil, ErrOrderAlreadyPlaced
    }

    total := 0
    for _, item := range cmd.Items {
        total += item.UnitPrice * item.Quantity
    }

    var events Events
    events.Append(OrderPlaced{
        OrderID:    cmd.OrderID,
        CustomerID: cmd.CustomerID,
        Items:      cmd.Items,
        Total:      total,
        PlacedAt:   time.Now(),
    })

    return events, nil
}

This handler is a pure function disguised as a method. Same inputs, same output, every time. No internal state. No hidden dependencies. Testable in complete isolation.

Validation in two layers

Look at the structure: validation is split into two distinct levels with different responsibilities.

Layer 1: cmd.Validate() — structural validation. Required fields, correct formats, simple constraints. This validation can be called before even loading the aggregate from the store. No business context needed. It belongs to the command itself.

Layer 2: business invariants in the handler — these validations require the current state. "Can we cancel this order?" depends on the order's current status. Without state, you can't answer.

type CancelOrderCmd struct {
    OrderID uuid.UUID
    Reason  string
}

func (cmd CancelOrderCmd) Validate() error {
    if cmd.Reason == "" {
        return fmt.Errorf("cancellation reason required")
    }
    return nil
}

func (h CancelOrderHandler) Handle(ctx context.Context, state Order, cmd CancelOrderCmd) (Events, error) {
    if err := cmd.Validate(); err != nil {
        return nil, err
    }

    // Business invariants — depend on state
    if state.Status == OrderStatusShipped {
        return nil, fmt.Errorf("cannot cancel shipped order")
    }
    if state.Status == OrderStatusCancelled {
        return nil, fmt.Errorf("order already cancelled")
    }

    var events Events
    events.Append(OrderCancelled{
        OrderID:     cmd.OrderID,
        Reason:      cmd.Reason,
        CancelledAt: time.Now(),
    })

    return events, nil
}

The separation is clean. Structural validation errors surface early, before any loading from the store. Business invariant errors surface later, when context is available.

Tests — 10 lines, no mocks

This is where the design pays off. Here's a complete test for the happy path:

func TestPlaceOrder_Success(t *testing.T) {
    handler := PlaceOrderHandler{}
    state := Order{} // empty state = new order

    cmd := PlaceOrderCmd{
        OrderID:    uuid.New(),
        CustomerID: uuid.New(),
        Items:      []OrderItem{{ProductID: "SKU-1", Quantity: 2, UnitPrice: 1500}},
    }

    events, err := handler.Handle(context.Background(), state, cmd)

    if err != nil {
        t.Fatalf("unexpected error: %v", err)
    }
    if len(events) != 1 {
        t.Fatalf("expected 1 event, got %d", len(events))
    }

    placed, ok := events[0].(OrderPlaced)
    if !ok {
        t.Fatalf("expected OrderPlaced, got %T", events[0])
    }
    if placed.Total != 3000 {
        t.Errorf("expected total 3000, got %d", placed.Total)
    }
}

No mocks. No DB. No Docker. No setup/teardown. This test runs in microseconds on any machine.

Business invariants test just as simply — just pass a state with the right status:

func TestPlaceOrder_AlreadyPlaced(t *testing.T) {
    handler := PlaceOrderHandler{}
    state := Order{Status: OrderStatusPlaced} // order already placed

    _, err := handler.Handle(context.Background(), state, PlaceOrderCmd{
        OrderID: uuid.New(),
        Items:   []OrderItem{{ProductID: "SKU-1", Quantity: 1, UnitPrice: 100}},
    })

    if err != ErrOrderAlreadyPlaced {
        t.Fatalf("expected ErrOrderAlreadyPlaced, got %v", err)
    }
}

And table-driven tests are particularly readable because the only variable between cases is the initial state:

func TestCancelOrder(t *testing.T) {
    tests := []struct {
        name    string
        state   Order
        wantErr bool
    }{
        {"draft order", Order{Status: OrderStatusDraft}, false},
        {"placed order", Order{Status: OrderStatusPlaced}, false},
        {"shipped order", Order{Status: OrderStatusShipped}, true},
        {"already cancelled", Order{Status: OrderStatusCancelled}, true},
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            _, err := CancelOrderHandler{}.Handle(context.Background(), tt.state, CancelOrderCmd{
                OrderID: uuid.New(),
                Reason:  "changed mind",
            })
            if (err != nil) != tt.wantErr {
                t.Errorf("got err=%v, wantErr=%v", err, tt.wantErr)
            }
        })
    }
}

Four status cases, four lines in the table. The test logic is entirely in the data — which is exactly what table-driven tests aim for.

The Events type and Append()

The Events type is intentionally simple:

type Events []DomainEvent

func (e *Events) Append(event DomainEvent) {
    *e = append(*e, event)
}

It's a slice. No event bus, no dispatcher, no middleware. Just a slice with a helper method so the calling code is a bit more readable than *e = append(*e, event).

Why so simple? Because the handler doesn't need to know what happens next. It emits events. That's it. What happens after — persistence, projections, sagas — is the command gateway's problem.

The full flow — from HTTP to storage

For the complete picture, here's what the command gateway does when it receives a command. The handler sees none of this:

// What the CommandGateway does when it receives a command:
// 1. Load the aggregate's existing events from the store
// 2. Replay events to reconstruct state (via Transition())
// 3. Call handler.Handle(ctx, state, cmd)
// 4. Store the new events in the store
// 5. Notify event handlers (projections, sagas)

func (g *CommandGateway) Dispatch(ctx context.Context, aggregateID uuid.UUID, cmd Command) error {
    // Load and reconstruct the aggregate
    storedEvents, err := g.store.Load(ctx, aggregateID)
    if err != nil {
        return fmt.Errorf("loading aggregate %s: %w", aggregateID, err)
    }

    state := g.aggregate.Rebuild(storedEvents)

    // The handler knows nothing of what came before
    newEvents, err := g.handler.Handle(ctx, state, cmd)
    if err != nil {
        return err
    }

    // Store the new events
    if err := g.store.Append(ctx, aggregateID, newEvents); err != nil {
        return fmt.Errorf("appending events: %w", err)
    }

    // Notify projections and sagas (async)
    g.bus.Publish(newEvents)

    return nil
}

The handler is called at step 3. It receives an already-reconstructed state, returns events, and doesn't know what comes before or after. This deliberate ignorance is what makes it testable.

The details of the PostgreSQL store and event-based reconstruction are covered in Part 4. What matters here: the handler is pure.

Summary

Pure handler: no dependency on repos, DBs, or external services. Receives a state, returns events.
Signature Handle(ctx, state, cmd) (Events, error): that's the contract. The entire architecture follows from it.
Two-layer validation: structural on the command (cmd.Validate()), business in the handler. Each where it has the necessary context.
Tests without mocks: build a state, call the handler, verify the events. No setup, no infrastructure, no Docker.
Events as a slice: type Events []DomainEvent. No bus built into the handler. Routing complexity belongs elsewhere.
The gateway orchestrates: loading from the store, state reconstruction, handler call, event persistence. The handler sees none of this — and that's intentional.

Part 3 covers sagas: how to react to an event to trigger other commands, and how to orchestrate business processes that span multiple aggregates.

📄 Associated CLAUDE.md

View • Download • Catalogue

CQRS in Go — Part 1: the aggregate, Transition() and the Clone() you forget

Odilon HUGONNOT — Mon, 20 Apr 2026 09:00:01 +0000

CQRS in Go series:

Part 1: the aggregate, Transition() and Clone()
Part 2: command handlers without side effects
Part 3: sagas and event choreography
Part 4: PostgreSQL as an event store

What an aggregate is

The aggregate is the guardian of business consistency. It's the one that says "no" when an operation is invalid: you can't cancel an order that's already been shipped, you can't add an item to a closed cart. This logic lives in the aggregate and nowhere else.

In Event Sourcing, the aggregate has no directly persisted state. Its state is the result of replaying all its events from the beginning. Each event represents something that happened — immutable, factual, in the past. The aggregate replays them in order and reconstructs its current state.

What the aggregate does not do: I/O. It doesn't read from the database, doesn't make HTTP requests, doesn't send emails. It receives a command, validates whether it can be applied to its current state, and emits events. That's it. This constraint is what makes it testable at zero cost.

The structure in Go

An Order aggregate in Go looks like this:

type OrderStatus string

const (
    OrderStatusDraft     OrderStatus = "draft"
    OrderStatusPlaced    OrderStatus = "placed"
    OrderStatusShipped   OrderStatus = "shipped"
    OrderStatusCancelled OrderStatus = "cancelled"
)

type OrderItem struct {
    ProductID string
    Quantity  int
    UnitPrice int // in cents
}

type Order struct {
    OrderID    uuid.UUID
    CustomerID uuid.UUID
    Status     OrderStatus
    Items      []OrderItem
    Total      int
    CreatedAt  time.Time
}

Two deliberate choices here. First, value struct, not a pointer. When you pass an Order, Go makes a copy of it. That's exactly what we want: two versions of a state don't share memory by default (except for slices and maps — we'll get to that). An aggregate passed by pointer between multiple goroutines is a source of silent bugs.

Second, fields are exported here to simplify the example, but in production you keep them private to the package and only expose what's needed through methods. The aggregate is not a DTO.

Events as the source of truth

Events are facts that have occurred. They don't describe an intention — that's the role of commands — they describe what happened.

type OrderPlaced struct {
    OrderID    uuid.UUID
    CustomerID uuid.UUID
    Items      []OrderItem
    Total      int
    PlacedAt   time.Time
}

type OrderItemAdded struct {
    OrderID  uuid.UUID
    Item     OrderItem
    NewTotal int
}

type OrderCancelled struct {
    OrderID     uuid.UUID
    Reason      string
    CancelledAt time.Time
}

Once emitted, an event never changes. You don't correct a past event — you emit a new event that compensates. That's the fundamental difference from an UPDATE in a database.

The aggregate declares the event types it handles via EventKinds(). This is useful for the event store which needs to know how to deserialize stored events:

func (o Order) EventKinds() []DomainEvent {
    return []DomainEvent{
        OrderPlaced{},
        OrderItemAdded{},
        OrderCancelled{},
    }
}

Transition() — the heart of the pattern

Transition() is the method that applies an event to the current state and returns a new state. It's the only place where the aggregate's state evolves.

func (o Order) Transition(event DomainEvent) Order {
    switch e := event.(type) {
    case OrderPlaced:
        o.OrderID = e.OrderID
        o.CustomerID = e.CustomerID
        o.Status = OrderStatusPlaced
        o.Items = e.Items
        o.Total = e.Total
        o.CreatedAt = e.PlacedAt

    case OrderItemAdded:
        o.Items = append(o.Items, e.Item)
        o.Total = e.NewTotal

    case OrderCancelled:
        o.Status = OrderStatusCancelled
    }
    return o
}

The receiver is a value, not a pointer. When Go calls this method, it copies the current Order into o. We modify this copy and return it. If you ignore the return value, nothing has changed — the original Order is intact.

It's a pure reducer, exactly like Array.prototype.reduce in JavaScript or fold in Haskell. The current state is a deterministic function of all past events:

state = events.reduce(initialState, transition)

The practical consequence: you can reconstruct the aggregate's state at any historical point in time by replaying events up to that point. To debug a bug that occurred on February 14th at 2:37 PM, replay the events up to that timestamp and inspect the state. No logs to dig through, no snapshots to restore.

The Clone() trap — the silent bug

This is where most CQRS implementations in Go fall apart. And it's silent: no panic, no error, just corrupted data.

The problem comes from Go's memory model for slices. A slice is a three-field structure: a pointer to the backing array, a length, and a capacity. When Go copies a struct containing a slice, it copies these three fields — but not the backing array. Both copies point to the same array in memory.

append is particularly treacherous. If the slice has enough capacity to accommodate the new element, append writes directly into the existing backing array without allocating a new one. Result: the old state is modified without the code knowing it.

package main

import "fmt"

type OrderItem struct {
    ProductID string
    Quantity  int
}

type Order struct {
    Items []OrderItem
}

func main() {
    // Slice with capacity 4, length 1 — enough room for append without reallocation
    original := Order{
        Items: make([]OrderItem, 1, 4),
    }
    original.Items[0] = OrderItem{ProductID: "A", Quantity: 1}

    // Simulate a naive Transition: copy the struct, append to the copy
    modified := original
    modified.Items = append(modified.Items, OrderItem{ProductID: "B", Quantity: 2})

    // The trap: append wrote into the shared backing array.
    // original.Items still points to len=1, but index [1] in memory
    // now contains product "B". Reslice and it reappears.
    fmt.Println("original length:", len(original.Items))  // 1 — reassuring
    fmt.Println("modified length:", len(modified.Items))  // 2

    // But:
    ghost := original.Items[:2] // Reslice within existing capacity
    fmt.Println("ghost item:", ghost[1].ProductID) // "B" — it's there
}

In practice in a CQRS system, this scenario occurs as soon as you store a snapshot or pass a state between goroutines. The old state you thought was immutable has been silently modified.

The solution is Clone(). Before any Transition(), explicitly clone the state to break the link with the backing array:

func (o Order) Clone() Order {
    c := o
    if o.Items != nil {
        c.Items = make([]OrderItem, len(o.Items))
        copy(c.Items, o.Items)
    }
    return c
}

copy allocates a new backing array and copies the elements into it. After that, c.Items and o.Items are completely independent. Any subsequent append on one doesn't touch the other.

The same constraint applies to maps. Go copies the reference, not the content. A Cart with a map requires an explicit deep clone:

type Cart struct {
    Items map[uuid.UUID]CartItem
}

func (c Cart) Clone() Cart {
    clone := c
    clone.Items = make(map[uuid.UUID]CartItem, len(c.Items))
    for k, v := range c.Items {
        clone.Items[k] = v
    }
    return clone
}

If CartItem itself contains slices or maps, you need to go one level deeper. That's the only case where the depth of the clone depends on the actual data structure — there's no safe generic shortcut in Go.

The production rule: always call Clone() before Transition(). You can also integrate it directly into Transition() as the first instruction, which makes the pattern foolproof at the cost of a systematic copy even when it's not needed. For most aggregates, this overhead is negligible.

Replay — reconstructing state from events

With Clone() and Transition() in place, the replay fits in five lines:

func Replay(events []DomainEvent) Order {
    var state Order
    for _, event := range events {
        state = state.Clone().Transition(event)
    }
    return state
}

Start from an empty Order, apply each event in order, and get the final state. Proof that the aggregate is pure: this function is deterministic. Same list of events, same result, every time. No network calls, no cache, no system clock. You can call it from a test without any mocks.

Replay is also what allows you to fix bugs after the fact. If the logic in Transition() was incorrect and you fix it, you can replay all historical events and get corrected states — provided the events themselves are correct, which they are by definition since they represent what actually happened.

Summary

The aggregate is the guardian of business consistency. It does no I/O.
Its state is a pure function of the events it has received since its creation.
Transition(event) applies an event and returns a new state — method on a value, not a pointer.
Go copies structs by value, but slices and maps share their backing store. Without Clone(), mutations via append can silently corrupt old states.
Clone() must deep copy all collections in the struct.
Replay reconstructs any historical state by replaying events — no I/O, therefore trivially testable.

Part 2 covers command handlers: how to validate a command against the aggregate's current state, emit the appropriate events, and keep everything free of side effects and implicit dependencies.

📄 Associated CLAUDE.md

View • Download • Catalogue