Forem: Andre Faria

Building a Python Display Framework for Raspberry Pi OLED Screens

Andre Faria — Sun, 12 Apr 2026 23:51:02 +0000

1. The Original Inspiration

This project started with an article by Michael Klements on The DIY Life: Add an OLED Stats Display to Raspberry Pi OS Bookworm. The article walks through connecting a small SSD1306 OLED display to a Raspberry Pi and writing a Python script that shows live system statistics (CPU usage, memory, disk, temperature, and IP address).

The original script, available at github.com/mklements/OLED_Stats, is a clear and working piece of code. It does exactly what it says on the tin. For a single-purpose stats screen, it is perfectly fine.

But the more I looked at the script, the more I noticed a pattern I have seen in many embedded display projects: the same boilerplate repeated everywhere. Every example in the repo wires up busio.I2C, initializes adafruit_ssd1306.SSD1306_I2C, creates a PIL.Image, sets up ImageDraw, and then tears it all down at the end. If you want to show something different on the screen like a clock, a network status, an animation you will need to write almost the same scaffolding again from scratch.

That made me want to build something better.

2. What I Built Instead

The result is rpi-display-core: a small Python framework for SSD1306 and SH1106 OLED displays on the Raspberry Pi. The goal was to eliminate all the display boilerplate and replace it with a clean, composable API.

Instead of wiring up I2C every time you want to show something, you write:

from rpi_display.displays.ssd1306 import SSD1306Display
from rpi_display import Runner
from rpi_display.widgets.clock import ClockWidget

Runner(SSD1306Display(), ClockWidget()).run()

That is it. Four lines. A running clock on the OLED display.

The framework provides:

Specialized display classes for SSD1306 and SH1106 backends
A canvas context manager that gives you a PIL.ImageDraw surface and automatically flushes it to the display on exit
A Widget base class with a consistent render(draw, x, y) interface
A Runner class that drives a render loop at a fixed FPS
A MockDisplay for testing without hardware
Multiple built-in widgets covering the most common display use cases

The framework is available on PyPI, has a full pytest suite, and includes examples such as a systemd service so you can run your display as a persistent background service.

The repository is at: https://github.com/andremmfaria/rpi-display-core

3. Hardware You'll Need

The hardware side of this project is minimal. You need a Raspberry Pi, a small OLED display, and four jumper wires.

Raspberry Pi — any Pi with I2C support will work. (The Pi 5 is the current recommended board, but i used an Rpi 4b for this)
Raspberry Pi Power Supply — the official USB-C power supply for the Pi
32GB MicroSD Card — any class-10 card works; 32GB is more than enough
I2C OLED Display 128×64 — the 0.96-inch SSD1306 module or 1.3-inch SH1106 module, four pins (GND, VCC, SCL, SDA)
4-Wire Female-to-Female Jumper Cables — for connecting the display to the GPIO header

The framework supports both SSD1306 and SH1106 I2C displays. SPI variants are out of scope.

4. Wiring It Up

The OLED module connects directly to the Raspberry Pi GPIO header using four wires. No breadboard required.

OLED Pin	Pi Header Pin	Description
GND	Pin 9	Ground
VCC	Pin 1	3.3V power
SCL	Pin 5	I2C clock
SDA	Pin 3	I2C data

Before running anything, I2C must be enabled on the Pi. Use raspi-config → Interface Options → I2C, or add dtparam=i2c_arm=on to /boot/firmware/config.txt. After enabling I2C, verify the display is detected:

i2cdetect -y 1

You should see 3c appear in the output grid, which is the default I2C address for SSD1306 and SH1106 displays.

5. Installing the Framework

The framework is available on PyPI and can be installed using uv:

uv add rpi-display-core

You can find the project on PyPI at: https://pypi.org/project/rpi-display-core

The adafruit-blinka, adafruit-circuitpython-ssd1306, and adafruit-circuitpython-sh1106 packages provide the I2C and display drivers. pillow handles image composition. The rpi-display-core package itself manages these dependencies for you.

To verify the installation:

python -c "from rpi_display.displays.ssd1306 import SSD1306Display; from rpi_display import canvas, Widget, Runner; print('ok')"

6. Core Concepts

The framework has four building blocks: Display backends, canvas, Widget, and Runner.

Display Backends

The framework provides specialized classes for different display controllers. Hardware imports are deferred inside __init__, so the module can be imported on any machine without crashing.

from rpi_display.displays.ssd1306 import SSD1306Display
from rpi_display.displays.sh1106 import SH1106Display

display = SSD1306Display()    # default: address=0x3C, 128×64
# OR
# display = SH1106Display()
display.clear()              # fill with black and flush

canvas

canvas is a context manager that creates a fresh PIL.Image and ImageDraw, yields the draw surface, and then flushes the image to the display on exit, even if an exception is raised inside the block.

from rpi_display.displays.ssd1306 import SSD1306Display
from rpi_display import canvas
from PIL import ImageFont

display = SSD1306Display()
with canvas(display) as draw:
    font = ImageFont.load_default()
    draw.text((10, 20), "Hello, World!", font=font, fill=255)

This pattern comes directly from luma.oled, which uses the same with canvas(device) as draw idiom.

Widget

Widget is the base class for all display components. Every widget implements a single method:

def render(self, draw, x: int = 0, y: int = 0) -> None:
    raise NotImplementedError

The draw argument is a PIL.ImageDraw.ImageDraw. The x and y offsets let you position widgets anywhere on the 128×64 canvas.

Runner

Runner drives the render loop. It accepts either a Widget instance or a plain callable, and calls it at a fixed FPS:

from rpi_display.displays.ssd1306 import SSD1306Display
from rpi_display import Runner
from rpi_display.widgets.system import SystemStatsWidget

Runner(SSD1306Display(), SystemStatsWidget(), fps=1).run()

The loop runs until interrupted. A try/finally block ensures display.clear() is always called on exit, leaving the screen blank rather than frozen on the last frame.

7. Built-in Widgets

The framework ships several widgets out of the box.

Text

Text renders a single line of text at a given position and font size. MultiLineText renders a list of strings as stacked lines with configurable spacing. ScrollingText scrolls a string horizontally across the screen, advancing by a configurable number of pixels per render call.

from rpi_display.widgets.text import Text, MultiLineText, ScrollingText

Text("Hello").render(draw, 0, 0)
MultiLineText(["Line 1", "Line 2", "Line 3"]).render(draw, 0, 0)
ScrollingText("This is a long message that scrolls...", speed=3).render(draw, 0, 50)

All three widgets load DejaVu Sans from /usr/share/fonts/truetype/dejavu/DejaVuSans.ttf when available, and fall back to the PIL bitmap font otherwise.

ProgressBar

ProgressBar draws a filled rectangle representing a value between 0.0 and 1.0. Values outside that range are clamped at construction time.

from rpi_display.widgets.shapes import ProgressBar

ProgressBar(0.72, label="CPU").render(draw, 0, 26)

ClockWidget

ClockWidget shows the current time in large type, a horizontal divider, the date in smaller type below, and an optional seconds progress bar at the bottom of the screen. It reuses ProgressBar internally for the seconds indicator.

SystemStatsWidget

SystemStatsWidget is a composite widget that stacks five individual sub-widgets in a column:

IpWidget — local IP address via hostname -I
CpuWidget — CPU usage via /proc/stat two-snapshot method
RamWidget — memory usage via free -m
DiskWidget — disk usage via df -h
TempWidget — CPU temperature via /sys/class/thermal/thermal_zone0/temp

Each sub-widget fetches its data fresh on every render() call and returns "N/A" if the data source is unavailable, rather than raising an exception.

The CPU widget deliberately avoids top -bn1 because it is slow and creates its own CPU load. Reading /proc/stat twice with a 0.1-second gap gives an accurate idle-time delta at a fraction of the cost.

NetworkWidget

NetworkWidget shows the hostname, local IP, and internet reachability (a 1-second ping to 8.8.8.8). All three lookups are wrapped in exception handlers and return graceful fallback values on failure.

Spinner

Spinner cycles through |, /, -, \ characters, advancing one frame per render() call. The caller controls speed by adjusting the Runner's FPS.

8. A Complete Example

Here is a full script using SystemStatsWidget with Runner. This is also what the systemd service example uses:

from rpi_display.displays.ssd1306 import SSD1306Display
from rpi_display import Runner
from rpi_display.widgets.system import SystemStatsWidget

Runner(SSD1306Display(), SystemStatsWidget(), fps=1).run()

When the script runs, it updates the display once per second with the current IP, CPU, RAM, disk, and temperature. Press Ctrl+C to stop. The display clears cleanly on exit.

For testing without a physical display, swap in MockDisplay:

from rpi_display import Runner
from rpi_display.mock import MockDisplay
from rpi_display.widgets.system import SystemStatsWidget

d = MockDisplay()
Runner(d, SystemStatsWidget(), fps=10).run()
# d.last_image holds the most recent PIL Image after each render

9. Running as a systemd Service

For a persistent display that survives reboots, create a systemd unit file that runs examples/09_systemd_service.py. That script runs SystemStatsWidget at 1 FPS and is designed to be the entry point for a service.

Create /etc/systemd/system/rpi-display.service with the following contents, replacing YOUR_USERNAME and the path to match your installation:

[Unit]
Description=rpi-display-core stats display
After=network.target

[Service]
User=YOUR_USERNAME
WorkingDirectory=/home/YOUR_USERNAME/rpi-display-core
ExecStart=uv run python examples/09_systemd_service.py
Restart=on-failure

[Install]
WantedBy=multi-user.target

Then enable and start it:

sudo systemctl daemon-reload
sudo systemctl enable rpi-display
sudo systemctl start rpi-display

Restart=on-failure means a clean exit (e.g. Ctrl+C in a terminal) will not trigger a restart. Only unexpected crashes will.

To check logs:

journalctl -u rpi-display -f

If the service restarts repeatedly, the most common cause is the display not being detected. Run i2cdetect -y 1 to confirm 3c appears.

10. Development Workflow

If you want to contribute to the project, I use uv for development. The following commands are used for linting, formatting, and testing:

uv run ruff check .
uv run ruff format --check .
uv run black --check .
uv run isort --check-only .
uv run mypy
uv run pytest

11. Future Improvements

The framework covers the common cases for I2C OLED displays, but there are a number of directions it could grow.

Support for additional display controllers: Potential future display backends include SPI displays and e-ink panels.
Additional widgets: I am considering adding BitmapWidget for rendering 1-bit PNG or BMP files and a QRCodeWidget for generating codes on the fly.
Enhanced scrolling: The ScrollingText widget currently wraps at the end of the text. Supporting bidirectional bounce scrolling is a planned improvement.

Conclusion

Starting from Michael Klements' original stats display script, this project built a composable Python framework that replaces display boilerplate with clean abstractions. The specialized display classes, canvas, Widget, and Runner primitives cover the full rendering lifecycle, and the built-in widgets handle the most common display use cases.

The framework is available on PyPI at https://pypi.org/project/rpi-display-core, is fully tested, and is designed for production use on the Raspberry Pi.

The repository is available at: https://github.com/andremmfaria/rpi-display-core

Credits:

Original article: Add an OLED Stats Display to Raspberry Pi OS Bookworm by Michael Klements
Original repo: github.com/mklements/OLED_Stats

I just wanted a desk clock I accidentally built a Home Assistant dashboard

Andre Faria — Sun, 22 Mar 2026 03:46:30 +0000

1. The Unexpected Device

I wasn’t trying to build anything.

I just wanted a desk clock. Something small, clean, and with Wi-Fi so it would always have the correct time. No tinkering, no integrations, no dashboards. Just something I could plug in, place on my desk, and forget about.

What I ended up buying was the GeekMagic Ultra on Amazon. The ad marketed it as a generic “smart weather clock,” which sounded close enough to what I needed. The design is nice, the screen is sharp, and on paper it looks like a slightly more capable version of a normal digital clock.

Out of the box, that’s exactly what it feels like. You connect to it using its own Wi-Fi network, then it provides you with a web interface so you can configure it to connect to your Wi-Fi. It shows time, weather, and a few widgets, and generally behaves like a polished consumer device. But after a few minutes of using it, something feels off.

The customization is limited. You can change what’s displayed, but not how it works. It’s flexible in appearance, but rigid in behavior. That’s usually a sign that the hardware underneath is either heavily locked down or far more capable than the software allows. In this case, it was the latter.

Once you dig a bit deeper, you realize this isn’t really a “smart clock” at all. It’s an ESP8266 with a 240×240 display attached to it. That’s it. No magic, no proprietary silicon. Just a very familiar microcontroller in a nicely packaged form factor.

That realization changes the entire perspective. Because if it’s an ESP8266:

it can be reflashed
it can run ESPHome
it can integrate directly with Home Assistant

At that point, it stops being a product and starts being a platform.

What I thought was a simple desk accessory turned out to be a small, hackable display node that fits perfectly into a home automation setup. Not by design, but by accident.

2. Peeling It Open: Hardware Reality

Once you accept that the device is hackable, the next step is understanding what you’re actually working with. And in this case, that means ignoring the marketing entirely and looking at the hardware.

In this device's case, the chip is soldered on the board with the other components and cannot be removed easily. Otherwise, the device is very simple:

an ESP8266
a 240×240 ST7789 TFT display
SPI wiring between them
a PWM-controlled backlight

Like this:

There’s no extra compute layer, no buffering chip, no hidden abstraction. Everything you draw goes straight through the ESP8266 to the display. That simplicity is both the reason this works and the reason it can fail so easily.

The ESP8266 is a capable chip, but it is also extremely constrained. You are working with a small amount of usable RAM, no PSRAM, and a heap that can become unstable if pushed too far. On the other side, the display is not trivial. A 240×240 screen sounds small, but it still requires a meaningful amount of memory to render properly.

That creates a constant tension:

the display wants memory, and the ESP8266 does not have much of it.

This is why so many initial attempts fail. The natural instinct is to treat it like a modern embedded system, allocate buffers, use large fonts, redraw frequently. On this device, that approach leads straight to crashes, boot loops, or a screen that just flickers black.

The wiring itself also comes with a few quirks. Through community reverse engineering, the common mapping looks like this:

GPIO14 → SPI clock
GPIO13 → SPI MOSI
GPIO0 / GPIO2 → display control (DC / RESET)
GPIO5 → backlight (PWM)

This mapping is also referenced by the GeekMagic owner in issue #4 of the smalltv repository, where they shared the same pin definitions for the device (TFT_DC=0, TFT_RST=2, SCK=14, MOSI=13, TFT_BL=5, TFT_CS=-1).

One detail that catches people off guard is the lack of a proper chip select line. Because of that, the display only behaves correctly when the SPI bus is configured in a specific mode (mode3). This is not documented anywhere official, it’s something the community figured out by trial and error.

And that pattern repeats across the entire device.

Nothing here is particularly complex, but almost nothing is documented either. Every working configuration is the result of small discoveries layered on top of each other.

The important takeaway is that this is not a forgiving platform. You don’t have the headroom to brute-force your way through problems. Every decision, buffer size, font size, update interval, has a direct impact on stability.

Once you understand those constraints, the device becomes predictable and surprisingly capable. Until then, it just looks like it’s broken.

3. The Real Work: Community Reverse Engineering

If you try to approach this device using only official documentation, you won’t get very far.

There is no proper datasheet for the product as a whole (although there is a GH repo with some manuals). There is no “supported ESPHome configuration.” There isn’t even a clear description of how the display is wired internally. What exists instead is a long trail of people experimenting, breaking things, and slowly converging on what works.

The starting point for me was a YouTube video from Maker HQ, which provides a basic working configuration. This video was really useful because it gave me a working config file as a starting point. Without it, the proper way to set the display parameters becomes a guessing game. It gets the screen to light up and things to render, but it doesn’t explain why certain settings matter or what happens when you deviate from them.

The real work happened in the forum thread on the Home Assistant Community.

An important detail for context: the hardware shown in post #8 of that thread is exactly the same as my unit, which places mine in the clone/counterfeit variant discussed there rather than the official SmallTV Ultra hardware.

That thread is long, messy, and full of partial solutions, but it’s also where most of the important details were uncovered. Not in a single place, but spread across dozens of posts. You don’t read it linearly, you piece it together.

A few of the key findings that came out of that effort:

The display works reliably only with spi_mode: mode3
The newer mipi_spi driver behaves better than older alternatives
color_depth: 8 is effectively mandatory on ESP8266
Full buffering is not viable, partial buffers must be used
Small mistakes in configuration lead to hard crashes, not soft failures

None of these are obvious if you just look at ESPHome documentation. They only become clear when you see multiple people hitting the same issues and gradually narrowing down the causes. Another important detail is that there isn’t a single “correct” configuration. There are working configurations, but they depend on trade-offs:

stability vs visual quality
buffer size vs responsiveness
font size vs memory usage

That’s why copying a YAML file blindly often doesn’t work. Small differences, even something like a slightly larger font, can push the device over the edge.

This is one of those cases where the community didn’t just provide examples. It effectively reverse engineered the behavior of the device through collective experimentation. Without that, this would have been a dead end.

Huge thanks to MakerHQ for publishing the video walkthrough, and to everyone in the Home Assistant forum thread who shared tests, pin mappings, and working configs. That collective effort is what made this project practical.

4. Step by Step: Connect, Flash, and Configure

If you have the same hardware revision I got, the process is easier than many guides suggest.

I did not need to solder anything at all. Flashing worked by simply plugging the device into my computer over USB and using the ESPHome web flasher.

Here is the exact flow that worked for me:

Connect the device to your computer with a USB cable.
Open https://web.esphome.io/ in a Chromium-based browser (Chrome, Edge, Brave, etc.).
Click Connect, then select the serial device that appears for the clock.
Install ESPHome onto the device from the web installer.
Wait for the first boot to complete, then join the temporary Wi-Fi AP created by the device if prompted.
Join the ap through your phone or something, enter the webpage on the device and configure the WiFi connection to your network.
Provide your Wi-Fi credentials so the device can join your network.
Add it to Home Assistant and upload your YAML configuration.
Reboot once after the first successful upload and confirm that the display renders correctly.

One important browser caveat: Firefox did not work for me because this flow depends on Web Serial support, which is available in Chromium-based browsers.

If you prefer to follow a visual walkthrough, there is also a step-by-step in the MakerQH video.

After this initial flash, updates are much easier because you can usually do OTA uploads from ESPHome without reconnecting USB.

5. Making It Work: ESPHome + Home Assistant Integration

Once the display is stable, the problem shifts from “how do I make this work” to “what do I actually want it to show.”

In my case, the answer was straightforward: I wanted a simple network status panel that still functioned as a desk clock.

The architecture ended up being very simple, given that I already had the UniFi integration in Home Assistant:

UniFi Dream Machine → Home Assistant → ESPHome → Display

The key decision here was to let Home Assistant do all the heavy lifting.

Instead of pushing data via MQTT or building custom logic on the ESP8266, I used the homeassistant: platform in ESPHome to pull values directly. That means the device is not calculating anything complex, it’s just rendering whatever Home Assistant already knows.

The data flowing into the display includes:

WAN status (up/down)
External IP address
Total data received and sent
Current download and upload speeds
Uptime

All of these come from existing Home Assistant entities. The ESP simply reads them and turns them into text on the screen. That approach keeps the system simple and, more importantly, stable.

Take a look at the result on this gist: https://gist.github.com/andremmfaria/7d060df2771cc90815e220d1a5440b85

There are still a few transformations that need to happen locally, but they are lightweight:

Uptime arrives as raw seconds → converted into days/hours/minutes
Byte counters → converted into KB/MB/GB for readability
Speed values → relabeled to match expected units

Nothing here is computationally heavy. It’s mostly formatting. This is important, because the ESP8266 doesn’t have much headroom. The more logic you move out of it, the more reliable the system becomes.

Rendering is done using a display lambda, updated every 15 seconds. That interval is deliberate. Faster updates are possible, but they start to introduce timing warnings and unnecessary load. Slower updates keep things smooth and predictable.

Another small but important choice was avoiding unnecessary state. The device does not cache values, track deltas, or maintain history. It simply redraws the current state each cycle. That makes it effectively stateless:

if Home Assistant updates, the display reflects it
if the ESP reboots, it just reconnects and resumes

No synchronization problems, no drift, no edge cases. In the end, the ESP8266 isn’t acting like a smart device. It’s acting like a very small, very focused display terminal for Home Assistant. And that’s exactly what makes it work.

6. The UI: Constraints Drive Design

Once everything is wired and talking properly, the next question is simple: what should this actually look like?

That’s where the constraints start shaping everything.

A 240×240 screen sounds like enough space, but it fills up quickly. Add to that the ESP8266 limitations, limited RAM, slow redraws, and occasional watchdog warnings, and you’re not designing freely anymore. You’re designing within a tight box.

Early on, it becomes clear that you can’t treat this like a modern UI. There’s no room for heavy layouts, large assets, or frequent updates. Even small changes, like increasing font sizes or adding extra text, can have a noticeable impact on performance.

So the layout has to be intentional.

The final structure ended up being simple and functional:

[ TIME            DATE ]
[ WAN STATUS      IP   ]
-----------------------
[ Down / Up            ]
[ RX / TX              ]
-----------------------
[ Uptime               ]

The time is the primary element, so it gets the largest font and the most visual weight. The date sits opposite it, using the same horizontal space to balance the layout without competing for attention.

Below that, the WAN status and IP address are split across the screen. This was a deliberate choice. Keeping them on the same line but on opposite sides avoids clutter while still grouping related information together.

The middle section is purely data:

download and upload speeds
total received and transmitted data

These are aligned in a predictable way, so your eyes don’t need to search. Labels on the left, values on the right. No surprises.

At the bottom, uptime sits on its own, separated by a line. It’s useful, but not something you need to glance at constantly, so it gets the least visual emphasis.

The biggest trade-offs showed up in small details:

Large fonts improve readability, but reduce available space
Right-aligned text looks better, but is slightly more expensive to render
Frequent updates feel “live,” but increase CPU load

Even color choices matter. Bright colors for data, white for labels, muted tones for separators. Not for aesthetics alone, but to keep the information readable at a glance.

There’s also no use of images or complex graphics. Everything is drawn using basic primitives, text, lines, and simple shapes. Not because it looks better, but because it’s cheaper to render and more stable over time.

The end result isn’t flashy, but it doesn’t need to be. It’s fast enough, stable enough, and clear enough to do its job.

And on a device like this, that’s the real definition of a good UI.

7. What This Became (and Why It’s Better Than a Clock)

At some point, this stopped being about fixing a device and started becoming something else entirely.

I set out to get a clock. What I ended up with is a small, always-on display that reflects the state of my network in real time.

The difference is subtle, but important.

A clock is passive. It shows time, maybe the weather, and that’s it. This device, once integrated with Home Assistant, becomes part of the system. It reacts to changes, reflects status, and gives you information you didn’t realize you wanted in that form.

Right now, it shows:

time and date
WAN status
external IP
live bandwidth usage
total traffic
uptime

But that’s just a starting point.

Because it’s running ESPHome, it can be extended in any direction:

flash the screen when WAN goes down
display alerts or notifications
switch between different pages of data
integrate other sensors from Home Assistant
react to events instead of just polling

None of that requires changing the hardware. It’s all software.

What makes this particularly interesting is how accidental it is. The device wasn’t designed to be used this way. It just happens to expose enough of its internals to make it possible.

That’s a recurring pattern with these kinds of products. They sit in a space between consumer electronics and development boards. Most people use them as intended. A few people look inside and realize they can do much more. This ended up being one of those cases.

It’s still sitting on my desk, still acting as a clock. But now it’s also a live view into my network, something I can glance at without opening a dashboard or checking an app. And that’s the part that makes it better.

Not because it’s more complex, but because it’s more useful.

Improving the ESP32 Wiimote Library - From Prototype to Production-Ready Arduino Library

Andre Faria — Tue, 10 Mar 2026 19:58:21 +0000

1. Why I Needed a Better ESP32 Wiimote Library

Nintendo’s Wii controllers are still surprisingly capable input devices. They are inexpensive, widely available, and include multiple sensors: digital buttons, a three-axis accelerometer, and support for extension controllers such as the Nunchuk. Because they communicate over Bluetooth, they can also be integrated into modern embedded systems without additional hardware.

For ESP32 projects, one of the few existing implementations is the ESP32Wiimote. The library provides a functional way to connect an ESP32 board to a Wiimote and exposes several core features:

Bluetooth pairing with Wii controllers
Button input events
Accelerometer data from the Wiimote
Support for extension controllers like the Nunchuk
A simple demonstration sketch

As a starting point, the library works well. It demonstrates how the ESP32’s Bluetooth stack can communicate with the Wiimote and decode controller data. For experimentation or small prototypes, it provides everything needed to get input from the controller.

However, once I began integrating the library into a larger project, some limitations became apparent. These are common challenges when a library evolves from a proof-of-concept into something used in real systems:

Limited runtime feedback – applications had little visibility into connection state or controller status.
Minimal documentation – most usage details were embedded only in the example sketch.
Lack of automated testing – making refactors risky and harder to validate.
Basic project structure – the repository layout did not fully follow modern Arduino library conventions.
Limited observability – debugging Bluetooth behavior required manual serial prints.

None of these issues prevented the library from working, but they made it harder to integrate into a reliable system. In particular, when building systems that run continuously or interact with other services, features like connection monitoring, structured logging, and predictable APIs become much more important.

Rather than starting from scratch, I decided to refactor and extend the original project while preserving its core functionality. The result is my fork of the library:

https://github.com/andremmfaria/ESP32Wiimote

The goal of the fork is not to replace the original work, but to evolve it into a more maintainable and production-ready Arduino library. The improvements focus on code organization, runtime features, testing infrastructure, and integration with the broader Arduino ecosystem.

2. The Real Project Behind This Work

The motivation for improving the library came from a practical project: using a Wii controller as a wireless input device for Home Assistant.

Home automation platforms often rely on smartphones or dedicated remotes for interaction. While these solutions work well, they do not always provide the flexibility of a programmable controller with physical buttons and motion sensors.

A Wiimote offers several advantages in this context:

multiple buttons for triggering actions
accelerometer input for gesture control
extension controllers such as the Nunchuk
reliable wireless connectivity

To integrate the controller with Home Assistant, I designed a small bridge architecture where an ESP32 acts as the Bluetooth interface to the Wiimote and forwards controller events to another system.

The high-level architecture looks like this:

Wiimote
   ↓ Bluetooth
ESP32
   ↓ Serial
Serial → MQTT bridge
   ↓ MQTT
Home Assistant

In this setup:

The ESP32 connects to the Wiimote over Bluetooth and decodes controller input.
Controller events are sent through the ESP32’s serial interface.
A small bridge service converts those events into MQTT messages.
Home Assistant consumes the MQTT events and triggers automations.

This design keeps the ESP32 firmware relatively simple while allowing the rest of the system to run on a more capable host.

For example, a button press could:

toggle lights
activate a scene
control media playback
navigate a dashboard
trigger custom automations

Before building the full integration, however, the underlying Wiimote library needed to be more robust. The ESP32 firmware had to be able to:

detect when controllers disconnect
expose battery status
provide clear debugging output
remain maintainable as new features are added

Improving the Wiimote library therefore became the first step toward enabling this architecture.

In a follow-up article, I will go deeper into the Home Assistant side of the project and describe how the ESP32 firmware, serial bridge, and MQTT integration work together to turn a Wiimote into a home automation controller.

3. Applying Arduino Library Best Practices

The original ESP32Wiimote already provides a solid implementation for connecting ESP32 boards to Wii controllers. The core Bluetooth functionality, input decoding, and extension support were all present and working well.

The goal of this fork was therefore not to redesign the library, but to apply common Arduino ecosystem best practices and make the project compliant with the expectations of the Arduino Library Manager.

The first step was aligning the repository with the standard structure expected by Arduino libraries.

A typical Arduino library layout looks like this:

ESP32Wiimote
 ├── src/
 │   ├── ESP32Wiimote.cpp
 │   └── ESP32Wiimote.h
 ├── examples/
 │   └── wiimote_demo/
 ├── test/
 ├── docs/
 ├── library.properties
 ├── keywords.txt
 └── README.md

This structure is recommended by Arduino because it clearly separates different parts of the project:

src/ contains the library implementation
examples/ provides sketches demonstrating how to use the library
docs/ contains additional documentation
test/ holds automated tests for development

Two metadata files were also added:

library.properties

This file describes the library for the Arduino ecosystem, including its name, version, architecture compatibility, and author information. The Arduino Library Manager uses this metadata to index and distribute the library.

keywords.txt

This file enables syntax highlighting for library classes and functions inside the Arduino IDE, improving the developer experience.

In addition to the structural changes, the repository was cleaned up to follow common Arduino library practices:

ensuring headers and source files are organized inside src/
improving documentation and examples
adding consistent formatting to the codebase
preparing the project for automated testing

These changes do not alter the fundamental behavior of the library. Instead, they make the project easier to maintain, easier to install through Arduino tooling, and easier for other developers to understand and contribute to.

Aligning the project with these conventions also made it possible to submit the library to the Arduino Library Manager, which significantly improves accessibility for users of the Arduino ecosystem.

4. New Runtime Features

Beyond structural improvements, the fork introduces several runtime capabilities that make the library easier to integrate into real applications.

When working with wireless controllers, especially over Bluetooth, applications often need more visibility into the state of the device. The new features focus on improving observability and control.

Connection State Detection

Bluetooth peripherals can disconnect for many reasons: signal loss, power issues, or the controller simply turning off. Applications therefore need a reliable way to determine whether a device is currently connected.

The library now exposes a simple method for checking connection status:

isConnected()

This allows firmware to react appropriately when a controller disconnects. For example, a program can:

trigger reconnection logic
reset controller state
update user feedback such as LEDs or displays
disable actions until the controller reconnects

This functionality becomes particularly important for long-running systems where the ESP32 may stay powered on for days or weeks.

Battery Monitoring

Another addition is access to the Wiimote’s battery level.

The library now provides two related functions:

getBatteryLevel()
requestBatteryUpdate()

This allows applications to monitor controller battery status in real time. In systems where controllers are used frequently, battery monitoring enables useful behaviors such as:

displaying battery status on a dashboard
sending alerts when battery levels are low
preventing unexpected controller shutdown during operation

For home automation scenarios, battery information can also be forwarded to monitoring systems through MQTT or similar telemetry mechanisms.

Improved Logging and Debugging

Debugging Bluetooth communication can be difficult when limited to raw serial output. To make troubleshooting easier, the library introduces a configurable logging system.

Different logging levels allow developers to control how much information is printed during operation. This provides insight into key events such as:

pairing and connection setup
controller initialization
input parsing
extension controller detection

Structured logging makes it much easier to diagnose issues during development or integration.

Expanded Example Sketch

The example sketch included in the repository was also expanded to better demonstrate the library’s capabilities.

The updated example now illustrates:

the full connection lifecycle
button input decoding
accelerometer readings
Nunchuk extension data
battery reporting
periodic update statistics

Instead of acting only as a minimal demo, the example now serves as a reference implementation for developers integrating the library into their own projects.

This combination of new runtime features and improved examples makes the library more suitable for real-world systems where reliability, observability, and maintainability are essential.

5. Adding Automated Testing

One improvement I wanted to introduce early was automated testing. While testing is common in most software projects, it is still relatively uncommon in Arduino libraries, largely because embedded systems interact with hardware and peripherals that are difficult to simulate.

However, even when hardware is involved, there are still many parts of a library that benefit from automated validation. For example, data parsing logic, internal structures, and event handling can often be tested independently of the physical device.

To support this, the project now includes a test/ directory with a basic testing setup. The goal is not to simulate the entire ESP32 environment, but to create a framework where core components of the library can be validated as the code evolves.

This approach provides several benefits:

Safer refactoring – changes can be validated before running them on hardware.
Regression prevention – previously fixed issues are less likely to reappear.
Improved contributor confidence – developers can verify their changes locally.

In addition to automated tests, the example sketch serves as a hardware validation reference. By running the example on a real ESP32 connected to a Wiimote, developers can quickly verify that button events, sensors, and extensions behave as expected.

Testing embedded software will always involve some interaction with real hardware, but combining automated tests with structured examples makes it much easier to maintain the project over time.

6. Publishing to the Arduino Library Manager

After aligning the repository structure with Arduino conventions and improving the library itself, the final step was to make the project easier for others to install and use.

The Arduino ecosystem distributes libraries through the Arduino Library Manager, which indexes libraries from a central repository:

Arduino Library Registry

To make a library available there, it must meet several requirements, including:

a valid library.properties file
a repository layout compatible with Arduino tooling
semantic versioning
a tagged release

Once those requirements were met, the library was submitted to the registry through a pull request:

ESP32Wiimote Arduino Library Manager submission: https://github.com/arduino/library-registry/pull/7883

After the submission was reviewed and the automated checks passed, the library was accepted into the index.

This means the library can now be installed directly from the Arduino IDE using the Library Manager, without needing to manually clone the repository.

For developers, this provides several advantages:

simple installation directly from the IDE
automatic updates when new versions are released
easier discovery within the Arduino ecosystem

Making the library available through the Library Manager helps ensure that ESP32 developers who want to use Wii controllers can install and use the project with minimal setup.

7. Future Improvements

While the library is now easier to use and integrates well with the Arduino ecosystem, there are still several areas where it could evolve further.

One potential improvement is support for multiple Wiimotes connected to a single ESP32. The current implementation focuses on managing a single controller, which is sufficient for many projects. However, some use cases—such as robotics, gaming interfaces, or interactive installations—could benefit from handling multiple controllers simultaneously.

Supporting multiple Wiimotes would likely require improvements in areas such as:

connection management and pairing workflows
tracking controller identities and connection states
handling concurrent input streams
managing Bluetooth resource limits on the ESP32

Another area that could be explored is expanded support for Wiimote extensions. The Nunchuk is already supported, but the Wii ecosystem includes several other extension devices, such as the Classic Controller and MotionPlus. Adding support for these devices would expand the range of inputs available to ESP32-based projects.

There is also room for improving event handling abstractions. Currently, applications interact with decoded controller state and events directly. A higher-level event system could make it easier to write applications that react to button presses, motion events, or controller changes without having to process low-level state updates.

Additional improvements could include:

improving reconnection behavior after controller disconnects
adding optional callback-based input handling
expanding the test suite to cover more scenarios
providing additional example sketches for common use cases

As with many open-source projects, the direction of these improvements will largely depend on the needs of the community and the projects that adopt the library.

Conclusion

The original ESP32Wiimote already provided a solid implementation for connecting Wii controllers to ESP32 boards. This work focused on building on top of that foundation by applying Arduino ecosystem best practices and introducing several practical improvements.

The fork aligns the project with the expectations of the Arduino Library Manager, improves maintainability, and introduces new runtime capabilities that make the library easier to integrate into real applications.

Some of the key improvements include:

Arduino-compliant project structure and metadata
improved documentation and examples
code quality and formatting improvements
automated testing support
improved logging and debugging
runtime features such as connection state detection and battery monitoring
availability through the Arduino Library Manager

The result is a library that keeps the strengths of the original implementation while making it easier for developers to install, use, and extend.

The library is available here:

ESP32Wiimote: https://github.com/andremmfaria/ESP32Wiimote

If you are interested in using Wii controllers with ESP32 boards, this library provides a solid starting point—and hopefully a foundation for even more creative projects in the future.

Mastering Technical Interviews A Practical Guide to the Algorithms That Appear Again and Again

Andre Faria — Mon, 02 Mar 2026 18:03:30 +0000

Technical interviews are often framed as a test of memorization: recognize a pattern, recall a solution, write it under time pressure. This framing has fuelled an entire industry around grinding problem sets and rehearsing answers, as if strong engineers were pattern-recognition machines trained to replay known solutions on demand. Technical interviews are generally designed to evaluate problem-solving ability, reasoning, and coding skills rather than rote recall. Research has shown that many candidates prepare in ways that do not reflect real engineering work, often relying on memorization rather than authentic problem-solving practice.

That isn’t how real engineering works. In practice, developers are expected to analyze incomplete information, reason about trade-offs, gather additional data when needed, and choose an approach that fits the constraints at hand. The best solutions rarely come from recalling a memorized template verbatim; they emerge from understanding the problem deeply and applying the right tools deliberately.

The algorithmic patterns discussed in this article (two pointers, sliding windows, heaps, traversals, dynamic programming, and others) are not meant to be memorized as answers. They are mental models: reusable ways of structuring thought when facing certain classes of problems. When understood properly, they guide reasoning rather than replace it. Many interview-preparation guides emphasize that patterns are meant to teach structured problem decomposition, not memorized solutions.

This guide focuses on those patterns not as a checklist to grind through, but as a toolbox to support problem analysis. The goal is not to “pass interviews by rote”, but to approach technical problems (interview or real-world) with clarity, structure, and sound judgement. Pattern-based preparation is most effective when it builds reasoning skills rather than memorization, reinforcing a problem-solving mindset instead of recall.

1. Two Pointers

Two pointers are useful when an array or string must be processed from two directions or when you need to maintain a pair of indices representing a candidate solution. This approach reduces nested loops into linear scans. It is most effective when the input is sorted, or when the problem involves distances, sums, comparisons between ends, or in-place modifications without extra memory.

Use when:

The array is sorted or can be sorted.
The task involves pairwise relationships: sum to target, maximize or minimize distance, compare left vs. right properties.
The problem asks for in-place rearrangement or partitioning.
You want to eliminate a nested loop and reduce complexity from O(n²) to O(n).

Typical patterns:

Opposite-direction pointers moving toward each other (summing, container area, water trapping).
Same-direction pointers, where one pointer marks the “write” position (Move Zeroes, Dutch Flag sorting).

Example template (sum-based):

l, r = 0, len(nums) - 1
while l < r:
    s = nums[l] + nums[r]
    if s == target:
        return [l, r]
    elif s < target:
        l += 1
    else:
        r -= 1

Example template (in-place compaction):

def moveZeroes(nums):
    insert = 0
    for i in range(len(nums)):
        if nums[i] != 0:
            nums[insert], nums[i] = nums[i], nums[insert]
            insert += 1

2. Sliding Window

Sliding windows handle problems involving contiguous subarrays or substrings. The key idea is maintaining a window [l, r] with properties that can be updated as r expands and l contracts. This avoids recomputation and typically yields O(n) complexity. Sliding windows come in fixed-size and variable-size forms.

Use when:

The problem explicitly requires considering contiguous sequences.
The goal is to maximize/minimize length, find the longest substring with constraints, or compute sums efficiently.
There is a property that can be updated incrementally when the window expands or shrinks.
Hash maps or counters are used to track window validity.

Fixed-size window:

Used when the window size is given (e.g., “subarray of size k”).
Simply slide by removing leftmost element and adding rightmost.

Variable-size window:

Used when the window grows until invalid and then shrinks to restore validity.
Common in distinct-character constraints or frequency-based problems.

Example fixed-size:

def max_sum_subarray(nums, k):
    window_sum = sum(nums[:k])
    best = window_sum
    for i in range(k, len(nums)):
        window_sum += nums[i] - nums[i - k]
        best = max(best, window_sum)
    return best

Example variable-size:

def lengthOfLongestSubstring(s):
    seen = {}
    l = 0
    best = 0
    for r, ch in enumerate(s):
        if ch in seen and seen[ch] >= l:
            l = seen[ch] + 1
        seen[ch] = r
        best = max(best, r - l + 1)
    return best

3. Intervals

Interval problems revolve around operations on ranges [start, end]. Solutions almost always begin with sorting intervals, and reasoning about overlaps, merges, or gaps. Correct management of boundaries is essential. Many problems reduce to merging, insertion, or counting overlapping intervals.

Use when:

Input consists of ranges and you must merge, insert, or count overlaps.
You are asked whether intervals overlap or conflict.
You must determine available or free time.
Greedy techniques become effective after sorting by start or end times.

Core techniques:

Sort by start time when merging or inserting.
Sort by end time when minimizing conflicts.
Maintain a running "current end" to detect overlap or free space.

Example merge:

def merge(intervals):
    intervals.sort(key=lambda x: x[0])
    res = []
    for s, e in intervals:
        if not res or s > res[-1][1]:
            res.append([s, e])
        else:
            res[-1][1] = max(res[-1][1], e)
    return res

Example non-overlapping (minimum removals):

def eraseOverlapIntervals(intervals):
    intervals.sort(key=lambda x: x[1])
    count = 0
    last_end = float('-inf')
    for s, e in intervals:
        if s >= last_end:
            last_end = e
        else:
            count += 1
    return count

4. Stack

Stacks are suitable for problems involving nested structures, reversing order, parsing, or tracking monotonic sequences. A stack keeps context: what has been seen but not yet closed or resolved. Monotonic stacks allow efficient next-greater-element or histogram computations.

Use when:

Parentheses or encoded strings must be validated or decoded.
You need "previous greater/smaller" or "next greater/smaller".
Problems require evaluating expressions or parsing nested formats.
You want to track elements in sorted order while maintaining O(n) amortized time.

Patterns:

Classic push/pop for matching delimiters.
Monotonic stack: maintain increasing or decreasing order to compute ranges efficiently.

Example parentheses:

def isValid(s):
    stack = []
    pair = {')': '(', ']': '[', '}': '{'}
    for ch in s:
        if ch in '([{':
            stack.append(ch)
        else:
            if not stack or stack[-1] != pair[ch]:
                return False
            stack.pop()
    return not stack

Example monotonic (Daily Temperatures):

def dailyTemperatures(T):
    res = [0] * len(T)
    stack = []
    for i, temp in enumerate(T):
        while stack and T[stack[-1]] < temp:
            j = stack.pop()
            res[j] = i - j
        stack.append(i)
    return res

5. Linked List

Linked list techniques rely on pointer manipulation, often requiring careful handling of node references. Many solutions hinge on using fast/slow pointers to detect cycles, identify midpoints, or perform operations relative to the end of the list. Extra memory is usually unnecessary, and elegance depends on pointer management.

Use when:

You must detect cycles or intersections.
The task involves reversing part or all of a list.
Operations depend on the nth node from the end.
You must reorder nodes without converting to arrays.

Patterns:

Fast/slow pointer to find cycles or midpoints.
Dummy nodes to simplify edge-case manipulation.
Two-pointer offset technique for “remove nth from end”.

Example cycle detection:

def hasCycle(head):
    slow = fast = head
    while fast and fast.next:
        slow = slow.next
        fast = fast.next.next
        if slow == fast:
            return True
    return False

Example remove nth:

def removeNthFromEnd(head, n):
    dummy = ListNode(0, head)
    slow = fast = dummy
    for _ in range(n):
        fast = fast.next
    while fast.next:
        slow = slow.next
        fast = fast.next
    slow.next = slow.next.next
    return dummy.next

6. Binary Search

Binary search applies to sorted arrays or to problems where the answer lies in a monotonic search space. You can binary-search over indices, values, or even abstract answers (binary search on “feasibility”). A solution is valid if increasing or decreasing the parameter changes feasibility in a predictable (monotonic) way.

Use when:

The array is sorted, rotated, or partially sorted.
The problem asks for first/last occurrence, boundary, or pivot index.
You can express the question as: “Is x feasible?” and feasibility changes monotonically.
You must optimize or minimize some parameter, such as speed, capacity, or rate.

Patterns:

Standard binary search on sorted arrays.
Modified binary search for rotated sorted arrays.
Binary search on answer when the value domain is large but checking feasibility is O(n).

Example binary search:

def binary_search(nums, target):
    l, r = 0, len(nums) - 1
    while l <= r:
        mid = (l + r) // 2
        if nums[mid] == target:
            return mid
        elif nums[mid] < target:
            l = mid + 1
        else:
            r = mid - 1
    return -1

Example binary search on answer (Koko Eating Bananas):

import math

def minEatingSpeed(piles, h):
    l, r = 1, max(piles)
    while l < r:
        m = (l + r) // 2
        hours = sum(math.ceil(p / m) for p in piles)
        if hours <= h:
            r = m
        else:
            l = m + 1
    return l

7. Heap (Priority Queue)

Heaps are ideal when the problem requires repeatedly extracting the minimum or maximum element, or maintaining a dynamic set where only the top-k items matter. They guarantee O(log n) insertion and extraction and are essential when selecting the smallest/largest elements without fully sorting. Heaps shine in multi-way merging, streaming problems, and any scenario where you need efficient “best candidate” retrieval.

Use when:

The task asks for the k smallest/largest items.
You need to continuously push/pop values while keeping only the top k.
You must merge multiple sorted lists or streams.
A greedy algorithm relies on always selecting the current minimum or maximum.

Patterns:

Min-heap for selecting smallest; use negative values for max-heap behavior.
Size-k heaps to ensure O(n log k) solutions.
Tuples in heaps for ordering by multiple properties.

Example: Kth Largest Element

import heapq

def findKthLargest(nums, k):
    heap = nums[:k]
    heapq.heapify(heap)
    for x in nums[k:]:
        if x > heap[0]:
            heapq.heapreplace(heap, x)
    return heap[0]

Example: Merge K Sorted Lists

import heapq

def mergeKLists(lists):
    heap = []
    for i, node in enumerate(lists):
        if node:
            heapq.heappush(heap, (node.val, i, node))
    dummy = ListNode(0)
    cur = dummy
    while heap:
        val, i, node = heapq.heappop(heap)
        cur.next = node
        cur = node
        if node.next:
            heapq.heappush(heap, (node.next.val, i, node.next))
    return dummy.next

8. Depth-First Search (DFS)

DFS is used for exploring deep paths in trees or graphs, inspecting components, and performing recursive structural computations. It is especially useful when the problem requires visiting all nodes in a connected component, generating all possible paths, or computing metrics that depend on recursive aggregation. DFS works on both trees and general graphs, using visited sets to avoid cycles.

Use when:

The problem requires exploring all paths or all nodes in a region.
Tree problems that involve computing depth, height, tilt, diameter, or checking validity.
Graph problems involving connected components, cloning, or traversal.
Grid problems identifying islands, regions, or flood fill behavior.

Patterns:

Recursive DFS for tree or grid problems.
Stack-based DFS for graph problems.
Mark visited nodes to prevent infinite loops.

Example: Maximum Depth of Binary Tree

def maxDepth(root):
    if not root:
        return 0
    return 1 + max(maxDepth(root.left), maxDepth(root.right))

Example: Number of Islands (grid DFS)

def numIslands(grid):
    rows, cols = len(grid), len(grid[0])

    def dfs(r, c):
        if r < 0 or c < 0 or r >= rows or c >= cols or grid[r][c] != '1':
            return
        grid[r][c] = '0'
        dfs(r+1, c)
        dfs(r-1, c)
        dfs(r, c+1)
        dfs(r, c-1)

    count = 0
    for r in range(rows):
        for c in range(cols):
            if grid[r][c] == '1':
                count += 1
                dfs(r, c)
    return count

9. Breadth-First Search (BFS)

BFS excels at shortest-path problems on unweighted graphs, level-order processing in trees, and multi-source propagation (spreading effects over steps). BFS processes nodes layer by layer, guaranteeing the minimum number of steps to reach targets. It is the appropriate choice when the question involves minimum distances, time steps, or systematic level traversal.

Use when:

The problem asks for the shortest number of steps in an unweighted setting.
You must process a tree or graph level by level.
Multi-source diffusion problems: rotting oranges, spread of signals, BFS from multiple starting states.
Grid problems requiring finding the minimal distance to something.

Patterns:

Use a queue and process nodes per level.
Use visited sets for cycles in graphs.
Push all initial sources before starting (multi-source BFS).

Example: Level Order Traversal

from collections import deque

def levelOrder(root):
    if not root:
        return []
    q = deque([root])
    res = []
    while q:
        level = []
        for _ in range(len(q)):
            node = q.popleft()
            level.append(node.val)
            if node.left: q.append(node.left)
            if node.right: q.append(node.right)
        res.append(level)
    return res

Example: Rotting Oranges (multi-source BFS)

from collections import deque

def orangesRotting(grid):
    rows, cols = len(grid), len(grid[0])
    q = deque()
    fresh = 0

    for r in range(rows):
        for c in range(cols):
            if grid[r][c] == 2:
                q.append((r, c, 0))
            elif grid[r][c] == 1:
                fresh += 1

    minutes = 0
    while q:
        r, c, t = q.popleft()
        minutes = max(minutes, t)
        for dr, dc in ((1,0),(-1,0),(0,1),(0,-1)):
            nr, nc = r+dr, c+dc
            if 0 <= nr < rows and 0 <= nc < cols and grid[nr][nc] == 1:
                grid[nr][nc] = 2
                fresh -= 1
                q.append((nr, nc, t + 1))

    return minutes if fresh == 0 else -1

10. Backtracking

Backtracking is the algorithmic backbone for generating all valid configurations under constraints. It searches through the solution space using depth-first exploration while pruning invalid options as early as possible. This allows concise solutions for combinatorial problems, exhaustive enumeration, and constructing sequences step-by-step while maintaining validity.

Use when:

The problem requires generating all subsets, permutations, or combinations.
There is a need to explore choices step-by-step while respecting constraints.
Validity can be checked incrementally, allowing pruning of branches.
Search space is exponential and requires efficient pruning.

Patterns:

Recursive function with state path and decision index.
Undo action (path.pop()) after exploring each branch.
Prune early when the partial solution already violates constraints.

Example: Subsets

def subsets(nums):
    res = []
    def dfs(i, path):
        if i == len(nums):
            res.append(path[:])
            return
        dfs(i+1, path)
        path.append(nums[i])
        dfs(i+1, path)
        path.pop()
    dfs(0, [])
    return res

Example: Generate Parentheses

def generateParenthesis(n):
    res = []
    def backtrack(path, open_count, close_count):
        if len(path) == 2*n:
            res.append(path)
            return
        if open_count < n:
            backtrack(path + "(", open_count + 1, close_count)
        if close_count < open_count:
            backtrack(path + ")", open_count, close_count + 1)
    backtrack("", 0, 0)
    return res

11. Graphs (Topological Sort)

Topological sort is applied to directed acyclic graphs when you must determine an order of tasks respecting prerequisites. Cycle detection is inherent: if no valid ordering exists, the graph contains a cycle. It is frequently used for scheduling, dependency resolution, and course prerequisite problems.

Use when:

The problem mentions prerequisites, dependencies, ordering, or sequence validity.
You must determine if a cycle exists in a directed graph.
You must output a valid order of completion.
Nodes represent tasks; edges represent dependencies.

Patterns:

Compute in-degree of nodes.
Use a queue to process nodes with in-degree zero.
Remove edges gradually and collect nodes in order.

Example: Can Finish (detect feasibility)

from collections import defaultdict, deque

def canFinish(numCourses, prerequisites):
    graph = defaultdict(list)
    indegree = [0] * numCourses
    for a, b in prerequisites:
        graph[b].append(a)
        indegree[a] += 1

    q = deque(i for i in range(numCourses) if indegree[i] == 0)
    taken = 0
    while q:
        u = q.popleft()
        taken += 1
        for v in graph[u]:
            indegree[v] -= 1
            if indegree[v] == 0:
                q.append(v)
    return taken == numCourses

Example: Course Schedule II (return ordering)

def findOrder(numCourses, prerequisites):
    from collections import defaultdict, deque
    graph = defaultdict(list)
    indegree = [0] * numCourses
    for a, b in prerequisites:
        graph[b].append(a)
        indegree[a] += 1

    q = deque(i for i in range(numCourses) if indegree[i] == 0)
    order = []
    while q:
        u = q.popleft()
        order.append(u)
        for v in graph[u]:
            indegree[v] -= 1
            if indegree[v] == 0:
                q.append(v)
    return order if len(order) == numCourses else []

12. Dynamic Programming (DP)

Dynamic programming is appropriate when a problem can be decomposed into overlapping subproblems with optimal substructure. DP trades space for time, storing intermediate results to avoid recomputation. Problems involving counting ways, optimizing values, or building solutions from smaller components often map directly to DP formulations.

Use when:

Optimal solutions depend on solutions to smaller subproblems.
The problem has overlapping subproblems and cannot be solved greedily.
You recognize patterns like knapsack, subsequences, paths, decoding, or interval DP.
The recurrence relation naturally emerges from the problem statement.

Types:

1D DP for sequences (Decode Ways, Word Break).
2D DP for grids (Unique Paths, Maximal Square).
DP + binary search for LIS-style problems.
DP on intervals or structure-dependent DP when combining segments.

Example: Decode Ways

def numDecodings(s):
    if not s or s[0] == '0':
        return 0
    dp = [0] * (len(s)+1)
    dp[0] = dp[1] = 1
    for i in range(2, len(s)+1):
        if s[i-1] != '0':
            dp[i] += dp[i-1]
        if 10 <= int(s[i-2:i]) <= 26:
            dp[i] += dp[i-2]
    return dp[-1]

Example: Longest Increasing Subsequence (DP + binary search)

import bisect

def lengthOfLIS(nums):
    dp = []
    for x in nums:
        i = bisect.bisect_left(dp, x)
        if i == len(dp):
            dp.append(x)
        else:
            dp[i] = x
    return len(dp)

13. Greedy Algorithms

Greedy algorithms make locally optimal decisions at each step with the expectation that these choices lead to a global optimum. They rely on the problem having a structure where greedy-choice and optimal substructure properties naturally hold. Once you commit to a choice, you do not revisit it, making solutions efficient and typically O(n) or O(n log n).

Use when:

The problem can be solved by repeatedly taking the best immediate option.
Sorting helps reveal an order that makes greedy decisions valid.
You are maximizing or minimizing a metric such as profit, number of intervals, or fuel balance.
Backtracking or DP is unnecessary because future steps do not depend on alternative past choices.

Patterns:

Track running min/max (Best Time to Buy/Sell Stock).
Maintain cumulative resource balance (Gas Station).
Advance by the farthest reachable index each step (Jump Game).

Example: Best Time to Buy and Sell Stock

def maxProfit(prices):
    min_price = float('inf')
    best = 0
    for p in prices:
        min_price = min(min_price, p)
        best = max(best, p - min_price)
    return best

Example: Jump Game

def canJump(nums):
    reachable = 0
    for i, jump in enumerate(nums):
        if i > reachable:
            return False
        reachable = max(reachable, i + jump)
    return True

14. Trie

Tries efficiently store and query large sets of strings, especially when prefix operations are frequent. They organize characters in a tree-like structure where each path from root to node represents a prefix. Tries allow O(m) lookup where m is the word length, independent of how many words exist. They are fundamental for autocomplete, prefix filtering, and dictionary checks.

Use when:

The task involves prefix search or prefix matching.
You must repeatedly query or insert strings with overlapping prefixes.
Problems ask whether any word starts with a given prefix.
Searching character-by-character offers more efficiency than scanning all strings.

Patterns:

Each node contains a map of children.
Mark end = True for completed words.
Walk the trie for searching or prefix validation.

Example: Trie Implementation

class TrieNode:
    def __init__(self):
        self.children = {}
        self.end = False

class Trie:
    def __init__(self):
        self.root = TrieNode()

    def insert(self, word):
        node = self.root
        for ch in word:
            node = node.children.setdefault(ch, TrieNode())
        node.end = True

    def search(self, word):
        node = self.root
        for ch in word:
            if ch not in node.children:
                return False
            node = node.children[ch]
        return node.end

    def startsWith(self, prefix):
        node = self.root
        for ch in prefix:
            if ch not in node.children:
                return False
            node = node.children[ch]
        return True

Example use case indicator:

Input: many words, many queries → trie fits.
Task: “return the number of words with a given prefix” or “determine if any word begins with prefix”.

15. Prefix Sum

Prefix sums transform cumulative operations into O(1) queries by precomputing running totals. They allow rapid calculation of subarray sums, difference queries, and frequency-based insights. Instead of recomputing from scratch, you subtract two prefix values to get the sum of any range.

Use when:

The problem involves frequent sum-of-subarray queries.
You must detect subarrays with a target sum or pattern.
Overlapping subarrays need efficient comparison.
A running balance or cumulative measure is helpful.

Patterns:

prefix[i] = nums[0] + ... + nums[i-1]
Subarray sum from i to j: prefix[j+1] – prefix[i]
Hash map of prefix sums to detect subarrays with specific targets.

Example: Subarray Sum Equals K

from collections import defaultdict

def subarraySum(nums, k):
    prefix = 0
    count = 0
    freq = defaultdict(int)
    freq[0] = 1
    for x in nums:
        prefix += x
        count += freq[prefix - k]
        freq[prefix] += 1
    return count

Example use cases:

“Count subarrays with sum k.”
“Find how many substrings satisfy some cumulative constraint.”

16. Matrices

Matrix problems require structured 2D traversal, manipulation, or transformation. Many tasks involve row/column operations, rotation, flooding, or spiral traversal. Solutions often rely on systematic scans or in-place transformations to maintain O(1) space. Index manipulation is the core challenge: understanding how rows and columns shift relative to one another.

Use when:

The question involves grid-based movement or transformations.
Problems require rotating, flipping, zeroing rows and columns.
Spiral-order traversal or layer-by-layer operations apply.
2D constraints create natural boundaries for iteration.

Patterns:

Use boundary pointers for spirals.
Matrix transpositions and reversals for rotations.
Row/column flags for operations like Set Matrix Zeroes.

Example: Spiral Matrix

def spiralOrder(matrix):
    res = []
    top, bottom = 0, len(matrix)-1
    left, right = 0, len(matrix[0])-1

    while top <= bottom and left <= right:
        for c in range(left, right+1):
            res.append(matrix[top][c])
        top += 1

        for r in range(top, bottom+1):
            res.append(matrix[r][right])
        right -= 1

        if top <= bottom:
            for c in range(right, left-1, -1):
                res.append(matrix[bottom][c])
            bottom -= 1

        if left <= right:
            for r in range(bottom, top-1, -1):
                res.append(matrix[r][left])
            left += 1

    return res

Example: Rotate Image (90° clockwise)

def rotate(matrix):
    n = len(matrix)
    for i in range(n):
        for j in range(i+1, n):
            matrix[i][j], matrix[j][i] = matrix[j][i], matrix[i][j]  # transpose
    for row in matrix:
        row.reverse()

Example: Set Matrix Zeroes

First pass: mark zero rows and columns.
Second pass: zero out cells in marked rows/columns.

Conclusion

Technical interviews should not reward the ability to memorize solutions or replay patterns on cue. Engineering is not an SAT exam, and developers are not pattern-recognition machines. In real systems, problems are ambiguous, data is incomplete, and the correct approach often emerges only after careful analysis or after asking better questions and gathering more information.

The algorithmic techniques covered in this article are best understood as tools, not answers. They are ways of shaping thought, reason about constraints, structure data, and reduce complexity. Used correctly, they help engineers arrive at solutions; used mechanically, they become blunt instruments.

For candidates, this means focusing less on grinding problems and more on understanding why a technique applies, when it does not, and how to adapt it when conditions change. Short, deliberate practice sessions that reinforce reasoning and trade-off analysis are far more valuable than endless repetition.

For interviewers, it means designing interviews that reflect real engineering work: encouraging exploration, validating assumptions, and thoughtful decision-making, rather than forcing candidates to perform another memorization exercise under time pressure. There is growing discussion in the engineering community about moving beyond purely LeetCode-style interviews toward formats that better reflect real-world problem solving.

Master the concepts, not the scripts. Treat patterns as a toolbox, not a collection of hammers. The goal isn’t luck or recall. It’s clarity, judgement, and the ability to reason your way to a solution.

When Chat Turns into Control - Security Lessons from Running a Local AI Agent using OpenClaw

Andre Faria — Sun, 22 Feb 2026 01:49:59 +0000

Running large language models locally is easier than ever. With tools like Ollama and frameworks such as OpenClaw, it’s now trivial to deploy AI agents that reason, keep state, and execute actions on private hardware.

That convenience comes with a catch.

Once an LLM is wired to tools and exposed through a platform like Discord, it stops being “just a chatbot.” It becomes a control surface driven by natural language, where user input can directly influence system behaviour. In that context, traditional security assumptions like clear trust boundaries, strict input validation, predictable execution no longer hold ground.

This article is not an installation guide. It’s a security-focused reflection on running a local AI agent: where the real risks appear, why “self-hosted” does not automatically mean “safe,” and which design choices actually reduce the blast radius when things go wrong.

1. Context and setup

Running LLMs locally has become easy enough that many people now treat them like “just another service.” Tools like OpenClaw push this further by turning an LLM into an agent: something that can reason, keep state, and execute actions.

In this setup, the agent is controlled through Discord, backed by a local Ollama instance. The deployment looks like this:

Ollama runs on a dedicated TrueNAS host with an RTX 3070, handling all LLM inference.
Model: Qwen3 8B, chosen for being fast and efficient on consumer GPUs.
OpenClaw runs on a separate Linux VM, acting as the agent control plane.
The two hosts communicate over the local network.
Discord is the primary user interface.

Everything is self-hosted and not directly exposed to the internet. At first glance, this feels “safe enough.” But once you let an agent do things, not just chat, you’re no longer dealing with a toy system. You’re running automation driven by natural language, which changes the security model completely.

2. Architecture and trust boundaries

At a high level, the system has three layers:

Discord – where humans talk to the agent
OpenClaw – where decisions, memory, and tool execution happen
Ollama + LLM – where language is generated

Each layer crosses a trust boundary.

Discord is an untrusted input surface, even if the users themselves are trusted. Messages can include pasted text, links, logs, or content copied from elsewhere. Research on prompt injection shows that attackers don’t need direct access to the model—indirect injection through user-supplied content is often enough to override intended behaviour (MDPI, 2024).

OpenClaw sits in the middle as a control plane. It turns text into actions. The problem is that LLMs don’t distinguish between “instructions” and “data.” Everything is just language. This is a known and well-documented weakness of LLM systems, and it’s why prompt injection keeps showing up as the dominant failure mode in agent-based designs (arXiv:2601.09625).

Finally, when the agent can execute tools (filesystem access, memory writes, or web fetches) the risk escalates. Academic and industry analyses consistently show that once an injected prompt can chain actions, the impact is no longer limited to bad answers; it can affect the system itself (arXiv:2410.23308).

One important takeaway: running Ollama and OpenClaw on separate hosts improves performance and resilience, but it does not automatically solve these security problems. The weakest link is still the language interface.

3. The security problem with small models

Qwen3 8B is a great fit for a home lab: it’s fast, it runs well on a consumer GPU (RTX 3070), and it’s cheap to keep online. The downside is that small-ish models are easier to steer off course.

That matters because agents don’t just “answer questions.” They can call tools, update memory, and sometimes fetch or interpret external content. Prompt injection is now widely treated as a top-tier LLM risk for exactly this reason: language is both data and instructions, and the model can be tricked into treating untrusted text as “policy.” OWASP calls this out directly as a primary risk category for LLM apps. (OWASP)

Where it gets nasty is indirect prompt injection: the attacker doesn’t need to DM your bot with an obviously malicious prompt. They just need your agent to consume content that contains hidden instructions (HTML, docs, logs, etc.). This has been demonstrated repeatedly for web agents, where malicious strings embedded in a page can hijack agent behaviour. (arXiv:2507.14799)

So the core issue isn’t “Qwen is bad.” It’s:

Small model + tool access = higher chance of bad tool calls
Small model + web/content ingestion = bigger prompt injection surface
Once it’s an agent, you have to assume the model will occasionally do the wrong thing

That’s why the security posture for small models tends to be: contain the blast radius (sandbox) and remove the easiest injection paths (web fetch / browser). (OWASP Cheat Sheet Series)

4. Discord as an attack surface

Discord feels like a friendly UI, but from a security perspective it’s an untrusted command channel. Anything users paste (logs, URLs, config snippets) can become “model input,” and that’s enough for prompt injection to show up.

The two main problems are:

Scope creep: “it’s only our server” slowly becomes “it’s in more channels than intended”
Permission drift: roles change, new channels get created, people invite the bot elsewhere

So the safe baseline is: deny by default, then allow only what you actually need.

In practice, that means:

Lock the bot to specific guild(s) (server allowlisting)
Restrict usage to a specific role (role gating)
Decide whether normal messages must be mention-gated (reduce accidental triggers)
Handle slash commands explicitly (they have their own permissions model in Discord)

Discord itself supports controlling who can use slash commands through its permissions system (and it’s worth doing that at the Discord layer, not just in the bot). (Discord)

This is the key mental shift: even if the model runs locally and the gateway isn’t public, Discord is still a big input funnel. Treat it like an API surface: least privilege, explicit allowlists, and “assume someone will paste something dumb eventually.” OWASP’s guidance maps well here: prompt injection is not rare, and the best defenses are limiting what the model can do when it gets it wrong. (OWASP)

5. Sandboxing and tool restriction

Once the agent was wired to Discord and running a small model, the real risk wasn’t wrong/bad answers. It was uncontrolled side effects. This is where sandboxing becomes essential.

In OpenClaw, sandboxing means session-level isolation for tool execution. Each conversation runs inside a constrained environment, with no access to the host filesystem or other sessions. If the model does something wrong, the impact is contained.

Enabling sandboxing globally is a single configuration change:

openclaw config set agents.defaults.sandbox.mode all
openclaw config set agents.defaults.sandbox.scope session
openclaw config set agents.defaults.sandbox.workspaceAccess none

This follows OpenClaw’s sandboxing model, which prioritizes containment over perfect prevention (docs.openclaw.ai/sandbox).

The second part of the fix was disabling web-based tools. Web access is the most common prompt-injection vector in agent systems: arbitrary, attacker-controlled text gets fed directly into the model. This has been repeatedly demonstrated in both academic work and industry analyses of indirect prompt injection (arXiv:2507.14799).

In practice, this meant explicitly turning off web fetch and denying the entire web tool group:

openclaw config set tools.web.fetch.enabled false
openclaw config set tools.deny '["group:web","browser"]'

The last item to complete the fix was to add a rate limiting on the auth attempts on the gateway

openclaw config set gateway.auth.rateLimit '{ "maxAttempts": 10, "windowMs": 60000, "lockoutMs": 300000 }'

This means that there are a max of 10 failed attempts per minute and it locks out for 5 minutes after that.

After these changes:

Tool execution became more predictable
Web-based injection paths were removed
OpenClaw’s built-in security audit reported zero critical or warning findings

This matches OWASP’s guidance for LLM applications: assume prompt injection will eventually happen, and focus on reducing blast radius instead of relying on model behaviour alone (OWASP LLM Top 10).

6. Takeaways

A few clear lessons came out of this setup:

Local LLMs are not automatically safe just because they are self-hosted
Discord is an attack surface, not just a chat UI
Small models like Qwen3 8B are efficient, but need more guardrails
Sandboxing matters more than model choice
Removing web access dramatically reduces risk
Separating Ollama and OpenClaw hosts improves resilience, not security

Most of these conclusions line up with existing research and security guidance. Prompt injection, permission drift, and over-trusted tools are expected failure modes, not edge cases (OWASP Prompt Injection Cheat Sheet), (arXiv:2410.23308).

The takeaway is simple: once an LLM can act, it must be treated like infrastructure. With sandboxing, explicit allowlists, and tool restrictions, a local agent can be both powerful and reasonably safe — but only if security is part of the design from the start.

I wanted to know how malware works, so I built an analyser

Andre Faria — Wed, 10 Dec 2025 11:41:32 +0000

1. Introduction & Motivation

When I began thinking about what to do for my Master’s thesis, one question kept resurfacing: How do people actually classify malware? I had always been curious about the internal logic behind malware categorization, not just at a high level, but at the level of processes, features, and decision-making.

In the end, the thesis became more of a means to an end: a structured excuse to finally build something I’d wanted for years, my own static malware analyser.

To do that, I needed a system that was:

Reproducible, so others could follow the same steps
Interpretable, so each decision had a clear explanation
Automated, so large numbers of samples could be processed
Modular, so rules, enrichment, or extraction could evolve over time

This article describes how I designed the baseline analysis pipeline, what I learned from it, and why building it was the most effective way to understand how malware works (see survey: ResearchGate).

Why Static Analysis not Dynamic analysis or both?

I chose static analysis because it’s the simplest, safest way to make progress fast. You can point mature tools like Ghidra at a binary and immediately get structure, imports and strings—no sandbox to provision, no risk of executing the sample, and results that are easy to trace back to rules. That makes static ideal for batch triage and for learning: it’s repeatable, quick, and interpretable.

Of course, static has blind spots. Dynamic analysis shows what the program actually does at runtime—process creation, network I/O, registry and file changes—and it can expose unpacking or decryption that static won’t see. The trade‑off is overhead and fragility: running malware safely requires instrumentation and isolation, it’s slower per sample, and many families try to evade sandboxes. My approach was to start with static to build a clear baseline, then layer enrichment (and later, hybrid methods) where deeper behaviour visibility is needed.

2. High-Level Architecture of the Baseline Pipeline

The baseline pipeline is intentionally simple. It follows a straight, modular workflow:

Feature Extraction – gather structural and semantic information from the PE file.
Heuristic Evaluation – apply rule-based checks to detect suspicious patterns.
Optional Data Enrichment – pull external intelligence (e.g., VirusTotal) for reference.
Decision Fusion – combine heuristic signals with enrichment (if available).
Reporting – output structured evidence, classification, and metadata.

Each component has a narrow purpose and produces structured data that the next stage consumes. This keeps the design predictable and transparent.

On the optional enrichment step

The enrichment layer is intentionally optional. In theory, it makes the classification stronger because the heuristic output can be cross-checked against external intelligence.

But enrichment also introduces an unexpected trade-off:

If the heuristic analysis is roughly aligned with the enrichment data, the result improves.
If the heuristic analysis is far off from the enrichment (e.g., near-random heuristics), the fusion process can skew the final label in unhelpful ways.

So enrichment is useful, but only when the baseline heuristics are not too noisy. This became a recurring theme in the project.

3. Extracting Features from Malware Samples

Static analysis begins with extraction gathering every meaningful property of a file without running it (overview: IJRASET). This includes:

PE metadata
Section layout
Import tables
Strings
Function signatures and decompiler output
Embedded resources
Other structural features

In the baseline, the decompiler stage writes a per-sample features JSON you can reuse downstream. Typical fields include program (name, format, language, compiler, image_base, size, sha256), functions, imports, sections, strings, and optional decompiled function records. For runs, artifacts are written under a run folder (e.g., decompile-<RUN_ID>/<sha256>.features.json).

Why only PE binaries (and how to adapt)

For the experiments in this article, I focused on PE (Portable Executable) binaries (.exe, .dll, .sys). Two practical reasons guided this decision:

PE is the most widespread format in desktop malware telemetry (Windows dominance in consumer endpoints).
Tooling and ecosystem maturity are strongest around PE (Ghidra processors, import table conventions, common packers/obfuscators), which reduces ambiguity when building a baseline.

That focus simplified feature extraction (e.g., sections, imports, entry points) and made heuristic authoring more reliable (static vs dynamic context: SJSU ScholarWorks).

Adapting to other formats is feasible with incremental changes:

ELF (Linux): switch language/processor in Ghidra, adjust extractors for ELF sections/segments, symbol tables, and libc/syscall imports; re-map heuristics to Linux TTPs (e.g., ptrace, /proc tampering, init/systemd persistence).
Mach-O (macOS): use Mach-O program metadata, dyld imports, code signatures/entitlements; adapt persistence/networking rules to macOS paths and launch agents.
Android APK/Dex: pivot to bytecode/decompiled Java/Kotlin; extract manifest, permissions, receivers/services; heuristics on exfil domains, trackers, sensitive API calls.
Scripted binaries (e.g., .NET, Python, JS packagers): add language-aware parsers, focus on runtime loaders, reflection/dynamic resolution, embedded payloads.

Concretely, the baseline needs:

A format detector in the orchestrator to choose the right decompiler/extractor path.
Format-specific feature schemas with a shared core (program info, strings, imports/exports, sections), plus optional blocks per format.
A heuristic ruleset per format (or parametric rules) and a tagging map aligned to each platform’s taxonomy.

With these adaptations, the same pipeline (decompile → heuristics → optional enrichment → fusion → report) extends beyond PE with minimal structural changes.

Why Ghidra (and not radare2, IDA, or Ada-based tools)?

A few people ask why I didn’t use Ada or other specialized tools. The answer is simple:

Ghidra is fully open source
It provides Python bindings through PyGhidra
It integrates a powerful decompiler
It can be automated in a pipeline without licensing issues

That said, Ghidra’s Python bindings are not trivial to use. Because Python and Java operate differently (different memory models, threading assumptions, and API expectations), interacting with Ghidra programmatically can become clunky. But it remained the most practical option.

Limits of the extraction approach

Because this is a lightweight baseline pipeline, the extraction steps are intentionally simple. This leads to a major limitation:

The analysis depends heavily on readable strings and predictable patterns. If the malware is encrypted, packed, or obfuscated, the extracted data becomes almost useless.

This constraint shapes everything downstream in the pipeline.

4. The Heuristics Engine: How the Rules Work

The heuristics engine is the simplest component of the pipeline by design. A heuristic rule is just:

A pure function
That examines extracted features
And returns structured evidence if a condition is met

The logic behind the rules is intentionally basic. Most rules rely on simple string-matching or pattern detection, such as:

Suspicious API calls
Writable/executable sections
Unusual import patterns
Indicators in metadata or strings

A double limitation

Because rules depend on literal string matching:

The input must closely match what the rule expects, or the rule will not fire.
Cryptographed, packed, or obfuscated malware evades the heuristics almost completely.

The upside is interpretability: every rule hit produces clear evidence.
The downside is coverage: many modern malware families will not match at all.

Rule shape and evidence contract

Rules are pure functions that take extracted features and return either Evidence or a miss reason. In REXIS they follow a signature like:

def rule_example(features: dict, rule_score: float = 0.2, params: dict = {}):
 # return (Evidence, "reason") on hit, or (None, "miss reason") on miss
 ...

Evidence is structured with id, title, detail, severity (info|warn|error) and a raw score in [0,1]. The analyser attaches a reason and per-evidence categories (derived from a tagging map) to aid traceability.

Tuning is externalized: a rules config (YAML/JSON) can reweight rules, pass per‑rule params via rule_args, filter by allow_rules/deny_rules, and define label_overrides for strong signals. Tag inference (e.g., ransomware, stealer, backdoor) is computed from evidence via a configurable tagging section.

Tip: Always return a miss reason; it surfaces in rule_misses and makes rule calibration easier.

Authoring and wiring a rule (concrete example)

Here is a simplified example that flags mutex creation APIs, showing the recommended return contract and tunable rule_score:

from typing import Any, Dict, Optional, Tuple
from rexis.utils.types import Evidence
from rexis.tools.heuristics_analyser.utils import get_imports_set

def rule_suspicious_mutex_creation(
 features: Dict[str, Any], rule_score: float = 0.10, params: Dict[str, Any] = {}
) -> Tuple[Optional[Evidence], Optional[str]]:
 imps = get_imports_set(features)
 mutex_apis = {"createmutexa", "createmutexw", "openmutexa", "openmutexw"}
 hits = imps & mutex_apis
 if not hits:
  return None, "no mutex-related imports found"
 return (
  Evidence(
   id="suspicious_mutex_creation",
   title="Mutex creation/manipulation",
   detail=f"Imports include: {', '.join(sorted(hits))}",
   severity="info",
   score=float(rule_score),
  ),
  f"matched mutex imports: {', '.join(sorted(hits))}",
 )

To wire it, register the function with a stable id in the analyser’s ruleset and add a default weight in the config. At runtime, you can raise/lower its impact via weights.suspicious_mutex_creation and pass parameters through rule_args.

Tuning via config (weights, thresholds, tags)

In your rules YAML/JSON you can control:

scoring.combine (weighted_sum|max) and label_thresholds.{malicious,suspicious}
weights: per‑rule caps; contribution is min(1.0, ev.score * weight)
allow_rules / deny_rules: enable/disable subsets
label_overrides: force a label if a rule fires
rule_args: (rule_score, params) per rule
tagging: map evidence to tags (e.g., ransomware, stealer, backdoor) with tag_weights, threshold, top_k

This keeps rule code simple while giving you environment‑specific control.

Testing a rule quickly (ad‑hoc)

from rexis.tools.heuristics_analyser.main import heuristic_classify

features = {
 "program": {"name": "sample.exe", "size": 200_000, "sha256": "...", "format": "pe", "language": "x86"},
 "imports": ["CreateMutexA", "GetProcAddress"],
 "sections": [{"name": ".text", "size": 3500, "flags": ["exec", "write"]}],
 "strings": ["http://example.com", "VirtualBox"],
}

result = heuristic_classify(features)
print(result["score"], result["label"])      # inspect overall score/label
print(result.get("evidence", []))             # list of evidence with reasons
print(result.get("tags", []))                 # tag candidates with scores
print(result.get("rule_misses", []))          # why a rule didn’t fire

5. Enrichment Through External Intelligence (Optional Step)

Enrichment was added only after early experiments revealed a problem:

The heuristics alone generated output that was “too weak” to stand on its own.

Not because the system was flawed, but because simple static heuristics have very limited visibility into modern malware. To counter that, enrichment allows the analyser to pull external data, such as (background: VirusTotal docs):

Hash reputation
Threat vendor classifications
Historical submissions
Known malicious families associated with a SHA-256
Community tags or detection ratios

This creates a baseline to compare the heuristic output against. But enrichment was never meant to override the heuristics, only to contextualize them (what enrichment adds and caveats: Wiz Academy).

Why enrichment is useful but imperfect

It improves confidence when heuristics are directionally correct.
It destabilizes classification when heuristics are very noisy.
It introduces dependency on an external service (API, rate limiting, coverage gaps).

Despite its imperfections, enrichment helped ground the pipeline’s outputs and made the entire system more meaningful.

6. Decision Fusion: Combining All Signals

Once both the heuristic engine and the optional enrichment layer produce their outputs, the pipeline needs a final step that decides:
What is the most reasonable label for this sample?

The decision fusion module combines the available signals:

Heuristic evidence (rule hits, counts, weights)
Optional enrichment (external reputation, vendor labels, known families)

The fusion logic uses a simple, weighted approach:

If heuristics show strong, consistent evidence → they carry more weight.
If heuristics are weak but enrichment is strong → enrichment influences the decision more.
If both are weak → the sample defaults to suspicious or unknown.
If they strongly disagree → the system emits a warning, and the final label is conservative.

This prevents the analyser from being “overconfident,” which is a real risk when combining noisy static heuristics with external reputation data.

Confidence-weighted fusion (with disagreement penalty)

The reconciler computes a final score using per-source confidences and weights:

S_final = clip_01( w_h C_h S_h + w_vt C_vt S_vt − penalty(|S_h − S_vt|) )

S_h, S_vt: heuristics and VT scores in [0,1]
C_h, C_vt: confidences in 0,1
w_h, w_vt: relative weights
penalty(...): applied when both signals exist and disagree beyond a policy threshold

When both sources are high‑confidence yet strongly disagree, a conservative hard‑override can force a mid score and an abstain/suspicious label. Final labels are then chosen via calibrated thresholds (e.g., T_mal=0.70, T_susp=0.40).

The core idea

The fusion layer isn’t meant to be clever, just balanced.
It ensures that neither heuristics nor enrichment dominate blindly, and that the final classification reflects the overall confidence of the system rather than any individual signal.

7. Output, Reporting, and Traceability

Every run of the baseline pipeline produces structured output that makes the analysis reproducible and auditable. For each sample, the system stores:

Extracted features
All heuristic rule evidence
Optional enrichment results
The fused classification label
Metadata (hashes, timestamps, config parameters)
A JSON report representing the entire reasoning chain

This traceability was crucial for the thesis.
It allowed me to re-run experiments, refine rules, compare outputs, and understand how every decision was made. When you are building an analyser from scratch, having visibility into why something happened is as important as the result itself.

Why reporting matters

It makes the pipeline reproducible.
It allows for manual inspection when results are unclear.
It provides ground truth for later LLM/RAG experiments.
It helps identify weak rules, noisy features, or misaligned fusion logic.

The reporting layer ended up being one of the most valuable parts of the pipeline, even though it was initially treated as a simple output function.

Concrete artifact paths

For a run directory like baseline-analysis-<RUN_ID>/, you’ll typically see:

Features: decompile-<RUN_ID>/<sha256>.features.json
Heuristics: <sha256>.baseline.json
Final report (fusion): <sha256>.report.json
Batch runs: baseline_summary.json plus a per‑run baseline-analysis-<RUN_ID>.report.json

8. Lessons Learned from Building a Static Malware Analyser

Building a malware analyser, even a simple baseline one, teaches you a lot about both malware and tooling. A few reflections stood out.

What worked well

The architecture was clear, modular, and easy to extend.
The rule engine was transparent and interpretable.
The pipeline could analyse large sets of files quickly.
It established a solid foundation for later ML and LLM-based experiments.

What didn’t work as well

Static analysis alone struggles with packed or cryptographed malware (see recent studies: ScienceDirect, MDPI).
The heuristic engine is only as good as the extracted strings and it often isn’t enough.
Simple string matching has obvious limits in modern malware ecosystems.
Enrichment, while useful, can distort results when heuristics are too weak.

What surprised me

How quickly the heuristics break when input patterns change.
How hard it is to design “general” rules that work across many families.
How often malware authors rely on simple tricks that defeat static inspection.

How this shaped the next phase of the thesis

These lessons directly informed the development of the LLM + RAG-enhanced pipeline (which will be covered on a dedicated article).
Static heuristics gave me structure, data, and understanding. But not enough depth.
The next logical step was to use LLMs to interpret extracted features more flexibly, grounded by retrieval to avoid hallucinations.

The baseline pipeline provided the scaffolding needed to move forward.

Analysis Results & Repository Structure

The complete artefacts from my experiments live in the repository under analysis/. It has two main branches of outputs and a simple aggregate:

Note: The LLM + RAG pipeline is only referenced here for structure and comparison; I’ll cover its design, prompts, retrieval strategy, and results in a dedicated follow‑up article.

analysis/baseline/: results from the baseline static pipeline (with and without VirusTotal enrichment) (link)
analysis/llmrag/: results from the LLM + RAG pipeline (link)
analysis/aggregation-output.json and analysis/aggregation-report.csv: quick roll‑ups of the per‑run outputs (link)

Directory layout (overview)

analysis/baseline/baseline-analysis-<family>-run-2508/: baseline runs per family (e.g., botnet, ransomware, rootkit, trojan) (examples)
analysis/baseline/baseline-analysis-<family>-run-vt-2508/: same families with VirusTotal enrichment enabled (examples)
analysis/llmrag/llmrag-analysis-<family>-run-2508/: LLM + RAG runs per family (examples)

Inside each run directory you’ll find the per‑sample artefacts described earlier:

decompile-<RUN_ID>/<sha256>.features.json: extracted features
<sha256>.baseline.json: heuristics output
<sha256>.report.json: fused final report (label, score, trace)S
baseline-analysis-<RUN_ID>.report.json: batch‑level summary for the run
baseline_summary.json: compact summary across all processed samples

Baseline analysis: what the runs show

Across the baseline folders, you can inspect how the simple heuristics behave for different malware families and how optional VirusTotal enrichment shifts confidence and labels:

Without enrichment (baseline-analysis-<family>-run-2508/), evidence is driven purely by structural/string‑based signals; many samples land in suspicious or unknown when strings are sparse or obfuscated.
With enrichment (baseline-analysis-<family>-run-vt-2508/), labels tend to stabilize when external reputation aligns with the heuristics; disagreement cases are explicitly noted in the fused reports via the reconciliation policy.

For a quick, high‑level view across runs, open analysis/aggregation-report.csv or the machine‑readable analysis/aggregation-output.json. These aggregate files summarize per‑run counts and label distributions without having to traverse each directory.

If you want to reproduce similar outputs, run the commands in Section 9 and point -o to a top‑level analysis/ directory; the pipeline will create run‑specific folders and the same artefact structure.

9. How to Run It Yourself

The analyser is open-source and can be run with only a few prerequisites:

Requirements

Python environment (follow the installation setup on the repository's README.md for setting up the environment)
Ghidra + PyGhidra (Ghidra installed at /opt/ghidra on Linux). If you need a fast, distro‑agnostic setup, follow my guide: Ghidra on Linux: Zero Fuss Install
A directory of PE files
(Optional) VirusTotal API key for enrichment (set [baseline].virus_total_api_key in config/settings.toml)

Basic usage

Once installed, running the baseline pipeline is straightforward (Typer CLI):

pdm run rexis analyse baseline -i ./data/samples/<file>.exe -o ./data/analysis

or for batch mode:

pdm run rexis analyse baseline -i ./data/samples -o ./data/analysis --parallel 4

Common options:

-i, --input: file or directory to analyse (required)
-o, --out-dir: output directory (defaults to CWD)
-r, --run-name: logical run name (default: UUID)
-y, --overwrite: overwrite existing artifacts
-p, --parallel: workers for directory mode
--rules: path to heuristics rules config (YAML/JSON)
-m, --min-severity: filter returned evidence (info|warn|error)
--vt: enable VirusTotal enrichment (requires API key in config/settings.toml)
--vt-timeout, --vt-qpm: timeout and queries-per-minute budget

Rule customization

Users can:

Add new heuristic rules
Tune weights and thresholds
Enable or disable individual rules
Adjust fusion parameters
Add their own enrichment sources

Where to start

All documentation is available in the repository:

Baseline pipeline guide: https://github.com/andremmfaria/rexis/blob/main/guides/BaselinePipeline.md
Heuristic rule‑writing guide: https://github.com/andremmfaria/rexis/blob/main/guides/WritingHeuristicRules.md
Reconciliation (fusion) details: https://github.com/andremmfaria/rexis/blob/main/guides/Reconciliation.md
Example configurations and sample reports in the repo

This makes it easy to experiment, modify, or build your own extensions.

10. Conclusion: Why Building Tools Is the Best Way to Learn

I started this project because I wanted to understand how malware classification works.
Building my own analyser forced me to confront all the assumptions, shortcuts, limitations, and edge cases that textbooks and blog posts never mention.

What I gained was not just a working pipeline, but a practical understanding of:

how static analysis actually behaves
where heuristics break
why enrichment matters
how evidence should be combined
and how analysts think about classification

The baseline pipeline is not perfect. It was never meant to be.
But it gave me the foundation I needed to build more advanced approaches, including the LLM + RAG pipeline that became the core of the second half of my thesis. This will be covered in a future article.

Most importantly, it taught me this:

If you want to learn how something works, build a tool that does it.
You’ll understand the entire problem far more deeply.

Building a Transparent LAGG (LACP) Bridge with OPNsense, UDM, and UniFi — A Practical Guide

Andre Faria — Mon, 01 Dec 2025 02:52:40 +0000

1. Introduction

I have a pretty heavy network topology at my home. The result of years of IoT devices, user devices, servers and miscellaneous IP enabled things. Those pose some security threats that I longed to correct or, at the very least, monitor. So I had this idea to place a transparent firewall between my UniFi Dream Machine (UDM) and the rest of my network.

I wanted to do this without replacing/reconfiguring the UDM, without breaking VLANs, without touching DNS/DHCP or anything else other than what was really necessary. The goal was simple:

perform minimal changes
stay fully transparent to the network
add filtering/inspection intelligence
do it using LAGG (LACP) to increase throughput (and because it is cool).

This article documents how I built a Layer 2–only transparent bridge using OPNsense with two aggregated links toward the UDM and two aggregated links toward my UniFi switch. It also covers the mistakes I made so you don’t repeat them.

To set the stage, here is the physical setup as it exists today.

On the top shelf we have, from left to right, a pod charger for xbox controller batteries, the USW-16 we will be using for this guide, the BACKUP bay and a small application host. And on the bottom shelf we got the OPNSense box and the UDM.

That host and the BACKUP bay are part of a Home Assistant installation I did with their HAOS system. If you'd like to know more about it let me know.

Sources

I followed these EXCELLENT guides in order to do this:

Video:
How to Configure LAG-LACP
https://www.youtube.com/watch?v=Rb4vlN_Hf-U
Article:
Configure LAG/LACP on SFP Ports (TP-Link Example)
https://homenetworkguy.com/how-to/configure-lag-lacp-on-sfp-ports-two-tp-link-switches-with-vlans/

2. Network Topology

This entire design operates strictly at Layer 2. OPNsense is merely a bump-in-the-wire — it does not route, it does not NAT, and it does not participate in DHCP or DNS.

Those services remain entirely on the UDM:

UDM: DNS, DHCP, routing, VLANs, firewall
OPNsense: Transparent inline bridge with LACP
USW: Downstream switching with LACP

Traffic Flow

UDM → ingresslagg → laggbridge → egresslagg → UniFi Switch

Because the bridge sits inline and has no IP addresses assigned to it, the rest of the network behaves normally. All VLAN tags pass through untouched, and the UDM continues to see the whole network exactly as before.

3. System Setup (Hardware & Software Overview)

Hardware Origin

This OPNsense box came to life after my old laptop suddenly died. Instead of throwing away perfectly good components, I salvaged:

the 32 GB DDR4 RAM,
the two SSDs I later mirrored as RAID-1,
and the Wi-Fi card,

and reused them inside a barebones mini-PC I purchased from Amazon UK.

The chassis ships without RAM or storage, making it ideal for a rebuild using recycled parts.

System Specs

CPU: Intel® Celeron® J6413 @ 1.80 GHz (4 cores / 4 threads)
RAM: 32 GB DDR4-2166
Storage: 128 GB SSD (RAID-1)
NICs: 6 × Intel 1G
OS: OPNsense 25.7.5-amd64

And here is the firewall itself, with all six Ethernet interfaces populated:

4. Why Use LAGG in Transparent Mode?

I chose a transparent firewall because I wanted:

redundancy — two links on each side, loss-tolerant
throughput headroom — LACP distributes sessions across NICs
zero impact on existing network design — no routing changes
full compatibility with UniFi — VLANs, DHCP, and DNS remain untouched

Layer 2 Only

This cannot be overstated:
OPNsense in this setup is 100% Layer 2.
It is not acting as a router, DHCP server, DNS resolver, or gateway.

It simply passes frames, while optionally filtering or inspecting them inline.

5. Configuring LAGG on OPNsense

On the OpnSense management web interface go to Interfaces → Devices → LAGG

There, two LAGGs are created:

LAGG	Interface	Direction	Members
`ingresslagg`	`lagg0`	toward UDM	`igc1`, `igc2`
`egresslagg`	`lagg1`	toward UniFi Switch	`igc4`, `igc5`

Click the "+" sign on the far right side of the table in the interface to add an entry. Repeat this for both entries on the table above.

Physical interface 1
Physical interface 2
Hash layer set for L2. Following the bump-in-the-wire approach.
Provide a meaningful description. Believe me, you will need it later.

Both LAGGs are configured in LACP mode. You can select more than 2 interfaces here, it only depends on how much interfaces you have. I only had 6 so in order to have a management interface (outside of the scope of this guide) i opted to have it be 2x2 (2 interfaces in and 2 interfaces out).

After this is set, it will look something like this:

Click apply.

Next, we need to create an interface from the newly created devices. Go to Interfaces → Assignments and assign a new interface to each of the devices on the Assign a new interface menu. Remember to provide a name in the description field. That will be the interface names.

Here we have lagg-test and lagtest but you can pretend that is either ingress-lagg/egress-lagg and ingresslagg/egresslagg. Click add to create it on the interfaces table like so:

Next, we need to add both interfaces to the bridge configuration. Go to Interfaces → Devices → Bridge and click the "+" sign on the far right side of the table in the interface to add an entry:

1st LAGG interface
2nd LAGG interface
GOOD description of what this device is.

Click Save to add it to the bridge interface table. From here, the newly created LAGG interfaces are added to a single bridge device. This device needs to be assigned to an interface by repeating the process we did for the lagg interface creation. After this is done, we have something like this:

After all of this is done, we now have laggbridge = ingresslagg + egresslagg.

Now, we enable the interfaces. After this is done, under the Interfaces menu, there should appear all interfaces we created (i.e. laggbridge, ingresslagg, egresslagg). Enable them all by clicking on each of the interface names under the Interfaces menu, check the Enable checkbox and click Save.

Management and Services Interfaces

I host some services on this OPNSense box like DynamicDNS and AdGuard. The installation/management of those are outside of the scope of this article, but if you would like for me to write something about those, let me know.

These are independent interfaces I set with fixed IPs (on the UDM side). They stay outside of the bridge and ensure I can always reach OPNsense even if the bridge goes offline:

management (igc3) → 192.168.1.x/24
services (igc0) → 192.168.1.y/24

⚠️ Common mistakes to avoid

Don’t assign IPs to lagg0, lagg1, or the bridge: Assigning IPs would make the bridge participate in Layer 3 (routing), breaking its transparent Layer 2 operation. This could disrupt network traffic, cause routing conflicts, and interfere with VLANs and DHCP.
Don’t enable the physical NICs individually, only the LAGGs: Enabling NICs outside the LAGG group can cause duplicate connections, loops, or flapping, as the LAGG protocol expects to manage all member interfaces. This can destabilize the link aggregation and the bridge.
Don’t mix LACP Active/Passive between devices: LACP requires both ends to be in compatible modes (usually Active). Mixing Active and Passive can prevent proper negotiation, causing the aggregated link to fail or operate unreliably.
Don’t skip a reboot, LAGG + bridge changes apply more cleanly afterward: Network interface and bridge changes may not fully apply until after a reboot. Skipping this step can leave the system in a partially configured state, leading to unpredictable behavior or connectivity issues.

6. Configuring LACP on the UDM (Ingress Side)

The UDM handles the WAN, DHCP, DNS, routing, and VLAN assignments. In this setup, we only need it to expose two LACP ports toward the OPNsense box.

On the updated UniFi interface:

Go to Settings → Profiles → Switch Ports
Create a new Aggregate profile
Set LACP Mode: Active
Apply this profile to ports 7 and 8 (or whichever pair you use)

Follow this very useful guide from Hostify if anything is not clear:

👉 https://support.hostifi.com/en/articles/6454249-unifi-how-to-enable-link-aggregation-on-switches-lag

Only after the profile is in place should you plug in the cables.

⚠️ Don’t #1 — Don’t plug cables in before creating the LACP group

UniFi auto-profiling may assign them to something else (like LAN, PoE, or VLAN-only), breaking negotiation.

⚠️ Don’t #2 — Don’t assume LACP comes up instantly

Give it 5–10 seconds to negotiate.
If one side is up and the other is not, it will flap.

7. Configuring LACP on the UniFi Switch (Egress Side)

On the USW-16, the process is similar:

Open UniFi Controller → Devices → USW
Select ports 7 and 8
Click Aggregate
Set LACP Active
Save and wait for synchronization

These two ports will form the downstream side of the bridge.

⚠️ Don’t #3 — Don’t mix up NIC-to-port mapping

Label the NICs and cables before you start.
One swapped cable is enough to break the bundle or cause intermittent LACP flaps.

8. Testing the Transparent LAGG Bridge

Once everything is connected, verify the bridge and both LAGGs:

On OPNsense

Run:

ifconfig lagg0
ifconfig lagg1
ifconfig laggbridge

You should see all LACP members in ACTIVE state.

In the UniFi Controller

UDM ports 7/8 should show Aggregate — Active
USW ports 7/8 should show Aggregate — Active
No errors, no flapping, no “Blocking” states

Optional: Throughput Testing

You can run an iperf3 test from a LAN device to something outside the bridge. Traffic should flow, no bottleneck should be observed and failover should work if you temporarily unplug one cable from each LAGG

⚠️ Don’t #4 — Don’t enable hardware offloading prematurely

Keep the following disabled until you verify stable operation:

TSO
LRO
Checksum offload
VLAN offload

Some Intel i225/i226 NICs misbehave with offloading in bridge mode.

9. Troubleshooting

Here are the most common issues — and their causes:

Issue: LAGG fails to negotiate

UDM or USW set to Passive instead of Active
Cables on the wrong ports
One side Active, the other Disabled
An interface accidentally assigned directly instead of via LAGG

Issue: Bridge appears up but traffic drops

Hardware offloading left enabled
A NIC is flapping or mismatched
Member NIC enabled individually by mistake
Duplicate MAC confusion (if using the same NIC model)

Issue: Network meltdown (loop)

This is the most catastrophic one.

It happens when something like this occurs:

UDM → OPNsense → USW
 ↑────────────────↓
  accidental loop

A single extra cable or an unconfigured LAGG member can create a broadcast storm.

⚠️ Don’t #5 — Don’t create a second path between UDM and USW

There must be exactly one traffic path:

UDM → ingresslagg → laggbridge → egresslagg → USW

No more, no less.

10. Final Working Architecture

Once configured, the final architecture looks like this:

          WAN
           │
        [ UDM ] Ports 7+8 (LACP Active)
          │  │
          ╰╮╭╯
          ingresslagg (lagg0)
           │
          [ laggbridge ]
           │
          egresslagg (lagg1)
          ╭╯╰╮
          │  │
        [ UniFi USW ] Ports 7+8 (LACP Active)
           │
         rest of LAN

Key properties:

Fully transparent Layer 2
No IPs on LAGGs or the bridge
UDM retains all core services (DHCP, DNS, routing, VLANs)
Redundant links on both sides
Clean inline filtering option for OPNsense

11. Conclusion and results

Building a transparent bridge with LAGG on both sides is a great way to introduce a firewall into your network without redesigning the entire topology. OPNsense, combined with UniFi hardware, handles this setup surprisingly well — once everything is wired and configured correctly.

Along the way, I learned (the hard way) that LACP is extremely sensitive to:

mismatched modes
cable swaps
auto-profiling quirks
enabling NICs individually
and forgetting to disable hardware offloading

But once the pieces fall into place, the result is a rock-solid, redundant inline bridge that just works.

With this setup you are able to see and/or filter EVERYTHING that is happening on your network (at least what passes through the bridge). You can set firewall rules for traffic that passes through the bridge and see cool graphs like this one:

This setup has a lot that can be improved upon. I am open for feedback and would love ideas on how to make this better. Leave a comment below and tell me what you think. Thank you!!!

Ghidra on Linux Zero Fuss Install

Andre Faria — Tue, 11 Nov 2025 00:04:13 +0000

A straightforward, distro-agnostic way to install Ghidra under /opt without touching your system Java alternatives, plus environment wiring that makes upgrades safe and re-runs idempotent.

Disclaimer:
This post assumes you’re comfortable using the terminal with sudo, have basic knowledge of environment variables, and can install a few command‑line tools (wget, unzip, tar) with your distro’s package manager.

What you’ll get

Ghidra 11.4.1 installed under /opt/ghidra (atomic replace with backups on re-run)
A local Temurin (Adoptium) JDK 21 under /opt/java-temurin without touching system alternatives
GHIDRA_INSTALL_DIR and GHIDRA_JAVA_HOME added to your shell RC (~/.bashrc or ~/.zshrc)
A process that is safe to re-run for upgrades and repairs

Note: The approach works across common Linux distributions. The automated script tries to install helper tools via apt when available, but it gracefully skips on other distros—so on RPM/Arch-based systems, just install wget, unzip, and tar via your package manager first.

Customization tip:
You can change the Ghidra version, release date, download URL, or even inject checksum verification simply by editing the constants at the top of the referenced script (GHIDRA_VERSION, GHIDRA_DATE, GHIDRA_URL, GHIDRA_SHA256, plus the Java bits like TEMURIN_MAJOR, TEMURIN_API_URL, TEMURIN_SHA256). Adjust them, re-run the script, and it will perform an atomic upgrade while backing up previous installs.

1) Prerequisites

Install the few tools we’ll use to fetch and unpack files.

Debian/Ubuntu:
- sudo apt-get update && sudo apt-get install -y wget unzip tar
Fedora/RHEL/CentOS:
- sudo dnf install -y wget unzip tar (or yum on older releases)
Arch/Manjaro:
- sudo pacman -S --needed wget unzip tar

Also ensure you have sudo privileges.

2) Install a private Temurin JDK 21 under /opt

Why: Ghidra runs best on a modern LTS JDK. Installing a local Temurin JDK avoids messing with system-wide Java alternatives and keeps Ghidra isolated.

Key details:

Location: /opt/java-temurin/<jdk-version> with a stable symlink at /opt/java-temurin/current
Source: Adoptium API for the latest GA JDK 21 (linux x64, HotSpot) — or browse Temurin releases
Optional integrity: You can verify checksums if you provide TEMURIN_SHA256

Steps (high level):

Create the base directory: /opt/java-temurin
Download the latest GA tarball for JDK 21 from Adoptium’s API
(Optional) Verify the tarball with sha256sum
Extract to a versioned folder under /opt/java-temurin and update the current symlink atomically

On re-run, the previous version is backed up and replaced. The stable current symlink is what we’ll point Ghidra to.

3) Install Ghidra 11.4.1 under /opt/ghidra

Ghidra publishes zip archives per release. We’ll unpack into /opt/ghidra and keep upgrades atomic.

Key details:

Download from the official NSA GitHub Ghidra releases page for 11.4.1
Optional integrity: provide GHIDRA_SHA256 from the release page to verify with sha256sum
The process backs up any existing /opt/ghidra before replacing it

Steps (high level):

Download the Ghidra 11.4.1 zip
(Optional) Verify its checksum
Unzip into a temporary directory, then move to /opt/ghidra atomically

4) Wire your shell environment

Two environment variables make life easier:

GHIDRA_INSTALL_DIR: points to /opt/ghidra
GHIDRA_JAVA_HOME: points to the Temurin JDK (preferred) or falls back to an existing valid JAVA_HOME

What gets added to your shell RC (~/.bashrc for bash, ~/.zshrc for zsh):

export GHIDRA_INSTALL_DIR=/opt/ghidra
export PATH="$PATH:$GHIDRA_INSTALL_DIR/support" (so analyzeHeadless is on PATH)
export GHIDRA_JAVA_HOME=/opt/java-temurin/current (or another detected JDK)

Open a new shell or source ~/.bashrc / source ~/.zshrc to apply the changes.

Optional: add Ghidra binaries to PATH

If you want to call both the headless tools and the GUI launcher from anywhere, add both the support directory and the root install dir to PATH.

Bash:

echo 'export GHIDRA_INSTALL_DIR=/opt/ghidra' >> ~/.bashrc
echo 'export PATH="$PATH:$GHIDRA_INSTALL_DIR/support:$GHIDRA_INSTALL_DIR"' >> ~/.bashrc
source ~/.bashrc

Zsh:

echo 'export GHIDRA_INSTALL_DIR=/opt/ghidra' >> ~/.zshrc
echo 'export PATH="$PATH:$GHIDRA_INSTALL_DIR/support:$GHIDRA_INSTALL_DIR"' >> ~/.zshrc
source ~/.zshrc

5) Verify the installation

Quick checks you can run:

Print Ghidra’s internal version from its properties file
Run a short, non-interactive headless/CLI command to validate scripts are reachable

Examples:

grep -E '^application\.version=' /opt/ghidra/Ghidra/application.properties | cut -d'=' -f2

"$GHIDRA_INSTALL_DIR"/support/analyzeHeadless -version

"$GHIDRA_INSTALL_DIR"/ghidraRun -h

If those commands execute without errors, you’re set.

6) Upgrades and re-runs

This setup is designed to be re-run safely:

Temurin: new JDK drops into a versioned folder, current symlink is updated atomically
Ghidra: any existing /opt/ghidra is backed up to /opt/ghidra.bak-<timestamp> before replacement
Env: GHIDRA_JAVA_HOME is updated or inserted once; subsequent runs update it if needed

To upgrade to a newer Ghidra release, adjust the version/date variables (or pull the latest script) and re-run. Your environment variables remain compatible.

7) Uninstall or roll back

Remove Ghidra: sudo rm -rf /opt/ghidra
Remove Temurin: sudo rm -rf /opt/java-temurin
Clean your shell RC: remove the lines that export GHIDRA_INSTALL_DIR, modify PATH, and GHIDRA_JAVA_HOME
To roll back: if you see a recent backup like /opt/ghidra.bak-YYYYmmddHHMMSS, you can move it back to /opt/ghidra

8) Troubleshooting

Missing tools on non-Debian distros: install wget, unzip, tar via your package manager
Permission denied under /opt: you need sudo privileges for system locations
GUI on servers/WSL: Ghidra’s GUI requires an X server; use headless mode with analyzeHeadless if you don’t have a display
Java detection: if Ghidra prompts for a JDK, ensure GHIDRA_JAVA_HOME is set and points to a valid JDK with bin/java

9) Optional: headless workflow teaser

Ghidra ships analyzeHeadless for automation. For example, to run a simple analysis in a CI runner, you can call it with a temporary project directory and scripts. This is out of scope for this post, but the steps above already place the tool on your PATH for easy use.

Full automation script (reference)

All the steps above are automated in a single idempotent script that you can read and run:

Script: https://github.com/andremmfaria/rexis/blob/main/scripts/install-ghidra.sh

What it does for you:

Installs Temurin JDK 21 under /opt/java-temurin and creates a stable current symlink
Downloads and installs Ghidra 11.4.1 under /opt/ghidra with atomic replace and timestamped backups
Wires GHIDRA_INSTALL_DIR, updates PATH, and sets GHIDRA_JAVA_HOME
Verifies the installation non‑interactively

Safe to re-run. If you want checksum verification for either download, export GHIDRA_SHA256 and/or TEMURIN_SHA256 before running it.

Thanks for reading, and happy reversing!

Further references

Ghidra official site: https://ghidra-sre.org/
Ghidra GitHub repository: https://github.com/NationalSecurityAgency/ghidra
Ghidra releases (official zips + checksums): https://github.com/NationalSecurityAgency/ghidra/releases
Ghidra Getting Started: https://github.com/NationalSecurityAgency/ghidra/blob/master/GhidraDocs/GettingStarted.md
Adoptium Temurin releases: https://adoptium.net/temurin/releases
Adoptium API (programmatic downloads): https://api.adoptium.net/

Continuous integration with containers and inceptions

Andre Faria — Mon, 10 Nov 2025 23:50:19 +0000

How to create a CI system using container recursion and how to create a CI system using container recursion

As many of you already know, containers are something of wonder. They exist since the old days of computing in a concept called OS-level virtualization. Since then, for their flexibility, they have been used in an orchestrated manner by many awesome tools, like Kubernetes, DC/OS, Apache Mesos and many more. This provides not only an abstraction layer on OS-Level but also enables a great deal of automation where there was none before.

I used the Semantic Versioning but you can easily replace this for any versioning patterns that you like. E.g., Calendar Versioning or any other.

I am now going to show you a model of continuous integration (CI) with some "inception".

Disclaimer:
This article assumes you have some knowledge about versioning, CI, CD, containers and orchestration, along with the tooling associated with those concepts.

Tools

For code versioning we will be using the so popular Git on GitHub. You can use pretty much any versioning system in the market with this, I only used git for familiarity. And GitHub for popularity.

Jenkins is an orchestration tool first devised for CI, but since the creation of the pipeline plugin it has become more of a general-purpose orchestrator.

The docker project, or just Docker, is a very neat application that lets you manage containers in a easy and inexpensive way using their command line and API.

For container registry I used Nexus. The free version is quite nice and is more than enough to everything we need. It has a plugin for storage on Amazon's S3 which makes your storage virtually limitless. (Use at your own discretion)

Note1: For container storage you can use any registry available in applications like Artifactory but you can also use cloud services like AWS's ECR, AZURE's Container Registry or GCP's Container Registry.

Note2: There are many other steps you can input on the pipeline for testing, validation, hardening, etc. But I will be covering only the general steps of the pipeline, like code checkout, container pull, app build, container build and container storage.

Model

Some time ago I made a presentation on this topic on a meetup that happened in São Paulo - Brazil. The slides for that presentation can be found here.

This model outlines the basic actors and actions made on the process. The orchestrator, in our case Jenkins, contacts the container engine's, in our case Docker, API over TCP and creates the build container. Then it connects to a random SSH port directly on the build container. From there, the orchestrator has a few workflows based on the build workflow. The workflows on this model build two structures, the app container and other container builders.

Structures

This section shows the results of the mentioned workflows.

Container builds container builders (Recursions are always nice :D)

The build container is a specialized image that is built specifically to build other things, among them other build containers.

From this image, the orchestrator pulls container definitions for specific application languages/build engines, like Java/gradle, c#/.NETcore or javascript/npm, builds and stores them onto the container registry.

Container builds app containers

From the definitions outlined above, the applications can be built from a container specific for their case. I will be using java's case as an example. It should be similar for other cases.

Jenkins provides a nice way to bundle pipelines definitions, among other actions, using shared libraries. Those are basically Groovy scripts and exist to extend Jenkins functionality.

You also need to have a Jenkinsfile on your code's repo for it to know which app engine is to be used for the build. Here is an example Jenkinsfile:

@Library('execPipeline') _

def config = [:]
config.'engine' = 'java'
config.'type' = 'app'

execPipeline(config)

Those are the configurations the pipeline needs to know where to go.

Configuration

These are the configurations for each tool. This might change a bit overtime but the documentation is (or should) be always true.

Jenkins

As stated, above Jenkins is using shared libraries. Its config is fairly simple, you just input the git repo for the shared lib on the config and you're good to go. I used the global shared lib definitions for this example. The full steps can be found here.

Jenkins also talks to docker on a remote host for the container's management. You need to install a docker plugin and configure it properly. The steps on Jenkins's plugin side can be found here. On the docker's side you can follow these steps to enable remote access using various methods.

This assumes that you are using Jenkins multibranch pipelines to create your pipeline's definition. From this, just input your Git repo on the multibranch pipeline and it should roll out by itself.
You can follow this example to configure everything: Build a multibranch pipeline project

Nexus

Configuring Nexus is easy. Just install it following the official documentation and make sure you have storage space. You'll need it.

Set up repositories as you feel like. You can separate per environment (gamma, beta, prod, etc), per app engine type (java, node, .net core, etc), both or none. A good recommendation is to separate each artefact repository on a different port. E.g.: beta repo on port 5000, gamma repo on port 5001, etc.

Docker

Depending on your situation Docker's installation and usage, is be "easy". I mean, if your infrastructure still uses windows for a .net framework app, I feel sorry for you but this method still works.

For Linux you can follow this guide. For windows you can follow this one. For mac… well… simply don't.

Git

You don't need any specific configuration on git's side. Just make sure that Jenkins can clone your repository and your repository contains the Jenkinsfile mentioned in section 3.2 of this article. If both are good, you're golden.

Conclusion

After these configurations are setup, you will see the pipelines for each branch rollout, and your nexus registry being populated with your app containers.

Thanks for reading this article till the end. Have a nice day/afternoon/evening :D

Service metrics and its meanings

Andre Faria — Mon, 10 Nov 2025 23:50:18 +0000

Whenever you have a system, it doesn't really matters what it does, it is of good measure to… well… measure it.

The purposes are multiple, make sure that it is performing well, measure its overall cost, measure its latency to costumers, etc. It is a fact that it is quite important to do so, but then the following question arises: how?

Well, the answer to that question is not easy, as there are aspects of each service/case/app that are to be considered.

A good example of that is Google itself, and I'm not saying that because they have THE book for Site Reliability Engineering, I am talking about relations between the metrics and their inherent meaning on a myriad of services so big that it covers the entire world. On those books they cover a great deal of trial and error on how to measure production systems and how to focus on what is important. See also the companion SRE Workbook and background on postmortem culture.

Albeit to that, you might find that those examples are irrelevant for your use case and you'll have to figure everything out for yourself. And that is OK. But one thing is undeniable, those concepts help a bit when they fall on a use case that fits.

From my point of view, there are two basic types of metrics that you can apply those concepts to. They are inherently interconnected, in the sense that one does not exist without the other. Related framing: product "North Star" metrics (see Amplitude guide) versus engineering reliability KPIs (e.g. Four Golden Signals).

Business meaning

When we talk about metrics from a business standpoint we are trying to quantify whether the service is succeeding in creating (and keeping) value. A metric is a proxy for an outcome; no proxy is perfect, but good ones are:

Connected to customer impact (they move when users are happier or churn when they are not)
Hard to game (improving the number tends to improve the underlying reality)
Stable enough to trend yet responsive enough to detect meaningful change

Leading vs. lagging indicators

Lagging indicators describe results you usually discover too late (monthly revenue, quarterly churn). Leading indicators move earlier and let you correct course (signup conversion, time-to-value, task success rate). A healthy metric set pairs each critical lagging metric to at least one leading metric that you can act upon. For a primer, see Leading vs. Lagging Indicators.

Outcome (Lagging)	Example Leading Metrics	Intervention Window
Customer retention	Onboarding completion %, First action latency	Days/weeks
Revenue growth	Trial -> paid conversion, Expansion feature adoption	Weeks
NPS / Satisfaction (What is NPS?)	Error-free session %, Page performance (p95) (Web Vitals)	Minutes/hours

North Star and guardrails

The North Star metric is the clearest expression of sustained value creation (e.g., "weekly active collaborative documents"). Guardrail metrics prevent optimizing the North Star at the expense of health (support tickets backlog, cost per transaction, reliability SLO compliance). If a push increases the North Star but violates a guardrail, you slow down.

Metric layers

Think in concentric layers:

North Star (one, maybe two)
Strategic pillars (fairly stable: acquisition, activation, retention, efficiency)
Operational KPIs (change more often, tie to teams OKRs)
Diagnostic metrics (rich, detailed; used to explain movement, rarely reported upward)

Business questions to validate

Before adopting a business metric ask: What decision will change if this metric moves? Who owns reacting to it? How fast must we respond? What thresholds define success vs. acceptable vs. alert?

Common business metric mistakes

Measuring everything and prioritizing nothing
Declaring a vanity metric (raw signups) as success without a quality filter
Lacking a clear owner; metrics without owners decay
Setting targets without historical baselines or variance analysis
Not revisiting metrics when the product stage changes (growth vs. efficiency phase)

Translating to technical metrics

Each business metric should map (not 1:1, but traceably) to technical signals. If "time-to-value" matters, you must instrument latency of first key workflow and session error rates. This translation is the handshake between product and engineering.

Technical meaning

On the technical side, metrics become the nervous system of operating the service. Their meaning comes from how precisely they reflect user-visible behavior and how actionable they are for engineers.

Observability pillars vs. service metrics

Observability often cites three pillars: metrics (numeric aggregations), logs (discrete events with context), traces (distributed request flows). Service metrics sit at the top as interpreted numbers distilled from raw telemetry. You rarely alert on raw logs; you derive counters, rates, percentiles. See OpenTelemetry for standardized telemetry and this discussion on Monitoring vs. Observability.

Golden signals

Borrowing from SRE practice, the four golden signals of a user-facing system:

Latency – How long it takes to serve a request (track both success and error paths, p50/p95/p99).
Traffic – Demand size: requests/second, concurrent sessions.
Errors – Failure rate: explicit errors, timeouts, correctness failures.
Saturation – Resource exhaustion proximity: CPU, memory, queue length. Add a fifth in many modern systems: Cost – Unit economics per request/job.

SLIs, SLOs, SLAs

SLI (Service Level Indicator): Precisely defined measurement of user experience (e.g., "fraction of read API requests completed under 300 ms and returning 2xx").
SLO (Service Level Objective): Target for SLI over a window ("99.9% weekly").
SLA (Service Level Agreement): Contractual externally visible commitment; breaching may have penalties. Always set SLO tighter than SLA.

Error Budget = 1 - SLO. It is the allowed unreliability used for change velocity (deploys, experiments). If you burn budget too fast: slow releases, add reliability work. If you never spend budget: you may be over-investing. For alerting strategy, consider multi-window, multi-burn rate SLO alerts.

Metric types (Prometheus style)

Counter: Monotonic increase (e.g., total requests). Alert on rate not raw value. (Prometheus counter)
Gauge: Arbitrary up/down (e.g., memory usage, queue depth). (Prometheus gauge)
Histogram: Buckets of observations (latency). Enables percentiles & tail analysis. (Prometheus histogram)
Summary: Client-side calculated quantiles; use sparingly due to aggregation limits. (Prometheus summary) Prefer histograms for latency & size; counters for events; gauges for states. Ensure unit consistency (seconds, bytes). Document each metric: name, type, unit, cardinality dimensions. Good primers: Prometheus histograms and summaries and Gil Tene on latency percentiles.

Cardinality discipline

High-cardinality labels (user_id, session_id) explode storage and slow queries. Guidelines:

Reserve high cardinality for traces/logs, not primary metrics. (Prometheus best practices)
Dimension by stable groupings (region, API endpoint, plan tier). (Grafana labels guide)
Keep total series per metric under sane thresholds (e.g., < 10k) unless justified. (Cardinality explained)

Instrumentation checklist

Define SLIs first (user perspective) then choose raw signals.
Standard naming: service_namespace_subsystem_metric_unit (e.g., checkout_api_request_latency_seconds). See Prometheus naming best practices.
Include outcome labels: status="success|error|timeout".
Separate pathologically slow from normal via buckets (e.g., 50,100,200,300,500,800,1200,2000 ms).
Emit from a well-tested middleware layer to ensure coverage.

Aggregation & rollups

Store both raw series and periodic rollups (1m, 5m, 1h) to enable long-range trends affordably. Tail metrics (p99) need raw-ish resolution; cost/traffic can tolerate coarser granularity. See Recording rules and continuous aggregates.

Dashboards vs. alerts

Dashboards are for exploration & storytelling; alerts for actionable interruption. An alert should meet: clear owner, severity classification, runbook link, deduplication logic, and auto-silence conditions (maintenance windows, downstream known incidents). Too many unactionable alerts create alert fatigue; measure mean time to acknowledge (MTTA) and percent of alerts yielding tickets. See PagerDuty on alert fatigue.

Dependency metrics

Track upstream SLIs you depend on (e.g., database p99 latency). Decompose incidents faster by correlating service SLI dip with dependency saturation metrics.

Continuous improvement loop

Observe SLI trends and error budget consumption.
Perform weekly reliability review: anomalies, top regression sources.
Propose experiments (cache change, index) with expected movement.
Deploy and compare before/after via change-focused dashboards.
Feed learnings into next quarter's reliability & efficiency OKRs.

Bridging business and technical metrics

The most powerful metrics tell a dual story: "p95 checkout latency improved 20%, and conversion rose 3%". Maintain a mapping document linking each business KPI to:

Primary SLI(s)
Key technical driver metrics (cache hit rate, DB lock wait)
Leading indicator hypotheses (client render time) Revisit mapping quarterly; prune metrics that no longer explain variance.

Metric taxonomy cheat sheet

Layer	Example	Owner	Cadence
North Star	Weekly active collaborative docs	Product leadership	Weekly
SLI	Successful doc save latency p95 < 400ms	SRE/Engineering	Continuous
SLO	99.95% saves < 400ms weekly	SRE	Weekly review
Guardrail	Infra cost per save < $0.001	Finance/Eng	Monthly
Diagnostic	DB write lock wait time	Eng team	Ad hoc

Common anti-patterns

Alerting on averages (hide tail pain) instead of percentiles. See ACM Queue on percentiles.
Chasing "100%" reliability (diminishing returns) vs. defined SLO + error budget. See Error Budgets.
Overloading a single metric with too many labels causing cardinality blow-up. See Prometheus instrumentation pitfalls.
Building dashboards no one uses: track dashboard views, retire stale boards. See Grafana dashboard tips.
Confusing throughput with performance (more requests could mean retries from errors). Contrast RED method vs. USE method.

Practical example (API service)

Scenario: Public REST API for document editing.
Defined SLIs:

Read latency: fraction of GET /doc/{id} served < 250ms.
Write success: fraction of PUT /doc/{id} returning 2xx.
Editing session stability: sessions without disconnect > 5 minutes. SLOs: 99.9%, 99.95%, 99% respectively (weekly). Error budget alarms at 50%, 75%, 100% consumption. Business KPI mapped: active editing minutes per user. Hypothesis: Improving read latency p95 will raise active minutes by reducing initial load friction. Run experiment: introduce edge caching; monitor cache hit rate (target > 80%), origin latency drop (expect -30%). Outcome metrics decide rollout.

Getting started checklist

List top 3 user journeys; define one SLI each.
Set initial SLOs using historical 4-week median performance minus a modest stretch.
Instrument counters for requests, errors; histograms for latency.
Create one "golden dashboard" with SLIs + dependency saturation metrics.
Define 3 alerts only: SLI burn rate high (short & long window), dependency slowdown, error spike.
Write runbooks before enabling alerts.
Review after 2 weeks: adjust buckets, remove noisy labels, refine SLO.

Conclusion

Metrics are a language; alignment happens when business and engineering speak a shared dialect rooted in user experience. Start small, be explicit, iterate continuously. The goal is not more graphs; it's faster, better decisions.