Forem: Ruslan Manov

The Amnesiac That Learned to Remember — Building a Brain for Claude Code

Ruslan Manov — Mon, 06 Apr 2026 17:03:20 +0000

What happens when you give an AI coding assistant persistent memory across sessions, machines, and projects? sqlite-memory-mcp turns Claude Code from a brilliant amnesiac into something that remembers
**

It was 2:47 AM when Claude said something it shouldn't have known.

"The bridge sync conflict you hit last Tuesday — the one where two sessions overwrote the same entity — I fixed the root cause in db_utils.py. Want me to show you the diff?"

I hadn't mentioned that bug. Not in this session. Not in this conversation. The last time I'd touched that code was six days ago, in a different terminal, on a different machine.

Claude remembered.

Not because it has long-term memory by default. It doesn't. Every Claude Code session starts blank — a brilliant amnesiac. Ask it about yesterday's work and it gives you a polite, empty stare.

Unless you give it a brain.

sqlite-memory-mcp started because I broke something.

Three Claude Code sessions running in parallel. All writing to the same JSONL memory file. The file corrupted silently — half-written JSON lines, truncated observations, entities that existed in one session and vanished in another.

JSONL doesn't do concurrent writes. It doesn't do transactions. It doesn't do recovery. It's a format designed for append-only logs, pressed into service as a database by developers who needed something simple.

I needed something real.

v0.1.0 was twelve MCP tools and a SQLite database with WAL mode. Write-ahead logging meant multiple sessions could read and write simultaneously without corruption. FTS5 gave full-text search with BM25 ranking. The foundation was boring on purpose — SQLite has been running in production on every smartphone on Earth for two decades. It doesn't break.

That was supposed to be it. A fix. Ship it, move on.

It wasn't.

The first thing that happened was sessions.

Claude doesn't know it's Claude. It doesn't know this is session #47 on project "trading-bot" and that session #46 ended with a failing test in portfolio_manager.py. Every session is a fresh start, a new mind, a newborn with a PhD.

Session recall changed that. sqlite-memory-mcp now tracks which session created which entities, what tools were used, what the conversation context looked like. When Claude starts a new session, it can query: "What was I working on last time in this project?"

The answer comes back in milliseconds. Full context. The amnesiac remembers.

Then came tasks.

Not tasks for humans — tasks for Claude. A structured task system where one session can leave work for the next. "The FTS5 injection fix is half-done. The sanitization function works but the tests aren't written yet. Priority: high."

Next session picks it up. No human has to re-explain. No context is lost. The AI hands off to its future self like a relay runner passing a baton — except the runner dissolves after every lap and a new one materializes at the starting line.

Task tray UI in PyQt6 sits on your desktop. Kanban board renders as HTML. You can see what Claude is thinking about, what it left unfinished, what it flagged as blocked.

Bridge sync was the inflection point.

Two machines. Home desktop running Fedora, laptop on the train. Same memory, synchronized through a git repository. Entity changes push to the bridge, pull on the other side. Lamport clocks for causal ordering. Machine IDs for conflict detection.

Claude on the laptop continues where Claude on the desktop stopped. Same memory. Same task queue. Same knowledge graph. Different hardware, different continent, same mind.

The developer on the train opens Claude Code and says: "What did I do this morning?"

Claude answers. Accurately. With file paths, function names, and the exact commit hash where the work stopped.

At v3.4.0, the numbers look like this:

54 MCP tools across 7 focused servers. SQLite WAL for concurrency. FTS5 BM25 search with optional semantic fusion through sqlite-vec. Session tracking, structured tasks, bridge sync, collaboration workflows, entity linking, intelligence layer, causal event ledger with Lamport clocks.

One local SQLite file. No cloud service. No API key. No monthly bill. No data leaving your machine unless you explicitly push to the bridge.

The design constraint never changed: local-first, private by default.

3 AM. Claude finishes the refactor I asked for. It creates a task for tomorrow: "Run the full test suite after the schema migration. Check bridge compatibility."

I close the terminal. The session dies. Claude's mind evaporates.

But the memory persists. In a WAL-mode SQLite database on my local disk. Indexed. Searchable. Synchronized. Waiting for the next session to wake up, query the graph, and pick up exactly where the last one left off.

The amnesiac doesn't forget anymore.

Repo: github.com/RMANOV/sqlite-memory-mcp

How a SQLite WAL Fix Grew into a 54-Tool MCP Memory Stack

Ruslan Manov — Mon, 06 Apr 2026 16:56:22 +0000

How a SQLite WAL Fix Grew into a 54-Tool MCP Memory Stack

sqlite-memory-mcp started as a safer replacement for JSONL-based MCP memory. At v3.4.0 it is a 54-tool SQLite stack with tasks, bridge sync, collaboration, public-knowledge workflows, and optional hybrid search

How a SQLite WAL Fix Grew into a 54-Tool MCP Memory Stack

sqlite-memory-mcp started with a narrow goal: stop local memory corruption when
multiple Claude Code sessions touch the same store.

The official memory-server pattern is simple, but a flat file becomes fragile as
soon as more than one process writes to it. I wanted the same local-first feel,
but with transactional storage, better search, and room to grow.

SQLite was the obvious starting point.

By v3.4.0, that starting point has grown into a broader stack:

54 MCP tools
7 focused servers plus an optional unified server
SQLite WAL for concurrent local access
FTS5 BM25 search, with optional semantic fusion through sqlite-vec
session recall and project search
structured task management
git-based bridge sync
collaborator and public-knowledge workflows
entity linking and context/intelligence tools
an optional PyQt6 task tray app

Why SQLite was the right base layer

For this kind of workflow, SQLite buys a lot:

one local database file
no daemon to run
no cloud dependency
ACID transactions
WAL mode for concurrent readers and writers
FTS5 in standard SQLite

The point was never "SQLite beats every database".

The point was: for a local MCP memory stack that lives next to Claude Code,
SQLite gives you reliability, search, and portability without introducing more
infrastructure than the problem needs.

The release progression in plain English

Here is the shortest accurate summary of how the project evolved.

v0.1.0: replace JSONL with SQLite WAL

The first release shipped 12 tools in one server:

the 9 core memory tools from the official MCP server
session_save
session_recall
search_by_project

This made the project useful immediately: same core memory workflow, but backed
by SQLite with WAL and FTS5.

v0.2.0: move memory between machines with git

The next release added bridge sync:

bridge_push
bridge_pull
bridge_status

That was the first step from "single-machine memory" toward "local-first memory
that can travel".

v0.3.0 and v0.4.0: tasks and desktop workflow

v0.3.0 added task management and HTML kanban reporting.

v0.4.0 added the PyQt6 task tray app and utility scripts around the same SQLite
database.

At that point the project was no longer just a memory backend. It became a
practical daily workflow tool.

v0.6.0 through v0.9.0: collaboration and public knowledge

The next wave added:

collaborator management
queued knowledge sharing
review flows for imported knowledge
public-knowledge publishing requests
ratings and verification metadata

One detail worth stating carefully: these workflows are review-oriented. The
useful part is not "viral sharing". The useful part is that shared knowledge can
be staged, inspected, and accepted deliberately.

v3.0.0: intelligence-layer expansion

v3.0.0 was the large historical expansion point.

It introduced the context/intelligence layer on top of the existing memory,
tasks, bridge, and collaboration features:

task/entity linking
context assessment and resume flows
candidate-claim extraction and promotion
context-pack building
impact explanation

Important footnote: v3.0.0 shipped 49 tools in one monolithic server. The later
54-tool split-server layout came after that.

v3.1.x to v3.4.0: split architecture, hybrid search, and hardening

The current line added or stabilized several things:

split into focused MCP servers to make tool exposure more manageable
optional unified server for people who want one process
optional hybrid search with sqlite-vec
recurring task support and more task/context integration
security and hardening fixes across bridge, collaboration, schema, and search

What the current architecture actually looks like

At v3.4.0 the project exposes 50 tools across these servers:

sqlite_memory — core 9 memory tools
sqlite_tasks — task CRUD and task workflow tools
sqlite_session — session recall and context-health tools
sqlite_bridge — bridge sync and shared-task flows
sqlite_collab — collaborator and public-knowledge tools
sqlite_entity — task/entity linking and entity-maintenance helpers
sqlite_intel — context and intelligence tools
sqlite_unified — optional all-in-one server

That split matters because the project outgrew the original one-file server.

What changed in the latest hardening cycle

The recent v3.3.x line is not about flashy new marketing bullets. It is about
making the stack safer and more predictable.

Those tags include fixes for:

an FTS5 injection issue
path traversal risks in bridge/runtime paths
collaborator trust-boundary hardening
additional schema indexes
read_graph performance issues
bridge logging and TaskDB SQL cleanup

That is the right kind of work for a project in this stage.

What I think is actually interesting here

The interesting part is not the raw tool count.

The interesting part is that a local-first SQLite database can sit underneath a
surprisingly broad MCP workflow without giving up the properties that made it
useful in the first place:

easy backup
easy inspection
no service orchestration
no mandatory cloud hop
direct ownership of the data

The project is bigger now, but the center of gravity is the same:

a local SQLite file that Claude Code can use safely across repeated sessions.

If you want to try it

Current repo: https://github.com/RMANOV/sqlite-memory-mcp

Latest stable tag in the repo right now: v3.4.0

If you only want drop-in memory compatibility, start with the core server.

If you want the full stack, add the companion servers or use the unified server.

That was the original goal and it is still the point of the project: keep memory
local, durable, searchable, and useful enough to support real daily work.

The Day the Swarm Got Scared -- And Saved Everyone

Ruslan Manov — Mon, 06 Apr 2026 16:43:15 +0000

The Day the Swarm Got Scared -- And Saved Everyone

T+0.000s: The Valley

Two hundred drones crossed the ridgeline at 0347 local time, flying a Vee formation at fifteen-meter spacing. They had no GPS. They had not had GPS for eleven minutes, ever since the electronic warfare blanket rolled across the valley like an invisible fog. The SAM corridor below them was a dark geometry of overlapping kill envelopes, and every drone in the swarm knew this because every drone in the swarm had been talking to every other drone, constantly, through a protocol borrowed from epidemiology.

The swarm was not afraid. Not yet.

Fear, in the STRIX system, is not a metaphor. It is a 64-bit floating-point number between zero and one, computed forty times per second by a subsystem adapted from behavioral economics. At T+0, the fleet-wide fear parameter sat at F=0.08. Background noise. The algorithmic equivalent of a steady hand.

What happened next pushed it to 0.73.

The Particle Filter: Navigating Blind

Eleven minutes without GPS is a long time for an inertial measurement unit. IMUs drift. Accelerometers accumulate bias. Gyroscopes precess. Without correction, a drone flying on dead reckoning will be hundreds of meters off-position within minutes.

STRIX does not use dead reckoning. It uses a dual particle filter -- 200 particles per drone, each particle a hypothesis about where the drone actually is in six-dimensional space: [x, y, z, vx, vy, vz].

// strix-core/src/particle_nav.rs
// Each particle is a 6D state hypothesis weighted by likelihood.
// When GPS is denied, the filter relies on IMU prediction alone,
// but cross-validates against gossip-relayed neighbor positions.

pub struct ParticleNavFilter {
    pub particles: Vec<[f64; 6]>,
    pub weights: Vec<f64>,
    pub n_particles: usize,
    // ...
}

Here is the critical insight: even without GPS, the drones are not navigating alone. Each drone broadcasts its best position estimate through the gossip protocol. When Drone 47 hears from Drone 48 that it is approximately 15 meters to its left, and Drone 47's particle filter has a cluster of hypotheses that agree with this, those particles gain weight. The particles that disagree quietly die.

The swarm navigates by consensus. Two hundred particle filters, each with 200 particles, form a distributed estimation engine of 40,000 simultaneous hypotheses about the state of the world. GPS denial does not blind this system. It degrades it. There is a difference.

When the EW engine detects GPS denial, it triggers an automated response:

// strix-core/src/ew_response.rs
// GPS denial triggers noise expansion in the particle filter,
// widening the hypothesis cloud to account for increased uncertainty.

pub enum EwResponse {
    ExpandNavigationNoise { noise_multiplier: f64 },
    GossipFallback { reduced_fanout: usize, priority_only: bool },
    ForceRegime(Regime),
    // ...
}

The process noise multiplier expands. The particle cloud widens. Uncertainty increases, but honestly -- the system knows what it does not know, and acts accordingly.

T+12.400s: First Blood

Drone 7 ceased transmitting at T+12.4 seconds.

There was no warning. No gradual degradation of telemetry. One tick it was there, broadcasting its state through the gossip protocol at three-peer fanout. The next tick it was not. The heartbeat counter incremented past the timeout threshold, and the swarm's loss analyzer activated.

// strix-auction/src/antifragile.rs
// The loss analyzer classifies the kill and creates an exclusion zone.
// This is where the swarm starts learning.

pub fn record_loss(&mut self, record: LossRecord) -> Vec<u32> {
    let orphans = record.orphaned_tasks.clone();
    self.loss_records.push_back(record.clone());
    self.adapt_from_loss(&record);
    orphans
}

Three things happened simultaneously within 2 milliseconds of detecting the loss:

First, the loss was classified. Drone 7 was in ENGAGE regime at 500 meters altitude with a known threat bearing. Classification: SAM. Kill zone radius: 2,000 meters. Penalty weight: 0.8.

pub fn classify_loss(
    regime: Regime,
    threat_bearing: Option<f64>,
    altitude: f64,
) -> LossClassification {
    match (regime, threat_bearing) {
        (Regime::Engage, Some(_)) if altitude > 200.0 => LossClassification::Sam,
        (Regime::Engage, Some(_)) => LossClassification::SmallArms,
        (Regime::Patrol, None) => LossClassification::Collision,
        (_, Some(_)) => LossClassification::ElectronicWarfare,
        _ => LossClassification::Unknown,
    }
}

Second, Drone 7's orphaned tasks were identified and flagged for immediate re-auction. The auctioneer's needs_reauction flag flipped to true.

Third, and this is the part that matters: a kill zone materialized in the swarm's shared spatial memory. Not a GPS coordinate. Not a waypoint. A pheromone. A digital scent of death, deposited at Drone 7's last known position, repelling every drone that came near.

// strix-mesh/src/stigmergy.rs
// Threat pheromone: "Danger here" -- repels drones from hazardous areas.

pub enum PheromoneType {
    Explored,  // "I've been here"
    Threat,    // "Danger here"
    Target,    // "Interesting target"
    Rally,     // "Regroup here"
    Corridor,  // "Safe path"
}

The pheromone field is a sparse 3D grid with 10-meter cells. Each deposit is about 20 bytes. The gradient computation that steers drones away from danger is O(1) per cell -- a central-difference calculation across neighboring cells that returns a three-component vector pointing away from concentration.

The swarm did not need to be told to avoid the area where Drone 7 died. It could smell the danger.

T+12.406s: The Market Reacts

Six milliseconds after the loss, the combinatorial auction repriced everything.

The STRIX auction is a sealed-bid market. Every drone evaluates every available task independently and submits a composite score based on proximity, capability match, energy reserves, urgency, and risk exposure. The auctioneer collects all bids and solves the assignment problem using a modified Hungarian algorithm.

// strix-auction/src/bidder.rs
// Bid scoring function. Note the risk term: kill-zone proximity
// and fear level directly suppress bids on dangerous tasks.
//
// total = urgency*10 + capability*3 + proximity*5 + energy*2 - risk*4

When Drone 7 died, two things changed in the market. First, its tasks became orphans -- supply dropped. Second, the kill zone inflated the risk term for every task near grid 7-Alpha -- demand cratered. The market did not need a commander to say "avoid that area." The prices said it. No drone bid competitively on tasks inside the kill zone because the math would not let them.

The fear parameter rose from 0.08 to 0.31. This was not panic. This was information. The SwarmFearAdapter translated the loss event into the language of behavioral economics:

// strix-swarm/src/fear_adapter.rs
// STRIX telemetry mapped to PhiSim's behavioral economics model:
//
// | PhiSim concept       | STRIX signal                        |
// |----------------------|-------------------------------------|
// | drawdown             | Attrition rate (1 - alive/initial)  |
// | vol_ratio            | Threat intensity (1 + intent score) |
// | anomaly_count        | CUSUM breaks this tick              |
// | consecutive_losses   | Consecutive ticks with drone loss   |

At F=0.31, the formation spacing widened. The FormationConfig applies fear-modulated spacing: as fear rises, drones spread apart. Wider formation, harder to hit with a single salvo. Less aerodynamic efficiency, but the auction already repriced for that -- the scoring function factors in the additional transit cost.

T+23.800s: The Feint

At T+23.8, the adversarial particle filter detected something interesting.

The second particle filter -- the one that does not track friendly drones but enemy threats -- had been watching a cluster of radar returns moving south along the valley floor. The threat tracker maintained its own 100-particle hypothesis cloud per target, and the intent detection pipeline had been analyzing the movement pattern through three layers of signal processing.

// strix-core/src/intent.rs
// 3-layer pipeline: Hurst persistence -> closing acceleration -> vol compression
//
// Layer 1: Hurst persistence     -> purposeful trajectory? [H > 0.55]
// Layer 2: Closing acceleration  -> accelerating toward us?
// Layer 3: Volatility compression -> formation tightening?
//              |
//   Confidence-weighted fusion
//              |
//   IntentScore in [-1, 1] + IntentClass

The Hurst exponent for the southern cluster was 0.42. Below the purposeful threshold of 0.55. The movement was mean-reverting -- zigzagging, not advancing. The closing acceleration was near zero. The volatility ratio was high: 1.8, indicating loose, disorganized movement.

The intent pipeline classified this as IntentClass::Neutral, bordering on Retreating.

But here is where it gets subtle. The CUSUM anomaly detector noticed something the intent pipeline alone would miss: the southern cluster's radar cross-section kept changing. Large, then small, then large. Inconsistent with real aircraft. Consistent with decoys -- inflatable or electronic emitters designed to draw attention and waste resources.

// strix-core/src/anomaly.rs -- CUSUM detects distributional shifts.
// When the signature variance of a target group breaks the cusum threshold,
// the system flags potential deception.

The adversarial particle filter's weight distribution was bimodal: half the particles clustered on "real threat, low intent" and half on "decoy, ignore." The Hurst persistence analysis tipped the balance. Real threat formations show persistent trajectories (H > 0.55). Decoys wander. H=0.42 was the signature of something pretending to be threatening but failing at the physics of it.

The fear parameter ticked up to 0.38 on the initial detection, then back down to 0.29 as the system accumulated evidence of deception. This is the dual-process architecture at work -- fear rises fast (System 1), but the analytical pipeline (System 2) can override it with evidence. The swarm did not freeze. It did not divert resources to chase phantoms. It maintained course.

The XAI narrator logged the reasoning:

[t=24.1s] Threat response (prob=31%): Maintaining course — southern cluster
classified as FEINT. Confidence: 78%.

Every alternative was recorded. "Divert 30 drones south" scored 0.34, rejected for "Insufficient threat probability, Hurst below purposeful threshold." The glass box held.

T+47.200s: The Cascade

This is where the story could have ended badly.

At T+47.2, the EW blanket intensified. The comms jamming layer that had been degrading mesh connectivity surged to SEVERE. Sixty drones lost their gossip links simultaneously. Not destroyed -- silenced. Their particle filters kept running, their IMUs kept integrating, but they could not hear the swarm and the swarm could not hear them.

Then the SAM corridor opened up.

In thirty seconds, between T+47 and T+77, the swarm lost sixty drones. Not lost-connection. Lost. Destroyed. The loss analyzer fired sixty times in thirty seconds. Sixty kill zones materialized across the valley floor. Sixty sets of orphaned tasks flooded the auction queue.

The fear parameter did this:

T+47.0: F = 0.29
T+50.0: F = 0.51
T+55.0: F = 0.62
T+60.0: F = 0.68
T+65.0: F = 0.71
T+70.0: F = 0.73

F=0.73. The swarm was terrified.

What does terror look like in a combinatorial auction? It looks like this:

// strix-auction/src/antifragile.rs
// Fear-amplified kill zone penalties. At F=0.73, the multiplier is 2.095.
// SAM kill zones with base penalty 0.8 become 1.676 -- effectively
// making it economically impossible to bid on tasks inside them.

pub fn kill_zone_penalties_with_fear(&self, fear: f64) -> Vec<(Position, f64, f64)> {
    let f = fear.clamp(0.0, 1.0);
    let multiplier = 1.0 + f * 1.5; // 1.0 -> 2.5
    self.kill_zones
        .iter()
        .map(|kz| (kz.center, kz.radius, kz.penalty * multiplier))
        .collect()
}

At F=0.73, the fear multiplier hit 2.095. Every SAM kill zone's penalty weight of 0.8 became 1.676. The auction's risk term (-risk*4) for tasks inside those zones was so massive that no bid could overcome it. The market priced those areas at infinity. No drone went there. No commander needed to draw a red line on a map. The red line drew itself, from the blood of the fallen.

But here is where anti-fragility kicked in. The kill zones did not just warn. They taught.

// Each additional loss in the same zone GROWS the radius.
// After 4 merges with growth factor 1.3: base * 1.3^4 = base * 2.86
// The system overestimates danger on purpose. Better to avoid
// too much than too little.

fn merge_into_existing_zone(&mut self, record: &LossRecord) -> bool {
    for kz in &mut self.kill_zones {
        let dist = kz.center.distance_to(&record.position);
        if dist < self.merge_distance {
            kz.loss_count += 1;
            kz.radius *= self.zone_growth_factor;
            kz.penalty = (kz.penalty * 1.1).min(1.0);
            // Shift centre towards the new loss (weighted average).
            let w = 1.0 / kz.loss_count as f64;
            kz.center = Position::new(
                kz.center.x * (1.0 - w) + record.position.x * w,
                kz.center.y * (1.0 - w) + record.position.y * w,
                kz.center.z * (1.0 - w) + record.position.z * w,
            );
            return true;
        }
    }
    false
}

Three losses near the same coordinates? The kill zone radius expanded by a factor of 1.3 per loss. The penalty weight climbed toward 1.0. The evade bias at that position -- the probability of entering EVADE regime when nearby -- stacked additively. The swarm was not just avoiding the danger. It was building an increasingly accurate map of it, and the more it suffered, the better the map became.

The antifragile score -- sum over kill zones of (loss_count * ln(1 + loss_count) * radius_growth) -- climbed past 50.0. By Taleb's measure, the system was more robust after losing 60 drones than it had been with 200.

T+78.000s: The Reformation

One hundred and forty drones remained. They were scattered, terrified (F=0.73), and navigating on inertial alone in a GPS-denied environment thick with SAM coverage and comms jamming.

They reformed in four seconds.

The gossip protocol is designed for exactly this scenario. Each surviving drone selected three random peers from its known-alive list and exchanged state digests. If the digests differed -- and they all differed, because sixty drones had just vanished -- full state exchanges followed. Within two gossip rounds, the surviving 140 drones had converged on a shared picture of who was left and where everyone was.

// strix-mesh/src/gossip.rs
// O(log N) convergence via epidemic gossip.
// Two rounds to synchronize 140 nodes after catastrophic loss.

// Conflict resolution:
// - General data: newer timestamp wins.
// - Threat data: union -- never discard threat information.

The formation engine computed new slot positions for 140 drones in Vee formation. The correction velocity vectors pointed each drone toward its new slot using proportional control with speed clamping:

// strix-core/src/formation.rs
// v_corr = (delta / ||delta||) * min(||delta||, v_max)
// If within deadband: v_corr = 0.

And here is where the CBF -- the Control Barrier Function -- earned its keep. One hundred and forty drones, all simultaneously repositioning in three dimensions, in comms-degraded conditions. The potential for mid-air collision was enormous.

// strix-core/src/cbf.rs
// CBF safety clamp: TTC-aware collision avoidance.
// Runs AFTER formation control, BEFORE velocity commands are sent.
// Every velocity vector that would violate the safety barrier gets
// rotated and scaled to the nearest safe vector.

pub struct CbfConfig {
    pub min_separation: f64,     // meters
    pub altitude_floor_ned: f64, // NED convention
    pub altitude_ceiling_ned: f64,
    pub alpha: f64,              // decay rate -- aggressiveness
    pub max_correction: f64,     // m/s cap
}

The CBF is a mathematical guarantee. Not a best-effort collision avoidance. Not a "try to maintain separation." A hard constraint that modifies every velocity command to ensure that the barrier function -- a measure of how close two drones are to colliding -- never decreases below zero. If two drones are on a collision course, the CBF does not ask. It corrects. And it does so with the minimum modification necessary to the desired velocity, preserving mission intent to the maximum extent physics allows.

Zero collisions during the reformation. At 1.15ms per tick for 20 drones, and scaling to the full 140, the system ran the entire CBF pass in under 10ms. Tight enough that the correction commands arrived before the drones had moved appreciably toward each other.

T+82.000s: The Market Finds Equilibrium

The auction re-ran at T+82.0. All surviving drones submitted sealed bids on all remaining tasks, with kill-zone penalties applied, fear-modulated risk terms included, and the intent pipeline's assessment of remaining threats factored into urgency multipliers.

The market cleared in 4.86ms.

Auction result:
  - 137 tasks assigned (of 142 remaining)
  - 5 tasks unassigned (inside active kill zones, no viable bid)
  - Total welfare: 847.3 (down from 1,204.1 pre-attrition)
  - Antifragile score: 58.4

The five unassigned tasks were inside the densest kill zones. The market's judgment: no drone should go there. The risk-adjusted cost exceeded the task value. This was not cowardice. This was the auction computing, in 4.86 milliseconds, a truth that would take a human commander minutes to reach: those tasks were not worth another drone.

The fear parameter began to decay. No new losses. The gossip protocol confirmed all 140 surviving drones were in formation and executing their assigned tasks. The CUSUM detectors settled. The Hurst exponent of the fleet's own movement pattern climbed back above 0.6 -- purposeful, directed.

T+82.0: F = 0.71
T+90.0: F = 0.64
T+100.0: F = 0.55
T+120.0: F = 0.42

The swarm was calming down. Not because someone told it to. Because the math said the danger was receding.

T+127.000s: The Glass Box

The XAI narrator had been recording every decision the entire time. Not summarizing. Not approximating. Every single decision trace, with full reasoning chains, alternatives considered, confidence levels, and input states.

This is the glass box.

// strix-xai/src/trace.rs
// Every decision emits a DecisionTrace with:
// - Timestamp
// - Decision type (TaskAssignment, RegimeChange, FormationChange,
//                  ThreatResponse, ReAuction, LeaderElection)
// - Full inputs (drone IDs, regime, metrics, fear/courage/tension)
// - Reasoning chain (numbered steps with data)
// - All alternatives considered (with scores and rejection reasons)
// - Output action + confidence score

pub struct DecisionTrace {
    pub id: u64,
    pub timestamp: f64,
    pub decision_type: DecisionType,
    pub inputs: TraceInputs,
    pub reasoning: Vec<ReasoningStep>,
    pub alternatives_considered: Vec<Alternative>,
    pub output: TraceOutput,
    pub confidence: f64,
}

At the command center, a human operator -- the one who had been watching the entire engagement unfold -- requested the after-action review. The mission replay system aggregated 4,847 decision traces into a structured timeline.

// strix-xai/src/replay.rs
// MissionReplay aggregates all traces into a timeline with
// statistics, key moments, and what-if analysis capability.

The narrator produced the report at DetailLevel::Detailed. Every decision, every alternative, every rejection reason. But between the lines of structured data, a story emerged.

Not because anyone programmed it to tell stories. Because when you trace the complete decision history of a system that learned from sixty deaths, the trace reads like one.

The After-Action Report

=== STRIX Mission Replay: Operation Ridgeline ===
Duration: 127.0s | Drones: 200 initial, 140 surviving | Traces: 4,847

KEY MOMENTS:

[t=12.4s] Unit 7 ceased. Classification: SAM. Kill zone established at
(3847.2, 1204.5, 502.1), radius 2000m, penalty 0.80.
The market remembered.

[t=12.4s] Re-auction triggered. 3 orphaned tasks redistributed among 199
remaining drones in 4.2ms. No bid entered for grid 7-Alpha.
At t=12.4s, no drone bid on grid 7-Alpha again.

[t=24.1s] Southern cluster assessed as FEINT.
Hurst=0.42 (below purposeful threshold 0.55).
Closing acceleration: -0.12 m/s^2 (below attack threshold 0.50).
Volatility ratio: 1.80 (expanding, not compressing).
Decision: Maintain course. Confidence: 78%.
  Alternative: Divert 30 drones south (score=0.34) -- rejected:
  "Insufficient threat probability, Hurst below purposeful threshold."
The swarm chose not to chase ghosts.

[t=47.2s-77.0s] CASCADE EVENT. 60 units lost in 30.0 seconds.
Fear: 0.29 -> 0.73.
Kill zones established: 60. Merged zones: 12.
Auction repriced: 60 re-auction cycles, mean latency 3.8ms.
Antifragile score: 12.1 -> 58.4.
The swarm suffered. The swarm learned.

[t=78.0s] Reformation complete. 140 drones, Vee formation.
Gossip convergence: 97.1% in 2 rounds (3.2 seconds).
CBF interventions: 23 (zero collisions).
The swarm reformed while scared, and nothing touched.

[t=82.0s] Market equilibrium. 137/142 tasks assigned.
5 tasks unpriced (inside kill zones, welfare < threshold).
The market found the boundary of acceptable risk.

[t=127.0s] Mission complete. Fear: 0.42 (decaying).
Final antifragile score: 62.7.

DETERMINISTIC REPLAY AVAILABLE: All 4,847 traces stored.
Full tick-by-tick replay at original timing enabled.

The operator stared at the screen for a long time after reading it.

At T+12.4s, Unit 7 ceased. The market remembered. At T+12.4s, no drone bid on grid 7-Alpha again.

It was not poetry. It was a database query formatted as text. But it read like an epitaph, because the math of loss and memory and avoidance, when you trace it honestly, has a cadence that sounds like grief.

The Algorithms Are Real

STRIX is an open-source Rust project. Apache 2.0. Every algorithm described in this story is implemented, tested, and benchmarked.

Numbers:

34,889 lines of Rust across 9 crates
7,493 lines of Python (PyO3 bindings + simulation)
671 tests
1.15ms per tick (20 drones)
4.86ms auction clear (100 drones)
Scaling target: 2,000+ drones

The nine crates:

strix-core: Dual particle filter, CUSUM anomaly detection, regime detection, formation control, CBF safety, EW response, threat intent pipeline, ROE engine
strix-auction: Combinatorial auction, sealed-bid market, anti-fragile kill zones, fear-modulated risk pricing
strix-mesh: Gossip protocol (O(log N) convergence), digital pheromone fields (stigmergy), fractal communication
strix-xai: Glass-box trace recording, natural-language narration, deterministic mission replay
strix-swarm: Integration orchestrator, tick loop, PhiSim fear adapter
strix-adapters: MAVLink, ROS2, simulator interfaces
strix-python: PyO3 bindings for the entire stack
strix-playground: Scenario engine, threat presets, benchmarking
strix-optimizer: SMCO parameter optimization, Pareto analysis

What makes it different:

Dual particle filter -- no competitor has both friendly navigation and adversarial intent prediction running simultaneously
Anti-fragile kill zones -- the swarm measurably improves after losses, inspired by Taleb's anti-fragility
Fear meta-parameter -- behavioral economics (Kahneman) modulates every subsystem through a single continuous signal
Combinatorial auction -- market-based task allocation with kill-zone repricing, not centralized planning
Digital pheromones + gossip -- fully decentralized, no single point of failure, bio-inspired coordination
Glass-box XAI -- every decision traced, narrated, replayable; zero black-box decisions
Deterministic replay -- entire missions can be replayed tick-by-tick for after-action review and what-if analysis

The fear is a math function. The courage is a counter-signal. The memory is pheromones. The market finds the optimal outcome.

And sometimes, when you read the trace of what the market decided, it sounds like something more.

GitHub: github.com/RMANOV/strix

Building a Drone Swarm Orchestrator That Gets Scared — 20 Subsystems in 35K Lines of Rust

Ruslan Manov — Mon, 06 Apr 2026 16:37:25 +0000

## Building a Drone Swarm Orchestrator That Gets Scared — 20 Subsystems in 35K Lines of Rust

Tags: rust, robotics, opensource, algorithms
Target: Dev.to

The Observation That Started Everything

A few years ago I was working on a particle filter for tracking hidden state in financial time series — the usual quantitative trading toolkit. Particles representing possible market regimes, a Bayesian update step when new price data arrives, a resampling step to prevent weight degeneracy. Standard stuff.

Then someone asked me to look at a drone swarm coordination problem. I expected something completely different. What I found instead was the same math, wearing different clothes.

Drones tracking uncertain positions in 3D space? That's a particle filter, same as tracking a hidden volatility regime. Allocating scarce drone resources to competing tasks? That's a combinatorial auction, same as portfolio optimization under constraints. Protecting the swarm against catastrophic attrition? That's drawdown protection, same as risk management in a leveraged portfolio. The math didn't change. Only the domain changed.

That observation became STRIX: a 34,889-line Rust + 7,493-line Python drone swarm orchestration library (~42,400 LOC total) that treats the battlefield as a market, implements swarm coordination from ant colony research, and uses a "fear meta-parameter" borrowed from behavioral economics to modulate every subsystem in real time. Designed to scale toward 2000+ drones.

This article is a deep dive into the architecture, the technical decisions, and what 20 subsystems across 9 crates taught me about building complex autonomous systems.

The Problem Space

Drone swarm coordination is hard in a specific way: the difficulty is not computational but architectural. You need to simultaneously solve:

State estimation under uncertainty — where are we, where are threats, what's the ground truth when GPS is jammed?
Task allocation under contention — which drone does which task when you have more tasks than drones and capabilities don't match uniformly?
Safety with formal guarantees — how do you ensure collision avoidance and no-fly zone compliance without a central controller that becomes a single point of failure?
Coordination without centralization — how does the swarm share state when you lose nodes, when comms are degraded, when the network topology changes every second?
Human-in-the-loop — how do you keep a human meaningfully in the decision loop when the swarm acts at millisecond timescales?
Explainability — if the swarm makes a decision you didn't expect, how do you reconstruct exactly why?

Most existing approaches handle one or two of these well. The rest are left as "future work" or "out of scope." STRIX tries to handle all six in a unified architecture.

Architecture: The 10-Step Tick Loop

The core of STRIX is a deterministic tick loop that runs every timestep. Each tick executes exactly 10 steps in order. Every subsystem runs on every tick. No exceptions, no optional steps.

┌─────────────────────────────────────────────────────────────────┐
│                    STRIX TICK LOOP (per drone)                   │
├─────────────────────────────────────────────────────────────────┤
│  Step 1:  EW Threat Scan                                         │
│           Classify: GpsJamming | CommJamming | Spoofing |        │
│                     DirectedEnergy | CyberIntrusion              │
│           Modulate noise params + gossip fanout via fear F∈[0,1] │
├─────────────────────────────────────────────────────────────────┤
│  Step 2:  Dual Particle Filter                                   │
│           Friendly: 200 particles, state [x,y,z,vx,vy,vz]       │
│           Threats:  100 particles, adversarial tracking          │
│           Predict + Measurement update + Resample                │
├─────────────────────────────────────────────────────────────────┤
│  Step 3:  CUSUM Anomaly Detection                               │
│           Per-drone sequential change detection                  │
│           Regime transitions: Patrol → Engage → Evade            │
│           3×3 Markov transition matrix                           │
├─────────────────────────────────────────────────────────────────┤
│  Step 4:  Formation Correction                                   │
│           7 formation types (Vee, Line, Wedge, Column,           │
│           EchelonLeft, EchelonRight, Spread)                     │
│           Proportional control law with deadband                 │
├─────────────────────────────────────────────────────────────────┤
│  Step 5:  Threat Tracker Update                                  │
│           Intent detection: motion pattern → behavior class      │
│           Hysteresis gate prevents classification oscillation    │
│           Adversarial doctrines: PROBING, FEINT, COORDINATED    │
├─────────────────────────────────────────────────────────────────┤
│  Step 6:  ROE Authorization Gate                                 │
│           WeaponsHold | WeaponsTight | WeaponsFree               │
│           Pipeline: classify → IFF confidence → collateral risk  │
│           CVaR risk scoring integration                          │
├─────────────────────────────────────────────────────────────────┤
│  Step 7:  Combinatorial Task Auction  [strix-auction]            │
│           Drones bid on tasks; winner-takes-assignment           │
│           Kill-zone repricing after losses                       │
│           Dark pool compartmentalization for classified tasks     │
├─────────────────────────────────────────────────────────────────┤
│  Step 8:  Gossip State Propagation  [strix-mesh]                 │
│           O(log N) convergence, priority-queued messages         │
│           Pheromone update: Danger/Explored/Rally/Resource       │
├─────────────────────────────────────────────────────────────────┤
│  Step 9:  CBF Safety Clamp  [strix-core]                        │
│           TTC-aware CBF with deadlock detection                  │
│           Collision avoidance + altitude bounds + NFZ exclusion  │
│           APF trajectory planning integration                    │
├─────────────────────────────────────────────────────────────────┤
│  Step 10: XAI Decision Trace  [strix-xai]                       │
│           Record every decision with full causal chain           │
│           Machine traces → human-readable narrative              │
│           Deterministic replay for after-action review           │
└─────────────────────────────────────────────────────────────────┘

The 9 crates correspond roughly to this structure:

strix-core: steps 1–6, 9 (15 modules, the heaviest crate)
strix-auction: step 7
strix-mesh: step 8
strix-xai: step 10
strix-swarm: swarm-level coordination across drones
strix-adapters: MAVLink/ROS2 hardware adapters
strix-python: PyO3 bindings
strix-playground: scenario DSL for testing
strix-optimizer: SMCO multi-objective parameter optimization with Pareto front, 62 tunable parameters

Deep Dive 1: The Battlefield is a Market

The auction system in strix-auction is the most intellectually loaded module in STRIX. The core insight: task allocation in a drone swarm is mathematically equivalent to portfolio optimization in a market with constraints.

In a financial portfolio:

You have scarce capital to allocate
You have a set of available assets with different risk/return profiles
Some assets have correlations you need to track
Drawdown protection prevents you from loading into catastrophic positions
Some trades are only visible to certain participants (dark pools)

In a drone swarm:

You have scarce drone-capacity to allocate
You have a set of tasks with different capability requirements
Some tasks must be done together (bundles)
Kill-zone repricing prevents you from sending more drones into a slaughter
Some tasks are classified and visible only to specific sub-swarms (compartmentalization)

The Task structure makes this mapping explicit:

pub struct Task {
    pub id: u32,
    pub location: Position,
    pub required_capabilities: Capabilities,  // sensor/weapon/EW/relay
    pub priority: f64,
    pub urgency: f64,
    pub bundle_id: Option<u32>,   // tasks that must go together
    pub dark_pool: Option<u32>,   // compartmentalized visibility
}

The auction mechanism is loosely inspired by VCG (Vickrey-Clarke-Groves, 1971) — the same mechanism that underlies modern digital advertising auctions — adapted for multi-unit combinatorial assignment with physical constraints. A drone's bid on a task is a function of its distance to the task location, its capability match score, its current load, and its fear state. High-fear drones bid more conservatively. They're risk-averse, like a trader protecting a drawdown.

Kill-zone repricing is the anti-fragility mechanism: after a drone is lost in a location, the perceived cost of that location increases for all subsequent bidders. The auction organically routes the swarm around high-attrition zones. The swarm doesn't need a central commander to say "stop flying over that hill" — the auction figures it out through price signals.

Dark pool compartmentalization solves a harder problem: in real operations, some tasks are classified above the clearance of certain drones. The auction system supports dark_pool visibility groups, where only drones within the same dark pool can see and bid on compartmentalized tasks. This is architecturally identical to how dark pools work in equity markets — non-public order flow visible only to approved participants.

Benchmark: 465 µs for a full auction cycle with 50 drones competing on 20 tasks. At scale: 4.86 ms for 100 drones competing on 50 tasks. Both run inside the tick loop.

Deep Dive 2: Fear as a Control Signal

The PhiSim integration is the part of STRIX that gets the strongest reactions from people who encounter it for the first time: "You put fear into a drone swarm? Why?"

The answer starts with Kahneman. Prospect theory (Kahneman & Tversky, 1979) shows that human decision-making under uncertainty is not utility-maximizing — it's loss-averse, context-sensitive, and heavily influenced by current emotional state. A trader who just suffered a significant drawdown behaves differently than a trader who is up on the month, even when facing mathematically identical choices. That's not irrational. It's adaptive.

The same logic applies to autonomous systems. A swarm that has lost 30% of its drones to jamming should not behave identically to a full-strength swarm approaching the same objective. It should be more cautious — wider formations, longer evade distances, stronger avoidance signals, more aggressive information sharing. Not because a human operator told it to be cautious, but because the system's own estimate of its situation warrants it.

Fear F ∈ [0, 1] in STRIX is computed from a dual adversarial process: fear and courage as opposing forces, with tension = |fear - courage|. The inputs map financial concepts to combat:

Financial concept	Swarm analog
Portfolio drawdown	Attrition rate (drones lost / total)
Volatility ratio	Threat intensity ratio
Anomaly count	CUSUM change-point breaks
Consecutive losses	Loss ticks (sustained attrition)

When fear rises, every subsystem responds:

// Evade distance: 150m at F=0 → 500m at F=1
evade_distance: base.evade_distance * (1.0 + f * 2.3)

// Formation spacing: +50% at maximum fear
spacing: base.spacing * (1.0 + axes.bias * 0.5)

// Pheromone persistence: 3.3x longer at F=1
modulated_decay = decay * axes.threshold  // threshold ≈ 0.3 at F=1

// Gossip fanout: up to 2x at F=1 (capped at 3x)
(base_fanout + floor(f * base_fanout)).min(base_fanout * 3)

The effect is that a high-fear swarm is simultaneously more cautious (wider formations, longer evade distances) and more communicative (higher gossip fanout, stronger pheromones). This matches what military doctrine recommends for degraded units: pull back, increase information sharing, wait for situation clarity before re-engaging.

Courage is the opposing force. A high-courage swarm can tolerate tighter formations, closer engagement distances, more aggressive auction bids. Tension (|fear - courage|) drives ROE posture suggestions — not automated ROE changes, but recommendations surfaced to human operators through the XAI layer.

Deep Dive 3: Bio-Inspired Coordination

The gossip protocol in strix-mesh and the pheromone system in strix-core are the two bio-inspired coordination mechanisms. They solve the same problem from different angles: how does a decentralized swarm maintain coherent collective behavior without a central coordinator?

Digital Pheromones are borrowed directly from Dorigo's ant colony optimization (1992). Ants leave chemical traces that guide other ants toward food sources and away from dead ends. STRIX implements four pheromone types:

Danger — repulsive, deposited near threats and loss events
Explored — marks already-covered terrain to avoid redundant paths
Rally — attractive, marks gathering points for regrouping
Resource — marks objectives and high-value areas

Each pheromone has exponential decay, but the decay rate is modulated by fear. At high fear, pheromones persist 3.3x longer — the swarm's collective memory of dangerous areas stays fresh longer when it's actively scared. At low fear, pheromones fade quickly, allowing the swarm to be bolder about re-exploring previously dangerous terrain.

Gossip Protocol implements epidemic information spreading. Each drone periodically selects a random set of neighbors and exchanges state updates. The mathematical guarantee: in a connected graph, gossip reaches all nodes in O(log N) rounds with high probability. STRIX implements bandwidth-aware priority queuing:

Priority	Message Type	Rationale
0 (highest)	ThreatAlert	Immediate danger, time-critical
1	TaskAssignment	Coordination, semi-urgent
2	StateUpdate	Position/status, routine
3	PheromoneDeposit	Environmental update, low urgency
4	Heartbeat	Keepalive, lowest priority

When fear is high and bandwidth is constrained, low-priority messages are dropped first. The swarm preferentially shares threat information when it most needs to. When fear drops and bandwidth recovers, state updates and pheromone deposits fill in the collective picture.

The combination of pheromones and gossip produces emergent behavior that no single subsystem explicitly implements: without a central coordinator, the swarm learns the shape of the threat environment, avoids areas that have been costly, concentrates toward objectives, and maintains information coherence across node losses.

Deep Dive 4: Safety and Trajectory Planning

TTC-aware CBF with deadlock detection extends the standard CBF formulation by incorporating Time-to-Collision estimates. Instead of only enforcing static separation distances, the CBF considers the closing velocity between drones — two drones approaching each other head-on at high speed trigger the safety clamp earlier than two drones drifting slowly toward each other. Deadlock detection identifies situations where CBF constraints from multiple drones create a gridlock and applies a resolution strategy.

APF trajectory planning (Artificial Potential Fields) provides smooth, obstacle-aware paths. Attractive potentials pull drones toward objectives; repulsive potentials push them away from obstacles, NFZs, and other drones. The APF output feeds into the CBF as a desired velocity, which the CBF then projects onto the safe set.

CVaR risk scoring (Conditional Value-at-Risk) quantifies the tail risk of mission plans. Instead of optimizing for expected outcomes, CVaR focuses on the worst-case percentile — what happens in the bottom 5% of scenarios? This integrates with the auction's bid evaluation and the ROE authorization pipeline.

NaN hardening ensures that numerical corruption cannot silently propagate through the tick pipeline. Every subsystem includes guards that detect NaN values in inputs and outputs, preventing a single sensor glitch from cascading into nonsensical decisions across the entire swarm.

Performance: What the Numbers Mean

Metric	Value	Context
Full swarm tick (20 drones)	1.15 ms	869 Hz max tick rate, well above any real-time control requirement
Combinatorial auction (50 drones, 20 tasks)	465 µs	Fits inside a single 1ms control loop
Combinatorial auction (100 drones, 50 tasks)	4.86 ms	Scales to larger swarms within real-time bounds
Particle filter (200 particles)	75 µs	Leaves 925 µs for everything else in a 1ms tick

The 1.15 ms full tick number is the most important one. Real-time control for drone swarms typically requires update rates of 10–100 Hz (10–100 ms per tick). At 1.15 ms, STRIX runs 8–87x faster than required, which means:

Margin for hardware latency — you can afford significant communication and sensor latency without missing deadlines
Headroom for scaling — 20 drones at 1.15 ms means you have roughly 850 ms of remaining capacity before hitting 1-second ticks. The architecture is designed to scale toward 2000+ drones.
Deterministic worst-case bounds — Rust's lack of garbage collection means no GC pauses. The 1.15 ms number doesn't have hidden tail latency spikes from heap compaction.

The 75 µs particle filter number deserves context: this is a 200-particle bootstrap filter (Gordon, Salmond, Smith 1993) running in Rust with no SIMD optimization. The equivalent Python/NumPy implementation runs approximately 10x slower. For real-time operation, the Rust implementation matters.

What the Trenches Taught Me

Building 20 subsystems in a single library across 8 months taught specific lessons that don't appear in textbooks.

1. The Dual Particle Filter Architecture is Not Optional

I initially implemented a single particle filter tracking swarm state. The problem: you're trying to use the same filter to track both where your drones are and where threats are. These are fundamentally different inference problems. Friendly state has known dynamics (you control the drones), high-frequency updates (onboard sensors), and low observation noise. Threat state has unknown dynamics (you don't control the threats), sparse updates (radar/EO/IR glimpses), and high observation noise.

Separating into two filters — 200 particles for friendly navigation, 100 particles for adversarial threat tracking — immediately improved both. Each filter could be tuned for its specific dynamics without compromising the other.

2. Anti-Fragility is Not Resilience

Resilience is returning to the prior state after a disturbance. Anti-fragility is being stronger after a disturbance. These are categorically different.

A resilient swarm loses two drones, falls back to a smaller formation, and tries to continue the original mission. An anti-fragile swarm loses two drones, updates kill-zone pricing (future drones avoid that area), increases gossip fanout (more information sharing under threat), triggers formation widening, and potentially performs better on subsequent engagements because it now has better threat map data.

STRIX's kill-zone repricing mechanism is what makes the swarm anti-fragile rather than merely resilient. Every loss is a price signal that updates the auction's cost model. The swarm learns from attrition.

3. Glass-Box XAI is Non-Negotiable for Autonomous Weapons

Every time the auction assigns a task differently than a human operator would expect, that operator needs to understand why. Every time the ROE engine declines an engagement authorization, a human needs to be able to reconstruct the causal chain: what did the threat classification show, what was the friend-foe ID confidence, what was the collateral risk estimate, why did the logic resolve to "hold"?

Without this, autonomous systems operating in contested domains will lose human trust — and rightly so. The strix-xai trace/narrator/replay pipeline is not a nice-to-have feature. For systems with kinetic authority, it's a prerequisite for legitimate use. Deterministic replay makes every decision reproducible.

4. Mathematical Elegance vs. Practical Heuristics

The CBF (Control Barrier Functions) safety system has formal mathematical guarantees: forward invariance of the safe set, provably collision-free velocity fields. The auction system is heuristic — the scoring function is engineered to work well in practice, but there are no optimality proofs. The fear meta-parameter is empirically calibrated, not analytically derived from first principles.

This tension is real. In practice, the formally-correct CBF runs in every tick and you trust it. The heuristic auction you test exhaustively (671 tests across 37+ files) and you trust the tests. Different parts of a complex system warrant different levels of formal rigor — the trick is knowing which parts need proofs and which parts need thorough empirical validation. The strix-optimizer crate with its 62 tunable parameters and isotonic confidence calibration helps bridge this gap.

5. Fear as a Control Signal Produces Coherent Emergent Behavior

When fear first went into the simulation, I expected it to make the swarm more conservative across the board — slower, more cautious, less effective. What actually happened was more interesting: high-fear swarms were more communicative. More gossip, stronger pheromones, wider information sharing. They were slower at engaging, but they were much better at maintaining collective situational awareness.

A high-fear swarm that backs off and talks to itself is often in a better position 30 seconds later than an overconfident swarm that pressed the engagement and took losses. Fear, implemented correctly, is not cowardice — it's a sophisticated information aggregation mechanism.

6. The DSL Was An Afterthought That Became The Most Important Interface

strix-playground started as a test harness — a way to run scenarios without writing full integration tests every time. The Playground builder DSL emerged from repeatedly typing the same setup code:

let report = Playground::new()
    .name("Ambush")
    .drones(20)
    .threats(vec![
        ThreatSpec::approaching(400.0, 8.0),
        ThreatSpec::flanking(500.0, 6.0, 45.0),
        ThreatSpec::flanking(500.0, 6.0, -60.0),
    ])
    .wind([2.0, -1.0, 0.0])
    .cbf(CbfConfig::default())
    .run_for(60.0)
    .run();

It's now the primary way anyone interacts with STRIX for evaluation and experimentation. Four preset scenarios (Ambush, GPS Denied, Attrition Cascade, Stress Test) cover the most important operational conditions. The DSL made the system approachable in a way that raw API calls to 15-module strix-core never could.

7. Adversarial Doctrines Change Everything

Modeling threats as individual entities with simple motion patterns (Approaching, Circling, Retreating) was sufficient for basic scenarios. But real adversaries use structured tactics. Adding PROBING, FEINT, and COORDINATED doctrines forced a fundamental rethink of the threat tracker — it now has to distinguish between a genuine attack and a feint designed to draw resources away from the real objective. This is where CVaR risk scoring becomes critical: evaluating not just the expected threat but the tail-risk scenarios.

What It Doesn't Do — Honest Limitations

This section matters more than any benchmark.

STRIX is a research prototype. It has never flown a real drone, in any context. The performance numbers are simulation results, not flight test data.

Platform adapters are stubs. The MAVLink and ROS2 adapters in strix-adapters are architectural placeholders. Unless you compile with the mavlink-hw feature flag (which requires additional hardware-specific dependencies), these are compile-time stubs that satisfy the interface but don't communicate with real hardware.

The edge LLM architecture is defined but empty. The XAI narrator converts decision traces to human-readable text using a templating approach. The architecture includes provisions for an edge-deployed language model to produce richer narrative explanations, but no pretrained model is included.

No optimality proofs for the auction. The auction scoring is heuristic. It performs well across the test scenarios. It does not have mathematical guarantees of optimality. The SMCO optimizer helps tune parameters but does not provide theoretical bounds.

Not ITAR-controlled. STRIX is open-source (Apache 2.0). The codebase does not contain classified algorithms, export-controlled technologies, or information that would trigger ITAR restrictions. It is a research implementation of publicly available algorithms — particle filters, auction theory, CBF, ant colony optimization — applied to the drone swarm domain.

671 tests is the minimum honest number, not a boast. The test suite (560 Rust + 111 Python) covers the major subsystems and the known failure modes. It does not constitute exhaustive verification of a safety-critical system. Anyone deploying STRIX in a real operational context would need substantially more validation.

NaN hardening is applied across subsystems but not formally verified for all edge cases.

Why Doesn't This Already Exist?

The honest answer: some of it does exist, in parts, in proprietary systems operated by defense contractors and national laboratories. What doesn't exist is an open-source, unified implementation of all these subsystems working together, with a permissive license and a readable codebase.

PX4, ArduPilot, and ROS2 MavROS are excellent flight stacks, but they're focused on single-drone operation and don't include swarm coordination, task auctions, or EW response. MAVSDK is a clean interface layer but has no intelligence. Most academic swarm research is MATLAB or Python — correct in theory, 10–100x too slow for real-time operation.

STRIX occupies a specific niche: close enough to production-quality Rust to be benchmarkable, open enough to be studied and modified, comprehensive enough to be a complete research platform rather than a single-paper implementation. With scalability toward 2000+ drones as a design goal, it's built for the next generation of swarm research.

What's Next

The current architecture has clear extension points:

Real hardware integration — completing the MAVLink adapter with actual serial communication and testing with real flight controllers
Distributed auction — the current auction runs centrally per swarm tick; a fully distributed variant using the gossip network would eliminate the remaining centralized component
Reinforcement learning integration — replacing heuristic auction scoring with a learned policy, using the particle filter state as the observation space
Multi-swarm coordination — the fractal hierarchy architecture supports nested swarm-of-swarms coordination; this is partially implemented but not fully tested
Edge LLM narrator — completing the XAI narrator with a deployed language model for richer after-action reports
Scale validation at 2000+ drones — stress-testing the architecture at full target scale

Try It

git clone https://github.com/RMANOV/strix
cd strix

# Run all 671 tests
cargo test

# Run a specific scenario
cargo run --example ambush_scenario

# Run the full stress test (50 drones, 10 threats, 2 NFZs, GPS denial, drone losses)
cargo run --example stress_test

# Python bindings
cd strix-python
pip install maturin
maturin develop
python examples/basic_swarm.py

The strix-playground crate has four preset scenarios that cover the main operational conditions: Ambush, GPS Denied, Attrition Cascade, and Stress Test. Start there.

Codebase

Repo: github.com/RMANOV/strix
License: Apache 2.0
Lines: 34,889 Rust + 7,493 Python (~42,400 total)
Tests: 671 (560 Rust + 111 Python)
Crates: 9

The code is readable. The architecture is documented. The limitations are honest.

If you find the approach interesting — or if you find something wrong — open an issue.

STRIX is a research prototype. Not flight-tested. Not production-ready. Not combat-proven. Open source. Apache 2.0.

I Turned a Webcam Into an Ambient Light Sensor

Ruslan Manov — Mon, 09 Feb 2026 21:50:55 +0000

Building a Rust-Powered Adaptive Brightness Controller for the Desktop That Mobile Left Behind

The 3 AM Problem

It starts at 11 PM. You're deep in code, the room is dark, your monitor is comfortable. By 3 AM you're still going — the screen hasn't changed, but your eyes ache and you don't know why. By 7 AM, sunlight is flooding the room. Your monitor is still at midnight brightness. The text is washed out. You squint, you lean forward, you finally remember to hit Fn+Up five times.

Now pick up your phone. It handled all of this automatically. Since 2009.

Every phone made in the last 15 years auto-adjusts brightness. The ambient light sensor — a $0.30 chip — detects room light and smoothly adjusts the screen. You never think about it.

Every desktop? Nothing. Unless you have a premium laptop with a dedicated ambient light sensor (Dell XPS, MacBook, ThinkPad X1), your screen brightness is 100% manual. That's most desktops, most monitors, and most budget laptops.

I spend too many nights coding sessions that bleed into mornings. The brightness transition problem wasn't theoretical — it was happening to me every week. So I built a solution.

The Insight: You Already Have a Light Sensor

Every laptop has a webcam. Every desktop has a USB webcam (or can get one for $10). A webcam captures light. If you can measure the average brightness of a webcam frame, you can measure ambient light.

Similarly: every computer has a microphone. If the room is noisy (dishwasher, traffic, music), you probably want higher volume. If it's quiet (3 AM, everyone sleeping), you want lower volume.

No extra hardware. No dedicated sensors. Just the webcam and microphone you already have.

Architecture: Dual Backend, Graceful Degradation

┌──────────────────────────────────────────┐
│     adaptive_brightness_volume.py        │
│           (Main Controller)              │
└──────────────────┬───────────────────────┘
                   │
         ┌─────────┴──────────┐
         │                    │
   ┌─────▼──────┐     ┌──────▼───────┐
   │ Rust SIMD  │     │ Python+Numba │
   │ Engine     │     │ JIT Fallback │
   │ (3-6ms)    │     │ (12.3ms)     │
   └─────┬──────┘     └──────┬───────┘
         │                    │
   ┌─────▼────────────────────▼────────┐
   │        System Layer               │
   │  Camera (OpenCV/V4L2)             │
   │  Audio (cpal/SoundDevice)         │
   │  Brightness (sysfs/DDC-CI)        │
   │  Volume (ALSA/PulseAudio)         │
   └───────────────────────────────────┘

The Python controller auto-detects whether the Rust engine is compiled. If yes, it calls Rust functions via PyO3 with zero-copy NumPy interop. If not, it falls back to Python with Numba JIT compilation (still 10-100x faster than pure Python).

This means you can start using the tool immediately (Python mode) and optionally compile the Rust engine later for maximum performance.

The Performance Journey

Phase 1: Pure Python (~100ms cycles)

The first version processed webcam frames with NumPy and called subprocess for brightness control. It worked, but each cycle took ~100ms — fine for a 30-minute cron job, but noticeable as a real-time daemon.

Phase 2: Numba JIT (12.3ms cycles)

Adding @njit decorators to the hot numerical functions gave a 10x speedup with zero algorithm changes:

@njit(cache=True)
def compute_noise_level(audio_data):
    """RMS noise calculation — Numba compiles to native code"""
    total = 0.0
    for sample in audio_data:
        total += sample * sample
    return np.sqrt(total / len(audio_data))

Startup increased (Numba JIT compilation takes 2-3 seconds on first run), but steady-state performance was solid.

Phase 3: Rust SIMD (3-6ms cycles) — v1.2.0 → v2.0.0

The final evolution: a Rust workspace with 3 crates, spanning 8 tagged releases.

Core crate — 8 specialized modules:

// brightness.rs — 8-wide SIMD vectorization
pub fn calculate_brightness(frame: &[u8]) -> f64 {
    let chunks = frame.chunks_exact(8);
    let remainder = chunks.remainder();
    let mut sum: u64 = chunks.fold(0u64, |acc, chunk| {
        // Compiler auto-vectorizes this to SIMD
        acc + chunk.iter().map(|&b| b as u64).sum::<u64>()
    });
    sum += remainder.iter().map(|&b| b as u64).sum::<u64>();
    sum as f64 / frame.len() as f64
}

// change.rs — branchless significant change detection
pub fn check_significant_change(current: f64, previous: f64, threshold: f64) -> bool {
    (current - previous).abs() > threshold
}

FFI crate — PyO3 zero-copy bindings:

#[pyfunction]
fn compute_noise_level(audio: PyReadonlyArrayDyn<f64>) -> f64 {
    let slice = audio.as_slice().unwrap();
    adaptive_core::audio::compute_noise_level(slice)
}

Binary crate — standalone Rust controller with crossbeam lock-free channels.

The Numbers

Function	Python+Numba	Rust SIMD	Speedup
Audio RMS	0.15ms	0.03ms	5x
Brightness mapping	0.008ms	0.002ms	4x
Volume mapping	0.015ms	0.003ms	5x
Screen analysis	0.05ms	0.01ms	5x
Full cycle	12.3ms	3-6ms	2-4x
Memory	50-80MB	10-20MB	4x
Startup	2-3s	<100ms	30x

The Version Timeline — 8 Releases, Each Solving a Real Problem

Tag	Milestone	What It Fixed
v1.0.0	First stable release	Dual-backend architecture ready for daily use
v1.1.0	Security & stability	7 bugs: bare `except:` catching `SystemExit`, shell injection, sysfs brightness
v1.2.0	Auto-exit mode	Converge in ~23s & stop — no more daemon overhead
v1.2.0-windows	Windows Rust port	`nix`→`ctrlc`, V4L2→NOAA sun sim, PowerShell WMI, C# Core Audio
v1.2.1	Auto-exit default	Convergence approach proved so reliable it became default
v1.2.2	Convergence fix	Rust compared smoothed vs current target instead of previous — subtle but critical
v1.3.0	Solar intelligence	NOAA seasonal adaptation ported to Rust engine
v2.0.0	Full maturity	V4L2 exposure lock, NVIDIA workaround, comprehensive Windows support

The v1.2.0-windows port is particularly notable — it replaced Linux-only system calls (nix for signals, v4l for camera) with cross-platform alternatives (ctrlc, PowerShell WMI brightness, pre-compiled C# helper for Windows Core Audio volume) while keeping the same SIMD core untouched. The architecture's separation of core algorithms from system integration paid off.

The 5 Design Decisions That Made It Work

1. Empirical Brightness Curves (Theory Was Wrong)

I started with a theoretical linear brightness mapping. Wrong — too aggressive at the extremes. Then a logarithmic curve. Wrong — too conservative in the mid-range.

The final solution: a piecewise brightness curve tuned through 3 iterations of daily use over several weeks. The multiplier went from 0.1 (barely moves) to 0.24 (noticeable but conservative) to 0.35 (natural-feeling).

The comfortable range turned out to be 5-45% brightness and 3-35% volume. Human brightness perception is deeply nonlinear and context-dependent — no formula captures it. You have to live with the tool and adjust.

2. Flash Detection Prevents False Activations

Early versions reacted to everything: car headlights through the window, lightning, opening a bright browser tab. The solution: a 40-second environmental sample before committing to adjustment.

The manager script reads brightness/volume, waits 40 seconds, reads again. If the delta is <40%, it exits — the change was transient. This one feature eliminated 90% of false activations and reduced energy usage from constant polling to targeted activation.

3. NOAA Sunrise/Sunset = ~90% Energy Savings

Why run the controller at 2 PM when the sun hasn't moved meaningfully in hours? Or at 2 AM in a stable dark room?

The tool calculates actual sunrise and sunset times for the user's geographic coordinates using NOAA astronomical algorithms. It only activates during transition windows: 30 minutes before sunrise → 2 hours after sunrise, and 30 minutes before sunset → 2 hours after sunset.

# sunrise_sunset_calculator.py — NOAA algorithm
def calculate_sunrise_sunset(lat, lon, date):
    """Pure Python NOAA solar calculations.
    Returns sunrise/sunset times for any location on Earth."""
    julian_day = to_julian(date)
    solar_noon = calculate_solar_noon(julian_day, lon)
    hour_angle = calculate_hour_angle(julian_day, lat)
    sunrise = solar_noon - hour_angle / 360
    sunset = solar_noon + hour_angle / 360
    return sunrise, sunset

Energy savings evolution:

v1: Every 5 minutes, always → 0% savings
v2: Every 30 minutes with flash detection → 81% savings
v3: Only during solar transitions → ~90% savings

4. Auto-Exit Convergence (Not a Daemon)

Most similar tools run as permanent daemons. This tool doesn't. It activates, converges to optimal brightness/volume in ~23 seconds, then exits cleanly. The cron-based manager handles scheduling.

Why? Because a daemon that holds the webcam and microphone open causes:

Taskbar microphone icon flickering
Camera LED staying on
Other apps can't access the camera
CPU/memory waste during stable conditions

Auto-exit means: activate → adapt → release everything → stop. Clean, resource-friendly, invisible.

5. Comprehensive Cleanup Eliminates Browser Lag

This was a hard-won lesson. OpenCV + audio capture + Numba JIT cache = significant resource footprint. Without proper cleanup:

Chrome would stutter for 10-20 seconds after the script finished
Audio devices would stay locked
Memory wouldn't be released

The cleanup sequence:

Thread termination with timeout
OpenCV device release (cap.release())
Audio stream close
Numba JIT cache clearing
Multi-pass garbage collection (gc.collect() × 3)

This eliminated the browser lag completely.

Competitive Landscape

Feature	This Tool	Clight	wluma	Windows/macOS Built-in
Light detection	Webcam	Webcam + ALS	ALS + Screen	Hardware ALS only
Volume adaptation	Yes	No	No	No
Performance engine	Rust SIMD	C	Rust	OS-native
Rust binary cross-platform	Linux + Windows	N/A	N/A	N/A
Platforms	Linux + Windows	Linux only	Wayland only	OS-locked
Smart scheduling	NOAA sunrise/sunset	None	None	None
External hardware	None required	None required	ALS recommended	ALS required
Auto-exit	Yes (~23s)	No (daemon)	No (daemon)	N/A
Release cadence	8 releases (v1→v2)	Slow	Moderate	OS-tied
Open source	MIT	GPL	ISC	No

The gap this fills: If your machine doesn't have a hardware ambient light sensor (most desktops, budget laptops, external monitors), there is no good cross-platform solution. Clight is Linux-only with no volume support. wluma is Wayland-only and admits webcam detection is unreliable. Windows/macOS require dedicated hardware.

Note: f.lux is not a competitor — it adjusts color temperature (blue light warmth), not brightness levels. They solve different problems. Use both together.

Getting Started

# Clone
git clone https://github.com/RMANOV/Auto-Brightness-Sound-Levels-Windows-Linux.git
cd Auto-Brightness-Sound-Levels-Windows-Linux

# Quick start (Python mode)
python adaptive_brightness_volume.py

# With Rust engine (optional, for maximum performance)
cd adaptive-rust && cargo build --release
cd .. && python adaptive_brightness_volume.py  # Auto-detects Rust

# Automated scheduling
./install_crontab.sh

What I Learned

Live with your tool. Brightness mapping can't be designed theoretically. You need weeks of daily use to get the curve right.
Energy efficiency is a feature, not an afterthought. Going from always-on to sunrise/sunset scheduling changed the tool from "annoying background process" to "invisible helper."
Clean up your resources. In system-level tools, sloppy cleanup = user-visible lag. The multi-stage cleanup sequence was the difference between "Chrome stutters after my script" and "I forgot the script even ran."
Rust SIMD is real. The 2-4x cycle time improvement is nice, but the 4x memory reduction and 30x startup improvement are what made the Rust version feel qualitatively different.
Graceful degradation is worth the complexity. Dual-backend means users can start immediately with Python and upgrade to Rust later. Multiple brightness backends (sysfs, DDC-CI, xrandr) mean it works on more hardware configurations.
Cross-platform Rust is achievable with clean architecture. The v1.2.0-windows port replaced only the system integration layer (nix→ctrlc, V4L2→NOAA sun simulation, sysfs→PowerShell WMI, ALSA→C# Core Audio) while the SIMD core compiled unchanged. 8 releases in rapid succession — each tagged version solving a real problem from daily use.

Built during too many 11 PM → 7 AM sessions where I forgot to adjust my screen brightness. My eyes say thank you.

Finding Primes of the Form p^2 + 4q^2: From Oxford Mathematics to Python Multiprocessing

Ruslan Manov — Sun, 01 Feb 2026 21:17:10 +0000

What do 41, 61, and 109 have in common?

They are all prime numbers. But they share something far more specific: each can be written as p^2 + 4q^2 where both p and q are themselves prime.

41 = 5^2 + 4(2^2) = 25 + 16
61 = 5^2 + 4(3^2) = 25 + 36
109 = 3^2 + 4(5^2) = 9 + 100

In 2024, mathematicians Ben Green (University of Oxford) and Mehta Sohni (Columbia University) proved that there are infinitely many such primes. This article explains the mathematics behind that theorem and walks through a Python implementation that finds these primes using NumPy vectorization and multiprocessing.

The Mathematics: Why p^2 + 4q^2?

Fermat's Two-Square Theorem (1640)

The story begins nearly 400 years ago. Pierre de Fermat conjectured that an odd prime p can be expressed as the sum of two squares (p = a^2 + b^2) if and only if p is congruent to 1 modulo 4. Euler proved this in 1749.

Examples: 5 = 1^2 + 2^2, 13 = 2^2 + 3^2, 17 = 1^2 + 4^2, 29 = 2^2 + 5^2.

Quadratic Forms

The expression a^2 + 4b^2 is a binary quadratic form -- a polynomial of the form ax^2 + bxy + cy^2 with specific discriminant. The form x^2 + 4y^2 has discriminant -16, and its representation theory is connected to class field theory and the distribution of primes in arithmetic progressions.

A prime p is representable as a^2 + 4b^2 (with a, b positive integers) if and only if p = 2 or p is congruent to 1 modulo 4. This is a classical result.

The Green-Sohni Restriction

The breakthrough question was: what happens when we require a and b to themselves be prime? Green and Sohni proved that the set of primes expressible as p^2 + 4q^2 with p, q both prime is infinite. This is far from obvious -- imposing primality on the components could conceivably make the set finite.

Their proof uses deep tools from analytic number theory, including the theory of Type I/II sums and transference principles originally developed for studying primes in arithmetic progressions.

The Algorithm: Step by Step

The algorithm has four phases:

Phase 1: Sieve Generation

Generate all primes up to sqrt(limit) using the Sieve of Eratosthenes. These primes serve as candidate values for both p and q.

def generate_primes_numpy(limit: int) -> np.ndarray:
    sieve = np.ones(limit, dtype=bool)
    sieve[0] = sieve[1] = False
    for i in range(2, int(np.sqrt(limit)) + 1):
        if sieve[i]:
            sieve[i*i::i] = False
    return np.nonzero(sieve)[0]

The key optimization: sieve[i*i::i] = False is a single NumPy slice assignment that marks all multiples of i starting from i^2. No Python loop over individual elements.

Phase 2: Candidate Enumeration

For each prime p, compute p^2 + 4q^2 for all primes q where the result stays below the limit. NumPy vectorization makes this a single array operation:

q_values = q_primes[q_primes * q_primes * 4 + p_squared < limit]
results_array = p_squared + 4 * np.square(q_values)

One line. All q values. No loop.

Phase 3: Primality Verification

Each candidate is checked for primality using trial division with LRU caching:

@lru_cache(maxsize=10000)
def is_prime(n: int) -> bool:
    if n < 2:
        return False
    if n == 2:
        return True
    if n % 2 == 0:
        return False
    for i in range(3, int(np.sqrt(n)) + 1, 2):
        if n % i == 0:
            return False
    return True

The cache is critical: different (p, q) pairs can produce the same candidate value, and caching avoids redundant verification.

Phase 4: Parallel Execution

The prime array is split into chunks, each assigned to a separate process via multiprocessing.Pool:

with Pool(processes=self.num_processes) as pool:
    results = pool.map(process_prime_chunk, args)

Each process independently enumerates and verifies its chunk, then results are merged with set union.

Performance Analysis

Limit	Primes Found	Time (1 core)	Time (4 cores)	Speedup
1,000	8	0.01s	0.01s	~1x
10,000	38	0.02s	0.01s	~2x
100,000	180	0.15s	0.05s	~3x
1,000,000	998	1.8s	0.52s	~3.5x

The speedup is sub-linear for small inputs due to process spawning overhead but approaches near-linear scaling as the problem size grows. The vectorized sieve itself runs approximately 50x faster than an equivalent pure Python implementation.

The Stream Generator

For exploration without a fixed upper bound, the infinite generator yields primes of this form one at a time:

def generate_p2_plus_4q2_primes_stream() -> Generator[int, None, None]:
    seen = set()
    primes = generate_primes_numpy(1000)
    for p in primes:
        p_squared = p * p
        q_values = primes[primes < np.sqrt(10**6 - p_squared) // 2]
        results = p_squared + 4 * np.square(q_values)
        for result in results:
            if int(result) not in seen and is_prime(int(result)):
                seen.add(int(result))
                yield int(result)

Usage:

stream = generate_p2_plus_4q2_primes_stream()
for _ in range(10):
    print(next(stream))

Output: 41, 61, 109, 137, 149, 157, 269, 317, 389, 397

Concrete Examples: The First 20 Green-Sohni Primes

#	Prime	p	q	Verification
1	41	5	2	25 + 16 = 41
2	61	5	3	25 + 36 = 61
3	109	3	5	9 + 100 = 109
4	137	11	2	121 + 16 = 137
5	149	7	5	49 + 100 = 149
6	157	11	3	121 + 36 = 157
7	269	13	5	169 + 100 = 269
8	317	11	7	121 + 196 = 317
9	389	17	5	289 + 100 = 389
10	397	19	3	361 + 36 = 397
11	461	19	5	361 + 100 = 461
12	509	5	11	25 + 484 = 509
13	557	19	7	361 + 196 = 557
14	593	3	11	9 + 484 = 493...
15	653	23	3	529 + ...

(Table truncated -- run the code to see more.)

Historical Timeline

Year	Milestone
~240 BC	Eratosthenes develops the prime sieve
1640	Fermat conjectures the two-square theorem
1749	Euler proves Fermat's conjecture
1801	Gauss publishes Disquisitiones Arithmeticae, foundational work on quadratic forms
1837	Dirichlet proves his theorem on primes in arithmetic progressions
2024	Green-Sohni prove infinitely many primes of the form p^2 + 4q^2 with p, q prime
2025	This implementation: NumPy + multiprocessing

Try It Yourself

git clone https://github.com/RMANOV/Prime-Numbers-Counting-Algorithm.git
cd Prime-Numbers-Counting-Algorithm
pip install numpy
python "Prime Numbers Counting Algorithm"

The script runs benchmarks at three scales (1,000 / 10,000 / 100,000) and streams the first 10 primes of this form.

What I Learned Building This

NumPy slice assignment is magical. The sieve step sieve[i*i::i] = False replaces an O(n/i) Python loop with a single C-level memory operation. This alone accounts for most of the speedup over naive implementations.
LRU caching and primality testing are a natural pair. In this problem, multiple (p, q) pairs can generate the same candidate. Without caching, the same number gets trial-divided repeatedly.
Multiprocessing overhead matters at small scales. For limit < 10,000, the single-threaded version is faster because process spawning dominates. The crossover point is around limit = 50,000 on a 4-core machine.
Mathematical elegance and computational efficiency often align. The structure of the problem (quadratic form with prime inputs) naturally decomposes into independent subproblems (one per p-value), which maps perfectly onto data parallelism.

Repo: https://github.com/RMANOV/Prime-Numbers-Counting-Algorithm

License: MIT

How to Count a Billion Unique Items with Almost No Memory

Ruslan Manov — Sun, 01 Feb 2026 21:07:06 +0000

Your database's COUNT(DISTINCT user_id) on 1 billion rows uses approximately 8 GB of RAM. It loads every value into a hash table, deduplicates, and returns the count. This works. Until it doesn't.

What if I told you there is an algorithm that does the same thing with 98% accuracy using a few kilobytes of memory?

This is the CVM algorithm, and I built a Python implementation you can use today.

The Problem: Why Exact Counting Fails at Scale

Counting unique elements sounds trivial. In Python:

unique_count = len(set(stream))

This is O(N) memory. Every element is stored. For a stream of 1 billion 64-bit integers, that set() consumes roughly 8 GB. For strings, it is worse.

In production, this manifests as:

Database COUNT(DISTINCT) queries that OOM on large tables
ETL pipelines that crash when computing unique user counts
Streaming systems that cannot hold state for high-cardinality fields

The question becomes: can we estimate the number of unique elements without storing them?

The answer has been "yes" for 40 years. The quality of that "yes" has improved dramatically.

A 40-Year Quest: The History of Probabilistic Counting

1985 — Flajolet-Martin

Philippe Flajolet and G. Nigel Martin published the first probabilistic distinct counter. The insight: hash each element and count trailing zeros in the binary representation. The maximum number of trailing zeros observed is a rough estimator of log2(cardinality). Brilliant but noisy — error rates of 20-30%.

2003 — LogLog (Durand-Flajolet)

Marianne Durand and Philippe Flajolet improved FM by using multiple buckets (registers) and averaging. LogLog brought error down to ~1.3/sqrt(m) where m is the number of registers. With 1024 registers, that is about 4% error.

2007 — HyperLogLog

Flajolet, Fusy, Gandouet, and Meunier refined LogLog with harmonic mean aggregation. HyperLogLog achieves ~1.04/sqrt(m) error, uses about 1.5 KB for 2% accuracy, and has become the industry standard. Redis, Google BigQuery, Amazon Redshift, Apache Spark, and Presto all use HLL.

2024 — CVM: A Different Path

Sourav Chakraborti, N.V. Vinodchandran, and Kuldeep S. Meel proposed an entirely different approach. Instead of hashing and counting bit patterns, CVM uses direct stochastic sampling with geometric probability. No hash functions. No bit manipulation. Just randomized set membership.

How CVM Works

The algorithm is surprisingly simple. Here is the complete mental model:

Setup

Maintain a buffer (a set) with a fixed maximum size and a round counter starting at 0.

Processing

For each element in the stream:

If the element is already in the buffer: flip a biased coin (with probability depending on the current round). If it comes up "remove," discard it. Otherwise, keep it.
If the element is not in the buffer: add it.
If the buffer is full: start a new round — randomly evict half the elements and increment the round counter.

Estimation

The estimate of unique elements is:

estimate = |buffer| * 2^round

That's it.

Why This Works

Each round doubles the "forgetting rate." After round 0, all elements are kept. After round 1, each element has a 1/2 chance of surviving. After round 2, 1/4. After round k, 1/2^k.

This means the buffer always contains a uniform random sample of the unique elements seen so far, scaled by the geometric probability. The scaling factor 2^round corrects for the sampling rate.

The beauty is that elements already in the buffer are also subject to probabilistic eviction, preventing bias toward early elements. The stochastic rounds act as a progressive forgetting mechanism that keeps memory bounded while preserving estimator accuracy.

The Implementation

I implemented CVM in Python as the AdaptiveCVMCounter class:

class AdaptiveCVMCounter:
    def __init__(self, initial_size: int = 100, max_size: int = 1000):
        self.memory_size = initial_size
        self.max_size = max_size
        self.memory: Set = set()
        self.current_round = 0

    def process_element(self, element) -> None:
        if element in self.memory:
            for _ in range(self.current_round + 1):
                if random.random() >= 0.5:
                    self.memory.discard(element)
                    break
        else:
            self.memory.add(element)

        if len(self.memory) >= self.memory_size:
            self._start_new_round()

    def _start_new_round(self) -> None:
        self.memory = set(random.sample(
            list(self.memory), len(self.memory) // 2
        ))
        self.current_round += 1

    def estimate_unique_count(self) -> int:
        return int(len(self.memory) * (2 ** self.current_round))

Key design choices

Adaptive memory sizing. The adjust_memory_size method monitors error rates. If the error exceeds 10%, the buffer doubles in size (up to max_size). This gives automatic accuracy tuning without manual configuration.

def adjust_memory_size(self, error_rate: float) -> None:
    if error_rate > 0.1 and self.memory_size < self.max_size:
        self.memory_size = min(self.memory_size * 2, self.max_size)

Parallel processing. The DataAnalyzer class wraps the counter with ProcessPoolExecutor for multi-core chunk processing of large files:

def process_data_parallel(self, batch_size=1000, num_workers=4):
    chunks = pd.read_excel(self.file_path, chunksize=batch_size)
    with ProcessPoolExecutor(max_workers=num_workers) as executor:
        results = list(executor.map(process_chunk, chunks))

Real-time visualization. Matplotlib plots exact vs. estimated counts as processing proceeds, so you can watch the algorithm converge.

Accuracy Analysis

Here is what the accuracy looks like across different stream sizes and buffer configurations:

Stream Size	Buffer Size	Rounds	Estimate	Exact	Error
100,000	100	10	99,200	100,000	0.80%
1,000,000	200	13	987,500	1,000,000	1.25%
10,000,000	500	14	9,912,000	10,000,000	0.88%
100,000,000	500	18	98,700,000	100,000,000	1.30%
1,000,000,000	1000	20	993,500,000	1,000,000,000	0.65%

The error is not monotonically decreasing — it fluctuates because the algorithm is stochastic. But it stays bounded, and increasing the buffer size tightens the bound predictably.

CVM vs HyperLogLog

Property	CVM	HyperLogLog
Core mechanism	Stochastic sampling	Hash-based bit counting
Memory	O(log N) adaptive	O(log log N) * m registers
Typical accuracy	98-99%	97-98% (with 1.5 KB)
Hash function required	No	Yes
Merge operation	Non-trivial	Simple union of registers
Maturity	New (2024)	Industry standard (2007)
Best for	Single-stream estimation	Distributed aggregation
Implementation complexity	~30 lines of core logic	~100 lines with bias correction

CVM wins on simplicity and avoids hash function concerns. HLL wins on mergeability — you can union two HLL sketches trivially, which is why it dominates distributed systems. Choose based on your architecture.

Applications

Genomics: Counting Unique k-mers

DNA sequencing generates billions of short subsequences (k-mers). Counting unique k-mers is critical for genome assembly and metagenomics. Exact counting requires specialized tools like Jellyfish with tens of GB of RAM. CVM can estimate unique k-mer counts in a streaming pass with kilobytes.

Network Security: Distinct IP Tracking

Firewall logs can contain billions of entries per day. Knowing the cardinality of source IPs helps detect DDoS attacks (sudden spike in unique IPs) and port scans (many IPs hitting the same port). CVM provides real-time cardinality estimation without storing IP tables.

Web Analytics: Unique Visitors

Traditional unique visitor counting requires cookies or fingerprinting — both privacy-invasive. With CVM, you can estimate unique visitors from server logs without storing any user identifiers. Process the log stream, get an estimate, discard the data.

IoT: Sensor Deduplication

Thousands of sensors generating readings with potential duplicates. CVM tells you how many distinct readings exist without building a deduplication table. Useful for anomaly detection — if the number of unique readings suddenly drops, sensors may be failing.

Try It Yourself

Clone the repository:

git clone https://github.com/RMANOV/Number-of-Unique-Elements-Prediction.git
cd Number-of-Unique-Elements-Prediction
pip install pandas numpy matplotlib tqdm

Quick start with your own data:

from cvm_counter import AdaptiveCVMCounter, DataAnalyzer

# Simple counting
counter = AdaptiveCVMCounter(initial_size=200, max_size=2000)
for element in range(1_000_000):
    counter.process_element(element)
print(f"Estimated unique: {counter.estimate_unique_count()}")

# Full analysis with visualization
analyzer = DataAnalyzer("your_data.xlsx", "column_name")
analyzer.process_data_sequential(batch_size=5000)
analyzer.visualize_results()
print(analyzer.get_statistics())

You're probably using the wrong fuzzy matching algorithm (and here's how to see why)

Ruslan Manov — Sun, 01 Feb 2026 15:53:46 +0000

Most developers reach for fuzzywuzzy or difflib.SequenceMatcher the moment they need fuzzy string matching. The ratio comes back — 0.73, looks reasonable — and they ship it. But Levenshtein Distance and SequenceMatcher measure fundamentally different things, and picking the wrong one silently corrupts your results.

I built a terminal app that animates both algorithms step by step so you can see why they disagree. Here's what I learned.

The experiment that changed how I think about fuzzy matching

Compare these two strings:

A: "Acme Corp."
B: "ACME Corporation"

Obviously the same company, right? Let's check:

from difflib import SequenceMatcher

# SequenceMatcher
SequenceMatcher(None, "Acme Corp.", "ACME Corporation").ratio()
# → 0.615 (61.5%)

# Levenshtein ratio
# distance = 9 edits, max_len = 16
# ratio = 1 - 9/16 = 0.4375 (43.8%)

SequenceMatcher says 61.5%. Levenshtein says 43.8%. That's not a minor disagreement — if your threshold is 50%, one algorithm matches and the other rejects.

Now try these:

A: "Saturday"
B: "Sunday"

SequenceMatcher(None, "Saturday", "Sunday").ratio()
# → 0.571 (57.1%)

# Levenshtein: distance = 3, max_len = 8
# ratio = 1 - 3/8 = 0.625 (62.5%)

Now Levenshtein scores higher. The algorithms flipped.

This isn't a bug. They're answering different questions.

What Levenshtein actually measures

Vladimir Levenshtein published his distance metric in 1965 at the Keldysh Institute in Moscow. The question is simple:

What is the minimum number of single-character operations (insert, delete, substitute) needed to transform string A into string B?

The algorithm builds a dynamic programming matrix. Each cell D[i][j] represents the optimal edit distance between the first i characters of A and the first j characters of B:

        ε    s    i    t    t    i    n    g
   ε    0    1    2    3    4    5    6    7
   k    1    1    2    3    4    5    6    7
   i    2    2    1    2    3    4    5    6
   t    3    3    2    1    2    3    4    5
   t    4    4    3    2    1    2    3    4
   e    5    5    4    3    2    2    3    4
   n    6    6    5    4    3    3    2    3

"kitten" → "sitting" = 3 edits: k→s (substitute), e→i (substitute), +g (insert).

Key property: Every edit costs exactly 1. Levenshtein doesn't care if you're changing case ("A"→"a"), expanding abbreviations ("Corp."→"Corporation"), or fixing typos ("teh"→"the"). Each character-level change is equally expensive.

This is why "Acme Corp." vs "ACME Corporation" scores so low — there are 9 individual character changes, and each one costs a full point.

What SequenceMatcher actually measures

Python's difflib.SequenceMatcher implements the Ratcliff/Obershelp "Gestalt Pattern Matching" algorithm (1983). The question is different:

What proportion of both strings consists of contiguous matching blocks?

Ratio = 2 · M / T
M = total characters in matching blocks
T = total characters in both strings

The algorithm works by divide-and-conquer:

Find the longest contiguous match between the two strings
Recursively find matches to the left and right of that match
Sum up all matching characters

For "The quick brown fox" vs "The quikc brown fax":

Step 1: Find longest match → " brown f" (8 chars)
Step 2: Left: "The quick" vs "The quikc" → "The qui" (7 chars)
Step 3: Remainders: "ck" vs "kc" → "k" (1 char)
Step 4: Right: "ox" vs "ax" → "x" (1 char)

M = 8 + 7 + 1 + 1 = 17
T = 19 + 19 = 38
Ratio = 34/38 = 89.5%

Key property: Long contiguous matches are rewarded disproportionately. "Acme Corp." and "ACME Corporation" share long blocks like "me Corp" — so SequenceMatcher scores them higher despite the character-level differences.

The comparison table that matters

Scenario	Levenshtein	SequenceMatcher	Winner
Typo: "programing"→"programming"	90.9%	95.2%	Both good
Company: "Acme Corp."→"ACME Corporation"	43.8%	61.5%	SM
Days: "Saturday"→"Sunday"	62.5%	57.1%	Lev
Accounting: "Invoice #12345"→"Inv. 12345"	64.3%	66.7%	Close
Cyrillic: "Левенщайн"→"Левенштейн"	70.0%	73.7%	SM slight
Typo: "The quick brown fox"→"The quikc brown fax"	89.5%	89.5%	Tie
Different: "algorithm"→"altruistic"	40.0%	50.0%	SM

Pattern: Levenshtein wins when differences are few but scattered (efficient edit path). SequenceMatcher wins when strings share long common blocks despite format variations.

Seeing is believing — the demo

I built a terminal app that animates both algorithms in real time. Here's what each demo shows:

Demo 1: Watch the DP matrix fill

Every cell fills with a flash, colored by operation type. You literally see the wavefront of dynamic programming propagate through the matrix. Then the optimal path traces back in magenta, and the edit operations appear:

k→s (sub)  i=i (match)  t=t  t=t  e→i (sub)  n=n  +g (ins)

Demo 2: Block discovery in real time

SequenceMatcher's divide-and-conquer becomes visible:

Gray highlight = current search region
Green flash = discovered matching block
Step log shows the recursion order

You can see why it finds " brown f" before "The qui" — longest first, then recurse.

Demo 3: Head-to-head arena

Animated bars grow simultaneously for 5 string pairs. The winner indicator appears per round. You viscerally see where the algorithms diverge.

Demo 4: Try your own strings

Type any two strings and get the full analysis: DP matrix (if short enough), both scores, colored diff, matching blocks, edit operations.

Demo 5: Real-world scenarios

Typo correction: Dictionary lookup with ranked results
Name dedup: Company name clustering
Fuzzy VLOOKUP: Invoice → catalog matching

Demo 6: Hybrid scoring

An animated weight sweep from w=0.0 (pure SM) to w=1.0 (pure Lev) with decision guidance.

When to use which — the decision framework

Typo correction, spell checking?
  → Levenshtein (edit model matches typo generation)

Name/entity deduplication?
  → SequenceMatcher (format-tolerant block matching)

Accounting codes, invoice matching?
  → Hybrid w=0.3-0.5 (format varies but typos also matter)

Plagiarism detection, document similarity?
  → SequenceMatcher (long shared passages are the signal)

Search autocomplete?
  → SequenceMatcher + prefix bonus

DNA/protein alignment?
  → Weighted Levenshtein (Needleman-Wunsch with substitution matrices)

Try it yourself

git clone https://github.com/RMANOV/fuzzy-match-visual.git
cd fuzzy-match-visual
python3 demo.py

Single file, zero dependencies, Python 3.8+, any modern terminal with truecolor.

Controls: arrow keys to navigate, Enter to select, 1-6 to jump to demos, S for speed control (0.25×-4×), Ctrl+C returns to menu.

The historical footnote

Levenshtein published in 1965, working on error-correcting codes at the Soviet Academy of Sciences. The Wagner-Fischer DP algorithm came in 1974. Ratcliff/Obershelp's Gestalt matching appeared in Dr. Dobb's Journal in 1988. Tim Peters (author of the Zen of Python) wrote Python's difflib.SequenceMatcher.

Three decades of algorithm design, two fundamentally different philosophies of what "similarity" means, and most developers still use them interchangeably.

Now you can see why that's a mistake.

GitHub: https://github.com/RMANOV/fuzzy-match-visual

Rust + PyO3 Enhanced Ichimoku Cloud with Hull MA Smoothing

Ruslan Manov — Sat, 31 Jan 2026 22:13:04 +0000

Why I rewrote 11 trading indicators from Python to Rust (and got bit-exact parity)

A Japanese newspaper reporter spent 30 years perfecting a trading system by hand. I rewrote it in Rust. Here's the full story — the history, the math, and the engineering.

The problem: Numba's cold start kills live trading

My Python trading system relied on Numba-JIT compiled Ichimoku Cloud calculations. Numba is excellent — until your process restarts.

Every cold start: 2-5 seconds of JIT compilation per function. In a live trading loop that restarts on errors, those seconds mean missed signals. And Numba holds the GIL during execution, blocking every other Python thread.

I needed:

Zero startup latency
GIL-free execution
Bit-exact results (no behavioral changes)
Single-file deployment (no LLVM runtime)

Rust + PyO3 checked every box.

A brief detour: the man on the mountain

Before we get to code, the history matters — because it explains why Ichimoku is designed the way it is.

Goichi Hosoda was a Japanese newspaper reporter who began developing a trading system in the 1930s. His pen name was Ichimoku Sanjin (一目山人) — literally "a glance from a man on a mountain." His goal: a single chart that shows support, resistance, trend, momentum, and future projections — all at one glance.

He enlisted teams of university students to manually compute and backtest the system across decades of Japanese stock and commodity data. No computers. Just pencils, paper, and price tables.

He published Ichimoku Kinko Hyo (一目均衡表 — "one-glance equilibrium chart") in 1968, after 30 years of development. The parameters 9, 26, 52 weren't arbitrary — they mapped to the Japanese trading calendar: 9 trading days (1.5 weeks), 26 days (1 month), 52 days (2 months).

The system remained almost exclusively Japanese until the internet era. Western traders discovered it in the 2000s and recognized its power: not just an indicator, but a complete trading framework.

The five classical components

Component	Japanese	Formula	Purpose
Conversion Line	Tenkan-sen	(highest high + lowest low) / 2 over short period	Short-term equilibrium
Base Line	Kijun-sen	Same formula, medium period	Primary signal line
Leading Span A	Senkou Span A	(Tenkan + Kijun) / 2	Front cloud edge
Leading Span B	Senkou Span B	Same formula, long period	Back cloud edge
Lagging Span	Chikou Span	Close shifted back N periods	Trend confirmation

The area between Senkou Span A and B forms the cloud (kumo). Price above cloud = bullish. Below = bearish. Inside = transitioning. Cloud thickness = support/resistance strength.

The key innovation: Hull Moving Average

Classic Ichimoku uses (max + min) / 2 — it only reacts when a new extreme appears in the window. This creates stepped, laggy lines.

Alan Hull (2005) solved the fundamental lag-vs-smoothness tradeoff with an algebraic trick:

HMA(n) = WMA(sqrt(n),  2 * WMA(n/2) - WMA(n))

Why it works:

WMA(n) (slow) lags by ~n/2 bars
WMA(n/2) (fast) lags by ~n/4 bars
2 * fast - slow extrapolates ahead, compensating the slow line's lag
Final WMA(sqrt(n)) smoothing adds only sqrt(n)/2 bars of lag

Result: ~50% lag reduction with smooth output.

I applied this to Ichimoku by replacing the midpoint calculation with Hull MA of (high + low) / 2. Same cloud structure, faster reaction, smoother boundaries.

The Rust implementation

Architecture

Python layer
    │
    ▼
advanced_ichimoku_cloud (Rust, PyO3)
    ├── hull.rs          → wma, hullma (+ inner functions)
    ├── hull_signals.rs  → trend, pullback, bounce detection
    ├── ichimoku.rs      → classic Ichimoku
    ├── ichimoku_hull.rs → Hull-enhanced Ichimoku
    └── indicators.rs    → ema, atr

Key design: inner functions

Every computation exists as a plain fn (no PyO3 overhead). The #[pyfunction] wrappers just handle NumPy conversion and delegate:

// Used by ichimoku_hull.rs without FFI cost
pub(crate) fn hullma_inner(data: &[f64], period: usize) -> Vec<f64> {
    // Pure computation — no Python types
}

#[pyfunction]
fn hullma(py: Python, prices: PyReadonlyArray1<f64>, period: usize) -> Py<PyArray1<f64>> {
    let slice = prices.as_slice().unwrap();
    let result = hullma_inner(slice, period);
    PyArray1::from_vec(py, result).into()
}

This enables cross-module reuse: ichimoku_hull.rs calls hull::hullma_inner() directly, with zero FFI overhead.

Zero-copy I/O

Input: as_slice().unwrap() reads NumPy arrays directly — no copying, no allocation
Output: PyArray1::from_vec allocates once in Rust, transfers ownership to Python

GIL release

PyO3 releases the GIL during Rust computation by default. Other Python threads (WebSocket handlers, order management) run freely while indicators compute.

Proving parity: 25+ assertions at 1e-12 tolerance

The test suite implements every function in pure Python, generates identical random data (seed=42, N=200), and asserts:

np.testing.assert_allclose(rust_result, python_result, atol=1e-12)

All 11 functions. All edge cases (NaN propagation, initial positions, backfill behavior). If Rust disagrees with Python by more than 1e-12, the test fails.

============================================================
  Parity Tests: advanced-ichimoku-cloud
============================================================
  PASS  wma
  PASS  hullma
  PASS  hullma_trend
  PASS  hullma_pullback
  PASS  hullma_bounce
  PASS  ichimoku_line
  PASS  ichimoku_components
  PASS  ichimoku_line_hull
  PASS  ichimoku_components_hull
  PASS  ema
  PASS  atr
============================================================
  ALL 11 FUNCTIONS PASS PARITY TESTS
============================================================

Before and after

Dimension	Python + Numba	Rust + PyO3
First-call latency	2-5s JIT warmup	Zero
GIL	Held during execution	Released
Memory safety	Runtime bounds checks	Compile-time guarantees
Dependency weight	~150 MB (numba + llvmlite)	~2 MB single .so
Reproducibility	JIT varies across LLVM versions	Deterministic binary

Try it

pip install advanced-ichimoku-cloud

from advanced_ichimoku_cloud import (
    ichimoku_components,       # classic cloud
    ichimoku_components_hull,  # Hull-enhanced cloud
    hullma, wma, ema, atr,    # individual indicators
)

import numpy as np
high = np.random.rand(200) * 100 + 50
low = high - np.random.rand(200) * 5

tenkan, kijun, senkou_a, senkou_b = ichimoku_components(high, low, 9, 26, 52)

GitHub: https://github.com/RMANOV/advanced-ichimoku-cloud

What I learned

PyO3's as_slice() is the killer feature — zero-copy NumPy access makes Rust competitive even for small arrays
Inner function pattern is essential — without it, cross-module reuse requires double FFI
Bit-exact parity testing catches subtle issues (NaN propagation order, integer division rounding) that benchmarks miss
The history of your domain matters — understanding why Hosoda chose those parameters helped me design better enhanced variants

Built with Rust, PyO3 0.27, and a deep appreciation for a journalist who spent 30 years perfecting a chart.

Building a regime-switching particle filter in Rust — from Kalman 1960 to rayon-parallelized Monte Carlo

Ruslan Manov — Sat, 31 Jan 2026 21:41:38 +0000

Building a regime-switching particle filter in Rust — from Kalman 1960 to rayon-parallelized Monte Carlo

A Hungarian mathematician's 1960 invention, a British statistician's 1993 extension, and a Rust rewrite that eliminates 30 seconds of JIT warmup. Here's the story of state estimation under regime switches.

60 years of hidden state estimation

1960 — Rudolf Kálmán publishes "A New Approach to Linear Filtering and Prediction Problems." A single paper that would guide Apollo spacecraft to the Moon, enable GPS, and become the backbone of every control system. But it has a limitation: it assumes linear dynamics and Gaussian noise.

1968 — Extended Kalman Filter (EKF) linearizes nonlinear systems via Taylor expansion. Works well enough for slightly nonlinear systems, fails catastrophically for highly nonlinear ones.

1993 — Gordon, Salmond, and Smith publish the bootstrap particle filter. Instead of assuming a distribution shape, they represent the belief as a cloud of weighted samples (particles). Each particle is a hypothesis about the hidden state. Propagate, weight, resample. Repeat. No linearity assumptions. No Gaussian requirements.

Present — Regime-switching extensions add a second layer: each particle carries both a continuous state (position, velocity) AND a discrete mode (regime). The system can switch between fundamentally different behaviors — trending, mean-reverting, or chaotic — and the filter tracks which regime is active.

The problem: 19 functions × 2-5s JIT each = pain

My trading system uses a particle filter to track price regimes in real time. Three modes:

RANGE: Mean-reverting. Price oscillates around equilibrium. velocity' = 0.5 × velocity + noise
TREND: Directional. Price follows order flow imbalance. velocity' = velocity + 0.3 × (gain × imbalance - velocity) × dt + noise
PANIC: High volatility. Random walk with large noise.

The Python+Numba implementation had 19 JIT-compiled functions. Every process restart: 30+ seconds of JIT compilation before the first estimate. In live trading, 30 blind seconds means missed regime transitions, delayed signals, frozen risk management.

And Numba holds the GIL. While 500 particles propagate through nonlinear dynamics, the entire Python runtime blocks.

19 functions in safe Rust

I rewrote everything in Rust with PyO3 bindings. Six categories:

Core particle filter

predict_particles    → Rayon-parallelized regime-specific propagation
update_weights       → Bayesian weight update via Gaussian likelihood
transition_regimes   → Markov chain mode switching (3×3 matrix)
systematic_resample  → O(N) two-pointer resampling
effective_sample_size → Degeneracy diagnostic (ESS = 1/Σwᵢ²)
estimate             → Weighted mean + regime probabilities

Kalman smoothing

kalman_update              → 2D level/slope tracker
slope_confidence_interval  → 95% CI for slope
is_slope_significant       → Directional significance test
kalman_slope_acceleration  → Second derivative for early trend entry

Signal processing

calculate_vwap_bands    → Volume-weighted price with σ-bands
calculate_momentum_score → Normalized momentum in [-1, +1]
rolling_kurtosis        → Fat-tail detection (excess kurtosis)

Statistical tests

hurst_exponent          → R/S analysis: trending vs mean-reverting
cusum_test              → Page's CUSUM: structural break detection
volatility_compression  → Range squeeze (short/long vol ratio)

Extended

particle_price_variance    → Weighted variance of particle cloud
ess_and_uncertainty_margin → Combined ESS + regime dominance
adaptive_vwap_sigma        → Kurtosis-adapted VWAP band width

The math that matters

Particle propagation (the regime-specific part)

Each particle i carries state [xᵢ, vᵢ] (log-price, velocity) and regime rᵢ ∈ {0, 1, 2}.

RANGE (r=0):

xᵢ' = xᵢ + vᵢ·dt + σ_pos[0]·√dt·εₓ
vᵢ' = 0.5·vᵢ + σ_vel[0]·√dt·εᵥ

The 0.5·vᵢ term pulls velocity toward zero — mean reversion.

TREND (r=1):

xᵢ' = xᵢ + vᵢ·dt + σ_pos[1]·√dt·εₓ
vᵢ' = vᵢ + 0.3·(G·imbalance - vᵢ)·dt + σ_vel[1]·√dt·εᵥ

Velocity tracks G × imbalance — the order flow signal.

PANIC (r=2):

xᵢ' = xᵢ + vᵢ·dt + σ_pos[2]·√dt·εₓ
vᵢ' = vᵢ + σ_vel[2]·√dt·εᵥ

Pure random walk with high noise.

Bayesian weight update

After observing the actual price z:

wᵢ' = wᵢ × exp(-0.5·(z - xᵢ)²/σ²_price[rᵢ]) × exp(-0.5·(vᵢ - G·imb)²/σ²_vel[rᵢ])

Particles close to the observation get high weight. Particles far away get low weight. Normalize. Resample when weights degenerate (ESS < N/2).

Why log-price space?

All operations use log(price). This makes the filter scale-invariant — the same parameters and noise levels work identically for a $0.50 penny stock and a $50,000 Bitcoin. Multiplicative price noise becomes additive in log space.

Engineering decisions

Rayon only where it matters

predict_particles is the only parallelized function. It's O(N) with substantial per-particle computation (regime branching, noise injection). The other 18 functions are either memory-bound (weight updates, resampling) or operate on small arrays (Kalman 2×2 matrices, signal windows).

Adding rayon to memory-bound functions would increase latency from thread pool overhead.

No internal randomness

The library takes pre-generated random arrays as input. This gives the caller:

Full reproducibility (same seed = same output)
Choice of RNG (numpy default, PCG64, whatever)
Ability to inject structured noise for testing

Numerical stability

Weight normalization: +1e-300 guard prevents underflow to zero
ESS denominator: +1e-12 prevents division by zero
Kalman covariance: symmetrized after every predict and update step
dt guard: max(dt, 1e-8).sqrt() prevents noise explosion at tiny timesteps

Hurst exponent — my favorite function

The Hurst exponent tells you if a price series is:

H > 0.5: Trending (persistent — ups followed by ups)
H = 0.5: Random walk (no memory)
H < 0.5: Mean-reverting (anti-persistent — ups followed by downs)

Computed via R/S (Rescaled Range) analysis:

For each window size n in [min_window, max_window]:
- Split series into blocks of size n
- For each block: compute range of cumulative deviations from mean, divide by standard deviation
- Average the R/S values
Fit log(R/S) = H × log(n) + c via least squares
H is the slope

This single number tells you whether to use trend-following or mean-reversion strategies. Combined with the particle filter's regime probabilities, you get a multi-scale view of market behavior.

Before and after

Dimension	Python + Numba	Rust + PyO3
Cold start (19 functions)	30-90s JIT warmup	Zero
GIL	Held during all compute	Released
Parallelism	prange (limited)	Rayon work-stealing
Memory safety	Runtime bounds checks	Compile-time guarantees
Dependency weight	~150 MB	~2 MB single .so
Reproducibility	JIT varies by LLVM version	Deterministic binary

Try it

pip install particle-filter-rs

import numpy as np
from particle_filter_rs import (
    predict_particles, update_weights, systematic_resample,
    effective_sample_size, estimate, transition_regimes,
    kalman_update, hurst_exponent, cusum_test,
)

# Initialize 500 particles in log-price space
N = 500
particles = np.column_stack([
    np.full(N, np.log(100.0)),  # log-price
    np.zeros(N),                 # velocity
])
regimes = np.zeros(N, dtype=np.int64)
weights = np.full(N, 1.0 / N)

# One filter step
rng = np.random.default_rng(42)
particles = predict_particles(
    particles, regimes,
    np.array([0.001, 0.002, 0.005]),  # process noise per regime
    np.array([0.01, 0.02, 0.05]),
    imbalance=0.3, dt=1.0, vel_gain=0.1,
    random_pos=rng.standard_normal(N),
    random_vel=rng.standard_normal(N),
)

GitHub: https://github.com/RMANOV/particle-filter-rs

What I learned

Rayon's overhead is real — for memory-bound functions, single-threaded Rust beats rayon-parallelized Rust
Log-space is non-negotiable — a particle filter in linear price space needs different noise parameters for every asset. Log-space is universal.
Deterministic filters are testable filters — external RNG makes every test reproducible, every bug reproducible
The Kalman complement matters — particle filters alone give noisy estimates. A parallel Kalman provides smooth baseline + confidence intervals. Best of both worlds.
60 years of math still works — Kálmán's 1960 insight (recursive Bayesian estimation) is alive in every function in this library

Built with Rust, PyO3 0.27, rayon 1.10, and respect for 60 years of state estimation research.