Forem: Bill Tu

Completing the Picture: Adding Memory Diagnostics to a CPU Profiler

Bill Tu — Mon, 13 Apr 2026 06:38:56 +0000

loop-detective started as a CPU profiler. It told you which functions were blocking the event loop. Then we added I/O tracking — which network calls were slow. But there was always a gap in the story.

The analyzer would detect gc-pressure — garbage collection consuming 8% of CPU time — and suggest "Reduce object allocations. Reuse buffers. Check for memory leaks." Good advice. But then what? The user knows GC is a problem, but has no way to see what's actually in memory. They'd have to switch to a completely different tool, reconnect to the process, and start over.

With v2.1.0, loop-detective can now capture heap snapshots and track memory usage. The diagnostic workflow stays in one tool, from "something is slow" to "here's the object causing the memory leak."

The Two New Capabilities

--heap-stats: The Quick Check

loop-detective 12345 --heap-stats -d 30

This samples process.memoryUsage() before and after the profiling period:

  Memory (before profiling)
    RSS: 85.2MB  Heap: 42.1MB / 65.0MB  External: 1.2MB
  Memory (after profiling)
    RSS: 92.8MB  Heap: 58.3MB / 75.0MB  External: 1.2MB
  Memory delta
    Heap: +16.2MB  RSS: +7.6MB

Four numbers tell you a lot:

RSS (Resident Set Size): total memory the OS has allocated to the process
Heap Used / Heap Total: V8's JavaScript heap — where your objects live
External: memory used by C++ objects bound to JavaScript (Buffers, etc.)

The delta is color-coded: green for <1MB growth (normal), yellow for 1-10MB (worth investigating), red for >10MB (likely a leak or large allocation during the profiling window).

This is the "should I worry?" check. If heap grew 16MB during a 30-second profile, something is allocating a lot of objects. If it grew 0.2MB, memory is probably fine and the performance issue is elsewhere.

--heap-snapshot: The Deep Dive

loop-detective 12345 --heap-snapshot ./heap.heapsnapshot

This captures a full V8 heap snapshot — a complete inventory of every JavaScript object in memory, their sizes, and their reference chains. The .heapsnapshot file can be loaded in Chrome DevTools:

Open Chrome DevTools → Memory tab
Click "Load" and select the file
Explore:
- Summary view: objects grouped by constructor, sorted by retained size
- Comparison view: diff two snapshots to find what grew
- Containment view: see the reference chain from GC roots to any object
- Statistics: pie chart of memory by type

The snapshot is captured after CPU profiling completes, so you get both the CPU analysis and the memory snapshot from the same session.

How It Works Under the Hood

Heap Stats

The simplest possible implementation. We use Runtime.evaluate to call process.memoryUsage() in the target process:

async getHeapStats() {
  const result = await this.inspector.send('Runtime.evaluate', {
    expression: `(function(){
      const m = process.memoryUsage();
      return {
        rss: m.rss,
        heapTotal: m.heapTotal,
        heapUsed: m.heapUsed,
        external: m.external,
        arrayBuffers: m.arrayBuffers || 0
      };
    })()`,
    returnByValue: true,
  });
  return result.result?.value || null;
}

We call this twice — once before profiling starts, once after it ends — and emit both values:

async _singleRun() {
  let heapBefore = null;
  try { heapBefore = await this.getHeapStats(); } catch {}

  // ... profiling happens here ...

  let heapAfter = null;
  try { heapAfter = await this.getHeapStats(); } catch {}

  if (heapBefore || heapAfter) {
    this.emit('heapStats', { before: heapBefore, after: heapAfter });
  }
}

Both calls are wrapped in try/catch because heap stats are non-essential — if they fail (target exited, inspector disconnected), the CPU profile and I/O data are still valid.

Heap Snapshot

The CDP HeapProfiler domain handles snapshot capture. The snapshot is streamed in chunks via events:

async captureHeapSnapshot() {
  let chunks = [];
  const onChunk = (msg) => {
    if (msg.method === 'HeapProfiler.addHeapSnapshotChunk') {
      chunks.push(msg.params.chunk);
    }
  };
  this.inspector.on('event', onChunk);

  try {
    await this.inspector.send('HeapProfiler.enable');
    await this.inspector.send('HeapProfiler.takeHeapSnapshot', {
      reportProgress: false
    });
    await this.inspector.send('HeapProfiler.disable');
  } finally {
    this.inspector.removeListener('event', onChunk);
  }

  return chunks.join('');
}

The V8 heap snapshot format is JSON, but it can be large — 10MB to 200MB+ depending on heap size. The CDP protocol streams it in chunks to avoid sending one massive message. We collect all chunks and join them into the final string.

A few important details:

reportProgress: false — we skip progress events to keep the implementation simple. For very large heaps, you could listen for HeapProfiler.reportHeapSnapshotProgress to show a progress bar.
The snapshot is captured after CPU profiling, not during. Taking a heap snapshot pauses the target process briefly (it needs to walk the entire object graph), so we don't want it interfering with the CPU profile.
The finally block ensures we remove the event listener even if the snapshot fails.

The Diagnostic Workflow

Here's how the pieces fit together in a real debugging session:

Step 1: Quick diagnosis

loop-detective 12345 -d 10

The report shows gc-pressure — GC is consuming 8% of CPU. The event loop isn't blocked by user code, but GC pauses are adding latency.

Step 2: Confirm with memory stats

loop-detective 12345 --heap-stats -d 30

  Memory delta
    Heap: +24.3MB  RSS: +18.1MB

Red flag. The heap grew 24MB in 30 seconds. Something is allocating objects faster than GC can collect them.

Step 3: Capture the evidence

loop-detective 12345 --heap-snapshot ./before-fix.heapsnapshot --heap-stats

Open before-fix.heapsnapshot in Chrome DevTools. The Summary view shows 500,000 Object instances retained by a Map in /app/cache.js. The cache has no eviction policy — it grows forever.

Step 4: Fix and verify

After adding an LRU eviction policy to the cache:

loop-detective 12345 --heap-stats -d 30

  Memory delta
    Heap: +0.8MB  RSS: +0.3MB

Green. Memory is stable. The gc-pressure pattern is gone from the CPU profile.

When to Use Each

Situation	Use
Quick health check	`--heap-stats`
Analyzer reports `gc-pressure`	`--heap-stats` first, then `--heap-snapshot` if growth is high
Suspected memory leak	`--heap-snapshot` (capture two snapshots minutes apart, compare in DevTools)
Full diagnostic session	`--heap-stats --save-profile ./cpu.cpuprofile --html report.html`
Memory-only investigation	`--heap-snapshot ./heap.heapsnapshot --no-io -d 1` (minimal profiling, just get the snapshot)

Performance Impact

--heap-stats: negligible. Two process.memoryUsage() calls, each taking <1ms.
--heap-snapshot: moderate. The V8 heap walker pauses the target process while it traverses the object graph. For a 100MB heap, this takes 1-5 seconds. For a 1GB heap, it could take 10-30 seconds. The process is unresponsive during this time.

This is why the snapshot is captured after profiling, not during. And it's why it's opt-in (--heap-snapshot) rather than default.

Programmatic API

For custom tooling:

const { Detective } = require('node-loop-detective');
const fs = require('fs');

const detective = new Detective({ pid: 12345, duration: 10000 });

// Memory stats before/after profiling
detective.on('heapStats', (data) => {
  const growth = data.after.heapUsed - data.before.heapUsed;
  if (growth > 50 * 1024 * 1024) {
    alert('Heap grew ' + (growth / 1024 / 1024).toFixed(0) + 'MB during profiling!');
  }
});

detective.on('profile', async (analysis) => {
  // If GC pressure detected, auto-capture heap snapshot
  const hasGcPressure = analysis.blockingPatterns.some(p => p.type === 'gc-pressure');
  if (hasGcPressure) {
    const snapshot = await detective.captureHeapSnapshot();
    fs.writeFileSync('auto-heap-' + Date.now() + '.heapsnapshot', snapshot);
  }
});

await detective.start();

This pattern — auto-capture a heap snapshot when GC pressure is detected — is particularly powerful for automated monitoring. You get the snapshot exactly when it matters, without human intervention.

What's Next

The heap snapshot is a point-in-time view. It tells you what's in memory right now, but not how it got there. Future improvements could include:

Allocation tracking via HeapProfiler.startTrackingHeapObjects — shows which functions are allocating the most objects
Memory timeline — sample process.memoryUsage() every second during profiling and show a trend line in the HTML report
Automatic leak detection — compare heap stats at the start and end of each watch cycle, flag monotonic growth

For now, the combination of --heap-stats (quick check) and --heap-snapshot (deep dive) covers the most common memory debugging workflow.

Try It

npm install -g node-loop-detective@2.1.0

# Quick memory check alongside CPU profiling
loop-detective <pid> --heap-stats

# Full memory diagnosis
loop-detective <pid> --heap-stats --heap-snapshot ./heap.heapsnapshot

# Minimal: just grab the snapshot
loop-detective <pid> --heap-snapshot ./heap.heapsnapshot --no-io -d 1

Source: github.com/iwtxokhtd83/node-loop-detective

CPU tells you what's slow. I/O tells you what's waiting. Memory tells you what's accumulating. Now you have all three in one tool.

Shareable Diagnostics: Generating HTML Reports From Production Profiling

Bill Tu — Fri, 10 Apr 2026 07:22:11 +0000

You've just diagnosed a production performance issue. loop-detective told you that processPayload at /app/handlers/payment.js:127 consumed 54% of CPU, that three HTTP calls to the payment gateway averaged 1.8 seconds, and that the event loop lagged 12 times during the 30-second capture.

Now you need to share this with your team. You copy-paste the terminal output into Slack. The ANSI color codes turn into garbage. The bar charts become meaningless characters. The carefully formatted report becomes an unreadable wall of text.

This is why we built HTML report output for node-loop-detective v2.0.0.

loop-detective 12345 --html report.html

One flag. One file. Open it in any browser. Share it anywhere.

The Problem With Terminal Output

Terminal output is perfect for the person running the diagnostic. It's immediate, it's colorful, it's right there. But it has three fundamental limitations:

It doesn't travel well. ANSI escape codes render as [31m in Slack, email, Jira, and most text editors. You lose all formatting.
It's not interactive. You can't collapse a call stack you don't care about. You can't hover over a lag event to see its timestamp. You can't sort the function table by percentage.
It's ephemeral. Once the terminal scrolls past, it's gone. You can redirect to a file, but then you have a plain text file with escape codes.

JSON output (--json) solves the data portability problem but creates a readability problem. Nobody wants to read a 500-line JSON blob in a Slack thread.

What the HTML Report Looks Like

The report is a single self-contained HTML file with a dark theme inspired by GitHub's UI. No external CSS, no JavaScript CDN links, no images to load. It works offline, it works on any device, and it's about 15-20KB for a typical report.

It has six sections:

Summary Cards

Five cards at the top showing the key numbers at a glance: profiling duration, CPU sample count, number of hot functions, lag event count, and slow I/O operation count. You can tell in one second whether this report is interesting.

Diagnosis

Each detected pattern gets a colored severity badge (HIGH in red, MED in yellow, LOW in green), a description, the code location, and a suggested fix. This is the same information as the terminal output, but formatted for readability.

CPU-Heavy Functions Table

A sortable table with visual percentage bars. The bars are color-coded: red for >50% CPU, yellow for >20%, green for the rest. Each row shows the function name, a visual bar, self time in milliseconds, and the file location.

Collapsible Call Stacks

The top blocking functions have expandable call stacks. Click to expand, click again to collapse. This keeps the report compact by default while making the full detail available on demand. The target function is highlighted in yellow.

Slow I/O Summary

Grouped by type (HTTP, FETCH, DNS, TCP), then by target. Each group shows call count, total duration, average, and max. Caller stack traces are shown inline. Error counts get a red badge.

Event Loop Lag Timeline

A visual timeline of lag events represented as colored dots. The dot size scales with lag severity. Hover over any dot to see the exact lag duration and timestamp. Below the timeline, a table aggregates lag events by code location.

Design Decisions

Why self-contained?

The report must work when attached to a Jira ticket, emailed to a colleague, or opened six months later during a post-mortem. External dependencies (CDN-hosted CSS, JavaScript libraries, web fonts) break in all these scenarios. Corporate firewalls block CDNs. Offline access fails. Links rot.

Every byte of CSS and JavaScript is inline in the HTML file. You can put it on a USB drive and it works.

Why dark theme?

Developers spend their days looking at dark-themed editors and terminals. A bright white report is jarring. The GitHub-inspired dark theme (#0d1117 background, #c9d1d9 text) feels native to the developer workflow.

Why collapsible call stacks?

A report with 5 expanded call stacks, each 10-15 frames deep, is overwhelming. Most of the time you only care about one or two stacks. Collapsible sections let you scan the targets quickly and expand only what's relevant.

The implementation is vanilla JavaScript — no framework, no build step:

document.querySelectorAll('.collapsible').forEach(el => {
  el.addEventListener('click', () => {
    el.classList.toggle('open');
    el.nextElementSibling.classList.toggle('show');
  });
});

Why not a full dashboard?

We considered generating a report with charts (line graphs for lag over time, pie charts for CPU distribution). But charts require a charting library (Chart.js, D3, etc.), which would either bloat the file or require a CDN. The current approach — colored bars, sized dots, tables — conveys the same information with pure HTML and CSS.

If you need charts, use --json and feed the data to your preferred visualization tool.

How It Works

The HTML report generator (src/html-report.js) is a pure function that takes analysis data and returns an HTML string:

function generateHtmlReport(data) {
  const { analysis, lagEvents, slowIOEvents, config, timestamp } = data;
  return `<!DOCTYPE html>
<html>
<head>
  <style>/* all CSS inline */</style>
</head>
<body>
  ${renderSummary(analysis.summary, lagEvents, slowIOEvents)}
  ${renderPatterns(analysis.blockingPatterns)}
  ${renderHeavyFunctions(analysis.heavyFunctions)}
  ${renderCallStacks(analysis.callStacks)}
  ${renderSlowIO(slowIOEvents)}
  ${renderLagEvents(lagEvents)}
  <script>/* collapsible toggle */</script>
</body>
</html>`;
}

Each section is rendered by a dedicated function. All user-provided strings (function names, file paths, error messages) are HTML-escaped to prevent XSS — even though the report is generated locally, it's good practice.

In the CLI, the events are captured before the reporter clears them:

detective.on('profile', (analysis, rawProfile) => {
  // Capture before onProfile clears the arrays
  const capturedLags = [...reporter.lagEvents];
  const capturedIO = [...reporter.slowIOEvents];

  reporter.onProfile(analysis); // prints to terminal + clears arrays

  if (config.html) {
    const html = generateHtmlReport({
      analysis, lagEvents: capturedLags, slowIOEvents: capturedIO,
      config, timestamp: analysis.timestamp,
    });
    fs.writeFileSync(path.resolve(config.html), html);
  }
});

Combining Output Formats

The HTML report works alongside all other output options:

# Terminal output + HTML report
loop-detective 12345 --html report.html

# Terminal + HTML + CPU profile for flame graphs
loop-detective 12345 -d 60 --html report.html --save-profile profile.cpuprofile

# JSON + HTML (JSON goes to stdout, HTML to file)
loop-detective 12345 --json --html report.html > data.json

Each format serves a different purpose:

Terminal: immediate diagnosis while you're at the keyboard
HTML: sharing with team, attaching to tickets, archiving
JSON: feeding to monitoring systems, custom analysis scripts
.cpuprofile: deep visual analysis in Chrome DevTools or speedscope

Programmatic API

The HTML generator is exported for use in custom tooling:

const { generateHtmlReport } = require('node-loop-detective');

// Generate from your own data
const html = generateHtmlReport({
  analysis: myAnalysisResult,
  lagEvents: myLagEvents,
  slowIOEvents: mySlowIOEvents,
  config: { duration: 30000, threshold: 50 },
  timestamp: Date.now(),
});

// Serve it from an Express endpoint
app.get('/debug/report', (req, res) => {
  res.type('html').send(html);
});

// Or save to S3 for the incident timeline
await s3.putObject({
  Bucket: 'incident-reports',
  Key: `diagnostics/${Date.now()}.html`,
  Body: html,
  ContentType: 'text/html',
});

Real-World Usage

Incident Post-Mortems

During an incident, run loop-detective with --html:

loop-detective 12345 -d 60 --html incident-2025-03-15.html

Attach the HTML file to the post-mortem document. Six months later, anyone can open it and see exactly what was happening: which functions were hot, which I/O was slow, when the lag events occurred.

Code Review Evidence

Found a performance regression? Profile the old and new versions, generate HTML reports for both, and attach them to the pull request. The reviewer can open both files side by side and compare.

Automated Reporting

In a CI/CD pipeline or scheduled job:

loop-detective --port 9229 -d 30 --html /reports/daily-$(date +%Y%m%d).html --json > /reports/daily-$(date +%Y%m%d).json

Build a simple file server over the reports directory and you have a performance history dashboard.

Try It

npm install -g node-loop-detective@2.0.0

loop-detective <pid> --html report.html

Open report.html in your browser. Share it with your team. Attach it to the ticket. The diagnostic data is no longer trapped in your terminal.

Source: github.com/iwtxokhtd83/node-loop-detective

Profiling the Threads You Forgot About: Worker Thread Diagnostics in Node.js

Bill Tu — Fri, 10 Apr 2026 06:57:05 +0000

You've profiled your Node.js server. The main thread looks healthy. CPU usage is low. Event loop lag is minimal. But users are still reporting slow responses.

Then you remember: half your request processing happens in worker threads.

Node.js worker_threads have their own V8 instances, their own event loops, and their own performance characteristics. A worker thread can be completely blocked — running a tight loop, parsing a massive JSON payload, executing a catastrophic regex — and the main thread's profiler won't see any of it.

With node-loop-detective v1.9.0, you can now list all inspector targets and profile individual worker threads. Two new flags: --list-targets and --target.

The Invisible Threads

Worker threads are increasingly common in Node.js applications. They're used for:

CPU-intensive computation (image processing, data transformation, ML inference)
Parallel request processing in frameworks like Piscina
Background jobs (report generation, data export, batch processing)
Offloading blocking operations from the main thread

The irony is that worker threads are often created specifically to handle heavy work — which makes them the most likely threads to have performance problems. But until now, most diagnostic tools only profiled the main thread.

Here's what happens when you profile a Node.js app that uses worker threads:

loop-detective 12345 -d 30

────────────────────────────────────────────────────────────
  Event Loop Detective Report
────────────────────────────────────────────────────────────
  Duration:  30012ms
  Samples:   13521
  Hot funcs: 3

  Diagnosis
────────────────────────────────────────────────────────────
   LOW  healthy
        No obvious event loop blocking patterns detected
        → Try profiling for a longer duration or during peak load

The main thread is healthy. But the workers are drowning. You just can't see them.

How the V8 Inspector Handles Threads

When Node.js opens its inspector (via --inspect or SIGUSR1), it exposes a /json/list HTTP endpoint that returns all available debugging targets. Each target is a separate V8 isolate with its own WebSocket URL:

[
  {
    "id": "1a2b3c",
    "title": "Main thread",
    "url": "file:///app/server.js",
    "webSocketDebuggerUrl": "ws://127.0.0.1:9229/1a2b3c"
  },
  {
    "id": "4d5e6f",
    "title": "Worker #1",
    "url": "file:///app/worker.js",
    "webSocketDebuggerUrl": "ws://127.0.0.1:9229/4d5e6f"
  },
  {
    "id": "7g8h9i",
    "title": "Worker #2",
    "url": "file:///app/worker.js",
    "webSocketDebuggerUrl": "ws://127.0.0.1:9229/7g8h9i"
  }
]

Previously, loop-detective always connected to targets[0] — the main thread. The worker threads were right there in the list, but we never looked at them.

Discovering Targets

The first step is knowing what's available:

loop-detective --port 9229 --list-targets

  Available inspector targets:

  [0] Main thread
      file:///app/server.js
  [1] Worker #1
      file:///app/worker.js
  [2] Worker #2
      file:///app/worker.js

  Use --target <index> to connect to a specific target.

Each target has an index, a title, and a URL (the script that created it). The main thread is always index 0.

For automation, --json gives you structured output:

loop-detective --port 9229 --list-targets --json

[
  { "index": 0, "id": "1a2b3c", "title": "Main thread", "url": "file:///app/server.js" },
  { "index": 1, "id": "4d5e6f", "title": "Worker #1", "url": "file:///app/worker.js" },
  { "index": 2, "id": "7g8h9i", "title": "Worker #2", "url": "file:///app/worker.js" }
]

Profiling a Worker Thread

Once you know the target index, profile it like any other process:

loop-detective --port 9229 --target 1 -d 30

✔ Connected to Node.js process
  Profiling for 30s with 50ms lag threshold...

⚠ Event loop lag: 1245ms at 2025-03-15T14:23:45.123Z

────────────────────────────────────────────────────────────
  Event Loop Detective Report
────────────────────────────────────────────────────────────
  Duration:  30008ms
  Samples:   8234
  Hot funcs: 7

  Diagnosis
────────────────────────────────────────────────────────────
   HIGH  cpu-hog
         Function "transformData" consumed 72.1% of CPU time (21630ms)
         at /app/worker.js:45
         → Consider breaking this into smaller async chunks

   1. transformData
      ██████████████████░░ 21630ms (72.1%)
      /app/worker.js:45:1

There it is. Worker #1 has a transformData function consuming 72% of CPU. The main thread never saw this because the work was offloaded to the worker.

A Real-World Debugging Session

Here's a typical workflow for diagnosing a worker thread issue:

# Step 1: Profile the main thread — looks healthy
loop-detective 12345 -d 10
# Result: "healthy" — no blocking detected

# Step 2: List all targets
loop-detective --port 9229 --list-targets
# Shows: [0] Main, [1] Worker #1, [2] Worker #2, [3] Worker #3

# Step 3: Profile each worker
loop-detective --port 9229 --target 1 -d 10
# Worker #1: healthy

loop-detective --port 9229 --target 2 -d 10
# Worker #2: cpu-hog on transformData — 72% CPU!

loop-detective --port 9229 --target 3 -d 10
# Worker #3: healthy

# Step 4: Deep dive on Worker #2
loop-detective --port 9229 --target 2 -d 60 --save-profile ./worker2.cpuprofile
# Open worker2.cpuprofile in Chrome DevTools for flame graph

In a script:

#!/bin/bash
# Profile all worker threads
TARGETS=$(loop-detective --port 9229 --list-targets --json)
COUNT=$(echo "$TARGETS" | node -e "process.stdin.on('data',d=>console.log(JSON.parse(d).length))")

for i in $(seq 0 $((COUNT - 1))); do
  echo "=== Profiling target $i ==="
  loop-detective --port 9229 --target $i -d 10 --json > "target-${i}.json"
done

How It Works Internally

The implementation touches two layers:

Inspector: Target Selection

The Inspector class now accepts a targetIndex parameter and uses it to select the WebSocket URL:

class Inspector {
  constructor({ host, port, targetIndex = 0 }) {
    this.targetIndex = targetIndex;
    // ...
  }

  async getWebSocketUrl() {
    const targets = await this.getTargets();
    if (this.targetIndex >= targets.length) {
      throw new Error('Target index ' + this.targetIndex + ' out of range');
    }
    return targets[this.targetIndex].webSocketDebuggerUrl;
  }
}

Detective: Target Discovery

The Detective class exposes a listTargets() method that uses the retry logic (same exponential backoff as regular connections) to handle the case where the inspector is still starting:

async listTargets() {
  this._activateInspector();
  const port = this._getInspectorPort();
  const host = this.config.inspectorHost || '127.0.0.1';

  // Retry with backoff (inspector may still be starting)
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      const inspector = new Inspector({ host, port });
      return await inspector.getTargets();
    } catch (err) {
      if (attempt === maxRetries) throw err;
      await this._sleep(delay);
    }
  }
}

What Works and What Doesn't

Everything that works on the main thread works on worker threads:

✅ CPU profiling and heavy function detection
✅ Event loop lag detection
✅ Blocking pattern analysis (cpu-hog, json-heavy, regex, gc, sync-io, crypto)
✅ --save-profile for flame graphs
✅ --watch for continuous monitoring
✅ --json for structured output

I/O tracking (http, fetch, dns, net patches) also works, but with a caveat: worker threads typically don't make HTTP requests directly. They receive data from the main thread via postMessage and send results back. If a worker does make network calls, the I/O tracker will catch them.

When Workers Don't Show Up

A few situations where --list-targets might not show your workers:

Workers created after inspector activation. If a worker is spawned after you send SIGUSR1, it may not appear in the target list. Re-run --list-targets to refresh.
Workers that have already exited. Short-lived workers (process a task, then terminate) may not be visible when you list targets. Use --watch on the main thread to catch the pattern, then target long-lived workers.
Workers without inspector support. Workers created with { execArgv: [] } explicitly disable the inspector. They won't appear in the target list.

Try It

npm install -g node-loop-detective@1.9.0

# List all targets
loop-detective --port 9229 --list-targets

# Profile a worker thread
loop-detective --port 9229 --target 1 -d 30

The main thread is just one thread. In modern Node.js applications, the interesting performance problems are often hiding in the workers.

Source: github.com/iwtxokhtd83/node-loop-detective

Why One Second Wasn't Enough: Adding Retry Logic to a Diagnostic Tool

Bill Tu — Fri, 10 Apr 2026 06:25:27 +0000

For the first seven releases of node-loop-detective, connecting to a target process looked like this:

async _findInspectorPort() {
  await this._sleep(1000); // wait 1 second
  return 9229;
}

Send SIGUSR1. Wait one second. Try to connect. If it fails, give up.

This worked on our development machines. It worked in CI. It worked on lightly loaded staging servers. Then users started running it in production — on machines with 95% CPU utilization, inside containers with throttled resources, on servers handling 50,000 requests per second.

One second wasn't enough.

What Happens After SIGUSR1

When you send SIGUSR1 to a Node.js process, it triggers the V8 Inspector to start. This involves:

The signal handler fires on the next event loop tick
Node.js initializes the inspector agent
A TCP server starts listening on port 9229
The /json/list HTTP endpoint becomes available
WebSocket connections are accepted

On an idle machine, this takes 10-50 milliseconds. On a machine where the event loop is already blocked (which is exactly when you're trying to diagnose it), step 1 might not happen for several seconds. The signal is queued, but the event loop has to process it.

This creates a paradox: the more you need the tool, the longer it takes to connect.

The Failure Mode

With a fixed 1-second wait, users on loaded systems saw:

✖ Error: Cannot connect to inspector at 127.0.0.1:9229.
  Is the Node.js inspector active? (connect ECONNREFUSED 127.0.0.1:9229)

The inspector was starting. It just wasn't ready yet. The user would run the command again, and it would work — because by then the inspector had finished initializing. But "run it again" is not an acceptable UX for a production diagnostic tool.

The Fix: Exponential Backoff

The solution is retry with exponential backoff. Instead of one attempt after a fixed delay, we make up to 5 attempts with increasing wait times:

Attempt	Delay Before	Cumulative Wait
1	500ms	500ms
2	1000ms	1.5s
3	2000ms	3.5s
4	4000ms	7.5s
5	4000ms	11.5s

The delay doubles each time, capped at 4 seconds. Total maximum wait before giving up: about 11.5 seconds.

async _connectWithRetry(host, port) {
  const maxRetries = this.config.inspectorPort ? 1 : 5;
  const baseDelay = 500;
  const maxDelay = 4000;

  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      this.inspector = new Inspector({ host, port });
      // ... set up event listeners
      await this.inspector.connect();
      return; // success
    } catch (err) {
      if (attempt === maxRetries) {
        throw new Error(
          `Failed to connect to inspector at ${host}:${port} ` +
          `after ${maxRetries} attempts. Last error: ${err.message}`
        );
      }
      // Clean up failed inspector
      if (this.inspector) {
        this.inspector.removeAllListeners();
        this.inspector = null;
      }
      const delay = Math.min(baseDelay * Math.pow(2, attempt - 1), maxDelay);
      this.emit('retry', { attempt, maxRetries, delay, error: err.message });
      await this._sleep(delay);
    }
  }
}

Design Decisions

Why not just wait longer?

We could have changed _sleep(1000) to _sleep(5000). Problem solved, right?

No. On a fast machine, you'd wait 5 seconds every time for something that takes 50ms. The whole point of a diagnostic tool is speed — you're in the middle of an incident, every second counts. Exponential backoff gives you the best of both worlds: fast connection on healthy systems, patient retry on loaded ones.

On a typical machine, the first attempt at 500ms succeeds. The user never sees a retry. On a loaded machine, the second or third attempt usually succeeds. The user sees one or two retry messages and then gets connected.

Why 5 attempts?

Five attempts with our backoff schedule gives a maximum wait of ~11.5 seconds. This is long enough to handle even severely loaded systems (where the event loop might be blocked for 5-10 seconds), but short enough that a genuine failure (wrong PID, process already exited) doesn't leave the user waiting forever.

Why skip retry for --port?

When the user specifies --port, they're connecting to an inspector that's already open. There's no SIGUSR1, no startup delay. If the connection fails, it's because the inspector isn't there — retrying won't help.

const maxRetries = this.config.inspectorPort ? 1 : 5;

One attempt for --port. Five attempts for PID-based connections.

Why emit a retry event?

The CLI shows retry progress:

  Connecting to inspector... attempt 1/5 (retry in 500ms)
  Connecting to inspector... attempt 2/5 (retry in 1000ms)
✔ Connected to Node.js process

Without this feedback, the user would see nothing for up to 11 seconds. In an incident, silence is anxiety. The retry messages tell the user "I'm working on it, the system is just slow."

For the programmatic API, the retry event lets integrators log, alert, or implement their own timeout logic:

detective.on('retry', (data) => {
  logger.warn('Inspector connection retry', {
    attempt: data.attempt,
    maxRetries: data.maxRetries,
    delay: data.delay,
    error: data.error,
  });
});

The Cleanup Problem

There's a subtle issue with retry: each failed attempt creates an Inspector instance with event listeners. If we don't clean up, we accumulate orphaned listeners:

// Each attempt creates a new Inspector
this.inspector = new Inspector({ host, port });
this.inspector.on('disconnected', () => { ... });

// If connect() fails, we need to clean up before retrying
if (this.inspector) {
  this.inspector.removeAllListeners();
  this.inspector = null;
}

Without removeAllListeners(), the disconnected handler from attempt 1 would still be attached when attempt 2 creates a new Inspector. If the target later exits, both handlers would fire, causing duplicate targetExit events.

What Users See Now

On a fast machine (most cases):

✔ Connected to Node.js process
  Profiling for 10s with 50ms lag threshold...

No retry messages. Connection happens on the first attempt at 500ms — faster than the old fixed 1-second wait.

On a loaded machine:

  Connecting to inspector... attempt 1/5 (retry in 500ms)
  Connecting to inspector... attempt 2/5 (retry in 1000ms)
✔ Connected to Node.js process
  Profiling for 10s with 50ms lag threshold...

Two retries, then success. Total wait: about 1.5 seconds.

On a very loaded machine or wrong PID:

  Connecting to inspector... attempt 1/5 (retry in 500ms)
  Connecting to inspector... attempt 2/5 (retry in 1000ms)
  Connecting to inspector... attempt 3/5 (retry in 2000ms)
  Connecting to inspector... attempt 4/5 (retry in 4000ms)

✖ Error: Failed to connect to inspector at 127.0.0.1:9229 after 5 attempts.
  Last error: Cannot connect to inspector — Is the Node.js inspector active?

Clear feedback at every step. The final error message includes the attempt count and the specific failure reason.

The Broader Pattern

Retry with exponential backoff is one of the most well-known patterns in distributed systems. But it's easy to forget when building CLI tools. We think of CLI tools as synchronous, immediate, local. "Run command, get result."

But a diagnostic tool that connects to another process over a network protocol (even localhost) is a distributed system. The target process is an independent actor with its own timing. The inspector startup is an asynchronous operation we don't control. Network connections can fail transiently.

The same principles apply:

Don't assume the first attempt will succeed. Especially under load.
Back off exponentially. Linear retry hammers a system that's already struggling.
Cap the backoff. Waiting 32 seconds between attempts is too long for an interactive tool.
Give feedback. The user needs to know the tool is working, not frozen.
Know when to stop. Five attempts is enough. If the inspector isn't up after 11 seconds, something else is wrong.

Try It

npm install -g node-loop-detective@1.8.0

loop-detective <pid>

On most machines, you won't notice the change — it just connects slightly faster than before (500ms vs 1000ms). On loaded machines, it now works where it previously failed.

Source: github.com/iwtxokhtd83/node-loop-detective

Giving Kafka-Style Topics the Routing Power of RabbitMQ

Bill Tu — Fri, 10 Apr 2026 06:11:56 +0000

One of the most common complaints about Kafka is its routing model. You have topics. You have partitions. That's it. If you want to route order events differently based on region, priority, or event type, you're stuck with two bad options: create a separate topic for every permutation (orders-us-high, orders-eu-low, ...) or have every consumer receive everything and filter client-side.

RabbitMQ solved this years ago with its exchange model. But RabbitMQ trades routing flexibility for throughput — it can't match Kafka's append-only log performance.

TitanMQ has had both systems from the start: a Kafka-style commit log for storage and RabbitMQ-style exchanges for routing. The problem was they weren't connected. The exchange implementations sat in titanmq-routing, fully tested, doing nothing. Every PRODUCE request went straight to TopicManager.append(). The routing engine was dead code.

We just fixed that. Here's how.

The Design Problem

The naive approach would be: every message goes through an exchange, always. But that breaks backward compatibility and adds latency for the common case where you just want topic-partition semantics.

The approach we chose: exchanges are opt-in and overlay the existing topic model. The broker checks if the target topic name matches a declared exchange. If it does, route through the exchange. If it doesn't, write directly to the topic. Zero overhead for users who don't use exchanges.

Producer sends to "order-events"
         │
         ▼
  ┌──────────────────┐
  │ Exchange exists?  │
  │ "order-events"   │
  └──────┬───────────┘
     yes │        no
         │         │
         ▼         ▼
  ┌────────────┐  ┌──────────────┐
  │ Route via  │  │ Direct write │
  │ exchange   │  │ to topic     │
  └────┬───────┘  └──────────────┘
       │
       ▼
  Write to each destination topic
  (fulfillment, analytics, ...)

This means existing applications that use plain topics continue to work without any changes. Exchanges are a layer you add when you need them.

ExchangeManager

The central piece is ExchangeManager, which manages the lifecycle of named exchanges:

public class ExchangeManager {

    private final ConcurrentHashMap<String, Exchange> exchanges = new ConcurrentHashMap<>();

    public void declareExchange(String name, ExchangeType type) {
        Exchange exchange = switch (type) {
            case DIRECT -> new DirectExchange(name);
            case TOPIC -> new TopicExchange(name);
            case FANOUT -> new FanoutExchange(name);
            case CONTENT_BASED -> new ContentBasedExchange(name);
        };
        exchanges.put(name, exchange);
    }

    public void bind(String exchangeName, String destinationTopic, String routingKey) {
        exchanges.get(exchangeName).bind(destinationTopic, routingKey);
    }

    public List<String> route(String exchangeName, TitanMessage message, String routingKey) {
        Exchange exchange = exchanges.get(exchangeName);
        if (exchange == null) return List.of();
        return exchange.route(message, routingKey);
    }
}

Declaration is idempotent — declaring the same exchange with the same type twice is a no-op. Declaring with a different type throws an error. This matches RabbitMQ's behavior and prevents accidental misconfiguration.

The Four Exchange Types

Direct

Exact routing key match. The simplest and fastest.

mgr.declareExchange("order-events", ExchangeType.DIRECT);
mgr.bind("order-events", "fulfillment", "order.created");
mgr.bind("order-events", "billing", "order.paid");

// Message with routingKey="order.created" → fulfillment only
// Message with routingKey="order.paid" → billing only
// Message with routingKey="order.cancelled" → nowhere (error returned)

Topic

Wildcard pattern matching. * matches one word, # matches zero or more.

mgr.declareExchange("events", ExchangeType.TOPIC);
mgr.bind("events", "us-handler", "order.us.*");
mgr.bind("events", "all-errors", "#.error");

// "order.us.created" → us-handler
// "payment.processing.error" → all-errors
// "error" → all-errors (# matches zero words too)

The pattern matching uses a recursive algorithm that handles the # wildcard's zero-or-more semantics:

private static boolean matchParts(String[] routing, int ri, String[] pattern, int pi) {
    if (ri == routing.length && pi == pattern.length) return true;
    if (pi == pattern.length) return false;

    if (pattern[pi].equals("#")) {
        if (pi == pattern.length - 1) return true;
        for (int i = ri; i <= routing.length; i++) {
            if (matchParts(routing, i, pattern, pi + 1)) return true;
        }
        return false;
    }

    if (ri == routing.length) return false;
    if (pattern[pi].equals("*") || pattern[pi].equals(routing[ri])) {
        return matchParts(routing, ri + 1, pattern, pi + 1);
    }
    return false;
}

Fanout

Broadcast to all bound destinations. Routing key is ignored.

mgr.declareExchange("notifications", ExchangeType.FANOUT);
mgr.bind("notifications", "email-service", "");
mgr.bind("notifications", "sms-service", "");
mgr.bind("notifications", "push-service", "");

// Any message → all three services

Content-Based

This is where TitanMQ goes beyond RabbitMQ. Instead of matching routing keys, it evaluates predicates against message headers.

mgr.declareExchange("smart-router", ExchangeType.CONTENT_BASED);
mgr.bind("smart-router", "us-queue", "region=us");
mgr.bind("smart-router", "eu-priority", "region=eu,priority=high");

// Message with header region=us → us-queue
// Message with headers region=eu, priority=high → eu-priority
// Message with header region=eu, priority=low → nowhere

The wire protocol uses a simple key=value,key=value format for binding rules. But programmatically, you can pass arbitrary Predicate<Map<String, String>> lambdas:

ContentBasedExchange exchange = (ContentBasedExchange) mgr.getExchange("smart-router");
exchange.bind("vip-queue", headers ->
    Integer.parseInt(headers.getOrDefault("priority", "0")) > 8 &&
    "premium".equals(headers.get("tier"))
);

Integration into the Broker

The key change is in BrokerRequestHandler.handleProduce(). The original code was a straight write:

// Before: direct write, no routing
TitanMessage message = TitanMessage.deserialize(buffer);
long offset = topicManager.append(message);

The new code checks for an exchange first:

if (exchangeManager.hasExchange(message.topic())) {
    String routingKey = message.headers().getOrDefault("routingKey", message.topic());
    List<String> destinations = exchangeManager.route(message.topic(), message, routingKey);

    for (String destTopic : destinations) {
        TitanMessage routed = TitanMessage.builder()
                .id(message.id())
                .topic(destTopic)
                .key(message.key())
                .payload(message.payload())
                .headers(message.headers())
                .timestamp(message.timestamp())
                .build();
        topicManager.append(routed);
    }
} else {
    topicManager.append(message);
}

The routing key comes from the message header routingKey. If the header isn't set, the topic name itself is used as the routing key. This keeps the producer API simple — you don't need a separate routing key parameter.

Each destination topic gets its own copy of the message with the topic field rewritten. The message ID, key, payload, headers, and timestamp are preserved. This means consumers on different destination topics see the same message content.

Wire Protocol

Three new commands were added:

Command	Code	Payload
`DECLARE_EXCHANGE`	`0x24`	`[4B nameLen][NB name][1B type]`
`BIND_EXCHANGE`	`0x25`	`[4B exchangeLen][NB exchange][4B destLen][NB dest][4B keyLen][NB key]`
`UNBIND_EXCHANGE`	`0x26`	Same as BIND

The exchange type is encoded as a single byte: 0=DIRECT, 1=TOPIC, 2=FANOUT, 3=CONTENT_BASED.

All three commands return ADMIN_RESPONSE with a success byte on completion.

What This Enables

With the routing engine live, TitanMQ can now handle patterns that previously required either multiple topics or client-side filtering:

Event-driven microservices: Declare a topic exchange for domain events. Each service binds to the patterns it cares about. The order service gets order.*, the analytics service gets # (everything), the error handler gets #.error.

Multi-region routing: Declare a content-based exchange. Bind regional queues based on the region header. Messages are routed at the broker — no wasted bandwidth sending US events to EU consumers.

Notification fanout: Declare a fanout exchange. Every notification channel (email, SMS, push) gets every message. Add a new channel by adding a binding — no producer changes needed.

Priority routing: Use content-based exchange with priority header predicates. High-priority messages go to a fast-path queue with more consumer instances. Low-priority messages go to a batch queue.

All of this on top of Kafka-style append-only logs with offset tracking, consumer groups, and replay capability. The routing happens at write time, so consumers still get the full commit log semantics — they can seek, replay, and track offsets on their destination topics.

Trade-offs

Routing adds a small amount of work per PRODUCE request: one ConcurrentHashMap.get() to check if an exchange exists, plus the routing logic itself. For direct and fanout exchanges, this is O(1). For topic exchanges, it's O(n) where n is the number of bindings (pattern matching against each binding). For content-based exchanges, it's O(n) predicate evaluations.

Messages routed to multiple destinations are written multiple times — once per destination topic. This increases disk I/O proportionally. If a fanout exchange has 10 bindings, each message is written 10 times. This is the same trade-off RabbitMQ makes, and it's the correct one: consumers on different topics need independent offset tracking and retention.

The routing engine is entirely in-memory. Exchange declarations and bindings are not persisted to disk yet. A broker restart loses all exchange configuration. This is a known limitation tracked in a separate issue — the fix will persist exchange state to the Raft log so it survives restarts and is replicated across the cluster.

GitHub: https://github.com/iwtxokhtd83/TitanMQ
Release: v1.2.0
Issue: #10

When the Patient Dies on the Table: Handling Target Process Exit During Profiling

Bill Tu — Thu, 09 Apr 2026 14:03:43 +0000

You're profiling a Node.js process that's been crashing intermittently. You attach loop-detective, start a 60-second capture, and wait. Thirty seconds in, the process crashes again.

What should happen next?

Before v1.7.0, the answer was: nothing good. loop-detective would hang waiting for a CDP response that would never come, or exit with a cryptic WebSocket error. Any lag events and slow I/O data collected during those first 30 seconds? Gone.

This is the story of how we made loop-detective handle the worst-case scenario — the target process dying mid-profiling — and why the solution required changes across every layer of the tool.

The Failure Mode

To understand the problem, you need to know how profiling works internally:

1. Connect to inspector WebSocket
2. Send Profiler.start
3. Sleep for <duration> seconds     ← target can exit here
4. Send Profiler.stop               ← this never gets a response
5. Analyze and report

Step 3 is a setTimeout wrapped in a Promise. Step 4 is a CDP command sent over the WebSocket. If the target process exits during step 3, the WebSocket closes. When step 4 tries to send Profiler.stop, it throws "Inspector not connected." The error bubbles up, and the tool exits without reporting anything useful.

But the real loss is the data. During those 30 seconds before the crash, the lag detector was running. The I/O tracker was recording slow HTTP calls. That data exists in memory — it just never gets reported because the profiling pipeline assumes it will complete normally.

The Fix: Three Layers

Layer 1: Detect the Exit Immediately

The WebSocket close event fires when the target process exits. Previously, this just emitted a disconnected event. Now it also rejects all pending CDP callbacks and emits a targetExit event:

// inspector.js
this.ws.on('close', () => {
  // Reject all pending callbacks — target is gone
  for (const { reject, timer } of this._callbacks.values()) {
    clearTimeout(timer);
    try { reject(new Error('Target process exited')); } catch {}
  }
  this._callbacks.clear();
  this.emit('disconnected');
});

This is critical. Without rejecting pending callbacks, any in-flight CDP command (like Profiler.stop) would hang until its 30-second timeout. That's 30 seconds of the user staring at a frozen terminal.

Layer 2: Cancel the Sleep

The profiling duration is implemented as a sleep:

await this._sleep(duration);

If the target exits 10 seconds into a 60-second profile, we don't want to wait the remaining 50 seconds. The sleep needs to be cancellable.

_sleep(ms) {
  return new Promise((resolve, reject) => {
    this._sleepReject = reject;
    const timer = setTimeout(() => {
      this._sleepReject = null;
      resolve();
    }, ms);
    if (timer.unref) timer.unref();
  });
}

When the disconnected event fires, the Detective cancels the sleep:

this.inspector.on('disconnected', () => {
  if (!this._stopping) {
    this._targetExited = true;
    this.emit('targetExit', { pid, message: `Target process (PID ${pid}) exited during profiling` });

    // Cancel the sleep so _captureProfile can finish
    if (this._sleepReject) {
      this._sleepReject(new Error('Target process exited'));
      this._sleepReject = null;
    }
  }
});

Layer 3: Salvage What You Can

_captureProfile now handles the case where the sleep is cancelled and the profiler can't be stopped:

async _captureProfile(duration) {
  await this.inspector.send('Profiler.enable');
  await this.inspector.send('Profiler.setSamplingInterval', { interval: 100 });
  await this.inspector.send('Profiler.start');

  try {
    await this._sleep(duration);
  } catch {
    // Sleep was cancelled (target exited) — try to stop the profiler anyway
  }

  try {
    const { profile } = await this.inspector.send('Profiler.stop');
    await this.inspector.send('Profiler.disable');
    return profile;
  } catch {
    // Target exited — no profile data available
    return null;
  }
}

The callers check for null:

const profile = await this._captureProfile(this.config.duration);
if (profile) {
  const analysis = this.analyzer.analyzeProfile(profile);
  this.emit('profile', analysis, profile);
}
// If null, targetExit event was already emitted

The key insight: even though we can't get the CPU profile (it's inside the dead process), the lag events and slow I/O events were already emitted in real-time during the profiling period. The user has already seen them scroll by in the terminal. They're not lost.

What the User Sees

Before v1.7.0:

✔ Connected to Node.js process
  Profiling for 60s with 50ms lag threshold...

⚠ Event loop lag: 245ms at 2025-03-15T14:23:45.123Z
🌐 Slow HTTP: 1820ms GET api.example.com/users → 200

<hangs for 30 seconds, then>

✖ Error: Inspector not connected

After v1.7.0:

✔ Connected to Node.js process
  Profiling for 60s with 50ms lag threshold...

⚠ Event loop lag: 245ms at 2025-03-15T14:23:45.123Z
🌐 Slow HTTP: 1820ms GET api.example.com/users → 200

  ✖ Target process (PID 12345) exited during profiling
    Any lag or I/O events collected before the exit are shown above.

✔ Disconnected cleanly

No hang. Clear message. The partial data (lag event + slow HTTP call) is visible above the exit message.

Exit Codes

We introduced a three-code system:

Code	Meaning
0	Success — profiling completed normally
1	Error — connection failed, invalid arguments, etc.
2	Target exited — process died during profiling

This matters for automation. A monitoring script can distinguish between "profiling worked" (0), "loop-detective itself had a problem" (1), and "the target crashed" (2):

loop-detective 12345 -d 60
EXIT_CODE=$?

if [ $EXIT_CODE -eq 2 ]; then
  echo "Target process crashed during profiling — check partial output"
  # Trigger incident response
elif [ $EXIT_CODE -eq 1 ]; then
  echo "loop-detective failed to connect"
elif [ $EXIT_CODE -eq 0 ]; then
  echo "Profiling completed successfully"
fi

The Irony of Profiling Crashing Processes

There's an irony here: the processes most likely to exit during profiling are the ones you most need to profile. They're crashing. They're running out of memory. They're hitting unhandled exceptions. You attached loop-detective precisely because something is wrong.

This means the "target exits during profiling" scenario isn't an edge case — it's a primary use case. If your tool can't handle it gracefully, it fails exactly when it's needed most.

The partial data is often enough. If you see three slow HTTP calls to the same endpoint right before the crash, that's a strong signal. If you see event loop lag spiking to 2 seconds, that tells you the process was CPU-bound before it died. You don't always need the full CPU profile to understand what happened.

Watch Mode Behavior

In --watch mode, a target exit stops the watch loop instead of starting a new cycle:

if (this._running && !this._targetExited) {
  setTimeout(runCycle, 1000);
}

There's no point retrying — the process is gone. The tool reports what it has and exits cleanly.

Programmatic API

For tool builders integrating loop-detective:

const { Detective } = require('node-loop-detective');

const detective = new Detective({ pid: 12345, duration: 60000 });

// Collect events as they arrive
const events = { lags: [], slowIO: [] };
detective.on('lag', (data) => events.lags.push(data));
detective.on('slowIO', (data) => events.slowIO.push(data));

// Handle target exit
detective.on('targetExit', (data) => {
  console.log(data.message);
  console.log(`Collected ${events.lags.length} lag events and ${events.slowIO.length} slow I/O events before exit`);
  // Send partial data to your monitoring system
  sendToMonitoring({ partial: true, ...events });
});

detective.on('profile', (analysis) => {
  // Full profile available — target didn't crash
  sendToMonitoring({ partial: false, analysis, ...events });
});

await detective.start();

The Broader Lesson

Diagnostic tools need to be more resilient than the systems they diagnose. If your profiler crashes when the target crashes, you've lost the most valuable debugging moment. Every diagnostic tool that attaches to external processes should answer three questions:

What happens when the target exits? Don't hang. Don't lose data. Report what you have.
How does the user know what happened? Clear messages, distinct exit codes, structured output.
What data survives? Real-time events (lag, I/O) survive because they were already emitted. Batch results (CPU profile) may be lost. Design accordingly.

The fix in v1.7.0 is about 60 lines of code across three files. But it transforms the tool from one that fails at the worst possible moment to one that handles it gracefully.

npm install -g node-loop-detective@1.7.0

Source: github.com/iwtxokhtd83/node-loop-detective

Not All Requests Are Equal: Adding Variable Cost to a Node.js Rate Limiter

Bill Tu — Thu, 09 Apr 2026 13:51:48 +0000

Rate limiters typically treat every request the same. One request, one token. But in the real world, a request that fetches a single user profile and a request that exports 10,000 records are not the same thing. Charging them the same rate limit token is like charging the same toll for a bicycle and a semi-truck.

We just shipped variable cost support in node-rate-limiter-pro v1.2.0. This post covers why it matters, how we implemented it across two algorithms and two storage backends, and the one subtle bug that almost shipped with it.

The Problem

Consider a batch API endpoint:

POST /api/users/export
Body: { "ids": ["u1", "u2", ..., "u500"] }

With a flat rate limiter set to 100 requests per minute, a client can export 50,000 users per minute by sending 100 requests of 500 IDs each. But a client making simple single-user lookups gets the same 100 requests per minute — 100 users total.

That's a 500x difference in actual resource consumption for the same rate limit.

Variable cost fixes this. Instead of consume('user:123') always deducting 1 token, you can now write:

// Each ID in the batch costs one token
const result = await limiter.consume('user:123', ids.length);

The batch export of 500 IDs now costs 500 tokens. The single lookup costs 1. The rate limiter finally reflects reality.

The API

The change is minimal — one optional parameter:

// Before (still works, backward compatible)
await limiter.consume('user:123');

// After: consume 5 tokens
await limiter.consume('user:123', 5);

For Express middleware, we added a costFn option:

app.use(limiter.middleware({
  costFn: (req) => req.body?.items?.length || 1,
}));

Implementation: Four Layers Deep

The cost parameter touches every layer of the stack: the public API, two algorithm implementations, two store backends, and two Redis Lua scripts. Here's how each one changed.

Token Bucket

The Token Bucket was the easy one. The algorithm already works with fractional tokens — we just replaced the hardcoded 1 with cost:

// Before
if (bucket.tokens >= 1) {
  bucket.tokens -= 1;
  // ...
}
const retryAfter = Math.ceil(
  ((1 - bucket.tokens) / this.maxTokens) * this.refillIntervalMs
);

// After
if (bucket.tokens >= cost) {
  bucket.tokens -= cost;
  // ...
}
const retryAfter = Math.ceil(
  ((cost - bucket.tokens) / this.maxTokens) * this.refillIntervalMs
);

The retryAfter calculation scales naturally. If you need 5 tokens and only have 2, you need to wait for 3 tokens to refill. The math is (cost - currentTokens) / refillRate.

Sliding Window — Where It Got Interesting

Our sliding window uses a weighted counter approach. Instead of storing every timestamp, it keeps two sub-window counters and calculates a weighted estimate:

estimatedCount = prevWindowCount × weight + currentWindowCount

where weight decays linearly from 1 to 0 as time progresses through the current window.

The original admission check was:

if (estimatedCount < this.maxRequests) {
  state.currCount++;
  // allowed
}

The naive variable-cost version would be:

if (estimatedCount + cost <= this.maxRequests) {
  state.currCount += cost;
  // allowed
}

This is where the bug hid. More on that in a moment. The correct implementation:

const used = Math.floor(estimatedCount);
const remaining = Math.max(0, this.maxRequests - used);

if (cost <= remaining) {
  state.currCount += cost;
  // allowed
}

We floor the estimated count to get integer remaining capacity, then check if the cost fits. The remaining field in the response now accurately reflects how many tokens are available — even when the request is denied:

// Denied response now shows actual remaining capacity
return {
  allowed: false,
  remaining,  // e.g., 2 tokens left, but you asked for 5
  // ...
};

This is a meaningful improvement over the old code, which always returned remaining: 0 on denial regardless of actual capacity.

Redis Lua Scripts

Both Redis Lua scripts needed the same treatment. The cost arrives as an additional ARGV parameter.

Token Bucket in Lua — straightforward substitution:

local cost = tonumber(ARGV[4])

if tokens >= cost then
  tokens = tokens - cost
  -- ...
  return {1, math.floor(tokens), 0}
end

local retryAfter = math.ceil(((cost - tokens) / maxTokens) * refillInterval)
return {0, math.floor(tokens), retryAfter}

Sliding Window in Lua — the ZSET approach needs to add cost members:

local cost = tonumber(ARGV[4])

if count + cost <= limit then
  for i = 1, cost do
    local seq = redis.call('INCR', key .. ':seq')
    redis.call('ZADD', key, now, now .. '-' .. seq)
  end
  return {1, limit - count - cost, 0}
end

return {0, limit - count, retryAfter}

The loop adds cost individual entries to the sorted set. Each entry gets a unique member via the atomic INCR counter (which we introduced in v1.1.0 to fix a collision bug — see our previous post). This keeps the ZSET accurate for future window calculations.

Express Middleware

The middleware layer adds a costFn option that mirrors the existing keyFn pattern:

middleware(options?: {
  keyFn?: (req: any) => string;
  costFn?: (req: any) => number;
  onLimited?: (req: any, res: any) => void;
}) {
  const costFn = options?.costFn;

  return async (req, res, next) => {
    const key = keyFn(req);
    const cost = costFn ? costFn(req) : 1;
    const result = await this.consume(key, cost);
    // ...
  };
}

The Floating Point Bug That Almost Shipped

Remember the sliding window admission check? Here's the bug we caught in testing.

Setup: limit = 3, window = 1000ms. Consume 3 tokens, wait 1100ms (just past the window boundary), then try to consume 1 more.

After 1100ms, the algorithm advances the sub-windows:

prevCount = 3, currCount = 0
elapsedInCurrent ≈ 100ms
weight = 1 - 100/1000 = 0.9
estimatedCount = 3 × 0.9 + 0 = 2.7

The old check estimatedCount < maxRequests → 2.7 < 3 → true ✅

The naive variable-cost check estimatedCount + cost <= maxRequests → 2.7 + 1 = 3.7 <= 3 → false ❌

Same scenario, different result. The request that should be allowed gets blocked.

The root cause: the weighted counter produces fractional values. The old code compared a float against an integer with <, which implicitly gave ~0.99 tokens of headroom. The new code with <= eliminated that headroom.

The fix: floor the estimate before comparing, converting the float back to integer capacity:

const used = Math.floor(estimatedCount);     // floor(2.7) = 2
const remaining = Math.max(0, limit - used); // 3 - 2 = 1
if (cost <= remaining)                       // 1 <= 1 → true ✅

This preserves the original behavior for cost = 1 while correctly handling higher costs. If estimatedCount were 2.3 and cost were 2, we'd get remaining = 1, and 2 <= 1 → false — correctly blocking a request that would exceed the limit.

Real-World Usage Patterns

Here are some patterns we've seen (or expect to see) with variable cost:

Batch APIs:

app.post('/api/batch', limiter.middleware({
  costFn: (req) => req.body?.operations?.length || 1,
}));

File uploads by size:

app.post('/upload', limiter.middleware({
  costFn: (req) => Math.ceil(Number(req.headers['content-length'] || 1) / 1_000_000),
}));

Tiered pricing (premium users get cheaper requests):

app.use(limiter.middleware({
  costFn: (req) => req.user?.plan === 'premium' ? 1 : 3,
}));

GraphQL complexity:

app.use('/graphql', limiter.middleware({
  costFn: (req) => calculateQueryComplexity(req.body.query),
}));

Design Decisions

A few choices we made along the way:

Why cost as a parameter, not a constructor option? Because cost varies per request, not per limiter. A single limiter instance might handle requests with different costs. Making it a per-call parameter is the only design that works.

Why default to 1? Backward compatibility. Every existing consume('key') call continues to work without changes. The cost parameter is purely additive.

Why floor the sliding window estimate? The weighted counter is an approximation. Flooring converts it to a conservative integer count, which is the right behavior for admission control — when in doubt, allow rather than block. This matches the original behavior and avoids surprising users with stricter-than-expected limits.

Why not validate cost > 0? We considered it, but decided against adding runtime validation on the hot path. If you pass cost = 0, you get a free peek at the current state (which is actually useful). If you pass a negative number, you're doing something wrong, but we're not going to add a branch to every consume() call to catch it.

What's Next

Variable cost was the most requested feature in our issue tracker. With it shipped, we're looking at:

get(key) — peek at rate limit state without consuming (#6)
Fixed Window Counter algorithm (#7)
IETF standard rate limit headers (#8)

The full implementation is in v1.2.0. MIT licensed, zero dependencies for in-memory mode, optional Redis for distributed deployments.

Writing 89 Tests for a Quantitative Trading Framework: Strategy and Trade-offs

Bill Tu — Thu, 09 Apr 2026 13:25:06 +0000

Adding a test suite to an existing codebase is a different exercise than writing tests alongside new code. You're reverse-engineering the implicit contracts, discovering which behaviors are intentional and which are accidental, and deciding what's worth testing versus what's noise. This article covers how we built the test suite for QuantFlow, an open-source quantitative trading framework in Python, and the decisions behind each layer of tests.

The Testing Problem in Quant Systems

Quantitative trading code has a testing problem that most software doesn't: the outputs are floating-point numbers derived from financial time series, and "correct" is often a matter of degree. An SMA of [1, 2, 3, 4, 5] with period 5 should be exactly 3.0 — that's easy. But what should the RSI of a 50-bar synthetic price series be? You can't hardcode an expected value without coupling the test to the random seed and the exact implementation.

This creates a tension between two testing philosophies:

Test against known reference values (precise but brittle)
Test invariants and contracts (robust but less specific)

We use both, choosing based on the indicator.

Test Architecture

The suite is organized into five files, each targeting a different layer:

tests/
├── conftest.py          # Shared fixtures
├── test_indicators.py   # 46 tests — all 16 indicators
├── test_risk.py         # 12 tests — risk manager rules
├── test_portfolio.py    # 16 tests — execution and position tracking
├── test_strategies.py   # 8 tests  — built-in strategy signals
└── test_engine.py       # 9 tests  — end-to-end backtest integration

The dependency flows downward: indicators have no dependencies, risk and portfolio depend on core models, strategies depend on indicators, and the engine depends on everything. Tests follow the same order — if indicator tests fail, strategy and engine tests are meaningless.

Fixtures: Deterministic Randomness

The conftest.py provides four shared fixtures that generate synthetic market data with a fixed random seed:

@pytest.fixture
def sample_prices() -> np.ndarray:
    np.random.seed(42)
    returns = np.random.normal(0.001, 0.02, 50)
    prices = 100.0 * np.cumprod(1 + returns)
    return prices

The seed 42 makes tests deterministic — same data every run. The parameters (0.1% daily drift, 2% daily volatility) produce realistic-looking price series without being tied to any real market data. This matters because tests that depend on real market data break when the data source changes, and they can't be run offline.

The sample_ohlcv fixture generates correlated OHLCV data where high > close > low holds approximately (with small random noise). This is important for indicators like ATR and Stochastic that use all four price fields — feeding them random uncorrelated data would test the math but not the real-world behavior.

Indicator Tests: Three Categories

The 46 indicator tests fall into three categories.

Warm-up Guards

Every indicator should return None when given insufficient data. This is the most important contract — it prevents strategies from acting on garbage values during the first few bars of a backtest.

def test_insufficient_data_returns_none(self):
    sma = SMA(period=20)
    assert sma.calculate(np.array([1.0, 2.0, 3.0])) is None

We test this for every indicator. It's repetitive, but it catches a common class of bugs: an indicator that returns 0.0 or NaN instead of None when it doesn't have enough data.

Known-Value Tests

For indicators with simple formulas, we test against hand-calculated values:

def test_known_value(self):
    prices = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
    sma = SMA(period=5)
    assert sma.calculate(prices) == pytest.approx(3.0)

def test_known_value(self):
    prices = np.array([100.0] * 13)
    prices[-1] = 110.0
    roc = ROC(period=12)
    assert roc.calculate(prices) == pytest.approx(10.0)

These are the strongest tests — they verify the actual computation. But they're only practical for indicators where you can compute the expected value by hand. For complex indicators like ADX or Ichimoku, hand-calculation is error-prone, so we use invariant tests instead.

Invariant Tests

For indicators where exact values are hard to predict, we test mathematical invariants that must always hold:

def test_range_0_to_100(self, sample_prices):
    rsi = RSI(period=14)
    val = rsi.calculate(sample_prices)
    assert 0.0 <= val <= 100.0

def test_all_gains_returns_100(self):
    prices = np.arange(1.0, 20.0)  # monotonically increasing
    rsi = RSI(period=14)
    assert rsi.calculate(prices) == pytest.approx(100.0)

def test_all_losses_returns_0(self):
    prices = np.arange(20.0, 1.0, -1.0)  # monotonically decreasing
    assert rsi.calculate(prices) == pytest.approx(0.0, abs=0.01)

RSI must be between 0 and 100. All-gain series must produce RSI of 100. All-loss series must produce RSI near 0. These invariants hold regardless of the implementation details.

For Bollinger Bands: upper > middle > lower (when prices have non-zero variance). For Stochastic: %K between 0 and 100. For Williams %R: between -100 and 0. These are mathematical properties of the formulas, not implementation details.

Consistency Tests

One particularly valuable test pattern verifies that calculate() and series() agree:

def test_calculate_matches_series_last(self, sample_prices):
    ema = EMA(period=12)
    calc_val = ema.calculate(sample_prices)
    series_val = ema.series(sample_prices)[-1]
    assert calc_val == pytest.approx(series_val, rel=1e-10)

After the v0.2.0 optimization where EMA.calculate() was rewritten to avoid building the full series, this test ensures the optimized path produces the same result as the reference implementation. It's a regression test disguised as a unit test.

The NaN Leakage Test

One test specifically targets a bug we fixed in v0.2.0:

def test_no_nan_leakage(self):
    prices = np.arange(1.0, 40.0)
    macd = MACD()
    m, s, h = macd.calculate(prices)
    if m is not None:
        assert not np.isnan(m)
        assert not np.isnan(s)
        assert not np.isnan(h)

The contract is: calculate() returns either valid floats or None. Never NaN. This test exists because NaN is insidious — it passes through arithmetic silently and makes comparisons return False, causing strategies to silently skip signals without any error.

The Stochastic Alignment Test

Another test targets the %D alignment bug from v0.2.0:

def test_d_alignment(self, sample_ohlcv):
    stoch = Stochastic(k_period=5, d_period=3)
    k_series, d_series = stoch.series(...)
    first_k = np.where(~np.isnan(k_series))[0][0]
    first_d = np.where(~np.isnan(d_series))[0][0]
    assert first_d == first_k + stoch.d_period - 1

%D is a moving average of %K, so it must start exactly d_period - 1 bars after %K begins. This is a structural invariant — if the alignment is wrong, the %D values are computed from the wrong %K windows.

Risk Manager Tests: State Machine Behavior

The risk manager is a stateful component — once the drawdown circuit breaker triggers, it stays triggered until explicitly reset. The tests model this as a state machine:

def test_drawdown_halts_trading(self):
    signal = Signal(SignalType.BUY, size=10)
    result = self.rm.check(signal, 100.0, 75000.0, 100000.0, 0)
    assert result.type == SignalType.HOLD

def test_drawdown_halt_is_sticky(self):
    self.rm.check(Signal(SignalType.BUY, size=10), 100.0, 75000.0, 100000.0, 0)
    # Even if equity recovers, still halted
    result = self.rm.check(Signal(SignalType.BUY, size=10), 100.0, 95000.0, 100000.0, 0)
    assert result.type == SignalType.HOLD

def test_drawdown_reset(self):
    self.rm.check(Signal(SignalType.BUY, size=10), 100.0, 75000.0, 100000.0, 0)
    self.rm.reset()
    result = self.rm.check(Signal(SignalType.BUY, size=10), 100.0, 95000.0, 100000.0, 0)
    assert result.type == SignalType.BUY

Three tests, three states: normal → halted → reset. The "sticky" test is critical — it verifies that the halt persists even when equity recovers. Without this test, a bug that resets the halt flag on every call would go undetected.

The position sizing test uses exact arithmetic:

def test_position_sizing_caps_size(self):
    # equity=100000, max_pct=10% → max $10000 → at $100/share → max 100 shares
    signal = Signal(SignalType.BUY, size=500)
    result = self.rm.check(signal, 100.0, 100000.0, 100000.0, 0)
    assert result.size <= 100

The comment documents the math. This is intentional — risk management tests should be readable by someone who isn't a developer, because the rules they encode are business logic that a portfolio manager or risk officer should be able to verify.

Portfolio Tests: Isolating Execution Mechanics

Portfolio tests use zero slippage to make assertions predictable:

def setup_method(self):
    self.portfolio = Portfolio(
        initial_capital=100000.0,
        commission_rate=0.001,
        slippage_rate=0.0,  # zero slippage for predictable tests
    )

This is a deliberate choice. Slippage adds randomness to fill prices, which makes exact assertions impossible. By setting it to zero, we can test the execution logic in isolation. Slippage behavior is tested separately at the engine level.

The most important portfolio test verifies the v0.2.0 fix for equity calculation:

def test_equity_reflects_current_price(self):
    signal = Signal(SignalType.BUY, size=100)
    self.portfolio.execute_signal(signal, "TEST", 100.0, self.ts)
    self.portfolio.update_market_prices("TEST", 120.0)
    assert self.portfolio.equity > self.portfolio.cash + 100 * 100.0

If equity used entry price (the old bug), equity would equal cash + 100 * 100. With the fix, it equals cash + 100 * 120, which is strictly greater. The test doesn't check the exact value — it checks the relationship, which is more robust.

The short margin test verifies the v0.2.0 safety fix:

def test_short_margin_check(self):
    self.portfolio.short_margin_rate = 1.0
    signal = Signal(SignalType.SELL, size=2000)  # $200k margin > $100k cash
    result = self.portfolio.execute_signal(signal, "TEST", 100.0, self.ts)
    assert result is None

This test exists because the absence of a margin check was a real bug. The test documents the expected behavior: shorts that exceed available margin are rejected.

Strategy Tests: Signal Generation

Strategy tests face a unique challenge: strategies are stateful (they track previous indicator values for crossover detection), and their outputs depend on the full price history. Testing them requires constructing price series that produce known indicator states.

def test_buy_on_oversold(self):
    strategy = RSIMeanReversion(period=14, oversold=30, size=100)
    portfolio = Portfolio()
    prices = np.linspace(200, 100, 50)  # strong downtrend
    bar = make_bar(100.0, prices)
    signal = strategy.on_bar(bar, portfolio)
    assert signal.type == SignalType.BUY

A monotonically decreasing price series guarantees RSI will be very low (near 0), which is well below the oversold threshold of 30. We don't need to know the exact RSI value — we just need to know it's below 30, which is guaranteed by the price pattern.

The SMA crossover test is more involved because it requires a crossover event:

def test_generates_buy_on_golden_cross(self):
    strategy = SMACrossover(fast_period=3, slow_period=5, size=100)
    portfolio = Portfolio()
    prices_down = np.linspace(120, 100, 20)
    prices_up = np.linspace(100, 130, 10)
    prices = np.concatenate([prices_down, prices_up])
    signals = []
    for i in range(len(prices)):
        bar = make_bar(prices[i], prices[:i + 1])
        signals.append(strategy.on_bar(bar, portfolio))
    buy_signals = [s for s in signals if s.type == SignalType.BUY]
    assert len(buy_signals) > 0

The price series goes down then sharply up. At some point during the upturn, the fast SMA (3-period) will cross above the slow SMA (5-period). We don't assert exactly when — just that it happens at least once. This makes the test resilient to minor changes in the crossover detection logic.

Engine Tests: End-to-End Integration

Engine tests verify that all components work together. They use an InMemoryDataFeed to avoid file I/O:

class InMemoryDataFeed(DataFeed):
    def __init__(self, df: pd.DataFrame):
        self._df = df

    def load(self) -> pd.DataFrame:
        return self._df

The most valuable engine test verifies that commission has a measurable effect:

def test_commission_reduces_equity(self, data_feed):
    r1 = BacktestEngine(..., commission=0.0, ...).run()
    r2 = BacktestEngine(..., commission=0.01, ...).run()
    assert r2.equity_curve[-1] <= r1.equity_curve[-1]

This is an invariant test at the system level: higher commission should never increase final equity. It doesn't test the exact commission calculation — that's covered in portfolio tests — but it verifies that commission flows through the entire pipeline correctly.

Edge case tests verify the system doesn't crash on degenerate inputs:

def test_single_bar_no_crash(self):
    df = pd.DataFrame({...}, index=pd.DatetimeIndex([datetime(2024, 1, 1)]))
    result = BacktestEngine(data_feed=InMemoryDataFeed(df), ...).run()
    assert len(result.equity_curve) == 1

def test_zero_volume_no_crash(self):
    df = pd.DataFrame({..., "volume": np.zeros(n), ...})
    result = BacktestEngine(data_feed=InMemoryDataFeed(df), ...).run()
    assert result.total_return == pytest.approx(0.0)

These tests exist because real market data contains edge cases: single-bar datasets from API errors, zero-volume bars from illiquid instruments, gaps in timestamps. The system should handle all of these without raising exceptions.

What We Chose Not to Test

A few deliberate omissions:

We don't test the YahooDataFeed because it makes HTTP requests to an external API. Testing it would require either mocking the HTTP layer (which tests the mock, not the code) or hitting the real API (which is slow, flaky, and rate-limited). The CSVDataFeed is tested indirectly through the engine tests.

We don't test the LiveEngine because it runs an infinite loop with time.sleep(). Testing it properly would require threading, timeouts, and mock brokers — significant infrastructure for a component that's architecturally identical to the backtest engine but with a different data source.

We don't test the plot() method on PerformanceReport because visual output is best verified by humans. We could assert that it doesn't raise an exception, but that's a weak test that adds maintenance burden.

Running the Suite

pip install pytest
python -m pytest tests/ -v

============================= 89 passed in 4.87s ==============================

All 89 tests run in under 5 seconds. No external dependencies, no network calls, no file I/O (except the fixtures that generate in-memory DataFrames). This matters for CI — fast tests get run; slow tests get skipped.

The test suite is part of QuantFlow v0.3.0. Contributions and additional test cases are welcome.

Who Should Own the Order ID? Moving ID Generation Inside the Matching Engine

Bill Tu — Thu, 09 Apr 2026 13:04:07 +0000

Order IDs seem trivial. Every order needs one, they need to be unique, and they show up in every trade record. In the first version of MatchEngine, the caller provided the ID as a string. It worked. Then it didn't.

This article explains why we moved ID generation inside the engine, the design choices behind the implementation, and the three new API methods that replaced the old approach.

The Problem With Caller-Provided IDs

The original API looked like this:

order := model.NewLimitOrder("my-order-1", model.Buy, price, quantity)
trades, err := e.SubmitOrder("BTC/USD", order)

The caller chose the ID. The engine accepted it. This created three problems:

1. Uniqueness is the caller's problem. The engine added duplicate detection in v0.2.0, but that just turned a silent corruption into an error. The caller still had to generate unique IDs — and every caller had to solve the same problem independently. In a system with multiple order sources (REST API, WebSocket, FIX gateway), coordinating unique IDs across all of them is a real engineering challenge.

2. No ordering guarantee. Caller-provided IDs are arbitrary strings. There is no way to determine which order came first by looking at the IDs. For debugging, auditing, and replay, monotonically increasing IDs are invaluable. They give you a total ordering of events without consulting timestamps.

3. The API is noisy. Every call site has to construct an Order struct with an ID, a side, a type, a price, a quantity, and a timestamp. The ID and timestamp are boilerplate — the caller doesn't care about them. They care about "sell 5 BTC at 50,000."

The Design: Atomic Counter

We chose the simplest possible ID generator: an atomic uint64 counter.

type Engine struct {
    nextID   uint64  // atomic counter
    idPrefix string  // optional prefix
    // ...
}

func (e *Engine) generateID() string {
    id := atomic.AddUint64(&e.nextID, 1)
    return e.idPrefix + strconv.FormatUint(id, 10)
}

atomic.AddUint64 increments the counter and returns the new value in a single atomic operation. No mutex needed. No allocation. The result is converted to a string with strconv.FormatUint, which is the fastest integer-to-string conversion in Go's standard library.

Why Not UUID?

UUIDs (google/uuid) are globally unique, which sounds appealing. But they have drawbacks for this use case:

Property	Atomic Counter	UUID v4
Uniqueness	Unique within engine instance	Globally unique
Ordering	Monotonically increasing	Random (no ordering)
Size	1-20 chars (`"1"` to `"18446744073709551615"`)	36 chars (`"550e8400-e29b-41d4-a716-446655440000"`)
Speed	~1ns (atomic add + format)	~200ns (crypto/rand + format)
Readability	`"ORD-42"`	`"550e8400-e29b-41d4-a716-446655440000"`

For a single-instance matching engine, the atomic counter is strictly better. It is faster, smaller, ordered, and human-readable. If the engine were distributed across multiple nodes, UUIDs or a distributed ID service (like Twitter's Snowflake) would be necessary. But that is a different problem for a different architecture.

Why Not Inside the Mutex?

The generateID method uses atomic.AddUint64, which is lock-free. It runs before the engine acquires its mutex in SubmitOrder. This is deliberate:

func (e *Engine) SubmitLimitOrder(symbol string, side model.Side, price, quantity decimal.Decimal) (string, []model.Trade, error) {
    id := e.generateID()  // lock-free, outside mutex
    order := model.NewLimitOrder(id, side, price, quantity)
    trades, err := e.SubmitOrder(symbol, order)  // acquires mutex internally
    return id, trades, err
}

If ID generation were inside the mutex, it would add contention for no reason. The atomic operation is already thread-safe on its own. Keeping it outside the mutex means the critical section is no longer than before.

The Prefix Option

IDs can be prefixed for readability or multi-engine disambiguation:

e := engine.New(engine.WithIDPrefix("ORD-"))
// Generates: "ORD-1", "ORD-2", "ORD-3", ...

e := engine.New(engine.WithIDPrefix("ENGINE-A-"))
// Generates: "ENGINE-A-1", "ENGINE-A-2", ...

The prefix is concatenated at generation time. No extra allocation — strconv.FormatUint returns a string, and + on two strings is a single allocation.

The New API

Three new methods replace the old pattern:

SubmitLimitOrder

func (e *Engine) SubmitLimitOrder(
    symbol string,
    side model.Side,
    price, quantity decimal.Decimal,
) (string, []model.Trade, error)

The caller provides only what they care about: symbol, side, price, quantity. The engine generates the ID and returns it.

id, trades, err := e.SubmitLimitOrder("BTC/USD", model.Sell, price, qty)
// id = "1" (or "ORD-1" with prefix)

SubmitMarketOrder

func (e *Engine) SubmitMarketOrder(
    symbol string,
    side model.Side,
    quantity decimal.Decimal,
) (string, []model.Trade, error)

Same pattern, no price parameter (market orders don't have one).

id, trades, err := e.SubmitMarketOrder("BTC/USD", model.Buy, qty)

SubmitRequest

For cases where the order type is determined at runtime (e.g., from a JSON payload), SubmitRequest accepts an OrderRequest struct:

type OrderRequest struct {
    Side     Side
    Type     OrderType
    Price    decimal.Decimal
    Quantity decimal.Decimal
}

No ID field. No timestamp. Just the intent.

req := model.NewLimitOrderRequest(model.Buy, price, qty)
id, trades, err := e.SubmitRequest("BTC/USD", req)

This is particularly useful when deserializing from external input:

// From a REST API handler
var req model.OrderRequest
json.NewDecoder(r.Body).Decode(&req)
id, trades, err := e.SubmitRequest(symbol, req)

The caller never touches an order ID. The engine owns the lifecycle.

Backward Compatibility

The original SubmitOrder method is unchanged:

func (e *Engine) SubmitOrder(symbol string, order *model.Order) ([]model.Trade, error)

It still accepts caller-provided IDs, still checks for duplicates, and still works exactly as before. The new methods are built on top of it — they generate an ID, construct an Order, and call SubmitOrder internally.

This means existing code continues to work without modification. Migration is opt-in.

How the Pieces Fit Together

The call chain for SubmitLimitOrder:

SubmitLimitOrder("BTC/USD", Sell, 50000, 1)
    │
    ├── generateID()              ← atomic, lock-free
    │   └── "ORD-7"
    │
    ├── NewLimitOrder("ORD-7", Sell, 50000, 1)
    │   └── Order{ID: "ORD-7", ...}
    │
    └── SubmitOrder("BTC/USD", order)
        ├── normalize symbol      ← "BTC/USD"
        ├── acquire mutex
        ├── validate symbol
        ├── check duplicate ID    ← "ORD-7" not in index
        ├── match against book
        ├── add to book if unfilled
        ├── record trades
        └── release mutex

The ID is generated before the mutex is acquired. If SubmitOrder returns an error (invalid price, bad symbol, etc.), the ID is "wasted" — the counter has already incremented. This is fine. Gaps in the ID sequence are harmless, and avoiding them would require either rolling back the atomic counter (unsafe) or generating the ID inside the mutex (unnecessary contention).

Testing Concurrency

The most important property of the ID generator is uniqueness under concurrent access. We test this with 10 goroutines each submitting 20 orders simultaneously:

func TestIDsUniqueAcrossGoroutines(t *testing.T) {
    e := New()

    idCh := make(chan string, 200)
    var wg sync.WaitGroup
    for g := 0; g < 10; g++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for i := 0; i < 20; i++ {
                id, _, _ := e.SubmitLimitOrder("BTC", model.Sell, d("100"), d("1"))
                idCh <- id
            }
        }()
    }
    wg.Wait()
    close(idCh)

    seen := make(map[string]bool)
    for id := range idCh {
        if seen[id] {
            t.Errorf("duplicate ID generated: %s", id)
        }
        seen[id] = true
    }
    if len(seen) != 200 {
        t.Errorf("expected 200 unique IDs, got %d", len(seen))
    }
}

200 orders, 10 goroutines, zero duplicates. The atomic counter guarantees this without any locking.

We also verify monotonic ordering:

func TestIDsAreMonotonicallyIncreasing(t *testing.T) {
    e := New()

    var ids []string
    for i := 0; i < 10; i++ {
        id, _, _ := e.SubmitLimitOrder("BTC", model.Sell, d("100"), d("1"))
        ids = append(ids, id)
    }

    for i := 1; i < len(ids); i++ {
        if ids[i] <= ids[i-1] {
            t.Errorf("IDs not monotonically increasing: %s <= %s", ids[i], ids[i-1])
        }
    }
}

Note: monotonic ordering is guaranteed only within a single goroutine. Across goroutines, the IDs are unique but the observation order depends on scheduling. Goroutine A might generate ID 5 before goroutine B generates ID 4, but B might write to the channel first. The IDs themselves are still strictly ordered by their numeric value.

Before and After

Before (caller manages IDs)

e := engine.New()

// Caller must generate unique IDs
sellID := fmt.Sprintf("sell-%d", time.Now().UnixNano())
sell := model.NewLimitOrder(sellID, model.Sell, price, qty)
trades, err := e.SubmitOrder("BTC/USD", sell)

buyID := fmt.Sprintf("buy-%d", time.Now().UnixNano())
buy := model.NewLimitOrder(buyID, model.Buy, price, qty)
trades, err = e.SubmitOrder("BTC/USD", buy)

// Hope the IDs are unique...
e.CancelOrder("BTC/USD", sellID)

After (engine manages IDs)

e := engine.New(engine.WithIDPrefix("ORD-"))

sellID, _, _ := e.SubmitLimitOrder("BTC/USD", model.Sell, price, qty)
buyID, trades, _ := e.SubmitLimitOrder("BTC/USD", model.Buy, price, qty)

// IDs are guaranteed unique
e.CancelOrder("BTC/USD", sellID)

Less code, no uniqueness bugs, and the returned IDs are useful for logging, cancellation, and client responses.

Summary

Aspect	Before	After
ID source	Caller-provided string	Engine-generated atomic counter
Uniqueness	Caller's responsibility (error on duplicate)	Guaranteed by engine
Ordering	Arbitrary	Monotonically increasing
Thread safety	N/A (caller's problem)	Lock-free atomic operation
API surface	`SubmitOrder(symbol, *Order)`	`SubmitLimitOrder`, `SubmitMarketOrder`, `SubmitRequest`
Backward compat	N/A	`SubmitOrder` still works

The change is small — one new field on the engine, one three-line method, three thin wrapper methods, and one new OrderRequest type. But it shifts the responsibility for a correctness-critical property (ID uniqueness) from every caller to a single, tested, atomic implementation inside the engine. That is the right place for it.

GitHub: https://github.com/iwtxokhtd83/MatchEngine
Release: v0.4.0
Issue: #6 — Engine should generate unique order IDs internally

Less Is More: Why We Added a Flag to Disable Our Best Feature

Bill Tu — Thu, 09 Apr 2026 06:00:31 +0000

We spent three releases building async I/O tracking into node-loop-detective. HTTP request timing. DNS lookup monitoring. TCP connection tracking. Global fetch() support. dns.promises.lookup coverage. It's the feature that fills the blind spot every other Node.js profiler has.

Then we added a flag to turn it all off.

loop-detective 12345 --no-io

This isn't a contradiction. It's a design principle: diagnostic tools that attach to production processes must give operators full control over what gets injected. Here's why.

What --no-io Does

When you pass --no-io, loop-detective skips the entire _startAsyncIOTracking() phase. No monkey-patching of http.request. No wrapping of dns.lookup. No touching net.Socket.connect. No fetch() interception. None of it.

What still works:

Event loop lag detection (the lightweight setInterval probe)
CPU profiling (V8's built-in sampling profiler via CDP)
All analysis: heavy functions, call stacks, blocking pattern detection
--save-profile for flame graph export

What's disabled:

Slow HTTP/HTTPS request tracking
Slow fetch() tracking
Slow DNS lookup tracking
Slow TCP connection tracking
The "Slow Async I/O Summary" section in the report

Why Someone Would Want This

Reason 1: The Monkey-Patching Risk

I/O tracking works by monkey-patching core Node.js modules inside the target process. We replace http.request with a wrapper that calls the original, measures the time, and records slow operations. Same for dns.lookup, net.Socket.connect, and globalThis.fetch.

This is inherently more invasive than the other diagnostic methods:

Method	How it works	Invasiveness
Event loop lag	`setInterval` with `unref()`	Minimal — one timer
CPU profiling	V8's built-in sampler via CDP	Minimal — native V8 feature
I/O tracking	Monkey-patches 6+ core functions	Moderate — modifies module behavior

For most applications, the monkey-patches are safe. We store originals, we restore them on cleanup, we don't alter the return values or error behavior. But "most" isn't "all."

Some environments have strict policies about runtime code modification. Financial systems, healthcare platforms, and security-critical services may have compliance requirements that prohibit monkey-patching production code — even temporarily, even from a diagnostic tool.

Reason 2: The CPU-Only Investigation

Sometimes you already know the problem is CPU-bound. A developer pushed a commit with an O(n²) algorithm. A regex is backtracking. A JSON.parse is choking on a 50MB payload.

In these cases, I/O tracking is noise. The report shows "No slow I/O operations detected" (because there aren't any), but the I/O patches are still active, adding a tiny overhead to every network call. With --no-io, you get a cleaner report focused on what matters:

# CPU-only investigation
loop-detective 12345 --no-io -d 30

# Output focuses entirely on:
# - Event loop lag events
# - CPU heavy functions
# - Call stacks
# - Blocking patterns (cpu-hog, json-heavy, regex, gc, sync-io, crypto)

Reason 3: Reducing Attack Surface

When you attach to a production process, every piece of injected code is a potential risk. The I/O patches interact with the application's network stack — the most security-sensitive part of most services.

A conservative operator might want to start with the least invasive option:

# Step 1: CPU profile only — zero monkey-patching
loop-detective 12345 --no-io -d 10

# Step 2: If CPU looks fine, enable I/O tracking
loop-detective 12345 -d 30 --io-threshold 500

This graduated approach lets you get initial diagnostics with minimal risk, then opt into deeper instrumentation only if needed.

Reason 4: Isolating Variables

When debugging a complex performance issue, you want to change one thing at a time. If you're seeing unexpected behavior and you're not sure whether it's caused by the I/O patches or by the application itself, --no-io lets you eliminate the patches as a variable.

The Implementation

The implementation is deliberately simple. In detective.js:

async _singleRun() {
  try {
    await this._startLagDetection();
    if (!this.config.noIO) {
      await this._startAsyncIOTracking();
    }
    const profile = await this._captureProfile(this.config.duration);
    // ...
  } finally {
    await this.stop();
  }
}

One if statement. That's it. The _startAsyncIOTracking method — with its 200+ lines of monkey-patching code — is simply never called. No patches are injected. No cleanup is needed. The target process is never touched beyond the lag detector and the CPU profiler.

The same guard exists in _watchMode():

async _watchMode() {
  await this._startLagDetection();
  if (!this.config.noIO) {
    await this._startAsyncIOTracking();
  }
  // ... profiling cycles
}

In the CLI, it's a boolean flag:

const boolMap = {
  '--no-io': 'no-io',
  // ...
};

const config = {
  noIO: values['no-io'],
  // ...
};

In the programmatic API:

const detective = new Detective({
  pid: 12345,
  duration: 10000,
  noIO: true,
});

The Design Philosophy

Every feature in a diagnostic tool should be opt-out-able. This is different from application features, where you want sensible defaults that "just work." Diagnostic tools operate in a different trust model:

You're modifying someone else's process. The target application didn't ask to be profiled. The operator is making a judgment call about acceptable risk.
Production is not staging. What's safe in development might not be acceptable in production. Giving operators granular control respects this reality.
Less can be more. A focused CPU-only report is sometimes more useful than a comprehensive report with I/O data that isn't relevant to the current investigation.

This is why node-loop-detective has --no-io but doesn't have --no-lag or --no-profile. The lag detector and CPU profiler are minimally invasive (a single unref'd timer and V8's native sampler). The I/O tracker is qualitatively different — it modifies the behavior of core modules. That difference warrants an opt-out.

When to Use Each Mode

Scenario	Command
General diagnosis (default)	`loop-detective <pid>`
Known CPU issue	`loop-detective <pid> --no-io`
Sensitive production system	`loop-detective <pid> --no-io` (start here, add I/O later if needed)
Suspected slow I/O	`loop-detective <pid> --io-threshold 200`
Full investigation with export	`loop-detective <pid> -d 60 --save-profile ./profile.cpuprofile`
Minimal risk, maximum info	`loop-detective <pid> --no-io --save-profile ./profile.cpuprofile`

Try It

npm install -g node-loop-detective@1.6.0

# Full diagnostics (default)
loop-detective <pid>

# CPU and event loop only — no monkey-patching
loop-detective <pid> --no-io

Source: github.com/iwtxokhtd83/node-loop-detective

The best diagnostic tool is one you trust enough to run in production. Sometimes that means giving you the option to use less of it.

The Invisible Network Calls: Tracking fetch() and dns.promises in Node.js

Bill Tu — Wed, 08 Apr 2026 13:24:57 +0000

You've instrumented your Node.js app with I/O tracking. You're catching slow HTTP requests, slow database connections, slow DNS lookups. Your monitoring dashboard looks comprehensive.

Then a developer refactors a service client from axios to the built-in fetch(). Or switches from dns.lookup callbacks to dns.promises.lookup. The code is cleaner. The tests pass. The deployment goes out.

And your I/O tracking goes blind.

This is the story of two API surfaces that silently bypass traditional Node.js instrumentation — and how we patched them in node-loop-detective v1.5.0.

The Problem: Two Parallel Worlds

Node.js has evolved its networking APIs over the years, creating parallel paths that don't share the same internal plumbing.

fetch() vs http.request

Before Node.js 18, if you wanted to make an HTTP request, you used http.request (or a library built on top of it like axios, got, or node-fetch). Every HTTP client in the npm ecosystem ultimately called http.request or https.request under the hood.

Then Node.js 18 introduced a global fetch() function. It looks like the browser Fetch API. It's convenient. It's modern. And it's backed by undici — a completely separate HTTP client that does NOT use http.request internally.

This means if you monkey-patch http.request to track slow HTTP calls, fetch() calls are completely invisible. They go through undici's own connection pool, its own socket management, its own DNS resolution. Your instrumentation sees nothing.

// This is tracked (goes through http.request)
const axios = require('axios');
await axios.get('https://api.example.com/users');

// This is NOT tracked (goes through undici, bypasses http.request)
const res = await fetch('https://api.example.com/users');

As more codebases adopt fetch() — it's the standard API, it's built-in, it doesn't need a dependency — this blind spot grows.

dns.promises.lookup vs dns.lookup

A similar split exists in the DNS module. The original dns.lookup uses callbacks:

const dns = require('dns');
dns.lookup('example.com', (err, address) => {
  // ...
});

Node.js 10.6 introduced dns.promises.lookup, the promise-based equivalent:

const dns = require('dns');
const { address } = await dns.promises.lookup('example.com');

These are separate functions with separate code paths. Patching dns.lookup doesn't intercept dns.promises.lookup. Modern async/await code increasingly uses the promise API, making callback-only instrumentation incomplete.

Why This Matters in Practice

Consider a typical Node.js microservice in 2025:

// auth.js — uses fetch (Node.js 18+)
async function verifyToken(token) {
  const res = await fetch(`${AUTH_SERVICE_URL}/verify`, {
    method: 'POST',
    headers: { 'Authorization': `Bearer ${token}` },
    body: JSON.stringify({ token }),
  });
  return res.json();
}

// dns-cache.js — uses dns.promises
const dns = require('dns');
async function resolveService(hostname) {
  const { address } = await dns.promises.lookup(hostname);
  return address;
}

If the auth service starts responding slowly (800ms instead of 20ms), or DNS resolution starts hanging (2 seconds in a misconfigured container), your I/O tracker reports nothing. The event loop isn't blocked. The CPU profile looks healthy. But every request is paying a multi-second tax.

This is exactly the scenario users reported: "The report says healthy, but my app is slow."

The Fix: Patching Both Worlds

In v1.5.0, node-loop-detective now patches both the old and new API surfaces.

Patching fetch()

The global fetch is a regular function on globalThis. We wrap it the same way we wrap http.request — call the original, measure the time, record if it's slow:

(function patchFetch() {
  if (typeof globalThis.fetch !== 'function') return;
  const origFetch = globalThis.fetch;

  globalThis.fetch = function patchedFetch(input, init) {
    const startTime = Date.now();
    const callerStack = captureCallerStack();

    // Extract URL and method
    let target = 'unknown';
    let method = 'GET';
    if (typeof input === 'string') {
      target = input;
    } else if (input && typeof input === 'object') {
      target = input.url || input.href || String(input);
      method = (input.method || 'GET').toUpperCase();
    }
    if (init && init.method) {
      method = init.method.toUpperCase();
    }

    return origFetch.call(this, input, init).then(
      (res) => {
        const duration = Date.now() - startTime;
        if (duration >= threshold) {
          recordSlowOp({
            type: 'fetch', method, target,
            statusCode: res.status, duration,
            timestamp: Date.now(), stack: callerStack,
          });
        }
        return res;
      },
      (err) => {
        const duration = Date.now() - startTime;
        if (duration >= threshold) {
          recordSlowOp({
            type: 'fetch', method, target,
            error: err.message, duration,
            timestamp: Date.now(), stack: callerStack,
          });
        }
        throw err;
      }
    );
  };
})();

Key details:

Graceful detection: if (typeof globalThis.fetch !== 'function') return — on Node.js 16/17 where fetch doesn't exist, the patch is silently skipped.
Input parsing: fetch() accepts a string URL, a URL object, or a Request object. We handle all three.
Promise wrapping: fetch() returns a Promise, so we chain .then() to measure timing without altering the response.
URL shortening: We parse the URL to show host + pathname instead of the full URL with query parameters, keeping the output readable.
Cleanup: The original fetch is stored and restored when loop-detective disconnects.

Patching dns.promises.lookup

The promise-based DNS API wraps similarly:

if (dns.promises && dns.promises.lookup) {
  const origPromiseLookup = dns.promises.lookup;

  dns.promises.lookup = function patchedPromiseLookup(hostname, options) {
    const startTime = Date.now();
    const callerStack = captureCallerStack();

    return origPromiseLookup.call(dns.promises, hostname, options).then(
      (result) => {
        const duration = Date.now() - startTime;
        if (duration >= threshold) {
          recordSlowOp({
            type: 'dns', target: hostname, duration,
            timestamp: Date.now(), stack: callerStack,
          });
        }
        return result;
      },
      (err) => {
        const duration = Date.now() - startTime;
        if (duration >= threshold) {
          recordSlowOp({
            type: 'dns', target: hostname, duration,
            error: err.message, timestamp: Date.now(), stack: callerStack,
          });
        }
        throw err;
      }
    );
  };
}

This sits alongside the existing dns.lookup callback patch. Both are active simultaneously, covering code that uses either API style.

What the Output Looks Like

With both patches active, the report now catches everything:

🌐 Slow FETCH: 1820ms GET api.example.com/users → 200
    at loadUsers (/app/services/user.js:23:18)
    at handleRequest (/app/routes/api.js:45:5)

🌐 Slow HTTP: 950ms POST payment-gateway.com/charge → 200
    at processPayment (/app/services/payment.js:67:12)

🔍 Slow DNS: 2100ms lookup internal-service.local
    at Agent.createConnection (node:_http_agent:321:5)

🔌 Slow TCP: 1520ms db-server:5432
    at createConnection (/app/db.js:12:5)

  ⚠ Slow Async I/O Summary
    Total slow ops: 8

    🌐 FETCH — 3 slow ops, avg 1400ms, max 1820ms
      GET api.example.com/users
        3 calls, total 4200ms, avg 1400ms, max 1820ms

    🌐 HTTP — 2 slow ops, avg 900ms, max 950ms
      POST payment-gateway.com/charge
        2 calls, total 1800ms, avg 900ms, max 950ms

    🔍 DNS — 2 slow ops, avg 1800ms, max 2100ms
      internal-service.local
        2 calls, total 3600ms, avg 1800ms, max 2100ms

    🔌 TCP — 1 slow ops, avg 1520ms, max 1520ms
      db-server:5432
        1 calls, total 1520ms, max 1520ms

Notice that FETCH and HTTP are reported as separate types. This is intentional — it tells you which API surface the code is using, which matters when you're deciding how to add timeouts or caching.

The Full Coverage Map

Here's what node-loop-detective now tracks across Node.js versions:

I/O Type	What's Patched	Node.js Version	Covers
HTTP/HTTPS	`http.request`, `http.get`, `https.*`	All	axios, got, node-fetch, superagent, needle, etc.
Fetch	`globalThis.fetch`	18+	Built-in fetch, any code using the Fetch API
DNS (callback)	`dns.lookup`	All	Internal hostname resolution for http/https
DNS (promise)	`dns.promises.lookup`	10.6+	Modern async/await DNS code
TCP	`net.Socket.connect`	All	MySQL, PostgreSQL, Redis, MongoDB, AMQP, etc.

On Node.js 16-17, the fetch patch is silently skipped (it doesn't exist). On Node.js < 10.6, the dns.promises patch is skipped. Everything degrades gracefully.

A Note on undici Internals

You might wonder: if fetch() uses undici internally, and undici uses TCP sockets, wouldn't the net.Socket.connect patch catch undici's connections?

It depends. undici maintains a connection pool. If a connection is already established and pooled, a fetch() call reuses it without creating a new socket. The TCP patch only fires on new connections. So:

First fetch() to a new host: TCP connect is tracked + fetch timing is tracked
Subsequent fetch() calls to the same host: only fetch timing is tracked (connection is reused)

This is actually the correct behavior. You want to know "this fetch call took 1.8 seconds" regardless of whether it created a new connection or reused one. The fetch-level timing captures the full request lifecycle.

Upgrading

npm install -g node-loop-detective@1.5.0

# Track everything including fetch() and dns.promises
loop-detective <pid> --io-threshold 200

If your Node.js app uses fetch() or dns.promises, the slow calls that were previously invisible will now appear in the report. No configuration needed — the patches are applied automatically based on what's available in the target process.

Source: github.com/iwtxokhtd83/node-loop-detective

From O(n log n) to O(log p): Rebuilding the Order Book for Performance

Bill Tu — Wed, 08 Apr 2026 13:02:24 +0000

When we first built MatchEngine, the order book used a sorted slice. Every time an order was inserted, the entire slice was re-sorted. It was simple, correct, and easy to understand. It was also O(n log n) per insertion, which means it got slower with every order added to the book.

This article walks through the redesign: replacing the sorted slice with a price-level map backed by binary search insertion. The result is O(log p) insertion where p is the number of distinct price levels — typically 50-200 even when the book holds tens of thousands of orders.

The Original Design

The order book stored all orders in a flat slice, sorted by price-time priority:

type OrderBook struct {
    Bids []*model.Order  // highest price first
    Asks []*model.Order  // lowest price first
}

Insertion appended the order and re-sorted:

func (ob *OrderBook) AddOrder(order *model.Order) {
    ob.Asks = append(ob.Asks, order)
    sort.SliceStable(ob.Asks, func(i, j int) bool {
        if ob.Asks[i].Price.Equal(ob.Asks[j].Price) {
            return ob.Asks[i].Timestamp.Before(ob.Asks[j].Timestamp)
        }
        return ob.Asks[i].Price.LessThan(ob.Asks[j].Price)
    })
}

sort.SliceStable is O(n log n). For a book with 10,000 orders, every single insertion triggers a comparison-sort across all 10,000 elements. The 10,001st order is just as expensive to insert as rebuilding the entire book from scratch.

Cancellation was O(n) too — a linear scan to find the order by ID:

func removeFromSlice(orders []*model.Order, id string) []*model.Order {
    for i, o := range orders {
        if o.ID == id {
            return append(orders[:i], orders[i+1:]...)
        }
    }
    return orders
}

This was fine for a prototype. But the issue was clear: the data structure didn't match the access pattern.

The Access Pattern

A matching engine's order book has a very specific access pattern:

Orders cluster at price levels. A book with 10,000 orders might have only 100 distinct prices.
The most frequent operations are insert and best-price lookup.
Orders at the same price are processed in FIFO order.
Removal by ID needs to be fast (for cancellation).

The sorted slice treats every order as an independent element. It doesn't exploit the fact that orders group by price. It re-sorts everything even when the new order's price already exists in the book.

The New Design: Price-Level Map

The new architecture has three layers:

OrderBook
├── bids: orderSide (highest price first)
├── asks: orderSide (lowest price first)
└── orders: map[string]*Order (ID index)

orderSide
├── prices: []decimal.Decimal (sorted, binary search)
├── levels: map[string]*priceLevel (price -> queue)
└── count: int

priceLevel
└── orders: []*Order (FIFO queue)

Each side of the book (orderSide) maintains:

A sorted slice of distinct price keys
A map from price string to a priceLevel
A total order count

Each priceLevel is a simple FIFO queue — a slice of orders at that exact price, in arrival order.

Why This Structure?

It matches the access pattern exactly:

Insert at an existing price: O(1) map lookup + O(1) append to the queue. No sorting.
Insert at a new price: O(log p) binary search to find the insertion point in the price slice + O(p) shift to make room. This happens rarely compared to inserts at existing prices.
Best price: O(1) — it's prices[0].
Cancel by ID: O(1) map lookup to find the order, then O(k) scan within its price level (where k is typically small).

The Implementation

priceLevel: The FIFO Queue

The simplest component. Each price level holds orders in arrival order:

type priceLevel struct {
    orders []*model.Order
}

func newPriceLevel() *priceLevel {
    return &priceLevel{orders: make([]*model.Order, 0, 4)}
}

func (pl *priceLevel) append(order *model.Order) {
    pl.orders = append(pl.orders, order)
}

func (pl *priceLevel) front() *model.Order {
    if len(pl.orders) == 0 {
        return nil
    }
    return pl.orders[0]
}

The initial capacity of 4 is a small optimization — most price levels have a handful of orders, and this avoids the first few slice growths.

The removeFilled method uses an in-place compaction pattern to avoid allocating a new slice:

func (pl *priceLevel) removeFilled() int {
    n := 0
    for _, o := range pl.orders {
        if !o.IsFilled() {
            pl.orders[n] = o
            n++
        }
    }
    pl.orders = pl.orders[:n]
    return n
}

This overwrites filled orders with unfilled ones from left to right, then truncates. Zero allocations, single pass.

orderSide: Sorted Prices + Level Map

This is where the interesting logic lives. Each side of the book is an orderSide:

type orderSide struct {
    prices []decimal.Decimal
    levels map[string]*priceLevel
    less   func(a, b decimal.Decimal) bool
    count  int
}

The less function determines sort order — GreaterThan for bids (highest first), LessThan for asks (lowest first). This lets both sides share the same code with different comparators.

Insertion: Binary Search

The core of the optimization. When an order arrives:

func (s *orderSide) insert(order *model.Order) {
    key := order.Price.String()
    pl, exists := s.levels[key]
    if !exists {
        pl = newPriceLevel()
        s.levels[key] = pl
        idx := sort.Search(len(s.prices), func(i int) bool {
            return s.less(order.Price, s.prices[i]) || order.Price.Equal(s.prices[i])
        })
        // Insert price at idx
        s.prices = append(s.prices, decimal.Zero)
        copy(s.prices[idx+1:], s.prices[idx:])
        s.prices[idx] = order.Price
    }
    pl.append(order)
    s.count++
}

Two paths:

Path 1: Price level exists (common case). Map lookup is O(1). Append to the queue is O(1) amortized. Total: O(1).

Path 2: New price level (rare case). sort.Search does a binary search over the price slice — O(log p). Then we insert the price at the found index using the standard Go slice insertion pattern (append + copy). The copy is O(p) in the worst case, but p is the number of distinct prices, not the number of orders.

In a real order book, most orders arrive at prices that already exist. The common path is O(1). The rare path is O(log p + p). Compared to the old O(n log n) on every insert, this is a massive improvement.

Best Price: O(1)

func (s *orderSide) best() *model.Order {
    if len(s.prices) == 0 {
        return nil
    }
    key := s.prices[0].String()
    return s.levels[key].front()
}

The best price is always prices[0]. The best order at that price is the front of the FIFO queue. Two lookups, both O(1).

Removal

When an order is cancelled, the engine already knows the order (via the order ID index). The orderbook looks up the price level and removes the order from the queue:

func (s *orderSide) remove(order *model.Order) {
    key := order.Price.String()
    pl, exists := s.levels[key]
    if !exists {
        return
    }
    if pl.remove(order.ID) {
        s.count--
        if pl.len() == 0 {
            s.removePrice(key, order.Price)
        }
    }
}

If the price level becomes empty after removal, it is cleaned up — deleted from the map and removed from the price slice. This keeps the data structure compact.

Cleanup: removeFilled

After each matching cycle, filled orders need to be removed. The new implementation scans each price level, compacts it in-place, and collects empty levels for removal:

func (s *orderSide) removeFilled() []*model.Order {
    var filled []*model.Order
    pricesToRemove := make([]int, 0)

    for i, price := range s.prices {
        key := price.String()
        pl := s.levels[key]
        for _, o := range pl.allOrders() {
            if o.IsFilled() {
                filled = append(filled, o)
                s.count--
            }
        }
        remaining := pl.removeFilled()
        if remaining == 0 {
            delete(s.levels, key)
            pricesToRemove = append(pricesToRemove, i)
        }
    }

    // Remove empty price levels in reverse order to preserve indices
    for i := len(pricesToRemove) - 1; i >= 0; i-- {
        idx := pricesToRemove[i]
        s.prices = append(s.prices[:idx], s.prices[idx+1:]...)
    }

    return filled
}

The reverse-order removal of empty price indices is a subtle but important detail. Removing index 3 before index 7 would shift index 7 to 6, invalidating the stored index. Removing in reverse (7 first, then 3) avoids this.

The OrderBook: Putting It Together

The public OrderBook struct wires the two sides together:

type OrderBook struct {
    bids   *orderSide
    asks   *orderSide
    orders map[string]*model.Order
}

func New() *OrderBook {
    return &OrderBook{
        bids: newOrderSide(func(a, b decimal.Decimal) bool {
            return a.GreaterThan(b)  // highest first
        }),
        asks: newOrderSide(func(a, b decimal.Decimal) bool {
            return a.LessThan(b)     // lowest first
        }),
        orders: make(map[string]*model.Order),
    }
}

The orders map provides O(1) lookup by ID, used by RemoveOrder (cancellation) and HasOrder (duplicate detection).

One notable change: Bids and Asks are no longer public fields. They are now methods that return flattened slices:

func (ob *OrderBook) Bids() []*model.Order {
    return ob.bids.flatten()
}

This is a deliberate API change. The internal representation is no longer a flat slice, so exposing it as one would either leak the abstraction or require materializing the slice on every access. Making it a method makes the cost explicit.

Complexity Comparison

Operation	Sorted Slice (before)	Price-Level Map (after)
AddOrder (existing price)	O(n log n)	O(1)
AddOrder (new price)	O(n log n)	O(log p + p)
RemoveOrder	O(n)	O(1) + O(k)
BestBid / BestAsk	O(1)	O(1)
RemoveFilled	O(n)	O(n)
Snapshot	O(n)	O(n)

Variables: n = total orders, p = distinct price levels, k = orders at a given price.

The key insight is that p is much smaller than n. A liquid market might have 10,000 resting orders spread across 100 price levels. The old design paid O(10000 × log 10000) ≈ 130,000 comparisons per insert. The new design pays O(1) for the common case (existing price) or O(log 100 + 100) ≈ 107 operations for a new price level.

What Didn't Change

The matching algorithm is completely untouched. It still calls BestAsk(), checks the price, calls executeTrade(), and loops. The engine code changed only in one place — cleanFilled now calls book.Bids() and book.Asks() methods instead of accessing public fields.

The test suite passes without modification to test logic (only field access syntax changed to method calls). This is the benefit of a clean abstraction boundary — the order book's public API (AddOrder, RemoveOrder, BestBid, BestAsk, Depth, Spread) stayed the same.

Trade-offs

Nothing is free:

Memory: The price-level map uses more memory than a flat slice. Each price level has its own slice header, and the map has per-bucket overhead. For a book with 100 price levels and 10,000 orders, the overhead is roughly 100 × (slice header + map entry) ≈ a few KB. Negligible.

Flatten cost: Bids() and Asks() now allocate a new slice and copy all orders into it. The old design returned the slice directly (O(1)). This matters for Snapshot(), which calls flatten() twice. For a 10,000-order book, that is two allocations of 10,000 pointers. Acceptable for a snapshot operation that is already O(n).

Complexity: The code is more complex. Three types (orderSide, priceLevel, OrderBook) instead of one. The binary search insertion with slice shifting is trickier than sort.SliceStable. But the complexity is localized — the matching engine doesn't know or care about price levels.

Benchmarks

We added three benchmarks to measure the impact:

func BenchmarkInsertOnly(b *testing.B) {
    e := New(WithMaxTradeLog(0))
    price := decimal.NewFromInt(100)
    qty := decimal.NewFromInt(1)
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        id := fmt.Sprintf("s%d", i)
        e.SubmitOrder("BTC", model.NewLimitOrder(id, model.Sell, price, qty))
    }
}

BenchmarkInsertOnly: All orders at the same price. Tests the O(1) common path.
BenchmarkInsertAndMatch: Alternating buy/sell at the same price. Tests insert + match + cleanup.
BenchmarkInsertMultiplePriceLevels: Orders spread across 100 price levels. Tests the O(log p) path.

Run with:

go test -bench=. -benchmem ./pkg/engine/

The single-price insert benchmark is the most dramatic improvement. With the old sorted slice, each insert re-sorted the growing slice. With the new design, each insert is a map lookup + slice append — constant time regardless of book depth.

Conclusion

The optimization boils down to one idea: exploit the structure of the data. Orders in a matching engine cluster by price. A flat sorted list ignores this structure and pays O(n log n) to maintain global order. A price-level map respects the structure and pays O(1) for the common case.

The change touched two files (orderbook.go rewritten, pricelevel.go added) and required only syntax changes in consumers (book.Bids → book.Bids()). The matching algorithm, the engine, and all existing tests remained functionally identical.

For most open-source matching engines, this is the right level of optimization. It eliminates the obvious bottleneck without introducing the complexity of a red-black tree or skip list. If profiling later shows that the O(p) slice shift during new-price insertion is a problem, the prices slice can be replaced with a balanced tree — but that is a localized change within orderSide, invisible to the rest of the system.

GitHub: https://github.com/iwtxokhtd83/MatchEngine
Release: v0.3.0
Issue: #4 — Order book uses O(n log n) sort on every insert