Forem: Christopher Hiller

[Boost]

Christopher Hiller — Tue, 02 Sep 2025 20:48:19 +0000

naugtur

Aug 27 '25

Sometimes the best thing to do is an explicit nothing

Comments

4 min read

Testing in parallel with Mocha v8.0.0

Christopher Hiller — Wed, 10 Jun 2020 21:03:01 +0000

With the release of Mocha v8.0.0, Mocha now supports running in parallel mode under Node.js. Running tests in parallel mode allows Mocha to take advantage of multi-core CPUs, resulting in significant speedups for large test suites.

Read about parallel testing in Mocha’s documentation.

Before v8.0.0, Mocha only ran tests in serial: one test must finish before moving on to the next. While this strategy is not without benefits — it's deterministic and snappy on smaller test suites — it can become a bottleneck when running a large number of tests.

Let's take a look at how to take advantage of parallel mode in Mocha by enabling it on a real-world project: Mocha itself!

Installation

Node.js v8.0.0 has reached End-of-Life; Mocha v8.0.0 requires Node.js v10, v12, or v14.

Mocha doesn’t need to install itself, but you might. You need Mocha v8.0.0 or newer, so:

npm i mocha@8 --save-dev

Moving right along...

Use the `--parallel` flag

In many cases, all you need to do to enable parallel mode is supply --parallel to the mocha executable. For example:

mocha --parallel test/*.spec.js

Alternatively, you can specify any command-line flag by using a Mocha configuration file. Mocha keeps its default configuration in a YAML file, .mocharc.yml. It looks something like this (trimmed for brevity):

# .mocharc.yml
require: 'test/setup'
ui: 'bdd'
timeout: 300

To enable parallel mode, I'm going to add parallel: true to this file:

# .mocharc.yml w/ parallel mode enabled
require: 'test/setup'
ui: 'bdd'
timeout: 300
parallel: true

Note: Examples below use --parallel and --no-parallel for the sake of clarity.

Let's run npm test and see what happens!

Spoiler: It didn't work the first time

Oops, I got a bunch of "timeout" exceptions in the unit tests, which use the default timeout value (300ms, as shown above). Look:

  2) Mocha
       "before each" hook for "should return the Mocha instance":
     Error: Timeout of 300ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/boneskull/projects/mochajs/mocha/test/node-unit/mocha.spec.js)
      at Hook.Runnable._timeoutError (lib/runnable.js:425:10)
      at done (lib/runnable.js:299:18)
      at callFn (lib/runnable.js:380:7)
      at Hook.Runnable.run (lib/runnable.js:345:5)
      at next (lib/runner.js:475:10)
      at Immediate._onImmediate (lib/runner.js:520:5)
      at processImmediate (internal/timers.js:456:21)

That’s weird. I run the tests a second time, and different tests throw "timeout" exceptions. Why?

Because of many variables — from Mocha to Node.js to the OS to the CPU itself — parallel mode exhibits a much wider range of timings for any given test. These timeout exceptions don't indicate a newfound performance issue; rather, they're a symptom of a naturally higher system load and nondeterministic execution order.

To resolve this, I'll increase Mocha's default test timeout from 300ms (0.3s) to 1000ms (1s):

# .mocharc.yml
# ...
timeout: 1000

Mocha's "timeout" functionality is not to be used as a benchmark; its intent is to catch code that takes an unexpectedly long time to execute. Since we now expect tests to potentially take longer, we can safely increase the timeout value.

Now that the tests pass, I'm going to try to make them pass more.

Optimizing parallel mode

By default, Mocha's maximum job count is n - 1, where n is the number of CPU cores on the machine. This default value will not be optimal for all projects. The job count also does not imply that "Mocha gets to use n - 1 CPU cores," because that's up to the operating system. It is, however, a default, and it does what defaults do.

When I say "maximum job count" I mean that Mocha could spawn this many worker processes if needed. It depends on the count and execution time of the test files.

To compare performance, I use the friendly benchmarking tool, hyperfine; I'll use this to get an idea of how various configurations will perform.

Regarding hyperfine usage:

In the below examples, I’m passing two options to hyperfine: -r 5 for "runs," which runs the command five (5) times. The default is ten (10), but this is slow, and I'm impatient. The second option I pass is --warmup 1, which performs a single "warmup" run. The result of this run is discarded. Warmup runs reduce the chance that the first k runs will be significantly slower than the subsequent runs, which may skew the final result. If this happens, hyperfine will even warn you about it, which is why I'm using it!

If you try this yourself, you need to replace bin/mocha with node_modules/.bin/mocha or mocha, depending on your environment; bin/mocha is the path to the mocha executable relative to the working copy root.

Mocha's integration tests (about 260 tests over 55 files) typically make assertions about the output of the mocha executable itself. They also need a longer timeout value than the unit tests; below, we use a timeout of ten (10) seconds.

I run the integration tests in serial. Nobody ever claimed they ran at ludicrous speed:

$ hyperfine -r 5 --warmup 1 "bin/mocha --no-parallel --timeout \
10s test/integration/**/*.spec.js"
Benchmark #1: bin/mocha --no-parallel --timeout 10s test/integration/**/*.spec.js
  Time (mean ± σ):     141.873 s ±  0.315 s    [User: 72.444 s, System: 14.836 s]
  Range (min … max):   141.447 s … 142.296 s    5 runs

That's over two (2) minutes. Let’s try it again in parallel mode. In my case, I have an eight-core CPU (n = 8), so by default, Mocha uses seven (7) worker processes:

$ hyperfine -r 5 --warmup 1 "bin/mocha --parallel --timeout 10s \
test/integration/**/*.spec.js"
Benchmark #1: bin/mocha --parallel --timeout 10s test/integration/**/*.spec.js
  Time (mean ± σ):     65.235 s ±  0.191 s    [User: 78.302 s, System: 16.523 s]
  Range (min … max):   65.002 s … 65.450 s    5 runs

Using parallel mode shaves 76 seconds off of the run, down to just over a minute! That's almost a 53% speedup. But, can we do better?

I can use the --jobs/-j option to specify exactly how many worker processes Mocha will potentially use. Let's see what happens if I reduce this number to four (4):

$ hyperfine -r 5 --warmup 1 "bin/mocha --parallel --jobs 4 --timeout 10s \
test/integration/**/*.spec.js"
Benchmark #1: bin/mocha --parallel --jobs 4 --timeout 10s \
test/integration/**/*.spec.js
  Time (mean ± σ):     69.764 s ±  0.512 s    [User: 79.176 s, System: 16.774 s]
  Range (min … max):   69.290 s … 70.597 s    5 runs

Unfortunately, that's slower. What if I increased the number of jobs, instead?

$ hyperfine -r 5 --warmup 1 "bin/mocha --parallel --jobs 12 --timeout 10s \
test/integration/**/*.spec.js"
Benchmark #1: bin/mocha --parallel --jobs 12 --timeout 10s test/integration/**/*.spec.js
  Time (mean ± σ):     64.175 s ±  0.248 s    [User: 80.611 s, System: 17.109 s]
  Range (min … max):   63.809 s … 64.400 s    5 runs

Twelve (12) is ever-so-slightly faster than the default of seven (7). Remember, my CPU has eight (8) cores. Why does spawning more processes increase performance?

I speculate it's because these tests aren't CPU-bound. They mostly perform asynchronous I/O, so the CPU has some spare cycles waiting for tasks to complete. I could spend more time trying to squeeze another 500ms out of these tests, but for my purposes, it's not worth the bother. Perfect is the enemy of good, right? The point is to illustrate how you can apply this strategy to your own projects and arrive at a configuration you're happy with.

When to avoid parallel mode

Would you be shocked if I told you that running tests in parallel is not always appropriate? No, you would not be shocked.

It's important to understand two things:

Mocha does not run individual tests in parallel. Mocha runs test files in parallel.
Spawning worker processes is not free.

That means if you hand Mocha a single, lonely test file, it will spawn a single worker process, and that worker process will run the file. If you only have one test file, you'll be penalized for using parallel mode. Don't do that.

Other than the "lonely file" non-use-case, the unique characteristics of your tests and sources will impact the result. There's an inflection point below which running tests in parallel will be slower than running in serial.

In fact, Mocha's own unit tests (about 740 tests across 35 files) are a great example. Like good unit tests, they try to run quickly, in isolation, without I/O. I'll run Mocha's unit tests in serial, for the baseline:

$ hyperfine -r 5 --warmup 1 "bin/mocha --no-parallel test/*unit/**/*.spec.js"
Benchmark #1: bin/mocha --no-parallel test/*unit/**/*.spec.js
  Time (mean ± σ):      1.262 s ±  0.026 s    [User: 1.286 s, System: 0.145 s]
  Range (min … max):    1.239 s …  1.297 s    5 runs

Now I'll try running them in parallel. Despite my hopes, this is the result:

$ hyperfine -r 5 --warmup 1 "bin/mocha --parallel test/*unit/**/*.spec.js"
Benchmark #1: bin/mocha --parallel test/*unit/**/*.spec.js
  Time (mean ± σ):      1.718 s ±  0.023 s    [User: 3.443 s, System: 0.619 s]
  Range (min … max):    1.686 s …  1.747 s    5 runs

Objectively, running Mocha's unit tests in parallel slows them down by about half a second. This is the overhead of spawning worker processes (and the requisite serialization for inter-process communication).

I'll go out on a limb and predict that many projects having very fast unit tests will see no benefit from running those tests in Mocha's parallel mode.

Remember my .mocharc.yml? I yanked that parallel: true out of there; instead, Mocha will use it only when running its integration tests.

In addition to being generally unsuitable for these types of tests, parallel mode has some other limitations; I'll discuss these next.

Caveats, disclaimers and gotchas, Oh my

Due to technical limitations (i.e., "reasons"), a handful of features aren't compatible with parallel mode. If you try, Mocha will throw an exception.

Check out the docs for more information and workarounds (if any).

Unsupported reporters

If you're using the markdown, progress, or json-stream reporters, you're out of luck for now. These reporters need to know how many tests we intend to execute up front, and parallel mode does not have that information.

This could change in the future, but would introduce breaking changes to these reporters.

Exclusive tests

Exclusive tests (.only()) do not work. If you try, Mocha runs tests (as if .only() was not used) until it encounters usage of .only(), at which point it aborts and fails.

Given that exclusive tests are typically used in a single file, parallel mode is also unsuitable for this situation.

Unsupported options

Incompatible options include --sort, --delay, and importantly, --file. In short, it's because we cannot run tests in any specific order.

Of these, --file likely impacts the greatest number of projects. Before Mocha v8.0.0, --file was recommended to define "root hooks." Root hooks are hooks (such as beforeEach(), after(), setup(), etc.) which all other test files will inherit. The idea is that you would define root hooks in, for example, hooks.js, and run Mocha like so:

mocha --file hooks.js "test/**/*.spec.js"

All --file parameters are considered test files and will be run in order and before any other test files (in this case, test/**/*.spec.js). Because of these guarantees, Mocha "bootstraps" with the hooks defined in hooks.js, and this affects all subsequent test files.

This still works in Mocha v8.0.0, but only in serial mode. But wait! Its use is now strongly discouraged (and will eventually be fully deprecated). In its place, Mocha has introduced Root Hook Plugins.

Root Hook Plugins

Root Hook Plugins are modules (CJS or ESM) which have a named export, mochaHooks, in which the user can freely define hooks. Root Hook Plugin modules are loaded via Mocha's --require option.

Read the docs on using Root Hook Plugins

The documentation (linked above) contains a thorough explanation and more examples, but here’s a straightforward one.

Say you have a project with root hooks loaded via --file hooks.js:

// hooks.js
beforeEach(function() {
  // do something before every test
  this.timeout(5000); // trivial example
});

To convert this to a Root Hook Plugin, change hooks.js to be:

// hooks.js
exports.mochaHooks = {
  beforeEach() {
    this.timeout(5000);
  }
};

Tip: This can be an ESM module, for example, hooks.mjs; use a named export of mochaHooks.

When calling the mocha executable, replace --file hooks.js with --require hooks.js. Nifty!

Troubleshooting parallel mode

While parallel mode should just work for many projects, if you're still having trouble, refer to this checklist to prepare your tests:

✅ Ensure you are using a supported reporter.
✅ Ensure you are not using other unsupported flags.
✅ Double-check your config file; options set in config files will be merged with any command-line option.
✅ Look for root hooks (they look like this) in your tests. Move them into a root hook plugin.
✅ Do any assertion, mock, or other test libraries you're consuming use root hooks? They may need to be migrated for compatibility with parallel mode.
✅ If tests are unexpectedly timing out, you may need to increase the default test timeout (via --timeout)
✅ Ensure your tests do not depend on being run in a specific order.
✅ Ensure your tests clean up after themselves; remove temp files, handles, sockets, etc. Don't try to share state or resources between test files.

What's next

Parallel mode is new and not perfect; there's room for improvement. But to do so, Mocha needs your help. Send the Mocha team your feedback! Please give Mocha v8.0.0 a try, enable parallel mode, use Root Hook Plugins, and share your thoughts.

Introducing report-toolkit for Node.js Diagnostic Reports

Christopher Hiller — Mon, 06 Jan 2020 18:51:11 +0000

In this article, I introduce you to report-toolkit, show you its coolest features, and help you get up and running with the new technology. Since Diagnostic Reports is a relatively new feature in Node.js and is still considered experimental, I'll start with a brief overview.

Node.js Diagnostic Reports: The basics

In this section, I explain what Diagnostic Reports are and how to generate them.

For a more in-depth overview, check out Easily Identify Problems in your Node.js Apps with Diagnostic Report, written by Gireesh Punathil, a Node.js TSC member.

What are Node.js Diagnostic Reports?

Originally introduced in Node.js v11.7.0 as an experimental feature, a Diagnostic Report is a JSON file (or JavaScript object) containing a diagnostic summary of a Node.js process. Diagnostic Reports are especially helpful for post-mortem debugging or situations in which it's impractical to attach a debugger.

A Diagnostic Report's summary contains information about the state of a node process, including:

Process information (static information which lives in the Process object) including the version of Node.js and the versions of its dependencies
Operating system, platform, and CPU information
The state of the JavaScript and native (C++) stacks
Memory and resource usage
The state of the libuv event loop
Environment variables
Shared libraries
Metadata about the report file itself

As of this writing (the current version of Node.js is v13.5.0), Diagnostic Reports is still considered experimental. Practically speaking, that means the usage, API, or output may introduce breaking changes at any time in both the LTS (v12.x) and current (v13.x) release lines.

That being said, the underlying implementation has proved itself robust in the months since its arrival, and I don’t expect it to be a source of problems for early adopters.

Next, I show you the magic spell you need to generate a report. And by magic spell, I mean command-line flags.

How do I generate a Node.js Diagnostic Report?

For this section, I recommend you use a version of Node.js between v12.5.0 and v13.1.0, inclusive. The API and output has not changed within this range. Newer versions may break the following assumptions, due to Diagnostic Reports' experimental status.

As with other experimental features, you need to supply node a flag. That flag is --experimental-report. In your shell, use it like so:

node --experimental-report <my-script.js>

The above command configures the node process to expose the global process.report API, and allows you to use other command-line flags to configure automatic report generation.

The most straightforward way to create a Diagnostic Report is to use the API. You can use node to run an inline script which calls process.report.writeReport(), like so:

node --experimental-report --eval "process.report.writeReport('report.json')"

You will see console output (on STDERR) like this:

Writing Node.js report to file: report.json
Node.js report completed
(node:66875) ExperimentalWarning: report is an experimental feature. This feature could change at any time

A file, report.json, will now live in your current directory. Open it in your editor, take a gander, and ponder the arcane wisdom contained therein.

You should now have a basic grasp of the what, why and how of Diagnostic Reports in Node.js. Given these fundamentals, you’ll better understand the motivation behind report-toolkit.

Introducing report-toolkit for real this time

While Diagnostic Reports are useful in solving a certain class of problems, the reports themselves can be impenetrable and tedious to work with. With feedback from the Node.js community, I designed report-toolkit to pave over the common speedbumps, and arrive more quickly at solutions.

report-toolkit is packaged as a command-line app (rtk), providing subcommands (think git) which map to a set of unique, purpose-built features. I’m going to start with what I believe is the most rad feature: automated analysis.

Analyze Diagnostic Reports with report-toolkit

Deep in the forest of a production filesystem, a developer happens upon a Diagnostic Report file. Taking stock of the surroundings, the developer discovers a process is not running. Experience and reasoning leads the developer to deduce, “These are the remains of a Node.js process.”

What happened here? How did this process meet its untimely demise? report-toolkit can help with the investigation.

rtk’s inspect feature runs a set of rules on one or more reports. Each rule is a function, and aims to provide analysis which is “good enough” — in other words, these are heuristics. If a function finds something bad, smelly, or otherwise dubious, rtk inspect will report this to the developer.

These rules are inspired by ESLint’s concept of a rule. Each rule is modular, each rule is configurable, and best of all: you can write your own and share!

report-toolkit ships with a small set of built-in rules to recognize some known problems — these include CPU and resource usage thresholds, long timeouts in the event loop, and mismatched shared library versions. As more developers adopt Diagnostic Reports (and, hopefully, report-toolkit) to solve problems, we aim to ship widely applicable heuristics as they are uncovered by the community. For other more environment-specific needs, rtk can use heuristics published as third-party modules (as “plugins”) or even just a script on disk.

Let's look at rtk inspect in action.

The following image is the output from running rtk inspect on a report file which notes that the system shared libraries in use by openssl (libcrypto and libssl) are not from the version of openssl which Node.js expects. It expects v1.1.1b, but it has been compiled against v1.1:

As another example, the rtk inspect element shows a Diagnostic Report that indicates suspiciously high CPU usage:

The above check takes the number of CPU cores into account, which otherwise would require the developer to do math, and nobody wants to do that.

Please stay tuned for tutorials on creating your own plugins!

The next feature we'll look at is less flashy, but extremely helpful: redaction of secrets.

Automatically redact secrets from Diagnostic Reports using report-toolkit

As mentioned above, Diagnostic Report contains the entire contents of your environment variables. This includes things like cloud provider API keys, tokens, session IDs, etc. The data is in a .json file, and as we are all careful and security-conscious developers, we must take great care of how and where we transmit this file.

You may be tempted to hand-edit these secrets out of the file, but
report-toolkit provides a redact feature, which — as you may have guessed — redacts commonly known secrets and phrases from the environment variables in a Diagnostic Report file.

Secrets include those used by major cloud providers, including IBM Cloud, AWS, Azure, and Google Cloud.

If you like to look at regular expressions, you can take a look at the matching.

Instead of obliterating the key outright, the value is replaced with the string [REDACTED].

Let's look at this feature in action. Take a Diagnostic Report containing the following:

{
  "environmentVariables": {
    "CLOUD_PROVIDER_ACCESS_KEY": "MY_SUPER_SECRET_KEY"
  }
}

Now, run rtk redact /path/to/report.json. rtk will dump the entire report to your terminal. The result will contain:

{
  "environmentVariables": {
    "CLOUD_PROVIDER_ACCESS_KEY": "[REDACTED]"
  }
}

I recommend using the --replace option to rtk redact, which overwrites the file in-place instead of printing to STDOUT. If you’re leery about that, try --output <new-filename>, which will create a new file from the redacted Diagnostic Report.

Another task report-toolkit helps with is comparison of Diagnostic Reports.

Comparing Diagnostic Reports with report-toolkit

Suppose you have two Diagnostic Reports generated from a single node process. Or two reports generated from the same script, but on different machines. Or two reports from different processes on the same machine. Or whatever—you have two reports, OK?

You could use diff report-1.json report-2.json. Or diff it in vim or some other GUI tool. That will (eventually) get the job done. But these tools weren’t made to compare Diagnostic Reports; they were made to compare arbitrary text files and source code.

rtk, on the other hand, provides a diff command purpose-built for Diagnostic Reports. It hides generally uninteresting information (certain timestamps and JSON syntax) and provides a helpful side-by-side view, noting what has been added, modified, or removed between the left and right reports.

Example output from rtk diff <report-1.json> <report-2.json> looks like this:

Above, note the differences between header.processId (good for checking if the same process created both reports), the current working directory, the command-line flags to node, and finally the different versions of Node.js used to create the reports.

rtk diff allows explicit including and excluding of field names (using “dot” syntax), in case there’s something you want to zero in on, or ignore; rtk diff report-1.json report-2.json -i header.nodejsVersion would only compare the Node.js version values.

Having worked with Diagnostic Report files, Alice & Bob notice the files contain much JSON; each is around 25KiB. Alice and Bob may not need all that information, or maybe they need it in a different format; this is a problem report-toolkit helps solve via transformers. Read on, Alice and Bob!

Transformation of Diagnostic Reports with report-toolkit

Transformers, in report-toolkit parlance, can be thought of as “mapping” functions. When performing a transformation, report-toolkit starts with a Diagnostic Report, maps it through one or more transformers, and finally produces output.

report-toolkit ships with a handful of built-in transformers intended for general-purpose use. However, I’d like to train the spotlight on a transformer tailored to a specific use case: identification of unique uncaught exceptions.

When an uncaught exception is thrown in Node.js, best practices recommend that the process does not attempt to resume normal operation. Instead, it should exit. A typical recovery strategy, then, is to just restart the process. The service is back online, and a developer can begin a post-mortem investigation as appropriate.

To aid in post-mortem debugging, Node.js can be configured to automatically generate a report in the case of an uncaught exception (using --report-uncaught-exception).

The reason behind any given uncaught exception may very well be a bug — but it also might be due to circumstances outside of developer control, such as network outages or Martian invasion. By examining the Diagnostic Report and its stack trace, a developer can identify an exception as “known” or, less charitably, “somebody else’s problem.”

Yet, this doesn’t answer the questions: “How many of these are out of my control and how often?” or “How many JIRA tickets do I need to create and assign to myself?”

To count chickens, one must know a chicken.

report-toolkit can help developers count chickens using the stack-hash transformer. This transformer computes a SHA1 hash of the exception—establishing unique exceptions — and outputs the hash along with the complete stack trace. Pipe this into a data store, and now you’ve got some lovely metrics for your favorite dashboard.

The output looks something like this:

{
  "dumpEventTime": "2019-11-21T15:18:47Z",
  "filepath": "report.json",
  "message": "Error: your problem",
  "sha1": "9c1d91a8e0f6944e0c0bc920c55e64145c3823a8",
  "stack": [
    "at Object.<anonymous> (/path/to/script.js:1:7)",
    "at Module._compile (internal/modules/cjs/loader.js:956:30)",
    "at Object.Module._extensions..js (internal/modules/cjs/loader.js:973:10)",
    "at Module.load (internal/modules/cjs/loader.js:812:32)",
    "at Function.Module._load (internal/modules/cjs/loader.js:724:14)",
    "at Function.Module.runMain (internal/modules/cjs/loader.js:1025:10)"
  ]
}

In a future release, report-toolkit will allow a user to customize which information is used to compute the hash.

We've just scratched the surface of transformers in report-toolkit. To learn more — and see a list of built-in transformers — check out report-toolkit’s quick start guide.

Conclusion

Ideally, this article gave you an understanding of the fundamentals of Diagnostic Reports in Node.js, and the ways in which report-toolkit can help you use them more effectively to solve problems. Yes.

Give feedback about report-toolkit

report-toolkit is a brand new (announced October 2019) Apache-2.0-licensed OSS project from IBM, created and maintained by me, Christopher “boneskull” Hiller.

While I'm busy writing more docs, I hope you can give it a try — I’d love your feedback.
These are my questions for you:

What worked well?
What didn’t work well? How could it be better?
Found a bug? Any feature requests?
Other questions?

Please drop an issue in report-toolkit's issue tracker. All contributions are welcome!

This article originally appeared December 19, 2019 on developer.ibm.com.

Recursive Directory Removal in Node.js

Christopher Hiller — Mon, 09 Sep 2019 17:14:50 +0000

Recursive directory removal has landed in Node.js v12.10.0!

This has been a long-standing feature request. New Node.js developers often express incredulity when they discover this particular “battery” isn’t included in Node.js.

Over the years, userland modules (rimraf, rmdir, del, fs-extra, etc.) have heroically provided what core did not. Thanks to the superbad maintainers-of and contributors-to these packages!

Here's a little story about how it came to pass—and why something so seemingly simple as rm -rf isn’t necessarily so.

About Node.js’ Filesystem Operations

First, I want to explain a bit about how Node.js works under the hood with regards to filesystem operations.

libuv provides filesystem operations to Node.js. Node.js’ fs module is just a JavaScript file which provides the fs.* APIs; those APIs call into an internal C++ binding (you could think of this as a “native module”). That binding is glue between libuv and the JavaScript engine ( V8 ).

Here’s an example. At the lowest level, libuv provides a C API (uv_fs_rmdir) to make the a system call to remove a directory.

const fs = require('fs');

// `rmdir` is just a function which calls into a C++ binding.
// The binding asks libuv to remove the "/tmp/foo" directory.
// Once libuv returns a result, the binding calls `callback`
fs.rmdir('/tmp/foo', function callback(err) {
  if (err) {
    // handle error
  }
});

Importantly, Node.js makes only a single call to libuv above_._

In fact, until recently, Node.js’ fs bindings follow a pattern: single calls into libuv. fs.readFile, fs.stat, fs.unlink; these are all just one call.

Oh, that recent change? It was recursive fs.mkdir. I’ll explain what makes it different.

Shell Operations vs. System Operations

Developers may not think about this much because it’s so well-abstracted by our tools. Take mkdir, for example:

$ mkdir ./foo

mkdir is a command-line utility (which flavor, exactly, depends on your operating system). It’s not a system call. The above command may only execute a single system call, but the following may execute several:

# creates dirs foo, then bar, then baz, ignoring dirs that already exist
$ mkdir -p ./foo/bar/baz

Unless our tools have transactional behavior—they can “commit” or “roll back” operations—it’s possible for this command to partially succeed (though maybe not obvious in this case, but trust me).

What happens if mkdir -p fails halfway through? It depends. You get zero or more new directories. Yikes!

If that seems weird, consider that the user may want to keep the directories it did create. It’s tough to make assumptions about this sort of thing; cleanup is best left to the user, who can deal with the result as they see fit.

How does this relate to Node.js? When a developer supplies the recursive: true option to fs.mkdir, Node.js will potentially ask libuv to make several system calls—all, some, or none of which may succeed.

Previous to the addition of recursive fs.mkdir, Node.js had no precedent for this behavior. Still, its implementation is relatively straightforward; when creating directories, the operations must happen both in order and sequentially—we can’t create bar/baz/ before we create bar/!

It may be surprising, then, that a recursive rmdir implementation is another beast entirely.

There Was An Attempt

I was likely not the first to attempt to implement a recursive rmdir in Node.js at the C++ level, but I did try, and I’ll explain why it didn’t work.

The idea was that a C++ implementation could be more performant than a JavaScript implementation—that’s probably true!

Using mkdir as a template, I began coding. My algorithm would perform a depth-first traversal of the directory tree using libuv ’s uv_fs_readdir; when it found no more directories to descend into, it would call uv_fs_unlink on each file therein. Once the directory was clear of files, it would ascend to the parent, and finally remove the now-empty directory.

It worked! I was very proud of myself. Then I decided to run some benchmarks against rimraf. Maybe I shouldn't have!

I found out that my implementation was faster for a very small N, where N is the number of files and directories to remove. But N didn’t have to grow very large for userland's rimraf to overtake my implementation.

Why was mine slower? Besides using an unoptimized algorithm, I used recursive mkdir as a template, and mkdir works in serial (as I mentioned above). So, my algorithm only removed one file at a time. rimraf , on the other hand, queued up many calls to fs.unlink and fs.rmdir. Because libuv has a thread pool for filesystem operations, it could speedily blast a directory full of files, only limited by its number of threads!

Note that when we say Node.js is single-threaded, we're talking about the programming model. Under the hood, I/O operations are multithreaded.

At this point, I realized that if it was going to be “worth it” to implement at the C++ layer—meaning a significant performance advantage which outweighs-the- maintenance-costs-of-more-C++-code—I’d have to rewrite the implementation to manage its own thread pool. Of course, there’s no great precedent for that in Node.js either. It’d be possible, but very tricky, and best left to somebody with a better handle on C++ and multithreaded programming.

I went back to the Node.js tooling group and explained the situation. We decided that the most feasible way forward would be a pure-JavaScript implementation of recursive directory removal.

Let’s Write It In JavaScript!

Well, that was the idea, but we didn’t get very far. We took a look at the source of rimraf, which is the most popular userland implementation. It’s not as straightforward as you’d expect! It covers many edge cases and peculiarities (and all of those hacks would need to be present in a Node.js core implementation; it needs to work like a consumer would expect).

Furthermore, rimraf is stable, and these workarounds have proven themselves to be robust over the years that it’s been consumed by the ecosystem.

I won’t attempt to explain what rimraf must do to achieve decent performance in a portable manner—but rest assured it’s sufficiently non-trivial. So non-trivial, in fact, that it made more sense to just pull rimraf into Node.js core instead of trying to code it again from scratch.

So that’s what we did.

It’s Just rimraf

Ian Sutherland extracted the needed code from rimraf. In particular, rimraf supplies a command-line interface, and we didn’t need that. For simplicity (and to eliminate dependencies) glob support (e.g., foo/**/*.js) was also dropped (though it may still have a future). After this, it was a matter of integrating it into a Node.js-style API, and the needed docs and tests.

To be clear, recursive directory removal in Node.js does not make rimraf obsolete. It does mean that for many use cases, Node.js’ fs.rmdir can get the job done. Stick with rimraf if you need globs or a portable command-line utility.

Thanks to Isaac Schlueter for rimraf —and to bless Node.js’ copy-and-paste efforts.

In Conclusion

That’s the story of Node.js’ recursive rmdir thus far. Want to help write the rest? Come participate in the Node.js Tooling Group, where we’re looking to make Node.js the best platform it can be for building CLI apps.

Acknowledgements to Isaac Schlueter and Ian Sutherland for reviewing this post.

This post originally appeared on boneskull.com on Sep 9, 2019.

PRO TIPS for devs working at home

Christopher Hiller — Tue, 11 Jun 2019 19:13:17 +0000

Having spent the better part of the last decade as a work-from-home developer, I have discovered or adopted a few LIFE HACKS which I am going to share with you now. In no way am I claiming that these WEIRD TRICKS will work for everyone; however, they work for me.

I want to also be clear: unless I'm traveling, I work almost exclusively out of my home office, and this LISTICLE is written from such a perspective. Here goes...

1. You need a home office

Look, we both know you can code while sitting in front of the TV just fine. But if you're looking to reach new, towering heights of productivity, you're going to want a home office. Because:

Health

Despite being called a "laptop", your lap makes for profoundly un-ergonomic computing. Your lap, the couch, the bed, the floor—you can work in these places for short sprints. But it's hard on your eyes, your wrists, your back, your neck, your spleen, and you could catch a cold that way.

Psychology

There's some sort of psychological THING that makes it so you're more likely to focus on work when you are in a working space. No, I'm not a psychologist. But I know that when I sit behind the wheel of a car, I am going to be focused on driving.

Control

You wield total control over your environment. Your office will have a door which you can—and often do—keep shut. You will avoid interruption by your terrible family, inconsiderate roommates, or bumbling henchmen. If you want the lights on, turn them on. If you want a window open, open it.

2. Professionals need professional tools

Contractors don't build houses with cut-rate power tools. You shouldn't skimp on your equipment either.

Meetings

Unfortunately, you will be videoconferencing. You need some headphones, a decent microphone, and—often neglected—an OS that "just works" with these and your videoconferencing app of choice. A "decent microphone" is subjective, but expect to pay around $50 USD minimum. You will also want a boom (the extendable "arm" thing) so you can position the microphone correctly; don't just set it on the desk a couple feet from your face! The headphones can be crap, but using them will mean that any echoing or feedback on your conference calls won't be your fault.

Finally, if possible, use a gigabit ethernet wired connection to reduce latency on calls.

Ergonomics

Get an adjustable sit-stand desk or converter. Get an external keyboard and trackpad/mouse with wrist rests. Use an external monitor & laptop stand. Ergonomics!!

3. In the morning, BEGIN; when you get to the end, STOP

The biggest complaint I hear from those unaccustomed to working from home is the struggle to separate work from home. A home office absolutely helps, but it's only part of a solution.

BEGIN-ing

You should have some sort of schedule. While you may have extra flexibility with when you work, pick some "core hours." Work with your teammates/manager to find what works best, and announce when you will be available.

STOP-ing

At the end of your workday, physically shut your laptop. Unplug it and put it away and don't open it until tomorrow. If you want to work on personal projects or use a computer for games or movies or whatever, use a personal computer, not the one your employer gave you.

Separation

Yes! You should have your own computer. Not only can it help with managing work/life balance, it works the other way, by keeping your work machine free of personal data and distractions. Also, "your own computer" becomes crucial if you happen to be "between jobs."

4. Some other good habits

Because I'm starting to sound like a self-help guru here, let me add: do what you can. These are not rules, and you should not feel bad if they don't stick or if they don't work for you.

Get dressed, but comfort should be top priority. Ideally, this is something you wouldn't be embarrassed to leave the house in, but feel free to reassess what you consider to be shameful.
Often, those unaccustomed to working from home will feel there are too many distractions. Personally, I don't feel I have a problem with this, but if I had to pick the biggest distraction, it'd be the bed. The bed is in your house. And it's comfy. If you fall prey to its siren call, it should be with intent (e.g., a short, planned "power nap", complete with alarm). To discourage casual use, make the bed before beginning work for the day.
Get up and move around often, even for just a few minutes. Walk out and get the mail, take the dog to pee, water some plants, etc.
Don't eat in your office. Eat in the kitchen. In other words, don't work through lunch. When you sit back down to revisit whatever it was you were hacking on, this can give you a fresh perspective.
You (probably) aren't wearing an ankle monitor. So if you feel your social needs aren't being met, for crying out loud, leave the house! Go work in a coffee shop or shared workspace or something. Schedule meetups with other remote teammates. If there's an actual office nearby, give it a visit.

5. There was an attempt

If you currently work in an office and have the option to work from home, I'd strongly suggest easing in to it. Start with a day or two a week, try some of the above suggestions, and go from there.

Despite having given it time and effort, you might still feel unproductive or just downright lonely. This is not unheard of! Working from home—or even just working "remotely"—is not for everyone. Reaching this conclusion doesn't mean you've failed; it means you've learned about yourself and what you need from a job.

I hope these ideas are helpful! Could you at least put some pants on, though?

Oh, and if you think these tips are not awful, I also spoke them with my mouth on JS Party #76 alongside Emma Wedekind and KBall. Take a listen.

You don’t have to dress up

JS Party

Your browser does not support the audio element.

1x initializing... ×