Forem: Nicholas Cloud

Goodbye Evernote

Nicholas Cloud — Sun, 31 Jan 2021 23:09:41 +0000

(This article originally published at nicholascloud.com.)

I've been a paying Evernote user for years; and before that, a "free" Evernote user for even longer. Evernote has some seriously powerful features, among which is the Evernote Web Clipper extension available on all major browsers, the excellent PDF markup features, flawless sync across devices, etc. It is solid software, backed by a solid service.

But I've left Evernote, likely for good.

In the wake of efforts by "Big Tech" companies to censor, deplatform, or control the data that belongs to customers, I've been cutting ties with as many Big Tech companies as I'm able. And Evernote, though I have never experienced any negative service from them, Evernote does not offer end-to-end encryption for it's services (meaning notes stored on Evernote servers are accessible by Evernote employees), and that has become a deal-breaker for me. I value my privacy, and my personal notes are where I can explore ideas and record my thoughts. And I don't want to use any service that doesn't respect and protect that.

But what to do with GIGABYTES of notes, clipped articles, recipes, photos, and annotated PDFs?

When I ditched GMail for ProtonMail, it took me months to dig through all my archived mail. I deleted, exported, or printed each piece, then had to update the sender's settings with my new email address (or unsubscribe, if it was a newsletter). I transferred all of my contacts to ProtonMail, then reviewed them all to put the exported information into ProtonMail's custom contact fields. It was a long, tedious, tiresome process, but I did it.

I expected my transition away from Evernote to be just as challenging. I came up with a list of goals for my new notebook scheme:

My notes should be plain text (well, technically Markdown) files that link to any relevant external assets, such as images, PDF files, etc.
Markdown files and assets should be organized in a uniform way.
Markdown files should have a uniform naming convention (all lower-case words, separated by hyphens).
I should be able to easily search for note content.
My notes should be available on multiple devices.
I should be able to clip content from the web and easily add it as a Markdown file to my notebook.

Step 1: Get my notes out of Evernote

There are two Evernote applications you can use on your device: the slick, newer version, and the old, legacy version. The new version looks nice, but it dropped a significant feature I thought would be available to me, and that is the ability to export all notes at once in HTML format. Even the legacy version seems to lack this feature (maybe just on OSX?), though in the legacy version you can still export individual notebooks as HTML collections. In the newer version you can only export notebooks as Evernote's own ENEX file format (a kind of XML archive of notebook content). This seemed like it was going to be a show-stopper, since I had no clue how I would convert ENEX files into Markdown files, but a friend pointed me to the excellent utility evernote2md which does exactly that. Since I had 132 individual notebooks, it took me a while to export them all, then to convert them to Markdown with this utility, but once done, I had all of my saved notes in Markdown format (along with their attachments).

The total size of my exported Markdown notes and attachments is around 2GB.

Step 2: Fixing file names

I noticed two things pretty quickly after my initial export:

Most Markdown file names were derived from the Evernote note title, which means they were typically in Title Case with underscores for space separators.
MANY Markdown files, for one reason or another, had leading or trailing underscores.
Some Markdown files -- mostly ones clipped from Reddit threads -- had strange naming conventions, e.g., _r_<subreddit-name>_<super-long-post-title>. This is because Evernote automatically uses the webpage title tag as the title of a note imported with it's web clipper.
MANY Markdown files were named "untitled-XX.md" (where XX is some number). Did these notes not have titles?
MANY Markdown files had the word undefined randomly peppered through the file name, e.g., A_Historyundefined_and_Timeline_of_the_World.md. (I later realized this only occurred immediately preceding the word and, of which I speculate that the ampersand was actually used in the original title and evernote2md has a bug that does not translate it correctly.)

So I had my work cut out for me. The first thing I decided to tackle was normalizing the file name case and space separator concerns. I prefer all lower-case file names, with hyphens for space separators. I hacked together a simple node.js script to traverse all of my exported notebooks and make this change.

// npm install globby
const globby = require("globby");
const path = require("path");
const renameSync = require("fs").renameSync;
const execSync = require("child_process").execSync;

const dirs = [
    // top-level notebook names omitted for REASONS
].map(dir => path.join(__dirname, dir, "**/*.md"));

(async () => {
    const filePaths = await globby(dirs);
    const fixedPaths = filePaths.map(filePath => {
        const pathDir = path.dirname(filePath);
        const pathName = path.basename(filePath, ".md");
        let newPathName = (pathName.replace(/[^\w\d-]/g, "") + ".md").toLowerCase();
        newPathName = newPathName.replace(/_/g, "-");
        const newPath = path.join(pathDir, newPathName);
        return {
            oldPath: filePath,
            newPath,
        };
    });
    fixedPaths.forEach(fixedPath => {
        console.info(`fixing path ${fixedPath.oldPath}...`);
        try {
            renameSync(fixedPath.oldPath, fixedPath.newPath);
        } catch (e) {
            console.error(e);
            process.exit(1);
        }
    });
    console.info("all done.");
    process.exit(0);
})();

This script worked well and addressed the first naming problem -- all files names are lowercase, and instead of underscores, hyphens delimit words -- but there were still problems to address.

My initial gut instinct was to begin modifying my script to handle the remaining naming problems, but that just made me tired, so I turned to the INTERNET to figure out if there was a better way to do this.

Sweet Baby Jesus there is.

There is a wonderful utility called perl-rename that uses Perl's regular expression engine to bulk rename files in-place. It's very similar to how Vim performs find/replace, and it helped me solve two of my other problems in quick order.

** Getting rid of undefined **

To get rid of the pesky word undefined in my note file names, I used the find command to traverse my entire notebook structure, find all the Markdown files that contained that word, then pass along those file paths to the perl-rename utility which renamed the file without its troublesome intruder.

cd $NOTEBOOK
find . -iname "*undefined*.md" -exec perl-rename --verbose --dry-run -- 's/undefined//g' '{}' \;

The actual heavy lifting is done in the substitution string: s/undefined//g, which reads like this: <substitute>/<the word undefined>/<with nothing>/<anywhere in the file name (globally)>.

(Note that the --dry-run flag will show you what would happen if the perl-rename command succeeded; to actually make the changes permanent the flag must be removed from the command.)

So far so good -- no more undefined in file names. What about leading and trailing spaces? Easy peasy.

cd $NOTEBOOK
find . -iname "*.md" -exec perl-rename --verbose --dry-run -- 's/^-//' '{}' \;
find . -iname "*.md" -exec perl-rename --verbose --dry-run -- 's/-$//' '{}' \;

Again, the magic is in the substitution.

In the first command, the substitution reads: <substitute>/<a dash at the beginning of the file name>/<with nothing>. (The caret ^ symbol represents the beginning of a series of characters.)
In the second command, the substitution reads: <substitute>/<a dash at the end of the file name>/<with nothing>. (The dollar sign $ symbol represents the end of a series of characters.)

Now for those pesky Reddit notes. Since I'd eliminated leading dashes in file names, clipped notes from Reddit would now have a file name like r-<subreddit>-<note-title>. I still wanted to know these notes were from Reddit, so I decided the following substitution was best.

cd $NOTEBOOK
find . -iname "r-*.md" -exec perl-rename --verbose --dry-run -- 's/^r-/reddit-/' '{}' \;

The substitution reads (as you probably know by now): <substitute>/<an r- at the beginning of the file name>/<with reddit- >.

But Nick, what about all those untitled-XX.md notes?

I'm glad you asked. There's nothing to do with those notes but manually examine them and rename them according to their content. Which would absolutely be a pain in the ass if not for the terminal file manager ranger.

Step 3: Renaming untitled notes

Ever since I watched Luke Smith demonstrate the ranger file manager I've had major boner for it, and wanted a real chance to kick its tires. The challenge of renaming all these untitled files gave me the opportunity.

Briefly, ranger is a terminal file manager that emulates some of Vim's modal editor behavior. For example, to move through directories you use the home-row keys h, j, k, and l. To run commands you press the colon key, then enter the command name. It's both sexy and dangerous, and since I'm a kinky guy it was love at first sight.

Since I had never used ranger for any serious file system work before, this was a great way to get used to its navigation controls and command capabilities. I quickly figured out that the home row was my navigation center, but ALSO that I wanted to move through pages of files at a time rather than just hitting j and k repeatedly. Turns out if you hold shift and hit those same keys ranger will move you half-page at a time. Excellent. I traversed each notebook and used ranger's find command -- hitting / followed by a file name string -- to quickly jump to the first instance of a file named untitled.... Ranger has a great file preview pane that immediately let me inspect the contents of each file, from which I could easily determine what the real file name should be. Renaming each file was easy enough -- I typed the command :rename <new-file-name> and that did the trick. If I perchance needed to edit the file, I simply hit the l key to enter the file itself, which opened my default text editor (set by the EDITOR and VISUAL environment variables) for immediate access. Quitting the editor returned me immediately to ranger. Hitting the n key repeated my search. And so it went, until I had renamed all untitled-XX.md files in each notebook directory.

Occasionally I realized that a note I was viewing in ranger really needed to be in another directory (notebook). So I initiated an external shell command by typing ! (alternatively I could have typed :shell) and then typed my typical shell command: mv <file-name> <other-directory>/.

All without leaving ranger.

Step 4: Prune unused assets

By far the bulk of the disk space in each notebook is allotted to assets attached to notes -- be they images, or PDFs, or audio files. Markdown files, being plain text, require little space to store -- but assets, being binary, are pigs.

When I exported my notes to markdown, evernote2md created two directories in each notebook for assets: file and image. This was uniform across notebooks, which worked to my advantage. After I exported my notes I started rummaging through each notebook directory, purging notes that were either no longer important, or too badly mangled by the export process to be of any value. But how to remove their assets as well? I hacked together another node.js script to help me find assets that were no longer referenced by any notes in a given notebook.

#!node
const path = require("path");
const execSync = require("child_process").execSync;

const args = process.argv.slice(2);
const assetDir = path.resolve(args[0] || "."); // e.g., file, or image -- assume this script is executed in an asset directory
const mdDir = path.resolve(args[1] || "..");

const lsResults = execSync(`ls ${ assetDir }`).toString().split("\n").filter(n => !!n);

const noResults = [];
lsResults.forEach(a => {
  const cmd = `grep -c -H -l "${ a }" ${ mdDir }/*.md`;
  let grepResults = [];
  try {
    grepResults = execSync( cmd ).toString().split("\n").filter(n => !!n);
  } catch ( e ) {
    // empty
  }
  console.info(">>", a);
  console.info(grepResults);
  if ( grepResults.length === 0 ) { // does not appear in any file
    noResults.push( a );
  }
});

console.info( noResults );
if ( noResults.length > 0 ) {
  console.info(`to remove - rm ${ noResults.join(" ") }`);
}

This script uses the grep command to determine if an asset filename appears in the text content of any note; if it does not, it is included in output at the end that builds up a long rm command string that can be copied and then run to eliminate unused assets for a given notebook directory.

The grep command flags are important here:

-c means to generate a count of the matching lines in a file for the given search string (in this case, the asset file name)
-H means to print the file name in which the match occurred
-l means to restrict output to matching file names only (instead of matching lines within a file)

This combination of flags produces one line per search file that will only be present if the asset name is found within the file, allowing the script to know how many times the asset itself is referenced. If it isn't referenced at all, it's safe to delete. And so it goes.

This process is still a work in progress. As I review each notebook, I'm pruning its assets, and keeping track of those I've completed.

Step 5: Add front-matter to notebooks

Several Evernote alternatives (e.g., Boostnote) and many static website generators use YAML metadata markup at in Markdown files to render them appropriately. This front-matter appears at the top of the file, and follows a schema similar to the following:

---
link:
title:
description:
keywords:
author:
date:
publisher:
stats:
tags:
---

My exported Evernote notes do not have this front-matter, but as will be demonstrated later, it is critically important for targeted note searches.

So since you're wondering, yes, I did hack together another script to inject this front-matter into every existing Markdown note within a notebook directory.

#!node
const path = require("path");
const execSync = require("child_process").execSync;
const { writeFileSync, readFileSync } = require("fs");

const args = process.argv.slice(2);
const mdDir = path.resolve(args[0] || "."); // assume this command is in a notebook directory


const frontMatter = `
---
link:
title: <title>
description:
keywords:
author:
date:
publisher:
stats:
tags:
---
`.trim();

const capitalize = (s) => {
  return s.charAt(0).toUpperCase() + s.slice(1);
};

const lsResults = execSync(`ls ${ mdDir }/*.md`).toString().split("\n").filter(n => !!n);
console.info(lsResults);

lsResults.forEach(m => {
  const fileContent = readFileSync( m ).toString();
  if ( fileContent.startsWith("---\n") ) {
    console.info(`${ m } has front matter, skipping...`);
    return;
  }

  const fileName = path.basename( m );
  const formattedFrontMatter = frontMatter.replace(
      '<title>',
      capitalize(fileName.replace(/-/g, " ").replace(".md", ""))
  )

  const newContent = `${ formattedFrontMatter }\n${ fileContent }`;

  writeFileSync( m, newContent )
});

Since most notes already had filenames derived from their Evernote titles, I took advantage of that fact and turned those filenames into the note's front-matter title -- sans hyphens, and with sentence case. It's rough, I know, but better than nothing. The rest of the information I will have to add manually. The most important fields to me are title, author, and tags. (Are tags different than keywords? I don't know.) On these fields -- and the note's file name -- I will most frequently perform targeted searches.

Step 6: Searching for notes

Searching for files by name is easy. If I want to search for a file with the word taxes in it, I simply use the find command:

cd $NOTEBOOK
find . -iname "*taxes*"

This will give me a list of file paths in which the word taxes appears. I try to name my notes intelligently so this kind of search can be productive. But sometimes I want to be more specific.

In that case I can rely on the tags front-matter that I've added to each note. For example, I have a recipe for a mixed drink that my brother recommends. I've tagged this mixed drink with alcohol, and can quickly find it using the ack command (you could use grep as well, but I prefer ack):

$ ack '^tags.*alcohol.*'
alcohol/super-complex-highly-rewarding-concotion-to-drink.md
10:tags: alcohol, don-julio, grand-marnier, kombucha, cocktail

This simple command reveals that the file I'm looking for is alcohol/super-complex-highly-rewarding-concotion-to-drink.md.

In fact, I could use ack to search for any front-matter field, simply by using the correct search expression. (The observant reader will notice that the expression resembles those I used when renaming files with perl-rename. The syntax is very similar.) In this case, the search expression reads: files that contain a line that beings with 'tags' (^tags) followed by any other characters (.*) but ALSO that has the world 'alcohol' in it, followed by any other characters (.*).

If I want to cast a wider net, I can also use ack to search for any term that occurs in any of my notes with the simple command: ack <search-term>.

Step 7: Creating new notes

Now that my notes are exported, cleaned, organized, and "front-mattered", how do I add new notes to my notebooks?

Adding a new Markdown file is as simple as using your favorite text editor to save a file with the .md extension. Because the evernote2md export favored file and image directories for external assets, I use those same conventions for my own notes. If I had a notebook directory called economics, for example, and I had a note called the-history-of-economics.md, I might reference assets like this:

The Author Adam Smith wrote the seminal work, The Wealth of Nations.

<!-- this is an image of Adam Smith -->
![Picture of Adam Smith](image/adam-smith.png)

<!-- this is a link to the das-kapital.pdf file -->
Later, Karl Marx challenged Adam Smith's ideas in his work, [Das Kapital](files/das-kapital.pdf).

Now, the vast majority of my notes are articles clipped from the Internet with the Evernote Web Clipper. As I've stated, this is one of the strongest features of Evernote, and the one I'll probably miss the most.

However, I've since discovered the clean-mark npm package which will do the exact same thing a) by exporting a web page in well-formatted Markdown, and b) adding front-matter by default. This is now my go-to method of snipping articles from the Internet. The only caveat is that all assets referenced by an article will not be downloaded, but instead referenced by their individual Internet URLs. If images or external files are an integral part of an article, it will be up to me to download them manually and adjust the links accordingly.

Step 8: Accessing notes from multiple devices

So far I've entertained two methods for accessing my notes on multiple devices.

I use the MegaSync cloud storage service to back up and synchronize files across devices. It is, by far, the best cloud storage service I've used. Mega has clients that work on Windows, OSX, Linux, and Android -- which is awesome since I have devices that run each of those. (Also supports iOS but I have an Android phone so I don't care :D ). Synchronizing notes and files is flawless -- the only downside is that the Android client does not render Markdown files, or show their plain text content, which obviously makes it an non-ideal mobile client solution. This is my only real gripe about Mega's mobile offering. It's 2021: everything should render Markdown.

As an alternative, I've also contemplated using Github to manage all of my notes. I am very familiar with Git and letting it manage versions of my files and track individual commits is very appealing to me. Synchronizing across devices is trivial, as Github's web interface will render Markdown files (including embedded images) in any web browser -- mobile or not. My only hesitation is that Github (unlike Mega) does not offer end-to-end encryption (my original issue with Evernote) which does not offer me the measure of privacy I desire.

This is the last big issue I need to solve before I have a complete Evernote replacement that meets all of my needs.

Conclusion

Leaving Evernote has been an adventure, but I've learned a lot along the way -- mostly that the tools I need to achieve my values are already within my reach, and they demand nothing but the time to learn. It's amazing how much of our personal lives we heft into the "cloud", to Big Tech services that don't actually give a crap about our privacy, and will use our own data against us when we don't bring our thoughts in line with whatever pre-established narrative to which they beat their drums. If you don't control your data, you roll the dice on ever more tenuous odds.

Reclaiming my data -- and making it my own again -- has been one of the most humanizing experiences I've had in a long time. I hope this inspires others to embark on a similar quest, for freedom -- for knowledge -- for autonomy.

import facepalm;

Nicholas Cloud — Wed, 11 Mar 2020 23:35:54 +0000

This article was originally published at nicholascloud.com.

Sometimes bugs can be particularly evasive, and today I had such a one.

A module in deep in our codebase was throwing an Error, but only in Mozilla's Firefox browser.

The error was NS_ERROR_ILLEGAL_VALUE.

I did some quick DuckDuckGoing and found that the error occurs when a native DOM function in Firefox is passed a value of a type it does not expect.

The stack trace led back to this line in our application code:

const hit = find( cache, c => c.original === obj );
if ( hit ) {
    return hit.copy;
}
// ...some time later...
return someUncachedObject;

"@-E$!&&@#", I thought. "Why is lodash's find() function passing a bad value to a native function?"

You see, I use lodash all the time. So much, in fact, that I made one fatal error in my diagnosis.

I assumed that because the find() function was defined, that lodash had indeed been imported.

How. Wrong. I. Was.

It turns out that window.find() is, in fact, a non-standard, but nevertheless nearly omnipresent function that is designed to search a DOM document for a particular string. And since any function attached to window is global, a missing import of the same name -- say, a missing lodash/find import -- would not raise any alarms. The code built. The code ran. And it ran without error in every browser but Firefox. Why?

The window.find() function expects a first argument of type String. In modern browsers other than Firefox (pretty much all the Chromium-based browsers), passing a non-String argument to window.find() will simply cause the function to return false. As you can see in the snippet above, though rendering the cache useless, the application nevertheless continued to work. In Firefox, however, window.find() will throw if its first argument is not of type String. Thus my bug.

I am ashamed to say how long it took me to realize lodash/find was not the function being called.

In the end I applied the great wisdom of Spock's ancestors, and started considering things that could not possibly be the case, until it dawned on me that perhaps -- just perhaps -- find() was not what it appeared to be after all.

And a single import find from "lodash/find"; statement fixed the bug.

Fun with Homebrew casks

Nicholas Cloud — Tue, 07 Jan 2020 03:31:58 +0000

This post was originally published at nicholascloud.com.

One of my favorite utilities for OSX is Homebrew, a package manager that lets you easily install programs from the terminal.

One of my favorite pastimes is thumbing through the Homebrew Cask recipes to find new programs and utilities to install. Some are pretty nifty, like Zotero which manages research bibliographies. Or Electric Sheep which harnesses the power of your "sleeping" devices to crowdsource digital artwork. Or Finicky, which lets you specify which of your web browsers you want to open specific links. (Maybe you use Brave for normal browsing but want to open all google.com links in Chrome.)

Unfortunately the Cask recipe files have no real descriptions, so I usually just fish through them and dig out the homepage links of any Cask recipe file that has an interesting name. It's kind of like a digital treasure hunt.

To make things even more fun, I cloned the homebrew-cask repo and came up with a simple shell script that will randomly choose a recipe and open its homepage for me.

find ~/projects/homebrew-cask/Casks -type f | \
shuf -n 1 | \
xargs cat | \
ack homepage | \
awk '{print $2}' | \
xargs open

Protip - return from exceptional conditions early

Nicholas Cloud — Tue, 03 Dec 2019 02:04:24 +0000

This article was originally posted on nicholascloud.com.

During a recent code interview, I noticed a React component with a render method written in the following (abbreviated) form,

render() {
  return this.state.items.length > 0 ? (
    <ComponentWithLotsOfProps
      prop1={}
      prop2={}
      propN={}
      ...
    />
  ) : (
    ''
  );
}

where ComponentWithLotsOfProps had at least a dozen props, some of which were not simple primitive values.

While there is nothing technically wrong with this render method, it could be better. It suffers from a few deficiencies.

First, ternaries are objectively difficult to read when they are not short. It is difficult to grok what the method actually produces because the whole ternary is returned, requiring the reader to do double work to find the "implicit" returns (there are two) rather than looking for the easily identifiable return keyword.

Second, one must read the entire method to know what gets returned if there are no items in state. Is it a component? Is it null? Is it an empty string? That is unknown until the whole method has been read.

Third, if additional conditions are required in future work to determine what will be rendered, they cannot easily be introduced in this method.

A better alternative is to omit the ternary, and explicitly return the exceptional condition values first.

render() {
  if (this.state.items.length === 0) {
    return '';
  }

  return (
    <ComponentWithLotsOfProps
      prop1={}
      prop2={}
      propN={}
      ...
    />
  );
}

Due to reduced nesting, this is far easier to read, and return values are also easily identifiable. If additional conditions must be evaluated in the future, modifying this method becomes much simpler:

render() {
  if (this.state.items.length === 0) {
    return '';
  }

  if (this.state.items.length == 1) {
    return (<SingleItemComponent item={this.state.items[0]} />);
  }

  return (
    <ComponentWithLotsOfProps
      prop1={}
      prop2={}
      propN={}
      ...
    />
  );
}

As with most things in programming: the simpler, more explicit, the better.

Book Review - The Mythical Man-Month

Nicholas Cloud — Mon, 02 Dec 2019 22:43:41 +0000

This book review originally posted at nicholascloud.com.

The Mythical Man-Month is one of those books that is, well, mythical in the circles to which it pertains -- that is, the software engineering and computer science fields. It is mythical because many people have heard of it, most agree that it is "classic", but not many remember exactly why. Perhaps they have never read it, only absorbed its ideas through hearsay. Or perhaps they did read it, but so long ago that its principles have been taken only in the tide of time.

Either way, I have finally finished the book in what I consider to be an unreasonable amount of time. It's not overly long, or overly verbose, but I have a bad habit reading a little from a lot of books at the same time, which means I don't finish a book for a while. I took notes as I went so that hopefully time will be more gracious to my mind when someone asks me, in the years to come, if I've read Frederick Brooks.

Widely considered the central theme of the book, Brooks's Law, in summary, is that adding programmers to a late software project will not make it go faster, but rather slower. This was a pattern Brooks saw during his years as a manager in larger companies that needed many engineers to write software either for internal use, or eventually for sale as products. Managers assumed that the central problem of software development -- why projects did not finish on time or on budget -- was a mechanical one that could be resolved with mechanical solutions. Maybe there just wasn't enough manpower. Or maybe the tooling was inferior, and retarded progress. Or maybe the wrong language had been chosen for the task. While all of these things can, and do, affect software engineering endeavors, Brooks's major insight was that they were the accidents of software engineering, and not its essence; and that the essence is what causes projects to fail.

The essence of "systems programming" (as Brooks called it) is one of complexity -- an irreducible complexity¹ -- that of itself cannot be fixed by mechanical solutions. This complexity arises from the fact that software development is a creative, human process. The engineer must, to write a program, conceptualize the problem space correctly and then use the tools at his disposal to create a solution. As projects grow, engineers are added, the consequence of which, as Brooks keenly observed, tends to make the project slower because it increases the number of communication pathways among team members (with every addition), and the conceptual foundation of the project becomes spread among many minds, in ways that are often fragmented and incorrect. This, Brooks argues, is the core problem, and the solution to the problem is to adapt to it rather than try to conquer it.

"Complexity is the business we are in, and complexity is what limits us."

How does one adapt to the problem of conceptual complexity in software engineering? Brooks proposed a number of solutions.

Conceptual integrity and communication

Brooks proposed that the conceptual integrity of a project -- the core ideas about what the problems are and the models used to represent those problems -- are of primary importance and must be safeguarded. The most efficient way to ensure this happens is to reduce the responsibility of that integrity to one, or at most, a couple of individuals, that will be responsible for enforcing that conceptual integrity by vetting the work of other team members on the project. They become the source of conceptual truth.

Communication and team structure

Because complexity scales with the number of communication pathways in a team, Brooks proposed that "surgical teams" be used in most software projects. These teams will be composed of the conceptual guardian(s) (the "surgeon"), and as few people as possible to get the work done. These teams are part of an organization as a whole, however, and there is always a management structure with which they must integrate. The key to good management, according to Brooks, is to realize that management is about action and communication. The person at the top should rely on his subordinate program managers to take action when needed, and he should give them the authority to do so. He should never, ever demand action when reviewing a general status report, however, because this will debilitate his program managers, and move the decision making power further from the decisions that needs to be made. Project managers should be concerned almost exclusively with managing the lines of communication in the team, and not with making decisions at all. The whole process of pushing decision making "down" to the program manager is effective because it gives program managers a stake in the total authority of the company, and therefore preserves the total authority of the company.

"The purpose of organization is to reduce the amount of communication and coordination necessary..."

"...the center gains in real authority by delegating power, and the organization as a whole is happier and more prosperous."

Complexity in code

Complexity can be addressed in the code itself by reducing the mental burden a programmer has to carry while implementing code that has conceptual integrity. In the first edition of Brook's book, he insisted that all programmers on a team be familiar with all modules (or entities) within a software project. In Brooks's mind, this was a good way to safeguard the integrity of the system, because everyone would have a working understanding of all code. In a subsequent edition of the book, he backtracked on this position, because it essentially suffered from the mental equivalent of the communication problem. Code changes over time; no programmer ever has a complete and accurate understanding of system because it is not static. Brooks eventually came around to a view promoted by Canadian engineer David Parnas:

"[David] Parnas argues strongly that the goal of everyone seeing everything is totally wrong; parts should be encapsulated so that no one needs to or is allowed to see the internals of any parts other than his own, but should see only the interfaces... [I initially proposed that] Parnas's proposal is a recipe for disaster [but] I have been quite convinced otherwise by Parnas, and totally changed my mind."

Information hiding, or encapsulation, allows a programmer to "use" or take advantage of, code, without having to know how it works internally, only how to ask it for something. The mental footprint of understanding an interface (the way to ask code for something) is orders of magnitude smaller than the mental footprint required to understand the implementation behind the interface. And interfaces don't change nearly as often as implementations (in a well-designed system).

Side-effects (changes created by a piece of code that change things beyond that code, or even system), likewise, should all be identified, well understood, and encapsulated (or eliminated) to reduce the mental burden of worrying about tangential consequences to implementations, which are often causes for bugs, and project delay.

Documentation

Documentation is central to adapting to complexity. Documenting decisions made in a software project is part of fostering the creative process itself:

"...writing the decisions down is essential. Only when one writes do the gaps appear and the inconsistencies protrude. The act of writing turns out to require hundreds of mini-decisions, and it is the existence of these that distinguishes clear, exact policies from fuzzy ones."

Documentation need not be overly verbose, however; although overly verbose documentation is better than no documentation, Brooks believed. And the documentation regarding technical decisions -- design and implementation -- should be as close to the code itself as possible (even within the same code files) to ensure the documentation will be maintained and updated as the code itself changes. The goal of documentation should be twofold: 1) to create an overview of the particular system concern the documentation addresses, and 2) to identify the purpose (the why) of the decisions made in regard to that concern. Documentation is not only for other programmers to read; it is often to benefit the original author as well.

"Even for the most private of programs, prose documentation is necessary, for memory will fail the user-author."

Conclusion

There are many more things I could say about Mythical Man-Month. More subtle insights, longer expositions on points articulated above. But these points are the ones that stuck with me most, the ones I felt were immediately relevant to my own work and career. Mythical Man-Month is one part history, one part knowledge, and one part wisdom. It lets us know that great minds in past struggled with, devised solutions to, and retracted and revised solutions to, the problems that make programming a struggle of genuine creativity.

This last quote is taken from a critique of Brooks, one that he found to be excellent, and expresses the same conclusion (albeit maybe a bit more pessimistic) at which Brooks himself arrived: the central problem of software is complexity, not tools or processes, and that will never change.

"Turski, in his excellent response paper [to No Silver Bullet] at the IFIP Conference, said eloquently: 'Of all misguided scientific endeavors, none are more pathetic than the search for the philosopher's stone, a substance supposed to change base metals into gold. The supreme object of alchemy, ardently pursued by generations of researchers generously funded by secular and spiritual rulers, is an undiluted extract of wishful thinking, of the common assumption that things are as we would like them to be. It is a very human belief. It takes a lot of effort to accept the existence of insoluble problems. The wish to see a way out, against all odds, even when it is proven that it does not exist, is very, very strong. And most of us have a lot of sympathy for those courageous souls who try to achieve the impossible. And so it continues. Dissertations on squaring a circle are being written. Lotions to restore lost hair are concocted and sell well. Methods to improve software productivity are hatched and sell very well. All too often we are inclined to follow our own optimism (or exploit the hopes of our sponsors). All too often we are willing to disregard the voice of reason and heed the siren calls of panacea pushers.'"

Not to be confused with the same term used by some creationists to defend their particular ideas about the origin of the universe.

Destructuring Reconsidered

Nicholas Cloud — Fri, 03 May 2019 13:42:01 +0000

(This post originally appeared on nicholascloud.com.)

While working with React for the last five months, I've noticed that React developers make extensive use of object destructuring, especially in function signatures. The more I use React the less I like this trend, and here are a few, short reasons why.

There are countless books by wise industry sages¹ that discuss how to write good functions. Functions should do one thing, and one thing only; they should be named concisely; their parameters should be closely related; etc. My observation is that destructured function parameters tend to quickly lead to violations of these best practices.

First, destructuring function parameters encourages "grab bag" functions where the destructured parameters are unrelated to each other. From a practical point of view, it is the destructured properties of the actual parameters that are considered, mentally, as parameters to a function. At least, the signature of a destructured function reads as if they are:

function foo({ bar, baz }, buzz) {}

A developer will read this as if bar, baz, and buzz are the actual parameters of the function (you could re-write the function this way, so they might as well be), but this is incorrect; the real parameters are buzz and some other object, which, according to best practice should be related to buzz. But because the first parameter (param1) is destructured, we now have properties bar and baz which are one step removed from buzz, and therefore the relationship between param1 and buzz is obscured.

This can go one of three ways:

if param1 and buzz are related, we do not know why;
if param1 and buzz are not related (but bar and baz are related to buzz) then the function is poorly written;
if bar, baz, param1, and buzz are all closely related, then the function is still poorly written, as it now has three "virtual parameters" instead of just two actual parameters.

Second, destructured functions encourage an excessive number of "virtual parameters". For some reason developers think this function signature is well written:

function sendMail({ firstName, lastName, email}, { address1, city, state, zip}, { sendSnailMail }) {}
// function sendMail(user, address, mailPreferences) {}

"But it only has three parameters!", they say. While technically true, the point of short function signatures is to scope the function to a single, tangible task and to reduce cognitive overhead. For all practical purposes this function has eight parameters. And while the purpose of this function is fairly obvious based on its name, less expressive functions are far more difficult to grok.

Third, destructuring makes refactoring difficult. Sure, our tools will catch up some day. But from what I've seen modern editors and IDEs cannot intelligently refactor a function signature with destructured parameters, especially in a dynamic/weak typed language like JavaScript. The IDE or editor would need to infer the parameters passed into the function by examining invocations elsewhere in code, and then infer the assignments to those parameters to determine which constructor function or object literal produced them, then rewrite the properties within those objects... and you can see how this is a near impossible feat. Or at the very least, how even the best IDEs and editors would introduce so many bugs in the process that the feature would be avoided anyway.

Fourth. Often developers must trace the invocation of a function to its definition. In my experience, code bases typically have many functions with the same name used in different contexts. Modern tools are smart, and examine function signatures to try and link definitions to invocations, but destructuring makes this process far more difficult. Given the following function definition, the invocations would all be valid (since JS functions are variadic), but if a code base had more than one function named foo, determining which invocation is linked to which definition is something of a special nightmare.

// in the main module
function foo({ bar, baz}, { bin }, { buzz }) {}

// in the bakery module
function foo(bar, { baz }) {}

// invocations
foo({ bar, baz });

foo(anObject, anotherObject);

foo(1, { bin }, null);

In contrast, functions with explicitly named parameters (usually the signature parameters are named the same as the variables and properties used to invoke the function) make these functions an order of magnitude easier to trace.

Fifth, destructured parameters obscure the interfaces of the objects to which they belong, leaving the developer clueless as to the related properties and methods on the actual parameter that might have use within the function. For example:

function handle({ code }) {}

What else, besides code may exist in the first parameter that will allow me to more adequately "handle" whatever it is that I'm handling? The implicit assumption here is that code will be all I ever need to do my job, but any developer will smirk knowingly at the naivety of that assumption. To get the information I need about this parameter I have to scour the documentation (hahahahaha documentation) in hopes that it reveals the actual parameter being passed (and doesn't just document the destructured property), or manually log the parameter to figure out what other members it possesses. Which brings me to my last point:

Logging. I cannot count the number of times I have had to de-destructure a function parameter in order to log the complete object being passed to the function, because I needed to know some contextual information about that object. The same applies for debugging with breakpoints. (I love when Webpack has to rebuild my client code because I just wanted to see what actual parameter was passed to a function. Good times.)

Don't get me wrong -- I'm not completely against destructuring. I actually like it quite a bit when used in a way that does not obscure code, hinder development, or hamstring debugging. Personally I avoid destructuring function parameters in the signature, and instead destructure them on the first line of the function, if I want to alias properties with shorter variable names within the function.

function sendEmail(user, address, mailPreferences) {
  const { firstName, lastName, email } = user;
  const { address1, city, state, zip } = address;
  const { sendSnailMail } = preferences;
  //...
}

This pattern both conforms to best practices for defining functions, and also gives me a lightweight way to extract the bits of information I need from broader parameters, without making it painful to get additional information from those parameters if I need it.

Don't use the new shiny just because it's what all the cool kids do. Remember the wisdom that came before, because it came at a cost that we don't want to pay again.

Clean Code, Code Complete, etc.

Creating Reusable Code

Nicholas Cloud — Tue, 08 Jan 2019 21:49:18 +0000

Create reusable software is challenging, especially when that software may be reused in situations or scenarios for which it may not necessarily have been designed. We've all had that meeting where a boss or manager asked the question: "What you've designed is great, but can we also use it here?"

In the last month I've had this exact experience, from which I've learned a number of valuable lessons about crafting reusable software.

eTexts and annotations

When I first started working for eNotes, my initial task was to fix some code related to electronic texts that we displayed on our site (e.g., Shakespeare, Poe, Twain, etc.). We have a significant collection of annotations for many texts, and those annotations were displayed to users when highlights in the text were clicked. A couple of years ago we spun this technology off into a separate product, Owl Eyes, with additional teacher tools and classroom management features. Because of my experience with the existing eText and annotation code, and because I am primarily responsible for front-end JavaScript, I was tasked with building a "Kindle-like" experience in the browser for these eTexts. (This is one of the highlights of my career. The work was hard, and the edge cases were many, but it works very well across devices, and has some pretty cool features.)

Filtering, serializing, and fetching annotation data

The teacher and classroom features introduced some additional challenges that were not present when the eText content was first hosted on enotes.com. First, classrooms had to be isolated from one another, meaning that if a teacher or student left an annotation in an eText for their classroom, it would not be visible to anyone outside the classroom. Also, a teacher needed the ability to duplicate annotations across classrooms if they taught multiple courses with the same eText. Eventually we introduced paid subscriptions for premium features, which made annotation visibility rules even more complicated. All Owl Eyes Official annotations are available for free, public viewing, but certain premium educator annotations are restricted to paid subscribers. (Also, students in a classroom taught by a teacher with a paid subscription are considered subscribers, but only within that classroom's texts!) It was complicated.

We devised a strategy whereby a chain of composable rules could be applied to any set of annotations, to filter them by our business requirements. These rules implemented a simple, identical interface, and each could be passed as an argument to another to form aggregates. The filtered annotation data was then serialized as JSON and emitted onto the page server-side. When the reader renders in the client this data is deserialized and the client-side application script takes over.

The role that a given user possesses in the system often determines if they can see additional meta-data related to annotations, or whether they can perform certain actions on those annotations. These were communicated to the front-end to enable/disable features as needed, and then enforced on the back-end should a clever user attempt to subvert the limitations of his own role. To keep the data footprint as light as possible on the page, we developed a composable serialization scheme that could be applied to any entity in our application. The generic serialization classes break down an entity's data into a JSON structure, while more specialized serialization classes add or remove data based on a user's role and permissions. In this way a given annotation might contain meta-data of interest to teachers, but would exclude that meta-data for students. Additional information is added if the user is an administrator, to give them better control over the data on the front-end.

The end result is that, from a user's perspective, the annotations visible to them, and the data within those annotations, are tailor-made to the user when the eText reader is opened.

Fast-forward to the present day. I have recently been tasked with bringing our eTexts and annotations full circle, back to enotes.com. We brainstormed about the best way to make this happen, as enotes.com lacks the full eText and annotation data, as well as the rich front-end reading experience.

We decided that since the eText and annotation data was already being serialized as JSON for client-side consumption in owleyes.org, it would be trivial to make that same data available via an API. I implemented a simple controller that made use of Symfony's authentication mechanisms for authenticating signed requests via API key pair, and returned annotation JSON data in the exact same manner that would be used for rendering that data in the eText reader. On inspection, I realized that some of the annotation data wasn't relevant to what we wanted to display on enotes.com, so I quickly created new serialization classes that made use of existing serialization classes, but plucked unwanted data from their generated JSON structures before returning it. No changes were necessary to the annotation filtering rules, as an API user is, from the ruleset's perspective, a "public user", and so would see the same annotation data that users who aren't logged in on the site would see.

Fetching this data on enotes.com was a simple matter of using PHP's CURL classes to request data from the owleyes.org endpoint.

The user interface

The eText reader JavaScript code on owleyes.org is complex; it is composed of many different modules -- view modules, state modules, utility modules, messaging modules, etc. -- that interact together to form a smooth, reading experience. It is far more interactive than the pages we wanted to display on enotes.com, so I initially worried that the code would not be entirely reusable because of its complexity.

I was pleasantly wrong.

When I write software I take great pains to decouple code, favor composition over inheritance, and observe clear, strict, and course API boundaries in my modules and classes. I, as every programmer does, have a particular "style" of programming -- the way I think about and model problems -- which, in this case, served me very well.

I copied modules from the owleyes.org codebase into the enotes.com codebase that I knew would be necessary for the new eText pages to function. With some minor adjustments (mostly related to DOM element identifiers and classes) the code worked almost flawlessly. Where I needed to introduce new code (we're using a popup to display annotations in enotes.com, whereas in owleyes.org we use a footer "flyout" that cycles through annotations in a carousel) the APIs in existing code were so well defined that I was able to adapt to them with few issues. Where differing page behavior was desired (e.g., the annotation popup shifts below the annotation when it gets too close to the top of the screen as the reader scrolls, and above otherwise) the decoupled utility modules that track window and page state already provided me with the events and information I needed to painlessly implement those behaviors. And because the schema of the serialized annotation data delivered over the API was identical to the JSON data embedded in the owleyes.org reader, the modules that filtered, sorted, and otherwise manipulated that data did not change at all.

Why it worked

Needless to say, this project left me very satisfied as a developer. When your code is painlessly reused in other contexts it means you've something right. I've made some observation about what made this reuse possible.

First, reusable code should model a problem, or a system, in such a way that the constituent components of that model can act together, or be used in isolation, without affecting the other parts of the model. Modules, classes, and functions are the tangible building blocks we use to express these models in software, and they should correspond with the way we think about these models in our heads. Each should be named appropriately, corresponding to some concept in the model, and the connections between them should be well understood and obvious. For example, in the eText reader, a tooltip is a highlighted portion of text that may be clicked on to display an annotation popup, which displays annotation information. The tooltip and annotation popup are components in the visual model; they are named appropriately, and the relationship between them is one-way, from tooltip to popup.

Second, a given problem may in fact be composed of multiple models that are being run at the same time. Modules that control the UI are part of the visual or display model; modules that control the access to, and filtering of, data are part of the domain model. Modules track mouse movements, or enable/disable features based on user interaction, are part of the interaction model. Within these models, objects or modules should only perform work that makes sense within the purpose of the model. Objects in the visual model should not apply business rules to data, for example. When one or more objects exhibit behaviors from multiple models, extracting and encapsulating the behavior that is not part of each object's primary model makes that object more reusable.

Third, objects within a model should have well-defined, coarse APIs. (In the context of objects, an API is an object's "public" methods to outside callers, or to the objects that extend it.) A coarse API is one that provides the least amount of functionality that its responsibilities require. Yes, the least. An object either stands alone, or makes use of other objects to do its work. If the methods on an object are numerous the object can likely be broken down into several smaller objects to which it will delegate and on which it will depend to do its work internally. Ask: what abstraction does this object represent, and which methods fulfill that abstraction. Likewise the parameters to an object's methods can often be reduced by passing known state to the object's constructor (or factory function, or whatever means are used to create the object). This chains the behavior of the object to a predetermined state -- all remaining method arguments are only augmentations to this state. If the state needs to change, another object of the same type, with different state, is created and used in its stead. The API is coarse because the methods are few, and their parameters are sparse.

Fourth an object's state should be stable at all times. Its initial state should be set, completely, through the object's source of construction (whether by data provided via parameters, or sensible defaults, or both). Properties on objects should be considered read-only, as they represent a "window" into the object's state. Computed properties should be calculated whenever an object's relevant internal state changes, usually the result of a method invocation. I avoid exposing objects that can be manipulated by reference through properties; properties are always primatves that can be re-produced or re-calculated, or collections of other "data" objects that have the same characteristics (usually cloned or reduced from some other source). If an object needs to expose information from one of its internal children, I copy that information from the internal source to a primitive property on the external object itself. If the information is itself in the form of an object with multiple properties, I flatten those into individual properties on the external object. The end result is that an object's state is always generated internally, as a consequence of method invocations, and cannot be manipulated externally, except by way of its public API (methods).

Finally, shared data should exist in "bags" -- objects that jealously guard data and only deliver data by value to callers when asked. For example, on owleyes.org a given chapter in Hamlet may contain hundreds of annotations. Annotations may be crated, edited, deleted, and receive replies in client code. The annotation bag is responsible for holding the annotation data and delivering it, in read-only format, to other modules as requested so that they can render themselves (or perform computations) accordingly. When an annotation changes -- when an owleyes.org PUT request is sent to the API and a successful response is received -- a method on the bag is invoked to update the annotation. Because annotations are only fetched by value, it does no good for the module that initiated the update to directly manipulate the properties on its own annotation object. No other module will receive the change. Instead, the responsible module tells the bag to update the annotation by passing it the new annotation deserialized from the API response. The bag replaces the annotation in its internal collection and then raises an event to notify listening modules that the given annotation has changed. Any module interested in that annotation -- or all annotations -- then requests the updated data (in read-only format) and re-renders itself (or re-computes its internal state). The bag, then, is the shared resource among modules (not the data, directly) and it is the source of Truth for all data requests.

Epilogue

There is more I could say on the patterns and principles that arose during the execution of this project, but those enumerated above were of the most import and consequence while porting existing code into its new context. Reusable code is not easy to write. It is not automatic. It is the result of thought and discipline that slowly become habit as exercised.

Not all code will be reused; most won't, in fact. But writing code with a view of extension and reuse in mind can pay off in time and effort in the long run. This is a trade-off, though. The more reusable code tends to be, the more layers of redirection it will possess, necessitating an increase in the number of modules, classes, functions, etc. that need be created. This is a trade-off that can be mitigated by keeping code as simple as possible. Code can be navigated with relative ease if one can reason about it, divining what modules (etc.) do and how they are related through inference.

While I can't guarantee your experience will be as pleasant as mine, I do believe that if you think about and put these patterns and principles into action you will one day experience the joy of truthfully telling your manager, "oh, that will only take two weeks!" because your diligence produced well-crafted, reusable code.

This article originally appeared at nicholascloud.com.

Using the find command

Nicholas Cloud — Fri, 09 Nov 2018 21:30:40 +0000

The find command is used to recursively locate files in a directory hierarchy. Since programmers and system administrators spend a great deal of time working with files, familiarity with this command can make each more efficient at the terminal.

The command is composed of four parts:

the command name
options that control how the command searches (optional)
the path(s) to search (required)
expressions (composed of "primaries) and "operators" that filter files by name (optional)

A primary is a switch, such as -name or -regex that may or may not have additional arguments. An operator is a primary such as -or and -not that combines expressions in logical ways.

Basic usage

Finding files by name is perhaps the most common use of the find command. Its output consists of all paths in which the file name appears within the directory structure it searches.

$ find . -name README.md
./node_modules/hexo-renderer-stylus/README.md
./node_modules/is-extendable/README.md
./node_modules/striptags/README.md
...
./README.md

NOTE: All terminal examples were generated within the directory structure of my blog, created by the Hexo static site generator.

The path given to find in the first argument is the path prepended to each result. The . path instructs find to search under the working directory and generate relative paths in the output. If an absolute path to the working directory is used, however, the full path will appear in results. Command substitution may be used to strike a compromise between brevity and more detailed output.

$ find `pwd` -name README.md
/Users/nicholascloud/projects/nicholascloud.com/node_modules/hexo-renderer-stylus/README.md
/Users/nicholascloud/projects/nicholascloud.com/node_modules/is-extendable/README.md
/Users/nicholascloud/projects/nicholascloud.com/node_modules/striptags/README.md
...
/Users/nicholascloud/projects/nicholascloud.com/README.md

If a filtering expression is omitted, find will return all file paths within its search purview.

$ find .
.
./scaffolds
./scaffolds/draft.md
./scaffolds/post.md
./scaffolds/page.md
...

Other scenarios

The find command can do much more than locate files by exact name, however. It can find files according to a wide swath of criteria in many different scenarios.

We may not know the exact name of the file for which we are searching

The -name primary supports wildcard searches.

an asterisk (*) can replace any consecutive number of characters
a question mark (?) can replace any single character

$ find . -name '*javascript*.md'
./source/_posts/javascript-frameworks-for-modern-web-dev.md
./source/_posts/l33t-literals-in-javascript.md
./source/_posts/historical-javascript-objects.md
./source/_posts/maintainable-javascript-book-review.md

By default, the -name primary is case-sensitive. To conduct a case-*in*sensitive search, we can use the -iname primary.

$ find . -iname '*JAVA*.md'
./source/_posts/java-4-ever.md
./source/_posts/javascript-frameworks-for-modern-web-dev.md
./source/_posts/l33t-literals-in-javascript.md
./source/_posts/historical-javascript-objects.md
./source/_posts/maintainable-javascript-book-review.md

For more complex searches, we can use the power of regular expressions with the -regex and -iregex (case-insensitive) primaries.

NOTE: Use the -E option to specify that extended regular expressions should be used instead of basic regular expressions.

 $ find -E . -regex '.*package(-lock)?\.json'
./node_modules/hexo-renderer-stylus/package.json
./node_modules/is-extendable/package.json
./node_modules/striptags/package.json
./node_modules/babylon/package.json
...
./package-lock.json
./package.json

We may want to limit our results to a specific path, or a pattern that matches multiple paths

To filter results by a path mask, we can specify a pattern with -path and -ipath (case-insensitive). Both support the asterisk and question mark wildcards.

$ find . -path './node_modules/*/lib/*' -regex '.*hexo.*'
./node_modules/hexo-renderer-stylus/lib/renderer.js
./node_modules/hexo-renderer-marked/lib/renderer.js
./node_modules/hexo-generator-archive/lib/generator.js
./node_modules/hexo-migrator-wordpress/node_modules/async/lib/async.js
./node_modules/hexo-log/lib/log.js
./node_modules/hexo-generator-category/lib/generator.js
./node_modules/hexo-i18n/lib/i18n.js
./node_modules/hexo-pagination/lib/pagination.js
./node_modules/hexo-generator-index/lib/generator.js
./node_modules/hexo-util/lib/pattern.js

Using the -path primary does not change the top-level directory in which find begins its search; it merely filters results by sub-directory.

We may want detailed information about the files we find

To see detailed information about a file, in a manner similar to ls -l, the -ls primary may be appended to the list of primaries.

$ find . -name *.md -path '*/_posts/*' -ls
8636130637  8 -rw-r--r--  1 nicholascloud  staff  490 Oct 24 12:25 ./source/_posts/strange-loop-2010-video-release-schedule-posted.md
8637286945  8 -rw-r--r--  1 nicholascloud  staff  1570 Oct 24 12:45 ./source/_posts/god-mode-in-windows-7-not-as-cool-as-rise-of-the-triad.md
8636130716  8 -rw-r--r--  1 nicholascloud  staff  204 Oct 24 12:25 ./source/_posts/what-writing-fiction-taught-me-about-writing-software.md
...

(As an alternative to the -ls primary, the -exec primary may be used to invoke ls, or the xargs command may be used for the same purpose.)

We may want to stop descending into a hierarchy once we've found the file(s) for which we've searched

The -prune primary causes find to stop traversing a particular directory path once it has found a result that matches its expression. It will, however, continue to search at the same directory level as a found result for other potential matches.

 $ find . -regex '.*middleware.*' -prune
./source/_posts/new-appendto-blog-post-streams-and-middleware-in-strata-js.md
./node_modules/stylus/lib/middleware.js
./node_modules/hexo-server/lib/middlewares
./public/2013/06/new-appendto-blog-post-streams-and-middleware-in-strata-js
./.deploy_git/2013/06/new-appendto-blog-post-streams-and-middleware-in-strata-js

By using the diff tool and IO redirection we can compare the output of a "pruned" result set with the output of unpruned results to see what paths were omitted. For example, in the diff below, the remaining paths that matched /node_modules/hexo-server/lib/middlewares/* were omitted once /node_modules/hexo-server/lib/middlewares had been added to the result set.

$ diff <(find . -regex '.*middleware.*') <(find . -regex '.*middleware.*' -prune)
4,9d3
< ./node_modules/hexo-server/lib/middlewares/route.js
< ./node_modules/hexo-server/lib/middlewares/redirect.js
< ./node_modules/hexo-server/lib/middlewares/logger.js
< ./node_modules/hexo-server/lib/middlewares/gzip.js
< ./node_modules/hexo-server/lib/middlewares/header.js
< ./node_modules/hexo-server/lib/middlewares/static.js
11d4
< ./public/2013/06/new-appendto-blog-post-streams-and-middleware-in-strata-js/index.html
13d5
< ./.deploy_git/2013/06/new-appendto-blog-post-streams-and-middleware-in-strata-js/index.html

We may only want to search to a particular depth OR search beyond a particular depth

Several primaries control depth traversal, or how far find will go to locate results.

-maxdepth controls the path depth to which find will traverse before stopping.

$ find . -name *.css -maxdepth 3
./public/fancybox/jquery.fancybox.css
./public/css/style.css
./.deploy_git/fancybox/jquery.fancybox.css
./.deploy_git/css/style.css

-mindepth controls the path depth at which find will start to search.

$ find . -name *.css -mindepth 6
./node_modules/hexo/node_modules/hexo-cli/assets/themes/landscape/source/fancybox/jquery.fancybox.css
./node_modules/hexo/node_modules/hexo-cli/assets/themes/landscape/source/fancybox/helpers/jquery.fancybox-thumbs.css
./node_modules/hexo/node_modules/hexo-cli/assets/themes/landscape/source/fancybox/helpers/jquery.fancybox-buttons.css
./themes/landscape/source/fancybox/helpers/jquery.fancybox-thumbs.css
./themes/landscape/source/fancybox/helpers/jquery.fancybox-buttons.css

-depth specifies the exact depth at which find will search.

$ find . -name *.css -depth 5
./node_modules/async-limiter/coverage/lcov-report/prettify.css
./node_modules/async-limiter/coverage/lcov-report/base.css
./themes/landscape/source/fancybox/jquery.fancybox.css

We may want to find files that are newer/older relative to another file

The -newer primary will find files that are newer than the specified file by comparing the modification times of each.

$ ls -l source/_drafts/
-rw-r--r--  1 nicholascloud  staff  189 Nov    7 11:51:20 2018 the-importance-of-names.md
-rw-r--r--  1 nicholascloud  staff  353 Nov    7 11:50:49 2018 the-most-satisfying-thing.md
-rw-r--r--  1 nicholascloud  staff  10812 Nov  8 19:13:09 2018 using-the-find-command.md

$ find . -newer source/_drafts/the-importance-of-names.md -path '*_drafts*'
./source/_drafts/using-the-find-command.md

For more fine-grained control, use the -newer[XY] primary, where values of X and Y represent different kinds of file timestamps (see table below). The timestamp for X applies to the files that find evaluates; that of Y applies to the file path argument supplied for comparison

`X`/`Y` flags	value
`a`	access time
`B`	inode creation time
`c`	change time (file attributes)
`m`	modification time (file contents)
`t` (`y` only)	`file` is interpreted as a date understood by `cvs(1)`

For example, the command find . -neweram foo.txt will find all files that have a newer access time than the modification time of foo.txt.

For each X flag there are shortcut primaries that make a comparison against the modification time of the file argument.

-anewer compares the access time of each file in the result set to the modification time of the specified file.
-bnewer compares the inode creation time of each file in the result set to the modification time of the specified file.
-cnewer compares the change time of each file in the result set to the modification time of the specified file.
-mnewer compares the modification time of each file in the result set to the modification time of the specified file, and is identical to -newer.

We may want to search for a particular kind of "file" (file, directory, symlink, etc.)

In Unix-like systems, "everything is a file", and these files have types. The find command can detect file type, and filter results accordingly. Regular files (for which we search most often) have a type of f; directories have a type of d. Block files -- disks, for example -- have a type of b.

In OSX it is easy to find all block files that represent disks (physical and logical).

$ find /dev -name 'disk*' -type b
/dev/disk0
/dev/disk0s1
/dev/disk0s2
/dev/disk1
/dev/disk1s2
/dev/disk1s3
/dev/disk1s1
/dev/disk1s4

The table below lists each file type that the find command may detect.

Flag	Meaning
`b`	block special
`c`	character special
`d`	directory
`f`	regular file
`l`	symbolic link
`p`	FIFO
`s`	socket

We may want to search for files that a particular user or group owns (or inversely, that are not owned by a known user or group)

Users and groups are identified by name and numeric ID on Unix-like systems. In OSX the id command tells me my user ID and group ID(s).

$ id
uid=501(nicholascloud) gid=20(staff)...

The find command accepts primaries that filter file results by user and/or group.

-uid <uid> and -user <username> filter results by owning user. If the argument to -user is numeric, and no group exists with that name, it is assumed to be a user ID.
-gid <id> and -group <groupname> filter results by owning group. The same caveat applies to groupname as username.

I write code for a website called OwlEyes.org, which is a PHP application served by the apache2 web server. If I search for files in my home directory owned by the www-data user (the typical apache2 user), I see some interesting results.

$ find . -user www-data
./projects/enotes/owleyesorg/app/logs/apache-error.log
./projects/enotes/owleyesorg/app/logs/apache-custom.log

Every other file in my project directory is owned by my user, but the apache2 log files are written by the web server, and are therefore owned by its user.

To find files that aren't owned by any known user and/or group, the inverse primaries may be used.

-nouser <username> shows results that do not belong to a known user.
-nogroup <groupname> shows results that do not belong to a known group.

We may want to find empty files or directories

To find empty (0 byte files or directories with no files) files append the -empty primary to the find command. This can be useful, for example, to see what log files are empty on your system.

$ sudo find /var/log -empty
./appfirewall.log
./ppp
./alf.log
./apache2
./com.apple.xpc.launchd
./cups
./CoreDuet
./uucp

We may want to run a utility on the files identified by `find`

The -exec and -ok primaries may be used to run a command on each file in find's result set. The two primaries are identical but -ok will request user confirmation for each file before executing the specified command.

The syntax for executing a command with find is:

find <expression(s)> -exec <command> \;

The command is written in standard form, as you would type it in a terminal. If the string '{}' appears anywhere in the command, it will be replaced by the file path of each result as find iterates over them. Commands must be terminated by a \;. (The escape character is necessary when executing within a shell environment.)

The command find . -newer db.json -type f -exec cp '{}' ~/tmp \;:

starts in the current directory
finds files that were modified after db.json (the database file that stores blog post information)
finds files of type "regular file"
and copies each one to ~/tmp

$ ls -lh db.json
-rw-r--r--  1 nicholascloud  staff   2.6M Nov  8 19:15 db.json

$ find . -newer db.json -type f -exec cp '{}' ~/tmp \;

$ ls -l ~/tmp
-rw-r--r--   1 nicholascloud  staff    61B Nov  9 10:46 README.md
-rw-r--r--   1 nicholascloud  staff    14K Nov  9 10:46 using-the-find-command.md

Two corresponding primaries, -execdir and -okdir do the same thing as -exec and -ok, however '{}' is replaced with as many file paths as possible from the result set, making these primaries akin to xargs. For example, to archive files in a find result set, one could use -execdir to create a tarball.

$ find . -newer db.json -type f -execdir tar cvzf ~/tmp/back.tar.gz '{}' \;
a using-the-find-command.md
a README.md

We may want to format `find`'s output

The output from find can be formatted in two ways.

By specifying the -print primary, the file path of each result in find's result set is printed to standard output, terminated by a newline. This is the way find displays results by default. However, some primaries, such as -exec, might not print each file to the terminal. The command find . -newer db.json -type f -print -exec cp '{}' ~/tmp \; will copy all files newer than db.json to ~/tmp, but the output will remain empty (the default behavior of the cp command). To force each file to be displayed, the -print primary may be added before -exec.

$ find . -newer db.json -type f -print -exec cp '{}' ~/tmp \;
./source/_drafts/using-the-find-command.md
./README.md

The -print0 primary creates a space-delimited string of all file paths, and can be useful when piping the output of find to xargs or some similar command that expects input in such a format.

We may want to combine expressions to refine our search

By default primaries are combined and applied together to form an expression, but find supports two operators that change the way expressions are applied. If two expressions are separated by the -or operator, then they will be applied in a boolean OR fashion; results will be returned that match either expression, or both.

$ find . -name '*eclipse*' -or -name '*clean*'
./source/images/2011/10/eclipse-example.png
./source/images/2011/10/eclipse-example-150x127.png
...
./source/images/2011/08/clean-coders1.png
./source/images/2011/08/clean-coders-150x117.png
...

If the -not (or !) operator precedes an expressison, it will negate it and remove matching file paths from the result set.

$ find . -name '*eclipse*'
./source/images/2011/10/eclipse-example.png
./source/images/2011/10/eclipse-example-150x127.png
./source/images/2011/10/eclipse-example-2-300x97.png
./source/images/2011/10/eclipse-example-2.png

$ find -E . -name '*eclipse*' ! -regex '.*[0-9]+x[0-9]+.*'
./source/images/2011/10/eclipse-example.png
./source/images/2011/10/eclipse-example-2.png

(Recall that the -E option in the example above forces find to use extended regular expressions when evaluating the -regex primary.)

We may want to delete found files

While possible to use -execdir rm '{}' \; to delete files in a result set, find supports a shorter primary, -delete that accomplishes the same task. By default, -delete will not show output for each file that is removed; use the -print primary in conjunction with -delete to see which files were removed from the file system.

Forem: Nicholas Cloud

Goodbye Evernote

Step 1: Get my notes out of Evernote

Step 2: Fixing file names

Step 3: Renaming untitled notes

Step 4: Prune unused assets

Step 5: Add front-matter to notebooks

Step 6: Searching for notes

Step 7: Creating new notes

Step 8: Accessing notes from multiple devices

Conclusion

import facepalm;

Fun with Homebrew casks

Protip - return from exceptional conditions early

Book Review - The Mythical Man-Month

Conceptual integrity and communication

Communication and team structure

Complexity in code

Documentation

Conclusion

Destructuring Reconsidered

Creating Reusable Code

eTexts and annotations

Filtering, serializing, and fetching annotation data

The user interface

Why it worked

Epilogue

Using the find command

Basic usage

Other scenarios

We may not know the exact name of the file for which we are searching

We may want to limit our results to a specific path, or a pattern that matches multiple paths

We may want detailed information about the files we find

We may want to stop descending into a hierarchy once we've found the file(s) for which we've searched

We may only want to search to a particular depth OR search beyond a particular depth

We may want to find files that are newer/older relative to another file

We may want to search for a particular kind of "file" (file, directory, symlink, etc.)

We may want to search for files that a particular user or group owns (or inversely, that are not owned by a known user or group)

We may want to find empty files or directories

We may want to run a utility on the files identified by find

We may want to format find's output

We may want to combine expressions to refine our search

We may want to delete found files

We may want to run a utility on the files identified by `find`

We may want to format `find`'s output