Forem: Bryan Hughes

Introducing React Strapped: a Bootstrap Data Management Library

Bryan Hughes — Mon, 06 Nov 2023 19:38:33 +0000

I've been using Next.js in production since version 10, and I'm a big fan overall. Like everything in life though, Next.js isn't perfect. One of my biggest pain points isn't actually due to Next.js itself, but rather a misalignment between Next.js and modern state management tools: managing page-specific bootstrap data.

So I create an open source library to fix it: React Strapped

Background

Single-Page Applications (SPAs) have been the predominant method of writing applications for some time in the web world, and for good reason. SPAs brought us high performance when navigating around the sight, as well as the ability for more complex state and context across routes. SPAs have their own downsides though, and is why we're starting to migrate away from them.

Next.js is a SPA-like system in the sense that Next.js provides client-side routing, instead of doing full page loads when navigating between routes. However, Next.js isn't really a SPA system because each route has its own code bundles, bootstrap data, etc. that are by default independent between pages. This allows each route to only need to load the data and code that route needs, increasing performance. Due to this hybrid nature, I've personally started calling Next.js' a "client-side MPA" framework, or csMPA.

I really appreciate Next.js' csMPA implementation because it greatly simplifies implementing and maintaining applications with multiple and varied routes compared with previous SPA architectures.

There's one area though that I've historically struggled with while using Next.js: how to access bootstrap data in a reliable and scalable manner. To be clear, accessing bootstrap data is outside of Next.js' jurisdiction and is instead the responsibility of state management libraries. Unfortunately I haven't found any state management libraries that handle bootstrap data from Next.js cleanly.

So I implemented my own library specifically to solve this use case: React Strapped.

Problem statement ^[1]

Before I dive into how React Strapped works, let's spend some time really defining the problem, because it wasn't immediately obvious to me.

React has a number of popular state management libraries, such as Redux, Zustand, Jotai, and Recoil. While each of these state management libraries brings their own opinions to state management, none of them have opinions on how to load bootstrap data from Next.js. Specifically, bootstrap data in Next.js a) varies from page to page and b) is not available to state management libraries until first render. As we'll see, this creates friction.

Bootstrap data in csMPAs typically have the following characteristics:

Some data is always available on all routes.
- Examples include: data about the current user, feature/experimentation flags, etc.
- This data is always available, and can be consumed "no questions asked."
- In Next.js, this data is typically populated via a centralized helper function that is called from getServerSideProps.
Other data is only available on certain routes.
- Examples include settings information that's only available on a settings page, but not on the home page.
- This data is not available on all pages, and so care must be take to only access this data on the settings page.
- In Next.js, this data is typically populated via one-off code in a specific route(s)' getServerSideProps implementation.
Bootstrap data is not available until the first render of the application.
- In Next.js, bootstrap data generated in getServerSideProps is passed as component properties to the top level component.
- The implication of this pattern is that any state management declared at module scope, such as atom() constructors in Jotai and Recoil, do not have access to this data at time of construction.
- This limitation means we can't use these libraries preferrerd state initialization techniques.
Data during SSR needs to be scoped to a React context and not be available globally.
- A Next.js server is rendering multiple requests from multiple users more or less at once, meaning global data is not a pragmatic option since it wouldn't be "scoped" to a specific request.
- While it's possible to use global data safely in Next.js applications, it's very tricky to completely prevent all data "leaks" at all times. As such, I think it's best to avoid this pattern as whole.
- This means we can't do tricks for Jotai/Recoil like creating a global promise we attach to each atom's initializer, and then resolving it once we get the bootstrap data.

Taking these characteristics into account when considering state management libraries, we end up with these requirements for state management:

Initialize the store/atoms/etc. synchronously during the first render
Allow global data to be used anywhere, but prevent page specific data from being used outside of that page
Only load state management code on a page if they are used on that page

Redux/Zustand handle requirement 1 well. Jotai/Recoil handle requirements 1 and 3 well, but only if we're not trying to solve requirement 2 (see below for an explanation). None of these four handle requirement 2 well.

React Strapped exists to handle all 3 requirements well. That said, React Strapped is not a replacement for the four state management libraries mentioned, and is indeed intended to be used in conjunction with them.

Wait, why doesn't Jotai/Recoil handle requirement 2? ^[2]

It may not be immediately obvious why Jotai/Recoil don't handle all requirements well. It certainly wasn't obvious to me at first, and my frustrations getting them to work specifically is what led me to create React Strapped. But why is that? It turns out that initializing Recoil with runtime bootstrap data is tricky and non-obvious

Specific challenges include:

In multi-page apps, Jotai/Recoil must be initialized with bootstrap data that's been prop-drilled to a component
- Thus, bootstrap data is not available to use with the default value mechanisms Jotai/Recoil provide, preventing us from using the standard way of initializing atoms.
- The recommended way to solve this challenge in Recoil is to initialize atoms with the initializeState property in RecoilRoot. As we'll see in challenge 2, this is not viable in multi-page apps.
- Similarly, Jotai provides the useHydrateAtoms hook, but also isn't viable for the same reasons
Bootstrap data varies from page to page
- Thus, we need to initialize different atoms depending on which route we're on.
- While it's possible, if unwieldy, to use a switch(currentRoute) statement to initialize only the atoms needed per route, we have to statically import every atom from every page to do the initialization. In other words, we're including code in our bundles that's never used.
- Dynamic imports are typically how we include dependencies conditionally, but their asynchronous nature is incompatible with the synchronous initializeState prop/useHydrateAtoms hook, which is otherwise the only way to conditionally import atoms based on the current route.
State needs to be scoped to a React context and not used globally
- A Next.js server is rendering multiple requests from multiple users more or less at once, meaning global data is not an option since it wouldn't be "scoped" to a specific request.
- This means we can't do tricks like creating a global promise we attach to each atoms default prop, and then resolving it once we get the bootstrap data.

All of these challenges together mean that Jotai/Recoil do not currently include any mechanisms for conveniently initializing themselves with bootstrapped data in csMPAs.

How React Strapped works ^[3]

React Strapped works by creating a React context to hold bootstrap data, and special hooks for accessing this data. We call an instance of provider+hooks associated with a piece of bootstrap data a "strap." Unlike other state management libraries, these providers+hooks are intentionally designed so that more than one provider can be used at a time without impacting other providers. We'll see what this means and why it's important in the next section.

Simple example

This example shows a minimal example using React Strapped. It's written in TypeScript to a) demonstrate how TypeScript types flows through the library and b) to give a sense of what data is expected where. You can absolutely use this library without using TypeScript though.

First, let's create our provider and hooks in a file called state.ts:

import { createStrappedProvider } from 'react-strapped';

export interface MyBootstrapData {
  currentUser: {
    name: string;
    age: number;
  };
}

// First, we create the strap, which includes the context
// provider and some helper functions for creating hooks.
const myStrap = createStrap<MyBootstrapData>();

// Next, export the provider, which is used to make data 
// available to components.
export const MyStrapProvider = myStrap.Provider;

// Finally create a hook for accessing the current user 
// included in the bootstrap data. The callback is called 
// once on first render to initialize the strap value 
// returned by the hook.
export const useCurrentUser = myStrap.createUseStrappedValue(
  ({currentUser}) => currentUser
);

Now let's create some UI in a Next.js page component:

import type { MyBootstrapData } from './state';
import { MyStrapProvider, useCurrentUser } from './state';

interface PageProps {
  bootstrapData: MyBootstrapData;
}

// If you're not familiar with Next.js, this function runs
// on a server and is responsible for fetching bootstrap 
// data. The value of the `props` property is passed as 
// props to the default export React component in this file.
export function getServerSideProps() {
  const props: PageProps = {
    bootstrapData: {
      currentUser: {
        name: 'Philip J Fry',
        age: 1_026,
      },
    },
  };
  return { props };
}

// This default export is the root component in a Next.js 
// page. The props passed to this component come from the 
// server via `getServerSideProps`.
export default function MyApp({ bootstrapData }: PageProps) {
  return (
    // We include our strap provider and give it the 
    // bootstrap data.  This initializes data and make it 
    // immediately available for  use via strap hooks, e.g. 
    // `useCurrentUser`.
    <MyStrapProvider bootstrapData={bootstrapData}>
      <MyComponent />
    </MyStrapProvider>
  );
}

function MyComponent() {
  // We use the hook created above, which makes sure that 
  // we're calling this hook in a component with 
  // <MyStrapProvider> as a  parent in the component tree.
  const currentUser = useCurrentUser();
  return (
    // Prints "Hello Philip J Fry"
    <div>Hello {currentUser.name}</div>
  );
}

Multi-page Apps

React Strapped is designed specifically for csMPAs, which React Strapped supports via multiple Provider components. You can have as many providers as you want with any amount of nesting. All hooks associated with providers are available for use, regardless of where that provider sits in relation to other providers.

This nesting works differently than in Recoil and Jotai. In Jotai, you can only use atom values associated with the closest Provider. In Recoil, you can either only access the root-most RecoilRoot, or the closest RecoilRoot if the override property is specified for that root. Neither of these allow you to access all values from all providers, regardless of nesting.

Why does this distinction matter? In multi-page applications, we often have a set of bootstrap data that is common to all pages as well as bootstrap data that is specific to a page. The natural setup to handle these two sets of data is to create one provider for the common bootstrap data that exists on all pages, and then per-page providers that contain just those pages' data. React Strapped supports exactly this provider setup by allowing any number of straps to be nested, and propagating data appropriately between them.

In code, we set this up like so:

export default function MyPage({
  commonBootstrapData,
  myPageBootstrapData
}) {
  return (
    // Add the common provider, which is used on all pages
    <MyCommonStrapProvider
      bootstrapData={commonBootstrapData}
    >
      {/* SomeCommonComponents only has access to hooks from 
          MyCommonStrapProvider */}
      <SomeCommonComponents />
      {/* Add the page specific provider here, which is only 
          used on this page */}
      <MyPageSpecificStrapProvider
        bootstrapData={myPageBootstrapData}
      >
        {/* SomePageSpecificComponents has access to all 
            hooks from both MyCommonStrapProvider _and_ 
            MyPageSpecificStrapProvider */}
        <SomePageSpecificComponents />
      </MyPageSpecificStrapProvider>
    </MyCommonStrapProvider>
  )
}

With this setup, the SomePageSpecificComponents component tree has access to all hooks associated with both the common and page specific straps, but the SomeCommonComponents component tree only has access to hooks associated with the common strap.

If bootstrap data exists across two or more pages, but not all pages, you can create a third strap that is shared between these pages that sits between the common strap and the page specific strap.

To ensure that SomeCommonComponents code cannot access the page specific hooks, these hooks provide guardrails against accessing data from the wrong place. If you try and call a hook based on MyPageSpecificStrapProvider from SomeCommonComponents, then you'll get an error saying you're trying to access it from the wrong place, like so:

Conclusion

And that's that. The library is very simple at only 138 lines of code with zero dependencies. I'm really excited about this approach because it:

Is technically and conceptually very simple
Removes a lot of boilerplate otherwise required to create safe access to bootstrap data
Provides strong guardrails for how we access page-specific bootstrap data in a csMPA

I'd love to hear what you think about this library!

Footnotes

[1] Most of this content comes from the Motivation section of the React Strapped README.

[2] Most of this content comes from the Motivation section of the Recoil Bootstrap README, an early version of what eventually became Recoil Strapped.

[3] Most of this content comes from the Getting Started section of the React Strapped README.

Interacting with Microsoft's LED art piece at JSConf EU

Bryan Hughes — Thu, 30 May 2019 12:41:15 +0000

Hi CSSConf/JSConf EU friend! If you're here, then you've probably heard that we brought an LED art piece with us to the conferences. This article will tell you all about this project at a high level, and how you can engage with it.

This art piece is very loosely inspired by bamboo forests. This device features multiple LED "shoots" that are controlled from the cloud, by you. The hardware was built by my amazing teammate and Microsoft Cloud Advocate PM for Europe Jan Schenk from designs I created. Here is what the finished version of the art piece looks like:

Creating animations for this piece may look daunting at first glance, but don't worry. I put a lot of work into this project over the years to make it approachable for a wide variety of folks.

This project is based on another project I created called Raver Lights. I started this project back in 2016 to control LEDs on art pieces destined for Burning Man and similar events. Suffice to say, it's pretty well tested at this point, having survived the extremes of the desert multiple times now.

There are two ways to control the animations in this piece. The first way is at our booth using a web interface running on the new Chromium-based version of Edge (ask us about it!). The second is by creating your own Azure Function-based Serverless endpoint, which gives you a lot more control over the animation than the booth interface.

Basic Animations

Don't let the name fool you, there's nothing boring about these animations! I'm just bad at naming things, to be completely honest.

At the booth, you can submit an animation from tablets we'll have on hand running the new Chromium-based version of Edge. If you haven't heard yet, we're rewriting our browser to be based on Chromium! We're still hard at work getting it ready for general use, but you can download canary and dev channel builds today for Windows 10 and macOS.

I decided to have a little fun building this app. My design skills may not be the best, but I did get to use some pretty new technologies here: CSS Grid, the new CSS filter property, and Web Assembly. If I'm at the booth, ask me all about how I wrote it (I'm the one with purple hair). You can also check out the code on GitHub.

While these animations are certainly pretty, you don't quite get the same control as custom animations.

Custom

You can also write a custom Azure Function to gain complete control of the animation. For the curious, here is the code to create the default animation you see running at the booth:

import {
  createWaveParameters,
  createMovingWave,
  createSolidColorWave,
  createPulsingWave
} from 'rvl-node-animations';

const animation = createWaveParameters(
  // Create the moving purple wave on top
  createMovingWave(215, 255, 8, 2),

  // Creating a pulsing green on top of the blue, but below the purple
  createPulsingWave(85, 255, 2),

  // Create the solid blue on bottom
  createSolidColorWave(170, 255, 255, 255)
);

This code uses the same tools you'll use to create a custom animation. Not so bad, right? Trust me, writing the animation engine that takes these parameters was considerably more difficult.

To get started writing a custom animation, head over to our starter repo and read the instructions there.

Happy Hacking!

A New Coding Style for Switch Statements in JavaScript/TypeScript

Bryan Hughes — Thu, 16 May 2019 16:08:42 +0000

I've recently adopted a new coding style for switch statements in JavaScript (well, technically TypeScript), and I'm really curious to hear what you all think about it!

Before I discuss the style, let me explain the problem I'm trying to solve. It's not uncommon for switch statements to have somewhat-repeated code between cases, but with variations in each case such that you can't easily abstract it. Let's take this example, written in ES5 (I promise there's a reason why).

switch (body.type) {
  case 'isBasic':
    var entry = getBasicEntry(body.id);
    console.log('Processing basic entry "' + entry.name + '"');
    doBasicProcessing(entry);
    break;

  case 'isCustom':
    var entry = getCustomEntry(body.id);
    console.log('Processing custom entry "' + entry.name + '"');
    doCustomprocessing(entry);
    break;

  default:
    throw new Error('Unknown flag ' + myFlag);
}

Cool, it's a basic switch statement and it works fine. Now, let's convert this to ES2015+.

switch (body.type) {
  case 'isBasic':
    const entry = getBasicEntry(body.id);
    console.log(`Processing basic entry "${entry.name}"`);
    doBasicProcessing(entry);
    break;

  case 'isCustom':
    const entry = getCustomEntry(body.id);
    console.log(`Processing custom entry "${entry.name}"`);
    doCustomprocessing(entry);
    break;

  default:
    throw new Error(`Unknown flag ${myFlag}`);
}

Uh-oh, we now get a SyntaxError exception when we run this code! The problem here is that const variable name entry cannot be declared twice in the same scope. Because of how scoping rules work, the entire switch statement is one single scope.

We could solve this by naming each entry with a different variable name, but honestly I find that kind of annoying. We could also create a helper function for each case, but that feels excessive to me for such short case statements.

So here's what I figured out, and I've started using it in all my code:

switch (body.type) {
  case 'isBasic': {
    const entry = getBasicEntry(body.id);
    console.log(`Processing basic entry "${entry.name}"`);
    doBasicProcessing(entry);
    break;
  }

  case 'isCustom': {
    const entry = getCustomEntry(body.id);
    console.log(`Processing custom entry "${entry.name}"`);
    doCustomprocessing(entry);
    break;
  }

  default: {
    throw new Error(`Unknown flag ${myFlag}`);
  }
}

I wrap each case statement in {}, which creates a new scope on a per-case basis. This solves the problem, and I think is kind of elegant myself. It's not very common though.

What do you all think? Does this seem like a good approach? Do you have a different approach that you use? I'd love to hear it in the comments!

Update:

Thank you all for the discussion so far, it's been great! I ended up creating a new module called conditional-reduce that automates another approach I hadn't thought of before:

const { reduce } = require('conditional-reduce');

console.log(reduce('dog', {
  dog: () => 'Dogs are great pets',
  cat: () => 'Cat\'s are also great'
})); // Prints "Dogs are great pets"

console.log(reduce('bird', {
  dog: () => 'Dogs are great pets',
  cat: () => 'Cat\'s are also great'
})); // Throws 'Invalid conditional value "bird"' exception

const { curry } = require('conditional-reduce');

const dogReducer = curry({
  dog: () => 'Dogs are great pets',
  cat: () => 'Cat\'s are also great'
});

console.log(dogReducer('dog')); // Prints "Dogs are great pets"
console.log(dogReducer('bird')); // Throws 'Invalid conditional value "bird"' exception

Thank you @avalander and @john_papa for the discussion that prompted this!

Building an Interactive LED Art Piece for JSConf EU

Bryan Hughes — Fri, 10 May 2019 18:02:30 +0000

JSConf EU is arguably the most important JavaScript conference of the year. Many important technologies have been announced at JSConf EU, such as Node.js, and a wide array of developers come from all over the globe to attend. Microsoft is sponsoring the conference this year, and we're bringing something fun along: an LED powered art piece that you can control from the booth and with your own custom Serverless implementation!

This series of blog posts will dive into how we built this project.

I've been involved in LED art for a while now after I was inspired by my experiences with Burning Man and related events. I've been developing a wireless lighting control system for programmable LED art for a few years now, and it's pretty rock-solid at this point. This is the perfect system to base our booth code on. I took on the lead role for the project since I created the base infrastructure for it, and I'm fortunate to work with my amazing teammates Jan Schenk, Tanya Janca, Tierney Cyren, and Suz Hinton to bring it to life. If you're curious what this system looks like, here's a tech demo:

For JSConf EU, I designed an art piece loosely inspired by bamboo forests. This device will feature multiple LED "shoots" that are controlled from the cloud.

This piece uses custom hardware I originally designed for Burning Man based on the ESP8266 12-S WiFi enabled microcontroller. These little boards control the lights themselves and connect to a Raspberry Pi over WiFi. The Raspberry Pi will connect to Azure using Azure IoT Hub, and from there we can hook into all sorts of cloud goodness!

The system is comprised of three sets of code: browser UI code, server code in the form of Azure Functions, and controller code running on the Raspberry Pi. Here is the complete architecture diagram (minus the ESP8266s ):

The Browser UI code consists of two clients. The first client will be running on tablets at the booth, and will be running the new Chromium-based version of Edge. This client will allow people to walk up to the booth and give them basic control over the animations. This interface will be intentionally limited though. For more advanced animations, we'll have another web UI that will allow people to submit custom animations, implementing using their own Azure Function, where they have full control over the animations.

The server code consists of four Azure Functions. One Azure Function will implement the endpoint for submitting basic animations, and another will implement the endpoint for submitting custom animations. Another Azure Function will be used to return the current queue of animations, so folks can see when their animation will run. The final Azure Function will run on a timer, and pulls the next animation off of the queue and sends it to the Raspberry Pi for display using Azure IoT Hub. The animation queue is implemented using Azure Storage Queues. Once nice side effect of using a queue for animations is that we don't need a database at all! This simplifies the code and improves security and privacy of user data (there's nothing to leak if it doesn't exist).

The last piece is the controller code running on the Raspberry Pi. This code is the smallest piece of code, and mostly just glues IoT Hub to the RVL-Node package I wrote to communicate with the ESP8266s.

All of the code for this project is written using TypeScript. The web UIs are written using React.js and Redux as well. The RVL-Node module is written with a combination of TypeScript for the Node.js side of things, and C++ via Web Assembly for the logic side of things.

All of these pieces together should create a fun, unique booth experience. I can't wait to show it to you all!

Passing structured data between C++ and JavaScript in Web Assembly

Bryan Hughes — Mon, 22 Apr 2019 16:53:38 +0000

I recently finished getting the messaging stack of my wireless LED control system running in Node.js via Web Assembly (WASM for short). The last major hurdle I encountered was how to pass a lot of structured data between JavaScript and C++.

The Scenario

The core data that is passed around through my messaging stack is a set of control parameters needed to control LED animations. This information is defined in C++ using the following structs:

#define NUM_WAVES 4

struct RVLWaveChannel {
  uint8_t a = 0;
  uint8_t b = 0;
  int8_t w_t = 0;
  int8_t w_x = 0;
  int8_t phi = 0;
};

struct RVLWave {
  RVLWaveChannel h;
  RVLWaveChannel s;
  RVLWaveChannel v;
  RVLWaveChannel a;
};

struct RVLWaveSettings {
  uint8_t timePeriod = 255;
  uint8_t distancePeriod = 32;
  RVLWave waves[NUM_WAVES];
};

My animation algorithm uses a bunch of coefficients to calculate a series of superimposed sin waves. The details of how this works are for another day though, just know that it looks really really pretty! What you do need to know for this blog post is that there are 82 (!!) numerical values that need to be passed from JavaScript to C++, and vice versa.

As I mentioned in the previous post in this series, you can only pass numbers between C++ and JavaScript in WASM. This structured data is numerical, but it's also structured. We must preserve this structure as well as the numerical values when passing data around.

We could implement a function that takes 82 arguments...but I mean, c'mon, really? I just know I'd mess it up! This approach would also make it really hard to update if the data changed. So we need something else. I thought about serializing to a JSON string and then deserializing it, but that takes a lot of work, processing time, and code bloat on the C++ side.

What I needed was something clever...

My Solution

And clever was what I found! I remembered that structs in C/C++ are laid out in memory in a deterministic manner. I realized that with this knowledge, I could directly marshal and unmarshal the data from the memory array in JavaScript, just like I did with strings!

To illustrate what I'm talking about, let's take a very simple struct:

struct MyStruct {
  uint8_t a = 0x12;
  uint16_t b = 0x3456;
  uint32_t c = 0x789ABCDE;
};

MyStruct str;

If we inspect the memory that str points to (e.g. the numerical value of &str in C/C++ parlance), we will see the following:

str + 0	str + 1	str + 2	str + 3	str + 4	str + 5	str + 6
0x12	0x34	0x56	0x78	0x9A	0xBC	0xDE

By using the sizeof() operator in C++, we know that this struct is 7 bytes large, which matches the layout above. We can also see that the values are stacked right next to each other in memory! All we need to know is the "memory offset" of each value relative to the base pointer, i.e. the + n part in the table.

So how do we determine this offset? C/C++ always arranges these properties in memory in the order they are declared in the struct in the source code. In this example, a comes first, followed by b, followed by c, because I declared them in that order in the code. If we switched the order of b and c so that b was at the end of the source code, then b would also be at the end of the memory block.

This means that we can calculate each offset summing up the size of every entry that came before it.

Automating the calculation of offsets

Calculating these by hand is error prone though, especially when structs reference other structs as I do. I would also have to recalculate these offsets if I ever changed the data in the structs. This is a perfect opportunity to automate the process with a build-time script!

You can see the (admittedly poorly commented) Node.js script I wrote on GitHub.

The first thing I did was write a quick-n-dirty C++ parser using regex's. This parse produces a data structure that looks like this:

const structs = {
  RVLWaveChannel: [
    { name: 'a', type: 'uint8_t', initialValue: 0 },
    { name: 'b', type: 'uint8_t', initialValue: 0 },
    { name: 'w_t', type: 'int8_t', initialValue: 0 },
    { name: 'w_x', type: 'int8_t', initialValue: 0 },
    { name: 'phi', type: 'int8_t', initialValue: 0 }
  ],
  RVLWave: [
    { name: 'h', type: 'RVLWaveChannel', initialValue: undefined },
    { name: 's', type: 'RVLWaveChannel', initialValue: undefined },
    { name: 'v', type: 'RVLWaveChannel', initialValue: undefined },
    { name: 'a', type: 'RVLWaveChannel', initialValue: undefined }
  ],
  RVLWaveSettings: [
    { name: 'timePeriod', type: 'uint8_t', initialValue: 255 },
    { name: 'distancePeriod', type: 'uint8_t', initialValue: 32 },
    { name: 'waves', type: 'array', subType: 'RVLWave', arraySize: 4 }
  ]
};

Now we have a representation of the C++ structs in JavaScript. We're not quite ready to start calculating offsets just yet though. We have references in two of our structs to other structs, and we also have an array. When this struct is instantiated in C++, these different structs and arrays aren't represented as pointers to multiple memory blocks. Rather, the structs and arrays are "flattened" such that they all sit in a single 82 byte memory block.

To represent this flattening in memory accurately, we must flatted our own representation of these structs and arrays too. I accomplished this by writing a while loop that iterates through each entry in the "root" struct (RVLWaveSettings in this case). We then replace any entry whose type value is not a primitive from stdint.h (e.g. something of the form [u?]int[8|16|32]_t) with it's "referenced" type. The way we do this replacement depends on whether it's a struct or an array. The while loop keeps running until there are no more replacements to be made.

When the loop encounters an array of items, it "unrolls" the array. In other words, it replaces:

{ name: 'waves', type: 'array', subType: 'RVLWave', arraySize: 4 }

with:

{ name: 'waves[0]', type: 'RVLWave', initialValue: undefined }
{ name: 'waves[1]', type: 'RVLWave', initialValue: undefined }
{ name: 'waves[2]', type: 'RVLWave', initialValue: undefined }
{ name: 'waves[3]', type: 'RVLWave', initialValue: undefined }

When each loop of the iteration encounters a struct type, it replaces the reference to the struct with the entire entry of the struct. In other words, it replaces:

{ name: 'waves[0]', type: 'RVLWave', initialValue: undefined }

with:

{ name: 'waves[0].h', type: 'RVLWaveChannel', initialValue: undefined }
{ name: 'waves[0].s', type: 'RVLWaveChannel', initialValue: undefined }
{ name: 'waves[0].v', type: 'RVLWaveChannel', initialValue: undefined }
{ name: 'waves[0].a', type: 'RVLWaveChannel', initialValue: undefined }

If we keep running this algorithm, we eventually end up with a set of entries that look like this:

{ name: "timePeriod", type: "uint8_t", initialValue: 255, size: 1 }
{ name: "distancePeriod", type: "uint8_t", initialValue: 32, size: 1 }
{ name: "waves[0].h.a", type: "uint8_t", initialValue: 0, size: 1 }
{ name: "waves[0].h.b", type: "uint8_t", initialValue: 0, size: 1 }
{ name: "waves[0].h.w_t", type: "int8_t", initialValue: 0, size: 1 }
{ name: "waves[0].h.w_x", type: "int8_t", initialValue: 0, size: 1 }
{ name: "waves[0].h.phi", type: "int8_t", initialValue: 0, size: 1 }
{ name: "waves[0].s.a", type: "uint8_t", initialValue: 0, size: 1 }
...

With this, we can now loop through and calculate the offsets! I iterate through each entry and keep a running sum of the sizes, which is the memory offset for each entry. I then write this information to a JSON file that looks like this:

{
  "totalSize": 82,
  "entryDictionary": {
    "timePeriod": {
      "name": "timePeriod",
      "type": "uint8_t",
      "initialValue": 255,
      "size": 1,
      "index": 0
    },
    "distancePeriod": {
      "name": "distancePeriod",
      "type": "uint8_t",
      "initialValue": 32,
      "size": 1,
      "index": 1
    },
    "waves[0].h.a": {
      "name": "waves[0].h.a",
      "type": "uint8_t",
      "initialValue": 0,
      "size": 1,
      "index": 2
    },
    ...
  }
}

Using offsets to read from a C++ struct in JavaScript

Now that we have our offsets, we can finally start passing data back and forth! Let's start by talking about how we read data from C++ into JavaScript. We start the same as we did with strings: by creating a Node.js Buffer object that represents the area of memory containing the struct we want to read. Then we iterate through each element in the offset data and read the value at the given offset:

const view = Buffer.from(memory.buffer, waveSettingsPointer, structData.totalSize);
for (const entryName in structData.entryDictionary) {
  const structEntry = structData.entryDictionary[entryName];
  let value = 0;
  switch (structEntry.type) {
    case 'uint8_t':
      value = view.readUInt8(structEntry.index);
      break;
    case 'int8_t':
      value = view.readInt8(structEntry.index);
      break;
    case 'uint16_t':
      value = view.readUInt16BE(structEntry.index);
      break;
    case 'int16_t':
      value = view.readInt16BE(structEntry.index);
      break;
    case 'uint32_t':
      value = view.readUInt32BE(structEntry.index);
      break;
    case 'int32_t':
      value = view.readInt32BE(structEntry.index);
      break;
    default:
      throw new Error(`Unexpected struct type "${structEntry.type}"`);
  }
  // Assign the value we just read to a JavaScript mirror object
  // using some dense code I'd rather not show here :-P
}

We then end up with a data structure in JavaScript defined using the following TypeScript interfaces:

export interface IWaveChannel {
  a: number; // Default 0
  b: number; // Default 0
  w_t: number; // Default 0
  w_x: number; // Default 0
  phi: number; // Default 0
}

export interface IWave {
  h: IWaveChannel;
  s: IWaveChannel;
  v: IWaveChannel;
  a: IWaveChannel;
}

export interface IWaveParameters {
  timePeriod?: number; // Default 255
  distancePeriod?: number; // Default 32
  waves: IWave[];
}

Looks familiar, right?

Writing to a C++ struct from JavaScript is effectively the reverse of the above. To see all the code that does the marshalling and unmarshalling, check out bridge.ts on GitHub.

And that's that, we can now pass structs from C++ into JavaScript and vice versa! It may sound like a whole lot of work for something that you might think would be simple, but that's kinda turning out to be par for the course with WASM. Regardless, this mechanism marks the next big step towards integrating this system with Azure IoT Edge via Node.js!

If I have time in the future, I'd love to beef up my parsing script to use a proper C++ AST parser so it can work with a wider range of code, and publish all of this as an easy-to-consume module on npm.

You can check out the complete code for the WASM powered Node.js messaging library in the RVL-Node repository on GitHub.

Passing strings from C++ to JavaScript in Web Assembly

Bryan Hughes — Thu, 21 Mar 2019 19:04:52 +0000

I'm moving right along with my experiments of getting the messaging stack of my wireless LED control system running in Node.js via Web Assembly (WASM for short). I'm now ready to start integrating the stack into a Node.js library.

The very first thing I decided to wire up was some logging functionality. This meant passing strings from C++ to JavaScript. Sounds straightfoward, right? I thought so until I spent a day and a half struggling to get it to work 😅.

The Scenario

You can only pass numbers between JavaScript and WASM. This is just how the runtime is designed. So how do you pass more complex data?

There is some emscripten documentation on interacting with code across languages that discusses how to do just that. If you're fully immersed in the emscripten world, then you can use the functions ccall and cwrap to neatly and tidily pass strings from one language to another. There's a catch though: you must be running a full C++ application to make use of these functions, not just a library.

I tried to hack the output so I could tie into these functions without making it a full application, similar to how I hacked the output to tie into emscripten's WASM bootstrap code. It didn't work this time though. emscripten is set up so that these functions are only available once int main() {} has been run in C++ land. I don't have a main function though, since this is a library. Even adding an empty main function didn't work for some reason. emscripten threw an error stating that ccall is not available until the app has been initialized and main had been run.

So back to the drawing board. I searched high and low for other sorts of emscripten tricks, but no such luck. Then it hit me! I was way over-complicating the problem.

WASM creates a chunk of memory within JavaScript for its use. This memory chunk is created by calling const memory = new WebAssembly.Memory({ initial: 256, maximum: 256 }). A WebAssembly.Memory instance is a wrapper around an ArrayBuffer, and exposes this buffer to us via the buffer property on the memory instance. This is a long winded way of saying that WASM memory is just a typed array that we can access!

The Solution

You might have read that last paragraph and thought "ok, cool, but what does that have to do with strings?" In C, strings are typically defined as a character array, e.g. const char* myString. const char* indicates that we have an array of characters, which is really an array of signed 8-bit integers. This means we can look at where the string is stored in the typed array buffer mentioned above, and interpret a contiguous block of memory as the array of characters relative to the string's memory pointer. A pointer is, at a high level, an index into a block of memory representing a value. We can represent a string in memory with a pointer called str as such:

str	str + 1	str + 2	str + 3
72	105	33	0

This block of memory forms the string "Hi!". See how there's a "fourth character" with a value of 0? This is what we call a "null terminator," which signifies the end of the string in memory. It's often easier to work explicitly with string length instead of looping through memory looking for a 0. We can get the length of any string in C/C++ with the strlen function. With a pointer and string length in hand, we can iterate over the memory and reconstruct the string with the following code:

const view = new Uint8Array(memory.buffer, pointer, length);
const string = '';
for (let i = 0; i < length; i++) {
  string += String.fromCharCode(view[i]);
}
console.log(string);

Now we're ready to write the code to bind them together! First, let's write the following C++ to use a JavaScript function:

extern "C" void jsPrintString(const char *s, uint16_t len);

void print() {
  const char* str = "Hello from C++!";
  jsPrintString(str, strlen(str));
}

Note the first extern "C" line. This does two things: 1) defines the function signature for a function named jsPrintString that we will implement in JavaScript, and 2) tells the compiler to use C name mangling instead of C++ name mangling. C and C++ compilers change the name of function signatures so that overloaded versions can be easily identified. This is a simple algorithm in C because it doesn't allow much overloading, and only prepends a _ to the name. C++ is a lot more complicated though, and you can end up with names like _Z16RVLMessagingLoopv for a function called RVLMessagingLoop in code. We'll see why this is important in a minute.

Note: Make sure to add -s ERROR_ON_UNDEFINED_SYMBOLS=0 to your em++ build command. This will prevent the compiler from erroring when a defined function's implementation cannot be found in C++. This is expected since the function is defined in JavaScript, not C++. Note: be careful with this option, as you may miss actual issues in your C++ code with this option enabled. Always be sure to compare the list of symbols it didn't find with what you expect not to be found.

Then we have our print function, which will invoke the JavaScript function. We define a character array with const char* and assign it a string value. str is now a pointer to the string in memory. Pointers are also numbers! This means we can pass the pointer straight from C++ to JavaScript without having to do anything special.

Now it's time to modify the JavaScript code. We're going to wrap our string reconstruction code in a function called handlePrintString. Then, we inject it into C++ code by modifying the env object we pass to the WASM instantiation. We assign this function to the env object with the key _jsPrintString (note the leading underscore). This name in env is the mangled name of the function in C/C++. This is why we want to use C mangling instead of C++ mangling. Finally, we can invoke the print function in C++ from JavaScript, which calls back into JavaScript to log the string.

function handlePrintString(ptr: number, len: number) {
  const view = new Uint8Array(memory.buffer, ptr, len);
  let string = '';
  for (let i = 0; i < len; i++) {
    string += String.fromCharCode(view[i]);
  }
  console.log(string);
}

const env = {
  ...
  _jsPrintString: handlePrintString,
  ...
};
WebAssembly.instantiate(bytes, { env }).then((result) => {
  result.instance.exports._print();
});

And there we have it, we can now passing strings from C++ into JavaScript! It may sound small, but this marks a big step forward towards integrating this system with Azure IoT Edge via Node.js.

Embedding emscripten in a Node.js library

Bryan Hughes — Fri, 15 Mar 2019 20:42:27 +0000

I've been experimenting with Web Assembly lately, and right now I'm in the early stages of getting the messaging stack of my wireless LED control system running in Node.js via WASM. I've gotten it up and running (yay!), but it's pretty hacky right now (boo!).

The Scenario

My library is written in C++. I intentionally avoided using anything from the C++ Standard Library and instead used the C Standard Library exclusively (it's more portable and I suspect less complicated to compile). Either way though, both of these standard libraries are runtime libraries that need to be compiled in to the output. If you're a JavaScript developer and have no idea what I'm talking about, imagine if jQuery or Underscore/Lodash were defined as part of the JavaScript spec and were bundled with Node.js, but were still separate libraries. That's the C/C++ Standard Libraries.

All C++ compilers come with these libraries built-in, and you don't have to do anything special to use them. emscripten comes with implementation for these as well, but IME they're still tricky to use. You have to change the compile flags. Specifically, you have to remove the -s ONLY_MY_CODE=1 flag that I mentioned in my previous blog post on WASM. If this is the only change you make to everything in that post, your app will crash with the following:

[LinkError: WebAssembly Instantiation: Import #1 module="env" function="nullFunc_ii" error: function import requires a callable]

Uhm, excuse me, what exactly is nullFunc_ii and why should I care?

So here's what's going on, to the best of my understanding. emscripten compiles your code and injects any and all runtime libraries necessary to run a C++ application. This includes the aforementioned standard libraries, but also includes some other things. Most notably, emscripten injects some runtime libraries to handle things like stack overflows, exception handling, segfaults, etc. I'm about 75% certain that the nullFunc_xxx methods are part of the latter.

These methods are all defined in JavaScript, not C++, and so are not included in the output.wasm file. Instead, they're included with a JavaScript runtime file called output.js (given my -o flag value).

My Hacky Solution

So how did I get around this? My first step was to check out the emscripten docs on output files and formats and the various emscripten specific config flags.

As far as I can tell, what I want to do isn't possible. emscripten allows you to either compile code on it's own (via the -s ONLY_MY_CODE=1 flag), or to compile a complete application that includes a void main() {} (i.e. not a library). Uhmmm...ok...?

After a lot of trial and error, I found a really hacky solution that seems to work.

First, here's my complete compile command I'm now using (note: you can ignore the ERROR_ON_UNDEFINED_SYMBOLS part, I'll talk about that in a later post):

em++ -s WASM=1 -s ERROR_ON_UNDEFINED_SYMBOLS=0 -s EXPORTED_FUNCTIONS=\"['_init','_loop','_setWaveParameters']\" -std=c++11 -Isrc -g4 -o js-dist/output.js js/*.cpp

Note how we have -o js-dist/output.js in the command. This tells emscripten to generate a JavaScript runtime file. This file is intended to be used as a "main" file, i.e. an entire application. We want to use it as a library though, not an application. There are a lot of things in here we need though, most notably two variables it creates called asmGlobalArg and asmLibraryArg. These variables define all of the nullFunc_xxx methods, among others. These variables are not exported in any way though, and as far as I can tell are not meant to be consumed directly.

We're going to do it anways, damn the consequences! I wrote a script to automatically hack this file with the following contents:

const { readFileSync, writeFileSync } = require('fs');
const { join } = require('path');

const OUTPUT_FILE = join(__dirname, '..', 'js-dist', 'output.js');

let source = readFileSync(OUTPUT_FILE).toString();

source = source.replace('var asmGlobalArg =', 'var asmGlobalArg = module.exports.asmGlobalArg =');
source = source.replace('var asmLibraryArg =', 'var asmLibraryArg = module.exports.asmLibraryArg =');

writeFileSync(OUTPUT_FILE, source);

Now we can import these variables into our main file (which is now written in TypeScript FWIW):

import { readFile } from 'fs';
import { join } from 'path';

import { asmGlobalArg, asmLibraryArg } from './output';

let wasmExports: WebAssembly.ResultObject | undefined;
const memory = new WebAssembly.Memory({ initial: 256, maximum: 256 });

readFile(join(__dirname, 'output.wasm'), (readErr, buf) => {
  const bytes = new Uint8Array(buf);
  const env = {
    ...asmLibraryArg,
    table: new WebAssembly.Table({
      'initial': 192,
      'maximum': 192,
      'element': 'anyfunc'
    }),
    __table_base: 0,
    memory,
    __memory_base: 1024,
    STACKTOP: 0,
    STACK_MAX: memory.buffer.byteLength
  };
  const global = {
    ...asmGlobalArg
  };
  WebAssembly.instantiate(bytes, { env, global })
    .then((result) => {
      wasmExports = result;
      wasmExports.instance.exports._init(deviceId);
    })
});

And it works! But it's also pretty ugly IMO. I'm excited though, cause I'm one step closer to integrating this system with Azure IoT Edge so I can control my LEDs from the cloud!

Question for all of you

Am I missing something here? Is there a proper way to do this? I find it hard to believe that emscripten doesn't have an option for compiling a library that includes all the necessary runtime libraries, but I can't seem to figure out how.

Experimenting with Web Assembly and Node.js

Bryan Hughes — Tue, 12 Mar 2019 23:42:50 +0000

I've been wanting to play around with Web Assembly for a while, and finally had a chance to. Since I'm a mostly Node.js developer, I wanted to play around with it here instead of in the browser. There's not a lof of documentation on Web Assembly though, and the APIs are changing rapidly. I finally got it to work though, and here's how I did it.

I mostly followed the emscripten Getting Started guide, and tweaked it to fit both Node.js and to account for parts of this doc that are out of date.

Installing emscripten

The first step is to install emscripten, which is well documented on emscripten's website. I installed it inside of Windows Subsystems for Linux running Ubuntu, so YMMV if you're using a different OS.

The first step is to install some dependencies. This takes a while, so I recommend brewing a nice cup of tea during this time. I chose a nice Silver Needle this time around :)

# Install Node.js using NodeSource's Ubuntu PPA, if not already installed
curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -

# Install dependencies, as defined in emscripten's getting started guide
sudo apt-get install python2.7 nodejs cmake default-jre

Then we're ready to install emscripten itself. This also takes some time, so enjoy some more tea.

git clone https://github.com/emscripten-core/emsdk.git
cd emsdk
./emsdk install latest
./emsdk activate latest
source ./emsdk_env.sh

I ran into a bit of an issue with this approach. I don't know if I made a mistake, or things are just locked in. Either way, emscripten installed version 8.x.x of Node.js on it's own and overrode my 10.x.x install of Node.js. Since I'm using async/await for my Node.js code (which requires Node 10 in practice, if not in theory), I needed to manually override this by editing ~/.emscripten so the following line matches the code below:

NODE_JS = '/usr/bin/node' # This line used to have a super long path to the emsdk directory

I never ran into any problems swapping out the version of Node.js that emscripten expected, but as always, YMMV.

Creating the code

First, I created a very simple C file, courtesy of an older blog post at Dynamsoft, named test.c. (Note: the instructions in this blog post no longer work).

int add(int a, int b) {
  return a + b;
}

Then, I created the JavaScript file that consumes this C module:

const fs = require('fs').promises;
const util = require('util');

async function run() {
  async function createWebAssembly(bytes) {
    const memory = new WebAssembly.Memory({ initial: 256, maximum: 256 });
    const env = {
      abortStackOverflow: (err) => { throw new Error(`overflow: ${err}`); },
      table: new WebAssembly.Table({ initial: 0, maximum: 0, element: 'anyfunc' }),
      __table_base: 0,
      memory,
      __memory_base: 1024,
      STACKTOP: 0,
      STACK_MAX: memory.buffer.byteLength,
    };
    return WebAssembly.instantiate(bytes, { env });
  }

  const result = await createWebAssembly(new Uint8Array(await fs.readFile('./output.wasm')));
  console.log(util.inspect(result, true, 0));
  console.log(result.instance.exports._add(9, 9));
}
run();

A few things to note about this JavaScript code. First, the code is a masheup of the code from the Dynamsoft blog post and the emscripten getting started guide. I recommend reading the emscripten getting started guide for an in-depth explanation of what's going on here. The next thing to note is the __table_base and __memory_base entries. These are named tableBase and memoryBase in the emscripten documentation and in almost all other documentation I've found, but crashes when run. I'm guessing the API changed very recently. Hat tip to this gist for pointing me in the right direction.

Running the code

Now that we've written our files, it's time to run the code. First we compile this code with the following command:

emcc -s WASM=1 -s ONLY_MY_CODE=1 -s EXPORTED_FUNCTIONS="['_add']" -o output.js *.c

This command will generate the file output.wasm that's referenced in the JavaScript code. Now we're ready to run our code!

> node index.js
{ instance: Instance [WebAssembly.Instance] {},
  module: Module [WebAssembly.Module] {} }
18

And it works! We can indeed add 9 and 9 together to get 18, calling from JavaScript to C++ and back again, all using Web Assembly!

My next step is to take the C++ messaging stack from my LED synchronization and animation project, wrap it in Web Assembly so Node.js can talk to my boards, and integrate it with Azure IoT Edge so I can control my LEDs from the cloud!

twitter-replies-parser - A Tool for Converting Twitter Replies into JSON

Bryan Hughes — Thu, 07 Mar 2019 19:24:59 +0000

Some of my teammates and I recently asked Node.js folks on Twitter what tools they used and why the liked them. We're collecting this so we can help our product teams at Microsoft create better Node.js tools for Azure that serve you, the community. Turns out, lots of folks have a lot of great opinions and feedback on this subject. This is great!

What's not so great is that we had so many replies we couldn't just copy-paste them from the web client 😱. So what to do? Write some JavaScript, of course!

It was a little easier said than done though. As it turns out, Twitter doesn't have any APIs available to get the replies to a tweet. Instead, I had to use the search API and then create the reply tree by hand. With a little coding work and a handful of recursive calls, I got it all working to a high degree of accuracy. Due to the approach used, it's not 100% accurate. I noticed about 2-3 tweets were missing out of the ~60 replies to my request for feedback, but close enough!

This tool is a Node.js application that spins up a web server with a simple web app for entering the URL of the tweet you want to get the replies to. It's definitely not pretty, but get's the job done:

You can check out the tool on GitHub, which is released under the MIT open source license. I'd love to hear what you think, and if you run into any issues using it.

Writing Unit Tests For A Rewrite: A Case Study

Bryan Hughes — Tue, 26 Feb 2019 18:56:01 +0000

This blog post is the first post in a series that discusses my efforts to convert Raspi IO to TypeScript and modernize its architecture. This blog post series will explore how to write unit tests specifically for rearchitecting or rewriting a project, how to create TypeScript base classes and functionality that is shared across multiple TypeScript and non-TypeScript projects, and how to convert an existing code base to TypeScript all in one go.

This post was first published to the Azure Medium publication.

All codebases age and mature over time. With age brings stability, and older projects are typically more reliable as a result.

However, age also brings with it creaks and groans as the original architecture struggles to keep up with modern user needs. Time also brings newer, better ways of developing these projects, and what was once cutting edge often becomes clunky and slow.

So the question for these projects becomes: to rewrite, or not to rewrite? I faced such a question with my oldest project still under development: Raspi IO.

Raspi IO is a plugin for the Johnny-Five Node.js robotics and IoT framework that enables Johnny-Five to run on the Raspberry Pi. I first created it as a monolith in 2014, but the original architecture quickly ran into limitations as I added more features. I rewrote the library the following year and split it into multiple modules. This rewrite allowed the project to scale as more features were added.

Raspi IO currently consists of 11 modules. Nine of these modules comprise what I call Raspi.js, which can be used independently of Raspi IO and Johnny-Five. These modules together provide a complete API for interacting with hardware on the Raspberry Pi in a uniform way. Raspi IO and Raspi IO Core together provide a translation layer from Raspi.js to the IO Plugin spec for Johnny-Five. Raspi IO Core is platform independent, and Raspi IO injects Raspi.js into Raspi IO Core to create a Raspberry Pi specific IO plugin.

Over time, all of Raspi.js has been converted to TypeScript and updated to modern coding practices. Raspi IO and Raspi IO Core, however, have remained more or less unchanged for three years. This is fine for Raspi IO, which only contains 32 lines of code, but not for Raspi IO Core. Inside, there are 1000 lines of dense JavaScript, replete with hacks for strange edge cases and bugs. This codebase definitely falls under the classic case of “afraid to make changes because it might break everything.” It’s also in dire need of updating to TypeScript and modern coding conventions.

With the need clear in my head, I sat down and devised a plan to rewrite Raspi IO Core without breaking it for my users. The first step in this rewrite was to implement unit tests with a high degree of code coverage, as Raspi IO Core did not have unit tests for historical reasons (unit tests involving hardware are tough).

While major refactors and rewrites bring a lot of advantages to them, such as state-of-the-art best practices and modern tooling, they are inherently risky from the standpoint of breaking your users. Unit tests act as insurance to make sure that the rewrite is as transparent to users as possible.

Methodology

So how does one implement unit tests for a project that has no unit tests and needs to be rewritten? Very methodically, and following a specification.

As previously mentioned, Raspi IO Core implements a published specification called the IO Plugin Spec. This spec provides a blueprint for how the module is supposed to behave, and in effect provides a blueprint for the unit tests themselves.

Not all projects implement an API spec, but hopefully there are design documents or other documentation describing what the project is supposed to do. If not, then the first step in implementing unit tests is to write such a spec. It’s a lot of work, but I promise it will help tremendously down the road. In addition to making it easier to implement unit tests, it provides a place for all stakeholders, not just coders, to provide input on the project and make it better. If you’re unsure where to start, Read the Docs has good content on writing quality specifications.

Next up was to decide on a unit testing tech stack. I decided to go with a common stack for open source Node.js modules because I’m already familiar with them, and didn’t want to learn new tools or platforms at this time:

Jasmine: a Behavior Driven Development (BDD) test framework.
Istanbul: a JavaScript code coverage tool. Code coverage tools measure how much of your codebase is executed by your unit tests, and provides a useful proxy measure of how much of your code is tested by unit tests.
Travis CI: a hosted unit testing platform that makes it easy to run unit tests on GitHub activity (e.g. when a PR is submitted, when pushing/merging to master, etc). Although not strictly required for the rewrite, it’s generally a good idea to wire up unit tests to a hosted platform such as Travis CI. This allows developers who are considering using your library to see unit test results without having to download your code and run tests themselves.
Coveralls: a hosted code coverage platform that integrates with Travis CI, and provides all the value that Travis CI does, except for code coverage instead of unit tests themselves.

With the specification and unit testing infrastructure in place, it was time to write my unit tests!

Walkthrough of a Unit Test

To illustrate how to write an effective unit test, I’m going to do a deep-dive walkthrough for one part of the IO spec: the digitalRead method. The IO Plugin spec has this to say about the digitalRead method:

digitalRead(pin, handler)

Initiate a new data reading process for pin
The recommended new data reading frequency is greater than or equal to 200Hz. Read cycles may reduce to 50Hz per platform capability, but no less.
Invoke handler for all new data reads in which the data has changed from the previous data, with a single argument which is the present value read from the pin.
A corresponding digital-read-${pin} event is created and emitted for all new data reads in which the data has changed from the previous data, with a single argument which is the present value read from the pin (This can be used to invoke handler).

We can break the things this spec says we must do down into a few different things we need to test, which will become our set of unit tests. Reading through the spec, I identified the following five tests:

The third bullet point indicates we need to test reading a value via the handler argument as the pin value changes over time.
The fourth bullet point indicates we need to test reading a value via the digital-read-${pin} event as the pin value changes over time.
The second bullet point indicates we need to test that handler is called at 50hz or faster.
The third and fourth bullet points indicate we need to test that the method doesn’t report the same value twice in a row.
Implicit in this and other parts of the spec is that we need to test that digitalRead continues to read even when the mode is changed to output mode and reports the output value that was set via digitalWrite.

Now that we’ve identified five unit tests we want to write, the next step is to figure out how to write them. At the end of the day, unit tests exist to confirm that the correct outputs are generated given a reasonably complete sampling of inputs. So the first step in any unit test is to identify the inputs and outputs.

We tend to think of inputs and outputs as the arguments we pass to functions, and the values they return. These are not the only inputs that exist though. For example, if we’re testing a function that saves a value to the database, then the call to the database is also an output, in addition to what the function returns or the callback it calls. In the case of digitalRead, we’re calling other modules that talk to hardware (more outputs and inputs!). In general, it’s quite common for there to be two or more sets of inputs and outputs.

The trick in unit testing is to figure out how to measure the inputs and outputs on the “back end” of the diagram below. Most often, this is done using mocking, and is the solution I chose to use here. The architecture of Raspi IO Core makes this pretty straightforward to do because we can pass in mocked versions of all the modules in Raspi.js. The full set of inputs and outputs we’re testing are shown below:

These mocked versions include a virtual implementation of hardware, and expose the inputs/outputs to this module such that we can verify them in our unit tests. For this unit test, we use the DigitalInput mock, which has the following code:

class DigitalInput extends Peripheral {
  constructor(...args) {
    super([ 0 ]);
    this.value = OFF;
    this.args = args;
  }
  read() {
    return this.value;
  }
  setMockedValue(value) {
    this.value = value;
  }
}

We’ve added an extra method called setMockedValue that doesn’t exist in the real Raspi GPIO DigitalInput class. This allows us to precisely control what Raspi IO Core will be reading. We also add a new property called args that we can use to see what parameters were passed to the class constructor. With this in place, we can measure all of the inputs and outputs to the “back end” of the black box we’re testing.

Now it’s time for the unit tests themselves. We’re going to take a look at a single unit test that tests using the callback to read the value:

it('can read from a pin using the `digitalRead` method',
    (done) => createInstance((raspi) =>
{
  const pin = raspi.normalize(pinAlias);
  raspi.pinMode(pinAlias, raspi.MODES.INPUT);
  const { peripheral } = raspi.getInternalPinInstances()[pin];

  let numReadsRemaining = NUM_DIGITAL_READS;
  let value = 0;
  peripheral.setMockedValue(value);
  raspi.digitalRead(pinAlias, (newValue) => {
    expect(value).toEqual(newValue);
    if (!(--numReadsRemaining)) {
      done();
      return;
    }
    value = value === 1 ? 0 : 1;
    peripheral.setMockedValue(value);
  });
}));

We start with some initialization code to get a test pin ready to read. We then call getInternalPinInstances, which is a special hook method that’s only exposed when we’re running unit tests. This returns the mocked instance of DigitalInput so we can access the hooks in DigitalInput we discussed above.

Then, we set up some state monitoring variables. Since this method is supposed to read data continuously, we must test that it can read more than once. numReadsRemaining tracks how many reads we’ve performed and how many we have left to go. We toggle the value each callback since it won’t call the callback if the value doesn’t change. In each callback, we test that the value that Raspi IO Core reports is the same value that we set in the mocked DigitalInput class.

And with that, the unit test is complete! If you’d like to see all of the unit tests that comprise the DigitalInput tests, you can find them on GitHub.

Lessons Learned

Throughout this process, I’ve learned several important lessons about unit tests and rewrites.

Edge cases are more important than common cases.

We test our common cases a lot, and our code is written with these common cases in mind. Edge cases, more often than not, are found through trial and error, or user reports. As such, when we’re rewriting an existing codebase, we want to make sure that we port the edge cases over as they’re much less likely to be fixed “out of the gate.” Getting unit tests to test these edge cases is the most effective way to ensure we get these edge cases included in the rewrite.

Always be specific, not general

When writing unit tests, it’s easy to write something quick that more or less tests what we want. For example, if we’re testing whether or not a function throws an exception when it’s given an incorrect parameter, we could write something like this:

expect(() => {
  add(NaN, `I'm not a number`);
}.toThrow();

This will indeed pass, but how do we know it passed because the add method correctly detected that we tried to add two non-numbers? What if there was a legitimate bug in the code that coincidentally threw on the same inputs? We should instead write this test as:

expect(() => {
  add(NaN, `I'm not a number`);
}.toThrow(new Error(`non-numbers passed as arguments to "add"`);

This way, we can ensure that it’s throwing the way we expect. This also helps us to prevent typos if we aren’t copy-pasting the error message over. This may not seem like a big deal, but sometimes user’s code depends on the content of the error message because they need to make a decision based on which error is thrown. If we change our error message, we break this code. For an in-depth discussion of why error messages are important (and tricky), I recommend reading how the Node.js project itself is changing how it does error handling.

Good code coverage is more important for rewrites than it is for day-to-day development.

In an ideal world we’d all have 100% code coverage. In practice, however, 100% code coverage is rarely ideal, and sometimes impossible. Indeed, Raspi IO Core sits at 93% coverage because most of the code not being testing is dead code. Most of this dead code is runtime code introduced by Babel itself, which is admittedly an outdated version. The rest is code that I thought was necessary but is most likely dead code in practice. There are also cases where some code is so tightly bound to something not present during testing (like, say, an external sensor), that mocking everything necessary would lead to a unit test that really is only testing the mocks, not the code itself.

It’s expected not to have 100% code coverage, but it’s more important to have high code coverage for a rewrite than for day-to-day coding. This is because of statistics. During a rewrite, we are changing vast swaths of our code which end up being covered by large numbers of unit tests, and thus large numbers of edge cases. Day to day coding rarely has such far reaching changes though. As such, the chance of regressions is higher during a rewrite. Having high code coverage is the most effective way at preventing regressions in general, and so high code coverage is especially important when we’re dealing with changes that have high risk of regressions, such as a rewrite.

Writing unit tests against a spec also improves the spec

As much as we want to view specifications as infallible, they’re created by humans. And just like humans who create code, humans who create specifications sometimes make mistakes and introduce bugs in the spec. Writing unit tests against a spec will often highlight areas of the spec that are ambiguous or contain errors. In creating the unit tests for Raspi IO Core, I uncovered multiple issues with the spec. In three of the cases, we simply forgot to update the spec with some new features that were added. In two other cases, the spec was ambiguous. Going through the process of writing unit tests can be a surprisingly effective way at sussing out issues in the specification.

Conclusion

I’ve attempted to convert Raspi IO Core to TypeScript probably 4 or 5 times in the past. Each previous attempt failed because I quickly became uncertain that I could provide a painless upgrade path for my users. Without unit tests, I was not confident in my changes. Writing these unit tests was the key missing ingredient in these previous attempts, and now I’m set to move forward with converting Raspi IO Core to TypeScript, and re-architecting major parts of it in the process.

This effort has really reiterated the importance of unit tests, as well as the importance of understanding what we test, how we test it, and why.

Forem: Bryan Hughes

Introducing React Strapped: a Bootstrap Data Management Library

Background

Problem statement [1]

Wait, why doesn't Jotai/Recoil handle requirement 2? [2]

How React Strapped works [3]

Simple example

Multi-page Apps

Conclusion

Footnotes

Interacting with Microsoft's LED art piece at JSConf EU

Basic Animations

Custom

A New Coding Style for Switch Statements in JavaScript/TypeScript

Update:

Building an Interactive LED Art Piece for JSConf EU

Passing structured data between C++ and JavaScript in Web Assembly

The Scenario

My Solution

Automating the calculation of offsets

Using offsets to read from a C++ struct in JavaScript

Passing strings from C++ to JavaScript in Web Assembly

The Scenario

The Solution

Embedding emscripten in a Node.js library

The Scenario

My Hacky Solution

Question for all of you

Experimenting with Web Assembly and Node.js

Installing emscripten

Creating the code

Running the code

twitter-replies-parser - A Tool for Converting Twitter Replies into JSON

Writing Unit Tests For A Rewrite: A Case Study

Methodology

Walkthrough of a Unit Test

Lessons Learned

Edge cases are more important than common cases.

Always be specific, not general

Good code coverage is more important for rewrites than it is for day-to-day development.

Writing unit tests against a spec also improves the spec

Conclusion

Problem statement ^[1]

Wait, why doesn't Jotai/Recoil handle requirement 2? ^[2]

How React Strapped works ^[3]