Forem: 🎙 m4dz

Hold On to Your Socks: High-Speed Data Stream Hosting with Websockets

🎙 m4dz — Tue, 14 Mar 2023 15:48:16 +0000

Created by WHATWG in 2011 and standardized by IETF in the same year, the WebSocket protocol has been providing web application developers with the ability to break free from the traditional TCP Client/Server model for real-time data transfer for over 10 years.

In the initial model, the client requests a resource from the server,the server returns it, and the communication ends. It is not possible for the server to easily "push" messages to the client (such as notifications).

WebSocket provides support for bidirectional communication (known as full-duplex) in which the server can now spontaneously "push" data to the client, and where the client can subscribe to messages that interest them and react to them later. Application interactivity, directly in the browser!

In 2023, how does hosting a WebSocket server application work?

When A Stanger Calls

Let's get back to basics: WebSocket is a network protocol in the application layer (in the famous OSImodel). You don't need to be a networking expert to use it, so everything will happen at the code level. It uses a traditional TCP connection between the client and server and the HTTP architecture.

To put it simply:

The client will request the initial application resource (HTML + JS) from the server.
This resource will be responsible for establishing communication with the WebSocket server to receive notifications.
The WebSocket server will register the client as a recipient and push relevant data to them when necessary.
The client, which was previously waiting, will receive data streams from the server and can process this data.

Note that in WebSocket, there is Web: the architecture is no different from traditional web applications. We still use HTTP/TCP, we just use a different application protocol. This means that we will need to use dedicated libraries.

The Service That Does "Pong"

There are plenty of libraries that can provide WebSocket support in almost all available web languages. For the purpose of this demonstration¹ we will implement a small WebSocket server in Node.js and then in Python. You are free to implement it in PHP, Ruby, Java, Perl, etc.

Our example is very simple: once connected, our client will be able to send the ping message to the server, which will then send pong back to all connected clients one second later. This simple example demonstrates the interactivity between all connected elements (i.e.broadcasting) of our application.

With Node.js…

Currently, the most reputable library for creating a WebSocket server application in Node.js is websockets/ws. Add the ws package to your project using npm² and create a wss.js file for your WebSocket server:

import WebSocket, { WebSocketServer } from 'ws';

const wss = new WebSocketServer({
  host: process.env.IP || '',
  port: process.env.PORT || 8080
});

We can attach the WebSocket server to the IP/PORT pair exposed in the environment variables for more flexibility, with a fallback to port 8080 for ease of development.

Our server is ready, now it needs to be equipped with its functionalities. First and foremost, it should be able to receive client connections.

wss.on('connection', (ws) => {
  // log as an error any exception
  ws.on('error', console.error);
});

Next, when a client sends the ping message, the server should respond with pong to all active clients.

wss.on('connection', (ws) => {
  /* ... */
  ws.on('message', (data) => {
    // only react to `ping` message
    if (data != 'ping') { return }
    wss.clients.forEach((client) => {
      // send to active clients only
      if (client.readyState != WebSocket.OPEN) { return }
      setTimeout(() => client.send('pong'), 1000);
    });
  });
});

That's all. To launch the WebSocket server, simply run the wss.js file with Node.js:

$ node wss.js

… or with Python!

To change things up from the usual examples, let's create the same WebSocket server in Python using asyncio and websockets. Start by installing the websockets package using pip in your venv, and then create a wss.py file:

#!/usr/bin/env python

import os

import asyncio
import websockets


async def handler(websocket): pass


async def main():
    async with websockets.serve(
        handler,
        os.environ.get('IP', ''),
        os.environ.get('PORT', 8080)
    ):
        # Run forever
        await asyncio.Future()


if name == "__main__":
    asyncio.run(main())

Now let's define our functionalities: registering clients, and broadcasting the message when a ping is received. Add to the handler method which contains the logic for our server:

connected = set()


async def handler(websocket):
    if websocket not in connected:
        connected.add(websocket)
    async for message in websocket:
        if message == 'ping':
            await asyncio.sleep(1)
            websockets.broadcast(connected, 'pong')

The Web Client Side

Our client will be simple: a web page with JavaScript that connects to the WebSocket server, sends a ping on connection, and has a method to manually send a new ping.

Create an index.html file for your client:

<!DOCTYPE html>
<script>
    WS_SERVER = 'localhost:8080'
    dateFormatter = new Intl.DateTimeFormat('en-US', {
        hour: "numeric",
        minute: "numeric",
        second: "numeric"
    })

    const websocket = new WebSocket(`ws://${WS_SERVER}/`)

    const ping = (msg) => {
        msg = msg || 'ping'
        console.log("Send message", msg)
        websocket.send(msg)
    }

    websocket.addEventListener('message', ({data}) => {
        console.log("Recv message", data, dateFormatter.format(Date.now()))
    })

    websocket.addEventListener('open', () => ping())
</script>
<p>Open the developer console and run the <code>ping()</code> function</p>

Start your WebSocket server (Python or Node.js), and open this HTML page with the developer tools open. You should see ping/pong messages appear in the console. Try manually executing the ping() function in the developer tools.

Now open this HTML page again in another tab, with the developer tools open. Execute the ping() function in either tab. Both will receive the pong from the server.

Congratulations, you have achieved a basic level of bidirectional broadcast communication between the client and server in full-duplex via WebSocket!

Noot Noot: WebSite or WebService ?

We now need to deploy this WebSocket server and client in a production environment.

At alwaysdata, we offer several solutions for deploying tools that need to be executed for long periods of time and accessed by clients in the future: Sites and Services.

For Sites, it’s straightforward: deploying a web server (Apache, WSGI, Node, etc.) that will be requested later by a client to obtain various resources via HTTP. This is the traditional historical web.

Services, on the other hand, are designed to run long processes that can potentially be accessed other than via HTTP, within the environment of your account (such as a Git repository server over SSH with soft-serve, or a monitoring/filtering system for messaging.)

So, for a WebSocket server, should you use Sites or Services?

While one might imagine that Services are the right place — for a long-running process that can be accessed from outside — I will repeat this here: WebSocket includes Web! The WebSocket protocol is software that uses HTTP/TCP, just like any Site.

A plan that comes together

Our application consists of two parts: a client and a WebSocket server. The WebSocket server cannot serve as a resource like a traditional web server (that's not its role), so you will need two sites:

A Static files type site that will serve your index.html client.
A second site adapted to the language of your WebSocket server to execute it.

For the WebSocket server, use an address of the type: [my-account].alwaysdata.net/wss. Your WebSocket client must connect to this address. Make sure the Trim path option is enabled as the server is hosted behind a pathUrl. You may need to set the value of the idle time to 0 to ensure that it will never be stopped by the system, and to keep the connections to the clients active, even in case of prolonged inactivity.

Update the index.html file to specify the WebSocket server URL in theWS_SERVER variable. Then create a Static files site with the address [my-account].alwaysdata.net to serve this file.

Head to this address: your WebSocket communication is up and running!

This article is, of course, only an introduction to the concepts of WebSocket, but it highlights several fundamental elements:

WebSocket enables bi-directional and simultaneous multi-client communication.
A WebSocket server runs in parallel with the application's Web server. The latter is responsible for distributing the application to the browser, which will then start it; but it is the former that is responsible for the transit of business data flows.
Even though it seems to be a different protocol, WebSocket exploits the fundamentals of the Web, its underlying building blocks, and its robust protocols.

You don't need to be a network expert to develop with WebSocket. You will use what you already know well from the Web, with its event model.

It's up to you to find the right uses that suit your needs; to add data processing; to transit JSON or binary formats; to support authentication… Everything you already know how to do on the Web is applicable.

Happy coding!

and to avoid always doing the same thing ↩
or pnpm or yarn ↩

Devops React : We can be Heroes, just for one app

🎙 m4dz — Wed, 01 Feb 2023 14:01:04 +0000

In an increasingly advanced front-end ecosystem, with an ever-richer selection of frameworks and increasingly demanding tooling, one can sometimes find oneself in a not so comical situation: how to handle a reliable and effective deployment? A case study with Create React App, let’s go!

Lexical Note

This guide will take place in two environments:

Local: run these commands on your local development environment (likely: your laptop).
Server: on your hosting account. With alwaysdata, you can connect over SSH to run these commands.

For an account on alwaysdata, mentions of <account> in paths/URLs/etc. refer to your account name (i.e. if your account name is superman, replace <account> with superman).

Create React App, what’s that?

Whether you missed the last few years of front-end development or you’re a bit lost in your first steps with React, a quick reminder: React is a JS/TypeScript framework that provides a development environment for your websites and applications. I want to emphasize the front-end aspect of this topic: even though it’s possible to use React with server-side rendering, its primary use case is to generate front-end assets: HTML, CSS, and JavaScript. Nothing more.

Since a React app is just a set of static files, there’s no need for a complex back-end architecture to serve your front end. No Node.js server or anything. A simple HTTP web server is enough. As stated in the Create React App deployment documentation:

Set up your favorite HTTP server so that a visitor to your site is served index.html, and requests to static paths like /static/js/main.<hash>.js are served with the contents of the /static/js/main.<hash>.js file.

I’ll take this opportunity to mention Create React App. If React is a framework dedicated to designing Web applications, setting up all of its toolings can be complex. Tools that encapsulate all of this logic have appeared to make things easier for you.

Development vs. Production

Let’s start our tutorial by creating a new React-based application using Create React App. In your local environment, run the command:

$ npx create-react-app my-app
$ cd my-app

Congratulations! Here is your first functional React application (without having coded anything!). Create React App has generated the basic files and installed the various modules necessary for the development and compilation of your project.

Compilation? Yes! Your front-end application will not be served directly to your clients but will have to be transpiled into JavaScript. Create React App has equipped your project with two commands:

npm start : this is your development mode command; it will launch a web server on your machine to allow you to work on your application comfortably (with hot-reloading, etc.)
npm run build : this is the command dedicated to transpiling, the one that will produce the static assets that you will then need to deploy on the server side.

On the server side now: in your alwaysdata administration interface, create a new site of type Static files, with the following information:

address : <account>.alwaysdata.net/my-app
root directory: /www/my-app/build

Creating the “Static Files” website

As this directory does not exist yet, create it from your local environment over SSH:

$ ssh <account>@ssh-<account>.alwaysdata.net mkdir -p www/my-app/build

Manual Deployment: The Simple and Quick Option

Everything is ready on the server side to receive your React site! Let’s move on to deployment.

You first need to indicate the final URL of your project in the package.json file to allow the build task to generate the assets with the correct file paths. Add the following entry:

"homepage": "<account>.alwaysdata.net/my-app"

Still from your local environment, launch the build task:

$ npm run build

You get a build directory that contains your static assets. To deploy them, use rsync via SSH:

$ rsync -rz build <account>@ssh-<account>.alwaysdata.net:www/my-app

You can go to your site’s address and admire the first production release of your application!

Automated Deployment from Github: Be Professional

This manual deployment works perfectly, but it puts a burden on your shoulders. Each time you want to deploy your project, you will have to launch the build task and then deploy the files via rsync/SSH.

Let’s move on to the DevOps part by automating the process using GitHub Webhooks!

Webhooks, the magic bond

With Webhooks, GitHub will be on duty to notify your site that a new version is available every time you push new commits to the main branch.

To do so, you will need to equip your alwaysdata account with a webhook service that will execute the deployment task for you. The adnanh/webhook project will do the trick. Connect to your account with SSH and install it:

$ mkdir webhook
$ wget -O- https://github.com/adnanh/webhook/releases/download/2.8.0/webhook-linux-amd64.tar.gz \
  | tar -xz --strip-components=1 -C webhook

Create a new site in your alwaysdata admin interface of type User Program with the following information:

address: <account>.alwaysdata.net/webhook
root directory: webhook
command: ./webhook -port $PORT -hooks ./hooks.yaml -logfile webhook.log
remember to also check the Force HTTPS usage and Trim path checkboxes

GitHub, The Central Perk

On the GitHub side now, we’re going to set up a new repository for our project. Create it as usual by giving it the name of your project.

Then go to the Settings section to add a new Webhook to your server:

payload URL : https://<account>.alwaysdata.net/webhook/hooks/deploy
secret : a random password
content-type : application/json

Call me WALL·E

Our GitHub repository is configured, it will now trigger requests to your webhook server for new events, such as git push. All that’s left is to automate the deployment.

Connect with SSH and create the webhook configuration file by editing the webhook/hooks.yaml:

- id: deploy
  execute-command: /home/<account>/webhook/deploy.sh
  command-working-directory: /home/<account>
  pass-arguments-to-command:
    - source: payload
      name: repository.name
    - source: payload
      name: repository.clone_url
    - source: payload
      name: head_commit.id
  trigger-rule:
    and:
      - match:
          type: payload-hmac-sha1
          secret: <github_webhook_secret>
          parameter:
            source: header
            name: X-Hub-Signature
      - match:
          type: value
          value: refs/heads/main
          parameter:
            source: payload
            name: ref

Remember to replace the values of <account> and <github_webhook_secret> in this configuration, and restart the webhook site in the administration interface once done.

This deploy webhook defines the action to run when a GitHub notification is received. It will call the script webhook/deploy.sh with several parameters, but only when the event is a push to the main branch.

All that remains is the deployment script itself, which will be executed by the webhook server when a notification is received. It will be responsible of the following tasks:

Clone the repository in a temporary directory.
Retrieve the latest version specified by GitHub in its payload.
Perform the build task.
Clean up after itself.

Still, using SSH, add the script webhook/deploy.sh to your server account:

#!/bin/bash

NAME=$1
URL=$2
COMMIT=$3

CLONE_DIR=$(mktemp -d)
DEST_DIR=${HOME}/www/${NAME}

mkdir -p ${DEST_DIR}

git clone --bare ${URL} ${CLONE_DIR}
git --work-tree ${DEST_DIR} --git-dir=${CLONE_DIR} checkout --force ${COMMIT}
npm --prefix=${DEST_DIR} install
npm --prefix=${DEST_DIR} run build
rm -rf ${CLONE_DIR}

Remember to make the file executable with the command chmod +x webhook/deploy.sh.

You’re all set! Now just let everything run its course. Go back to your local environment and add the GitHub repository to your project before pushing the main branch:

$ git remote add origin https://github.com/<gh_name>/my-app.git
$ git push -u origin main

This push will trigger the notification through the Webhooks. Once notified, the alwaysdata side server will run the deployment script, and your site will be updated. You can check the progress by consulting the webhook/webhook.log file on the server side.

Make changes to your site, push to GitHub, reload your site at http://<account>.alwaysdata.net/my-app, and admire! You no longer have anything to do to manage your deployments.

The best PaaS is the one that suits you

You don’t need solutions like Netlify, Heroku, or others to manage the DevOps part of your deployments. Even though these platforms offer many practical tools to make your deployments easier (like atomic deployments, etc.), they’re not mandatory.

We have set up a simple solution here to automate the deployment of static front-end applications on your alwaysdata account without using third-party tools. Our deployment tool is independent of the application itself, so you can now use it to deploy other projects: in another GitHub repository, configure a webhook pointing to the same endpoint with the same secret, and watch your deployments happening on their own in your www directory.

It is now up to you to adapt the solution to more complex cases, integrating migration tasks, updating databases, and atomic builds to prevent service interruptions during redeployment… the list is far from its end. Happy deployment 🤓 !

A story of how we migrated to pnpm

🎙 m4dz — Wed, 26 Jan 2022 09:13:13 +0000

It all started with me trying to improve our Continuous Integration pipeline. I'm a strong believer in having proper CI - the threshold for how much to invest in unit & integration tests is always tricky to set, but to me the bare minimum should be to have linting and type checking run on every commit.

Now, having that bare minimum is great, but it also needs to be as fast as possible. When you want commits & reviews to be fast, CI cannot be the one thing holding you back.

Yet... This is what we would see in the best case scenario on that bare minimum linting & type checking job:

1 minute and 11 seconds just to install dependencies. Obviously, the job has to do more afterwards, and that's where I'd prefer it to spend time.

But wait, there's more. This was the best case scenario. You may know that package managers have caches, and a known trick to speed installs is to save that cache after CI runs, so it can be reused for subsequent runs. An easy way to do that nowadays is to use actions/node-setup's caching capabilities.

However the cache cannot always be used. As soon as the lock file changes, typically when adding dependencies, the cache isn't reused because the cache's hash is usually computed based on the lock file. We would then get:

6 minutes and 31 seconds 🐌.
That's when we really thought we needed to do something.

Where we stood with Yarn

We have been using Yarn 2 for quite some time, having originally switched to it for its native workspace support which is great for monorepos as we happen to have one. Because we use a lot of different dev tools (in no particular order - Vite, Vitepress, Astro, esbuild, Webpack, Eleventy, Firebase tools, Tailwind...) and many more actual dependencies. It's easy to understand how many dependencies we're bound to have when you see all the frameworks we support, whether on WebComponents.dev or on Backlight.

You may know Yarn 2 for introducing the Plug'n'Play linker. To make it short, it completely forfeits the idea of the node_modules resolution mechanism and tells Node to depend on Yarn for dependency resolution.
It's a really interesting idea but dropping node_modules is a compatibility challenge which kept us away from trying it. We stuck and are sticking to node_modules for now.

Anyway, because Yarn 3 had been released for a few months with performances improvements, we decided to give it try to see if that would speedup our builds.

Trying yarn 3

Upgrading to Yarn 3 is fairly simple:

> yarn set version berry

➤ YN0000: Retrieving https://repo.yarnpkg.com/3.1.1/packages/yarnpkg-cli/bin/yarn.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-3.1.1.cjs
➤ YN0000: Done in 0s 758ms

And there we go, we were upgraded to Yarn 3.

I'll spare you another pair of screenshots, but that got us down a bit, to 4 minutes 50 seconds without cache and 57 seconds with cache.

I'm sparing you the screenshots for a good reason - I did mention we have been using Yarn 2 in that monorepo for a while. We've also been adding so many packages in different workspaces that we ended up with a lot of duplicated dependencies, ie with multiple versions of the same packages.

So just for the sake of the comparison and because our original point was to speedup install times, I went ahead and completely removed the yarn.lock file and tested again.

With cache, down to 50 seconds:

And without cache, we got down to 4 minutes and 1 second:

It's fair to say we've sped up our builds quite a lot already, but we wanted to go further yet.

Now, that's where we decided to try out pnpm. But as pointed out by @larixer, there are also options you can set to speedup Yarn. We tried them out to make it a fairer comparison.

@larixer mentions the 3 following options:

nmMode: hardlinks-global
enableGlobalCache: true
compressionLevel: 0

And they do help a lot, especially without cache where we go down to 1 minute 10 seconds:

It is also slightly faster with a cache, yielding 45 seconds:

So if you are running Yarn, do considering trying them out! Chances are they will greatly improve your install times.

Anyhow, let's jump into pnpm!

Enter pnpm

pnpm stands for Performant NPM. Its adoption has been really steady as its close to the 15k stars at the moment on Github. It also comes with out of the box support for workspaces, making it easier for us to consider.

As its name indicates, it really emphasizes performances, both regarding disk space and installation times. In all the figures provided, whether from pnpm or from Yarn you can see that pnpm really comes out faster most of the time.

There seems to be two main reasons for it.

One, being performance-oriented, its implementation targets speed. You may have seen when installing with yarn or npm timings for each of the resolution/fetch/link steps. It appears that pnpm is not doing those steps sequentially globally, but sequentially for each package in parallel which explains why it's so efficient.

The other reason is the way it deals with the node_modules folder.

Centralized addressable cache

pnpm calls it a content addressable filestore, and we know other package managers like yarn or npm have caches as well, which allow you not to have to re-download.

The difference with pnpm's is that this cache is also referenced by your node_modules files, which are effectively hard-links to that cache. A hard-link means your OS will report those files as being actual files - but they're not. So the actual disk usage occurs in pnpm's cache, not in your node_modules folder. You save space, and installation time, because there's way less IO involved in setting up that infamous node_modules folder! 🪄

Non-flat node_modules

What's also interesting is the way the node_modules is organized with pnpm. npm and yarn (when using the node_modules linker) tend to do hoisting to save space as they're not using links. Hoisting is the act of installing a dependency in a parent directory rather than where it is depended on. So if you have a dependency that can be resolved to the same version pulled by two other packages, they'll try to hoist that dependency to avoid storing that same dependency twice in your node_modules.

The behavior of pnpm is different, somewhat more consistent. It's always setting up the node_modules structure the same way. First, it's non-flat. So running pnpm install vite in an empty folder will result in the following node_modules:

> tree node_modules -L 1
node_modules
└── vite -> .pnpm/vite@2.7.10/node_modules/vite

So our node_modules only contains vite and not all of its dependencies. This may seem unusual, but this avoids phantom dependencies. Phantom dependencies are dependencies that you end up being able to use without explicitly depending on them. This is a rather dangerous practice, because you do not control those - you may update the original dependency, just upgrading it to a new patch, but its dependencies may have been upgraded to major versions breaking your own code!

In our previous example, my source code won't be able to require any other dependency but vite as it's the only one which was effectively installed to the top of my node_modules.

Now we can see that this folder is actually linking to another folder in node_modules/.pnpm: this is pnpm's Virtual Store where you will find all the packages installed in your project.

If we take a peek at this folder:

> tree node_modules/.pnpm/vite@2.7.10 -L 2
node_modules/.pnpm/vite@2.7.10
└── node_modules
    ├── esbuild -> ../../esbuild@0.13.15/node_modules/esbuild
    ├── postcss -> ../../postcss@8.4.5/node_modules/postcss
    ├── resolve -> ../../resolve@1.21.0/node_modules/resolve
    ├── rollup -> ../../rollup@2.63.0/node_modules/rollup
    └── vite
        ├── bin
        ├── CHANGELOG.md
        ├── client.d.ts
        ├── dist
        ├── LICENSE.md
        ├── node_modules
        ├── package.json
        ├── README.md
        ├── src
        └── types

So, vite itself and its dependencies were installed to node_modules/.pnpm/vite@2.7.10/node_modules.
The magic that makes it all work is that Node, when resolving packages, considers the target of the symlink instead of using the symlink's path itself. So when I do require('vite') from an src/index.js file, Node finds the node_modules/vite file by iterating on parent directories looking for a node_modules folder containing vite but actually resolves it to the source of the symlink:

> node -e "console.log(require.resolve('vite'))
/tmp/foobar/node_modules/.pnpm/vite@2.7.10/node_modules/vite/dist/node/index.js

That means that any further package resolutions needed will effectively be done from this folder - so if that /tmp/foobar/node_modules/.pnpm/vite@2.7.10/node_modules/vite/dist/node/index.js file requires esbuild it will find it in node_modules/.pnpm/vite@2.7.10/node_modules/esbuild!

This is also why some dependencies don't play nice with pnpm: because they don't resolve symlink targets. But we'll get to that later.

Note: peer dependencies have a special handling in pnpm, if you're interested about it I suggest this read.

Now that we have a rough understanding of how pnpm works, let's try using it! 🚀

Migrating to pnpm

pnpm import

pnpm comes with a command to import yarn's locked dependencies:

https://pnpm.io/cli/import

There's just one gotcha when you're using it in a monorepo: the workspaces have to be declared in your pnpm-workspace.yaml first. If you don't, then at best pnpm import will only import the dependencies declared in your root file.

Dependencies which have undeclared dependencies

Another kind of problem we ran into is some dependencies having undeclared dependencies. When using yarn it was not a problem because those undeclared dependencies are sometimes very used. For example, after the migration we realized mdjs-core had not declared its dependency on slash.

A simple way to fix this is again through the readPackage hook we mentioned in the previous section. There, you can simply declare the dependency explicitly for mdjs-core:

if (pkg.name === '@mdjs/core') {
  pkg.dependencies = {
    ...pkg.dependencies,
    slash: '^3.0.0',
  };
}

shamefully-hoist when tools don't play along

We talked about the non-flat node-modules earlier. This structure is unfortunately not compatible with every Node tool.

An example of this is Astro which at the moment recommends using shamefully-hoist.
Kind of a funny name, meant to dissuade you from using it :-)

As the name implies, this one will hoist all your dependencies in your root node_modules, fixing any incompatibility you may have with dev tools not playing along with the nested node_modules. This typically happens because they don't resolve symlinks to their target.

At the time of this writing, Astro requiring it, if you're not using it will fail at loading its dependencies, with a

Error: The following dependencies are imported but could not be resolved:

  react (imported by /not-relevant/testimonial-card/src/index.tsx)
  svelte/internal (imported by /not-relevant/double-cta/dist/DoubleCta.svelte.js)

Instead of going this way, I preferred adding manually the missing dependencies to the workspace using Astro. It's a hack, but one I prefer living with than using shamefully-hoist globally as it would cancel out the advantages of the non-flat node-modules.

How fast is it

I know, that was the whole point of us trying out pnpm - let's see how fast it is!

So, when the cache is hit, we get down to 24 seconds:

And when the cache cannot be used, we get down to a whopping 53 seconds:

Summarizing the results:

	Without cache	With cache
yarn 2 (without dedupe)	6min 31s	1min 11s
yarn 3 (without dedupe)	4min 50s	57s
yarn 3	4min 1s	50s
yarn 3 (optimized)	1min 10	45s
pnpm	58s	24s

Honestly, I'm particularly impressed about the results when there's no cache.
I would have expected network to be the bottleneck for both yarn or pnpm in that case, but somehow pnpm still really shines there, while also being faster (at least for us) when the cache is used too!

Now I'm happy - the CI is snappy, at least way snappier than it was, and our local install times also benefitted from it. Thank you pnpm!

Vite in the browser

🎙 m4dz — Wed, 19 Jan 2022 17:39:06 +0000

TL;DR

We made browser-vite - a patched version of Vite running in the browser with Workers.

How it works - in a nutshell

A Service Worker: replaces Vite's HTTP server. Capturing the HTTP calls of an embedded iframe from example.
A Web Worker: Run browser-vite to process off the main thread.
Calls to the file system are replaced by an in-memory file system.
Import of files with special extensions (.ts, .tsx, .scss...) are transformed.

The challenges

No real File System

Vite does a lot with files. The files of the project but also config files, watchers, and globs. These are difficult to implement in the browser with a shimmed in-memory FS. We removed watchers, globs, and config file calls to limit the complexity and the surface API.

The project files stay in the in-memory FS that browser-vite and vite plugins can access normally.

No "node_modules"

Vite relies on the presence of node_modules to resolve dependencies. And it bundles them in a Dependencing Pre-Bundling optimization at startup.

We didn't want to run a node_modules folder in the browser's memory because we think it's just too much data to download and store into the browser's memory. So we carefully stripped out node resolvers and Dependencing Pre-Bundling from Vite.

Users of browser-vite have to create a Vite plugin to resolve bare module imports.

Our products: Backlight.dev, Components.studio and WebComponents.dev, are running a server-side bundler optimizer for the past 2 years now. We created a Vite plugin for browser-vite to resolve node dependencies automatically. As of the date of this post, this server-side bundler is not open-sourced.

Regex "lookbehind"

Some regexs in Vite are using lookbehind. This works great locally when executed by Node.js, but it's not supported in Safari.

So we rewrote the regexs for more browser compatibility.

Hot Module Reload (HMR)

Vite uses WebSockets to communicate code changes from the server (node) to the client (browser).

In browser-vite, the server is the ServiceWorker + Vite worker and the client is the iframe. So we changed the communication from WebSockets to a post message to the iframe.

For this, the client side code of Vite in iframe has been replaced by a special browser version handling messages outside of WebSockets.

How to use it

As of the time of this writing, it's not a plug and play process. There is a lot to figure out by reading Vite's internal processing in order to use browser-vite.

Note: This post may become obsolete over time, so make sure you check
browser-vite's README for always up to date information on browser-vite's usage.

Installation

Install the browser-vite npm package.

$ npm install --save browser-vite

$ npm install --save vite@npm:browser-vite

To channel "vite" imports to "browser-vite".

iframe - window to browser-vite

You need an iframe that will show the pages served internally by browser-vite.

Service Worker - the in-browser web server

The Service Worker will capture certain URLs requests coming from the iframe.

Here is an example using workbox.

workbox.routing.registerRoute(
  /^https?:\/\/HOST/BASE_URL\/(\/.*)$/,
  async ({
    request,
    params,
    url,
  }: import('workbox-routing/types/RouteHandler').RouteHandlerCallbackContext): Promise<Response> => {
    const req = request?.url || url.toString();
    const [pathname] = params as string[];
    // send the request to vite worker
    const response = await postToViteWorker(pathname)
    return response;
  }
);

Mostly posting a message to the "Vite Worker" using postMessage or broadcast-channel.

Vite Worker - processing request

The Vite Worker is a Web Worker that will process requests captured by the Service Worker.

Example of creating a Vite Server:

import {
  transformWithEsbuild,
  ModuleGraph,
  transformRequest,
  createPluginContainer,
  createDevHtmlTransformFn,
  resolveConfig,
  generateCodeFrame,
  ssrTransform,
  ssrLoadModule,
  ViteDevServer,
  PluginOption
} from 'browser-vite';

export async function createServer(
  const config = await resolveConfig(
    {
      plugins: [
        // virtual plugin to provide vite client/env special entries (see below)
        viteClientPlugin,
        // virtual plugin to resolve NPM dependencies, e.g. using unpkg, skypack or another provider (browser-vite only handles project files)
        nodeResolvePlugin,
        // add vite plugins you need here (e.g. vue, react, astro ...)
      ]
      base: BASE_URL, // as hooked in service worker
      // not really used, but needs to be defined to enable dep optimizations
      cacheDir: 'browser',
      root: VFS_ROOT,
      // any other configuration (e.g. resolve alias)
    },
    'serve'
  );
  const plugins = config.plugins;
  const pluginContainer = await createPluginContainer(config);
  const moduleGraph = new ModuleGraph((url) => pluginContainer.resolveId(url));

  const watcher: any = {
    on(what: string, cb: any) {
      return watcher;
    },
    add() {},
  };
  const server: ViteDevServer = {
    config,
    pluginContainer,
    moduleGraph,
    transformWithEsbuild,
    transformRequest(url, options) {
      return transformRequest(url, server, options);
    },
    ssrTransform,
    printUrls() {},
    _globImporters: {},
    ws: {
      send(data) {
        // send HMR data to vite client in iframe however you want (post/broadcast-channel ...)
      },
      async close() {},
      on() {},
      off() {},
    },
    watcher,
    async ssrLoadModule(url) {
      return ssrLoadModule(url, server, loadModule);
    },
    ssrFixStacktrace() {},
    async close() {},
    async restart() {},
    _optimizeDepsMetadata: null,
    _isRunningOptimizer: false,
    _ssrExternals: [],
    _restartPromise: null,
    _forceOptimizeOnRestart: false,
    _pendingRequests: new Map(),
  };

  server.transformIndexHtml = createDevHtmlTransformFn(server);

  // apply server configuration hooks from plugins
  const postHooks: ((() => void) | void)[] = [];
  for (const plugin of plugins) {
    if (plugin.configureServer) {
      postHooks.push(await plugin.configureServer(server));
    }
  }

  // run post config hooks
  // This is applied before the html middleware so that user middleware can
  // serve custom content instead of index.html.
  postHooks.forEach((fn) => fn && fn());

  await pluginContainer.buildStart({});
  await runOptimize(server);

  return server;
}

Pseudo code to process requests via browser-vite

import {
  transformRequest,
  isCSSRequest,
  isDirectCSSRequest,
  injectQuery,
  removeImportQuery,
  unwrapId,
  handleFileAddUnlink,
  handleHMRUpdate,
} from 'vite/dist/browser';

...

async (req) => {
  let { url, accept } = req
  const html = accept?.includes('text/html');
  // strip ?import
  url = removeImportQuery(url);
  // Strip valid id prefix. This is prepended to resolved Ids that are
  // not valid browser import specifiers by the importAnalysis plugin.
  url = unwrapId(url);
  // for CSS, we need to differentiate between normal CSS requests and
  // imports
  if (isCSSRequest(url) && accept?.includes('text/css')) {
    url = injectQuery(url, 'direct');
  }
  let path: string | undefined = url;
  try {
    let code;
    path = url.slice(1);
    if (html) {
      code = await server.transformIndexHtml(`/${path}`, fs.readFileSync(path,'utf8'));
    } else {
      const ret = await transformRequest(url, server, { html });
      code = ret?.code;
    }
    // Return code reponse
  } catch (err: any) {
    // Return error response
  }
}

Check Vite's internal middlewares for more details.

How does it compare to Stackblitz WebContainers

"WebContainers:
Run Node.js natively in your browser"

Stackblitz's WebContainers can also run Vite in the browser. You can elegantly go to vite.new to have a working environment.

We are not experts in WebContainers but, in a nutshell, where browser-vite shims the FS and the HTTPS server at the Vite level, WebContainers shims the FS and a lot of other things at the Node.js level, and Vite runs on it with a few additional changes.

It goes as far as storing a node_modules in the WebContainer, in the browser. But it doesn't run npm or yarn directly because it would take too much space (I guess). They aliased these commands to Turbo - their package manager.

WebContainers can run other frameworks too, like Remix, SvelteKit or Astro.

It's magical ✨ It's mind-blowing 🤯 We have massive respect for what the Stackblitz team has built here.

One downside of WebContainers is that it can only run on Chrome today but will probably run on Firefox soon. browser-vite works on Chrome, Firefox and Safari today.

In a nutshell, WebContainers operates at a lower level of abstraction to run Vite in the browser. browser-vite operates at a higher level of abstraction, very close to Vite itself.

Metaphorically, for the retro-gamers out there, browser-vite is a little bit like UltraHLE 🕹️😊

(*) gametechwiki.com: High/Low level emulation

What's next?

browser-vite is at the heart of our solutions. We are progressively rolling it out to all our products:

Backlight.dev
Components.studio
WebComponents.dev
Replic.dev (New app coming up very soon!)

Going forward, we will continue to invest in browser-vite and report back upstream. Last month, we also announced that we sponsored Vite via Evan You and Patak to support this wonderful project.

Want to know more?

GitHub Repository: browser-vite
Join our Discord server, we have a #browser-vite channel going on 🤗

How ESLint Can Enforce Your Design System Best Practices

🎙 m4dz — Wed, 19 Jan 2022 17:34:31 +0000

If you're creating a design system component library for your company or the open-source community, there's a good chance that you have strong opinions on how end-users should consume your design system.

To ensure that your design system is used in its intended way, and to reduce the number of possible bugs, you might want your users to adhere to your best practices. The following are two examples of possible best practices:

Avoiding inline styles in your elements
Ensuring that tooltips don't contain interactive content.

If you're the only person designing, developing, and consuming your design system, then you can sleep comfortably knowing that your design system is being used exactly as intended.

Chances are you're not the only person developing the design system and you'll certainly not be present when someone consumes it. How can you feel sure that everyone abides by your design system's best practices? You could cross your fingers and trust that your end-users read the documentation, heed your warnings, and never fail to abide by your rules.

Unfortunately, this is often not the case and it's very easy to miss warnings or misunderstand how to properly use a tool. I've been there!

Fortunately, a great way to encourage your consumers to follow your best practices is through the use of ESLint, a static analysis tool to find problems in your code.

By default, ESLint ships with a handful of general best practices, called rules and will display red squigglys in your IDE if the rules have been violated. Some of these rules include:

No duplicate keys in objects
No unreachable code
No unused variables

However, the rules you enable in your project don't need to come directly from ESLint. Popular libraries like Cypress, Lodash, and React have ESLint configurations that anyone can use in their own projects to ensure users adhere to best practices. If you're an intrepid explorer of the JavaScript language you can go a step further and create custom rules specific to your design system which you can export for other people to use in their projects. That's exactly what we'll do in these articles.

In this article, we'll spend a bit of time understanding how tools like ESLint parse JavaScript down into a data structure called an abstract syntax tree (AST). We'll then touch on how ESLint rules work and how to parse our Lit templates into HTML. Finally we'll start creating our rules. We'll even use ESLint's built-in testing tool to make sure our rules work under a variety of conditions.

The pre-requisite for this article is some JavaScript + HTML knowledge. A little experience using ESLint and Lit may come in handy but isn't necessary.

What’s an Abstract Syntax Tree?

For those, like me, who haven't gotten their hands dirty with compilers before, conceptualising how the human-readable language we write in our IDE gets understood (and transformed) by tools like Webpack, Prettier, and Babel can feel like magic.

Under the hood, when a tool like ESLint wants to start performing actions against your JavaScript it parses your code. Parsing is the process of taking the JavaScript you've written and turning it into a tree representation of the code, an abstract syntax tree (AST).

This process of parsing is split into two parts, tokenization and tree construction.

Tokenization takes the code and splits it into things called tokens which describe isolated parts of the syntax.

Tokens for a JavaScript program like:

const helloWorld = 'hello world';

will look something like this:

[
  { type: 'IdentifierName', value: 'const' },
  { type: 'WhiteSpace', value: ' ' },
  { type: 'IdentifierName', value: 'helloWorld' },
  { type: 'WhiteSpace', value: ' ' },
  { type: 'Punctuator', value: '=' },
  { type: 'WhiteSpace', value: ' ' },
  { type: 'StringLiteral', value: "'hello world'", closed: true },
];

JS Tokens

I used js-tokens as a quick way of tokenizing my JS for this example but we won't be dealing with tokenization directly ourselves in this article.

The second step in the parsing process is tree construction, which reformats the tokens into an AST. The AST describes each part of the syntax and its relationship to the others.

We can visualise this relationship by parsing the following JavaScript statement:

const component = html`<h1>Creating custom ESLint rules</h1>`;

It would get transformed into an AST, with the following structure:

Tools like Babel and Prettier turn your written JavaScript into an AST to analyse and transform the code we've written. Babel uses the AST to transpile our code into a browser-friendly version of JavaScript, while Prettier uses the AST to reformat your code.

Getting Curious with the AST Explorer

To really explore what an AST looks like, have a play with the AST explorer. Get familiar with the AST explorer as we're gonna be using it loads later in the article.

Write a simple statement, like the following:

const helloWorld = 'hello world';

You'll see that the top level of the tree describes the entire program and we can look into the body array to see the individual constituents of our above statement represented in the AST.

If you hover over the VariableDeclaration you can see that the entire statement on the left gets highlighted. If we go a level deeper into the declarations array you'll see an additional node VariableDeclarator. If we keep going we'll eventually reach rock bottom. In the case of our hello world statement, it's with the variable's Identifier and the variable's Literal value.

Let's revisit our component from earlier:

const component = html`<h1>Creating custom ESLint rules</h1>`;

If you step through the tree in the AST explorer you can see that the structure matches our image from earlier. Pay particular attention to the TaggedTemplateExpression node and the TemplateLiteral node. These are the ones that will come in handy when we write our ESLint rules.

Our call to the html function is an expression, but it looks a little different from other function definitions. Let's see how the AST differs with an expression like the following:

function heyThere() {
  return 'hey';
}

heyThere();

If we hover over the heyThere() ExpressionStatement, we see that the properties match our html ExpressionStatement. The main difference is that the value in the expression property looks different. The expression this time is a CallExpression, which has a set of properties different to that of our TaggedTemplateExpression.

If we look back at our TaggedTemplateExpression, we can see that we have properties like tag and quasi.

The tag gives us some details about the function name. Which in this case is html.

This means when writing our ESlint rule we'll be able to do something like this:

// Some ESLint psuedo-code
function createRule() {
  return {
    TaggedTemplateExpression(node) {
      const isLitExpression = node.tag.name === 'html';

      if (isLitExpression) {
        // rest of the rule
      }

      // do nothing
    },
  };
}

Finally, if you look into the TaggedTemplateExpression object, you'll see a property named quasi. This property contains our two noteworthy properties expressions and quasis. Take the following expression:

The blue underlines, the first and third respectively, will live in the quasis array and they'll be in the order that they're written in your template literal.

The green underline, the second one, will live in the expressions array, and provides a reference to the name of the variable. Like the quasis, the items in the array are in the order that they're defined. This makes it very easy to reconcile your template literal later on.

Quasis

When accessing the value in your quasis, you'll see the string available as either raw or cooked. These values determine whether escape sequences are ignored or interpreted. Axel Rauschmayer covers this in a little more detail in this article.

Here's a question for you, what happens if the first character of our template literal is an expression? How is this represented in our AST? Try the following snippet in the AST explorer:

const helloWorld = `${name}, how you doin'?`;

Take a little more time exploring quasis and expressions if they still feel unfamiliar to you.

Fortunately, we won't need to directly deal with the parsing process when writing our ESLint rules. We've covered a lot of ground because having a high-level understanding of how the tooling works, makes for a much more intuitive development experience later on.

Super Tiny Compiler

If you're interested in learning a little more about the whole compilation process, the Super Tiny Compiler is a really fun way of building your own JavaScript compiler using only a couple of hundred lines of code.

How Do Eslint Rules Work?

The Visitor Pattern

Fortunately, we don't need to make any transformation when writing ESLint rules and instead we write our checks against specific node types in our code. These nodes are slices from our code's AST.

Once ESLint has parsed your code into an AST, it then traverses your tree, visiting each node along the way. For those familiar with programming design patterns, you might recognise this pattern as the visitor pattern.

The visitor pattern is a way of running some new logic against an object without modifying the object. ESLint uses the visitor pattern to separate the code used to run checks against your code from the AST.

Let's take a look at the visitor pattern in action.

You can see, that I've implemented the visitor using 3 blocks of code:

ast.js: The AST for const name = 'andrico'
traverser.js: An algorithm that traverses the nodes of our AST.
visitors.js: An object of methods where a given method fires once the traverser reaches its corresponding node. In our case, when the traverser reaches a VariableDeclarator node, it fires off our visitor function.

Let's break down the traverser a little more:

We begin in index.js by creating an instance of our Traverser class and passing through our AST and our visitors. Under the hood, our Traverser class stores our AST and visitors as instance variables for us to use later.

We then invoke the instance's traverse method. If you move to the traverser.js file, you can see that when we invoke traverse 5 things can happen:

The node is null, which will happen when we manually invoke the traverse method without any arguments. When this happens, we kick off the traversal function using the AST we stored during the class's initialisation.
The node has a type of Program, which will happen for the top-level nodes in our AST. When this happens we recursively call the traversal method on the child nodes.
The node has a type that matches a visitor function. When this happens, we fire our visitor function and pass through the node as an argument.
The node has additional declarations, so we continue calling our traversal function on those child declarations.
Our node satisfies none of these conditions, which will cause our traversal method to exit.

In the context of our const name = 'andrico' example, our traversal function will continue making its way through the AST until it reaches the VariableDeclarator, where it will invoke the visitor we defined in visitors.js. In this visitor we check to see if the value is Andrico and if it is, we log a message saying that it's an invalid name (though I kinda like it).

Open the console in the CodeSandbox and see what it outputs. Try changing the check in your visitor and see what happens, if anything.

The good news is that ESLint handles the traversal logic for our JavaScript. The other good news is that we'll need to implement the traversal logic for our parsed HTML. 😄

Open WC's eslint-plugin-lit-a11y

This section was heavily informed by my recent involvement with Open WC's eslint-plugin-lit-a11y and eslint-plugin-lint. If you'd like to learn more about (or try a handful of) ESLint rules focused on web components, these are your go-to repos.

What Does an Eslint Rule Look Like?

Writing an ESLint rule doesn't require anything fancy, it's just a plain ol' JavaScript object. The object's top-level can receive two properties: meta and create.

meta provides the metadata for the rule.

The create property is a function that returns an object of visitors that ESLint calls when it visits each node. This follows the same principle as the snippets in the codesandbox. And much like the demo in our codesandbox, the name of each visitor function is the name of the node that we want to visit.

In fact, we can even repurpose the pseudo-code from earlier, and decorate it with the ESLint specific boilerplate:

module.exports = {
  create: function create() {
    return {
      TaggedTemplateExpression(node) {
        const isLitExpression = node.tag.name === 'html';

        if (isLitExpression) {
          // rest of the rule
        }

        // do nothing
      },
    };
  },
};

The create function also provides a context object, which provides some additional helpers and information about the current rule. The helper we're most concerned with right now is the report() method. We can call report whenever we want an ESLint error to display in the console or the IDE.

Context.report takes an object with a handful of properties, but we're most interested in the following:

message: the description of the problem
node: the AST node related to the problem

Additional information

We can pass through additional information, like the line of code we want to report the error on, but that's outside the scope of this tutorial.

Before continuing, why not think about adjusting the pseudocode above to display an ESLint error when a tagged template is invoked, and the template literal has no content, like this:

const expression = html``;

With a basic understanding of JavaScript's AST, the visitor pattern, and the anatomy of an ESLint rule, the only thing left to cover is how to parse our template string into HTML before we can start creating our rules.

For a more in-depth read into the anatomy of an ESLint rule, there's no better place to look than the official docs.

How Can We Transform Our Templates into HTML?

When using ESLint, we have the luxury of ESLint providing us with our parsed JavaScript AST. And while ESLint can't parse our HTML, we can use a library like [parse5](https://github.com/inikulin/parse5) to parse a valid HTML string into a data structure, not unlike our JavaScript AST.

The AST explorer we've spent so much time exploring even has settings for displaying HTML ASTs.

Since one of our rules is going to prevent us from passing through inline styles, let's see how the following gets represented as an AST:

<div style="display:inline;">Main content</div>

If we dive into the AST and look for our div, we can see we're presented with some useful information. The most notable are:

tagName: Which is the name of the html element. (in this case div).

attrs: Which is an array of attributes, represented as a key-value pair. Our div's attrs property holds a single item. The item has a name of style and a value of display:inline;.

Using this information we can already start seeing how to piece together everything we've learned to create our first lint rule.

Here's how we can parse our JavaScript templates using the parse5 library:

import parse5 from 'parse5';

// We're defining out HTML templates
const htmlString = `<div style="display:inline;">Main content</div>`;

// We're passing through an HTML snippet to parseFragment, which returns our HTML AST
const parsedFragment = parse5.parseFragment(htmlString);

// We access the first child because the top-level contains metadata we don't need right now.
const div = parsedFragment.childNodes[0];

// We check to see if there are any style attributes in our div
const hasStyleAttr = div.attrs.some((attr) => attr.name === 'style');

// If there are, we report an error
if (hasStyleAttr) console.log('FAIL');

Thanks to tools like parse 5 and ESLint, we can offload a lot of the complex processing, and focus on writing the code for our specific rules. This is what we'll start doing from the next article onwards.