Forem: Thomas Klein

Introducing the Node.js package analyzer

Thomas Klein — Wed, 07 Apr 2021 15:51:02 +0000

Introduction

I'm proud to present to you the Node.js package analyzer.

tmkn / packageanalyzer

A framework to introspect Node.js packages

It's a framework designed to easily introspect a Node.js package.

Think of it as eslint for your Node.js package/project.

It's also what powered the data behind npmbomb.

Package Analyzer

While the JS ecosystem is quite mature and diverse, there is no linter for a Node.js project. If you want to do a license compliance check, it's a separate tool. If you wan't to check the health of a dependency, that's a separate tool. If you want to check for unneeded polyfills/deprecated packages, that's yet another tool.

The aim of the package analyzer is to provide the framework and toolset to easily answer questions like above within a single tool.

All while being flexible enough to answer any questions surrounding a Node.js package like:

calculate the number of all dependencies
find newest/oldest dependency
find most included dependency
get weekly downloads
check for a new version
release velocity
etc

That's why the package analyzer is both an API and a CLI

Another reason why I started this project was security.

Personally I think there's to much trust put into adding dependencies.

Any Node.js package that you install can run a postinstall script and people tried to extract credentials via this way for some time now, yet there is no straight forward way to highlight packages with a postinstall script.

I also envision a system that highlights which API's a particular dependency is using. fs, http etc. or highlights differences between version upgrades: A new sub dependency was added? By whom?

Ideally the package analyzer will be able to answer those questions.

Architecture

The package analyzer is written in TypeScript so TypeScript types are a first class citizen.

Package

At the heart of all of it, is the Package class. After you've traversed a package and all of its dependencies you'll get a single instance of the Package class back. Important points of the API are as follows:

interface IPackage<T> {
    //each package parent's can be easily accessed
    parent: T | null;

    //vital information is directly accessible
    name: string;
    version: string;
    fullName: string; //`name@version`

    //dependencies are listed here
    directDependencies: T[];

    //convenience functions to iterate over the dependency tree
    visit: (callback: (dependency: T) => void, includeSelf: boolean, start: T) => void;

    ///easily find dependencies
    getPackagesBy: (filter: (pkg: T) => boolean) => T[];
    getPackagesByName: (name: string, version?: string) => T[];

    //access package.json data
    getData(key: string): unknown;

    //access custom data
    getDecoratorData<E extends IDecoratorStatic<any, []>>(decorators: E): DecoratorType<E>;
}

For more information about the architecture you can checkout the Architecture.md in the GitHub repository.

CLI

If you install the package analyzer globally:

npm install -g @tmkn/packageanalyzer@0.9.4

You will get a pkga command where you can easily introspect packages.

Some of the things it does currently are as follows:

Print Metadata

You can use the analyze option to print metadata.
If you don't provide a version number it will use the latest release.

pkga analyze --package react //use latest version
pkga analyze --package react@16.12.0 //use specific version

Use the --full option to print additional data like oldest/newest package.

pkga analyze --package react --full

If you want to analyze a local project use the --folder option:

pkga analyze --folder path/to/your/package.json

Print Dependency Tree

pkga tree --package react

Print Weekly Downloads

The downloads command will print the weekly downloads for a package for NPM

pkga downloads --package react

Cyclic Dependencies

Use the loops command to print cyclic dependencies in the dependency tree:

pkga loops --package webpack@4.46.0

API

In addition to the CLI, the Package Analyzer also offers an API.
All the commands in the CLI are done via this API as well.

Example

Here's how you can use it to list all dependencies of fastify that come with built int TypeScript support.
Type declarations are either marked via the types or typings field in the package.json, so all we need to do is ask if those fields are set and collect them:

const { Visitor, npmOnline, OraLogger } = require("@tmkn/packageanalyzer");

(async () => {
    try {
        const visitor = new Visitor(["fastify"], npmOnline, new OraLogger());
        const pkg = await visitor.visit();

        const matches = new Set();

        pkg.visit(dep => {
            if (dep.getData("types") || dep.getData("typings"))
                matches.add(dep.fullName);
        }, true);

        console.log("Built in TypeScript support:")
        for (const name of matches)
            console.log(name);
    }
    catch (e) {
        console.log(e);
    }
})();

        const visitor = new Visitor(["fastify"], npmOnline, new OraLogger());

First we need to create a Visitor that traverses the dependency tree.

The 1st argument is a tuple that specifies the package, if you don't provide a version. e.g.["fastify", "3.14.1"] it will default to the latest like here.

The 2nd argument is the Provider. Whenever the Visitor wants data about a package it will ask the Provider. In this case we are asking the NPM registry, but you could also write a Provider that gets the information from the filesystem e.g. node_modules folder.

The 3rd argument is the logger. All output is routed to this logger.

        const pkg = await visitor.visit();

Then we simply call the async visit function from the Visitor to start traversing the dependency tree. We will end up with a single top level Package class.

        pkg.visit(dep => {
            if (dep.getData("types") || dep.getData("typings"))
                matches.add(dep.fullName);
        }, true);

This Package class provides utility functions like visit to iterate over each dependency and the getData method to access the respective package.json. So here we iterate over each dependency and check if the respective package.json contains entries for types or typings. If yes, collect dep.fullName which is a string formatted to packagename@version.

        console.log("Built in TypeScript support:")
        for (const name of matches)
            console.log(name);

Afterwards we just print out our findings, done.

If you want to know more I recommend to check out the Architecture.md in the GitHub repository. But do note that the API is not yet stable and will likely undergo changes as it's a first preview.

What's next

Since I have a lot of ideas where to take this and since it's a first preview I would really appreciate feedback :)

In the beginning I said to think of it as eslint for packages, however defining your own set of checks that you want to run is not yet possible and something that I want work on next.

I'm also thinking about a web interface, that allows to better visually present the results and where you can jump back and forth between packages/reports in an easy manner.

Apart from that, any suggestions are highly welcomed.

For updates you can follow me on Twitter or follow the project on GitHub: https://github.com/tmkn/packageanalyzer

What's new in npmbomb?

Thomas Klein — Thu, 27 Aug 2020 19:39:25 +0000

After the initial release I found some time to do some quality of life changes, among them are:

Visualizing the dependency tree
Refactored search to be future proof
Updated the dataset

What is npmbomb?

https://npmbomb.tmkn.dev/

npmbomb is a little website that I built where you can try and guess the number of total dependencies for popular npm modules to shine a light on the growing transitive dependencies number of npm modules. If you add 1 dependency to your project, that dependency might contain a mountain of other dependencies that you don't know of but now will be part of your project.
It does this by simply following every dependency of the dependencies field in the package.json down to the very last one. As such, the calculated numbers might surprise you.
If you want to know more, you can find an introduction here

Visualizing the dependency tree

There is now a Dependency Tree tab, switching to it will reveal the 🎉 dependency tree.
It will show the transitive dependency count as well as provide a link to npm by clicking on the link icon.

How are the numbers calculated?

The number on the right on each line is the transitive dependency count or total dependency count.
In case of React that number is 8.
Here's how it is calculated:

3 direct dependencies of React itself + all transitive dependencies of loose-envify (1) and prop-types (4). object-assign doesn't itself define any other dependencies so doesn't contribute to the transitive dependencies count.

To keep things snappy, the tree uses react-virtualized.
Apart from that the tree component is custom built.

While most trees would work just fine without react-virtualized since they are not that deep and big, Jest for example made the browser sweat.

Since the react-virtualized module was already in use on the search page, it was also used for the tree rendering to solve potential rendering bottlenecks.

Coming back to Jest and its humongous dependency tree, the very first working version resulted in a 20MB JSON payload just for the tree data.
The straight forward format looked like this:

interface IDependencyTree {
    name: string;
    version: string;
    transitiveCount: number;
    loop: boolean;
    dependencies: IDependencyTree[];
}

Even though a dependency tree like Jest is not the norm, clearly that's too much. Even gzipped it still was 5MB.
When I reduced the length of the keys to single chars it was still 16MB.

So I changed it to provide a lookup table instead and only reference numbers(id) for the actual nested format:

The tree value looks like this:

export interface IDependencyTreeConfig {
    //lookup
    data: ITreeData[];
    //nested tree structure
    tree: IDependencyTreeStructure;
}

export interface ITreeData {
    id: number;
    name: string;
    version: string;
    count: number;
}

export interface IDependencyTreeStructure {
    id: number;
    dependencies?: IDependencyTreeStructure[];
}

I could potentially reduce it further by reducing the length of the keys here as well but already with this approach the payload went down to 7MB, while compressed with Brotli it's now around 47kb. For an outlier like Jest, I think it's acceptable.

Refactored search to be future proof

The long term goal of npmbomb is to have data for any(most) npm modules.
As of now the dataset is limited to a handful of the most popular modules.
As such the architecture of the search was tailored for this limited data set so it wouldn't scale with growing data.

To remedy this, the search now uses react-virtualized to display its results (just like the dependency tree), supporting any number of search results.

With the old search architecture everything was in memory so it wasn't prone to ajax race conditions and such, so it's implementation was really simple but again with a growing dataset this approach wouldn't be feasible any more. Now you would need to take care of Ajax race conditions, only fire the search request after the user stopped typing etc, dismiss the ongoing ajax request if the user starts typing again, etc, to handle all of this cleanly the search architecture is now powered by RxJs.

Here's the new search:
In fact, it looks like the old one, all the changes are behind the scenes ;)

Updated the dataset

The original data was based on July 2019 so I thought it was fitting to provide an update after a year, so now the data is based on July 2020. While this is already "outdated", the goal of npmbomb is not to provide up to date data. In fact, this would be very challenging to provide, as any new version of any module could alter any existing dependency tree. It's highly volatile. Instead the goal of npmbomb is to provide a ballpark number from mostly up to date data.

Interestingly the npm dataset grew within this 1 year from 23.9GB to 42.2GB.
And the total number of modules grew from 1 007 928 to 1 332 134. So within this 1 year npm saw 324 206 brand new modules.

What's next?

Things that I have on my mind for npmbomb:

Increase dataset

One of the immediate next steps is to increase the dataset aka looking at options to host the data. The dataset is currently hosted together with the webapp via netlify. While an awesome service, it eats into the free tier usage, I would rather like to move the data hosting to somewhere else and only host the web app on netlify. I'm open for suggestions.

Dependency tree improvements

Breadcrumbs for tree view

Show the path upon hovering over tree nodes, as with large trees you can get easily lost.

Filtering

Allow the user to search for specific modules.
Highlight modules where they appear in the tree

More information

Display more information for a module, like

maintainer
release date
show source code
etc.

More E2E testing

While there is E2E testing, it's not on a level where I would like it to be. The bulk of the testing is done via unit tests.

A11Y

Since the project is past the prototype stage, with a growing audience it makes sense to invest into accessibility.

Housekeeping

With the tree view added and other small tweaks here and there, it is now a good opportunity to take a step back and do some housekeeping.

You see there are a lot of ideas and things that I want to do. This is also a friendly reminder that npmbomb is totally Open Source.

Go checkout the new tree view: https://npmbomb.tmkn.dev/
Feedback highly appreciated, whether on Twitter or GitHub or here 🙃

Building npmbomb

Thomas Klein — Sun, 19 Jan 2020 16:08:53 +0000

This is a technical writeup of how I built npmbomb, which technologies I chose and why. The problems encountered along the way and how I solved them.

npmbomb is a purely client side app where you can guess the number of dependencies for popular NPM packages 😈
I took the packages from the first 3 pages here and calculated the dependency tree for each of them.

How

For the past few months, after work, I've been working on and off on a NodeJS package analyzer. I wanted to know what is the oldest dependency in my dependency tree, what are the licenses, are there new versions for my dependencies etc. You just point it to a package.json or hand it a package name and it will resolve the dependencies and calculate all sorts of data. Once the analyzer is more polished I'll write more in-depth about it. At one point parts of it used C# as the NodeJS stream API was unstable but that's for another time.

Ever since the addition of React Hooks, I always wanted to create a little web app that exclusively uses them just to see how it feels and Emotion CSS for the styling. I just needed a worthwhile idea, ...this was it.

Tech Stack

In a nutshell it's a purely client side app, leveraging React, React Router, Emotion.

Normally I opt for MobX and CSS Modules. While I like MobX, I wanted to see if React Context in conjunction with React Hooks would be a suitable replacement in this case.

As for CSS Modules I never really quite liked them because I can't easily access CSS properties in my code and vice versa like animation delays etc. Also it's just too much magic, while it works, "importing" CSS files in JavaScript just feels wrong, it's a clever hack. Also using them with TypeScript requires some extra effort. By using Emotion I hoped to improve on all of these pain points.

In its infancy I used NextJS because I wanted to try it out as they added TypeScript support but dropped it because adding Emotion was giving me some trouble and overall I could just move faster by rolling my own setup instead. While people hate configuring webpack I don't find it that bad as everything I need can be found in their documentation. For NextJS I had to look at stackoverflow etc. Never-mind I will revisit NextJS in the future, it just wasn't a great fit at the time.

TypeScript

All the code is written in TypeScript with the strict option, for me the most invaluable tool for writing maintainable JavaScript. You could replace any part of the tech stack and I'm cool with it except TypeScript.

I still remember one time at work where I had to replace instances of TextNode with HTMLNode. It took me several commits to catch and fix everything. Later I converted the whole code base to TypeScript(just after they announced support for JS files) and even though the compiler complained A LOT and thus I had to touch much more code than when I did the TextNode to HTMLNode conversion, the dev experience was just so much better.

When the compiler said 0 errors, he really meant it. Ever since then TypeScript became invaluable, if TypeScript complains you listen.

React

As I said I wanted to build the app solely with Function Components and Hooks/Context.
Overall I found it quite pleasing, however defining Contexts is a little awkward in conjunction with TypeScript. I didn't want to make the actual Context members optional so I had to initialise them with temporary values...

interface IAppContext {
    appState: IAppState;
    setAppState: (state: IAppState) => void;
}

export const AppContext = React.createContext<IAppContext>({
    appState: {
        inGameMode: false,
        guesses: [],
        remaining: [],
        packages: []
    },
    setAppState: () => {}
});

Here all the values are just placeholders to satisfy the TypeScript compiler, setAppState will essentially do nothing, if I later down the road forget to provide the actual data, it will use my placeholder data with no real warning.

export const App: React.FC = () => {
    const [appState, setAppState] = useState<IAppState>({
        inGameMode: false,
        guesses: [],
        remaining: [],
        packages: []
    });

    const context: IAppContext = {
        appState,
        setAppState
    };

    return (
        <AppContext.Provider value={context}>
        //have to make sure I use AppContext.Provider to provide
        //the actual data for child components
        </AppContext.Provider>
    );
};

I could have simply defined AppContext like this:

export const AppContext = React.createContext<IAppContext | undefined>(undefined);

Which would allow me to initialise AppContext with undefined and be done with it, but then every-time I would like to consume it I would have to do a check for undefined which leads to very repetitive code.

    const appContext = useContext(AppContext);

    if(appContext) {
        //do something with appContext.appState
    }

By not having it undefined I can consume it much more elegantly, but I have to initialise it with temp data first like shown above.

    const { setAppState } = useContext(AppContext);

Other than that I like the flexibility that Contexts provide, no more passing on props to child components, definitely an improvement.
For this use case it could replace external state management completely.

React Router

Just like React I used Hooks for React Router. In conjunction with TypeScript this also simplified typing. Previously you had to setup much more boilerplate code to access router related props in your components like:

interface IProps extends RouteComponentProps<{}> {
  package: string
}

With hooks it just boils down to using the appropriate useHistory or useParams hooks:

import { useParams, useHistory } from "react-router-dom";

    //in your function component...
    const history = useHistory();
    const { pkgName } = useParams<{ pkgName: string; }>();

Emotion

In short it basically solved all the pain points that I had with CSS Modules and it will be my preferred CSS solution going forward.

I like that with the spread operator it makes for very concise code.
The only downside was that since I was using it directly with TypeScript
and not with Babel, I had to place a pragma comment at the top of each file where I wanted to use Emotion:

/** @jsx jsx */
import { jsx, css } from "@emotion/core";

This also meant that I couldn't use the shorthand notation <></> anywhere in the code and had to fall back to <React.Fragment></React.Fragment>. While a bummer it wasn't that much of a bummer that I would introduce Babel into the mix.

Another surprise was that it auto prefixed the CSS out of the box, without me doing anything:

const titleStyle = css({
    "@media (min-width: 320px)": {
        flex: 1
    }
});

turned into

@media (min-width: 320px)
.css-bof962-Header {
    -webkit-flex: 1;
    -ms-flex: 1;
    flex: 1;
}

Quite nice!

Sentry

Sentry monitors your application for runtime errors. If there's an error, Sentry collects it and sends it to their servers where you can inspect them and take further actions:

Here some error happened in my app, and I get to see details about the browser environment and also a stacktrace etc.

I also use it to propose me new packages. As said I only analyzed the most popular NPM packages and I'm using Sentrys functionality to gather feedback too, not just runtime errors.

I've used datadog in the past, which offers similar functionality but the onboarding was much smoother with Sentry, going forward this will be my preferred choice for collecting runtime errors.

Testing

No application is complete without testing, here are the tools that I used:

Unit Tests

For unit tests I used React Testing Library. Not much needs to be said, I like the API that they are offering to test various parts of your app.

E2E Tests

For E2E tests I used cypress.

Visual Regression Testing

For a per pixel check of the app I use Percy. It's a cloud service that compares screenshots on a per pixel basis. I instrument cypress to take screenshots at certain points in the app and with different breakpoints and Percy then shows me the differences, if there are any.

    //take a screenshot of the homepage with
    //3 different breakpoints
    cy.visit("http://localhost:8080");
    cy.percySnapshot("home page", { widths: [320, 768, 1200] });

I can then approve it or go back into the code and fix it.

What is also nice is that it's integrated into GitHub, it checks every pull request for changes automatically

Code Coverage

Code Coverage is visualized via Codecov, I like their UI better than coveralls, it's less cluttered.
It's also integrated into GitHub and run on every pull request:

Continuous Integration

To make sure the tests are working and to upload the code coverage and percy results I'm using Azure Pipelines.
I specified to run the tests on Linux, Mac and Windows even though there is no explicit platform specific code since it's a web app I do it because I use all of these 3 operating systems to develop.
It's also integrated into GitHub, every push to master/pull request triggers the pipelines:

Setting it up was fairly straight forward, the biggest problem I had was to tell it to only upload the code coverage report on the Linux run. It took me several tries to get it correct.

Deployment

The whole app is hosted on Netlify. It's the first time using the platform and it was very easy to integrate. Each push to master deploys it automatically.
You just have to specify the build command and which folder contains the compiled output, it couldn't be easier!

I only had to add a netlify.toml to specify redirect rules so urls like https://npmbomb.tmkn.dev/package/jest@24.8.0 resolve correctly since I'm using client side routing.

[[redirects]]
  from = "/*"
  to = "/index.html"
  status = 200
  headers = {X-From = "Netlify"}

I also enabled Netlify to run on pull requests:

I just need to click the link and get a live version of the pull request, super nice!

That's basically it about the tech stack and how I wrote npmbomb.
All the tools that I introduced can be used for free, I run all of them on the free tier and they are more than enough for my use case and make my life as a developer much easier.

Here I instantly see that the visual design changed thanks to Percy, but everything else like tests ran through and Netlify was able to successfully deploy the pr. So once I fix the issue reported by Percy I can confidently merge this back into master (never-mind the fact that codecov couldn't find a report here).

Whats next

Greater test coverage, ~~just kidding, we all know it won't happen. 😐~~ I actually did it:

By using the app (at least for me) I found it quite addictive, currently you can guess only 4 packages before it starts over again. I think an endless mode would be cool.

I also would like to show the dependency tree visually, so that you see which dependencies it pulls in from where etc.

Thanks for making it to the end and have fun guessing NPM dependencies!
All the code can be found on GitHub.

JavaScript and bit masks

Thomas Klein — Sun, 24 Nov 2019 16:08:04 +0000

Even though you can use bit masks in JavaScript they are not widely (if at all) known. If you don't know what bit masks are, and want to learn a little bit about programming history, this post is for you:

Introduction

The primary use case for bit masks is to save memory. They come from a time where memory was scarce, long before the internet.
Programmers back then stored multiple values into a single value. How did they do this? With bit masks.

They took advantage that the computer uses bits. 1 bit just stores 0 or 1. You might as well think of it as set or unset, true or false or simply a flag.

If you have 2 bits you can store them in 4 ways, this could be 4 different flags:

Bit2	Bit1
0	0
0	1
1	0
1	1

If we have 3 bits we can store them in 8 different ways:

Bit3	Bit2	Bit1
0	0	0
0	0	1
0	1	0
0	1	1
1	0	0
1	0	1
1	1	0
1	1	1

They follow a power of two:
2bit -> 2^2 = 4
3bit -> 2^3 = 8
32bit -> 2^32 = 4 294 967 296
and so on.

Unfortunately if you wanted to have a flag in your code, a true-false value, early computer systems didn't allow you to just use 1 bit to specify that flag. For the sake of this introduction the computer always allocated 4bits for every kind of variable.

Since you only want to store a flag that theoretically only needs 1bit the remaining 3bits are going to waste.

If you add yet another flag, you will again allocate 4bits where 3bits are wasted.
So now you wasted 6bits because you created 2 separate flags.
This was a huge deal back in the day.

The solution to this problem was to store the 2 flags(2bits) not as separate variables but inside a single (4bit) variable. This way we only waste 2bits out of 4bits in this concrete example.

Bit Operators

To achieve this you have to use bit operators. I'm sure you've already seen || or && in action in some JavaScript code so you know they relate to logical or and logical and.

But did you know there is also this syntax?:

let or = var1 | var2;
let and = var1 & var2;

They operate on a bit level and are the whole secret to bit masks.

& (AND)

Given 2 bit sequences they produce the following output:

0 1 0 1 1
1 1 0 1 0
..............
0 1 0 1 0

If both are 1, the output will also be 1.
If only one of them is 1 it will be set to 0.

| (OR)

Given 2 bit sequences they produce the following output:

0 1 0 1 1
1 1 0 1 0
..............
1 1 0 1 1

If at least one of them is 1, the output will also be set to 1.

This is all you need to know to work with bit masks.
Pretty similar to logical AND and logical OR.
To enter an if clause with logical AND (&&), both conditions need to be true.
To enter an if clause with logical OR (||), only 1 condition needs to be true.

Example

Say we want to have a simple permission system that differentiates between a user and admin.

We can specify the bit sequences as follows:

Bit2	Bit1	Permission
0	0	Unused
0	1	User
1	0	Admin

The person is a user, if the bit sequence is 01.
The person has admin rights, if the bit sequence is 10.

With this info we can create our bit masks in JavaScript, we can specify the binary value right away with binary literals:

const USER = 0b01;
const ADMIN = 0b10;

Lets create a permission that specifies that someone is a user and admin:

let userPermission = USER | ADMIN;

Notice the | operator to set values.
This is the "store 2bits in 1 variable" that I talked about before.
Now userPermission contains info whether USER is set and whether ADMIN is set.

But how do we check the permission? With the & operator:

if((userPermission & ADMIN) === ADMIN) {
  //confirmed admin
}

How did this work?
Let's look at the & operation:

1 1 //userPermission
1 0 //ADMIN
.....
1 0

We get the bit sequence 10, then we check if this sequence matches ADMIN and yes it does, thus we are confirmed admin :)

To check for a user we use the same pattern:

if((userPermission & USER) === USER) {
  //confirmed user
}

That's how programmers back in the day tried to save memory, while it's not that useful on the web today due to the way JavaScript works and overall technological advancements it's always nice to know the history of things.

If you're feeling intrigued, here's more info about bit operators (in JavaScript): https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Bitwise_Operators
I only covered the very top, there are more operators that allow more use cases but that would exceed the scope for this post.

A better puppeteer.evaluate

Thomas Klein — Mon, 28 Oct 2019 21:41:56 +0000

Execute JavaScript with puppeteer

To execute JavaScript code on your puppeteer instance, you can use the provided evaluate method. e.g.

const title = await page.evaluate(() => document.title);

This returns the document title from the puppeteer instance back to the node process.

But the way this works is actually quite limited and error prone.
This already errors at runtime:

const selector = `p`;
const paragrapahs = await page.evaluate(() => {
    const elements = document.body.querySelectorAll(selector); //runtime error
});

(╯°□°)╯︵ ┻━┻

Why?

Because the callback that you provide in the evaluate method gets serialized.
Meaning by the time it arrives at the browser it will have lost all information about closures/imports etc. What a bummer.

If you're like me and want to separate your evaluate code to an external file, use node modules etc, you're out of luck. It just doesn't work.

Worse, you will get no indication that this will not work, the IDE says all good, go ahead run it, only to be hit with a runtime error :(

To be fair though, the above fairly simple example can be made to work, as you can also specify arguments:

const selector = `p`;
const paragrapahs = await page.evaluate(selector => {
    const elements = document.body.querySelectorAll(selector);
}, selector /* <- pass selector as argument */);

Still, this is very cumbersome to work with, as you just expect closures to be available, after all, that's how JavaScript works.

A better puppeteer evaluate

For that reason I published a little library(my first) that lets me do just that:

Aptly named puppeteer-evaluate2, it allows me to write my puppeteer callback code like any other JavaScript code, where closures/imports are available.

It's signature looks like this:

export function evaluate2<T = any>(page: puppeteer.Page, jsPath: string): Promise<T>

You just pass in your puppeteer page object as the 1st parameter and the path to your JavaScript file as the 2nd parameter.

The library then does the rest, it will create a JS bundle, taking the 2nd parameter as the entry point. The finished bundle is then passed on to the actual page.evaluate method. Closures and imports are then available.

Here's an example importing lodash:

//code.js
export default function() {
    const chunk = require("lodash/chunk");

    return chunk([1, 2, 3, 4], 2);
}

let response = await evaluate2(page, `./code.js`);
console.log(response); //[[1, 2], [3, 4]]

voila 🎉

The only requisite is that your entry file must export a default function!
Thanks to the TypeScript API there are checks in place that tell you if you're missing a default function export 💪

Creating a bundle, entry file,.. this sounds a lot like webpack, and you're right! It is actually using the webpack API under the hood to create the bundle, but in memory and on the fly.

Final

If you're asking why did I import lodash with require and not via import? Because I still consider it more of a POC, hence it only has a beta version right now, as I also want to have TypeScript support. But I wanted to share it early to get some feedback, so let me know what you think :)