Forem: Sergey Melyukov

Brand new toolkit to analyze and validate your webpack bundle

Sergey Melyukov — Tue, 14 Sep 2021 17:40:31 +0000

Hi! 🖐
In this post I would like to share Statoscope with you.

Statoscope is a modern tooklit for analyzing and validating your bundle.
You can try Statoscope by simply dropping the stats of your build into https://statoscope.tech or using a webpack plugin that will generate a UI-based report about your build.

This report will show you full information about your build, NPM packages duplicates, and even the download speed of your bundle to a client browser. With this UI report, you can make your own requests to stats and generate your own reports about your bundle. This is very similar to python notebook, but for webpack stats and is possible thanks to a special query language.

Also, Statoscope is able to validate your build on CI, there is a special CLI tool for this. You can make sure that, for example, the time of building your bundle or loading the bundle to a client browser does not exceed the limit you have defined.

This is very useful on CI
Also, you can generate a UI-report with validation result.

You could learn more on Github or ask your question here or with Github issue.
Also, there are many plans to improve Statoscope features.

Your feedback is so precious 😊

Webpack 5: Progress percentage calculation

Sergey Melyukov — Mon, 17 Feb 2020 15:41:07 +0000

When you build a huge project with webpack, you might notice that progress percentage may get stuck on 66-68% for a long time. This is because by default webpack calculates progress percentage as the ratio of built modules count and total modules count. It's ok at first sight, but the problem is that the total modules count is unknown in advance. Any loader may add a few modules at any moment. Therefore the total modules count is changing during the build and it causes progress percentage jerking.
To solve this problem you can specify an approximate quantity of modules in your project:

new ProgressPlugin({ modulesCount: 10000 });

It means that you need to start the build one time, get the total module count somehow and set this value to the plugin settings and then update it manually during the time.
Another solution is to write a script that will count all the files in the project folder and then pass it value to the modulesCount-field. This is based on the assumption that one module corresponds to one file, and this is not always the case. Moreover, we can't count node_modules correctly because not all of these may be using in your project.
These are dubious solutions...

A year and a half ago, during migrating a huge project to webpack I was thinking that calculating of a progress percentage for the big projects should based on the entry points, not on the modules. There are a lot of entry points in huge projects. Most likely you will split a huge codebase into many entry points, not a single entry point. I made a PR to webpack. Its meaning was to calculate a progress percentage as the ratio of the number of the built entry points to the total number of entry points. The main benefit of this approach is that the total number of the entry points is known in advance and doesn't change during the build.
This PR was merged half taken - it means that the ratio built/total will be shown only in a console and will not affect percentage calculation: 252/420 - it means that we have 420 entry points and only 252 was built at this moment.

A couple of days ago another my PR was merged to webpack 5. Its meaning was to choose a percentage calculation strategy. By default, it calculates by number of modules as before. But with percentBy-options you can change it. For example, to calculate the progress percentage totally by the number of entry points you may write:

new ProgressPlugin({ percentBy: 'entries' });

But it's not the end. while I was writing this post, another idea that seemed to be obvious came to my mind.
I was thinking that we can save total modules count to persistent cache at the end of a build, restore it before a next build will start and initialize modulesCount-options of the plugin by this value.
The first compilation will warm the cache, and the next compilations will use and update this value and will show a more realistic percentage, even if we calculate the percentage by modules count.
It is unlikely that between the builds the codebase will change so that the number of modules will be changed so dramatically, well, or it is unlikely that this will happen too often.

This PR was merged today and you can try it with beta.14 (not released yet) 🎉

Webpack 5 - Asset Modules

Sergey Melyukov — Sun, 16 Feb 2020 10:47:36 +0000

Hello. This post starts a series of articles about the new features of coming webpack 5. Why do I want to tell about the webpack? At least because I take an active part in its development and constantly delve its source code. In this article, I want to tell you about the Asset Modules - an experimental feature of webpack 5, which makes it possible to throw out a few habitual loaders, but not to cut these functionalities.

Let's imagine that we need to bundle a page with some images and styles.

With webpack 4

Webpack 4 configuration for this purpose may look like this:
webpack.config.js

module.exports = {
  module: {
    rules: [
      {
        test: /\.svg$/,
        use: [
          'file-loader',
          'svgo-loader'
        ]
      },
      {
        test: /\.css$/,
        use: [
          'style-loader',
          'css-loader'
        ]
      }
    ]
  }
};

src/index.js

import './styles.css';

// ...

src/styles.css

.logo {
  background: url("/images/logo.svg") no-repeat;
  background-size: cover;
  width: 75px;
  height: 65px;
}

Output:

/dist/main.js
/dist/eb4c5fa504857.svg

As the result all the svg-files will be handled by svgo and emitted into a bundle output path by file-loader. A styles will be transformed into:

.logo {
  background: url("eb4c5fa504857.svg") no-repeat;
  background-size: cover;
  width: 75px;
  height: 65px;
}

But later we may want to inline the images into css to optimize our page. Let's replace file-loader to url-loader for this:

      {
        test: /\.svg$/,
        use: [
-         'file-loader',
+         'url-loader',
          'svgo-loader'
        ]
      },

Output:

/dist/main.js

A bundled css will look like this:

-   background: url("eb4c5fa504857.svg") no-repeat;
+   background: url("data:image/svg+xml;base64,....") no-repeat;

Next we would want to inline only small svg (e.g, less than 8kb) and emit the rest files into the output directory. url-loader has the limit-option for this:

      {
        test: /\.svg$/,
        use: [
-         'url-loader',
+         'url-loader?limit=8192',
          'svgo-loader'
        ]
      },

After that only a small svg (less than 8kb) will be inlined, the rest svg-files will be emitted into the output directory, url-loader will implicitly use file-loader for it.

The issue is solved... but wait a minute, with webpack 5 and its Asset Modules feature this issue may be resolved easier, without url-loader and file-loader (url-loader implicitly use it for the files with size less than specified in limit-option).

With webpack 5

First, we need to explicitly specify that we want to use the Asset Modules. Let's specify it in our configuration:

module.exports = {
  // ...
+ experiments: {
+   asset: true
+ }
};

Asset Modules is an experimental feature at this moment and we are collecting a feedback from the users.

Now, we just need to mark our svg-files as asset and that's it. All that I was described related with file-loader and url-loader above will work out of the box, without any loaders:

      {
        test: /\.svg$/,
-       use: [
-         'url-loader?limit=8000',
-         'svgo-loader'
-       ]
+       type: 'asset',
+       use: 'svgo-loader'
      },

That's it, for the modules that match a rule with type: 'asset' the next logic will be applied: If a module size is less than 8kb (by default), then inline it to the bundle, otherwise emit its file to the output directory.

Note that use-property is also taken into account (if specified)

It's not the end, there are some other module-types besides asset.

asset/inline

Just like the url-loader. All the modules that match a rule with type: 'asset/inline' will be inlined to the bundle as data-url:

      {
        test: /\.svg$/,
-       type: 'asset',
+       type: 'asset/inline',
        use: 'svgo-loader'
      },

Also we can specify a custom data-url generator for type: 'asset/inline'.
For example, we can use mini-svg-data-uri for the svg-files. It transforms a svg as the data-url but without base64. It reduces a bundle size a little bit:

+ const miniSVGDataURI = require('mini-svg-data-uri');
// ...
      {
        test: /\.svg$/,
        type: 'asset/inline',
+       generator: {
+         dataUrl(content) {
+           content = content.toString();
+           return miniSVGDataURI(content);
+         }
+       },
        use: 'svgo-loader'
      },

Our css will be transformed into this:

-   background: url("data:image/svg+xml;base64,....") no-repeat;
+   background: url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg'....") no-repeat;

We even can use a custom data-url generator and the use-property

asset/resource

Just like the file-loader. All the modules that match a rule with type: 'asset/resource' will be emitted into the output directory:

      {
        test: /\.svg$/,
-       type: 'asset/inline',
+       type: 'asset/resource',
-       generator: {
-         dataUrl(content) {
-           content = content.toString();
-           return miniSVGDataURI(content);
-         }
-       },
        use: 'svgo-loader'
      },

Output path for Asset Modules

By default, all the modules with type asset/resource are emitting into the root of the output directory (dist by default). But with output.assetModuleFilename you can change this behaviour:

module.exports = {
+ output: {
+   assetModuleFilename: 'assets/[name][ext]'
+ },
  // ...
};

Output:

/dist/main.js
/dist/assets/logo.svg

By the way if we replace [name] to [hash] then we get a perfect base for assets long term caching:

module.exports = {
  output: {
-    assetModuleFilename: 'assets/[name][ext]'
+    assetModuleFilename: 'assets/[hash][ext]'
  },
  // ...
};

Output:

/dist/main.js
/dist/assets/eb4c5fa504857.svg

Also we can chage assetModuleFilename only for specific asset-rule. E.g. we may emit the svg-icons into the dist/icons and the rest asset-modules into the dist/assets:

      {
        test: /\.svg$/,
        type: 'asset/resource',
+       generator: {
+         filename: 'icons/[hash][ext]'
+       },
        use: 'svgo-loader'

Output:

/dist/main.js
/dist/assets/fd441ca8b6d00.png
/dist/icons/eb4c5fa504857.svg

asset/source

Just like raw-loader. All the modules that match a rule with type: 'asset/source' will be inlined without any transformations (as is):
file.txt

hello world

webpack.config.js

module.exports = {
       // ...
      {
        test: /\.svg$/,
        type: 'asset/resource',
        generator: {
          filename: 'icons/[hash][ext]'
        },
        use: 'svgo-loader'
      },
+     {
+       test: /\.txt$/,
+       type: 'asset/source'
+     },
      // ...

index.js

import './styles.css';
+ import txt from './file.txt';

+ console.log(txt); // hello world

Output:

/dist/main.js
/dist/icons/eb4c5fa504857.svg

asset

Combines asset/resource and asset/inline. It chooses a strategy by next logic: if a module size is greater than 8kb (by default), then it uses asset/resource strategy and asset/inline otherwise.

module.exports = {
       // ...
      {
        test: /\.svg$/,
-       type: 'asset/resource',
+       type: 'asset'
-        generator: {
-          filename: 'icons/[hash][ext]'
-        },
        use: 'svgo-loader'
      },
      {
        test: /\.txt$/,
        type: 'asset/source'
      },
      // ...

The limit for using asset/inline strategy may be overrided:

      {
        test: /.svg$/,
        type: 'asset',
+       parser: {
+         dataUrlCondition: {
+           maxSize: 20 * 1024 // 20kb
+         }
+       },
        use: 'svgo-loader'
      },

Totally: webpack 5 Asset Modules makes possible to throw out some habitual loaders because its features works out of the box.
The full example may be found here.

When webpack 5 will come out?

We have no known date. At the moment when I writing this guide, webpack 5 has beta.13 version and a user's feedback is collecting. You can help with it by trying to migrate your projects to webpack 5 (for production on your own risk). You can read more here

P.S

I plan to continue talking about the new features of webpack 5 and about the webpack itself. Some of the articles will be larger, some smaller. Very small notes (not only about webpack) can be seen in my twitter.

Thanks