Forem: Rahul Soni

Friendship ended with Webpack, now ESBuild is my best friend

Rahul Soni — Sat, 30 Sep 2023 17:03:27 +0000

In this post, I will be discussing about the current configuration of our JavaScript/React setup.
Last year we were struggling with very long webpack build times. Due to our project's constraints, every page in our app is a separate React App, which when compiled and bundled is injected into it's own HTML page, which gets served from a Perl Server.
Our previous webpack configuration used to look like this, where after running npm run build the files would be generated into the reactbuild folder.

require("babel-polyfill");

module.exports = {
  entry: {
    page_name: ['path_to_page'],
    ...,
    ...
  },
  output: {
    filename: "[name].main.js",
    path: path.resolve(__dirname, "./reactbuild/"),
  },
}

At this point our build times were around a minute for npm run build, even while running in watch mode it would take around 5-8 seconds on average for webpack to compile all the changes, which was also affecting our developer experience. Moreover, the longer feedback loop between making changes and seeing the results made it challenging to maintain a high level of productivity and responsiveness in addressing issues and implementing new features.

After some digging around we came to know about esbuild-loader for webpack, which was supposed to be faster than babel-loader that we were using currently. After replacing babel-loader with esbuild-loader, our build times were reduced to 1/10th, along with our bundle sizes. At that point our webpack.config.js looked like this

const path = require("path");
const { ESBuildMinifyPlugin } = require("esbuild-loader");
const { ProvidePlugin } = require("webpack");

module.exports = {
 entry: {
  page_name: ['path_to_page'],
   ...,
   ...
 },
 output: {
   filename: "[name].main.js",
   path: path.resolve(__dirname, "./reactbuild/"),
 },
 plugins: [
    new ProvidePlugin({
      React: "react",
    }),
  ],
  optimization: {
    minimizer: [new ESBuildMinifyPlugin()],
  },
 module: {
    rules: [
      {
        test: /\\.css$/,
        use: [
          "style-loader",
          "css-loader",
          {
            loader: "esbuild-loader",
            options: {
              loader: "css",
              minify: true,
            },
          },
        ],
      },
      {
        test: /\\.(js|jsx)$/,
        exclude: /node_modules/,
        loader: "esbuild-loader",
        options: {
          loader: "jsx",
          target: "es2015",
        },
      },
    ],
  },
}

Everything was working great until we encountered a performance bottleneck with the esbuild-loader when we began adding new pages to our entry points array. As a result, our build times started to increase. This became a significant issue for us, as our project progressed. We noticed that our build process started to slow down considerably, affecting our development efficiency. We had to find a solution to address this challenge and optimize our build process for better performance and faster build times. Hence, we had to take immediate action to resolve this bottleneck and improve the overall build performance. At one point, our build process looked like this, with slow build times and a noticeable impact on our development speed.

In addition to the excessively long build times mentioned earlier, we encountered another issue in our CI pipeline. Specifically, when running the npm run build command, it would sometimes fail to complete execution, thereby causing the subsequent steps in the pipeline to fail as well. To address this problem, we brainstormed the idea of dividing the build step into multiple stages within our CI process, hoping that it would solve the issue. Unfortunately, this approach also proved to be unsuccessful.

Enter ESBuild

While looking for a solution to our build time problem, we stumbled upon ESBuild. We were a little skeptical about it's shown performance, as it was shown 100x faster than webpack, but we decided to give it a try and we were amazed by the results. ESBuild lived up to its promise and significantly improved our build times. With ESBuild, our build times were reduced even further, allowing us to compile and bundle our React apps in a matter of seconds. The performance boost was a game-changer for our development workflow, and we quickly became big fans of ESBuild.

The below image showcases our current build times with ESBuild. From 90 seconds in webpack to less than 2 seconds with ESBuild.

Our current ESBuild Config files looks like this

const sassPlugin = require("esbuild-sass-plugin");
const autoprefixer = require("autoprefixer");
const postcss = require("postcss");

const path = require("path");
const esbuild = require("esbuild");
const entries = {
 page: "path_to_page",
 ...,
 ...,
 ...
}
const config = {
  entryPoints: Object.values(entries),
  bundle: true,
  logLevel: "info",
  outExtension: {
    ".js": ".main.js",
  },
  outbase: path.resolve(__dirname, "./react/"),
  outdir: path.resolve(__dirname, "./reactbuild/"),
  format: "esm",
  splitting: true,
  mainFields: ["browser", "module", "main"],
  incremental: process.argv.includes("--watch"),
  inject: ["./mui-shim.js"],
  loader: {
    ".js": "jsx",
    ".locale.json": "file",
    ".json": "json",
    ".png": "file",
    ".jpeg": "file",
    ".jpg": "file",
    ".svg": "file",
  },
  watch: process.argv.includes("--watch") && {
    onRebuild(error) {
      if (error) console.error("watch build failed:", error);
      else console.log("watch build succeeded:");
    },
  },
  minify:
    !process.argv.includes("--watch") &&
    !process.argv.includes("--development"),
  metafile: true,
  plugins: [
    sassPlugin.sassPlugin({
      async transform(source) {
        const { css } = await postcss([autoprefixer]).process(source);
        return css;
      },
    }),
  ],
  target: ["safari12", "ios12", "chrome92", "firefox88"],
};
esbuild.build(config).catch(() => process.exit(1));

With splitting code shared between multiple entry points is separated into a dedicated shared file that can be imported by both entry points.

The outdir options allows us to specify the path where the generated files will be stored.

The incremental option allows us to speed up rebuilding files in the development environment.

We highly recommend giving ESBuild a try if you're facing similar performance issues with your webpack builds.

In conclusion, the journey from struggling with prolonged webpack build times to embracing ESBuild has been nothing short of transformative for our development team. ESBuild emerged as the game-changing solution we needed to overcome the performance bottlenecks that were hindering our progress.

Building an ETL Job with Python and DuckDB: Analyzing Electric Vehicle Data

Rahul Soni — Fri, 09 Jun 2023 20:38:41 +0000

In this blog post, we will explore how to build an ETL (Extract, Transform, Load) job using Python and DuckDB to analyze electric vehicle data. We will walk through the code step by step, explaining each part and highlighting key concepts along the way. By the end of this post, you will have a some understanding of how to process and analyze datasets efficiently using Python and DuckDB.

Prerequisites

Before we begin, make sure you have the following prerequisites:

Python installed on your machine
DuckDB Python library installed (pip install duckdb)

Introduction

As the popularity of electric vehicles continues to rise, there is a growing need to analyze and extract insights from electric vehicle data. This blog post aims to demonstrate how to build an ETL job to process electric vehicle data using Python and DuckDB.

Setting Up DuckDB and Python

To get started, we need to import the necessary libraries and establish a connection to DuckDB. Open your favorite Python editor and create a new Python script. Let's begin by importing the required libraries:

import duckdb
from duckdb import DuckDBPyConnection
import os
import logging

We also need to configure logging for better visibility into the execution of our ETL job. Add the following code to configure the logging module:

logging.basicConfig(
    level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s"
)
logger = logging.getLogger(__name__)

Creating the Database Table

The first step in our ETL process is to create a table in DuckDB to store the electric vehicle data. We'll define the schema of the table and execute the SQL statements to create it. Here's the code for creating the "vehicles" table:

CREATE_TABLE_VEHICLES = """
        DROP TABLE IF EXISTS vehicles CASCADE;
        CREATE TABLE vehicles (
            VIN VARCHAR(10),
            County VARCHAR,
            City VARCHAR,
            State VARCHAR(2),
            Postal_Code VARCHAR(10),
            Model_Year INTEGER,
            Make VARCHAR,
            Model VARCHAR,
            Electric_Vehicle_Type VARCHAR,
            CAFV_Eligibility VARCHAR,
            Electric_Range INTEGER,
            Base_MSRP DECIMAL,
            Legislative_District INTEGER,
            DOL_Vehicle_ID VARCHAR(15),
            Vehicle_Location VARCHAR,
            Electric_Utility VARCHAR,
            Census_Tract VARCHAR(11)
    );
"""

conn = duckdb.connect(database="electric_vehicles_data")
cursor = conn.cursor()

# Create vehicles table
try:
    cursor.execute(CREATE_TABLE_VEHICLES)
    logger.info("Vehicles table created.")
except Exception as e:
    logger.error(f"Failed to create vehicles table: {str(e)}")
    conn.close()
    return

In the code above, we define the SQL statement CREATE_TABLE_VEHICLES as a multi-line string. It drops the existing "vehicles" table if it exists and creates a new table with the specified columns and data types.

We establish a connection to DuckDB using duckdb.connect() and create a cursor to execute SQL statements. Then, we execute the CREATE TABLE statement using the cursor. If any exception occurs, we log an error message and close the connection.

Extracting and Loading Data

Now that we have our table structure in place, let's proceed to extract data from a CSV file and load it into our "vehicles" table. We'll define a function called insert_data() to handle this task:

def insert_data(cursor: DuckDBPyConnection, csv_file_path: str):
    try:
        cursor.sql(
            f"

INSERT INTO vehicles SELECT * FROM read_csv_auto('{csv_file_path}');"
        )
        cursor.commit()
        logger.info("Data insertion successful.")
    except Exception as e:
        logger.error(f"Failed to insert data: {str(e)}")

The insert_data() function takes two parameters: the cursor object and the path to the CSV file containing the data. Inside the function, we use the read_csv_auto() method provided by DuckDB to read the CSV file and insert the data into the "vehicles" table using the INSERT INTO statement.

To call this function and load the data, add the following code to your script:

csv_file_path = "data/Electric_Vehicle_Population_Data.csv"
insert_data(cursor, csv_file_path)

Make sure to replace "data/Electric_Vehicle_Population_Data.csv" with the actual path to your CSV file.

Data Analysis Queries

With the data loaded into our "vehicles" table, we can now perform various data analysis queries. Let's define a few functions to run these queries and retrieve the results.

Counting Cars per City

Our first query will count the number of cars per city. Define a function called count_cars_per_city():

def count_cars_per_city(cursor: DuckDBPyConnection):
    cars_per_city_query = """
    SELECT city, COUNT(VIN) AS cars_in_city 
    FROM vehicles 
    GROUP BY city
    """

    try:
        cursor.execute(cars_per_city_query)
        results = cursor.fetchall()
        return results
    except Exception as e:
        logger.error(f"Failed to count cars per city: {str(e)}")
        return []

The count_cars_per_city() function takes the cursor object as a parameter. It executes the SQL query to count the cars per city and fetches the results. The results are returned as a list of tuples, where each tuple contains the city name and the corresponding car count.

To retrieve the cars per city, add the following code to your script:

cars_per_city = count_cars_per_city(cursor)
logger.info(f"Cars per city: {cars_per_city}")

Finding the Top Three Popular Vehicles:

The find_top_three_popular_vehicles function executes an SQL query that groups the data by make and model, counts the number of vehicles for each combination, and retrieves the top three most popular vehicles based on the vehicle count. This information helps us identify the most sought-after electric vehicles.

Determining the Most Popular Vehicle by Postal Code:

The find_most_popular_vehicle_by_postal_code function executes an SQL query that groups the data by postal code, model, and make. It then calculates the vehicle count for each combination and selects the most popular vehicle for each postal code. This analysis reveals the preferred electric vehicle models in different areas.

Counting Cars by Model Year:

The count_cars_by_model_year function executes an SQL query to count the number of cars for each model year. This analysis provides an overview of the electric vehicle market's growth over time.

Saving Results

As we generate the results of our data analysis queries, it's important to save them for future reference. Let's define a directory structure to store the results and write the results to files.

output_dir = "data/results"
os.makedirs(output_dir, exist_ok=True)

In the code above, we define the output_dir variable as the path to the directory where the results will be saved. We use os.makedirs() to create the directory if it doesn't exist.

Next, we'll save the results of the "count_cars_by_model_year" query to separate Parquet files for each year. Add the following code after the count_cars_by_model_year() function:

for row in rows:
    year, count = row

    year_dir = os.path.join(output_dir, str(year))
    os.makedirs(year_dir, exist_ok=True)

    file_path = os.path.join(year_dir, f"{year}.parquet")

    try:
        with open(file_path, "w") as file:
            file.write(f"Count: {count}")
        logger.info(f"File saved: {file_path}")
    except Exception as e:
        logger.error(f"Failed to save file: {str(e)}")

In the code above, we iterate over the rows returned by the count_cars_by_model_year() function. For each row, we extract the year and count values. We create a subdirectory for each year within the output_dir and save a file named "{year}.parquet" with the count value inside.

Conclusion

Congratulations! You have successfully built an ETL job using Python and DuckDB to analyze electric vehicle data. We covered the steps involved in creating a database table, loading data from a CSV file, running data analysis queries, and saving the results.

Python and DuckDB provide powerful tools for processing and analyzing large datasets efficiently. With the knowledge gained from this blog post, you can now apply these concepts to other data analysis tasks and gain valuable insights from your data.

Feel free to experiment with different queries and analysis techniques to further explore the electric vehicle data. Happy coding!

Note: Don't forget to close the database connection at the end of your script: conn.close().

Github Repo

Additional Resources

How to cleanly import your files.

Rahul Soni — Fri, 15 Oct 2021 08:27:26 +0000

We all have sometime imported files or components like this in multiple lines, this would be okay if there is only one component or file to be imported, but as the codebase grows, writing import statements for every single file would get tedious as we're just repeating ourselves.

We're importing the components using the relative paths by leaving the src folder and going into the components folder and then going into the specific component folder.

We could easily replace that by importing all the files from their respective folder in a single line like this using non-relative import.

relative and non-relative imports

Note: In the first image components are exported using default export and in the latter image non default export is used.

First we need to define jsconfig.json file in the root of our project.
jsconfig.json has compilerOptions object which sets the baseUrl to be the src folder for resolution of non-relative imports.

Each component gets it's own folder because components have several files involved like css or sometimes even more so it makes sense to do have a separate folder for each one.
It would look something like this with the component file, a css file and index.js file for exporting the component

In the root of components folder we need to define an index.js file which will export all the components defined in it.

The App component is imported in the main index.js file and the final result is this.

We can also use our import our components inside other component files using the same non-relative paths like this.

By using this way we can reduce our cleanly import our files.

How to create your own custom VS code snippets.

Rahul Soni — Thu, 14 Oct 2021 16:57:52 +0000

Code snippets can be very useful and can potentially increase productivity and reduce input errors.

With snippets we can insert commonly used code blocks and can create boilerplate templates saving us keystrokes.

Snippets in VS code are written in JSON and to create your own custom snippet.

Snippets can be configured through Preferences (Ctrl + Shift + P) -> Configure User Snippets.

It should look something like this.

Snippets can be configured for every file or specific languages in their JSON file.

Every snippet has 3 main properties

prefix - Shortcut by which it will be called.
body - Actual code to be inserted
description - Description of the code

This is a custom snippet with prefix arfc for generating a React component. It works in following steps.

Imports React
Creates a functional React component with placeholder name functionName
Returns an empty div
exports the component

It will yield this code.

Another one of my favorite is

The ftcData will inject this piece of code

A very good tool to generate custom snippets is this Snippet Generator, which works for Sublime and Atom as well.

A very popular VS code extension for JavaScript, React, React-Native, Redux snippets

How to set up a new React app without create-react-app.

Rahul Soni — Thu, 14 Oct 2021 10:59:28 +0000

Generally for spinning up a React project we use the command.

npx create-react-app app-name

create-react-app is good for starter projects but it comes with it's own disadvantages.

Difficult to add custom build configs.
Comes with a lot of extra dependencies which might not be needed.

There are 2 ways to get around this.

Eject the app by running [npm run eject]
- This will copy all the config files and dependencies to your package.json, which will include webpack, babel, ESlint etc.
Setup your own webpack config with babel.
- This allows you to customize the entire app according to your needs.
- The only disadvantage is there is a learning curve for setting up a webpack and to configure it properly.

One disadvantage while using webpack that it uses bundled development. That means while in development mode, whenever a file is changed and is saved, it builds and rebundles the entire application. If the project codebase is large enough even a small change can take a while before reflecting in the browser. This introduces additional complexity in development workflow which is unnecessary.

The opposite approach to that is unbundled development. That's where Snowpack comes in.

What is Snowpack ?

Snowpack is a lightning-fast frontend build tool, designed for the modern web. It is an alternative to heavier, more complex bundlers like webpack or Parcel in your development workflow. Snowpack leverages JavaScript's native module system (known as ESM) to avoid unnecessary work and stay fast no matter how big your project grows.

During development mode once the files are built it is cached indefinitely, and whenever a file is changed and saved only the changed file is rebuilt by snowpack. Also a point to note that it only rebuilds the file when it is required in the browser. Which mean no matter the size of the codebase because the files are cached forever only the changed files are re-built when it is needed changes are always lightning fast.

Snowpack's guide to create a react app

First we'll initialize a npm project in a empty directory with the command.

npm init

We'll add Snowpack as a development dependency and add it to start script

npm install --save-dev snowpack

The package.json file would look like this.

Now we'll add react and react-dom in the project with the command

npm install react react-dom

Now we'll create a index.html file at the root level of our directory and add a div tag with id = root in the body section. This is the div where our entire react app will be injected.
Also we need to reference an entry point for our react application in a script tag which would be index.js.

We'll add index.jsx file in the src folder, which would look something like this.
The jsx will be transformed to js by snowpack automatically and injected into the index.html file.
At last we'll run the npm start command and we should see this.

Additional configurations can also be done, I recommend checking out Snowpack documentation and learn more.

This is how we can easily setup our React app without using create-react-app or webpack.