Forem: Alireza Ebrahimkhani

Node package managers (npm, yarn, pnpm) - All you need to know

Alireza Ebrahimkhani — Thu, 02 May 2024 11:35:05 +0000

A week ago, one of my friends told me he had joined a new front-end project and needed to install dependencies. However, instead of npm, his manager told him to use yarn. In the project folder, he saw a pnpm-lock.yaml file. He asked me what these are and what I should do.

So I thought this might be a problem for many others, and that's why I decided to talk about this topic in this blog in a simple but in-depth way, OK let's start from the beginning.

What is Nodejs?

Node.js is an open-source, cross-platform runtime environment for executing JavaScript outside of a web browser. It's built on the V8 JavaScript engine, developed by Google for the Chrome browser, which compiles JavaScript into native machine code for efficient execution. Node.js uses an event-driven, non-blocking I/O model that makes it particularly well-suited for building scalable network applications, such as web servers, real-time communication systems, and collaborative tools. Its single-threaded event loop can handle numerous simultaneous connections, which provides high throughput and excellent responsiveness.

Node.js also popularized the use of JavaScript on the server, enabling developers to use a single programming language throughout their entire stack. This has streamlined web development processes, reducing context switching and making it easier to transfer code between the client and server. Node.js's module system, based on the CommonJS specification, allows developers to encapsulate code into reusable packages, which can be shared and updated independently. This modularity is key to managing the complexity of large systems and contributes to the rich ecosystem of third-party modules available through npm, Node.js's package manager.

npm (Node Package Manager)

npm is the default package manager for Node.js and the world’s largest software registry. It was introduced in 2010 and quickly became the cornerstone of the Node.js ecosystem. npm facilitates the installation, updating, and management of Node.js packages, which are modules of JavaScript code that can be reused across projects. The npm registry hosts hundreds of thousands of packages, making a vast array of solutions readily available to developers. npm uses a package.json file to track project dependencies and their versions, ensuring consistency and compatibility.

Features and Advantages of npm

npm automates the process of installing, updating, and managing dependencies, which helps to avoid "dependency hell." It supports semantic versioning (semver) that automatically handles patch and minor updates without breaking the existing code, thus maintaining stability across projects. npm also provides the capability to run scripts and commands defined in package.json, which can automate common tasks such as testing, building, and deployment.

Yarn

Introduced by Facebook in 2016, Yarn emerged as an alternative to npm with the promise of faster, more secure, and more reliable dependency management. Yarn provides deterministic installations by using a yarn.lock file, which precisely records the versions of all installed packages and their dependencies. This ensures that the same dependencies are installed in the same way on every machine, eliminating discrepancies in development environments that might lead to bugs.

Improvements Brought by Yarn

Yarn improves upon npm’s performance by parallelizing operations where possible and caching downloaded packages locally. This means that once a package has been downloaded, it doesn’t need to be downloaded again for other projects using the same package, speeding up subsequent installations. Yarn also checksums all packages to ensure their integrity before executing code, enhancing security.

pnpm

pnpm is another alternative to npm that focuses on performance and efficiency. Introduced in 2016, pnpm uses a unique node_modules approach that employs symlinks to save disk space and significantly reduce the amount of data transferred during installations. Unlike npm and Yarn, pnpm creates a single copy of a module on a disk and then creates a symbolic link to it in the node_modules directory where it’s needed. (I will explain pnpm structure and its algorithms in separate blog)

Key Features of pnpm

pnpm’s strictness in linking packages only if they are explicitly declared in the project’s package.json helps in avoiding accidental usage of transitive dependencies (dependencies of dependencies), thus promoting better practices in dependency management. This also results in a cleaner and more efficient module resolution, reducing potential confusion and conflicts among versions of packages.

npm vs. Yarn vs. pnpm

While npm and Yarn share a similar approach to managing packages, pnpm differs significantly in its strategy by using symlinks to manage node_modules. npm is generally easier to integrate into projects due to its status as the default package manager for Node.js. Yarn’s focus on performance and security makes it particularly suited for large-scale applications or those with stringent reliability requirements. pnpm is often favored in environments where disk space and bandwidth are critical considerations.

Evolution and Current Trends

The evolution of these package managers reflects ongoing efforts to address the complexities of managing dependencies in modern web development. Continuous improvements are being made to enhance security features, reduce overheads, and ensure compatibility with the burgeoning array of tools and frameworks in the JavaScript ecosystem.

For the last section, I want to explain the purpose of lock files in package managers:

Lock Files in Package Managers

Lock files are a crucial component in modern package management, providing a snapshot of all dependencies installed at a given time. This snapshot includes the exact versions and configurations of the packages used in a project. npm uses package-lock.json while Yarn utilizes yarn.lock and pnpm uses pnpm-lock.yaml to maintain these details. These lock files are automatically generated when dependencies are installed and are crucial for ensuring that installations are consistent across different environments, be it development, testing, or production.

Purpose and Benefits of Lock Files

The primary purpose of lock files is to record the exact version of each package and its dependencies as they were installed. This ensures that subsequent installations, regardless of when or where they occur, produce the same structure in node_modules by using the exact versions listed in the lock file. This deterministic behavior is vital for addressing issues related to dependency drift, where different team members or deployment environments might otherwise use slightly different versions of packages, potentially leading to bugs or inconsistencies that are hard to trace.

Lock Files and Dependency Management

Lock files play a significant role in continuous integration and deployment pipelines. By committing lock files into version control systems, teams can ensure that every member and every deployment environment uses the exact same set of dependencies. This not only reduces "works on my machine" problems but also enhances security by ensuring that only audited and approved versions of packages are used in production. Furthermore, lock files allow developers to review changes in dependencies via version control systems, adding an additional layer of oversight before changes are merged into production codebases.

And, in the end hope you enjoy this blog and learn new things about Nodejs and its package management ecosystem.

Be aware of Arrays - V8 engine advice

Alireza Ebrahimkhani — Tue, 13 Feb 2024 12:03:35 +0000

In this blog, I decided to talk about arrays and their behavior inside v8. By understanding these you can write efficient code that is good for v8 to optimize.

In V8(the JavaScript engine behind Google Chrome and Node.js), "elements kinds" refer to the internal classifications used to optimize array operations. V8 uses these classifications to make assumptions about the types of elements an array contains, which in turn allows it to optimize access to and manipulation of these arrays. Understanding elements kinds can be crucial for developers looking to write high-performance JavaScript code, as certain operations can cause an array to transition between kinds, potentially impacting performance.

While running JavaScript code, V8 keeps track of what kind of elements each array contains. This information allows V8 to optimize any operations on the array specifically for this type of element. For example, when you call reduce, map, or forEach on an array, V8 can optimize those operations based on what kind of elements the array contains.

V8 categorizes arrays into different "elements kinds" based on the types of values they store and whether they have "holes" (missing elements). This classification allows V8 to use more efficient storage and access methods for arrays, depending on their content. Here's a closer look at each type:

Packed vs. Holey

Packed Elements: These arrays have no missing elements between the first and last defined positions. Access to packed arrays is typically faster because V8 can optimize memory layout and access patterns, assuming a continuous block of elements for example: const array = [1, 2, 3];

Holey Elements: These arrays contain holes, or undefined positions, which can occur if elements are deleted or if an array is declared with a larger initial size than the number of elements it contains. Operations on holey arrays are generally slower because V8 must check for the presence of elements before accessing them for example if you write code like the below sample v8 will treat this array like a Holey element:

Types of Elements

Within the packed and holey classifications, arrays are further categorized by the types of elements they store:

Smi Elements: "Smi" stands for "small integer," referring to a specific optimization for storing 31-bit integers directly within pointers, saving space and access time. Arrays that exclusively contain Smi values are optimized differently from those containing other types of values.

Double Elements: These arrays store floating-point numbers. Because floating-point numbers require a different storage format than integers or other types, V8 optimizes arrays that solely contain doubles differently.

Elements: This category is for arrays that can contain elements of any type, including objects, strings, and symbols, in addition to numbers. These arrays are the most flexible in terms of the types of values they can contain but may not benefit from some of the optimizations that more specialized arrays do.

The elements kinds lattice

V8 implements this tag transitioning system as a lattice. Here’s a simplified visualization of that featuring only the most common elements kinds:

It’s only possible to transition downwards through the lattice. Once a single floating-point number is added to an array of Smis, it is marked as DOUBLE, even if you later overwrite the float with a Smi. Similarly, once a hole is created in an array, it’s marked as holey forever, even when you fill it later.

In general, more specific elements kinds enable more fine-grained optimizations. The further down the elements kind is in the lattice, the slower manipulations of that object might be. For optimal performance, avoid needlessly transitioning to less specific types — stick to the most specific one that’s applicable to your situation.

Performance tips

In most cases, elements kind tracking works invisibly under the hood and you don’t need to worry about it. But here are a few things you can do to get the greatest possible benefit from the system.

Avoid reading beyond the length of the array

Nowadays, the performance of both for-of and forEach is on par with the old-fashioned for loop and when the collection you’re looping over is iterable just use for-of and for arrays specifically, you could use the forEach built-in.

Avoid elements kind transitions

In general, if you need to perform lots of operations on an array, try sticking to an elements kind that’s as specific as possible, so that V8 can optimize those operations as much as possible for example, just adding 1.1 to an array of small integers is enough to transition it to PACKED_DOUBLE_ELEMENTS.

The same thing goes for NaN and Infinity. They are represented as doubles, so adding a single NaN or Infinity to an array of SMI_ELEMENTS transitions it to DOUBLE_ELEMENTS.

Avoid polymorphism

If you have code that handles arrays of many different elements kinds, it can lead to polymorphic operations that are slower than a version of the code that only operates on a single elements kind.

Consider the following example, where a library function is called with various elements kinds. (Note that this is not the native Array.prototype.forEach)

Built-in methods (such as Array.prototype.forEach) can deal with this kind of polymorphism much more efficiently, so consider using them instead of userland library functions in performance-sensitive situations.

Avoid creating holes

This is one of the most important tips you have to consider because once the array is marked as holey, it remains holey forever — even if all its elements are present later!
Holey arrays occur when there are missing elements in the array, leading to less efficient access and manipulation due to the engine's need to handle potential "holes." Here’s how you can avoid creating holey arrays:

If you know the size of the array in advance but initialize it with empty slots, it becomes a holey array. Instead, pre-initialize it with known values. For integers, you might use 0 or another placeholder value. For a more concise way to initialize arrays without creating holes, use the fill method. This is especially useful when the array size is known, but you want to avoid holes.

Also deleting elements from an array can introduce holes, transforming it into a holey array. If you need to remove an element, consider setting it to undefined or null if you cannot use methods like splice to restructure the array without leaving gaps.

and for the last one when adding new elements to an array, use methods like push or unshift instead of directly setting them by index, which could create holes if the index is beyond the current array length.

When writing performance-sensitive JavaScript code, understanding how these array types affect performance can guide how you structure your data. Keeping arrays homogenous (all Smi or all doubles) when possible can leverage V8's optimizations for faster access and manipulation.

Fun part :))

For debugging elements kinds to figure out a given object’s “elements kind”, get a debug build of v8 (either by building from source in debug mode or by grabbing a precompiled binary using jsvu), and run:
out/x64.debug/d8 --allow-natives-syntax
Note that “COW” stands for copy-on-write, which is yet another internal optimization. :))

And, in the end hope you enjoy this blog and learn new things about v8 and its magics behind javascript.

What JIT really is ? (Part 2)

Alireza Ebrahimkhani — Fri, 25 Mar 2022 22:48:00 +0000

In the previous part, I talked about what compilers and interpreters are, and in this part, I want to continue the content. If you have not read the previous part, you can read it from this link.

Let's have a look at the pros and cons of each. First of all, interpreters are quick to get up and running, right? Because if we go to our example with an interpreter, we don't have to convert into another language.There's no compilation step before you can start running your code, you just give this to an interpreter and the interpreter starts translating their first line and just runs the code for us because of this, an interpreter is a natural fit for something like JavaScript, right, a JavaScript file gets sent from the server to the browser to the client on the front end and we want that JavaScript to execute right away because our users are going to be waiting on the Web page trying to interact with our application. And JavaScript originally was created for the browser, so being able to interpret JavaScript and run it as fast as possible.

Well, that was ideal and this is exactly why it JavaScript used interpreters at the beginning but there's a con with using an interpreter and this is the same problem that Google had back in the day when they had Google Maps running a lot of JavaScript, but it will get slower and slower and slower because the problem with interpreters is that when you're running the same code more than once, for example, if you're in a loop like this one over here where we're running this piece of code over and over and over, even though it gives us the same result.

It can get really, really slow. The compiler actually helps us here.It takes a little bit more time to start up because it has to go through that compilation step at the beginning, go through our code, understand it and spit it out into a another language.But the compiler will be smart enough and if it sees code like that we just loop over and it has the same inputs returning the same outputs it can actually just simplify this code and instead of calling function multiple times, it can just replace this function with something like nine because a compiler doesn't need to repeat the translation for each pass through in that loop, the code generated from it is actually faster and these sort of edits that compilers do are called optimization.

So let's go back to that question, interpreter or a compiler?
Which one is better?

They both have their pros and cons compiler obviously takes a little bit longer to get up and running,but the code is going to eventually run faster or interpreter that is really fast to get up and running but unfortunately doesn't do any optimizations.

Is there a way that we can get the best of both worlds and this is what some engineers came up with in late 2000, and if we use Google as our example with the V8 engine, what if we combine the best of both worlds? What if instead of using the compiler and interpreter, we combine these two and create something called a JIT compiler or just in time compiler ?

This is exactly what Browsers started doing, Browsers started mixing compilers, specifically these JIT compilers for Just in Time compilation to make the engines faster.

So let's see how V8 engine does this underneath the hood in order to use the best of both worlds remember, we passed the code, we turn it into an abstract syntax tree (AST), and then this code initially goes to an interpreter. Initially, the code gets sent to the interpreter and in the V8 engine, this is called ignition.I know they have a really cool sounding names to make them sound like they do things fast but this interpreter and the V8 engine is called ignition and it takes AST and it spits out bytecode and remember, bytecode is code that's not as low level as machine code, but it's code that is able to be interpreted by the JavaScript engine in order to run our programs.

So that's the first step, but there's something called a profiler that's checking out our code. This profiler also called a monitor, monitors and watches our code as it runs.
It makes notes on how we can optimize this code, such as how many times it is being run, what types are used and how we can possibly optimize code and using this profiler as the code is running through our interpreter, which tells our browser what to do.

If the same lines of code are run a few times we actually pass off some of this code to the compiler or a JIT compiler, because as the code is running the interpreter is going to say, hey, here's some code for you to optimize, passes it off to the compiler and the compiler as the application is running, takes a code and compiles it or modifies it.
So it does what you ask it to, but trying to make optimizations so it runs faster and it then replaces the sections where it could be improved of the bytecode with optimized machine code.

So that code, that optimized code is used from that point on instead of the slower bytecode, so it mixes and matches things and it constantly runs through this loop and this means that the execution speed of JavaScript code will be improving.

This means that the execution speed of the JavaScript code that we entered into the engine is going to gradually improve because the profiler and the compiler are constantly making updates and changes to our bytecode in order to be as efficient as possible so interpreter allows us to run the code right away and the compiler and profiler allows us to optimize this code as we're running.

That's where the name comes from JIT compiler and this compiler for V8 is called Turbo Fan again, a cool sounding name to make it sound like things are going fast and by the way, just a bit of a fun fact, previous versions of V8 Engine actually used JIT compilers with really cool sounding names, but they changed it to this method because there are some faster optimizations that they could do.

So why do we just learn all of this, you can write JavaScript without knowing any of this stuff and you'll be fine. Now that we know how the engine works underneath the hood, we can write more optimized code that the compiler can take and run faster than our regular JavaScript and we also can use this knowledge to make sure that we don't confuse the compiler because the compiler is imperfect, it can make mistakes and it can try to optimize code that actually does the opposite and if it makes a mistake and it does something unexpected, it does which takes even longer time to reverted back to the interpreter.

I hope this paper is useful for you and helps you better understand the JIT compiler.

What JIT really is ? (Part 1)

Alireza Ebrahimkhani — Fri, 25 Mar 2022 22:15:00 +0000

In this article i want to explain what JIT really is. Before that i want to talk about interpreters and compilers which as we can see are important pieces in our JavaScript engine.

You see in programming there are generally two ways of translating to machine language or something that our computers can understand. And what we're going to talk about here actually applies to most programming languages not just JavaScript,Python, Java, C++, any language you can think of is going to use some of these concepts.

Interpreter

Let's start with the first one, the interpreter. With an interpreter what we do is we translate and read the files line by line on the flight. Let's have a look at how this works.

I have a piece of code here, I have a function that does some calculation and we just loop over this calculation, which is five plus four right over here. And we do that a thousand times. That's a long time and you see that we get the result nine over here, but the loop is happening over and over.

Now with an interpreter, if I give this file to an interpreter.The translation happens line by line on the fly.And this is how you think the code should be run, right?The interpreter sees the first line and says, all right,this is a function.And then it goes to the next part and says, all right, I should loop over this a bunch of times and it starts doing some calculation.It's going to go and see.That's what the function is.I'm going to run five plus four and then I'm looping again.So I'm going to go once again, run some calculation five plus for some calculation five plus four.And it keeps running.Because interpreting code simply means taking a set of instructions like we have over here and returning an answer and doing something with that code, it's just like me telling a computer to do this, then do that, then do this.And initially, that's how JavaScript worked, it was interpreting using an interpreter, which makes sense.

Compiler

Well, a compiler like an interpreter doesn't translate on the fly. What it does is it works ahead of time to create a translation of what code we've just written. And it compiles down to usually a language that can be understood by our machines.

Let's have a look at the above code in more detail with a compiler.

This time around, it's going to take one pass through the code and try to understand what the code does.And it's going to take the program in JavaScript or any type of language and write a new program in your new language.
If we interpret it this language that is going line by line one at a time and running it, it's going to create the same results as our previous language, so Compiler tries to understand what we want to do and takes our language and changes it into something else and this usually happens into something called a lower level language, such as machine code.Now, I know you're still a little bit confused.
You see, the definition itself is actually a little bit fuzzy in some respects, all languages have to be interpreted and compiled because it has to run.

It has to be interpreted and it also has to most likely get translated into something low level like machine code, for example we have a high level language here like JavaScript, and we run a piece of code here an interpreter will take this code line by line and spit out bytecode that will execute code for us or a compiler might take code and go through and spit out machine code so that it can give it to a CPU so that the CPU can actually run the code.

There are two ways to run JavaScript using an interpreter or a compiler.Now, I know it's still a little bit fuzzy, but if I ask you, why would we want to do one over the other? What are the pros and cons on each? Why would you pick one over the other?

In the next part going to explore that and say what JIT really is ...

You can read the next part from here