Forem: StackBlitz

What is pnpm? Is it really so fast and space-efficient?

Sylwia Vargas — Thu, 13 Oct 2022 12:57:41 +0000

We announced at ViteConf that our WebContainers now support pnpm. It was a major achievement in our commitment to support the Vite ecosystem as many projects running on Vite also use pnpm (examples include Vue, Astro, SvelteKit).

In this post, I’d like to talk about what sets pnpm apart from other package managers - and why we need package managers at all. As a person who best operates through analogy, I will also talk about: 🍣 sushi, 🪦 cemetery, 🎁 birthdays, and 🖼 memes.

What are package managers?

I've also written an “Explain like I’m 5” version of this section!

Let’s start with the terminology.

package, dependency, node module

A “package” (or “dependency”) is one or a few files bundled neatly together that can be downloaded from a package registry. Once it’s downloaded and installed, it resides in the node_modules folder in a project’s directory. In this post, I will use the words a Node module, a package, and a dependency interchangeably as many folks in the tech community do. That being said, the npm docs offer a differing opinion so check it out if you’re interested in trivia and precision. 🙂

package.json

Each package has a file called package.json which contains information on the package author, its version, its dependencies, and so on. A package can have many dependencies, which in turn can also have many dependencies (called "sub-dependencies"). This sounds like a lot of files, doesn’t it! It really is, especially if your project requires bigger packages.

In the olden times, you’d manually download, install, and configure the dependencies of your app. Keep in mind that most packages are frequently updated, which means that you’d have to always make sure that your project's dependencies (and their sub-dependencies!) are up to date - or that in a given project you use a correct version of a given package. This sounds not only stressful but also unnecessarily repetitive! This is where a package manager comes in.

package managers

A package manager is a tool that helps you keep track of all dependencies of your app in a consistent way. It automates the tasks of dependency installation, upgrading, configuration, and removal. It makes sure that all the packages that your app needs are installed, of the correct version, and up-to-date. This frees so much time for coding… and procrastination 🤪

The first package manager for the Node runtime was introduced only in 2010 and it was npm, (which, contrary to popular belief, doesn’t stand for “Node package manager” and, in fact, is an empty acronym). Twelve years later, npm is now managed by Microsoft and there are two other contenders to the iron throne of the package universe: yarn and pnpm.

What do package managers do?

A package manager grabs the published source files from the online registry and installs them into a directory (a folder) called node_modules. This is the folder where your app will look for the packages when you call the require() method.

You can think about the node_modules folder as a box with helpful tools that you can plug into your app so it works in a desired way. Except that all the tools are made of iron (or osmium!) and the box is massive. What I mean to say is that, sadly, prior to pnpm the node_modules folder would usually be bulky and take a lot of disk space. This resulted in a collection of memes such as:

Imagine that you have numerous projects on your local machine, and each project comes with its own node_modules folder. Most likely, many of these projects use a similar stack. In such a case, you end up with the same packages in numerous projects. Duplication and redundancy! Think about all those inspired side projects that never left the boilerplate stage, all the past attempts to check out all the hot new frameworks - all of them are chilling on your precious disk space.

Who cursed the `node_modules` folder?

To understand these jokes, we need to take a stroll down memory lane and look at how package management has traditionally been handled.

To the best of its intentions and abilities, npm handled the dependencies by splitting the installation (which is triggered by, for example, npm install ) process into three phases:

Resolving, when the package manager is checking all the project’s dependencies (and their sub-dependencies) listed in the package.json file, finds a version that satisfies the version specifier (for instance, ^1.0.0 would install any future minor/patch versions). Afterwards, it creates a file like package-lock.json for npm and pnpm-lock.yaml for pnpm. you can think about it like a project’s equivalent of a birthday gift wishlist.
Fetch, when the package manager takes the list of resolved dependencies and fetches all the packages from the package registry, which is an equivalent of the feverish shopping in a crowded mall two hours before the party for the gifts all of your friends will give to the birthday person.
Linking, when the package manager writes all the dependencies into the project’s node_modules folder, which, finally, is an equivalent of placing all the gifts in the corner of the party room for when they are needed.

In this scenario, each phase needs to end for the next one to begin. This means that if one dependency has twenty dependencies itself or if one package takes forever to be downloaded, you may have to wait for a long time. If we follow the birthday analogy, imagine that the wishlist included a new version of a popular phone and a box of chocolates. You end up queuing for a few days in front of the phone store to only then buy the chocolates. In such a system, you can’t “save” your spot in the queue, go buy the chocolates and join the party, even if that looked like a better use of your time. You see that t*he way npm manages its tasks is just not efficient.*

As we said earlier, it is also the matter of the disk space. You will end up with numerous duplications of the same packages. And even if there are different versions of the same package, it is rarely the case that all of the files have been changed from one version to another. You still end up with some number of copies. We can agree that the way npm manages the disc space is not efficient either.

How is pnpm a solution to all our worries and troubles?

On its docs page you can read that pnpm is a “fast, disk space efficient package manager.” It really is fast - locally, up to three times faster than the alternatives - and space-efficient. But - how?

pnpm is fast

Do you remember the birthday wishlist analogy? As a gofer, npm first collected all the wishes, then bought all the gifts, then placed them in the corner of the party room. Each phase needed to end for another to begin. This is how the npm gofer organizes its way through life.

Well, if pnpm were the gofer, they’d buy the gift as soon as they’ve read the wish, and designated the spot in the room as soon as they placed the order. They may not even have the physical gift yet to already be able to mark the space in the room where the gift will stand. This happens for each gift separately and independently from others. By design, pnpm doesn’t have the blocking stages of installation - the processes run for each of the packages independently.

pnpm is disc space efficient

Let’s talk about food. I love veggie sushi. I order a lot of veggie sushi. I also eat a lot of wasabi and I don’t like waste. Even though it would be fair to assume that such portions surely are to be shared, I usually order just for myself. My order always arrives with two pairs of chopsticks even though I say on the phone that I don’t need them - I have metal ones at home. I wouldn’t just throw unused chopsticks away and now I have a drawer full of them, and you can also see them laying around in the most surprising places in the kitchen. Similarly, I have my own wasabi tube but would I throw away those little sachets? Never. They chill on the shelf in my fridge in an ever-expanding fashion.

My chopsticks and wasabi could work as an analogy for how npm manages disc space. “Oh, you’ve already installed React two hundred times? I’m SURE you NEED the 201st copy!”

To my relief, pnpm checks what is already available on your disk and only adds what you additionally need for your project to run. All the dependencies are located in one global location (for instance, ~/.pnpm-store/ - you can check it by running pnpm store path in the terminal) called “a content-addressable store”. In your project’s node_modules folder there is a .pnpm file that contains the “virtual store” with many so-called “hard links”. it creates one hard link for each file of each package. I like this explanation that pnpm docs offer:

So, for example, if you have foo in your project as a dependency and it occupies 1MB of space, then it will look like it occupies 1MB of space in the project's node_modules folder and the same amount of space in the global store. However, that 1MB is the same space on the disk addressed from two different locations. So in total foo occupies 1MB, not 2MB.

This makes the node_modules folder more like a portal (like the wardrobe in Narnia) to files located in different nooks of the global store, and not like a bloated storage unit. This diagram illustrates the strategy employed by pnpm:

But, does pnpm “know” if there are duplicate files between two versions of the same package? Yes, because, just like git, it identifies the files by a hash id (called also “content integrity” or “checksum”) and not by the filename. This means that two same files will have identical hash id and pnpm will determine that there’s no reason for duplication.

pnpm has got your back

There are so many ways that pnpm looks out for you. One of them is that it is impossible to invite silly bugs by trying to use modules that are not directly specified in the project's package.json but, for instance, are required by your project’s dependencies. While it’s not the end of the world if everything works well, what happens if your project’s dependency no longer requires one of the packages and as a result it is no longer available in the the node_modules folder? Well, everything crashes and you don’t even know why.

Such bugs may happen in npm and Yarn Classic because of the flat node_modules directory because during installation, they hoist (elevate) all of the packages to the node_modules , regardless of whether they are directly required by your app or not. As a result, your project gets access to distant “relatives” of its dependencies.

…but what if my project doesn’t support symbolic links?

And even if your project presents an edge case and doesn’t work well with symbolic linking, worry not! You can still use pnpm but set it to an npm-like mode. While it will not be space-efficient, it will still be faster.

But… what about Yarn?

There are two editions of Yarn:

“Yarn Classic”, which comprises versions below v2 and is no longer maintained,
“Yarn Berry”, which is v2 and higher.

Yarn Classic operates similarly to npm with regards to managing the node_modules folder. Yarn Berry, on the other hand, offers three solutions:

the npm-like mode
the “Plug'n'Play” mode
the pnpm-like mode, which uses hard links to reduce the disc space (not available by default and not thoroughly documented)

The Plug’n’Play seems like an intriguing option but it has been met with the community’s apprehension as the .zip files are not so easily accessible during the dev workflow. The earlier-mentioned post by TakeShape explains some of the other challenges.

Closing remarks

Considering how space-efficient and fast pnpm is, it is not a surprise that more and more projects make a move towards it and that its popularity is rapidly growing. This year, the weekly download rate for pnpm was seven times as high as last year!

If you’re interested in learning about why projects choose to switch to pnpm, check out these posts by Gestalt and TakeShape.

Come work with us on package managers!

While you’re here, an announcement: we are looking for a new team member who will work on package managers including pnpm, npm, yarn, and our own Turbo. Interested? Apply today or reach out to me on Twitter!

Hacktoberfest: docs updates in three steps

Sylwia Vargas — Wed, 12 Oct 2022 14:57:36 +0000

Hacktoberfest is a celebration of Open Source contributions and new developers are eagerly encouraged to join in. However, many projects require so much setup, which not only stressed me out but also discouraged me from considering contributing.

Of course, there is the Markdown Preview on GitHub - however, it only changes Markdown to HTML and doesn't render the whole page. You can't see how the changes you're introducing will look on the page unless you close, install dependencies, start the dev server.

But what if updating docs wasn't this complicated? Well, good news is that it doesn't have to be!

Below, I'm offering a workflow that helps you focus on the content update or creation, without the hassle of the setup. If you prefer to watch a video instead, I got you covered as well - check the end of the post!

Content editing hack

Follow these three steps for a productivity hack:

In the docs repository, find the file you want to edit - can be markdown, can be anything, really. Click "edit" icon.

In the browser's address tab, add "pr.new" to the beginning of the URL address, press enter, and enjoy updating the content.

Once you're happy with the result, propose the changes and submit a PR!

What is this tool?

What you see above is a tool called Web Publisher. It is - and will remain - free for Open Source projects and for public repositories.

Web Publisher allows you to edit any file and see the changes rendered on the live preview. You don't have to do any special setup - you can even use it on your phone (on Chrome or Firefox). Now even the new developers can contribute their talents to Open Source by proofreading, translating, editing the text without the stress of the environment setup.

I have been using Web Publisher for weeks now and it really helped me a lot in my daily workflow. The other day I corrected a dead link on our page while waiting in line for an appointment 😂

Wanna try it out?

You can either edit a page of any of the public repos or... Add your name to this page!

See it in action

In case you'd like to see how I use the tool, see this video:

At StackBlitz, we love our community. Don't be a stranger - join our Discord or reach out to me on Twitter with your StackBlitz ideas, dreams, and wishes!

One-click treat for Open Source maintainers

Sylwia Vargas — Tue, 11 Oct 2022 18:18:21 +0000

It's October. For some, a month of beautiful foliage, Halloween, and pumpkin spice. Some Open Source maintainers live through their own version of Halloween - the stream of PRs coming their way to celebrate Hacktoberfest🎃

At ViteConf earlier today, StackBlitz has just unveiled Codeflow. It is a tool that I have been using for the past few weeks to build our docs site, to quickly fix typos, and to review PRs. Codeflow is and will be free for Open Source projects and for public repos, just like other StackBlitz tools.

So, what's the charm? One click and the entire IDE for the PR boots up inside your browser. No more cloning, committing, running dependencies, stashing your local changes.

If you prefer to watch people do things instead of reading about it, let me show you how I reviewed a PR from my colleague today:

One-click solution to PR reviews

CodeflowApp bot is a part of StackBlitz Codeflow. Once you install it on your account, organization, or a single repository, it will detect PRs (and issues containing bug reproductions in StackBlitz) and immediately spin up an entire IDE so you can explore and review it.

Don't believe it? See it yourself on this pull request I prepared for you.

NOTE: you need to have a StackBlitz account.

But what is Codeflow?

Codeflow is a one-click integration with GitHub for seamless coding workflows.

StackBlitz now helps you switch contexts seamlessly:

Codeflow IDE is a fully fledged desktop-grade dev environment capable of running your whole workflow - from production-level dev work to reviewing pull requests.
Web Publisher is a publishing tool that makes editing docs or blogs pleasant thanks to a realtime preview of the changes.
pr.new is a short URL that can be used on any repository to explore code, a branch or an issue, to review a PR, or even to edit a file, all in a live environment with a realtime preview.
CodeflowApp is a friendly bot, which provides a one-click link that spins up the whole environment for pull requests and issues. No more context-switching or branch-checkouts, just a new browser tab with a full IDE and a dev server running.

At StackBlitz, we love our community. Don't be a stranger - join our Discord or reach out to me on Twitter with your StackBlitz ideas, dreams, and wishes!

Explain like I’m five: package managers

Sylwia Vargas — Sun, 09 Oct 2022 17:01:44 +0000

For a long time, the subject of package managers was for me an intimidating one. As a bootcamp grad, I knew enough to build a website or set up a database but any configuration issues stressed me out. Every now and then, I'll encounter a discussion on which package manager is the superior one and I'd immediately feel humbled by all the things I still don't know about web dev.

Now that at StackBlitz we are about to make a big announcement regarding package managers, I thought I'd revisit this topic and take an “Explain like I’m five” route 💕

Let’s look into what a Node module, a package, and a package manager are.

Metaphor: LEGO sets

I have recently visited a friend who loves LEGO. His whole house looks like an exhibition hall with colorful sets displayed in the most surprising places. There’s a flower standing on a bathroom mirror shelf, a dragon guarding the stairs, a TV series set near the kitchen table, and a wizardry school near his TV stand. I’ve noticed that in most cases one big creation was, in fact, a collection of smaller sets.

Imagine that the massive wizardry school castle is your app and for it to be complete, you need to add a bunch of extras like a train station, a magic tree, or a phoenix. Each of them is a separate entity that you can move around as you wish.

Node modules and packages

If the wizardry school castle is your app, then each of the extras would be a Node module or a package.

So your castle needs a magic tree. You could design it yourself but it is way easier get the whole set designed and packaged together in a box with an instruction. A ready-made set in this analogy is a “package” and the instruction is the “package.json” file.

A Node module, or a package, is a piece of code that helps you add some functionality to your app. The package.json file tells us who created the package, when it was created, what other “dependencies” it needs to function, and so on.

To satisfy the folks who like to be very precise, please note that the npm docs page makes a distinction between "Node modules" and "packages. However, in most cases “Node module” and “package” are used interchangeably.

Package managers

Now, if you only have one or two LEGO sets in your home, managing an inventory of all the blocks, sets, and figurines you have would not be that difficult. But what if you are a lifelong collector, pieces break often, there are new sets constantly coming out, and the available color scheme changes faster than the company manages to announce it? Well, ideally you would find a system to keep track of all the pieces you own, need to replace, no longer need, and that that you need to buy?

This is why in software there is a need for the so-called “package managers”. A package manager is a tool that helps you keep track of all dependencies of your app in a consistent way. It automates the tasks of dependency installation, upgrading, configuration, and removal. It makes sure that all the packages that your app needs are installed, in the correct version, or up-to-date.

tl;dr package managers automate all the tasks regarding dependencies, saving your precious time 💕

We will share big news regarding package managers at the upcoming ViteConf. Grab your free ticket and tune in this week (Oct 12-13) if you're curious 👀

StackBlitz Monthly Update Post: September 2022

Sylwia Vargas — Thu, 06 Oct 2022 23:04:47 +0000

Hello everyone!

At StackBlitz, we have been working hard on the new tools that will premiere at ViteConf and that I believe will be truly game-changing for Open Source maintainers.

The release will happen before a significant crowd: at this moment, 21,000 devs registered to attend ViteConf! If you haven’t yet, click here to get your free ticket. Here's mine:

Meanwhile, we have been pushing updates and cheering for our community!

Today we are bringing you:

StackBlitz in the wild: news from our community
DX & Compatibility improvements: the change log
New Teammates: our team has grown - do you wanna join, too?
Quick bites for the road

I hope you’ll have a wonderful month!

StackBlitz in the wild

Does running WordPress in the browser without a PHP server sound like fun? Here’s a delightful writeup on spinning it on StackBlitz thanks to WASM. We may soon be able to run an in-browser WordPress IDE - something to look forward to!
Wanda has just released a beautiful Design System components playground powered by StackBlitz - have a look:
tRPC, a library used for building fully typesafe APIs, added a StackBlitz demo on their homepage and we love to see it!
Matt Pocock is soon releasing Total TypeScript, a comprehensive course on TypeScript, which uses StackBlitz to help users learn as they watch the videos without leaving the browser.
Token CSS is a new tool that seamlessly integrates Design Tokens into your development workflow - you can launch a Token CSS demo with a single click!
Josh Morony created a bash script to instantly launch a fresh Angular app for test code, which publishes the app on StackBlitz 🚀 // Detect dark theme var iframe = document.getElementById('tweet-1567483037906259976-489'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1567483037906259976&theme=dark" }
neodrag has just dropped. It is a lightweight plain JS library that makes your elements draggable. You can play with it in this demo.
And finally, oh so many StackBlitz starters to choose from in the whyframe docs! whyframe is a library that “gives iframes superpowers, making it easy to render anything in isolation”.

DX & compatibility improvements

As we’re working on new tools, we are finishing the month with a whole plethora of improvements:

Bug fixes and improvements
- Updated our dashboard with new project starters: effector, NativeScript (mobile section), Quasar, Remotion, VuePress, VitePress.
- Our dashboard now features a description of each of the projects instead of the project slug.
- New repositories now have their name as a header in the README instead of the project slug.
- Introduced improvements to the Console components (for EngineBlock projects):
- Added support for console.table
- Fixed the display of NaN, Infinity and -Infinity values
- Improved the color contrast and readability of console log items.
- Improved the responsiveness of the Console when logging over hundreds of messages.
- Improved StackBlitz EngineBlock error messaging when a library referenced from the code has an entry point file that does not exist in the package.
- Fixed issue in the EngineBlock projects with importing ace-builds package due to the way that it handles AMD and the way that it specifies a webpack file-loader in its webpack-loader.js file.
- The ace editor library entry point overrides the require function with its own custom logic (amd loader).
- Fixed a bug Ctrl+S / Cmd+S not saving project if all tabs are closed.
- Fixed an issue where a standalone preview tab would get stuck in the loading state if the StackBlitz project had some sort of error. These errors are now shown to the user.
- When reloading the page, now the previously displayed screen is copied and displayed for up to 3 seconds.
- Updated our icon set to use Microsoft’s Codeicons in more places.
- Fixed an issue with the dashboard's project list not showing the “Delete this project” menu entry.
- Now user can open the commit dialog in the auto commit mode by alt+click on the commit button (GitHub Issue #2034 fix).
- Clarified the UI shown when opening a binary file in the editor.
- Binaries are now preserved when you connect git repository to a project with binaries (Github Issue #787 fix).
- Fixed the storage quota exceeded error, by using indexeddb instead of localstorage.
- Clarified context for options in the user menu.
- Now user can use git integration with GitHub "Keep my email addresses private" setting on (GitHub Issue #1963 fix).
Blog
- Replaced the generic social image and added the new social image for category index pages.
- A specific title to each Category page, instead of showing generic StackBlitz::Blog title.
- Implemented the new Netlify build image (the new Ubuntu 20.04 images).
Docs
- Created and added numerous demos, like for example this project that showcases all open file options.
- Updated the list of the custom project starters with these friends:
- analogjs.org/new
- ladle.dev/new
- NativeScript:
  - TypeScript: nativescript.new
  - JavaScript: nativescript.new/javascript
  - Angular: nativescript.new/angular
  - Vue: nativescript.new/vue
  - React: nativescript.new/react
  - Svelte: nativescript.new/svelte
- node.new/nx
- qwik.new
- rakkasjs.org/new
- storybook.new
- vitepress.new
- Refreshed content across the docs platform.

New Teammates

Our team has grown! Please meet:

Yuka Masuda, a front-end engineer, a community organizer, a talented knitter, an inspiring baker, and a loving human to a shiba inu 🐕
Dave McChesney, an Enterprise Account Executive, a skilled archer, a true enthusiast of the outdoors, and a pet parent to two wonderful poodles 🐩

Are you looking for a ✨ nurturing ✨ team to join? We have roles open for software engineers (a senior Rails engineer and a senior TypeScript/Rust/Nodde.js engineer), as well as a Product Marketing Manager.

Here are three tweet threads on how it is to work with us:
// Detect dark theme var iframe = document.getElementById('tweet-1479224585899888640-172'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1479224585899888640&theme=dark" }

// Detect dark theme var iframe = document.getElementById('tweet-1493877657699819523-130'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1493877657699819523&theme=dark" }

// Detect dark theme var iframe = document.getElementById('tweet-1533109972518715393-515'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1533109972518715393&theme=dark" }

Wanna learn more? Feel free to reach out!

Quick bites for the road

Are you ready for ViteConf? Our team member, Patak, spoke recently to the Storyblok folks about some of the marvels awaiting you there!
Here’s an article predicting a looming end of the localhost, inspired by Swyx’ blog post.
Puru Vijay, our team member spoke recently at Svelte Summit on how to migrate React libraries to Svelte
Do you already have a ticket for ViteConf? Just in case, it’s an online and free conference with an all-star speaker list celebrating the web ecosystem at large. Excited? See you on 11-12 October 2022!

What’s next?

We are so excited for what the nearest future will bring — we are cooking something really special for you! In the meantime, stay tuned on Twitter or join our Discord community.