Forem: Matt Biilmann

Distributed Persistent Rendering: A new Jamstack approach for faster builds

Matt Biilmann — Fri, 14 May 2021 04:14:05 +0000

I’d like to share a new concept we’ve been exploring for the Jamstack at Netlify, born out of our work with large enterprise dev teams deploying hundreds of thousands of pages at a time.

We wanted a way to make those deployments faster—without introducing unwanted complexity for Jamstack developers. Some solutions have come up before now, but they tend to lock you into a specific framework or break some of the most compelling reasons to use Jamstack in the first place. A Jamstack-focused approach was needed that would work across the ecosystem. After discussion both within Netlify and some members of the Jamstack community, we feel we’ve landed the right concept—something we’ve dubbed “Distributed Persistent Rendering” or “DPR” for short.

This post is designed as an early introduction to the approach we’re proposing as well as an invitation for the entire community to offer feedback and collaboration. Our hope is to create a new standard that maintains the core principles and benefits of Jamstack and works across a wide variety of site generators and frameworks, bringing an even wider range of websites and use cases to the Jamstack.

The power of atomic deploys

We think the very best tools are built around a simple mental model that helps you easily reason about the state of your application as you build it. Modern frontend libraries (like React and Vue) make a simple, powerful contract with the developer: UI becomes a function of state. Change the state, and the UI will rerender in response.

The Jamstack has thrived because it too centers around an intuitive mental model: every git push runs a build process to create its own atomic deployment. This approach keeps it incredibly easy to reason about the current state of your site or application, even as many changes are committed daily from around your team. It makes both new deploys and rollbacks painless. It keeps you always confident in what any visitor will see at any given URL. And, most importantly, it avoids all the deep layers of caching, complexity, and infrastructure that came out of scaling the legacy web.

The fight against complexity

For any technology, the hardest part is not establishing simplicity, but protecting it over time.

As the Jamstack evolves, new features like dynamic server side rendering, rehydration, tiered CDN caching, and stale while revalidate seem to be creeping us back towards all the complexity we fought to escape. Can you still be confident in a rollback? Do you really know how your site will behave if you push deploy preview #110?

Of course the momentum to add features to a platform is the desire to solve real challenges. And one of the challenges of the atomic deployment model is the time it takes to rebuild an entire site each for each deploy. Larger teams and projects are feeling this impact, especially now that Jamstack sites are scaling to 100,000 pages and beyond.

Speeding up deployments... without weighing down the Jamstack

Netlify would like to work with the entire Jamstack community to help bring a solution for faster large site deployments—but importantly an approach that stays true to the atomic deployment model of the Jamstack. We’re calling our proposal Distributed Persistent Rendering, and today we’ve posted details about the approach in a Request For Comments (RFC). Our hope is that community members and site generator authors will weigh in and help us build something that is easily adoptable by multiple frameworks. Already, we’ve seen early engagement from Nuxt and 11ty and we hope many more will join the collaboration.

Distributed Persistent Rendering (or DPR) allows developers to defer rendering any given URL or asset until it’s first requested. With DPR, you can (and should) still prerender critical pages at build time—perhaps your homepage or recent blog posts. But that huge archive containing years of older posts? Using DPR, those pages can be rendered only when and if they are requested. And once rendered, they remain as a persistent asset until the next deploy—just as if they had been part of the initial build. It brings the benefit of faster builds without introducing the complexity of scaling and caching server-side rendering. And unlike a caching-based strategy, there’s no risk of confusion from stale assets or fallback pages.

We think DPR strikes the right balance—new capabilities without new complexities. We’re so excited we’ve even built a new service offering to help accelerate us towards DPR and we’re announcing today that any Netlify customer can gain early access to explore the approach and the simple API behind it.

Please let us know what you think in the RFC and show us what you build!

Open source parallel processing for Gatsby

Matt Biilmann — Tue, 25 Feb 2020 15:12:37 +0000

To help the greater Gatsby ecosystem shorten the time it takes from commit to deploy, today I've just submitted a freshly baked Pull Request as my first larger contribution to the Gatsby open source project.

The new gatsby-parallel-runner plugin builds on some of the existing work in the Gatsby project to allow both plugins and core parts of Gatsby to parallelize certain tasks by delegating the work to a large pool of serverless functions.

Gatsby Cloud pioneered this approach, and with this plugin I hope that we can open an even more generalized approach to parallelization. While it's in early development, my goal is that this can one day be made available on any CI/CD environment, and empower individual plugin developers in the ecosystem to build in support for parallelization when a task is well suited for it.

Gatsby and "External Jobs"

A bit under a month ago Ward Peeters from Gatsby's team landed a Pull Request to "enable external jobs with ipc". The idea behind the pull request is that instead of running gatsby build directly, an orchestrating parent process can fork gatsby build and make sure an environment variable called ENABLE_GATSBY_EXTERNAL_JOBS is set. When that is done certain jobs will be sent via node's IPC protocol to the parent process as "external jobs" in order to allow the parent orchestrater to efficiently parallelize the execution of them.

In itself nodejs is single threaded, so out of the box any CPU intensive jobs in Gatsby will only ever take advantage of one CPU core, but this ipc based delegation opens up the possibility of taking advantage of external worker processes for parallelization.

The only plugin that currently hooks into this is gatsby-plugin-sharp which is used for image transformation. At Netlify we've often seen image transformations be a huge source of build slowdowns for Gatsby sites, since doing lots of image processing within a single threaded build is really inefficient.

The open source Gatsby process has until now not offered any implementation of an orchestrator for Gatsby that can help you take advantage of external jobs, and that's where our new gatsby-parallel-runner helps.

Gatsby Parallel Runner

I've been having fun building out this new Gatsby plugin that acts as an alternative build command. Once installed in a project, you'll run gatsby-parallel-runner instead of gatsby build.

Out of the box it comes with a parallelized implementation of the Sharp image plugin based on Google Cloud Functions. It includes an easy script to get your own functions and queues setup in a Google Cloud project with just one command. It's built to be extensible so the community can easily add alternative implementations of the actual execution layer. Obvious candidates would be AWS Lambda functions or a nodejs cluster implementation. It should also be easy to use with new plugins that want to add external jobs outside of just image processing.

My hope is that this can help pave the way for more innovations in the ecosystem around build parallelization – and of course we're seeing a lot of opportunity in adding a more generalized form of this to Netlify's own build layer.

Our philosophy has always been to keep the build layer fundamentally open - our core build image has always been Open Source as is our new Build Plugin layer, and we've always believed that a healthy Open Source ecosystem in the build tool space is vital to the growth of the whole JAMstack category. So we're happy to contribute this project back to the Open Source community.

I ran a few benchmarks with the official Gatsby image benchmark repository on Netlify's build environment both with and without the gatsby-parallel-runner and was thrilled to see the gatsby-parallel-runnner consistently outperform the normal gatsby build command:

Running 3 times from a clear cache with gatsby build:

Run 1:
11:10:27 PM: success Generating image thumbnails - 351.218s - 3234/3234 9.21/s

Run 2:
1:45:43 PM: success Generating image thumbnails - 384.171s - 3234/3234 8.42/s

Run 3:
11:18:22 PM: success Generating image thumbnails - 322.853s - 3234/3234 10.02/s

Avg time for image generation: 352.747s

Running 6 times from a clear cache with gatsby-parallel-runner:

Run 1:
10:51:31 PM: success Generating image thumbnails - 158.438s - 3234/3234 20.41/s

Run 2:
5:33:33 PM: success Generating image thumbnails - 68.016s - 3234/3234 47.55/s

Run 3:
3:03:48 AM: success Generating image thumbnails - 75.731s - 3234/3234 42.70/s

Run 4:
10:54:47 PM: success Generating image thumbnails - 64.478s - 3234/3234 50.16/s

Run 5:
10:58:31 PM: success Generating image thumbnails - 66.021s - 3234/3234 48.98/s

Run 6:
11:01:58 PM: success Generating image thumbnails - 71.416s - 3234/3234 45.28/s


Avg time for image generation: 84.017s

The first run after deploying the functions was a bit slower than the subsequent runs, as Google worked on scaling up the number of concurrent function executions, but even then it was still more than twice as fast in the worst case as the standard Gatsby build command.

And on average - even with the initial outlier included - the parallel runner gave more than a 4.2x speedup over the single threaded runtime.

For curiosity I repeated the same benchmark on Gatsby Cloud:

Run 1:
03:02:38 AM: success Generating image thumbnails - 98.472s - 3234/3234 32.84/s

Run 2:
07:34:53 AM: success Generating image thumbnails - 328.141s - 3234/3234 9.86/s

Run 3:
22:19:50 PM: success Generating image thumbnails - 85.101s - 3234/3234 38.00/s

Run 4:
22:34:22 PM: success Generating image thumbnails - 134.721s - 3234/3234 24.01/s

Run 5:
23:02:37 PM: success Generating image thumbnails - 82.822s - 3234/3234 39.05/s

Run 6:
23:07:31 PM: success Generating image thumbnails - 60.532s - 3234/3234 53.43/s


Avg time for image generation: 131.631s

These tests had a lot more variability in build times than my Netlify based tests, and while the average was more than twice as fast as the single threaded build performance, the open source parallel runner performed significantly better in the tests I ran. So hopefully the Gatsby Cloud team can also benefit from looking into the source code behind this implementation.

Setting Up

Install in your gatsby project:

npm i gatsby-parallel-runner

To use with Google Cloud, set relevant env variables in your shell:

export GOOGLE_APPLICATION_CREDENTIALS=~/path/to/your/google-credentials.json

export TOPIC=parallel-runner-topic

Deploy the cloud function:

npx gatsby-parallel-runner deploy

Then run your Gatsby build with the parallel runner instead of the default gatsby build command.

npx gatsby-parallel-runner