Forem: Michael Esteban

How to use @next/font globally

Michael Esteban — Mon, 20 Feb 2023 04:13:09 +0000

The @next/font package can download Google Fonts at build time and self-host them as a static asset. This is useful because no requests are made to Google by the client to request fonts used on the page.

However, it wasn't obvious to me how to apply the font globally. This can be achieved as follows.

app.js

import "@/styles/globals.css";

import { DM_Sans } from "@next/font/google";

const dm_sans = DM_Sans({
  display: "swap",
  subsets: ["latin"],
  weight: ["400", "500", "700"],
});

export default function App({ Component, pageProps }) {
  return (
    <>
      <style jsx global>{`
        :root {
          --dm-font: ${dm_sans.style.fontFamily};
        }
      `}</style>
      <Component {...pageProps} />
    </>
  );
}

And then use the CSS variable in your stylesheet.

global.css

body {
  font-family: var(--dm-font);
}

How to build an AI assistant with GPT-3 for your website

Michael Esteban — Tue, 07 Feb 2023 23:47:32 +0000

I've seen a lot of announcements recently for something peculiarly similar - an AI powered assistant that can answer questions about a product, library, framework, website or blog - you name it and it can provide answers about it.

I was curious. How were these AI assistants being built? Everyone knows about ChatGPT. But when I tried asking ChatGPT very specific questions about these products or libraries the response that I received didn’t match up. Intriguingly, these custom built assistants could respond with answers that included recent knowledge - things that had changed well after ChatGPT’s training cutoff date in 2021.

So I did what any sane person would do and spent several nights after work making my own AI assistant. This is a summary of everything I have learnt.

If you’re just interested in the final code, you can find it right here.

It starts with a clever prompt

Most of the time when you ask ChatGPT a question, it draws its response from its vast knowledge of possibly everything that was publicly available on the internet at the time the model was trained. However, it turns out that through a clever prompt we can ask ChatGPT to instead only draw its answer from a body of text that we provide. You can easily try this yourself through the ChatGPT interface.

Answer the question as truthfully as possible using the source text, and if the answer is not contained within the source text, say "I don't know".

Question: Which mountain did I climb on Monday?

Source text: On Monday I climbed Mount Yari. On Tuesday I rested. On Wednesday I travelled by train to Hiroshima.

You climbed Mount Yari on Monday

By using this prompt technique we can leverage ChatGPT’s language prowess without relying on its (at times dated) knowledge base. I wanted to try the same prompt but programmatically with Python. Interestingly, ChatGPT doesn’t have a public API at the time of writing. All the AI assistants we see being announced are instead using GPT-3. It’s perhaps a slight simplification - but we can think of ChatGPT as being built on top of GPT-3, and fine tuned heavily with conversational text. Hence why it is a better fit for chatting. OpenAI has several iterations of GPT-3. The most advanced publicly released model is called text-davinci-003, and it’s this one that I am going to be using to create my AI assistant.

After pip installing the openai package, I ended up with a small Python script.

import openai

openai.api_key = "XXXX"

response = openai.Completion.create(prompt="""Answer the question as truthfully as possible using the source text, and if the answer is not contained within the source text, say "I don't know".
Question: Which mountain did I climb on Monday?
Source text: On Monday I climbed Mount Yari. On Tuesday I rested. On Wednesday I travelled by train to Hiroshima.""",
model="text-davinci-003"
)

print(response["choices"][0]["text"])

# Answer: I climbed Mount Yari on Monday.

Great! This returns a very similar response aside from the I/You difference which can perhaps be attributed to ChatGTP’s superior conversational skills.

I then used Python to read a text file. It’s a draft article I’ve been working on. Perhaps I could ask GPT-3 some questions about what I have been writing.

This model's maximum context length is 4097 tokens, however you requested 17748 tokens.

😔 Building an AI assistant wasn’t going to be as easy as I thought.

The context window

GPT-3 has a limit on the context that it can handle. This is something that seems to significantly increase with each generation of GPT. GPT-4 is expected to at least double the current context window. For now, however, the maximum request limit is 4000 tokens. A token is approximately 4 characters. If we do some rough back of the napkin maths: 1000 tokens is ~750 words, and 4000 tokens therefore equates to a max input of ~3000 words.

Three thousand words is nothing to sneeze at. But what happens if want to ask questions about a source text that exceeds it. There’s been a number of clever solutions developed to get around the context window limitation. What I have learnt is that everyone’s approach is slightly different. Often the approach taken is highly influenced by the kind of source text being worked with (e.g. a book vs API documentation vs research papers). However, in most cases we can say that the process generally consists of three steps:

Divide the source text into chunks (parts)
Calculate which chunks are likely to contain the answer to the user’s question
Only send these specific chunks, along with the question, to GPT-3 as a prompt. This way the request token limit is not exceeded.

As a small aside — a completely alternative approach is to fine-tune GPT-3 so that the model itself learns new things. This is however quite expensive and it would need to be done every time the source text changes. So I personally haven’t seen too many examples of this.

Chunking

Chunking is the process of splitting the source text into parts. For our purposes, there isn’t necessarily a one-size-fits-all approach. The most rudimentary method would be to simply split the source text into blocks, of say, 1024 characters. However, the disadvantage of such a rudimentary approach is that it’s likely to split words in the middle.

text = "This is a sample text to illustrate the point."

# Split the text into chunks of 10 characters

['This is a', ' sample te', 'xt to ill', 'ustrate th', 'e point.']

This means that we lose the meaning of the text, which for reasons we are about to explore soon is not ideal. Instead, what I decided to do was to use the NLTK library, and in particular the sentence tokeniser method, to break the source text into sentences. This way each chunk contains complete sentences and the meaning of the text is better preserved.

import nltk

def chunk_text(text, chunk_size=1024):
    sentences = nltk.sent_tokenize(text)
    chunks = []
    chunk = ""
    for sentence in sentences:
        if len(chunk + " " + sentence) <= chunk_size:
            chunk = chunk + " " + sentence
        else:
            chunks.append(chunk)
            chunk = sentence
    chunks.append(chunk)
    return chunks

To reiterate - how you go about chunking the text is going to be highly dependent on what it is. For a research paper, you may consider chunking on paragraphs or sections. For poem/lyrics, you may want to chunk on each stanza.

Semantic search and embeddings

At this stage we have a list containing the chunked source text. We now need to decide which chunks are likely to contain the answer to the user’s question. What we might be tempted to do is a form of keyword-based search. If the question asked is:

What are some good movies about time travel?

We could find the chunks with the most frequent occurrences of the words movies, time and travel. However, this is unlikely to give us the desired response because the user isn’t interested in travel movies in the broad sense, they are only specifically interested in time travel movies.

Alternatively, it’s also possible that the user could ask a question that can be answered by the source text but they use a synonym in the question that doesn’t appear in the source text. For example, if the user asks:

What are some of the shortcomings of Python as a programming language?

Our source text may contain a very detailed section on the limitations of Python, but the word shortcomings is no where to be found.

So what we need instead of a keyword-based search is semantic search. We want to find chunks of the source text that match the meaning of what the user has asked. In order to do semantic search we first need to create an embedding for each chunk. An embedding is a numerical representation of words.

Imagine for a moment that we have a big box full of words. We want to give each word a number that tells us what it means and how it relates to other words. To do this, we can use a neural networked-based language model that has learnt how to map words to numerical representations. Then, whenever we want to know if two words are related, we can compare their numbers to know if they are similar or not.

To get a bit more technical - the chunked source text can be represented in a multi-dimensional vector space. We can calculate distances between the vectors to find the closest matches. We need to use a language model that can calculate the embeddings of text. There’s a few different options here but OpenAI itself has a popular model for this purpose called text-embedding-ada-002. It lets us send a string of text (up to ~8,000 words) and it turns it into a list of 1,536 floating point numbers.

import openai
from time import sleep

def get_embeddings(chunked_text):
  embeddings = []
  count = 0
  for chunk in chunked_text:
    count += 1
    embeddings.append(openai.Embedding.create(
        model="text-embedding-ada-002", input=chunk)["data"][0]["embedding"])
    if count % 30 == 0:
      sleep(60)

  return embeddings

You might have noticed the use of the sleep function here. I’ve included it because the API enforces a rate limit which is quite stringent if you’re not on a paid plan.

This what the embeddings look like for one chunk.

[-0.012661292217671871, -0.005988267716020346, -0.006203093566000462, 0.0012268563732504845, 0.018662987276911736, 0.02103949710726738, ...1530 more items]

The distance between two embeddings represents how semantically similar the text is to each other. This distance can be calculated using cosine similarity.

import numpy as np

def cosine_similarity(a, b):
    dot_product = np.dot(a, b)
    norm_a = np.linalg.norm(a)
    norm_b = np.linalg.norm(b)
    return dot_product / (norm_a * norm_b)

This is computationally intensive task, particularly if working with a lot of vectors. To be honest, the above function is probably just fine for our purposes. But, in the spirit of over engineering our AI assistant - we can use a library called Faiss written by the smart folks at Facebook (Meta) Research which specialises in comparing vectors. It uses an indexing structure to reduce the number of vectors that need to be compared, and also implements a number of different algorithms depending on the dimensionality of the data.

import faiss
import pickle
import numpy as np

index = faiss.IndexFlatL2(len(embeddings[0]))
index.add(np.array(embeddings))
pickle.dump(index, open('index.pickle', 'wb'))

I’m also taking advantage of Python’s pickle function as a quick way of storing the index to a file. The process of creating the embeddings can be slow and it also has a small associated cost, so it makes sense to store the embeddings somewhere if you’re planning on working again with the source text.

What would you like to know?

The final part to our AI assistant is the main loop. We ask the user to enter a question. Then, we get the embeddings for that question. Having obtained the question’s embeddings, we can now search for the most similar chunks. The faiss search method allows us to pass in the question’s embeddings and also the number of similar chunks to return. In the code below I am asking for the indices of the 4 closest chunks which will be sorted by increasing distance to the question's embeddings.

We construct the prompt with the relevant source text, send it to GPT-3 for completion and then print out its response.

import openai
import numpy as np

while True:
  question = input("Please ask your question: ")

  question_embedding = openai.Embedding.create(
    model="text-embedding-ada-002", input=question)
    ["data"][0]["embedding"]
  _, indices = embeddings.search(np.array([question_embedding]), 4)

  relevant_text = []

  for i in indices[0]:
      if i == -1:
        break
    relevant_text.append(chunked_text[i])

  relevant_text = "\n".join(relevant_text)

  answer = openai.Completion.create(
    prompt=f"""Answer the question as truthfully as possible using the source text, and if the answer is not contained within the source text, say "I don't know".
    Question: {question}
    Source text: {relevant_text}""",
  model="text-davinci-003"
  )

  print(answer["choices"][0]["text"].strip())

The example code in full can be found here.

Pricing

The OpenAI API is not free so I encourage you to check out their pricing page. When you sign up you receive a complimentary $18 worth of credit to begin with. Completions are significantly more expensive than calculating embeddings. I spent roughly $1.40 in the process of writing this article. The free credits are certainly enough to dip your toes in without handing over your credit card.

Shout outs

There’s a few things I would like to acknowledge as they helped solidify my understanding while working on this and perhaps they can do similarly for you.

Build a GitHub support bot with GPT3, LangChain, and Python by Pete Hunt.
LangChain. This was actually one of the first things I looked at as I had seen plenty praise for it. Personally, I found that it abstracted too much of what was going on for my liking. But, it is a cool library and I found the source code to be very readable.
How to implement Q&A against your documentation with GPT3, embeddings and Datasette by Simon Willison.

Client-side image compression with Supabase Storage

Michael Esteban — Mon, 12 Dec 2022 03:45:31 +0000

One of the projects I have been working on allows users to upload and share images. Super novel, I know. We are using Supabase for the backend, and the images are stored in Supabase Storage - an S3 compatible object store. I’ve used Supabase a few times now and found it overall to be a neat product (#notsponsored). However, this was my first time using the storage offering and we ran into an issue from the outset. Users were uploading photos from their phones with very large resolutions and consequently huge file sizes. This wasn’t good because we have limits on storage and bandwidth, plus the gallery view of the app was taking forever to load.

Unfortunately, Supabase does not currently support any kind of image optimization. It’s been commonly requested, and quite likely on their roadmap, but at least for now there wasn’t a setting I could quickly toggle to solve the problem. I’ve run into a similar issue before using Amazon S3, and in that project we used a lambda function to do image optimisation when an image was uploaded to the bucket. But this didn’t seem particularly straightforward with Supabase, so I went searching for a different solution.

After researching a couple of different options, what I landed on was a package called compressorjs written by Chen Fengyuan. This library performs compression and resizing of images in the browser. I was a bit skeptical at first, but it's a cool project and has good browser support. How Compressorjs works is that it uses the HTML5 canvas element to read the original image data and perform lossy transformations to compress and resize the image. There's a whole bunch of transformation options. For my project, what I have found is simply changing the maximum width of the image to 600px and slightly reducing the quality has greatly reduced the file sizes.

Here is how you might go about using compressorjs with Supabase:

import Compressor from "compressorjs";

document.getElementById("file").addEventListener("change", (e) => {
  const image = e.target.files[0];

  if (!image) {
    return;
  }

  new Compressor(image, {
    quality: 0.9,
    maxWidth: 600,

    success(compressedImage) {
      // compressedImage contains the Blob
      // Upload the compressed image to Supabase storage
      supabase.storage
        .from("images")
        .upload(`public/filename`, compressedImage);
    },
    error(err) {
      console.log(err.message);
    },
  });
});

Cover photo thanks to Tamanna Rumee

Next.js & MDX w/ code highlighting

Michael Esteban — Fri, 16 Jul 2021 00:48:59 +0000

A walkthrough on how to configure Next.js to render MDX files with code highlighting.

I use this as a quick way to write documentation for small projects. It can also be useful for creating nicely formatted PDFs with code highlighting built in.

Let's get started!

Using MDX

Assuming you have a fresh Next.js project - begin by installing the Next.js MDX package:

npm install @next/mdx @mdx-js/loader

In next.config.js, configure Next.js to handle MDX files:

// next.config.js
const withMDX = require('@next/mdx')({
    extension: /\.mdx?$/,
});

module.exports = withMDX({
    pageExtensions: ['js', 'jsx', 'mdx'],
});

This means we can now create a MDX file in the pages directory and Next will magically do the routing and rendering for us. Let's try it. In the pages folder, create a new onions.mdx file:

# Onions

Layers. Onions have layers. Ogres have layers. Onions have layers. You get it? We both have layers.

Oh, you both have layers. Oh. You know, not everybody like onions.

```js
const year = "2001";
const movie = "Shrek";

console.log(`${movie}, ${year}`);

Run the project with npm run dev and navigate to /onions to see the rendered MDX page:

Syntax highlighting

Currently our code block has no highlighting - let's change that!

Install the remark-prism package. Then update next.config.js to include the plugin:

// next.config.js
const withMDX = require("@next/mdx")({
  extension: /\.mdx?$/,
  options: {
    remarkPlugins: [require("remark-prism")],
  },
});

module.exports = withMDX({
  pageExtensions: ["js", "jsx", "mdx"],
});

Then find a Prism highlighting theme to use - it's essentially a CSS file. As a starting point, check out this prism-theme repository for a list of options.

I'm a fan of the Dracular theme. I will add it to the project by creating a _document.js file and linking to the stylesheet in the head:

import Document, { Html, Head, Main, NextScript } from "next/document";

class MyDocument extends Document {
  render() {
    return (
      <Html>
        <Head>
          <link
            rel="stylesheet"
            href="https://unpkg.com/dracula-prism/dist/css/dracula-prism.css"
          ></link>
        </Head>
        <body>
          <Main />
          <NextScript />
        </body>
      </Html>
    );
  }
}

export default MyDocument;

Restart the dev server. The code block should now be highlighted!

Styling with Water.css

As an added bonus, I like to add a few quick styles. Water.css provides a great default stylesheet and saves us having to write our own. This can be added to the _document.js file as a stylesheet:

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/water.css@2/out/dark.css"></link>

By default, the generated Next.js styles/globals.css file sets all padding and margin to zero. I also remove these rules to get some padding and margin back.

The end result should look something like:

🎉

Print as a PDF

Aside from hosting the project and sharing the URL - you can also print the page and set the destination as a PDF. This will produce a nicely formatted PDF file to share with others.

Example repository

Here's an example of everything covered above!

Thanks to Dee @ Copper and Wild on Unsplash for the cover photo.

Create React App Turns Four

Michael Esteban — Thu, 29 Oct 2020 03:40:36 +0000

Create React App v4 is live! This post is a quick overview of what has changed.

Creating & Migrating

A v4 project can be created with: npx create-react-app myapp . You still have time to grab a coffee while it installs - some things never change!

To update an existing project from 3.4.x to 4.0.0, use npm install --save --save-exact react-scripts@4.0.0 . If you run into issues, try deleting the node_modules folder and reinstall the project's dependencies with npm install.

Fast Refresh

Fast Refresh is arguably the headline feature in CRA v4. It was available in v3, hidden behind an experimental feature flag, but has now officially replaced React Hot Loader. Fast Refresh is noticeably quicker, automatically reloads when syntax and run time errors are resolved, and - drum roll please - preserves the state of functional components and hooks on reloads.

If you want to quickly try it out, go ahead and create a small counter component:

function App() {
  const [count, setCount] = useState(0);
  return (
        <>
          <button onClick={() => setCount((count) => count + 1)}>{count}</button>
          {/* <p>The state is preserved!</p> */}
        </>
  );
}

Click the button a few times, uncomment the paragraph element, save and watch as the count value stays the same. Super neat!

React v17

CRA v4 now uses React v17. This release famously has no new developer features. Things are changing under the hood though, in particular to event delegation and making React easier to upgrade.

One big advantage to v17 is when writing React components: the new JSX transform means that we no longer need to import React into components to use JSX 🎉 . Unless you are using TypeScript, in which case you still need to wait a little longer to take advantage of this.

Typescript v4

Speaking of which, support for the latest version of Typescript now comes out of the box. TypeScript v4 has no breaking changes, which should make upgrading a breeze, and brings with it some type changes and inference improvements.

Webvitals Support

One of the more intriguing additions to CRA v4 is the inclusion of Chrome's web-vitals library. This tiny library can provide performance metrics on your app such as first input delay and cumulative layout shift.

One of the first things you may notice is a new file within your project's src directory: reportWebVitals.js. This file is imported in index.js:

// If you want to start measuring performance in your app, pass a function
// to log results (for example: reportWebVitals(console.log))
// or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals
reportWebVitals();

In addition to simply logging out the metrics, they can also be passed to an analytics endpoint such as Google Analytics.

Jest v26 and ESLint 7

CRA v4 now comes with Jest v26. A current and ongoing goal for the Jest project is to reduce its package size, with Jest v26 being the first step towards that. ESLint has also been upgraded to v7, adding with it several new recommended rules such as no-dupe-else-if , no-import-assign and no-setter-return .

Goodbye Node 8

Node 8 reached end of life at the end of 2019, and with CRA v4 is no longer supported. If you're still using Node 8, it's time to look for an upgrade path!

Thanks as always to all the contributors for this release.

Top Level Await in Node

Michael Esteban — Sat, 15 Aug 2020 08:09:31 +0000

Node v14.8.0 was released this week and with it came the unlocking of a commonly requested feature that I am excited about!

// Detect dark theme var iframe = document.getElementById('tweet-1293229801570881537-218'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1293229801570881537&theme=dark" }

You may have run into the dreaded await is only valid in async function syntax error when trying to write code like this:

const res = await fetch("https://dev.to");

// SyntaxError: await is only valid in async function

As a workaround, it was common to see codebases use an immediately invoked function expression:

(async function() {
  const res = await fetch("https://dev.to");
}());

Ugly - but functional! Alternative options include transpiling with Babel or using the command line flag --harmony-top-level-await.

With v14.8.0, top level await has been unflagged and now just works. The only catch is that top level await is only supported in ES modules. This means either adding "type": "module" to your package.json file or renaming your .js file to .mjs.

If your project can work with v14.8.0, you can take advantage of this feature today. For everyone else, you will still need to await a while.

The Screen Wake Lock API

Michael Esteban — Thu, 23 Jul 2020 04:27:38 +0000

Chrome v84, released on 14th July, brought with it the Screen Wake Lock API - a way for developers to stop a device's screen from dimming and turning off. Previously this functionality was only possible by using a native app or a clever but hacky workaround such as NoSleep.js that continuously plays a hidden video on the webpage [UPDATE: NoSleep.js now includes support for the native API]. With the release of v84, an always on display can now be triggered natively in the browser.

Progressive Web Applications in particular stand to profit from this new API, as Microsoft and Google recently announced their commitment to forge ahead with improving the PWA experience. The types of use cases that could benefit include waiting to scan a boarding pass, presentation slides, recipe pages and e-books.

Requesting a screen wake lock is achieved through a single method call.

navigator.wakeLock.request('screen');

However, the device can refuse a wake lock request if, for example, the battery of the device is low and discharging, or the user has turned on some kind of power conservation mode. The request will also be refused if the browser does not support the Wake Lock API so it's worthwhile considering what browser compatibility you are comfortable with. As the request can be refused, wrapping it within a try/catch block is recommended.

try {
  navigator.wakeLock.request('screen');
  console.log('Screen Wake Lock is active');
} catch (err) {
  console.error(err);
}

There's nothing worse than a battery depleted device. It's therefore encouraged to either let the user manually turn off the wake lock, or to turn it off after a defined period. The W3C Editor's Draft also recommends ensuring that the user is aware that the wake lock is active on their device by displaying some form of unobtrusive notification.

In order to release the wake lock, we need to store the WakeLockSentinel object that is returned from the request and call the release method on it.

let wakeLock = null;

const requestWakeLock = async () => {
  try {
    wakeLock = await navigator.wakeLock.request('screen');
    console.log('Screen Wake Lock is active');
  } catch (err) {
    console.error(err);
  }
};

await requestWakeLock();

// Elsewhere, perhaps when a user clicks a button...
wakeLock.release();
wakeLock = null;

Alternatively, the wake lock could be released after a defined period by using a timeout.

// Release wake lock after 10 minutes
window.setTimeout(() => {
  wakeLock.release();
  wakeLock = null;
}, 600000);

Note that a screen wake lock will be automatically released when the user minimises the tab or window, or switches away from a tab or window where the screen wake lock is active.

If you are developing with React, you might want to try using my useWakeLock hook. Suggestions and contributions are welcome.

Further information on the Wake Lock API can be found in the W3C Editor's Draft and this web.dev article. If you have a cool use case for the screen wake lock API - share it with me in the comments below!

Thanks to Johannes Plenio for the cover photo from Unsplash.

Incrementally building the web

Michael Esteban — Fri, 10 Jul 2020 12:02:56 +0000

A few weeks ago I rewrote Junior Developer Jobs. It's a modest website that lists recent tweets advertising junior developer positions. If you're looking for a gig in these turbulent times, I hope it helps!

Little did I know that this small side project would heavily influence my mindset on the future of developing web apps. It's an exciting time to be a web developer. I want to share with you my predictions, as well as some practical tips on how to dip your toes into this what-is-old-is-new-again static world.

For a moment, think about how you would create a site like Junior Developer Jobs. There's two approaches that might come to mind:

Fetch a list of tweets from Twitter each time the site is accessed; or
Fetch a list of tweets from Twitter. Store them in a database. Retrieve the tweets from the database when the site is accessed. Periodically update the stored tweets.

What are the advantages of the first approach? We can be certain that the tweets shown are as recent as possible. Disadvantages? A request needs to be made to the Twitter API each time the site is accessed. This request takes some additional time. More worryingly, we run the risk of being rate limited if our site has a spike in views or is maliciously targeted.

So, which approach did I opt for? Initially I settled on the second. Then I tried a third approach, and my view on web development has never quite been the same.

A static resurgence

Static website frameworks have seen a remarkable surge in popularity. In the React ecosystem, arguably the forerunner in this space (and the one I am most familiar with) is Gatsby. It leverages React and a community driven ecosystem of plugins - packages that make it easy to integrate external data sources (e.g. APIs) with a Gatsby website. Gatsby sites need to be built before they are deployed. At compile time, the build system makes all the API calls necessary to transform every React page into a HTML document.

The benefits of static sites are well documented. They don't require complex servers, they can utilise distributed CDNs, the perceived performance is (in Gatsby's own words) blazing fast, and they make SEO a breeze compared to client-side JavaScript heavy Create React App-esque sites.

You'd be forgiven for thinking that static sites can do no wrong. That is, until you realise that you need the data on your site to be dynamic. At face value this seems problematic - each time the data is updated, the site needs to be rebuilt. It's therefore unsurprising that Gatsby is commonly associated with blogs, marketing sites and wikis. These are the kinds of websites where the content is updated infrequently and new content is typically added manually.

How can we use Gatsby as a dynamic, data driven website?

Hooks, actions & plugins

For Junior Developer Jobs to work with Gatsby, I needed to automate the process of fetching new tweets and rebuilding the site. I'll explain how I did this using Netlify Build Hooks and GitHub Actions. Similar services will work just as well too.

Netlify build hooks

Netlify's build hook is a uniquely generated URL that when triggered with a HTTP POST request rebuilds the site. You can generate a new build hook by clicking the Add build hook button at Settings > Build & Deploy > Continuous Deployment > Build Hooks.

If you want to test making a POST request, you can use Postman, Postwoman or cURL:

curl -X POST -d '{}' https://api.netlify.com/build_hooks/XXXXXXXXXXXXXXX

I didn't have a use for it in this project, but it's also possible to send a string in the payload of the POST request which can then be accessed in the build process via an environmental variable.

GitHub actions

Netlify's build hook lets us trigger a new build but we still need to automate and schedule when the hook should be triggered - that's where GitHub Actions comes in. GitHub Actions is essentially GitHub's solution for continuous integration and deployment. We will create a new workflow that uses cURL to make a POST request to the Netlify build hook URL every hour. If a scheduled event is not the best fit for your project, there are many other events to choose from.

One of the benefits of using GitHub Actions is that the workflow can be created in the same repository as the project. In the root of your project, create a new YAML file at .github/workflows/hourly-build.yml

name: Hourly build
on:
  schedule:
  - cron: '0 * * * *'
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Trigger Netlify build hook
      run: curl -X POST -d {} "https://api.netlify.com/build_hooks/${TOKEN}"
      env:
        TOKEN: ${{ secrets.NETLIFY_CRON_BUILD_HOOK }}

To briefly explain this workflow - the name of the workflow is Hourly build. It runs on a schedule. 0 * * * * is the cron syntax for run every hour. You can change this to a different interval if needed, and I've found Crontab Guru to be a useful tool to help with the syntax. The workflow runs a Ubuntu container that makes a cURL request to the Netlify build hook.

You might have also noticed that I removed the last part of the URL from the build hook that identifies the Netlify project and replaced it with an environmental token. This is because anyone who has access to the full URL can start a new build of your site. It's safer to store this identifier as a GitHub secret which is only accessible to collaborators of the repository. In a GitHub repository, you can add a new secret by going to Settings > Secrets. Name the secret NETLIFY_CRON_BUILD_HOOK. The value should be the extracted part of the Netlify build hook URL.

As we've included our workflow in the special .github/workflows folder, Github will recognise it automatically. You should see the workflow listed under the Actions tab in the repository. With this workflow set up, our project will rebuild and deploy on Netlify every hour.

Netlify plugins

When Gatsby builds a site, it generates assets and places them in a public folder. The public folder, along with a cache folder, are used to keep track of the assets that make up the site. By default, Netlify does not keep the Gatsby cache. This means that each build process starts from scratch.

We can use a Netlify build plugin by Jason Lengstorf to persist the Gatsby cache between builds. This can significantly reduce the build time and is incredibly easy to do - it's a simple one-click install through the UI.

Building on a budget

How far can we get on a shoestring budget for a hobby project?

Netlify can host the website and also offers 300 build minutes per month for free. If we assume a site takes one minute to build (we'll come back to whether this is a realistic assumption), Netlify can build the site 300 times a month or roughly 10 times a day. If you're willing to go all in on Gatsby - their own offering, Gatsby Cloud, offers a sizeable bump to 25 builds a day. Reassuringly, both Netlify and Gatsby Cloud don't require credit card registration for their free tiers.

GitHub Actions offers a generous 2,000 minutes per month for private repositories and free usage for public repositories. In my own experience, I've found that making a cURL request typically takes around 2 seconds. That's a staggering ~1,296,000 requests per month for private repositories, dwarfing the number of monthly requests that we would actually ever make.

In short, budget is no barrier to entry.

A look to the future

I'm bullish that frequently rebuilt websites will continue to grow in popularity and be a better fit for a wider range of use cases. However, I'm also not oblivious to the challenges. Unlike Junior Developer Jobs, most applications will need a database or headless CMS. This will require some additional initial set up, but generally they can be treated as just another data source that the Gatsby build process draws upon. There are already plugins for popular options such as PostgreSQL and Wordpress.

Nonetheless, there are some things that simply cannot be generated at build time. For example, the options selected on a settings page is something that is unique to the user that is currently logged in. The typical approach to solving this issue is to statically render as much of the page as possible, and then hydrate the application state using a client-side fetch request. It's not as good as a pure static page, but the first initial render (which is often the bulk of the page) remains blazing fast. For certain use cases, such as real time applications, a different website architecture may be preferable. And you know what? That's fine too - I'm a strong believer in using the best tool for the job.

In the previous section I made the assumption that a Gatsby site takes around a minute to build. For a small site, with a few data sources, that's in the ball park. For larger sites, think of an e-commerce site with hundreds of product pages and images, I've heard of initial builds taking anywhere between 20 minutes to over an hour.

The Gatsby team is actively working on reducing build times. One of the main ways they are tackling this problem is to ensure that Gatsby only rebuilds the parts of a website that have actually been modified. This is commonly referred to as incremental or conditional page builds. For now, it's an experimental feature that you can try today through an environmental variable. If you're willing to go all in with Gatsby Cloud, the weather gods offer up to 20x faster build times and 10-1000x faster incremental builds.

I'm excited for the future when incremental builds for large websites might well be in the range of milliseconds. Infrastructure challenges that have long haunted web developers such as scaling, database replication and website stability may face a new kryptonite.

Go forth and incrementally build!

In the 2019 State of JavaScript survey, 35% of the respondents indicated that they had heard of Gatsby and would like to learn it. If you're in a similar position, there's never been a better time. Not only for blogs and personal portfolios, but dynamic incrementally built data driven websites too.