Forem: Joe Kent

Remembering how to make a website without React

Joe Kent — Wed, 28 Dec 2022 19:34:51 +0000

Making a boring static website with modern frontend tooling is fun!

First, you install React & React DOM… actually no, you probably won’t do that. You likely wind up choosing between create-react-app, parcel, vite, or a full fledged framework like NextJS and Gatsby. Oh and if you need Typescript you’ll have to install and configure a bunch of more stuff. Then there’s styling, you can install a component library like styled-components, or use a CSS system like Tachyons. You still need to ingest content from somewhere, so you can also make a Contentful account. Since you’re doing server side rendering, you’ll probably want a host like Netlify or Vercel that supports serverless functions. I think we’re done configuring our new project now? There isn’t anything on the page yet but the size of the build bundle you just created would probably indicate otherwise.

Don't get me confused, I'm not a hater of the modern frontend stack. If you’re building a full fledged application, all of this makes sense. But I’m just trying to refresh my personal website, which has 2 templates, 5 pages, and 2 interactive buttons. This site probably won’t get much bigger or more complicated than that in the next 5-10 years (or, ever…). So I decided to go old school and render some html templates using content stored in the file system. This blog is a quick run through of how I did it, and I’m sorry in advance that this project has serious “Gen-Z rediscovers what record players are” vibes.

First, I setup mustache (npm package) as the templating language (the npm package hasn’t been updated in two years, so that’s great). Mustache is really simple, it provides a few helper utilities, like replacing variables, or iterating over data,

<ul>
  {{#links}}
    <li><a href="{{href}}">{{label}}</a></li> 
  {{/links}}
</ul>

{
  "links": [
    { "href": "itsjoekent.substack.com", "label": "Coder Brain" }
  ]
}

You can also define custom functions (“lambdas”) in your Javascript code that are then accessible in your Mustache template. For example, I created a lambda to inject the proper url prefix for all of my media assets,

function mediaLambda() {
  function _mediaLambda(mediaFile, render) {
    return `https://cdn.joekent.nyc/${render(mediaFile)}`;
  }

  return _mediaLambda;
}

Next, I setup a markdown parser, markdown-it. I then created another Mustache lambda for converting markdown to HTML and injecting it in the template output. I also ran the Markdown string through the Mustache parser, so I could use my “media” lambda inside markdown. This is where things got gross in the code.

Unfortunately, the mustache package I’m using seems bugged, because when I call a Lambda with a parameter that is a variable, and the variable value calls more Lambda’s (eg: the media lambda being used in my markdown text), it doesn’t do recursive replacement.

<div class="markdown"
  {{#markdown}}{{blogContent}}{{/markdown}}
</div>

{
  "blogContent": "The media URL is {{#media}}example.png{{/media}}"
}

// The markdown function outputs, "The media URL is {{#media}}example.png{{/media}}"

Oh and to make everything worse, I decided to use TOML (npm package, last published 4 years ago) to store my content because TOML is prettier than JSON and YAML (big brain decision making), and the TOML package I’m using is replacing forward slashes in multi-line strings with their HTML entity hex code.

# TOML
myMarkdownVariable = """
{{#media}}example.png{{/media}} // input
"""

# TOML node package output
{{#media}}example.png{{&#x2f;media}} // output

This obviously does not work with the Mustache parser. So I had to uninstall another package to decode the HTML entities, and then deal with the original recursive replacement problem. Here’s what the final code looks like:

// 'markup' is the markdown string from the TOML file
// 'render' is the mustache renderer
markdown.render(render(he.decode(render(he.decode(markup)))), {
  html: true,
  linkify: true,
});

Thankfully that is the only really gross code in this entire project.

From here, everything was really straight forward. In local development, I run an express server that handles serving the frontend bundle (generated by webpack, intentionally boring choice on my part), and matching the requested path name to a TOML file in the content directory. Once the file is matched, it can pull the HTML template required for the content type and run the HTML through the Mustache engine with the content data, and return the HTML output to the client.

For deploying to production, I just repeat the above process for every content file and instead write the resulting HTML to the build folder.

// Development
app.use('/dist', express.static('www/dist'));
app.get('/*', async (request, response) => {
  try {
    const content = await readAllContent();
    const match = content.find(
      (compare) => compare.path === trimLastSlash(request.path)
    );

    if (!match) {
      return response.status(404).send('not found');
    }

    const templateHtml = await renderTemplate(match);
    const html = await renderTemplate({
      ...match,
      template: 'index',
      templateHtml,
    });

    response.set('content-type', 'text/html').send(html);
  } catch (error) {
    console.error(error);
    response.status(500).send('server error');
  }
});

// Production
const allContent = await readAllContent();

for (const content of allContent) {
  const templateHtml = await renderTemplate(content);
  const html = await renderTemplate({
    ...content,
    template: 'index',
    templateHtml,
  });

  const indexFolderPath = path.join(process.cwd(), 'www', content.path);
  await fs.mkdir(indexFolderPath, { recursive: true });
  await fs.writeFile(path.join(indexFolderPath, 'index.html'), html);
}

You can find the full source code for this here.

Going into this project, I wrote down 3 goals,

Write HTML templates, CSS files, and JavaScript that uses native DOM API's.
Store my content in the repo, media assets in S3 compatible storage.
Deploy static files to a CDN.

And I think I succeeded on all fronts (for #3, I am using Cloudflare Pages).

But here’s the million dollar question… Do I regret rolling this custom file system based CMS & template engine, instead of just using React & some framework like NextJS, attached to a real CMS like Contentful?

…Not really. This was more fun than googling some opaque NextJS error for an hour. And this tiny website doesn’t need anything that fancy anyway. In hindsight, the only things I would do differently,

Use a templating engine that doesn't require my ugly hack (or just make my own).
Ditch TOML in favor of YAML or JSON 5.

Oh and you find my updated website here. Hope you enjoyed this journey back in time!

Super simple file based "CMS" for NextJs projects

Joe Kent — Fri, 24 Apr 2020 19:31:34 +0000

Tell me if you've been here before.

You're coding up a new personal blog or a tiny website for a friend using a React framework like NextJs, because it's a quick way to render an entire static website built with React. But you don't want to deal with setting up a CMS or writing an integration for Contentful and setting up proper content models. You just need some key/values and maybe Markdown support.

File system to the rescue!

Below is a quick guide to loading arbitrary key/value pairs from .txt files at build time, and injecting them into your React components.

Much of this will reference specific NextJS features, but the principles are applicable to any React application (checkout babel-plugin-preval for example).

First, create a content directory in the root of your project and make some .txt files that look like this,

@hello
World

@key
Value value value!

*Tap mic*, is this still working?

Then create a src/content.js file to store all of your content helper functions.

To start, you'll need a function that can read the file and parse it correctly. There is certainly a lot one could expand upon here (comments, inline variables, ...), but this was enough for me.

export function parseContent(text) {
  const content = {};

  const lines = text.split('\n');

  let target = null;

  lines.forEach((line) => {
    if (line.startsWith('@')) {
      target = line.replace('@', '').trim();
      content[target] = '';
    } else if (!!target && typeof content[target] === 'string') {
      if (!!line.trim().length) {
        if (!!content[target].length) {
          content[target] += '\n';
        }

        content[target] += line;
      }
    }
  });

  return content;
}

export async function loadContentFile(fs, path, file) {
  const fsPromises = fs.promises;

  const filePath = path.join(process.cwd(), 'content', `${file}.txt`);
  const fileContents = await fsPromises.readFile(filePath, 'utf8');

  return parseContent(fileContents);
}

Depending on how you architect your site content, you might need multiple files per page for global components like a navigation bar or footer. In which case, it would be helpful to also have a wrapper function for loading many files,

export async function loadManyContentFiles(fs, path, files) {
  const result = await Promise.all(files.map((file) => loadContentFile(fs, path, file)));

  return result.reduce((acc, data) => ({ ...acc, ...data }), {});
}

This next part will vary depending on how your application is structured, but the general principle is that you'll want to call these content loading functions from the root component of your page that is being server side rendered.

Here is an example with NextJS,

import { loadManyContentFiles } from '../content';

export async function getStaticProps(context) {
  const content = await loadManyContentFiles(fs, path, [
    'pages/demo',
    'components/footer',
  ]);

  return {
    props: {
      content,
    },
  }
}

export default function Homepage(props) {
  const { content } = props;

  const {
    hello,
    key,
  } = content;

  ...

But now we have a new problem, how do we get content to the child components?

Simple, use React context!

In our content helper file, setup a reusable context provider and custom hook function that child components can use.

import { createContext, useContext } from 'react';

export const ContentContext = createContext({});

export function useContent() {
  const content = useContext(ContentContext);

  return content;
}

Then in your page you need to setup the provider,

import { ContentContext, loadManyContentFiles } from '../content';

export async function getStaticProps(context) {
  ...
}

export default function Homepage(props) {
  const { content } = props;

  const {
    hello,
    key,
  } = content;

  return (
    <ContentContext.Provider value={content}>
    ...

Now within any child component, you can reference the useContent hook and pull in key/value pairs.

import { useContent } from '../content';

export default function Footer() {
  const content = useContent();

  const {
    footerLists,
  } = content;

And now you're fully setup with a file system based "CMS"!

Here are a few additional tricks you can use to get more out of this,

Dynamic lists

Want to make a list? Or better yet, want to tie multiple key/value pairs together in a list? No problem!

@post1Title
...

@post1Description
...

@post2Title
...

@post2Description
...

@post3Title
...

@post3Description
...

@posts
1,2,3

function mapPostList(posts, content) {
  return posts.split(',').map((key) => ({
    title: content[`post${key}Title`],
    description: content[`post${key}Description`],
  }));
}

Markdown

Plaintext is cool, but have you ever tried Markdown?

There are a lot of great Markdown parsers for Javascript (eg: marked, markdown-it) and React (eg: marksy). And with this "CMS", you can use any of them! Just write Markdown as the value, and pass it into a parser.

My one flag is that because of how the functions load in content, I had to write this small helper function to make sure line breaks were correctly applied when using Marksy.

  const finalContent = (pageContent || '')
    .split('\n').map((line) => `${line}\n\n`).join('\n');

Load a directory to create a list of paths

When generating a static site, you need a list of paths to build for. To do that, just create another helper function in your content.js file,

export function loadAllPagePaths(fs, path) {
  const postsDirectory = path.join(process.cwd(), 'content/pages');
  const filenames = fs.readdirSync(postsDirectory);

  return filenames.map((name) => `${name.replace('.txt', '')}`);
}

If you're using NextJs, you can then toss this into the static paths function,

export async function getStaticPaths() {
  const pages = loadAllPagePaths(fs, path);

  return {
    paths: pages.map((page) => ({ params: { slug: [page] } })),
    fallback: false,
  };
}

I hope the internet finds this helpful and let me know if you end up using this for your own project!

Server Side Rendering React In Realtime Without Melting Your Servers

Joe Kent — Mon, 16 Mar 2020 19:25:52 +0000

This post is a dive into how ElizabethWarren.com was eventually able to re-render content updates from a headless CMS on the fly, cache everything, and serve the entire website from an S3 bucket. The post presumes a baseline understanding of web technologies such as React.

I wanted to share this post largely because throughout the campaign I was quite frustrated by the lack of content online (that I could find) about scaling React server side rendering in production, and I hope some of the ideas in this post help someone one day.

Related note: I think this post would be an interesting talk to give and if you know of a great conference looking for proposals get in touch!

It's quite fascinating how the web has evolved from just being static files sitting on web servers, to monolithic web servers, to micro-services, and now the trend arriving back to deploying static files.

Static websites are wonderful from an operational perspective, they have virtually no cost compared to web servers and have the high uptime guarantee of the object storage provider you choose to use (most commonly AWS S3). A static site today just requires a build system that can build and push your website to object storage on command. If you want to get really fancy, you can even setup multiple buckets across regions or cloud providers for added redundancy. And if you need to maintain some light custom routing in conjunction with your static website, you can run code at the edge with a service like Cloudflare Workers or Lambda@Edge.

Often, the first hurdle in deploying your React application as a static website is server-side rendering all of the pages.

What is server-side rendering you ask? Rendering a React application within a NodeJS process is referred to as server side rendering (SSR), it's just a fancy way saying you want to generate all of the HTML on the page outside of a browser context. While not a requirement in every React project (for example, an internal dashboard would be sufficient with just client side rendering), server side rendering is necessary if you want your site visitors to see the content of the page immediately on load (eg: an article, or a landing page), or if you want Google to crawl your webpage.

React is fundamentally a UI library though, so you need to hook-up a few additional wires to server-side render your React components into static files. And there are many wonderful frameworks out there to choose from that make this very easy, such as NextJs and GatsbyJs.

But if your business requirements include being able to push out content updates as fast as humanly possible, you're going to run into a problem. Server side rendering an entire website is not an instantaneous process. And if your website is more than just a content site (eg: you have a million profile pages), exporting all of those user profiles as static pages with Next or Gatsby is not going to be a trivial problem to solve. So for the purposes of this post, let's just keep the scope to content websites.

There isn't really an average time for how long server side rendering takes, as it depends entirely on the components being rendered. But it's certainly possible a complex page could take well over 100ms to render. In the context of a static site, you can optimize this by having your build system divide work across multiple cores (eg: take a look at how Gatsby does this), but ultimately you're likely to run into another problem, the latency in fetching content.

Every web stack is different, but a common pattern in modern web development is using a headless cms. A headless cms simply means that the content management system storing all of your content is detached from the application(s) powering your interfaces, and content is fetched from the CMS via an API.

If you're using a headless cms, such as Contentful, your static build system can only render pages as quickly as it can fetch content from the CMS over the wire. In practice, this is adding a few hundred milliseconds before you can begin rendering a page.

A simple way to speed this up a little is utilizing pagination and requesting many pages at once. But if your website contains pages (or plans) with thousands of words, pagination starts to become problematic for both the network payload size and running out of memory in the NodeJS process.

Another way to reduce the time to get content is by caching these CMS responses in a database that the build system can access, but now you've just created an extremely "fun" cache invalidation problem to solve.

For example, imagine you had a content model for a blog post such as the following,

{
  "title": String,
  "publishedAt": Date,
  "content": String,
  "author": <Reference:Author>,
}

Every time the author changes, you'll need to invalidate the cache for every single blog post attributed to that author. And this is a simple one-to-many relationship, a sufficiently content-rich website will have content references that go several layers deep. And even if you put in all of that effort to maintain a tree of content relationships, the next time you go to rebuild the site, you're still looking at a significant latency hit having to refetch all of that content again.

But all of this in the grand scheme of things is a super unnecessary optimization conversation to have. For most teams, as long as you can revert a bad deploy quickly, the difference between rendering your site in one minute or five doesn't really matter. But on elizebthwarren.com, we had to carefully coordinate our site updates at the same pace as the rest of the campaign (AKA, everything had to go out ASAP, and often without any significant heads-up).

This meant for most of the campaign, the website architecture was fundamentally a cache layer sitting in-front of web servers that would always spit out the latest version of the website. Clear the cache and voilà!

Over the course of the campaign we still went through several architecture evolutions as increased traffic, technical requirements and web content continued to force our stack to scale. Here is a brief overview,

Launch (Feb. 2019): Wordpress backend, React client side rendering

Spring 2019: Started integrating Contentful with client side React components

June 2019: NodeJS backend on Heroku, converted Wordpress templates to Mustache templates, continued client side rendering existing React components

Mid summer 2019: Redis caching Contentful data

Late summer 2019: Server side rendering React components

Fall 2019: Storing server-side-rendered pages in Redis

Dec. 2019: Background workers doing SSR, moved assets to cdn.

Feb. 2020: Moved to fully static website.

Once we left Wordpress, the common theme was to put more things in Redis and add more servers when we experienced high traffic (eg: debates or other viral moments). While it "worked" for the most part, I didn't like constantly worrying things were going to fall apart at the worst possible moment.

The overall SSR strategy we implemented was nevertheless still a success for our requirements of updating things as soon as possible, and ultimately continued to be the backbone of how rendering the static website would work.

The premise is that we should never attempt to re-render the entire website at once, and instead use site traffic to trigger incremental re-renders if cached content is stale. At a high level, it looks like the following:

Keep a set of key value pairs for the "build version" and "content version".
If anything is published in the CMS, a webhook fires and the "content version" is incremented.
If the website is deployed, increment the build version.
If the last time a page was rendered was for an older build or content version, re-render the page and clear the cache.

The "content version" is kind of naive, as it leads to lots of unnecessary re-rendering, but it's 10x simpler than trying to use Contentful webhooks to maintain a consistent graph database of our CMS content references that would be required to do more selective re-rendering (as I explained earlier with the "author" reference problem).

During the winter of 2019, mostly in anticipation of Iowa and the rest of the primaries kicking off, we started a series of architecture improvements.

First, we moved all of the frontend assets to a cdn subdomain. This is common practice already in high-traffic websites, and it's one of those things that was on my to-do list forever but never made it to the sprint.

We did something interesting, however. Every deploy created a new, uniquely named and immutable folder in the cdn that all of the assets would go in. For example,

https://cdn.elizabethwarren.com/deploy/1cc2e8207789dc8c0a3f83486cae16a3cd3effa8b970f6306c1435c31014a560890f5236722af8d7ed3cfec76107508ffd82b2eb872b00e3ddf3f88012ead904/build/6.5d30e50ab08bb11f9cf8.js

This ensured that regardless of whether you saw a stale version of the site from your browser cache, or a stale version of the site that was served on our end, the asset would always exist, as it was originally deployed. As we get more into the server side rendering strategy we used, this point will become increasingly more important.

The other benefit of this unique folder name is it allowed us to safely apply a high max-age value to the cache-control header, ensuring your browser kept the file for quite a long time instead of re-asking for it the next time you visit. Using max-age on files that change content between deploys, but don't necessarily change filenames is a quick way to bury your users in very bad cache problems. Our webpack configuration hashed the names of our Javascript chunk files, but certain files do not have uniquely hashed file names (in particular, webpack manifest files). (*I should also note, certain files such as fonts that did not change between deploys were kept in a consistent place, and not redeployed under unique build folders).

Once we got all of the fonts, images, CSS and Javascript served by the CDN, the next step was to perform server-side rendering on background workers and store the html in Redis, making the web servers only responsible for serving HTML that was already put in Redis. The new SSR strategy looked like the following,

Keep a set of key value pairs for the "build version" and "content version".
If anything is published in the CMS, a webhook fires and the "content version" is incremented.
If the website is deployed, increment the build version and push build files to the CDN.
When a request comes in, web server immediately serves whatever page we have in Redis cache.**
If the page we served was stale, add an item to a Redis queue to notify a background worker the page needs to be re-rendered.
Background worker eventually re-renders the page, pushes the HTML to Redis cache and clears Cloudflare cache for that page.

** It's possible that the stale pages were from a prior build of the site, which is why it's important we had the unique build folders I mentioned earlier! It's kind of like a mini wayback machine.

These two architectural changes had immediate, noticeable improvements that improved the stability of our stack. Due to how successful they were, this was supposed to be the last architectural change we made before primary season started early February. Unfortunately, Heroku experienced several significant outages during the month of January. This included service outages to important features such as logging that lasted over 24 hours, and complete platform failures. So just over a week before the Iowa caucuses, scared out of my mind Heroku was off the rails, I convened some members of the team to decide if we should move to a static website, and we ultimately decided to do so.

Part of the safety net in making the decision was that most of the work involved in making this switch was creating the edge routing on Cloudflare Workers, as our backend web servers merely had to point the data they were already generating at S3 instead of Redis. Here is what the new SSR strategy was, and the last architectural change we shipped to production.

Keep a set of key value pairs for the "build version" and "content version".
If anything is published in the CMS, a webhook fires and the "content version" is incremented.
If the website is deployed, increment the build version and push build files to the CDN.
When a request comes in, Cloudflare Worker pulls the HTML from the CDN (*also had simple retry logic, as S3 requests rarely but occasionally fail).
After serving the request, the Cloudflare Worker forwards the request to the web server.
Web server receives the request, if the page is marked as stale, web server adds an item to a Redis queue to notify a background worker the page needs to be re-rendered.
Background worker eventually re-renders the page, pushes the HTML to the CDN and clears Cloudflare cache for that page.

This strategy solidified every aspect of the website was served from a CDN first, and all of the computational and network load of server side rendering was off-loaded to background workers. Just as important, it continued to fulfill our goal of allowing CMS editors to publish changes and see the update made to production in seconds.

And then it came, the night of the Iowa caucuses. As we were in the thick of prime-time, just before 9pm Eastern to be exact, Heroku experienced another major platform failure... but jokes on them because ElizabethWarren.com was fully static! 😎

If you'd like to read more about the work we did on ElizabethWarren.com, checkout this Twitter thread!

Write unit tests for React Hooks

Joe Kent — Sat, 12 Oct 2019 22:32:30 +0000

React hooks are an elegant pattern for sharing logic between your React components. If you're not familiar with the concept, you can read more about them here.

One of the problems I recently ran into with creating custom React hooks was figuring out a clear strategy for testing the function independently of all the different components consuming the hook.

For example, imagine we were using the following hook in 10 different components.

import { useState, useEffect } from 'react';

function useExample() {
  const [value, setValue] = useState(0);

  useEffect(() => setValue(1), []);

  return value;
}

This hook will initially return a value equal to 0, and after the initial render will return a value equal to 1. Rather than unit test this logic in several components, it would be great to write tests specific to this hook.

To write a unit test for this hook, there are two challenges to overcome.

We can't simply run something to the effect of expect(useExample()).toBe..., as it will return a React error stating we invoked the hook outside of a component being rendered.
We want a clear pattern to test the initial hook value, and the value after it renders.

To solve these challenges, this I was able to work out a simple test utility function,

import React from 'react';
import { shallow, mount } from 'enzyme';

export default function testHook(runHook, flushEffects = true) {
  function HookWrapper() {
    const output = runHook();

    return (
      <span output={output} />
    );
  }

  const wrapperFunc = flushEffects ? mount : shallow;
  const wrapper = wrapperFunc(<HookWrapper />);

  return wrapper.find('span').props().output;
}

You can now easily unit test your custom React hooks by passing them to this function. Here is an example of how I would test the custom hook example I created earlier using Jest,

import useExample from '../src/js/hooks/useExample';
import testHook from './_test-hook';

test('Initial value is 0', () => {
  const value = testHook(() => useExample(), false);
  expect(value).toBe(0);
});

test('Value after render is 1', () => {
  const value = testHook(() => useExample());
  expect(value).toBe(1);
});

If your application uses Hooks that rely on context values, you can also wrap the HookWrapper with that same context provider, and pass in the context value you want to leverage.

import React from 'react';
import { shallow, mount } from 'enzyme';
import { ApplicationContext, defaultApplicationState } from '../src/js/Application';

export default function testHook(runHook, flushEffects = true, useApplicationContext = null) {
  function HookWrapper() {
    const output = runHook();

    return (
      <span output={output} />
    );
  }

  let Container = null;

  if (useApplicationContext) {
    const value = {
      ...defaultApplicationState,
      state: useApplicationContext(defaultApplicationState),
    };

    Container = ({ children }) => (
      <ApplicationContext.Provider value={value}>
        {children}
      </ApplicationContext.Provider>
    );
  }

  const wrapperFunc = flushEffects ? mount : shallow;

  const wrapper = wrapperFunc(Container ? (
    <Container>
      <HookWrapper />
    </Container>
  ) : (
    <HookWrapper />
  ));

  return wrapper.find('span').props().output;
}

To demonstrate, the following is a simple hook that relies on a context value. By default, it will set the return value to 0. If the context value is a valid number, it will set that to be the state value.

import { useState, useEffect, useContext } from 'react';
import { ApplicationContext } from '../Application'; 

function useExampleContext() {
  const [value, setValue] = useState(0);
  const { contextValue } = useContext(ApplicationContext);

  useEffect(() => {
    if (contextValue !== value && typeof contextValue === 'number') {
      setValue(contextValue);
    }
  }, [contextValue]);

  return value;
}

And now you can test the hook in the following fashion,

test('Initial value is 0', () => {
  const value = testHook(() => useExampleContext(), false);
  expect(value).toBe(0);
});

test('Value matches context value', () => {
  const value = testHook(() => useExampleContext(), true, (state) => ({ ...state, contextValue: 1 }));
  expect(value).toBe(1);
});

I hope this post was helpful for anyone else looking to unit test their custom React hooks, and I am curious to see how others have handled this problem!