Forem: Paul Scanlon

How to Build a Survey With KwesForms and Astro

Paul Scanlon — Thu, 22 Feb 2024 02:54:00 +0000

In this post I'll explain how I built a survey form using KwesForms and Astro, complete with some fancy data viz to me help better understand the results.

To give you a taste of what's to come, below you'll find links to a live preview and the GitHub repo.

Getting Started

To get started with KwesForms, you'll first need to Sign up, then you can create your first form. Here's the Getting Started guide from the KwesForms docs.

When using KwesForms with Astro, I've found it easier to load KwesForms from the CDN and use Astro's is:inline script directive, E.g.

<script is:inline src="https://kwesforms.com/v2/kwes-script.js"></script>

Add the KwesForms script to the bottom of any Astro page that will use KwesForms. You can see the src for my survey page here: index.astro.

Building a Form

When following the setup guide from the KwesForms docs you'll be given code snippets to copy and paste into your page. A very minimal example looks like the below.

<form redirect="/submitted" action="https://kwesforms.com/api/foreign/forms/swj...">
  <label for="name">Your Name</label>
  <input type="text" name="name" />

  <button type="submit">Submit</button>
</form>

I've added an extra attribute to the above code snippet to control the behavior of my form after data has been been submitted. Using the redirect attribute I can redirect to a /submitted page.

Form Fields

I've mainly used HTML select and radio buttons for my form, and below i'll explain how each of them work.

Select

The first field in my survey form is a standard HTML select. Nothing extra is required here from the KwesForms side of things.

<label for="time">Time</label>
<select id="time" name="time">
  <option hidden>Please select</option>
  <option>🌞 Daytime</option>
  <option>🌚 Nighttime</option>
  <option>🌓 Anytime</option>
</select>

Radio Group

The second set of fields in my form are a "radio group" and have been wrapped in an HTML fieldset element. You'll also notice I've given the fieldset an attribute of data-kw-group. Defining a group using this data- attribute allows KwesForms to apply any rules that may have been defined to each element within the group.

<fieldset data-kw-group rules="required">
  <div>
    <input type="radio" id="novice" name="experience" value="1-5" />
    <label for="novice">1-5</label>
  </div>
  <div>
    <input type="radio" id="intermediate" name="experience" value="5-10" />
    <label for="intermediate">5-10</label>
  </div>
  <div>
    <input type="radio" id="proficient" name="experience" value="10-20" />
    <label for="proficient">10-20</label>
  </div>
  <div>
    <input type="radio" id="expert" name="experience" value="20+" />
    <label for="expert">20+</label>
  </div>
</fieldset>

Repeater Fields

The third set of fields in my form are super cool! KwesForms calls them repeater fields. These are fields that you define once in your code, but can be duplicated or removed by the user to enable multi-input answers.

<fieldset repeater name="technology">
  <div repeater-group>
    <div>
      <label for="technology">Technology</label>
      <select id="technology" name="technology">
        <option hidden>Please select</option>
        <option>React</option>
        <option>Qwik</option>
        <option>Astro</option>
        <option>Remix</option>
        <option>Serverless</option>
        <option>Edge</option>
        <option>Postgres</option>
        <option>SQL</option>
        <option>CSS</option>
        <option>SVG</option>
      </select>
    </div>
  </div>
</fieldset>

You'll see from the image below that KwesForms automatically injects an "add" button, and in cases where a field has been added, KwesForms will automatically inject a "remove" buttons — I told you they were super cool!

Validation

KwesForms is client-side form solution and will perform client-side validation and comes built in with some handy announce fields which you can style however you like, but what's really cool is that KwesForms also handles server-side validation for you automatically!

Spam Protection

Another feature I really like about KwesForms is it's built-in spam protection, powered by AI!

Data Collection

Now that you have a form set up and capturing data, you might be asking, where does the data go?

Every form submission can be viewed in the KwesForms Dashboard and there are a number of filters and actions available to you for viewing, sorting or deleting data.

More Data

But... whilst looking at a list of data is good, being able to transform this data into something more fancy is what i'm really interested in.

KwesForms API

KwesForms has an API which can be used to "fetch" all data submitted by any form. Once you have a method in place for requesting data, you can do whatever you like with it. Which is what i've done on the /results page.

By using a little bit of High School Maths and the SVG element I've created some Data Visualization to make it easier to understand the data.

To achieve this I'm making a request to the KwesForms API using the Get All Submissions endpoint.

The request I've used in my survey runs server-side inside of Astro's code fences and looks a bit like the below.

// src/pages/results.astro
---
const response = await fetch(
  `https://kwes.io/api/v1/forms/${import.meta.env.KWESFORMS_FORM_ID}/submissions?mode=production`,
  {
    method: 'GET',
    headers: {
      Accept: 'application/json',
      'Content-Type': 'application/json',
      Authorization: `Bearer ${import.meta.env.KWESFORMS_API_KEY}`,
    },
    redirect: 'follow',
  }
);

const data = await response.json();
---

There's two additional bits of information you'll need to make an API call, they are as follows.

FORM_ID: You'll find this in your browser's address bar and it'll be a four digit number E.g. kwesforms.com/websites/4411/forms/4694
API_KEY: You can find this under the account dropdown and it'll be a long alpha-numeric number E.g. gwEvpqTVnGDA4s5vOHeIAcOxSO566jXZ5EPjfFj...

The response for the above endpoint will look similar to the below but naturally, it'll depend on the fields you have in your form.

[
  {
    ...
    "time": "🌞 Daytime",
    "experience": "1-5",
    "technology": [
      {
        "technology__1": "CSS"
      }
    ]
  },
  {
    ...
    "time": "🌚 Nighttime",
    "experience": "20+",
    "technology": [
      {
        "technology__1": "Edge"
      },
      {
        "technology__2": "Remix"
      }
    ]
  }
]

Data Visualization

To create a data visualization from this data it first needs to be grouped, and then counted. For the time and experience data the process is pretty much the same but I've created two functions to group each of these.

groupByTime

This function groups the data by the time key from the response.

export const groupByTime = (data) => {
  const groupedData = data.reduce((result, item) => {
    const { time } = item;
    const existingItemIndex = result.findIndex((obj) => obj.name === time);

    if (existingItemIndex !== -1) {
      result[existingItemIndex].count++;
    } else {
      result.push({ name: time, count: 1 });
    }

    return result;
  }, []);

  return groupedData;
};

// usage
const byTime = groupByTime(data);

You can see the src for this function here: group-by-time.js

The result of this function will look similar to the below.

[
  {
    "name": "🌞 Daytime",
    "count": 5
  },
  {
    "name": "🌚 Nighttime",
    "count": 7
  },
  {
    "name": "🌓 Anytime",
    "count": 1
  }
]

groupByExperience

This function groups the data by the experience key from the response.

export const groupByExperience = (data) => {
  const groupedData = data.reduce((result, item) => {
    const { experience } = item;
    const existingItemIndex = result.findIndex((obj) => obj.name === experience);

    if (existingItemIndex !== -1) {
      result[existingItemIndex].count++;
    } else {
      result.push({ name: experience, count: 1 });
    }

    return result;
  }, []);

  return groupedData;
};

// usage
const byExperience = groupByExperience(data);

The result of this function will look similar to the below.

[
  {
    "name": "1-5",
    "count": 4
  },
  {
    "name": "20+",
    "count": 4
  },
  {
    "name": "5-10",
    "count": 2
  },
  {
    "name": "10-20",
    "count": 3
  }
]

You can see the src for this function here: group-by-experience.js

groupByTechnology

This function uses a forEach to iterate over the technology array and then groups the data by the key name.

export const groupByTechnology = (data) => {
  const groupedData = data.reduce((result, item) => {
    const { technology } = item;

    technology.forEach((key) => {
      const name = Object.values(key)[0];
      const existingItemIndex = result.findIndex((obj) => obj.name === name);

      if (existingItemIndex !== -1) {
        result[existingItemIndex].count++;
      } else {
        result.push({ name, count: 1 });
      }
    });

    return result;
  }, []);

  return groupedData;
};

// usage
const byTechnology = groupByTechnology(data);

The result of this function will look similar to the below.

[
  {
    "name": "CSS",
    "count": 2
  },
  {
    "name": "SQL",
    "count": 1
  },
  {
    "name": "Edge",
    "count": 1
  },
  {
    "name": "Remix",
    "count": 3
  },
  {
    "name": "Qwik",
    "count": 5
  }
]

You can see the src for this function here: group-by-technology.js

In call cases, the returned shape is an array of objects containing a name and count value. This data can now be used to drive the data visualization.

Creating the Bar Chart

The Bar Chart I've used in my survey has been "hand cranked". You could install a charting library if you want but, you might find it harder to style, and in some cases it might only work when JavaScript is enabled in the browser.

Using my "hand-cranked" approach, the chart will be super lightweight, you won't be sending any additional client-side JavaScript to browser, it can be rendered on the server, and will work even if JavaScript is disabled in the browser.

Data Fetching

The first step is to create a Bar Chart component, fetch the data and pass it on to the component via a prop named data E.g.

// src/pages/results.astro

---
import { groupByTechnology } from '../utils/group-by-technology'
import BarChart from '../components/bar-chart.astro';

const response = await fetch('https://kwes.io/api/v1/forms/...');

const data = await response.json();

const byTechnology = groupByTechnology(data);
---

<BarChart data={byTechnology} />

Setting Up Defaults

The next step is destructure the data from Astro.props, install / import d3-scale and setup some defaults.

// src/components/bar-chart.astro
---
const { data } = Astro.props;

import { scaleLinear } from 'd3-scale';

const chartWidth = 1920;
const chartHeight = 1080;

const paddingL = 100;
const paddingR = 100;
const paddingT = 100;
const paddingB = 180;
const gap = 20;
const maxValue = data.reduce((items, item) => Math.max(items, item.count), 0);
const xAxis = new Array(8).fill(null);
const axisPaddingB = 80;
const axisPaddingT = 100;
const xValues = scaleLinear().domain([0, maxValue]).range([0, maxValue]).ticks(xAxis.length);
const colorValues = scaleLinear().domain([0, data.length]).range(['#f056c7', '#0091f7', '#58e6d9']);
---

Bar Chart Properties

This step is used to create a properties object that will drive the chart. Each value, x, y, width, height, etc can be calculated by using values from the data set.

// src/components/bar-chart.astro
---

const { data } = Astro.props;

import { scaleLinear } from 'd3-scale';

const chartWidth = 1920;
const chartHeight = 1080;

const paddingL = 100;
const paddingR = 100;
const paddingT = 100;
const paddingB = 180;
const gap = 20;
const maxValue = data.reduce((items, item) => Math.max(items, item.count), 0);
const xAxis = new Array(8).fill(null);
const axisPaddingB = 80;
const axisPaddingT = 100;
const xValues = scaleLinear().domain([0, maxValue]).range([0, maxValue]).ticks(xAxis.length);
const colorValues = scaleLinear().domain([0, data.length]).range(['#f056c7', '#0091f7', '#58e6d9']);

+ const properties = data
+  .sort((a, b) => b.count - a.count)
+  .map((property, index) => {
+    const { name, count } = property;

+    const x = paddingL;
+    const width = ((chartWidth - paddingL - paddingR) / maxValue) * count;
+    const height = (chartHeight - paddingT - paddingB) / data.length;
+    const y = height * index + paddingT;

+    return {
+      name: name,
+      count: count,
+      font: 40,
+      x: x,
+      textX: x - paddingL / 2,
+      y: y + gap,
+      width: width,
+      height: height - gap,
+      color: colorValues(index * 2),
+    };
+  });
---

Bar Chart Axis

This step is to create some properties to help display the xAxis on the chart.

// src/components/bar-chart.astro
---

const { data } = Astro.props;

import { scaleLinear } from 'd3-scale';

const chartWidth = 1920;
const chartHeight = 1080;

const paddingL = 100;
const paddingR = 100;
const paddingT = 100;
const paddingB = 180;
const gap = 20;
const maxValue = data.reduce((items, item) => Math.max(items, item.count), 0);
const xAxis = new Array(8).fill(null);
const axisPaddingB = 80;
const axisPaddingT = 100;
const xValues = scaleLinear().domain([0, maxValue]).range([0, maxValue]).ticks(xAxis.length);
const colorValues = scaleLinear().domain([0, data.length]).range(['#f056c7', '#0091f7', '#58e6d9']);

const properties = data
 .sort((a, b) => b.count - a.count)
 .map((property, index) => {
   const { name, count } = property;
     ...
 });

+ const axis = xAxis.map((_, index) => {
+  const x = ((chartWidth - paddingR) / xAxis.length - 1) * index;
+  const width = 2;
+  const height = chartHeight - axisPaddingT - axisPaddingB;
+  const y = axisPaddingT;

+  return {
+    x: x + paddingL,
+    y: y,
+    width: width,
+    height: height,
+    value: xValues[index],
+  };
+ });

---

Display The Data

This last step is the finished SVG element complete with two Array.maps that iterate over the properties and axis arrays defined above.

// src/components/bar-chart.astro
---

const { data } = Astro.props;

...

---

+ <svg
+  xmlns='http://www.w3.org/2000/svg'
+  viewBox={`0 0 ${chartWidth} ${chartHeight}`}
+ >

+ {
+   axis.map((axi) => {
+     const { value, x, y, width, height } = axi;

+      return (
+        <>
+          <rect x={x} y={y} width={width} height={height} />
+          <text x={x} y={y + height} text-anchor='middle' font-size={30}>
+            {value}
+          </text>
+        </>
+      );
+    })
+  }

+  {
+    properties.map((property) => {
+      const { name, count, font, x, textX, y, width, height, color } = property;

+     return (
+       <>
+          <rect x={x} y={y} width={width} height={height} fill={color} />
+          <text
+            x={textX}
+            y={y + height / 2 + font / 2.6}
+            class='fill-brand-text font-bold'
+            text-anchor='start'
+            font-size={font}
+          >
+            {`x${count}`}
+          </text>
+          <text
+            x={chartWidth - paddingR - 20}
+            y={y + height / 2 + font / 2.6}
+            text-anchor='end'
+            font-size={font}
+          >
+           {name}
+          </text>
+        </>
+      );
+    })
+  }
+ </svg>

You can see the src for the Bar Chart here: bar-chart.astro

Finished

And that's it, a fully custom survey form complete with data visualization, and all data is stored safely and securely within KwesForms in case you wish to do anything else with it.

The Gatsby Changelog Prototype

Paul Scanlon — Tue, 07 Dec 2021 20:35:28 +0000

My Workflow

build-me.yml

A scheduled workflow for Jamstack Static Site Generators.

Demo: https://changelog.gatsbyjs.io
Repo: https://github.com/gatsby-inc/changelog

You can read all about how I created this workflow and the demo site on the link below: 👇
Building The Gatsby Changelog Prototype

Submission Category:

DIY Deployments

Yaml File or Link to Code

build-me.yml

name: Build Site

on:
  schedule:
    - cron: '* */12 * * *'

jobs:
  Trigger-Build:
    runs-on: ubuntu-latest
    steps:
      - run: echo "set env using GATSBY_CLOUD_BUILD_WEBOOK"
        env:
          GATSBY_WEBHOOK: ${{secrets.GATSBY_CLOUD_BUILD_WEBOOK}}
      - name: Post to Gatsby Cloud Webhook
        id: rebuild
        uses: fjogeleit/http-request-action@master
        with:
          url: ${{secrets.GATSBY_CLOUD_BUILD_WEBOOK}}
          method: 'POST'

Additional Resources / Info

fjogeleit/http-request-action@master

Building The Gatsby Changelog Prototype

Paul Scanlon — Mon, 06 Dec 2021 15:22:11 +0000

Hi 👋

In this post i'll be documenting my journey as i develop a prototype Gatsby Changelog site for the GitHub Actions Hackathon.

Repo: gatsby-inc/changelog
Gatsby Changelog: changelog.gatsbyjs.io
Twitter Account: @GatsbyChangelog
Submission Blog post: The Gatsby Changelog Prototype

Day 8 - Wednesday 8th December

Twitter Feed is in! This is a bonus feature and i'm almost out of time but... i thought it might be nice to demonstrate Gatsby's Serverless Functions. Serverless Functions are written and live along side your front end code but they run on "the server". What this means for Jamstack devs is that you can write a function as you normally would and Gatsby will run it on the server. This is particulary helpful when communicating with API's that require secret API keys and will only accept server-side requests. The Twitter API is one instance where using Serverless Functions makes perfect sense!

I'm using the Twitter V2 API for Node.js to request Tweet information from the new @GatsbyChangelog Twitter account and display them on a new page on the site which can be see here: https://changelog.gatsbyjs.io/twitter-feed.

To use Serverless Fucntions all you need to do is export a default function from somewhere in the src/api dir. Here's the one i'm using in the Changelog. It's called get-tweets.js

//src/api/get-tweets

const Twitter = require('twitter-v2');

const twitter = new Twitter({
  consumer_key: process.env.TWITTER_CONSUMER_KEY,
  consumer_secret: process.env.TWITTER_CONSUMER_KEY_SECRET,
  access_token_key: process.env.TWITTER_ACCESS_TOKEN,
  access_token_secret: process.env.TWITTER_ACCESS_TOKEN_SECRET
});

export default async function hander(req, res) {
  const { id } = JSON.parse(req.body);

  try {
    if (!id) {
      res.status(400).json({ message: 'id not found' });
    }

    const { data } = await twitter.get(`users/${id}/tweets`);

    res.status(200).json({
      message: 'A ok!',
      tweets: data
    });
  } catch (error) {
    res.status(500).json({ error: 'Error' });
  }
}

... and i make the HTTP request from my React page component inside a useEffect

//src/pages/twitter-feed.js

  const [isLoading, setIsLoading] = useState(true);
  const [tweets, setTweets] = useState([]);

  useEffect(() => {
    const getTweets = async () => {
      const response = await fetch('/api/get-tweets', {
        method: 'POST',
        body: JSON.stringify({ id: '1456240477783695360' })
      });

      const data = await response.json();

      setTweets(data.tweets);
      setIsLoading(false);
    };

    getTweets();
  }, []);

When the response comes back i set the returned data into React's state using setTweets and then iterate over the response in Jsx.

You can see the function src code here: get-tweets.js,
and the React component src code here: twitter-feed.js

So there you have it. SSG for data that doesn't change a lot but is kept fresh using a GitHub Action, SSR for data that changes frequently and is statically baked into a page and Serverless Functions for real time data requests.

I'm pretty sure that no matter what you're trying to build you can achieve it using Gatsby and Gatsby Cloud.

Day 7 - Tuesday 7th December

Events are in! I wasn't sure if i could pull this off in time but decided i'd give it a go and see what happens. I wanted to demonstrate where using SSR rather than SSG is a good idea and Repository Events seemed like a good fit.

Where SSR differs to SSG is when data is requested each and every time a users visits a page. The data is still statically baked into the page so SEO doesn't suffer as much as it would if you were to make a typical client side request (CSR). However like with CSR, SSR will exhibit is a slight delay before the page loads.

In my opinion compared to SSG, SSR is the second best option.

That said i feel SSR is the right method for dealing with data that frequently change, like GitHub Repository Events.

By contrast SSG and the Cron Job GitHub Action is the best way to deal with data that updates less frequently.

To request "events" i've opted for using the GitHub REST API. I had a play with the GraphQL API for this but as with SSR vs SSG, GraphQL compared to REST also has its advantages and disadvantages. The REST API does a lovely job of returning a whole lump of data which is perfect for the Events page. If i'd tried to do the same with the GraphQL API i would have had to create quite a specific query to pull all the different event types. Pull Requests, Issues, Watch Events etc.

Here's a snippet of the REST request i'm using.

  const { data } = await octokit.request(
    `GET /repos/{owner}/{repo}`,
    {
      owner: 'gatsbyjs',
      repo: 'gatsby'
    }
  );

I've also added an intro section to the site which explains a little more about the project and links to this blog post. The "content" for the intro section is sourced from a local markdown file and i've created a new node in the Gatsby Data layer called "intro".

I think i'm pretty much there with this project now... and think i'm ready to submit my entry for the Hackathon -- Wish me luck!

Day 6 - Monday 6th December

I've had an absolute belter of a day.

I have now implemented my GitHub Action. It's effectively a Cron Job which runs every 12 hours. 🥳

... and here it is

name: Build Site

on:
  schedule:
    - cron: '* */12 * * *'

jobs:
  Trigger-Build:
    runs-on: ubuntu-latest
    steps:
      - run: echo "set env using GATSBY_CLOUD_BUILD_WEBOOK"
        env:
          GATSBY_WEBHOOK: ${{secrets.GATSBY_CLOUD_BUILD_WEBOOK}}
      - name: Post to Gatsby Cloud Webhook
        id: rebuild
        uses: fjogeleit/http-request-action@master
        with:
          url: ${{secrets.GATSBY_CLOUD_BUILD_WEBOOK}}
          method: 'POST'

When the schedule kicks in i call a Gatsby Cloud webhook url which is stored as a GitHub Secret. This webhook URL notifies Gatsby Cloud to give off a new build. When a new build is started the data i'm querying using the GitHub GraphQL API is returned and pumped into Gatsby's data layer. If no new date is found Gatsby Cloud is smart enough to know it can use a cached build rather than start a fresh one so builds that contain existing data are super speedy.

Now the Cron Job is running this cycle will repeat from now... until the end of time!

I think this is a great way to ensure that a static websites can remain "fresh" and so long as the data you're requesting doesn't change too often, running a Cron Job every 12 hours should be sufficient.

Markdown update!! After trying and failing to work out how to deal with this pesky frontmatter issue i dropped a note into the Gatsby Engineering Slack Channel and within seconds Kyle Mathews came to my aid and suggested i try gray-matter... and as luck would have it, it worked a treat!

To recap, the problem i was having was that the .md files i'm sourcing from GitHub contain frontmatter. You can see one of them here: v4.3.

I wanted to do two things:

remove the frontmatter from the main "body" of the markdown and create a new html object on the node
transform and isolate just the frontmatter and create a new frontmatter object on the node

Here's the two functions i'm using to achieve this

transform frontmatter

//gatsby-node.js
const matter = require('gray-matter');

const transformFrontmatter = (markdown) => {
  const grayMatter = matter(markdown);

  return grayMatter.data;
};

Convert markdown to HTML

// gatsby-node.js

const remark = require('remark');
const remarkParse = require('remark-parse');
const remarkRehype = require('remark-rehype');
const rehypeStringify = require('rehype-stringify');
const rehypeAutoLinkHeadings = require('rehype-autolink-headings');
const rehypeSlug = require('rehype-slug');
const matter = require('gray-matter');

const convertToHTML = async (markdown) => {
  const grayMatter = matter(markdown);

  const response = await remark()
    .use(remarkParse)
    .use(remarkRehype)
    .use(rehypeSlug)
    .use(rehypeAutoLinkHeadings)
    .use(rehypeStringify)
    .process(grayMatter.content);

  return String(response);
};

I can then use both of these functions when i create the new node

    createNode({
      id: object.id,
      index: index,
      name: name,
      frontmatter: await transformFrontmatter(object.text),
      html: await convertToHTML(object.text),
      internal: {
        type: 'changelog',
        contentDigest: createContentDigest(data)
      }
    });

It's a bit tricky to explain without showing the output but rest assured it does what i needed.

I've also spent some time today tidying up the CSS so the site looks a little nicer and refactored the jump links.

I actually needed to use two additional rehype plugins to achieve this. rehypeSlug and rehypeAutoLinkHeadings this adds the name of the heading to an href so that i can "jump" to any heading contained within the markdown file.

The onRouteUpdate now looks like this

// gatsby-browser.js

export const onRouteUpdate = ({ location }) => {
  const heading = document.getElementById(location.hash.split('#')[1]);
  const nav = document.querySelector(`header`);

  if (heading) {
    window.scrollTo({
      top: heading.offsetTop - nav.offsetHeight - 12
    });
  }
  return true;
};

Oh and I added a link to the @GatsbyChangelog Twitter account and did a little bit of work around SEO / Opengraph data

... and that just about wraps things up for today.

Day 5 - Sunday 5th December

Had a better day today. I tried again to play with the frontmatter but still haven't quite worked out what's going on so shifted my focus to some CSS. I have a nice looking site now, the one issue i did encounter is adding the hash links to the sidebar because what i want to happen is when any nav link in the sidebar is clicked the page will scroll to the heading for that release... but i also want each heading to work the same way. I also want to highlight the current nav link in the sidebar.

This is probably a post on its own but here's the main bit that allows a page to "jump" to a heading.

// gatsby-browser.js

export const onRouteUpdate = ({ location }) => {
  const jumplink = document.querySelectorAll(
    `a.jumplink[href="/${location.hash}"]`
  )[0];

  if (jumplink) {
    window.scrollTo({
      top: jumplink.offsetTop
    });
  }
  return true;
};

this "jumps" to the offsetTop of any HTML <a> that has the class name jumplink and of course, the correct location.hash E.g


<a class="jumplink" href="/#v4.2">
   <span class="text-lg text-gray-300">#</span>v4.2
</a>

I've got a little more CSS to do today because i'll need a burger menu so that sidebar can open and close and i'll probably need to adjust some font sizes so the site is usable on "mobile"

Day 4 - Saturday 4th December

Not a great day today, i was partly walking around NYC trying to find a coffee shop so i could sit down and work out a problem i've been having with the frontmatter contained within each release note.

Each release note uses frontmatter which contains a date, version number and title...

---
date: "2021-11-30"
version: "4.3.0"
title: "v4.3 Release Notes"
---

The trouble is when i use use remark-rehype each item from the frontmatter gets converted to either a <p> or <h2> and really i want to display this information the same way GitHub does, e.g in an HTML table. I haven't solved this yet so i'll pop it in the backlog and come back to it later.

Day 3 - Friday 3rd December

Today was mainly having a think and trying to workout a few things. It had previously occurred to me that using gatsby-transformer-remark to transform remotely sourced markdown files might be "taking the long route"... and it turns out it is.

The reason is that gatsby-tranfromer-remark is great for transforming files or "nodes" that already exists on disk. But, in my case i'm souring markdown from a remote source. In order for the plugin to do it's thing i have to make GraphQL understand the markdown files are of type "File"

All of this is completely unnecessary if i transform the markdown nodes to html before i add them to Gatsby's data layer. To achieve this i'll use the same remark transformers the plugin uses. The only slight snag is that a lot of the remark node modules are ESM only and i need to use them in gatsy-node which at the time of writing only compiles common.js

It's kinda ok for now because i've rolled back the versions of remark i'm using which will work as cjs. You can see where the updates to ESM happened by inspecting the commits on each of the repos.

Here's the full gatsby-node.js file showing how i'm currently sourcing remote markdown files and adding them to Gatsby data layer.

gatsby-node.js

const { graphql } = require('@octokit/graphql');
const remark = require('remark');
const remarkParse = require('remark-parse');
const remarkRehype = require('remark-rehype');
const rehypeStringify = require('rehype-stringify');

const graphqlWithAuth = graphql.defaults({
  headers: {
    authorization: `token ${process.env.OCTOKIT_PERSONAL_ACCESS_TOKEN}`
  }
});

const CHANGELOG = 'changelog';

const convertToHTML = async (string) => {
  const response = await remark()
    .use(remarkParse)
    .use(remarkRehype)
    .use(rehypeStringify)
    .process(string);

  return String(response);
};

exports.sourceNodes = async ({
  actions: { createNode },
  createContentDigest
}) => {
  const {
    repository: {
      folder: { entries }
    }
  } = await graphqlWithAuth(`
  query {
    repository(name: "gatsby", owner: "gatsbyjs") {
      folder: object(expression: "master:docs/docs/reference/release-notes") {
        ... on Tree {
          entries {
            name
            repository {
              createdAt
            }
            object {
              ... on Tree {
                entries {
                  name
                  object {
                    ... on Blob {
                      id
                      text
                    }
                  }
                }
              }
            }
            object {
              ... on Blob {
                id
                text
              }
            }
          }
        }
      }
    }
  }
  `);

  const createMarkdownNode = async (data, name, repository, index) => {
    const { object } = data;

    createNode({
      id: object.id,
      index: index,
      name: name,
      date: repository.createdAt,
      html: await convertToHTML(object.text), // this was markdown and is now html
      internal: {
        mediaType: 'text/markdown',
        type: CHANGELOG,
        contentDigest: createContentDigest(data)
      }
    });
  };

  entries.forEach((item, index) => {
    const { object, name, repository } = item;
    if (Array.isArray(object.entries)) {
      const markdown = object.entries.find((item) => item.name === 'index.md');
      createMarkdownNode(markdown, name, repository, index);
    } else {
      createMarkdownNode(item, name, repository, index);
    }
  });
};

Day 2 - Thursday 2nd December

I had a few hours to spare today to investigate how to query GitHub using the GraphQL API. I had a small problem with some of the release-notes markdown files. Specifically ones nested in sub directories. Over time we've moved each .md into it's own directory and in order to query both .md at the root of the release-notes dir and sub directories I needed to get a bit creative with GraphQL.

Here's the query i've settled on. I found it best to try out the query using GitHub's GraphiQL Explorer before implementing it into my project.

  query {
    repository(name: "gatsby", owner: "gatsbyjs") {
      folder: object(expression: "master:docs/docs/reference/release-notes/") {
        ... on Tree {
          entries {
            name
            repository {
              createdAt
            }
            object {
              ... on Tree {
                entries {
                  name
                  object {
                    ... on Blob {
                      id
                      text
                    }
                  }
                }
              }
            }
            object {
              ... on Blob {
                id
                text
              }
            }
          }
        }
      }
    }
  }

Now that I can query the data I can iterate over the response and using createNode add the markdown / HTML content to Gatsby's Data layer.

Adding the response data to Gatsby's data layer went ok, the difficulty was converting the sourced nodes which are of type markdown and transforming them to HTML using gatsby-transformer-remark. Usually if you have .md files on disk the plugin takes over and transforms the files nodes to HTML, however in my case the response which is of type markdown doesn't yet exist as a file node, and if it doesn't exist as a file node gatsby-transformer-remark can't do it's job.

So there was a quite a bit of work then went into creating the nodes on disk and then ensuring they are of the correct type using Gatsby's createSchemaCustomization.

Here's how I create the nodes

// gatsby-node.js

exports.onCreateNode = async ({
  node,
  actions: { createNode, createNodeField },
  createNodeId,
  createContentDigest
}) => {
  if (node.internal.type === CHANGELOG) {
    const markdownNode = {
      id: createNodeId(node.id),
      parent: node.id,
      internal: {
        mediaType: 'text/markdown',
        type: 'markdown',
        content: node.object.text,
        contentDigest: createContentDigest(node)
      }
    };

    createNode(markdownNode);

    if (markdownNode) {
      createNodeField({ node, name: 'text', value: markdownNode.id });
    }
  }
};

... and here's how i ensure they are of the correct type

// gatsby-node.js

exports.createSchemaCustomization = ({ actions: { createTypes } }) => {
  createTypes(`
    type changelog implements Node {
      text: markdown @link(from: "fields.text")
    }
  `);
};

This is all i need for now, and querying the data from the page component can now be done as i normally would, and here's what that looks like.

// index.js

import React from 'react';
import { graphql } from 'gatsby';

const Page = ({ data }) => {
  const {
    allChangelog: { nodes }
  } = data;

  return (
    <main className="container mx-auto max-w-5xl grid gap-16 p-8">
      {nodes.map((node, index) => {
        const {
          name,
          repository: { createdAt },
          text: {
            childMarkdownRemark: { html }
          }
        } = node;
        return (
          <div>
            <div className="text-brand-primary text-5xl font-black">{name}</div>
            <div
              className="prose lg:prose-xl"
              dangerouslySetInnerHTML={{ __html: html }}
            />
          </div>
        );
      })}
    </main>
  );
};

export const query = graphql`
  {
    allChangelog(sort: { fields: repository___createdAt, order: DESC }) {
      nodes {
        name
        repository {
          createdAt
        }
        text {
          childMarkdownRemark {
            html
          }
        }
      }
    }
  }
`;

export default Page;

The class names on the HTML elements are from Tailwind, and if you're looking to add TailwindCSS to your Gatsby site there's an excellent guide in the TailwindCSS Docs

Day 1 - Wednesday 1st December

I'm so late to the party but I saw this Tweet from Colby Fayock and thought... ah, ha. I wonder if I can cobble something together to demonstrate a Cron Job GitHub Action i've used on a number of projects for the GitHub Actions Hackathon.

I also wondered what kind of project I could build that would best demonstrate why a Cron Job is quite handy as a GitHub Action.

Then it struck me. We (at Gatsby) have launched a new Twitter Account. It's called GatsbyChangelog and we plan to use it to announce the changes to the Gatsby framework every couple of weeks. Along with the new Twitter Account i'll also be prototyping a Changelog site: https://changelog.gatsbyjs.io. We have plans to bring more content to support the descriptions contained within the Changelog, but we need a home for this content.

I plan to build a Gatsby site, hosted on Gatsby Cloud that will source markdown files from the Gatsby GitHub Account docs folder. The markdown files within the release-notes will be sourced and then rendered to a single page. I'll be using @octokit/graphql and a few of Gatsby's data handling methods: sourceNodes and createNode.

The Cron Job Action is kinda crucial for this project because the site will be statically rendered, but in order to ensure the data sourced from the release-notes is always "fresh" I'll use a Cron Job to call a webhook provided by Gatsby Cloud which will re-build the site every 12 hours. I could use Gatsby's newest page rendering method SSR to achieve the same thing but this approach it is way more fun!