Forem: Sean C Davis

Netlify Dynamic Site Challenge - Help Thread!

Sean C Davis — Thu, 02 May 2024 21:01:44 +0000

Looking for help with your Netlify DEV Challenge Submission?

We're here to assist anyone who encounters issues or has questions about Netlify!

Feel free to post your questions or concerns in the comments below, and we'll do our best to provide timely and helpful responses.

Just hearing about this challenge? Read the announcement post here:

Join us for the Netlify Dynamic Site Challenge: $3,000 in Prizes!

dev.to staff for The DEV Team ・ May 1

#netlifychallenge #devchallenge #webdev #javascript

Five Reasons I'm Excited about Astro

Sean C Davis — Tue, 12 Oct 2021 18:56:24 +0000

When Astro first appeared on the scene, I rolled my eyes.

Another static site generator. Really? Don't we already have enough to choose from?

Astro Fills in the Gaps

At first, it felt like it could have been an Eleventy plugin (similar to what Slinkity is doing). I was frustrated that the Astro team felt like they needed to add a whole new thing to the ecosystem.

But then I started tinkering with it and realized that Astro fills in the gaps that I've found in the other tools I've been working with recently. And that's when I got super excited!

The Right Amount of Opinion

For me, what it boils down to is that I feel like Astro brings the right degree of opinion to its framework. It knows that there are widespread preferences for lower-level tooling, and so it leaves options. But it provides a solid framework for implementing those options.

Now, granted, I've only built a few proofs of concept, but these are the five reasons I'm so excited about Astro:

Static-first, with progressive enhancement
Supporting multiple component frameworks
Native markdown support
A foundation for styling
Reduced boilerplate code

Let's take a quick look at each of these ideas.

1. Progressive Enhancement / Partial Hydration

Astro lets you build your website with modern front-end component frameworks, like React and Vue, but it doesn't assume you want to run JavaScript in the browser.

That's my struggle with frameworks like Gatsby and Next.js. They are super powerful, but they assume interactivity and always include JavaScript files in your built site.

Astro assumes the opposite. It makes you, the developer, tell it when you need interactivity. Otherwise it renders static HTML. That means that even though there is some magic that happens during the build, you are still in control of your HTML code for the most part.

2. Bring Your Own Framework (BYOF)

It's unlikely we're going to see one component framework to rule them all. React is hugely popular, yes, but Vue and Svelte have a major following that isn't going away.

Astro doesn't just say you can choose one of these frameworks, but that you can use any of them at any point. Meaning you could have a Vue component and a React component in the same project. That's not necessarily a great idea, but it leaves the developer to make that decision. Astro just provides the foundation.

3. Native Markdown Support

Pages can be .astro components or .md (markdown) files. It also provides a markdown component for use in Astro components, when you only need a little markdown in a larger component.

This feels like an inspiration from Eleventy — which will render markdown pages right out of the box, with the option to inject plugins as necessary.

I like this in comparison to Gatsby, which requires a clunky GraphQL query and custom JavaScript code to be able to render markdown pages. Or Next.js, which intentionally takes no opinion on data fetching or page content and requires that you do all the work.

4. A Foundation for CSS

Like BYOF, Astro provides an array of options for implementing styles in the project, including directly in a component, using CSS modules, PostCSS, Tailwind, Sass, and more.

This is great because, well, CSS is hard. And everyone has their own way of staying organized. And yet, in comparison to Eleventy, which makes no opinion on styling, a foundation of the popular options today is available for you in Astro.

5. Reduced Boilerplate Code

I love that all the logic for Astro components lives in the component. I'm not a Vue guy, and I've only tinkered with Svelte, so the concept was a bit odd to me at first.

But after playing around with it, I'm in love. With support for top-level await, you can write only the JavaScript you need within a component.

A Place for Eleventy

While it originally felt to me like Astro used Eleventy as a basis and then built something else on top of it, I'm looking at the relationship of the two differently after working with both.

Eleventy is very much about reducing the amount of code you have to write in general. It supports a number of server-side templating languages, and does not go after front-end component frameworks. There's a lot of power in this simplicity. It makes it a little easier to focus on the content.

But the big remaining benefit of Eleventy is that you have 100% control over the HTML that is rendered to the page. You, the developer, are in absolute control of the performance of your site. And that can't be understated. You'll have to do a bit of work to maintain that and to stay organized as your site grows, but you have the control.

Will I settle on one over the other? Maybe. But I think both have their place for now.

Power in Developers' Hands

So that's it. Astro provides a foundation that combines popular opinions today. It shows developers what those options are, and then asks the developer to pick one, or more. It says, "Here are three different hammers. You can pick the one that feels right, or you can use all three at different times. You just have to tell me what you want to do."

And that is really exciting!

A Guide on Transforming an Idea into a Website

Sean C Davis — Thu, 09 Sep 2021 18:42:32 +0000

At its core, Jamstack was a revolution. It took the best part of the first 25+ years of website development and combined them into a powerful pattern that is used widely today. It has created an explosion of ideas and innovations that have helped developers across the world build websites that are more performant, more secure, less expensive, easier to scale, and (most importantly) fun to build!

Instead of taking you on a theoretical journey and digging into the technical nuances of the Jamstack, we're going someplace else. We'll take a more tangible journey, as we step through the entire process of building a Jamstack website, from the little light bulb in your brain all the way to a physical website that real people can visit.

These are the stops we'll make along the way:

Bring the design to life
Find a developer
Build the site (write the code)
Review and test the site
Deploy, edit, and repeat, repeat, repeat
Or ... do it the Stackbit way!

Buckle up. Here we go!

Stop #1: Design

The first stop on the Jamstack journey is design — the process of visually representing your ideas.

From Spec to Delivery

The way this typically works is that you provide some form of spec to a designer. The designer then takes that spec and turns it into one or more design files that show what should be built. That can include page layouts, individual components, or global patterns like colors, fonts, and typography.

Delivering a Good Spec

The “spec” you provide the designer should be more then, “I need a website.” Before you bring a designer into your project, consider spending time to gather the following:

A list of pages on the site, along with the purpose of each page. Bonus points if you also include content, as you'll reduce the variability of the design when it is transformed into code.
Brand assets or other visual specifications you have — logos, colors, typography, etc.
A list of websites that you really like and why you like them. The designer will want you to be happy, and the more you can tell them what you like, the higher the chance that they will deliver on your vision.
A list of competitors. Even if the designer doesn't use inspiration from your competitors' sites, it's nice for them to know where the competition stands and what it'll take to be noticed in the crowd.

The designer may ask you for more information, but this is a good start to get the conversation rolling.

Locating the Right Designer

If you have a designer already working with you, that's a bonus! If not, the cost and abilities of designers vary widely. You could use a service like Fiverr and have a decent-sized website designed for a few hundred dollars (USD). Or you could hire a fancy agency and spend tens of thousands of dollars.

Generally speaking, cost tends to be somewhat proportional to experience and the quality you're going to get back. But that doesn't mean the inexpensive designer is going to provide an awful design. This is how I usually think about it:

If I hire a designer on the cheap, I'm going to have to do more work if I want to get something worthwhile out of it. I'll have to drive the process. I'll have to deliver a super tight spec so that they give me exactly what I'm looking for.
If I hire a fancy and expensive firm, I should expect to still have to put in time and effort, but should be led through the process and be left with something truly great.

Looking Toward Development

What's interesting about this process is that while design tends to be a fraction of the cost of development, it's also largely responsible for determining the complexity of the build. Granted, a developer can spend your money in a silly way and over-engineer a website for you. But, most of the time, the time a developer is going to spend building your site is determined by their specification — the design files.

The more interactive the design, the more it's going to cost to develop. The more unique each page is from the others, the more it's going to cost to develop.

You don't have to be a fortune teller. Instead, leave a little room in your budget for going back to the designer after consulting with your developer. If the developer says you can save a thousand dollars by changing a feature, maybe you'd want to spend another hundred to have the designer reimagine and simplify it.

The Deliverable

After going through the design phase, you're left with the deliverables — artifacts you can pass on to the developer as their specification. Here's an example for a simple site that includes just four files:

Home page for large screens
Home page for small screens
Interior pages
Style tiles (i.e. global style definitions)

And this is an example of what a style tiles spec might look like:

(This example is from Unmute, a side project I'm involved with.)

Stops #2-4: Development

Toot toot! This train keeps on rolling. The next several steps encompass the development phase of the project. Our Jamstack train makes three stops here because there are usually three steps in this process, though the amount to which you are involved in each will vary depending on the developer you hire.

The Process

This process itself is similar to the design process in that it goes through these steps:

Locate & Estimate: Developer provides estimate based on specification.
Build: Developer builds the site.
Review: The developer delivers the code and you review to make sure it works.

Locate a Developer

You can go through much the same process as you did with design. If you have an in-house developer, great! If not, you can use a service like Fiverr and hire devs on the cheap. Or you can look to a more formal agency or dev shop to suit your needs.

In this case, you don't have to build a spec for the developer. You already did that! The design files are your specification.

Finding the right developer is crucial to this process, as well. And that's why I've broken out the build portion (below) as three steps. Some developers (usually those you hire on the cheap) may only complete the first step in the process. But for you to truly achieve a useful and powerful Jamstack site, you'll want someone who can take you through all three.

The Build

These are the steps in the process:

Static build
Templatizing the code
Separating content from presentation

You can spend very little on web development and still get something that you can deploy out of it. Often, that means the developer only went through the first step and gave you static files. That's totally fine, but it will make adding new pages or editing existing content super difficult.

The second step — templatizing — aims to take the static code and turn it into modular pieces that can be reused. This process is explained in great length and technical detail in this guide.

After the code has been modularized, it can be further adjusted to extract all the content and put it in a single place, such as a content management system. This process is explained in great length in this guide.

And the end of these three steps, you'll be left with a system that you can use to edit content without worrying about messing up the code. In fact, you ought to be able to edit the content without even seeing the code.

Reviewing the Site

While the review process for design was more conceptual, with development, it's up to you to actually test that the website behaves as you'd expect. In other words, you're trying to break it. The developer should respond by fixing any bugs discovered within a certain timeframe.

While the developer may not deploy the code for you, they ought to be able to help you get it running so that you can test and send notes back to them.

Stop #5: Deploy & Use!

At this point you have a site that is ready to go to production, so that's the next step. I recommend using that same developer to help you through that process, but I've broken it out here because it tends to require more effort from you than the development phase of the project.

Here are a few of the crucial steps in this process:

Choose a host and setup an account
Buy and/or configure a domain name
Add analytics tooling
Add SEO content
Use it!

Let's look at each of these briefly.

Hosting

The term "Jamstack" was coined by a startup called Netlify. And they continue to lead the way in terms of deploying and hosting Jamstack websites. But there are other great tools out there, like Vercel, GitHub Pages, Azure Static Web Apps, and many more.

Whatever you choose, you'll want to set up an account for yourself. Most of these services have a free (or low cost) tier that is enough to accommodate small websites.

Once you have an account, you can add your developer, and they will hook it up so that the code is deployed. And then you will have a working website!

Domain

You'll want a domain name for your site, otherwise you'll be left with default names from your hosting provider. If you already have one, that's great!

Your developer can tell you what you need to do to point the domain to your host, and then you'll be able to use your domain name to access your new website!

Analytics

Analytics is a crucial piece of the puzzle, because you want to know who is visiting your website. Some hosting providers (like Netlify) offer what they call server-side analytics, which tend to be the most accurate, though there are limitations to them. But if you're just getting started, it's easy enough to start with Google Analytics.

Tell your developer you want to install Google Analytics and they can help you through that process.

SEO

Of all these pre-launch steps, perhaps the trickiest to deal with is SEO. Usually I'd expect a developer to prepare my site to support SEO. That means automatically generating a sitemap file and providing the ability to customize SEO meta values for any given page. (It would be worthwhile to mention this when first setting up the arrangement with the developer so that they are prepared.)

You'll want to:

Register your site (and sitemap) with Google Search Console. This will help provide you with analytics on where you land in search results. It'll also help Google know where your site is and what its pages are.
Add custom SEO values for every page on your site, including adding an image.

If you don't feel confident in this step, you can also hire an SEO expert to help you through the process. These types of consultants also have a huge price range, from Fiverr to big agency. If you're just getting started, I recommend toward the cheaper option — just have someone give you a quick audit and make the changes they suggest.

(Insider secret: Mostly what these SEO pros are doing is using some expensive tool, hooking up your site, running a report, and giving you the output of that report. The funny thing is, some of the people are often less expensive than buying the tools and doing the work yourself.)

Editing

When all of those last bits are in place, you should have a live site and it's time to start promoting it and keeping it up to date. Make a few changes to the content. Tell the world about it. and give yourself a pat on the back.

BUT WAIT! Before we wrap this up, I want to spend a little time giving you what I think is a better way to get started.

Stop #6: An Easier Approach

If you're new to the world of creating websites, I think you should take a look at Stackbit! (Of course I do, I work there and am writing on their blog.)

Stackbit is a site builder that specializes in Jamstack websites.

Instead of going through that entire process (the first five stops on the Jamstack journey), you could do it all yourself in much less time for much less money. After creating an account, Stackbit will give you a list of templates from which you can build your site. You'll then walk through the process of customizing it just for you.

The catch here is that Stackbit isn't just another WordPress or Squarespace, even though it may feel like it at first. Stackbit works to be super transparent about where your code and content live. What that means is that you can start this journey on your own, without a designer or a developer, at no cost other than your time. And you'll still have access to the code and content that powers your site. Once you're up and running, if you need a little help, that's when you can bring in a designer, developer, or SEO expert to make some suggestions or additions to your new site.

If that sounds like a walk in the park, give it a try!

In any case, I hope you've enjoyed your ride, and I'd love to learn more about your Jamstack Journey. Let's chat.

Separate Content from Website Code for Easier Editing

Sean C Davis — Tue, 24 Aug 2021 18:10:55 +0000

When it comes to turning a website design into a real thing (a website), a static site generator is an invaluable tool. It provides you with the ability to abstract reusable parts so you can write code more efficiently and more accurately.

We covered the process of moving from static HTML files to a static site generator (SSG) in the precursor to this article. Here we're going to look at taking your SSG game to the next level by transforming it into content-driven engine. Together, these two steps make up a few parts of The Jamstack Journey.

Let's dive in!

How a Content-Driven Website Works

There are two main steps in the process of transforming a templatized site into one that is content-driven:

Separate content from presentation
Abstract repeatable components

Separate Content from Presentation

The first step in the process is to separate content from presentation. Your templatized website likely has a number of HTML files representing the content of your site. (In our templatized example, we also used .njk files.)

For example, you might have a blog post represented as my-first-post.html that looks something like this:

It doesn't have any layout elements because those have all been abstracted away in your template, but it still has HTML code, which effects the structure — the presentation — of the page.

Converting this to a content driven approach means we can convert the HTML file into a content-based file, like markdown (e.g. my-first-post.md). And then that content can be fed into the main post layout. Like this:

If this doesn't make sense, don't worry yet. We're going to go through a real-world example.

Abstract Repeatable Components

This process forces us to develop a shape for our content. Doing so will help us uncover opportunities to further simplify templates by combining and abstracting repeatable structures. If this also has your head spinning, it's okay. We'll look at an example of this in action, too.

Benefits of a Content-Driven Website

The process of separating content from presentation has a number of benefits. These are the primary reasons I go through through this effort:

It reduces the potential for introducing new bugs through content. When you force your editors to write HTML code just to add content to your site, you are greatly increasing your risk for new bugs that come from unexpected or bad HTML code.
Less technical content editors can contribute without learning code.
It's more pleasant to author content. Markdown is built to get out of your way. Instead of wrapping a top-level heading in an <h1> tag, you just precede the line with #. There are no paragraph tags — they are inferred.
Markdown is faster to write, too! (At least once you get the hang of it.)

What about Content Management Systems?

If we're talking about separating content from presentation, shouldn't we be talking about content management systems (CMS), too?

(Great question, Sean.)

Yes, this would be the perfect time to introduce a CMS into your project. What we're doing here is substituting a more formal CMS with markdown files. Think of markdown files as the data coming from your CMS. In fact, you could technically pull content from a CMS and convert it to markdown files. I like this approach because a) it's simpler than wiring up a whole CMS, but b) can still work well with your setup if you choose to use a CMS.

Step 1: The Templatized Site

With that, let's get started!

Following the templatizing tutorial, we were left with this code. That's where we're going to begin for this exercise.

Add the contents of the example templatized project to some directory on your machine. Here's a link to download the larger example. After doing that, you can find the appropriate files in the 03-templated-site directory. Move these files into a new directory on your machine. Your folder's contents should look like this.

Now we're ready to get started. We're going to do a series of abstractions to get to our final content-driven site. What we'll focus on below are four steps that represent a smaller part of a larger effort. The code I'll share in the end will have made more abstractions, but we'll aim to arm you with the tools you need to get there.

To make sure everything is in order before we get going, it'd be safe to reinstall your dependencies and then start the development server.

npm install
npm run dev

Then visit localhost:8000 in your browser and you should see the site.

Step 2: Simplify Pages

The easiest step in this process is setting up our interior pages. Right now, our privacy and terms pages are .njk files, which means they are littered with HTML code. The beauty of Eleventy is that if we simply change the file extension in these files to .md they will work immediately.

Give it a try. Rename privacy.njk to privacy.md and terms.njk to terms.md. Notice that you don't even have to do anything and they work perfectly fine! That's the magic of markdown. Technically, HTML is valid markdown code.

But, even though it works, we have an opportunity to drastically simplify this content by removing the HTML. I ran the HTML code through an online converter. You could do the same or you could reference the new files (Terms here, and Privacy here).

⚠️ Notice that the little YAML markdown snippet at the top of these files did not change. That piece is still relevant so that Eleventy knows which template to use and how to pass on the title of each page to that template.

After these changes, your code should look like this and you should be able to visit localhost:8000/terms and localhost:8000/privacy with both looking as they did before you made the change.

Step 3: Shared Global Data

We could jump to the home page next, but that one is a doozy, so let's move on to something a little simpler first — shared global data. There is often content that you want to share across multiple pages or templates. Eleventy is built to support this with its global data files.

One example I see in looking at the site as it is right now are the details about the next event. So what I would do is place these details in their own data file in _data/next_event.json. Create that file and add the details:

{
  "date": "October 1, 2021",
  "time": "7 PM ET",
  "location": "Virtual",
  "cost": "Free"
}

We're using these in two places today — the home page jumbotron and the header in the content page. Since both files (index.njk and _includes/content-page.njk) are using the Nunjucks templating language, you can render this value using a Nunjucks variable.

For example, the code in the content page template would look like this:

<span class="hidden lg:inline-block lg:mr-2 xl:mr-3"
  >{{ next_event.date }}</span
>
<span class="hidden lg:inline-block lg:mx-2 xl:mx-3"
  >{{ next_event.time }}</span
>
<span class="hidden xl:inline-block lg:mx-2 xl:mx-3"
  >{{ next_event.location }}</span
>
<span class="hidden lg:inline-block lg:mx-2 xl:mx-3"
  >{{ next_event.cost }}</span
>

The home page can follow a similar pattern. Here is a summary of the changes I made.

It's a good idea to keep this pattern in mind and employ it whenever you find opportunities to share content (not code) among multiple pages or templates. For example, another use case might be to build a list of social media account URLs so that you can separate the links from the markup and style of the icons.

Give your browser another look to make sure that those values were updated as you'd expect.

Step 4: The Home Page

Let's take a look at the home page next. We won't abstract this in quite the same way as the interior pages. This is because when we templatized the site, we used a super generic template and put all the unique stuff directly in the home page.

What we want to do here is now extract the content from the home page so that it is trivial to edit without scrolling through the hundreds of lines of code. But we're in a good position to do that.

First, move your index.njk file to _includes/home.njk. Doing this effectively removes our home page. We still need that index page. But now we want it to be a content page. So let's create a new file in the root of the project and call in index.md.

Refresh your browser and see that there's nothing there! That's because Eleventy is now reading this as your home page, but there's no content and you haven't specified a layout. So let's do that. Change the file to look like this:

---
layout: home
---

Now refresh your browser and we're back.

Then we can move through the file meticulously and extract elements one at a time until we have separated all our content from the template.

For example, say we wanted to adjust the copy under "Unmute yours" in the jumbotron.

Maybe we make a new object for jumbotron and then add a body section to it. Your index.md file would look like this:

---
layout: home
jumbotron:
  body: |-
    Join us for an evening of storytelling with some of your favorite fellow
    nerds. We'll laugh. We'll cry. We may even hear a story without any
    industry jargon!
---

And you could replace the section in _includes/home.njk to look like this:

<p class="max-w-md mb-8 mx-auto md:mx-0">
  {{ jumbotron.body }}
</p>

This is a long and often tedious process, but it's a game-changer when it comes to editing your site down the road. I'll leave the rest up to you and share where I ended up after going through the exercise. But first I want to talk about the process of abstracting components.

Step 5: Abstracting Components

Extracting content from your pages and templates requires that you define the structure of your data. In Step 4, we could see the beginning of that structure — e.g. jumbotron was going to be its own section, presumably with multiple properties within it.

The resulting structure can help you determine where you can tighten up your template code further by looking for repeatable patterns.

For example, in this home page, each of the speakers has a similar shape to its content. So, even though you could define each speaker individually, it probably makes sense to develop a repeatable pattern, like this:

speakers:
  - name: Speaker Name
    position: Title / Company
    fun_fact: Fun Fact
    image: /images/speakers/speaker-01.png
    social:
      twitter: "#"
      facebook: "#"
      instagram: "#"
  - name: Speaker Name
    position: Title / Company
    fun_fact: Fun Fact
    image: /images/speakers/speaker-02.png
    social:
      twitter: "#"
      facebook: "#"
  - name: Speaker Name
    position: Title / Company
    fun_fact: Fun Fact
    image: /images/speakers/speaker-03.png
    social:
      twitter: "#"
  - name: Speaker Name
    position: Title / Company
    fun_fact: Fun Fact
    image: /images/speakers/speaker-04.png
    social:
      twitter: "#"
      facebook: "#"
      instagram: "#"

This repeatable pattern hints that we could also abstract the code. Instead of having individual HTML for each speaker, we could create another include template and render the content of each speaker.

Our _includes/home.njk file would include this:

<div class="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-{{ lineup.speakers.length }}">
  {% for speaker in speakers %}
    {% include "speaker.njk" %}
  {% endfor %}
</div>

And then _includes/speaker.njk could look like this:

<div class="mx-4 mb-8 lg:mb-0 template-home--speaker">
  <span class="block mb-2">
    <img src="{{ speaker.image }}" alt=""/>
  </span>
  <strong class="block mb-2 text-lg">{{ speaker.name }}</strong>
  <span class="block text-sm">
    <em>{{ speaker.position }}</em>
  </span>
  <span class="block text-sm">{{ speaker.fun_fact }}</span>

  <div class="flex items-center justify-center mt-4">
    {% if speaker.social.twitter %}
      <a href="{{ speaker.social.twitter }}" class="mx-1 hover:opacity-75 transition-all duration-300">
        <span class="inline-block bg-red text-white rounded-full p-2 overflow-hidden h-8 w-8 component--icon">
          {% include "svg/twitter.svg" %}
        </span>
      </a>
    {% endif %}
    {% if speaker.social.facebook %}
      <a href="{{ speaker.social.facebook }}" class="mx-1 hover:opacity-75 transition-all duration-300">
        <span class="inline-block bg-red text-white rounded-full p-2 overflow-hidden h-8 w-8 component--icon">
          {% include "svg/facebook.svg" %}
        </span>
      </a>
    {% endif %}
    {% if speaker.social.instagram %}
      <a href="{{ speaker.social.instagram }}" class="mx-1 hover:opacity-75 transition-all duration-300">
        <span class="inline-block bg-red text-white rounded-full p-2 overflow-hidden h-8 w-8 component--icon">
          {% include "svg/instagram.svg" %}
        </span>
      </a>
    {% endif %}
  </div>
</div>

That's 30-some lines of HTML, instead of more than 100! And if you want to change the presentation of a speaker, you only have to change it in one place.

Here are the changes I made to the home page. And here is what the resulting code looks like. At this point, you have a website that is not just easy to edit its structure or presentation, but it's also trivial to create new pages or adjust values on the home page without searching through a bunch of HTML code to get there.

Go ahead, try it! Create a new .md file. Give it a content-page layout (like Terms and Privacy) and title, then see it come to life!

Now that you now how to templatize a static HTML site and you can separate its content from presentation, you have two very important tools necessary to create rock solid websites with the Jamstack.

How to Convert Static HTML into Powerful Templates

Sean C Davis — Tue, 10 Aug 2021 17:54:52 +0000

You've gone through the design process in building a website, but now you're left with a handful of HTML files, along with some images and one or more CSS and JavaScript files, and you want to know what to do with them? Great! Let's talk about it.

You could deploy (i.e. publish) them, and eventually you will, otherwise no one will see your site. But deploying at this point is only going to get you so far. Inevitably, you’re going to want to change something with the site, and then what? Are you going to go all the way through the design process again? That sounds … painful.

A more sustainable approach is to take the static HTML content you were given and templatize it! Make it easier for you to work with — to make changes and to create new pages — when the need arises.

Why templatize a website?

This process has one major benefit — reusability. It makes the act of adjusting structure or content relatively trivial when compared with having to work with individual HTML files. That process can be further enhanced by separating content from presentation, so that when all you need to do is change a few words, you know right where to go and never have to mess with any of the code. (We have a detailed guide for that, but you’ll want to follow this one first.)

Consider two typical and simple content pages that most sites have — Terms & Conditions and Privacy Policy. They often have the same header and footer. It’s just the content in the middle of the page that differs (though the structure is likely similar). With those pages as separate HTML files, when you want to change something in a shared section — like the header or footer — you have to make the change in both places.

Expand that example to a site that has hundreds of pages with a similar structure (like blog posts). You’d be making that change hundreds or thousands of times, depending on the size of your site.

Following a software development principle called don’t repeat yourself (often referred to as DRY code), you could build the site with reusable pieces. Then you can make each change once and have it work everywhere. This is the magic of templatizing.

How templatizing works

When you have no templates and only HTML, the code on each page is unique and lives within its own file. Take a super simple site that has a home page (index.html), along with Terms (terms.html) and Privacy (privacy.html) pages. Your files look like this, where the dark blue boxes represent the code unique to each page:

Now let’s say the header on the home page is unique, but the header on the Terms and Privacy (i.e. interior) pages could be shared. And maybe the footer is the same on every page. Then we could make the header and footer into their own files and share them among multiple pages. Like this:

After creating these templates, changes to the footer only need to be made once and every page is updated to reflect the new changes.

This is a super simplistic view to show the power that even small abstractions can add to the longevity of your site. What we’ll do here will be slightly different in practice.

But how do we actually do this?

Static site generators are here to save the day!

Unfortunately, the web isn’t built to work with templates. We can’t actually just add a footer.html file and then tell every page to include it and have the browser work. In the end, the browser expects a single HTML file. In other words, the browser actually wants the non-templatized version of the two scenarios above.

But that’s not how we want to work. It’s tedious and prone to errors. If you want the footer to look the same everywhere, it should be the same everywhere. Otherwise, you risk making inadvertent changes on random pages. To help with this process we can use a tool called a static site generator.

There are many to choose from, perhaps including some popular ones you may have heard of, such as Jekyll, Gatsby, Next.js, or Hugo. Here we’re going to use one called Eleventy.

What’s so great about Eleventy?

Eleventy isn’t as popular as some of its competition, but it is supreme in its approach, which is simplicity. It is modeled after Jekyll, which provides a super low barrier to getting started when compared with the rest of the field. But unlike Jekyll, it is written entirely in JavaScript, which makes it a great fit for folks getting started with building websites.

Eleventy can also be super powerful. It takes more customization as your site grows in complexity, but it can grow with you. My site has several hundred pages and, at the time of writing this, is built entirely through Eleventy.

Okay, now are you convinced you should templatize those HTML files?

Hooray! Me too.

Let’s do it!

Step 1: The Static Site

Let’s work through this process together using a real-world example. We’re going to build a super simple version of the Unmute website. (Unmute is a real thing — a side project I’m working on with a few nerdy friends.)

We’ll have a unique home page along with two similar content pages, mimicking a Terms & Conditions page and a Privacy Policy page.

Let’s say the output of the whatever process you went through to obtain the files for your site left you with this mess:

css/styles.css to hold all your styling for the site.
js/bundle.js which handles the carousel at the bottom of the home page.
images as a house for all visual assets.
index.html to represent your home page.
content-page.html to represent all other internal pages.

Gotcha!
Take note that we’re making a big assumption here. We’re assuming that you won’t need to mess with your styles.css or bundle.js file. They were bundled up nicely for you by the freelance dev and you won’t need to make changes.

In the real world, it can be a tricky process to take big, bulky, obfuscated CSS and JS files and create a method for adding to them. And it gets even more complicated if you ever have to go back to the developer for changes after you’ve customized one of these.

As a result, we’re considering that process outside the scope of this guide. However, if this need arises, I’ve written a couple relatively simple guides on how to achieve this for both CSS and JavaScript.

Step 2: Create a New Project

Add the contents of the example static project to some directory on your machine. Here’s a link to download the larger example. After doing that, you can find the appropriate files in the 02-static-site/www directory. Move these files into a new directory on your machine. Your folder’s contents should look like this.

Step 3: Setup Eleventy

We’re going to assume you have a computer that is setup for web development. (If not, here’s a guide I wrote on setting up a new Mac for development.)

Once you’re ready to go, open up your command line or terminal application, change into the project directory and install Eleventy:

# navigate to your new project
cd path/to/my/project

# setup project for Eleventy
npm init -y

# install eleventy
npm install -D @11ty/eleventy

Note: If you’re tracking your changes with Git, this is a great spot to initialize the repository (git init) and add node_modules to a .gitignore file. If you’re not working with Git, don’t worry about this right now.

I like to add a few shortcuts after this installation to make working with sites more consistent across my machine. Open your package.json file and add the following to the scripts section:

{
    // ...
    "scripts": {
        "build": "eleventy",
        "clean": "rm -rf _site",
        "dev": "eleventy --serve --port 8000"
    }
}

Here’s what that file should look like at this point.

Now we have a way to run an Eleventy development server. Run the following command:

# start eleventy server
npm run dev

You should now have a dev server running at localhost:8000. And you’ll notice you now have a new _site directory in your project. Eleventy did this automatically for you.

You can open your browser and visit localhost:8000 to see your site, and … something doesn’t look right.

That’s because we didn’t tell Eleventy where our assets are. To do this, we must add an Eleventy config file at .eleventy.js:

module.exports = function (eleventyConfig) {
  // Copy static assets over to _site directory.
  eleventyConfig.addPassthroughCopy("css");
  eleventyConfig.addPassthroughCopy("images");
  eleventyConfig.addPassthroughCopy("js");
  // Return configuration object.
  return {}
};

Give the browser a refresh and everything should look good again!

Note: Throughout this guide, when making changes, you may have to clear the cache on your browser. Most browsers have an option to reload while clearing the cache for that site.

Step 4: The Default Layout

In Eleventy (and most static site generators), each page is wrapped in a layout. A layout is just a fancy term for a template. We’re going to begin by creating a default layout. This will be code that every page uses. We do this because, as we’ll soon see, Eleventy supports nesting layouts within one another. So what we’re going to do here is define our base layout.

In looking at index.html and content-page.html, the code that is consistent between the two is mostly contained within the <head> tag, but also includes a few lines at the very bottom of the file. Here’s what it looks like:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8"/>
    <meta name="viewport" content="width=device-width, initial-scale=1.0"/>
    <link rel="apple-touch-icon" sizes="180x180" href="/images/favicon/apple-touch-icon.png"/>
    <link rel="icon" type="image/png" sizes="32x32" href="/images/favicon/favicon-32x32.png"/>
    <link rel="icon" type="image/png" sizes="16x16" href="/images/favicon/favicon-16x16.png"/>
    <link rel="manifest" href="/images/favicon/site.webmanifest"/>
    <link rel="mask-icon" href="/images/favicon/safari-pinned-tab.svg" color="#5bbad5"/>
    <meta name="msapplication-TileColor" content="#da532c"/>
    <meta name="theme-color" content="#ffffff"/>
    <title>Stackbit Demo: Unmute</title>
    <link rel="stylesheet" href="/css/styles.css"/>
    <script>
      function onInit(callback) {
        if (typeof App !== "undefined")
          return callback();
        setTimeout(onInit, 250, callback);
      }
    </script>
  </head>
  <body class="font-default">

    <!-- This was the unique code -->

    <script src="/js/bundle.js"></script>
  </body>
</html>

Put that code in a new file: _includes/default.njk. And in the spot where the unique code existed, add the Nunjucks variable content. (More on this in a moment.) Your resulting file should look like this.

Three items are important to note in this new template:

We changed the file extension from .html to .njk. This means we’re using Nunjucks as our templating language. Eleventy supports a number of languages. Nunjucks is nice for staying with our theme of using JavaScript. And it’s also fairly minimalistic, which is nice for our example.
We added a single Nunjucks variable, content and passed a safe filter to it. content tells Nunjucks to render the content of the each page in that area. That means all of our page content will fall where you see {{ content | safe }}.
The _includes directory has some special characteristics that make it easy to … well, include files. Read more here.

Nothing has changed if we refresh the browser because we haven’t wired these up yet. When we go to the home page, we’re still just looking at index.html, which isn’t using our new layout yet.

Step 5: Wrap the home page in the new layout

Let’s adjust our home page to use the new layout.

Begin by changing index.html to index.njk. This is going to help us in the future when we want to use Nunjucks variables or tags in the page.

Now, remove all the code you extracted to create the layout and refresh your browser.

What the heck? You’re back to a page without styles. What happened?

What happened is that you removed your references to the CSS and JS files, which were in the <head> and near the bottom of the <body>, but you didn’t tell Eleventy to use a layout. So let’s change that. Add the following at the top of your index.njk file:

---
layout: default
---

Now refresh the browser and you should see your styles come back!

This style code — three hyphens, then some code, then three more hyphens — is called frontmatter. It’s like code before the code. It provides us a space to place meta information about that file that won’t ultimately be written to your browser screen.

The last thing to note here is that you will want to either remove or change the comments in your code. Nunjucks comments look a little different than HTML comments (although they will ultimately still be treated as comments, so you can ignore this if you’d like).

Any code between  is an HTML comment. Either remove these or change them to {# and #}.

In the end, this is what your new index.njk file should look like.

Step 6: Content Pages

Now let’s add a couple plain content pages. Our developer delivered a generic content page to us called content-page.html. We want to use that to create multiple pages. In this case, maybe those are Terms & Conditions and Privacy Policy pages.

To do this we first want to make our generic content page a layout. To do that rename content-page.html to content-page.njk and move it into the _includes directory.

Now, here’s a really weird and super cool thing about Eleventy — we can nest layouts in other layouts. What that means is that because the content page has code that we’ve already used in _includes/default.njk, we can reuse that with the content page layout.

To do this, remove the shared code from _includes/content-page.njk and specify the layout in the frontmatter at the top of the file.

Now you can create two new files, terms.njk (or terms.html — we aren’t doing anything special at this point) and privacy.njk. Add some content (I used a generator for the content and you can use what I have) and tell Eleventy to use the “content-page” layout:

layout: "content-page"

Now you can go to localhost:8000/terms and localhost:8000/privacy and you should see your nicely-formatted content.

Here are the links to these files at this stage:

Step 7: Extracting Shared Code

Now you’re in good shape and have a pattern to create new pages with a nice layout. Any new file you create with the “content-page” layout will now have a header and footer wrapping your content.

But, we still have an opportunity for some improvement. While we are using a layout to create multiple pages, which means we’re sharing code, we do still have some duplicated code. Now the home page (index.njk) and the content page layout (_includes/content-page.njk) have separate footers. Even though they look the same, you’d have to edit the content in both if you wanted to make a global change.

Let’s make that easier. Pull the footer content into its own file in the _includes directory, _includes/footer.njk (see here). Then remove that shared code from the home page and content page layout and replace it with the include Nunjucks tag:

{% include "footer.njk" %}

(See the new index.njk and _includes/content-page.njk files.)

Now you can make the change once and see it work everywhere. For example, you could add links to the terms and privacy pages.

Note: You could have chosen to include the footer directly in the default layout, or even reference in in the default layout. This is entirely up to you and your project. In this case, I’m accounting for some future template that won’t want the footer. But if we’re sure every page wants the footer, maybe it makes sense to put it in the default layout.

Other Opportunities

As your site grows, you’ll find other opportunities for improvement and abstraction (abstraction being the process of cleaning up code to be shared).

Shared Images

For example, many of the shapes you see on the pages are SVG elements. They are images, but represented with HTML code. And many of them are used more than once.

You could remove the <svg> elements and turn them into their own file in the _includes directory. (You could even put them in their own subdirectory so they are grouped together.) Then you can reference them with a Nunjucks tag.

Unique Titles

You may also run into the opposite issue at some point, where you’ll have code or content in a shared space, but you want it to be unique to some particular template. For example, you’ll want the title of the page (contained in the <title> attribute) to be unique to each page. But that code is nestled up in the default layout.

This is where Nunjucks variables and page frontmatter comes into play. Take the terms page, for example. You could add a title attribute in its frontmatter:

layout: "content-page"
title: "Terms & Conditions"

Then, in the _includes/default.njk layout, you can use the title variable.

<title>{{ title }}</title>

You could make the same change to the _includes/content-page.njk body so that you don’t have to write the <h1> on every page. Something like this:

<h1>{{ title }}</h1>

The world is your oyster when it comes to finding these efficiencies. It’s all about how you want to work and how you want the framework (Eleventy) to work for you.

I bucketed a handful of these changes together and did them all at once. Here are the changes I made and the files in the project at this point.

Wrapping Up

Phew! Take a break, give yourself a pat on the back. This process can be a harrowing one, especially the first time around. If you’ve made it through, you deserve a break and some ice cream.

To quickly recap. What we started with was a jumbled mess of HTML files, handed to you by some developer. We took those files and turned them into layouts (or templates) that we could use to quickly create new pages with the same design.

Where do we go from here?

I’m glad you let me ask that question on your behalf, because we have a great next step! Instead of using .html or .njk files for our content, you can double-down on frontmatter and markdown to create a truly powerful editing experience that gets all the code out of your way and keeps you focused on only the content.

And, lucky for you, we have a guide to keep this thing going and to walk you through that process.

But first, go get that ice cream.

Export Bear Notes to Markdown Files

Sean C Davis — Wed, 16 Jun 2021 13:10:20 +0000

I love Bear. The only other writing experience I've had that rivaled it was Ulysses. It's an absolute joy to use to author content with markdown.

The biggest issue I have with Bear is that it's not easy to do anything with the content outside of the application. But in the end, what you're authoring within Bear is just markdown (or it can be), and that content should be portable to wherever you can make it most actionable.

Here's a guide to get you started in building a workflow around your Bear notes by exporting them to markdown files.

The Example

We're going to write a single Node.js script that will export notes from Bear into markdown files with frontmatter metadata elsewhere on your machine.

In this specific example, we're going to target active notes (i.e. not trashed). And in the frontmatter, we'll add five key-value pairs:

id: The ID of the note within the database.
title: The title of the note, which comes from the note's main <h1> tag.
slug: A slugified version of the title that we'll use to name the file.
updatedAt: The last time the note was updated in the Bear app.
tags: An array of tags used within the note.

You can always adjust to store the data that you care about.

We're going to keep this super simple. We'll find the database, make a copy, and query it directly with the sqlite3 package. If I were going to build a real workflow that I wanted to last, I'd probably look at putting a more formal system together using an ORM like Sequelize or Prisma. But I want to keep us focused here.

One Quick Gotcha!

Since Bear is setup to author in markdown, we're not actually going to change the content in any way, although I'll mention where you could do that work if you would like.

However, the current version of Bear has its own style of markdown. Therefore, your mileage may vary if you're using that default markdown style. You may have to build a parser to convert it to actual markdown.

An easy way around that is to use markdown compatibility mode. Personally, I prefer to do this anyways because it has parity with authoring in other applications.

Step 1: Locate and Examine the Database

Before we write any code, we have to find the database. Fortunately, someone had the same question and there's an answer that I've found success with so far.

As part of this process we will copy the database each time before we run the command. But, before we begin, go ahead and manually copy the database to some more-easily accessible location. Then open it up to look at the content and schema. My favorite app for this is TablePlus, but there's also the popular DB Browser for SQLite.

You'll notice the database is littered with obscure table and column names. Here's what's important to know for this exercise:

The notes are located in a table called ZSFNOTE.
Within that table, ZTRASHED tells us if the note is active.
We can grab the other attributes we need (except tags) directly:
- Z_PK is the ID
- ZTITLE is the note title
- ZTEXT is the main body content
- ZMODIFICATIONDATE is the last updated date
Tags are located in ZSFNOTETAG where Z_PK is its ID and ZTITLE is its title.
Tags can be joined to notes through the Z_7TAGS table, where Z_7NOTES is the note's ID and Z_14TAGS is the tag's ID.

Phew! Are you already tired?

Hopefully not, because we haven't even written any code yet. If you are, take a break and rest that brain. And then let's write some code!

Step 2: Setup

Let's begin by creating a new directory for your project and setting it up. Here are the steps I take when adding a new JavaScript project.

Follow steps 1-3. In Step 3, add the following to your .gitignore file, in addition to node_modules:

database.sqlite
tmp/

For Step 4, we're going to work with a few libraries:

We can install them with one command:

$ npm install sqlite3 slugify js-yaml

We'll deal with the package.json scripts later on.

Step 3: Get Notes from the Database

First thing we're going to do is add a script to copy the database. Put this in a utils directory.

// utils/copyDatabase.js

const fs = require("fs")
const path = require("path")
const HOME = require("os").homedir()

const srcPath = path.join(
  HOME,
  "/Library/Group Containers/9K33E3U3T4.net.shinyfrog.bear/Application Data/database.sqlite"
)
const destPath = path.join(__dirname, "../database.sqlite")

if (!fs.existsSync(srcPath)) {
  console.error(`Could not find Bear database: ${srcPath}`)
  process.exit(1)
}

fs.copyFileSync(srcPath, destPath)
console.log(`Copied Bear database: ${destPath}`)

This will copy the database to the root of your project, or it will let you know that it couldn't find the database.

Then let's add our main script at index.js:

// index.js

const sqlite3 = require("sqlite3").verbose()
const db = new sqlite3.Database("./database.sqlite")
const { promisify } = require("util")

const query = promisify(db.all).bind(db)

const getNotesQuery = `
  SELECT
    Z_PK as id,
    ZTITLE as title,
    ZTEXT as body,
    ZSUBTITLE as subtitle,
    ZTRASHED as deleted,
    ZMODIFICATIONDATE as updatedAt
      FROM ZSFNOTE
      WHERE deleted = 0;`

const main = async () => {
  const notes = await query(getNotesQuery)
  console.log(notes)
}

main()
  .finally(() => {
    console.log("Done.")
  })
  .catch(err => {
    throw new Error(err.message)
  })

There's not much going on here. We connect to a database then run the SQL query shown to grab all the active notes and log them to the console.

There's some fanciness in here that protects us from madness later on. The SQLite library uses callbacks by default, which means it will run a function we give it after the query is complete.

A more modern JavaScript pattern is to use promises through async and await. That's what's going on here. We promisify the SQLite query and then run the main part of our code within an async function so that we can be sure each line is resolved before moving on to the next.

Next, now we can add the scripts to package.json so we can run these two commands:

{
  // ...
  "scripts": {
    "preexport": "node ./utils/copyDatabase.js",
    "export": "node index.js"
  }
}

While it looks a little funny, NPM has this handy method of running scripts before and after other scripts. When we run the export command, the preexport will automatically be run first. Thus, we copy the database every time, and then run the main script.

We could have combined this into a single script. But I like this because it feels like two very different actions — copy the database, then export markdown files from it.

You can test it all out now by running the export script:

$ npm run export

You should see content from your active notes logged to your console.

Here is the code at this point in time if you'd like to look. There may be a few extra things in there.

Step 4: Add Tags

Next, let's do the same thing (continue to log our notes to the console), but collect the tags.

The SQL query gets much more complex here. It looks like this:

SELECT
  'Note'.'Z_PK' AS 'id',
  'Note'.'ZTITLE' AS 'title',
  'Note'.'ZTEXT' AS 'body',
  'Note'.'ZTRASHED' AS 'deleted',
  'Note'.'ZMODIFICATIONDATE' AS 'updatedAt',
  'Tags'.'Z_PK' AS 'Tags.id',
  'Tags'.'ZTITLE' AS 'Tags.title',
  'Tags->NoteTag'.'Z_7NOTES' AS 'Tags.NoteTag.NoteId',
  'Tags->NoteTag'.'Z_14TAGS' AS 'Tags.NoteTag.TagId'
    FROM 'ZSFNOTE' AS 'Note'
    LEFT OUTER JOIN 'Z_7TAGS' AS 'Tags->NoteTag' ON 'Note'.'Z_PK' = 'Tags->NoteTag'.'Z_7NOTES'
    LEFT OUTER JOIN 'ZSFNOTETAG' AS 'Tags' ON 'Tags'.'Z_PK' = 'Tags->NoteTag'.'Z_14TAGS'
    WHERE 'Note'.'ZTRASHED' = 0;

This is doing some fancy join stuff. It will ultimately provide a line representing every instance of every tag in every note. That means that some rows returned will be duplicate notes. So we have to collect notes in a bit of a different way.

Our main() function gets updated to this, with some comments for context:

const main = async () => {
  // Reference to store note data.
  let notes = []
  // Query the database for notes and their tag. There will be a row returned
  // for each tag that a note contains.
  const queryResult = await query(getNotesQuery)
  // Get a unique set of IDs for the notes returned, as more than one row may
  // contain the same note.
  const noteIds = new Set(queryResult.map(res => res.id))
  // Collects all notes matching the passed ID and builds an object to represent
  // that note.
  const buildNoteObject = noteId => {
    // Find all rows from the query result matching the passed ID.
    const rows = queryResult.filter(row => row.id === noteId)
    // Return a null object if we were given a bad ID.
    if (rows.length === 0) return null
    // Extract relevant attributes out of the first row. Each of these is
    // assumed to be the same value in any row. We're picking the first one
    // because we know there will always be a first one.
    const { id, title, body, deleted, updatedAt } = rows[0]
    // Collect the tag names. Each row in the query result has its own unique
    // tag name, assuming the tag was only used once in the document.
    const tags = rows.map(row => row["Tags.title"])
    // Build the object and return it.
    return { id, title, body, deleted, updatedAt, tags }
  }
  // Loop through the notes and store the result in the notes object.
  noteIds.forEach(id => {
    notes.push(buildNoteObject(id))
  })
  // Log our result.
  console.log(notes)
}

Here is the file in its entirety at this point.

Step 5: Write Notes to File

Now we have just about everything we need to write the notes to file.

First, we need one more attribute to write the file — the filename! We'll get that by slugifying the title. Here's the change to make that happen. And the resulting file looks like this.

Now we're ready to write the note to file! Let's add another function inside our main() function to export the note. This will take a prepared note object, build its fronmatter and body, and then write it to file.

Then we can iterate over each note and run these functions.

First, add the js-yaml dependency to the top of your file:

const yaml = require("js-yaml")

Then add a few lines to create the export directory if it doesn't exist:

const outputDir = path.join(__dirname, "./tmp/export")
if (!fs.existsSync(outputDir)) fs.mkdirSync(outputDir, { recursive: true })

Note that we're going to write these files to a tmp/export directory within your current project.

And add some new lines to the main() function:

const main = async () => {
  // ...
  // Builds frontmatter and then writes the note to file.
  const exportNote = note => {
    const filePath = path.join(outputDir, `${note.slug}.md`)
    const { id, title, slug, body, tags } = note
    const frontmatter = yaml.dump({ id, title, slug, tags })
    const content = `---\n${frontmatter}---\n\n${body}`
    fs.writeFileSync(filePath, content)
    return { filePath, content }
  }
  // Loop through the notes and store the result in the notes object.
  noteIds.forEach(id => {
    const note = buildNoteObject(id)
    const { filePath } = exportNote(note)
    console.log(`Wrote note to file: ${filePath}`)
  })
}

Altogether, the file should now look like this.

Run the command again:

$ npm run export

And then check your tmp/export directory for these new files.

Now you should be able to take a note from Bear that looks like this:

# Hello World

#export-me

I'm so cool!

And the resulting file, tmp/export/hello-world.md, looks like this:

---
id: 203
title: Hello World
slug: hello-world
tags:
  - export-me
---

# Hello World

#export-me

I'm so cool!

Pretty cool, right!?

Next Steps

That's a start to something that could be super powerful and enable you to bring Bear into a more efficient workflow for you personally.

But on its own, it's not much. Maybe it's good enough for you. But you'll probably want to do more with it if it's really going to be effective. Here are some thoughts I have on what I might do next:

If this workflow grows in complexity for you, it'd likely be easier to work with an ORM. It could help keep the code a little simpler and keep you away from nasty SQL statements.
Instead of simply targeting notes that weren't trashed, you might want to target through some other means, like a specific tag or set of tags.
This doesn't extract files from the content. Images that you've dropped inline are still hidden away on your machine. You may want to extract them.
Syncing seems like a dangerous game to play here. But if you do proper backups of the database and know what you're doing, it might be something to explore.
I'm not protecting against duplicate tags or filenames, both of which could happen.
Write some tests if you're going to take this into a production-level workflow!

A Closing Question

I hope you got something out of this and it has helped you work more efficiently. I'll leave you with a question before I go:

Is there any value in this being a library of sorts? In other words, if instead of going through this entire process, you could have installed an NPM package and run something like this:

$ bearapp export

I've considered building something like this, but I'm not sure of the usage it would get. I'd love to know what you think or what you are doing with your Bear exporter.

Keep writing, keep exporting, keep being awesome!

Generate Dynamic JSON Pages with Next.js

Sean C Davis — Fri, 11 Jun 2021 10:28:15 +0000

Next.js is a super powerful tool for building scalable websites and web applications. Building dynamic web pages is no big thing with Next.

I had a scenario pop up in which I wanted to generate and deliver JSON pages. I wanted to retrieve the data from elsewhere and then output it to a file that didn't have to change between builds.

Limitations of Pages in Next.js

Part of the reason Next is equal parts powerful and easy to use is a result of the opinions it brings along. One such opinion is the way in which pages are delivered.

While there are options to fetch data prior to rendering a page, pages are rendered as React components. And they are wrapped in application-level components.

That means there isn't an easy way for me to follow the Next.js pages pattern to generate statically dynamic JSON pages.

Fortunately, I found two ways in which I could still accomplish what I wanted in two other ways.

The Setup

Before we walk through these two examples, I'm assuming you have a Next.js project ready to go. If you don't you can use create-next-app to start with the default template.

$ npx create-next-app

We're going to install a single dependency for this example, axios:

$ npm install axios

Once it seems you're ready to go, boot that development server. With the default template, that command is:

$ npm run dev

And the server runs at localhost:3000 in your browser.

If you had an existing Next project, you may have a different command to start the dev server and a different port on which the front end runs.

Method #1: API Routes

Now that you're up and running, let's look at our first option for generating JSON pages: API routes.

Okay, I lied. A little. Pages in next don't have to be React components. Next also supports what they call API routes. These are methods that run on the server side and return data back to the user. That feels like a really good use case for our scenario.

Let's pretend that we want to return a single random dad joke.

Starting Simple

As a quick introduction to API routes, let's first create a page at pages/api/joke.js with the following content:

// pages/api/joke.js

export default (req, res) => {
  res.status(200).json({ hello: "World" });
};

Now visit http://localhost:3000/api/joke in your browser, or make a GET request to that same URL through an API client. You'll see on screen (or in your client) the object we sent:

{ "hello": "world" }

Great!

Making it Dynamic

Now let's make it dynamic by adding axios and querying the icanhazdadjoke.com API:

// pages/api/joke.js

import axios from "axios";

export default async (req, res) => {
  const { data } = await axios.get("https://icanhazdadjoke.com/", {
    headers: { Accept: "application/json" },
  });

  res.status(200).json(data);
};

There's not much to that, really. We're asking the icanhazdadjoke.com API for a response and passing that response on to the user.

Now refresh your browser or make a request through your API client again and you'll be sent something classically witty like this:

{
  "id": "xXg3LZLZDd",
  "joke": "*Reversing the car* \"Ah, this takes me back\"",
  "status": 200
}

Note that if you were going to take this into production, you'd want to put some checks in place to guard against the icanhazdadjoke.com API being down or giving you something you didn't expect.

Method #2: Static File

The first method is powerful and all, but it's also forcing you into a solution in which you have to run that method (i.e. do some work, like hit another API) every time you want the data in this file.

Recall in the intro that I mentioned a nuance of not needing the file to change in between builds. Thus, the JSON file itself should be generated dynamically, but could be delivered statically. (This has all the makings of a static API.)

While we know we can't make pages as static JSON files, we could generate a static JSON file prior to building the site and serve it as a static asset.

Generate Script

To do that, let's put together a little script at scripts/getJoke.js which retrieves the dad joke and then writes it to a file at public/joke.json.

Note: We're putting it in the public directory because these files get copied over directly. A file at public/joke.json would be available at /joke.json on our website.

// scripts/getJoke.js

const path = require("path");
const fs = require("fs");
const axios = require("axios");

const filePath = path.join(__dirname, "../public/joke.json");

const main = async () => {
  const { data } = await axios.get("https://icanhazdadjoke.com/", {
    headers: { Accept: "application/json" },
  });

  fs.writeFileSync(filePath, JSON.stringify(data, null, 2));
};

main().then(() => console.log("Done."));

Notice this is very similar to our API function. But instead of returning the response, we write it to file.

To see it in action, you can run the script like this from the root of your project:

$ node ./scripts/getJoke

Check your public directory for a joke.json file.

{
  "id": "xXg3LZLZDd",
  "joke": "*Reversing the car* \"Ah, this takes me back\"",
  "status": 200
}

You can verify that it will be available on your website by visiting localhost:3000/joke.

Automate It!

To automate this process, we can add a pre script to hook into the appropriate script in our package.json file.

For example, let's say we want this to run before we run the npm run dev script. To do that, add a predev script to your package.json file:

// package.json

{
  // ...
  "scripts": {
    "predev": "node ./scripts/getJoke",
    "dev": "next dev"
    // ...
  }
}

Now, whenever you run npm run dev, your getJoke script will run, producing a new static file.

Try it out. Stop your server (if it's still running) and start it back up. Then visit localhost:3000/joke to see new content!

There are two methods for taking dynamic data and rendering it as JSON in your Next.js application. One updates itself on every request, the other on every build. Choose the best path for you and keep building cool things!

Simple JavaScript Pipeline with webpack

Sean C Davis — Fri, 28 May 2021 15:17:32 +0000

webpack Introduction

webpack has a reputation for being pretty gnarly. If you've dug through the code of an established project using webpack, it's likely mind-boggling at best. Shoot, take a look at the source code for Next.js — they have an entire directory to manage webpack configuration.

That complexity is due, in large part, to its power. webpack can do a lot.

Fortunately, the fine folks building this free and open source tool have been working hard to make each version a little easier to use than the previous. And now, you can start very simply, with very little config. Thus, you can justify the power of webpack in the smallest and simplest of projects.

What We're Going to Build

Let's do that. Let's build a super simple build pipeline to bundle multiple JavaScript modules together into a single file that we can load from any HTML page.

You can take a look at the full code example at any point if you get stuck.

Gotcha!

There's one big gotcha we'll have to overcome along the way. The output bundle is obfuscated and anonymous. That means we can't access it by default. And even if we could, we likely wouldn't know how to navigate it.

In our case, we want to access our bundled code from external places (like an HTML file), so we're going to load our main exports into an App object that we can then access from that main HTML file.

Specifically in this example, we want to be able to call App.Logger.sayHi() and see it print "Hi!" to the console.

Let's go!

Step 1: Setup

If you have a project ready to go, great! If not, feel free to follow my steps to get started, with the following notes:

These are the dependencies we're going to use:

http-server
webpack
webpack-cli

And here are the scripts to add to package.json:

// package.json

{
  // ...
  "scripts": {
    "build": "WEBPACK_ENV=production webpack",
    "dev": "webpack",
    "serve": "http-server dist -p 8000"
  }
}

Step 2: Add JavaScript Files

Now let's add a couple JavaScript files. First, our Logger at src/modules/logger.js:

// src/modules/logger.js

const sayHi = () => {
  console.log("Hi!")
}

export { sayHi }

And our main file (src/main.js), which will be responsible for exporting the Logger object.

// src/main.js

import * as Logger from "./modules/logger"

export { Logger }

Note: If this were a bigger web project where you have more files in your src directory, you'd probably want to put these files in some other nested place, like a js directory.

Step 3: Add webpack Config

Next, let's add our webpack config. This code example is commented so you can see what's going on:

// webpack.config.js

const path = require("path")
// Used to determine whether to watch the files or build.
const env = process.env.WEBPACK_ENV || "development"

module.exports = {
  // The main file for the bundle.
  entry: "./src/main.js",
  output: {
    // Name of the bundle file.
    filename: "bundle.js",
    // Directory in which the bundle should be placed.
    // Here we're saying `dist/js/bundle.js` will be our bundled file.
    path: path.resolve(__dirname, "dist/js"),
    // These two library items tells webpack to make the code exported by main.js available as a variable called `App`.
    libraryTarget: "var",
    library: "App"
  },
  mode: env,
  // If we're in development mode, then watch for changes, otherwise just do a single build.
  watch: env !== "production"
}

To summarize:

main.js is the primary targeted file, which will be bundled to dist/js/bundle.js.
The exports from main.js will be available globally in an App variable.
When the WEBPACK_ENV is set to something other than production, webpack will watch for changes. When WEBPACK_ENV is set to production, it will build the bundle and then stop running. We're setting this variable automatically in the scripts added to package.json.

Step 4: Add HTML

Now let's add a simple index.html file to the dist directory, which is where the bundled JS file is going to be placed.

<!-- dist/index.html -->

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Simple Webpack ES6 Pipeline</title>
  </head>
  <body>
    <p>Hello there.</p>

    <script src="/js/bundle.js"></script>
    <script>
      App.Logger.sayHi()
    </script>
  </body>
</html>

In most real-world cases, you're probably going to have some sort of a build step that would place the file here, likely provided by the static site generator or other framework you're using.

In this case, we're placing the file in here as though it was already built so we can stay focused and not worry about all that setup.

Step 5: Run it!

We actually have two commands we have to run to get this to work. First, build the JavaScript bundle:

$ npm run build

Then you can run the web server:

$ npm run serve

Now visit localhost:8000, open your browser's console, and you should see the message "Hi!" printed!

If you want to to make changes to JavaScript and see them reflected without reloading your web server, you can use two terminal tabs. Run npm run serve in one to run the web server, and npm run dev in the other, which will watch for JavaScript changes and rebuild.

References

Full code example

That's all it really takes to get up and running with webpack. Starting with a simple foundation is the key to understanding and working with webpack. Now you can build on this base and create something truly awesome and unique to you.

Use Netlify Functions to Send Email Notifications

Sean C Davis — Thu, 20 May 2021 13:49:48 +0000

While Netlify supports email notifications around triggers and features within its system (e.g. deploy events, form submissions), you may want to add custom email notification triggered by actions from users on your site.

Notifications sound tricky, though, don't they?

Well, they don't have to be. We can use Netlify Functions, along with some email-sending service, to make that process a walk in the park.

To keep this example as simple as possible, we're going to use Nodemailer with Ethereal as our email sending service. That means we will have to configure very little, but the emails will be caught, not sent.

Let's dive into a quick example! (You can view a full version of the example code here at any time.)

Step 1: Setup Project

Before we get started, you'll want a new project. Create a directory for your project. My first steps in a new project are usually these:

$ npm init -y
$ echo 'node_modules' >> .gitignore

Then you can install the only dependency we need, Nodemailer:

$ npm install nodemailer

Step 2: Add the Netlify Function

Let's add a Node-based Netlify function that will handle sending our email message.

The function will expect a stringified JSON object as the body, containing two key-value pairs:

email: The email address to use to send the message.
body: The message to use as the body of the email.

Place the following code in netlify/functions/send-email.js.

Note: netlify/functions is the default location for functions. If you have overridden this value for you site, be sure to place the file in the appropriate location.

Also note that we're having Nodemailer make use of Ethereal, which is a mail-catching service. That means none of the messages are going to actually be delivered. I'll talk a little more about this when we get to Next Steps at the end of this post.

// netlify/functions/send-email.js

const nodemailer = require("nodemailer")

exports.handler = async function (event, context, callback) {
  // Parse the JSON text received.
  const body = JSON.parse(event.body)

  // Build an HTML string to represent the body of the email to be sent.
  const html = `<div style="margin: 20px auto;">${body.body}</div>`

  // Generate test SMTP service account from ethereal.email. Only needed if you
  // don't have a real mail account for testing
  let testAccount = await nodemailer.createTestAccount()

  // create reusable transporter object using the default SMTP transport
  let transporter = nodemailer.createTransport({
    host: "smtp.ethereal.email",
    port: 587,
    secure: false, // true for 465, false for other ports
    auth: {
      user: testAccount.user, // generated ethereal user
      pass: testAccount.pass // generated ethereal password
    }
  })

  try {
    // send mail with defined transport object
    let info = await transporter.sendMail({
      from: '"☁️ The Cloud ☁️" <thecloud@example.com>',
      to: body.email,
      subject: "New Form Submission",
      text: body.body,
      html: html
    })
    // Log the result
    console.log(info)
    callback(null, { statusCode: 200, body: JSON.stringify(info) })
  } catch (error) {
    // Catch and log error.
    callback(error)
  }
}

This will log the result to the console, regardless of whether it is successful or not (callback(error) will print feedback), so you can have an idea of what's going on.

Step 3: Wire up the Front End

Next, let's build a simple HTML page that gives you the ability to set the email and body fields. Then we'll add just a little JavaScript to make it all work.

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Send Email Notifications with Netlify Functions</title>
  </head>
  <body>
    <form onsubmit="submitForm(event)">
      <div style="margin-bottom: 1rem">
        <label for="email">Email Address</label>
        <input
          id="email"
          type="text"
          name="email"
          placeholder="Where should I send the message?"
          required
        />
      </div>

      <div style="margin-bottom: 1rem">
        <label for="body">Message</label>
        <textarea id="body" name="body" cols="30" rows="10" required></textarea>
      </div>

      <input type="submit" value="Send email" />
    </form>

    <script>
      function submitForm(event) {
        // Stop the browser's default behavior.
        event.preventDefault()
        // Retrieve data from the form.
        const formData = new FormData(event.target)
        const request = new XMLHttpRequest()
        // Convert data to JSON object.
        var jsonData = {}
        formData.forEach((value, key) => (jsonData[key] = value))
        // Send the data to the Netlify function.
        request.open("POST", "/.netlify/functions/send-email")
        request.send(JSON.stringify(jsonData))
        // Clear the form.
        alert("Email request submitted!")
        event.target.reset()
      }
    </script>
  </body>
</html>

WARNING! It's not good practice to use an onsubmit attribute on a form element to call a global function when submitting a message. This is just a very simple example for demonstration purposes.

Not Getting Emails?

If you're not getting the email messages in the inbox you specified, it's because we're not actually delivering them!

WTF?

Yes. Ethereal, which we're using as our email server, is a mail-catching service, which means that it catches the mail requests and lets you read them, but it doesn't actually send them.

If you want to see the messages being caught, then instead of creating a test account in the function, go to Ethereal and click Create Ethereal Account. Then plug the username and password in. You can then visit the inbox for that account and see everything that was caught.

Next Steps

Being that this is just the beginning of something that you'd actually put into practice, here are some ideas on where you could take it from here to get it ready for production:

Instead of submitting through an HTML page and relying on user input to control the recipient and the message, you could submit through an API request platform, like Postman.
Add a legit emailing service in place of Nodemailer and Ethereal, such as Mailgun or SendGrid.
Adjust which values you accept and which you hard-code, like subject or from email.
Add some validation or authentication so not just anyone can trigger the action.
Be mindful of limitations or quotas on your accounts. Netlify Functions enables you to have a generous number of requests per day. Email services are generally not as generous, and you could run up a bill quickly.

No matter where you go with it, I'd love to learn more about your approach or any questions you have. Let's chat.

References

Demo code

Testing sessionStorage and localStorage with Selenium (Node)

Sean C Davis — Mon, 17 May 2021 18:05:56 +0000

I recently added a feature to the Grouparoo site that is using sessionStorage to send analytics data. Being that it's an important feature, we should write test(s) to cover the use case(s), right?

Okay, fine. Let's do it!

This website is a Next.js application that uses Jest as the test runner and Selenium WebDriver for integration test help.

What I wanted to do with Jest and Selenium was to read from sessionStorage after visiting a series of pages. After a bit of perusing, I finally uncovered a (goofy) way to achieve what I wanted.

We can use the executeScript method to run a JavaScript expression and capture the result. Our test looks like this:

declare var browser: any;

async function getSessionItem(key) {
  return await browser.executeScript(
    `return window.sessionStorage.getItem("${key}");`
  );
}

test("stores page history in the session data", async () => {
  await browser.get(url + `/docs/config`);
  expect(await getSessionItem("prevPath")).toBe("null");
  expect(await getSessionItem("currentPath")).toBe("/docs/config");
  await browser.get(url + `/meet`);
  expect(await getSessionItem("prevPath")).toBe("/docs/config");
  expect(await getSessionItem("currentPath")).toBe("/meet");
});

Here are a few of the key items to note:

You must return the JavaScript expression or you'll end up with undefined.
It's a much cleaner approach to run tests as async functions so you can use await to retrieve the result of the script, rather than ending up in a nightmarish Promise chain.
browser is often referred to as driver in other documentation and implementations. This comes from the library we're using to connect Jest and Selenium.

This now works like a charm! You could take a similar approach if you wanted to read from any other JavaScript object, including localStorage.

Change the Primary Key Data Type with Sequelize

Sean C Davis — Wed, 12 May 2021 15:45:43 +0000

We recently adjusted how we handle primary keys. Previously they were UUIDs with a max length of 40 characters. With our Declarative Sync feature, we allow developers to set primary key values from their configuration files. Thus, we needed to lengthen the maximum number of characters allowed on primary keys in our database.

Seems simple, right?

I thought so, too. We're using Sequelize as our ORM tool, and I found a handy changeColumn method.

So that's what I did. It looked like this:

await migration.changeColumn(tableName, columnName, {
  type: DataTypes.STRING(191),
});

Note: In these examples, I'm accessing Sequelize methods through an object called migration. This is because we use Actionhero to run our database migrations. Your objects will look different, but the methods on them should be the same.

I first tested with SQLite and voila! It did exactly as I expected. All the primary keys were changed and working just lovely.

Changing PostgreSQL Primary Keys

Since we support both Postgres and SQLite as our application database, I moved on to test in Postgres, and that's when, instead of the database being properly migrated, I was presented with this lovely message:

column "id" is in a primary key

I thought: Yes, true. That is correct. And ... ?

It turns out Sequelize doesn't handle this action well with Postgres. After going down a rabbit hole in playing around with constraints, I ended up just writing the SQL statement directly. It looked something like this:

const query = `ALTER TABLE "${tableName}" ALTER COLUMN "${columnName}" SET DATA TYPE varchar(${maxIdLength}); `;
await migration.sequelize.query(query);

That worked!

Consistency is Awesome! (SQLite is Weird.)

It made sense to me to try to use the same approach with both databases. So I tried my Postgres solution with SQLite.

It didn't work. (Sensing a theme yet?)

That seemed odd. But, of course, we already know that SQLite is weird. And it turns out SQLite's ALTER TABLE methods are extremely (and intentionally) limited.

Which meant I was stuck with two solutions. And when that happens, we tend to look at the current dialect and execute the appropriate code. And that's why this is the weird function that alters the primary key column in both Postgres and SQLite:

const changeColumn = async (tableName, columnName) => {
  if (config.sequelize.dialect === "postgres") {
    const query = `ALTER TABLE "${tableName}" ALTER COLUMN "${columnName}" SET DATA TYPE varchar(${maxIdLength}); `;
    await migration.sequelize.query(query);
  } else {
    await migration.changeColumn(tableName, columnName, {
      type: DataTypes.STRING(191),
    });
  }
};

You can see the complete set of changes that came along with this code in this pull request.

Understanding Types with SQLite and Node.js

Sean C Davis — Thu, 29 Apr 2021 19:27:11 +0000

Two fun facts about SQLite:

The initial release was more than 20 years ago!
It is the most widely used database (and likely one of the most widely deployed pieces of software).

And here are a few of my opinions on SQLite:

It's super cool.
We don't talk about it enough.
It's actually really easy to use (which is likely why it's so widely used).
It is a little quirky.

So let's talk about this super cool thing. Let's look at how easy it is to use with Node.js before spending some time uncovering its quirks, which mostly have to do with how it handles data types.

A Quick SQLite Example with Node.js

If you're able to run Node on your machine and install packages via NPM, then you can very easily create and manipulate a SQLite database. Let's go through an example in which we do just that — create a database, users table, and a few users.

(Note: You can also work with SQLite without Node or NPM. That's just what we're using for our examples here today.)

To get started, create a new directory for your project:

$ mkdir my-sqlite-project
$ cd my-sqlite-project
$ npm init -y

Pro Tip: This is the point at which I usually drop a .gitignore file that ignores the node_modules directory. And then I run git init so I can start tracking my changes.

Now that you have an empty directory, let's install our dependencies:

$ npm install sqlite3 faker

Notice this installs Faker.js in addition to the SQLite library. Faker is a nice and simple library that will help us add random data when we create new users.

Next, create an index.js file with the following code:

const sqlite3 = require("sqlite3").verbose();
const faker = require("faker");
const path = require("path");
const { promisify } = require("util");

// Create a new database named mydb.sqlite in the root of this project.
const dbFilePath = path.join(__dirname, "mydb.sqlite");
const db = new sqlite3.Database(dbFilePath);

// Use the promise pattern for SQLite so we don't end up in callback hell.
const query = promisify(db.all).bind(db);

// SQL query for creating a users table if it doesn't already exist.
const createTableQuery = `
  CREATE TABLE IF NOT EXISTS users (
    "id" INTEGER PRIMARY KEY,
    "email" TEXT NOT NULL,
    "first_name" TEXT,
    "last_name" TEXT,
    "created_at" TEXT
  )
`;

// Generate user attributes using faker.
const newUser = {
  email: faker.internet.email(),
  first_name: faker.name.firstName(),
  last_name: faker.name.lastName(),
  created_at: Date.now(),
};

/**
 * Run an INSERT query on some given table and insert the given object.
 */
const create = async ({ table, object }) => {
  const keys = Object.keys(object).join(",");
  const values = Object.values(object)
    .map((v) => `"${v}"`)
    .join(",");
  const res = await query(`INSERT INTO ${table} (${keys}) VALUES (${values})`);
  return res;
};

/**
 * Read all records and all their columns from some given table.
 */
const read = async ({ table }) => {
  const res = await query(`SELECT * FROM ${table}`);
  return res;
};

/**
 * The main controller of this script. This is inside an async function so we
 * can use the promise pattern.
 */
const run = async () => {
  // Create users table if it doesn't exist.
  await query(createTableQuery);
  // Create a new user.
  await create({ table: "users", object: newUser });
  // Read all the users.
  const users = await read({ table: "users" });
  // Print to the console.
  console.log(users);
};

run();

Then run the script:

$ node index.js

After you do that, two things should happen:

You should see some output in the console that is an array containing a single user with the values you just randomly generated. Something like:

[
  {
    id: 1,
    email: "Dawson39@yahoo.com",
    first_name: "Dorris",
    last_name: "West",
    created_at: "1619034411275",
  },
];

You should have a new file in your project called mydb.sqlite. This is your database!

You can open up your database file with a SQLite browser. I'm a big fan of TablePlus, which has a free version. But you can also use the simple-but-popular DB Browser for SQLite.

If you open your users table, you should see the same record represented as was printed to the console.

See how easy that was?

Now that we have that basic example in place, let's keep it around while we dig into some of SQLite's quirks.

SQLite Type Quirks

SQLite is weird. Well, it's not really weird, it's just ... simple. So it feels weird in comparison to other beefy (and structured) databases like PostgreSQL or MySQL.

Most of the quirks with SQLite have to do with how it handles types. And that's in large part because of how SQLite stores its data (as a file on your file system). That's right. That mydb.sqlite file you created in the simple example above is your entire database. Cool, right?

Let's dig in to the basics of SQLite's data types before we look at how we can better work with them when writing a program with Node.js.

Understanding SQLite Types

SQLite only has five data types:

NULL: The value is a NULL value.
INTEGER: The value is a signed integer, stored in 1, 2, 3, 4, 6, or 8 bytes depending on the magnitude of the value.
REAL: The value is a floating point value, stored as an 8-byte IEEE floating point number.
TEXT: The value is a text string, stored using the database encoding (UTF-8, UTF-16BE or UTF-16LE).
BLOB: The value is a blob of data, stored exactly as it was input.

What the heck does that mean? Reading between the lines, it means that the data in your database is either going to be (in JavaScript speak) a string or a number, at least for the majority of cases.

That's right. That means there are no dates or booleans in SQLite. WTF?

SQLite does not store date objects

SQLite has date functions, but it will ultimately store the actual value as either a string or number.

For example, I could run this query against the users table from above:

INSERT INTO users (email,created_at) VALUES ('hello@example.com',date('now'))

And the value will be stored as 2021-04-21, as a string.

SQLite also doesn't store booleans

Instead, booleans used in SQL statements are converted to either 0 or 1. In other words, true in an insert statement becomes 1 when it's stored in the database.

How Type Affinity Works in SQLite

SQLite uses what is called dynamic typing. Where most other databases set the data type of some given value by the column definition, SQLite looks more directly at the value itself.

But, there is some magic happening through what SQLite calls type affinity on columns in a database. You can set a type for a column, but it's really just a recommended type. SQLite will do its best to convert the value to that type, but it may not always work as you'd expect.

The goofy thing here is that aside from matching a type affinity name when setting the type of a column, the name is virtually meaningless. It can be anything. Here are a set of types that map to type affinities in SQLite.

Let's go through some specific examples to see what happens when we use the Node sqlite3 library to interact with a SQLite database.

Type Affinity Examples with SQLite and Node.js

Let's go through a few examples together to show how type affinity works with SQLite and how we can use JavaScript to gain more control over types by manually casting them. Then we'll close with a cool approach for getting around all this nonsense.

Using the example from the beginning of the post, let's add a few columns to our users table. Edit the createTableQuery in index.js:

const createTableQuery = `
  CREATE TABLE IF NOT EXISTS users (
    "id" INTEGER PRIMARY KEY,
    "email" TEXT NOT NULL,
    "first_name" TEXT,
    "last_name" TEXT,
    "created_at" TEXT,
    "a" TEXT,
    "b" INTEGER,
    "c" CHEESE
  )
`;

Here we've recreated the users table with three new columns:

a of type TEXT. This matches the TEXT affinity.
b of type INTEGER. This matches the INTEGER affinity.
c of type CHEESE. This has no affinity.

Then, let's set a, b, and c on our newUser object to be a random number.

const newUser = {
  email: faker.internet.email(),
  first_name: faker.name.firstName(),
  last_name: faker.name.lastName(),
  created_at: Date.now(),
  a: faker.datatype.number(),
  b: faker.datatype.number(),
  c: faker.datatype.number(),
};

Before we load the content into the database, let's check the type. And when we retrieve it from the database, we'll check the types again.

const run = async () => {
  await query(createTableQuery);

  const user = newUser();
  // Log the data types going into the database.
  Object.entries(user).map(([key, value]) =>
    console.log(`${key}: ${typeof value}`)
  );
  await create({ table: "users", object: user });

  const users = await read({ table: "users" });
  // Log the types coming out of the database.
  Object.entries(users[0]).map(([key, value]) =>
    console.log(`${key}: ${value} (${typeof value})`)
  );
};

Delete the database mydb.sqlite and run the script:

$ node index.js

Notice what you see on the console. Before the data went into the database, each of our new values was a number:

a: number
b: number
c: number

That's what we should have expected, considering we set those attributes as random numbers.

For me, when they came out of the database, they looked different:

a: '91057' (string)
b: 9807 (number)
c: 31711 (number)

Notice that a is now a string. Why? Because we set the column's type affinity to a value that SQLite will use to try to convert to a string. Meanwhile, nothing happened with the CHEESE column because it has no affinity, so it left it as a number.

Try again, but with strings

If we do that again, but change faker.datatype.number() to faker.datatype.string(), we will see something slightly different.

The types going in are all strings:

a: string
b: string
c: string

But coming out they are still all strings:

a: 'i_`kneb8|]' (string)
b: '/@adUCVEV3' (string)
c: '@8eMpbKoFk' (string)

The reason b wasn't converted to a number is because SQLite doesn't know how to convert that random string to a number. There's no logical choice for what the number should be. So it leaves the original value as is.

Using a number as a string

Now, one last time. Let's use a number again, but let's convert it to a string. Change faker.datatype.string() to faker.datatype.number.toString().

Now, going in we still have all strings:

a: string
b: string
c: string

But coming out, it looks a little different:

a: '42995' (string)
b: 22072 (number)
c: 3466 (number)

Whoa! Something weird happened this time. Now our affinities match for a and b as they did in the first example. This is example SQLite knows how to turn "22072" into an integer.

But c changed from being a string to a number. That's because SQLite is trying to do the best it can't with what it is given. And in this case it figured it could turn the string into a number, so it did that. The only way to have kept it as a string would have been to use an appropriate affinity mapping value.

Typecasting SQLite Queries using Node.js

This quirk is pretty interesting and clever, but it's dangerous in real-world scenarios when you don't have bounds around how it can behave. You don't want your database turning strings into numbers without knowing why that's happening.

One approach is to write your own getters and setters that typecast values more strictly. That way your program has more control over what is going into and coming out of the database.

For example, consider a function that takes user data retrieved from the database and parses a created_at field to convert it from an integer into a JavaScript date object. That might look something like this:

const normalizeUser = (obj) => {
  return { ...obj, created_at: new Date(parseInt(obj.created_at)) };
};

The list goes on and on, but that could be a lot to manage as your application grows.

Or Use an ORM!

As your application grows in size and complexity, it likely will make sense to make use of some other library that can handle this typecasting logic for you. These libraries are called ORMs, which stands for object-relational mapping. That's a fancy term for a tool that makes it easier to move from database to database without needing to change the syntax used to manipulate data in the database.

At Grouparoo, we use Sequelize to interact with our application database.

Here's a simple example that does essentially what we were doing in the introductory example using Sequelize:

const faker = require("faker");
const path = require("path");

const { Sequelize, Model, DataTypes } = require("sequelize");

const sequelize = new Sequelize({
  dialect: "sqlite",
  storage: path.join(__dirname, "mydb.sqlite"),
});

const userAttrs = {
  email: faker.internet.email(),
  firstName: faker.name.firstName(),
  lastName: faker.name.lastName(),
};

class User extends Model {}

User.init(
  {
    email: {
      type: DataTypes.STRING,
      allowNull: false,
    },
    firstName: DataTypes.STRING,
    lastName: DataTypes.STRING,
  },
  { sequelize, modelName: "user" }
);

const run = async () => {
  await sequelize.sync();
  const users = await User.findAll();
  users.map((user) =>
    Object.entries(user.dataValues).map(([key, value]) =>
      console.log(`${key} (${typeof value}): ${value}`)
    )
  );
};

run();

Notice here that I'm more explicit about the column types. Sequelize then handles the typecasting for me. For example, running this once, this was the object returned to me from the database:

id (number): 1
email (string): Erling_Friesen50@gmail.com
firstName (string): Easton
lastName (string): Kub
createdAt (object): Tue Apr 20 2021 13:50:17 GMT-0400 (Eastern Daylight Time)
updatedAt (object): Tue Apr 20 2021 13:50:17 GMT-0400 (Eastern Daylight Time)

Notice that it actually sent date objects for createdAt and updatedAt. (Also notice that Sequelize handled setting those values when I created the record. I didn't have to do anything.)

There are plenty of other ORMs out there. Sequelize is among the most popular for Node. Another I came across recently is Prisma.

That's all for this exploration through how types work with SQLite when using a Node.js library. I hope you learned something!