Forem: Steven Mercatante

Use Custom Paths in Gatsby

Steven Mercatante — Mon, 18 Nov 2019 16:00:38 +0000

This post was originally published at stevemerc.com

By default, Gatsby generates URLs for posts from their filename (or the directory name, depending on how you organize your posts). For example, if your post is named gatsby-custom-paths.md, its URL will be https://mysite.com/gatsby-custom-paths.

But what if you want to use a different URL (say, use-custom-paths-in-gatsby) without changing the post's filename? That's the problem I ran into. I have a specific naming and organization convention for my posts that should be independent of their URL.

Luckily, we can change how these URLs are generated.

The Solution

First, add a path attribute to the frontmatter for any post whose URL you want to customize. Here's the frontmatter for this post:

---
title: Use Custom Paths in Gatsby
path: 'use-custom-paths-in-gatsby'
---

Make sure you restart the Gatsby dev server after adding new fields to frontmatter for the first time, otherwise you won't be able to query them.

You can control how page URLs are generated by overriding the onCreateNode function in gatsby-node.js. Here's what my implementation looks like (yours may differ; the important thing is to see that slug is generated by createFilePath.)

// gatsby-node.js

exports.onCreateNode = ({ node, getNode, actions }) => {
  const { createNodeField } = actions

  if (node.internal.type === 'Mdx') {
    const slug = createFilePath({ node, getNode })
    createNodeField({
      node,
      name: 'slug',
      value: slug,
    })
  }
}

Remember when we added that path attribute to our post's frontmatter ealier? We're gonna refer to it here. Update your slug definition to use node.frontmatter.path or fallback to the createFilePath invocation.

// gatsby-node.js

exports.onCreateNode = ({ node, getNode, actions }) => {
  ...

  if (node.internal.type === "Mdx") {
    let slug = node.frontmatter.path || createFilePath({ node, getNode })
    ...
  }
}

Restart your Gatsby dev server again, and it'll regenerate the slugs. If you added a custom path to any of your posts's frontmatter, you'll now be able to access them via that custom path. In our case, we can now access this post by going to https://stevemerc.com/use-custom-paths-in-gatsby.

Caveats

One of the nice things about having Gatsby autogenerate slugs/paths is that you'll never have two identical URLs for posts since you can't have identically named files or directories. Using this custom method, you need to make sure you don't use the same path for different posts.

👋 Enjoyed this post?

Join my newsletter and follow me on Twitter @mercatante for more content like this.

How to Map Over a Tree in JavaScript

Steven Mercatante — Tue, 12 Nov 2019 16:09:23 +0000

Steven Mercatante

@mercatante

🔥 You can map over a tree by using lodash's cloneDeepWith function and passing it a customizer function

15:51 PM - 12 Nov 2019

0 1

Check out the runnable example below:

Should You Learn Redux?

Steven Mercatante — Thu, 07 Nov 2019 23:25:38 +0000

This post was originally published at stevemerc.com

Dan Abramov

@dan_abramov

Reading some Redux example code I wrote four years ago and I have no effing idea what's going on there

22:47 PM - 04 Nov 2019

398 2771

(In case you're unaware, Dan co-created Redux.)

Ever since React's context API became stable, and especially after the release of hooks, a boatload of posts told us that Redux was on its deathbed. Who needs Redux when we've got the context API? Who wants to write all that boilerplate code? It's way too verbose when all I want to do is push an item to an array!

I don't necessarily disagree with those sentiments. Redux can be a pain to use, but sometimes it's the right choice for a project.

I want to pause here and say that I'm really grateful for Redux. Its creators and maintainers have spent a ton of time and energy in making a powerful tool. I've made good money working on projects that use it. And it's probably single handedly responsible for teaching more people how reduce works than anything else.

So, Should You Learn Redux?

When a post asks if you should learn something, the answer is usually "it depends." But not this post. Yes, you should learn Redux.

Regardless of the flak it gets, it's still the most popular state management library for React apps. It's battle tested, has incredible tooling, and there's a large community around it. If you've got a question, someone's already answered it.

This means that you're likely going to encounter and work on a project that uses it. And no matter how much you protest that there's a better option (maybe there is, maybe there isn't), you're gonna have to use it. So, rather than fight it, embrace it.

The official docs are great. Spend some time with them. Redux isn't the easiest library to grok, which is why knowing how it works is useful before you find yourself neck deep working in a project that uses it.

I'd also recommend learning how Reselect and Immer work. They're not necessary to use Redux, but they can improve performance and readability, and it's not uncommon to see them used alongside Redux.

[Edit: also check out Redux Starter Kit, mentioned by Mark Erikson in the comments below.]

Should You Use Redux?

This is where I respond with "it depends". Personally, I've stopped using Redux for new projects. Local state and the context API can get you very far. And when I need something more powerful, I use MobX. The balance it strikes between Redux and not-Redux is great. But, as a consultant, I often find myself working on projects that use Redux, and knowing how it works is essential.

In the end, the choice to learn Redux and the choice to use it are two different things. I think it's at least worth learning.

👋 Enjoyed this post?

Join my newsletter and follow me on Twitter @mercatante for more content like this.

Publish Posts After a Date in Gatsby

Steven Mercatante — Thu, 07 Nov 2019 00:38:16 +0000

This post was originally published at stevemerc.com

I like writing articles ahead of time and only have them listed on my site based on whether the current date is greater than or equal to the article's date attribute. This means I get to write a bunch of content at once, but release it over time so there's a constant stream of new material on my site without having to rush to write it the night before.

Note: This lesson builds upon my previous Add a Published Field to Your Gatsby Posts to Control Their Visibility article.

The Solution

First, make sure your article/post/whatever has a date attribute in its frontmatter. For example, here's this post's frontmatter:

---
title: Publish Posts Based After a Date in Gatsby
date: '2019-10-08'
published: true
---

I'm already filtering my query to check the article's published attribute, so I need to add an additional check against its date. Let's add a filter that only returns articles whose date is less than or equal to the current date.

query AllArticles($currentDate: Date!) {
  allMdx(
    filter: {
      frontmatter: { published: { eq: true }, date: { lte: $currentDate } }
    }
    sort: { fields: [frontmatter___date], order: DESC }
  ) {
    edges {
      node {
        fields {
          slug
        }
        frontmatter {
          date(formatString: "MMMM DD, YYYY")
          title
        }
      }
    }
  }
}

But now we've gotta pass in that $currentDate variable, so open up your gatsby-node.js file and add the following:

// gatsby-node.js

exports.onCreatePage = ({ page, actions }) => {
  const { createPage, deletePage } = actions
  deletePage(page)
  createPage({
    ...page,
    context: {
      ...page.context,
      currentDate: getCurrentDate(),
    },
  })
}

onCreatePage is provided by Gatsby and is called whenever new pages are created. By overriding it here, we can pass additional data (currentDate) via its context object that'll be made available to all downstream components.

Here's the getCurrentDate implementation:

// gatsby-node.js

/**
 * Returns the current date in YYYY-MM-DD format
 */
function getCurrentDate() {
  const d = new Date()

  let month = (d.getMonth() + 1).toString()
  if (month.length < 2) {
    month = `0${month}`
  }

  let day = d.getDate().toString()
  if (day.length < 2) {
    day = `0${day}`
  }

  return `${d.getFullYear()}-${month}-${day}`
}

One More Thing...

In my Add Previous and Next Article Buttons in Gatsby article I show how to add "Previous Article" and "Next Article" buttons that depend on the published flag. For this date filtering, I needed to update this to account for the date constraint, so posts that are set to be released in the future don't appear in the pagination links before they're ready.

// gatsby-node.js

const publishedPosts = posts
  .filter(post => {
    return (
      post.node.frontmatter.published === true &&
      new Date(post.node.frontmatter.date) <= new Date(getCurrentDate())
    )
  })
  .reduce((acc, post) => {
    acc[post.node.fields.slug] = post
    return acc
  }, {})

Wrapping Up

With this new logic in place, I can write as much content as I want and set it to publish on a certain date. The only thing I need to do is rebuild and deploy my site (I'm looking for a way to automate this...)

👋 Enjoyed this post?

Join my newsletter and follow me on Twitter @mercatante for more content like this.

Python Comprehensions for PHP Developers

Steven Mercatante — Wed, 30 Oct 2019 20:43:05 +0000

This post was originally published at stevemerc.com

When moving from PHP to Python, your experience with many of the basic language features and constructs are easily carried over. Sure, the syntax is different, but not radically so. Things like loop constructs, basic data structures and even OOP don't take long to feel familiar. One area where this is not true, however, is Python's list and dict comprehensions. PHP doesn't have an equivalent here, and in my opinion, this is an area where Python really shines in terms of functionality, ease of use and succinctness.

In this post, I'll explain what list and dict comprehensions are, and show side-by-side examples of PHP and Python to demonstrate how and why you'd use them.

First, the definition from the official docs:

"List comprehensions provide a concise way to create lists. Common applications are to make new lists where each element is the result of some operations applied to each member of another sequence or iterable, or to create a subsequence of those elements that satisfy a certain condition."

Let's begin with a simple example. We'll use an array of associative arrays in PHP and a list of dicts in Python to represent (some of) Pink Floyd's discography.

PHP:

<?php
$albums = [['name' => 'The Piper at the Gates of Dawn', 'year' => 1967],
            ['name' => 'Atom Heart Mother', 'year' => 1970],
            ['name' => 'The Dark Side of the Moon', 'year' => 1973],
            ['name' => 'Animals', 'year' => 1977],
            ['name' => 'The Wall', 'year' => 1979]];

Python:

albums = [{'name': 'The Piper at the Gates of Dawn', 'year': 1967},
            {'name': 'Atom Heart Mother', 'year': 1970},
            {'name': 'The Dark Side of the Moon', 'year': 1973},
            {'name': 'Animals', 'year': 1977},
            {'name': 'The Wall', 'year': 1979}]

The first thing we want to do is grab a list of just the album names:

PHP:

<?php
$album_names = [];
foreach ($albums as $album) {
    $album_names[] = $album['name'];
}
// or, preferably using array_map()...
$album_names = array_map(function($album) {
    return $album['name'];
}, $albums);

Every PHP developer should be intimately familiar with the above. For developers new to Python, it's natural to do something like this:
Python:

album_names = []
for album in albums:
    album_names.append(album['name'])

But, there's an easier (and arguably better) way to do this, using a list comprehension:

Python:

album_names = [album['name'] for album in albums]

Try this out in your REPL of choice and you'll end up with:

['The Piper at the Gates of Dawn', 'Atom Heart Mother', 'The Dark Side of the Moon', 'Animals', 'The Wall']

In Python, a list comprehension takes the following form:

[(expression) (for clause) (zero or more if statements)]

Say we wanted to return a list of lists, where each inner list contains two elements: the year first and then the album name:

PHP:

<?php
$by_year_name = array_map(function($album) {
    return [$album['year'], $album['name']];
}, $albums);

Python:

by_year_name = [[album['year'], album['name']] for album in albums]
# [[1967, 'The Piper at the Gates of Dawn'], [1970, 'Atom Heart Mother'], [1973, 'The Dark Side of the Moon'], [1977, 'Animals'], [1979, 'The Wall']]

Only want albums released after 1970?

PHP:

<?php
$filtered = array_map(function($album) {
    if ($album['year'] > 1970) {
        return [$album['year'], $album['name']];
    }
}, $albums);

Python:

filtered = [[album['year'], album['name']] for album in albums if album['year'] > 1970]
# [[1973, 'The Dark Side of the Moon'], [1977, 'Animals'], [1979, 'The Wall']]

Don't like 'The Wall'?

PHP:

<?php
$filtered = array_map(function($album) {
    if ($album['year'] > 1970 && $album['name'] != 'The Wall') {
        return [$album['year'], $album['name']];
    }
}, $albums);

Python:

# We'll move the if statements to a new line for the sake of readability
filtered = [[album['year'], album['name']] for album in albums
            if album['year'] > 1970 and album['name'] != 'The Wall']
# [[1973, 'The Dark Side of the Moon'], [1977, 'Animals']]

By now you should see how much more terse the Python examples are compared to PHP. This continues to hold true as your logic becomes more complex.

So, what are dict comprehensions? Turns out they're like list comprehensions, but create dicts instead (shocking, I know.)

Let's index the albums by year, but only for those that were released before 1973:

PHP:

<?php
$indexed = [];
foreach ($albums as $album) {
    if ($album['year'] < 1973) {
        $indexed[$album['year']] = $album['name'];
    }
}

Python:

indexed = {album['year']: album['name'] for album in albums if album['year'] < 1973}
# {1970: 'Atom Heart Mother', 1967: 'The Piper at the Gates of Dawn'}

The syntax may look different, but it still follows the form of:

[(expression) (for clause) (zero or more if statements)]

You can work with dicts in list comprehensions, and vice versa. Let's return the original list of dicts, but with an uppercased version of the album names:

uppercased = [{'name': album['name'].upper(), 'year': album['year']} for album in albums]
# [{'name': 'THE PIPER AT THE GATES OF DAWN', 'year': 1967}, {'name': 'ATOM HEART MOTHER', 'year': 1970}, {'name': 'THE DARK SIDE OF THE MOON', 'year': 1973}, {'name': 'ANIMALS', 'year': 1977}, {'name': 'THE WALL', 'year': 1979}]

An important thing to remember is the symbols on the outside of the comprehension determine what type of data structure you'll get back. Square braces are used for list comprehensions, and curly brackets are used for dict comprehensions.

Let's finish with one more example: returning a generator:

# Notice how we use parens instead of square braces
album_gen = (album for album in albums if album['year'] > 1970)
# <generator object <genexpr> at 0x109d4ff50>

There's still alot more you can do with list and dict comprehensions that's outside the scope of this post. I'll leave it as an exercise for the reader to play around (hint: try nesting comprehensions or figuring out when you would want to use map or filter instead of a comprehension.)

List and dict comprehensions are a terrific feature of Python that I wish PHP had. They make working with data collections much easier. Add them to your arsenal and you'll be glad you did!

👋 Enjoyed this post?

Join my newsletter and follow me on Twitter @mercatante for more content like this.

📢 Announcing React Timeline

Steven Mercatante — Wed, 23 Oct 2019 16:45:03 +0000

This post was originally published at stevemerc.com

I'm excited to announce the release of React Timeline, an open source library for creating responsive, customizable, and themable timelines in React apps.

Originally conceived as a way for developers and designers to showcase their work and achievements (e.g. articles published, videos of conference talks, designs created, etc.), the library is also useful for any type of timeline where you want to display chronological events.

Its API provides high level Event components that lets users quickly build timelines, as well as low level components used to compose higher level events.

Timelines are fully themable — every visual element, from the event wrapper, date, timeline, marker, and event contents can be customized via a theme object. Each component also exposes semantic CSS class names, making styling via traditional stylesheets possible. The goal here was to give the user as much control over the visual style as possible. A few themes are provided out of the box, and creating your own is easy.

React Timeline supports various layouts. By default, events alternate around the timeline (assuming the viewport is wide enough), but you can also provide a layout prop that changes how events are arranged.

Check out the docs and demo here, and the source code here.

You can check out the CodeSandbox demo below, but I suggest opening it in a new tab to see how the responsive layouts behave.

👋 Enjoyed this post?

Join my newsletter and follow me on Twitter @mercatante for more content like this.

Done Is Better Than Perfect

Steven Mercatante — Thu, 17 Oct 2019 14:51:30 +0000

This post was originally published at stevemerc.com

How many side projects have you started, but never released? What about articles? Designs? Open source libraries? I have way more than I can count. They're collecting dust on external hard drives and in Github repos. Things that I got excited about, started, and never released.

This is nothing new for me, and maybe not for you, either. It's easy to fall into the trap of "this design needs to be pixel perfect before I can release it", or "this one last feature is essential; I can't ship without it." These thoughts are roadblocks. They prevent your creation from seeing the light of day.

"Done is better than perfect", "Good enough is good enough", "Perfect is the enemy of good". I didn't coin any of these phrases, and I'm not the first one to repeat them. I've been hearing them for years, but have only recently internalized them, and put them into practice. And that's led to me shipping the following in the last 6 weeks:

7 blog posts
9 videos
2 open source libraries
6 features to my site

For every one of the above items, there was a moment when I thought "let me just tweak this one thing, then I can release it." Every single one. But I knew that if I kept tweaking them until they were perfect, they'd never be done, and I never would have shared them.

A funny thing about shipping, is that it gets easier the more you do it. Patrick McKenzie has a great quote:

Being able to successfully ship things is a useful muscle to develop.

The more you ship, the easier it becomes. You start to doubt yourself less, and you put more out into the world.

Here's another funny thing about shipping: it can bring about unexpected opportunities. I've had people email me after reading or watching my content, and checking out my open source libraries. I now have ideas for more content to produce, related to the stuff I recently shipped. It's a virtuous cycle.

This makes me wonder: how many opportunities have I missed from not releasing my previous work? How many cool work opportunities and new clients could there have been? How many people could have learned something new from reading my source code and articles?

Starting things is easy. Shipping them is hard. So start exercising that muscle.

👋 Enjoyed this post?

Join my newsletter and follow me on Twitter @mercatante for more content like this.

Add Custom JavaScript to a Gatsby Site

Steven Mercatante — Mon, 14 Oct 2019 20:28:55 +0000

This post was originally published at stevemerc.com

I recently found myself needing to add some custom JavaScript to the head of a Gatsby project. In this article I'll show you how to do this.

Gatsby doesn't actually use an index.html file. Instead, it uses an html.js file, which isn't exposed by default.

Following their docs, we can expose html.js by invoking the following command in terminal:

cp .cache/default-html.js src/html.js

This copies html.js to our src folder so we can edit it.

Now, let's add our custom JavaScript. Below is a contrived example for demonstration purposes.

// html.js
<script>var foo = 'bar'; console.log(foo);</script>

Since our HTML doc is a JavaScript file, we can't just copy and paste the above example, otherwise React will complain about a syntax error.

The trick is to use React's dangerouslySetInnerHTML attribute. As its name suggests, you're going to set the inner HTML of whatever tag you apply it to. Let's see it in action!

// html.js
<script
  dangerouslySetInnerHTML={{
    __html: `
      var foo = 'bar';
      console.log(foo);
    `,
  }}
/>

Now if you refresh your app, you should see bar logged to the console.

I admit, this is more work than I care for to just add something to the head of a site, but as with all tech, there are tradeoffs we must make. Since Gatsby makes so many other things easy, we can overlook this minor invonvenience.

👋 Enjoyed this post?

Join my newsletter and follow me on Twitter @mercatante for more content like this.

Add Previous and Next Article Links in Gatsby

Steven Mercatante — Tue, 08 Oct 2019 14:40:22 +0000

This post was originally published at stevemerc.com

Yo! You might want to check out this article first on Managing Draft Articles in Gatsby.

Lots of sites include "Previous Article" and "Next Article" buttons/links on the bottom of their articles. I wanted some, too!

Updating the UI

Let's start with the fun stuff first - updating the UI. I've got a template that renders the actual markdown content, and I want to add the previous and next buttons to it:

// templates/article.js

class ArticleTemplate extends React.Component {
  // These represent the previous and next articles...
  // ... we'll pass these via GraphQL later on
  const { previous, next } = this.props.pageContext

  // Boolean letting us know if we should show prev and next links
  const needsPagination = Boolean(previous) || Boolean(next)

  render() {
    // Imagine the article's body is rendered here...

    {needsPagination && (
      <Pagination>
        {previous && (
          <Link to={previous.fields.slug}>
            <span>← {previous.frontmatter.title}</span>
          </Link>
        )}

        {next && (
          <Link to={next.fields.slug}>
            <span>{next.frontmatter.title} →</span>
          </Link>
        )}
      </Pagination>
    )}
  }
}

(Pagination is a styled component - here's the source if you're interested.)

Refresh your site and you'll see... nothing new! We haven't passed the previous and next articles via GraphQL yet, so needsPagination will be false and that conditional won't render anything. Let's fix that.

Updating The Content Pipeline

Note: I've made some major edits to my gatsby-node.js, but the general idea translates to whatever you've got.

I'm also excluding all the logic that's not related to previous and next articles.

// gatsby-node.js

function createArticlePages() {
    return new Promise((resolve, reject) => {
      const articleTemplate = path.resolve(`./src/templates/article.js`)

      resolve(
        query.then(result => {
          const articles = result.data.allMdx.edges

          /**
           * We only want to check published articles when
           * fetching the `previous` and `next` ones. So,
           * let's convert the `articles` array into an object
           * whose keys are the article slugs.
           *
           * You don't need this step if you're not setting a
           * `published` flag on articles.
           */
          const publishedArticles = articles
            .filter(article => {
              return article.node.frontmatter.published === true
            })
            .reduce((acc, article) => {
              acc[article.node.fields.slug] = article
              return acc
            }, {})

          articles.forEach((article, index) => {
            const [previous, next] = getPrevAndNextArticles(
              publishedArticles,
              article.node.fields.slug
            )

            createPage({
              path: article.node.fields.slug,
              component: blogArticleTemplate,
              context: {
                slug: article.node.fields.slug,
                previous: previous ? previous.node : null,
                next: next ? next.node : null,
              },
            })
          })
        )
      }
    )
}

Most of this is standard Gatsby stuff, but I'm taking an extra step to only fetch the previous and next articles from the subset of published articles (you can skip this step if you don't have a published flag for your articles).

If you're wondering why I'm only using publishedArticles to fetch the previous and next articles, and not using it instead of articles in the forEach below, it's because I want to create article pages regardless of their published status - I just don't want to link to unpublished ones.

Near the bottom of the example is where we pass the previous and next articles via Gatsby's context object, making them available to our Article template.

Let's take a look at the getPrevAndNextArticles function:

function getPrevAndNextArticles(articles, slug) {
  const currentArticleIndex = Object.keys(articles).findIndex(
    slug => slug === slug
  )
  const articlesArray = Object.values(articles)
  let prevArticle
  let nextArticle

  if (currentArticleIndex < articlesArray.length - 1) {
    prevArticle = articlesArray[currentArticleIndex + 1]
  }

  if (currentArticleIndex > 0) {
    nextArticle = articlesArray[currentArticleIndex - 1]
  }

  return [prevArticle, nextArticle]
}

This function takes an object of articles (where each key is the article's slug) and the slug of the article we're currently iterating over in the forEach loop of createArticlePages. We need to find the index of the current article in relation to its siblings, and then we can return the previous and next articles. If the current article is either the first or the last one in the collection, previous or next will be null, respectively.

Now when you refresh your site, you should see previous and next article links (assuming you have enough published articles!)

👋 Enjoyed this post?

Join my newsletter and follow me on Twitter @mercatante for more content like this.

Web Scraping in Elixir

Steven Mercatante — Tue, 01 Oct 2019 19:02:17 +0000

This post was originally published at https://stevemerc.com

As a programmer, two of the most useful tools in my day-to-day work are the terminal and a bowl of hot soup. As the weather gets colder, I get soup almost daily from a local Hale and Hearty shop. After realizing how often I was going to their website to view the menu, I decided to do what any self respecting programmer would do and wrote a command line tool to print the list of a specific location's daily soups right in my terminal.

I used this as an opportunity to become more familiar with the Elixir programming language. I find that building small command line applications is a great way to learn a new language while also creating something useful.

By the end of this tutorial, you'll have built your own command line web scraper in Elixir from start to finish. You'll be able to just type ./soup and be presented with a list of delicious (and sometimes nutritious) bowls of goodness.

The complete source code for this app can be found here.

Spec'ing Out Our App

Our app just needs to print the list of soups for a given location. However, we must tell it which one we care about, and we also want to set a default, because once you have a favorite location you'll probably want to view their menu in the future. I'm not going to go to one location one day, and then another the next day - that'd be crazy.

Our app should have the following API:

./soup --locations: Display a list of all the locations, and ask you to save a default.
./soup: Display the list of soups for your default location.

If you run ./soup without having set a default location, you should be prompted to select a default.

We can break this down with the following flow chart:

Let's Start Building!

Create a new Elixir app and cd into its directory:

$ mix new soup
$ cd soup

We'll need to add two dependencies:

HTTPoison: an HTTP client
Floki: a simple HTML parser

Open up mix.exs and add the following dependencies:

defmodule Soup.Mixfile do

  ...

  defp deps do
  [
    {:httpoison, "~> 0.10.0"},
    {:floki, "~> 0.11.0"}
  ]
  end

end

Install the dependencies:

$ mix deps.get

List httpoison as an application dependency:

defmodule Soup.Mixfile do

  ...

  def application do
    [applications: [:logger, :httpoison]]
  end

end

Scraping the Data

The scraping logic is the most important part of our app, so let's start there. Our first task will be to fetch a list of the locations that we can display to the user.

Fetch the List of Locations

Create lib/scraper.ex and add the following:

defmodule Scraper do

  @doc """
  Fetch a list of all of the Hale and Hearty locations.
  """
  def get_locations() do
    case HTTPoison.get("https://www.haleandhearty.com/locations/") do
      {:ok, response} ->
        case response.status_code do
          200 ->
            locations =
              response.body
              |> Floki.find(".location-card")
              |> Enum.map(&extract_location_name_and_id/1)
              |> Enum.sort(&(&1.name < &2.name))

            {:ok, locations}

          _ -> :error
        end
      _ -> :error
    end
  end

end

Let's break this function down:

We first make a GET request to https://www.haleandhearty.com/locations/ and pattern match on the response. If there was any type of error, we return an :error atom.
Assuming we got back a valid response, we then pattern match on its status code. If it was anything other than 200, we return an :error atom.
Assuming we got back a 200 status code, we're ready to parse the HTML and extract the names and IDs of the locations. We do this by mapping the extract_location_name_and_id/1 (which we'll implement soon) function over the .location-card elements. Each .location-card contains the name and ID of the location.
We sort the locations alphabetically.
The last thing we do is return a tuple of an :ok atom and the list of locations.

If you're wondering about the & character above, read up on the capture operator.

Implement extract_location_name_and_id/1:

defmodule Scraper do

  ...

  defp extract_location_name_and_id({_tag, attrs, children}) do
    {_, _, [name]} =
      Floki.raw_html(children)
      |> Floki.find(".location-card__name")
      |> hd()

    attrs = Enum.into(attrs, %{})

    %{id: attrs["id"], name: name}
  end

end

The one argument is a destructured tuple of the tag name, attributes and children nodes. We convert the children back into HTML so we can use Floki to drill down further and grab the .location-card__name elements. hd() is used to grab the first element of the list, which happens to be the name of the location.

Next we convert attrs from a list of tuples into a map, which makes it easier to pull out attributes by their name. Finally, we return a map of the locaton's name and ID.

That was a lot of typing! Let's see if it works by trying it out in an interactive Elixir session:

$ iex -S mix
iex> Scraper.get_locations()
{:ok,
 [%{id: "17th-and-broadway", name: "17th & Broadway"},
  %{id: "21st-and-6th", name: "21st & 6th"},
  %{id: "29th-and-7th", name: "29th & 7th"},
  %{id: "33rd-and-madison", name: "33rd & Madison"},
  ...
]}

You should see something similar to the above (I've omitted some of the results for the sake of brevity.) If not, you may have a typo in the above code.

Fetch the List of Soups

Now that we've got that working, let's fetch a list of soups for a given location. Fortunately this will be easier than fetching the list of locations. Add the following to lib/scraper.ex:

defmodule Scraper do

  ...

  @doc """
  Fetch a list of the daily soups for a given location.
  """
  def get_soups(location_id) do
    url = "https://www.haleandhearty.com/menu/?location=#{location_id}"

    case HTTPoison.get(url) do
      {:ok, response} ->
        case response.status_code do
          200 ->
            soups =
              response.body
              # Floki uses the CSS descendant selector for the below find() call
              |> Floki.find("div.category.soups p.menu-item__name")
              |> Enum.map(fn({_, _, [soup]}) -> soup end)

            {:ok, soups}

          _ -> :error
        end

      _ -> :error
    end
  end

end

Once again, let's break this function down:

Similar to get_locations/0, the first things we do are make a GET request and then do some pattern matching to make sure we got back a valid response.
We then use Floki.find/2 to find all the p tags with a menu-item__name class. The HTML actually has a bunch of these elements, but not all of them are for soups, so we tell Floki that we only care about the p tags that are descendents of div tags that have a category and soups class.
Using Enum.map/2, we return a list of the soup names.
We return a tuple of an :ok atom and the list of soups.

Let's test our new function:

$ iex -S mix
iex> Scraper.get_soups("17th-and-broadway")
{:ok,
 ["7 Herb Bistro Chicken ", "Broccoli Cheddar", "Chicken & Sausage Jambalaya",
  "Chicken And Rice", "Chicken Pot Pie",
  "Chicken Vegetable With Couscous Or Noodles", "Classic Mac & Cheese",
  "Crab & Corn Chowder", ...]}

We've made a lot of progress! Let's build the command line interface.

Building the Command Line Interface

Note: I've adapted this style of writing the command line interface from the great Programming Elixir book by Dave Thomas.

The first thing we need to do is create a module that'll be called from the command line and can handle arguments.

Create a new module, lib/soup/cli.ex and add the following:

defmodule Soup.CLI do

  def main(argv) do
    argv
    |> parse_args()
    |> process()
  end

end

The main function is what will be triggered when we call our application from the command line. Its sole purpose is to take in an arbitrary number of arguments (like --help), parse them into something our app understands, and then perform the appropriate action.

Let's parse the arguments. Add the following to the Soup.CLI module:

defmodule Soup.CLI do

  ...

  def parse_args(argv) do
    args = OptionParser.parse(
      argv,
      strict: [help: :boolean, locations: :boolean],
      alias: [h: :help]
    )

    case args do
      {[help: true], _, _} ->
        :help

      {[], [], [{"-h", nil}]} ->
        :help

      {[locations: true], _, _} ->
        :list_locations

      {[], [], []} ->
        :list_soups

      _ ->
        :invalid_arg
    end
  end

end

parse_args/1 uses the built in OptionParser module to convert command line arguments into tuples. We then pattern match on these tuples and return an atom that will be passed to one of the process/1 implementations defined below:

defmodule Soup.CLI do

  ...

  def process(:help) do
    IO.puts """
    soup --locations  # Select a default location whose soups you want to list
    soup        # List the soups for a default location (you'll be prompted to select a default location if you haven't already)
    """
    System.halt(0)
  end

  def process(:list_locations) do
    Soup.enter_select_location_flow()
  end

  def process(:list_soups) do
    Soup.fetch_soup_list()
  end

  def process(:invalid_arg) do
    IO.puts "Invalid argument(s) passed. See usage below:"
    process(:help)
  end

end

Let's test this out:

$ mix run -e 'Soup.CLI.main(["--help"])'
soup --locations  # Select a default location whose soups you want to list
soup        # List the soups for a default location (you'll be prompted to select a default location if you haven't already)

I know what you're thinking: typing mix run -e 'Soup.CLI.main(["--help"])' doesn't seem very user-friendly. You're right; it's not. But we'll only be using it like this while we develop our app. The final product will be much more user-friendly, I promise.

Building the Main Module

Now that we've got our command line interface in place, we can focus on our main module. The first thing we want to do is to prompt the user to select a default location. Add the following to lib/soup.ex:

defmodule Soup do

  def enter_select_location_flow() do
    IO.puts("One moment while I fetch the list of locations...")
    case Scraper.get_locations() do
      {:ok, locations} ->
        {:ok, location} = ask_user_to_select_location(locations)
        display_soup_list(location)

      :error ->
        IO.puts("An unexpected error occurred. Please try again.")
    end
  end

end

This function will be called if the user passes the --locations argument to the CLI, or if there's no default location set when calling the CLI without any arguments. It uses our Scraper module to fetch the list of locations and then pattern matches on the result. If get_locations/1 returned an :error we print "An unexpected error occurred. Please try again.". Otherwise we:

Ask the user to select a default location, and then,
Display the list of soups for the location the user just selected

Let's take a look at ask_user_to_select_location/1:

defmodule Soup do

  ...

  @config_file "~/.soup"

  @doc """
  Prompt the user to select a location whose soup list they want to view.

  The location's name and ID will be saved to @config_file  for future lookups.
  This function can only ever return a {:ok, location} tuple because an invalid
  selection will result in this funtion being recursively called.
  """
  def ask_user_to_select_location(locations) do
    # Print an indexed list of the locations
    locations
    |> Enum.with_index(1)
    |> Enum.each(fn({location, index}) -> IO.puts " #{index} - #{location.name}" end)

    case IO.gets("Select a location number: ") |> Integer.parse() do
      :error ->
        IO.puts("Invalid selection. Try again.")
        ask_user_to_select_location(locations)

      {location_nb, _} ->
        case Enum.at(locations, location_nb - 1) do
          nil ->
            IO.puts("Invalid location number. Try again.")
            ask_user_to_select_location(locations)

          location ->
            IO.puts("You've selected the #{location.name} location.")

            File.write!(Path.expand(@config_file), to_string(:erlang.term_to_binary(location)))

            {:ok, location}
        end
    end
  end

end

Notice the @config_file "~/.soup" line - this is a module attribute. It refers to the location of the local file in which we'll save our default location.

The first thing ask_user_to_select_location/1 does is take the list of locations, and prints them in the following format:

 1 - 17th & Broadway
 2 - 21st & 6th
 3 - 29th & 7th
 4 - 33rd & Madison
 ...

According to the docs for Enum.with_index/2, this function:

Returns the enumerable with each element wrapped in a tuple alongside its index.

If an offset is given, we will index from the given offset instead of from zero.

Since it returns a list of each location wrapped in a tuple with a number, we can iterate over this using Enum.each/2 to print out each location with the number we've assigned it.

We then use IO.gets/1 to ask the user to select a location. If they've entered a valid value (an integer that's in range of the list), we confirm their selection, save the location name and ID to @config_file, and return a {:ok, location} tuple. If they've made an invalid selection, we ask them to try again.

Notice the use of :erlang.term_to_binary/1. Because Elixir is a hosted language and Erlang already provides functions to serialize and unserialize data, we call the Erlang function directly.

The last step needed to complete enter_select_location_flow/0 is to implement display_soup_list/1.

defmodule Soup do

  ...

  def display_soup_list(location) do
    IO.puts("One moment while I fetch today's soup list for #{location.name}...")
    case Scraper.get_soups(location.id) do
      {:ok, soups} ->
        Enum.each(soups, &(IO.puts " - " <> &1))
      _ ->
        IO.puts("Unexpected error. Try again, or select a location using `soup --locations`")
    end
  end

end

We pass this function a location map (which if you remember contains the location's name and ID). We fetch the list of soups for that location using Scraper.get_soups/1 and pattern match on the result. If we get back a {:ok, soups} tuple we iterate over soups and print each one. Otherwise we alert the user that an unexpected error has occurred and ask them to try again.

Let's try what we've built so far. Run this in your terminal: mix run -e 'Soup.CLI.main(["--locations"])' and you should see something like the below:

One moment while I fetch the list of locations...
 1 - 17th & Broadway
 2 - 21st & 6th
 3 - 29th & 7th
 4 - 33rd & Madison
 ...
Select a location number:

When it asks you to Select a location number:, enter 3 and you should then see something similar to:

Select a location number: 3
You've selected the 29th & 7th location.
One moment while I fetch today's soup list for 29th & 7th...
 - Chicken Vegetable With Couscous Or Noodles
 - Cream of Tomato with Chicken And Orzo
 - Ten Vegetable
 - Three Lentil Chili
 - Tomato Basil With Rice

Success!

Let's move on to building the functionality to fetch the list of soups if a default location has been set. First we need a function to fetch the location's name and ID.

defmodule Soup do

  ...

  @doc """
  Fetch the name and ID of the location that was saved by `ask_user_to_select_location/1`
  """
  def get_saved_location() do
    case Path.expand(@config_file) |> File.read() do
      {:ok, location} ->
        try do
          location = :erlang.binary_to_term(location)

          case String.strip(location.id) do
            # File contains empty location ID
            "" -> {:empty_location_id}

            _ -> {:ok, location}
          end
        rescue
          e in ArgumentError -> e
        end

      {:error, _} -> :error
    end
  end

end

We use Path.expand/1 (docs) to transform a path like ~/.soup into the absolute path of /Users/steven/.soup. Next we read the contents of @config_file and attempt to unserialize the location's data using Erlang's binary_to_term/1 function. If the unserialization is successful, we return a tuple of an :ok atom and the unserialized location map, otherwise we return an :error atom. A failure to unserialize the data likely means that the user manually edited the file's contents.

defmodule Soup do

  ...

  def fetch_soup_list() do
    case get_saved_location() do
      {:ok, location} ->
        display_soup_list(location)

      _ ->
        IO.puts("It looks like you haven't selected a default location. Select one now:")
        enter_select_location_flow()
    end
  end

end

This is a pretty simple function. We call get_saved_location/1 and match on its return value. If we get back a location map we display the list of soups for that location, otherwise we prompt the user to select a location. This acts as a catch-all for the following error conditions:

@config_file can't be read for whatever reason
the location data in @config_file couldn't be unserialized for whatever reason

Time to try it out. In your terminal, type: mix run -e 'Soup.CLI.main([])' and you should see something similar to:

One moment while I fetch today's soup list for 29th & 7th...
 - Chicken Vegetable With Couscous Or Noodles
 - Cream of Tomato with Chicken And Orzo
 - Ten Vegetable
 - Three Lentil Chili
 - Tomato Basil With Rice

Creating an Executable File

Remember when I told you we'd make the process of using this app more user-friendly than typing something like mix run -e 'Soup.CLI.main(["--locations"])'? That's what we'll do now.

We'll use escript to do this. From the Elixir docs:

An escript is an executable that can be invoked from the command line. An escript can run on any machine that has Erlang installed and by default does not require Elixir to be installed, as Elixir is embedded as part of the escript.

Open up your mix.exs file and add the following line to the project macro block:

escript: [main_module: Soup.CLI],

The entire macro should look something like the following:

  def project do
  [app: :soup,
   version: "1.0.0",
   elixir: "~> 1.3",
   build_embedded: Mix.env == :prod,
   start_permanent: Mix.env == :prod,
   escript: [main_module: Soup.CLI],
   deps: deps]
  end

Now we just need to build the executable. Run this in your terminal:

$ mix escript.build

You should now see a soup executable in your project directory. Try it out with the following commands:

$ ./soup --help
$ ./soup -h
$ ./soup --locations
$ ./soup

And there we go, a fully functional command line web scraper written in Elixir. Now go out and eat soup until you burst!

👋 Enjoyed this post?

Join my newsletter and follow me on Twitter @mercatante for more content like this.

Automating Gatsby Post Creation

Steven Mercatante — Wed, 25 Sep 2019 17:12:24 +0000

I'm a lazy programmer. I love building stuff, but I hate repeating myself. Anytime I start a new article for my site, I do the following:

navigate to my site's directory in terminal
open up VS Code
create a new file
set the title
set the slug
set the date
set its published flag to false

By no means is that a lot of work, but when I have an idea for something to write, I want to start writing it immediately. So, like any self-respecting lazy programmer, I put together a process to remove as much friction as possible. And, to be honest, I used this as an example to learn more about Alfred workflows and bash scripting.

So, What's the Goal?

I use Alfred all the time, so I thought this would be the perfect interface for my new process. I wanted to simply open the Alfred
prompt and type art <some article title> and have it perform the steps listed above.

The Implementation

Gatsby Post Manager has a posts new command that I could use to create the actual file. The command itself looks like this:

$ gpm posts new path/to/posts "hello world"

This command automatically creates the file and assigns it the title, slug, and date, but I still need to pass it the path to the posts. This path never changes for me, so I shouldn't need to type it every time.

My first thought was to create an Alfred workflow that passes that command to terminal. That would work, but it has two downsides:

Alfred will open a new terminal window anytime I invoke the command
I still need to open VS Code to actually write the article, which means I need to pipe the output of gpm to code

The better approach is to write a bash script that invokes gpm and then opens VS Code.

The Bash Script

#!/bin/bash

# Alfred sometimes has trouble finding Node; the below fixes that
export PATH=$PATH:/usr/local/bin

# Require a single argument for the article name
if [ $# -ne 1 ]; then
  echo 1>&2 "Usage: $0 article-name"
  exit 3
fi

# Create the article file
# Uses Gatsby Post Manager (gpm): https://github.com/steven-mercatante/gatsby-post-manager
RESULT=`/usr/local/bin/gpm posts new /Users/steve/sites/stevemerc.com/content/posts "$1"`

# Split $RESULT into an array
# $RESULT normally looks like: "/path/to/posts/post.md successfully created", but we only want the filename

# This is the delimiter that $RESULT will be split on
IFS=' '

# Creates $ARTICLE_ARR array from $RESULT string
read -ra ARTICLE_ARR <<< $RESULT

# Grab just the filename from the success or error message
ARTICLE=${ARTICLE_ARR[0]}

# Open up VS Code on the proper project, then open up the newly created article
/usr/local/bin/code /Users/steve/sites/stevemerc.com && /usr/local/bin/code $ARTICLE

Here's a link to the repo.

I've done my best to document what the script does. If you have any questions, tweet at me.

Now that we've got our script that orchestrates everything, we need to wire it up to Alfred.

Creating the Alfred Workflow

I'm using Alfred v4, so your setup may differ a bit from mine. But the gist is:

open Alfred's preferences
navigate to the Workflows section
click the + button near the bottom and select Templates > Essentials > Keyword to Script

Name your workflow whatever you want and click the Create button. In my case, I called it "Create Article". This is used for organization purposes.

You should then see your newly created workflow and two nodes:

Double click that Keyword node and you'll be prompted to define the command name and title.

Next, double click the Run Script node and you'll be prompted to either write the bash script right there, or you can select External Script (<- do that).

As soon as you enter a path to an existing bash script in the Script File: input, its contents will appear below.

Everything should be set up now, and you can type art hello world in the Alfred prompt to create your new article.

This is what the contents of the newly created hello-world.md file looks like:

---
title: Hello World
slug: 'hello-world'
tags: []
published: false
date: '2019-09-24'
---
Add your content here

Debugging

I had some trouble getting my bash script and Alfred to work together. Something to do with paths... who knows. Luckily, Alfred lets us debug our workflows.

Once you click that little bug icon, it opens a console window that'll print out the steps Alfred takes, as well as any output or errors from your bash script. Getting this process to work without the debugger would've definitely taken longer.

Wrapping Up

I don't expect many people to want to solve the same problem that I just solved, but, knowing how to wire up command line tools, bash scripts, and Alfred to orchestrate common and repetitive tasks is a valuable skill.

👋 Enjoyed this post?

Join my newsletter and follow me on Twitter @mercatante for more content like this.

Add a Published Field to Your Gatsby Posts to Control Their Visibility

Steven Mercatante — Tue, 24 Sep 2019 17:15:15 +0000

The Problem

After converting my site to Gatsby, I quickly realized I needed a way to manage draft posts - pieces of writing I started, but for whatever reason weren't ready to be released publicly. Gatsby makes your posts public by default - as soon as you create that Markdown file and build your site, your post is available for all to see.

The Solution?

Easy! Throw a draft: true|false flag in the post's frontmatter, update the page query to only fetch pages where draft === false, and call it a day. Or, maybe not...

The Real Solution

The problem with the draft flag is that it's not explicit enough for this use case. The post will still be made public if I either forget to include the draft flag, or if it has a typo (e.g dratf: false).

Let's flip the conditional on its head and check if a published flag is true instead. What if I forget to add that flag, or have a typo (👋 publisehd)? The post doesn't appear until I fix the flag. This is a minor annoyance, but not as bad compared to if I accidentally publish something that isn't ready to be released.

The Code

Here's the frontmatter for this post - nothing crazy going on:

---
title: Manage Draft Posts in Gatsby
slug: '/manage-draft-posts-in-gatsby/'
tags: gatsby
published: false
---

And here's my page query:

query {
  allMdx(
    filter: { frontmatter: { published: { eq: true } } }
    sort: { fields: [frontmatter___date], order: DESC }
  ) {
    edges {
      node {
        fields {
          slug
        }
        frontmatter {
          date(formatString: "MMMM DD, YYYY")
          title
        }
      }
    }
  }
}

That frontmatter: { published: { eq: true } } line in the filter block is where the magic happens.

A few things to keep in mind:

you'll need to restart your Gatsby dev server after adding new fields to frontmatter before you can query them
adding a published flag has implications if you have Previous Article and Next Article links: you need to make sure you're not linking to unpublished articles. I'll show you how I handle that in the future article. Stay tuned, friends!

👋 Enjoyed this post?

Join my newsletter and follow me on Twitter @mercatante for more content like this.