Forem: gjdickens

Choosing an ETL Tool in 2026

gjdickens — Mon, 16 Mar 2026 11:46:43 +0000

As the founder of Dark Seahorses, a new agency that provides data engineering and analysis services to small businesses, I had the opportunity to survey the current ETL (extract, transform, load) landscape in 2026 and pick the best tool out there for us.

When picking any tool, it’s important to start with your vision for where the marketplace is heading and how you think you can add value to that. Only then can you can see clearly what tools fit with that vision. What it comes to ETL, I see a marketplace that is heading towards customization, and as an agency it is our job to try to deliver that in a way that scales for us and our clients.

Why ETL Will Become More and More Customized

Previously, a lot of data engineering was about piping data from one predictable source to another limited number of destination databases / data warehouses. This led to several companies succeeding in making connector platforms that provided these pipes in a low or no-code interface. This freed up developer resources while empowering non-technical analysts to have more control over their data pipeline.

Recently, however, two trends are disrupting the dynamics of this structure in the marketplace. The first of those is the rise of AI-assisted programming which is reducing the amount of time it takes for developers to create custom integrations. And the second of those is the proliferation of APIs and potential data sources to connect to.

What was once a problem that could have a very centralized solution, has now turned into something with a much longer tail. Every company has a slightly different set of digital platforms that they rely on. Many of these will be common across industries, while others not so much. Legacy centralized platforms naturally prioritized the most profitable types of data sources to integrate, usually focused on digital marketers, but that led to all the big platforms offering the same limited amount of data sources.

The demands of the marketplace require a lot more customization in order to cover all of the different data sources that are out there. But how do you do that at scale while keeping costs under control?

Delivering Customized ETL at Scale

A key tenant of our philosophy as an agency, is to get our clients onto open-source software that prevents them from being locked into un-affordable platforms while enabling the customization they need. We can scale this up, with an assist from AI, but we also have another trend that helps us scale down services when not required.

Serverless computing services (like AWS Lambda) have provided the infrastructure to pay for only the services that are being used at any one time, and are perfectly suited to ETL jobs that often run for only a few minutes every day or week. By taking advantage of serverless architectures, we can keep client costs down to the absolute minimum of what they actually need.

Key Decision Criteria Based on Our Vision

So with this vision in mind, we had the following key criteria for choosing the platform we will focus on for new client projects:

Criteria	Rationale
Open-source	Need full control and pricing that scales up
Code-first (as opposed to no-code)	Need the customization ability and increasingly can be accelerated with AI assistance
Lightweight, can run on serverless	Need the ability to scale up and down on demand

Proprietary No-Code Platforms

The first category is not one that we seriously considered, as it goes against two of our main criteria: open-source and code-first. But we did think it was important enough to understand the offering of the primary proprietary no-code platforms.

Fivetran

As far as no-code platforms go, Fivetran is considered the industry standard. They have over 300 pre-built connectors and a UI that seems simple enough to get started. But with no options for custom code and pricing that can add up quickly it just wasn’t the right fit for our vision.

To give you an idea of how the quickly the pricing can add up, for our first sample project we were looking at moving 15k rows of data which prices out a $12/month already for just a demo that we can run on AWS for pennies.

No-code
Pre-built connectors
Enterprise standard
No customization
Expensive

Stitch

We saw Stitch as a similar option to Fivetran but with the added con that it seems to be lagging behind in the marketplace in terms of connections. On the plus side, the pricing seems more reasonable than Fivetran’s but again it’s not a fit for us. For someone who can find the connector they’re looking for and just wants a simple no-code connection, this could be a good fit.

No-code
Pre-built connectors, but limited compared to Fivetran
Acquired, not much sign of new development
Better pricing than Fivetran but can get expensive at scale

Open-Source No-Code Platforms

The next category fits closer to our vision and we seriously thought about the trade-offs. Would it make sense for us to operate some connections through a no-code interface and then only escape out to code when necessary? Or maybe the orchestration capabilities of a fully baked platform would come in handy and save us some AWS tooling.

Airbyte

Airbyte is the name to know in this category and we had a good long look at it. In the end, we decided that it was a bit overbuilt for what we needed, and we didn’t want to give up the flexibility of being able to run on serverless. We had other ideas for orchestration (like using simple AWS cron jobs) so that aspect of the platform didn’t offer us much value. Also, debugging issues seems like it can be challenging based on some developer comments we saw. Getting closer, but heavier weight than we needed.

Largest selection of pre-built connectors
Can’t run serverless, needs a dedicated server
Data source configuration can be done with no-code
Customization possible but main focus is on pre-built connectors

Code-first Tools

Custom Python scripts

I’ve gone down this route before with prior projects and know that you can get a long way with Python and libraries like Pandas and Polars. But some things would be nice to have out of the box, like interfaces with AWS or Google BigQuery as well as logic for managing incremental loading and keeping track of load metrics. Ideally we wanted to stay as close to vanilla Python and these core libraries to avoid learning frameworks that might fall out of favor later, but some additional tools would be nice.

Fully customizable
Libraries like Pandas and Polars make it easy to work with tabular data
Better to have some help with schemas, state and incremental loading patterns

Meltano

Meltano comes really close to delivering everything we were looking for. It’s open-source, code-first and can be run on serverless. Its primary strength is also its primary weakness: it’s built on the open-source Singer specification which was developed by Stitch. This means there are lots of pre-built connectors, but the developer opinions are mixed as to how well these work. The specification is also quite complex and requires a lot of up-front work to get a new data source set up.

We thought about investing the time to learn the specification, but didn’t see much enthusiasm from the developer community going in this direction and saw it as too big of a risk that it would be throw-away work. A viable option, but we found something else that fit our needs just a bit better.

Python first
Open-source
Has pre-built connectors
Lots of up-front effort to understand Singer specification (risk of throw-away work)

dlt

In the end, we chose dlt, a relatively new library for Python that handles a lot of the ETL legwork, like incremental loading and schema inference, while still allowing for full customization of the actual data extraction. You can use as much Python as you want in writing your connections, with very little new syntax to learn.

For us, it worked out well because we weren’t looking for lots of pre-built connections or orchestration capability and really valued how lightweight and easy it was to run on serverless.

Available anywhere Python runs
Lightweight
Handles incremental loading and schema inference
Code your own connections and orchestration

Conclusion

This is the thought process that led us to choose dlt as the primary ETL tool that we are running client projects on. It meets all the criteria of our vision of customization that scales up and down, delivering control and affordability to our clients.

For sure, other businesses will have different needs that might make a proprietary platform a better fit or lead someone to value no-code interfaces or orchestration features more highly, but that’s why it’s important to have your vision and choose your tools based on that. For us and our vision of the marketplace, dlt is the right tool for ETL in 2026.

Getting Started With the Mailchimp Marketing API

gjdickens — Fri, 09 Sep 2022 07:43:33 +0000

Mailchimp is one of the oldest, most reliable email marketing platforms out there. Even though there are newer, cheaper options for email marketing out there, Mailchimp still is the tool of choice for lots of online publishers and small businesses.

One of the reasons that Mailchimp stands out from the other email marketing options, is that they have a comprehensive API. This makes for endless customization and extension possibilities if you are comfortable with a little bit of coding.

If you are not as comfortable coding, there are plenty of no-code tools that are popping up to help you make use of the Mailchimp Marketing API. One of those tools is our own connector for Google Data Studio, which enables you to import your Mailchimp data in real-time to Data Studio, where you can then build custom reports and visualize your data with graphs and charts.

In this guide though, we will assume that you have some basic programming knowledge and are comfortable with APIs. This will be a very simple and practical demonstration of how to set up the Mailchimp API, how to get connected to it, and some basic examples of how you can pull down all of your Mailchimp lists using the Postman to access the API.

Before we dive into the details, let’s take a quick look at the tools we will be using.

Mailchimp Marketing API

This is the main API that Mailchimp offers where you can pull data about your Mailchimp lists, campaign performances, list growth and more. This is not to be confused with their transactional email API, which is different and is for sending one-off transactional emails.

Postman

For all of our examples, we will be using Postman. Postman is a simple, free tool that you can use for testing out API’s without having to write the rest of the code around it. This way, we can simply make sure that we understand how the Mailchimp API is working, before we look at writing a bunch of other code to do things with our Mailchimp data. If you are not familiar to Postman, you can get started here.

With that out of the way, let’s jump into it. We will be covering the following:

Creating your API Key
Finding your Mailchimp API Server Location
Using your Mailchimp API Key with Postman
Getting your Lists from the Mailchimp API

Creating Your Mailchimp API Key

In order to access the Mailchimp Marketing API, we need to create a Mailchimp API key. This can be done by logging into our Mailchimp account and going to the Account Profile.

From here, we will go to the dropdown Extras where we will see the option for API Keys:

If you do not yet have any API Keys listed, you will simply click on “Create a Key”. Then you will see it added to the list.

Note the key down since we will need it in our next step. And be very careful with is API key. Do not share it anywhere online, as anyone with this key will have full access to the Mailchimp API on your behalf.

This means someone could send lots of emails from your account or access all of your subscribers email addresses. In short, be careful! And if you do think that your API Key is compromised, you can simply delete it and create a new one.

Finding Your Mailchimp API Server Location

Next, we will need to find our Mailchimp server location. This is very simple as you just need to look up at the URL when you are logged into your Mailchimp account and see the first part of the URL.

For example, in this case, it would be “us13”. Ok, that was simple. Again, note that down, as we will need it to access the Mailchimp API.

Using Your Mailchimp API Key with Postman

Now we are ready to make our first API call to the Mailchimp Marketing API. To test this out without getting too complicated at first, we will just make a call to the easiest endpoint in the Mailchimp API: the Ping endpoint.

Essentially, this endpoint just let’s you know that you have successfully reached the API and received a response. Here’s how we access it.

The URL for the Ping endpoint is: https://[SERVER-LOCATION].api.mailchimp.com/3.0/ping

So if we follow our example from before where the server location was us13, the URL we would need to access to call the API would be: https://us13.api.mailchimp.com/3.0/ping

Great, so now that we know that, we can open up Postman and create a new GET request using this URL.

Now, if we press Send, we will see that we are halfway there - we have successfully received a message from the Mailchimp API, but it is telling us that we need to add our API key.

To do this, we will go to Headers and we will add the key “Authorization” with the value “apikey [OUR API KEY]”.

Once we’ve done this and clicked Send again, we will see a successful response from the Mailchimp Ping endpoint.

Getting Lists from the Mailchimp API

Now that we have successfully connected to the Mailchimp API using Postman, we just need to modify our request so that it does something a bit more useful. As a simple example, we will get all of our Mailchimp Lists from the Mailchimp API.

To do that, we will simply modify the URL in our Postman request, leaving the Authorization Headers as they are. First, let’s check the Mailchimp API Reference to see what the endpoint for fetching lists is.

Once we look it up, we will see it is: https://[SERVER-LOCATION].api.mailchimp.com/3.0/lists

So that means our URL will be https://us13.api.mailchimp.com/3.0/lists

Then if we enter that into Postman as we had done before:

We will see a response that looks like this:

{
  "lists": [
    {
      "id": "string",
      "web_id": 0,
      "name": "string",
      "contact": {
        "company": "string",
        "address1": "string",
        "address2": "string",
        "city": "string",
        "state": "string",
        "zip": "string",
        "country": "string",
        "phone": "string"
      },
      "permission_reminder": "string",
      "use_archive_bar": false,
      "campaign_defaults": {
        "from_name": "string",
        "from_email": "string",
        "subject": "string",
        "language": "string"
      },
      "notify_on_subscribe": false,
      "notify_on_unsubscribe": false,
      "date_created": "2019-08-24T14:15:22Z",
      "list_rating": 0,
      "email_type_option": true,
      "subscribe_url_short": "string",
      "subscribe_url_long": "string",
      "beamer_address": "string",
      "visibility": "pub",
      "double_optin": false,
      "has_welcome": false,
      "marketing_permissions": false,
      "modules": [
        "string"
      ],
      "stats": {
        "member_count": 0,
        "total_contacts": 0,
        "unsubscribe_count": 0,
        "cleaned_count": 0,
        "member_count_since_send": 0,
        "unsubscribe_count_since_send": 0,
        "cleaned_count_since_send": 0,
        "campaign_count": 0,
        "campaign_last_sent": "2019-08-24T14:15:22Z",
        "merge_field_count": 0,
        "avg_sub_rate": 0,
        "avg_unsub_rate": 0,
        "target_sub_rate": 0,
        "open_rate": 0,
        "click_rate": 0,
        "last_sub_date": "2019-08-24T14:15:22Z",
        "last_unsub_date": "2019-08-24T14:15:22Z"
      }
    }
  ],
  "total_items": 0,
  "constraints": {
    "may_create": true,
    "max_instances": 0,
    "current_total_instances": 0
  }
}

As you can see, we’ve successfully pulled down data from each of our Mailchimp lists. If you want to get more details about what each of the fields means, you can refer back to the Mailchimp API reference.

Wrapping Up

With that, we have successfully connected to the Mailchimp Marketing API using our newly generated API key, and we have pulled down data for all of our Mailchimp Lists.

This just scrapes the surface of what can be done using the Mailchimp Marketing API and in future guides we will go into more detail on some of the more interesting uses.

And if you are interested in using your Mailchimp data to build custom reports and data visualizations, be sure to check out out our Mailchimp connector for Data Studio.

Serverless Subscription Management with Fauna, Paddle, Gatsby and Netlify

gjdickens — Thu, 16 Sep 2021 12:45:08 +0000

In this article I will show you how you can extend a Gatsby static-site to manage memberships and paid subscriptions using Netlify and Paddle, with Fauna’s cloud API as the glue that keeps everything together.

We will be covering the following:

Setting up a basic Gatsby site
Configuring and deploying to Netlify
Membership management using Netlify Identity
Adding login and signup modals using React hooks
Adding Gatsby private routes
Creating subscription plans using Paddle
Subscription management from members’ accounts using Fauna

You can find all of the finished code for the project here along with a working demo here.

Before we get started, let’s do a quick visual tour of what we will building:

Creating your Gatsby site

To begin with, we need a basic Gatsby site to build from.

We will start by installing the Gatsby CLI (Command Line Interface) by running the command:

npm install -g gatsby-cli

Next we will spin up a Gatsby instance using their default Gatsby Starter Blog starter. To do this, just go to the folder you want to install in and run:

gatsby new <your-project-name> https://github.com/gatsbyjs/gatsby-starter-blog

Now if you run gatsby developyou can navigate to http://localhost:8000/ in your browser and see your site running on a development server.

Deploying to Netlify

With our Gatsby site up and running our next step is to deploy it to Netlify so that we can take advantage of Netlify’s serverless functions and identity storage. (if you don’t yet have a Netlify account you can create one for free here).

We will be deploying on Netlify directly from Github, so now is a good time to initialize your repository if you haven’t yet already. Now we will add a short netlify.toml file in our root directory that will tell Netlify where we are going to be keeping our serverless functions:

[build]
  command = "NODE_ENV=production gatsby build"
  publish = "public/"
  functions = "./functions"

[[redirects]]
  from = "/api/*"
  to = "/.netlify/functions/:splat"
  status = 200

Here we are simply telling Netlify that on build, any serverless functions we create will be in the ./functionsfolder and they will be served from the endpoint www.mysite/api/.

Now if we follow the Netlify instructions to link our Git repository, you will have a fully deployed site on Netlify that will be automatically rebuilt on any new commits and will be ready for our serverless functions that we will create in a little while.

While we are in Netlify, let’s also enable Netlify Identity which we will also be using soon. To do this, go to the Netlify dashboard and select the site that we just created. Then go to Identity and click “Enable Identity.”

Finally, we will install the Netlify plugin for Gatsby that will handle some configuration automatically for us when it comes to our private routes. To do that we will simply run npm install --save gatsby-plugin-netlifyand add it to our gatsby-config.jsfile.

With that, we are all done with our Netlify configuration. Note, from now on, if we use the command netlify devinstead ofgatsby develop, we will be able to see changes to our serverless functions in real-time. So I recommend that you use that as you are following along.

Member management with Netlify Identity

Let’s now move on to how we can use Netlify’s Identity to manage our members. To help us with this, we will use a Gatsby plugin called gatsby-plugin-netlify-identity-gotrue which is a wrapper for another package called react-netlify-identity-gotrue.

GoTrue is an open-source API maintained by Netlify that allows you to access Netlify’s identity membership signup and login features. These two packages directly integrate GoTrue into React and Gatsby allow us to easily use Netlify Identity in our project.

Let’s start by getting both of these installed with:

npm install --save gatsby-plugin-netlify-identity-gotrue react-netlify-identity-gotrue.

Next let’s add the plugin to our gatsby-config.js:

    {
      resolve: `gatsby-plugin-netlify-identity-gotrue`,
      options: {
        url: `https://YOUR-NETLIFY-APP-URL`
      }
    },

Now we have access to a variety of helpful functions along with the identity context of any logged in user. Let’s start using them by creating our user login process, beginning with some login buttons that will control our signup modals using React Hooks.

First we will create a simple button.js component in src/components:

import React from 'react';

import { handleKeyDown } from '../utils/utils';

const Button = ({ children, showPopup, setShowPopup }) => {

  const handleLoginClick = () => {
    setShowPopup(!showPopup);
  }

  const handleLoginKeyDown = (e) => {
    handleKeyDown(e, showPopup, setShowPopup);
  }

  return (
    <button
      onClick={ (e) => handleLoginClick() }
      onKeyDown={ (e) => handleLoginKeyDown(e) }
       >{children}</button>

  )
}

export default Button;

And then in a new folder src/utils we will create a file called utils.js and add this:

export const handleKeyDown = (ev, showPopup, setShowPopup) => {
  if (ev.keyCode === 13 && !showPopup) {
    // enter to open
    setShowPopup(true);
  } else if (ev.keyCode === 27 && showPopup) {
    // escape to close
    setShowPopup(false);
  }
}

In the component we are adding the ability for the button to control React Hooks that we will pass into it in order to determine whether or not our Signup/Login modal will be displayed. While the handleKeyDown function is checking for keyboard enter and escape events and setting showPopup to true or false respectively. We put the handleKeyDownfunction into a separateutills.js file since we will be using the same function later when we create our modal in order to let users close the modal using the escape key.

Next, let’s move on to create a new component file src/components/loginButtons.js:

import React from 'react';
import { Link } from "gatsby";
import { useIdentityContext } from 'react-netlify-identity-gotrue';

import Button from './button';

const LoginButtons = ({ children, showRegisterPopup, setShowRegisterPopup, showLoginPopup, setShowLoginPopup }) => {

  const identity = useIdentityContext();

  return (
    <>
      {!identity.user &&
        <>
        <div className="headerButton">
          <Button showPopup={showRegisterPopup} setShowPopup={setShowRegisterPopup}>Sign Up</Button>
          <Button showPopup={showLoginPopup} setShowPopup={setShowLoginPopup}>Login</Button>
        </div>
        </>

      }
      {identity.user &&
        <>
        <div className="headerButton">
          <button
             ><Link to="/account/profile">Profile</Link></button>
          <button
            className="link"
            onClick={identity.logout}
            onKeyDown={identity.logout}
             >Logout</button>
        </div>
        </>
      }
    </>
  )

}
export default LoginButtons;

At the same time, we will also make a few additions to our src/components/layout.js file:

import React, { useState } from 'react';
import { Link } from "gatsby"

import LoginButtons from './loginButtons';  

const Layout = ({ location, title, children }) => {
  const rootPath = `${__PATH_PREFIX__}/`;
  const isRootPath = location.pathname === rootPath
  let header

  const[showRegisterPopup, setShowRegisterPopup] = useState(false);
  const[showLoginPopup, setShowLoginPopup] = useState(false);

  if (isRootPath) {
    header = (
      <>
        <h1 className="main-heading">
          <Link to="/">{title}</Link>
        </h1>
        <LoginButtons showRegisterPopup={showRegisterPopup} setShowRegisterPopup={setShowRegisterPopup} showLoginPopup={showLoginPopup} setShowLoginPopup={setShowLoginPopup} />
      </>
    )
  } else {
    header = (
      <>
        <Link className="header-link-home" to="/">
          {title}
        </Link>
        <LoginButtons showRegisterPopup={showRegisterPopup} setShowRegisterPopup={setShowRegisterPopup} showLoginPopup={showLoginPopup} setShowLoginPopup={setShowLoginPopup} />
      </>
    )
  }

In our component, we are initializing our Netlify Identity context helper. With this we can check if a user is logged in by checking if identity.user exists. If it doesn’t, we will show two buttons: one for signing up and one for logging in.

These buttons will toggle the state that is controlled in the layout.jsfile using the React Hooks showRegisterPopup and showLoginPopup. We will manage this state from the Layout component since we will also have to pass these hooks into the modals themselves so we can close them.

When a user is logged in, we will show him a link to his profile page (we will create this a bit later using Gatsby private routes), as well as the option to log out. Using our Netlify Identity context helper we can simply call identity.logout which will handle all the backend of the logout process for us.

Now we need to create our actual modal, which we will do in two parts: First, the popup itself in a new component called popup.js and after that the form component that we will call registerForm.js which will be served through the popup and will be used to collect sign up and login information.

Let’s look at the popup modal first:

import React from 'react';
import { X } from 'react-feather';

import { handleKeyDown } from '../utils/utils';

import './styles/popup.css';

const Popup = ( { children, showPopup, setShowPopup } ) => {

  return (
    <>
      {showPopup ? (
        <div className="popup-overlay">
          <div
            className="popup-background"
            onClick={ (e) => setShowPopup(false) }
            onKeyDown={ (e) => handleKeyDown(e, showPopup, setShowPopup) }
            tabIndex={0}
            aria-label="Toggle Popup"
            role="button"
          ></div>
          <div className="popup-inner">
            <X
              className="popup-close"
              onClick={ (e) => setShowPopup(false) }
              onKeyDown={ (e) => handleKeyDown(e, showPopup, setShowPopup) }
              tabIndex={0}
              aria-label="Toggle Popup"
              role="button"
            />
            {children}
          </div>
        </div>
      ) : null}
    </>
  )
}
export default Popup;

First, note that we are importing a custom styles file here as well as the icons package react-feather. You can find the CSS file in the source code for the example here and simply copy and paste it into your project, while react-feather can be installed with:

npm install --save react-feather

Styling aside, you will see that we are passing in the React hook showPopup which will control whether or not anything is actually returned. Then you will see that if someone clicks (or presses the escape key) on either the background overlay of the modal or the X icon, showPopup will be set to false and the modal will be hidden again. Finally, you will see that we pass in children to allow us to display our form inside the popup.

Let’s create the component for that form in src/components/registerForm.js:

import React, { useState } from 'react';
import { useIdentityContext } from 'react-netlify-identity-gotrue';

const RegisterForm = ({ children, loginType, setShowPopup }) => {
  const identity = useIdentityContext();

  const [formValues, setFormValues] = useState({
    email: '',
    password: ''
  });

  const handleEmailInputChange = (event) => {
    setFormValues((values) => ({
        ...values,
        email: event.target.value,
    }));
  };

  const handlePasswordInputChange = (event) => {
    setFormValues((values) => ({
      ...values,
      password: event.target.value,
    }));
  };

  const handleSubmit = (e) => {
    e.preventDefault();
    if (loginType === 'login') {
      handleLogin(formValues);
    }
    else {
      handleSignup(formValues);
    }

  }

  const handleSignup = async ( formValues ) => {
    let newUser = {
      email: formValues.email,
      password: formValues.password
    }

    await identity.signup(newUser)
    .then(() => {
      console.log('registration successful');
      setShowPopup(false);
    })
    .catch((e) => {
      console.log(e);
    });

  }

  const handleLogin = async ( formValues ) => {
      await identity.login({
        email: formValues.email,
        password: formValues.password
      })
    .then(() => {
      setShowPopup(false);
    })
    .catch((e) => {
      console.log(e);
    });

  }

  return (

    <div className="member-form">
      <form onSubmit={handleSubmit}>
        <div className="member-form-group">
          <label htmlFor="username">Email</label>
          <input type="email" className='member-input' placeholder="youremail@example.com" value={formValues.email} onChange={handleEmailInputChange} />
        </div>
        <div className="member-form-group">
          <div>
            <label htmlFor="password">Password</label>
            <input type="password" className='member-input' placeholder="Password" value={formValues.password} onChange={handlePasswordInputChange} />
          </div>
        </div>

        <div className="member-form-group" style={{textAlign: 'center', paddingTop: '2rem'}}>
          <button className="button member-submit" type="submit">
            <span className="button-content">{loginType ==='login' ? 'Login' : 'Register'}</span>
          </button>
        </div>
      </form>
    </div>
)

}

export default RegisterForm;

Here we are using helper functions from our Netlify Identity context in order to submit our form data. Let’s leave those to the side for a minute while we see how the form itself is actually working.

We have a React hook called formValues which is initiated with empty values for “email” and “password.” Whenever one of the inputs on the form changes, these values are updated via setFormValueswhich is called from both handleEmailInputChange and handlePasswordInputChange. Finally, when the form is submitted the function handleSubmit is called which checks to see whether this is a signup or login form based on the property we have passed in called loginType.

Now let’s look at where we use the Netlify helper functions. First, we can see in handleSignup that we are getting the form values and storing them in an object that has an “email” and a “password” key. Then we simply need to provide this object to identity.signup() and Netlify automatically logs the user in for us. (as well as creating a cookie that persists the login state) We use await here so that when the function returns successfully we can close the popup modal automatically by calling setShowPopup(false).

If we look at handleLogin, we see that it is very similar. The only difference is that now we are calling identity.login() instead. One thing to note is that since we are using await, both of these functions need to be async functions.

Finally, we just need to wire up our popup modal and form in our layout.js file. Here’s the updated version:

import React, { useState } from 'react';
import { Link } from "gatsby"

import Popup from './popup';
import RegisterForm from './registerForm';
import LoginButtons from './loginButtons';

const Layout = ({ location, title, children }) => {
  const rootPath = `${__PATH_PREFIX__}/`;
  const isRootPath = location.pathname === rootPath
  let header

  const[showRegisterPopup, setShowRegisterPopup] = useState(false);
  const[showLoginPopup, setShowLoginPopup] = useState(false);

  header = (
    <>
      {isRootPath ?
        <h1 className="main-heading">
          <Link to="/">{title}</Link>
        </h1>
         :
         <Link className="header-link-home" to="/">
           {title}
         </Link>
       }
      <LoginButtons showRegisterPopup={showRegisterPopup} setShowRegisterPopup={setShowRegisterPopup} showLoginPopup={showLoginPopup} setShowLoginPopup={setShowLoginPopup} />
      <Popup showPopup={showRegisterPopup} setShowPopup={setShowRegisterPopup} >
        <h1 style={{textAlign: 'center', paddingBottom: '2rem'}}>Join now</h1>
        <RegisterForm loginType='register' setShowPopup={setShowRegisterPopup} />
      </Popup>
      <Popup showPopup={showLoginPopup} setShowPopup={setShowLoginPopup} >
        <h1 style={{textAlign: 'center', paddingBottom: '2rem'}}>Sign in to access your account</h1>
        <RegisterForm loginType='login' setShowPopup={setShowLoginPopup} />
      </Popup>
    </>
  )

  return (
    <div className="global-wrapper" data-is-root-path={isRootPath}>
      <header className="global-header">{header}</header>
      <main>{children}</main>
      <footer>
        © {new Date().getFullYear()}, Built with
        {` `}
        <a href="https://www.gatsbyjs.com">Gatsby</a>
      </footer>
    </div>
  )
}

export default Layout;

First, note that I’ve refactored the code of the original template in order to avoid duplication in the header component. Instead of the previous If / else statement, I’ve put a conditional statement directly in the HTML to control what is displayed when isRootPath is true.

Then after that we simply import our Popup and RegisterForm components and pass in the React hooks for signing up and logging in, as well as a title for each popup. Our finished popup will now look like this:

Gatsby Private Routes

Next we will set up private routes in Gatsby that will allow us to have a private account management section of the site for each user.

Essentially, a private route in Gatsby is a route that is only generated on client-side when certain conditions are met. (rather than the majority of a Gatsby site, or any other static site for that matter, that will be generated in advance on the server side) This prevents someone who is not entitled to see some content from being served it along with all of the other resources of a site. You can read more about it here.

There’s also a great tutorial on how to get this setup using Auth0 instead of Netlify Identity over on the Auth0 blog. (credit to the author Sam Julien, as a lot of the following code was adapted from his examples)

To get this set up, we first need to edit our gatsby-node.js file in the root of our project. We will be adding the following code to what’s already there:

exports.onCreatePage = async ({ page, actions }) => {
  const { createPage } = actions

  if (page.path.match(/^\/account/)) {
    page.matchPath = "/account/*"

    createPage(page)
  }
}

What this is doing is it is telling Gatsby to look for any pages that fall under the path /account/ and that these pages should be rendered on the client-side only. (rather than generated as static content on the server side.

With that done, we can now create our account page as a new file in the page directory called account.js:

import React from "react"
import { Link, graphql } from "gatsby"
import { Router } from "@reach/router"
import Layout from "../components/layout"
import Profile from "../components/profile"
import PrivateRoute from "../components/privateRoute"

const Account = ({ data, location }) => {
  const siteTitle = data.site.siteMetadata?.title || `Title`

  return (
    <Layout location={location} title={siteTitle}>
      <Router>
        <PrivateRoute path="/account/profile" component={Profile} />
      </Router>
    </Layout>
  )
}

export default Account

export const pageQuery = graphql`
  query {
    site {
      siteMetadata {
        title
      }
    }
    allMarkdownRemark(sort: { fields: [frontmatter___date], order: DESC }) {
      nodes {
        excerpt
        fields {
          slug
        }
        frontmatter {
          date(formatString: "MMMM DD, YYYY")
          title
          description
        }
      }
    }
  }
`

Most of this is boilerplate and copied from our index.js page, including the GraphQL query and the metadata that is passed into our Layout component. The important thing to notice here is that we are using Router imported from @reach/router (no NPM install required since it comes by default with Gatsby) to wrap a new component called PrivateRoute that we will create now.

So let’s create a new file in our components folder called privateRoute.js:

import React from "react";
import { useIdentityContext } from 'react-netlify-identity-gotrue';

const PrivateRoute = ({ component: Component, path, ...rest }) => {
  const identity = useIdentityContext();

  return identity.user ? <Component {...rest} /> : null;
}

export default PrivateRoute;

This is a super simple and straight to the point component. We are importing our Netlify Identity context and then using that, we are returning a component if a user is logged in and null if they are not.

So essentially this component serves as a wrapper for another component that we specify and only serves that component if a user is logged in.

If you scroll back up to our account.js page, you will see that the component we are wrapping here is called Profile. So let’s create a new component file called profile.js and in it we will put:

import React from "react"
import { useIdentityContext } from 'react-netlify-identity-gotrue';

const Profile = () => {
  const identity = useIdentityContext();

  if (!identity.user) {
    return <div>Loading...</div>;
  }
  return(
    <>
      <section className="blog-wrapper" style={{textAlign: 'center'}}>
        <h2>Account Details</h2>
        <p>Email: {identity.user.email}</p>
      </section>
    </>
  )
}

export default Profile;

This component is the actual content that will be served to a logged in user browsing to /account/profile/. Here we are using our Identity context to check if a user’s details are available, if not they are shown a simple loading message. Then once the logged in user’s details have been loaded, the profile page simply displays their email address. (which is the only information we have collected from them in this simple example)

And with that, we have a basic but fully functioning member sign up and login flow. Let’s look at how we can add paid subscriptions to that next using Paddle and Fauna.

Creating subscription plans using Paddle

Next we will look at how we can let our users manage subscription plans using their account profiles that we just created. To manage our subscriptions, we will use a service called Paddle.

Essentially Paddle takes care of all of the billing and invoicing issues that come with selling a SaaS product and it has an easy-to-use dashboard as well as lots of API functionality that makes it very developer friendly.

For the purposes of our example, we will create a new Paddle sandbox account here.

Once we have our account, we will create a new Subscription Plan by going to Catalog > Subscription Plans. Create a new plan and give it whatever name and price you like - just take note of the Plan ID that is generated once you create the plan.

Now let’s get Paddle connected to our Gatsby build. There is a helpful plugin called gatsby-plugin-paddle that injects Paddle’s script into your site automatically, but unfortunately it doesn’t support Paddle’s sandbox environment. So instead of using the plugin, I have taken the code from this plugin, modified it slightly and we will use it directly in our project.

To do that, we will create a new file in our root folder called gatsby-ssr.js and inside it we will paste:

const React = require(`react`);

exports.onRenderBody = ({ setPostBodyComponents }) => {
    if (process.env.NODE_ENV === `production`) {
        return setPostBodyComponents([
            <script key="gatsby-plugin-paddle" src="https://cdn.paddle.com/paddle/paddle.js" />,
            <script
                key={`gatsby-plugin-paddle-vendor`}
                type="text/javascript"
                dangerouslySetInnerHTML={{
                    __html: `Paddle.Setup({ vendor: ${process.env.PADDLE_ID} });`
                }}
            />
        ]);
    }
    else {
        return setPostBodyComponents([
            <script key="gatsby-plugin-paddle" src="https://cdn.paddle.com/paddle/paddle.js" />,
            <script
                key={`gatsby-plugin-paddle-vendor`}
                type="text/javascript"
                dangerouslySetInnerHTML={{
                    __html: `Paddle.Environment.set('sandbox'); Paddle.Setup({ vendor: ${process.env.PADDLE_ID} });`
                }}
            />
        ]);
    }
};

We will also need to create a .env.development file in our root directory and put inside of that:

PADDLE_ID=YOUR_VENDOR_ID
PADDLE_PRODUCT_ID=YOUR_PADDLE_PRODUCT_ID

The gatsby-ssr.js file is hooking into Gatsby’s build process and every time a page is generated it injects Paddle’s script into the body of that page. The Paddle script requires your Vendor ID which you can find in the Paddle dashboard under Developer Tools > Authentication. We are storing that as an .env variable since it should be kept secret and not logged into your source control.

The original plugin only added the Paddle script to production builds, so I modified the code to set Paddle to sandbox mode if you are not running in production. This way, you could create an .env.production file and put your Paddle production Vendor ID there and be able to simultaneously test with the sandbox while you are in dev mode and then deploy smoothly to production.

With that configuration out of the way, let’s look at how we can actually use Paddle in our site. Let’s open up our utils.js file and add the following:

export const openPaddleCheckout = (user) => {

  let options = {product: process.env.PADDLE_PRODUCT_ID};
  if(user && user.email) {
    options.email = user.email;
  }
  let Paddle = window['Paddle'];
  if (Paddle) {
    Paddle.Checkout.open(options);
  }

}

Here we are creating a utility function that we can call from anywhere in our app, that will open up a new Paddle checkout session and serve it as an overlay to our site. Using Paddle’s helper functions that were set up with our script injection previously, we just need to define some options and pass these into Paddle.Checkout.open().

For the options, we are pulling the Paddle Production ID from our .env.development file and then we are checking if we have an email passed in with our user object comes from Netlify Identity. If we have an email, this will be passed into our options which tells Paddle to pre-populate the email address so the user doesn’t have to type it in again.

Finally, we will check that the Paddle script is available and loaded using window['Paddle'] before we actually call it.

Now let’s go back to our index.js file in order to use our Paddle checkout. First, we will make some additions to the top half of the code:

import React, { useState, useEffect } from 'react';
import { Link, graphql } from "gatsby";
import { useIdentityContext } from 'react-netlify-identity-gotrue';

import Bio from "../components/bio";
import Layout from "../components/layout";
import Seo from "../components/seo";
import Button from "../components/button";
import Popup from "../components/popup";
import RegisterForm from "../components/registerForm";
import { openPaddleCheckout } from "../utils/utils";

const BlogIndex = ({ data, location }) => {
  const siteTitle = data.site.siteMetadata?.title || `Title`;
  const posts = data.allMarkdownRemark.nodes;
  const identity = useIdentityContext();

  const[showPaddleCheckout, setShowPaddleCheckout] = useState(false);

  useEffect(() => {
    if (identity.user && showPaddleCheckout) {
      openPaddleCheckout(identity.user);
      setShowPaddleCheckout(false);
    }
  }, [identity.user, showPaddleCheckout])

…

Here we are importing our openPaddleCheckout function as well as React’s useEffect. We will also create a React hook called showPaddleCheckout.

Next we will create a useEffect that checks if there is a logged in user and if our showPaddleCheckout hook is set to true. If both of these are the case, then we will call our openPaddleCheckout function while passing in our user object from Netlify Identity. Finally, we will call setShowPaddleCheckout and set it to false, so that we don’t initiate another checkout session if the page is refreshed.

Note that we have passed an array of dependencies to our useEffect that includes identity.user and showPaddleCheckout. This means that if the value of either of these two dependencies changes the useEffect will be called again.

Now we just have to add a button where we can set the value of our showPaddleCheckout hook. Let’s add it right beneath where we are showing all of our blog posts on the index.js page:

  return (
    <Layout location={location} title={siteTitle}>
      <Seo title="All posts" />
      <Bio />
      <ol style={{ listStyle: `none` }}>
        {posts.map(post => {
          const title = post.frontmatter.title || post.fields.slug

          return (
            <li key={post.fields.slug}>
              <article
                className="post-list-item"
                itemScope
                itemType="http://schema.org/Article"
              >
                <header>
                  <h2>
                    <Link to={post.fields.slug} itemProp="url">
                      <span itemProp="headline">{title}</span>
                    </Link>
                  </h2>
                  <small>{post.frontmatter.date}</small>
                </header>
                <section>
                  <p
                    dangerouslySetInnerHTML={{
                      __html: post.frontmatter.description || post.excerpt,
                    }}
                    itemProp="description"
                  />
                </section>
              </article>
            </li>
          )
        })}
      </ol>
      {
        identity.user &&
        <>
          <h1>Add Subscription</h1>
          <Button showPaddleCheckout={showPaddleCheckout} setShowPaddleCheckout={setShowPaddleCheckout} user={identity.user}>Subscribe</Button>
        </>
      }

    </Layout>
  )

After the list of posts, we are checking if a user is logged in then simply serving a button with some header text.

Notice though that we will need to update our button.js component to handle this new case of adding a subscription. So we will modify our code to the following:

import React from 'react';

import { handleKeyDown } from '../utils/utils';

const Button = ({ children, showPopup, setShowPopup, showPaddleCheckout, setShowPaddleCheckout, user }) => {

  const handleLoginClick = () => {
    if (!user) {
      setShowPopup(!showPopup);
    }
    if (setShowPaddleCheckout && !showPaddleCheckout) {
      setShowPaddleCheckout(true);
    }
  }

  const handleLoginKeyDown = (e) => {
    if (!user) {
      handleKeyDown(e, showPopup, setShowPopup);
    }
    if (setShowPaddleCheckout && !showPaddleCheckout) {
      setShowPaddleCheckout(true);
    }
  }

  return (
    <button
      onClick={ (e) => handleLoginClick() }
      onKeyDown={ (e) => handleLoginKeyDown(e) }
       >{children}</button>

  )
}

export default Button;

We are now passing in showPaddleCheckout, setShowPaddleCheckout and our Netlify user as props. In both handleLoginClick and handleLoginKeyDown we are first checking if we have a prop for setShowPaddleCheckoutand that showPaddleCheckout is not already set to true. Then we are calling setShowPaddleCheckout(true) which will then trigger our useEffect to serve our Paddle checkout.

Subscription management from member accounts using Fauna

Finally, let’s tie everything together using Fauna. Credit to Jason Lengstorf and his great article on connecting Stripe to Netlify using Fauna that formed the basis of a lot of the following code.

If you haven’t yet already, head over to fauna.com and get signed up for a free account. Then you will need to create a new database, give it whatever name you want.

Once you have the database created, we will need to define its schema. I’m following Jason’s advice here and storing our schema in a new db folder in our project’s root directory. Call this file schema.gql and paste in the following:

type User {
  netlifyID: ID!
  paddleSubID: ID!
}

type Query {
  getUserByNetlifyID(netlifyID: ID!): User!
}

This is telling our Fauna instance that our database will have a data type of User that will have two ID types: netlifyID and paddleSubID. We are also defining one query that will expect a netlifyID and will return the User object.

Now we will upload this into Fauna by going to GraphQL > Import Schema.

Next we need to generate a Server Key from Fauna that we can use to connect from our Gatsby site. On the left-hand menu go to Security and then New Key. Now we will save this key in our .env.development file:

PADDLE_ID=YOUR_VENDOR_ID
PADDLE_PRODUCT_ID=YOUR_PADDLE_PRODUCT_ID
FAUNA_SERVER_KEY=YOUR_SERVER_KEY

As a last configuration step, we will install the node-fetch package to our project so that we can use fetch in our Netlify Serverless functions. We will need to use Serverless functions to connect to our Fauna instance, since we don’t want to expose our secret key in our client-side code.

So we will run:

npm install --save node-fetch

Now with that set up, we will start writing our Netlify serverless functions. Let’s start with a new file in functions/utils/fauna.js:

const fetch = require('node-fetch');

exports.faunaFetch = async ({ query, variables }) => {
  return await fetch('https://graphql.fauna.com/graphql', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.FAUNA_SERVER_KEY}`,
    },
    body: JSON.stringify({
      query,
      variables,
    }),
  })
    .then((res) => res.json())
    .catch((err) => console.error(JSON.stringify(err, null, 2)));
};

Credit to Jason again, as this was his creation in his tutorial. Here we are just creating a boilerplate fetch function that we will be able to reuse in order to access our Fauna instance. Note that the function expects query and variables objects and then converts these into JSON in the body of the request. Also note we are pulling our Fauna Server Key from our .env.development file.

Now let’s use our faunaFetch utility function in a new file functions/addSub.js:

const { faunaFetch } = require('./utils/fauna');

exports.handler = async (event) => {
  // Only allow POST
  if (event.httpMethod !== 'POST') {
    return {
      statusCode: 200,
      body: JSON.stringify('Method not allowed')};
  }

  const { netlifyID, paddleSubID } = JSON.parse(event.body);

  // link the Netlify and Paddle Sub IDs in Fauna
  await faunaFetch({
    query: `
      mutation ($netlifyID: ID!, $paddleSubID: ID!) {
        createUser(data: { netlifyID: $netlifyID, paddleSubID: $paddleSubID }) {
          netlifyID
          paddleSubID
        }
      }
    `,
    variables: {
      netlifyID: netlifyID,
      paddleSubID: paddleSubID,
    },
  });

  return {
    statusCode: 200,
    body: JSON.stringify('Sub Added')
  };
};

In order to create a new record in our Fauna we will specify our mutation query that expects both a Netlify User ID and a Paddle Subscription ID. We will then be storing both of those in our Fauna database so we can link one to the other. That way, we can fetch our users by their subscription ID and vice versa.

Now we need to create a function on the client-side to call our serverless function to add a new user. Let’s go back to our src/utils/utils.js file and add the following:

const addPaddleSubscription = ( netlifyID, paddleSubID ) => {
  async function postData(url) {
     const response = await fetch(url, {
       method: 'POST',
       mode: 'cors',
       headers: {
         'Content-Type': 'application/json',
       },
       body: JSON.stringify({netlifyID: netlifyID, paddleSubID: paddleSubID})
     });
     return response.json();
   }
   postData('/api/addSub/')
     .then(data => {
       console.log("Success: ", data);
     })
     .catch((error) => {
        console.log('Error: ', JSON.stringify(error));
      });
  }

Here we are creating an async function that will call our serverless function at /api/addSub/ passing our Netlify ID and Paddle Subscription ID’s into the body.

Now we will just need to modify our openPaddleCheckout function in our same file:

export const openPaddleCheckout = (user) => {

  const checkoutComplete = (data) => {
    var checkoutId = data.checkout.id;

    Paddle.Order.details(checkoutId, function(data) {
      addPaddleSubscription(user.id, data.order.subscription_id)
    });
  }
  let options = {product: process.env.PADDLE_PRODUCT_ID, successCallback: checkoutComplete};
  if(user && user.email) {
    options.email = user.email;
  }
  let Paddle = window['Paddle'];
  if (Paddle) {
    Paddle.Checkout.open(options);
  }

}

Here we are adding an additional option to our Paddle.Checkout.open called successCallbackwhich runs after a payment is made successfully.

In our checkoutComplete function we get the checkout ID that is passed to the callback as data.checkout.id. Then with this ID, we can get further details from Paddle by calling Paddle.Order.details(checkoutId). Finally, this function also has a callback function where we will call addPaddleSubscription() while passing in our Netlify User ID and the Paddle Subscription ID that we just accessed from the details via data.order.subscription_id.

Now that we’ve linked our Netlify ID’s with Paddle subscriptions using Fauna, let’s build a way to display a Paddle subscription in a user’s profile. To do this, we will need to create another serverless function and accompanying client-side API call to get a subscription ID by Netlify user ID.

Let’s start with our serverless function we will create in a new file at functions/getSub.js:

const { faunaFetch } = require('./utils/fauna');

exports.handler = async (event) => {
  // Only allow POST
  if (event.httpMethod !== 'POST') {
    return {
      statusCode: 200,
      body: JSON.stringify('Method not allowed')};
  }

  try {
    const { netlifyID } = JSON.parse(event.body);

    const result = await faunaFetch({
      query: `
        query ($netlifyID: ID!) {
          getUserByNetlifyID(netlifyID: $netlifyID) {
            netlifyID
            paddleSubID
            _id
          }
        }
      `,
      variables: {
        netlifyID: netlifyID,
      },
    });

    return {
      statusCode: 200,
      body: JSON.stringify({subs: true, data: result.data.getUserByNetlifyID}),
    };

  }
  catch (err) {
    console.log(err); // output to netlify function log
    return {
      statusCode: 200,
      body: JSON.stringify({subs: false, data: {} }),
    };
  }
};

Most of this function is the same as addSub, we are just passing in different data and using a different Fauna query. Here we are parsing the Netlify ID via the body of our POST call and then using that in a Fauna query (rather than a mutation). We are calling the query that we previously defined in our schema (getUserByNetlifyID) and we are asking that query to return the paddleSubID as well as _id which is the Fauna ID for the record in the database. (we will use this a bit later when we want to delete records)

Note the structure of the data that we are returning. We are returning an object that has a key “subs” that will either be true or false, depending on if any subs currently exist for that user ID and then we are returning the actual data under the key “data.”

Now that we have our query and serverless function ready, let’s create our POST request in src/utils/utils.js by adding the following:

export const  getPaddleSubscription = async ( netlifyID ) => {
  async function postData(url) {
     const response = await fetch(url, {
       method: 'POST',
       mode: 'cors',
       headers: {
         'Content-Type': 'application/json',
       },
       body: JSON.stringify({netlifyID: netlifyID})
     });
     return response.json();
   }
  return await postData('/api/getSub/');
}

This is a very straight-forward function that simply calls our new serverless function while passing in the Netlify ID and returns the response, which will be the Netlify ID, Paddle Subscription ID and Fauna ID.

Now let’s use this data in our user profile component. We will replace our src/components/profile.js with the following:

import React, { useState, useEffect } from 'react';
import { useIdentityContext } from 'react-netlify-identity-gotrue';
import { getPaddleSubscription } from '../utils/utils';

const Profile = () => {
  const identity = useIdentityContext();

  const [loadingSubscription, setLoadingSubscription] = useState(true);
  const [subscription, setSubscription] = useState(null);

  useEffect(() => {
    if (loadingSubscription) {
      getPaddleSubscription(identity.user.id)
        .then(newSub => {
          if (newSub.subs) {
            setSubscription(newSub.data);
          }
          setLoadingSubscription(false);
        })
        .catch(err => {
          setLoadingSubscription(false);
        })
    }
  }, [loadingSubscription, setSubscription, setLoadingSubscription, subscription, identity.user.id])

  if (!identity.user) {
    return <div>Loading...</div>;
  }
  return(
    <>
      <section className="blog-wrapper" style={{textAlign: 'center'}}>
        <h2>Account Details</h2>
        <p>Email: {identity.user.email}</p>
        <h3>Subscriptions</h3>
        {loadingSubscription && <p>Loading Subscriptions</p>}
        {!loadingSubscription && subscription !== null &&
          <p>Test Subscription: {subscription.paddleSubID} </p>
        }
        {!loadingSubscription && subscription === null &&
          <p>No current subscriptions</p>
        }

      </section>
    </>
  )
}

export default Profile;

Here we have two React Hooks that are controlling the action: loadingSubscription and subscription. By default, loadingSubscription is set to true which will cause our useEffect to be triggered when the component is initially loaded. In the useEffect, we then call our getPaddleSubscription() function passing in our user ID from our Netlify Identity context. Once that is returned, we will check if any subscription was found by checking if subs is true, then we set the subscription ID that is returned using setSubscription(). Finally, whether a subscription was found or not, we will set loadingSubscription to false.

Note that we have the following as dependencies in our useEffect: subscription, setSubscription, loadingSubscription, setLoadingSubscription, identity.user.id. So if any of these change our useEffect will run again, although it will not actually fetch new data unless loadingSubscription is set to true.

Then to display our data, we are using a few conditionals in our HTML. If loadingSubscription is true, we are simply displaying a message that says “Loading Subscriptions.” Then once loadingSubscription is complete and set to false, we have two options:

If a subscription is found, it will be displayed with it’s Paddle ID. (You could do a lot more here by displaying other subscription data to your user, but for the purpose of this example we will just show the ID)
If no subscription is found, it simply says “No current subscriptions”

Finally, let’s take things one step further and let our user’s cancel their subscription from their account profile.

First we will need an API key from Paddle which you can find in the Paddle Dashboard under Developer Tools > Authentication, then you can just use the Default API Key and press Reveal Auth Code. Paste this code into your .env.development file:

PADDLE_ID=YOUR_VENDOR_ID
PADDLE_PRODUCT_ID=YOUR_PADDLE_PRODUCT_ID
FAUNA_SERVER_KEY=YOUR_SERVER_KEY
PADDLE_AUTH_CODE=YOUR_PADDLE_API_KEY

Now let’s create a new serverless utility function in functions/utils called paddleCancel.js:

const fetch = require('node-fetch');

exports.paddleCancel = async (subID) => {
  return await fetch('https://sandbox-vendors.paddle.com/api/2.0/subscription/users_cancel', {
    method: 'POST',
    body: new URLSearchParams({
        'vendor_id': process.env.PADDLE_ID,
        'vendor_auth_code': process.env.PADDLE_AUTH_CODE,
        'subscription_id': subID
    })
  })
    .then((res) => res.json())
    .catch((err) => console.error(JSON.stringify(err, null, 2)));
};

Here we are calling the Paddle API (more details here) using fetch and passing in form encoded data in the body with three items: vendor_id and vendor_auth_code simply come from our .env file while subscription_id will be provided as an argument to our function.

Now let’s use this utility function in a new serverless function functions/deleteSub.js:

const { faunaFetch } = require('./utils/fauna');
const { paddleCancel } = require('./utils/paddleCancel');

exports.handler = async (event) => {
  // Only allow POST
  if (event.httpMethod !== 'POST') {
    return {
      statusCode: 200,
      body: JSON.stringify('Method not allowed')};
  }

  const { netlifyID } = JSON.parse(event.body);

  const result = await faunaFetch({
    query: `
      query ($netlifyID: ID!) {
        getUserByNetlifyID(netlifyID: $netlifyID) {
          netlifyID
          paddleSubID
          _id
        }
      }
    `,
    variables: {
      netlifyID: netlifyID,
    },
  });

  const paddleResult = await paddleCancel(result.data.getUserByNetlifyID.paddleSubID);
  console.log(paddleResult);

  await faunaFetch({
    query: `
      mutation ($faunaID: ID!) {
        deleteUser(id: $faunaID) {
          netlifyID
          paddleSubID
        }
      }
    `,
    variables: {
      faunaID: result.data.getUserByNetlifyID._id,
    },
  });

  return {
    statusCode: 200,
    body: JSON.stringify('Sub Deleted'),
  };
};

In this function we have three important things happening one after the other:

First, we are querying Fauna by Netlify ID and returning the Paddle Subscription ID as well as the Fauna ID. (same as we did in getSub.js)
Then we are using that Paddle Subscription ID to call our paddleCancel function. (we are just logging the response from Paddle in the example)
After that, we are calling Fauna with a new mutation that deletes our User data item

Note, this is why we needed to return our Fauna _id before, since it is required in order to delete the item.

As with all of our serverless functions, we need to add an accompanying client-side API call, so let’s add that to our src/utils/utils.js file:

export const cancelPaddleSubscription = ( netlifyID ) => {
  async function postData(url) {
     const response = await fetch(url, {
       method: 'POST',
       mode: 'cors',
       headers: {
         'Content-Type': 'application/json',
       },
       body: JSON.stringify({netlifyID: netlifyID })
     });
     return response.json();
   }
   postData('/api/deleteSub/')
     .then(data => {
       console.log("Success: ", data);
     })
     .catch((error) => {
        console.log('Error: ', JSON.stringify(error));
      });
  }

Again, very straightforward - we are just passing our Netlify ID into the body and calling /api/deleteSub/ with a POST request.

Finally to get all this online we just need to update our UI, so let’s go back to our src/components/profile.js and make a couple of quick changes:

import { getPaddleSubscription, cancelPaddleSubscription } from '../utils/utils';

…

  return(
    <>
      <section className="blog-wrapper" style={{textAlign: 'center'}}>
        <h2>Account Details</h2>
        <p>Email: {identity.user.email}</p>
        <h3>Subscriptions</h3>
        {loadingSubscription && <p>Loading Subscriptions</p>}
        {!loadingSubscription && subscription !== null &&
          <p><span>Test Subscription: {subscription.paddleSubID} </span><button onClick={e => {cancelPaddleSubscription(identity.user.id); setSubscription(null); }}>Cancel</button></p>
        }
        {!loadingSubscription && subscription === null &&
          <p>No current subscriptions</p>
        }

      </section>
    </>
  )

We are adding our new cancelPaddleSubscription to our imports and then we are using it in the HTML that is served when a user has a subscription. Then we have put a simple button inside a span next to the Paddle Subscription ID which when clicked will call cancelPaddleSubscription with the Netlify User ID as an argument. It will also set subscription back to null so the subscription will disappear as soon as the button is clicked.

And that’s it! We’ve now got a fully functioning account management service that let’s our users subscribe and cancel subscriptions on their own.

Summary

In this post, you learned how to:

Set up a basic Gatsby site
Configure and deploy it to Netlify
Manage memberships using Netlify Identity
Add login and signup modals using React hooks
Add private member areas using Gatsby private routes
Create subscription plans using Paddle
Manage subscriptions from users’ accounts using Fauna

This post gives you the basic flow of how you can incorporate memberships and subscriptions into your static site and how a simple and flexible database like Fauna can help you pull it all together.

You can get even more ideas of what you can do with them together by checking out more of my work at Epilocal where I am building technology for local news and other small online publishers using this same tech stack to manage my memberships and SaaS subscriptions.

Written in connection with the Write with Fauna Program.

Collect Serverless Email Sign Ups with a Simple AWS Lambda

gjdickens — Thu, 10 Jun 2021 14:23:51 +0000

I run my project Epilocal on Gatsby and AWS. Static sites are great for a lot of reasons, but one of the draw-backs they have is that it's more difficult to handle tasks that would typically be done on the server, like form submissions.

One of the most common situations for this is when you are collecting email sign ups on your site. In a typical set-up with a server, you would be sending your form submission to your server and your server would have a PHP script that would hit the API of your email marketing provider.

But in a serverless set-up, you don't have that option. You might be tempted to just call the email marketing service directly, but you shouldn't be exposing the API key for your email service on the client-side, and many email services prevent you from calling their API from the front-end anyway.

There are some services that have started to pop up and fill this need, but it is pretty easily solved with an AWS Lambda.

Here's the simple Lambda that I use on the Epilocal site:

//index.js
const https = require('https');
const JsonResponse = require('./response');

const defaultOptions = {
    host: 'api.mailerlite.com',
    port: 443,
    headers: {
        'Content-Type': 'application/json',
        'X-MailerLite-ApiKey': 'YOUR_API_KEY',
    }
}

const post = (path, payload) => new Promise((resolve, reject) => {
    const options = { ...defaultOptions, path, method: 'POST' };
    const req = https.request(options, res => {
        let buffer = "";
        res.on('data', chunk => buffer += chunk)
        res.on('end', () => resolve(JSON.parse(buffer)))
    });
    req.on('error', e => reject(e.message));
    req.write(JSON.stringify(payload));
    req.end();
})

exports.handler = async (event) => {
    const { queryStringParameters } = event;
    let postData = { "email": queryStringParameters.email };
    if (queryStringParameters.form_type) {
        postData["fields"] = {"form_type": queryStringParameters.form_type}
    }
    const response = await post("/api/v2/groups/YOUR_GROUP_NUMBER/subscribers", postData);
    let lamdaResponse = new JsonResponse(200, response);
    return lamdaResponse.build();
};

//response.js
class JsonResponse {
  constructor(statusCode, message) {
    this.statusCode = statusCode;
    this.message = message;
  }

  build() {
    return {
      statusCode: this.statusCode,
      headers: {
        "Access-Control-Allow-Origin": "*",
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        statusCode: this.statusCode,
        message: this.message,
      }),
    };
  }
}

module.exports = JsonResponse;

You can see the file response.js is just some boilerplate for building a JSON response.

All of the action is happening in index.js. And even here, defaultOptions and the post function are just boilerplate for creating a POST request.

So the only code you really need to pay close attention to is in exports.handler. Here we are parsing the Lambda's queryStringParameters for an "email" parameter and saving that to a new object called postData. Then we are checking if the parameter "form_type" has been provided and if so, we save that to the postData object as well.

Then we are simply passing the postData object into our post function. Note, we are using await here, so exports.handler needs to be an async function.

Finally, we are taking the response that we get from our email marketing service and passing that to our boilerplate JSON response imported from response.js.

I use Mailerlite but you could easily adapt this to whatever email marketing service you use, as long as they have a similar API for adding new subscribers.

Checklist for Deploying Gatsby on AWS S3 with Cloudfront

gjdickens — Thu, 27 May 2021 09:02:46 +0000

At Epilocal, we are big believers in static site generators and in particular we use Gatsby. Although Gatsby plays pretty well together with AWS, there are some tricks to getting it set-up correctly.

I’ve put together a checklist that we use internally for all of our new set-ups that we wanted to share with anyone else who is deploying Gastby on AWS.

Tools used:

Gatsby with gatsby-plugin-s3
AWS S3
AWS Cloudfront
AWS Route 53
AWS Certificate Manager

Steps:

Step 1: Set up S3 bucket for static hosting

Create a new S3 bucket with public access allowed for access point policies only. This means that you will only be blocking public access for access control lists.

Your set-up should look like this:

Go to properties and select: Static Website Hosting. Select the option “Use this bucket to host a website”.

You will also see the endpoint url for your bucket on this page, copy it for later. (you will use it in Step 5 to setup CloudFront)

Step 2: Give IAM User CLI permission to deploy to bucket by adding a new bucket policy

Navigate to bucket policies in S3 by going to Permissions > Bucket Policy and add the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AddPerm",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::[YOUR BUCKET NAME HERE]/*"
        }
    ]
}
```



### Step 3: Install and configure gatsby-plugin-s3 for your Gatsby build
Install gatsby-plugin-s3 with either NPM or Yarn.

In your gatsby-config.js file, you will need to add the following:


```
    {
      resolve: `gatsby-plugin-s3`,
      options: {
        protocol: "https",
        hostname: "[YOUR CANONICAL DOMAIN NAME HERE (example:  www.epilocal.com)]",
        bucketName: "[YOUR BUCKET NAME HERE]",
      },
    }
```



### Step 4: Deploy Gatsby build to S3 bucket
From the command line run: `gatsby deploy`.

If you have multiple AWS profiles on your machine, you can prefix the id of the profile that you want to use by using this command instead: `AWS_PROFILE=[YOUR PROFILE NAME HERE] npm run deploy`.

### Step 5: Connect Cloudfront to S3 bucket
Create a new Cloudfront distribution using all defaults except for changing the following options:

* Origin Domain Name: copy and paste the endpoint url from your S3 bucket here rather than picking the drop-downs. It should look something like this: `[YOUR BUCKET NAME].s3.amazonaws.com`
* Viewer Protocol Policy: Redirect HTTP to HTTPS
* CNAME: `www.[YOURDOMAINNAME].com`
* Customer SSL Certificate: follow the links to AWS Certificate Manager to enable SSL on your domain if you haven’t yet already. (if you are not using Route 53 for your domain name, you can import your SSL certificate here instead). Note: be sure that you specify your SSL certificate to cover *.com and your root domain.

### Step 6: Create A and AAAA zones in Route 53 for your www domain and point it to your Cloudfront distribution
Define a new simple record in your Route 53 Hosted Zones.

Ensure that you type “www” in record name and select “Alias to CloudFront Distribution”. Then you should see your CloudFront region and distribution name and be able to select it from a drop-down. (where it says “Choose Distribution” in the image below).

![AWS Route 53 Zone Record](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b440s5bkfzsigbh4rq25.png)

Repeat the same process and create a AAAA record instead of an A record.

### Step 7: Create a second S3 bucket to redirect root domain requests to www
Create a new S3 bucket, this time you don’t need to allow public access.

Go to properties and select: Static Website Hosting. Select the option “Redirect requests” and under “Target bucket or domain” enter your www domain name.

![AWS S3 Static Redirect Properties](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zukttxa69z4bxszkfvp0.png)

### Step 8: Create second Cloudfront distribution for S3 redirect bucket
Create a new Cloudfront distribution using all defaults except for changing the following options:

* Origin Domain Name: copy and paste the url from your second S3 bucket here rather than picking the drop-downs. It should look something like this `[YOUR BUCKET NAME].s3.amazonaws.com`
* Viewer Protocol Policy: HTTP and HTTPS
* CNAME: [YOURDOMAINNAME].com
* Custom SSL Certificate: the same SSL certificate should also cover this distribution if you specified before that your SSL certificate would cover *.com and your root domain.

### Step 9: Set-up A and AAAA zones for root domain pointed to second Cloudfront distribution
Define a new simple record in your Route 53 Hosted Zones.

This time leave the record name blank. Select “Alias to CloudFront Distribution” and this time you should see your second CloudFront region and distribution name and be able to select it from the drop-down.