Forem: Anders Ramsay

How to save anonymous content to a database

Anders Ramsay — Sat, 02 Sep 2023 17:13:45 +0000

I recently released a new version of logd, a side project where part of the vision is to enable just starting to type notes without the ceremony of creating a new document, etc.

In that spirit, I made it possible for anonymous users to create notes that are saved locally, making it a kind of ad-hoc scratch pad. You can try that out now if you'd like.

However, I also wanted to provide the option to save those local notes to the cloud on login.

I did quite a bit of googling around for a solution that would support this but did not find any patterns I liked. One reason might be that this is a somewhat non-standard user flow. In a conventional model, you sign in first, and then create a note.

Since others might be interested in supporting saving anonymous content to a database, I decided to write a blog post about how I implemented this feature.

Solution overview

For this solution, I decided to use a combination of localStorage and query params to manage the process of saving anonymous content to a database. (At the end, I'll talk a bit about some other option I considered.)

Here is an overview of the solution, which I'll walk through in detail below:

Persist changes locally: While a user is anonymous, their note content is persisted via local storage. Nothing unusual there.
Use a param to trigger saving to the database: When a user clicks login or signup, we check to see if any local content has been entered, and if so append a query param with the "save anon content" route where we'll handle saving to the database. We reuse the same query param we'd use if a user tried to access a restricted page.
After login, redirect to the "save anon content" page: After login completes, our redirect param will send the user to the "save anon content" page.
Save local content to the database: On the "save anon content" page, we retrieve the local content, save it to the database, which generates an id, which we then use to redirect the user to the newly created note page.
Redirect to our new note and delete the local content: Finally, after redirecting to the new note page, we also include a param that will enable deleting the localStorage content, since it is now safely in the database.

Prerequisites and Setup

I implemented this version of Logd notes using the Remix framework with Node, and I'll therefore also use the same stack for this walk-through. I'm also assuming you already have basic knowledge of React with TypeScript.

You can find a fully working solution in this repo.

I'll be using the Remix Indie Stack as a starting point since it already includes both authentication as well as notes CRUD functionality. If you're following along writing code, you'll want to first create a new app using the Remix Indie Stack, as follows: npx create-remix@latest --template remix-run/indie-stack.

Let's get started!

1. Persist changes locally

This part is mostly basic functionality for persisting changes locally, with an added simple check for if content has been entered, and a function for appending a query param to the login link. Open up app/routes/_index.tsx and make the following updates:

// app/routes/_index.tsx
...
export default function Index() {
...
  // hook for persisting changes locally
  const [value, setValue] = useLocalStorage(
    ANON_USER_LOCAL_STORAGE_CONTENT,
    "",
  );

  // simple check for if content has been entered
  const noteHasContent = (value as string).trim() !== "";

  // display current save status
  function displaySaveStatus() {
    if (noteHasContent) {
      return (
        <span>
          Saved locally.{" "}
          <Link to={`/login${setParam()}`} className="text-gray-400 underline">
            Save to my notes
          </Link>
          .
        </span>
      );
    }
    return "";
  }

  // set the param if content has been entered
  function setParam() {
    return noteHasContent
      ? `?${REDIRECT_TO_PARAM}=${encodeURIComponent(SAVE_ANON_ROUTE)}`
      : "";
  }

  // replace the current view with the following
  return (
    <div className="flex h-full min-h-screen flex-col">
      <header className="flex items-center justify-between bg-slate-800 p-4 text-white">
        <h1 className="text-3xl font-bold">Anon Note To Db Example</h1>
        {user ? (
          <div>
            <Link to="/notes" className="">
              View Notes for {user.email}
            </Link>
          </div>
        ) : (
          <div className="">
            <Link
              to={`/join${setParam()}`}
              className="rounded bg-slate-600 px-4 py-2 text-blue-100 hover:bg-blue-500 active:bg-blue-600 mr-4"
            >
              Sign up
            </Link>
            <Link
              to={`/login${setParam()}`}
              className="rounded bg-slate-600 px-4 py-2 text-blue-100 hover:bg-blue-500 active:bg-blue-600"
            >
              Log In
            </Link>
          </div>
        )}
      </header>
      <main className="p-8">
        <div className="w-[400px] mx-auto">
          <textarea
            className="w-full rounded-md border-2 border-blue-500 px-3 py-2 text-lg leading-6"
            value={value}
            rows={8}
            onChange={(e) => setValue(e.target.value)}
            placeholder="Write something..."
          />
          <div className="py-2 text-sm text-slate-500">
            {displaySaveStatus()}
          </div>
        </div>
      </main>
    </div>
  );
}

With the above update, a redirect param will be passed when clicking signup or login, which we'll make use of in the next step.

2. Use a param to trigger saving to the database

Now, when the user clicks on login or signup, if they've typed in any content, a param will be appended to the URL. Note the use of REDIRECT_TO_PARAM and SAVE_ANON_ROUTE. I prefer to store anything pases as a param as a constant since it will be used in at least two places (the originating link and the handler on the target page), so we want to ensure they match.

Also, note the use of encodeURIComponent, which ensures passed values are properly formatted for appearing in a URL.

3. After login, redirect to the "save anon content" page

On the login page, we shouldn't need to make any updates, since we're re-using the existing redirectTo functionality. After login, the app should redirect to the route we passed in (SAVE_ANON_ROUTE). Note that the actual route can be whatever you want. In my case it's /save-anon-note.

Why a separate page?

Let's talk a bit about why we have a separate view just for saving local content to a database. One fair question is why we can't just complete this task right on the login page after the user signs in? However, that would not be a good idea, since, once the user is authenticated, they need to be redirected away from the login page, as that page should only be accessible to anonymous users.

Ok, so why not just handle everything on the server after the user is signed in? Unfortunately, that will not work, because localStorage is, by definition, only accessible on the client. We therefore need an intermediate page that we fully load so that we can retrieve the local content on the client and then save it to the database on the server.

4. Save local content to the database

Thanks to the param we passed on the homepage, we are redirected to the "save anon" page. This view contains the core part of the solution.

Go ahead and create a new page at app/routes/save-anon-note.tsx and add the code below.

// app/routes/save-anon-note.tsx
import type { ActionArgs, LoaderFunction } from "@remix-run/node";
import { json, redirect } from "@remix-run/node";
import { useNavigate, useSubmit } from "@remix-run/react";
import { useEffect } from "react";
import { LiaSpinnerSolid } from "react-icons/lia";
import { createNote } from "~/models/note.server";
import { getUserId, requireUserId } from "~/session.server";
import {
  ANON_USER_LOCAL_STORAGE_CONTENT,
  LOCAL_NOTE_SAVED_PARAM,
} from "~/shared";

// redirect users who are not signed away from this page
export const loader: LoaderFunction = async ({ request }) => {
  const userId = await getUserId(request);
  if (!userId) return redirect("/");
  return json({});
};

// handle the programmatic form submit
export const action = async ({ request }: ActionArgs) => {
  const userId = await requireUserId(request);

  const formData = await request.formData();
  const title = formData.get("title");
  const body = formData.get("body")?.toString() || "";

  if (typeof title !== "string" || title.length === 0) {
    throw redirect("/", 400);
  }

  const note = await createNote({ body, title, userId });

  return redirect(`/notes/${note.id}?${LOCAL_NOTE_SAVED_PARAM}=true`);
};

export default function SaveAnonNote() {
  const navigate = useNavigate();
  const submit = useSubmit();

  useEffect(() => {
    // after the page has mounted, look for content in local storage
    const localContent = window.localStorage.getItem(
      ANON_USER_LOCAL_STORAGE_CONTENT,
    );

    if (localContent) {
      handleAnonNote(localContent);
    } else {
      console.warn("no local content found");
      navigate("/");
    }

    function handleAnonNote(localContent: string) {
      try {
        const noteContent = JSON.parse(localContent);
        if (noteContent.trim() === "") {
          throw new Error("No note content");
        }

        const lines = noteContent.split("\n");
        const title = lines[0];
        const body = lines.length > 0 ? lines.slice(1).join(" ") : "";

        let formData = new FormData();
        formData.append("title", title);
        formData.append("body", body);

        submit(formData, { method: "post" });
      } catch (error) {
        console.warn("anon note error: ", error);
        navigate("/");
      }
    }

  }, [navigate, submit]);

  // on the page itself, we're just displaying a spinner while this brief process completes
  return (
    <div className="flex h-full flex-col items-center justify-center">
      <LiaSpinnerSolid
        size={"30px"}
        title="Loading"
        className="text-primary animate-spin text-6xl"
      />
    </div>
  );
}

There's quite a bit going on here, so let's break it down step-by-step:

On the server side (in the loader method), we confirm the user is authenticated and redirect away if they are not.
We use the useEffect method to determine that the page has mounted and that we are able to read from localStorage.
If no local content is found, we redirect to the homepage, canceling the process.
If local content is found, we do some basic validation and then parse out the content title vs body. (This part is not really specific to saving anonymous content to a database.) Then, we use Remix's useSubmit hook to programmatically submit our note values to the server, after which point we handle the process very similarly to as if a user had created a new note.
See the action method for how we handle the form submit, create the note and redirect to the newly created note. Note also that we are passing in a LOCAL_NOTE_SAVED_PARAM which we'll use to remove the local content after the redirect.
Finally, on the page itelf, we just display a spinner while we complete the above process.

5.Redirect to our new note and delete the local content

If all goes well, we redirect to the new note and pass our LOCAL_NOTE_SAVED_PARAM param.
On the note detail view, we check for this param and use it to safely clear local storage, knowing that it has been saved to the database. And with that we are done 🎉.

// Only relevant snippets included
// See app/routes/notes.$noteId.tsx for the complete code
...
import {
  ANON_USER_LOCAL_STORAGE_CONTENT,
  LOCAL_NOTE_SAVED_PARAM,
} from "~/shared";
...
export default function NoteDetailsPage() {
  ...
  const navigate = useNavigate();
  const [searchParams] = useSearchParams();
  const localNoteSaved = searchParams.get(LOCAL_NOTE_SAVED_PARAM);

  useEffect(() => {
    // if the param is found, remove the local content and redirect to the current page, and remove the url that had the param from this history stack with replace set to true.
    if (localNoteSaved) {
      window.localStorage.removeItem(ANON_USER_LOCAL_STORAGE_CONTENT);
      navigate(location.pathname, { replace: true });
    }
  }, [localNoteSaved, location.pathname, navigate]);
...

Final Thoughts

I considered a few other solutions before going wih the one described above.

For example, I considered creating a guest session for every anonymous user. This would have the advantage of allowing for just storing everything in the database. However, it also comes with some security implications and also felt a bit more complex. I also considered using a JWT stored as cookie as a temporary note id.

But in the end, I felt the above solution, while containing many small steps, was the simplest and fastest to implement. Additionally, there are fewer security concerns, since I do not write to the database until the user has been authenticated. And last but not least, I am using query params to manage the flow, which is a tried and true pattern.

This is not a very complex solution, but it does involve a series of small steps, which means there is quite a bit of surface area for failure. For this reason, it's a great candidate for one or more E2E tests, something I might write about in another post.

How to set up a local E2E test development environment

Anders Ramsay — Mon, 05 Sep 2022 18:58:49 +0000

This post is part of the E2E Testing Series. If you haven't already, I suggest going back and reading previous posts before continuing.

Just give me the code...

Here is the "tl;dr" version of the article below, with just the commands to run and the code you'll need to add. If anything isn't clear, I suggest reading the full post.

Install the sample app:

 npx create-next-app e2e-testing --ts --use-npm

Create a src directory and move the pages and styles directories into it.
Install these packages:

 npm i -D cypress cypress-watch-and-reload start-server-and-test

Rename the dev script in your package.json to dev:next.
Add these scripts to your package.json:

 "cy:e2e": "cypress open --e2e --browser chrome --config baseUrl=http://localhost:3000",
 "dev": "start-test dev:next 3000 cy:e2e"

Run npm run dev to initialize Cypress.
Update/add these files (click on the file link to view the code):
Run the dev script to enable E2E test development with hot reloading of test and source files.

Introduction: Keeping E2E tests front and center

One of most common anti-patterns I see relating to E2E testing is that, while a test suite has been set up and some tests have been written, it is almost never run locally and new tests are only rarely added for new features. This obviously is not a good thing, since we effectively are wasting all the effort that went into setting up the test suite.

But why would we let such a valuable resource go to waste?

One simple answer is that it is out of sight, out of mind. In other words, while the command to run the E2E test locally does exist, it's something you have to remember to do, and it's kind of a hassle, since you usually have to first fire up the dev environment, and then start the E2E runner in a second terminal.

One way to address this problem is to automate running of the E2E test suite whenever you start up your development environment. As we'll see, this can have a significant positive impact on your E2E practice.

Add a single command for starting both dev environment and test runner

To help ensure E2E tests get written, we want to make it is easy to have both our app running and the test runner in a hot-reloading state. In other words, as we write tests, we want to be able to get quick feedback on if tests are passing or not.

When we are done with the steps below, you'll be able to just run npm run dev to both start you dev environment and also start up your E2E test runner in dev mode, meaning that it will rerun tests automatically if changes are made either to the tests or your code. As an added bonus, it will reduce the amount of terminal windows you have open in your IDE.

Setting up our sample app

We'll be using NextJS as the basis for our sample app. One reason for this is that our newsletter feature will require an API endpoint, and NextJS makes it super easy to add that.

To get started, go ahead and run the following to install the sample app:

npx create-next-app e2e-testing --ts --use-npm

You can replace e2e-testing with any app name you prefer. Both the --ts and --use-npm are optional. NPM is just my personal preference, and I am a strong proponent of using TypeScript.

Install the E2E test runner and utility packages

For this tutorial, we're going to use Cypress, which in my opinion currently is the best end-to-end testing solution. (My one gripe with Cypress is the lack of support for testing of older browsers, which are the ones often causing regressions.)

Let's install all the packages we'll need, as development dependencies:

npm i -D cypress cypress-watch-and-reload start-server-and-test

Add the dev scripts

Next, let's update the scripts section of our package.json as follows:

First, rename the dev script to dev:next. Then, add the following scripts:

"scripts": {
    ...
    "cy:e2e": "cypress open --e2e --browser chrome --config baseUrl=http://localhost:3000",
    "dev": "start-test dev:next 3000 cy:e2e"
    ...
}

Configuring Cypress

We need to configure Cypress to use TypeScript and also to hot reload when our source files change.

Add Typescript support

Cypress has built-in Typescript support. To enable it, and allow for writing TS-based tests, we need to add a tsconfig.json to the root of the cypress folder.

Add the following code to cypress/tsconfig.json and you should now be able to create test files using Typescript.

{
  "compilerOptions": {
    "target": "es5",
    "lib": ["es5", "dom"],
    "types": ["cypress"]
  },
  "include": ["**/*.ts"]
}

Also, update the main tsconfig.json in the root directory to exclude the cypress directory, so we do not compile test files as part of compiling the main source code.

{
  ...
  "exclude": ["node_modules", "cypress"]
}

Get Cypress to hot reload if your source files change

By default, the Cypress runner will only re-run if you modify test files. However, we really also want it to restart when we make changes to our code. To achieve this, we need to make the following updates:

First, update cypress.config.ts as follows:

import Cypress, { defineConfig } from "cypress";

interface Config extends Cypress.UserConfigOptions<any> {
  "cypress-watch-and-reload": {
    watch: string | string[]
  }
}

const config: Partial<Config> = {
  "cypress-watch-and-reload": {
    watch: ["src/**/*"],
  },
  e2e: {
    setupNodeEvents(on, config) {
      return require("cypress-watch-and-reload/plugins")(on, config);
    },
  },
}

export default defineConfig(config);

Next, add the following to the end of the file cypress/support/e2e.ts.

require("cypress-watch-and-reload/support");

Your local environment and dev command should now be set up.

Go ahead and run npm run dev and you should see the dev server first start up, then the test runner will start. The first time you run Cypress, you may be asked to click through their startup wizard.

Btw, a huge shout-out to Gleb Bahmutov who is the author of both the utility packages we are using here! 👏 👏 👏

Next Up: A Suggested Process for writing E2E tests

A Suggested Process for Writing E2E Tests

Anders Ramsay — Mon, 05 Sep 2022 18:57:14 +0000

This post is part of the E2E Testing Series. If you haven't already, I suggest going back and reading previous posts before continuing.

Process Overview

We're going to walk through a sample process for how to write E2E tests and touch on some best practices along the way. Here's an overview of the general process I tend to follow:

Create a test file and add pending tests for all the feature requirements.
Write the minimum code needed to get each test to pass.
With the test runner in watch mode, go back and refactor and clean up code as needed.

If you're familiar with Test-Driven Development (TDD), the above might look familiar. I've found that writing E2E tests is a great opportunity to put TDD into practice.

As our sample feature, we'll be walking through writing the tests for the Newsletter signup form I discussed in the series intro.

I picked it because it's a small but still complete feature, so a great basis for conveying the E2E test-writing process.

To get started, we'll create our main test file.

Create the test file and add pending tests

If possible, I suggest collecting all tests for each feature in the same test file, and then match the name of file to the test title. Let's do that now.

(I'm assuming you've already set up your local environment.)

Create a new file at cypress/e2e/newsletterSignup.cy.ts.

Note that in version 10 of Cypress E2E test files are placed in the directory cypress/e2e and use the .cy extension in the file name, which replaces the previous .spec extension for E2E tests.

Add the following code to this file:

// cypress/e2e/newsletterSignup.cy.ts

describe("Newsletter signup", () => {  // ** 1 **
  beforeEach(() => {  // ** 2 **
    cy.visit("/newsletter");
  });

  it("displays a message explaining the form's purpose");  // ** 3 **

  it("allows for completing a newsletter signup");

  it("allows for submitting the form by using the enter key");

  it("displays an error message if input value is not a valid email address");

  it("displays an error message if the input field is empty");

  it("clears the email input if signup is successful");

  it("persists the email input value if there is an error");
});

Code notes

This is the describe block, which is the method used to group test specs. I recommend matching the name of this block (e.g. here the name is "Newsletter signup") to the name of the test file and, if possible, also to what this feature is named elsewhere, eg in a User Story. This is a convention I like to follow to encourage only having one set of feature tests per file and also to create less confusion when trying to locate a test. The value of this will become more apparent when you start to have large numbers of tests.
The beforeEach block will run before every test in the describe block. This is a great place to tell the test runner where it should navigate to. Also, when applicable it can be a good place to place the app in desired state, eg using a login command.
This is our list of pending tests. The test descriptions for these tests should together make up the requirements for this feature. If you are working off a User Story, the list of tests should match your acceptance criteria.

Using your E2E tests as your "live" todo list

At this point, your Cypress console should look something like this:

Go ahead and click on "newsletterSignup" and you should now see the test runner with a list of pending tests.

With the test runner now in watch mode, as you update your tests and source code, our tests will re-run automatically and update tests to passing or failing.

I personally find it to be a great approach to have this type of "live" todo list, helping me remain assured that I am fully aligned and building what actually is being requested, and having a clear sense of my progress.

Adding our first test

We're going to start by getting the first pending test to pass, which also happens to be fairly basic.

In cypress/e2e/newsletterSignup.cy.ts, go ahead and replace the first pending test with the following code:

it("displays a message explaining the form's purpose", () => {
  cy.get('[data-test="newsletter"]').contains("Sign up for our newsletter");
});

Cypress tests allow for chaining of methods, which means they often are one-liners like the one above.

The test divides into two parts:

The cy.get part, which is where we select the element we want to test.
The .contains part which is one of many types of assertions. In this case, we are asserting that we will find a specific text block in the previously selected element.

(At the risk of causing some confusion, .get is also an assertion that an element exists, and .contains can also be used to select an element, for use in other assertions.)

You might have noticed the use of '[data-test="newsletter"]' to identify our newsletter DOM element, which might look a bit odd. We'll talk about that in more detail in a bit, but first let's get the above test to pass.

Getting our first test to pass

If we look at our test runner, we see that the test we added above is failing.

This is expected, since we haven't yet added the code to make it pass. Let's do that now.

Add a new file at src/pages/newsletter.tsx. Since we are placing a file in the special pages directory, NextJS will create the new corresponding route /newsletter.

Add the following code to the new file:

// src/pages/newsletter.tsx

import type { NextPage } from "next";

const Newsletter: NextPage = () => {
  return (
    <form data-test="newsletter">
      <h3>Sign up for our newsletter!</h3>
    </form>
  );
};

export default Newsletter;

If you now go back to the test runner, you should see that there is a green checkmark next to your first test, which means it's passing! :celebrate:

Making tests more resilient with Test IDs

Note the data-test="newsletter" attribute added to the form tag above.

<form data-test="newsletter">

"newsletter" is our "test id" and what we use to select this element in our test with the selector [data-test="newsletter"].

cy.get('[data-test="newsletter"]')...

This is a fairly common pattern in E2E testing, where we make use of a custom data attribute to identify the element(s) we want to test. Note that you can put whatever you want after the data- part, but I prefer using test.

Why would we want to do this?

I think the easiest way to understand the benefits of using test ids, is to look at the alternatives to identifying elements on a page:

Use a DOM selector, such as maybe adding an ID to the element, eg <form id="signupForm">.
Using a class name, eg <form class="signup-form">.
Create a selector based on the current DOM structure, such as main form h3.

The main issue with these approaches is that they're all fragile.

Every time a developer touches this code, it is possible they might modify any of these due to some implementation need, and unknowingly cause a false negative test failure.

Use of test ID allows us to separate test-specific code and implementation code, making tests more resilient to unrelated code changes.

Adding the main user flow test

Next, let's add a test for the main user flow: completing and submitting the signup form.

This test will be a bit more advanced. Go ahead and update the next pending test to be as follows:

// cypress/e2e/newsletterSignup.cy.ts

it("allows for completing a newsletter signup", () => {
  const testEmail = "test@email.com";

  cy.intercept("POST", "/api/newsletter", (req) => { // ** 1 **

    expect(req.body.email).to.contain(testEmail); // ** 2 **

    req.reply({  // ** 3 **
      statusCode: 200,
      body: {
        success: true,
        message: "Success message",
      },
    });
  });

  cy.get('[data-test="emailInput"]').type(testEmail); // ** 4 **
  cy.get('[data-test="formSubmit"]').click(); // ** 5 **
  cy.get('[data-test="successMessage"]').should("exist"); // ** 6 **
});

The goal of this test is to ensure that a "happy path" newsletter signup flow works as expected.

Code Notes

There's quite a bit going on in this test. Let's walk through it step by step:

cy.intercept("POST"... - here, we are telling Cypress to intercept a network request that will happen later in the test. With the intercept feature, Cypress basically masquerades as the given endpoint we defined and returns the response we expect. The intercept feature is truly powerful and I highly recommend reading more about it in their docs. In our case, it allows us to fully test our client side form by actually submitting an email address without having to worry about clogging up some database with fake test email addresses.
This is where we assert that the client side sent the payload we expected to the back-end.
If the previous assertion passes, we send the expected success response.
cy.get('[data-test="emailInput... - this is where we type in the test email address.
cy.get('[data-test="formSubmit"]').click(); - here, we click the form submit button and, which results in an API call, that then is intercepted as per step 1 above.
cy.get('[data-test="successMessage"]')... - assuming our form submit was successful, we expect a success response from the server.

Getting the main user flow test to pass

Let's add the code needed to get the above test to pass. Update src/pages/newsletter.tsx with the following:

import type { NextPage } from "next";
import { useState } from "react";
import { ServerResponse } from "../pages/api/newsletter";

const Newsletter: NextPage = () => {
  const [emailInput, setEmailInput] = useState("");
  const [confirmMessage, setConfirmMessage] = useState("");

  const handleSubmit = async (e: React.FormEvent<HTMLFormElement>) => {
    setConfirmMessage("");
    e.preventDefault();

    const raw = await fetch("/api/newsletter", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        email: emailInput,
      }),
    });

    const response: ServerResponse = await raw.json();

    if (response.success) {
      setConfirmMessage(response.message);
      setEmailInput("");
      return;
    }
  };

  return (
     <form data-test="newsletter" onSubmit={handleSubmit}>
        {confirmMessage && (
          <div data-test="successMessage">{confirmMessage}</div>
        )}
        <h3>Sign up for our newsletter!</h3>
        <input
          type={"email"}
          value={emailInput}
          onChange={(e) => setEmailInput(e.target.value)}
          data-test="emailInput"
          id="emailInput"
          formNoValidate
        />
        <input type={"submit"} data-test="formSubmit" />
    </form>
  );
};

export default Newsletter;

We also need to add an API endpoint for submitting our form data. Create a new file at src/pages/api/newsletter.ts and add the following:

import type { NextApiRequest, NextApiResponse } from "next";

export type ServerResponse = {
  message: string;
  success: boolean;
};

export default function handler(
  req: NextApiRequest,
  res: NextApiResponse<ServerResponse>
) {
  const { email } = req.body;

  res.status(200).json({
    message: \`\"\$\{email\}\" was added to the mailing list. Thanks for signing up!\`,
    success: true,
  });
}

(I won't be going through this code in detail, as this is not a post about TS or React.)

Note that this API endpoint doesn't actually store the submitted email in a data store. However, if this were a real newsletter signup, the place to do it would likely be in the above file.

If we now check our test runner, we should see that the main user flow test ("allows for completing a newsletter signup") test is passing.

Getting remaining tests to pass

We've written the minimum code needed to get the current tests to pass. However, there are still several requirements that need to be fulfilled, such as ensuring that emails are validly formed. You'll find the completed tests and the corresponding code to get them to pass in the github repo.

Refactor and clean up your code with confidence

Now that we have passing tests for all the feature requirements, we can reap the benefits of our efforts when refactoring.

For example, we may want to DRY up our code, which, at least in my experience, can often lead to unintentionally breaking things. Additionally, I haven't added much in the way to styling, and that is also something we now can do with far less concern about breaking something.

Next Up: Ensuring tests run automatically on every code push

Now that we've written some tests, let's be sure and put them to work. In the next post, we'll show how to make sure your full test suite runs every time you push new code, and why that matters.

Ensuring Your E2E Tests Run On Every Code Push

Anders Ramsay — Mon, 05 Sep 2022 18:56:11 +0000

This post is part of the E2E Testing Series. If you haven't already, I suggest going back and reading previous posts before continuing.

Writing tests is only the beginning

We've walked through the process of how to write E2E tests. We showed the benefits of writing tests even before you write any code. And we showed the value of having your E2E tests running in watch mode to get quick feedback.

While this is all great, it's unfortunately not enough. While we're doing a great job of writing our tests we're still not going to get quick feedback on a general app regression. (Remember, when you are running tests in watch mode, that is only covering the current feature you are working on, and not on the full test suite.)

To achieve that, we need to ensure that our full test suite runs when we add new code and that we are notified about any failures before we merge the code.

These are the three main steps we'll complete in this post to achieve that goal:

Enable running the full test suite against a production build.
Automatically run the full suite when we push new code.
Prevent merge of code into the main branch unless all tests pass.

There are a number of tools and techniques we can use to achieve the above. In our case, we're going to use a combination of npm scripts, Github Actions, and Github branch protection.

Let's get started!

Enable running the full test suite against a production build

Up until now, we've only been running the specific tests associated with the feature we are working on, and we've been running them against the development server. However, to prevent regression, we need to run all the tests, and run them against a production build. This is the version of your code that is served to users on the live site, and that is what we want to run our full test suite against.

Add full test suite scripts

Most of what we need to do is to add the necessary scripts to run the full suite. Go ahead and make the following updates to scripts in your package.json file:

Rename the "build" script to "prestart". This leverages a scripts feature in npm, in which adding pre to another script will run that script first. In this case, we want to run a build before we run the start script, which is what is run to start the production server.
Update the start script to use a different port: Add -p 8080 to the end of the "start" script. This will allow for running the production server locally while also running the dev server, eg for debugging purposes. (Port 8080 is just my personal preference, you can pick a different port if you wish.)
Add a script for running the full Cypress test suite: "cy:run": "cypress run --e2e --browser chrome --config baseUrl=http://localhost:8080". In contrast to the "cy:open" script, this will run the full test suite in headless mode, and we can use it both for local debugging and to run in a deployed environment. Note that the port used in baseUrl has to match what we use in the start script.
Add a single command E2E script: "e2e": "start-test start 8080 cy:run". This script pulls together all the other scripts above. We are first running the start script (which in turn will be preceded by the prestart build script), then waiting for a server to be listening on port 8080, and finally running the full test suite against that port. As before, this port has to match the two other ports you set previously.

Your scripts section should now include the following scripts:

"scripts": {
     ...
    "prestart": "next build",
    "start": "next start -p 8080",
     ...
    "cy:run": "cypress run --e2e --browser chrome --config baseUrl=http://localhost:8080",
    "e2e": "start-test start 8080 cy:run"
  },

To make sure everything is working as expected, go ahead and run the e2e script, and you should see the entire above sequence being run, concluding with an 'All specs passed!' message in your terminal:

We'll be using this last script in the next section when pushing our code, but you can also use it locally for debugging purposes, such as when you are getting a test failure only when pushing code, but not seeing an error when running in watch mode.

Automatically run the full suite when we push new code

As a reminder, this and the next step in this tutorial assumes you are using github to store your code. There are many other options out there for achieving the same goal.

To ensure our e2e script runs on all new code that we push, we're going to add a github action. Add the following to .github/workflows/e2e.yml (You must place the file in this directory, though the name e2e is my personal preference.)

name: E2E Tests

on: [push] # ** 1 **

jobs:
cypress-run:
runs-on: ubuntu-20.04 # ** 2 **
steps: - name: Checkout
uses: actions/checkout@v2 - name: Cypress run
uses: cypress-io/github-action@v2 # ** 3 **
with:
command: npm run e2e # ** 4 **

This code will be triggered when we push to a github repo.

Code Notes

This is telling github to run this action every time we push new code.
This is the OS and version used for the virtual environment in which run our tests. I recommend always using a specific version, such as the latest stable version, rather than latest, which is risky because you may then suddenly start to see test failures caused by a version update that has nothing to do with your tests. See virtual-enviroments for the latest stable version.
We are using a ready-made action created by Cypress to run our tests.
This script name needs to match the e2e script we created earlier.

Go ahead and commit this code and push it to a github repo.

Open up your repo in your browser and click on the "Actions" tab, and you should see your action running.

You can click into the action and see a detailed log of all the action steps.

Prevent merge of code into the main branch unless all tests pass.

Just one more step to go!

Now that we've got our test running automatically on push, we want ensure that we don't merge code with failing tests. Similar to above, these instructions are specific to Github, but the general principles should be applicable regardless of the service you are using.

Achieving this requires setting up a branch protection rule.

Protecting the main branch

The main branch (ie the branch containing code that will be pushed to production) is the one we need to protect from any code that contains regressions.

In github, we can add this protection by creating a branch rule, which should:

Not allow pushing code directly to the main branch, but instead require a pull request.
Require that all checks, in our case just the E2E tests, pass before merge is allowed.

You can find instructions for how to do add branch rules in the github docs, but very briefly, in your repo, click on Settings > Branches > Add Rule.

In "Branch name pattern", enter the name of your main branch, eg "main" and check the box "Require a pull request before merging". (I'll also uncheck "Require reviews" for the purposes of this tutorial, but it's probably a good idea to leave it checked.)

Next,check off "Require status check to pass before merging", which will prompt you to "search for status checks". Somewhat confusingly, a "status check" corresponds to the name of job within an action rather than the name of the action itself. In our case, the name is cypress-run (which you'll see under jobs: in .github/workflows/e2e.yml.) Go ahead and type that in and you should see the job name to select.

Your form should look something like this:

Scroll down and click on "Create".

Testing that everything works

Let's make sure that our new branch rule works as expected. Go ahead and make a change to a test that you know will cause a failure, eg by adding an extra character to a test id.

To confirm that your test will fail, run your test suite locally using the e2e script.

Next, go ahead and commit the change and push your branch and create a pull request.

You should now see a message showing your checks have failed with the merge button disabled.

Now, undo the change you made and push again and this time you should (eventually) see all green checkmarks.

You've now set up a basic E2E testing system for writing tests and preventing code regressions! :celebrate: :celebrate: :celebrate:

Final Thoughts

This wraps up the E2E Testing Series. I hope you found it useful. If you want to keep learning, I recommend watching this video, in which the founder of Cypress talks about E2E testing best practices.

E2E Testing Series

Anders Ramsay — Mon, 05 Sep 2022 18:55:49 +0000

Introduction

This is a series of posts I've written about the process and benefits of writing and using End to End Tests (E2E).

We'll walk through how to set up your system to write them, how to use the tests you've created to stay focused, and last but not least, how to use E2E tests to prevent code regression.

I've broken the series into the following parts:

Series intro and prerequisites (current post).
Setting up your local environment for writing E2E tests.
A suggested process for writing tests.
Ensuring your test suite runs every time you push code.

In this post, I'll offer a general intro to E2E testing and assumed prerequisites. I hope you find it useful. Please feel free to post feedback in the comments.

What are E2E tests and why should you care?

One good way to understand E2E testing is to look at what can happen without it.

A project without E2E testing

You're working on a project that doesn't have any formal E2E testing in place.

Let's say you're adding a newsletter signup form to your web app.

To keep your code DRY, you re-use existing input and form submit components to implement your form. You test the form locally and everything is working great. You also click around your web app and everything else seems to be working great as well.

However, after deploying your code, you start getting notices that users no longer can reset their passwords.

But wait, you were working on newsletter signup, how could that possibly break password reset?

Actually, as it turns out, the password reset form uses some of the same components that you used when implementing newsletter signup.

When you made some tweaks to those components (maybe you decided to rename a prop) while adding the newsletter feature, you unintentionally broke the reset feature. Worse, since that feature is out of sight, out of mind, you didn't think to test it before shipping newsletter signup.

This is an example of what we call a code regression, where something that previously worked unexpectedly stops working, and it is one of the main reasons we write E2E tests.

E2E testing to the rescue

If you'd had an E2E testing system in place, such as the one we'll be creating, and you had good feature coverage in your tests, after modifying the components being used in password reset, you'd have received a test failure notification, and then be able to quickly address the problem, preventing a reset password failure in production.

As apps get more complex and more feature-rich, and as we get better at re-using code across our apps, the risk of regression increases, so writing these types of tests and ensuring we run them regularly becomes increasingly important.

In this series, we're going to walk through the process of setting yourself up for success by enabling quickly and effectively writing these types of tests, running them locally against your development environment to get quick feedback on possible issues, and then automatically running the full suite of tests every time you push code, to prevent merging in code that will lead to regressions.

We'll also discuss how integrating E2E tests into your development process can you help you write better code overall and help ensure that you remain focused on your goals.

In my opinion, E2E tests are the most valuable test type, because they simulate actual end users and are focused on outcomes rather than implementation details. However, while valuable, these types of tests can also be costly, in that setting up your system to run them and writing the tests themselves can be complex and time-consuming. And as your test suite grows in size, they can take a long time to run.

But here's the thing: if done right, you can overcome these challenges, and get the best of both worlds. Achieving that is one of the top goals of this series.

A quick note about terminology

I am using the term "E2E" or End to End Testing a bit loosely here. Some might call the types of tests we are writing a functional test, or you may have a different name. There are countless names and nuances for different types of tests, such as smoke test, integration test, regression test, and more, many of them overlapping.

It's easy to get caught up in the terminology and lose sight of the actual goals, such as writing higher quality code and preventing stuff from breaking. I prefer just using E2E as an umbrella term for any type of test that simulates a user using the app and which treats the app itself as a black box, just as a real user would. (I.e. they don't know and don't care about the inner workings of the app.)

Prerequisites

I am assuming you already have some experience with JavaScript generally and specifically with React and Node.
I'm also assuming you already have the latest stable version of Node and npm installed on your machine.
I'll be writing the sample code mostly in Typescript so I am assuming you are familiar with that as well.
Finally, I am assuming you have some experience using git, as we'll be using that in the last part of the series, to create branches, pull requests, and more.

However, to be clear, many of the principles we discuss will be agnostic to whatever language or tool you're using.

Code Repo

You can find all the source code we'll be walking through below in this repo. I recommend writing your own code rather than just copying from the repo, but you might find it to be a good reference in case you get stuck.

Forem: Anders Ramsay

How to save anonymous content to a database

Solution overview

Prerequisites and Setup

1. Persist changes locally

2. Use a param to trigger saving to the database

3. After login, redirect to the "save anon content" page

Why a separate page?

4. Save local content to the database

5.Redirect to our new note and delete the local content

Final Thoughts

How to set up a local E2E test development environment

Just give me the code...

Introduction: Keeping E2E tests front and center

Add a single command for starting both dev environment and test runner

Setting up our sample app

Install the E2E test runner and utility packages

Add the dev scripts

Configuring Cypress

Add Typescript support

Get Cypress to hot reload if your source files change

Next Up: A Suggested Process for writing E2E tests

A Suggested Process for Writing E2E Tests

Process Overview

Create the test file and add pending tests

Code notes

Using your E2E tests as your "live" todo list

Adding our first test

Getting our first test to pass

Making tests more resilient with Test IDs

Adding the main user flow test

Code Notes

Getting the main user flow test to pass

Getting remaining tests to pass

Refactor and clean up your code with confidence

Next Up: Ensuring tests run automatically on every code push

Ensuring Your E2E Tests Run On Every Code Push

Writing tests is only the beginning

Enable running the full test suite against a production build

Add full test suite scripts

Automatically run the full suite when we push new code

Code Notes

Prevent merge of code into the main branch unless all tests pass.

Protecting the main branch

Testing that everything works

Final Thoughts

E2E Testing Series

Introduction

What are E2E tests and why should you care?

A project without E2E testing

E2E testing to the rescue

A quick note about terminology

Prerequisites

Code Repo

Next up: Setting up your local testing environment