Forem: Uroš Štok

Extracting verification codes with Cypress

Uroš Štok — Sat, 03 Feb 2024 16:26:12 +0000

When it comes to automating signups, verification codes are almost always used. These codes serve as an additional layer of security, ensuring that the user signing up is genuine and has access to the email associated with the account.

Verification codes range from only digits to mixed characters; it all depends on the platform.

How do we go about parsing and extracting these codes?

There’s two ways: we either focus on the text and search based on that, or we look for html elements in the email that contain the said code. Sometimes, a bit of both is required. In this post, we’ll explore both methods and show some common examples.

If you're looking for a more general guide on how to automate the verification flow, check out Email verification with Cypress

Searching with Regex

If our verification code consists solely of digits, we can write a simple regex that searches for a digit string of a certain length. If the verification code is 6 digits, we can use the following regex:

\b(\d{6})\b

it("Should verify email", () => {
  let code;

  cy.mailiskSearchInbox(Cypress.env("MY_NAMESPACE"), {
    to_addr_prefix: "testuser",
    subject_includes: "verification code",
  }).then((response) => {
    const emails = response.data;
    const email = emails[0];
    // we search the text version of the email
    const match = email.text.match(/\b(\d{6})\b/);
    expect(match.length).toBeGreaterThan(0);
    code = match[1];
    expect(code).to.not.be.undefined;
  });

  console.log("Your code is: " + code);
  ...
});

In the above code, we used the text version of the email. This doesn't contain a lot of the HTML clutter like styles, tags and classes, making it less likely we'd match the wrong content.

But what if the code contains both digits and letters? In this case, the Regex method isn't quite enough unless there's a prefix before the code that we can use. In the above example, the text version looks like this:

Please enter this verification code to get started on CoolApp: \n> \n> 065406 \n> \n>

Let's modfiy the code so it includes both numbers and letters, making it harder to extract:

Please enter this verification code to get started on CoolApp: \n> \n> A3G56L \n> \n>

Since we know that the code is after verification code to get started on CoolApp: we can include this in our regex, modifying it like so:

verification code to get started on CoolApp:.\*([A-Z0-9]{6})

it("Should verify email", () => {
  let code;

  cy.mailiskSearchInbox(Cypress.env("MY_NAMESPACE"), {
    to_addr_prefix: "testuser",
    subject_includes: "verification code",
  }).then((response) => {
    const emails = response.data;
    const email = emails[0];
    // we search the text version of the email
    const match = email.text.match(/verification code to get started on CoolApp:.*([A-Z0-9]{6})/);
    expect(match.length).toBeGreaterThan(0);
    code = match[1];
    expect(code).to.not.be.undefined;
  });

  console.log("Your code is: " + code);
  ...
});

We constructed our regex in three parts. First the prefix which ends on CoolApp:, then any characters .* and finally the format of our code [A-Z0-9]{6}.

Of course, in this example, simply using \b[A-Z0-9]{6}\b would have been enough. However, this can fail depending on the text content and lenght of the code, which is why we used the prefix.

Searching with HTML

Using HTML is usually a surefire way to easily extract verification codes. The reason is that most codes are marked differently (e.g., bolded), making them easy for the user to see and for us to extract.

Let's use the cheerio package. This will allow us to use selectors to fetch certain parts of the HTML.

In this example the email's html contains the following:

...
<tr>
  <td class="h1 black" align="left" dir="ltr" style="...">065406</td>
</tr>
...

This is quite convenient, as in this example there is only one td element with h1 black. If there were more then we'd have to filter from higher in the hierarchy (e.g. tr, table, etc.).

it("Should verify email", () => {
  let code;

  cy.mailiskSearchInbox(Cypress.env("MY_NAMESPACE"), {
    to_addr_prefix: "testuser",
    subject_includes: "verification code",
  }).then((response) => {
    const emails = response.data;
    const email = emails[0];
    // we load the email html into cheerio
    const $ = cheerio.load(email.html);
    // we find the element using the selector
    const nodesText = $("td.h1.black").text();
    // since there's one element and it only contains the code we can extract it directly
    const code = nodesText;
    expect(code).to.not.be.undefined;
  });

  console.log("Your code is: " + code);
  ...
});

And there we have it – two ways verification codes can be parsed and extracted from emails.

Password reset with Selenium in Node

Uroš Štok — Sun, 08 Jan 2023 11:58:16 +0000

Testing password reset functionality is an important part of ensuring the security and reliability of any web application. In this tutorial, we will learn how to test password reset in Selenium with Node.js.

In order to test this functionality we'll also need some demo app which supports password reset. For ease of use the final code including the test app for this post is available on GitHub.

Selenium Setup

We'll be using the selenium-webdriver library in Node. You can add this into your project using the following command

npm install selenium-webdriver

We're not setup to use selenium quite yet though, we'll also need to install the relevant driver. In this example we'll be using chrome and install the ChromeDriver. For an easier setup let's use the chromedriver npm package which uses one of the latest ChromeDriver versions. It's installed like so

npm install chromedriver

This will make it easier for other developers to checkout our repository since they won't need to do a separate installation step for the driver, doing npm install will be enough.

Writing the test

We'll write a test which focuses on the key aspects of resetting passwords

Create a new user, this will make it easier to run the test by itself with no other dependent logic.
Login as this user, here we'll just confirm that the selected password works.
Reset the password for this user, this will be a two step process. First we'll trigger the password reset email, after it arrives we'll enter our new password and reset it.
Login with new password, finally we confirm that the new password is accepted and the password reset works as expected.

As mentioned before, we're using the app available in the repository which has both the frontend and backend already prepared.

Let's start by getting the parts not related to emails out of the way. Here are the contents of our password-reset.test.js

const { Builder, By } = require("selenium-webdriver");
const { MailiskClient } = require("mailisk");

(async function passwordResetTest() {
  let resetLink;
  const namespace = "mynamespace";
  const testEmailAddress = `test.${new Date().getTime()}@${namespace}.mailisk.net`;

  const mailisk = new MailiskClient({ apiKey: "YOUR_API_KEY" });

  // Create a new Chrome driver
  let driver = await new Builder().forBrowser("chrome").build();

  try {
    // Let's visit the signup page and create a new user
    await driver.get("http://localhost:3000/register");
    await new Promise((r) => setTimeout(r, 500));
    await driver.findElement(By.id("email")).sendKeys(testEmailAddress);
    await driver.findElement(By.id("password")).sendKeys("password");
    await driver.findElement(By.css("form")).submit();
    await new Promise((r) => setTimeout(r, 500));

    // We should have been redirected to the login page, so let's enter our credentials
    await driver.findElement(By.id("email")).sendKeys(testEmailAddress);
    await driver.findElement(By.id("password")).sendKeys("password");
    await driver.findElement(By.css("form")).submit();
    await new Promise((r) => setTimeout(r, 500));

    // Let's send a password reset email
    await driver.get("http://localhost:3000/forgot");
    await new Promise((r) => setTimeout(r, 500));
    await driver.findElement(By.id("email")).sendKeys(testEmailAddress);
    await driver.findElement(By.css("form")).submit();
    // the reset email is sent here!

    // Now we wait for the email to arrive extract the link
    // TODO: we'll need to implement this part

    // We visit the reset link and set the new password
    await driver.get(resetLink);
    await new Promise((r) => setTimeout(r, 500));
    await driver.findElement(By.id("new-password")).sendKeys("newpassword");
    await driver.findElement(By.css("form")).submit();
    await new Promise((r) => setTimeout(r, 500));

    // Let's try logging in again, but this time with the new password
    await driver.get("http://localhost:3000");
    await new Promise((r) => setTimeout(r, 500));
    await driver.findElement(By.id("email")).sendKeys(testEmailAddress);
    await driver.findElement(By.id("password")).sendKeys("newpassword");
    await driver.findElement(By.css("form")).submit();
    await new Promise((r) => setTimeout(r, 500));

    // If we successfully log in, the app will redirect us to the dashboard
    let currentUrl = await driver.getCurrentUrl();
    if (currentUrl !== "http://localhost:3000/dashboard") {
      throw new Error(`Expected url to be 'http://localhost:3000/dashboard' got '${currentUrl}'`);
    }
  } catch (error) {
    console.error(error);
  } finally {
    // Close the browser
    await driver.quit();
  }
})();

The provided code above implements most of the key steps in resetting the password. We only need some way to receive the email programmatically and get the reset link.

Receiving Email

We'll use the mailisk library to fetch the password emails. First install it with

npm install mailisk

The test sample above already contains some relevant code, such as importing the library and creating a client

const { MailiskClient } = require("mailisk");
...
const mailisk = new MailiskClient({ apiKey: "YOUR_API_KEY" });

In order to use the client, you'll also need an api key, you can find yours in the dashboard.

Mailisk works based on namespaces you can think of these as a catch-all address which we can access programmatically. For example if we have the namespace mynamespace we can find all emails sent to john@mynamespace.mailisk.net (you can replace john with any valid email address).

...
const namespace = "mynamespace";
const testEmailAddress = `test.${new Date().getTime()}@${namespace}.mailisk.net`;

We use this snippet of code to create a unique email address per test. Every time the test starts it'll create a new user using a random email address, which will look like this

test.123456789@mynamespace.mailisk.net

This means we don't have to worry about clutter due to existing emails (e.g. it returning an old email with an invalid reset link).

searchInbox

In order to read the email we'll fill out the missing code using mailisk's searchInbox function

// Now we wait for the email to arrive extract the link
const { data: emails } = await mailisk.searchInbox(namespace, {
  to_addr_prefix: testEmailAddress,
});
const email = emails[0];
resetLink = email.text.match(/.*\[(http:\/\/localhost:3000\/.*)\].*/)[1];

The searchInbox function takes a namespace and options. We use the to_addr_prefix option to filter out emails sent to other addresses (e.g. john@mynamespace.mailisk.net). In combination with the email being random this ensures that we're guaranteed to only find the password reset email.

By default the searchInbox function filters out older emails and waits for at least one email to be returned. Which is why we don't need to pool/wait manually for the results.

As a response we'll get an array of emails, but we only care about the one. We'll run some regex to filter out the link and assign it to resetLink so it can be used later in the test.

And that's it. We've got an automated test that will reset the password and try logging in again with the new password.

Receive emails with Node

Uroš Štok — Tue, 01 Nov 2022 18:02:09 +0000

You've probably found a dozen posts about sending emails with Node. But what about receiving them?

Why?

When would you want to receive emails instead of sending them? The most common use cases are for testing password resets and email flows. A common email flow is signing up, receiving a code and entering that code to signup.

How?

We'll be using the mailisk library for receiving emails. This is a free email testing service which allows us to read emails. It works by creating a virtual email server that we can send emails to.

The email addreses look like this <anything>@mynamespace.mailisk.net. Where mynamespace is the name of your namespace. An email sent to an address that ends in @mynamespace.mailisk.net can be read using the API.

Setup

Install the library with

npm install mailisk

We'll interact with the API using the client, you can create one by passing your API Key

const { MailiskClient } = require('mailisk')

const mailisk = new MailiskClient({ apiKey: 'YOUR_API_KEY' })

Receiving Email

We can use the searchInbox function to receive emails. It only needs one parameter, our namespace.

const { MailiskClient } = require('mailisk')

const mailisk = new MailiskClient({ apiKey: 'YOUR_API_KEY' })

mailisk.searchInbox('mynamespace').then(console.log)

A note here is that, by default, this will only return the latest emails - any emails received in the past 5 seconds. This can be changed by overriding the from_timestamp parameter. You can check the docs for more information.

With that in mind, let's change it so it gives us all the emails in the inbox, regardless of when they were received

mailisk.searchInbox('mynamespace', { from_timestamp: 0 }).then(console.log)

In order to show this works we'll need to send an email. You can send one using your own email provider (e.g. Gmail). Any address that ends with your namespace will work, here let's simply send it to john@mynamespace.mailisk.net

After sending the email and executing the script, this is what we get in the console

{
  total_count: 1,
  options: { limit: 10, offset: 0, from_timestamp: 0, wait: true },
  data: [
    {
      id: '1667305800547-9KJHqVLG5',
      from: { address: 'test@example.com', name: '' },
      to: [ { address: 'john@mynamespace.mailisk.net', name: '' } ],
      subject: 'This is a test',
      html: 'false',
      text: 'Something goes here',
      received_date: '2022-11-01T12:30:00.000Z',
      received_timestamp: 1667305800,
      expires_timestamp: 1667565000,
      spam_score: 0.1
    }
  ]
}

Sending emails

To make testing a bit easier we can also use mailisk to send emails. These can only be sent to mailisk namespaces though

...
const namespace = 'mynamespace'

await mailisk.sendVirtualEmail(namespace, {
  from: 'test@example.com',
  to: `john@${namespace}.mailisk.net`,
  subject: 'Testing',
  text: 'This is a test.',
})

As an example, let's send ourselves an email and then read it

const { MailiskClient } = require('mailisk')

const namespace = 'mynamespace'

// create client
const mailisk = new MailiskClient({ apiKey: 'YOUR_API_KEY' })

async function main() {
  // send email
  await mailisk.sendVirtualEmail(namespace, {
    from: 'test@example.com',
    to: `john@${namespace}.mailisk.net`,
    subject: 'Testing',
    text: 'This is a test.',
  })

  // receive latest email
  const { data: emails } = await mailisk.searchInbox(namespace, {
    to_addr_prefix: `john@${namespace}.mailisk.net`,
  })

  // print out the subjects of all the recieved emails
  for (const email of emails) {
    console.log(`Received email with subject '${email.subject}'`)
  }
}

main().then(() => console.log('Done'))

You'll notice we also used to_addr_prefix here. This let's us filter the emails we fetch by their destination address. In this example we sent an email to john@..., and therefore we only care about emails sent to john@.... If we didn't filter this we would also return other emails, such as ones sent to somethingelse@...

Closing notes

This should be enough to get you up and started with receiving emails in Node.

If you're using Cypress take a look at our post on Password reset with Cypress, which features the mailisk cypress plugin (and an example repository). Otherwise there's additional information about options for fetching emails in the documentation and API Reference

Password reset with Cypress

Uroš Štok — Thu, 06 Oct 2022 17:10:42 +0000

Every website that has user accounts with passwords has a "Forgot password?" function. Suffice to say that this is a common test case that you usually want to be covered.

If you'd like to follow along or just want to see the final result checkout the repository.

With that out of the way. First, let's take a look at how this password reset functionality usually works:

User clicks "Forgot password", they're taken to a reset password screen
User inputs the email they want to reset the password for, an email is sent
User clicks the link from the email, taking them to a new page where they enter their new password
User can now login with their new password

The hard part here is getting the password reset link from an email. Automating the rest is pretty easy with Cypress.

Cypress

We're using Cypress as the test framework. If this is your first time using Cypress, take a look at their Installation guide. The provided repository for this post already has Cypress setup.

Mailisk

For "getting the password reset link from an email" part, we'll be using Mailisk and it's Cypress plugin. This is an email testing service that offers API endpoints for reading emails.

This lets us create a virtual email server that we can send emails to. The emails look like this <anything>@mynamespace.mailisk.net. Where mynamespace is the name of the namespace.

So any email we send to these addresses can be read using the API, which will allow us to automate the email password reset. But more on that when we get to it.

cypress-mailisk setup

Since we're using Cypress we'll be using the cypress-mailisk plugin. This will give us a simple command to access the received emails.

First install the plugin using

npm install -D cypress-mailisk

After installing the package add the following in your project's cypress/support/e2e.js

import 'cypress-mailisk'

In order to be able to use the API we'll also need to add our API key to cypress.config.js. An additional thing we'll be adding is the MAILISK_NAMESPACE. Doing this will let us easily use a different namespace.

module.exports = defineConfig({
  env: {
    MAILISK_API_KEY: 'YOUR_API_KEY',
    MAILISK_NAMESPACE: 'YOUR_NAMESPACE', // we're also adding this
  },
})

Writing the test

In the app/cypress/e2e folder we'll add a password-reset.cy.js file with the following contents

describe('Test password reset', () => {
  let resetLink
  const testEmailAddress = `test.${new Date().getTime()}@${Cypress.env(
    'MAILISK_NAMESPACE'
  )}.mailisk.net`

  it('Should sign up a new user', () => {
    cy.visit('http://localhost:3000/register')
    cy.get('#email').type(testEmailAddress)
    cy.get('#password').type('password')
    cy.get('form').submit()
    // if the register was successful we should be redirected to the login screen
    cy.location('pathname').should('eq', '/')
  })

  it('Should login as user', () => {
    cy.get('#email').type(testEmailAddress)
    cy.get('#password').type('password')
    cy.get('form').submit()
    // if the login was successful we should be redirected to the dashboard screen
    cy.location('pathname').should('eq', '/dashboard')
  })

  it('Should reset password', () => {
    cy.visit("http://localhost:3000/forgot");
    cy.get('#email').type(testEmailAddress)
    cy.get('form').submit()
    // this will send an email with a reset link to the provided email address

    // TODO: we need to fill this out
  })

  it('Should visit reset link and set new password', () => {
    cy.visit(resetLink)
    cy.get('#new-password').type('newpassword')
    cy.get('form').submit()
    // if the reset was successful we should be redirected to the login screen
    cy.location('pathname').should('eq', '/')
  })

  it('Should login as user with new password', () => {
    cy.get('#email').type(testEmailAddress)
    cy.get('#password').type('newpassword')
    cy.get('form').submit()
    // if the login was successful we should be redirected to the dashboard screen
    cy.location('pathname').should('eq', '/dashboard')
  })
})

This is a basic test which:

creates a user
tries logging in
resetting the password
logging in again.

You probably noticed the following line

const testEmailAddress = `test.${new Date().getTime()}@${Cypress.env(
  'MAILISK_NAMESPACE'
)}.mailisk.net`

Since we'll be registering and resetting the password of a user, we need an email for them. This will create a string similar to this

test.123456789@mynamespace.mailisk.net

This matches the pattern Mailisk uses for emails

<anything>@mynamespace.mailisk.net

The only difference is that we added the current time to the email. We're using this as a filtering trick. If we run the same test later, we can be sure there won't be any other emails for this user, which means checking for a new email will be really easy. And since with Mailisk we have unlimited email addresses, there's no worry of using them up.

To improve on this you could change from using the timestamp to using something like a uuid which would make the email address even more unique.

Reading the email

Now let's take a look at the password reset part

it('Should reset password', () => {
  cy.visit("http://localhost:3000/forgot");
  cy.get('#email').type(testEmailAddress)
  cy.get('form').submit()
  // this will send an email with a reset link to the provided email address

  // TODO: we need to fill this out
})

After we enter the email and hit submit and email is sent to that email address. We need to read the email, extract the link and save it in resetLink so we can visit it in the next step.

Let's use the cy.mailiskSearchInbox command for this

it('Should reset password', () => {
  cy.get('#email').type(testEmailAddress)
  cy.get('form').submit()
  // this will send an email with a reset link to the provided email address

  cy.mailiskSearchInbox(Cypress.env('MAILISK_NAMESPACE'), {
    to_addr_prefix: testEmailAddress,
  }).then((response) => {
    const emails = response.data
    const email = emails[0]
    resetLink = email.text.match(/.*\[(http:\/\/localhost:3000\/.*)\].*/)[1]
    expect(resetLink).to.not.be.undefined
  })
})

Let's break down what this does:

mailiskSearchInbox

The cy.mailiskSearchInbox command takes a namespace, options and callback as it's parameters. For the namespace we're using what we defined in the env earlier.

For the options, we pass our email in to_addr_prefix. This way, only emails sent to this email address are returned. A namespace can have unlimited email addresses, so if we don't filter, it would return emails from all addresses in this namespace. Including the ones we're not interested in, like john@mynamespace.mailisk.net.

By default this command also does a few quality of life things for us:

It calls the API with the wait flag, this means the call won't timeout until at least one email is received or 5 minutes pass. We can adjust the timeout by passing timeout in the options
It uses a default from_timestamp of current timestamp - 5 seconds. This simply means that old emails will be ignored, we wouldn't want the command to return with an old email just because it's still in the inbox.

The callback

In the callback we get the data which represents all the emails found. Since in this scenario only 1 email will be sent, we filter it out. Next, we use some regex to look for this string http://localhost:3000/anythingafterthis.

And et voilà we have our link, we just save this in the global variable resetLink so it can be used in the next test (where it vists the link and resets the password).

And that's it, we've automated the password reset using Cypress and Mailisk.

If we run the E2E spec all of our tests should pass.

Hiding ids with express

Uroš Štok — Fri, 18 Mar 2022 10:21:42 +0000

When building APIs we often want to hide or obfuscate certain information from the end user. The most common of which is probably hiding IDs.

Why hide IDs?

There are many reasons you might want to hide IDs depending on the type of data you're serving.

For example if your API involves sharing documents (public google docs) it's a bad idea to allow your user's files to be enumerable by simply incrementing the ID received from the API.

Or you could simply not want to allow others to see how many users or apps you have (or gain over time, if you're using ID offsets).

How do we hide IDs?

For performance reasons you generally want the IDs to be reversible, therefore it's best to avoid directly hashing the ID. Instead you want to encrypt it before sending it to the user, and decrypt it when trying to access it in the backend.

Since the point of this post isn't security I'll be using the hashids library. This gives us an easy way to generate unique strings from numbered IDs. Hashids is by no means secure, it's just a simple way to obfuscate IDs.

Here's how we use it in code:

const hashids = new Hashids("secret salt");

const encodedId = hashids.encode(42);
const [originalId] = hashids.decode(encodedId);

Express middleware

Let's assume that we want to hide every public ID. This would mean we encode/decode every id field in the request and response body. We could also expand this to relational fields, for example user_id. By doing the same to every field that ends in _id.

To achieve this in express, we want to create two middleware. encodeMiddleware and decodeMiddleware.

/** helper function to replace IDs inside object */
function replaceIds(obj, replaceFunc) {
  if (obj == null) return obj;

  for (const key of Object.keys(obj)) {
    if (obj[key] == null) continue;

    if (typeof obj[key] === "object")
      obj[key] = replaceIds(obj[key], replaceFunc);
    else if (key == "id" || (key.length >= 4 && key.endsWith("_id")))
      obj[key] = replaceFunc(obj[key]);
  }
  return obj;
}

function encodeMiddleware(req, res, next) {
  var _json = res.json;
  res.json = (obj) => {
    res.json = _json;
    obj = replaceIds(obj, (v) => hashids.encode(v));
    return res.json(obj);
  };
  next();
}
function decodeMiddleware(req, res, next) {
  try {
    req.query = replaceIds(req.query, (v) => hashids.decode(v)[0]);
    req.body = replaceIds(req.body, (v) => hashids.decode(v)[0]);
  } catch (e) {
    console.error(`Could not decode id:`, e);
    return res.sendStatus(404);
  }
  next();
}

In the encodeMiddleware we're assuming the response will always be JSON and modifying it by replacing all id occurances with the encoded version.

In the decodeMiddleware we're assuming that the data can be either in the body or query and replacing all id occurances with the decoded version.

For both of these we could add exceptions to certain endpoints by comparing the req.url. Or just not using the middleware globaly.

I've also added a helper function that takes in an object and recursively replaces all ids using the supplied function.

Example code

Now here's an example with everything put together:

const express = require("express");
const Hashids = require("hashids");
const hashids = new Hashids("secret salt", 6);

/** helper function to recursively replace ids inside object */
function replaceIds(obj, replaceFunc) {
  if (obj == null) return obj;

  for (const key of Object.keys(obj)) {
    if (obj[key] == null) continue;

    if (typeof obj[key] === "object")
      obj[key] = replaceIds(obj[key], replaceFunc);
    else if (key == "id" || (key.length >= 4 && key.endsWith("_id")))
      obj[key] = replaceFunc(obj[key]);
  }
  return obj;
}

function encodeMiddleware(req, res, next) {
  var _json = res.json;
  res.json = (obj) => {
    res.json = _json;
    obj = replaceIds(obj, (v) => hashids.encode(v));
    return res.json(obj);
  };
  next();
}
function decodeMiddleware(req, res, next) {
  try {
    req.query = replaceIds(req.query, (v) => hashids.decode(v)[0]);
    req.body = replaceIds(req.body, (v) => hashids.decode(v)[0]);
  } catch (e) {
    console.error(`Could not decode id:`, e);
    return res.sendStatus(404);
  }
  next();
}

const app = express();
app.use(express.json());

// we're using the middleware globaly here
app.use(encodeMiddleware);
app.use(decodeMiddleware);

// sample endpoints to demonstrate encoding, decoding
app.get("/get-id", (req, res) => {
  res.json({ id: 5, name: "John" });
});
app.post("/send-id", (req, res) => {
  console.log(req.body);
  res.sendStatus(200);
});

app.listen(3000);

Encoding and decoding

If we call the GET endpoint we should get back some JSON data with the id and name:

> curl GET http://localhost:3000/get-id

{"id":"OPZexb","name":"John"}%

Which we did, except the id was automatically encoded. If we were to return anything that ended with _id in the JSON body the middleware would automatically encode it.

Now let's try sending the encoded id to the POST endpoint:

> curl -X POST http://localhost:3000/send-id \
   -H 'Content-Type: application/json' \
   -d '{"id":"OPZexb"}'
...

[server log]: { id: 5 }

And on the server we should see { id: 5 }. Which means the middleware successfully decoded the id we sent. Similarly if we send any values that might include _id it will automatically decode it.

Closing notes

We were able to add global middleware whichs finds all ids in the request or response bodies and encodes or decodes them as needed.

In production you will likely want to add a filter to prevent this middleware running on webhooks from 3rd party services. As they might use the id, _id syntax themselves.

Input validation in Express from TypeScript

Uroš Štok — Wed, 22 Dec 2021 14:01:52 +0000

Since this is a bit of a long post with multiple files I've provided a sample repo so you can see the final code in action, in case the post gets a bit too hard to follow.

Most issues with Express crop up with badly (not) checked inputs to endpoints. The easiest way to show this is with an example:

type RequestBody<T> = Request<{}, {}, T>;

interface UserBody {name: string};
app.post("/user", (req: RequestBody<UserBody>, res) => {
    return addUser(req.body.name);
})

Here we're using typescript with express for typed routes which give us autocomplete.

Despite this being typesafe, it can still crash our whole backend!

Nothing is preventing someone from posting an empty body, which express will happily accept, which will cause an exception in the addUser function, which will crash the whole server as the error is not caught.

Solution?

We could simply add some code checks (or catch some exceptions), for example:

type RequestBody<T> = Request<{}, {}, T>;

interface UserBody {name: string};
app.post("/user", (req: RequestBody<UserBody>, res) => {
    if (!req.body.name) return res.sendStatus(400); // we added this
    return addUser(req.body.name);
})

This is not a good solution though. For every new parameter we add to the endpoint (request body), we will have to make sure the code won't crash later. When you have a large API, it's really easy to miss some edge-case that brings the whole server down.

So what can you do?

ajv

A library called ajv offers schema validation, which allows you to validate data based on some pre-defined schema. This schema looks like this:

const schema = {
  type: "object",
  properties: {
    foo: {type: "integer"},
    bar: {type: "string", nullable: true}
  },
  required: ["foo"],
  additionalProperties: false
}

Just as an example, if we were to describe this schema using a Typescript interface, we would get this:

interface UserPostRequest {
    foo: integer,
    foo: string
}

But more on that later. First let's see how we could use ajs to create a middleware that would only allow the request if the req.body matched the schema:

// function that creates middleware by compiling the supplied schema
function validateBody(schema) {
  // compile schema
  const validate = ajv.compile(schema);
  // middleware that returns error if schema is not ok
  return (req, res, next) => {
    if (!validate(req.body)) return res.status(400).json(validate.errors);
    return next();
  };
}

const schema = {
  type: "object",
  properties: {
    name: {type: "string"},
  },
  required: ["name"],
}

app.post("/user", validateBody(userSchema), (req, res) => {
    return addUser(req.body.name); // name will never be undefined
})

Now, wouldn't it be cool if we could use that typescript interface to... generate the schema? What if we were able to convert our typescript interface into a json schema, which we would could then use for validation.

typescript-json-schema

Now that cool thing we wanted to do? This library does exactly that!

It requires a bit of a setup.

We need to have a schema_definition.ts file contains the interfaces we would like to convert into json schemas.
Some schemaGenerator.js script that will use this library to convert said file into schema.
And finally, a _schema.ts file that will be generated, all it will do is export the json schema so we can import it elsewhere. Essentialy a .json file wrapped in a .ts file.

To follow along easier you can clone the provided repo, which has this stuff already setup.

schema_definition.ts

As mentioned, this will hold the typescript interfaces that will be converted into json schemas. For the user example let's just write:

// schema_definition.ts
export interface UserPostRequest {
  name: string;
}

schemaGenerator.js

Pretty much the glue that holds this implementation together:

// schemaGenerator.js
const path = require("path");
const tjs = require("typescript-json-schema");
const fs = require("fs");

const settings = {
  required: true,
  ref: false,
};
const compilerOptions = {
  strictNullChecks: true,
};

const program = tjs.getProgramFromFiles([path.resolve("schema_definition.ts")], compilerOptions, "./");

const schema = tjs.generateSchema(program, "*", settings);
fs.writeFileSync(
  "_schema.ts",
  "const schema = " + JSON.stringify(schema) + " as const;\nexport default schema.definitions;"
);

And to use it, let's add a script to package.json:

scripts: {
  "schema": "node schemaGenerator.js"
  ...
}

Running this with npm run schema will create a _schema.ts file in the root directory.

Final code

If we were to expand our previous code, to support input validation, we would get this:

// index.ts
import express, { Request, Response, NextFunction } from "express";
import _schema from "./_schema";
import { UserPostRequest } from "./schema_definition";
import Ajv from "ajv";

const app = express();
app.use(express.json());

const ajv = new Ajv();

// validation middleware
function validateBody(schema: object) {
  const validate = ajv.compile(schema);
  return (req: any, res: any, next: NextFunction) => {
    if (!validate(req.body)) return res.status(400).json(validate.errors);
    return next();
  };
}

// helper type
type RequestBody<T> = Request<{}, {}, T>;

function addUser(name: string) {}

app.post("/user", validateBody(_schema.UserPostRequest), (req: RequestBody<UserPostRequest>, res: Response) => {
  return addUser(req.body.name); // name will never be undefined
});

app.listen(3000);

This whole example is available as a repo if you're having issues setting things up.

Closing notes

I hope you found this useful! Personally, it's my favourite way to add input validation if you're using typescript. It's so nice and clean!

Another cool alternative that allows for more powerful definitions (at the cost of losing clean interfaces) is typebox.

Typed routes in Express

Uroš Štok — Sun, 19 Dec 2021 10:15:23 +0000

While Express wasn't built with Typescript, there are type definitions available - @types/express. This adds typings for routes (specifically for this post, Request and Response).

I've looked around for ways of properly doing Request and Response types, and haven't found anything that works without breaking something else or being complicated. So here's how I usually implement typesafety into express routes.

Let's say we had an endpoint for adding a new user:

import express from "express";

const app = express();

app.post("/user", (req, res) => {
    req.body.name; // autocomplete doesn't work
});

app.listen(3000);

This is pretty standard javascript, besides using ESM imports, there's no reason we need typescript for this. So let's add some types:

import express, {Request, Response} from "express";
...
app.post("/user", (req: Request, res: Response) => {
    req.body.name; // autocomplete doesn't work
});

Note that this is what happens normally even if we don't specify the types, typescript infers the Request and Response type from the function automatically. So we didn't really do much here.

Request.body type

What if this endpoint needs some input body data? Currently when we type req.body autocomplete doesn't offer anything special. Let's change that.

We can pass an interface to the Request type parameter list so that Typescript knows what variables are available in req.body. It would look something like this:

type UserRequestBody = { name: string };
app.post("/user", (req: Request<{}, {}, UserRequestBody>, res: Response) => {
    req.body.name; // autocomplete works
});

We need to put {} for the first two parameters as the thing we want (body) is actually the third type parameter. As we can see in the Request definition:

interface Request<
        P = core.ParamsDictionary,
        ResBody = any,
        ReqBody = any, // this is the Request.body
        ...

Now this is quite chunky code for simply passing an interface for the request body. Luckily there's a better way, we simply define a helper type:

type RequestBody<T> = Request<{}, {}, T>;

With our cleaner definition we can simply use:

type RequestBody<T> = Request<{}, {}, T>;

type UserRequestBody = { name: string };
app.post("/user", (req: RequestBody<UserRequestBody>, res: Response) => {
    req.body.name; // autocomplete works
});

Other defintions

Now with our new found knowledge of how to write clean route typed code we can declare helper types for all our use cases!

// for .body
type RequestBody<T> = Request<{}, {}, T>;
// for .params
type RequestParams<T> = Request<T>;
// for .query
type RequestQuery<T> = Request<{}, {}, {}, T>;
// and so on... similarly for Response

Multiple types

To cover everything, we need to be able to specify multiple types, for example .body and .params. We can do so by simply adding a new type:

type RequestBodyParams<TBody, TParams> = Request<TParams, {}, TBody>

Typed example

Here's the full example from the start, now with typed routes:

import express, {Request, Resposne} from "express";

const app = express();

type RequestBody<T> = Request<{}, {}, T>;
type UserRequestBody = { name: string };
app.post("/user", (req: RequestBody<UserRequestBody>, res: Response) => {
    req.body.name; // autocomplete works
});

app.listen(3000);

Closing notes

That's it! This should allow you to create proper typed routes. The next step would be to add schema validation for these routes.