Forem: Eddy Nguyen

Schema-driven development in 2021

Eddy Nguyen — Thu, 15 Jul 2021 10:25:28 +0000

Schema-driven development is an important concept to know in 2021. A lot of full-stack applications are built with schema-driven technologies. Schema-driven development could help teams build products better and faster. What exactly is schema-driven development? What are the benefits of schema-driven development? We will explore the answers to these questions in this article.

What is schema-driven development?

The schema is a contract

First, let's understand what a schema is. A schema is a contract between two sides of a system. The schema communicates the type of requests that can be made and the expected type of response.

Depending on the context where the schema is used, each side could play a different role. For example, in traditional web applications, the side making a request is the client or browser. The side returning a response is the application server.

Example of a client making request to a server based on schema specification

A schema follows a specific, unambiguous type of language defined by the technology that you use. Schema language is usually programming-language agnostic that communicates common software ideas such as objects, enumeration, field types (e.g. string, integer). Some common schema languages you may have encountered before are YAML and JSON. The client and server should understand these concepts to fulfil their part of the contract.

If we think a system works like a restaurant where the client is the "customer" and the server is the "wait staff". Then, the schema is the "menu". Given a menu, the customer can quickly scan through to find the "dish" (or resource) to consume. In some cases, the customer could see the ingredients and tell to the wait staff to remove items from the final dish.

The schema is the menu. Via Unsplash

Schema-driven development ( SDD ) definition

SDD prioritises the design of the schema and uses it as the first-class citizen to communicate the responsibilities of the client and the server. This contract usually becomes the API.

Let’s say we have a web application being built by two teams: Frontend and Backend.

A non-SDD development process may look like this:

Frontend team contacts Backend team for data that Backend team owns.
Backend team receives the request and starts to create the endpoint that returns the data Frontend team needs.
Backend team finishes the endpoint and lets Frontend team know.
Frontend team tries out the endpoint but it may not have what they need.
Backend team goes back and fixes the endpoint.
Repeat steps 3-6 until Frontend team gets what they need.

On the other hand, the SSD process may look like this:

Frontend team proposes changes to the schema.
Backend team understands exactly what they need to implement.
While Backend team is creating an endpoint that fulfil their part of the contract, Frontend team already knows what that looks like and can mock it out.
Both teams finish their side of the contract.

Note: Just because you are working with a technology that has a schema does not mean you are applying SDD. The schema design and discussions must be front and centre for the process to be called SDD.

Benefits of schema-driven development

There are many benefits of schema-driven development, which allows teams to build better and faster applications.

1. Better cross-team communications

Effective communication is unambiguous, concise and intentional. These are also the attributes of a schema language. When a team proposes changes to a schema, the team on the other side should know what must be done.

In fact, I was in numerous projects where complicated changes can be understood from reading the schema proposals.

Without writing a single line of code.
Without countless chat messages.
Without tortuous video call meetings.

We just nodded at each other, and knew exactly what must be done. We understood each other as if we somehow develop Professor X's level of telepathic power.

2. Better API design

SDD forces developers to think about designing the contract in abstract schema language. This frees up any implementation details from the conversation. If two teams are involved, they can see whether they can fulfil their part of the contract and suggest changes early. This usually results in well thought out API design for both sides of the system.

If a side cannot fulfil their part of the schema after the implementation phase has started, they can let the other side know and propose different schema changes. This saves a lot of time and money because errors caught in the design process or early in the implementation phase are much cheaper to fix.

3. Independent client and server development

Each side of a schema should know exactly what the other side can provide. This knowledge allows development to happen at the same time because we can mock out each side’s payload. In a scenario of a SDD web application, the frontend team should know an endpoint’s response from the schema and therefore can mock said response to code their part without needing to wait for the backend team to do any work.

4. Clear entity relationships

In software development, we usually need to identify various entities and how they interact with each other. Schema is a great way to represent said relationships.

For example, a Farm entity may house many Bull entities and one Bull entity can only stay in one Farm entity at a time. This relationship can be written in the following made-up Bull Schema Language:

// Bull Schema Language ( BS Language )
entity Bull {
  id: String
  name: String
}

entity Farm {
  id: String
  name: String
  bulls: Bull[] // An array of Bulls
}

5. Type-safety

Type-safety is important when building medium to large applications. A lot of bugs can be caught at compile time by using languages like Go or TypeScript.

A schema has information about the types and interfaces of entities, requests and responses. There are usually tooling in the ecosystem that could help generate code which conforms to the types declared in the schema.

When code can be generated from the schema, it helps cut down a lot of time and effort in development as developers can focus on the business logic, rather than writing boilerplate-y code to send requests and responses. Some examples of code generators are: GraphQL Code Generator, Swagger Codegen etc.

Examples of schema-driven development technologies

In 2021, SDD can be applied end-to-end when building applications. In this section, let’s look at some examples of SDD technologies.

GraphQL

GraphQL is developed by Facebook initially for their mobile app and has been adopted widely across web apps.

GraphQL's schema is called Schema Definition Language (SDL). Given a schema, GraphQL clients can declare the fields they want to query. This drastically reduces the amount of data that is sent over the internet because the client usually only needs a subset of an entity attributes. Learn more about GraphQL schema.

There are numerous implementations of GraphQL clients and servers. You will be spoiled for choice when you are in this space. You can check out a list of outstanding GraphQL related libraries here.

To build clients and servers, you will need consistent tooling and support. The GraphQL open-source community is truly astonishing in this regard. The Guild is the main powerhouse coordinating a myriad of GraphQL projects.

Remember how I was telling you code can be generated from the schema? @dotansimha takes home the cake for creating a highly versatile plugin-based GraphQL Code Generator that works for various clients and servers.

There is also gqlgen which is a Go-based server and a code generator. If you want the performance of Go, in a qualified GraphQL server, with type-safety and heaps of other functionalities, gqlgen is the one you’d want. 🤩

Here's an example in GraphQL schema:

query {
  bull(id: ID!): Bull
  farm(id: ID!): Farm
}

type Bull {
  id: ID!
  name: String!
}

type Farm {
  id: ID!
  name: String!
  bulls: [Bull!]!
}

Common use case	Browser/Mobile to Server/s communication
Clients	Relay, Apollo Client, urql, and more
Servers	Apollo Server, gqlgen, Nexus, and more

A quick note about GraphQL and SDD

If you are coming from the wonderful world of GraphQL, you might hear the terms schema-first development, Schema Definition Language first ( SDL-first ) and code-first being thrown around a lot. I will use this section to clarify what each term means.

Schema-first development is the process of building software where schema-design is prioritised. This is the same as schema-driven development.
SDL-first is an implementation approach where code is often generated from the schema.
Code-first is an implementation approach where resolvers are created first and the schema is generated from the code.

Both SDL-first and code-first approaches in the context of GraphQL can be schema-driven development if the schema design is the number one focus. It might seem like you are doing SDD by using the SDL-first approach because both processes start from making changes to the schema. However, it is not SDD if the team providing the resource ( i.e. the Backend team in the example in "schema-driven development definition" section ) does not want to discuss the schema changes up front with the team consuming the resource ( i.e. the Frontend team ).

SDL-first, code-first and schema-driven development - By @nikolasburk

For those who are not familiar with GraphQL, you have just stumbled upon something that I'd like to called GraphQL: Civil War. You can do a Google search on SDL-first vs code-first to learn more about the ideology of each approach.

gRPC/Twirp

gRPC and Twirp are Remote Procedure Call frameworks. gRPC is developed by Google and Twirp is developed by Twitch. They both use Protocol Buffers (Protobuf) as the Interface Definition Language. Protobuf is also the serialisation protocol for structured data.

gRPC and Twirp are commonly used in a micro-service architecture. The client is the service making the request and the server is the service returning the response. Services can choose to send data encoded using Protobuf or JSON. Protobuf can be serialised and sent faster than JSON. JSON is useful if you need to debug calls between services.

They both support code generation for clients and servers in many languages: Go, PHP, Ruby, TypeScript, etc. You can see gRPC supported languages here and Twirp supported languages here. This works out great as companies with micro-service architecture may choose different languages for each service.

At 99designs, we have backends in different languages: Go, PHP and Ruby. By using Twirp, we allow teams to code in their preferred language while maintaining a consistent, type-safe and convenient way to send data from one service to another.

Here's an example in Protobuf:

service FarmingApi {
  rpc GetBull(GetBullRequest) returns (Bull);
  rpc GetFarm(GetFarmRequest) returns (Farm);
}

message Bull {
  string id = 1;
  string name = 2;
}

message GetBullRequest {
  string id = 1;
}

message Farm {
  string id = 1;
  string name = 2;
  repeated Bull bulls = 3;
}

message GetFarmRequest {
  string id = 1;
}

Common use case	Server to server communication
Clients	Automatically generated for different languages
Servers	Automatically generated for different languages

Prisma

Prisma is an Object–relational mapping ( ORM ) library which represents database models through a central schema written in Prisma Schema Language ( PSL )

The Prisma schema is effortlessly easy to read. It clearly shows model relationships and field types. On top of that, other important information such as database connection is also stored in the schema. Learn more about Prisma schema.

From the schema, a TypeScript Prisma Client can be generated that can be used in Node.js applications - including Next.js! A Go Prisma Client is also in the works.

On the other side of the schema, Prisma has a sophisticated Prisma Migrate feature to keep the database in sync with the schema. What's great about this is that it works for multiple types of database: PostgreSQL, MySQL and SQLite with support coming for other types!

Personally, I normally don't care for traditional ORMs. I think they are a type of abstraction that requires a lot of work to maintain while hardly making the relationships of entities clearer to developers. I'm a lazy developer so anything that's remotely hard is a turn off.

Prisma is the only ORM that hits the sweet spot for me. It is easy to see the relationships of entities without connecting to the database and click through 200 tables to build up a mental map to connect all the pieces together. The TypeScript Prisma Client is generated with strong type-safety. It's almost like I have two other senior developers watching over me. And when I'm about to do something silly they would be like: "Tsk tsk don't do that, do this!". 😍

Here's an example in Prisma schema:

datasource db {
  provider = "mysql"
  url      = env("PRISMA_DATABASE_URL")
}

generator client {
  provider        = "prisma-client-js"
}

model Bull {
  id     String @id @default(uuid())
  name   String
  farm   Farm
  farmId String
}

model Farm {
  id   String @id @default(uuid())
  name String
  bulls Bull[]
}

Common use case	Server to database communication
Clients	TypeScript Prisma Client, Go Prisma Client
Supported databases	PostgreSQL, MySQL and SQLite ( as of July 2021 )

🙌 Summary

Schema-driven development is becoming more relevant in 2021 with many benefits: better cross-team communications, better API design, independent client and server development, clear entity relationships and type-safety. You can easily build a full-stack application with SDD technologies such as GraphQL, Twirp and Prisma.

I hope you had fun reading this article as much as I did researching and writing it. For similar posts about software engineering, follow me on Twitter @eddeee888. 🤙

I have worked with lazy, impatient & opinionated Software Engineers - here’s what happened

Eddy Nguyen — Wed, 21 Apr 2021 11:22:40 +0000

I have been working in the software industry for almost a decade now. As a result, I was in different teams, some large, some small, and sometimes all by myself. This post will be about my encounters with different types of engineers in the past.

😴 The "lazy" engineer

In one of my jobs, I was working with a particularly opinionated web framework. This was great for my team at the time because it enforced a consistent coding standard. However, this framework also required a lot of boilerplate code.

On one rainy morning, after getting assigned a problem to solve, I immediately created the boilerplate code by moving my fingers across the keyboards with the speed and agility comparable to an Olympic 100-metre runner. Then, I noticed one of the engineers barely touched the keyboard at all.

"How lazy!" - I thought to myself - "He won't have anything to show at due date!"

When the due date came, we presented our work and, lo and behold, his work is completed with the boilerplate code, business logic, the lot.

"Impossible! Did his fingers move so fast my eyes could not see them?" - I mumbled as I desperately tried to make sense of the situation.

I later met the "lazy" engineer at the water cooler to ask about his secret technique. He happily showed me the script he wrote to generate all the boilerplate files that I was furiously typing up. My job became much easier from that day onwards.

I love to work with "lazy" engineers because they want to work smart and finds ways to do their job more efficiently.

😤 The "impatient" engineer

On one of the earliest projects I worked on, we used to use FTP to connect to the server and copy files over to deploy ( I know! I'm a dinosaur! ). This was a manual and tedious process. So when I set up the CI/CD pipeline for said project, I was happy with myself. I was on cloud nine!

"I automated the process, I learned from the 'lazy' engineer!" - young me virtually patting myself on the back.

The process stayed like that for a couple of months. Then, an "impatient" engineer joined the team.

He complained. A lot.
He complained about how slow the build process was.
He complained about how long it took to completely deliver a new build.

The young me was very hurt. How dare he attacked the process that I poured my heart and soul into? How could he be so impatient? Building only took 15 minutes and delivery only took about 45 minutes!

Speed is relative. My "impatient" colleague patiently explained that we could improve the process further. Within an afternoon, he worked his magic and reduced the build time to 3 minutes and delivery time to 10 minutes.

As I took off my rose coloured glasses, I realised I had been foolishly patient with the existing process and stopped looking for ways to improve it.

I love to work with "impatient" engineers because they want to make things better and faster. They also do not mind getting their hands dirty and get the work done sooner than later.

🤨 The "opinionated" engineer

I am the type of person that hates arguments. So I spent a good portion of time earlier in my career not voicing my opinions in meetings, just so I don't have to argue with others.

One day, an "opinionated" engineer gave me feedback saying he agreed with a lot of my opinions and would love to hear them verbally.

"Verbally?" - I asked - "Do I normally communicate my opinions non-verbally?"

"Yes, in your code."

Mind blown. Coding is a way for software engineers to express themselves. Code does not lie, and it's loud and clear what our intentions and opinions are. Through code, we communicate how we think problems should be solved.

The "opinionated" engineer also explained to me that certain ideas are better communicated verbally. Having an opinion is good, voicing it is better and discussing it with others is a great way to materialise ideas into practices, standards and products.

I love to work with "opinionated" engineers because they effectively communicate their opinions and help others have their voice heard.

🧘 The respectful engineer

One common trait of the engineers I worked with is that they are respectful. A respectful engineer treats all their colleagues with respect, regardless of their job title, background, sex, age, etc.

An "opinionated" engineer without respect for others' opinions does not listen to others and only wants to do things their way.

An "impatient" engineer without respect for others' processes probably looks down on them without offering ways to make things better.

A "lazy" engineer without respect for others' time probably does the minimum just to not get fired.

A respectful engineer is who I always strive to be and whom I want to work with.

Summary

I have worked with "lazy", "impatient" and "opinionated" Software Engineers, and to be honest - I love working with them! Do you have your own favourites? Let me know via Twitter @eddeee888. 🤙

Want to be surrounded by amazing people you'd love to work with? Join the Duel of Doves community!

How to deploy Prisma in AWS Lambda with Serverless

Eddy Nguyen — Wed, 24 Feb 2021 20:46:55 +0000

I remember a time when I had to remotely connect to a server using FTP to deliver a web application source code. Those were the "Wild West" days of my career where almost all of the code I wrote were untyped and the manual deployment process could bring down the production website with one wrong keystroke. 😱

Fast forward to the present, I have complete confidence in the code I write in Node.js and TypeScript, strongly typed domain objects modelled with Prisma, and one keystroke is all I need to deploy a feature to AWS Lambdas, all without a dedicated server. What a time to be alive!

In this post, I will talk about one of my favorite setups to go from almost zero to deploying Prisma integrated AWS Lambdas with shared Lambda layers.

🔍 Overview

Prerequisites
How it works
Setting up the project
Setting up Prisma
Creating a Lambda
Creating Lambda layers
Deployment
General notes
Summary

🗝️ Prerequisites

To make full use of this guide, make sure you have the following items set up:

AWS account - free-tier
Serverless account - free-tier
One AWS RDS database
One AWS IAM user

🛠️ How it works

AWS Lambda and Lambda layers

An AWS Lambda is a computing service that allows you to run code without a server. A Lambda is normally small and has a size limit of 50MB. It may seem like a lot but if we build a Lambda with all the dependencies in node_modules, it would easily go over the limit.

A good practice is to keep only the main business logic in the Lambda function. Keeping a Lambda small also means it takes less time to deploy.

All imports should be treated as externals i.e. as if they come from node_modules. These externals usually come from Lambda layers. I like to split my layers into 3 ( you can use up to 5 layers ):

Normal runtime dependencies layer: This layer contains the runtime dependencies installed from package registries. These are typically declared in package.json's dependencies field.
Prisma Client layer: I like to keep Prisma-related dependencies as their own layer because the way they are generated is fairly different from others.
Libs layer: This layer includes custom utilities that can be shared between Lambdas and other apps.

We will explore how to create these layers for Node.js Lambdas later in this post.

Prisma binary

Prisma is an ORM where a TypeScript client is generated based on a schema so consuming apps have type-safety when querying databases. It also creates binaries to run in different environments. We will need to generate different binaries to run in the dev environment and in a Lambda.

Serverless Framework

Serverless Framework is a service that does a lot of heavy lifting when it comes to deploying to AWS.

🏡 Setting up the project

Let's start with a minimal setup. It should have the following structure:



prisma/
scripts/
src/
  -- lambdas/
  -- libs/
package.json

Now, install the following:



$ yarn add -D prisma @prisma/client @types/node ts-node typescript

Create a tsconfig.json that looks like this:



{
  "compilerOptions": {
    "lib": ["es2016", "esnext.asynciterable"],
    "baseUrl": "./src",
    "outDir": "./build",
    "paths": {
      "@libs/*": ["libs/*"]
    },
    "target": "ESNext",
    "module": "commonjs",
    "sourceMap": true,
    "strict": true,
    "skipLibCheck": true,
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true,
    "noUnusedLocals": true,
    "noImplicitAny": true
  },
  "include": ["./src"],
  "exclude": ["node_modules"]
}

The important setting here is @libs/*: ["libs/*"]. This is used so we can use @libs as the alias to import modules from src/libs locally.

🥞 Setting up Prisma

Let's create a minimal Prisma schema:



// ./prisma/schema.prisma

datasource db {
  provider = "mysql"
  url      = env("PRISMA_DATABASE_URL")
}

generator client {
  provider      = "prisma-client-js"
  binaryTargets = [env("PRISMA_BINARY_TARGET")]
}

model User {
  id   Int    @id @default(autoincrement())
  uuid String @unique
}

Notes:

There will be a User table when migrations are run.
env("PRISMA_DATABASE_URL") is used to provide different Prisma URLs depending on the environment.
env("PRISMA_BINARY_TARGET") is used to generate and run different binaries depending on where we run the code.

The default value of binaryTargets is native. This means a Prisma Client will be generated based on the current operating system where the command is run. We will use this in dev. You can learn about the native binary target here.

We want to set this value to rhel-openssl-1.0.x when we generate the Prisma Client to be used by the Lambda. You can find all of the available binaryTargets options here.

Docker is my first choice when it comes to separating environments but for this example, we will use a .env file for simplicity:



// ./.env

# Database URL could be different depending on your setup:
PRISMA_DATABASE_URL=mysql://root:root@localhost/lambda_test?schema=public

# `native` is used in dev so it will generate 
# the binary based on your current OS. 
# `rhel-openssl-1.0.x` should be used for AWS lambda.
PRISMA_BINARY_TARGET=native

Running the following command should create a new database locally:



$ yarn prisma migrate dev

The rest of this guide assumes you have run the same migration on the prod RDS database. For more information, please refer to the Prisma migrate documentation. For this guide's purpose, you can export your dev database structure and content and import them into your RDS database.

Now, we can create a new lib to initialise a Prisma Client:



// ./src/libs/createPrismaClient.ts

import { PrismaClient } from "@prisma/client";

export const createPrismaClient = (): PrismaClient => {
  const prisma = new PrismaClient();
  return prisma;
};

We could directly do this in the Lambda but extracting this into a lib has many advantages:

Consistent logic when creating Prisma Clients e.g. we might want to add a middleware to all Clients.
Can be used by different services e.g. we can use this function in a Node.js web app, Lambdas, or tests.

👷 Creating a Lambda

Let's create a Lambda that when invoked will create a new user with a UUID into the database. Note that Prisma is able to create UUIDs for records natively but this will demonstrate how the runtime dependencies layer works.

First, let's install the packages to generate UUID:



$ yarn add uuid
$ yarn add -D @types/uuid

And here's the code for the Lambda:



// ./src/lambdas/insertUser/handler.ts

import { createPrismaClient } from "@libs/createPrismaClient";
import { v4 as uuidv4 } from "uuid";

const handler = async (): Promise<void> => {
  const prisma = createPrismaClient();

  try {
    await prisma.user.create({
      data: { uuid: uuidv4() },
    });
  } catch (e) {
    console.error(e);
  }

  prisma.$disconnect();
};

export default handler;

Note that we should always run prisma.$disconnect(); at the end of every Lambda to ensure we do not hold connections to the database.

We will use TypeScript to compile this code:



$ yarn tsc

This will be compiled into the build folder:



// ./build/lambdas/insertUser/handler.js

"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
const prismaClient_1 = require("@libs/createPrismaClient");
const uuid_1 = require("uuid");
const handler = async () => {
    const prisma = prismaClient_1.createPrismaClient();
    try {
        await prisma.user.create({
            data: { uuid: uuid_1.v4() },
        });
    }
    catch (e) {
        console.error(e);
    }
    prisma.$disconnect();
};
exports.default = handler;
//# sourceMappingURL=handler.js.map

This file can be uploaded to AWS as a Lambda function. However, it's importing @libs/createPrismaClient and uuid from node_modules. We must create the Lambda layers that hold these dependencies.

🧅 Creating Lambda layers

As mentioned above, there are 3 Lambda layers we want to create:

Runtime dependencies layer
Prisma layer
Libs layer

A Lambda layer is a zip file that looks like this:



layer.zip
  -- nodejs
     -- node_modules
        -- lib1
        -- lib2
        ...

You can read more about Lambda layers here. We will need to create one such file for each layer:



// Runtime dependencies layer

lambda-layers-node_modules.zip
  -- nodejs
     -- node_modules
        -- uuid



// Prisma layer

lambda-layers-prisma-client.zip
  -- nodejs
     -- node_modules
        -- .prisma
        -- @prisma



// Libs layer

lambda-layers-libs.zip
  -- nodejs
     -- node_modules
        -- @libs
           -- createPrismaClient.js

Let's create 3 separate scripts that can prepare these layers for us.

Runtime dependencies layer

This script should be in ./scripts/ci/prepare-node-modules-lambda-layer.sh



#!/bin/bash
function prepare_node_modules_lambda_layer() {
  echo "Cleaning up workspace ..."
  rm -rf lambda-layers-node_modules

  echo "Creating layer ..."
  mkdir -p lambda-layers-node_modules/nodejs

  echo "Prepare server node_modules lambda layer ..."
  cp -r node_modules lambda-layers-node_modules/nodejs

  echo "Compressing ..."
  pushd lambda-layers-node_modules && tar -zcf /tmp/nodejs.tar.gz . && mv /tmp/nodejs.tar.gz ./nodejs.tar.gz

  echo "Remove unzipped files ..."
  rm -rf nodejs

  echo "Stats:"
  ls -lh nodejs.tar.gz

  popd
}
prepare_node_modules_lambda_layer

Note: this script creates a zip file that contains everything in node_modules so if you generated a Prisma Client here, it will be included in this layer. This script is intended to be used in CI/CD in a step where we only install production package.json dependencies.

Prisma layer

This script should be in ./scripts/ci/prepare-prisma-client-lambda-layer.sh



#!/bin/bash
function prepare_prisma_client_lambda_layer() {
  echo "Cleaning up workspace ..."
  rm -rf lambda-layers-prisma-client

  echo "Creating layer ..."
  mkdir -p lambda-layers-prisma-client/nodejs/node_modules/.prisma
  mkdir -p lambda-layers-prisma-client/nodejs/node_modules/@prisma

  echo "Prepare Prisma Client lambda layer ..."
  cp -r node_modules/.prisma/client lambda-layers-prisma-client/nodejs/node_modules/.prisma
  cp -r node_modules/@prisma lambda-layers-prisma-client/nodejs/node_modules

  echo "Remove Prisma CLI..."
  rm -rf lambda-layers-prisma-client/nodejs/node_modules/@prisma/cli

  echo "Compressing ..."
  pushd lambda-layers-prisma-client && tar -zcf /tmp/nodejs.tar.gz . && mv /tmp/nodejs.tar.gz ./nodejs.tar.gz

  echo "Remove unzipped files ..."
  rm -rf nodejs

  echo "Stats:"
  ls -lh nodejs.tar.gz

  popd
}
prepare_prisma_client_lambda_layer

When this script is run, it will create a Lambda layer with the .prisma and @prisma directories:

@prisma is where the generators and wiring happen.
.prisma contains the generated TypeScript interfaces.

It also removes @prisma/cli to keep the layer smaller since we won't be running commands in the Lambda. In later versions of Prisma ( >=2.16 ), this package is no longer needed so you can omit this line.

This should be run in CI/CD after the Prisma packages have been installed and the Prisma Client generated.

Libs layer

This script should be in ./scripts/ci/prepare-libs-lambda-layer.sh



#!/bin/bash
function prepare_libs_lambda_layer() {
  echo "Cleaning up ..."
  rm -rf lambda-layers-libs

  echo "Creating layer ..."
  mkdir -p lambda-layers-libs/nodejs/node_modules/@libs
  mv build/libs build/@libs

  echo "Prepare libs lambda layer ..."
  cp -r build/@libs lambda-layers-libs/nodejs/node_modules

  echo "Compressing ..."
  pushd lambda-layers-libs && tar -zcf /tmp/nodejs.tar.gz . && mv /tmp/nodejs.tar.gz ./nodejs.tar.gz

  echo "Remove unzipped files ..."
  rm -rf nodejs

  echo "Stats:"
  ls -lh nodejs.tar.gz

  popd
}
prepare_libs_lambda_layer

This script should be run after we have compiled the libs using yarn tsc. Note that this layer will be built as build/libs but we rename it into node_modules/@libs to match the module import path that we have in the Lambda.

☁️ Deployment

At this point, we could manually upload the Lambda and its layers' zips to AWS but it would take forever to do it this way. I got you fam, don't worry. 😉

We'll set up some CI/CD goodness in this section.

Serverless

This is where Serverless Framework comes in. We can deploy everything with one command. Create a serverless yml like this:



# ./serverless.yml

service: prisma-aws-lambda-deployment

provider:
  name: aws
  runtime: nodejs12.x
  stage: prod
  region: ${env:AWS_REGION}
  # vpc:
  #   securityGroupIds:
  #     -  # FILLME
  #   subnetIds:
  #     -  # FILLME
  #     -  # FILLME

layers:
  TopicPrismaAwsNodeModules:
    path: lambda-layers-node_modules
  TopicPrismaAwsLibs:
    path: lambda-layers-libs
  TopicPrismaAwsPrismaClient:
    path: lambda-layers-prisma-client

functions:
  insertUserCron:
    handler: insertUser/handler.default
    memorySize: 512
    timeout: 290 # 4 minutes 50 seconds
    events:
      - schedule: rate(5 minutes)
    environment:
      NODE_ENV: production
      PRISMA_DATABASE_URL: ${env:PRISMA_DATABASE_URL}
      PRISMA_BINARY_TARGET: ${env:PRISMA_BINARY_TARGET}
    layers:
      - { Ref: TopicPrismaAwsNodeModulesLambdaLayer }
      - { Ref: TopicPrismaAwsLibsLambdaLayer }
      - { Ref: TopicPrismaAwsPrismaClientLambdaLayer }

In this file, we basically deploy the insertUser Lambda with all the layers attached as a cron and run it every 5 mins. Remember to set your VPC settings so your Lambda can send requests to the RDS instance!

Other ways to invoke a Lambda function:

Send a request to the Lambda URL
Execute the AWS CLI invoke command
Use the test feature in the AWS Lambda UI

Also, this file is intended to be run in the build/ directory. You will see an error if you try to run this from the project root. Don't worry, it'll make sense in CI/CD.

Github action

In the previous sections, I have been alluding to deploying in CI/CD. We will be using GitHub action but you can do this in any other providers:

Build runtime dependencies layer
Build Prisma layer
Build libs layer
Build Lambda(s)
Once all the previous steps are done, download the artifacts and deploy

Here's the GitHub action file for it:



# ./.github/workflows/deploy-lambdas.yml

name: Deploy lambdas
on:
  push:
    branches:
      - "master"

jobs:
  build-node_modules-lambda-layer:
    name: Bld. node_modules layer
    runs-on: ubuntu-18.04
    steps:
      - name: Check out repository
        uses: actions/checkout@v2

      - name: Set up Node.js 12.x
        uses: actions/setup-node@v1
        with:
          node-version: 12.x

      # Only install PROD packages i.e. no `@types/*` packages or dev-related packages
      - name: Install PROD packages
        run: yarn --production

      - name: Prepare lambda layer
        run: ./scripts/ci/prepare-node-modules-lambda-layer.sh

      - uses: actions/upload-artifact@v2
        with:
          name: lambda-layers-node_modules
          path: ./lambda-layers-node_modules

  build-prisma-client-lambda-layer:
    name: Bld. @prisma/client layer
    runs-on: ubuntu-18.04
    steps:
      - name: Check out repository
        uses: actions/checkout@v2

      - name: Set up Node.js 12.x
        uses: actions/setup-node@v1
        with:
          node-version: 12.x

      - name: Install ALL packages
        run: yarn --frozen-lockfile

      # Generate Prisma Client and binary that can run in a lambda environment
      - name: Prepare prisma client
        run: yarn PRISMA_BINARY_TARGET=rhel-openssl-1.0.x prisma generate

      - name: Prepare "@prisma/client" lambda layer
        run: ./scripts/ci/prepare-prisma-client-lambda-layer.sh

      - uses: actions/upload-artifact@v2
        with:
          name: lambda-layers-prisma-client
          path: ./lambda-layers-prisma-client

  build-libs-lambda-layers:
    name: Bld. @libs layer
    runs-on: ubuntu-18.04
    steps:
      - name: Check out repository
        uses: actions/checkout@v2

      - name: Set up Node.js 12.x
        uses: actions/setup-node@v1
        with:
          node-version: 12.x

      - name: Install ALL packages
        run: yarn --frozen-lockfile

      - name: Prepare prisma client
        run: yarn PRISMA_BINARY_TARGET=rhel-openssl-1.0.x prisma generate

      - name: Build assets
        run: yarn tsc

      - name: Prepare "@libs/*"" lambda layer
        run: ./scripts/ci/prepare-libs-lambda-layer.sh

      - uses: actions/upload-artifact@v2
        with:
          name: lambda-layers-libs
          path: ./lambda-layers-libs

  build-lambdas:
    name: Build lambdas
    runs-on: ubuntu-18.04
    steps:
      - name: Check out repository
        uses: actions/checkout@v2

      - name: Set up Node.js 12.x
        uses: actions/setup-node@v1
        with:
          node-version: 12.x

      - name: Install ALL packages
        run: yarn --frozen-lockfile

      - name: Prepare prisma client
        run: yarn PRISMA_BINARY_TARGET=rhel-openssl-1.0.x prisma generate

      - name: Build lambdas
        run: yarn tsc

      - uses: actions/upload-artifact@v2
        with:
          name: build-lambdas
          path: ./build/lambdas

  deploy-lambdas:
    name: Deploy lambdas
    needs:
      [
        build-node_modules-lambda-layer,
        build-prisma-client-lambda-layer,
        build-libs-lambda-layers,
        build-lambdas,
      ]
    runs-on: ubuntu-18.04
    steps:
      - name: Check out repository
        uses: actions/checkout@v2

      - name: Set up Node.js 12.x
        uses: actions/setup-node@v1
        with:
          node-version: 12.x

      - uses: actions/download-artifact@v2
        with:
          name: build-lambdas
          path: ./build/lambdas

      - uses: actions/download-artifact@v2
        with:
          name: lambda-layers-node_modules
          path: ./build/lambdas/lambda-layers-node_modules

      - uses: actions/download-artifact@v2
        with:
          name: lambda-layers-libs
          path: ./build/lambdas/lambda-layers-libs

      - uses: actions/download-artifact@v2
        with:
          name: lambda-layers-prisma-client
          path: ./build/lambdas/lambda-layers-prisma-client

      - name: Unzip layers
        run: |
          tar -C ./build/lambdas/lambda-layers-node_modules -xf ./build/lambdas/lambda-layers-node_modules/nodejs.tar.gz
          rm -rf ./build/lambdas/lambda-layers-node_modules/nodejs.tar.gz
          tar -C ./build/lambdas/lambda-layers-libs -xf ./build/lambdas/lambda-layers-libs/nodejs.tar.gz
          rm -rf ./build/lambdas/lambda-layers-libs/nodejs.tar.gz
          tar -C ./build/lambdas/lambda-layers-prisma-client -xf ./build/lambdas/lambda-layers-prisma-client/nodejs.tar.gz
          rm -rf ./build/lambdas/lambda-layers-prisma-client/nodejs.tar.gz

      - name: Move serverless.yml
        run: mv serverless.yml ./build/lambdas/serverless.yml

      - name: Deploy lambdas and layers
        uses: aaronpanch/action-serverless@master
        with:
          args: deploy --debug
        env:
          SERVICE_ROOT: ./build/lambdas
          SERVERLESS_ACCESS_KEY: ${{ secrets.SERVERLESS_ACCESS_KEY }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID_CI }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY_CI }}
          AWS_REGION: ${{ secrets.AWS_REGION }}
          PRISMA_DATABASE_URL: ${{ secrets.PRISMA_DATABASE_URL }}
          PRISMA_BINARY_TARGET: rhel-openssl-1.0.x

Notes:

rhel-openssl-1.0.x is the AWS Lambda binary target for Prisma Client.
serverless.yml is moved into ./build/lambdas/serverless.yml to make sure it only compresses the built Lambda folder.
Serverless zips layers automatically so we need to unzip the layers before running the serverless deploy command.
secrets.* values can be set using GitHub action secrets
SERVERLESS_ACCESS_KEY can be created through the Serverless dashboard
AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY can be created when you create an IAM user in AWS

You will see something like this in the "Actions" tab of your Github repo if everything went according to plan:

And... that's it! If the gods of AWS are on your side, you'll start seeing new users being inserted into your User table with Prisma! 🎉

📚 General notes

Make sure to secure your VPC, database, IAM, etc. When I wrote this post, I made everything public to make it easy to test. My database was hacked with a ransom note saying I should pay 1 Bitcoin to recover it! Who would have thought root and password123 as credentials is an open invitation to evil hackers?! 🤭
You can find your Lambda logs in AWS CloudWatch of your chosen region
If you want to add a new Lambda using this setup, you can create the Lambda in the src/lambdas directory and update serverless.yml
Always take care of your keys and secrets and do not store them in plain text in your app or config files.
Always set ?connection_limit=1 to the Prisma database URL if you are planning to use it in a Lambda to avoid exhausting the connection pool. Read more about the recommended connection limit

✌️ Summary

I hope you enjoyed reading this post! If you have tips & tricks regarding any of the topics discussed or general feedback, don't be shy to send me a holler at @eddeee888

The full repo with working CI/CD can be found here: https://github.com/eddeee888/topic-prisma-aws-lambda-deployment

How to write a Bash shortcut script to enhance your terminal experience

Eddy Nguyen — Tue, 26 Jan 2021 21:56:01 +0000

I love shortcuts. If I had to pick the most impactful invention in human history, it would be Ctrl+C and Ctrl+V. We would still be living in caves if we didn't find ways to make our lives easier.

Living in caves is what I felt about using the Terminal for a long time. Everything was so hard but felt so... magical at the same time. One day, I got tired of repeatedly bashing my keyboard in, putting down my club, learned Bash, and made my first script.

Now, dear reader, gather around the fire (figuratively, of course), and let me tell you about my approach to writing a Bash script for shortcuts that got me from the stone age to 2021.

(*) The commands you see in this post are executed on a Mac. If you are using Windows or Linux, you may have to adjust the commands slightly.

(**) I'll stop mentioning cavemen.

🧐 What the Bash?

Bash is a command language that allows users to type in text commands to tell the computer to do specific actions. Mac and Linux usually come with Bash installed.

If you go to Application > Utilities on a Mac, you will see an app called Terminal

Open the Terminal, and you will see a black window with white texts - kinda like what the hackers use on TV. This is the command-line interface (CLI) that allows you to run text commands (including Bash commands and scripts).

🛠 Starting with the basics

You may have seen something like this before:



# This is an example, do not execute this command.
$ yarn install

In this case, yarn is the command name, and install is the first parameter. We will want our Bash command to work in a similar way. For us, let's say if we execute the following command, it will open Netflix in the browser:



# Hell yea - a shortcut to watch Netflix. I'm down!
$ ss tv nf

(*) Executing a command means we type the command into the terminal and press return/enter.

Let's create a new file with the following content:



#/bin/bash

function ss() {
  echo "Hello World"
  return 0
}

Notes

This file is called a script.
The name of the function is ss because we want the command name to be short ( it's a shortcut script after all! ).
You can save this anywhere but I normally save all my custom scripts in ~/bin. For our script, we can save it as ~/bin/ss.sh.
#/bin/bash is needed at the top so your operating system knows how to run this script i.e. using Bash.
If this function is executed, it will print "Hello World" in the terminal.
return 0 is how we tell the terminal that the command has run successfully.

🚗 Executing the script

Normally, if you try to execute the script by typing in the path to the script and hit return/enter, it will say that you don't have permission to do so:

😤 Hmm, I'm a grown-ass man. No one or nothing can "permission denied" me like this. Let's find a way to fix this.

At this point, we could mess around with script permissions but I prefer sourcing the script for many reasons:

No need to update the script permission. 😌
The function name i.e. ss can be used as the command name. 🥰
The script works regardless of its location. 😍

You can add the following line at the end of your ~/.zshrc or ~/.bashrc file:



source ~/bin/ss.sh

Doing this will load the script into the terminals that you open in the future so you can execute the ss command directly.

Now, open a new terminal, type ss into the command prompt, and hit return/enter, you will see something like this:

✂️ Creating shortcuts

Now we get to the fun part! We want to check the parameters that come after the command name and assign them to an action we want to take. For example, we can update the script to something like this:



#/bin/bash

function ss() {
  case $1 in
    "open")
      vi ~/bin/ss.sh
      return 0
    ;;
    "tv")
      case $2 in
        "nf")
          open https://www.netflix.com/
          echo "Opening Netflix"
          return 0
        ;;
        "az")
          open https://www.primevideo.com/
          echo "Opening Amazon Prime"
          return 0
        ;;
        *)
          open /System/Applications/TV.app
          echo "Opening Apple TV"
          return 0
        ;;
      esac

    ;;
  esac

  echo "\nERROR - Invalid command\n"
  return 1
}

There's quite a bit to take in here but don't worry, here's how it works:

There are multiple Bash case statements, each starting with case and ending with esac.
$1 and $2 are the first parameter and second parameter (that comes after the script name) respectively.
When executed, this function will check if the first parameter is open or tv. If it is tv, it would go into its own case statement, step through the second parameter and check if it is nf or az. *) is to catch all other options.
If a correct combination is found, it will execute the code inside the respective block.
If no combination is found, return 1 is executed to tell the terminal that an error occurred.

In other words:

If the first parameter is open, we will use vi command to make changes to the shortcut file. Remember to start a new terminal after making changes to load the new script!
If the first parameter is tv and the second parameter is nf, it will open Nextflix in the default browser.
If the first parameter is tv and the second parameter is az, it will open Amazon Prime in the default browser.
If the first parameter is tv and the second parameter is anything ( including no second parameter ), it will open the Apple TV app.

Try opening a new Bash script and executing the following command and see if it's working:



$ ss tv nf

If it opens Netflix in a new browser, congratulations! You have created your first Bash shortcut script. 🎉

The above commands are pretty simple but you can add more commands in each block. In another script that I use, I start 5-6 web services locally on my laptop with one command. That is, like, 10000% productivity increase.. or something. Yea. 😎

If you enter an invalid command, you will see an error message:

🍕 BONUS - Add pizzazz to your script

Error messages just have bad vibes. If I fail to type a command, I want encouragement from my computer, not getting yelled at! If you feel the same way, update the script to the following:



#/bin/bash

function sayStuff(){
  stuff_to_say=("You may have a fat finger but you also have a freaking fat heart." "Take a deep breath and try again man. You got this!" "You will stop sucking one day." "Your command is invalid. But you are valid.")
  length=${#stuff_to_say[@]}
  random=$$$(date +%s)

  random_stuff=${stuff_to_say[$random % $length + 1]}

  echo "\nERROR - Invalid command"
  echo "$random_stuff\n"
}

function ss() {
  case $1 in
    "tv")
      case $2 in
        "nf")
          open https://www.netflix.com/
          return 0
        ;;
        "az")
          open https://www.primevideo.com/
          return 0
        ;;
        *)
          open /System/Applications/TV.app
          return 0
        ;;
      esac

    ;;
    "open")
      vi ~/bin/ss.sh
      return 0
    ;;
  esac

  sayStuff
  return 1
}

sayStuff is a function that picks one of the few given quotes to cheer you up every time you enter an invalid shortcut combination. For me, some aggressive positivity is exactly what I want to hear during trying times:

"You may have a fat finger but you also have a freaking fat heart."
"Take a deep breath and try again man. You got this!"
"You will stop sucking one day."
"Your command is invalid. But you are valid."

🙏 Summary

Writing a Bash shortcut script is a fun way to get used to scripting as well as saving yourself from typing too much ( everyone hates that right? ). Do you have other ways that make your daily workflows easier? I would love to hear from you. 🙂

The full code example is available on GitHub: https://github.com/eddeee888/topic-simple-bash

How to write tests for Prisma with Docker and Jest

Eddy Nguyen — Mon, 11 Jan 2021 00:55:35 +0000

I have been using Prisma for a couple of my projects and I'm absolutely loving it. With generated types and easy to use API, I can effortlessly build features without having to worry about the data shapes anymore.

🌴 Trouble in ( testing ) paradise

One small problem with Prisma is that it is not always clear how to write unit and functional tests. .env file is used by default but it takes a bit of work to get .env.test working as mentioned in this issue.

Docker is great to separate development and testing environment. With Docker, .env files are not needed because environment variables can be set when the containers are created. Since I was using Docker for development already, setting up a testing environment was very easy.

In this post, I will talk about my approach to writing tests for Prisma-integrated applications.

⚡ TLDR;

Create and run tests in Docker containers.
Set up and reset the database before and after tests.
For unit tests, create a Prisma client and disconnect after each test.
For functional tests, start a server and close it after each test.
Full example with working CI here: https://github.com/eddeee888/topic-prisma-testing

💻 Setup

NPM Packages

First, let's install the npm packages that we need. Run this in your host terminal:

$ yarn -D @prisma/cli @prisma/client @types/jest jest node-fetch ts-jest ts-node typescript

Prisma schema

Let's get started with a very simple Prisma schema:

// ./src/prisma/schema.prisma
datasource db {
  provider = "mysql"
  url      = env("PRISMA_DATABASE_URL")
}

generator client {
  provider = "prisma-client-js"
}

model User {
  id    Int    @id @default(autoincrement())
  email String @unique
}

Notes:

We use env("PRISMA_DATABASE_URL") for the url because we will give it different values based on whether we are in a testing or development environment.
A user's email is also unique so Prisma should throw an error if we try to add two users with the same email

App Docker container

We will need a Node container to run migrations and tests in. We do this in containers so the environment is consistent for everyone - no more "but it works on my machine" problems!

Create a Dockerfile to store what we need:

# ./Dockerfile
FROM node:12.18.0-alpine3.11 AS base
WORKDIR /usr/src/app
RUN apk update \ 
  && apk add bash \
  && rm -rf /var/cache/apk/*
COPY . . 
RUN yarn install --frozen-lockfile
RUN yarn prisma generate

docker-compose

docker-compose is a tool to manage multi-container apps. In our case, we will need something like this:

# ./docker-compose.test.yml
version: "3.7"

services:
  server:
    build:
      context: "."
      target: base
    environment:
      SERVER_DATABASE_NAME: test_db
      PRISMA_DATABASE_URL: mysql://root:root@database:3306/test_db?schema=public
    ports:
      - 9999:80
    volumes:
      - ./src:/usr/src/app/src
      - ./package.json:/usr/src/app/package.json
    networks:
      - test_vm
    depends_on:
      - database

  database:
    image: mysql:5.7
    restart: always
    environment:
      - MYSQL_ROOT_PASSWORD=root
      - MYSQL_PORT=3306
    volumes:
      - database:/var/lib/mysql
    expose:
      - 3307
    ports:
      - 3307:3306
    networks:
      - test_vm
volumes:
  database:
networks:
  test_vm:

The file above is quite long but don't fret! The most important things to note here are:

There are 2 services: server and database
server which is a server with node v12.18.0 ( and a few other things installed as stated in the Dockerfile above )
server has PRISMA_DATABASE_URL set, which means it can run Prisma commands against the database.
database is a mysql database ( which matches Prisma schema ).

🧑‍🍳 Prepare the testing environment

Let's start by building our Node image. We will use this image to manage migrations for the test database.

Run the following command on your host terminal:

$ docker-compose -f docker-compose.test.yml build --no-cache

You can check if your image has been built successfully by running the docker images command. It will look something like this:

Now, let's create a new migration:

$ docker-compose -f docker-compose.test.yml run --rm server yarn prisma migrate save --experimental --name add-user-model

Then, we apply the migration:

$ docker-compose -f docker-compose.test.yml run --rm server yarn prisma migrate up --experimental --create-db --auto-approve

🧪 Unit tests

Writing unit tests

We can't run tests unless we write a function to test first 😛. Let's add a simple function:

// ./src/actions/createUserAction.ts
import { PrismaClient, User } from "@prisma/client";
export interface CreateUserActionParams {
  prisma: PrismaClient;
  email: string;
}
const createUserAction = async ({
  prisma,
  email,
}: CreateUserActionParams): Promise<User> => {
  return await prisma.user.create({ data: { email } });
};
export default createUserAction;

This is a very contrived example that just calls Prisma functions underneath. The thing to note here is that a Prisma client is injected from the callsite to make it easy to test.

We will need to install the following packages to generate unique emails for our tests:

$ yarn add -D uuid @types/uuid

And here's our test file:

// ./src/actions/createUserAction.test.ts
import createUserAction from "./createUserAction";
import { v4 as uuidv4 } from "uuid";
import { PrismaClient } from "@prisma/client";

const prisma = new PrismaClient();
afterAll(async (done) => {
  await prisma.$disconnect();
  done();
});

describe("createUserAction() - unit", () => {
  it("creates new user correctly", async () => {
    const email = `${uuidv4()}@test.com`;

    await createUserAction({ prisma, email });

    const [savedUser] = await prisma.user.findMany({
      where: { email },
      take: 1,
    });

    expect(savedUser.email).toBe(email);
  });

  it("fails if tries to create records with the same user twice", async () => {
    const email = `${uuidv4()}@test.com`;

    await createUserAction({ prisma, email });

    const [savedUser] = await prisma.user.findMany({
      where: { email },
      take: 1,
    });

    expect(savedUser.email).toBe(email);

    await expect(() => createUserAction({ prisma, email })).rejects.toThrow(
      "Unique constraint failed on the constraint: `email_unique`"
    );
  });
});

Ok, let's inspect the important parts of this file:

const prisma = new PrismaClient();
afterAll(async (done) => {
  await prisma.$disconnect();
  done();
});

Here, we create a new client for this test file ( and other files too ). This is fairly inexpensive so we can run it for every file. After all of the tests in this file, we will disconnect the Prisma client from the database to avoid hogging connections.

  it("creates new user correctly", async () => {
    const email = `${uuidv4()}@test.com`;

    await createUserAction({ prisma, email });

    const [savedUser] = await prisma.user.findMany({
      where: { email },
      take: 1,
    });

    expect(savedUser.email).toBe(email);
  });

In this test, we create a user with a unique email and make sure we can query it.

  it("fails if tries to create records with the same user twice", async () => {
    const email = `${uuidv4()}@test.com`;

    await createUserAction({ prisma, email });

    const [savedUser] = await prisma.user.findMany({
      where: { email },
      take: 1,
    });

    expect(savedUser.email).toBe(email);

    await expect(() => createUserAction({ prisma, email })).rejects.toThrow(
      "Unique constraint failed on the constraint: `email_unique`"
    );
  });

In this above test, we test that if we try to create a user with the same email, it will throw an error the second time!

Running tests

Finally, here's the moment we are all waiting for. Let's run the tests!

$ docker-compose -f docker-compose.test.yml run --rm server yarn jest -i

Note that -i flag is used to make sure we run tests one by one to avoid race conditions in tests.

Sometimes, our tests may fail because the database container is not ready before tests are run. It is highly recommended to be using something like wait-for-it.sh. We can copy the file into ./scripts/wait-for-it.sh. Then, we can run the following instead of the previous command:

$ docker-compose -f docker-compose.test.yml run --rm server ./scripts/wait-for-it.sh database:3306 -- yarn jest -i

🚗 Functional tests

Functional tests are specifications of how a system works. For example, if our app receives a request at a certain URL, a new user is created.

Let's create an app server. First, we need to install a few packages:

$ yarn add express
$ yarn add -D @types/express node-fetch @types/node-fetch

Then, we can create a server. Note that we don't start the server yet.

// ./src/createServer.ts

import express, { Express } from "express";
import { PrismaClient } from "@prisma/client";
import createUserAction from "./actions/createUserAction";

export interface CreateServerParams {
  prisma: PrismaClient;
}

const createServer = ({ prisma }: CreateServerParams): Express => {
  const server = express();

  server.get("/new-user/:email", async (req, res) => {
    const { email } = req.params;

    try {
      await createUserAction({ prisma, email });
      return res.status(200).send("ok");
    } catch (e) {
      res.status(403).send(`Cannot create new user for email: ${email}`);
    }
  });

  return server;
};

export default createServer;

In here, our createServer function also takes a Prisma client to make it easier to test. If a GET request is sent to /new-user/:email ( e.g. http://website.com/new-user/cool.personl@zmail.com ), then we will call createUserAction to create a new user and send back 200 if is successful or 403 if encountered errors.

NOTE: Please DO NOT - I REPEAT, DO NOT - have a URL that can create new users on GET requests without input validation/authentication/authorization, etc. or you will get an army of angry pelicans delivering spams to your app! ☠️

Writing functional tests

Now, we can start a new server for our tests to run against:

// ./src/actions/createUserAction.functional.test.ts
import { v4 as uuidv4 } from "uuid";
import fetch from "node-fetch";
import { PrismaClient } from "@prisma/client";
import createServer from "./createServer";

const prisma = new PrismaClient();
const server = createServer({ prisma });
const internalConfig: any = {};
beforeAll(async (done) => {
  const instance = await server.listen({ port: 80 });
  internalConfig.server = instance;
  done();
});
afterAll(async (done) => {
 internalConfig.server.close();
 await prisma.$disconnect();
 done();
});

describe("createUserAction() - functional", () => {
  it("creates new user correctly", async () => {
    const email = `${uuidv4()}@test.com`;

    const res = await fetch(`http://localhost/new-user/${email}`);

    expect(res.ok).toBe(true);
  });

  it("fails if tries to create records with the same user twice", async () => {
    const email = `${uuidv4()}@test.com`;

    await prisma.user.create({ data: { email } });

    const res = await fetch(`http://localhost/new-user/${email}`);

    expect(res.ok).toBe(false);
  });
});

Again, let's break this down:

const prisma = new PrismaClient();
const server = createServer({ prisma });
const internalConfig: any = {};
beforeAll(async (done) => {
  const instance = await server.listen({ port: 80 });
  internalConfig.server = instance;
  done();
});
afterAll(async (done) => {
 internalConfig.server.close();
 await prisma.$disconnect();
 done();
});

This snippet of code creates a new Prisma client for the server. Before the tests in this file start, start the server at port 80. After the tests in this file end, stop the server and disconnect Prisma client.

  it("creates new user correctly", async () => {
    const email = `${uuidv4()}@test.com`;

    const res = await fetch(`http://localhost/new-user/${email}`);

    expect(res.ok).toBe(true);
  });

In the above test, we send a request to our server, and if it is a new user, then it's all g!

  it("fails if tries to create records with the same user twice", async () => {
    const email = `${uuidv4()}@test.com`;

    await prisma.user.create({ data: { email } });

    const res = await fetch(`http://localhost/new-user/${email}`);

    expect(res.ok).toBe(false);
  });

In the second test, we are trying to create a user who already exists, which causes the response to fail. Perfect! 🕺

Then, we can run the same test command again:

$ docker-compose -f docker-compose.test.yml run --rm server ./scripts/wait-for-it.sh database:3306 -- yarn jest -i

👋 Summary

Testing Prisma is not simple because it is hard to separate an environment for testing. Using Docker solves this issue for me. Do you know of a different way to test Prisma? I would love to hear from you 😊

For the full development and test environment examples, including CI ( GitHub actions ), check out this repository: https://github.com/eddeee888/topic-prisma-testing.