Forem: Timo Schinkel

Native TypeScript with Node

Timo Schinkel — Mon, 09 Mar 2026 07:40:18 +0000

Starting with Node 22.18.0 it is possible to run TypeScript natively from the Node interpreter. This feels like the best of both worlds - the benefits of type inference in your IDE without need to first transpile. I have been using this feature for the small scripts that I use during development and during inspections. Think of scripts to generate or validate a keypair, or a script to build a configuration file for local development. Needing to run tsc is a bit of a nuisance for these scripts, and therefore solutions like ts-node and tsx were introduced. But now this is built-in in Node, and that means that we have one dependency less in our package.json.

But before you start using this in your codebases there are a few caveats that you should probably be aware of.

Type safety

First a word of caution; Node does not actually perform the type checks that TypeScript performs. It merely strips the type information from the TypeScript code. This does mean an added, although barely noticable, latency to the execution of your scripts. More importantly it means that the type safety is not guaranteed. This is actually the same thing that happens with bundlers like esbuild. That's why you should always also run tsc --noEmit on your TypeScript code when you use a bundler or when running it natively with Node.

Restricted syntax

But more importantly it means that you cannot use any syntax that is TypeScript only. Two example of this that I personally ran into were enums and parameter properties:

export enum MyEnum {
  OptionA = 'option-a',
  OptionB = 'option-b',
};

export class MyClass {
  public constructor(
    public readonly name: string,
    private readonly value: number,
  ) {
  }
}

There is a way to let TypeScript block this via the tsconfig.json:

{
  "compilerOptions": {
     "erasableSyntaxOnly": true
  }
}

Keep in mind that - unlike package.json - tsconfig.json files are not hierarchical read by tsc. I personally tend to use the native TypeScript feature for specific scripts, which are located outside the TypeScript files meant for production. Contrary to linters like eslint, oxlint and biome TypeScript does not have a way to specify compiler options per folder. So you have three options if you want to use this option:

Enable this flag in your root tsconfig.json, with the side effect that you are limited in your TypeScript syntax. But this approach will give you the option to validate the typing of your scripts.
Create a separate tsconfig.json inside the folder where the scripts are located and run a second tsc --project scripts/tsconfig.json --noEmit. By using the $extends option from TypeScript you can still keep most of your configuration centralized.
Don't explicitly check for erasable syntax.

CommonJS vs ESM

Already since version 12 NodeJS has had support for ESM - ECMAScript modules -, while also keeping support for CommonJS modules. We are now at version 25 of NodeJS and still there are libraries that are only available as CommonJS modules, and some libraries have made the step towards ESM. But if you want to use native TypeScript this becomes an even more concious decision, because we cannot rely on tsc to convert ESM syntax to CommonJS syntax - or vise versa - based on the compiler options in tsconfig.json.

// CommonJS way of including files and modules
const { method } = require('./method');
const module = require('module');

// ESM way of including files and modules
import { method } from './method.js';
import module from 'module';

ESM has some advantages over CommonJS, such as a better capability of tree shaking, but it also has a downside; you will need to specify the extension of the file for relative imports. If you're using tsc to build your production code you will end up with multiple .js files, and therefore you will need to explicitly use the .js extension in your relative imports. Even in your TypeScript code, because tsc does not rewrite those imports (yet?).

How NodeJS decides if a file is to be interpreted as ESM or as CommonJS is well-documented. In a nutshell; It checks the extension, then the nearest package.json with a type property, then the --input-type flag, and finally it guesses.

If you use a bundler then this usually is corrected by the bundler, but this is not the case for native TypeScript. So, you better make an explicit choice and go for one or the other.

NB This really only is an issue if you want to use code from another file in your codebase. If you only use modules from NodeJS itself then this is simple choice of syntax, and I would recommend to steer the module resolution by using either the extension .mts or .cts. When importing from an (P)NPM package you are dependent of the package maintainers as they how they offer their library; as ESM, as CJS or as both.

ESM

The moment you want to introduce relative imports into your script I would recommend using ESM. Not only is it more future-proof, and the syntax is used in the code examples on the TypeScript website, but I have yet to find a nice way to make relative imports work using CommonJS. Since the files that are used via native TypeScript in my repositories are typically grouped in a single folder I tend to add a minimal package.json to that folder to indicate ESM. Of course, this is only relevant if your root package.json explicitly indicates CommonJS usage:

{
  "type": "module"
}

Now I can split my code in file - just like I would do for production code -, and for relative imports I add the explicit extension .ts:

import { method } from './method.ts';

method();

There is one small extra step to take for this to work properly; we need to tell TypeScript that we're going to use these extensions:

{
  "compilerOptions": {
    "allowImportingTsExtensions": true
  }
}

Using a linter like eslint, oxlint or biome you can enforce usage of extensions in imports, or not, per folder.

CommonJS

If you decide to use CommonJS as module resolution you can do so via package.json:

{
  "module": "commonjs"
}

There is a big caveat for this however; you will still need to include relative paths with an extension when using native TypeScript. Where this will work in plain JavaScript:

const { sum } = require('./sum');

sum(1, 2);

This will not work when using native TypeScript. In that situation you are forced to specify the extension:

const { sum } = require('./sum.ts');

sum(1, 2);

Mixing "script" files and "production" files

So you've built a nice script, and you really want to use a class, method or type definition that is also used in the code that is used in your "production" code. This could become problematic if you don't use a bundler or postprocessor. Since you need to specify the .ts - or .mts or .cts - extension on your imports, you will also need to do this on your "production" code. This can become a problem, because TypeScript will not rewrite the extension for you. And that means that the transpiled code will have the extension .js, but files will try to include it using the extension .ts.

There are three workarounds for this:

Use a bundler to build your production artifact as the bundler will take care of this translation.
Do not mix production code and native TypeScript code.
Use a postprocessor to correct the imports after TypeScript transpilation.

My setup

This may sound like a lot of caveats and a lot of stuff to deal with, just so you don't have to introduce a dependency on ts-node/tsx or add a tsc step to your workflow. And you might be right. But I have found that with a "modern" codebase it is just a handful of configurations to make native TypeScript possible.

It is 2026, so I opt for ESM for module resolution. I think it is the way forward. This means that my ./package.json configures this for the entire codebase:

{
  "type": "module"
}

This allows me to use the import from and export syntax, and to use .ts extensions on all my TypeScript files. I try to avoid mixing native TypeScript code and production code by separating them in my filesystem. Typically, the native TypeScript code is for development and/or CI/CD only, and therefore I tend to keep it in a separate scripts folder.

Because size always matters I use a bundler like esbuild to bundle, minify and tree shake my code. This will minimize my code artifact, and that can mean better performance. Especially on serverless architectures a smaller artifact can lead to much lower cold start times. This also makes that I can configure my linter to always require relative imports to have a .ts extension. I am currently using Biome for linting:

{
  "linter": {
    "rules": {
      "correctness": {
        "useImportExtensions": "error"
      }
    }
  }
}

Lastly I configure TypeScript to allow .ts extensions in my imports and - because I use a bundler - I explicitly disable emitting of transpiled code:

{
  "compilerOptions": {
    "noEmit": true,
    "allowImportingTsExtensions": true
  }
}

I don't have any additional safeguards in place to prevent TypeScript-only syntax in the code that is run via native TypeScript. These scripts are either used for local development or as an inspection step in a CI pipeline, so if they fail the fallout is not noticable for my production environment.

tl/dr;

Since NodeJS version 22 it is possible to have NodeJS run TypeScript code without the need to transpile. This feature is called native TypeScript. It adds a minimal latency and should therefore probably not be used for production purposes, but it allows typing and typesafety in your scripts. When you create larger scripts, and you want to perform relative imports then you are best off switching your entire codebase to ESM due to the requirement of extensions in your relative imports.

Peer dependencies in (P)NPM

Timo Schinkel — Mon, 24 Nov 2025 15:52:12 +0000

If you dive into the specification of package.json you might be surprised to learn that next to the options dependencies and devDependencies there are also the options bundleDependencies, optionalDependencies and peerDependencies. Recently we have been migrating our main application to an architecture with multiple frontends based on NextJS. Since we have shared components and logic we expose these as packages with peer dependencies. And I found that a good number of my colleagues are unfamiliar with peer dependencies.

What are peer dependencies

In order to explain peer dependencies, it is beneficial to first look at how NPM and PNPM work. Whenever you install a dependency in your project, this dependency can have other dependencies (these are typically referred to as non-root dependencies or transitive dependencies). Within the scoping model of (P)NPM these dependencies are tied to this specific dependency. The big benefit of this is that you hardly ever have version conflicts in your project, but a big downside of this is that you cannot guarantee that the dependency and your project will have the same version of a dependency, and you might get a good amount of duplicate dependencies in your node_modules folder. Let's look at an example.

NB. For the example I am using existing dependencies that I found in one of the projects of my employer. These dependencies are deliberately outdated, this will allow you to reproduce this yourself. The examples use NPM 10.9.4 and PNPM 10.20.0.

npm install chalk@4.1.2 wrap-ansi@8.1.0 ansi-styles@2.2.1 --save-dev

We can now use npm ls ansi-styles to get a list of usages of this package:

├── ansi-styles@2.2.1
├─┬ chalk@4.1.2
│ └── ansi-styles@4.3.0
└─┬ wrap-ansi@8.1.0
  └── ansi-styles@6.2.3

We have installed three different versions of the ansi-styles package. How NPM stores this is as follows:

└─┬ node_modules
  ├── ansi-styles       // version 2.2.1
  ├─┬ chalk
  │ └─┬ node_modules
  │   └── ansi-styles   // version 4.3.0
  └─┬ wrap-ansi
    └─┬ node_modules
      └── ansi-styles   // version 6.2.3

We can do the same in PNPM - pnpm install chalk@4.1.2 wrap-ansi@8.1.0 ansi-styles@2.2.1 --save-dev - with a similar result for pnpm why ansi-styles:

ansi-styles 2.2.1
chalk 4.1.2
└── ansi-styles 4.3.0
wrap-ansi 8.1.0
└── ansi-styles 6.2.3

In the folder structure we see that we still actually have three versions of ansy-styles installed:

└─┬ node_modules
  ├─┬ .pnpm
  │ ├── ansi-styles@2.2.1
  │ ├── ansi-styles@4.3.0
  │ ├── ansi-styles@6.0.1
  │ ├─┬ chalk@4.1.2
  │ │ └─┬ node_modules
  │ │   └── ansi-styles -> ../../ansi-styles@4.3.0/node_modules/ansi-styles
  │ └─┬ wrap-ansi@8.1.0
  │   └─┬ node_modules
  │     └── ansi-styles -> ../../ansi-styles@6.0.1/node_modules/ansi-styles
  └── ansi-styles -> .pnpm/ansi-styles@2.2.1/node_modules/ansi-styles

Although PNPM makes heavily use of symlinks to save disk space, but we still have three copies of the ansi-styles package. Even the pnpm dedupe won't change this.

Now we need to get a little bit hypothetical. Let's say that wrap-ansi expects an object that is defined in ansi-styles as argument to a function. This means that we need to instantiate this object in our application and pass this to wrap-ansi. And let's say that between version 2.2.1 of ansi-styles - which we have defined in our project - and version 4.3.0 of ansi-styles - which is required by chalk - the type definition of this object has changed. There's a difference of two major versions, so backwards incompatibility is expected. What will happen? Our application will probably break at runtime.

This is where peer dependencies come into play; a dependency that is marked as a peer dependency is required to be installed next to the dependency, as a peer. What if in this example chalk marks ansi-styles@^4.0.0 as peer dependency. Then NPM would have thrown an error when we would try to install ansi-styles@2.2.1 as 2.2.1 does not match the version range ^4.0.0. So, peer dependencies allow you to guarantee that a transitive dependency is the same version range as the non-transitive dependency of the application.

A concrete example of this is NextJS. NextJS is a framework that uses React, and as such the package next has a peer dependency on react, so when we want to install NextJS we als need to install React - and a number of additional packages. Be aware! NPM and PNPM do not automatically install peer dependencies. They will happily let you add next as a dependency without adding react as dependency, and without giving an error message. So, this is something you will need to do yourself. Installing NextJS is done as follows:

npm install next@^15 react@^19 react-dom@^19 @opentelemetry/api@^1.1 @playwright/test@^1.51.1 babel-plugin-react-compiler sass@^1.3  --save

(P)NPM is an outlier in this behavior compared to package managers of other languages. With package managers like Composer (PHP), pip (Python) and NuGet (.NET) dependencies are by default peer dependencies. That means that in those package managers it is not possible to have multiple versions of the same dependency in your application¹.

When to use peer dependencies

Peer dependencies are used when your application and one or more dependencies will be relying on the same definitions of methods or objects. A good rule of thumb is: when instantiation and usage are in two separate parts of the code then you might want to consider using peer dependencies. Let's highlight three concrete examples to illustrate this.

Shared type definitions

We currently operate in three countries, but in some countries we have multiple activities. Within our application landscape these activities are indicated as subsidiaries. For the majority of code it is completely irrelevant how these subsidiaries are represented, but they do need to have knowledge about subsidiaries, for example when we want to make an API call to retrieve information for the current subsidiary. We need to make sure that the subsidiary in the application is defined in the same way as it is in the package that makes the API call - let's call this package product-information with a single method with the following signature:

type GetProduct = (sku: string, subsidiary: Subsidiary) => Promise<Product>;

Where should the type Subsidiary be defined? And what if we also had another package to retrieve contact information - let's calls this package contact-information? That is also localized per subsidiary. It is not desirable to have separate definitions for the same entity. For this we introduced an internal package called constants. That package contains the definition of Subsidiary, and by making this package a peer dependency of product-information and or contact-information we guarantee that the definition of Subsidiary is the same in all the placed we need it.

Plugins

We are using Axios for HTTP requests, and Apollo Client for GraphQL requests. For both of these clients we have built extensions/plugins, for example to send correlation headers². For such an extension you are dependent on the version of the client. As such we mark axios a peer dependency of the extensions for Axios, and apollo-client of the extensions for Apollo Client. That way we can guarantee that the plugin will be compatible with the version of the client.

Sending these headers is not enough, they need to be added to the logs as well. That's why we also have an extension for our logger that automatically adds any correlation values to all logs. This extension therefore also has a peer dependency on the package that contains the logger.

NB. These extensions also have a peer dependency on a package that contains a type definition for CorrelationContext, which is another example of the previous use case.

Frameworks

We have recently migrated away from our monolithic approach for our main application in favor of a decentralized approach, where separated parts of our application are handled by separate frontends. We have adopted NextJS as our framework of choice for these frontends. Now, just because we know that our application exists of multiple smaller frontends does not mean that our customers need to have this knowledge as well. In fact, for our customers we want to present our application as exactly that: a unified application.

To achieve this idea of a single application we have introduced a slew of packages that build parts of the application; there's a package for the shared layout items - header, footer -, there's a package with all design elements (our design system), and there are packages for things like product cards. All these packages use features defined in both NextJS, but also in React - which is the foundation for NextJS. All these packages can only be guaranteed to work when they know what version of NextJS and React is being used. That is why for all these packages next and react are peer dependencies.

For the latter example we can see how NPM responds to a deliberate violation of the peer dependencies:

> npm i next@^15.0 react@^15.0 --save
npm error code ERESOLVE
npm error ERESOLVE unable to resolve dependency tree
npm error
npm error While resolving: npm-peer-dependencies@1.0.0
npm error Found: react@15.7.0
npm error node_modules/react
npm error   react@"^15.0" from the root project
npm error
npm error Could not resolve dependency:
npm error peer react@"^18.2.0 || 19.0.0-rc-de68d2f4-20241204 || ^19.0.0" from next@15.5.6
npm error node_modules/next
npm error   next@"^15.0" from the root project

But let's have a look at the folder structure when we use the correct versions:

npm i next@^15 react@^19 next-auth@^4 --save

I have added next-auth for illustration purposes, because next-auth also has a peer dependency on next, and this way we can illustrate that only one version of next is installed:

└─┬ node_modules
  ├── next
  ├── next-auth
  └── react

So, instead of having potentially multiple version of the same package, we force NPM - and PNPM - to have a single version of the same package.

A note on PNPM

PNPM by default emits a warning when there's an issue with peer dependencies, but it does not actually exit with a non-success exit code:

> pnpm add next@^15.0 react@^15.0 --save

Progress: resolved 70, reused 33, downloaded 8, added 23, done
 WARN  Issues with peer dependencies found
.
├─┬ react-dom 19.2.0
│ └── ✕ unmet peer react@^19.2.0: found 15.7.0
└─┬ next 15.5.6
  ├── ✕ unmet peer react@"^18.2.0 || 19.0.0-rc-de68d2f4-20241204 || ^19.0.0": found 15.7.0
  └─┬ styled-jsx 5.1.6
    └── ✕ unmet peer react@">= 16.8.0 || 17.x.x || ^18.0.0-0 || ^19.0.0-0": found 15.7.0

> echo $?
0

And also when installing the dependencies we PNPM will respond with a non-error exit code:

> pnpm install --frozen-lockfile

Lockfile is up to date, resolution step is skipped
Already up to date

> echo $?
0

The consequence of this is that automated pull requests from Dependabot or Renovate might pass inspections with invalid peer dependencies³. Looking at some issues on the PNPM repository - e.g. #6893 and #7087 - this is a deliberate choice, and not a high priority issue. There is a setting called strictPeerDependencies for this that you can enable for your repository:

echo 'strict-peer-dependencies=true' >> .npmrc

The problem is that this setting does not work very consistently; it will not work when running pnpm add nor when running pnpm install with the --frozen-lockfile flag - which should be the default in your CI/CD pipelines. For that to work you need to run an additional command:

> pnpm install --resolution-only --prefer-frozen-lockfile

 ERR_PNPM_PEER_DEP_ISSUES  Unmet peer dependencies

.
├─┬ react-dom 19.2.0
│ └── ✕ unmet peer react@^19.2.0: found 15.7.0
└─┬ next 15.5.6
  ├── ✕ unmet peer react@"^18.2.0 || 19.0.0-rc-de68d2f4-20241204 || ^19.0.0": found 15.7.0
  └─┬ styled-jsx 5.1.6
    └── ✕ unmet peer react@">= 16.8.0 || 17.x.x || ^18.0.0-0 || ^19.0.0-0": found 15.7.0

> echo $?
1

NB. You might notice the missing --frozen-lockfile flag and the presence of the --prefer-frozen-lockfile flag. There's an issue open on the PNPM repository about this: #8433. What my experiments have shown is that this command will correctly use the lockfile for resolution.

Keep in mind that this does not actually install the dependencies! This will only perform a resolution step. You will still need to run pnpm install --frozen-lockfile after running this command, but it will exit with a non-success exit code, and thus any code inspections should fail accordingly.

Conclusion

Looking at dependency managers for similar (web focused) languages the JavaScript ecosystem stands out; where Composer, Nuget and Pip install a package only once, NPM, PNPM and Yarn scope a package to their direct dependency. This can become an issue in the scenarios where you reuse definitions between one of multiple packages. For these scenarios you can use the concept of peer dependencies.

Proper handling of peer dependencies will give you guarantees about versions of packages being used within your application, but also guarantees for the maintainer of packages that their package works as expected. Neither NPM and PNPM automatically install peer dependencies, nor do they notify you of missing peer dependencies. NPM does give you the guarantee that the versions of peer dependencies are valid by default, but for PNPM some extra steps are required.

Of course, with enough trickery you might be able to install two versions of the same package, but it is far from trivial. ↩
Correlation headers are headers that are sent with (internal) interactions. When added to logs it allows for correlating all the logs/operations across applications for a single request. The W3C has a recommendation in Trace Context that has a similar purpose. ↩
Of course every codebase has proper test coverage, so the inspections would fail in case invalid peer dependencies were to cause problems, right?! ↩

Friday Thoughts on email validation

Timo Schinkel — Fri, 14 Jun 2024 08:10:57 +0000

While working on a new authentication system I was getting alerts that account creation was failing. After diving into the logs I learned that the accounts were rejected because the email addresses were not deemed valid. But I made sure standardized email validation was in place. What's going on here?

So many systems and so many specifications

The authentication system is web based and thus uses HTML¹. There is a backend written in JavaScript (actually TypeScript), which in turn - for some operations - talks to a service written in .NET that stores data in AWS Cognito.

Because the front-end is web based we use <input type="email">. This has a number of benefits from the perspective of usability; we get out-of-the-box validation on the format of the input, for some devices a customized keyboard is shown, and password managers are inclined to prefill the field. The validation rules of this input type are well-defined in the HTML specification. There's even a handy regular expression for you to use, which is nice as JavaScript does not have out-of-the-box email validation.

The backend of our system is dotnet and uses a feature called Data Annotations for Model Validation that can be used to validate incoming models, including a validation for email addresses. Microsoft is nice enough to share the code for this validation with us: https://github.com/microsoft/referencesource/blob/master/System.ComponentModel.DataAnnotations/DataAnnotations/EmailAddressAttribute.cs#L48

The storage for our system is AWS Cognito, and this was the actual source of our errors. Looking at the documentation of Cognito AWS tells us the following:

Generally email , Value must be a valid email address string following the standard email format with @ symbol and domain, up to 2048 characters in length.

Luckily we give AWS enough money that they are willing to answer our questions. AWS told me that they use RFC 3696 to validate email addresses.

An interesting fact is that all the specifications for email related address also need to be compliant with RFC 1035. That RFC describes how a domain name should be constructed.

To validate or not to validate

According to David Gilbertson the only proper way to validate an email address is by sending an email to that address containing a link. When that link is clicked then we know for sure that the email address is valid. But we are in the e-commerce business, and we want to remove as many blockers from our customer journey as possible. As such we don't want to interrupt the checkout process with a verification email.

So, yes ideally we would not validate the email syntax, and yes ideally we would send a verification email, but in the real world we are sometimes faced with non-ideal scenarios. In our situation it did not help that the error messages coming from Cognito do not contain distinctive error codes. And even if Cognito did then our .NET service would have to respond with a Bad Request status code, and we still have to interpret the response. Validation of the syntax of an email address is not ideal, but for our scenario it allows us to quickly give feedback to our user and prevent "invalid" email addresses to make their way into the rest of our system.

Comparison

The email addresses that were marked as invalid can be grouped in three scenarios:

invalid . usage; the local part of an email address cannot start of end with . and two of more . are not allowed. So .name@example.com, name.@example.com, and na..me@example.com are not allowed
missing TLD extensions; according to RFC 5233 the domain part can be any internet address, which makes name@localhost a valid email address, however this is in violation of RFC 3696

Let's make a comparison of how the different layers handle these addresses:

                                    HTML    .NET    Cognito  
name@example.com                     ✅      ✅       ✅
name.name@example.com                ✅      ✅       ✅
"name..name"@example.com             ❌      ✅       ❌
name@localhost                       ✅      ✅       ✅
nåme@example.com                     ❌      ✅       ✅
aA0!#$%&'*+-/=?^_`{|}~@example.com   ✅      ✅       ✅

name.example.com                     ❌      ❌       ❌
.name@example.com                    ✅      ✅       ❌
name.@example.com                    ✅      ✅       ❌
name..name@example.com               ✅      ✅       ❌
<name>@example.com                   ❌      ✅       ❌

name@-example.com                    ❌      ✅       ❌
name@example-.com                    ❌      ✅       ❌
name@example.com-                    ❌      ✅       ❌

# extra examples from the 2024-6-28 update
name@x.com                           ✅      ✅       ✅
name@e.mail.com                      ✅      ✅       ✅
name@12mail.com                      ✅      ✅       ✅
name@1.1                             ✅      ✅       ❌
name@1.com                           ✅      ✅       ✅
name@1.2.com                         ✅      ✅       ✅

The code used to test this using .NET and Cognito can be found in this Gist: https://gist.github.com/timoschinkel/fe409ce4e019138778d4f0d9d1879e1e

I was surprised by this outcome; AWS had told me that Cognito required an RFC 3696 compliant email address, but it still rejected "name..name"@example.com, which is a valid address. At least how I interpret the specification.

tl/dr;

Although we would love to, we don't actually write code under perfect circumstances. Sometimes we are bound by limitations outside our influence. It was our choice to strive for a frictionless customer journey, without email verification via an actual email, and the consequence of this choice is that we depend on Cognito accepting our data, and when Cognito rejects it we introduce friction in our customer journey. By matching the validation rules in all layers with the layer that has the strictest rules we can at least tell our customers that their email address has been rejected and why.

At the end of the day we created a regular expression that allowed addresses that are accepted in any layer of our application. It is not fully compliant with any of the specifications mentioned, but it will allow us to explain to our customers that their email address was rejected based on the structure. If a customer has an address that is blocked by our system we will find out where it is blocked and we'll try to find a way around it.

Addendum

I did not manage to write a regular expression that meets all criteria - the maximum length is still an issue - but I did manage to create one that at works against the test set from this article:

^[\p{L}\d!#$%&'*+\-/=?^_`{|}~]+(?:\.[\p{L}\d!#$%&'*+\-/=?^_`{|}~]+)*@(?:(?!-[a-z0-9]+\.)(?![a-z0-9]+-\.)(?![a-z0-9]+--[a-z0-9]+\.)[a-z0-9-]+\.)+[a-z][a-z0-9]+$

or for your HTML input²:

<input type="email" pattern="[\p{L}\d!#$%&\x27*+\-\/=?^_`\{\|\}~]+(?:\.[\p{L}\d!#$%&\x27*+\-\/=?^_`\{\|\}~]+)*@(?:(?!-[a-z0-9]+\.)(?![a-z0-9]+-\.)(?![a-z0-9]+--[a-z0-9]+\.)[a-z0-9\-]+\.)+[a-z][a-z0-9]+" required name="email">

Concessions have been made in creating this pattern. It does not completely match any of the specifications mentioned in this article, but it does filter out all email addresses that would have been blocked by any of the layers in our application. And because the pattern element does not have any flags the pattern is by definition case-sensitive.

Update 2024-06-28

After deploying this validation rule our observability platform detected a rise in errors; my regular expression missed a number of scenarios. As we don't log privacy-sensitive data to our observability platform we decided to run all our existing users against our pattern. In hindsight, we should have done this earlier in the process.

What we found is that we had email addresses with accented characters like è that were now blocked. We also had missed one character domain names - x.com and domain names that started with a numeric value.

This shows that email validation using a pattern is very difficult and that David Gilbertson was right all along. But because we are still bound by the requirements from Cognito, and because we feel that notifying a visitor that their email address is likely invalid is still a better customer journey then risking the email not reaching the customer due to email reasons we still use a pattern validation.

I have updated my test suite with email addresses that follow the same pattern as the email addresses that were falsely rejected, as well as the gists and the regular expressions. The regular expression now uses \p{L}. This matches all characters that belong to the "letter" category, and this includes special characters like é. But because this has a larger match, it is only used for the name part of the email address. See https://www.regular-expressions.info/unicode.html for more explanation.

This Friday Thought is also applicable if your frontend is built using React or Vue. ↩
The regular expression for the pattern attribute requires some changes; it is case-sensitive, ' and " need to be encoded as \x27 and \x22 respectively, and more characters need to be escaped. ↩

Friday Thoughts on abstraction and not wanting to know

Timo Schinkel — Fri, 02 Feb 2024 15:57:16 +0000

Twice this week I found myself in a discussion about how to abstract a multicast-like behavior. Both times the discussion revolved around "what part of the code should have what knowledge"?

The first situation was about retrieving information. To abstract the retrieval we introduced an interface - let's say IRetrievalService. For reasons beyond the scope of this text some of this information can still be in a legacy system. When we built this application we decided to ignore the legacy system as it had not been updated in years. But analysis showed we would lose revenue, and so the only solution was to also query the legacy service. One colleague introduced the switch logic in the consumer of the service, in our case a controller:

try {
    const information = await this.retrievalService.retrieve()
        .catch(async (error) => {
            if (error instanceof NotFoundError) {
                return this.legacyRetrievalService.retrieve();
            }
            throw error;
        });
} catch (error) {
    // handle error scenario
}

Another colleague suggested solving this "within" the interface; by introducing a fallback implementation that calls both the current and - if needed - the legacy system:

export class FallbackRetrievalService implements IRetrievalService
{
    constructor(
        private readonly retrievalService: IRetrievalService,
        private readonly fallbackRetrievalService: IRetrievalService,
    ) {
    }

    public async retrieve(): Promise<unknown> {
        return this.retrievalService.retrieve()
            .catch(async (error) => {
                if (error instanceof NotFoundError) {
                    return this.fallbackRetrievalService.retrieve();
                } 
                throw error;
            });
    }
}

Both approaches do exactly the same. The difference is that for the first every part of the codebase that calls the IRetrievalService needs to have knowledge about when and how to fall back to the legacy service. The latter approach abstracts that knowledge away with the result that none of the parts of the codebase that call the IRetrievalService needs to know about the legacy service. In fact only two parts of the code need to have some knowledge about this behavior; the actual FallbackRetrievalService and the code that handles dependency injection/initialization.

I think that for every abstraction that you write you need to ask yourself the question: Who needs to know what? I have seen too many occurrences where an interface required a table name as parameter - should the code calling the interface have the knowledge that the data is stored in a database? -, and too many occurrences where an interface could throw a database exception or an HTTP exception - again; should the code calling the interface have the knowledge that that is coming from a database or API? No. At least, not in my opinion.

By consequently asking yourself the question "where should this knowledge be?" you can make your code much more flexible and concise. If a year from now we decide to remove the legacy service, all that is to be done is change the dependency injection/initialization and remove a few files. None of the calling code needs to be changed. It also helps you with testing; you want to be sure that the fallback scenario is working as expected. With a FallbackRetrievalService you only need to test that class, and you don't need to test all possible variations at every location where it is called. As a bonus it is much easier to ensure the fallback (or multicast) is applied everywhere.

Constructor promotion in serializable objects

Timo Schinkel — Fri, 16 Jun 2023 11:42:27 +0000

I am not one to easily get excited about new syntactical sugar in languages. As such I was also sceptical about the introduction of constructor promotion in PHP 8.0. But I have grown to really like it. It is a great way to shorten the definition of you objects, and I actually miss it now when I'm working in C# or TypeScript. I use constructor promotion in both services and in value objects, but recently we suffered an outage caused by applying constructor on an object that was serialized for caching.

NB Serialization does not necessarily have to happen explicitly. Session handlers and cache handlers often serialize and deserialize objects implicitly.

Our object

In order to demonstrate the issue let's assume an object:

final class Client
{
    public function __construct(
        private string $id,
    ) {
    }

    public function getId(): string
    {
        return $this->id;
    }
}

No, what happens when we serialize this object?

file_put_contents('client.serialized', serialize(new Client('abc')));

This will output the following string:

O:6:"Client":1:{s:10:"Clientid";s:3:"abc";}

NB Clientid is not a string with a length of 10. Because Client::$id is a private property the property in the string representation is surrounded by special byte characters.

A new property appears

As time goes by we find that we want to add another property to our object, so we update the definition:

final class Client
{
    public function __construct(
        private string $id,
        private string $name = '',
    ) {
    }

    public function getId(): string
    {
        return $this->id;
    }

    public function getName(): string
    {
        return $this->name;
    }
}

After deployment the codebase tries to deserialize the previous data, and we

$client = unserialize('O:6:"Client":1:{s:10:"Clientid";s:3:"abc";}');
$client->getName();

This will break on the line where we call Client::getName():

Fatal error: Uncaught Error: Typed property Client::$name must not be accessed before initialization in {filename}

What's happening

When unserializing PHP interprets the serialized string and tries to rebuild the object. The serialized string contains the name of the class and all the properties. This includes private and protected properties. The rebuilding of the object is not done via the constructor, but the values are set directly on the object instance. Because the constructor is not called, the new property name is not defined. This is the cause of the error.

How to prevent

There are a number of ways to prevent this.

Don't use constructor promotion

Easiest solution is to not use constructor promotion for objects that can be serialized. That will add some code, but it will prevent these errors:

final class Client
{
    private string $id;

    private string $name = '';

    public function __construct(
        string $id,
        string $name = '',
    ) {
        $this->id = $id;
        $this->name = $name;
    }

    public function getId(): string
    {
        return $this->id;
    }

    public function getName(): string
    {
        return $this->name;
    }
}

In this example I opted to declare all properties for consistency reasons. Keep in mind that this can potentially still cause issues if you add properties without a default value. Those will result in the same error that the property must not be accessed before initialization.

Use the magic methods `serialize()` and `unserialize()`

By using the __serialize() and __unserialize() you can steer how your object is serialized and unserialized:

class Client 
{
    public function __serialize(): array
    {
        return get_object_vars($this);
    }

    public function __unserialize(array $data): void
    {
        // instantiate new properties:
        $this->name = '';

        foreach ($data as $key => $value) {
            $this->{$key} = $value;
        }
    }
}

This adds more difficulties as you still need to update your __unserialize() method body with every new property you add.

NB PHP has the Serializable interface, but that has been deprecated in favor of __serialize() and __unserialize(), so I will not mention that as a solution.

Run your own serialization

You can of course build your own serialization mechanism. You can opt to export to JSON, but you will have to handle your unserialization yourself as well. In which case this whole article might not be relevant to you.

tl/dr;

Object serialization, although often used, has some pitfalls that you as a developer should be aware of, but that are often overlooked. One of those pitfalls is constructor promotion. I found that the easiest way around that is to not use constructor promotion for properties that you add while you may still have string representations that were created before the property was added.

Make your Cloudformation conditions mean something

Timo Schinkel — Mon, 10 Oct 2022 11:22:33 +0000

Within AWS Cloudformation it is possible to create conditions. You can use these conditions to change behavior of the stack, like create a resource only in some situations. It is tempting to use the environment as a base for your condition. Even the documentation of AWS has an example with the condition CreateProdResources. While performing some migrations I realised that these conditions don't mean anything, and so I renamed my conditions to have meaningful names.

Environment

A typical list of conditions as I've seen so far might look something like this:

Conditions: 
  IsTesting: !Equals [ !Ref Environment, "testing" ]
  IsProductionOrAcceptance: !Or [ !Equals [ !Ref Environment, "production" ], !Equals [ !Ref Environment, "acceptance" ]]
  IsProduction: !Equals [ !Ref Environment, "production" ]

These conditions are typically used like so:

Resources:
  RdsClusterNew:
    Type: "AWS::RDS::DBCluster"
    Condition: IsTesting
    Properties:
      # ...
      DBClusterParameterGroupName: !If [ IsTesting, !Ref "RdsClusterParameterGroupWithPerformanceInsights", !Ref "RdsClusterParameterGroup" ]
      # ...

This is a random resource and when you look at this example it makes sense; Only on the testing stack do we want to create this database cluster and only on the testing stack do we want to enable performance insights for the database. But a feature rarely has a single resource; When creating an RDS cluster you'll need a cluster, a parameter group, instances, instance parameter groups, and possibly even other resources. Enabling the RDS on another environment would mean changing multiple lines. This has an increased risk of changing a condition too many or too few.

Meaningful conditions

One way to avoid this is to give your conditions meaningful names, for example:

Conditions:
  CreateNewRdsCluster: !Equals [ !Ref Environment, "testing" ]
  EnablePerformanceInsights: !Equals [ !Ref Environment, "testing" ]

Resources:
  RdsClusterNew:
    Type: "AWS::RDS::DBCluster"
    Condition: CreateNewRdsCluster
    Properties:
      # ...
      DBClusterParameterGroupName: !If [ EnablePerformanceInsights, !Ref "RdsClusterParameterGroupWithPerformanceInsights", !Ref "RdsClusterParameterGroup" ]
      # ...

Now also creating the new RDS cluster on the acceptance environment as well is a single line change; You only need to change the condition. And the same thing goes for enabling Performance Insights.

tl/dr;

Naming is hard, and just like with your methods and variables in code you will benefit from meaningful and descriptive names when working with (conditions in) Cloudformation. Not only does it make the resource definitions better to read and understand, it also makes enabling or disabling resources easier as they are one-line changes.

The case for immutability

Timo Schinkel — Fri, 29 Apr 2022 19:05:21 +0000

Some common questions I get on pull requests are "what's with those with-methods?" and "why do you use DateTimeImmutable?". The reason is I am a fan of immutability. I became a fan of immutability the hard way; By introducing bugs caused by (unintentional) mutability.

What is immutability

When an object is immutable its state can not be modified after the object has been created. Concrete this means that an immutable object does not have any writable properties nor any setters to update the properties.

For clarification; The most simple implementation of a mutable object:

final class MutableObject
{
    public string $type = '';
}

A more elaborate implementation could have a constructor and a setter. Using a constructor and a setter allows for validation on the value for both initialization and for mutation:

final class MutableObject
{
    private string $type;

    public function __construct(string $type)
    {
        $this->setType($type);
    }

    public function setType(string $type): void
    {
        if (empty($type)) {
            throw new InvalidArgumentException('Type cannot be empty');
        }

        $this->type = $type;
    }

    public function getType(): string
    {
        return $this->type;
    }
}

Why make an object immutable

In short; To guarantee the validity of an object.

In PHP scalar values - integer, float, string and boolean are passed by value by default. That means that in a function or method where I pass an integer as argument we can assign the value of that integer without any side effects on the code that made the call:

function doSomething(int $number): void
{
    $number += 5;
}

$number = 12345;
echo $number . PHP_EOL;

doSomething($number);
echo $number . PHP_EOL;

This above example will output:

12345
12345

Objects and arrays¹ are by default passed by reference. That means that any changes I apply on an object inside a function or method that was passed as an argument will have side effects on the case that made the call:

function doSomething(DateTime $date): void
{
    $date->add(new DateInterval('P1D'));
}

$date = new DateTime("2022-04-26");
echo $date->format('Y-m-d') . PHP_EOL;

doSomething($date);
echo $date->format('Y-m-d') . PHP_EOL;

This above example will output:

2022-04-26
2022-04-27

The contents of $date has been mutated. And there is no way to prevent this in PHP. Apart from making sure your objects can not be mutated. With an immutable object you can be sure that another piece of code does not (accidentally) change the object causing unexpected side effects. Let's assume another example - a scenario I actually encountered myself:

$today = new DateTime(); // this could be injected from a clock

$openingHours = $this->openingHoursService->getOpeningHoursForUpcomingDays($store, $today, 7);

if ($this->isCurrentlyClosed($today, $openingHours)) {
    // try to find an alternative that is currently open
    $currentlyOpenStores = $this->openingHoursService->getCurrentlyOpen($today);
    // ...
}

After releasing this change it took a few days for one of the stakeholders to start complaining about wrong information being shown. We were unable to reproduce this until a colleague decided to set $today to the moment the stakeholder reported the issue. Now we saw the wrong information as well! Stepping through the code we discovered that properties of $today were changing after calling the opening hours service. We had a look at the implementation and found that it queried the database as follows:

select
    open, closed
from
    opening_hours
where 
    store = {store_id}
    and open >= {$today}
    and open < {$today->add(new DateInterval("P${days}D"))}

Because $today is an instance of DateTime calling the add() method on it actually updated the internal value of $today.

We can now get into a discussion about this implementation. If the developer that implemented this method had used the date manipulation features of the database this issue would not have existed. More important is that unexpectedly an unrelated piece of code changed "my object". This is exactly the scenario where immutability can help; If we were to pass an immutable object we can be absolutely certain that no matter how getOpeningHoursForUpcomingDays() is implemented the object that we pass to it will never change.

The solution for this specific scenario was to replace the DateTime with DateTimeImmutable. It required a change in signature for the service we called, but from that moment on our bug was fixed.

How to implement an immutable object

The immutable equivalent of MutableObject could look something like this:

final class ImmutableObject
{
    private string $type;

    public function __construct(string $type)
    {
        if (empty($type)) {
            throw new InvalidArgumentException('Type cannot be empty');
        }

        $this->type = $type;
    }

    public function getType(): string
    {
        return $this->type;
    }
}

So what about those "with-methods" mentioned earlier. Ideally all properties are to be part of the constructor. This can lead to some readability difficulties when working with a large amount of properties and when some properties are optional. Another scenario is when you want to change one or more values after instantiation of the object. The convention we have been using - and that is used in PSR's as well - is to use methods that start with with:

final class ImmutableObject
{
    private string $type;
    private ?string $label = null;

    public function __construct(string $type, ?string $label = null)
    {
        if (empty($type)) {
            throw new InvalidArgumentException('Type cannot be empty');
        }

        $this->type = $type;
        $this->label = $label;
    }

    public function getType(): string
    {
        return $this->type;
    }

    public function withLabel(string $label): static
    {
        return new self($this->type, $label);
    }
}

Another implementation uses clone. This implementation is useful when an object had a high number of properties or when the validity of your object requires a combination of arguments to be specified²:

final class ImmutableObject
{
    private string $type;
    private ?string $label = null;

    public function __construct(string $type)
    {
        if (empty($type)) {
            throw new InvalidArgumentException('Type cannot be empty');
        }

        $this->type = $type;
    }

    public function getType(): string
    {
        return $this->type;
    }

    public function withLabel(string $label): static
    {
        $clone = clone $this;
        $clone->label = $label;

        return $clone;
    }
}

Instantiating these objects becomes a bit more verbose, but will almost resemble prose:

$object = (new ImmutableObject('my-type'))
    ->withLabel('My type');

In the examples shown so far I have opted for verbosity. With the syntactic sugar that is introduced in PHP 8 and PHP 8.1 we can shorten object definitions drastically:

final class ImmutableObject
{
    public function __construct(
        public readonly string $type
    ) {}
}

This example will make the property available for usage by other objects. If you prefer to use methods for retrieval of data all you need to do is make the property private and introduce a getter similar to the examples above.

Should every object be immutable?

Short answer: no. In my opinion writing code is like being in traffic; You need to be predictable. I think most data containers would benefit from immutability. There are however some scenarios where a mutable object makes more sense as some objects are expected to maintain state. For example a form abstraction; A form contains objects that represent the fields and these fields have a value. This value can be pre-populated, but will be updated based on the info that is submitted. In that scenario it makes sense that the form instance has some form of state.

tl/dr;

Mutable objects can cause bugs that might be difficult to find. By making object immutable you have the guarantee that you are in control of your objects and their state. Making objects in immutable requires only a relative small amount of code.

Arrays are technically passed by reference, but because most operations on an array creates a new array instance this is not very apparent. Calling methods that change the internal pointer of an array - like reset() and next() - do have effects on the array instance and can therefore "leak". ↩
PHP 8 has introduced named arguments. This makes working with a large number of arguments easier. The downsides of this are that your argument names have now become part of your public facing API and you will have to add more complex validation rules when arguments can only be set in combination with other arguments. ↩

Keeping dependencies up-to-date in Composer

Timo Schinkel — Wed, 28 Jul 2021 11:58:23 +0000

In Good practices when working with Composer I discussed some basics about Composer. Something that is not discussed in that article is the importance of keeping your dependencies up-to-date. This goes for both libraries and applications. In this article I will focus on applications - codebases that also maintain a composer.lock file in source control. The majority of the tips and tricks however can also be used for libraries.

Why should I keep my dependencies up-to-date?

Not updating your dependencies has its benefits - you are guaranteed that the interface of the dependency will not change for example. It has some downsides as well however; You will miss out on security updates, new features and improvements.

Another reason to keep your dependencies up-to-date is to keep your migrations small. It is - usually - simpler to migrate a minor version than to migrate a major version¹ as minor upgrades usually contain fewer changes and should not introduce breaking changes.

composer outdated

The simplest way to find out if you have dependencies that are not up-to-date is to run composer outdated. This command will output a list of all dependencies - both direct dependencies and indirect dependencies² -, their current version in your composer.lock and their most recent version. Dependencies that are up-to-date are by default not shown. If you do want to show these add the flag -a or --all. This is an excerpt of the output of composer outdated --all I ran on one of my codebases:

phpunit/phpunit                       9.5.6     9.5.6     The PHP Unit Testing framework.
psr/container                         1.1.0     2.0.1     Common Container Interface (PHP FIG PSR-11)
psr/log                               1.1.3     1.1.4     Common interface for logging libraries

It is good to know that composer outdated does not take any version constraints into account when compiling the list. The version constraint for psr/container is set to ^1.1, but composer outdated shows 2.0.1 as the most recent version.

Apart from the --all flag a few other useful flags are available for composer outdated. The documentation of Composer explains these flags very well.

Updating packages

So what if one or more dependencies is outdated? You update them! To update dependencies two commands can be used: composer update and composer require. The difference between these two commands is that composer update will try to update a dependency based on the current constraints in composer.json and will only update composer.lock. With composer require however Composer will try to install the latest version - keeping existing dependencies and platform constraints in mind - and will update both composer.json and composer.lock. This difference also means that composer update typically will not update a package to a new major version.

composer update

Taking the output from composer outdated from before we can run composer update. This will try to update all dependencies:

> composer update
Loading composer repositories with package information
Updating dependencies
Lock file operations: 0 installs, 2 updates, 0 removals
  - Upgrading psr/container (1.1.0 => 1.1.1)
  - Upgrading psr/log (1.1.3 => 1.1.4)
Writing lock file

When you're working on a large codebase with a lot of dependencies, updating all of them might result in a lot of new packages. This can be undesirable as you might not know how and where all these dependencies are used. If that is the case you can limit the dependencies to be updated by specifying them:

composer update psr/container psr/log

The side effect of specifying what dependencies should be updated is that Composer will not update any other packages. This can be a problem when you are trying to update to a new minor version, but one of the indirect dependencies needs a newer version as well. Assume that I have installed timoschinkel/codeowners-cli with the constraint ^1.0 and I have currently installed 1.0.0. This library depends on timoschinkel/codeowners:^1.0 and version 1.0.0 is therefore installed. A new version of timoschinkel/codeowners-cli is released, tagged 1.1.0 and now it requires timoschinkel/codeowners:^1.1. Running composer update timoschinkel/codeowners-cli will result in Composer not updating the dependency:

> composer update timoschinkel/codeowners-cli
Loading composer repositories with package information
Updating dependencies
Nothing to modify in lock file
Writing lock file
Installing dependencies from lock file (including require-dev)
Nothing to install, update or remove

Composer will see that the constraint timoschinkel/codeowners:^1.1 does not match the already installed version 1.0.0. We can specify a specific version to be installed. Maybe Composer just needs some help:

> composer update timoschinkel/codeowners-cli:1.1.0
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires timoschinkel/codeowners-cli ^1.0, 1.1.0 -> satisfiable by timoschinkel/codeowners-cli[1.1.0].
    - timoschinkel/codeowners-cli 1.1.0 requires timoschinkel/codeowners ^1.1.0 -> found timoschinkel/codeowners[1.1.0] but the package is fixed to 1.0.0 (lock file version) by a partial update and that version does not match. Make sure you list it as an argument for the update command.

Use the option --with-all-dependencies (-W) to allow upgrades, downgrades and removals for packages currently locked to specific versions.

One option to solve this would be to add timoschinkel/codeowners to the list of packages to update as well. This could however lead to just another dependency that is causing the same problem. The solution is to use either -W or -w with the composer update command - as Composer tells us already in the output. What this flag will do is also update dependencies of the dependencies that you are updating. The difference between -w (lowercase) and -W (uppercase) is that the latter will also update any direct dependencies.

> composer update timoschinkel/codeowners-cli -w
Loading composer repositories with package information
Updating dependencies
Lock file operations: 0 installs, 2 updates, 0 removals
  - Upgrading timoschinkel/codeowners (1.0.0 => 1.1.0)
  - Upgrading timoschinkel/codeowners-cli (1.0.0 => 1.1.0)
Writing lock file

composer require

As mentioned before, running composer update will seek newer versions of your dependencies while taking the version constraints into account. That means that a new major version is not likely to be installed. I found that the easiest way to upgrade a dependency to a new major version is to use composer require. This will work in the same way as when you would require a dependency that is not already present in your composer.json; It will find the most recent version - taking any existing constraints into account - and updates the composer.json with the closest minor version:

> composer require psr/log
Using version ^3.0 for psr/log
./composer.json has been updated
Running composer update psr/log
Loading composer repositories with package 
Lock file operations: 0 installs, 1 update, 0 removals
  - Upgrading psr/log (1.1.4 => 3.0.0)
Writing lock file

Similar to composer update you can specify a specific version, require multiple dependencies, and you can use -w and -W if needed. Nice side effect is that Composer will verify if any indirect dependencies are now no longer needed and will remove them from your composer.lock.

NB When using composer require on an existing development dependency don't forget to use the --dev flag, otherwise Composer will mark the dependency a production dependency.

When to update a dependency constraint

Let's assume your codebase uses PHPUnit via the constraint ^9.4. You are currently using version 9.4.4 and you are updating to version 9.5.7. Should you update the constraint in composer.json to ^9.5?

Your constraint should reflect the versions that your codebase supports. Does your codebase require a feature introduced in 9.5? In that case you should definitely update the constraint to ^9.5 and thus update via composer require --dev phpunit/phpunit:^9.5 -w. Are you updating just the dependency with the purpose of running the most recent version? In that case there is no need to update the constraint and thus update via composer update phpunit/phpunit -w.

Debugging updates

Composer offers two commands that are instrumental when maintaining larger codebases; composer why and composer why-not³. Just as the names imply can you use these commands to ask Composer why a certain dependency is available or why a certain dependency cannot be installed.

The main use case for composer why is to find out why an indirect dependency is present in your composer.lock:

> composer why paragonie/random_compat
ramsey/uuid  3.9.3  requires  paragonie/random_compat (^1 | ^2 | 9.99.99)

composer why-not can tell you the exact opposite. I have found this to be very useful when I find myself in a situation where I'm unable to update an indirect dependency to a specific version:

> composer why-not paragonie/random_compat:9.99.100
ramsey/uuid  3.9.3  requires  paragonie/random_compat (^1 | ^2 | 9.99.99)

Hidden power

composer why-not has a "hidden power"; It also works for requirements like PHP version. This realisation was a very big help in preparing our codebases for the migration to PHP 8:

> composer why-not php:8.0.0
paragonie/random_compat  v9.99.99  requires  php (^7)

Automation

I've used different approaches trying to keep dependencies up-to-date. For a while I blocked an afternoon every month to update outdated dependencies. This works great as long as you have a limited number of codebases and dependencies to keep up-to-date. As these grow a few hours every month is no longer enough. Luckily this can be automated to a certain point.

The most known tool for this is Dependabot. Dependabot integrates seemlessly into Github and is able to create pull requests for outdated dependencies. If you have set up automated tests on your codebase all you have to do is merge the pull request created by Dependabot. It does not get any easier.

If you're not using Github or you don't want to use Dependabot, there's also the possibility to build something yourself. composer oudated can be "configured" to help you with this:

composer outdated --format json --strict

The flag --strict will make that the Composer executable will return a non-zero status code if any dependency is outdated, where --format json switches the output from text to json. The latter is much easier to parse. With the output of this call you can create a pull request, an issue or an alert to whatever communication channel you prefer. All you need to do is run this periodically.

tl/dr;

There are multiple reasons why it's a good idea to keep your dependencies as up-to-date as possible, ranging from preventing vulnerabilities to making future migrations easier. Composer is well equipped to detect outdated dependencies and to update them without you needing to open an editor to change composer.json. If you want to update to a version supported by the constraints you can use composer update. If you need - or want - to update the constraints as well you can use composer require.

If Composer gives you an error trying to update a dependency you can find the culprit by running composer why-not. A good way pf preventing blocking dependencies is to use either -w or -W when running composer update or composer require.

See Semantic Versioning for the difference between minor and major versions. ↩
Direct dependencies are dependencies that are specified in the composer.json of your project ↩
Officially these commands are called depends and prohibits, but I have a personal preference for using why and why-not as it almost makes your command into a grammatically correct sentences. ↩

Using Swagger UI in AWS' serverless stack

Timo Schinkel — Tue, 08 Jun 2021 09:47:29 +0000

Few things are as frustrating as working with an API that is not properly documented. That is why we aim to equip our applications with Swagger / OpenAPI documentation. This exposes the url /swagger where a nice documentation for the API is available. At the same time have we been using more and more serverless technologies, which in our situation means AWS Lambda. In order to use an AWS Lambda function as an HTTP API the function needs to be triggered by a route defined in AWS API Gateway. I want to share the setup we use to quickly expose Swagger documentation for our serverless stacks.

NB The code examples use NodeJS and Cloudformation as those are the tools we currently use at Coolblue. This trick is not very complex, so it should be easily translated to other languages.

Swagger. Or OpenAPI?

OpenAPI is a specification that is used to describe an API. It is the de facto industry standard for describing REST APIs and as such there is a lot of tooling available to generate or interpret OpenAPI specifications.

For clarification: the terms OpenAPI and Swagger are both used. As the OpenAPI Initiative explains themselves:

The easiest way to understand the difference is:

OpenAPI = Specification

Swagger = Tools for implementing the specification

In order to serve a nice documentation we need three things; an OpenAPI specification of the API, a way to generate html from that specification and a way to serve the generated html.

Creating an OpenAPI specification is outside the scope of this text - although that might be an interesting topic for a followup. For this text I will assume you already have an OpenAPI specification. Either generated from code (annotations) or created by hand, maybe using the online Swagger Editor.

Exposing the generated documentation

Given that you have an OpenAPI specification the next step is to convert this to html and expose it. There are a couple of ways to build up html based on an OpenAPI specification; The OpenAPI Initiative themselves offer two NPM packages to do this: swagger-ui and swagger-ui-dist. Both these packages offer an example html and the required javascript and css resources. The benefit of using swagger-ui-dist over swagger-ui is that it is dependency free and ready to use.

Using a serverless architecture has a downside; For every url we need to create an endpoint. We can either create a function and an endpoint for every resource or work with some form of wildcards. The first means additional work, where the latter introduces some additional complexity. Both result in Lambda invocations for static content.

Another option that is actually suggested on the installation instructions of swagger-ui is to use a CDN. There are multiple CDNs that offer these files, like jsdelivr and cdnjs. I'll be using unpkg in these examples, just like in the installation instructions.

In the scenario where you don't want to use an external CDN you can also expose these files through your own CDN solution, for example an S3 bucket. This approach has an additional bonus as you will be able to keep all of your Swagger documentations running on the same version of swagger-ui and you might be able to host additional CSS files to apply some house styles to your Swagger documentations.

Cloudformation

I'll be following the convention to offer the documentation on the /swagger url:

Parameters:
  Environment:
    Type : "String"
    Default: "development"
    Description: "Environment in which resources are deployed."

Resources:
  # Lambdas
  SwaggerFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: swagger.swagger
      FunctionName: !Sub "${AWS::StackName}-swagger"
      CodeUri: .
      Events:
        GetHtml:
          Type: Api
          Properties:
            Path: /swagger
            Method: get
            RestApiId: !Ref "ServerlessApi"
        GetSpec:
          Type: Api
          Properties:
            Path: /swagger.json
            Method: get
            RestApiId: !Ref "ServerlessApi"

  # Api gateway
  ServerlessApi:
    Type: "AWS::Serverless::Api"
    Properties:
      StageName: !Ref "Environment"

I'm applying a little trick here; It is possible to add multiple events to the same AWS::Serverless::Function resource. The reason I am also routing /swagger.json to the Lambda function is to enable consumers to download the specification. This specification can be used to autogenerate client code for consuming the API.

Javascript

Now that Cloudformation end of things is set up we need to actually generate the documentation, which is where swagger-ui comes in play:

'use strict';

const { readFileSync } = require('fs');

const applicationName = 'My Awesome Application';

exports.swagger = async (event) => {
    if (event.path === '/swagger.json') {
        return {
            statusCode: 200,
            headers: {
                'content-type': 'application/json'
            },
            body: readFileSync('./etc/swagger.json')
        }
    }

    const body = `<!DOCTYPE html>
        <html lang="en">
        <head>
            <meta charset="UTF-8">
            <title>${applicationName}</title>
            <link rel="stylesheet" href="https://unpkg.com/swagger-ui-dist@3/swagger-ui.css">
        </head>
        <body>
            <div id="swagger"></div>
            <script src="https://unpkg.com/swagger-ui-dist@3/swagger-ui-bundle.js"></script>
            <script>
              SwaggerUIBundle({
                dom_id: '#swagger',
                url: '/swagger.json'
            });
            </script>
        </body>
        </html>`;

    return {
        statusCode: 200,
        headers: {
            ['Content-Type']: 'text/html',
        },
        body
    };
}

Conclusion

Writing documentation is just as much part of the development process as writing code. OpenAPI offers a standardized way of describing an API and swagger-ui offers an off-the-shelf solution that can generate a very readable documentation. Exposing Swagger documentation on a standardized url - /swagger - adds a consistent way for any consumers of the API to know what endpoints the API exposes and how these endpoints can be exposed. With just a handful lines of code it is possible to expose Swagger documentation for your serverless stack.

Continue reading

There are a lot of great resources about OpenAPI. I want to point out a couple of resources that have been very helpful to me while working with OpenAPI.

Swagger Petstore - A demo and showcase of all the features offered by OpenAPI and Swagger. A great resource if you prefer to read example code over reading the documentation of the OpenAPI specification.
OpenAPI specification - An extensive documentation of all properties and their allowed values for the OpenAPI specification.

Testing with immutable PSR-7 objects and Prophecy

Timo Schinkel — Thu, 04 Mar 2021 16:32:46 +0000

Recently I have been working a lot with PSR-15 and PSR-18 and one of the characteristics of these recommendations is that it uses the immutable objects specified in PSR-7. Soon after we introduced PSR-18 in our codebase a colleague implemented a client including a unit test that was passing. And yet the code was failing when run in the browser. The cause of this was that the fact that RequestInterface objects are immutable by specification was overlooked. Due to the use of Prophecy and prophesized objects we were struggling to properly test it. Until another colleague introduced me to Argument::that() that is.

Let's consider the following class as a simplified example of the situation described above:

<?php

declare(strict_types=1);

use Psr\Http\Client\ClientInterface;
use Psr\Http\Message\RequestInterface;
use Psr\Http\Message\ResponseInterface;

final class ApiKeyMiddleware implements MiddlewareInterface
{
    /** @var string */
    private $apiKey;

    public function __construct(string $apiKey)
    {
        $this->apiKey = $apiKey;
    }

    public function process(RequestInterface $request, ClientInterface $client): ResponseInterface
    {
        $request->withHeader('Api-Key', $this->apiKey);

        return $client->sendRequest($request);
    }
}

This is an example of a PSR-18 middleware we created and introduced into our codebase shortly after the approval of PSR-18¹. Seems correct at first glance. Now let's have a look at a simple unit test to test the happy path using Prophecy:

<?php

declare(strict_types=1);

use PHPUnit\Framework\TestCase;
use Prophecy\Argument;
use Psr\Http\Client\ClientInterface;
use Psr\Http\Message\RequestInterface;
use Psr\Http\Message\ResponseInterface;

final class ApiKeyMiddlewareTest extends TestCase
{
    public function testApiKeyIsAddedAndResponseIsReturned(): void
    {
        $apiKey = 'api key';

        $client = $this->prophesize(ClientInterface::class);
        $client
            ->sendRequest(Argument::type(RequestInterface::class))
            ->shouldBeCalled()
            ->willReturn($this->prophesize(ResponseInterface::class)->reveal());

        $request = $this->prophesize(RequestInterface::class);
        $request
            ->withHeader('Api-Key', $apiKey)
            ->shouldBeCalled()
            ->willReturn($request);

        $middleware = new ApiKeyMiddleware($apiKey);
        $response = $middleware->process($request->reveal(), $client->reveal());

        self::assertInstanceOf(ResponseInterface::class, $response);
    }
}

At first glance the code and the unit test look good. But while the test will pass, the code does not actually do what it should; when running this code the remote server will most likely respond with a 403 Forbidden response, because we are not sending the Api-Key header with the request. This is caused by the immutability of the \Psr\Http\Message\RequestInterface. The header is being set, but this operation returns a new instance of the request. Unfortunately the new object is not assigned to anything and therefore the request object without the header is passed to the actual client. We have found the culprit and the fix is simple, but how can we make sure this will not happen again in the future?

Prophecies, Prophecies everywhere

The first approach one can take is to make sure every operation return a new prophesized object:

public function testApiKeyIsAddedAndResponseIsReturned(): void 
{
    $apiKey = 'api key';

    $requestWithApiKey = $this->prophesize(RequestInterface::class)->reveal();

    $request = $this->prophesize(RequestInterface::class);
    $request
        ->withHeader('X-Api-Key', $apiKey)
        ->shouldBeCalled()
        ->willReturn($requestWithApiKey);

    $client = $this->prophesize(ClientInterface::class);
    $client
        ->sendRequest(Argument::is($requestWithApiKey))
        ->shouldBeCalled()
        ->willReturn($this->prophesize(ResponseInterface::class)->reveal());

    $middleware = new ApiKeyMiddleware($apiKey);
    $response = $middleware->process($request->reveal(), $client->reveal());

    self::assertInstanceOf(ResponseInterface::class, $response);
}

This test will fail, as it should. But now consider a situation where not just one operation is done, but multiple. Maybe a header is set, another is modified. Or - shifting away from PSR-7 objects - look at other immutable objects that might have multiple operations. The amount of prophesized objects would grow and every time the order in which the object methods are called changes the unit tests have to be changed. Every call to the immutable object adds three methods on the prophesized object and decreases readibility.

Value object do not need prophesizing

The objects for the example are not just immutable objects, but can also be seen as value objects². And there is some discussion whether one would even mock value objects. Maybe instantiating these objects is a solution. In that case the order in which the object is modified is irrelevant and our test will focus more on what comes out, than on the order methods are called.

This works great if a method is tested that modifies and returns an object; we'll get an instance of that object and we are able to run all sorts of assertions on it. But what if this is not the case. What if we have a situation where a value object is passed as a parameter, some modifications are done and the value object is passed to another object. Something like the middleware from the example earlier. Some testing frameworks like Mockery offer spies to test these situations. When using Prophecy this situation can be handled using Argument::that():

public function testApiKeyIsAddedAndResponseIsReturned(): void
{
    $apiKey = 'api key';

    $request = new Request('/url', 'GET');

    $client = $this->prophesize(ClientInterface::class);
    $client
        ->sendRequest(
            Argument::that(function (RequestInterface $request) use ($apiKey): bool {
                return $request->getHeaderLine('X-Api-Key') === $apiKey;
            })
        )
        ->shouldBeCalled()
        ->willReturn($this->prophesize(ResponseInterface::class)->reveal());

    $middleware = new ApiKeyMiddleware($apiKey);
    $response = $middleware->process($request, $client->reveal());

    self::assertInstanceOf(ResponseInterface::class, $response);
}

Instead of testing against mocked value objects this code check if sendRequest() is called with an instance of RequestInterface that has the expected value for the X-Api-Key header. The nice thing about this is that we are no longer testing whether the code makes a certain set of calls in a given order. We are actually testing whether an object is in the correct state when passed.

PSR-7 - a special case

This unit test will fail when ran against the implementation from earlier. And in this situation a failing test is a good thing. But for this specific situation another "issue" has made it's way into the testcase; the unit test is now depending on the implementation that is being used for PSR-7. When working in a large codebase - like I currently am - one might have more than just a few usages of these interfaces and thus unit tests. Using the suggested approach, instantiating value objects instead of prophesizing, will lead to a large amount of object instantiations. And this will make switching to another implementation more work. Ideally this instantiating is centralized as much as possible. This is exactly why PSR-17 - HTTP Factories - was introduced. Typically dependencies - like these factories - are injected. For unit tests this is not feasible without plugins. My solution is a bit less fancy; a trait:

<?php

declare(strict_types=1);

use Psr\Http\Message\RequestInterface;

trait Psr7Factories
{
    public function createRequest(string $method, $uri): RequestInterface
    {
        return (new RequestFactory())->createRequest($method, $uri);
    }
}

This can be applied in the unit test:

<?php

declare(strict_types=1);

use PHPUnit\Framework\TestCase;
use Prophecy\Argument;
use Psr\Http\Client\ClientInterface;
use Psr\Http\Message\RequestInterface;
use Psr\Http\Message\ResponseInterface;

final class ApiKeyMiddlewareTest extends TestCase
{
    use Psr7Factories;

    public function testApiKeyIsAddedAndResponseIsReturned(): void
    {
        $apiKey = 'api key';

        $request = $this->createRequest('GET', '/url');

        $client = $this->prophesize(ClientInterface::class);
        $client
            ->sendRequest(
                Argument::that(function (RequestInterface $request) use ($apiKey): bool {
                    self::assertEquals($request->getHeaderLine('X-Api-Key'), $apiKey);

                    return true;
                })
            )
            ->shouldBeCalled()
            ->willReturn($this->prophesize(ResponseInterface::class)->reveal());

        $middleware = new ApiKeyMiddleware($apiKey);
        $response = $middleware->process($request, $client->reveal());

        self::assertInstanceOf(ResponseInterface::class, $response);
    }
}

When testing a class that creates the PSR-7 objects via a PSR-17 factory this approach can also be used:

$requestFactory = $this->prophesize(RequestFactoryInterface::class);
$requestFactory
    ->createRequest('GET', '/url')
    ->shouldBeCalled()
    ->willReturn($this->createRequest('GET', '/url'));

More usages

In the examples up until now instances of \Psr\Http\Message\RequestInterface have been created. But this approach can also be used for instances of Psr\Http\Message\ResponseInterface or other PSR-7 objects, simply by adding the methods you want to use to the trait. We have opted to follow the PSR-17 method signatures for consistency. This will make it easy to inject different response fixtures to test how the code responds to them:

$client = $this->prophesize(ClientInterface::class);
$client
    ->sendRequest(Argument::type(RequestInterface::class))
    ->willReturn(
        $this->createResponse(200)
            ->withBody(
                $this->createStream(
                    file_get_contents('path/to/response/fixture.json')
                )
            )
    );

And when testing controllers or request handlers it allows for constructing instances of ServerRequestInterface in a more readable way than prophesizing the object:

$request = $this->createServerRequest('GET', '/uri')
    ->withAttribute('attribute', 'value')
    ->withHeader('content-type', 'application/json');

Conclusion

Working with immutable objects is tricky; forget one assignment and your code stops behaving the way you expect it should. By testing for this the right way the resilience of a codebase is improved. Using Prophecy and prophesized objects does not immediately remedy this.

When opting to not mock value objects, but rather use them as is in the unit test Argument::that() can act as a spy allowing for more specific assertions on the value objects. This will also allow assertions to determine the immutability of the object has been satisfied.

The PSR-7 objects are a special case; you could consider them as value objects, but they are also interfaced. Using instances of these objects instead of mocks or prophecies allows for these same immutability asserts. In order to prevent a lot of object instantiations of these objects in unit tests one can use the PSR-17 factories. But since the unit tests don't normally allow for dependency injection an alternative way of centralizing this object instantiation is by creating a trait that handles this.

With the described approach we have managed to reduce the references to the PSR-17 implementation we used to two locations; the dependency injection container and one trait. If for whatever reason a decision is made to move away from the current PSR-7 or PSR-17 implementation this should be a relative small task.

This approach can also be used for packages. The nice thing of the PHP-FIG HTTP foundation is that you can create packages relying just on interfaces. But you also want to test your code. By adding an implementation of PSR-7 as a development dependency consumers of the package won't notice this dependency. Only maintainers and contributors will need to notice.

Middleware is not part of the PSR-18 specification. It is a layer on top of PSR-18 that we built ourselves. ↩
According to Martin Fowler are value objects comparable to eachother based on the values. Since the PSR-7 objects are interfaced there is no documentation on how the object values should be represented internally, but there is a standardized format to represent HTTP requests as strings. ↩

Private serverless APIs in AWS

Timo Schinkel — Wed, 17 Feb 2021 14:29:39 +0000

Our application landscaping is growing to more interfacing applications; A website, mobile apps and customer service tooling just to give a few examples. We've been using the AWS Serverless Application Model to quickly create stacks that are consumable by multiple applications. One downside of using AWS SAM is that the API Gateway endpoints are public by default. We protect these endpoints against abuse via an authentication, but for some endpoints we would like to have an additional layer of protected by making these endpoints only available from inside our VPC. In other words make our endpoints private.

NB This article specifically outlines making an AWS Serverless Application Model stack private. A serverless infrastructure has a great amount of similarities with AWS::ApiGateway::RestApi, but is still subtly different. I like to look at it as a simplification of AWS::ApiGateway::RestApi and AWS::Lambda::Function.

A serverless stack

A typical serverless stack consists of two parts: an AWS Lambda function and a trigger. This trigger can be anything from an upload on an S3 bucket, a message being pushed on a SQS or a call to an HTTP endpoint. In this last scenario the endpoint is defined as an API Gateway resource.

We use Cloudformation to define our stacks. Let's consider a minimal function that allows retrieval of entities of the type entity via the call GET /entities:

Parameters:
  Environment:
    Type : "String"
    Default: "development"
    Description: "Environment in which resources are deployed."

Resources:
  # Lambdas
  GetEntitiesFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.getEntities
      FunctionName: !Sub "${AWS::StackName}-get-entities"
      CodeUri: .
      Role: "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
      Events:
        GetResource:
          Type: Api
          Properties:
            Path: /entities
            Method: get
            RestApiId: !Ref "ServerlessApi"

  # Api gateway
  ServerlessApi:
    Type: "AWS::Serverless::Api"
    Properties:
      StageName: !Ref "Environment"

When deployed this should result in a serverless stack with a single endpoint that is publicly available for everyone that knows the url.

VPC endpoint

Making an API Gateway private requires a VPC. Every stack that requires access to the private API must be part of the VPC. Benefit of having a VPC is that access control is centralized in the VPC configuration.

In order to make the API private the traffic will have to pass through the VPC. This is done via a VPC endpoint:

Resources:
  APIGatewayVpcEndpoint:
    Type: "AWS::EC2::VPCEndpoint"
    Properties:
      SubnetIds:
        - !ImportValue "exported-subnet-id"
      SecurityGroupIds:
        - !Ref "resource of type AWS::EC2::SecurityGroup"
      ServiceName: "com.amazonaws.eu-west-1.execute-api"
      VpcId: !ImportValue "exported-vpc-id"
      VpcEndpointType: "Interface"
      PrivateDnsEnabled: false

Disabling the private DNS is a deliberate choice. The reason for that is explained later in this article.

Marking an API private

By default API Gateway resources are publicly available. In 2018 AWS introduced the possibility to mark API Gateway resources as private. Marking an API Gateway as private is easily done in the Cloudformation definition of our stack:

Resources:
  # ...

  # Api gateway
  ServerlessApi:
    Type: "AWS::Serverless::Api"
    Properties:
      StageName: !Ref "Environment"
      EndpointConfiguration:
        Type: "PRIVATE"
        VPCEndpointIds:
          - !ImportValue "infrastructure-api-gateway-vpc-endpoint"
      Auth:
        ResourcePolicy:
          IntrinsicVpceWhitelist:
            - !ImportValue "infrastructure-api-gateway-vpc-endpoint"

NB This example imports a VPC endpoint. We try to separate stacks as much as possible for ownership and maintainability. In a separate infrastructure stack the VPC endpoint(s) are defined and the required identifiers are exported from those stacks. That way every other stack can use them while the hosting team still has the possibility to make changes to the endpoint.

The EndpointConfiguration tells AWS that the API should no longer be publicly available. A required property of EndPointConfiguration is a list of VPC endpoints.

Next to marking the API private we also need to tell the API it is allowed to be called from the endpoint. This is done via a resource policy. In a AWS::Serverless::Api this is done via the Auth property.

When we now deploy this stack, the API will no longer be publicly available.

Consuming the private API

The downside of making an API Gateway resource is that consuming becomes a little bit more complicated. There are two methods that work out-of-the-box. Those methods use the VPC endpoint as entry for the request:

curl -v https://{public-dns-hostname}.execute-api.{region}.vpce.amazonaws.com/{stage}/{path}

The {public-dns-hostname} of the VPC endpoint is visible in the AWS Console. For the following examples I will call the /entities endpoint defined in the Cloudformation for region eu-west-1 and stage testing.

The first method uses the Host header:

curl -v https://vpce-01234567abcdef012-01234567.execute-api.eu-west-1.vpce.amazonaws.com/testing/entities \
-H 'Host: 01234567ab.execute-api.eu-west-1.amazonaws.com'

The second method uses the x-apigw-api-id header:

curl -v https://vpce-01234567abcdef012-01234567.execute-api.eu-west-1.vpce.amazonaws.com/testing/entities \
-H 'x-apigw-api-id: 01234567ab'

Both the host and the API id can be found in the AWS Console after deploying your stack.

There are other options to call your private endpoints that a bit more user-friendly, but that have their own caveats. You can enable the "Private DNS Name" while creating an interface VPC endpoint for API Gateway, but this also means that the VPC where the VPC endpoint is present will no longer be able to access public APIs:

When you select the Enable Private DNS Name option while creating an interface VPC endpoint for API Gateway, the VPC where the VPC Endpoint is present won't be able to access public (edge-optimized and regional) APIs. For more information, see Why can't I connect to my public API from an API Gateway VPC endpoint?.

Source: Invoking your private API using private DNS names

As we have both public and private APIs and we also consume public APIs from within our VPC we are not able to use this approach.

Another option is to use a Route53 alias. We have not yet tried this, so I cannot make any comments on this option.

Conclusion

The SAM model offered by AWS allows for fast development of applications as you don't have to worry too much about your infrastructure choices. AWS will scale if needed. Building a modest sized API can be achieved using the tandem Lambda and API Gateway, but this will result in a public endpoint by default.

If you want to utilize the benefits of the SAM model, but you don't want to expose your API to the public you can make your API private. This will increase security on your endpoint, but it will also introduce some downsides like the necessity to have a VPC and a VPC endpoint.

Mocking third party classes in TypeScript

Timo Schinkel — Mon, 19 Oct 2020 06:54:11 +0000

TypeScript is becoming increasingly popular as a replacement for plain JavaScript; the strict type checking should result in more resilient code. Initiatives like Definitely Typed make that the majority of often used NPM packages have type definitions. This is great when writing code, but can become a nuisance when writing tests. Especially when mocking these sometimes large third party classes.

Mocking with type definitions

Let's consider the following code that takes a payload and stores it on an AWS S3 bucket using the AWS SDK:

import { S3 } from 'aws-sdk';

export class MyS3Repository implements MyRepositoryInterface {
    private readonly s3: S3;
    private readonly bucketName: string;

    public constructor(s3: S3, bucketName: string) {
        this.s3 = s3;
        this.bucketName = bucketName;
    }

    async store(entity: MyEntity): Promise<void> {
        return this.s3.putObject({
            Body: JSON.stringify(entity), 
            Bucket: this.bucketName, 
            Key: entity.getIdentifier()
        }).promise()
        .then((response: S3.PutObjectOutput) => undefined);
    }
}

NB This is a simplified example, but it will work as an example.

After creating an implementation the next logical step is to create a unit test for this class. We don't want to actually make a connection with an S3 bucket every time we run our test suite. One way to prevent this is to mock the S3 object that is injected into the repository. Coming from JavaScript Jest is my go-to tool to write tests and that means jest.fn():

describe('MyS3Repository', () => {
    describe('store()', () => {
        it('should store on S3', async () => {
            const s3 = {
                putObject: jest.fn().mockReturnValue({
                    promise: jest.fn().mockResolvedValue({})
                })
            }
            const repository = new MyS3Repository(s3);
            const entity = new MyEntity(/* ... */);

            expect.assertions(1);
            await expect(repository.store()).resolves.toBe(undefined);
        })
    });
});

When working with "plain" JavaScript this would work as expected, and the unit test would pass. But when using TypeScript you will encounter an error:

Argument of type '{ putObject: jest.Mock; }' is not assignable to parameter of type 'S3'.
Type '{ getObject: Mock; putObject: Mock; }' is missing the following properties from type 'S3': config, abortMultipartUpload, completeMultipartUpload, copyObject, and 98 more.ts(2345)

What happens here is that TypeScript is detecting that the object that should impersonate the S3 object is missing 98 methods. A trade off needs to be made here; do I use an existing package and try to work with it or do I write the necessary code myself. When not working on personal projects the scale typically tilts towards the first. So let's see if we can make this work.

Pick me!

A number of solutions for this problem are described like using the __mocks__ functionality, introducing interfaces and using jest.mock().

For multiple reasons none of these really worked well for me. As I don't have control over the code introducing an interface is not possible. jest.mock() seems to only work on modules that have a default export and __mocks__ still requires mocking the entire signature of a class.

Another option is to use additional tooling like typemoq, ts-mockito or substitute. But I am a bit reluctant to add a full-fledged mocking library if it is not completely necessary.

Here I find myself with a very useful object with 99 methods and I only use one. Enter the Pick<Type, Keys> utility!

Constructs a type by picking the set of properties Keys from Type.

It allows us to tell TypeScript that even if we have an object with 99 methods we only really care about a small subset of those methods:

import { S3 } from 'aws-sdk';

export class MyS3Repository implements MyRepositoryInterface {
    private readonly s3: Pick<S3, 'putObject'>;
    private readonly bucketName: string;

    public constructor(s3: Pick<S3, 'putObject'>, bucketName: string) {
        this.s3 = s3;
        this.bucketName = bucketName;
    }
}

Result of this is that in our unit test we only have to mock putObject() without TypeScript complaining about missing types.

Conclusion

If you find yourself with a large third party class that you want to mock you can resort to tooling like typemoq, ts-mockito or substitute. Or you can leverage features offered by TypeScript. By using Pick you are able to specify exactly what properties you need in your code. Doing that will make your mocking work a lot simpler.

Forem: Timo Schinkel

Native TypeScript with Node

Type safety

Restricted syntax

CommonJS vs ESM

ESM

CommonJS

Mixing "script" files and "production" files

My setup

tl/dr;

Peer dependencies in (P)NPM

What are peer dependencies

When to use peer dependencies

Shared type definitions

Plugins

Frameworks

A note on PNPM

Conclusion

Friday Thoughts on email validation

So many systems and so many specifications

To validate or not to validate

Comparison

tl/dr;

Addendum

Update 2024-06-28

Friday Thoughts on abstraction and not wanting to know

Constructor promotion in serializable objects

Our object

A new property appears

What's happening

How to prevent

Don't use constructor promotion

Use the magic methods __serialize() and __unserialize()

Run your own serialization

tl/dr;

Make your Cloudformation conditions mean something

Environment

Meaningful conditions

tl/dr;

The case for immutability

What is immutability

Why make an object immutable

How to implement an immutable object

Should every object be immutable?

tl/dr;

Keeping dependencies up-to-date in Composer

Why should I keep my dependencies up-to-date?

composer outdated

Updating packages

composer update

composer require

When to update a dependency constraint

Debugging updates

Hidden power

Automation

tl/dr;

Using Swagger UI in AWS' serverless stack

Swagger. Or OpenAPI?

Exposing the generated documentation

Cloudformation

Javascript

Conclusion

Continue reading

Testing with immutable PSR-7 objects and Prophecy

Prophecies, Prophecies everywhere

Value object do not need prophesizing

PSR-7 - a special case

More usages

Conclusion

Private serverless APIs in AWS

A serverless stack

VPC endpoint

Marking an API private

Consuming the private API

Conclusion

Mocking third party classes in TypeScript

Mocking with type definitions

Pick me!

Conclusion

Use the magic methods `serialize()` and `unserialize()`