Forem: Nick Matantsev

Getting TypeORM to work with Next.js and TypeScript

Nick Matantsev — Sun, 14 Feb 2021 22:10:15 +0000

I was setting up a new Next.js app (using NextAuth for SSO) and decided to use TypeORM as the database persistence layer. Next.js is great at running TypeScript code without hiccups and TypeORM, while new to me, seemed promising in its minimalism. However, I ran into some non-trivial error messages and bugs trying to get them to play together. And even though most answers were already discovered by other kind folks on the internet, it took me a while to track it all down and combine together into one solution.

There were three key areas to resolve:

importing TypeScript-formatted entities in Next.js and CLI contexts
support for TypeORM's decorator syntax + reflect-metadata in Next.js build pipeline
preventing hot-module-reload (HMR) class instance confusion during development runtime

Importing Entities and Connection Config

Normally, TypeORM runs inside a vanilla Node.js environment, which means it cannot consume TypeScript files (such as entity class definitions) without precompilation. For example, there is a common error that happens when the entities path in TypeORM config refers to TS source files (i.e. src/entity/*.ts):

Error during schema synchronization:
/project/path/src/entity/User.ts:1
import {
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at wrapSafe (internal/modules/cjs/loader.js:979:16)
...

With Next.js and its built-in support for TypeScript I did not expect this problem to happen, but still ended up encountering the above error message.

The reason is that when TypeORM creates a new connection, it tries to load all the entity class files dynamically based on a path wildcard. That process bypasses the entire Next.js Babel bundling pipeline and falls back on Node.js's built-in module loader. So even though my Next.js server code can import and run entity class TS files just fine, the TypeORM connection initializer lives in a "parallel universe" and naively tries to load them from scratch on its own, which then fails.

I tried to use ts-node to compile TS modules on the fly as they get loaded by TypeORM, but then I got a different kind of error:

RepositoryNotFoundError: No repository for "User" was found. Looks like this entity is not registered in current "default" connection?

In this scenario I ended up having two clones of each entity class co-existing in memory: one loaded and instantiated by TypeORM + ts-node, and another one bundled by Next.js pipeline with the rest of server code. Hence the class reference confusion.

Instead, my approach was to name and import all the entity files explicitly, without an entities wildcard, like so:

import { User } from './entity/User';
import { Account } from './entity/Account';
// etc

And then pass an explicit options object to createConnection() with entity classes directly referenced like this:

createConnection({
  entities: [
    User,
    Account,
    // etc
  ]
})

Correspondingly, I removed the ormconfig.js file to avoid any further conflicts.

I did keep around a separate ormconfig.cli.js file just for CLI schema sync and migrations. For that to work, I installed ts-node and added require('ts-node/register') to the top of the config file so that TS entity definitions can be loaded with no extra fuss. The command-line script looks like this:

typeorm --config ormconfig.cli.js schema:sync

Decorator Syntax and `reflect-metadata` in Next.js

TypeORM entity class definitions use decorator syntax (e.g. @Entity(), @Column(), etc). Also, there has to be a bit of special plumbing to let TypeORM read TypeScript field types such as string and infer database column types such as varchar. To make the above work, the TypeORM documentation asks to install a package called reflect-metadata and also to tweak tsconfig.json to set emitDecoratorMetadata and experimentalDecorators to true.

However, Next.js does not use TSC (the original TypeScript compiler) and instead relies on Babel's @babel/preset-typescript package. Because of that, those tsconfig.json tweaks do not have any effect.

Instead, I added custom Babel configuration in my Next.js project and included the equivalent options for Babel (see this issue about decorator support and this issue about metadata). This is what's in the resulting .babelrc file:

{
  "presets": [
    [
      "next/babel",
      {
        "class-properties": {
          "loose": true
        }
      }
    ]
  ],
  "plugins": [
    "babel-plugin-transform-typescript-metadata",
    ["@babel/plugin-proposal-decorators", { "legacy": true }]
  ]
}

Extra packages needed to be installed (class-properties plugin is already included with Next.js): @babel/plugin-proposal-decorators babel-plugin-transform-typescript-metadata @babel/core.

Note: you can omit installing reflect-metadata and babel-plugin-transform-typescript-metadata if you specify database column types explicitly. Then TypeORM does not have to infer anything from TS types. For some folks that might be preferable from a stability perspective, but at the cost of being more verbose.

Next.js HMR and TypeORM Entity Classes

Hot-module-reloading (HMR) throws another monkey-wrench in the works.

During development, every time you edit your Next.js pages, API routes or other files like entity classes your code ends up being recompiled and reloaded from scratch. Because TypeORM connection manager is not aware of entity class reloads, the connection object quickly gets out of sync and stops being useful.

E.g. if you have a User entity class, Next.js will load and create a class reference for it - let's call it "User v1". That reference is passed to createConnection and of course the rest of your code uses it too. Now, once you edit that class file, Next.js will perform a hot reload, and there are now two different class references living inside runtime memory. One is the original "User v1" and another one is the freshly recompiled "User v2". Your route code is now using the "User v2" class reference, but the connection still has "User v1" in its list of known entities. When you try to e.g. call getRepository(User) in your code, you will not be passing the same class reference as what TypeORM "knows", so you will get this error again:

RepositoryNotFoundError: No repository for "User" was found. Looks like this entity is not registered in current "default" connection?

I saw a few GitHub issues discussing solutions to this (such as this workaround). For me, the ultimate answer was to simply get any prior connection and close it before opening a new one.

Here is a sample code snippet; I put it in a shared central file like src/db.ts:

let connectionReadyPromise: Promise<void> | null = null;

function prepareConnection() {
  if (!connectionReadyPromise) {
    connectionReadyPromise = (async () => {
      // clean up old connection that references outdated hot-reload classes
      try {
        const staleConnection = getConnection();
        await staleConnection.close();
      } catch (error) {
        // no stale connection to clean up
      }

      // wait for new default connection
      await createConnection({
        // connection options go here
      });
    })();
  }

  return connectionReadyPromise;
}

Then in any function that calls out to TypeORM I add this before performing any database actions:

await prepareConnection();

It may seem a bit onerous to include this in many different spots but there always has to be some sort of wait-until-ready logic for database usage anyway, so this ends up serving that exact purpose.

Conclusion

I hope that the TypeORM docs eventually include Babel-specific config recipes and HMR-friendly "connection refresh" helpers like the above. Also, the dynamic wildcard loader for entities would ideally be pluggable into a bundler pipeline like Next.js's. But for now this combination of settings worked pretty well, and I hope it helps you too!

TypeScript front-end library compilation for publishing on NPM

Nick Matantsev — Thu, 26 Nov 2020 16:53:28 +0000

I have recently been going through the process of packaging and publishing a React UI widget on NPM (React CSV Importer) and wanted to document some of the technical nuances encountered along the way.

Please note that there are more comprehensive publishing guides out there, such as this one; here I want to focus on my experience with a specific aspect of the process: the compilation pipeline for the library.

Overview

I am a big fan of TypeScript and use it almost exclusively when writing React front-end apps. There are plenty of tools and patterns that help compile and bundle TypeScript for final deployment to the browser. However, when publishing a library, the build/packaging pipeline has key differences in requirements.

A published front-end library should provide the following:

JavaScript code included by apps
TypeScript typings (.d.ts file for TS apps)

This will be installed and referenced by the applications that are consuming our library (referred to as "consuming app" further on). Because those consuming apps have their own build pipelines and language expectations, we have to keep the above output compliant with those expectations. Let's go over them in some detail.

Generating JavaScript Code

In the most minimal case, one could simply package up and publish the original TypeScript source code; of course, that excludes a large chunk of consuming apps that cannot use TypeScript for various reasons. This is why we need to compile to JavaScript output before publishing.

Unlike a regular app, our library's JavaScript code does not have to be bundled and minified into a single file. We can assume that whichever app consumes our library will have its own Webpack/Rollup/etc setup, so we do not need to perform any of that ourselves.

The simplest build pipeline then is to just run tsc:

# output goes into dist folder (cleaned first using rimraf)
rimraf dist && tsc --outDir dist

To produce the correct "flavour" of JavaScript output, the tsconfig.json file should include the following in addition to your other settings:

{
  "compilerOptions": {
    "target": "ES6", // change to ES5 for much older browsers
    "module": "CommonJS", // change to ES2015 or ESNext for ES module syntax output
    "isolatedModules": true, // may help catch isolation issues
    ... other options ...
  },
  "include": ["src"] // change as needed
}

The generated JavaScript files will get bundled by the consuming app, but they will most likely not get transpiled for legacy browser compatibility. In other words, what you produce is what will run in the browser or server-side Node process directly (as happens while unit testing or pre-rendering page contents). This is why TypeScript target should be fairly conservative: e.g. ES6 is probably good enough for most browsers/environments that will run your code at the moment.

Your TypeScript source files reference each other and third-party module dependencies via import statements. The module setting controls what happens to that import syntax in the resulting JS output. This matters because this will be parsed by the consuming app's Webpack/Rollup/etc bundler, and older versions of bundlers may not recognize the import keyword. Also, if your code runs in a server-side Node process, the runtime might not support it either. Setting module to CommonJS will result in imports being output as require() calls, which is supported most broadly at the moment.

Once you produce your output (in the dist folder in the above example), you might want to refer to the main entry point of your library by adding this to your published package.json:

{
  ...
  "main": "dist/index.js" // change to your entry .js output
  ...
}

This way when the consuming app imports your library it will be loading the correct file under dist.

There might be more complex situations where simple tsc is not enough to build your library. You may want to set up Babel to perform the transpilation for TypeScript alongside other source formats, e.g. PostCSS for stylesheet theming. Or, you may want to rely on Webpack or Rollup to do the same plus also bundle the files together (which is especially useful for libraries that allow a "raw" option - inclusion via script tags). This post cannot document all these possible advanced use cases, of course, but hopefully this provides a starting point for further research.

Generating Typings

When your tsc produces JavaScript output, all the type information (interface declarations, function args and return types) is lost. Hence we want to gather up the typings that were lost and expose them to the consuming app - that is usually referred to as the .d.ts or "DTS" file.

The TypeScript compiler has an option to produce typings for every file that it processes, but this is not very useful for us! A lot of internal types should never be exposed to the consuming app, but tsc has no awareness of what is "internal" versus "external" to the library - so its output will be way too big and include all the unnecessary internal type information.

For small libraries, the simplest thing to do is to "cheat" a bit. Move externally-visible type declarations in your source code to a central file named something like exports.ts and import it in your other source files as usual. Then, before publishing, do the following:

cp src/exports.ts dist/index.d.ts

That's it. All you need to do then is add this to your package.json:

{
  ...
  "types": "dist/index.d.ts"
  ...
}

The consuming app's TypeScript compiler will now consult your typings file and will be able to perform necessary type safety checks downstream.

For more advanced use cases, there are helpers such as dts-bundle-generator. This type of tool will "intelligently" read through your source code, starting with your library entry-point, and collect exposed type information while discarding anything that is purely internal to the library. There are plenty of new caveats that come with this approach, so that deserves a separate write-up.

Review

This post has described a very basic starter build pipeline for a TypeScript-based front-end NPM module. You will need to figure out a few other things before running "npm publish", but hopefully this provides a reference point for further work. Happy publishing!