Forem: Hamish Milne

Zig + CMake

Hamish Milne — Mon, 16 Sep 2024 16:29:33 +0000

I love Zig. I'm a big fan of many (though not quite all!) of the design choices and philosophies behind the language.

One particular capability I love is how easy it makes cross-compiling - both for Zig and C/C++ code. I have vivid memories of the pain involved in setting up cross-compilation toolchains, or just giving up and directly compiling directly on whatever Raspberry Pi I had to hand.

Zig makes this as easy as -DTarget=arm-linux-musleabihf, even from the locked-down Windows laptop I'm forced to use in my day job.

Clang, which zig cc wraps, has the same -target option - but Zig goes a step further and includes dependencies for all the LibC variants it supports, removing the need for any kind of cross-toolchain, headers, or system configuration.

It's magical. I genuinely believe that this will completely change developers' perspectives on cross-compilation. The "Cross compiling" section in every project's documentation will go from a multi-page guide to "here's a list of targets we've tested - go nuts".

To illustrate this, let's take a use case that doesn't involve Zig-the-language at all. Say we're building a pre-existing app for some non-native platform - maybe changing some configuration, adding a patch or two, you know the drill.

In many - dare I say most - cases, projects that proport to support multiple architectures, operating systems, and compiler toolchains will use CMake as a build system.

(CMake has many, many faults. It is not 'good'. But it is currently the best 'mature', 'general-purpose' C/C++ build system by a long, long way.)

In this case, you might define some top-level CMakeLists.txt and add your dependencies as Git submodules:

cmake_minimum_required(VERSION 3.22)
project(my-project)
# Adjust any configuration...
add_subdirectory(my-dependency)

Normally, you'd now need to figure out how to get CMake to interact with your compiler toolchain of choice. This might involve running from a special command shell (i.e. Visual Studio), setting environment variables, relying on the built-in auto-detection, 'kits', et-cetera. It's a mess.

However, with Zig available on the PATH, we can create a file we'll call toolchain.cmake:

set(CMAKE_C_COMPILER zig cc)
set(CMAKE_CXX_COMPILER zig c++)
set(CMAKE_C_COMPILER_TARGET ${TARGET})
set(CMAKE_CXX_COMPILER_TARGET ${TARGET})

And then add a single line to our main project file, just above the project() statement:

set(CMAKE_TOOLCHAIN_FILE "${CMAKE_CURRENT_SOURCE_DIR}/toolchain.cmake")

This tells CMake: don't rely on any system configuration. Use the compiler I've specified, and nothing else.

We can then build the project in the usual way, but specifying whatever target we like:

cmake -S . -B build -G Ninja -DTARGET=arm-linux-musleabihf
cmake --build build

And that's all there is to it. zig cc ensures that LibC headers are present if needed, and since it's just Clang underneath, all the options that CMake passes in are well understood. We've just taken hours of awkward setup and debugging and replaced it with 5 lines of configuration.

Just to make this really clear, this applies not just to cross-compiling, but native compiling as well. As in, your development system setup for any and all targets is now just:

sudo apt install cmake ninja-build
sudo snap install zig --classic --beta

Likewise for Windows, you only need to download binaries of Zig, CMake, and Ninja, and put them on the PATH. No installers, no Visual Studio, no eternally-cursed Developer Command Prompt.

If that's not magical, I don't know what is.

A long-winded Primer to the JavaScript Packaging Situation

Hamish Milne — Sat, 22 Jun 2024 11:20:35 +0000

If you've been dealing with the JavaScript ecosystem for more than 5 minutes you're probably aware that the situation with modules, build systems, runtimes, bundlers, and packaging code for distribution is kind of a mess right now. The aim of this guide is to cut through some of the bullshit, give a bunch of somewhat opinionated recommendations and end with a simple decision tree to let anyone determine what to write, what to distribute, and how to do it.

First off, some key assumptions (aka, my opinions):

0. These are (just) my opinions

I believe, quite strongly, that the approach outlined here should be used for all new JS projects. There will, inevitably, be certain niche or legacy situations where you will have to, for example, use UMD modules or TypeScript's 'import require' syntax. I'll try to note any examples where I've personally come across them, and of course feel free to comment your own if you have any.

1. We are using TypeScript

Seriously, at this point there is precious little reason to not use TypeScript as your source language. The tooling is robust, the language itself is excellent, there's wide support in the ecosystem, and it's got first-class support in Bun (i.e. the probable NodeJS replacement).

If you really love runtime errors and want to see more of them in your code, you can set "strict": false in tsconfig and just... write JavaScript. Then, to be serious for a moment, you can gradually adopt type annotations as your project grows - and you'll have the systems in place to support it.

In a few places, I'll be mentioning ts-node, which is a module for NodeJS that adds a special 'loader' for TypeScript files. In essence, if you run Node with the right options (node --loader ts-node/esm typically) you'll get pretty robust, seamless TypeScript support - no need for separate build steps. If it's at all applicable, you should use it.

With all that said, I highly recommend against distributing raw TypeScript. At the end of the day, TypeScript is still an additional dependency that we don't want to impose on library consumers. There's also a plethora of configuration options that may cause distributed TypeScript to simply not work in a given downstream project.

2. We don't care about UMD, AMD, SystemJS etc.

For those not in the know, there are two 'module systems' (i.e. ways of telling the runtime what values to import and export from a given file) in common use right now. The normal, standard-since-2015, fully-featured, widely-adopted 'ECMAScript Module' syntax, or ESM for short:

import { foo } from "foo";

export async function bar() {
    const { baz } = await import("baz");
}

and the old, outdated, legacy system used by NodeJS, called CommonJS (or CJS):

const { foo } = require("foo");

function bar() {
    const { baz } = require("baz");
}

module.exports = { bar }

There were once a whole plethora of alternative module systems: UMD, AMD, and SystemJS, but these are now historical footnotes. The only time you will encounter them is if someone has packaged a library using them (God help you!). You should never, in the current year, distribute code that uses them.

And to be absolutely clear: the only reason we are even mentioning CommonJS is because the NodeJS implementation of ESM has, at the time of writing, several critical compatibility issues that are unlikely to be resolved in the near future, which will be explained in the relevant sections later on.

And if you're using Bun, rejoice - you can mix and match both systems in the same file:

import { foo } from "foo";
const { bar } = require("bar");

Not that you would want to, of course, but it's probably the best solution to the problems we'll be discussing.

3. We are aiming for simplicity, compatibility, and robustness

We want our emitted JS code to be as close to our TypeScript as possible.
We don't want superfluous distribution files, or long build processes, or a load of extra dependencies.
We want the code to work in every situation - as long as it doesn't conflict with the first two goals.

The problem with ESM in NodeJS

NodeJS uses CommonJS internally, and probably always will. Node's ESM support uses a special API hook called a 'loader', which is responsible for essentially translating your ESM files into CJS under the hood.

There are, of course, several problems with this approach. Notably, you can't really configure it so that multiple loaders act on the same file, as a user. ts-node has to use a special implementation to get ESM compatibility on Node, which is why you need ts-node/esm. Likewise for Yarn's PnP system (though in that case it's the default). If you want to use both together, you're SOL.

The second thing to remember is that ESM imports are inherently asynchronous, with top-level await and so on, whereas Node's require is a synchronous call. Evidently, Node's execution pipeline is inflexible enough that this presented an enormous problem for the devs, and the result: you can't require ESM files in Node. The reverse isn't true, fortunately - you can import CJS files no problem (well, some problems, but that's for later).

In short:

import foo from "./foo.cjs"; // Works!
const { bar } = require("./bar.mjs"); // Error!

The upshot is if you're writing apps and tools, you can, and should run ESM natively. If you transpile to CJS, you won't be able to use ESM-only dependencies (without using await import, which is a whole other can of worms and will probably make your APIs unusable).

But for libraries, where you have no idea if they'll be imported from an ESM or CJS context, you'll need to distribute your package as CJS by transpiling your ESM code (with tsc, ideally). Note that in this case, ESM-only dependencies will not work but they are, fortunately, quite rare.

In all cases I've found, this transpiled CJS code will work just fine in bundlers like Webpack and Vite - in the latter case it'll get converted back to ESM, funnily enough. The sole exception is Deno, which does use ESM internally and won't like your CJS code. If you want to support Deno, bundlers, and Node at runtime, there's no way around it: you'll have to multi-target CJS and ESM. But we'll get to that later.

In conclusion: Node's ESM support sucks. At the very, very least, it's now officially 'stable'; the 'experimental' warnings you might remember in the Node CLI have gone away as of v20.

Configuration recommendations

I'm going to be covering a few different types of JS project:

Frontend/web apps, IIFE bundles
Backend/server-side apps, tools, and local scripts
Libraries
- Some of which may need to run on NodeJS

Each of these has different requirements, so keep them in mind as you follow along.

A note on import statements and file extensions

What's the correct way to write imports of local files? It's easy, right?

import { foo } from "./foo"; // Works... sometimes!

The thing to remember is that module resolution is not standardised across consumers. All you're doing is telling whatever program is consuming your code - tsc, Node, Webpack, etc. - to please give you the module object for ./foo, whatever that is. This is how when you import react, Node will look inside node_modules and load node_modules/react/index.js or whatever.

First off, if you're importing a local file, always begin the import specifier with ./ or ../; this ensures there's no ambiguity with packages, and in my experience, makes tooling much more reliable. Don't try to mess with tsconfig to add multiple source roots, or do src/foo or somesuch - you'll end up tying yourself in knots and run into all sorts of errors down the line. Stick to relative paths.

Secondly, you need to be extremely cognizant of file extensions. This applies both to local files, and when deep-linking into a package to import a particular file (but not to directories or package roots). TypeScript, bundlers, Bun, and Node's require statements will search for the file in question, trying different file extensions in a particular order until they get a match.

Deno and Node's ESM implementation, however, do not do this. They require the file to be specified exactly, including the specific file extension to use.

Sounds reasonable - in the former cases, you aren't forbidden from specifying the extension, it's just optional. So we just need to start using import "./foo.js" and everything should still work... right?

But we aren't writing JavaScript. We're writing TypeScript. So what should we use to import a local file - .ts, or .js?

The sad reality is that no option will work in all cases.

In the IDE, you can use either: for .js, TypeScript will 'magically know' to convert this to .ts under the hood... ugh. You can also leave off the extension entirely, if TypeScript is configured to let you do this.

If you're using ts-node with type: module, or Deno with TypeScript directly, you have to use .ts because the .js files don't exist on disk (the runtime will take care of the conversion).

If you're manually transpiling to ESM JavaScript, then running on Node, you have to use .js, because of course the .ts files don't exist (and Node wouldn't understand them in any case).

If you're manually transpiling to CJS, you can use .js, or leave off the extension.

If you're using a bundler that accepts TypeScript directly, like Webpack, you'll have to use .ts because the .js doesn't exist on disk. There are probably ways to rectify this with custom resolution rules etc. but that's out of scope for now.

One 'simple' fix that comes to mind is simply rewriting import statements at build time from *.ts (or blank) to *.js. The TypeScript devs, however, have explicitly rejected this as a solution - and their reasons do seem pretty solid.

So, what's the TL;DR here?

For libraries, you should specify .js because you'll be building JS files from your TS source anyway.

For bundlers/front-end, you should leave off the extension, or use .ts. In this case you may need to set "moduleResolution": "Bundler" as described below.

For server-side that uses ts-node, you should specify .ts.

For server-side where you're pre-building your JS, you should specify .js.

`type`, `module`, and `moduleResolution`

These three fields, the first in package.json and the others in tsconfig.json, determine and are determined by the module system your compiled JavaScript will use. Despite them being three separate fields they are in fact very tightly linked, and only certain combinations will work.

type is used by both Node and TypeScript to decide what module system .js and/or .ts files use. If left unset, or set to "commonjs" it will assume they're CJS. If set to "module" it will assume they're ESM.

For libraries targeting Node and therefore emitting CJS, you should set it to commonjs.
In all other cases, use module.

The setting can be overridden on a per-file basis by using .cjs and .cts to for CJS code, and .mjs and .mts for ESM code. This is useful, but these extensions are not as widely-supported and there are pitfalls associated with using them.

moduleResolution determines how TypeScript resolves type information from an import operand, and ensures that these operands are specified correctly.

For server-side, we need to use Node16, since this allows us to import both ESM and CJS and will give compile errors when we fail to specify a file extension.
For libraries building to CJS, we need to use Node10, since that's the only option allowed when doing so.
For front-end, or libraries only targeting front-end, you should use Bundler, which is similar to Node16 but more permissive.

module determines which module system tsc will use in the emitted JS.

For server-side, we need to use Node16 because this is the only option when moduleResolution is correctly set for this.
For libraries building to CJS we should, of course, use CommonJS.
For other libraries, you should use ES2015.
And for front-end you can use ES2022, or otherwise the latest version.

So, what do I save my source files as?

If you follow the steps above, you should save your files as .ts, as normal.

`target` and `lib`

target tells tsc whether it needs to down-level newer JavaScript features to support older runtimes.

For any app or tool in which you control the runtime environment, you should set it to the latest: ES2022 at the time of writing.

For libraries, or code that gets distributed like a dev tool, you should start with ES2018 (essentially ES2015 but with async functions and object-spread expressions, both supported since Node v10) and only increment it as needed.

The lib option should be kept in sync with target (aside from the addition of DOM, DOM.Iterable for front-end).

`isolatedModules` and `verbatimModuleSyntax`

Turning TypeScript into JavaScript is, in most cases, quite straightforward: find any syntax that related to the type system, and delete it. This works because TypeScript was always designed to be a strict superset of JavaScript, and it's great because the JavaScript we emit is extremely close to the source code. So if sourcemaps aren't working you can still debug, and you don't need to worry about tsc introducing bugs into your code.

Of course if you're targeting CJS you need to turn all your imports into requires, but that's pretty straightforward to do, and in any case we'll deal with the pitfalls later on.

When tsc runs, it needs to load up every file in the project in order to type-check everything. If successful, it then emits the JS code for every file. Other tools, like swc, esbuild, and Vite, don't do this: they process each file individually and in parallel, without type-checking, often in response to those specific files being changed. As you can imagine this is enormously faster. You can then run tsc on a slower cadence, without emitting JS. This is known as 'SFC' for Single-File Compilation.

However, there are a couple of TypeScript language features that don't fit this pattern: namespace and const enum. I won't go into details here, but suffice it to say that these are now largely considered mistakes, and will often cause errors if your TypeScript is consumed by anything other than tsc.

To fix this, you should always enable isolatedModules in tsconfig, which makes it an error to use any of these features.

The second problem relates to importing types. The documentation goes into detail, but the short version is that, when importing, you should be explicit about importing a type, compared to a runtime value.

// Instead of this:
import { some_type, some_value } from "foo";
import { other_type } from "bar";

// Use this:
import { type some_type, some_value } from "foo";
import type { other_type } from "bar";

What's the difference between import type { foo } and import { type foo }? The first one will get removed entirely from the output, whereas the second one will become import {}. This is crucial to remember if module initialization order matters - for example if a module has any side effects (it's extremely bad practice, but... it happens).

My personal advice: if you're importing modules with side effects, use import "foo"; with a comment to explain; ideally at the very top of your program.

The setting verbatimModuleSyntax will helpfully enforce the use of import type for us, so you should absolutely enable it in tsconfig.

BUT!

When verbatimModuleSyntax is enabled, tsc will - bafflingly - refuse to emit CJS code! The documentation implies that this is a deliberate design choice, which is very silly indeed.

This means that, if we're targeting CJS, we need to do a little dance with the compiler by defining a secondary tsconfig that we'll call tsconfig-cjs.json:

{
    "extends": "./tsconfig.json",
    "compilerOptions": {
        "verbatimModuleSyntax": false,
        "module": "CommonJS"
    }
}

We'll then need to set "module": "ES2018" in the main tsconfig, so that the IDE doesn't complain. The build command will be npx tsc -p ./tsconfig-cjs.json.

LFANs, `esModuleInterop`, and `allowSyntheticDefaultImports`

Normally, when converting ESM to CJS the rules are pretty straightforward:

// src/index.ts
import { foo } from "foo";
import * as bar from "bar";
import baz from "baz";
import bat1, { bat2 } from "bat";

// dist/index.cjs
// ... Okay, it's not *exactly* this, there's normally some extra cruft, but semantically this is what you get.
const { foo } = require("foo");
const bar = require("bar");
const { default: baz } = require("baz");
const { default: bat1, bat2 } = require("bat");

Note a couple of things here:

a 'Default Import' (import baz) in ESM is the same as accessing the default property of the module object in CJS
a 'Namespace Import' (* as bar) in ESM is the same as just assigning the whole module to a variable in CJS

These are both true because the CJS export syntax is simply:

module.exports = { foo, default: bar };

This is all fine, but CJS doesn't restrict module.exports to only objects - it can be anything you like, a string, a number, a function. Indeed, a common pattern historically was to define modules like so:

// package.js
module.exports = function PackageFactory() { }

// index.js
const PackageFactory = require("package");
const myPackage = PackageFactory();

Let's call this a... Legacy Function-As-Namespace Module. LFAN for short.

Of course, it's not possible to define LFANs in ESM, but we can at least consume them: simply use import * as PackageFactory from "package".

However... this is technically invalid according to the ESM spec! ESM namespaces are immutable object-like things and must never be callable!!!1!

Therefore, transpilers like tsc, esbuild, swc, and Babel, will usually generates a lot of cruft in CJS output in order to 'fix' this 'problem', which is controlled with the esModuleInterop setting.

There are, roughly speaking, two methods used to do this:

When importing a CJS module which has no default field, set module.exports.default = module.exports, OR
Create a new, immutable object, copying over only the owned properties of module.exports, if any, and setting the value of default to module.exports itself.

tsc, esbuild, and swc use the first option, which is more efficient and compatible, since it ends up that a Default Import and a Namespace Import will provide the same value. Node's ESM runtime and Babel use the second, which is more spec-compliant, since only a Default Import will work - a Namespace Import will return an object like { default: moduleFunction }, which is probably not what you expect.

From what I can tell, only tsc allows you to turn this 'feature' off entirely, by disabling esModuleInterop.

TypeScript's allowSyntheticDefaultImports mirrors this runtime behaviour in the type system. When enabled, TypeScript relaxes its checking of import statements very slightly, such that for any module that lacks an explicit 'default' export, a 'synthetic' one is created that just aliases the whole module. Unfortunately there's no way to forbid Namespace Imports in this situation, so if you're targeting Node's ESM or Babel you just need to be careful.

import * as moment from "moment";
import moment from "moment"; // allowed when allowSyntheticDefaultImports=true

So, in general:

If you know that you have no LFAN dependencies, OR
You are only targeting CJS, and don't intend to run your source code through esbuild, swc, or Babel,

then you can (and should) disable both esModuleInterop and allowSyntheticDefaultImports. Otherwise, you will have to enable them both to prevent compatibility issues down the line, and be sure to only use Default Imports for LFANs.

Note that, for CJS emits, TypeScript will always output the following:

Object.defineProperty(exports, "__esModule", { value: true });

This just sets a flag that allows consumers that use esModuleInterop (and similar) to be a little more efficient.

Other, miscellaneous options

You should enable composite, and set your include patterns appropriately. This will also enable incremental (making your builds faster), so remember to add .tsbuildinfo to gitignore. If all your code is in some directory, e.g. src, make sure to set rootDir to that aforementioned directory.

You will probably want to enable skipLibCheck. Packages can include all sorts of TypeScript code which may or may not be valid for your current TypeScript version and compilation options, so using this saves a lot of headaches. Note you'll still have type-checking on your imports of library code - this just tells tsc to not bother checking the library declaration files themselves for correctness.

There are various other useful options like forceConsistentCasingInFileNames but these all come with sensible defaults.

And finally you should, of course, enable strict in tsconfig.

Build, run, distribute

You should use `ts-node`

For server-side apps, or local scripts, this is without a doubt the best approach. It's simple, robust, and removes whole classes of potential errors.

With everything set up as described above, you'd just run node --loader ts-node/esm ./my_file.ts. Debugging in VSCode 'just works'. I like to use the JavaScript Debug Terminal as it saves having to create launch.json.

If ts-node is all you need, you should enable noEmit in tsconfig, and use tsc as essentially a linter. This prevents JavaScript files from being created accidentally.

If you're using Yarn, remember to set nodeLinker: node-modules in yarnrc, for the reasons mentioned above about Node's ESM support sucking.

The only ts-node specific option I'd recommend is transpileOnly. You probably don't need swc, but if startup performance is really critical you might consider it.

There are of course certain situations where ts-node isn't an option. Maybe you really want to use Yarn PnP, or you're distributing a dev tool where you want to minimise your runtime dependencies. In that case...

Emitting ESM

You'll need to do this for dev tools, and for libraries which don't target Node as a runtime (or do, but have ESM-only dependencies).

With everything configured as above, just set outDir, and run tsc to build. If you take a look at the emitted JavaScript you'll see that it's extremely close to your source code.

For an app, you can then run node ./dist/main.js as normal. You should explicitly set declaration to false, since you're not expecting anyone to consume your type information, and it'll speed up your build process.

For a library, you'll want to set two fields in package.json, main and types, which should point to /dist/index.js and /dist/index.d.ts respectively. This allows consumers to import your library and get the root module.

Emitting CJS

You'll need to do this for libraries that target Node, but not Deno, and don't have any ESM-only dependencies.

After following the steps described in the verbatimModuleSyntax section above, you should already have a tsconfig-cjs.json file. Simply run npx tsc -p ./tsconfig-cjs.json and you'll have your output.

Multi-targeting CJS and ESM

You'll need to do this if you want to target both Node and Deno.

Since CJS is the 'lowest common denominator' it will be your primary target, with ESM being built specifically for Deno. Follow all the steps above assuming you're targeting CJS alone.

In tsconfig-cjs.json, set your outDir to dist/cjs or similar. Set main and types in package.json accordingly. Then in your main tsconfig.json, set outDir to dist/esm or similar.

We can then run npx tsc for ESM, and npx tsc -p ./tsconfig-cjs.json for CJS.

Remember: since we've set type: commonjs, if you try to import the contents of dist/esm in Node you'll get an error. Of course there's no reason to do this when the CJS target works just fine and is more compatible in any case. The only consumers of this target will be those that outright do not support CJS (i.e. Deno).

Newer versions of Node use a field called exports in package.json, which is supposed to allow CJS contexts to get a CJS module, and ESM contexts to get an ESM module. I don't see much value in this when we know CJS works all the time; it just seems like it'll make testing harder.

Things to avoid

Don't use `export =` or `import foo = require`

Before TypeScript supported ESM, there was some special syntax to define a CJS module's imports and exports:

// .ts
import foo = require("foo");
export = { bar };

// .js
const foo = require("foo");
module.exports = { bar };

... Yep, that's really all it does. Honestly I don't see the point.

In any case, this syntax is still available today, and it's the only way to define a 'raw' CJS module in TypeScript, rather than an ESM module which is then converted to CJS. As you might imagine, you can only target CJS when using this syntax.

Furthermore, though the documentation doesn't call this out, there's no way to import or export type-only declarations (type and interface) with this syntax.

There are exactly two reasons why you'd want to use this rather than ESM:

You really hate the presence of Object.defineProperty(exports, "__esModule", { value: true }); in the output (with this syntax it won't be emitted)
You want to define an LFAN, so you need a way to directly control the value of module.exports

Neither of these are very good reasons.

Don't bundle libraries for distribution

Some library authors follow the practice of bundling some or all of a library's local files and/or package dependencies into the output. There is really only one valid purpose for doing this, and that's when you're creating a bundle for legacy web development - the sort where you write <script> tags to include your dependencies then manually bash out the JavaScript.

This is, of course, a very outdated technique and should be discouraged at every opportunity in favour of just... using NPM. However, if it does become necessary, you need to create a global-setting IIFE bundle, as with Webpack's type: global or esbuild's global name. This bundle can be distributed in addition to a standard NPM package. You should under no circumstances use something like UMD; CommonJS consumers should always use packages.

In all other cases, there is absolutely no reason to do this. It complicates dependency management, makes debugging harder, and requires additional tooling for no real benefit in kind.

Be extremely careful with `await import`

The await import syntax is ESM's method of defining a 'dynamic import' - that is, an import that happens during the program flow, rather than immediately on startup.

The most common use of this syntax is for 'code splitting' on the front-end: JS bundles tend to be rather large, so this allows you to lazy-load parts of your app to make your initial page load faster (the bundler is responsible for figuring out how this works underneath).

The other use case is specific to NodeJS. The Node runtime provides await import in ESM contexts... but also in CJS contexts. And, unlike require, you can import any kind of module using it, both ESM and CJS. So the upshot is that, under Node, this is the only way to import an ESM module into a CJS context.

Let's consider an example dependency diagram:

main.mjs
┣━ import 'A.mjs'
┃  ┗━ import 'B.cjs'
┃     ┗━ require('C.cjs')
┗━ import 'D.cjs'
   ┗━ await import('E.mjs')

In order to import module E in the context of module D, we have to use await import. It's a useful escape hatch for the very few cases where you have no other options.

However, you do have other options!

If you're making a server-side app, just use ESM directly as recommended earlier. There's no reason not do do this now that Node's ESM is considered stable.
If you're making a library, and you really can't get rid of this ESM-only dependency, just make your library ESM-only too. While this does just kick the problem further up the chain, your consumers can also just use ESM directly. In other words, if anyone complains, direct them to the point above.

The problem is that await import is really, really awful to use outside of code-splitting. You end up with code like this:

// there are other ways, but this is probably the most elegant
async function doImports() {
  return {
    bar: await import("bar.mjs"),
    baz: await import("baz.mjs"),
  }
}

// note that all our exports need to be async functions now.
// want to export sync functions, or constants? You're SOL.
export async function myFunc() {
  const { bar } = await doImports();
  // etc.
}

This is, to put it lightly, horrible.

Don't downlevel to ES5

ES2015, aka ES6, has been standardised since (as the name implies) 2015. NodeJS has been 99% compliant since version 8. Any browser version released after 2016, eight years ago, fully supports it.

The only reason to target ES5 is to support ancient, insecure, unsupported browsers like IE11. If this applies to you, I understand that it's unlikely to be by choice, so you have my sympathies :)

A note on unit testing

Traditionally, the received wisdom for TypeScript unit testing was to use ts-jest. However, Jest is kind of enormous and brittle with a huge amount of configuration for some reason, and in my experience getting both TypeScript and ESM to work with Jest really quite painful.

For that reason, I highly recommend Vitest. It's compatible with Jest's API and has both TypeScript and ESM support out of the box.

If you really have to use Jest for some reason, you should target CJS and save yourself a lot of headaches.

In summary...

In all cases

Install typescript as a dependency
Save your files as .ts
Write normal TypeScript using the ESM syntax
When importing a local file, always use a relative path starting with ./
In tsconfig:
- Set strict to true
- Set isolatedModules to true
- Set verbatimModuleSyntax to true
- Set skipLibCheck to true
- Set composite to true, making sure to set include and rootDir

Frontend/web apps (or anything using a bundler)

Refer to your specific bundler (e.g. Webpack, Vite, esbuild) for rules about file extensions in import statements
In package.json, set type to "module"
In tsconfig:
- Set module and target to "ES2022" (or newer)
- Set moduleResolution to "Bundler"
- Set allowSyntheticDefaultImports to true
- Set noEmit to true
Let the bundler consume your TypeScript directly

Libraries that are only used by Web apps, Bun, Deno etc., or target Node but have ESM-only dependencies

Use .js file extensions when importing local modules
In package.json:
- Set type to "module"
- Set main to "./dist/index.js"
- Set types to "./dist/index.d.ts"
In tsconfig:
- Set module and moduleResolution to "Node16"
- Set target to "ES2018" (or ES2015 if absolutely required)
- Set lib to the smallest required set of libraries
- Set allowSyntheticDefaultImports to true
- Set outDir to dist
- Set sourceMap to true
Run npx tsc to build

Libraries that may be used by the NodeJS runtime

Use .js file extensions when importing local modules
In package.json:
- Set type to "commonjs"
- Set main to "./dist/index.js"
- Set types to "./dist/index.d.ts"
In tsconfig:
- Set module to "ES2015"
- Set target to "ES2018" (or ES2015 if absolutely required)
- Set moduleResolution to "Node10"
- Set lib to the smallest required set of libraries
- Set outDir to dist
- Set sourceMap to true
- Consider setting esModuleInterop and allowSyntheticDefaultImports to false, making sure to use Namespace Imports for LFANs
Create a file tsconfig-cjs.json, extending the primary one, and in it:
- Set verbatimModuleSyntax to false
- Set module to "CommonJS"
Run npx tsc -p ./tsconfig-cjs.json to build

Libraries that may be used by the NodeJS runtime, but also want to target Deno

Use .js file extensions when importing local modules
In package.json:
- Set type to "commonjs"
- Set main to "./dist/cjs/index.js"
- Set types to "./dist/cjs/index.d.ts"
In tsconfig:
- Set module to "ES2015"
- Set target to "ES2018" (or ES2015 if absolutely required)
- Set moduleResolution to "Node"
- Set lib to the smallest required set of libraries
- Set outDir to dist/esm
- Set sourceMap to true
- Set allowSyntheticDefaultImports to true
- If you have any LFANs as dependencies, or may do in future, set esModuleInterop to true; otherwise false
Create a file tsconfig-cjs.json, extending the primary one, and in it:
- Set verbatimModuleSyntax to false
- Set module to "CommonJS"
- Set outDir to dist/cjs
Run npx tsc -p ./tsconfig-cjs.json to build the CJS target, and npx tsc to build the ESM

Server-side apps and local scripts where you can use `ts-node`

Use .ts file extensions when importing local modules
Install ts-node as a dependency
If using Yarn, set nodeLinker to node-modules
In package.json, set type to "module"
In tsconfig:
- Set module and moduleResolution to "Node16"
- Set target to "ES2022"
- Set verbatimModuleSyntax to true
- Set allowSyntheticDefaultImports to true
- Set noEmit to true
Start the app with node --loader ts-node/esm src/main.mts

Dev tools, server-side apps that can't use `ts-node`

Use .js file extensions when importing local modules
In package.json, set type to "module"
In tsconfig:
- Set module and moduleResolution to "Node16"
- Set target to "ES2022"
- Set verbatimModuleSyntax to true
- Set allowSyntheticDefaultImports to true
- Set outDir to dist
- Set sourceMap to true
Use npx tsc to build your JavaScript before running your app as normal

Making a better 2D asset pipeline

Hamish Milne — Tue, 21 Dec 2021 20:54:10 +0000

I hate image formats.

Maybe it's an emotional reaction to hours upon hours of searching for obscure specifications, finding bizarre proprietary blobs in supposedly standardised files, wondering if the tool I'm running is even doing anything, and having my (fairly high end) PC completely run out of memory several times - but it's an understandable one, I think.

As a warning, this post is going to be pretty long, boring, and rant-y. If that's not for you, feel free to skip it and go look at some cool WebGL demo, but if you like this sort of thing then read on!

A bit of background.

After a long day of coding, I like to sit down for a nice evening of coding - specifically, working on a to-be-announced game project along with a couple of long-time friends. Naturally, I've ended up in the architect's role, which means I'm responsible for our asset pipelines, and in the (primarily) 2D game we're making, that means a lot of sprites.

In our game, each character bust is built from a whole load of individual pieces. Arms, feet, noses, shins, eyebrows, breasts, clothes - everything can be mixed and matched, like a space-age Mr. Potato Head. This is obviously more difficult to set up in the short term, but it allows us to quickly create expressive, high-quality character busts that hopefully don't look too creepily alike.

Our artist has a particular creative process that I make no claim of understanding, but the important thing here is the file structure, which looks like this:

For Photoshop neophytes (like myself): the folder-looking things are exactly that, and are called 'groups'. Groups can be infinitely nested. The leaf nodes of this tree structure are 'layers', and they contain image data. This could include vector shapes, text, or other effects, but for our purposes we can assume it's raster (bitmap) data.

Each of these serves a different purpose: the lowest level layers are where the actual work gets done - so, we might have a layer for some line art, then another one for some shading. Directly above that (usually!) we have a group that collects all the layers for one 'piece', like 'SS_Ears_Kobold_Front_1'. Above that, there are groups used for organisation - gathering together the Kobold ears, all the ears, all the head pieces and so on. This allows the artist to hide the pieces that aren't relevant while working, but making them visible when he needs to check that they fit with the other adjacent pieces. There are also layers and groups used as references, temporary sketches, you get the idea.

For practical reasons, the artist splits his work into multiple PSD (Photoshop Document) files which, as we'll see later, is probably a good thing.

So! How do we get that into a format we can use in Unity?

Attempt #1: keep it simple, stupid.

The solution we first jumped to was the one built in to Photoshop: "Quick export layers as PNG". The artist need only individually mouse-select the 434 groups corresponding to the pieces, hit the aforementioned option, and generate a load of images like this:

And nicely cropped as well, how thoughtful! So I can just drop them all into Unity, and...

Hmm. Okay, maybe I can make this work...

Yeah.

In hindsight, I spent an embarrassingly high number of hours, in and out of voice calls with the artist, trying to pixel-perfectly align each piece with some 433 others, before coming to the eminently sensible conclusion that this would not do.

Attempt #2: Not enough RAM

After looking into a few possible solutions, including things like adding a nearly-transparent pixel in two corners of every layer to trick it into not cropping the images, the artist found a plugin for Photoshop that provided a slightly more configurable PNG exporter. The new exports would all be identical in size, matching the size of the canvas, with the graphic correctly positioned within it. The downside: you have to merge (combine into a single layer) each of the 434 groups before export - a process that takes several times as many clicks as the previous one. Safe to say, he was not pleased.

I, however, was ecstatic!

...at least until I tried to shift-select all the images in Unity. After minutes of waiting for the unresponsive editor, my PC locked up, having completely run out of memory.

The problem is the sheer size of the images: 3500x5200 (enough to stay sharp on a 4k monitor). The size on disk isn't much changed with the addition of all this empty space--PNG provides pretty good lossless compression after all--but in Unity (both in the editor, and at runtime) things aren't so simple.

In order to display a texture on screen, a desktop GPU requires it to be ultimately stored in one of a few different formats. For RGBA images like ours, we've got essentially two options: raw, uncompressed data at 32 bits per pixel, or the DXT5 fixed-rate compression at 8 bits per pixel. Since blank pixels now cost just the same as filled ones, it adds up to 70MiB or 17MiB for a single image. All together it's 30GiB or 7.5GiB, and that could be for both system and GPU memory depending on the exact operation. The cost when selecting assets in the editor is probably twice that due to internal buffering.

As an aside, crunch compression is a bit of a red herring here. While it does reduce the size on disk, it needs to be expanded back to regular DXT when it's first displayed, so it won't solve our memory issues by itself.

Now as much I love wasting my development headspace with caveats like "Don't ever touch Ctrl+A", this wouldn't do for the eventual players of the game, who would need something like a 2080 Ti in order to run our 2D RPG. If you're familiar with Unity you've probably just yelled the solution at the screen: sprite atlases! With one of these, we can pack all our sprites together into as few textures as possible, cropping out the empty space and even nestling them in each others' concave spaces, while preserving their positional data.

A tip: you can drag a folder into the 'packable objects' list, and Unity will recursively add all the sprites within it to the atlas. Saves a lot of clicking and dragging!

After adding the folder, I hit the "Pack Preview" button, and--you guessed it--got another out of memory crash. Perhaps I could create multiple atlases, limited to a few dozen pieces each, but that would compromise memory efficiency, download size, and draw call count, all because I insisted on having a load of empty space around each sprite. And the problem would only get worse; we projected our final sprite count to be at least 10 times what we currently have.

Attempt #3: Process and reprocess.

So we can't have cropped sprites because we lose the positional data, and we can't have uncropped sprites due to memory issues. But since the only thing the empty space provides us is an offset coordinate, perhaps we can extract this information from the raw images, and store them cropped in the project?

[MenuItem("Tools/Trim sprites")]
public static void TrimSprites()
{
    // For each selected Texture asset...
    var textures = Selection.objects.OfType<Texture2D>();
    foreach (var texture in textures) {

        var path = AssetDatabase.GetAssetPath(texture);
        var importer = (TextureImporter)AssetImporter.GetAtPath(path);

        // Disable compression, and make it 'readable';
        // this allows us to get a pointer to its data later on.
        importer.isReadable = true;
        importer.maxTextureSize = 8192;
        importer.textureCompression = TextureImporterCompression.Uncompressed;

        // Re-import the asset.
        importer.SaveAndReimport();

        // Find the bounds of the graphic by looking for non-transparent pixels
        var rect = GetNonTransparentBounds(texture);

        if (rect.min != Vector2Int.zero || rect.size != new Vector2Int(texture.width, texture.height)) {

            // Calculate the new sprite pivot based on the computed bounds
            var sourcePivot = importer.spritePivot;
            var sourcePivotPixels = sourcePivot * new Vector2(texture.width, texture.height);
            importer.spritePivot = (sourcePivotPixels - rect.min) / rect.size;

            // Copy the graphic to a new, correctly-sized texture
            var trimmed = new Texture2D(rect.width, rect.height, TextureFormat.ARGB32, false);
            trimmed.SetPixels(texture.GetPixels(rect.x, rect.y, rect.width, rect.height));
            trimmed.Apply();

            // Write the texture to the original file path in the PNG format
            File.WriteAllBytes(path, trimmed.EncodeToPNG());
        }

        // Undo the previous changes to the import settings
        importer.isReadable = false;

        // Re-import the asset, again.
        importer.SaveAndReimport();
    }
}

These new cropped, transformed sprites will happily pack into an atlas, while staying correctly aligned, so... yay?

Well as you might imagine, this import/calculate/re-import process isn't exactly quick, and if the assets need updating we'd need to reset the pivot point and re-run the process. Plus, if we needed recover a layered PSD file from these images, it's more difficult to do so (though admittedly not impossible), and after all of this we still don't have the group structure of the original file. It "works", but surely, surely, there's a more sustainable solution out there.

Attempt #4: Just use PSD bro!

In the past, Unity had very poor support for layered image files, at best flattening the entire image into a single texture. This is rapidly changing, however, with the addition of the 2D PSD Importer package. This adds a scripted importer which takes the original image, extracts all the layers, automatically crops and packs them into an atlas (not as efficiently as a regular sprite atlas, but good enough to save on memory use in the editor!), while keeping the group structure. You can even share skeleton rigs between different sprite groups, and (in the beta version) individually enable or disable layers in the import settings.

The artist, however, was sceptical. In order to get sprites for the pieces (instead of each art layer) he would still need to manually merge all the piece layer groups like he does now (the importer can sort of do this, but it's not very flexible), but with the added downside of having to upload a much larger file: the uncropped PNGs total about 20MiB, where the PSD was around 250!

A bizarre limitation of scripted importers is that's impossible to handle any file extension (like .psd) that Unity handles natively - even if the author of said importer is Unity themselves. Thus, the 2D PSD importer actually imports PSB files - a very similar, but much less well-supported format. Before you send the devs any strongly-worded letters though, you can simply rename your .psd files to .psb and it'll work fine (a feature that remains undocumented at the time of writing, naturally).

I persuaded the artist to send me his work file, and, with reckless curiosity, dropped it into Unity, which spun on the importer for about half an hour before crashing (probably due to out of memory, but I was disinclined to confirm this by trying again). Given the sheer number of art layers I'm not too surprised, but in any case I'd have to rule out postprocessing the file in Unity.

Attempt #5: In a tiff.

As much as Adobe might pretend otherwise, PSD isn't the only layered raster format in existence. The TIFF format supports multiple layers (called 'pages' or 'directories'), and both Photoshop and GIMP can save a TIFF file that they claim preserves layer information.

Since I'm a cheap bastard I don't have a Photoshop license, so I used GIMP to start with. Exporting to TIFF gives you some extra settings, which I filled out like so:

The whole process was rather quick, just a few seconds. I then opened the exported file, and got some more settings:

And here's the layer structure that resulted:

So, it looks like the TIFF export process had individually merged each top-level layer group. This was... not as helpful as I'd hoped, but it's something! The artist would need to un-structure his work, moving each piece group to the top level, but I theorised that it would be much faster than manually merging and selecting each of them.

Of course, Unity won't handle these layered TIFFs in a useful way, so I had to make a scripted importer of my own!

// Remember: Unity won't let us handle '.tif' or '.tiff'!
[ScriptedImporter(1, "tif2")]
public class MyTiffImporter : ScriptedImporter
{
    public override void OnImportAsset(AssetImportContext ctx)
    {
        // This uses BitMiracle.LibTiff.NET from nuget
        using (var tif = Tiff.Open(ctx.assetPath, "r"))
        {
            var textures = new List<Texture2D>();
            var pivots = new List<Vector2>();

            int maxWidth = 0, maxHeight = 0;

            // For each 'page' within the TIFF file...
            do {
                // Get some metadata (width, height, name)
                var width = tif.GetField(TiffTag.IMAGEWIDTH)[0].ToInt();
                var height = tif.GetField(TiffTag.IMAGELENGTH)[0].ToInt();

                maxWidth = Mathf.Max(maxWidth, width);
                maxHeight = Mathf.Max(maxHeight, height);

                if (tif.GetField(TiffTag.PAGENAME) == null) {
                    Debug.Log("No page name");
                    continue;
                }
                var name = tif.GetField(TiffTag.PAGENAME)[0].ToString();

                // Read the 'raster' (pixel data)
                var raster = new int[width * height];
                tif.ReadRGBAImage(width, height, raster);

                var bounds = GetNonTransparentBounds(raster, width, height);
                // Skip all-transparent pages
                if (bounds.width <= 0 || bounds.height <= 0) {
                    continue;
                }

                // Calculate the page's X/Y offset
                var xres = tif.GetField(TiffTag.XRESOLUTION)[0].ToDouble();
                var yres = tif.GetField(TiffTag.YRESOLUTION)[0].ToDouble();
                var xpos = tif.GetField(TiffTag.XPOSITION)?[0].ToDouble() ?? 0;
                var ypos = tif.GetField(TiffTag.YPOSITION)?[0].ToDouble() ?? 0;
                var pageBounds = new RectInt(
                    (int)(xpos * xres),
                    (int)(ypos * yres),
                    width,
                    height
                );

                // Calculate the pivot based on the page's position and calculated bounds
                var srcPivot = (new Vector2(0, 1)*pageBounds.size + new Vector2(-1, 1)*pageBounds.position)/pageBounds.size;
                var pivot = ((srcPivot * pageBounds.size) - bounds.min) / bounds.size;
                pivots.Add(pivot);

                // Create a new texture for the cropped image
                var tex = new Texture2D(bounds.width, bounds.height, TextureFormat.RGBA32, false) {
                    name = name,
                    alphaIsTransparency = true
                };

                // Copy the pixels from the raster into the texture
                var data = tex.GetPixelData<int>(0);
                // data.CopyFrom(raster);
                for (int i = 0; i < bounds.height; i++) {
                    for (int j = 0; j < bounds.width; j++) {
                        data[i * bounds.width + j] = raster[(bounds.x + j) + (bounds.y + i) * width];
                    }
                }

                textures.Add(tex);

                // ReadDirectory returns false when there are no more pages to read
            } while (tif.ReadDirectory());

            // Create a new texture for the combined image
            var atlas = new Texture2D(64, 64, TextureFormat.DXT5, false) {
                alphaIsTransparency = true
            };

            // This function packs the textures somewhat loosely, but good enough for now!
            var rects = atlas.PackTextures(textures.ToArray(), 0, 4096, true);
            EditorUtility.CompressTexture(atlas, TextureFormat.DXT5, TextureCompressionQuality.Best);

            ctx.AddObjectToAsset("atlas", atlas);
            ctx.SetMainObject(atlas);

            for (int i = 0; i < textures.Count; i++) {
                // Add the Sprite to the asset
                var sprite = Sprite.Create(atlas,
                    rects[i].TransformSpace(new Rect(0, 0, 1, 1), new Rect(0, 0, atlas.width, atlas.height)),
                    pivots[i], 100);
                sprite.name = textures[i].name;
                ctx.AddObjectToAsset(sprite.name, sprite);
            }
        }
    }

    public static RectInt GetNonTransparentBounds(int[] raster, int w, int h)
    {
        int x = w;
        int y = h;
        int maxX = 0;
        int maxY = 0;
        for (int i = 0; i < w; i++)  {
            for (int j = 0; j < h; j++) {
                int c = raster[i + j * w];
                int alpha = (c >> 24) & 0xff;
                if (alpha != 0)
                {
                    if (i < x) x = i;
                    if (i > maxX) maxX = i;
                    if (j < y) y = j;
                    if (j > maxY) maxY = j;
                }
            }
        }
        return new RectInt(x, y, maxX - x, maxY - y);
    }
}

It's rough, but it sort of works!

More importantly, though: how easy would it be for our artist to create a valid file? At first it seemed the answer was 'extremely' - as simple as Save As -> TIFF - but when I opened the file in GIMP it seemed to only have a single layer.

Opening the file in the very retro-looking TIFF Inspector gave me this:

One directory (i.e. layer), and a lot of 'privately defined' tags. Using a very helpful reference, we find that most of these tags are irrelevant--colour profiles, thumbnail data and suchlike--but tag 37724 seems to have some proprietary Photoshop-related data, which is corroborated by the Photoshop TIFF spec.

Hang on, Photoshop TIFF spec? Yeah, the TIFFs that Photoshop creates are, for our purposes, totally proprietary, so it's essentially a PSD file but even less well supported. Great! Apparently ImageMagick has support for getting the layer data out of this 'variant', so if you have a bunch of files in the format already, you can still make use of them.

I could get the artist to open his work file in GIMP and go through the export process there, but by this point it seemed like a bit of a hassle for not much benefit.

Attempt #6: Can't I just ZIP a bunch of PNGs together or something?

Before I resorted to making my own format out of chewing gum and string, I thought I'd have a quick browse of the other layered raster formats out there, just to see if there were any other options - and wouldn't you know it, there are! The OpenRaster format (.ora) is an open standard, is supported in GIMP, and is literally a ZIP of PNGs, with an XML file describing the layer structure - groups and all:

So it appeared (foreshadowing!) that OpenRaster was a good candidate for our 'final' asset format. However, we still had the problem of how to go from our many-layered PSD work file to a few-layered OpenRaster file. Merging all the layers was still a manual process, and I have to confess I wouldn't be happy if I had to do several thousand clicks just to get my art in a nice format for the programmers.

So I made a plugin!

(define (merge-and-optimize-recursive image items)
    (for-each (lambda (item)
        (define merged-layer item)
        (if (= TRUE (car (gimp-item-is-group item))) (let* (
            (children (vector->list (cadr (gimp-item-get-children item))) )
        )
            ; If any children are not groups, merge the item
            ; Otherwise, recurse.
            (if (= 0 (foldr * 1 (
                map car (map gimp-item-is-group children)
            ) ) )
                (set! merged-layer (car (gimp-image-merge-layer-group image item)))
                (merge-and-optimize-recursive image children)
            )
        ) )
        ; Auto-crop the (possibly merged) layer
        (if (= FALSE (car (gimp-item-is-group merged-layer))) (let* ()
            (gimp-image-set-active-layer image merged-layer)
            (plug-in-autocrop-layer RUN-NONINTERACTIVE image merged-layer)
        ) )
    ) items )
)

(define (script-fu-merge-and-optimize image layer)
    (gimp-image-undo-group-start image)
    ; The final assets will be in 8-bit RGBA, so convert the image to that if needed.
    (if (not (= PRECISION-U8-GAMMA (car (gimp-image-get-precision image))))
        (gimp-image-convert-precision image PRECISION-U8-GAMMA)
    )
    (merge-and-optimize-recursive image (vector->list (cadr (gimp-image-get-layers image))) )
    (gimp-image-undo-group-end image)
)

(script-fu-register
    "script-fu-merge-and-optimize"
    "Merge layer groups and optimize"
    "Merge layer groups and optimize"
    "Hamish Milne"
    "Hamish Milne"
    "2021"
    "*"
    SF-IMAGE "Image" 0
    SF-DRAWABLE "Layer" 0
)
(script-fu-menu-register "script-fu-merge-and-optimize" "<Image>/Image")

... Or, more specifically a 'Script-Fu Script'. The -ahem- 'code' above is Scheme, specifically TinyScheme, GIMP's scripting runtime of choice. It's also possible to use Python 2.7 (ugh), or compile an executable from scratch, but Scheme is a lot less painful for simple scripts like this one.

There's a few things to note here:

The input files will generally use high-precision colours for more accurate composition while editing. Before everything, I change the image precision to 8-bit gamma encoding, since this is what will ultimately be used by Unity when importing; skipping this step will result in needlessly large output files.
My heuristic for whether to merge layers is somewhat specific to our pipeline. I merge the groups directly above the leaf layers and leave the rest alone.
I run the Auto Crop function on every layer, which cuts it down to the smallest rectangle that encloses the graphic. This obsoletes my Unity-based solution, and naturally makes the output files smaller still.

I have to say, it was certainly satisfying watching layer groups flick down the screen as the script did its thing. If you're sensitive to flashing images, though, I recommend looking away from the screen for a bit...

On exporting to the OpenRaster format, everything seemed to work! I had, effectively, a ZIP of all the cropped layers, in PNG format, along with the layer structure. That's just about everything I'd been looking for, right?

Right?

Well... For starters, Unity has zero support for OpenRaster, so I'd need to make another scripted importer. Not too difficult, since the format's so simple, but I couldn't help feeling some chagrin that I'd have to essentially re-implement all the features of the Unity PSD importer, just in a much more janky way.

Also, it took GIMP several minutes to do the export. On a more reasonably-specced machine, and with a larger file, it might push half an hour. I don't know exactly why this is the case, when exporting to TIFF or PSD takes seconds, but it's probably to do with the OpenRaster exporter being a Python plugin, where those other ones are built in to the main program.

Man, it'd be cool if we could just use PSD, huh?

Attempt #7: Seriously, just use PSD.

The main issue we had with using PSD wasn't the format itself, it was the total effort required to prepare the file for export. With the plugin I'd made, that just became a non-issue; I'd reduced the preparation time to almost nothing, and we could export in whatever format we pleased. Why wouldn't PSD do?

In fact, with all my optimizations, the PSD file exported after running the script was only about 20% larger than the equivalent OpenRaster file, and about half the size of all the un-cropped PNGs. The PSD shrunk from 250MiB to a little over 10. And this time, Unity didn't crash on the importer!

The only major caveat with this approach is the size of the intermediary atlas. Unlike regular sprite atlases, the PSD importer will create one big atlas texture per file. Unity (and indeed, most GPUs) has a maximum texture size of 16k square, even in the editor. If your sprites don't fit, they'll be shrunk until they do, and the sprite atlas won't be able to un-shrink them later on. So if the intermediary atlas looks pretty full, you might want to break up the PSD into smaller files.

Another thing to watch out for is this:

Neither GIMP nor Unity's PSD importer will perfectly handle every Photoshop feature. The effect above is caused by a mask being incorrectly applied, so if your layers are doing anything beyond being linearly composed together, it's a good idea to rasterize them just before you export.

Conclusion: What have we learned?

If you find yourself in the incredibly specific position of having a huge amount of structured, fixed-position sprites, created in Photoshop, that need to be imported into Unity, you've got a few options:

Quick export layers as PNGs:

Pro: Quick and simple
Pro: Small file size (thanks to the cropping)
Con: You lose the positional data and group structure (thanks to the cropping)

Use SuperPNG or similar to export un-cropped PNGs:

Pro: No special software needed to process the output
Pro: Keeps position data
Con: Need to manually merge each sprite group
Con: No group structure
Con: Requires post-processing in Unity
Con: Very easy to accidentally run out of memory in the editor

Use the PSD work file directly:

Pro: Easiest of the bunch for the artist
Pro: You use exactly the same file, making it easy to stay in sync
Pro: Keeps the position data and group structure
Pro: Importer made by Unity, will get new features over time
Con: Massive file size
Con: Importing takes forever and might crash if the file is too big
Con: If you have a weird layer structure, make use of masks, smart objects etc. you might still need to pre-process the file for Unity to display it correctly

Bring each sprite's group to the top level, then export to TIFF in GIMP:

Pro: Fewer clicks than merging each sprite group individually
Pro: Fairly small file size
Pro: Keeps position data
Con: No group structure
Con: Requires a custom scripted importer
Con: Multi-page TIFFs not well supported

Use a GIMP script to optimize the file, then export to OpenRaster:

Pro: Smallest file size of the lot
Pro: Automatic processing saves a lot of time
Pro: Simple format, easy to parse and use elsewhere
Pro: Keeps the position data and group structure
Con: Requires a custom scripted importer
Con: Format not well supported
Con: Takes a long time to export

Use a GIMP script to optimize the file, then export to PSD (our solution of choice):

Pro: Fairly small file size
Pro: Automatic processing saves a lot of time; quick to export
Pro: Keeps the position data and group structure
Pro: Importer made by Unity, will get new features over time
Con: Need to rasterize masks etc. for Unity to display it correctly

Recap: How we do it

For reference (or if you've just skipped to the end), here's our full 2D pipeline, step by step:

Artist setup:
- Install Photoshop (obviously)
- Install GIMP
- Copy the Scheme code above to %APPDATA%\GIMP\<GIMP version>\scripts\script-fu-merge-and-optimize.scm
While creating:
- Structure the layers as you like, but make sure that the group for each sprite only contains layers, and not other groups.
To export:
- Delete (not hide!) any layers that you don't want in the final output (references and so on)
- If you're using masks, smart objects, patterns etc., rasterize and/or merge the layers as appropriate so that only simple layers remain
- Open the PSD file in GIMP
- Run the script by going to 'Image -> Merge layer groups and optimize'
- NB, it's not necessary to make all the layers visible at this stage
- Check the results, then export the file as PSD
To import:
- Install the "2D PSD Importer" package, if it's missing
- Change the image file's extension to .psb, and copy it into the project
- Check the import settings - in particular, the texture size, hidden layers, and layer group options
- Ensure your sprites are added to an atlas before building

And... that's it! After a lot of trial and error, we've got what I think is a pretty powerful and robust asset pipeline, which hopefully won't make our artist pull his hair out every time he needs to do an export.

All the code in this post is ISC licensed. Feel free to use it if you find it useful!