Forem: Roman K

Type-safe Express validations with Zod

Roman K — Tue, 25 Apr 2023 11:46:19 +0000

TypeScript is known for its ability to catch the bugs early when writing the code, rather than letting users to find problems for us in production. But TypeScript has one downside: its checks are easily disabled by any type. When we have any types in our code, TS becomes just a wordy version of JavaScript, and it won't provide any guarantees.

I want to share a library which was inspired by the fact that all or almost all examples you can find on the internet for how to do validations with Express are not type safe, which means we can still make mistakes that aren't caught by TypeScript.

At the moment, my library only supports Zod which is easy to use and it can infer types for us.

To see the problem, try googling "express typescript zod" to see how it's commonly done.

Example from the first link:

app.post("/create",
  validate(dataSchema),
  (req: Request, res: Response): Response => {
    return res.json({ ...req.body });
  }
);

Here req.body is of type any, so we can easily break the code inside of route handler by changing a dataSchema and have a bug in production:

app.post("/create",
  validate(dataSchema),
  (req: Request, res: Response): Response => {
    // access any property, no matter how `dataSchema` looks like
    req.body.one.two.three

    return res.json({ ...req.body });
  }
);

We can use z.infer<typeof dataSchema> to let typescript know that this any type should be treated as dataSchema type. And it actually solves the problem. But this is a workaround, and we can still make a mistake, we can forget to specify this as, we can accidentally write typeof otherSchema, and it's more code to write.

Second link is express-zod-api, it looks similar to my library, but instead of validating req.params, req.query, and req.body, it offers input and output instead. This makes it very different from usual Express workflows.

The rest of google results seems pretty irrelevant, but if you try to search express zod typescript on dev.to you'll also see a lot of post of how to make it in the unsafe way. I didn't find a single example of how make it properly.

When we are making a public API interfaces, it seems to be important to return correct responses. It seems so, but again, you won't find any examples over the internet of how to validate and type-check the response type with Express.

Another issue is req.user which is commonly used for storing a current user record. No matter if you have used the middleware or not, all existing examples have it defined always.

express-ts-handler solves all of the mentioned problems:

app.post('/path/:id',
  handler({
    // middleware is setting req.user
    use: authorizeUser,

    // coerce and validate id from route path
    params: {
      id: z.coerce.number().int(),
    },
    // validate query string
    query: {
      key: z.string().optional(),
    },
    // validate request body
    body: {
      name: z.string(),
    },
    // validate route response
    result: {
      prop: z.boolean(),
    },
    // finally, route handler. It may be sync or async
    async handler(req) {
      // req.user has the exact type which was set by the middleware
      req.user.id

      // all the data is typed properly
      const { id } = req.params // { id: number }
      const { key } = req.query // { key: string }
      const { name } = req.body // { name: string }

      // TS error when trying to return extra fields:
      // return { prop: true, extra: true }

      // returned data is validated when NODE_ENV !== production
      return { prop: true }
    },
  }),
);

Let me know what you think about it!

Announcing a new TypeScript ORM

Roman K — Fri, 03 Feb 2023 20:58:29 +0000

There are already a lot of ORMs for node.js with different strong and weak sides, but still, this is not enough, there is still not a single tool to cover typical needs simultaneously:

Type safety
Flexibility
Easiness of use for more complex cases
Performance

This is why Orchid ORM was created - to answer all of these challenges!

Before creating a new ORM, I examined 5 top popular node.js ORMs and 2 query builders to find out if there is a "go to" solution I can safely pick for any project without worrying, and have concluded there is no such tool I was looking for.

In this post, I'll cover how different ORMs and query builders are responding to expectations, and what Orchid can offer in comparison.

Raw SQL vs query builders vs ORMs

But why do we need ORMs at all, isn't raw SQL or a query builder all we need?

They operate on different levels of abstraction:

Raw SQL means writing SQL manually, with no abstraction at all. This makes it hard to work with dynamic queries. Imagine you're building an endpoint for fetching a list of posts, and the client can send various parameters: how to order, filtering, pagination, and more. It is all possible with Raw SQL, but you'll end up with a much larger amount of code, that will look messy, and be vulnerable to SQL injections unless done very carefully.
Query Builder solves the problem of dynamic queries, but it doesn't abstract away relations. When you do a join on two tables, and if one record in the left table has many records on the right, this will cause duplications of the left table data. This is solvable by using GROUP BY, or by using a subquery, or by de-duplicating data on the JS side, but this is cumbersome. In addition, if the relationship is complex (multiple tables are involved), you may want to use it in various places, it causes duplicating of quite complex code parts.
ORMs solve the problems above, but they are often criticized for generating inefficient code and being not flexible, while Raw SQL and Query Builders are better at this. The goal of Orchid ORM is to prove that ORM can have all the benefits of Query Builder and ORMs combined.

Type safety

I believe that TypeScript is the best thing that happened to JavaScript. If before it was normal to be extremely careful when changing something, we had to keep everything in our head and search for all places where some object is used to update the code accordingly, now we have the luxury of TS pointing to where we need to update the code.

Ideally, ORM must be capable of:

Preserving a correct type when selecting specific columns
Knowing which relations were included
Different operations are available for different types: "lower than" and "greater than" are available for numeric types, "contains", and "starts with" are available for text types, etc.
Mixing a raw statement into a query with the ability to specify its type

Most of the ORMs always have a full record as a returned type, ignoring selecting a subset of columns. This is done by design in OOP-inspired ORMs, where you have an "Entity" and it must be always fully loaded. So for some people, this point might be not a problem, but a valid and preferred approach. But for me, an ideal ORM allows you to select only what is needed and preserve the correct types. Among all popular ORMs in node.js, only Prisma serves this purpose well.

Knowing which relations were included: slowly but steadily things are improving, and some ORMs are gaining this ability. MikroORM gained this last year, and Sequelize has an actively developed alpha version where it probably can do this.

Mixing a raw statement into a query with the ability to specify its type: as far as I know, not a single ORM can do this.

Let me share a use case to demonstrate how Orchid ORM is solving the points from above:

Imagine two tables, posts, and likes. Users can like the post.

We want to load multiple posts with specific fields (point #1).

If this is an authorized request we want to load a boolean whether the post is liked by a current user (including data from the related table, point #2).

If this request is not authorized, we don't have a currentUserId, so we're selecting a constant false instead of a related table (point #4). t.boolean() here indicates the returning type of SQL expression.

Filter posts by title containing a word to show #3.

const result = await db.post.select(
  'title',
  'body',
  {
    liked: currentUserId
      ? (q) => q.likes.where({ userId: currentUserId }).exists()
      : db.article.raw((t) => t.boolean(), 'false'),
  },
).where({
  title: {
    contains: 'word',
  },
});

The resulting type is completely inferred and equals to this:

type Result = {
  title: string,
  body: string,
  liked: boolean
}[]

Flexibility

Except they're not helping with relations, query builders are great at their flexibility, they are allowed to build complex queries with nested sub-queries.

ORMs are handling this differently:

exposing only a limited interface (Prisma, Sequelize), so they become unusable when you need more control over a query.
switch between a limited interface and a query builder (MikroORM, TypeORM). This feels like using two different libraries, switching between two different sets of limitations. Kysely is a nice query builder with good TS support, but MikroORM is using Knex instead so you're losing TS, and TypeORM has a custom query builder, less user-friendly than Knex.
Built on top of query builder (Objection, OrchidORM) - in such a way, querying feels natural and remains powerful.

Query builder-based ORM allows building the query step by step, adding pieces based on conditions. Orchid ORM example code:

let q = db.post.select('id', 'title')

if (params.search) {
  q = q.or([
    { title: { contains: params.search } },
    { body: { contains: params.search } },
  ]);
}

if (params.order === 'newer') {
  q = q.order({ createdAt: 'DESC' })
} else if (params.order === 'older') {
  q = q.order({ createdAt: 'ASC' })
}

const posts = await q

Orchid ORM has a custom query builder, specifically designed to be as TypeScript-friendly as possible. Query builder here is inspired by Knex and supports all the same query methods (and more).

To make things even cleaner, Orchid ORM has a repository feature, that allows hiding complex or repeated parts under custom methods. In the following example, search and customOrder are custom methods defined somewhere else, and you can build nice looking easily readable queries:

const posts = await postRepo
  .search(params.search)
  .customOrder(params.order)

Easiness of use for more complex cases

Let's say we want to load post records, include the post author, tags, and a few last comments, and comments should include the author. Here is how it looks with Orchid:

await db.post
  .select('id', 'title', 'description', {
    author: (q) => q.author.select('id', 'firstName', 'lastName'),
    tags: (q) => q.postTags.pluck('tagName'),
    lastComments: (q) =>
      q.comments
        .select('id', 'text', {
          author: (q) =>
            q.author.select('id', 'firstName', 'lastName'),
        })
        .order({ createdAt: 'DESC' })
        .limit(commentsPerPost),
  })
  .order({ createdAt: 'DESC' });

14 lines of code.

The same with Prisma took me 48 lines of code (source), and it looks clean and well, but requires mapping of result, as it doesn't support naming the fields as you need in the query, so we can't just load comments as "lastComments", but this may be required by our API spec.

Sequelize code for the same query (source) looks more complicated and includes inevitable dangerous type casts.

The limited interface of MikroORM and TypeORM would not be enough for this query, so if in case of using them, we would have to switch to a query builder.

But, how is it even possible with a query builder? Using MikroORM, TypeORM, Knex, Kysely for such a query would much more time and effort, resulting in something really scary to maintain. We would probably end up with one separate query per table, and then combining records on the JS side. And need to do this carefully to not introduce the N + 1 problem. Let me know if I'm wrong, I'd love to be wrong at this and to see how it could be done with a query builder properly.

Performance

Premature optimization is evil, so the performance doesn't matter as much as other characteristics, but still, in some cases, it can be critical.

The initial version of OrchidORM was built to test the idea: if Postgres can include nested resources on its end via sub-queries, and return them as JSON columns, why not a single ORM is using this? Would it be efficient to follow this way?

SQL example to demonstrate this approach: loading posts with JSON array of comments:

SELECT
  *,
  (
    SELECT json_agg(t.*)
    FROM (
      SELECT * FROM comments WHERE postId = posts.id
    ) AS t
  ) AS comments,
FROM posts

This is how OrchidORM handles relations under the hood, so all nested select queries are turned into a single SQL query.

Prisma loads each new relation as a new query, then combines results on the JS side.

Sequelize produces huge queries with joins.

OrchidORM is the fastest in comparison with other ORMs and even query builders, see benchmarks here, and other benchmarks in my other article.

Writing a model

Defining columns in OrchidORM may look familiar if you ever have used Zod, Yup, Joi, and similar. It is more compact than defining types separately (Sequelize) or using TypeScript decorators and ! signs as in TypeORM and MikroORM, and it doesn't require generating code on each schema change as in Prisma.

It's possible to convert the table schema to Zod to use for validations later.

All columns are required (not nullable) by default.

text type requires min and max arguments, so our table is protected from empty or billion chars long texts. All column types can be customized in BaseTable, this can be used to set a common min and max for all columns together.

Supported relation types are belongsTo, hasOne, hasMany, hasAndBelongsToMany.

hasOne and hasMany supports through the option for a table in the middle.

export class ArticleTable extends BaseTable {
  table = 'article';
  columns = this.setColumns((t) => ({
    id: t.serial().primaryKey(),
    userId: t.integer().foreignKey('user', 'id').index(),
    title: t.text(10, 200),
    body: t.text(100, 100000),
    favoritesCount: t.integer(),
    ...t.timestamps(),
  }));

  relations = {
    author: this.belongsTo(() => UserTable, {
      primaryKey: 'id',
      foreignKey: 'userId',
    }),
  };
}

// convert columns schema to Zod, use it later for validations
export const ArticleSchema = tableToZod(ArticleTable);

What's the catch?

only Postgres is supported at this point
not possible to have a relation between tables in different databases (to be done in the future)
though it works on demo projects, it is too green for production
though it contains all methods from Knex and inserting/updating in the style of Prisma, many features are to be done yet
it can generate migrations from the existing database, it can generate table files from running migrations. But generating migrations from table schemas (as in Prisma) or updating the database schema on the fly (as in many ORMs) is yet to be done.

Summary of OrchidORM

Type safety is a top priority
Query-builder interface enables writing complex custom queries
Creating and updating nested records is as powerful as in Prisma
Zod-like defining columns makes it simpler than the others
Cares about performance

Try it out!

I hope you could appreciate OrchidORM and try it on non-critical personal projects and share feedback, so the ORM could grow and prosper.

To speed up the setup, I added a script to do all routine preparations automatically, simply run this in a new directory and check out quickstart:

npx orchid-orm

Or you can clone this examples repo and play with the Blog API code.

Don't forget to star the project if you liked it, and share feedback.

Does it look interesting to you? Do you agree existing tools are not good enough?

React.js localStorage library showcase

Roman K — Sun, 16 Jan 2022 20:05:15 +0000

localStorage is used almost in every React project, but is it easy to manage it, do we need a lib for this and what do we have?

There are popular ways of handing auth: one way we store JWT token in localStorage, in other way we store it in httpOnly cookie which is more secure. But even in second way it still makes sense to store authorizedUserId id or at least isAuthorized boolean in localStorage to not make redundant requests to a server side later, remove it when logging out or when server responds with '401 Not Authorized' code.

If after performing authorization request we simply set a value of authorized user to the localStorage, other components won't be updated automatically, because React is not aware of state change.

How localStorage is usually handled in React

One popular option is Context API, documentation even says it's good for storing current user (here).

Another popular way is to use a state manager.

Why I'd like to avoid both of them:

boilerplate: they both requires writing much more code than we want to, yes, even with Redux Toolkit, need to write slice, add it to global store, if we are using TypeScript it's even more code since both Context and state manager cannot infer type from initial data
synchronization of two sources of truth is error prone: each change of value in the store must be reflected by setting value to localStorage, if localStorage was updated in a separate browser tab the change must be saved to the store
efficiency: Context is known to be less efficient than a state manager. With state manager you have to read, parse and validate the value at every page load when store or context is initializing. In case of global Redux it means that even if we want to handle some specific value only in some distant page of our app, it has to be read, parsed and validated to initialize our global monolithic store, no matter what page is currently open.

The simplest solution

Here is my library: react-local-storage-manager

Let's consider example of storing authToken:

first, create a separate file for a 'store', it may be named auth.store.js or as you like:

// auth.store.js

import createLocalStore from 'react-local-storage-manager'

const store = createLocalStore(
  'authToken', // local storage key
  (id) => isNaN(id) ? undefined : id, 
)

// export setter function:
export const saveAuthToken = store.set

// export getter function:
export const getAuthToken = store.get

// export hook to use a value:
export const useAuthToken = store.use

// export other useful hooks:
export const useIsAuthorized = () =>
  Boolean(useAuthToken()) // token to boolean

And now it's so easy to get, set, use:

// set token after successful authorization result:
const result = await callApiToAuthorize(loginData)
saveAuthorizedUserId(result.token)

// to keep track of authorized state in any component:
const isAuthorized = useIsAuthorized()

// get token outside of component:
const token = getCurrentUserToken()

With the library values are read, parsed and validated only when accessing them and only for the first time, second time it will get value from cache. Validation is enforced to not be forgotten, multiple tabs will be synchronized.

To see it in action check this codesandbox, one more example of shopping cart is in readme on github.

Please share you thoughts!