Forem: Gemma Black

Understanding Sequelize's 'escape' function and why ideally we should avoid using it manually

Gemma Black — Sat, 23 Nov 2024 10:32:56 +0000

So a disclaimer. As a software engineer, I'm not a security expert. I rely heavily on those who know better, especially OWASP. So all feedback is welcome to help fix any flaws in the article or improve it.

Sequelize is a very robust database ORM for Node.js. Knowing how to use it is one thing. Knowing how it works under the hood is another.

So when I was asked recently, why I had not used the escape function in Sequelize's where clause, I said, properties in the where clause are already escaped. But was I correct? Or was that an assumption? Even more, was I following the best practices for preventing SQL injections.

To answer, first, let's explore how SQL injection attacks happen.

Preparing our database

We want to prepare a database to test how a SQL injection could happen.

You can use any database you like to do this but it'll be easier if you use SQLite or Postgres as they have similar escaping syntaxes whereas MySQL is a law unto itself 😁.

Create a new SQLite database.

sqlite3 database.sqlite

Create a basic users table.

CREATE TABLE users (
    id INTEGER PRIMARY KEY,
    name TEXT NOT NULL,
    email TEXT UNIQUE NOT NULL
);

Insert some users into the database.

INSERT INTO users (name, email) VALUES ('John Doe', 'john@example.com');
INSERT INTO users (name, email) VALUES ('Bruch Wayne', 'bruce@example.com');

Select those users from the database.

SELECT * FROM users;

Now you should see two users.

id	name	email
1	John Doe	john@example.com
2	Bruch Wayne	bruce@example.com

Selecting a single user

If you have a multi-tenant application, where users can only see their data, you'll usually do lookups by a single user ID. That way, you don't expose information to the wrong person.

Query a user by their id.

SELECT * FROM users where id = 1;

Now we only see one user.

id	name	email
1	John Doe	john@example.com

Attempting a SQL injection attack

But what if we pretend to we have a SQL injection attack? Add or '1' = '1' to the end of the SQL query, and now that the last statement is always true, all users are returned.

SELECT * FROM users where id = '1' or '1' = '1';

id	name	email
1	John Doe	john@example.com
2	Bruch Wayne	bruce@example.com

Users shouldn't have direct access to our database, so how would this disastrous situation happen in reality.

Creating a Node.js app as an example

Initalise npm in a new folder.

npm init -y

Install Sequelize and its dependencies.

npm install sequelize sequelize-cli sqlite3

Create a basic index.js file which connects to the sqlite database (or your flavour of it).

const { Sequelize, DataTypes } = require('sequelize');

const sequelize = new Sequelize({
    dialect: 'sqlite',
    storage: './database.sqlite',
});

(async () => {
    try {
        // test the connection is Ok
        await sequelize.authenticate();

        // Get the argument from the input
        const id = process.argv[2];

        const [results] = await sequelize.query(`SELECT * FROM Users WHERE id = ${id}`);

        console.log('Raw query results:', results);

    } catch (error) {

        console.error('Unable to connect to the database:', error);

    } finally {

        await sequelize.close();

    }
})();

It works by taking an argument from the input in the terminal.
It runs the query.
It outputs the results.

Running a safe query against our script

Running the query produces the following output:

node index.js 1

Executing (default): SELECT 1+1 AS result
Executing (default): SELECT * FROM Users WHERE id = 1

Raw query results: [ { id: 1, name: 'John Doe', email: 'john@example.com' } ]

That successfully returned the user.

Running a SQL injection attack against our script

Repeating the SQL injection attack we tried before, the outcome shows the disaster. We can access all users when we shouldn't be able to!

node index.js "1 or '1' = '1'" 

Executing (default): SELECT 1+1 AS result
Executing (default): SELECT * FROM Users WHERE id = 1 or '1' = '1'

Raw query results: [
  { id: 1, name: 'John Doe', email: 'john@example.com' },
  { id: 2, name: 'Bruch Wayne', email: 'bruce@example.com' }
]

So how can we fix this?

First attempt trying to escape user input

Sequelize wraps the query in quotes. So we'll do just that. Replace the const id line with this.

const id = "'" + process.argv[2] + "'";

And voila. That's fixed the injection...partly.

node 02_index.js "1 or 1=1"

Executing (default): SELECT 1+1 AS result
Executing (default): SELECT * FROM Users WHERE id = '1 or 1=1'

Raw query results: []

Selecting users by id with 1 or 1=1 doesn't match anything. So nothing is returned.

But what if we were to try and inject some apostrophes to end the first statement and add our own OR statement.

node 02_index.js "1' or 1=1 or '"

Executing (default): SELECT 1+1 AS result
Executing (default): SELECT * FROM Users WHERE id = '1' or 1=1 or ''

Raw query results: [
  { id: 1, name: 'John Doe', email: 'john@example.com' },
  { id: 2, name: 'Bruch Wayne', email: 'bruce@example.com' }
]

Now the disaster returns! We can get all users again!

Escaping user inputted quotes

We need to escape the quotes as well, just like Sequelize does.

const id = "'" + process.argv[2].replace(/'/g, "''") + "'";

node 03_index.js "1' or 1=1 or '"

Executing (default): SELECT 1+1 AS result
Executing (default): SELECT * FROM Users WHERE id = '1'' or 1=1 or '''

Raw query results: []

You can see how the whole paramater is a string in the last query.

And this is what the Sequelize escape function does but is far more comprehensive, covering different sql dialects.

Sequelize escape function

Instead of doing our own escaping, it's best to leave it to libraries that are battle-tested and proven.

It is safer just to do this.

const id = sequelize.escape(process.argv[2]);

Escaping is not the best option against SQL injection attacks

OWASP puts escaping user input as last on its list of acceptable ways of preventing sql injections. It's strongly discouraged!

Why?

Potentially the responsibility is on the developer to remember to escape each and every single user input. Even using Sequelize here, if the developer forgot, then this would be a security breach waiting to happen.

What's better than escaping

Using Sequelize's query builder is better than escaping manually.

As an example, create a user model.

npx sequelize-cli model:generate --name User --attributes name:string,email:string

Add the user model to the main script and replace the raw query with the model-based one.

const { Sequelize, DataTypes } = require('sequelize');

const sequelize = new Sequelize({
    dialect: 'sqlite',
    storage: './database.sqlite',
});

const User = require('./models').User;

(async () => {
    try {
        await sequelize.authenticate();

        const [results] = await User.findAll({ 
            where: {
                id: process.argv[2]
            }
        });

        console.log('Raw query results:', results);

    } catch (error) {

        console.error('Unable to connect to the database:', error);

    } finally {

        await sequelize.close();

    }
})();

When we run our SQL injection attack again, we now get undefined. No need to use Sequelize escape, the where clause does it for us.

node 05_index.js "1' or 1=1 or '"

Executing (default): SELECT 1+1 AS result
Executing (default): SELECT `id`, `name`, `email` FROM `Users` AS `User` WHERE `User`.`id` = '1'' or 1=1 or ''';

Raw query results: undefined

And what about if we use Op.like in a where statement?

We would never do this on an ID but the principle is the same.

        const [results] = await User.findAll({ 
            where: {
                id: {
                    [Sequelize.Op.like]: '%' + process.argv[2] + '%'
                }
            }
        });

The final query shows the entire statement is escaped.

Bind parameter. Better than escaping.

Parameterised queries are great because we send the sql query separately from the data.

Note that was use a bind parameter, '%:id%'.

        const [results] = await User.findAll({ 
            where: {
                id: {
                    [Sequelize.Op.like]: '%:id%'
                }
            }
        },
        {
            bind: { id: process.argv[2]}
        });

Notice what is sent to the database.

Executing (default): SELECT `id`, `name`, `email` FROM `Users` AS `User` WHERE `User`.`id` LIKE '%:id%';

Raw query results: undefined

The database engine knows exactly what to do with the data, and how to escape it. If we send the query syntax plus the data all as one, then we have to escape it before it gets to the database.

Parameterised queries are the recommended way (according to OWASP) of making queries of any kind with user input, in the database.

Conclusion

It seems using the where statement without bind parameters doesn't follow best practices from OWASP. So that's something I need to improve there.

And thank you for reading. If I've made any mistakes, do let me know.

Reading material

16 Principles for Tech-led Start-ups as a Software Engineer

Gemma Black — Mon, 29 Jul 2024 11:46:26 +0000

After a few years at a start-up turned scale-up, learning a few hard lessons along the way, I decided to share some principles I think are universal. I’m sure there'll be exceptions, like start-ups with near-unlimited runways funded by a generous benefactor, but for the rest of us, we have only so much time before we have to call it quits.

Two goals will influence all the principles I share.

1) Achieving speed.
2) Propelling growth.

I’ll start with general principles, and then more technical ones as I go on.

Focus

Even if we're a one-person startup, our singular goal can get lost. It's even more difficult to keep focused with a multi-person team. But we have to pull in one direction.

We can't make progress by going around in circles. So there needs to be regular reminders of what the focus is for the month, week, day and even hour. It's too easy to forget our goal and get side-tracked.

And what things do we need to focus on? It depends on where we are:

Getting the first customer. E.g. Do we need to get that first feature out the door? How quickly can we iterate?
Growing to N customers. E.g. Do we need more features? How quickly can we release more?
Preventing churn. E.g. Do we need to fix bugs and improve stability?

Essentialism

"Essentialism isn't about getting more done in less time. It's about getting only the right things done" - says a review for the book, Essentialism by Greg McKeown. How is this different from focus? I believe we need to set what the focus is first, then we can make sure all our actions work towards our goal.

Greg McKeown's diagram explains his concept perfectly. It's what happens when we put all our energy into moving in one direction. We'll get further than trying to spread our attention across too many things.

What does that mean for us?

If we're trying to reach product-market fit, we're probably trying different features that resonate with customers, building things fast.

Should we worry about infrastructure?
Should we even be worrying about scaling?
Should we be worrying about the cleanest and most maintainable code?

If doing those things slow us down, and we can't even get one customer because our time is spent worrying about things that are not our core focus, then we're wasting time and resources. We're solving a problem we don't have and may never have.

Release small, release often

From a developer's perspective, large releases are downright terrifying.

So much code going into production can go all-kinds-of wrong, not necessarily at first, but in the hours and days to come. Having lots of small releases reduces the risk of introducing bugs. But there's another perspective.

When we're trying to get early adopters, even having a little tiny piece of a product in hours or days is better than having the finished super mammoth product in months or years! If we can get the product into someone's hands and get feedback, we'll learn fast from a technical point of view and from a product one. With this fast feedback, we can decide whether we should keep walking down that road to this big goal, or whether we should pivot or quit while we're ahead.

Don't waste time on developer problems

I believe in hiring tools like hiring people.

If we have a software engineer on our team, they're there to build products unique to us. So don't be afraid to allow our software engineers to leverage tools for generic problems, whether paid or free. Using these tools is almost like hiring the person who built them, but for much lower costs.

But you might think, why have a developer use tools when they should be able to build them?

For example, very few people would consider building their own payment gateway directly with banks, unless our focus is on becoming the next Stripe or Paddle!

So for instance, instead of letting our developer build a payment gateway integration from scratch to collect payments, which apparently took Stripe 6 months, we can set up a beautiful payment UI in a day or less using payment services that do it for us. The difference?

Cost using payment SaaS company = transaction costs + 1 developer day
Cost using our own developer = 120 developer days + tonnes of maintenance time!

Once we reach a scale where we're now cost optimising, then we can worry about all these other things. We may indeed have a cheaper way of handling infrastructure, like DHH did when he left AWS. But in the meantime, we as developers have to frequently question, am I working on a developer problem and is there a tool I can use now so I have time to work on business problems?

Basically, don't reinvent the wheel, at least, not prematurely. Build things that are unique to the business.

Hire tools to avoid wasting time on maintenance

Some people say real developers roll out their own authentication. But let's take authentication as an example. It's not just a user table. It's managing emails. It's handling email bouncebacks. It's SMS and 2FA if we really want to protect customer data. It's handling customer passwords and secrets correctly.

Alternatively, we can use open source tools or hire third-party tools from companies. Really, these companies could have dozens if not hundreds of developers that work on the solution they provide. When we use these tools, we're hiring these people! We're hiring their expertise.

For example, with Okta or Auth0, they keep up with security best practices and make improvements to their product all the time. If there's a security patch they need to apply, chances are, without us even knowing about it, those changes will be rolled out and working for us. Unless we're trying to be the next authentication provider, why not leverage these companies and tools?

It's the same with infrastructure, which is a hard one for me as I really like my servers. It's no problem to set up Let's Encrypt, write a GitHub Action and have full CI/CD. It takes me at most, an hour or two if that from scratch. But I also have to admit, what about DDoS attacks, and handling that? What about server patches, which will be needed at some point? What about scaling the infra to meet customer needs? These are all things I'd have to worry about by making the decision to have to maintain the thing.

Alternatively, I could just leverage infrastructure-as-a-service that does it all for me, and some of those have free tiers which is just enough to carry us through until we start getting enough users to worry about how to deal with costs at scale.

So we have a choice. We can have a developer spending weeks if not months per year worrying about these things that our customers didn't hire us for. We might make an allowance here or there, but we really have to decide if we can afford to or not.

Monitoring as QA

I've worked with some of the best QA Engineers, and they've literally prevented hundreds of bugs going into production, if not thousands. However, if we're starting out and don't have enough customers yet, hiring a great QA Engineer is not going to be our focus. So what can we do instead?

Using something like Bugsnag, an observability platform that checks for bugs and issues users experience in production means we use our monitoring as QA. As soon as there's an unhandled error, Bugsnag alerts us and gives us a digest of issues. This doesn't mean not testing as a developer, because even for a relatively simple project, I used TDD to write the core logic. However, it means not spending huge amounts of time trying to QA our product and slowing down releases. Besides, a QA Engineer would probably do a better job of testing our code anyway, especially with a pair of fresh eyes.

Write down your principles

If we're the founding engineer or CTO, we should write down our principles so that when we onboard new developers, they don't have to ask us millions of questions to make decisions. They'll have the autonomy to just move ahead with things.

This saves us time, because we share our core principles with our team, and it saves them time too. If there's no one else working with us, then it might not be a priority to care about, but there's nothing stopping us from having a quick list of 3 or 4 things in a sentence or less. For example:

Use a third-party tool with a free tier > open-source > roll out own.
Use existing tools unless one doesn't exist.
Use modular monolith over separate service.
Scale vertically before worrying about scaling horizontally.

Have transparent roadmaps

Roadmaps shouldn't just be for the tech team but for the whole company. Everyone should be able to take a look at it and know what we are working on right now, what's been done, and what's to come.

Why? Two things:

It gives our team focus. If a piece of work is in progress on the roadmap, but we're working on other things, we have to ask ourselves, why?
If we're talking to current and potential customers, it'll make the conversations a lot easier, as we'll know what's in the pipeline.

Now whether we share this roadmap with the outside world is up to us, but some companies have done it with great effect! Take Supermetrics. Their customers know their roadmap, what integrations are being built and they can even offer suggestions for tools to integrate with.

Choose what we can be productive in

I love using new tools. I can't wait to learn Zig and Rust. I want to work with Go on a product with customers, not just personal projects. I know everyone is clearly using Next.js. After years of working with Vue.js, knowing React is still the rage, I feel like I should pick it up again, but do we really need to build a single-page application? Can we use a Go or Node.js server with some HTMX and Alpine.js sprinkled on top? If we go down the React route, then we need to create an API. Do we need an API? Then we need validation on the client and the backend. Can we just have validation on the backend? Do we need two separate Git[Hub|Lab]/Bitbucket repositories? Do we need Docker? I mean, do we really? We might even ask, do we even need a server? Could we just have something like Firebase and just a static app hosted on Netlify or Vercel or something? Could we just use functions with functions as a service?

Depending on our skills and core competencies, starting with what we know means we can get productive fast. But sometimes, picking up a framework that does most of the work for us, especially if we know we'll need it will make our life a lot simpler, if we're willing to give ourselves a small amount of time to learn it upfront. I did this with HTMX like David Guillot did. But if we have one day to get that feature or part of that feature released, and we need to use what we know, that's probably the simplest and most productive thing that we can do, even if it's not the most efficient.

Now if we have all the time in the world, we can make the thing as complicated as we like: https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition.

Make it work, make it right, maybe make it fast

There's a saying, hindsight is 20/20. We know more about the project at the end rather than at the beginning. And maybe many of us have many things we'd do differently if we could work on a project again. So building the thing quickly gives us exactly that, hindsight.

Instead of trying to code a feature properly the first time, unless we know exactly what we're doing, get the thing built, then make it right. If we try to build something properly up front, not only do we risk building something that the customer might not use, but we risk wasting time on building it because we make mistakes anyway the first time round and spend longer doing so.

Forget ceremonies and the processes

"Agile processes" are the rage, but one of the fundamental principles from the Founding Fathers of the Agile Manifesto themselves is optimise for "individuals and interactions over processes and tools".

I've found that it's better to just take regular intervals to identify what's the focus, then double down and get back to building and doing whatever activity is needed to grow the business. So I wouldn't worry about sprints, stand-ups, ceremonies and story points. If they help us build software faster, by all means, but chances are, we don't need to do any of that, but instead, can do things that work for us.

Push to prod

Unless it's critical for us not to do so, push directly to prod. If we have to do manual deployments, we will find ourselves trying to avoid deployments all the time, and so our releases will get bigger, slower and more risky.

"And sacrilege. Pushing directly to production is irresponsible!" Or is it? If we're doing something that is sensitive, like working with payments or data, yes, have our test environments. We need to use our judgement of course, but if we can take away barriers to deploy to production, then we make it easier, not just to release features, but to fix bugs which will certainly happen at some point anyway.

If Michael Bryzek can do it with a company processing thousands of transactions per day on a mature website and platform, we can do it for our start-ups if there's very little risk.

https://www.youtube.com/watch?v=z-ATZTUgaAo

Of course, we need to test our features before the push.

Measure your product usage

We can interview our customers, which is a great idea. But what customers say and do are not always the same. When they have the product in their hand, using something like Hotjar or another tool to measure how the product is used will give us real-time insights. If we thought a set of features would help our users, and then after release, realised no one uses them, we can stop wasting time building on that feature even more or approach the feature from a different angle or move on to something else.

Make IntelliSense your friend

While I love TypeScript, for a new tool and hopefully product, I used JavaScript with JSDoc. It allows me to build quickly, but with JSDoc I get IntelliSense helping me to build the tool. In fact, after spending 2 hours writing the core logic of feature v0 for a product, I took the JSDoc comments, gave it to Claude.ai, then I asked it to write me the glue code. And it did!

When we optimise for IntelliSense, it's like documentation to help us write our code in real time without ever having to come out of the IDE. And writing code becomes faster at that too!

Document your tools

We all are leveraging a suite of different tools. If we're fortunate to get new hires and developers, they'll start introducing tools of their own. Before we know it, we've got overlap, and people don't know where to find stuff. Even if we're a one-person band, we might even forget all the tools we're using.

For me, it's things like analytics, email hosting, domain hosting, uptime monitoring. I even used a README to document it. We can use whatever we like. That saves time asking, what are we using.

Finally, don't stress

Stress prevents creativity. While we just need to get things done, sometimes creativity and realising there's a simpler, faster and better way to do something versus the way we've always done it can be beneficial. So avoid deadlines, especially if they create stress. Instead, we should try to choose the feature or work that we think will have the biggest impact on our goal that can be achieved in the fastest possible time, and then go do that.

How to dump and restore a Postgres DB with new table ownership

Gemma Black — Thu, 14 Mar 2024 08:20:13 +0000

I've used MySQL for years. But recently, I found myself working with PostgreSQL and simple things like dumping and restoring a database are different enough that I decided to document the process. It's straightforward enough once I knew how.

So first, I'll do a quick overview and then share the explanations afterwards.

Quick Overview

There are several ways to dump and restore data in Postgres if you have more specialist needs. I've chosen one almost analogous to how I would do it with MySQL.

Requirements

You need the correct privileges to both dump the database and recreate all the database objects.

Set the user credentials

Instead of passing options to pg_dump and psql, we can also set up our access credentials as environment variables. There are more secure alternatives to handling database access secrets though.

export PGHOST=<host>
export PGUSER=<user>
export PGPASSWORD=<password>

Dumping the data

Notice how we dump the data with the --no-privileges and --no-owner. Don't include this if you want to keep the ownership as is.

pg_dump \                                                                          
  --no-privileges \
  --no-owner \ 
  dbname > dbname.sql

Creating the new database

Create a new database from a template.

psql \
    -c "CREATE DATABASE newdb TEMPLATE template0;"

Restoring the data

Notice I used --single-transaction to import the data. If you don't mind importing the data where there are some errors, remove that option.


psql \
    --single-transaction \
    new_db < dbname.sql

Changing table ownership

Finally, when working with MySQL, I don't have to worry about table ownership. I just use grants and privileges. But when you import data in Postgres, it assumes the owner is the user associated with the connection. One way I've seen how to change table ownership to one we want is to generate the ALTER commands and then run that.

psql newdb

Then within the psql shell, we create the ALTER commands.

## :psql >>

select 'ALTER TABLE ' || t.tablename || ' OWNER TO new_owner;' 
 from  pg_tables t
 where t.tableowner != 'rdsadmin';

It will print out something like the following, which you will need to copy-paste, and run within the shell again.

## :psql >>

ALTER TABLE groups OWNER TO new_owner;
ALTER TABLE images OWNER TO new_owner;
ALTER TABLE jobs OWNER TO new_owner;

And that's it.

We've dumped the database.
We've restored it.
And we've changed the table ownership to what we want.

Now, I'll share a few things I found in the documentation.

Notes on how the dump and restore work

Libpq

Libpq is the "C application programmer's interface to PostgreSQL". So I needed to install Libpq before being able to run some commands on my local environment. For me on MacOS, it was not installed with Postgres and had to be done separately.

Table locks

With MySQL, dumping the database with table locks means users may experience slow responses or even downtime. To prevent table locks in MySQL, we can do the following:

mysqldump \
    --single-transaction \
    --skip-lock-tables \
    dbname > dbname.sql

The --single-transaction tells the mysqldump to "put everything into a transaction", and "read the database in the current state and create a consistent data dump". But it will still lock the tables unless we provide --skip-lock-tables. That way, we get a consistent data output and no table locks.

↗️ Run mysqldump without locking the tables - https://mysqldump.guru/run-mysqldump-without-locking-the-tables.html

In Postgres, this is not a problem as "dumps created by pg_dump are internally consistent, that is, updates to the database while pg_dump is running will not be in the dump. pg_dump does not block other operations on the database while it is working." Therefore, no extra options are needed. Of course, there are exceptions.

↗️ Chapter 9. Backup and Restore - https://www.postgresql.org/docs/7.2/backup.htm

Using options instead of environment variables for connection settings

Similar to MySQL, we have -h host, -p port , but we have -W password for the prompt and a capital U for -U user.

Removing ownership

If you want the table ownership and privileges to remain the same, providing --no-privileges and --no-owner is not necessary. However, without this, you get the following SQL in your SQL dump:

CREATE SCHEMA abc;

ALTER SCHEMA abc OWNER TO existing_user;

Database Templates

Database Templates in Postgres is a foreign concept to me in MySQL. However, when we do, CREATE DATABASE in Postgres, it's an alias for CREATE DATABASE dbname TEMPLATE template1.

Now, there are two templates by default. template0 and template1. In the earlier example, I used template0, but why?

template1 can be amended so you can create a new database with your customisations.
template0 doesn't have any customisations. So it's a pure, unadulterated database that you can clone and copy.

All-or-nothing imports

Something interesting we can do in Postgres is ensure our imported data is consistent. If there are any errors when importing the dumped file, it will roll back all changes. To do this, we use --single-transaction on the import:

psql \
    --single-transaction \
    < dumpfile.sql

Note: "When using this mode, be aware that even a minor error can rollback a restore that has already run for many hours. However, that might still be preferable to manually cleaning up a complex database after a partially restored dump." - postgresql.org

That wasn't so bad

MySQL and Postgres are similar but they're not the same. Hopefully dumping and restoring in Postgres will be simpler (even if it's just for me) in future.

NodeJS: 4.8x faster if we go back to callbacks!

Gemma Black — Sat, 10 Feb 2024 10:08:00 +0000

Yeah, I said it!

Callbacks are 4.8x faster when running them parallel over async/await in parallel. And only 1.9x faster when we run sequential callbacks.

I've modified this article somewhat after I got some helpful and kind comments about my dodgy test. 😂🙏

Thank you to Ricardo Lopes and Ryan Poe for taking the time to steer the benchmarks in the right direction. My first faux pas was I wasn't actually waiting for the execution of the code to finish, which crazily skewed the results. The second was I was comparing parallel to sequential runtimes, which make the benchmarks worthless.

So this is round 2 which addresses my initial errors. Previously, I said:

NodeJS: 34.7x faster if we go back to callbacks! I shouldn't have believed it. 🤣

Not as impressive as my bad benchmarks before (and see comments for context), but still a sizeable difference.

So what exactly did I test?

I compared callbacks to promises and async/await when reading a file, 10,000 times. And maybe that's a silly test but I wanted to know, which is faster at I/O.

Then I finally compared callbacks in Node.js to Go!

Now, guess who won?

I won't be mean. TLDR. Golang!

Lower is better. Results are in ms.

Now, true benchmarkers? Go easy on me on this one. But please do leave your comments to make me a better person.

Everyone keeps saying that Node.js is slow!

And it bugs me out.

Because what does slow mean? As with all benchmarks, mine is contextual.

I started reading about the Event Loop, just to even begin to understand how it works.

But the main thing I've understood is that Node.js passes I/O tasks onto a queue that sits outside the main Node.js executable thread. This queue runs on pure C. A number of threads could potentially handle these I/O operations. And that's where Node.js can shine, handling I/O.

Promises, however, get handled in the main, single executable thread. And async/await, is well, promises but now with blocking added.

So are callbacks faster than promises?

Let's put it to the test.

First off. My machine! Complements of working with Kamma. It's important to note what resources we're working with. Plenty memory and CPU.

MacBook Pro (14-inch, 2021)
Chip      Apple M1 Pro
Memory    32 GB
Cores     10
NodeJS    v20.8.1
Go        1.21.0

So we have a text.txt file with an original message, Hello, world.

echo "Hello, world" > text.txt

And we'll read this text file using native Node.js, which means, zero node module dependencies because we don't want to drag the speed down with the heaviest objects in the universe.

Callbacks

Parallel callbacks

First, let's start with parallel callbacks. I'm interested in how quickly the same file can be read as quickly as possible, all at once. And what's faster than parallel?

// > file-callback-parallel.test.mjs
import test from 'node:test';
import assert from 'node:assert';
import fs from "node:fs";

test('reading file 10,000 times with callback parallel', (t, done) => {
    let count = 0;
    for (let i = 0; i < 10000; i++) {
        fs.readFile("./text.txt", { encoding: 'utf-8'}, (err, data) => {
            assert.strictEqual(data, "Hello, world");
            count++
            if (count === 10000) {
                done()
            }
        })
    }
});

Sequential callbacks

Second, we have callbacks again, but sequential (or rather blocking). I'm interested in how quickly the same file can be read sequentially. Having not done callbacks calling callbacks for ages, this was fun to try again. Albeit, it doesn't look pretty.

// > file-callback-blocking.test.mjs
import test from 'node:test';
import assert from 'node:assert';
import fs from "node:fs";

let read = (i, callback) => {
    fs.readFile("./text.txt", { encoding: 'utf-8'}, (err, data) => {
        assert.strictEqual(data, "Hello, world");

        i += 1

        if (i === 10000) {
            return callback()
        }

        read(i, callback)
    })
}

test('reading file 10,000 times with callback blocking', (t, done) => {
    read(0, done)
});

Async/Await

Then we have async/await. My favourite way of working with Nodejs.

Parallel async/await

It's as parallel as I can get with async/await. I load all the readFile operations into an array and await them all using Promise.all.

// > file-async-parallel.test.mjs
import test from 'node:test';
import assert from 'node:assert';
import fs from "node:fs/promises";

test('reading file 10,000 times with async parallel', async (t) => {
    let allFiles = []
    for (let i = 0; i < 10000; i++) {
        allFiles.push(fs.readFile("./text.txt", { encoding: 'utf-8'}))
    }

    return await Promise.all(allFiles)
        .then(allFiles => {
            return allFiles.forEach((data) => {
                assert.strictEqual(data, "Hello, world");
            })
        })
});

Sequential Async/Await

This was the easiest and most concise one to write.

// > file-async-blocking.test.mjs

import test from 'node:test';
import assert from 'node:assert';
import fs from "node:fs/promises";

test('reading file 10,000 times with async blocking', async (t) => {
    for (let i = 0; i < 10000; i++) {
        let data = await fs.readFile("./text.txt", { encoding: 'utf-8'})
        assert.strictEqual(data, "Hello, world");
    }
});

Promises

Finally, we have promises without async/await. I've long stopped using them in favour of async/await but I was interested in whether they were performant or not.

Parallel promises

// > file-promise-parallel.test.mjs
import test from 'node:test';
import assert from 'node:assert';
import fs from "node:fs/promises";

test('reading file 10,000 times with promise parallel', (t, done) => {
    let allFiles = []

    for (let i = 0; i < 10000; i++) {
        allFiles.push(fs.readFile("./text.txt", { encoding: 'utf-8'}))   
    }

    Promise.all(allFiles)
        .then(allFiles => {
            for (let i = 0; i < 10000; i++) {
                assert.strictEqual(allFiles[i], "Hello, world");
            }

            done()
        })
});

Sequential promises.

Again, we want to wait for the execution of all readFile operations.

// > file-promise-blocking.test.mjs
import test from 'node:test';
import assert from 'node:assert';
import fs from "node:fs/promises";

test('reading file 10,000 times with promises blocking', (t, done) => {
    let count = 0;
    for (let i = 0; i < 10000; i++) {
        let data = fs.readFile("./text.txt", { encoding: 'utf-8'})
            .then(data => {
                assert.strictEqual(data, "Hello, world")
                count++
                if (count === 10000) {
                    done()
                }
            })
    }
});

And voila! Results 🎉! I even ran it a few times to get a better reading.

I ran each test by doing:


node --test <file>.mjs

Reading a file 10,000 times with callbacks is over 5.8x faster than with async/await in parallel! It's also 4.7x faster than with promises in parallel!

So, in Node.js land, callbacks are more performant!

Now is Go faster than Node.js?

Well, I don't write in Go, so this may be truly terrible code because I asked ChatGPT to help me and yet, it seems pretty decent.

Hey ho. Let's go. Our Golang code.

package main

import (
    "fmt"
    "io/ioutil"
    "time"
)

func main() {
    startTime := time.Now()

    for i := 0; i < 10000; i++ {
        data, err := ioutil.ReadFile("./text.txt")
        if err != nil {
            fmt.Printf("Error reading file: %v\n", err)
            return
        }
        if string(data) != "Hello, world" {
            fmt.Println("File content mismatch: got", string(data), ", want Hello, world")
            return
        }
    }

    duration := time.Since(startTime)
    fmt.Printf("Test execution time: %v\n", duration)
}

And we run it as so:

go run main.go

And the results?

Test execution time: 58.877125ms

🤯 Go is 4.9x faster than Node.js using sequential callbacks. Node.js only comes close with parallel execution.

Node.js Async/await is 9.2x slower than Go.

So yes. Node.js is slower. Still, 10,000 files in sub 300ms isn't to be scoffed at. But I've been humbled by Go's speediness!

Now just a side note. Do I have bad benchmarks?

I really did have terrible Benchmarks. Thank you again to Ricardo and Ryan.

Yes, I did. Hopefully now they're better.

But you may ask, who's really going to read the same file, over and over again? But for a relative test between things, I hope it's a helpful comparison.

I also don't know how many threads Node.js is using.

I don't know how my CPU cores affect Go vs Node.js performance.

I could just rent an AWS machine with one core and compare.

Is it because I'm on Mac M1?

How would Node.js perform on a Linux or...Windows? 😱

And there's the practicality of, yes, reading a file is one thing, but at some point, you have to wait anyway for the file to be read to do something with the data in the file. So, speed on the main thread is still pretty important.

Now, do you really want to use callbacks?

I mean, do you really, really want to?

I don't know. I definitely don't want to tell anyone what to do.

But I like the clean syntax of async/awaits.

They look better.

They read better.

I know better is subjective here but I remember callback-hell, and I was grateful when promises came into existence. It made Javascript bearable.

Now, Golang is clearly faster than Node.js at its optimum, with callbacks, and with async/await, by 9.2x! So if we want good readability and performance, Golang is the winner. Although, I'd love to learn how Golang looks under the hood.

Anywho. This was fun. It was more of an exercise to help me understand how callbacks and I/O work in the Event Loop.

So to sign out

Is Node.js slow? Or are we just using Node.js on slow mode?

Probably where performance matters, Golang is worth the jump. I'll certainly be looking more at using Golang in future.

Updates

Sequential promises

If I rewrite it as below, I'm still using callbacks 😬 but at least it's sequential:

import test from 'node:test';
import assert from 'node:assert';
import fs from "node:fs/promises";

let read = (i, callback) => {
    let data = fs.readFile("./text.txt", { encoding: 'utf-8'})
        .then(data => {
            assert.strictEqual(data, "Hello, world")
            i += 1

            if (i === 10000) {
                return callback()
            }

            read(i, callback)
        })
}

test('reading file 10,000 times with promises blocking', (t, done) => {
    read(0, done)
});

I didn't include the above in the benchmarks but it is slower and not surprisingly so at 532.777667ms.

NodeJS: We can run tests natively!

Gemma Black — Wed, 07 Feb 2024 10:37:55 +0000

Once upon a time, there was Mocha.

Then there was Jasmine. Or perhaps it was the other way around. But Mocha was first for me.

There was Jest.

Now there's Vitest.

If you want to write tests for your Nodejs app, you're spoilt for choice. You just run npm install for the pleasure.

Some worked well with Grunt and Gulp.

Others worked great with Webpack.

Now there are those that work perfectly with Vite.

But unless we're doing some compilation for the front end or we're working with Typescript, why aren't we using NodeJS's native test runner? It's been around since version 18! That's April 2022. Almost 2 years in!

So what does it look like? 💯 zero node modules required:

And straight from the docs.

// > node test.mjs
import test from 'node:test';
import assert from 'node:assert';

test('synchronous passing test', (t) => {
    assert.strictEqual(1, 1);
});

That's it!

Assertions? They're native.

Test runners? Also native.

The output? Delightful. 🤩

✔ synchronous passing test (0.618667ms)
ℹ tests 1
ℹ suites 0
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 4.6255

We can even run tests spanning multiple files. As long as the file ends with an acceptable glob pattern like *.test.mjs , you can run the following:

node --test

Default globs are:

**/*.test.?(c|m)js

**/*-test.?(c|m)js

**/*_test.?(c|m)js

**/test-*.?(c|m)js

**/test.?(c|m)js

**/test/**/*.?(c|m)js

We even have watch mode for goodness' sake. How did I not know about this?

node --test --watch

Code coverage? Yes. 😯

# node --test --experimental-test-coverage

ℹ tests 6
ℹ suites 2
ℹ pass 6
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 54.586833
ℹ start of coverage report
ℹ --------------------------------------------------------------
ℹ file          | line % | branch % | funcs % | uncovered lines
ℹ --------------------------------------------------------------
ℹ test.mjs      | 100.00 |   100.00 |  100.00 | 
ℹ test.test.mjs | 100.00 |   100.00 |  100.00 | 
ℹ --------------------------------------------------------------
ℹ all files     | 100.00 |   100.00 |  100.00 |
ℹ --------------------------------------------------------------
ℹ end of coverage report

describe/it blocks? Yes, too! 🤯

describe('A thing', () => {
    it('should work', () => {
        assert.strictEqual(1, 1);
    });

    it('should be ok', () => {
        assert.strictEqual(2, 2);
    });

    describe('a nested thing', () => {
        it('should work', () => {
            assert.strictEqual(3, 3);
        });
    });
});

And it looks beautiful may I add.

▶ A thing
  ✔ should work (0.082667ms)
  ✔ should be ok (0.051292ms)
  ▶ a nested thing
    ✔ should work (0.05575ms)
  ▶ a nested thing (0.126541ms)

▶ A thing (0.512584ms)

So why aren't we using more of Nodejs natively?

One reason is that the test runner didn't exist until April 2022, probably when I was already super excited about Vite. Another reason is we rarely just work with pure Nodejs. Normally we work with a frontend framework, be it React or Vue, Typescript, Purescript, or a fully-fledged framework like AdonisJS to make our life easier.

Reducing our dependencies and relying on native

Having recently fallen foul of dependency hell with an old project, I came to the realisation, that reducing dependencies makes it easier to update a project. Try and fix a Node version 8 script with a dozen node modules that are no longer maintained, and you realise it's probably easier to just write the whole thing again.

The downside is native Nodejs will be more verbose to write, it will lack the Typescript types that I love so much, and it will be missing the prettier documentation that comes with beautiful frameworks like Nuxt.

However, if I want my projects to last a long time with a minimum of fuss and maintenance, it might be worth forgoing these shinier tools that abstract away NodeJS's NodeJSness, and just use it with its fullest purity.

It's not just a NodeJS problem. Every language has this problem. We've developed tools to make our lives easier because changing and improving the underlying language isn't always possible.

Yet, Ricardo Lopes proved you can have a sane zero-dependency mini web application without even a package.json written in NodeJS:

"There's a place for complex frameworks and architectures, sure. But for many projects, they may be an overkill."

So if we're developing a project that's 1) small, 2) not really going to change much, and 3) only needs NodeJS, why not avoid using any dependencies at all, and go native with node --test?

I'll sign off with a Flavio Tweet in favour of reducing dependencies.

// Detect dark theme var iframe = document.getElementById('tweet-1754784031307084015-865'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1754784031307084015&theme=dark" }

Making Preons

Gemma Black — Mon, 25 May 2020 16:47:44 +0000

This is my first serious article on dev.to. Critique is most welcomed. Nevertheless, I love how-to guides but there are so many good ones out there. Instead, I hope to merely share my experience.

The Story - 2 years and 4 iterations

I loved writing code. I still do.

In front of my laptop, I was studying yet another project. The design was beautiful as always. All I had to do was make the HTML. But here I was again, writing the navigation, the card elements, the buttons, the grid, the footer. Déjà vu or not, I knew there had to be a way of systemising the creation of websites.

All I wanted was to build beautiful user interfaces faster without rewriting the same code every time. So I created Preons.

In this article I share:

The principles I learned along the way
And how Preons became a css generating system

Iteration 0 - Tachyons

The story

It's a bit cheeky saying Tachyons is the first iteration. I didn't build it. Adam Morse and co did.

All I know is I cottoned on to it either at the end of 2017 or the beginning of 2018.

But this very library changed my thinking about how stylesheets could be written. I was doing BEM. And there was a cycle. Every UI element needed both handwritten css and afterwards, HTML.

Tachyons was different. Tachyons was about writing a stylesheet once, then building the interface with that stylesheet right there in the DOM.

5 minutes later, I copy-pasted up a nice looking user interface. No CSS written. The hard work went solely into finding components that looked good. A nice header. A fancy footer. And something in between.

This is what I mocked up again (circa 2020):

edit on repl.it.

Change paddings. Add margins. Increase font sizes right there in the browser. Underneath the hood, it looks like this:

<footer class="pa4 pa5-l black-70 bt b--black-10"></footer>

The only thing was, I wanted a different font. A Google font to be exact. And then the colours weren't what I liked. These were the things not in the stylesheet.

Being able to make these little customisations quickly and easily are important. Not every project has the same vertical rhythm. Not every project needs the same CSS properties let alone values.

So I did the easiest thing, but the wrong thing. I hardcoded classes into a new stylesheet that "extended" the Tachyons library. But there was a priority problem. Can you spot it?

/** 
 * In tachyons.css
 */

@media screen and (min-width: 60em) {
  .aspect-ratio-l {
    height: 0;
    position: relative;
  }

  .aspect-ratio--16x9-l {
    padding-bottom: 56.25%;
  }

  .aspect-ratio--9x16-l {
    padding-bottom: 177.77%;
  }
}

/**
 * Then...in a custom .css
 */

.aspect-ratio--4-3 {
  padding-bottom: 75%;
}

So what happens if I try to have applied different aspect ratios at different breakpoints?

<div class="aspect-ratio--4-3 aspect-ratio--16x9-l">
  <img src="some-image.png" />
</div>

Lesson 1 - Framework or library

Library

When you use a library, you are in charge of the flow of the application. You are choosing when and where to call the library.

Framework

When you use a framework, the framework is in charge of the flow. It provides some places for you to plug in your code, but it calls the code you plugged in as needed.

- The Difference Between a Framework and a Library

Updating Tachyons felt like I was using it as a framework rather than as a dependency. To prevent priority issues, you need to insert your new css classes at particular places inside the stylesheet. But then the line is blurred between what is mine and added, and what is Tachyons'.

Then I thought, maybe it's better instead, not to extend Tachyons, but generate it instead.

Lesson 2 - Use what exists

As a developer, I must admit, there's always the temptation to write functionality from scratch instead of tweaking something that does a good job already. So I found a tachyons-scss library. It was perfect! Except, I still felt like I was hard-coding styles.

.b--dotted { border-style: dotted; }
.b--dashed { border-style: dashed; }
.b--solid {  border-style: solid; }
.b--none {   border-style: none; }

@media #{$breakpoint-not-small} {
  .b--dotted-ns { border-style: dotted; }
  .b--dashed-ns { border-style: dashed; }
  .b--solid-ns {  border-style: solid; }
  .b--none-ns {   border-style: none; }
}

Iteration 1 - Preonize

Generate a library

What if I defined all my rules using sass maps like that:

$colors: (
  'blue': #365e86,
  'white': #ffffff,
  'grey': #eef0ee,
  'grey-dark': #89969d
);

Then applied them to a css property like background-color to get this:

.bg-blue { background-color: #365e86; }
.bg-white { background-color: #ffffff; }
.bg-grey { background-color: #eef0ee; }
.bg-grey-dark { background-color: #89969d; }

The icing on the cake is having each rule at different breakpoints in the right order. Priority problem solved.

.bg-blue { background-color: #365e86; }

@media screen and (min-width: 40em) {
  .bg-blue-m { background-color: #365e86; }
}

@media screen and (min-width: 60em) {
  .bg-blue-l { background-color: #365e86; }
}

And so I wrote the preonize function. And that was it. Almost.

@mixin preonize($name, $prop, $map, $breakpoints) {
  @each $label, $value in $map {
    .#{$name}#{$label} {
      #{$prop}: $value;
    }
  }

  @each $breakpoint, $breakpoint-value in $breakpoints {
    @media #{$breakpoint-value} {
      @each $label, $value in $map {
        .#{$name}#{$label}-#{$breakpoint} {
          #{$prop}: $value;
        }
      }
    }
  }
}

All I had to do was define all my reusable rules, and apply them using preonize.

@include preonize('bg-', background-color, $colors, $breakpoints);
@include preonize('fill-', fill, $colors, $breakpoints);
@include preonize('', color, $colors, $breakpoints);

// etc

Preonize took 4 things:

a class prefix like bg-
a css property like background-color
a sass map of rules like $colors: (white: #ffffff, black: #000000)
and an array of breakpoints to apply those rules

I could generate an entire functional CSS library from a base sass file. Minor changes were now trivial. Update the sass maps; compile to css.

If I left it at this, I'd be happy. I already built carolblackmusic.co.uk, pixelexaspect.com and kammadata.com this way. But I had come across some issues.

Lesson 3: Opinions and conventions

There are definitely wrong ways to do things, but usually, there are no single right ways. But it's also true, that not having an opinion for a project will cause confusion eventually, even with one maintainer.

So some of the classes I didn't like were the ones that redeclared rules for the same css property, like padding-bottom. For example, padding-bottom could override a completely different class prefix aspect-ratio.

This causes the priority issue again.

I felt that there should be one class prefix for one css property.

The other thing I didn't want was for a class to represent two CSS properties. One example is margins. So instead of one class representing margin-left and margin-right at the same time, it's more explicit to just use two.

<!-- Instead of doing this -->
<img src="some.jpg" class="hv-auto" />

<!-- I'd do this -->
<img src="some.jpg" class="hl-auto hr-auto" />

This decision would eventually help me to write simpler code generators for Preons, though, at the time, I didn't know it.

Lesson 4 - Eat your own dog food

I have Gary Gale to thank for the phrase. All it means is, what you're giving others to use, are you using it yourself?

So it's all good building a CSS library but could I use it? In fact, using it highlighted limitations of functional css in general very quickly.

Displaying user generated articles
Hovers and animations

Limitation 1 - Displaying user-generated articles

How could I expect a user to write an article and add margins to every paragraph?

<p class="mb2">My wonderful article paragraph.</p>
<h2 class="fs1 fs2-m blue">Next headline</h2>

Doing so:

would actually cause inconsistencies in the design
make writing in WYSIWYGs difficult
writing markdown would be no different from writing html
just in general, the writing experience would be awful and take longer

So the solution to this is to create scoped article classes such as s-article, something I learned from working with Milad Alizadeh and Chris Boakes.

Here's a snippet I use for this Preons' documentation website:

.s-article {
  @extend .lh0;
  @extend .fs0;
  @extend .lh1-m;

  li,
  p {
    @extend .mb1;
  }

  h1 {
    @extend .fs2;
    @extend .lh2;
    @extend .mb2;
    @extend .fwb;
  }

  h2 {
    @extend .fs1;
    @extend .lh2;
    @extend .pt2;
    @extend .mb1;
    @extend .fwb;
    @extend .bwb1;
    @extend .bsb-solid;
    @extend .bca-greyll;
    @extend .lh4-m;
  }
}

Then I apply it where it is needed so it doesn't affect the entire site:

<div class="s-article">
  <nuxt-content :document="page" />
</div>

Limitation 2 - Hovering and animations

Animations are the life-blood of interactive websites. At a minimum, we need to hover and change things like colours of text and backgrounds on buttons. Okay, it's not essential, but great for user experience. So a new preonize function would be needed.

@mixin preonize-hover($name, $prop, $map, $breakpoints) {
  @each $label, $value in $map {
    .#{$name}#{$label}:hover {
      #{$prop}: $value;
    }
  }

  @each $breakpoint, $breakpoint-value in $breakpoints {
    @media #{$breakpoint-value} {
      @each $label, $value in $map {
        .#{$name}#{$label}-#{$breakpoint}:hover {
          #{$prop}: $value;
        }
      }
    }
  }
}

Iteration 2 - A CLI

The Story

After building a few sites, I had a problem. There were differences in the class conventions I used between them. What was max-width again?

.mw1 {
  max-width: 1rem;
}

/** or **/

.maxw1 {
  max-width: 1rem;
}

And what about colours? What's the convention? The colour first, or the modifier first?

.dark-grey {
  color: grey;
}

/** or **/

.grey-dark {
  color: grey;
}

I had very few conventions and was mainly inspired by Tachyons. But switching between projects I was forgetting what rule applied where.

Looking up classes wasn't easy using my sass file. I had to find the CSS class, then look up the corresponding global style.

I could look through the CSS, but then I thought, that's not a great user experience. If I could config my rules in yaml or JSON, I could generate documentation.

Still my favorite #dev discovery of the year. Cyrille's concepts on living documentation. pic.twitter.com/psmtpI2czF
— Gemma Black (@GemmaBlackUK) October 31, 2019

It was reading Cyrille Martraire's book on Living Documentation that cemented the idea of turning existing code into docs. And your code would always be in sync with your documentation. But it was a Michael Bryzek video that took the concept one step further, generating code from your config.

And he had a point.

With a config, I could generate styles themselves from a yaml config into any CSS style language I liked.

Sass
Postcss
Less.

One single config would be the definition for both the library and the documentation.
So preons.yaml was born.

preons:
  baseline: 1.2rem;

  gutter: 2.5rem;

  rules:

  # Preon breakpoints
  breakpoints:

  # Preon classes
  classes:
    - label: bg-
      css-property: background-color
      rule:
        - color
    - label: bg-
      css-property: background-color
      rule:
        - color

Then I stopped working on the project...

For ages.

Limitation 3 - Mixing and matching rules

For widths, I wanted to use gradations of both percentages and rems.

For margins, I wanted the same, but also negative rems. So unless I repeated myself, I needed a way of mixing different global rules together.

$scaled: (
  n1: -1rem,
  n2: -2rem,
  n3: -3rem,
  n4: -4rem,
  n5: -5rem,
  n6: -6rem,
  n7: -7rem,
  n8: -8rem,
  n9: -9rem,
  n10: -10rem,
  0: 0,
  1: 1rem,
  2: 2rem,
  3: 3rem,
  4: 4rem,
  5: 5rem,
  6: 6rem,
  7: 7rem,
  8: 8rem,
  9: 9rem,
  10: 10rem
);

So I found map-collect.

@function map-collect($maps...) {
  $collection: ();

  @each $map in $maps {
    $collection: map-merge($collection, $map);
  }
  @return $collection;
}

Then I could mix-and-match them across different css properties:

@include preonize(
  'pa', 
  padding, 
  $scaled, 
  $breakpoints
);
@include preonize(
  'pl',
  padding-left,
  map-collect($scaled, $discrete),
  $breakpoints
);

Iteration 3 - Tweaking the config

Array vs object of rules

It occurred to me. Why am I using an array of properties when it should just be an object? I can't redeclare the same property twice because of the priority problem and the convention to solve that problem of 1 css property to 1 class prefix.

Yaml linters highlight when you have duplicate properties in an object. It is not allowed. So that's a win in applying the convention.

Global vs standalone rules

I also didn't like having to declare global rules for each CSS property. Sometimes rules only have to be applied once, eg. for display: flex

align-content:
  class: content-
  values:
    start: flex-start
    end: flex-end
    center: center
    between: space-between
    around: space-around
    stretch: stretch

vs colors:

color:
  class:
  rule: theme-colors

background-color:
  class: bg-
  rule: theme-colors

Generating documentation

Doing preons config would spit out JSON based on the preons.yaml to make creating documentation easier:

{
  "border-top-color": {
    "class": "bct-",
    "rule": "theme-colors",
    "values": {
      "black": "#242027",
      "white": "#fefeff",
      "greyll": "#f6f5f9",
      "greyl": "#beb9cc",
      "grey": "#7d778e",
      "greyd": "#47454c",
      "transparent": "transparent",
      "hotpink": "#ea2889"
    },
    "mappings": {
      "bct-black": "#242027",
      "bct-white": "#fefeff",
      "bct-greyll": "#f6f5f9",
      "bct-greyl": "#beb9cc",
      "bct-grey": "#7d778e",
      "bct-greyd": "#47454c",
      "bct-transparent": "transparent",
      "bct-hotpink": "#ea2889"
    }
  }
}

I use this json to build the reference for the docs.

Lesson 5 - Release fast and automate versions

Whether you adhere to romantic versioning, sentimental versioning or semantic versioning, managing versions manually is a bottleneck. Furthermore, it's like doing paperwork. But if it can be automated, it saves so much time.

Why?

Because moving fast allows fixes and features to be released quickly. The developer writing the code should know if their update is breaking existing functionality or not, whether they are adding a feature, a fix, or an improvement or not.

So I used Intuit's Auto. Running npm run release along with Angular style commits:

bumps npm
tags the project
makes a release in GitHub
creates the changelog

Done.

Lesson 6 - Distribute files easily

Michael Jackson's Unpkg allows you to access any file from an npm library at any version and it's free to use 🙏. It made creating the example repl easy.

Here's a great intro to it:

https://kentcdodds.com/blog/unpkg-an-open-source-cdn-for-npm

Lesson 7 - Keeping the cli and stylesheets core

"I don't know any good reason to split my codebase between stylesheet and cli over multiple repositories". So I package them together. Maybe this is bad. But until it becomes a problem, I decided to not worry about it.

It means, preons-theme 0.3.28 was released at the same time as the cli which is also 0.3.28, even if the cli didn't change.

Lesson 8 - Skip the tests

Now as of May 2020, there aren't any tests. There are no linters either. Dangerous? Maybe. But normally I'd fuss over assuring my code passes all sorts of code quality bells and whistles, not just because I'm a badge junkie. This time, I was determined that version 0.0.z would just work and be functional.

I think Eric Elliot says something like:

Make it work
Make it stable
Make it fast

Even though, he starts with tests, making it work is a lot more archaic for me. Make it stable is where tests come in and solidifying them under the right design.

“Coding faster: Make it work, then make it good” by Michael Parker https://link.medium.com/SRPxvd76F6

If anything, this is the biggest shift in my thinking. Having gone through several iterations, I finally designed a version of Preons that I actually like.

If I started with TDD, I think I'd feel too precious about changing anything because TDD sometimes eats into the design phase, rather than just being a development tool.

I genuinely think design comes before TDD and static type analysis. That's why JavaScript is so powerful. You can iterate fast without worrying about correctness of code.

An MVP is a viable prototype, not a final product. Don't worry about perfection. The idea is to get fast feedback from users.
— Eric Elliott (@_ericelliott) May 19, 2020

Lesson 9 - Design first

I saw a video where James Clear talked about two sets of photography students. One group had to come up with the perfect photo, one time. Another group had to take lots of photos and present their best one. The latter scored better apparently because applying their knowledge came from trail and error, learning what worked through practice versus theorizing what worked. That is pragmatism. Practice over theory.

My fear is, in our desire to follow best practice, we use tools that can hinder us at the wrong time. During the design phase, the learning phase, the prototyping phase, getting something to work is most important. Proving the possibility is most important.

Now I've blitzed through the most basic of features, I have a choice:

to start firming up what I have with tests
try a different design
add more features

In conclusion

I hope you enjoyed the article or at least learned that your open source project doesn't have to be right the first time around. By no means am I a decent writer or even correct about everything in this article. So please share feedback.

If you're on Twitter, just @GemmaBlackUK and tell me what I did wrong 😅.