Forem: Muhammad Bin Zafar

The fuss with CRLF and LF in Git 😕

Muhammad Bin Zafar — Thu, 04 Jan 2024 14:08:24 +0000

This problem arises with Git when team members are using different operating systems, e.g., Windows vs Mac/Linux.

A non-AI-generated summary of the solution:

Step 1. Decide whether to save changes in the Git index (the Git commit database) in CRLF or LF. Git wants to save in LF. The world wants to save in LF. Linux and Mac have the default of LF. Windows is alone in having CRLF. So the decision is made: LF.

Step 2. In git-config, core.autocrlf=input means when generating files from Git index for the user, Git will do CRLF for Windows and LF for Mac/Unix. Also, while writing to Git index from files, it will use LF. Having core.autocrlf=false means Git won't do anything. Just save stuff as is. So, make sure you don't change the default setting.

Another case to be aware of is that code formatters like Prettier may format the code and save the file with LF as line-endings in Windows, although the file had CRLF. In this case, Git will notice it and show a warning. Safely ignore this warning, not important at all, forget it.

warning: LF will be replaced by CRLF in <filename>.

Step 3. Usually, that's all. But some teammates might add an ESLint (or similar code formatter) rule to have all line-endings as LF.

This is tricky because tools like ESLint check the current file buffer and informs you of warnings and errors on the fly. And, in Windows, Visual Studio Code defaults to saving files with CRLF.

Now, if you have an ESLint rule that enforces line-endings to be LF, then your entire file buffer in Visual Studio Code will be underlined due to warnings/errors! Overwhelming as hell!!!

This is pain, and that's what happened to me once. Untangling that tuxedo situation is how I started investigating the case of CRLF vs LF in Git and finally wrote a LinkedIn post about it!

To fix:

Try not having such an ESLint rule. One that strictly checks line-endings to be LF or CRLF.
Create an .editorconfig file in project root with:

root = true
[*]
end_of_line = lf

Then install the EditorConfig extension. This way, Visual Studio Code will not save the file with CRLF (the default in Windows), rather will follow the end_of_line option.

And that's it. Another wrap on another fussy issue. If you have a question, let me know. I'd love to answer and try helping you out!

Thanks for reading and have a great one 👋

The fuss with ESM vs CJS in Node.js

Muhammad Bin Zafar — Thu, 26 Oct 2023 06:13:10 +0000

If you are learning to code in Node.js or have learnt to code in Node.js, then you are familiar with callback hell. However, that's a case solved and closed. The one at hand is the dilemma of whether to write our code in the ECMAScript Module format or the CommonJS one. So, let's talk about this!

The Prelude

Node.js 🐢 started in 2009 with the standard of its time, the CommonJS format. Later the ECMAScript specification picked up pace and garnered more stability. For this, ECMAScript modules were first introduced to Node.js in v8.5.0 in 2017 and it continued to receive improvements over the years.

The default format in Node.js is CommonJS, and it is usually not explicitly mentioned in package.json such as { "type": "commonjs" }. However, the standard format for code reuse in JavaScript is ESM, which needs to be explicitly mentioned in the package.json like so { "type": "module" }.

The Clash

Many articles and docs including the Node.js docs talk in length about this. The summary, in my opinion, is the following:

You can use CommonJS code from ESM.
You can use ESM code from CommonJS.

Firstly, CommonJS code from ESM. It is dead simple from ESM to import CommonJS code that has a default export or named exports.

import express from 'express' // importing the default export
import {Router, Route} from 'express' // importing named exports

However, it gets tricky when there's no named exports from the CommonJS package, and the default-exported variable is an object. This is tricky because in CJS-to-CJS code reuse, we are accustomed to this convenience. But for CJS-ESM interoperability, this is an obvious bug.

The following is perfectly fine between CJS files:

// common.cjs
module.exports = {
  a: 10,
  b: 20
}

// main.cjs
const {a,b} = require('./common.js')

However, while importing common.cjs from ESM, we cannot do named imports of the properties a and b.

import {a,b} from './common.cjs'   // SyntaxError: Named export 'a' not found. 

import common from './common.cjs'  // Rather, first import the default
const {a,b} = common               // Then, access object properties!

Secondly, ESM code from CommonJS. This is not convenient. Importing ESM code from CommonJS is only done through the async import() function. Since the ESM package is loaded asynchronously, coding pattern and linear reason-ability is hurt.

const importFileType = import('file-type')

exports.isGzip = async function (filename) {
  const fileType = await importFileType
  const {ext} = fileType.fileTypeFromFile(filename)
  return ext == 'gz'
}

exports.isPng = async function (filename) {
  const fileType = await importFileType
  const {ext} = fileType.fileTypeFromFile(filename)
  return ext == 'png'
}

In the code above, the gist is, only to import an ESM module all dependent functions are now being converted to async. CommonJS does not support top-level awaits for us to do const importFileType = await import('file-type'). For this inconvenience, you should not write library code in ESM that will be reused.

I disagree with Sindre Sorhus on this in terms of writing library code. However, for writing application code, ESM is quite nice and convenient.

Note that, in the code example above, the file-type package will be imported only once and the promise is also resolved only once - causing no performance issues.

The Solution

This dilemma has multiple frontiers to solve for. For me, the summary is as follows:

Case	Suggestion	Reason	Example project
Node.js app	ESM	App code are never reused	An express.js backend server e.g.
Node.js app that will bundled, minified, and packaged	CJS	Better support for CJS in these tools	A command-line app e.g.
Library package	CJS	Best for the most swift reuse by both	Express.js
TypeScript	Same rules	Same cases	Read this.
Babel/Transpilers	Try not using them.

Thanks for reading! Found it interesting? Give it a ❤️ or share your thoughts/questions in 💭! Did I miss anything? Let me know in the comments.

Have a great day!

Ryan Dahl, and the story of his Wizardry in Opensource

Muhammad Bin Zafar — Thu, 07 Sep 2023 10:07:00 +0000

On 16 February 2009, a GitHub commit was made. By a person who would later turn out to be the creator of two of the most successful, impactful, and standard tech platforms the world has ever seen.

It's the story of Ryan Dahl, the person behind Node.js and Deno. The mastermind behind the non-blocking asynchronous I/O model of Node.js - a radical approach at that time to process thousands of requests per second. A runtime that made JavaScript successful and gave it legs (i.e. through file system access) to pave the path for modern frontend development frameworks e.g. React from Facebook and Angular from Google.

The struggle, however, was real. First, he had to give up his addiction towards Abstract Mathematics - some people apparently loved doing maths, a rare breed! Second, he had to find a company which will be willing to fund this research and development for the first few years. With these two figured out after much efforts, came the big third one. Fork!

Forks in opensource are often good things, but not so much for an emerging technology that attempts to be a standard in handling web requests and developing server-side applications. Difference of opinions in certain aspects may cause a project to be forked and have the dedicated pool of contributors be divided into forks, competing against each other with a more limited and divided group of people. This isn't good news for Ryan.

With much negotiations, he was able to bring teams back together to focus on the main project. The remnants, however, of the fork still is found in some historical corners of GitHub. Its name was io.js.

nodejs / node-convergence-archive

Archive for node/io.js convergence work pre-3.0.0

Node.js Foundation - Node.js

This repository is the working repository for the proposed convergence of the http://github.com/iojs/io.js and http://github.com/joyent/node projects under the Node.js Foundation.

Contributions, releases, and contributorship are under the proposed Governance and Developer Policy for the soon-to-be-launched Node.js Foundation:

This project is operating with the oversight of the joint Node.js and io.js core technical teams.

Note: The original io.js README.md is temporarily renamed to iojs_README.md.

View on GitHub

The days were going swift with Node.js and Ryan, until... one day. Golang from Google came across Ryan, and its newer concurrency model to process web requests through goroutines and channels. It made a realization for Ryan:

Node.js is not the most effective approach to handling HTTP requests.

For more distributed applications or DNS servers, Node.js is no match for Golang. Despite having garbage collection built-in, the performance of Golang is unmatched and it compiles code into a easily-distributable single binary executable file.

Soon after this realization, Ryan left the Node.js opensource project 😕

Since then, Node.js is being governed by the OpenJS Foundation. Now, the project is in good hands having people like James Snell from Snyk, Michael Dawson from Red Hat, and Matteo Collina formerly from NearForm in its Technical Steering Committee - tirelessly driving success, assessing latest developments in the field, and keeping the legend of Ryan alive!

Notes: Advanced Node.js Concepts by Stephen Grider

Muhammad Bin Zafar — Sat, 19 Aug 2023 15:17:14 +0000

Here's one of the most common interview questions you'll face when looking for a Node.js job: "Can you explain Node's Event Loop?" There are two types of engineers: those who can describe the Event Loop and those who cannot! This course will ensure that you are incredibly well prepared to answer that most important question.

This is the preface to an advanced course from Stephen Grider, an Engineering Architect and a distinguished Udemy Instructor Partner based in San Francisco Bay Area, authoring engineering courses including the 3rd highest-rated course on React.js in all of Udemy, among 1000s of courses. Stephen has received over 400,000 reviews, and over a million students studied his published materials.

The Advanced Node.js Concepts course contains 16 aggregate hours of lectures divided into several sections:

The Internals of Node
Enhancing Node performance
Project Setup
Data caching with Redis
Automated Headless Browser Testing
Wiring Up Continuous Integration
Scalable Image/File Upload

The first two explores solely the Node.js anatomy in considerable depth, while the rest address several challenges using standard libraries and approaches in a suspensive manner. While benefitting from the course materials, I have greatly enjoyed the discoveries and solidified what I already knew.

Without taking notes information flies away, which is why I made sure to do that maintaining a quite high signal-to-noise ratio as Chuck Zerby mentions in his book The Devil's Details: A History of Footnotes. The notes may be the most helpful, if you already have an introductory idea on Node.js and willing to further it. It may also be advisable to follow Stephen alongside. However, I have included addendum wherever seemed relevent.

So, here we go, the notes, enjoy!

The Internals of Node.js

Node.js is not a new programming language, rather a runtime built on top of JavaScript. A programming language is primarily distinguished through its syntax. Node.js does not introduce any new syntax. In such a manner, TypeScript is a new programming language since it introduces new syntax.

In computing, a program can be defined by an executable file stored on disk, containing code or binary instructions for the processor. A process can be defined as an instance of a program, when it is loaded into memory and executed by the processor. The operating system allocates and manages resources, e.g. heap, stack pointers, registers, for the process in a secure isolated manner.

Threads are units of instructions to be executed from a process. Scheduling is the order of execution, and it refers to the ability of an operating system to decide which thread to process/execute in any given time. Some threads are more important than others. However, note the distinctions that:

In Node.js, threads are best suited for CPU-intensive tasks, not I/O-intensive tasks,^[1] for example any cryptographic computation, e.g. crypto.pbkdf2() will run fast and fs.writeFile() will not.
Also note that threads are not used in Node.js for network I/O, rather network requests are managed by libuv using underlying functions provided by the operating system, e.g. epoll for Linux, kqueue for Mac.
Further note that, having more number of threads does not necessarily improve the file I/O performance. File read-write performance depends on completely different set of factors, such as bus speed, bus bandwidth, storage device, etc.
Inspite of that, using a thread from the thread pool to perform file I/O is necessary, in order to enable asynchronicity. Having more threads does in fact mean that we can read more files simultaneously, but it doesn't mean the act of reading an individual file becomes faster 🤯

Thread pool (also known as the Worker pool) is a collection of worker threads that are managed by the libuv library. By default 4 threads are deployed by Node.js, and the long-running operations as referenced in the event loop, in truth, refer to thread pool tasks. Note that the event loop itself does not assign tasks to the thread pool, but only checks and runs pending callbacks when the thread pool tasks are complete. Thread pool tasks are assigned by the standard libraries e.g. fs module, themselves. Custom thread pools tasks can be assigned from a Node.js C++ addon through N-API and WebWorker Threads. Despite its relative feasibility, it may be suggested to use the builtin thread pool instead of creating new workers using the worker_threads module.^[2]

In the source code of the Node.js opensource project, lib folder contains JavaScript code, mostly wrappers over C++ and function definitions. On the contrary, src folder contains C++ implementations of the functions, which pulls dependencies from the V8 project, the libuv project, the zlib project, the llhttp project, and many more - which are all placed at the deps folder.

For example, the latest implementation of the exported function named pbkdf2 in the file lib/internal/crypto/pbkdf2.js contains a reference to a class named PBKDF2Job. However that class is never found among the JavaScript source code, rather found at the file src/crypto/crypto_pbkdf2.h - which is made available to internal JavaScript code through const {PBKDF2Job} = internalBinding('crypto');.

When does libuv and V8 come into play? The purpose of V8 is to translate C++ values into their V8 JavaScript equivalence. The libuv project provides file system access, some aspects of concurrency, and a lot of processing constructs on the C++ side.

To process threads well, multiple cores are introduced. A single core can process multiple threads, which is called multi-threading or hyperthreading. For example, let's say Thread 1 is reading from a file, and Thread 2 is multiplying 3x3. Reading a file always takes a non-zero amount time, which the OS can detect and put the thread on pause, to allow running Thread 2.

Threads inside the event loop. Node.js creates 1 thread by default, and executes all code in that one thread. There is exactly one Event Loop per Node.js process, however that Event Loop may initiate other smaller event loops within itself. Each iteration in an Event Loop is called a tick.

Stephen Grider mentions 5 major phases, checked by the event loop:

Pending callbacks of timers to run? e.g. setTimeout, setInterval
Pending callbacks of system operations and long-running operations to run? e.g. epoll(), inotify_init1(), kevent() are system operations, and Thread Pool tasks are long-running operations
Pause execution, continue only when:
- a new event done for system operations
- a new event done for long-running operations
- a timer is about to complete
Pending callbacks for setImmediate?
Pending close events to cleanup?

So, the Event Loop makes one more iteration if it detects any pending timers, or system operations, or long-running operations.

Is Node.js single-threaded? The answer isn't straightforward. The Event loop of Node.js is single-threaded, however file I/O operations, cryptographic operations, and some network operations run in multiple threads through the Thread pool.

Conclusion

I can't say I don't yearn for another deeper more advanced series of materials from Stephen Grider or someone alike. Nevertheless, I do think I have been well-packed enough to pursue the next steps in this learning journey by myself.

^{[1] Node.js API Docs: Worker threads}
^{[2] Node.js Guide Docs: Don't Block the Event Loop}

My journey from Vim/Neovim to VS code 🥺

Muhammad Bin Zafar — Mon, 12 Jun 2023 18:36:36 +0000

Disclaimer: potential standard-inclined content, reader discretion advised.

The following anti-monkeypatch standard-inclined mindset originated from the frustrations from prolonged exposure to and usage of semi-standard technologies, e.g. code editors, operating systems, desktop workspace, etc.

Opensource is good. As a movement to empowering developers, enabling innovations, and receiving critique to gradually agree on an industry standard - there is simply no alternative of opensource. Our world runs on opensource! However the aspect of opensource regarding its very nature, that is, most opensource projects do not receive the funding they deserve and may lack a strong leadership to govern the project. This may yield a project deficit in supporting its ever-evolving consumer base. Opensource software projects witnessed tremendous success in the early ages of commercial computer software - this is beyond doubts. However in the contemporary era, in which the complexity, diversity, and expectations of software is vital - a distinction is to be made between FAANG-backed or Fortune 500-backed opensource initiatives and other individual/communitarian ones.

This was a regular day in the life of mine as a Software Engineer - however, yet again, I caught myself searching for decentralized extensions individually developed on an average of 9 years ago for Vim/Neovim. Similar to Visual Studio Code, however... (checking vscode vs vim user count worldwide...)

(In a survey of 86,544 responses...)

Code editor	% Users
Visual Studio Code	73.71%
Visual Studio	28.43%
IntelliJ IDEA	26.82%
Vim	22.29%
Neovim	11.86%

... more likely that, Visual Studio Code extensions will be better 🙃. Plus, centralized "verified" extension support provide an upper hand in some cases.

In reddit, a question was asked, why shouldn't I use Vim? Arguments against Vim. The first reply received 92 upvotes. Feb 2019.

"Every once in a while you'll go through the rabbit hole of addng something cool, configuring another plugin, or changing colorscheme/syntax - that you forget doing actual productive work."

(This is too true! May I be forgiven! Unbelievable! Feels sad!)

One of the reply to the first reply was:

"That's why I started relearning vanilla Vim. Native Vim. Without plugins. For Java, Go, Swift - I use JetBrains products with ideavim [a plugin that enables Vim keybindings]."

I almost feel like taking one step further,

"I may just use Visual Studio Code, instead of Vim. Maybe Windows 11, instead of a Linux distribution that was created by a guy over a decade ago. Maybe the magic developed by JetBrains. Maybe Slack, instead of Matrix. Maybe Microsoft 365 w/ Copilot, instead of LibreOffice. These are products well-built and polished and proven through rigorous testing and extensive real-world usage.

Maybe, just maybe, I need to learn to appreciate professionals investing their time and life and company to build these incredible, impossible products."

The Fuss of Cross-Compiling C/C++ 😒

Muhammad Bin Zafar — Sun, 20 Nov 2022 04:04:30 +0000

This article was put together from my findings when I tried to compile libcpy for Windows 11.

Let's say we have some C/C++ code already written for gcc in linux. We want to compile them for Windows 11 as well. To have greater portability. How can we achieve this? Which toolchain to select? What are the obstacles? In this article we'll explore.

🧩 Les problèmes

How everything other than VS2022 fails? In CMake for VS Code, you can have some preset CMake configurations in CMakeSettings.json file. Also known as toolchain or "kits" for clang, mingw, cl, etc. However:

mingw doesn't have address sanitizers.
clang throws syntax errors for multi-line macro, same as cl.
cl doesn't fully support C99, so unsigned stack[va_arg_count] is an error because it "expected a constant expression" of int
all projects may not be portable in visual studio; if you clone a project using cmake from github and Open that folder in VS2022, then you're almost on your own with CMakeSettings.json or CMake presets
in VS code, you may have quite a hard time in larger projects, e.g. linking external shared libraries (i.e. vld_x64) with the compiled executable using CMake

🌟 La lumière de la solution

Enter Microsoft Visual Studio 2022. The pinnacle of excellency from Microsoft. An under-appreciated humble big-friendly-giant. A champ. A software specimen.

If you ask any airline "is this flight safe?", they reply something to the effect of "oh, your trip to the airport was the most risky part".

If you ask me, "how does it feel coding in VS2022?", I'd brighten up my charm and uplift my morale to say "double-clicking this app was the hardest part". T is liketh wat'r.

L'appréciation

So, a major reason behind migrating to Microsoft VS2022 was to have great support, great tooling, and ease in development. VS2022 has a ton of features/tooling/extensions/options/configurations. So much so that, in the beginning it may feel like an overkill or a bloat. However, each feature has a purpose of life, and each of them are helpful. Each of them are built over years of demand from low-level developers from around the global over more than a decade of feedback.

La confusion?

Let's say I have a Project for dll in C/C++ in a Solution. In the same solution, I have another Console App Project to use the library.

You turn on "Enable Address Sanitizer" from Project Properties on the console app - and the whole app crashes for some people. Apparently Visual Studio developers tried fixing this flaw. However this is not fixed yet.

The error output will almost spam your debug console with the following message in enormous quantity:

Exception thrown at 0x00007FFB12B64F69 (KernelBase.dll) in
main.exe: 0xC0000005: Access violation 
writing location 0x0000001F3966FCCE.

Please do note that this major problem is for x64 only. However, a dev from the Visual Studio team said that the silent violations are normal, and not a bug. This is incorrect because, these "silent normal access violations" cause:

all memory leaks to be undetected! (when compiled with Visual Studio's lovely library to detect memory leaks, <crtdbg.h>, which we'll discuss later)
causes deadlier, more messy, unfixable Stack overflow 😎 errors. (when compiled with Visual Leak Detector, <vld.h>)
crashes the application immediately for some people

La good news :D

Just don't turn on "Enable Address Sanitizer" for the console app. That's it. That's how all of the problems in the previous section can be completely mitigated!

Les armes 🛠️

There are quite a number of libraries and tools to detect memory leaks, but the following two worked great for me. Both of the library masterpieces work without enabling --fsantize=address, hence no access violation errors. Let's say we have a piece of code, and let's see how they turn out.

`#include <crtdbg.h>`

This is a system library from the C Runtime library of Microsoft Visual Studio. Therefore this library is available is no other platform except Windows. The library works exceptionally well, and it automatically binds itself to all app exit points, and outputs memory leak info.

Let's say the code we have is:

#define _CRT_SECURE_NO_WARNINGS
#define _CRTDBG_MAP_ALLOC
#include <stdlib.h>
#include <crtdbg.h>

int main() {
  _CrtSetDbgFlag(_CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF);
  char* str = malloc(6);
  strcpy(str, "hello");
  // free(str);
  return 0;
}

The corresponding output:

Detected memory leaks!
Dumping objects ->
main.c(8) : normal block at 0x0000020C7981FB50, 6 bytes long.
 Data: <hello> .... some hex values .... 
Object dump complete.

`#include <vld.h>`

This library is developed by KindDragon. Another acme of human intelligence. However the fork from Azure seems to fix an issue.

Let's say the code we have is:

#include <stdlib.h>
int main() {
  char* str = malloc(6);
  strcpy(str, "hello");
  // free(str);
  return 0;
}

The corresponding output:

WARNING: Visual Leak Detector detected memory leaks!
---------- Block 1 at 0x00000000C7C0D970: 6 bytes ----------
Leak Hash: 0x9C8CAD7D, Count: 1, Total 6 bytes
Call Stack (TID 13088):
ucrtbased.dll!malloc()
.... more call stack ....                                                                                             
.... more call stack ....
C:\Users\USER\main.c
.... more call stack ....
.... more call stack ....

Data: 
.... The readable value of the pointer: "hello" ....
.... outputs the surrounding heap ....
.... outputs the surrounding heap ....

💭 La final thoughts

Msys2 may help while coding in Visual Studio. And as we discussed, you may get VS Code to work correctly on a large project, however Microsoft Visual Studio 2022 would be the best practice! Apparantly you can also write code for linux from VS 2022, however gcc and gdb has complete support for address sanitizers.

📦 Bundle Node.js code into single executable binary

Muhammad Bin Zafar — Sat, 20 Aug 2022 12:13:00 +0000

Node.js 🐢, the asynchronous event-driven JavaScript runtime, has unparalleled support for file-system access, among other things - opening up the door to endless possiblities! However, Node.js often loses out to other runtimes/languages in cases where being able to package a single, executable application simplifies distribution and management of what needs to be delivered.

While there are components/approaches for doing this, they need to be better documented and evangelized so that this is not seen as a barrier for using Node.js in these situations. This is important to support the expansion of where/when Node.js is used in building solutions.

This article addresses 2 major concerns in the Node.js ecosystem: bundling and packaging. Let's talk about them briefly.

Bundling is the concept of merging the code, and all its dependencies into a single file. This is commonly seen for frontend development.

However, using the ESM packaging format has one advantage than CJS: tree-shaking. Tree-shaking is the concept of removing unused code from a dependency. Tools: esbuild, parcel, webpack, rollup, terser.

Packaging in Node.js is concept of creating a single executable binary, which includes the source code and the Node.js runtime. This way, Node.js will not be needed to be installed on end-user's machine.

During the process, the tool parses the source code, detects calls to require(), traverses the dependencies, and includes them into executable. Usually the source code is compiled into bytecode using the V8 engine. Tools: pkg, ncc, nexe.

`esbuild` to bundle

An extremely fast JavaScript and CSS bundler and minifier
Most convenient
Fastest in comparison
Support for TypeScript syntax, ESM, and CJS
Supports tree-shaking for ESM
Supports minification and source maps

# Output CommonJS bundle
$ npx esbuild index.js  --bundle --outfile=build.cjs \
--format=cjs --platform=node

# Output ESM bundle
# Note that, you may not need the --banner flag.
# But, in some cases, require() and __dirname are needed.
$ npx esbuild index.js  --bundle --outfile=build.mjs \
--format=esm --platform=node --banner:js="
import {createRequire} from 'module';
const require = createRequire(import.meta.url);
import { dirname } from 'path';
import { fileURLToPath } from 'url';
const __dirname = dirname(fileURLToPath(import.meta.url));"

`pkg` to package

Package your Node.js project into an executable
Instantly make executables for Windows, Mac, Linux, etc
No need to install Node.js, or hundreds of dependencies

# Packaging tools work best with CJS. 
# These tools don't go well with ESM.

# To package into executable, just take the file outputted
# by `esbuild`, and pass it to `pkg`, and we're done!
$ npx pkg build.cjs

This command will output 3 binary exectuable files build-linux, build-macos, and build-win.exe. You might want to run the executable file for your platform. Now you can simply distribute these files to your end-users or deploy in production - without installing Node.js or any dependencies or anything - just this one file!

Thanks for reading! Found it interesting? Give it a ❤️ or 🦄! Any topic you'd want to me cover? Let me know in the comments.

Have a great day!

How to correctly design REST APIs

Muhammad Bin Zafar — Thu, 18 Aug 2022 17:45:00 +0000

REST stands for "Representational State Transfer". This is an architectural style. A web API conforming to this style is a REST API.

REST stands on 6 guiding principles, important among them are:

Client-server: separate backend and frontend.
Stateless: each http request contains enough data (e.g. auth, session, user data) to understand it without knowing what the previous request was.
Cacheable
- GET: always
- POST: using http headers expire, cache-control, etag, last-modified
- PUT, DELETE: never

🧠 Concepts

The following are some important, concise, and simplified REST concepts a good backend dev must be familiarized with.

Resource

REST APIs are modeled as a resource hierarchy - where each node is either a collection, or a single resource. A single resource has some state, or sub-resources.

Example of a stateful resource is the following, where a user resource has data/states:

POST /users/:id

{
  id: 4e7d418a70f,
  name: 'muhammad',
  country: 'earth',
  verified: true
}

Example of a resource with sub-resources is the following, where a user has multiple projects, each of which are a resource.

GET /users/:id/projects
content-type: application/json

[
  {
    id: 'postman cli',
    desc: 'postman implemented in command-line'
    url: 'github.com/midnqp/postman-cli'
  },
  {
    id: 'typescript',
    desc: 'typescript is a superset of javascript',
    url: 'github.com/microsoft/typescript'
  }
]

Idempotency

API Idempotency means "a client can make a request multiple times, returning same response - without having any side-effect".

Making the following requests multiple times produces the same result, and has no further side-effect - thus are "idempotent" APIs:

GET /users/:id - just retrieving same user data
PUT /users - just updating same json data
DELETE /users/:id - just deleteing the same user

A non-idempotent API:

POST /users/:id - because a new user will be created everytime this request is made

🎨 Designs

Some constraints in REST API naming ensures a design of scalable API endpoints:

Use plural nouns, e.g. users, orders, categories.
Use hyphen, not underscores. Use lowercase, never camel-case, e.g. GET /food-categories, not GET /foodCategories.
Never use CRUD function names, e.g. GET /users/list or POST /users/create.

Some perfect API endpoints:

GET /users - get a collection/list of users.
GET /users/:id - get a single user.
POST /users/:id - create a single user.
PUT /users/:id - update a single user.
DELETE /users/:id - delete a single user.

For sub-resources:

GET /users/:id/projects - get projects of given user
DELETE /users/:id/projects/:id - remove a project of given user

Adding some features:

GET /users?country=earth&verified=true - search users by country and verified
GET /users?$fields=name,country - list users, but return only 2 data attributes: name and country
GET /users?$sort=name&$order=asc - list users and sort by name in ascending order
GET /users/$page=1&$limit=10 - list users and paginate with page and limit

🚫 Errors

Most Google APIs use resource-oriented API design. Instead of defining different NOT_FOUND errors, the server uses one standard NOT_FOUND status code, and tells the client which specific resource was not found.

The smaller error space has advantages:

reduces the complexity of documentation
better mapping
reduces client logic complexity

Model

type Error = {
  code: number
  message: string
  details: any[]
}

Error.code: simple code, easily handled by client
Error.message: Developer-facing human-readable error
Error.details: Additional error information for client, such as retry info, help link, etc.

Code

Individual APIs must avoid defining additional error codes.
Developers must use canonical error codes.
Standard error codes for Google APIs:
- 200 OK, not an error; returned on success.
- 500 UNKNOWN, internal server error; unexpected and insufficient info
- 400 INVALID_ARGUMENT, invalid/problematic user data
- 404 NOT_FOUND, something doesn't exist globally, for everyone
- 409 ALREADY_EXISTS, something already exists
- 403 PERMISSION_DENIED, access denied for a user; relevant people have access
- 401 UNAUTHENTICATED, no bearer token; no valid auth creds
- 429 RESOURCE_EXHAUSTED, too many requests; rate-limit exceeded
- 400 FAILED_PRECONDITION, the operation was rejected; business logic unmet
- 400 UNAVAILABLE, the operation was rejected; user should retry

Message

Error messages should help users understand & resolve API errors easily.

Do not assume the user to an expert user of the API.
Do not assume user knows anything about service implementation.
Should be constructed such that technical users can respond, and correct it.
Keep the message brief. If needed, provide a link to more info, questions, feedback. Otherwise, use "details" field.