Forem: Poorshad Shaddel

Why Implementing Your Own Username Password Authentication System Could Be a Big Mistake!

Poorshad Shaddel — Tue, 26 Dec 2023 16:17:29 +0000

This is an awareness about the dangers of implementing your authentication system!

Why I wrote this article?

In almost all the projects and companies that I have worked with, we had our own Username Password authentication system, as a result, we have faced a lot of challenges and this article is a summary of the challenges we had and what to consider before jumping into it.

Some of the reasons I think using our own Username-Password authentication could be a mistake:

Cybernews analyzed 15 billion passwords and found that nearly 30 percent of them featured just 8 characters , 20 percent had just 6 characters!
“123456” is still the most popular password (NordPass)
Nearly 6 in 10 Americans have used personal information such as their name or birth date in passwords (Google)
52% of internet users admit to resetting passwords regularly (ExpressVPN)
Weak passwords are the root cause of 81% of all data breaches (Verizon)
Just 45% of US consumers change their passwords after a data breach (Google)
Hackers leaked over 721 million passwords in 2022

Another super important factor in my opinion is the massive amount of time that we need for implementing a good quality authentication system. An authentication System is not something that you can deliver without a high test coverage, and that needs a lot of effort. In the next section we will see which features are needed for a authentication system and what to consider in implementation.

Authentication Features and Challenges

Choosing a Hash Function

Some of the hash functions are considered weak, this is a small list:

MD5, SHA-1, RIPEMD & RIPEMD-128 , Whirlpool

But probably this is not a big issue, since there are some other strong hash algorithms like Argon2id, bcrypt, scrypt.

Secure Storing of Secret

For hashing the password you probably need a secret. Keeping this secret secure is also really important. To avoid having this secret in your code, if you are using Kubernetes, you can use K8s Secrets.

Login with Email

If you want to allow users to Login with their email, you already need a new implementation for verification of email. You already need a mail service even before starting your project! This verification could be done by sending a code, or a link that results in verification and a valid user session. Testing this is doable by mocking the mail service.

Duplicated Emails

It is easy to implement but the problem is that you are giving the anonymous request sender some information about existing users emails! So as a result you have to use something like a rate limiter to avoid someone making a list of your application user emails by brute force.(preventing username or email enumeration attack)

Login or Register with Phone

You need an SMS service to verify that the user owns the number. If you are allowing register with a phone number, also here you should be careful about Enumeration attacks, the attacker can get some existing phone numbers and try to target those users with social engineering or phishing attacks.

Password Policy

Choosing the standard password policy is also important, For example:

Not less than 8 Characters
Up to 64 Characters
Allow usage of all characters including unicode and whitespace. There should be no password composition rules limiting the type of characters permitted.
Include a password strength meter to show the user how strong is their password!

zxcvbn-ts is a nice package for showing the strength of the password to the user.

This is the most basic functionality of an Authentication System. the User should be able to log in with a username or password!

Reset Password

We usually send a reset password link or a code to allow the reset password process. What to consider?

Sending consistent message(not email not found!)
Response time in cases that the email does not exist!
Tokens should be safe, single use and expriable!
Use a Rate Limiter for avoiding Bruteforce

Locking Account on Multiple Try

This is a logic quite similar to cell phones, if you enter the wrong pin a few times, you have to wait for a while to be able to try it again. In order to avoid brute force on specific accounts, you have to lock the account when someone is sending a lot of wrong passwords to that specific user. Implementing this logic requires also a policy for unlocking the account. For some use cases, they unlock the user after a period of time. For some other use cases only support users are able to unlock these users.

Session Management

Google Showing List of Your Active Sessions

Session manager might also be something specific to Finance sector, but I had to implement it twice! But honestly this is not a complicated feature to implement, it just adds a bit of complexity and a bunch of code and more tests to your codebase.

Risk-Based Authentication (RBA)

You see an email once in a while, when you try to Login from a new Browser or device in your Google Account. This restriction might be neccessary if you are working in finance sector. You need tell the user that there was a suspicious activity. Another result of this feature is that you need to store IP Address(or a Hash of it) somewhere to check each Login against this list of IP Addresses.

Persisting Sessions

This is also a question that you should answer usually the moment you want to implement your own authentication system. Nowdays everyone is using containers and deploy their applications on K8s or Serverless, as a result you do not have a monolith and you might have more than once instance of the application, so you cannot use memory as a store for your sessions. Usually it is recommend to use a key-value store, because Databases are slow(I doubt this one! Databases are not slow specially if you have the correct index, they might not be as fast as key-value stores but they are definitely not slow!). So you need something like Redis.

Security Concerns

These are only some of the security concerns and attacks which you should be familir with before thinking about your own authentication system.

BruteForce Attacks

For fighting bruteforce attacks, having a complex rate limiter is vital. The user should not try a lot of username at registration, otherwise it can make a list of users and later target this group. User should not be able to send a lot of password reset emails at once. My favorite package on Nodejs is this node-rate-limiter-flexible that supports complicated cases.

If you are intereseted in checking out the details of BruteForce Attack you can check this article from me:

Prevent Brute Force Attacks in Node.JS

Username Enumeration

The username enumeration is an activity in which an attacker tries to retrieve valid usernames from a web application. The web applications are mostly vulnerable to this type of attack on login pages, registration form pages or password reset pages.

SQL/NoSQL Injection

You always have to validate data that comes from the user and frontend. Sanitizing is also vital in case of Login, Register and Reset Password. Sanitzing means that you are not letting the user to pass something like a regex that could do something you do not expect in your code(for example fetching all users instead of one).

How to Prevent No-SQL Injection in Node.js

CSRF Attack

Cross-Site Request Forgery (CSRF) is an attack that forces an end user to execute unwanted actions on a web application in which they’re currently authenticated. For example the attacker could execute change email, change password. Usually we solve this issue by using a CSRF Token.

Prevent CSRF Attacks in Node.JS application

Session Fixation

The attacker has his own valid session and tries to associate it with another user. The problems occur when we are not generating new sessionIds(unique identifier) on actions like Login. Passportjs the most famous authentication library in nodejs had this vulneribility until version 0.5.0. Read the Passportjs maintainer article here.

I have also covered this issue on my medium

What is Session Fixation and How to Prevent it in Node.js

More Code, More Bugs!

I think this is not 100% true, the more accurate sentence will be “The More Code, The More Possibility of Bugs”. If we implement all of these logics in our code, even if we are aware of security concerns there is a good posibility that we see some bugs, in both category of security, and functionality. This is another reason why I wrote this article, all these logic could be handled by somthing like Google, Apple, Github or any third party authentication service and we can focus on our business logic.

More code = more bugs

Is there a Solution?

Definitely!, you can use Google Login, Apple Login, Github Login or Third Party Authentication Services like Keycloak, Auth0, AWS Cognito, FusionAuth and a lot of other options(both open source and closed source). By not implementing authentication yourself you are saving a lot of time and energy. My personal experience was that we could not ignore a good test coverage for authentication features and bugs, as a result we had to implement a lot of tests to be sure that everything is working as expected.

What if we loose some users because of not having the traditional username password?

This is a question that only your specific product and targeted users can correctly answer, but it is always a trade off, you might loose some users but you gain time and energy for building more business related features that might attract some more users.

Conclusion

Implementing our own authentication system could be a big security risk, and even for those who do not care that much about security(Do you really?), the cost of implementing all the features, tests and manual testing is really high. I think this is a mistake beacuse there are a lot of easier options which their main focus is this topic and the possiblity that they make mistakes is much less than us!

https://docs.xperience.io/spaces/flyingpdf/pdfpageexport.action?pageId=62069615

What is Session Fixation and How to Prevent it in Node.js

Poorshad Shaddel — Mon, 13 Nov 2023 00:18:18 +0000

By Session Fixation attacker can hijack a valid user session and it is absolutely important to know this vulnerability and protection against it.

Session Fixation and How to Fix it

Before diving into this we need to know what, the session is and how session authentication works. If you are already familiar with this you can skip to the part: What is Session Fixation or How to Prevent Session Fixation

What is Session?

As we know HTTP requests are stateless, it means that when we send my Login request, and we have a valid username and password there is not a default mechanism to know that I am the same person who is sending the next request. For solving this and in other words make the requests stateful there are some suggested ways like Cookies , Hidden Form Fields , URL Parameters , HTML5 Web Storage , JWT and Session. In this article we are focusing on Session.

Session is a data stored on the server. Each client is given a unique identifier associated with this data on the server. The client must send this unique identifier on each request, so we know who is sending this request. This identifier could be sent in a cookie or URL parameter.

A simplified example to show the session and identifier (sessionId) in an expressjs application:

const app = require('express')();
const session = require('express-session');
app.use(require('cookie-parser')());
app.use(require('body-parser').json());

app.use(session({
    secret: 'secret',
    cookie: { maxAge: 60000 },
    name: 'sessionId'
}));

app.get('/', (req, res) => {
    res.send('ping');
});

app.listen(3000, () => {
    console.log('Server is running on port 3000');
});

When sending a request for the first time the express-session middleware is creating a new unique identifier and sets it as a cookie and stores this somewhere (in this case memory, but we can pass our custom store too). In the options of session middleware, we have used sessionId as the name of our key that stores this unique identifier. Now if we send a request, we see something like this:

Session Middleware setting a new Session Id

Browsers now set this cookie and store it automatically for further requests. If we send a request that contains a valid session (the session exists in our session store - memory in our case) we do not get the Set-Cookie header in the response back:

No Set-Cookie When we have a valid session.

When the user signs in we can store the user information in the cookie(serialized) or we can store it in a database and associate the data with the sessionId. Let’s use a Map as our database:

const db = new Map();
app.get('/me', (req, res) => {
    const user = db.get(req.sessionID);
    res.json({ mySessionId: req.sessionID, me: user ? user : 'anonymous' });
});
const users = [{ name: 'bob', age: 19 }, { name: 'joe', age: 20 }];
app.post('/login', (req, res) => {
    const { name } = req.body;
    const user = users.find(u => u.name === name);
    if (user) {
        db.set(req.sessionID, user);
        res.send('ok');
    } else {
        res.send('try again');
    }
});

If we login and then use the cookie to send another request to /me we get this result:

User Data associated with Session Id

This was a simplified summary of why we have to use session and how we can do that.

Can an Attacker Create a Valid Session Id?

In this case that we were using express-session . you saw that we passed a secret to session middleware. This secret is used for signing the value of our cookie. It simply means that we are sure that it was us who generated the sessionId. So as long as you are sending signed values to your clients, it is not possible.

A sample of session:

sessionId=s%3AL6j4T8hBwMk1ulJqGoisZbAxUOkOuQqP.x5UxPQEtKrj3sWrIy6S01CQRjAtp4biVs4H2zgqmSs

First Part: s%3A simply means: s: which is a prefix to indicate that our cookie-session is signed!

Second Part: L6j4T8hBwMk1ulJqGoisZbAxUOkOuQqP this is our sessionId, we are using this in our database for associating data.

Third Part: This is the third part: x5UxPQEtKrj3sWrIy6S01CQRjAtp4biVs4H2zgqmSs This is the signing part. We have generated this text by using our secret, so we can be sure that this cookie is generated by us.

We can simply regenerate this sign and check if it is valid or not:

const crypto = require('crypto');
const secret = 'secret';
const sessionId = 'L6j4T8hBwMk1ulJqGoisZbAxUOkOuQqP';
const hmac = crypto.createHmac('sha256', secret);
hmac.update(sessionId);
const signature = hmac.digest('base64').replace(/\=+$/, '');
console.log(signature); // x5UxPQEtKrj3sWrIy6S01CQRjAtp4biVs4H2zgqmSs

This is how express-session is checking it.

What is Session Fixation?

In Session Fixation attacks, the attacker hijacks a valid user session. We said that we sign the cookie in order to be sure that no one can hijack another user's valid session. But what if the attacker has his own valid session and tries to associate it with another user? in this case he can perform actions on behalf of the victim.

The problems occur when we are not generating new sessionIds(unique identifier) on actions like Login.

How can the Attacker do this?

One of the cases is when attacker has physical access to the computer. As an attacker, I go to the university and I choose one of the shared computers, then I sign into my account on the vulnerablewebsite.com and then without doing the logout (which normally destroys the session in the server store), I leave an open login page on vulnerablewebsite.com and before that I have to copy my valid sessionId. Now the victim is using this computer and if the victim signs in, the attacker sessionId is associated with the victim's account. Sounds complicated? Not at all, let’s see that in action:

Let’s login with our first user Bob(Attacker):

Now the browser sets this cookie for this website. It means if someone else tries to send the login request, express-session is not generating a new sessionId and what it does is overriding the existing sessionId.

Let’s imagine Joe (victim) decides to use this shared computer, the cookie and the valid session of Bob is also sent:

Login with sending an existing valid session

We did not get a new session or cookie!

The magic happened and now the sessionId of Bob is associated with Joe’s user. So, if the attacker (Bob) sends a request to /me he will get data of Joe back:

Using Bob Session Id for getting Joe Data

We were able to get Joe Data by using the Session of Bob. In this example the attacker had physical access, but it is possible to do it without physical access if there are some other vulnerabilities such as XSS.

Some websites are passing sessionId as URL parameters in requests. In this case if the attacker gives a link to login page with his sessionId on URL parameters there is a possibility to exploit.

Session Fixation by URL Parameter

Read more about security challenges of this method in this stack exchange question.

How to Prevent Session Fixation?

Generate New Sessions on Login!

The main solution is really easy, by doing that you are always sure that this session override is not happening!

Let’s change our code:

app.post('/login', (req, res) => {
    const { name } = req.body;
    req.session.regenerate(err => {
        if (err) {
            res.send('error');
        } else {
            const user = users.find(u => u.name === name);
            if (user) {
                db.set(req.sessionID, user);
                res.send('ok');
            } else {
                res.send('try again');
            }
        }
    });
});

We can use regenerate function, in order to assign a new session each time someone wants to log in. It does not matter anymore if you pass a session cookie or not, it will generate a new session Id and sends it to the client in the Set-Cookie Header.

Each Time We Log in we get a new sessionId and Set-Cookie Header

Only Use HTTP Only Cookies

When you are using HTTP Only, it means that only server can set cookies by Set-Cookie header and the client side (Browser JavaScript) cannot change it. So even if your app has a XSS vulnerability, the attacker cannot change sessionId (cookie).

Protect against XSS

Session Fixation can be combined with XSS attacks to be more effective, so it does make sense to take XSS attacks also serious if you are concerned about session fixation.

Sensible Session Expiration Time

The sessions expiration should match your application specific requirements, if you are more concerned about security then it should be shorter and vice versa.

Correct Logout Implementation

On Logout you have to correctly destroy the existing session and its association with any data. Otherwise, these sessions could be used after logout. (It is not enough to remove the cookie from client browser!)

Was Passportjs Vulnerable to Session Fixation?

Yes, in versions before 0.6.0 the issue was there, Passport maintainers thought that session regeneration should be done on the application side, but after a while they realized the importance of the issue and they fixed it in 0.6.0 version. If you are interested in the details of this fix, you can read all the details here.

Conclusion

Session fixation could occur in case of overriding an existing sessionId with another user data. The solution is quite straight forward by generating a new session each time someone signs in, using HTTP Only cookies, proper expiration time, correct logout implementation.

References

https://owasp.org/www-community/attacks/Session_fixation#

https://developer.mozilla.org/en-US/docs/Web/Security/Types_of_attacks#session_fixation

Implement a Simple Version Control with JavaScript to Understand Git Better!

Poorshad Shaddel — Sun, 29 Oct 2023 21:35:16 +0000

Learn Git Internals by trying to implement a simplified version of it!

What is Git or generally Version Control?

Simply something that helps us track our project over time. One good example is that we can easily go back in time in our source code and see how the code looked like at a specific time in the past.

Why do we need to understand it?

First of all, because when something goes wrong just blindly knowing some commands will not help. Second of all, if we do not understand how the thing, we are working with it every day is working, then where is the fun?

Now we will understand the concepts one by one in the way of implementing our super simplified version control called Gitj

Implementation

Keep in mind that git does so many more advanced things like compression and the way thing are stored that I might cover in separate articles.

Implement the first command: Init!

As you probably know we start the project by git init and git creates the .git folder and stores data in it. We are going to implement this:

Two important folders that git creates are refs and objects. Objects are building blocks of git, they can have 3 types(actually 4!): commit , tree and blob . We are going to see each one of these types in details. refs folder has a subfolder called heads that contains our branches and the latest commit of each branch(from the name it should be kind of obvious that it stores the head of the branch). We have also an important file called HEAD which keeps the current branch or commit.(sometimes we want to checkout on a commit and not a branch)

So let’s create these folders when someone is calling our init function.

const fs = require("fs");

function init() {
    // creates a folder called .gitj and creates a subfolder called .gitj/objects and .gitj/refs and .gitj/refs/heads
    fs.mkdirSync(".gitj");
    fs.mkdirSync(".gitj/objects");
    fs.mkdirSync(".gitj/refs");
    fs.mkdirSync(".gitj/refs/heads");
    // creates a file called .gitj/refs/heads/master
    fs.writeFileSync(".gitj/refs/heads/master", "");
    // creates a file called .gitj/HEAD
    fs.writeFileSync(".gitj/HEAD", "ref: refs/heads/master");
}

init();

Git Add! Put files into Staging

In git files can have three different stages:

Git Files Stages

Working Directory : your normal directory that you work on it and you change files and folders and structures.

Staging Directory: This is a snapshot of a moment in your working directory and when you use git add command it is actually copies your files to .git folder. Keep in mind that staging files are somehow final draft that you want to later commit to your repository.

Repository : By running commit command you are creating a new snapshot in your repository. Now you can point to this snapshot by using the SHA Hash of this commit(you could not point to staging files, staging files were kind of draft).

Now let’s implement git add command.

The steps we should do in git add:

Read the file content
Hash the content of the file
Use Hash as file name and store it in the objects folder
If the file is already there, do nothing. If you have 10 files with identical contents (even with different file name and folder location) git does not copy them 10 times, it can re-use the blob.

Also, git uses first two letters of hash and creates a folder name. For example, if the hash is 4f9be057f0ea5d2ba72fd2c810e8d7b9aa98b469 . git stores it in this folder: 4f and creates this file with the rest: 9be057f0ea5d2ba72fd2c810e8d7b9aa98b469 . The reason is that when over time there are a lot of files in a single folder accessing files could be slower and by using the first two letters as a folder git tries to prevent this problem.

const fs = require("fs");
const crypto = require("crypto");
function add(filename) {
    try {
        // file exists?
        fs.accessSync(filename);
        // read the file
        const content = fs.readFileSync(filename);
        // hash the file
        const hash = crypto.createHash("sha1");
        hash.update(content);
        const sha = hash.digest("hex");
        // create a folder with the first two characters of the hash if it doesn't exist
        if (!fs.existsSync(`.gitj/objects/${sha.slice(0, 2)}`)) {
            fs.mkdirSync(".gitj/objects/" + sha.slice(0, 2), { recursive: true });
        }
        if (fs.existsSync(`.gitj/objects/${sha.slice(0, 2)}/${sha.slice(2)}`)) {
            // a blob with the same content already exists
            process.exit(0);
        }
        // write the file to the objects folder
        fs.writeFileSync(`.gitj/objects/${sha.slice(0, 2)}/${sha.slice(2)}`, content);
    } catch (error) {
        console.log(error);
        console.log(`File ${filename} does not exist.`);
        process.exit(1);
    }
}

add('./sample/src/readme.md')

After running add.js and adding a file from your source code you should see a file in your objects repository like this.

How your Gitj folder should look like

Commit the Changes!

Commit itself is also an object type and as you are probably guessing it should create more files in our objects folder.

The intention of commit is to make a pointer to the current situation (files and folders) that we can point to and in the future, we can come back to this state.

A commit object in Git contains this information:

Author: The person who made the changes
Committer: The person who commit the changes (sometimes you receive a patch from someone else and you have to commit changes.)
Date of commit
Commit Message
Tree (which is itself another object that keeps the shape of working directory at the time of creation)
Parent(If exists)

Let’s see how the Git Commit looks like

If we use command git log we can see the list commit hashes and if we copy one of them, we can use the command git show --pretty=raw commitHash . Here is an example of result of this command(Date is represented as timestamp after the commiter and author name):

Result of Git Show Commit

In this example we have a parent but if the commit is the first commit ever, then we do not have a parent.

The reason the parent exists inside commit object is that by doing that we can chain these commits together and if we manually change a commit in the past everything goes wrong because all the commit hashes should be recalculated. This is something that we do by using commands like rebase .

What is Tree object type in Git?

It is a type of object that keeps the shape of folders and files. For exampe in the commit example that I showed, we can checkout the Tree object and see what is inside. For seeing the tree we have to use git ls-tree treeHash . Here is an example:

Git LS Tree Command

This is the base tree. It contains the files and folders of my working directory. As we can see it has two different types: blob which are files, and tree which points to another tree object that represents a sub folder in this case.

In the result second column is the type of object, third one is the SHA and the last one is file or folder name(We already mentioned that keeping file name out of blob helps to reuse the exact same content over and over). The only thing we do not know yet is the first column. First column is File Mode. The file mode specifies the type of the object (e.g., blob, tree) and its permissions. The leading numbers like 040000 or 100644 represent the file mode in octal notation. The most common modes are:

100644: Indicates a normal file (blob) with read-write permissions.
100755: Indicates an executable file (blob) with read-write-execute permissions.
040000: Indicates a directory (tree).

Actions needed to implement commit functionality:

Create a Tree of the current working directory
Create the commit object
Get the parent commit(Head), if there is not parent update the head(master) after this commit.

We are going to create a simple tree that generates the structure of files and folders in the same way that git does.

Implement Tree generator function!

Small function for getting the file mode of each file:

async function getTreeFileMode(fileType, fileOrFolder) {
    const { mode } = await fs.stat(fileOrFolder);
    return fileType === 'tree' ? '040000' : '100' + ((mode & parseInt("777", 8)).toString(8));
}

A function to get hash of a file

async function getHashOfFile(path) {
    const content = await fs.readFile(path);
    const hash = crypto.createHash("sha1");
    hash.update(content);
    const sha = hash.digest("hex");
    return sha;
};

Now the main function

async function createTreeObjectsFromPaths(folderPath) {
    let treeFileContent = '';
    let treeHash = ''
    // we want to create a tree object similar to git ls-tree
    const listOfFilesAndFolders = await fs.readdir(folderPath, { withFileTypes: true });
    // if it is a file we want to store the hash of the file, if it is a directory we want to call this function recursively
    for (const fileOrFolder of listOfFilesAndFolders) {
        const fileType = fileOrFolder.isDirectory() ? 'tree' : 'blob';
        const fileName = fileOrFolder.name;
        let fileHash = '';
        if (fileType === 'tree') {
            const treeHash = await createTreeObjectsFromPaths(`${folderPath}/${fileName}`);
            fileHash = treeHash;
        } else {
            // here we need to calculate the hash of the file
            fileHash = await getHashOfFile(`${folderPath}/${fileName}`);
        }
        const fileMode = await getTreeFileMode(fileType, `${folderPath}/${fileName}`);
        const fileModeAndName = `${fileMode} ${fileType} ${fileHash} ${fileName}`;
        treeFileContent += fileModeAndName + '\n';
    }
    const hash = crypto.createHash("sha1");
    hash.update(treeFileContent);
    treeHash = hash.digest("hex");
    // write the tree object to the objects folder
    if (!await folderOrFileExist(`.gitj/objects/${treeHash.slice(0, 2)}`)) {
        await fs.mkdir(`.gitj/objects/${treeHash.slice(0, 2)}`, { recursive: true });
    }
    if (await folderOrFileExist(`.gitj/objects/${treeHash.slice(0, 2)}/${treeHash.slice(2)}`)) {
        // a tree with the same content already exists
        console.log(`.gitj/objects/${treeHash.slice(0, 2)}/${treeHash.slice(2)}`);
        return treeHash;
    }
    // write the file to the objects folder
    await fs.writeFile(`.gitj/objects/${treeHash.slice(0, 2)}/${treeHash.slice(2)}`, treeFileContent);
    // console.log(`.gitj/objects/${treeHash.slice(0, 2)}/${treeHash.slice(2)} \n`, treeFileContent);
    return treeHash;
}

Explaination of this function:

1- We are iterating to result of fs.readdir .

2- If it is a directory then type is tree and we are calling this function recursively. Otherwise it is a file or blob in other words and we need to calculate the hash.

3- We get the file mode(040000, 100644, …) because we need it for creating our tree object.

4- We have the content of tree object. Now we can create the hash.

5- if the object(hash of current tree) exists we do not need to do anything otherwise we create the object and store it into our objects folder.

Let’s run this function and see if it is working correctly:

The folder structure look like this.

Folder Strucutre

Now we run createTreeObjectsFromPaths('.') . As a result two new objects are created in our Gitj folder:

Two new objects after creating our tree object

We expect the content of one of these objects to have package.json which was in the root folder as a type of blob and we expect another tree object which points to src folder:

Root Folder Tree Object

This tree commit hash now refers to another object that stores structure of src folder:

src Folder Tree Object

Now it’s time to implement Commit function.

const fs = require('fs').promises;
const crypto = require('crypto');
const { createTreeObjectsFromPaths, folderOrFileExist } = require('./tree');

async function commit(commitMessage) {
    const treeHash = await createTreeObjectsFromPaths('./sample');
    const parentHash = await getLatestCommitHash();
    const author = 'test';
    const committer = 'test';
    const commitDate = Date.now();
    const commitContent = `tree ${treeHash}\nparent ${parentHash}\nauthor ${author}\ncommitter ${committer}\ncommit date ${commitDate}\n${commitMessage}`;
    const hash = crypto.createHash("sha1");
    hash.update(commitContent);
    const commitHash = hash.digest("hex");
    // write the commit object to the objects folder
    if (!await folderOrFileExist(`.gitj/objects/${commitHash.slice(0, 2)}`)) {
        await fs.mkdir(`.gitj/objects/${commitHash.slice(0, 2)}`, { recursive: true });
    }
    if (await folderOrFileExist(`.gitj/objects/${commitHash.slice(0, 2)}/${commitHash.slice(2)}`)) {
        // a commit with the same content already exists
        console.log(`.gitj/objects/${commitHash.slice(0, 2)}/${commitHash.slice(2)}`);
        return commitHash;
    }
    // write the file to the objects folder
    await fs.writeFile(`.gitj/objects/${commitHash.slice(0, 2)}/${commitHash.slice(2)}`, commitContent);
    // set the head of current branch to the commit hash
    await fs.writeFile('.gitj/refs/heads/master', commitHash);
    return commitHash;
}

The hard part was the tree object, now we have all the data and we put them together and create a new object for commit. Also, we need to update the head of the branch and point to this new snapshot that we got (obviously the commit hash).

Git Checkout

What does branch Master or Main or other branches mean?

Branches are simply reference or bookmark to a commit. In the implementation of commit you saw that since we updated the content of file: refs/head/master , it is just a commit hash. And this commit hash has a parent(if it is not the first commit ever) and we can go back to history until there is no commit anymore. So, by using this branch name we can access the latest commit (Head!). Simply, being on a branch means that you are pointing to another head.

it does not store the file name with the blob, then the advantage is that git can use the blob even if the file names are different.

How to implement Git Checkout!

Since we already implemented commit functionality, we know that in a commit we have access to tree object (recursively all the folders and files) and we have files as blob in the .git(for us .gitj). So we should remove the working directory and then build the whole directory when someone is checking out on another commit(or branch — branch head points to a commit hash). But first of all we need to store the commit hash or the branch name to the file HEAD .

Before that we want to implement a small function to get the tree object of that commit:

async function getTreeHashFromCommit(commitHash) {
    const commitContent = await fs.readFile(`.gitj/objects/${commitHash.slice(0, 2)}/${commitHash.slice(2)}`, 'utf-8');
    const array = commitContent.split('\n').map(e=> e.split(' '))
    const elem = array.find(e => e[1] === 'tree');
    return elem[2];
};

Now by having the tree we have to rebuild the whole folder:

async function checkout(commitHash) {
    const listOfFilesToCreate = [];
    // store the commit hash in the refs folder
    await fs.writeFile('.gitj/HEAD', commitHash);
    const treeHash = await getTreeHashFromCommit(commitHash);
    // get tree file
    const baseTree = await convertTreeObject(treeHash);
    // clear the folder
    await removeAllFilesAndFolders(folderPath);
    // create the files and folders based on address on the blob
    await createFilesAndFolders(baseTree, folderPath);
}

First we are writing this commit to HEAD file to remember that the head is not on master anymore. We have to recursively get the tree and their blobs to re-create the whole folder. Here is the implementation:

async function convertTreeObject(treeHash, folderPrefix = '', files = []) {
    const treeObject = await fs.readFile(`.gitj/objects/${treeHash.slice(0, 2)}/${treeHash.slice(2)}`, 'utf-8');
    const array = treeObject.split('\n').map(e=> e.split(' '))
    for (const file of array) {
        if (!file || file.length < 2) continue;
        const [mode, type, hash, name] = file;
        if (type === 'tree') {
            await convertTreeObject(hash, folderPrefix + name + '/', files);
        } else {
            files.push({
                mode: mode,
                type: type,
                hash: hash,
                name: folderPrefix + name
            })
        }
    }
    return files;
}

If we have a file then we add the name file and the blob( content of the file) to an array. if the object type is tree it means that it is a folder and we need to call this function recursively and we have to path the parent folder to create the correct path to the file.

Some more functionalities that I would like to implement in future

Git Status
Git Diff

Conclusion

We have learned how git uses hashes and chaining of commit hashes to give us the ability of tracking the history. Personally, I find this kind of deep dive the best way to learn something really good, I hope it was useful. If you are interested in implementing more functionalities in git checkout this GitHub Repo.

References

https://www.youtube.com/watch?v=lG90LZotrpo&ab_channel=CS50

https://www.youtube.com/watch?v=P6jD966jzlk&pp=ygUgR2l0IGludGVybmFscyBob3cgaXQgc3RvcmVzIGRhdGE%3D

https://www.youtube.com/watch?v=dBSHLb1B8sw&ab_channel=GOTOConferences

https://www.youtube.com/watch?v=52MFjdGH20o&ab_channel=Brief

Comparing Prisma and Mongoose for MongoDB: A Comprehensive Analysis

Poorshad Shaddel — Sun, 03 Sep 2023 15:54:29 +0000

Mongoose is the most used ODM with MongoDB and Prisma is a modern ODM/ORM that supports MongoDB in this article we are going to do a complete comparison between these two in various areas!

Prisma vs Mongoose — Designed by Marisa Habibijouybari

I have used both Mongoose and Prisma in production for a few years and I want to share some experiences and information about using these two famous open source libraries. Prisma supports relational databases and it has been a while since it supports MongoDB and I think now is a good time for a comparison between these two!

What you will read in this article

We are going to compare these two libraries in various parameters. Here are the areas of comparison:

Learning Curve
Community
Documentation
Performance
Typescript Experience
Some Interesting Open Issues

Learning Curve

Mongoose Learning Curve

Mongoose has a bit shorter learning curve in my experiments and for using it there are only a few things that you need to know to run your first query using Mongoose! A simple piece of code which is on the home page of Mongoose documentation shows how easy it is to use:

const mongoose = require('mongoose');
// Connection
mongoose.connect('mongodb://127.0.0.1:27017/test');

// Schema
const kittySchema = new mongoose.Schema({
  name: String
});
// Model
const Cat = mongoose.model('Cat', kittySchema);

 // Option 1
const kitty = new Cat({ name: 'Zildjian' });
kitty.save().then(() => console.log('meow'));

// Option 2
const kitty2 = await Cat.create({ name: 'Zildijan' });
console.log(kitty2)

The first part is the database connection. It returns a promise but you do not need to wait for it.

The second part is the schema definition which is simply the structure of your collection. We use this schema object to create our database Model.

Now for creating a new document in our collection, we have two options that you can see. Option one creating a new object and calling .save() method and the second option is to call Model.create({...}) .

To use it we need to know two concepts of Schemaand Model. We use schema to create our database model. That is the only thing you need to learn to use Mongoose.

Prisma Learning Curve

For using Prisma first thing we need is a prisma.schema file. This is a file type with its own syntax which is not so complicated and you can quickly adapt. This is where we define our collections’ structure. They all exist in this single file(which sounds not good but it has some benefits, for example, you have a version of the database structure in your version control system that you can switch into). Our connection also exists in this file(address of environment variable if we want to be more accurate).

Let’s look at a schema file:

datasource db {
  url      = env("DATABASE_URL")
  provider = "postgresql"
}

generator client {
  provider = "prisma-client-js"
}

model Cat {
  id          String  @id @default(auto()) @map("_id") @db.ObjectId
  name        String?
}

In the database source, we define the name of the env variable that we want to use for our connection string.

The second part is the client and generator address which we did not pass here. Prisma creates a client API based on this schema. Generator is important because each time we change our schema we have to run this command npx prisma generate to update Prisma Client(and typing of client).

We need to run another script to also apply changes of schema to the database. An example is creating a new index. We have to manually apply it to the database using Migrate:

npx prisma migrate dev --name

Now we can use our Prisma client to create a new Cat:

import { PrismaClient } from '@prisma/client'

async function createNewKitty() {
  const prisma = new PrismaClient();
  const kitty = await prisma.user.create({
    data: {
      name: 'Elsa Prisma'
    },
  });
  console.log(kitty);
}

Summary of this part: So overall to use Mongoose we need to understand Mongoose Schema and Model. For using Prisma we need to understand schema definition in schema.prisma file, Generator, and Migration. Prisma has a longer learning curve compared to Mongoose.

Community

Both projects have quite a good community which is satisfactory. Mongoose has been around for a long time and it has a big community and eco-system. On the other hand, Prisma has a more active community from my perspective. They have an active Slack channel and also a Discord channel. We should not also underestimate the Prisma Youtube Channel since there is a lot of good content created and they have live sessions after each release and they are presenting changes and answering questions in live.

Probably the difference you see here is because Prisma is backed by a dedicated company that manages these channels.

Summary of this part: Both have a good enough community but Prisma‘s community is much more active(Supporting No-SQL is quite new).

Documentation

Index of Documents of Mongoose and Prisma

Mongoose documentation is nice and sufficient for doing the job. The style of the documentation looks a bit old-fashioned but nothing to complain about. Prisma documentation looks modern. One thing that I find attractive in Prisma documentation is the separation of Concepts and Guides. We have mentioned the Prisma learning curve, and I think good documentation covers you quite well in the process of learning Prisma.

Summary of this part: Both have really good documentation!

Performance

In order to compare these two libraries in terms of performance I am going to compare the queries generated by these two libraries.

Find By ID or something Unique

We are trying to find a document by using something unique like _id.

FindById Prisma vs. Mongoose

As we can see the User.findById Mongoose directly translates to findOne({ _id: "_id" }) which is the most efficient way of fetching a document.

For Prisma, it was translated to an aggregation pipeline. The first stage is $match and it contains a few single-element $and phrases and also one empty operation phrase {} . There is also a $ne with using $$REMOVE probably for conditional exclusion of fields. And in the end, we have a $project that exposes the fields that we have used in our Prisma Schema.

Does using an aggregation pipeline make it slower? Practically No, the speed of fetching documents highly depends on using an index(Check out my MongoDB Index Challenge). After the documents have been fetched doing some extra steps like projection or addFields does not take a lot of time unless we have an enormous number of documents.

Another interesting behavior is that if you have some extra fields in the database which does not exist on the schema, Prisma does not show them to you because it uses the projection even when you are not passing the projection object to it. But in Mongoose it returns the whole object from the database. it is not a positive or negative behavior(especially if we are using No-SQL) out of context.

Generally, the Mongoose query generator seems less complicated and straightforward, but in terms of performance, I think it does not make a big difference, since most of the time we want to project some of the fields and not all of them. Especially if we have a findOne like this the projection step in the aggregation pipeline is negligible.

Insert Many Documents

Queries are almost the same, on Mongoose we call insertMany (Be careful that create method generates some insertOne queries), and on Prisma we call createMany and we pass the documents.

Inserting many documents using Prisma and Mongoose

As we can see both generate the same insertMany queries on MongoDB, the only difference is that Mongoose generates the _id and passes it to MongoDB(We can access _id if we use new User([...])) and it can be sometimes really helpful when we want to use _id in some other objects or pass it to the response of a request.

Update Documents

Updating documents is a bit different. Prisma only allows you to use a unique field for fetching a document. Beforehand it does a find(in Prisma world it is an aggregation pipeline) and returns an error if it cannot find the document! It also does another fetch after the update. So normally you have 2 more steps if you want to use Prisma for updating documents. One fetch beforehand to guarantee that the document is there and one fetch afterward to get the document. Mongoose simply translates your query to a updateMany or updateOne in MongoDB nothing more and nothing less.

Someone might argue that this is not a common way when you have an ODM or ORM. That’s true and when you fetch the document it gives the object with extra methods and helpers on it that you can change the object and then you can call .save() method on the object to persist it to the database, but on Prisma, there is no option for that, however, you can pass your fetched user object to update method and reassign the object.

Conclusion of this part: Prisma builds queries quite differently (using the Aggregation pipeline). Prisma also does an extra fetch after create or update. For update Prisma forces you to only create a folder with unique fields and also it does the fetch twice once before it and once after it.

Mongoose queries are quite similar(at least now!) to the way that we use MongoDB native driver, which is why I like it. Prisma used to be a SQL-ORM and probably that is the reason why its query builder behaves a bit differently. Prisma could improve itself by letting you customize this fetch behavior or even customize the logic of forcing using unique fields for updates(Let me know in the comments section if there is a way!).

As an end-user, you might not feel that much performance difference in my opinion.

Typescript Experience

Hopefully, with the new version of Mongoose, we can create a type from a Schema.

const userSchema = new mongoose.Schema({
  age: { type: Number, required: true },
  name: { type: String, required: true },
  email: String,
});
type User = InferSchemaType<typeof userSchema>;

With Prisma the Prisma client has already our types based on field definition in Prisma schema:

User type in Prisma Client

For Model Types, both give nice Type definitions.

Let’s see the type definitions for queries.

Find Queries Typings

Mongoose gives you the User type as the result of the find query but it unfortunately does not support typing when you project only some of the fields:

Mongoose Typing in the result of Find Query

To implement projection typing in Mongoose, you need a little bit of Typescript tricks like this:

type FindParameters = Parameters<typeof User.findOne>;
type Filters = FindParameters[0];

function findOneWithTyping<T extends mongoose.ProjectionType<User>>(
  filter: Filters,
  projection: T
): Promise<Pick<
  User,
  keyof typeof projection extends keyof User ? keyof typeof projection : never
> | null> {
  return User.findOne(filter, projection) as any;
}

async () => {
  const user = await findOneWithTyping({ name: "John" }, { name: true });
  if (!user) throw new Error("User not found");

  const age = user.age; // Error
};

Prisma perfectly supports the typing of projection in find queries.

Prisma typing in find queries

Model Relationship Typing Support

Both support relation typings. But Mongoose has the same problem with projection and also you do not have nested projection typings. Prisma does have these capabilities:

Prisma Nested Projection

Aggregation Typings Support

Unfortunately, Mongoose does not support aggregation pipeline typings and it always types the result as any[].

Mongoose Aggregation Pipeline Typing is always an array of Any

There are two options for an aggregation pipeline building with Prisma. The first one uses aggregate method but it supports only a limited number of aggregation operations but it supports typing with this method:

Prisma Aggregate Method

You can alternatively use aggregateRaw method which supports all aggregation operations, but it does not support aggregations.

Prisma AggregteRaw Method does not support typing

Summary of this part : Both support Typescript to some extent but Prisma supports most use cases. Mongoose is not perfect but with some extra Typescript tricks, you can build what you need.

Some Interesting Open Issues

At the time of writing this article Prisma has 2.7k issues and Mongoose has only 257 open issues. One of the reasons is that Prisma supports different databases and each have a lot of speciel cases and bugs. Also Mongoose has been around for a long time and most bugs are already fixed.

Here we want to check out some of the debatable and interesting open issues of both.

Prisma Open Issues

Add runtime validation to models · Issue #3528: This issue suggests that Prisma could expose the validation function so people can check their object before trying to perform database actions. Also, some people might want to have some custom validations. This issue is open from 2020 I think. Mongoose already has this feature. I think if Prisma is not implementing this, then you could have a little bit of duplication for validation using Zod or something else.

Soft deletes (e.g. deleted_at) · Issue #3398: This is a very interesting idea! Prisma does not support it directly at the schema level but there are some ways to implement this using Middlewares. Mongoose has also some workarounds that work with hooks.

Support for splitting Prisma schema into multiple files · Issue #2377: This is open from 2020, people want to split their schema into multiple files. Some people have suggested some scripts to do that. You can even implement the script yourself for appending and mixing multiple files into one schema.

Add exclude in the query arguments · Issue #5042: This is also open for a long time and I think for MongoDB it is important to have this since it does support exclusion projection.

Support for a Union type · Issue #2505: Supporting union types is also one of desirable features that users asked Prisma. They want to use multiple types(table or collection).

Subscriptions/real-time API support · Issue #298: Having subscriptions and real time support is also another requested feature. In my perspective it is noramlly a ORM/ODM does not need to give this feature since its job is something else. But if they provide something it will save a lot of poeple a lot of time.

Nested relations using createMany · Issue #5455: This feature request is asking for creating nested records. For example a User could have some posts. Normally you create the user and then you create the posts and in the end you make the connection. But what if you could create the user and the posts at the same time? In my perspective this is not really common use case and I needed this feature at the time of implementing some database migrations and structural changes in the database, but looks like so many people supported the request.

Mongoose Open Issues

Mongoose execution time increased 2 times after upgrading 4 to 6 #11380 Mongoose had a performance issue after Mongoose 6 release and this was not due to mongo-native-driver upgrade. They are still investigating this issue.

new feature: virtual async #5762 Virtuals are simply virtual fields that you can set and they exist in your object but sometimes not in your database. This feature does not support async functions and as a result people have to use some other workarounds. In my opinion it is not a major issue and you can find some good workarounds in the issue discussions.

Multi-level discriminators #2851 Discriminators help implement inheritance in Mongoose model. This feature asks for the feature of multiple discriminators.

[Feature] option to disable running pre/post hooks #8768 This requests simply disable hooks.

Optimize nested populate #3812 In a case it generates two separate queries which could be improved but it is still not implemented.

Conclusion

Both libraries are really good and well-documented and well-supported and you can easily do the job with both of them. The choice really depends on your values and the context. If you value stability more than other things I think Mongoose could give you that and it will never surprise you(negative or positive). If can tolerate some minor bugs and you are curious, maybe Prisma is a better choice for you and it will definitely suprise you with new features with each release and after a while it could be your choice no matter which database you are using.

Comparing Prisma and Mongoose

Exploring Parallelism and Concurrency in Node.js:

Poorshad Shaddel — Tue, 22 Aug 2023 18:20:05 +0000

Exploring Parallelism and Concurrency in Node.js: Child Processes, Multithreading, Cluster, and More

In this article, we delve into the world of parallelism and concurrency in Node.js, exploring the concepts of child processes, multithreading, and their practical implementations.

What you will read in this article

You are going to read about the Node.js Child process, and how we can spawn a child process. Then we will look into Fork and the differences between Fork() and Unix Fork. Then we look into worker threads and clusters and in the end we shortly discuss concepts of Parallelism vs Concurrency on Nodejs.

What is a Child Process?

A child process in computing is a process created by another process (the parent process). This technique pertains to multitasking operating systems and is sometimes called a subprocess or traditionally a subtask.

Each Process could have multiple child processes and one parent. If it does not have a parent process it is probably created directly by Kernel.

Specifically in Nodejs: Each process(Spwan or Fork) has its own memory, with its own V8 instances.

You can see the process tree using this command on Mac or Linux:

ps -axjf

Example of process tree

Here PID is the process ID and PPID is the parent process id. If you want to see it visualized as a tree you can use This VSCode Extension or install pstree with brew install pstree .

For example by using this vscode extension you can see the current process tree of your machine:

Visualized Process Tree

Spawn

Spawn is one of the ways we can create a child process and The child_process.spawn() method spawns a new process using the given command .

Let’s create a subprocess in our Nodejs app and check it out. In order to prevent processes from termination I am going to use Unix sleep command so we can see the parent-child process in the console.

This is the code, we are going to use Spawn :

const { spawn } = require('node:child_process');

const sleep = spawn('sleep', ['10']);

console.log(`Current process PID: ${process.pid}`);

console.log(`Child process PID: ${sleep.pid}`);

sleep.on('close', (code) => {
    console.log(`child process exited with code ${code}`);
});

sleep.stdout.on('data', (data) => {
    console.log(`stdout: ${data}`);
});

sleep.stderr.on('data', (data) => {
    console.error(`stderr: ${data}`);
});

sleep.on('close', (code) => {
    console.log(`child process exited with code ${code}`);
});

By using spawn we have created a new process. Before the sleep command is over I am going to log the pid and ppid in the console:

It logs the current process and the child process

Child Parent Process

As we can see sleep 10 command has a parent which is the process node ./spawn.js .

Fork

Based on the official docs Fork() method is a special case of

Spawn() and it is used specifically to spawn new Node.js processes. But the child process that comes from Fork() will have an additional communication channel built-in that allows messages to be passed back and forth between the parent and child. This forked Nodejs process is independent of the parent and has its own V8 instance.

Let’s see a simple example of a parent and forked child:

// Parent.js
const { fork } = require('node:child_process');

const forkSort = fork('./fork_sort');

const numbers = [1, 5, 2, 3, 4];

forkSort.send(numbers);

forkSort.on('message', (sortedNumbers) => {
    console.log(`Sorted numbers: ${sortedNumbers}`);
    process.exit(0);
});

forkSort.on('close', (code) => {
    console.log(`Child process exited with code ${code}`);
}
);

// Child.js
process.on('message', (msg) => {

    console.log("CHILD: message received from parent process", msg);

    const sortedNumbers = msg.sort((a, b) => Number(a) - Number(b));
    process.send(sortedNumbers);

});

By running parent.js we can create a forked process and do the sorting in the child.js and get the result back.

Difference between UNIX Fork and Nodejs Child Process Fork

In UNIX when we are forking a process, that makes a shadow of the parent's memory (they share the memory as long as they are only reading from it). this kind of memory is called Copy on Write.

Here comes the big difference: Nodejs fork is not a clone of the parent process. It is like Spawn(fresh memory) with a built-in communication channel!

Cluster

Using Cluster, we can run multiple instances of Node.js that can distribute workloads among their application threads.

The important thing about cluster is by using it we can easily create child processes that share the same port.

Here is an example(from official docs) of using Cluster for creating multiple child processes that all listen to the same port.

import cluster from 'node:cluster';
import http from 'node:http';
import { availableParallelism } from 'node:os';
import process from 'node:process';

const numCPUs = availableParallelism();

if (cluster.isPrimary) {
  console.log(`Primary ${process.pid} is running`);

  // Fork workers.
  for (let i = 0; i < numCPUs; i++) {
    cluster.fork();
  }

  cluster.on('exit', (worker, code, signal) => {
    console.log(`worker ${worker.process.pid} died`);
  });
} else {
  // Workers can share any TCP connection
  // In this case it is an HTTP server
  http.createServer((req, res) => {
    res.writeHead(200);
    res.end('hello world\n');
  }).listen(8000);

  console.log(`Worker ${process.pid} started`);
}

As you can see Cluster module has spawned some child processes using cluster.fork() .

The default way that cluster module load balances the requests is that the primary process listens on port and accepts connections and uses round-robin for distributing them to other processes.

The second approach is where the primary process creates the listen socket and sends it to interested workers. The workers are able to accept incoming connections directly. ( In practice, distribution with this method tends to be very unbalanced due to operating system scheduler vagaries )

Routing using Cluster Module : Node.js does not provide routing logic. It is therefore important to design an application such that it does not rely too heavily on in-memory data objects for things like sessions and login. In other words, you cannot assign specific incoming requests to specific workers based on custom logic.

Worker Threads

Threads are smaller units of execution within a process. Multithreading is a technique used to achieve parallelism by splitting a process into multiple threads that can execute concurrently.

Threads vs Processes

The worker_threads module in Node.js does not create new processes; instead, it provides a way to create and manage multiple threads within the same Node.js process. This allows you to perform parallel computation without spawning separate processes.

Three important things about worker threads:

Workers (threads) are useful for performing CPU-intensive JavaScript operations.
They do not help much with I/O-intensive work.
The Node.js built-in asynchronous I/O operations are more efficient than Workers can be.

Let’s see a simple example before getting into the details.

We want to pass our sorting operation to a worker thread. In order to do that we can have two files:

1- Function that creates the worker and gets the computation result back

2- Worker thread that calls sort method on the array.

Here is the first file: It creates a new worker thread and resolves the promise when it gets the result back from the worker thread.

const {
    Worker, isMainThread, parentPort, workerData,
} = require('node:worker_threads');

function sortAsyncUsingWorkerThreads(array) {
    return new Promise((resolve, reject) => {
        const worker = new Worker('./sort_work_thread', {
            workerData: array,
        });
        worker.on('message', resolve);
        worker.on('error', reject);
        worker.on('exit', (code) => {
            if (code !== 0)
                reject(new Error(`Worker stopped with exit code ${code}`));
        });
    });
};

const numbers = [1, 5, 2, 3, 4];

const main = async () => {
    const sortedNumbers = await sortAsyncUsingWorkerThreads(numbers);
    console.log(`Sorted numbers: ${sortedNumbers}`);
}

main().catch(console.error);

Here is the implementation of this worker file:

const { parentPort, workerData } = require('worker_threads');
const array = workerData;
parentPort.postMessage(array.sort());

By just calling the postMessage method we can pass the data to the parent.

In the example, each time we call this function it creates a new thread. Which is not a good idea(because we do not have unlimited worker threads available). Generally using worker threads could be tricky. Take a look at this resource from Standford University:

https://web.stanford.edu/~ouster/cgi-bin/papers/threads.pdf

Parallelism vs Concurrency in Node.js

Concurrency in Node.js

Concurrency

Concurrency in Node.js refers to the ability of the system to handle multiple tasks or operations at the same time. Node.js does that by using Event Loop, it can handle a lot of requests while it is waiting for non-blocking I/O. The async-await pattern that we use daily as Node.js Developer is in order to achieve concurrency.

Parallelism in Node.js

Parallelism

Parallelism refers to the simultaneous execution of multiple tasks or processes to achieve better performance by utilizing multiple CPU cores. Node.js is inherently single-threaded, meaning it runs on a single thread of execution. However, Node.js can still achieve parallelism through various mechanisms, such as:

Child Processes : Node.js can spawn or fork multiple child processes, to take advantage of multiple CPU cores. These processes can communicate through inter-process communication mechanisms.
Worker Threads : The worker_threads module in Node.js allows developers to create and manage multiple threads within a single Node.js process. Each thread operates independently, and they can share memory, making it useful for CPU-bound tasks.

Conclusion

This was a quick review of some related topics. Spawn gives a new fresh isolated process. The Fork method gives you a new Nodejs process with a built-in communication channel. Cluster is using the fork method to distribute workloads. Worker threads are good for doing CPU-Intensive tasks inside the process with shared memory.

References

Child process - Wikipedia

Difference between processes and threads — YouTube

Building a non-blocking multi-processes Web Server (Node JS fork example) — YouTube

https://web.stanford.edu/~ouster/cgi-bin/papers/threads.pdf

MongoDB Index Challenge: Learn by Doing!

Poorshad Shaddel — Sun, 11 Jun 2023 23:47:15 +0000

In this article, you will learn how indexing in MongoDB works!

What you will see in this article?

Introduction to Indexes on MongoDB
A list of challenges that you should choose the right index

Introduction to Indexes on MongoDB

If you feel totally comfortable with indexes you can skip this part and get to the challenges.

Why do we need indexes?

The answer is quite simple and indexes on the database exist for the reason that you have indexes on your phone notebook(I know that probably you do not have it now! but it was a useful thing in the past). Take a look at this notebook and say if it makes sense to start checking the notebook from the beginning to the end to find last_name of someone like “John Smith”? The answer is NO and you are directly going to the “S” section and searching there and it saves a lot of time. This is the same thing on the database.

Old Phone Number Notebooks

Generally, if we do not have any index on any field, for each query we have to fetch all the documents from the Disk(which is a generally expensive operation, especially when there are a lot of documents) and then filter the desired results and return them. In MongoDB, it is called a COLLSCAN which means a collection scan.

For example, if this is our documents in a collection called users:

const users = [
  {
  "_id": "1",
  "name": "John",
  "last_name": "Smith"
  },
  {
  "_id": "2",
  "name": "Jason",
  "last_name": "Sterling"
  }
];

and this is our query:

db.users.find({ last_name: "Smith" });

This query has to perform a Collection Scan and fetch all the documents to check if they have this last_name or not.

On the other hand, if we create an index

db.users.createIndex({ last_name: 1 });

Now we have an index exactly like our phone book. Now instead of fetching all the documents from the disk we are just fetching 1 document. In a MongoDB index, the Record_ID is stored which is the address of the document on the Disk.

If you do not know a lot about indexes focus on the answers and useful links which are provided!

Index Challenges

In these challenges try to avoid Collscan and also in-memory sort!

In each challenge, you get the :

document structure
query(queries) that we want to perform on this collection
extra information about the data if needed

You can provide one or more indexes on the collection

Challenge Number 1

// Documents
const users = [
  {
  "_id": "1",
  "name": "John",
  "last_name": "Smith"
  },
  {
  "_id": "2",
  "name": "Jason",
  "last_name": "Sterling"
  }
];

// Queries
db.users.find({ last_name: "Smith" });

db.users.find({ }).sort({ last_name: 1 });

This is for warming you up!

Challenge 2 Answer :

As you probably know we can use the same index for searching and sorting at the same time! so as easy as that:

db.users.createIndex({ last_name: 1 });

Challenge Number 2

// Documents
const users = [
  {
  "_id": "1",
  "name": "John",
  "last_name": "Smith"
  },
  {
  "_id": "2",
  "name": "Jason",
  "last_name": "Sterling"
  }
];

// Queries
db.users.find({ last_name: "Smith", name: "John" }); // 1
db.users.find({ last_name: "Sterling" }); // 2
db.users.find({ name: "John" }); // 3

Challenge 2 Answer :

db.users.createIndex({ last_name: 1, name: 1 });
db.users.createIndex({ name: 1});

When we are matching two different fields that we are searching for we can use compound indexes:

db.users.createIndex({ last_name: 1, name: 1 });

This is like a phone book in that we have the indexes of family names but on each page, we are adding each person also sorted by their name.

Last_Name - First_Name
Anderson - Amy
Anderson - Jim
Davis - Amber
Smith - Adam 
Smith - Bruce
Smith - Jane

Pay attention that the order in the index is important!

But if you answered with this index: db.users.createIndex({ last_name: 1 }) this is also correct but it is not the best option. With this index, we will get all the documents and then Filter the desired name.

But do we need another index for the Second Query? no, we do not need it! If we set this index it will be a redundant index.

Let’s take a look at one index and supported queries by that index to understand prefix:

db.users.createIndex({ last_name: 1, name: 1, age: 1 });

db.findOne({ last_name: "Smith" });//Not needed index: ({ last_name: 1 })
db.findOne({ last_name: "Smith", name: "John" }); // not needed ({ last_name: 1, name: 1 })
db.findOne({ last_name: "Smith", name: "John", age: 21 });

All of the above queries are supported by the compound index.

Let’s get back to our challenge. Query 3: db.users.find({ name: "John" }) . If you think this could be supported by the existing index we had( ({ last_name: 1, name: 1})) You need to take a look at the Phone Book:

Last_Name - First_Name
Anderson - Amy
Anderson - Jim
Davis - Amber
Smith - Adam 
Smith - Bruce
Smith - Jane

As you can see there is no order for finding first names. So we have to put a single field index: db.users.createIndex({ name: 1});

Challenge Number 3

// Documents
const products = [
  {
  "_id": "1",
  "rating": 3,
  "createdAt": ISO_DATE("2023-05-11")
  },
  {
  "_id": "2",
  "rating": 3,
  "createdAt": ISO_DATE("2023-05-11")
  }
];

// Queries
db.products.find({ }).sort({ rating: -1, createdAt: -1 }); // Best Newest
db.products.find({ }).sort({ rating: -1 }); // Best
db.products.find({ }).sort({ rating: 1, createdAt: 1 }) // Worst Oldest

Challenge 3 Answer :

db.users.createIndex({ rating: -1, createdAt: -1 });

This compound index will be enough since the second one is a prefix of this index. about the third query, we must know that when we have a sorted list we can traverse it “forward” or “backward” and still use the index!

Challenge Number 4

// LeaderBoards
const leaderBoards = [
  {
  "_id": "1",
  "userId": "0",
  "competition": "Python Hackaton",
  "score": 100,
  "number_of_problems_solved": 8
  },
  {
  "_id": "2",
  "userId": "1",
  "competition": "Python Hackaton",
  "score": 98,
  "number_of_problems_solved": 6
  },
  ...
];

Here we have a leaderboard. Each document has a “competition field” which is the name of the competition and each person has a “score”. We also keep a number of problems that are solved and resulted in this score.

This is the query we want to perform.

Challenge 4 Answer :

// Queries
db.leaderBoards.find({ 
   competition: "Python Hackaton", 
   number_of_problems_solved: { $lte: 10 }  
}).sort({ score: -1 });

In this query, we want to get userId of the people who have the best score, but we expect them to have this score for solving less than 10 problems.

Our suggestion for the index is this one:

db.leaderBoards.createIndex({ 
  competition: 1, 
  score: -1, 
  number_of_problems_solved: 1 
});

But Why? There is a simple rule called: ESR (Equality, Sort, Range)

The first field we are indexing is competition. Since equality comparison usually can filter lots of documents it makes sense to put this as the first field in our compound index. (if for example, 95% of the documents have this field competetion with value of Python Hackaton then it might not make sense to use it as the first field — in other words, our equality filter is not Selective enough).

The second field in the compound is score. But why are we using the sort field as the second? The intention is to prevent In Memory Sort. If we use our range in the second field of the compound index then we cannot use the sort index anymore.

This is the result of the current index:

With the current index — no in-memory search

If we use this index:

{ competition: 1, number_of_problems_solved: 1, score: -1 }

This will be the result:

without ESR Rule we face in-memory sort

As you can see we had to do a in-memory sort.

Do we have to always follow ESR Rule to prevent in-memory sorting?

The answer is NO! If the situation is somehow that the range field is really selective and eliminates most of the documents and you do not have a lot of documents left to sort, then it makes more sense to put this field as the second field of the index. In this case, we are sorting in memory but we might have better results. If you are still not sure, you can have both indexes for a period of time and see which one is chosen by the query optimizer(the query optimizer runs queries with these indexes at the same time to check which one is better — By default, it avoids the in-memory sort).

Challenge Number 5

// Documents
const products = [
  {
  "_id": "1",
  "title": "Desk",
  "weight": 1,
  "createdAt": ISO_DATE("2023-05-11")
  },
  {
  "_id": "2",
  "title": "IPhone 14",
  "createdAt": ISO_DATE("2023-05-11")
  },
  {
  "_id": "3",
  "title": "IPhone 14",
  "createdAt": ISO_DATE("2023-05-11")
  },
  {
  "_id": "4",
  "title": "IPhone 14",
  "createdAt": ISO_DATE("2023-05-11")
  },
];

// Queries
db.products.find({ weight: { $lte: 2 } }); // Filter products with less than 2 kg weight

Extra Information : Field weight does not exist on more than half of the products(but we do not want to have these documents in the result)

Challenge 5 Answer :

// Partial Index
db.products.createIndex(
  { weight: 1 }, 
  { partialFilterExpression: { weight: { $exists: true } } }
);
// Alternative Solution
db.products.createIndex(
  { wight: 1 }, 
  { sparse: true }
);

By using a partial index and adding a partial filter expression we can ask MongoDB to not index documents that we do not want. If you have a collection with a few million documents and you are querying on a specific field that only a minority of the documents have this field, Partial Index can be helpful with two benefits: 1- Smaller Index 2- Faster Look Up

Another alternative is using the sparse option. Only indexes if the field exists.

Another usage of Partial Indexes is in a situation where you are running a range query. For example, you never filter products that have a rating of less than 2(worst products). So why index them? we can use this index on rating:

db.products.createIndex(
  { ...ourIndex }, 
  { partialFilterExpression: { rating: { $gte: 2 } } }
);

But keep in mind that if now you run a query like this one:

db.products.find({ rating: 1 });

This results in a collection index!

Challenge Number 6

// Documents
const products = [
  {
  "_id": "1",
  "title": "Desk",
  "tags": [
    { name: "office" }, 
    { name: "wood" }, 
    { name: "laptop_desk"}
   ],
  "createdAt": ISO_DATE("2023-05-11")
  },
  {
  "_id": "2",
  "title": "IPhone 14",
  "tags": [
    { name: "phone" }, 
    { name: "14" }, 
    { name: "apple"}
   ],
  "createdAt": ISO_DATE("2023-05-11")
  },
];

// Queries
db.products.find({ "tags.name": "apple" }); // Filter apple products

Challenge 6 Answer :

// Multikey Index
db.products.createIndex({ tags.name: 1 });

This index will index each element of the array! this works on fields that are arrays like: [1, 2, 3] and also fields that are documents or nested documents .

Limitations

You cannot have 2 multikey indexes in a compound index(the number of indexes will be n*m)
Cannot use covered queries. (Covered queries are the queries that all needed fields that exist in the index you do not need to fetch documents from the disk)

There are some other limitations too that you can check here.

Challenge Number 7

// Documents
const products = [
  {
  "_id": "1",
  "title": "Desk",
  "description": "a brown desk for usage in office or home"
  "createdAt": ISO_DATE("2023-05-11")
  },
  {
  "_id": "2",
  "title": "IPhone 14",
  "description": "Hight quality product",
  "createdAt": ISO_DATE("2023-05-11")
  },
];

// Queries
db.products.find( { $text: { $search: "brown desk" } });

Challenge 7 Answer :

// Text Index
db.products.createIndex({ description: "text" })
// Text Index with another collation to support lowercase uppercase
db.products.createIndex({ description: "text" }, { collation: {
  local: "en",
  strength: 2,
});

The text index is also supported by MongoDB and we can use it. But keep in mind that it works similarly to the multikey index! It means that it is kind of similar to having an array of strings and putting a multikey index on this array.

And the limitation is that you can use it on only a field in your collection. But if you are using Atlas Search is a much much better option to use.

Challenge Number 8

// Documents
const users = [
  {
  "_id": "1",
  "first_name": "Poorshad",
  "last_name": "Shaddel"
  "userMetadata" : { "likes" : ["dogs", "cats"] }
  },
  {
  "_id": "2",
  "first_name": "John",
  "last_name": "Smith",
  "userMetadata" : { "dislikes" : "pickles", "inactive": false }
  },
  ...
];

// Queries
db.users.find( { userMetadata.likes: 'cats'});
db.users.find( { userMetadata.dislikes: 'pickles'});
db.users.find( { userMetadata.age: 45});
db.users.find( { userMetadata.inactive: false});

Extra Information : userMetadata has an arbitrary structure but we want to have an index on it.

Challenge 8 Answer :

// Wild Card Index
db.users.createIndex( { "userMetadata.$**" : 1 } )

For this case, we have to use a Wild Card Index. We can index all the fields that fit in this pattern. If you are putting a lot of stuff into this object and you are not querying most of them, it might not be the best idea to use userMetadata.

Another pattern we could use to store this arbitrary data in the index was to Attribute Pattern.

userMetadata: [
  { "attribute": "likes", value: ["cats"] },
  { "attribute": "age", value: 45 }  
]

And we can use an index like this to support the queries on this:

db.users.createIndex({ userMetadata.attribute: 1, userMetadata.value: 1 })

Summary Of Challenges

Challenge Number 1(Single Index)

Challenge Number 2(Compound Index)

Challenge Number 3(Sort Direction)

Challenge Number 4(ESR)

Challenge Number 5(Partial Index)

Challenge Number 6(Multikey Index)

Challenge Number 7(Text Index)

Challenge Number 8(Wildcard index)

Next Steps to Explore More

Hashed Indexes

2D Indexes

How MongoDB Clustered Collection Can Boost Query Performance

Poorshad Shaddel — Sun, 28 May 2023 17:08:33 +0000

Mongodb Clustered Collection is a relatively new feature released in MongoDB 5.3 that could improve query performance!

What you will read in this article

How does MongoDB fetch documents in a collection
How does Mongodb fetch documents in a clustered collection
Limitation
Playing with Clustered and Normal Collection
When to use Clustered Collection

How does MongoDB fetch documents in a normal collection?

Here we want to discuss how MongoDB fetches documents using indexes. If there is no index MongoDB has to fetch all documents in a process called COLLSCAN if you look at the query explainer. COLLSCAN fetches all documents of a collection from the disk and then starts doing your desired operation. Here we focus on what happens when you are using an index.

Imagine you are using _id which is indexed(by default) and you want to fetch one document. Here is what happens:

Find the _id in the MongoDB BPlus Tree

Indexes in MongoDB are stored in a BPlus Tree. If you do not know what is a BPlus Tree it is similar to a BTree with two major differences:

1- Values are at leaves.

2- Leaves have pointer to the next one(which makes it fast for range queries)

This is a simple BTree consisting of these numbers: (1,2,3,4,5,6,9,10)

BTree consisting 1,2,3,4,5,6,9,10

This is on the other hand a B+Tree consisting of the same numbers

B+Tree consisting 1,2,3,4,5,6,9,10

MongoDB stores indexes in B+ Tree so when we are running this query:

db.users.findOne({ _id: ObjectID("...") });

The first thing MongoDB does is it goes to this B+ Tree and finds the _id.

Now it got the Leaf. But the important thing is that your document is not the value of this Leaf.

the value is another thing called RecordID. MongoDB has a hidden clustered index which is another B+ Tree that you can find the document by using RecordID. You have to traverse the tree once again and you have your document.

So the summary of what happens with this simple search query:

1- Traverse the B+ Tree and find the Leaf regarding _id or any indexed field you have

2- Get RecordID from the Leaf.

3- Traverse the hidden clustered index for RecordID

4- Get the document from Value of RecordID

How does Mongodb fetch documents in a clustered collection?

In a clustered collection you do not have a RecordID anymore!

You need to traverse the B+ Tree only once and you have the document!

As simple as that. We should be careful with this since we know everything comes at a cost.

Cost of using Clustered Documents

In a normal collection you only had RecordID, as a result when we added more indexes we do not see that many changes in the size of the collection. But now if we add more indexes we see big changes. If you do not add more indexes the size of the clustered collection is smaller than the normal collection because we do not have this RecordID.

Limitations

By default, if a secondary index exists on

a clustered collection and the secondary index is usable by your

query, the secondary index is selected instead of the clustered

index.

You can use clustered index only on _id field!

You cannot convert a normal collection to a clustered collection or vice versa.

Playing with Clustered and Normal Collection

I want to create two collections on MongoDB Atlas and run some queries on them:

For creating a clustered collection we can use the UI and easily choose the clustered collection:

Create a Clustered Collection

Also, let’s create a normal collection:

Normal Collection

Now if we run a findOne({ _id: “”}) we can see the difference in explain query.

The first one is for transactions_original which uses the _id index and chooses IDHACK(the name of the plan when we use _id to find a document)

IDHACK

For the clustered collection we can see that the query plan selected was CLUSTERED_IXSCAN

CLUSTERED_IXSCAN

Let’s also see the difference in the size of the index for 100k documents:

Clustered Index Size

Normal Index Size

Let’s perform some queries on a new collection called products. Our usage is that we fetch products that have better ratings(mostly range queries).

Then we put 1 million products. For comparison, we create also products_original . In order to use clustered index we put rating of product into _id of products .

For products_original this is the schema and the query we want to perform:

// products_original 
{
  _id: ObjectID("..."),
  rating: number => index(-1),
  name: string
}
Query = db.collection.find({ rating: { $gte: 0.9 } })

// products_clustered
{
  _id: clusteredIndex(we put rating into this field),
  productId: uuidv4(),
  name: string,
}
Query = db.collection.find({ _id: { $gte: 0.9 } })

Then I created 1M products and for making the comparison fair I used the same ratings for these 1M products. The data in both collections is the same. This is not a benchmark that you can rely on but the result and query explain data might give you some idea that when it is worth using.

Result of the range query in the Clustered Collection

Range Query Using Clustered Index

Result of the range query in the Normal Collection

Range Query Result in Normal Collection

In normal collection one apparently expensive step was Fetch! Because for each _id we have to look up the RecordID and then get the documents.

When to use Clustered Collection

There is not a special description for this in the documentation but based on the list of benefits we can summarize that like this:

Queries with range scans and equality comparisons on the clustered index key
Faster bulk insert

https://www.youtube.com/watch?v=OhJ3xcjtpis&ab_channel=HusseinNasser

Clustered Collections

Level Up Coding

Thanks for being a part of our community! Before you go:

👏 Clap for the story and follow the author 👉
📰 View more content in the Level Up Coding publication
💰 Free coding interview course ⇒ View Course
🔔 Follow us: Twitter | LinkedIn | Newsletter

🚀👉 Join the Level Up talent collective and find an amazing job

From Java and Microservices to Dgraph: A Big Bang Migration Journey of a 14M Users App

Poorshad Shaddel — Thu, 25 May 2023 02:46:47 +0000

The story of a big bang migration that we had in one of the companies where I worked!

Dgraph

What you will read in this article?

About the application and the team
Why we had to migrate the technology?
What is Dgraph? Could a Database replace the Backend?
How We Have Done the Migration with New Data Structure
The problem of Caching in GraphQL
Security Issues in GraphQL
How we Built a Custom UI using GraphQL
How We Have Used Elastic Search Directly on Dgraph
Regrets and Expriences

About the application

Before getting to the details I want to clarify what was the application. It was a video streaming application and also users had access to a selected archive of the content(each content stored in the platform based on its program’s priority). Most people have used this application for watching important football games or famous local TV series. This application had about ~14M users at that time and in the pick time, we were experiencing around 800K-900k concurrent users.

Our Backend Before Migration: Java/MySQL Microservice Architecture

After Migration: Dgraph/ Elixir Phoenix for Authentication

Our Frontend was Angular and remained the same.

Size of the Team

1 Backend Developer(I) — 2 Frontend Developers — 3 Site Reliability Engineers — Engineering Manager and CTO. Overall 8 persons.

Why We Had to Migrate the Technology?

If you are reading here to see that the performance issue was related to the programming language, I have to say you are in the wrong place, and without a doubt Java was not the problem.

These were the problems in my personal opinion:

1- Lack of coherence in the Backend Repository

This was the result of having a small team and most of the time it was one person or in some cases, two persons working on the backend side of this project and it resulted in a code base with different strategies. Probably because the developer was the only one working on the code base it was hard to standardize the code. Also in teams without enough resources for growth, there are lots of tasks and bugs that are expected to be tackled before other important things like documentation, tests or standardization.

2- Lack of Documentation

There was no documentation about the code and how it works and even why some features exist. We had to communicate with different people to find out something is needed or not. The management has never asked for something like this in the first place so no one created any documentation about the code.

3- Inflexibility

Based on the estimation that the backend team(with different seniority levels and range of skills) was giving for small features or bug fixings it was obvious that the code has lost its flexibility and they are not able to ship fast and change the code.

4- No Knowledge Transfer

After the last Backend Developer left the company we had to deal with a Backend Repository without documentation or any clues. unfortunately there was no process of knowledge transfer so we had no choice except to go to into maintenance mode for a while.

What is Dgraph? Could a Database Could Replace the Backend?

We wanted to ship a lot of features fast and the number of users was increasing so we needed also a scalable solution. We finally decided to rewrite the whole backend but there were some problems:

It was huge and had a lot of logic
We did have only a few months at most for rewriting the whole thing and the Team was super small
At the same time, there were expected new features

A year before this migration our CTO told me about an interesting Database called Dgraph and I started following this database newsletter and joined the forums of the community.

At that time it just introduced the GraphQL APIs which was amazing. If you’ve never heard about Dgraph it is a Graph Database written with Go Lang and besides being a database it offers API’s based on your Schema. This schema is a simple GraphQL Schema with some extra directives specific to Dgraph. When you define your schema you already have the API’s.

For example, if you define this schema in your Dgraph GraphQL Schema(The environment is Dgraph Cloud):

Dgraph GraphQL Schema Deployment and Definition

ID is a unique type and it is created by Dgraph DB itself.

Now we have a GraphQL Schema that we can use:

GraphQL API’s Based on GraphQL Schema

For each type by default you have these API’s generated:

1- aggregate[TypeName]

2-get[TypeName]

3-query[TypeName]

4-add[TypeName]

5-update[TypeName]

6-delete[TypeName]

At the time I started following Dgraph it did not support authorization and customized access to these functionalities but after six months they introduced Authentication Directive on GraphQL Schema which made it production ready in my perspective and we could handle a lot of cases.

Why did Our Team think Dgraph was a good solution?

1-Time Restriction : First of all we did not have that much time and we had to do this migration in a few months and in between a big Cyber Attack happened and moved the deadline to a sooner date. So it was not possible to implement more than hundred different routes with high test coverage. On the other hand, using GraphQL API of Dgraph could save us a lot of time.

2-Our Application Behavior : Our application had more reads than writes and most of these reads were the result of complicated queries from joining more than 3 tables. A lot of joins and heavy queries send us the signal that we might have some benefits if we use a Graph Database .

Another important thing that motivated us was that Dgraph had introduced something called Custom DQL . DQL is the language of Dgraph Database. This feature lets us implement custom complicated queries(Something like MongoDB aggregation pipeline) and put these queries behind GraphQL API’s. Pagination was also included without any problem and in our case, a lot of already existing Backend API’s were just returning a complicated query from MySQL. So this feature was also saving us a lot of time.

3- Lambda Resolvers For Our Custom Logic: We knew that we need some to implement some custom logic. For this need, Dgraph offered Lambda Resolvers which are equivalent to Lambda Functions but with the difference that Lambda Resolvers are web workers.

Could Dgraph Database Replace the Backend?

The answer is Yes and No!. There are more options these days that are working in the same way as Dgraph. Neo4J got the good idea(in my opinion maybe from Dgraph or Postgraphile) and now offers a super nice GraphQL API for the database with a lot of features. It seems that it is easier for Graph Databases to offer a GraphQL API but Hasura offers it with Relational Databases.

The point here is that people are using API’s from these databases(or platforms) instead of implementing every single route in a backend. We can have our own custom logic in each one of these platforms that I mentioned. Even if we are not satisfied with the database custom logic handling we can have a small backend on the side that works with the database and does some of the complicated things for us.

How We Have Done the Migration with new Data Structure

We started with a proof of concept to see if we are able to do this or not and let’s see how is the performance with some of our complicated queries.

For migrating SQL data to Dgraph there is a tool that Dgraph implemented but we did not want to use it because we had to also change the data structure.

We were providing Video content and everything was implemented in a hierarchy. For example:

Hierarchy of Categories

Each of these categories had some subcategories. Each content had one of the leaves in this tree. We decided to go with the concept of Tags instead of using Tree(or both at the same time). We had a lot of tables and structures to assign different attributes, categories and subcategories to each content, but from a Tag perspective, all of these attributes could be considered as some Tags that we could assign to the Content and this drastically reduced the complexity .

In the older system content had a leaf called: PrimaryLeague but in the new system it had a tags fields that should contain all parents of content:


tags: [
  {uid: 0x1, name: 'primaryLeague'},
  {uid: 0x2, name: 'football'},
  {uid: 0x3, name: 'international'},
]

Database Converter and Challenges

For migrating Data, We have used Node.js and Prisma. The reason we used Prisma was the Database Pull Feature. We had the production database structure and with this command, I was able to convert all the tables to a Prisma Schema: npx prisma db pull .

After that running npx prisma generate updated the Prisma Client and we had access to all tables and their relations in Typescript so I was able to fetch data and convert them to the new structure and insert them into the new database. Dgraph had a official Nodejs Client library for using DQL queries and mutations that we have used for this matter.

One of the biggest challenges was connecting Dgraph Nodes and creating the reverse edges. For connecting the Graph Nodes you need UID of the Nodes and the problem was that you cannot set this UID, dgraph generates this UID. So I had to create both Nodes at one step, then fetch the nodes and finally run an update query to connect these nodes. This was mostly the reason why this Database Converter was not so fast.

The Problem of Caching in GraphQL

Caching is usually not easy for GraphQL API since most caching processes are working with URL rather than details of the request. Parsing the request content was expensive for us since the number of concurrent requests were always high(most of the time more 10K). So we came up with the idea of hashing the queries and sending them from the client side. For each common query we were hashing the query content and we added this as queryId to the endpoint. Now we were able to do the caching. We could also set the caching time of each endpoint by using @cache directive on Dgraph Schema.

Security Issues in GraphQL

Failure to Appropriately Rate-limit

This is one the queries that could cause problem:

query Recurse {
  allUsers {
    posts {
      author {
        posts {
          author {
            posts {
              author {
                posts {
                  id
                }
              }
            }
          }
        }
      }
    }
  }
}

The attacker can nest this query a lot and as a result it puts a lot of pressure on the database.

For solving this issue we had to put a minimal Shield for protection against this kind of queries. We limited the size of query and also response to a small number and as a result this query which is passed could not be so big. We set the query timeout number in Dgraph to 1 Second(it means it drop the queries that take more than this number).

Also Dgraph have a feature for rate limiting that we have used along side these options I explained and you can check the details here: https://dgraph.io/docs/deploy/cli-command-reference/#limit-superflag

User Could Get All Documents From Dgraph GraphQL API

This was also another issue that the client could set page size. We limited the response size and as a result no one can easily get this data but they could harm the database performance. So we had to handle this also in our Go Lang Shield. We set the page size in there and as a result you cannot pass your desired page size to the system.

How We Built a Custom UI

One of the pains we had in there was too many changes in the UI of the application both in the Website and Mobile Apps. Not major changes but moving components around. We already had some components which were well implemented. We came up with a page, component system to handle these changes and at the same time introduced the feature of custom pages for different channels and events. These custom pages usually took a lot of time from our Fronted Developers so it having this feature could save them a lot of time.

This is a simplified version to show you the idea of how this UI was implemented:

Simple UI Component Structure

Our Frontend Developers and Mobile Developer used the same component names and the design system was somehow that they could both read our pages and generate the UI. Support users were able to do all the customization with different kinds of components.

How We Have Used Elastic Search Directly on Dgraph

First we started the new version by using Dgraph Fuzzy Matching and later with some complicated queries to get related contents. It was not efficient enough and we were not able to get related content when the search phrase was not so accurate or it was more than a word. As a result we decided to use Elastic Search for our search functionality. One of the teamates has implemented a small service with Elixir for syncing our Dgraph Database with Elastic Search(Not the whole database just 2 types).

Now we had to find a way to perform our search queries. The first way that came to mind was to implement a minimal backend to access these search queries, but we had to think about scaling this small service and again we were facing all the problems. In the end we came up with a solution to expose one GraphQL API that requested data from our Elastic Search.

By using Dgraph Custom Queries you can make HTTP Requests to other endpoints and parse the result and return it in a GraphQL Resolver.

Check out this article on Dgraph for the details of how we connected Dgraph to Elastic Search. Below you can see a schematic of how this is handled.

How Dgraph is Connected to Elastic

Regrets and Experiences

Not enough Documentation

Dgraph did not have a perfect documentation at the time and I had to learn all the tricks from Dgraph Forum.

Investors Fired CEO(Founder) of Dgraph

After we almost finished our migration this happend and the whole Dgraph Team resigned and It was super scary for me. Now it has another CEO and they are going in a good path and their steadily improving features and fixing bugs.

Read more about this:

https://www.reddit.com/r/golang/comments/sfl7kj/quitting_dgraph_labs/

https://discuss.dgraph.io/t/going-out-of-business/16631

A Lot of Try and Error to Optimize Queries Based on Data

After a while some our queries changed and the reason behind it was the shape of our data. We had some Types with more than a few Million records and in queries we had to avoid running queries like Regex or Full-Text Search as much as possible.

An example of design change because of optimizing query:

First we had Content and Tag but querying contents with specific tags was really expensive and we had to touch all contents. We made the relation two way and then we were able to get the contents from tag(Indexes are removed for simplicity).

Two Different Approach for getting contents that have specific tag

With the first design for getting contents that have specific tag we must touch all contents(which was over a few million nodes). With second approach we directly get the tag and the contents are attached to this node so it is super fast compared to previous approach.

Two different queries for getting contents that have specific tag

Some Dgraph Issues that made us Slow

https://discuss.dgraph.io/t/dgraph-manual-reverse-index-is-gone-after-each-graphql-schema-upload/14829

https://discuss.dgraph.io/t/cannot-query-after-deleting-an-entity-that-is-used-in-a-union-type/15483

https://discuss.dgraph.io/t/bug-cannot-limit-number-of-results-using-auth-directive-to-prevent-malicious-queries/14828/5

In the end Kudos to Teammates who made this migration possible!

GraphQL Security: The complete guide

Level Up Coding

Thanks for being a part of our community! Before you go:

👏 Clap for the story and follow the author 👉
📰 View more content in the Level Up Coding publication
💰 Free coding interview course ⇒ View Course
🔔 Follow us: Twitter | LinkedIn | Newsletter

🚀👉 Join the Level Up talent collective and find an amazing job

How to Prevent No-SQL Injection in Node.js

Poorshad Shaddel — Thu, 20 Apr 2023 16:32:14 +0000

This article explains what is No-SQL Injection attack, how usually attackers make it and, how we can prevent it.

The attack is called SQL Injection but the same concept applies to No-SQL databases too.

SQL or No-SQL Injection

Why it is important to know (No-)SQL Injection?

By taking a look at HackerOne Top Ten Vulnerabilities we can quickly understand the importance of SQL Injection and why we should care about it.

SQL Injection Bounty ranking

There are two main reasons that make (No-)SQL Injection very tempting for attackers:

Abundance of (No-)SQL Injection vulnerabilities
Database is a very attractive target since it contains critical data

What is a (No-)SQL Injection?

What we do on the backend side is mostly based on the data that comes from the request(user). An example is when a user does Login. So we have to get the email of the user and check if it exists and matches the password. We store data in the database and as a result, we have to make a query to the database to find the email. So the Email that the User gave to us is part of the Query. This is where the attacker can do some tricks to extract or manipulate data by giving us something other than a simple Email. This is how an SQL Injection is done.

The Intention of (No-)SQL Injection is access to the database or parts of the database that it should not have.

Examples of No-SQL Injection Attacks

No-SQL Injection(MongoDB)

In the first example, we want to work on Express app that uses MongoDB Native Driver. Why anyone might want to use the native driver? Because in terms of performance, it is the best!(As you can see in the chart below)

Mongoose vs MongoDB Driver

import { MongoClient } from "mongodb";
import { Router } from "express";
const url = "mongodb://localhost:27017";
const client = new MongoClient(url);
const dbName = "raw-mongodb";

export const userMongoDBRouter = Router();

userMongoDBRouter.post("/login", async (req, res) => {
  const { email, password } = req.body;
  await client.connect();
  const db = client.db(dbName);
  const collection = db.collection("users");
  const user = await collection.findOne({
    email,
    password: hashPassword(password)
  });
  if (!user) {
    res.status(401).send("Invalid credentials");
    return;
  } else {
    res.json({ message: "Login successful" });
  }
});

userMongoDBRouter.post("/register", async (req, res) => {
  const { email, password } = req.body;
  await client.connect();
  const db = client.db(dbName);
  const collection = db.collection("users");
  const user = await collection.insertOne({
    email,
    password: hashPassword(password)
  });
  res.json({ message: "User created", user });
});

const hashPassword = (password: string) => {
  return password;
};

We have a login route that does not have Validation and passes email and password directly to MongoDB Driver.

First, let’s create a user with the register route and check the Login logic:

Create a Valid User with Password and Email

Now we can login with this valid user that we just created but If we use an invalid user, obviously we cannot Login:

Failed Login with User: Attacker@gmail.com

How to do the No-SQL Injection

We know that MongoDB supports regex with the operator $regex so we can pass this operator instead of a String and use a regex that accepts anything ,*:

{
  "email": {
    "$regex": ".*" 
  },
  "password": {
    "$regex": ".*" 
  }
}

As you can see in the picture we were able to Login without having a user:

Using Regex Operator to Bypass MongoDB findOne

This was done with MongoDB native driver. Using other ODM’s like Mongoose will not solve this issue:

userMongooseRouter.post("/login", async (req, res) => {
  const { email, password } = req.body;
  await mongoose.connect(url, { dbName });
  const db = mongoose.connection;
  const collection = db.collection("users");
  // mongoose enable debug mode
  mongoose.set("debug", true);
  const user = await collection.findOne({
    email,
    password
  });
  if (!user) {
    res.status(401).send("Invalid credentials");
    return;
  } else {
    res.json({ message: "Login successful" });
  }
});

Mongoose is behaving the way we expect. We can bypass login by using $regex .

Bypassing Login when using Mongoose

Is it the responsibility of ODM to Prevent (No-)SQL Injections?

I was mad at my ODM(Mongoose) at first but after a little bit of thinking I realized that Mongoose or Prisma are just libraries for easier database interactions. How could they distinguish that it is you that wants to use the regex or an attacker?

Long story short: It is not the responsibility of ODM or ORM and even if some of them are doing it you cannot be so sure that your app is safe against Injections(Sequalize Security Issue that resulted in SQL Injection).

How to Prevent No-SQL Injection

Avoid Passing Request Objects Directly To ODM or ORM Functions

The worst thing we can do is to pass something like req.body or req.query directly to our ODM/ORM functions:

const user = await collection.findOne(req.body); // Bad Practice

The least we can do is to use the specific fields

const user = await collection.findOne({ userId: req.body.id });

Use Input Validation

Using something like Zod, Yup, and other options.

Example of validating with Zod:

const loginValidator = z.object({
  email: z.string().email(),
  password: z.string()
});

By using objectStrict it will return an error if someone tries to add extra fields.

const loginValidator = z.strictObject({
  email: z.string().email(),
  password: z.string()
});
userMongooseRouter.post("/login", async (req, res) => {
  const result = loginValidator.safeParse(req.body);
  if (!result.success) {
    res.status(401).send("Invalid credentials");
    return;
  }
  await mongoose.connect(url, { dbName });
  const db = mongoose.connection;
  const collection = db.collection("users");
  // mongoose enable debug mode
  mongoose.set("debug", true);
  const user = await collection.findOne(result.data);
  if (!user) {
    res.status(401).send("Invalid credentials");
    return;
  } else {
    res.json({ message: "Login successful", user: user });
  }
});

Now sending a request like this will result in an error:

Sending Extra Field in the Body of Request

Sanitize User Inputs and Filters

Hopefully, Mongoose 6 introduced a sanitizer that you can use: “Sanitizes query filters against query selector injection attacks by wrapping any nested objects that have a property whose name starts with $ in a $eq.”

const obj = { username: 'val', pwd: { $ne: null } };
sanitizeFilter(obj);
obj; // { username: 'val', pwd: { $eq: { $ne: null } } });

By wrapping phrases in $eq it prevents injections that are trying to get more data.

If we are not using Mongoose there are other options to consider:

express-mongo-sanitize is a library that checks req.body, req.params, req.query and req.headers for prohibited characters.

Another option is mongo-sanitize that strip out any keys which start with $ in the input.

Conclusion

We know that (No-)SQL Injection is important and common since the target is the database. We need to do some simple steps to prevent this attack: 1- Avoid passing req objects directly, use strong validation, and in the end use sanitizers.

These other articles might be also interesting for you:

Level Up Coding

Thanks for being a part of our community! Before you go:

👏 Clap for the story and follow the author 👉
📰 View more content in the Level Up Coding publication
💰 Free coding interview course ⇒ View Course
🔔 Follow us: Twitter | LinkedIn | Newsletter

🚀👉 Join the Level Up talent collective and find an amazing job

How to prevent SSRF attacks in Node.js

Poorshad Shaddel — Mon, 10 Apr 2023 00:43:28 +0000

This articles explains what is SSRF attack, how usually attackers make it and how we can prevent it.

Why it is important to know SSRF?

By taking a look at HackerOne Top Ten Volunuriblities we can quickly understand the importance of SSRF and why me and you as developers should care about it.

HackerOne Bounty Table By Type

What is a SSRF Attack?

SSRF is the abbreviation for Server Side Request Forgery. Let’s explain it with an example instead of classic definitions. Imagine that you have a application, and your users need profile picture. Suddenly you decide that just uploading the picture is not enough for my users should be able to pass a URL, and I will download the picture and use it as their avatar. The problem begins in here. Downloading the picture means a GET request to a server that we do not have that much information about. Now that the attacker nows that you are making these GET requests he passes a URL like this to you: http://localhost/admin . If your system did not consider this attack you will send a GET request to your host admin which is definitely dangerous when someone can somehow interact with your Internal Systems or Services. Now if you are showing the result of the image download request it could make it much easier for attacker to explore your internal services and do something. I should mention that this type of attack is not limited to Internal Services and it could affect also External Systems.

The intention of the SSRF Attack is usually to exploit trust relationships to escalate an attack from the vulnerable application and perform unauthorized actions.

Different Types of SSRF Attacks

Server Attacks : In the example of downloading user Avatar from a URL if we pass something like localhost or 127.0.0.1 or the IP of the server we are attacking the server itself.
Internal Backend Systems : In this case attacker tries to communicate with other backend systems and for doing that he uses URL’s that contain private network IPs like http://192.168.0.68/admin or even using names like https://payment/something .
External Systems : This could be an external system that only your server or service is able to interact. This could be a bank API or anything that is restricted their service to your IP address with some other forms of authentication and authorization.

Schema of a SSRF attack in attempt to get access to 10.0.0.1

What is Basic SSRF and Blind SSRF?

In the example of downloading user picture from a URL if we return the response directly to the user, then the attacker is able to see the response and explore our services much faster. If we do not show any response from the request then the attack type becomes blind SSRF which becomes harder for the attacker to explore our systems without seeing the response.

An Example of a SSRF Attack

What This Service Is Doing : It is a resume builder website and when you are filling a field for your personal website it checks if it is a valid website or not and gives you a feedback to fix your URL if it cannot make a request to it! This application has a base part which has routes and you have to authenticate yourself to access these routes. We have a internal service called Payment that only the main app have access to it and it is not public.(This is full of simplification and the intention is to understand a situation that this could happen).

Let’s see the main app. It consists of :

personal-website: for checking the url and it gives you the message that says success and also returns response that came from your personal website(This is not a common thing to do but we are doing this to not make it a blind attack) to show a preview of the website to you.
me/balance: for checking current user’s balance. It makes a request to our payment service.
me/payin: for adding money to my wallet. It makes a request to our payment service.

const express = require('express');
const app = express();
const axios = require('axios');
app.use(express.json());
const paymentServiceAddress = 'localhost:3011'
app.post('/personal-website', async (req, res) => {
    const url = req.body?.url;
    try {
        const websiteHomePageTest = await axios.get(url);
        await saveAddressInProfile(url);
        res.json({ message: 'success', url, data: websiteHomePageTest.data });
    }
    catch (error) {
        if (axios.isAxiosError(error)) {
            return res.json({ message: 'failed', error: error.message }).status(500);
        } else {
            console.error(error);
            res.json({ message: 'failed' }).status(500);
        }
    }
});

app.post('/me/balance', someAuthenticationMiddleware, async (req, res) => {
    const response = await axios.post(`http://${paymentServiceAddress}/1/balance`);
    const balance = response.data.balance;
    res.json({ balance });
})

app.post('/me/balance/payin', someAuthenticationMiddleware, async (req, res) => {
    const payIn = await payIn()
    // now we want to update the balance
    const response = await axios.post(`http://${paymentServiceAddress}/1/balance/increase`);
    const balance = response.data.balance;
    res.json({ balance });
})

app.listen(3010, () => console.log(`Example app listening at http://localhost:3010`));
async function saveAddressInProfile(image) { console.log('save the image'); }
function someAuthenticationMiddleware(req, res, next) { next(); }
async function payIn() { }

If we use a valid URL for our personal website we get this result:

HTTP Client that shows the message is success

At this point we are sure that this app is making a request.

Now if we try some other URL’s we can check other services that could be exploit.

Using local URL for SSRF

It gives us information about the server and internal services.

Now the attacker should start playing with the URL to find some other services. Finally if we try an address like http://localhost:3011 we see that the result becomes success and it means that there is a service behind this URL. Let’s see the implementation of payment service. We cannot make direct requests to this service since it is in a private network but we can make requests from the main app.

const express = require('express');
const app = express();
app.use(express.static('static'));
app.use(express.json());

app.get('/', (req, res) => {
    res.send('pong');
});

app.get('/:userId/balance', async (req, res) => {
    const balance = getUserBalance(req.params.userId);
    res.json({ balance });
});

const users = [
    { id: 1, name: 'John', balance: 100 },
    { id: 2, name: 'Jane', balance: 200 },
    { id: 3, name: 'Jack', balance: 300 }
]
function getUserBalance(userId) {
    return users.find(user => user.id == userId)?.balance;
}

app.listen(3011, () => {
    console.log(`Example app listening at http://localhost:3011`);
});

After a little bit of trying the attacker could find that there is a route that gives balance to the user.(when the URL is not correct the payment service returns 404 and we see the result in the response).

By using the balance route which is a GET request we can access all users balance by using their userId. Normally by using this route on main app /me/balance we were able to see our own balance but now we have unauthorized access to other users’ balance too.

Unauthorized Access to Users’ Balance

How to Prevent SSRF Attacks

Now it is time to think about solution and prevent this attack step by step.

1- Validation

Black List

You can use Regex for validating the URL or having a black list of forbidden phrases like 127.0.0.1 or localhost. You can use regex directly or you can use validators like Zod ,hapi , validatorjs and so many other options.

IP Address

If you are accepting IP addresses or if you have a whitelist or black list that you want to check against, you can use this library for that: ip-address

Domain

If you are only accepting Domain names then this library could be helpful: is-valid-domain. It is archived but it does not have any dependency to other projects.

Warning

Be aware of the problem that there are so many ways to bypass the validation. For example:

Using an alternative IP representation of 127.0.0.1, such as 2130706433, 017700000001, or 127.1.
Registering your own domain name that resolves to 127.0.0.1. You can use spoofed.burpcollaborator.net for this purpose.
Obfuscating blocked strings using URL encoding or case variation.

So this input validation is just the beginning.

2- Network Layer

If a service does not interact with another service it should not have access to that service in internal private network. Since we are working with docker and k8s a lot this might happen that two pod or container do not have any relation but they can make requests to each other. This could be a potential issue. If a pod is using another pod called database and other pods do not need this database then these other pods should not be able to make requests to this pod(Azure Example).

3- Use SSRF Agents

By using Nodejs libraries like ssrf-req-filter or ssrf-agent you can prevent passing private URL’s to your app.

4- Use Proper Authentication and Authorization

In our example we saw that the main application had access to make any kind of requests to payment service. A better approach could be to pass in the user authentication information(it might be a JWT or session or anything) and then payment gets the userId from JWT payload and then we are sure that this user has access to this data. Even backend services should have limited access on each other. If they rely on user access then it could be harder to get unauthorized access.

5- Do Not Expose Response of Unknown Requests

Another mistake in the example was that it was exposing a http request response both in success and failure mode. Any extra information could be a potential clue for the attacker. So when the personal website is not available or gives internal server we should not care. We should always return something similar, for example a bad request.

Conclusion

Overall making a request based on client url is a risk and by doing validation, checking that it is not a private IP address or not and testing against some black list phrases we can reduce the risk. Best thing to do is to have a concrete authentication authorization system that could stay resilient even when the systems are making requests that they should’t.

Thanks for reading this article.

In case you like it you can check my previous articles about security in Nodejs:

Level Up Coding

Thanks for being a part of our community! Before you go:

👏 Clap for the story and follow the author 👉
📰 View more content in the Level Up Coding publication
💰 Free coding interview course ⇒ View Course
🔔 Follow us: Twitter | LinkedIn | Newsletter

🚀👉 Join the Level Up talent collective and find an amazing job

Boost Your Productivity: Simple Tips for Developers

Poorshad Shaddel — Mon, 13 Mar 2023 02:26:45 +0000

This article provides useful tips for maintaining productivity during development work.

Continue reading on Level Up Coding »

Build your First Node.js Web Framework Step by Step

Poorshad Shaddel — Mon, 02 Jan 2023 01:27:22 +0000

We want to create a super simple web framework in order to get an idea of how web framework libraries are working.

Build Your First Node.js Web Framework

In the previous article, we took a look at Expressjs and tried to understand how this library works. In this article, we want to build something super simple in order to feel the challenges and feel comfortable when we want to take a look at the source code of other libraries.

Let’s get to coding without hesitation.

Handle Requests using Node.js HTTP Library

We are going to use the Node.js HTTP library which is one of the core libraries and you do not need to install anything.

const http = require('node:http');

const server = http.createServer((req, res) => {
  res.end('pong');
});

server.listen(8000);

Now if we send a request to any route in localhost:8000/anything/you/want we get pong as a result. Now let’s implement our features.

Our Strategy for Handling Requests

For handling a request in a normal application most of the time we are doing totally different operations on the same incoming request. As an example, a request wants to update a user. And imagine this is how the request looks like:

Method: POST
URL: "http://ourAddress/users/userId"
JSON Body of request: { name: "something New" }

As a backend service I want to do different operations on this request:

Authorization
Parse Body of the request
Log the operation and the person who did the operation
Update the User in the Database

function myHandler(req, res) {
    // Authorization
    const isAuthorizedUser = authorizeRequest(req);
    if (!isAuthorizedUser) {
        // return some bad error
        res.send(...);
        return;
    }

    // Parse Body of the Request
    bodyParser(req);

    // Log the error
    logger(req);


    // Update the user
    updateUser(req);

    res.end();
}

This code is totally OK. But imagine there are more than twenty different routes in the same application and all of them need almost the same operations. All of them need to log every action or authorize the request. So we need a way to reduce code duplications. One common strategy for solving this issue is the concept of middleware. ExpressJS uses this concept and we also want to implement something like this.

Now look at this code:

const http = require('node:http');

const server = http.createServer((req, res) => {
    function authorizationMiddleware(req, res) {
        const isAuthorizedUser = authorizeRequest(req);
        if (!isAuthorizedUser) {
            // return some bad error
            res.send(...);
            // How to tell next functions that we already send the request to the user?
        } else {
            // What to do? should go to the next function
        }
    }
    function bodyParserMiddleware(req, res) {
        bodyParser(req);
    };
    function logger(req) {
        console.log(req);
    };
    function updateUser(req, res) {
        updateUser(req);
        res.end();
    }
    authorizationMiddleware(req, res);
    bodyParserMiddleware(req, res);
    logger(req, res);
    updateUser(req, res);
});

server.listen(8000);

In this case, we only have one route in our application and it updates the user.

Why Do You Need a Next Function?

There is a big difference between authorizationMiddleware and the other ones, if the user is unauthorized we are returning the response to the user and it does not make sense to send the request to other handlers like body-parser, logger, or update user. So we need a mechanism for knowing when the request is sent and we should stop handing over the request and response object to the middlewares.

For doing this the first thing we need to do is to store these middlewares somewhere. Then we can implement the next function to pass the request to these middlewares. At the same time, we can solve the problem of having different handlers for different functions.

In Expressjs they keep middlewares(handler functions) in an array. Let’s do it kind of the same way:

const http = require('node:http');

function FastFramework() {
    // An array for keeping the middlewares
    const middlewares = [];
    const server = http.createServer((req, res) => {
      let middlewareCounter = 0;
      function nextFunction() {
        while(middlewareCounter < middlewares.length) {
          console.log("Next Function While Loop- Current Counter:", middlewareCounter);
          if (match(req, middlewares[middlewareCounter])) {
            return middlewares[middlewareCounter++].handler(req, res, nextFunction);
          } else {
            middlewareCounter++
            }
          }
            // If we are here then no middleware matched!!!!
        }
        nextFunction();
    });
    // A method that we can add middelwares
    server.use = function use(url, method, handler) {
        middlewares.push({
          url,
          method,
          handler
        })
    }
    return server;
}

// if the request is a match for the middleware or not
function match(req, { url, method }) {
    if (req.method !== method) return false;
    if (req.url !== url) return false;
    return true;
}

module.exports = { FastFramework }

The function FastFrameWork is just for returning the HTTP instance.

In line 5 we see an array for keeping all the middlewares. After creating an HTTP server we defined a variable for keeping track of the current middleware that should be handled.

Next Function

In line 8 we implemented the nextFunction and later on, we called this function so it can start processing the request and response, and we intentionally put the variable middlewareCounter outside the next function. In line 10 we have a while loop, it loops over middlewares until it finds a match.

If it is a match in line 12 we are calling the handler function of the middleware. The most important thing happens here:

return middlewares[middlewareCounter++].handler(req, res, nextFunction)

We are also passing the next function as an argument to the handler, so we can use this argument to call this function and let other middlewares handle the request. If we are sending the response in the implementation of our middleware handlers and we are not calling the next function which means that this is the last stop for the request. If do not call the next function and we do not return any response then the request will timeout after a while. Also, we are increasing the counter variable because we want the next middleware next time we are in this piece of code.

If it was not a match in line 14 we just increase the variable so the while loop will check the next middleware or if it was the last middleware it gets out of the loop and it does nothing.

Use Function

Like Express I named it use method. This function just adds the middleware to the array of middlewares. The arguments are url , method and handler should be a function like this: (req, res, next) => {...} .

Match Function

Now let’s implement something for this match function. We can check if the method and URL are the same or not.

function match(req, { url, method }) {
    if (req.method !== method) return false;
    if (req.url !== url) return false;
    return true;
}

We know that right now it cannot handle URL patterns like this:

/api/users/:id but that is OK.

Usage

We can import this library like this:

const { FastFramework } = require("./lib");

const app = FastFramework();

app.use('/users', 'GET', (req, res, next) => {
    console.log('authrized the user...');
    next();
});
app.use('/users', 'GET', (req, res, next) => {
    console.log('logged the user...');
    next();
});
app.use('/users', 'GET', (req, res, next) => {
    console.log('prepating the list of users...');
    res.end('some users...');
});
app.use('/otherRoute', 'GET', (req, res, next) => {
    console.log('other route');
    res.end('other info');
});
app.listen(8000);

If we send a request we must see some users...

Response to the request.

We expect to see logs of all the routes in order except the /otherRoute which does not match the request:

Log of response to the request: GET /users

As expected first middleware that is handled is the authorization, then the log, and finally the users.

Now if we send a request to /otherRoute we should only see the log of otherRoute:

Log of the request to otherRoute

How to Handle 404

For handling not found requests we need a middleware that matches every request. And we put this middleware as the last middleware so if we know that this request did not match with any of the middlewares. Let’s do this small change to our framework. It is just your choice how you want to implement this:

function match(req, { url, method }) {
    if(!url && !method) return true;
    if (req.method !== method) return false;
    if (req.url !== url) return false;
    return true;
}

Now if we do not pass URL and Method to the use function it returns true which means that it going to be a match.

Let’s add our not found middleware as the last middleware.

app.use(null, null, (req, res) => {
    res.end('NOT_FOUND');
});

Now if we send a request to an unknown URL:

Request to a route that does not exist

Implement res.json Method

We need to add this code after creating the server(where we have the request and response):

res.json = function(data) {
  res.setHeader('Content-Type', 'application/json');
  res.end(JSON.stringify(data));
}

Our FastFramework function looks like this after this change:

function FastFramework() {
    // An array for keeping the middlewares
    const middlewares = [];
    const server = http.createServer((req, res) => {
        res.json = function(data) {
            res.setHeader('Content-Type', 'application/json');
            res.end(JSON.stringify(data));
        }
        let middlewareCounter = 0;
        function nextFunction() {
            while(middlewareCounter < middlewares.length) {
                console.log("Next Function While Loop- Current Counter:", middlewareCounter);
                if (match(req, middlewares[middlewareCounter])) {
                    return middlewares[middlewareCounter++].handler(req, res, nextFunction);
                } else {
                    middlewareCounter++
                }
            }
            // If we are here then no middleware matched!!!!
        }
        nextFunction();
    });
    // A method that we can add middelwares
    server.use = function use(url, method, handler) {
        middlewares.push({
          url,
          method,
          handler
        })
    }
    return server;
}

Let’s use this function in a simple middleware:

app.use('/users', 'GET', (req, res, next) => {
    console.log('prepating the list of users...');
    res.json({ list: ['user1', 'user2'], count: 2 });
})

This is the result.

Body Parser

We can use the body-parser library under the Expressjs project:

npm i body-parser

It is usable because our Framework also has the next function:

const bodyParser = require('body-parser');

const app = FastFramework();

app.use(null, null, bodyParser);

Note that we did not pass a URL or method body parser will be used for every request after this middleware.

Let’s see the request body before and after using body-parser.

const { FastFramework } = require("./lib");
const bodyParser = require('body-parser');

const app = FastFramework();

app.use('/user', 'POST', (req, res, next) => {
    console.log("body before:", req.body);
    next();
});

app.use(null, null, bodyParser.json());

app.use('/user', 'POST', (req, res) => {
    console.log("body after:", req.body);
    res.json({});
});

The response is as expected an empty object and as we can see in the logs the body of the request is parsed correctly in it is in the req.body

Logs of request body before and after using body parser.

Summary

As you saw the most difficult thing was to add a concept like middleware for reusing the logic. For that, we implemented the next function and we were able to handle requests and pass the request and response objects between these middlewares. Feel free to add more functionality to your own HTTP Library.