Forem: Sumit Bhanushali

Production Ready NodeJS build using Docker and npm

Sumit Bhanushali — Tue, 12 Dec 2023 07:34:19 +0000

An app with a smooth deployment process is reliable and has fewer bugs because it is easy to release fixes and is easier to set up in any environment. An app can lose all its charm if it is not reliable. When users want your app to work, it should work.
To set up a smooth deployment process, we will divide our process into three stages which will benefit Developers, DevOps and Users alike.

Build
The build Stage is when we are done with developing code and are confident for deployment. We thus run a build command that outputs an executable file that cannot be modified by us.
Release
Release Stage is where we combine our build with config which makes the app able to run. These releases should be tagged with a unique ReleaseID.
Run
Run Stage is where we run our release and make our app accessible to the outside world.

These stages should be strictly separated and any code change should trigger a new build, release and run stage. The Run stage should be as simple as possible. Any process or system restart can easily rerun our run stage without requiring manual intervention. This also simplifies rolling back to the previous release when required. Let's start with the first stage

Build

We will use Docker to build our app. We will be able to use the image generated from docker to deploy on any machine where docker is installed.
We will need a few npm scripts which will be used in our Dockerfile.

"prisma:generate": "prisma generate",
"start": "node .dist/index.js",
"build": "tsc",

Next we will create Dockerfile. We will use a multi-stage build technique to keep our image size minimal.

FROM node:20-alpine AS builder

WORKDIR /usr/src/app

COPY package*.json ./

RUN npm ci --only=production

COPY . .

RUN npm run build && npm run prisma:generate

##### STAGE 2 #####

FROM node:20-alpine

WORKDIR /usr/src/app

COPY --from=builder /usr/src/app/.dist ./.dist
COPY --from=builder /usr/src/app/node_modules ./node_modules
COPY --from=builder /usr/src/app/package.json ./package.json

EXPOSE 3000

CMD ["npm", "start"]

These commands RUN npm run build && npm run prisma:generate is where our Typescript code gets converted to Javascript and necessary prisma client files will be created.

In Stage 2, we will only copy files from .dist folder and discard typescript files.
This config.env file will be different for different environments like UAT, Production, etc. We will be required to manually create/update this file before creating a Docker container.

We will use this command to build a Docker image, we are using the version from our package version to tag our images. In this case, our image name will be bloggo:0.1.0

docker image build -t bloggo:$(node -p \"require('./package.json').version\") .

We can further simplify the build process by introducing some npm scripts.

Build image by just running npm run build

"build:image": "docker image build -t bloggo:$(node -p \"require('./package.json').version\") ."

Upgrade the version accordingly and build an image. Note that npm version requires that all changes are already committed and this command will create a new commit with the version number as the tag

"build:major": "npm version major && npm run build:image",
"build:minor": "npm version minor && npm run build:image",
"build:patch": "npm version patch && npm run build:image"

Full npm scripts for reference

"scripts": {
    "prisma:generate": "prisma generate",
    "start": "node .dist/index.js",
    "start:dev": "nodemon --exec node -r ts-node/register --env-file=config.env index.ts",
    "build": "tsc",
    "build:image": "docker image build -t bloggo:$(node -p \"require('./package.json').version\") .",
    "build:major": "npm version major && npm run build:image",
    "build:minor": "npm version minor && npm run build:image",
    "build:patch": "npm version patch && npm run build:image"
}

Release

Release = Build + Config

This command creates a Release version for us. Here Docker's ContainerID can be treated as ReleaseID. We are using --env-file to load config.env file in our container

docker container create -p 3000:3000 --env-file config.env bloggo:0.1.0

Run

To fetch ContainerID, which we will use to run our container

docker container ps

To start container and add restart policy

docker container start a897dbba5329 && docker update --restart unless-stopped a897dbba5329

With this setup, we will be easily be able to generate deployment builds and deploy to any number of servers with just few changes in config (if required) reliably

Link to the project that will be used in this post: https://github.com/sumitbhanushali/bloggo

Production Ready NodeJS build using Docker and npm

Sumit Bhanushali — Tue, 12 Dec 2023 07:34:18 +0000

Build
The build Stage is when we are done with developing code and are confident for deployment. We thus run a build command that outputs an executable file that cannot be modified by us.
Release
Release Stage is where we combine our build with config which makes the app able to run. These releases should be tagged with a unique ReleaseID.
Run
Run Stage is where we run our release and make our app accessible to the outside world.

Build

"prisma:generate": "prisma generate",
"start": "node .dist/index.js",
"build": "tsc",

Next we will create Dockerfile. We will use a multi-stage build technique to keep our image size minimal.

FROM node:20-alpine AS builder

WORKDIR /usr/src/app

COPY package*.json ./

RUN npm ci --only=production

COPY . .

RUN npm run build && npm run prisma:generate

##### STAGE 2 #####

FROM node:20-alpine

WORKDIR /usr/src/app

COPY --from=builder /usr/src/app/.dist ./.dist
COPY --from=builder /usr/src/app/node_modules ./node_modules
COPY --from=builder /usr/src/app/package.json ./package.json

EXPOSE 3000

CMD ["npm", "start"]

These commands RUN npm run build && npm run prisma:generate is where our Typescript code gets converted to Javascript and necessary prisma client files will be created.

We will use this command to build a Docker image, we are using the version from our package version to tag our images. In this case, our image name will be bloggo:0.1.0

docker image build -t bloggo:$(node -p \"require('./package.json').version\") .

We can further simplify the build process by introducing some npm scripts.

Build image by just running npm run build

"build:image": "docker image build -t bloggo:$(node -p \"require('./package.json').version\") ."

Upgrade the version accordingly and build an image. Note that npm version requires that all changes are already committed and this command will create a new commit with the version number as the tag

"build:major": "npm version major && npm run build:image",
"build:minor": "npm version minor && npm run build:image",
"build:patch": "npm version patch && npm run build:image"

Full npm scripts for reference

"scripts": {
    "prisma:generate": "prisma generate",
    "start": "node .dist/index.js",
    "start:dev": "nodemon --exec node -r ts-node/register --env-file=config.env index.ts",
    "build": "tsc",
    "build:image": "docker image build -t bloggo:$(node -p \"require('./package.json').version\") .",
    "build:major": "npm version major && npm run build:image",
    "build:minor": "npm version minor && npm run build:image",
    "build:patch": "npm version patch && npm run build:image"
}

Release

Release = Build + Config

This command creates a Release version for us. Here Docker's ContainerID can be treated as ReleaseID. We are using --env-file to load config.env file in our container

docker container create -p 3000:3000 --env-file config.env bloggo:0.1.0

Run

To fetch ContainerID, which we will use to run our container

docker container ps

To start container and add restart policy

docker container start a897dbba5329 && docker update --restart unless-stopped a897dbba5329

With this setup, we will be easily be able to generate deployment builds and deploy to any number of servers with just few changes in config (if required) reliably

Link to the project that will be used in this post: https://github.com/sumitbhanushali/bloggo

12 Factor App in NodeJS Part 1 (1-4)

Sumit Bhanushali — Thu, 07 Dec 2023 10:02:23 +0000

In Part 0, we got an overview of the 12-factor app methodology and its importance in building a scalable and reliable backend. In this Part, we will learn in detail about the first 4 factors and simultaneously implement them by building a blogging platform using NodeJS

1. Codebase

Nowadays, everyone is familiar with version control systems aka VCS, such as Git, Mercurial, or Subversion, Git being the most popular one. It has become second nature to us that the first thing we do after initializing our project is run git init && git add . && git commit -m "Initial commit". To understand how important VCS is let's assume we don't know about VCS or something like VCS doesn't even exist.

Let's take 4 scenarios to understand how difficulty escalates exponentially
1) Working Solo on a Project with Single Deployment
2) Working Solo on a Project with Multiple Versions and Deployments
3) Working in a Team of 4 with a Single Version and Deployment
4) Working in a Team of 4 with Multiple Versions and Deployments

Here, deployment means a version of the codebase. Single Deployment means only one version of the codebase that will be deployed. Multiple Deployments means multiple versions(including client-specific) of codebase in multiple environments(staging, UAT, production)

Scenario 1: Working Solo on a Project with Single Deployment

As a solo developer without version control, working on a project with a single deployment becomes a risky situation. Any code changes made are irreversible, and there's no safety net in case of errors. If a bug is introduced, there's no straightforward way to revert to a previous, stable state. The lack of versioning makes it challenging to track changes over time, and it's easy to lose sight of the project's evolution.

Collaboration with past iterations is impossible, making it difficult to understand the reasoning behind certain decisions or to revisit successful implementations. The absence of branching and merging capabilities means that experimenting with new features or fixes could disrupt the entire codebase.

Deployments are a nerve-wracking process. If an issue arises during deployment, rolling back to a working state is a manual and error-prone task. Without a VCS, there are no efficient means of documenting the deployed versions or managing configurations for different environments.

Scenario 2: Working Solo on a Project with Multiple Versions and Deployments

In this scenario, the absence of version control worsens the challenges faced in the first scenario. Managing multiple versions and deployments without VCS becomes a logistical nightmare. Keeping track of different code states for various features or environments is practically impossible.

Each deployment is a high-stakes endeavor, as there is no systematic way to isolate changes specific to a version or deployment. Debugging becomes a herculean task, and the risk of introducing new bugs while fixing existing issues looms large. Without branching, trying out experimental features or implementing temporary fixes without affecting the main codebase becomes nearly impossible.

Documentation is limited to external notes or comments within the code, making it hard to understand the rationale behind specific versions or deployments. Coordinating changes across different branches or versions requires meticulous manual effort, leading to a high probability of errors.

Scenario 3: Working in a Team of 4 with Single Version and Deployment

Without version control in a team setting, collaboration turns chaotic. Coordinating efforts among team members becomes a communication-heavy process, relying on constant updates and manual file sharing. The risk of overwriting each other's work is ever-present, and resolving conflicts is an arduous manual task.

There's no clear history of changes, making it difficult to attribute modifications to specific team members, there will be blame games of faulty code as we won't be able to pinpoint who introduced the faulty code. Debugging and troubleshooting are cumbersome, as there's no easy way to identify when and why a particular change was made.

Deployment is a risky and unpredictable process, with no efficient rollback mechanism. Coordinating releases requires meticulous planning and synchronization among team members, increasing the likelihood of errors and downtime.

Scenario 4: Working in a Team of 4 with Multiple Versions and Deployments

In a team setting with multiple versions and deployments but without version control, the challenges escalate exponentially. Collaboration becomes a logistical nightmare, with each team member potentially working on a different version or deployment. Coordinating changes and ensuring a cohesive codebase is a constant struggle.

Without branching and merging capabilities, integrating features or fixes from different team members becomes an error-prone process. The risk of introducing conflicts and breaking the codebase during integration is high. Keeping track of changes across multiple branches or versions requires meticulous manual effort and is prone to oversight.

Deployments are high-stakes events, and rolling back to a stable state in case of issues is a manual and time-consuming task. The lack of versioning makes it challenging to manage configurations for different environments, leading to potential deployment errors.

It was just a Nightmare

To sum it up, without version control, it's like navigating a coding maze with a blindfold. Adding a version control system is like turning on the lights — suddenly, everything just clicks. I hope after reading this, your appreciation for VCS increases tenfold

Now that we know what it's like without VCS, let's create our project and initialize it with git. Our project will be a simple blog application that will be built using ExpressJS, Typescript and MySQL. We will name our project will be bloggo.

Let The Coding Begin

Create a Project folder with README.md

mkdir bloggo && cd bloggo && touch README.md && echo -e "# Bloggo \n\nSimple Blog Application" > README.md

Initialize Git Repo


git init && git add . && git commit -m "initial commit"

Link to commit for changes up to this section https://github.com/sumitbhanushali/bloggo/commit/a5f14218eab0fb5a84fdf1e31fe31965925e81d1

2. Dependencies

12 Factor app is easy to setup and run. This will be possible when all dependencies are declared in the dependency declaration manifest. These dependencies will be isolated i.e these dependencies will only apply to the current app and no global or system-wide package will be accessible inside it unless it is explicitly declared in the dependency declaration manifest.
This reduces friction to setup project on a developer's machine or even on production and leads to increased productivity and thus a better app will be delivered.
NodeJS has few popular package managers available like npm, pnpm and yarn. We will be using npm since it is the official package manager of NodeJS and most developers are already well-versed with it

This generates a default package.json file which will be used by npm to store metadata, dependencies and scripts

npm init -y

We will use express to create our app

npm i express

We don't want node_modules to be committed to git since it will unnecessarily increase our repo size which will make it longer to clone

echo "node_modules" > .gitignore

Our final package.json file will look like this

{
  "name": "bloggo",
  "version": "1.0.0",
  "description": "Simple Blog Application",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "express": "^4.18.2"
  }
}

Link to commit for changes in this post https://github.com/sumitbhanushali/bloggo/commit/a91c1424aa8a7279740338276af563bf1c0c5ee6

3. Config

An app’s config is everything likely to vary between deploys (staging, production, developer environments, etc) which are credentials to external services such as Database(MySQL, Redis), Cloud Providers or External APIs like Twitter, Dropbox, etc.
A beginner might hardcode such config in a codebase or might create separate versions of such file for each deployment. Such a person might not be aware of problems that can arise because of this. If such configs get in the wrong hands (hacker or fresher), that person can bring down the whole system by deleting the database or misconfigure something accidentally or purposely.
Configs should never be committed to VCS even if it is a private repository. A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials.

Some teams maintain a config file with defaults and edit this file manually at the server level according to environment needs. This is a good approach but there is still room for errors.

Another approach is to batch config into named groups (often called “environments”) named after specific deploys, such as the development, test, and production environments. This method does not scale cleanly: as more deploys of the app are created, new environment names are necessary, such as staging or QA. As the project grows further, developers may add their special environments like joes-staging, resulting in a combinatorial explosion of config which makes managing deploys of the app very brittle.

The best way is to make use of environment variables which are provided by every OS out there. In NodeJS, you can access these environment variables using process.env.ENV_VARIABLE where ENV_VARIABLE can be any variable name. dotenv is a popular package to simplify using environment variables which is used by almost every NodeJS project out there. From Node v20.6.0, support for reading environment variables from .env file is added which removes the dependency from dotenv` package.

We can choose to add this file into .gitignore and pass this file manually across team members or provide a default file for staging and editing this file on sensitive(internet-facing) environments

environment variables in NodeJS are accessed from process.env
`ts
//index.js

console.log(process.env.foo);
`

node only supports reading environment variables from files having .env extension
`
//config.env

foo=bar
`

add file to .gitignore, as we don't want to commit this file in our git

bash echo "config.env" >> .gitignore

we need to pass --env-file=config.env for NodeJS process to read our .env file

`bash
node --env-file=config.env index.js

`

Link to commit for changes in this post https://github.com/sumitbhanushali/bloggo/commit/56c3e46168a9146912db2814cb0030905d3d8300
references: https://nodejs.org/en/blog/release/v20.6.0

4. Backing Services

Make your app scalable

If you want your app to be scalable, you should keep components as decoupled as possible. Components like API Servers, Database, Message Brokers, Caching Servers, etc. This way you can independently scale them according to the application's needs. Our API Server should be prepared from the beginning for such changes to come Otherwise, It would take months of refactoring efforts depending on how big your codebase is.

Your App should make no distinction between local and the third party services. Each component must be pluggable where you can attach and detach them at your will. These components can be accessed via URL or some type of credentials which should be stored in the Config. This way, if your business needs require you to swap your local MySQL database instance to one managed by a third party service such as Amazon RDS, it would just require a change in URL stored in Config. The same applies to if you need to shard your database or need to replace a faulty database.

In previous posts, we initialized git repository, and did setup for adding dependencies and config. For this post, we require CRUD App Setup, scope of which is outside of this post hence I have created another post where I have added detailed setup instructions on how to setup Typescript with ExpressJS and Prisma as ORM which will communicate with our database. We will focus on connecting database using URL in this post using Prisma. You can checkout code at commit https://github.com/sumitbhanushali/bloggo/commit/03792df70faf7182c7ff88c62d9f5426d27627a9 and continue with below steps

bash npm install prisma --save-dev npx prisma init --datasource-provider mysql

This will generate prisma folder in our root directory where in schema.prisma will contain database connection details and prisma models which will be used to create/update tables in our database. We will add two models in schema.prisma

`ts
model User {
id Int @id @default(autoincrement())
email String @unique
name String?
blogs Blog[]
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
}

model Blog {
id Int @id @default(autoincrement())
title String
content String?
published Boolean @default(false)
author User @relation(fields: [authorId], references: [id])
authorId Int
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
}
`

This model files are self-explanatory. You can check more about prisma model here: https://www.prisma.io/docs/reference/api-reference/prisma-schema-reference
Next step is to execute prisma script to run migration which will create these model files inside our database. But before this we need to setup our mysql instance

`bash
docker pull mysql

docker run -d --name mysqldb -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password -d mysql
`
This will pull mysql image from docker hub and create container from it, we have set container name as mysqldb and root password as password. Our db will listen on port 3306

`bash
docker exec -it mysqldb sh

mysql -u root -p
// enter "password" when prompted

create database bloggo;
`
This way we can attach to our db container and execute our bash command to connect to mysql and create database named "bloggo". type exit to exit from mysql and container respectively
We will update our environment variable to our credentials in config.env file

DATABASE_URL="mysql://root:password@localhost:3306/bloggo"

Next we will execute prisma script to run migrations, this command also generates prisma client files which comes in handy when using ORM
bash npx prisma migrate dev --name init

You can now check that tables have been connected by connecting to mysql instance. Run this commands after connecting to mysql instance
`bash
use bloggo;
show tables;

describe Blog;
describe User;
`

Now we can use ORM to perform CRUD operations on our database. Let's update our route files to use prisma client to run operations.

`ts
// user.route.ts

import { PrismaClient } from '@prisma/client';
const prisma = new PrismaClient();

router.get('/', async (req: Request, res: Response) => {
const users = await prisma.user.findMany();
res.send(users);
});

router.get('/:id', async (req: Request, res: Response) => {
const user = await prisma.user.findUnique({
where: {
id: parseInt(req.params.id),
},
})

res.send(user);

});

router.post('/', async (req: Request, res: Response) => {
const user = await prisma.user.create({
data: req.body
})

res.send(user);

});

router.delete('/:id', async (req: Request, res: Response) => {
await prisma.user.delete({
where: {
id: parseInt(req.params.id),
},
})
res.send(User with id: ${req.params.id} deleted);
});
`

`ts
//blog.route.ts

import { PrismaClient } from '@prisma/client';
const prisma = new PrismaClient();

router.get('/', async (req: Request, res: Response) => {
const blogs = await prisma.blog.findMany();
res.send(blogs);
});

router.get('/:id', async (req: Request, res: Response) => {
const blog = await prisma.blog.findUnique({
where: {
id: parseInt(req.params.id),
},
})

res.send(blog);

});

router.post('/', async (req: Request, res: Response) => {
const createdBlog = await prisma.blog.create({ data: req.body });
res.send(createdBlog);
});

router.delete('/:id', async (req: Request, res: Response) => {
await prisma.blog.delete({
where: {
id: parseInt(req.params.id),
},
})
res.send(Blog with id: ${req.params.id} deleted);
});
`

Now, we can create user and their blog using your favorite tool like Postman.

`bash
// First we create a dummy user
curl --location 'localhost:3000/user' \
--header 'Content-Type: application/json' \
--data '{
"email": "abc@test.com"
}'

// Then blog from userId returned from previous command. In this case, authorId 1
curl --location 'localhost:3000/blog' \
--header 'Content-Type: application/json' \
--data '{
"title": "Title",
"content": "Content",
"authorId": 1
}'
`

Link to commit for changes in this post https://github.com/sumitbhanushali/bloggo/commit/a91c1424aa8a7279740338276af563bf1c0c5ee6

Up Next

This is where our Part 1 ends. We have learnt about VCS like git that how it can make a developer's life a lot simpler, about dependency declaration manifest that how it can make setting up project smoother, about storing config in env vars which makes deployment secure and benefits of using backing services as pluggable resources.
On next part we will continue with next 4 steps i.e 5. Build, Setup and Run, 6. Processes, 7. Port Binding, 8. Concurrency according to 12 Factor App's methodology

12 Factor App in NodeJS Part 0: Intro

Sumit Bhanushali — Thu, 30 Nov 2023 09:18:34 +0000

We have all heard that attachments are bad, well this applies to software development too. Your app maybe too dependent on OS or Cloud Provider. Your libraries might only work on specific OS or your app must be too much dependent on cloud sdks. Some OS are better suited in certain tasks than other and a policy change in some cloud provider can lead to huge expenses.

This leads to huge obstacle when you need to scale your app and can lead to huge expenses since moving out of particular cloud or OS is a herculean task. Your App should be flexible enough to make sure you dodge such bullets easily deployments must be as smooth as possible to be able to ship more features faster and without any bugs with minimal intervention of any human resource.

App must be able to scale automatically up or down on demand which leads to savings on cloud bills on non active hours and better User experience by delivering faster responses at peak times.

That’s why engineers from heroku came up with 12 Factor app methodology which applies to web apps in general. These methodologies when applied will help scale your app easily, make your app platform agnostic and make deployments smoother by maintaining proper build, test and deploy stages and keeping minimal difference between dev and prod environment. You can easily deploy your apps on aws, azure, gcp, heroku or even on on-premise servers

As mentioned in the name 12 Factor app is a list of 12 factors, I will list them here briefly and will write a detailed post with code implementation in NodeJS; ExpressJS + TS to be specific in later posts

I. Codebase

Track your codebase in revision control system such as git. Why? It helps maintaining multiple versions of an app in single codebase

II. Dependencies

Language runtime and its Package manager should be the only prerequisite to run the app. Other dependencies should be declared explicitly in a dependency declaration manifest file and should be isolated i.e no global packages can be used inside the app. This results in deterministic builds and thus makes it extremely simple to setup dev and prod environments

III. Config

An app’s config is everything that is likely to vary between deploys (staging, production, developer environments, etc) which are credentials to external services such as Database(MySQL, Redis), Cloud Providers or External APIs like twitter, dropbox, etc. Such configs should be stored in environment variables. This prevents accidental leaking of sensitive information and reduces hassle of updating such configs by just updating environment variables instead of manually editing file on server

IV. Backing Services

A backing service is any service the app consumes over the network as part of its normal operation. Examples include datastores (such as MySQL or MongoDB), messaging/queueing systems (such as RabbitMQ or Kafka), caching systems (such as Redis). App should be able to swap out a local MySQL database with one managed by a third party (such as Amazon RDS) without any changes to the app’s code. This makes app reliable and more easy to scale since Resources can be easily attached to and detached from deploys at will. For example, if the app’s database is misbehaving due to a hardware issue, the app’s administrator might spin up a new database server restored from a recent backup. The current production database could be detached, and the new database attached — all without any code changes.

V. Build, Release, Run

Build Stage is when we are done with developing code and are confident for deploy. We thus run a build command which outputs an executable file which cannot be modified by us.

Release Stage is where we combine our build with config which makes app able to run. This releases should be tagged with unique ReleaseID.

Run Stage is where we run our release and make our app accessible to outside world.

This stages should be strictly seperate and any change in code should trigger new build, release and run stage. Run stage should be as simple as possible, process or system restarts can easily rerun our run stage without requiring manual intervention. This also simplifies rolling back to previous release when required

VI. Processes

Our App runs on processes. It can be single process on developer’s machine or multiple processes on production. This processes must be stateless, all states should be stored in a storage solution. Temporary state can be stored in caching solution like redis and permanent state in Database like mysql. App should be able to handle process or system restarts and recover to right state. Also, since our app is running on multiple processes, same user can be routed to different process and its state information becomes inaccessible. Making our processes stateless makes our app reliable and ready to scale

VII. Port binding

Your service must be self contained and be able to bind to a port to serve requests without relying on other software such as nginx to create a web-facing service. This must happen entirely in user space, that is, within the app’s code. [why]

VIII. Concurrency

In the twelve-factor app, processes are a first class citizen. Using this model, the developer can architect their app to handle diverse workloads by assigning each type of work to a process type. For example, HTTP requests may be handled by a web process, and long-running background tasks handled by a worker process. Since our apps are already running on stateless processes, fearless concurrency is easily achieved and our app can scale without making any changes to code

IX. Disposability

Stateless processes are disposable, meaning they can be started or stopped at a moment’s notice. This facilitates fast elastic scaling and makes zero downtime deployments possible. Our app should take minimal time to startup and shutdown gracefully. Our app should also be prepared for sudden death. All these makes sure our app is reliable, easy to deploy and scale elastically

X. Dev/Prod Parity

To prevent “This works on my machine”. This happens because developers by using adapter solution use lightweight solution on development machine and different solution on prod. Some inconsistencies between these two can cause production to break which makes our app unreliable and thus damaging business

XI. Logs

Our app should not worry about routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout. It will be responsibility of execution environment on how it should handle stdout. In case when dev is developing on hist local machine, the developer will view this stream in the foreground of their terminal.

In staging or production deploys, each process stream will be captured by the execution environment, collated together with all other streams from the app, and routed to one or more final destinations for viewing and long-term archival.

XII. Admin Processes

Admin processes such as running database migration or running some script to debug should be run in an environment that is identical to target environment. Twelve-factor strongly favors languages which provide a REPL shell out of the box, and which make it easy to run one-off scripts. In a local deploy, developers invoke one-off admin processes by a direct shell command inside the app’s checkout directory. In a production deploy, developers can use ssh or other remote command execution mechanism provided by that deploy’s execution environment to run such a process.

Benefits of using Typescript with ExpressJS with Project Template

Sumit Bhanushali — Sat, 25 Nov 2023 14:39:10 +0000

In this post, we will setup our Backend service. We will be using ExpressJS for APIs, MySQL for storage and Prisma for ORM. We will be using Typescript as primary language. We will be using Docker to host MySQL on our machine. This will only work on Node Version: v20.10.0 and above.

Why Typescript?

TypeScript enhances Express.js web development by introducing static typing, catching potential bugs early and improving code maintainability. With superior IDE support and modern JavaScript features, TypeScript offers a productive development experience.TypeScript with Express.js combines the simplicity of Express.js with TypeScript's static typing benefits, resulting in more reliable and maintainable web applications.

Our project will be simple blog application named bloggo.

mkdir bloggo && cd bloggo && touch README.md && echo -e "# Bloggo \n\nSimple Blog Application" > README.md

npm init -y

To Add Typescript support

npm i -D typescript @types/node ts-node

npx tsc --init --rootDir ./ --outDir .dist --target es2020 --allowJs --noImplicitAny false

npm i -g nodemon

Here we have installed 3 packages as dev dependencies, their functionality is:

typescript: this package is required to transpile ts to js
@types/node: this provides type definitions like Request, Response for NodeJS
ts-node: this is combination of two package ts transpiles given ts files to js files and node executes those js files

we use tsc package to generate tsconfig.json for us. We need tsconfig.json file to instruct how to handle our ts files

--rootDir: location to all ts files
--outDir: location to place transpiled js files
--target: javascript version for js files to be generated
--allowJs: whether it should also allow js files, if disabled it will only allow ts files
--noImplicitAny: if true, will not allow any types to have ambigious 'any' declaration

nodemon will be used to watch our files and refresh automatically when any changes detected.

Next, we will install express and @types/express for building APIs

npm i express
npm i -D @types/express

Let's create a basic server which will just respond with Hello World on port 3000

//index.ts

import express, { Request, Response } from 'express';

const app = express();
const port = 3000;

app.get('/', (req: Request, res: Response) => {
    res.send("hello world");
})

app.listen(port, () => {
    console.log(`listening on ${port}`);
})

We will run this using nodemon we had previously installed, we will add some npm scripts in package.json to simplify our efforts

// code elided

"scripts": {
    "start:dev": "nodemon --exec node -r ts-node/register --env-file=config.env index.ts",
    "build": "tsc",
    "build:watch": "tsc --watch"
}

// code elided

- start:dev: we need to use ts-node we had installed previously to run ts files without rebuilding for every change done while development. Since ts-node does not directly support --env-file yet, we are using -r to require (import) ts-node/register

build: tsc which is part of typescript package will transpile our ts into js and place it in .dist folder as mentioned in tsconfig.json file
build⌚ This will run build in watch mode

On running npm run start:dev on terminal, our app will start running on port 3000. When you will navigate to localhost:3000 you will receive hello world.

Next, we will add some routes for interacting with blog and user resource. Create a folder named routes and add two files named blog.router.ts and user.router.ts

// blog.router.ts

import express, { Request, Response } from 'express';
const router = express.Router()

router.get('/', (req: Request, res: Response) => {
    res.send('List of blogs');
});

router.get('/:id', (req: Request, res: Response) => {
    res.send(`Blog: ${req.params.id} `)
});

router.post('/', (req: Request, res: Response) => {
    res.send(`Blog created`);
});

router.delete('/:id', (req: Request, res: Response) => {
    res.send(`Blog with id: ${req.params.id} deleted`);
});

export default router

//user.router.ts

import express, { Request, Response } from 'express';
const router = express.Router()

router.get('/', (req: Request, res: Response) => {
    res.send('List of users');
});

router.get('/:id', (req: Request, res: Response) => {
    res.send(`User: ${req.params.id} `)
});

router.post('/', (req: Request, res: Response) => {
    res.send(`User created`);
});

router.delete('/:id', (req: Request, res: Response) => {
    res.send(`User with id: ${req.params.id} deleted`);
});

export default router

Import these routes in our main index.ts file

//index.ts

import blogRouter from './routes/blog.route';
import userRouter from './routes/user.route';

app.use('/blog', blogRouter);
app.use('/user', userRouter);

You can now access respective routes at /blog/ and /user/

Next We need to add Database for storing our data. We will use image of MySQL from docker hub and prisma for ORM for performing CRUD operations on MySQL.
We will start with prisma.

npm install prisma --save-dev
npx prisma init --datasource-provider mysql

model User {
  id        Int      @id @default(autoincrement())
  email     String   @unique
  name      String?
  blogs     Blog[]
  createdAt DateTime @default(now())
  updatedAt DateTime @updatedAt
}

model Blog {
  id        Int      @id @default(autoincrement())
  title     String
  content   String?
  published Boolean  @default(false)
  author    User     @relation(fields: [authorId], references: [id])
  authorId  Int
  createdAt DateTime @default(now())
  updatedAt DateTime @updatedAt
}

docker pull mysql

docker run -d --name mysqldb -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password -d mysql

This will pull mysql image from docker hub and create container from it, we have set container name as mysqldb and root password as password. Our db will listen on port 3306

docker exec -it mysqldb sh

mysql -u root -p 
// enter "password" when prompted

create database bloggo;

This way we can attach to our db container and execute our bash command to connect to mysql and create database named "bloggo". type exit to exit from mysql and container respectively
We will update our environment variable to our credentials in config.env file

DATABASE_URL="mysql://root:password@localhost:3306/bloggo"

Next we will execute prisma script to run migrations, this command also generates prisma client files which comes in handy when using ORM

npx prisma migrate dev --name init

You can now check that tables have been connected by connecting to mysql instance. Run this commands after connecting to mysql instance

use bloggo;
show tables;

describe Blog;
describe User;

Now we can use ORM to perform CRUD operations on our database. Let's update our route files to use prisma client to run operations.

// user.route.ts

import { PrismaClient } from '@prisma/client';
const prisma = new PrismaClient();

router.get('/', async (req: Request, res: Response) => {
    const users = await prisma.user.findMany();
    res.send(users);
});

router.get('/:id', async (req: Request, res: Response) => {
    const user = await prisma.user.findUnique({
        where: {
            id: parseInt(req.params.id),
        },
    })

    res.send(user);
});

router.post('/', async (req: Request, res: Response) => {
    const user = await prisma.user.create({
        data: req.body
    })

    res.send(user);
});

router.delete('/:id', async (req: Request, res: Response) => {
    await prisma.user.delete({
        where: {
            id: parseInt(req.params.id),
        },
    })
    res.send(`User with id: ${req.params.id} deleted`);
});

//blog.route.ts

import { PrismaClient } from '@prisma/client';
const prisma = new PrismaClient();

router.get('/', async (req: Request, res: Response) => {
    const blogs = await prisma.blog.findMany();
    res.send(blogs);
});

router.get('/:id', async (req: Request, res: Response) => {
    const blog = await prisma.blog.findUnique({
        where: {
            id: parseInt(req.params.id),
        },
    })

    res.send(blog);
});

router.post('/', async (req: Request, res: Response) => {
    const createdBlog = await prisma.blog.create({ data: req.body });
    res.send(createdBlog);
});

router.delete('/:id', async (req: Request, res: Response) => {
    await prisma.blog.delete({
        where: {
            id: parseInt(req.params.id),
        },
    })
    res.send(`Blog with id: ${req.params.id} deleted`);
});

Now, we can create user and their blog using your favorite tool like Postman.

// First we create a dummy user
curl --location 'localhost:3000/user' \
--header 'Content-Type: application/json' \
--data '{
    "email": "abc@test.com"
}'

// Then blog from userId returned from previous command. In this case, authorId 1
curl --location 'localhost:3000/blog' \
--header 'Content-Type: application/json' \
--data '{
    "title": "Title",
    "content": "Content",
    "authorId": 1
}'

This finishes up our setup of CRUD app. We started by first creating an empty directory for our blog application and initializing it with npm to setup package.json. We added typescript support by installing few packages and generating tsconfig.json. We added few CRUD apis for Blog and User. We then added Prisma for ORM and created instance for MySQL, connected the two, ran migrations and in the end we got full functioning CRUD app.
This app is far from production like we don't have auth, request body validations, pagination, docker setup for our api and many other things. These things will be covered in later post.

What Would Life Without Git Be Like

Sumit Bhanushali — Wed, 22 Nov 2023 07:41:49 +0000

Nowadays, everyone is familiar with version control system aka VCS, such as Git, Mercurial, or Subversion, Git being the most popular one. It has become second nature to us that the first thing we do after initializing our project is run git init && git add . && git commit -m "initial commit". To understand how important VCS is let's assume we don't know about VCS or something like VCS doesn't even exist.

Let's take 4 scenarios to understand how difficulty escalates exponentially
1) Working Solo on a Project with Single Deployment
2) Working Solo on a Project with Multiple Versions and Deployments
3) Working in a Team of 4 with Single Version and Deployment
4) Working in a Team of 4 with Multiple Versions and Deployments

Here, deployment means version of codebase. Single Deployment means only one version of codebase that will be deployed. Multiple Deployments means multiple versions(including client specific) of codebase in multiple environments(staging, uat, production)

Scenario 1: Working Solo on a Project with Single Deployment

Deployments are a nerve-wracking process. If an issue arises during deployment, rolling back to a working state is a manual and error-prone task. Without a VCS, there's no efficient means of documenting the deployed versions or managing configurations for different environments.

Scenario 2: Working Solo on a Project with Multiple Versions and Deployments

Scenario 3: Working in a Team of 4 with Single Version and Deployment

Scenario 4: Working in a Team of 4 with Multiple Versions and Deployments

It was just a Nightmare

REST API Best Practices: Design & Security

Sumit Bhanushali — Tue, 21 Nov 2023 06:27:58 +0000

REST in Brief

A RESTful (Representational State Transfer) API (Application Programming Interface) is an interface that allows systems to communicate with each other over the internet. REST is often used in the context of web services and APIs to enable the communication and exchange of data between systems.

REST API gives freedom to design API however you want but for building secure, scalable, reliable and maintainable backend system it is important to follow conventions accross backend system. Here are some of those Best Practices which can help you achieve that

Best Practices

Use Resource Names

Resource name is typically name of the model/table in a database, it can also represent some virtual logical entity. Using Resource name makes the API more human readable and understandable and it makes it clear what the API endpoint represents.
Also including resource names in the URL allows for a hierarchical structure that reflects the relationships between resources. For example, /users/123/orders indicates a user with ID 123 and their associated orders.

GET /querycarts/123 -> GET /carts/123

use plurals

Pluralization is a natural part of human language. When dealing with multiple instances of a resource, using the plural form in the URL can make the API more intuitive and easier to understand. It aligns with how we typically describe multiple instances of a noun in English. Using plurals helps distinguish between individual resource instances and collections of resources. For example, /user/123 might represent a single user, while /users represents the entire collection of users.

HTTP methods like GET, POST, PUT, and DELETE are often used with plural resource names. For example, a GET request to /users might retrieve a list of users, and a POST request to /users might create a new user. This aligns with the convention of using plurals for resource names. In many cases, RESTful APIs are built on top of databases, where tables are often named in the plural form to represent collections of entities. Aligning resource names with database table names can simplify the mapping between the API and the underlying data structure.

GET /cart/123 -> GET /carts/123

Idempotency

Idempotence ensures that the same operation can be repeated multiple times without causing different outcomes.

In unreliable networks, requests may fail or be sent multiple times. Idempotence ensures that if a request is duplicated due to network issues or client retries, the system's state remains consistent.
It allows clients to implement a safe retry mechanism. If a request fails, the client can simply retry it without the fear of causing unintended changes or inconsistencies in the system.
In distributed systems where multiple components may process a request, idempotence helps maintain consistency. Even if a request is processed by multiple nodes, the end result should be the same as if it were processed only once.

POST /carts -> POST /carts {requestId: 123}

Use Versioning

API versioning allows developers to introduce changes, improvements, or new features to an API without breaking existing clients. By maintaining backward compatibility, existing applications can continue to function as expected even as the API evolves.
Versioning provides a way for clients to transition from one version of the API to another at their own pace. Clients can choose to adopt a new API version when they are ready, reducing the risk of disruptions to their services.

GET /carts/123 -> GET /v1/carts/123

Query After Soft Deletion

Soft deletion typically involves marking entries as deleted instead of physically removing them from the system
By default, the API should exclude soft-deleted entries from the response. This ensures that clients not explicitly requesting deleted entries receive the expected behavior.
Introduce a query parameter or filter to allow clients to specify whether they want to include soft-deleted entries in the response.

GET /carts -> GET /carts?includeDeleted=true

Pagination

Pagination is a technique used in web development and APIs to break down large sets of data into smaller, more manageable chunks or pages. Loading and displaying a large dataset all at once can lead to slow response times and increased server and network loads. Pagination allows you to retrieve and display smaller subsets of data, improving overall performance and user experience.

GET /carts -> GET /carts?pageSize=xx&pageToken=xx

Sorting

When presenting data to users in a user interface, it's often helpful to allow them to choose how the data is sorted. For example, in an e-commerce application, users might want to sort products by price, name, or popularity.
In scenarios where the API is used for reporting or data analysis, clients may want to retrieve data in a specific order to facilitate easier analysis. For example, sorting by timestamp for time-series data or sorting by a numeric field for ranking purposes.

GET /carts -> GET /carts?sortyBy=time

Filtering

Filters are often used to implement search functionality within an API. Clients can specify search criteria to retrieve only the resources that match the specified conditions. For example, /users?filter=name:John might return a list of users with the name "John."

Clients may need to filter data based on specific attributes or fields. For instance, in an e-commerce API, clients might want to retrieve products within a certain price range, such as /products?filter=price:10-50 to get products with prices between $10 and $50.

Filters enable conditional retrieval of resources. Clients can request resources that satisfy a particular condition, allowing them to fetch data based on dynamic criteria.

GET /carts -> GET /carts?filter=color:red

Secure Access

The X-API-KEY header is often used for API key-based authentication. Each client is issued a unique API key, and the server validates incoming requests by checking the presence and correctness of the API key. This helps control access to the API and ensures that only authorized clients can make requests.

The X-EXPIRY header is used to specify the expiration time of a request. It helps prevent replay attacks by indicating that a request is only valid for a certain period. Servers can check the timestamp in the X-EXPIRY header and reject requests that have expired, enhancing the security of the API.

The X-SIGNATURE header is used to include a cryptographic signature or hash of the request data. This signature is generated using a secret key known only to the server and the client. The server can verify the integrity and authenticity of the request by recalculating the signature on its end and comparing it with the provided X-SIGNATURE. This helps ensure that the request has not been tampered with during transit.

Together, these headers contribute to establishing a secure communication channel between clients and servers, which is crucial when dealing with sensitive or private data.

X-API-KEY=xxx -> X-API-KEY=xxx X-EXPIRY=xxx X-REQUEST_SIGNATURE=xxx(hmac(URL + Query + Expiry + Body))

Nested Resources For Accessing Cross Reference Resource

Nested resources can represent a logical hierarchy, making the structure of the API more intuitive. This is particularly useful when there is a clear parent-child relationship between resources. For example, /orders/123/items signifies that you are retrieving items belonging to the order with ID 123.

The use of nested resources provides context and identification for the relationships between entities. When you see a nested resource like /users/456/posts, it's clear that you are retrieving posts associated with the user with ID 456.

GET /carts/123?item=321 -> GET /carts/123/items/321

POST: Use Request Body Instead Of Query Params

Query parameters are typically appended to the URL and are limited in terms of length. Sending data in the request body allows for larger and more complex data payloads. This is important when dealing with substantial amounts of data or when the data structure is hierarchical.
Placing sensitive information, such as passwords or user credentials, in the request body provides a more secure approach compared to query parameters. Query parameters are visible in the URL, which may expose sensitive information in logs, browser history, or other places.

POST /carts/123?addItem=321 -> POST /carts/123/items:add {itemId: 123}

Rate Limit

Rate limiting helps protect an API against abuse, misuse, or malicious attacks. By restricting the number of requests a client can make, it mitigates the risk of denial-of-service (DoS) attacks, brute-force attacks, and other forms of abuse that could overwhelm the API servers.
It promotes fair usage of API resources by preventing any single client from monopolizing server resources. This ensures that resources are distributed more equitably among all clients,preventing one client from negatively impacting the experience of others.

Conclusion

It's essential to note that these are conventions. The key is to choose a convention and stick to it consistently throughout your API. Whichever approach you choose, clarity, consistency, and adherence to RESTful principles should be the guiding principles in your API design. Additionally, documenting your API well helps developers understand the naming conventions you've chosen. preventing one client from negatively impacting the experience of others.

Microservices: Why You Need To Use npm Workspaces

Sumit Bhanushali — Fri, 06 Oct 2023 17:24:47 +0000

If you are using microservice architecture to organize your project, you most likely will be having common helper functions which are copied across all microservices. Helper functions likes date utils, error handlers, customized loggers, response normalization, etc. This leads to code duplication. Maintaining, testing and versioning such codebase becomes a nightmare if not handled well

This problem is typically solved by either using npm link or by publishing private packages on Package Managers like npm.

1) npm link works by creating symbolic links between packages. This can cause issues with certain project setups or operating systems. Also, these are more cumbersome to setup and maintain

2) Publishing to, and maintaining, a private package repository requires additional overhead – both in terms of time (publishing, updating) and infrastructure (maintaining the repository, ensuring its security, backup, etc.). Handling authentication, permissions, and access for a private npm registry can introduce complexities. Maintaining a private package repository might involve costs – both in terms of infrastructure and potential licensing fees, depending on the solution.

All such issues and more are solved efficiently by npm workspace

npm Workspaces

npm workspace is a feature introduced in npm v7 that allows for managing multiple packages within a single top-level, root package. It enables you to work on interdependent packages as a single unit, making it easier to develop and test them together. This can be particularly useful in monorepo setups or projects with complex dependency structures.

With npm workspace, you can:

Manage Interdependent Packages: npm workspace allows you to work on multiple packages simultaneously, making it easier to develop and test them together. For example, you can make changes to a shared library used by multiple projects and test those changes in the context of each project.
Share Dependencies: npm workspace enables you to share dependencies between packages. This means that if multiple packages in your project require the same dependency, you can install it once at the workspace level and have it shared across all packages. This helps to optimize disk space and simplify dependency management.
Run Commands Across Packages: npm workspace allows you to run commands across all packages in your project. For instance, you can run a build command or a test command that will be executed in each package, ensuring consistency and efficiency in your project's development process.
Manage Versioning: npm workspace provides a unified versioning approach for packages within the workspace. You can easily update versions for all packages at once, ensuring compatibility and consistency across the project.

These are just a few examples of what can be done with npm workspace. It provides a powerful toolset for managing multiple packages within a project, promoting collaboration, modularity, and efficiency.

Code Walkthrough:

Setup Project

mkdir or cd into your project

mkdir test-project && cd test-project && npm init -y

Parent Workspace Configuration (root package.json):
Add workspaces key which will be array of packages. This is the configuration that you'll set up at the root of your workspace to include both of the packages.

{
  "name": "test-project",
  "workspaces": [
    "libraries/*"
  ],
    "scripts": {
    "install": "npm install --ignore-scripts",
    "test": "npm run test:libraries",
    "test:libraries": "npm run test --workspaces"
  }
}

In this example, the "workspaces" field specifies the directory pattern for the packages within the workspace. In this case, it is set to "packages/*", which means that all packages within the packages directory will be part of the workspace.

The "scripts" section includes two example scripts. The "install" script is used to install dependencies across all packages in the workspace, while ignoring any individual package scripts. The "test:packages" script is used to run tests in each package within the workspace. The --workspaces flag is added to the "test" script to ensure that the test command is executed in each package.

The directory structure will look like:

/test-project
|-- /libraries
|   |-- /package-a
|   |-- /package-b
|-- package.json

Package A:

//file: /libraries/package-a/package.json

{
  "name": "package-a",
  "version": "1.0.0",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: run tests from root\" && exit 1"
  }
}

//file: /libraries/package-a/index.js

module.exports = () => {
  console.log('Hello from Package A!');
};

Package B:

//file: /packages/package-b/package.json

{
  "name": "package-b",
  "version": "1.0.0",
  "main": "index.js",
  "dependencies": {
    "package-a": "1.0.0"
  },
  "scripts": {
    "test": "echo \"Error: run tests from root\" && exit 1"
  }
}

//file: /packages/package-b/index.js

const packageA = require('package-a');

module.exports = () => {
  packageA();
  console.log('Hello from Package B!');
};

With the above setup, package-b depends on package-a. When using npm workspaces, you can use local versions of your packages within the workspace without having to publish them to npm.

After creating this structure, you can run npm install in the root directory, and npm will install all the dependencies for all packages, linking them where appropriate.

Remember that the actual code in each package is minimal for illustration purposes, and you can extend them as needed for your projects.

Helpful Commands for Reference:

Installing Workspace Dependencies:

This will install dependencies for all the packages defined in your workspace configuration.
```
npm install
```
Running Scripts Across Workspaces:

If you want to run a specific script defined in a package.json script section of one of your workspaces, you can use the -w or --workspace flag.

For instance, to run the test script for a workspace named package-a:
```
npm run test -w package-a
```
To run the test script for all workspaces:
```
npm run test --workspaces
```
Listing Dependencies Across Workspaces:

You can list dependencies across all workspaces using the ls or list command.
```
npm ls
```

Conclusion:
npm workspaces are designed to make managing monorepos easier. This means that while you can manage dependencies, scripts, and other tasks at the individual workspace level, it's often beneficial to handle shared tasks at the root level to maintain consistency and simplicity.

3 Beginner Level Quirky Leetcode Problems That Made Me Go WOW!!

Sumit Bhanushali — Sun, 03 Sep 2023 06:53:06 +0000

242. Valid Anagram

Demonstrates how we can leverage limited set of inputs and mapping technique to save on both space and time

Solution

We can solve this problem by using a array of size 26(alphabet size) with counts 0. We can traverse the first string s, find exact index in array of each char using (charCode - 97) and store the frequency of each character in a array. Then we can traverse the second string t and decrement the frequency of each character in the array. If the array contains only 0 values at the end, then the second string t is an anagram of the first string s.

/**
 * @param {string} s
 * @param {string} t
 * @return {boolean}
 */
var isAnagram = function(s, t) {
    if (s.length !== t.length) {
        // If the lengths are different, then t cannot be an anagram of s
        return false;
    }

    // Initialize a array with 0 frequencies for each character
    const freqTable = Array(26).fill(0);

    // Traverse the first string s and update the frequency table
    for (let i = 0; i < s.length; i++) {
        const charCode = s.charCodeAt(i) - 97;
        freqTable[charCode]++;
    }

    // Traverse the second string t and decrement the frequency table
    for (let i = 0; i < t.length; i++) {
        const charCode = t.charCodeAt(i) - 97;
        freqTable[charCode]--;

        if (freqTable[charCode] < 0) {
            // If the frequency goes below 0, then t cannot be an anagram of s
            return false;
        }
    }

    // If all frequencies are 0, then t is an anagram of s
    return true;
};

console.log(isAnagram("anagram", "nagaram"));   // Output: true
console.log(isAnagram("rat", "car"));   // Output: false

Time Complexity: O(n) where n is the length of the strings s and t. We traverse each string only once.
Space Complexity: O(1) since the array has a fixed size of 26 characters (assuming only lowercase letters are used in the strings).

1. Two Sum

This is a beginner problem which helps demonstrate how we can use hashmap and a little Math sense to take our time complexity from O(n^2) to O(n)

Solution:

We iterate over the array of integers and for each number, we check if its complement (target - number) already exists in the hash map. If it does, we have found a pair of numbers that add up to the target. If it doesn't, we can add the current number to the hash map and continue iterating.

function twoSum(nums, target) {
  const map = new Map();
  for (let i = 0; i < nums.length; i++) {
    const complement = target - nums[i];
    if (map.has(complement)) {
      return [map.get(complement), i];
    }
    map.set(nums[i], i);
  }
  return [];
}

console.log(twoSum([3,2,4], 6)) //outputs [1, 2]

Time Complexity:

The time complexity of the above algorithm is O(n), where n is the length of the array of integers nums. This is because we traverse the array only once.

Space Complexity:

The space complexity of the above algorithm is O(n), because we store at most n key-value pairs in the hash map.

205. Isomorphic Strings

This is a beginner problem demonstrating how intelligent mapping technique can be used to solve complex problems

Solution:

To solve this problem, we need to find a way to map each character in s to a character in t. We can do this by using two hash maps - one to map characters from s to t, and another to map characters from t to s.

We iterate over each character in s and t, and for each character, we check if it has been mapped before. If it has been mapped before, we check if the mapping is the same in both s and t. If it is not, we return false. If the character has not been mapped before, we add it to the hash map.

We return true if all characters have been checked and the condition is satisfied.

const isIsomorphic = function(s, t) {
    if (s.length !== t.length) {
        return false;
    }

    const mapST = {};
    const mapTS = {};

    for (let i = 0; i < s.length; i++) {
        const charS = s[i];
        const charT = t[i];

        if (mapST[charS] !== undefined && mapST[charS] !== charT) {
            return false;
        }

        if (mapTS[charT] !== undefined && mapTS[charT] !== charS) {
            return false;
        }

        mapST[charS] = charT;
        mapTS[charT] = charS;
    }

    return true;
};

Time Complexity:

The time complexity of this solution is O(n), where n is the length of the input strings s and t. We iterate over each character in s and t only once.

Space Complexity:

The space complexity of this solution is also O(n), where n is the length of the input strings s and t. We create two hash maps to store the mappings between characters from s to t and from t to s. The maximum size of these hash maps is n.

What makes PostgreSQL king of databases

Sumit Bhanushali — Fri, 28 Jul 2023 06:27:43 +0000

According to the StackOverflow survey, PostgreSQL is the most loved database in 2022. What makes it so special? I mean it is just another SQL database, right? let’s find out

Well first, let's clear out that PostgreSQL is ACID-compliant which is the primary requirement of relational databases and it supports foreign keys, stored procedures, joins, and views too just like MySQL

When we think of relational databases, MySQL is the first thing that comes to mind. When we google MySQL vs PostgreSQL, it spits out “MySQL is a purely relational database, whereas PostgreSQL is an object-relational database.”

Object Relational Database?
Objects, Classes and Inheritance are directly supported in database schemas and in the query language. In addition, just as with pure relational systems, it supports extension of the data model with custom data types and methods. This is what makes PostgreSQL so useful, it helps in solving several use cases that would have otherwise required us to learn new database systems like

OLTP (Online Transaction Processing):
Traditional CRUD (Create-Read-Update-Delete) operations.

OLAP (Online Analytical Processing):
OLAP is used by data engineers to analyze and get insights into data.
Since PostgreSQL is based on HTAP (Hybrid transactional/analytical processing) architecture, it can handle OLAP pretty well.

FDW (Foreign Data Wrapper):
We can create or use FDW to connect to different databases like Redis, MySQL, Neo4j and perform data flow operations.

Streaming:
PipelineDB extended from PostgreSQL is a time series database that can be used for real time reporting and analytics applications.

Geospatial:
PostGIS is a spatial database extender for PostgreSQL. It adds support for geographic objects, allowing location queries to be run in SQL.

Time Series:
Timescale extended from PostgreSQL for time series and analytics. It is used to combine relentless streams of financial and tick data with other business data to build new apps and uncover unique insights.

Distributed Tables:
Distributed Tables are tables whose rows are spanned across multiple tables to speed up read and write operations. This is done using a consistent hashing technique. CitusData extended from PostgreSQL gives this feature out of the box

Scaling your Backend: Database (Master - Slave Replication)

Sumit Bhanushali — Mon, 24 Jul 2023 06:28:46 +0000

When you notice that your server is taking more time to respond to the client, your first step should be to find out any wrongly written slow performing code on the API server. If all checks out well, next you look at any slow performing database queries. Generally, queries can be optimized by reducing unnecessary joins or by adding indexes where required (too many indexes can slow down write operations).

After performing these checks and fixing these if you still want to improve speed then the next step would be to upgrade your database server because in a typical web app, databases become the bottleneck, so databases should be the first which should be scaled vertically.

After you hit the vertical limit of the server, your next step would be to scale horizontally. Databases are usually read heavy, having a read to write ratio of approximately 90:10. The easiest and best way to scale such a database would be to adopt master slave replication architecture. In this setting, all writes are applied to the master database and all reads go to slave databases. We can have any number of slave replicas as we need. Writes happening at master are synchronized to slave databases in a few milliseconds. In a case when the Master database goes down, one of the slave database is promoted to master using the leader election algorithm.

There are 3 types of Replication Strategies, you can choose one of them depending on your use case:

Synchronous Replication:

In Synchronous Replication, once the Master node updates its copy of the data, it initiates the write operation on its replicas. Once the Replicas receive the update, they apply the change to their copy of data and then send the confirmation to the Master. Once the Master receives the confirmation from all the Replicas, it responds to the client and completes the operation.

Asynchronous Replication:

In Asynchronous Replication, once the Master node updates its copy of the data, it immediately completes the operation by responding to the Client. It does not wait for the changes to be propagated to the Replicas, thus minimizing the block for the Client and maximizing the throughput.

Semi-synchronous Replication:

In Semi-synchronous Replication, which sits right between the Synchronous and Asynchronous Replication strategies, once the Master node updates its copy of the data, it synchronously replicates the data to a subset of Replicas and asynchronously to others.

Never forget ACID again!!

Sumit Bhanushali — Sun, 23 Jul 2023 08:55:20 +0000

We often forget how ACID properties function individually since these are the core of Relational Databases it is important we know their importance to ensure clear communication between team members and to use those properties efficiently

Atomicity: All or nothing. This property makes sure that in a transaction either all changes are committed or if something fails all changes are reverted.

A great example of seeing why it is critical to have atomicity is Money Transfers.

Imagine transferring money from bank account A to B. The transaction involves subtracting the balance from A and adding the balance to B. If any of these changes are partially applied to the database, it will lead to money either not being debited or credited, depending on when it failed.

Consistency: This property makes sure that every data in the database is correct. Any attempt to add incorrect data will cause failure.

What does inconsistent data look like? The balance of an account can be updated as negative which is not possible in the real-world Leftover of orphaned data such as comments on a blog post. If a blog post is deleted, its comments must also be deleted from the database

Correctness is ensured by defining rules such as Foreign Key constraints, Check constraints, On Delete Cascades, On Update Cascades, etc.

Isolation: This property ensures that no more than one transaction can have write(exclusive lock) or read(shared lock) access to a particular row to prevent incorrectly updating data and phantom reads.

Take a case when two or more transfers happen on the same account simultaneously, without isolation there will be a high probability that the wrong amount is updated in the database and there will be a mismatch between the amount debited and the amount credited

Durability: This property ensures that once the transactions commit, the changes survive any outages, crashes, and failures, which means any writes that have gone through as part of the successful transaction should never abruptly vanish.

A typical example of this is your purchase order placed on Amazon, which should continue to exist and remain unaffected even after their database faced an outage. So, to ensure something outlives a crash, it has to be stored in non-volatile storage like a Disk; and this forms the core idea of durability.

Now that you know what ACID properties are and how to use them effectively, Do you ever wonder how they are implemented under the hood?

Atomicity:

Most databases implement Atomicity using logging. When a transaction starts, the engine starts logging all changes in a file. When the engine receives COMMIT it then applies those changes permanently. If it receives ROLLBACK or some error happens, that log file is discarded and thus no changes get applied.

Alternatively, Atomicity can also be implemented by keeping a copy of the data before starting the transaction and using it during rollbacks.

Consistency:

Integrity constraints like primary key, foreign key and datatype checks are checked at runtime when the changes are being applied to the data.

Cascade operations i.e operations to be performed when data containing foreign keys are modified are performed synchronously along with the running transaction. Most database engines also provide a way to make them asynchronous, allowing us to keep our transactions leaner.

Isolation:

A transaction before altering any row takes a lock (shared or exclusive) on that row, disallowing any other transaction to act on it. The other transactions might have to wait until the first one either commits or rollbacks.

The granularity and the scope of locking depend on the isolation level configured. Every database engine supports multiple Isolation levels, which determines how stringent the locking is. The 4 isolation levels are

Serializable:

The Serializable isolation level provides the strictest transaction isolation. This level emulates serial transaction execution for all committed transactions; as if transactions had been executed one after another, serially, rather than concurrently.

Repeatable reads:

The Repeatable Read isolation level only sees data committed before the transaction began; it never sees either uncommitted data or changes committed during transaction execution by concurrent transactions.

Read committed:

Read Committed is the default isolation level in PostgreSQL. When a transaction uses this isolation level, a SELECT query (without a FOR UPDATE/SHARE clause) sees only data committed before the query began; it never sees either uncommitted data or changes committed during query execution by concurrent transactions.

Read uncommitted:

The uncommitted read isolation level allows an application to access the uncommitted changes of other transactions. Moreover, UR does not prevent another application from accessing a row that is being read, unless that application is attempting to alter or drop the table.

Durability:

The most fundamental way to achieve durability is by using a fast transactional log. The changes to be made to the actual data are first flushed on a separate transactional log and then the actual update is made.

This flushed transactional log enables us to reprocess and replay the transaction during database reboot and reconstruct the system's state to the one that it was in right before the failure occurred - typically the last consistent state of the database. The write to a transaction log is made fast by keeping the file append-only and thus minimizing the disk seeks.