Forem: Dantis Mai

Postgres organizes Indexes

Dantis Mai — Sat, 03 Sep 2022 17:16:13 +0000

Index

what is Index?

The index is a datatype, separated from our data from the table. It likes a map of locations of data that points to a value located in which page on heap AND row id (for Postgres, it is tupleId). Heap is a table of our data with its page after another.

(index value) => (Page, rowId)
(index value) => (Page, rowId)

Page

At first, I thought when we make a query to get a single row, it just fetches that single row for us. Actually, it will fetch the whole page that contained our row from the heap, then collect and return the row we want. For each page, we call it an I/O. So the performance of our query depends on how many pages the DB fetches. In case each row of data contains a huge string or a lot of columns, then a page will include fewer rows, and then we will need more IO, so the performance of our query will decrease.

Clustered Index

Sometimes, we may hear about Clustered index. When we define a column as a clustered index, the table now will organize around that key. In case a new row is added to the table, the row will be added to the location on the heap that follows the index order. For example, we have a user table below with the id column is uuid, and priority_level is a clustered index. Although, we all see the rows ordered Dantis > Vu > Ngan > John, the actual order in the heap is Dantis > Ngan > Vu > John.

id	name	priority_level
awes	Dantis	10
zwqe	Vu	5
tydb	Ngan	9
mnbv	John	4

The Clustered index is really beneficial when we perform a range query, like getting all users that are older than 90 with column age is a clustered index. The DB just need to find the start page and the end page, then fetch all of them.

However, in Postgres, we just have an eventually clustered database. Because even if we create a clustered index for our table, then a new row comes, Postgres does not guarantee that the row will be followed our order.

Organizing Indexes (Postgres vs Mysql)

For Postgres, we still can do that, but actually, those indexes are all secondary indexes. Different from other RDBMS, all indexes will point directly to the heap. So, you may ask where is the primary index, Postgres uses that for tupleId (it likes rowId) to manage the data.

For MySQL, we can create both primary index and secondary index, then the primary index will point to the heap, and the secondary index will point to the primary one.

Because of that design, Postgres and Mysql have their own behaviors when querying, inserting, and updating (or deleting) a row. Basically, they are around the idea that the index needs to be aware of any change related to it.

Querying

Because Postgres point the index directly to the heap, the DB just needs to use an index scan or another heap scan to get our targeted row. Meanwhile, Mysql will perform 2 index scans on the secondary index and the primary index. So, Postgres can process just a little bit faster than Mysql.

Inserting

For inserting, Postgres and Mysql do the same thing that they need to update all the existing indexes to make those indexes aware of the new row coming. For example, if our table has 10 indexes, then the table needs to perform 10 updates. That is the problem when we have too many indexes in a table.

Updating (Deleting) This is the most interesting part

When updating a row, instead of changing the data, Postgres create a new row from the original data with the changes and a new tupleId, then it points all the indexes to that new row. You may ask about the old tupleId (dead tuple). Postgres uses that to revert a transaction, but then if we forget to clean up those dead tuples, they will take a large resource for nothing. That is why we need the 'vacuum', not to free our memory but to reuse it.

For Mysql, the thing is quite more simple. it just needs to update the primary and the related index if necessary.

Datatype

The problem of the index datatype mostly comes from the randomness of the data. In case our index is Sequential data, when we create or update a row, it is easy for the DB to locate the next location on the heap for the row. In addition, B-tree ,which is the most popular index datatype by default, is also need that sequential. Because it takes a lot of step to re-balance the structure.

Paypal.

I am really happy to receive your feedback on this article. Thanks for your precious time reading this.

Enhance Logger Using Inversify Context, Decorators, and Reflect-metadata

Dantis Mai — Sat, 19 Feb 2022 11:15:08 +0000

Generally, Log is a footprint of a system, it tells us about the state of the service at a moment. With it, in case there is an issue on the server, we can trace the root cause.

Nowadays, the logger is an important feature for any service, either small or big. However, we shouldn't write too much or too few records, that will lead to overwhelming or not enough information. Besides, those records need to follow the same format within a codebase.

In this post, we will discuss how to enhance them by using inversify context and Decorators. The logger package used is winston, and the below code I used to create log group

export const logger = LoggerWrapper();

export const createChildLogger = (messagePrefix: string): Logger =>
  Object.create(logger, {
    write: {
      value(info) {
        info.message = `[${messagePrefix}] ${info.message}`;
        logger.write(info);
      },
    },
  });

Inversify and Its Context

A powerful and lightweight inversion of control (IoC) container
for JavaScript & Node.js apps powered by TypeScript

We already talk about how to break the dependencies by IoC in this post, the Inversify are made in order to help us follow IoC rule.

We can create a Logger component bind to our container, and use toDynamicValue API to create a logger instance for each other components. The input argument for the API is context, it represents the called component and all its dependencies.

const getClassNameFromRequest = (context: interfaces.Context) =>
  (context.currentRequest.parentRequest &&
    context.currentRequest.parentRequest.bindings.length &&
    context.currentRequest.parentRequest.bindings[0].implementationType &&
    (context.currentRequest.parentRequest.bindings[0].implementationType as any)
      .name) ||
  'Undefined';

container
    .bind<Logger>(types.Logger)
    .toDynamicValue((context: interfaces.Context) => {
      const namedMetadata = getClassNameFromRequest(context);
      return createChildLogger(namedMetadata);
    });

When we are here, let say we have a below service to get service info. Then whenever we call the request, a logger instance will be created for ServiceInfoServiceImpl, and the className will be added to the logger message.

Decorators

Decorators are a stage 2 proposal for JavaScript and are >available as an experimental feature of TypeScript.

Decorator is farmiliar concept for Java, C++, and so on. But it is just a experiment feature of Typescript. According to TS documentation, we must to endable experimentalDecorators compiler option to use Decorator.

The goal of using decorator is the same with Inversify, which is creating and customizing logger property for each class. Because the logger is a property of a class so we need to use class decorator.

export function classDecorator<T extends { new (...args: any[]): {} }>(constructor: T) {
  return class extends constructor {
    logger = createChildLogger(constructor.name);
  };
}

The class decorator will overide the class constructor, so we need to define a logger property, and be careful with other constructor params.

Dive Deeper

what if we also need called method?

It will look like [className] [methodName] message ...

I already tried method and property decorators, but it didn't work well. The reason for this is that the decorators just run at the runtime, while the logger instance is still not created yet.

After a lot of googling, I found this post. That is a wonderful idea, I take that then modify logger property, so I have what I need.

However, because the solution change the original definition of the called method, I wonder about that, although I haven't faced any issue while using it.

Ok, so how does it work??! The first thing we do is define const originalMethod = descriptor.value;, the originalMethod now belongs to the closure of our brand new descriptor.value. Yes, it will work as its definition

export const createChildLogger = (messagePrefix: string,subPrefix?: string): Logger =>
  Object.create(logger, {
    write: {
      value(info) {
        info.message = `[${messagePrefix}]${subPrefix ? ` [${subPrefix}]` : ''} ${info.message}`;
        logger.write(info);
      },
    },
  });

export function logGroup() {
  return (target: any) => {
    for (const propertyName of Object.getOwnPropertyNames(target.prototype)) {
      const descriptor = Object.getOwnPropertyDescriptor(
        target.prototype,
        propertyName,
      );
      if (!descriptor) {
        continue;
      }

      const originalMethod = descriptor.value;
      const isMethod = originalMethod instanceof Function;
      if (!isMethod) {
        continue;
      }

      descriptor.value = function (...args: any[]) {
        (this as any).logger = createChildLogger(target.name, propertyName);
        return originalMethod.apply(this, args);
      };

      Object.defineProperty(target.prototype, propertyName, descriptor);
    }
  };
}

Note: this class decorator doesn't work on the controller layer of Inversify

So, What if we need a prefix for a function outside of a class?

The solution below just work on service that process one request at a time
Because the decorators just can work with class, whenever a method from a class calls a util function, the function will lose the context.

When we install inversify, we always need a package called reflect-metadata. Generally, it is like a global variable, but it just is applied to a specific target scope. you can take a look at this repository.

The general idea is that we will store the caller className, methodName to the metadata of the logger property, then whenever we call an outside function, we structure the logger message using the context from metadata.

First of all, we need a customLogger as below and use it as the default logger. Then, we modify a few lines in our class decorator logGroup, and container config. When an outside function calls logger.debug or something, the new logger will check for the context then structure the message.

Inversify context

As mentioned above, the Inversify Context show us the connection between the called component with its dependencies. we can use the code below to get it.

export const getContextName = (node : interfaces.Request | null):string => 
  (node?.bindings[0].implementationType as any).name

export const getContextTree = (context:interfaces.Context): string[] => {
  let node = context.currentRequest.parentRequest
  const listNode = [];

  while(node){
    listNode.unshift(getContextName(node));
    node = node.parentRequest;
  }

  return [... new Set(listNode)];
}

Then call it in api onACtivation or in creating logger instance when initilize container

  container
    .bind<Logger>(types.Logger)
    .toDynamicValue((context: interfaces.Context) => {
      const namedMetadata = getClassNameFromRequest(context);
      const listNode = getContextTree(context);
      console.log(listNode);
      return createChildLogger(namedMetadata);
    });

  container
    .bind<ServiceInfoService>(types.Service.SERVICE_INFO)
    .to(ServiceInfoServiceImpl)
    .onActivation((context: interfaces.Context, service: ServiceInfoService) => {
      const listNode = getContextTree(context);
      console.log(listNode)
      return service
    });

To be honest, I still don't have any idea for that listTree, just collect them then be proud of myself 🤣.

Play Around

We can use this npm package to init inversify project with build in the logger above, just install it then run the command below

typescript-maker init -ioc test-repo

Paypal.

I am really happy to receive your feedback on this article. Hope you have fun with it. Thanks for your precious time reading this.

Create New Project Using NPM package

Dantis Mai — Mon, 03 Jan 2022 16:37:16 +0000

In the last article, we can create a new project by using Express Typescript Boilerplate template, it saves us pretty much time to do it from scratch. However, it is still not the best, we can optimize it using NPM and just 1 command line, we get everything.

Setup account

we will need an account in npm
Login our account on our pc
After that, we update our profile by access profile > edit profile. There are 2 things we need to pay attention GitHub Username and Email, they will cause some problems when publishing a package. For example, after running command npm publish, it returns 403 Forbidden - You do not have permission to publish. Are you logged in as the correct user?. In case that we can fix it by changing email to whatever, then revert it back to our main email.

Setup project

If we config npm account successfully, just running npm publish, then we will see the log below

Then it will be shown on our npm packages, and the other developer can reach to the package also

In order to make our package more reliable, we should enable security. If there is any issue Github will show us as below.

Otherwise, it will be green.

Config command

In this post, we will use our last template as source code for the npm package. And below is my package.json.

{
  "name": "typescript-maker",
  "version": "1.1.1",
  "description": "Minimalistic boilerplate to quick-start Node.js development in TypeScript.",
  "engines": {
    "node": ">=14 <15"
  },
  "bin": {
    "typescript-maker": "./bin/generateApp.js"
  },
  "scripts": {
    "start": "node src/index",
    "dev": "nodemon --config restart.json",
    "clean": "rm -rf coverage build tmp",
    "prebuild": "npm run lint",
    "build": "tsc -p tsconfig.build.json",
    "build:watch": "tsc -w -p tsconfig.build.json",
    "lint": "eslint . --ext .ts,.tsx",
    "test": "jest"
  },
  "author": "Dantis Mai",
  "license": "Apache-2.0",
  "dependencies": {
    "commander": "^8.3.0",
    "express": "^4.17.1",
    "module-alias": "^2.2.2",
    "tslib": "~2.3.0",
    "winston": "^3.3.3"
  },
  "devDependencies": {
    "@tsconfig/recommended": "^1.0.1",
    "@types/express": "^4.17.13",
    "@types/jest": "^26.0.24",
    "@types/node": "~14.14.45",
    "@typescript-eslint/eslint-plugin": "~4.28.2",
    "@typescript-eslint/parser": "~4.28.2",
    "dotenv": "^10.0.0",
    "eslint": "~7.30.0",
    "eslint-config-prettier": "~8.3.0",
    "eslint-plugin-jest": "~24.3.6",
    "jest": "^27.0.6",
    "jest-html-reporter": "^3.4.1",
    "nodemon": "^2.0.12",
    "prettier": "~2.3.2",
    "rimraf": "^3.0.2",
    "supertest": "^6.1.5",
    "ts-jest": "^27.0.3",
    "ts-node": "^10.2.0",
    "ts-node-dev": "^1.1.8",
    "tsconfig-paths": "^3.10.1",
    "tsutils": "~3.21.0",
    "typescript": "~4.3.5"
  }
}

In the package.json file, there are 3 fields, we need to update:

name: npm package name. This name will be our npm package name, ignoring the GitHub repository name. For example, my repository name is express-typescript-boilerplate, while the package name is typescript-maker.
version: npm package version. By versioning, we can update the new features with backward compatibility.
bin: command configuration. We will direct the source code for the command. As you can see the bin field in my package.json, I define "typescript-maker": "./bin/generateApp.js", it means that typescript-maker is the command name, and the options and arguments are described in ./bin/generateApp.js.

Now, let jump to config our command.
For a sample command, there are 4 steps:

Verify arguments: verify the number of arguments to make sure, we have enough information.
Parse arguments and options: get argument value from input
Validate existing folder: prevent issues when creating a new folder or file. It will work like we clone a repository 2 times at the same directory.
Define workflow: define what thing we gonna do when we call the command.
Clear unused files: keep the result clean to not distract the user after running the command.
Trigger workflow.

Combine everything and we have a sample config for typescript-maker below

# Verify arguments
if (process.argv.length < 3) {
  console.log('You have to provide a name to your app.');
  console.log('For example :');
  console.log('    typescript-maker my-app');
  process.exit(1);
}

# Parse arguments and option
const projectName = process.argv[2];
const currentPath = process.cwd();
const projectPath = path.join(currentPath, projectName);
const git_repo = 'https://github.com/Maithanhdanh/express-typescript-boilerplate.git';

# Validate existing folder
try {
  fs.mkdirSync(appPath);
} catch (err) {
  if (err.code === 'EEXIST') {
    console.log('Directory already exists. Please choose another name for the project.');
  } else {
    console.log(error);
  }
  process.exit(1);
}

# define steps in workflow
async function main() {
  try {
    console.log('Downloading files...');
    execSync(`git clone --depth 1 ${git_repo} ${projectPath}`);


    // Change directory
    process.chdir(appPath);

    // Install dependencies
    const useYarn = await hasYarn();
    console.log('Installing dependencies...');
    if (useYarn) {
      await runCmd('yarn install');
    } else {
      await runCmd('npm install');
    }
    console.log('Dependencies installed successfully.');
    console.log();

    # Clean unused files
    console.log('Removing useless files');
    execSync('npx rimraf ./.git');
    fs.rmdirSync(path.join(projectPath, 'bin'), { recursive: true});

    console.log('The installation is done, this is ready to use !');

  } catch (error) {
    console.log(error);
  }
}

# trigger workflow
main();

In case we want a more complicated command, we can use module commander, which supports us pretty much thing when we design the architecture of the command. After using the commander, I structure my command like this.

This is mine, you can enjoy it.

Paypal.

I am really happy to receive your feedback on this article. Thanks for your precious time reading this.

Integrate Database Migration Using Flyway Into CI/CD

Dantis Mai — Mon, 11 Oct 2021 18:30:09 +0000

Practically, Database Migration is changing the structure or data of a database, which includes some activities on the database, like create a table, update or insert data to a table.

Without DB migration, if it is not establihsing new datbase, we need to consider whether the script has been run on the target DB or not. In case we accidentally re-run the script, it will causes duplicate records on the table for inserting data. That is one of reasons for manual deployment.

With the development of CI/CD, there are a lot of great DB migration tools developed. In this post, I will use Flyway for CircleCI pipeline to deploy on Heroku, you can find their configurations in my last article.

Ideas

We may use different platforms for DB migration, but overall they are just around the idea of generating the connection configuration from the connection string. In this session, I will realize that idea by using flyway on Heroku PostgreSQL

Get the DB connection string using platform CLI: For Heroku, it is heroku config:get DATABASE_URL -a <APP_NAME>, then the console will print out postgres://DB_USER:DB_PASSWORD@DB_HOST:5432/DB_NAME.
Extract DB user, password, host, name from connection string: We may any technique to get that info out, in this article, I used sed, cut.
Generate a connection config for DB migration tool: Depending on the DB migration tool, the config formats may be different. In this case, I created a config file like below. You may wonder about the locations, and table, they are SQL scripts directory for the former, and table for version control for the later.

flyway.url=jdbc:postgresql://DB_HOST/DB_NAME
flyway.locations=SQL_FILE_LOCATION
flyway.user=DB_USER
flyway.password=DB_PASSWORD
flyway.table=schema_history
flyway.outOfOrder=true

Call DB migration tool CLI: If we use flyway, the command will be flyway -configFiles=<CONFIG_FILE_DIRECTORY> migrate
Combine all things into a bash file

# Extract data from DB connection
export CONNECTION_URL="$(heroku config:get DATABASE_URL -a ${HEROKU_APP_NAME})"
export SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE}" )" && pwd)"
export SQL_FILE_LOCATION="filesystem:/${SCRIPT_DIR}/migrations/scripts/sql"
export CONFIG_DIR="${SCRIPT_DIR}/migrations/config"
export DB_NAME="$(echo $CONNECTION_URL | sed "s|.*:5432/||")"
export DB_USER="$(echo $CONNECTION_URL | cut -d ":" -f 2 | sed "s|.*//||")"
export DB_PASSWORD="$(echo $CONNECTION_URL | cut -d ":" -f 3 | cut -d "@" -f 1)"
export DB_HOST="$(echo $CONNECTION_URL | cut -d ":" -f 3 | cut -d "@" -f 2)"

# Generate config file
sed -i -e "s|DB_HOST|${DB_HOST}|g" ${CONFIG_DIR}/flyway.conf
sed -i -e "s|DB_NAME|${DB_NAME}|g" ${CONFIG_DIR}/flyway.conf
sed -i -e "s|DB_USER|${DB_USER}|g" ${CONFIG_DIR}/flyway.conf
sed -i -e "s|DB_PASSWORD|${DB_PASSWORD}|g" ${CONFIG_DIR}/flyway.conf
sed -i -e "s|SQL_FILE_LOCATION|${SQL_FILE_LOCATION}|g" ${CONFIG_DIR}/flyway.conf

# Run flyway
flyway -configFiles=${CONFIG_DIR}/flyway.conf migrate

Note: This file should be run by the pipeline, but not manually. Because the information in the config file will change and lead to wrong information for the next run.

Integrate DB migration to CI/CD

Let re-use our CI/CD pipeline from the last post.
CirlceCI supports post-steps, which is the last part of the job. If post-step is failed, the whole job will be marked as failed.
Because CircleCI doesn't support flyway, we need to install it independently by command line. After installing flyway, we are good to call our bash file above to start the migration process.

The pipeline config now will look like this.

orbs:
  node: circleci/node@4.1.0
  heroku: circleci/heroku@1.2.6

version: 2.1

commands:
  deploy-command:
    parameters:
      BRANCH:
        type: string
      HEROKU_APP_NAME:
        type: string
    steps:
      - run: heroku config:set YARN_PRODUCTION=false -a <<parameters.HEROKU_APP_NAME>>
      - heroku/deploy-via-git:
          app-name: <<parameters.HEROKU_APP_NAME>>
          branch: <<parameters.BRANCH>>

jobs:
  test:
    executor: node/default
    steps:
      - checkout
      - node/install-packages:
          cache-path: ~/project/node_modules
          override-ci-command: npm install
      - run: npm test

  deploy:
    executor: heroku/default
    steps:
      - checkout
      - heroku/install

      - when:
          condition:
            equal: [master, << pipeline.git.branch >>]
          steps:
            - deploy-command:
                HEROKU_APP_NAME: production-server
                BRANCH: master

      - when:
          condition:
            equal: [staging, << pipeline.git.branch >>]
          steps:
            - deploy-command:
                HEROKU_APP_NAME: staging-server
                BRANCH: staging

      - when:
          condition:
            equal: [develop, << pipeline.git.branch >>]
          steps:
            - deploy-command:
                HEROKU_APP_NAME: develop-server
                BRANCH: develop

workflows:
  heroku_deploy:
    jobs:
      - test

      - deploy:
          requires:
            - test
          filters:
            branches:
              only:
                - master
                - develop
                - staging
          post-steps:
            - run: 
                name: install flyway
                command: wget -qO- https://repo1.maven.org/maven2/org/flywaydb/flyway-commandline/7.14.0/flyway-commandline-7.14.0-linux-x64.tar.gz | tar xvz && sudo ln -s `pwd`/flyway-7.14.0/flyway /usr/local/bin
            - run: 
                name: run migration scripts
                command: sh migrations/scripts/migrations-run.sh
                environment:
                  GIT_BRANCH: << pipeline.git.branch >>

In this new pipeline, you can see that I have the environment part, which is used to declare environment variables. We have totally 3 environments (develop, staging, production), so, with that variables, we can change APP_NAME depending on GIT_BRANCH. By adding the below condition at the very first of the bash script, the script will get the appropriate DB info for each environment.

if [ "$GIT_BRANCH" = "master" ]; then
  export HEROKU_APP_NAME=production-server
elif [ "$GIT_BRANCH" = "staging" ]; then
  export HEROKU_APP_NAME=staging-server
else
  export HEROKU_APP_NAME=develop-server
fi

# Extract data from DB connection ...

Paypal.

I am really happy to receive your feedback on this article. Thanks for your precious time reading this.

CI/CD Pipeline with CircleCI and Heroku

Dantis Mai — Sat, 02 Oct 2021 18:13:42 +0000

Introduction to CircleCI

CircleCI is one of the greatest CI/CD platforms that helps a single developer or small team can quickly set up. CircleCI can be really flexible from a simple pipeline with build, test, and deploy to a complex one with caching, docker layer, and so on.

After authenticating CircleCI with Github and register our repository, every integration will trigger the jobs. For a Free plan, CircleCI provides us up to 2500 credits per week, I can try a lot of fun with that much.

Introduction to Heroku

Heroku is a cloud service that allows developers to build and manage web services without getting headaches about configuration servers. We just need to focus on developing our product and let the other things for Heroku. Unlike other cloud services, Heroku computes bills by dynos, which is a kind of a virtual machine that can auto-scale up or down depending on the size of our product.

The biggest advantage of Heroku is Free within 550h per month, but still provides numerous addons like Database, SSL, and so on. Besides, it also supports Nodejs, Python, Java, Go, .... Heroku will be a great choice to test an idea, or hosting a small website, for me, I host my portfolio there.

Make it work

Authenticate CircleCI with GitHub

First of all, we need to have a pipeline config in our repository, otherwise, CircleCI won't allow us to go further. This documenttation will help to config our own pipeline. Once finishing the pipeline in the directory .circleci/config.yml and pushing it to GitHub, we are ready to go. I will show you my pipeline below.
After sign up CircleCI by Github account, we can see all repositories listed out in the middle panel. By clicking set up project, CircleCI will scan the pipeline config in the repository, because we have already done the first step, it will let us go, otherwise, the guard will ask us to create one. set up project will be toggled to unfollow project, if the scanning is successful, and CircleCI will lead us to a dashboard of the project.

Now, we can try to commit a small change. In case the pipeline is triggered, congratulation!

Note: CircleCI editor supports pretty well for creating, as well as debugging pipelines. I always use it, and I hope that it also can help you.

Authenticate Heroku with GitHub, and CircleCI

From the top right corner, we can access account setting by clicking our avatars, then come to Applications > Third-party Services. This step just helps Heroku can interact with GitHub, but not each repository.

Every time we create a new app, we need to choose Deployment method, this way will help Heroku can use GitHub as a Version Control. There are 2 more steps needed to be done:
- App connected to GitHub: reach to the repository that we want to deploy
- Automatic deploys: For every integration to the chosen branch, the new code will be deployed. In case we have already had CI for the app, we tick the option Wait for CI to pass before deploy. And don't forget to Enable Automatic Deploys.
After a lot of configuration, let click on Deploy Branch in the Manual deploy session. To validate it is successful or not, we can check the status of the deployment appeared here. If successful, the Environments session of the repository on GitHub will list out the Heroku app name.
To let CircleCI pipeline can deploy to Heroku, we need to 2 things, they are:
- API Key: This key is really important and we'd better let no one know it. With the key, authenticated CircleCI pipelines can process deployment to Heroku. In Project Settings, We can set it by adding a new Environment Variables with key name HEROKU_API_KEY, and the value will be placed in Account Settings > Account > API Key.
- App name: This key is the target app name needed to be deployed.

Config pipeline

Below is pipeline for CircleCI and NodeJS platform to deploy to Heroku.

orbs:
  node: circleci/node@4.1.0
  heroku: circleci/heroku@1.2.6

version: 2.1

commands:
  deploy-command:
    parameters:
      BRANCH:
        type: string
      HEROKU_APP_NAME:
        type: string
    steps:
      - run: heroku config:set YARN_PRODUCTION=false -a <<parameters.HEROKU_APP_NAME>>
      - heroku/deploy-via-git:
          app-name: <<parameters.HEROKU_APP_NAME>>
          branch: <<parameters.BRANCH>>

jobs:
  test:
    executor: node/default
    steps:
      - checkout
      - node/install-packages:
          cache-path: ~/project/node_modules
          override-ci-command: npm install
      - run: npm test

  deploy:
    executor: heroku/default
    steps:
      - checkout
      - heroku/install

      - when:
          condition:
            equal: [master, << pipeline.git.branch >>]
          steps:
            - deploy-command:
                HEROKU_APP_NAME: production-server
                BRANCH: master

      - when:
          condition:
            equal: [staging, << pipeline.git.branch >>]
          steps:
            - deploy-command:
                HEROKU_APP_NAME: staging-server
                BRANCH: staging

      - when:
          condition:
            equal: [develop, << pipeline.git.branch >>]
          steps:
            - deploy-command:
                HEROKU_APP_NAME: develop-server
                BRANCH: develop

workflows:
  heroku_deploy:
    jobs:
      - test

      - deploy:
          requires:
            - test
          filters:
            branches:
              only:
                - master
                - develop
                - staging

For every integration, the pipeline will run 2 jobs, test for all branches, and deploy for just branch master, develop, and staging. The jobs are applied to feature branches gonna be shown in Pull Request.

There are 4 parts that I think you need to notice:

orbs: This session is where we declare "modules", like in npm in case we use Javascript. By adding orbs, we are free to use the commands inside.
commands: we define reusable commands, and they are called by jobs.
jobs: we define what they will do for each job, they may call commands on orbs or commands session.
workflows: jobs will be called here, followed by the arrangement from top to bottom. Besides, we can require another successful running job, and which branch will be applied for the job. As you can see in the figures below, the branch feature/demo is just required to pass the test job. In case the job is failed, the developers need to fix it before merging, That helps us save develop branch by preventing potential bugs. It's looked cool, right?

Let merge develop branch to staging branch, which are required successful both jobs

Paypal.

I hope that this post helps you have some idea for your pipeline. I am really happy to receive your feedback on this article. Thanks for your precious time reading this.

CI/CD pipeline

Dantis Mai — Sat, 25 Sep 2021 09:57:35 +0000

What is CI?

CI is Continuous Integration. It is a way-of-work that required teammates to integrate their work regularly. For each integration, the host server will do a series of steps to detect potential issues, automate repetitive manual processes, like test, build....

What is CD?

CD is Continuous Delivery. This process will automatically deploy every change in team code to the testing or staging environment. Besides, It also allows developers to set up other steps like system testing, performance testing... before actually deploy it to the production environment.

In addition, CD can stand for Continuous Deployment (they are all CD in short form =) ). Continuous Delivery is a technique to auto-deploy code change to the staging environment, but manually deploy to the production, and Continuous Deployment is a technique to auto-deploy to production.

Basically, the staging and production environments have the same configuration. However, the production is a serious case, we don't want to make any mistakes on it. So, we usually choose manual deployment for the production.

CI/CD process

Different processes are depending on companies. Below one is applied in most of my projects.

Step 1 [Hand]: Init repository with, usually, master, staging, and develop branch.
Step 2 [Hand]: Developer will do their jobs in a new branch, then create Pull Request (PR) to start Peers Review process. The new feature will be pushed to the develop branch when the PR is accepted.
Step 3 [Pipeline]: After pushing code to the develop branch, CI/CD pipeline is triggered to process testing. If it is passed, The feature will be deployed to the develop environment.
Step 4 [Hand]: Tester/QE will perform testing before confirming that the new feature is good to go.
Step 5 [Hand]: Developer create PR to merge code from the develop branch to the staging branch.
Step 6 [Pipeline]: CI/CD pipeline is triggered to process another test suite. If it is passed, the new feature will be deployed to the staging environment.
Step 7 [Hand]: Tester/QE will perform testing on the staging environment.
Step 8 [Hand]: Developer create PR to merge code from the staging branch to the master branch.
Step 9 [Pipeline]: CI/CD pipeline finishes its job, and the new feature will be deployed on production.
Step 10 [Hand]: Tester/QE need to deal with the production environment. If it is not ok, developer will roll back the previous version.
Step 11 [Hand]: Pray!!!🙏🤞

Pipeline tools

With the movement of DevOps methodologies and Agile culture, a lot of good CI/CD tools are developed. Depending on your favor, you can find one here. In this series, I will use CircleCI.

Paypal.

this is the first artical of CI/CD with CircleCI on Heroku series. Honetly, I had a lot of fun when trying to build a CI/CD pipeline for my side project, and I hope you will too.
I am really happy to receive your feedback on this article. Thanks for your precious time reading this.

Applying Domain-Driven Design in Developing Software

Dantis Mai — Sun, 19 Sep 2021 13:36:08 +0000

Domain-Driven Design (DDD) is a concept of developing software that the structure of the codebase deeply matched the business domain.

For more information about DDD, you can find it here

It took me pretty much time to understand and apply it to daily tasks, so this article is my sharing all I know about DDD.

Rule

That rule specifies that something declared in an outer circle >must not be mentioned in the code by an inner circle.
--- The Clean Architecture - Uncle Bob ---

Dependency is the only thing we can't avoid in software development because a component/class either uses another component/class or is being used by another one. But we can control the direction of dependency.

Let say we have a chain A -> B -> C

A depends on B, B depends on C, then C depends on A. So if C changes, B has to change, which leads to A has to change. We call this circular dependency, which is always a bad idea.
We can control this direction of dependency by Dependency Inversion. Adding additional component A', we have the new chain A -> B -> C, C -> A', A -> A'. Now, everything depends on A'. In this case, C may change, A may change, but when A' is the same, the changes won't affect others.
A' could be an interface in the context of Java language, and domain at the module level.

Layers role

Entity

Description: represent the entry point of requests, messages from SQS, SNS, or Kafka
Roles:
- Middleware (Request validation, error handler).
- Router.
- Build response.

Repository

Description: call request to external resources like external APIs.
Roles:
- Middleware (Request validation, authentication) for external API.
- No business logic here.

Infrastructure

Description: define resources replaced to infrastructure, like Database, Queues.
Roles:
- Establish connection.
- No business logic here.

Application

Description: contain orchestration/process logic.
Roles:
- Handle business flow.
- Make external calls to retrieve data.

Domain

Description: contain shared logic across process, mainly focus on a specific domain.
Roles:
- validation for business logic.
- implement business logic.

Sometime, to simplify the codebase structure, We can combine infrastructure to repository, I do it in my example below.

When we talk about DDD, we talk about interfaces, and all of them are declared in the domain layer for each specific domain.
It's too much for theory, now come back to reality!!

How to apply it to project

Let assume we have a business flow below, which is about booking a seat in a cinema:

IsInvestor: We don't want to cause any trouble for investors, so if they ask for a seat, we will take it for them.
IsEligible: in this step, we will check whether the user's age is old enough.
Register seat: register the user for the seat.
Notification: send email to let them know whether the request is success or fail.

Now we structure our codebase from outer layer to inner layer based on the onion model above.

Controller: This layer handles 3 things
- Mapping router to the controller.
- Validate required data for the request.
- Get suitable business flow in the Application layer to handle this request
Repository: This layer makes requests to external services. In this case, we may need user information from User downstream, seat information from Seat database.
Application: this layer just handles business logic of a specific domain, otherwise the Domain layer will do the job.
Domain: this layer handles business logic about User, Seat, and Email service. There is a lot of work to do here:
- Retrieve required data for business logic from downstream.
- Return result of isInvestor, isEligible, and so on.

After all all codebase will look like:

├── controller
|  └── register-seat
|     ├── registerSeat.controller.ts
|     ├── registerSeat.router.ts
|     └── registerSeat.validator.ts
├── application
|  └── register-seat
|     └── registerSeatService.ts
├── repository
|  └── email
|  |  ├── type.ts
|  |  ├── emailBuilder.ts
|  |  └── emailService.ts
|  ├── user
|  |  ├── type.ts
|  |  └── userService.ts
|  └── seat
|     ├── type.ts
|     ├── seatSchema.ts
|     └── seatService.ts
└── domain
   ├── user
   |  └── eligibility
   |     ├── isSeatAvailable.ts
   |     └── isCurrentUserOldEnough.ts
   ├── email
   |  └── sendEmail.ts
   └── seat
      └── seatInfoProvider.ts

Note: sub-domains name may have different of mine, depended on project business domain.

As we can see in the onion model, the registry is the most outer layer, and the domain is the most inner one. How on earth the domain will call repositories to integrate with outside resources, so that violates our rule, right?

If we stay still applying the same thing for domain and repository, the answer for that question is yes 😑, we definitely break out the golden rule. It is the place which Dependency Inversion plays its role. After creating an interface for a repository, we inject that service into domain logic. So that, we separate 2 of them.x

Paypal.

This article is all thing I know about DDD, I may misunderstand some points, please help me fix it by leaving your feedback, I am really happy. Thanks for your precious time reading this.

Create Express Typescript Boilerplate

Dantis Mai — Sat, 11 Sep 2021 12:03:05 +0000

What is Git Template?

Git Template is a frame for us to make numerous clones with the same configuration

Create boilerplate

Init git repository

Depends on familiarity, we can init repository by using Git Interface then clone it back or executing Git CLI on local folder.

Add configuration

First of all, we need to initialize nodejs project by answering some questions after executing the command npm init

Then we install Typescript by npm or yarn. I suggest installing it with --save-dev flag, because usually, the production package is built to Javascript*



#For npm
npm install --save-dev typescript

#For yarn
yarn add --dev typescript

Now, we need to config our project. If you have followed me until this post, you will have the configuration of tsconfig.json, Prettier, ESLint, Jest, and Husky

We come to the main guy, Express server

Install Express module. As I mentioned in Jest, Express can't understand TS, so we need an additional module,ts-node, to run the server on local, and 2 other modules @types/express, @types/node to get types of Express.



npm install express
npm install --save-dev @types/express @types/node ts-node

There are some other ones you may need nodemon for watching the changes in the resource folder, dotenv for loading environment variables files, or cors for solving error "access-control-allow-origin".
Create our server. From my experience, we need to create 2 files in src folder placed at root level. The first one is src/config/express.ts which is used to config our express server, and the second one is src/index.ts for starting the server. If we merge 2 of those files, we will violate the SOLID theory.
If you ask about errorhandler middleware, I have an example for you below. And about @controller, it depends on your domain business.
Add scripts to package.json to start server. Thanks to ts-node we can directly start server without continuous complier.

"start": "ts-node -r tsconfig-paths/register src/index"

- Try `npm start` to make sure it can start the server successfully
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8rw1uoj0nx9llk98xpo6.png)

- Add **Unit tests** to make sure everything works as expected. In case you followed my configuration in this [post](https://dev.to/maithanhdanh/configuration-for-javascript-testing-framework-jl1), then push test files into folder `__tests__` placed at root level with the same location in `src` (your folder tree will look like below). I love using [supertest](https://www.npmjs.com/package/supertest) to test my express server, you can make this [page](https://www.albertgao.xyz/2017/05/24/how-to-test-expressjs-with-jest-and-supertest/) as your reference
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5kju7jqs3yh9oepc0777.png)
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8d1gj71p1s9hwaqkpb9c.png)
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jzzw7n1nerio8q51k76g.png)

- Now we can try to commit the changes to init our repository. If we config `Husky`, then it will run `npm test` before actually commit
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uijd9rycullsi9pzi3nd.png)

###Mark repository as template
Finally, we come to the last part. After access our repository on [github](https://github.com/), we tick the box **template repository** in tab **setting**
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vvkmjqhjirq5b5qxjl67.png)
**CONGRATULATION!!! EXPRESS TYPESCRIPT BOILERPLATE ACHIEVED**

You can try to use it by clicking on **Repository template** on **New repository** page

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dms70q4vf3ihcpa4e0y1.png)
This is my [Template](https://github.com/Maithanhdanh/express-typescript-boilerplate), I am glad if you give me a star 😍.
And [here](https://www.npmjs.com/package/typescript-maker) is my brand new npm 😍.

_We have gone on a long journey with the **Create Your Own TypeScript Express Template** series. Thank you very much. If You need GitHub template, you can refer [here](https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/creating-a-repository-on-github/creating-a-repository-from-a-template)_

[Paypal](https://paypal.me/DantisMai).

_I am really happy to receive your feedback on this article. Thanks for your precious time reading this._

Configuration for Husky + pre-commit

Dantis Mai — Mon, 06 Sep 2021 13:22:41 +0000

What is Husky?

Husky improves your commits and more 🐶 woof!

Husky helps us do more things along with git commands. For example, we can run npm test in pre-commit phase and do something else in post-commit phase

Setup

There is a bit difference between npm version below and above 7, so please check it by



npm -v

Automatic (require npm version > 7)



#For npm
npx husky-init && npm install

#For Yarn 1
npx husky-init && yarn

#For Yarn 2
yarn dlx husky-init --yarn2 && yarn

After executing the command successfully, we need to take a look at the directory tree to make sure .husky/pre-commit is there.

Manual

Install Husky ```

npm install --save-dev husky

- Enable git hooks

npx husky install

- Add `prepare` script to `package.json`, this script will be trigger enable Git hooks after install. This step also depeneds on our npm version
  - npm > 7: `npm set-script prepare "husky install"`
  - npm < 7: copy `"prepare": "husky install"` to `scripts` in `package.json` 
 `
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s0o6oq1dqngrjpttsiu2.png)

Once here, our folder tree look like below. 
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/avhpmlo0rqafqel6o3j1.png)

- Now we need to create a hook by command `husky add`. After executing the below command, a line `npm test` is added to a new file `pre-commit` in `.husky`, which means `npm test` will be run before we actually commit.

```
npx husky add .husky/pre-commit "npm test"
```

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/khzirhq9u0dvov4y931q.png)

*If you use yarn2, [here](https://typicode.github.io/husky/#/?id=yarn-2) is your reference*.

##Have fun
Now, depending on our needs, we list out commands in the file `pre-commit`. In my case, I want to verify **branch name pattern**, **lint**, and **test cases**
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fhgy3ws6egqh57nupnh0.png)

[Paypal](https://paypal.me/DantisMai).

_I hope you had fun with git hook. I am really happy to receive your feedback on this article. Thanks for your precious time reading this._

Configuration for Jest

Dantis Mai — Sat, 04 Sep 2021 12:24:04 +0000

Testing framework

Besides development, testing plays an important role, it helps us to make sure any developed feature works as expected. Especially in the refactoring codebase process, the business has to be the same, so we just basically go through all the test cases without changing them. If all of them are passed, our new codebase is good to go.

There are a lot of testing framework for Javascript (JS), which has thier own strength and weakness. Please take a look on this Comparing JS testing framework to find one. In this post I will use Jest.

Configuration

My configuration

I use the configuration below in most of my Typescript projects. Because there are some important parts, I will explant them later, but for others, you can search in configuration files

import path from 'path';
const rootDirector = path.resolve(__dirname);

export default {
  clearMocks: true,
  collectCoverage: true,
  coverageDirectory: 'coverage',
  coverageProvider: 'v8',
  coverageThreshold: {
    global: {
      branches: 70,
      function: 80,
      lines: 80,
      statements: 80,
    },
  },
  globals: {
    'ts-jest': {
      tsconfig: path.resolve(__dirname, 'tsconfig.json'),
    },
  },
  moduleDirectories: ['node_modules'],
  moduleFileExtensions: ['ts', 'tsx', 'js', 'jsx', 'json', 'node'],
  moduleNameMapper: {
    '@server(.*)$': `${rootDirector}/src$1`,
    '@config(.*)$': `${rootDirector}/src/config$1`,
    '@tests(.*)$': `${rootDirector}/__tests__$1`,
    '@domain(.*)$': `${rootDirector}/src/domain$1`,
    '@controller(.*)$': `${rootDirector}/src/controller$1`,
    '@middleware(.*)$': `${rootDirector}/src/middleware$1`,
  },
  reporters: [
    'default',
    [
      path.resolve(__dirname, 'node_modules', 'jest-html-reporter'),
      {
        pageTitle: 'Demo test Report',
        outputPath: 'test-report.html',
      },
    ],
  ],
  rootDir: rootDirector,
  roots: [rootDirector],
  setupFilesAfterEnv: [`${rootDirector}/__tests__/setup.ts`],
  testPathIgnorePatterns: [
    '/node_modules/',
    '<rootDir>/build',
    `${rootDirector}/__tests__/fixtures`,
    `${rootDirector}/__tests__/setup.ts`,
  ],
  transform: {
    '^.+\\.ts$': 'ts-jest',
  },
  testRegex: ['((/__tests__/.*)|(\\.|/)(test|spec))\\.tsx?$'],
};

To use

Yup, first step is always installing the package, we can use the below command to do it. Besides, we are going to setup Jest config for TS project, so remember to install ts-test also.

#For npm
npm install --save-dev jest ts-jest

#For yarn
yarn add --dev jest ts-jest

Create a file jest.config.ts or jest.config.js at the root level, and copy the configuration above to this file.
Add a script "test": "jest" to scripts in our package.json. By default, without the flag --config, jest will search in root level for the configuration file, otherwise, we need to evaluate value for --config, and also update options in the config, like ts-jest, reporters, and so on.

# Root level
"scripts": {
    "test": "jest",
},

# With config flag
"scripts": {
    "test": "jest --config='./jest-config/jest.config.ts'",
},

Explaination

Importing Path module: the module helps us navigate directory easier. In case we let jest.config.ts in other folder, let say jest-config:
- <rootDir> in this case is the directory that we put in flag --config, so now the value for setupFilesAfterEnv must be ['<rootDir>/../__tests__/setup.ts']
- if we use path module, rootDirector will point to root level.

#Without path module
setupFilesAfterEnv: [`<rootDir>/../__tests__/setup.ts`]

#With path module
setupFilesAfterEnv: [`${rootDirector}/__tests__/setup.ts`]

globals: because Jest can't undestand TS, we need to install additional module ts-test, and also define ts-test config in this option.
moduleNameMapper: in the post tsconfig.json, I use module-alias to simlify importing modules, so I need to register those mappings again.
transform: it is the place for us to define pattern filename for each testing framework. In my config, I just use ts-test to test my TS modules.
testRegex: this option is used to let Jest know how our test files look like, where they are.

To be honest, Mock is the most important term in testing. If you want to follow developer journey, I suggest you fully understand this term

Paypal.

I hope this post helps you with config testing environment for your project. I am really happy to receive your feedback on this article. Thanks for your precious time reading this.

Configuration for ESLint

Dantis Mai — Fri, 03 Sep 2021 05:18:31 +0000

What and Why ESLint

ESLint is a code analyzer, it will prevent errors by finding and fixing problems in our codebase from variable to syntax

Configuration

My configuration

I just use the default configuration from command eslint --init.

{
  "env": {
    "browser": true,
    "es2021": true
  },
  "parser": "@typescript-eslint/parser",
  "parserOptions": {
    "ecmaVersion": 12,
    "sourceType": "module"
  },
  "plugins": ["@typescript-eslint"],
  "extends": [
    "eslint:recommended",
    "plugin:@typescript-eslint/recommended",
    "prettier"
  ],
  "rules": {}
}

To use

Just as my post about prettier, we need to install ESLint with flag --save-dev. You can reach to the first step of part To Use in the last post to get an answer for question why we need the flag.

# for npm
npm install --save-dev eslint 

# for yarn
yarn add --dev eslint

After installing the package, we initialize the configuration file for the project.

# for npm
$ npx eslint --init

# for yarn
$ yarn run eslint --init

Then, it will ask us a lot of questions to generate a suitable config.

Notes:

Because I've already installed @typescript-eslint\eslint-plugin and @typescript-eslint\parser, I selected No for the question Would you like to install them now with npm.
ESLint also supports pretty much formats of the configuration file. Configuartion Files

Other Options

ESLint is a great tool for us, and well worth for our digging into the Documentation. To understand more how it behaves, you can test ESLint on demo. By clicking on Rules Configuration, you can change the configuration.

After enjoy while playing around, ESLint allows us to download the current config.

Paypal.

Now, we already have tsconfig, prettier and eslint, I hope they can you guys with your current projects. I am really happy to receive your feedback on this article. Thanks for your precious time reading this.

Configuration for prettier

Dantis Mai — Wed, 01 Sep 2021 17:29:05 +0000

What and Why prettier?

Prettier is a code formatter, which supports many kinds of language from Programming languages (JXS, TS), to Style sheets (CSS, SCSS), and also Serialization languages (YAML).

As we can see in the gif above, it will take a lot of time to arrange those lines, and even more for HTML. Prettier is the light from heaven, just saving the file everything will be in order

Configuration

My configuration

My beloved configuration is below



{
  "singleQuote": true,
  "trailingComma": "all",
  "useTabs": false,
  "tabWidth": 2,
  "overrides": [
    {
      "files": "*.yaml",
      "options": {
        "tadWidth": 2,
        "printWidth": 40,
        "singleQuote": true
      }
    }
  ]
}

To use

First of all, we need to install Prettier with flag --save-dev, because it is just useful for development, but not production. Without --save-dev, our production package will containts many unused modules (reference about dependencies vs devdependencies) ```

npm

npm i --save-dev prettier

yarn

yarn add --dev prettier

- Create 2 files `.prettierrc`, `.prettierignore` in our root level, the former is for configuration and the latter is for ignoring directories. Usually, I ignore `build` and `coverage`, which is generated from build production package and test result corresponding.
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a2x74kcrk7v3nl2p6pph.png)
- Done!!

_Notes:_ Instead of `.prettierrc`, `prettier` also supports a wide range of types. Please take a look at [Configuration File](https://prettier.io/docs/en/configuration.html).

##Other options
You can find more interesting ways to use it in [prettier documentation](https://prettier.io/docs/en/options.html), such as lint-staged, pre-commit, and so on.

I usually due with CloudFormation, so besides a general configuration, I `overrides` it with my favorite style for `.yaml`. You can find your own one while [playing around](https://prettier.io/playground/) with it.
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xbaqumw7c2ykkwgxbfxi.png)

[Paypal](https://paypal.me/DantisMai).

_I hope this post helps you find your style in your developer journey. I am really happy to receive your feedback on this article. Thanks for your precious time reading this._