Forem: Thy Pham

A simple decision tree on how to choose communication styles between microservices

Thy Pham — Mon, 04 Apr 2022 13:00:34 +0000

When I design a microservices system, one of the questions I always ask myself: how will service A communicate with service B?

This one question seems simple, but the right (maybe just reasonable) answer can depend on many things. Such as synchronous or asynchronous? Request-response or event-driven? Queue or Topic? REST or GraphQL or RPC? etc.
That's why I decided to put some effort into building a decision tree that can somehow help choose a good communication style between two services.

Note: We can use a mix of styles in our microservices, and don't take it for granted. My decision tree can have mistakes and can be incomplete, and I very much appreciate it if you could help me correct them :)

You can open the original size image of the decision tree here: https://live.staticflickr.com/65535/51981736503_9eb0839e08_h.jpg

Build a service catalog for your software system using Backstage

Thy Pham — Wed, 30 Mar 2022 19:40:15 +0000

Problem

A software system often contains multiple components such as backend services, database, API, documentation, etc. Is usually hard for us to have an overview, such as how many components we have? which component is connected to which? Who is the owner? Where can I find the documentation? Where can I see the API? etc.
One way to solve this problem is to build a catalog for our infrastructure. We can achieve this quickly with the help of Backstage.

What is Backstage?

Backstage is an open source project launched by a team at Spotify and is currently a CNCF Incubation project. Its purpose is to manage the microservices and infrastructure of the software systems.

Backstage has a Dashboard UI. It uses plugins for different features. Each plugin is presented on a separate page in the Dashboard and has its own URL. So, for example, we can access the catalog plugin using /catalog and the documentation plugin using /docs.

In the scope of this post, I will demonstrate the use of the catalog plugin. More plugins can be found in the market here. In addition, I suggest visiting its Docs page for more details on the Backstage project.

Catalog plugin

The catalog plugin lists all the services, libraries, databases, etc. It also shows the ownership, like who owns what and the relationship between the entities. We can describe this info using YAML files committed together with the code and link these files to the catalog to show them on the UI.

Let's get our hands dirty.

Step 1: Installation

Install Backstage using npx:



> npx @backstage/create-app

Enter y to proceed

Type in the name of your app:

Choose database for Backstage backend. Here I choose SQLite just for demonstration. Note that the data will be wiped out whenever we stop the app.

After the app is created, we can go to its directory and start it locally with yarn dev command.



> cd my-system
> yarn dev

Backstage runs at localhost:3000 by default. Open the catalog page http://localhost:3000/catalog, and we will see this screen below.

As you can see, Backstage already includes three services playback-order, searcher, and shuffle-api, just for demonstration. Backstage defines these services in app-config.yaml file located in the root directory of the Backstage project code. You can see them at catalog.locations. Feel free to remove them if you like. I will just let them there. insert lazy meme :))

Step 2: Define our services

We can define our service by writing a YAML file. We should name it catalog-info.yaml, as Backstage suggests.

Let's say we have an order service written in Node JS. We can define this service like this:



apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: order-service
  description: This service receives new orders and processes them
  tags:
    - nodejs
    - backend
spec:
  type: service
  lifecycle: production
  owner: backend-team

Backstage has explained the meanings of all the fields in the file above very clearly in its documentation. Commit this file to your Github order repository.

Step 3: Add the service to the Backstage catalog

Go to your catalog page at localhost:3000/catalog.
Click on "Create component" -> "Register existing component".
Paste the Github URL to your catalog-info.yaml file in the "Repository URL" input field. Then review and tap on import. And we're done!

Go back to localhost:3000/catalog. Select the tags nodejs, and we can see our order service is listed there!

Some thoughts

I like the feature of Backstage that allows us to define our infrastructure using YAML files and commit together with the code. This way, we can track the changes in our system by reviewing the commits.

Having everything (backend services, API, docs, resources...) in one place is super helpful. That helps dev teams have an overview and look up the info when necessary.

In my humble opinion, it would be even better to have a feature that automatically discovers our software components instead of manually writing the YAML files, so that the catalog is always up-to-date with the changes of our system.

Build microservices with Dapr in Kubernetes

Thy Pham — Sun, 20 Feb 2022 08:57:32 +0000

Introduction

Dapr is a Cloud Native Computing Foundation project currently at the Incubating stage. It was created to help us as developers build microservices quickly with ease.

Dapr runs as a sidecar and brings us some benefits, such as:

Provides an API to make it easier to build the connections between services like REST/gRPC, Pub/Sub, etc.
Includes best practices for building distributed applications.
Make your app portable, which means it can connect to different databases and secret stores, send/receive messages to/from various Pub/Sub brokers, etc., without changing the code.

To see how Dapr works, let's build a simple Order service with a REST endpoint for creating new orders. The service stores the orders into a Redis database.

We will also build a client service that consumes this endpoint. All two services are written using Node.js and Express.

We will then run the two apps in Kubernetes with Dapr sidecar. There are many other use cases of Dapr. I suggest going to its homepage for more details.

Implementation

Before following the steps below, I assume that we all have a running Kubernetes cluster. I use minikube to run a Kubernetes cluster on my local machine for testing purposes.

Step 1: Run Redis in Kubernetes

To make it simple, we can use Helm to install Redis.

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install redis bitnami/redis

Step 2: Deploy Dapr control plane to the cluster

First, follow this instruction to install Dapr CLI.

After the installation is done, let's deploy the Dapr control plane to our k8s cluster:

> dapr init --kubernetes --wait

We can now see Dapr has been installed in namespace dapr-system. You can also install Dapr in another namespace by using flag -n <yournamespace>.

Step 3: Create a Dapr state store component to let Dapr connect to Redis.

To make Dapr recognize and connect to Redis, we have to create a Dapr component by writing a YAML file:

# file Redis.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  # We will use this name to configure the dapr-client SDK
  # in our Order service code so Dapr sidecar can connect
  # to this Redis store
  name: redis-store
spec:
  type: state.redis
  version: v1
  metadata:
    # The Redis host and password below are the default
    # values after installing Redis using Helm. You should
    # provide your values if you have your own Redis instance.
    - name: redisHost
      value: redis-master:6379
    - name: redisPassword
      secretKeyRef:
        name: redis
        key: redis-password

Apply the component:

> kubectl apply -f redis.yaml

Step 4: Build Order service.

Now it's time to write our Order service. In the code below, I use Dapr SDK to store data in Redis through Dapr sidecar running locally. Note that at this line: const STATE_STORE_NAME = "redis-store". The store name here must be the same as the component name defined in Redis.yaml file.

const express = require("express");

// Dapr SDK
// > npm install dapr-client
const { DaprClient, CommunicationProtocolEnum } = require("dapr-client");

const app = express();
const port = 3000;

// Our Order service only interacts with Dapr sidecar running in local.
const daprHost = "127.0.0.1";

// Create a new Dapr client
const client = new DaprClient(
  daprHost,
  // The HTTP port that the Dapr sidecar is listening on.
  // We should use this env var to connect to Dapr sidecar
  process.env.DAPR_HTTP_PORT,
  CommunicationProtocolEnum.HTTP
);

// Name of the state store component, defined in Redis.yaml file.
const STATE_STORE_NAME = "redis-store";

app.use(express.json());

app.post("/orders", async (req, res) => {
  const order = req.body;

  // We can use this Dapr client to store data into database.
  const response = await client.state.save(STATE_STORE_NAME, [
    {
      key: new Date().getTime().toString(), // Generate a unique key.
      value: order,
    },
  ]);

  res.json(response);
});

app.listen(port, () => console.log(`server listens on port ${port}`));

As we can see, our Order service doesn't need to know anything about Redis, and it can still interact with it. This makes our Order service portable.

I have built a Docker image for the Order service, and it is available on Docker Hub at tphamdev/dapr-example-order-service. For more details about the steps to create a Docker image and push it to the Docker Hub, you can have a look at this post here.

Step 5: Create a K8s Deployment for our Order service

Let's write a Deployment file for our Order service. You can see I use three annotations, dapr.io/enabled, apr.io/app-id and dapr.io/app-port in the file below. These annotations let the Dapr control plane knows that it should deploy a sidecar to our Order service Pod.

# file order-service-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      # We add the annotations below to let Dapr recognize
      # and deploy the sidecar together with our service in the pod.
      annotations:
        dapr.io/enabled: "true"
        # The client service will use this name to locate
        # the Order service through the Dapr sidecar.
        dapr.io/app-id: "order-service"
        # The port that your application is listening on
        dapr.io/app-port: "3000"
      labels:
        app: order-service
    spec:
      containers:
        - name: order-service
          image: tphamdev/dapr-example-order-service

Apply the deployment:

> kubectl apply -f order-service-deployment.yaml

Step 6: Build a client to consume the Order service endpoint

Let's create a simple client service that calls the /orders endpoint from the Order service above to create new orders.

This client service will use Dapr SDK to interact with the Dapr sidecar running on localhost. The Dapr sidecar will take care of the rest (connect to the Order service, also through another Dapr sidecar).

// Dapr SDK
// > npm install dapr-client
const { DaprClient, HttpMethod } = require("dapr-client");

async function main() {
  // Our client service only connects to the Dapr sidecar running locally.
  const daprHost = "127.0.0.1";

  // The HTTP port that the Dapr sidecar is listening on.
  // We should use this env var to connect to Dapr sidecar
  const daprPort = process.env.DAPR_HTTP_PORT;

  // Create a new Dapr client
  const client = new DaprClient(daprHost, daprPort);

  // This is the value of the dapr.io/app-id annotation,
  // which we defined in order-service-deployment.yaml file.
  const serviceAppId = "order-service";

  // The REST Endpoint /orders
  const serviceMethod = "orders";

  for (let i = 1; i < 100; i++) {
    // New order
    const order = {
      name: `order-${i}`,
    };

    // Invoke the Order service using Dapr SDK (through Dapr sidecar)
    const response = await client.invoker.invoke(
      serviceAppId,
      serviceMethod,
      HttpMethod.POST,
      order
    );

    console.log(`order-${i} was created:`, response);

    await sleep(2000);
  }
}

function sleep(time) {
  return new Promise((resolve) => setTimeout(resolve, time));
}

main();

The Docker image of the client is tphamdev/dapr-example-order-client

Step 7: Create K8s Deployment for the client service.

The deployment file is the same as the order-service-deployment.yaml file in Step 5 above. We also add some annotations to let Dapr recognize and deploy the sidecar.

# file order-client-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-client-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-client
  template:
    metadata:
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "order-client"
      labels:
        app: order-client
    spec:
      containers:
        - name: order-client
          image: tphamdev/dapr-example-order-client

Apply the deployment:

> kubectl apply -f order-client-deployment.yml

Confirm that the pods are running:

> kubectl get pod

❯ kubectl get pod
NAME                                        READY   STATUS    RESTARTS       AGE
order-client-deployment-55948499c8-4snfg    2/2     Running   0              11s
order-client-deployment-55948499c8-b9g68    2/2     Running   0              11s
order-client-deployment-55948499c8-fz5rr    2/2     Running   0              11s
order-service-deployment-59f7b4c5cd-7g87k   2/2     Running   0              2m33s
order-service-deployment-59f7b4c5cd-ddlrs   2/2     Running   0              2m33s
order-service-deployment-59f7b4c5cd-hg52f   2/2     Running   0              2m33s
redis-master-0                              1/1     Running   1 (112m ago)   19h
redis-replicas-0                            1/1     Running   2 (111m ago)   19h
redis-replicas-1                            1/1     Running   2 (111m ago)   19h
redis-replicas-2                            1/1     Running   2 (111m ago)   19h

Let's see the logs of order-client:

> kubectl logs order-client-deployment-55948499c8-4snfg order-client -f

order-1 was created:
order-2 was created:
order-3 was created:
order-4 was created:
order-5 was created:
order-6 was created:
order-7 was created:
order-8 was created:
order-9 was created:
...

Now let's confirm what we have in Redis.

> kubectl exec -it redis-master-0 -- sh

$ redis-cli -a "password from Helm"

127.0.0.1:6379> keys *


  1) "order-service||1645345085506"
  2) "order-service||1645345149800"
  3) "order-service||1645345188626"
  4) "order-service||1645345150485"
  5) "order-service||1645344965614"


127.0.0.1:6379> hgetall "order-service||1645345085506"

1) "data"
2) "{\"name\":\"order-8\"}"
3) "version"
4) "1"

127.0.0.1:6379>

Yay! The Order service has successfully stored data into Redis.

How to run Node app in Kubernetes

Thy Pham — Thu, 17 Feb 2022 16:24:10 +0000

Summary

We can follow these main steps to run our Node app in Kubernetes. In this post, I use Docker as the container runtime for Kubernetes.

Write the app in Node.js.
Write a Dockerfile and build a Docker image from it.
Push the image to Docker Hub.
Use the image above to create a Kubernetes Deployment.

Step 1: Write the app in Node.js

To make it simple, I wrote a web app using Express, which has only one API Endpoint:

const express = require("express");
const app = express();
const port = process.env.PORT || 3000;

app.use(express.json());
app.get("/greeting", (req, res) => res.send({ message: "hello" }));

app.listen(port, () => console.log(`server listens on port ${port}`));

Step 2: Build a Docker image

We can use the Dockerfile below to instruct Docker to build a container image for our app:

FROM node:16-alpine
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
CMD [ "node", "server.js" ]

We also need a .dockerignore file so Docker will not copy unnecessary files to our image.

node_modules
.DS_Store
npm-debug.log

Now let's build the image by running this command. (<your username> is your username on Docker Hub)

> docker build . -t <your username>/myapp

Step 3: Push the image to Docker Hub

The image is now created. Assume you have already had a Docker Hub account. You can log in using:

> docker login

We can now push it to Docker Hub:

> docker push <your username>/myapp

Step 4: Run the app in Kubernetes

I use minikube to set up a local Kubernetes cluster for testing purposes. If you are interested, you can follow the instruction on its homepage to set up on your machine.
So now we have a Docker image on Docker Hub, we can use it to create a Deployment:

# file deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: myapp
          image: <your username>/myapp
          ports:
            - containerPort: 3000

Create the deployment by executing

> kubectl create -f deployment.yml

Check if your pod is running

> kubectl get pod -l app=myapp
NAME                                READY   STATUS    RESTARTS   AGE
myapp-deployment-54865d44bf-h8t9r   1/1     Running   0          18m
myapp-deployment-54865d44bf-llsfs   1/1     Running   0          18m
myapp-deployment-54865d44bf-p7dkq   1/1     Running   0          18m

Check the log:

> kubectl logs myapp-deployment-54865d44bf-h8t9r myapp
server listens on port 3000

Enforce coding style check and run tests before each code commit using Git hooks

Thy Pham — Fri, 04 Feb 2022 20:28:24 +0000

Intro

Whenever we develop a new feature for our software, it's crucial to identify the problems in the code as soon as possible, ideally before committing the code to VCS. Because the later we discover bugs, the more expensive to fix them.

We can easily detect common problems like failing unit tests and wrong coding styles locally. We can even go further by enforcing it before every commit so that the code is clean and passes the tests before going to the Pull Request phase.

Sample project

Let's say I have a Node project with a function called sum() like below

const sum = (a: number, b: number): number => a + b;

export { sum };

I wrote a unit test for this function using Jest:

import { sum } from './math';

test('sum of 2 + 3 is 5', () => {
  expect(sum(2, 3)).toBe(5);
});

Furthermore, I have also set up ESLint and Prettier to check for coding style. (I will write a separate post to show you how to enable ESLint and Prettier in the Node project. This post mainly focuses on how to set up Git hook).

I added some scripts to my package.json file to run unit tests and check coding style with ESLint and Prettier

{
  "name": "sample",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "jest",
    "lint": "eslint ./src/**/*.ts",
    "prettier": " prettier --check ./src"
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "@types/jest": "^27.4.0",
    "@typescript-eslint/eslint-plugin": "^5.10.2",
    "@typescript-eslint/parser": "^5.10.2",
    "eslint": "^8.8.0",
    "eslint-config-airbnb-base": "^15.0.0",
    "eslint-plugin-import": "^2.25.4",
    "husky": "^7.0.4",
    "jest": "^27.4.7",
    "prettier": "2.5.1",
    "ts-jest": "^27.1.3"
  }
}

The commands we want to use are:

To run unit tests:

> npm test

Checking code with ESLint:

> npm run lint

Format code with Prettier

> npm run prettier

Create Git hook

To enforce running the three commands above before every code commit, we will use Husky to set up a pre-commit hook:

> npm i -D husky
> npx husky install
> npx husky add .husky/pre-commit "npm run prettier"

After running the commands above, you will see file .husky/pre-commit with command npm run prettier at the end. It meant, npm run prettier will run before every code commit.

#!/bin/sh
. "$(dirname "$0")/_/husky.sh"

npm run prettier

Append npm run lint and npm test to the end of the file

#!/bin/sh
. "$(dirname "$0")/_/husky.sh"

npm run prettier
npm run lint
npm test

Now we are done! Let's test our setup by creating our first commit:

> git add .
> git commit -m "my first commit."

The output will look like below:

> sample@1.0.0 prettier
> prettier --check ./src

Checking formatting...
All matched files use Prettier code style!

> sample@1.0.0 lint
> eslint ./src/**/*.ts


> sample@1.0.0 test
> jest

 PASS  src/math.spec.ts
  ✓ sum of 2 + 3 is 5 (2 ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        1.526 s, estimated 2 s
Ran all test suites.
[branch_1 ae6dfd0] my first commit.
 1 file changed, 1 insertion(+), 1 deletion(-)

As you can see in the result above, the three commands are automatically run. If one of the commands fails, the code will not be committed.

One JSON Schema rules them all: Typescript type, API validation, OpenAPI doc, and Swagger UI.

Thy Pham — Sun, 30 Jan 2022 17:08:17 +0000

Problem

Let's say we have an API endpoint to create a new user. The request body includes information about the user's name, age, and optional address.

This endpoint must have a request/response validator and OpenAPI documentation. It must be shown on the Swagger page as well.

To achieve this goal, we will have to create a user type, a user validation schema for validating the request and response, and another user schema for the OpenAPI doc and Swagger page.

// Typescript type
type User = {
  address?: string | undefined;
  name: string;
  age: number;
};

// Validation schema
app.post(
  '/users',
  body('address').isString(),
  body('age').isNumber().notEmpty(),
  body('name').isString().notEmpty(),
  (req: Request, res: Response) => {
    // ...
  },
);

---
openapi: 3.0.0
info:
  title: Sample API Spec
  version: 1.0.0
servers:
- url: http://localhost
paths:
  "/users":
    post:
      summary: Create new user
      requestBody:
        required: true
        content:
          application/json:
            schema:
              additionalProperties: false
              type: object
              properties:
                name:
                  type: string
                age:
                  type: number
                address:
                  type: string
              required:
              - name
              - age
      responses:
        '200':
          description: successful response
          content:
            application/json:
              schema:
                type: object
                properties:
                  message:
                    type: string
                required:
                - message
        '500':
          description: error response
          content:
            application/json:
              schema:
                type: object
                properties:
                  message:
                    type: string
                  error:
                    type: string
                required:
                - message
                - error

Defining three schemas for the user is code redundancy. The problem will come when we have to, for example, add a new field named job in the request body. Then we will have to modify all three places in our code for that update.

Solution

There is a way that allows us to create just one schema and use it for static type, API Validation, OpenAPI doc, and Swagger page. The answer is JSON Schema, with the help from these libraries:

express-openapi-validator: Validating API request and response using OpenAPI specification.
swagger-ui-express: Generate Swagger page using OpenAPI specification.

As you might have already known, OpenAPI uses JSON Schema to define its data types. So the last missing piece of our solution is:

@sinclair/typebox: this lib helps us define in-memory JSON Schema and use it as Typescript type.

So the main idea is to use Typebox to create a user JSON Schema. Then use this schema in the OpenAPI specification. Finally, use the OpenAPI spec in API validation and build the Swagger page.

Create user JSON schema

import { Static, Type } from '@sinclair/typebox';

/**
 * The Schema below is the same as
 * {
 *   additionalProperties: false,
 *   type: 'object',
 *   properties: {
 *     name: { type: 'string' },
 *     age: { type: 'number' },
 *     address: { type: 'string' }
 *   },
 *   required: [ 'name', 'age' ]
 * }
 */
const UserSchema = Type.Strict(
  Type.Object(
    {
      name: Type.String(),
      age: Type.Number(),
      address: Type.Optional(Type.String()),
    },
    { additionalProperties: false },
  ),
);

/**
 * The type below is the same as
 * type User = {
 *     address?: string | undefined;
 *     name: string;
 *     age: number;
 * }
 */
type User = Static<typeof UserSchema>;

export { User, UserSchema };

Use user JSON schema to create OpenAPI specification

import { OpenAPIV3 } from 'express-openapi-validator/dist/framework/types';
import { ErrorResponseSchema } from './ErrorResponse';
import { SuccessResponseSchema } from './SuccessResponse';
import { UserSchema } from './User';

const apiSpec: OpenAPIV3.Document = {
  openapi: '3.0.0',
  info: {
    title: 'Sample API Spec',
    version: '1.0.0',
  },
  servers: [
    {
      url: 'http://localhost',
    },
  ],
  paths: {
    '/users': {
      post: {
        summary: 'Create new user',
        requestBody: {
          required: true,
          content: {
            'application/json': {
              schema: UserSchema as OpenAPIV3.SchemaObject,
            },
          },
        },
        responses: {
          200: {
            description: 'successful response',
            content: {
              'application/json': {
                schema: SuccessResponseSchema as OpenAPIV3.SchemaObject,
              },
            },
          },
          500: {
            description: 'error response',
            content: {
              'application/json': {
                schema: ErrorResponseSchema as OpenAPIV3.SchemaObject,
              },
            },
          },
        },
      },
    },
  },
};

export { apiSpec };

Use the api spec above for validating the api request/response and build Swagger page

import express from 'express';
import * as OpenApiValidator from 'express-openapi-validator';
import * as swaggerUi from 'swagger-ui-express';
import { apiSpec } from './api';

const app = express();
app.use(express.json());
app.use(express.urlencoded());
app.use('/api-docs', swaggerUi.serve, swaggerUi.setup(apiSpec));
app.use(
  OpenApiValidator.middleware({
    apiSpec,
    validateRequests: true,
    validateResponses: true,
  }),
);
app.post('/users', (req, res) => {
  res.json({
    message: 'successful',
  });
});
app.listen(3000);

As you can see in the code above, we only have to define the user schema once using Typebox. Whenever we need to update the user schema, we only have to change the code in one place. The API validation and the OpenAPI doc, Swagger page will be updated accordingly.

What we need to know about the database

Thy Pham — Thu, 27 Jan 2022 18:43:23 +0000

In this post, I'm not trying to explain every point in detail, but rather giving an overview with a brief description of some important aspects that we need to keep in mind while working with databases.

Types of databases

Relational databases

Data are structured using tables with columns and rows. The relationship between tables is predefined. SQL is used to query the data in the database.

E.g. PostgreSQL, MySQL, MariaDB, SQLite.

Non-Relational Databases (NoSQL - aka Not Only SQL)

Data are not organized in tables but rather have a more flexible structure. These databases are designed for easy scaling horizontally and ease of use. There are some main types of Non-Relational databases:

Document databases

Document databases store each item in a JSON, BSON, or XML encoded object called a document.

E.g. MongoDB, CouchDB,

Key-Value store

Each item has a key and a value. E.g. name=John, last_name=Doe.

E.g., Redis, Memcached

Column-oriented databases

Data is stored in columns. It is normally used when our data items have a huge amount of properties (each property is stored in one column), and we just want to query some of the properties to avoid unnecessary data.

E.g., Cassandra, HBase

Graph databases

Data is held in nodes and edges. The main focus of this type of database is the connection between nodes (using edges).

E.g., Neo4j, Amazon Neptune

Consistency model

ACID

Atomic, Consistent, Isolated, Durable

BASE

Basic Availability, Soft-state, Eventual consistent

Database replication

The data has its copies on multiple interconnected machines. These machines can be in different locations. To reduce access latency, increase availability and read throughput. Some typical database replication techniques are:

Single-Leader

New data is written on only one single machine called Leader. The data will then be replicated to other machines. These machines are called Replicas. Reading data can be done on both Leader and Replicas.

Multi-Leader

Same as Single-Leader, but there are multiple Leaders. Each data center has one Leader. Data written to one Leader will be replicated to the other Leaders.

Leaderless

There is no Leader. The Replicas receive read and write requests directly.

Transaction

We use transactions when we want to execute a set of operations that can either be successful altogether or not at all.

Index

An index is a data structure that helps read data faster with a charge of slower write operations and more space needed.

Database backup

Database backup is a way to make a copy of the data. In case of a disaster that causes data loss, we can use this copy to restore our important data.

Full backup

Backup everything

Differential backup

Only backup what was changed since the last full backup

Incremental backup

Backup what was changed since the last incremental backup

Database migration

Our software often requires changes in the code and the database schema. When we deploy new code to production, the database should be updated as well. This process is called database migration. We have to make sure that during the migration progress, the database can always be rolled back to its original state if something wrong happens.

(Bonus) Serverless databases

Cloud providers like Amazon or Google will manage everything (provisioning, scaling, maintenance, etc.) for us. We just need to pay for what we use (pay-per-use).

How my company follows Agile + DevOps in daily work

Thy Pham — Tue, 25 Jan 2022 19:18:41 +0000

Summary

We form the dev teams by business domains, for example, Order, Article, or Inventory team.
Each team consists of 1 Product Owner and 4 to 5 devs.
The cross-functional teams can do all Frontend, Backend, DevOps, and QA tasks.
We develop new features within every 2-weeks, called "Sprint."
All teams hold a daily standup in the morning to quickly talk about what we did yesterday, what we're going to do today, and what is blocking us at the moment.
At the beginning of each Sprint, we have a "Sprint-planning" meeting where we plan and estimate new features that we will develop in the next two weeks.
Our Product Owner writes each new feature in one Story ticket on the Kanban board. After that, the devs will split the ticket into smaller sub-tasks. Each sub-task should be small enough to build, test and deploy within one day.
The tickets will move through 5 main columns on Kanban board: Backlog -> In Progress -> In Code Review -> In QA -> Done
We only have one main git branch (also called main). Everyone uses this branch to deploy the code to the development environment for testing. We will deploy the code from this main branch to production at the end.
(Note) We containerize all the services and use Terraform or CloudFormation to define the infrastructure.
When devs start with a new feature, a new feature branch will be branched off from main with the ticket name at the beginning of the branch name, for example, FEAT-2192-build-create-order-endpoint. All code commits must have the ticket name too.
After implementing the new feature, the devs will make a Pull Request for others to review. At this point, the CI/CD pipeline will run the tests and linting. Only when all pass, then the Pull Request is ready.
Code reviewers will check for coding style, tests, potential bugs, etc. And approve it if everything is ok.
The devs merge new code to the main branch. After that, CI/CD pipeline will deploy the code to the development environment.
Other devs or QA engineers in the same team will test the new feature on development environment.
Finally, when QA engineers approve, someone can click the deploy button and trigger CI/CD pipeline to deploy the code to production.
At the end of the Sprint, we have a Retrospective meeting where the team discusses what went well and what not to improve the process next Sprint.
Continue to Step 1. and start a new Sprint.

The tech stack that we use

Git, Github for version control.
Github Action to build CI/CD pipelines.
Kubernetes and Docker.
Terraform to define infrastructure.
AWS services.
Jira for Kanban board, Confluence for writing documents and Wiki pages.
Slack for communication.
Bitwarden for storing password.

Dependency Inversion in practice (example code in typescript)

Thy Pham — Sun, 23 Jan 2022 15:33:03 +0000

Summary

Dependency Inversion is a technique (as the name suggests) to invert the dependency from one module to another.
Without DI: A -> B (A depends on B)
With DI: (A -> I) <- B (B depends on A through interface I)
We can use DI to protect our core business logic module (and its tests) from depending on other modules (Database, HTTP API, etc...)

What is Dependency Inversion (DI)?

Imagine a class called UserService that needs to interact with MongoDB to save new user data. UserService will call createUser() function provided by UserRepoMongoDB class to serve this purpose. That means UserService depends on UserRepoMongoDB.

// file ./UserRepoMongoDB.ts
interface MongoDBUserSchema {
  // MongoDB user schema fields
}

class UserRepoMongoDB {
  createUser(name: string, age: number): Promise<MongoDBUserSchema> {
    // Some MongoDB specific logic to store user data in MongoDB
  }
}

export { UserRepoMongoDB };

// file ./UserService.ts
import { UserRepoMongoDB } from "./UserRepoMongoDB";

class UserService {
  createUser() {
    return new UserRepoMongoDB(). createUser("Max Mustermann", 20);
  }
}

export { UserService };

To make UserRepoMongoDB depends on UserService, we can create an interface IUserRepo with the createUser() function. UserService will use this function from IUserRepo instead. And UserRepoMongoDB will implement this interface:

// file ./IUserRepo.ts
interface User {
  name: string;
  age: string;
}

interface IUserRepo {
  createUser(name: string, age: number): Promise<User>;
}

export { User, IUserRepo };

// file ./UserRepoMongoDB.ts
import { IUserRepo, User } from "./IUserRepo";

interface MongoDBUserSchema {
  // MongoDB user schema fields
}

class UserRepoMongoDB implements IUserRepo {
  createUser(name: string, age: number): Promise<User> {
    // 1. Store user into MongoDB.
    // 2. Convert result from MongoDBUserSchema type to User type and return.
  }
}

export { UserRepoMongoDB };

// file ./UserService.ts
import { IUserRepo } from "./IUserRepo";

class UserService {
  constructor(private userRepo: IUserRepo) {}

  createUser() {
    return this.userRepo.createUser("Max Mustermann", 20);
  }
}

export { UserService };

// file ./main.ts
import { UserRepoMongoDB } from "./UserRepoMongoDB";
import { UserService } from "./UserService";

function executeCode() {
  new UserService(new UserRepoMongoDB()).createUser();
}

When DI comes to the rescue

One day the team decides to use DynamoDB instead of MongoDB, so we must change the code to make it works. Without DI, we need to create a new class UserRepoDynamoDB with the createUser() function and change our UserService to use this new function. That means we have to change our core business logic code (UserService) every time there is an update on the database module.

// file ./UserRepoDynamoDB.ts
interface DynamoDBUserSchema {
  // DynamoDB user schema fields
}

class UserRepoDynamoDB {
  createUser(
    Id: string, // DynamoDB needs this field in user Table
    name: string,
    age: number
  ): Promise<DynamoDBUserSchema> {
    // store user data in DynamoDB
  }
}

export { UserRepoDynamoDB };

// file ./UserService.ts
import { randomUUID } from "crypto";
import { UserRepoDynamoDB } from "./UserRepoDynamoDB";

class UserService {
  // This function is updated to adapt DynamoDB
  createUser() {
    const Id = randomUUID();
    return new UserRepoDynamoDB().createUser(Id, "Max Mustermann", 20);
  }
}

export { UserService };

But if we use DI, all we need to do is make the new class UserRepoDynamoDB implement IUserRepo, and that's it!
There is no need to change anything in UserService.

// file ./UserRepoDynamoDB.ts
import { IUserRepo, User } from "./IUserRepo";

interface DynamoDBUserSchema {
  // DynamoDB user schema fields
}

class UserRepoDynamoDB implements IUserRepo {
  createUser(name: string, age: number): Promise<User> {
    // 1. Generate new Id and Store user into DynamoDB.
    // 2. Convert result from DynamoDBUserSchema type to User type and return.
  }
}

export { UserRepoDynamoDB };

// file ./main.ts
import { UserRepoDynamoDB } from "./UserRepoDynamoDB";
import { UserService } from "./UserService";

function executeCode() {
  // Use UserRepoDynamoDB instead of old UserRepoMongoDB
  new UserService(new UserRepoDynamoDB()).createUser();
}

executeCode();