Forem: Bello Ladi

Building Context-Aware Applications in Node.js: A Deep Dive

Bello Ladi — Tue, 03 Oct 2023 15:09:04 +0000

So you're building your awesome application and would like to have some context-aware code. For example, you want all the logs generated in your app (or across your microservices) to contain the requestId that started the operation. Or perhaps you would like to always have access to the userId of whoever initiated the operation, such as to generate audit logs. But how can you keep track of this context and make it available to all downstream function calls, or even across microservices communicated via message queues, gRPC, REST, and more?

In this article, we will explore the answers to these questions, focusing on Node.JS applications. We will cover various aspects of building context-aware applications and discuss how to implement and leverage AsyncLocalStorage to store and propagate the context throughout your codebase. Additionally, we'll see how to automatically add this context to logs and share the context between microservices.

Building the Context
Leveraging AsyncLocalStorage for Context Storage
Accessing Context Downstream
Automatically Adding Context to Logs
Passing Context between Microservices

1. Building the Context

Before diving into the technical details, let's clarify what we mean by "context" in this context (pun intended). Context refers to the information that you want to propagate across various parts of your application or microservices. This can include request-related data, user-related data, or any other relevant information.

To build the context, you need to identify the specific data points that are crucial to your application's functionality. Make a list of the key data points you want to track and ensure they are accessible throughout your codebase.

Example: Express Middleware for Retrieving Request and User ID

Let's consider an example where you want to extract the requestId and userId from the headers of an incoming request in an Express application. To achieve this, you can create a custom middleware that runs before your route handlers and populates the context with the necessary information.

Here's an example implementation of the middleware:

import { RequestContext } from "../requestContext";

// Express middleware for retrieving request and user ID
function contextMiddleware(req: Request, res: Response, next: NextFunction): void {
  // Retrieve the request ID from the headers
  const requestId = req.headers['x-request-id'];

  // Retrieve the user ID from the headers
  const { userId } = decodeToken(req.headers['authorization']);

  // Create the context object
  const context = {
    requestId,
    userId,
  };

  // Store the context; More on this below
  RequestContext.start(context);

  next();
}

In this example, the middleware function contextMiddleware is responsible for extracting the requestId and userId from the headers of the incoming request. It then creates a context object containing this information.

By adding the contextMiddleware before your route handlers, you ensure that the context is populated with the requestId and userId from the headers of each incoming request. Now to make it possible for the context to be accessed downstream in your application.

2. Leverage AsyncLocalStorage for Context Storage

In newer versions of node.js, we can leverage the power of AsyncLocalStorage to store and manage the context throughout the execution flow. AsyncLocalStorage provides a mechanism to associate contextual data with the execution of asynchronous functions and enables the data to be automatically propagated as the functions are called.

In this section, we will explore how to integrate AsyncLocalStorage into your application and use it to store and retrieve the desired context. We will create a ContextStore class to hold the context data and a RequestContext class to create and access the current execution context.

// requestContext.ts
import { AsyncLocalStorage } from 'async_hooks';

export interface IInitContext {
  userId?: string;
  requestId?: string;
}

export class ContextStore {
  private _userId?: string;
  private _requestId: string;

  constructor({ userId, requestId }: IInitContext) {
    this._userId = userId;
    this._requestId = requestId;
  }

  toJSON() {
    return {
      userId: this._userId,
      requestId: this._requestId,
    };
  }

  get userId() {
    return this._userId;
  }
  get requestId() {
    return this._requestId;
  }
}

export class RequestContext {
  static als = new AsyncLocalStorage();

  static start = (ctx: IInitContext): void => {
    this.als.enterWith(new ContextStore(ctx));
  };

  static getStore(): ContextStore {
    const store = this.als.getStore();
    return store as ContextStore;
  }
}

3. Accessing Context Downstream

Now that we have successfully set the context using RequestContext, we can simply call the getStore() function within your downstream functions. This will retrieve the context associated with the current execution flow.

import { RequestContext } from "../requestContext";

// Downstream function
function randomDownstreamFunction() {
  const requestContext = RequestContext.getStore();

  // Access and use the context as needed
  console.log(requestContext.userId);
}

By retrieving the context, you can access all the relevant data points and use them for logging, auditing, or any other requirements specific to your application.

4. Automatically Adding Context to Logs

Logging is an essential aspect of any application, and having the context included in your logs can greatly assist in troubleshooting and understanding the flow of operations. To automatically add the context to your logs, you can utilize popular logging libraries such as winston, pino, or bunyan.

Here's an example using the winston library:

import { format, createLogger, transports } from "winston";
import { RequestContext } from "../requestContext";

const addContext = format((info) => {
  const requestContext = RequestContext.getStore();
  if (requestContext) {
    const oldCtx = info.ctx || {};
    info.ctx = { ...oldCtx, ...requestContext.toJSON() };
  }

  return info;
});

const logger = createLogger({
  format: format.combine(
    format.json(),
    addContext(),
  ),
  transports: [
    new transports.Console(),
  ],
});

// Usage example
function processRequest(request) {
  logger.info('Processing request...');
}

This will create logs like this

By customizing the log format and retrieving the context using getStore(), you can add the relevant context information to your logs effortlessly.

5. Passing Context between Microservices

In a microservices architecture, passing the context from one service to another becomes vital. Context propagation ensures that downstream services have access to the same context data as the originating service. This allows for end-to-end tracking and maintains the integrity of the context throughout the system.

To pass the context between microservices, you can include it in the payload of the message or the headers of the HTTP request. The specific implementation details will depend on the communication mechanism you are using (message queues, gRPC, REST, etc.).

For example, when using message queues, you can serialize the context data and include it as a part of the message payload. On the receiving end, the consumer can deserialize the data and set it as the context using RequestContext class we created.

import { RequestContext } from "../requestContext";

// publisher
function publish(data: Record<string, any>): Promise<void> {
    // add ctx to message
    let _ctx = {};
    const requestContext = RequestContext.getStore();
    if (requestContext) {
      _ctx = requestContext.toJSON();
    }
    const publishData = { data, _ctx };

    // publish via rabbitMq
}

// listener
async function onMessage(message) {
  const messageBody = msg.content.toString();

  const { data, _ctx = {} } = JSON.parse(messageBody);

  // build context from message
  RequestContext.start(_ctx);

  // process the message here
}

Similarly, when using RPC / HTTP-based communication, you can include the stringified context as Metadata or headers respectively in the request. The receiving service can extract the context and set it using AsyncLocalStorage to make it accessible throughout its execution flow.

By ensuring context propagation, you enable seamless communication and consistent context availability across your microservices.

Conclusion

Building context-aware applications in node.js requires careful consideration of the data points you want to track and their availability throughout your codebase. By leveraging AsyncLocalStorage, you can easily store and retrieve the context, making it accessible to downstream functions or microservices. Additionally, integrating the context with your logging mechanism and ensuring context propagation between microservices further enhances the effectiveness of your application.

In this article, we have covered the fundamental aspects of building context-aware applications in node.js. We hope this guide serves as a valuable resource to help you create engaging and contextually rich applications.

Happy coding!

The Easiest Way To Setup A MongoDB Replica Set on Docker-compose/Kubernetes

Bello Ladi — Mon, 23 Aug 2021 07:54:29 +0000

Let's say you're building an app, and you need to implement multi-document transactions to ensure updates across multiple collections succeed or fail together.

This article will help you set up everything you need to take advantage of this feature of MongoDB in your dev environment or when running your integration tests.

This article will not go into how MongoDB Transactions work or how to use them.
There are already several awesome resources covering that. Links to some of my favorites are included at the bottom of this article.

Beginning at 4.0, MongoDB added support for multi-document ACID transactions.

This is awesome but there's a caveat:

To utilize transactions, MongoDB must be configured as a replica set or a sharded cluster. Transactions are not supported on standalone deployments. If you are using a database hosted on Atlas, you do not need to worry about this as every Atlas cluster is either a replica set or a sharded cluster.

A replica set in MongoDB is a group of mongod processes that are used to maintain the same data set. They provide high redundancy and high availability.

If you aren't using Docker, There is an NPM package called run-rs, which can help you set this up. However, if you are developing using docker-compose/Kubernetes, or need to quickly spin up a zero-config replica set to run in a CI suite, this is for you.

Alright, lets get into it.

With Docker-compose 🐳

# docker-compose.yml

version: "3"
services:
  mongodb:
    image : davybello/mongo-replica-set:5.0.12
    container_name: mongodb
    volumes:
      - "./docker/.data/mongo1:/var/lib/mongo1"
      - "./docker/.data/mongo2:/var/lib/mongo2"
      - "./docker/.data/mongo3:/var/lib/mongo3"
    ports:
      - 27017:27017
      - 27018:27018
      - 27019:27019

That's it!!

Really, that's all you need to have a replica set up and running.

A quick breakdown of what that does:

Pulls the davybello/mongo-replica-set:5.0.12 image
Mounts the volumes for each of the individual mongoDB instances to their respective folders ./docker/.data/mongo1, /mongo2 and /mongo3
Exposes the ports for each of the MongoDB instances on ports 21017, 21018, and 21019 respectively

and it's just as easy when;

Working with kubernetes ☸️

Here's a sample pod config

apiVersion: v1
kind: Pod
metadata:
  name: mongo-pod
spec:
  containers:
    - name: mongo
    image: davybello/mongo-replica-set:5.0.12

Pretty neat, but how do you connect to this?

The ports can be exposed through a service like so

apiVersion: v1
kind: Service
metadata:
  name: mongo-service
spec:
  selector:
    app: mongo
  ports:
    - name: db1
      protocol: TCP
      port: 27017
      targetPort: 27017
    - name: db2
      protocol: TCP
      port: 27018
      targetPort: 27018
    - name: db3
      protocol: TCP
      port: 27019
      targetPort: 27019

This exposes each MongoDB instance on ports 21017, 21018, and 21019 respectively

You can connect to the replica-set from any app in your cluster using this service.

To persist your data here's a sample config using a PersistentVolumeClaim named mongo-volume

apiVersion: v1
kind: Pod
metadata:
  name: mongo-pod
spec:
  containers:
    - name: mongo
    image: davybello/mongo-replica-set:5.0.12
    volumeMounts:
      - mountPath: '/var/lib/mongo1'
        subPath: "mongo1"
        name: mongo-data
      - mountPath: '/var/lib/mongo2'
        subPath: "mongo2"
        name: mongo-data
      - mountPath: '/var/lib/mongo3'
        subPath: "mongo3"
        name: mongo-data
  volumes:
    - name: mongo-data
      persistentVolumeClaim:
        claimName: mongo-volume

Conclusion 🏁

In the end, We've seen how to easily set up a replica set, with basically zero config, using docker-compose and Kubernetes.
We've also seen how to expose the ports for each MongoDB instance. As well as persist the data with volume mounts.

If you are curious about how to set up a replica set manually, or you want to do some further reading, links are included below.

Let me know what you think about this?
And feel free to share any improvement tips for this. ✌️

Image Configuration 🛠

There are some more configuration options available on the Image.
For example, you can change the server host by passing the HOST env variable.
Check out the Readme to see what is possible.

https://github.com/DavyBello/mongo-replica-set