Forem: John Biundo

Part 5: Completing the Client Component

John Biundo — Wed, 04 Mar 2020 14:37:21 +0000

John is a member of the NestJS core team

Introduction

This is Part 5 of a six-part series. If you landed here from Google, you may want to start with Part 1.

In this article, we build the final iteration of the client component of the Faye Custom Transporter.

Reminder: Many of the concepts and terminology here are introduced and explained in this article series. That series serves as a good foundation for understanding the more advanced concepts covered in this series.

Get the Code

All of the code in these articles is available here. As always, these tutorials work best if you follow along with the code. The README covers all the steps you need to get the repository, build the apps, and follow along. It's easy! I strongly encourage you to do so. Note that each article has a corresponding branch in the repository. For example, this article (Part 5), has a corresponding branch called part5. Read on, or get more details here on using the repository.

Git checkout the current version

For this article, you should git checkout the branch part5. You can get more information about the git repository branches here.

Build the Apps for This Part

For each article in the series, we introduce some new components (and sometimes entirely new projects). For convenience, at the start of each article, you should run the following command from the top-level directory (where you cloned the repo): *

$ # from root directory of project (e.g., transporter-tutorial, or whatever you chose as the root)
$ sh build.sh

*This is a shell script that runs on Linux-like systems. You may have to make an adjustment for non-Linux systems. Note that the script is simply a convenience that runs npm install && npm run buildinside each top-level directory, so you can always fall back to that technique if you have trouble with the script.

Overview

I ended the last article with a discussion of the shortcomings of our Take 1 (first iteration) implementation of the Faye Custom Transporter client component. In this article, we'll address those shortcomings. Since this chapter is considerably more complex than the previous ones, we'll spend a fair amount of time clarifying the issues and discussing the strategy and approach before we get into the code. My intention is to clarify the concepts you'll need to address in any custom transporter, rather than focusing entirely on the code for the Faye-flavored Custom Transporter. Hopefully we're creating re-usable conceptual building blocks, and creating a big picture you can rely on in building any custom transporter!

Understanding the Multiplexing Challenge

The big problem we found with our first implementation is that it's a little naïve about handling multiple simultaneous requests. Let's make sure we understand why. The first thing to think through is the premise of multiplexing. What we're essentially doing is sharing a single pair of channels (per pattern) for sending and receiving unrelated requests. Let's focus on the behavior of the response channel, as it seems like the potentially problematic one (after all, the request channel is just sort of "fire and forget").

We uncovered the problem in our "race test" at the end of the last chapter. The triggering scenario for our problem, in basic terms, is: we sent two requests, where the second request finished before the first one. Our shared response channel just isn't prepared to handle this. It's well-behaved in the context of one-at-a-time requests: it subscribes and waits for a response, then unsubscribes. But in the face of multiple overlapping requests, its behavior is clearly incorrect. Let's try to describe why, as this will guide us to a solution.

The problem is this: we can have only a single active Faye subscription handler¹ for each pattern no matter how many messages we send on that pattern (this is by definition, since our design is committed to a single response channel). Each request for a given pattern causes us to overwrite the previous Faye subscription handler for that pattern when we re-issue the fayeClient.subscribe(...) call. Moreover, that active one, with its built-in Observer, notifies every subscriber for that pattern.

¹The generic term "subscription handler" is unfortunately overloaded in our discussion. In the last chapter we introduced our nifty observable subscriber function. Here we're talking about a Faye subscription handler. These are two separate concepts, and we need to try to be very careful with our language. Especially since we have been, and will continue to be, building objects that create a tight relationship between these two! I'll do my best to keep the terminology straight, and you please do too!

Strategy for Solving the Multiplexing Challenge

Let's start working on a strategy to solve this challenge. Given the complexity of the problem, it's helpful to approach a solution starting from a somewhat abstract level, and successively refine our understanding and implementation. We'll walk through the details of handling this in our Faye client component, but some of the details will vary from one transport library (e.g., broker) to the next because each has its own message protocol and API.

If you understand the concepts, there is a certain amount of boilerplate material that you can adapt to a particular transport library. Inevitably, if you need to build a new transporter client, you'll have to work through a few knotholes. In the mean time, pay more attention to the concepts than to understanding every line of code.

The Observable Part

We know ClientProxy#send() returns an Observable. The job of that Observable is to consume incoming Faye messages (boxes in the diagram below) and emit them as a stream (circles in the diagram below) so that our user-land subscriber can deal with them as a nice, well-behaved RxJS Observable.

Figure 1: Convert Messages to Stream

One valuable lesson we learned in the last chapter is that we can use the Observable/observer pattern to essentially "wrap our Faye client inside an Observable". This was our observable subscriber function pattern — implemented in the handleRequest() method (review the source code here) — which builds the observable subscriber function (the bit of code that actually emits values through our Observable). We're going to build on this lesson, and take it to the next level, to deal with the complexities of the multiplexing challenge.

Handling Multiple Requests: The Union of Observables and Correlation Ids

The problem we have to solve is how to associate a unique observable subscriber function with each Observable (i.e., with each send() request that returns that Observable as a response). Furthermore, we need to do this while having only a single active Faye subscription handler for each pattern. This is going to require a little higher order programming. Stick with me — this is the hardest part of the tutorial, but we can get through it.

Let's propose defining our problem as follows: we are binding the observable subscriber function logic to our Faye subscription handler too soon/too statically. Our solution needs to do late/dynamic binding of the observable subscriber function logic. To be precise, it needs to delay binding the the observable subscriber function into the Faye subscription handler until a request is made so it can associate a unique one to each request. This is tough, because there is only a single Faye subscription handler for each pattern. What to do? We have to make a little leap here.

Conceptually, what we're going to do is the following. We'll make our single observable subscriber function logic dynamic by extracting a chunk of what was previously static code and instead have it produced dynamically by a factory function. We'll invoke the factory function for each send() request. The code we extract and produce from the factory will be the code that actually translates inbound messages to Observer emits. We'll call that code the response emitter and the factory that produces it the response emitter factory. We'll see the factory in a moment.

We'll store a unique instance of the response emitter function each time we create an Observable. Later, when we get a response, we'll retrieve that stored function and plug it back into the single static observable subscriber function, giving us a unique observable subscriber function for each request.

Let's get started. The response emitter factory function is actually provided by the framework (it's called createObserver, but we're going to stick with our response emitter factory label in this article):

// from https://github.com/nestjs/nest/blob/master/packages/microservices/client/client-proxy.ts#L82
  protected createObserver<T>(
    observer: Observer<T>,
  ): (packet: WritePacket) => void {
    return ({ err, response, isDisposed }: WritePacket) => {
      if (err) {
        return observer.error(err);
      } else if (response !== undefined && isDisposed) {
        observer.next(response);
        return observer.complete();
      } else if (isDisposed) {
        return observer.complete();
      }
      observer.next(response);
    };
  }

We've gotten quite meta here! Let's take a quick timeout to clarify all our terms!

Faye subscription handler: A callback registered with the Faye client library that is invoked whenever a message with a particular topic is received.

Observable subscriber function: A callback defined during Observable creation to convert application events (e.g., inbound Faye messages) to Observer callbacks. For example, this is the mechanism that enables us to call the user-land handler with the value emitted by our Observer every time an inbound message is received.

Response emitter factory: A higher order function, provided by the framework, that returns a response emitter (defined below).

Response emitter: A function that delivers the dynamic part of our observable subscriber function. It handles the actual Observer emit calls to produce the Observable stream. We will have a unique instance of the response emitter for each Observable, overcoming the multiplexing problem.

Let's head toward implementation. The approach we'll take to implement this goes something like this:

At the time a user-land send() call is made, we'll call the response emitter factory to create a unique instance of the response emitter; we'll also generate a unique identifier* (id field) for the request, and store an association between the unique id and the response emitter in a map.

*At last, we see the genesis of the id field we've been carrying along with our messages all this time!
We'll subscribe to the response channel.
When a response message is received, we'll extract the id from the message, and use it to look up the response emitter. We'll then call this unique instance of the response emitter from within the single shared observable subscriber function, and use this to emit the result. Since the only Observable associated with this response emitter is the originator of the request, that's the only one that receives the emitted result.
To tie these things together, we need to pass the id all the way through the request/response flow. The requesting code generates the id property, it's copied (by the server side code) to the corresponding response message, and finally retrieved again on the corresponding inbound message. Then we use it as described in step 3 above.

The pattern we described in step 3 above is often referred to as using correlation ids (for example, here's a tutorial showing how a similar concept can be used for writing native RabbitMQ client apps).

Other Bookkeeping

One related issue we kind of glossed over is managing the Faye (response channel) subscription process intelligently. We discussed the problem that we are essentially overwriting the Faye subscription handler each time we handle a new request for a given pattern. So an additional consequence of our design goal of sharing a single inbound response channel per pattern is that we we must have only a single active subscription for the response channel, per pattern.

When all active requests for a given pattern/topic have completed, we unsubscribe. When a new request comes in, we subscribe again, and leave the subscription open until the channel quiesces (i.e., there are no more inflight requests) again. We'll need to add some bookkeeping to improve our current behavior (of simply overwriting the handler with each new request on a pattern).

Connection Management

One thing we'll find, as we integrate our code with the framework, is that we need to adhere to its expectations for how we provide a client library connection (i.e., the connection to the broker). Since we're packaging up a bunch of stuff inside observable subscriber functions, the framework expects us to provide access to the connection in a particular (RxJS friendly) way. For the most part, we can just utilize some boilerplate to handle this. There's only a small bit that is specific to a client library. This is all packaged up in a connect() method that we must implement in our ClientFaye class.

Odds and Ends

We'll also see sprinkled in some functions that deal with other loose ends:

we should return channel names using a function, rather than hard coding them with the _ack and _res modifiers to avoid scattering magic strings throughout the code
since user-land patterns can be arbitrary objects (we use strings in these articles, but Nest allows constructs like client.send({cmd: 'get-customers'}, {})), we'll use a built-in inherited function to "normalize" (essentially flatten) the pattern for internal usage

Implementation

With this strategy in mind, here's the outline of how we'll implement it.

Note that there's a fair amount of interaction back and forth between our ClientFaye subclass and the ClientProxy superclass it extends, so pay close attention to that, and I'll try to give clear signposts to guide you.

Unlike our Take 1 version, where we created a new standalone class, we're now going to start by extending the ClientProxy class. This is, of course, how we "plug in" to the framework.
We're going to let the framework take over the implementation of send() for us (at least the shell of the implementation), rather than creating a concrete implementation ourselves, as we did in Part 4. The superclass send() method is now the entry point for processing a user-land send() call, and dictates how we plug in our pieces without disrupting the overall machinery.
The superclass send() method calls upon a concrete implementation of publish(), which is where we'll implement the strategy we've been discussing, dealing with our response emitter factory construct, unique identifiers, and so forth, as described above.
We're going to implement a method to unsubscribe from a Faye topic.
We're going to handle events (ClientProxy#emit()) by providing a concrete implementation of dispatchEvent().
We're going to beef up connection management, as we mentioned above, to enable the framework to efficiently share a Faye client library connection across multiple calls.
We'll take care of a few other minor details.

We have our shopping list, so let's get started!

Take 2 Code Review

The Superclass `send()` Method

Let's begin at a logical place, the send() method. As mentioned, we'll now let the superclass handle this for us, and plug in our code appropriately. Let's take a look:

// From the ClientProxy superclass.  See source code here:
// https://github.com/nestjs/nest/blob/master/packages/microservices/client/client-proxy.ts#L82
  public send<TResult = any, TInput = any>(
    pattern: any,
    data: TInput,
  ): Observable<TResult> {
    if (isNil(pattern) || isNil(data)) {
      return _throw(new InvalidMessageException());
    }
    return defer(async () => this.connect()).pipe(
      mergeMap(
        () =>
          new Observable((observer: Observer<TResult>) => {
            const callback = this.createObserver(observer);
            return this.publish({ pattern, data }, callback);
          }),
      ),
    );
  }

Stripping this method to the bone, here's what it's doing:

Reusing a connection if it exists, otherwise obtaining a new one. Note that since it depends on client library particulars to deal with obtaining a connection, it depends on the connect() method — which we are responsible for implementing and will get to soon.
Creating and returning the Observable that is effectively the "container" within which we'll implement the strategy we discussed above.

Let's look at that Observable creation step for a moment. The first line const callback = this.createObserver(observer) is the call to the response emitter factory, wherein we create our request-specific response emitter.

The second line calls publish(), which is an abstract method on the superclass that we must implement. This is where we'll implement the custom details of our strategy. Let's tackle that next. But first, let's note that send() calls publish() with two parameters:

the arguments from the user-land client.send() call (e.g., if the user wrote client.send('/get-customers', {}), the first argument would contain {pattern: '/get-customers', data: {}}).
the newly minted response emitter

The `ClientFaye#publish()` Method

This is where it all comes together. Let's look at the whole method, then we'll break it down. For now, I suggest you read the comments and lightly scan the code below. We'll dive into the code in a moment.

// nestjs-faye-transporter/src/requestor/clients/faye-client.ts
protected publish(
    partialPacket: ReadPacket,
    callback: (packet: WritePacket) => any,
  ): Function {
    try {
      // prepare the outbound packet, and do other setup steps
      const packet = this.assignPacketId(partialPacket);
      const pattern = this.normalizePattern(partialPacket.pattern);
      const serializedPacket = this.serializer.serialize(packet);
      const responseChannel = this.getResPatternName(pattern);

      // start the Faye subscription handler "bookkeeping"
      let subscriptionsCount =
        this.subscriptionsCount.get(responseChannel) || 0;

      // build the call to publish the request; in addition to making the Faye API
      // `publish()` call, when this function is called, it:
      // * updates the bookkeeping associated with the Faye subscription handler
      //   count
      // * stashes our *response emitter* (called `callback` below) in the map
      const publishRequest = () => {
        subscriptionsCount = this.subscriptionsCount.get(responseChannel) || 0;
        this.subscriptionsCount.set(responseChannel, subscriptionsCount + 1);
        this.routingMap.set(packet.id, callback);
        this.fayeClient.publish(
          this.getAckPatternName(pattern),
          serializedPacket,
        );
      };

      // build the Faye subscription handler
      // this function retrieves the *response emitter*
      // and binds it to the rest of the Faye subscription handler logic
      const subscriptionHandler = this.createSubscriptionHandler(packet);

      // perform Faye `subscribe()` first (if needed), then Faye `publish()`
      if (subscriptionsCount <= 0) {
        const subscription = this.fayeClient.subscribe(
          responseChannel,
          subscriptionHandler,
        );
        subscription.then(() => publishRequest());
      } else {
        publishRequest();
      }

      // remember, this is an *observable subscriber function*, so it returns
      // the unsubscribe function.
      return () => {
        this.unsubscribeFromChannel(responseChannel);
        this.routingMap.delete(packet.id);
      };
    } catch (err) {
      callback({ err });
    }
  }

With the time we put in on the strategy discussion, this code should hopefully make sense, at least at a high level. Let's first dispense with a couple of the minor details so they don't distract, then we can focus on the core functionality.

At the top of the method (remember, this method is called synchronously when a user-land request is made), we prepare the outbound packet (the request message to be published), including assigning the packet id and serializing the packet.
Response channel subscription management should be straightforward to follow. We essentially keep a counter of response channel subscriptions per pattern. We (logically speaking) do a count++ when publishing a request on that channel (pattern), and a count-- when receiving a response from the response channel. This let's us decide whether or not we need to subscribe to the response channel before publishing a request (solving our "we can only have one active subscription per pattern at a time" issue).

The `createSubscriptionHandler` Method: Binding the Response Emitter

Finally, let's talk about the call to createSubscriptionHandler() — first at a high level, and then the details. First, let's recognize that this a factory that returns the actual Faye subscription handler that gets bound to the Faye subscribe() call on the response channel (i.e., <message-pattern>_res). In it, we do the late binding of the observable subscriber function. We do the late binding by looking up the response emitter by id, matching it with the request Observable, and calling it with the destructured inbound message fields. Here's the code:

  public createSubscriptionHandler(packet: ReadPacket & PacketId): Function {
    return (rawPacket: unknown) => {
      const message = this.deserializer.deserialize(rawPacket);
      if (
        message.id &&
        message.id !== (rawPacket as ReadPacket & PacketId).id
      ) {
        return undefined;
      }
      const { err, response, isDisposed, id } = message;

      const callback = this.routingMap.get(id);
      if (!callback) {
        return undefined;
      }

      if (isDisposed || err) {
        return callback({
          err,
          response,
          isDisposed: true,
        });
      }
      callback({
        err,
        response,
      });
    };
  }

Let's explore a little further. The function returned by this factory, as we said, is the Faye subscription handler. As such, it has only one argument (as dictated by the Faye API) — an actual Faye inbound message. Here are the steps the returned function performs:

Deserialize the packet.
Destructure the message so we can deal with its constituent values: err, response, isDisposed and id
Use the map to lookup the correct response emitter
Discard any messages which are not destined for this client¹
Finally, return the actual observable subscriber function. Note: we automatically add isDisposed: true if there's an error, to force the closure of the Observable.

¹This is the code fragment:
if (!callback) {
return undefined;
}

We take up discussion of this snippet in its own section immediately below.

With this, we've completed the process of converting to a late binding of the observable subscriber function! We have a couple more details to work through next.

Discarding Messages for Other Clients

Consider that in a distributed microservice-based architecture, we may have multiple clients (e.g., instances of nestHttpApp or other requestor apps), connected via the broker, to the same responder (e.g., nestMicroservice). In such a configuration, each client may issue the same requests (i.e., utilize the same message pattern). When that happens, we'll have multiple clients communicating using the same Faye topic. This is, of course, completely natural for Faye (and any message broker). However, since we may have multiple Faye subscribers on the same pattern/topic (e.g., '/get-customers'), each client (each subscriber) will be notified with any response message that matches the response topic. Only those that come from the originating client will have a matching id and response emitter. We can surmise that non-matching responses are properly destined to be handled by other client instances. We can simply ignore these, returning undefined to our Faye subscription handler, which is essentially a no-op.

Loose Ends

Back in our massive discussion of the publish() method, we briefly mentioned unsubscribing our Faye subscription handler for a particular message pattern after the response channel quiesces.

Let's talk about Observable life-cycle hooks briefly. While it's possible for us to unsubscribe from an Observable in user-land (a subscribe() call on an Observable returns a function to unsubscribe), that's rarely needed. The RxJS Observer takes care of this automatically for us when the final event in a stream is emitted, and has a hook for any further cleanup we need to do. In Nest transporter-land, this life-cycle event happens when we receive the last message in a stream, with the isDisposed property set to true.

From the Observable unsubscribe hook, we can run any cleanup code we need upon stream completion, such as determining whether to unsubscribe our Faye subscription handler. To do this, we call our custom cleanup code from that unsubscribe hook. Remember, this is the function returned from the observable subscriber function, so we can expect to find it back in our publish() method. Let's take a look. Note the function returned at the bottom of the publish() method body. That's our unsubscribe hook. From it, we can do any cleanup, as shown below:

// nestjs-faye-transporter/src/requestor/clients/faye-client.ts
      ...
      return () => {
        this.unsubscribeFromChannel(responseChannel);
        this.routingMap.delete(packet.id);
      };
      ...

We do two important things at the termination of each message stream:

We decide whether to unsubscribe for messages on that topic on the Faye response channel
We delete the relevant response emitter function from the map

Deciding whether to unsubscribe requires us to revisit the subscription management topic. You should be able to connect the dots by looking at the unsubscribeFromChannel() method shown below, and the discussion we had earlier about subscription bookkeeping. Basically, we unsubscribe when the count of in-flight requests for a given pattern/topic reaches 0.

  protected unsubscribeFromChannel(channel: string) {
    const subscriptionCount = this.subscriptionsCount.get(channel);
    this.subscriptionsCount.set(channel, subscriptionCount - 1);

    if (subscriptionCount - 1 <= 0) {
      this.fayeClient.unsubscribe(channel);
    }
  }

Connection Management

The framework follows a specific protocol for accessing the broker client library connection object. This protocol deals with connection establishment and connection errors as Observable events. Rather than review this code in detail, we can safely say that it is mostly boilerplate. Let's call out the parts that are specific to Faye.

We need to implement a connect() method, which is tied to a class member called connection. Let's take a look:

  public async connect(): Promise<any> {
    if (this.fayeClient) {
      return this.connection;
    }
    const { url, serializer, deserializer, ...options } = this.options;
    this.fayeClient = new faye.Client(url, options);
    this.fayeClient.connect();
    this.connection = await this.connect$(
      this.fayeClient,
      ERROR_EVENT,
      CONNECT_EVENT,
    )
      .pipe(share())
      .toPromise();
    this.handleError(this.fayeClient);
    return this.connection;
  }

We populate this.connection by first generating a live connection using the the library-specific protocol. For Faye, this requires instantiating a Client object, then calling connect().

Once we have a connection, we run this through the framework by calling this.connect$() as shown above. This method takes the connection object (this.fayeClient), and constants defining the events the client library defines for "success" and "failure". In the case of Faye, we define them in the file nestjs-faye-transporter/src/constants.ts, and import those constants to this (the faye-client.ts) file.

In this manner, the framework uses a connection if it exists, or creates one if needed, and handles the connection lifecycle events in a uniform and efficient way.

The call to this.handleError() is optional, and let's us emit a log message in the event the connection fails.

As mentioned, most of this is boilerplate; if you're integrating another broker, you can mostly just plug in the client library-specific calls where indicated for the Faye library calls above.

Event Handling

The final feature to implement is event handling — handling user-land client.emit(...) calls. As we found on the server side in Part 3, this is far simpler than dealing with request/response style messaging because we are dealing with one-way messages: we simply send a message over the outbound (request) channel, and we're done. No subscribing to responses necessary.

Because this is straightforward and follows a predictable pattern, the framework handles most of this for us in the ClientProxy#emit method. Here's what that code looks like:

// from https://github.com/nestjs/nest/blob/master/packages/microservices/client/client-proxy.ts
  public emit<TResult = any, TInput = any>(
    pattern: any,
    data: TInput,
  ): Observable<TResult> {
    if (isNil(pattern) || isNil(data)) {
      return _throw(new InvalidMessageException());
    }
    const source = defer(async () => this.connect()).pipe(
      mergeMap(() => this.dispatchEvent({ pattern, data })),
      publish(),
    );
    (source as ConnectableObservable<TResult>).connect();
    return source;
  }

Here, once again, the framework is handling connection management for us, and all we really need to do is provide a concrete implementation for dispatchEvent(). The superclass defines an abstract dispatchEvent method as follows:

// from https://github.com/nestjs/nest/blob/master/packages/microservices/client/client-proxy.ts
protected abstract dispatchEvent<T = any>(packet: ReadPacket): Promise<T>;

So our implementation simply needs to accept a request packet and publish the appropriate message using the Faye client API. We need to wrap that in a promise and return the promise to match the signature above.

Here's how we implement the dispatchEvent() method in our Faye client component:

  protected dispatchEvent(packet: ReadPacket): Promise<any> {
    const pattern = this.normalizePattern(packet.pattern);
    const serializedPacket = this.serializer.serialize(packet);

    return new Promise((resolve, reject) =>
      this.fayeClient.publish(pattern, serializedPacket),
    );
  }

The logic should be easy to follow. We simply extract the pattern, normalize it, serialize the outbound packet (no need for an id on this one), and then return a promise that resolves to the results of the fayeClient.publish() call. The framework handles the rest (which is really very little, other than efficient connection management).

Acceptance Testing

At this point, we should have a complete implementation of the Faye Custom Transporter! 🍺 🍺 🍺!

To test it, you'll want to start up the nestHttpApp and nestMicroservice, and run any of the routes. See these notes for more help in running the tests.

Conclusion

We've come a long way! Thanks for hanging in there so long! I hope you found this series useful. Please hit me up in the comments with any questions!

For those of you braving the task of writing your own Custom Transporter, please share your experiences in the comments.

For those just interested in a deeper understanding of this key aspect of Nest Microservices, I hope this was helpful, and feel free to share your thoughts or questions in the comments!

What's Next

I have a sixth article planned, in which I'll compare several of Nest's built-in transporter implementations to provide further insight into the nuances of each implementation. I'll gauge from the responses to this series whether that's something people would find useful, so please comment below to encourage me to write that final article! 😄

Feel free to ask questions, make comments or suggestions, or just say hello in the comments below. And join us at Discord for more happy discussions about NestJS. I post there as Y Prospect.

Part 4: Basic Client Component

John Biundo — Wed, 04 Mar 2020 14:06:40 +0000

John is a member of the NestJS core team

Introduction

This is Part 4 of a six-part series. If you landed here from Google, you may want to start with Part 1.

In this article, we build the first iteration of the client component of the Faye Custom Transporter.

Get the Code

All of the code in these articles is available here. As always, these tutorials work best if you follow along with the code. The README covers all the steps you need to get the repository, build the apps, and follow along. It's easy! I strongly encourage you to do so. Note that each article has a corresponding branch in the repository. For example, this article (Part 4), has a corresponding branch called part4. Read on, or get more details here on using the repository.

Git checkout the current version

For this article, you should git checkout the branch part4. You can get more information about the git repository branches here.

Build the Apps for This Part

$ # from root directory of project (e.g., transporter-tutorial, or whatever you chose as the root)
$ sh build.sh

Overview

It's time to turn our attention to the client component. But let's start by recalling that we spent a lot of effort ensuring that the server component can return an observable stream of results to a requestor. Let's have a quick visual to cement that idea, since we'll start right out with consuming that stream on the client side.

Figure 1: Remote Observable Stream

What's Next

In this animated sequence, our nestMicroservice responder app responds to a request message (ClientProxy#send) by emitting a stream of three results (circles). We saw in the last article how our transporter server component converts that stream to a sequence of 3 messages (boxes) to the broker, sending those messages on the response channel. Our first meaningful task with the client component will be to consume those broker messages and convert them back into an Observable stream (circles) in the ClientProxy layer so that our user-land controllers and services can use them as regular RxJS observables.

In this iteration (Take 1), we'll take a similar approach to our first iteration of the server. We'll get a basic functioning client working so we can focus on its responsibilities and the main code path. Later, we'll add more robustness. As with the server side, we'll ignore events (the ClientProxy#emit path) for now.

First Iteration (Take 1) of the Client Component

Let's get started. First, let's look at how someone would use the client component of our Faye Custom Transporter. Open the app.controller.ts file for our nestHttpApp.

// nestHttpApp/src/app.controller.ts
import {
  Controller,
  Get,
  Logger,
  Param,
  Response,
  Post,
  Body,
} from '@nestjs/common';

import {
  ClientFaye,
  InboundResponseIdentityDeserializer,
  OutboundMessageIdentitySerializer,
} from '@faye-tut/nestjs-faye-transporter';


import { tap, reduce, filter } from 'rxjs/operators';

@Controller()
export class AppController {
  logger = new Logger('AppController');
  client: ClientFaye;

  constructor() {
    this.client = new ClientFaye({
      url: 'http://localhost:8000/faye',
      serializer: new OutboundMessageIdentitySerializer(),
      deserializer: new InboundResponseIdentityDeserializer(),
    });
  }

We simply instantiate the client in the class constructor. We could also use constructor injection. Note: one thing we can not currently do is use the @Client() decorator. That's not really a significant limitation, as the decorator is merely a convenience, and is not the preferred way of instantiating a ClientProxy.

The JSON options object we pass to ClientFaye should be quite similar to the options object we support on the server side, as it is either configuring the Faye connection, or passing some generic transporter options, like serializer.

Once we instantiate a client like this, we can use it just like a ClientProxy object to make requests and emit events. Let's take a look at one such request. Further down in the app.controller.ts file, notice this call. Here, we make a call to client.send() and — since, in this case, we want to manipulate the entire stream of results (what we do with those results isn't important here, we're just showing that we are wanting to access the stream) — we pipe that stream through the RxJS tap() operator.

// nestHttpApp/src/app.controller.ts
@Get('jobs-stream1/:duration')
  stream(@Param('duration') duration) {
    return this.client.send('/jobs-stream1', duration).pipe(
      // do notification
      tap(step => {
        this.notify(step);
      }),
    );
  }

Take 1 Requirements

The request above is the one represented in the animation at the top of the article. Feel free to take a look at the server-side code (nestMicroservice/src/app.controller.ts) to get familiar with what's happening here. You can run this now by issuing an HTTP request to the nestHttpClient app as shown below. Assuming you've got your multi-terminal setup going, you'll be able to follow the full flow of this request and response through all of the components.

$ # HTTPie command to request `/jobs-stream1` with base duration of 1
$ http get localhost:3000/jobs-stream1/1

You can also read the deep dive on Observables for a thorough treatment of the elegance of streaming server side observable results back to the client. For now, all we need to know is what we see in the animation above: when we make this request, we get a stream of results in the form of a sequence of messages.

Let's define our acceptance criteria for Take 1 as follows. Our ClientFaye object should let us send the '/jobs-stream1' request, and handle the response, converting the message stream back into an Observable.

In part5 we'll complete the implementation and have a fully functioning Faye Custom Transporter — both client and server side.

Take 1 Strategy

Before diving into the code, let's review our overall strategy. At a top level, it's pretty simple. For each user-land request (e.g., the this.client.send(...) call in the app.controller.ts file shown above), we want to issue the remote request and wait for the results, converting the response message(s) into an Observable stream.

Issuing the request and waiting for the results using the broker API is familiar territory. We'll just implement our very familiar STRPTQ (subscribe-to-the-response-then-publish-the-request) pattern to accomplish this.

How do we convert the broker message stream into an Observable stream? We can use the Observable/observer pattern to do this. While I won't go into a lot of detail (you can read more, including simple examples, here), the basic idea is simple, and is the primary use case for Observables. The easiest way to see this is through the code review below.

Take 1 Code Review

Let's dive into the code. Open the file nestjs-faye-transporter/src/requestor/clients/faye-client.ts.

You might notice that in this iteration, we're not actually extending any framework class. We'll need to do that in the next iteration, but it makes our code smaller and easier to focus on the core logic if we omit that in this step and just define a new class.

Class Definition

Let's start with the class members and constructor:

// nestjs-faye-transporter/src/requestor/clients/faye-client.s
export class ClientFaye {
  private readonly serializer;
  private readonly deserializer;
  protected fayeClient;

  constructor(protected readonly options) {
    this.fayeClient = this.connect();
    this.serializer = options.serializer || new IdentitySerializer();
    this.deserializer =
      options.deserializer || new IncomingResponseDeserializer();
  }

We should be familiar with the serializer and deserializer properties, and how they're initialized, from the server side. The fayeClient holds our connection to the Faye broker and is the object we'll use to make Faye client API calls. The constructor just sets this stuff up. It defaults to using the built-in (essentially no-op) serializer and deserializer if the user doesn't provide one in the options.

Implementing `send()`

With that covered, we can look at the API our class exposes. First, recall that the user-land send() call we need to support is:

// nestHttpApp/src/app.controller.ts
@Get('jobs-stream1/:duration')
  stream(@Param('duration') duration) {
    return this.client.send('/jobs-stream1', duration).pipe(
      // do notification
      tap(step => {
        this.notify(step);
      }),
    );
  }

We expose this feature, of course, with the send() method right at the top of the faye-client.ts file, as shown here:

// nestjs-faye-transporter/src/requestor/clients/faye-client.ts
  public send(pattern, data): Observable<any> {
    return new Observable(observer => {
      return this.handleRequest({ pattern, data }, observer);
    });
  }

The `send()` Method Returns an Observable

As required, send() returns an Observable. We need to delve into Observable-land here for a while. This can be confusing/frustrating if you're not familiar with the topic, but by the time you're finished here, I think the 💡 will go on.

In the send() method body, we construct an Observable using the normal Observable creation pattern¹ and return it to user-land. Applying the normal Observable creation pattern in our context means we pass the constructor a function that accesses the Faye message stream and emits the values through our Observable. Traditionally, this callback is called the observable subscriber function.

Simple Observables (the ones you'll find in a lot of Medium articles 😉) often define this observable subscriber function body right in place. For example:

return new Observable(observer => {
  observer.next('hello');
  observer.next('world');
  observer.complete();
});

But we already know that ours is going to be a little complex because the source of the data we are going to emit comes from the Faye broker. Specifically, it comes from the callback we're going to register in the Faye client API subscribe() call which is invoked each time a message comes in on the response channel. That code — which listens through Faye for inbound messages — is the code that needs to run our observer.next() (or equivalent) function call.

In these more complex scenarios, to keep the code clean, we delegate this sophisticated observer subscriber function to a specialized function. That's what we're doing in the code above by calling handleRequest().

For the moment, you need to know two things:

handleRequest() is our observable subscriber function
handleRequest() is going to make a Faye client API call to subscribe to the response topic, causing it to register to receive future inbound requests. As it receives them, it's going to emit the results in an Observable stream, analogous to our observer.next('hello') statement above. We'll see that code in a minute, but let's keep it abstract for another minute.

This can all seem kind of strange if you're not familiar with the Observable pattern¹. Let's see if it helps to think about it this way. We're setting up callbacks at several levels in a sort of cascade. At the "innermost" level, we have the Faye subscription handler callback that will run when we get an inbound message on the response channel. When this handler runs, we want to in turn trigger a callback through the Observer mechanism to a higher-level callback — the user-land response handler (the code that looks like pipe(tap(step) => {...}) in the app.controller.ts file, shown above).

Here are the connections you need to make:

inner callback (broker) called when inbound message received...
this triggers calling the observable subscription function...
which in turn calls back to our user-land client.send(...) subscription handler

That's what the send() method is setting up with the Observable it creates. One more thing about Observables to keep in mind. What we just described effectively "sets up the cascade of callbacks", but these functions are cold in the sense that all we've done is register callbacks. The act of subscribing to the Observable kicks things in motion. The act of subscribing is the event that triggers the observable subscriber function to run; when you think about that, you realize that "Aha! So this in turn causes the Faye API subscribe() call inside handleRequest() to run", and we see that this is how we start the stream flowing and turn the process from cold to hot.

Another thing to know is that if the user-land client.send() call doesn't explicitly subscribe, but instead just returns the result, Nest automatically subscribes to the result. As a ClientProxy developer, you don't have to worry about this. But as a quick aside, note that this behavior is useful as a default, and works great in most situations.

Now if you've clicked ahead to the next question and you're wondering how does Nest return a potential stream of results over HTTP? The answer is that it returns the last result in the stream. Again, in many cases this is fine. In our user-land client.send(...) call (shown again below), however, we want to access the values in the stream, thus we add a pipe() operator that allows us to operate on the stream itself, once it's subscribed:

// nestHttpApp/src/app.controller.ts
@Get('jobs-stream1/:duration')
  stream(@Param('duration') duration) {
    return this.client.send('/jobs-stream1', duration).pipe(
      // do notification
      tap(step => {
        this.notify(step);
      }),
    );
  }

If this still isn't completely intuitive, I feel your pain, but I advise you to just let it marinate in the background and proceed. Trust me, you can still follow most of this even if this still feels strange. If you really must scratch that itch, see the footnote below.

¹I'm assuming familiarity with this pattern. If you're an RxJS junky, this should be easy to follow. If you're not, I'll try not to wave my hands too much. I too struggled with this concept for a while, so I've created a small side excursion here to help with this concept. This should take you no more than 5-10 minutes to work through, and might be a helpful pre-requisite if some of this section has you a little boggled.

So let's dig into that requestHandler() method.

The Request Handler

The handleRequest() method is where we marry up our Observable pattern with our request/response handling pattern. It's easiest if we decompose this and look at a version that abstracts out the Observable code first. Then we'll see how it all wires together.

In the first pseudocode-ish version below, we can focus directly on the request/response handling pattern. You'll quickly recognize this as our old friend the STRPTQ pattern. (As a refresher, feel free to review how we implemented this functionality in our native Faye customerApp way back in Episode 1. Here's a link to that code. Or just open up customerApp/src/customer-app.ts).

With that in mind, the pseudocode-ish version of handleRequest() below should be pretty clear. I'll make a few explanatory comments below the code sample.

  public handleRequest(partialPacket, observer): Function {
    const packet = Object.assign(partialPacket, { id: uuid() });
    const serializedPacket = this.serializer.serialize(packet);

    const requestChannel = `${packet.pattern}_ack`;
    const responseChannel = `${packet.pattern}_res`;

    const subscriptionHandler = ... /* we'll fill this in shortly!  This function
      will provide the code we want to plug in as our *Faye subscription handler*.
      Guess what?  This is where we'll wire in our Observable :)
    */

    // Execute the familiar subscribe-to-response-then-publish-the-request pattern
    const subscription = this.fayeClient.subscribe(
      responseChannel,
      subscriptionHandler,
    );

    subscription.then(() => {
      this.fayeClient.publish(requestChannel, serializedPacket);
    });

    /*
      Handle any other Observable requirements
    */
  }

A few other notes from the code:

The partialPacket parameter has most of the elements of our outbound request message. It's been invoked from send() with an object like:
```
{
  pattern: '/jobs-stream1/',
  data: 1
}
```
We complete the formation of our outbound packet by using uuid() to add that pesky id parameter (that we still haven't talked about — I promise to address this in Part 5!).
After running our serializer, the request packet is ready to send.
In Faye, the subscribe() call returns a promise that resolves when the Faye server returns an acknowledgement.
Upon that acknowledgement (promise resolution), we publish the serialized packet we built earlier.

Now let's deal with marrying that up with our Observable. Here's the final version of handleRequest().

  public handleRequest(partialPacket, observer): Function {
    const packet = Object.assign(partialPacket, { id: uuid() });
    const serializedPacket = this.serializer.serialize(packet);

    const requestChannel = `${packet.pattern}_ack`;
    const responseChannel = `${packet.pattern}_res`;

    const subscriptionHandler = rawPacket => {
      const message = this.deserializer.deserialize(rawPacket);
      const { err, response, isDisposed } = message;
      if (err) {
        return observer.error(err);
      } else if (response !== undefined && isDisposed) {
        observer.next(response);
        return observer.complete();
      } else if (isDisposed) {
        return observer.complete();
      }
      observer.next(response);
    };

    const subscription = this.fayeClient.subscribe(
      responseChannel,
      subscriptionHandler,
    );

    subscription.then(() => {
      this.fayeClient.publish(requestChannel, serializedPacket);
    });

    return () => {
      this.fayeClient.unsubscribe(responseChannel);
    };
  }

Let's focus in on the newly added part:

    const subscriptionHandler = rawPacket => {
      const message = this.deserializer.deserialize(rawPacket);
      const { err, response, isDisposed } = message;
      if (err) {
        return observer.error(err);
      } else if (response !== undefined && isDisposed) {
        observer.next(response);
        return observer.complete();
      } else if (isDisposed) {
        return observer.complete();
      }
      observer.next(response);
    };

Let's review the purpose of this chunk of code. This is (in terms of the concepts we started out with), our inner callback — the function that will be passed in our Faye client API subscribe() call to handle inbound messages. As we handle each message in this function, this code converts the message back into a value in the Observable stream. We do that by taking the following steps for each inbound message received from Faye:

Deserialize it and destructure it, leaving us with an error, response and isDisposed value for each inbound message
Based on the contents of the message, determine if the inbound message stream has errored or is complete
Emit an Observable value, based on step 2

As in any normal observable subscriber function, we handle these cases with the observer.error(), observer.next() and observer.complete() calls. If this confuses you, remember that all of this plugs into the Observable we created in our send() method. You might take a minute to grok all of this, and if it's still a little fuzzy, you can visit my mini Observable factory tutorial here.

Handling Unsubscribe

Any observable subscriber function must return a way to unsubscribe from the Observable. This is part of the Observable creation pattern. The unsubscribe method can be called by the user; more often, it is called automatically when the Observable stream ends (e.g., when the final message is returned from our remote responder).

The last bit of code in handleRequest() fulfills this contract for our observable subscriber function:

    return () => {
      this.fayeClient.unsubscribe(responseChannel);
    };

The net effect of this is that when the stream completes (e.g., when the inbound message has isDisposed set to true), we unsubscribe from the response channel.

The Rest of the Code

The only other thing left is the connection management to the Faye broker. That's handled in the connect() method, which for now we're calling in the constructor. This should be pretty understandable. The only slightly funky thing is that we use destructuring to extract out the url and the options object from the constructor options so we can easily format our call to fayeClient.connect().

Connection management will become more important and consequently more sophisticated in our next iteration.

  public connect() {
    if (this.fayeClient) {
      return this.fayeClient;
    }
    const { url, serializer, deserializer, ...options } = this.options;
    this.fayeClient = new faye.Client(url, options);
    this.fayeClient.connect();
    return this.fayeClient;
  }

Acceptance Testing

We're ready to test our code. In previous articles, I gave a lot of suggestions about managing your multiple terminals. Going forward, I'll simply indicate the steps you should take, and assume you've got a working environment with the appropriate logical terminals set up. Here are the steps to run our test:

Make sure our Faye server is up and running in one terminal.
Make sure the nestMicroservice app is running in one terminal.
Make sure the nestHttpApp app is running in one terminal.

Now, issue a request (expressed below as an HTTPie request, but use your favorite HTTP testing tool):

$ # HTTPie request to get the jobs stream with a base duration of 1
$ http get localhost:3000/jobs-stream1/1

If all is working, you should get a coordinated set of logs in each of the terminal windows indicating that we have, indeed, successfully subscribed to the results of our remote request. You'll see our RxJS pipe come into play, making simulated calls to our notifier as each step in the job completes (as each message in the stream arrives). 🍺!

Limitations of Take 1

We knowingly deferred a few things to Take 2 to keep focused on the core requirements and implementation of a client. We'll address them in Part 5. But there's one big issue lurking. Any guesses on what that might be? I've definitely foreshadowed this along the way (hint: repeatedly deferring the discussion of that pesky message id).

OK, let's get right to it. Ask yourself this: since our client is embedded in an HTTP app that is itself responding to multiple incoming requests, many of which in turn are triggering these client.send(...) remote requests, how does the transporter client keep track of all of the outstanding remote requests and route them back to the correct originating (HTTP) request handler (i.e., the code with the original client.send() call?

Let's try it out. You'll need one extra terminal window for this to enable launching two simultaneous requests. You'll notice that in this branch, there is an extra route in nestHttpApp. That route makes a new request ('/race'), which has a corresponding @MessagePattern() in nestMicroservice. The point of these new features is to enable us to launch multiple requests that overlap in their execution. Feel free to take a look at the code (these new routes/handlers are at the bottom of their respective Controller files), but all you really need to know is this: the race route in our nestHttpApp takes two parameters:

a requestId
a delay period (in seconds) specifying how long the server should wait before responding

For example, try making the following request (HTTPie format):

$ http get localhost:3000/race/1/10

This will return the following response after 10 seconds (cid in the response corresponds to the first route parameter, and delay in the response corresponds to the second parameter (expressed in milliseconds)):

{
    "cid": "1",
    "customers": [
        {
            "id": 1,
            "name": "fake"
        }
    ],
    "delay": 10000
}

With this in place, we can run two overlapping requests. Let's go ahead and run the following two requests. You'll need to do this in two separate terminals, and you should launch the second request within 5 seconds of launching the first (to achieve the desired overlap):

$ # run this in terminal 1, and be prepared to run the second one in terminal 2 a few seconds later
$ http get localhost:3000/race/1/10
$ # run this one in terminal 2, within a couple of seconds of running the top 1
$ http get localhost:3000/race/2/5

The results may (well, maybe not with all the foreshadowing 😉) surprise you. Both requests return the following result:

{
  "cid": "2",
  "customers": [
      {
          "id": 1,
          "name": "fake"
      }
  ],
  "delay": 5000
}

This is obviously incorrect for the first request. The issue demonstrates what I'll refer to as the multiplexing challenge. That is, we are multiplexing our requests for each pattern through a single pair of channels. When we issue request #1, it subscribes to a response on the '/race_res' channel. Request #2 does the same thing, and since request #2 finishes first, the request #1 subscription gets request #2's result (the wrong answer!) on the response channel.

Also, there's another nagging question... why do both subscribers get the result? Hmmm... clearly something isn't working quite right here. We'll have to dig in and work on this in our next (and final!) revision!

What's Next

With these issues in mind, we're ready to build a final version of the Faye Custom Transporter client component, completing our development project. In Part 5 we cover:

Solving the multiplexing issue
Client library connection management
Event handling on the client side

Feel free to ask questions, make comments or suggestions, or just say hello in the comments below. And join us at Discord for more happy discussions about NestJS. I post there as Y Prospect.

Part 3: Completing the Server Component

John Biundo — Mon, 02 Mar 2020 18:02:41 +0000

John is a member of the NestJS core team

Introduction

This is Part 3 of a six-part series. If you landed here from Google, you may want to start with Part 1.

In this article, we build the final iteration of the server component of our Faye Custom Transporter.

Get the Code

All of the code in these articles is available here. As always, these tutorials work best if you follow along with the code. The README covers all the steps you need to get the repository, build the apps, and follow along. It's easy! I strongly encourage you to do so. Note that each article has a corresponding branch in the repository. For example, this article (Part 3), has a corresponding branch called part3. Read on, or get more details here on using the repository.

Git checkout the current version

For this article, you should git checkout the branch part3. You can get more information about the git repository branches here.

Build the Apps for This Part

$ # from root directory of project (e.g., transporter-tutorial, or whatever you chose as the root)
$ sh build.sh

Overview

I ended the last article with a discussion of the shortcomings of our Take 1 (first iteration) implementation of the Faye Custom Transporter server component. In this article, we'll address those shortcomings. Most of this work is fairly straightforward, but I want to take a moment to discuss the "Observable issue" a bit further.

You could actually ignore this issue entirely. The server component we built in the last chapter is functional, and we'll clean up a few loose ends here to make it typesafe, etc. However, by ignoring proper handling of Observables, you'd be leaving out a feature that you may not miss until you really need it. Plus, let me hit you with the full commercial:

It's super fun to understand how it works, and if you're not an RxJS whiz, don't worry, that won't stop you (and you might even pick up a few nifty tricks about RxJS along the way)
It builds a deeper appreciation and understanding of the elegance of the NestJS framework
It's really, really easy to implement, once you get on the happy path
The feature provides some amazing benefits, virtually for free
Your transporter really should be plug-and-play compatible with other transporters to be a good Nest citizen (Nestizen? 😃)

I could go on and on, but instead, I've encapsulated all the concrete examples and detailed analysis here to try to keep this tutorial on track. Suffice to say, I strongly encourage you to follow that link and take that little side trip to both understand why this is an important step — and because you might even get inspired to use Nest microservices in some new and interesting ways.

With all that said, let's jump in!

Addressing Observables

After that build-up, you might think we're about to embark on some form of brain surgery (maybe I should say "🚀 science" in keeping with the trekkie theme? 😉). Fortunately, that's not really the case. Let's start by considering the requirements.

At the highest level, we need to consider the possibility that the user-land handler returns a stream (Observable that emits multiple values). Our earlier version won't handle that (as we saw at the end of the last article). The problem lies with our response message publishing logic being unaware of a response stream. It simply awaits a single response and publishes. Let's take a quick look at the part of the code that publishes to refresh ourselves:

// nestjs-faye-transporter/src/responder/transporters/server-faye.ts
// on branch part2
public getMessageHandler(pattern: string, handler: Function): Function {
  return async message => {
    const inboundPacket = this.deserializer.deserialize(message);
    const response = await handler(inboundPacket.data);
    const outboundRawPacket = {
      err: null,
      response,
      isDisposed: true,
      id: (message as any).id,
    };
    const outboundPacket = this.serializer.serialize(outboundRawPacket);
    this.fayeClient.publish(`${pattern}_res`, outboundPacket);
  };
}

We could try to handle this streaming ourselves, but it's a tad tricky bit of code, and as such, a great candidate for the framework to handle for us. Which it does! As always with frameworks, if we want to delegate a responsibility, we have to invert our thinking a bit. Rather than publishing directly, we instead need to build a publish function and hand it to the framework

Now, here's the new, improved, Observable-aware version. This is what you'll see now (on the current branch, part3) when you open the nestjs-faye-transporter/src/responder/transporters/server-faye.ts file:

//nestjs-faye-transporter/src/responder/transporters/server-faye.ts
// on branch part3
public getMessageHandler(pattern: string, handler: Function): Function {
  return async (message: ReadPacket) => {
    const inboundPacket = this.deserializer.deserialize(message);
    const fayeCtx = new FayeContext([pattern]);

    const response$ = this.transformToObservable(
      await handler(inboundPacket.data, fayeCtx),
    ) as Observable<any>;

    const publish = (response: any) => {
      Object.assign(response, { id: (message as any).id });
      const outgoingResponse = this.serializer.serialize(response);
      return this.fayeClient.publish(`${pattern}_res`, outgoingResponse);
    };

    response$ && this.send(response$, publish);
  };
}

Let's discuss the differences. For now, ignore the fayeCtx object — we'll cover that below.

The reorganized code does two main things:

It converts our handler response to an Observable
It builds a publish function that it can delegate to the framework (which will then use it to publish a stream if that's what the user-land handler returns)

For the first item, we call a built-in (inherited from Server) utility to convert the response to an Observable.

const response$ = this.transformToObservable(
  await handler(inboundPacket.data, {}),
) as Observable<any>;

This step is pretty intuitive, and the utility is correspondingly pretty simple. Feel free to take a moment to review the transformToObservable() method's source code.

Building the publish function is also fairly straightforward.

const publish = (response: any) => {
  Object.assign(response, { id: (message as any).id });
  const outgoingResponse = this.serializer.serialize(response);
  return this.fayeClient.publish(`${pattern}_res`, outgoingResponse);
};

This code is very similar to the way we published in the Take 1 version (from branch part2), but now packaged as a function we can hand off to the framework. Study it for a moment and make sure you understand it — it should be pretty straightforward. The Object.assign(...) code's job is to copy the id field from the inbound request to the outbound response. We did that in the earlier version as well, but there we were defining the response object in place, so the technique was a little different.

Note: We still haven't discussed why we need that id. For now, just trust the process 😃. Don't worry, we'll cover this in Part 5.

Finally, we delegate the publishing step to the framework with the final line:

response$ && this.send(response$, publish);

The send() method is inherited from Server; it takes care of the details of dealing with streams.

The significance of the previous sentence should not be overlooked. To fully appreciate both why this is so useful and how the framework makes this as easy as delegating a publish() function, you really should read this deep dive.

Anyway, let's discuss what the framework is doing for us. As mentioned, send() is inherited from Server. See here for the full source code of the Server. Let's reproduce it below to get a feel for it.

Note: we don't need to fully understand how this method works, and I won't belabor it here. It's fair to treat it as a black box, as long as we understand how to use it. We'll make sure we do!

// from https://github.com/nestjs/nest/blob/master/packages/microservices/server/server.ts
  public send(
    stream$: Observable<any>,
    respond: (data: WritePacket) => void,
  ): Subscription {
    let dataBuffer: WritePacket[] = null;
    const scheduleOnNextTick = (data: WritePacket) => {
      if (!dataBuffer) {
        dataBuffer = [data];
        process.nextTick(() => {
          dataBuffer.forEach(buffer => respond(buffer));
          dataBuffer = null;
        });
      } else if (!data.isDisposed) {
        dataBuffer = dataBuffer.concat(data);
      } else {
        dataBuffer[dataBuffer.length - 1].isDisposed = data.isDisposed;
      }
    };
    return stream$
      .pipe(
        catchError((err: any) => {
          scheduleOnNextTick({ err, response: null });
          return empty;
        }),
        finalize(() => scheduleOnNextTick({ isDisposed: true })),
      )
      .subscribe((response: any) =>
        scheduleOnNextTick({ err: null, response }),
      );
  }

While at first a little obscure, this really isn't that hard to understand if you work through it, though it does require some understanding of observables. Essentially, we are subscribing to the response coming back from our user-land handler. If the subscription produces multiple values (a stream), we buffer them (dataBuffer), then publish each value using the publish() function we built a moment ago (inside this method the publish() function is accessed as the respond parameter). When the stream completes, we signify that by setting isDisposed to true on the final emitted response.

Handle Events

Thus far we've skipped handling events — those user-written handlers that are decorated with @EventPattern(...). Let's take care of those now. They turn out to be a lot easier than request/response type messages precisely because they don't require any response. Take a look at the bindHandlers() method of server-faye.ts. The comments should pretty much explain the simple change we've made here to handle events.

// nestjs-faye-transporter/src/responder/transporters/server-faye.ts
public bindHandlers() {
  /**
   * messageHandlers is populated by the Framework (on the `Server` superclass)
   *
   * It's a map of `pattern` -> `handler` key/value pairs
   * `handler` is the handler function in the user's controller class, decorated
   * by `@MessageHandler()` or `@EventHandler`, along with an additional boolean
   * property indicating its Nest pattern type: event or message (i.e.,
   * request/response)
   */
  this.messageHandlers.forEach((handler, pattern) => {
    // In this version (`part3`) we add the handler for events
    if (handler.isEventHandler) {
      // The only thing we need to do in the Faye subscription callback for
      // an event, since it doesn't return any data to the caller, is read
      // and decode the request, pass the inbound payload to the user-land
      // handler, and await its completion.  There's no response handling,
      // hence we don't need all the complexity of `getMessageHandler()`
      this.fayeClient.subscribe(pattern, async (rawPacket: ReadPacket) => {
        const fayeCtx = new FayeContext([pattern]);
        const packet = this.parsePacket(rawPacket);
        const message = this.deserializer.deserialize(packet, {
          channel: pattern,
        });
        await handler(message.data, fayeCtx);
      });
    } else {
      this.fayeClient.subscribe(
        `${pattern}_ack`,
        this.getMessageHandler(pattern, handler),
      );
    }
  });
}

Two additional (unrelated) changes that show up on this branch are:

The use of the FayeContext object. We'll cover that in the next section.
Passing { channel: pattern } to the deserialize() call. We'll cover that later in the article.

Subscription Context

Consider what happens when we subscribe to a topic via the broker API. With Faye, it's straightforward. We register a Faye subscription handler on a topic, and when a message matching that topic arrives, our handler is called with the message payload containing only what the publisher sent.

For some brokers, the subscription handler can receive additional context about the message — sometimes in the callback parameters, and sometimes in the message payload. For example, with NATS, the message payload packs the actual content sent by the publisher in a top-level data property, and adds two additional top-level properties with context metadata: one describing the topic the caller was subscribed to, and one containing the optional reply topic. We refer to this metadata as Context, and Nest allows you to pass this information through to the user-land handler (method decorated by @MessagePattern() or @EventPattern()).

Since Faye simply doesn't provide any such metadata, we'll demonstrate this behavior by passing the channel in this context object. This is not particularly useful in this case, as we're not actually providing any new information, but with Faye it's the best we can do to demonstrate the concept of the Context object.

Creating the context object

For any given broker, the technique for creating the context object may vary, based on the discussion above. In the end, you'll need to extract any context information (from the message payload, if context is encoded there, or from the subscribe() callback parameters) and pass it to the user-land handlers. We use a custom object to pass context. Open the file nestjs-faye-transporter/src/responder/ctx-host/faye-context.ts. It's a pretty simple object that extends BaseRpcContext.

// nestjs-faye-transporter/src/responder/ctx-host/faye-context.ts
import { BaseRpcContext } from '@nestjs/microservices/ctx-host/base-rpc.context';

type FayeContextArgs = [string];

export class FayeContext extends BaseRpcContext<FayeContextArgs> {
  constructor(args: FayeContextArgs) {
    super(args);
  }

  /**
   * Returns the name of the Faye channel.
   */
  getChannel() {
    return this.args[0];
  }
}

Essentially, it keeps an array of context values — in our case just strings, though you can specify a more complex type if you need to. For example, MQTT uses:

// from https://github.com/nestjs/nest/blob/master/packages/microservices/ctx-host/mqtt.context.ts
type MqttContextArgs = [string, Record<string, any>];

export class MqttContext extends BaseRpcContext<MqttContextArgs> {
  constructor(args: MqttContextArgs) {
    super(args);
  }

For each context value type (e.g., "channel", in our case, used for carrying the channel/pattern name), you supply a getter function. Remember, this is a simple data structure as there are generally very few context values, so your getter is just pulling out a particular row of the array.

Passing Context to User-land Handlers

With this in place, we then need to simply find the right places to:

Instantiate a FayeContext object with the correct values
Pass it to the user-land handlers

We already encountered this behavior in the previous sections. Let's look again at the getMessageHandler() method, focusing on the relevant snippet from that method:

  return async (message: ReadPacket) => {
    const inboundPacket = this.deserializer.deserialize(message);
    const fayeCtx = new FayeContext([pattern]);

    const response$ = this.transformToObservable(
      await handler(inboundPacket.data, fayeCtx),
    ) as Observable<any>;

It should be clear how we're using an instance of FayeContext here. We instantiate it with the relevant context, and then pass the instance to our user-land handler method. The code above takes care of context for request/response type handlers. We also need to handle this appropriately for events. We saw where that happened in the last section, in the bindHandlers() method. Here's the relevant snippet:

    if (handler.isEventHandler) {
      // The only thing we need to do in the Faye subscription callback for
      // an event, since it doesn't return any data to the caller, is read
      // and decode the request, pass the inbound payload to the user-land
      // handler, and await its completion.  There's no response handling,
      // hence we don't need all the complexity of `getMessageHandler()`
      this.fayeClient.subscribe(pattern, async (rawPacket: ReadPacket) => {
        const fayeCtx = new FayeContext([pattern]);
        const packet = this.parsePacket(rawPacket);
        const message = this.deserializer.deserialize(packet);
        await handler(message.data, fayeCtx);
      });

Remember, where and how you gather the context for any particular broker may vary, but the concept remains the same.

Testing Context

In this branch (part3), we provided a new project, nestHttpApp. This is a simple Nest HTTP app that includes a few routes for testing our nestMicroservice over the Faye transporter. It's worth mentioning that this branch also includes a full final version of the ClientFaye subclass of ClientProxy — this is necessary for our nestHttpApp to instantiate a client to be able to run ClientProxy#send() and ClientProxy#emit() calls. We'll actually build that ClientFaye component from scratch in the next two articles, but it's sitting there in this branch to help you test. With this in place, you can see the context by following these steps:

Start up the nestHttpApp

If you ran the build.sh script as described at the beginning of this chapter, all of this project's dependencies have been installed. If not, simply open a terminal in the top level nestHttpApp directory and run npm install. Then, start the application with npm run start:dev.
Issue an HTTP request like:

GET /customers

Note: You can do this in any number of ways — via a browser, browsing to localhost:3000/customers, via something like HTTPie - my favorite, or with a desktop program like Postman. See here for more help on running these requests.

Keep an eye on the nestMicroservice app's console log. You should see a line like the following, demonstrating that we've successfully passed the FayeContext data to the handler:

[AppController] Faye Context: {"args":["/get-customers"]}"

Deserialization Options

The transporter main class we inherit from (Server) has hooks for calling custom serialization and deserialization methods to encode/decode broker messages. I cover this topic extensively in this article. The main use case is for translating between non-Nest (i.e., external) message formats and the internal Nest microservice transporter message protocol. One thing we need to do to fully enable custom deserialization (i.e., let users hook their own custom deserializer class) with our custom transporter is fully implement the call to the Deserializer#deserialize() method. Here's its interface:

export interface Deserializer<TInput = any, TOutput = any> {
  deserialize(value: TInput, options?: Record<string, any>): TOutput;
}

This means we can pass in some context via a second argument that the user can access as the options parameter to utilize in their custom deserialization code. Usually this context is just the channel (pattern), but it can also include other contextual information. For example, the NATS transporter passes the following information (note the replyTo property):

const message = this.deserializer.deserialize(rawMessage, { channel, replyTo });

In our case, with Faye, we'll stick to the usual pattern and pass the channel. This is done in bindHandlers(), where we added the deserialization context as shown below:

    const message = this.deserializer.deserialize(packet, {
      channel: pattern,
    });

With this in place, any user-written deserializer will have access to the channel name to make decisions about transforming inbound messages. I encourage you to read this article for a deeper understanding of deserialization to help inform what you should pass as deserialization options.

Type Safety

In this branch, we added types to several calls to help ensure type safety when end users use the custom transporter. You can examine the code and explore how we use the following newly imported interfaces:

// nestjs-faye-transporter/src/responder/transporters/server-faye.ts
import { FayeClient } from '../../external/faye-client.interface';
import { FayeOptions } from '../../interfaces/faye-options.interface';

Let's take a quick moment to examine the FayeOptions interface:

// nestjs-faye-transporter/src/interfaces/faye-options.interface.ts
import { Serializer, Deserializer } from '@nestjs/microservices';

export interface FayeOptions {
  /**
   * faye server mount point (e.g., http://localhost:8000/faye)
   */
  url?: string;
  /**
   * time in seconds to wait before assuming server is dead and attempting reconnect
   */
  timeout?: number;
  /**
   * time in seconds before attempting a resend a message when network error detected
   */
  retry?: number;
  /**
   * connections to server routed via proxy
   */
  proxy?: string;
  /**
   * per-transport endpoint objects; e.g., endpoints: { sebsocket: 'http://ws.example.com./'}
   */
  endpoints?: any;
  /**
   * backoff scheduler: see https://faye.jcoglan.com/browser/dispatch.html
   */
  // tslint:disable-next-line: ban-types
  scheduler?: Function;
  /**
   * instance of a class implementing the serialize method
   */
  serializer?: Serializer;
  /**
   * instance of a class implementing the deserialize method
   */
  deserializer?: Deserializer;
}

These options (modulo serializer and deserializer, which are consumed by the ServerFaye class directly) are passed through to the Faye client library in createFayeClient() to customize the Faye connection:

  public createFayeClient(): FayeClient {
    // pull out url, and strip serializer and deserializer properties
    // from options so we conform to the `faye.Client()` interface
    const { url, serializer, deserializer, ...options } = this.options;
    return new faye.Client(url, options);
  }

The other benefit of this interface is that it provides intellisense for users constructing the options object in the call to NestFactory.createMicroservice(...) in the main.ts file.

Figure 1: Editor Intellisense

Error Handling

What should we do if the server loses its connection to the broker (e.g., the Faye broker goes offline)? This is a good question that probably deserves deeper discussion than what we'll cover here. The Faye client library appears to be fairly robust at re-establishing a connection, including buffering failed attempts, so we are going to simply defer to it, and log an error message to the console whenever a connection goes down. To handle this, we simply add a listener for a disconnect event and log the message:

  public handleError(stream: any) {
    stream.on(ERROR_EVENT, (err: any) => {
      this.logger.error('Faye Server offline!');
    });
  }

Checkpoint

We covered a lot of ground in this chapter. Because we provided the nestHttpApp and a full implementation of the Faye flavored ClientProxy, you now have a fully working Faye Custom Transporter. You can feel free to experiment at large with the nestHttpApp and nestMicroservice. Try issuing HTTP requests to the nestHttpApp like:

GET /customers

and

POST /customer // <-- with a payload

See here for more help on running these and similar tests..

As mentioned earlier in this article, there's also a series of routes related to exploring how microservices handle promises and observables when they are returned from a request. You can read more here.

What's Next

In Part 4), we turn our attention to the "client" side of the transporter. We'll build the Faye-flavored ClientProxy from scratch. Like the server construction we've done, we'll break this into two parts so we can work through the main concepts first, then handle some edge cases and details in the subsequent part.

Feel free to ask questions, make comments or suggestions, or just say hello in the comments below. And join us at Discord for more happy discussions about NestJS. I post there as Y Prospect.

Part 2: Basic Server Component

John Biundo — Mon, 02 Mar 2020 17:40:05 +0000

John is a member of the NestJS core team

Introduction

This is Part 2 of a six-part series. If you landed here from Google, you may want to start with Part 1.

In this article, we build the first iteration of the server component of our Faye Custom Transporter. We'll then test it with a simple Nest responder (microservice) app.

Get the Code

All of the code in these articles is available here. As always, these tutorials work best if you follow along with the code. The README covers all the steps you need to get the repository, build the apps, and follow along. It's easy! I strongly encourage you to do so. Note that each article has a corresponding branch in the repository. For example, this article (Part 2), has a corresponding branch called part2. Read on, or get more details here on using the repository.

Git checkout the current version

For this article, you should git checkout the branch part2. You can get more information about the git repository branches here.

Build the Apps for This Part

$ # from root directory of project (e.g., transporter-tutorial, or whatever you chose as the root)
$ sh build.sh

Organizing the Code

Even though the scope of this tutorial is relatively small — at the end of the day we'll just be creating a few main classes — we're going to be accumulating a number of assets along the way to test and exercise our code. It's time to think a little bit about code organization. First, let's take a look at what we'll end up with:

Our Faye Custom Transporter is going to consist of some interfaces and constants, some serializers/deserializers (discussed extensively in the previous article series), and the main components: a client component (class) providing the ClientProxy subclass we can use in any Nest requestor apps and a server component (class) we can use in any Nest responder apps.
The native apps we wrote in the previous article that let us explore the Faye client API and give us some test drivers.
A pair of regular Nest apps that will host and exercise our transporter code as we build it: a Nest requestor that is a simple Nest HTTP app called nestHttpApp and a Nest responder that is a classic Nest microservice, called nestMicroservice. (Note: see the previous article series for more on this terminology, but think of a Nest requestor as an app that makes remote requests over some non-HTTP transport (like Faye) and a Nest responder as a Nest application that listens for inbound requests over that transport).

Where to keep all of these assets?

The main thing to think about now is where to keep the Faye Custom Transporter components. Since we'll want to easily re-use them across different Nest apps, the logical place is an NPM package. To that end, when we work on any of those components, they'll go into that package.

The Custom Transporter Package

At this point, you should be on the part2 branch of the repo that you cloned (read more here for details). Once you've checked out that branch, you'll notice a new folder called nestjs-faye-transporter. This is organized as an NPM package (you can learn more about creating NestJS-friendly NPM packages in my article Publishing NestJS Packages with npm, including how the tsconfig.json file, package.json scripts and properties, and file organization all interact to create a reusable NPM package).

Here's a run-down on the contents of the nestjs-faye-transporter package, and how we'll use each:

The scripts (package.json) let you build (npm run build or npm run build:watch) and, if you want, publish (npm publish) the package.
Our workflow, in dev, will be to build the package and then use npm link to install it in our Nest apps. More on this soon.
The code is organized in a directory structure as follows:
- Code common to our client and server is in top level files or folders, like src/external (holds things like interfaces to the Faye client library which are external to the Nest environment), src/interfaces (holds interfaces used within the Nest constructs), and src/constants.
- All code defining the client is in the src/requestor folder. Ultimately, this will consist of a sub-class of ClientProxy from the @nestjs/microservices package — the class which provides methods like client.send(...) and client.emit(...)) — and a variety of related classes.
- All code defining the server is in the src/responder folder. Ultimately, this will consist of a sub-class of Server from the @nestjs/microservices package — the class which implements a transporter's server-side strategy — and a variety of related classes.

First Iteration (Take 1) of the Server Component

Let's get started building the "server" side of the equation — the part of the custom transporter that you'll use to help build Nest microservices (AKA Nest responders).

Using a Custom Transporter Strategy

Before we start writing transporter code, let's first take a look at how we're going to use a custom transporter. To see this in action, let's look at the nestMicroservice app (this app has been added in this branch). The code for it is in the top-level nestMicroservice folder.

Instantiating the Custom Transporter

Open the src/main.ts file. Notice the structure of the createMicroservice call:

// nestMicroservices/src/main.ts
const app = await NestFactory.createMicroservice(AppModule, {
  strategy: new ServerFaye({
    url: 'http://localhost:8000/faye',
    serializer: new OutboundResponseIdentitySerializer(),
    deserializer: new InboundMessageIdentityDeserializer(),
  }),
});

This is very similar to the structure used for any built-in Nest transporter. For example, with MQTT it looks like:

const app = await NestFactory.createMicroservice(AppModule, {
  transport: Transport.MQTT,
  options: {
    host: 'localhost',
    port: 1883,
    serializer: new OutboundResponseIdentitySerializer(),
    deserializer: new InboundMessageIdentityDeserializer(),
  },
});

It should be clear what we're doing: instead of passing the normal transporter options object with transport and options properties, we pass a single property, strategy, whose value is an instance of our custom transporter class. This class takes a single options object in its constructor, pretty much mirroring the options you pass to the options property with the built-in transporters.

Using the Custom Transporter

Let's briefly review how our nestMicroservice app works. Its only real job is to take in a '/get-customers' message and return a list of customers. It performs the exact same function as the customerApp we built in Part 1. The action is in the nestMicroservices/src/app.controller.ts file:

// nestMicroservices/src/app.controller.ts
@Controller()
export class AppController {
  logger = new Logger('AppController');

  /**
   * Register a message handler for '/get-customers' requests
   */
  @MessagePattern('/get-customers')
  async getCustomers(data: any): Promise<any> {
    const customers =
      data && data.customerId
        ? customerList.filter(cust => cust.id === parseInt(data.customerId, 10))
        : customerList;
    return { customers };
  }
}

All it really does is check for the existence of a customerId in the inbound message, and if it exists, uses it to filter the customer list it returns.

Take 1 Requirements

Alright, we're finally ready to look at our ServerFaye class — the one we'll instantiate and pass as the value of that strategy property above. We'll take this in a couple of steps.

This iteration (Take 1) is going to be the absolute bare-bones class needed to implement the server component of the Faye transporter. We're going to keep our first version basic so that we can focus on the core flow. Our first cut will:

minimize error handling
not yet utilize some of the nice features of the framework that make our code really robust
not handle events (e.g., inbound messages coming from client.emit(...))
not really be type safe (we omit a bunch of typing to declutter the code)

To state it in terms of requirements: the goal is to respond to a well-formed inbound request (i.e., a request from a request-response style message). We'll test this requirement by replacing our native customerService responder app from the last article with our nestMicroservices app running our new Faye Custom Transporter, and sending it the same '/get-customers' request from our native customerApp.

In Part 3, we'll complete the implementation and have a fully functioning Faye Custom Transporter (server component). At that point, you'll also have all of the concepts in place to write your own custom transporter server component, as well as the ability to look inside the built-in transporters (like the MQTT transporter server) and understand what's going on. That will prepare you for even more adventures, like customizing the Nest built-in transporters to add features— the subject of my next NestJS microservice tutorial (already underway, and coming very soon 💥)!

Take 1 Code Review

Let's dive into the code. Open the file nestjs-faye-transporter/src/responder/transporters/server-faye.ts.

An important concept is that this class extends Server (view this parent class here: @nestjs/microservices/server/server.ts). We won't walk through all the details of the Server superclass, but the main things to know are:

1) Our ServerFaye class inherits some properties and methods from the built-in Server class
2) After being instantiated in the main.ts file, the ServerFaye instance is essentially managed for us as part of the Nest lifecycle. This means that when the application bootstraps, and our custom ServerFaye class is instantiated, its inherited properties are populated with data, and the framework calls the entry point of our class. That entry point is the listen() method.

So let's dig into the ServerFaye class, starting with the listen() entry point. Its job is to make a connection to the broker, and then run start(), which is where the fun begins.

The interesting thing in start() is the call to this.bindHandlers(). Take a look at that method:

// nestjs-faye-transporter/src/responder/transporters/server-faye.ts
  public bindHandlers() {
    /**
     * messageHandlers is populated by the Framework (on the `Server` superclass)
     *
     * It's a map of `pattern` -> `handler` key/value pairs
     * `handler` is the handler function in the user's controller class, decorated
     * by `@MessageHandler()` or `@EventHandler`, along with an additional boolean
     * property indicating its Nest pattern type: event or message (i.e.,
     * request/response)
     */
    this.messageHandlers.forEach((handler, pattern) => {
      // only handling `@MessagePattern()`s for now
      if (!handler.isEventHandler) {
        this.fayeClient.subscribe(
          `${pattern}_ack`,
          this.getMessageHandler(pattern, handler),
        );
      }
    });
  }

  public getMessageHandler(pattern: string, handler: Function): Function {
    return async message => {
      const inboundPacket = this.deserializer.deserialize(message);
      const response = await handler(inboundPacket.data);
      const outboundRawPacket = {
        err: null,
        response,
        isDisposed: true,
        id: (message as any).id,
      };
      const outboundPacket = this.serializer.serialize(outboundRawPacket);
      this.fayeClient.publish(`${pattern}_res`, outboundPacket);
    };
  }

The comments should help with understanding what's happening, but at a high level, the strategy should be pretty clear:

Iterate over all of our "message handlers"*. These are the user-land Controller methods decorated with @MessagePattern(); a list of them is made available to us (in this.messageHandlers) by the framework, which discovers them using introspection with the Reflect API during the bootstrap process.
For each one, subscribe to the inbound channel (the _ack form of the topic). Remember, a Faye client's subscribe() call registers a Faye subscription handler to be invoked whenever the Faye client library receives an inbound message matching this topic. So this is the step where we map patterns to handlers. In short, this is our "router".
The Faye subscription handler in step 2, when invoked, runs the actual user supplied handler (the user-land handler method decorated with something like @MessagePattern('/get-customer')), and returns the result it gets from that method. When we say "returns", we mean publishes a reply on the outbound channel (the _res form of the topic).
Along the way, we run our deserializer on the inbound message, our serializer on the outbound response, and we package up the data produced by the user's pattern handler in an appropriately shaped standard Nest transporter message object.

*For Take 1, we are omitting event handlers (methods decorated with @EventPattern(...)). We'll handle these in Take 2 of our server component, in the next article.

The only slightly tricky part is the call to getMessageHandler(). This is just a higher-order function that is returning us the actual Faye subscription handler function.

If all this looks somewhat familiar, it's because we're basically following the same STRPTQ (subscribe-to-the-response-then-publish-the-request) pattern we used in the native customerService app in the previous article (Part 1).

Acceptance Testing

We should be ready to test out our code. We need to do a tiny bit of setup first. This is going to run best if you have four separate terminals open so you can watch things unfold across the whole process as the various pieces communicate.

Now's a good time to mention a couple of things about the development setup:

I run this kind of stuff on an Ubuntu machine. There's nothing platform-specific anywhere in the Nest-related code, but, I find running builds, starting and stopping processes, running multiple terminal sessions, etc., to be much smoother on Linux. You can run this wherever you want, but if you're not on Linux, you may have to make slight adjustments to your package.json, or other actual build steps.
Since you kind of need to run multiple terminal sessions to see the full impact, I strongly recommend Tmux. I mentioned this briefly at the end of the last article, so you now know I feel strongly about it 😃. You can instead start multiple terminal programs, or use a tabbed console program if you prefer, but if you want what I think is the best DX for this kind of work, checkout Tmux. I covered some detailed Tmux recommendations in the last article series.

In the following steps, I'll reference these (logical) terminals as:

Terminal 1: run the Faye broker here
Terminal 2: run live builds (npm build:watch) of the transporter server code we're working on here
Terminal 3: run the "requestor code" here. This is usually the customerApp; in the future, we'll also interact with nestHttpApp (making it the requestor) using HTTPie commands from the OS prompt (you can also use something like Postman or curl to issue HTTP requests, of course). We can use one terminal for this since we don't typically run both the customerApp and the nestHttpApp at the same time
Terminal 4: run the nestMicroservice Nest responder application here (this is the plain old Nest microservice app that will be using our new Faye Custom Transporter)

Primary Acceptance Test

Going back to our requirements, our main acceptance test is simple: send a well-formed message from our native customerApp (acting as a requestor) to our nestMicroservice (acting as a responder), and get back a proper response.

Figure 1: Faye Transporter Acceptance Test

Using `npm link` in Our Development Environment

Before we can run the test, we need to cover one more preliminary. Ask yourself this: how does our nestMicroservice app know how to find the code in our nestjs-faye-transporter package? The answer is pretty simple (though it belies the awesomeness of NPM!).

Note: If you ran the build.sh script at the beginning of this article, you probably do not need to run the npm link commands below. Doing so again is harmless, and it's useful to understand what's happening so I include the steps below. Also, sometimes things just get flakey with the links, and you need to re-run them. I'd actually suggest you go ahead and do so even if you did already run the build.sh script, just to see the behavior. These steps are prety quick anyway.

We're going to use the npm link command (read details here, but the following is all you need to know for this tutorial). There are two simple steps:

In terminal 2, make sure you're in the folder nestjs-faye-transporter (our NPM package folder). Then run npm link at the OS level. You should see output like this (I use nvm; if you don't your output will look slightly different):

$ # from directory nestjs-faye-transporter
$ npm link

@faye-tut/nestjs-faye-transporter@1.0.1 prepare /home/john/code/nest-micro/nestjs-faye/nestjs-faye-transporter
npm run build

@faye-tut/nestjs-faye-transporter@1.0.1 build /home/john/code/nest-micro/nestjs-faye/nestjs-faye-transporter
tsc

 (... some lines omitted ...)

/home/john/.nvm/versions/node/v10.18.1/lib/node_modules/@faye-tut/nestjs-faye-transporter ->
/home/john/code/nest-micro/nestjs-faye/nestjs-faye-transporter

In terminal 4, make sure you're in the folder nestMicroservice. Then run npm link @faye-tut/nestjs-faye-transporter at the OS level. This command references the NPM package name (found in nestjs-faye-transporter/package.json) for our custom transporter. You should see output like this:
```
$ # from directory nestMicroservice
$ npm link @faye-tut/nestjs-faye-transporter
/home/john/code/nest-micro/nestjs-faye/nestMicroservice/node_modules/@faye-tut/nestjs-faye-transporter ->
/home/john/code/nest-micro/nestjs-faye/nestjs-faye-transporter
```

At this point, our nestMicroservice testing app is live-linked to the custom transporter code in the @faye-tut/nestjs-faye-transporter NPM package we're working on. Any changes we make to that code will trigger a rebuild, and because we linked, those changes are immediately visible. Very cool stuff there.

Running the Test

We're ready to rock and roll 🎵!

In terminal 1, make sure the Faye broker is running. First make sure you're in the faye-server directory, then run npm run start. You should see something like this:

$ # from directory faye-server
$ npm run start
simple-faye-server@1.0.0 start /home/john/code/nest-micro/nestjs-faye/faye-server
node server.js

listening on http://localhost:8000/faye
========================================

In terminal 4, start the nestMicroservice. First make sure you're in the nestMicroservice directory, then run npm run start:dev. You should see the usual Nest startup logs, followed by the message Microservice is listening....

In terminal 3, we'll run the native customerApp to make the request. Earlier, we used this app to send requests to the native customerService app. Now we're sending those exact same messages (via the Faye broker, of course) to the nestMicroservice. Make sure you're in the customerApp directory, then...

... keep your eyes on all three terminal windows so you don't miss the magic, and...

# in terminal 3 window...
# in folder customerApp
npm run get-customers

If all went well, you should see a flurry of log messages. I strongly encourage you to take the time to look at them and make sure you can follow the flow of what's happening. In terminal 3, we should get a nice response that looks like this:

$ npm run get-customers

> faye-customer-app@1.0.0 get-customers /home/john/code/nest-micro/nestjs-faye/customerApp
> node dist/customer-app get

Faye customer app starts...
===========================
<== Sending 'get-customers' request with payload:
{"pattern":"/get-customers","data":{},"id":"415fdcef-88e8-42e7-a464-ebdb3b10ff57"}

==> Receiving 'get-customers' reply:
{
  "customers": [
    {
      "id": 1,
      "name": "nestjs.com"
    }
  ]
}

Terminals 1 and 4 show the corresponding message flow in the Faye broker and nestMicroservice app respectively. They all should weave together nicely to let you trace the entire saga through the layers of the system. By the way, if you're wondering where the message level logging in terminal 4 (nestMicroservice) comes from, take a look at the serializer/deserializer implementations in nestjs-faye-transporter/src/responder/serializers/outbound-response-identity-serializer.ts and nestjs-faye-transporter/src/responder/deserializers/inbound-message-identity-deserializer.ts respectively. These are our so-called "identity" serializer/deserializer from the previous article series, which are very helpful for visualizing message flow.

Hooray! We're done right? 🍺 🍺 🍺?

Not so fast... did you forget this is a six-part series 😃? Read on to see where we've still got work to do.

Understanding the Limitations of Take 1

We already know we have to clean up a few things, like adding event handling (e.g., handlers decorated with @EventPattern(...)), adding TypeScript types, and plugging in more cleanly to the framework. But the biggest limitation of our Take 1 implementation is its (lack of) handling of RxJS Observables. To fully appreciate this, we'll have to take a bit deeper dive into the overall flow and handling of requests through the Nest system.

We'll explore this in greater detail in the next article, but let's start with a picture. The following animation shows the path a hypothetical inbound HTTP request would take through our application. Inner boxes with a red background are part of the Nest infrastructure. User supplied code lives in the user-land space with a white background. Controllers are in light blue, and "Services" are in yellow.

Figure 2: Nest Request Handling

An inbound HTTP request kicks off the following sequence of events. Bolded words represent Nest system responsibilities. Underlined words represent user code. There's probably nothing terribly surprising going on here, but let's just briefly walk through it.

The request is routed to a route handling method (like getCustomers in our controller).
The route handling method can either make a remote request directly, or call a service that makes a remote request.
The remote request is handled by the broker client library and sent to the broker.
The remote request is received by the broker client library in the nestMicroservice app.
The request is routed to the correct handler (e.g., a method decorated with @MessagePattern('/get-customers')) based on matching the pattern in the request with the pattern in the method decorator.
The request handling method may make a call to other services (which in turn, could make their own internal calls or remote calls). Let's say it does make such a call, to the custService.getCustomers() method, and that method has a signature like this:
```
getCustomers(id: integer): Observable<Customer>
```

Once the getCustomers method returns, we start the return trip, where things get more interesting. This "more interesting" part is mainly because Nest is very Observable-aware.

Figure 3: Nest Response Handling

In this sequence, I'll introduce a part of the transporter infrastructure responsible for what I'm informally calling "Marshalling" (there's no such official term or single component inside Nest that does Marshalling). Conceptually, it's the part(s) of the system that handle(s) dealing with transferring Observable streams over the network.

In Part 3 we'll go through a few use cases for why Observables are so cool, why they're a perfect fit in this flow, AND how they're actually really easy to use. I know this diagram doesn't make it seem that way, but hey, we're building our own transporter (my inner trekky can't help but giggle over that 🚀). The beauty of it is that once we handle this case properly — and the framework will make this easy as we'll see in the next chapter — everything we might want to do with Observables (and their potential is just, well... mind bending) just works.

Here's the walk through of the return trip flow:

Our handler in nestMicroservice (perhaps directly, or perhaps by calling a service) returns the result. In terms of our example, it's returning a list of customers.
Now, if the response is a "plain" value (JavaScript primitive, array or object), there's not much work to be done other than send it back to the broker. But, if the response is an Observable stream, Nest steps in and makes this extremely easy for everything downstream to work with. That's where marshalling comes in.
Once the response is prepared from nestMicroservice, it's delivered to the Faye broker via the broker client library.
The broker takes care of publishing the response.
On the nestHttpApp side, the broker client library (which has previously subscribed to the broker on the response channel — though we haven't coded that part yet), receives the response message.
The ClientProxy class (again, we haven't built this yet) takes care of routing (and marshalling, if it's dealing with a stream of responses).
This routes the response to the originating service, then back to the originating controller.
Finally, the Nest HttpAdapter software again, if needed, marshalls responses to a suitable form for HTTP transport. For example, if the response is an Observable or a promise, it converts it to an appropriate form for return over HTTP.

So what exactly is the issue?

As you might expect from the above, things work fine if our handlers return plain objects. For example, we currently return an object in our nestMicroservices getCustomers() handler (last line below):

// nestMicroservice/src/app.controller.ts
@MessagePattern('/get-customers')
async getCustomers(data: any): Promise<any> {
  const customers =
    data && data.customerId
      ? customerList.filter(cust => cust.id === parseInt(data.customerId, 10))
      : customerList;
  return { customers };
}

But what happens if our handler returns an Observable? The framework enables this for all built-in transporters, so we should handle it too. Let's test this really quickly. Replace that last line in app.controller.ts with:

return of({customers});

You'll also have to add the following line to the top of the file:

import { of } from 'rxjs';

This construct uses the RxJS of operator to convert our getCustomers() method handler response to an Observable — a stream (containing only a single value in our case, but still, a stream) of values.

If you make this change, then re-issue the /get-customers message (run npm run get-customers in terminal 3), you'll get a rather ugly failure in the nestMicroservice window. This is our fault! If you think about it, we aren't doing any marshalling anywhere in our code. So we aren't handling this case, which, again, is expected of any Nest microservice transporter.

What's Next

With these issues in mind, we're ready to step up our game and make the Faye Custom Transporter server component much more robust. We'll tackle that in the next article. In Part 3 we cover:

A little side expedition on how and why you should care about the "Observables issue" we just uncovered
Addressing that issue
Handling events (e.g., @EventPattern(...) decorated methods) in our responder app
Adding TypeScript types
A few other minor items to clean up our app and make it production worthy

Feel free to ask questions, make comments or suggestions, or just say hello in the comments below. And join us at Discord for more happy discussions about NestJS. I post there as Y Prospect.

Part 1: Introduction and Setup

John Biundo — Mon, 02 Mar 2020 17:18:40 +0000

John is a member of the NestJS core team

Introduction

This article series covers the topic of building a custom transporter for the NestJS microservices subsystem. Is this article right for you? It is if you fit into one of the following two audiences:

You're trying to build or modify a NestJS microservices transporter. That one was probably pretty obvious 😄.
You're a NestJS application developer (someone who writes code in what I'll often refer to as "user-land"), and want a deeper understanding of how NestJS microservices work. Candidly, this is where I came from, and a main motivator for doing the research to write this article series. Even if you never intend to build your own transporter, you can still benefit from this deeper understanding, as I have done. Along the way, I'll try to share insights on how you can become a better user of Nest microservices.

Why Custom Transporters?

Let's start with the Why? question. The Nest microservices subsystem provides a communications layer abstraction that makes it easy for applications (both Nest and non-Nest) to communicate over a wide variety of what are called transporters. This provides several key benefits, covered fully in the previous article series. Nest comes with a variety of built-in transporters, including NATS, RabbitMQ, Kafka, and others. But what if you want to build your own transporter — say for ZeroMQ, Google Cloud Pub/Sub, Amazon Kinesis or Apache ActiveMQ? If that's your desire, you've landed on the right article! Or, if you just want to understand more about how this key part of the Nest infrastructure works, read on!

A related question is "How can I extend the capabilities of existing Nest transporters?" For example, maybe you'd like to use MQTT's QoS feature with the MQTT transporter. That's a topic I'll cover in another upcoming series (already under development 💥), but this series provides a baseline for addressing that requirement.

Article Series Overview

Part 1 (this article): Sets the background for the entire series. We'll begin by examining the simple Faye message broker to set the stage for building a custom Nest transporter that sits on top of it. By the time we're done with Part 5, we'll have a fully functional Faye Transporter!
Part 2: Tackles building an initial version of the server side of our custom transporter. The result is a working version that should familiarize you with the main concepts needed to understand how to build the server component of any custom transporter. It defers implementation of a few important details to the next article so as to keep us focused on the main path.
Part 3: Cleans up some loose ends, and dives deep on a few of the more advanced and impressive features of the framework, and shows you how to enable these features in your custom transporter.
Part 4: Switches to the client side. As with the server side, this article walks through a basic functional implementation for Faye, deferring a few of the more complex features to the next article.
Part 5: Completes the client-side journey, resulting in a finished, fully functional Custom Transporter for Faye.
Part 6: (Coming soon!) Surveys a few of the built-in NestJS transporters and compares their implementation to the Faye implementation to shed light on some broker-specific nuanced implementation details.

Let's get started!

Get the Code

All of the code in these articles is available here. As always, these tutorials work best if you follow along with the code. The README covers all the steps you need to get the repository, build the apps, and follow along. It's easy! I strongly encourage you to do so. Note that each article has a corresponding branch in the repository. For example, this article (Part 1), has a corresponding branch called part1. Read on, or get more details here on using the repository.

Git checkout the current version

For this article, you should git checkout the branch part1. You can get more information about the git repository branches here.

Build the Apps for This Part

$ # from root directory of project (e.g., transporter-tutorial, or whatever you chose as the root)
$ sh build.sh

Overview for Part 1

For this case study, we'll build a custom transporter to work with the Faye message broker. Faye is a simple OSS JavaScript publish/subscribe message broker that runs nicely on Node.js. Getting it up and running and understanding its API are simple, making it a convenient target. At the same time, it provides all the features Nest needs to build a transporter.

Faye uses a very simple — almost canonical — publish/subscribe protocol, as depicted in Figure 1 below.

Figure 1: Faye Message Protocol

Building Basic Faye Client Apps

As we did in the NestJS Microservices in Action series, we'll start by building simple native requestor and responder apps. These are TypeScript apps that exercise the Faye client API directly. This will help us make sure we understand the Faye client API, and give us a convenient test bench.

As covered in the previous article series, one of the challenges the Nest microservices package deals with is to layer a request/response message style on top of publish/subscribe semantics. In other words, Nest requestors need to be able to run code like this, which issues a request and returns a response from a remote microservice:

  @Get('customers')
  getCustomers(): Observable<any> {
    this.logger.log('client#send -> topic: "get-customers"');
    // the following request returns a response as an Observable
    return this.client.send('/get-customers', {});
  }

Supporting Request/Response

Since Faye (as well as most brokers) only supports publish/subscribe, we need to follow a simple recipe to provide request/response semantics. Here's a recap of the generic pattern for how this is often done.

Let's say component A wishes to "get customers" from component B, which has access to a customer DB. Component A can publish a '/get-customers' message, and (assuming it has subscribed to that topic) component B receives it, queries the customer DB for a list of customers, and sends a response message which is delivered back to A. Handling the response message is where the magic happens. In order for B to respond to A, they both must do a few things, agreed upon by convention:

A chooses a response topic (sometimes called a reply subject)
A subscribes to the response topic
A publishes a request, passing the response topic as part of the message
B publishes a response, using the response topic as the topic of its message

💡 We've seen this subscribe-to-the-response-then-send-the-request pattern a few times now, and we'll see it again. It's a vital pattern, and worth permanently etching into your synapses. To that end, let's invent a simple mnemonic. That way we can mentally conjure this pattern quickly whenever we need it. I'll refer to it as STRPTQ. The letters come from "Subscribe To the Response, then Publish The ReQuest". I use Q to make it easier to remember reQuest (vs. Response), in the proper order.

As we'll see in developing our Faye transporter, Nest makes this a bit easier by automatically choosing the response topic name for you. In fact, Nest builds two topics from each pattern you declare. For example, assume you define a Nest microservice responder message pattern like this:

@MessagePattern('/get-customers')

(Note: Nest uses the term channel as a generic internal name for topics (also called subjects and channels by some brokers), and to help disambiguate them from the user-land concept of a pattern, from which they are derived.)

Internally, Nest builds two channels from the above pattern:

'/get-customers_ack' - this is the physical channel name we'll use in the Faye transporter to publish/subscribe to request messages on the topic '/get-customers'
'/get-customers_res' - this is the physical channel name we'll use in the Faye transporter to publish/subscribe to response messages resulting from '/get-customers' requests

Let's build those native apps we mentioned. Note: later we'll want to use these native apps to test our Nest transporter, so to make them interact seamlessly with Nest, we'll utilize the Nest channel names described above. We'll start with the responder app.

Native Responder App (customerService)

This is the app that responds to inbound messages (requests) and generates responses. Assuming you're following along on the part1 branch, open the file customerService/src/service.ts.

Here's the implementation of the getCustomers Faye subscription handler. This is the code we register to run when our app receives an inbound message on the '/get-customers_ack' channel (see the subscribe() call lower in the file):

// customerService/src/service.ts
function getCustomers(packet): void {
  const message = parsePacket(packet);
  console.log(
    `\n========== <<< 'get-customers' message >>> ==========\n${JSON.stringify(
      message,
    )}\n=============================================\n`,
  );

  // filter customers list if there's a `customerId` param
  const customers =
    message.data && message.data.customerId
      ? customerList.filter(
          cust => cust.id === parseInt(message.data.customerId, 10),
        )
      : customerList;

  client.publish('/get-customers_res', getPayload({ customers }, message.id));
}

The logic should be easy to understand. Refer to the full listing and make sure you see how we are using the '/get-customers_ack' and '/get-customers_res' channels here. We'll be seeing a lot of that pattern, so make sure it makes sense to you.

One detail to point out is the call to getPayload(). Let's discuss that. As mentioned, we're building these native apps to conform to Nest protocols. This includes using the Nest-standard channel names, as well as matching the internal Nest message format (message format is covered extensively in the NestJS Microservices in Action series). Because we take this approach, the apps provide a convenient test bench for the Nest transporter Server and ClientProxy we're building in this article series. The message format requirement means we need to wrap responses in a standard object we can depict like this:

   {
     /**
      *  Request error message, if any
      */
     err: any,
     /**
      *  The response message payload (return value of a
      *  message pattern handler)
      */
     response: any
     /**
      *  Status of an Observable response. `true` once the final
      *  stream value has been retrieved.
      */
     isDisposed: boolean
     /**
      *  Unique identifier, corresponding to the id field received
      *  from the initial request
      */
     id: string
   }

So getPayload() is a helper that wraps payloads in this standard Nest message structure. If you're paying close attention, you'll notice that getPayload() (not shown here — look at the service.ts source code to see it) copies the message.id field from the inbound request to the outbound message. We haven't discussed these id fields yet, but they'll become very important when we dive into the "client side" of our transporter starting in Part 5.

Native Requestor App (customerApp)

This is the user-facing app that makes outbound requests. The requestor is in customerApp/src/customer-app.ts. Below is the implementation of the getCustomers() function from that app.

There's a little bit of boilerplate, but the main logic should be easy to see. It implements the requestor side of that STRPTQ request/response protocol we walked through a couple of minutes ago.

// customerApp/src/customer-app.ts
async function getCustomers(customerId) {
  // build Nest-shaped message
  const payload = getPayload('/get-customers', { customerId }, uuid());

  return new Promise((resolve, reject) => {
    // subscribe to the response message
    const subscription = client.subscribe('/get-customers_res', result => {
      console.log(
        `==> Receiving 'get-customers' reply: \n${JSON.stringify(
          result.response,
          null,
          2,
        )}\n`,
      );
    });

    // once response is subscribed, publish the request
    subscription.then(() => {
      console.log(
        `<== Sending 'get-customers' request with payload:\n${JSON.stringify(
          payload,
        )}\n`,
      );
      const pub = client.publish('/get-customers_ack', payload);
      pub.then(() => {
        // wait .5 second to ensure subscription handler executes
        // then unsubscribe and resolve
        setTimeout(() => {
          subscription.cancel();
          resolve();
        }, 500);
      });
    });
  });
}

We first subscribe to '/get-customers_res' (the response channel). We pass it a simple Faye subscription handler that prints out the result when a message on this channel comes back. This output becomes the "final response" to an end-user command (request) that's been issued from the command line (received by way of the basic command line processing logic at the bottom of the file).

Then we publish the request on '/get-customers_ack'. Note that the call to subscribe() returned a subscription which is a Promise that resolves once the Faye server acknowledges that the subscribe operation has completed.

Because we're running this as a standalone Node.js command line program, and thus want to exit after sending a request (which let's us use it as a command processor to conveniently send requests), we wait 500ms after publishing the request, then cancel the subscription and exit. Cancelling the subscription is important hygiene — if we don't, subscriptions hang around (though Faye will eventually detect them and clean them up). This is a bit brute force, and we'll want to do more sophisticated subscription management in our transporter client, but suffices for now.

Finally, you'll notice that much like the customerService responder app, we have a helper getPayload() utility to wrap requests in the Nest message protocol. You'll also notice that we are generating the id field — the same id field we are handling on our responder app above. For now, just think of the id field as something we need to have in order to speak native Nest message protocol. Later, we'll dive into this in more detail.

For completeness, let's take a very quick look at how we run the Faye server. Though you can treat it as a black box, it's worth a moment to look at how it runs, and appreciate its native node.js-ness and simplicity. Open up faye-server/server.js. The code is shown below. You can read more about running the Faye server here, but we're basically just starting it up, and for instrumentation purposes, listening for various events that we can log to the console to make it easy to trace the handshaking and message exchange. You shouldn't have to mess with this at all, but if you do, it's pretty basic stuff.

// faye-server/server.js
const http = require('http');
const faye = require('faye');

const mountPath = '/faye';
const port = 8000;

const server = http.createServer();
const bayeux = new faye.NodeAdapter({ mount: mountPath, timeout: 45 });

bayeux.attach(server);
server.listen(port, () => {
  console.log(
    `listening on http://localhost:${port}${mountPath}\n========================================`,
  );
});

bayeux.on('handshake', clientId =>
  console.log('^^ client connect (#', clientId.substring(0, 4), ')'),
);

bayeux.on('disconnect', clientId =>
  console.log('vv client disconnect (#', clientId.substring(0, 4), ')'),
);

bayeux.on('publish', (clientId, channel, data) => {
  console.log(
    `<== New message from ${clientId.substring(0, 4)} on channel ${channel}`,
    `\n    ** Payload: ${JSON.stringify(data)}`,
  );
});

bayeux.on('subscribe', (clientId, channel) => {
  console.log(
    `++ New subscription from ${clientId.substring(0, 4)} on ${JSON.stringify(
      channel,
    )}`,
  );
});

bayeux.on('unsubscribe', (clientId, channel) => {
  console.log(
    `-- Unsubscribe by ${clientId.substring(0, 4)} on ${JSON.stringify(
      channel,
    )}`,
  );
});

Testing the Apps

We've now completed our requestor and responder, and can test them out. While I only reviewed the '/get-customers' message flow above, the code in this branch also implements the '/add-customer' message. Take a minute to run the code now by following these simple instructions. Here's what it looks like. In this video loop, the Faye broker is running in the top pane, the customerService app is running in the middle pane, and the customerApp app is running in the lower pane (and if you're curious, this is running in the tmux virtual terminal; you can get my setup here if you're interested).

Screen Capture: Native App Demo

What's Next

In Part 2, we'll write and test a basic version of the Faye transporter's server component. This is the component that runs in the context of a microservice listener to enable apps to function as Nest responders (these terms and concepts are discussed extensively in the NestJS Microservices in Action series). We'll test this server component by using it in a simple Nest responder (microservice) app. That Nest microservice app will replace the native customerApp we just built, and we'll make sure it responds to the exact same customer-related messages.

Feel free to ask questions, make comments or suggestions, or just say hello in the comments below. And join us at Discord for more happy discussions about NestJS. I post there as Y Prospect.

Part 3: Techniques for Application Integration with NestJS and NATS

John Biundo — Wed, 12 Feb 2020 14:25:30 +0000

John is a member of the NestJS core team

Part 3: Solving the Message Incompatibility Problem

In Part 2 of this series, we described the challenges of integrating Nest apps with non-Nest apps using NATS as an intermediary.

In this article, we explore solutions to that problem.

Get the Code

Reminder: all the code for these articles is available here, with complete instructions here.

Serialization and Deserialization

Serialization and deserialization are the steps Nest uses to translate between message formats. In other words, Nest receives inbound messages (from the broker), deserializes them (converts arbitrary message formats into Nest message formats), and then processes them. For outbound messages, the process is reversed: the last step before sending a message to the broker is to (optionally) serialize it (convert from Nest message format to some other format).

The nifty part is that while Nest does all this automatically, it also provides hooks where you can easily plug in your own serializers and deserializers to handle arbitrary message formats. Serializers and deserializers are simply classes that implement a defined interface.

Since requestors and responders perform their own unique roles (each with their own context) and each are both message senders and message receivers, there are four hooks where serialization/deserialization take place. These are examined in the recipes below. If there's one section of this article series that you should bookmark for future reference, this is it. Each recipe describes exactly where you should look to handle a serialization/deserialization task, including the full context you need to understand why and how to do it.

Serialization/Deserialization Hooks and Recipes

Let's examine the two different contexts (requestors and responders) in which we need to serialize/deserialize. For each, we'll examine handling inbound and outbound messages.

Nest as Requestor

Nest requestors live in the context of a ClientProxy instance. It follows that we customize serialization/deserialization behavior by modifying the ClientProxy behavior. For example:



// app.controller.ts
@Client({
  transport: Transport.NATS,
  options: {
    url: 'nats://localhost:4222',
    serializer: new OutboundRequestSerializer(),
    deserializer: new InboundResponseDeserializer(),
  }
})
client: ClientProxy;

In the code fragment above, OutboundRequestSerializer is the class that answers the question "how do does my Nest requestor format outgoing requests so that an external app can process them?", while InboundResponseDeserializer is the class that addresses "how does my requestor translate incoming external responses so that Nest can process them?".

Note: you can choose any name for these classes, but trust me, it's highly recommended that you choose a naming convention similar to the ones above to keep things straight!

Nest as Responder

Nest responders live in the context of a microservice instance. It follows that we customize serialization/deserialization behavior by modifying the microservice instance behavior. For example:



// main.ts
async function bootstrap() {
  const app = await NestFactory.createMicroservice(AppModule, {
    transport: Transport.NATS,
    options: {
      url: 'nats://localhost:4222',
      deserializer: new InboundMessageDeserializer(),
      serializer: new OutboundResponseSerializer(),
    },
  });
  app.listen(() => console.log('Microservice is listening...'));
}

So InboundMessageDeserializer is the class that answers the question "how does my responder translate incoming external messages so that Nest can process them?", while OutboundResponseSerializer is the class that handles "how does my responder format responses so that the external app can understand them?".

This feature results in the following diagrams — ones that remove all those ugly red X's in the last diagram!

Here's the diagram to keep in mind when writing serializers/deserializers for Nest responders.

Figure 1: Nest Responder (De)serialization

In the above diagram, the Nest responder serializer and deserializer are configured in the NestFactory.creatMicroservice() call.

And here's the corresponding diagram for Nest requestors.

Figure 2: Nest Requestor (De)serialization

In the above diagram, the Nest requestor serializer and deserializer are configured in the ClientProxy configuration (decorator or injectable, depending on which method you use).

Implementing Serializers and Deserializers

Before we start writing serializer/deserializer classes, let's take a look at their interfaces.

Serializer Interface

A serializer class needs to implement the serialize() method, which takes a single argument — the outbound payload to be serialized — and returns the serialized payload.



export interface Serializer<TInput = any, TOutput = any> {
  serialize(value: TInput): TOutput;
}

We'll now implement an identity serializer for our Nest responder as an exercise. An identity serializer simply returns the same message it receives. In addition to exercising the interface, we can use it to "spy on" the otherwise silent serialization function that Nest always performs on our behalf.

Responder Identity Serializer Implementation

Here, we'll be working in the nestMicroservice project (i.e., nest-nats-sample/nestMicroservice directory). Take a look at the src/common/serializers/outbound-response-identity.serializer.ts file:



// src/common/serializers/outbount-response-identity.serializer.ts
import { Serializer, OutgoingResponse } from '@nestjs/microservices';
import { Logger } from '@nestjs/common';

export class OutboundResponseIdentitySerializer implements Serializer {
  private readonly logger = new Logger('OutboundResponseIdentitySerializer');
  serialize(value: any): OutgoingResponse {
    this.logger.debug(
      `-->> Serializing outbound response: \n${JSON.stringify(value)}`
    );
    return value;
  }
}

Now simply plug this serializer into the main.ts file for the Nest responder app. If you're following along with the github repository, the main.ts file will have all of this code (and more that we'll get to soon) in place, but commented out. Simply uncomment the line shown below.



// src/main.ts
const app = await NestFactory.createMicroservice(AppModule, {
  transport: Transport.NATS,
  options: {
    queue: 'customers',
    url: 'nats://localhost:4222',
    /**
      * Use the "Identity" (de)serializers for observing messages for
      * nest-only deployment.
      */
    serializer: new OutboundResponseIdentitySerializer(),
  },
});

If you now send some requests to the nestMicroservice app from the nestHttpApp (here's how), you'll see logging information that looks something like this, showing you the internal layout of the outbound message:



[Nest] 8786   - 02/04/2020, 10:19:04 AM   [OutboundResponseIdentitySerializer] -->> Serializing outbound response:
{"err":null,"response":[],"isDisposed":true,"id":"41da72e8-09d0-4720-9404-bd0977b034a0"}

Deserializer Interface

Now let's take a look at the required interface for a deserializer.



export interface Deserializer<TInput = any, TOutput = any> {
  deserialize(value: TInput, options?: Record<string, any>): TOutput;
}

A deserializer class needs to implement the deserialize() method, which takes a two arguments — the payload to be deserialized and an optional options object — and returns the deserialized payload.

The options object contains metadata about the incoming message. For NATS, the object contains the value of the replyTo message property, if one exists.

Let's implement a deserializer. To start with, we'll build an "identity deserializer" that just logs out the contents of any incoming message without doing any transformation on it.

Responder Identity Deserializer Implementation

We're still working on the Nest responder — the nestMicroservice project (i.e., nest-nats-sample/nestMicroservice directory). Take a look at the src/common/deserializers/inbound-message-identity.deserializer.ts file:



// src/common/serializers/inbound-message-identity.deserializer.ts
import { ConsumerDeserializer, IncomingRequest } from '@nestjs/microservices';
import { Logger } from '@nestjs/common';

export class InboundMessageIdentityDeserializer
  implements ConsumerDeserializer {
  private readonly logger = new Logger('InboundMessageIdentityDeserializer');

  deserialize(value: any, options?: Record<string, any>): IncomingRequest {
    this.logger.verbose(
      `<<-- deserializing inbound message:\n${JSON.stringify(
        value
      )}\n\twith options: ${JSON.stringify(options)}`
    );
    return value;
  }
}

Like our identity serializer, we can quickly plug in an identity deserializer to spy on the incoming messages from our requestor. Again, if you're following along using the github repository, simply comment out the lines so the active (de)serializers match those shown below.



// src/main.ts
const app = await NestFactory.createMicroservice(AppModule, {
  transport: Transport.NATS,
  options: {
    queue: 'customers',
    url: 'nats://localhost:4222',
    /**
      * Use the "Identity" (de)serializers for observing messages for
      * nest-only deployment.
      */
    serializer: new OutboundResponseIdentitySerializer(),
    deserializer: new InboundMessageIdentityDeserializer(),
  },
});

If you again send some requests to the nestMicroservice app from the nestHttpApp (like this), you'll see logging information that looks something like this, showing you the layout of the inbound message:



[Nest] 8786   - 02/04/2020, 10:19:04 AM   [InboundMessageIdentityDeserializer] <<-- deserializing inbound message:
{"pattern":"get-customers","data":{},"id":"41da72e8-09d0-4720-9404-bd0977b034a0"}
        with options: {"channel":"get-customers","replyTo":"_INBOX.IDML09N4JB6W0MF4P6GBPB.IDML09N4JB6W0MF4P6GCX2"}

Notice the value for the replyTo field in the logging output. You can probably guess what that is. This is a detail that's going to come in handy down the line, so file that away for future reference.

Implementing the Integration Use Cases

We've nearly arrived at our final destination. With the understanding we've gained, we can specify what we need to complete our integration use cases from Figure 1 in Part 1.

Here are the requirements:

The nestHttpApp, in its role as a requestor, must implement:
- A) an outbound message external serializer that translates a Nest formatted request into a request understood by our external service. For example:
  - From Nest format
    - topic: 'get-customers'
    - reply topic: '_INBOX.XVRL...'
    - payload: {pattern: 'get-customers', data: {}, id: 'abc...'}
  - To external format
    - topic: 'get-customers'
    - reply topic: '_INBOX.XVRL...'
    - payload: {}
- B) an inbound response external deserializer that translates an external response into a format understood by Nest. For example:
  - From external format
    - topic: '_INBOX.XVRL...'
    - payload: {customers: [{id: 1, name: 'nestjs.com'}]}
  - To Nest format
    - topic:'_INBOX.XVRL...'
    - payload: {err: undefined, response: {customers: [{id: 1, name: 'nestjs.com'}]}, isDisposed: true}
The nestMicroservice app, in its role as a responder, must implement:
- A) an inbound message external deserializer that translates an external request into a format understood by Nest. For example:
  - From external format
    - topic: 'get-customers'
    - reply: '_INBOX.XVRL...'
    - payload: {}
  - To Nest format
    - topic 'get-customers'
    - reply topic: '_INBOX.XVRL...'
    - payload: {pattern: 'get-customers', data: {}, id: 'abc...'}
- B) an outbound response external serializer that translates a Nest formatted response into a response understood by our external service. For example:
  - From Nest format
    - topic: '_INBOX.XVRL...'
    - payload: {err: undefined, response: {customers: [{id: 1, name: 'nestjs.com'}]}, isDisposed: true}
  - To external format
    - topic: '_INBOX.XVRL...'
    - payload: {customers: [{id: 1, name: 'nestjs.com'}]}

With that in mind, let's get busy! The bolded descriptions above inform the names of our classes.

For requirement 1-A, we have src/common/serializers/outbound-message-external.serializer.ts in our nestHttpApp project. Here's the code. The comments explain its intent.



// src/common/serializers/outbound-message-external.serializer.ts
import { Logger } from '@nestjs/common';
import { Serializer } from '@nestjs/microservices';

export class OutboundMessageExternalSerializer implements Serializer {
  private readonly logger = new Logger('OutboundMessageExternalSerializer');
  serialize(value: any) {
    this.logger.debug(
      `-->> Serializing outbound message: \n${JSON.stringify(value)}`,
    );

    /**
     * Here, we are merely "unpacking" the request payload from the Nest
     * message structure and returning it as a "plain" top-level object.
     */
    return value.data;
  }
}

For requirement 1-B, we have src/common/deserializers/inbound-response-external.deserializer.ts in our nestHttpApp project. Here's the code. The comments explain its intent.



// src/common/deserializers/inbound-response-external.deserializer.ts
import { WritePacket, Deserializer } from '@nestjs/microservices';
import { Logger } from '@nestjs/common';

export class InboundResponseExternalDeserializer implements Deserializer {
  private readonly logger = new Logger('InboundResponseExternalDeserializer');

  deserialize(value: any): WritePacket {
    this.logger.verbose(
      `<<-- deserializing inbound response:\n${JSON.stringify(value)}`,
    );

    /**
     * Here, we wrap the external payload received in a standard Nest
     * response message.  Note that we have omitted the `id` field, as it
     * does not have any meaning from an external responder.  Because of this,
     * we have to also:
     *   1) implement the `Deserializer` interface instead of the
     *      `ProducerDeserializer` interface used in the identity deserializer
     *   2) return an object with the `WritePacket` interface, rather than
     *      the`IncomingResponse` interface used in the identity deserializer.
     */
    return {
      err: undefined,
      response: value,
      isDisposed: true,
    };
  }
}

For requirement 2-A, we have src/common/deserializers/inbound-message-external.deserializer.ts in our nestMicroservice project. Here's the code. The comments explain its intent.



// src/common/serializers/inbound-message-external.deserializer.ts
import { Logger } from '@nestjs/common';
import * as uuid from 'uuid/v4';
import { ConsumerDeserializer } from '@nestjs/microservices';

export class InboundMessageExternalDeserializer
  implements ConsumerDeserializer {
  private readonly logger = new Logger('InboundMessageExternalDeserializer');
  deserialize(value: any, options?: Record<string, any>) {
    this.logger.verbose(
      `<<-- deserializing inbound external message:\n${JSON.stringify(
        value,
      )}\n\twith options: ${JSON.stringify(options)}`,
    );

    /**
     * Here, we merely wrap our inbound message payload in the standard Nest
     * message structure.
     */
    return {
      pattern: undefined,
      data: value,
      id: uuid(),
    };
  }
}

For requirement 2-B, we have src/common/serializers/outbound-response-external.serializer.ts in our nestMicroservice project. Here's the code. The comments explain its intent.



// src/common/serializers/outbound-response-external.serializer.ts
import { Serializer, OutgoingResponse } from '@nestjs/microservices';
import { Logger } from '@nestjs/common';

export class OutboundResponseExternalSerializer implements Serializer {
  private readonly logger = new Logger('OutboundResponseExternalSerializer');
  serialize(value: any): OutgoingResponse {
    this.logger.debug(
      `-->> Serializing outbound response: \n${JSON.stringify(value)}`,
    );

    /**
     * Here, we are merely "unpacking" the response payload from the Nest
     * message structure, and returning it as a "plain" top-level object.
     */

    return value.response;
  }
}

Enabling the External (De)serializers

The last step is to plug in these (de)serializers in at the appropriate "hook points". We've seen this before. For the nestHttpApp, this is done wherever you configure the ClientProxy. In our case, for ease of code organization, we've done this with the @Client() decorator in the src/app.controller.ts file. To enable the external (de)serializers, update that file to look like this:



// src/app.controller.ts
  @Client({
    transport: Transport.NATS,
    options: {
      url: 'nats://localhost:4222',
      /**
       * Use the "Identity" (de)serializers for observing messages for
       * nest-only deployment.
       */
      // serializer: new OutboundMessageIdentitySerializer(),
      // deserializer: new InboundResponseIdentityDeserializer(),

      /**
       * Use the "External" (de)serializers for transforming messages to/from
       * (only) an external responder
       */
      serializer: new OutboundMessageExternalSerializer(),
      deserializer: new InboundResponseExternalDeserializer(),
    },
  })
  custClient: ClientProxy;

For the nestMicroservice, this is done in the src/main.ts file. To enable the external (de)serializers, update that file to look like this:



// src/main.ts
  const app = await NestFactory.createMicroservice(AppModule, {
    transport: Transport.NATS,
    options: {
      queue: 'customers',
      url: 'nats://localhost:4222',
      /**
       * Use the "Identity" (de)serializers for observing messages for
       * nest-only deployment.
       */
      // serializer: new OutboundResponseIdentitySerializer(),
      // deserializer: new InboundMessageIdentityDeserializer(),

      /**
       * Use the "External" (de)serializers for transforming messages to/from
       * an external requestor
       */
      serializer: new OutboundResponseExternalSerializer(),
      deserializer: new InboundMessageExternalDeserializer(),
    },
  });

At this point, you should be able to send and receive requests between any combination of apps. For example, from nestHttpApp to customerService, from customerApp to nestMicroservice, and all other combinations. I highly recommend you do so now, using the instructions from here and here to run all components at the same time*. Pay close attention to the log outputs which show how (de)serialization is happening at each step.

*We haven't described the use of queues, but when you start running the full mixed configuration (Nest and non-Nest components), one thing you'll notice is that requests will be randomly routed between the nestMicroservice app and the customerService app. This is because they are members of a NATS queue. If you want to send a request to a particular responder, you'll need to shut down the other one to guarantee the message is sent there. I'll discuss NATS queues in more detail the final article of this series.

Here's a nice surprise! We've magically handled Figure 1 Case D along the way. If you think carefully about it, this is both not a surprise, and has a somewhat strict limitation. The reason this works is that we're now serializing and deserializing to and from a canonical external format. So all requests look like they come from an external requestor, and all responses look like they come from an external responder. With this in place, our serializers and deserializers work in all combinations. We've created a "lingua franca" message protocol. The limitation with this approach is perhaps a subtle one, and is something we'll address later (see bullet point #2 in What's Next? below).

Conclusion 🚀

We've covered a lot of ground. Hopefully you've got both a conceptual framework for how to think about integrating external applications with Nest apps using tools like Nest microservices and the NATS message broker, and some boilerplate code you can work with. Some of these concepts also translate directly to other Nest microservice transporters, but as you can imagine, the details are different. Holler in the comments if you'd like me to cover other transporters in this fashion in another series, and I'll see if I can do so.

What's next? ❓

There are a few remaining subtle topics to cover, which I'll do in the next installment of this series. As a teaser, these include:

Can we run both the nestMicroservice app and the external customerService app at the same time, and load balance requests across them? Spoiler: yes! In fact, we already did so, as mentioned above. This is simple to do with NATS distributed queues, which we'll cover briefly in the next article.
We covered the case where we're running our modified nestHttpApp in this hybrid (Nest and non-Nest) environment. What happens if we have a mixture of Nest apps that are unmodified — that is, they are deployed apps that we don't want to touch to implement our "lingua franca" message protocol (i.e., they're running out-of-the-box standard de(serializers))? Can they be made to play nicely in this hybrid environment? Spoiler: yes!
What about events? We've covered what seems to be the trickier case here with request/response style messaging, but how about plain old events? In fact, you may have noticed we've built a feature for adding a customer (see the 'add-customer' messages and handlers sprinkled through our code). Go ahead and try them now. You'll see you have mixed results. Can we make events cooperate in this hybrid environment? Spoiler: yes!

Stay tuned for the answers to these and other exciting questions in our next installment! 😃

Feel free to ask questions, make comments or suggestions, or just say hello in the comments below. And join us at Discord for more happy discussions about NestJS. I post there as Y Prospect.

Part 2: Challenges of External Integration

John Biundo — Tue, 11 Feb 2020 15:32:30 +0000

John is a member of the NestJS core team

Part 2: Digging into Transporter Communications

This is Part 2 of a series on using Nest microservices as an integration technology. Part 1 lays the foundation with an introduction to the basic communication concepts used by Nest microservices.

This article lays out the concepts and challenges of integrating Nest apps with non-Nest apps.

Get the Code

Reminder: all the code for these articles is available here, with complete instructions here.

Roles in Action

In the prior article, I covered the roles a Nest app plays in various communication scenarios. Let's dig further into those roles and how they matter in each integration use case.

Nest as Requestor

Let's start with a quick review of Nest's transporter application-level protocol. This is covered in depth in the Nest documentation, but we'll summarize briefly here, marrying Nest concepts with our freshly minted terminology covered at the end of Part 1.

When a Nest component is in the role of Nest as requestor, it performs its role through an instance of the ClientProxy class which has been configured to work with a specific transporter. Such a requestor may be housed in any sort of Nest app. For example, to make a 'get-customers' request from a Nest HTTP-based app (e.g., from within a REST route handler), via NATS, we would instantiate a ClientProxy with code something like this:*

// app.controller.ts
@Client({
  transport: Transport.NATS,
  options: {
    url: 'nats://localhost:4222',
  }
})
client: ClientProxy;

*Note: we use the @Client() decorator as a convenience, but Nest recommends using the dependency injection method for creating a ClientProxy instance. Both work for our purposes, but the DI method has some advantages for production development. See the Nest documentation for more details.

Using this ClientProxy instance, we would then add code to our REST route handler (e.g., one that responds to the REST GET customers request) to issue the NATS request with code something like this:

// app.controller.ts
@Get('customers')
async getCustomers(): Observable<customers> {
  return this.client.send('get-customers', {});
}

Let's update the diagram from Figure 1, Case C to reflect this terminology.

Figure 1: Nest as Requestor

Nest as Responder

When a Nest component is in the role of Nest as responder, things are slightly more complex. We need the component to function inside the context of a network listener (in order to receive inbound messages from remote senders). This concept gives us an understanding of what we can now refer to as a Nest microservice. A Nest microservice is a component that binds some behavior to incoming network messages bearing specific topics (and, optionally, payloads). Obviously the microservice listener must connect to the correct broker, and possibly be configured with other parameters. Making this association between a microservice listener and a particular broker, and specifying configuration parameters for that association, is the job of the transporter. For example, a Nest component that can respond to a NATS 'get-customer' message would need several moving parts.

First we need to start up a network listener. This is covered in detail in the Nest documentation here, but the code is straightforward if you're familiar with a typical Nest main.ts file:

// main.ts
async function bootstrap() {
  const app = await NestFactory.createMicroservice(AppModule, {
    transport: Transport.NATS,
    options: {
      url: 'nats://localhost:4222',
    },
  });
  app.listen(() => console.log('Microservice is listening...'));
}

All the transporter-specific details — such as how to connect to the NATS broker — are passed in as an options object to the createMicroservice() call.

Then we need to register the handler for any inbound request we want to handle. This is covered in detail here, but the handler code, running inside a microservice controller, is straightforward if you're familiar with typical Nest HTTP controllers:

// app.controller.ts
@MessagePattern('get-customers')
getCustomers(@Payload() data: any) {
  return customerService.getCustomers();
}

We can now update Figure 1 Case B to reflect this understanding.

Figure 2: Nest as Responder

The other roles discussed earlier — emitter and subscriber — are similar (and simpler). For brevity, we'll omit the diagrams and just describe the differences. An emitter is like a requestor except it does not expect a response, so it does not subscribe to a response message. Like a requestor, it's issued via a ClientProxy instance. A subscriber is like a responder except it does not issue a response message. Like a responder, it's housed as a handler within a microservice listener.

Running an All Nest Stack

Full implementations of these apps are included in the repository available here. Running these two Nest apps as shown in Case A of Figure 1, which you can do now by following these instructions, is easy. Following those steps will start up an instance of a NATS broker, in verbose mode, which is very helpful for watching the message flow. Take a few minutes to run through the full installation and to examine the code.

Of course so far, we haven't really had to deal with any external app components. This means there are no messy details, and the Nest components "just work". Let's move on to the topic of external app components now.

External NATS app

To give us something concrete to look at, let's quickly construct basic implementations of our external apps. We'll build two of them: one as a requestor and one as a responder. These will serve as sandboxes for examining live message flow behavior, and play the role of the "external apps" we see in Figure 1.

As native NATS apps, these can immediately and seamlessly communicate with each other.

Figure 3: External NATS Apps

The customerApp application

This app functions as a requestor, and is quite simple. When the app is called with the command line argument "get-customers", it first connects to the NATS server, then runs the getCustomers() function. Here's the essence of that function (slightly simplified, and minus a little logging and error handling).

// customer-app.ts
async function getCustomers() {
  const response = await nats.request('get-customers', 1000, {});
  console.log(
    'getCustomers reply: \n',
    JSON.stringify(JSON.parse(response.data), null, 2)
  );
}

The customerService application

This app functions as a responder. The code below is the essence of the main() function, which simply starts up, connects to NATS, and subscribes to a couple of topics. Note: to stay focused on the important elements, the code below is slightly simplified from the code in the repository, though the overall structure and intent remains the same.

// service.ts
async function main() {
  nats = await connect({
    servers: [NATS_URL],
    timeout: 1000,
  });
  console.log('NATS customer service starts...');

  nats.on('connect', (client, url, serverInfo) => {
    nats.subscribe('get-customers', getCustomers);
    nats.subscribe('add-customer', addCustomer);
  });
}

The code below is the essence of the getCustomers() callback function (again, simplified slightly from the repo code), which has been registered as the handler (line #9 above) to call when the 'get-customers' message is received.

// service.ts
async function getCustomers(err, message): Promise<void> {
  if (err) {
    return;
  }

  if (message.reply) {
      const customers =
        message.data && JSON.parse(message.data).id
          ? customerList.filter(
              cust => cust.id === parseInt(JSON.parse(message.data).id, 10)
            )
          : customerList;
      await nats.publish(message.reply, JSON.stringify({ customers })
  } else {
    console.error('Malformed request.  No reply field included in request.');
  }
}

If you look at the full repository source code, you'll see that our "database" is just a stub with an array of customers stored in a local customerList variable.

Most importantly, the getCustomers() method shows the request-response implementation clearly. The logic is straightforward:

check to see if a customer id is supplied. If so retrieve that customer, otherwise retrieve all customers.
publish a response message using the reply topic passed in on the request message. Include the customer list as the payload in that response.

The full source code for these apps is available here. Read more about running them here. I highly encourage you to do this to get familiar with their behavior, and to observe the NATS message flows first-hand.

NATS Client Library Native Support for Requests

You may have noticed we made a call to request() in the customerApp above, and wondered why (perhaps you were expecting us to call something like publish()). Nice catch! Here's a quick geeky aside on that topic that also provides a little insight into the Nest transporter abstraction layer.

If you dig into the NATS client API libraries (for example, the TypeScript client), you'll notice that they provide API calls to both the NATS PUB verb (usually publish()) and SUB verb (usually subscribe()), but also offer a request() method that doesn't have a direct counterpart in the NATS protocol (see an example here). By now, you should probably be able to guess what request() does. It uses essentially the same pattern we described in the previous article — implemented by Nest to provide request/response semantics on top of publish/subscribe messaging — within the NATS TypeScript client library itself. To be clear: this is a convenience provided by the client library (not a native feature of NATS).

In the case of the NATS transporter, Nest takes advantage of this client library feature directly, rather than emulating it. For other transporters, where no such "request/response abstraction" exists, Nest emulates this functionality. In the end, both Nest and the client libraries share a similar need to add request/response behavior on top of a publish/subscribe model.

Clash of the Message Formats

Different Message Formats

One thing we can do, now that we have an all-Nest version and a native TypeScript/NATS version of the requestor and responder, is examine the actual NATS messages exchanged. You can and should do this yourself (for now, you can examine the NATS logs produced by following these instructions and these instructions), but let's cut right to the chase. Nest encodes message payloads in a format that is probably not directly compatible with the format used by your external app. We'll address why that is, and how to resolve the incompatibility, below. Now we're getting to the meaty part of the article series!

Note: the following representations of the actual messages as seen in the NATS server log take a few small liberties to simplify and clarify. What you'll actually see in the logs is slightly more verbose. To see the NATS log messages yourself, follow these instructions.

Native App Messages

The native customerApp, when it issues the 'get-customers' request, emits a message that looks like this:

PUB get-customers _INBOX.6EADK
MSG_PAYLOAD: {}

Here, you see the message topic (get-customers) and the reply topic (_INBOX.6EADK) on the first line (they're part of the message "header", not the payload), and an empty message payload on the second line.

The customerApp receives a response back from customerService (because it included a response topic in the request - _INBOX.6EADK - that it subscribed to) that looks like this:

MSG_PAYLOAD: {"customers": [{ "id": 1, "name": "nestjs.com" }]}

Here, payload is just a "stringified" version of our JSON response.

Nest App Messages

On the other hand, the Nest components produce slightly different messages. The nestHttpApp, when it runs client.send('get-customers', {}), emits a message that looks like this:

PUB get-customers _INBOX.9FEAM
MSG_PAYLOAD: {"pattern": "get-customers","data": {},
"id": "84d9259e-fd00-4456-83b8-408311ca72cc"}

It receives a response back from nestMicroservice that looks like this:

MSG_PAYLOAD: {"err": null,"response": {"customers": [{ "id": 1, "name":"nestjs.com" }]},
"isDisposed": true,"id": "84d9259e-fd00-4456-83b8-408311ca72cc"}

The differences should be clear. Nest wraps your message payloads inside a JSON object. For requests, your payload is wrapped in a data property. For responses, your payload is wrapped in a response property.

Why the differences? Consider that Nest must properly route and manage the lifetime of messages within and between Nest apps (e.g., our "Pure NestJS" Case A in Figure 1). Nest needs to pass some metadata, along with the actual application-specific message content, with each message. Nest encodes this metadata in the payload itself (because NATS doesn't allow you to add fields anywhere else in a message), resulting in the extra fields we see.

With this in mind, we can layout the standard format for all Nest messages, thus defining Nest's transporter message protocol.

Nest Transporter Message Protocol

Request messages (coming from Nest requestors) are wrapped in a structure that we can depict as follows:

   {
     /**
      *  The message topic, also known inside Nest as
      *  the "message pattern"
      */
     pattern: string,
     /**
      *  The message payload (the first argument of
      *  ClientProxy#send or ClientProxy#emit)
      */
     data: any,
     /**
      *  A unique identifier, assigned by Nest.  Present
      *  only when the message is a request (created with
      *  ClientProxy#send)
      */
     id: string
   }

Response messages (coming from Nest responders) are wrapped in a structure we can depict as follows:

   {
     /**
      *  Request error message, if any
      */
     err: any,
     /**
      *  The response message payload (return value of a
      *  message pattern handler)
      */
     response: any
     /**
      *  Status of an observable response. False once the final
      *  stream value has been retrieved.
      */
     isDisposed: boolean
     /**
      *  Unique identifier, corresponding to the id field received
      *  from the initial request
      */
     id: string
   }

Clearly, we'll have a problem communicating between our Nest and non-Nest apps based on these different message formats. For example, in the Nest as responder case, we have the following issue, where an external request is not understood by the Nest responder due to the message format incompatibility.

Figure 4: Message Format Mismatch

As you can imagine, we have the reverse issue in the Nest as requestor case, where Nest issues requests wrapped in the Nest request format, which aren't understood by the external app, and the external app also responds with an incompatible message format (missing fields expected by Nest).

Now the big question: How do we reconcile these message format differences to connect Nest and external NATS apps, as in Figure 1 Cases B, C and D?

The good news is that Nest anticipates this need and provides a neat solution. We now have all the pieces in place to start seeing how Nest solves this problem and how to craft a solution. We'll dive into this in Part 3!

Feel free to ask questions, make comments or suggestions, or just say hello in the comments below. And join us at Discord for more happy discussions about NestJS. I post there as Y Prospect.

Part 1: Introduction to Microservices and Transporters

John Biundo — Mon, 10 Feb 2020 15:24:31 +0000

John is a member of the NestJS core team

Introduction

This article series covers the topic of integrating NestJS applications with other applications and services. This can be accomplished in several different ways; the approach covered in this series is to use a combination of Nest microservices and a message broker as the glue. In other words, Nest apps and external apps communicate with each other asynchronously through a message broker.

In these examples, we use NATS as the message broker. Note: some of the concepts here are common to all Nest microservices transporters, while others are specific to NATS. I try to identify those differences in the text.

Source Code

Working examples and all source code is available here. The articles work best if you follow along with the repository. Full installation instructions, along with many other details, are available here.

Nest Microservices Background

Nest's microservices package is one of those technologies that can be a little hard to wrap your mind around. Not that it's overly complex — to the contrary, it presents an application programming model that is quite simple. I think the confusion comes from the use of the term "microservices", which has become so over-used as to be virtually meaningless, and from the fact that Nest's microservices package supports several considerably different use cases:

Used internally as a transport layer, there is little beyond the surface level that a developer needs to understand to effectively use it. It simply "works", and provides a mechanism for adding features such as scalability and fault tolerance to your Nest application(s).
Used as an integration technology, one needs to dig a bit deeper. Because the microservices package plays this dual role, when using it for integration there are important implementation details that surface and can get in the way. Fortunately, as is usually the case, the elegance of the Nest architecture provides mechanisms that make this use case possible (if not immediately obvious). This topic is the focus of this article series.

Article Series Overview

This article series covers the integration use case. To get there, it first covers some Nest microservices basics. Some of these basics might be of interest to you even if you are using the package for the first use case described above (inter-Nest-app communication layer). The articles are organized so you can skip the basics if you want, or if you want to come back later for reference, you can just read the parts you need.

Part 1 (this article!) covers introductory material, including the basic microservices communication model, brokers, and how Nest uses publish/subscribe and request/response communication models. It introduces external integration use-cases that will be covered in the series. It also lays out a vocabulary for describing all the moving parts in these sometimes-complex interactions.
Part 2 digs a little deeper, extending the vocabulary to describe the modalities in which Nest operates when interacting with external systems. It introduces a pair of simple external (non-Nest, NATS-based) apps that we will use as part of the integration case study. It introduces the potential challenges arising from Nest's use of a proprietary message format.
Part 3 describes the Nest approach for solving the message format challenge. It goes on to work through code samples that implement this approach. It uses the vocabulary introduced earlier to help you keep a simple cognitive model in mind as you address this problem.
Part 4 (coming soon) covers some leftovers, some advanced use cases, and some tips and tricks.

I hope you find this article series helpful!

Part 1: Introduction to Nest Transporters

NestJS comes with a built-in versatile network communications-oriented subsystem called (perhaps somewhat confusingly) microservices. The core of Nest microservices, and the key to understanding them, is what Nest calls transporters. Transporters enable you to connect components over a network using a pluggable communications layer and a very simple application-level message protocol. Nest delivers a variety of transporters out-of-the-box, as well as an API allowing developers to build new custom transporters. This combination of architecture and features is extremely powerful.

A concrete example of a built-in Nest transporter is NATS. With the NATS-flavored transporter plugged in, Nest applications can communicate using the NATS messaging system. Since NATS is the intermediary for these communications, Nest can communicate not only with other Nest apps, but also with non-Nest apps, or a combination of both. This leads to four interesting use cases (Figure 1 below).

Figure 1: Transporter Use Cases

The current Nest microservices documentation focuses almost entirely on the first use case (Case A). And rightly so. If you simply want to benefit from a robust communication layer (e.g., easily adding load balancing and fault tolerance to increase the horizontal scalability of your Nest application), you really don't need to know about 90% of the material in this article series! (You may still benefit from a deeper understanding of the Nest microservices architecture, however, so I invite you to read on. Much of the material is still relevant background for leveraging Nest microservices).

The rest of this article series focuses mainly on the last three use cases — namely, using Nest microservices running on NATS to integrate with external systems.

To get there we need to lay some conceptual groundwork. Let's get started.

Concepts

There's a lot going on in that diagram, even viewed at the highest level. As soon as we begin diving into the details, the terminology soup can easily get in the way of deeper understanding. So without further ado, let's take a crack at some basic concepts and terminology.

First, let's acknowledge that there are two main flavors of Nest transporters, which I'll refer to as:

broker-based: this includes NATS, as well others like Redis, RabbitMQ, MQTT, and Kafka
point-to-point: this includes TCP and gRPC

This article focuses exclusively on the broker-based transporters, so let's discuss that flavor a bit further, using NATS as our case study example. We start with a description of a broker in general terms. An installed broker is comprised of several parts:

the broker server: this is an active server process (or possibly replicated servers) that manages publishing and subscribing (and the bookkeeping that goes with it), and delivers messages to clients.
the broker client API: this is delivered in a language-specific package (e.g., a JavaScript flavor, a Java flavor, a Go flavor, etc.) providing an API for accessing the broker from client programs. Client programs include both "native NATS applications" you write with the API (and without benefit of a framework), as well as frameworks like NestJS which use the client API to communicate to the broker.

Message flow diagrams typically only label the broker component (in our case, for example, as a green circle in the middle of various other components), but think of the client API as embedded in the other boxes, connecting them to the broker.

The key advantage of broker-based messaging systems is they allow you to decouple the various application components from each other. Each need only connect to the broker, and can remain unaware of the existence of, location of, or implementation details of other components. The only thing the components need to share is a message protocol.

Nest makes use of a small set of features that are typically available across most brokers. This is a bit of an over-generalization — one we'll revisit later on — but for now, let's examine the common broker features that Nest relies on.

Broker Message Protocol

From Nest's perspective, brokers provide a basic message-oriented communication protocol that is usually described as publish/subscribe. Publish/subscribe can be generally understood in terms of the following diagram, with the red circles indicating the order of events in a publish/subscribe "conversation".

Figure 2: Broker Message Protocol

In that diagram, each client component is either a publisher or a subscriber. The salient point is that subscribers "register interest in a topic" and publishers publish messages about a topic. The broker sits in the middle and performs the following functions:

keeps track of subscribers (by managing a list of subscriptions)
receives published messages
forwards published messages to all interested subscribers (based on matching topics between published messages and subscriptions)

One thing obviously missing from this diagram is the request/response style communication model. This communication style is useful when we want to verify receipt of a published message and/or receive a response from whoever consumes the message. In fact, this is the style of message implicit in Figure 1 above (where we can infer that 'get-customers' is a request and the array of customers is a response). How do we get from publish/subscribe to request/response?

Nest (as well as some of the brokers natively, including NATS as we'll soon see) solve this problem in a handy way, by building a request/response capability on top of the publish/subscribe model. Let's say component A wishes to "get customers" from component B, which has access to a customer DB. Component A can publish a "get customers" message, and (assuming it has subscribed to that topic) component B receives it, queries the customer DB for a list of customers, and sends a response message. The response message is where the magic happens. In order for B to respond to A, they both must do a few things, agreed upon by convention:

A chooses a response topic (sometimes called a reply subject)
A subscribes to the response topic
A passes the response topic as part of the initial message
B uses the response topic as the topic of its own subsequent response message

In other words, if a response topic is included in a message received by a subscriber, the subscriber publishes a response message on that response topic. Thus, in our example, B publishes its response, including a payload containing the customer list, on the response topic it received in the "request" message. Because A previously subscribed to the response topic, it receives the response as a normal message. All very tidy!

A Vocabulary for Transporter Use Cases

With this in mind, we can give names to the two different types of messages our system handles, and also describe the roles of the participants in those messages. Don't worry, we're almost done with the preliminaries! These terms will all become second nature pretty quickly, but it turns out to be extremely useful to have a vocabulary that will let us quickly navigate a bunch of similar-looking data flow diagrams, and keep track of where our custom code fits in!

The first kind of message we can exchange based on the above capabilities is called an event. Any given component participating in event-based messaging can be either:

an event emitter - meaning it publishes a message with a topic (and an optional payload). An event emitter is a pure message publisher.
an event subscriber - meaning it registers interest in a topic, and receives messages (forwarded by the broker) when a message matching that topic is published by an emitter.

The second kind of message we can exchange is called a request/response message. In this exchange, a participating component can be either:

a requestor - meaning it publishes a message it intends to be treated as a request, and it also takes the conventional steps described above — namely, subscribing to a response topic and including that response topic in the message it publishes.
a responder - meaning it subscribes to a topic it intends to treat as an incoming request, it produces a result, and it publishes a response, including the result payload, to the response topic it received on the inbound request.

So requestors are a special case of publishers, and we use this term to remember that they also await a response. And responders are a special case of subscribers, and we use this term to remember that they are also publishers of a response. Make sense? If not, don't worry, it will all come together soon.

Integration Use Cases

Most of the above is background that "just works" in an all-Nest world. Let's turn our attention now to integrating Nest apps with non-Nest apps using NATS as the communication intermediary.

In Figure 1, cases B, C, and D illustrate the different topologies for Nest-based apps to integrate with non-Nest apps using NATS. Note, we'll use the generic term "app" to describe these communicating components. As we'll see, these "apps" could be web apps, app services written in NestJS or some other platform, "microservices", etc., but the rules we'll describe are common regardless of the type of app involved. Let's briefly motivate the use cases.

In case B, we could be migrating a legacy app service, responsible for managing our customer DB, to Nest. In this case, our Nest app needs to play the role of responder that we described above. Going forward, we call this the Nest as responder use case.

In case C, we could be writing a new app, in Nest, that needs to query our customer DB using the existing (legacy) service. In this case, our Nest app needs to play the role of requestor that we described above. Going forward, we call this the Nest as requestor use case.

In case D, we could be somewhere in the middle of a complex migration process, where both types of communications are happening. Nest apps need to be able to speak to both other Nest apps and non-Nest apps, all via NATS. Non-nest apps similarly need to be able to talk to both types of apps. Going forward, we call this the Nest as duplex requestor/responder use case.

We'll examine each use case in more detail, including sample code you need to make this work (reminder full repository with usage notes here). Before we can do that, we need to understand a little more about how Nest components interact with the broker. We cover this in the next installment in the series.

Feel free to ask questions, make comments or suggestions, or just say hello in the comments below. And join us at Discord for more happy discussions about NestJS. I post there as Y Prospect.

Build a NestJS Module for Knex.js (or other resource-based libraries) in 5 Minutes

John Biundo — Mon, 23 Sep 2019 14:45:55 +0000

John is a member of the NestJS core team

Ever wanted to integrate your favorite library into NestJS? For example, while Nest has broad built-in support for database integration, what if you want to use your own favorite library and no Nest package exists? Well, why not build your own?

This might seem like a daunting task at first. But if you've been following my blog posts, you saw a design pattern for NestJS dynamic modules in my last post that points you in the right direction. OK, you say, but it still seems like a lot of work to integrate a library.

I have good news! Using the awesome power of the Nest CLI, you can generate a complete, custom dynamic module template, following NestJS best practices, with a single command! You can then literally have your library integrated in about 5 minutes! This technique works for many resource-based libraries that export what I'll call an API object, which naturally work well with Nest's built-in singleton provider model for sharing a global resource. Follow along with such an adventure as we build a quick integration to Knex.js below.

And stay tuned for more on the wonders of the vastly under-appreciated NestJS CLI. Built on top of the Angular CLI, the potential for this tool to revolutionize the way you use Nest is limitless. I have lots more planned for this topic in the near future!

Intro

In my How to build completely dynamic NestJS modules article last week, I covered a design pattern that is used throughout standard Nest packages like @nestjs/passport, @nestjs/jwt, and @nestjs/typeorm. The point of the article was two-fold:

Provide a roadmap to a basic pattern used within common NestJS packages to help you be a better consumer of those packages.
Encourage Devs to think about how to make use of this pattern in their own code to enable more easily composable modules.

Some might consider the pattern a bit complex, and I wouldn't necessarily argue. However, gaining a good understanding of it is useful, and helps in the quest for mastery of some core concepts of Nest, especially around leveraging the module system and dependency injection. Nevertheless, there's a fair amount of boilerplate code in the pattern. Wouldn't it be great to simplify the process?

Buckle up! NestJS CLI custom schematics to the rescue! 🚀

What We'll Build

In this tutorial, we'll build a module that exports a direct API to the full Knex.js library. This is a powerful DB integration library used widely across the Node.js ecosystem. Here's what we'll do. The following represent the exact same steps you can use to integrate any other basic callable API (for example, ioredis, Cassandra, Neo4J, Elasticsearch, LevelDb to name just a few).

Use the dynpkg custom schematic to generate a customized package (the schematic automates the dynamic module pattern I've been discussing above).
Fill in a few quick details to customize the package for Knex.js.
Use the auto-generated test client to demonstrate that it works.

Thanks to the power of the Nest CLI, coupled with a custom schematic, we can complete these steps in about 5 minutes! As a bonus, we'll cover a few more advanced features at the end of this article, including how to publish the generated package on npm with a single click.

If you want to see a completed version of the steps presented in the remainder of this article (a fully useable NestJS/Knex.js module), you can find a github repo with the completed code here.

Nest CLI and Schematics

You probably already use the Nest CLI on a regular basis. nest new myProject and nest generate controller user are common examples. They use the @nestjs/schematics package that ships with the CLI out-of-the-box. What's very cool is that you can easily use other schematics to do highly customized work. A feature common to all schematics is that they understand your project, and have a smart way of scaffolding new architectural components and wiring them into your project. Think of an individual schematic as a blueprint, and of the CLI machinery as a set of standards for how each blueprint works. Because of this, new schematics "just work", and inherit all of the features provided by the CLI and its standard capabilities.

You can write your own schematics, and I plan to show you how in some upcoming blog posts. For now, I've built one that you can use to do the library integration project we've undertaken today. Let's get started!

To use any external schematics, you of course need to install them. One important note: you must install a schematics collection as a global package, due to the way the CLI works.

Installing a Custom Schematics Collection

Schematics are packaged as collections and can be bundled up as npm packages, so installing them is simple. You can do so now with:

npm install @nestjsplus/dyn-schematics -g

Using a Custom Schematics Collection

Using schematics from a custom collection is straightforward. The standard Nest CLI command structure looks like this:

nest commandOrAlias [-c schematicCollection] requiredArg [options]

commandOrAlias is: new or generate (alias: g) or add
schematicCollection is optional, and defaults to the built-in NestJS schematics; you can optionally specify a globally installed npm package
requiredArg is the architectural element being generated/added, e.g., controller, module, or soon, in our case dynpkg
options are global to all schematics; they can be --dry-run, --no-spec, or --flat

Once it's installed, you use the @nestjsplus/dyn-schematics package like this:

Make sure you're in the folder you want to have as the parent of the project. With this schematic, we're creating a new complete nest package, meaning it's a new standalone project. So it will create a folder using the name you provide, and put all of the component parts inside that folder.
Run the schematic with the CLI

nest g -c @nestjsplus/dyn-schematics dynpkg nest-knex

This runs using the custom schematics collection from @nestjsplus/dyn-schematics and specifying the dynpkg schematic to execute (the schematic identifies the thing to generate - in this case, a dynamic module package identified as dynpkg), and giving it a name of nest-knex. If you want to see what this would do without adding files to your filesystem, just add --dry-run at the end of the command.

This should prompt you with the question Generate a testing client?. Answer yes to this to make testing easier. You'll see the following output, and you'll then be able to open this project in your IDE from the nest-knex folder.

► nest g -c @nestjsplus/dyn-schematics dynpkg nest-knex
? Generate a testing client? Yes
CREATE /nest-knex/.gitignore (375 bytes)
CREATE /nest-knex/.prettierrc (51 bytes)
CREATE /nest-knex/README.md (2073 bytes)
CREATE /nest-knex/nest-cli.json (84 bytes)
CREATE /nest-knex/package.json (1596 bytes)
CREATE /nest-knex/tsconfig.build.json (97 bytes)
CREATE /nest-knex/tsconfig.json (430 bytes)
CREATE /nest-knex/tslint.json (426 bytes)
CREATE /nest-knex/src/constants.ts (54 bytes)
CREATE /nest-knex/src/index.ts (103 bytes)
CREATE /nest-knex/src/main.ts (519 bytes)
CREATE /nest-knex/src/nest-knex.module.ts (1966 bytes)
CREATE /nest-knex/src/nest-knex.providers.ts (262 bytes)
CREATE /nest-knex/src/nest-knex.service.ts (1111 bytes)
CREATE /nest-knex/src/interfaces/index.ts (162 bytes)
CREATE /nest-knex/src/interfaces/nest-knex-module-async-options.interface.ts (532 bytes)
CREATE /nest-knex/src/interfaces/nest-knex-options-factory.interface.ts (194 bytes)
CREATE /nest-knex/src/interfaces/nest-knex-options.interface.ts (409 bytes)
CREATE /nest-knex/src/nest-knex-client/nest-knex-client.controller.ts (732 bytes)
CREATE /nest-knex/src/nest-knex-client/nest-knex-client.module.ts (728 bytes)

At this point, you can install the generated code:

cd nest-knex
npm install

Now start up the app with:

npm run start:dev

If you now browse to http://localhost:3000 you should get:

Hello from NestKnexModule!

Great! At this point, we've scaffolded the framework of our dynamic module, and we're ready to begin the Knex.js integration.

Integrating with Knex.js

You now have a project scaffolded to do the library integration. We have to make just a few edits to perform the actual integration. Let's talk them through:

We need to install knex as a dependency. Note, in order to run this project, you'll need access to a live SQL database. Knex.js doesn't bundle any database libraries, so you'll need to install the appropriate one (you can read more here). In this tutorial, I'll use PostgreSql, which I have available on localhost. If you do not have a local DB available, you can consider using a docker setup.
We need a way to provide the configuration options to the Knex.js API.
We need to call the Knex.js API in the way it wants to be initialized, returning a handle to a knex object which we can use to access Knex.js features.
To test things out, we can use the client controller that was auto-generated by the schematic.
Once it works, we can use this package directly. Or we can publish the package to a registry (e.g., an internal package registry or publicly to npmjs.com)

Install Dependencies

The knex package is required. You will also need a database API library to run the module. Below, I use pg for PostgreSql, but you can choose whatever you want (from the list here).

cd nest-knex
npm install knex pg

Knex.js Options

As discussed in the dynamic modules article, we provide options to the service using an async options provider. The code to do all this has been scaffolded for you, but to pass the appropriate options in a native TypeScript way, we need to modify the NestKnexOptions interface. This is contained in the file src/interfaces/nest-knex-options.interface.ts.

We want to describe the available Knex.js options that will be passed to the library API. The easiest way to do this is to use the types already provided by Knex.js. Since this interface is exported by the knex package, we can simply import it, and our job is nearly done! We'll simply alias it to make it visible to our generated package. Open up src/interfaces/nest-knex-options.interface.ts and edit it to look like this:

// src/interfaces/nest-knex-options.interface.ts
import { Config } from 'knex';

export interface NestKnexOptions extends Config {}

Knex Connections

The job of our module is pretty simple: connect to the API and return a re-usable knex object for interacting with the database. In the case of PostgreSql, knex (on top of pg) returns a connection pool, which means we can connect once and return a singleton knex object that will automatically balance multiple queries across the connection pool. This works seamlessly with NestJS and its notion of singleton providers!

Note: I'm less familiar with Knex than other DB libraries, and also less familiar with other DBs like MySQL, so if you are using this module for such databases, be sure you understand the right pattern for sharing a knex connection object within your application. This topic is not a NestJS issue -- it's all about the architecture of the Knex.js library and the database client library.

Now we'll implement a simple pattern to return our knex object. Open up src/nest-knex.service.ts. Have a quick look to see what the generated code looks like, then replace that code with the following:

// src/nest-knex.service.ts
import { Injectable, Inject, Logger } from '@nestjs/common';
import { NEST_KNEX_OPTIONS } from './constants';
import { NestKnexOptions } from './interfaces';

const Knex = require('knex');

interface INestKnexService {
  getKnex();
}

@Injectable()
export class NestKnexService implements INestKnexService {
  private readonly logger: Logger;
  private _knexConnection: any;
  constructor(@Inject(NEST_KNEX_OPTIONS) private _NestKnexOptions: NestKnexOptions) {
    this.logger = new Logger('NestKnexService');
    this.logger.log(`Options: ${JSON.stringify(this._NestKnexOptions)}`);
  }

  getKnex() {
    if (!this._knexConnection) {
      this._knexConnection = new Knex(this._NestKnexOptions);
    }
    return this._knexConnection;
  }
}

Almost all of this is unchanged from the generated boilerplate. We simply added a property to store our connection object (_knexConnection), and a getKnex() method to instantiate the object the first time it's requested, and then cache it for future use.

Test the Module

As mentioned, you'll need a local database instance available to test the module. If you don't have one handy, consider using docker to do so. It's easy!

The @nestjsplus/dyn-schematics dynpkg schematic built a small test client module for us (assuming you answered yes to the prompt). We can use this to quickly test our nest-knex module.

First, open src/nest-knex-client/nest-knex-client.module.ts and add the needed Knex.js options in the register() method. Adjust yours appropriately:

// src/nest-knex-client/nest-knex-client.module.ts
...
@Module({
  controllers: [NestKnexClientController],
  imports: [
    NestKnexModule.register({
      client: 'pg',
      connection: {
        host: 'localhost',
        user: 'john',
        password: 'mypassword',
        database: 'nest',
        port: 5432,
      },
    }),
  ],
})
...

Now, open src/nest-knex-client/nest-knex-client.controller.ts and plug in some queries. Of course this is not really a great design pattern (invoking database services directly from your controller), but is really just a quick test that the Knex.js integration works. In reality, you'll want to delegate any such access to true NestJS services as per NestJS best practices.

Here's an idea of what you can try. This test relies on the following database table being available (syntax below works with PostgreSql, but may require slight tweaks for other databases):

CREATE TABLE cats
(
   id     serial    NOT NULL,
   name   text,
   age    integer,
   breed  text
);

ALTER TABLE cats
   ADD CONSTRAINT cats_pkey
   PRIMARY KEY (id);

And here's a sample of what you could try in your test controller. Note that you have the full power of Knex.js available via the knex object. See here for lots more interesting samples of things you can do with Knex. Since you have a handle to the knex object, all of those Knex Query Builder methods should just work! In this admittedly silly example, we are both creating and querying some cats in our index route, using the knex object to access the database.

// src/nest-knex-client/nest-knex-client.controller.ts
import { Controller, Get } from '@nestjs/common';
import { NestKnexService } from '../nest-knex.service';

@Controller()
export class NestKnexClientController {
  constructor(private readonly nestKnexService: NestKnexService) {}

  @Get()
  async index() {
    const knex = this.nestKnexService.getKnex();
    const newcat = await knex('cats').insert({
      name: 'Fred',
      age: 5,
      breed: 'tom cat',
    });

    const cats = await knex.select('*').from('cats');

    return cats;
  }
}

Now, fire up the app with npm run start:dev, and browse to http://localhost:3000, and you should see the results of your query!

If you want to test the module with a separate app -- really demonstrating the power of the reusable library module you just built -- read on to see how to upload it as an npm package in one step. And for convenience, you can download a full client app that imports the module here and use it to test your newly minted Knex.js module. In fact, knex-cats is a fully working example, so if you've been reading along without coding, that's the easiest way to see the finished product.

Bonus Section

Hopefully I kept my promise of showing you how quickly you can generate a dynamic NestJS module that wraps an external resource library in just a few minutes! We're done -- you have a full-fledged Knex.js API at your disposal now! The queries we did in the previous section are just the tip of the iceberg, as Knex.js has a boat load of cool features.

Let's cover a couple more topics to round out the discussion.

A Better API

As mentioned, it's not good practice to access the knex object directly from our controller. We can make things a little bit better by creating a service to house our database logic. We can also make our Knex.js module a little easier to use by adding a higher level API. Let's do that. We're going to add a provider that let's us directly inject the knex object into any service. You'll see how this cleans the API up in a moment.

First, create a file called nest-knex-connection.provider.ts inside the src folder. Add the following code:

// src/nest-knex-connection.provider.ts
import { KNEX_CONNECTION } from './constants';
import { NestKnexService } from './nest-knex.service';

export const connectionFactory = {
  provide: KNEX_CONNECTION,
  useFactory: async nestKnexService => {
    return nestKnexService.getKnex();
  },
  inject: [NestKnexService],
};

To follow best practices for providers, we are using the KNEX_CONNECTION constant as our provider injection token. Be sure to add this line to src/constants.ts:

export const KNEX_CONNECTION = 'KNEX_CONNECTION';

See what we're doing here? We're binding an injection token (KNEX_CONNECTION) to a factory so that we can use our knex API object directly, avoiding the need to instantiate the service. For example, once we complete one more step (below), we'll be able to access the knex object in our controller as follows (notice the slightly simplified technique between old and new in the sample code):

// src/nest-knex-client/nest-knex-client.controller.ts
import { Controller, Inject, Get } from '@nestjs/common';
import { KNEX_CONNECTION } from '../constants';

@Controller()
export class NestKnexClientController {
  // new
  constructor(@Inject(KNEX_CONNECTION) private readonly knex) {}
  // old
  // constructor(private readonly nestKnexService: NestKnexService) {}

  @Get()
  async index() {
    // following line no longer needed
    // const knex = this.nestKnexService.getKnex();
    const newcat = await this.knex('cats').insert({
      name: 'Fred',
      age: 5,
      breed: 'tom cat',
    });

    const cats = await this.knex.select('*').from('cats');

    return cats;
  }
}

To wire this new provider into our module, open up src/nest-knex.module.ts and make the following changes:

Import the connectionFactory
Add the connectionFactory to the @Module metadata providers and exports properties. This section of the file will now look like this:

import { connectionFactory } from './nest-knex-connection.provider';

@Global()
@Module({
  providers: [NestKnexService, connectionFactory],
  exports: [NestKnexService, connectionFactory],
})

The complete implementation of our best practices for database access would also have us create a service that accesses the KNEX_CONNECTION provider, rather than doing that in a controller, but I'll leave that as an exercise for the reader. 😉

Dynamic Registration (with a Config Factory)

In the dynamic modules article, we talked about using asynchronous options providers. By that, we meant that rather than use a statically declared object containing our connection params like this:

    NestKnexModule.register({
      client: 'pg',
      connection: {
        host: 'localhost',
        user: 'john',
        password: 'mypassword',
        database: 'nest',
        port: 5432,
      },
    }),

We'd instead like to supply our connection options in a dynamic fashion. Good news! This is already built-in to the generated package. Let's test it out. For simplicity, we'll just use an in-place factory, but you can use the full power of class-based, factory based, and existing providers. To test it, we'll update the registration of the NestKnexModule in src/nest-knex-client/nest-knex-client.module.ts to look like this:

// src/nest-knex-client/nest-knex-client/module.ts
    NestKnexModule.registerAsync({
      useFactory: () => {
        return {
          debug: false,
          client: 'pg',
          connection: {
            host: 'localhost',
            user: 'john',
            password: 'mypassword',
            database: 'nest',
            port: 5432,
          },
        };
      },
    }),

It's not a terribly exciting example, but you get the idea. You can take advantage of any of the asynchronous provider methods to supply the configuration data to the module. If you check out the knex-cats sample, you'll see some more robust use of a config module to configure our Knex module dynamically.

Publish the Package

If you read my earlier Publishing NestJS Packages with npm article, you may have noticed that the structure of this package (e.g., the package.json file, the tsconfig.json file, the presence of the index.ts file in the root folder) follows that pattern exactly. As a result, publishing this package is as simple as this (assuming you have an npmjs.com account, that is):

npm publish

Seriously, that's it!

Of course, you can always use this ready-made @nestjsplus/knex package, built exactly as describe in this article.

Not Just External APIs!

The @nestjsplus/dyn-schematics package not only supports generating a standalone package, as described so far, but also generating a regular dynamic module inside an existing project. I won't cover that in detail here, but the use-case is simple. Let's say you're building a ConfigModule for use within a project. Simply scaffold it with the dynmod schematic (instead of dynpkg), and it will generate a new dynamic module into your existing project, fully ready for you to implement the details of your register() and registerAsync() static methods.

This works exactly like the normal CLI commands for adding elements like controllers, services, and modules. In fact, its behavior is very parallel to nest generate module. It builds the dynamic module and all its parts (which look quite similar to what we've seen in this article, minus the top level Nest application components) in a folder inside your project, and wires them up to the rest of the project just as the normal module schematic does. Read more about it at the github page for @nestjsplus/dyn-schematics. This use-case (adding a dynamic module to an existing project) is covered here. You can easily give it a quick dry run inside an existing Nest project with:

nest g -c @nestjsplus/dyn-schematics dynmod myNewModule --dry-run

Conclusion

If you've been following this series, you've previously learned a powerful architectural pattern for building modular NestJS services and modules, and composing them into applications. Now, with the power of schematics, you can automatically generate the boilerplate code to implement this powerful pattern. You can use this schematic to customize your own internal modules, or to easily integrate external APIs. In future articles, I'll show you how to build your own schematics to extend the power of NestJS and the NestJS CLI even further!

Feel free to ask questions, make comments or suggestions, or just say hello in the comments below. And join us at Discord for more happy discussions about NestJS. I post there as Y Prospect.

Advanced NestJS: How to build completely dynamic NestJS modules

John Biundo — Mon, 16 Sep 2019 15:02:26 +0000

John is a member of the NestJS core team

So you've noticed the cool ways in which you can dynamically configure some of the out-of-the-box NestJS modules like @nestjs/jwt, @nestjs/passport and @nestjs/typeorm, and want to know how to do that in your own modules? This is the right article for you! 😃

Intro

Over on the NestJS documentation site, we recently added a new chapter on dynamic modules. This is a fairly advanced chapter, and along with some recent significant improvements to the asynchronous providers chapter, Nest developers have some great new resources to help build configurable modules that can be assembled into complex, robust applications.

This article builds on that foundation and takes it one step further. One of the hallmarks of NestJS is making asynchronous programming very straightforward. Nest fully embraces Node.js Promises and the async/await paradigm. Coupled with Nest's signature Dependency Injection features, this is an extremely powerful combination. Let's look at how these features can be used to create dynamically configurable modules. Mastering this skill lets you build modules that are re-usable in any context. This enables fully context-aware, re-usable packages (libraries), and lets you assemble apps that deploy smoothly across cloud providers and throughout the DevOps spectrum - from development to staging to production.

Basic Dynamic Modules

In the dynamic modules chapter, the end result of the code sample is the ability to pass in an options object to configure a module that is being imported. After reading that chapter, we know how the following code snippet works via the dynamic module API.

import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { ConfigModule } from './config/config.module';

@Module({
  imports: [ConfigModule.register({ folder: './config' })],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

If you've played around with any NestJS modules - such as @nestjs/typeorm, @nestjs/passport or @nestjs/jwt - you'll have noticed that they go beyond the features described in that chapter. In addition to supporting the register(...) method shown above, they also support a fully dynamic and asynchronous version of the method. For example, with the @nestjs/jwt module, you can use a construct like this:

@Module({
  imports: [
    JwtModule.registerAsync({ useClass: ConfigService }),
  ]
})

With this construct, not only is the module dynamically configured, but the options passed to the dynamic module are themselves constructed dynamically. This is higher order functionality at its best. Configuration options are provided based on values extracted from the environment by the ConfigService, meaning they can be changed completely external to your feature code. Compare that to hardcoding a parameter, as with ConfigModule.register({ folder: './config'}), and you can immediately see the win.

In this article, we'll further explore why you may need this feature, and how to build it. Make sure you have a firm grasp on the concepts in the Custom providers and Dynamic modules chapters before moving on to the next section.

Async Options Providers Use-case

The section heading above is quite a mouthful! What the heck is an async options provider?

To answer that, first consider once again that our example above (the ConfigModule.register({ folder: './config'}) part) is passing a static options object in to the register() method. As we learned in the Dynamic modules chapter, this options object is used to customize the behavior of the module. (Review that chapter before proceeding if this concept is not familiar). As mentioned, we're now going to take that concept one step further, and let our options object be provided dynamically at run-time.

Async (Dynamic) Options Example

To give us a concrete working example for the remainder of this article, I'm going to introduce a new module. We'll then walk through how to use async options providers in that context.

I recently published the @nestjsplus/massive package to enable a Nest project to easily make use of the impressive MassiveJS library to do database work. We'll study the architecture of that package and use parts of it for the code we analyze in this article. Briefly, the package wraps the MassiveJS library into a Nest module. MassiveJS provides its entire API to each consumer module (e.g., each feature module) through a db object that has methods like:

db.find() to retrieve database records
db.update() to update database records
db.myFunction() to execute scripts or database procedures/functions

The primary function of the @nestjsplus/massive package is to make a connection to the database, and return the db object. In our feature modules, we then access the database using methods hung off the db object, like those shown above. Right off the bat, it should be clear that in order to establish a database connection, we need to pass in some connection parameters. In our case, with PostgreSQL, those parameters would be something like:

{
  user: 'john',
  password: 'password',
  host: 'localhost',
  port: 5432,
  database: 'nest',
}

What we quickly realize is that it's not optimal to hard-code those connection parameters in our Nest app. The conventional solution is to supply them through some sort of Configuration Module. And that's exactly what we can do with Nest's @nestjs/jwt module as shown in the example above. That, in a nutshell, is the purpose of async options providers. Now let's figure out how to do that in our Massive module.

Coding for Async Options Providers

To begin with, we can imagine supporting a module import statement using the same construct we found in @nestjs/jwt, like this:

@Module({
  imports: [
    MassiveModule.registerAsync({ useClass: ConfigService })
  ],
})

If this doesn't look familiar, take a quick look at this section of the Custom providers chapter. The similarities are deliberate. We're taking inspiration from the concepts learned from Custom providers to build a more flexible means of supplying options to our dynamic modules.

Let's dig in to the implementation. This is going to be a long section, so take a deep breath, maybe refill your coffee cup, but don't worry - we can do this! 😄. Bear in mind that we are walking through a design pattern that, once you understand it, you can confidently cut-and-paste as boilerplate to start building any module that needs dynamic configuration. But before the cutting and pasting starts, let's make sure to understand the template so we can customize it to our needs. Just remember that you won't have to write this from scratch each time!

First, let's revisit what we expect to accomplish with the registerAsync(...) construct above. Basically, we're saying "Hey module! I don't want to give you the option property values in code. How about instead I give you a class that has a method you can call to get the option values?". This would give us a great deal of flexibility in how we can generate the options dynamically at run-time.

This implies that our dynamic module is going to need to do a little more work in this case, as compared to the static options technique, to acquire its connection options. We're going to work our way up to that result. We begin by formalizing some definitions. We're trying to supply MassiveJS with its expected connection options, so first we'll create an interface to model that:

export interface MassiveConnectOptions {
  /**
   * server name or IP address
   */
  host: string;
  /**
   * server port number
   */
  port: number;
  /**
   * database name
   */
  database: string;
  /**
   * user name
   */
  user: string;
  /**
   * user password, or a function that returns one
   */
  password: string;
  /**
   * use SSL (it also can be a TSSLConfig-like object)
   */

  ...
}

There are actually more options available (we can see what they are from examining the MassiveJS connection options documentation), but let's keep the choices basic for now. The ones we modelled are required to establish a connection. As a side note, we're using JSDoc to document them so that we get a nice Intellisense developer experience when we later use the module.

The next concept to grapple with is as follows. Since our consumer module (the one calling registerAsync() to import the MassiveJS module) is handing us a class and expecting us to call a method on that class, we can surmise that we'll probably need to use some sort of factory pattern. In other words, somewhere, we're going to have to instantiate that class, call a method on it, and use the result returned from that method call as our connection options, right? Sounds (kind of) like a factory. Let's go with that concept for now.

Let's describe our prospective factory with an interface. The method could be something like createMassiveConnectOptions(). It needs to return an object of type MassiveConnectOptions (the interface we defined a minute ago). So we have:

interface MassiveOptionsFactory {
  createMassiveConnectOptions(): Promise<MassiveConnectOptions> | MassiveConnectOptions;
}

Nice! We can return the options object directly, or return a Promise that will resolve to an options object. Nest makes it super easy to support either. Hence, we now see the "async" part of our async options provider coming into play.

Now, let's ask the following: what mechanism is going to actually call our factory function at run time, take the resulting options object, and make it available to the part of our code that needs it? Hmmm... if only we had some general purpose mechanism. Maybe a feature that we could register an arbitrary object (or function returning an object) with at run-time, and then have that object passed into a constructor. Anybody got any ideas? 😉

Well of course, we've got the awesome NestJS Dependency Injection system at our disposal. That seems like a good fit! Let's figure out how to do that.

The recipe for binding something to the Nest IoC container, and later having it injected, is captured in an object called a provider. Let's whip up an options provider that will meet our needs. If you need a quick refresher course on custom providers, go ahead and re-read the Custom providers chapter now. It won't take long. I'll wait right here.

OK, so now you remember that we can define our options provider with a construct like the following. We already have an intuition that we need a factory provider, so this seems like the right construct:

{
  provide: 'MASSIVE_CONNECT_OPTIONS',
  useFactory: // <-- we need to get our options factory inserted here!
  inject:     // <-- we need to supply injectable parameters for useFactory here!
}

Let's tie a few things together. We're in pretty deep, so now's a good time to do a quick refresher on the big picture, and assess where we're at:

We're writing code that constructs, then returns, a dynamic module (our registerAsync() static method will house that code).
The dynamic module it returns can be imported into other feature modules, and provides a service (the thing that connects to the database and returns a db object).
That service needs to be configured at the time the module is constructed. A more helpful way to say this is that the service depends on a dynamically constructed options object.
We're going to construct that configuration options object at run-time, using a class that the consuming module hands us.
That class contains a method that knows how to return an appropriate options object.
We're going to use the NestJS Dependency Injection system to do the heavy lifting to manage that options object dependency for us.

OK, so we're working on steps 4, 5, and 6 right now. We're not yet ready to assemble the entire dynamic module. Before we do that, we have to work out the mechanics of our options provider. Returning to that task, we should be able to see how to fill in the blanks in the skeleton options provider we sketched out earlier (see the lines annotated <-- we need to... above). We can fill in those values based on how the registerAsync() call was made:

@Module({
  imports: [
    MassiveModule.registerAsync({ useClass: ConfigService })
  ],
})

Let's go ahead and fill them in now based on what we know. We'll sketch a static version of the object, just to see what we're trying to generate dynamically in the code we're about to write:

{
  provide: 'MASSIVE_CONNECT_OPTIONS',
  useFactory: async (ConfigService) =>
    await ConfigService.createMassiveConnectOptions(),
  inject: [ConfigService]
}

So we've now figured out what our generated options provider should look like. Good so far? It's important to remember that the 'MASSIVE_CONNECT_OPTIONS' provider is just fulfilling a dependency inside the dynamic module. Now that I mention it, we haven't really looked at the service that depends on the 'MASSIVE_CONNECT_OPTIONS' provider we're working so hard to supply. Let's connect a few more dots there and take a quick moment to consider that service. The service -- the one that connects and returns a db object -- is, predictably enough, declared in the MassiveService class. It's surprisingly straightforward:

@Injectable()
export class MassiveService {
  private _massiveClient;

  constructor(@Inject('MASSIVE_CONNECT_OPTIONS') private _massiveConnectOptions) {}

  async connect(): Promise<any> {
    return this._massiveClient
      ? this._massiveClient
      : (this._massiveClient = await massive(this._massiveConnectOptions));
  }
}

The MassiveService class injects the connection options provider and uses that information to make the API call needed to asynchronously create a database connection (await massive(this._massiveConnectOptions)). Once made, it caches the connection so it can return an existing connection on subsequent calls. That's it. That's why we're jumping through hoops to be able to pass in our options provider.

We've now worked out the concepts and sketched out the piece parts of our dynamically configurable module. We're ready to start assembling them. First we'll write some glue code to pull this all together. As we learned in the dynamic modules chapter, all that glue should live in the module definition class. Let's create the MassiveModule class for that purpose. We'll describe what's happening in this code just below it.

@Global()
@Module({})
export class MassiveModule {

  /**
   *  public static register( ... )
   *  omitted here for brevity
   */

  public static registerAsync(
    connectOptions: MassiveConnectAsyncOptions,
  ): DynamicModule {
    return {
      module: MassiveModule,
      providers: [
        MassiveService,
        ...this.createConnectProviders(connectOptions),
      ],
      exports: [MassiveService],
    };
  }

  private static createConnectProviders(
    options: MassiveConnectAsyncOptions,
  ): Provider[] {
    return [
      {
        provide: 'MASSIVE_CONNECT_OPTIONS',
        useFactory: async (optionsFactory: MassiveOptionsFactory) =>
          await optionsFactory.createMassiveConnectOptions(),
        inject: [options.useClass],
      },
      {
        provide: options.useClass,
        useClass: useClass,
      }
    ];
  }

Let's get a firm handle on what this code does. This is really where the rubber meets the road, so take time to understand it carefully. Consider that if we were to insert a logging statement displaying the return value from the following call:

registerAsync({ useClass: ConfigService });

we'd see an object that looks pretty much like this:

{
  module: MassiveModule,
  providers: [
    MassiveService,
    {
      provide: 'MASSIVE_CONNECT_OPTIONS',
      useFactory: async (optionsFactory: MassiveOptionsFactory) =>
        await optionsFactory.createMassiveConnectOptions(),
      inject: [ ConfigService ],
    },
    {
      provide: ConfigService,
      useClass: ConfigService,
    }
  ],
  exports: [ MassiveService ]
}

This should be pretty recognizable, as it would plug right into a standard @Module() decorator to declare module metadata (well, all except for the module property, which is part of the Dynamic module API). To describe it in English, we're returning a dynamic module that declares three providers, exporting one of them for use in other modules that may import it.

The first provider is obviously the MassiveService itself, which we plan to use in our consumer's feature modules, so we duly export it.
The second provider ('MASSIVE_CONNECT_OPTIONS') is only used internally by the MassiveService to ingest the connection options it needs (notice that we do not export it). Let's take a little closer look at that useFactory construct. Note that there's also an inject property, which is used to inject the ConfigService into the factory function. This is described in detail here in the Custom providers chapter, but basically, the idea is that the factory function takes optional input arguments which, if specified, are resolved by injecting a provider from the inject property array. You might be wondering where that ConfigService injectable comes from. Read on 😉.
Finally, we have a third provider, also used only internally by our dynamic module (and hence not exported), which is our single private instance of the ConfigService. So, Nest is going to instantiate a ConfigService inside the dynamic module context (this makes sense, right? We told our module to useClass, which means "create your own instance"), and that will be injected into the factory.

If you made it this far - congrats! That was the hardest part. We just worked out all the mechanics of assembling a dynamically configurable module. The rest of the article is gravy!

One other thing that should be obvious from looking at the generated useFactory syntax above is that the ConfigService class must implement a createMassiveConnectOptions() method. This should be a familiar pattern if you're already using some sort of configuration module that implements various functions to return options of a particular shape for each service it gets plugged into. Now perhaps you can see a little more clearly how this all fits together.

Variant Forms of Asynchronous Options Providers

What we've built so far allows us to configure the MassiveModule by handing it a class whose purpose is to dynamically provide connection options. Let's just remind ourselves again how that looks from the consumer perspective:

@Module({
  imports: [
    MassiveModule.registerAsync({ useClass: ConfigService})
  ]
})

We can refer to this as configuring our dynamic module with a useClass technique (AKA a class provider). Are there other techniques? You may recall seeing several other similar patterns in the Custom providers chapter. We can model our registerAsync() interface based on those patterns. Let's sketch out what those techniques would look like from a consumer module perspective, and then we can easily add support for them.

Factory Providers: useFactory

While we did make use of a factory in the previous section, that was strictly internal to the dynamic module construction mechanics, not a part of the callable API. What would useFactory look like when exposed as an option for our registerAsync() method?

@Module({
  imports: [MassiveModule.registerAsync({
    useFactory: () => {
      return {
        host: "localhost",
        port: 5432,
        database: "nest",
        user: "john",
        password: "password"
      }
    }
  })]
})

In the sample above, we supplied a very simple factory in place, but we could of course plug in (or pass in a function implementing) any arbitrarily sophisticated factory as long as it returns an appropriate connections object.

Alias Providers: useExisting

This sometimes-overlooked construct is actually extremely useful. In our context, it means we can ensure that we re-use an existing options provider rather than instantiating a new one. For example, useClass: ConfigService will cause Nest to create and inject a new private instance of our ConfigService. In the real world, we'll usually want a single shared instance of the ConfigService injected anywhere it's needed, not a private copy. The useExisting technique is our friend here. Here's how it would look:

@Module({
  imports: [MassiveModule.registerAsync({
    useExisting: ConfigService
  })]
})

Supporting Multiple Async Options Providers Techniques

We're in the home stretch. We're going to focus now on generalizing and optimizing our registerAsync() method to support the additional techniques described above. When we're done, our module will support all three techniques:

useClass - to get a private instance of the options provider.
useFactory - to use a function as the options provider.
useExisting - to re-use an existing (shared, SINGLETON) service as the options provider.

I'm going to jump right to the code, as we're all getting weary now 😉. I'll describe the key elements below.

@Global()
@Module({
  providers: [MassiveService],
  exports: [MassiveService],
})
export class MassiveModule {

  /**
   *  public static register( ... )
   *  omitted here for brevity
   */

  public static registerAsync(connectOptions: MassiveConnectAsyncOptions): DynamicModule {
    return {
      module: MassivconnectOptions.imports || [],eModule,
      imports:
      providers: [this.createConnectProviders(connectOptions)],
    };
  }

  private static createConnectProviders(
    options: MassiveConnectAsyncOptions,
  ): Provider[] {
    if (options.useExisting || options.useFactory) {
      return [this.createConnectOptionsProvider(options)];
    }

    // for useClass
    return [
      this.createConnectOptionsProvider(options),
      {
        provide: options.useClass,
        useClass: options.useClass,
      },
    ];
  }

  private static createConnectOptionsProvider(
    options: MassiveConnectAsyncOptions,
  ): Provider {
    if (options.useFactory) {

      // for useFactory
      return {
        provide: MASSIVE_CONNECT_OPTIONS,
        useFactory: options.useFactory,
        inject: options.inject || [],
      };
    }

    // For useExisting...
    return {
      provide: MASSIVE_CONNECT_OPTIONS,
      useFactory: async (optionsFactory: MassiveConnectOptionsFactory) =>
        await optionsFactory.createMassiveConnectOptions(),
      inject: [options.useExisting || options.useClass],
    };
  }
}

Before discussing the details of the code, let's cover a few superficial changes to make sure they don't trip you up.

We now use the constant MASSIVE_CONNECT_OPTIONS in place of a string-valued token. This is a simple best practice convention, covered at the end of this section of the docs.
Rather than listing MassiveService in the providers and exports properties of the dynamically constructed module, we promoted them up to live in the @Module() decorator metadata. Why? Partly style, and partly to keep the code DRY. The two approaches are equivalent.

Fully Understanding the Code

You should be able to trace the path through this code to see how it handles each case uniquely. I highly recommend you do the following exercise. Construct an arbitrary registerAsync() registration call on paper, and walk through the code to predict what the returned dynamic module will look like. This will strongly reinforce the patterns and help you firmly connect all the dots.

For example, if we were to code:

@Module({
  imports: [MassiveModule.registerAsync({
    useExisting: ConfigService
  })]
})

We could expect a dynamic module to be constructed with the following properties:

{
  module: MassiveModule,
  imports: [],
  providers: [
    {
      provide: MASSIVE_CONNECT_OPTIONS,
      useFactory: async (optionsFactory: MassiveConnectOptionsFactory) =>
        await optionsFactory.createMassiveConnectOptions(),
      inject: [ ConfigService ],
    },
  ],
}

(Note: because the module is preceded by an @Module() decorator that now lists MassiveService as a provider and exports, it our resulting dynamic module will also have those properties. Above we are just showing the elements that get added dynamically.)

Consider another question. How is the ConfigService available inside the factory for injection in this useExisting case? Well - heh heh - that's kind of a trick question. In the sample above, I assumed that it was already visible inside the consuming module -- perhaps as a global module (one declared with @Global()). Let's say that wasn't true, and that it lives in ConfigModule which has not somehow registered ConfigService as a global provider. Can our code handle this? Let's see.

Our registration would instead look like this:

@Module({
  imports: [MassiveModule.registerAsync({
    useExisting: ConfigService,
    imports: [ ConfigModule ]
  })]
})

And our resulting dynamic module would look like this:

{
  module: MassiveModule,
  imports: [ ConfigModule ],
  providers: [
    {
      provide: MASSIVE_CONNECT_OPTIONS,
      useFactory: async (optionsFactory: MassiveConnectOptionsFactory) =>
        await optionsFactory.createMassiveConnectOptions(),
      inject: [ ConfigService ],
    },
  ],
}

Do you see how the pieces fit together?

Another exercise is to ponder the difference in the code paths when you use useClass vs. useExisting. The important point is how we either instantiate a new ConfigService object, or inject an existing one. It's worth working through those details, as the concepts will give you a full picture of how NestJS modules and providers fit together in a coherent way. But this article is already too long, so I'll leave that as an exercise for you, dear reader. 😄

If you have questions, feel free to ask in the comments below!

Conclusion

The patterns illustrated above are used throughout Nest's add-on modules, like @nestjs/jwt, @nestjs/passport and @nestjs/typeorm. Hopefully you now see not only how powerful these patterns are, but how you can make use of them in your own project.

As a next step, you may want to consider browsing through the source code of those modules, now that you have a roadmap. You can also see a slightly evolved version of the code in this article in the @nestjsplus/massive repository (while you're there, maybe give it a quick ⭐ if you like this article 😉). The main difference between the code in this article and that repo is that the production version needs to handle multiple asynchronous options providers, so there's a tiny bit more plumbing.

Now you can confidently start using these powerful patterns in your own code to create robust and flexible modules that work reliably in a wide variety of contexts.

As a final bonus, if you're building an Open Source package for public use, just combine this technique with the steps described in my last article on publishing NPM packages, and you're all set.

Feel free to ask questions, make comments or suggestions, or just say hello in the comments below. And join us at Discord for more happy discussions about NestJS. I post there as Y Prospect.

Publishing NestJS Packages with npm

John Biundo — Mon, 19 Aug 2019 17:50:38 +0000

John is a member of the NestJS core team, primarily contributing to the documentation.

This is the first in a multi-part series on building re-usable npm packages for NestJS. This article focuses on the mechanics of the process:

building a simple npm package
consuming and testing it locally in a NestJS app
publishing it to NPM

This article lays the foundation for future articles in this series, which delve into more complex NestJS packages.

Intro

We all interact with the npm registry on a daily basis. Usually the registry itself, and how it works, are conveniently relegated to the background - we don't have to think about the details when we run npm install. But sometimes we need to work with the registry more directly - perhaps we're publishing a core infrastructure/utility package for private use across teams within an organization, or maybe we're releasing an open source package for public use.

Publishing a package for the first time, then seeing it install with npm install is pretty fun.

And it's surprisingly easy. While there are plenty of general npm tutorials out there, I've created this tutorial, along with a package starter repository that you can quickly clone to set things up in a standard way, and skip past some of the little hurdles. While there really isn't anything Nest-specific about this, TypeScript can introduce a wrinkle or two not typically covered in other npm tutorials. And combined with the the NestJS environment, you can enjoy a productive iterative development/deployment workflow if you follow a few basic steps. That's what I'll cover here. Follow along, and you should be able to publish a basic NestJS package in about 15 minutes.

As to why you want to do this, I can think of at least two great reasons:
1) Learning how to publish a package demystifies one of the things we tend to take for granted. It probably ought to be on the resume of any self-respecting full stack developer. 😃
2) Whether you publish to the npmjs.com registry, or a private one, publishing an npm package is the standard way of creating and sharing re-usable packages.

Again, while the sample package we'll build in this tutorial is trivial, this article lays the foundation for several more articles that will go into detail on much more sophisticated modules. Stay tuned!

Pre-requisites

If you plan to follow the steps in this article, make sure you have an account at www.npmjs.com (you can use a free one for this exercise, and indeed, unless you want to publish private packages, you can stick with a free account). You could also publish to a private npm registry like verdaccio, but setting that up is beyond the scope of this article.

Setting up for development

If you're live-coding along with reading this article, I have a couple of brief recommendations.

First, note that we'll actually be working on two separate but related bits of code. One for the npm package, and one for a companion NestJS app that will consume and exercise the package.

I recommend sticking with the names used below to keep things properly synchronized. If you're building your own real package with its own unique name, just make sure to synchronize the name you choose across the various places it occurs (folders, package.json, npmjs, git, etc.).

Next, I recommend you open two command line windows (as well as two editing sessions) to work on both bits of code simultaneously. This will help you get the feel of a useful development pattern for building a package in an iterative fashion. The rest of this tutorial will refer to these two command line windows. If you prefer to work in a single command line window, just pay careful attention to what folder you're in as you execute each step!

Setting up the folders

In terminal window 1, create a new folder. This will be the parent folder for both parts of the project: the package we're building, and our small companion NestJS app.



mkdir nestmod && cd nestmod

Clone the module starter repo

In terminal window 1, start by cloning the nestjs-package-starter repo. This is a starter repo that sets up a lot of the default details for your NestJS related npm package. These details can be a bit tedious to get right.



git clone https://github.com/nestjsplus/nestjs-package-starter.git

Once completed, this step should result in a brand new package template in the nestjs-package-starter sub-folder.

Now install its dependencies. Still in terminal window 1:



cd nestjs-package-starter
npm install

Create the test app

In your second terminal window, make sure you start out in the top level folder you created (I created nestmod above so that's what I would use; use whatever folder you created to follow along with the tutorial). Scaffold the small NestJS app we'll be using to exercise our package.



nest new test-app

Choose your preferred package manager (npm or yarn), and wait a moment while the Nest CLI builds your starter app.

Your folder structure should look similar to this now:



nestmod
└─── nestjs-package-starter
│   └───node_modules
│   └───src
│   ...
└─── test-app
│   └───node_modules
│   └───src
│   ...

Build the package

Take a moment to poke around in the nestjs-package-starter folder. Here are a few things to notice:

1) The src folder has two files:

test.ts is the entire functionality of our package; it exports one simple test function we'll import into our Nest test-app to prove to ourselves that we've actually installed and are using the package.
index.ts exports the test.ts function from the folder (it's a barrel file).

2) The package.json file has a number of interesting parts. We'll delve into several of these throughout the tutorial. One to notice now is the different types of package dependencies that are apparent:

regular dependencies are those required to run our code; these should be anything in addition to, but not including NestJS itself. For example, if we were using the dotenv package in our code, this is something that is not provided by NestJS itself, so it would belong in dependencies.
peerDependencies: since this is a NestJS-based package, we use this section to declare that it is compatible with Nest version 6 (a minimum of 6.0.0). This means we assume that the person using the package has a compatible NestJS environment that we can rely on. Mentioning just the @nestjs/common package is sufficient for this purpose. Specifying a version of ^6.0.0 gives us broad compatibility with any minor versions of NestJS the user may have installed.
devDependencies: these are the packages we need only for development; since we expect to build a Nest-based package, we need Nest in our development environment, along with testing utilities, TypeScript, etc. These dev dependencies should be pretty much identical to the regular devDependencies you have when building a normal Nest app. For example, compare these to the devDependencies entry in the test-app and they should be identical.

In terminal window 1, make sure you're still in the folder where you cloned the nestjs-package-starter, and build the package with:



npm run build

The npm build script is trivially simple: it just runs the TypeScript compiler in the folder configured as the rootDir in our tsconfig.json file. Speaking of which, let's take a quick look at tsconfig.json, which kind of "teams up" with package.json to control how our package is built.

The most notable things in the tsconfig.json file are:

declaration: true ensures that our type files are generated, which helps consumers of the package benefit from TypeScript type checking and editor features like intellisense.
the two Decorators flags ensure that TypeScript decorators work properly.
outDir controls where our compiled code is published.
rootdir, along with the top-level include and exclude entries, control which source code is compiled.

Take another quick peek at package.json, and note the "main": "dist/test.js" entry. This is the last piece of the "packaging puzzle". The build script compiles our TypeScript into JavaScript in the dist folder; the barrel file (index.ts) ensures that our function is exported and publicly visible; the "main" entry in our package.json tells the module loader where to discover the exported symbols from our package when it's imported.

So we've now compiled the TypeScript and prepared it for deployment in a package.

Install the package into the test app

You now have a full-fledged npm package, though it's only available locally. You can use the package in test-app with a familiar-looking npm command.

In terminal window 2, change to the folder created by the nest new test-app command, where our test app lives.



cd nestmod/test-app

Use npm to install the package we just built into our test app.



npm install ../nestjs-package-starter

The main difference from a normal npm install, I'm sure you've noticed, is that you are referring to a combination of a path (the .. part) and a name (the nestjs-package-starter part), rather than the typical usage (when pulling from the npmjs.com registry) which uses just a package name. This illustrates the simple process enabled by npm for developing packages: npm install works exactly the same for local packages as it does for remote packages downloaded from npmjs.com. You can see this clearly by noticing that in testapp/package.json, there's an entry like:
"@nestjsplus/nestjs-package-starter": "file:../nestjs-package-starter"

Use the package in the test app

The template package exports a single simple test function. Examine nestjs-package-starter/src/test.ts to see it:



// nestjs-package-starter/src/test.ts
export function getHello(): string {
  return 'Hello from the new package!';
}

Now that you've installed the new package in test-app, it's available like any npm package, and you can use it in the normal fashion. Open test-app/src/app.controller.ts and import the function; make sure the file looks like this:



// test-app/src/app.controller.ts
import { Controller, Get } from '@nestjs/common';
import { getHello } from '@nestjsplus/nestjs-package-starter';

@Controller()
export class AppController {
  @Get()
  getHello(): string {
    return getHello();
  }
}

In terminal window 2, start test-app (using start:dev is recommended here so we can make iterative changes):



npm run start:dev

In a browser, head on over to http://localhost:3000 and see the message coming from the imported package function.

If you've followed along so far, let's do one more thing that shows how easy it is to continue to iterate. Make a simple change to the getHello() function exported by the package (I like to change it to return 'Buon Giorno!' 😄).

Now, make sure the test app is still running in dev mode (e.g., npm run start:dev) in terminal window 2, then, in terminal window 1, rebuild the package with:



npm run build

Notice that in terminal window 2, since we're linked to the local package, the dev server will automatically restart as the package is rebuilt (it may show errors for a few seconds while the process completes). This highlights a key thing to keep in mind: whenever you make local changes to the package, you must rebuild it before it's visible to consumers of the package. Obviously this means you must rebuild before you publish it.

To make sure this last bit happens, take another look at the package.json file. The prepare script is a special script recognized by the npm publish command. Whenever you publish (which we'll do in a moment), the prepare script will be run before publishing. Note that there are several other such npm "hooks" available that will let you do things like run lint, check in/tag source code to your git repo, etc. See here for details.

When you refresh your page at localhost:3000, you should see your nice warm Italian greeting!

Publish the package

We're now ready to push our newborn package out of the nest and onto the 'net (ugghh, sorry for the bad pun). To complete this step, you'll need a free account at www.npmjs.com. If you don't have one, go to the signup page to get one.

Give the package a unique scoped name for npm

A few notes about npm

npm names
All npm packages have a name. If the name starts with @, it's a scoped name. Scoped names are useful to create a unique namespace for your packages. If you and I both publish nestjs-package-starter to npmjs.org, there will be a collision. Whoever publishes first owns the name. But if I publish a scoped package name, such as @johnbiundo/nestjs-package-starter, you can publish your own copy at @yournpmjsname/nestjs-package-starter, and they both happily co-exist. Anyone that wants mine will do npm install @johnbiundo/nestjs-package-starter, and anyone that wants yours will do npm install @yournpmjsname/nestjs-package-starter. Read more about scopes here.
npm public packages
npm lets you publish packages for free as long as they're public. If you want to publish private packages (say for sharing only within your company), you'll need a private account. Alternatively, you can set up an internal private registry with a registry server like verdaccio.
npm orgs
In addition to publishing packages scoped under your name, you can create a free npm org to give your packages a more custom name (scope). For example, I publish several packages under the org @nestjsplus. Read more about npm orgs here.

Once you're ready, open the package.json file for the package and change the following entries:

name: your scoped package name, something like @mynpmjsname/my-new-package.
version: choose a starting version number; I comment a bit more about this below, but something like 1.0.0 will work fine for now.
author: your name and email, in a format like John Biundo <john@email.com>.

Later, you'll want to customize other things like the description, license, keywords, repository links, etc. None of those are very TypeScript or NestJS-specific, so we won't cover them here, but a little Googling will turn up lots of good articles and help for them.

Once ready, publish the package

Note: Each time you re-publish the package, you'll have to change the version field in package.json (or the publish step will fail). It's best practice to use semantic versioning ("semver") for npm packages.

In terminal window 1, run:



npm publish

The output of this step shows you exactly what files are bundled up and sent to npmjs.org to create your package, as well as the version number and some other metadata.

Search for your package on www.npmjs.com

Head on over to https://www.npmjs.org, and search for your package.

Install the published npm package in your test app

Now that the package is live on npm, you can install it as you would any package.

In terminal window 2, first, uninstall the local one we created and installed about 10 minutes ago.



npm uninstall @nestjsplus/nestjs-package-starter

Now, install your shiny new npm package from the internet. Change the names as appropriate below:



npm install @yournpmjsname/your-package

Finally, edit your test app to reflect the new npm package name (paying attention to using the correct "scoped" name if you published that way). For example:



// src/app.controller.ts
import { Controller, Get } from '@nestjs/common';
import { getHello } from '@yournpmjsname/your-new-package';

@Controller()
export class AppController {
  @Get()
  getHello(): string {
    return getHello();
  }
}

Fire up the app, and you'll be using your new package!

Now that you know how easy it is to publish an npm package, stay tuned for the next articles in this series where we'll cover building more interesting NestJS modules.

Feel free to ask questions, make comments or suggestions, or just say hello in the comments below. And join us at Discord for more happy discussions about NestJS. I post there as Y Prospect.

Authentication and Sessions for MVC Apps with NestJS

John Biundo — Mon, 22 Jul 2019 14:29:22 +0000

John is a member of the NestJS core team, primarily contributing to the documentation.

Note: You can find all of the source code from this article on github here.

Intro

There's a lot of good information on building API servers in NestJS (AKA Nest). That use case is certainly a very strong fit for Nest.

Perhaps surprisingly, Nest is also very good at building traditional web applications - what Nest refers to as MVC apps in the documentation. By this I mean apps that you might otherwise consider building in ExpressJS + Handlebars, Laravel, or Ruby on Rails, with template-driven server-side rendering of pages. Users considering Nest for this use case might find the resources in this area to be a little lighter than the API server use case.

This is probably partially due to the fact that most of the information you need to build say, an ExpressJS + Handlebars app, is already well covered and easy to find. And most of that information applies directly to building such an app in Nest. But leaving it at that still leaves a few questions for the Nest newcomer to sort out on their own. This article attempts to provide a little more color on one aspect of the challenge: how to manage session-based authentication with Nest. Why is this different with Nest?

Fully answering that question probably needs one or more complete articles on its own. The short answer is: if you really want to take advantage of Nest, it helps to try to learn to do things "the Nest way". Nest makes no apologies for being opinionated. It has a certain way of approaching application architecture. If you're on-board with this approach, it helps to go all-in and try to build things the Nest way. This gives you amazing productivity advantages, helps you build DRY maintainable code, and lets you spend most of your time focusing on business logic, not boilerplate.

The Nest way will be familiar to many coming from a Java or .NET background, as many of the concepts - such as OO and dependency injection - are similar. But it should be equally accessible to anyone who has a background in TypeScript, Angular, or any modern FE or BE framework/enviroment. As with any such framework, there's plenty of room for individual choice and style, but following the general patterns of the framework can make you feel like you have the wind at your back instead of swimming against the tide.

Requirements

Phew! With that out of the way, let's get a little more specific. When it comes to the problem at hand, there are only a few Nest-specific techniques that you need to know to get a quick jump out of the starting blocks:

How to set up a template engine like Handlebars to work with Nest
How to integrate template rendering with Nest controllers
How to integrate express-session with Nest
How to integrate an authentication system (we'll specifically use Passport) with Nest

Items 3 and 4 are the main focus of this article. To get there, we'll wade slightly into items 1 and 2 (providing code samples and references to Nest docs that can help you go deeper), but we won't spend a lot of time there. Let me know in the comments if these are areas you'd like to see more about in future articles.

Let's take a quick look at the app we're going to build.

Our requirements are simple. Users will authenticate with a username and password. Once authenticated, the server will use Express sessions so that the user remains "logged in" until they choose to log out. We'll set up a protected route that is accessible only to an authenticated user.

Installation

If you haven't installed Nest before, the pre-requisites are simple. You just need a modern Node.js environment (>=8.9 will do) and npm or yarn. Then install Nest:



npm i -g @nestjs/cli

We start by installing the required packages, and building our basic routes.

Passport provides a library called passport-local that implements a username/password authentication strategy, which suits our needs for this use case. Since we are rendering some basic HTML pages, we'll also install the versatile and popular express-handlebars package to make that a little easier. To support sessions and to provide a convenient way to give user feedback during login, we'll also utilize the express-session and connect-flash packages.

Note For any Passport strategy you choose (there are many available here), you'll always need the @nestjs/passport and passport packages. Then, you'll need to install the strategy-specific package (e.g., passport-jwt or passport-local) that scaffolds the particular authentication strategy you are building.

With these basic requirements in mind, we can now start by scaffolding a new Nest application, and installing the dependencies:



nest new mvc-sessions
cd mvc-sessions
npm install --save @nestjs/passport passport passport-local express-handlebars express-session connect-flash 
npm install --save-dev @types/express @types/express-session @types/connect-flash @types/express-handlebars

Web interface

Let's start by building the templates we'll use for the UI of our authentication subsystem. Following a standard MVC type project structure, create the following folder structure (i.e., the public folder and its sub-folders):



mvc-sessions
└───public
│   └───views
│       └───layouts

Now create the following handlebars templates, and configure Nest to use express-handlebars as the view engine. Refer here for more on the handlebars template language , and here for more background on Nest-specific techniques for server-side rendered (MVC style) web apps.

Main layout

Create main.hbs in the layouts folder, and add the following code. This is the outermost container for our views. Note the {{{ body }}} line, which is where each individual view is inserted. This structure allows us to set up global styles. In this case, we're taking advantage of Google's widely used material design lite component library to style our minimal UI. All of those dependencies are taken care of in the <head> section of our layout.



<!-- public/views/layouts/main.hbs -->
<!DOCTYPE html>
<html>
  <head>
    <script src="https://code.getmdl.io/1.3.0/material.min.js"></script>
    <link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons">
    <link rel="stylesheet" href="https://code.getmdl.io/1.3.0/material.indigo-pink.min.css">
    <style>
      .mdl-layout__content {
        padding: 24px;
        flex: none;
      }
      .mdl-textfield__error {
        visibility: visible;
        padding: 5px;
      }
      .mdl-card {
        padding-bottom: 10px;
        min-width: 500px;
      }
    </style>
  </head>
  <body>
    {{{ body }}}
  </body>
</html>

Home page

Create home.hbs in the views folder, and add the following code. This is the page users land on after authenticating.



<!-- public/views/home.hbs -->
<div class="mdl-layout mdl-js-layout mdl-color--grey-100">
  <main class="mdl-layout__content">
    <div class="mdl-card mdl-shadow--6dp">
      <div class="mdl-card__title mdl-color--primary mdl-color-text--white">
        <h2 class="mdl-card__title-text">Welcome {{ user.username }}!</h2>
      </div>
      <div class="mdl-card__supporting-text">
        <div class="mdl-card__actions mdl-card--border">
          <a class="mdl-button" href='/profile'>GetProfile</a>
        </div>
      </div>
    </div>
  </main>
</div>

Login page

Create login.hbs in the views folder, and add the following code. This is the login form.



<!-- public/views/login.hbs -->
<div class="mdl-layout mdl-js-layout mdl-color--grey-100">
  <main class="mdl-layout__content">
    <div class="mdl-card mdl-shadow--6dp">
      <div class="mdl-card__title mdl-color--primary mdl-color-text--white">
        <h2 class="mdl-card__title-text">Nest Cats</h2>
      </div>
      <div class="mdl-card__supporting-text">
        <form action="/login" method="post">
          <div class="mdl-textfield mdl-js-textfield">
            <input class="mdl-textfield__input" type="text" name="username" id="username" />
            <label class="mdl-textfield__label" for="username">Username</label>
          </div>
          <div class="mdl-textfield mdl-js-textfield">
            <input class="mdl-textfield__input" type="password" name="password" id="password" />
            <label class="mdl-textfield__label" for="password">Password</label>
          </div>
          <div class="mdl-card__actions mdl-card--border">
            <button class="mdl-button mdl-button--colored mdl-js-button mdl-js-ripple-effect">Log In</button>
            <span class="mdl-textfield__error">{{ message }}</span>
          </div>
        </form>
      </div>
    </div>
  </main>
</div>

Profile page

Create profile.hbs in the views folder and add the following code. This page displays details about the logged in user. It's rendered on our protected route.



<!-- public/views/profile.hbs -->
<div class="mdl-layout mdl-js-layout mdl-color--grey-100">
  <main class="mdl-layout__content">
    <div class="mdl-card mdl-shadow--6dp">
      <div class="mdl-card__title mdl-color--primary mdl-color-text--white">
        <h2 class="mdl-card__title-text">About {{ user.username }}</h2>
      </div>
      <div>
        <figure><img src="http://lorempixel.com/400/200/cats/{{ user.pet.picId }}">
          <figcaption>{{ user.username }}'s friend {{ user.pet.name }}</figcaption>
        </figure>
        <div class="mdl-card__actions mdl-card--border">
          <a class="mdl-button" href='/logout'>Log Out</a>
        </div>
      </div>
    </div>
  </main>
</div>

Set up view engine

Now we tell Nest to use express-handlebars as the view engine. We do this in the main.ts file, which by convention is where your Nest app will be bootstrapped from. While we won't go into a lot of detail here, the main concepts are:

Pass the <NestExpressApplication> type assertion in the NestFactory.create() method call to gain access to native Express methods.
Having done that, you can now simply access any of Express's native app methods. For the most part, the configuration of the view engine, and any other traditional Express middleware, proceeds as normal as with the view engine setup shown below.

Modify the main.ts file so that it looks like this:



// src/main.ts
import { NestFactory } from '@nestjs/core';
import { NestExpressApplication } from '@nestjs/platform-express';
import { join } from 'path';
import { AppModule } from './app.module';
import * as exphbs from 'express-handlebars';

async function bootstrap() {
  const app = await NestFactory.create<NestExpressApplication>(AppModule);
  const viewsPath = join(__dirname, '../public/views');
  app.engine('.hbs', exphbs({ extname: '.hbs', defaultLayout: 'main' }));
  app.set('views', viewsPath);
  app.set('view engine', '.hbs');

  await app.listen(3000);
}
bootstrap();

Authentication routes

The final step in this section is setting up our routes. This is one area where the MVC architecture starts to shine through. It can be a little unfamiliar if you're used to bare Express routes, so let's wade in a little. Feel free to take a few minutes to read the Nest documentation on controllers for lots more details on how Nest Controllers work.

We'll modify app.controller.ts so that it looks like the large code block below.

Before we do that, let's take a moment to look at the GET / route handler in the large code block below (the line starting with @Get('/'). The work is done by the @Render() decorator. Since all we're doing in this first iteration of the code is rendering our templates, the code is trivial. @Render() takes a single argument, which is the name of the template to render. Think of this as the Nest equivalent of Express code like:



app.get('/', function (req, res) {
    res.render('login');
});

Methods decorated with @Render() can also return a value that will supply template variables. For now, we'll just do an empty return (implicitly returning undefined). Later, we'll use this feature to pass template variables.

Go ahead and update src/app.controller.ts with this code:



// src/app.controller.ts
import { Controller, Get, Post, Res, Render } from '@nestjs/common';
import { Response } from 'express';

@Controller()
export class AppController {
  @Get('/')
  @Render('login')
  index() {
    return;
  }

  @Post('/login')
  login(@Res() res: Response): void {
    res.redirect('/home');
  }

  @Get('/home')
  @Render('home')
  getHome() {
    return;
  }

  @Get('/profile')
  @Render('profile')
  getProfile() {
    return;
  }

  @Get('/logout')
  logout(@Res() res: Response): void {
    res.redirect('/');
  }
}

At this point, you should be able to run the app:



$ npm run start

Now, browse to http://localhost:3000 and click through the basic UI. At this point, of course, you can click through the pages without logging in.

Implementing Passport strategies

We're now ready to implement the authentication feature. It's helpful to have a good understanding of how Nest integrates with Passport. The Nest docs covers this in some depth. It's worth taking a quick detour to read that section.

Note: I authored that section of the NestJS docs, so you'll see some similarities/overlaps with the code and documentation below

The key takeaways are:

Nest provides the @nestjs/passport module which wraps Passport in a Nest style package that makes it easy to treat Passport as a provider.
You implement Passport strategies by extending the PassportStrategy class, where you implement the strategy-specific initialization and callback.

As mentioned, we'll utilize the passport-local strategy for this use case. We'll get to that implementation in a moment. Start by generating an AuthModule and in it, an AuthService:



nest g module auth
nest g service auth

We'll also implement a UsersService to manage our User store, so we'll generate that module and service now:



nest g module users
nest g service users

Replace the default contents of these generated files as shown below. For our sample app, the UsersService simply maintains a hard-coded in-memory list of users, and a method to retrieve one by username. In a real app, this is where you'd build your user model and persistence layer, using your library of choice (e.g., TypeORM, Sequelize, Mongoose, etc.).



// src/users/users.service.ts
import { Injectable } from '@nestjs/common';

@Injectable()
export class UsersService {
  private readonly users: any[];

  constructor() {
    this.users = [
      {
        userId: 1,
        username: 'john',
        password: 'changeme',
        pet: { name: 'alfred', picId: 1 },
      },
      {
        userId: 2,
        username: 'chris',
        password: 'secret',
        pet: { name: 'gopher', picId: 2 },
      },
      {
        userId: 3,
        username: 'maria',
        password: 'guess',
        pet: { name: 'jenny', picId: 3 },
      },
    ];
  }

  async findOne(username: string): Promise<any> {
    return this.users.find(user => user.username === username);
  }
}

In the UsersModule, the only change is to add the UsersService to the exports array of the @Module decorator so that it is visible outside this module (we'll soon use it in our AuthService).

You can read more here about how Nest uses modules to organize code and to understand more about the exports array and other parameters of the @Module() decorator.

Make sure src/users/users.module.ts looks like this:



// src/users/users.module.ts
import { Module } from '@nestjs/common';
import { UsersService } from './users.service';

@Module({
  providers: [UsersService],
  exports: [UsersService],
})
export class UsersModule {}

Our AuthService has the job of retrieving a user and verifying the password. Replace the default contents of the src/auth/auth.service.ts file with the code below:



// src/auth/auth.service.ts
import { Injectable } from '@nestjs/common';
import { UsersService } from '../users/users.service';

@Injectable()
export class AuthService {
  constructor(private readonly usersService: UsersService) {}

  async validateUser(username, pass): Promise<any> {
    const user = await this.usersService.findOne(username);
    if (user && user.password === pass) {
      const { password, ...result } = user;
      return result;
    }
    return null;
  }
}

Warning Of course in a real application, you wouldn't store a password in plain text. You'd instead use a library like bcrypt, with a salted one-way hash algorithm. With that approach, you'd only store hashed passwords, and then compare the stored password to a hashed version of the incoming password, thus never storing or exposing user passwords in plain text. To keep our sample app simple, we violate that absolute mandate and use plain text. Don't do this in your real app!

We'll call into our validateUser() method from our passport-local strategy subclass in a moment. The Passport library expects us to return a full user if the validation succeeds, or a null if it fails (failure is defined as either the user is not found, or the password does not match). In our code, we use a convenient ES6 spread operator to strip the password property from the user object before returning it. Upon successful validation, Passport then takes care of a few details for us, which we'll explore later on in the Sessions section.

And finally, we update our AuthModule to import the UsersModule.



// src/auth/auth.module.ts
import { Module } from '@nestjs/common';
import { AuthService } from './auth.service';
import { UsersModule } from '../users/users.module';

@Module({
  imports: [UsersModule],
  providers: [AuthService],
})
export class AuthModule {}

Our app will function now, but remains incomplete until we finish a few more steps. You can restart the app and navigate to http://localhost:3000 and still move around without logging in (after all, we haven't implemented our passport-local strategy yet. We'll get there momentarily).

Implementing Passport local

Now we can implement our passport-local strategy. Create a file called local.strategy.ts in the auth folder, and add the following code:



// src/auth/local.strategy.ts
import { Strategy } from 'passport-local';
import { PassportStrategy } from '@nestjs/passport';
import { Injectable, UnauthorizedException } from '@nestjs/common';
import { AuthService } from './auth.service';

@Injectable()
export class LocalStrategy extends PassportStrategy(Strategy) {
  constructor(private readonly authService: AuthService) {
    super();
  }

  async validate(username: string, password: string) {
    const user = await this.authService.validateUser(username, password);
    if (!user) {
      throw new UnauthorizedException();
    }
    return user;
  }
}

We've followed the recipe described in the NestJS Authentication Chapter. In this use case with passport-local, there are no configuration options, so our constructor simply calls super(), without an options object.

We've also implemented the validate() method. For the local-strategy, Passport expects a validate() method with a signature like



validate(username: string, password:string): any

Most of the work is done in our AuthService (and in turn, in our UserService), so this method is quite straightforward. The validate() method for any Passport strategy will follow a similar pattern. If a user is found and valid, it's returned so request handling can continue, and Passport can do some further housekeeping. If it's not found, we throw an exception and let Nest's exceptions layer handle it.

With the strategy in place, we have a few more tasks to complete:

Create guards used to decorate routes so that the configured Passport strategy is invoked
Add @UseGuards() decorators as needed
Implement sessions so that users stay logged in across requests
Configure Nest to use Passport and session-related features
Add a little polish to the user experience

Let's get started. For the following sections, we'll want to adhere to a best practice project structure, so start by creating a few more folders. Under src, create a common folder. Inside common, create filters and guards folders. Our project structure now looks like this:



mvc-sessions
└───src
│   └───auth
│   └───common
│       └───filters
│       └───guards
│   └───users
└───public

Implement guards

The NestJS Guards chapter describes the primary function of guards: to determine whether a request will be handled by the route handler or not. That remains true, and we'll use that feature soon. However, in the context of using the @nestjs/passport module, we will also introduce a slight new wrinkle that may at first be confusing, so let's discuss that now. Once again, the NestJS Authentication chapter has a section which describes this scenario which would be good to read now. The key takeaways are:

Passport supports authentication in two modes. First, you need to perform the authentication step (i.e., logging in)
Subsequently, you need to verify a user's credentials. In the case of passport-local, this means ensuring that the user's session is valid.

Both of these steps are implemented via Nest guards.

Initial Authentication

Looking at our UI, it's easy to see that we'll handle this initial authentication step via a POST request on our /login route. So how do we invoke the "login phase" of the passport-local strategy in that route? As suggested, the answer is to use a Guard. Similar to the way we extended the PassportStrategy class in the last section, we'll start with a default AuthGuard provided in the @nestjs/passport package, and extend it as needed. We'll name our new Guard LoginGuard. We'll then decorate our POST /login route with this LoginGuard to invoke the login phase of our passport-local strategy.

Create a file called login.guard.ts in the guards folder and replace its default contents as follows:



// src/common/guards/login.guard.ts
import { ExecutionContext, Injectable } from '@nestjs/common';
import { AuthGuard } from '@nestjs/passport';

@Injectable()
export class LoginGuard extends AuthGuard('local') {
  async canActivate(context: ExecutionContext) {
    const result = (await super.canActivate(context)) as boolean;
    const request = context.switchToHttp().getRequest();
    await super.logIn(request);
    return result;
  }
}

There's a lot going on in these few lines of code, so let's walk through it.

Our passport-local strategy has a default name of 'local'. We reference that name in the extends clause of the LoginGuard we are defining in order to tie our custom guard to the code supplied by the passport-local package. This is needed to disambiguate which class we are extending in case we end up using multiple Passport strategies in our app (each of which may contribute a strategy-specific AuthGuard).
As with all guards, the primary method we define/override is canActivate(), which is what we do here. You can read more about guards and customizing the canActivate() method here.
The critical part happens in the body of canActivate() , where we set up an Express session. Here's what's happening:
- We call canActivate() on the super class, as we normally would in extending a class method. Our super class provides the framework for invoking our passport-local strategy. Recall from the Guards chapter that canActivate() returns a boolean indicating whether or not the target route will be called. When we get here, Passport will have run the previously configured strategy (from the super class) and will return a boolean to indicate whether or not the user has successfully authenticated. Here, we stash the result so we can do a little more processing before finally returning from our method.
- The key step for starting a session is to now invoke the logIn() method on our super class, passing in the current request. This actually calls a special method that Passport automatically added to our Express Request object during the previous step. See here and here for more on Passport sessions and these special methods.
- The Express session has now been set up, and we can return our canActivate() result, allowing only authenticated users to continue.

Sessions

Now that we've introduced sessions, there's one additional detail we need to take care of. Sessions are a way of associating a unique user with some server-side state information about that user. Let's delve briefly into how Passport rides on top of Express sessions to provide some context.

Passport adds properties to the session object to keep track of information about the user and their authentication state. The user details are populated by a serializeUser() call to the Passport library. This function is called automatically by Nest with the user object created in the validate() method (which we implemented in src/auth/local.strategy.ts a few minutes ago). This approach may feel a little complex at first, but it supports a completely flexible model for how you can manage interactions with your User store. In our case, we are simply passing the user object through untouched. In advanced scenarios, you may find yourself calling out to your database or caching layer to augment the user object with more information (e.g., roles/permissions). Unless you're doing something advanced like that, you can usually just use the following boilerplate for the serialization process.

Create the session.serializer.ts file in the auth folder, and add the following code:



// src/auth/session.serializer.ts
import { PassportSerializer } from '@nestjs/passport';
import { Injectable } from '@nestjs/common';
@Injectable()
export class SessionSerializer extends PassportSerializer {
  serializeUser(user: any, done: (err: Error, user: any) => void): any {
    done(null, user);
  }
  deserializeUser(payload: any, done: (err: Error, payload: string) => void): any {
    done(null, payload);
  }
}

We need to configure our AuthModule to use the Passport features we just defined. Of course AuthService and LocalStrategy make sense as providers (read more about providers here if you need to). Note that the SessionSerializer we just created is also a pluggable provider, and needs to be included in the providers array. Don't worry if this isn't 100% clear at the moment. Just think of providers as a generic way to inject customizable services into your application structure (including 3rd party modules you've included).

Update auth.module.ts to look like this:



// src/auth/auth.module.ts
import { Module } from '@nestjs/common';
import { AuthService } from './auth.service';
import { UsersModule } from '../users/users.module';
import { PassportModule } from '@nestjs/passport';
import { LocalStrategy } from './local.strategy';
import { SessionSerializer } from './session.serializer';

@Module({
  imports: [UsersModule, PassportModule],
  providers: [AuthService, LocalStrategy, SessionSerializer],
})
export class AuthModule {}

Now let's create our AuthenticatedGuard. This is a traditional guard, as covered in the NestJS Guards chapter. Its role is simply to protect certain routes. Create the file authenticated.guard.ts in the guards folder, and add the following code:



// src/common/guards/authenticated.guard.ts
import { ExecutionContext, Injectable, CanActivate } from '@nestjs/common';

@Injectable()
export class AuthenticatedGuard implements CanActivate {
  async canActivate(context: ExecutionContext) {
    const request = context.switchToHttp().getRequest();
    return request.isAuthenticated();
  }
}

The only thing to point out here is that in order to determine whether a user is authenticated or not, we use the convenient isAuthenticated() method that Passport has attached to the request object for us. Passport will return true only if the user is authenticated (i.e., has a valid session).

Configure Nest to bootstrap features

We can now tell Nest to use the Passport features we've configured. Update main.ts to look like this:



// src/main.ts
import { NestFactory } from '@nestjs/core';
import { NestExpressApplication } from '@nestjs/platform-express';
import { join } from 'path';
import { AppModule } from './app.module';

import * as session from 'express-session';
import flash = require('connect-flash');
import * as exphbs from 'express-handlebars';
import * as passport from 'passport';

async function bootstrap() {
  const app = await NestFactory.create<NestExpressApplication>(AppModule);

  const viewsPath = join(__dirname, '../public/views');
  app.engine('.hbs', exphbs({ extname: '.hbs', defaultLayout: 'main' }));
  app.set('views', viewsPath);
  app.set('view engine', '.hbs');

  app.use(
    session({
      secret: 'nest cats',
      resave: false,
      saveUninitialized: false,
    }),
  );

  app.use(passport.initialize());
  app.use(passport.session());
  app.use(flash());

  await app.listen(3000);
}
bootstrap();

Here, we've added the session and Passport support to our Nest app.

Warning As always, be sure to keep secrets out of your source code (don't put your session secret in the code, as we did here; use environment variables or a config module (such as NestJS Config Manager instead).

Note carefully that the order is important (register the session middleware first, then initialize Passport, then configure Passport to use sessions). We'll see the use of the flash feature in a few minutes.

Add route guards

Now we're ready to start applying these guards to routes. Update app.controller.ts to look like this:



// src/app.controller.ts
import { Controller, Get, Post, Request, Res, Render, UseGuards } from '@nestjs/common';
import { Response } from 'express';

import { LoginGuard } from './common/guards/login.guard';
import { AuthenticatedGuard } from './common/guards/authenticated.guard';

@Controller()
export class AppController {
  @Get('/')
  @Render('login')
  index() {
    return;
  }

  @UseGuards(LoginGuard)
  @Post('/login')
  login(@Res() res: Response) {
    res.redirect('/home');
  }

  @UseGuards(AuthenticatedGuard)
  @Get('/home')
  @Render('home')
  getHome(@Request() req) {
    return { user: req.user };
  }

  @UseGuards(AuthenticatedGuard)
  @Get('/profile')
  @Render('profile')
  getProfile(@Request() req) {
    return { user: req.user };
  }

  @Get('/logout')
  logout(@Request() req, @Res() res: Response) {
    req.logout();
    res.redirect('/');
  }
}

Above, we've imported our two new guards and applied them appropriately. We use the LoginGuard on our POST /login route to initiate the authentication sequence in the passport-local strategy. We use AuthenticatedGuard on our protected routes to ensure they aren't accessible to unauthenticated users.

We're also taking advantage of the Passport feature that automatically stores our User object on the Request object as req.user. With this handy feature, we can now return values on our routes decorated with @Render() to pass a variable into our handlebars templates to customize their content.
For example return { user: req.user } to display information from the User object in our home template.

Finally, we have added the call to req.logout() in our logout route. This relies on the Passport logout() function, which, like the logIn() method we discussed earlier in the Sessions section, has been added to the Express Request object by Passport automatically upon successful authentication. When we invoke logout(), Passport tears down our session for us.

You should now be able to test the authentication logic by attempting to navigate to a protected route. Restart the app and point your browser at http://localhost:3000/profile. You should get a 403 Forbidden error. Return to the root page at http://localhost:3000, and log in, and you should be able to browse around (though the app is still missing a couple of features). Refer to src/users/users.service.ts for the hard-coded usernames and passwords that are accepted.

Adding polish

Let's address that ugly 403 Forbidden error page. If you navigate around the app, trying things like submitting an empty login request, a bad password, and logging out, you'll see that it's not a very good UX. Let's take care of a couple of things:

Let's send the user back to the login page whenever they fail to authenticate, and when they log out of the app
Let's provide a little feedback when a user types in an incorrect password

The best way to handle the first requirement is to implement a Filter. Create the file auth-exceptions.filter.ts in the filters folder, and add the following code:



// src/common/filters/auth-exceptions.filter.ts
import {
  ExceptionFilter,
  Catch,
  ArgumentsHost,
  HttpException,
  UnauthorizedException,
  ForbiddenException,
} from '@nestjs/common';
import { Request, Response } from 'express';

@Catch(HttpException)
export class AuthExceptionFilter implements ExceptionFilter {
  catch(exception: HttpException, host: ArgumentsHost) {
    const ctx = host.switchToHttp();
    const response = ctx.getResponse<Response>();
    const request = ctx.getRequest<Request>();

    if (
      exception instanceof UnauthorizedException ||
      exception instanceof ForbiddenException
    ) {
      request.flash('loginError', 'Please try again!');
      response.redirect('/');
    } else {
      response.redirect('/error');
    }
  }
}

The only new element here from what's covered in the NestJS Filters chapter is the use of connect-flash. If a route returns either an UnauthorizedException or a ForbiddenException, we redirect to the root route with response.redirect('/'). We also use connect-flash to store a message in Passport's session. This mechanism allows us to temporarily persist a message upon redirect. Passport and connect-flash automatically take care of the details of storing, retrieving, and cleaning up those messages.

The final touch is to display the flash message in our handlebars template. Update app.controller.ts to look like this. In this update, we're adding the AuthExceptionFilter and adding the flash parameters to our index (/) route.



// src/app.controller.ts
import { Controller, Get, Post, Request, Res, Render, UseGuards, UseFilters } from '@nestjs/common';
import { Response } from 'express';

import { LoginGuard } from './common/guards/login.guard';
import { AuthenticatedGuard } from './common/guards/authenticated.guard';
import { AuthExceptionFilter } from './common/filters/auth-exceptions.filter';

@Controller()
@UseFilters(AuthExceptionFilter)

export class AppController {
  @Get('/')
  @Render('login')
  index(@Request() req): { message: string } {
    return { message: req.flash('loginError') };
  }

  @UseGuards(LoginGuard)
  @Post('/login')
  login(@Res() res: Response) {
    res.redirect('/home');
  }

  @UseGuards(AuthenticatedGuard)
  @Get('/home')
  @Render('home')
  getHome(@Request() req) {
    return { user: req.user };
  }

  @UseGuards(AuthenticatedGuard)
  @Get('/profile')
  @Render('profile')
  getProfile(@Request() req) {
    return { user: req.user };
  }

  @Get('/logout')
  logout(@Request() req, @Res() res: Response) {
    req.logout();
    res.redirect('/');
  }
}

We now have a fully functional authentication system for our server side Web application. Go ahead and fire it up and try logging in and out, accessing protected routes while logged out, and typing bad/missing username/password fields, and notice the more friendly error handling.

We're done! Kudo's for hanging in through a long tutorial. I hope this helps you understand some techniques for implementing sessions and authentication with NestJS.

Resources

You can find all of the source code from this article on github here.

Feel free to ask questions, make comments or suggestions, or just say hello in the comments below. And join us at Discord for more happy discussions about NestJS. I post there as Y Prospect.

Acknowledgements

Thanks to Jay McDoniel, Livio Brunner and Kamil Myśliwiec for their help reviewing this article.

Forem: John Biundo

Part 5: Completing the Client Component

Introduction

Get the Code

Git checkout the current version

Build the Apps for This Part

Overview

Understanding the Multiplexing Challenge

Strategy for Solving the Multiplexing Challenge

The Observable Part

Handling Multiple Requests: The Union of Observables and Correlation Ids

Other Bookkeeping

Connection Management

Odds and Ends

Implementation

Take 2 Code Review

The Superclass send() Method

The ClientFaye#publish() Method

The createSubscriptionHandler Method: Binding the Response Emitter

Discarding Messages for Other Clients

Loose Ends

Connection Management

Event Handling

Acceptance Testing

Conclusion

What's Next

Part 4: Basic Client Component

Introduction

Get the Code

Git checkout the current version

Build the Apps for This Part

Overview

What's Next

First Iteration (Take 1) of the Client Component

Take 1 Requirements

Take 1 Strategy

Take 1 Code Review

Class Definition

Implementing send()

The send() Method Returns an Observable

The Request Handler

Handling Unsubscribe

The Rest of the Code

Acceptance Testing

Limitations of Take 1

What's Next

Part 3: Completing the Server Component

Introduction

Get the Code

Git checkout the current version

Build the Apps for This Part

Overview

Addressing Observables

Handle Events

Subscription Context

Creating the context object

Passing Context to User-land Handlers

Testing Context

Deserialization Options

Type Safety

Error Handling

Checkpoint

What's Next

Part 2: Basic Server Component

Introduction

Get the Code

Git checkout the current version

Build the Apps for This Part

Organizing the Code

The Custom Transporter Package

First Iteration (Take 1) of the Server Component

Using a Custom Transporter Strategy

Instantiating the Custom Transporter

Using the Custom Transporter

Take 1 Requirements

Take 1 Code Review

Acceptance Testing

Primary Acceptance Test

Using npm link in Our Development Environment

Running the Test

The Superclass `send()` Method

The `ClientFaye#publish()` Method

The `createSubscriptionHandler` Method: Binding the Response Emitter

Implementing `send()`

The `send()` Method Returns an Observable

Using `npm link` in Our Development Environment