Forem: Antonio Santiago

How to slice or get symbols from a unicode string with emojis in JavaScript? Lets learn how JavaScript represent strings

Antonio Santiago — Tue, 14 Dec 2021 10:23:06 +0000

Originally I wrote this article in https://www.acuriousanimal.com

Everybody loves JavaScript (or not)! It's easy, powerful and fun (or not) but then one day you found something weird that makes you learn something new and that's fine (or not).

If you don't know why "😄".charAt(0) returns � or why "🌍".length is equal 2 then this article if for you. Do you really know how to handle strings in JavaScript?

Take a look to the next code, what would you expect as a result?

"😄".charAt(0);
"hello🌍world".slice(0, 6);

If you know the result and why is that result, congratulations you can skip this post. Otherwise, if like me you get a surprise then this post can clarify you a bit how JavaScript handles strings and how you must handle string when working with unicode characters.

Unicode vs UTF

No, unicode isn't the same as UTF.

Unicode is a standard for representing characters from many human languages. It assigns characters to ordinal numbers in decimal form, which are called code points.
UTF is a method for encoding the Unicode characters in decimal form to binary representations. Here we can find UTF-8, UTF-16, etc where the number represents the number of bits used to encode each unicode character.

For example:

à                 -> Unicode character
224               -> Decimal code point
00000000 11100000 -> Binary representation with 16 bits
00E0              -> Hex representation with 4 digits
U+00E0            -> Hex code point

The thing you need to remember is a code point is a number assigned to a single character.

Unicode planes

Unicode allows to represent 1,114,112 code points which ranges from U+0000 to U+10FFFF and only 144,697 has an associated character.

The first plane goes from U+0000 to U+FFFF, that is 16⁴ (or 2¹⁶ if you think in binary), which results in 65,536 characters. Now multiply 65,536 by the 17 planes and you get the 1,114,112.

In addition, the unicode space is divided in 17 planes:

Plane 0, Basic Multilingual Plane (BMP), contains code points from U+0000 to U+FFFF. It contains characters from most of the modern languages (Basic Latin, Cyrillic, Greek, etc) and a big number of symbols.
Plane 1, Supplementary Multilingual Plane (SMP), contains code points from U+10000 to U+1FFFF
Plane 2, Supplementary Ideographic Plane (SIP), contains code points from U+20000 to U+2FFFF
...
Plane 16 contains code points from U+100000 to U+10FFFF.

The 16 planes beyond the BMP (from plane 1 to plane 16) are named supplementary or astral planes.

Note, while the code points at BPM plane have all 4 digits the code points in supplementary planes can have 5 o 6 digits, for example:

🌍       -> character
127757   -> code point (decimal)
U+1f30d  -> code point (5 hex)

UTF-16, code units and surrogates

Although each character is represented by a code point we need a way to encode that number in a machine friendly way, here is where UTF-16 comes in action.

Most JavaScript engines use UTF-16 encoding, so no matter if a file is encoded in UTF-8 or any other format, the JS engine will convert it to UTF-16.

UTF-16 is a variable-length encoding, that means, not all code points are encoded using the same number of digits (basically to reduce space):

Code points from BMP are encoded using a single code unit of 16-bit, i.e.
Code points from supplementary planes are encoded using two code units of 16-bit each.

For example here you can see examples of codifications:

A            -> character
65           -> code point (decimal)
U+0041       -> code point (4 hex)
\u0041       -> 1 code unit 

🌍            -> character
127757        -> code point (decimal)
U+1f30d       -> code point (5 hex)
\uD83C\uDF0D  -> 2 code units

Note when you write encoded character in HTML you are using the decimal notation, while in JavaScript you usually use the hexadecimal one.

So, what is a code points? A code unit is a bit sequence used to encode each character within a given encoding form, so we found the unicode character could be represented in JavaScript using 1 or 2 code units. When you need 2 code units to represent a code point they are called a surrogate pair, where the first value of the pair is a high-surrogate code unit and the second value is a low-surrogate code unit.

Ways to write characters in JavaScript

Of course the best way to write characters is writing them directly with the keyboard, but there could be some of them difficult to write (like emojis or math symbols). Fortunately JavaScript has a special syntax to represent characters both using their code point or code unit values:

console.log('A')             -> writing the letter
console.log('\x41')         -> writing the 2 digits hex code
console.log('\u{0041}')      -> writing the code point
console.log('\u0041')        -> writing the code unit

console.log('🌍')            -> writing the letter
console.log('\u{1f30d}')     -> writing the code point
console.log('\uD83C\uDF0D')  -> writing the code unit

Note ES5 only accepts the code unit notation while ES6 introduces the brackets notation that allows us to use code points. This is useful when, i.e., representing emojis, the 🐶 symbol can be written as code point \u{1F436} or with code units \uD83D\uDC36.

Things to take into account when working with strings

Normalization

We can find situations where different code units represents the same characters, for example:

let string1 = '\u00F1';
let string2 = '\u006E\u0303';

console.log(string1);  //  ñ
console.log(string2);  //  ñ

string1 === string2    // false

We can see the strings are different but they represent the same character so we would like the comparison string1 === string2 returns true instead false. How we can work in this situations? The answer is normalization.

👉 The normalize() method helps solve this problem by converting a string into a normalized form common for all sequences of code points that represent the same characters.

let string1 = '\u00F1';
let string2 = '\u006E\u0303';

console.log(string1);  //  ñ
console.log(string2);  //  ñ

string1.normalize() === string2.normalize()    // false

Note how lengths differ from previous normalization:

let string1 = '\u00F1';
let string2 = '\u006E\u0303';

string1.length               // 1
string2.length               // 2

string1.normalize().length   // 1
string2.normalize().length   // 1

Working with code points

ES6 introduce two methods to works directly with code points codePointAt() and fromCodePoint(). You just can imagine what they do looking at the next code:

'Aloha'.codePointAt(0)      // 65
String.fromCodePoint(65)    // A


'😅Aloha'.codePointAt(0)      // 128517
String.fromCodePoint(128517)  // 😅

Length of string

See the next code:

'A'.length    // 1
'🌍'.length   // 2

Shocked? Well, this is more easy to understand if we see the definition of String that ES6 does:

The String type is the set of all ordered sequences of zero or more 16-bit unsigned integer values ("elements") up to a maximum length of 2⁵³-1 elements. The String type is generally used to represent textual data in a running ECMAScript program, in which case each element in the String is treated as a UTF-16 code unit* value.

In the first case, the letter A is encoded using 1 code unit of 16 bits while the emoji 🌍 requires 2 code units of 16 bits to be represented. The important thing here is know if the methods we are using works with code points or code units.

The length of a String is the number of elements within it. Where ECMAScript operations interpret String values, each element is interpreted as a single UTF-16 code unit.

How I get the length (in symbols) of a string?

Ok, nice to know JavaScript count code units but what if I just want to know the number of symbols (or characters) within a string?

console.log('A 🌍'.length)   // 4 bit I want to get 3

The easiest way is to convert your string into an array:

Array.from('A 🌍').length   // 3
[...'A 🌍'].length          // 3

Why this works? Easy, because the String iterator When you create an iterator over a string you get new iterator object that iterates over the code points of a String value, returning each code point as a String value.

In addition to the two previous options you can also iterate using a for..of loop.

The answer to the original question: How can I slice ot get a character?

Well, If you have read til this point, now it's easy to answer our original question how do I get a single symbol from string?

Given the string 🐶 this is my dog if we want to get the character 🐶 we can't use the chatAt method because that would return one of the surrogate code unit that represent that character instead the symbol as you expect:

'🐶 this is my dog'.charAt(0)        // � Invalid char
[...'🐶 this is my dog'][0]          // 🐶
Array.from('🐶 this is my dog')[0]   // 🐶

For the second question how do I slice a string? the answer is similar, if you want to slice the string taking into account code points (the symbols) instead the code units you need a way to iterate of them:

'👋🌍😊🐶'.slice(0,3)                // 👋�
[...'👋🌍😊🐶'].slice(0,3).join('')  // 👋🌍😊

References

How I migrated my site from Jekyll to Gatsby

Antonio Santiago — Sun, 03 Nov 2019 17:42:02 +0000

This post was originally published at my blog www.acuriousanimal.com

I think it was a matter of time to make this change. Jekyll was one of the first static site generates and it's awesome but for me, as a JavaScript developer that know little about ruby language, has more sense to stay close JS world.

I heard about Gatsby some time ago but never spent enough time to get a good ideas about its philosophy and capabilities. Finally this weekend I decided to make the change. What I found is while Jekyll is extremely good doing one thing --mainly focused on blogs--, Gatsby is a more general solution that can be good for many other things and not only a blog.

This post is about my experience in the migration process and how I implemented some features.

Differences between Jekyll & Gatsby

Disclaimer: I'm not a Jekyll expert so don't flame 🔥 next is my description of Jekyll for a blogger user

Jekyll is mainly focused on static sites like blogs, because of this it divides the type of content between pages and posts. If you write a markdown file in the root folder or under _site thay are considered pages, meanwhile if you write .md files in _posts folder they are considered posts.

In addition the post files must be named following the next rule YEAR-MONTH-DAY-title.md i.e. 2012-09-12-how-to-write-a-blog.md. This is a nice rule that make your post files appears ordered in disk.

Both pages and posts have a main section so called frontmatter used to specified some attributes of the content like title, date, tags, ...:

--------
layout: post
title:  "Welcome to Jekyll!"
--------

On the other hand Gatsby is a more general purpose tool that reads data from different sources, process them and generate pages and the great is what it reads, process and generates is up to you.

Well, don't worry most of them are done automatically but you are allowed to be involved in the previous process and tune them to achieve what you need.

Note, one really important difference between Jekyll and Gatsby is designed to work with markdown content by default, both pages and posts can directly be written in markdown format, while Gatsby understand JavaScrip and React, but anyway it is easy configurable to accept other data sources like markdown files.

Before start

I hate those shitty posts title Migrating from X to Y in 10 minutes so How to build X in 20 min, so to be honest I want to say it clear: I spent many ours last weekend migrating my site from Jekyll to Gatsby, and that includes some hours playing with Gatsby, learning its API, looking into others source code and existent staters.

Further more, I'm not a great designer. I love programming the client side but after years I have finally admitted to myself I'm not able to do visual beautiful things 😅 so I spend some time looking for a theme for my site and finally decided for the one used in the julia starter.

One thing a loved about this theme is it used the emotion library for CSS styles.

One important consideration in my migration is all the content from previous site made in Jekyll is written in markdown, so I don't want to rewrite everything as JavaScript, React and HTML, the goal is simply copy&paste all these files and make them work in the new Gatsby site.

How Gatsby works

I'm going to try to synthesize the content of the Gastby Internal section in a few sentences:

The flow to build a Gatsby site is as follow:

first read data sources: this can be the JS files in the src/pages or any file read by a plugin like gatsby-source-filesystem. All this content is transformed into gatsby nodes. For example, each markdown file es read and converted into a node.

process the previous gatsby node: as example, here we can create and attach new properties to the node. For example here we compute a slug for each node that corresponds to a blog post.

create pages: we tell Gatsby engine to create a page for the nodes we desire. For example, we create a page for each blog posts, or a page for each node that correspond to a markdown marked as page,

Analyzing the new site

After the selecting the theme for the new site I was thinking how to organize it by sections:

Home: Simple landing page with slogan and some links to my social networks
Blog: This should contain all my previous posts written in markdown
Books, Projects and About: those sections are also written in markdown
Archive: This was automatically generated in Jekyll and must also by automatically generated in Gatsby

So at the end I have:

sections written natively in JavaScript and sections written in markdown
sections that are pages and blog posts

Write pages in Gatsby is really easy, by default all React components under src/pages will be transformed to a page.

Migrating the markdown pages

I left src/pages as default folder to generate pages from JavaScript files and created a new folder src/pages-markdown for those pages writing in markdown format.

In addition, these pages must be have next properties within the frontmatter section:

layout: page
path: /some_path

For example:

--------
layout: page
title: About
date: 2010-11-23 14:48
path: /about
--------

To make Gatsby engine take into account these files in the build process first stop is to inform where to read them:

// gatsby-config.js
{
  resolve: `gatsby-source-filesystem`,
  options: {
    name: `pages-markdown`,
    path: `${__dirname}/src/pages-markdown/`,
  },
}

For each file found at src/pages-markdown Gatsby will create a node. Now, we can extend the createPages method and make Gatsby create a new page for each node. The steps can be summarized as:

Query for all the markdown nodes read by gatsby
Filter those nodes with layout=page
Create a new page invoking action.createPage method and using the template file src/templated/page.js

// gatsby-node.js
exports.createPages = ({ graphql, actions }) => {
  const { createPage } = actions
  const pageTemplate = path.resolve(`src/templates/page.js`);

  return graphql(
    `
    {
      allMarkdownRemark {
        edges {
          node {
            fields {
              slug
            },
            frontmatter {
              path
              layout
            }
          }
        }
      }
    }
    `
  ).then(result => {
    if(result.errors) {
      return Promise.reject(result.errors)
    }

    const markdownItems = result.data.allMarkdownRemark.edges

    // Create pages and blog post pages
    markdownItems.forEach(({ node }) => {
      if (node.frontmatter.layout === 'page') {  
        createPage({
          path: node.frontmatter.path,
          component: pageTemplate,
        })
      }
    })
  })
}

Migrating posts

I have copied all my blog posts into blog folder, so we need to tell gatsby where to find them:

// gatsby-config.js
{
  resolve: 'gatsby-source-filesystem',
  options: {
    name: 'blog',
    path: `${__dirname}/blog/`,
  },
},

While the files are named YYYY-MM-DD-some-title.md we want the posts have the path /blog/YYYY-MM-DD-some-title in the browser so we need to create that slug from the file name. The way we can do is using the onCreateNode method:

exports.onCreateNode = ({ node, getNode, actions }) => {
  const { createNodeField } = actions

  if (node.internal.type === `MarkdownRemark`) {
    const filename = createFilePath({ node, getNode, basePath: `blog`})

    // Blog files must have format name YYYY-MM-DD-title.md
    if (node.frontmatter.layout === 'post') {
      const match = filename.match(/^\/([\d]{4}-[\d]{2}-[\d]{2})-{1}(.+)\/$/)
      if (match) {
        const [, date, title] = match
        if (!date || !title) {
          console.error(`Invalid filename ${filename}. Change name to start with a valid date and title`)
        } else {
          const slug = `/blog/${slugify(date, "/")}/${title}/`
          createNodeField({
            node,
            name: `slug`,
            value: slug
          })
        }
      }
    }
  }
}

Finally, we need to tell gatsby to create a page for each blog post node. We can do it extending a bit the previous createPages method:

exports.createPages = ({ graphql, actions }) => {
  const { createPage } = actions
  const blogPostTemplate = path.resolve(`src/templates/blog-post.js`);
  const pageTemplate = path.resolve(`src/templates/page.js`);

  return graphql(`...`).then(result => {
    if(result.errors) {
      return Promise.reject(result.errors)
    }

    const markdownItems = result.data.allMarkdownRemark.edges

    // Create pages and blog post pages
    markdownItems.forEach(({ node }) => {
      if (node.frontmatter.layout === 'page') {  
        createPage({
          path: node.frontmatter.path,
          component: pageTemplate,
        })
      } else if (node.frontmatter.layout === 'post') {
        createPage({
          path: node.fields.slug,
          component: blogPostTemplate,
          context: {
            slug: node.fields.slug,
          },
        })
      }
    })
  })
}

Paginating the blog posts

What we desire is a blog section that lists all posts, really a paginated list of posts, and then let user select one to be read.

The previous section creates a page for each post so we need to create the pages that list the posts.

Again, we only need to update a bit the createPages method to create a new page for each set of six posts:

exports.createPages = ({ graphql, actions }) => {
  const { createPage } = actions
  const blogListTemplate = path.resolve("./src/templates/blog-list.js")
  const blogPostTemplate = path.resolve(`src/templates/blog-post.js`)
  const pageTemplate = path.resolve(`src/templates/page.js`)

  return graphql(`...`).then(result => {
    if(result.errors) {
      return Promise.reject(result.errors)
    }

    const markdownItems = result.data.allMarkdownRemark.edges

    // Create blog-list pages
    const posts = markdownItems.filter(item => item.node.frontmatter.layout === 'post')
    const postsPerPage = 6
    const numPages = Math.ceil(posts.length / postsPerPage)
    Array.from({ length: numPages }).forEach((_, i) => {
      createPage({
        path: i === 0 ? `/blog` : `/blog/${i + 1}`,
        component: blogListTemplate,
        context: {
          limit: postsPerPage,
          skip: i * postsPerPage,
          numPages,
          currentPage: i + 1,
        },
      })
    })

    // Create pages and blog post pages
    ...
  })
}

Other things to take into consideration

As we saw, Gatsby knows where to read files using the plugin gatsby-source-filesystem but how does it understand markdown format?

The answer is it doesn't, we need to use the plugin gatsby-transformer-remark to transform markdown files into something gatsby can understand. This plugin is the responsible to read markdown files a extract information like the frontmatter, the excerpt and convert the text into HTML ready to be rendered.

The plugin gatsby-transformer-remark is really powerful and also has many plugins to use within it. I have used three main plugins:

gatsby-remark-prismjs: responsible to syntax highlight code within markdown files
gatsby-remark-images: optimize the images used in markdown files to be production ready
gatsby-remark-reading-time: adds some metadata about the estimated reading time

Finally, note in Jekyll I used the text  to mark the excerpt of the post. We can easily set it configuring the plugin gatsby-transformer-remark:

{
  resolve: `gatsby-transformer-remark`,
  options: {
    excerpt_separator: `<!--more-->`
  }
}

Express API with autogenerated OpenAPI doc through Swagger

Antonio Santiago — Fri, 09 Nov 2018 07:31:00 +0000

This post was originally published in the acuriousanimal.com blog.

In past years OpenAPI has arise as the preferred way to document APIs. In this article we will see how easy is document an API created with NodeJS and Express through the Swagger tools.

If you are working in an REST API you more probably will desire to have some API doc where your users could find what are the endpoints of your API, what they do, which parameters they accept and which output they generate.

See accelerated video at: https://www.youtube.com/watch?v=CE01dwNEkEU

Swagger

Do not confuse OpenAPI with Swagger. OpenAPI is an specification that says how APIs are documented. Swagger is a collection of tools to implement the specification.

Thanks to Swagger, see the editor example, writing API docs is not difficult, you only need to write a bunch of YAML lines.

Among other tools, swagger offers the swagger-ui, which is nothing more than a collection of HTML, Javascript, and CSS assets that dynamically generate beautiful documentation from a Swagger-compliant API, or in other words, swagger-ui is the beautiful web page ou see in the previous swagger editor link.

To serve the swagger-ui content you only need a web server and the API documentation bundled in a JSON or YAML file.

ExpressJS and API documentation

When developing APIs with ExpressJS I tend to write my API endpoints in route files (usually also stored under a routes folder) which contains the rules to attend requests and, because of this, it is a perfect location to document the endpoints of my API, for example:

/**
 * @swagger
 * /users:
 *    get:
 *      description: This should return all users
 */
app.get('/users', (req, res) => res.end('This sould return all users'));

The problem is swagger-ui requires all my API doc reside in a single file, i.e. swagger.json, to be beautified as a web page.

How to autogenerate swagger doc ?

All the magic resides in the next packages:

swagger-ui-express: Adds middleware to your express app to serve the Swagger UI bound to your Swagger document. This acts as living documentation for your API hosted from within your app.
swagger-jsdoc: Generates swagger doc based on JSDoc comments in your code to keep a live and reusable OpenAPI (Swagger) specification.

So the idea is:

Document your routes with jsdoc using the swagger YAML notation (as you can see in the previous route).
Let swagger-jsdoc to autogenerate a swagger.json file with all the documentation you set in your files.

const swaggerJsdoc = require('swagger-jsdoc');

const options = {
  swaggerDefinition: {
    // Like the one described here: https://swagger.io/specification/#infoObject
    info: {
      title: 'Test API',
      version: '1.0.0',
      description: 'Test Express API with autogenerated swagger doc',
    },
  },
  // List of files to be processes. You can also set globs './routes/*.js'
  apis: ['endpoints.js'],
};

const specs = swaggerJsdoc(options);

Use the swagger-ui-express to add a new endpoint where your user can see the documentation.

const swaggerUi = require('swagger-ui-express');

app.use('/api-docs', swaggerUi.serve, swaggerUi.setup(specs));

That is all the magic. At this point if you start and open your browser at http://localhost:3000/api-docs you will see the Swagger UI web page with the documentation of your endpoint.

Of course you can create much more complex documentations. You can include files which only defines includes type definitions (i.e. about input/output data), group your endpoints by tags, etc.

The important thing is how the convination of swagger-ui-express and swagger-jsdoc can incredible help and simplify the way to have your API documentation always up to date.

Deploy on Friday Festival !!! #deployFridayFest

Antonio Santiago — Wed, 31 Oct 2018 11:42:38 +0000

Talking about deploying to production on Fridays is a controversial issue. Some companies has strict policies about that, while others are much more agile and has no problem with it.

Personally, I think it depends on many factors: the kind of change, the impact it could have, how good and automated is your process (I mean applying a change, test and deploy, etc), how company is organised in teams, etc.

I would like to start an "initiative" 😅 to know how many developers deploys to production on Fridays, because of this, every time you deploy something to production on Friday publish a tweet with the #deployFridayFest hashtag.

Cheers !

Graceful shutdown NodeJS HTTP server when using PM2 (4/4)

Antonio Santiago — Fri, 19 Oct 2018 10:12:00 +0000

So you have created a NodeJS server that receives tons of requests and you are really happy but, as every piece of software, you found a bug or add a new feature to it. It is clear you will need to shutdown your NodeJS process/es and restart again so that the new code takes place. The question is: how can you do that in a graceful way that allows continue serving incoming requests?

More on this series:

Understanding the NodeJS cluster module
Using cluster module with HTTP servers
Using PM2 to manage a NodeJS cluster
Graceful shutdown NodeJS HTTP server when using PM2

Starting a HTTP server

Before see how we must shutdown a HTTP server lets see how usually create one. Next code shows a very basic code with an ExpressJS service that will return Hello World !!! when accessing the /hello path. You can also pass a path param, i.e. /hello/John with a name so it returns Hello John !!!.

const express = require('express')

const expressApp = express()

// Responds with Hello World or optionally the name you pass as path param
expressApp.get('/hello/:name?', function (req, res) {
  const name = req.params.name

  if (name) {
    return res.send(`Hello ${name}!!!`)
  }

  return res.send('Hello World !!!')
})

// Start server
expressApp.listen(3000, function () {
  console.log('App listening on port 3000!')
})

What app.listen() function does is start a new HTTP server using the core http module and return a reference to the HTTP server object. In concrete, the source code of the listen() is as follows:

app.listen = function listen() {
  var server = http.createServer(this);
  return server.listen.apply(server, arguments);
};

NOTE: Another way to create an express server is pass our expressApp reference directly to the http. createServer(), something like: const server = http.createServer(app).listen(3000).

How to shutdown properly an HTTP server ?

The proper way to shutdown an HTTP server is to invoke the server.close() function, this will stop server from accepting new connections while keeps existing ones until response them.

Next code presents a new /close endpoint that once invoked will stop the HTTP server and exit the applications (stopping the nodejs process):

app.get('/close', (req, res) => {
  console.log('Closing the server...')

  server.close(() => {
    console.log('--> Server call callback run !!')

    process.exit()
  })
})

It is clear shutting down a server through an endpoint is not the right way to it.

Graceful shutdown/restart with and without PM2

The goal of a graceful shutdown is to close the incoming connections to a server without killing the current ones we are handling.

When using a process manager like PM2, we manage a cluster of processes each one acting as a HTTP server. The way PM2 achieves the graceful restart is:

sending a SIGNINT signal to each worker process,
the worker are responsible to catch the signal, cleanup or free any used resource and finish the its process,
finally PM2 manager spawns a new process

Because this is done sequentially with our cluster processes customers must not be affected by the restart because there will always be some processes working and attending requests.

This is very useful when we deploy new code and want to restart our servers so the new changes take effect without risk for incoming requests. We can achieve this putting next code in out app:

// Graceful shutdown
process.on('SIGINT', () => {
  const cleanUp = () => {
    // Clean up other resources like DB connections
  }

  console.log('Closing server...')

  server.close(() => {
    console.log('Server closed !!! ')

    cleanUp()
    process.exit()
  })

  // Force close server after 5secs
  setTimeout((e) => {
    console.log('Forcing server close !!!', e)

    cleanUp()
    process.exit(1)
  }, 5000)
})

When the SINGINT signal it catch we invoke the server.close() to avoid accepting more requests and once it is closed we clean up any resource used by our app, like close database connection, close opened files, etc invoking the cleanUp() function and, finally, we exits the process with process.exit(). In addition, if for some reason our code spends too much time to close the server we force it running a very similar code within a setTimeout().

Conclusions

When creating a HTTP server, no matter if a web server to serve pages or an API, we need to take into account the fact it will be updated in time with new features and bug fixes, so we need to think in a way to minimize the impact on customers.

Running nodejs processes in cluster mode is a common way to improve our applications performance and we need to think on how to graceful shutdown all them to not affect incoming requests.

Terminating a node process with process.exit() is not enough when working with an HTTP server because it will terminate abruptly all the communications, we need to first stop accepting new connections, free any resource used by our application and, finally, stop the process.

Best services for email verification

Antonio Santiago — Wed, 17 Oct 2018 15:05:00 +0000

Hi, what are best and, if possible, cheapest 😅 services for email verification? What are your experiences and opinions ?

Note: I'm not talking about a library to validate emails. I mean a service to check the email is real (the address exists).

Using PM2 to manage NodeJS cluster (3/4)

Antonio Santiago — Tue, 09 Oct 2018 17:57:00 +0000

The cluster module allows us to create worker processes to improve our NodeJS applications performance. This is specially important in web applications, where a master process receives all the requests and load balances them among the worker processes.

But all this power comes with the cost that must be the application who manages all the complexity associated with process managements: what happens if a worker process exists unexpectedly, how exit gracefully the worker processes, what if you need to restart all your workers, etc.

In this post we present PM2 tool. although it is a general process manager, that means it can manage any kind of process like python, ruby, ... and not only NodeJS processes, the tool is specially designed to manage NodeJS applications that want to work with the cluster module.

More on this series:

Understanding the NodeJS cluster module
Using cluster module with HTTP servers
Using PM2 to manage a NodeJS cluster
Graceful shutdown NodeJS HTTP server when using PM2

Introducing PM2

As said previously, PM2 is a general process manager, that is, a program that controls the execution of other process (like a python program that check if you have new emails) and does things like: check your process is running, re-execute your process if for some reason it exits unexpectedly, log its output, etc.

The most important thing for us is PM2 simplifies the execution of NodeJS applications to run as a cluster. Yes, you write your application without worrying about cluster module and is PM2 who creates a given number of worker processes to run your application.

The hard part of cluster module

Lets see an example where we create a very basic HTTP server using the cluster module. The master process will spawn as many workers as CPUs and will take care if any of the workers exists to spawn a new worker.

const cluster = require('cluster');
const http = require('http');
const numCPUs = require('os').cpus().length;

if (cluster.isMaster) {
  masterProcess();
} else {
  childProcess();  
}

function masterProcess() {
  console.log(`Master ${process.pid} is running`);

  for (let i = 0; i < numCPUs; i++) {
    console.log(`Forking process number ${i}...`);

    cluster.fork();
  }

  cluster.on('exit', (worker, code, signal) => {
    console.log(`Worker ${worker.process.pid} died`);
    console.log(`Forking a new process...`);

    cluster.fork();
  });
}

function childProcess() {
  console.log(`Worker ${process.pid} started...`);

  http.createServer((req, res) => {
    res.writeHead(200);
    res.end('Hello World');

    process.exit(1);
  }).listen(3000);
}

The worker process is a very simple HTTP server listening on port 3000 and programmed to return a Hello World and exit (to simulate a failure).

If we run the program with $ node app.js the output will show something like:

$ node app.js

Master 2398 is running
Forking process number 0...
Forking process number 1...
Worker 2399 started...
Worker 2400 started...

If we go to browser at URL http://localhost:3000 we will get a Hello World and in the console see something like:

Worker 2400 died
Forking a new process...
Worker 2401 started...

That's very nice, now lets go to see how PM2 can simplify our application.

The PM2 way

Before continue, you need to instal PM2 on your system. Typically it is installed as a global module with $ npm install pm2 -g or $ yarn global add pm2.

When using PM2 we can forget the part of the code related with the master process, that will responsibility of PM2, so our very basic HTTP server can be rewritten as:

const http = require('http');

console.log(`Worker ${process.pid} started...`);

http.createServer((req, res) => {
  res.writeHead(200);
  res.end('Hello World');

  process.exit(1);
}).listen(3000);

Now run PM2 with $ pm2 start app.js -i 3 and you will see an output similar to:

Note the option -i that is used to indicate the number of instances to create. The idea is that number be the same as your number of CPU cores. If you don't know them you can set -i 0 to leave PM2 detect it automatically.

$ pm2 start app.js -i 3

[PM2] Starting /Users/blablabla/some-project/app.js in cluster_mode (3 instances)
[PM2] Done.

| Name      | mode    | status | ↺ | cpu | memory    |
| ----------|---------|--------|---|-----|-----------|
| app       | cluster | online | 0 | 23% | 27.1 MB   |
| app       | cluster | online | 0 | 26% | 27.3 MB   |
| app       | cluster | online | 0 | 14% | 25.1 MB   |

We can see the application logs running $ pm2 log. Now when accessing the the http://localhost:3000 URL we will see logs similar to:

PM2        | App name:app id:0 disconnected
PM2        | App [app] with id [0] and pid [1299], exited with code [1] via signal [SIGINT]
PM2        | Starting execution sequence in -cluster mode- for app name:app id:0
PM2        | App name:app id:0 online
0|app      | Worker 1489 started...

We can see how PM2 process detects one of our workers has exit and automatically starts a new instance.

Conclusions

Although the NodeJS cluster module is a powerful mechanism to improve performance it comes at the cost of complexity required to manage all the situations an application can found: what happens if a worker exists, how can we reload the application cluster without down time, etc.

PM2 is a process manager specially designed to work with NodeJS clusters. It allow to cluster an application, restart or reload, without the required code complexity in addition to offer tools to see log outputs, monitorization, etc.

References

Node.js clustering made easy with PM2

Using cluster module with HTTP servers (2/4)

Antonio Santiago — Wed, 19 Sep 2018 20:10:00 +0000

The cluster module allow us improve performance of our application in multicore CPU systems. This is specially important no matter if working on an APIs or an, i.e. ExpressJS based, web servers, what we desire is to take advantage of all the CPUs on each machine our NodeJS application is running.

The cluster module allow us to load balance the incoming request among a set of worker processes and, because of this, improving the throughput of our application.

In the previous post Understanding the NodeJS cluster module I introduced the cluster module and show some basic usages of it to create worker processes and communicate them with the master process. In this post we are going to see how to use the cluster module when creating HTTP servers, both using plain HTTP module and with ExpressJS.

More on this series:

Understanding the NodeJS cluster module
Using cluster module with HTTP servers
Using PM2 to manage a NodeJS cluster
Graceful shutdown NodeJS HTTP server when using PM2

Using cluster module with HTTP servers

Lets go to see how we can create a really basic HTTP server that takes profit of the cluster module.

const cluster = require('cluster');
const http = require('http');
const numCPUs = require('os').cpus().length;

if (cluster.isMaster) {
  masterProcess();
} else {
  childProcess();  
}

function masterProcess() {
  console.log(`Master ${process.pid} is running`);

  for (let i = 0; i < numCPUs; i++) {
    console.log(`Forking process number ${i}...`);
    cluster.fork();
  }
}

function childProcess() {
  console.log(`Worker ${process.pid} started...`);

  http.createServer((req, res) => {
    res.writeHead(200);
    res.end('Hello World');
  }).listen(3000);
}

We have divided the code in two parts, the one corresponding to the master process and the one where we initialize the worker processes. This way the masterProcess function forks a worker process per CPU code. On the other hand the childProcess simply creates an HTTP server listening on port 3000 and returning a nice Hello World text string with a 200 status code.

If you run the code the output must show something like:

$ node app.js

Master 1859 is running
Forking process number 0...
Forking process number 1...
Forking process number 2...
Forking process number 3...
Worker 1860 started...
Worker 1862 started...
Worker 1863 started...
Worker 1861 started...

Basically our initial process (the master) is spawning a new worker process per CPU that runs an HTTP server that handle requests. As you can see this can improve a lot your server performance because it is not the same having one processing attending one million of requests than having four processes attending one million requests.

How cluster module works with network connections ?

The previous example is simple but hides something tricky, some magic NodeJS make to simplify our live as developer.

In any OS a process can use a port to communicate with other systems and, that means, the given port can only be used by that process. So, the question is, how can the forked worker processes use the same port?

The answer, the simplified answer, is the master process is the one who listens in the given port and load balances the requests among all the child/worker processes. From the official documentation:

The worker processes are spawned using the child_process.fork() method, so that they can communicate with the parent via IPC and pass server handles back and forth.

The cluster module supports two methods of distributing incoming connections.

The first one (and the default one on all platforms except Windows), is the round-robin approach, where the master process listens on a port, accepts new connections and distributes them across the workers in a round-robin fashion, with some built-in smarts to avoid overloading a worker process.

The second approach is where the master process creates the listen socket and sends it to interested workers. The workers then accept incoming connections directly.

As long as there are some workers still alive, the server will continue to accept connections. If no workers are alive, existing connections will be dropped and new connections will be refused.

Other alternatives to cluster module load balancing

Cluster module allow the master process to receive request and load balance it among all the worker processes. This is a way to improve performance but it is not the only one.

In the post Node.js process load balance performance: comparing cluster module, iptables and Nginx you can find a performance comparison among: node cluster module, iptables and nginx reverse proxy.

Conclusions

Nowadays performance is mandatory on any web applications, we need to support high throughput and serve data fast.

The cluster module is one possible solution, it allows us to have one master process and create a worker processes for each core, so that they run an HTTP server. The cluster module offers two great features:

simplifies communication among master and workers, by creating an IPC channel and allowing send messages with process.send(),
allow worker processes share the same port. This is done making the master process the one which receives requests and multiplexe them among workers.

Understanding the NodeJS cluster module (1/4)

Antonio Santiago — Thu, 13 Sep 2018 10:14:00 +0000

This post was originally written at A Curious Animal blog.

NodeJS processes runs on a single process, which means it does not take advantage from multi-core systems by default. If you have an 8 core CPU and run a NodeJS program via $ node app.js it will run in a single process, wasting the rest of CPUs.

Hopefully for us NodeJS offers the cluster module that contains a set of functions and properties that help us to create programs that uses all the CPUs. Not a surprise the mechanism the cluster module uses to maximize the CPU usage was via forking processes, similar to the old fork() system call Unix systems.

More on this series:

Understanding the NodeJS cluster module
Using cluster module with HTTP servers
Using PM2 to manage a NodeJS cluster
Graceful shutdown NodeJS HTTP server when using PM2

Introducing the cluster module

The cluster module is a NodeJS module that contains a set of functions and properties that help us forking processes to take advantage of multi-core systems. It is propably the first level of scalability you must take care in your node application, specifally if you are working in a HTTP server application, before going to a higher scalability levels (I mean scaling vertically and horizontally in different machines).

With the cluster module a parent/master process can be forked in any number of child/worker processes and communicate with them sending messages via IPC communication. Remember there is no shared memory among processes.

Next lines are a compilation of sentences from the NodeJS documentation I have taken the liberty to copy&pasta to put it in a way I think can help you understand thw whole thing in a few lines.

A single instance of Node.js runs in a single thread. To take advantage of multi-core systems the user will sometimes want to launch a cluster of Node.js processes to handle the load.

The cluster module allows easy creation of child processes that all share server ports.

The worker (child) processes are spawned using the child_proces.fork() method, so that they can communicate with the parent via IPC and pass server handles back and forth. The child_process.fork() method is a special case of child_process.spawn() used specifically to spawn new Node.js processes. Like child_process.spawn(), a ChildProcess object is returned. The returned ChildProcess will have an additional communication channel built-in that allows messages to be passed back and forth between the parent and child, through the send() method. See subprocess.sen() for details.

It is important to keep in mind that spawned Node.js child processes are independent of the parent with exception of the IPC communication channel that is established between the two. Each process has its own memory, with their own V8 instances. Because of the additional resource allocations required, spawning a large number of child Node.js processes is not recommended.

So, most of the magic is done by the child_process module, which is resposible to spawn new process and help communicate among them, for example, creating pipes. You can find a great article at Node.js Child Processes: Everything you need to know.

A basic example

Stop talking and lets see a real exampe. Next we show a very basic code that:

Creates a master process that retrives the number of CPUs and forks a worker process for each CPU, and
Each child process prints a message in console and exit.

const cluster = require('cluster');
const http = require('http');
const numCPUs = require('os').cpus().length;

if (cluster.isMaster) {
  masterProcess();
} else {
  childProcess();  
}

function masterProcess() {
  console.log(`Master ${process.pid} is running`);

  for (let i = 0; i < numCPUs; i++) {
    console.log(`Forking process number ${i}...`);
    cluster.fork();
  }

  process.exit();
}

function childProcess() {
  console.log(`Worker ${process.pid} started and finished`);

  process.exit();
}

Save the code in app.js file and run executing: $ node app.js. The output should be something similar to:

$ node app.js

Master 8463 is running
Forking process number 0...
Forking process number 1...
Forking process number 2...
Forking process number 3...
Worker 8464 started and finished
Worker 8465 started and finished
Worker 8467 started and finished
Worker 8466 started and finished

Code explanation

When we run the app.js program an OS process is created that starts running our code. At the beginning the cluster mode is imported const cluster = require('cluster') and in the if sentence we check if the isMaster property.

Because the process is the first process the isMaster property is true and then we run the code of masterProcess function. This function has not much secret, it loops depending on the number of CPUs of your machine and forks the current process using the cluster.fork() method.

What the fork() really does is to create a new node process, like if you run it via command line with $node app.js, that is you have many processes running your app.js program.

When a child process is created and executed, it does the same as the master, that is, imports the cluster module and executes the if statement. Once of the differences is for the child process the value of cluster.isMaster is false, so they ends running the childProcess function.

Note, we explicitly terminate the master and worker processes with process.exit(), which by default return value of zero.

NOTE: NodeJS also offers the Child Processes module that simplifies the creation and comunication with other processes. For example we can spawn the ls -l terminal command and pipe with another process that handles the results.

Comunicating master and worker processes

When a worker process is created, an IPC channel is created among the worker and the master, allowing us to communicated between them with the send() method, which accepts a JavaScript object as parameter. Remember they are different processes (not threads) so we can't use shared memory as a way of communcation.

From the master process, we can send a message to a worker process using the process reference, i.e. someChild.send({ ... }), and within the worker process we can messages to the master simply using the current process reference, i.e. process.send().

We have updated slighly the previous code to allow master send and receive messages from/to the workers and also the workers receive and send messages from the master process:

function childProcess() {
  console.log(`Worker ${process.pid} started`);

  process.on('message', function(message) {
    console.log(`Worker ${process.pid} recevies message '${JSON.stringify(message)}'`);
  });

  console.log(`Worker ${process.pid} sends message to master...`);
  process.send({ msg: `Message from worker ${process.pid}` });

  console.log(`Worker ${process.pid} finished`);
}

The worker process is simply to understand. First we listen for the message event registering a listener with the process.on('message', handler) method. Later we send a messages with process.send({...}). Note the message is a plain JavaScript object.

let workers = [];

function masterProcess() {
  console.log(`Master ${process.pid} is running`);

  // Fork workers
  for (let i = 0; i < numCPUs; i++) {
    console.log(`Forking process number ${i}...`);

    const worker = cluster.fork();
    workers.push(worker);

    // Listen for messages from worker
    worker.on('message', function(message) {
      console.log(`Master ${process.pid} recevies message '${JSON.stringify(message)}' from worker ${worker.process.pid}`);
    });
  }

  // Send message to the workers
  workers.forEach(function(worker) {
    console.log(`Master ${process.pid} sends message to worker ${worker.process.pid}...`);
    worker.send({ msg: `Message from master ${process.pid}` });    
  }, this);
}

The masterProcess function has been divided in two parts. In the first loop we fork as much workers as CPUs we have. The cluster.fork() returns a worker object representing the worker process, we store the reference in an array and register a listener to receive messages that comes from that worker instance.

Later, we loop over the array of workers and send a message from the master process to that concrete worker.

If you run the code the output will be something like:

$ node app.js

Master 4045 is running
Forking process number 0...
Forking process number 1...
Master 4045 sends message to worker 4046...
Master 4045 sends message to worker 4047...
Worker 4047 started
Worker 4047 sends message to master...
Worker 4047 finished
Master 4045 recevies message '{"msg":"Message from worker 4047"}' from worker 4047
Worker 4047 recevies message '{"msg":"Message from master 4045"}'
Worker 4046 started
Worker 4046 sends message to master...
Worker 4046 finished
Master 4045 recevies message '{"msg":"Message from worker 4046"}' from worker 4046
Worker 4046 recevies message '{"msg":"Message from master 4045"}'

Here we are not terminating the process with process.exit() so to close the application you need to use ctrl+c.

Conclusion

The cluster module offers to NodeJS the needed capabilities to use the whole power of a CPU. Although not seen in this post, the cluster module is complemented with the child process module that offers plenty of tools to work with processes: start, stop and pipe input/out, etc.

Cluster module allow us to easily create worker processes. In addition it magically creates an IPC channel to communicate the master and worker process passing JavaScript objects.

In my next post I will show how important is the cluster module when working in an HTTP server, no matter if an API or web server working with ExpressJS. The cluster module can increase performance of our application having as many worker processes as CPUs cores.