Forem: Akshendra Pratap Singh

Iterators in JavaScript

Akshendra Pratap Singh — Sun, 29 Jul 2018 14:16:12 +0000

Many a times you wake up and realize that today you will be traversing through an array or maybe many arrays. But you don't worry about it, you have done it before and it wasn't that difficult. You also have a lot of options, you can use the good old loops, or the wonderful map, reduce, fitler. OR you can use the iterators.

Iterator is a design pattern that allows us to traverse over a list or collection. In JavaScript, like most things, they are implemented as objects. Before going in detail, here is a simple example.

const arr = [1, 2, 3, 4, 5];

for (const num of arr) {
  console.log(num);
}

Using for..of loop, you can iterate over any object that implements the iterable protocol.

`Iterable` Protocol

To follow this protocol, the object must define a special method @@iterator (as Symbol.iterator key) which takes zero arguments and returns an object which itself should follow the iterator protocol.

`Iterator` Protocol

To follow this protocol, the object must define a method named next, which itself returns an object with two properties:

value: the current item in iteration
done: a boolean, that represents whether the iteration is finished or not. done=true means iteration is finished.

Array, String, Map, Set, TypedArrays follow the iterator protocol.

Implementing the protocols

Here is a function that returns an iterable which allows us iterate over first n natural numbers.

function numbers(till = 100) {
  let i = 0;
  const iteratorFx = () => {
    const iterator = {
      next() {
        i += 1;
        if (i <= till) {
          return { done: false, value: i };
        }
        return { done: true };
      },
    };
    return iterator;
  };
  return {
    [Symbol.iterator]: iteratorFx,
  };
}

const numbersTill10 = numbers(10);
for (const i for numbersTill10) {
  // 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
}

The @@iterator method is only called once at the beginning of the for..of loop. So the following is same as above

for (const i for numbers(10)) {
  // 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
}

Since we know that the object has a next method which implements the details of iteration, we can simple call this method ourselves.

const numbersTill5 = number(5);
numbersTill5.next(); // { done: false, value : 1 }
numbersTill5.next(); // { done: false, value : 2 }
numbersTill5.next(); // { done: false, value : 3 }
numbersTill5.next(); // { done: false, value : 4 }
numbersTill5.next(); // { done: false, value : 5 }
numbersTill5.next(); // { done: true }

This can be used to control the iteration externally, by passing values to next method.

We can implement our custom iterators like above. However, JavaScript provides another way to create iterables.

Generators

Generators are special function which when called return a Generator object. The generator object follows the iteration protocols. So to implement the above example using generators,

function* generateNumbers(till = 100) {
  let i = 1;
  while (i <= till) {
    yield i;
    i += 1;
  }
}

const numbersTill10 = generateNumbers(10); // iterator
// rest is same

The value sent by yield (here i), will be the value stored in object returned by the next method. And when the generator finishes it returns { done: true }.

It is very clear from the above example that Generators provide a concise way to create iterables. They abstract away the protocols, and we need to worry about the iteration logic only.

Conclusion

Since we started this post with a hyperbole about traversing array. Its only fair that we include an example involving arrays. Arrays are already iterable, so we will create an iterable value mapper.

function* mapOver(arr, mapper = (v) => v) {
  for (let i = 0; i < arr.length; i += 1) {
    yield mapper(arr[i]);
  }
}

const twices = mapOver([...numbers(5)], (v) => v + 2);
for (const num of twices) {
  // 2, 4, 6, 8, 10
}

Spread (...) operator, by definition works on iterables.

The beast that is Array.prototype.reduce

Akshendra Pratap Singh — Sat, 23 Jun 2018 14:50:47 +0000

reduce() is an absolute beast of a method when it comes to functional style programming in JavaScript. The more you use it, the more you see use cases popping everywhere for it.

I recently realized, that it has become my goto method, whenever I have to deal with arrays. So I looked through bunch of my code and found a lot of examples, some of which I will list in this post. But before that - lets start with a short review the method itself.

Signature

arr.reduce((acc, current, index, array) => {
  // work goes here
}, initial);

reduce() takes two parameters.

A callback function, would be the first. reduce() will go through every element of the array and pass callback the following values.
- acc or accumulator, this value is like state that gets updated on every call to keep track of the result
- For the first call, it is equal to initial value provided as second parameter.
- And in subsequent calls, acc will be the value returned by the previous callback call.
- current, the element of the array we are dealing with.
- index, the current index of array
- array, the array itself
The second parameter is initial, the first value of acc. This is optional and in case it is not provided, acc will be the first element of the array.

Simple example

A very common example of reduce() is to calculate the sum of an array of integers.

[1, 2, 3, 4, 5].reduce((sum, integer) => sum + integer, 0);

In this example, we don't need index and array, which is a case in general with reduce(). And sum, integer and 0 play the parts of acc, current and initial respectively.

Now some practical examples

I mentioned above that I went through some of my code to find examples of reduce(). I have listed below some of those, which were different enough to represent a new use case.

I have trimmed the project specific code from these examples to keep the short

1. Reducing to a boolean

I have a file path (id) and I want to know, if the path belongs to any of the directories or files from the watching array.

return watching.reduce((acc, curr) => {
  return acc || id.startsWith(path.join(__dirname, curr));
}, false);

2. Converting an array of objects into a map using a specific property / key of the objects

I have an array of objects that I received from a database. But I want to convert them into a simple map for later processing. All these objects have a common structure and a key that stores a unique identifier (primary key).

Example of data,

// docs array
const docs = [{
  id: 'id-1',
  name: 'K Dilkington',
  style: 'orange',
}, {
  id: 'id-2',
  name: 'Lanky Fellow',
  style: 'googly',
}];

// result
const result = {
  'id-1': {
    id: 'id-1',
    name: 'K Dilkington',
    style: 'orange',
  },
  'id-2': {
    id: 'id-2',
    name: 'Lanky Fellow',
    style: 'googly',
  },
};

function makeMap(docs, key) {
  return docs.reduce((map, doc) => {
    map[doc[key]] = doc;
    return map;
  }, {});
}

We can now call the this function using makeMap(docs, 'id'), to build the map we desire.

3. Flatten an array of arrays

A very common case. I have an array of arrays and I want to combine them into a single array.

function flatten(arr) {
  return arr.reduce((acc, current) => {
    return acc.concat(current);
  }, []);
}


flatten([['1', '2'], ['3', 4], [{}, []]]) // => [ '1', '2', '3', 4, {}, [] ]

4. Doing the job of `filter()` - quite unnecessary :)

From an array of players, filter those with with valid ids (mongoId here).

game.players.reduce((acc, val) => {
  if (is.existy(val.mongoId)) {
    acc.push(val.mongoId);
  }
  return acc;
}, []);

5. A deep `Object.assign`

Object.assign copies values from source objects to given object, but it does a shallow copy and also mutates the given object.

I want a function (deepAssign), that would do a deep copy and would not mutate the given object.

const source = {
  l1: {
    inside: true,
    prop: 'in',
  },
  prop: 'value',
};
const target = {
  prop: 'out',
  l1: {
    prop: 'inisde',
  },
}

const shallow = Object.assign(source, target);
/*
shallow = {
  "l1": {
    "prop": "inisde"
  },
  "prop": "out"
}
*/

const deep = deepAssign(source, target);
/*
deep = {
  "l1": {
    "inside":true,
    "prop": "inisde"
  },
  "prop": "out"
}

function deepAssign(object, update, level = 0) {
  if (level > 5) {
    throw new Error('Deep Assign going beyound five levels');
  }

  return Object.keys(update).reduce((acc, key) => {
    const updatewith = update[key];
    if (is.not.existy(updatewith)) {
      return acc;
    }

    // lets just suppose `is` exists
    if (is.object(updatewith) && is.not.array(updatewith)) {
      acc[key] = deepAssign(object[key], updatewith, level + 1);
      return acc;
    }

    acc[key] = updatewith;
    return acc;
  }, Object.assign({}, object));
}

We are using recursion here and don't want to kill the stack, hence a simple check for - how many levels deep inside the source object we should care about.

6. Chaining Promises

I have four async functions that have to be executed in series, feeding the result of the previous function into next.

const arr = [fetchData, updateData, postData, showData];
const response = arr.reduce((acc, current) => {
  // (cue alarm sirens) no error handling
  return acc.then(current));
}, Promise.resolve(userId));

response.then(data => {
  // data is final response
});

That's it folks.

I found several more examples, however they were following more or less the same storylines with a twist or two of their own.

Finally, thanks for reading and if you have any magical use case of reduce() or if I have made any mistake in this post, I would love to know.

Generating documentation on the fly in express

Akshendra Pratap Singh — Sat, 09 Jun 2018 21:18:25 +0000

I little while ago, I decided to generate docs for some HTTP APIs I was about to build. The APIs were going to be used internally, so I didn't want to commit to any of the online solutions, like API Blueprint, Apiary or Swagger. And setting up something from scratch would have been, ... bit much. Also, I wanted the docs integrated within the code, avoiding any kind of context switch just to write documentation.

I was aware of jsdoc and esdoc, both of them allows us to write documentation in comments. However, their job is to document javascript code and not HTTP APIs. Then, I found a tool swagger-jsdoc, which generates swagger / OpenAPI specification from comments. This was just what I was looking for.

Let's see some `code` now

Just a simple server that lists animals and you can add your favorite animal too. Quite a novel concept.

const express = require('express');
const bodyparser = require('body-parser');

const app = express();
app.use(bodyparser.json({
  strict: false,
}));

const animals = [
  'panda', 'racoon', 'python',
];

app.get('/list', (req, res) => {
  return res.json(req.query.sort === 'yes' ? Array.from(animals).sort() : animals); // why is .sort inplace 😠
});

app.post('/add', (req, res) => {
  animals.push(...req.body.animals);
  return res.json({
    message: 'Added',
  });
});

app.listen(3000, () => {
  console.log('Server started at port 3000');
});

swagger-jsdoc requires comments to follow OpenAPI Specification, which is quite intuitive.

Adding documentation comments for /list route.

/**
 * @swagger
 * /list:
 *   get:
 *     summary: List all the animals
 *     description: Returns a list of all the animals, optionally sorted
 *     tags:
 *       - animals
 *     parameters:
 *       - in: query
 *         name: sort
 *         type: string
 *         required: false
 *         enum:
 *           - yes
 *           - no
 *     responses:
 *       200:
 *         description: List of animals
 *         schema:
 *           type: object
 *           properties:
 *             animals:
 *               type: array
 *               description: all the animals
 *               items:
 *                 type: string
 */

app.get('/list', (req, res) => {
  // ...
});

The first line is @swagger which helps swagger-jsdoc identify this comment block as swagger (OpenAPI) specification. Next few lines define the path, the method, a little summary, and description. tags are used to group the APIs.

The expected parameters, both query and path are described next. Our /list API expects an optional sort query parameter, which is used to decide whether the list of animals should be sorted or not, before sending.

Then we define the response. Status coming first, a little description and then the schema of the response. We are returning JSON here. However, it is easy to document other content types as well.

Same we will do for the /add request.

/**
 * @swagger
 * /add:
 *   post:
 *     summary: Add more animal
 *     description: Add animals to the list
 *     tags:
 *       - animals
  *     requestBody:
 *       content:
 *         application/json:
 *           schema:
 *             type: object
 *             properties:
 *               animals:
 *                 type: array
 *                 items:
 *                   type: string
 *     responses:
 *       200:
 *         description: Adds the animals in body
 *         schema:
 *           type: object
 *           properties:
 *             message:
 *               type: string
 *               default: 'Added'
 */
app.post('/add', (req, res) => {
  // ...
});

Now that we have the comments ready, we will hook up the swagger-jsdoc module.

// ... other modules
const swaggerJSDoc = require('swagger-jsdoc');

const app = express();
app.use(bodyparser.json({
  strict: false,
}));

const animals = [
  'panda', 'racoon', 'python',
];

// -- setup up swagger-jsdoc --
const swaggerDefinition = {
  info: {
    title: 'Animals',
    version: '1.0.0',
    description: 'All things animlas',
  },
  host: 'localhost:3000',
  basePath: '/',
};
const options = {
  swaggerDefinition,
  apis: [path.resolve(__dirname, 'server.js')],
};
const swaggerSpec = swaggerJSDoc(options);

// -- routes for docs and generated swagger spec --

app.get('/swagger.json', (req, res) => {
  res.setHeader('Content-Type', 'application/json');
  res.send(swaggerSpec);
});

// other routes

This will serve a swagger specification at /swagger.json. All that is left to do is render this spec in a more human-friendly way. I choose ReDoc for that. It has a simple setup.

Include an HTML file

<!DOCTYPE html>
<html>
  <head>
    <title>Quizizz Docs</title>
    <!-- needed for adaptive design -->
    <meta charset="utf-8"/>
    <link rel="shortcut icon" type="image/x-icon" href="https://quizizz.com/favicon.ico" />
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link href="https://fonts.googleapis.com/css?family=Montserrat:300,400,700|Roboto:300,400,700" rel="stylesheet">

    <!--
    ReDoc doesn't change outer page styles
    -->
    <style>
      body {
        margin: 0;
        padding: 0;
      }
    </style>
  </head>
  <body>
    <!-- we provide is specification here -->
    <redoc spec-url='http://localhost:3000/swagger.json' expand-responses="all"></redoc>
    <script src="https://cdn.jsdelivr.net/npm/redoc@next/bundles/redoc.standalone.js"> </script>
  </body>
</html>

We have set the http://localhost:3000/docs/swagger.json as the place to server JSON specification already. Let's set up a route to serve this HTML as well.

app.get('/docs', (req, res) => {
  res.sendFile(path.join(__dirname, 'redoc.html'));
});

The result,

There are more components to OpenAPI and swagger-jsdoc which make the process easier. You can write definitions for schemas/requests/responses that are used more than once and then use them in the docs. Check out Components Section | Swagger and Getting Started | Swagger JSDoc to define them in a JavaScripty way.

Code can be found here

Reloading the express server without nodemon

Akshendra Pratap Singh — Tue, 05 Jun 2018 15:46:49 +0000

I have been using nodemon for reloading express server and any other NodeJs code since I started writing backend NodeJS code. It does what it says on the label and does it pretty well. However, the problem with nodemon is lack of control and the fact that it seems to kill the process. You write a console.log statement and it will restart your whole server, which is all fine and dandy if your server starts quickly. But the situation becomes frustrating when restarting the server means reconnecting to a lot of external services.

Code to explain, what I am talking about

We start with a pretty simple project with the following directory structure

.
├── boot.js
├── index.js
├── package.json
├── routes
│   └── index.js
└── server.js

index.js is the main script. We call boot() here with makes the connection to external services. Once we are connected, we start the server() and listen at port 3000.

const boot = require('./boot');
const server = require('./server');
const debug = require('debug')('app');

boot()
  .then(() => {
    const app = server();
    app.listen(3000, () => {
      debug('Started on 3000');
    });
  })
  .catch((err) => {
    debug(err);
  });

boot.js makes the connections to external service, which can be a database or a queue. To simulate that, I am just using a promise that will resolve in 10 seconds.

const debug = require('debug')('app');

module.exports = function boot() {
  debug('Connecting to the satellites...');
  return new Promise((resolve) => {
    setTimeout(() => {
      debug('Connected to satellites...');
      resolve();
    }, 10000);
  });
};

server.js create an express app, adds all the middleware required and simply return the app.

const express = require('express');

const routes = require('./routes');

module.exports = function () {
  const app = express();

  app.get('/', (req, res) => {
    res.send('Nothing here...');
  });

  app.use('/', routes);

  return app;
};

Now the route/index.js, a simple route that is just silly.

const express = require('express');

const router = new express.Router();

router.get('/silly', (req, res) => {
  res.send('Now, I’ve noticed a tendency for this programme to get rather silly');
});

I like your code, but what now?

Well, to start the server we can use nodemon index.js.

As it is clearly visible, the app connects to external service (satellites) every time any change to the code is being made (noted by nodemon), which takes 10s + whatever extra time needed to restart the server.

Now the solution

To build something that can restart the server when the code is changed, we need a way to listen to file changes. NodeJS fs module does give facility to watch over files, but there is something better, chokidar.

Using chokidar we are going to listen for any changes to routes/index.js.

chokidar.watch('./routes').on('all', (event, at) => {
  if (event === 'add') {
    debug('Watching for', at);
  }
  if (event === 'change') {
    debug('Changes at', at);
    restart(); // assume that this exists
  }
});

Pretty straightforward. Now we need to figure out what to do in case of a restart. One of the first thing that comes to my mind is a way to restart the express server. As shown in index.js, we are starting an express app at port 3000. Surely we can't start an express app at port 3000 again. We need to stop this app first.

From express documentation, app.listen is basically doing this

app.listen = function() {
  var server = http.createServer(this);
  return server.listen.apply(server, arguments);
};

So, http.Server is what we need to stop. And by the grace of god we have a close method. Let's read the docs,

Stops the server from accepting new connections and keeps existing connections.
This function is asynchronous, the server is finally closed when all connections are ended and the server emits a 'close' event.

Oh, so all the connection need to be closed before we attempt to close the server. Okay, we need a way to monitor all the connections and manually destroy them if required. We will use server.on('connection') to get access to all the connections.

Now that we have a little bit of state to maintain, we will use this very simple object for that,

const state = {
  server: null,
  sockets: [],
};

We will start the server like this (remember server.js will return express() app).

function start() {
  state.server = require('./server')().listen(3000, () => {
    debug('Started on 3000');
  });
  state.server.on('connection', (socket) => {
    debug('Add socket', state.sockets.length + 1);
    state.sockets.push(socket);
  });
}
At the end will destroy all the sockets.
state.sockets.forEach((socket, index) => {
  debug('Destroying socket', index + 1);
  if (socket.destroyed === false) {
    socket.destroy();
  }
});

Before we go any further notice, the require('./server') inside the function start

This is done to avoid require cache

We also need to take care to require (CommonJS) cache. As an optimisation, require caches your code at the module level. Once it encounters a require it will compile the code inside the file and put the result in a cache. Next time it encounters the same require it will use the result saved in the cache.

This breaks all our plans. Since the changed code will never be loaded again. We should invalidate the cache, which is basically as simple as deleting the cached result.

function pathCheck(id) {
  return (
    id.startsWith(path.join(__dirname, 'routes')) ||
    id.startsWith(path.join(__dirname, 'server.js'))
  );
}

Object.keys(require.cache).forEach((id) => {
  if (pathCheck(id)) { // delete selectively
    debug('Reloading', id);
    delete require.cache[id];
  }
});

That's it basically, we have all the ingredients ready. All we have to do now is put them in the correct order.

const path = require('path');
const debug = require('debug')('app');
const chokidar = require('chokidar');

const boot = require('./boot');

const state = {
  server: null,
  sockets: [],
};

function start() {
  state.server = require('./server')().listen(3000, () => {
    debug('Started on 3000');
  });
  state.server.on('connection', (socket) => {
    debug('Add socket', state.sockets.length + 1);
    state.sockets.push(socket);
  });
}

function pathCheck(id) {
  return (
    id.startsWith(path.join(__dirname, 'routes')) ||
    id.startsWith(path.join(__dirname, 'server.js'))
  );
}

function restart() {
  // clean the cache
  Object.keys(require.cache).forEach((id) => {
    if (pathCheck(id)) {
      debug('Reloading', id);
      delete require.cache[id];
    }
  });

  state.sockets.forEach((socket, index) => {
    debug('Destroying socket', index + 1);
    if (socket.destroyed === false) {
      socket.destroy();
    }
  });

  state.sockets = [];

  state.server.close(() => {
    debug('Server is closed');
    debug('\n----------------- restarting -------------');
    start();
  });
}

boot()
  .then(() => {
    start();
    chokidar.watch('./routes').on('all', (event, at) => {
      if (event === 'add') {
        debug('Watching for', at);
      }

      if (event === 'change') {
        debug('Changes at', at);
        restart();
      }
    });
  })
  .catch((err) => {
    debug(err);
  });