Forem: Kashyap Patel

Node Fundamentals: Modules

Kashyap Patel — Wed, 08 Apr 2020 04:56:25 +0000

This is a series of posts that will illustrate the what, why and how of Node. I'll be sharing my learnings from a course on Advanced NodeJS by Samer Buna offered on PluralSight. Any code samples tagged or attached will be available at the following repo.

jscomplete / advanced-nodejs

For help, ask in #questions at slack.jscomplete.com

Node Modules

Modularity is a first-class concept in Node.

There are two core modules involved.

require - It is a global function, but each module gets its own require function
module - It is also available globally and is used to manage all the modules we require with require.

Requiring a module in node is very simple concept.

To execute a require call, node goes through a following sequence of steps:

Resolving: Find the absolute file path of the module required.
Loading: Determined by the content of the file at the resolved path.
Wrapping: Gives every module its private scope and what makes require local to every module.
Evaluating: Eventually, VM does something to code.
Caching: When we require again, we don't go over all steps above mentioned.

`module` Object

Some interesting properties:

id: String Identifier, Usually full path to the module except for the root module. . identifier is used for the root module.
filename: String path to the file containing the module. So when you require a module from node_modules, it loads the content of a file into memory.
path: Array of paths that will used to find a module that is required. It starts with node_modules folder in current directory and goes all the way to the root directory. If it can't find a module in any of those directories, it will throw a Cannot find module 'module' error. Core node modules are an exception. When you require a core node module, it resolves immediately.

Let's consider following example.

// ./index.js
console.log("In ./index.js");
require("find-me");

// ./node_modules/find-me.js
console.log("In find-me.js");

This will result in output

In ./index.js
In find-me.js

Note, not only it loads the file, it also evaluates when you require it.

If you only want to load a file, and not evaluate it, you can use require.resolve(package). It will also throw an error if it can't find the package in any of the paths. This is used to determine if an optional package is installed or not.

If you have package installed in multiple folders present in path property, it will only resolve the first one it finds.

Usually, packages are not files, but are folders, with multiple files. It will use index.js or main property in package.json of the package. We can require any module, with a relative or absolute path.

Note module object available in index.js and in package index.js are different. module object in package index.js will have a reference to root index.js, and will be attached to it's parent property.

Module Wrapper

index.js

exports.id = 1 // this is ok
exports = { id: 1 } // this is not ok

module.exports = { id: 1 } // this is ok, why?

var  = 42; // local to this file

Only the things we export are available outside the module. How come variables we declare are magically scope. The answer is simple.

Before compiling a module, Node will wraps the module code in a function as follows.

> require('module').wrapper

(function (exports, require, module, __filename, __dirname) { ',
    '\n});

This is how each module gets its own require, exports and module object. These are just function arguments that are provided by wrapped function by node.

To see the values of these arguments you can just run the following code.

// index.js

console.log(arguments);

This will print all 5 arguments passed to the wrapper function.

The wrapping function's return value is the exports object reference. Note, exports is just a variable reference to module.exports. So, if we modify the whole of exports by assignment operator, we lose the module.exports module.

So, there's nothing special about require function. It takes the module name or path and returns the exports object. So in test case scenarios, where one might need to overwrite/mock require, we can do quite easily as follows.


require = () => {
  return { mocked: true };
};

console.log(require("somepackage")) // { mocked: true }

Let's say we have this simple function that takes an integer and a string and prints something.

// printStars.js
const print = (stars, header) => {
  console.log("*".repeat(stars));
  console.log(header);
  console.log("*".repeat(stars));
}

We want to run this function in two ways.

Through the command line as follows

$ node printStars.js 5 hello

Through require in another file as a module as follows.

// index.js

const printStars = require("printStars");
print(5, "hello");

To achieve this, we can leverage wrapping.
When it is run through Node CLI, require.main will be the same as module.

//printStars.js

const print = (stars, header) => {
  console.log("*".repeat(stars));
  console.log(header);
  console.log("*".repeat(stars));
}

if (require.main == module) {
  // When run as script
  print(process.argv[2], process.argv[3]);
} else {
  // being required by other file
  // export the module
  module.exports = print;
}

Caching

Imagine this case

// index.js

require("printFancy"); // Prints
console.log(require.cache); // Entry for `printFancy` module
require("printFancy"); // Nothing happens 

//printFancy.js

console.log("Hello Fancy");

Note, when we require printFancy first time, it will resolve, load, evaluate and cache the module.

However, when we require again, node has cached the module and so will repeat earlier steps again.

To circumvent this we can delete the cache on require.cache object, before the second call, with delete require.cache['absModulePath'] and printFancy will be called twice. But it is not the most efficient solution.

The easiest solution is to wrap the console log in printFancy.js in a function and export it.

// printFancy.js
module.exports = () => {
  console.log("Hello Fancy");
};

Now every time you require the module, just execute the exports.

// index.js

require('printFancy')() // Prints 
require('printFancy')() // Prints

Node Fundamentals: Buffer

Kashyap Patel — Wed, 08 Apr 2020 04:44:21 +0000

jscomplete / advanced-nodejs

For help, ask in #questions at slack.jscomplete.com

Buffer

Buffer is heavily used in Node to work with binary streams of data. It is a low-level object to represent a sequence of binary data.

A buffer is essentially a chunk of memory allocated outside v8 heap and we can put some data in memory, which can be interpreted in many ways based on the length of each character. That's why there is always a corresponding character-encoding associated with that buffer.

Whatever we place inside a buffer, doesn't have any character encoding, so to read it we need to specify an encoding.

Unlike arrays, once the buffer is allocated, it can't be resized. We can create a buffer in 1 of 3 major ways.

Buffer.alloc(n) - Allocates a 0-filed buffer of n bytes in memory.
Buffer.allocUnsafe(n) - Allocates a buffer of n byte in memory. This can cause a vulnerability as it can contain sensitive information.
Buffer.from() - Allocates a buffer with a value passed in argument.

const string = "touché";
const buffer = Buffer.from("touché");
console.log(string.length) // 6
console.log(buffer.length) // 7

Buffers are useful when we want to read an image file from a TCP stream or a compressed file or any other form of binary data.

Just like arrays and string, we can use operations like includes, slice, indexOf.

In the case of slice, unlike arrays, a sliced copy will use the same memory space.

String Decoder

When converting streams of binary data, use String Decoder module as it handles multi-byte characters much better. It gracefully handles incomplete characters, while calling toString method on buffer doesn't do that.

Node Fundamentals: CLI and Repl

Kashyap Patel — Wed, 08 Apr 2020 04:30:36 +0000

jscomplete / advanced-nodejs

For help, ask in #questions at slack.jscomplete.com

Node CLI and REPL

Node CLI comes with a variety of options to expose built-in debugging, multiple ways to execute scripts and other helpful runtime options.

Running a node command without any arguments starts a REPL.

R - Read
E - Eval
P - Print
L = Loop

When in REPL, you hit enter, it reads the command, executes it, prints the result and waits for the next command.

Helpful CLI Tips and Tricks

-c - Syntax check
-p - Print Command. e.g node -p "process.argv.slice(1) test 42" will print ['test', '42']

Helpful Repl Tricks and Tips

Autocomplete by Tab
rlwrap utility to track reverse search.

NODE_NOREADLINE=1 rlwrap node

_ is used to capture the last evaluated value.
Special commands that begin with a dot.
- .help to print all such commands.
- .break to break out of a multiline session.
- .load to load external script file
- .save to save current session
You can create your own repl with custom options by requiring a repl module and starting it with custom options. You can also control repl's global context in case of preloading a library of data.

Example below will start the repl in strict mode and doesn't print anything when result is undefined. Also, it will have lodash available globally.

  const repl = require("repl");
  const lodash = require("lodash");

  let r = repl.start({
    ignoreUndefined: true,
    replMode: repl.REPL_STRICT_MODE
  });

  r.context.lodash = lodash;

Node Fundamentals: Process

Kashyap Patel — Wed, 08 Apr 2020 04:26:36 +0000

jscomplete / advanced-nodejs

For help, ask in #questions at slack.jscomplete.com

`process` Object

process object provides a bridge between a Node application and its running environment.

process object is an instance of event-emitter. So we can emit and listen to events on the object

Useful properties and events

pid: Outputs the process id on OS level.
version: Read dependencies and versions

$ node -p "process.version"
{
  node: '13.12.0',
  v8: '7.9.317.25-node.30',
  uv: '1.35.0',
  zlib: '1.2.11',
  brotli: '1.0.7',
  ares: '1.16.0',
  modules: '79',
  nghttp2: '1.40.0',
  napi: '6',
  llhttp: '2.0.4',
  openssl: '1.1.1e',
  cldr: '36.1',
  icu: '66.1',
  tz: '2019c',
  unicode: '13.0'
}

env - Lists all environment variables. Better to always read values from a layer above it using config.
release.lts - node -p "process.release.lts" will be undefined if not on LTS version of node.
exit event - Will be invoked when the application has nothing else to do or a manual process.exit call is made.

process.on("exit", (code) => {
  // do one final synchronous operation
  // before node process terminates.
  // can't stop termination at this point
});

uncaughtException event - Will invoke when an error is not caught in the application. If not registered, node will print the stack trace and terminate. When it is registered, node will not terminate and will stay in an unstable state. It's better to forcefully terminate the process when this event is invoked.

process.on("uncaughtException", (err) => {
  // something wnet unhandled
  // Do any cleanup and EXIT forcefully.
});

Node Fundamentals: Architecture

Kashyap Patel — Wed, 08 Apr 2020 04:15:07 +0000

jscomplete / advanced-nodejs

For help, ask in #questions at slack.jscomplete.com

Node VM

v8, Default, Single-Threaded
Chakra by Microsoft, powers Edge browser

VM executes the Javascript code. So, v8 actually executes your JS code when you run a node application.

v8 Feature Groups

Shipping - Available by default
Staged - Not quite ready yet. flag --harmony
In-Progress - Less stable.

You can see the list of all group features by

node --v8-options | grep "in progress"
node --v8-options | grep "staged"

Overview

Node is more than a wrapper for v8, it provides APIs for working with the file system, operating system, binary data, networking and much more. It mainly constitutes of the following components.

v8

Node v8 uses v8 C++ API. So, node API eventually executes C++ code using V8 object and function templates.

When node is done waiting for IO or timers, it usually has callback functions to invoke back to. When it's time to invoke these callbacks, node passes control from libuv to the v8 engine. When v8 executes the callback, it passes control to Node itself.

When control is with v8, Node can not execute any Javascript code as v8 is single-threaded. Node will simply wait until v8 is ready for more operations.

This is actually programming in node so simple yet powerful as we don't have to worry about locks or race conditions. There is only one thread where our Javascript code runs.

libuv

libuv is a C library developed for Node, but now it's used by Rust, Julia and others. It's used to abstract the non-blocking I/O operations. It provides a consistent interface across many operating systems.

It handles operations on the file system, TCP/UDP sockets, child processes, and others. libuv includes a thread pool to handle what can't be done asynchronously at OS level.

libuv also provides node with Event Loop.

More Dependencies

Apart from v8 and libuv, node has a few more dependencies.

http-parser: Library for parsing HTTP messages. It works for requests and responses, works with a very small memory footprint.
c-ares: Enables performing asynchronous DNS queries.
OpenSSL: Mostly used in tls and crypto modules. Provides the implementation for many cryptographic functions
zlib: Used for fast streaming compression and decompression interfaces.

End to End Testing with Cypress

Kashyap Patel — Wed, 26 Jun 2019 22:47:05 +0000

End-to-end testing lets us test our whole application from start to finish. It involves ensuring that all the integrated pieces of an application function and work together as expected. E2E tests simulate real user scenarios, essentially testing how a real user would use the application.

Selenium

Selenium is an application written in Java that lets us test our apps from the point of view of a browser. It’s a little bit old but lets us do a lot of wonderful things that we might not otherwise be able to do with a nice ecosystem of integration tools.

Here’s how it works, Selenium has a Java Service that listens for commands. You write test code in any language, which then uses a library to speak the language of Selenium called “webdriver API” and send these commands to the service. The service then queues all these commands and tries to send them to any given browser, the real browsers where tests are being executed.

Unfortunately, there are a number of different abstractions, integrations, and interfaces where things could go wrong. When a test fails, it could be any one of these issues:

Our test code has an issue
Platform dependent API that works with Selenium might be behaving differently
A different version of Selenium might work differently based on the usage
Things could go wrong in Java service
Browser web driver can have bugs, Message multiplexing can go wrong if socket connection to browser goes haywire resulting in funny front-end behaviors

There are a lot of places where things can go wrong with this architecture. Plus the damn thing is SLOW.

We, humans, have good intuition skills, but computers and code do not. When our test code is running, it does exactly what it is told to do without taking any guesses. And if it clicks a button and starts to re-render, we need to be very careful about knowing whether it finished rendering before we try to do the next thing. Any slowness in asynchronous services like DB, network, disk read/writes can break the order of the tests to be executed, which can cause a cascading failure of tests.

When we take this heavy architecture and try to merge with our apps, things get extra juicy. It’s a recipe for frustration. We’d write a commit, run tests locally, everything looks good, we’d push it to the build and, alas, some test would fail. Then we’d run that test again, as it’s not something the commit should have broken, and it would pass! Hurrah!

Our app should be simple, fast and consistent. Right? Our pages make complex interaction to provide rich user experience consistently so our users can trust our service more which leads to increased user retention and returns.

Overall, Selenium is an extremely popular piece of software that has become the basis of many automation strategies. It comes with an excellent community, support, and tools, and allows us to do cross-browser testing. However, it comes with its limitations. Can we do better?

Cypress

Welcome Cypress! Cypress is an OSS written in Javascript which lets us write E2E tests like Selenium. At PerfectTense we have E2E tests for dreaded contenteditable. I chose not to use Selenium due to its setup process and I had a general impression of it being slow and brittle. In contrast, Cypress can be started with two simple commands, and running tests is simple. When I first ran Cypress, my mind was blown by the speed at which sample tests ran. Cypress also creates several E2E tests when it starts for the very first time. These are helpful guidelines on what, why and how to test and how to use. They are also great for learning how to use their APIs to visit a page, find and interact with an element, deal with cookies, ajax requests, and more.

What makes Cypress different, is that is built on a new architecture that eliminates flakiness. It’s able to achieve by running in the same run-loop as your application whereas Selenium executes remote commands through a network. It works with any server-rendered pages (like our HAML pages) or a web app written in any modern framework like React, Vue, etc.

Cypress tests are written in Javascript with familiar matchers from Mocha, Chai, and Sinon. They are executed in the same environment of the browser itself and require no special setup, language or driver bindings. JS to rule them all.

Because of no network latency and no need for any setup, Cypress runs extremely fast. You can actually do TDD with full end-to-end tests for the very first time. You can develop faster while driving the entire dev process with tests because you can see your application, you have access to the developer tools, you have access to server-side code, and changes are reflected in real time. It’s so fast, it’ll let you find bugs you didn’t know existed.

Here is a comparison of timings of tests written in Cypress and Capybara that visit the same page, perform the same actions with the DOM and have the same expectations in the end.

Some other useful, nice-to-have features and benefits that Cypress provides:

Reliably test viewport and scroll behaviors.
Replay DOM snapshots and walk through our tests.
Built-in test runner with a web UI to run any test individually or all together.
Easily pause and debug in between expectations.
Browser screenshots are taken automatically on failure, a video is recorded of your entire test suite when tests are run from the CLI.
Actual window object from the browser to expect messages sent to globals.
AGAIN It’s blazing fast. Our tests in cypress run 3–4x faster compared to Selenium Webdriver.

Current Limitations

Only works with Chromium-based browsers and tests can only to be written in Javascript. Cypress team has been actively working on making it compatible for other browsers. You can track the progress here.
Testing iframes requires some workarounds but it's certainly possible.

To summarize, Cypress is still a new tool with a growing community, a unique architecture that manipulates DOM fast and reliably without any network latency, and is built with developer-friendliness in mind. It also has excellent documentation, with a lot of snippets that can help us write more manageable tests and stick to best practices.

Hope this helped!

Forem: Kashyap Patel

Node Fundamentals: Modules

jscomplete / advanced-nodejs

For help, ask in #questions at slack.jscomplete.com

Node Modules

module Object

Module Wrapper

Caching

Node Fundamentals: Buffer

jscomplete / advanced-nodejs

For help, ask in #questions at slack.jscomplete.com

Buffer

String Decoder

Node Fundamentals: CLI and Repl

jscomplete / advanced-nodejs

For help, ask in #questions at slack.jscomplete.com

Node CLI and REPL

Helpful CLI Tips and Tricks

Helpful Repl Tricks and Tips

Node Fundamentals: Process

jscomplete / advanced-nodejs

For help, ask in #questions at slack.jscomplete.com

process Object

Useful properties and events

Node Fundamentals: Architecture

jscomplete / advanced-nodejs

For help, ask in #questions at slack.jscomplete.com

Node VM

v8 Feature Groups

Overview

v8

libuv

More Dependencies

End to End Testing with Cypress

Selenium

Cypress

Current Limitations

Further Readings:

`module` Object

`process` Object