Forem: Fabrizio Lallo

4 FREE Resources To Boost Your Node.js Developer Career ⚡

Fabrizio Lallo — Sun, 04 Dec 2022 23:00:00 +0000

Do you want to speed up your Node.js career?

These are the resources I wish I knew when I started learning Node.js, and guess what? They are completely FREE:

1. Event Loop 🔁

An animated guide to understanding how Event Loop works under the hood.

What's better than learning through animations and graphics?

✍🏻 Andrew Hu - https://dev.to/nodedoctors/an-animated-guide-to-nodejs-event-loop-3g62
✍🏻 Fabrizio Lallo - https://dev.to/nodedoctors/animated-nodejs-event-loop-phases-1mcp

2. Broken Promises 💔

Using promises correctly is crucial to maintain Node.js performance and writing clean code.

✍🏻 James Snell & Matteo Collina- https://www.youtube.com/watch?v=yRyfr1Qcf34&t=1181s

3. Async Iterators 🔃

Knowing iteration protocols is essential to write more modern and idiomatic Javascript code.

✍🏻 Luciano Mammino - https://github.com/lmammino/iteration-protocols-workshop

4. Streams 🚰

Learning streams allow us to build efficient Node.js applications.

You can build a video streaming service or ingest and transform terabytes of data, which is in great demand nowadays.

✍🏻 Luciano Mammino - https://www.youtube.com/watch?v=yRyfr1Qcf34&t=1181s

Let's Connect 🌎
Follow us on Twitter @fabrizio.lallo and @AndrewHu368 for more Node.js content

5 Top Node.js Streams Resources You Should Learn to Level Up Your Node.js Skills 🚀💯

Fabrizio Lallo — Fri, 02 Dec 2022 12:27:31 +0000

If you want to master Node.js, stop screwing around 🛑.

Learn Streams.

Streams are the fundamental block of the essential Node.js features:

HTTP server
File management
Encryption and Decryption

This means you can build (TODAY) a video streaming service or ingest and transform terabytes of data just by learning streams.

You can use these resources and unlock all the Node.js power 🔐🔥.

1. Streams workshop 📒

Don’t you know anything about theory and practice about streams? This workshop is the gold standard and the perfect place to get started with streams—an ideal combination of theory and practical exercises.

Author ✍🏻: Luciano Mammino

Link 🔗: https://github.com/lmammino/streams-workshop

2. Everything you should know about Node.js Streams 📽️

If you are more of a “video guy,” these are fantastic walkthrough about how streams work and how you can use them in your project.

PLUS 🆕: the first video contains the new stream features introduced in Node.js v18.

Author ✍🏻: Erik Wendell

Link 🔗: https://youtu.be/BdePYKgrMh0

EXTRA:
Author ✍🏻: Luciano Mammino

Link 🔗: https://youtu.be/ldcfYB_mo6Q

3. Understanding Streams in Node.js 🔁

Modern Node.js streams with async iterator.

Author ✍🏻: Liz Parody

Link 🔗: https://nodesource.com/blog/understanding-streams-in-nodejs/

4. NodeJS Backpressuring in Streams 🚱

Backpressure is an important technique to prevent your application from being overwhelmed by data. It is essential to understand and master backpressure.

Author ✍🏻: Vladimir Topolev

Link 🔗: https://enlear.academy/nodejs-backpressuring-in-streams-52638f505e1b

5. How To Code A Video Streaming Server in NodeJS 🕹️

All Node.js stream knowledge is put into practice to build a real-world example: a video streaming service.

Author ✍🏻: Abdisalan Mohamud

Link 🔗: https://youtu.be/ZjBLbXUuyWg

If you liked the article, follow us on Twitter @fabrizio.lallo and @AndrewHu368

Credits to Conny Schneider
https://unsplash.com/photos/pREq0ns_p_E

Node.js animated: Event Loop Phases

Fabrizio Lallo — Sat, 19 Nov 2022 20:52:26 +0000

In the previous article, Node.js animated: Event Loop, you explored how Libuv and the event loop enable asynchronous task handling in Node.js' single-threaded environment. We simplified the event loop as a mechanism that pushes callbacks from a single event queue to an empty call stack. In reality, the event loop is composed of multiple phases, each responsible for specific asynchronous tasks.

The event loop phases

A phase is a FIFO queue of callbacks to execute. When the event loop enters a given phase, it executes callbacks until the queue is exhausted or the maximum number of callbacks is run and moves to the next stage.

The event loop has 6 phases, and they run in the following order:

Timers: executes callbacks scheduled by setTimeout() and setInterval().
Pending callbacks: executes I/O callbacks deferred to the next loop iteration.
Idle-prepare: only used internally.
Poll: retrieve new I/O events and execute I/O-related callbacks
Check: executes callbacks scheduled by setImmediate().
Close callbacks: some close callbacks, e.g., socket.on('close', …).

The event loop doesn’t constantly keep spinning. If there are no active I/O handlers and the event loop doesn’t need to process any callbacks, the Node.js program automatically exits. On the other hand, the event loop stops on the poll phase to capture incoming requests whenever you create a web server and the phases are empty.

Timers and check phase

You are ready to apply your newly acquired knowledge. Still, you can be disappointed when running a setTimeout after a setImmediate, or vice versa, because the result is not deterministic.

setTimeout schedules a callback after at least the defined milliseconds have elapsed. The CPU could be busy and Libuv is unable to set the task as complete before the event loop processes the timer phase. Consequently, the setImmediate callback could be run in advance, although the timer is the first phase.

In this example, the CPU is not busy, and the setTimeout callback is immediately added to its dedicated queue before setImmedate.

In this second example, the event loop starts processing the timer phase, but the timer has not expired yet, so setImmediate callback is run in advance.

Exploring the poll phase

You are now wondering how I can test the phases processing order. The following code snippet will help you prove the order of the phases.

Once Node.js starts the program, the event loop processes all the queues and blocks on the poll phase waiting for incoming requests. When someone makes a GET HTTP request to the server, the event loop goes through these steps:

The event loop pushes the request handler on the call stack.
A setImmediate callback is added in the check phase.
A setTimeout callback is scheduled in the timers phase.
The event loop moves to the check phase. The setImmediate callback is popped off from the queue and pushed on the call stack for execution.
The event loop does not move any callbacks from empty queues.
The setTimeout callback is popped off from the timer queue and pushed on the call stack for processing.

The code is available at:

Animated Node.js-02-event-loop-phases repository

Conclusion

Now, you understand the event loop and its phases. You can also tell the difference between setTimeout and setImmediate and expect a non-deterministic behavior when running them in sequence. In addition, you know that event loop blocks on the poll phase waiting for new I/O requests on a Node.js web server.

But where are promises handled in Node.js?

You will explore it in more detail in the following articles.

Resources

If you liked the article, follow us on Twitter @fabrizio.lallo and @AndrewHu368

What is Node.js?

Fabrizio Lallo — Sun, 30 Oct 2022 21:38:22 +0000

JavaScript had the reputation of being a toy language. A tool to add interactivity to web pages. For a long time, its use cases were limited to the browser, but the creation of Node.js completely changed its role in the modern web development world.

Node.js allows us to run JavaScript outside the browser, and out of nowhere, a limited language became the favorite companion for thousands of developers working on small and big projects.

Its vibrant community created many tools that power many applications on servers, desktops, and embedded systems.

Node.js is commonly used with Express or Fastify to power complex backend services.
Electron package Node.js and Chrome to create cross-platform desktop applications. It is used in production to power programmers’ favorite editor VSCode and messaging applications like Microsoft Teams.
Embedded systems can leverage low.js to develop applications.

Why Node.js?

Before Node.js

Contrary to popular belief, Node wasn’t invented to allow JavaScript outside the browser. The latter was a positive outcome. It was developed to create more performant web servers.

Ten years ago, handling multiple user connections involved the creation of a new thread, an independent unit of execution within the web server process. This methodology implied two problems:

If you perform a database query, it blocks the entire thread from executing other operations until it receives the result.

Handling concurrent connections with associated threads requires high memory consumption and increased difficulty in managing synchronization between them.

The Node.js way

JavaScript and Node.js solve the problem of threads' high memory usage and synchronization by using a single thread paired with an event loop mechanism.

In the code snippet above, Node.js can handle concurrent connections and run other pieces of code because the database query execution is delegated to the event loop. The database operation has a callback function which is run when the event loop notifies the single thread that the query is completed.

Node.js internal overview

To understand Node.js, we need to understand V8.

V8 is a C++ program that runs JavaScript code and converts it into instructions understandable by the computer microprocessor. V8 is included in Google Chrome and other browsers to run our websites JavaScript and enable our rich web experience.

The V8 engine is not limited to modern browsers but can also be embedded into any C++ program and extended with more functionalities. Not by chance, Node.js is a C++ program that integrates the V8 engine and other libraries to enrich JavaScript with new features:

libuv, the C++ library responsible for the event loop, TCP and UDP connection sockets, DNS resolution, file system operations, data encryption, etc.
C++ bindings are responsible for invoking C++ code when running Node.js-specific JavaScript code.
The JavaScript core API.

Conclusion

Node.js clever design and event loop has empowered thousands of developers and products to create web servers capable of handling thousands of concurrent connections without the hassle of threads’ high memory consumption and synchronization. Node.js event loop capabilities are not limited to database queries and concurrent connections, but they are also used in production to process large files, encrypt/decrypt data, perform network requests, etc.

You will explore in more detail the event loop in the following article.

If you liked the article, follow us on Twitter @fabrizio.lallo and @AndrewHu368

ORM Database Fundamentals

Fabrizio Lallo — Tue, 18 Oct 2022 17:07:42 +0000

What is an ORM Database?
Query Builder
Identity Map
Unit of Work
Conclusion

What is an ORM Database?

An ORM Database (Object Relational Mapping) is concerned with helping your application achieve persistence. Persistence simply means we would like our application’s data to outlive the application process. But, expressly, Object Relational Mapping is a technique to achieve a mapping between database data and objects used with your programming language.

Object Relational Mapping achieves this goal through fundamental concepts:

Query Builder
Identity Map
Unit of Work

Query Builder

A Query Builder provides an API designed for constructing a DQL (Domain Query Language) query in several steps. It provides a set of classes and methods that can build queries programmatically. Usually, you can use Query Builder API to build a query quickly or just use the raw DQL language if you need something more complex that it’s easy to write in raw DQL language.

So let’s imagine we need to execute the following SELECT query in our application:

SELECT * FROM Users
WHERE name='John';

A Query Builder API to select rows in a table could be:

orm.select('*')
  .where({
    name: 'John',
  })

Most of the time, people think this is the definition of ORM, but as we’ll see, two other fundamental concepts are essential for an ORM to be defined as such.

Identity Map

An Identity Map keeps a record of all objects that have been read from the database in a single business transaction. Whenever you want an object, the ORM checks the Identity Map first to see if that object is already contained. So, it acts as a cache.

Having an identity map is crucial for avoiding cases when you read the same database record in two different objects. In this case, when you update them both, you'll have a lousy time writing the changes out to the database correctly.

Furthermore, reading the same record several times can cause an expensive cost in remote calls to the database. So, not loading the same data twice helps avoid profound problems writing changes to the database and speeds up your application.

Let’s put it into practice with some javascript code:

// Both queries return the same user
const user1 = orm.select('*')
  .where({
    name: 'John',
      surname: 'Doe'
  })

const user2 = orm.select('*')
  .where({
    name: 'John',
      surname: 'Doe',
    age: 27
  })

// identity map in action
console.log(user1 === user2) // true

Unit of Work

A Unit of Work keeps track of every change you make during a business transaction that can affect the database. When you're done, it figures out everything that needs to be done to alter the database due to your work.

Keeping track of what you've changed when pulling data in and out of a database is fundamental. Otherwise, that data won't be written back into the database.

You can change the database with each change to your object model, but this can lead to many very slow and very small database calls. Furthermore, it requires you to have a transaction open for the whole interaction, which is impractical if you have a business transaction that spans multiple requests.

Let’s see what a unit of work looks like:

const user1 = orm.select('*')
  .where({
    name: 'John',
      surname: 'Doe',
    age: 27
  })

user1.age = 30;

const user2 = orm.select('*')
  .where({
    name: 'Jane',
      surname: 'Doe',
    age: 25
  })

user2.age = 32;

// it will apply all the changes done in this unit of work
// so user1 and user2 age will be updated in the database
// using a single transaction
orm.flush();

Usually, when there are a lot of changes, an ORM should take care to batch the changes in several transactions in order not to create a single heavy transaction.

Conclusion

In this blog post, we explored the fundamental concepts of a Database ORM: Query Builder, Identity Map and Unit of work.

Certainly, an ORM has many features to make easy interaction with a database, but these are the main building blocks.

Stay tuned for the next blog posts. 👀

Follow me on Twitter if you enjoyed this blog post @fabri.lallo

5 tips to improve software development and release process

Fabrizio Lallo — Sun, 16 Oct 2022 17:47:07 +0000

Every software engineering team follow specific development practices to build and release software.

Here are 5 tips to improve the software development and release process.

1) Stop using pull requests and code reviews

If you write tests, you don't need code reviews. Leave pull requests and code reviews where they belong so to open-source projects.

You can still rely on disciplined pair programming.

2) Use Trunk Based Development

Trunk Based Development is a version control management practice where developers merge small, frequent updates to a core “trunk” or main branch.

Stop leaving those branches open for days, weeks, or months. Stop making it hard to merge changes between branches.

Push small changes at a time.

3) Write tests

Writing tests is the most powerful weapon to protect your application from bugs.

Cover every surface of your application through unit tests, e2e test etc..

4) Setup a CI/CD pipeline

Let the CI/CD pipeline take care of building your application, executing tests and deploying it.

Automate as much as possible.

5) Trust your team

Last but not least, trust your team.

Every team member should trust each other and feel autonomous.

Follow me on Twitter if you enjoyed this blog post @fabri.lallo

Moon phases with Node.js streams over HTTP

Fabrizio Lallo — Thu, 13 Oct 2022 08:30:24 +0000

Introduction to Node.js streams

Streams are a fundamental building block of many Node.js functionalities. They provide a way to consume data as soon as it’s available and send the output as soon as the application produces it.
Just think about data-intensive applications which can’t afford to accumulate all data in memory and process them all at once. In this case, applications must chunk data not to overflow the memory, which is what streams do.

A crucial example of stream usage in Node.js is the HTTP server.
HTTP server use streams in order never to buffer entire requests or responses so that the user can stream chunked data.

There are four fundamental stream types within Node.js:

Writable: streams to which data can be written
Readable: streams from which data can be consumed
Duplex: streams that are both Readable and Writable
Transform: Duplex streams that can transform the data as it is written and read

In this first blog post, we will explore Readable and Writable streams reproducing moon phases over an HTTP server.

🌕 🌖 🌗 🌘 🌑🌘🌗🌖🌕

Let’s code

The goal is to create a simple server which continuously shows moon phases with emojis. So we need to define two streams:

A Readable stream from which we can read the emojis
A Writable stream which is represented by the HTTP server response

Let's start by initializing the project and installing the required dependencies.
For this project, I'm using Node.js v16.17.1.

npm init -y

npm install node-emoji

The library node-emoji adds emojis support to Node.js

We need only one file, so let's create an index.js file. Firstly, we focus on building the Readable stream, which is the core task of this project:

class MoonPhasesStream extends Readable {
  constructor (options) {
    super(options);
    this.moonPhases = [];
    this.index = 0;
    this.moonPhases.push(emoji.get(':full_moon:'));
    this.moonPhases.push(emoji.get(':waning_gibbous_moon:'));
    this.moonPhases.push(emoji.get(':last_quarter_moon:'));
    this.moonPhases.push(emoji.get(':waning_crescent_moon:'));
    this.moonPhases.push(emoji.get(':new_moon:'));
    this.moonPhases.push(emoji.get(':waxing_crescent_moon:'));
    this.moonPhases.push(emoji.get(':first_quarter_moon:'));
    this.moonPhases.push(emoji.get(':moon:'));
  }

  _read () {
    const LAST_MOON_PHASE = 8;
    const FIRST_MOON_PHASE = 0;
    this.push(this.moonPhases[this.index++]);
    if (this.index === LAST_MOON_PHASE) {
      this.index = FIRST_MOON_PHASE;
    }
  }
}

We've just created a custom Readable stream; wow.

Let's break it down:

First thing first, to define a custom Readable stream, we need to create a class that extends the Readable stream base class
In the constructor, we initialize an empty array and fill it with the eight moon phases emojis. We also need an index to restart the moon phases when it ends.
Now we can override the _read() method, which is responsible for producing data (emojis in this case). In this method, we just 'produce' data using the push method of the Readable stream. When the moon phases flow is over, we restart it setting the index again to 0

Okay, so now that we have our MoonPhasesStream, it's time to define the consumer of these emojis streams.
Let's define our Writable stream using the Node.js HTTP server:

const server = http.createServer((req, res) => {
  const moonPhasesStream = new MoonPhasesStream();

  pipeline(moonPhasesStream, res, (error) => console.log(error));
});

server.listen(3000);

Very simple, no?

Let's explain:

In Node.js, the request object is a Readable stream so that we can consume data from it. But the object we are interested in is the response which is the Writable stream on which the emoji will flow. So, we won't need to build a custom Writable stream as we did for the Readable one.
There are several ways to combine streams, and the most used are pipe()and pipeline(). The difference is that with pipe() if there's an error, we need to end and destroy each stream (to avoid memory leaks) manually:

readableFileStream
  .on('error', (err) => readableFileStream.destroy())
  .pipe(decompressStream)
  .on('error', (err) => decompressStream.destroy())
  .pipe(decryptStream)
  .on('error', (err) => decryptStream.destroy())

pipeline() solve this problem by taking care of the stream destruction:

pipeline(
  readableFileStream,
  decompressStream,
  decryptStream,
  (err) => {
    if (err) {
      console.error(err)
    } else {
      console.log('Process completed')
    }
  }
)

That's why we also use pipeline() to compose Readable and Writable streams:

  const moonPhasesStream = new MoonPhasesStream();

  pipeline(moonPhasesStream, res, (error) => console.log(error));a

Now we're ready to try our server. Just start it:

node index.js

and then open a terminal and run:

curl localhost:3000

You will see an infinite moon phases cycle.

If you want to show just one moon phases cycle, it's possible to stop the stream by pushing a null value.

Conclusion

In this first blog post, we created a custom Readable stream to produce data and used the HTTP server response, a Writable stream, to consume them.

Here is my GitHub repository: https://github.com/fabrilallo/moon-phases-stream

Stay tuned for the next blog posts. 👀

Follow me on Twitter if you enjoyed this blog post @fabri.lallo

Forem: Fabrizio Lallo

4 FREE Resources To Boost Your Node.js Developer Career ⚡

1. Event Loop 🔁

2. Broken Promises 💔

3. Async Iterators 🔃

4. Streams 🚰

5 Top Node.js Streams Resources You Should Learn to Level Up Your Node.js Skills 🚀💯

1. Streams workshop 📒

2. Everything you should know about Node.js Streams 📽️

3. Understanding Streams in Node.js 🔁

4. NodeJS Backpressuring in Streams 🚱

5. How To Code A Video Streaming Server in NodeJS 🕹️

Node.js animated: Event Loop Phases

The event loop phases

Timers and check phase

Exploring the poll phase

Conclusion

Resources

What is Node.js?

Why Node.js?

Before Node.js

The Node.js way

Node.js internal overview

Conclusion

ORM Database Fundamentals

Table of Contents

What is an ORM Database?

Query Builder

Identity Map

Unit of Work

Conclusion

5 tips to improve software development and release process

1) Stop using pull requests and code reviews

2) Use Trunk Based Development

3) Write tests

4) Setup a CI/CD pipeline

5) Trust your team

Moon phases with Node.js streams over HTTP

Introduction to Node.js streams

Let’s code

Conclusion