Forem: Gian felipe theodorowicz

Data processing on-demand with Node.js streams

Gian felipe theodorowicz — Mon, 27 Mar 2023 16:29:39 +0000

For sure you already heard some phrases like, "node.js don't handle to process data" but, is it real?

A topic that ghost the life of seniors programmers, and not even is talk by juniors, Node.js streams.

What is

Node.js streams is data clusters, as arrays, but, streams don't is fully allocated in memory, what I mean is that they are allocated in pieces, this pieces is called chunks, basically is like you take a data, and break up it:

// static, alocated in memory
let nodejs = "node.js isn't single thread"

// stream pseudo-code
stream = ["nodejs", "isn't", "single", "thread"]

Notice that by this pseudocode the phrase is broking by fragments, they are called chunks, they are utils when we want to work on-demand data, as an example to who uses that type of on-demand data, is the chatgpt, which as the bot write a letter, is displayed to the user.

Transforming in code

Let's simule that we have a big reservoir of water purify, which extract water, and purify, to serve it to drink

The first step to transform it in to code to justly have a watter source


// this is a generator function, I will explain this in next topic
function * waterGenerator() {
  // 1e5 is 100000
  for(let i = 0; i < 1e5; i++){
    // here I create matter like god
    const matter = {
      // matter have water, so let's create it
      water: {
        // we need a way to represent the quantity in numbers, so let's use 'ml' (milliliters), that's physics baby
        mlQuantity: Math.random()
      }
    }
  }
}

Now that we have where take water, we have to find a way to extract it.

But I might have to explain the usability of generators function by the context of this example

Generators function

that's isn't my words, it's from mdn docs:

Generators are functions that can be exited and later re-entered. Their context (variable bindings) will be saved across re-entrances.

Generators in JavaScript — especially when combined with Promises — are a very powerful tool for asynchronous programming as they mitigate — if not entirely eliminate -- the problems with callbacks, such as Callback Hell and Inversion of Control. However, an even simpler solution to these problems can be achieved with async functions.

Transforming in code

proceeding...

To extract our water from our water fountain, we will to do this:

// I will use the new node's import so the module 'stream' will be 'node:stream', the prefix is use to explain that this is a native module from node
import { Readable } from 'node:stream'

// this 'new' word is opcional in this context, but if you choose to don't use, you won't have the IntelliSense help.

// it's important to say that readale streams is responsible to break our data to fragments (chunks).
const extract = new Readable({
  // the class 'Readable' from the stream module receive in contructor and config object.
  // inside the config object, I am passing a read module, this is responsible to the dirty work

  read() {
    // here we will consume
    for(const data of waterGenerator()){
      // an inportant topic that we have to know before start writing code is that streams only accepts buffers and strings as yours data so if we want to use objects as data...

      // we have to use JSON.stringify()
      const stringified = JSON.stringify(data)


      // okay, the way to pass the data throught the 'pipe' <i will explain pipe later> in readable streams is using the *this.push()*
      this.push(stringified)
    }

    // here we push null to represents that our data ended
    this.push(null)
  }
})

Now that we completed the extraction part of our water purifier, we have to clean this water, right?

When we want to transform our data, in the nodejs stream context, we have the class Transform that's responsible to trasnform the data and pass it throught the data pipeline

import { Transform } from 'node:stream'

const betterWater = new Transform({
  // notice that every class that have configs in their contructors at stream module receive a method which name is exact the same or close to the class name


  // the transform method receive 3 params that is:
  /**
   * @param('chunk') => represents the chunk data
   * @param('encoding') => represents the encoding format in the chunk
   * @param('cb') => represents the way that you return the data
  */

  // we won't use the encoding so I am passing it with a _ as prefix
  transform(chunk, _ec, cb) {

    // remember that when we implement the readable stream we pass the data as string? now we have to parse it to work with objects
    const data = JSON.parse(chumk)

    // let's think that water when is purified, loss quantity so:
    data.water.mlQuantity = data.water.mlQuantity - 0.1

    // remember, we have to parse the data to string to pass throught
    const stringfied = JSON.stringify(data)

    // here is the callback, the responsible to pass the data throught the pipe 
    // pipe receive an error as the first param, and the data as the second param
    // as I ain't handling any error, i will pass null as error, to represents that don't have any error
    cb(null, stringfied)
  }
})

Okay so now we have all the pipe... wait... where is the pipe?

// to create a pipeline, we extract the pipeline function from the stream module
import { pipeline } from 'node:stream'
// here we are extracting the promisify from util module to transform the pipeline in async pipeline
import { promisify } from 'node:util'

// transforming the pipeline function in to an async pipeline function
const asyncPipeLine = promisify(pipeline)


// notice that I am using await as top level, you don't need it as well but I like it,
await asyncPipeLine(
  // the first argument is the readableStream
  extract,
  // the middle arguments is the transforms, it can be 1, 2, 3 transforms, all you want.
  betterWater,
  // the last argument have to be a writable stream, in this case i will use the 'process.stdout', yes, console.log() is a writable stream and can be print as demand
  process.stdout
)

Conclusion

Wow, that's too much information, right? No.

Node.js streams it's a dense content but it's kind easy, read again as much you need, you will notice that I ain't pass tech specification about node.js streams, this article was wrote to who is diving in to node.js streams but still don't got the concepts in pratical therms.

the github source code:

github

Node.js HTTP server without any frameworks.

Gian felipe theodorowicz — Mon, 20 Mar 2023 13:17:19 +0000

Ok, so to create an API using Node.Js, first, it's necessary to install Express as dependency and then... Wait, we actually need 'Express' to build an http server in Node.Js?

Node.Js might needs external frameworks to build an API? see above.

What is HTTP

HTTP is a transfer protocol that enables people to... wait, let's go straight to the point, HTTP is text-based, no connection basically means people who access a site x, send requests to a server x that returns to who ordered, images, texts, and other media, after the request is answered, the connection between who ordered and who responded, is cut off.

Usually when we deal with HTTP requests in general, we guide ourselves with something, basically when something makes a request for a server, it expects an answer, but an answer is something very broad, so the famous status-codes were created, now we stop get guiding only by data and now we get guiding by number codes as well, But wait, where did the data go? Calm they are still sent by the server to whom you requested them, but now they are present within the request's content, the 'body'.

Now that I introduced to what is HTTP protocol, and clarified how it works, let's write and get out of the world of ideas and enter the world of codes, where there is no right or wrong, just what works and does not work... You may be wondering how my computer will send a request from my home to a server on the other side of the country or even the mainland, and I'll still get back an answer... Calm down, these and other headaches are already solved by modern technologies present both in programming languages natively, and even within browsers such as Firefox, Google, Opera, etc. Since we don't have to worry about the way these requests will be made, we will then start thinking uniquely and exclusively about our code. For this, we will use Node.js 19.8.1.

We should start questioning, how to create an API?

Probably, you when I asked, instantly aswered happy and smilling, that's easy!

fisrt we instantiate the Express using:

import express from 'express';
const app = express();

or by fastify:

import fastify from 'fastify';
const app = fastify()

... I say we couldn't limit ourselves only by using frameworks, justly by the fact they are built by something, that's why they are called, frameworks.

But after all, what precisely is this 'something'?

I can't answer with such certainty by what frameworks like 'express' or 'fastify' are built. But, natively, inside Node.js we have a way to build an HTTP server, that's inside the module 'http' or 'node:http'. Particuly this module have a native function, createServer, so to answer the question above mentioned...

// we started by calling inside Node.js the module 'node:http', and extracting the 'createServer'
import { createServer } from 'node:http'

// after, we instantiate it inside a constant, by any name.
const app = createServer()

This way, by theory, we already have a server receiving HTTP, but we will a little deeper:

import { createServer } from 'node:http'

// the function createServer, by default receive a property 'requestHandler', so we create a function and pass it as property
async function HttpHandler(
  request, response
){}

const app = createServer(httpHandler)

Now looks more beauty, right? well, we have an HTTP server that is receiving HTTP requests, and we are handling the requests... But what if I say that isn't working yet? yes, the createServer function have to be called with some arguments... we have the .listen(port) argument that is passed to the createServer can listen and send request on some network port.

import { createServer } from 'node:http'


async function HttpHandler(request, response){}

const app = createServer(HttpHandler)
.listen(3333)
.on('listening', () => console.log('http server ready'))

Notice that I add one more thing beyond .listen(), and to understand exactly what is this .on(), we might know what is events in Node.js, but don't worry, I will make a summary suitable to this article.

Events are notifications that can be handled by calling a function that emits these notifications, and that's notifications should have some data and headers inside, so we can consume them!

Now that you already know something about events, will should notice that the .on() is an event, and their headers or flag, is 'listening', so we handle it and order console.log() to notice it on our terminal.

Good right? we have our own HTTP server that is listening and aswering!

wait... you ran the code and isn't working with you? yes... I forgot to wrote a return haha!

import { createServer } from 'node:http'

async function HttpHandler(request, response){
  try {
    const data = request
    response.writeHead(200)
  } catch (e) {
    response.writeHead(500)
  }
}

const app = createServer(HttpHandler)
.listen(3333)
.on('listening', () => console.log('http server ready'))

Notice that now, I am really interacting with HTTP requests, in truth, I am litering returning differents status for whom requested, based on the state of my code, but we have a little problem when we use const data = requests, justly by the request is the result of an event, i.e, inside of my createServer, what is an event emmiter, not all events are requests, so to resolve it, we can use a solution provided by Node.js as well:

import { createServer } from 'node:http'

import { once } from 'node:events'

async function HttpHandler(request, response){
  try { 
    // now we are handle just when we have request data!
    const data = await once(request, 'data')
    response.writeHead(200)
  } catch (e) {
    response.writeHead(500)
  }
}

const app = createServer(HttpHandler)
.listen(3333)
.on('listening', () => console.log('http server ready'))

See how easy building an HTTP server in Node.js without any framework can be? But I would let highlight those frameworks help us to build applications faster, but I believe that we should know what we are doing before using any framework, I hope you liked it.