Forem: Rahul Ramteke

AI for software architecture

Rahul Ramteke — Thu, 01 Feb 2024 14:40:09 +0000

Designing a system, from small to big, starts as a vague process and then incrementally you narrow down
on a particular approach. It'd be absurd to ask someone:

Design a local first webapp with auto and manual saving

Design a system which can onboard users, teams and organisations and manage role based access to resources

and expect a fully usable answer right out of the gate in 5 minutes.
But the urge to do this with AI is quite present, if not immense.

And to do so, would set you up for failure, more than it'd have by doing the same to you team member. Because
they would understand context, and read between the lines and might actually give something valuable!

And yet, I find myself talking to ChatGPT endlessly about how to build the features I mentioned above.

Your experience is context

Before I dive in, let me show you what I mean. The local sync problem I mentioned before, well that was real.

I am quite new to frontend and its mysterious ways of working, but I understand systems. I understand the
fundamental blocks enough to reason about things, or at least get a sense of direction.

I am also, a Functional Programming enthusiast, and tried my hand at FRP(Functional Reactive Programming), which led me to RxJS.

Now, the folks who use RxJS as a daily driver must have gotten their wheels turning and good chance they solved it already.

But not me, I don't use RxJS, I just understand pipelines!

Pipelines can be mapped, filtered, buffered, merged but wait, the events that my app is generating is a stream
and hence an Observable, even auto save is just a stream of button presses. I think there's something here that can solve my problem.

And that's it!

A general sense of direction but with concrete ideas is just enough, that current LLMs require to help you with design.

Lemme show you the final code which me and ChatGPT arrived at:

const manualSaveTrigger$ = typeof window === 'undefined' ? from([]) : fromEvent<KeyboardEvent>(document, 'keydown').pipe(
    filter(event => event.key === 's' && (event.metaKey || event.ctrlKey)),
    map(() => [] as StateChangeEvent<any>[])
);

export function setupSyncPipeline(
    projectId: string,
    eventSource$: NonNullable<SyncManagerState['eventSource$']>,
    syncManagerStore: UseBoundStore<StoreApi<SyncManagerState>>,
    bufferTimeInMs: number,
) {
    const syncManagerState$ = new BehaviorSubject<SyncManagerState>(syncManagerStore.getState());
    syncManagerStore.subscribe(state => syncManagerState$.next(state));

    const sharedEventSource$ = eventSource$.pipe(
        share(),
    );

    const bufferedEvents$ = sharedEventSource$.pipe(
        tap(() => syncManagerStore.getState().setSyncStatus('unsynced')),
        bufferWhen(() => merge(
          timer(bufferTimeInMs), // Emit when buffer time elapses
          manualSaveTrigger$    // Or when manual save is triggered
        )),
        filter(events => events.length > 0)
    );

    const result = bufferedEvents$.pipe(
        map(mergeEvents),
        withLatestFrom(syncManagerState$),
        filter(([_, state]) => state.isSyncingAllowed), // Check if syncing is allowed
        tap(() => syncManagerStore.getState().setSyncStatus('syncing')),
        concatMap(([mergedEvents, state]) => 
            ajax<{server_sequence: number}>({
                url: getSyncApiUrl(projectId),
                method: 'POST',
                headers: {
                    'content-type': 'application/json',
                },
                body: { updated_artifacts: mergedEvents, server_sequence: state.serverSequence },
                withCredentials: true,
            }).pipe(
                tap({
                    next: (response) => {
                        syncManagerStore.getState().setSyncStatus('synced');
                        syncManagerStore.getState().setServerSequence(response.response.server_sequence);
                    },
                    error: () => {
                        syncManagerStore.getState().setSyncStatus('errored');
                        syncManagerStore.getState().setIsSyncingAllowed(false);
                    }
                })
            )
        )
    );

    const subscription = result.subscribe({
        next: response => console.log('Merged event synced:', response),
        error: err => console.error('Error syncing merged event:', err),
        complete: () => console.log('All merged events synced')
    });

    syncManagerStore.getState().setSubscription(subscription);
}

const mergeEvents = (events: StateChangeEvent<any>[]) => {
    // Merge logic here
    return events.reduce(
        (merged, event) => {
            merged[event.source] = event.newState;
            return merged
        }, 
        {} as Record<string, any>
    );
};

function getSyncApiUrl(id: string) {
    return `${getBaseAPIPath()}/project/sync/${id}`;
}

That's a lot of code, but that's not important. What's important is it works, I understand how it works, and I got this by understanding basics.

It's the new engineer in your team

The one who is quite eager to execute but needs some hand holding, and with good direction
can deliver exactly what you want. That's what current LLMs are when it comes to architecture.

You have to be the one boucing ideas, guiding and more importantly verifying, because they'll definitely make mistakes. Here are the ones it made in my case:

When it missed an operation:

Indeed, you are correct. Let's integrate the merging of events into the updated pipeline, ensuring that the merged events and the sequence number are correctly handled in each API call.

After reviewing that it missed a crucial check:

Yes, you are correct. The current implementation does not check if syncing is allowed before proceeding with the API call. This check is crucial, especially in the scenario where the state is set to 'errored' and further syncing is supposed to be halted.

When it just assumed how TS works:

You're correct to question the typecasting of useStore to Observable<StoreState> in the RxJS pipeline.

When it came up with the wrong implementation:

You're correct in your understanding. Placing the tap operator after bufferTime means it will only execute after the specified time interval (n seconds) has passed and the events have been buffered.

Once again, wrong implementation:

Yes, you're correct. Setting the 'unsynced' status after bufferTime would introduce the same issue as discussed earlier. The 'unsynced' status would only be set after the buffering period has elapsed, not immediately when events start arriving.

When it fumbled manual save:

You're absolutely correct. If the manualSave$ observable emits an empty array, and it's merged with the event source observable, there's a risk of syncing that empty array instead of the actual buffered events.

But what it lacks in precision, it makes up for in speed. The chat interaction might look long but the conversation lasted
for barely an hour and by the end of it, I had exactly what I needed.

Here are some more design problems I could rush through quickly:

That user, projects, organisation and resource problem I mentioned before, very real.
I also needed to create a basic file system abstraction, I know how trees work but need help with applying that knowledge to the app I am building.
In the metz runtime I needed a better way to detect connections, I had an idea around a graph based approach but needed some oversight.

Compiler of Ideas

A programming language compiler, takes our intention of what we want to do, and then does all the hard work of
making it a reality.

And that's how I see my interactions with AI for design. It too has "syntax" errors, where if you don't start
with something concrete it'll "crash" and give garbage answers.

Designing a system needs in-depth analysis, but not just of one idea, rather of many ideas.

System design is not just about choosing the right thing, but knowing there are other right things and this thing is the most right for us.

This is where iteration speed matters. The ability to try things out quickly, create a thesis and compare, is crucial to arriving at an informed decision.

And this is exactly were AI shines.

PS: This post appeared first at metz's blog.
We at metz are building a platform where you can just code to create architecture diagrams.

Saving cost with timeout latches in JavaScript

Rahul Ramteke — Mon, 18 Jul 2022 09:11:38 +0000

I maintain this app called:

casset.app

which essentially live streams what song I am listening on spotify and lets users join in and sync with me.
It doesn't really stream audio buffers, just the song title.

And the architecture is pretty straight forward:

Poll spotify api in background
maintain state in memory
push state change to client via websockets

And it works quite well, acceptable delay, no rate limits hit(hard to when it's just my user getting polled).
But every month, I get an invoice of 25$ from Digital Ocean.

That happens because I use their k8s cluster and a load balancer. Which I know is an overkill for something like this.

And that's why I decided to not only move away to something simpler and cheaper but also explore can we only pay for the work that we do?

Architecture Overhaul

Firstly we need to move away from the "always on" mode to "on demand" mode.
Does the tree fall in the forest only when we observe it? In our case, yes it does. Because then we don't have to render/calculate anything unnecessarily.

So we remove the socket layer, and let the client poll our system.

Now we have two polls:

Client polling our system
System polling Spotify

Is it ok if we take the shortcut and couple these two polling? That is, call Spotify API only if client calls our system.

That can work in theory, but with this our rate of API requests to spotify gets coupled with number of clients. One client is ok, maybe 10 even, but 100? 100,000?

That's when we would have problems. Problems like, rate limit and quota exhaustion.
Also, let's say I am listening to a song for a few minutes, does it even make sense to call spotify API 100,000k times to check current track?

With this, we see that coupling those two systems isn't a good idea. So what do we do next?
We need to poll spotify but not unnecessarily, only when we have intent, but we don't want to tightly couple with incoming requests.

Weak coupling?
Let's say we have a timer with us for polling spotify, say 10s. Our system will poll spotify every second for 10s and then stop. Unless a request comes, then the timer gets reset to 10s and the cycle starts again.
If there are 100,000 requests coming to the system, the max a request can do is reset the timer, it doesn't affect the poll rate for spotify, just elongates the process.

It's best described visually here:

Timeout latch / iostreamer | Observable

It's a leaky bucket which leaks at constant rate unless reset by some action. In this example, clicking! Checkout the project here

observablehq.com

Timeout latch

It's straight forward to model this with simple timers. For example, we start a setTimeout and whenever we want to reset, we clear that timeout and start again.

But I didn't want the overhead of creating and deleting timers just for resetting the clock.

So I created a custom scheduler, which ticks every 1ms. On the other hand, we have a latch, essentially just an object with a counter.
The scheduler's job is to decay the latches by decrementing the counters.
In this universe, scheduler creates time using ticks, and latches experience that time using associated counters.

This is fairly simple and straightforward, but as I mentioned before, I wanted to explore can we only pay for whatever actual work we do?

Running a scheduler indefinitely still means we are in an "always on" mode. What's the point of creating time if there are no entities to observe it?

And that's final thing that we needed to do, stop the scheduler if all latches are done, or cancelled.

If something new is added, or an old latch gets reset then start the engine once again.

The code is maintained here and published to npm as well.

iostreamer-X / timeout-latch

A simple timeout latch. Like a reverse leaky bucket.

timeout-latch

A simple timeout latch. Like a reverse leaky bucket.

It's a simple callback based mechanism to get notified when a timeout has occurred and reset that timeout if need arises Checkout the visual demo here

Why not plain timeout?

Plain timeouts might not work that easily. For example, one might start a timeout and manage state around it.

function run() {
    setTimeout(
        () => {
            // your callback       
        },
        3000
    )
}
function reset() { // when you want to reset the timer
    run();
}

But we realise that resetting only works when the timeout has completed its work. If we reset before that then we simply have two timeouts!

To counter this we can clear the previous timer and start again. And it works but has the overhead of cancelling and creating new timers. Also, it's a timer…

View on GitHub

Why is this better?

So we already established that in this case "on demand" is better than "always on".
And in "on demand" mode as well, we wanted to be truly "on demand", that is expend resources only when necessary.

With the current setup, especially with timeout-latch, we are in a state where nothing runs unless necessary, and it halts if the intent to run isn't there anymore.

This property is extremely beneficial if we look at serverless or edge functions.
That's a model where you do pay for whatever you execute.

Even for simple apps on Digital Ocean there's a minimum of 5$ you'd have to pay.
But for a platform that takes the "server" away from you, one can truly embrace the burstiness nature of work.

And this serves as an example where we transformed polling(very continuous) to something that's bursty and "on demand".

And that concludes my cost cutting process, the things you have to do in this economy 😁

Seamlessly offloading typescript compilation

Rahul Ramteke — Tue, 26 Apr 2022 09:11:18 +0000

I picked the problem of my macbook pro being an absolute snail under load, mostly because I wanted to play around and experiment while learning something.

Sure I got my standard VS Code, a terminal, a browser, slack, postman and some other programs running, but there comes a time when I change something in my code and in response the macbook starts preparing for a lift off.

You see I have this typescript project, not too big, but not too small either. I keep a build task running in background which incrementally compiles code whenever I change it.

And here is the usual ram usage:

Looking up and fixing VS Code ram usage is fairly easy, but what about that node process? That’s the one I have no idea how to fix. I mean it compiles stuff and it takes a lot of juice, that’s all I know.

I can’t throw hardware at this problem! Or so I thought. Well I can’t upgrade my RAM or my CPU even for that matter. But, I do have nice PC sitting idle when I work. It has 16G of RAM, and a Ryzen 1600 for CPU. Fun fact, Ryzen 1600 is supposed to be a 6-core processor but somehow AMD manafacturing plants messed up and released a few 1600s with 8 cores. And somehow I got lucky.

The problem

The problem is straightforward, use the PC to compile the code residing on my macbook.

But I have come to depend on nodemon and auto reload on change. And that’s something I don’t want to miss out on. So let’s throw in this constraint as well. The solution should work as good as shown below where nodemon restarts my server as soon as I change a file and save it:

The solution

What if we listened for changes in our filesystem, especially if done to a particular directory, and then we create a zip, send it to PC over network, the PC compiles it, and sends back a zip, we unzip it? Voila, we have got ourselves a working solution! And along with that we have got ourselves a whole lot of lag! This would slow down to the point of defeating the whole purpose.

I am quite convinced that sending code back and forth will be a bottleneck if I come up with something of my own. It’d be better if we lean on something native, I mean this is a problem someone else must have solved. And sure someone had. It’s called NFS(Network File system), and it’s quite old and widely used.

The idea is simple, mount macbook’s disk on PC and then ask PC to compile.

                          +--------------------------+
                          | +--------------+         |
                          | |Compiler      |         |
                          | +-+------------+         |
                          |   |                      |
                          |   |                      |
                          |   |Compile mounted       |
                          |   |project               |
                          |   |                      |
                          |   |                      |
                          |   |                      |
                          +---v---------+            |
                          |             |            |
+---------+               |             |            |
|         +-------------->+ Mounted     |            |
| Macbook |   Mount       | Project     |            |
|         |               |             |            |
+---------+               |             |            |
                          |             |            |
                          +-------------+            |
                          |                          |
                          |                          |
                          |                          |
                          |                          |
                          |                          |
                          |                          |
                          |      PC                  |
                          +--------------------------+

Figure 1

And as the name Network File System suggests, it lets you access remote machines’ disk as if you are accessing it locally. And that’s the beauty of it. Anything I change on my machine, gets transmitted to the remote machine instantly(well ignoring some network latency) and vice versa.

Explaination

Figure 1 sheds some light on this simple architecture. The remote machine has the compiler installed, and to actually compile it, all that needs to be done is compile the directory.

And yes, that directory is a mounted one but that doesn’t matter! It’s a directory nonetheless and the compiler is well versed in talking to it.

Well, ok, sure the compiler can handle a mounted directory but does it help our cause? Absolutely!

The directory is mounted on remote machine, and it has the compiler as well. It’s pretty straightforward for the remote machine to go ahead and provide some resources to the compiler and let it do its job.

Offloading succesfull!

Setup

I tried this solution and first I needed to setup NFS server on my mac. NFS Manager is a nice little free tool that came handy. It’s easy to configure and lets you control a lot of things.

Now that my macbook is mountable, I needed to tell my PC to actually mount it. My PC runs a Ubuntu, so all the commands for PC will be linux specific.

In essence, I need to use the mount command and tell it the IP address of my macbook and the directory it’s sharing.

Zero Conf

It enables any device to find other devices on the network and contact them. So if a device has the zero-conf daemon running, and is brought on to a network, it has the capability to find other zero-conf enabled devices.

But what does it mean to ‘know’ devices on the network? Well a part of the answer is the name of the device, or more specifically the hostname.

mDNS(Multicast DNS) is another service on which Zero Conf relies on and it powers the hostname resolution for connected devices.

Each device on the network has a name it can use, and by default other devices can access it through the hostname: <device-name>.local

Well access is not quite right, just like DNS, mDNS is also a protocol which maps hostnames to IP addressess. And that’s all it does, it resolves the hostname into a valid IP address.

Once you have the IP you can basically do all the networking you want to do.

The beauty is, you don’t have to remeber ip addresses anymore, if your device restarts and your router gives it a different IP address, it simply doesn’t matter. We have a hostname for the device now, and mDNS to resolve it.

It comes by default in macOS, and in fact you can test it right away. <your-laptop's-name>.local is the hostname of your macbook. And if you have a server running(can try running with python -m SimpleHTTPServer), you can access it using that hostname.

Setting up NFS

Let’s continue our NFS journey with this newfound knowledge of .local domains. In my setup, the name of my PC is kaer-morhen and my macbook is named bianco.

Once again:

PC/Remote machine = kaer-morhen

Macbook = bianco

I have created a directory on PC as /mnt/bianco. If a new device comes up it’ll follow the same pattern. Now, mounting is simple, as discussed before, we will have to use the mount command, like this:

sudo mount bianco.local:/System/Volumes/Data/Users/iostreamer /mnt/bianco

This comes with a limitation that one’d have to mount it everytime I boot the PC. To automate it, I followed the steps mentioned here and created an entry in /etc/fstab and let the OS handle it for me. Now all that mounting stuff is done behind the scenes automatically!

Voila! Our macbook will be automatically mounted on PC whenever the PC boots.

Setting up Compiler

For this particular project, I need the typescript compiler. Installing it is as simple as:

npm i -g typescript

To compile the project all one needs to do is run the command tsc in the project directory. This parses your tsconfig.json, picks up source directories, compiles file and puts them in the output directory. But we also need it to watch for changes done to the source directories and then recompile if anything changes. That can be done with compiling in watch mode: tsc -w. This would block your shell and start watching for changes.

Here’s a script I wrote which SSHs into the remote machine, goes to the appropriate directory and starts compilation in watch mode. I named this script rcw(Remote compile watch). It basically picks up the current working directory as string, and replaces folder names such that it matches the mount folder on remote machine:

#!/usr/bin/env zsh

cwd=`pwd`
cwd="${cwd/Users/mnt}" // Syntax for replacing substrings
cwd="${cwd/iostreamer/bianco}"
ssh -tt iostreamer@kaer-morhen.local "cd $cwd && tsc -w"

Let’s try to run it, change the files and see if automatically compiles the changes or not:

Lo and behold, it doesn’t work!

What did we miss? Did we miss something? Let’s find out!

The catch

Why did it not work? We have mounted correctly, it even compiles properly and one can even see the compiled files locally. Then what’s stopping it to react to changes done to files?

Well, to be blunt, it does work. It’s just clunk and slow in my experience(YMMV).

What is inotify?

Well inotify is a Linux API/ Sub system. You can ask this API to tell you whenever a change is done to a directory or its files. When you run tsc -w, i.e. compile in watch mode, that’s exactly what the typescript compiler does. It sets up a watcher, which is notified when source directories change.

In our case, the NFS client running on linux(PC) is the one handling changes done by macbook. And these changes are all immediate, very snappy. You can verify by creating a file locally and then SSHing in to remote machine and using the watch command to see the contents of the file at let’s say an interval of 0.1s. As soon as you change the contents of the file locally, you’d see it getting reflected immediately through the watch command.

The issue seems to be with NFS client and inotify. It’s not that it doesn’t work, but in my experience it’s quite slow, to the extend you can’t depend on it. From what I observed, the content of the file change immediately but the NFS client takes its time to contact inotify. Eventually it does, but who wants to wait?

The solution: Part 2

After Googling for like 5 minutes I came across this interesting project called notify-forwarder. It works on a very simply principle, forward it manually.

It’s a program which works in 2 modes:

watch
- When working in watch mode, it listens for changes locally and transmits those changes to the specified machine.
- When watching, you have to specify the remote ip, as well as the remote directory path.
receive
- When working in receive mode, it simply listens for changes done to the directory.

To be more precise, this program simply tells some change has been done to a file. And when listening for events(receive mode), it simulates an ATTRIB event. This event is fired when an attribute of the file is changed.

And the obvious drawback is that, not all compilers/build systems respect ATTRIB events.

Thankfully, typescript does.

Now, all we have to do is run this program in receive mode on my PC. And then in watch mode locally. For PC, I didn’t want to run it manually, so I created a service, which can be handled by systemd.

Note: I had no prior experience with systemd services and followed this guide to create it.

Script to run in receive mode

notify-forwarder receive

.service file for the script

[Unit]
Description=Notify Forwarder

[Service]
Type=simple
ExecStart=/bin/bash /usr/local/bin/notify_receive

[Install]
WantedBy=multi-user.target

For my macbook, I needed a script which runs notify-forwarder in watch mode, watches the current directory and sends the changes to PC(kaer-morhen). I named it nk (Notify Kaer-Morhen)

Here’s the script for same:

ip=`dscacheutil -q host -a name kaer-morhen.local | grep ip_address | awk -v FS=': ' '{print $2}'`
cwd=`pwd`
cwd="${cwd/Users/mnt}"
cwd="${cwd/iostreamer/bianco}"
notify-forwarder watch -c $ip . $cwd

And this is how it looks when it all comes together:

Conclusion

This was a day’s hack and I got to learn so much! But all of it fades when compared to this one learning. You see, before starting I thought achieving something like this smoothly would need some genius level hacks or code.

But it did not. I just stumbled from one problem to another till I managed to hack together a part elegant, a part messy solution. This is definitely not a genius level code.

Getting it done > Waiting to learn some godsent tools and creating that perfect thing.

Future

This is definitely not a Compiler as a Service product and I doubt it’s going to be for a long time. But that’s not going to stop me from monetising it in my home. My roommate has a similar setup, and I am thinking of an AWS like model where 1 CPU minute = 2 dishes they wash.

Threads in NodeJS: Going beyond eventloop using Rust

Rahul Ramteke — Sat, 09 Apr 2022 20:08:18 +0000

Circumventing the single thread bottleneck

Index:

NodeJS Refresher
- A brief overview of how eventloop works internally
Let's block the main thread
- How a simple code can bring down the performance of NodeJS
A qr generator service
- A realistic example and the results of load testing
How to improve?
- Can we do better than Node?
Rust solution
- Using rust and neon to save the day
Comparison
- It's a number's game
Conclusion
- It's all about choosing the best tool for the job

NodeJS refresher

At this point, we have all heard and read how nodejs is singlethreaded but not really. But just in case, here's a refresher:

NodeJS relies on the concept of event loop. The idea is to ask the os/kernel to do the heavylifting and expect a signal saying 'hey, this is done'.
- Each os has their own thing going on, linux has epoll_wait, osx has kqueue and windows has something weird.
- These kernel api calls are the ones doing the actual job. It kinda looks like this

//pseudocode
while(event=epoll_wait()) {
    if(event.type === 'socket') {
        // do something
        // or in our case, execute the relevant callback
    }
}

NodeJS doesn't have a one size fit all event loop, rather it has a phased setup.
- For example, it checks timers(setTimeout etc) first.
  - Here it's the OS's show again, and it uses epoll or equivalent to know if it needs to execute a callback or not.
- Then we have the microtask queue, which handles promises and nextTicks
- ...And more, checkout this video for full picture
- At the end of the phased setup, it checks if there are still any more events that it needs to handle or wait for. If yes, the loop continues, if not the loop and the program exits.
After getting a signal saying 'hey, this is done', the associated callback that you provided is executed.
- Now mind you, the loop itself is what's single threaded. The tasks that node does in the loop, all on one thread.
- And the associated callback that it needs to run? Well, you guessed it, the same event loop thread.

And now you case see why there might be some confusion around the execution. Afterall, it's singlethreaded but not really.

Also, what happens if the callback you provided is trying to calculate the meaning of life? That's when we have a problem, because now our eventloop isn't going to do anything until the callback function's execution is complete.

That's what we mean by blocking the main thread in NodeJS.

Let's block the main thread

Let's say we have a NodeJS express server with us. And on each request, we calculate a cryptographic hash of given query parameters. And just to stress the system, we do this hashing 500k times and then return the result back.

const express = require('express')
const app = express()
const port = 3000

function getHash(text) {
    let hashedString = text;
    for(const i=0; i<500000; i++) {
        // do fancy hashing
    }
    return hashedString;
}

app.get('/', async (req, res) => {
    const text = req.query?.text;
    const result  = getHash(text);
    res.send({ result });
})

app.listen(port, () => {
  console.log(`App listening on port ${port}`)
})

Based on what we discussed in previous section, we can see how this setup can backfire and undermine the performance of NodeJS. But to show again:

NodeJS starts up, and starts executing our script
It asks OS to tell when the server starts
It asks OS to also tell when that server receives a connection request
And now the grand loop runs in phased manner, checking timer first, then i/o and so on
Since NodeJS still has some events that it's waiting for(server connection requests), the loop doesn't quit
Let's say someone hits our api, then the os tells NodeJS of that event
In the next iteration/tick of the grand phased loop, it checks timers first, finds nothing and then it checks i/o
It finds that there's a request, and promptly starts executing associated callback
Once the execution of callback is finished, the grand phased loop is iterated again and the queues are checked for more connection requests.

Now, our callback isn't very easy breezy, it can take a good amount of time to execute, relatively speaking.

And that will delay the next iteration of the grand phased loop, which will delay knowing if there's a new connection or not. And that's one very good way of losing i/o performance in NodeJS.

If you look at the code, it's quite innocent looking, nothing weird about it. But one nefarious loop or thread blocking operation is all it takes.

A qr generator service

The previous example of hash calculation isn't very realistic. So let's say we have to build a service which can create a qr image of any given text.

This service will have a simple GET api which will take text in query params. After that it will return a base64 string representing the QR version of given text.

Let's use NodeJS and commonly used libraries for this service. Here's how it looks in code:

const QRCode = require('qrcode')
const express = require('express')
const app = express()
const port = 3000

app.get('/', async (req, res) => {
    const text = req.query?.text || 'QR TEST';
    const result  = await QRCode.toDataURL(text);
    res.send({ result });
})

app.listen(port, () => {
  console.log(`App listening on port ${port}`)
})

Voilà! We have what we needed. A very simple script which does what we planned to do. But here's the catch, if you look at the source code of qrcode library, you'll find there are no async calls. It's all done in one synchronous function.

And now our code looks a lot like the 500k hashing one. But how bad can it really be?

To answer that, I setup pm2 for some advanced monitoring and artillery for load testing. Here's how it went:

┌─ Custom Metrics ───────────────────────────────────────────┐┌─ Metadata ────────────────────────────────────────────────────────────┐
│ Used Heap Size                                  23.74 MiB  ││ App Name              index                                           │
│ Heap Usage                                        40.33 %  ││ Namespace             default                                         │
│ Heap Size                                       58.87 MiB  ││ Version               1.0.0                                           │
│ Event Loop Latency p95                            10.78 ms ││ Restarts              0                                               │
│ Event Loop Latency                                3.2 ms   ││ Uptime                2m                                              │
│ Active handles                                       1608  ││ Script path           /home/iostreamer/projects/node-qr-test/index.js │
│ Active requests                                         0  ││ Script args           N/A                                             │
│ HTTP                                       269.86 req/min  ││ Interpreter           node                                            │
│ HTTP P95 Latency                                    16 ms  ││ Interpreter args      N/A                                             │
│ HTTP Mean Latency                                    7 ms  ││ Exec mode             fork                                            │
│                                                            ││ Node.js version       16.13.2                                         │

--------------------------------
Summary report @ 16:49:34(+0530)
--------------------------------

http.codes.200: .............................49994
http.request_rate: ..........................356/sec
http.requests: ..............................49994
http.response_time:
  min: ......................................1
  max: ......................................97
  median: ...................................15
  p95: ......................................29.1
  p99: ......................................47

Some important stats out of this exercise:

event-loop-latency:
p95                     10.78 ms
current                 3.2 ms

http.response_time:
  min: ................ 1 ms
  max: ................ 97 ms
  median: ............. 15 ms
  p95: ................ 29.1 ms
  p99: ................ 47 ms

The response times that we are seeing, a median of 15ms and p95, p99 of ~30ms and ~50ms respectively, seem like a lot. It's a fairly simple service, it makes sense to expect better.

We know that we have a performance bottleneck, and apparently this is how it crops up. But we still don't if this is really bad or not, or if we can do better or not and if so then by how much?

How to improve?

We know the bottleneck is that we only have one thread, and if we block it, we are doomed. We need more threads for this. What if we tried worker_threads?

Introduced in node 10, these are separate threads with their own eventloops, but they share the same node and v8 instance, unlike child processes. This is what makes
them analogous to standard threads in other runtimes.

Well, we probably can use them and it might even work, but I wanted to go all in and have a much leaner solution.

That's why I went with Rust, to get some near native performance.

Architecture

The idea is to use NodeJS for what it's known for, i.e brilliant i/o and async performance, and rust for managing threads. This way we get to have best of both the worlds.

NodeJS has n-api/node-api as a layer which enables FFI(Foreign Function Interface). Essestially, it allows node to call functions running in entirely different runtime, written in some other language.

Here are the steps involved in this new architecture for our service:

NodeJS will still handle the http connection aspect
On a new request, it will call our rust program to create qr
- This will be an async call, where our rust program can be viewed like an os/kernel call
- Like registering a callback for event, except the event is that our rust program is ready with qr base64 string.
Once in rust domain, we will parse and clean our input given by NodeJS process
In rust runtime, we will spawn a new thread
- We will create a qr for given text where
- Once done, we will intimate that we have a result for the event, and pass it back to NodeJS runtime.
Once NodeJS knows there's data for the event, it will execute the registered callback with given data.

The result is that we have simulated qr creation as an os/kernel api which epoll_wait or equivalent can take care of!

This is huge because our NodeJS program is now about handling http requests as fast as it can, without worrying about doing something heavy on its main thread.

Rust solution

We are using neon to help us with creating a Rust binding for NodeJS. They have pretty good docs and example for you to start tinkering with it.

I started with their hello-world example and then used that as a template.

Neon creates a node compatible binary, which our NodeJS program then loads as a library and runs.

Here's the rust code:

use neon::prelude::*;
use image::{DynamicImage, ImageOutputFormat, Luma};
use base64::{encode as b64encode};
use qrcode::QrCode;
use neon::event::Channel;

fn create_qr(
    text: String,
) -> Result<String, String> {
    let width = 128;
    let height = 128;

    if let Ok(qrcode) = QrCode::new(text.as_bytes()) {
        let qrcode_image_buffer = qrcode
            .render::<Luma<u8>>()
            .max_dimensions(width, height)
            .build();

        let qrcode_dynamic_image = DynamicImage::ImageLuma8(qrcode_image_buffer);

        let mut image_bytes: Vec<u8> = Vec::new();

        if let Ok(_v) = qrcode_dynamic_image.write_to(&mut image_bytes, ImageOutputFormat::Png) {
            Ok(b64encode(image_bytes))
        } else {
            Err("Error: Cannot get image bytes".to_string())
        }
    } else {
        Err("Error: Cannot encode this text".to_string())
    }
}


fn create_qr_and_send_back(text: String, callback: Root<JsFunction>, channel: Channel) {
    let result = create_qr(text);

    channel.send(move |mut cx| {
        let callback = callback.into_inner(&mut cx);
        let this = cx.undefined();
        let args = match result {
            Ok(imageString) => {

                // Save the data in a result object.
                let obj = cx.empty_object();
                let str = cx.string(imageString);
                obj.set(&mut cx, "imageString", str)?;
                vec![
                    cx.null().upcast::<JsValue>(),
                    obj.upcast(),
                ]
            }
            Err(err) => {
                let err = cx.string(err.to_string());
                vec![
                    err.upcast::<JsValue>(),
                ]
            }
        };

        callback.call(&mut cx, this, args)?;

        Ok(())
    });
}

fn parse_js_and_get_qr(mut cx: FunctionContext) -> JsResult<JsUndefined> {
    // The types `String`, `Root<JsFunction>`, and `Channel` can all be
    // sent across threads.
    let text = cx.argument::<JsString>(0)?.value(&mut cx);
    let callback = cx.argument::<JsFunction>(1)?.root(&mut cx);
    let channel = cx.channel();

    // Spawn a background thread to complete the execution. The background
    // execution will _not_ block the JavaScript event loop.
    std::thread::spawn(move || {
        // Do the heavy lifting inside the background thread.
        create_qr_and_send_back(text, callback, channel);
    });

    Ok(cx.undefined())
}

#[neon::main]
fn main(mut cx: ModuleContext) -> NeonResult<()> {
    cx.export_function("createQR", parse_js_and_get_qr)?;
    Ok(())
}

Here's the js code which uses it:

const lib= require('.');
const createQR = require('util').promisify(lib.createQR);

const express = require('express')
const app = express()
const port = 3000

app.get('/', async (req, res) => {
    const text = req.query?.text || 'QR TEST';
    const { imageString }  = await createQR(text);
    res.send({ imageString });
})

app.listen(port, () => {
  console.log(`App listening on port ${port}`)
})

And it works! If we run this code, we will get our base64 representation of a qr code.

But is it any good? Does this perform better than our main thread blocking version?

┌─ Custom Metrics ───────────────────────────────────────────┐┌─ Metadata ─────────────────────────────────────────────────────────────────────┐
│ Used Heap Size                                  22.00 MiB  ││ App Name              index                                                    │
│ Heap Usage                                        36.74 %  ││ Namespace             default                                                  │
│ Heap Size                                       59.87 MiB  ││ Version               0.1.0                                                    │
│ Event Loop Latency p95                            2.29 ms  ││ Restarts              0                                                        │
│ Event Loop Latency                                0.17 ms  ││ Uptime                96s                                                      │
│ Active handles                                       1604  ││ Script path           /home/iostreamer/projects/node-rust-hello-world/index.js │
│ Active requests                                         0  ││ Script args           N/A                                                      │
│ HTTP                                       240.11 req/min  ││ Interpreter           node                                                     │
│ HTTP P95 Latency                     9.549999999999955 ms  ││ Interpreter args      N/A                                                      │
│ HTTP Mean Latency                                    1 ms  ││ Exec mode             fork                                                     │
│                                                            ││ Node.js version       16.13.2                                                  │

--------------------------------
Summary report @ 16:55:55(+0530)
--------------------------------

http.codes.200: .............................50005
http.request_rate: ..........................356/sec
http.requests: ..............................50005
http.response_time:
  min: ......................................0
  max: ......................................58
  median: ...................................1
  p95: ......................................12.1
  p99: ......................................22

Important stats:

event-loop-latency:
p95                     2.29 ms
current                 0.17 ms

http.response_time:
  min: ................ 0 ms
  max: ................ 58 ms
  median: ............. 1 ms
  p95: ................ 12.1 ms
  p99: ................ 22 ms

Comparison

HTTP performance: Latency in ms

Eventloop performance: Latency in ms

Conclusion

We see a tremendous performance increase, especially in p95 and p99 cases. We successfully modified our app such that not only is it faster on average, but the users facing hiccups aren't far by a huge margin. This ~2-3x increase in performance tells a lot about where node shines and where it shouldn't be used.

This ability to create native addons has huge implications for JS projects. Imagine you have your entire stack in typescript and all the engineers well versed with TS/JS ecosystem, but you finally hit the limit. Now you can rewrite and retrain, or you can simply create a fast, low surface area library which anyone can plug and play as easily as downloading it from npm.

All in all, it's looking good for NodeJS with projects like neon and languages like Rust. Given that NodeJS democratized server side development, it has been fascinating to see how the pitfalls have been plugged over the years.

We now have typescript to instill confidence and now wasm and ffi backed by reliable, safe and blazing fast languages. It's fair to say, NodeJS now has almost everything for everyone.

Fixing Null Pointer Exceptions with Proxies

Rahul Ramteke — Sat, 15 Jun 2019 02:02:25 +0000

No more Cannot set property of undefined

A proxy, as the name suggests, acts as an interceptor to an object. Every property one accesses, every property one sets, every operation, goes through the proxy.

Here are some articles to get you to speed on proxies:

Now that we have some idea of using proxies, we can try to solve this problem which came up while implementing a multitenant architecture I was working on.

Let’s say we have an empty object x = {} and for some reason we want to do x.y.z = 23; and not just x.y.z we want to do add keys and subkeys, infintely to x. So, x.y.z = 23 and x.y.z.a.i.v.b.n.s = 99 should not throw an error but rather should create the following objects respectively:

x = {
  y: {
    z: 23
  }
}

and

Problem is, if we were to do x.y.z = 23 on x = {}, we’d be trying to access property z assuming y exists on x, but actually it doesn’t, all we have in x is {}.

But what if our x was smart, so much so, that if we tried to access a property which doesn’t exist, it will create that property on the fly and our code won’t even know!

What if we intercepted every property access call on x, checked if the property actually exists, if it doesn’t, initialize it with empty object and then go about our usual business?!

So, doing x.y.z = 23 will first create y on x and then will set z

And this is where proxies shine, we could do this magic with a simple proxy like this:

function getUndefinedHandlerProxy() {
    const undefinedHandlerProxy = new Proxy({}, {
        get(target, key) {
            if (target[key] === undefined) {
                target[key] = {};
            }
            return target[key];
        },
    });
    return undefinedHandlerProxy;
}

The first argument to new Proxy($1, $2) is basically x, or the object we want a layer on, the object we want to intercept. The second argument is our handler, a collection of traps and interceptors.

Here, we decided to intercept property access calls using get trap. As we planned before, we will check if the property(or key in this case) exists on our object(called target here). This is done using the line if (target[key] === undefined) and if it doesn’t, we return an empty object, like we planned.

Let’s see how this helps us solve our problem:

const x = getUndefinedHandlerProxy();
x.y.z = 23;
console.log(x.y.z);

Our x is a proxy now, x.y.z was performed on our proxy, which caught us when we tried to access y. It checked if y existed, and since it did not, x.y returned an empty object, after which, z was simply set on that returned empty object!

One little problem though, what if we had x = getUndefinedHandlerProxy(); and did x.y.z.a = 78? Since x is proxy, y would be automatically created as a blank object. But since y is simple plebian, accessesing y.z.a would be problematic, as z doesn’t actually exist, so, we’d be setting aon undefined, and now we are back to our original problem. We need y to be as smart as x and not a simple object, same goes for all the sub properties that get created. What if y was actually a proxy, just like x?

We could do that by changing our proxy a bit, rather than doing target[key] = {}, we could dotarget[key] = getUndefinedHandlerProxy().

And now our final code would be:

function getUndefinedHandlerProxy() {
    const undefinedHandlerProxy = new Proxy({}, {
        get(target, key) {
            if (target[key] === undefined) {
                target[key] = getUndefinedHandlerProxy({});
            }
            return target[key];
        },
    });
    return undefinedHandlerProxy;
}

And this, handles everything.

One last caveat though, our proxy would always create an object when we try to access a property and it doesn’t exist. Now, what if we wanted to check if a property exists or not, which is a very common scenario in day to day javascript.

Doing:

if(ourSpecialProxy.a && ourSpecialProxy.a.x) {
  // Some special code
}

would not work as intended, the condition will always return true.

One way to avoid this, is not checking property existence directly, we need to check if the property is an actual object/string/number. Or if the property is anything but a proxy.

For this, we have:

function getActualValueFromUndefinedhandlerProxy(target, key) {
    if (util.types.isProxy(target[key])) {
        return undefined;
    }
    return target[key];
}

It’s a simple function which takes a target and key, if targey[key] is proxy, then the whole thing is a facade, it returns undefined. If not, then target[key] is returned.

And that’s it, we achieved what we set out to.

No more Cannot set property of undefined!