Forem: about14sheep

Understanding the Monotonic Stack To UP Your Data Structure Game

about14sheep — Mon, 08 Jan 2024 23:50:36 +0000

The Monotonic Stack be very difficult to understand, and even hard to actually utilize!

I wrote up a solution over on leetcode that solves the problem 1475. Final Prices With a Special Discount in a Shop.

I am testing out using DEV to get feedback on solutions I write up (over on leetcode), so please check it out!

If this helps feel free to let me know either here or there! If you have any suggestions, or see things I can make more clear, don't hesitate to let me know!

ESLint Understand By Doing Part 1: Abstract Syntax Trees

about14sheep — Thu, 30 Mar 2023 02:39:34 +0000

Like everything in software development, its best to learn by doing. In this blog series we will go over what is happening, under the hood, when you lint your JavaScript files by writing our very own custom eslint rule!

Setting the stage

Like many I am always searching for answers to life's greatest questions. Also like many, I can narrow these questions down to the three most elusive:

What happens when we die?
Do aliens exist?
How does ESLint work?

Due to recent advances in modern sciences we can finally begin to understand the most important of these questions: How does ESLint work?

TL:DR

I have no idea what I am talking about.

Abstract Syntax Trees

When you run ESLint, the first thing it does is create an Abstract Syntax Tree (AST) of your code. Basically an AST is a description of your codes syntax. A tool can use this description to evaluate your codes structure.

ESLint is not the only tool that uses ASTs. The TypeScript parser creates an AST in its own format to check for incomplete code, type checking, and if you have used the any type implicitly (hehe).

For this blog we will focus on how ESLint uses an AST to understand your code.

Learn by doing

The best way to understand what an AST is.. is to simply play with them over on AST Explorer. I like to switch the right side panel to json as I find it easier to follow, but you do you.

You can follow along with this example:

// Simply setting a variable `foo` to a string 'bar'
const foo = 'bar';

ESLint's AST format, ESTree, would represent this line of code as:

{
  "type": "Program",
  "start": 0,
  "end": 16,
  "body": [
    {
      "type": "VariableDeclaration",
      "start": 0,
      "end": 16,
      "declarations": [
        {
          "type": "VariableDeclarator",
          "start": 6,
          "end": 15,
          "id": {
            "type": "Identifier",
            "start": 6,
            "end": 9,
            "name": "foo"
          },
          "init": {
            "type": "Identifier",
            "start": 12,
            "end": 15,
            "name": "bar"
          }
        }
      ],
      "kind": "const"
    }
  ],
  "sourceType": "module"
}

You can gather a lot of information from this. To help we can break it down, then build it back up while we describe whats going on.

The Root

{
  "type": "Program",
  "start": 0,
  "end": 16,
  "body": [
    // Not important right now
  ],
  "sourceType": "module"
}

The root of this object has a type of 'Program'. This is standard and encapsulates the the rest of the program code. It shows what byte the program code starts and ends, in the file, as well as describes the type of structure your file has, like whether it is a commonJS file or a ES module file.

The Body

{
  "type": "Program",
  "start": 0,
  "end": 16,
  "body": [
    {
      "type": "VariableDeclaration",
      "start": 0,
      "end": 16,
      "declarations": [
        // Not important right now
      ],
      "kind": "const"
    }
  ],
  "sourceType": "module"
}

In the body of the program you have an array of objects. This is where everything you write lives. Since we are only declaring a variable in our program, there is only one object whos type is of "VariableDeclaration". If we were to add a function to the file, a new object would pop up in the body array (specifically it would be a "FunctionDeclaration" object).

Looking back at the "VariableDeclaration" object you will notice it has the familiar start and end, as well the kind of variable it is (its a constant).

The Meat

{
  "type": "Program",
  "start": 0,
  "end": 16,
  "body": [
    {
      "type": "VariableDeclaration",
      "start": 0,
      "end": 16,
      "declarations": [
        {
          "type": "VariableDeclarator",
          "start": 6,
          "end": 15,
          "id": {
            "type": "Identifier",
            "start": 6,
            "end": 9,
            "name": "foo"
          },
          "init": {
            "type": "Identifier",
            "start": 12,
            "end": 15,
            "name": "bar"
          }
        }
      ],
      "kind": "const"
    }
  ],
  "sourceType": "module"
}

Now to bring it all back by adding the meat, the "declarations" property. Inside of this property is where the meat is. The id object is where the name of our variable is housed. Likewise, the init property is where the value our variable was initialized too is house. Each of these nested objects also has a type property and their own start and end properties. Can you see a pattern?

That's all an AST is. It simply describes what and where the individual pieces of your code are. This is the map ESLint uses to run its rules off of.

In part 2 of this series we will show how ESLint does this by creating rules of our own.

Type Assertions, Trucks, and Aliens

about14sheep — Sun, 04 Dec 2022 14:46:23 +0000

Buckle up, yo.

I'm listening to R&B right now

It's extremely early where I am, and I couldn't sleep. So like any logical person would be doing, I was reading through the TypeScript Handbook. As I am reading the part about type assertions (the as keyword); I couldn't help but think back to a StackOverflow answer I wrote about multiple type assertions. Don't get me wrong, I like my answer. However, it's just that its... well its...

While laying there I had an epiphany, the likes of which would make every Ancient Astronaut Theorist proud. I could spice up the explanation of type assertions (and make it more understandable) if I added aliens.

What even is TypeScript?

If you don't know what TypeScript is, you should. That's all I have to say about that.

Type Assertions

Now that it's just the TS nerds; lets talk about Type Assertions. In short (because obviously I am going to explain it WAY cooler further down), a type assertion is you telling the compiler whats up. The compiler isn't the great Giorgio A. Tsoukalos, sometimes you just know better. Type Assertions allow you to tell the compiler that a particular type will behave like another (i.e. have the same shape). I won't bog us down with code yet, and the handbook has a great example so i'll just steal that:

For example, if you’re using document.getElementById, TypeScript only knows that this will return some kind of HTMLElement, but you might know that your page will always have an HTMLCanvasElement with a given ID.

In this situation, you can use a type assertion to specify a more specific type:
const myCanvas = document.getElementById("main_canvas") as HTMLCanvasElement;

That's all well and good, pretty easy to understand. However, there are obvious limitations to this. What if I tried to tell the stoopid computer machine this:

const myVar = 42 as string; // Error, sending nuclear launch codes to Russia!

TypeScript will throw a fit, and its correct to do so. The compiler is smart enough to know there is no possible way this number 42 can behave like a string. Instead of me grinding this down, you go try it in plain old JavaScript. Try and read the .length property of 42 and see if you don't cause World War 3.

¿¿¿Where Aliens???

If the compiler is so smart, why does asserting 42 to the type any (or unknown) allow me to then assert it as a string??? Am I bamboozling the compiler???

const myVar = 42 as any as string;

Can I now magically call the .length property of 42 and not cause World War 3?

Slow down on the questions, it's early and the coffee is still brewing.

The answer to 42.length is still no. That will NEVER be a thing. Give it up already, kid.

However, you are bamboozling the compiler. When you cast the multiple type assertion spell, it makes the compiler immediately believe you (there is not even a saving roll). It throws its little compiler hands up and says "ok dungeon master, you know best" and goes about its day.

Why you ask, again? That's where the aliens come in.

No more memes, just facts

Now that it's really just the TS nerds; let's start with two data types, one is a truck:

interface Truck {
  color: string;
  numDoors: number;
  drive: () => void; // truck method to drive, cause all trucks drive
}

the other is a plane:

interface Plane {
  color: string;
  numPropellers: number;
  fly: () => void; // plane method to fly, cause all planes fly
}

Lets just focus on the truck for now. If I were to show you this truck (const truck: Truck) you would be able to assume it does truck things. You would expect a number of doors to access it, its color, and other truck things. If I told you to go and drive the truck (truck.drive()) you would see no problem thinking you could hop in and drive it.

What if I told you to fly (truck.fly()) the truck? You would throw a syntax error and remind me that it is physically impossible for trucks to fly; that's what planes do.

Even if I said, think of this truck as a plane:

const truck: Truck = <insert truck object> as Plane

You would probably slap me and leave the room. You can not be fooled so easily, you are a smart compiler!

Now what if I told you it was an alien truck?

const truck: Truck = <insert truck object> as any

Well now it could be anything (the any type you might say?). You have no idea what them aliens be doing these days. It is perfectly plausible that an alien truck can fly!

const truck: Truck = <insert truck object> as any as Plane

You would have no problem getting in the truck and attempting to make it fly. You do so truck.fly(), and the truck explodes.

That's what multiple Type Assertions are doing (and the dangers!). By broadening the type to any anything is possible. So its then perfectly valid (syntactically) to narrow an any type down to a string in our example:

const myVar = 42 as any as string;

BUT WHAT ABOUT `42.length`??

You would be correct in worrying about this. Since the compiler does not care, you could run this code. In code the "alien" veil will come off revealing a regular old truck trying to fly, when it can't. Then the program crashes just as the truck would.

However, there is a correct way to do this. Instead of going to aliens for a flying truck, you could just invent it like the genius you are!

In TypeScript you can extend interfaces to add functionality to them, so to make your special truck you would simply:

interface FlyingTruck extends Truck {
  // You will get all the properties of the Truck type
  fly: () => void // Plus the ability to fly!
}

And Presto! You have now invented the flying truck. When you then instantiate a FlyingTruck object you will be able to fly that bad boy without any issue. The compiler is happy, the JavaScript interpreter is happy, Giorgio A. Tsoukalos is happy, and you know what? I'm happy.

Felt clever, might delete later

If you liked this article (or just love TypeScript as much as I do) let's talk about it! In the comments, though, to drive engagement on the post ;)

Make your own Online Identity Mapping site

about14sheep — Sat, 16 Apr 2022 00:00:06 +0000

Online Identity Mapping

Making a developer website for yourself takes a certain type of person. I am one of those that hate it, but I like the idea of having a centralized place for all my links. So, I made an Online Identity Mapping!

This simple website does exactly what it says on the tin, it takes all the profiles (or at least the ones I want to share) and links them to my name visually. I can then send one link, and know they have everything they need to sleuth.

You can check mine out here

How to make an Online Identity Mapping

I have kept everything very simple. Most of the magic comes from main.css. It is in "card" style on large screens then moves to vertical on smaller ones.

Setting it up

To start, simply fork or clone this repo onto your machine. From there you can rename it whatever you want by running this in your terminal:

// replace <whatever> with what you want to call it.
mv about14sheep.github.io <whatever>
// example - rename this project to myproject
mv about14sheep.github.io myproject

Now you can go into your new project to do the rest with cd myproject replacing myproject with whatever you choose to name it.

Now, you don't want all my git history so you can remove it with:

rm -r .git

You will also want to remove my CNAME file, or updating it if you have your own domain:

rm CNAME

Now that you have a clean slate, lets get git back (you will need it later):

git init

Adding your details

To keep it simple, I have added a template html file. This file is an exact copy of the index.html file, but has spaces for you to insert your details. What you are looking for is big { PUT YOUR _____ HERE } where _____ is a title or image path etc.. Don't forget to also remove the brackets { }!

If you get stuck, you can peep into index.html to compare!

Once you have added all of your information delete my index.html:

rm index.html

Then rename the template file to index.html:

mv template.html index.html

For an example, to change the websites title you would look for this:

<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link rel="stylesheet" href="./main.css">
    <title>{PUT YOUR TITLE HERE}</title>
</head>

And if you wanted to change this to 'John Smith | Terminator' it would look like:

<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link rel="stylesheet" href="./main.css">
    <title>John Smith | Terminator</title> // remove the {brackets}!
</head>

Images and path to images

You can download the image you want for your link, then save them into the images folder. From there the path becomes ./images/your-image.png. An example being:

<img src="./images/stack-overflow.svg" width="48" height="48">

Changing the pipe color

You can change the pipe accent color directly in the html. If you look for the <p> element where your name is you will see a <span> element:

<span style="color: gray; border-left: solid; border-color: #F47F24; padding-left: 5px;">{YOUR ONLINE HANDLE HERE}</span>

Change the border-color to whatever you like

Make it yours

This is a simple html file with not a lot going on. Have fun with it, or keep it the way it is. Its up to you! If you do, come back and leave a link in the comments so I can see too!

Hosting your Online Identity Mapping

To host this site in all its glory I went the easy route. I turned it into a github pages site. This will be pretty simple from where you're at now and that link should explain everything else.

If you run into any issue let me know here and we can figure it out!

For Recursion: See learnrecursion.com!

about14sheep — Sun, 13 Feb 2022 22:22:32 +0000

What are you waiting for?

learnrecursion.com

Firing on all cylinders (Part 2): Understanding Hidden Classes to optimize your JavaScript code

about14sheep — Mon, 17 Jan 2022 19:40:56 +0000

In the first part of this series we went over the differences between dynamic and non-dynamic languages. We also went over the difference between how the two approach object storage lookup. We discussed the meaning of offset, the displacement integer in memory between an object and its properties. We then looked into how JavaScript interpreters combine all of that through the use of hash tables.

We left on a cliff hanger. Realizing that the use of hash tables is inefficient, we hinted at the way v8 mitigates this: Hidden Classes.

In part 2 of this series we learn what hidden classes are, how they work, and how the v8 JavaScript interpreter handles object storage look up efficiently.

Along the way I will stop at where, I think, the best understanding of the one-liners (mentioned in part 1) can come from.

Before we begin

Although the concepts mentioned here may not be required to get value from this post. If you are confused with the term offset, how hash tables work, or how JavaScript interpreters handle object storage lookup; I encourage you to go back and read part 1 of this series.

I have always felt that in order to understand a solution you must first understand the problem the solution solves.

The Rise of Hidden Classes

Hidden Classes are based on the same principles behind the fixed offset mapping in non-dynamic languages (see part 1). The difference is that they are created at runtime, but the outcome is the same. Hidden Classes allow the v8 interpreter to optimize property access time on objects. Hidden Classes are created for each and every object in your program.

We will go back to our example from part one of the series, the employee constructor function:

// Define a simple constructor function for an employee
const employee = function(salary, position) {
  this.salary = salary;
  this.position = position;
};

When the v8 interpreter reads this code, it first creates a pointer to a location in memory where the call signature for the employee function is (this 'shell' does not include the properties as we learned in part 1). So you end up with your first hidden class (we can call this HC0):

Now, when the interpreter reads the next line (this.salary) it creates a new hidden class for employee that includes the offset value for the property this.salary. It then updates the pointer to now point to this new hidden class. Also, it adds a transition from the first hidden class (HC0) to the new hidden class (HC1):

Nest, just like previously, when the interpreter reads the next line (this.position) it creates a new hidden class (and updates the pointer) for employee that includes the offset value for the property this.position along with the already added offset value for the property this.salary. It then, also just like previously, adds a new transition from (HC1) to (HC2):

All of these together in one big happy Harry Potter family tree looks like this:

In this image you can see the final state of the hidden classes and transitions that make up the employee constructor function.

What it all means

The transitions between the hidden classes are important. They allow for hidden classes to be shared among similar objects. What this means is that if two objects share a hidden class and you add a new property to both of them, transitions ensure that both of the objects will have the same hidden class.

This is important because being able to share hidden classes between object is what removes that need to have a hash map with each instance. Instead you have one hidden class, accessed by one quick lookup, shared among all objects of the employee type.

Now here's the catch...

The order in which you add dynamic properties to an object matters. Changing this order between two similar objects creates two different hidden classes, omitting the optimization we just discussed!

More on the catch

Let's look at what we just discussed in code. We will create two employee objects and dynamically add some properties to both of them, but we will do it out of order:

// Instantiate the two employees
const salesEmployee = new employee(50000, 'sales');
const ceoEmployee = new employee(1000000, 'ceo');

// add two new properties to salesEmployee
salesEmployee.payDay = 'Saturday';
salesEmployee.phoneNumber = 8675309;

// add the same two properties to ceoEmployee but in a different order
ceoEmployee.phoneNumber = 9087654;
ceoEmployee.payDay = 'Monday'

This looks the same, after this is ran you have two employee's with the same structure, all conforming to the employee constructor function shape. Since the shape of the objects seems identical it seems logical to assume they will share the same hidden class and all the optimization that comes with it... right?

Nope, as it turns out the v8 interpreter will create two separate hidden classes. One for each, as the offset for the two dynamically added properties will be different. To better explain this i'll use a food analogy.

Same, Same; but Different

Imagine you are cooking a roast. There are many possible ways to cook a roast, however we will limit this discussion to just two. You might use a crock-pot and let it simmer all day pulling it out at the end to flash sear the edges. You might first sear the edges before you leave it to simmer. In both of these scenarios the ingredients are the same, however the technique is different. Both of them result in a delicious dinner, but both of them has their own distinct recipe.

This is how optimizing hidden classes works in v8. The order in which you dynamically add properties to an object matters. Either way, it is valid JavaScript just like both are valid roasts. Also just like the roasts; although the outcome is the same, the recipe is different. You have to memorize the two different techniques in cooking, so too does the v8 interpreter have to store (memorize) the two different objects and the offset of their property values.

TL;DR

When you use TypeScript, you are required to do this. The TypeScript compiler will throw an error if you try and add a property to an object dynamically. This is one of the many reasons we love TypeScript. You could almost say that the TL;DR for this post is "use TypeScript".

Final thoughts

With a better understanding of hidden classes and the catch with how you apply properties dynamically, I think this one-liner might make more sense:

always add dynamic properties to an instantiation of a class (object) in the same order

I had originally thought to add inline caching to this part of the series, however this post is already a long one. No worries though, we can just do a part 3!

Thank you for reading and if you have any questions don't hesitate to leave a comment.

Firing on all cylinders(Part 1): Understanding object value lookup in JavaScript interpreters and the rise of hidden classes

about14sheep — Sun, 16 Jan 2022 23:23:07 +0000

There is a lot of information around writing performant JavaScript and optimizing your code for the v8 engine. When you are reading through this information you will see a lot of phrases like inline caching, hidden classes, and memory offset; but what does it all mean? You'll come across quick one-liners about "always add properties to a particular instantiation of a class in the same order" or, better yet, "assign all of the objects properties in the constructor". You try to dive into the documentation only to see branching charts with a million tiny words in it that look more like Harry Potter's family tree. At the end of all this, you end up just trying to commit those one liners to memory without fully understanding why.

In this series, I am going to attempt to explain these concepts in a way where we don't need pages of flow charts, and with approachable examples (not just look at this code).

What to expect

We will start by going over the difference between dynamic and non-dynamic languages (mostly pertaining to how they store objects in memory). Then, in part 2, we will dive into the v8 engine and the methods it uses to efficiently handle the concepts we discuss in part 1. Also, in part 2, I will describe the common pitfalls and ways you can increase the performance of your code. However, you can't make the code better without first understanding the why.

Before we begin

Although I am going to try and explain these concepts in an approachable way, these are not easy concepts to grasp. Most developers can go their entire career without digging into the minute details of how a particular JavaScript engine accesses objects in memory.

Modern JavaScript Interpreters, like v8, are amazing tech and mostly handle all of this for you. Furthermore, with TypeScript, you have a compiler that can help keep you from making a lot of the common mistakes that can lead to a decrease in performance. However, taking the time to try and understand what is happening under the hood can go a long way.

The dynamic language

JavaScript is a dynamic programming language. This means that you can add, remove, and change (the type) property values of objects after they are initialized. Let's look at the following code:

// Define a simple constructor function for an employee
const employee = function(salary, position) {
  this.salary = salary;
  this.position = position;
};

// Instantiate a new employee
const newHire = new employee(50000, 'sales');

// Dynamically add the employee's desired pay day
newHire.payDay = 'Saturday';

After the employee object is created, their preferred payDay is added dynamically. This is all perfectly valid JavaScript. It will run just fine and the newly hired employee will get paid every Saturday.

The difference between a non-dynamic programming language (meaning all of an objects properties are fixed before compilation) is that new properties cannot be added or removed at runtime. The benefit to a language being non-dynamic is that the values of these properties (or pointers) can be stored in memory with a fixed offset (an integer indicating the displacement between the beginning of an object in memory and any given property). This offset can be easily determined based on the properties type.

Skrrtttt... offset?!?! displacement?!?! You said this would be easy to follow and approachable!

You're right, this is why I decided to do this blog in two parts.

Memory storage displacement (offset)

The easiest way to explain this is with a simple data structure like an array:

const array = ['value1', 'value2', 'value3', 'value4'];

We know we can access a value in that array using its index:

array[2]; // this will get us the item at index 2 ('value3')

If the first value ('value1') is at memory position 0, moving two places to the right will give you ('value3'). So 'value3' has an offset of 2 from the start of where the array is stored in memory.

This is simple enough for an array, however not all objects are stored in memory sequentially like an array is. With more complex objects, like the employee function above, you can't be sure where the object, and its properties, will be stored. Thus making it harder to determine the offset between the objects 'shell' (to keep it simple) and its properties. You could have the 'shell' of the object (function employee() {}) at position 0, then its property this.salary at position 6 with other objects in between.

Back to dynamic vs. non-dynamic

In order to keep up with these offset values, non-dynamic languages (i'll use Java in this case) create a fixed object layout. This layout (or mapping) cannot be changed (as in changing the type), added too, or removed from at runtime. The offset is written in stone, making it easy (usually one instruction) to grab any property value of a given object.

Since you can add, remove, and even change a properties type in JavaScript at runtime, the interpreter has to allocate a new space in memory and then add a new mapping (offset value) back to the object for every change. It will then have to go back through and clean up the memory by a process called garbage collection. In order to keep up with these changes, JavaScript interpreters needed a data structure that can change at runtime.

Hash Tables

Instead of using a fixed object layout, like in the non-dynamic Java, most JavaScript interpreters use a dictionary like structure (based on a hash function) to store the objects property values in memory. This is often referred to as a hash table.

A hash table, simply put, is a collection of key/value pairs. A over-simplified version of what the employee object might look like in a hash table would be keys mapped to the values of where the employee object starts in memory, along with the offset value of each of it properties. We can attempt to replicate this using a Plain Old JavaScript Object:

const hashTable = {
  objects: {
    employee: {
      employeeInstance1: {
        memoryStart: 0,
        properties: {
          salary_offset: 4,
          position_offset: 5
        }
      },
      employeeInstance2: {
        memoryStart: 12,
        properties: {
          salary_offset: 8,
          position_offset: 12
        }
      }
    }
  } 
}

If you wanted to add a property to the table (after the value is added in memory), you can simply update that objects 'bucket' in the table.

Adding payDay to employee 1 would change the table to this:

const hashTable = {
  objects: {
    employee: {
      employeeInstance1: {
        memoryStart: 0,
        properties: {
          payDay_offset: 19, // added property and offset
          salary_offset: 4,
          position_offset: 5
        }
      },
      employeeInstance2: {
        memoryStart: 12,
        properties: {
          salary_offset: 8,
          position_offset: 12
        }
      }
    }
  } 
}

By doing this, the interpreter is able to add properties to the object anywhere in memory, because it has a dictionary (the hash table) keeping track of where the properties are located and what object they belong too.

The downside of this is that grabbing the properties value from a hash table is more computationally expensive (more instructions) than the fixed object layout of a non-dynamic language. Instead of having a direct one-to-one mapping of object/property to the offset of where the value is located, the interpreter must search through the hash table for the employee instance, grab the correct property, then use the offset value to finally get the property's value from memory!

This is an extremely over-simplified explanation of how object memory storage hash tables work, but I think it helps in understanding at a very low level (and we are trying to keep this approachable). If you wanted to learn more you can start here and I have added a very nice blog at the bottom of this post. Just know it is extremely inefficient compared to what is coming next.

Firing on all cylinders

Since using hash tables to get property values is so inefficient, the JavaScript engine NodeJS uses, 'v8', takes a different approach. This approach is built around using Hidden Classes and made faster by Inline Caching.

OK! Now that the gritty stuff is out of the way. In part 2 of this series we will dive into hidden classes and inline caching. Once you better understand the concepts, it well help you understand those one-line suggestions, mentioned everywhere, that can make your JavaScript code more performant.

Streaming files from AWS S3 using NodeJS Stream API with Typescript

about14sheep — Sun, 02 Jan 2022 20:35:31 +0000

Checkout the new npm package!

This simple solution just got easier! I also added some upgrades like the ability to adjusted the size of the range mid stream! Now you can speed up and slow down at will!

https://www.npmjs.com/package/s3-readstream

You can use this package as a drop in replacement for AWS.S3.getObject().createReadStream()!

Original Blog

AWS s3 SDK and NodeJS read/write streams makes it easy to download files from an AWS bucket. However, what if you wanted to stream the files instead?

Before we begin

I am assuming you have used AWS s3 SDK to download files successfully and are now wanting to convert that functionality to a proper stream. As such, I will omit the AWS implementation and instead show a simple example of how, and where, to instantiate this "smart stream" class.

I am also assuming you have a (basic) understanding of NodeJS and NodeJS read/write streams.

The pipe dream

I know that pun was bad, but it's the only one in the article so work with me.

The first solution you will probably come across when implementing your stream (and why I decided to write this article) is to simply take the read stream created off your S3 instance and plug that guy where you need it. Boom streaming!

...not so fast.

There is a timeout on connections to an AWS s3 instance set to 120000ms (2 minutes). Unless you have very small files, this just won't cut it for streaming.

One option is to simply raise that timeout, but then how much should you raise it? Since the timeout is for the total time a connection can last; you would have to either make the timeout some ridiculous amount, or guess how long it will take to stream the file and update the timeout accordingly. This is also not taking into account the stream closing due to HTTP(S)'s own timeout reasons as well.

Bite by Byte

I'm sorry. It was right there... I had too!

Timeouts are not the only things that can cause you problems, there's latency too. You can't be certain that your stream isn't going to slow to a crawl in the middle of it, and everyone hates waiting for the buffer (if you should so choose to stream video). Although this problem can't be solved outright, you can make it a lot easier on yourself.

Instead of just connecting a hose and feeding the beast, you can use a "smart stream" that fetches a range of data in a single request. Grabbing data as you need it can help you avoid latency, while also keeping you away from those nasty timeouts.

Smart Streaming

The idea is to create a stream that uses the power of AWS s3
ability to grab a range of data with a single request. We can then grab another range of data with a new request and so on. This stream will pause when its buffer is full, only requesting new data on an as needed basis. This allows us to take all the time we need to process the data (or pause the video, in the middle of it, to go to the bathroom). When the process is done (and hands are washed), it picks right back up where it left off and the show goes on.

Put your darkest shades on, you're in!

Instead of making guesses and fighting random bugs, we can make use of the NodeJS Stream API and create our very own custom readable stream.

We will start by creating the "smart stream" class:

import {Readable, ReadableOptions} from 'stream';
import type {S3} from 'aws-sdk';

export class SmartStream extends Readable {
    _currentCursorPosition = 0; // Holds the current starting position for our range queries
    _s3DataRange = 64 * 1024; // Amount of bytes to grab
    _maxContentLength: number; // Total number of bites in the file
    _s3: S3; // AWS.S3 instance
    _s3StreamParams: S3.GetObjectRequest; // Parameters passed into s3.getObject method

    constructor(
        parameters: S3.GetObjectRequest,
        s3: S3,
        maxLength: number,
        // You can pass any ReadableStream options to the NodeJS Readable super class here
        // For this example we wont use this, however I left it in to be more robust
        nodeReadableStreamOptions?: ReadableOptions
    ) {
        super(nodeReadableStreamOptions);
        this._maxContentLength = maxLength;
        this._s3 = s3;
        this._s3StreamParams = parameters;
    }

    _read() {
        if (this._currentCursorPosition > this._maxContentLength) {
            // If the current position is greater than the amount of bytes in the file
            // We push null into the buffer, NodeJS ReadableStream will see this as the end of file (EOF) and emit the 'end' event
            this.push(null);
        } else {
            // Calculate the range of bytes we want to grab
            const range = this._currentCursorPosition + this._s3DataRange;
            // If the range is greater than the total number of bytes in the file
            // We adjust the range to grab the remaining bytes of data
            const adjustedRange = range < this._maxContentLength ? range : this._maxContentLength;
            // Set the Range property on our s3 stream parameters
            this._s3StreamParams.Range = `bytes=${this._currentCursorPosition}-${adjustedRange}`;
            // Update the current range beginning for the next go 
            this._currentCursorPosition = adjustedRange + 1;
            // Grab the range of bytes from the file
            this._s3.getObject(this._s3StreamParams, (error, data) => {
                if (error) {
                    // If we encounter an error grabbing the bytes
                    // We destroy the stream, NodeJS ReadableStream will emit the 'error' event
                    this.destroy(error);
                } else {
                    // We push the data into the stream buffer
                    this.push(data.Body);
                }
            });
        }
    }
}

Let's break this down a bit

We are extending the Readable class from the NodeJS Stream API to add some functionality needed to implement our "smart stream". I have placed underscores (_) before some of the properties to separate our custom implementation from functionality we get, right out of the box, from the Readable super class.

The Readable class has a buffer that we can push data in too. Once this buffer is full, we stop requesting more data from our AWS s3 instance and instead push the data to another stream (or where ever we want the data to go). When we have room in the buffer, we make another request to grab a range of bites. We repeat this until the entire file is read.

The beauty of this simple implementation is that you have access to all of the event listeners and functionality you would expect from a NodeJS readStream. You can even pipe this stream into 'gzip' and stream zipped files!

Now that we have the SmartStream class coded, we are ready to wire it into a program.

Implementing with AWS S3

For this next part, as I am assuming you understand the AWS s3 SDK, I am simply going to offer an example of how to establish the stream.

import {SmartStream} from <Path to SmartStream file>;

export async function createAWSStream(): Promise<SmartStream> {
    return new Promise((resolve, reject) => {
        const bucketParams = {
            Bucket: <Your Bucket>,
            Key: <Your Key>
        }

        try {
            const s3 = resolveS3Instance();

            s3.headObject(bucketParams, (error, data) => {
                if (error) {
                    throw error;
                }
                // After getting the data we want from the call to s3.headObject
                // We have everything we need to instantiate our SmartStream class
                // If you want to pass ReadableOptions to the Readable class, you pass the object as the fourth parameter
                const stream = new SmartStream(bucketParams, s3, data.ContentLength);

                resolve(stream);
            });
        } catch (error) {
            reject(error)
        }
    });
}

You can check this out in an HD video streaming app on my github!

The most simple HD video streaming app

Thank you for reading! If you liked this blog let me know in the comments below!

Forem: about14sheep

Understanding the Monotonic Stack To UP Your Data Structure Game

ESLint Understand By Doing Part 1: Abstract Syntax Trees

Setting the stage

TL:DR

Abstract Syntax Trees

Learn by doing

The Root

The Body

The Meat

Type Assertions, Trucks, and Aliens

I'm listening to R&B right now

What even is TypeScript?

Type Assertions

¿¿¿Where Aliens???

No more memes, just facts

BUT WHAT ABOUT 42.length??

Felt clever, might delete later

Make your own Online Identity Mapping site

Online Identity Mapping

How to make an Online Identity Mapping

Setting it up

Adding your details

Images and path to images

Changing the pipe color

Make it yours

Hosting your Online Identity Mapping

For Recursion: See learnrecursion.com!

Firing on all cylinders (Part 2): Understanding Hidden Classes to optimize your JavaScript code

Before we begin

The Rise of Hidden Classes

What it all means

More on the catch

Same, Same; but Different

TL;DR

Final thoughts

Further reading

Firing on all cylinders(Part 1): Understanding object value lookup in JavaScript interpreters and the rise of hidden classes

What to expect

Before we begin

The dynamic language

Memory storage displacement (offset)

Back to dynamic vs. non-dynamic

Hash Tables

Firing on all cylinders

Further reading

Streaming files from AWS S3 using NodeJS Stream API with Typescript

Checkout the new npm package!

Original Blog

Before we begin

The pipe dream

Bite by Byte

Smart Streaming

Put your darkest shades on, you're in!

Let's break this down a bit

Implementing with AWS S3

Further Reading

BUT WHAT ABOUT `42.length`??