Forem: Rakesh

Demystifying DOM(s) : Everything you need to know about DOM, V-DOM, Shadow DOM

Rakesh — Sat, 26 Dec 2020 09:46:06 +0000

Back Story

Wondering why this picture ?

Because if you have asked me about DOM few months back , I would have told you about him. As he is the only "Dom" I knew, till I started learning web dev at neoG Camp.

In our third week at neoG Camp, I got introduced to another DOM formally by our mentor .Although it was just enough intro to work upon the assignment , I got more confused when virtual DOM is introduced in our fourth week . And when I heard that there is something called shadow DOM too , I could not stop myself to dive deeper.

Here is what I am able to understand about DOM(s) .

What is this DOM ?

DOM or Document Object Model is a web api that models the entire html document in an object oriented way , so that programming languages like Javascript can access and manipulate it's properties .

Why ?

Because without representing the elements of html document as objects, Javascript won't be able to recognise them,let alone accessing it's properties .

How ?

By taking all the elements of the doument as objects and then representing them in a hierarchical tree . In that tree each branch starts and ends with nodes and by using these nodes javascript control the whole document .

so only elements are represented as nodes ?

Not Necessarily . A node is just the generic name for any type of object in DOM tree .
It could be one of the built-in DOM element like document or document.body or it could be an element node or text node or even attribute node too .

so if it's all objects underneath , why updating it slow ?

Actually updating the DOM is not slow , It's like updating any javascript object . It's what comes after the manipulation that's where the problem lies . So in order to answer what is making the whole process slow we need to understand browser workflow first .

Broswer Workflow

From the above image , you can see it's happening in four stages mainly .
But you must be thinking who is responsible for all of it . The answer is "Rendering Engine" .

The Rendering Engine

Like the name suggests, all it does is Render, that is displaying requested contents on the browser screen. Below we are going to see the work flow of webkit engine that powers browsers like safari ,chrome and edge .

As you can see from the above image before painting each modification it has to go through all those time consuming stages like layout and paint .

So for example if you have modified 40 nodes, one by one. That would mean 40 potential re-calculations of the layout, 40 potential tree modifications and 40 potential re-renderings too .

And that's what makes it look like direct DOM Manipulation slow .

It's like creating your bike from scratch for every little fix or color change or even stickering . Then imagine if you have to do the same thing for a BUS .

Yes it's insane , If you are even thinking about creating something like facebook and twitter in this way .

then solution ?

Virtual DOM a.k.a VDOM .

Although it is not the only way we can address this issue , We are going to talk about this only .

Virtual DOM

Virtual DOM is nothing but a light-weight copy of actual DOM that exist only in memory . Like actual DOM it is a object too . You can think of it as a blueprint to the actual thing .

then how does it different from actual DOM?

It lacks the ability to directly change what is displayed on the screen until the real DOM is updated . Which is why, it does not have to go through all the heavy weight stages like re-layout or re-paint etc . It just observes the changes then draws out in blueprint and passes it down to the real DOM .

Sounds fascinating , right ? Even if it's not ..can u please pretend and ask something like below question..

how does it do it ?

As you can see from above pictures , the whole process can be divided broadly into three parts .

State Change - Due to changing state of red color node , virtual DOM is generated in memory with the updated tree .
Compute diff - Comparison is carried out to detect the changes and it's effects.
Re-render - Finally , the change in real DOM is consolidated and updated at once.

Now let's get to the million dollar question..

How does react use it to solve our problem ?

In short , by using following things :

Observable instead of dirty checking to detect the change
Efficient diff algorithm
Batched update operations
Efficient update of subtree only

Using observable's instead of dirty checking

In React, each piece of the UI is a component and each component has an internal state. This state is observed by the library in order to detect changes in it. So whenever state changes , React mark that component dirty and re-renders it.

So unlike angular that uses dirty checking to find modified models in regular intervals , React uses observables to find the exact modified components . If there are no state changes then it won't have to do anything .

The Diffing Algorithm

Before rendering a new component , React must figure out the specific parts that need to be updated . And to figure out this React must compare between two Virtual DOMs it maintains at any given time , one with the updated state and the other with previous state .

And To achieve it efficiently , React uses a heuristic approach known as the Diffing Algorithm of the order O(n) instead of using state of the art algorithm of order [O(n³)] .

HOW ? by relying on below assumptions ..

Assumption 1 : Two elements of different types will produce two different trees.
Assumption 2 : The developer can hint at which child elements may be stable across different renders with a key prop.

Batch Update operations

React uses the diffing process to find minimum no. of steps to update the Real DOM and once it has those , it executes all of it in one event loop . So if there are more elements that needs to get updated in meantime , React will wait for the event loop to finish first and then in bulk will update the DOM with updated elements .

After all these steps are done, React will repaint the Real DOM .

As you can see DOM is getting repainted only once during the whole process, thus all the layout , re-calculation process will run only once .

Updating the subtree only

ReactJS traverses the tree using BST. So if the state of a component has changed, then ReactJS re-renders all the child components even if they are not modified .

Consider the tree below.

States of element B and H have changed. So when using BST ReactJS reached element B it will by default re-render the element H. This is the reason to use BST for tree traversal.

Is shadow DOM is same as Virtual DOM ?

NO..they are completely different things..

Shadow DOM

Shadow DOM refers to the ability of the browser to include a subtree of DOM elements into the rendering of a document, but not into the main document DOM tree.

you can say it simply as “ DOM within a DOM” .

As it remains hidden and separate from the scope of the other codes , we use it as a meanning of creating web component .

Can you give a example that we know ?

think of <video> element with all it's default controls . For us it maybe
just <video> element that we can put anywhere in our document without any hassle.

But if you look closely in it's shadow DOM , you are gonna find out a series of buttons , controls and other settings hidden and separated from the outer code .

How did I remove the rate limit from my minion translator and how you can do it too : a neogCamp story

Rakesh — Mon, 21 Dec 2020 15:59:37 +0000

Introduction

For those who don't know what I am talking about , Neog Camp is a project oriented bootcamp under the supervision of tanay pratap , where you can only advance to the Level ONE, if you have finished all the assignments from Level ZERO .

And Minion Translator is one of those assignment in which, we have to use the fun translation api to translate from english to minionese .

so what's the problem ?
The problem is rate limiting .

The Problem

Fun translation or similar public apis ratelimit the api calls to maintain their server level.So you can imagine the problems that can arise if 500 folks uses the same api that accepts only 5 calls per hour .

And not only it was causing problem during the devlopment, it was gonna be there even after the deployment too. Imagine sharing your app to your friends or employers and all they get is error occured ! try after sometime message .

The Solution

There was only two possible solutions to these problems and they were

Finding another API that is free and unlimited
Building your own API

I choose the second option after wasting a day in search for the first one .
Before that I have known about APIs in theories only, never used one of them not even a mock one before Neog Camp tutorial. But still somehow It seems feasible after using the Mock Server created by Tanay Pratap .

He have already taken care of the input and output part of the program, so all I have to do was figure out the processing part . When I thought about the inner-workings of fun translation API, It seems like all it was doing underneath is taking our values , comparing it to values in the database and returning whatever the corresponding values as result and If something is not on the database it returns the same.

So first of all I needed to find a database of minion words and thanks to the popularity of minion it was not that hard. I found a extreme large list of minionese words compiled by Bryce Dorn and available under MIT license in github .

After getting the database all I had to do was write a program that can map through the list and compared the input to the words in the list . As we all have done similar program in our second CLI app , I tried similar approach here too . But only could translate single words even after a lot of attempt . So I took the help from our mentors Rohit Gaur and Sreetam Das and Voila , our own api was ready .

Now only one more thing was left that is finding a way to keep it running forever . Which was taken care by Repl.it very efficiently as once deployed, the HTTP Server will continue to run in the background, even after you close the browser tab .

you can read more about it from here .

End Result

I have used it in my own minion translator app . so you can test it here.

Sometimes it may take some time waking a idle server which can be fixed by a quick refresh but normally you won't need it .

How can you use it in your own app ?

Step one

Go to the below link and fork the repl

 https://repl.it/@r4k3sh/minionsapi#index.js

Step Two

Run the Repl

Step Three

Grab the link it provide

add /translate/minion.json?text= to it

so for me it becomes :

https://minionsapi.r4k3sh.repl.co/translate/minion.json?text=

You will have similar links based on your repl name and id .

Step Four

Test the api on browser after typing anything after the above url .

If everything goes well you will receive the translated text instantly like below .

Step Five

Replace the link in your app instead of fun translation api url .

Similarly you can make any other translation API too ,If you can manage to find the wordlist.

That's it neoGrammers .

Thank you.

CREDITS

Bryce Dorn: For Compiling the Word list

Tanay Pratap: For Creating the Server

Sreetam Das & Rohit Gaur: For Writing the final piece of code

Neog Camp : For Existing

Getting started with Eloquent Javascript : Chapter one summary

Rakesh — Sun, 06 Dec 2020 15:29:57 +0000

“Below the surface of the machine, the program moves. Without effort, it expands and contracts. In great harmony, electrons scatter and regroup. The forms on the monitor are but ripples on the water. The essence stays invisibly below.”

— Master Yuan-Ma, The Book of Programming

Introduction

If we deep dive into the computer's world , we are going to findout it's all Data in there , It always has been . So if something is not Data neither we can use it nor mention it within computers .It's important to remember here every Data is fundamentally same as they are all stored in long sequence of Bits .

so you must be thinking what are these Bits..so let's talk about it .

What is Bits ?

Although we usually describe it as ones and zeros it can take any forms like high or low electrical charge , a strong or weak signal etc .

If any form of information can be reduced to a sequence of zeros and ones, It can be represented in Bits .

Fun Fact : Modern computers have more than 30 million bits in its volatile memory itself .

How can a human work with this ocean of bits without getting lost in them ?

The answer is Divide and Conquer . We group them according to their type so that they can represent particular pieces of information according to their role .

These chunks of Bits are also known as Values and they can be of different types like functions, text , number etc .

So how do we create those values and use it according to our requirement ?

All you have to do is call it ..or you can say invoke its name and voila , you will have it .

One thing we must keep in mind while using Values is, every one of them is getting stored somewhere in memory and that is memory is limited .

So if you are not needing them simultaneously there should not be any problems as values will disipate as soon as you stop needing them .

Different Types of Values

Numbers

As you can guess from the name, values of this type are numeric values like 13,15,100 etc. As javascript uses a fixed number of bits i.e 64 bits to represent Numbers, there is a certain limit to the number of different numbers we can represent, although that limit is about 18 quintillion.

There is something important here to remember that not only whole numbers but also negative(like -13 or -20) and fractional numbers (like 5.2 or 2.99e8 ) are there too, to be represent by the same 64 bits pattern .

However calculations involving with fractional numbers are generally not precise like it's with whole numbers .

which lead us to the main thing that numbers do, that is arithamtic operations.

Arithmatic Operations
It basically take two numbers , perform some operations with the help of operator like '+' ,'-', '*' ,'/', '%'and return another number .

The order in which the operations are going to be executed is decided by Operator Precedence. Most of the time they follow the BODMAS rule generally .

for example : 100+10*45 = 550

You can always change the precendce by wrapping the operands in parantheses .

for example : (100+10)*45 = 4950

Fun Fact : There is a difference between modulo('%') and
remainder ('%').Remainder can be negative but modulo is
always positive.

Special Numbers
There are three special values are there too in javascript which are considered as numbers but don't behave a like a normal one .

They are iInfinity ,-Infinity and NaN ( stands for "not a number") .

Strings

After Number strings are the next common data types , which are mainly used to represent text . If we want anything to be perceived as strings, we must wrap them around in quotes .

It can be single quotes,double quotes or backticks .Take a look at below examples.

   `Down on the sea`
   "Lie on the ocean"
   'Float on the ocean'

so you must be thinking why are using backticks ?

Because when required they can embed other values too .
let's have a look at below example

half of 100 is ${100 / 2} // "half of 100 is 50"

you can see when we write something inside ${}within a backtick, the results will be computed first before converting into a string.They are known as Template Literals .

Fun Fact : we can use '+' on strings which concatenates or glues them together .

Boolean Values

Javascript has a Boolean data type which only has just two values i.e true or false . It is very useful in controlling the program flow when using conditional statements like if..else .

console.log(3 > 2) // true
console.log(3 < 2) // false
console.log(NaN == NaN) // false

Fun fact : NaN is only value in js that is not equal to itself.

Empty values

You may have known them as undefined or null . These values are only used in the absence of meaningful value .

Generally undefined means the variable or the memory location is not defined or instantiated but null however means lack of a value , however you can use them interchangeable as it does not matter most of the time .

Operators

Operators are normally special symbols which are used to perform operations on values or variables a.k.a Operands . But not all operators are symbols some are written as words too .

for example type of operator . It is also known as unary operator as it only operates on one value .

for example :

console.log(typeof 6 ) // Number

There are also Binary and Ternary operators too , which operates on two or three operands respectively .

Binary operators examples :

console.log (6 + 3) // 9

Ternary operator syntax :

(conditon) ? statement-if-true: statement-if-false.

Conclusion

So we have discused briefly about various types of javascript values and operators that can transform them . But if you want to know more about them you can read it from here.

Thank you ..

How does javascript really work : a peek under the hood..

Rakesh — Tue, 24 Nov 2020 13:35:26 +0000

Javascript can be found everywhere nowadays ,from browser to server, from mobile to desktop, even in hybrid apps as well. Also It has a vast community behind it with an equally incredible eco-system which is contineously evolving in a proper way .

All of this led it to become one of the most preferred language to learn for newbies like me for getting a job in near future . But a lot of us never try to find out how it is actually working behind the scene , unless they have a interview scheduled next week .

That's why this article is an attempt to give everyone a heads up about the basics ..so without further ado ..let's get going then..

Have you ever thought how come all the modern browsers understand javascript ? who is translating it for them ?

They all have something similar...A Javascript Engine built into them...

What Is Javascript Engine now ?

A Javascript Engine nothing but a program written normally in C and C++ , which goes through the javascript code one line at a time and convert it to the machine readable format so that the cpu can understand and execute it .

Some famous javascript engines are :

V8 – engine used in Opera & Chrome.
SpiderMonkey – engine used in Firefox.
SquirrelFish – engine used in Safari.

Ok, So What's in this Engine ?

It is mainly consist of two main parts :

Memory Heap
Call Stack

What are those now ?

Memory Heap- Whenever you define a variable, constant,
object,etc in your javascript program, you need some place to
store it first , That place is nothing but the memory heap.

So Whenever we define a variable likevar a = 10, a
location in the memory is assigned to store the value of a.

However there is something important here to remember , the
available memory in Memory Heap is limited . So even if
Javascript offers a effective garbage collection mecanism
automatically, Memory leaks can still occur due to various
reasons like Using Global Variables or Using Too Many Event
Listeners etc etc.
Call stack - The Call Stack is nothing but a data
structure,which records basically where in the program we are.
As JavaScript is a single-threaded programming language, it
has only one Call Stack .

Which means whenever we execute a function i.e we put it on
the top of the stack or when we return from a function i.e we
pop it off from the top of stack , all of it happens one at a
time.

Call stack size is also limited like Memory Heap . So whenever
a program attempts to use more space than there is available,
Stack Overflow occurs .

Why does Javascript choose to be single threaded ?

Because it is easy and less complicated than multi-threaded environment,
where sometimes you have to deal with scenarios like Deadlocks and all .

But isn't it limiting ? What's gonna happen when things are slow ?

Yes . If we process things like image processing or network request synchronously, It can be a cause of browser getting stuck or block .
And that's not the only problem you are going to encounter here . Once the browser starts to process all the remaining tasks, It may stop being responsive for a quite long time .

I think all of you must remeber this page unresponsive error .

So How to prevent Blocking then ?

Well , the answer is Asynchronous Callbacks .

Wait , you just said it is a single threaded langunage,then how does it handle Asynchronous Callbacks ?

Well it turns out I have been telling you partial truth this whole time . There's more to the browser than just the engine . It also consist of Event Loop and Call back Queue along with various Web APIs like the DOM,
AJAX (XMLHttpRequest), setTImeout() and many more .

So, you are saying setTimeout() is not a part of javascript then ?

Yes , Technically it's not a part of javascript engine . As it provided by the
Web APIs .

Can you give us a example ?

Sure , Consider the following the codes..

console.log('1');
setTimeout(()=>{
   console.log('2');
},5000);
console.log('3');

As you all know, the output gonna be like this

1
3
2

but do you know how does it arrive at this output ...let me walk you through this ..step by step..

Five seconds later...

I have a doubt , What if the setTimeout() is set to 0 ?

The result is still going to be same , As the callback by any means not going to pass through the Event Loop untill the stack is complete empty .

Ok,.this looks interesting , Where can I find out about this more ?

I have choose to not going into too much depth about it because if you are only starting out it might be overwhelming for you already . But for those who are into the game a little bit longer, I believe the following resources can help them for sure .

https://medium.com/better-programming/how-javascript-works-1706b9b66c4d

https://blog.bitsrc.io/how-does-javascript-really-work-part-1-7681dd54a36d

https://www.youtube.com/watch?v=8aGhZQkoFbQ&feature=emb_title