Forem: Atishay Jain

What exactly happens when you enter a URL into your Browser

Atishay Jain — Tue, 18 Jul 2023 10:22:17 +0000

Have you ever wondered what happens when you type a website address like google.com or youtube.com into your browser’s address bar and hit enter? Sure, if you are connected to the internet, you are presented with the desired web page. But exactly what goes behind the scenes is what we will be discussing in this blog.

Structure

This blog will be divided into two parts, the first part will give a complete high-level overview of the whole process that starts with a client typing in a URL then, then converting the URL to an IP address (using DNS), establishing a connection to share resources and finally rendering the web page.

In the second part, we will learn about DNS and how it actually resolves the client-provided domain name into the corresponding IP address.

Part 1: The Overview

1. URL Parsing

Let’s just first understand what a URL is-

URL stands for Uniform Resource Locator. It is composed of four parts namely- Scheme, Domain, Path and Resource, as shown in the image below.
If you are confused about path and resource, consider a file system and think of path as the directory path and the resource as the actual file (.txt / .cpp file or anything).

The process starts with the client entering a URL into the browser’s address bar. The browser then parses this URL, which basically means it destructures it into components (see the diagram above). Now that the browser has the domain name extracted, the process of DNS resolution can start.

2. DNS Resolution

The machine does not understand English, that’s what we humans understand and thus are provided to type the domain name of whatever site we need to refer to. To fulfill our request, the browser needs to know at exactly what address the web page is stored, and that address must be a bunch of 0’s and 1’s!

In other words, it is required to convert the domain name into an IP address of the server that currently stores the files to render the required web page. After obtaining the IP, a connection can be established between the client’s machine and the server.

As soon as we hit enter, the browser first looks into something known as a DNS Cache. A cache is something where recently used information is stored for retrieval to reduce the load on servers. They are of two types-

Browser DNS Cache — Domain name and IP address mapping stored in the browser. If you are using Chrome, you can see your browser's DNS cache at chrome://net-internals/#dns
OS DNS Cache — You can see the stored IP addresses in your computer by running ipconfig/displaydnson your terminal.

If the value is not found in these caches, the request goes on to the DNS resolver. For now, let’s treat it as a black box that receives a domain name and sends the corresponding IP address back to the browser. The actual working of DNS will be discussed in detail in Part 2 below.

3. Establishing a Connection

With the IP address obtained from the DNS resolution process, the browser can establish a TCP (Transmission Control Protocol) connection with the web server through a three-way handshake.

If the website uses HTTPS (secure), an additional step called SSL/TLS handshake occurs to establish a secure connection. You can read more about it here.

4. Sending an HTTP Request

Once the connection is established, the browser sends an HTTP (Hypertext Transfer Protocol) request to the web server. The request includes information about the resource you want to access (e.g., the path in the URL) and any additional data if you’re submitting a form or interacting with the website.

5. Receiving the HTTP Response

After processing the request, the web server sends back an HTTP response to the browser. The response contains an HTTP status code, headers (e.g., Content-Type, Cache-Control), and the requested content itself (e.g., HTML page, images, CSS files).

6. Rendering the Web Page

Upon receiving the HTTP response, the browser starts rendering the web page. It interprets the HTML content, loads any additional resources referenced in the HTML (e.g., images, stylesheets, JavaScript files), and executes any JavaScript code.

Finally you can interact with your favourite website and all this happens in few seconds, pretty neat isn’t it?

Part 2: DNS Resolution

Now let's expand the black box we created in step 2.

DNS stands for Domain Name System, it is essentially a distributed and decentralized collection of servers that works together in order to convert the domain name into a corresponding IP address.

There are mainly four types of servers in DNS -

DNS Resolver
The Root Server
Top Level Domain Server (TLD)
Authoritative Name Server

We will be going through each of their functionality in detail. But before that, we need to know how our computer gets access to this DNS system. So, to answer that let’s learn about ISPs.

1. ISP

After a hurtful cache miss in the browser and OS cache storage, the browser turns to an ISP (Internet Service Provider). The ISP looks into its own cached records and if it is not able to find the domain name, it passes the IP address of the DNS Resolver.

Some examples of ISP’s are verizon, AT&T, etc.
In India some of the popular ones are Airtel, Reliance Jio, etc.

2. DNS Resolver

A DNS resolver is anything that returns an IP address corresponding to a domain name, previously our browser, computer, and ISP acted as a DNS resolver. Since we met with cache miss in all the above three, we need to contact a higher authority, a place where we will be definitely able to get our IP address.

This place is nothing but the network of servers that we call DNS, and the thing that passes our query to this DNS system is the DNS resolver, it basically acts as an entry point to the DNS.

The address of a DNS resolver is provided by your ISP. But you can also change it by tinkering with the network settings. Changing your DNS can lead to a faster and more secure browsing experience. Maybe now’s a good time to learn what a VPN does (but we will go off-topic, instead learn about it here :)).

Some popular examples are Cloudfare's 1.1.1.1 and Google’s 8.8.8.8

We will be discussing a ‘recursive’ type of DNS where we basically tell the DNS resolver to not come back until it has the IP address.

There is another type which is called — ‘iterative’. In this, the client makes a series of requests itself until it found a match. In case of a miss, the current server directs the client to try on some other server. Learn more about it here.

3. Root Server

The root nameserver maintains an up-to-date index of TLD nameservers, each of which handles a specific TLD, such as .com, .org, or .net. The root nameserver also maintains a cache of records, if it is not able to find the queried domain name, it sends back the the address of the appropriate TLD server. (For example, the address of the .in TLD server is sent if the query was for amazon.in).

The resolver then queries the specified TLD server.

There are a total 13 sets of these root servers (A-M) which are strategically placed around the globe. They are managed by 12 organization. The map below shows the distribution of these root servers

https://root-servers.org/

4. Top-Level Domain (TLD) Servers

The TLD servers manage information about domain names under their specific TLDs. When the DNS resolver queries the TLD server for the IP address of the requested domain (e.g., “google.com”), the TLD server either provides the IP address directly(cache system) or refers the resolver to the authoritative name server responsible for that domain.

5. Authoritative Name Servers

These are the final stop in the DNS resolution process. The authoritative name servers store the DNS records for specific domain names. Each domain typically has at least two authoritative name servers for redundancy and fault tolerance. When the DNS resolver contacts the authoritative name server for the requested domain, it receives the IP address of the server hosting the website associated with that domain.

6. IP Address Returned

The client (finally) gets back the IP address from the DNS resolver, which caches the IP address in its local cache for future use.

Step 2 described in Part 1 is now complete, and we can move on with the rest of the steps as described.

Notice how caching mechanism is involved in every step of DNS resolution. And how the DNS system is distributed among various servers. This distributed network and the caching mechanisms are necessary in order to make the URL resolution process seamless and efficient.

If you learned something new, be sure to show your support and check out my other blogs —

The Backend Development Primer [Part 1/2]
The Backend Development Primer [Part 2/2]

Find me on LinkedIn and Github.

References

All the illustrations in this article are taken from ByteByteGo. Check them out :)
A short video by ByteByteGo (Part 1)
DNS Resolution explanation by PowerCert Animates Videos (Part 2)
A short video by ByteByteGo (Part 2)
https://www.computerhope.com/jargon/d/dns-resolver.htm
https://www.cloudflare.com/learning/dns/what-is-recursive-dns

Database and Scaling : Backend Development [Part 2/2]

Atishay Jain — Fri, 07 Jul 2023 08:15:30 +0000

In this article we will continue where we left off. If you haven’t checked the ‘Part 1’ out, I highly recommend to read that first, so you get a basic understanding of how things work at the backend :) Now let’s get started.

Databases
Authentication and Authorisation
Performance Optimisation
Caching
Scaling
Serverless Architecture

1. Databases

Data persistence is integral to the vast majority of web applications. Being able to understand how to structure, build, and query your own database are important skills for any full stack developer to have.

Databases are split into “relational” and “non-relational” types of databases, and each handles data and scaling in different manners.

SQL(Structured query language) databases are relational databases that store and manage data using tables. They ensure data integrity, support powerful querying with SQL, and offer scalability, ACID transactions, and data security. Popular implementations include MySQL, PostgreSQL, and Oracle Database.

NoSQL (Not only SQL) databases offer flexible data models, scalability, and high performance. They don’t enforce strict schemas (like SQL) and excel in handling large volumes of unstructured data. Popular implementations include MongoDB, Cassandra, and Redis. They are ideal for real-time data, high-speed transactions, and large-scale analytics.

Ultimately, which DB you chose depends on the type of functionality you want in your application. This article goes in depth with the difference between the SQL and NoSQL DB’s.

2. Authentication and Authorisation

Are they the same thing ? Definitely not.

Authentication is the process by which we determine whether a user is who he claims to be. Whereas authorization is the process of verifying what specific applications, files, and data a user has access to. Authorisation always comes after authentication.

There are various way of providing authentication, some of them being -

Classic email and password verification
Google’s OAuth 2.0 protocol (sign in with google feature)
JWT tokens. Learn More.
Using libraries such as passport and session. Learn More.

Note that hashing users' passwords before storing it is a very important step, storing user’s password in plain text in your database is very dangerous flaw. A library called bcrypt helps with hashing and comparing the passwords.

If you want to learn more about authentication, then I recommend reading about zero-knowledge proofs, which is a core concept in authentication.

3. Performance Optimisation

In today’s fast-paced digital world, a slow-loading website or application can frustrate users and cause them to abandon it altogether. Therefore, it’s crucial to optimize the performance of MERN stack applications to ensure they can handle a high volume of traffic without compromising on speed and performance.

Performance optimisation can be categorised in to two parts -

Code Optimisation : This includes techniques like minification of code, lazy loading, server-side rendering, code splitting and load balancing.

Database Optimisation : This includes techniques like indexing, denormalisation, caching, partitioning and database sharding.

Note that these optimisation techniques becomes important when developing enterprise level applications. We will discuss caching and database sharding in detail in the upcoming sections.

If you want to know more about the techniques mentioned above. Check out this article.

4. Caching

As discussed above, caching is database optimisation technique. A cache layer or server acts as a secondary storage layer, faster and highly efficient to temporarily store a subset of data which doesn’t change often and is frequently accessed by the client. To maintain it’s small size and freshness, the data items also have an expiration time.

Two caching paradigms-

Cache Aside Pattern : Also known as lazy loading, is the most common caching pattern available. Here, the cache is updated after the data is requested. In order to read data from the database, the cache is first checked to determine whether the data is available. If the data is available (also known as a cache hit), the cached data is returned. Otherwise (cache miss), the database is queried for the data and then the cache is the populated with this data.

Write-Through Pattern : Here, when the application or backend process updates the primary database, the data is also updated in the cache. In case of cache miss, lazy loading pattern is deployed. Here, the chances of a cache miss is reduced but, the size of cache has increased and it’s also storing items which are infrequently accessed.

‘node-cache’ and ‘memcache’ are two popular Nodejs packages. The most popular option for caching is using ‘Redis’. Being an in-memory database, its data access operations are faster than any other disk-based database, which makes Redis the perfect choice for caching.

Check out this article for difference between on-disk and in-memory databases.

5. Scaling

While developing an application in our local machine we have one server, to which we make a single request at a time. Also, our application acts as a single service. This type of system is called a monolithic system ((0,0,0) in the cube shown below)

However, an enterprise level application needs to handle multiple requests at the same time from possibly million of users. Surely, there is a need to scale our application so that it can perform in these type scenarios.

The diagram below shows three dimensions of scaling, let’s discuss each of them in brief below-

Cloning Services: To prevent one server to become swarmed with requests and thus take a lifetime to respond, we create multiple instances of our server listening for the same set of requests. We can implement this in our local machine, we use cluster module which is a inbuilt module in Nodejs, having multiple cores (or CPU’s) by running each instance in each of the core.

const express = require('express');
const cluster = require('cluster');
const totalCpus = require('os').cpus.length;
//totalCpus will give total number of CPU's in your machine

if (cluster.isMaster) {
for (let i = 0; i < totalCpus; i++) 
  // create worker threads
  cluster.fork();
}
else {
  //each worker thread will execure this
  const PORT = 3000;
  const app = express();
  app.listen(PORT, () => {console.log(`Server running on ${PORT}`)};
}

//This will result in multiple servers listening on the same PORT and also 
//connected to the database seperately.
//Output
/* Server running on 3000
   Server running on 3000
   Server running on 3000
   ....8 times, if running on octa-core machine.
*/

Check out this video to see the difference in response time of multiple requests while using one versus multiple servers.

Database Sharding: We saw above that we can create multiple server instances, but the problem is that all these server access the same database, thus querying a DB results in a bottleneck in our application’s performance. Database sharding provides the solution for this, using this technique each server is only responsible for only a subset of data.

For example, let’s take the example of Amazon. Let’s assume that amazon servers are running on 3 servers (let’s call them server A,B and C). These servers are designed in such a way that each cater to the request of only a subset of the user. For instance, Servers A handles users whose names' start with ‘A’ to ‘I’. Server B handles ‘J’ to ‘R’ and C handles ‘S’ to ‘Z’. Now suppose of server B requests data fromt he database, the database will now know that it needs to look in that partition that store users’ information with initial letter from ‘J’ to ‘R’
.

Thus, data partitioning, as shown in the above example can improve the application’s capacity and availability.

Microservices: Microservices are an architectural and organizational approach to software development where software is composed of small independent services that communicate over well-defined APIs. Where as Z-axis scaling splits things that are similar, Y-axis scaling splits things that are different. At the application tier, Y-axis scaling splits a monolithic application into a set of services. Each service implements a set of related functionality such as order management, customer management etc.

Now our multiple server instances can assigned to a different microservices. This way multiple independent functions in our app need not to wait for each other and can run at the same time. Note that how we divide our application into microservices are important and require careful planning.

Two ways to run your application that has been divided into microservices are using- container or serverless. A platform that provides container service is ‘Amazon elastic container service’. We will discuss serverless architecture in detail next.

If you are interested to know more about backend scalability, you can check out ‘The Art of Scalability’. Or you can check out this youtube playlist.

6. Serverless Architecture

As you may have noticed by now, maintaining a server, making it scalable and secured among other things is not an easy job. Thus, by adopting serverless architecture, developers can offload these responsibilities to a third-party provider, enabling them to focus on writing application code.

Function as a Service (FaaS), a popular type of serverless architecture, allows developers to focus on writing application code.Function as a Service (FaaS), a popular type of serverless architecture, allows developers to focus on writing application code.

One of the most popular serverless architectures is Function as a Service (FaaS), where developers write their application code as a set of discrete functions. Each function will perform a specific task when triggered by an event, such as an incoming email or an HTTP request. Developers then deploy their functions, along with their triggers, to a cloud provider account.

When a function is invoked, the cloud provider either executes the function on a running server, or, if there is no server currently running, it spins up a new server to execute the function. This execution process is abstracted away from the view of developers, who focus on writing and deploying the application code.

Few of the major FaaS providers are AWS Lambda, Google Cloud Functions and AWS Functions.

References

Image Source

The Basics : Backend Development [Part 1/2]

Atishay Jain — Fri, 07 Jul 2023 07:18:02 +0000

This article will include all the major terms you hear when backend development is discussed. You can use this to skim through all the important concepts before an interview ;)
Note that this article is written with the MERN stack in view, but the underlying concepts are universal.

Introduction
HTTP
Middleware Functions
REST Architecture
CRUD Operations
MVC Architecture
CORS
Frameworks

1. Introduction to Backend

The back-end is the code that runs on the server, that receives requests from the clients, and contains the logic to send the appropriate data back to the client. The back-end also includes the database, which will persistently store all of the data for the application.

The backend is made up of three parts :

The server : This is the computer that receives requests. To make your machine a server, install Node.js which is runtime environment for Javascript and listen on desired port number.

The app : This is the application running on the server that listens for requests, retrieves information from the database, and sends a response. A framework like express helps in making these apps.

The database : Databases are used to organize and persist data. They can be classified into SQL and NoSQL databases. MERN stack follows MongoDB database which is a NoSQL database.

2. HTTP

HTTP stands for Hypertext Transfer Protocol and is used to structure requests and responses over the internet. HTTP requires data to be transferred from one point to another over the network. The resources itself are transferred with the help of a TCP connection.

Analogy

Suppose you are at restaurant and you call a waiter for him to take your order. You dictate your order in english and the waiter notes it down, he then makes a track (at the speed of light) to the kitchen and tells the chef to prepare the order. He comes back, while ripping of the track, with the dish or some unfortunate news that the dish can’t be prepared.

Here, you are the client, the waiter the server and the chef is the database. To dictate your order you chose english, while the request on web uses HTTP, the track is the TCP connection to deliver resources, and whatever the waiter comes back with is the response. The dish you want to eat is the URL (uniform resource locator), this tells the server what actually the client needs.

How it is different from HTTPS ?

This ‘S’ in HTTPS stands for security. Coming to the analogy above, if you tell the waiter to include a secret sauce in the dish, it might not be a good idea for the waiter to note it in plain text, ( it can be read by anyone at the track junctions !) To secure sensitive and personal content, some servers may provide an encryption facility.

3. Middleware Functions

Middleware is any code that executes between the server receiving a request and sending a response. These functions might modify the request object, query the database, or otherwise process the incoming request.

In the analogy, the actual preparation of the dish can be thought of a middleware. Middleware functions typically end by passing control to the next (using next()) middleware function, rather than by sending a response. You often see them recieve three parameters — req, res and next.

const express = require('express')
const app = express()

//our middleware functions
const myLogger = function (req, res, next) {
  console.log('LOGGED')
  next()
}

app.use(myLogger)

app.get('/', (req, res) => {
  res.send('Hello World!')
})

app.listen(3000)
// code excerpt taken from express documentation

//Output
/* Every time the app receives a request, 
it prints the message “LOGGED” to the terminal.*/

4. REST Architecture

REST, or REpresentational State Transfer, is an architectural style for building API’s , providing standards between computer systems on the web, making it easier for systems to communicate with each other.

A RESTful API uses commands to obtain resources. In REST, each resource is identified by a unique URI (Uniform Resource Identifier) and can be manipulated using standard HTTP methods such as:

GET to retrieve a resource;
PUT to change the state of or update a resource, which can be an object, file or block;
POST to create that resource; and
DELETE to remove it.

It is based on the idea that the REST APIs are stateless, meaning that calls can be made independently of one another, and each call contains all of the data necessary to complete itself successfully.

It also separates the concern of server and client, allowing their implementation to be done independently of each other.

By using a REST interface, different clients hit the same REST endpoints, perform the same actions, and receive the same responses.

5. CRUD Operations

CRUD stands for Create, Read, Update and Delete operations. Think of a simple blog website, the basic function you want to give your users are — creating a blog, reading blogs of other people, updating content and deleting a blog. It’s crucial for a developer to know and implement these CRUD operations.

Corresponding to each type of operation there is a different HTTP verb that is included in the request that the client makes. HTTP verbs for CRUD are POST, GET, PUT, and DELETE respectively. Apart from HTTP verbs, a client’s request have certain other fields like — header and accept parameter, a path to resource and optional body content. Read about it here.

Similarly, server’s response consists of — content type, status code and any data the client requested for (read operation). Client will only accept the data if it’s type is one of the mentioned types in header of the request.

Example

//Clients request :
GET http://fashionboutique.com/customers
Accept: application/json
/*
HTTP verb : GET
Resource Locator : fashionboutique.com/customers
Accept files which are application/json format.
*/

//Servers response
Status Code: 200 (OK)
Content-type: application/json
/*
Status code 200 implies that response is successfully generated. 
Content type : the response is in application/json format
followed by the actual customers data
*/

6. MVC Architecture

MVC stands for Model, View, Controller and refers to the architecture of your code. It is a way to organize your application by separating all of the actions into 3 main components: Models, Views and Controllers.

Models are the basic building blocks of your database. So for every entry in your DB , you’ll create a model that holds the details of that entry. Models define the types of information that get used by your views and controllers. (Mongoose)

Views are the component that generates the UI for your application. It uses data supplied by a controller, plug that data in html with the help of a templating engine and display the desired information.(EJS or React-engine).

Controllers are the components that decide what view to display and what information is going to be put into it. It’s where the main logic of your server resides.

7. CORS

The Same Origin Policy is an important security measure that basically says “Only requests from the same origin (the same IP address or URL) should be allowed to access this API”.

But, we are specifically trying to set up our API so that we can access it from different origins, so to enable that we need to set up Cross-origin resource sharing, or CORS.

Setting up CORS is very easy by using a npm library of the same name.

//Simple Usage (Enable All CORS Requests)
var cors = require('cors')
var app = express()

app.use(cors())

app.get('/products/:id', function (req, res, next) {
  res.json({msg: 'This is CORS-enabled for all origins!'})
})

8. Frameworks

A framework is just boilerplate code bundled in an organised way. A obvious advantage of using a framework is that you don’t have to repeatedly write the same code. Another is that it gives you the organisation of files from the start according to MVC principles. Some good examples are —

Express
Django,
Ember
Rails and many more.

Using frameworks can reduce the verbose and provide necessary abstraction to speed up your development .

Check out the ‘Part 2' of this primer to know about how to scale and optimise your application and other cool stuff.

References :

Image Source :

Forem: Atishay Jain

What exactly happens when you enter a URL into your Browser

Structure

Part 1: The Overview

1. URL Parsing

2. DNS Resolution

3. Establishing a Connection

4. Sending an HTTP Request

5. Receiving the HTTP Response

6. Rendering the Web Page

Part 2: DNS Resolution

1. ISP

2. DNS Resolver

3. Root Server

4. Top-Level Domain (TLD) Servers

5. Authoritative Name Servers

6. IP Address Returned

References

Database and Scaling : Backend Development [Part 2/2]

Table of Contents

1. Databases

2. Authentication and Authorisation

3. Performance Optimisation

4. Caching

5. Scaling

6. Serverless Architecture

References

Image Source

The Basics : Backend Development [Part 1/2]

Table of Contents

1. Introduction to Backend

2. HTTP

Analogy

How it is different from HTTPS ?

3. Middleware Functions

4. REST Architecture

5. CRUD Operations

6. MVC Architecture

7. CORS

8. Frameworks