Forem: Bilal Ul Haque

Ngrok: Exposing local server on the internet

Bilal Ul Haque — Tue, 19 Mar 2024 05:58:08 +0000

Have you ever wanted to expose your local server to the internet for development purposes in a simple way? The answer is probably yes. I faced the exact same issue when I needed to expose my local server. Today, I’m going to discuss one of the solutions that caught my attention and is easy to use. First of all, let me briefly explain my problem.

As a Backend Engineer. I was working on a product that involved integration of payment gateways like Stripe, PayPal, etc. As some of you may know, we need to listen to webhooks to trigger various events in our application based on the events sent by the webhook. The main issue occurred was that the project was still in development phase and the payment gateway webhook couldn’t access the local server as it requires a public HTTPS URL. That’s when I had to use ngrok to expose my local webserver to the internet allowing the local server to listen to the events sent via the webhook.

Understanding WebHooks:

Webhook is a mechanism that allows one application to send real-time data to another when a specific event occurs. This asynchronous communication is useful for scenarios where immediate updates or responses are needed, such as in chat apps, payment gateways, etc.

Webhooks operate on a simple principle that when an event happens in the source application, it sends an HTTP POST request to a predefined URL in the destination application. This URL is often called the “callback URL” or “endpoint.” The destination application then processes the incoming data and executes the necessary actions.

What is Ngrok?

Ngrok is a multi-platform tunnelling tool that creates secure tunnels to localhost, making local servers accessible from the internet. It simplifies the process of exposing local services, such as web servers, to external networks, enabling developers to test and interact with webhooks in a real-world environment.

Key Features of Ngrok:

Secure Tunnelling: Establishes secure tunnels using HTTPS, ensuring that data transferred between the local server and external services is encrypted.
Port Forwarding: Allows developers to expose specific local ports, redirecting incoming requests to the corresponding services.
Public URLs: It generates public URLs for the exposed local servers, eliminating the need for a public IP address or domain name.
Inspection and Replay: Offers a web-based interface for developers to inspect HTTP traffic and replay requests. This feature is particularly useful for debugging and analysing webhook payloads.
Authentication and Access Control: Enables developers to add authentication to their tunnels, ensuring that only authorized users can access the exposed services.
Custom Subdomains: By providing the option to use custom subdomains for the public URLs, it adds flexibility to the testing environment.

Ngrok Use Cases:

Web Development: Allows to make your local server accessible via a public URL, so that any device from any location can access it.
Webhooks Testing: Most of our application relies on webhooks to receive data or events from external services. Ngrok enables the exposure of your local server to the internet, allowing the application to listen to incoming webhook endpoints.
Demonstration Purposes: It gives a temporary public URL, which enables us to showcase local development environment for demonstrations.

Conclusion:

Ngrok serves as a powerful tool for simplifying webhook development and testing. Its ability to create secure tunnels, generate public URLs, and provide advanced features makes it an amazing resource for developers working on applications that require real-time communication.
Integrating Ngrok into your development process streamlines the process of exposing local servers, testing webhooks, and debugging applications.

Exploring DNS

Bilal Ul Haque — Tue, 05 Dec 2023 06:32:47 +0000

Exploring DNS: Understanding the Communication Between Domain Names and IP Addresses!

Ever wondered how, when you type "example.com" in your browser, what happens behind the scenes? 🌐 That's where DNS (Domain Name System) steps in.

DNS, which stands for Domain Name System, plays a crucial role in facilitating user access to websites by translating domain names into IP addresses.
When a user enters a website like example.com in the search bar, the DNS server performs the essential task of converting the website name into its corresponding IP address and promptly delivers it to the client. This IP address enables the web browser to establish communication with the web server, ultimately retrieving the requested data.

Unlocking Efficiency: A Deep Dive into Redis as an In-Memory Datastore

Bilal Ul Haque — Tue, 21 Nov 2023 06:18:56 +0000

Introduction:

In today’ s world, web applications require a lot of things up and running on the server to serve the purpose of users. Unfortunately, many developers might not be aware of factors that could ruin the user experience. The quality of code optimization, server response, and database integration all play crucial roles, and this can vary depending on the chosen framework or programming language. However, developers have the power to boost their application’s performance significantly through optimization. One key optimization is ensuring an efficient database connection for speedy responses. This is where an In-memory datastore comes into play.

An in-memory data store is a type of database or data storage system that primarily relies on keeping data in the system’s main memory for data storage and retrieval, in contrast to traditional databases that store data on disk drives. This approach allows for faster data access and retrieval since reading and writing to system’s memory is much quicker than accessing data from disk storage. In-memory data stores are commonly used in various applications, ranging from caching mechanisms to real-time analytics and high-performance computing.

In this article we’ll be looking at one of the most popular in-memory data store Redis.

Redis is an open-source, in-memory data structure store, used as a distributed, in-memory key-value database, cache, and message broker, with optional durability. Redis supports different kinds of abstract data structures, such as strings, lists, maps, sets, sorted sets, etc.

Use Cases:

Caching: Redis is an ideal choice for establishing a high-performance, in-memory cache to reduce data access latency and boost throughput. Its ability to deliver responses in milliseconds contributes to its impressive speed, and its scalability allows for handling increased loads without substantial backend costs. Utilizing Redis for caching purposes is common in scenarios such as caching database query results, persisting session data, storing web pages, and caching frequently accessed objects like images, files, and metadata.

Messaging and Queue: Redis has a great support for Pub/Sub (Publish and Subscribe) along with pattern matching and a diverse set of data structures. This functionality empowers Redis to facilitate high-performance messaging, chat rooms, and server intercommunication. Utilizing Redis List data structure, we’ve implemented a lightweight queue that proves advantageous for applications requiring a dependable message broker. Lists, with their great operations and blocking capabilities, are well-suited for various scenarios where reliability is important.

Session Store: For scalable web applications, Redis stands out as the optimal selection due to its in-memory support. It delivers sub-millisecond latency, scalability, and resilience essential for handling session data such as session state, credentials, and other critical information.

Different types of Redis Architecture:

Redis provides multiple architectural options to deal with different scalability and fault-tolerance requirements.

Standalone:
In its standalone mode, Redis operates as a singular instance, offering a straightforward and uncomplicated deployment choice that is well-suited for smaller-scale applications or development environments. However, it presents a single point of failure, implying that if the instance experiences downtime, the entire Redis system would immediately stop functioning.

Master-Slave Replication:
Redis incorporates master-slave replication, allowing for the existence of multiple instances. In this configuration, one Redis instance serves as the master, managing both read and write operations, while several slave instances replicate data from the master and are exclusively utilized for read operations. This arrangement improves data availability and enhances read scalability.
But only one node can be the master and the number of slaves nodes can be incremented according to the requirement.

Redis Sentinel:
This architecture serves as a high-availability solution that introduces automatic failover and monitoring capabilities to the master-slave replication architecture. In simple terms, in a traditional Master-slave setup, if the master goes down, human intervention is needed to promote one of the slaves to become the new master since write operations are halted. Sentinel streamlines this process by automating the promotion of a slave node to the master, eliminating the need for manual intervention. Redis Sentinel utilizes a special TCP port, typically set to 26379 by default. This port facilitates communication among sentinels, allowing them to monitor the health of the Redis node. If the master node becomes unresponsive via the TCP port, sentinels collaborate, and the one receiving the most votes assumes the leadership role. The leader Sentinel then initiates the promotion of one of the slaves to take over as the new master.

Redis Cluster:
Redis Cluster represents a more advanced architecture compared to Master-Slave and Redis Sentinel. It surpasses them in power, robustness, and efficiency, particularly during failovers. Redis Cluster features multiple master nodes, enhancing its overall performance.

In this architecture, data is divided into 16384 hash slots, each assigned to a single master node. The number of hash slots increases with the addition of more master nodes. When a client sends a request to Redis, the request is directed to a specific hash slot based on the shard key provided.

The sharding process involves generating a hash key (shard key) from the key sent to Redis. This shard key determines the slots to which the request should be directed. It’s important to note that each slot functions as a Redis sentinel, enabling automatic recovery of the master node in case of a failure. This design enhances the resilience and reliability of Redis Cluster.

Redis cluster is highly scalable when it comes for large enterprise application since the load is now distributed to the different slots based on sharding and can process more number of request at once.

Conclusion:

In conclusion, Redis is a robust and versatile in-memory data store, renowned for its speed, efficiency, and adaptability. Whether serving as a caching solution or a message broker, Redis excels in providing low-latency access, scalability, and resilience. With features like Pub/Sub, master-slave replication, and Redis Sentinel for high availability, it caters to a wide range of applications. From small-scale to internet-scale deployments, Redis remains a top choice for developers and businesses seeking a reliable and high-performance data management solution. Its simplicity, speed, and flexibility position Redis as a fundamental component in the realm of modern computing.

Cross-Origin Resource Sharing (CORS)

Bilal Ul Haque — Mon, 13 Nov 2023 15:55:21 +0000

Consider the following scenario: You attempt to use your website’s GET() endpoint to retrieve some data from an API, but an error results.

The Access-Control-Allow-Origin header has a value of “some URL” that is not equal to the supplied origin,” or “The Access-Control-Allow-Origin header has a value of “No Access-Control-Allow-Origin header is present on the requested resource,” written in red text, indicating that your request was denied due to CORS policy.

What is CORS?

CORS (Cross-Origin Resource Sharing) is a method that permits resources to be loaded from one origin to another while preserving the security and integrity of the website. Popular web browsers like Chrome and Mozilla Firefox use this security technique to determine which cross-site requests are secure.

Modern browsers use the same-origin policy to limit access from scripts to resources outside their domain for security concerns. This policy was created in reaction to malicious activities including cross-site request forgery (CSRF) assaults and theft of personal information from other web servers.

Why does the same origin exist?

You might use cookies, like many websites, to remember login information or session details. When they are created, those cookies are restricted to a specific domain. The browser will include the cookies made specifically for that domain with each HTTP request to that domain. This occurs on every HTTP request, whether those for AJAX, HTML sites, or static images.

This indicates that a cookie is stored for https://exampleapi.com when you log into https:// exampleapi.com. The bank may have developed a REST API at https:// exampleapi.com/api if it is a single-page React application so that the SPA can interface with it using AJAX.

Unfortunately, developers that want to fetch different resources from various origins found this same-origin constraint to be somewhat restrictive. The CORS HTTP protocol was created to inform browsers to allow restricted resources on a web page to be requested from other domains in order to loosen restrictions a little. Here is an example of a case where information could be requested from an external source, such as an API (this is a very standard practice for client-side JavaScript code):

Using CORS headers, the resource origin sends a preflight request to the external web server.
The preflight request is then validated by the external web server to ensure that scripts are authorized to make the request.
The external web server replies with its own set of HTTP headers that specify the allowed request methods, origins, and custom headers after the request has been confirmed. Information on whether it is appropriate to pass along credentials, such as authentication headers, may also be included in this final server response.

Why do we need CORS?

A script that runs in a user’s browser would typically only ever need to access resources from the same origin (think about API calls to the same backend that served the JavaScript code in the first place). Therefore, it is good for security that JavaScript typically cannot access resources on other origins.

“Other origins” here refers to the fact that the URL being viewed is different from the location where the JavaScript is being executed by having:

A different scheme (HTTP or HTTPS)
An alternative domain or port

Cross-origin access, however, may be required or even desired in certain situations. As in the case of a React SPA that communicates with an API backend that is hosted on a different domain.

As you can see, CORS is an important protocol for making cross-domain requests possible, in cases where there’s legitimate need.

How does CORS work?

CORS requests come in two types: simple requests and preflighted requests. The criteria for determining if a request is preflighted are discussed later:

Simple requests:

A CORS request that doesn’t need to go through any preflight requests (initial checks) is referred to as a simple request.
AJAX request is started by a browser tab that is open to https://www.mydomain.com. GET widgets at https://api.mydomain.com
The browser automatically inserts the Origin Request Header along with headers like Host for cross-origin requests
The Origin request header is examined by the server. The Access-Control-Allow-Origin is set to the value in the request header Origin if the Origin value is allowed.
The Access-Control-Allow-Origin header is checked by the browser to verify if it matches the origin of the tab when it receives the answer. If not, the reply is suppressed. If the Access-Control-Allow-Origin either contains the wildcard * operator or exactly matches the single origin, the check succeeds, as in this example.

Preflight requests:
The other kind of CORS request is a preflight. A preflight request is a CORS request in which the browser must first make a preflight request (also known as a preliminary probe) to the server to determine whether the original CORS request can be processed. It is an additional HTTP request using the OPTIONS method. The browser performs this for every unsafe request that is intended to alter data — for example, POST, PUT, or DELETE requests.

We refer to the original CORS request as preflighted because it was preceded by a preflight request.

Common circumstances necessitating preflighted requests:

The Content-Type header for a website’s AJAX call to POST JSON data to a REST API is application/json.

A website makes an AJAX call to an API and includes a token in the request header to authenticate the API. Authorization
This means that a REST API frequently has the bulk of AJAX requests preflighted when it powers a single-page application.

Simple requests vs. pre-flighted requests:
Simple requests are defined as those that do not result in a CORS preflight. However, after the request has been determined to be a basic request, some requirements must be met. These circumstances are:

The request’s HTTP method needs to be one of the following: head, Post, or Get
Except for the headers that are automatically set by the user agent, the request headers should only contain CORS safe-listed headers like Accept, Accept-Language, Content-Language, and Content-Type.
Only one of these three values, total, should be present in the Content-Type header: Multipart/form-data, application/x-www-form-URLencoded, or text/plain
The XMLHttpRequest returned object has no registered event listeners.
XMLHttpRequest must use the upload attribute.
There shouldn’t be any Readable Stream objects in the request.

The request is regarded as a pre-flighted request if either of these requirements is not met. For these requests, the browser must first submit an OPTIONS method request to the alternative origin.

This is used to determine whether the actual request can be sent to the server without risk. The response headers to the pre-flighted request determine whether the real request is approved or denied. The request is not sent if the main request’s headers and these response headers conflict.

Enabling CORS:
Suppose we faced the CORS error. There are many ways we could solve this issues depending on our access to the server on which the resources are hosted. There are two ways:

Access to the backend server
Access to only the frontend

Access to the backend:
You may set up the server to respond with the proper headers to enable resource sharing across various origins because CORS is essentially an HTTP header-based method. Look at the CORS headers we covered previously, and adjust the headers as necessary.

For example in node.js+express.js application:

The default configuration will be applied if a CORS configuration object is not given, which is identical to:

The following describes how to set up CORS on your server such that it will only accept GET requests from https://example.com with the headers Content-Type and Authorization and a preflight cache time of 10 minutes:

Access to the front end:
If you don’t have access to the backend server like a public API. Due to this, you cannot add headers to the response received. So in that case you could use a proxy server that’ll add the CORS headers to the proxied request. The proxy server is available at https://cors-anywhere.herokuapp.com/. You can also build your own proxy server.

Instead of directly making the request to the server, simply append the proxy server’s URL to the start of the API’s URL.

Common Pitfalls:

Using the Access-Control-Allow-Origin (all) operator:
The same-origin criterion is overridden by CORS in order to ensure security. While utilising *, the majority of CORS’ security rules are disabled. Wildcard is permissible in some circumstances, such as with an open API that is integrated into various websites hosted by third parties. The security may be enhanced by giving the API its own domain. For instance, your open API https://api.example.com may respond with Access-Control-Allow-Origin while your main website’s API https://www.example.com/api may still respond with Access-Control-Allow-Origin: https://www.example.com.
Multiple domains are returned for the Access-Control-Allow-Origin header:
Unfortunately, the standard does not allow the Access-Control-Allow-Origin: https:// example.com, https://www.example.com. Although the server can only return one domain or *, the Origin request header can be used.
Choosing a wildcard, such as *.example.com:
This is not covered by the CORS specification because wildcard can only be used to imply that all domains are permitted.
Not including non-standard or protocol ports:
Access-Control-Allow-Origin: example.com is invalid since the protocol is absent.
Similar to this, you will encounter problems with Access-Control-Allow-Origin: localhost unless the server is actually running on a normal HTTP port like: 80.
Ignoring the Origin header in the Vary response:
Most CORS frameworks do this automatically, but you still need to let clients know that server responses differ based on where the request came from.
Failure to define Access-Control-Expose-Headers:
If a required header is absent, the CORS request will still be successful, but any response headers that are not whitelisted will be obscured in the browser tab. The following common response headers are always made available for CORS requests:
Cache-Control, Content-Type, Language, Expires, and Last-Modified
When Access-Control-Allow-Credentials is set to true, using a wildcard:
Many people become entangled in this complex subject. The wildcard operator cannot be used on any of the response headers, including Access-Control-Allow-Origin, if the response includes Access-Control-Allow-Credentials: true.