Forem: Solomon Eseme

How To Reduce Page Load Time In JavaScript

Solomon Eseme — Mon, 10 Jul 2023 12:13:32 +0000

Page speed is a significant challenge in modern web applications. It is an important part of any user’s experience.

According to Forbes, 40% of people bounce from a website that takes more than 3 seconds to load. It is also important to note that a website that achiеvеs a loading timе of just 1 sеcond еxpеriеncеs a convеrsion ratе 5x highеr comparеd to a wеbsitе that takеs 10 sеconds to load.

This shows that users care more about page speed than the new shining design added to the website. To understand how your users will interact with your web pages, you need to further test websites under different network conditions, different screen sizes, browsers, and platforms.

This information will expose how users will interact with your webpage, and it will also show the UI components that need improvement even before moving your webpage to the production environment.

This blog will explore how to reduce page load time in JavaScript and the tips you can implement to increase your page speed.

What is Page Load Time?

Page load time is the average time it takes for a page to appear on your screen. It is calculated from when the page link is clicked or loaded to when it is completed. In enterprise applications, reducing the amount of time it takes for a page to load is an important investment, as we have seen in examples of how it could minimize conversion rate and affect business revenue.

*Here’s 295+ Selenium Interview Questions with Answers that will help you boost your confidence in an Interview. *

Why Reduce Page Load Time in JavaScript?

As we have discussed above, there are financial and other implications, such as a drop in traffic, drop in conversions, or increased bounce rate, etc., for not reducing the page load time. Below are some benefits why you should reduce the page load time of your software application.

Improve user experience: Faster page loading times lead to improved user experience, as users can find the information they need quickly and easily.
Increase conversion rates: When users can load a page quickly, they are more likely to stay on the page and complete the desired action.
Improve SEO rankings: From Search Engine Journals, search engine algorithms take page loading times into account when ranking websites, so faster page loading times can help improve SEO rankings.
Reduce server load: Faster page loading times can reduce the load on the server, leading to improved performance and reliability.
Reduce bounce rates: Faster page loading times can reduce bounce rates, as users are more likely to stay.

When a website takes a long time to load, users may be quick to abandon it and switch to a faster-performing one. On the other hand, a website with a fast loading speed will give the impression of being user-friendly and appealing, thus increasing the potential for more website traffic.

People who come to the website would likely enjoy looking at other content as well. The more people who look at the website’s content, the lower the bounce rate will be. The bounce rate is the percentage of people who leave the website after viewing only one page. The lower the bounce rate, the higher the quality of the website, both in terms of content and performance.

In the next section of this tutorial on how to reduce page load time in JavaScript, we will explore the different strategies you can implement to reduce the page load time of your software application.

How to Reduce Page Load Time in JavaScript?

Different strategies can be used to reduce page load time. In this section on how to reduce page load time in JavaScript, we will elucidate some strategies you can use to reduce page load time.

Minimize HTTP Requests

One of the major causes of slow page load time is the number of HTTP requests the page has to complete before painting the page to the screen.

The larger the number of requests, the more the page loads slowly, especially if the requests contain large data to load, such as large images, videos, or HTTP payloads.

Therefore, to reduce your page load time, you need to inspect the different HTTP requests and the number of these requests your page is loading and reduce it.

The most effective way to reduce the number of requests is to combine all JavaScript resources into one, and you can do that with different applications, libraries, and online tools. However, we will demonstrate how to automate the building process using Apache Ant.

In Ant, the concat task can be used to join multiple files together. It is important to remember that the JavaScript files must be combined in the correct order to ensure that any dependencies are respected. To specify the order in which the files should be combined, you can use either a filelist or a combination of fileset elements.

*If you are preparing for a Playwright automation interview, this list of Playwright interview questions can help you get a list of the most asked questions with detailed answers. *

Here is an example of how this could be done.

<target name="js.concatenate">
    <concat destfile="${bld.dir}/result.js">
    <filelist dir="${src.dir}"
        files="script1.js, script2.js"/>
    <fileset dir="${src.dir}"
        includes=".js"
        excludes="script1.js, script2.js"/>
    </concat>
</target>

Ant is instructed by this code to generate a file called result.js in the build directory. The contents of this file will be the combination of script1.js, then script2.js, and then all the files in the src directory in alphabetical order.

Enable Compression

You can use HTTP transfer encodings such as brotli or gzip to compress JavaScript assets in transmission when you enable compression in your request and response to the live cycle.

This process allows efficient transmission of files and therefore results in a faster page load time. Generally, the compression ratio can reach 75%, that is, KB; the compression size is only 25 kB.

Using a Content Delivery Network

Content Delivery Network (CDN) is a great way to reduce the latency of your users and deliver content to them faster because the content will be served from a location closer to them.

In addition, distributing static assets with a Content Delivery Network (CDN) can serve website visitors from a place closer to them geographically and also cache the content for the user during revisit.

Below are some of the CDN you can choose from to host your static files.

Akamai
Cloudflare
Amazon CloudFront
Microsoft Azure CDN
Google Cloud CDN
Fastly
Level 3
Limelight Networks
CacheFly
EdgeCast

*Here’s Top 30+ WebdriverIO Interview Questions and Answers that will help you boost your confidence in an Interview. *

Minify Resources

Minifying resources is another important strategy used to reduce the code size of your application without compromising any of its functionality. You can use different modification tools to minify the code, such as Closure Compiler, JSMin, or Uglify.

To minify a large JavaScript file on the go, you can install the uglify library globally using the command below:

npm install uglify-js -g

Here’s a screenshot of a successful installation.

Next, you can use this command below to minify your JavaScript files as shown below:

uglifyjs jquery-3.2.1.js — output jquery-3.2.1.min.js

The original file size before minifying with the Uglify library.

The current file size after minifying with the Uglify library. UglifyJS default settings determine the name of the input file and the output file. Already, there is a significant decrease in size; the original file was 268 KB, but after running it through UglifyJS, it is now only 136 KB.

Optimize Images

With more than 3.2 billion photos uploaded to the Internet daily, according to Photodoto, it is clear that images occupy the majority of your web content and, therefore, should be properly optimized to improve the load time of your web pages.

You can optimize your images in different ways. However, below are some other ways you can optimize your web images.

Use the correct file format: JPEG for photographs, PNG for graphics and illustrations, and GIF for simple graphics and animations.
Resize images to the correct dimensions: The larger the image, the more data it contains and the longer it takes to load.
Compress images: Compressing images reduces the file size without sacrificing quality. Tools like TinyPNG let you reduce the size of images without losing their quality.
Also, to customize the size of the image with CSS and media queries, it is recommended to use a tool to reduce the image rather than doing it in real time with CSS.
Use a content delivery network (CDN): A CDN is a server network that delivers images to users based on their geographic location.

Reduce Redirects

Reducing redirects on a web page can reduce page load time by eliminating the extra time it takes for the browser to make multiple requests to the server.

When a browser requests a page, it must first request the server, then wait for a response. If the response from the server is to redirect to another website, the browser must then make another request to the new page and wait for a response.

This process takes time, and if there are multiple redirects, the page load time can be significantly increased. By reducing redirects, the browser can make fewer requests and receive responses faster, resulting in a faster page load time.

*Prepare to ace your interviews with our thorough set of Jest interview questions and solutions that will enable you to prove your command of the Jest testing framework. *

Leverage Browser Caching

Leveraging browser caching is a technique used to reduce page load time by allowing the browser to store certain web page resources locally.

When a user visits a website, the browser will download the resources needed to render the page, such as images, CSS, and JavaScript.

By leveraging browser caching, the browser will store these resources locally. When the user revisits the same page, the browser will not have to download the resources again, thus reducing the page load time.

For example, the browser will fetch the HTML page from the web server but consult its cache for static assets (JavaScript, CSS, images) and return the content of the cache if found. If not, it will return the server’s content.

Here’s an image that illustrates the concept of browser caching.

The importance of browser caching is that it reduces the amount of data that is returned from the server, thereby making the request respond faster.

You can look at the example of a website with Caching and No Caching to see the amount of data transferred.

Without caching, we notice a very high amount of data transferred from the server to the browser causing the web page to slow a little.

The above image shows that the load time has drastically reduced because of browser caching by reducing the amount of data that is loaded from the server rather than the cached data.

In Chrome, you can view the contents of the cache. This will display a page of links to a detailed view of each cached file.

In the next section of this tutorial on how to reduce page load time in JavaScript, we will explore how to test website speed on different devices. In addition, we will indicate some pointers to help you improve your website page speed.

Importance of Testing Websites for Speed Optimization

In this section on how to reduce page load time in JavaScript, we will explore the role of website testing for page speed optimization.

Developing websites is not easy and requires a lot of effort from the development teams; however, what use is a website that does not solve a user’s problem and generates revenue as fast as possible? This makes developers expend effort and time optimizing slow and unfriendly websites, making it a top priority if the website is slow and losing users.

Nevertheless, developers need a way to test the speed of a website before deploying to production, or instead of waiting for the traffic to fall and users to start losing interest because of the speed problems, using a tool to test the website before deploying is an ideal solution.

However, it is general knowledge that a website can behave in a certain manner depending on the network, geo-location, or device because of the device configurations. Therefore, if you want to see real results, you need to test your website with a tool that uses real devices.

Several tools are available at your disposal to test the speed of your website.

In this tutorial on how to reduce page load time in JavaScript, we will use Google PageSpeed and a mobile-friendly tester tool by LambdaTest called LT Browser 2.0.

Using Google PageSpeed

Google PageSpeed allows you to test the speed of your website for both desktop and mobile devices. It gives you the overall percentages and lists all the issues that need to be fixed.

Visit Google PageSpeed, type in the website URL you want to test, and wait for the results to appear.

In this example, we are checking the performance of the LambdaTest website, and below are the results and some of the
issues we can fix to improve the performance and usability of the website. In this image below, you can experience how your users are experiencing and how the pages of your website are being delivered to them.

Next, we will look at the “Diagnose performance issues” and the percentages for each category.

From the screenshot above, we can see a 97% performance, with all the metrics that optimize page load time. These metrics are part of the Core Web Vitals, and Core Web Vitals are specific factors that Google considers important in a webpage’s overall user experience.

First content paint: This indicatеs thе momеnt whеn thе first tеxt or imagе is displayеd on thе pagе.
Speed index: Thе spееd indеx mеasurеs thе spееd at which thе pagе’s contеnt bеcomеs visiblе to usеrs.
Largest contentful paint: Thе largеst contеntful paint rеprеsеnts thе point in timе whеn thе large tеxt or imagе is rеndеrеd on thе pagе.
Time to interactive: It rеfеrs to thе duration rеquirеd for thе pagе to bеcomе fully intеractivе.
Total blocking time: Sum of all time periods between First Content Paint (FCP) and Time to Interactive (TTI) when task length exceeds 50ms, expressed in milliseconds.
Cumulative Layout Shift: This measures the movement of visible elements within the viewport.

These metrics and more are the ways your website performance and speed are measured. In addition, Google PageSpeed also lists suggestions on how you can further improve your page speed and performance, amongst other diagnoses.

The above screenshot shows that the static assets do not have an efficient cache policy. Therefore causing the resource to be downloaded every time and increasing latency.

Additionally, depending on your website, you might see different results and diagnostic issues you can fix to improve your website performance.

Also, Google PageSpeed ‘s a free tool to use quickly when running website checks; it does not allow you to simulate network latencies or select different devices to test with. That’s why using a mobile responsive test tool like LT Browser 2.0 will enable you to set all these parameters and is a good choice for understanding your user’s real experience based on different configurations.

This questionnaire of Jasmine Interview questions will aid you in giving good insight about Jasmine and how to crack the interview.

Using LT Browser 2.0

LT Browser 2.0 is the next-gen browser to build, test, and debug mobile websites. It allows you to test your website on different mobile device viewports. Additionally, you can create and test your websites on your custom device resolution and viewport and save it for future use. LT Browser 2.0 supports all kinds of platforms and devices ranging from desktops, mobile, tablets, iPhones, etc.

In addition, LT Browser 2.0 allows you to test your websites against different configurations, such as setting network throttling and performing a full lighthouse check on your website based on these configurations.

The first step is to download and install LT Browser on your machine. You can follow the documentation to get started with LT Browser.

After downloading and installing, open the LT Browser and enter the URL of your website, select your favorite screen size, and start debugging.

I have selected Desktop and typed in LambdaTest website for my case. Next, we will generate Lighthouse reports.

Click on the Performance Report tab to see the performance of your website based on the device size, network throttling, etc., you have selected.

After a few minutes, the results came in. From the screenshot below, you can see a performance reduction based on my browser configuration, network throttling settings, and Internet provider.

In addition, you can also see the metrics that need improvements and where the website is lacking speed.

If there are any issues, the report will show them and list the opportunities to improve the website’s performance. Your development team can go through the list and fix most of the bugs listed below for better performance on a slow network.

Subscribe to the LambdaTest YouTube Channel and stay up to date with the latest tutorials around Selenium testing, Cypress testing, and more.

Summary

In this tutorial on how to reduce page load time in JavaScript, we explored how reducing page load time in JavaScript can be achieved by minifying code, using caching, optimizing images, and reducing the number of HTTP requests.

Additionally, using a content delivery network (CDN) improves page load times by reducing latency and improving performance. By following these steps, developers can ensure that their web pages load quickly and efficiently.

Prepare to ace your NUnit interviews questions with our thorough set of solutions that will enable you to prove your command of the NUnit testing framework.

Frequently Asked Questions (FAQs)

How do I reduce page loading time?

Providing a better user experience and improving website performance depends on reducing page load time. To decrease page loading time, you can follow the below strategies:

Optimize and compress images
Minify CSS and JavaScript
Enable browser caching
Use Content Delivery Networks (CDNs)
Reduce HTTP requests

How can I make JavaScript load faster?

To make JavaScript load faster, you can implement several techniques, including minifying and compressing JavaScript, using asynchronous loading, combining JavaScript files, using caching, and others.

How to use Date Picker in Selenium using JavaScript

Solomon Eseme — Wed, 17 May 2023 12:50:42 +0000

Selenium is widely recognized as one of the top-performing web automation tools. However, its potential extends beyond testing, with numerous use cases leveraging its capabilities. One such application involves automating the process of completing a form within a browser.

In this Selenium JavaScript tutorial, we will cover how to use Date Picker in Selenium using JavaScript when filling out a form. We will discuss the best pattern and approach to achieve the solution and explore how we solved this problem in our application.

So, let’s get started.

In this tutorial, learn what is Regression testing, its importance, types, and how to perform it: https://www.lambdatest.com/learning-hub/regression-testing

What is Selenium?

Selenium is a popular open-source test automation framework used for automated browser testing. This framework helps you write Selenium test scripts that can be used to automate the testing of websites and web applications, then execute them in different browsers on multiple platforms using any programming language of your choice.

Date Picker in Selenium

If your website requires users to fill out their date of birth or any date-related fields and your web page is using any of these date picker components available.

You might want to test out the approach it will take your users to fill out the form or possibly automate the process of filling web forms that require a Date Picker in any website using Selenium with JavaScript.

How do you achieve this using the Selenium automation tool? I had this problem testing out my web page, and I will share with you how to automate testing with Selenium JavaScript.

First, let’s build a web page that contains a Date Picker.

Creating the Webpage with Date Picker

To learn how to automate Date Picker in Selenium using JavaScript, we’ll use jQuery’s Date Picker demo URL as the test URL to demonstrate the same.

In this Selenium JavaScript testing tutorial, I’ll be using JavaScript to automate the Date Picker using Selenium WebDriver

Below is a sample of the form we have created for this demo:

Next, we are going to set up Selenium and automate the process of filling out this form. The process is seamless and very simple to get started.

Setting up Selenium with JavaScript

Installing Selenium in your JavaScript project requires some installations on your local machine. Here are some prerequisites you need to have:

Download and install NodeJS from official NodeJS documentation. You should have NodeJS v6 or newer.
Make sure you are using the latest version of JavaScript.
Install npm from th*e official npm website.*
Download Selenium JavaScript bindings from the official website. The latest versions of Selenium Client and WebDriver are ideal for running your JavaScript automation testing script.

Set up Selenium with Express

If you have successfully installed and set up the following prerequisites, you’re ready to move to the next step. Next, create a project folder for your Selenium Date Picker project. We will call it “selenium-date-picker”

Step 1: Install the required dependencies.

Below are the required dependencies to install:

npm install express selenium-webdriver

**Step 2: **Create an “index.js” file that will add the configuration for Selenium for using the Date Picker in Selenium using JavaScript.

Here’s the code snippet for setting up Selenium in the Express framework:

const express = require("express");
const { By, Builder, Select } = require("selenium-webdriver");
const app = express();
const port = 3002;
let driver;
const URL = "https://jqueryui.com/datepicker/#date-range";
const expected_fr_date = "20";
const expected_to_date = "26";
app.get("/", async (request, response) => {
 try {
   const data = await DatePickerWithLambdaTest();
   response.status(200).json(data);
 } catch (error) {
   console.log(error);
   response.status(500).json({
     message: "Server error occurred",
   });
 }
});
app.listen(port, () => {
 console.log(`Example app listening at http://localhost:${port}`);
});

The above code snippet shows a simple web server created with Express.js to request from a Date Picker in Selenium using JavaScript.

Code Walkthrough:

Let’s walk through the code together and understand the nitty-gritty of it.

*Step 1: **Add the required packages and set up the Express server.
First, we need the *express *and *selenium-webdriver packages. Next, we create an Express server and initialize variables which will be used below.

Step 2: Create Express Route.

Next, we create an endpoint to fill out the Date Picker in Selenium using JavaScript and display the result as JSON data. This endpoint will call the DatePickerWithLambdaTest() function to fill out the date picker and return the data.

Test native, hybrid, and web apps on any mobile OS with our free Android emulator online. Sign up to optimize app performance.

How to use Date Picker in Selenium using JavaScript?

After setting up the Express server, we will create the two functions to fill out the Date Picker and return the data.

Here’s the code snippet:

async function DatePickerWithLocal() {
 try {
   const driver = await new Builder().forBrowser("chrome").build();
   return selectDatePicker(driver);
 } catch (error) {
   throw error;
 } finally {
   await driver.quit();
 }
}
async function DatePickerWithLambdaTest() {
 const USERNAME = ""; //replace with your username
 const KEY = ""; //replace with your accesskey
 const GRID_HOST = "hub.lambdatest.com/wd/hub";
 const capabilities = {
   browserName: "Chrome",
   browserVersion: "107.0",
   "LT:Options": {
     username: USERNAME,
     accessKey: KEY,
     geoLocation: "US",
     platformName: "Windows 10",
     build: "Date Picker",
     project: "Date Picker Example",
     w3c: true,
     plugin: "node_js-node_js",
   },
 };


 const gridUrl = "https://" + USERNAME + ":" + KEY + "@" + GRID_HOST;
 try {
   driver = await new Builder()
     .usingServer(gridUrl)
     .withCapabilities(capabilities)
     .build();


   return await selectDatePicker(driver);
 } catch (error) {
   throw error;
 } finally {
   await driver.quit();
 }
}


async function selectDatePicker(driver) {
 try {
   if (!driver) return;
   await driver.get(URL);
   /**
    * Switching Frames with iframe
    */
   const frame = await driver.findElement(
     By.xpath("//*[@id='content']/iframe")
   );
   await driver.switchTo().frame(frame);


   /**
    * ################################# Steps for the From Date ####################
    */
   const from_date = await driver.findElement(By.xpath('//*[@id="from"]'));
   await from_date.click();


   const from_month = await driver.findElement(
     By.className("ui-datepicker-month")
   );
   const selected_from_month = new Select(from_month);
   await selected_from_month.selectByVisibleText("Jan");
   const from_day = await driver.findElement(
     By.xpath(
       "//td[not(contains(@class,'ui-datepicker-month'))]/a[text()='" +
         expected_fr_date +
         "']"
     )
   );
   await from_day.click();


   /**
    * ################################# Steps for the To Date #####################
    * The same steps like the ones in From Month are repeated except that now the operations are performed on a different web element.
    *   
    */
   const to_date = await driver.findElement(By.xpath("//input[@id='to']"));
   await to_date.click();


   const to_month = await driver.findElement(
     By.xpath("//div/select[@class='ui-datepicker-month']")
   );
   const selected_to_month = new Select(to_month);
   await selected_to_month.selectByVisibleText("Feb");
   const to_day = await driver.findElement(
     By.xpath(
       "//td[not(contains(@class,'ui-datepicker-month'))]/a[text()='" +
         expected_to_date +
         "']"
     )
   );
   await to_day.click();


   /**
    * Get Date values
    */
   const selected_from_date_str = await from_date.getAttribute("value");
   const selected_to_date_str = await to_date.getAttribute("value");
   return {
     from: selected_from_date_str,
     to: selected_to_date_str,
   };
 } catch (error) {
   throw error;
 }
}

The code snippet above is the complete section of the index.js that contains the Express server and the routing configuration.

Code Walkthrough:

Let’s walk through the code together and understand the nitty-gritty of using Date Picker in Selenium using JavaScript.

Step 1: Selenium with local setup.

The code snippet in the DatePickerWithLocal shows how to use locally installed Selenium to perform the Date Picker operation. This method runs on your local machine and requires installing the Selenium driver and setting up the correct browser configuration.
We configured and selected a specific browser we wanted to perform the task with and passed the new driver instance to the ‘selectDatePicker‘ function to operate.

You can learn more about configuring and installing Selenium and its browser driver locally by performing web scraping with Selenium JavaScript.

Step 2: Selenium with LambdaTest Grid

The code snippet in the DatePickerWithLamdaTest shows how to use LamdaTest Grid to perform the Date Picker operation with Selenium on the cloud.

JavaScript automation on a cloud-based testing digital experience testing platform like LambdaTest helps you achieve improved browser coverage while performing automation testing on over 3000+ browsers and OS combinations.

By running Selenium JavaScript tests in parallel using LambdaTest cloud-based infrastructure, you can speed up your release cycle significantly, with the potential to increase the overall test execution pace by up to 10 times.

Please subscribe to the LambdaTest YouTube Channel and stay updated with the latest tutorials on Selenium testing, Cypress testing, CI/CD, and more.

No installations are required, only a few configurations, as shown below:

Before running the operation, please set the following environment variables in your .env file. You can find the username and access key on your LambdaTest Profile page.

bash
LT_USERNAME=
LT_ACCESS_KEY=
GRID_HOST=hub.lambdatest.com/wd/hub

Next, generate your Selenium configuration using the Lambdatest Capabilities Generator. Set your configuration and copy the JavaScript object to your code, as shown below.

Next, we created a driver using the configuration and capacity before passing the driver to the ‘selectDatePicker‘ function.

Step 2: Performing the Operation.

Lastly, we call the selectDatePicker function to operate. Below is a breakdown of the function.

First, we check the driver instance to make sure it’s created. Next, we open the webpage where the date picker is located using the ‘driver.get(URL)’ function. Lastly, we find the first iframe on the page and switch to it using the ‘driver.switchTo().frame()’ function.

Furthermore, still inside the function, we started the first operation for the From date by interacting with the input element using Selenium functions as shown below:

The expected_fr_str comes from the constants we declared above to specify the specific date we want to select from the calendar.

Next, we repeat the same thing for the to-date instance as shown below:

The expected_to_str comes from the constants we declared above to specify the specific date we want to select from the calendar.

Lastly, we retrieve the chosen date from the input field and return it as an object.

If you configure everything correctly, you should see a JSON response from your server when you run it on your browser.

Running the Date Picker

To run your test for Date Picker in Selenium using JavaScript, type the following command into your root terminal.

bash
npm start

Open your browser and locate the web address. After successfully performing running the Date Picker, you should be greeted with a JSON response from your server, like the screenshot below:

So far, we have successfully used a Date Picker with Selenium using JavaScript. You can clone the entire code repository from this GitHub Repository.

If you are a developer or a tester, this Selenium JavaScript 101 certification can help equip you with advanced skills and extensive knowledge in automation testing, enabling you to excel in any JavaScript automation role.

Perform browser automation testing on the most powerful cloud infrastructure. Leverage LambdaTest automation testing for faster, reliable and scalable experience on cloud.

Summary

Selenium is one of the most popular browser automation tools. Therefore achieving simple tasks such as automating the process of filling out a Date Picker can be daunting.

This article covers in detail how to use Date Picker in Selenium using JavaScript when filling out a form. We discussed the best pattern and approach to achieve the solution and finally explored how we solved this problem in our application.

Frequently Asked Questions (FAQs)

1. How to automate Date Picker in JavaScript?

Automating a date picker in JavaScript can be achieved using the following steps:

Locate the date picker element on the web page using the DOM.
Use the .click() method to trigger the date picker to open.
Use JavaScript to set the desired date in the date picker element. Depending on the date picker plugin used, you can directly set the date value using the value attribute or similar property. Alternatively, you may need to simulate user actions such as clicking the previous/next month buttons, selecting a date from a calendar, etc.

2. How to set Date Picker in Selenium?

To set a date picker in Selenium, you can follow the following steps:

Locate the date picker element on the web page using any of the Selenium locators like id, name, class, xpath, etc.
Use the .click() method to open the date picker widget.
Depending on the date picker widget used, you can either directly set the date value using the sendKeys() method, or you may need to simulate user actions like clicking on the previous/next month buttons, selecting a date from a calendar, etc.

How To Perform Web Scraping With JavaScript And Selenium

Solomon Eseme — Thu, 24 Nov 2022 12:50:51 +0000

Web scraping with JavaScript is a very useful technique to extract data from the Internet for presentation or analysis. However, in this era of dynamic websites, it becomes difficult to accurately extract data from the web because of the ever-changing nature of data.

In the past, we used libraries such as urllib or requests to read data from webpages, but things started falling apart with dynamic websites. For websites that are not static, using a common ajax call or requests will simply not work.

Selenium framework was introduced for such automation to help in the aspect of dynamic data and other important features.

When we were developing an automated job portal for backend engineers, we needed a reliable web scraping framework to help us scrape over 20 job portals and return over a thousand jobs each, and Selenium was the answer.

But before we delve into what is Selenium and perform web scraping with JavaScript and Selenium, let’s explore what web scraping is all about?

What is Web Scraping?

Web scraping is the process of extracting data from the Internet for storage in different formats or for further processing and analysis. It can also be referred to as web harvesting or web data extraction.

By Web scraping with JavaScript and Selenium, we basically automates the manual process of extracting data from the Internet and storing it for further processing.

It can be useful to extract data for different purposes such as

Academic or business Research
Historical Data Analysis
Price comparisons
SEO
Brand protection
Ad verification
and many other use cases

You can use web scraping to generate product reviews from review websites to help you decide to buy a particular product.

You can also use web scraping to scrape thousands of job portals for specific jobs you’re interested in applying for.

Benefits of Web Scraping with JavaScript

Below are a few benefits of web scraping:

Extracting product details of e-commerce websites such as prices, product names, images, etc.
Web scraping is very useful in research as it can help to gather structured data from multiple websites.
Gathering data from different sources for analysis can be automated with web scraping easily.
It can be used to collect data for testing and training machine learning models.

The benefit of web scraping is endless and introduces different thoughtful insights and possibilities to businesses.

Where to check for data protection?

In general, web scraping is not illegal, but there are some caveats to it, and to some extent, it has become illegal.

It’s always very important to be careful and check out some of the different terms you should consider when web scraping to determine if your actions are illegal or not.

Copyright Infringement
Every website should have a document stating whether the website’s content can be web scraped or not. You can check the copyright document to gain more insights into the possibilities provided by the company on web scraping.
Robot.txt
The robot.txt file also provides strict information on web scraping, and you should adhere to this file’s information or legal stand. If it says no web scraping, then it should be respected.
Using API
If a business provides a public API that users can access the data in the business, then you should adhere to using the API instead of web scraping it to avoid legal action.
Terms of Services
Terms of service is a good place to find all the information on how the website’s content should be used and accessed. It states the possibilities of web scraping the stand the website is taking on.
Scraping Public Content
Web scraping of public content is not a crime, but if the content owner forbids it — web scraping companies should comply.

Now that we understand our stand on the legality of web scraping, what methods are there to web scrape data either for a small use case or web scraping at scale? Next, let’s explore the different methods and tools available for web scraping.

Check this out: Complete WebDriver Tutorial- A Guide on Selenium WebDriver Automation

Web Scraping Tools

This is the most popular web scraping method where a business deploys an already made software for all their web scraping use cases.

If you want to access and gather data at scale, you need good web scraping tools that can surpass IP blocking, cloaking, and ReCaptcha. There are popular tools such as Scrapy, Beautiful Soup, Scrapebox, Scrapy Proxy Middleware, Octoparse, Parsehub, and Apify.

These tools help you with your web scraping task at scale and can surpass different obstacles to help you achieve your goals.

Selenium is a popular open-source web automation framework used for automated browser testing. This framework helps you write Selenium test scripts that can be used to automate testing of websites and web applications, then execute them in different browsers on multiple platforms using any programming language of your choice. However, it can be adapted to solve dynamic web scraping problems, as we will demonstrate in the blog on how you can do web scraping using JavaScript and Selenium.

Selenium has three major components:

Selenium IDE: It is a browser plugin — a faster, easier way to create, execute, and debug your Selenium scripts.
Selenium WebDriver: It is a set of portable APIs that help you write automated tests in any language that runs on top of your browser.
Selenium Grid: It automates the process of distributing and scaling tests across multiple browsers, operating systems, and platforms.

When writing this blog on web scraping with JavaScript and Selenium, the latest version of Selenium is Selenium 4. If you are intrigued to know more about what is new in Selenium 4, you can go through this video:

Check this out: Selenium Testing Automation Cloud — Test on Selenium Grid Cloud of 3000+ Desktop & Mobile Browsers.

Benefits of using Selenium with JavaScript for Web Scraping

The benefits of using Selenium are limitless and can be used in different aspects of web scraping with JavaScript — from scraping static to dynamic websites. Below are a few benefits of using Selenium WebDriver for your next web scraping project.

The Selenium WebDriver can be configured to scrape a website on browsers whose browser drivers are available.
Selenium WebDriver to scrape complicated websites with dynamic contents since some needed data requires HTTP requests.
Perform automated screenshot testing by taking numerous screenshots of the web pages as you scrape the web pages.
With the Selenium WebDriver, you can simulate a real user working with a browser by loading all resources and downloading browser cookies.

In this Selenium JavaScript tutorial, we will learn how to perform web scraping with JavaScript and Selenium on our local machine and also on LambdaTest cloud Selenium Grid.

We will be web scraping the LambdaTest YouTube channel, displaying the list of videos on a webpage using Selenium, and converting the information to JSON data.

If you are new to this channel, please subscribe to the LambdaTest YouTube Channel and stay updated with the latest tutorials on Selenium testing, Cypress testing, CI/CD, and more.

Accelerate JavaScript Automation with Parallel testing. Try LambdaTest Now!

Implementing Web Scraping with JavaScript and Selenium

One of the benefits of Selenium is that it can web-scrape dynamic JavaScript pages where there are dynamic interactions such as hovering over menu items. In this section of this blog on web scraping with JavaScript and Selenium, we will demonstrate how to web scrape YouTube videos and display the data as JSON using Selenium and JavaScript.

Selenium encapsulates the complexity of building an automation tool for dynamic pages and makes it easy by simplifying installing and configuring the package.

Check this out: Selenium Testing tool Automation Cloud — Test on Selenium Grid Cloud of 3000+ Desktop & Mobile Browsers.

Prerequisites

Before you continue below, you should be familiar with Node.js and Express.js — a Node.js framework. If you are new to Express testing, you can learn more about it through this guide on getting started quickly with Express testing.

Installation

We will continue by installing the necessary packages and software to implement web scraping with JavaScript and Selenium.

Visual Studio Code: This is the code editor we will use in this demo, though you can go with any code editor of your choice. You could use any other IDE of your choice.
Browser Driver: To run the Selenium testing drivers on your machine, you need to download the drivers from the following locations according to the browser of your choice.

BROWSER	DOWNLOAD LOCATION
Opera	https://github.com/operasoftware/operachromiumdriver/releases
Firefox	https://github.com/mozilla/geckodriver/releases
Chrome	http://chromedriver.chromium.org/downloads
Internet Explorer	https://github.com/SeleniumHQ/selenium/wiki/InternetExplorerDriver
Microsoft Edge	https://blogs.windows.com/msedgedev/2015/07/23/bringing-automated-testing-to-microsoft-edge-through-webdriver/

After downloading any of the drivers, make sure to install it and set it up properly according to your operating system.

After successfully installing and configuring the browser driver, we will explore how to create a new project and start scraping YouTube videos in the next section of this blog on web scraping with JavaScript and Selenium.

Creating a Project

After the installation, we will create a new VSCode Node.js project and add the following libraries to our package.json file.

Ensure you have Node.js installed and configured properly in your operating system. You can download and install it from the Node.js official website.

Let’s install the following packages:

```bash
npm install express selenium-webdriver
```

3- After installing the libraries, let’s create an **index.js **file inside the root directory and set up our express server.

    ```bash
    touch index.js
    ```

Create the express server to listen to incoming requests:

    ```js
    const express = require('express')
    const { Builder, By } = require('selenium-webdriver');

    const app = express()
    const port = 3000
    app.get('/', (request, response) => {
     // Web Scraping Code here
    })
    app.listen(port, () => {
      console.log(`Example app listening at http://localhost:${port}`)
    })

4- The code snippet above creates a new Express server and listens to incoming requests from the user. In the next section of this blog on web scraping with JavaScript and Selenium, we will use the selenium-webdriver library we installed to web scrape our YouTube videos.

YouTube Web Scraping with JavaScript and Selenium

In this demonstration, we scrape the following data from the LambdaTest YouTube channel and display it as JSON:

Video Title
Video Views
Video Published Date

You can implement it using the cloud-based Selenium automation grid provided by LambdaTest, which will take away all the manual configuration both in development and in production.

LambdaTest is a cross browser testing platform that allows you to automate browsers with Selenium, Appium, and other frameworks. With LambdaTest’s Selenium Grid, you can run Selenium tests in parallel over an online browser farm of 3000+ browsers and platform configurations.

Below is a simple walkthrough of how the scraping is going to happen before delving into the implementation.

Open the YouTube channel videos page.
It scrolls until the end of the page to scrape all the videos available.
Retrieves the video title, views, and upload details using a loop.
Convert the scraped information to JSON.

Implementation:

const express = require('express');
const { Builder, By } = require('selenium-webdriver');

const app = express();
const port = 3000;
app.get('/', async (request, response) => {
 // Web Scraping Code here
 try {
   const data = await WebScrapingLocalTest();
   response.status(200).json(data);
 } catch (error) {
   response.status(500).json({
     message: 'Server error occurred',
   });
 }
});
app.listen(port, () => {
 console.log(`Example app listening at http://localhost:${port}`);
});

async function WebScrapingLocalTest() {
 try {
   driver = await new Builder().forBrowser('chrome').build();
   await driver.get('https://www.youtube.com/c/LambdaTest/videos');
   const allVideos = await driver.findElements(
     By.css('ytd-grid-video-renderer.style-scope.ytd-grid-renderer')
   );
   return await getVideos(allVideos);
 } catch (error) {
   throw new Error(error);
 } finally {
   await driver.quit();
 }
}

Code Walkthrough:

Let’s walk through the code together and understand the nitty-gritty of it.

Step 1: Add required packages

First, we require the express and selenium-webdriver packages, and next we created an Express server to handle the incoming request. The request is handled by calling the WebScrapingTest function.





```js

    app.get('/', async (request, response) => {
     // Web Scraping Code here
     try {
       const data = await WebScrapingLocalTest();
       response.status(200).json(data);
     } catch (error) {
       response.status(500).json({
         message: 'Server error occurred',
       });
     }
    });

    async function getVideos(videos) {
     let videoDetails = [];
     try {
       for (const video of videos) {
         const title = await video.findElement(By.id('video-title')).getText();
         const views = await video
           .findElement(By.xpath(".//*[@id='metadata-line']/span[1]"))
           .getText();
         const date = await video
           .findElement(By.xpath(".//*[@id='metadata-line']/span[2]"))
           .getText();
         videoDetails.push({
           title: title ?? '',
           views: views ?? '',
           publishedDate: date ?? '',
         });
       }
     } catch (error) {
       console.log(error);
     }
     return videoDetails;
    }

Step 2: Create an instance of Selenium WebDriver

Inside the openChromeTest function, we created an instance of the Builder class from the Selenium Webdriver package we imported earlier. We also pipe the forBrowser method used to indicate the browser to use for the web scraping and lastly, we call the build method, which creates a new WebDriver client based on this builder’s current configuration.

With the instance of the builder, we passed in the official LambdaTest YouTube videos URL in the get method of the builder class.

```js

    driver = await new Builder().forBrowser('chrome').build();
       await driver.get('https://www.youtube.com/c/LambdaTest/videos');

Step 3: Load all the YouTube Videos on the page
Next, we will load all the videos within the page and collect specific information using a loop in JavaScript.

To do that, we need to target a specific element in the YouTube Videos page that contains all the videos using the Chrome inspector tool, we can select any element within the webpage. Below is a screenshot showing the element and the CSS selector to use.

As of the time of writing, the class attribute to target is ytd-grid-video-renderer style-scope ytd-grid-renderer, which is a combination of the CSS class property of the parent container and the individual video container.

Next, we are going to convert the classes to CSS QuerySelector and target all the videos using the FindElements method in the Selenium WebDriver and By instance class.
Here is how it looks:

 const allVideos = await driver.findElements(
         By.css('ytd-grid-video-renderer.style-scope.ytd-grid-renderer')
       );

Selenium uses the By instance class to represent different query selectors, so you can select elements using Selenium locators like CSS name, ID, Class, etc.

Finally, the findElements method returns an object of all the selected HTML DOM Nodes called WebElement, which we will loop through to retrieve metadata of the videos while performing web scraping with JavaScript and Selenium.

Step 4: Converting to JSON

Lastly, we loop through the WebElement and retrieve metadata of the videos such as the video title, video views, and video published date.

 async function getVideos(videos) {
 let videoDetails = [];
 try {
   for (const video of videos) {
     const title = await video.findElement(By.id('video-title')).getText();
     const views = await video
       .findElement(By.xpath(".//*[@id='metadata-line']/span[1]"))
       .getText();
     const date = await video
       .findElement(By.xpath(".//*[@id='metadata-line']/span[2]"))
       .getText();
     videoDetails.push({
       title: title ?? '',
       views: views ?? '',
       publishedDate: date ?? '',
     });
   }
 } catch (error) {
   console.log(error);
 }
 return videoDetails;
}

In the code above, we are using both the XPath selectors to accomplish our task of retrieving basic video information. This screenshot below shows you how to retrieve the XPath of a particular element in a webpage.

Secondly, We are also using the ID selector to retrieve the video title. This screenshot shows where to get the ID for the video title element on YouTube using the Inspector tool:

Check this out: Complete WebDriver Tutorial- A Guide on Selenium WebDriver Automation

Display the JSON data

After successfully performing YouTube web scraping with JavaScript and Selenium, we convert them into JSON and send them as a response to our request using the Express server. Finally, we are ready to test the new web scrapper.

Start a new development server using the following command:
```
bash
    node index.js


    ```
```

2- Open the Chrome browser and visit localhost:3002, assuming you use the same port number from the demo.

3- You should be greeted with JSON data of the YouTube videos after a few seconds, like the screenshot below:

If you love testing as much as I do, you’re going to enjoy learning more about one of the most powerful tools I know of — Selenium WebDriver with JavaScript. This video tutorial covers everything you need to know.

Web Scraping with JavaScript and Selenium on cloud-based Selenium Grid

In the previous section, we performed web scraping with JavaScript and Selenium on our local machine. One of the shortcomings of this approach is that you need to install and configure the Browser drivers locally.

Assuming you need to use multiple browsers to perform that automation, you must also manually install and configure each browser. This is where a cloud-based grid can be super impactful, as you can perform JavaScript automation testing in parallel on various browser and platform combinations.

Cloud testing platforms like LambdaTest allow you to perform live testing and run parallel tests on 3000+ real browsers and their different versions. It provides you access to thousands of browsers for mobile and web testing to help you gain the maximum test coverage during the process of Selenium web automation.
Let’s look at how to run the Selenium automation test with JavaScript on LambdaTest:

Step 1: Let’s start by creating a free account on the LambdaTest platform. Once you log in, fetch your unique username and access key from the LambdaTest Profile Section of the website.

**Step 2: **Make the following configuration changes to your existing code:
Provide username and access key.

Provide username and access key.

const USERNAME = 'YOUR_USERNAME'; //replace with your username
const KEY = 'YOUR_KEY'; //replace with your accesskey
Provide the host corresponding to the LambdaTest platform.
```
const GRID_URL = 'hub.lambdatest.com/wd/hub';
```
Generate the desired browser capabilities using LambdaTest Capabilities Generator.



const capabilities = {
         build: 'Web Scraping with JavaScript and Selenium',
         name: 'Youtube Videos',
         platform: 'Windows 10',
         browserName: 'Chrome',
         version: '90.0',
         selenium_version: '3.13.0',
         'chrome.driver': '90.0',
        };

Provide the gridUrl. You can also find this value from the LambdaTest Dashboard.
```
const gridUrl = 'https://' + USERNAME + ':' + KEY + '@' + GRID_URL;
```

Finally, edit the code to build and instantiate the browser on the designated platform.

async function WebScrapingWithLambdaTest() {
 try {
   driver = await new Builder()
     .usingServer(gridUrl)
     .withCapabilities(capabilities)
     .build();
   await driver.get('https://www.youtube.com/c/LambdaTest/videos');
   const allVideos = await driver.findElements(
     By.css('ytd-grid-video-renderer.style-scope.ytd-grid-renderer')
   );
   return await getVideos(allVideos);
 } catch (error) {
   throw new Error(error);
 } finally {
   await driver.quit();
 }
}

Here is the final implementation of Selenium automation with JavaScript that will be run on LambdaTest’s cloud-based Selenium Grid:



app.get('/lambdatest', async (request, response) => {
         try {
           const data = await WebScrapingWithLambdaTest();
           response.status(200).json(data);
         } catch (error) {
           response.status(500).json({
             message: 'Server error occurred',
           });
         }
        });
        const USERNAME = 'YOUR_USERNAME'; //replace with your username
        const KEY = 'YOUR_KEY'; //replace with your accesskey
        const GRID_URL = 'hub.lambdatest.com/wd/hub';
        const capabilities = {
         build: 'JavaScript and Selenium Testing',
         name: 'Google search',
         platform: 'Windows 10',
         geoLocation: 'US',
         browserName: 'Chrome',
         version: '90.0',
         selenium_version: '3.13.0',
         'chrome.driver': '90.0',
        };

        const gridUrl = 'https://' + USERNAME + ':' + KEY + '@' + GRID_URL;

        async function WebScrapingWithLambdaTest() {
         try {
           driver = await new Builder()
             .usingServer(gridUrl)
             .withCapabilities(capabilities)
             .build();
           await driver.get('https://www.youtube.com/c/LambdaTest/videos');

           const allVideos = await driver.findElements(
             By.css('ytd-grid-video-renderer.style-scope.ytd-grid-renderer')
           );
           return await getVideos(allVideos);
         } catch (error) {
           throw new Error(error);
         } finally {
           await driver.quit();
         }
        }

Step 3: Finally, it’s time to run the test on the LambdaTest grid. Trigger the following command on the terminal to run the automation testing with Selenium JavaScript.





 ```js
        node index.js
        ```

Check this out: Selenium Automation Cloud - Test on Selenium Grid Cloud of 3000+ Desktop & Mobile Browsers.

Drawbacks of using Selenium for Web Scraping with JavaScript

In this blog, we have seen how to perform web scraping with JavaScript and Selenium. However, there are some drawbacks to using any particular technology (or tool/framework), and Selenium is not exempted. Below are some of the drawbacks of using Selenium for web scraping.

When using Selenium for web scraping, it can automatically download unnecessary files. For instance, if you’re only interested in the text data of a webpage, Selenium will also download the assets (i.e., CSS, HTML, JS, and other files) needed to render the web page.
It consumes time and resources when using Selenium WebDriver because it downloads unnecessary files to render the web page.
Web Scraping with Selenium is slow since it waits for the entire page to be loaded before you can start accessing data. To learn more about waits, you can read through this blog on types of waits in Selenium.

Watch this video to learn what are waits in Selenium and how to handle them using different methods like hard-coded pauses and by combining explicit waits with different design patterns.

So far, we have detailed a few drawbacks you need to look out for when web scraping with JavaScript and Selenium. Let’s also explore the limitless possibilities and benefits of using Selenium for web scraping.

If you’re a JavaScript developer serious about automation testing, this certification will give you the hard-core skills and in-depth knowledge needed to excel in any JavaScript automation role.

Here’s a short glimpse of the Selenium JavaScript 101 certification from LambdaTest:

Check this out: Online Safari Browser For Windows - Say No To Safari VM! Perform Cross Browser Compatibility Testing On All Safari Browser Versions Across Real Browsers And Operating Systems. https://www.lambdatest.com/safari-browser-for-windows

Summary

Web scraping is an important technique to extract data. It eliminates the manual process of extracting public data. Web scraping is also very accurate and less time-consuming. It can also be configured to retrieve dynamic websites using Selenium and JavaScript.

In this blog on web scraping with JavaScript and Selenium, we have explored the benefits of web scraping, which can not be overemphasized, the different web scraping tools, and Selenium which stands as the best tool for automated testing and dynamic web scraping tools.

We scraped all the content of the official LambdaTest YouTube channel and converted them into JSON, automatically extracting the titles and authors to generate an author matrix, among other useful insights.

If you are interested in learning more about web scraping using Selenium using other languages like C#, you can go through this article on scraping dynamic web pages using Selenium and C#.